One Packet, Then Silence: Debugging PAwR on nRF54L15

Result

I was moving the access point (AP) of an ESL-style sensor network from a working nRF5340 / Thingy:91 X gateway to an nRF54L15. Every step the AP could report on succeeded: pairing, GATT configuration, sync transfer, subevent data. The tags still never answered. Three days of testing in May (10–12 May 2026) moved the fault from "the ESL app doesn't respond" to the receiving controller, with a sharp interval boundary. In October I found what sits behind it:

Why this was hard to see

The tags are sensor nodes that sleep on a Periodic Advertising with Responses (PAwR) train. Each periodic event carries one or more subevents. The AP puts addressed commands into a subevent, and the addressed tag answers in a numbered response slot shortly after. Tags don't scan for the train. The AP connects to each tag, writes the ESL configuration over GATT and hands over the sync with PAST (Periodic Advertising Sync Transfer), and then the connection is closed.

None of the three sources is enough by itself. An API call that returns success on the AP says nothing about timing on air. Packets on air don't prove the tag accepted them, and a quiet tag log doesn't prove the AP stopped sending. Every conclusion below is checked against at least two of the three sources.

One packet, then silence

On the AP everything returned success: the GATT writes (err=0x00), PAST (#PAST_TRANSFER:ok) and every call to bt_le_per_adv_set_subevent_data(). The tag got a sync callback with the right parameters and then received either exactly one periodic event or none.

At first the tag log just went quiet. Enabling CONFIG_BT_PER_ADV_SYNC_BUF_SIZE=64 on the tag makes Zephyr call recv_cb() with buf=NULL when the controller fails to receive a periodic event. With that change, the silence became an explicit controller report for every event (rssi=127 means no packet was received). One more trap: the AP registry reported synced=1 for the tag, but that flag is set when PAST is sent, not when a response arrives. It had nothing to do with the tag's real state.

# tag RTT, run 20260511_181151Z (abridged)
#SYNC_CB:idx=0,sid=8,phy=1,int=1408,num=1,sub_int=11,rsd=8,rss=3,svc=0x1857
#PAWR_RX:122855 sub=0 ev=90
#PAWR_RX_FAIL:ev=91,…,rssi=127
…  ev=92 … ev=114, all rssi=127
#PAWR_RX_FAIL:ev=115,…,rssi=127
#PAWR_SILENT:45000 ms - recovering
#SYNC:0
#STATE:UNSYNC

Making the RF evidence trustworthy

The first question was whether the AP transmitted anything after PAST. Answering it turned into debugging the measurement first. These four steps changed what the captures could be trusted for.

1. CRC-bad frames only look like evidence. The first correlated run (115809Z, 420 s, 73,751 records) seemed to contain SID 8 and SyncInfo. Every one of those frames had failed CRC. Filtering strictly to CRC-good records left no AP evidence at all, so the run was inconclusive rather than proof of a silent AP. From then on, only CRC-good records count.
2. Validate against a known source. I wrote a small nRF54L15 signal generator (SID 8, 2M secondary, SyncInfo, PAwR data, service UUID 0x1857). It exposed a bug in the probe's parser. It masked the PDU length byte with the legacy 6-bit mask, but extended advertising PDUs use all 8 bits. A 71-byte packet (0x47) was parsed as 7 bytes (0x07), and SyncInfo was cut off with it. After the fix, the probe passed all five checks on the known source: 1,094 CRC-good target records, of which 26 carried SyncInfo.
3. An AP-only capture found nothing, for a reason. With the tag out of the loop, the sniffer still saw no SID 8. The cause was that the AP stops its extended advertising once PAwR starts. Keeping it running gave 31 CRC-good primary packets but no secondary ones: at the production cadence (14080 × 0.625 ms = 8.8 s), the host-driven AuxPtr follow had too few chances. A diagnostic cadence of 100 ms gave 2,502 CRC-good target records, 57 of them SyncInfo.
4. Gate every run. From then on, each correlated run started with a 90 s AP-only capture that had to show CRC-good SID 8 before the main run could start. Before each run, AP bonds, AP tag storage and the tag (chip erase) were all cleared. Clearing the bonds on both sides was not enough: the AP's separate tag storage still held the old ESL address and key, and pairing failed with BT_SECURITY_ERR_PIN_OR_KEY_MISSING.

# host_spi/pawr_probe.py: the length fix
def adv_payload_length(pdu):
    encoded = pdu[1]
    masked = encoded & 0x3F          # legacy PDUs: 6 bits
    if encoded != masked and len(pdu) >= 2 + encoded:
        return encoded              # extended: full byte fits
    return masked

# only CRC-good target packets may retune a radio
should_follow = crc_ok and secondary_workers and (
    args.follow_all_aux or bool(reasons))
if should_follow and host_phy is not None:
    worker = secondary_workers[follow_index % len(secondary_workers)]
    follow_index += 1
    worker.follow(aux["channel"], host_phy,
                  args.follow_hold_ms / 1000.0)

The second guard matters as much as the first. A CRC-bad AuxPtr sends a radio to the wrong channel at the wrong time, and the capture that follows looks plausible but is wrong.

Limits of the sniffer as used here: it proves CRC-good packets with channel, RSSI and decoded AuxPtr/SyncInfo fields. Timestamps are per radio and not calibrated across radios, and the direct access-address mode does not predict which channel each event should use. Details are in Sniffer method and limits.

Ruling things out, one variable per run

Each of these runs used the gated setup above, with one change at a time.

One earlier observation pointed the same way. In a previous attempt, with the nRF54L15 on the tag side and the old nRF5340 gateway as AP, archived logs show the same signature: one #PAWR_RX, then repeated receive failures and HCI 0x2084 status 0x3a. One archived run in that role did sustain responses, so this is not a blanket incompatibility.

A reproducer without the product

The unmodified Zephyr periodic_adv_rsp / periodic_sync_rsp samples on the same two boards and the same NCS v2.8.0 worked: 6 PAST handovers and 176 responses in 150 s. So the hardware and controller can do PAwR with PAST. I forked the samples into isolated diagnostics and changed one parameter group per run, moving them towards the ESL profile. The pass/fail signal is AP-side responses, because the tag board had no usable console in this setup.

#DIAG_ADV_PROFILE:sid=8,interval=303,num_subevents=1,
  subevent_interval=48,rsp_delay=5,rsp_spacing=80,
  rsp_slots=5,ext_adv_interval=160,phy=1M

# diagnostics/pawr_esl_profile_adv/CMakeLists.txt
foreach(diag_define
    DIAG_NUM_RSP_SLOTS
    DIAG_NUM_SUBEVENTS
    DIAG_PAWR_SID
    DIAG_EXT_ADV_INTERVAL
    DIAG_PAWR_INTERVAL
    DIAG_PAWR_SUBEVENT_INTERVAL
    DIAG_PAWR_RSP_SLOT_DELAY
    DIAG_PAWR_RSP_SLOT_SPACING
    DIAG_PAWR_FORCE_1M
)
  if(DEFINED ${diag_define})
    target_compile_definitions(app PRIVATE
        ${diag_define}=${${diag_define}})
  endif()
endforeach()

"Data sets" are successful bt_le_per_adv_set_subevent_data() calls; runs differ in length, so compare pass/fail rather than raw counts.

Interval sweep, one subevent, wide slots

Back in the product at interval 288

The sweep gave a working region. Interval 288 (360 ms) is below the cut-off, with margin from the weaker 300/302 runs. Following the subevent result, I moved the first tag to group/subevent 1. With that profile, the full ESL product got its first real responses through the nRF54L15 AP: #PAWR_RSP, #NUS_STATUS and #NUS_SENSORS from ESL 0x0101 on subevent 1. With one subevent at the same interval, the product got none.

A clean repeat of that run with the tag's RTT attached got no responses at all, and the log shows why. The controller delivers subevent 0 first. The tag rejects it as the wrong group and asks to switch to subevent 1, and the controller refuses:

# tag RTT, clean group-1 repeat (logs/deep_esl_group1_acl/20260512_rtt01)
#SYNC_CB:idx=0,sid=8,phy=1,int=288,num=2,sub_int=48,rsd=5,rss=80,svc=0x1857
#PAWR_SUB:sync_cb=0
#PAWR_RX:378274 sub=0 ev=1431
#PAWR_GROUP_MISMATCH:ev=1431,got=0x00,want=0x01
opcode 0x2084 status 0x3a          LE Set Periodic Advertising Sync Subevent → Controller Busy
#PAWR_SUB:retry1 err=-5 sched=360 force=0
#PAWR_RX_FAIL:sub=1,rssi=127,state=3   … repeated until #PAWR_SILENT:45000 ms

I then tested two changes in isolated copies. In the first, the tag requested subevent 1 directly in sync_cb; the call returned 0, but the controller still delivered subevent 0 first. In the second, the AP re-sent the response request up to 64 times (#PAWR_RETRY:sub=1,left=63…). Neither changed the outcome: the tag never received subevent 1. That points below the application again, at how the controller picks up a subevent after PAST.

October: it was the clock

I came back to it in October with a new bench: an nRF5340 Audio DK as the access point (the proven gateway configuration, with only the host link changed) and a second nRF54L15 Connect Kit as the tag. The nRF54L15 showed the May signature from the receiving side: one good event after sync, then buf=NULL on every event. So the question changed from "which controller is wrong" to "what does this board do differently".

1. Remove the application and the handover. A passive probe that syncs straight from a scan, with no connection and no PAST, still got exactly one good subevent per sync: 15 syncs, 15 good events, 60 failures in 12 minutes. Getting there had its own trap: the controller only builds in PAwR receive support when both PAST features are enabled. Without them, Kconfig prints a warning, the sync comes up as plain periodic advertising, and every subevent request fails with -EIO.
2. Same code, other chip. The identical probe source on an nRF52840, listening to the same AP at the same time: 137 of 137.
3. One variable on the nRF54L15. Low-frequency clock from the RC oscillator instead of the crystal: 157 of 157. Crystal kept, but its accuracy declared as 500 ppm instead of 50: 181 of 182.
4. Put a number on it. I wrote a small meter that times the kernel clock, which the 32 kHz clock keeps through sleep, against a timer on the 32 MHz crystal, in 30-second windows with the CPU asleep. The 32 kHz crystal reads +174.17 ppm relative to the board's 32 MHz crystal. The reading follows the internal load setting (17 pF: +148.70 ppm, the 18 pF maximum: +126.03 ppm), so it is the 32 kHz crystal that is off; even the maximum setting leaves it far outside 50 ppm. The 32 MHz reference itself isn't calibrated, which is why the next step matters.
5. Read the answer out of May's data. Periodic packets have no random delay, so they sit on an exact grid, and the sniffer stamps each one with a 1 MHz timer on its own crystal. Fitting time against grid index in the May captures gives the May AP's clock: +167 to +182 ppm across 12 runs and 3 sniffer radios, a second unit with the same fault. That is relative to the sniffers' own crystals, but three independent ones agreeing within about 8 ppm leaves no room for a 170 ppm error on their side. The per-radio timestamps that May's notes listed as a limit turned out to be all this needed.

# diagnostics/lfclk_meter/src/main.c
static void read_pair(uint32_t *grtc, uint32_t *t)
{
	unsigned int key = irq_lock();

	counter_get_value(tmr, t);   /* HFXO, 16 MHz */
	*grtc = k_cycle_get_32();    /* kept by the LF clock */
	irq_unlock(key);
}
…
k_sleep(K_SECONDS(WINDOW_S));
read_pair(&g1, &t1);
double ppm = (((double)(uint32_t)(g1 - g0) /
		sys_clock_hw_cycles_per_sec()) / (dt / f) - 1.0) * 1e6;

# board file load (15.5 pF), then 17 pF
#CLK n=10 win=30s grtc_d=30007057 tmr_d=480029288 err_ppm=174.21
#CLK n=10 win=30s grtc_d=30007056 tmr_d=480041508 err_ppm=148.71
# probe, 12 min, crystal declared 50 ppm
#RAW total=75 data=15 null=60      15 syncs
# same board, accuracy declared 500 ppm
#RAW total=182 data=181 null=1     1 sync

Why it depends on the pair

A receiver widens its window by the declared accuracy of both ends: 50 + 50 ppm allows about 180 µs of drift over a 1.76 s interval. What it actually has to absorb is the real drift between the two crystals. I measured every board on the bench (Audio DK +39 ppm, nRF52840 Pro Micro +60, Connect Kit +174) and tested each pairing. The link breaks between about 110 and 135 ppm of relative drift. An unfixed nRF54L15 AP even works with the Pro Micro, because that tag's crystal runs fast too. A drift that is constant from event to event against a fixed margin also gives a step, not a slope, which fits May's abrupt 302/303 cut-off.

With CONFIG_CLOCK_CONTROL_NRF_K32SRC_500PPM=y, a deliberately conservative class (+174 ppm falls in the 151–250 ppm class, but spread over units, temperature and ageing isn't characterised yet), the nRF54L15 runs the production ESL tag service on the same gateway as the nRF52840 tags: 18 of 18 status requests answered in 2.0–3.5 s, and over a 10.3-hour soak one sync, 21,006 events received, 97 failed (0.46 %), no sync loss. The access point build for this board carries the same line.

Where it goes next

The cause is settled. What remains is cost and coverage. Two May observations are not explained by the clock alone: why five subevents rescued interval 1408, and why subevent 1 answered when subevent 0 didn't.

How the work was run: the known-good gateway was never modified, tag experiments lived in separate copies of the source, and boards were restored from saved images afterwards. Each run has its raw logs, and every number here has a log path in the notes.

Research notes

The working documents from the investigation. The May notes are unedited apart from links and a pointer to the October result.

• Root cause: the 32 kHz clock (October)
• Investigation overview (May)
• Root-cause evidence packet (support request draft)
• Chronological test log, 10–12 May
• Sniffer method and limits
• Stock Zephyr sample control
• Adapted reproducer matrix and interval sweep
• Interval-288 release-candidate trial
• Raccoon: the sniffer hardware