t4as

Research note

nRF54L15 PAwR/PAST Desync Test Log

This file records concrete test runs for the Makerdiary nRF54L15 Connect Kit AP port. The goal is to keep each attempt reproducible and separate observed facts from interpretation.

Baseline

  • Date: 2026-05-10
  • AP board: Makerdiary nRF54L15 Connect Kit, nrf54l15_connectkit/nrf54l15/cpuapp
  • AP firmware: samples_nrf54/central_esl, clean debug image after temporary LFCLK/RX-failure diagnostics were removed
  • Tag: ESL 0x0001, BLE random E9:8A:BE:0D:B2:D9, name CTrack_B2D9
  • AP config confirmed before this round:
    • CONFIG_CLOCK_CONTROL_NRF_K32SRC_XTAL=y
    • CONFIG_CLOCK_CONTROL_NRF_K32SRC_50PPM=y
    • CONFIG_ESL_CLIENT_PAWR_WQ_PRIORITY=-1
    • CONFIG_BT_CTLR_SDC_PERIODIC_ADV_RSP_RX_FAILURE_REPORTING disabled
  • Real sync proof markers remain #PAWR_RSP, #NUS_STATUS, #NUS_SENSORS, #SENSOR_REPORT, or #SENSOR_SYNCED. #SENSOR_PASTED means PAST/probe-ready only.

2026-05-10 Tag Power-Cycle Retest

User power-cycled the tag before this test round.

Linux advertising scan

Command:

python3 scripts/ble_ctrack_scan.py --timeout 60 --name CTrack_B2D9 --address E9:8A:BE:0D:B2:D9

Result:

  • #LINUX_BLE_SCAN_DONE:found=1,target_found=1
  • Multiple sightings of CTrack_B2D9, RSSI about -41 to -56 dBm.

Interpretation:

  • Power-cycle restored connectable advertising.
  • The previous no-advertising state was tag runtime state, not scanner/AP inability to see the device.

AP PAwR diagnostic after tag power-cycle

Command:

python3 scripts/host_console_pawr_diag.py --observe 120 --linux-adv-confirm --linux-adv-timeout 90 --linux-adv-name CTrack_B2D9 --linux-adv-address E9:8A:BE:0D:B2:D9

Key observations:

  • AP scan saw and accepted the tag: seen=1,accepted=1.
  • ACL connected to E9:8A:BE:0D:B2:D9.
  • Discovery completed: #DISCOVERY: 1,0x00.
  • Auto path hit ESL address write timeout:
    • #STALE_BOND:0x0001,reason=write_timeout,recover=unbond_esl 0x0001
    • PAST guard held for about 30033 ms, return -11.
  • Image path PAST transfer succeeded:
    • #PAST_TRANSFER:ok,conn_idx=0,dt_ms=10
    • #SENSOR_PASTED:0x0001
  • ACL disconnected with reason 0x13.
  • AP registry then showed synced=1, but seen=0 and no real response markers appeared.
  • Marker summary included #PAWR_REQ: 50, #PAWR_PUSH_OK: 36, #PAWR_SKIP_PAST: 16, real_pawr_response=False.
  • Linux confirmation immediately afterward still saw the tag advertising:
    • #LINUX_BLE_SYNC_FAILED_CONFIRMATION:advertising=1
    • #DIAG_SYNC_FAIL_CONFIRMED_BY_LINUX_ADV:1

Interpretation:

  • AP can scan, connect, discover, and perform controller PAST transfer.
  • The tag does not behave as PAwR-synced afterward.
  • The registry synced=1 state is stale/optimistic in this failure mode because it was based on PAST, not a real PAwR response.
  • Continuing advertisement after PAST confirms the tag did not accept or remain in synchronized PAwR state.

AP unbond plus gateway hardware reset

Commands:

esl_c acl disconnect 0
esl_c unbond_esl 0x0001
pyocd reset --target nrf54l --frequency 4000000 --method hw

Post-reset AP state:

  • #AP_RADIO:up=8s,scan=0,scan_ms=0,pawr=1,pawr_ms=7786,acl=0
  • #SENSOR_REG:none
  • Group 000 status 2 SYNC_PUSHED len 16

Interpretation:

  • Gateway runtime state and registry were clean after reset.
  • PAwR advertiser restarted normally.

AP PAwR diagnostic after AP unbond/reset

Command:

python3 scripts/host_console_pawr_diag.py --observe 120 --linux-adv-confirm --linux-adv-timeout 90 --linux-adv-name CTrack_B2D9 --linux-adv-address E9:8A:BE:0D:B2:D9

Key observations:

  • AP again scanned, accepted, connected, and discovered the tag.
  • Auto path again failed at ESL address write:
    • #PAST_FAIL: write_esl_addr ret=-11 - will retry on next connection
    • #STALE_BOND:0x0001,reason=write_timeout,recover=unbond_esl 0x0001
    • PAST guard held for about 30032 ms, return -11.
  • Image path PAST transfer again succeeded:
    • #PAST_TRANSFER:ok,conn_idx=0,dt_ms=11
    • #SENSOR_PASTED:0x0001
  • No real PAwR response: real_pawr_response=False.
  • Linux confirmation still saw advertising:
    • #LINUX_BLE_SYNC_FAILED_CONFIRMATION:advertising=1
    • #DIAG_SYNC_FAIL_CONFIRMED_BY_LINUX_ADV:1

Interpretation:

  • Clearing the AP-side bond and resetting the gateway did not remove the failure.
  • The repeated timeout is specifically on the protected ESL configuration/association path before meaningful PAwR sync can be trusted.

Linux direct GATT and NUS probes

Read-only/harmless Linux checks after the AP failure:

  • Linux connected and discovered services successfully.
  • Services included GAP, DIS, NUS 6e400001-b5a3-f393-e0a9-e50e24dcca9e, and ESL 00001857-0000-1000-8000-00805f9b34fb.
  • Reads of normal GAP/DIS characteristics succeeded, including device name CTrack_B2D9.
  • First ESL-service readable characteristic probe failed with EOFError; the next ESL read reported service discovery no longer available.
  • Direct NUS STATUS command over Linux succeeded:
    • STATUS:UP=85s,ADV=0,CONN=0,TEMP=,BAT=0
  • AP bond_dump after the failed retest showed a bond entry for E9:8A:BE:0D:B2:D9.

Interpretation:

  • The tag’s basic BLE connection and application NUS channel are responsive.
  • The failing path is narrowed to the ESL service security/configuration path, especially around write_esl_addr/protected ESL characteristics.
  • This looks more like a BLE security/bond or tag association-state problem than an LFCLK-only or PAwR workqueue priority problem.

2026-05-10 Tag-Side Bond Wipe And Security Marker Test

The tag firmware NUS recovery command was inspected in 1wire_blesensro_prod/src/bt/services/nus_service.c. It uses the service key B7E3A591 and clears all tag-side BLE bonds with bt_unpair(BT_ID_DEFAULT, BT_ADDR_LE_ANY) before rebooting.

After explicit approval, direct Linux NUS sent:

SRVCKEY:B7E3A591

Result:

  • The tag replied SRVCKEY_OK:CLEARING_BOND_REBOOTING.
  • AP-side esl_c unbond_all and hardware reset were also run.
  • AP bond_dump was empty after reset.

Post-wipe AP security-marker run:

  • AP scanned and accepted CTrack_B2D9.
  • ACL connected: #ACL_CONNECTED:idx=0,err=0,from=1,past=0.
  • AP requested security: #SEC_REQ:idx=0,level=2,ret=0.
  • AP still loaded a stored tag entry despite empty BLE bonds:
    • #TAG0x0001,E9:8A:BE:0D:B2:D9 connect_from=1 past_needed=1
    • #RSP_KEY:0x2c041f30a4aa6b942c380b17974ab8319c302506bc7e847a
  • Security failed before MTU exchange/discovery:
    • #SEC_CHANGED:idx=0,level=1,err=2
    • Disconnect reason 0x16.

Interpretation:

  • Zephyr enum bt_security_err maps err=2 to BT_SECURITY_ERR_PIN_OR_KEY_MISSING.
  • The AP BLE bond table and AP tag-storage database had diverged: no BLE bond existed, but tag storage still identified the device as ESL 0x0001 with an old response key.
  • Clearing only BLE bonds is insufficient; AP tag storage must also be cleared when the tag-side bond database is wiped.

2026-05-10 AP Tag-Storage Cleanup Retest

AP cleanup commands:

esl_c acl disconnect 0
esl_c unbond_all
esl_c remove_all_tags
esl_c bond_dump
esl_c list_tags_storage_map
pyocd reset --target nrf54l --frequency 4000000 --method hw

Cleanup result:

  • esl_c remove_all_tags printed #REMOVETAGSDATA:1.
  • bond_dump was empty.
  • list_tags_storage_map printed #TAG_IN: with no entries.

Fresh AP scan/configure/PAST capture after cleanup:

  • AP scan accepted the tag: #SCAN_SUMMARY:stopped,seen=1,accepted=1,ignored=0.
  • Fresh pairing/security succeeded:
    • #SEC_REQ:idx=0,level=2,ret=0
    • #SEC_CHANGED:idx=0,level=2,err=0
    • #MTU_EXCHANGE:idx=0,err=0,mtu=247
  • Discovery and new-tag configure path ran:
    • #NEW_TAG_CONFIG: conn_idx=0x00
    • #DISCOVERY: 1,0x00
    • #PAST_SKIP: conn_idx=0 past_needed=0
    • #CONFIGURED: 1,0x0000,0x0001
  • Image-path PAST transfer was accepted by the controller:
    • #PAST_GUARD:set,path=image,conn_idx=0
    • #PAST_START: conn_idx=0 esl_addr=0x0001 (image path)
    • #PAST_TRANSFER:attempt=0,conn_idx=0,dt_ms=0
    • #PAST:1,00
    • #PAST_TRANSFER:ok,conn_idx=0,dt_ms=10
    • #PAST_GUARD:clear,path=image,conn_idx=0,held_ms=1529,ret=0
    • #SENSOR_PASTED:0x0001
  • AP persisted the fresh bond and tag mapping:
    • Bonded TAG E9:8A:BE:0D:B2:D9 (random)
    • #TAG_IN: 0001 E9:8A:BE:0D:B2:D9

Remaining failure after clean security/configuration:

  • Over 180 seconds after PAST, marker counts remained:
    • #PAWR_RSP: 0
    • #NUS_STATUS: 0
    • #NUS_SENSORS: 0
    • #SENSOR_REPORT: 0
    • #SENSOR_SYNCED: 0
  • Polling produced #SENSOR_POLL_FAIL:0x0001,consec=1.
  • AP registry showed synced=YES, but this was only from #SENSOR_PASTED; last_seen=0ms and no real response marker appeared.

Manual ACL disconnect after the successful PAST did not change the result:

  • Before disconnect: CONN[0x00]:Connected E9:8A:BE:0D:B2:D9 (random).
  • Disconnect produced reason 0x16.
  • Another 120 seconds of PAwR requests produced no #PAWR_RSP, no NUS response, no #SENSOR_REPORT, and no real #SENSOR_SYNCED.

Post-PAST advertising checks:

  • Linux scan for CTrack_B2D9 / E9:8A:BE:0D:B2:D9 for 60 seconds returned #LINUX_BLE_SCAN_DONE:found=0,target_found=0.
  • AP one-shot scan for 30 seconds returned #SCAN_SUMMARY:stopped,seen=0.
  • AP registry later showed synced=NO, last_seen=0ms, desync#=1.

Interpretation:

  • AP tag-storage cleanup fixed the post-wipe security failure and restored normal fresh pairing, MTU exchange, discovery, configuration, bond creation, tag storage, and controller-accepted PAST.
  • The remaining nRF54 failure is now cleanly isolated past GATT security/configuration: after a successful PAST API return, the tag stops connectable advertising but still does not produce any real PAwR response.
  • #SENSOR_PASTED/registry synced=YES must continue to be treated as optimistic AP state, not proof of sensor synchronization.

2026-05-10 Inline PAwR Data-Request Diagnostic

Firmware change under test:

  • Added default-off shared Kconfig CONFIG_ESL_PAWR_PUSH_INLINE_IN_CALLBACK.
  • Enabled it only in the nRF54 AP board config.
  • With the option enabled, pawr_request() calls bt_le_per_adv_set_subevent_data() inline through esl_c_push_sync_to_controller() instead of first scheduling pawr_workq.
  • Generated nRF54 config confirmed CONFIG_ESL_PAWR_PUSH_INLINE_IN_CALLBACK=y.

Build/flash result:

  • ./build.sh clean completed.
  • Flash programmed build_nrf54l15_connectkit/merged.hex successfully.
  • Only compile warning observed was the existing const-qualifier warning in esl_client_tag_storage.c.

Fresh retest after AP tag-storage cleanup/reset:

  • AP scan saw and accepted the tag: seen=1,accepted=1.
  • Security and MTU were clean: #SEC_CHANGED:idx=0,level=2,err=0, #MTU_EXCHANGE:idx=0,err=0,mtu=247.
  • Discovery and configuration ran: #DISCOVERY: 1,0x00, #CONFIGURED: 1,0x0000,0x0001.
  • PAST was accepted: #PAST:1,00, #PAST_TRANSFER:ok,conn_idx=0,dt_ms=10, #SENSOR_PASTED:0x0001.
  • During the 90 second diagnostic window, inline PAwR pushes matched data requests: marker summary showed #PAWR_REQ: 38, #PAWR_PUSH_OK: 38, #PAWR_SKIP_PAST: 2.
  • No real response markers appeared: #PAWR_RSP: 0, #NUS_STATUS: 0, #NUS_SENSORS: 0, #SENSOR_REPORT: 0, #SENSOR_SYNCED: 0.

Longer observation after a subsequent fresh clean run:

  • AP connected, security succeeded, configured ESL 0x0001, transferred PAST, then kept ACL as configured.
  • AP stayed at acl=1 until manual disconnect; another 120 seconds after manual disconnect still produced no #PAWR_RSP, no NUS response, no sensor report, and no real sync marker.
  • Linux scan after the post-PAST failure found no CTrack_B2D9 in 60 seconds.
  • AP one-shot scan after that also found no tag: #SCAN_SUMMARY:stopped,seen=0.

Interpretation:

  • The PAwR workqueue hop is not the root cause. nRF54 accepts inline subevent data on each request, but the tag still never produces an occupied response slot.
  • The failure remains after successful BLE security, MTU exchange, ESL configuration, PAST API success, and inline PAwR subevent-data submission.
  • The strongest next discriminator is tag-side evidence: whether the tag receives PAST, creates PA sync, enables PAwR response, selects the same subevent/slot as the AP, and retains the expected ESL address/response key.

Follow-up diagnostic prepared:

  • scripts/host_console_pawr_diag.py now has --ping-after-past.
  • When enabled, the script stops observing at #SENSOR_PASTED, waits briefly, runs esl_c acl list_conn, then runs esl_c ping 0 over the still-open ACL link.
  • Expected discriminator:
    • If the ping response prints #BASIC_STATE with SYNCHRONIZED:off, the tag did not enter PA sync after PAST.
    • If SYNCHRONIZED:on appears but PAwR responses remain absent, the next suspect is tag PAwR subevent/response-slot subscription or PA receive delivery.
    • If the ACL is already disconnected, that itself suggests the tag entered SM_SYNCHRONIZED and called its disconnect path; tag-side logs should then focus on missing #PAWR_RX / #PAWR_RSP.

2026-05-11 Gemini Research Hypothesis Matrix

Source hypothesis document:

  • Researchnotes, focused on nRF54L15 single-core SDC timing, PAST/PAwR overlap, SDC buffer capacity, and PAwR callback timing.

Precondition from the latest fixed AP path:

  • The discovery continuation workqueue bug is fixed: GATT config callbacks now arrive cleanly.
  • Auto config writes complete for handles 0x001f, 0x0025, 0x0021, 0x0023, and 0x002b.
  • Auto PAST succeeds in the clean baseline: #PAST_TRANSFER:ok,conn_idx=0,dt_ms=10 and #SENSOR_PASTED:0x0001.
  • User-provided tag log after the queue fix proved the tag can reach #SYNC:1 and receive one PAwR payload: #PAWR_RX:... sub=0 ev=6, then later #PAWR_SILENT:45000 ms.

Hypotheses tested:

  1. Defer PAwR subevent data instead of calling bt_le_per_adv_set_subevent_data() inline from pawr_request().
  • Test config: CONFIG_ESL_PAWR_PUSH_INLINE_IN_CALLBACK=n, ACL keep still enabled, pre-PAST delay 6000 ms.
  • Result: PAST succeeded, #PAWR_REQ: 56, #PAWR_PUSH_OK: 56, no #PAWR_RSP, and Linux found CTrack_B2D9 advertising afterward.
  • Verdict: disproven as a standalone fix on this NCS v2.8.0 AP. The workqueue path still accepts pushes but does not sustain real PAwR reception.
  1. Remove post-PAST ACL overlap.
  • Test config: restored inline push, CONFIG_ESL_PAST_KEEP_ACL_AFTER_TRANSFER=n, CONFIG_ESL_PAST_POST_TRANSFER_DISCONNECT_DELAY_MS=2500.
  • Result: PAST succeeded, AP emitted #PAST_DONE_DISCONNECT, disconnect reason 0x16, #PAWR_REQ: 58, #PAWR_PUSH_OK: 58, no real response, and Linux found the tag advertising afterward.
  • Verdict: disproven as a standalone fix. It is still cleaner for scaling/production, but it does not fix the one-RX-then-silence failure.
  1. Relax PAwR timing using the exact research values.
  • Test config: CONFIG_ESL_SUBEVENT_INTERVAL=16, CONFIG_ESL_RESPONSE_SLOT_DELAY=20.
  • Result: invalid timing combination for this controller/config. The AP booted with pawr=0; PAST failed with #PAST:0,00,-5 in both auto and image paths.
  • Verdict: exact suggested values are invalid here because response-slot delay 20 is 25 ms while subevent interval 16 is 20 ms.
  1. Relax PAwR timing with a valid widened interval.
  • Test config: CONFIG_ESL_SUBEVENT_INTERVAL=32, CONFIG_ESL_RESPONSE_SLOT_DELAY=20, ACL keep disabled.
  • Result: PAwR started, PAST succeeded, #PAWR_REQ: 58, #PAWR_PUSH_OK: 58, no real response, and Linux found the tag advertising afterward.
  • Verdict: disproven as a standalone fix.
  1. Increase SDC/host buffer capacity.
  • First build attempt used research value CONFIG_BT_CTLR_SDC_PERIODIC_ADV_RSP_TX_MAX_DATA_SIZE=255.
  • Result: Kconfig rejected 255; NCS v2.8.0 active range is [1, 249], so the valid maximum is 249.
  • Retest config: CONFIG_BT_CONN_TX_MAX=16, CONFIG_BT_BUF_ACL_TX_COUNT=16, CONFIG_BT_CTLR_SDC_PERIODIC_ADV_RSP_TX_MAX_DATA_SIZE=249, CONFIG_BT_CTLR_SDC_PERIODIC_ADV_RSP_RX_BUFFER_COUNT=24, default PAwR timing restored.
  • Result: PAST succeeded, #PAWR_REQ: 58, #PAWR_PUSH_OK: 56, no real response, and Linux found the tag advertising afterward.
  • Verdict: disproven as a standalone fix. The specific 255 value is invalid in this SDK; the maximum valid 249 plus larger queues did not fix the failure.
  1. Send an addressed PAwR command immediately after PAST.
  • Test command: host_console_pawr_diag.py --ping-after-past on the buffer-capacity build.
  • Result: PAST succeeded, the script issued esl_c ping 0 after #SENSOR_PASTED, PAwR pushes continued, but no #PAWR_RSP or #BASIC_STATE response arrived. Linux again found the tag advertising afterward.
  • Verdict: the failure is not explained only by the AP sending broadcast NOPs. An addressed ping after PAST still did not get a response.

Clean baseline restored/flashed after tests:

  • CONFIG_ESL_PAWR_PUSH_INLINE_IN_CALLBACK=y
  • CONFIG_ESL_SUBEVENT_INTERVAL=11
  • CONFIG_ESL_RESPONSE_SLOT_DELAY=8
  • CONFIG_ESL_PAST_PRE_TRANSFER_DELAY_MS=6000
  • CONFIG_ESL_PAST_POST_TRANSFER_DISCONNECT_DELAY_MS=2500
  • CONFIG_ESL_PAST_KEEP_ACL_AFTER_TRANSFER=n
  • CONFIG_BT_CONN_TX_MAX=6
  • CONFIG_BT_BUF_ACL_TX_COUNT=6
  • CONFIG_BT_CTLR_SDC_PERIODIC_ADV_RSP_TX_MAX_DATA_SIZE=73
  • CONFIG_BT_CTLR_SDC_PERIODIC_ADV_RSP_RX_BUFFER_COUNT=12

Interpretation:

  • The broad research claim remains plausible: HCI/API success does not prove on-air PAwR subevent delivery on nRF54L15 SDC v2.8.0.
  • The concrete mitigations tested today do not resolve the issue on this codebase and hardware.
  • num_events 1 in the tag sync log is not suspicious for the single-group ESL setup; it aligns with one PAwR subevent.
  • The next useful evidence needs to come from either tag-side PAwR logs during the same run, a minimal Zephyr PAwR/PAST reproducer, or an NCS 3.x controller comparison.

2026-05-11 Correlated AP Payload Trace And Tag RTT Run

Instrumentation added before this run:

  • AP payload staging/push markers in service/esl_client.c:
    • #PAWR_STAGE:... plain_len=...,enc_len=...,op=...,addr=...,resp=...,nop_overwrite=...,seq=...
    • #PAWR_PUSH_DETAIL:... len=...,op=...,addr=...,rsp=...,ead=...,seq=...
  • scripts/host_console_pawr_diag.py now prints those markers and has --nus-status-after-past to force a unicast NUS-over-PAwR status request after #SENSOR_PASTED.
  • scripts/tag_rtt_collector.py was added for continuous tag RTT collection through J-Link serial 684638462.

Validation before the hardware run:

  • python3 -m py_compile scripts/tag_rtt_collector.py scripts/host_console_pawr_diag.py passed.
  • git diff --check passed earlier for the AP trace/collector changes.
  • nRF54 AP build succeeded and was flashed with pyOCD. App RAM was about 89.12%; the only build warning was the existing esl_client_tag_storage.c const discard.
  • RTT collector smoke test attached to the tag and captured live #W1_BEGIN/#W1_END markers.

Correlated capture commands:

python3 scripts/tag_rtt_collector.py --serial 684638462 --duration 240 --quiet --summary-every 30 --out logs/tag_rtt/nrf54_trace_20260511_110203.log

python3 scripts/host_console_pawr_diag.py --port /dev/ttyACM0 --observe 150 --nus-status-after-past --target-esl-addr 0x0001 --linux-adv-confirm --linux-adv-timeout 45 --linux-adv-name CTrack_B2D9 --linux-adv-address E9:8A:BE:0D:B2:D9

AP-side observations:

  • AP started from a clean state: PAwR active, no ACL, no sensor registry entry.
  • Scan saw and accepted one tag, then connected to E9:8A:BE:0D:B2:D9.
  • GATT config writes completed cleanly for the ESL config path.
  • Basic State before PAST showed SERVICE_NEEDED:on, SYNCHRONIZED:off, ACTIVE_LED:off.
  • PAST transfer succeeded:
    • #PAST_TRANSFER:ok,conn_idx=0,dt_ms=10
    • #SENSOR_PASTED:0x0001
  • Before the forced NUS command, AP pushed broadcast/NOP payloads:
    • #PAWR_PUSH_DETAIL:sub=0,len=16,op=0x00,addr=0xff,rsp=0,...
  • The forced esl_c nus status 0x0001 staged a unicast response-producing payload:
    • #PAWR_STAGE:grp=0,plain_len=4,enc_len=17,op=0x1f,addr=0x01,rsp=1,nop_overwrite=0,seq=1
  • The controller accepted that addressed payload on the PAwR request path:
    • #PAWR_PUSH_DETAIL:sub=0,len=17,op=0x1f,addr=0x01,rsp=1,ead=1031eb8b,seq=100
  • Afterward, the AP repeatedly staged/pushed additional addressed response-producing payloads (addr=0x01,rsp=1, op 0x10) but still never received a real response.
  • Marker summary from the AP run:
    • #PAWR_STAGE: 14
    • #PAWR_PUSH_DETAIL: 64
    • #PAWR_REQ: 62
    • #PAWR_PUSH_OK: 64
    • real_pawr_response=False
  • End-state AP registry was optimistic only: #SENSOR_REG:0,addr=0x0001,synced=1,...,seen=0,....
  • Linux BLE confirmation immediately afterward found the tag advertising again:
    • #LINUX_BLE_SYNC_FAILED_CONFIRMATION:advertising=1
    • #DIAG_SYNC_FAIL_CONFIRMED_BY_LINUX_ADV:1

Tag RTT observations from logs/tag_rtt/nrf54_trace_20260511_110203.log:

  • Tag connected, secured, accepted ESL address 0x0001, and entered configured state.
  • Tag subscribed to PAST/PAwR:
    • #PAST_SUB:start
    • #PAST_SUB:done
    • #PAWR_SUB:sync_cb=0
    • #SYNC:1
  • The tag received exactly one PAwR event immediately after sync:
    • #PAWR_RX:2339304 sub=0 ev=99
  • That received event decoded as broadcast/no-addressed TLV:
    • All broadcast TLV or no addressed TLV, no need replied
    • tlv counts 0 rsp_slot -1
  • The tag did not log any later #PAWR_RX for the AP’s addressed addr=0x01,rsp=1 pushes.
  • After 45 seconds without another PAwR receive, the tag recovered from silence:
    • #PAWR_SILENT:45001 ms - recovering
    • #SYNC:0
    • #STATE:UNSYNC
  • RTT collector final summaries matched the file: one connection/security/configure, one #PAWR_RX, one #PAWR_SILENT, no #CONFIGURED_TIMEOUT in this run.

Interpretation:

  • This run answers the main software-trace question: the AP is not merely sending broadcast NOPs. It stages and pushes addressed, response-producing PAwR payloads for ESL low address 0x01, and bt_le_per_adv_set_subevent_data() accepts them.
  • The tag proves it can sync once and receive the first PAwR event, but after that it receives no further PAwR events at all, including the AP’s addressed pushes.
  • The failure is now most consistent with nRF54 controller/on-air PAwR delivery after PAST, periodic advertising anchor/continuation, or an incompatibility around PAwR subevent data updates after the initial event. It is less consistent with an AP application bug that never stages unicast payloads.
  • The 7-channel synchronized sniffer system becomes useful at this point, but it should be aimed narrowly: verify whether the AP actually transmits the PAwR train after event 99 and whether the addressed EAD-bearing payload appears on air after the #PAWR_PUSH_DETAIL len=17,addr=0x01,rsp=1 markers.

2026-05-11 Seven-Sniffer Correlated PAwR Run

Run ID: 20260511_115809Z.

Artifacts saved locally:

  • logs/correlated_pawr/20260511_115809Z/ap_diag.log
  • logs/correlated_pawr/20260511_115809Z/tag_rtt.log
  • logs/correlated_pawr/20260511_115809Z/sniffer/sniffer.jsonl
  • logs/correlated_pawr/20260511_115809Z/sniffer/sniffer_summary.json
  • logs/correlated_pawr/20260511_115809Z/analysis_report.md

Pi-side original sniffer artifacts:

  • ~/raccoon/captures/correlated_20260511_115809Z/sniffer.jsonl
  • ~/raccoon/captures/correlated_20260511_115809Z/sniffer_summary.json

Commands used:

# Pi, 7-device raccoon sniffer
python3 host_spi/pawr_probe.py --duration 420 --secondary-phy 2m --output-prefix captures/correlated_20260511_115809Z/sniffer --save-all

# Local tag RTT via J-Link 684638462
python3 scripts/tag_rtt_collector.py --serial 684638462 --device nRF52840_xxAA --speed 4000 --duration 420 --out logs/correlated_pawr/20260511_115809Z/tag_rtt.log --summary-every 60

# Local nRF54 AP host console on /dev/ttyACM0
python3 scripts/host_console_pawr_diag.py --port /dev/ttyACM0 --observe 180 --ping-after-past --nus-status-after-past --target-esl-addr 0x0001 --linux-adv-confirm --linux-adv-timeout 90 --linux-adv-name CTrack_B2D9 --linux-adv-address E9:8A:BE:0D:B2:D9

AP-side observations:

  • AP started with PAwR already running and no ACL. Registry had ESL 0x0001 with synced=0, seen=0, desync=15.
  • AP scan saw and accepted the tag, connected to E9:8A:BE:0D:B2:D9, completed discovery, and completed the protected ESL config writes.
  • PAST transfer succeeded: #PAST_TRANSFER:ok,conn_idx=0,dt_ms=10, followed by #SENSOR_PASTED:0x0001.
  • Before the forced unicast command, AP pushed broadcast NOP payloads (op=0x00,addr=0xff,rsp=0,len=16).
  • Forced NUS status staged and pushed an addressed response-producing PAwR payload:
    • #PAWR_STAGE:... op=0x1f,addr=0x01,rsp=1,seq=182
    • #PAWR_PUSH_DETAIL:... len=17,op=0x1f,addr=0x01,rsp=1,seq=6144
  • Subsequent addressed ping/status traffic also staged and pushed response-producing payloads (op=0x10,addr=0x01,rsp=1,len=17).
  • No real response markers appeared: no #PAWR_RSP, no #NUS_STATUS, and no #SENSOR_REPORT; diagnostic summary ended with real_pawr_response=False.
  • End-state AP registry again showed optimistic synced=1 with seen=0.
  • Linux BLE confirmation found the tag advertising after the failure: #DIAG_SYNC_FAIL_CONFIRMED_BY_LINUX_ADV:1.

Tag RTT observations:

  • Tag connected to AP EF:D8:FF:83:BD:0C, security reached level 2, ESL address 0x0001 was written, and PAST subscription completed.
  • Tag created PA sync to the nRF54 AP:
    • AP address: 04:7B:F1:93:33:D5 (random)
    • interval: 0x0580 / 1408 units / about 1.76 s
    • PHY: 2
    • SID: 8
    • service data: 0x1857
    • num_events 1, sub_int 11, resp delay 8, resp spacing 3
  • Tag logged #SYNC:1 and exactly one real PAwR receive: #PAWR_RX:878432 sub=0 ev=6151.
  • That one received event decoded as broadcast/no-addressed TLV: All broadcast TLV or no addressed TLV, no need replied, tlv counts 0 rsp_slot -1.
  • The tag did not log a receive for the AP’s addressed addr=0x01,rsp=1 payloads.
  • After 45 seconds with no further PAwR receive, the tag recovered: #PAWR_SILENT:45000 ms - recovering, then #SYNC:0 and #STATE:UNSYNC.
  • Final RTT collector summary for the actual run was one connection/security sequence, one #PAWR_RX, one #PAWR_SILENT, no #PAWR_GAP, and no #CONFIGURED_TIMEOUT.

Sniffer observations:

  • Seven-device sniffer ran the full 420 seconds and saved 73751 records, including 969 ADV_EXT_IND records.
  • Summary counters: aux_ptr_count=954, aux_ptr_2m_count=2, sync_info_count=3, secondary_crc_ok_count=0.
  • A strict CRC-good filter found 896 CRC-good extended advertising records, but only SID 1 (851) and SID 0 (45).
  • Strict CRC-good evidence found no SID 8, no SyncInfo, no 2M AuxPtr, no secondary packets, and no raw AP address hit for 04:7B:F1:93:33:D5.
  • The many CRC-good 0x1857 service-data hits were normal tag connectable advertisements (ADV_IND, name CTrack_B2D9), not AP PAwR extended advertising.
  • Some decoded SID 8/SyncInfo examples existed in the raw report, but all were CRC-failed frames and must be treated as noise/misdecode.

Interpretation:

  • This run cleanly reproduced the core failure with AP, tag, and sniffer all recording at the same time.
  • AP-side application logic is still proven to stage and controller-push addressed response-producing payloads after PAST.
  • Tag-side logic is still proven to create PA sync and receive exactly one initial PAwR event, then receive no further PAwR events until the silence watchdog fires.
  • The seven-sniffer sweep did not capture CRC-good AP PAwR/SID-8 evidence in this run. That makes the sniffer result inconclusive for AP on-air PAwR delivery, not proof that the AP is silent.
  • The next sniffer iteration needs either a true PAwR follower with valid SyncInfo/access-address/channel-map timing, or a simpler AP PHY/setup that the current sweep can reliably catch.

2026-05-11 Controlled nRF54 Sniffer Validation

Follow-up work built a deterministic nRF54L15 signal generator at samples_nrf54/sniffer_signal and improved the racoon host_spi/pawr_probe.py path before returning to the real ESL AP.

Validation setup:

  • Signal source: Makerdiary nRF54L15 Connect Kit running samples_nrf54/sniffer_signal.
  • Advertiser profile: SID 8, service UUID marker 0x1857, primary LE 1M extended advertising, secondary LE 2M, periodic advertising SyncInfo, and PAwR subevent data.
  • Sniffer command path: run_validation.py deployed the current pawr_probe.py to the Pi and ran with --follow-aux --follow-hold-ms 120 --secondary-phy 2m.

Issues fixed during validation:

  • The signal generator initially stopped before advertising because bt_le_per_adv_set_data() returned -13 on the PAwR advertiser path. The generator now leaves normal periodic advertising data unset and supplies PAwR payloads through bt_le_per_adv_set_subevent_data() from the data-request callback.
  • The sniffer initially missed CRC-good SyncInfo and service UUID evidence because secondary extended advertising length byte 0x47 was incorrectly masked to 0x07. The parser now accepts the valid raw captured length for secondary PDU payloads and preserves remaining extended-header bytes as ACAD.

Known-good 2M validation run:

  • Artifact directory: samples_nrf54/central_esl/logs/sniffer_validation/20260511_125546Z/
  • crc_good.ext_count = 1194
  • crc_good.sid_counts = {"8": 1094, "1": 97, "0": 3}
  • crc_good.sync_info_count = 26
  • crc_good.aux_ptr_count = 1168
  • crc_good.aux_ptr_2m_count = 1068
  • crc_good.secondary_count = 26
  • target_crc_good.total = 1094
  • target_crc_good.by_reason = {"sid": 1094, "service_uuid": 26}
  • follow_aux_requests = 1068

Validation gates all passed:

  • CRC-good target SID evidence
  • CRC-good AuxPtr evidence
  • CRC-good secondary-channel evidence
  • CRC-good SyncInfo evidence
  • CRC-good service UUID marker evidence

Interpretation:

  • The seven-device racoon sniffer path is now proven against a controlled nRF54 source for the exact evidence needed before ESL AP conclusions: SID 8, 2M AuxPtr follow, CRC-good secondary decode, SyncInfo, and service UUID marker.
  • Future ESL AP captures should use the validated probe and continue treating only CRC-good summary fields as proof.

2026-05-11 Validated-Sniffer Real nRF54 ESL AP Retest

Run ID: 20260511_131239Z.

Artifacts saved locally:

  • logs/correlated_pawr/20260511_131239Z/ap_diag.log
  • logs/correlated_pawr/20260511_131239Z/tag_rtt.log
  • logs/correlated_pawr/20260511_131239Z/sniffer/sniffer.jsonl
  • logs/correlated_pawr/20260511_131239Z/sniffer/sniffer_summary.json

AP setup before the run:

  • The Makerdiary nRF54L15 board was reflashed from samples_nrf54/central_esl after the controlled signal-generator validation.
  • AP console confirmed PAwR was running and no tag registry was present before scanning: #AP_RADIO:...pawr=1...acl=0, #SENSOR_REG:none.

Runner command:

python3 scripts/run_correlated_ap_sniffer.py --duration 240 --ap-observe 180

The runner deployed the current pawr_probe.py, spi_host_driver.py, and device_profiles.py to the Pi, then ran the sniffer with:

python3 host_spi/pawr_probe.py --duration 240 --secondary-phy 2m --sid 8 --service-uuid 0x1857 --follow-aux --follow-hold-ms 120 --output-prefix captures/esl_ap_validated_20260511_131239Z/sniffer --save-all

AP-side observations:

  • AP scan saw and accepted the tag: #SCAN_SUMMARY:stopped,seen=1,accepted=1,ignored=0.
  • ACL connected to E9:8A:BE:0D:B2:D9, discovery completed, and protected ESL config writes all completed with err=0x00.
  • Basic State before PAST showed SERVICE_NEEDED:on, SYNCHRONIZED:off, ACTIVE_LED:off.
  • PAST transfer was accepted by the controller:
    • #PAST_TRANSFER:ok,conn_idx=0,dt_ms=11
    • #SENSOR_PASTED:0x0001
  • Forced NUS status after PAST staged an addressed response-producing payload:
    • #PAWR_STAGE:grp=0,plain_len=4,enc_len=17,op=0x1f,addr=0x01,rsp=1,...
    • #PAWR_PUSH_DETAIL:sub=0,len=17,op=0x1f,addr=0x01,rsp=1,...
  • Subsequent addressed ping/status traffic also staged and pushed response-producing payloads (op=0x10,addr=0x01,rsp=1,len=17).
  • Marker summary: #PAWR_STAGE: 14, #PAWR_PUSH_DETAIL: 76, #PAWR_REQ: 74, #PAWR_PUSH_OK: 76, real_pawr_response=False.
  • AP registry again ended optimistic only: #SENSOR_REG:0,addr=0x0001,synced=1,...,seen=0,....
  • Linux BLE confirmation found the tag advertising after the failure: #DIAG_SYNC_FAIL_CONFIRMED_BY_LINUX_ADV:1.

Tag RTT observations:

  • The RTT file starts with a buffered earlier sync sequence, including #PAWR_SUB:sync_cb=0, #SYNC:1, and AP SyncInfo for address 04:7B:F1:93:33:D5, SID 8, service data 0x1857, interval 0x0580, PHY 2.
  • The actual diagnostic attempt in this capture began a few seconds later: the tag connected, reached security level 2, accepted ESL address 0x0001, entered configured state, and called #PAST_SUB:start / #PAST_SUB:done.
  • The tag then disconnected with reason 0x13 and never logged a new PA sync callback for this diagnostic attempt.
  • After 30 seconds, the configured-state watchdog fired: SM_CONFIGURED watchdog fired - PAST sync_cb never arrived, #CONFIGURED_TIMEOUT, followed by #STATE:UNSYNC.
  • Final collector summary included #PAWR_RX=0, #PAWR_GAP=1, #PAWR_SILENT=0, and #CONFIGURED_TIMEOUT=1. The #PAWR_SUB=1 / #SYNC:=1 counts are from the buffered earlier sequence at the start of the RTT capture, not from the current AP diagnostic connection.

Sniffer observations with the validated probe:

  • The capture ran for 240 seconds and saved 40020 records.
  • Strict CRC-good extended advertising evidence was limited to SID 1 and SID 0:
    • crc_good.extended_count = 487
    • crc_good.sid_counts = {"0": 34, "1": 453}
    • crc_good.aux_ptr_count = 487
    • crc_good.aux_ptr_2m_count = 0
    • crc_good.secondary_count = 0
    • crc_good.sync_info_count = 0
  • Target evidence was completely absent: target_crc_good.total = 0, with empty by_reason, by_role, and examples.
  • AuxPtr follow did not issue any target follows: follow_aux.requests = 0, with skips not_target = 487 and crc_bad = 37.
  • The top-level raw decode had one SID 8 candidate, but direct inspection showed it was CRC-failed noise: crc_ok=false, RSSI -90 dBm, channel 38, 1M AuxPtr, and no service data.
  • The top-level raw decode also counted one 2M AuxPtr, but the strict CRC-good 2M AuxPtr count was zero.

Interpretation:

  • This run is different from the earlier one-RX-then-silence reproductions: during the actual AP diagnostic connection, the tag did not receive a fresh PAST sync callback and timed out in configured state.
  • AP-side control flow still reports success for scan, GATT configuration, PAST transfer, and addressed PAwR subevent-data pushes.
  • The validated sniffer did not see any CRC-good real-AP SID 8, SyncInfo, service UUID 0x1857, 2M AuxPtr, or secondary-channel evidence during the same capture.
  • Therefore, for this specific run, the highest-value conclusion is that AP/API success still does not prove an observable on-air SID 8 PAwR chain. The current failure presented as PAST sync creation/delivery failure on the tag, with the sniffer also failing to find the expected AP advertising evidence.
  • Because the tag RTT file also contains a buffered earlier successful sync to AP SID 8, the AP may be intermittent or state-dependent rather than permanently unable to advertise. The next test should deliberately split AP radio-advertising proof from tag PAST behavior.

2026-05-11 AP-Only Validated Sniffer Preflight

Run ID: 20260511_132823Z.

Artifacts saved locally:

  • logs/correlated_pawr/20260511_132823Z/preflight/ap_status.log
  • logs/correlated_pawr/20260511_132823Z/preflight/sniffer.jsonl
  • logs/correlated_pawr/20260511_132823Z/preflight/sniffer_summary.json
  • logs/correlated_pawr/20260511_132823Z/preflight/sniffer_stdout.log

Purpose:

  • Prove or disprove AP idle radio evidence without triggering the tag association path.
  • This test intentionally does not run sensor_mgmt mode open, esl_c acl scan, tag RTT collection, or the AP PAST diagnostic.

Command:

python3 scripts/run_correlated_ap_sniffer.py --preflight-duration 90 --preflight-only

AP status during the no-connection preflight setup:

  • #AP_RADIO:up=1281s,scan=0,scan_ms=0,pawr=1,pawr_ms=1280998,acl=0
  • #AP_RADIO:up=1281s,scan=0,scan_ms=0,pawr=1,pawr_ms=1281098,acl=0
  • Existing registry state was stale/optimistic only: #SENSOR_REG:0,addr=0x0001,synced=1,...,seen=0,desync=2,name=?.
  • No tag scan/connect command was issued; the sniffer was the only active test component.

Validated-sniffer result:

  • Sniffer command used SID 8, service UUID 0x1857, secondary PHY 2m, --follow-aux, --follow-hold-ms 120, and --save-all.
  • Strict CRC-good summary:
    • crc_good.extended_count = 301
    • crc_good.sid_counts = {"0": 54, "1": 247}
    • crc_good.aux_ptr_count = 301
    • crc_good.aux_ptr_2m_count = 0
    • crc_good.secondary_count = 0
    • crc_good.sync_info_count = 0
  • Target evidence was absent again: target_crc_good.total = 0, with empty by_reason and by_role.

Interpretation:

  • The user hypothesis was correct: we can test AP PAwR radio output without disabling the sensor/tag, as long as we do not mark the tag for connection by starting the AP scan/config path.
  • This AP-only preflight removes tag PAST behavior from the equation. While the AP console reported pawr=1 and no ACL/scan activity, the validated sniffer still found no CRC-good AP SID 8, service UUID 0x1857, 2M AuxPtr, secondary packet, or SyncInfo evidence.
  • This shifts the immediate focus earlier than PAST: either the AP is not actually advertising the expected SID 8 PAwR chain despite pawr=1, the AP parameters differ from the expected SID/service/PHY, or the real AP RF placement/power/channel behavior differs from the controlled nRF54 signal generator in a way the current sniffer still does not capture.
  • The next discriminator should be a direct AP console/firmware audit of the extended/periodic advertising parameters and, if those are correct, a forced-1M nRF54 AP build to remove the 2M secondary path from the failure.

2026-05-11 AP-Only Preflight With Extended Advertising Kept Running

Run ID: 20260511_134248Z.

Firmware change under test:

  • nRF54 AP board config enabled CONFIG_ESL_PAWR_KEEP_EXT_ADV_RUNNING=y.
  • nRF54 AP board config enabled CONFIG_ESL_PAWR_START_PERIODIC_BEFORE_EXT_ADV=y.
  • The AP extended advertising interval was still the legacy 0x3700 value in this run.

Artifacts saved locally:

  • logs/correlated_pawr/20260511_134248Z/preflight/ap_status.log
  • logs/correlated_pawr/20260511_134248Z/preflight/sniffer.jsonl
  • logs/correlated_pawr/20260511_134248Z/preflight/sniffer_summary.json
  • logs/correlated_pawr/20260511_134248Z/preflight/sniffer_stdout.log

AP status during the no-connection preflight setup:

  • #AP_RADIO:up=57s,scan=0,scan_ms=0,pawr=1,pawr_ms=56255,acl=0
  • #AP_RADIO:up=57s,scan=0,scan_ms=0,pawr=1,pawr_ms=56356,acl=0
  • #SENSOR_REG:none
  • No tag scan/connect command was issued; the sniffer was the only active test component.

Validated-sniffer result:

  • Strict CRC-good summary:
    • crc_good.extended_count = 338
    • crc_good.sid_counts = {"0": 59, "1": 248, "8": 31}
    • crc_good.aux_ptr_count = 338
    • crc_good.aux_ptr_2m_count = 31
    • crc_good.secondary_count = 0
    • crc_good.sync_info_count = 0
  • Target evidence was present on primary only: target_crc_good.total = 31, target_crc_good.by_reason = {"sid": 31}, target_crc_good.by_role = {"primary": 31}.
  • Target examples show AP address 0C:2C:66:22:DA:2E, SID 8, DID 3643, and 2M AuxPtr offsets of 510, 810, and 1110 us.

Interpretation:

  • Keeping the PAwR extended advertiser running restored CRC-good AP SID 8 primary evidence without any tag interaction.
  • Therefore, the earlier 20260511_132823Z no-target preflight should not be read as proof that the AP had no PAwR radio chain. It mostly proved that the default AP implementation stops the extended-advertising breadcrumb after PAwR starts.
  • The remaining gap after this run was secondary/SyncInfo capture. The AP was emitting target primary extended advertisements with 2M AuxPtr, but the short AuxPtr offsets and slow 0x3700 primary cadence gave the host-driven follow path too few chances to catch secondary packets.

2026-05-11 AP-Only Preflight With Short Diagnostic Ext-Adv Interval

Run ID: 20260511_140333Z.

Firmware change under test:

  • Added shared Kconfig CONFIG_ESL_PAWR_EXT_ADV_INTERVAL, default 14080 to preserve the legacy 0x3700 behavior.
  • service/esl_client.c now uses CONFIG_ESL_PAWR_EXT_ADV_INTERVAL for the AP PAwR extended advertiser.
  • nRF54 AP board config overrides it to CONFIG_ESL_PAWR_EXT_ADV_INTERVAL=160, matching the validated nRF54 signal generator cadence.
  • Generated config confirmed:
    • CONFIG_ESL_PAWR_EXT_ADV_INTERVAL=160
    • CONFIG_ESL_PAWR_KEEP_EXT_ADV_RUNNING=y
    • CONFIG_ESL_PAWR_START_PERIODIC_BEFORE_EXT_ADV=y

Build/flash result:

  • ./build.sh clean completed.
  • Flash programmed build_nrf54l15_connectkit/merged.hex successfully using pyOCD UID 820D9A5F0E1231A458F5D.
  • Only compile warning observed was the existing const-qualifier warning in esl_client_tag_storage.c.

Artifacts saved locally:

  • logs/correlated_pawr/20260511_140333Z/preflight/ap_status.log
  • logs/correlated_pawr/20260511_140333Z/preflight/sniffer.jsonl
  • logs/correlated_pawr/20260511_140333Z/preflight/sniffer_summary.json
  • logs/correlated_pawr/20260511_140333Z/preflight/sniffer_stdout.log

AP status during the no-connection preflight setup:

  • #AP_RADIO:up=44s,scan=0,scan_ms=0,pawr=1,pawr_ms=43483,acl=0
  • #AP_RADIO:up=44s,scan=0,scan_ms=0,pawr=1,pawr_ms=43584,acl=0
  • #SENSOR_REG:none
  • No tag RTT, AP scan, ACL connection, association, or PAST diagnostic was run.

Validated-sniffer result:

  • Strict CRC-good summary:
    • crc_good.extended_count = 2734
    • crc_good.sid_counts = {"0": 12, "1": 220, "8": 2502}
    • crc_good.aux_ptr_count = 2677
    • crc_good.aux_ptr_2m_count = 2445
    • crc_good.secondary_count = 57
    • crc_good.sync_info_count = 57
  • Target evidence: target_crc_good.total = 2502, target_crc_good.by_reason = {"sid": 2502}, target_crc_good.by_role = {"primary": 2445, "secondary": 57}.
  • Secondary examples were CRC-good LE 2M ADV_EXT_IND records with SID 8, name ESL_AP, SyncInfo interval 1408 units / 1760.0 ms, access address 0xA64E6569, CRC init 0x5C6A82, and the full channel map ffffffff1f.
  • The summary does not count service UUID 0x1857 as a target match reason for this AP run; target matching is by SID. The CRC-good secondary records carry the complete name ESL_AP and SyncInfo.

Interpretation:

  • The real nRF54 AP can emit CRC-good primary SID 8, 2M secondary extended advertising, and valid periodic advertising SyncInfo on air when the diagnostic image keeps extended advertising running at a fast cadence.
  • The older AP/API success versus RF-silence question is now split: AP PAwR SyncInfo exists on air in this diagnostic condition, while the original tag failure still remains unresolved.
  • The previous lack of secondary/SyncInfo evidence was primarily a capture/observability problem caused by slow primary cadence plus short 2M AuxPtr offsets, not proof that the AP could never emit SyncInfo.
  • Future tag/PAST runs should keep this AP-only RF preflight as a gate. If the AP-only gate passes and the tag still fails after PAST, the next focus returns to tag PAST synchronization, PAwR train reception, response-slot timing, or controller/application state after association.

2026-05-11 Gated Full AP/Tag/Sniffer Run With 2M SyncInfo Proven

Run ID: 20260511_145757Z.

Command:

python3 scripts/run_correlated_ap_sniffer.py --preflight-duration 90 --preflight-require-target --duration 240 --ap-observe 180

Firmware under test:

  • nRF54 AP diagnostic image with extended advertising kept running.
  • Periodic advertising started before extended advertising.
  • Diagnostic extended advertising interval CONFIG_ESL_PAWR_EXT_ADV_INTERVAL=160.
  • Secondary PHY remained the default 2M path.

Artifacts saved locally:

  • logs/correlated_pawr/20260511_145757Z/ap_diag.log
  • logs/correlated_pawr/20260511_145757Z/tag_rtt.log
  • logs/correlated_pawr/20260511_145757Z/preflight/sniffer_summary.json
  • logs/correlated_pawr/20260511_145757Z/sniffer/sniffer_summary.json

No-connection preflight gate:

  • AP status before sniffer capture: pawr=1, scan=0, acl=0, and #SENSOR_REG:none.
  • Strict CRC-good preflight summary:
    • crc_good.extended_count = 2593
    • crc_good.sid_counts = {"0": 2, "1": 208, "8": 2383}
    • crc_good.aux_ptr_count = 2524
    • crc_good.aux_ptr_2m_count = 2314
    • crc_good.secondary_count = 69
    • crc_good.sync_info_count = 69
  • Target evidence passed the gate: target_crc_good.total = 2383, target_crc_good.by_reason = {"sid": 2383}, target_crc_good.by_role = {"primary": 2314, "secondary": 69}.

Full correlated run RF evidence:

  • Strict CRC-good main sniffer summary:
    • crc_good.extended_count = 5087
    • crc_good.sid_counts = {"0": 2, "1": 458, "8": 4627}
    • crc_good.aux_ptr_count = 4860
    • crc_good.aux_ptr_2m_count = 4400
    • crc_good.secondary_count = 227
    • crc_good.sync_info_count = 227
  • Target evidence during tag interaction: target_crc_good.total = 4627, target_crc_good.by_reason = {"sid": 4627}, target_crc_good.by_role = {"primary": 4400, "secondary": 227}.

AP diagnostic result:

  • Scan found and accepted CTrack_B2D9.
  • ACL connected to E9:8A:BE:0D:B2:D9 and discovery completed.
  • Protected ESL GATT writes completed with err=0x00 for handles 0x001f, 0x0025, 0x0021, 0x0023, and 0x002b.
  • Basic state before PAST still reported SYNCHRONIZED:off.
  • PAST succeeded: #PAST_TRANSFER:ok,conn_idx=0,dt_ms=11 and #SENSOR_PASTED:0x0001.
  • After PAST, AP staged and pushed response-requesting addressed PAwR payloads:
    • op=0x1f, addr=0x01, rsp=1, encrypted length 17
    • repeated op=0x10, addr=0x01, rsp=1, encrypted length 17
  • Marker summary included #PAWR_STAGE: 14, #PAWR_PUSH_DETAIL: 74, #PAWR_REQ: 74, and #PAWR_PUSH_OK: 74.
  • No real response markers appeared: real_pawr_response=False.
  • Linux BLE confirmation still found the tag advertising afterward: #LINUX_BLE_SYNC_FAILED_CONFIRMATION:advertising=1, #DIAG_SYNC_FAIL_CONFIRMED_BY_LINUX_ADV:1.

Tag RTT result:

  • Fresh PAST subscription completed: #PAST_SUB:start then #PAST_SUB:done.
  • Tag got a fresh sync callback: #SYNC:1.
  • Sync details reported interval 0x0580 / 1408, PHY 2, SID 8, service data 0x1857, one event, subevent interval 11, response delay 8, and response spacing 3.
  • The tag received one PAwR event: #PAWR_GAP:ev=1950,skipped=1108 and #PAWR_RX:11707036 sub=0 ev=1950.
  • That event contained only broadcast/no-addressed TLV, so no response was sent.
  • The tag then disconnected, skipped re-subscribe because PAwR was considered established, and went silent: #PAWR_SILENT:45000 ms - recovering, #SYNC:0, #STATE:UNSYNC.
  • Collector summary showed #PAWR_RX=1, #PAWR_GAP=1, #PAWR_SILENT=1, and no #PAWR_RSP.

Interpretation:

  • This run removes AP RF/SyncInfo absence as the leading explanation in the diagnostic 2M configuration. The AP emitted CRC-good primary SID 8, 2M secondary extended advertising, and SyncInfo before tag interaction and throughout the full correlated run.
  • AP GATT configuration, PAST API completion, and addressed response-requesting PAwR pushes all succeeded from the AP side.
  • The remaining failure is tag/controller PAwR sustain after PAST: the tag accepts or reaches one PAwR event, then no longer receives usable subevents and eventually recovers as unsynchronized.
  • The next discriminator should force the AP secondary PHY to 1M. If the tag sustains PAwR on 1M, the suspicion moves toward 2M secondary/controller interoperability. If 1M still produces one-RX-then-silence with RF proven, the suspicion moves beyond PHY to nRF54 PAwR/PAST synchronization state or response-subevent handling.

Cross-Role nRF54 PAwR Failure Clue

User-provided context: a previous attempt placed nRF54L15 on the sensor side while keeping the old nRF5340 Thingy91x gateway code intact, and the observed symptom was very similar: one sync or one PAwR receive, then silence.

Archive support:

  • logs/archive/log80: nRF54L15 sensor reached #SYNC:1, saw one #PAWR_RX, then produced repeated controller failures around opcode 0x2084 with status 0x3a, #PAWR_SUB:give_up err=-5, and many #RX_FAIL markers.
  • logs/archive/log86: nRF54L15 sensor reached #SYNC:1, saw one #PAWR_RX, then repeated #RX_FAIL:ev=... sub=0 rssi=127 txpwr=127.
  • logs/archive/log83: nRF54L15 sensor showed one #PAWR_RX, resubscribe attempts, and repeated RX-failure behavior.
  • Caveat: logs/archive/log89 showed sustained PAwR RX and real #PAWR_RSP responses in at least one nRF54 sensor configuration/run. The cross-role clue is therefore a recurring failure signature, not proof that nRF54 can never sustain PAwR.

Interpretation:

  • The same one-RX-then-silence signature has now appeared with nRF54 involved on either side of the AP/tag relationship.
  • Because there is at least one sustained-response counterexample, the likely issue is configuration, timing, controller version, PHY, or PAwR/PAST state-machine interaction rather than an absolute hardware incapability.

2026-05-11 Forced-1M AP/Tag/Sniffer Discriminator

Run ID: 20260511_150808Z.

Command:

python3 scripts/run_correlated_ap_sniffer.py --preflight-duration 90 --preflight-require-target --duration 240 --ap-observe 180 --secondary-phy 1m

Firmware under test:

  • nRF54 AP diagnostic image with extended advertising kept running.
  • Periodic advertising started before extended advertising.
  • Diagnostic extended advertising interval CONFIG_ESL_PAWR_EXT_ADV_INTERVAL=160.
  • CONFIG_ESL_PAWR_FORCE_1M_PHY=y, which adds BT_LE_ADV_OPT_NO_2M to the AP PAwR extended advertising set.
  • Generated config confirmed CONFIG_ESL_PAWR_FORCE_1M_PHY=y, CONFIG_ESL_PAWR_EXT_ADV_INTERVAL=160, CONFIG_ESL_PAWR_KEEP_EXT_ADV_RUNNING=y, and CONFIG_ESL_PAWR_START_PERIODIC_BEFORE_EXT_ADV=y.

Build/flash result:

  • ./build.sh clean completed.
  • Flash programmed build_nrf54l15_connectkit/merged.hex successfully using pyOCD UID 820D9A5F0E1231A458F5D.
  • Only compile warning observed was the existing const-qualifier warning in esl_client_tag_storage.c.

Artifacts saved locally:

  • logs/correlated_pawr/20260511_150808Z/ap_diag.log
  • logs/correlated_pawr/20260511_150808Z/tag_rtt.log
  • logs/correlated_pawr/20260511_150808Z/preflight/sniffer_summary.json
  • logs/correlated_pawr/20260511_150808Z/sniffer/sniffer_summary.json

No-connection preflight gate:

  • AP status before sniffer capture: pawr=1, scan=0, acl=0, and #SENSOR_REG:none.
  • Strict CRC-good preflight summary:
    • crc_good.extended_count = 2378
    • crc_good.sid_counts = {"0": 36, "1": 188, "8": 2154}
    • crc_good.aux_ptr_count = 2274
    • crc_good.aux_ptr_2m_count = 0
    • crc_good.secondary_count = 104
    • crc_good.sync_info_count = 97
  • Target evidence passed the gate: target_crc_good.total = 2154, target_crc_good.by_reason = {"sid": 2154}, target_crc_good.by_role = {"primary": 2057, "secondary": 97}.
  • Target examples reported AuxPtr phy_name = "1M"; no 2M AuxPtr was present.

Full correlated run RF evidence:

  • Strict CRC-good main sniffer summary:
    • crc_good.extended_count = 5109
    • crc_good.sid_counts = {"0": 11, "1": 455, "13": 1, "8": 4642}
    • crc_good.aux_ptr_count = 4857
    • crc_good.aux_ptr_2m_count = 0
    • crc_good.secondary_count = 251
    • crc_good.sync_info_count = 222
  • Target evidence during tag interaction: target_crc_good.total = 4642, target_crc_good.by_reason = {"sid": 4642}, target_crc_good.by_role = {"primary": 4420, "secondary": 222}.
  • This proves the forced-1M AP image emits CRC-good SID 8, 1M secondary extended advertising, and SyncInfo before and during tag interaction.

AP diagnostic result:

  • Scan found and accepted CTrack_B2D9.
  • ACL connected to E9:8A:BE:0D:B2:D9 and discovery completed.
  • Protected ESL GATT writes completed with err=0x00 for handles 0x001f, 0x0025, 0x0021, 0x0023, and 0x002b.
  • Basic state before PAST still reported SYNCHRONIZED:off.
  • PAST succeeded from the AP/controller side: #PAST_TRANSFER:ok,conn_idx=0,dt_ms=10 and #SENSOR_PASTED:0x0001.
  • After PAST, AP staged and pushed response-requesting addressed PAwR payloads:
    • op=0x1f, addr=0x01, rsp=1, encrypted length 17
    • repeated op=0x10, addr=0x01, rsp=1, encrypted length 17
  • Marker summary included #PAWR_STAGE: 14, #PAWR_PUSH_DETAIL: 78, #PAWR_REQ: 74, and #PAWR_PUSH_OK: 78.
  • No real response markers appeared: real_pawr_response=False.
  • Linux BLE confirmation still found the tag advertising afterward: #LINUX_BLE_SYNC_FAILED_CONFIRMATION:advertising=1, #DIAG_SYNC_FAIL_CONFIRMED_BY_LINUX_ADV:1.

Tag RTT result:

  • Tag configured and subscribed for PAST: #PAST_SUB:start then #PAST_SUB:done.
  • ACL disconnected with reason 0x13 about nine seconds later, while the tag was still in configured state.
  • The configured watchdog fired: SM_CONFIGURED watchdog fired - PAST sync_cb never arrived, followed by #CONFIGURED_TIMEOUT and #STATE:UNSYNC.
  • Collector summary showed #PAWR_SUB=0, #SYNC:=0, #PAWR_RX=0, #PAWR_GAP=0, #PAWR_SILENT=0, #CONFIGURED_TIMEOUT=1, and no #PAWR_RSP.

Interpretation:

  • Forced 1M does not restore PAwR operation.
  • The forced-1M failure is not identical to the 2M gated baseline. In 2M, the tag reached #SYNC:1 and received exactly one PAwR event before silence. In 1M, the AP RF/SyncInfo gate passed, AP-side PAST returned OK, but the tag never received the fresh PAST sync callback.
  • This weakens a simple “2M secondary PHY only” explanation. The failure is now broader: nRF54 can emit valid RF on both 2M and 1M diagnostic paths, but the tag still fails to transition into sustained PAwR after PAST.
  • The next useful split is controller/API level rather than another AP RF proof: capture more tag-side PAST/sync failure status, compare against a minimal Zephyr PAwR sample pair, or repeat the ESL AP on a newer controller stack.

2026-05-11 RX_FAILED Reporting And Direct Periodic-AA Proof

Run ID: 20260511_172925Z.

Firmware/instrumentation under test:

  • Tag debug config enabled CONFIG_BT_PER_ADV_SYNC_BUF_SIZE=64, so Zephyr reports PAwR receive failures by calling recv_cb(..., buf=NULL).
  • Tag recv_cb() logs #PAWR_RX_FAIL:ev=...,sub=...,rssi=...,state=... for those controller receive-failure reports.
  • nRF54 AP remained in the forced-1M diagnostic profile with extended advertising kept running and fast diagnostic extended advertising cadence.

Correlated run result:

  • AP PAST succeeded: #PAST_TRANSFER:ok,conn_idx=0,dt_ms=10.
  • Sniffer preflight and main captures both proved CRC-good AP SID 8, 1M secondary, and SyncInfo evidence:
    • Preflight: sid_counts = {"1": 148, "8": 2225}, secondary_count = 75, sync_info_count = 63.
    • Main: sid_counts = {"1": 432, "8": 4672}, secondary_count = 314, sync_info_count = 292.
  • Tag received a fresh PA sync callback with SID 8, PHY 1, interval 1408, one subevent, response delay 8, and response spacing 3.
  • Tag received exactly one valid PAwR event: #PAWR_RX:520149 sub=0 ev=71.
  • Every following event reported controller receive failure: #PAWR_RX_FAIL:ev=72 through ev=96, all with rssi=127.
  • The tag silence watchdog fired after 45 seconds and recovered to unsynchronized state.
  • AP never received a real PAwR response, and Linux BLE confirmation found the tag advertising afterward: #DIAG_SYNC_FAIL_CONFIRMED_BY_LINUX_ADV:1.

Direct periodic-AA capture after the run:

  • Remote capture: ~/raccoon/captures/esl_pa_periodic_20260511_post172925/.
  • SyncInfo-derived periodic access address/CRC were AA=0x647D6EBE, CRC=0xD40629.
  • The direct AA filter found 6 records, all CRC-good, all secondary LE 1M ADV_EXT_IND packets on the periodic access address.
  • Captured channels were {15: 3, 27: 2, 7: 1}.
  • Example PAwR payloads began with 0712001031... or 0711000f31..., matching EAD-bearing periodic payloads rather than only primary/SyncInfo breadcrumbs.

Interpretation:

  • This is the strongest failure signature so far: the AP continues to advertise valid PAwR periodic packets on the SyncInfo-derived access address, while the tag controller opens subsequent PAwR receive attempts and reports RX_FAILED after the first event.
  • The failure is not simply absence of AP RF, not merely AP host/API optimism, and not a tag application parser issue. The tag controller is missing or rejecting the later periodic events before the ESL payload parser sees them.

2026-05-11 Slow Extended-Advertising Cadence Discriminator

Run ID: 20260511_181151Z.

Purpose:

  • Test whether the fast diagnostic extended advertising cadence (CONFIG_ESL_PAWR_EXT_ADV_INTERVAL=160) was perturbing sustained PAwR receive.

Firmware under test:

  • nRF54 AP board config changed CONFIG_ESL_PAWR_EXT_ADV_INTERVAL from 160 back to the legacy low-pressure value 14080.
  • Generated config confirmed CONFIG_ESL_PAWR_EXT_ADV_INTERVAL=14080 with forced 1M, extended advertising kept running, and periodic advertising started before extended advertising.

Clean-state procedure before the run:

  • Tag was reflashed with chip erase.
  • AP state was cleared with esl_c acl disconnect 0, esl_c unbond_all, esl_c remove_all_tags, esl_c bond_dump, esl_c list_tags_storage_map, and esl_c ap_status.
  • AP confirmed empty bonds/storage and no registered sensor before scanning.

Correlated run result:

  • AP PAST succeeded: #PAST_TRANSFER:ok,conn_idx=0,dt_ms=10.
  • AP staged and pushed addressed response-producing PAwR payloads (op=0x1f followed by repeated op=0x10, addr=0x01, rsp=1).
  • No real AP response marker appeared: real_pawr_response=False.
  • Because the extended advertising cadence was slow, preflight/main SyncInfo evidence was sparse:
    • Preflight: sid_counts = {"1": 240, "8": 31}, secondary_count = 10, sync_info_count = 0.
    • Main: sid_counts = {"1": 468, "8": 55}, secondary_count = 28, sync_info_count = 1.
  • Tag received a fresh PA sync callback: #SYNC_CB:idx=0,sid=8,phy=1,int=1408,num=1,sub_int=11,rsd=8,rss=3,svc=0x1857.
  • Tag received exactly one valid PAwR event: #PAWR_RX:122855 sub=0 ev=90.
  • Every following event reported RX_FAILED: #PAWR_RX_FAIL:ev=91 through ev=115, all with rssi=127.
  • The tag silence watchdog fired after 45 seconds and Linux BLE confirmation found the tag advertising afterward.

Direct periodic-AA capture after the run:

  • Remote capture: ~/raccoon/captures/esl_pa_periodic_20260511_181151Z/.
  • SyncInfo-derived periodic access address/CRC were AA=0xD9522E15, CRC=0x10888A.
  • The direct AA filter found 7 records, all CRC-good, all secondary LE 1M ADV_EXT_IND packets on the periodic access address.
  • Captured channels were {0, 13, 15, 18, 21, 32, 35}.
  • Example PAwR payloads began with 0711000f31..., again matching EAD-bearing periodic payloads.

Interpretation:

  • Slowing extended advertising back to the legacy low-pressure cadence does not fix the failure.
  • The failure persists with the same shape: one valid PAwR receive, then controller RX_FAILED on every subsequent periodic event.
  • The slow cadence reduces SyncInfo observability and is not useful as a default diagnostic baseline. Future evidence-gated tests should use the fast 160 cadence unless specifically retesting production-like pressure.

2026-05-11 Diagnostic Channel-Map Override Attempt

Run ID: 20260511_184712Z.

Purpose:

  • Test whether constraining the AP’s periodic data-channel map could distinguish channel-selection divergence from timing/anchor failure after the first PAwR event.

Firmware change under test:

  • Added default-off AP Kconfig options:
    • CONFIG_ESL_PAWR_CHANNEL_MAP_OVERRIDE
    • CONFIG_ESL_PAWR_CHANNEL_MAP_LOW
    • CONFIG_ESL_PAWR_CHANNEL_MAP_HIGH
  • service/esl_client.c calls bt_le_set_chan_map() before PAwR advertising-set creation when the override is enabled, and emits #PAWR_CHMAP:low=...,high=...,err=....
  • nRF54 AP board config was temporarily set to CONFIG_ESL_PAWR_CHANNEL_MAP_LOW=0x00000001, CONFIG_ESL_PAWR_CHANNEL_MAP_HIGH=0x00, with fast extended advertising cadence restored to 160.

Build/flash and clean state:

  • AP build succeeded and generated config confirmed the one-channel override symbols.
  • AP and tag were flashed/reset, tag was chip-erased, and AP bonds/tag storage were cleared before the correlated run.

Correlated run result:

  • AP PAST succeeded: #PAST_TRANSFER:ok,conn_idx=0,dt_ms=21.
  • AP continued staging and pushing addressed PAwR payloads, but no real response marker appeared.
  • Tag received a fresh PA sync callback with the same expected PAwR timing values, but did not receive even the first valid PAwR payload in this run.
  • Tag reported #PAWR_RX_FAIL:ev=718 through ev=742, all with rssi=127, then the silence watchdog recovered.
  • Linux BLE confirmation again found the tag advertising afterward.

Sniffer result:

  • Preflight passed strongly after returning to fast cadence:
    • sid_counts = {"1": 130, "8": 1389}, secondary_count = 58, sync_info_count = 56.
  • Main run also passed strongly:
    • sid_counts = {"0": 5, "1": 390, "8": 3841}, secondary_count = 233, sync_info_count = 207.
  • Every captured SyncInfo still advertised the full channel map ffffffff1f:
    • Preflight: 56/56 SyncInfo records had channel_map_hex = ffffffff1f.
    • Main: 207/207 SyncInfo records had channel_map_hex = ffffffff1f.
  • SyncInfo access address/CRC for the run were AA=0x1BAFA4DB, CRC=0x237B60.

Interpretation:

  • The attempted bt_le_set_chan_map() override did not affect the nRF54 AP periodic advertising SyncInfo in this SDC/NCS v2.8.0 configuration.
  • Because the controller continued to advertise all 37 data channels, this run does not test the single-channel workaround hypothesis.
  • The default-off channel-map diagnostic code remains available for future experiments, but the nRF54 AP board config was returned to the fast-observability baseline with the override disabled.

2026-05-11 Stock Zephyr PAwR/PAST Positive Control

Artifact doc: NRF54L15_MINIMAL_ZEPHYR_SAMPLE_RESULT.md.

Purpose:

  • Separate a generic nRF54L15/nRF52840 SoftDevice Controller PAwR+PAST failure from an ESL-profile-specific failure.
  • Avoid further edits to the production tag application while still using the same physical AP/tag hardware.

Temporary firmware under test:

  • AP / PAST sender: stock Zephyr ~/ncs/v2.8.0_clear/zephyr/samples/bluetooth/periodic_adv_rsp, built for nrf54l15_connectkit/nrf54l15/cpuapp in build_minimal_periodic_adv_rsp.
  • Tag / PAST receiver: stock Zephyr ~/ncs/v2.8.0_clear/zephyr/samples/bluetooth/periodic_sync_rsp, built for adafruit_itsybitsy/nrf52840 in build_minimal_periodic_sync_rsp.
  • Both samples use the SoftDevice Controller PAwR paths in this NCS v2.8.0 tree.

Capture result:

  • AP serial capture: ../logs/minimal_pawr_past/20260511_stock_samples/ttyACM0.log.
  • In a 150-second capture, the AP log contained:
    • PAST sent: 6
    • PAwR config written: 6
    • Connected (err 0x00): 7
    • Disconnected, reason: 7
    • Response: subevent: 176
  • Representative sustained response payloads were visible after PAST, for example 0xFF: 590018, 0xFF: 59001D, and later incrementing values.

Post-test restore:

  • nRF52840 tag restored from 1wire_blesensro_prod/build_debug/merged.hex with chip erase, verify, and reset.
  • nRF54L15 AP restored from samples_nrf54/central_esl/build_nrf54l15_connectkit/merged.hex with pyOCD flash and hardware reset.

Interpretation:

  • The same hardware and NCS v2.8.0 controller family can sustain PAwR responses after PAST with the stock Zephyr sample profile.
  • Therefore, the ESL failure should not be escalated as a generic “PAwR+PAST never works” issue.
  • The next minimal reproducer should start from the passing stock samples and move one parameter group at a time toward the ESL profile: interval 1408, one subevent, subevent interval 11, response slot delay 8, response slot spacing 3, response slots 8, SID 8, service UUID 0x1857, then payload/update/disconnect behavior.

2026-05-11 Adapted Minimal Reproducer

Artifact doc: NRF54L15_ADAPTED_MINIMAL_REPRODUCER_RESULT.md.

Purpose:

  • Reproduce the failure with a Zephyr-derived sample and no production ESL app source.
  • Split the passing stock PAwR/PAST profile from the failing ESL-like PAwR profile.

Diagnostic sources:

  • AP / PAST sender: ../diagnostics/pawr_esl_profile_adv/src/main.c.
  • Tag / PAST receiver: ../diagnostics/pawr_esl_profile_sync/src/main.c.

ESL-profile timing run:

  • Capture: ../logs/minimal_pawr_past/20260511_esl_profile_timing/ttyACM0.log.
  • AP profile: sid=8,interval=1408,num_subevents=1,subevent_interval=11,rsp_delay=8,rsp_spacing=3,rsp_slots=8,ext_adv_interval=160,phy=1M.
  • Marker counts: 8 PAST sent, 8 PAwR config written, 103 #DIAG_ADV_SET_OK, 1 #DIAG_ADV_RESPONSE, 8 disconnects.
  • Only response observed: #DIAG_ADV_RESPONSE:subevent=0,slot=6,len=5, payload 0xFF: 590024.

Long-interval / wide-slot comparison:

  • Capture: ../logs/minimal_pawr_past/20260511_long_interval_wide_slots_real/ttyACM0.log.
  • AP profile: sid=8,interval=1408,num_subevents=1,subevent_interval=48,rsp_delay=5,rsp_spacing=80,rsp_slots=5,ext_adv_interval=160,phy=1M.
  • Marker counts: 5 PAST sent, 5 PAwR config written, 85 #DIAG_ADV_SET_OK, 0 #DIAG_ADV_RESPONSE, 5 disconnects.

Interpretation:

  • The failure is reproducible with a tiny Zephyr-derived sample. Production ESL association, encrypted ESL payload parsing, and application-specific sensor logic are not required.
  • The tight ESL response-slot spacing is not the only trigger, because the long-interval / wide-slot comparison still failed.
  • The next split should isolate long interval 1408 from one-subevent PAwR.

Stock-interval / one-subevent comparison:

  • Capture: ../logs/minimal_pawr_past/20260511_stock_interval_one_subevent/ttyACM0.log.
  • AP profile: sid=8,interval=255,num_subevents=1,subevent_interval=48,rsp_delay=5,rsp_spacing=80,rsp_slots=5,ext_adv_interval=160,phy=1M.
  • Marker counts: 4 PAST sent, 4 PAwR config written, 376 #DIAG_ADV_SET_OK, 270 #DIAG_ADV_RESPONSE, 3 disconnects.
  • Interpretation: one-subevent PAwR is not sufficient to trigger the failure. The long periodic interval 1408, or its interaction with PAST/re-onboarding, is now the strongest suspect.

Long-interval / five-subevent comparison:

  • Capture: ../logs/minimal_pawr_past/20260511_long_interval_5_subevents/ttyACM0.log.
  • AP profile: sid=8,interval=1408,num_subevents=5,subevent_interval=48,rsp_delay=5,rsp_spacing=80,rsp_slots=5,ext_adv_interval=160,phy=1M.
  • Marker counts: 15 PAST sent, 15 PAwR config written, 170 #DIAG_ADV_SET_OK, 12 #DIAG_ADV_RESPONSE, 14 disconnects.
  • Interpretation: long periodic interval 1408 is not sufficient by itself to trigger the failure. With stock-style 5 subevents, the isolated diagnostic sample repeatedly produces responses. The strongest suspect is now the long-interval + one-subevent combination.

Long-interval / one-subevent / 2M comparison:

  • Capture: ../logs/minimal_pawr_past/20260511_long_interval_one_subevent_2m/ttyACM0.log.
  • AP profile: sid=8,interval=1408,num_subevents=1,subevent_interval=48,rsp_delay=5,rsp_spacing=80,rsp_slots=5,ext_adv_interval=160,phy=2M.
  • Marker counts: 5 PAST sent, 5 PAwR config written, 69 #DIAG_ADV_SET_OK, 0 #DIAG_ADV_RESPONSE, 5 disconnects, 1 Maximum number of syncs onboarded.
  • Interpretation: the failing long-interval / one-subevent profile also fails on 2M secondary. Forced 1M is not the trigger.

Long-interval / two-subevent comparison:

  • Capture: ../logs/minimal_pawr_past/20260511_long_interval_2_subevents/ttyACM0.log.
  • AP profile: sid=8,interval=1408,num_subevents=2,subevent_interval=48,rsp_delay=5,rsp_spacing=80,rsp_slots=5,ext_adv_interval=160,phy=1M.
  • Marker counts: 10 PAST sent, 10 PAwR config written, 69 #DIAG_ADV_SET_OK, 5 #DIAG_ADV_RESPONSE, 10 disconnects, 1 Maximum number of syncs onboarded.
  • Configured pairs: subevent 0 slots 0..4 once each, and subevent 1 slots 0..4 once each.
  • Response pairs: subevent 1 slots 0..4 once each; no AP-side responses for subevent 0 in this run.
  • Interpretation: this partially clears a strict num_subevents=1 explanation, but it also introduces a subevent-index/phase clue. The long-interval profile is not uniformly broken with two subevents, but the first subevent remained silent in this capture.

Long-interval / two-subevent / repeated-subevent-0 comparison:

  • Capture: ../logs/minimal_pawr_past/20260511_long_interval_2_subevents_subevent0/ttyACM0.log.
  • AP profile: sid=8,interval=1408,num_subevents=2,subevent_interval=48,rsp_delay=5,rsp_spacing=80,rsp_slots=5,ext_adv_interval=160,phy=1M.
  • Marker counts: 10 PAST sent, 10 PAwR config written, 68 #DIAG_ADV_SET_OK, 0 #DIAG_ADV_RESPONSE, 10 disconnects, 1 Maximum number of syncs onboarded.
  • Configured pairs: subevent 0 slots 0..4 twice each.
  • Interpretation: subevent 0 stayed silent when repeatedly configured. The two-subevent evidence now points at subevent index/phase, not only total subevent count.

Long-interval / two-subevent / repeated-subevent-1 comparison:

  • Capture: ../logs/minimal_pawr_past/20260511_long_interval_2_subevents_subevent1/ttyACM0.log.
  • AP profile used for the build: sid=8,interval=1408,num_subevents=2,subevent_interval=48,rsp_delay=5,rsp_spacing=80,rsp_slots=5,ext_adv_interval=160,phy=1M.
  • Marker counts: 8 PAST sent, 8 PAwR config written, 68 #DIAG_ADV_SET_OK, 8 #DIAG_ADV_RESPONSE, 8 disconnects, 1 Maximum number of syncs onboarded.
  • Configured pairs: subevent 1, slots 2, 3, 4, 0, 1, 2, 3, 4.
  • Response pairs: every configured cycle produced an AP-side response; slots 0 and 1 responded once each, and slots 2..4 responded twice each.
  • Interpretation: subevent 1 stayed responsive under the same two-subevent long-interval profile where repeated subevent 0 produced no responses. This confirms that subevent index/phase is the strongest current lead.

Long-interval / one-subevent / single-sync comparison:

  • Capture: ../logs/minimal_pawr_past/20260511_long_interval_one_subevent_single_sync_retry/ttyACM0.log.
  • AP profile: sid=8,interval=1408,num_subevents=1,subevent_interval=48,rsp_delay=5,rsp_spacing=80,rsp_slots=5,ext_adv_interval=160,phy=1M.
  • Marker counts: 1 Connected, 1 PAST sent, 1 PAwR config written, 1 #DIAG_KEEP_SYNC_ACTIVE, 86 #DIAG_ADV_SET_OK, 0 #DIAG_ADV_RESPONSE, 0 disconnects, 0 Maximum number of syncs onboarded.
  • Configured pair: subevent 0, slot 0, held active for the rest of the capture.
  • Interpretation: repeated PAST/re-onboarding is not required. A single successful PAST/config write with the sync held active still produced no AP-side responses while the AP continued setting subevent data.

Current Next Test Candidates

  1. Repeat the 302/303/304 one-subevent threshold on a newer controller stack if practical.
  2. Repeat the product group-1 profile with tag RTT attached and focus on the moment around ACL disconnect.
  3. Add a direct periodic-AA follower mode to the sniffer, using SyncInfo AA/CRC/channel map/event counter to listen on the predicted PAwR channel schedule instead of sweeping. This would turn sparse CRC-good direct periodic evidence into event-by-event channel/timing evidence.

2026-05-11 One-Subevent Interval Sweep

Artifact doc: NRF54L15_ADAPTED_MINIMAL_REPRODUCER_RESULT.md.

Purpose:

  • Identify whether the one-subevent failure has a threshold between the passing stock interval and the failing long ESL interval.
  • Keep the production tag source untouched by using the Zephyr-derived diagnostic pair.

Common profile:

  • SID 8.
  • One subevent.
  • Subevent interval 48.
  • Response slot delay 5.
  • Response slot spacing 80.
  • Response slots 5.
  • Extended advertising interval 160.
  • Forced LE 1M.

Sweep results:

Interval Capture PAST sent #DIAG_ADV_SET_OK #DIAG_ADV_RESPONSE Result
288 ../logs/minimal_pawr_past/20260511_interval_0288_one_subevent/ttyACM0.log 5 223 9 pass
296 ../logs/minimal_pawr_past/20260511_interval_0296_one_subevent/ttyACM0.log 5 268 9 pass
300 ../logs/minimal_pawr_past/20260511_interval_0300_one_subevent/ttyACM0.log 5 263 4 pass, weaker
302 ../logs/minimal_pawr_past/20260511_interval_0302_one_subevent/ttyACM0.log 5 262 5 pass, weaker
303 ../logs/minimal_pawr_past/20260511_interval_0303_one_subevent/ttyACM0.log 4 201 0 fail
303 retry ../logs/minimal_pawr_past/20260511_interval_0303_one_subevent_retry/ttyACM0.log 2 262 0 fail
304 ../logs/minimal_pawr_past/20260511_interval_0304_one_subevent/ttyACM0.log 5 261 0 fail
320 ../logs/minimal_pawr_past/20260511_interval_0320_one_subevent/ttyACM0.log 5 193 0 fail
384 ../logs/minimal_pawr_past/20260511_interval_0384_one_subevent/ttyACM0.log 5 206 0 fail
512 ../logs/minimal_pawr_past/20260511_interval_0512_one_subevent/ttyACM0.log 5 155 0 fail
768 ../logs/minimal_pawr_past/20260511_interval_0768_one_subevent/ttyACM0.log 5 104 0 fail

Interpretation:

  • The one-subevent diagnostic has a sharp response boundary near 303/304 in this setup.
  • Interval 288 was chosen as the conservative product-app trial value because it sits below the threshold and gives margin below the weak 300/302 responses.
  • The threshold does not by itself prove product stability; it only identifies a diagnostic parameter region where the minimal sample can answer.

2026-05-12 nRF54L15 288 Product Release-Candidate Trial

Artifact doc: NRF54L15_STABLE_288_RELEASE_CANDIDATE.md.

Purpose:

  • Convert the diagnostic interval result into a product-app candidate for controlled release-environment use.
  • Keep edits confined to the experimental nRF54L15 board config under samples_nrf54/central_esl.
  • Preserve production tag source by restoring the existing production debug image with chip erase before each active test.

One-subevent 288 product result:

  • Candidate values: CONFIG_ESL_CLIENT_MAX_GROUP=1, interval min/max 288, subevent interval 48, response delay 5, spacing 80, slots 5, ext advertising interval 160, forced 1M.
  • Active test: ../logs/stable_288_profile/20260512_release_candidate/host_console_test_clean_state.log.
  • Clean state was confirmed before the run: AP bonds/tag storage cleared and tag chip-erased/reflashed.
  • Security succeeded: #SEC_CHANGED:idx=0,level=2,err=0.
  • PAST succeeded: #PAST_TRANSFER:ok,conn_idx=0,dt_ms=10.
  • Sensor was marked pasted: #SENSOR_PASTED:0x0001.
  • No real #PAWR_RSP, #NUS_STATUS, #NUS_SENSORS, or #SENSOR_NAME response appeared.
  • The run ended with ERROR: timed out waiting for response to esl_c nus status 0x0001.

Interpretation: interval 288 and clean state are not sufficient to make the one-subevent full ESL app stable.

Two-subevent / group-1 288 product result:

  • Candidate values: CONFIG_ESL_CLIENT_MAX_GROUP=2, CONFIG_ESL_CLIENT_DEFAULT_GROUP_ID=1, interval min/max 288, subevent interval 48, response delay 5, spacing 80, slots 5, ext advertising interval 160, forced 1M.
  • Build: ../logs/stable_288_profile/20260512_release_candidate/build_stable_288_group1.log.
  • AP cleanup: ../logs/stable_288_profile/20260512_release_candidate/ap_cleanup_group1.log.
  • Active test: ../logs/stable_288_profile/20260512_release_candidate/host_console_test_group1.log.
  • Generated config confirmed group 1 and interval 288.
  • Tag configured as ESL 0x0101: #CONFIGURED: 1,0x0000,0x0101.
  • PAST succeeded: #PAST_TRANSFER:ok,conn_idx=0,dt_ms=10.
  • Initial subevent-1 PAwR/NUS responses worked:
    • #PAWR_RSP:sub=1,slot=0,rssi=-45
    • #NUS_STATUS:ADDR=0x0101,UP=12s,BATT=4168mV,FLAGS=0x03
    • #PAWR_RSP:sub=1,slot=0,rssi=-41
    • #NUS_SENSORS:ADDR=0x0101,TEMP=27.18,BATT=4168mV,CNT=2
  • The run did not pass long-term validation:
    • ACL disconnected with reason 0x13 after PAST.
    • Later polling produced #SENSOR_POLL_FAIL:0x0101,consec=1.
    • get_name timed out with ERROR: timed out waiting for response to esl_c nus get_name 0x0101.

Interpretation:

  • Group/subevent 1 is a real partial product workaround because it produces real full-app PAwR/NUS responses.
  • It is not a release-stable long-term workaround yet because sustained PAwR-only operation stops after the initial responses.
  • Follow-up trials attached tag RTT and AP retry instrumentation; they are recorded below.

2026-05-12 AP Retry And V3 Target-Subevent Split

Artifact directory: ../logs/pawr_resub_exp/20260512_exp03_ap_retry.

Purpose:

  • Test whether the group-1 product failure is caused by AP command lifetime: a response-requesting PAwR payload being overwritten by NOP before the tag can answer.
  • Test whether the tag-side failure is caused by initially requesting/defaulting to subevent 0 instead of the configured target group/subevent 1.
  • Keep tag changes isolated in 1wire_blesensro_prod_pawr_resub_exp; do not edit production tag app 1wire_blesensro_prod.

AP bounded retry result:

  • AP retry build enabled CONFIG_ESL_PAWR_RESPONSE_RETRY_COUNT=64 only for the nRF54L15 AP experiment board config; global default remains 0.
  • Evidence: ../logs/pawr_resub_exp/20260512_exp03_ap_retry/host_console_test.log and ../logs/pawr_resub_exp/20260512_exp03_ap_retry/tag_rtt.log.
  • Clean AP state was confirmed before the run with #SENSOR_REG:none.
  • Security, GATT configuration, and PAST succeeded; AP registered ESL 0x0101.
  • AP staged NUS status for subevent/group 1: #PAWR_STAGE:grp=1,plain_len=4,enc_len=17,op=0x1f,addr=0x01,rsp=1.
  • AP retry fired repeatedly: #PAWR_RETRY:sub=1,left=63,op=0x1f,addr=0x01 and later counts.
  • No real PAwR response arrived; the host test timed out waiting for esl_c nus status 0x0101.
  • Tag RTT showed #PAWR_RX ... sub=0, #PAWR_GROUP_MISMATCH:ignore_first, then repeated #PAWR_RX_FAIL:...sub=1.

Interpretation: AP retry worked mechanically but did not recover NUS-over-PAwR. The failure is not explained solely by one-shot AP response payload lifetime.

V3 target-subevent result:

  • Evidence: ../logs/pawr_resub_exp/20260512_exp03_ap_retry/host_console_test_v3_target_sub.log, ../logs/pawr_resub_exp/20260512_exp03_ap_retry/tag_rtt_v3_target_sub.log, ../logs/pawr_resub_exp/20260512_exp03_ap_retry/tag_build_v3_target_sub.log, and ../logs/pawr_resub_exp/20260512_exp03_ap_retry/tag_flash_v3_target_sub.log.
  • The isolated tag was patched to request its configured target group/subevent in sync_cb rather than starting from subevent 0.
  • V3 built and flashed successfully.
  • AP cleanup initially showed remove_all_tags cleared storage but left stale in-memory registry state; after pyOCD reset, AP reported #SENSOR_REG:none.
  • Commissioning and PAST succeeded again.
  • Tag RTT still showed #PAWR_SUB:sync_cb=0, then first delivery as #PAWR_RX ... sub=0, wrong-group ignore, and repeated #PAWR_RX_FAIL:...sub=1.
  • AP again emitted #PAWR_RETRY but received no #PAWR_RSP or NUS response.

Interpretation: requesting target subevent 1 immediately in sync_cb is not sufficient. The controller still reports a successful subevent request while the first delivered payload is subevent 0, and subsequent target-subevent receives fail.

Updated conclusion:

  • AP command lifetime is not the primary remaining root cause.
  • Immediate target-subevent request from tag sync_cb is not a sufficient workaround.
  • The interval-288, two-subevent, group-1 product profile remains not release-cleared.
  • Next discriminators should focus on AP profile shape and controller subevent phase/tracking: group 0 versus group 1, one versus two subevents, and/or the same tests on a newer controller stack.