t4as

Research note · 2026-05-11

nRF54L15 PAwR/PAST Investigation Overview

Update, 1 October 2026: the cause was found. The board’s 32 kHz crystal runs about 175 ppm fast while the firmware declared 50 ppm, which narrows the PAwR receive windows below what this note describes. See Root cause: the 32 kHz clock. This note is kept as written in May.

Date: 2026-05-11

This branch documents a focused embedded Bluetooth investigation: porting and validating an ESL-style PAwR access point path on nRF54L15 while preserving a known-working nRF5340/Thingy91x gateway and a production nRF52840 sensor tag firmware.

The work is intentionally evidence-heavy. The goal is not to claim a final controller bug before it is proven. The goal is to show a disciplined process for isolating a complex wireless failure across application code, Zephyr/NCS configuration, Bluetooth controller behavior, hardware state, and RF evidence.

System Under Test

The system uses Bluetooth LE ESL-style communication where an access point synchronizes tags through Periodic Advertising with Responses and PAST:

  • AP under investigation: Makerdiary nRF54L15 Connect Kit running NCS v2.8.0 with SoftDevice Controller.
  • Production tag target: nRF52840-based sensor tag firmware, protected from diagnostic source edits.
  • Known-good gateway baseline: existing nRF5340/Thingy91x path, treated as the working reference and not modified for this investigation.
  • Diagnostic tooling: AP host-console scripts, tag RTT collection, stock Zephyr PAwR/PAST samples, Zephyr-derived minimal reproducer samples, and an external multi-device RF sniffer rig.

The active research branch is deliberately separate from the production firmware. Diagnostic changes are isolated under samples_nrf54/central_esl, and production tag recovery state is tracked in PRODUCTION_TAG_FIRMWARE_STATE.md.

Failure Being Investigated

The observed failure is not simply that an application callback is quiet. The AP can complete GATT configuration, report successful PAST, and stage PAwR subevent data. The tag can receive PA sync information. The failure appears after that handoff: the receiver gets either one valid PAwR event or no valid PAwR event, then reports receive failures at the periodic receive layer.

Current best narrowing is documented in NRF54L15_PAWR_ROOT_CAUSE_PACKET.md: the strongest suspect is a controller/scheduling boundary around PAwR + PAST with a small subevent set, especially the first/only response-capable subevent or subevent phase. An interval sweep found a sharp one-subevent threshold near 303/304, and a product-app group-1 trial produced initial real responses but did not sustain long-term operation after ACL disconnect. Later AP retry and isolated tag target-subevent experiments did not recover the failure, which further points away from AP command lifetime alone. That is not yet a final root cause; it is the best-supported boundary as of this branch.

Method

The investigation used a layered approach:

  1. Preserve known-good firmware before experimenting.
  2. Clear AP/tag state before interpreting failures.
  3. Add AP logs for PAST, PAwR staging, subevent-data pushes, and registry state.
  4. Add tag RTT markers for sync callbacks, PAwR RX, RX_FAILED, and silence timeouts.
  5. Validate the external sniffer against a controlled nRF54 signal before trusting AP captures.
  6. Use stock Zephyr PAwR/PAST samples as a positive control on the same boards.
  7. Create isolated Zephyr-derived diagnostic samples and move one profile group at a time toward the ESL-like configuration.
  8. Keep raw logs and summarize marker counts in docs so later conclusions can be audited.

This process matters because any single layer can lie by omission. AP API success does not prove RF timing. RF presence does not prove the tag accepted sync. A tag-side timeout does not prove the AP stopped transmitting. The useful evidence comes from correlating all layers.

Evidence Produced

The branch contains a compact evidence packet rather than every raw capture produced during debugging:

Selected results:

  • Stock Zephyr PAwR/PAST samples sustained responses on the same nRF54L15/nRF52840 hardware.
  • ESL-like long interval 1408, one subevent, and tight timing reproduced the non-sustained behavior in an isolated diagnostic sample.
  • Long interval 1408 plus one subevent still failed with wider response slots.
  • Long interval 1408 plus one subevent still failed when the first PAST/config was kept active instead of repeatedly re-onboarding.
  • Stock interval 255 plus one subevent worked.
  • Long interval 1408 plus stock-style five subevents produced repeated responses.
  • Long interval 1408 plus one subevent failed on both forced 1M and 2M secondary PHY.
  • Long interval 1408 plus two subevents partially recovered, but only subevent 1 produced AP-side responses in that capture.
  • Repeating only subevent 0 in the long-interval two-subevent profile produced no AP-side responses.
  • Repeating only subevent 1 in the same long-interval two-subevent profile produced AP-side responses for every configured cycle.
  • One-subevent interval sweep with wide slots produced responses at 288, 296, 300, and 302, then zero responses at 303 and above in the captured runs.
  • Full product-app interval 288 with one subevent did not produce real post-PAST responses.
  • Full product-app interval 288 with two subevents and default group 1 produced initial real subevent-1 PAwR/NUS responses, but did not pass sustained long-term validation after ACL disconnect.
  • AP bounded response retry kept subevent-1 response payloads alive and emitted repeated #PAWR_RETRY, but still produced no PAwR response because the tag stayed in subevent-1 RX_FAILED.
  • An isolated V3 tag change requested target subevent 1 immediately in sync_cb, but the controller still delivered subevent 0 first and then repeated subevent-1 RX_FAILED.

Those splits reduce the search space without overstating the answer.

What This Demonstrates

This project shows a practical debugging style for embedded wireless systems:

  • Protect production firmware and known-good baselines before changing variables.
  • Treat controller/API success as a starting point, not as proof.
  • Build positive controls before diagnosing a negative result.
  • Use minimal reproducers to separate application behavior from stack/controller behavior.
  • Validate measurement tools before using them as evidence.
  • Keep documentation close to raw artifacts so conclusions remain reviewable.

The result is a research trail that can be handed to another engineer or to vendor support without requiring them to trust intuition. They can inspect the logs, compare the test matrix, and see which hypotheses were cleared.

Current State

As of this branch, the likely failure boundary is PAwR + PAST with subevent index/phase behavior and a one-subevent interval threshold on the tested NCS v2.8.0 controller stack. The clearest paired split so far is that repeated subevent 0 is silent while repeated subevent 1 responds under the same long-interval two-subevent profile, and the one-subevent interval sweep stops responding around 303/304. Repeated PAST/onboarding is not required to trigger the one-subevent failure. A clean product group-1 RTT repeat also exposed a local tag-side timing hazard, but AP retry and immediate target-subevent subscription did not recover the failure. The current nRF54L15 product gateway profile is not yet cleared for long-term release use. The next useful tests are:

  • A newer-controller-stack repeat of the 302/303/304 threshold.
  • A controlled AP profile split for group/subevent 0 versus 1 at interval 288, with tag RTT attached.
  • A one-subevent versus two-subevent product split with retry instrumentation, clean AP reset, and tag RTT attached.
  • A newer-controller-stack repeat of the minimal subevent-index split, if practical.
  • A SyncInfo-following sniffer mode that predicts event channel/time and records event-by-event RF presence.

The project is therefore in a productive research state: enough evidence exists for a high-quality technical support question, and the remaining work is clearly scoped.