Yocto for Small Boards: One Image, Remote Access, and an FPGA as the Camera

Result

This started as a small image for one board and is becoming the base for devices that get installed where nobody can reach them. That sets the one requirement everything here is measured against: a deployed device must never need a visit. It has to come back from a fault on its own, stay updatable, and still be the same device afterwards.

• One layer, several boards. The distro policy, package list and services live in a single project layer. A board is a short kas file that names a machine. Images for the Pi Zero W, the Pi Zero 2 W and the Banana Pi M2 Zero come out of the same recipes.
• Remote access without an open port. The device makes outbound connections only. It enrols once over HTTPS, then takes commands over MQTT with a client certificate. A shell is a reverse SSH tunnel that the device opens on request and closes on a deadline.
• It renews and recovers by itself. The device renews its certificate before it expires, and after it has expired, using the key it enrolled with. A health check proves once a minute that commands can still reach it, and escalates step by step when they can’t. A hardware watchdog covers the case where nothing runs at all.
• Identity outside the root filesystem. Keys, certificates and Wi-Fi credentials sit on their own partition. I replaced the root partition on an enrolled device and it came back as the same device, with no re-enrolment.
• An FPGA in place of the camera. A Tang Nano 20K drives the Banana Pi’s parallel camera input through the kernel’s normal capture path. Every byte of the test stream was checked at eight clock rates up to 54 MHz.

One layer, several boards

The project layer defines a distro (systemd, systemd-networkd, no X11, Wayland or audio), one package group and one image recipe. Board-specific things stay in the BSP layers: meta-raspberrypi for the Pis and meta-sunxi with mainline Linux and U-Boot for the Banana Pi.

Target	MACHINE	Notes
Pi Zero W	raspberrypi0-wifi	the fleet device used for all hardware tests
Pi Zero 2 W	raspberrypi0-2w-64	64-bit
Banana Pi M2 Zero	bananapi-m2-zero	Allwinner H2+, mainline kernel and U-Boot
QEMU	qemuarm64	boots the same image for smoke tests: systemd, DHCP, DNS, TLS, SSH

Every upstream layer is pinned to a full commit in kas, so a build next year fetches the same sources (kas/base.yml):

repos:
  poky:
    url: https://git.yoctoproject.org/poky.git
    tag: yocto-5.0.19
    commit: bb98354685781296e3b3737e7762412100f359c2
    layers:
      meta:
  meta-openembedded:
    url: https://git.openembedded.org/meta-openembedded.git
    branch: scarthgap
    commit: bec755063a8b5da65df626f5749496aadaa4f4bb
    layers:
      meta-oe:
      meta-python:

The release is Scarthgap (5.0 LTS) rather than the newest one for a concrete reason: the Banana Pi BSP layer has a Scarthgap branch and none for the newer release. I would rather run one release behind than edit LAYERSERIES_COMPAT in someone else’s layer and pretend it was tested.

Features are kas fragments that are added to a board: the fleet client, the OV5647 camera, the U-Boot boot path. The plain image has none of them, and no credentials. The build host is an Orange Pi 4 Pro, so everything compiles natively on aarch64, with BitBake held to four threads because the board has 5.7 GiB of RAM. Downloads and shared state are kept on the NVMe drive and shared by all targets.

Remote access without an open port

Enrolment. A fresh device generates a UUID, a TLS key with a signing request, and a separate SSH key for the tunnel. It sends the public parts with a single-use token; the server stores only the token’s hash and binds it to the first key material it sees, so a retry with the same keys is idempotent and a replay with different keys is refused. The device gets back a client certificate whose MQTT permissions cover its own five topics and nothing else.

Commands. MQTT is only the transport. Each message goes through one dispatcher that checks the envelope, the expiry time and an allowlist: camera start, stop, restart, status and profile, telemetry, shell open and close, reboot. There is no “run this” command. The arguments of shell.open are checked the same way (fleet-dispatch):

fleet_is_uint "$duration" || { reply_error invalid_duration; exit 0; }
[ "$duration" -ge 60 ] && [ "$duration" -le 3600 ] || {
    reply_error invalid_duration
    exit 0
}
fleet_is_uint "$server_port" || { reply_error invalid_server_port; exit 0; }
[ "$server_port" -ge 31000 ] && [ "$server_port" -le 31999 ] || {
    reply_error invalid_server_port
    exit 0
}
"$FLEET_TUNNEL" open "$duration" "$server_port" >/dev/null

A command can choose how long the tunnel lives and which server port it uses, within bounds. It cannot choose the host, the user, the key or the forwarding target: those come from the enrolled configuration, and the target is always the device’s own 127.0.0.1:22. Replies are cached by command ID, so an MQTT QoS 1 duplicate does not reboot the device twice. Commands that change state are refused while the clock is unsynchronised, because expiry cannot be checked without one.

Shell and camera. The operator logs in through the server’s loopback port with a short-lived SSH certificate. The same tunnel carries the camera: an OV5647 on the Pi Zero W, H.264 over TCP, started and stopped by command. Through the tunnel, ffprobe read 1280×720 at 15 fps.

Tests on the Pi Zero W, with the device untouched after each fault:

Fault	Outcome
Device rebooted while the server was down for 5 minutes	same device ID, reconnected by itself, healthy telemetry 25 s after the server returned
Wi-Fi taken down and restored	reconnected without a reboot, healthy telemetry after about 39 s
Agent process killed with SIGKILL	restarted by systemd within 10 s, telemetry resumed
Tunnel opened with a 60 s lifetime	closed at the deadline, server listener gone, 0 failed units

Identity that survives a reflash

On the first fleet image, flashing a new build wiped the device’s identity, and it had to be enrolled again as a stranger. That makes updates impossible, so before any update mechanism I moved everything that makes a device that device onto its own partition (myproject-raspberrypi-state.wks):

bootloader --ptable msdos

part /boot --source bootimg-partition --fstype=vfat --label boot --active --align 4096 --size 130
part /     --source rootfs --fstype=ext4 --label root --align 4096
part state --source rawcopy --sourceparams="file=myproject-state-image-${MACHINE}.rootfs.ext4"
           --fstype=ext4 --label MYPROJECT_STATE --use-label --align 4096 --fixed-size 256

The state partition holds the device ID, the TLS and tunnel keys, the trust anchors, the Wi-Fi configuration, the SSH host keys and the duplicate-command cache. Two rules keep it honest. A marker separates factory-empty from provisioned, and a provisioned device whose state is missing or unreadable stops in a degraded mode: it never quietly generates a new identity. And the fleet services declare RequiresMountsFor on the mount, so they cannot start on an empty directory in the root filesystem.

The test was to overwrite only the root partition of an enrolled device and boot it. A guarded script does the write: it resolves the target with lsblk, refuses any disk the host itself runs from, and touches partition 2 only.

Keeping a device reachable without help

Device certificates last 90 days, and MQTT is the only way in: the shell is opened through it. So an unrenewed certificate is a lock-out with a date on it. The bench unit enrolled in September would have gone silent in early December. Renewal was the first thing to fix.

Renewal that still works after expiry. The device asks for a new certificate at two thirds of the lifetime. It proves who it is with the signing request itself: the request is signed by the device’s private key, and the server checks that key against the fingerprint it stored at enrolment. The old certificate is deliberately not required (fleet_server/devices.py):

bound_key = (record.get("binding") or {}).get("tls_public_key_sha256")
if not bound_key or public_key_fingerprint_from_csr(csr) != bound_key:
    raise EnrollmentError("CSR key does not match the enrolled key", 403)

A device that stayed offline past its certificate’s expiry comes back, renews with the key it has always had, and is reachable again. A revoked device is refused. A replayed request only yields a certificate for a key the replaying party doesn’t hold.

On the device, a new certificate replaces the old one only after it has been used once to reach the broker. A bad certificate from the server therefore can’t cut the device off. The exception is a certificate that has already expired: then there is nothing left to protect (fleet-renew):

fleet_mqtt_pub "$(fleet_topic events)" 1 0 "$event" && broker_ok=1
MQTT_CERT_FILE=$current_cert_file
if [ "$broker_ok" -eq 1 ]; then
    log "new certificate accepted by the broker"
elif [ "$expired" -eq 1 ]; then
    log "broker test failed, but the current certificate is expired; installing the new one"
else
    log "broker refused or was unreachable with the new certificate; keeping the current one"
    exit 1
fi

A health check that tests the real path. A process that exists isn’t proof that commands arrive. Once a minute the device publishes a ping to its own topic, and its command subscriber has to receive it. A broker acknowledgement counts as contact. When contact goes stale, the device escalates, one step at a time and each step rate-limited:

No contact for	Action
ping not echoed for 5 min, broker reachable	restart the agent (back-off up to 6 h)
15 min	restart networking
1 h	reboot, then 2 h, 4 h, … at most once a day

The reboot counter lives on the state partition, so a device whose server is gone for weeks reboots once a day instead of looping, and any contact resets it. All the timing uses seconds of uptime rather than the wall clock, which a board without a battery-backed clock can’t trust at boot. Underneath this, systemd drives the SoC’s hardware watchdog with a 10 s timeout, and a kernel panic reboots after 10 s. The time is saved to the state partition and restored at the next boot, only ever forwards.

since() {
    marker=$(fleet_marker_read "$1")
    if [ -z "$marker" ]; then
        echo -1
    elif [ "$marker" -ge "$now" ]; then
        echo 0
    else
        echo $((now - marker))
    fi
}

Results, first against the real server software with a simulated device and certificates that live two minutes, then on the Pi Zero W:

Test	Outcome
6 min soak, a command every few seconds	4 renewals, 44 commands, none lost
server stopped until the device’s certificate expired, then restarted	renewed and taking commands 13 s after the server returned
revoked device asks for renewal	refused, installed certificate unchanged
Pi Zero W: renewal against the live server	2.7 s; new certificate accepted by the broker before it was installed
Pi Zero W: ping through its own subscriber	echoed within 2 s
Pi Zero W: hardware watchdog	armed by systemd with a 10 s timeout

An FPGA as the camera

The Banana Pi M2 Zero has a parallel camera input (8 data lines, PCLK, HREF, VSYNC). I wired a Tang Nano 20K to it with eleven hand-soldered wires, with one goal: get data from FPGA logic into Linux through the SoC’s real capture path, sun6i-csi with DMA and V4L2. Sampling GPIOs would have been easier and would have proved nothing about speed.

Wiring first. Before any fast signal, the FPGA drove one line at a time while the others floated, and the Banana Pi read all eleven pins, once with pull-downs and once with pull-ups. All 11 lines arrived on the intended pin, with no swaps or bridges.

A sensor that isn’t there. The capture driver only starts when a sensor sub-device is linked to it in the media graph. I wrote a small V4L2 sub-device that negotiates the format and does nothing else (kernel/fpga_dvp_testsrc.c), and a device-tree overlay that links it to the CSI:

static int fpga_dvp_s_stream(struct v4l2_subdev *sd, int enable)
{
	/* The FPGA is already free-running; the callback only satisfies the
	 * normal bridge contract. */
	return 0;
}

With that, /dev/video1 behaves like a camera, and captures use the stock driver.

A stream that can be checked. The FPGA sends 32×32 frames with a 16-byte header (magic, pattern ID, clock rate, frame counter) and eight payload patterns in rotation: a counter, walking ones and zeros, 55/AA, 00/FF for maximum simultaneous switching, static levels and a pseudo-random sequence. The host regenerates each frame from its counter and compares every byte, with an error count per data line. Blanking is long on purpose: the edges run at the full pixel clock while the average data rate stays low, which separates wire and timing problems from throughput problems.

At 45 MHz the first full sweep showed 2,112 byte errors, and they were not spread out: every one was at the start of a line, and the rest of each line was clean. That pattern points at the first sample after HREF rises, not at the wires. The fix is in the FPGA: put byte 0 on the bus for one full clock with HREF still low, then open the line (stress/stress_top.v):

// Put byte zero on the bus for a complete low-HREF PCLK before
// opening the active window. This gives the CSI input pad a full
// cycle of setup at the line/frame boundary.
state <= STATE_ACTIVE_START;
href_r <= 1'b0;
byte_index <= 6'd0;
line_index <= 6'd0;
data_r <= stream_byte(frame_count, 6'd0, 6'd0);

After rebuilding all eight images, the sweep ran with the stock kernel driver:

PCLK	Frames	Bytes checked	Byte errors	Sync or dropped frames
1 MHz	64	131,072	0	0
5 MHz	64	131,072	0	0
10 MHz	64	131,072	0	0
20 MHz	64	131,072	0	0
27 MHz	64	131,072	0	0
36 MHz	64	131,072	0	0
45 MHz	64	131,072	0	0
54 MHz	64	131,072	0	0

The slow clocks come from a fractional divider on the board’s 27 MHz oscillator, the fast ones from the FPGA’s PLL. The whole flow is open source: Yosys, nextpnr and openFPGALoader, loading the FPGA’s SRAM only, with a Python harness that programs each image, captures over SSH and writes the report.

Where it started

The first version was a single image for the Pi Zero W on Yocto Kirkstone, built in Docker: join a known Wi-Fi network, accept SSH, nothing else. It taught me the two things the current project is built around. On a minimal systemd image nothing brings the interface up unless a unit says so, and the DHCP client has to wait for wpa_supplicant to associate. And credentials baked into an image at build time are the wrong place for them, which is why the current default image contains none and devices are provisioned at runtime. That repository is public and linked above.

The same image ran an IP camera on a two-servo pan-tilt mount, shown on the 3D prints & CAD page.

Where it goes next

The aim is a base that other projects build on and then leave unattended, so the remaining work is ordered by what could strand a device.

• A test rig that does the visiting. A Raspberry Pi 3 as the tester: it switches each board’s power, reads its serial console and acts as its Wi-Fi access point, so it can cut power at any moment, drop the network or block the clock source, thousands of times, and count the boots that fail to come back. Each test has a pass criterion a script can check.
• Storage that survives power cuts. A read-only root filesystem and checked state files, measured on that rig. The target is zero failed boots.
• Updates with rollback. Two system slots behind U-Boot, with signed bundles. A new system is confirmed only after it has reached the server by itself; one that cannot is rolled back, which becomes the last step of the escalation above. The U-Boot image builds and passes its artefact checks, and booting it on the board is the next step.
• A release gate. The rig’s fault tests before every release, plus devices that run for weeks with nightly updates.

Board specifics stay in the BSP layers and a short machine file, so the same base can move to an industrial board when a deployment calls for one. On the FPGA side the sweep covered edge timing with short frames; the next runs raise the duty cycle towards real video frames, and the receiver module moves into the Yocto build for the Banana Pi.