What should run on HIL instead of host tests or QEMU?

Run only behaviour that exists on real silicon on HIL: drivers, sleep and wake, brownout, the OTA bootloader path, RS-485 turnaround, and the network stack against a real access point. Logic belongs in host unit tests. Boot flow and panic handling can run in QEMU.

Why does power-cycling over USB fail to reset my ESP32 board?

With VBUS cut, the board can still be powered through the USB-UART TX line into the ESP32 RX pin, or by an external supply, so the chip half-boots. To fix it, switch the board's main supply, or add series resistance or a buffer on the UART lines.

Should CI automatically retry failed HIL tests?

Retry only classified infrastructure errors, such as a missing port, an esptool connection failure or a failed rig health check, and power-cycle the board before retrying. Never retry an assertion failure. Quarantine a flaky test with an owner and an expiry date instead of adding blanket reruns.

Which OTA rollback cases belong in the merge request gate?

Run a good image that is marked valid and survives a second reboot, plus the core rollback cases: an image that panics at boot, one that fails its self-test, and one that hangs before marking itself valid. Build the bad images from the same commit.

ESP32 Hardware-in-the-Loop Testing: A CI Pipeline for ESP-IDF

Most ESP32 teams have unit tests, but few have a CI job that can prove the OTA rollback path still works on the commit they're about to merge. Field failures come from that gap: a device bricks after a power cut, a gateway stops reconnecting after the broker kicks it. None of that shows up on a laptop.

This post describes how we build ESP-IDF hardware-in-the-loop (HIL) pipelines at GizanTech: what to test on real silicon, how to wire the rig so it doesn't lie to you, and which tests run on every merge request (MR).

What hardware-in-the-loop testing is, and where it fits

Hardware-in-the-loop testing is automated testing in which CI flashes firmware onto real target hardware and checks its behaviour against simulated peripherals, networks and injected faults.

It's one layer of a pyramid, and it is the slowest and most fragile one. The rule we follow is to push every test as far down the pyramid as it will go.

Layer	What it catches	What it misses
Host unit tests (ESP-IDF `linux` target, preview, plus CMock)	Logic bugs, quickly and deterministically	ISR timing, DMA, cache and flash interaction, RF, power, bootloader
QEMU (Espressif fork: ESP32, ESP32-C3, ESP32-S3)	Boot flow, partition table, panic handling	Most peripherals, the Wi-Fi radio, real timing
HIL on real silicon	Drivers, sleep and wake, brownout, the OTA bootloader path, RS-485 turnaround, the network stack against a real AP	RF quality, EMC, anything irreversible
Manual bench	RF, EMC, thermal, eFuse burns	Regressions between releases

HIL minutes are the scarcest in the pipeline. Don't spend them on a parser test.

The tooling stack

pytest-embedded and Unity

Espressif's pytest-embedded plugin family is the backbone. Its services (esp, idf, serial, jtag, qemu, wokwi) give you a dut fixture. dut.expect() matches a regex and dut.expect_exact() matches a literal string. dut.run_all_single_board_cases() drives the Unity test menu over serial. You select hardware with --target, --port, --app-path and --build-dir.

On the firmware side, ESP-IDF's Unity integration registers tests with TEST_CASE("name", "[tag]") and exposes them through unity_run_menu(). Two macros matter most for HIL:

TEST_CASE_MULTIPLE_STAGES for tests that cross a reset: deep sleep, restart, OTA reboot.
TEST_CASE_MULTIPLE_DEVICES for master/slave scenarios across two boards.

Tags such as [ignore], [leaks] and your own custom tags separate the MR suite from the nightly suite.

Put each test app in components/<x>/test_apps/, with a pytest_*.py beside it. Give each config variant its own sdkconfig.ci.<variant> file and let idf-build-apps find and build the whole matrix. Output --junitxml so that GitLab MR test reports and GitHub check annotations show results without extra tooling.

Crash capture that fails the run

A passing regex doesn't prove the board stayed healthy. Fail the run globally whenever the log contains Guru Meditation Error, abort() was called or Task watchdog got triggered. Do the same for Brownout detector was triggered unless the test expects a brownout. Keep the ELF as a build artifact so you can decode backtraces later. Enable CONFIG_ESP_COREDUMP_ENABLE_TO_FLASH and run idf.py coredump-info to get post-mortems for resets that happened while serial wasn't watching.

Split build from test

Build in a container job using the espressif/idf image with a pinned tag, and pass the binaries on as artifacts. The HIL job flashes and tests, and nothing else. Never compile on the rig host.

Rig architecture: where most flakiness comes from

Runners and isolation. Route HIL jobs by runner tag on GitLab (for example hil-esp32s3) or by self-hosted label on GitHub. Never let fork PRs run on a self-hosted HIL runner. A fork PR on that runner is arbitrary code executing on a host with USB access to your hardware.

Exclusive access. Give each rig a GitLab resource_group or a GitHub concurrency group. Without one, two jobs can flash the same port at once and corrupt each other without any error.

Stable port names. /dev/ttyUSB* renumbers every time a device re-enumerates. Address boards through /dev/serial/by-path/..., which follows the physical hub port, or through udev rules keyed on the USB serial number. Many CH340 adapters have no unique serial number, so for them by-path is the only stable handle. If the runner uses containers, bind-mount /dev and add a cgroup device rule rather than passing a static --device. Native USB parts re-enumerate during a job, and the node a static --device pointed at can vanish.

Native USB re-enumeration. On the S2, S3, C3 and C6, the USB-Serial-JTAG port disappears on reset, panic and deep sleep, then comes back. The harness has to reopen the port and tolerate the gap. Anything the chip prints during re-enumeration is lost. For boot-critical assertions, tap UART0 through a separate USB-UART adapter.

Power: control the rail, not the USB cable

Per-port power switching with uhubctl only works on hubs that implement real per-port power switching. Many hubs advertise the feature but switch all ports together (ganged switching), or never actually cut VBUS. Check with a meter before buying in bulk, or use a MOSFET or relay on VBUS driven by a GPIO expander.

Switching only USB VBUS has a deeper problem. With VBUS off, the board can still draw power through the adapter's TX line into the ESP32's RX pin, or from an external supply. The chip then half-boots or never fully resets, and your power-cycle recovery does nothing. Switch the board's main supply instead, or add series resistance or a buffer on the UART lines. Treat the PSU as a test instrument.

Current headroom matters too. A USB 2.0 port supplies 500 mA and a USB 3.x port 900 mA, while ESP32 Wi-Fi TX peaks reach several hundred mA. When several boards transmit at once on a weak or bus-powered hub, the hub causes brownouts that look exactly like firmware bugs. Use powered hubs and spread the boards across them.

Flashing without false alarms

The ROM bootloader prints at 115200 baud with a 40 MHz crystal. ESP32 boards with a 26 MHz crystal print ROM output at 74880, which looks like garbage if the port is open at the wrong speed. Flash at 921600 or higher with compression, which is esptool's default. CH340 parts and other cheap adapters get unreliable at high speed, while CP2102N and FTDI parts hold up better.

Flash only the partitions that changed. NOR sectors survive roughly 100k erase cycles, so a full chip erase on every run wastes time and wears out rig boards.

Treat failed to connect and timed out waiting for packet header as infrastructure failures: power-cycle the board and retry. Never retry a failed test assertion.

Simulated peripherals: Modbus RTU as the worked example

Industrial ESP32 products usually depend on RS-485. The Modbus RTU details that matter:

A character is 11 bits: start, 8 data, parity or a second stop bit, and stop.
Frames are delimited by t3.5, and the maximum gap between characters is t1.5. Above 19200 baud the spec fixes these at 1.75 ms and 750 µs.
Addresses run from 1 to 247, and 0 is broadcast. The maximum PDU is 253 bytes and the maximum RTU ADU is 256 bytes.
The CRC-16 uses polynomial 0xA001 (0x8005 reflected) and is sent low byte first.

In ESP-IDF, UART_MODE_RS485_HALF_DUPLEX lets RTS drive DE/RE in hardware, and uart_set_rx_timeout(), measured in symbol times, is how you tune t3.5 detection. The bugs to hunt are DE released too early, which truncates the last byte, and DE released too late, which collides with the slave's reply.

On the other end of the bus, run a pymodbus slave on a USB-RS485 adapter and script it per test to:

Delay the response past the master's timeout.
Corrupt the CRC.
Return each exception code: 01 illegal function, 02 illegal data address, 03 illegal data value, 04 slave device failure.
Drop the response entirely.
Answer from the wrong address.
Split a frame with a gap longer than t1.5.

Test the simulator before you trust it. FTDI's default latency timer is 16 ms. Unless you set it to 1 ms through sysfs latency_timer, the simulator itself breaks t3.5 timing, and you end up chasing a DUT bug that doesn't exist. Use the TIOCSRS485 ioctl where the adapter supports it. Check the harness once with a cheap logic analyzer and sigrok/PulseView on DE and TX, then pin the adapter model and latency setting the same way you pin the IDF version. Terminate both ends with 120 Ω and add failsafe biasing. Even a one-metre unbiased bench bus produces framing errors while idle.

Fault injection: the reset path is the product

Brownouts, watchdog resets, panics, OTA rollbacks and power cuts during writes all end with the same question: what happens on the next boot? Multi-stage tests that assert the reset reason, the persisted state and the running partition after each fault catch the field-only bugs that host tests can't see.

Brownout. Set the threshold with CONFIG_ESP_BROWNOUT_DET_LVL_SEL. The supply is nominally 3.3 V with a 3.0 to 3.6 V operating range. Sag the rail with a programmable PSU (a SCPI ramp or dip), or with a MOSFET and series resistor during Wi-Fi TX. On the next boot, assert esp_reset_reason() == ESP_RST_BROWNOUT, check that NVS state survived, and confirm the device doesn't boot-loop.
Power cut during writes. Cut power at random offsets during NVS writes and during OTA download and flash write, hundreds of times per nightly run. NVS itself is designed to tolerate power loss. Your multi-key transaction layer on top of it usually isn't.
Flash corruption. Overwrite an OTA slot, NVS, otadata or a data partition using esptool's write_flash or erase_region. The bootloader checks the image header, checksum and appended SHA-256, and it should reject a corrupt app. For NVS, exercise the erase-and-reinit recovery path after ESP_ERR_NVS_NO_FREE_PAGES or ESP_ERR_NVS_NEW_VERSION_FOUND. Flash encryption makes raw writes meaningless, so run these tests on unencrypted CI builds and record that gap explicitly.
Network chaos. Run hostapd on a USB Wi-Fi dongle, with Mosquitto and an HTTP server behind toxiproxy or tc netem. Inject a vanishing AP, a silent DHCP server, DNS failure, a stalled TLS handshake, a broker kick and a half-open TCP connection. Assert reconnection within your SLA, bounded backoff, and no heap growth across N cycles.
Power states. Assert the wake cause and RTC-retained memory with TEST_CASE_MULTIPLE_STAGES. The chip's deep-sleep current is around 10 µA, but dev boards read far higher because of the LDO, LEDs and USB-UART. Assert absolute µA only on product boards. On dev boards, assert relative regressions.

OTA rollback, on every merge request

Enable CONFIG_BOOTLOADER_APP_ROLLBACK_ENABLE. A new image boots in PENDING_VERIFY. The app must call esp_ota_mark_app_valid_cancel_rollback() after its self-test passes, or esp_ota_mark_app_invalid_rollback_and_reboot() if it fails. If the device resets while the image is still pending, the bootloader marks it ABORTED and boots the previous slot.

The MR matrix we run:

A good image downloads, reboots, passes its self-test, is marked valid, and survives a second reboot.
An image that panics at boot rolls back automatically.
An image that boots but fails its self-test calls invalid and rolls back.
An image that hangs before marking itself valid gets a watchdog reset and rolls back.
A truncated or corrupted download is rejected, and the running slot is untouched.
A power cut mid-download or mid-otadata write still leaves the device booting a valid image.
An image for the wrong chip or project is rejected.

Build the bad images in CI from the same commit, using sdkconfig or compile-time fault flags, so rollback is tested against this MR's bootloader and partition table. Keep anti-rollback eFuses, Secure Boot v2 and release-mode flash encryption off the fleet rig. eFuse burns are irreversible, so test those on a labelled sacrificial board at release time.

Tiering and run time

MR gate: aim for 10 to 20 minutes wall-clock. That covers builds in parallel, flashing each test app, Unity smoke suites, the core OTA good and rollback cases, and the Modbus happy path plus one fault.
Nightly: the full Unity menu across the sdkconfig matrix, every power-cut loop, every Modbus fault case, and network chaos.
Weekly soak: multi-hour runs that track heap and reconnect statistics over time. Leak and fragmentation bugs only show up at that length.

Run time is dominated by flashing, resets, boot waits and network timeouts, not by test logic. Parallelise across rigs, never within one, and shorten timeouts with CI-only Kconfig values.

Hardware is the cheap part

A useful single-target rig is a runner host, dev boards with spares for flash wear, a switchable hub, a programmable PSU, a power profiler, USB-RS485 adapters, a logic analyzer, helper MCUs and a Wi-Fi dongle. The engineering time needed to make it reliable outweighs all of that hardware.

Leave these on the bench

RF and antenna tuning, certification, ESD and EMC, RS-485 surge, thermal cycling, eFuse and Secure Boot provisioning, long multi-drop cable runs, and anything that needs a human eye. Automate a proxy where you can, such as an RSSI sanity bound or an LED GPIO state, but don't treat it as the real test.

The point that matters most

A HIL failure is only useful if you can tell a rig failure from a firmware failure. Run a pre-suite health check: flash a known-good image, confirm it boots, and check the PSU and hub. If the check fails, mark the rig offline instead of failing the MR. Quarantine flaky tests with an owner and expiry; no blanket --reruns. Until that classification is reliable, developers learn to press retry and the gate is effectively off.

If you want help designing a rig or a pipeline like this for an industrial ESP32 product, our firmware development team builds them, and our industrial IoT solutions show where they fit in a product.