ESP32 Core Dump Analysis: Remote Crash Diagnostics for Fleets

A backtrace on a serial console is a bench tool. It stops being useful once the board is bolted into a pump station or a roadside cabinet. Nobody has a laptop plugged in when a field unit panics at 3 a.m. On most devices the panic output goes nowhere, the watchdog reboots the unit, and all you have is a gap in telemetry.

My position is simple. A fleet crash pipeline has to do four things: keep the dump across the reboot, prove which build produced it, get it to a server safely, and decode, group and act on it without a person in the loop. If any one of these is missing, you are guessing.

This post covers each stage using ESP-IDF's own mechanisms, and points out the places where teams usually go wrong.

Stage 1: Capture the dump on the device

Pick flash, ELF and SHA-256

ESP-IDF can send a core dump to three destinations: CONFIG_ESP_COREDUMP_ENABLE_TO_FLASH, TO_UART or NONE. UART only helps when someone is watching the port, which in practice means the bench. Fleets need flash.

Choose the ELF data format (CONFIG_ESP_COREDUMP_DATA_FORMAT_ELF). The legacy binary format is deprecated. More importantly, only ELF supports esp_core_dump_get_summary() and embeds the app ELF SHA note, and the rest of this pipeline depends on both. ELF offers CRC32 or SHA-256 checksums. Choose SHA-256: the dump is about to cross a network and several storage layers, and you want integrity checking stronger than a CRC.

Add the partition yourself

The stock single-app and OTA partition tables have no coredump partition, so you need a custom CSV with a line like this:

coredump, data, coredump, , 64K

The type is data and the subtype is coredump (0x03). The partition must be aligned to a 4 KB sector. Turn on CONFIG_ESP_COREDUMP_CHECK_BOOT so the stored image is validated at startup.

What happens at panic time

The panic handler writes the dump with interrupts disabled, then resets according to CONFIG_ESP_SYSTEM_PANIC. For fleets, use print-and-reboot. Erasing and writing flash adds reboot latency, and the latency grows with dump size, so a larger dump means a longer outage on every crash. Set CONFIG_ESP_COREDUMP_STACK_SIZE so the dump writer runs on its own stack. Otherwise a stack overflow in the crashing task can also stop the dump from being written.

After boot, the application works with the stored dump through a small API:

• esp_core_dump_image_check() checks whether a valid dump exists.
• esp_core_dump_image_get(&addr, &size) returns its location and size.
• esp_core_dump_image_erase() clears it once the server has acknowledged receipt.
• esp_core_dump_get_summary() fills an esp_core_dump_summary_t with the crashing task, PC, backtrace info, app ELF SHA, dump version and exception registers.

The summary differs by architecture. On Xtensa parts (ESP32, S2, S3) it contains a backtrace of up to about 16 PCs and a "corrupted" flag. On RISC-V parts (C3, C6, H2) it contains a raw stack dump, not PCs. Unwinding then happens on the server using DWARF, unless you build with CONFIG_ESP_SYSTEM_USE_EH_FRAME. Your ingestion code has to handle both shapes.

Crashes that leave no dump

In some cases no dump is written at all:

• Brownout resets (ESP_RST_BROWNOUT) and hard power loss.
• Faults that break the flash or cache path during the panic. "Cache disabled but cached memory region accessed" is the classic example.
• A dump write that fails for its own reasons.

Every crash still produces a reset reason. Log esp_reset_reason() as its own fleet metric, independent of the dump pipeline. If brownout resets on one hardware revision are rising, you want to see that even though no core dump will ever show it.

Breadcrumbs

A stack trace tells you where the device died, not what it was doing just before. COREDUMP_DRAM_ATTR and COREDUMP_RTC_ATTR place chosen variables inside the captured region. Separately, an RTC_NOINIT_ATTR ring buffer of recent state-machine events survives a soft reset, so you can send it along with the dump. Keep the entries small and meaningful: state transitions, not log lines.

Stage 2: Size the partition by measurement

Dump size is roughly the number of tasks times (task stack plus TCB), plus headers and notes. CONFIG_ESP_COREDUMP_MAX_TASKS_NUM limits the task count.

The tempting option is CONFIG_ESP_COREDUMP_CAPTURE_DRAM, which adds .data, .bss and the heap. The post-mortem is much better, but the dump grows by an order of magnitude, and heap memory contains data you should not be shipping off the device (see Stage 6). My advice: leave it off in production.

64 KB covers typical stack-only apps. Apps with many tasks or large stacks need more. Measure instead of guessing:

1. Build firmware with the fleet's real task set and real stack sizes.
2. Force a panic on a bench unit.
3. After reboot, read size from esp_core_dump_image_get().
4. Add headroom for the tasks you expect to add over the product's life.
5. Commit the partition CSV. Changing it later means a partition-table migration across the fleet.

If you skip this, the dump is truncated or refused with "Not enough space…", and you learn about it from the one field crash you most needed to understand.

The partition holds only one dump. In a crash loop, each new dump overwrites the previous one, so the first crash, which is usually the informative one, is lost. Keep a crash counter and the reset reason of the first crash in NVS or RTC memory so you can still tell how the loop started.

Stage 3: Identify which build produced the dump

All the steps that follow depend on one field.

Every app image embeds an esp_app_desc_t containing version[32] (from PROJECT_VER), project_name, time, date, idf_ver, secure_version and app_elf_sha256[32]. The ELF core dump also carries that same app ELF SHA-256 in its info note, and espcoredump.py warns when it does not match the ELF passed with --prog.

The app ELF SHA-256 is the primary key of the whole diagnostics system.

The hash is computed from the unstripped ELF before image generation and before Secure Boot v2 signing, so signing does not change it. The signed .bin has a different hash. Store both, but index on the ELF SHA.

For people reading dashboards, put the git SHA, the CI pipeline ID and a dirty flag into PROJECT_VER. It is limited to 32 characters, so abbreviate. Turn on CONFIG_APP_REPRODUCIBLE_BUILD so path and timestamp differences are removed and a rebuild can be checked against the stored ELF SHA.

A hard rule for CI: if uploading the symbol artefacts fails, the build fails. A dump without a matching ELF cannot be read. A dump decoded against the wrong ELF gives symbols that look right and are wrong, which costs days of debugging in the wrong place. That is worse than having no symbols.

Stage 4: Upload after reboot, carefully

When to upload

On boot, queue an upload if esp_core_dump_image_check() passes or if the reset reason is PANIC, INT_WDT, TASK_WDT or WDT. Don't upload from the start of app_main. If the upload code path is itself what crashes, you have built a boot loop. Wait until the network is up and the main loop has run cleanly for a while.

Two tiers

The summary is enough to group crashes. For most known bugs you never need the full dump, and when the full dump is blocked by bandwidth limits, privacy policy or a crash loop, the summary still gets through.

Chunking, compression, idempotency

MQTT allows payloads up to 256 MB, but that number doesn't matter. The broker limit does: AWS IoT Core caps messages at 128 KB. Split the dump into chunks of a few KB. Give each chunk {dump_id, offset, total_len, chunk_crc} and send a SHA-256 of the whole dump. Alternatively, send an HTTPS PUT to a pre-signed object-store URL so reassembly doesn't have to happen behind the broker.

Stack memory compresses well. Use a small streaming compressor so the device never has to hold the whole dump in RAM.

Set dump_id to the hash of the dump, and have the server deduplicate on it. Erase only after the server acknowledges. A successful publish is not an acknowledgement. If the device crashes mid-upload, the dump is still in flash and the next attempt picks it up.

Also protect the backend from your own fleet. Use jittered exponential backoff and a per-device daily cap. A bad OTA that crashes every unit will otherwise make the whole fleet upload at once and flood your own ingestion endpoint.

Stage 5: Decode and group on the server

The symbol store

For each signed OTA build, keyed by app_elf_sha256, store:

• the unstripped app ELF
• the bootloader ELF
• sdkconfig
• the partition table
• the IDF version and toolchain version
• the chip target

Also store the ROM ELF for each chip and silicon revision, passed with --rom-elf. Without it, frames inside ROM functions will not resolve.

Decoding

Decode with esp-coredump / espcoredump.py info_corefile --core <file> --core-format elf|raw|b64 --prog app.elf, or with idf.py coredump-info and coredump-debug. Under the hood this runs GDB (xtensa-esp32*-elf-gdb or riscv32-esp-elf-gdb). Run it in a container pinned to the toolchain that built that ELF. The output includes per-task backtraces, registers, and locals where DWARF information allows.

Classify by exception cause

Keep separate classes for Interrupt WDT, Task WDT, "Stack canary watchpoint triggered" (stack overflow), CORRUPT HEAP, and abort()/assert. They have different root causes and need different fixes.

The stack-hash signature

A crash signature is a hash of symbolized application frames, the exception class and the task name, so one bug maps to one group across every build that contains it.

1. Symbolize every frame to a function name. Never hash raw addresses, because they change with every build.
2. Remove generic frames: the panic handler, abort, __assert_func, esp_system_abort, vPortTaskWrapper and similar.
3. Take the top N remaining application frames.
4. Add the exception class and the task name.
5. Hash the result.

Don't trust damaged unwinds. Xtensa's |<-CORRUPTED marker and truncated backtraces go into an "unwind-failed" group keyed by PC and function. Without that group, one memory-corruption bug can produce thousands of unique signatures and bury the real trend.

Stage 6: Flash encryption and privacy

Mark the coredump partition encrypted in the CSV so the dump is not stored in plaintext. On the device, esp_partition_read() decrypts transparently, so the app uploads plaintext. Off the device, esptool read_flash returns ciphertext. In practice, flash encryption means the upload path is your only diagnostic channel. Reading the flash on the bench no longer tells you anything.

Encryption at rest is not encryption in transit. Use TLS, and mutual TLS for MQTT. For stronger protection, encrypt the dump on the device to a server public key (ECDH plus an AEAD cipher) so brokers, logs and queues only ever see ciphertext.

Security note: a decoded dump contains live memory. Stacks and heap can hold Wi-Fi PSKs, MQTT/TLS credentials and session keys inside mbedTLS contexts, tokens, customer payloads, MAC addresses and location. Treat every raw dump as a secret.

The redaction policy I recommend:

• Upload summaries only by default, and make full dumps opt-in per device.
• Leave CAPTURE_DRAM off in production.
• Zero secret buffers after use, and keep long-lived secrets outside the captured regions.
• On the server: decode, keep the symbolized report, delete the raw dump after a short retention window, restrict access, encrypt at rest and audit every read.

If you're working through the rest of the device security posture, it's covered here:

Stage 7: Feed crashes into OTA rollout gates

On-device rollback covers the early window

With CONFIG_BOOTLOADER_APP_ROLLBACK_ENABLE, a new image moves through NEW → PENDING_VERIFY → VALID / INVALID / ABORTED. If the device resets before calling esp_ota_mark_app_valid_cancel_rollback(), the bootloader goes back to the previous image. That only catches early failures. Mark the image valid after real health checks pass, such as a cloud connection and a stable main loop, not as the first line of app_main.

The fleet gate covers the later window

A crash that first appears six hours after boot never reaches the bootloader's rollback logic. That is the fleet gate's job:

• Roll out to staged cohorts.
• For each build, compute the crash rate normalized by device-hours of exposure and the crash-free-device rate. Compare them with the incumbent build on matched hardware and region.
• Require a minimum exposure before deciding anything, and use a proper rate test such as a Poisson test. Small cohorts produce noisy rates, and noisy rates cause wrong rollbacks.
• Treat a new signature that appears only on the new build as a stronger signal than any rate change. This is why the signature pipeline in Stage 5 matters.

When the gate trips, it should halt the rollout automatically, then roll back fleet-wide by pointing OTA at the last good build.

The anti-rollback trap

With CONFIG_BOOTLOADER_APP_ANTI_ROLLBACK, devices reject any image whose secure_version is lower than the value burned into eFuse. Raise secure_version only for security fixes. Otherwise, "rollback" means rebuilding the old code with the current secure_version and shipping that. The rebuild has a new ELF SHA, so it needs its own symbol-store entry. Build this "roll forward to old code" release path before you need it.

Key takeaways

• The app ELF SHA-256 is the primary key. Archive the unstripped ELF and ROM ELF before signing, and fail CI if that upload fails.
• Send the summary first and the full dump second. A summary of a few hundred bytes is enough for grouping and gating, even during a crash loop.
• Flash encryption makes the uplink mandatory, and the uplink carries secrets. Design encryption, transport and retention together.
• Local rollback catches boot-time faults. Only fleet crash-rate gating with minimum exposure catches crashes that appear hours later.
• Erase the dump only after a server acknowledgement, and cap uploads so a bad OTA can't flood your own backend.

Where to start

If your fleet has nothing yet, do it in this order: the coredump partition and ELF/SHA-256 config, then PROJECT_VER and symbol-store upload in CI, then the summary-record upload with ack-before-erase, then server-side signature grouping. Full-dump transfer and rollout gating come after that. Each step is useful by itself, and each one depends on the step before.

At GizanTech we build this pipeline into the ESP32 products we ship, because our industrial clients' devices are installed where nobody can reach them with a cable. If you're planning a fleet and want crash diagnostics designed in from the start, our firmware development team can help. The same thinking applies across industrial IoT systems more generally: plan for failures in the field, because every deployed fleet will have them.