Files

T

evan 42c96a31a2 more

2026-07-20 16:34:19 -04:00

18 KiB

Raw Permalink Blame History

Handoff — xteink-web-emulator

Everything below is hard-won context from the session that built the native X3 emulator. If it isn't in this repo or this doc, it's gone. Read this before touching QEMU internals. Pairs with ROADMAP.md (the plan) — this is the "how it actually works and what will bite you" file.

0. TL;DR of current state

One xteink QEMU machine selects variant=x3|x4 at launch. Unmodified canonical firmware detects X3 through I²C fingerprints and boots the home screen; X4 detects through all-NAK probes and reaches a blank SSD1677 stub.
Phase-1 device models live in qemu/patches/0001-xteink-machine.patch; 0002-qemu-wasm.patch ports qemu-wasm's host backend onto the same pinned Espressif QEMU commit 40edccac415693c5130f91c01d84176ae6008566.
make qemu-wasm produces a smoke-tested ES module, WASM binary, and pthread worker under dist/wasm/; the static browser UI boots firmware, renders X3, drives buttons, and exposes SD files/debug metrics. Full X4 rendering waits.

1. Environment / hardware realities (READ FIRST)

A full QEMU build is large and can take several minutes. Build parallelism defaults to nproc; set JOBS lower if memory pressure requires it and use long timeouts for any ninja/pio/nix build. Do not treat a long-running build as hung.
First full QEMU build ≈ many minutes; incremental after a patch edit is seconds (only the touched .c + relink). Iterate incrementally; never rm -rf build.
Nix sandbox capability leak: running FHS/bwrap-wrapped tools (PlatformIO, and the prebuilt release QEMU) fails with bwrap: Unexpected capabilities but not setuid, old file caps config?. Workaround — clear ambient caps:
```
SP=$(nix build nixpkgs#util-linux.bin --no-link --print-out-paths)/bin/setpriv
"$SP" --inh-caps=-all --ambient-caps=-all nix develop <flake> --command <cmd>
```
Our source-built QEMU is a normal nix-linked ELF and does NOT need this (that's a bonus reason we build from source, not just to patch).

2. Building espressif-qemu from source (the fiddly part)

scripts/build-qemu.sh encodes all of this, but here's why:

Reuse nixpkgs' QEMU build env: nix develop nixpkgs#qemu --command … (or the repo devShell, which has inputsFrom = [ pkgs.qemu ]). Getting glib/ pixman/meson/ninja right by hand is painful; borrow qemu's.
slirp: the fork unconditionally compiles net/slirp.c even with --disable-slirp (build-system bug), so you must enable it. nixpkgs provides libslirp; --enable-slirp + libslirp on PKG_CONFIG_PATH works, or it falls back to the bundled subproject.
libgcrypt: the ESP32-C3 crypto devices (esp32c3_aes.c, rsa, ds, xts) are gated on gcrypt.found(). QEMU prefers gnutls and skips gcrypt unless you pass --enable-gcrypt. Without it the build succeeds but the machine fails at runtime: unknown type 'misc.esp32c3.aes'.
Meson caches pkg_config_path: exporting PKG_CONFIG_PATH after the first configure is ignored on reconfigure. Use meson configure build -Dpkg_config_path=… or pass it to ./configure.
Config that works: --target-list=riscv32-softmmu --enable-gcrypt --enable-slirp --disable-werror --disable-docs --disable-tools.
/tmp/qsrc is where I built it this session and is EPHEMERAL (lost on reboot). The patch is the durable artifact; make qemu reclones+rebuilds.

3. How espressif-qemu's esp32c3 machine is structured

File: hw/riscv/esp32c3.c. It hand-builds the SoC (no device tree).
Catch-all MMIO: esp32c3_io_ops (esp32c3_io_read/write) covers the whole peripheral window from ESP32C3_IO_START_ADDR (== DR_REG_UART_BASE == 0x60000000) for 0xd1000 bytes. Any peripheral without a dedicated device falls here and reads 0 / ignores writes. That's why unmodeled peripherals silently misbehave instead of faulting.
Turn on #define ESP32C3_IO_WARNING 1 (top of the file) to log every unsupported access as [ESP32-C3] Unsupported read/write to $ADDR. This is the single most useful debugging lever. It's noisy (megabytes/sec), so run ~12 s and bucket by address:
```
grep 'Unsupported read' log | sed -E 's/.*\$//' | sort | uniq -c | sort -nr | head
```
The hottest address is almost always the next thing to model. Leave it 0 in committed code.
Devices that already exist in the fork: UART, GPIO (barebones — we fleshed it out), SPI1 (flash only), GDMA, AES/SHA/RSA/HMAC/DS/XTS, RTC_CNTL, systimer, timg, efuse, intmatrix, cache, JTAG, TWAI, esp_rgb (a QEMU-specific virtual framebuffer, NOT real X3 hardware — we disable it on the xteink machine).
Devices that did NOT exist and we added: SAR ADC, SPI2, the UC8253 panel, and real GPIO register behavior. There is no I²C controller model — this matters for X3/X4 detection (see §7).

4. Peripheral register facts (discovered, verified against firmware)

Base addresses from ESP-IDF soc/esp32c3 headers; all confirmed live.

SAR ADC — `DR_REG_APB_SARADC_BASE = 0x60040000`

The original boot blocker: firmware polls INT_RAW (+0x44) bit 31 (ADC1_ONESHOT_DONE) after starting a conversion; the unmodeled reg returned 0 forever → busy-loop with interrupts off → interrupt-WDT panic (Guru Meditation … Interrupt wdt timeout). The panic dump's S0/FP = 0x60040000 literally pointed at the ADC base.
Model (hw/adc/esp32c3_adc.c): INT_RAW always returns done bits 31|30; ONETIME_SAMPLE (+0x20) bits [28:25] select the channel; 1_DATA_STATUS (+0x2c) / 2_DATA_STATUS (+0x30) return the selected channel's 12-bit value. Per-channel values are QOM props adci[*], default 0xfff.
adc_oneshot_ll_get_raw_result reads adc1_data & 0xfff.

SPI2 (display + SD) — `DR_REG_SPI2_BASE = 0x60024000`

Completely different register map from SPI1 (hw/ssi/esp32c3_spi.c is flash-specific — do NOT reuse it). GP-SPI2 layout: CMD 0x00 (USR = bit24, UPDATE = bit23), MS_DLEN 0x1c (bits[17:0] = bitlen-1), data buffer W0..W15 at 0x98..0xd4.
Model (hw/ssi/esp32c3_spi2.c): on CMD write with USR set, clock (MS_DLEN/8)+1 bytes out of W0.. onto an SSIBus via ssi_transfer, writing RX back into the same words. DMA path is intentionally unimplemented — the firmware's CPU-buffer path is enough; add DMA only if a future firmware needs it (it'll show up as SPI2 traffic that the CPU-buffer model can't explain).

GPIO — `DR_REG_GPIO_BASE = 0x60004000`

The stock model only returned the strap register. We added real OUT 0x04 / OUT_W1TS 0x08 / OUT_W1TC 0x0c / ENABLE 0x20 (+W1TS/W1TC) / IN 0x3c, with 32 named input + output qemu_irq lines (ESP32_GPIO_INPUT/ESP32_GPIO_OUTPUT).
CRITICAL SUBTLETY: an output line must read HIGH when the pin is configured as input (output_enable bit clear), not follow the (zero) OUT latch. Real boards have pull-ups; the X3 display CS is pulled high while SD is being probed. Without this, CS floated LOW during SD init and the panel greedily consumed the SD command bytes (FF … 40 00 … 95) as garbage commands. This one bit cost real debugging time.

SD-over-SPI — QEMU's `ssi-sd` + `sd-card-spi`

These exist upstream; we attach them to SPI2 at CS12. But the 9.2 fork's ssi-sd.c has a bug: CMD58 (read OCR) hardcodes the R1 byte to 0x01 (idle) even after ACMD41 reported ready → SdFat rejects the card → firmware shows "SD card error". Fix: track idle state, clear it when ACMD41 returns 0, return it for CMD58. (Upstream qemu master rewrote this path entirely; we did the minimal equivalent + a vmstate version bump to 8.)

5. The e-ink panel model (`hw/display/xteink_x3_eink.c`)

X3 uses a UC8253 controller, 792×528, 99 bytes/row, 52 272 bytes/plane.
Protocol (from chip/eink-x3.chip.c, the old Wokwi model — keep it as the reference): DC low = command, DC high = data. 0x10=DTM1 (old plane), 0x13=DTM2 (new plane), 0x12=refresh (render + pulse BUSY), 0x04/0x02= power on/off (pulse BUSY). Bit 1 = white, 0 = black. Rows arrive bottom-to-top (wire row i → framebuffer row H-1-i).
BUSY is a GPIO output from the panel back into GPIO6 (active low; we pulse it low for 5 ms of virtual time via a QEMUTimer, then high).
Rendered to a QEMU graphical console (not a custom file callback) so the WASM front-end reuses QEMU's display surface. Capture headless with QMP screendump → PPM.
Orientation: the controller framebuffer (792×528 landscape) is rotated 90° clockwise to the physical portrait panel (528×792). We bake that into the render loop (screen_x = H-1-y, screen_y = x) so the console is already physical-orientation. Verified by eye: text reads upright at 528×792.
SSI gotcha: a TYPE_SSI_PERIPHERAL subclass must set ssi->realize even if empty — the base class calls it unconditionally and a null pointer segfaults at machine init. Also set cs_polarity = SSI_CS_LOW and user_creatable = false.

6. Machine wiring (`xteink_machine_init`)

xteink is a subclass of the esp32c3 machine. variant=x3 is the default; variant=x4 selects the sibling wiring before boot. After the base init it:

creates the selected panel on spi2.bus, wires GPIO out 21→CS, 4→DC, 5→RST, and panel BUSY→GPIO in 6 (X3 UC8253 renders; X4 SSD1677 is a blank stub);
creates ssi-sd at CS index 1 on spi2.bus, wires GPIO out 12→its CS, and a sd-card-spi backed by drive_get(IF_SD, 0, 0). Native uses a raw image; WASM uses QEMU vvfat over the JS-created MEMFS /sdcard directory;
defaults both CS lines high at init (qemu_set_irq(cs, 1)) so nothing is selected before the firmware drives GPIO.

Pin facts (X3, from freeink-sdk BoardConfig.h XTEINK_X3): display SCLK8/MOSI10/CS21/DC4/RST5/BUSY6; SD shares SCLK8/MOSI10, MISO7, CS12; button ADC ladders on GPIO1 & GPIO2; power button GPIO3; UC8253 @ 16 MHz.

7. X3 vs X4 detection — the thing that wasted time

Both X3 and X4 ship in ONE ESP32-C3 binary. Which panel/battery profile is used is decided at runtime by an I²C fingerprint on SDA20/SCL0: presence of BQ27220 gauge (0x55), DS3231 RTC (0x68), QMI8658 IMU (0x6B). ≥2 of 3 on two passes ⇒ X3; 0 ⇒ X4. Result is cached in NVS (cphw/dev_det) and can be overridden (cphw/dev_ovr: 1=X4, 2=X3).
esp32.i2c from the Espressif fork now runs on C3. The register layout matches, but C3 command opcodes differ (RESTART=6, WRITE=1, READ=3, STOP=2, END=4); a c3 controller property normalizes them without changing xtensa.
variant=x3 attaches minimal BQ27220/DS3231/QMI8658 targets returning exactly the probe signatures. variant=x4 leaves the bus empty, so reads NAK and the firmware selects X4. The resulting Arduino errors are noisy but harmless.
Validation used upstream commit 556b8ae with the old forced-X3 line removed. X3 rendered the home screen (~16,054 black pixels); X4 emitted SSD1677 bytes 12 18 80 0c ..., proving the guest selected its X4 driver.
Detection persists cphw/dev_det; use a clean flash or QEMU -snapshot when testing both variants so one run's NVS cache cannot affect the next.

8. Reproducibility model

Durable = the patches: apply 0001-xteink-machine.patch, then 0002-qemu-wasm.patch for the browser build. Both target exact QEMU commit 40edccac415693c5130f91c01d84176ae6008566 (tag esp-develop-9.2.2-20260417). When editing _scratch/qemu-src, regenerate:
```
cd <qemu-src> && git add -N <new files> && git diff --binary > <repo>/qemu/patches/0001-xteink-machine.patch
git apply --reverse --check <repo>/qemu/patches/0001-xteink-machine.patch
```
(Reverse-check on the dirty tree proves patch == pristine→current diff, which is equivalent to forward-applying to pristine.)
Firmware/SD/flash images and /tmp/qsrc are all disposable; make qemu + make firmware sdimage rebuild them.

9. Debugging workflow cheat-sheet

Headless run + capture (canonical loop this session):

qemu-system-riscv32 -machine xteink,variant=x3 -L <src>/pc-bios -display none -serial null \
  -qmp unix:/tmp/q.sock,server=on,wait=off \
  -drive file=flash.bin,if=mtd,format=raw -drive file=sd.img,if=sd,format=raw &
sleep 200   # this CPU is slow — give boot real time
scripts/qmp-screendump.py /tmp/q.sock out.ppm

Inspect the screen: convert with nix shell nixpkgs#imagemagick -c magick out.ppm out.png; the black-pixel count is a cheap "did the screen change" signal (home ≈ 15914, browse ≈ 11962, boot-white = 962).
Inject a button (proper edge = set then clear): scripts/qmp-qom.py /tmp/q.sock set /machine/adc "adci[1]" 2760, then 4095 to release. Ladder-1: Back 3512 / Confirm 2694 / Left 1493 / Right ~5; ladder-2: Up 2242 / Down ~5. >3900 = no button. Power is active-low: qmp-qom.py ... set /machine/gpio "input-level[3]" false then true.
Cold-boot power behavior: stock firmware expects Power held after a POWERON reset. On this ~24× slow host, hold GPIO3 low ~90 real seconds. For X4 battery boot also set GPIO20 low; high means USB and causes intentional sleep.
SD protocol trace: flip //#define DEBUG_SSI_SD 1 in hw/sd/ssi-sd.c.
Panel trace: add an info_report in xteink_x3_eink_transfer (bounded with a static counter — it's hot).
Symbolize a panic PC: llvm-addr2line -f -C -e firmware.elf 0xADDR (ignore DWARF "premature terminator" warnings).
Flash images must be 2/4/8/16 MB. The partition table spans 16 MB (app is <4 MB), so always pad to 16 MB (truncate -s 16M); a mismatched size errors with "only 2, 4, 8, and 16MB images are supported".

10. WASM build and next steps

make qemu-wasm uses Podman or Docker and the pinned Emscripten 3.1.50 image in qemu/wasm/Dockerfile. It builds only riscv32-softmmu with the xteink device graph; unrelated boards, networking, audio, virtfs, tools, and docs are disabled. Outputs go to dist/wasm/.
qemu-wasm commit 0ef7b4e2814b231705d8371dd7997f5b72e70baf is based on QEMU 8.2; 0002-qemu-wasm.patch is the resolved port to Espressif QEMU 9.2. -sEMULATE_FUNCTION_POINTER_CASTS=1 is required for QEMU's QOM callbacks.
The WASM dependency image omits libgcrypt. AES/RSA/DS/XTS devices are therefore not instantiated; their MMIO falls through to the existing zero-return stub.
Real firmware now boots past the earlier crashes and initializes the X3 panel. It stalls before painting a full home screen (display generation freezes at 2 after early init) — the next investigation, likely timer/interrupt or slow TCG.
Function-pointer signature fixes (required after dropping EMULATE_FUNCTION_POINTER_CASTS, which broke the wasm32 TCG ffi_call_js path): object_class_cmp → GCompareFunc, esp_sha hash-callback wrappers, and an xts_aes != NULL guard in the ESP32-C3 cache (XTS is gcrypt-gated and absent in WASM; unencrypted flash is unaffected).
Emscripten pthread startup has a rare transient trap at proxied entry (invokeEntryPoint → dynCall_ii); the build retries the bridge smoke once.
The upstream Emscripten RAM allocator fix is backported: anonymous guest RAM uses qemu_memalign instead of the broken partial-unmap mmap workaround.
Emscripten pthreads require cross-origin isolation (COOP/COEP) in the browser. make web runs the stdlib server with those headers and serves /web/ plus the sibling dist/wasm/ artifacts.
The narrow browser bridge exports panel surface metadata/pixels and atomic ADC/GPIO setters. JS polls a frame generation counter and forces alpha to 255; X3's native surface stores RGB with a zero alpha byte.
Firmware and ROM are written to MEMFS before QEMU starts. Browser SD uses QEMU vvfat to expose /sdcard as a writable 32 GiB FAT32 volume; memory scales with actual directory files plus about 5 MiB of FAT/write bookkeeping, not logical capacity. The initial JS provider creates Test/example.txt.
The inspector has Logs, Files, and Debug tabs. Debug reports committed WASM heap (176.3 MiB after a 90-second validation run), visible MEMFS bytes, SD file bytes, and browser JS heap when supported.
A second boot needs a page reload because QEMU machine selection and global state are process-wide.

Browser SD editing/persistence: add file controls and optionally OPFS. Live host-side changes to a mounted FAT volume are intentionally accepted as unsafe; editing is not implemented yet.

Patch workflow: scripts/build-qemu-wasm.sh resets/reapplies patches when either patch is newer than .xteink-patch-stamp. Regenerate patch 0002 from _scratch/qemu-fix-src with git diff --binary 610f8c69 > qemu/patches/0002-qemu-wasm.patch.

Full X4 rendering later: SSD1677 is 800×480 controller / 480×800 portrait, BUSY active-high. Implement its RAM/window/update commands when needed.

11. Key paths

What	Where
Emulator repo	`/home/evanreichard/Development/git/personal/xteink-web-emulator`
Native/WASM patches	`qemu/patches/0001-xteink-machine.patch`, `0002-qemu-wasm.patch`
WASM environment	`qemu/wasm/Dockerfile`
WASM build	`scripts/build-qemu-wasm.sh` → `dist/wasm/`
Browser UI	`web/`; run with `make web`
QEMU source (ephemeral)	`_scratch/qemu-src` (via `make qemu`)
Canonical firmware	`https://github.com/crosspoint-reader/crosspoint-reader`
Tested firmware checkout	`/tmp/crosspoint-reader-main` @ `556b8ae` (unmodified)
espressif/qemu commit	`40edccac415693c5130f91c01d84176ae6008566`
espressif/qemu tag	`esp-develop-9.2.2-20260417`
qemu-wasm	`github.com/ktock/qemu-wasm`

18 KiB Raw Permalink Blame History Unescape Escape