xteink-web-emulator/HANDOFF.md

# Handoff — xteink-web-emulator

Everything below is hard-won context from the session that built the native X3
emulator. If it isn't in this repo or this doc, it's gone. Read this before
touching QEMU internals. Pairs with `ROADMAP.md` (the plan) — this is the
"how it actually works and what will bite you" file.

---

## 0. TL;DR of current state

- One `xteink` QEMU machine selects `variant=x3|x4` at launch. **Unmodified
  canonical firmware** detects X3 through I²C fingerprints and boots the home
  screen; X4 detects through all-NAK probes and reaches a blank SSD1677 stub.
- Phase-1 device models live in `qemu/patches/0001-xteink-machine.patch`;
  `0002-qemu-wasm.patch` ports qemu-wasm's host backend onto the same pinned
  Espressif QEMU commit `40edccac415693c5130f91c01d84176ae6008566`.
- `make qemu-wasm` produces a smoke-tested ES module, WASM binary, and pthread
  worker under `dist/wasm/`; the static browser UI boots firmware, renders X3,
  drives buttons, and exposes SD files/debug metrics. Full X4 rendering waits.

---

## 1. Environment / hardware realities (READ FIRST)

- A full QEMU build is large and can take several minutes. Build parallelism
  defaults to `nproc`; set `JOBS` lower if memory pressure requires it and use
  long timeouts for any `ninja`/`pio`/`nix build`.
  Do not treat a long-running build as hung.
- **First full QEMU build ≈ many minutes; incremental after a patch edit is
  seconds** (only the touched `.c` + relink). Iterate incrementally; never
  `rm -rf build`.
- **Nix sandbox capability leak:** running FHS/bwrap-wrapped tools (PlatformIO,
  and the prebuilt release QEMU) fails with
  `bwrap: Unexpected capabilities but not setuid, old file caps config?`.
  Workaround — clear ambient caps:
  ```sh
  SP=$(nix build nixpkgs#util-linux.bin --no-link --print-out-paths)/bin/setpriv
  "$SP" --inh-caps=-all --ambient-caps=-all nix develop <flake> --command <cmd>
  ```
  Our **source-built** QEMU is a normal nix-linked ELF and does NOT need this
  (that's a bonus reason we build from source, not just to patch).

---

## 2. Building espressif-qemu from source (the fiddly part)

`scripts/build-qemu.sh` encodes all of this, but here's *why*:

- **Reuse nixpkgs' QEMU build env**: `nix develop nixpkgs#qemu --command …`
  (or the repo devShell, which has `inputsFrom = [ pkgs.qemu ]`). Getting glib/
  pixman/meson/ninja right by hand is painful; borrow qemu's.
- **slirp**: the fork *unconditionally compiles* `net/slirp.c` even with
  `--disable-slirp` (build-system bug), so you must **enable** it. nixpkgs
  provides libslirp; `--enable-slirp` + libslirp on `PKG_CONFIG_PATH` works, or
  it falls back to the bundled subproject.
- **libgcrypt**: the ESP32-C3 crypto devices (`esp32c3_aes.c`, rsa, ds, xts) are
  gated on `gcrypt.found()`. QEMU prefers gnutls and **skips gcrypt unless you
  pass `--enable-gcrypt`**. Without it the build succeeds but the machine
  **fails at runtime**: `unknown type 'misc.esp32c3.aes'`.
- **Meson caches `pkg_config_path`**: exporting `PKG_CONFIG_PATH` after the
  first `configure` is ignored on reconfigure. Use
  `meson configure build -Dpkg_config_path=…` or pass it to `./configure`.
- Config that works: `--target-list=riscv32-softmmu --enable-gcrypt
  --enable-slirp --disable-werror --disable-docs --disable-tools`.
- **`/tmp/qsrc` is where I built it this session and is EPHEMERAL** (lost on
  reboot). The patch is the durable artifact; `make qemu` reclones+rebuilds.

---

## 3. How espressif-qemu's esp32c3 machine is structured

- File: `hw/riscv/esp32c3.c`. It hand-builds the SoC (no device tree).
- **Catch-all MMIO**: `esp32c3_io_ops` (`esp32c3_io_read`/`write`) covers the
  whole peripheral window from `ESP32C3_IO_START_ADDR` (== `DR_REG_UART_BASE` ==
  `0x60000000`) for `0xd1000` bytes. **Any peripheral without a dedicated
  device falls here and reads 0 / ignores writes.** That's why unmodeled
  peripherals silently misbehave instead of faulting.
- Turn on `#define ESP32C3_IO_WARNING 1` (top of the file) to log every
  unsupported access as `[ESP32-C3] Unsupported read/write to $ADDR`. **This is
  the single most useful debugging lever.** It's noisy (megabytes/sec), so run
  ~12 s and bucket by address:
  ```sh
  grep 'Unsupported read' log | sed -E 's/.*\$//' | sort | uniq -c | sort -nr | head
  ```
  The hottest address is almost always the next thing to model. Leave it `0` in
  committed code.
- Devices that **already exist** in the fork: UART, GPIO (barebones — we fleshed
  it out), SPI1 (flash only), GDMA, AES/SHA/RSA/HMAC/DS/XTS, RTC_CNTL, systimer,
  timg, efuse, intmatrix, cache, JTAG, TWAI, `esp_rgb` (a QEMU-specific virtual
  framebuffer, NOT real X3 hardware — we disable it on the xteink machine).
- Devices that **did NOT exist** and we added: SAR ADC, SPI2, the UC8253 panel,
  and real GPIO register behavior. There is **no I²C controller model** — this
  matters for X3/X4 detection (see §7).

---

## 4. Peripheral register facts (discovered, verified against firmware)

Base addresses from ESP-IDF `soc/esp32c3` headers; all confirmed live.

### SAR ADC — `DR_REG_APB_SARADC_BASE = 0x60040000`
- The **original boot blocker**: firmware polls `INT_RAW` (`+0x44`) bit 31
  (`ADC1_ONESHOT_DONE`) after starting a conversion; the unmodeled reg returned
  0 forever → busy-loop with interrupts off → **interrupt-WDT panic**
  (`Guru Meditation … Interrupt wdt timeout`). The panic dump's `S0/FP =
  0x60040000` literally pointed at the ADC base.
- Model (`hw/adc/esp32c3_adc.c`): `INT_RAW` always returns done bits 31|30;
  `ONETIME_SAMPLE` (`+0x20`) bits [28:25] select the channel; `1_DATA_STATUS`
  (`+0x2c`) / `2_DATA_STATUS` (`+0x30`) return the selected channel's 12-bit
  value. Per-channel values are QOM props `adci[*]`, default `0xfff`.
- `adc_oneshot_ll_get_raw_result` reads `adc1_data & 0xfff`.

### SPI2 (display + SD) — `DR_REG_SPI2_BASE = 0x60024000`
- **Completely different register map from SPI1** (`hw/ssi/esp32c3_spi.c` is
  flash-specific — do NOT reuse it). GP-SPI2 layout: `CMD 0x00` (USR = bit24,
  UPDATE = bit23), `MS_DLEN 0x1c` (bits[17:0] = bitlen-1), data buffer
  `W0..W15` at `0x98..0xd4`.
- Model (`hw/ssi/esp32c3_spi2.c`): on `CMD` write with USR set, clock
  `(MS_DLEN/8)+1` bytes out of W0.. onto an `SSIBus` via `ssi_transfer`, writing
  RX back into the same words. **DMA path is intentionally unimplemented** — the
  firmware's CPU-buffer path is enough; add DMA only if a future firmware needs
  it (it'll show up as SPI2 traffic that the CPU-buffer model can't explain).

### GPIO — `DR_REG_GPIO_BASE = 0x60004000`
- The stock model only returned the strap register. We added real
  `OUT 0x04` / `OUT_W1TS 0x08` / `OUT_W1TC 0x0c` / `ENABLE 0x20` (+W1TS/W1TC) /
  `IN 0x3c`, with 32 named input + output qemu_irq lines
  (`ESP32_GPIO_INPUT`/`ESP32_GPIO_OUTPUT`).
- **CRITICAL SUBTLETY**: an output line must read **HIGH when the pin is
  configured as input** (`output_enable` bit clear), not follow the (zero) OUT
  latch. Real boards have pull-ups; the X3 display **CS is pulled high** while
  SD is being probed. Without this, CS floated LOW during SD init and the panel
  greedily consumed the SD command bytes (`FF … 40 00 … 95`) as garbage
  commands. This one bit cost real debugging time.

### SD-over-SPI — QEMU's `ssi-sd` + `sd-card-spi`
- These exist upstream; we attach them to SPI2 at CS12. But the 9.2 fork's
  `ssi-sd.c` has a **bug**: CMD58 (read OCR) hardcodes the R1 byte to `0x01`
  (idle) even after ACMD41 reported ready → SdFat rejects the card → firmware
  shows "SD card error". Fix: track `idle` state, clear it when ACMD41 returns
  0, return it for CMD58. (Upstream qemu master rewrote this path entirely; we
  did the minimal equivalent + a vmstate version bump to 8.)

---

## 5. The e-ink panel model (`hw/display/xteink_x3_eink.c`)

- X3 uses a **UC8253** controller, 792×528, 99 bytes/row, **52 272 bytes/plane**.
- Protocol (from `chip/eink-x3.chip.c`, the old Wokwi model — keep it as the
  reference): DC low = command, DC high = data. `0x10`=DTM1 (old plane),
  `0x13`=DTM2 (new plane), `0x12`=refresh (render + pulse BUSY), `0x04`/`0x02`=
  power on/off (pulse BUSY). Bit 1 = white, 0 = black. Rows arrive
  **bottom-to-top** (wire row i → framebuffer row H-1-i).
- BUSY is a GPIO output from the panel back into GPIO6 (active low; we pulse it
  low for 5 ms of virtual time via a QEMUTimer, then high).
- **Rendered to a QEMU graphical console** (not a custom file callback) so the
  WASM front-end reuses QEMU's display surface. Capture headless with QMP
  `screendump` → PPM.
- **Orientation**: the controller framebuffer (792×528 landscape) is **rotated
  90° clockwise** to the physical portrait panel (**528×792**). We bake that
  into the render loop (`screen_x = H-1-y`, `screen_y = x`) so the console is
  already physical-orientation. Verified by eye: text reads upright at 528×792.
- **SSI gotcha**: a `TYPE_SSI_PERIPHERAL` subclass **must set `ssi->realize`**
  even if empty — the base class calls it unconditionally and a null pointer
  segfaults at machine init. Also set `cs_polarity = SSI_CS_LOW` and
  `user_creatable = false`.

---

## 6. Machine wiring (`xteink_machine_init`)

`xteink` is a subclass of the `esp32c3` machine. `variant=x3` is the default;
`variant=x4` selects the sibling wiring before boot. After the base init it:
- creates the selected panel on `spi2.bus`, wires GPIO out 21→CS, 4→DC, 5→RST,
  and panel BUSY→GPIO in 6 (X3 UC8253 renders; X4 SSD1677 is a blank stub);
- creates `ssi-sd` at CS index 1 on `spi2.bus`, wires GPIO out 12→its CS, and a
  `sd-card-spi` backed by `drive_get(IF_SD, 0, 0)`. Native uses a raw image;
  WASM uses QEMU `vvfat` over the JS-created MEMFS `/sdcard` directory;
- **defaults both CS lines high** at init (`qemu_set_irq(cs, 1)`) so nothing is
  selected before the firmware drives GPIO.

Pin facts (X3, from `freeink-sdk` `BoardConfig.h` `XTEINK_X3`): display
SCLK8/MOSI10/CS21/DC4/RST5/BUSY6; SD shares SCLK8/MOSI10, MISO7, CS12; button
ADC ladders on GPIO1 & GPIO2; power button GPIO3; UC8253 @ 16 MHz.

---

## 7. X3 vs X4 detection — the thing that wasted time

- **Both X3 and X4 ship in ONE ESP32-C3 binary.** Which panel/battery profile is
  used is decided **at runtime** by an I²C fingerprint on SDA20/SCL0: presence of
  BQ27220 gauge (0x55), DS3231 RTC (0x68), QMI8658 IMU (0x6B). ≥2 of 3 on two
  passes ⇒ X3; 0 ⇒ X4. Result is cached in NVS (`cphw/dev_det`) and can be
  overridden (`cphw/dev_ovr`: 1=X4, 2=X3).
- `esp32.i2c` from the Espressif fork now runs on C3. The register layout
  matches, but C3 command opcodes differ (RESTART=6, WRITE=1, READ=3, STOP=2,
  END=4); a `c3` controller property normalizes them without changing xtensa.
- `variant=x3` attaches minimal BQ27220/DS3231/QMI8658 targets returning exactly
  the probe signatures. `variant=x4` leaves the bus empty, so reads NAK and the
  firmware selects X4. The resulting Arduino errors are noisy but harmless.
- Validation used upstream commit `556b8ae` with the old forced-X3 line removed.
  X3 rendered the home screen (~16,054 black pixels); X4 emitted SSD1677 bytes
  `12 18 80 0c ...`, proving the guest selected its X4 driver.
- Detection persists `cphw/dev_det`; use a clean flash or QEMU `-snapshot` when
  testing both variants so one run's NVS cache cannot affect the next.

---

## 8. Reproducibility model

- **Durable = the patches**: apply `0001-xteink-machine.patch`, then
  `0002-qemu-wasm.patch` for the browser build. Both target exact QEMU commit `40edccac415693c5130f91c01d84176ae6008566` (tag
  `esp-develop-9.2.2-20260417`). When editing `_scratch/qemu-src`, regenerate:
  ```sh
  cd <qemu-src> && git add -N <new files> && git diff --binary > <repo>/qemu/patches/0001-xteink-machine.patch
  git apply --reverse --check <repo>/qemu/patches/0001-xteink-machine.patch
  ```
  (Reverse-check on the dirty tree proves patch == pristine→current diff, which
  is equivalent to forward-applying to pristine.)
- Firmware/SD/flash images and `/tmp/qsrc` are all disposable; `make qemu` +
  `make firmware sdimage` rebuild them.

---

## 9. Debugging workflow cheat-sheet

- **Headless run + capture** (canonical loop this session):
  ```sh
  qemu-system-riscv32 -machine xteink,variant=x3 -L <src>/pc-bios -display none -serial null \
    -qmp unix:/tmp/q.sock,server=on,wait=off \
    -drive file=flash.bin,if=mtd,format=raw -drive file=sd.img,if=sd,format=raw &
  sleep 200   # this CPU is slow — give boot real time
  scripts/qmp-screendump.py /tmp/q.sock out.ppm
  ```
- **Inspect the screen**: convert with `nix shell nixpkgs#imagemagick -c magick
  out.ppm out.png`; the black-pixel count is a cheap "did the screen change"
  signal (home ≈ 15914, browse ≈ 11962, boot-white = 962).
- **Inject a button** (proper edge = set then clear):
  `scripts/qmp-qom.py /tmp/q.sock set /machine/adc "adci[1]" 2760`, then `4095`
  to release. Ladder-1: Back 3512 / Confirm 2694 / Left 1493 / Right ~5;
  ladder-2: Up 2242 / Down ~5. **>3900 = no button.** Power is active-low:
  `qmp-qom.py ... set /machine/gpio "input-level[3]" false` then `true`.
- **Cold-boot power behavior:** stock firmware expects Power held after a
  POWERON reset. On this ~24× slow host, hold GPIO3 low ~90 real seconds. For X4
  battery boot also set GPIO20 low; high means USB and causes intentional sleep.
- **SD protocol trace**: flip `//#define DEBUG_SSI_SD 1` in `hw/sd/ssi-sd.c`.
- **Panel trace**: add an `info_report` in `xteink_x3_eink_transfer`
  (bounded with a static counter — it's hot).
- **Symbolize a panic PC**: `llvm-addr2line -f -C -e firmware.elf 0xADDR`
  (ignore DWARF "premature terminator" warnings).
- **Flash images must be 2/4/8/16 MB.** The partition table spans 16 MB (app is
  <4 MB), so always pad to 16 MB (`truncate -s 16M`); a mismatched size errors
  with "only 2, 4, 8, and 16MB images are supported".

---

## 10. WASM build and next steps

- `make qemu-wasm` uses Podman or Docker and the pinned Emscripten 3.1.50 image
  in `qemu/wasm/Dockerfile`. It builds only `riscv32-softmmu` with the xteink
  device graph; unrelated boards, networking, audio, virtfs, tools, and docs are
  disabled. Outputs go to `dist/wasm/`.
- qemu-wasm commit `0ef7b4e2814b231705d8371dd7997f5b72e70baf` is based on
  QEMU 8.2; `0002-qemu-wasm.patch` is the resolved port to Espressif QEMU 9.2.
  `-sEMULATE_FUNCTION_POINTER_CASTS=1` is required for QEMU's QOM callbacks.
- The WASM dependency image omits libgcrypt. AES/RSA/DS/XTS devices are therefore
  not instantiated; their MMIO falls through to the existing zero-return stub.
- Real firmware now boots past the earlier crashes and initializes the X3 panel.
  It stalls before painting a full home screen (display generation freezes at 2
  after early init) — the next investigation, likely timer/interrupt or slow TCG.
- Function-pointer signature fixes (required after dropping
  `EMULATE_FUNCTION_POINTER_CASTS`, which broke the wasm32 TCG `ffi_call_js`
  path): `object_class_cmp` → `GCompareFunc`, `esp_sha` hash-callback wrappers,
  and an `xts_aes != NULL` guard in the ESP32-C3 cache (XTS is gcrypt-gated and
  absent in WASM; unencrypted flash is unaffected).
- Emscripten pthread startup has a rare transient trap at proxied entry
  (`invokeEntryPoint` → `dynCall_ii`); the build retries the bridge smoke once.
- The upstream Emscripten RAM allocator fix is backported: anonymous guest RAM
  uses `qemu_memalign` instead of the broken partial-unmap `mmap` workaround.
- Emscripten pthreads require cross-origin isolation (COOP/COEP) in the browser.
  `make web` runs the stdlib server with those headers and serves `/web/` plus
  the sibling `dist/wasm/` artifacts.
- The narrow browser bridge exports panel surface metadata/pixels and atomic
  ADC/GPIO setters. JS polls a frame generation counter and forces alpha to 255;
  X3's native surface stores RGB with a zero alpha byte.
- Firmware and ROM are written to MEMFS before QEMU starts. Browser SD uses
  QEMU `vvfat` to expose `/sdcard` as a writable 32 GiB FAT32 volume; memory
  scales with actual directory files plus about 5 MiB of FAT/write bookkeeping,
  not logical capacity. The initial JS provider creates `Test/example.txt`.
- The inspector has Logs, Files, and Debug tabs. Debug reports committed WASM
  heap (176.3 MiB after a 90-second validation run), visible MEMFS bytes, SD file
  bytes, and browser JS heap when supported.
- A second boot needs a page reload because QEMU machine selection and global
  state are process-wide.

Next:

1. **Browser SD editing/persistence**: add file controls and optionally OPFS.
   Live host-side changes to a mounted FAT volume are intentionally accepted as
   unsafe; editing is not implemented yet.

> Patch workflow: `scripts/build-qemu-wasm.sh` resets/reapplies patches when
> either patch is newer than `.xteink-patch-stamp`. Regenerate patch 0002 from
> `_scratch/qemu-fix-src` with `git diff --binary 610f8c69 >
> qemu/patches/0002-qemu-wasm.patch`.
2. **Full X4 rendering later**: SSD1677 is 800×480 controller / 480×800 portrait,
   BUSY active-high. Implement its RAM/window/update commands when needed.

---

## 11. Key paths

| What | Where |
|------|-------|
| Emulator repo | `/home/evanreichard/Development/git/personal/xteink-web-emulator` |
| Native/WASM patches | `qemu/patches/0001-xteink-machine.patch`, `0002-qemu-wasm.patch` |
| WASM environment | `qemu/wasm/Dockerfile` |
| WASM build | `scripts/build-qemu-wasm.sh` → `dist/wasm/` |
| Browser UI | `web/`; run with `make web` |
| QEMU source (ephemeral) | `_scratch/qemu-src` (via `make qemu`) |
| Canonical firmware | `https://github.com/crosspoint-reader/crosspoint-reader` |
| Tested firmware checkout | `/tmp/crosspoint-reader-main` @ `556b8ae` (unmodified) |
| espressif/qemu commit | `40edccac415693c5130f91c01d84176ae6008566` |
| espressif/qemu tag | `esp-develop-9.2.2-20260417` |
| qemu-wasm | `github.com/ktock/qemu-wasm` |