xteink-web-emulator/HANDOFF.md

# Handoff — xteink-web-emulator

Everything below is hard-won context from the session that built the native X3
emulator. If it isn't in this repo or this doc, it's gone. Read this before
touching QEMU internals. Pairs with `ROADMAP.md` (the plan) — this is the
"how it actually works and what will bite you" file.

---

## 0. TL;DR of current state

- The `xteink-x3` QEMU machine boots **unmodified canonical firmware** to the
  home screen, mounts a FAT32 SD image, and takes button input. Verified with
  QMP `screendump`.
- All Phase-1 device models live in **one patch**:
  `qemu/patches/0001-xteink-x3-machine.patch`, applied onto espressif/qemu tag
  `esp-develop-9.2.2-20260417` by `scripts/build-qemu.sh`.
- Next: X4 sibling machine → compile to WASM (qemu-wasm) → static browser UI.

---

## 1. Environment / hardware realities (READ FIRST)

- **The dev box is ~15 yr old, 4 GB RAM.** A full QEMU build is ~1660 objects
  and takes a long time. **Always `-j2`** (more OOMs the linker) and use
  **very long timeouts (I used 7200 s)** for any `ninja`/`pio`/`nix build`.
  Do not treat a long-running build as hung.
- **First full QEMU build ≈ many minutes; incremental after a patch edit is
  seconds** (only the touched `.c` + relink). Iterate incrementally; never
  `rm -rf build`.
- **Nix sandbox capability leak:** running FHS/bwrap-wrapped tools (PlatformIO,
  and the prebuilt release QEMU) fails with
  `bwrap: Unexpected capabilities but not setuid, old file caps config?`.
  Workaround — clear ambient caps:
  ```sh
  SP=$(nix build nixpkgs#util-linux.bin --no-link --print-out-paths)/bin/setpriv
  "$SP" --inh-caps=-all --ambient-caps=-all nix develop <flake> --command <cmd>
  ```
  Our **source-built** QEMU is a normal nix-linked ELF and does NOT need this
  (that's a bonus reason we build from source, not just to patch).

---

## 2. Building espressif-qemu from source (the fiddly part)

`scripts/build-qemu.sh` encodes all of this, but here's *why*:

- **Reuse nixpkgs' QEMU build env**: `nix develop nixpkgs#qemu --command …`
  (or the repo devShell, which has `inputsFrom = [ pkgs.qemu ]`). Getting glib/
  pixman/meson/ninja right by hand is painful; borrow qemu's.
- **slirp**: the fork *unconditionally compiles* `net/slirp.c` even with
  `--disable-slirp` (build-system bug), so you must **enable** it. nixpkgs
  provides libslirp; `--enable-slirp` + libslirp on `PKG_CONFIG_PATH` works, or
  it falls back to the bundled subproject.
- **libgcrypt**: the ESP32-C3 crypto devices (`esp32c3_aes.c`, rsa, ds, xts) are
  gated on `gcrypt.found()`. QEMU prefers gnutls and **skips gcrypt unless you
  pass `--enable-gcrypt`**. Without it the build succeeds but the machine
  **fails at runtime**: `unknown type 'misc.esp32c3.aes'`.
- **Meson caches `pkg_config_path`**: exporting `PKG_CONFIG_PATH` after the
  first `configure` is ignored on reconfigure. Use
  `meson configure build -Dpkg_config_path=…` or pass it to `./configure`.
- Config that works: `--target-list=riscv32-softmmu --enable-gcrypt
  --enable-slirp --disable-werror --disable-docs --disable-tools`.
- **`/tmp/qsrc` is where I built it this session and is EPHEMERAL** (lost on
  reboot). The patch is the durable artifact; `make qemu` reclones+rebuilds.

---

## 3. How espressif-qemu's esp32c3 machine is structured

- File: `hw/riscv/esp32c3.c`. It hand-builds the SoC (no device tree).
- **Catch-all MMIO**: `esp32c3_io_ops` (`esp32c3_io_read`/`write`) covers the
  whole peripheral window from `ESP32C3_IO_START_ADDR` (== `DR_REG_UART_BASE` ==
  `0x60000000`) for `0xd1000` bytes. **Any peripheral without a dedicated
  device falls here and reads 0 / ignores writes.** That's why unmodeled
  peripherals silently misbehave instead of faulting.
- Turn on `#define ESP32C3_IO_WARNING 1` (top of the file) to log every
  unsupported access as `[ESP32-C3] Unsupported read/write to $ADDR`. **This is
  the single most useful debugging lever.** It's noisy (megabytes/sec), so run
  ~12 s and bucket by address:
  ```sh
  grep 'Unsupported read' log | sed -E 's/.*\$//' | sort | uniq -c | sort -nr | head
  ```
  The hottest address is almost always the next thing to model. Leave it `0` in
  committed code.
- Devices that **already exist** in the fork: UART, GPIO (barebones — we fleshed
  it out), SPI1 (flash only), GDMA, AES/SHA/RSA/HMAC/DS/XTS, RTC_CNTL, systimer,
  timg, efuse, intmatrix, cache, JTAG, TWAI, `esp_rgb` (a QEMU-specific virtual
  framebuffer, NOT real X3 hardware — we disable it on the xteink machine).
- Devices that **did NOT exist** and we added: SAR ADC, SPI2, the UC8253 panel,
  and real GPIO register behavior. There is **no I²C controller model** — this
  matters for X3/X4 detection (see §7).

---

## 4. Peripheral register facts (discovered, verified against firmware)

Base addresses from ESP-IDF `soc/esp32c3` headers; all confirmed live.

### SAR ADC — `DR_REG_APB_SARADC_BASE = 0x60040000`
- The **original boot blocker**: firmware polls `INT_RAW` (`+0x44`) bit 31
  (`ADC1_ONESHOT_DONE`) after starting a conversion; the unmodeled reg returned
  0 forever → busy-loop with interrupts off → **interrupt-WDT panic**
  (`Guru Meditation … Interrupt wdt timeout`). The panic dump's `S0/FP =
  0x60040000` literally pointed at the ADC base.
- Model (`hw/adc/esp32c3_adc.c`): `INT_RAW` always returns done bits 31|30;
  `ONETIME_SAMPLE` (`+0x20`) bits [28:25] select the channel; `1_DATA_STATUS`
  (`+0x2c`) / `2_DATA_STATUS` (`+0x30`) return the selected channel's 12-bit
  value. Per-channel values are QOM props `adci[*]`, default `0xfff`.
- `adc_oneshot_ll_get_raw_result` reads `adc1_data & 0xfff`.

### SPI2 (display + SD) — `DR_REG_SPI2_BASE = 0x60024000`
- **Completely different register map from SPI1** (`hw/ssi/esp32c3_spi.c` is
  flash-specific — do NOT reuse it). GP-SPI2 layout: `CMD 0x00` (USR = bit24,
  UPDATE = bit23), `MS_DLEN 0x1c` (bits[17:0] = bitlen-1), data buffer
  `W0..W15` at `0x98..0xd4`.
- Model (`hw/ssi/esp32c3_spi2.c`): on `CMD` write with USR set, clock
  `(MS_DLEN/8)+1` bytes out of W0.. onto an `SSIBus` via `ssi_transfer`, writing
  RX back into the same words. **DMA path is intentionally unimplemented** — the
  firmware's CPU-buffer path is enough; add DMA only if a future firmware needs
  it (it'll show up as SPI2 traffic that the CPU-buffer model can't explain).

### GPIO — `DR_REG_GPIO_BASE = 0x60004000`
- The stock model only returned the strap register. We added real
  `OUT 0x04` / `OUT_W1TS 0x08` / `OUT_W1TC 0x0c` / `ENABLE 0x20` (+W1TS/W1TC) /
  `IN 0x3c`, with 32 named input + output qemu_irq lines
  (`ESP32_GPIO_INPUT`/`ESP32_GPIO_OUTPUT`).
- **CRITICAL SUBTLETY**: an output line must read **HIGH when the pin is
  configured as input** (`output_enable` bit clear), not follow the (zero) OUT
  latch. Real boards have pull-ups; the X3 display **CS is pulled high** while
  SD is being probed. Without this, CS floated LOW during SD init and the panel
  greedily consumed the SD command bytes (`FF … 40 00 … 95`) as garbage
  commands. This one bit cost real debugging time.

### SD-over-SPI — QEMU's `ssi-sd` + `sd-card-spi`
- These exist upstream; we attach them to SPI2 at CS12. But the 9.2 fork's
  `ssi-sd.c` has a **bug**: CMD58 (read OCR) hardcodes the R1 byte to `0x01`
  (idle) even after ACMD41 reported ready → SdFat rejects the card → firmware
  shows "SD card error". Fix: track `idle` state, clear it when ACMD41 returns
  0, return it for CMD58. (Upstream qemu master rewrote this path entirely; we
  did the minimal equivalent + a vmstate version bump to 8.)

---

## 5. The e-ink panel model (`hw/display/xteink_x3_eink.c`)

- X3 uses a **UC8253** controller, 792×528, 99 bytes/row, **52 272 bytes/plane**.
- Protocol (from `chip/eink-x3.chip.c`, the old Wokwi model — keep it as the
  reference): DC low = command, DC high = data. `0x10`=DTM1 (old plane),
  `0x13`=DTM2 (new plane), `0x12`=refresh (render + pulse BUSY), `0x04`/`0x02`=
  power on/off (pulse BUSY). Bit 1 = white, 0 = black. Rows arrive
  **bottom-to-top** (wire row i → framebuffer row H-1-i).
- BUSY is a GPIO output from the panel back into GPIO6 (active low; we pulse it
  low for 5 ms of virtual time via a QEMUTimer, then high).
- **Rendered to a QEMU graphical console** (not a custom file callback) so the
  WASM front-end reuses QEMU's display surface. Capture headless with QMP
  `screendump` → PPM.
- **Orientation**: the controller framebuffer (792×528 landscape) is **rotated
  90° clockwise** to the physical portrait panel (**528×792**). We bake that
  into the render loop (`screen_x = H-1-y`, `screen_y = x`) so the console is
  already physical-orientation. Verified by eye: text reads upright at 528×792.
- **SSI gotcha**: a `TYPE_SSI_PERIPHERAL` subclass **must set `ssi->realize`**
  even if empty — the base class calls it unconditionally and a null pointer
  segfaults at machine init. Also set `cs_polarity = SSI_CS_LOW` and
  `user_creatable = false`.

---

## 6. Machine wiring (`xteink_x3_machine_init`)

`xteink-x3` is a subclass of the `esp32c3` machine that, after the base init:
- creates the panel on `spi2.bus`, wires GPIO out 21→CS, 4→DC, 5→RST, and panel
  BUSY→GPIO in 6;
- creates `ssi-sd` at CS index 1 on `spi2.bus`, wires GPIO out 12→its CS, and a
  `sd-card-spi` backed by `drive_get(IF_SD, 0, 0)` (QEMU block layer — the SD
  "interface" you asked for; local `-drive` now, browser blockdev later);
- **defaults both CS lines high** at init (`qemu_set_irq(cs, 1)`) so nothing is
  selected before the firmware drives GPIO.

Pin facts (X3, from `freeink-sdk` `BoardConfig.h` `XTEINK_X3`): display
SCLK8/MOSI10/CS21/DC4/RST5/BUSY6; SD shares SCLK8/MOSI10, MISO7, CS12; button
ADC ladders on GPIO1 & GPIO2; power button GPIO3; UC8253 @ 16 MHz.

---

## 7. X3 vs X4 detection — the thing that wasted time

- **Both X3 and X4 ship in ONE ESP32-C3 binary.** Which panel/battery profile is
  used is decided **at runtime** by an I²C fingerprint on SDA20/SCL0: presence of
  BQ27220 gauge (0x55), DS3231 RTC (0x68), QMI8658 IMU (0x6B). ≥2 of 3 on two
  passes ⇒ X3; 0 ⇒ X4. Result is cached in NVS (`cphw/dev_det`) and can be
  overridden (`cphw/dev_ovr`: 1=X4, 2=X3).
- **QEMU models no I²C**, so detection currently returns X4 by default. The
  **crosspoint-reader-lua** firmware never calls `setDisplayX3()` at all → it is
  **X4-only in emulation** and drives SSD1677 commands (`0x18/0x0c/0x44/0x45…`),
  which our UC8253 panel ignores (→ blank screen). Don't chase that as a bug;
  it's the wrong firmware for an X3 machine.
- To test the X3 display this session I built the **canonical** firmware with a
  **one-line temporary hack**: `detectDeviceTypeWithFingerprint()` in
  `/tmp/crosspoint-reader-main/lib/hal/HalGPIO.cpp` returns `X3` at the top.
  **THIS EDIT IS IN /tmp AND WILL BE LOST ON REBOOT.** The merged test image is
  `/tmp/crosspoint-main-x3-flash.bin` (also ephemeral).
- **Production plan (no custom firmware):** the `xteink-x3` machine should expose
  the three X3 I²C fingerprint chips (needs an esp32c3 I²C controller model +
  trivial BQ27220/DS3231/QMI8658 stubs that return plausible WHO_AM_I/values),
  OR the emulator seeds the NVS `dev_ovr` key. Either makes stock firmware
  auto-select X3. The `xteink-x4` machine simply omits the fingerprint. This is
  how launch-time X3/X4 selection stays firmware-agnostic.

---

## 8. Reproducibility model

- **Durable = the patch** (`qemu/patches/0001-xteink-x3-machine.patch`), verified
  to `git apply` cleanly onto a pristine checkout of the pinned tag. When you
  edit `/tmp/qsrc`, **regenerate it** and re-verify:
  ```sh
  cd <qemu-src> && git add -N <new files> && git diff --binary > <repo>/qemu/patches/0001-xteink-x3-machine.patch
  git apply --reverse --check <repo>/qemu/patches/0001-xteink-x3-machine.patch   # must succeed
  ```
  (Reverse-check on the dirty tree proves patch == pristine→current diff, which
  is equivalent to forward-applying to pristine.)
- Firmware/SD/flash images and `/tmp/qsrc` are all disposable; `make qemu` +
  `make firmware sdimage` rebuild them.

---

## 9. Debugging workflow cheat-sheet

- **Headless run + capture** (canonical loop this session):
  ```sh
  qemu-system-riscv32 -machine xteink-x3 -L <src>/pc-bios -display none -serial null \
    -qmp unix:/tmp/q.sock,server=on,wait=off \
    -drive file=flash.bin,if=mtd,format=raw -drive file=sd.img,if=sd,format=raw &
  sleep 200   # this CPU is slow — give boot real time
  scripts/qmp-screendump.py /tmp/q.sock out.ppm
  ```
- **Inspect the screen**: convert with `nix shell nixpkgs#imagemagick -c magick
  out.ppm out.png`; the black-pixel count is a cheap "did the screen change"
  signal (home ≈ 15914, browse ≈ 11962, boot-white = 962).
- **Inject a button** (proper edge = set then clear; hold-from-boot gives no
  press edge): `scripts/qmp-qom.py /tmp/q.sock set /machine/adc "adci[1]" 2760`
  then `… 4095` to release. Ladder-1 midpoints: Back 3512 / Confirm 2694 /
  Left 1493 / Right ~5; ladder-2: Up 2242 / Down ~5. **>3900 = no button.**
- **SD protocol trace**: flip `//#define DEBUG_SSI_SD 1` in `hw/sd/ssi-sd.c`.
- **Panel trace**: add an `info_report` in `xteink_x3_eink_transfer`
  (bounded with a static counter — it's hot).
- **Symbolize a panic PC**: `llvm-addr2line -f -C -e firmware.elf 0xADDR`
  (ignore DWARF "premature terminator" warnings).
- **Flash images must be 2/4/8/16 MB.** The partition table spans 16 MB (app is
  <4 MB), so always pad to 16 MB (`truncate -s 16M`); a mismatched size errors
  with "only 2, 4, 8, and 16MB images are supported".

---

## 10. Next steps (in priority order)

1. **X3 fingerprint or NVS seeding** (§7) so stock firmware auto-selects X3 with
   no source edit. Smallest path to a genuinely firmware-agnostic X3 machine.
   Needs a minimal esp32c3 I²C controller + 3 tiny I²C target stubs, OR NVS
   pre-seed. This is the last thing standing between "works with a hacked build"
   and "works with any downloaded binary."
2. **`xteink-x4` machine**: SSD1677 panel (different command set — `0x18` set-RAM,
   `0x44/0x45` window, `0x4e/0x4f` cursor, `0x24` write, `0x22/0x20` update;
   BUSY active-HIGH), X4 battery via ADC (not I²C gauge), 792×528? confirm X4
   geometry from `BoardConfig.h XTEINK_X4`. Sibling machine + a second panel
   device; reuse ADC/SPI2/GPIO/SD unchanged.
3. **Compile to WASM**: combine this patched fork with `ktock/qemu-wasm`'s WASM
   TCG/TCI backend + Emscripten. Expect toolchain friction; the device models
   themselves are portable C and shouldn't need changes. De-risk by first
   building an *unpatched* esp32c3 target under qemu-wasm, then apply our patch.
4. **Static browser front-end**: `<canvas>` fed from the display surface; DOM
   buttons → the ADC/GPIO QOM injection (same calls as `qmp-qom.py`); file input
   → emulated flash; preloaded/uploaded SD image → block backend; X3/X4 chosen
   before boot. Must run fully from static assets (airgapped).

---

## 11. Key paths

| What | Where |
|------|-------|
| Emulator repo | `/home/evanreichard/Development/git/personal/xteink-web-emulator` |
| The patch | `qemu/patches/0001-xteink-x3-machine.patch` |
| QEMU source (ephemeral) | `_scratch/qemu-src` (via `make qemu`); was `/tmp/qsrc` this session |
| Canonical firmware (X3 hack in /tmp, ephemeral) | `/tmp/crosspoint-reader-main` (`lib/hal/HalGPIO.cpp` forced X3) |
| Lua firmware (X4-only in emu) | `/home/evanreichard/Development/git/personal/crosspoint-reader-lua` |
| Upstream firmware refs | `crosspoint-reader/crosspoint-reader` (canonical, cleaner) + its `freeink-sdk` submodule |
| espressif/qemu tag | `esp-develop-9.2.2-20260417` |
| qemu-wasm | `github.com/ktock/qemu-wasm` |