initial commit

HANDOFF.md

@@ -0,0 +1,290 @@
+# Handoff — xteink-web-emulator
+
+Everything below is hard-won context from the session that built the native X3
+emulator. If it isn't in this repo or this doc, it's gone. Read this before
+touching QEMU internals. Pairs with `ROADMAP.md` (the plan) — this is the
+"how it actually works and what will bite you" file.
+
+---
+
+## 0. TL;DR of current state
+
+- The `xteink-x3` QEMU machine boots **unmodified canonical firmware** to the
+  home screen, mounts a FAT32 SD image, and takes button input. Verified with
+  QMP `screendump`.
+- All Phase-1 device models live in **one patch**:
+  `qemu/patches/0001-xteink-x3-machine.patch`, applied onto espressif/qemu tag
+  `esp-develop-9.2.2-20260417` by `scripts/build-qemu.sh`.
+- Next: X4 sibling machine → compile to WASM (qemu-wasm) → static browser UI.
+
+---
+
+## 1. Environment / hardware realities (READ FIRST)
+
+- **The dev box is ~15 yr old, 4 GB RAM.** A full QEMU build is ~1660 objects
+  and takes a long time. **Always `-j2`** (more OOMs the linker) and use
+  **very long timeouts (I used 7200 s)** for any `ninja`/`pio`/`nix build`.
+  Do not treat a long-running build as hung.
+- **First full QEMU build ≈ many minutes; incremental after a patch edit is
+  seconds** (only the touched `.c` + relink). Iterate incrementally; never
+  `rm -rf build`.
+- **Nix sandbox capability leak:** running FHS/bwrap-wrapped tools (PlatformIO,
+  and the prebuilt release QEMU) fails with
+  `bwrap: Unexpected capabilities but not setuid, old file caps config?`.
+  Workaround — clear ambient caps:
+  ```sh
+  SP=$(nix build nixpkgs#util-linux.bin --no-link --print-out-paths)/bin/setpriv
+  "$SP" --inh-caps=-all --ambient-caps=-all nix develop <flake> --command <cmd>
+  ```
+  Our **source-built** QEMU is a normal nix-linked ELF and does NOT need this
+  (that's a bonus reason we build from source, not just to patch).
+
+---
+
+## 2. Building espressif-qemu from source (the fiddly part)
+
+`scripts/build-qemu.sh` encodes all of this, but here's *why*:
+
+- **Reuse nixpkgs' QEMU build env**: `nix develop nixpkgs#qemu --command …`
+  (or the repo devShell, which has `inputsFrom = [ pkgs.qemu ]`). Getting glib/
+  pixman/meson/ninja right by hand is painful; borrow qemu's.
+- **slirp**: the fork *unconditionally compiles* `net/slirp.c` even with
+  `--disable-slirp` (build-system bug), so you must **enable** it. nixpkgs
+  provides libslirp; `--enable-slirp` + libslirp on `PKG_CONFIG_PATH` works, or
+  it falls back to the bundled subproject.
+- **libgcrypt**: the ESP32-C3 crypto devices (`esp32c3_aes.c`, rsa, ds, xts) are
+  gated on `gcrypt.found()`. QEMU prefers gnutls and **skips gcrypt unless you
+  pass `--enable-gcrypt`**. Without it the build succeeds but the machine
+  **fails at runtime**: `unknown type 'misc.esp32c3.aes'`.
+- **Meson caches `pkg_config_path`**: exporting `PKG_CONFIG_PATH` after the
+  first `configure` is ignored on reconfigure. Use
+  `meson configure build -Dpkg_config_path=…` or pass it to `./configure`.
+- Config that works: `--target-list=riscv32-softmmu --enable-gcrypt
+  --enable-slirp --disable-werror --disable-docs --disable-tools`.
+- **`/tmp/qsrc` is where I built it this session and is EPHEMERAL** (lost on
+  reboot). The patch is the durable artifact; `make qemu` reclones+rebuilds.
+
+---
+
+## 3. How espressif-qemu's esp32c3 machine is structured
+
+- File: `hw/riscv/esp32c3.c`. It hand-builds the SoC (no device tree).
+- **Catch-all MMIO**: `esp32c3_io_ops` (`esp32c3_io_read`/`write`) covers the
+  whole peripheral window from `ESP32C3_IO_START_ADDR` (== `DR_REG_UART_BASE` ==
+  `0x60000000`) for `0xd1000` bytes. **Any peripheral without a dedicated
+  device falls here and reads 0 / ignores writes.** That's why unmodeled
+  peripherals silently misbehave instead of faulting.
+- Turn on `#define ESP32C3_IO_WARNING 1` (top of the file) to log every
+  unsupported access as `[ESP32-C3] Unsupported read/write to $ADDR`. **This is
+  the single most useful debugging lever.** It's noisy (megabytes/sec), so run
+  ~12 s and bucket by address:
+  ```sh
+  grep 'Unsupported read' log | sed -E 's/.*\$//' | sort | uniq -c | sort -nr | head
+  ```
+  The hottest address is almost always the next thing to model. Leave it `0` in
+  committed code.
+- Devices that **already exist** in the fork: UART, GPIO (barebones — we fleshed
+  it out), SPI1 (flash only), GDMA, AES/SHA/RSA/HMAC/DS/XTS, RTC_CNTL, systimer,
+  timg, efuse, intmatrix, cache, JTAG, TWAI, `esp_rgb` (a QEMU-specific virtual
+  framebuffer, NOT real X3 hardware — we disable it on the xteink machine).
+- Devices that **did NOT exist** and we added: SAR ADC, SPI2, the UC8253 panel,
+  and real GPIO register behavior. There is **no I²C controller model** — this
+  matters for X3/X4 detection (see §7).
+
+---
+
+## 4. Peripheral register facts (discovered, verified against firmware)
+
+Base addresses from ESP-IDF `soc/esp32c3` headers; all confirmed live.
+
+### SAR ADC — `DR_REG_APB_SARADC_BASE = 0x60040000`
+- The **original boot blocker**: firmware polls `INT_RAW` (`+0x44`) bit 31
+  (`ADC1_ONESHOT_DONE`) after starting a conversion; the unmodeled reg returned
+  0 forever → busy-loop with interrupts off → **interrupt-WDT panic**
+  (`Guru Meditation … Interrupt wdt timeout`). The panic dump's `S0/FP =
+  0x60040000` literally pointed at the ADC base.
+- Model (`hw/adc/esp32c3_adc.c`): `INT_RAW` always returns done bits 31|30;
+  `ONETIME_SAMPLE` (`+0x20`) bits [28:25] select the channel; `1_DATA_STATUS`
+  (`+0x2c`) / `2_DATA_STATUS` (`+0x30`) return the selected channel's 12-bit
+  value. Per-channel values are QOM props `adci[*]`, default `0xfff`.
+- `adc_oneshot_ll_get_raw_result` reads `adc1_data & 0xfff`.
+
+### SPI2 (display + SD) — `DR_REG_SPI2_BASE = 0x60024000`
+- **Completely different register map from SPI1** (`hw/ssi/esp32c3_spi.c` is
+  flash-specific — do NOT reuse it). GP-SPI2 layout: `CMD 0x00` (USR = bit24,
+  UPDATE = bit23), `MS_DLEN 0x1c` (bits[17:0] = bitlen-1), data buffer
+  `W0..W15` at `0x98..0xd4`.
+- Model (`hw/ssi/esp32c3_spi2.c`): on `CMD` write with USR set, clock
+  `(MS_DLEN/8)+1` bytes out of W0.. onto an `SSIBus` via `ssi_transfer`, writing
+  RX back into the same words. **DMA path is intentionally unimplemented** — the
+  firmware's CPU-buffer path is enough; add DMA only if a future firmware needs
+  it (it'll show up as SPI2 traffic that the CPU-buffer model can't explain).
+
+### GPIO — `DR_REG_GPIO_BASE = 0x60004000`
+- The stock model only returned the strap register. We added real
+  `OUT 0x04` / `OUT_W1TS 0x08` / `OUT_W1TC 0x0c` / `ENABLE 0x20` (+W1TS/W1TC) /
+  `IN 0x3c`, with 32 named input + output qemu_irq lines
+  (`ESP32_GPIO_INPUT`/`ESP32_GPIO_OUTPUT`).
+- **CRITICAL SUBTLETY**: an output line must read **HIGH when the pin is
+  configured as input** (`output_enable` bit clear), not follow the (zero) OUT
+  latch. Real boards have pull-ups; the X3 display **CS is pulled high** while
+  SD is being probed. Without this, CS floated LOW during SD init and the panel
+  greedily consumed the SD command bytes (`FF … 40 00 … 95`) as garbage
+  commands. This one bit cost real debugging time.
+
+### SD-over-SPI — QEMU's `ssi-sd` + `sd-card-spi`
+- These exist upstream; we attach them to SPI2 at CS12. But the 9.2 fork's
+  `ssi-sd.c` has a **bug**: CMD58 (read OCR) hardcodes the R1 byte to `0x01`
+  (idle) even after ACMD41 reported ready → SdFat rejects the card → firmware
+  shows "SD card error". Fix: track `idle` state, clear it when ACMD41 returns
+  0, return it for CMD58. (Upstream qemu master rewrote this path entirely; we
+  did the minimal equivalent + a vmstate version bump to 8.)
+
+---
+
+## 5. The e-ink panel model (`hw/display/xteink_x3_eink.c`)
+
+- X3 uses a **UC8253** controller, 792×528, 99 bytes/row, **52 272 bytes/plane**.
+- Protocol (from `chip/eink-x3.chip.c`, the old Wokwi model — keep it as the
+  reference): DC low = command, DC high = data. `0x10`=DTM1 (old plane),
+  `0x13`=DTM2 (new plane), `0x12`=refresh (render + pulse BUSY), `0x04`/`0x02`=
+  power on/off (pulse BUSY). Bit 1 = white, 0 = black. Rows arrive
+  **bottom-to-top** (wire row i → framebuffer row H-1-i).
+- BUSY is a GPIO output from the panel back into GPIO6 (active low; we pulse it
+  low for 5 ms of virtual time via a QEMUTimer, then high).
+- **Rendered to a QEMU graphical console** (not a custom file callback) so the
+  WASM front-end reuses QEMU's display surface. Capture headless with QMP
+  `screendump` → PPM.
+- **Orientation**: the controller framebuffer (792×528 landscape) is **rotated
+  90° clockwise** to the physical portrait panel (**528×792**). We bake that
+  into the render loop (`screen_x = H-1-y`, `screen_y = x`) so the console is
+  already physical-orientation. Verified by eye: text reads upright at 528×792.
+- **SSI gotcha**: a `TYPE_SSI_PERIPHERAL` subclass **must set `ssi->realize`**
+  even if empty — the base class calls it unconditionally and a null pointer
+  segfaults at machine init. Also set `cs_polarity = SSI_CS_LOW` and
+  `user_creatable = false`.
+
+---
+
+## 6. Machine wiring (`xteink_x3_machine_init`)
+
+`xteink-x3` is a subclass of the `esp32c3` machine that, after the base init:
+- creates the panel on `spi2.bus`, wires GPIO out 21→CS, 4→DC, 5→RST, and panel
+  BUSY→GPIO in 6;
+- creates `ssi-sd` at CS index 1 on `spi2.bus`, wires GPIO out 12→its CS, and a
+  `sd-card-spi` backed by `drive_get(IF_SD, 0, 0)` (QEMU block layer — the SD
+  "interface" you asked for; local `-drive` now, browser blockdev later);
+- **defaults both CS lines high** at init (`qemu_set_irq(cs, 1)`) so nothing is
+  selected before the firmware drives GPIO.
+
+Pin facts (X3, from `freeink-sdk` `BoardConfig.h` `XTEINK_X3`): display
+SCLK8/MOSI10/CS21/DC4/RST5/BUSY6; SD shares SCLK8/MOSI10, MISO7, CS12; button
+ADC ladders on GPIO1 & GPIO2; power button GPIO3; UC8253 @ 16 MHz.
+
+---
+
+## 7. X3 vs X4 detection — the thing that wasted time
+
+- **Both X3 and X4 ship in ONE ESP32-C3 binary.** Which panel/battery profile is
+  used is decided **at runtime** by an I²C fingerprint on SDA20/SCL0: presence of
+  BQ27220 gauge (0x55), DS3231 RTC (0x68), QMI8658 IMU (0x6B). ≥2 of 3 on two
+  passes ⇒ X3; 0 ⇒ X4. Result is cached in NVS (`cphw/dev_det`) and can be
+  overridden (`cphw/dev_ovr`: 1=X4, 2=X3).
+- **QEMU models no I²C**, so detection currently returns X4 by default. The
+  **crosspoint-reader-lua** firmware never calls `setDisplayX3()` at all → it is
+  **X4-only in emulation** and drives SSD1677 commands (`0x18/0x0c/0x44/0x45…`),
+  which our UC8253 panel ignores (→ blank screen). Don't chase that as a bug;
+  it's the wrong firmware for an X3 machine.
+- To test the X3 display this session I built the **canonical** firmware with a
+  **one-line temporary hack**: `detectDeviceTypeWithFingerprint()` in
+  `/tmp/crosspoint-reader-main/lib/hal/HalGPIO.cpp` returns `X3` at the top.
+  **THIS EDIT IS IN /tmp AND WILL BE LOST ON REBOOT.** The merged test image is
+  `/tmp/crosspoint-main-x3-flash.bin` (also ephemeral).
+- **Production plan (no custom firmware):** the `xteink-x3` machine should expose
+  the three X3 I²C fingerprint chips (needs an esp32c3 I²C controller model +
+  trivial BQ27220/DS3231/QMI8658 stubs that return plausible WHO_AM_I/values),
+  OR the emulator seeds the NVS `dev_ovr` key. Either makes stock firmware
+  auto-select X3. The `xteink-x4` machine simply omits the fingerprint. This is
+  how launch-time X3/X4 selection stays firmware-agnostic.
+
+---
+
+## 8. Reproducibility model
+
+- **Durable = the patch** (`qemu/patches/0001-xteink-x3-machine.patch`), verified
+  to `git apply` cleanly onto a pristine checkout of the pinned tag. When you
+  edit `/tmp/qsrc`, **regenerate it** and re-verify:
+  ```sh
+  cd <qemu-src> && git add -N <new files> && git diff --binary > <repo>/qemu/patches/0001-xteink-x3-machine.patch
+  git apply --reverse --check <repo>/qemu/patches/0001-xteink-x3-machine.patch   # must succeed
+  ```
+  (Reverse-check on the dirty tree proves patch == pristine→current diff, which
+  is equivalent to forward-applying to pristine.)
+- Firmware/SD/flash images and `/tmp/qsrc` are all disposable; `make qemu` +
+  `make firmware sdimage` rebuild them.
+
+---
+
+## 9. Debugging workflow cheat-sheet
+
+- **Headless run + capture** (canonical loop this session):
+  ```sh
+  qemu-system-riscv32 -machine xteink-x3 -L <src>/pc-bios -display none -serial null \
+    -qmp unix:/tmp/q.sock,server=on,wait=off \
+    -drive file=flash.bin,if=mtd,format=raw -drive file=sd.img,if=sd,format=raw &
+  sleep 200   # this CPU is slow — give boot real time
+  scripts/qmp-screendump.py /tmp/q.sock out.ppm
+  ```
+- **Inspect the screen**: convert with `nix shell nixpkgs#imagemagick -c magick
+  out.ppm out.png`; the black-pixel count is a cheap "did the screen change"
+  signal (home ≈ 15914, browse ≈ 11962, boot-white = 962).
+- **Inject a button** (proper edge = set then clear; hold-from-boot gives no
+  press edge): `scripts/qmp-qom.py /tmp/q.sock set /machine/adc "adci[1]" 2760`
+  then `… 4095` to release. Ladder-1 midpoints: Back 3512 / Confirm 2694 /
+  Left 1493 / Right ~5; ladder-2: Up 2242 / Down ~5. **>3900 = no button.**
+- **SD protocol trace**: flip `//#define DEBUG_SSI_SD 1` in `hw/sd/ssi-sd.c`.
+- **Panel trace**: add an `info_report` in `xteink_x3_eink_transfer`
+  (bounded with a static counter — it's hot).
+- **Symbolize a panic PC**: `llvm-addr2line -f -C -e firmware.elf 0xADDR`
+  (ignore DWARF "premature terminator" warnings).
+- **Flash images must be 2/4/8/16 MB.** The partition table spans 16 MB (app is
+  <4 MB), so always pad to 16 MB (`truncate -s 16M`); a mismatched size errors
+  with "only 2, 4, 8, and 16MB images are supported".
+
+---
+
+## 10. Next steps (in priority order)
+
+1. **X3 fingerprint or NVS seeding** (§7) so stock firmware auto-selects X3 with
+   no source edit. Smallest path to a genuinely firmware-agnostic X3 machine.
+   Needs a minimal esp32c3 I²C controller + 3 tiny I²C target stubs, OR NVS
+   pre-seed. This is the last thing standing between "works with a hacked build"
+   and "works with any downloaded binary."
+2. **`xteink-x4` machine**: SSD1677 panel (different command set — `0x18` set-RAM,
+   `0x44/0x45` window, `0x4e/0x4f` cursor, `0x24` write, `0x22/0x20` update;
+   BUSY active-HIGH), X4 battery via ADC (not I²C gauge), 792×528? confirm X4
+   geometry from `BoardConfig.h XTEINK_X4`. Sibling machine + a second panel
+   device; reuse ADC/SPI2/GPIO/SD unchanged.
+3. **Compile to WASM**: combine this patched fork with `ktock/qemu-wasm`'s WASM
+   TCG/TCI backend + Emscripten. Expect toolchain friction; the device models
+   themselves are portable C and shouldn't need changes. De-risk by first
+   building an *unpatched* esp32c3 target under qemu-wasm, then apply our patch.
+4. **Static browser front-end**: `<canvas>` fed from the display surface; DOM
+   buttons → the ADC/GPIO QOM injection (same calls as `qmp-qom.py`); file input
+   → emulated flash; preloaded/uploaded SD image → block backend; X3/X4 chosen
+   before boot. Must run fully from static assets (airgapped).
+
+---
+
+## 11. Key paths
+
+| What | Where |
+|------|-------|
+| Emulator repo | `/home/evanreichard/Development/git/personal/xteink-web-emulator` |
+| The patch | `qemu/patches/0001-xteink-x3-machine.patch` |
+| QEMU source (ephemeral) | `_scratch/qemu-src` (via `make qemu`); was `/tmp/qsrc` this session |
+| Canonical firmware (X3 hack in /tmp, ephemeral) | `/tmp/crosspoint-reader-main` (`lib/hal/HalGPIO.cpp` forced X3) |
+| Lua firmware (X4-only in emu) | `/home/evanreichard/Development/git/personal/crosspoint-reader-lua` |
+| Upstream firmware refs | `crosspoint-reader/crosspoint-reader` (canonical, cleaner) + its `freeink-sdk` submodule |
+| espressif/qemu tag | `esp-develop-9.2.2-20260417` |
+| qemu-wasm | `github.com/ktock/qemu-wasm` |