evan
376d302e1d
Bumps third_party/qemu to the wasm32 store-TB TCI workaround so the browser reader no longer panics in uzlib font decompression. Adds docs/wasm-inflate-panic.md with the full investigation (native control, non-deterministic reject, JIT promotion) and next steps. Updates ISSUES: the panic is worked around; the reader now inverts into muddled grayscale after the initial BW page (new top reader defect). tests navigate the 4th Open press into the reader.
121 lines
6.5 KiB
Markdown
121 lines
6.5 KiB
Markdown
# WASM-only reader panic in uzlib inflate
|
|
|
|
Living record of the investigation into the browser-only crash when opening a
|
|
book. Native QEMU (same submodule) does **not** crash, so this is a wasm32 TCG
|
|
backend bug. Update as things are ruled in/out.
|
|
|
|
## Symptom
|
|
|
|
Opening a book in the browser panics; native opens the same book fine.
|
|
|
|
```text
|
|
Invalid read at addr 0x3FCB7999, size 1, region '(null)' size 0x0 owner '(none)' ram=0, reason: rejected
|
|
Guru Meditation Error: Core 0 panic'ed (Load access fault). Exception was unhandled.
|
|
MEPC : 0x4201e210 RA : 0x4201e18c SP : 0x3fcb3810
|
|
```
|
|
|
|
## Established facts
|
|
|
|
- **Fault site (via same-version local ELF addr2line, not byte-identical):**
|
|
`uzlib_uncompress` (tinflate.c:613) ← `tinf_inflate_block_data` (tinflate.c:476)
|
|
← `InflateReader::read` ← `FontDecompressor::decompressGroup` ← `prewarmCache`.
|
|
"Indexing" = the reader inflating compressed glyph groups from flash into a
|
|
`malloc`'d temp buffer.
|
|
- **Input is static flash data**, independent of SD. The compressed font bank is
|
|
in the firmware image; the SD file only selects *which* glyph groups get
|
|
prewarmed (the characters present in the text).
|
|
- **Native renders the reader** ("chapter one") with no fault — proven with
|
|
`_scratch/native-reader-repro.py` + QMP screendumps. Confirms the firmware,
|
|
the font data, and the decompression logic are all correct.
|
|
- **Fault address is non-deterministic:** `0x3FCB7999` and `0x3FCB7997` across
|
|
runs. A static machine-map boundary would fault at a fixed address. Drift ⇒ a
|
|
computed pointer is slightly wrong, i.e. runtime/codegen, not the memory map.
|
|
- **Rejected region is bogus:** `region '(null)' size 0x0 owner '(none)' ram=0`.
|
|
The address lands in mapped DRAM (`0x3fc80000`+`0x60000`) but the wasm softmmu
|
|
TLB routed it to the IO slow path with a zero-size section. Hallmark of a bad
|
|
address/TLB result from a JIT-compiled TB.
|
|
- **Menus work.** Home/file-browser render crisp text through the same font
|
|
system and the same TLB fast path, so the TLB codegen is not broadly broken —
|
|
the bug is pattern/data specific in a hot, JIT-promoted TB.
|
|
- **User observation:** firmware-written crash logs open fine; prepopulated
|
|
files do not. Consistent with data-dependence — a crash log's character set
|
|
prewarms different glyph groups than lorem-ipsum, avoiding the group whose
|
|
compressed byte pattern trips the miscompiled path.
|
|
|
|
## Architecture notes (why JIT is implicated)
|
|
|
|
- `tcg/wasm32.c`: TBs run on the TCI interpreter until executed
|
|
`INSTANTIATE_NUM` times, then get compiled to native WASM (`instantiate_wasm`)
|
|
and run as added table functions. The inflate hot loop gets promoted.
|
|
- `WASM_TCI_ONLY_ENTRY` forces a TB to stay on TCI forever;
|
|
`wasm_tci_only_tb` (per-TB, reset each translation) requests that.
|
|
- Dynamic promotion must stay on for usable performance (all-TCI starves the
|
|
guest watchdog), so a permanent fix must correct codegen, not disable JIT.
|
|
|
|
## Hypotheses
|
|
|
|
| # | Hypothesis | Status |
|
|
|---|-----------|--------|
|
|
| H1 | A store-containing TB is miscompiled (the cleanup removed `wasm_tci_only_tb = true` from `tcg_out_qemu_st`); the compiled TB corrupts a pointer a later load dereferences | **confirmed** (E1) |
|
|
| H2 | Compiled **load** codegen bug (`tcg_wasm_out_qemu_ld` / `_direct`) | untested |
|
|
| H3 | TLB fast-path codegen edge case (`tcg_wasm_out_tlb_load`) hit only by certain access patterns | untested |
|
|
| H4 | asyncify/fiber unwind corrupts vCPU state mid-inflate | unlikely (inflate makes no async import calls) |
|
|
| H5 | mimalloc / linear-memory / threads issue | unlikely (native heap layout differs, no crash) |
|
|
|
|
## Experiments
|
|
|
|
### E1 — force store-containing TBs to TCI (tests H1) ✅ CONFIRMED
|
|
Re-added `wasm_tci_only_tb = true;` to `tcg_out_qemu_st` in
|
|
`tcg/wasm32/tcg-target.c.inc`.
|
|
|
|
- Result: **panic gone.** `make browser-test` reaches the reader and renders a
|
|
correct page (black lorem-ipsum on white, footer `80% example 1/3 33%`). No
|
|
reject, no `Guru Meditation`, 1 boot, 0 watchdog panics. New refresh frames
|
|
(`dtm2_white=0.891`) appear past the old `0.960` stall.
|
|
- **Conclusion: the compiled `qemu_st` codegen is the bug.** Forcing store TBs
|
|
to TCI is a working *workaround*, not the fix — every store-containing TB
|
|
then runs interpreted (broad perf cost; prior attempts reported watchdog
|
|
starvation under sustained load, though this finite-burst run was clean).
|
|
- The reader first paints a correct BW page (black text on white), then a
|
|
follow-up **grayscale** refresh pass inverts it into muddled white-on-black
|
|
text. That second issue is real and still open (see ISSUES.md "Reader renders
|
|
inverted grayscale"); it is independent of this panic fix. The `browser-test`
|
|
wait-for-stable happened to screenshot the initial white page.
|
|
|
|
### Caveat on E1's specificity
|
|
Forcing store-*containing* TBs to TCI drops the whole TB (its loads, stores,
|
|
and arithmetic) to the interpreter. So the bug is *in a TB that also has a
|
|
store*, not provably the store op. `tcg_wasm_out_qemu_st_direct` and
|
|
`tcg_wasm_out_qemu_ld_direct` are byte-for-byte symmetric and look correct in
|
|
isolation, which points away from the direct emit and toward the store path's
|
|
surrounding block/rewind/helper scaffolding in `tcg_wasm_out_qemu_st`, or an
|
|
interaction with a co-located op.
|
|
|
|
### Next options
|
|
1. **Ship E1 as a guarded workaround** — reader works now; measure real
|
|
navigation/page-turn perf before judging it unacceptable.
|
|
2. **Localize precisely** — bisect by making the TCI trigger conditional
|
|
(store width; presence of the miss/helper path; TB size) to shrink the
|
|
suspect set, then read the generated wasm for that TB.
|
|
|
|
## Diagnostics currently in the tree (uncommitted)
|
|
|
|
- `system/memory.c`: reject log prints region size/owner/ram to identify the
|
|
faulting region.
|
|
- `hw/display/xteink_x3_eink.c`: `[eink] refresh` popcount trace (per-plane
|
|
white ratio) — also used for the separate grayscale-rendering issue.
|
|
|
|
## Reproduction
|
|
|
|
- **WASM:** `make browser-test` (the test does 4 confirm presses: Browse Files →
|
|
Test → highlight → Open). Panic appears in `_scratch/browser-raw-sd.log`.
|
|
- **Native (control, no panic):** `nix develop -c python3 _scratch/native-reader-repro.py`,
|
|
then inspect `_scratch/native-step4-reader.ppm`.
|
|
|
|
## Ruled out
|
|
|
|
- Not SD / filesystem: fixed separately (raw FAT32, SSI-SD token, CMD13); the
|
|
file reads correctly and "Indexing" begins.
|
|
- Not firmware / font data: native decompresses and renders identical input.
|
|
- Not the machine memory map: fault address drifts; native map is the same.
|