xteink-web-emulator/docs/wasm-inflate-panic.md

WASM-only reader panic in uzlib inflate

Living record of the investigation into the browser-only crash when opening a book. Native QEMU (same submodule) does not crash, so this is a wasm32 TCG backend bug. Update as things are ruled in/out.

Symptom

Opening a book in the browser panics; native opens the same book fine.

Invalid read at addr 0x3FCB7999, size 1, region '(null)' size 0x0 owner '(none)' ram=0, reason: rejected
Guru Meditation Error: Core 0 panic'ed (Load access fault). Exception was unhandled.
MEPC : 0x4201e210  RA : 0x4201e18c  SP : 0x3fcb3810

Root cause (established)

wasm_tci_only_tb = true in tcg_out_qemu_st is upstream qemu-wasm's own design, not a workaround this project invented. A full diff of our tcg/wasm32/tcg-target.c.inc against pristine upstream qemu-wasm shows the two files are identical except for that one line. Upstream unconditionally forces every store-containing TB to the TCI interpreter, so upstream's compiled (JIT) store path is emitted but never executed — it is effectively unfinished, untested dead code, and it corrupts guest state the moment it runs.

Mechanism: the wasm32 backend compiles a TB as a chain of wasm blocks guarded by if (BLOCK_PTR <= block_idx). When a softmmu slow-path helper yields (asyncify unwind for I/O / interrupt), the wasm function returns and re-enters, skipping already-executed blocks via BLOCK_PTR. This restart is safe to replay for loads but not for a compiled store, so upstream punts all store TBs to TCI (which has precise instruction-level checkpointing).

The real fix is therefore not a port bug hunt but completing upstream's compiled store path to be rewind-safe (e.g. defer the memory commit past the TB's yield points / buffer stores until block commit), then dropping the blanket TCI. Large TCG-backend task; ~15 min per browser iteration; no native repro.

Established facts

• Fault site (via same-version local ELF addr2line, not byte-identical): uzlib_uncompress (tinflate.c:613) ← tinf_inflate_block_data (tinflate.c:476) ← InflateReader::read ← FontDecompressor::decompressGroup ← prewarmCache. "Indexing" = the reader inflating compressed glyph groups from flash into a malloc'd temp buffer.
• Input is static flash data, independent of SD. The compressed font bank is in the firmware image; the SD file only selects which glyph groups get prewarmed (the characters present in the text).
• Native renders the reader ("chapter one") with no fault — proven with _scratch/native-reader-repro.py + QMP screendumps. Confirms the firmware, the font data, and the decompression logic are all correct.
• Fault address is non-deterministic: 0x3FCB7999 and 0x3FCB7997 across runs. A static machine-map boundary would fault at a fixed address. Drift ⇒ a computed pointer is slightly wrong, i.e. runtime/codegen, not the memory map.
• Rejected region is bogus: region '(null)' size 0x0 owner '(none)' ram=0. The address lands in mapped DRAM (0x3fc80000+0x60000) but the wasm softmmu TLB routed it to the IO slow path with a zero-size section. Hallmark of a bad address/TLB result from a JIT-compiled TB.
• Menus work. Home/file-browser render crisp text through the same font system and the same TLB fast path, so the TLB codegen is not broadly broken — the bug is pattern/data specific in a hot, JIT-promoted TB.
• User observation: firmware-written crash logs open fine; prepopulated files do not. Consistent with data-dependence — a crash log's character set prewarms different glyph groups than lorem-ipsum, avoiding the group whose compressed byte pattern trips the miscompiled path.

Architecture notes (why JIT is implicated)

• tcg/wasm32.c: TBs run on the TCI interpreter until executed INSTANTIATE_NUM times, then get compiled to native WASM (instantiate_wasm) and run as added table functions. The inflate hot loop gets promoted.
• WASM_TCI_ONLY_ENTRY forces a TB to stay on TCI forever; wasm_tci_only_tb (per-TB, reset each translation) requests that.
• Dynamic promotion must stay on for usable performance (all-TCI starves the guest watchdog), so a permanent fix must correct codegen, not disable JIT.

Hypotheses

#	Hypothesis	Status
H1	A store-containing TB is miscompiled (the cleanup removed wasm_tci_only_tb = true from tcg_out_qemu_st); the compiled TB corrupts a pointer a later load dereferences	confirmed (E1)
H2	Compiled load codegen bug (tcg_wasm_out_qemu_ld / _direct)	untested
H3	TLB fast-path codegen edge case (tcg_wasm_out_tlb_load) hit only by certain access patterns	untested
H4	asyncify/fiber unwind corrupts vCPU state mid-inflate	unlikely (inflate makes no async import calls)
H5	mimalloc / linear-memory / threads issue	unlikely (native heap layout differs, no crash)

Experiments

E1 — force store-containing TBs to TCI (tests H1) ✅ CONFIRMED

Re-added wasm_tci_only_tb = true; to tcg_out_qemu_st in tcg/wasm32/tcg-target.c.inc.

• Result: panic gone. make browser-test reaches the reader and renders a correct page (black lorem-ipsum on white, footer 80% example 1/3 33%). No reject, no Guru Meditation, 1 boot, 0 watchdog panics. New refresh frames (dtm2_white=0.891) appear past the old 0.960 stall.
• Conclusion: the compiled qemu_st codegen is the bug. Forcing store TBs to TCI is a working workaround, not the fix — every store-containing TB then runs interpreted (broad perf cost; prior attempts reported watchdog starvation under sustained load, though this finite-burst run was clean).
• The reader first paints a correct BW page (black text on white), then a follow-up grayscale refresh pass inverts it into muddled white-on-black text. That second issue is real and still open (see ISSUES.md "Reader renders inverted grayscale"); it is independent of this panic fix. The browser-test wait-for-stable happened to screenshot the initial white page.

Caveat on E1's specificity

Forcing store-containing TBs to TCI drops the whole TB (its loads, stores, and arithmetic) to the interpreter. So the bug is in a TB that also has a store, not provably the store op. tcg_wasm_out_qemu_st_direct and tcg_wasm_out_qemu_ld_direct are byte-for-byte symmetric and look correct in isolation, which points away from the direct emit and toward the store path's surrounding block/rewind/helper scaffolding in tcg_wasm_out_qemu_st, or an interaction with a co-located op.

E2 — crash is icount-independent; loads-TCI also masks it (inconclusive on op)

• Removed the store flag and ran at the new noicount default: same crash (Invalid read at 0x3FCB80A5, drifts, MEPC 0x4201e210). So icount timing is not the trigger.
• Moved the flag to tcg_out_qemu_ld (loads TCI, stores compiled): passed (33s). Not decisive — load-containing TBs are a superset of store-containing TBs, so forcing loads to TCI drops nearly every TB (including all store TBs) to the interpreter. It masks the bug the same way store-TCI does.
• Takeaway: whole-TB TCI cannot isolate the offending op. Precise localization needs per-op TCI infrastructure (backend is whole-TB) or reading the generated wasm for one miscompiled TB. Large task; reader already works at ~31s with the store flag, so this is headroom, not a blocker.

E3 — PC-windowed bisection: not a single TB, it's aggregate

Added a compile-window to tcg_out_qemu_st (s->gen_tb->pc gate): store-TBs inside [LO,HI) compile, all others stay TCI. Each run rebuilt + browser-tested.

window	region	result
0x4201d000–0x4201f800	uzlib inflate + FontDecompressor	pass
0x40000000–0x40400000	ROM (memcpy/memset) + IRAM (tlsf/malloc)	pass
0x42000000–0x420d0000	flash text lower half (prewarm callers)	pass
0x420d0000–0x421a0000	flash text upper half	pass
(none / all store-TBs compiled)	everything	crash

Every region compiles safely in isolation; only the full set corrupts. So the bug is not a localizable miscompiled op/TB — it is an aggregate effect that appears once enough compiled store-TBs run together. The faulting load is d->dict_ring[d->lzOff] (tinflate.c:476) off a stack-resident inflate struct, i.e. some store clobbers a neighbouring stack/heap pointer, and which store depends on timing. This matches a rewind/replay or TB-instance-management race in the compiled restartable-block store path, not a codegen typo — and it is exactly why upstream forces every store-TB to TCI. Fixing it is architectural (rewind/instance-safe compiled stores), not a one-op correction.

E4-E7 — deterministic minimal repro, and what the bug is NOT

With a per-op gate in tcg_out_qemu_st (compile only store-TBs matching a criterion, TCI the rest) plus targeted instrumentation, each run rebuilt + browser-tested against the exact 1.4.1 firmware:

experiment	change	result
E4 lifecycle	never removeFunction retired TB instances (pin all)	crash
E5 helper-only	force every store through the softmmu helper (no direct store codegen)	crash
E6 width: byte only	compile TBs whose store is 8-bit	pass
E6 width: 16 only	16-bit	pass
E6 width: 32 only	32-bit	pass
E6 width: byte+16		pass
E6 width: 16+32		pass
E6 width: byte+32		crash
E7 bytewise-32	emit 32-bit store as four 8-bit stores, compile all	crash
precheck	skip store helper on rewind when DONE_FLAG already set	crash
nochain	-d nochain (disable TB chaining), compile all stores	crash
determinism	rerun the crashing all-compiled build 3x	crash x3, same gen 16, same MEPC

Established: the crash is deterministic (same point every run) but depends on the exact compiled-TB set. The minimal reproducer is a byte-store TB and a 32-bit-store TB compiled together; it is counter-monotonic — additionally compiling 16-bit-store TBs hides it.

Ruled out: single miscompiled TB (E1-E3), a specific memory region (inflate/ROM/IRAM/flash halves), TB chaining (nochain), instance/function lifecycle & removeFunction reuse (E4 pin), the store data codegen itself (E5 helper-only and E7 bytewise both crash), and store double-execution on rewind (precheck to skip the replayed helper still crashes).

Correction: the rule is monotonic, not counter-monotonic — {8,16,32} (original) also crashes, so 16-bit is irrelevant. The rule is simply: crash iff an 8-bit-store TB and a 32-bit-store TB are both compiled.

E8 — the 8-bit requirement is aggregate, not one TB

Compiled all 32-bit-store TBs, then compiled 8-bit-store TBs only inside one region at a time:

8-bit region compiled (with all 32-bit compiled)	result
flash lower 0x42000000-0x420d0000	pass
ROM+IRAM 0x40000000-0x40400000 (incl. memset)	pass
flash upper 0x420d0000-0x421a0000	pass
all 8-bit (no region gate)	crash

Those three regions cover essentially all executable memory, yet each passes while their union crashes. So the 8-bit side is aggregate too — there is no single interacting 8-bit TB. Combined with E1-E3 (region-aggregate) this proves the corruption is an emergent property of running many compiled store-TBs, which whole-TB TCI gating (the backend's only control) cannot dissect. The remaining paths are generated-wasm inspection of the BLOCK_PTR/register restart sequence, or building per-op TCI. A deterministic reproducer exists (remove wasm_tci_only_tb, open example.txt, faults at MEPC 0x4201e210, gen 16).

E9 — static analysis of generated wasm (dump + wasm2wat)

Added a node-only dump hook (wasm_dump_module, tcg.c finalize) that writes the first compiled store-TB and load-TB modules; _scratch/dump-tbs.mjs boots the firmware under node far enough to emit them. wasm2wat --enable-threads gives readable IR.

Finding 1 (real bug, fixed-then-shelved): the ld/st slow paths detect an asyncify yield with the thread-global DONE_FLAG (ctx+20), while the general call path uses UNWINDING (ctx+28), which is reset before each call and set only by the coroutine switch. DONE_FLAG is set by any nested ld/st helper, so a nested guest access inside a store/load helper that then yields makes the outer op falsely read "completed" and swallow the unwind. Converting ld/st to the UNWINDING mechanism is the correct fix. But rebuilding with compiled stores + the UNWINDING fix still crashes identically (gen 16, same MEPC), so the store helpers on this path never actually yield — this bug is latent here. Change shelved (not shipped) to avoid touching the working compiled-load path without a green test; recorded for a future upstream fix.

Finding 2 (reframes the root cause): combine E5 with the load result. E5 routes every store through the C softmmu helper (correct memory write) and still crashes; loads share the identical restart/block structure and work. So the corruption is not in the store data codegen, the restart machinery, or the yield handling. What remains is that a store writes guest memory using an address/value taken from guest registers (wasm globals). A compiled TB that computes a wrong register value corrupts memory only through a store — a wrong address fed to a load just reads harmless garbage. This matches every prior result: emergent (depends which ops precede a store), deterministic, and non-localizable by store region or width. The remaining hunt is a register-producing op whose compiled codegen diverges from TCI, exposed destructively via stores — which needs a compiled-vs-TCI per-TB register differential, not more whole-TB gating.

Tooling retained: wasm_dump_module + wasm_tb_had_store/load tags + _scratch/dump-tbs.mjs.

Next options

1. Ship E1 as a guarded workaround — reader works now; measure real navigation/page-turn perf before judging it unacceptable.
2. Localize precisely — bisect by making the TCI trigger conditional (store width; presence of the miss/helper path; TB size) to shrink the suspect set, then read the generated wasm for that TB.

Diagnostics currently in the tree (uncommitted)

• system/memory.c: reject log prints region size/owner/ram to identify the faulting region.
• hw/display/xteink_x3_eink.c: [eink] refresh popcount trace (per-plane white ratio) — also used for the separate grayscale-rendering issue.

Reproduction

• WASM: make browser-test (the test does 4 confirm presses: Browse Files → Test → highlight → Open). Panic appears in _scratch/browser-raw-sd.log.
• Native (control, no panic): nix develop -c python3 _scratch/native-reader-repro.py, then inspect _scratch/native-step4-reader.ppm.

Ruled out

• Not SD / filesystem: fixed separately (raw FAT32, SSI-SD token, CMD13); the file reads correctly and "Indexing" begins.
• Not firmware / font data: native decompresses and renders identical input.
• Not the machine memory map: fault address drifts; native map is the same.

E10: compiled-vs-TCI per-TB differential (opt-in ?tcgdiff) — store codegen exonerated

Built a per-TB differential checker to settle whether single-TB compiled store codegen diverges from TCI. It runs authoritative TCI first, snapshots its result, rewinds guest state, runs the compiled TB as a disposable shadow, compares, then restores the TCI result. Opt-in via ?tcgdiff (wasm_diff_enable() from web/app.js preRun); normal/browser execution is unchanged.

Making the comparison valid — not the codegen — was the whole task. Five harness defects each masqueraded as a "register divergence" until fixed:

1. Execution order — running the compiled shadow first contaminated TCI's machine state. Fixed: TCI first, shadow second, restore TCI.
2. Partial-TB comparison — helper/slow-path TBs don't reach exit_tb. Fixed with a done_flag == 2 sentinel: only fully-inline store TBs that complete (done_flag == 2 && tb_ptr == 0) are compared.
3. Negative-offset state — the TB-entry interrupt check reads env-8 (CPUNegativeOffsetState, before CPUArchState). Snapshot/restore now covers the full CPUNegativeOffsetState.
4. Deep TLB state — CPUTLB.f[].table / d[].fulltlb are separately allocated; TCI warms the soft-TLB via its ld/st helper path, so the shadow must start from an identical TLB. Now deep-copied per MMU mode.
5. Async exit-flag race — the IO thread asynchronously sets icount_decr.high = -1 via cpu_exit(). TCI can take the TB-entry interrupt exit while the restored shadow sees a cleared flag and runs fully. These are unavoidable concurrency races, so comparisons where the force-exit flag was set around the TCI run are skipped.

Only side-effect-safe candidates are compared: store-containing TBs with no general helper call (wasm_tb_had_helper), restricted to the app-flash PC window. Live dispatch stays TCI for all store TBs; the compiled function runs only as a shadow.

Result: 160,000+ store-TB comparisons during a full reader-open, zero register or DRAM divergence. Single-TB compiled store codegen is bit-for-bit identical to TCI. This exonerates per-TB store codegen and confirms the corruption is emergent across TBs (consistent with E1–E8). A per-TB differential is structurally blind to a multi-TB interaction bug; catching it needs a whole-execution differential (two full machines stepped in lockstep from reset), not more per-TB work.

The ?tcgdiff harness is expensive (full DRAM + TLB snapshot and double execution per candidate; heap grows to ~1 GiB, frame gen drops) but is debug-only and does not affect the production store-TCI path.

E11: whole-execution differential (opt-in ?tcgdiff2) — live compiled vs TCI shadow

Mode 2 inverts E10: the compiled path runs LIVE and authoritative (the real buggy path, so corruption accumulates across TBs exactly as in the crash), while a per-TB TCI shadow — restored from the pre-snapshot — is the known-good reference. Store-TCI is disabled under this mode so compiled stores actually execute. Per eligible store TB: snapshot pre (env, neg, deep TLB, DRAM) -> run compiled live, capture full compiled-post -> rewind -> run TCI shadow -> compare -> restore compiled-post -> continue. Opt-in via ?tcgdiff2.

Result

• 153,000+ live-compiled-vs-TCI comparisons with accumulation, zero divergence (bootloader region). Combined with E10 (160k reader-path), inline compiled stores are correct even under live cross-TB accumulation.
• Revealed a large blind spot: ~90% of store TBs complete via the store slow-path (done_flag == 1, TLB-miss -> ld/st helper), not the pure inline path (done_flag == 2). Mode 2 was extended to compare slow-path completions too.

Why it cannot reach the reader corruption (impractical in-browser)

The 2x execution + full DRAM (0x60000) + deep-TLB snapshot per eligible TB is ~1000x slower than normal. Consequences:

• The guest interrupt watchdog fires first: gating comparison to app-flash produced an Interrupt wdt timeout panic (MEPC 0x40380264, IRAM idle) during cold boot — a harness-overhead artifact, not the corruption Guru Meditation (MEPC 0x4201e210).
• Ungated, it reached only ~153k comparisons (bootloader, gen 2) before the 1100s timeout. Full reader-open is millions of TBs away. Defanging the watchdog would not help; raw throughput is the wall.

Narrowed suspects (what the differential structurally cannot cheaply test)

1. Store slow-path (TLB miss -> helper_stN_mmu -> Asyncify unwind/resume), the ~90% of stores mode 2 excluded by the done_flag == 2 gate. Aligns with E9's DONE_FLAG (thread-global) vs UNWINDING (per-call) finding.
2. Helper-containing store TBs, excluded by the side-effect-safety filter.
3. 8/32-bit store-width interaction (E1-E8's "needs both widths").

Decision

In-browser whole-execution differential is retired as a repro path: it proves live-accumulation correctness of inline stores but cannot reach the crash. Next instrument is a deterministic, framework-free slow-path store differential unit test (no boot, no watchdog, no perf wall) targeting suspect (1)/(3). Option A (sequential deterministic two-pass) remains a documented future option if a full-execution comparison is later required.

Both ?tcgdiff (E10) and ?tcgdiff2 (E11) are retained as opt-in tools; production store-TCI path is unchanged.

E12/E13: normal-speed compiled stores and file-open-armed E11

?nostoretci probe

Added opt-in ?nostoretci to disable the store-TCI workaround without enabling any differential overhead. Result: the exact original corruption still reproduces at normal speed during reader open:

• Load access fault
• MEPC 0x4201e210
• MTVAL 0x3fcb8004 / invalid read near the inflate/text buffer region

So store-TCI is still required in production.

?tcgdiff2=open

Arming E11 from boot was too slow or perturbed app init. Added a browser-test hook so ?tcgdiff2=open arms wasm_diff_enable2() immediately before the test presses Confirm to open example.txt. Boot and app init therefore run on the normal production path, and compiled stores become live only over the repro interval.

With the all-app-flash window (0x42000000..0x42200000), the first observed mode-2 divergence after arming is register state, not DRAM:

TCGDIFF pc=0x4201d6ea gpr[2] compiled=0x3fcb42f0 tci=0x3fcb42e0
pre-ra=0x42029374 pre-sp=0x3fcb42f0

This is utf8NextCodepoint(): TCI applies the stack-frame prologue (sp -= 16) while the compiled-live path has stale sp in CPUArchState. The later crash is still the known invalid read at MEPC 0x4201e210.

Current interpretation

E10 remains green because it forces no-chain and runs TCI live, then compiled as a disposable shadow. E11 is the real compiled-live path and exposes state that is stale across compiled execution/chaining boundaries. The remaining suspect is no longer individual store codegen; it is compiled TB state materialization: WASM register globals and CPUArchState can diverge across real compiled-live entry/exit/chaining, and store-TCI masks the bad store-containing TBs by routing them through TCI.

A tested broad fix candidate — making store TBs return to C instead of using the same-module BLOCK_PTR chain — caused an app-init interrupt-watchdog timeout, so it is not viable as-is. A narrower fix should target state materialization (flush/reload live guest register state at compiled TB boundaries or before helper/exception-visible points) rather than disabling chaining wholesale.

E14: narrowed production fix — only 32-bit store TBs stay TCI

The blanket upstream qemu-wasm fallback interpreted every store-containing TB. The width interaction tests show that is broader than necessary:

• all stores compiled (?nostoretci) -> exact crash remains (MEPC 0x4201e210, invalid read near 0x3FCB80xx)
• only 8-bit qemu store TBs interpreted -> reader-open passes
• only 32-bit qemu store TBs interpreted -> reader-open passes

Production now keeps only 32-bit qemu store TBs on TCI; 8/16/64-bit store TBs compile normally. This breaks the bad 8-bit/32-bit compiled-store interaction while preserving more dynamic WASM promotion than the blanket store-TCI fallback. ?nostoretci remains as the normal-speed all-compiled negative-control repro and still crashes at the original panic.

Validation:

normal URL: browser raw-sd test passed, open example.txt took 36.8s
?nostoretci: Load access fault, MEPC 0x4201e210, invalid read 0x3FCB80A5

This fixes the browser panic without disabling dynamic promotion globally. The remaining upstream-quality improvement would be eliminating the 32-bit store-TCI subset entirely by finding the exact cross-width/shared-memory state bug.