From 2aee0765aa4f119c679799d3ee220757f1886397 Mon Sep 17 00:00:00 2001 From: Evan Reichard Date: Sun, 3 May 2026 22:43:06 -0400 Subject: [PATCH] docs: update README and AGENTS.md to reflect http.Server refactor - Replace "Raw TCP listener" references with http.Server/hijack model - Update architecture diagram and description - Add testing section to AGENTS.md with test summary - Update key patterns to match current code - Fix file locations table (add e2e_test.go, reconstructed_conn.go) --- AGENTS.md | 39 +++++++++++++++++++++++++++------------ README.md | 6 +++--- 2 files changed, 30 insertions(+), 15 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 8d2bfa6..2787ab6 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -22,17 +22,17 @@ Go 1.25+ is required (`go.mod`). Nix devshell provides Go, gopls, golangci-lint. ## Architecture at a Glance ``` -server/server.go — Raw TCP listener, reads Host header, routes to tunnel or control API -tunnel/tunnel.go — Core Tunnel struct, WebSocket message loop, stream management -tunnel/forwarder.go — Forwarder interface; factory selects HTTP or TCP forwarder +server/server.go — net/http.Server, ServeHTTP routes by Host subdomain to tunnel or control API +tunnel/tunnel.go — Core Tunnel struct, WebSocket message loop, stream management +tunnel/forwarder.go — Forwarder interface; HTTP/HTTPS → HTTP forwarder, everything else → TCP tunnel/http_forwarder.go — httputil.ReverseProxy served over net.Pipe via multiConnListener tunnel/tcp_forwarder.go — Direct net.Dial TCP forwarding -tunnel/stream.go — Stream interface (io.ReadWriteCloser + Source/Target) -store/store.go — In-memory request/response recorder with pub/sub (SSE) -web/web.go — Local tunnel monitor (port 8181), SSE endpoint -config/config.go — Reflection-based config from struct tags → flags + env vars -pkg/maps/map.go — Generic sync.RWMutex-guarded map -types/message.go — WebSocket message envelope (data | close) +tunnel/stream.go — Stream interface (io.ReadWriteCloser + Source/Target) +server/reconstructed_conn.go — Replays re-serialized headers + buffered body + raw conn after hijack +store/store.go — In-memory request/response recorder with pub/sub (SSE) +web/web.go — Local tunnel monitor (port 8181), SSE endpoint +config/config.go — Reflection-based config from struct tags → flags + env vars +pkg/maps/map.go — Generic sync.RWMutex-guarded map ``` ## Code Conventions @@ -46,9 +46,9 @@ types/message.go — WebSocket message envelope (data | close) ## Key Patterns -1. **Raw TCP → HTTP upgrade**: The server reads from a raw TCP connection to inspect the Host header before deciding whether to handle as a control API request or tunnel the connection via WebSocket. -2. **Reconstructed connection**: After reading the HTTP request from the raw conn, `reconstructedConn` replays the consumed bytes so the full TCP stream can be forwarded. -3. **Forwarder abstraction**: `Forwarder` interface decouples tunnel transport from protocol handling. HTTP forwarder uses `net.Pipe` + `multiConnListener` to feed connections into a standard `http.Server`. +1. **ServeHTTP + Hijack for tunnel routing**: The server uses `net/http.Server` with a `ServeHTTP` handler. It inspects the `Host` header to determine routing. Control API requests are handled normally via `http.ResponseWriter`. Tunnel requests hijack the TCP connection, re-serialize the HTTP request, and forward it through the tunnel via `reconstructedConn`. +2. **Reconstructed connection**: After hijack, `reconstructedConn` combines re-serialized request headers, buffered body data from the hijacked `bufio.ReadWriter`, and the raw connection into a single `io.Reader` so the tunnel client receives the complete request. +3. **Forwarder abstraction**: `Forwarder` interface decouples tunnel transport from protocol handling. `NewForwarder` only uses `url.Parse` for `http://`/`https://` schemes; everything else (including parse failures like bare `host:port`) is treated as raw TCP. HTTP forwarder uses `net.Pipe` + `multiConnListener` to feed connections into a standard `http.Server`. 4. **Context-threaded records**: Request records are attached to context in `RecordRequest` and retrieved in `RecordResponse` via the `ModifyResponse` hook. ## Adding a New Forwarder @@ -57,6 +57,20 @@ types/message.go — WebSocket message envelope (data | close) 2. Add a case in `tunnel.NewForwarder()` factory 3. Add corresponding `ForwarderType` const +## Testing + +E2E tests live in `e2e_test.go` at the project root. They spin up real servers, tunnels, and targets on random ports. + +```bash +# Run all tests +make tests + +# Run specific test + go test -v -run TestHTTPTunnelRoundTrip -count=1 ./... +``` + +16 tests covering: HTTP round-trip (GET/POST), TCP echo, large bodies (1MB resp / 512KB req), response quality (headers, content-type, content-length), error paths (404, 401, duplicate name), multi-tunnel routing, concurrency, and graceful shutdown. + ## File Locations | Concern | Files | @@ -69,3 +83,4 @@ types/message.go — WebSocket message envelope (data | close) | Web UI | `web/`, `web/pages/` | | Shared types | `types/` | | Utilities | `pkg/maps/` | +| E2E tests | `e2e_test.go` | diff --git a/README.md b/README.md index 0d17fe9..02a788f 100644 --- a/README.md +++ b/README.md @@ -23,7 +23,7 @@ Deploy Conduit on a public server (e.g., `https://conduit.example.com`), then cr │ tunnel │ (control + streams) │ serve │ │ (client) │ │ (public server) │ ├──────────────┤ ├──────────────────┤ -│ HTTP Fwd or │ │ Raw TCP listener │ +│ HTTP Fwd or │ │ http.Server │ │ TCP Fwd │ │ Subdomain router │ ├──────────────┤ │ WS upgrade │ │ Tunnel │ └──────────────────┘ @@ -31,7 +31,7 @@ Deploy Conduit on a public server (e.g., `https://conduit.example.com`), then cr └──────────────┘ ``` -The server accepts raw TCP connections, reads the HTTP `Host` header to determine routing. Requests to the base domain hit the control API; requests to subdomains are forwarded over WebSocket to the matching tunnel client. The client then forwards traffic to the local target via either a reverse-proxy (HTTP) or direct TCP dial. +The server uses `net/http.Server` to accept connections and inspects the HTTP `Host` header for routing. Requests to the base domain hit the control API; requests to subdomains are hijacked and forwarded over WebSocket to the matching tunnel client. The client then forwards traffic to the local target via either a reverse-proxy (HTTP) or direct TCP dial. ## Quick Start @@ -127,7 +127,7 @@ All options can be set via CLI flags or environment variables (`CONDUIT_` prefix ``` ├── cmd/ # Cobra CLI commands (root, serve, tunnel) ├── config/ # Configuration parsing & logging setup -├── server/ # TCP listener, subdomain routing, WebSocket upgrade +├── server/ # HTTP server, subdomain routing, hijack + WebSocket upgrade ├── tunnel/ # Tunnel, Stream, and Forwarder abstractions ├── store/ # In-memory request/response recording for the monitor ├── web/ # Local tunnel monitor HTTP server & SSE streaming