fix(api): bound SSE relay writes with a deadline to sever stalled readers

relayLogStream copied a pod-log follow to the client with a plain
flusher.Flush per event. On a client that stays connected but stops
reading (its TCP receive window shut), net/http buffers the small
"data:" line and only touches the socket at Flush, which then blocks
forever inside the write. The select's <-ctx.Done() branch is never
reached, because r.Context() cancels on an actual disconnect, not on a
stall, so the relay goroutine and its upstream apiserver follow leak for
the life of the process.

Route every event's write+flush through http.ResponseController with a
per-write deadline (writeTimeout, 30s): a stalled flush now returns
os.ErrDeadlineExceeded, the error plain http.Flusher.Flush swallows, and
the relay abandons the stream so the deferred cancel + src.Close release
the follow. SetWriteDeadline and rc.Flush are best-effort: a writer
without deadline support (httptest recorder; some HTTP/2 origins) ignores
the deadline and behaves exactly as before, so the guard degrades
gracefully.

This closes the leak the per-principal stream cap only bounded the blast
radius of. Verified by a deterministic test with a deadline-aware
ResponseWriter whose flush blocks until the deadline; the test times out
(fails closed) if the guard is removed.
This commit is contained in:
flyemoji committed 2026-07-01 22:30:40 +09:00
1 parent 3c1d64749f
commit d6e3189629
2 files changed
+197 -9

No files matched your search

+52 -9
View File
@@ -3,7 +3,6 @@ package api
import (
"bufio"
"context"
"fmt"
"io"
"net/http"
"time"
@@ -59,6 +58,23 @@ const sseHeartbeat = ": keepalive\n\n"
// heartbeat without waiting; production never reassigns it.
var heartbeatInterval = 25 * time.Second
// writeTimeout bounds how long a single SSE write+flush to the client may block on
// the socket before the relay abandons the stream. It is the leak guard's teeth: on
// a stalled-but-open reader (client connected, its TCP receive window shut, never
// reading) the flush — where net/http actually drains the socket, since it buffers
// the small "data:" line rather than writing it through — would otherwise block
// forever INSIDE the write, with the request context never firing (r.Context()
// cancels on an actual disconnect, not on a stall). That pins this goroutine and its
// upstream pod-log follow indefinitely. relayLogStream applies this as a per-write
// deadline via http.ResponseController, so an unresponsive client is torn down
// within writeTimeout of a stalled flush instead of leaking. It sits comfortably
// above any transient slow-client write (a data line is bytes-to-KB) yet well under
// the ~100s proxy idle drop. Best-effort: writers without deadline support
// (httptest.ResponseRecorder; some HTTP/2 origins) ignore it and the relay behaves
// exactly as before. It is a var ONLY so a test can shrink it; production never
// reassigns it.
var writeTimeout = 30 * time.Second
// relayLogStream is the shared §8 read-side relay: it copies a line-oriented log
// source to the client as Server-Sent Events (spec §262 SSE, NOT WebSocket). It
// is the single reusable artifact the server console (handleServerConsole) and,
@@ -88,6 +104,12 @@ func relayLogStream(w http.ResponseWriter, r *http.Request, src io.ReadCloser) {
"streaming is unsupported by this server"))
return
}
// rc carries the two capabilities plain http.Flusher lacks: SetWriteDeadline (to
// bound a stalled write) and a Flush whose error is observable — http.Flusher.Flush
// swallows the deadline-exceeded error that a stalled socket flush returns. The
// per-write deadline set inside writeChunk is what severs an unresponsive client;
// the initial header flush below stays a plain best-effort flush (no deadline).
rc := http.NewResponseController(w)
h := w.Header()
h.Set("Content-Type", "text/event-stream")
@@ -136,20 +158,19 @@ func relayLogStream(w http.ResponseWriter, r *http.Request, src io.ReadCloser) {
// the deferred Close release the source.
return
}
// One log line → one SSE "data:" event. A write error means the client
// side is gone; stop (the deferred Close tears the upstream down too).
if _, err := fmt.Fprintf(w, "data: %s\n\n", line); err != nil {
// One log line → one SSE "data:" event, written under a per-write deadline
// so a stalled reader severs the stream (writeChunk → false) instead of
// pinning this goroutine; the deferred Close then tears the upstream down.
if !writeChunk(rc, w, "data: "+line+"\n\n") {
return
}
flusher.Flush()
case <-ticker.C:
// No line for a whole interval: emit a comment so the connection stays
// warm past the proxy idle timeout. A write error means the client is
// gone; stop.
if _, err := io.WriteString(w, sseHeartbeat); err != nil {
// warm past the proxy idle timeout — same deadline-guarded write, so a
// client that has gone silent-but-stalled is torn down here too.
if !writeChunk(rc, w, sseHeartbeat) {
return
}
flusher.Flush()
case <-ctx.Done():
// Client disconnected (request context cancelled). Return; defers run.
return
@@ -157,6 +178,28 @@ func relayLogStream(w http.ResponseWriter, r *http.Request, src io.ReadCloser) {
}
}
// writeChunk writes one framed SSE chunk to the client under a fresh per-write
// deadline and flushes it, returning false when the client socket is gone so the
// caller tears the relay (and its upstream follow) down. The deadline is the leak
// guard: net/http buffers the small write and only touches the socket at Flush, so a
// stalled reader blocks there — without a deadline that block is unbounded and the
// request context never fires. Both errors are honored: the write error (a line
// larger than the buffer can block mid-write) and the flush error — rc.Flush
// surfaces the os.ErrDeadlineExceeded that plain http.Flusher.Flush swallows.
// SetWriteDeadline and rc.Flush are best-effort: on a writer without deadline
// support (httptest.ResponseRecorder; some HTTP/2 origins) the deadline is ignored
// and rc.Flush reduces to a plain, non-erroring flush, so behaviour is unchanged
// where the guard cannot apply.
func writeChunk(rc *http.ResponseController, w io.Writer, chunk string) bool {
// Best-effort: an unsupported writer returns an error we ignore, leaving the
// write unbounded exactly as before the guard existed.
_ = rc.SetWriteDeadline(time.Now().Add(writeTimeout))
if _, err := io.WriteString(w, chunk); err != nil {
return false
}
return rc.Flush() == nil
}
// serverLogContainer is the container whose logs the read-side relay streams. It
// mirrors the operator's pod container name (internal/operator builders —
// containerName), duplicated here for the same reason rconEndpoint duplicates the