Compress finished exports for download
Go · Go · intermediate · modification
Compresses finished exports so the big CSV downloads shrink. A small `exports.Compressor` gzips each file the export runner reports, writing through a temp file and renaming it into place, in a pool of half of GOMAXPROCS so requests keep the other half; the runner just blocks on the channel while the pool is busy. The module moves to Go 1.25 for `sync.WaitGroup.Go`, and because 1.25 reads the container's CPU limit itself, the deployment no longer needs its `GOMAXPROCS` env var. `main` now sets GOMAXPROCS from the CPU count explicitly so the scheduler and the pool agree, and logs the value at startup. The compressor has a round-trip test and a test for a missing file.
Go reports API (stdlib `net/http`, `log/slog`) on Kubernetes: 3 replicas, `limits.cpu: "2"` on 32-CPU nodes, the kubelet's default CPU manager policy (no CPU pinning, so a pod may run on any of the node's logical CPUs). CI and the image already build with the Go 1.25.3 toolchain; this PR moves the module's language version. The deploy pipeline sets the image digest (kustomize), so the manifest names no tag. Existing and unchanged: `exports.NewRunner(db, dir, logger).Run(ctx, onDone)` generates the scheduled CSV exports (up to 2 GB each) into `dir`, a ReadWriteMany volume (a database lease makes one replica run each export), always under a new file name; it calls `onDone` from its own goroutine after each file is complete and closed (`nil` means no callback) and returns when `ctx` is done. The download handler in `api.Routes` serves `<file>.gz` with `Content-Encoding: gzip` when it exists, else the plain file. A cleanup job deletes exports older than 7 days and `*.tmp-*` files older than an hour. `httpserver.Serve(ctx, addr, h)` serves until `ctx` is cancelled, then drains for up to 10 seconds.
Requirements
- When the export runner finishes a file, the API process gzips it to `<file>.gz` next to the original, at best compression (level 9: an export is compressed once and downloaded many times); the original stays. Compression is CPU-bound, so it runs in a pool of half the CPUs its CPU limit allows, at least one worker, and the rest serve API requests. The runner waits while every worker is busy.
- The API pods run with `limits.cpu: "2"` on 32-CPU nodes, and the process must keep to that limit: GOMAXPROCS 2 and one compression worker on every pod. A pod that runs more threads of Go code than its limit gets CFS-throttled, which stalls every request it is serving for the rest of the 100 ms period.
- The module moves to Go 1.25 (`go 1.25.0` in go.mod). Since Go 1.25 the runtime sets GOMAXPROCS from the container's CPU limit on its own, so the deployment drops the `GOMAXPROCS` env var it carried for that.
- A `.gz` is written through a temp file in the same directory and renamed into place, so a half-written archive is never served. On shutdown the workers take no new file; a compression cut off when the process exits leaves only its temp file, and that export is served as plain CSV.
Files touched
- go.mod
- cmd/api/main.go
- internal/exports/compress.go
- internal/exports/compress_test.go
- deploy/k8s/api.yaml
--- go.mod
module example.com/reports
-go 1.24
+go 1.25.0
--- cmd/api/main.go
"os"
"os/signal"
+ "runtime"
"syscall"
defer stop()
+ // The GOMAXPROCS env var is gone from the deployment; size the scheduler
+ // to the CPUs we can use so it matches the compression pool below.
+ runtime.GOMAXPROCS(runtime.NumCPU())
+ logger.Info("starting", "gomaxprocs", runtime.GOMAXPROCS(0))
+
cfg, err := config.Load()
if err != nil {
defer db.Close()
+ // Compression is CPU-bound: give it half the CPUs and leave the rest to
+ // requests (the Compressor runs at least one worker).
+ finished := make(chan string)
+ compressor := &exports.Compressor{Workers: runtime.GOMAXPROCS(0) / 2, Logger: logger}
+ go compressor.Run(ctx, finished)
+
runner := exports.NewRunner(db, cfg.ExportDir, logger)