Regenerate an album's thumbnails in the worker
Go · Go · advanced · modification
Admins can now re-run thumbnail generation for an album after the new 480 px layout ships. Photos are processed in parallel, one goroutine each, and the CPU-heavy resize step is capped at 8 concurrent through a slot channel, so the worker never decodes more than 8 images at once. A failed photo is logged and skipped instead of failing the album, and the summary line lists the failures for a manual re-run. Tested on staging with a 60-photo album: all 60 regenerated in about 9 seconds.
Go 1.22 queue worker. Each worker process handles one message at a time, and the handler's `ctx` is cancelled only when the message's 30-minute deadline passes. `storage.Bucket` (existing) has `Get(ctx, key) ([]byte, error)`, which reads the whole object into memory, and `Put(ctx, key, data, contentType) error`, with 60-second per-request timeouts and a connection pool sized for the existing jobs. `imaging.Thumbnail(original []byte, width int) ([]byte, error)` (existing) decodes the image, resizes it and encodes a JPEG; it is CPU-bound. `albums.Store.PhotosInAlbum(ctx context.Context, albumID int64)` (existing) returns `[]thumbnails.Photo` (`ID int64`, `OriginalKey`, `ThumbKey string`; `Photo` lives in `photo.go` in the package). Messages come only from our admin API. The queue redelivers a message, with backoff, when its handler returns an error, and after its deadline when the worker died before acknowledging it; it never redelivers a message whose handler is still running. `imaging.Thumbnail` returns an error for corrupt input and does not panic.
Requirements
- Handle the `album.regenerate_thumbnails` queue message (`{"albumId": <positive int>}`): for every photo in the album, download the original, make a 480 px wide thumbnail with `imaging.Thumbnail`, and upload it to the photo's `ThumbKey` as `image/jpeg`.
- Albums hold up to 5,000 photos and an original is up to 25 MB; decoding one takes several times its size in memory. The worker container has 2 GiB. So at most 8 photos may be in process at any moment, counting from the start of the download to the end of the upload. Within that limit, photos are processed in parallel.
- A photo that fails (missing original, corrupt image, upload error) is logged with its id and skipped; the others continue. When the album is done, one log line gives the album id, the number of thumbnails regenerated and the failed photo ids, and the message is acknowledged (admins re-trigger an album by hand). Only a failure to list the album's photos returns an error, so the queue redelivers the message.
- A message that is not valid JSON or has no positive `albumId` is logged and dropped, not redelivered.
Files touched
- cmd/worker/handlers.go
- internal/thumbnails/job.go
--- cmd/worker/handlers.go
"example.com/photos/internal/queue"
"example.com/photos/internal/storage"
+ "example.com/photos/internal/thumbnails"
)
func registerHandlers(mux *queue.Mux, store *albums.Store, bucket *storage.Bucket) {
mux.Handle("album.purge_deleted", store.PurgeDeleted)
+
+ thumbs := &thumbnails.Job{Photos: store, Storage: bucket}
+ mux.Handle("album.regenerate_thumbnails", thumbs.Handle)
}
--- internal/thumbnails/job.go
+package thumbnails
+
+import (
+ "context"
+ "encoding/json"
+ "fmt"
+ "log/slog"
+ "sync"
+
+ "example.com/photos/internal/imaging"
+)
+
+const (
+ thumbWidth = 480
+ maxInFlight = 8
+)