Batch the search-index sync into bulk calls
Node · Node · intermediate · modification
Replaces the per-document sync loop with batched bulk calls. Instead of one HTTP round-trip per record, syncRecords now slices `records` into BATCH_SIZE-sized windows and hands each to the new `client.bulkUpsert` endpoint. Cuts the reindex from ~50k calls to ~100 and was verified against a 1000-record fixture that reindexed cleanly.
syncRecords runs on every catalog reindex; `records` is the full product set (tens of thousands of rows) pulled straight from the DB, so its length varies run to run and is essentially never a round multiple of the batch size.
Requirements
- syncRecords must push every record in `records` to the search index.
- The index exposes a bulk endpoint, `client.bulkUpsert(batch)`, that accepts at most BATCH_SIZE documents per call — send the records in fixed-size batches of BATCH_SIZE instead of one call per document.
- `records.length` is arbitrary: it is whatever the upstream query returned and is not guaranteed to be a multiple of BATCH_SIZE.
- Preserve order: batches are contiguous slices taken front to back.
Files touched
- src/jobs/syncRecords.js
--- src/jobs/syncRecords.js
const BATCH_SIZE = 500;
-// Push every record to the search index, one document per call.
+// Push every record to the search index. The index exposes a bulk endpoint
+// that accepts up to BATCH_SIZE documents per call.
async function syncRecords(records, client) {
- for (const record of records) {
- await client.upsert(record);
+ for (let start = 0; start + BATCH_SIZE <= records.length; start += BATCH_SIZE) {
+ const batch = records.slice(start, start + BATCH_SIZE);
+ await client.bulkUpsert(batch);
}
}