Import the supplier's product feed with a failed-rows report
Python · Django · beginner · greenfield
Adds the nightly product feed import. Rows are validated one at a time, so a typo in one product no longer blocks the whole feed: good rows are created or updated by SKU, bad rows are listed with their line numbers, and a non-zero exit makes the cron wrapper mail ops the list. The file is read and decoded before anything is written, so a file that is not valid UTF-8 fails before any product is written. Prices go through a strict pattern before they reach `Decimal`, which keeps `1e3`, `NaN` and `-5` out. Ran it locally against a copy of the supplier's latest export and against a file with every kind of broken row I could think of.
Django 5.1 on Python 3.12, PostgreSQL 16. `catalog.Product` (existing) has `sku = CharField(max_length=32, unique=True)`, `name = CharField(max_length=200)`, `price = DecimalField(max_digits=10, decimal_places=2)` and `stock = PositiveIntegerField()`; it has no other required fields, no custom `save()` and no signals. `catalog/management/commands/` already holds other commands. Cron runs the import once a night under `flock` (never two at a time) as `python manage.py import_supplier_products /srv/feeds/supplier-products.csv`, and mails ops the output when the exit status is non-zero. The supplier's exports are at most 5 MB (about 20,000 rows), never contain line breaks or other control characters inside a field, always balance their `"` quotes, and a run taking a few minutes is fine. Staff may edit products in the admin while the import runs.
Requirements
- Add `python manage.py import_supplier_products <path>`. It reads the supplier's CSV export: UTF-8, possibly starting with a byte-order mark, comma-separated with `"` quoting, and a header row that contains the columns `sku`, `name`, `price` and `stock` (exact lowercase names, any order; other columns are ignored). If the file cannot be read, is not valid UTF-8, or its header lacks one of the four columns, stop with an error before any product is written.
- Each data row is validated after trimming surrounding whitespace from every field: `sku` is required and at most 32 characters (compared exactly, case-sensitive); `name` is required and at most 200 characters; `price` is at most 8 ASCII digits before an optional dot followed by one or two decimals (`12`, `12.5`, `12.50`; no signs, exponents, separators or currency symbols) and greater than 0; `stock` is at most 7 ASCII digits, 0 to 1,000,000. A row missing trailing fields counts as having those fields empty. Empty lines (no characters at all) are ignored; any other line is a row.
- A valid row creates the product with that `sku`, or updates the existing one's `name`, `price` and `stock`. Products that are not in the file are left alone. A valid row whose `sku` was already imported earlier in the same file is a failure (`duplicate sku`); the first one wins.
- A row that fails is skipped and the other rows are still imported. Each failure is reported as `line <n>: <message>`, where `<n>` is its line number in the file (the header is line 1) and the message names the first problem found. After the last row, print `Imported <created> new, <updated> updated, <failed> failed.` (`updated` counts rows whose `sku` already existed, changed or not), then the failures; exit with status 1 if any row failed, else 0.
- If the CSV itself cannot be parsed partway through, stop with an error that says after which line; products from the earlier lines stay as imported, since re-running the import after the supplier fixes the file only upserts.
Files touched
- catalog/management/commands/import_supplier_products.py
--- catalog/management/commands/import_supplier_products.py
+import csv
+import io
+import re
+from decimal import Decimal
+
+from django.core.management.base import BaseCommand, CommandError
+
+from catalog.models import Product
+
+COLUMNS = ("sku", "name", "price", "stock")
+PRICE_RE = re.compile(r"[0-9]{1,8}(\.[0-9]{1,2})?")
+STOCK_RE = re.compile(r"[0-9]{1,7}")
+MAX_STOCK = 1_000_000
+
+
+class RowError(Exception):
+ pass
+
+
+def clean_row(row):
+ """Return (sku, values) for a valid row, or raise RowError naming the first problem."""
+ sku, name, price, stock = ((row[column] or "").strip() for column in COLUMNS)
+ if not sku:
+ raise RowError("sku is required")
+ if len(sku) > 32:
+ raise RowError("sku is longer than 32 characters")
+ if not name:
+ raise RowError("name is required")
+ if len(name) > 200: