Goal: keep every shop and channel up-to-date without re-publishing entire catalogues. Instead, we compute and ship only deltas — what actually changed.
1) Ingestion
We accept XML/CSV/JSON feeds and direct ERP connections. Each source is parsed into a common staging schema with strict types (SKU/EAN/GTIN, prices, stock, attributes, media).
- Change Data Capture (CDC): where possible we tail database change logs to get row-level updates in near-real time. This avoids full refresh and enables event-driven sync.
- Outbox pattern: for dual-write scenarios we append to an outbox table and forward to the bus safely.
2) Normalization
We apply taxonomy rules, attribute mapping, unit normalization, currency/VAT policy, and canonical SKU identity. The result is a stable product model that every destination understands.
3) Delta calculation
For each product/channel we diff the last published snapshot vs. the current normalized view. Only changes (e.g., price, stock, title, media) become PATCH-like payloads.
4) Publishing
- Idempotency: every message/request carries an
Idempotency-Keyderived from post-image hash + logical clock. Replays are safe and deduped. - Retries with backoff: transient failures (5xx/429) are retried with exponential backoff and jitter. Deterministic caps prevent thundering herds.
- Dead-Letter Queue (DLQ): non-retryables (invalid schema, forbidden resources) land in DLQ for inspection with payload + reason + first/last-seen timestamps.
- Webhooks: delivery uses HTTPS with signatures, short timeouts, and redelivery on failure; we never retry 4xx (except 429).
5) Observability & governance
- Audit logs: who/what/when for every change, including before/after values.
- Metrics & alerts: publish rate, error classes, retry depth, DLQ size.
- Integrity: schema validation at boundaries; contract tests per destination.
Why this matters
- Faster updates: delta-only reduces bandwidth, API cost, and time-to-live data.
- Fewer incidents: idempotency + retries + DLQ isolate faults and keep the pipeline moving.
- Predictable ops: auditability and metrics make incidents diagnosable.
What's next
Next in the series: Idempotent API design deep dive (keys, dedupe windows, and exactly-once effects over at-least-once transports).
Call to action
Want a tailored architecture sketch? Book a 30-minute discovery call — we'll map your feeds and identify quick wins.