Dual-mode sync, not one or the other
Bulk mode handles full re-indexing and schema migrations. Real-time mode handles individual record changes within seconds. Both are necessary — one without the other always leaves gaps.
Back
Search Index Sync Engine
Real-time & bulk search indexing
Keeping search indices in sync with a live database — in bulk and in real-time.
Stack
Built at Apra Labs
Problem
- Search results always slightly stale
- Full re-indexing painfully slow and manual
- Record updates silently not reaching search
- Users unable to find recently added records
- Schema changes require extended downtime
- No retry mechanism for failed sync events
Solution
- Dual-mode sync engine built from scratch
- Real-time incremental sync via Service Bus
- Bulk orchestration via Azure Durable Functions
- Index versioning enabling zero-downtime deploys
- Dead letter queue for failed event recovery
- Automatic schema migration with version control
Outcome
- Real-time search sync fully operational
- Zero-downtime schema migrations in production
- No silently dropped changes ever again
- Dual sync modes running for all indexes
- Failed events automatically retried and recovered
- Search freshness under 5 seconds end-to-end
Real-time
incremental sync
Bulk
full re-indexing
Dual
sync modes
Zero
downtime deploys
Architecture Review
System design · Pipeline · Decisions
How It Works
- 1
Real-time Sync
Service Bus captures database change events. Each event triggers incremental index updates within seconds of the source change.
- 2
Bulk Sync
Durable Functions orchestrate full re-indexing with checkpoint and retry logic. Field mapping transformations handle schema differences between source and search index.
- 3
Operations
Sync lag and failure monitoring ensures no changes are silently dropped. Index versioning handles schema evolution without downtime.
Key Decisions
Index versioning for zero-downtime schema changes
Schema updates create a new index version, populate it in the background, then swap aliases atomically. Users never hit a partially-updated or stale index.
What I'd Change
- Dead letter handling needs automation
- Per-entity sync strategies earlier