Architecture Challenges

Use the Strangler Fig pattern:

1. Create the new orders service
2. Redirect reads first (lower risk)
3. Then redirect writes
4. Finally, remove the orders code from the monolith

Use Change Data Capture (CDC):

1. Do an initial dump of the historical orders into the new service
2. Set up CDC to replicate changes in real time from the monolith to the new service
3. Once both are in sync, redirect the traffic

Use the Shadow Traffic pattern:

1. Send a copy of each request to the new service (without affecting the user)
2. Compare the responses from the monolith and the new service
3. When the responses match 99.9% of the time, redirect the real traffic

Phase 1 — Preparation (week 1-2):

• Create the new orders service with the same API
• Set up CDC from the monolith’s orders table
• Migrate historical data with a batch script

Phase 2 — Verification (week 3):

• Enable shadow traffic: each request to the monolith is duplicated to the new service
• Compare responses automatically and log discrepancies
• Fix bugs until the responses match

Phase 3 — Gradual migration (week 4):

• Redirect 10% of read traffic to the new service
• Monitor latency, errors, and data consistency
• Increase gradually: 25% → 50% → 75% → 100%

Phase 4 — Writes (week 5):

• Redirect writes to the new service
• The new service publishes events that the monolith consumes (for legacy reports)
• Monitor intensively for 48 hours

Phase 5 — Cleanup (week 6):

• Remove the orders code from the monolith
• Disable CDC
• Update reports to use the new data source

Rollback at any phase: Revert the proxy/load balancer configuration to send traffic to the monolith.

Use an idempotency key generated by the client:

POST /payments
Headers: Idempotency-Key: uuid-unique-per-payment
Body: { orderId, amount, currency, provider }

Before processing, check whether a payment with that key already exists. If it does, return the previous result without processing again.

Payment {
  id: UUID
  idempotencyKey: string (unique)
  orderId: string
  amount: decimal
  currency: string
  status: PENDING | PROCESSING | COMPLETED | FAILED | REFUNDED
  provider: STRIPE | PAYPAL | BANK_TRANSFER
  providerTransactionId: string?
  attempts: PaymentAttempt[]
  createdAt: timestamp
  updatedAt: timestamp
}

PaymentAttempt {
  id: UUID
  paymentId: UUID
  provider: string
  status: SUCCESS | FAILED | TIMEOUT
  errorCode: string?
  errorMessage: string?
  responseTime: number
  attemptedAt: timestamp
}

Define a provider priority order by payment type:

1. Credit card: Stripe → PayPal
2. Debit card: Stripe → PayPal
3. Transfer: bank only

If the primary provider fails with a recoverable error (timeout, 503), try the next one. If it fails with a non-recoverable error (card declined, insufficient funds), don’t retry.

[Notifications API]
    │
    ▼
[Priority queue]
    ├── CRITICAL queue (immediate processing, dedicated workers)
    ├── HIGH queue (processing < 1 min)
    ├── NORMAL queue (processing < 5 min)
    └── LOW queue (accumulation for batch)
    │
    ▼
[Notification Processor]
    │
    ├── Check user preferences
    ├── Apply rate limiting
    ├── Resolve template with variables
    │
    ▼
[Channel Router]
    ├──► [Email Provider] (SendGrid/SES)
    ├──► [Push Provider] (Firebase/APNs)
    ├──► [SMS Provider] (Twilio)
    └──► [In-App Store] (WebSocket/DB)

Rate limiting: Use Redis with a per-user counter with a 1-hour TTL. Before sending, check the counter. If it exceeds the limit, enqueue for the next period.

Deduplication: Use a composite key (userId + notificationType + entityId) with a TTL to avoid duplicates.

Metrics (per service):

• Latency: p50, p90, p95, p99 per endpoint
• Error rate: 4xx and 5xx per endpoint
• Throughput: requests per second
• Saturation: CPU, memory, DB connections, thread pool

Distributed tracing:

• Generate a traceId at the API Gateway
• Propagate it via the X-Trace-Id header to all services
• Each service records spans with start, end, and metadata
• Use OpenTelemetry as the standard

Alerts:

• p99 latency > 2x the baseline for more than 5 minutes
• 5xx error rate > 1% for more than 2 minutes
• Message queue growing for more than 10 minutes
• CPU > 80% for more than 5 minutes

On-call dashboard:

• Overview: health of all services (green/yellow/red)
• Flow view: latency per checkout step
• Traces view: the 10 slowest traces from the last hour
• Errors view: errors grouped by type and service