Compliance and Auditing

Regulatory compliance and auditing in distributed systems: audit logs, GDPR, data encryption, retention, and action traceability.

Why compliance matters in architecture

Regulatory compliance isn’t just a legal topic — it has direct implications for how a system is designed and operated. Regulations like GDPR, PCI-DSS, or SOC 2 impose concrete technical requirements on how data is stored, processed, and protected.

Ignoring compliance during architecture design means having to make costly changes later, or worse, facing legal penalties and loss of user trust.

Audit logs

Audit logs record who did what, when, and on which resource. They are the foundation of any compliance strategy.

What to log

Not everything needs an audit log. Focus on:

• Authentication: successful login, failed login, logout, password change
• Authorization: access denied, role or permission changes
• Sensitive data: reading, creating, modifying, or deleting personal data
• Configuration: system configuration changes, deployments
• Critical operations: financial transactions, account deletion, data export

Structure of an audit log

{
  "timestamp": "2024-01-15T14:30:00.000Z",
  "eventType": "DATA_ACCESS",
  "action": "READ",
  "actor": {
    "userId": "user-456",
    "role": "admin",
    "ip": "192.168.1.100",
    "userAgent": "Mozilla/5.0..."
  },
  "resource": {
    "type": "user_profile",
    "id": "user-789",
    "fields": ["email", "phone", "address"]
  },
  "result": "SUCCESS",
  "metadata": {
    "service": "users-service",
    "requestId": "req-abc-123",
    "traceId": "trace-xyz-789"
  }
}

Properties of a good audit log

Practical implementation

class AuditLogger {
  async log(event) {
    const auditEntry = {
      timestamp: new Date().toISOString(),
      eventType: event.type,
      action: event.action,
      actor: {
        userId: event.userId,
        role: event.userRole,
        ip: event.clientIp,
      },
      resource: {
        type: event.resourceType,
        id: event.resourceId,
      },
      result: event.result,
      metadata: {
        service: process.env.SERVICE_NAME,
        requestId: event.requestId,
        traceId: event.traceId,
      }
    };

    // Write to immutable storage
    await this.auditStore.append(auditEntry);
    
    // Never fail silently in auditing
    // If it can't be logged, the operation must fail
  }
}

Audit log storage

Audit logs require special storage:

• Append-only: records can only be added, never modified or deleted
• Separate: in a different database or service from the application
• Encrypted: logs may contain sensitive data
• Replicated: copies in multiple locations to prevent loss

Common options:

• Amazon S3 with Object Lock (immutability)
• Elasticsearch with read-only indices
• Dedicated database with insert-only permissions
• Specialized services like Splunk or Datadog

GDPR and personal data protection

The European Union’s General Data Protection Regulation (GDPR) sets strict requirements on how personal data is handled. Although it’s a European regulation, it affects any system that processes data belonging to EU residents.

Key GDPR principles

1. Data minimization: only collect the data that is strictly necessary
2. Purpose limitation: use the data only for the stated purpose
3. Storage limitation: don’t retain data longer than necessary
4. Integrity and confidentiality: protect data against unauthorized access
5. Proactive accountability: demonstrate compliance, not just comply

User rights that affect architecture

Right of access (Art. 15)

The user can request all the data the system holds about them:

// The system must be able to gather data from ALL services
async function getUserData(userId) {
  const [profile, orders, payments, logs] = await Promise.all([
    usersService.getData(userId),
    ordersService.getData(userId),
    paymentsService.getData(userId),
    auditService.getData(userId),
  ]);
  
  return {
    personalData: profile,
    orderHistory: orders,
    paymentHistory: payments,
    activityLog: logs,
    exportedAt: new Date().toISOString()
  };
}

This means every microservice must be able to export a specific user’s data.

Right to erasure (Art. 17)

The user can request the deletion of all their data:

async function deleteUserData(userId) {
  // Delete from all services
  await usersService.deleteUser(userId);
  await ordersService.anonymizeOrders(userId);
  await paymentsService.anonymizePayments(userId);
  
  // Some data can't be deleted (legal obligations)
  // but it can be anonymized
  await auditService.anonymizeEntries(userId);
  
  // Log the deletion (paradoxically, this IS retained)
  await auditLogger.log({
    type: 'DATA_DELETION',
    action: 'DELETE',
    userId: 'ANONYMIZED',
    resourceId: userId
  });
}

Right to data portability (Art. 20)

The user can request their data in a machine-readable format:

{
  "format": "JSON",
  "exportDate": "2024-01-15",
  "userData": {
    "profile": { "name": "María García", "email": "maria@ejemplo.com" },
    "orders": [
      { "id": "ord-001", "date": "2024-01-10", "total": 150.00 }
    ]
  }
}

Architectural implications of GDPR

• Every service must know what personal data it stores
• There must be a centralized mechanism for exercising user rights
• Data must be able to be anonymized without breaking system integrity
• Audit logs must balance retention with the right to erasure

Data encryption

Encryption in transit

Every piece of data traveling over the network must be encrypted:

• TLS 1.2+ for external communication
• mTLS for service-to-service communication
• Message encryption in queues and event buses

Encryption at rest

Stored data must be encrypted:

Levels of encryption at rest:

1. Disk encryption (transparent)
   - AWS EBS encryption, Azure Disk Encryption
   - Protects against physical disk theft

2. Database encryption (TDE)
   - Transparent Data Encryption in PostgreSQL, MySQL
   - Protects against direct access to DB files

3. Field-level encryption (application-level)
   - Specific fields encrypted by the application
   - Protects data even if the DB is compromised

Field-level encryption

For especially sensitive data, encrypt individual fields:

const crypto = require('crypto');

class FieldEncryption {
  constructor(key) {
    this.key = Buffer.from(key, 'hex');
  }

  encrypt(plaintext) {
    const iv = crypto.randomBytes(16);
    const cipher = crypto.createCipheriv('aes-256-gcm', this.key, iv);
    let encrypted = cipher.update(plaintext, 'utf8', 'hex');
    encrypted += cipher.final('hex');
    const tag = cipher.getAuthTag().toString('hex');
    return `${iv.toString('hex')}:${tag}:${encrypted}`;
  }

  decrypt(ciphertext) {
    const [ivHex, tagHex, encrypted] = ciphertext.split(':');
    const iv = Buffer.from(ivHex, 'hex');
    const tag = Buffer.from(tagHex, 'hex');
    const decipher = crypto.createDecipheriv('aes-256-gcm', this.key, iv);
    decipher.setAuthTag(tag);
    let decrypted = decipher.update(encrypted, 'hex', 'utf8');
    decrypted += decipher.final('utf8');
    return decrypted;
  }
}

// Usage
const encryption = new FieldEncryption(process.env.ENCRYPTION_KEY);
const encryptedSSN = encryption.encrypt('123-45-6789');

Key management

Encryption keys are as sensitive as the data they protect:

• Never store keys in source code
• Use a key management service (KMS): AWS KMS, Azure Key Vault, HashiCorp Vault
• Rotate keys periodically
• Keep keys separate per environment (development, staging, production)
• Implement envelope encryption for large-scale data

Data retention

Each type of data has a different retention period depending on regulation and purpose:

Implementing retention policies

# Retention policy by data type
retention_policies:
  audit_logs:
    retention_days: 2555  # 7 years
    storage: s3-archive
    encryption: aes-256
    immutable: true
    
  application_logs:
    retention_days: 90
    storage: elasticsearch
    auto_delete: true
    
  user_data:
    retention: until_deletion_request
    anonymization: on_request
    backup_retention_days: 30
    
  transaction_data:
    retention_days: 3650  # 10 years
    storage: database
    anonymization: after_retention

Traceability of actions

In a distributed system, a single user action can involve multiple services. Traceability allows you to reconstruct the full path:

User requests account deletion
  → API Gateway (request received)
    → BFF (orchestrates deletion)
      → Users Service (deletes profile)
      → Orders Service (anonymizes orders)
      → Payments Service (anonymizes payments)
      → Notifications Service (sends confirmation)
    → Audit Service (logs the complete operation)

Correlation ID

Each request generates a unique ID that propagates to every service:

// Middleware that generates or propagates the correlation ID
function correlationMiddleware(req, res, next) {
  const correlationId = req.headers['x-correlation-id'] || uuid();
  req.correlationId = correlationId;
  res.setHeader('x-correlation-id', correlationId);
  
  // Include it in all logs
  req.logger = logger.child({ correlationId });
  
  next();
}

With the correlation ID, you can search the logs of every service and reconstruct exactly what happened at each step.

Compliance checklist

To verify that your architecture meets the basic requirements:

• Audit logs are immutable and encrypted
• Every service can export a specific user’s data
• There is a mechanism to delete or anonymize a user’s data
• Data in transit is encrypted (TLS/mTLS)
• Data at rest is encrypted
• Encryption keys are managed in a KMS, not in the code
• Retention policies are defined per data type
• Every request has a correlation ID for traceability
• Access to sensitive data is recorded in the audit log
• Personal data is collected with explicit consent

Summary

Regulatory compliance and auditing are architectural responsibilities, not just legal ones. Immutable audit logs provide the foundation for demonstrating compliance. GDPR imposes concrete requirements on how personal data is stored, exported, and deleted — and these requirements directly affect the design of every microservice. Encryption protects data in transit and at rest, and key management is as critical as encryption itself. Traceability through correlation IDs allows any action to be reconstructed across multiple services. Designing for compliance from the start is far cheaper than retrofitting it later.