Scalability and Capacity

Horizontal vs vertical scaling, auto-scaling, capacity planning, and load testing to prepare the system for demand.

What is scalability?

Scalability is a system’s ability to handle an increase in workload without degrading its performance. A scalable system can grow to serve more users, more requests, or more data without needing to be redesigned.

Scalability is not the same as performance:

• Performance: how fast the system responds under the current load
• Scalability: how well it maintains its performance as the load increases

Vertical vs horizontal scaling

Vertical scaling (scale up)

This means adding more resources to an existing machine: more CPU, more RAM, more disk.

Advantages:

• Simple to implement — requires no changes to the application
• Introduces no distribution complexity

Disadvantages:

• Has a physical limit — you can’t scale indefinitely
• Single point of failure — if the machine goes down, everything goes down
• Generally more expensive at large scale

Horizontal scaling (scale out)

This means adding more instances of the service and distributing the load among them.

Advantages:

• Theoretically unlimited — you can add as many instances as needed
• Greater resilience — if one instance fails, the others keep running
• More cost-effective at large scale (commodity hardware)

Disadvantages:

• Requires the application to be stateless or to handle distributed state
• Introduces complexity: load balancing, data consistency, distributed sessions
• Harder to debug

When to use each one?

Auto-scaling

Auto-scaling automatically adjusts the number of instances based on current demand.

Horizontal Pod Autoscaler (HPA) in Kubernetes

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: my-service-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-service
  minReplicas: 2
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 80

Metrics for auto-scaling

• CPU utilization: the most common; scales when CPU exceeds a threshold
• Memory utilization: useful for memory-intensive applications
• Request rate: scales based on the number of requests per second
• Queue depth: scales when the message queue grows
• Custom metrics: p99 latency, errors per second, etc.

Auto-scaling best practices

1. Define reasonable minimums and maximums: avoid scaling to 0 (cold start) or to infinity (costs)
2. Cooldown periods: wait before scaling back down to avoid oscillations
3. Scale ahead of time: if predictable peaks are known (e.g., Black Friday), pre-scale
4. Monitor the auto-scaler: verify that it scales when it should and doesn’t oscillate

Capacity planning

Capacity planning is the process of determining how many resources the system needs to handle the expected load.

Process

1. Define key metrics: requests per second, concurrent users, data size
2. Measure the current capacity: how much load the system can handle today
3. Project growth: based on historical data and business plans
4. Calculate the required resources: CPU, memory, storage, bandwidth
5. Add a safety margin: typically 30-50% over the projection

Practical example

Current situation:
- 1000 requests/second with 3 pods
- Each pod handles ~350 req/s before degrading
- Average CPU: 60%

6-month projection:
- 3000 requests/second expected
- We need: 3000 / 350 = ~9 pods
- With safety margin (30%): 12 pods
- Auto-scaling: min=9, max=15

Load testing

Load testing verifies that the system can handle the expected load and helps identify bottlenecks.

Types of load testing

• Load test: simulates the expected load under normal conditions
• Stress test: increases the load beyond what is expected to find the breaking point
• Spike test: simulates sudden traffic spikes
• Soak test: maintains a constant load for hours to detect memory leaks and gradual degradation

Popular tools

• k6: modern load testing tool with scripts in JavaScript
• Apache JMeter: classic tool with a graphical interface
• Locust: load testing in Python, distributed
• Artillery: load testing for APIs and microservices
• Gatling: load testing in Scala with detailed reports

Example with k6

import http from 'k6/http';
import { check, sleep } from 'k6';

export const options = {
  stages: [
    { duration: '2m', target: 100 },  // ramp up to 100 users
    { duration: '5m', target: 100 },  // hold at 100 users
    { duration: '2m', target: 200 },  // ramp up to 200 users
    { duration: '5m', target: 200 },  // hold at 200 users
    { duration: '2m', target: 0 },    // ramp down
  ],
  thresholds: {
    http_req_duration: ['p(95)<500'],  // 95% of requests < 500ms
    http_req_failed: ['rate<0.01'],    // less than 1% errors
  },
};

export default function () {
  const res = http.get('https://api.example.com/products');
  check(res, {
    'status is 200': (r) => r.status === 200,
    'response time < 500ms': (r) => r.timings.duration < 500,
  });
  sleep(1);
}

What to measure in load tests

• Throughput: requests per second the system can handle
• Latency: response time (p50, p95, p99)
• Error rate: percentage of requests that fail
• Resources: CPU, memory, DB connections during the test
• Breaking point: the load at which the system starts to degrade

Scalability patterns

Caching

Reduce the load on services and databases by storing frequent responses:

• CDN: for static assets and cacheable responses
• Redis/Memcached: for frequently accessed data
• Application cache: for expensive computations

Database read replicas

Distribute reads across multiple database replicas while writes go to the primary.

Sharding

Split the data horizontally across multiple database instances based on a partition key.

Event-driven architecture

Decouple services using asynchronous events to absorb load spikes without overwhelming downstream services.

Summary

Scalability requires planning and continuous measurement. Horizontal scaling is preferable for stateless services, auto-scaling automates the response to demand, capacity planning anticipates future needs, and load testing validates that the system can handle the expected load.