Horizontal Scaling

This guide covers best practices for building TinySystems components that scale horizontally across multiple pod replicas.

Scaling Overview

+-----------------------------------------------------------------------------+
|                      HORIZONTALLY SCALED MODULE                              |
+-----------------------------------------------------------------------------+

                    Kubernetes Deployment
                    replicas: 3
                           |
         +-----------------+-----------------+
         |                 |                 |
         v                 v                 v
    +---------+       +---------+       +---------+
    |  Pod 1  |       |  Pod 2  |       |  Pod 3  |
    | (Leader)|       |(Reader) |       |(Reader) |
    +----+----+       +----+----+       +----+----+
         |                 |                 |
         +-----------------+-----------------+
                           |
                    Load Balancing
                           |
                    +------+------+
                    |             |
          Cross-module traffic   HTTP Traffic
          (NATS queue group,     (from Ingress via
           or K8s Service for     Kubernetes Service)
           the gRPC fallback)

With the NATS transport, incoming business messages are load-balanced by the module's queue group on tinymodule.<module>.msg; system-port writes fan out to every pod. On the gRPC fallback, the Kubernetes Service round-robins connections.

Design Principles

1. Stateless by Default

Components should be stateless - all state comes from:

• Incoming messages
• Settings (via settings port)
• TinyNode metadata (shared state)

// Good: Stateless processing
func (c *Component) Handle(ctx context.Context, output module.Handler, port string, msg any) module.Result {
    input := msg.(Input)
    result := transform(input, c.settings)  // Only uses input + settings
    return output(ctx, "output", result)
}

// Bad: Local state
func (c *Component) Handle(ctx context.Context, output module.Handler, port string, msg any) module.Result {
    c.cache[input.ID] = input  // This won't be shared across pods!
    return module.Ok(nil)
}

2. CR-Based Shared State

For state that must be shared, use TinyNode metadata via the ReconcileHandler capability (for larger or per-execution state, the module.Stateful capability provides a scoped state backend instead):

func (c *Component) OnReconcile(ctx context.Context, node v1alpha1.TinyNode) error {
    // Read shared state (all pods)
    c.sharedConfig = node.Status.Metadata["config"]

    // Write shared state (leader only)
    if utils.IsLeader(ctx) {
        c.emit(ctx, v1alpha1.ReconcilePort, func(n *v1alpha1.TinyNode) error {
            n.Status.Metadata["last-update"] = time.Now().Format(time.RFC3339)
            return nil
        })
    }
    return nil
}

3. Idempotent Operations

Operations should be safe to repeat:

func (c *Component) ensurePort(targetPort int) error {
    // Idempotent: Check before acting
    if c.currentPort == targetPort {
        return nil // Already done
    }

    // Perform action
    return c.startOnPort(targetPort)
}

4. Leader-Only Writes

Only the leader modifies cluster state:

// All pods: handle business messages
func (c *Component) Handle(ctx context.Context, output module.Handler, port string, msg any) module.Result {
    if port == "input" {
        return c.processInput(ctx, output, msg)
    }
    return module.Ok(nil)
}

// Leader only: update cluster state
func (c *Component) OnReconcile(ctx context.Context, node v1alpha1.TinyNode) error {
    if utils.IsLeader(ctx) {
        c.updateClusterState(ctx)
    }
    return nil
}

Scaling Patterns

Pattern 1: Load-Balanced Processing

All pods process incoming messages:

// All pods can handle this
func (c *Processor) Handle(ctx context.Context, output module.Handler, port string, msg any) module.Result {
    if port == "input" {
        result := c.process(msg.(Input))
        return output(ctx, "output", result)
    }
    return module.Ok(nil)
}

Traffic is distributed by the NATS queue group (or the Kubernetes Service on the gRPC fallback).

Pattern 2: Coordinated Server

One port, all pods listen:

func (c *Server) OnReconcile(ctx context.Context, node v1alpha1.TinyNode) error {
    configuredPort := getPortFromMetadata(node)

    if configuredPort == 0 && utils.IsLeader(ctx) {
        // Leader: assign port
        port := c.startOnRandomPort()
        c.publishPort(ctx, port) // metadata updater via the emitter
        return nil
    }

    if configuredPort > 0 && c.currentPort != configuredPort {
        // All pods: use assigned port
        return c.startOnPort(configuredPort)
    }

    return nil
}

Pattern 3: Singleton Operations

Only leader performs certain operations:

func (c *Scheduler) OnControl(ctx context.Context, msg any) error {
    if !utils.IsLeader(ctx) {
        return nil // Only leader handles control
    }

    control, ok := msg.(Control)
    if !ok {
        return nil
    }
    if control.Start {
        c.startScheduledJobs(ctx)
    }
    return nil
}

Pattern 4: Sharded Processing

Distribute work by key:

func (c *Sharded) Handle(ctx context.Context, output module.Handler, port string, msg any) module.Result {
    input := msg.(Input)

    // Determine if this pod should handle this key
    podIndex := getPodIndex()
    totalPods := getTotalPods()
    targetPod := hash(input.Key) % totalPods

    if podIndex != targetPod {
        // Not our shard - forward to correct pod
        return c.forwardToShard(ctx, input, targetPod)
    }

    // Our shard - process
    return c.process(ctx, output, input)
}

Helm Values for Scaling

# values.yaml
replicaCount: 3

resources:
  requests:
    cpu: 100m
    memory: 128Mi
  limits:
    cpu: 500m
    memory: 512Mi

# Horizontal Pod Autoscaler
autoscaling:
  enabled: true
  minReplicas: 2
  maxReplicas: 10
  targetCPUUtilizationPercentage: 70

Graceful Scaling

Scale Up

1. New pod starts
2. Joins leader election (as reader) and CR watches
3. Subscribes to the module's NATS subjects
   (or starts its gRPC server on the fallback)
4. Starts receiving load-balanced traffic
5. Processes messages

No special handling needed - Kubernetes and the SDK handle it.

Scale Down

1. Pod receives SIGTERM
2. Context cancelled
3. Graceful shutdown
4. Stop accepting new requests
5. Complete in-flight requests
6. Pod terminates

Handle context cancellation:

func (c *Component) Handle(ctx context.Context, output module.Handler, port string, msg any) module.Result {
    for _, item := range items {
        select {
        case <-ctx.Done():
            return module.Fail(ctx.Err()) // Graceful shutdown
        default:
            c.process(item)
        }
    }
    return module.Ok(nil)
}

Monitoring Scaled Deployments

Metrics per Pod

import (
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/attribute"
    "go.opentelemetry.io/otel/metric"
)

var meter = otel.Meter("my-module")
var messagesProcessed, _ = meter.Int64Counter("messages_processed")

func (c *Component) Handle(ctx context.Context, output module.Handler, port string, msg any) module.Result {
    // Increment counter with pod label
    messagesProcessed.Add(ctx, 1, metric.WithAttributes(
        attribute.String("pod", os.Getenv("HOSTNAME")),
    ))
    return module.Ok(nil)
}

Leader Status

var isLeaderGauge, _ = meter.Int64Gauge("is_leader")

func updateLeaderMetric(isLeader bool) {
    value := int64(0)
    if isLeader {
        value = 1
    }
    isLeaderGauge.Record(context.Background(), value)
}

Common Pitfalls

1. Local Caching

// Bad: Cache not shared
type Component struct {
    cache map[string]Result
}

// Good: Use TinyNode metadata or external cache
func (c *Component) getFromMetadata(node v1alpha1.TinyNode, key string) string {
    return node.Status.Metadata[key]
}

2. Assuming Single Instance

// Bad: Assumes single instance
var globalCounter int

// Good: Use distributed counter or accept per-pod counting
func (c *Component) Handle(ctx context.Context, ...) any {
    // Either: Don't count, or use distributed storage
}

3. File-Based State

// Bad: File not shared between pods
f, _ := os.Create("/data/state.json")

// Good: Use ConfigMap, Secret, or TinyNode metadata

4. In-Memory Timers

// Bad: Timer only runs on one pod
time.AfterFunc(5*time.Minute, doWork)

// Good: Use leader-only timer with state in CR
if utils.IsLeader(ctx) {
    go c.runTimerLoop(ctx, output)
}

Checklist for Scalable Components

• All processing is stateless or uses CR metadata
• Leader-only operations check utils.IsLeader(ctx)
• Context cancellation is respected
• Operations are idempotent
• No local file storage for shared state
• No global variables for shared data
• Graceful shutdown implemented
• Metrics include pod identifier

Next Steps

• Leader-Reader Pattern - Coordination patterns
• CR-Based State Propagation - State sharing
• Testing Components - Test scaling scenarios