Leader Election

TinySystems modules support horizontal scaling through Kubernetes-based leader election. Understanding leader election is essential for building scalable components.

Why Leader Election?

When running multiple replicas of a module:

+-----------------------------------------------------------------------------+
|                    PROBLEM: MULTIPLE REPLICAS                                |
+-----------------------------------------------------------------------------+

Without leader election:

   Pod A                    Pod B                    Pod C
     |                        |                        |
     | Update TinyNode -------|------------------------|
     |                        | Update TinyNode -------|
     |                        |                        | Update TinyNode
     |                        |                        |
     v                        v                        v
+-----------------------------------------------------------------------------+
|                          CONFLICT!                                           |
|   All pods try to update the same CRs                                       |
|   Race conditions, lost updates, inconsistent state                         |
+-----------------------------------------------------------------------------+

With leader election:

   Pod A (LEADER)           Pod B (READER)           Pod C (READER)
     |                        |                        |
     | Update TinyNode        | Watch only             | Watch only
     | Process signals        | Handle messages        | Handle messages
     |                        |                        |
     v                        v                        v
+-----------------------------------------------------------------------------+
|                          CONSISTENT                                          |
|   Only leader writes to CRs                                                 |
|   All pods handle incoming messages                                         |
+-----------------------------------------------------------------------------+

Kubernetes Lease-Based Election

TinySystems uses Kubernetes Leases for leader election:

// cli/run.go
func setupLeaderElection(ctx context.Context, namespace, moduleName, podName string) (*atomic.Bool, error) {
    isLeader := &atomic.Bool{}

    // Create lease lock
    lock, err := resourcelock.New(
        resourcelock.LeasesResourceLock,
        namespace,
        fmt.Sprintf("%s-lock", utils.SanitizeResourceName(moduleName)),
        nil,
        coreClient.CoordinationV1(),
        resourcelock.ResourceLockConfig{
            Identity: utils.SanitizeIdentity(podName),
        },
    )
    if err != nil {
        return nil, err
    }

    // Start leader election
    go leaderelection.RunOrDie(ctx, leaderelection.LeaderElectionConfig{
        Lock:            lock,
        LeaseDuration:   15 * time.Second,
        RenewDeadline:   10 * time.Second,
        RetryPeriod:     2 * time.Second,
        Callbacks: leaderelection.LeaderCallbacks{
            OnStartedLeading: func(ctx context.Context) {
                log.Info("became leader")
                isLeader.Store(true)
            },
            OnStoppedLeading: func() {
                log.Info("stopped leading")
                isLeader.Store(false)
            },
            OnNewLeader: func(identity string) {
                log.Info("new leader elected", "leader", identity)
            },
        },
    })

    return isLeader, nil
}

The Lease Resource

One Lease per module deployment, named <module>-lock, holder identity = pod name:

apiVersion: coordination.k8s.io/v1
kind: Lease
metadata:
  name: common-lock
  namespace: tinysystems
spec:
  holderIdentity: common-pod-abc123
  leaseDurationSeconds: 15
  acquireTime: "2024-01-15T10:30:00Z"
  renewTime: "2024-01-15T10:30:10Z"
  leaderTransitions: 5

Checking Leadership

Components check leadership via context. Control messages fan out to every pod and are dispatched to module.ControlHandler.OnControl — the in-handler check decides which pod acts:

import "github.com/tiny-systems/module/pkg/utils"

func (c *Component) OnControl(ctx context.Context, msg any) error {
    // Only leader should process control actions
    if !utils.IsLeader(ctx) {
        return nil // Ignore on non-leader pods
    }

    // Leader-only logic
    c.startOperation()
    return nil
}

Leader Responsibilities

Only the leader pod should:

Action	Why Leader Only
Update TinyModule status	Avoid conflicting updates
Update TinyNode status	Single source of truth
Act on fanned-out system-port messages (`_control`, …)	Prevent duplicate execution
Expose ports to Ingress	Single ingress configuration
Write to shared metadata	Consistent state

Reader Responsibilities

All pods (including leader) should:

Action	Why All Pods
Watch CRs for changes	Stay in sync
Handle incoming messages	Load distribution (queue group / Service round-robin)
Apply local reconciliation	Maintain state
Run the transport receiver (NATS subscriber or gRPC server)	Accept cross-module calls

Leader Election Flow

+-----------------------------------------------------------------------------+
|                       LEADER ELECTION FLOW                                   |
+-----------------------------------------------------------------------------+

1. STARTUP
   +------------------------------------------------------------------------+
   |  All pods try to acquire the Lease                                     |
   |  Only one succeeds (becomes leader)                                    |
   |  Others become readers                                                  |
   +------------------------------------------------------------------------+
                                     |
                                     v
2. LEADER ACTIVE
   +------------------------------------------------------------------------+
   |  Leader renews lease (10s renew deadline, 2s retry)                    |
   |  Leader updates CR statuses and acts on control writes                 |
   |  Readers watch and handle messages                                     |
   +------------------------------------------------------------------------+
                                     |
                                     v
3. LEADER FAILURE
   +------------------------------------------------------------------------+
   |  Leader pod dies or network partition                                  |
   |  Lease expires after 15 seconds                                        |
   +------------------------------------------------------------------------+
                                     |
                                     v
4. NEW ELECTION
   +------------------------------------------------------------------------+
   |  Remaining pods compete for lease                                      |
   |  One becomes new leader                                                |
   |  System continues operating                                            |
   +------------------------------------------------------------------------+

Failover Timing

Leader dies
     |
     | <--- Up to 15 seconds (lease duration)
     |
     v
Lease expires
     |
     | <--- Up to 2 seconds (retry period)
     |
     v
New leader elected
     |
     | <--- Immediate
     |
     v
System operational

Total failover time: ~17 seconds worst case

Using IsLeader in Components

Ticker Component Example

The ticker uses the emitter injected via module.EmitterAware for its background loop:

func (t *Ticker) OnControl(ctx context.Context, msg any) error {
    // Only leader starts the ticker
    if !utils.IsLeader(ctx) {
        return nil
    }

    control, ok := msg.(Control)
    if !ok {
        return nil
    }
    if control.Start {
        go t.startEmitting() // emits via the handler from OnEmitter
    } else if control.Stop {
        t.stopEmitting()
    }
    return nil
}

HTTP Server Example

func (s *Server) OnReconcile(ctx context.Context, node v1alpha1.TinyNode) error {
    // Read port from metadata (all pods)
    port := node.Status.Metadata["http-server-port"]

    if utils.IsLeader(ctx) && port == "" {
        // Leader starts server and publishes port
        actualPort := s.startServer()
        s.emit(ctx, v1alpha1.ReconcilePort, func(n *v1alpha1.TinyNode) error {
            n.Status.Metadata["http-server-port"] = strconv.Itoa(actualPort)
            return nil
        })
    } else if port != "" {
        // All pods use the published port
        s.startOnPort(port)
    }
    return nil
}

Controller-Level Leadership

Controllers also check leadership:

func (r *TinyNodeReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    // All pods reconcile locally
    r.Scheduler.Update(ctx, node)

    // Only leader patches status — non-leaders requeue so a pod that
    // wins leadership later still publishes fresh status
    if !r.IsLeader.Load() {
        return ctrl.Result{RequeueAfter: time.Minute}, nil
    }

    // Leader-only: patch status
    r.Status().Patch(ctx, node, client.MergeFrom(originNode))
    return ctrl.Result{RequeueAfter: 5 * time.Minute}, nil
}

When a pod wins leadership, the SDK requeues every TinyNode of the module so the new leader republishes all statuses immediately.

Testing Leadership

With a single replica, the pod acquires the Lease automatically — no special flag needed. In unit tests, stamp leadership onto the context directly:

ctx := utils.WithLeader(context.Background(), true)  // leader
ctx = utils.WithLeader(context.Background(), false)  // non-leader

Best Practices

1. Don't Assume Leadership

// Bad: Assumes will always be leader
func (c *Component) OnControl(ctx context.Context, msg any) error {
    c.updateClusterState() // May not be leader!
    return nil
}

// Good: Check leadership
func (c *Component) OnControl(ctx context.Context, msg any) error {
    if utils.IsLeader(ctx) {
        c.updateClusterState()
    }
    return nil
}

2. Handle Leadership Changes

type Component struct {
    cancelFunc context.CancelFunc
    mu         sync.Mutex
}

func (c *Component) OnReconcile(ctx context.Context, node v1alpha1.TinyNode) error {
    c.mu.Lock()
    defer c.mu.Unlock()

    if utils.IsLeader(ctx) && c.cancelFunc == nil {
        // Just became leader
        var workCtx context.Context
        workCtx, c.cancelFunc = context.WithCancel(context.Background())
        go c.startLeaderOnlyWork(workCtx)
    } else if !utils.IsLeader(ctx) && c.cancelFunc != nil {
        // Lost leadership
        c.cancelFunc()
        c.cancelFunc = nil
    }
    return nil
}

3. Idempotent Leader Operations

func (c *Component) storeInit(ctx context.Context, output module.Handler) {
    if utils.IsLeader(ctx) {
        // Idempotent: safe to call multiple times
        output(ctx, v1alpha1.ReconcilePort, func(n *v1alpha1.TinyNode) error {
            if n.Status.Metadata["initialized"] != "true" {
                n.Status.Metadata["initialized"] = "true"
                // Do initialization...
            }
            return nil
        })
    }
}

Next Steps

Leader-Reader Pattern - Full pattern documentation
CR-Based State Propagation - Sharing state
Multi-Replica Coordination - Coordination patterns