Controller Reconciliation

Reconciliation is the core pattern in Kubernetes operators. Understanding how TinySystems controllers reconcile ensures your components work correctly.

Reconciliation Pattern

+-----------------------------------------------------------------------------+
|                      RECONCILIATION PATTERN                                  |
+-----------------------------------------------------------------------------+

Desired State (Spec)         Actual State (Status/Runtime)
       |                              |
       |                              |
       v                              v
+-------------+              +-------------+
|   TinyNode  |              |   Runner    |
|   .Spec     |--------------|   Instance  |
|             |  Reconcile   |             |
| - module    |------------->| - component |
| - component |              | - ports     |
| - edges     |              | - state     |
+-------------+              +-------------+
       |                              |
       |                              |
       +--------------+---------------+
                      |
                      v
              Update Status
              (ports, metadata, error)

TinyNode Reconciliation

The TinyNodeReconciler handles node lifecycle:

// tinynode_controller.go (simplified from the SDK)
func (r *TinyNodeReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    // 1. The node NAME carries the module it needs — parse and match.
    //    Matching is tolerant of an optional publisher prefix.
    m, _, err := module.ParseFullName(req.Name)
    if err != nil {
        return ctrl.Result{}, nil // don't requeue invalid names
    }
    if !module.NameMatches(m, r.Module.GetNameSanitised()) {
        return ctrl.Result{}, nil // not our node
    }

    // 2. Fetch the TinyNode
    node := &v1alpha1.TinyNode{}
    if err := r.Get(ctx, req.NamespacedName, node); err != nil {
        if errors.IsNotFound(err) {
            return ctrl.Result{}, nil
        }
        return ctrl.Result{}, err
    }

    // 3. Deletion: finalizer drives cleanup so OnDestroy runs exactly once
    if !node.ObjectMeta.DeletionTimestamp.IsZero() {
        if controllerutil.ContainsFinalizer(node, nodeFinalizer) {
            if err := r.Scheduler.Destroy(req.Name); err != nil {
                return ctrl.Result{}, err
            }
            controllerutil.RemoveFinalizer(node, nodeFinalizer)
            if err := r.Update(ctx, node); err != nil {
                return ctrl.Result{}, err
            }
        }
        return ctrl.Result{}, nil
    }

    // 4. Add finalizer if needed
    if !controllerutil.ContainsFinalizer(node, nodeFinalizer) {
        controllerutil.AddFinalizer(node, nodeFinalizer)
        if err := r.Update(ctx, node); err != nil {
            return ctrl.Result{}, err
        }
    }

    originNode := node.DeepCopy()
    node.Status.Module = v1alpha1.TinyNodeModuleStatus{
        Name: r.Module.Name, Version: r.Module.Version, SDKVersion: r.Module.SDKVersion,
    }
    node.Status.Status = "OK"
    node.Status.Error = false

    // 5. Update runner instance (leadership flag rides the ctx for components)
    ctx = utils.WithLeader(ctx, r.IsLeader.Load())
    if err := r.Scheduler.Update(ctx, node); err != nil {
        return ctrl.Result{}, err
    }
    node.Status.ObservedGeneration = node.ObjectMeta.Generation

    // 6. Only the leader patches status. Non-leaders REQUEUE rather than
    //    drop: at pod start reconciles run before leadership is won.
    if !r.IsLeader.Load() {
        return ctrl.Result{RequeueAfter: time.Minute}, nil
    }

    t := metav1.NewTime(time.Now())
    node.Status.LastUpdateTime = &t
    if err := r.Status().Patch(ctx, node, client.MergeFrom(originNode)); err != nil {
        return ctrl.Result{}, err
    }

    // 7. Periodic heartbeat
    return ctrl.Result{RequeueAfter: 5 * time.Minute}, nil
}

Reconciliation Triggers

Controllers reconcile on these events:

Event	Trigger	Action
Create	New TinyNode CR	Create runner instance
Update	Spec changes (generation) or `status.metadata` changes	Update runner, rebuild ports
Delete	Deletion timestamp set	Finalizer runs `Scheduler.Destroy`, then is removed
Periodic	Timer (5 min, leader)	Refresh state, heartbeat status
Leadership won	Elector callback	Requeue ALL nodes so the new leader republishes status

Reconciliation must be idempotent — running it multiple times produces the same result. The SDK achieves this not by skipping generations but by making Scheduler.Update an upsert: re-running it against an unchanged spec re-dispatches the lifecycle callbacks with the same values and re-sends the same settings (which the runner dedups). status.observedGeneration is stamped on every pass so external tooling (e.g. WaitForNodeSync) can tell whether the controller has processed the current spec:

// after Scheduler.Update succeeds
node.Status.ObservedGeneration = node.ObjectMeta.Generation

Scheduler Update

Scheduler.Update upserts the runner instance for the node, then dispatches the typed lifecycle callbacks in framework-enforced order — system ports never reach Component.Handle:

1. Upsert runner (create with component.Instance() on first sight)
2. OnIdentity   — IdentityAware      (node name, namespace, flow, project)
3. OnClient     — ClientAware        (module.K8sClient)
4. OnNATS       — NATSAware          (JetStream handle; nil without TINY_NATS_URL)
5. OnState      — Stateful           (state backend; first update only)
6. OnEmitter    — EmitterAware       (long-lived Handler; first update only)
7. OnReconcile  — ReconcileHandler   (full TinyNode)
8. _settings message — dispatched to SettingsHandler.OnSettings
   (settings always run AFTER reconcile, so they win over restored state)

Reconciliation in Components

Components implement module.ReconcileHandler to react to node reconciles:

func (c *Component) OnReconcile(ctx context.Context, node v1alpha1.TinyNode) error {
    // Read shared metadata
    if val, ok := node.Status.Metadata["key"]; ok {
        c.cachedValue = val
    }

    // Cleanup stale resources
    c.cleanupOldConnections()

    // Update shared state (leader only) via the injected emitter
    if utils.IsLeader(ctx) {
        c.emit(ctx, v1alpha1.ReconcilePort, func(n *v1alpha1.TinyNode) error {
            n.Status.Metadata["last-reconcile"] = time.Now().Format(time.RFC3339)
            return nil
        })
    }

    return nil
}

Requeue Strategies

Controllers use different requeue strategies:

// Immediate requeue (error recovery)
return ctrl.Result{Requeue: true}, nil

// Delayed requeue (rate limiting)
return ctrl.Result{RequeueAfter: 10 * time.Second}, nil

// Periodic requeue (refresh)
return ctrl.Result{RequeueAfter: 5 * time.Minute}, nil

// No requeue (wait for next event)
return ctrl.Result{}, nil

// Error requeue (exponential backoff)
return ctrl.Result{}, err

Status Building

The controller stamps status.module, status.status, status.error, and status.observedGeneration; the runner's ReadStatus publishes the port list (simplified from internal/scheduler/runner):

func (c *Runner) ReadStatus(status *v1alpha1.TinyNodeStatus) error {
    status.Status = "OK"
    status.Error = false

    var ports []m.Port
    for _, p := range c.component.Ports() {
        // system plumbing ports never appear in status
        if p.Name == v1alpha1.ReconcilePort || p.Name == v1alpha1.ClientPort || p.Name == v1alpha1.IdentityPort {
            continue
        }
        ports = append(ports, p)
    }

    for _, p := range ports {
        // Schema: reflected from p.Configuration (Go type), or taken
        // verbatim from p.Schema for runtime-authored forms
        status.Ports = append(status.Ports, v1alpha1.TinyNodePortStatus{
            Name:     p.Name,
            Label:    p.Label,
            Position: v1alpha1.Position(p.Position),
            Source:   p.Source,
            Schema:   schemaBytes(p),
            // ...
        })
    }
    return nil
}

Watch Configuration

Controllers configure watches for efficient reconciliation. The TinyNode controller also consumes a leadership-change channel that requeues every node when this pod becomes leader:

func (r *TinyNodeReconciler) SetupWithManager(mgr ctrl.Manager) error {
    return ctrl.NewControllerManagedBy(mgr).
        For(&v1alpha1.TinyNode{}).
        WithEventFilter(GenerationOrMetadataChangedPredicate{}).
        WatchesRawSource(source.Channel(
            r.leadershipCh, // requeue-all when leadership is won
            handler.EnqueueRequestsFromMapFunc(r.allNodesOfThisModule),
        )).
        Complete(r)
}

GenerationOrMetadataChangedPredicate

The SDK uses a custom predicate rather than plain GenerationChangedPredicate. It fires on:

Spec changes (generation bump) — standard behavior
status.metadata changes — so non-leader pods pick up state the leader published (see CR-Based State Propagation)
Generic events — so the leadership requeue-all channel gets through

Status-only updates that do not touch metadata are still filtered out, preventing infinite loops when the leader writes status.

Error Handling in Reconciliation

func (r *TinyNodeReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    // Transient errors: return error for exponential backoff
    if err := r.Scheduler.Update(ctx, node); err != nil {
        if isTransient(err) {
            return ctrl.Result{}, err  // Will retry with backoff
        }
        // Permanent error: log and don't requeue
        log.Error(err, "permanent error")
        return ctrl.Result{}, nil
    }

    // Rate limit errors: explicit requeue delay
    if err := r.Status().Update(ctx, node); err != nil {
        if errors.IsConflict(err) {
            return ctrl.Result{RequeueAfter: 1 * time.Second}, nil
        }
        return ctrl.Result{}, err
    }

    return ctrl.Result{RequeueAfter: 5 * time.Minute}, nil
}

Best Practices

1. Check Generation

if node.Status.ObservedGeneration >= node.Generation {
    return ctrl.Result{RequeueAfter: 5 * time.Minute}, nil
}

2. Leader-Only Writes (Requeue, Don't Drop)

if !r.IsLeader.Load() {
    // Requeue: at pod start reconciles run BEFORE leadership is won
    return ctrl.Result{RequeueAfter: time.Minute}, nil
}

3. Clean Up via Finalizer

if !node.ObjectMeta.DeletionTimestamp.IsZero() {
    if controllerutil.ContainsFinalizer(node, nodeFinalizer) {
        r.Scheduler.Destroy(req.Name) // calls OnDestroy exactly once
        controllerutil.RemoveFinalizer(node, nodeFinalizer)
        r.Update(ctx, node)
    }
    return ctrl.Result{}, nil
}

4. Use Appropriate Requeue

// Error: exponential backoff
return ctrl.Result{}, err

// Conflict: short delay
return ctrl.Result{RequeueAfter: 1 * time.Second}, nil

// Periodic: long delay
return ctrl.Result{RequeueAfter: 5 * time.Minute}, nil

Next Steps

Leader Election - Multi-replica coordination
CR-Based State Propagation - Sharing state
Horizontal Scaling - Scaling patterns