Kubernetes Architecture
TinySystems modules are Kubernetes operators built with Kubebuilder. Understanding the Kubernetes architecture is essential for advanced module development.
Operator Pattern
Each module runs as a Kubernetes operator:
+-----------------------------------------------------------------------------+
| MODULE OPERATOR |
| |
| +------------------------------------------------------------------------+ |
| | CONTROLLER MANAGER | |
| | | |
| | +-------------+ +-------------+ | |
| | | TinyNode | | TinyModule | | |
| | | Controller | | Controller | | |
| | +------+------+ +------+------+ | |
| | | | | |
| | +------------+-----------+ | |
| | | | |
| +----------------------------------+-------------------------------------+ |
| | |
| +------------------------------------+-----------------------------------+ |
| | SCHEDULER | |
| | | | |
| | +-------------------------+-------------------------+ | |
| | | | | |
| | +-----+-----+ +-----------+ +-----------+ | | |
| | | Runner | | Runner | | Runner | ... | | |
| | |(node-abc) | |(node-def) | |(node-ghi) | | | |
| | +-----------+ +-----------+ +-----------+ | | |
| | | | |
| +--------------------------------------------------------------+ | |
| |
| +------------------------------------------------------------------------+ |
| | TRANSPORT | |
| | NATS receiver (TINY_NATS_URL set) / gRPC server (fallback) | |
| | (Cross-module communication) | |
| +------------------------------------------------------------------------+ |
+-----------------------------------------------------------------------------+
|
| Watch/Update
v
+-----------------------------------------------------------------------------+
| KUBERNETES API SERVER |
| |
| TinyNode CRDs TinyModule CRDs TinyFlow CRDs (platform-managed) |
+-----------------------------------------------------------------------------+
Custom Resource Definitions (CRDs)
TinySystems defines several CRDs. Module operators run controllers only for the first two; the rest are organizational resources managed by the platform:
| CRD | Purpose | Managed by |
|---|---|---|
| TinyNode | Component instance configuration | TinyNodeReconciler (module) |
| TinyModule | Module registration and discovery | TinyModuleReconciler (module) |
| TinyFlow | Group nodes in a flow | Platform |
| TinyProject | Top-level project container | Platform |
| TinyWidgetPage | Dashboard widget pages | Platform |
| TinyScenario | Test scenarios | Platform |
External triggers no longer use a CRD — the former TinySignal was replaced by direct NATS publishes (see Signals).
Controller-Runtime
Modules use controller-runtime:
import (
ctrl "sigs.k8s.io/controller-runtime"
)
// Simplified from the SDK's cli/run.go
mgr, err := ctrl.NewManager(config, ctrl.Options{
Scheme: scheme,
HealthProbeBindAddress: probeAddr,
// Controller-runtime's built-in leader election is DISABLED.
// The SDK runs its own leader elector (a Lease named
// "<module>-lock") so that non-leader pods still run their
// controllers and schedulers — they only skip status writes.
LeaderElection: false,
})
// Register controllers
(&controller.TinyNodeReconciler{ /* Module, IsLeader, Scheduler, ... */ }).SetupWithManager(mgr)
(&controller.TinyModuleReconciler{ /* Module, IsLeader, ClientPool, ... */ }).SetupWithManager(mgr)
// Start manager (after leadership is first resolved)
mgr.Start(ctx)
Reconciliation Loop
Controllers implement the reconciliation pattern:
+-----------------------------------------------------------------+
| RECONCILIATION LOOP |
+-----------------------------------------------------------------+
+---------------+
| Watch Events |
| (Create/Update|
| /Delete) |
+-------+-------+
|
v
+---------------+
| Work Queue |
| (Deduplication|
| Rate limiting)|
+-------+-------+
|
v
+---------------+
| Reconcile() |
| |
| 1. Get CR |
| 2. Check state|
| 3. Take action|
| 4. Update |
| status |
+-------+-------+
|
+-------------+-------------+
| |
v v
+-------------+ +-------------+
| Success | | Requeue |
| (No requeue)| | (After delay)|
+-------------+ +-------------+
Module Startup Flow
1. INITIALIZATION
+----------------------------------------------------------------+
| - Load kubeconfig |
| - Create controller manager |
| - Register component handlers |
| - Setup leader election |
+----------------------------------------------------------------+
|
v
2. LEADER ELECTION
+----------------------------------------------------------------+
| - Acquire Lease "<module>-lock" (custom elector, identity = |
| pod name; 15s lease / 10s renew / 2s retry) |
| - All pods run controllers and schedulers |
| - Only the leader performs status updates |
+----------------------------------------------------------------+
|
v
3. TRANSPORT + MODULE REGISTRATION
+----------------------------------------------------------------+
| - TINY_NATS_URL set: subscribe to tinymodule.<name>.msg |
| (queue group) and .sysmsg (per-pod fan-out); optional |
| JetStream durable transport via TINY_NATS_TRANSPORT |
| - Otherwise: start gRPC server (fallback transport) |
| - Create TinyModule CR; leader updates status with: |
| - Address (NATS subject or gRPC address) |
| - Version, SDK version |
| - Available components (with port schemas) |
+----------------------------------------------------------------+
|
v
4. CONTROLLER STARTUP
+----------------------------------------------------------------+
| - Start watching TinyNode CRs |
| - Start watching TinyModule CRs (for discovery) |
+----------------------------------------------------------------+
|
v
5. RECONCILIATION
+----------------------------------------------------------------+
| - Process TinyNode CRs matching this module |
| - Create Runner instances for each node |
| - Handle incoming messages and signals via the transport |
| - Periodic reconciliation (every 5 minutes, leader only) |
+----------------------------------------------------------------+
Key Components
Controller Manager
Manages all controllers and shared resources. Note that controller-runtime's leader election stays off; the SDK's own elector feeds an IsLeader flag into both reconcilers:
mgr, _ := ctrl.NewManager(config, ctrl.Options{
Scheme: scheme,
Logger: l,
HealthProbeBindAddress: probeAddr,
Cache: cache.Options{
DefaultNamespaces: map[string]cache.Config{namespace: {}},
},
LeaderElection: false, // custom elector, see Leader Election
})
Scheduler
Routes messages to component instances (simplified from internal/scheduler):
// Handle dispatches a message to the runner for its target node,
// or hands it to the message router for cross-module delivery.
func (s *Schedule) Handle(ctx context.Context, msg *runner.Msg) (any, error) {
nodeName, _, _ := utils.ParseFullPortName(msg.To)
instance, ok := s.instancesMap.Get(nodeName)
if !ok {
// Not local — cross-module delivery via NATS (or gRPC fallback)
return s.msgHandler(ctx, msg)
}
return instance.DataHandler(s.msgHandler)(ctx, msg)
}
Runner
Wraps a component instance. The component's Handle returns module.Result, which propagates back through the transport so blocking callers receive their synchronous response:
// Simplified
func (r *Runner) input(ctx context.Context, port string, data any) module.Result {
ctx, span := r.tracer.Start(ctx, "handle-message")
defer span.End()
// Deserialize, apply port configuration, evaluate {{expressions}}
msg := r.buildPortMessage(port, data)
// Call component — Result carries payload or error
return r.component.Handle(ctx, r.outputHandler, port, msg)
}
RBAC Requirements
Modules need these Kubernetes permissions:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: tinysystems-module
rules:
# CRD access
- apiGroups: ["operator.tinysystems.io"]
resources: ["tinynodes", "tinymodules"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete", "deletecollection"]
# Status updates
- apiGroups: ["operator.tinysystems.io"]
resources: ["tinynodes/status", "tinymodules/status"]
verbs: ["get", "update", "patch"]
# Finalizers
- apiGroups: ["operator.tinysystems.io"]
resources: ["tinynodes/finalizers", "tinymodules/finalizers"]
verbs: ["update"]
# Events
- apiGroups: [""]
resources: ["events"]
verbs: ["create", "patch"]
# Leader election
- apiGroups: ["coordination.k8s.io"]
resources: ["leases"]
verbs: ["get", "create", "update"]
# Service/Ingress management (for modules using ExposePort)
- apiGroups: [""]
resources: ["services"]
verbs: ["get", "list", "update", "patch"]
- apiGroups: ["networking.k8s.io"]
resources: ["ingresses"]
verbs: ["get", "list", "update", "patch"]
Components that touch other cluster resources declare their needs via the module's requirements; the SDK's tools rbac-values command generates the matching RBAC overlay.
Health Probes
Modules expose health endpoints:
// Liveness probe
mgr.AddHealthzCheck("healthz", healthz.Ping)
// Readiness probe
mgr.AddReadyzCheck("readyz", healthz.Ping)
Kubernetes probes:
livenessProbe:
httpGet:
path: /healthz
port: 8081
initialDelaySeconds: 15
periodSeconds: 20
readinessProbe:
httpGet:
path: /readyz
port: 8081
initialDelaySeconds: 5
periodSeconds: 10
Next Steps
- TinyNode CRD - Node configuration details
- Controller Reconciliation - Reconciliation patterns
- Leader Election - Multi-replica coordination