Debugging

This guide covers techniques for debugging TinySystems components and flows, from development to production issues.

Development Debugging

Local Development

Run your module locally with verbose logging:

# Enable debug logging
export LOG_LEVEL=debug

# Run module
go run ./cmd/main.go

IDE Debugging

VS Code launch.json:

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Debug Module",
            "type": "go",
            "request": "launch",
            "mode": "auto",
            "program": "${workspaceFolder}/cmd/main.go",
            "env": {
                "LOG_LEVEL": "debug",
                "KUBECONFIG": "${env:HOME}/.kube/config"
            }
        }
    ]
}

GoLand:

Run -> Edit Configurations
Add Go Build
Set environment variables
Add breakpoints

Print Debugging

Quick debugging with structured logging. Handle returns module.Result (built with module.Ok / module.Fail), never a bare error:

func (c *Component) Handle(ctx context.Context, output module.Handler, port string, msg any) module.Result {
    log := log.FromContext(ctx)

    log.Info(">>> HANDLE CALLED",
        "port", port,
        "msgType", fmt.Sprintf("%T", msg),
        "msg", fmt.Sprintf("%+v", msg),
    )

    result := c.process(ctx, output, port, msg) // returns module.Result

    log.Info("<<< HANDLE RESULT",
        "value", fmt.Sprintf("%+v", result.Value()),
        "error", result.Err(),
    )

    return result
}

Component Debugging

Debug Port

Add a debug port for introspection:

type DebugInfo struct {
    Settings   any            `json:"settings"`
    State      string         `json:"state"`
    IsLeader   bool           `json:"isLeader"`
    NodeName   string         `json:"nodeName"`
    Metadata   map[string]any `json:"metadata"`
}

func (c *Component) Ports() []module.Port {
    return []module.Port{
        // ... other ports
        {
            Name:          "debug",
            Label:         "Debug",
            Source:        false, // input port — receives the trigger
            Position:      module.Top,
            Configuration: struct{}{},
        },
        {
            Name:          "debug_out",
            Label:         "Debug Info",
            Source:        true, // output port — emits the dump
            Position:      module.Bottom,
            Configuration: DebugInfo{},
        },
    }
}

func (c *Component) Handle(ctx context.Context, output module.Handler, port string, msg any) module.Result {
    if port == "debug" {
        return output(ctx, "debug_out", DebugInfo{
            Settings: c.settings,
            State:    c.state,
            IsLeader: utils.IsLeader(ctx),
            NodeName: c.nodeName,
            Metadata: c.metadata,
        })
    }
    // ...
    return module.Ok(nil)
}

Note: Source: true marks an OUTPUT port; the port schema is reflected from the Go type in Configuration. Port.Schema (raw json.RawMessage) exists only for forms whose shape is unknown at compile time.

Control Port Debugging

Add debug buttons. Control messages never reach Handle — implement the module.ControlHandler capability interface instead:

type Control struct {
    Start      bool `json:"start,omitempty" title:"Start" format:"button"`
    Stop       bool `json:"stop,omitempty" title:"Stop" format:"button"`
    DumpState  bool `json:"dumpState,omitempty" title:"Dump State" format:"button"`
    Reset      bool `json:"reset,omitempty" title:"Reset" format:"button"`
}

func (c *Component) OnControl(ctx context.Context, msg any) error {
    control, ok := msg.(Control)
    if !ok {
        return nil
    }
    if control.DumpState {
        c.dumpStateToLog()
        return nil
    }
    // ...
    return nil
}

func (c *Component) dumpStateToLog() {
    log.Info("COMPONENT STATE DUMP",
        "settings", fmt.Sprintf("%+v", c.settings),
        "isRunning", c.isRunning,
        "currentPort", c.currentPort,
        "connections", len(c.connections),
    )
}

Flow Debugging

Message Tracing

Track message flow:

func (c *Component) Handle(ctx context.Context, output module.Handler, port string, msg any) module.Result {
    traceID := trace.SpanFromContext(ctx).SpanContext().TraceID().String()

    log.Info("MESSAGE TRACE",
        "traceID", traceID,
        "component", c.GetInfo().Name,
        "port", port,
        "msgPreview", preview(msg, 100),
    )

    // Wrap output to trace outgoing — module.Handler returns module.Result
    tracedOutput := module.Handler(func(ctx context.Context, port string, msg any) module.Result {
        log.Info("MESSAGE OUT",
            "traceID", traceID,
            "toPort", port,
            "msgPreview", preview(msg, 100),
        )
        return output(ctx, port, msg)
    })

    return c.process(ctx, tracedOutput, port, msg)
}

func preview(v any, maxLen int) string {
    s := fmt.Sprintf("%+v", v)
    if len(s) > maxLen {
        return s[:maxLen] + "..."
    }
    return s
}

Debug Component

Create a debug component that logs everything:

type Debug struct {
    settings Settings
}

type Settings struct {
    Label   string `json:"label" title:"Label" default:"debug"`
    Verbose bool   `json:"verbose" title:"Verbose Output"`
}

func (d *Debug) Handle(ctx context.Context, output module.Handler, port string, msg any) module.Result {
    if port == "input" {
        log.Info("DEBUG: "+d.settings.Label,
            "type", fmt.Sprintf("%T", msg),
            "value", func() any {
                if d.settings.Verbose {
                    return msg
                }
                return preview(msg, 200)
            }(),
        )
        return output(ctx, "output", msg)
    }
    return module.Ok(nil)
}

Kubernetes Debugging

Check Pod Status

# List pods
kubectl get pods -n tinysystems -l app=my-module

# Describe pod for events
kubectl describe pod my-module-abc123 -n tinysystems

# Check logs
kubectl logs my-module-abc123 -n tinysystems
kubectl logs my-module-abc123 -n tinysystems --previous  # Crashed pod

# Follow logs
kubectl logs -f my-module-abc123 -n tinysystems

Check CRDs

# List TinyNodes
kubectl get tinynodes -n tinysystems

# Describe specific node
kubectl describe tinynode my-flow-abc123 -n tinysystems

# Get TinyNode YAML
kubectl get tinynode my-flow-abc123 -n tinysystems -o yaml

# Check TinyModules
kubectl get tinymodules -n tinysystems

Debug with Shell

# Exec into pod
kubectl exec -it my-module-abc123 -n tinysystems -- sh

# Port forward for local access
kubectl port-forward my-module-abc123 8080:8080 -n tinysystems

# Check environment
kubectl exec my-module-abc123 -n tinysystems -- env

Network Debugging

# Check services
kubectl get svc -n tinysystems

# Test DNS resolution
kubectl run debug --rm -it --image=busybox -- nslookup my-module-v1.tinysystems

# NATS transport (primary when TINY_NATS_URL is set): check the broker
kubectl run debug --rm -it --image=natsio/nats-box -- \
  nats --server nats://tinysystems-nats:4222 sub 'tinymodule.>'

# gRPC fallback (TINY_NATS_URL unset): test the module's gRPC port
kubectl run debug --rm -it --image=busybox -- \
  nc -zv my-module-v1.tinysystems 8483

Common Issues

Component Not Receiving Messages

Symptoms: Handle() never called

Check:

TinyNode exists, status.status is OK and status.error is false
Edges are properly configured
Source component is sending

# Check node status
kubectl get tinynode my-node -n tinysystems -o yaml

# Check edges
kubectl get tinynode my-node -n tinysystems -o jsonpath='{.spec.edges}'

Messages Lost Between Modules

Symptoms: Cross-module messages not arriving

Check:

Both modules discovered each other (TinyModule status populated)
NATS transport: TINY_NATS_URL set on both modules and the broker reachable; gRPC fallback: connection healthy
No network policies blocking

# Check TinyModules
kubectl get tinymodules -n tinysystems

# Check module addresses (nats://tinymodule.<name>.msg with the NATS
# transport, host:port with the gRPC fallback)
kubectl get tinymodule my-module-v1 -n tinysystems -o jsonpath='{.status.addr}'

Leader Not Processing Controls

Symptoms: Control buttons don't work

Check:

Leader election working
Component checking utils.IsLeader(ctx) in OnControl

# Check leases (each module holds one Lease named <module>-lock)
kubectl get leases -n tinysystems

# Check which pod is leader
kubectl get lease my-module-lock -n tinysystems -o jsonpath='{.spec.holderIdentity}'

Settings Not Applied

Symptoms: Component ignores settings

Check:

Component implements module.SettingsHandler (OnSettings(ctx, settings any) error) — the _settings port never reaches Handle
TinyNode has a _settings port configuration in spec.ports
Settings schema matches

# Check TinyNode spec.ports for the _settings configuration
spec:
  ports:
    - port: _settings
      configuration: '{"timeout": 5000}'

Port Not Exposed

Symptoms: HTTP endpoint not accessible

Check:

Port added to the module release's Service (ExposePort mutates the existing operator Service, it does not create one per node)
Ingress rule and TLS entry added for the hostname
DNS resolving
Certificate ready (secret <hostname>-tls)

kubectl get svc,ingress -n tinysystems -l app.kubernetes.io/name=tinysystems-operator
kubectl get certificate -n tinysystems

Debugging Checklist

Flow Not Working

TinyNode exists, status.status is OK, status.error is false
All referenced components available
Edges properly configured
Expression syntax correct
Port schemas match data

Component Not Starting

Settings delivered before other messages
Required settings have values
No initialization errors in logs
Context not cancelled prematurely

Cross-Module Issues

Both TinyModules have addresses
NATS broker reachable (or gRPC port accessible on the fallback path)
No network policies blocking
Client pool has connection (gRPC fallback only)

Multi-Replica Issues

Leader election working
Only leader writes to CRs
All pods watching CR changes
Metadata propagating correctly

Debug Tools Summary

Tool	Use Case
`kubectl logs`	View component logs
`kubectl describe`	Check events and status
`kubectl get -o yaml`	View full resource
`kubectl exec`	Interactive debugging
`kubectl port-forward`	Local access
IDE debugger	Step through code
Debug component	Inspect flow data
Control buttons	Trigger debug actions

Next Steps

Testing Components - Test before debug
Observability - Production monitoring
FAQ - Common problems and solutions