Module Discovery
Module discovery enables modules to find each other for cross-module communication. TinySystems uses TinyModule CRDs as a distributed service registry.
Discovery plays two roles depending on the transport:
- NATS transport (
TINY_NATS_URLset — the primary wire): message routing needs no address lookup at all, since the subjecttinymodule.<module>.msgis derived from the module name. TinyModule status still matters for tooling — it publishes the module's version, SDK version, and component catalog (including port schemas). - gRPC fallback (no NATS):
status.addris the dial target, registered into the ClientPool as described below.
How Discovery Works
+-----------------------------------------------------------------------------+
| MODULE DISCOVERY ARCHITECTURE |
+-----------------------------------------------------------------------------+
+-----------------+ +-----------------+ +-----------------+
| common-module | | http-module | | my-module |
| | | | | |
| Creates: | | Creates: | | Creates: |
| TinyModule CR | | TinyModule CR | | TinyModule CR |
+--------+--------+ +--------+--------+ +--------+--------+
| | |
| | |
v v v
+-----------------------------------------------------------------------------+
| KUBERNETES API SERVER |
| |
| +---------------------------------------------------------------------+ |
| | TinyModule CRs | |
| | | |
| | common http my-module | |
| | +------------------+ +------------------+ +--------------+ | |
| | | status: | | status: | | status: | | |
| | | addr: nats://..| | addr: nats://..| | addr: ... | | |
| | | components: | | components: | | components:| | |
| | | - router | | - server | | - mycomp | | |
| | | - split | | - client | | | | |
| | +------------------+ +------------------+ +--------------+ | |
| +---------------------------------------------------------------------+ |
+-----------------------------------------------------------------------------+
^ ^ ^
| | |
| Watch all TinyModule CRs |
| | |
+--------+--------+ +--------+--------+ +--------+--------+
| common-module | | http-module | | my-module |
| | | | | |
| ClientPool: | | ClientPool: | | ClientPool: |
| - http-module | | - common-module| | - common-module|
| - my-module | | - my-module | | - http-module |
+-----------------+ +-----------------+ +-----------------+
Discovery Flow
1. MODULE STARTUP
+------------------------------------------------------------------------+
| Module creates its TinyModule CR (name sanitized, AlreadyExists |
| ignored — every pod calls this): |
| |
| resourceManager.CreateModule(ctx, module.Info{ |
| Name: "my-module", |
| Version: "1.0.0", |
| }) |
+------------------------------------------------------------------------+
|
v
2. LEADER ELECTION
+------------------------------------------------------------------------+
| Multiple pods compete for leadership |
| Winner becomes leader, others become readers |
+------------------------------------------------------------------------+
|
v
3. LEADER PUBLISHES STATUS
+------------------------------------------------------------------------+
| Only leader updates TinyModule.Status: |
| |
| instance.Status.Addr = "nats://tinymodule.my-module.msg" |
| // or "host:8483" on the gRPC fallback |
| instance.Status.Name = "my-module" |
| instance.Status.Version = "1.0.0" |
| instance.Status.SDKVersion = "0.13.59" |
| instance.Status.Components = [...] // incl. per-port schemas |
| r.Status().Update(ctx, instance) |
+------------------------------------------------------------------------+
|
v
4. OTHER MODULES DISCOVER
+------------------------------------------------------------------------+
| TinyModuleReconciler watches ALL TinyModule CRs |
| |
| For each remote module: |
| if instance.Status.Addr != "" { |
| r.ClientPool.Register(req.Name, instance.Status.Addr) |
| } |
+------------------------------------------------------------------------+
|
v
5. READY FOR COMMUNICATION
+------------------------------------------------------------------------+
| NATS transport: publish straight to tinymodule.<module>.msg |
| gRPC fallback: ClientPool holds connections to remote modules |
+------------------------------------------------------------------------+
TinyModule Controller Implementation
// tinymodule_controller.go
func (r *TinyModuleReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
instance := &v1alpha1.TinyModule{}
if err := r.Get(ctx, req.NamespacedName, instance); err != nil {
if errors.IsNotFound(err) {
// Module removed - cleanup connections
r.ClientPool.Deregister(req.Name)
return ctrl.Result{}, nil
}
return ctrl.Result{}, err
}
// Is this a REMOTE module?
if req.Name != r.Module.GetNameSanitised() {
// Register remote module for cross-module communication
if instance.Status.Addr != "" {
r.ClientPool.Register(req.Name, instance.Status.Addr)
log.Info("discovered remote module",
"name", req.Name,
"address", instance.Status.Addr)
}
return ctrl.Result{}, nil
}
// This is OUR module - only leader updates
if !r.IsLeader.Load() {
return ctrl.Result{RequeueAfter: 5 * time.Second}, nil
}
// Leader: publish our address, versions, and components
instance.Status.Addr = r.Module.Addr
instance.Status.Version = r.Module.Version
instance.Status.Name = r.Module.Name
instance.Status.SDKVersion = r.Module.SDKVersion
instance.Status.Components = r.buildComponentStatus(l)
if err := r.Status().Update(ctx, instance); err != nil {
return ctrl.Result{}, err
}
return ctrl.Result{}, nil
}
Client Pool (gRPC fallback)
The ClientPool manages gRPC connections when the NATS transport is not in use (simplified):
type Pool struct {
connections map[string]*grpc.ClientConn
mu sync.RWMutex
}
func (p *Pool) Register(moduleName, address string) {
p.mu.Lock()
defer p.mu.Unlock()
// Already registered?
if conn, exists := p.connections[moduleName]; exists {
if conn.Target() == address {
return // Same address, skip
}
// Different address - reconnect
conn.Close()
}
// Create new connection
conn, err := grpc.Dial(address,
grpc.WithTransportCredentials(insecure.NewCredentials()),
grpc.WithKeepaliveParams(keepalive.ClientParameters{
Time: 10 * time.Second,
Timeout: 3 * time.Second,
PermitWithoutStream: true,
}),
)
if err != nil {
log.Error("failed to connect to module",
"module", moduleName,
"address", address,
"error", err)
return
}
p.connections[moduleName] = conn
log.Info("connected to module", "module", moduleName, "address", address)
}
func (p *Pool) Get(moduleName string) (*grpc.ClientConn, bool) {
p.mu.RLock()
defer p.mu.RUnlock()
conn, ok := p.connections[moduleName]
return conn, ok
}
func (p *Pool) Unregister(moduleName string) {
p.mu.Lock()
defer p.mu.Unlock()
if conn, exists := p.connections[moduleName]; exists {
conn.Close()
delete(p.connections, moduleName)
}
}
Service Discovery vs DNS
TinySystems uses CR-based discovery rather than Kubernetes DNS:
| Aspect | CR-Based | DNS-Based |
|---|---|---|
| Updates | Real-time via watch | Cached, delayed |
| Metadata | Components, version | Just address |
| Leader awareness | Only leader publishes | N/A |
| Custom data | Extensible status | Fixed format |
Kubernetes Service (gRPC fallback)
On the fallback transport, the gRPC address typically points to a Service (the chart exposes port 8483):
apiVersion: v1
kind: Service
metadata:
name: common
namespace: tinysystems
spec:
type: ClusterIP
selector:
app: common
ports:
- name: grpc
port: 8483
targetPort: 8483
Module Naming
Module names are sanitized for Kubernetes:
func SanitizeResourceName(name string) string {
// github.com/tiny-systems/common-module -> common-module
// Lowercase, remove special chars, truncate to 63 chars
name = strings.ToLower(name)
name = regexp.MustCompile(`[^a-z0-9-]`).ReplaceAllString(name, "-")
if len(name) > 63 {
name = name[:63]
}
return strings.Trim(name, "-")
}
Cross-Module Message Routing
There is no separate "which module owns this node" registry — the node's name carries its module. The message router parses it and picks the wire (simplified from cli/run.go):
func route(ctx context.Context, msg *runner.Msg) (any, error) {
// msg.To is "flowID.module.nodename:port" — the module is in the name
targetModule, _, err := m.ParseFullName(msg.To)
if err != nil {
return nil, err
}
// Destination inside this module: dispatch to the local scheduler
if targetModule == moduleInfo.GetNameSanitised() {
return scheduler.Handle(ctx, msg)
}
// Cross-module: NATS subject delivery when the transport is enabled,
// gRPC AddressPool otherwise. Same semantics either way.
if natsTransport != nil {
return natsTransport.Handler(ctx, msg) // publish to tinymodule.<module>.msg
}
return pool.Handler(ctx, msg) // gRPC fallback via ClientPool
}
Dynamic Discovery
Modules can be added/removed at runtime:
1. New module deployed
+-> Creates TinyModule CR
+-> Other modules see watch event
+-> ClientPool.Register() called
+-> Cross-module communication enabled
2. Module removed
+-> TinyModule CR deleted
+-> Other modules see delete event
+-> ClientPool.Unregister() called
+-> Connections cleaned up
Best Practices
1. Don't Hand-Roll Discovery Waits
With the NATS transport there is nothing to wait for — publishes to tinymodule.<module>.msg succeed as soon as the broker is up, and the JetStream transport even buffers them durably. On the gRPC fallback, a send to a not-yet-discovered module fails and the edge surfaces the error (edges are single-shot by default). If a flow must tolerate module-warmup blips, put an explicit retryPolicy on the edge rather than sleeping in components.
2. Handle Connection Failures
func (p *Pool) Get(moduleName string) (*grpc.ClientConn, error) {
conn, ok := p.connections[moduleName]
if !ok {
return nil, fmt.Errorf("module not discovered: %s", moduleName)
}
// Check connection state
if conn.GetState() == connectivity.Shutdown {
return nil, fmt.Errorf("connection shutdown: %s", moduleName)
}
return conn, nil
}
Next Steps
- CR-Based State Propagation - Share state across pods
- Cross-Module Communication - gRPC details
- Horizontal Scaling - Scaling patterns