tiny

gRPC Fundamentals

gRPC is TinySystems' fallback cross-module wire. When TINY_NATS_URL is set, cross-module messages travel over NATS instead (see Cross-Module Communication); when it is unset, modules talk to each other over the gRPC path described here. Understanding it helps when building and debugging distributed flows.

Overview

+-----------------------------------------------------------------------------+
|                    GRPC IN TINYSYSTEMS (fallback wire)                       |
+-----------------------------------------------------------------------------+

  Module A                     Kubernetes                     Module B
  (common-module)              Network                        (http-module)
       |                          |                               |
       |  Message to              |                               |
       |  http-module             |                               |
       |       |                  |                               |
       |       v                  |                               |
       |  Scheduler               |                               |
       |       |                  |                               |
       |       | Not local?       |                               |
       |       |                  |                               |
       |       v                  |                               |
       |  AddressPool.Handler     |                               |
       |       |                  |                               |
       |       v                  |                               |
       |  gRPC Client -------------------------------> gRPC Server
       |                          |                               |
       |                          |                   Scheduler.Handle()
       |                          |                               |
       +--------------------------+-------------------------------+

What is gRPC?

gRPC is a high-performance RPC framework that uses:

  • Protocol Buffers: Efficient binary serialization
  • HTTP/2: Multiplexing, streaming, header compression
  • Service Definition: Strongly-typed APIs

TinySystems gRPC Service

The SDK defines a single-RPC service (internal/server/proto):

service ModuleService {
  rpc Message(MessageRequest) returns (MessageResponse);
}

message MessageRequest {
  string ID = 1;
  string From = 2;      // source full port name
  string To = 3;        // target full port name
  bytes Payload = 4;    // JSON-serialized message data
  string EdgeID = 5;    // edge that carried the message
}

message MessageResponse {
  bytes Data = 1;       // synchronous response payload (empty if none)
}

There is no error field in the response — handler failures are returned as regular gRPC status errors.

Server Implementation

Each module runs a gRPC server that dispatches into the scheduler. Simplified from internal/server/server.go:

modulepb.RegisterModuleServiceServer(srv, module.NewService(
    func(ctx context.Context, req *modulepb.MessageRequest) (*modulepb.MessageResponse, error) {
        // Extract message depth from gRPC metadata for cycle detection
        var depth int
        if md, ok := metadata.FromIncomingContext(ctx); ok {
            if vals := md.Get("x-message-depth"); len(vals) > 0 {
                depth, _ = strconv.Atoi(vals[0])
            }
        }

        res, err := handler(ctx, &runner.Msg{
            EdgeID: req.EdgeID,
            To:     req.To,
            From:   req.From,
            Data:   req.Payload,
            Depth:  depth,
        })
        if err != nil {
            return nil, err // surfaces as a gRPC status error
        }
        if utils.IsNil(res) {
            return &modulepb.MessageResponse{Data: nil}, nil
        }
        if resData, ok := res.([]byte); ok {
            return &modulepb.MessageResponse{Data: resData}, nil
        }
        data, err := json.Marshal(res)
        if err != nil {
            return nil, err
        }
        return &modulepb.MessageResponse{Data: data}, nil
    }))

handler is the scheduler's entry point:

Handle(ctx context.Context, msg *runner.Msg) (any, error)

The any return is the synchronous response for blocking I/O — it flows back to the caller as MessageResponse.Data.

The server also registers the standard gRPC health service and reflection, so grpcurl works out of the box.

Client Implementation

The sending side lives in the AddressPool (see Client Pool):

func (p *AddressPool) Handler(ctx context.Context, msg *runner.Msg) ([]byte, error) {
    moduleName, _, err := module.ParseFullName(msg.To)
    if err != nil {
        return nil, err
    }

    addr, ok := p.addressTable.Get(moduleName)
    if !ok {
        return nil, fmt.Errorf("%s module address is unknown", moduleName)
    }

    client, err := p.getClient(ctx, addr)
    if err != nil {
        return nil, err
    }

    // Propagate message depth for cross-module cycle detection
    if msg.Depth > 0 {
        md := metadata.Pairs("x-message-depth", strconv.Itoa(msg.Depth))
        ctx = metadata.NewOutgoingContext(ctx, md)
    }

    resp, err := client.Message(ctx, &module.MessageRequest{
        From:    msg.From,
        Payload: msg.Data,
        EdgeID:  msg.EdgeID,
        To:      msg.To,
    })
    if err != nil {
        return nil, err
    }
    return resp.Data, nil
}

The Message RPC is the blocking hop: the caller waits until the remote handler (and everything it triggers downstream) returns.

Connection Management

Keepalive

Both sides use keepalive to detect dead connections:

// Client side (AddressPool)
conn, err := grpc.NewClient(addr,
    grpc.WithTransportCredentials(insecure.NewCredentials()),
    grpc.WithStatsHandler(otelgrpc.NewClientHandler()),
    grpc.WithKeepaliveParams(keepalive.ClientParameters{
        Time:                10 * time.Second, // ping every 10s if idle
        Timeout:             3 * time.Second,  // wait 3s for ping ack
        PermitWithoutStream: true,             // ping even without active RPCs
    }),
)

// Server side
srv := grpc.NewServer(
    grpc.StatsHandler(otelgrpc.NewServerHandler()),
    grpc.KeepaliveParams(keepalive.ServerParameters{
        Time:    10 * time.Second,
        Timeout: 3 * time.Second,
    }),
    grpc.KeepaliveEnforcementPolicy(keepalive.EnforcementPolicy{
        MinTime:             5 * time.Second,
        PermitWithoutStream: true,
    }),
)

Connections are created lazily on first use, then conn.Connect() is triggered immediately to eliminate first-message latency. One connection per remote address is shared by all messages to that module.

Connection States

+---------------------------------------------------------------------------+
|                        GRPC CONNECTION STATES                              |
+---------------------------------------------------------------------------+

    IDLE ------> CONNECTING ------> READY
      |              |                 |
      |              |                 |
      |              v                 v
      |         TRANSIENT_FAILURE <-------
      |              |
      |              | Retry
      |              v
      +--------> CONNECTING
                     |
                     v
                 SHUTDOWN
StateDescription
IDLENo activity, will connect on demand
CONNECTINGEstablishing connection
READYConnected and healthy
TRANSIENT_FAILUREFailed, will retry
SHUTDOWNConnection closed

Error Handling

gRPC Status Codes

CodeNameDescription
0OKSuccess
1CANCELLEDOperation cancelled
2UNKNOWNUnknown error (includes remote handler failures)
4DEADLINE_EXCEEDEDTimeout
14UNAVAILABLEService unavailable

Handling Errors

import "google.golang.org/grpc/status"

resp, err := client.Message(ctx, req)
if err != nil {
    st, ok := status.FromError(err)
    if ok {
        switch st.Code() {
        case codes.Unavailable:
            // Module is down
        case codes.DeadlineExceeded:
            // Timeout
        default:
            // Remote handler error or other failure
        }
    }
}

Addressing

Modules advertise their listen address in the TinyModule CR's status.addr; peers watch those CRs and register the address in their pool. There is no fixed well-known port — the address is whatever the module bound and published. Traffic is plaintext gRPC inside the cluster; use NetworkPolicies to restrict which namespaces can reach module pods.

Debugging gRPC

Enable Logging

export GRPC_GO_LOG_VERBOSITY_LEVEL=99
export GRPC_GO_LOG_SEVERITY_LEVEL=info

Check Connectivity

Server reflection and the health service are registered, so:

# List services on a module (use the addr from TinyModule status)
grpcurl -plaintext <module-addr> list

# Health check
grpcurl -plaintext <module-addr> grpc.health.v1.Health/Check

Monitor Connections

Look for these log patterns from the pool:

"address pool: registering module" module=http-module addr=...
"address pool: connection pre-warmed" module=http-module addr=...
"grpc client: connection failed" addr=...

Performance Considerations

Message Size

Default gRPC limits apply (4MB send/receive). For larger payloads, chunk the data or pass a reference.

Timeouts

Cross-module calls block until the remote subtree finishes. Set a deadline on the context when the caller cannot wait indefinitely:

ctx, cancel := context.WithTimeout(ctx, 30*time.Second)
defer cancel()

Next Steps