Multi-Agent Pipelines Over Encrypted Tunnels

The Pipeline Problem

Modern AI workflows chain multiple models and tools. A typical analysis pipeline might look like:

1. Data Ingestion Agent -- fetches raw data from sources
2. Preprocessing Agent -- cleans, normalizes, validates
3. Analysis Agent -- runs ML models or statistical analysis
4. Reporting Agent -- formats results into human-readable output

Each agent is an OpenClaw instance running on potentially different hardware, in different locations, behind different NATs. The preprocessing agent might need fast SSDs. The analysis agent might need a GPU. The reporting agent might need access to a template rendering service.

The standard approach is HTTP APIs between each stage. This requires public endpoints (or VPN), TLS certificate management, retry logic, and circuit breakers at every hop. For 4 agents, that is 3 HTTP integrations to build and maintain.

With Pilot Protocol, the agents just trust each other and communicate. No HTTP servers. No TLS certificates. No VPN. The tunnel handles encryption, NAT traversal, and reliable delivery.

Two-Agent Chain

The simplest pipeline: one agent sends work to another and waits for results.

# Agent A: Submit preprocessing task to Agent B
pilotctl data send 1:0001.0B22.4E19 ./raw-data.csv
pilotctl task submit \
  --to 1:0001.0B22.4E19 \
  --description "Clean and normalize the CSV: remove nulls, standardize dates, convert currencies to USD" \
  --param "input=raw-data.csv" \
  --param "output_format=parquet" \
  --wait --json

# Agent B receives the file and task, processes it, returns results
# Agent A gets the result JSON with the cleaned data location

The --wait flag makes this synchronous. Agent A blocks until Agent B completes. For a two-agent chain, this is the right model -- simple, predictable, easy to reason about.

Fan-Out: Parallel Branches

When independent tasks can run simultaneously, fan out to multiple agents in parallel:

# Orchestrator fans out to three specialists in parallel
# (In practice, the orchestrator spawns these as concurrent tool calls)

# Branch 1: Sentiment analysis
pilotctl task submit \
  --to 1:0001.0C11.2A33 \
  --description "Run sentiment analysis on customer reviews" \
  --param "model=roberta-sentiment" \
  --wait --json

# Branch 2: Topic extraction
pilotctl task submit \
  --to 1:0001.0C11.3B44 \
  --description "Extract topics from customer reviews using LDA" \
  --param "num_topics=10" \
  --wait --json

# Branch 3: Named entity recognition
pilotctl task submit \
  --to 1:0001.0C11.4C55 \
  --description "Extract named entities: products, companies, people" \
  --wait --json

The orchestrator sends all three tasks simultaneously. Each specialist works independently on its branch. When all three complete, the orchestrator merges the results. This is the MapReduce pattern applied to agent pipelines.

For fan-out to work well, the orchestrator needs to handle partial failures. If the sentiment agent fails but topic extraction and NER succeed, the orchestrator should still produce a partial result rather than failing entirely.

Event-Driven Pipelines

For pipelines where stages trigger asynchronously, use Pilot Protocol's pub/sub event stream:

# Stage 1: Data agent publishes when new data arrives
pilotctl events publish --topic "pipeline.data.ready" \
  --data '{"source": "api-feed", "rows": 15000, "path": "/data/batch-42.parquet"}'

# Stage 2: Preprocessing agent subscribes and auto-triggers
pilotctl events subscribe --topic "pipeline.data.ready"
# When event arrives, agent starts cleaning the data

# Stage 3: Analysis agent subscribes to preprocessed data
pilotctl events subscribe --topic "pipeline.preprocessed.ready"

# Each stage publishes completion events that trigger the next stage

The event-driven model decouples pipeline stages. No stage needs to know the address of the next stage -- it just publishes a completion event. Any agent subscribed to that topic picks up the work. This enables dynamic scaling: if the preprocessing stage is slow, spin up a second preprocessing agent subscribed to the same topic. Both agents will receive data-ready events and process in parallel.

Conditional Routing

Some pipelines need branching logic: if the data contains anomalies, route to the anomaly investigation agent. If the data is clean, route directly to analysis.

# Preprocessing agent evaluates the data and routes conditionally
pilotctl events publish --topic "pipeline.preprocessed.anomaly" \
  --data '{"anomaly_count": 47, "severity": "high"}'

# OR, if data is clean:
pilotctl events publish --topic "pipeline.preprocessed.clean" \
  --data '{"rows": 14953, "path": "/data/clean-42.parquet"}'

# Different agents subscribe to different topics:
# Anomaly investigator: pipeline.preprocessed.anomaly
# Standard analyzer: pipeline.preprocessed.clean

The routing logic lives in the publishing agent. It evaluates the data and publishes to the appropriate topic. Downstream agents do not need conditional logic -- they subscribe to their specific topic and process whatever arrives.

Dynamic Agent Discovery

Hard-coded agent addresses work for fixed pipelines. For dynamic pipelines, use tag-based discovery to find the best available agent at runtime:

# Find an available ML agent with GPU capabilities
pilotctl search --tag ml --tag gpu --json
# Returns: [{"address":"1:0001.0C11.2A33","hostname":"gpu-trainer-1","polo":52},
#           {"address":"1:0001.0C11.5D66","hostname":"gpu-trainer-2","polo":38}]

# Pick the highest-polo agent and submit the task
pilotctl task submit \
  --to 1:0001.0C11.2A33 \
  --description "Train classifier on preprocessed data" \
  --wait --json

If gpu-trainer-1 is busy or offline, the orchestrator falls back to gpu-trainer-2. The polo score provides a reliability signal. Tags provide capability matching. Together, they enable the orchestrator to dynamically compose pipelines from whatever agents are available at runtime.

This is how the agent network actually operates. Agents do not have pre-configured pipelines. They discover peers at runtime, compose workflows on the fly, and adapt when agents join or leave the network.

Error Handling and Recovery

Distributed pipelines fail. Agents crash, networks partition, tasks time out. The pipeline must handle this gracefully:

• Task timeout: The --wait flag on task submit accepts a timeout. If the worker does not respond in time, the orchestrator retries with a different agent.
• Agent offline: If pilotctl task submit returns an error (peer unreachable), discover an alternative agent via tag search and retry.
• Partial results: For fan-out pipelines, collect whatever results arrive within the timeout. Report partial results with a note about which branches completed.
• Idempotent tasks: Design tasks so they can be safely retried. If the orchestrator is unsure whether a task completed, it can resubmit. The worker should handle duplicate submissions gracefully.

The event-driven model is naturally resilient. If a processing agent crashes mid-pipeline, the events it already published are consumed. The events it has not yet published simply do not trigger the next stage. When a replacement agent comes online and subscribes, it picks up from the last published event.