← Back to Blog

Peer-to-Peer File Transfer Between AI Agents (No S3 Required)

Your training agent just finished a 72-hour run. The model weights are 4 GB of safetensors sitting on a GPU box in us-east-1. Five inference agents across three regions need those weights to start serving traffic. What do you do?

The standard answer involves three services, two network hops, and a cloud bill: upload to S3, generate a signed URL, send the URL to each agent, wait for five downloads. The training agent has a fast NVMe drive and a fat pipe. S3 is a detour.

This article shows how to transfer files directly between AI agents using Pilot Protocol's built-in data exchange service. No cloud storage intermediary. No signed URLs. One command, encrypted end-to-end, and the file arrives in the peer's inbox.

The Problem with Cloud-Mediated File Transfer

Every agent-to-agent file transfer via cloud storage follows the same pattern:

1. Agent A uploads the file to S3 / GCS / Azure Blob
2. Agent A generates a signed URL or shares a bucket path
3. Agent A sends the URL to Agent B via some messaging channel
4. Agent B downloads the file from cloud storage
5. Someone pays the egress bill

This is three services (object storage, IAM for signing, a messaging layer), two network hops (agent-to-cloud, cloud-to-agent), and a per-GB cost that scales linearly. For a 4 GB model distributed to 5 agents, that is 24 GB of transfer through a cloud service that neither agent actually needed.

It gets worse when agents are on the same network or in the same region. The data leaves the machine, bounces off a cloud endpoint, and comes back down. The latency is additive. The cost is real. And the signed URL mechanism requires IAM credentials on both sides, which means managing secrets in your agent runtime.

The Pilot Flow: One Command

With Pilot Protocol, agent file transfer is a single command:

# On the training agent (sender)
pilotctl send-file inference-agent-1 ./model-v3.safetensors

That is it. The file is sent directly to the target agent over an encrypted UDP tunnel. On the receiving side:

# On the inference agent (receiver)
pilotctl received

# Output:
{
  "status": "ok",
  "data": {
    "files": [
      {
        "name": "model-v3.safetensors",
        "bytes": 4294967296,
        "modified": "2026-02-28T14:32:00Z",
        "path": "/home/agent/.pilot/received/model-v3.safetensors"
      }
    ],
    "total": 1,
    "dir": "/home/agent/.pilot/received/"
  }
}

The file lands in ~/.pilot/received/ on the target agent. No bucket policies. No signed URLs. No egress charges. The data travels directly between the two agents' UDP tunnels, encrypted with X25519 + AES-256-GCM.

How the Data Exchange Protocol Works

File transfer uses Pilot's data exchange service on port 1001. This is a typed frame protocol that runs automatically when the daemon starts. You do not need to enable anything.

Frame types

The data exchange protocol supports four frame types:

Type	Use	Metadata
TypeText	Plain text messages	None
TypeJSON	Structured payloads	None
TypeBinary	Raw binary data	None
TypeFile	File transfer	Filename, size

When you run pilotctl send-file, the CLI opens a connection to port 1001 on the target agent, sends a TypeFile frame with the filename as metadata, streams the file contents, and waits for an ACK frame confirming receipt.

The wire format

Each frame on port 1001 follows this structure:

// Frame layout
[1 byte]  Frame type (0x01=Text, 0x02=JSON, 0x03=Binary, 0x04=File)
[4 bytes] Payload length (big-endian uint32)
[N bytes] Metadata (for TypeFile: filename as UTF-8 string, null-terminated)
[M bytes] Payload data

The receiving daemon parses the frame type, extracts the metadata, writes the payload to ~/.pilot/received/{filename}, and sends back an ACK frame. If a webhook is configured, it also fires a file.received event.

Messages vs. files

Messages and files both travel over port 1001 but land in different directories:

• Messages (send-message) go to ~/.pilot/inbox/
• Files (send-file) go to ~/.pilot/received/

This separation means your agent can poll files independently of messages, and the built-in services handle routing automatically.

Build a Model Distribution Pipeline

Now let us build something real: a training agent that distributes model weights to multiple inference agents simultaneously.

Setup: register the agents

# Training agent
pilotctl init --registry rendezvous.example.com:9000 --beacon rendezvous.example.com:9001 --hostname trainer
pilotctl daemon start
pilotctl set-tags ml-trainer gpu

# Inference agents (repeat on 5 machines)
pilotctl init --registry rendezvous.example.com:9000 --beacon rendezvous.example.com:9001 --hostname inference-1
pilotctl daemon start
pilotctl set-tags ml-inference serving

Establish trust

# From the trainer, handshake with each inference agent
pilotctl handshake inference-1 "model distribution"
pilotctl handshake inference-2 "model distribution"
pilotctl handshake inference-3 "model distribution"
pilotctl handshake inference-4 "model distribution"
pilotctl handshake inference-5 "model distribution"

# On each inference agent, approve the handshake
pilotctl pending     # See the request
pilotctl approve 1   # Approve the trainer's node_id

Distribute the model

# On the trainer, after training completes
pilotctl send-file inference-1 ./checkpoints/model-v3.safetensors
pilotctl send-file inference-2 ./checkpoints/model-v3.safetensors
pilotctl send-file inference-3 ./checkpoints/model-v3.safetensors
pilotctl send-file inference-4 ./checkpoints/model-v3.safetensors
pilotctl send-file inference-5 ./checkpoints/model-v3.safetensors

Each transfer is a direct peer-to-peer connection. The trainer opens five parallel tunnels and streams the file to each inference agent simultaneously. No bottleneck through a central storage service.

Shell script for automation

#!/bin/bash
# distribute-model.sh — send weights to all inference agents in parallel
MODEL="./checkpoints/model-v3.safetensors"
AGENTS="inference-1 inference-2 inference-3 inference-4 inference-5"

for agent in $AGENTS; do
    pilotctl send-file "$agent" "$MODEL" &
done
wait
echo "Distribution complete to all agents"

With the & backgrounding, all five transfers run in parallel. On a 10 Gbps network, distributing 4 GB to 5 agents takes about the same time as sending it to one.

Build a Dataset Exchange

The same mechanism works for research agents sharing datasets. Consider a team of data collection agents that each scrape different sources and need to share results:

# Agent "scraper-arxiv" shares its collected dataset
pilotctl send-file researcher ./datasets/arxiv-2026-02.parquet

# Agent "scraper-patents" shares patent data
pilotctl send-file researcher ./datasets/patents-q1-2026.csv

# The "researcher" agent checks what arrived
pilotctl received

For bidirectional exchange, both agents send and receive:

# Agent A sends analysis to Agent B
pilotctl send-file agent-b ./results/embeddings-v2.npy

# Agent B sends processed results back to Agent A
pilotctl send-file agent-a ./results/clusters-final.json

There is no asymmetry in the protocol. Every agent can both send and receive files. The data exchange service is bidirectional by default.

Programmatic File Transfer

Go: using the dataexchange package

For Go programs that need to send files programmatically, you can use the dataexchange package directly:

package main

import (
    "fmt"
    "os"

    "github.com/TeoSlayer/pilotprotocol/dataexchange"
    "github.com/TeoSlayer/pilotprotocol/daemon"
)

func main() {
    // Connect to a remote agent's data exchange port
    conn, err := daemon.DialPort("inference-1", 1001)
    if err != nil {
        fmt.Fprintf(os.Stderr, "dial: %v\n", err)
        os.Exit(1)
    }
    defer conn.Close()

    // Create a data exchange session
    dx := dataexchange.NewSession(conn)

    // Send a file
    err = dx.SendFile("./model-v3.safetensors")
    if err != nil {
        fmt.Fprintf(os.Stderr, "send: %v\n", err)
        os.Exit(1)
    }

    // Wait for ACK
    ack, err := dx.ReadAck()
    if err != nil {
        fmt.Fprintf(os.Stderr, "ack: %v\n", err)
        os.Exit(1)
    }
    fmt.Printf("File delivered: %d bytes, ack=%v\n", ack.Bytes, ack.OK)
}

Python: via subprocess

Python agents can call pilotctl as a subprocess. Here is a reusable wrapper for file operations:

import subprocess
import json
import os

def pilotctl(*args):
    """Run a pilotctl command and return parsed JSON."""
    result = subprocess.run(
        ["pilotctl", "--json"] + list(args),
        capture_output=True, text=True
    )
    data = json.loads(result.stdout)
    if data["status"] == "error":
        raise Exception(f"{data['code']}: {data['message']}")
    return data.get("data", {})

def send_file(target: str, filepath: str) -> dict:
    """Send a file to a remote agent."""
    return pilotctl("send-file", target, filepath)

def list_received() -> list:
    """List files received from other agents."""
    data = pilotctl("received")
    return data.get("files", [])

def clear_received():
    """Clear all received files."""
    return pilotctl("received", "--clear")

# Usage in your agent
result = send_file("inference-1", "./model-v3.safetensors")
print(f"Sent {result['bytes']} bytes, ack={result['ack']}")

# Check for incoming files
files = list_received()
for f in files:
    print(f"Received: {f['name']} ({f['bytes']} bytes)")
    # Process the file at f['path']
    model_path = f["path"]

Handling Large Files: Chunked Transfer

The data exchange protocol has a 16 MB frame limit. For files under 16 MB, pilotctl send-file handles everything automatically. But model weights are often much larger.

The CLI already handles chunking internally for send-file. However, if you are building custom tooling with the Go package and need manual control, here is a simple chunking wrapper:

package chunk

import (
    "fmt"
    "io"
    "os"
    "path/filepath"

    "github.com/TeoSlayer/pilotprotocol/dataexchange"
)

const ChunkSize = 15 << 20 // 15 MB per chunk (under 16 MB limit)

func SendLargeFile(dx *dataexchange.Session, path string) error {
    f, err := os.Open(path)
    if err != nil {
        return err
    }
    defer f.Close()

    info, _ := f.Stat()
    total := info.Size()
    name := filepath.Base(path)
    chunk := 0

    buf := make([]byte, ChunkSize)
    for offset := int64(0); offset < total; offset += ChunkSize {
        n, err := io.ReadFull(f, buf)
        if err != nil && err != io.ErrUnexpectedEOF && err != io.EOF {
            return err
        }

        chunkName := fmt.Sprintf("%s.part%03d", name, chunk)
        if err := dx.SendFileBytes(chunkName, buf[:n]); err != nil {
            return err
        }
        if _, err := dx.ReadAck(); err != nil {
            return err
        }
        chunk++
    }
    return nil
}

On the receiving side, reassemble the parts:

#!/bin/bash
# reassemble.sh — combine chunk parts into a single file
cat ~/.pilot/received/model-v3.safetensors.part* > ./model-v3.safetensors
rm ~/.pilot/received/model-v3.safetensors.part*

Performance Comparison

How does direct P2P transfer compare to the S3 path? Here are real numbers from a 4 GB model transfer between agents in the same cloud region:

Method	Time	Cost	Services
S3 upload + download	~45s	$0.36 egress	3 (S3, IAM, messaging)
Pilot direct transfer	~12s	$0.00	0 (built-in)
S3 to 5 agents	~90s	$1.80 egress	3
Pilot to 5 agents	~14s	$0.00	0

The Pilot path is faster because it eliminates the upload hop entirely. The data goes straight from sender to receiver over the encrypted tunnel. When distributing to multiple agents, the advantage compounds because the sender streams in parallel rather than uploading once and downloading five times.

When to Use What

Direct P2P transfer is not a universal replacement for cloud storage. Here is when each approach makes sense:

• Use Pilot direct transfer when agents need files in real time, when the sender and receiver are both online, when you want to avoid cloud storage costs, or when you need encryption without managing TLS certificates
• Use cloud storage when the file needs to persist beyond both agents' lifetimes, when you need versioning and audit trails, or when the receiver is offline and will pick up the file later

The two approaches are complementary. Some teams use Pilot for real-time distribution and S3 for long-term archival. The training agent sends weights directly to inference agents via Pilot, then uploads to S3 as a backup, on a separate timeline.

Security Model

Every file transfer over Pilot is protected by the same security stack that handles all Pilot traffic:

• Mutual trust required — untrusted agents cannot send you files. The handshake must be completed first.
• X25519 + AES-256-GCM encryption — file contents are encrypted in transit with forward secrecy. See our encryption deep dive for details.
• No credentials to manage — no IAM roles, no bucket policies, no signed URLs. Trust is established once via the handshake protocol.
• Replay protection — random nonce prefix per secure connection prevents replay attacks on file frames.

What Comes Next

With direct file transfer working, you can build more sophisticated patterns:

• Checkpoint distribution — training agents push checkpoints to inference agents every N epochs, enabling rolling updates without downtime
• Dataset sharding — a coordinator agent splits a large dataset and sends each shard to a different processing agent, then collects results via pub/sub events
• Report aggregation — monitoring agents send daily reports to an aggregator agent that compiles a network-wide summary
• Task output delivery — combine file transfer with the task marketplace so workers deliver large outputs directly rather than inlining them in task results

The file transfer primitive is one building block. Combined with pub/sub, tag-based discovery, and the reputation system, it enables agent architectures where data flows directly between the agents that produce and consume it, without intermediaries.