P2P Communication for Federated Learning Nodes

The FL Communication Bottleneck

A typical federated learning round works like this: each client trains a local model on its private data, computes gradients (or updated model weights), sends them to a central aggregation server, and receives the aggregated global model. The cycle repeats for hundreds or thousands of rounds.

The bottleneck is structural. Every client must upload its full gradient update to the same server. With a ResNet-50 model (about 100 MB of parameters) and 100 clients, the server must receive 10 GB of data per round. With a large language model at 7 billion parameters (roughly 14 GB in FP16), the numbers become absurd: 100 clients uploading 14 GB each means 1.4 TB of inbound traffic per round.

The server's inbound bandwidth becomes the chokepoint. Clients queue up waiting to upload. The slowest client (the straggler) determines when the round completes. And the server itself is a single point of failure -- if it goes down, all clients halt.

Gradient compression and quantization help reduce the per-client payload, but they do not fix the topology. The star topology -- every client talks to one server -- is the root cause. What you actually want is for clients to exchange gradients directly, peer to peer, and aggregate locally. This is decentralized federated learning, and its unsolved problem has always been communication: how do clients find each other, traverse NAT, and exchange data securely?

Why gRPC Falls Short

gRPC is the default transport for federated learning frameworks like Flower, PySyft, and TensorFlow Federated. It works well in controlled environments (all nodes in the same data center, no NAT, pre-configured TLS), but it breaks down in realistic FL deployments.

The 2 GB message size limit. gRPC defaults to a maximum message size of 4 MB, and even when you increase it, the practical ceiling is roughly 2 GB due to protobuf serialization constraints. A 7B parameter model in FP16 is 14 GB. You have to manually chunk, stream, reassemble, and handle partial failures. This is doable, but it is custom plumbing that every FL framework reimplements slightly differently.

No NAT traversal. gRPC requires the server to be reachable on a known IP and port. When FL clients are hospitals behind corporate firewalls, IoT devices on cellular networks, or laptops on residential WiFi, they cannot accept inbound connections. The standard workaround is a cloud-hosted relay server, which reintroduces the central bottleneck you were trying to avoid.

Server infrastructure required. Even for decentralized FL algorithms like gossip-based aggregation, you still need every node to run a gRPC server that other nodes can reach. This means public IPs, open firewall ports, and TLS certificate management for each participant. The operational burden scales linearly with the number of nodes, and most FL research teams do not want to become infrastructure teams.

No peer discovery. gRPC has no built-in mechanism for nodes to find each other. You need an external service registry, hardcoded peer lists, or a coordination server. In dynamic FL where nodes join and leave between rounds, this becomes its own distributed systems problem.

Pilot as FL Transport

Pilot Protocol is an overlay network for AI agents that runs over UDP. Each node gets a 48-bit virtual address, an Ed25519 identity, and automatic NAT traversal. For federated learning, it provides exactly the pieces that gRPC lacks: peer-to-peer connectivity through NAT, encrypted data exchange without infrastructure, and tag-based peer discovery.

The architecture is straightforward. Each FL client runs a Pilot daemon alongside its training process. The daemon handles networking: NAT traversal via STUN, hole-punching, or relay (automatically selected based on NAT type), encrypted tunnels using X25519 key exchange and AES-256-GCM, and peer discovery through a lightweight registry.

Gradient exchange happens over port 1001 (data exchange). This port supports file transfer and arbitrary message passing. A gradient tensor serialized to a file can be sent directly to a peer with a single CLI command or API call, with no size limit imposed by the protocol layer.

# Node A sends its gradient file to Node B
pilotctl send-file 1:0001.0002.0003 ./gradients_round_42.pt

# Node B sends its gradient file to Node A
pilotctl send-file 1:0001.0001.0001 ./gradients_round_42.pt

Both commands work regardless of NAT type. If both nodes are behind NAT, Pilot automatically negotiates hole-punching or falls back to relay through the beacon server. The gradient data is encrypted in transit with AES-256-GCM. No TLS certificates, no firewall rules, no cloud relay to configure.

Architecture: FL Nodes on Pilot

Here is the architecture for a decentralized FL system with N nodes, no central aggregation server, and gossip-based gradient exchange.

Each FL node consists of three components:

1. Training process -- PyTorch, TensorFlow, or JAX. Computes local gradients on private data.
2. Pilot daemon -- handles networking, runs in the background, uses 10 MB of memory.
3. FL coordinator script -- orchestrates the round: exports gradients, sends them to peers, receives peer gradients, aggregates, and updates the local model.

The round proceeds as follows:

1. Each node trains on its local data and serializes gradients to a file.
2. Each node discovers its peers using tags (see next section).
3. Each node sends its gradient file to K peers (where K is a configurable fan-out).
4. Each node receives gradient files from its peers.
5. Each node locally averages the received gradients with its own.
6. Repeat from step 1.

There is no central server. The aggregation is fully distributed. Each node converges toward the global average through repeated local averaging, which is mathematically equivalent to centralized FedAvg under standard assumptions.

Code Example: Python FL Node with Gradient Exchange

Here is a Python FL node that trains a model, exchanges gradients with peers via Pilot, and aggregates. It uses subprocess to call pilotctl for file transfer and peer discovery.

import subprocess
import json
import torch
import torch.nn as nn
import os
import time

# Simple model for demonstration
model = nn.Sequential(
    nn.Linear(784, 256),
    nn.ReLU(),
    nn.Linear(256, 10),
)
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)

def discover_peers():
    """Find all FL nodes in the network using tags."""
    result = subprocess.run(
        ["pilotctl", "find-by-tag", "fl-node", "--json"],
        capture_output=True, text=True,
    )
    peers = json.loads(result.stdout)
    return [p["address"] for p in peers if p["address"] != my_address()]

def my_address():
    result = subprocess.run(
        ["pilotctl", "status", "--json"],
        capture_output=True, text=True,
    )
    return json.loads(result.stdout)["address"]

def send_gradients(peer_address, round_num):
    """Send gradient file to a peer over Pilot port 1001."""
    grad_path = f"/tmp/gradients_round_{round_num}.pt"
    torch.save(
        {k: v.grad for k, v in model.named_parameters() if v.grad is not None},
        grad_path,
    )
    subprocess.run([
        "pilotctl", "send-file", peer_address, grad_path,
    ])

def receive_gradients(round_num, timeout=30):
    """Wait for gradient files from peers."""
    recv_dir = f"/tmp/received_round_{round_num}/"
    os.makedirs(recv_dir, exist_ok=True)
    # pilotctl receive writes incoming files to the specified directory
    subprocess.run([
        "pilotctl", "receive-file", "--output-dir", recv_dir,
        "--timeout", str(timeout),
    ])
    grad_files = [os.path.join(recv_dir, f) for f in os.listdir(recv_dir)]
    return [torch.load(f) for f in grad_files]

def aggregate(local_grads, peer_grads_list):
    """Average local gradients with peer gradients (FedAvg)."""
    n = len(peer_grads_list) + 1
    averaged = {}
    for key in local_grads:
        total = local_grads[key].clone()
        for peer_grads in peer_grads_list:
            if key in peer_grads:
                total += peer_grads[key]
        averaged[key] = total / n
    return averaged

# Main FL loop
NUM_ROUNDS = 100
FAN_OUT = 3  # Send to 3 random peers per round

for round_num in range(NUM_ROUNDS):
    # Step 1: Local training
    train_one_epoch(model, optimizer, local_dataloader)

    # Step 2: Discover peers
    peers = discover_peers()
    selected = peers[:FAN_OUT]

    # Step 3: Exchange gradients
    for peer in selected:
        send_gradients(peer, round_num)

    peer_grads = receive_gradients(round_num)

    # Step 4: Aggregate
    local_grads = {
        k: v.grad for k, v in model.named_parameters()
        if v.grad is not None
    }
    averaged = aggregate(local_grads, peer_grads)

    # Step 5: Apply averaged gradients
    with torch.no_grad():
        for name, param in model.named_parameters():
            if name in averaged:
                param.grad = averaged[name]
    optimizer.step()

    print(f"Round {round_num}: aggregated with {len(peer_grads)} peers")

This is a simplified example. A production FL system would add round synchronization, gradient compression, Byzantine fault tolerance for malicious peers, and differential privacy noise injection. But the transport layer -- discovering peers, sending gradient files, receiving them, and doing this through NAT -- is handled entirely by Pilot.

Discovery: FL Nodes Find Each Other via Tags

In centralized FL, the server knows all clients because they registered with it. In decentralized FL, you need a discovery mechanism. Pilot uses tags: key-value labels that agents attach to themselves in the registry.

# Mark this node as an FL participant working on a vision model
pilotctl set-tags fl-node vision-model round-robin

# Find all nodes participating in this FL task
pilotctl find-by-tag fl-node --json

The output gives you each peer's virtual address, hostname, and tags:

[
  {"address": "1:0001.0002.0003", "hostname": "hospital-a", "tags": ["fl-node", "vision-model"]},
  {"address": "1:0001.0004.0005", "hostname": "hospital-b", "tags": ["fl-node", "vision-model"]},
  {"address": "1:0001.0006.0007", "hostname": "clinic-c", "tags": ["fl-node", "vision-model"]}
]

Tags allow for flexible grouping. You can run multiple FL tasks on the same network by using different tags: fl-node vision-model for one task, fl-node nlp-model for another. Nodes discover only their relevant peers. When a node leaves the FL task, it removes its tag and disappears from discovery results.

This replaces the registration server, the peer list configuration file, and the manual endpoint management that decentralized FL frameworks typically require.

Security: Encrypted Gradient Exchange

Gradient leakage is a well-documented attack vector in federated learning. Research has shown that raw gradients can be inverted to reconstruct training data -- including images, text, and tabular records. If gradients are transmitted in plaintext over the network, any observer on the path can capture them and attempt reconstruction.

Pilot encrypts all data in transit using X25519 key exchange and AES-256-GCM. When two nodes establish a connection, they perform an ECDH handshake to derive a shared secret, then use that secret to encrypt every byte of payload data. This happens automatically, with no configuration.

Beyond encryption, Pilot's trust model adds access control. Nodes are private by default -- they are not discoverable unless they explicitly opt in. For FL, this means a node must both tag itself as an FL participant and approve trust handshakes from peers before any gradient exchange can occur.

# Node A requests trust with Node B (includes justification)
pilotctl trust request 1:0001.0002.0003 --reason "FL gradient exchange for vision-model task"

# Node B approves (after verifying the request)
pilotctl trust approve 1:0001.0001.0001

Once trust is established, it persists across sessions. If a node is compromised or behaves maliciously (sending poisoned gradients), any peer can instantly revoke trust:

# Revoke trust from a suspected malicious node
pilotctl trust revoke 1:0001.0002.0003

Revocation is immediate. The revoked node can no longer send data to or receive data from the revoking peer. This provides a practical defense against model poisoning attacks at the transport layer: if your Byzantine fault detection algorithm flags a node, you can cut it off at the network level without waiting for the next aggregation round.

Performance: Pilot vs gRPC vs Custom P2P

We compared three transport options for exchanging a 100 MB gradient file between two nodes, one in us-east and one in eu-west, both behind NAT.

The gRPC relay path adds latency because data goes through a third-party server instead of directly between peers. Pilot's direct P2P path (via hole-punching) avoids this extra hop, resulting in lower transfer times for large payloads. The custom UDP approach is slightly faster on raw throughput, but requires thousands of lines of transport code to handle reliability, congestion control, and NAT traversal -- exactly the code that Pilot already provides.

For large models (7B+ parameters, 14+ GB), the difference is more pronounced. gRPC requires chunking and streaming logic that adds complexity and failure modes. Pilot's file transfer sends the data as a stream over the encrypted tunnel with built-in congestion control and segmentation, handling large payloads natively.

Getting Started

To set up an FL node with Pilot transport:

# Install Pilot on each FL node
go install github.com/TeoSlayer/pilotprotocol/cmd/pilotctl@latest

# Start the daemon
pilotctl daemon start

# Join the network and tag as FL participant
pilotctl join --network 1
pilotctl set-tags fl-node vision-model

# Verify connectivity to another FL node
pilotctl ping 1:0001.0002.0003

# Exchange a test file
pilotctl send-file 1:0001.0002.0003 ./test_gradients.pt

The daemon handles NAT traversal automatically. You do not need to configure firewall rules, open ports, or set up relay servers. If both nodes are behind symmetric NAT (the hardest case), Pilot falls back to relay through the beacon server transparently.

For a full decentralized FL system, integrate the pilotctl commands into your training loop as shown in the Python example above, or use the Go driver package directly for tighter integration:

import "github.com/TeoSlayer/pilotprotocol/pkg/driver"

func exchangeGradients(peerAddr string, gradFile string) error {
    d, err := driver.Connect()
    if err != nil {
        return err
    }
    return d.SendFile(peerAddr, gradFile)
}

Communication overhead is the dominant bottleneck in federated learning. Removing the central server from the communication path, enabling direct peer-to-peer gradient exchange, and handling NAT traversal automatically cuts that overhead substantially. Pilot provides the transport layer so you can focus on the learning algorithms instead of the networking plumbing.