Run HTTP Services Over an Encrypted Agent Overlay

A 15-Line HTTP Server on Pilot

Here is a complete HTTP server that runs on Pilot port 80. Fifteen lines of application logic.

package main

import (
    "encoding/json"
    "fmt"
    "net/http"

    "github.com/TeoSlayer/pilotprotocol/pkg/driver"
)

func main() {
    d, err := driver.Connect()
    if err != nil {
        panic(err)
    }

    // Listen on Pilot port 80
    ln, err := d.Listen(80)
    if err != nil {
        panic(err)
    }
    defer ln.Close()

    mux := http.NewServeMux()

    mux.HandleFunc("/api/status", func(w http.ResponseWriter, r *http.Request) {
        w.Header().Set("Content-Type", "application/json")
        json.NewEncoder(w).Encode(map[string]any{
            "status":  "healthy",
            "agent":   d.Address().String(),
            "version": "1.0.0",
        })
    })

    mux.HandleFunc("/api/echo", func(w http.ResponseWriter, r *http.Request) {
        w.Header().Set("Content-Type", "application/json")
        var body map[string]any
        json.NewDecoder(r.Body).Decode(&body)
        json.NewEncoder(w).Encode(body)
    })

    fmt.Printf("HTTP server listening on pilot port 80 (%s)\n", d.Address())
    http.Serve(ln, mux)
}

The key line is d.Listen(80). This returns a net.Listener that accepts connections on Pilot virtual port 80. The listener is backed by the Pilot daemon's connection infrastructure -- the same encrypted tunnel, the same NAT traversal, the same trust model. But to http.Serve, it looks like any other listener.

Accessing the Service from Another Agent

Agent B wants to call Agent A's /api/status endpoint. It uses the driver to dial the connection and then wraps it in a standard HTTP client.

package main

import (
    "bufio"
    "fmt"
    "io"
    "net/http"

    "github.com/TeoSlayer/pilotprotocol/pkg/driver"
)

func main() {
    d, err := driver.Connect()
    if err != nil {
        panic(err)
    }

    // Dial Agent A's address on port 80
    conn, err := d.DialAddr("1:0001.0002.0003", 80)
    if err != nil {
        panic(err)
    }
    defer conn.Close()

    // Send a standard HTTP request over the Pilot connection
    req, _ := http.NewRequest("GET", "http://agent-a/api/status", nil)
    req.Write(conn)

    // Read the standard HTTP response
    resp, err := http.ReadResponse(bufio.NewReader(conn), req)
    if err != nil {
        panic(err)
    }
    defer resp.Body.Close()

    body, _ := io.ReadAll(resp.Body)
    fmt.Printf("Status: %d\nBody: %s\n", resp.StatusCode, body)
}

The HTTP request and response travel over the encrypted Pilot tunnel. Agent B does not need Agent A's IP address, does not need to open firewall ports, and does not need TLS certificates. The Pilot address (1:0001.0002.0003) is the only identifier needed.

Using net/http.Client Directly

For a more ergonomic approach, you can create an http.Transport that dials through Pilot:

package main

import (
    "fmt"
    "io"
    "net"
    "net/http"

    "github.com/TeoSlayer/pilotprotocol/pkg/driver"
)

func main() {
    d, _ := driver.Connect()

    // Custom transport that dials through Pilot
    transport := &http.Transport{
        DialContext: func(ctx context.Context, network, addr string) (net.Conn, error) {
            // addr is "hostname:port" - resolve hostname via registry
            host, _, _ := net.SplitHostPort(addr)
            resolved, err := d.Resolve(host)
            if err != nil {
                return nil, err
            }
            return d.DialAddr(resolved, 80)
        },
    }

    client := &http.Client{Transport: transport}

    // Now use standard HTTP client methods
    resp, err := client.Get("http://agent-a/api/status")
    if err != nil {
        panic(err)
    }
    defer resp.Body.Close()

    body, _ := io.ReadAll(resp.Body)
    fmt.Printf("%s\n", body)
}

With this transport, you can use client.Get(), client.Post(), and all the standard HTTP client methods. The hostname in the URL is resolved through the Pilot registry, and the connection goes through the encrypted tunnel. Existing code that accepts an *http.Client works without any changes.

Gateway: Bridge to Standard HTTP Clients

The Go driver approach works for agent-to-agent communication. But what about external tools -- curl, Postman, a Python requests call, or a browser? The Pilot gateway bridges the overlay network to your local machine.

Starting the Gateway

# Map Agent A's address to a local IP and expose port 80
pilotctl gateway start --ports 80 1:0001.0002.0003

# The gateway assigns a loopback alias (e.g., 10.4.0.1)
# Output: Gateway active: 1:0001.0002.0003 -> 10.4.0.1:80

The gateway does three things:

1. Adds a loopback alias: On macOS, ifconfig lo0 alias 10.4.0.1. On Linux, ip addr add 10.4.0.1/32 dev lo.
2. Listens on the local IP: TCP port 80 on 10.4.0.1.
3. Bridges to Pilot: Incoming TCP connections are forwarded over the encrypted Pilot tunnel to the agent's port 80. Responses come back the same way.

Now any standard HTTP client works:

# curl works
curl http://10.4.0.1/api/status
{"status":"healthy","agent":"1:0001.0002.0003","version":"1.0.0"}

# Python requests works
python3 -c "import requests; print(requests.get('http://10.4.0.1/api/status').json())"

# Even browsers work (navigate to http://10.4.0.1/api/status)

Multiple Agents, Multiple Local IPs

You can map multiple agents simultaneously. Each gets its own local IP:

pilotctl gateway start --ports 80 1:0001.0002.0003  # -> 10.4.0.1
pilotctl gateway start --ports 80 1:0001.0002.0004  # -> 10.4.0.2
pilotctl gateway start --ports 80 1:0001.0002.0005  # -> 10.4.0.3

# List active gateway mappings
pilotctl gateway list
ADDRESS              LOCAL IP      PORTS
1:0001.0002.0003     10.4.0.1      80
1:0001.0002.0004     10.4.0.2      80
1:0001.0002.0005     10.4.0.3      80

Build a REST API Mesh

Now for the real use case. Three agents running microservices: Users, Orders, and Inventory. Each exposes a REST API on Pilot port 80. The gateway makes them all accessible as local services.

Agent 1: Users Service

package main

import (
    "encoding/json"
    "net/http"

    "github.com/TeoSlayer/pilotprotocol/pkg/driver"
)

type User struct {
    ID    string `json:"id"`
    Name  string `json:"name"`
    Email string `json:"email"`
}

var users = map[string]User{
    "u1": {ID: "u1", Name: "Alice", Email: "[email protected]"},
    "u2": {ID: "u2", Name: "Bob", Email: "[email protected]"},
}

func main() {
    d, _ := driver.Connect()
    ln, _ := d.Listen(80)

    mux := http.NewServeMux()

    mux.HandleFunc("/api/users/", func(w http.ResponseWriter, r *http.Request) {
        id := r.URL.Path[len("/api/users/"):]
        w.Header().Set("Content-Type", "application/json")
        if user, ok := users[id]; ok {
            json.NewEncoder(w).Encode(user)
        } else {
            w.WriteHeader(404)
            json.NewEncoder(w).Encode(map[string]string{"error": "not found"})
        }
    })

    mux.HandleFunc("/api/users", func(w http.ResponseWriter, r *http.Request) {
        w.Header().Set("Content-Type", "application/json")
        all := make([]User, 0, len(users))
        for _, u := range users {
            all = append(all, u)
        }
        json.NewEncoder(w).Encode(all)
    })

    http.Serve(ln, mux)
}

Agent 2: Orders Service

package main

import (
    "encoding/json"
    "net/http"

    "github.com/TeoSlayer/pilotprotocol/pkg/driver"
)

type Order struct {
    ID     string  `json:"id"`
    UserID string  `json:"user_id"`
    Item   string  `json:"item"`
    Total  float64 `json:"total"`
    Status string  `json:"status"`
}

var orders = []Order{
    {ID: "o1", UserID: "u1", Item: "GPU Cluster Hours", Total: 250.00, Status: "completed"},
    {ID: "o2", UserID: "u2", Item: "Dataset License", Total: 75.00, Status: "pending"},
    {ID: "o3", UserID: "u1", Item: "Model Training Run", Total: 1200.00, Status: "active"},
}

func main() {
    d, _ := driver.Connect()
    ln, _ := d.Listen(80)

    mux := http.NewServeMux()

    mux.HandleFunc("/api/orders", func(w http.ResponseWriter, r *http.Request) {
        w.Header().Set("Content-Type", "application/json")
        userID := r.URL.Query().Get("user_id")
        if userID == "" {
            json.NewEncoder(w).Encode(orders)
            return
        }
        var filtered []Order
        for _, o := range orders {
            if o.UserID == userID {
                filtered = append(filtered, o)
            }
        }
        json.NewEncoder(w).Encode(filtered)
    })

    http.Serve(ln, mux)
}

Agent 3: Inventory Service

package main

import (
    "encoding/json"
    "net/http"

    "github.com/TeoSlayer/pilotprotocol/pkg/driver"
)

type InventoryItem struct {
    SKU       string `json:"sku"`
    Name      string `json:"name"`
    Available int    `json:"available"`
}

var inventory = []InventoryItem{
    {SKU: "gpu-a100", Name: "GPU Cluster Hours (A100)", Available: 48},
    {SKU: "dataset-01", Name: "ImageNet License", Available: 999},
    {SKU: "train-spot", Name: "Spot Training Instance", Available: 12},
}

func main() {
    d, _ := driver.Connect()
    ln, _ := d.Listen(80)

    mux := http.NewServeMux()

    mux.HandleFunc("/api/inventory", func(w http.ResponseWriter, r *http.Request) {
        w.Header().Set("Content-Type", "application/json")
        json.NewEncoder(w).Encode(inventory)
    })

    mux.HandleFunc("/api/inventory/check", func(w http.ResponseWriter, r *http.Request) {
        sku := r.URL.Query().Get("sku")
        w.Header().Set("Content-Type", "application/json")
        for _, item := range inventory {
            if item.SKU == sku {
                json.NewEncoder(w).Encode(map[string]any{
                    "sku":       item.SKU,
                    "available": item.Available > 0,
                    "count":     item.Available,
                })
                return
            }
        }
        w.WriteHeader(404)
        json.NewEncoder(w).Encode(map[string]string{"error": "sku not found"})
    })

    http.Serve(ln, mux)
}

Accessing the Mesh via Gateway

Start gateway mappings for all three agents:

# Map each service to a local IP
pilotctl gateway start --ports 80 1:0001.0002.0001  # Users  -> 10.4.0.1
pilotctl gateway start --ports 80 1:0001.0002.0002  # Orders -> 10.4.0.2
pilotctl gateway start --ports 80 1:0001.0002.0003  # Inventory -> 10.4.0.3

Now query the services with standard HTTP tools:

# Get all users
curl http://10.4.0.1/api/users
[{"id":"u1","name":"Alice","email":"[email protected]"},{"id":"u2","name":"Bob","email":"[email protected]"}]

# Get orders for user u1
curl "http://10.4.0.2/api/orders?user_id=u1"
[{"id":"o1","user_id":"u1","item":"GPU Cluster Hours","total":250,"status":"completed"},{"id":"o3","user_id":"u1","item":"Model Training Run","total":1200,"status":"active"}]

# Check inventory for a SKU
curl "http://10.4.0.3/api/inventory/check?sku=gpu-a100"
{"sku":"gpu-a100","available":true,"count":48}

These are standard HTTP requests to local IP addresses. The encryption, NAT traversal, and peer authentication all happen transparently at the Pilot tunnel layer. Your Python script, your JavaScript frontend, your Go microservice -- they all use their native HTTP clients unchanged.

Service-to-Service Calls

The services can also call each other. The Orders service can look up user details by dialing the Users service through Pilot:

// Inside the Orders service: enrich order with user details
func enrichOrder(d *driver.Driver, order Order) map[string]any {
    conn, _ := d.DialAddr("1:0001.0002.0001", 80) // Users service
    defer conn.Close()

    req, _ := http.NewRequest("GET",
        fmt.Sprintf("http://users/api/users/%s", order.UserID), nil)
    req.Write(conn)

    resp, _ := http.ReadResponse(bufio.NewReader(conn), req)
    defer resp.Body.Close()

    var user User
    json.NewDecoder(resp.Body).Decode(&user)

    return map[string]any{
        "order": order,
        "user":  user,
    }
}

The mesh works the same way whether all three agents are on the same machine, on different machines in the same datacenter, or on different continents behind different NATs. The code does not change.

Port 443: Double Encryption

Pilot port 80 uses the tunnel's AES-256-GCM encryption. Port 443 adds a second layer: a per-connection X25519 key exchange followed by AES-256-GCM encryption on the connection itself. This is double encryption -- the tunnel encrypts the outer packet, and the connection encrypts the inner payload.

Why Double Encryption?

The tunnel encryption key is shared across all connections between two daemons. If an attacker compromises the tunnel key, they can read all connections. Port 443's per-connection encryption means each connection has its own key derived from a fresh X25519 exchange. Compromising one connection does not affect others.

This is the same defense-in-depth approach as running TLS inside a VPN tunnel. It protects against key compromise at any single layer.

Setting Up a Secure HTTP Service

package main

import (
    "encoding/json"
    "fmt"
    "net/http"

    "github.com/TeoSlayer/pilotprotocol/pkg/driver"
)

func main() {
    d, err := driver.Connect()
    if err != nil {
        panic(err)
    }

    // Listen on secure port 443
    // The driver automatically performs X25519 key exchange
    // for each incoming connection
    ln, err := d.ListenSecure(443)
    if err != nil {
        panic(err)
    }
    defer ln.Close()

    mux := http.NewServeMux()

    mux.HandleFunc("/api/secrets", func(w http.ResponseWriter, r *http.Request) {
        w.Header().Set("Content-Type", "application/json")
        json.NewEncoder(w).Encode(map[string]string{
            "message": "This payload is double-encrypted",
            "tunnel":  "AES-256-GCM (shared tunnel key)",
            "conn":    "AES-256-GCM (per-connection X25519 key)",
        })
    })

    fmt.Printf("Secure HTTP server on pilot port 443 (%s)\n", d.Address())
    http.Serve(ln, mux)
}

The client side uses DialSecure instead of DialAddr:

// Client: connect to secure port
conn, err := d.DialSecure("1:0001.0002.0003", 443)
if err != nil {
    panic(err)
}
defer conn.Close()

// Standard HTTP over the double-encrypted connection
req, _ := http.NewRequest("GET", "http://agent/api/secrets", nil)
req.Write(conn)
resp, _ := http.ReadResponse(bufio.NewReader(conn), req)

The gateway also supports secure ports:

# Gateway with secure port mapping
pilotctl gateway start --ports 443 --secure 1:0001.0002.0003

# Access via curl (still plain HTTP on the local side)
curl http://10.4.0.1:443/api/secrets

Note that the gateway terminates the Pilot encryption and serves plain HTTP locally. The double encryption protects the data in transit between agents. The local loopback segment is unencrypted, which is standard practice (your local loopback is trusted).

Why This Matters

The HTTP ecosystem is enormous. Decades of tools, libraries, frameworks, middleware, and monitoring solutions all speak HTTP. By making Pilot connections compatible with net.Conn, every one of these tools works over the encrypted overlay without modification.

• Go: net/http, Gin, Chi, Echo -- all work. Pass the Pilot listener to your framework's Serve method.
• Gateway clients: Python requests, JavaScript fetch, Rust reqwest, Java HttpClient -- all work through the gateway. They see a local IP.
• Monitoring: Prometheus can scrape http://10.4.0.1/metrics. Grafana dashboards work unchanged.
• Load testing: wrk, hey, k6 -- point them at the gateway IP.

The encryption is automatic. The NAT traversal is automatic. The peer authentication is automatic. You write HTTP handlers and they just work, whether your agents are on the same LAN or on different continents behind symmetric NAT.

For the performance characteristics of HTTP over Pilot versus direct HTTP, see the benchmark comparison. For event-driven alternatives to REST, see the event stream tutorial. For the full API reference, check the documentation.