When the Egress Proxy Rotates Its Credentials

The problem

The sandbox puts the proxy URL in the shell environment, HTTPS_PROXY and friends, with the credentials embedded in the URL. It also replaces those credentials every few minutes. A new shell always sees the current ones. A process that has been running for a while still holds whatever was in its environment when it started, and the proxy answers its next CONNECT with 407 Proxy Authentication Required.

Two things make this unusually hard to see:

• The 407 does not look like a 407. Inside the Muse VM, HTTP clients surfaced it as malformed HTTP status code, which sends you debugging the application, the server, or TLS: everything except the proxy credentials.
• The failure is partial. Connections opened when the process started were authenticated with credentials that were valid at the time, and they stay up. For a Pilot daemon that is the registry TLS tunnel and the beacon WebSocket, so lookups work and heartbeats flow. Every new HTTPS request, such as an app-store call or a fresh fetch, fails.

Detect it without printing a secret

Compare hashes of the proxy setting the daemon has and the one a fresh shell has. Never print the values themselves.

D=$(pgrep -f pilot-daemon | head -1)
tr '\0' '\n' < /proc/$D/environ | grep '^https_proxy=' | sha256sum   # what the daemon has
printf 'https_proxy=%s\n' "$(bash -c 'printf %s "$https_proxy"')" | sha256sum  # what a fresh shell has

Different hashes mean the sandbox rotated the credentials and the daemon's copy is stale. To confirm directly, open a TCP connection to the proxy, send CONNECT for any allowed host on port 443 with the daemon's Proxy-Authorization header, and read the status line: stale credentials get 407, fresh ones get 200 Connection Established.

The fix: a relay that re-stamps fresh credentials

Do not hand long-lived processes the raw proxy URL. Run a small relay on 127.0.0.1:3128 and point them at that instead. For every request it:

• strips whatever Proxy-Authorization the client sent;
• stamps credentials read from a fresh shell, which always sees the current ones, cached for 60 seconds;
• on a 407, re-reads the credentials immediately and retries once;
• forwards to the real proxy and then copies bytes in both directions.

It never terminates TLS. After the CONNECT handshake it is a byte pipe, so certificate verification stays end to end between your process and the real server. And it never logs credential values: only the method and target host, and the upstream status line. Python 3, standard library only:

#!/usr/bin/env python3
"""Credential-refreshing egress relay. Listens on 127.0.0.1:3128.

For every request: strips the client's Proxy-Authorization, stamps fresh
credentials read from a fresh shell (the sandbox rotates them; long-lived
processes otherwise start getting 407s), forwards to the real egress proxy,
and retries once with re-read credentials on a 407.

Never terminates TLS. Never logs credential values.
"""
import base64, os, socket, subprocess, threading, time, urllib.parse

LISTEN = ("127.0.0.1", 3128)
# How to read the *current* proxy URL. A fresh shell picks up rotated creds.
CRED_CMD = os.environ.get("RELAY_CRED_CMD",
                          'printf %s "${https_proxy:-$HTTPS_PROXY}"')
CACHE_SECONDS = 60
LOG = open("/tmp/egress_relay.log", "a", buffering=1)

_lock = threading.Lock()
_cache = {"auth": None, "upstream": None, "at": 0.0}


def log(msg):
    LOG.write(f"[{time.strftime('%H:%M:%S')}] {msg}\n")


def current_proxy(force=False):
    """Return (upstream (host, port), basic-auth token or None), cached."""
    with _lock:
        if not force and _cache["upstream"] and time.time() - _cache["at"] < CACHE_SECONDS:
            return _cache["upstream"], _cache["auth"]
        try:
            url = subprocess.run(["bash", "-c", CRED_CMD], capture_output=True,
                                 timeout=10).stdout.decode().strip()
            pu = urllib.parse.urlsplit(url if "://" in url else "http://" + url)
            auth = None
            if pu.username is not None:
                # userinfo is percent-encoded in the URL; the header wants it raw.
                user = urllib.parse.unquote(pu.username)
                pw = urllib.parse.unquote(pu.password or "")
                auth = base64.b64encode(f"{user}:{pw}".encode()).decode()
            if pu.hostname:
                _cache.update(upstream=(pu.hostname, pu.port or 3128), auth=auth,
                              at=time.time())
        except Exception as e:
            log(f"cred-refresh failed: {type(e).__name__}")
        return _cache["upstream"], _cache["auth"]


def read_head(sock, limit=65536):
    """Read up to the end of the HTTP header block. Returns (head, rest)."""
    buf = b""
    while b"\r\n\r\n" not in buf:
        chunk = sock.recv(4096)
        if not chunk or len(buf) > limit:
            return None, None
        buf += chunk
    head, _, rest = buf.partition(b"\r\n\r\n")
    return head.decode("latin-1"), rest


def pipe(src, dst):
    """Copy src -> dst until EOF, then half-close dst so the other
    direction can finish (TLS close_notify and late replies survive)."""
    try:
        while True:
            data = src.recv(65536)
            if not data:
                break
            dst.sendall(data)
    except OSError:
        pass
    finally:
        try:
            dst.shutdown(socket.SHUT_WR)
        except OSError:
            pass


def handle(client):
    up = None
    try:
        head, rest = read_head(client)
        if head is None:
            return
        lines = [ln for ln in head.split("\r\n")
                 if not ln.lower().startswith("proxy-authorization:")]
        log(lines[0].split(" ", 2)[0] + " " + lines[0].split(" ", 2)[1])  # method + target only
        for attempt in (1, 2):
            upstream, auth = current_proxy(force=(attempt == 2))
            if not upstream:
                client.sendall(b"HTTP/1.1 502 Bad Gateway\r\n\r\n")
                return
            hdrs = lines + ([f"Proxy-Authorization: Basic {auth}"] if auth else [])
            up = socket.create_connection(upstream, timeout=20)
            up.sendall(("\r\n".join(hdrs) + "\r\n\r\n").encode("latin-1") + rest)
            resp_head, resp_rest = read_head(up)
            status = (resp_head or "").split("\r\n", 1)[0]
            log(f"upstream -> {status[:60]!r}")
            if status.split(" ")[1:2] == ["407"] and attempt == 1:
                up.close()
                continue  # creds rotated under us: re-read them and retry once
            break
        if resp_head is None:
            client.sendall(b"HTTP/1.1 502 Bad Gateway\r\n\r\n")
            return
        client.sendall((resp_head + "\r\n\r\n").encode("latin-1") + resp_rest)
        # The 20 s timeout was for the handshake only. Tunnels (registry TLS,
        # beacon WebSocket) sit idle far longer than that.
        up.settimeout(None)
        client.settimeout(None)
        t = threading.Thread(target=pipe, args=(up, client), daemon=True)
        t.start()
        pipe(client, up)
        t.join()
    except Exception as e:
        log(f"ERR {type(e).__name__}")
    finally:
        for s in (client, up):
            if s is not None:
                try:
                    s.close()
                except OSError:
                    pass


def main():
    current_proxy()
    srv = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    srv.bind(LISTEN)
    srv.listen(128)
    log(f"listening on {LISTEN[0]}:{LISTEN[1]}")
    while True:
        c, _ = srv.accept()
        threading.Thread(target=handle, args=(c,), daemon=True).start()


if __name__ == "__main__":
    main()

This is not quite the relay that ran in the Muse VM. We reviewed that version and ran it against a test proxy that rotates its password on a schedule, and fixed three things before publishing it:

• Idle tunnels were torn down after 20 seconds. The 20-second socket timeout meant for the proxy handshake stayed set while piping bytes, so any tunnel that sat idle that long, such as a quiet registry connection or a WebSocket between pings, was closed by the relay itself. The timeout is now cleared once the tunnel is up.
• Passwords with reserved characters failed every time. The password in a proxy URL is percent-encoded, and the original sent it to the proxy still encoded, so a password containing @, % or / produced a 407 on every attempt, retry included. It is now decoded first. In our test the original failed and this version succeeded.
• Error handling and shutdown. A failed upstream reply is now returned to the client as a failure instead of being piped as if the tunnel were open. When one side finishes sending, the relay half-closes rather than tearing down both directions, so a TLS close or a late reply still gets through. The upstream address is cached along with the credentials instead of being re-read from a new shell on every connection.

Wire it up

# 1. The relay first. It reads fresh credentials itself.
nohup python3 egress_relay.py > /dev/null 2>&1 &

# 2. Then anything long-lived, pointed at the relay, with no credentials in the URL.
PROXY=http://127.0.0.1:3128
env https_proxy=$PROXY HTTPS_PROXY=$PROXY http_proxy=$PROXY HTTP_PROXY=$PROXY \
  NO_PROXY=localhost,127.0.0.1 <your daemon launch>

If you also run the SNI router from the pilot-sandbox skill, the router has the same problem: it reads HTTPS_PROXY once when it starts. Start it with HTTPS_PROXY=http://127.0.0.1:3128 so its upstream CONNECTs go through the relay, which stamps fresh credentials on them.

The subtlety that costs an hour

The SNI router only carries traffic addressed to 127.0.0.1, which is the registry and beacon traffic the hosts-file trick redirects. The daemon's other HTTPS clients, which serve app-store and broker calls, resolve real hostnames and use the proxy environment directly. They never touch the router. Fixing the router alone leaves that path broken, so the daemon itself must point at the relay too.

Verify with a call that forces a new HTTPS request through that path, not a lookup that reuses the long-lived registry tunnel. If it fails with malformed HTTP status code or a 407, the daemon is still on stale credentials.

Diagnostics cheat sheet

Rules

• Never print, log, or persist proxy credential values. Hashes only.
• Never terminate TLS in the relay. CONNECT, then a byte pipe, end to end.
• Anything that restarts the daemon, such as a watcher, a cron job, or a respawn loop, must launch it with the relay proxy settings. Otherwise the next automatic restart quietly brings the bug back.

A native fix is in progress: the daemon will re-read proxy credentials itself when the proxy rejects them, so a sandbox like this one will not need a relay. Until that ships, the relay works with any Pilot version, and with any other long-lived process in the same situation. The Muse install guide and the firewalls and compat mode page cover the rest of the setup.