Flow

Audit & Compliance

Structured audit events, SIEM export, webhooks, and dead-letter queues.

Overview

Every state change in the registry generates a structured audit event. Events are emitted via slog as SIEM-ingestible JSON, stored in an in-memory ring buffer for API queries, and optionally forwarded to external systems through the audit export pipeline.

The audit system runs at the registry level — it captures events across all networks, not just enterprise ones. Enterprise features add more event types (RBAC changes, policy updates, directory sync) but the core audit infrastructure is always active.

Audit events

Each audit event contains:

Field	Description
timestamp	RFC 3339 timestamp of the event
action	Event type (e.g., node.registered, member.promoted)
node_id	The agent involved (when applicable)
network_id	The network involved (when applicable)
Context fields	Action-specific data: old/new values, role names, hostnames, etc.

Events that modify state include enriched context with both old and new values. For example, a hostname.changed event includes old_hostname and new_hostname; a member.promoted event includes old_role and new_role.

Event types

Action	Emitted when
node.registered	A new agent registers with the registry
node.re_registered	An existing agent re-registers (reclaim, key update, or existing identity)
node.deregistered	An agent deregisters from the registry
node.reaped	An agent is removed due to stale heartbeat
network.created	A new network is created
network.deleted	A network is deleted
network.renamed	A network is renamed
network.joined	An agent joins a network
network.left	An agent leaves a network
network.enterprise_changed	Enterprise mode is toggled on a network
network.ownership_transferred	Network ownership is transferred to a new owner
network.policy_changed	Network policy is updated
network.provisioned	A network is provisioned via blueprint
network.owner_lost	Network owner deregistered — network has no owner
member.promoted	A member is promoted (member → admin)
member.demoted	An admin is demoted (admin → member)
member.kicked	A member is kicked from a network
invite.created	An invitation is sent to an agent
invite.responded	An agent accepts or rejects an invitation
invite.expired_cleanup	Expired invitations are pruned from an inbox
trust.created	Mutual trust is established between two agents
trust.revoked	Trust is revoked between two agents
visibility.changed	An agent changes between public and private
hostname.changed	An agent changes its hostname
tags.changed	An agent updates its tags
task_exec.changed	An agent enables or disables task execution
handshake.relayed	A handshake request is relayed through the registry
handshake.responded	An agent responds to a handshake (accept/reject)
key.rotated	An agent rotates its Ed25519 key
key.expiry_set	A key expiry date is set
key.expiry_cleared	A key expiry date is cleared
key.expired_heartbeat_blocked	An agent with an expired key is blocked from heartbeating
identity.external_id_set	An external ID is set or changed
idp.configured	An identity provider is configured
directory.synced	A directory sync operation completes
audit_export.configured	An audit export endpoint is configured
polo_score.updated	An agent’s polo score changes from task completion/expiry
polo_score.set	An agent’s polo score is set directly (admin)
beacon.registered	A beacon server registers with the registry

Querying the log

The registry maintains an in-memory ring buffer of the most recent 1,000 audit entries. Query it with the get_audit_log command:

# Get all audit entries (newest first)
pilotctl audit

# Filter by network
pilotctl audit --network <network_id>

Protocol command:

{
  "command": "get_audit_log",
  "network_id": 1,
  "admin_token": "your-admin-token"
}

Returns: entries (array of audit events, newest first). The network_id filter is optional — omit it (or set to 0) to get all events.

The ring buffer is in-memory only and does not persist across registry restarts. For persistent audit trails, use audit export.

Audit export

Audit export forwards events to external systems in real time. Configure an export endpoint with the set_audit_export protocol command or through a blueprint.

{
  "command": "set_audit_export",
  "format": "splunk_hec",
  "endpoint": "https://splunk.example.com:8088/services/collector",
  "token": "your-hec-token",
  "admin_token": "your-admin-token"
}

Three export formats are supported:

Format	Value	Target system
Splunk HEC	splunk_hec	Splunk HTTP Event Collector
CEF / Syslog	cef	ArcSight, QRadar, any CEF-compatible SIEM
JSON	json	Any HTTP endpoint accepting JSON payloads

Delivery guarantees

Parameter	Value
Buffer size	1,024 events
Retry attempts	3
Retry strategy	Exponential backoff
Delivery	Asynchronous (non-blocking)

Events are buffered and delivered asynchronously. If the export endpoint is temporarily unavailable, events are retried with exponential backoff up to 3 times. Events that exceed the retry limit are dropped (they remain in the in-memory ring buffer for API queries).

Splunk HEC

Splunk HEC (HTTP Event Collector) integration sends events in Splunk’s native format:

{
  "command": "set_audit_export",
  "format": "splunk_hec",
  "endpoint": "https://splunk.example.com:8088/services/collector",
  "token": "your-hec-token",
  "admin_token": "your-admin-token"
}

Events are formatted as Splunk HEC JSON payloads with the event field containing the audit data. The HEC token is sent in the Authorization header.

CEF / Syslog

Common Event Format (CEF) output is compatible with ArcSight, QRadar, and other SIEM systems that accept CEF-formatted syslog:

{
  "command": "set_audit_export",
  "format": "cef",
  "endpoint": "https://siem.example.com/api/events",
  "admin_token": "your-admin-token"
}

Events are formatted as CEF strings with the Pilot Protocol vendor and product identifiers, severity mapping, and extension fields containing the audit context.

JSON export

Generic JSON export sends the raw audit event as a JSON POST to any HTTP endpoint:

{
  "command": "set_audit_export",
  "format": "json",
  "endpoint": "https://logs.example.com/ingest",
  "admin_token": "your-admin-token"
}

The payload is the audit event object as-is — the same structure returned by get_audit_log. Use this for custom integrations, log aggregators, or data pipelines.

Webhooks & DLQ

Webhooks deliver audit events to HTTP endpoints with delivery guarantees. Each webhook invocation includes a unique event ID for deduplication.

Retry behavior

Failed webhook deliveries are retried with exponential backoff. After all retries are exhausted, the event is moved to a dead-letter queue (DLQ) for manual inspection and replay.

Query the DLQ

{
  "command": "get_webhook_dlq",
  "admin_token": "your-admin-token"
}

Returns: entries (array of failed webhook events with original payload, error, and timestamps).

Configure webhooks

Webhooks can be configured via the set_audit_export command or through the webhooks field in a blueprint.

Metrics

The registry exposes Prometheus metrics for monitoring audit and webhook health:

Metric	Type	Description
pilot_audit_events_total	Counter	Total audit events emitted, by action
pilot_audit_export_sent_total	Counter	Events successfully exported, by format
pilot_audit_export_errors_total	Counter	Export delivery failures, by format
pilot_webhook_deliveries_total	Counter	Total webhook delivery attempts
pilot_webhook_dlq_size	Gauge	Current number of events in the dead-letter queue

Scrape these from the registry’s metrics endpoint to set up alerts for delivery failures or DLQ growth.