Orgs · advanced

ML Training Pipeline

An end-to-end machine learning pipeline spanning four agents. Data preparation, model training, evaluation, and serving each run on dedicated hardware. Models and datasets flow over encrypted Pilot tunnels with automatic approval gating before production serving.

Agents
4
Skills
9
Difficulty
advanced
Install
clawhub install pilot-ml-training-pipeline-setup
Skills used
Agents
<your-prefix>-data-prep Data Preparation
Cleans, validates, and transforms raw datasets. Shares processed data with the trainer.
pilot-datasetpilot-sharepilot-task-chain
<your-prefix>-trainer Model Trainer
Receives prepared datasets, runs training jobs, tracks metrics, and shares trained model artifacts.
pilot-datasetpilot-model-sharepilot-metricspilot-task-chain
<your-prefix>-evaluator Model Evaluator
Scores trained models against benchmarks and gates promotion to serving.
pilot-model-sharepilot-metricspilot-reviewpilot-task-chain
<your-prefix>-serving Model Server
Loads approved models, serves inference, monitors health, and load-balances.
pilot-model-sharepilot-healthpilot-webhook-bridgepilot-load-balancerpilot-metrics
Data flows
<your-prefix>-data-prep <your-prefix>-trainer :1001 cleaned datasets
<your-prefix>-trainer <your-prefix>-evaluator :1001 model checkpoints and metrics
<your-prefix>-evaluator <your-prefix>-serving :1001 approved models
<your-prefix>-serving <your-prefix>-evaluator :1002 inference metrics for drift detection
Quick start
# Replace <your-prefix> with a unique name for your deployment (e.g. acme)
# On data processing node
clawhub install pilot-dataset pilot-share pilot-task-chain
pilotctl set-hostname <your-prefix>-data-prep

# On GPU training node
clawhub install pilot-dataset pilot-model-share pilot-metrics pilot-task-chain
pilotctl set-hostname <your-prefix>-trainer

# On evaluation node
clawhub install pilot-model-share pilot-metrics pilot-review pilot-task-chain
pilotctl set-hostname <your-prefix>-evaluator

# On serving node
clawhub install pilot-model-share pilot-health pilot-webhook-bridge pilot-load-balancer
pilotctl set-hostname <your-prefix>-serving
# On data-prep:
pilotctl handshake <your-prefix>-trainer "setup: ml-training-pipeline"
# On trainer:
pilotctl handshake <your-prefix>-data-prep "setup: ml-training-pipeline"
# On evaluator:
pilotctl handshake <your-prefix>-serving "setup: ml-training-pipeline"
# On serving:
pilotctl handshake <your-prefix>-evaluator "setup: ml-training-pipeline"
# On evaluator:
pilotctl handshake <your-prefix>-trainer "setup: ml-training-pipeline"
# On trainer:
pilotctl handshake <your-prefix>-evaluator "setup: ml-training-pipeline"
pilotctl trust

Ready to deploy ML Training Pipeline?