Agentic HPC Console
active buildA console where research assistants describe the analysis they want and get a reviewable execution plan — sbatch arrays, partitions, cost estimate — before anything touches the cluster. The agent has a deliberately narrow tool surface, and every job it runs leaves provenance behind.
Console screenshot or agent/tool architecture diagram
problem
Research assistants need HPC, not HPC expertise. The gap between "run this pipeline on batch 2026-07" and a correct sbatch array — right partition, right GPU count, scratch quota, module versions — is where their work stalls and where the cluster gets misused. Handing them a shell teaches Slurm; handing them a ticket queue teaches waiting.
approach
- Plan, then approve. The agent never submits directly. It emits a concrete plan — command, resources, runtime and cost estimate — that a human approves, edits, or cancels.
- Narrow tools, not shell access. Six typed tools over the Slurm REST API and AWS Batch: submit, status, tail, cancel, fetch results, describe partitions. No arbitrary execution path.
- Provenance by default. Every run records the prompt, the plan, the resolved script, container digests, and inputs — so a result can be traced back to what produced it.
- Accounting built in. SlurmDBD plus Aurora gives per-team chargeback and capacity planning, which is what makes self-service acceptable to the people who own the budget.
outcome
Assistants launch, monitor, and retrieve their own multi-hundred-task jobs without an engineer in the loop, and the approval step catches the expensive mistakes — wrong partition, missing array bounds, oversized GPU requests — before they queue.
> run the CNV pipeline on the 48 samples from batch 2026-07plan sbatch cnv.sbatch --array=1-48 --partition=gpu --gres=gpu:1 est. 3h20m · 48 tasks · ~$18.40 (spot) · fsx scratch 1.2 TB [approve] [edit] [cancel]✓ submitted job 481203 · 48 tasks queued◍ running 31/48 complete · 4 gpu nodes · 1h02m elapsed✓ provenance run manifest → s3://mbbg-runs/481203/manifest.json$ stemfold monomers input[2026-08-08 17:06:01] Manifest found: 0/6 already completed[2026-08-08 17:06:01] Auto-detected precomputed alignments: /workspace/output/monomers/alignments[2026-08-08 17:06:01] Starting monomers inference with model: params_model_1_ptm[2026-08-08 17:06:01] Working directory: /workspace/input[2026-08-08 17:06:01] Output directory: /workspace/output/monomers[2026-08-08 17:06:01] Found 3 FASTA files to process[2026-08-08 17:06:01] GPUs for inference: 3 of 4 (capped to 3 input files)[2026-08-08 17:06:01][2026-08-08 17:06:01] -- Performance Profile -------------------------------------------------[2026-08-08 17:06:01] GPUs: 4x NVIDIA A10G (23028 MiB VRAM each), 3 active[2026-08-08 17:06:01] MSA mode: precomputed (skipped) (mmap (memory-mapped))[2026-08-08 17:06:01] Inference: round-robin across 3 GPU(s)[2026-08-08 17:06:01] Memory: 182G available[2026-08-08 17:06:01][2026-08-08 17:06:01] Optimization level: optimal[2026-08-08 17:06:01] ------------------------------------------------------------------------[2026-08-08 17:06:01][2026-08-08 17:06:02] Distributing 3 FASTA files across 3 GPUs[2026-08-08 17:06:02] GPU 0: 1 FASTA files[2026-08-08 17:06:02] GPU 1: 1 FASTA files[2026-08-08 17:06:02] GPU 2: 1 FASTA files[2026-08-08 17:06:02] Launching 3 parallel inference processes...[2026-08-08 17:06:02] GPU 0 (cuda:0): 1 files → /workspace/output/monomers/gpu_0[2026-08-08 17:06:02] GPU 1 (cuda:1): 1 files → /workspace/output/monomers/gpu_1[2026-08-08 17:06:02] GPU 2 (cuda:2): 1 files → /workspace/output/monomers/gpu_2[2026-08-08 17:07:08] PID 16298 completed successfully[2026-08-08 17:07:08] PID 16304 completed successfully[2026-08-08 17:07:08] PID 16310 completed successfully[2026-08-08 17:07:08] Merging results from 3 GPUs into /workspace/output/monomers...[2026-08-08 17:07:08][2026-08-08 17:07:08] Progress: 6/6 completed[2026-08-08 17:07:08][2026-08-08 17:07:08] Monomers: 6 prediction files created[2026-08-08 17:07:08] ----------------------------------------[2026-08-08 17:07:08] 512 /workspace/output/monomers/predictions/c3_model_1_ptm_relaxed.pdb[2026-08-08 17:07:08] 512 /workspace/output/monomers/predictions/c3_model_1_ptm_unrelaxed.pdb[2026-08-08 17:07:08] 512 /workspace/output/monomers/predictions/d3_model_1_ptm_relaxed.pdb[2026-08-08 17:07:08] 512 /workspace/output/monomers/predictions/d3_model_1_ptm_unrelaxed.pdb[2026-08-08 17:07:08] 65K /workspace/output/monomers/predictions/d4_model_1_ptm_relaxed.pdb[2026-08-08 17:07:08] 41K /workspace/output/monomers/predictions/d4_model_1_ptm_unrelaxed.pdb[2026-08-08 17:07:08] ---------------------------------------- Structure Summary ──────────────────────────────────────────────────────────────────────────────────── Name Chains Residues pLDDT (min/mean/max) pTM PAE(Å) c3_model_1_ptm_relaxed 1 (A) 208 31.2 / 96.5 / 98.8 N/A 2.9 c3_model_1_ptm_unrelaxed 1 (A) 208 31.2 / 96.5 / 98.8 N/A 2.9 d3_model_1_ptm_relaxed 1 (A) 90 31.1 / 62.1 / 80.3 N/A 11.9 d3_model_1_ptm_unrelaxed 1 (A) 90 31.1 / 62.1 / 80.3 N/A 11.9 d4_model_1_ptm_relaxed 1 (A) 110 33.3 / 54.5 / 68.2 N/A 13.9 d4_model_1_ptm_unrelaxed 1 (A) 110 33.3 / 54.5 / 68.2 N/A 13.9 ──────────────────────────────────────────────────────────────────────────────────── pLDDT Confidence Bands: c3: 97.1% very high (>90), 1.0% confident (70-90), 1.9% low (<70) c3: 97.1% very high (>90), 1.0% confident (70-90), 1.9% low (<70) d3: 0.0% very high (>90), 26.7% confident (70-90), 73.3% low (<70) d3: 0.0% very high (>90), 26.7% confident (70-90), 73.3% low (<70)[2026-08-08 17:07:09] Monomers inference completescreens
- Plan approval view
