Workload audit | Dagnam.AI Docs

Find the LLM calls a small model you own can replace, prove it on a holdout, and switch by changing three values.

Workload audit The audit reads a month of your LLM traces on your own machine, groups the calls into workloads, and tells you which ones are worth replacing with a small model you own. For the workloads that clear the bar, a second command trains candidate models, deploys them, replays a held-out slice of your real traffic against each one, and reports agreement, latency and cost side by side. Raw traces never leave your machine; only redacted, derived rows for the workloads you name are uploaded, and only after you confirm the list. Every workload gets one of three verdicts: Verdict Meaning ----------- ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- REPLACE A candidate cleared the quality floor agreement lower bound at or above 0.97 for labels, 0.95 for JSON, by default and the workload's teacher-to-student cost ratio is at or above the 3× default. NOT YET Worth replacing on economics, but no candidate cleared the floor in this audit. The report shows the best achieved; the workload is retested on every audit. KEEP Not a small-model job: below the 3× ratio, free text, too few samples under 1,000 traces, or a holdout under 200 rows, by default , or unknown cost. The report opens with these rows, so the audit says no where it should. Install The audit extra adds keyring for storing deployment keys; everything else the audit needs is already part of dagnam . The API key is only used by run , status , cancel and delete ; see Install /docs/dag-lib/install for how keys are created. The commands dagnam audit scan Reads the export, discovers workloads, computes spend and the verdicts, and writes scan-report.json and scan-report.md to --out default ./audit . It opens no network connection. --map is for the generic jsonl and csv sources and maps a trace field to one of your columns; --window is the it; --price-table overrides the bundled vendor price table; --json prints scan-report.json to stdout. dagnam audit run Trains, deploys and scores the candidate models for the workloads the scan marked worth auditing, or only the ones you name with --workloads . Before uploading anything it prints exactly what will leave your machine: each workload with its row count, split, redaction counts per class and the classes scanned for, the project the rows go to, and the credit ceiling. Then it waits for your confirmation; --yes skips the prompt, and without it the command refuses to upload when stdin is not a terminal. --floor overrides the default quality floor on the agreement lower bound, and --max-credits default 500 stops the run before a submit that would exceed it. The pre-check is spent so far plus the largest candidate so far , not the exact cost of the next one, so a workable ceiling is at least twice your most expensive candidate — the bundled fixture needs 1200, because its head-tune costs about 116 credits to train and about 396 more to replay and measure. --no-wait returns after submitting; state is written after every step, so re-running run or an interrupted one resumes where it stopped. --local-only keeps the run off your account entirely — see Watch it on the website watch-it-on-the-website . The result is audit-report.json and audit-report.md in the audit directory; --json prints the JSON to stdout. dagnam audit status Shows every training job and endpoint the run created, one row per workload and candidate, with the requests served by each endpoint over the last 7 days. dagnam audit cancel Cancels queued or running training jobs through the API and pauses the deployments. Nothing is deleted. dagnam audit delete Lists what will be deleted from your account, asks for confirmation --yes skips it , removes every platform artifact the run created and writes a receipt; --json prints the receipt. See Deleting an audit deleting-an-audit . What leaves your machine, and when Command What is sent -------- ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ scan Nothing. The command opens no network connection; a test in the SDK fails if it ever does. run Redacted, derived rows for the workloads you name, after a confirmation that lists them, and the report itself as it is produced — unless --local-only . Raw traces, session ids and timestamps stay local. status A request for the state of your jobs and endpoints. cancel Cancel requests for your jobs; pause requests for your deployments. delete Delete requests for the datasets, models and deployments the run created, then a read of each id to confirm it is gone. A derived row is the rendered prompt plus the response for one call: a "input", "label" pair for label workloads, or a messages list for JSON and short-span workloads. Rows are redacted and exact-deduplicated before they are written to disk under the audit directory, and the training/holdout split is submitted as row indices, not timestamps. Watch it on the website run publishes the audit to your account as it goes, so the report is readable at Audits /audits while the command is still working — the same verdicts, the same candidates, each one's step as it reaches it. What is published is the report, not the data behind it: workload ids, verdicts, spend and the masked template excerpts the scan produced, then each candidate's status, agreement, latency and cost, and the winner with its switch values. Rows are never published, and neither is a deployment key. The consent listing run prints before it uploads anything says so in its own line: --local-only declines it. The run trains, deploys and scores exactly as it otherwise would and writes the same four files; nothing about it appears on the website. A published report is worth two things the files are not. Share it mints a token-addressed link to a read-only copy of the same page — the report, without the lifecycle actions, the retrain, and the key material — for someone with no account; you can revoke it at any time. Retrain this workload queues one more candidate from a recipe and a base you pick, scored against the same held-out traces as the CLI's, so it lands in the same comparison rather than beside it. Deleting is still one command. dagnam audit delete removes the published audit along with every artifact the run created, and the receipt it writes to deleted.json is the server's own. The rest of the boundary is unchanged; see Privacy commitments privacy-commitments . Switching a workload When a workload's verdict is REPLACE, the report carries its switch snippet. The endpoint is OpenAI-compatible, so the change is three values in the client you already have: the base URL, the model, and the key. model is the deployment id from the report. The deployment key is never printed; you read it into DEPLOYMENT KEY by the key ref the report names, which resolves through your OS keyring when one is available and otherwise through secrets.json in the audit directory, readable only by you. Privacy commitments - Redaction runs before anything is written. Every row passes the same detector set on your machine and on the platform, and each finding is replaced with REDACTED: . The classes are PII EMAIL , PII PHONE , PII PAYMENT CARD Luhn-checked and PII NATIONAL ID US SSN . The scan is best-effort assistance, not certification: it looks only for those classes, and both reports list the classes searched and the counts found so the boundary of the check is visible. - The two scans must agree. The platform rescans uploaded rows with the same implementation; if it finds a class the local scan missed, the run stops for that workload and the report says so. - Reports carry no raw prompts. At most a 200-character masked template excerpt per workload. - Deployment keys are secrets. They live in your keyring or a file only you can read, never in a report or on the terminal. - Retention is yours to end. Nothing is deleted on a schedule; the datasets, models and deployments a run creates stay in your account until you run dagnam audit delete , and that command is the deletion. Deleting an audit dagnam audit delete reads state.json , deletes every deployment, model version through its model entry , dataset and the project the run created, in that order, then re-reads each id and expects not-found. It writes deleted.json in the audit directory as the receipt, and only once every id is confirmed gone; then it forgets the deployment keys and removes the local workloads/ rows. An id that was already absent is recorded as already absent , so re-running delete is safe. Report files All four files live in the audit directory. The JSON files are the source of truth; each Markdown file is rendered from its JSON and never recomputes a number. File Contents --------------------------------------- ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ scan-report.json / scan-report.md schema: "dagnam.audit.scan/1" , the audit-report.json / audit-report.md The scan plus, per audited workload, the candidate list with agreement metric, value, 95% interval, holdout size , latency, serving cost labelled estimated or measured, training cost, the winner, and the switch values. The Markdown opens with the KEEP rows, then the frontier per candidate, the switch snippet per winner, artifact ids, and the method: thresholds, window, sample sizes, truncation and dedup rates. Every report is validated on read; an unknown schema value is an error naming the version the CLI understands.