You upgraded the model on Tuesday. On Wednesday the agents started producing malformed tool calls. The rollback plan: put the old weights back. That plan restores one of at least six things that define what your users are actually talking to. A served model is a release, not a file: - the weights, and their exact quantization - the tokenizer and the chat template - the generation defaults (temperature, stop tokens, max length) - the inference engine version and its flags - the system prompts and tool schemas your apps send - the routing config in front of it (gateway alias, fallbacks) Change any one and behavior changes. The chat template is the sneaky one: a revision that ships a different template changes how tool calls are formatted with the same weights and a perfect checksum. An engine upgrade can shift outputs with zero model change. And if you serve by a mutable tag ("latest", or any tag a publisher can re-push), a restart can quietly pull a different artifact than yesterday's. Three things to do before the next upgrade: 1. Write a release manifest. One file per release: weights revision (a commit hash, not a tag), quantization, tokenizer and template hashes, engine image digest, flags, prompt pack version, gateway route. Diffing manifest N against N-1 is your change log and your rollback spec. 2. Do the rollback arithmetic. Illustrative, not measured, arithmetic as of 2026-10-27: a 70B-parameter model in 16-bit is roughly 140 GB of weights (parameters x 2 bytes). Loading that at 1 GB/s from local NVMe takes a couple of minutes. Pulling it over a 1 Gbit link at roughly 0.1 GB/s takes about 25 minutes. Your rollback time is set by storage and network, not by your deploy tool. If N-1 is not staged on the node, "rollback" is a half-hour outage with a polite name. 3. Route through an alias. Apps call "assistant-prod", never a model name. Flipping the alias is the rollback: seconds, as long as N-1 is still resident or staged. That costs GPU headroom. The honest on-prem tax of safe releases is that canary and shadow traffic need VRAM your capacity plan probably did not include.

Passthrough a GPU to a VM and you've just turned one cluster node back into a pet. That's the part nobody puts in the "run your own LLM on Proxmox" tutorials. Getting the card into the VM takes an afternoon. Living with it is the real job. Here's what actually changes, and I checked each of these against current vendor docs. Passthrough (vSphere DirectPath I/O, Proxmox hostpci): • No live migration. VMware's docs say no vMotion, no suspend, no snapshots for a DirectPath VM. Proxmox's wiki says VMs with passed-through devices can't be migrated, full stop. • vSphere reserves the VM's full memory as soon as you add a PCI device. Your overcommit maths just broke. • Big-VRAM cards need 64-bit MMIO configured (pciPassthru.use64bitMMIO on vSphere, a bigger MMIO window on Proxmox/OVMF). Skip it and you'll be googling "DevicePowerOn failed" at 11pm. • The IOMMU group has to be clean. If your GPU shares a group with a SATA controller, the ACS override is a last resort, not a fix. • NUMA matters. Put the vCPUs on the socket the GPU is physically attached to. vGPU gives you some of the cluster back. vSphere 8 can vMotion vGPU VMs, with stun-time limits and some cluster settings to tune. Proxmox 8.4 added live migration for vGPU mediated devices. Here's what it costs you: • licensing and a license server you now have to keep alive • a supported-GPU list, and no GeForce cards on it • host driver and guest driver versions that have to stay compatible, so every upgrade becomes a choreographed one MIG only exists on specific datacenter cards (A100, A30, H100/H200, Blackwell datacenter parts and some RTX PRO Blackwell). Time-slicing works on more cards but gives you no memory or fault isolation between tenants. How I'd run it: 1. Treat the GPU VM as cattle that just happens to be pinned to one host. Run N+1 GPU hosts, not one. 2. Health-check the inference server and restart the process, not the VM. A wedged GPU sometimes only comes back with a host reboot. 3. Put GPU hosts in their own maintenance window, because DRS won't save you.

In July 2026 the EU published the "Digital Omnibus on AI". It pushes the AI Act's stand-alone (Annex III) high-risk obligations from 2 August 2026 to 2 December 2027. Some teams read that as "logging can wait". Wrong lesson. Most infra agents probably aren't Annex III high-risk anyway. But the first time an agent changes something in prod, your security reviewer, your ISO 27001 or SOC 2 auditor, or your own incident review will ask the same questions. Regulation or not. Here's the uncomfortable bit: logs are not an audit trail. Tokens, latency and a stack trace are observability. Audit is being able to prove, after the fact, who did what, as whom, on whose authority, and that nobody edited the record since. The questions you will actually get, per agent action: 1. Who asked for it, and who authorised it? 2. Which identity did it run as? Service account, not "the agent". 3. Which model, which version, which prompt and policy version? 4. What did it see? Inputs and retrieved docs, by reference or hash, not necessarily the content. 5. Which tool calls, with which arguments? 6. What actually changed, on which target? 7. Who approved the human-in-the-loop step, and when? 8. Can you rebuild the whole chain from one correlation ID: gateway, agent, MCP/tool, database? 9. How do you know the record wasn't edited or trimmed? 10. How long do you keep it, and does that clash with your GDPR retention? ILLUSTRATIVE example, invented org: ExampleCo's change-bot scales payments-api from 3 to 6 replicas. Their logs show a 200 from the Kubernetes API and a token count. They can't show that Bob approved it, which prompt version was loaded, or that the agent's own service account couldn't have deleted the log line. That's a vibe, not an audit trail. The traps I keep seeing: Logging full prompts and tool output "for audit", turning your log store into an unmanaged PII copy. Sampling. Fine for traces, fatal for audit. You don't get to sample which deletions you can prove. Logs writable by the same identity the agent runs as.

Every routing slide says the same thing: local model for the bulk, frontier API for the hard 5%. Nice slide. Here is what it is missing: nobody on that slide measured the 5%. If you cannot say how you know a request belongs to the hard 5%, you do not have a router. You have a hope with a YAML file. Three ways teams actually decide where a request goes, most reliable first: 1. Route by task class. Summarize a ticket, classify an alert, extract fields from an invoice: local. Plan a migration across 40 services: frontier. You decide per workflow, in advance, with an eval behind it. Boring, auditable, and it captures most of the savings. 2. Cascade with a verifier. Try local first, then check the result with something that is not the model's opinion of itself: the JSON schema validates, the Terraform plan parses, the unit test passes, the tool call returned 200. Escalate on failure. Great when output is checkable, weak when it is prose. 3. Learned router. A classifier predicts which model a prompt needs. Published research exists and the headline savings look great, but they were measured on someone else's traffic and someone else's model pair. Your router is stale the day either model changes. The math that decides whether to bother (ILLUSTRATIVE numbers, not anyone's price list, as of 2026-10-12): a frontier call averages $0.010 for you. Your local stack costs $3,000 a month fixed (hardware amortization, power, hosting, the person who keeps it alive) and almost nothing per request. - 20% escalation: break-even is about 380,000 requests a month. Below that, all-frontier is cheaper and you should stop reading routing posts. - At 1M requests a month: 5% escalation is about $3.6k vs $10k all-frontier. 30% is about $6.1k. At about 70% you built an expensive way to call the API. Two things the cost dashboard will never show you: The cascade tax. An escalated request pays for the local attempt AND the frontier call. The silent miss. A hard request wrongly kept local returns a plausible wrong answer. Nothing errors, nothing alerts. Only a graded sample catches it.

Your model weights are code. Inside an air gap, nobody is watching them load. Most "we run AI on-prem, nothing leaves the building" projects I read about fail the first real security review for one boring reason: the stack was built with internet access and then "disconnected" at the end. The gap hides the problem until the first restart, when the server hangs trying to reach a hub that no longer answers. Three things bite almost everyone: 1. Silent phone-homes. A model server resolves a model name by calling out to a hub. Usage stats, tokenizer fetches, embedding-model downloads, update checks in the web UI, tracing SaaS in the agent framework. Set the offline switches (HF_HUB_OFFLINE=1, TRANSFORMERS_OFFLINE=1, VLLM_NO_USAGE_STATS=1, DO_NOT_TRACK=1), load from local paths, and then do not trust the config. Enforce it at the network layer with deny-all egress. App-level flags are a hope, nftables is a fact. (Variable names as of Oct 2026 for the common stacks. Check them against your versions.) 2. Weights are an executable supply chain. Pickle-based checkpoint formats (.bin, .pt, .pkl) can run code when loaded. Security researchers have documented malicious models on public hubs. Inside the gap this is worse, not better, because your monitoring is thinner and your blast radius is "the box that has the GPU and the data". Rule: safetensors only, never trust_remote_code=True, hash every file at intake, verify the hash again on the inside, keep a provenance record. 3. Agents bring tools, and tools bring dependencies. An MCP server that calls a SaaS API, a Python package with telemetry, a container that pulls at start. Each tool needs its own least-privilege identity and its own allowlist entry. If it is not on the list, it does not run. The workflow that holds up is boring and I recommend it: intake zone outside the gap (download, reject unsafe formats, hash, scan, write a manifest), a transfer step with a second person, a verify step inside, then promotion into an internal registry. Serving images get built outside, saved as a tarball, loaded inside. Nothing pulls anything, ever.