I have twice told readers to self-host open-weight models inside a controlled VPC when the data legally cannot leave the perimeter. I caveated that advice on capability and never once on infrastructure, and near-frontier open weights have made the omission expensive.
This is a correction. In Design AI Inference for Model Disappearance I wrote that the pattern for perimeter-bound data is to run an open-weight model on inference instances inside a controlled VPC: no internet egress, private subnets, endpoint policies that deny anything you did not explicitly allow. A week earlier, in Model Selection Is Capacity Planning, I put the same thing in a routing table as the Tier 2 answer for anything touching money, health, or legal exposure. Both times my only stated caveat was capability, that open weights at a given size will not match a frontier model on the hardest tasks. Both times I said nothing about what it takes to actually run the weights.
I run security and DevOps for a fintech platform serving 1,500+ financial institutions, so I get to watch how advice like that lands. When an architect reads "self-host open weights in a controlled VPC," what they hear is an instance family and a Terraform module. That was a fair reading of what I wrote. It is not a fair reading of the hardware, and it was never a fair reading of who ends up owning model risk.
The floor for a near-frontier open model is a liquid-cooled rack, not an instance type
Moonshot AI put the Kimi K3 weights on Hugging Face in late July. The repository is a useful reality check because you can measure it yourself rather than argue about it: 2.8 trillion total parameters with roughly 104 billion activated per token, natively quantized to MXFP4, shipped as 96 safetensors shards totaling about 1.56 terabytes, under a custom Kimi K3 License (opens in new tab) rather than Apache or MIT.
The number that should have been in my earlier posts is in the deployment recipe that Moonshot's own model card points readers to. The official vLLM recipe for Kimi K3 (opens in new tab) states the prerequisite in one line: "At least 8x GB300. Multi-node for real production traffic." The AMD path is the same shape, at least eight MI355X or MI350X.
Read that as a procurement document instead of a README. GB300 is NVIDIA's Blackwell Ultra part, and it ships in rack-scale, fully liquid-cooled configurations. Eight of them is not an instance you launch on a Tuesday. It is a slice of a rack you must already have bought, powered, plumbed, and staffed. "Multi-node for real production traffic" is carrying even more weight, because multi-node means an interconnect fabric, NVLink domains or RDMA over InfiniBand, which is not something you retrofit into an existing private subnet because a data-classification policy told you to.
None of that appears in either post I wrote. My caveat was that the open model would be somewhat less capable. The real caveat is that a near-frontier open model is a data-center program: accelerators on allocation, high-bandwidth memory, an interconnect, cooling loops, and operators who can debug expert-parallel scheduling at three in the morning. For a platform team of a handful of people, that is not a backlog item, it is a capital plan with a hiring plan attached.
The software surface is its own review. The K3 repository ships custom modeling and tokenizer code alongside the tensors, which means loading it the ordinary way requires trust_remote_code, and the recommended serving paths are pinned to specific engine builds and all-to-all communication backends. You are not deploying a file. You are onboarding a third-party Python runtime into your own inference path, into the same VPC you built to keep the data in.
The narrow version of my advice survives, and I should say exactly how narrow. The model I actually named both times was Gemma 4, which does fit on a single multi-GPU node, and for classification, extraction, redaction, and retrieval over regulated records that deployment is often the right answer. What does not survive is the rule I wrote around it, the implied generality that "self-host open weights" answers residency at whatever capability tier the business asks for. Between the model that fits on one node and the one that needs a rack sits a gap I told readers to accept on quality grounds and never priced on infrastructure grounds.
Downloading the weights makes you the model provider
The chip count is the part that gets attention, and it is the less important half. The important half does not go away even if you win the capital argument, because the moment you pull the weights you stop being a customer of a model provider and become one.
Read the license before the architecture diagram. Section 5 of the Kimi K3 License is the standard grant of nothing: the software and any output are provided "ON AN 'AS IS' BASIS, WITHOUT WARRANTY OF ANY KIND," and "IN NO EVENT SHALL MOONSHOT AI OR ITS AFFILIATES OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY." That is not a criticism of Moonshot, it is what every open-weight license says. It is also the entire content of your model-provider relationship. There is no data processing agreement, because no data goes to them. There is no availability commitment, no support tier, no named security contact, no notification obligation, and no counterparty to sit across from at a renewal.
Notice how much of a closed-provider relationship is compensating control you never inventoried. Every AI vendor review I sit in spends time on the provider's certifications, its breach notification clock, and its incident contacts, and then treats those as evidence the model layer is covered. Delete the provider and those rows do not become someone else's problem. They become empty.
Then there is patching, which is where the asymmetry is sharpest. On a hosted API, a jailbreak class or a data-handling defect gets fixed upstream and you inherit the fix without a change ticket. A checkpoint on your disk is frozen the day you download it. Weights do not have a CVE feed, a version-bump advisory, or a coordinated disclosure process pointed at your inventory. If a behavioral defect surfaces in that specific checkpoint, remediation is your engineering project, on your calendar, validated against your own evaluation suite, and there is nobody to escalate to. I have never seen an AI vendor review that asked what the patch path is for a model artifact the company holds itself, and I have sat in a lot of them.
Regulators already reason about this in provider terms. The EU AI Act's Article 25 treats a deployer or other third party as a provider of a high-risk system once it puts its own name on that system or makes a substantial modification to it. The Commission's July 2025 guidelines on general-purpose AI models extend the logic downstream with an indicative criterion: spend more training compute on your modification than a third of the original model's training compute and you are considered the provider of the modified model. A model trained above 10^25 floating-point operations is presumed to carry systemic risk, and the open-source exemption in Article 53 explicitly does not reach those. Fine-tune a near-frontier open checkpoint on your own corpus and the paperwork does not shrink because the weights were free, it concentrates.
Your model-risk guidance just told you it does not cover this
US banking supervisors moved on this in April, and the move cuts against anyone hoping an existing framework absorbs the burden. OCC Bulletin 2026-13 (opens in new tab), issued April 17, 2026 by the OCC, the Federal Reserve Board, and the FDIC jointly, replaces the 2011 model risk management guidance everyone in this industry still calls SR 11-7. It sets a risk-based approach, aims primarily at organizations above 30 billion dollars in assets, and keeps a section on vendor and other third-party products holding that even where you cannot see the underlying code or development data, "the principles of model risk management remain applicable."
Then it says this: "Generative AI and agentic AI models are novel and rapidly evolving. As such, they are not within the scope of this guidance." The agencies signaled a request for information on AI to follow.
I do not read that as permission, I read it as the two clearest sentences a regulated buyer has been handed all year. The generic framework will not tell you how to validate a 2.8-trillion-parameter checkpoint you downloaded, and the vendor section cannot help either, because its whole premise is that there is a vendor. What is left is your own program: your inventory entry, your validation evidence, your monitoring, your documented limitations, your accountable owner. If an examiner asks who validated this model and who answers when it is wrong, "it is open weights, we run it ourselves" puts every one of those names inside your building.
The realistic residency path is a third-party host you never screened
Almost nobody clears the rack. So the practical path, and the one I watch teams take, is to rent the weights from somebody who did. That is where the residency argument quietly inverts.
Within days of the weights landing, OpenRouter listed nine endpoints for the same model (opens in new tab) across eight hosting companies, eight of the nine at an identical list price of three dollars per million input tokens and fifteen per million output. Moonshot's own endpoint matches that price, with cached input at thirty cents. Renting is cheap and instantaneous, which is exactly why it will be the decision.
Look at what you actually bought, though. Those endpoints report different quantizations for the same weights: some MXFP4, some FP8, some FP4, and several report nothing at all. Two hosts serving "Kimi K3" are not serving the same artifact, and a quantization change is a model change that your evaluation set, your calibration testing, and your prompt-injection suite have to re-clear. This is the same failure mode as the context custody problem, arriving from the opposite direction: there the vendor held your accumulated context, here a vendor you never diligenced holds the inference itself.
And the sentence in the policy still reads "we self-host open weights for data residency," while the traffic goes to a company nobody put through third-party risk review, under terms nobody negotiated, in a region nobody confirmed. That is not a control. It is an assertion with a receipt from a different vendor.
The amended rule, and what belongs in the classification policy
Here is the version of my own advice I would defend now. Self-hosting open weights is a legitimate residency control at the model sizes that fit inside a single node you already own. Above that size it becomes a hardware program, and the residency claim has to be tested against the hardware bill before it is written into a data-classification policy. Monday, open the data-classification policy and run every model it names through this test.
- Size the model against the rack, not the roadmap. Before "self-host" enters a policy, name the accelerator, the count, the node topology, the interconnect, the power envelope, and the two people who operate it. If any cell is blank, the control does not exist yet.
- Write down who the provider is. One line per model: provider of record, contractual assurance, security contact, notification path. Downloaded weights read "us, none, us, none," and that line is what makes the tradeoff visible to a risk committee.
- Pin the artifact, not the model name. Record the repository revision hash and the quantization, not "Kimi K3." Treat a quantization or revision change as a re-validation trigger, exactly as you would a floating container tag.
- Budget the patch as your own engineering work. Give every self-held checkpoint a named owner, a re-evaluation cadence, and a documented rollback target, because no upstream advisory is coming.
- If a host serves the weights, run the full third-party program on the host. Region, subprocessors, retention, logging, incident notification, and a data processing agreement. Open weights change nothing about that obligation except which company's name goes on the paperwork.
- Decide the fallback in advance. For each perimeter-bound data class, write what happens if neither the rack nor the host clears review: the work stays human, or it runs on a smaller model with narrower scope. Both are legitimate. Silence is not.
What I should have written the first two times is that open weights solve custody of the weights and nothing else. They do not solve custody of the compute, and they remove the accountable counterparty a closed API quietly supplied. Both earlier posts treated the model provider as a dependency to be routed around. That was right about availability and incomplete about accountability, because a provider is also the party who patches, who answers, and who is liable. Delete the dependency and you inherit all three jobs.
Downloadable is not deployable, and deployable is not governed. If you have an in-VPC open-weight deployment in production today, I would genuinely like to know which side of that line it fell on: did you clear the hardware floor in your own racks, or did the residency story end at a hosting provider's endpoint that never went through vendor review?