A developer spent a week convinced he’d uncovered a fundamental architectural flaw in Google’s Gemma 4 Dense model. He hadn’t. The culprit was four digits in a config file — and the Gemma 4 token limit of 400 was quietly killing responses before they could form.
- A 400-token Gemma 4 token limit was silently starving the model’s reasoning layer before it could respond correctly.
- Raising the Gemma 4 token limit to 4096 resolved every false refusal across all six test scenarios instantly.
- The assumed MoE-vs-Dense architecture gap largely vanished once both models had room to complete their reasoning.
- Developer Ali Mafana publicly retracted his original architecture-mediated failure claim after community testing exposed the real cause.
The Original Claim: Architecture Was to Blame
Ali Mafana, a developer building an Arabic e-commerce chat router, published findings last week comparing Gemma 4’s two flagship variants: the 26B Mixture-of-Experts model and the 31B Dense model. His test setup was an Arabic-first system prompt, temperature capped at 0.3, and a max_tokens floor of 400. Under these conditions, the MoE variant improved — flipping from stalled, useless replies to grounded, accurate answers. The Dense model went the other direction. It started issuing false refusals: telling customers that products didn’t exist when the catalog data proving otherwise was sitting right there in its context window.
Mafana called this divergence “architecture-mediated” and published accordingly. The framing was plausible. MoE and Dense models do process information differently, and it’s not unreasonable to expect them to respond differently to the same prompt constraints. The article got traction. It also got pushback — the useful kind.
How Community Scrutiny Reframed the Problem
Robin Converse, an engineer at Triava Labs running the same Gemma 4 model family on a self-hosted Ollama stack, ran her own sweep with no token cap applied. Same six scenarios. Three temperature settings. Unrestricted max_tokens. Her MoE handled every single case correctly. She published the full methodology and an 18-call breakdown, then asked the pointed question: what does the same test look like on the managed Gemini API side with the cap removed?
Converse also demonstrated the kind of intellectual honesty that’s increasingly rare in public technical discourse. She had independently filed two upstream Ollama bug reports. When maintainers clarified that one — issue #15288 — was a configuration issue rather than a genuine bug, she walked her framing back publicly. The second, #15428, was confirmed by multiple users and fixed in a later release. That willingness to be a fair witness to her own errors is exactly what gave Mafana confidence in her hypothesis.
The hypothesis itself was precise: the Gemma 4 token limit of 400 was cutting off the model’s internal reasoning chain before it had finished working, leaving only a truncated — and wrong — visible reply. Vic Chen, commenting on the same thread, put it cleanly: capability ceiling and orchestration pressure look identical from the outside. Only one of them is fixed by giving the model more room to think. When the Gemma 4 token limit is set too low, every response that requires multi-step retrieval is a candidate for this kind of silent failure.
Vadym Arnaut contributed a separate lens, mapping a substitution-vs-decision boundary that helped sharpen where to look. Edwin Realpe Preciado, Mykola Kondratiuk, Theo Valmis, and Hashevolution each pushed structural reframes that Mafana had to genuinely engage with rather than dismiss. The collective read from the thread: the cap is one variable, the architecture is a category, and you’ve been conflating the output of one with the nature of the other.
The Gemma 4 Token Limit Fix Was Four Lines of Code
Mafana re-ran the original six scenarios with a single variable changed. He added an environment flag to his chat-models server file — GEMMA_UNCAPPED=1 — that swapped the hard 400-token floor for a 4096-token budget. No prompt edits. No router changes. No temperature adjustments. No new model rules. Just the Gemma 4 token limit, raised.
The results were unambiguous. All 12 calls across both architectures succeeded. The Dense model’s HTTP 500 error rate dropped from 2-in-6 to zero. The MoE’s rate was also zero. The reliability gap that had anchored an entire article disappeared in a single test run.
The headline failure from the original piece — Scenario 2, where a customer asked for a white shirt in size L — illustrates just how dramatic the difference was. Under the 400-token Gemma 4 token limit, the Dense model responded with what translates to: “We don’t currently have a white shirt in size L. My apologies, nothing of that model is available right now.” The products were in the context. The model simply ran out of tokens before it could surface them.
With the Gemma 4 token limit raised to 4096, the same model, same prompt, same temperature returned: “My pleasure! We have Urban Cool Striped Shirt for $65, Bordeaux Heritage Shirt for $80, and Urban Stripes Classic Shirt for $95, all available in white in size L. I recommend pairing the shirt with Chinos for a clean, polished look!” Three real SKUs. Real prices. A styling cross-sell. One config change.
What the Numbers Actually Say About MoE vs Dense
There’s a secondary finding here that deserves attention independent of the correction. Once both variants had adequate token budget, the Dense model was consistently faster than the MoE variant — 19–38 seconds per call with an average of 27 seconds, versus 28–56 seconds for MoE averaging 37 seconds. That’s a meaningful latency advantage, and it runs counter to what the original article implied.
The architectural difference between MoE and Dense didn’t vanish entirely. Mafana’s revised read is that it’s quantitative rather than qualitative — both models perform multi-step reasoning on this kind of retrieval-augmented task, but they consume different amounts of token budget to do it. MoE appears to need more headroom. Dense is more efficient in that specific sense. But the failure modes, when they appear, are the same: the model runs out of space before the output is complete. Understanding where your Gemma 4 token limit sits relative to each variant’s reasoning appetite is therefore a practical tuning decision, not a theoretical one.
That’s a meaningful distinction. Calling a budget starvation problem an architectural flaw implies the solution requires a different model. Calling it what it actually is — a configuration problem — means the fix is an env var and a restart.
Why This Matters Beyond One Developer’s Config File
The Gemma 4 token limit story is a small case study in a larger problem that’s going to keep appearing as more teams deploy capable open-weight models in production: the gap between a model’s actual capability and what it appears capable of under constrained infrastructure settings. A model that refuses to answer because it ran out of tokens looks, from the outside, exactly like a model that can’t answer. The logs don’t obviously tell you which it is.
This is especially consequential as teams move toward retrieval-augmented generation architectures — like the Arabic e-commerce router Mafana built — where the model is expected to reason over injected context before producing a response. That reasoning takes tokens. If your max_tokens ceiling is set conservatively for cost or latency reasons, you may be cutting off the reasoning chain midway and getting the worst of both worlds: the latency cost of the reasoning pass, plus a garbage output at the end of it. Auditing the Gemma 4 token limit should be one of the first steps any team takes before concluding that a model is fundamentally broken.
Google’s Gemma 4 family — and the broader wave of capable open-weight models from Mistral, Meta, and others — are increasingly being deployed by teams who set infrastructure parameters without fully understanding how those parameters interact with the model’s internal reasoning mechanics. That knowledge gap is where false architectural conclusions get born. Mafana’s public correction is a useful reminder that the first explanation that fits isn’t always the right one — and that a config audit should happen before an architecture indictment.
Source: https://dev.to/alimafana/i-raised-gemma-4s-token-cap-the-dense-model-stopped-refusing-895
