Choosing the Right Model: Right-Sizing, Small Models, Routing and Cascades

Chapter 11 closed Part III with decisions that hold under either reading of the debate. This chapter opens Part IV, E3 Design, with the second lever of the optimisation hierarchy, sufficiency of model: the smallest model, with the lowest reasoning effort and the shortest context, that meets the requirement of the use. Chapter 4 placed it after necessity, because a use that should not exist needs no model, and before efficient serving, because a service that runs the wrong model efficiently is still running the wrong model. It teaches how to write the requirement first, what small models save and at what quality cost, why size is a poor guide to energy, how routing and cascades work and what drives their footprint, and how to record the choice for review.

In most uses a much smaller model meets the requirement, nobody wrote the requirement down, and the largest available model was chosen because it was available. Writing the requirement first turns the choice into a measurement. Where one model cannot meet it on every request, a cascade (Figure 12.1) lets a small model answer what it can and escalates the rest; the share that escalates, the escalation rate, drives the footprint.

A cascade: a policy point at the gateway, a small model first, an independent check, escalation to a large model, the escalation rate metered and the budget and stop path outside the model

Figure 12.1: A cascade. The policy point, the check, the meter and the stop path sit outside the models they govern. No figure is carried by the diagram.

The biggest model by default

A scene from the Region, illustrative. In 2032 its registry wants incoming letters sent to the right unit (benefits, planning, social services: 31 units in all). The platform team already runs an open-weight generative model of about 70 billion parameters for the staff assistant (FPR-NR-0003) behind the gateway. The first prototype of the correspondence classifier therefore calls that model with a prompt that lists the units and asks for the best match. It works, is accepted in a week, and nobody asks whether it is the right size.

Three things made the large model the default, and none was a decision. It was available; its quality on a handful of examples was good; and its cost was invisible to the person choosing it, because the energy of a request is on no invoice unless someone meters it (Chapter 6). AI Platform and Security, Chapter 22 named the pattern in cost terms: most requests are simple, a small model answers them as well as a large one, and a routing policy sends each request to the cheapest model that meets the quality objective (27, Nordqvist, 2026). This chapter takes the pattern in energy terms and adds a way to ask, before the choice, what the smallest adequate model is. A frontier-scale model is the right choice for open-ended drafting, for hard reasoning over long documents and for the request no small model handles; the point is that the choice is made per use, against a stated requirement (Chapter 6).

Writing the requirement first

A requirement is the written statement of what the use must achieve, fixed before any candidate is run. It has five parts, each answering a question that otherwise gets answered by accident.

The task and its functional unit. What the model does for one item, in the same unit as the footprint record: an item routed, a query answered, a letter drafted. Chapter 6 chose functional units for the footprint; the requirement uses the same one, so that quality and energy are measured on the same thing.

The quality threshold and the measure that decides it. The number and the instrument. For a classifier, accuracy on a held-out set of labelled letters, with a limit on the error that matters most (a letter with a legal deadline sent to the wrong unit). For an assistant that answers from policy wordings, the share of answers that are correct and cite the wording, judged by specialists. The threshold is set before the candidates run, by a person who owns the outcome, and the test set is never used to tune a candidate. This is the series' independence rule, that what verifies a result must be independent of what produced it, and AI Developer, Chapter 9 states the corollary for software, that the test oracle must be independent of the generator (36, Nordqvist, 2026).

Latency. A letter routed overnight needs no answer in two seconds; a handler waiting for a clause does. Latency decides whether a second pass fits.

Languages and data. The languages the use must handle and where data may go. Nordvik Region's letters contain personal data and stay in its own data centre, which removes every hosted candidate. Language matters for energy too: European languages use roughly 1.2 to 1.5 tokens per word and Chinese, Japanese and Korean more, so the same content costs a different number of tokens (129, UNESCO, 2025).

The settings that are part of the model choice. The reasoning effort, the longest context and the longest answer. These are not tuning details: Chapter 6 showed that reasoning, output length and context are among the largest multipliers of a use's footprint, and a requirement that does not state them leaves the largest lever to a default. For routing letters: reasoning off, a context of 512 tokens, an answer of one unit code.

With a threshold and a test set any model can be tested, and the smallest that passes is the answer.

What small and task-specific models save

The best-known public evidence is the report Smarter, Smaller, Stronger by UNESCO and University College London, published on 8 July 2025 (129, UNESCO, 2025). Its headline reached the press as small, task-specific models cutting energy use by up to 90 per cent. The press page and the report differ, and a buyer who relays the number needs the baseline, which the report states and the page does not. The report was read in full in November 2026 from a copy of the PDF at unesco.org.uk; the copy on UNESCO's own repository returned an access error.

What was measured. The authors tested one baseline, Meta's Llama 3.1 8B Instruct, an open-weight generative model with about 8 billion parameters, on three tasks: summarising news articles, translating between English and other languages, and answering questions from science textbooks. Energy was estimated with CodeCarbon, an open-source monitoring tool, and quality was scored with BERTScore, a measure of how close a generated text is in meaning to a human-written reference. The experiments ran on one consumer accelerator, an NVIDIA GeForce RTX 3090 Ti with 24 GB of memory (stated in the appendix), not a data-centre part; the passages read state no batch profile, and the energy boundary is that of the tool.

What the small models were. For each task the authors chose popular fine-tuned community models: summarisation of 61 to 570 million parameters, translation of 76 to 78 million and question-answering of 65 to 335 million. Several of the last are encoders that extract an answer and do not generate text, so the saving mixes size, design and output length.

The result. On datasets drawn from the same distribution as the fine-tuning data, the task-specific models used 15 to 50 times less energy than the unquantised baseline and produced higher-quality outputs by BERTScore. On a slightly varied dataset of the same task their performance fell, and the report adds that they suit multi-tasking applications less well. It concludes that a collection of small models, with a large general model where needed, gives the best balance. Fifteen to fifty times is a reduction of about 93 to 98 per cent; the up-to-90-per-cent figure of the press page matches the report's scenario calculation, not its model comparison: for a concept-explanation prompt, with an assumed 1 billion daily requests of which 35 per cent are of that type, smaller models with shorter responses cut the equivalent daily electricity from about 38,000 UK households to under 4,000. The same report puts the saving from halving the response length at 54 per cent, from a halved prompt at 5 per cent, and from three quantisation methods at 22, 35 and 44 per cent (Chapter 13). Its figure of 0.34 watt-hours per prompt is quoted from a provider's chief executive without a stated boundary, the kind Chapter 6 said cannot be compared with anything (129, UNESCO, 2025).

Read at its boundary, the report supports a narrower claim than the headline and a more useful one. For a defined task with fine-tuning data and a test set from the same source, fine-tuned models one fourteenth to one hundred and thirtieth of the size of a general 8 billion parameter model met or beat it on a similarity score at a fifteenth to a fiftieth of the energy, on one baseline, one tool, one consumer accelerator and one set of open models, with a measured loss of quality outside the task. It does not say that a small model is as good for any task, that 90 per cent applies to a given organisation, or that the saving holds on production hardware.

Luccioni, Jernite and Strubell (FAccT 2024) measured the energy of 1,000 inferences on representative datasets for many categories of model and found multi-purpose generative architectures orders of magnitude more expensive than task-specific systems for a variety of tasks, even when controlling for the number of parameters (130, Luccioni et al., 2024). The price of the saving is generality and the work of building the model: a fine-tuned classifier needs labelled data, must be re-tested when inputs change and has its own training footprint (Chapter 14).

Size is not a proxy for energy

The easiest way to choose by footprint would be by parameter count, and the evidence says that is not safe. Three results make the point.

First, the AI Energy Score v2 of Hugging Face (4 December 2025, read in November 2026) compared a new cohort of models with a reference model of similar active size from its February 2025 leaderboard, excluding reasoning and mixture-of-experts models (109, Luccioni & Gamazaychikov, 2025). Of the 14 models that met the criteria, 8 used the same or more GPU energy than the similar-sized model of nine months earlier; the range ran from 3 per cent of the energy to almost twice as much. The boundary is GPU energy per 1,000 queries on standardised datasets, measured with CodeCarbon, mostly for open models: independent, but the accelerator only.

Second, reasoning changed what size predicts. The same post reports that reasoning models use on average 30 times the energy of the same models without reasoning, 150 to 700 times for the same model with reasoning switched on, driven by 300 to 800 times more output tokens. Chapter 6 gave the cases in its Table 6.3; the point here is that a 3 billion parameter model with reasoning on (12,791 Wh per 1,000 queries, GPU energy) used far more than a 70 billion parameter model with it off (49.5 Wh), so a smaller model is not a cheaper one unless its settings are chosen as well (109, Luccioni & Gamazaychikov, 2025).

Third, the result of Luccioni and colleagues holds with parameters controlled (130, Luccioni et al., 2024).

The count that matters is the active parameters per token, which differs from the total in mixture-of-experts designs, and the energy per task depends on how many tokens a model writes to finish it. The working rule is to choose by the measured energy per functional unit on your own workload and hardware, at the settings of the requirement, and to use public benchmarks to shortlist, not to decide. The AI Energy Score suits shortlisting because it is independent and uses one method and one accelerator type; its documentation states that only GPU energy is scored and that proprietary models are submitted as a container with an attestation by the uploader, so a proprietary entry is a claim (133, Hugging Face, 2026).

Reasoning effort and context as model choices

Reasoning effort belongs in the model choice, not after it. Many recent models offer a switch or several levels, and a deployer who accepts the default has chosen by omission. The AI Energy Score v2 reports that the GPT-OSS models differ by 4.8 times between their high and low reasoning modes in the 20 billion parameter class and by 1.6 times in the 120 billion class, on its GPU-energy boundary, without reporting quality for the comparison (109, Luccioni & Gamazaychikov, 2025). The requirement states the lowest effort that meets it; candidates are tested at that effort and one above; and if the higher effort is needed only for a tail of hard requests, the tail goes to a cascade. A context limit works likewise. Chapter 13 treats effort, context, caching and quantisation as serving levers; here the settings are written into the requirement and the record, so that they are chosen once, on purpose.

Routing and cascades

Two patterns let several models serve one use. A router decides, before any model runs, which model should take the request: by the class of the request (a legal appeal always goes to a person), by a rule on its length or language, or by a small learned classifier. A cascade tries the small model first, checks the result, and escalates to a larger model only when the check fails. They combine: a router can send some classes straight to the large model and the rest into a cascade. Both belong in the gateway, the policy point that already meters and limits calls (AI Platform and Security, Chapter 22 (27, Nordqvist, 2026)), so the choice is made by code the model cannot change.

What the research reports. FrugalGPT, by Chen, Zaharia and Zou, proposed a learned cascade over commercial models and reported, on its benchmark tasks in 2023, that it could match the best single model, GPT-4, with up to 98 per cent lower cost, or improve accuracy over GPT-4 by 4 per cent at the same cost (131, Chen et al., 2023). Soiffer, Kolawole and Smith used agreement between several outputs of smaller models as a training-free signal for deferral and reported, across models from 500 million to 70 billion parameters, cascades that matched or surpassed the target model's quality at 40 per cent of the cost with up to 60 per cent lower latency (132, Soiffer et al., 2025). Both measured monetary cost and latency, not energy. Cost and energy often move together, but a provider's price is not its energy, and a cascade adds work that a price may hide. The sources read give no measured energy result for a cascade in production. The mechanism is established for cost and plausible for energy, so an organisation's own measurement at its own boundary is the evidence.

The arithmetic of a cascade. Let e_s be the energy of one answer by the small model, e_c the energy of the check (a second sample, a retrieval score, a citation test), e_l the energy of the large model, and r the escalation rate, the share of requests that the check sends upward. The expected energy per request is

E = e_s + e_c + r × e_l

and the cascade saves energy against always using the large model only while E < e_l, that is while r < (e_l − e_s − e_c) / e_l. Three consequences follow. The escalation rate drives the footprint, not the size of the small model: a cheap model with a poor check escalates most requests and costs more than the large model alone. The check is paid on every request, so one that runs the small model again is expensive and one that reads a score the retrieval step already computed is nearly free. And the escalation rate is a property of the traffic, metered continuously, not assessed once.

A second failure is one of quality, not energy. A small model can be confidently wrong, and a cascade that escalates only when the small model is unsure passes confident errors through. The check must be independent of the model it checks: a retrieval score from the search step, a test that the answer cites a passage that exists in the retrieved text, agreement between samples. None is perfect, so the cascade's quality is measured on the independent test set. An escalated request pays for two models in sequence, so the slow tail is the escalated tail.

Practical guide: a model-choice protocol

Purpose. Choose the smallest model that meets a written requirement, with measured quality and measured energy, and record the choice so that it can be reviewed.

Roles and inputs. The use owner signs the requirement; the platform engineer runs the candidates and holds the gateway meter; the sustainability lead reviews and is not the owner; specialists label the test set. Inputs are the footprint record of the use (Chapter 6), the measurement plan (Chapter 8) and a labelled test set that no candidate has been tuned on.

Steps.

  1. Write the requirement: task and functional unit, quality threshold and measure, latency, languages and data constraints, and the settings (reasoning effort, longest context, longest answer). The owner signs it before any candidate runs.
  2. Choose three candidates of different kinds: the default model, a smaller general model, and a task-specific or fine-tuned model where labelled data exists. All must satisfy the data constraints.
  3. Run the quality test on the independent test set and a second test on a shifted set (new forms, new wording, other languages). Tune on one set and decide on another.
  4. Run the energy test on your own hardware and at the production batch profile, with the same functional unit and the boundary of the footprint record. Record the boundary, the date and the statistic.
  5. Decide: the smallest candidate that passes both tests. If none passes alone, test a cascade, measuring the escalation rate on the test set and the errors the check lets through.
  6. Set the review triggers (a new form, a volume change, a much lower independent energy figure, a change of licence or supplier) and the age of the evidence: 90 days for the measured energy and quality of a running use (the illustrative convention of Chapter 4).
  7. Record the decision as an AI Decision Record and the choice as a model-choice record MCR-*, linked from the footprint record, and set an environmental budget (below).

Result. One model-choice record per use, with the requirement, the candidates, the numbers, the decision and its expiry.

Worked example (illustrative): correspondence routing at Nordvik Region

The Region's correspondence classifier (FPR-NR-0005; AIDR-0042, UC-2026-017) routes about 120,000 items a month. The platform team, with the sustainability lead Astrid Nyberg as reviewer, wrote the requirement in October 2032 and tested three candidates, recorded as MCR-NR-0001.

# model-choice/MCR-NR-0001.yaml (illustrative, Nordvik Region)
record: MCR-NR-0001
use: FPR-NR-0005                  # correspondence classifier (items routed)
decision: AIDR-1312
owner: maja-lindqvist
reviewer: astrid-nyberg           # sustainability lead, not the owner
requirement:                      # written and signed before any candidate was run
  task: route an incoming letter to one of 31 units
  functional_unit: item routed
  quality:
    held_out_accuracy_min: 0.94   # 2,000 letters labelled by registry staff
    shifted_set_accuracy_min: 0.90  # 400 letters on a form introduced in 2032
    deadline_case_misroute_max: 0.005
  latency_p95_s_max: 3.0
  languages: [sv, fi, en]
  data: personal data; stays in nordvik-dc
  abstain_below_confidence: 0.80  # such items go to registry staff
candidates:                       # energy: node boundary, reserved pool, 2032-10, measured
  - {id: general-70b, kind: open-weight generative, accuracy: 0.953, shifted: 0.948, p95_s: 2.4, wh_per_item: 0.80}
  - {id: general-8b, kind: open-weight generative, accuracy: 0.917, shifted: 0.903, p95_s: 0.9, wh_per_item: 0.12}
  - {id: tuned-110m, kind: fine-tuned encoder classifier, accuracy: 0.956, shifted: 0.901, p95_s: 0.08, wh_per_item: 0.004}
chosen: tuned-110m
reasoning_effort: not applicable  # no generation
context_tokens_max: 512
volume_per_month: 120000
monthly_kwh_node: {general-70b: 96.0, general-8b: 14.4, tuned-110m: 0.48}
measured_on: 2032-10-11
valid_to: 2033-01-09              # 90 days
review_when:
  - a new form or a new unit changes the shifted-set result by more than 3 points
  - volume above 180,000 items a month
  - a candidate with an independent energy figure at least 5 times lower
  - the supplier or the licence of the base model changes

Illustrative. Energy per item is measured at the node of the reserved pool (accelerators and host, before facility overhead), operational, October 2032; it is not comparable with the facility-boundary figures of the footprint records of Chapters 6 and 8.

The 70 billion and the tuned candidates pass the held-out threshold; the 8 billion candidate does not (91.7 per cent against 94.0). On the shifted set the tuned classifier is at 90.1 per cent, one tenth of a point above its threshold and well below the large model's 94.8: the trade-off the UNESCO report describes. The record turns that into a design: items with a confidence below 0.80 go to registry staff (6 per cent of the shifted letters in the test), and the shifted-set test is repeated whenever a new form is introduced. The choice is the 110 million parameter classifier.

The energy result is then put in proportion. At 120,000 items a month the large model would use about 96 kWh at the node boundary and the classifier about 0.48 kWh; at the Region's 32 g CO2e per kWh (location-based, illustrative) and the provider's overhead of 1.21 on whose reserved pool the candidates were tested, the difference is about 3.7 kg CO2e a month. (The classifier itself would run in nordvik-dc at a power usage effectiveness (PUE, the facility's total electricity divided by that of its IT equipment) of about 1.55, about 0.02 kg a month, which leaves the difference at about 3.7 kg.) The ratio is about 200 and the amount close to nothing, so the case for the choice is not the carbon of this one use. It is cost, latency, the capacity left on the pool for the staff assistant (410,000 requests a month, FPR-NR-0003), and the practice: the staff assistant, with more than three times the requests and generative output, is where the same protocol matters for the footprint. The one-off cost: fine-tuning took 14 accelerator-hours, about 8.4 kWh at the node boundary at 0.6 kW per accelerator, paid back against the 8 billion candidate within a month (Chapter 14).

Open-weight and proprietary models

An open-weight model is one whose trained parameters are published for anyone to run; a proprietary model is reached through the provider's service. The choice rests on many grounds (sovereignty, cost, quality, support), and AI Platform and Security, Chapter 10 treats build, buy and adapt (27, Nordqvist, 2026). For the footprint it decides one thing: who can measure.

With an open-weight model on hardware the organisation controls, the organisation can run the energy test above, read the rack and accelerator meters (Chapter 8) and attribute energy to the use. That is a capability, not a saving, and it brings the utilisation risk: a small model on a reserved pool at the Region's average utilisation of 0.58 spends the idle share on every request, and a shared hosted large model may use less energy per request than a dedicated small model on a half-idle accelerator. That is a question of utilisation (Chapter 13) and of the boundary, not of size.

With a proprietary model through a service, the buyer sees a price, a latency and, if lucky, a provider figure on a boundary the provider chose. The AI Act requires providers of general-purpose AI models to keep technical documentation with the information of Annex XI, for the AI Office and the national authorities on request (Article 53(1)(a)) (1, European Union, 2024). Article 53(2) lifts the obligations of points (a) and (b) for providers of models released under a free and open-source licence with public parameters, architecture information and usage information, except for models with systemic risk. Two architectural implications follow, and this is not legal advice. A deployer has no direct duty under these provisions to measure energy and the documentation is directed at the authorities, so a buyer who needs energy per request puts it into the contract (Chapter 23). And an open-weight model does not automatically fall under Article 53(2), because many open-weight licences are not free and open-source licences in the sense of the provision. The candidate list says, per candidate, who can measure its energy and on what boundary; a candidate nobody can measure is scored as unmeasured, not as low.

The model-choice record

The model-choice record MCR-* answers the question a reviewer will ask in six months: why this model, against what requirement, and what would make us change it. It is one decision for one use, small enough to write in an afternoon, beside the footprint record FPR-*; the example above shows the form.

Three fields carry most of its value. The requirement block is signed before the candidates run; a requirement written after the result is a justification. The shifted-set result records how the choice behaves outside the data it was tuned on, the quality cost that benchmarks hide. The review triggers make the choice re-open itself, because models and traffic change. AI Operations, Chapter 13 treats provider changes to a hosted model and the need to re-test after one; a record that names a model by an alias the provider can repoint will not notice (34, Nordqvist, 2026).

The budget and the stop path

A cascade is a design for less, and a design is a claim until it runs under a limit. Every chapter on E3 Design asks what budget applies, how it is measured outside the system it limits and who can stop the work. Hansa Mutual's claims assistant (FPR-HM-0001), which answers about 1.1 million queries a month from retrieval over policy wordings, is the worked case.

Worked example (illustrative): the claims assistant's cascade at Hansa Mutual

Anja Weiß, head of the group AI platform, owns the routing and carbon policy RP-HM-CLAIMS-2026-02, which the group revised in October 2032 to put a small model first. The small model is an 8 billion parameter class model and the large one a 70 billion parameter class model, both self-hosted in Frankfurt. The policy is a file that the gateway reads.

# routing/RP-HM-CLAIMS-2026-02.yaml, revision 3 (illustrative, Hansa Mutual)
policy: RP-HM-CLAIMS-2026-02
revision: 3
use: FPR-HM-0001            # claims assistant, retrieval over policy wordings
effective: 2032-11-02
owner: anja-weiss           # head of the group AI platform; not the service user
allowed_models: [claims-small-8b, claims-large-70b]   # the only models the route may reach
first_try: claims-small-8b
escalate_to: claims-large-70b
escalate_if_any:            # evaluated by the gateway, not by the model
  retrieval_score_below: 0.62
  cited_wording_missing: true
  samples_disagree: true    # two small-model samples differ in meaning (agreement check)
never_answer_alone:         # handled by a person, never by either model
  - appeal_or_complaint
  - legal_dispute
energy_wh_per_query:        # measured MCR-HM-0001, node boundary, October 2032
  claims-small-8b: 0.30
  claims-large-70b: 1.80
  second_sample: 0.30        # the agreement check runs the small model twice
expected_escalation_rate: 0.22
alert_escalation_rate: 0.35      # the router is frozen at the last good revision above this
freeze_escalation_rate: 0.50

A short script (Python 3.12, PyYAML) reads the policy, routes a query from its signals and computes the expected energy for any escalation rate, so that the arithmetic above can be run and read by a reviewer.

"""Check a cascade policy: route one query and bound the expected energy."""
import sys
import yaml

def load(path):
    with open(path, encoding="utf-8") as f:
        return yaml.safe_load(f)

def route(p, signals):
    if signals["category"] in p["never_answer_alone"]:
        return "handler"
    escalate = (signals["retrieval_score"] < p["escalate_if_any"]["retrieval_score_below"]
                or (p["escalate_if_any"]["cited_wording_missing"] and not signals["cited"])
                or (p["escalate_if_any"]["samples_disagree"] and signals["disagree"]))
    model = p["escalate_to"] if escalate else p["first_try"]
    assert model in p["allowed_models"], "model outside the allow-list"
    return model

def expected_wh(p, rate):
    e = p["energy_wh_per_query"]
    return e[p["first_try"]] + e["second_sample"] + rate * e[p["escalate_to"]]

def break_even(p):
    e = p["energy_wh_per_query"]
    return (e[p["escalate_to"]] - e[p["first_try"]] - e["second_sample"]) / e[p["escalate_to"]]

Run against the policy above, the functions give Table 12.1, illustrative and at the node boundary of October 2032 (accelerators and host, before facility overhead). The always-large reference is 1.80 Wh a query.

Table 12.1: Expected energy per query of the claims assistant's cascade at five escalation rates, against the always-large reference (illustrative; node boundary, October 2032).

Escalation rate Expected energy per query (Wh) Against always-large
0.10 0.78 57 per cent lower
0.22 (expected) 1.00 45 per cent lower
0.35 (alert) 1.23 32 per cent lower
0.50 (freeze) 1.50 17 per cent lower
0.80 2.04 13 per cent higher

The break-even escalation rate is 0.67: beyond two requests in three, the cascade costs more than the large model alone. The policy's thresholds are set well below it, because a rising rate first threatens the cascade's quality, not its energy.

The group's evaluation, MCR-HM-0001, rests on 1,500 questions written and labelled by four claims specialists, separate from the 1,000 on which the thresholds were tuned. The share of answers that are correct and cite the right wording was 88.4 per cent (95 per cent interval 86.8 to 90.0) for the small model alone, 97.2 per cent (96.4 to 98.0) for the large model alone and 96.8 per cent (95.9 to 97.7) for the cascade, against a requirement of 96.0. At the measured escalation rate of 22 per cent the cascade used 1.00 Wh a query against 1.80, 45 per cent less at the node boundary. At 1.1 million queries a month that is about 1,096 kWh against 1,980 kWh; with the Frankfurt power usage effectiveness of 1.38 and 350 g CO2e per kWh (location-based), about 529 kg CO2e a month against 956 kg, operational, illustrative, without embodied emissions or the fine-tuning of the models.

The budget that holds the use to this design is recorded as EBG-HM-0002. It covers the escalation rate and the energy of answer generation by the routed models at the node boundary on the evaluation set. The budget of the whole serving path of the same use, with retrieval context and effort settings and at the boundary of the accelerator racks, is EBG-HM-0003 in Chapter 13; the two stand at different boundaries and are never added or compared.

# budget/EBG-HM-0002.yaml (illustrative, Hansa Mutual)
id: EBG-HM-0002
use: FPR-HM-0001                  # claims assistant; routing policy RP-HM-CLAIMS-2026-02 revision 3
status: active                    # a stop drill is on record
period: monthly
set_by: rainer-lehmann            # group CIO; not the platform owner, not the assistant
reviewed_by: verena-lindner
unit: query answered
budget:
  energy_kwh_per_month: 1260      # node boundary, before facility overhead, 2032 (illustrative)
  escalation_rate_max: 0.35       # share of queries reaching the large model
  quality_floor: 0.96             # correct and cited, on the monthly sample; never traded for energy
basis: "MCR-HM-0001: 0.996 Wh per query at 22 per cent escalation x 1.1 million x 1.15 headroom"
measured_by:
  energy: GPU rack meters, Frankfurt (Katja Neumann), attribution by gateway call counts
  routes: AI gateway counters, outside the assistant (Jan Berger, showback)
stop:
  holders: [anja-weiss, platform-on-call]
  path: claims workbench flag switches the assistant off and shows the wording search
  independent_of_assistant: true  # the flag lives in the claims application, not in the gateway
  last_drill: 2032-10-27
  drill_valid_days: 90            # valid to 2033-01-25
on_breach:
  escalation_0.35: alert the owner and the register owner; no automatic change
  escalation_0.50: freeze the router at the last good revision; review before any change
  energy_1.00: report to the register owner; the owner decides scope, quality is not lowered
  energy_1.15: the owner and the budget setter decide together whether to narrow the use
rebound_check: monthly queries against 1.1 million; a rise above 20 per cent is reported

Four properties answer the three questions. The budget is set by Rainer Lehmann, the group CIO, who is neither the platform owner nor the assistant. It is measured outside the system it limits: the rack meters of the Frankfurt data centre and the gateway counters, not the assistant or the router. It is enforced by the gateway, which reads the allow-list and the policy and which neither the models nor their calling agents can write to; the policy changes by pull request, reviewed by someone other than its author and gated by the evaluation. And the stop path is outside the assistant and the gateway: a flag in the claims application that switches the assistant off and shows handlers the wording search that existed before. It was drilled on 27 October 2032 and counts for 90 days, so the budget counts as a cap until 25 January 2033 and then shows as expired until it is drilled again.

Two choices guard against the usual failure of an energy budget, that the use meets it by getting worse. The quality floor sits in the same record and is never traded for energy: when floor and budget conflict the use is narrowed in scope and handlers fall back to the wording search. And the ladder on breach reports and freezes; it does not retune. A router that raised its own thresholds when the budget ran short would be an agent that changes the limit it runs under, a Level 4 act this book prohibits.

The three tests applied

The model choice is a design lever, not a benefit claim. The three tests of the net-benefit principle (Chapter 4) show what this chapter's evidence does and does not establish.

Count. The footprint of each candidate is counted at the node boundary, operational, on the organisation's own measurement, with date and statistic. That passes for model energy. It does not count the training of the small model beyond one figure, the embodied emissions of the hardware, the check and the retrieval in full, or water: the saving is a statement about inference energy at one boundary.

Compare. The comparison here is between models, not between the use and its non-AI alternative. Whether the claims assistant is worth running against handlers searching the wordings, or the classifier against registry staff reading letters, is the question of Chapters 9 to 11 and needs benefit records, which neither use yet has (Chapter 9 wrote them for two other uses). A smaller model makes a use cheaper to run, not worth running, and may make it more used (Chapter 10); the volume check in the budget is there for that reason.

Cap. The budget, the escalation limits, the quality floor and a drilled stop path pass for the claims assistant. For the Region's classifier, EBG-NR-0002 holds it to 1 kWh a month at the node boundary and 180,000 items, with a stop path that returns letters to registry staff by the existing mail-room procedure. Neither budget offsets anything; one use's saving is not credited against another.

Across the modes and the organisations

Across the usage modes

Who chooses the model depends on the usage mode more than on the organisation (Table 12.2).

Table 12.2: Who chooses the model, by usage mode, and what this chapter asks.

Mode Who chooses the model What this chapter asks
Mode 0 Ambient AI The supplier of the host product Nothing can be chosen; ask for disclosure and record the gap
Mode 1 Conversational assistants The user, from a menu, or the provider's default A default of the smallest adequate tier; reasoning effort off unless needed
Mode 2 Embedded workplace assistants The supplier Contract clauses for model tier and disclosure (Chapter 23)
Mode 3 Agentic work tools The tool, with the organisation's allow-list An allow-list of models and efforts in the tool's settings
Mode 4 Built AI applications The builder, against a requirement The protocol above; a model-choice record per use
Mode 5 Autonomous production agents The mandate; the agent may route only among models the mandate lists Allow-list and effort ceiling in the mandate; the agent cannot widen them

In Mode 1 the choice is made by default and habit, which is why a default tier matters more than guidance; in Mode 4 it is made by requirement, the subject of this chapter; in Mode 5 an agent may route among models only inside a mandate that fixes the allowed set and the effort ceiling and enforces them at the gateway.

Across the degrees of delegation

At Level 0 Advisory and Level 1 Assisted a person chooses the model and an assistant may propose candidates or draft the record. At Level 2 Supervised an assistant may run the candidate tests and draft the record for review by a named person. At Level 3 Bounded a router may send requests among the models of the allow-list under the policy, within the budget. Level 4 Autonomous is never used for a router that changes its own allow-list, thresholds or effort ceiling, for an agent that provisions or scales compute beyond a budget set by someone else, or for any action that disables the meter, budget or stop path.

Nordvik Region and Hansa Mutual

Nordvik Region has the small-model case. Its correspondence classifier, a fine-tuned open-weight encoder of 110 million parameters, runs on premises in nordvik-dc and is recorded in MCR-NR-0001 and AIDR-1312 (dated 20 October 2032, owner Maja Lindqvist; reviewer Astrid Nyberg). The choice saves a factor of about 200 in inference energy against the general model, at the price of a quality drop outside the training distribution that the record bounds with a confidence threshold and a shifted-set test. The absolute saving is small, about 96 kWh a month at the node boundary, and the Region's lesson is the habit: the same protocol for the staff assistant and for the eligibility-explanation route (FPR-NR-0002), where the hosted model's energy is undisclosed and a candidate nobody can measure scores as unmeasured. The tender annex TA-NR-ENV-01 asks for that disclosure.

Hansa Mutual has the cascade case. The claims assistant routes between an 8 billion and a 70 billion parameter class model under RP-HM-CLAIMS-2026-02 revision 3, recorded in MCR-HM-0001 (measured on 18 October 2032, valid to 16 January 2033) and AIDR-1362 (dated 28 October 2032, owner Anja Weiß). The group's record carries the escalation rate as the number to watch, and the budget EBG-HM-0002 holds it, with a drilled stop path. Both organisations' figures are illustrative.

Observable, expected, speculation

Observable. Small fine-tuned models used 15 to 50 times less energy than a general 8 billion parameter model on three tasks in one study, and performed worse outside their fine-tuning distribution (129, UNESCO, 2025). Eight of 14 comparable recent models used the same or more GPU energy than similar-sized models of nine months earlier, on one benchmark's boundary (109, Luccioni & Gamazaychikov, 2025). Cascades have reported large cost reductions at matched quality on benchmark tasks (131, Chen et al., 2023) (132, Soiffer et al., 2025).

Expected. Routing among models is expected to become a standard feature of provider products and gateways, given the cost pressure the cascade papers address. Whether providers will expose each route's energy is not established.

Speculation. Whether small models will close the quality gap on complex tasks, so that the escalation rate falls over time, is open. No source read gives an energy measurement for a production cascade, and the claim that cascades save energy at an organisation's own boundary stays a hypothesis until it measures it.

Summary

This chapter concerned E3 Design and the second level of the optimisation hierarchy: the smallest model, with the lowest reasoning effort and the shortest context, that meets the requirement. The method is requirement first: write the task, the quality threshold and its instrument, the latency, the language and data limits and the settings; test three candidates on an independent test set and a shifted set; measure energy per functional unit on your own workload; choose the smallest that passes; record it with an expiry.

The evidence supports a smaller claim than the headline. Task-specific fine-tuned models used 15 to 50 times less energy than a general 8 billion parameter baseline in the UNESCO and University College London study, with a measured loss outside their task; the often-quoted 90 per cent comes from a scenario calculation. Parameter count is a poor guide: 8 of 14 recent models used as much or more energy as similar-sized predecessors, and reasoning can multiply a small model's energy by hundreds. Cascades are established for cost and plausible for energy; their footprint is driven by the escalation rate, which has a break-even and must be metered, and their quality depends on an independent check.

A choice of model runs under a budget set by someone other than the use, measured outside it, with a quality floor that energy cannot buy down and a stop path that works when the gateway does not. The choice passes Count at one boundary and Cap for the two uses, and Compare only between models: whether either use is worth running is the question of Chapters 9 to 11. Chapter 13 takes the next lever, efficient inference.

Sources

  1. European Parliament and Council of the European Union. Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act). Official Journal of the European Union, 12 July 2024. (1, European Union, 2024)
  2. Nordqvist, G. (2026). AI Platform and Security. The Transformed Architecture, Book 3. (27, Nordqvist, 2026)
  3. Nordqvist, G. (2026). AI Operations. The Transformed Architecture, Book 10. (34, Nordqvist, 2026)
  4. Nordqvist, G. (2026). AI Developer. The Transformed Architecture, Book 12. (36, Nordqvist, 2026)
  5. Luccioni, S. and Gamazaychikov, B. (2025). AI Energy Score v2. Hugging Face blog, 4 December 2025. (109, Luccioni & Gamazaychikov, 2025)
  6. UNESCO and University College London (2025). Smarter, Smaller, Stronger: Resource-Efficient Generative AI and the Future of Digital Transformation. Authors Verdadero, Drobnjak, Bosilkovski, Wu, Fischer and Pérez-Ortiz; UNESCO, 2025. (129, UNESCO, 2025)
  7. Luccioni, A. S., Jernite, Y. and Strubell, E. (2024). Power Hungry Processing: Watts Driving the Cost of AI Deployment? ACM FAccT 2024. (130, Luccioni et al., 2024)
  8. Chen, L., Zaharia, M. and Zou, J. (2023). FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance. arXiv 2305.05176. (131, Chen et al., 2023)
  9. Soiffer, D., Kolawole, S. and Smith, V. (2025). Semantic Agreement Enables Efficient Open-Ended LLM Cascades. EMNLP 2025 Industry Track, arXiv 2509.21837. (132, Soiffer et al., 2025)
  10. Hugging Face and partners (2026). AI Energy Score documentation. Page undated, retrieved November 2026. (133, Hugging Face, 2026)