Efficient Inference: Quantisation, Caching, Batching and Context Discipline
Chapter 12 settled which model a use needs. This chapter takes the next step down the optimisation hierarchy of Chapter 4, the third level, efficient serving. Given the model, it asks what else shapes the energy of one request. The chapter concerns E3 Design, the function that chooses and shapes the solution for less, and it serves the Count and Cap tests of the net-benefit principle: the footprint of a lever is counted at a stated boundary, and each lever is adopted only under a budget with a stop path. The levers are reasoning effort, context length, caching, batching, quantisation and the techniques of the serving stack, such as speculative decoding and mixture-of-experts models.
The chapter makes three arguments. The levers differ enormously in the strength of their evidence: reasoning effort has been measured in factors of tens and hundreds, a well-built serving stack in tens of per cent, and caching, in the sources read, not in energy at all. The levers fall into two groups that need different contracts: those a buyer of a hosted model can pull and those only the operator of a serving stack can pull. And a lever is not adopted until saving and quality loss have both been measured against a baseline, with the measurement and the stop outside the system being made efficient.

Figure 13.1: Levers on the path of one request and the party that controls each. Dark grey nodes are in the buyer's control; medium grey nodes belong to the supplier or to the operator of the serving stack; the green node, the meter and the gate, sits outside the use and reads the path along the dotted lines.
Where the energy goes in a request
A request to a language model has two phases. In the first, called prefill, the model reads the whole input, which is the instruction, the retrieved passages and the conversation so far. In the second, called decode, it writes the answer one token at a time, a token being a word or a part of a word. While it does so it keeps the intermediate results of its attention over everything already read in a memory structure called the key-value cache, so that it does not recompute them for each new token. A batch is a group of requests that the serving software processes together on the same accelerator, which raises the share of the accelerator's capacity that does useful work, its utilisation.
Two consequences follow from this description, and the rest of the chapter depends on them. The first is that the energy of a request is not a property of the model alone. It depends on how many tokens go in and come out, how many requests share the accelerator and how the software is built. A study of inference energy that varied exactly these things found that the effect of any optimisation depends heavily on the shape of the workload, the software stack and the hardware, and that estimates built from theoretical operation counts substantially underestimate real energy use (134, Fernandez et al., 2025). The second is that the two phases scale differently. Input tokens are read in parallel; output tokens are produced in sequence. Measurements bear this out in the extreme case. Reasoning models, which write a long chain of intermediate text before they answer, were reported to produce 300 to 800 times more output tokens than their non-reasoning counterparts in one benchmark (109, Luccioni & Gamazaychikov, 2025), and the energy followed the tokens.
Who can pull which lever
The party that controls a lever decides what a requirement can say. A buyer of a hosted model, reached through an application programming interface (API), sees the request and the bill and nothing of the machine. The buyer controls the content of the request, the choice of model tier, the effort setting where the supplier offers one, the output limit and whether the work is interactive or can wait. The supplier controls the rest. An operator of its own serving stack, which is what both organisations are for part of their estate (Chapter 6), controls all of it. Table 13.1 gives the division and what has been measured for each lever.
Table 13.1: The levers, who controls them and what has been measured. Effects are those of the cited studies at their own boundaries and are not comparable with each other. Sections below give boundary, baseline and source for each row.
| Lever | Hosted model (buyer) | Own stack (operator) | Measured effect, with boundary | Evidence |
|---|---|---|---|---|
| Reasoning effort, thinking | Setting, where offered | Setting and model choice | 1.6 to 697 times, GPU energy per 1,000 queries, 2025 | Independent benchmark |
| Prompt and context length | Yes | Yes | "Over 50 per cent", baseline not stated, 2025 | UNESCO and UCL report page |
| Output limit | Yes | Yes | Not isolated in the sources read | None |
| Provider caching | Opt in, as offered | Prefix cache in the stack | No energy measurement found | Cost and latency only |
| Deferred batch work | Choice of interface | Scheduler | Not measured in energy | Price only |
| Quantisation | No, the supplier decides | Yes | Up to about 4 times against FP32 on short outputs, 2026 | Preprint |
| Stack: compiler, batching, decoding | No | Yes | Up to 73 per cent, GPU energy, one GPU, simulated offline, 2025 | Peer-reviewed |
The division has a practical consequence that Chapter 23 turns into tender text: a buyer cannot require quantisation of a hosted model, but it can require that the supplier disclose the serving configuration, hold the model version still and give notice when the serving stack changes, because a change in serving is a change in footprint and in behaviour (AI Operations, Chapter 13) (34, Nordqvist, 2026).
Reasoning effort and the thinking budget
A reasoning model spends tokens on intermediate work before it writes its answer. Many providers now let the caller set how much. One provider's documentation, read in November 2026 and used here as an example of a pattern, offers an effort setting with five levels from low to max; it states that the setting trades response thoroughness against token efficiency, that it affects all output tokens including tool calls and thinking, that the default is high on most models and medium on two, and that it is a behavioural signal and not a strict token budget (138, Anthropic, 2026). The same page advises an effort sweep on the user's own evaluations before the setting is changed and notes that changing it between requests can invalidate a cached prefix, so effort is best varied across workloads and not within a long cached conversation.
The measured size of the lever is the largest of any in this chapter. The Hugging Face AI Energy Score, published on 4 December 2025, compared the same open models with reasoning off and on. The boundary is GPU energy in watt-hours per 1,000 queries, measured with the CodeCarbon library; the post states neither hardware nor batch size. DeepSeek-R1-Distill-Llama-70B used 49.53 Wh with reasoning off and 7,626.53 Wh on, a factor of 154; Phi-4-reasoning-plus 18.42 and 9,461.61 Wh, a factor of 514; SmolLM3-3B 18.35 and 12,791.22 Wh, a factor of 697. The post reports an average of about 30 times across reasoning models, and for the gpt-oss models the high setting against the low setting at about 4.8 times for the 20-billion-parameter model and about 1.6 times for the 120-billion-parameter one (109, Luccioni & Gamazaychikov, 2025). The post does not say whether answer quality was measured. The factors are therefore the cost side of a trade, not a saving: the quality that the extra tokens buy is not in the data.
A second, independent view comes from a modelling study by Microsoft authors, which estimates the median energy of a query on a frontier-scale model (more than 200 billion parameters, on H100 nodes) at 0.31 Wh and of a test-time-scaling query 15 times longer than a typical one at 3.91 Wh, thirteen times as much. The boundary is a modelled eight-GPU node's power times facility overhead, for 2026, and the study is an estimate and not a measurement (52, Oviedo et al., 2026). It agrees in direction with the benchmark but counts a different thing, so the two figures are never ranked together.
Three conclusions are the book's own. First, effort is the lever where a buyer has the most reach for the least engineering: it is a request parameter. Second, a single default is wrong in one direction or the other: high wastes tokens on lookups, low fails on multi-step questions. The right design is a default per class of request, an escalation path the user can invoke and a measurement of both. Third, the saving is only a saving after the quality check, because at lower effort the model "still thinks on sufficiently difficult problems, but thinks less" (138, Anthropic, 2026), and that is exactly the case where an unchecked setting fails.
Context discipline
Context discipline means sending the model what the task needs and nothing else, and asking for the shortest answer that serves. It covers the system prompt, the retrieved passages (the retrieval of less, treated architecturally in AI Platform and Security, Chapter 10) (27, Nordqvist, 2026), the conversation history that is carried forward and the length of the answer.
The evidence is thinner than the practice suggests. The UNESCO and UCL report page of July 2025, updated April 2026, states that shorter, more concise prompts and responses reduce energy use by over 50 per cent, with no baseline, model, hardware or boundary on the page; the report itself was not read (129, UNESCO, 2025). The figure is a direction, not a number for a footprint record, which needs a paired measurement on the organisation's own use, as the worked example below shows.
Four practices make the lever work. Set an output limit on every route, so that the longest answer is a design decision and not an accident. Retrieve fewer and better passages, since each passage is read on every request. Trim the history that is carried forward, summarising it or dropping what the task no longer needs. Name the context ceiling in configuration, as AI Platform and Security, Chapter 10 does with a maximum context in its retrieval configuration, so that growth is visible in review. Each has a quality side: a limit can truncate a legal answer mid-sentence, and fewer passages can drop the one that mattered. The quality gate below tests for that.
One interaction deserves a warning. Effort and output length are separate levers. The provider documentation read for this chapter states that on one model family changing effort does not reliably shorten replies and that length should be controlled in the prompt (138, Anthropic, 2026).
Caching
Caching reuses earlier work. Two kinds matter and they are often confused. A prompt cache, also called a prefix cache, stores the model's processed form of a fixed beginning of the input, such as a long system prompt or a document, so that the next request that starts the same way does not have to read it again. A response cache, sometimes called a semantic cache, stores answers and returns a stored one when a new question is the same or sufficiently similar, so that the model is not called at all.
For prompt caching, the provider documentation read for this chapter describes the mechanism and its terms: the cached prefix is the tools, the system text and the messages up to a marked point; the default lifetime is five minutes and is refreshed by use; reads are priced at a tenth of the base input price on the standard rate (with per-model exceptions) and writes at 1.25 times; there is a minimum prompt length below which nothing is cached and no error is returned; and caching has no effect on the output (136, Anthropic, 2026). The book's own production meets the same gap: the energy of a cache read is unknown for the models used (Chapter 8).
For response caching the position is worse. Searches found vendor and practitioner reports of cost reductions and hit rates, and one master's thesis measuring the energy of semantic caching in a Kubernetes environment, which could not be read. An earlier research dossier reported a review of 66 semantic caching papers, none of which measured the cache's energy against the inference it replaces; that review could not be found in November 2026 and is not cited. The sound statement is the narrower one: no measured energy comparison of caching against its alternative was found in the sources read for this chapter, and none is cited.
That does not make caching a bad idea: a cache that answers a share of requests without a model call plainly removes that inference. What is missing is the net: the energy of the lookup, the storage, the extra writes, the loss of batching efficiency and the wrong answers returned for questions that only look the same. For a reserved pool of accelerators there is a further consideration that is the book's own reasoning and not a cited result. An accelerator that is reserved draws power whether or not it is busy, so work removed from a pool raises idleness before it lowers energy, and the saving is booked only when the capacity is released or resized. The practical rule is to book the token saving that was measured and the energy saving that was metered, and to say which is which.
Batching and the choice between interactive and deferred work
A serving system raises efficiency by batching requests. Continuous batching is the technique of adding new requests to a running batch as earlier ones finish. In the study of inference energy cited above, it saved up to about 5 per cent at larger batch sizes, and at small batch sizes the overhead of the online scheduler outweighed the benefit; the boundary is GPU energy on one test GPU in 2025 (134, Fernandez et al., 2025). The operator controls it. The buyer's lever is different: it decides whether the work must be answered now.
A batch interface accepts many requests together and returns results later. The provider documentation read for this chapter describes one such interface: asynchronous processing, most batches finishing within an hour, results available when all have completed or after 24 hours, and a 50 per cent price reduction (137, Anthropic, 2026). The page presents price and throughput, not an energy benefit, and no source read measures the energy difference between interactive and deferred serving. The book's own conclusion rests on the structure of the problem: work that nobody waits for, such as nightly re-scoring, document classification or evaluation runs, should use the deferred interface where there is one, because it lets the supplier fill capacity that would otherwise idle, and it also lets the buyer set a time window that Chapter 17 can use for placement.
Quantisation and compression
Quantisation stores a model's numbers at lower precision, for instance in 8 bits where the original used 16 or 32. The model needs less memory and can move fewer bits for every token. Only the party that serves the model can apply it. A buyer of a hosted model gets whatever the supplier chose; the buyer's lever is the question it asks and the evaluation it runs.
A preprint of 13 October 2026 evaluated ten post-training weight-only quantisation methods on five models of 1 to 14 billion parameters, four benchmarks and four GPU types, and measured energy, accuracy and run time. It reports that the quantised models reach their highest efficiency relative to full precision when inputs are long and outputs short; that larger batches make them slightly more efficient; that the fused-kernel methods EETQ int8 and Bitsandbytes int8 gave the largest savings, up to about four times against FP32 on short generation, with little accuracy loss; and that energy tracks run time closely (135, Basta, 2026). Only the abstract was read. Three cautions follow. The baseline is FP32, whereas production serving normally starts from a 16-bit format, so the factor against a production baseline is smaller and the paper's figure for it was not read. The result is for models of 14 billion parameters and fewer. And the result is conditional on the workload: a gain that is largest for short outputs says little about a reasoning route that writes thousands of tokens. The UNESCO and UCL page credits compression with up to 44 per cent, again with no baseline stated (129, UNESCO, 2025).
The conclusion for a buyer is the narrow one. For a hosted model, ask whether the served model is quantised, at what precision, and whether a version change altered it, and test quality when it does. For the open-weight route that an organisation runs itself, treat quantisation as an engineering change with a measurement before and after, never as a free saving.
Speculative decoding, mixture of experts and the serving stack
Speculative decoding lets a small draft model propose several tokens that the large model then checks together, so that fewer slow steps are needed. In the study of inference energy, it reduced energy at small batch sizes (16 or fewer) and increased it at larger ones: at a batch size of 128 it needed 25.65 per cent more energy than ordinary decoding. The paper's own figure for the small-batch saving is written ambiguously, with a plus sign on a reduction, so it is not repeated here (134, Fernandez et al., 2025). A mixture-of-experts model activates only some of its parameters for each token. In the same study the mixture-of-experts model OLMoE used up to 54.24 per cent more energy than the dense OLMo 1B model, which the authors trace to a slower fused expert kernel in their set-up; the models were small, and the result does not transfer to large production models. It does show that the architecture is not efficient by label.
The stack as a whole is where the large measured gain was found. On replayed traffic traces, an optimised configuration (a dense model on the vLLM serving library with CUDA graphs) against unoptimised native PyTorch saved 73.00 per cent on a short-conversation trace, 72.18 per cent on a conversation trace and 37.58 per cent on a code trace. The boundary is GPU energy measured through NVIDIA's management library on one RTX A6000 GPU; the workloads were simulated offline, not measured in online service; the year is 2025 (134, Fernandez et al., 2025).
At the scale of a provider, one disclosure gives the order of magnitude and its limits. Google reported in August 2025 that the median Gemini Apps text prompt used 0.24 Wh in May 2025, on a whole-stack boundary that includes accelerators, host, idle capacity and data-centre overhead, and that the energy per prompt fell by a factor of 33 in twelve months, 23 times from model improvements and 1.4 times from better utilisation. The paper lists mixture-of-experts, speculative decoding, distillation, quantisation and batching among the measures but does not apportion the gain among them (51, Elsworth et al., 2025). The figures are vendor-reported and unaudited. A projection by Microsoft authors puts a combined 8 to 20 times reduction per query within reach from hardware, serving and models together (52, Oviedo et al., 2026); that is an expected, conditional figure and not an observation.
From lever to requirement
A lever becomes a requirement when it is written where it binds: in a configuration for a route that the organisation runs, in a contract clause for one it buys, and in a budget that names who may change it. The checklist below orders the levers by the strength of evidence and by who controls them. It is what the platform team and the procurement officer work through for each use at the start and at each renewal.
Practical guide: an inference efficiency checklist
Purpose. Decide, for one use, which levers apply, who pulls them and what evidence closes each. Roles. AI product owner, platform engineer, FinOps or GreenOps practitioner, procurement officer, sustainability lead. Inputs. The footprint record of the use (Chapter 6), a representative request set with labelled expected answers, and the hosting arrangement (own stack, reserved capacity or hosted model).
- Effort. For every route that uses a reasoning model, set a default effort per request class, define an escalation path and record both. Run a paired test on the request set at each setting. Controlled by the buyer where the supplier offers the parameter.
- Output limit. Set a limit per route that no valid answer in the request set exceeds, and count truncations. Controlled by the buyer.
- Context. Count prompt tokens per request by component (system text, passages, history). Reduce the largest component first and re-test. Controlled by the buyer.
- Deferral. List the requests nobody waits for and move them to a deferred interface or an off-peak window. Controlled by the buyer; the energy effect is asked of the supplier.
- Caching. Enable a prefix cache for fixed prompts above the minimum length, and a response cache only for questions with approved fixed answers. Record the hit rate. Do not record an energy saving unless metered.
- Quantisation and stack. For an own stack, change one thing at a time and measure energy and quality before and after. For a hosted model, put the disclosure and notice clauses below into the contract. Controlled by the operator or the supplier.
- Gate. Run the quality gate and adopt only what passes. Write the result, the date and the owner into the budget record, and set its expiry.
Result. An environmental budget record with its levers and an adopted or rejected decision for each.
Worked example: effort, context and caching on one assistant (illustrative)
Hansa Mutual's claims assistant (FPR-HM-0001, SVC-HM-007) answers about 1.1 million queries a month for about 2,900 handlers, retrieving eight passages from policy wordings for every query on the Frankfurt platform. It runs an open-weight model whose effort setting has been left at its default. Anja Weiß, head of the group AI platform, asks for three changes: effort low for wording lookups, with a handler-invoked escalation to high; four retrieved passages in place of eight; and an enabled prefix cache for the fixed system text. A sentinel set SENT-HM-007 of 400 labelled queries, never used for tuning, was drawn from the queries of September 2032 and split into 288 wording lookups and 112 multi-clause questions. The claims quality team, not the platform team, labelled the expected answers and grades the output (AI Operations, Chapter 13, treats such a fixed set as the sentinel of a model service) (34, Nordqvist, 2026).
Each configuration was run on a dedicated test pool whose racks are metered by Katja Neumann's data-centre team. The energy figure is the metered IT energy of the test pool divided by the queries served, before facility overhead, October 2032. All figures are illustrative.
Table 13.2: Paired runs on one request class. Energy: metered IT energy of the test pool per query, October 2032, illustrative. The prefix cache changed nothing that the meter could separate from noise of about 0.1 Wh.
| Configuration (wording lookups) | Energy per query (Wh) | Passes (of 288) | Change in passes |
|---|---|---|---|
| Baseline: default effort, 8 passages | 3.1 | 266 | none |
| 4 passages only | 2.6 | 265 | one fewer |
| Low effort only | 1.2 | 263 | three fewer |
| Low effort and 4 passages | 0.8 | 262 | four fewer |
| Same, with prefix cache on | 0.8 | 262 | none |
Table 13.2 gives the result for the wording lookups. For the 112 multi-clause questions, low effort left the energy per query at 6.0 Wh in the same test, because the model still wrote as many reasoning tokens as the problem needed, and it cost 10 of 96 passes (86 against 96). That result rejected the lever for that class. The gate is a short program that reads the paired runs and the limits and prints a decision for each class.
"""Decide whether a lever may be adopted: quality loss per class and energy change, from paired runs."""
import json, sys
def decide(runs, max_loss):
out = {}
for cls, r in runs.items():
loss = round(100 * (r["baseline_pass"] - r["lever_pass"]) / r["n"], 1)
wh_base, wh_lever = r["baseline_wh"], r["lever_wh"]
out[cls] = {
"loss_points": loss,
"energy_change_pct": round(100 * (wh_lever - wh_base) / wh_base, 1),
"adopt": loss <= max_loss[cls],
}
return out
if __name__ == "__main__":
runs = json.load(open(sys.argv[1]))
limits = json.load(open(sys.argv[2]))
print(json.dumps(decide(runs, limits), indent=2))
Run on the paired results, it reports a loss of 1.4 points and an energy change of minus 74.2 per cent for wording lookups, which is inside the limit of 2.0 points, and a loss of 8.9 points for multi-clause questions, which is not inside the limit of zero. The decision follows: low effort by default for wording lookups, high effort kept for multi-clause questions, and a handler button that escalates a lookup to high.
Two caveats belong to the example. Four extra failures in 288 cannot be told from chance, and the gate bounds the loss it can see without proving there is none, so the production sentinel run continues weekly and the budget expires the decision on any change of model, version, hardware, region or effort default. And the saving per query is not the saving per month. With the classes at 72 and 28 per cent of queries and one lookup in ten escalated, the expected energy per query is 3.91 Wh before and 2.42 Wh after (a fall of 38 per cent), which at 1.1 million queries is about 4,303 kWh and about 2,664 kWh of IT energy a month, a difference of about 1,639 kWh. That is the figure the budget carries; it is an estimate from the test pool and is replaced by the metered monthly figure when it exists. The levers are recorded with the budget:
# Illustrative. Hansa Mutual, claims assistant (FPR-HM-0001, SVC-HM-007), levers recorded with the budget.
id: EBG-HM-0003
status: proposed
use: FPR-HM-0001
period: monthly
set_by: anja-weiss # head of the group AI platform; not the assistant, not its supplier
approved_by: dr-annette-kraemer # chair of the group AI governance board
decision: AIDR-1363 # 2032-11-04, after the paired runs of 2032-10-28
budget:
energy_kwh_it: 2900 # accelerator racks, IT energy before facility overhead
basis: "2,664 kWh expected after levers, plus 9 per cent headroom (illustrative)"
volume_trigger: 1210000 # queries a month; 10 per cent above 1,100,000 reported to the register owner
levers:
- {id: effort-default, setting: low, applies_to: [wording-lookup], escalate: handler-request, owner: anja-weiss}
- {id: effort-escalated, setting: high, applies_to: [multi-clause], owner: anja-weiss}
- {id: retrieved-passages, max: 4, was: 8, owner: anja-weiss}
- {id: output-tokens, max: 600, owner: anja-weiss}
- {id: prefix-cache, enabled: true, energy_effect: not-measured, owner: anja-weiss}
quality_gate:
set: SENT-HM-007 # fixed regression set, 400 labelled queries, never used for tuning
max_loss_points: {wording-lookup: 2.0, multi-clause: 0.0}
independent_oracle: claims-quality-team # not the platform team and not the model that answers
measured_by: rack-level metering (Katja Neumann) joined to gateway volume counts; read outside the assistant
stop:
holders: [anja-weiss, platform-on-call]
path: "gateway route switched to the policy-wording search page; flag held in the gateway, not in the assistant"
last_drill: none # an undrilled stop does not count as a cap (Chapter 4)
evidence:
footprint_valid_days: 90
expires_on_change_of: [model, version, hardware, region, effort-default]
The record answers rule 22 for this use. It is the serving budget of the claims assistant at the boundary of the accelerator racks, and it stands beside EBG-HM-0002 of Chapter 12, which holds the escalation rate and the generation energy of the model cascade at the node boundary on the evaluation set (about 1.0 Wh a query there, against 3.9 and 2.4 Wh a query here for the whole serving path); the two are never added or compared. The budget is 2,900 kWh a month of metered IT energy, set by Anja Weiß and approved by the group AI governance board, so not by the assistant and not by its supplier. It is measured outside the assistant, by rack-level metering joined to the gateway's volume counts, read by people who do not run the model. The stop path is a switch of the route at the gateway to the policy-wording search page, held by the platform head and the platform on-call and kept in the gateway and not in the assistant. The record says that the stop has not been drilled, and under the rule of Chapter 4 an undrilled stop does not count as a cap until it has been. Rebound is watched by the volume trigger: if cheaper answers draw 10 per cent more queries, the register owner is told and the budget is reopened, as Chapter 10 describes.
Requirements for a hosted route
The assistant above runs on the organisation's own stack. A hosted route needs the same levers written as terms, because the buyer cannot pull most of them. The block below lists the requirements that Nordvik Region's procurement manager, Ida Pettersson, carries into the tender annex TA-NR-ENV-01 for the hosted route of its eligibility-explanation service (FPR-NR-0002); Chapter 23 completes the annex.
# Illustrative input to TA-NR-ENV-01 (Chapter 23), hosted route FPR-NR-0002.
route: FPR-NR-0002
requirements:
- id: INF-1
text: Effort or thinking depth is settable per request, and the response reports the tokens used for it.
evidence: API documentation and a test call at signature
- id: INF-2
text: Output length can be limited per request, and truncation is reported in the response.
evidence: API documentation
- id: INF-3
text: The served model version is pinned for the contract term; a change of version, precision or serving stack is notified 60 days ahead (or at once for an emergency) as a change request.
evidence: Contract clause; notice log
- id: INF-4
text: Energy per request or per million tokens is disclosed at a stated boundary and year, vendor-reported or independent, for the pinned version; where it is not, the supplier says so.
evidence: Supplier disclosure record SDR-NR-*, valid 12 months
- id: INF-5
text: A deferred interface is offered for work that is not urgent, with the energy basis of any price reduction stated.
evidence: API documentation; supplier statement
quality:
regression_set: sentinel set owned by the Region; the supplier gets no access to the labels
rule: a lever or a supplier change that lowers quality below the gate is refused or rolled back by the Region
The AI Act does not do this work for a deployer. Article 53(1)(a) obliges the provider of a general-purpose AI model to maintain technical documentation with the information of Annex XI (1, European Union, 2024), and as Chapter 8 records, the Commission's documentation form asks for training energy and inference compute and not for the energy of inference. A buyer who wants inference energy has to write it into the contract, which is what INF-4 does.
Testing quality after the lever
Three practices keep the trade honest. The first is a fixed regression set, as above, that the people tuning the lever never see, graded by a party independent of the one that applies the lever; this is the independent test oracle of AI Developer (36, Nordqvist, 2026) and the independence principle of AI Purple (33, Nordqvist, 2026), applied to efficiency. The second is a sentinel run in production at a stated interval, because a model, a provider's serving stack or the request mix can change underneath a decision (AI Operations, Chapter 13). The third is a record of each refused lever, so that nobody repeats the experiment.
Quality testing also guards against a lever that saves tokens per attempt but raises the number of attempts, as when lower effort makes an agent retry, and so raises energy per completed task. Chapter 15 treats that for agents; here the unit of measurement is the completed request that passes the gate, not the token.
The budget and the stop path
Every route in this chapter runs under an environmental budget (Chapter 4). For the claims assistant it is the energy record above; for the hosted route it is a token and cost cap at the gateway, because the Region cannot meter the supplier's energy, with the energy ask recorded as an open requirement. The measurement is read outside the use, by data-centre metering, the provider's meter or the gateway's counts, never by the assistant. The stop is a gateway route switch to a non-AI fallback, held by named people and drilled on a schedule, since an undrilled stop is not a cap.
The levers are subject to the same rule. An agent that could change its own effort, output limit or model, or switch off the meter or the cache under measurement, could disable the budget from the inside. The ceilings in the next subsection prevent that, and Level 4 is never used for an action that disables the measurement, the budget or the stop path, or for provisioning or scaling compute beyond a budget set by someone else.
Across the modes and the organisations
Across the usage modes
Table 13.3 shows which levers a buyer can pull in each usage mode.
Table 13.3: Levers by usage mode. Where the supplier exposes no control, the lever is a contract term (Chapter 23) or a reason to choose another route (Chapter 12).
| Mode | What the buyer can pull | What it cannot |
|---|---|---|
| Mode 0 Ambient AI | Nothing; a supplier-reported row | Effort, context, caching, serving |
| Mode 1 Conversational assistants | Account settings and user habits, where exposed | Effort and serving, as a rule |
| Mode 2 Embedded workplace assistants | Licence scope and admin settings, where exposed | Effort, serving |
| Mode 3 Agentic work tools | Effort and budget by administrator setting, if exposed | Serving stack |
| Mode 4 Built AI applications, hosted model | Context, output limit, effort, model tier, deferral | Quantisation, serving |
| Mode 4 Built AI applications, own stack | All levers, with measurement | Nothing; the operator carries the whole footprint |
| Mode 5 Autonomous production agents | Per-step effort and a token budget per task | Raising its own limits |
Across the degrees of delegation
The starting ceilings of Chapter 4 apply to the levers (the author's recommendation, not a cited standard). An agent may lower its own effort, output limit or context within the mandate's range at Level 3, because the action is reversible and lowers the spend. Raising effort above the default for the class, choosing a larger model or lengthening the output limit is a Level 2 action: the agent proposes and the gateway or a named person approves within the budget. Changing the mandate, the budget, the meter, the cache that is under measurement or the stop path is never the agent's action at any level, and Level 4 is never used for it. For an agent that decides about a named person's access to work, the prohibition of Chapter 4 applies in full and no efficiency lever alters it.
Nordvik Region and Hansa Mutual
Nordvik Region. The citizen-service assistant (SVC-NR-005) has two routes with opposite control. The open-weight generation route (FPR-NR-0001) runs on the Region's reserved accelerators at the hosting provider (NR-2026-031), so the platform team under Maja Lindqvist is the operator of its serving stack and can pull every lever. Henrik Dahl's decision of 25 October 2032 (AIDR-1313) sets an output-length limit on the route and adds an exact-match cache of approved answers to frequently asked questions, with the hit rate to be counted at the gateway. The answers in that cache are texts the Region has already approved and published, which is why a cache is safe here. The Region books the tokens saved as counted by the gateway and books no energy saving, because the pool of 24 reserved accelerators at average utilisation 0.58 keeps drawing power until the pool is resized (the plan of 2027 reduces the reserved pool to 16); the saving becomes an energy figure when the hosting provider's metering shows it, and the next supplier disclosure request (SDR-NR-0001) asks for it. The eligibility-explanation route (FPR-NR-0002) is hosted by a large provider in the EU that discloses neither site nor energy; the supplier controls its quantisation and serving, the Region controls the output limit, the request content and, where offered, the effort, and the requirements above go into the tender annex TA-NR-ENV-01, since the hosting contract is due for re-tender within the year.
Hansa Mutual. The claims assistant (FPR-HM-0001) follows the worked example: effort low for wording lookups, escalation on request, four passages, an output limit, the prefix cache enabled and not credited, and the budget EBG-HM-0003 approved by the group AI governance board under Dr. Annette Krämer on 4 November 2032 (AIDR-1363), after the paired runs of 28 October. For the correspondence assistant (FPR-HM-0003), which uses hosted models on non-sensitive routes and self-hosted models on sensitive ones, the group applies the same checklist to the self-hosted route and the clause set above to each hosted provider, and records the answers per provider in the supplier disclosure records that Stefan Wiedemann's procurement team holds. Group sustainability under Dr. Verena Lindner counts no efficiency claim until the claim record CLM-HM-0001 has its evidence: the board's statement that AI work became more efficient is not made on the strength of a lever that has been tested on one assistant.
Observable, expected, speculation
Observable. Switching reasoning on multiplied GPU energy per 1,000 queries by 154 to 697 times for three open models in a December 2025 benchmark, and the gpt-oss high setting cost 1.6 to 4.8 times the low one (109, Luccioni & Gamazaychikov, 2025). A serving stack changed measured GPU energy by up to 73 per cent in a simulated offline test, and speculative decoding cost 25.65 per cent more at a batch size of 128 (134, Fernandez et al., 2025). Providers document effort, caching and batch interfaces at the level of price, latency and behaviour, and say nothing about energy (136, Anthropic, 2026) (137, Anthropic, 2026) (138, Anthropic, 2026).
Expected. Effort and thinking controls are expected to be exposed on more models and by more providers, since the documentation read for this chapter already extends the control across several model families; the exact set will change. A combined reduction in energy per query of 8 to 20 times from hardware, serving and models together is expected on the conditions stated by its authors (52, Oviedo et al., 2026).
Speculation. The size of serving gains beyond that, and whether lower energy per query will be outrun by volume, are speculation here; Chapter 10 treats rebound and Chapter 25 gives the author's own view.
Summary
This chapter concerned E3 Design and the third level of the optimisation hierarchy. The levers of efficient inference differ in size and in evidence. Reasoning effort is the largest measured lever and a buyer can pull it with a request parameter, but the benchmark that sized it did not measure the quality it buys. Context and output discipline is next, with a direction and no usable baseline. Quantisation, the serving stack, batching, speculative decoding and mixture-of-experts models belong to the operator, give gains of tens of per cent in the studies read and can raise energy in the wrong conditions. Caching and deferred batch work are real mechanisms with documented prices, and no measured energy comparison for them was found.
A lever becomes a requirement as a configuration for a route that the organisation runs, as a contract clause for one it buys, and as a record in the environmental budget that names the owner, the measurement taken outside the use, the quality gate and a stop path that is drilled. The saving is booked at the level it was measured: tokens counted, or energy metered, never one reported as the other. The budget of this chapter holds for one route and one request class, and its stop path sits outside the assistant. Chapter 14 moves one step back to the choice between training, fine-tuning and retrieval, which sets the footprint before any request is served, and Chapter 15 gives agents the resource budget that this chapter's levers presuppose.
Sources
- European Parliament and Council of the European Union. Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act). Official Journal of the European Union, 12 July 2024. (1, European Union, 2024)
- Fernandez, J., Na, C., Tiwari, V., Bisk, Y., Luccioni, S. and Strubell, E. (2025). Energy Considerations of Large Language Model Inference and Efficiency Optimizations. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, long papers, pp. 32556-32569. (134, Fernandez et al., 2025)
- Luccioni, S. and Gamazaychikov, B. (2025). AI Energy Score v2. Hugging Face blog, 4 December 2025. (109, Luccioni & Gamazaychikov, 2025)
- Basta, P. (2026). Towards Green AI: Evaluating Energy Efficiency for Quantized LLM Inference. arXiv 2610.17531. (135, Basta, 2026)
- UNESCO and University College London (2025). Smarter, Smaller, Stronger: Resource Efficient Generative AI and the Future of Digital Transformation, report page. (129, UNESCO, 2025)
- Elsworth, C. et al. (2025). Measuring the environmental impact of delivering AI at Google Scale. arXiv 2508.15734. (51, Elsworth et al., 2025)
- Oviedo, F. et al. (2026). Energy use of AI inference, efficiency pathways, and test-time scaling. Joule 10(8), 102430. (52, Oviedo et al., 2026)
- Anthropic (2026). Prompt caching. Claude Platform documentation. (136, Anthropic, 2026)
- Anthropic (2026). Batch processing. Claude Platform documentation. (137, Anthropic, 2026)
- Anthropic (2026). Effort. Claude Platform documentation. (138, Anthropic, 2026)
- Nordqvist, G. (2026). AI Platform and Security. (27, Nordqvist, 2026)
- Nordqvist, G. (2026). AI Operations. (34, Nordqvist, 2026)
- Nordqvist, G. (2026). AI Purple. (33, Nordqvist, 2026)
- Nordqvist, G. (2026). AI Developer. (36, Nordqvist, 2026)