Day 2 · Invited talk · 13:30–14:00

Making AI inference more sustainable

Pierre-Luc Bacon · Université de Montréal, CIFAR, Mila

Tap a dashed word or a blue reference to open it below.

In one line

AI wastes most of its energy when it runs, not when it trains, and mostly by moving data rather than doing maths. Start by asking whether you need an LLM at all, then work down the levers, counting joules instead of tokens.

Training laws miss the running cost

, like DeepMind's work, describe how model size, data and compute trade off during training. Bacon pointed out that they say nothing about , the cost of running a model for its whole life. Work by extends the laws to include it.

Moving data costs more than maths

On a GPU, fetching data from off-chip memory costs orders of magnitude more energy than an add or a multiply. Reading the prompt is limited by compute, but writing the answer one token at a time is limited by memory (). Long answers make it worse: every new token rereads the whole , so an answer twice as long costs roughly four times as much.

Nine levers, from the task to the data centre

He broke the energy of one task into factors, each with its own lever:

≈ × × × × ( × / ( × ) + × )

What each symbol means
Etask
energy used to complete one task
η
data-centre overhead, such as cooling and idle machines
ebyte
energy to move one byte of data on the hardware
Ncall
how many times a model is called for the task
Ntok
tokens generated per call
bw
bytes per model weight (fewer with 8-bit or 4-bit weights)
Pact
parameters used for each token
B
batch size, how many requests share one read of the weights
γ
tokens produced per read of the weights
κ
bytes of KV cache stored per token
L
current length of the context
Nine levers on the energy of one AI task Nine levers listed from the task itself down to the data centre: whether you need a big model, which model, how much reasoning, weights read per token, memory per token, tokens per weight read, the serving system, hardware settings, and the data centre. Adapted from Pierre-Luc Bacon's talk at SMASH 2026. Nine levers on the energy of one AI task From the task itself down to the building it runs in task building 01Do you need a big model?A classical method or a small task model 02Which model?Size, open-weight family, routing 03How much reasoning?Thinking budget, shorter answers, stopping 04Weights read per tokenLower-precision weights, mixture of experts 05Memory per tokenLeaner attention, state-space models 06Tokens per weight readSpeculative decoding, multi-token prediction 07Serving systemBatching, cache paging, scheduling 08Hardware settingsGPU clock speed, choice of chip 09Data centreIdle machines, cooling, the power grid Highlighted: not using a big model when you don't need one saves the most.

Swipe sideways to see the whole diagram.

Adapted from Pierre-Luc Bacon's talk at SMASH 2026.

The first lever is the biggest: for scheduling or booking, a good old forecasting model often beats an LLM.

Billed by the token, not the joule

Most published efficiency methods aim to save tokens, because tokens are what customers pay for. Energy can be measured on the GPU with off-the-shelf tools, but per-token billing gives customers little reason to care about joules. He called that a policy question.

Measured properly, simple baselines win

Training models to switch experts less often looked promising, but measured at the hardware level it didn't pay off. The better answer was a classic caching rule from Belady: keep the experts you'll need soonest. He tested it on .

Agents explore well but certify badly

AI agents are good at finding ideas but prone to false discoveries, often because they choose a test after peeking at the data. His group built a tool that separates choosing a test from running it, records every claim, and uses so results stay valid even when someone checks early. He sees making agents do trustworthy discovery as the flip side of AI safety.

Unofficial notes by Rai Vaheed, written up after the talk. The ideas are the speaker's; any mistakes are mine. Corrections: vaheed.raihan@gmail.com · View the source of these notes