AI sustainability and what we should be measuring

Why she started measuring

As a Mila postdoc working on AI for climate, she was asked whether her models did more good than the energy they used, and she couldn't say: the numbers didn't exist. Per-query figures from big companies still aren't comparable, because nobody knows what "average" means. What users actually want to know is which model is best for their task, and what it costs. Meanwhile, disclosure of compute and energy in notable AI papers rose for a while and is now falling.

Training and inference

Early work estimated the emissions of training large models, such as Strubell et al. and her BLOOM study, which showed emissions vary hugely with the power grid and setup. For running models, her study across tasks found that generating an image can use thousands of times more energy than classifying a piece of text. That kind of measurement is only possible with open models you can run yourself.

What's being measured now

• The EU AI Act and the EU's data-centre rating scheme are starting to require reporting, such as PUE, water use and heat reuse. What's still missing is absolute numbers, and location matters.
• AI Energy Score gives models star ratings for energy on the same task, so you can compare like with like.
• CodeCarbon, a Python package she created, runs alongside any code and estimates its energy use and emissions.

What we're missing

• Data centres' impact is concentrated: nearby communities can face pollution from gas generators and higher electricity prices.
• The Jevons paradox: AI keeps getting more efficient, yet its total energy use keeps rising.
• On-device isn't always greener: simple tasks can run locally, but complex ones are often more efficient in the cloud.
• Model routers could send each query to the most efficient place to run it, weighing other factors such as security too.

Measure at every level

Researchers should report energy even for small models, so "AI" doesn't only mean big company LLMs. Organisations should put AI in their environmental reports: a company spending heavily on AI should report it alongside travel and heating.