More big numbers coming in. For NYT’s the Upshot, Aatish Bhatia demonstrates the millions of trillions (octillions!) of calculations required for training AI.
During its training, the A.I. is fed a portion of a sentence, and the knobs are adjusted using a series of math calculations to improve its guesses for the next word. This process is then repeated over trillions of words of training data.
It takes only a few additions and multiplications to adjust a single knob. But when trillions of knobs are adjusted trillions of times, the math quickly adds up.
And this is just the first step in training a language model, known as pre-training. It’s followed by additional steps of calculations that hone the A.I.’s behavior and improve its ability to code and solve math problems.
All those calculations require energy, data, and time. The big companies are taking what they can get with the first two things in efforts to reduce the third.
Visualize This: The FlowingData Guide to Design, Visualization, and Statistics (2nd Edition)
