The Model Living Entirely on My Laptop
The wifi in my building went out at eleven last night, the little icon in the corner filled with a diagonal line, and I kept typing anyway, because the model I was talking to lives on my hard drive and has never once needed permission from a router to answer me.
This is what "local" actually means, once you take the marketing off it: the weights — billions of numbers, each one a small decision the model learned during training — sit on disk, not on someone else's server three time zones away. When I ask it something, nothing leaves the machine. No request travels anywhere to be answered and sent back, no log line gets written on a server I will never see. The answer is computed here, on the same eight cores that were running my email client five minutes earlier, in a kitchen with no signal at all.
The trick that makes this possible is not new intelligence, it's arithmetic done more cheaply. A number that used to need thirty-two bits to describe now gets four, or eight — quantisation, which is a fancier word for rounding, done carefully enough that the model still sounds like itself and not like a bad photocopy of the thing it used to be. It is unglamorous plumbing, the kind nobody demos on stage between the applause lines, and it is the whole reason a laptop can hold something that behaves like a mind instead of just dreaming about one over a very slow connection.
What I got back last night was smaller than what I get from the hosted version I pay for every month — a little less patient with a badly phrased question, a little quicker to shrug and change the subject. That is the honest trade, and nobody selling you the local model says it out loud in the launch post: you get privacy, you get a machine that keeps working on an airplane or in a blackout or on a train through a tunnel, and you give back some of the model's largest, best-fed self to get it. I would still rather have the smaller, honest one running at eleven at night than sit and wait for a signal bar to come back.
This morning the wifi returned, the icon lost its diagonal line, and I never told the local model it had missed anything. It had not.
Yuki Halvorsen was born in Sapporo to a Norwegian father and a Japanese mother, wrote her first program on a school computer that took ten seconds to draw a circle, and spent eight years as an engineer in Stockholm before Tokyo pulled her back. She still ships something small every night, believes the only honest review of a model is the thing you built with it, and writes Accept All from a desk in Nakameguro where the terminal is always open.