PrismML's Bonsai 2 Squeezes a 27B Model Onto Your PC

PrismML, a startup founded by Caltech researchers, has released Bonsai 2 27B, a compressed version of Alibaba's widely used open source Qwen model. The company says the new model shrinks the original down to 5.9 GB, small enough to run on a PC and possibly a high-end smartphone.

The compression approach relies on what PrismML calls ternary weights, reducing each weight to one of three values: +1, -1, or 0. The company claims Bonsai 2 retains 98% of Qwen's aggregate benchmark scores, up from 95% for the first Bonsai, released in March. Those earlier models have collectively been downloaded more than 13 million times, according to PrismML.

CEO Babak Hassibi, a Caltech professor specializing in compression, told TechCrunch that compression will likely always cost some performance. He added that larger models should be easier to compress without losing capability, and said the next releases, expected within a couple of months, will target the several-hundred-billion-parameter range.

Adviser Ion Stoica, a Databricks co-founder and director of Berkeley's Sky Computing Lab, said the appeal is running advanced models directly on devices users already own, keeping them free and private. PrismML has raised a $22.25 million seed round from Khosla Ventures, Cerberus Capital, and Caltech. TechCrunch reported rumors of talks with Apple; Hassibi declined to comment on them.

Source: TechCrunch