LightTok Chip Turns Light Into AI Tokens On The Sensor Itself

Researchers in China have built a sensor that could cut the energy drones and other autonomous machines spend processing visual data.

Detailed Aug. 19 in the journal Nature Sensors, the chip is called LightTok. It converts raw light into tokens that AI models can read directly on the chip, skipping several power-hungry steps that conventional sensor systems rely on.

In standard visual perception, a sensor captures light, an analog-to-digital converter turns it into pixels, the data is stored, then shipped to a separate chip where the image is divided into tiles and each tile becomes a token. One widely cited study found an analog-to-digital converter accounts for 66% of an image sensor's energy use on average, and sending visual data to the cloud can push consumption higher still.

LightTok collapses those five stages into one by building sensing, memory and computation into the same pixel. The array uses a single-layer molybdenum disulfide floating-gate phototransistor that senses light, holds what it sensed, and factors that into a calculation.

"The chip physically eliminates data movement, which is the main source of energy waste," Liang Shi-Jun, a physics professor at Nanjing University, told Chinese state-run news agency Xinhua. "Light comes in, tokens come out."

In tests, LightTok hit 87.3% accuracy in image recognition while converting light into tokens 10 times more efficiently than the conventional multi-step process.

Its maximum resolution is currently just 32 by 32 photosensitive pixels - far below smartphone cameras, let alone drones. Miao Feng, director of Nanjing University's Institute of Brain-Inspired Intelligence, said the technology could be scaled up using the same complementary metal-oxide-semiconductor process used for smartphone and laptop chips.

Kumar Sokka, CEO of Acre Security, called the work a "small-scale demonstration" but said "the direction matters to anyone working in the physical world." Sokka, who was not involved in the research, said too much discussion of physical AI focuses on models, when the bigger challenge is the energy cost of getting what a sensor sees into a usable form at the point of sensing. Processing at the point of detection may enable pervasive physical AI, he added, "but it's not a cure for the whole problem."

Source: Llive Science