OpenAI's nightmare: Deepseek R1 on a Raspberry Pi

A Chinese hedge fund-backed startup built an open weights model called R1 that outperforms OpenAI’s best models across multiple metrics. They achieved this feat with only $6 million and GPUs operating at half the memory bandwidth of OpenAI’s, using a mere 1% of typical resources. The model is designed to distill and optimize other models for slower hardware, enabling advanced AI performance on devices like the Raspberry Pi. This development challenges the belief that cutting-edge AI requires massive energy and GPU resources, undermining the competitive edge that large providers have relied on.

Raspberry Pi 5 runs AI models like Deep seek R1, but it is not equivalent to the substantially larger Deep seek R1 671b that outperforms ChatGPT. The high-end model requires extensive GPU compute typically achievable with a few Nvidia 309 cards. Users can bypass external services by leveraging local GPU power, experimenting with sophisticated AI models at home. This approach makes advanced, cost-effective AI experimentation within reach.

Deep Seek 671b on AmpereOne proves that with sufficient RAM, any computer can run advanced deep learning algorithms. It operates on a 192-core server at around 4 tokens per second, a performance that, while modest, is enough to showcase its potential. The system offers exceptional cost efficiency, avoiding the steep price tag of high-end setups near $100,000. Its impressive low power consumption of roughly 100 Watts further underscores its practical innovation.

A smaller 14B model runs on a Raspberry Pi, demonstrating that resource-constrained devices can support AI applications despite modest performance levels. The model processes roughly 1.2 tokens per second, indicating it isn’t designed to win any speed contests. Its functionality is sufficient for specific uses like simple chatbot interactions or debugging tasks, emphasizing practicality over raw speed.

Employing an external graphics card dramatically speeds up AI inference compared to CPU processing. An AMD W7700 with 16GB of VRAM can hold the entire model in memory, achieving a performance boost up to ten times faster. Measured outputs indicate a token generation rate of 20 to 50 tokens per second, with monitoring tools confirming that the GPU efficiently manages the workload.

The discussion highlights robust GPU performance on ARM platforms like Raspberry Pi and other ARM boards. AMD GPUs are delivering excellent results, while Intel's new open source drivers are showing promising progress, with potential for Nvidia support in the future. Boards equipped with full-size PCI slots underscore a significant leap forward in GPU capabilities on ARM systems this year.

The AI market is mired in a bubble where conventional desktop systems are unlikely, making way for innovations like custom ARM chips and specialized PCs. Nvidia experienced a dramatic loss of over half a trillion dollars in a single day following a major product launch, yet its stock remains significantly above 2023 levels amid ongoing hype. This observation suggests that achieving computational tasks doesn’t require consuming unsustainable global energy resources or constructing colossal infrastructures. Additionally, emerging AI models are exhibiting basic identification errors, as seen in their confusion over familiar cultural figures.