DE — heise online
30 Million Parameters: Local AI Language Model for the ESP32
AI1 min read
A YouTuber named Marina trained her own small AI language model, called PocketGPT, to run on a tiny ESP32-S3 microcontroller. It can hold very short chats, but it is slow and very limited.
What happened
Marina built a small AI chat model from zero and made it run on an ESP32-S3, a tiny computer chip. When asked "how are you?", it answered "I'm doing okay." after almost 12 seconds. Her video had more than 158,000 views since October 13.
Key facts
- The model has about 30 million parameters (the number values that store what it learned).
- It was trained on about 300 million tokens (words and word parts).
- Most training text came from AI-made children's stories and educational web pages. About one tenth came from public chat logs.
- Most weights are stored with only 4 bits each, so the model fits in the chip's memory.
- On a developer board with 16 MB of extra memory, it makes about 1.4 tokens per second.
How it compares
Another ESP32 project, esp32-ai, has a model of similar size (28.9 million parameters). It is about seven times faster, at nearly 10 tokens per second. But most of its parameters sit in a kind of word list, so its real calculating part is much smaller. Its makers say it cannot answer questions or follow instructions.
Why it matters
The project shows that a language model can run on a very cheap, low-power chip with no internet connection. This is useful for learning how AI works. But PocketGPT has a memory of only 64 tokens, and its answers can be wrong, repeat themselves, or drift off topic. Marina says it should never compete with modern AI models.
Glossary
- ESP32-S3
- A small, cheap microcontroller chip that can run simple programs and devices.
- PocketGPT
- The small AI chat model Marina trained for this project.