←back to thread

Show HN: I open-sourced my AI toy company that runs on ESP32 and OpenAI realtime

(github.com)
177 points akadeb | 1 comments | 22 Apr 25 14:10 UTC | HN request time: 0.246s | source

Hi HN! Last year the project I launched here got a lot of good feedback on creating speech to speech AI on the ESP32. Recently I revamped the whole stack, iterated on that feedback and made our project fully open-source—all of the client, hardware, firmware code.

This Github repo turns an ESP32-S3 into a realtime AI speech companion using the OpenAI Realtime API, Arduino WebSockets, Deno Edge Functions, and a full-stack web interface. You can talk to your own custom AI character, and it responds instantly.

I couldn't find a resource that helped set up a reliable, secure websocket (WSS) AI speech to speech service. While there are several useful Text-To-Speech (TTS) and Speech-To-Text (STT) repos out there, I believe none gets Speech-To-Speech right. OpenAI launched an embedded-repo late last year which sets up WebRTC with ESP-IDF. However, it's not beginner friendly and doesn't have a server side component for business logic.

This repo is an attempt at solving the above pains and creating a great speech to speech experience on Arduino with Secure Websockets using Edge Servers (with Deno/Supabase Edge Functions) for fast global connectivity and low latency.

Show context
hoppp ◴[22 Apr 25 16:26 UTC] No.43763898[source]▶
Its great.lovely. but on the long run these toys rely on subscription payment?

Both the supabase Api and OpenAI billing is per api call.

So the lovely talking toys can die if the company stops being profitable.

I would love to see a version with decent hardware that runs a local model, that could have a long lifespan and work offline.

replies(2): >>43764050 #>>43791205 #
xp84 ◴[22 Apr 25 16:42 UTC] No.43764050[source]▶
> lovely talking toys can die if the company stops being profitable.

This is a good point to me as a parent -- in a world where this becomes a precious toy, it would be a serious risk of emotional pain if the child experienced this scenario like the death of a pet or friend.

> version with decent hardware that runs a local model

I feel like something small and efficient enough to meet that (today) would be dumb as a post. Like Siri-level dumb.

Personally, I'd prefer a toy which was tethered to a home device. Without a cloud (and thus commercial) dependency, the toy wouldn't be 'smart' outside of Wi-fi range, but I'd design it so that it got 'sleepy' when away from Wi-fi, able to be "woken up" and, in that state, to respond to a few phrases with canned, Siri-like answers. Perhaps new content could be made up for it daily and downloaded to local storage while at home, so that it could still "tell me a story" offline etc.

replies(1): >>43765586 #
1. scottmcf ◴[22 Apr 25 19:46 UTC] No.43765586[source]▶
> This is a good point to me as a parent -- in a world where this becomes a precious toy, it would be a serious risk of emotional pain if the child experienced this scenario like the death of a pet or friend.

We've already seen this exact scenario play out with "Moxie" a few months ago:

https://www.axios.com/2024/12/10/moxie-kids-robot-shuts-down