←back to thread

Show HN: I open-sourced my AI toy company that runs on ESP32 and OpenAI realtime

(github.com)
177 points akadeb | 8 comments | 22 Apr 25 14:10 UTC | HN request time: 0.001s | source | bottom

Hi HN! Last year the project I launched here got a lot of good feedback on creating speech to speech AI on the ESP32. Recently I revamped the whole stack, iterated on that feedback and made our project fully open-source—all of the client, hardware, firmware code.

This Github repo turns an ESP32-S3 into a realtime AI speech companion using the OpenAI Realtime API, Arduino WebSockets, Deno Edge Functions, and a full-stack web interface. You can talk to your own custom AI character, and it responds instantly.

I couldn't find a resource that helped set up a reliable, secure websocket (WSS) AI speech to speech service. While there are several useful Text-To-Speech (TTS) and Speech-To-Text (STT) repos out there, I believe none gets Speech-To-Speech right. OpenAI launched an embedded-repo late last year which sets up WebRTC with ESP-IDF. However, it's not beginner friendly and doesn't have a server side component for business logic.

This repo is an attempt at solving the above pains and creating a great speech to speech experience on Arduino with Secure Websockets using Edge Servers (with Deno/Supabase Edge Functions) for fast global connectivity and low latency.

1. drakenot ◴[22 Apr 25 15:41 UTC] No.43763477[source]▶
Something that really kills the 'effect' of most of the Voice > AI demos that I see is the cold start / latency.

The OpenAI "Voice Mode" is closer, but when we can have near instantaneous and natural back and forth voice mode, that will be a big in terms of it feeling magical. Today, it is say something, awkwardly wait N seconds then listen to the reply and sometimes awkwardly interrupt it.

Even if the models were no smarter than they are today, if we could crack that "conversational" piece and performance piece, it would be a big difference in my opinion.

replies(3): >>43763547 #>>43764293 #>>43770877 #
2. Sean-Der ◴[22 Apr 25 15:48 UTC] No.43763547[source]▶
I think it will always feel unnatural as long as 'AI Speech' is turn based. Right now developers used Voice Activity Detection to detect when the user has stopped talking.

What would be REALLY cool is if we had something that would interrupt you during conversation like talking with a real human.

replies(1): >>43763880 #
3. conductr ◴[22 Apr 25 16:24 UTC] No.43763880[source]▶
I can see how interruptions would prove even more unnatural and annoying pretty quick. There's a lot of nuance in knowing how to interrupt properly and often, people that interrupt only do so quickly, then yield, allow person to finish then resume - very situational and tons of nuance. Otherwise, with current level of sophistication, you'd just have the AI talking over you the entire time, not allowing you to complete your thoughts/questions/commands/etc and people would quickly be more frustrated and just turn it off.
replies(1): >>43771697 #
4. akadeb ◴[22 Apr 25 17:12 UTC] No.43764293[source]▶
Yeah the way I am handling this is turn detection which feels unnatural. I like how Livekit handles turn detection with a small model[0][1] [0]https://www.youtube.com/watch?v=EYDrSSEP0h0 [1]https://docs.livekit.io/agents/build/turns/turn-detector/

``` turn_detection: { type: "server_vad", threshold: 0.4, prefix_padding_ms: 400, silence_duration_ms: 1000, }, ```

5. dgellow ◴[23 Apr 25 11:32 UTC] No.43770877[source]▶
Have you recently tried OpenAI voice mode from ChatGPT Plus? It's basically what you describe
replies(1): >>43773429 #
6. mst ◴[23 Apr 25 13:05 UTC] No.43771697{3}[source]▶
I absolutely agree with your analysis wrt current tech - however, I suspect the person you're replying to is talking about "what would be really cool" in terms of it happening in a future where the relevant underpinnings had advanced to the point where it could actually manage the situational/nuance stuff properly.

I almost certainly wouldn't want to use something that tried to implement it now but it's a lovely dream and the state of the art keeps advancing at quite the speed (i.e. faster than I would have predicted, even when I do my best to take into account that it keeps advancing faster than I would have predicted ;).

7. drakenot ◴[23 Apr 25 15:43 UTC] No.43773429[source]▶
Yes, I mentioned this in the comment.

I think it is closer, although still even it has a cold start problem. Once you are connected and in-session, it is a better experience.

There is still some "turn based" conversational aspect to it that can be awkward but it is much better. It also helps that you can "tap and hold" to override, which is a bit of a hack but works well in practice for that mobile use-case.

replies(1): >>43777235 #
8. dgellow ◴[23 Apr 25 22:14 UTC] No.43777235{3}[source]▶
Sorry what I meant was: have you tried _recently_? I feel that what you describe is the old implementation. The current version, with a colorful bubble, has very low latency and starts almost instantly. The older version has a UI composed of a white dot on a black background, and is pretty slow. But the more recent one is pure magic IMHO.