MentraOS – open-source Smart glasses OS

Here's the deal, you don't need any of this.

I have Rayban Metas and the hardware is great...but the software borders on being unhelpful. If they merely served a dumb camera and bluetooth headset to my phone they'd be an unbelievably good product.

Meta won't do this because they want to capture _everything_ going on, but I don't want to chat with Meta's AI, it is very bad, I want to chat with Gemini or ChatGPT and I can do so with their glasses but I must initiate that on my phone (Meta won't give you wakewords for OpenAI/Google of course).

So my suggestion here would be don't? There is no need for an app store or anything like that, just the thinest software layer you need to make the sunglass hardware work as a dumb bluetooth headset and remote camera for the user's phone.

This comment reminds me of a simple `esp32` project I saw recently that lets you send your LLM requests via SMS. It basically offloads everything. Particularly useful when you don't have a decent data connection, but can still send SMS.

edit: found it https://www.reddit.com/r/arduino/comments/1n7r3vl/a_textbot_...

I was about to make the exact same comment. But then I remembered that there are billions of people who buy products advertised on Facebook and TikTok because it's "cool" and "fun". So what do I know about the future of smart glasses OS? Probably nothing.

META wants to be the Android of smart glasses because they know it will be the next dominate form factor when we have desirable devices (also why they are starting with less hardware but a form factor people feel comfortable wearing in public)

Android XR is coming out with Moohan next month, if Visor ever comes out, it is believed that will eventually by on AXR. Apple still seems hobbled since Jobs left

It is hard for me to swallow the promise of smart glasses and I was dev-ing for the original Google glass.

It's awkward, battery life is a pittance, the display can be useful but only in select cases. Controls are always an issue. LLMs won't actually fix that - voice control is not the answer.

Disagree with you there having used Rayban Metas for about six months.

Always-on access to an LLM via voice is a useful and novel way of interacting with computers.

From trivial things like asking it about a landmark I'm seeing or when I'm driving to tell me about some historical event (almost like an on tap podcast), to slightly more useful things like asking it to add stuff to my calendar/reminders when I'm biking home.

It certainly isn't a replacement for a more robust interface, but it is a very nice way of using a computer while I'm out and about and don't want to pull out my phone.

That sounds nice but there's problems in reality.

How do the glasses serve as a "dumb camera to your phone"? What protocol do they use to do this? It doesn't exist. It's something that must be solved at the OS layer.

What if you want to use multiple apps? Are you going to spend 2 minutes each time disconnecting Bluetooth from one phone app, connecting to another, and then using it? No, you need to runtime that lets multiple apps access the sensors as needed.

Do you want to make an app that accesses the microphone? If you want to have translation app running at the same time that you're taking notes, then again you need some way to allow multiple apps to run at once.

MentraOS solves those problems.

(obex or opp or ftp, if you don't care about live previews. Nokia S60 could just do it, so could Windows Mobile 6.x and under. iOS/most Android, nope)

> How do the glasses serve as a "dumb camera to your phone"? What protocol do they use to do this? It doesn't exist. It's something that must be solved at the OS layer.

USB webcams have been a thing for years ;)

I have a pair of Xreal glasses and, while they don’t have a camera, they do have the other components. They are entirely dumb. You plug the USB cable into your phone/laptop/portable gaming device and that’s literally it.

The cable runs discreetly from the back of the ear and has the additional benefit that you don’t need a heavy battery built into the frame of the glasses.

So you definitely can have a XR glasses that are “dumb”.

"How do the glasses serve as a dumb camera to the phone": just like a USB camera. USB protocol, or USBIP. "yeah, but what OS" - what OS does a USB webcam need to be a USB webcam? That OS.

"What if you want to use multiple apps?" for a headset that's a window to a phone, you see the phone screen, the phone handles multitasking. Want to switch between apps? Then switch between apps on your phone, and you see the result.

"Do you want to make an app that accesses the microphone?" again, the phone does it. What OS do my bluetooth earphones run to be accessible from my phone?

I agree with what the person you're responding to wants: just an screen/audio interface with my phone. MentraOS is obviously not* aiming to be that, otherwise it wouldn't have any apps at all, especially not things like a "notes" app or any other app I already have on my phone.

The issue is as soon as you start trying to build an app ecosystem, you inevitably create the sort of opportunities business loves to exploit, and then all of a sudden I've got another layer for big tech to try extract stuff from me, when all I wanted was to be able to see my phone screen without having my phone directly in front of me - as someone who uses apps rather than develops them, I don't need another app store or more apps!

*Edit: having read some of their work culture, and the people involved, this isn't a project that's intended to be owned by humans, this is going to become the worst kind of big tech, or nothing.

XREAL is DisplayPort Alt Mode + USB for gyros. It's also wired only. DP needs 10-40Gbps of bandwidth, doesn't work wireless.

USB cameras also aren't natively supported on iOS/Android. You need apps. With apps comes lock-in opportunities which are never not tapped.

So "just use USB" doesn't make technical sense at all.

A real user-centric OS (like a full-fledged Linux distribution, not something intentionally crippled as badly as Android) would use something like PipeWire[1] for this. It's a project designed entirely around managing multiple multimedia devices so they can be accessed by multiple applications, even concurrently.

[1] https://pipewire.org/

PipeWire? I don't think you want that kind of thing for raw video output. You want display content to be on VRAM. The void between software jockeys and hardware world sometimes makes me feel numb.

> XREAL is DisplayPort Alt Mode + USB for gyros. It's also wired only.

I know what Xreal uses. As I said, I have a pair

> DP needs 10-40Gbps of bandwidth, doesn't work wireless.

And as I also said, having a cable is a feature, not a problem.

VR headsets are heavy and uncomfortable. USB powered XR glasses are not. And the reason for that is because you don’t need to make those XR glasses as literal portable computers with heavy batteries.

You might relish the idea of an ugly monstrosity that weighs as much as a laptop strapped to your head. Myself, I’d much rather have something that look and feel like sunglasses. If that means I need a discreet UsB cable behind my ear, then thats a small price to pay because they’d still look less stupid than wearing anything bulkier out in public.

> USB cameras also aren't natively supported on iOS/Android. You need apps. With apps comes lock-in opportunities which are never not tapped.

That’s not a limitation for all platforms though. And you’d have that problem on Android whatever solution you opted for. So it’s a moot point.

> So "just use USB" doesn't make technical sense at all.

It does and plenty of people, myself included, owning a pair of Xreal glasses are proof of that.

The problem here is not USB, it’s that you have very specific differing requirements and thus are dismissing the practical value myself and others have shared.

> The void between software jockeys and hardware world sometimes makes me feel numb.

Comments like the above are completely unnecessary and against the rules on HN

> VR headsets are heavy and uncomfortable. USB powered XR glasses are not. And the reason for that is because you don’t need to make those XR glasses as literal portable computers with heavy batteries.

no it's lenses and chassis. Lenses work precisely because of their density difference against air, so the better they are, the heavier they are. Chassis weigh a lot because they use impact resistant ABS and don't make them in forged Al-Li or Ti or molded Mg, which they should consider for hilarity, but then the product will cost like a bad joke. The mobile computer part weighs nothing, they're like somewhat soggy potato crisps. Those 0.8mm PCBs, boy they feel like cardstocks. Batteries weigh a bit, but they're also usually lipo pouches, like 0.5kg/L. You're not putting dozen 18650 into a VR headset.

Especially VR lenses are heavy and bulky because they need short focal lenses with massive pupils for max FOV and max transmittance. The panels tend to be way bigger than that for smart glasses thanks to Palmer Luckey which he deserves credit for. Smart glasses tend to use way smaller panels and prisms with fractions of FOVs relative to VR, like 1/6th? 1/12th? They carry some amount of weight but not nearly as much, especially if it's waveguide or holographic or working as pure fresnels.

I'm not going into the second half of this response. I am sorry but I don't think it's worth anyone's time if I explained why DP Alt don't count as USB and all that stuffs.

You’re arguing a lot but ignoring what other people are saying. It’s pretty clear from your other comments that you have distain for “software jockies”, but that doesn’t mean that we are idiots who don’t understand how hardware works.

Case in point: even if you took the lenses out, they’d still weigh more than a pare of sunglasses. You even admitted that yourself, but then you quickly brushed over that point.

So does it really matter that lenses are also heavy when we are talking specifically about the battery?

I also happen to know a thing or two about mobile computing hardware and there’s a bunch of stuff you’ve also neglected to mention that would add weight. But ultimately the battery alone is a compelling enough argument.

Let’s also remember that I wasn’t just talking about weight but bulk too. Even if you could get the weight down so it’s comparable to a pare of glasses (you couldn’t, but let’s assume for the moment that you do manage to break the laws of physics), it’s still going to be bulkier than a pair of glasses.

So even if you were right that the lenses were the only thing that matter about weight (which you’re not), it’s still just a moot point.

> I'm not going into the second half of this response. I am sorry but I don't think it's worth anyone's time if I explained why DP Alt don't count as USB and all that stuffs.

I’m not an idiot. I’m well aware that Xreal are using DP, and I’ve already pointed out before.

My point is all I need to do as an end user is plug in a USB-C cable and everything “just works”. The underlying protocol is largely irrelevant. It’s like saying “you’re not using wireless, you’re using 802.11ac…” literally zero consumers give a shit because it’s completely irrelevant to the UX of the device.

I would agree if I could use ChatGPT's advanced voice mode with them. It's purely a software/lock-in issue that I can't, but it means that I never wear my glasses, vs wearing them all the time.

They're so unobtrusive for chatting, that it would be amazing if we could get a capable LLM on the other side. Too bad we can't because corporations like walled gardens.

OpenAI walled garden should have Jony Ive's upcoming $6B hardware device for voice chat.

> I also happen to know a thing or two about mobile computing hardware

You don't. You just don't. You haven't seriously thought about making an HMD. You haven't heard of those microdisplay vendors or had frantically searched how to cheat those electrical requirements. You haven't gutted a mobile device and held the shells and non-compute parts in hand. You haven't even torn apart a phone. You don't know how injection molded Mg chassis feel which don't sit right with your brain.

Your views and opinions and distributions of are based on user side stories and dopamine releasing qualities the elements offer, which is great for marketing existing or virtual products, but isn't well connected to underlying hardware. It's like human figures reconstituted from Penfield's Homunculus, made completely out of proportion.

I already brought up the shell. From the way you fixated on the battery after going through that part, it clearly didn't even come to your mind that the shells can be heavier than it, which I'm sure would be the case for a lot of battery powered HMDs.

You called DP Alt an underlying "protocol". It's DisplayPort. They're like one-way PCIe x1. Back of magazine Agilent vs Teledyne LeCroy stuffs. Besides you called XREAL displays "entirely dumb" when they have more than a 2D sprite engine on board. I know it does 3DoF warping when the host is not whispering the right command. It's a stupid feature but I'm doubtful PS2 can handle that.

And you keep insisting, what appear to boil down to, "use USB dumbass". Nothing about that make any sense.

You’re making a lot of assumptions about me when you don’t know anything about me.

> From the way you fixated on the battery after going through that part, it clearly didn't even come to your mind that the shells can be heavier than it, which I'm sure would be the case for a lot of battery powered HMDs.

I mentioned the battery because I was making a comparison between wired vs wireless.

It’s really that simple.

> You called DP Alt an underlying "protocol". It's DisplayPort. They're like one-way PCIe x1

I also called it a specification elsewhere. What you’re actually doing there is calling out my incompetence at the English language. And being dyslexic I’d agree with you on that. I guess you’re going to insult me over that now too?

> Besides you called XREAL displays "entirely dumb"

No I didn’t. I said ‘“dumb”’ not “entirely dumb” and the quotes are important too because it clearly demonstrates I wasn’t using the term literally. The context was that they’re dumb compared to “smart” devices. Like how people talk about smart phones and “dumb phone even though “dumb phones” still have more computing power than Apollo 11.

It was a contextual statement, not a literal one. Hence the quotes.

> when they have more than a 2D sprite engine on board.

That depends on the model of the glasses. Mine don’t really do much processing compared to the higher end ones. Xreal make several different XR glasses. Though they all are wired (from what I recall).

The higher end ones do feel a lot more like smart devices, from what I’ve seen. Whereas the lower end ones need a companion device if you want to do anything beyond screening mirroring and audio.

Again, talking about UX here rather than components.

> And you keep insisting, what appear to boil down to, "use USB dumbass". Nothing about that make any sense.

Once again you’re rephrasing my comments in intentionally unflattering ways to misrepresent what I was saying.

I made the point that XR glasses can work without a full blown smart OS when driven by USB-C. And the fact that I literally own a pair of XR glasses that do exactly that, proves what I said not only makes sense, but is also factually correct.

———

Now you can continue to build strawman arguments, misquote me, make unfounded assumptions about me, and generally show bad HN etiquette if you want. But I have a family were my time is better spent. So I’ll leave you to find some other chump to troll this weekend

"How do the glasses serve as a "dumb camera to your phone"? What protocol do they use to do this? It doesn't exist. It's something that must be solved at the OS layer."

No, Snapchat did this just fine in the software layer with their glasses looooooong ago.

Some people prefer wired because of external battery, bandwidth, signal stability and integration/interop (e.g. usbc hubs). There are always tradeoffs in designs.

"USB cameras also aren't natively supported on iOS/Android. You need apps."

The attachable endoscope for my Ulefone Armor would disagree. It works with the stock Android camera app.

Lo and behold, you just need DRIVERS.

> discreet UsB cable behind my ear

If someone could create a fashionable, transparent usbc cable, perhaps replacing copper with indium tin oxide, it might change people's minds about wired.

Except people don't want to wear shades indoors, or if their eyes are find, glasses either. That is its Achilles heel.

Did they finally implement UVC in Camera2/CameraX? I thought that had been Soon(TM) for years.

No, you just implement the Camera2 API and camera HAL interface in your driver so your device can talk to all the applications that call that API.