Steam Frame

Foveated streaming! That's a great idea. Foveated rendering is complicated to implement with current rendering APIs in a way that actually improves performance, but foveated streaming seems like a much easier win that applies to all content automatically. And the dedicated 6 GHz dongle should do a much better job at streaming than typical wifi routers.

> Just like any SteamOS device, install your own apps, open a browser, do what you want: It's your PC.

It's an ARM Linux PC that presumably gives you root access, in addition to being a VR headset. And it has an SD card slot for storage expansion. Very cool, should be very hackable. Very unlike every other standalone VR headset.

> 2160 x 2160 LCD (per eye) 72-144Hz refresh rate

Roughly equivalent resolution to Quest 3 and less than Vision Pro. This won't be suitable as a monitor replacement for general desktop use. But the price is hopefully low. I'd love to see a high-end option with higher resolution displays in the future, good enough for monitor replacement.

> Monochrome passthrough

So AR is not a focus here, which makes sense. However:

> User accessible front expansion port w/ Dual high speed camera interface (8 lanes @ 2.5Gbps MIPI) / PCIe Gen 4 interface (1-lane)

Full color AR could be done as an optional expansion pack. And I can imagine people might come up with other fun things to put in there. Mouth tracking?

One thing I don't see here is optional tracking pucks for tracking objects or full body tracking. That's something the SteamVR Lighthouse tracking ecosystem had, and the Pico standalone headset also has it.

More detail from the LTT video: Apparently it can run Android APKs too? Quest compatibility layer maybe? There's an optional accessory kit that adds a top strap (I'm surprised it isn't standard) and palm straps that enable using the controllers in the style of the Valve Index's "knuckles" controllers.

> Roughly equivalent resolution to Quest 3 and less than Vision Pro. This won't be suitable as a monitor replacement for general desktop use. But the price is hopefully low.

Question, what is the criteria for deciding this to be the case? Could you not just move your face closer to the virtual screen to see finer details?

Not only would it be a chore to constantly lean in closer to different parts of your monitor to see full detail, but looking at close-up objects in VR exacerbates the vergence-accommodation mismatch issue, which causes eye strain. You would need varifocal lenses to fix this, which have only been demonstrated in prototypes so far.

Foveated streaming is wild to me. Saccades are commonly as low as 20-30ms when reading text, so guaranteeing that latency over 2.4Ghz seems Sisyphean.

I wonder if they have an ML model doing partial upscaling until the eyetracking state is propagated and the full resolution image under the new fovea position is available. It also makes me wonder if there's some way to do neural compression of the peripheral vision optimized for a nice balance between peripheral vision and hints in the embedding to allow for nicer upscaling.

There's no precise criteria but the usual measure is ppd (pixels per degree) and it needs to be high enough such that detailed content (such as text) displayed at a reasonable size is clearly legible without eye strain.

> "Could you not just move your face closer to the virtual screen to see finer details?"

Sure, but then you have the problem of, say, using an IMAX screen as your computer monitor. The level of head motion required to consume screen content (i.e., a ton of large head movements) would make the device very uncomfortable quite quickly.

The Vision Pro has about ~35ppd and generally people seems to think it hits the bar for monitor replacement. Meta Quest 3 has ~25ppd and generally people seem to think it does not. The Steam Frame is specs-wise much closer to Quest 3 than Vision Pro.

There are some software things you can do to increase legibility of details like text, but ultimately you do need physical pixels.

They're doing it over 6GHz, if I understand correctly, which with a dedicated router gets you to a reasonable latency with reasonable quality even without foveated rendering (with e.g. a Quest 3).

With foveated rendering I expect this to be a breeze.

It was hard for me to believe as well but streaming games wirelessly on a Quest 2 was totally possible and surprisingly latency-free once I upgraded to wifi 6 (few years ago)

It works a lot better than you’d expect at face value.

Couldn't you get around that by having a "zoom" feature on a very large but distant monitor?

Even 5.8Ghz is getting congested. There's a dedicated router in this case (a USB fob), but you still have to share spectrum with the other devices. And at the 160Mhz symbol rate mode on WiFi6, you only have one channel in the 5.8GHz spectrum that needs to be shared.

Not so much of an issue when neighbors with paper thin walls see that 6ghz as a -87 signal

That said, in the US it is 1200MHz aka 5.925 GHz to 7.125 GHz.

Why hasn't Meta tried this given the huge amount of R&D they've put into VR and they had literally John Carmack on the team in the past?

> Roughly equivalent resolution to Quest 3 and less than Vision Pro. This won't be suitable as a monitor replacement for general desktop use.

The real limiting factor is more likely to be having a large headset on your face for an extended period of time, combined with a battery that isn't meant for all-day use. The resolution is fine. We went decades with low resolution monitors. Just zoom in or bring it closer.

They prioritized cost, so they omitted eye tracking hardware. They've also bet more on standalone apps rather than streaming from a PC. These are reasonable tradeoffs. The next Quest may add eye tracking, who knows. Quest Pro had it but was discontinued for being too expensive.

We'll have to wait on pricing for Steam Frame, but I don't expect them to match Meta's subsidies, so I'm betting on this being more expensive than Quest. I also think that streaming from a gaming PC will remain more of a niche thing despite Valve's focus on it here, and people will find a lot of use for the x86/Windows emulation feature to play games from their Steam library directly on the headset.

More of an issue when your phone's wifi or your partner watching a show while you game is eating into that one channel in bursts, particularly since the dedicated fob means that it's essentially another network conflicting with the regular WiFI rather than deeply collaborating for better real time guarantees (not that arbitrary wifi routers would even support real time scheduling).

MIMO helps here to separate the spectrum use by targeted physical location, but it's not perfect by any means.

Yes. You can make a low-resolution monitor (like 800x600px, once upon a time a usable resolution) and/or provide zoom and panning controls

I've tried that combination in an earlier iteration of Lenovo's smart glasses, and it technically works. But the experience you get is not fun or productive. If you need to do it (say to work on confidential documents in public) you can do it, but it's not something you'd do in a normal setup

The One Big Beautiful Bill fixed that. Now a large part of this spectrum will be sold out for non-WiFi use.

IMO there is not much reason to use WiFi 6 for almost anything else. I have a WiFi 6 router set up for my Quest 3 for PC streaming, and everything else sits on its 5GHz network. And since it doesn't really go through walls, I think this is a non-issue?

The Frame itself here is a good example actually - using 6GHz for video streaming and 5GHz for wifi, on separate radios.

My main issue with the Quest in practice was that when I started moving my head quickly (which happens when playing faster-paced games) I would get lag spikes. I did some tuning on the bitrate / beam-forming / router positioning to get to an acceptable place, but I expect / hope that here the foveated streaming will solve these issues easily.

Foveated streaming is cool. FWIW the Vision Pro does that for their Mac virtual display as well, and it works really well to pump a lot more pixels through.

2k X 2k doesn't sound low res it is like full HD, but with twice vertical. My monitor is 1080p.

Never tried VR set, so I don't know if that translates similarly.

VR does need a lot of resolution when trying to display text.

Can get away with less for games where text is minimized (or very large)

What do you mean? What part have they not tried?

Whether or not we used to walk to school uphill both ways, that won't make the resolution fine.

To your point, I'd use my Vision Pro plugged in all day if it was half the weight. As it stands, its just too much nonsense when I have an ultrawide. If I were 20 year old me I'd never get a monitor (20 year old me also told his gf iPad 1 would be a good laptop for school, so,)

It's just about what pixel per degree will get you close to the modern irl setup. Obviously it's enough for 80 char consoles but you'd need to dip into large fonts for a desktop.

Even the vision pro at 35ppd simply isn't close to the PPD you can get from a good desktop monitor (we can calculate PPD for desktop monitors too, using size and viewing distance).

Apple's "retina" HiDPI monitors typically have PPD well beyond 35 at ordinary viewing distances, even a 1080p 24 inch monitor on your desk can exceed this.

For me personally, 35ppd feels about the minimum I would accept for emulating a monitor for text work in a VR headset, but it's still not good enough for me to even begin thinking about using it to replace any of my monitors.

> https://phrogz.net/tmp/ScreenDensityCalculator.html

If you mean foveated streaming - It’s available on the Quest Pro with Steam Link.

Oh yeah for sure. Most people seem to accept that 35ppd is "good enough" but not actually at-par with a high quality high-dpi monitor.

I agree with you - I would personally consider 35ppd to be the floor for usability for this purpose. It's good in a pinch (need a nice workstation setup in a hotel room?) but I would not currently consider any extant hardware as full-time replacements for a good monitor.

The thing is that I'd expect foveated rendering to increase latency issues, not help them like it does for bandwidth concerns. During a lag spike you're now looking at an extremely down sampled image instead of what in non foveated rendering had been just as high quality.

Now I also wonder if an ML model could also work to help predict fovea location based on screen content and recent eye trackng data. If the eyes are reading a paragraph, you have a pretty good idea where they're going to go next for instance. That way a latency spike that delays eye tracking updates can be hidden too.

I worked on a foveated video streaming system for 3D video back in 2008, and we used eye tracking and extrapolated a pretty simple motion vector for eyes and ignored saccades entirely. It worked well, you really don't notice the lower detail in the periphery and with a slightly over-sized high resolution focal area you can detect a change in gaze direction before the user's focus exits the high resolution area.

Anyway that was ages ago and we did it with like three people, some duct tape and a GPU, so I expect that it should work really well on modern equipment if they've put the effort into it.

It will be interesting to see how the X86 emulation plays out. In the Verge review of the headset they mentioned stutters when playing on the headset due to having to 'recompile x86 game code on the fly', but they may offer precompiled versions which can be downloaded ahead of time, similar to the precompiled shaders the Steam Deck downloads.

If they get everything working well I'm guessing we could see an ARM powered Steam Deck in the future.

Despite the fact it uses a Qualcomm chip, I'm curious on whether it retains the ability to load alternative OS's like other Steam hardware.

Oh goody! I hope some of it can be used for DRM encrypted TV broadcasts too.

My understanding is that the foveated rendering would reduce bandwidth requirements enough that latency spikes become effectively non-existent.

We’ll see in practice - so far all hands-on reviewers said the foveated rendering worked great, with one trying to break it (move eyes quickly left right up down from edge to edge) and not being able to - the foveated rendering always being faster.

I agree latency spikes would be really annoying if they end up being like you suggest.

Foveated rendering very clearly works well with a dedicated connection, wiht predictable latency. My question was more about the latency spikes inherent in a ISM general use band combined with foveated rendering, which would make the effects of the latency spikes even worse.

The real trick is not over complicating things. The goal is to have high fidelity rendering where the eye is currently focusing so to solve for saccades you just build a small buffer area around the idealized minimum high res center and the saccades will safely stay inside that area within the ability of the system to react to the larger overall movements.

Picture demonstrating the large area that foveated rendering actually covers as high or mid res: https://www.reddit.com/r/oculus/comments/66nfap/made_a_pic_t...

> Mouth tracking?

What a vile thought in the context of the steam… catalogue.

For a small taste of what using that might be like turn on screen magnification on your existing computers. It's technically usable but not particularly productive or pleasant to use if you don't /have/ to use it.

I did the math on this site and I'd have to hunch less than a foot from the screen to hit 35 PPD on my work provided Thinkpad X1 Carbon with a 14" 1920x1200 screen. My usual distance is nearly double that so my ppd normally is more like 70 ppd, roughly.

https://phrogz.net/tmp/ScreenDensityCalculator.html#find:dis...

Yes but that can create major motion sickness issues - motion that does not correspond top the user's actual physical movements create a dissonance that is expressed as motion sickness for a large portion of the population.

This is the main reason many VR games don't let you just walk around and opt for teleportation-based movement systems - your avatar moving while your body doesn't can be quite physically uncomfortable.

There are ways of minimizing this - for example some VR games give you "tunnel vision" by blacking out peripheral vision while the movement is happening. But overall there's a lot of ergo considerations here and no perfect solution. The equivalent for a virtual desktop might be to limit the size of the window while the user is zooming/panning.

Your 2K monitor occupies something like a 20-degree field of view from a normal sitting position/distance. The 2K resolution in a VR headset covers the entire field of view.

So effectively your 1080p monitor has ~6x the pixel density of the VR headset.

And foveated streaming has a 1-2ms wireless latency on modern GPUs according to LTT. Insane.

They're probably thinking of it in comparison to the Apple Pro which attempts to do some facial tracking of the bottom of your face to inform their 'Personas', it notably still fails quite badly on bearded people where it can't see the bottom half of the face well.

The problem is that 2k square is spread across the whole FOV of the headset so when it's replicating a monitor unless it's ridiculously close to your face a lot of those pixels are 'wasted' in comparison to a monitor with similar stats.

Have a look at this video by Dave2D. In his hands-on, he was very impressed with foveated streaming https://youtu.be/356rZ8IBCps.

You're talking about "Wi-Fi 6" not "6 GHz Wi-Fi".

"6 GHz Wi-Fi" means Wi-Fi 6E (or newer) with a frequency range of 5.925–7.125 GHz, giving 7 non-overlapping 160 MHz channels (which is not the same thing as the symbol rate, it's just the channel bandwidth component of that). As another bonus, these frequencies penetrate walls even less than 5 GHz does.

I live on the 3rd floor of a large apartment complex. 5 GHz Wi-Fi is so congested that I can get better performance on 2.4 in a rural area, especially accounting for DFS troubles in 5 GHz. 6 GHz is open enough I have a non-conflicting 160 MHz channel assigned to my AP (and has no DFS troubles).

Interestingly, the headset supports Wi-Fi 7 but the adapter only supports Wi-Fi 6E.

Most people in what age group?

I'm 53 and the Quest 3 is perfectly good as a monitor replacement.

At 100fps (mid range of the framerate), you need to deliver a new frame every 10ms anyway, so a 20ms saccade doesn't seem like it would be a problem. If you can't get new frames to users in 30ms, blur will be the least of your problems, when they turn their head, they'll be on the floor vomiting.

Are you saying ppd requirements for comfortable usage vary with age?

Wow, that's just 1 frame of latency at 60 fps.

Edit: Nevermind, I'm dumb. 1/60th of a second is 16 milliseconds, not 1.6 milliseconds.

I'm in the same boat. Due to my vision not being perfect even after correction, a Quest 3 is entirely sufficient.

I guess there's a market for this but I'm personally disappointed that they've gone with the "cram a computer into the headset" route. I'd much rather have a simpler, more compact dumb device like the Bigscreen Beyond 2, which in exchange should prove much lighter and more comfortable to wear for long time periods.

The bulk and added component cost of the "all in one" PC/headset models is just unnecessary if you already have a gaming PC.

Enough bandwidth to absolve any latency issues over a wireless connection is not really a thing for a low latency use case like foveated rendering.

What do you do when another device on the main wifi network decides to eat 50ms of time in the channel you use for the eye tracking data return path?

You can get a Beyond if that's what you want. It's an amazing device, and will be far more comfortable and higher resolution than this one. Valve has supported Bigscreen in integrating Lighthouse tracking, and I hope that they continue that support by somehow allowing them to integrate the inside-out tracking they've developed for this device in the next version of the Beyond.

I gathered as much, but still.

I agree. Hopefully Bigscreen continues making hardware. I still have the original bigscreen beyond and im very happy with it besides the glare.

Much less than, 1 frame is 16ms

That would probably add a lot of extra weight and it would need to make the device bigger.

I don't think it would be too bad. Cameras are tiny. The processing would still happen on the PC, and you could delete the lighthouse tracking sensors. I guess the hardest part would be sending that much camera data back to the PC over the cable.

I think it works really well to pump the same amount of pixels, just focusing them on the more important parts.

60 fps is 16.67 ms per frame.

I'm personally quite hyped to see the first commercially available Linux-based standalone VR headset announced. This thing is quite a bit lighter than any of the existing "cram a computer in" solutions.

I believe all communication with the dongle is on 6GHz - both the video and the return metadata.

So again, you just make sure the 6GHz band in the room is dedicated to the Frame and its dongle.

The 5GHz is for WiFi.

Pretty funny to me that you're backseat engineering Valve on this one. If it didn't have a net benefit they wouldn't have announced it as a feature yet lmao

They crammed a computer into the headset, but UNLIKE Meta's offerings, this is indeed an actual computer you can run linux on. Perhaps even do standard computer stuff inside the headset like text editing, Blender modeling, or more.

We get by with lower resolution monitors with lower pixel density all the time.

This all sounds a bit like the “better horse” framing. Maybe richer content shouldn’t be consumed as primarily a virtualized page. Maybe mixing font sizes and over sized text can be a standard in itself.

I'm not sure most of us do anymore - see my 1080p/24 inch example.

No one who has bought almost any MacBook in the last 10 years or so has had PPD this low either.

One can get by with almost anything in a pinch, it doesn't mean its desirable.

Pixel density != PPD either, although increasing it can certainly help PPD. Lower density desktop displays routinely have higher PPD than most VR headsets - viewing distance matters!

> Despite the fact it uses a Qualcomm chip, I'm curious on whether it retains the ability to load alternative OS's like other Steam hardware.

I think it should: we have Linux support/custom operating systems on Snapdragon 8 Gen 2 devices right now today, and the 8 Gen 3 has upstream support already AFAIK

https://rocknix.org/devices/ayn/odin2/

No, thats between 0.06 and 0.12 frame latency on 60fps. It's not even a frame on 144Hz (1s/144≈7ms)

I think part of getting by with a lower PPD is the IRL pixels are fixed and have hard boundaries that OS affordances have co-evolved with.

(pixel alignment via lots of rectangular things - windows, buttons; text rendering w/ that in mind; "pixel perfect" historical design philosophy)

The VR PPD is in arbitrary orientations which will lead to more aliasing. MacOS kinda killed their low-dpi experience via bad aliasing as they moved to the hi-dpi regime. Now we have svg-like rendering instead of screen-pixel-aligned baked rasterized UIs.

Always PIP, Pump Important Pixels

MU-MIMO is very nice.

Different spectrum. They're grabbing old radar ranges.

Also talking about adding more spectrum to the existing ISM 6GHz band.

The battery isn't an issue if you're stationary, you can plug it in.

The resolution is a major problem. Old-school monitors used old-school OSes that did rendering suitable for the displays of the time. For example, anti-aliased text was not typically used for a long time. This meant that text on screen was blocky, but sharp. Very readable. You can't do this on a VR headset, because the pixels on your virtual screen don't precisely correspond with the pixels in the headset's displays. It's inevitably scaled and shifted, making it blurry.

There's also the issue that these things have to compete with what's available now. I use my Vision Pro as a monitor replacement sometimes. But it'll never be a full-time replacement, because the modern 4k displays I have are substantially clearer. And that's a headset with ~2x the resolution of this one.

I was worried about the built in computer as well, but then I found out it's only 185g. It is 78g more than the Bigscreen Beyond 2, but it's still pretty light.

I keep hearing this argument, and it baffles me. I find that, as I age and my vision gets worse, I need progressively finer text rendering. Using same-size displays (27") at the same distance, with text the same physical size on screen, 1440p gives me a much worse reading experience than 4k with 2x scaling.

That's pretty quick. I've heard that in ideal circumstances Wi-Fi 6 can get close to 5ms and Wi-Fi 7 can get down to 2ms.

I's impressive if they're really able to get below 2ms motion-to-photon latency, given that modern consumer headsets with on-device compute are also right at that same 2ms mark.

> Foveated streaming! That's a great idea.

Back when I was in Uni, so late 80s or early 90s, my dad was Project Manager on an Air Force project for a new F-111 flight simulator, when Australia upgraded the avionics on their F-111 fighter/bombers.

The sim cockpit had a spherical dome screen and a pair of Silicon Graphics Reality Engines. One of them projected an image across the entire screen at a relatively low resolution. The other projector was on a turret that pan/tilted with the pilot's helmet, and projected a high resolution image but only in a perhaps 1.5m circle directly in from of where the helmet was aimed.

It was super fun being the project manager's kid, and getting to "play with it" on weekends sometimes. You could see what was happening while wearing the helmet and sitting in the seat if you tried - mostly ny intentionally pointing your eyes in a different direction to your head - but when you were "flying around" it was totally believable, and it _looked_ like everything was high resolution. It was also fun watching other people fly it, and being able to see where they were looking, and where they weren't looking and the enemy was speaking up on them.

I think there is a missing number here: angular resolution of human eyeballs is believed to be ~60 ppd(some believes it's more like 90).

I'm guessing it's main use case will be VR chat syncing mouths to avatars.