←back to thread

3D + 2D: Testing out my cross-platform WASM graphics engine

579 points seanisom | 2 comments | 02 Apr 24 16:44 UTC | HN request time: 0s | source

I used to work at Adobe on the infrastructure powering big applications like Photoshop and Acrobat. One of our worst headaches was making these really powerful codebases work on desktop, web, mobile, and the cloud without having to completely rewrite them. For example, to get Lightroom and Photoshop working on the web we took a winding path through JavaScript, Google’s PNaCl, asm.js, and finally WebAssembly, all while having to rethink our GPU architecture around these devices. We even had to get single-threaded builds working and rebuild the UI around Web Components. Today the web builds work great, but it was a decade-long journey to get there!

The graphics stack continues to be one of the biggest bottlenecks in portability. One day I realized that WebAssembly (Wasm) actually held the solution to the madness. It’s runnable anywhere, embeddable into anything, and performant enough for real-time graphics. So I quit my job and dove into the adventure of creating a portable, embeddable WASM-based graphics framework from the ground up: high-level enough for app developers to easily make whatever graphics they want, and low-level enough to take full advantage of the GPU and everything else needed for a high-performance application.

I call it Renderlet to emphasize the embeddable aspect — you can make self-contained graphics modules that do just what you want, connect them together, and make them run on anything or in anything with trivial interop.

If you think of how Unity made it easy for devs to build cross-platform games, the idea is to do the same thing for all visual applications.

Somewhere along the way I got into YC as a solo founder (!) but mostly I’ve been heads-down building this thing for the last 6 months. It’s not quite ready for an open alpha release, but it’s close—close enough that I’m ready to write about it, show it off, and start getting feedback. This is the thing I dreamed of as an application developer, and I want to know what you think!

When Rive open-sourced their 2D vector engine and made a splash on HN a couple weeks ago (https://news.ycombinator.com/item?id=39766893), I was intrigued. Rive’s renderer is built as a higher-level 2D API similar to SVG, whereas the Wander renderer (the open-source runtime part of Renderlet) exposes a lower-level 3D API over the GPU. Could Renderlet use its GPU backend to run the Rive Renderer library, enabling any 3D app to have a 2D vector backend? Yes it can - I implemented it!

You can see it working here: https://vimeo.com/929416955 and there’s a deep technical dive here: https://github.com/renderlet/wander/wiki/Using-renderlet-wit.... The code for my runtime Wasm Renderer (a.k.a. Wander) is here: https://github.com/renderlet/wander.

I’ll come back and do a proper Show HN or Launch HN when the compiler is ready for anyone to use and I have the integration working on all platforms, but I hope this is interesting enough to take a look at now. I want to hear what you think of this!

Show context
wokwokwok ◴[03 Apr 24 05:31 UTC] No.39913917[source]▶
There is (surprisingly) no high level commentary on what this actually is, but people banging on about how nice it would be to have a high level cross platform GPU accelerated library.

...but this..?

> Graphics data and code can be developed together in the same environment, packaged together into a WebAssembly module called a renderlet, and rendered onto any canvas. With WebAssembly, we compile graphics code to portable bytecode that allows it to safely run on any processor and GPU

So what is a renderlet?

> The renderlet compiler is currently in closed preview - please contact us for more information.

Hm... what this seems to be is a C++ library that lets you take compiled WASM and run it to generate and render graphics.

Which, I think it is fair to say, it's surprising, because you can already render graphics using C++.

Only here, you can render graphics using an external WASM binary.

So, why?

Specifically, if you're already using C++:

1) Why use WASM?

2) Why use renderlet instead of webGPU, which is already a high level cross platform abstraction including shader definitions?

What is this even for?

> wander is designed to be a rendering engine for any high-performance application. It primarily is designed as the runtime to run renderlet bundles

...but, why would I use a renderlet, if I already need to be writing C++?

I. Get. It. A cross platform GPU accelerated rendering library you can use from any platform / browser / app would be great. ...but that is not what this is.

This is a C++ library runtime that you can use to run graphics in any circumstance where you you can currently use C++.

...but, in circumstances where I can use C++, I have many other options for rendering graphics.

Look at the workflow:

    Rendering code -> Renderlet compiler -> renderlet binary
    App -> load renderlet binary -> start renderlet runtime -> execute binary on runtime -> rendered
vs.

    App -> rendering code (WebGPU) -> rendered
or, if you writing a new cross platform API over the top of webGPU

    App -> Fancy api -> WebGPU -> rendered
I had a good read of the docs, but I honestly fail to see how this is more useful than just having a C++ library that renders graphics cross platform like SDL.

Shaders? Well, we also already have a good cross platform rendering library in webGPU; it already runs on desktop and browsers (maybe even some mobile devices); it already has a cross platform shader pipeline; it's already usable from C++.

I'm not going to deny the webGPU API is kind of frustrating to use, and the tooling for building WASM binaries is too, but... it does actually exist.

Is this like a 'alternative to webGPU' with a different API / easy mode tooling?

...or, have I missed it completely and there's something more to this?

replies(1): >>39914122 #
1. seanisom ◴[03 Apr 24 06:05 UTC] No.39914122[source]▶
Keeping it high level -

No, the goal is not to create a C++ API to give you GPU functions.

The C++ API for wander is used to embed the WebAssembly module of graphics code into the application. The API footprint is very small - load a file, pass parameters to it, iterate through the tree it produces.

This could be viewed as logically equivalent to programmatically loading a flash/swf file. Or similar to what Rive has built with a .riv, although this is static content, not code.

> 1) Why use WASM?

You're loading arbitrary, third-party code into an app - that is the renderlet. The benefit is to have a sandboxed environment to run code to put data on the GPU.

2) Why use renderlet instead of webGPU, which is already a high level cross platform abstraction including shader definitions?

WebGPU is a low-level API. If you are a graphics programmer, and want to build an app around WebGPU, go for it! A renderlet is more of a graphics plugin system than an entire first-party app.

> The renderlet compiler is currently in closed preview - please contact us for more information.

This is the system to build the renderlet. This is not writing raw C++ code to talk to WebGPU, this can be higher-level functions (build a grid, perform a geometric extrusion, generate a gradient) - you can see in the video it is a yaml specification. The compiler generate the necessary commands, vertex buffers, textures, etc, and soon, shaders to do this, and builds a Wasm module out of it.

> Is this like a 'alternative to webGPU' with a different API / easy mode tooling?

I certainly wouldn't describe it as an alternative to WebGPU, but easy(er) tooling to build graphics, yes.

> What is the use-case for 'I've compiled a part of my application only into a cross platform binary renderlet and I can now run that cross platform ... after I've compiled the rest of my application into a platform specific binary for the platform I'm running it on?'

Let's take an example - Temporal Anti-Aliasing. There are libraries that exist to implement this, or you can implement it through raw code. This requires changes structural changes to your pipeline - to your render targets, additional outputs to your shaders, running additional shaders, etc. Wouldn't it be nice to easy connect a module to your graphics pipeline that contains the code for this, and the shader stages, and works across platforms and graphics APIs, with data-driven configuration? That is the vision.

> ... rest of your application into WASM/platform native code... is that not strange? It seems strange to me

There is not really such a thing as a standalone Wasm application. It has seen great success as a data-driven plugin model. In a browser, it is hosted with / interacts with JavaScript. Even built for pure WASI, as a standalone app where everything is compiled into a single module, there is stil a runtime/host environment.

Does that help clarify?

replies(1): >>39914637 #
2. wokwokwok ◴[03 Apr 24 07:46 UTC] No.39914637[source]▶
> A renderlet is more of a graphics plugin system than an entire first-party app.

I see.

So this is basically flash?

A high level API to build binary application bundles (aka .swf files, ie. renderlets) and a runtime that lets you execute arbitrary applications in a sandbox.

renderlet = .swf file

wander = flash runtime

renderlet compiler = magic sauce, macromedia flash editor

yeah?

> Let's take an example - Temporal Anti-Aliasing. There are libraries that exist to implement this, or you can implement it through raw code.

Mhm. You can certainly do it in a cross platform way using webGPU, but I suppose I can see the vision of 'just download this random binary and it'll add SMAA' but it sounds a lot like "and then we'll have a marketplace where people can buy and sell GPU plugins" or "if you're building a web browser" rather than "and this is something that is useful to someone developing a visualization application from scratch".

The majority of these features could exist with just a C++ library and no requirement to 'pre-compile' some of your code into a renderlet... hosting external arbitrary 3rd party binaries in your application seems... niche.

Really, the only reason you would normally ever not just do it from source as a monolithic part of your application was if you didn't have the source code for some reason (eg. because you bought it as a WASM binary from someone).

Smells like Flash, and I'm not sure I like that, but I guess I can see the vision now, thanks for explaining.