Pretty cool. Do you have a list of features you plan to support and plan to cut? Also, how much does this differ from the DOM impls that test frameworks use? I recall Jest or someone sporting such a feature.
replies(1):
Why start over? We’ve worked a lot with Chrome headless at our previous company, scraping millions of web pages per day. While it’s powerful, it’s also heavy on CPU and memory usage. For scraping at scale, building AI agents, or automating websites, the overheads are high. So we asked ourselves: what if we built a browser that only did what’s absolutely necessary for headless automation?
Our browser is made of the following main components:
- an HTTP loader
- an HTML parser and DOM tree (based on Netsurf libs)
- a Javascript runtime (v8)
- partial web APIs support (currently DOM and XHR/Fetch)
- and a CDP (Chrome Debug Protocol) server to allow plug & play connection with existing scripts (Puppeteer, Playwright, etc).
The main idea is to avoid any graphical rendering and just work with data manipulation, which in our experience covers a wide range of headless use cases (excluding some, like screenshot generation).
In our current test case Lightpanda is roughly 10x faster than Chrome headless while using 10x less memory.
It's a work in progress, there are hundreds of Web APIs, and for now we just support some of them. It's a beta version, so expect most websites to fail or crash. The plan is to increase coverage over time.
We chose Zig for its seamless integration with C libs and its comptime feature that allow us to generate bi-directional Native to JS APIs (see our zig-js-runtime lib https://github.com/lightpanda-io/zig-js-runtime). And of course for its performance :)
As a company, our business model is based on a Managed Cloud, browser as a service. Currently, this is primarily powered by Chrome, but as we integrate more web APIs it will gradually transition to Lightpanda.
We would love to hear your thoughts and feedback. Where should we focus our efforts next to support your use cases?
But of course we plan to add others features, including
- tight integration with LLM
- embed mode (as a C library and as a WASM module) so you can add a real browser to your project the same way you add libcurl
So yes it could be used in a serverless platform like Cloudflare workers. Our startup time is a huge advantage here (20ms vs 600ms for Chrome headless in our local tests).
Regarding v8 in Cloudflare workers I think we can not used directly, ie. we still need to embed a JS engine in the wasm module.