←back to thread

Show HN: Magnitude – open-source, AI-native test framework for web apps

(github.com)
179 points anerli | 1 comments | 25 Apr 25 17:00 UTC | HN request time: 0.251s | source

Hey HN, Anders and Tom here - we’ve been building an end-to-end testing framework powered by visual LLM agents to replace traditional web testing.

We know there's a lot of noise about different browser agents. If you've tried any of them, you know they're slow, expensive, and inconsistent. That's why we built an agent specifically for running test cases and optimized it just for that:

- Pure vision instead of error prone "set-of-marks" system (the colorful boxes you see in browser-use for example)

- Use tiny VLM (Moondream) instead of OpenAI/Anthropic computer use for dramatically faster and cheaper execution

- Use two agents: one for planning and adapting test cases and one for executing them quickly and consistently.

The idea is the planner builds up a general plan which the executor runs. We can save this plan and re-run it with only the executor for quick, cheap, and consistent runs. When something goes wrong, it can kick back out to the planner agent and re-adjust the test.

It’s completely open source. Would love to have more people try it out and tell us how we can make it great.

Repo: https://github.com/magnitudedev/magnitude

Show context
arendtio ◴[26 Apr 25 07:52 UTC] No.43801710[source]▶
It looks pretty cool. One thing that has bothered me a bit with Playwright is audio input. With modern AI applications, speech recognition is often integrated, but with Playwright, using voice as an input does not seem straightforward. Given that Magnitude has an AI focus, adding a feature like that would be great:

  test('can log in and see correct settings')
    .step('log in to the app')
      .say('my username is user@example.com')
replies(1): >>43804506 #
anerli ◴[26 Apr 25 15:34 UTC] No.43804506[source]▶
Huh that’s an interesting use case. Yeah using an AI driven system definitely opens up some cool possibilities that aren’t possible with playwright alone. Would be curious to hear more about what you’re trying to test this way with audio.
replies(1): >>43810012 #
1. arendtio ◴[27 Apr 25 07:05 UTC] No.43810012[source]▶
I have this project planning prototype [1], where you can use audio to plan a project, like' Please create a project plan for an example project and make assumptions' or 'During the last two weeks of the year, nobody is available due to the holidays, please adjust the resource availability'.

To test this, you need an openai api key and add it in the settings (it will be stored in your browser's localstorage). After that, you can use the microphone icon in the ribbon menu (press it once to start the recording, press it again to stop the recording, and the processing begins).

You can also test most things via text input in this app, but for example, I have another app for kids that supports audio input only. There, the kid can say 'I want to learn about apple trees' and the system creates apple tree content ;-) However, it also has some content filters to allow only content suited for certain age levels. That is something you might want to include in automated tests.

[1]: https://critically.app