←back to thread

Magma: A foundation model for multimodal AI agents

(microsoft.github.io)
301 points SerCe | 2 comments | 20 Feb 25 02:11 UTC | HN request time: 0.48s | source
Show context
ygouzerh ◴[20 Feb 25 07:19 UTC] No.43111985[source]▶
The rate of progress on multimodal agents is impressive. OpenVLA was released in June 2024 and was state of the art at that time... 8 months later, on tasks like "Pick Place Hotdog Sausage" the success rate is passing from 2/10 to 6/10
replies(1): >>43113450 #
1. paulluuk ◴[20 Feb 25 11:22 UTC] No.43113450[source]▶
"Pick Place Hotdog Sausage" is such a bizarre name, though. Is it meant to be human readable? AI-readable? Just a label for the researchers? Same with "Put Mushroom Place Pot". As far as I can see both labels are only used in this Magma paper, nowhere else that Google can find.
replies(1): >>43114959 #
2. ekidd ◴[20 Feb 25 14:21 UTC] No.43114959[source]▶
"Pick & place" is a term for a kind of robot that can pick up scattered items from a conveyor belt and arrange them in a regular fashion.

The really fast multi-arm versions can be hypnotic to watch. You can see an example at 1:00 in this video: https://youtu.be/aPTd8XDZOEk

The limitation of industrial pick & place robots is that they're configured for a single task, and reconfiguring them for a new product is notoriously expensive.

Magma's "pick & place" demo is much slower and shakier than a specialized industrial robot. But Magma can apparently be adapted to a new task by providing plain English instructions.