←back to thread

Video scraping: extracting JSON from a 35s screen capture for 1/10th of a cent

(simonwillison.net)
309 points simonw | 2 comments | 17 Oct 24 12:43 UTC | HN request time: 0.419s | source
Show context
pridkett ◴[18 Oct 24 03:48 UTC] No.41876184[source]▶
Video scraping doesn’t need to be just screen captures. I’ve demoed a solution with Gemini where you take a video walking up and down aisles in a retail store and it captured 100% accurate data on product name, quantity/size, sku, and price for a little under 75% of the products. And that was back in January.

This has huge implications for everything from competitive pricing, to understanding store layouts, to creating your own grocery store inflation monitor. Just subtly take a video and process it.

And the models have only gotten better.

replies(4): >>41889606 #>>41891498 #>>41891541 #>>41894068 #
1. TechDebtDevin ◴[19 Oct 24 23:14 UTC] No.41891541[source]▶
I've done this as well with bookshelves at thrift stores that are completely unorganized. I don't want to lose my mind reading every title on every book binding, so I take a picture and ask the model to list all the books on the shelf then I can easily scan through the list to see if something catches my eye.
replies(1): >>41962136 #
2. pridkett ◴[27 Oct 24 12:35 UTC] No.41962136[source]▶
This is an old thread and will probably be lost now, but this sort of tool has been industrialized now. At the library book sales in my area, you'll see a handful of people there right as they open, they'll have either a scanner device or an app on their phone. They just scan over the spines of the books and it beeps when it finds something interesting. It highlights the book and they pick it out.

On the one hand, great application of technology. On the other hand, the folks using them have zero interest in reading the books. So, when the library happens to have a nearly brand new copy of "Harry Potter and the Order of the Phoenix: Illustrated Edition" there's almost no chance that someone in the community who wants to read the book will get it. Instead it gets snapped up by the resellers who make a couple of bucks off sending it to someone else who may be across the country.