←back to thread

93 points walz | 1 comments | | HN request time: 0.219s | source

I was walking around New York last month during some light rain and noticed about half the people had umbrellas open. When the rain picked up a few minutes later, that number jumped closer to 80%.

It got me thinking it'd be cool to track this somehow, so I built a website! I am taking a sidewalk livestream, feeding it into a YOLO model for people tracking, then sending a frame of each detected person to Gemini 2.0 Flash, which returns structured JSON about each person's clothing and if they're holding an umbrella. I also had fun making the site look like a TV weather channel.

I showed some friends this project and someone mentioned how the legendary Tasks xkcd comic (https://xkcd.com/1425) is out of date now. If you want to check whether a photo has birds in it (or if someone is holding an umbrella), you can just ask an inexpensive vision model for JSON.

1. Anthony-G ◴[] No.44375978[source]
Very cool project that I would have figured to be impossible to do in real-time.

I thought I'd let you know that the web page is rotated to the right on my desktop monitor so I have to tilt my head or drag the browser window to the landscape-oriented monitor. I'm guessing that this is an optimisation for mobile devices but I doubt I'm the only one with a portrait-oriented monitor (viewport dimensions are currently 1200×1779).