←back to thread

Launch HN: Flywheel (YC S25) – Waymo for Excavators

103 points jashmota | 4 comments | 24 Sep 25 16:48 UTC | HN request time: 0s | source

Hey HN, We're Jash and Mahimana, cofounders of Flywheel AI (https://useflywheel.ai). We’re building a remote teleop and autonomous stack for excavators.

Here's a video: https://www.youtube.com/watch?v=zCNmNm3lQGk.

Interfacing with existing excavators for enabling remote teleop (or autonomy) is hard. Unlike cars which use drive-by-wire technology, most of the millions of excavators are fully hydraulic machines. The joysticks are connected to a pilot hydraulic circuit, which proportionally moves the cylinders in the main hydraulic circuit which ultimately moves the excavator joints. This means excavators mostly do not have an electronic component to control the joints. We solve this by mechanically actuating the joysticks and pedals inside the excavators.

We do this with retrofits which work on any excavator model/make, enabling us to augment existing machines. By enabling remote teleoperation, we are able to increase site safety, productivity and also cost efficiency.

Teleoperation by the operators enables us to prepare training data for autonomy. In robotics, training data comprises observation and action. While images and videos are abundant on the internet, egocentric (PoV) observation and action data is extremely scarce, and it is this scarcity that is holding back scaling robot learning policies.

Flywheel solves this by preparing the training data coming from our remote teleop-enabled excavators which we have already deployed. And we do this with very minimal hardware setup and resources.

During our time in YC, we did 25-30 iterations of sensor stack and placement permutations/combinations, and model hyperparams variations. We called this “evolution of the physical form of our retrofit”. Eventually, we landed on our current evolution and have successfully been able to train some levels of autonomy with only a few hours of training data.

The big takeaway was how much more important data is than optimizing hyperparams of the model. So today, we’re open sourcing 100hrs of excavator dataset that we collected using Flywheel systems on real construction sites. This is in partnership with Frodobots.ai.

Dataset: https://huggingface.co/datasets/FlywheelAI/excavator-dataset

Machine/retrofit details:

  Volvo EC380 (38 ton excavator)
  4xcamera (25fps)
  25 hz expert operator’s action data
The dataset contains observation data from 4 cameras and operator's expert action data which can be used to train imitation learning models to run an excavator autonomously for the workflows in those demonstrations, like digging and dumping. We were able to train a small autonomy model for bucket pick and place on Kubota U17 from just 6-7 hours of data collected during YC.

We’re just getting started. We have good amounts of variations in daylight, weather, tasks, and would be adding more hours of data and also converting to lerobot format soon. We’re doing this so people like you and me can try out training models on real world data which is very, very hard to get.

So please checkout the dataset here and feel free to download and use however you like. We would love for people to do things with it! I’ll be around in the thread and look forward to comments and feedback from the community!

Show context
seabrookmx ◴[24 Sep 25 19:04 UTC] No.45364639[source]▶
> The joysticks are connected to a pilot hydraulic circuit, which proportionally moves the cylinders in the main hydraulic circuit which ultimately moves the excavator joints

I've actually spent a decent amount of time running an excavator, as my Dad owns a construction / road building company. It was a great summer job!

An important note about the pilot hydraulics is that they _provide feedback to the operator_. I would encourage any system that moves these controls on behalf of a remote human operator or AI to add strain gauges or some other way to measure this force feedback so that this data isn't lost.

The handful of "drive by wire" pieces of equipment that my Dad or other skilled operators in my family have ran were universally panned, because the operators are isolated from this feedback and have a harder time telling when the machine is struggling or when their inputs are not sufficiently smooth. In the automotive world, skilled drivers have similar complaints about fully electronic steering or braking systems, as opposed to traditional vacuum or hydraulic boosting approaches where your foot still has a direct hydraulic connection to the brake pads.

replies(4): >>45365403 #>>45365849 #>>45366303 #>>45366966 #
roamerz ◴[24 Sep 25 23:04 UTC] No.45366966[source]▶
I have a 20 ton Takeuchi and I don’t recall any feedback in the controls at all. The feedback I use is from the seat and sounds of the machine - well besides of course visual of course.

I cannot imagine this being useful to me unless the virtual operators cab closely mimicked an actual machine. It would have to have audio from the machine and be on a platform that tilted relative to the real thing. It would also need 270 degrees of monitors with a virtual mirror to see behind. On the front monitor, minimally, would need the to see vertically up and down too.

I also imagine all of this would be more useful to seasoned operators who can do most things on excavators in their sleep (definitely not me lol)

replies(1): >>45368579 #
jashmota ◴[25 Sep 25 02:17 UTC] No.45368579[source]▶
The way I think about this - we should not have multi screens. Human field of vision is 60 degrees for central and about 120 degrees binocular. The bucket of the excavator is way narrower than this which means actual task doesn't require wide vision.

So if we are able to have really good autonomous safety layers to ensure safe movements, and dynamically resize remote teleop windows, you'd make the operator more efficient. So while we stream 360 degree view, we get creative in how we show it.

That's on the vision side. We also stream engine audio, and do haptic feedback.

Takeuchi are interesting! Rare ones to have blades even on bigger sizes - is that why you got one?

replies(2): >>45368860 #>>45369335 #
roamerz ◴[25 Sep 25 04:39 UTC] No.45369335[source]▶
Well sure if you are just looking at where the bucket is digging but there is often a dump truck sitting on either your right or left flank waiting for what’s in your bucket (don’t forget the beep button lol). Having a monitor to either side duplicates what you are seeing out of your peripheral vision when operating the real thing. Would make transitioning from real to virtual much easier and imho safer.

Yes that is precisely why - makes for a much more versatile machine. TB180FR - it’s med-small, about 10 ton.

replies(1): >>45369362 #
1. jashmota ◴[25 Sep 25 04:47 UTC] No.45369362[source]▶
I think swinging (which is about 40% of dig and dump workflow by time spent) should not be manual. That's lowest levels of autonomy which requires roughly centering to the pit/truck which we have already achieved. Hence operator only has to look in front!

Those workflow numbers come from multiple observations at different sites, one of the examples is this: https://www.youtube.com/watch?v=orEsvu1CS64

I wish to talk to you because it's rare sight someone has a Takeuchi - is there a way to connect? My email is contact at useflywheel dot ai

replies(2): >>45369428 #>>45371300 #
2. roamerz ◴[25 Sep 25 05:04 UTC] No.45369428[source]▶
>>I think swinging should not be manual.

I disagree and here's a couple of reasons why I say that:

1. What am I going to do with the time between releasing control and regaining it from the autonomous control?

2. In that break of workflow my first thought is it will cause a break in my concentration.

3. When I am swinging back from the truck to the trench the bucket is naturally in my control. It seems that in autonomy mode the transition from autonomous to my control would be very unnatural and choppy. I suppose with time it would be okay but man seems to violate the whole smooth is fast concept.

I'll shoot you an email.

3. mschuster91 ◴[25 Sep 25 10:38 UTC] No.45371300[source]▶
> I think swinging (which is about 40% of dig and dump workflow by time spent) should not be manual.

It's been over a decade since I last operated an excavator, so grains of salt as usual - but I'd say it should be manual, or at least semi-automated. You need to take care where you unload the bucket on a truck, to avoid its weight distribution being off-center, or to keep various kinds of soil separated on the bed (e.g. we'd load the front with topsoil and fill the rear with the gravel or whatever else was below.

replies(1): >>45380786 #
4. jashmota ◴[25 Sep 25 23:46 UTC] No.45380786[source]▶
I agree - the dumping and digging itself (where you move boom, arm, bucket much more than swing/tracks) should be manual. But swinging to the truck and back to pit (pure swinging motion to center around these areas of interest) do not have to be manual. I agree with your and other comments that the transition has to be smooth and that's something we're working on.