←back to thread

Magic Wormhole: get things from one computer to another, safely

(github.com)
816 points tosh | 2 comments | 17 Aug 24 16:59 UTC | HN request time: 0.001s | source
Show context
geerlingguy ◴[17 Aug 24 18:23 UTC] No.41276702[source]▶
I've used this for years when passing large files between systems in weird network environments, it's almost always flawless.

For some more exotic testing, I was able to run my own magic wormhole relay[1], which let me tweak some things for faster/more reliable huge file copies. I still hate how often Google Drive will fall over when you throw a 10s-of-GB file at it.

[1] https://www.jeffgeerling.com/blog/2023/my-own-magic-wormhole...

replies(4): >>41277198 #>>41277682 #>>41277698 #>>41278657 #
bscphil ◴[17 Aug 24 20:28 UTC] No.41277698[source]▶
> For some more exotic testing, I was able to run my own magic wormhole relay[1], which let me tweak some things for faster/more reliable huge file copies.

The lack of improvement in these tools is pretty devastating. There was a flurry of activity around PAKEs like 6 years ago now, but we're still missing:

* reliable hole punching so you don't need a slow relay server

* multiple simultaneous TCP streams (or a carefully designed UDP protocol) to get large amounts of data through long fat pipes quickly

Last time I tried using a Wormhole to transmit a large amount of data, I was limited to 20 MB/sec thanks to the bandwidth-delay product. I ended up using plain old http, with aria2c and multiple streams I maxed out a 1 Gbps line.

IMO there's no reason why PAKE tools shouldn't have completely displaced over-complicated stuff like Globus (proprietary) for long distance transfer of huge data, but here we are stuck in the past.

replies(4): >>41278538 #>>41279150 #>>41279898 #>>41284508 #
croemer ◴[18 Aug 24 00:19 UTC] No.41279150[source]▶
20MB/sec is 160Mbps, so wormhole wasn't that far off the 1Gbps. Sure not maxing out but within a factor of 6.
replies(3): >>41279692 #>>41279911 #>>41282474 #
ghusbands ◴[18 Aug 24 02:31 UTC] No.41279692[source]▶
A factor of six is a very long way off, pretty much universally.
replies(2): >>41280240 #>>41282224 #
1. croemer ◴[18 Aug 24 13:21 UTC] No.41282224[source]▶
Not as far off as the casual reader might think 20MB vs 1Gb sounds way more than the actuall 160Mb vs 1Gb - one shouldn't use Bytes and bits in a direct comparison together. One or the other, otherwise it's misleading/confusing.
replies(1): >>41285306 #
2. bscphil ◴[18 Aug 24 20:39 UTC] No.41285306[source]▶
In this case transferring the data at the slow rate would have taken more than a week, so it's no small difference. Actually one side had a 10 Gbps line, so if the other side had had faster networking I could easily have exceeded the limit and gotten the transfer done more than 6x faster.

I used the term "1 Gbps line" just because it's a well known quantity - the limitation of Gigabit Ethernet. The point wasn't that multiplexing TCP can get you 6x better speeds, it's that it improved the speed so much that the TCP bandwidth-delay product was no longer the limiting factor in the transfer.