Most active commenters
  • croemer(3)
  • bscphil(3)

←back to thread

Magic Wormhole: get things from one computer to another, safely

(github.com)
816 points tosh | 14 comments | 17 Aug 24 16:59 UTC | HN request time: 1.058s | source | bottom
Show context
geerlingguy ◴[17 Aug 24 18:23 UTC] No.41276702[source]▶
I've used this for years when passing large files between systems in weird network environments, it's almost always flawless.

For some more exotic testing, I was able to run my own magic wormhole relay[1], which let me tweak some things for faster/more reliable huge file copies. I still hate how often Google Drive will fall over when you throw a 10s-of-GB file at it.

[1] https://www.jeffgeerling.com/blog/2023/my-own-magic-wormhole...

replies(4): >>41277198 #>>41277682 #>>41277698 #>>41278657 #
bscphil ◴[17 Aug 24 20:28 UTC] No.41277698[source]▶
> For some more exotic testing, I was able to run my own magic wormhole relay[1], which let me tweak some things for faster/more reliable huge file copies.

The lack of improvement in these tools is pretty devastating. There was a flurry of activity around PAKEs like 6 years ago now, but we're still missing:

* reliable hole punching so you don't need a slow relay server

* multiple simultaneous TCP streams (or a carefully designed UDP protocol) to get large amounts of data through long fat pipes quickly

Last time I tried using a Wormhole to transmit a large amount of data, I was limited to 20 MB/sec thanks to the bandwidth-delay product. I ended up using plain old http, with aria2c and multiple streams I maxed out a 1 Gbps line.

IMO there's no reason why PAKE tools shouldn't have completely displaced over-complicated stuff like Globus (proprietary) for long distance transfer of huge data, but here we are stuck in the past.

replies(4): >>41278538 #>>41279150 #>>41279898 #>>41284508 #
1. croemer ◴[18 Aug 24 00:19 UTC] No.41279150[source]▶
20MB/sec is 160Mbps, so wormhole wasn't that far off the 1Gbps. Sure not maxing out but within a factor of 6.
replies(3): >>41279692 #>>41279911 #>>41282474 #
2. ghusbands ◴[18 Aug 24 02:31 UTC] No.41279692[source]▶
A factor of six is a very long way off, pretty much universally.
replies(2): >>41280240 #>>41282224 #
3. dgoldstein0 ◴[18 Aug 24 03:21 UTC] No.41279911[source]▶
^ found the astronomer
replies(1): >>41280526 #
4. mrinfinitiesx ◴[18 Aug 24 05:10 UTC] No.41280240[source]▶
Yeah but with magic wormholes you see, there could be other universes where that's not the case and 160mbps is close to 1024mbps or 1000mbps whatever the cool kids call a gigabit now adays.
5. anonymousiam ◴[18 Aug 24 06:23 UTC] No.41280526[source]▶
I (Electrical + Software Engineer) once worked for a physicist who believed that anything less than an order of magnitude was merely an engineering problem. He was usually correct.
replies(4): >>41280572 #>>41281479 #>>41282602 #>>41285264 #
6. vasco ◴[18 Aug 24 06:35 UTC] No.41280572{3}[source]▶
I was taught the same. To not care a lot about things under an order of magnitude. Over the years when planning large software projects or assessing incidents and so on, the 1 order of magnitude threshold helped me often.
replies(1): >>41281098 #
7. croemer ◴[18 Aug 24 08:57 UTC] No.41281098{4}[source]▶
Bingo, I studied Physics!
8. elashri ◴[18 Aug 24 10:31 UTC] No.41281479{3}[source]▶
As a physicist, I think this is correct too :). You don't start to see problems with things under that, unless they are deviation from standard model predictions.
9. croemer ◴[18 Aug 24 13:21 UTC] No.41282224[source]▶
Not as far off as the casual reader might think 20MB vs 1Gb sounds way more than the actuall 160Mb vs 1Gb - one shouldn't use Bytes and bits in a direct comparison together. One or the other, otherwise it's misleading/confusing.
replies(1): >>41285306 #
10. CyberDildonics ◴[18 Aug 24 14:08 UTC] No.41282474[source]▶
If you sit down and need to wait one hour for one and 6 hours for the other to do the same thing I doubt you would say they are 'not that far off'
11. sva_ ◴[18 Aug 24 14:31 UTC] No.41282602{3}[source]▶
Variance in accuracy of this statement also safely within one order of magnitude
12. bscphil ◴[18 Aug 24 20:34 UTC] No.41285264{3}[source]▶
An order of magnitude isn't a defined quantity, it depends on what base you're working in.
replies(1): >>41292709 #
13. bscphil ◴[18 Aug 24 20:39 UTC] No.41285306{3}[source]▶
In this case transferring the data at the slow rate would have taken more than a week, so it's no small difference. Actually one side had a 10 Gbps line, so if the other side had had faster networking I could easily have exceeded the limit and gotten the transfer done more than 6x faster.

I used the term "1 Gbps line" just because it's a well known quantity - the limitation of Gigabit Ethernet. The point wasn't that multiplexing TCP can get you 6x better speeds, it's that it improved the speed so much that the TCP bandwidth-delay product was no longer the limiting factor in the transfer.

14. newaccount74 ◴[19 Aug 24 17:03 UTC] No.41292709{4}[source]▶
The difference between log 2, ln, and log 10 is less than an order of magnitude, so to a physicist it's all the same :)