An intro to DeepSeek's distributed file system

(maknee.github.io)

621 points sebg | 1 comments | 17 Apr 25 12:50 UTC | HN request time: 0s | source

Show context

londons_explore ◴[17 Apr 25 13:45 UTC] No.43716756[source]▶

This seems like a pretty complex setup with lots of features which aren't obviously important for a deep learning workload.

Presumably the key necessary features are PB's worth of storage, read/write parallelism (can be achieved by splitting a 1PB file into say 10,000 100GB shards, and then having each client only read the necessary shards), and redundancy

Consistency is hard to achieve and seems to have no use here - your programmers can manage to make sure different processes are writing to different filenames.

replies(2): >>43717054 #>>43717178 #

sungam ◴[17 Apr 25 14:05 UTC] No.43717054[source]▶

>>43716756 #

I wonder whether it may have been originally developed for the quantitive hedge fund

replies(2): >>43717714 #>>43724356 #

1. huntaub ◴[17 Apr 25 14:48 UTC] No.43717714[source]▶

>>43717054 #

Yes, I think this is probably true. I've worked with a lot of different hedge funds who have a similar problem -- lots of shared data that they need in a file system so that they can do backtesting of strategies with things like kdb+. Generally, these folks are using NFS which is kind of a pain -- especially for scaleability -- so building your own for that specific use case (which happens to have a similar usage pattern for AI training) makes a lot of sense.

↑