Data Version Control

How does it compare to Oxen?

https://github.com/Oxen-AI/Oxen

never heard of oxen but it looks like a super interesting alternative. would love to hear from someone who has experience with both.

my first impression: dvc is made to use with git where there are arbitrary folders handled by dvc INSIDE your git repo, where oxen is an alternative for a separate data repo. also oxen has lots of integration with dataframes and tabular, ai training and infernece data that dvc is missing. on the other hand dvc has a full DAG pipeline engine integrated as well as import/ export and pluggable backends.

Maintainer of Oxen here, we initially built Oxen because DVC was pretty painfully slow to work with, and had a lot of extra bells and whistles that we didn’t need. Under the hood we optimized the merkle tree structure, hashing algorithms, network protocols, etc to make it speedy when it came to large datasets. We have a pretty nice front end at https://oxen.ai for viewing and querying the data as well.

Happy to answer any thoughts or questions!

Can this be used with GitHub? If yes, I would shift from dvc immediately