←back to thread

287 points shadaj | 1 comments | | HN request time: 0.204s | source
1. th0ma5 ◴[] No.43199185[source]
Since multicore processing a ton of software you use or create is distributed you have to ask if you want to be in control of how it is distributed or not. If you want it easy and let the library figure it out then you have to accept the topological ideas it has. For instance H2O is a great machine learning package that even has its own transparent multi core processing. If you want to go across machines it has its own cluster built in. You can also install it into Hadoop, Spark, etc but once you start going that direction you're more and more on the hook for what that means and if it even is more effective for your problem and what your distributed strategy should be.

Things like re-entrant idempotence, software transactional memory, copy on write, CRDTs etc are going to have waste and overhead but can vastly simplify conceptually the ongoing development and maintenance of even non-distributed efforts in my opinion, and we keep having the room to eat the overhead.

There's a ton of bias against this for good reasons that the non distributed concepts still just work without any hassle but we'd be less in the mud in a fundamental way of we learned to let go of non-eventual consistency.