Open Riak – open, modern Riak fork

Speaking as a former tech evangelist/engineer at Basho, there were a few significant challenges.

Riak is horribly unfriendly as a database: no SQL, it exposes eventual consistency directly to the developer, it’s relatively slow, and Erlang is a fairly unusual language.

While you can run Riak on a single server, you’d have to really want to.

Its strength is the ability to scale massively, but not many projects need that scale, and by the time you do, you’re probably already using some friendlier database and you’d rather make that one work.

I wonder if some of these issues could be addressed sanely in an extension to the functionality

Thing is, Cassandra became and remained popular, with similar aspects (though in JVM instead of Erlang, so).

Though it had a couple years head start when there really no other options for people wanting that kind of kit.

Back in 2011 I was working on a project that involved Riak. The difficulty and slowness for doing stuff corresponding to basic SQL operations was certainly a giant strike against it, and helped sink that project before it was released.

https://howfuckedismydatabase.com/nosql/ this infamous comic is about riak

> corresponding to basic SQL operations

Ohhh, this brings memories of developers hitting the wall... Between different SQL databases!

Back in 2016 I was delegated at work to do ops on a project that had big data ambitions in Threat Intelligence space.

Part of how they intended to support that was Apache Phoenix, an SQL database backed by HBase, running on top of Hadoop that also provided object storage (annoyingly through WebHDFS gateway).

Constant problems with hung Phoenix queries and instability of Hadoop in entirety led me to propose moving over to PostgreSQL, which generally went quite well... Except several cases of "basic SQL operations" that turned to have wildly different performance compared to Phoenix and most importantly, to MySQL in MyISAM mode, like doing SELECT (*) on huge tables.

Fun times, got to meet a postgres core team member thanks to it.

We were working on ways of making it easier (such as CRDTs to reduce the amount of work developers had to do to leverage eventual consistency), but these were pretty challenging problems to solve.

One of our biggest disappointments: we had plans to add a way to enforce strong consistency leveraging (IIRC) something akin to multi-paxos, but couldn't get it to work.

That could also very well be about CouchDB which implemented indexes/views as MapReduce functions.

Back in the day we had a CouchDB MapReduce view (on Cloudant) which took a full month to rebuild (while an angry customer was waiting). The I/O inefficiency was absolutely off the charts.

that sounds like a painful session

I feel building a threat intelligence product on Cassandra is a bit on the nose. What's next, calling the TCB Palladium?

TBH, we shipped fully working strong consistency in 2014. It just had a limited feature set, was disabled by default, and was never promoted/marketed since it didn't fit the direction the new CEO/CTO was pushing.

The engineering exodus around that time sorta killed the project though, and we never were able to do the big follow-up work to make it really shine.

(Disclaimer: Former Basho Principal Engineer, primary author of strong consistency work, lead riak_core dev from 2011-2015)

I think another 18 months would have been enough too. But it just wasn't the right environment after the hostile take-over / leadership transition.

My apologies for misremembering. I’m glad you chimed in to correct the record.

There were customers happily using strong consistency in production, but somehow the idea that it wasn’t “finished” kept getting repeated over and over by management. I was well on my way to solving the biggest rough edge (tombstone reaping in SC buckets) but then I got pulled off to work on the infamous “data platform” and never got to finish that work :-(

I'm not sure though for how much longer it will continue to make sense for the project as-is to continue to roll riak_ensemble forward as part of future releases. As there are no contributors who have direct direct experience or knowledge of using it in production, so it is hard to claim it as being a supported part of the product in any real sense.

I apologise if we do eventually cut it. Having worked through the code when chasing unstable tests, I developed an appreciation for the quality of the work.

I joined a company that had some investment in Basho and managed to sell Riak as the data store for a large client. It never really worked out, not enough of the SREs had be properly trained on Riak, the developers hated it because getting help and support was somewhat difficult, especially in an emergency.

In the end more and more data was offloaded to MariaDB, until one day the last remaining data couldn't justify the cost of the Riak cluster. I think we swapped out an eight node Riak cluster for two largish MariaDB database (one being a hot-standby).

For one of the other clients it was the exact same scenario, only we had been contracted in to help run the Riak cluster, which we didn't do well. Once they had migrate of it, to Oracle I think, the client left.

To me it always felt like it was just the wrong tool for that particular job. Someone really wanted to be able to jump on the NoSQL hype and sell something. They picked Riak, because it honestly looked really good, and probably was, compared to MongoDB, CouchDB or whatever else happened to float around at the time. It just wasn't the right tool for the problems it was applied to.

I remember somebody whose competence I rate highly talking about ending up picking Cassandra over Riak (at a point where Riak was technically decently mature) simply because of availability of ops expertise - basically his take seemed to be that Basho support was solid but they hadn't succeeded at cultivating people he could *hire* to handle as much as possible in house before calling in the gurus.

(I can't, of course, speak to the truth of this, only that over a couple decades of knowing the dude in question and working with him on and off he had sufficient Clue that I expect he did put in the effort before coming to that conclusion)

If you don’t mind my asking, how did you find working with Erlang in such a context? I’ve always been curious about it, but these days it seems like most people are mostly interested in Elixir for REST APIs and I am content with my existing tools for that purpose. However, there are clearly places where OTP and other Erlang functionality would shine and I’ve always thought it might be fun to keep in my back pocket.