Build your own SQLite in Rust, Part 5: Evaluating queries

Making a basic database that works is not hard, but making a robust one is really hard. What makes SQLite shines is its extensive testsuite. I still don't understand the motivation of limbo, as without years of hardwork on test you can't say you are correct, and why i should pay for it...

At this point it's too early to even worry about correctness, it doesn't work yet.

But the years of work put into the existing project to make it robust don't mean the exact same years have to be spent on the reimplementation:

- there's been work spent on discovering the right architecture and evolving the db format. A new impl can copy the end result.

- hard lessons have been learned about dealing with bad disks, filesystems, fsync, flaky locks, etc. A new impl can learn from the solutions without having to rediscover them the hard way.

- C projects spend some time on compatibility with C compilers, OSes, and tweaking build scripts, which are relatively easy in Rust.

Testing will need a clever solution. Maybe they'll buy access to the official test suite? Maybe they'll use the original SQLite to fuzz and compare results?

Something being hard doesn't mean it's not worth trying.

It's up to the developers to decide if the outcome is worth the effort.

Maybe they're right, maybe they're wrong, but either way – if they succeed, we're all better off for it, aren't we?

The Limbo team seems to be leaning heavily into deterministic simulation testing (DST) and one of the cofounders on a recent podcast was very enthusiastic about the benefits of the approach.

https://github.com/tursodatabase/limbo/tree/main/simulator

https://changelog.com/podcast/626

> without years of hardwork on test you can't say you are correct

Before, I have the myth in my head that RDBMS development is like cryptography/kernels: Something not mean for humans to do.

Now working as part of one (http://spacetimedb.com) I see now that is hard, but doable.

Certainly MORE EASY than working in ERPs that is the thing that I have done for more than 20 years (now THAT is what is insane). Literally doing RDBMS is more relaxing than doing ERPs.

But what I have learned is that what make RDBMS truly hard is not the usual suspect: ACID.

You don't need 'years of hardwork' to reach it and to prove them is right (only some insanity when you need to target very weird os/archs and deal with faulty hardware).

What is truly hard and will eat your time is sql. SQL is hard. Is *bad*. Is *very hard* to optimize, and is so anemic that your time is expended more in the query optimizer (that has hard time because sql more than anything else) and because sql is bad, I tell you, it leads to insane stuff like having a single query with 1000 joins, and people have not means to good, optimal designs (and what you have like CTE and all that is poor bandaids) so now you get a bad input and have fun.

But if the RDBMS stay in the relational part and you could make a better lang to interface it then is far more easy. Still query optimizer is big part of your time, but if you have control of the input language you can do your life easy.

This is interesting.

I completely get why SQL is important and as an user I love it as a language, but as an application writer I have a very complicated relationship with it. It requires either inline SQL (which carries its own security risks and causes a bit of redundancy), or we gotta use complex abstractions on top of it, like ORMs. Tests also require spinning up a database or require some extra machinery.

It feels like we're making our lives more complicated by "requiring" a database to have SQL.

I would be totally ok with adopting a non-SQL relational database with a more structured API in greenfield projects. (Btw I will be definitely checking out your company).