Nubmq: A high performant key-value store engine built from first principles

This looks great, how are you doing the benchmarks? It claims to be way faster than Redis. Can you also measure it against Microsoft Garnet? Whats the secret sauce for beating in latency?

Redis:

    Write Latency ~1.1ms
    Read Latency ~700µs
    Max Throughput ~85,000 ops/sec

Nubmq:

    Write Latency 900µs
    Read Latency 500µs
    Max Throughput 115,809 ops/sec

also, 700µs for Redis reads sounds high to me. Running against memtier_bench also would be great - https://github.com/RedisLabs/memtier_benchmark

The benchmarks were done on an M2 MacBook Air (8-core) with 21M requests distributed across 100 concurrent clients sending requests as fast as possible(the test script is also in the github repo),

a few reasons for being fast that come to my mind now are:

1. reads are direct lookups, so distributing that across goroutines will result in them being faster 2. it is set requests where it gets complicated, if we're simply updating some key's value, it's essentially negligible, but if we're creating a new key value pair, that can increase per shard load under scale, which would trigger an store resizing, to avoid just stopping everything when that happens, the engine recognises when the per shard load starts to gets too high, in the backgroud creates a bigger store, and then essentially switches writes from the old engine to the new(bigger) one, while the old one keeps processing reads, and the older engine migrates it's keys to the newer one in backgroud, once it's done, we just dereference the older engine to be collected by GC :) this essentially makes sure that incoming requests keep getting served (oh shit, I spilled the secret sauce)

atleast on my machine,with the default setting, under that concurrent load, Redis starts slowing down due to single-threaded execution and Lua overhead.