Not flexible enough, performance problems, doesn't provide much ootb.
You're better off just writing a service from scratch, the time you are saving is minimal (this applies to most products before we get to django or wordpress imho)
We tried and abandoned keystone too.
Performance problems are usually something related to maybe missing indexes in your database for fields that you query on often, or similar. Payload itself is super thin. Can you give me an example of where you're seeing slowdowns? Maybe I can point you in the right direction.
Also I would love to hear about where Payload is not flexible enough. Extensibility is one of our priorities so if there is something you'd like to accomplish, but can't, I will see what I can do about it!