- The grammar is hard. I built a parser using lpeg and I'm incredibly glad I did--doing it ad hoc won't lead to good results.
- It's an asynchronous protocol. You can send lots of requests to a server and you have to tag them so you can match them up with responses later. You don't generally want to do that in a client; i.e. you don't want to do these transactional things over an async connection and track state across all of it. You want to like, block on deleting things, renaming things, sending things, etc.
- IMAP is multi-user--it's built around the idea of multiple clients accessing the same mailbox at the same time and streaming updates. Another thing you really don't want to deal with when building an email client.
- There's functionality that you basically shouldn't use; the big one is search. Even the specs more or less say "good luck using this".
You can group all this under the a heading of "we thought people would use this over telnet", but attachments and non-plain-text email I think made that non-viable.
I think this all means probably every non-web email client treats IMAP like POP and keeps its own store. I haven't done a survey or anything, but I'd be surprised if that weren't true.