How IMAP works under the hood

- The grammar is hard. I built a parser using lpeg and I'm incredibly glad I did--doing it ad hoc won't lead to good results.

- It's an asynchronous protocol. You can send lots of requests to a server and you have to tag them so you can match them up with responses later. You don't generally want to do that in a client; i.e. you don't want to do these transactional things over an async connection and track state across all of it. You want to like, block on deleting things, renaming things, sending things, etc.

- IMAP is multi-user--it's built around the idea of multiple clients accessing the same mailbox at the same time and streaming updates. Another thing you really don't want to deal with when building an email client.

- There's functionality that you basically shouldn't use; the big one is search. Even the specs more or less say "good luck using this".

You can group all this under the a heading of "we thought people would use this over telnet", but attachments and non-plain-text email I think made that non-viable.

I think this all means probably every non-web email client treats IMAP like POP and keeps its own store. I haven't done a survey or anything, but I'd be surprised if that weren't true.

Pretty sure mutt doesn't. It only caches the headers.

And as for treating IMAP like POP, yes, there are clients that only pay lip service to "having IMAP support", only so that they can have one more green checkbox in feature list that their present in their marketing. But there are also more serious clients.

Message sequence numbers. Every folder in IMAP has its emails numbered from 1-N, with no holes, so if you delete a message, everything after it has its message sequence number decremented to close the hole. Except IMAP is multiclient, and the message can be deleted by other connections than the one you're currently on. But the server is only allowed to tell you about message deletions at certain points, so now the server has to keep essentially per-client message sequence number state and carefully make sure that everyone is kept in sync... and it's a recipe for disasters in practice. Any sane client will instead use UIDs for everything (and any sane server will implement all the UID extensions to let UIDs be used for everything).

The other fun corner case I recall is that IMAP part numbering is a little unclear what happens around body parts of content-type message/rfc822. So I crafted a message that had a multipart/mixed with one leg being a message/rfc822 whose body was a message/rfc822, and tested the output on all 4 IMAP server implementations I had accounts on at the time to see how they handled the part numbering. I got back 4 different results. None of them were considered correct by the IMAP mailing list discussion on the experiment.

> I think this all means probably every non-web email client treats IMAP like POP and keeps its own store.

The distinction I would use is thin client versus thick client. Most clients like Outlook or Thunderbird are thick clients, which need to maintain their own local database for other reasons (like supporting offline mode or having database features not necessarily supported by an IMAP server, like arbitrary tagging). If you've got a local database, it's much saner to use IMAP essentially as a database synchronization protocol rather than trying to build a second implementation of all of your features on top of IMAP's native features, especially given that IMAP server implementation of these features is generally questionable at best.

IMAP was originally designed, it seems to me, to make thin clients easy to write (I can see how something like pine would be a very thin veneer over the protocol itself). But it's been clear over the past few decades that most clients are of the thick variety; things like QRESYNC and CONDSTORE were added to make the database synchronization steps a lot easier.

Seem like it can cache message bodies:

* http://www.mutt.org/doc/manual/#message-cachedir

* https://neomutt.org/guide/optionalfeatures.html#body-caching

Yes, you can match the final OK/NO/BAD responses with the original command based on the tag. The annoying thing is that each command typically sends most of its data in "untagged responses". And IMAP servers can send unsolicited untagged responses at any time too. It's quite tricky to handle all those responses correctly in email clients. Indeed, normally you just want to send a command, and get all the data for that command back until the finishing response. IMAP clients typically open multiple connections to isolate commands.

Yes! Since last year there's the experimental UIDONLY extension that allows clients & servers to operate entirely without message sequence numbers. Saves quite a bit of accounting and possibly memory (for large mailboxes). It's a bit surprising the RFC so recent.

> [... thick vs thin clients ...]

For some programmatic email access, IMAP seems quite easy to use. Just write a programmer-readable line of text as command (eg fetch/search/store/etc) and parse the response lines (that's a bit trickier in many cases, probably needing a library, but you may be able to get away without it some of the time) and you're done.

About thick clients: Storage capacity is growing much harder than my email, also on mobile. I would be fine with thick clients that sync all messages. Allows for simpler protocols and simpler implementations. An email access protocol is then not much more than a two-way sync of email message files and associated "dynamic data" like flags/keywords.

But IMAP and JMAP are doing many things at once: synchronization for thick clients, and online access for thin clients. JMAP is specifically targeting the web (thin clients), so what was old is new again (although base IMAP isn't too helpful to (web-based) thin clients either).

No, the grammar is too convoluted for that. It's more like "we thought text-based protocols are better".

POP3 and SMTP are relatively easy to use manually. IMAP is not.