How IT WorksIMAP4

R’ykandar Korra’ti

Over the previous two issues, I discussed the two most popular Internet message-moving protocols, SMTP and POP3. This month, I will approach the Internet Message Access Protocol Version 4, revision one (IMAP4rev1), often referred to just as IMAP4.

Unlike the previously discussed protocols such as POP3, IMAP4 does not move messages between message stores. It instead manipulates and displays messages that exist on a remote server without transferring them to the client for storage. In this respect, it mirrors portions of the Microsoft® Messaging Application Programming Interface (MAPI): a client and server-independent method of allowing clients to view messages stored on servers. Accordingly, IMAP4 is far more complex than SMTP or POP3. It must understand folders, messages, addresses, and the detailed internal structure and associated status information of each. It was also designed to handle Usenet newsgroups, public folders, and similar concepts, further complicating the picture.

Starting a Session

IMAP4 sessions start with a line-oriented TCP/IP connection initiated by the client over port 143 by default, and communicating by using command verbs in a send-and-reply exchange of 7-bit character text. Here, however, the client-server relationship is asynchronous; either side can, under certain circumstances, initiate a transaction. (The server might do so with the arrival of new mail, for example.) Because there is no guarantee that a previous transaction will be completed before the next command is sent, all client commands must be prefixed with an incrementing sequence number. Completion responses from the server include the sequence number of the command evoking them and one of the five response codes shown in Figure 1.

Figure 1 IMAP4 Response Codes

An IMAP4 session consists of a series of states, each allowing a different set of actions. Sessions last as long as the client is active. During the first state, Not Authenticated, the client must establish its identity and provide credentials showing that it has permission to access the server under that identity. (It is important to note that identity, server, and mailbox are not interchangeable terms under IMAP4; a single identity will typically have the right to use multiple mailboxes on a single server.)

The Not Authenticated state allows only six commands, three of which are allowed in all states. The three commands available in any state are CAPABILITY, which asks the server what protocols and authentication methods the server supports; NOOP, which is used to prevent timeouts; and LOGOUT, which ends the session.

Commands unique to the Not Authenticated state are STARTTLS, AUTHENTICATE, and LOGIN. STARTTLS triggers a Transport Layer Security (TLS) negotiation with the communications layer of the server, allowing channel-level encryption and validation of the channel for data security purposes. If successful, the IMAP4 server may then advertise additional capabilities not allowed on a clear channel.

Authenticating Clients

AUTHENTICATE triggers a Simple Authentication and Security Layer (SASL) exchange between the client and server to establish the client’s login credentials. SASL provides several methods for this. As with TLS, a full description of SASL is beyond the scope of this column, but all IMAP4rev1 servers must support the PLAIN method and should list other methods they support in response to the CAPABILITY command. PLAIN is not secure except over encrypted channels, so may only be used after a successful STARTTLS negotiation. Bear with me; it gets less abstract soon.

Finally, LOGIN—the last-gasp authentication method—allows a username and password pair to be sent to log in a user. This method is typically not allowed until after a successful STARTTLS; in fact, the spec recommends disabling it entirely. If it must be supported, servers should not present it as a CAPABILITY until after a secure channel has been negotiated.

After authentication, the client should reach the second state, Authenticated. In this state, mailboxes (which includes folders) may be manipulated, created, and deleted, though with one exception: their contents may not be changed. Also in this state, public folders, newsgroups, and the like may be joined/subscribed or abandoned/unsubscribed. The client can further pick which mailbox to open for message manipulation.

The Authenticated state mailbox manipulation commands are CREATE, DELETE, RENAME, SUBSCRIBE, UNSUBSCRIBE, LIST, and LSUB. CREATE, DELETE, and RENAME perform the functions you’d expect, with certain caveats, mostly involving the special INBOX mailbox. INBOX may not be created by a user, may not be deleted, and while it may be renamed, this actually results in the INBOX messages being moved to a new mailbox with the new name, leaving an empty INBOX ready to receive more messages.

SUBSCRIBE and UNSUBSCRIBE add and remove public-folder or Usenet-news style modified-access mailboxes to your subscription list. These commands will also cause the server to set up or break down any per-user flags associated with each subscribed folder. These flags indicate message status within the folders—which messages are recent, unread, have attachments, and so forth. IMAP4 specifies a minimum flag set that may be expanded.

Two commands identify available mailboxes. LSUB lists mailboxes to which the user has subscribed. LIST prompts the server to present the names of mailboxes the user could access—not just that user’s individual mailboxes and folders, but, depending upon the parameters given with the LIST command, public folders, newsgroups, and so on. Both take limiting parameters so the client does not have to parse a complete server tree.

The four other Authenticated state commands—SELECT, EXAMINE, STATUS, and APPEND—do not quite fit the model of mailbox manipulators. APPEND expects a mailbox and a message as parameters; the message will be appended to the end of the given mailbox. This is a manipulation of the mailbox, but as it is also manipulation of an individual message, the purity of the model is violated.

The SELECT, EXAMINE, and STATUS commands all provide gateways into mailbox contents. Each expects a mailbox name as a parameter. STATUS results in information about the messages within the specified mailbox being presented to the client, without affecting that mailbox’s status or contents. The information returned includes the message count, how many messages are recent, and how many are unseen, or haven’t been viewed by that individual user. However, it does not allow those messages to be manipulated.

SELECT and EXAMINE, on the other hand, both open (or select) the mailboxes they are given. The difference between the two is that SELECT opens the mailbox read/write; EXAMINE opens the mailbox’s messages read-only, though per-user message status flags may be changed. Both commands additionally provide important data about the mailbox: how many messages it holds, what flags are valid (or defined) in the mailbox, how many messages are flagged as recent, and the first unseen message in the mailbox.

Manipulating Mailboxes

Once a mailbox has been opened, the session enters the Selected state, with its message-manipulation commands of COPY, FETCH, SEARCH, STORE, and UID. Three commands affect mailbox status as a whole: CHECK, CLOSE, and EXPUNGE.

FETCH allows a client to retrieve all or parts of a message or group of messages, with surprising granularity. For example, a single FETCH command could retrieve all the FLAGS set for all the messages in a mailbox, or all the attachments, or all the plain-text portions of a subset of messages, or various header items and envelope data for a single message.

SEARCH takes search parameters and returns the index numbers of messages meeting the requested criteria. SEARCH, like FETCH, is deeply granular, requiring extensive searching capabilities on the server; it searches not just body and header content, but flag status, message size, envelope information, and more. Because of the intense server load this command can cause, it is often disabled.

The COPY command takes identifiers for one or more messages and a destination mailbox, and appends the messages to the end of that mailbox. STORE allows the client to overwrite flags associated with a message or set of messages, or change individual flags across a message set, such as the deleted flag. UID allows the client to use unique identifiers with the FETCH, COPY, STORE, and SEARCH commands instead of the message sequence numbers typically used.

The whole-mailbox commands are CHECK, EXPUNGE, and CLOSE. CHECK orders journaling mail servers to establish a checkpoint when the command is issued. Any housecleaning necessary should be committed before the server responds OK. EXPUNGE orders the server to delete all messages marked for deletion within the current mailbox. CLOSE combines both the CHECK and EXPUNGE commands (if applicable on the individual server) and then deselects the mailbox, enabling the client to switch work focus to another mailbox entirely, returning the session to the Authenticated state. A new mailbox may then be selected with the SELECT command.

Logging Out

The final state of any IMAP4rev1 session is the Logout state. This is not a server state entered after connection drop; rather, it is the time between a logout initiation and a server acknowledgment. During this time, the server does final housecleaning, deletes messages flagged for deletion, updates flags, deselects selected mailboxes, and then sends the BYE response before closing the connection.

This high-level overview of the spec sadly neglects specifics of command formats, and all but totally ignores details of server actions. For enough detail to implement either client or server, see RFC 3501.

Resources

The following sites will provide you with more IMAP information.

• IMAP - Version 4rev1
• The IMAP Connection

R’ykandar Korra’ti, postmaster for a small co-op ISP, lives near Seattle with her partner Anna. Having previously shipped mail products at Microsoft, she is now looking at grad school in a CS-related field so esoteric it doesn’t really have a name. Potential faculty advisors can reach her at darako@murkworks.net.

© 2008 Microsoft Corporation and CMP Media, LLC. All rights reserved; reproduction in part or in whole without permission is prohibited.