Cyrus

Syntax

Syntax used in this document includes the following.

• example.com - a domain name
• username - a username or local part of an email address
• cyradm> - a cyradm prompt (already authenticated as an admin user) - usually displayed as localhost.localdomain>
• cyrus$ - shell prompt for the cyrus user (in some cases you can also do the command using su cyrus -c "..." but be careful to correctly escape characters)
• root# - shell prompt for the root user

imapd.conf configuration options

The example commands given are for installations where the imapd.conf has the following options (among others)

admins: cyrusadmin
defaultdomain: cyrusdomain
allowusermoves: on

deletedprefix: DELETED
delete_mode: delayed
expunge_mode: delayed

unixhierarchysep: on
virtdomains: userid

hashimapspool: true
fulldirhash: on

cyrus.conf configuration

frontend

Frontend servers in a murder MUST have the mupdate line set to prefork=1, if you use prefork=0 then no updates will be fetched from the mupdate server.

SERVICES {
...
  # mupdate slave - needed on each frontend server
  mupdate cmd="mupdate" listen=3905 prefork=1
}

mupdate

Mupdate servers in a murder require the -m option, and can be set to prefork=0 or prefork=1

SERVICES {
...
  # Master mupdate process
  mupdate cmd="mupdate -m" listen=3905 prefork=1
}

backend

mupdate service should not be defined on backend servers but you can put it in the startup events to resync all the mailbox lists with the master.

START {
...
  # Make sure we re-synchronise with the mupdate server at boot time
  mupdatepush   cmd="ctl_mboxlist -m"
}

SERVICES {
...
  # Master mupdate process
  # mupdate     cmd="mupdate -m" listen=3905 prefork=1
}

If you are using delete_mode: delayed or expunge_mode: delayed, you'll want to purge deleted mailboxes/messages after a certain period. To do this, add this to the EVENTS section.

EVENTS {
...
  # Expire deleted mailboxes older than 28 days.
  deleteprune    cmd="cyr_expire -E 4 -D 28" at=0430
  # Expire expunged messages older than 28 days.
  expungeprune   cmd="cyr_expire -E 4 -X 28" at=0445
}

Notes on Cyrus build

# /etc/init.d/cyrus-imapd start

This will spawn listeners for those services listed in /etc/cyrus.conf - typically POP3S, POP3, IMAPS, and IMAP.

Basic IMAP functionality can be tested:

# /usr/lib/cyrus-imapd/imtest -a user@domain -w password

This should print the server capabilities and end up saying:

Authenticated.
Security strength factor: 0

Cyrus may need an extra entry in /etc/services if you're planning to use a murder (group of cyrus backends/frontends).

mupdate         3905/tcp                        # Cyrus mupdate

Sieve port on CentOS 6

The default port number for sieve on CentOS 6 has changed. If working in a mixed CentOS 5 / CentOS 6 environment, ensure that the port number is standardised to avoid strange problems connecting to sieve.

# grep sieve /etc/services
sieve      2000/tcp            # Sieve Mail Filter Daemon
sieve      2000/udp            # Sieve Mail Filter Daemon

# grep sieve /etc/services
sieve-filter   2000/tcp            # Sieve Mail Filter Daemon
sieve-filter   2000/udp            # Sieve Mail Filter Daemon
sieve          4190/tcp            # ManageSieve Protocol

Edit /etc/services so that it's consistently either like the top one or bottom. If using the bottom one, ensure the bottom line is commented out so there's nothing on port 4190.

Listing and fixing mailbox quotas (both use and limit)

Check mailbox quota use for the domain example.com

cyrus$ /usr/lib/cyrus-imapd/quota -d example.com

To check and fix the mailbox quota use for the domain example.com

cyrus$ /usr/lib/cyrus-imapd/quota -d example.com -f

If a mailbox quota is incorrect on the mailstore it can be changed from cyradm. The below example sets the quota to 40MB (the number is the mailbox quota in KB)

cyradm> setquota user/username@example.com 40960

To view the current quota for a mailbox

cyradm> listquota user/username@example.com

Deleting a mailbox

$ cyradm -u cyrusadmin localhost

localhost.localdomain> setacl user/username@example.com cyrusadmin all
localhost.localdomain> deletemailbox user/username@example.com

Undeleting a mailbox

When mailboxes are deleted via the control panel, they're prefixed with DELETED and are kept around for 7 days (as specified by cyr_expire -D 7 in cyrus.conf)

$ /usr/lib/cyrus-imapd/ctl_mboxlist -d | grep example.com
example.com!DELETED.user.username.4D88AF31  0 fs1 username@example.com    lrswipkxtecda cyrusadmin lrswipkxtecda
example.com!DELETED.user.username.Drafts.4D88AF33  0 fs1  username@example.com   lrswipkxtecda

The 4D88AF31 and 4D88AF33 are the timestamp of the deletion converted to hex. It can be converted back to a useful timestamp using this one-liner.

$ perl -le 'print scalar(gmtime(hex("4D88AF31")));'
Tue Mar 22 14:16:17 2011

Using cyradm, you should be able to rename the folders back into the original mailbox.

cyradm> lm DELETED/user/username*@example.com
DELETED/user/username/4D88AF31@example.com (\HasNoChildren)
DELETED/user/username/Drafts/4D88AF33@example.com (\HasNoChildren)

If the original mailbox hasn't been recreated, simply rename it back.

cyradm> renamemailbox DELETED/user/username/4D88AF31@example.com user/username@example.com
cyradm> renamemailbox DELETED/user/username/Drafts/4D88AF33@example.com user/username/Drafts@example.com

Alternatively, you could rename it to a sub-folder such as "restored"

cyradm> renamemailbox DELETED/user/username/4D88AF31@example.com user/username/restored@example.com
cyradm> renamemailbox DELETED/user/username/Drafts/4D88AF33@example.com user/username/restored/Drafts@example.com

Moving a whole mailbox into a sub-folder of a different mailbox

*** THIS ONLY WORKS WHEN YOU HAVE delete_mode:delayed ENABLED ***

A possible reason for needing this would be if user/username@example.com was deleted and then re-created, but then the user wanted their original mailbox undeleted without losing anything from their new mailbox either. In this instance, rename the new mailbox temporarily (eg user/username2@example.com) and then undelete the old mailbox as per the instructions above. Next rename the sub-folders from the new mailbox into the restored mailbox (eg user/username/restored/Sent@example.com) using the normal renamemailbox command.

It's not possible to rename user/username@example.com directly into user/username2/restored@example.com because it's a top-level mailbox but if you delete the mailbox, it becomes a sub-folder under DELETED/user/username/4D88AF31@example.com and then you can rename that to wherever you want.

Undeleting messages in a mailbox

List messages available to unexpunge:

su cyrus -c "/usr/lib/cyrus-imapd/unexpunge -l user/username@example.com"

Each message will give you something like the following:

UID: 4778
       Size: 6878
       Sent: Fri Oct 11 12:00:00 2013
       Recv: Fri Oct 11 16:45:21 2013
       Expg: Sun Oct 13 00:19:59 2013
       From: george’s tradition <sales@georgestradition.co.uk>
       To  : <username@example.com>
       Cc  : 
       Bcc : 
       Subj: {42}
kids eat for £1.99 at the thorn tree inn!

To unexpunge all the messages and mark them as undeleted as well:

su cyrus -c "/usr/lib/cyrus-imapd/unexpunge -adv user/username@example.com"

Sample output

restoring all expunged messages in mailbox 'example.com!user.username'
Unexpunged example.com!user.username: 163 => 518
Unexpunged example.com!user.username: 178 => 519
Unexpunged example.com!user.username: 179 => 520
...
Unexpunged example.com!user.username: 493 => 527
Unexpunged example.com!user.username: 494 => 528
Unexpunged example.com!user.username: 495 => 529
restored 12 expunged messages

NOTE: This isn't recursive

It will only restore the inbox. To find other folders use ctl_mboxlist

# su cyrus -c "/usr/lib/cyrus-imapd/ctl_mboxlist -d" | grep domain.com
example.com!user.username       0 fs1 username@example.com      lrswipkxtecda
example.com!user.username.Drafts        0 fs1 username@example.com      lrswipkxtecda
example.com!user.username.Sent  0 fs1 username@example.com      lrswipkxtecda
example.com!user.username.Trash 0 fs1 username@example.com      lrswipkxtecda

Run the unexpunge command for every folder that needs to have mail undeleted.

For folder names that have spaces ' ', the spaces need to be escaped with a backslash

su cyrus -c "/usr/lib/cyrus-imapd/unexpunge -adv user/username/Deleted\ Items@example.com"

Renaming a mailbox

In this example, all mailboxes at example.net need to be renamed to example.com - any subfolders can be renamed manually, or you can just do the top level INBOX and all the subfolders will also be renamed.

$ cyradm -u cyrusadmin localhost

cyradm> renamemailbox user/bob@example.net user/bob@example.com
cyradm> renamemailbox user/info@example.net user/info@example.com

NOTE - if renaming a mailbox or folder with a single quote (') in the name, you will need to escape it.

localhost> renamemailbox user/mick^o\'brian@example.org user/mick^o\'brian@example.com

If you are renaming or working with folders with spaces, you can either escape the spaces or put double quotes (") around the mailbox

localhost> renamemailbox "user/me/My Folder@example.org" user/me/MyFolder@example.org
localhost> renamemailbox user/me/My\ Folder@example.org user/me/MyFolder@example.org

Error: can't fork process to run service Resource temporarily unavailable

This is when the number of files/processes that cyrus is using has exceeded the ulimit variables.

To fix, add the following into /etc/security/limits.d/90-nproc.conf

cyrus      soft    nproc     unlimited
cyrus      hard    nproc     unlimited
cyrus      soft    nofile    30000
cyrus      hard    nofile    30000

And then reboot the server (can we just do this instead? service cyrus-imapd stop ; service cyrus-imapd start)

IOERROR: opening /var/lib/imap/user_deny.db: No such file or directory

To fix the error

Jul 10 12:04:35 servername imap[3431]: IOERROR: opening /var/lib/imap/user_deny.db: No such file or directory

The solution is as follows:

echo "" > /root/user_deny.flat
/usr/lib/cyrus-imapd/cvt_cyrusdb /root/user_deny.flat flat /var/lib/imap/user_deny.db skiplist
chown cyrus:cyrus /var/lib/imap/user_deny.db

Is there a tool to add/remove a user to/from the user_deny.db?

From Dan White via the cyrus mailing list back in 2010

You can use cyr_dbtool:

touch /var/lib/imap/user_deny.db
chown cyrus:mail /var/lib/imap/user_deny.db
su - cyrus
cyr_dbtool /var/lib/imap/user_deny.db flat set dwhite "2     pop3    Can't use pop."

For the format of the database value, see Cyrus IMAP docs

When using the shell, you will need to escape your tabs. In bash, I typed:

"2<ctrl-v><tab>pop3<ctrl-v><tab>Can't use pop."

And then I get:

telnet localhost 110
Trying 127.0.0.1...
Connected to zek.olp.net.
Escape character is '^]'.
+OK zek Cyrus POP3 v2.3.16 server ready
<17564574347538583243.1276265278@zek>
user dwhite
+OK Name is a valid mailbox
-ERR [SYS/TEMP] Can't use pop.
Connection closed by foreign host.

System I/O error on attempted delivery

The exim log will show something similar to this:

LMTP error after RCPT TO:<xxxx@example.com>: 451 4.3.0 System I/O error

If you get the above I/O error when trying to deliver to a mailbox, it probably has something wrong with the cache files in the spool. To fix this:

root# su - cyrus
cyrus$ rm /var/spool/imap/domain/Y/example.com/X/user/xxxx/cyrus.*
cyrus$ /usr/lib/cyrus-imapd/reconstruct user/xxxx@example.com

It could take a while depending how many messages are present in the mailbox, but none of them will be lost and in some cases, very recently deleted messages will be recovered as well.

Y/example.com/X/user/xxxx is the path to the spool - the Y and X vary depending on the hashing used and if you're not sure, you can use the following commands to try and find the path.

root# ls -d /var/spool/imap/domain/?/example.com/?/user/xxxx

MUPDATE/IMAP administration

In the event of broken mailboxes, there are a number of things that can be done by poking directly at the mupdate or imap servers. Using the mupdatetest program, you can query and modify the central database directly. Note the format of mailbox names is domain.com!user.user^name where the dots in the user part of the email address are replaced with ^

RFC for MUPDATE protocol - http://tools.ietf.org/html/rfc3656

MUPDATE

Connecting to the mupdate server is similar to connecting to the IMAP server.

root# mupdatetest -a cyrusadmin mupdatehost

MUPDATE - FIND

Query the mupdate server to see if a mailbox exists

mupdate> F01 FIND "example.com!user.username"
F01 MAILBOX {25+}
example.com!user.username {26+}
mailstore1.example.net!fs1 {20+}
username@example.com
F01 OK "Search completed"

MUPDATE - DELETE

Delete a mailbox

mupdate> D01 DELETE "example.com!user.username"
D01 OK "done"

MUPDATE - ACTIVATE

Create a mailbox (note this doesn't actually create it on the backend server, just within the mupdate server)

mupdate> A01 ACTIVATE "example.com!user.username" "mailstore1.example.net!fs1" "username@example.com lrswipkxtecda"
A01 OK "done"

IMAP

To create the folder, you need to login to the cyrusbe server where the mailbox needs to reside and run the following - this part is the IMAP server, not the MUPDATE server. It may also remove the need for the above ACTIVATE command to be run.

imtest -a cyrusadmin localhost

If you need to authenticate as a different user without knowing their password, imtest will also allow this provided you're using a SASL mechanism that supports proxying such as PLAIN or DIGEST-MD5.

imtest -a cyrusadmin -u someuser localhost

IMAP - GETANNOTATION

This command can be used to fetch the amount of space left on the partition.

0 GETANNOTATION "" "*" "value.shared"
* ANNOTATION "" "/vendor/cmu/cyrus-imapd/freespace" ("value.shared" "107410772")
0 OK Completed

The value returned is in KB. This is the same information as the following (but in a raw form).

cyradm> info
Server Wide:
  freespace: 107410772

IMAP - LOCALCREATE

This command causes the IMAP server to create a local folder. If there is a folder name rather than just INBOX, the folder name is inserted just before the @ i.e user/username/folder@example.com

LC1 LOCALCREATE user/username@example.com
LC1 OK Completed

It can also be written on a single line.

LC1 LOCALCREATE user/username@example.com

IMAP - SETACL

Next we have to set permissions on the newly created folder so that the user can access it.

ACL0 SETACL user/username@example.com username@example.com lrswipkxtea
ACL0 OK Completed

IMAP - MUPDATEPUSH

And finally we push the details to the mupdate server so that everything syncs up properly.

MP1 MUPDATEPUSH user/username@example.com
MP1 OK Completed

IMAP - LOCALDELETE

To remove a folder locally:

LD1 LOCALDELETE user/username@example.com
LD1 OK Completed

IMAP - SETQUOTA

When logged in as an admin, you can set/get quotas.

Cyrus IMAPd 2.5 (current git master branch) has the ability to set a limit on the number of folders a user can have as well as an overall storage quota.

With unixhierarchysep on:

>> A SETQUOTA user/username (X-NUM-FOLDERS 20)

<< A OK Completed

or with unixhierarchysep off:

>> A SETQUOTA user.username (X-NUM-FOLDERS 20)

<< A OK Completed

And to fetch the folder quota (with unixhierarchysep on, replace / with . if off):

>> A GETQUOTA user/username

<< * QUOTA user/username (X-NUM-FOLDERS 5 20)
<< A OK Completed

5 is the current number of folders within that mailbox (including the INBOX), and 20 is the maximum number.

IMAP - Other commands

Basic IMAP commands. All commands must be prefixed with an identifier, just a string that will match the response to the command. Typically for manual interaction, it's sufficient to just use a single character - in the following examples, I'll use "A". Data sent to the server will be marked with ">>" and data received will be marked using "<<" followed by the prefix "A"

You can use imtest or telnet to do these commands. Note that imtest will perform the initial login for you.

Logging in

>> A LOGIN username password

<< A OK [CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE ACL RIGHTS=kxte QUOTA
   MAILBOX-REFERRALS NAMESPACE UIDPLUS NO_ATOMIC_RENAME UNSELECT CHILDREN
   MULTIAPPEND BINARY CATENATE CONDSTORE ESEARCH SORT SORT=MODSEQ 
   SORT=DISPLAY THREAD=ORDEREDSUBJECT THREAD=REFERENCES ANNOTATEMORE
   LIST-EXTENDED WITHIN QRESYNC SCAN XLIST URLAUTH URLAUTH=BINARY 
   X-NETSCAPE LOGINDISABLED COMPRESS=DEFLATE IDLE] User logged in 
   SESSIONID=<bocks.com-15513-1374369517-1>

In this instance I've used a plaintext password, but I think this may vary depending on server settings (it's always worked for me so far though). It might have something to do with the imapd.conf setting "allowplaintext:on"

Opening a folder

>> A SELECT INBOX

<< * 673 EXISTS
<< * 0 RECENT
<< * FLAGS (\Answered \Flagged \Draft \Deleted \Seen $NotJunk NotJunk $Junk)
<< * OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen $NotJunk NotJunk $Junk \*)] Ok
<< * OK [UNSEEN 33] Ok
<< * OK [UIDVALIDITY 1374358277] Ok
<< * OK [UIDNEXT 675] Ok
<< * OK [HIGHESTMODSEQ 677] Ok
<< * OK [URLMECH INTERNAL] Ok
<< A OK [READ-WRITE] Completed

If you want to know what the response means, go read the documentation. An "OK" as the last line is a good sign though.

Reading a message

If you only know the UID, you have to convert it to a message id that the IMAP server will understand before you can read it.

>> A SEARCH UID 100

<< * SEARCH 99
<< A OK Completed (1 msgs in 0.000 secs)

This means that if we want to read the UID '100' you have to fetch 99 instead. Not sure why this happens, but I suspect it's because you'll get missing messages eventually as older ones are deleted and the message id is simply a sequential number.

In the above example, a little investigation reveals UID 1 to be missing.

>> A SEARCH UID 1

<< * SEARCH
<< A OK Completed (0 msgs in 0.000 secs)

UID 1 is missing.

>> A SEARCH UID 2

<< * SEARCH 1
<< A OK Completed (1 msgs in 0.000 secs)

UID 2 returns message id 1

Next step is to actually fetch the message

>> A FETCH 99 FULL

<< * 99 FETCH (FLAGS (\Deleted \Seen) INTERNALDATE " 6-Jul-2013 04:02:12 +0100" 
   RFC822.SIZE 5998 ENVELOPE ("Sat, 06 Jul 2013 04:02:12 +0100" 
   "Logwatch for wood.bocks.com (Linux)" ((NIL NIL "logwatch" "wood.bocks.com"))
   ((NIL NIL "logwatch" "wood.bocks.com")) ((NIL NIL "logwatch" "wood.bocks.com"))
   ((NIL NIL "root" "wood.bocks.com")) NIL NIL NIL "<E1UvIlB-0004xH-U4@wood.bocks.com>")
   BODY ("TEXT" "PLAIN" ("CHARSET" "iso-8859-1") NIL NIL "7BIT" 5406 174))
   A OK Completed (0.000 sec)

This provides some of the headers and the body part specifies something useful if you happen to speak IMAP! Next we'll just fetch the whole message content in an easy to read format.

>> A FETCH 99 BODY[]

<< * 99 FETCH (BODY[] {5998}
   Return-path: <root@wood.bocks.com>
   Envelope-to: root@wood.bocks.com
   ...
   )
   A OK Completed (0.000 sec)

The final ) is to match the one before BODY[] on the first line. I believe the 5998 is the number of bytes that follows.

Mark all messages in the current folder as read

>> A UID STORE 1:* +FLAGS.SILENT \Seen

In this context, * means the message with the highest uid, so 1:* means all messages (uid 1 to the highest uid).

Delete a message

>> A UID STORE 1 +FLAGS.SILENT (\Seen \Deleted)
>> A EXPUNGE

The above will mark message uid 1 as deleted, and then expunge the folder. Be aware that expunge will purge any other deleted messages in that folder, for example if you have previously deleted other messages via webmail, a mail client, or telnet.

Logging out (nicely)

Obviously you could just terminate the connection, but the better way is to logout nicely.

>> A LOGOUT

<< * BYE LOGOUT received
<< A OK Completed

At this point the connection is closed and you should be returned to a shell prompt.

Migration from dovecot

Because dovecot (configured to use mbox + /var/spool/mail/$USER) allows all kinds of extra characters in folder names, you may experience some issues migrating to cyrus. I found the best way was to setup cyrus on a separate port number and use imapsync with the following options:

# imapsync \
  --pidfile /tmp/${USERNAME}-imapsync.pid \
  --host1 127.0.0.1 \
  --host2 127.0.0.1 \
  --user1 $USERNAME \
  --user2 $USERNAME \
  --port1 143 \
  --port2 1143 \
  --delete2 \
  --nofoldersizes \
  --noreleasecheck \
  --subscribe_all \
  --password1 "$PASSWORD" \
  --password2 "$PASSWORD" \
  --regextrans2 's/[\(\)\!]/~/g'

The last one is the one that strips out ( ) and ! from folder names and replaces them with ~ instead. For the most part, this seemed to fix folders with bad names. You'll know you have them because without the last line you'll get errors like this appearing in the imapsync output.

Couldn't create folder [INBOX/!!Replies!!] from [!!Replies!!]: 8 NO Invalid mailbox name

If you also get errors saying the following error (typically in Sent Items), you can add "--addheader" to the options above to fix it by adding its own Message-Id if it's missing.

Host1 uid 6 no header by parse_headers so taking whole header with BODY.PEEK[HEADER]
Host1 Sent Items/6 size 1633 ignored (no wanted headers so we ignore this message)

With --addheader it says this instead.

Host1 uid 6 no header by parse_headers so taking whole header with BODY.PEEK[HEADER]
Host1 uid 6 no header found so adding our own [Message-Id: <6@imapsync>]

Using telemetry to debug IMAP issues

If you have a user joe@example.com who is having problems with some kind of IMAP client, enable telemetry and see what exactly is being exchanged with the client.

Find the config directory for cyrus and assign it to an environment variable - you will need this in a minute. Environment variables are per-shell so you would need to do this every time.

# grep ^configdirectory /etc/imapd.conf
configdirectory: /var/lib/imap
# export CONFDIR=/var/lib/imap

Alternatively, just type it out in full wherever you see $CONFDIR below.

To enable telemetry for joe@example.com:

# mkdir -p $CONFDIR/log/joe@example.com
# chown cyrus: $CONFDIR/log/joe@example.com

To disable telemetry for joe@example.com, very carefully:

# rm -rf $CONFDIR/log/joe@example.com

When an IMAP connection is established, as long as you did the creation correctly you will see any IMAP traffic being logged to files within this directory. It may help to determine if the IMAP client is doing something odd.

Sieve scripts

Uploading a sieve script can either be done as the normal IMAP user, or as an admin. To upload a script as an admin user, you can use sieveshell.

# sieveshell --user=normaluser --authname=cyrus localhost

When prompted for the password, it's the password for cyrus, not normaluser that you need to enter. It will however upload the script to the normaluser account.

Simple spam filter

Create the spam folder for normaluser.

cyradm> cm user/normaluser/spam

Create a sieve script to filter the email and then upload it.

$ cat > spam-score-sieve.script <<EOF
require "fileinto";

if header :contains ["X-Spam-Score"] ["++++++++"] {
   fileinto "INBOX/spam";
}
EOF
$ sieveshell --user=normaluser --authname=cyrus localhost
> put spam-score-sieve.script
> activate spam-score-sieve
> quit

The above script will filter any email with a spamassassin score of 8 or higher into the spam folder.

Stats

On the Cyrus frontends (in a murder) or on standalone servers you can generate stats as to how many unique addresses are connecting using TLS vs non-TLS with the following command. This only covers the duration of /var/log/maillog which is typically just from 4am today but you could easily apply it to previous logs by changing the filename.

grep "User logged in" /var/log/maillog | awk '{ email=$9;
 if (email !~ /@/) { email=$8; } total[email] = total[email] + 1;
 if ($0 ~ /plaintext\+TLS/) { ssl[email]=ssl[email]+1; } else
 if ($0 ~ /PLAIN\+TLS/) { ssl[email]=ssl[email]+1; } }
 END { print "total = " length(total); print "ssl = " length(ssl); }'

Assorted Links

Cyrus: sieve scripts in shared folders