I usually use Emacs’ gnus as my email reader; familiar key bindings, integration with the rest of my workflow, what’s not to like?
Gnus has a bit of a reputation of being hard to configure. On that reddit thread I counter that but, on reflection, my email set up uses quite a few moving parts – though it’s fair to say that the gnus configuration is probably the simplest part.
The underlying problem is that gnus, like most of emacs, is single threaded. Whilst it’s off trying to retrieve email it blocks all other activity, resulting in delays that encourage you not to check your email. Not checking may not be a bad thing, but it should be your choice!
Overall approach
I have two main email accounts I want to access: a personal one hosted on Gmail and a work one hosted on Office 365. My approach is to use mbsync to sync both of those accounts over IMAP to an email server, dovecot, on my computer. gnus is then pointed at the dovecot instance on localhost.
Email server
The dovecot website is the best source of information for obtaining and installing the email server. My use of it is limited – I’m only serving a couple of accounts, on localhost only, with only single user access to the computer. Because of this I can get by with a simplified configuration:
listen = 127.0.0.1
protocols = imap
# Disable SSL for now.
ssl = no
disable_plaintext_auth = no
# We're using Maildir format
mail_location = maildir:/path/to/mail/root/%u
mail_uid = 1001
mail_gid = 1001
# Authentication configuration:
auth_verbose = yes
auth_mechanisms = plain
passdb {
driver = passwd-file
args = /etc/dovecot/passwd
}
userdb {
driver = passwd-file
args = /etc/dovecot/passwd
}
Listening is constrained to localhost, SSL is disabled, and plain text authentication is allowed. Passwords for the virtual users are stored in plain text in /etc/dovecot/passwd in the form:
Dovecot password file: /etc/dovecot/passwd.
workacct:{PLAIN}secret1:::
personalacct:{PLAIN}secret2:::
This creates two users named workacct and personalacct. The %u in the setting for mail_location is replaced by the user name so I have two separate mail stores.
I’ve chosen maildir format for compatibility with gnus. These directories need to be created, along with the sub-directories cur, tmp, and new.
dovecot‘s logging is extremely useful in diagnosing any configuration issues and is decribed well in the administration manual.
Gnus
With a hopefully fully functional email server set up the next stage is to persuade gnus to connect with it.
Gnus config file.
(setq gnus-select-method
'(nnimap "personalacct"
(nnimap-stream network)
(nnimap-port 143)
(nnimap-address "127.0.0.1")))
(setq gnus-secondary-select-methods
'((nnimap "workacct"
(nnimap-stream network)
(nnimap-port 143)
(nnimap-address "127.0.0.1"))))
Authentication is handled by appropriate username and password settings in .authinfo.
I’ve chosen to use the nnimap mail source to access email over IMAP. An alternative approach would be to use the maildir source to access via the file system with less overhead. I’ve chosen the nnimap approach as I use additional features of dovecot, such as Full Text Search (fts) to speed up searching my email archive.
IMAP synchronisation
With a working email server and client it’s now time to transfer email from the remote servers to the local one. I do this using mbsync.
Simplifying greatly, mbsync takes the credentials for your remote server, logs in to that, checks for new emails and copies them over IMAP to a local server. Synchronisation also runs in the other direction allowing emails to be marked as read and so on.
Gmail synchronisation
Logging into Gmail is reasonably straight forwards, even with two factor authentication (2FA) enabled. Gmail allows you to generate an application specific password which bypasses 2FA. Logging into gmail then just requires your email address and the application specific password. Whilst they could be entered directly into .mbsyncrc, I don’t like putting credentials into config files if I can help it – they are all too likely to get pushed to github, copied to colleagues or other exposures. Where possible I try to store credentials in my authinfo file. Fortunately mbsync provides configuration options for a command to retrieve user name and password. I use this to extract the required credentials from authinfo.
My personal gmail IMAP server mbsync settings are then:
Partial mbsync configuration file: Gmail login.
IMAPAccount remote-personal-acct Host imap.gmail.com UserCmd "emacs --batch -l ~/.emacs.d/get-credentials.el --eval '(get-user \"personal\")'" SSLType IMAPS AuthMechs LOGIN CertificateFile /etc/ssl/certs/ca-certificates.crt PassCmd "emacs --batch -l ~/.emacs.d/get-credentials.el --eval '(get-passwd \"personal\")'"
UserCmd and PassCmd use emacs in batch mode to retrieve the username and password from authinfo. The incantation to pull out the credentials using emacs’ built in auth-source-search is a little convoluted to be entering on a command line, particularly as it gets repeated, so I’ve broken that out into a separate file, get-credentials.el which is loaded by the -l argument to emacs.
get-credentials.el needs to extract the credentials and print them out to stdout. If the server I’m looking for doesn’t appear in authinfo I print an error message to stderr and exit with an error code (255):
(require 'auth-source)
(defun get-user (host)
"Print the user for HOST if found by `auth-source-search'."
(interactive "sHost:")
(princ (format "%s\n"
(get-credentials host :user))))
(defun get-passwd (host)
"Print the password for HOST if found by `auth-source-search'."
(interactive "sHost:")
(let* ((password (get-credentials host :secret))
(password-string (if (functionp password)
(funcall password)
password)))
(princ (format "%s\n" password-string))))
(defun get-credentials (host cred)
"Find required credential CRED for HOST.
Signal an error if host record cannot be found."
(let ((record (car (auth-source-search :host host :max 1))))
(if record
(plist-get record cred)
(error "Could not find credentials for host %s" host))))
The corresponding entry in .authinfo is then:
Example authinfo entry.
machine personal login me@example.com password supersecret port 993
mbsync then needs the corresponding details for the local, dovecot, email account and to know that emails from one are to be synced with the other:
Partial mbsync configuration file: local account and chaining it to the remote account.
IMAPAccount local-personal-acct Host 127.0.0.1 UserCmd "emacs --batch -l ~/.emacs.d/get-credentials.el --eval '(get-user \"localpersonal\")'" PassCmd "emacs --batch -l ~/.emacs.d/get-credentials.el --eval '(get-passwd \"localpersonal\")'" AuthMechs PLAIN SSLType None IMAPStore remote-personal Account remote-personal-acct IMAPStore local-personal Account local-personal-acct Channel personal Far :remote-personal: Near :local-personal: Sync All Create Near
Looking at AuthMechs and SSLType you’ll notice the lax approach to security again, relying on the fact that all communication is limited to localhost on a single user machine.
Microsoft synchronisation
Our week email uses Microsoft’s Office365 with two factor authentication enabled. This does not allow the use of application passwords in the way that gmail does. Instead we have to work with Microsoft’s flavour of OAuth. This initiates two factor authentication through a web browser and proved an access token which is used as a password to logon. The token expires periodically and the 2FA process has to be conducted again to obtain a new token.
The mutt mail reader provides a python script mutt_oauth2.py which implements this protocol, returning the access token needed for logon. With the aid of this script, once configured, mbsync access to Office 365 email is enabled with:
Partial mbsync configuration file: accessing Office 365.
IMAPAccount microsoft-work-acct Host outlook.office365.com UserCmd "emacs --batch -l ~/.emacs.d/get-credentials.el --eval '(get-user \"mywork\")'" SSLType IMAPS AuthMechs XOAUTH2 CertificateFile /etc/ssl/certs/ca-certificates.crt PassCmd "$HOME/bin/mutt_oauth2.py $HOME/bin/o365.tokens"
There’s a miserly “once configured” statement in the previous paragraph. The authentication process requires both a client ID and client secret]] to enable connection. These are given as empty strings in mutt_oauth2.py. You will need to populate those strings with appropriate values.
The Microsoft documentation gives the official process for obtaining those values. This requires admin privileges which I do not have. I have however discovered that the Thunderbird email client can connect to my work email over IMAP. So judiciously adding Thunderbird’s values of client_id and client_secret into mutt_outh2.py we get:
'microsoft': {
'authorize_endpoint': 'https://login.microsoftonline.com/common/oauth2/v2.0/authorize',
< lines elided >
'client_id': '08162f7c-0fd2-4200-a84a-f25a4db0b584',
'client_secret': 'TxRBilcHdC6WGBee]fs?QR:SJ8nI[g82',
},
Using this, for me, mutt_oauth2.py can successfully generate tokens to be used for login by mbsync. Have a look at the help output from that script and in particular note:
This script obtains and prints a valid OAuth2 access token. State is maintained in aencrypted TOKENFILE. Run with “–verbose –authorize” to get started or whenever all tokens have expired, optionally with “–authflow” to override the default authorization flow.
Whilst this setup is working for me at the moment, it does appear that the Thunderbird client ID and secret may be being trusted as a result of a bug. They have been advertised to Microsoft as truly secret but are in reality widely available and used by many in the same way I’m describing. It’s possible that changes in the way these credentials are handled may break a lot of people’s email access, including Thunderbird users. Have a read through the bug report for full details and a sense of the developers’ exasperation.
And finally
Having got all the individual components working together I automate the process of checking for and fetching new emails through an appropriate entry in my crontab. When setting those don’t forget that at execution time crond will not have the same setting for $PATH as your user shell so use explicit paths.
Imperfect Software 