Hack 56. Read Yahoo! Mail from Any Email Client

Use your Yahoo! webmail account in a normal email client.

A few years back webmail exploded onto the scene. It was new, it was cool, and it enabled you to read your email from any computer with Internet access. In an age when people were increasingly using computers at work, school, and home, webmail was a natural progression for email access.

Despite the benefit of reading your email on any computer with a web browser, the very fact that the browser is your client is a tad limiting. Aside from the issue that webmail is slower than a normal email program, deleting and moving mail with webmail is a tedious process that involves selecting lots of checkboxes and waiting while the web site slowly updates your mailbox. Therefore, many people have opted to use alternative methods to access their webmail accounts using traditional email clients. However, not all webmail accounts allow access methods, such as POP3 and IMAP, so users have to use clever workarounds. This hack covers one such method to retrieve email from a Yahoo! email account.

8.3.1. Using FetchYahoo

To solve the problem of getting your email from the Yahoo! server into your own mailbox, the FetchYahoo script was written to screen-scrape the information from the web site. This screen-scraping involves connecting to the Yahoo! site, picking out the right bits of information from the HTML, and sending them back to your computer in the form of usable email. When combined with a few other tools, FetchYahoo is an invaluable piece of software.

To begin, grab FetchYahoo from http://fetchyahoo.twizzler.org/, and extract the tarball:

foo@bar:~$ tar xcvf  fetchyahoo-x-x-x .tar.gz

The directory that is extracted will have the version number in it, so you should rename it to a simpler directory name (this is useful if you create a cronjob later, as you can upgrade the script without having to update the cronjob entry):

foo@bar:~$ mv  fetchyahoo-x-x-x  fetchyahoo

You need to replace fetchyahoo-x-x-x with whatever directory name is extracted from the tarball.

8.3.2. Configure the Script

The FetchYahoo script needs to have a properly configured fetchyahoorc configuration file that specifies your login details and configures certain options in the script. A fetchyahoorc file is included within the main tarball, and you can configure it by loading it into a text editor. You can locate this file anywhere on your system as you will call it with an absolute path later, but keeping personal configuration files in your home directory is always a good idea.

The most important parts to configure are at the top of the file. Like many other Linux configuration files, each option has the format option_keyword = setting. For example, username = johnsmith sets the username option to johnsmith. To get started, you need to set the username and password options to your Yahoo! account settings (you don't need to put these settings in double quotes, as with other configuration files). It is also advisable that you set use-https = 1 to ensure secure communications between your computer and the mail server.

The spool option is another important setting. This indicates where your mail spool (the place where your mail is stored) is located on your Linux system. Most distributions place this in a file with your account name inside the /var/spool/mail/ directory. As an example, the user bob would have his mail spool at /var/spool/mail/bob. To check this, have a poke around /var/spool/mail and see if the username exists in thereevery user should have a mail spool file. If you don't have a mail spool (such as if your mail client cannot read mail from a spool), you can have the mail stored in maildir format by adding a slash (/) to the spool setting (e.g., /var/spool/mail/); this directory will contain the mailbox.

A variety of other options are in the configuration file, and each one is documented inside the file itself. These options control how much mail is delivered, if attachments are delivered, whether to empty the bulk mail directory, and other useful functions.

It is recommended that when you first use FetchYahoo, you should set no-delete = 1 so that your Yahoo! mail won't be deleted when running the script. When you are confident the script is working, set no-delete back to 0 so that it will delete the mail on the server, once it has been successfully received.

8.3.3. Run the Script

With the configuration file set up, execute the script by simply running:

foo@bar:~$ fetchyahoo --configfile=fetchyahoorc

The first time you run the script against a Yahoo! account containing lots of email messages, you might find the script downloads only some of the mail before exiting with an error. This is usually because the Yahoo! server is set up to deny malicious requests that bombard it with traffic, and it misinterprets your use of the FetchYahoo script as one of these malicious programs. If you get an error such as this, visit the web interface to Yahoo! mail, and if you cannot log in, this is likely what happened. To resolve this problem, simply leave the server alone for a few minutes and then try to access the web interface againif you can log in, the server has recovered and you can run FetchYahoo again. It might take a number of runs to get all your email transferred to your computer. You do not encounter this problem when downloading only a few emails at regular intervals.

8.3.4. Automate FetchYahoo with cron

Most people who use email want to check it regularly. With this in mind, you ideally want to run FetchYahoo every few minutes to update your mail spool with new emails, but you don't want to have to manually rerun the command every time; repetitive tasks such as this are what computers are for. To solve this problem, use cron to automate running the script [Hack #70] . First, you should edit your crontab:

foo@bar:~$ crontab -e

If you want to run the fetchyahoo command every three minutes, you can use the following cronjob:

*/3 * * * * /sources/fetchyahoo/fetchyahoo --configfile=/sources/fetchyahoo/
fetchyahoorc

In this cronjob, you are running the script every three minutes on every day, month, and year. It includes a full path to the location of the script because you have not installed the FetchYahoo script to your path, so you need to specify its path to the system. You also need to give a full path to the location of the configuration file.