Subversion Repositories sysadmin_scripts

CAUTION CAUTION CAUTION
=======================
This script has usernames and passwords in clear text. The output log file has information about your users accounts such as folders, number of messages, and how they log in. This is sensitive information.

This should be installed in a protected folder, owned by root:root, and permissions 0700. The scripts should be owned by root and permissions 0700. The output log file should stay in the protected folder.

YOU HAVE BEEN WARNED.

The script will run from any directory, but assumes all its little files are in that same directory. However, the sample cron job assumes the directory is /root/bin/imapsync. You will need to modify this if you put it someplace else.

Information:
===========
This is a wrapper around imapsync that allows you to rapidly and repeatedly get e-mail from one server to another. I use it to transfer e-mail accounts from one machine to another. There are tools that let you see when the DNS has changed over (no new traffic on source machine, but new traffic on the new one)

Installation
============
Create a folder, owned by root, with permissions 0700
Copy all files into it
chmod 0700 checkTraffic getStats imapsycn.sh
chmod 0600 users.list.example imapsync.include etc_cron.d_imapsync
mv etc_cron.d_imapsync /etc/cron.d/imapsync

Edit /etc/cron.d/imapsync to set the amount of time between runs, and the location of the scripts. Note, it will not run until a later step.

Use:
====

Following is suggested usage for moving one or more e-mail accounts from one server to another. We usually move a domain at a time, so the instructions are for that.

Set your TTL (time to live) to 6 hours on the DNS MX record for the domain you are moving. That means the "world" will check every six hours to see if that entry has changed.

Copy users.list.example to users.list.someidentifier. I generally name it after the domain I'm working on, so for example.com, I'd call it users.list.example.com. I'll use that in the following instructions

Edit users.list.example.com, adding a few lines. Each line is a four field record, separated by colons (:). The first column is the login on the source machine, the second column is the password, the third column is the login on the target machine, and the last column is the password on the target machine. See users.list.example for an, guess what, example :)

Run imapsync.sh in testing mode. Since this can take a while for any large e-mail account, I usually do a nohup and background. Since you are not using the default data file name, you will need to pass it in as the first parameter, with the second (and last) parameter containing literally anything to turn on test mode:
   nohup ./imapsync.sh users.list.example.com TEST &

At this point, the process is running, and you can follow the process by tail -f imapsync.log

When the process stops, look at your stats:
   ./getStats
The number of messages is meaningless (well, kinda), but what we are looking for is valid logins, making sure you have the correct usernames and passwords. You can look at the logins via the output of getStats, but a simple way is to run these two commands:
   ./getStats | grep Success | wc
   wc users.list.example.com
The output of the second command (number of e-mail accounts you are checking) should be HALF the number from the first command (each account has two logins, the source and the target). If these numbers match, you are likely configured correctly. If not, go figure out which ones are messing up and fix them.

Once getStats shows that you are working, you are ready to do a seed run. This will copy all e-mail from the source machine to the target machine. It will take a long time, depending on your connection speed, the speed of the machines, amount of memory, and number of e-mail messages. It will also likely saturate your network. Best done in the middle of the night. Do this run as:
   nohup ./imapsync.sh users.list.example.com &
this will overwrite your log file, and will do actual transfers. NOTE: I have specifically excluded any e-mail over 20 Meg (--maxsize 20971520) as this really puts a strain on things.

When the process is done, do a getStats to see what happened. The top part of the summary tells you how many messages you have transferred. NOTE: it is safe to run getStats while the process is running, and it give you an idea of which accounts have been done (and thus your progress).

At this point, you can change the DNS to point to the new server. Change your DNS record to point the MX to the new server (and any other records your clients use to check mail such as mail or smtp aliases). Be sure to reset the TTL to some reasonable value, like 24 or 48 hours.

You must update the new message store occasionally. For 84,000 messages, it takes about two hours to do a sync (gigabit network, two medium level machines). If you have configured your cron job correctly, simply copy users.list.example.com to users.list
   cp users.list.example.com users.list
the cron job will note that users.list exists, and run a sync every four hours. The summary (from getStats) will be e-mailed to the user in the MAILTO assignment near the top of the cron file.

Theoretically, after six hours (your TTL) the move will be complete. However, some large mail servers will not honor that. I usually run this for 24 hours. After that time, I feel only spammers will be still trying to use the old server. You can see the number of messages coming in on the old server via getStats, but NOTE: if a user deletes or moves an e-mail message on the new server, it will be copied over again on the next sync run.
You can also check the number of messages coming in on the new server. The script checkTraffic accepts one parameter, the domain name. It will look in /var/log/mail.log for any entry matching that domain name. Usually there is a simple word or phrase that the mail server uses to show that a message has been delivered, which is defined as KEYPHRASE at the top of the script. '(delivered via dovecot service)' is used on my DoveCot/Postfix server, so I use that.
   ./checkTraffic example.com
Should return the number of e-mails received in the day.

At the end of 24 hours, see if any new mail is coming in on the old server. If you are comfortable, you may now delete the old accounts. You should also delete users.list so the cron job will no longer run:
   rm users.list # you did save the original file, right???