Migrating from Google Apps to Office 365

Just over 6 years ago, I consolidated all my personal email accounts onto Google Apps back when the accounts were still free, however it wasn’t actually that long after that I switched up to a paid Google Apps for Business account – there were a number of useful benefits as part of the package, not just limited to the removal of advertising that plagued the interface. My view has always been that while free services are great, unless their is a sustainable business model behind it, don’t expect the service to always be around – especially important if you become heavily dependant on that service.

You might think that someone like Google are unlikely to drop free services? Take Google Reader as a prime example; an excellent online RSS reader, but dropped by the wayside in 2013, even though arguably it was the best service available at the time. Only in more recent years are there now services providing comparable equivalents, but you’ll notice the best ones are not 100% free. If you’re not paying for the product, you are the product.


Earlier this year I decided to move back into the Microsoft eco-system – the package of services within Microsoft Office 365 are hard to compete against. Even on a personal use level, for about £4 per month you get fully hosted email, calendar and contacts (Exchange), unlimited online file storage services (OneDrive for Business; aka Sharepoint) and a real-time communication tool (Lync/Skype for Business, which also provides a WebEx-like service). Believe it or not, take up of Office 365 has increased massively over the past 12 months, now exceeding Google Apps deployments, which many had thought would become the disrupter in this space to become the online office-suite ‘SaaS’ king, pushing Microsoft and it’s traditional software licensing model out of the market. However, Microsoft has adapted and is now unifying it’s approach across all platforms, not just providing software and services as first class citizens on Windows.

Once I was all setup, all said and done, the experience has been very good – the web interface is excellent, on my MacBook a new version of Outlook for Mac has been released which links into the same web interface, while with the acquisition of Acompli late last year, the now rebranded Outlook Mobile is now probably one of the best mobile email clients available. It’s not all been completely smooth sailing however, as migrating over all my historic data has brought about it’s own challenges, in particular the past 10+ years of email.

Migrating Email via IMAP

Within Office 365 there a Connected Accounts option which allows you to link to an existing (in this case, old) account, but for whatever reason it doesn’t support Google’s IMAP servers and so is effectively useless. Another option is to use the migration options available within Exchange’s administration interface, but again the file format of the CSV I created detailing my old account credentials were not accepted in any shape or form.

Googling (the irony) exposed many commercial software packages and chargeable services available to carry out a migration, but with some persistence I did come across a more hands-on approach, providing an ideal minimal cost option. Rick Sanders’ IMAP Tools are a set of perl scripts, one of which (imapcopy.pl) can be used to copy mail between two different IMAP servers. Rather than try and get them up and running on my MacBook, I decided to spin up an instance on DigitalOcean – this would take advantage of the high speed connectivity from the DigitalOcean data centre, but also not require my own MacBook to remain online and connected throughout the migration process.

There are a couple of steps that need to be completed before hand – first, within Google Apps, make sure that IMAP has been enabled, and if two factor authentication has been enabled an app password will also need to be generated. Similarly, if two factor authentication is enabled on Office 365, an app password needs to be generated there as well.

Next, create an account at DigitalOceanuse this link to get $10 free credit. Create a Droplet, assign a name (this can be anything), set to the smallest size ($5 per month) in the nearest region (in my case, London), and set the distribution to CentOS 7 x64 – no other options are required. It takes around 60 seconds to complete, at which point click on the ‘Console Access’ button. Type the user name as ‘root’, then check your email for a message from DigitalOcean, which contains the root password. When prompted, change the password, then you’ll have completed the login process.

There are a couple of dependancies to install, so type:

yum install -y perl openssl perl-IO-Socket-SSL screen

The IMAP Toolkit is no longer free, however the last free version of imapcopy.pl is still available on Google Code. This can be downloaded and prep’d with the following commands:

wget https://imaputils.googlecode.com/svn-history/r5/trunk/imapcopy.pl
chmod 755 imapcopy.pl

Next, to be able to convert between Google and Office 365’s folder structure, a mapping file needs to be created using the following command – I wanted to move all mail into a single folder on Office 365 called ‘migrated’, but you can add additional mappings as required:

echo [Gmail]/All Mail:migrated >> map

Finally, to start the migration process use the following commands – this will use the screen command so that if you loose your connection to the DigitalOcean host, the migration will still carry on in the background uninterrupted. Be sure to replace your Google Apps & Office 365 email address and password where highlighted below – if you did change the mapping file above, you’ll need to add the comma separated additional folders to be migrated to the -m command:

time ./imapcopy.pl -S imap.gmail.com:993/<google email address>/<google apps password> -D outlook.office365.com:993/<office 365 email address>/<office 365 password> -v -U -d -I -z -M map -m "[Gmail]/All Mail" -L migration.log

Depending on the size of your mailbox, this may take a few hours to run. Once complete, the time command will display how long the migration took to complete. The logfile ‘migration.log’ will have full details of every action taken, including any errors encountered – the migration command can be re-run exactly as-is and it will pick-up where it last left off. If you do get disconnected at any point, reconnect back to your Droplet and login as root as before, then type the following to reconnect back to the migration script:

screen -D -r

On a few occasions I found that the copy hit a problematic email, which prevented the migration from proceeding – I was able to resolve this by opening up the log file using the following command:

nano migration.log

Then pressing CTRL-W to initiate a search, then typing ‘BAD FETCH’. This would jump to the place in the log where the issue first occurred. Scrolling up a few lines would reveal the message ID (it looks like an email address) of the problematic email, which I then searched for in Google Apps with the search phrase ‘rfc822msgid:<message id>’. Once found, I deleted the email then restarted the migration script – fortunately, none of the problematic emails were important.

Checking for Duplicates

After the migration had completed, I noticed that a few disconnects/reconnects had taken place, so I decided to check that no duplicates had been created, just in case the migration had re-started from scratch on each reconnect. Another Google search later produced the answer with Quentin Stafford-Fraser’s IMAPdedup hosted on GitHub. This can be download and prep’d as follows:

wget https://raw.githubusercontent.com/quentinsf/IMAPdedup/master/imapdedup.py
chmod 755 imapdedup.py

Similar to the migration script, however this time it is executed against only the folder copied over to Office 365 (i.e. ‘migrated’). I prefer to do this, rather than against the original source on Google Apps because if there are any issues / lost mail etc, as a roll-back I can re-copy over the mail from Google Apps. As an extra pre-caution, it’s initially run with a -n, which will carry out a dry-run, not actually deleting any mail, but also with -c and -m does a check-sum against the To, From, Subject, Date, Cc, Bcc and Message-ID fields as the safest method to check for duplicates.

./imapdedup.py -s outlook.office365.com -u <office 365 email address> -x -c -m -n migrated

A final report details how many messages would have been deleted as duplicates, if any. To proceed with deletion of the duplicates, re-run the command without the -n:

./imapdedup.py -s outlook.office365.com -u <office 365 email address> -x -c -m migrated

Once you’re happy that all your email has successfully migrated over, log back into the DigitalOcean control panel, and destroy the Droplet – the account will only be charged the hourly usage while the Droplet was running, deducted from the free $10 credit if you used the sign up link previously mentioned.

Yesterday’s internet, today’s dependency

In today's connected society, access to the internet is now assumed. Everything we do, from paying taxes, doing shopping, watching TV, doing our job – it's all online, and it's only going to grow in ways we can't currently imagine or foresee. Just 10 years ago I had an ADSL 'Max' connection running into my home, providing a download speed of around 6Mbit per second, and 5 years before that I was one of the first ADSL customers in the UK with BT Broadband able to download at around 512Kbit per second – a huge jump compared to the dial-up modem of the 90s, literally 10 times faster (plus a lot quieter), marking out the very early foundations of the online social revolution.

Back to the present day, and I am now able to download at around 60Mbit per second through my BT Infinity connection (10 times faster than 10 years ago), or was, until my phone line went dead on Thursday. What is pretty much seen as a critical (or the 4th) 'utility' of the modern household, won't be back online until Wednesday at the earliest. To be fair, I've probably got a bit more tech than most, and I recognise that broadband speed still varies greatly across the UK, but in my house no internet means no TV (Netflix/Amazon Prime/YouView/YouTube), no Music (Spotify/Google Play), no gaming (Xbox/PC), no remote working, and the central heating can't check in for weather information / no remote access away from home (Nest), among other things. It is also fair to say I am not going to die due to lack of internet either, but just look how quickly our dependency on the internet has changed the way we run and operate our lives.

It also highlights a legacy that still exists – the phone line. I have a telephone number at home, but there are very few people who know it, and even less who ever call it, but the need to have one is still enforced because I need a fixed line internet connection. We've not got past needing to have a 'full' phone line, just to have a broadband connection – don't get me wrong, I need the physical wire into my property which also needs to be 'maintained', but I don't need the additional costs lumped into the line rental for services I don't use, which itself goes up every year without fail. The cost I pay for broadband should be all inclusive.

Fortunately, I have a mobile phone with a 3G internet connection, which means all is not lost. Although the experience of accessing the internet over a 3G connection feels like I have gone back 10 years (but being very much more dependent on reception), I am able to get a level of access which does not significantly change the way I live and work day to day. While my own use of technology is higher than most, it's also easy to see how most people can live without a fixed line connection. Research into this area is showing a year on year increase in the number of homes that have no phone line at all, as the use of mobile technology gradually negates the need for these services. With the advent of faster 4th generation (4G) data services, and beyond, what can be received over the air is becoming on-par with traditional fixed line broadband connections, except it's with you everywhere you go and not just at home.

Even Google gives bad support on PAID services

(updated: Thursday, 9th October, 2014)

I've hosted my personal domains on Google Apps for years without issue, to the point that I switched over to a paid account sometime back to get extra email routing options and to remove the adverts. However, just today, Google sent out the following email (I've added the highlight/bold):

Your Google Apps tax setting will change in the next few weeks


We're writing to let you know about a change to Google Apps that will affect your account for apps.andrewallen.co.uk. We will be changing your tax setting in Google Apps to “business” from its existing setting of “personal” in the next few weeks. After this change, we won’t add VAT to your Google Apps charges, and you’ll be responsible for determining taxes due in your country if you're not an Irish customer. If you are an Irish customer, you will still be charged VAT.

A “business” tax setting means that your Google Apps account is used for commercial purposes. We believe that your account is a business account, which includes corporates, affiliates, sole traders, self-employed merchants or professionals, and partnerships.

If you are using Google Apps for non-business reasons, then you will be allowed to sign up again as non-business account. We will send another communication with details about how to sign up again.


The Google Apps Team

No contact details were provided, and return emails just bounced. Makes you wonder, how do they decide (incorrectly in my case) if you're a business? Do they read your emails, your calendar and make assumptions? Is a human even involved, or is this a result of data mining all the information they hold on you from across all their services? Oh, and they graciously say I'll “be allowed” to sign up again as a non-business account.

It suddenly feels like, even though I pay for services, I have no control or say over my own account. Even I can't decide my own status. An Orwellian moment.

Update: Thursday, 9th October, 2014

I called up Google support over lunch to ask for clarification. The response was that all european Google Apps for Work customers will be switched over to a busines account setting, sometime in mid-November. I asked, what about all the other individuals who are not businesses, and use Google Apps for personal use, and the response after a couple of minutes on hold was that there will be an email sent out sometime afterwards, which will allow users to switch the setting back to personal.

Why would Google make such a bad broad assumption? Why cause unecessary work for it's customers? My immediate thought is this is some kind of accounting dodge – I am no way an expert and it's just personal opinion, but if Google no longer have to account for collecting tax from customers across all the various countries of Europe, and instead put the onus back onto their customers, it's got to same them some money, right? And with a number of large corporates being added to those being investigated by the European Commission for the way they operate their tax arrangements, including now Amazon, I don't believe it will be long before Google joins the same list.

In any case, the original email was terribly worded and insinuated that it's customers were acting nefariously in some way, and in my book that is totally unacceptable.

“I became an expert overnight” said no-one, ever!

It’s easy to believe that some people were born in life to do what they do, that somehow they are gifted at birth or it’s part of their DNA. The way that they make it look so easy, that their knowledge just flows from their lips or fingertips like it was gospel, or that their power and hold over an audience is natural and unrehearsed. In reality, when you see them in full flow of their ‘expertise’, it’s a common mistake to forget or to not even realise how much time they have spent repeating over and over and over, through trial and error, through failure and success, through sacrifice.

Is it our own mistake, or is that what we’re taught to believe? Today’s modern society can unfortunately deceive those who wish to take on a new skill or ability, with promises of success without the effort, achievements without the commitment, and reward without the risk. In turn, this causes those who fall at the first hurdle all too easily to give up and lose hope, never again opening up the opportunity to understand their own potential.

The truth is, anyone can achieve what they want in life by committing themselves to that outcome. Whatever it is you’re trying to learn or the goal you want to achieve, the investment of time in repeated rehearsal and then taking inspiration from failure (because there will be many opportunities), are two key ingredients for success.

For myself, I’ve learned that the only way to know, is to try. How do you know if you can, if you never try?

Back at school I used to dread those afternoon cross-country runs, since it meant trudging through the rain for a hour, trying to keep up with those who seemed to otherwise enjoy the experience. I couldn’t see the point, and found relief on returning to warm up with a hot shower, quickly forgetting what just took place. Or perhaps they understood. Not the meaning of today’s running experience, but the longer term benefits of running every week. The repeated experience – each time making small progress, small improvements, growing their ability.

Earlier this year I decided to put a stake in the ground and conquer my aversion to running – if there is one event that needs to be done, it’s the London Marathon. If there is a reason to run, it’s to see if it’s possible to run that 26.2 mile distance, as someone who has always avoided the experience all together. How hard could it be? How far I could run, how long, how quickly? It is something that is unknown to me, as I had not tested myself, or tried.

The first few times I felt like dying since my body was quite rejecting the idea. Sitting in a chair all day it seemed, was much easier, less effort, more reward. But pushing through, and doing it again, different day, different place. Even after the first few weeks, stop start, walking to running, I began to see an improvement. Small progress, small improvements, growing my ability. Encouraged, I looked for ways to keep focused, building a habit, maintaining consistency, as I was living proof that the only way to improve is to repeat, over and over and over.

It’s still a good 6 months away, but with the targets I’ve set, using the tools I’ve found to help me out along the way, I should be on target to complete the marathon within a reasonable time. To be honest, I am not looking to set the world on fire, as It isn’t about showing to other people what I can do. Just being able to complete it at all is proving to myself, that I can, because I tried.

Man of the match, and his future self

The autumn weather has today arrived in full force, ensuring that Ben’s first under 7’s friendly was a truly muddy affair. With some good enthusiasm across the whole team, though having lost to a number of goals in the first half, a really good goal run down the field by Ben’s best friend really lifted the spirits of the team, even as the heavy rain kicked in the second half. With Ben sitting in goal, maintaining his smile throughout, I did my bit by shouting with all the other dads from the sidelines, which obviously made a huge difference to game. After the final whistle, although thoroughly soaked to the skin, Ben was rewarded with the man of the match trophy for his efforts.

The truth is, while I have never been a follower of football even though throughout my own schooling it was a major part of the sports program, I can the see benefits it brings to Ben with the experiences of team play, communication, and commitment that are building important foundation skills for later in life. What was preached by our own parents seems obvious now, but when you’re young and free from the realities of the world, the bigger picture just isn’t clear.

As a parent, my role isn’t written down as a set of neatly printed instructions – to be honest there is a heavy dose of creativity required. It’s a bit like buying a new flat-pack unit where the instructions are in chinese – just when you think you’ve figured it out, you’ve actually put the shelf on backwards. Then when you feel like giving up because it’s taken so much time already, you just push through and make the pieces fit, because it’s the end result that matters.

Can I teach my kids to code? [Part 1]

With the change in air outside here in the UK and Autumn really beginning to kick in after the extended warm weather in September, it’s time to recognise that keeping the kids shut outside is no longer a humane method of childcare – besides, thawing a wet, cold child is not a peaceful affair, and certainly doesn’t afford the appropriate investment of time. In a shear stroke of coincidence, the fruits of a Kickstarter campaign I joined last year has just been delivered through my mailbox this week, which will again bring a sense of peace and calm to the household, at least if only for one day.

Good things come in small packages, which seems to hold true at least for this boxed up modern day electronics kit, as after taking out the bright orange box from the bland brown packaging, then sliding out the beige box within, the colourful picture of the printed circuit board on the front is revealed. With what appears to be an Apple like attention to detail, the front edge of the box lifts up, held in place with a satisfyingly grippy magnetic strip hidden away inside the cardboard interior, the neatly organised interior is finally exposed.

Nestled inside the top are a couple of manuals and sheets of stickers, while the rest is divided into different sections, each securly holding the various components and cables required to build out the computer itself. Since this is a UK version, it’s been supplied with an appropriate mains adapter, but otherwise requires that you supply the monitor or display, which does need to sport an HDMI connection.

Since it would defeat the purpose, I’ve managed to resist chucking out the manuals (who RTFMs anyway?) and slamming it all together myself, but instead had a quick flick through the coding manual to identify that my kids will be well and truly absorbed by the Minecraft section. It looks like it I might already be calling mission accomplished the quiet household front, since Minecraft has held their attention for many hours along with their friends, and to be honest some of the results of their virtual construction is very, very, impressive.

Closing back up the box ready for Sunday, the true test will be keeping them from fighting over who gets to touch it first.

Productivity on the Mac: aText

Recently I’ve been looking at a number of ways to improve productivity – this includes changing the way I work, live, and operate, but some of these improvements can be done in small steps as well. Having switched back to Mac at the end of last year as my primary device for work and play (the biggest improvements gained were through choosing high quality hardware with an SSD drive, by the way!) I have since built up a useful collection of software utilities which make those small improvements in productivity.

The latest of these is called aText, which I came across when originally looking at start using TextExpander. In simple terms, it will replace text automatically with pre-defined text, which you configure within the app. How is that useful? It takes some getting used to, but if you spent anytime thinking about it, you’ll find that you regularly type the same particular phrases all day long – this app allows you to, for example, type ‘tkvm’ in any application, and aText will replace that with ‘Thank you very much’.

Not an immediately obvious benefit, but as I’ve began to populate aText with business names, people, products, I do find myself writing emails, documents, meeting notes much quicker. In Evernote I recently built a template note which I use to document all my meetings – using aText, I can now quickly include attendees (e.g. ‘AA*’ becomes ‘Andrew Allen’), add dates (‘ddate’ becomes ‘Friday, 23 May 2014’), include timings (‘ttime’ becomes ’13:56’) etc.

Arranging meetings and including the conference bridge details is quick as well. I’ve setup ‘bridge*’ to be automatically replaced with:

To join the teleconference:

United Kingdom: 0800 xxxxxxx (freephone) or 0203 xxxx xxx
United States: 855 xxx xxxx (toll-free) or 404 xxx xxxx
Conference Code: 7785xxxxxx#

To view all global dial-in numbers, please click the link:

So why aText? Like I mentioned above, I was looking to use TextExpander as I heard it described on a recent productivity podcast, but I couldn’t justify ~$35 – fortunately a few Google searches later, I came across aText, which provides pretty much the same functionality and at only $5 on the App Store, it was a no brainer.