Migrating from Google Apps to Office 365

Just over 6 years ago, I consolidated all my personal email accounts onto Google Apps back when the accounts were still free, however it wasn’t actually that long after that I switched up to a paid Google Apps for Business account – there were a number of useful benefits as part of the package, not just limited to the removal of advertising that plagued the interface. My view has always been that while free services are great, unless their is a sustainable business model behind it, don’t expect the service to always be around – especially important if you become heavily dependant on that service.

You might think that someone like Google are unlikely to drop free services? Take Google Reader as a prime example; an excellent online RSS reader, but dropped by the wayside in 2013, even though arguably it was the best service available at the time. Only in more recent years are there now services providing comparable equivalents, but you’ll notice the best ones are not 100% free. If you’re not paying for the product, you are the product.


Earlier this year I decided to move back into the Microsoft eco-system – the package of services within Microsoft Office 365 are hard to compete against. Even on a personal use level, for about £4 per month you get fully hosted email, calendar and contacts (Exchange), unlimited online file storage services (OneDrive for Business; aka Sharepoint) and a real-time communication tool (Lync/Skype for Business, which also provides a WebEx-like service). Believe it or not, take up of Office 365 has increased massively over the past 12 months, now exceeding Google Apps deployments, which many had thought would become the disrupter in this space to become the online office-suite ‘SaaS’ king, pushing Microsoft and it’s traditional software licensing model out of the market. However, Microsoft has adapted and is now unifying it’s approach across all platforms, not just providing software and services as first class citizens on Windows.

Once I was all setup, all said and done, the experience has been very good – the web interface is excellent, on my MacBook a new version of Outlook for Mac has been released which links into the same web interface, while with the acquisition of Acompli late last year, the now rebranded Outlook Mobile is now probably one of the best mobile email clients available. It’s not all been completely smooth sailing however, as migrating over all my historic data has brought about it’s own challenges, in particular the past 10+ years of email.

Migrating Email via IMAP

Within Office 365 there a Connected Accounts option which allows you to link to an existing (in this case, old) account, but for whatever reason it doesn’t support Google’s IMAP servers and so is effectively useless. Another option is to use the migration options available within Exchange’s administration interface, but again the file format of the CSV I created detailing my old account credentials were not accepted in any shape or form.

Googling (the irony) exposed many commercial software packages and chargeable services available to carry out a migration, but with some persistence I did come across a more hands-on approach, providing an ideal minimal cost option. Rick Sanders’ IMAP Tools are a set of perl scripts, one of which (imapcopy.pl) can be used to copy mail between two different IMAP servers. Rather than try and get them up and running on my MacBook, I decided to spin up an instance on DigitalOcean – this would take advantage of the high speed connectivity from the DigitalOcean data centre, but also not require my own MacBook to remain online and connected throughout the migration process.

There are a couple of steps that need to be completed before hand – first, within Google Apps, make sure that IMAP has been enabled, and if two factor authentication has been enabled an app password will also need to be generated. Similarly, if two factor authentication is enabled on Office 365, an app password needs to be generated there as well.

Next, create an account at DigitalOceanuse this link to get $10 free credit. Create a Droplet, assign a name (this can be anything), set to the smallest size ($5 per month) in the nearest region (in my case, London), and set the distribution to CentOS 7 x64 – no other options are required. It takes around 60 seconds to complete, at which point click on the ‘Console Access’ button. Type the user name as ‘root’, then check your email for a message from DigitalOcean, which contains the root password. When prompted, change the password, then you’ll have completed the login process.

There are a couple of dependancies to install, so type:

yum install -y perl openssl perl-IO-Socket-SSL screen

The IMAP Toolkit is no longer free, however the last free version of imapcopy.pl is still available on Google Code. This can be downloaded and prep’d with the following commands:

wget https://imaputils.googlecode.com/svn-history/r5/trunk/imapcopy.pl
chmod 755 imapcopy.pl

Next, to be able to convert between Google and Office 365’s folder structure, a mapping file needs to be created using the following command – I wanted to move all mail into a single folder on Office 365 called ‘migrated’, but you can add additional mappings as required:

echo [Gmail]/All Mail:migrated >> map

Finally, to start the migration process use the following commands – this will use the screen command so that if you loose your connection to the DigitalOcean host, the migration will still carry on in the background uninterrupted. Be sure to replace your Google Apps & Office 365 email address and password where highlighted below – if you did change the mapping file above, you’ll need to add the comma separated additional folders to be migrated to the -m command:

time ./imapcopy.pl -S imap.gmail.com:993/<google email address>/<google apps password> -D outlook.office365.com:993/<office 365 email address>/<office 365 password> -v -U -d -I -z -M map -m "[Gmail]/All Mail" -L migration.log

Depending on the size of your mailbox, this may take a few hours to run. Once complete, the time command will display how long the migration took to complete. The logfile ‘migration.log’ will have full details of every action taken, including any errors encountered – the migration command can be re-run exactly as-is and it will pick-up where it last left off. If you do get disconnected at any point, reconnect back to your Droplet and login as root as before, then type the following to reconnect back to the migration script:

screen -D -r

On a few occasions I found that the copy hit a problematic email, which prevented the migration from proceeding – I was able to resolve this by opening up the log file using the following command:

nano migration.log

Then pressing CTRL-W to initiate a search, then typing ‘BAD FETCH’. This would jump to the place in the log where the issue first occurred. Scrolling up a few lines would reveal the message ID (it looks like an email address) of the problematic email, which I then searched for in Google Apps with the search phrase ‘rfc822msgid:<message id>’. Once found, I deleted the email then restarted the migration script – fortunately, none of the problematic emails were important.

Checking for Duplicates

After the migration had completed, I noticed that a few disconnects/reconnects had taken place, so I decided to check that no duplicates had been created, just in case the migration had re-started from scratch on each reconnect. Another Google search later produced the answer with Quentin Stafford-Fraser’s IMAPdedup hosted on GitHub. This can be download and prep’d as follows:

wget https://raw.githubusercontent.com/quentinsf/IMAPdedup/master/imapdedup.py
chmod 755 imapdedup.py

Similar to the migration script, however this time it is executed against only the folder copied over to Office 365 (i.e. ‘migrated’). I prefer to do this, rather than against the original source on Google Apps because if there are any issues / lost mail etc, as a roll-back I can re-copy over the mail from Google Apps. As an extra pre-caution, it’s initially run with a -n, which will carry out a dry-run, not actually deleting any mail, but also with -c and -m does a check-sum against the To, From, Subject, Date, Cc, Bcc and Message-ID fields as the safest method to check for duplicates.

./imapdedup.py -s outlook.office365.com -u <office 365 email address> -x -c -m -n migrated

A final report details how many messages would have been deleted as duplicates, if any. To proceed with deletion of the duplicates, re-run the command without the -n:

./imapdedup.py -s outlook.office365.com -u <office 365 email address> -x -c -m migrated

Once you’re happy that all your email has successfully migrated over, log back into the DigitalOcean control panel, and destroy the Droplet – the account will only be charged the hourly usage while the Droplet was running, deducted from the free $10 credit if you used the sign up link previously mentioned.


“I became an expert overnight” said no-one, ever!

It’s easy to believe that some people were born in life to do what they do, that somehow they are gifted at birth or it’s part of their DNA. The way that they make it look so easy, that their knowledge just flows from their lips or fingertips like it was gospel, or that their power and hold over an audience is natural and unrehearsed. In reality, when you see them in full flow of their ‘expertise’, it’s a common mistake to forget or to not even realise how much time they have spent repeating over and over and over, through trial and error, through failure and success, through sacrifice.

Is it our own mistake, or is that what we’re taught to believe? Today’s modern society can unfortunately deceive those who wish to take on a new skill or ability, with promises of success without the effort, achievements without the commitment, and reward without the risk. In turn, this causes those who fall at the first hurdle all too easily to give up and lose hope, never again opening up the opportunity to understand their own potential.

The truth is, anyone can achieve what they want in life by committing themselves to that outcome. Whatever it is you’re trying to learn or the goal you want to achieve, the investment of time in repeated rehearsal and then taking inspiration from failure (because there will be many opportunities), are two key ingredients for success.

For myself, I’ve learned that the only way to know, is to try. How do you know if you can, if you never try?

Back at school I used to dread those afternoon cross-country runs, since it meant trudging through the rain for a hour, trying to keep up with those who seemed to otherwise enjoy the experience. I couldn’t see the point, and found relief on returning to warm up with a hot shower, quickly forgetting what just took place. Or perhaps they understood. Not the meaning of today’s running experience, but the longer term benefits of running every week. The repeated experience – each time making small progress, small improvements, growing their ability.

Earlier this year I decided to put a stake in the ground and conquer my aversion to running – if there is one event that needs to be done, it’s the London Marathon. If there is a reason to run, it’s to see if it’s possible to run that 26.2 mile distance, as someone who has always avoided the experience all together. How hard could it be? How far I could run, how long, how quickly? It is something that is unknown to me, as I had not tested myself, or tried.

The first few times I felt like dying since my body was quite rejecting the idea. Sitting in a chair all day it seemed, was much easier, less effort, more reward. But pushing through, and doing it again, different day, different place. Even after the first few weeks, stop start, walking to running, I began to see an improvement. Small progress, small improvements, growing my ability. Encouraged, I looked for ways to keep focused, building a habit, maintaining consistency, as I was living proof that the only way to improve is to repeat, over and over and over.

It’s still a good 6 months away, but with the targets I’ve set, using the tools I’ve found to help me out along the way, I should be on target to complete the marathon within a reasonable time. To be honest, I am not looking to set the world on fire, as It isn’t about showing to other people what I can do. Just being able to complete it at all is proving to myself, that I can, because I tried.

Man of the match, and his future self

The autumn weather has today arrived in full force, ensuring that Ben’s first under 7’s friendly was a truly muddy affair. With some good enthusiasm across the whole team, though having lost to a number of goals in the first half, a really good goal run down the field by Ben’s best friend really lifted the spirits of the team, even as the heavy rain kicked in the second half. With Ben sitting in goal, maintaining his smile throughout, I did my bit by shouting with all the other dads from the sidelines, which obviously made a huge difference to game. After the final whistle, although thoroughly soaked to the skin, Ben was rewarded with the man of the match trophy for his efforts.

The truth is, while I have never been a follower of football even though throughout my own schooling it was a major part of the sports program, I can the see benefits it brings to Ben with the experiences of team play, communication, and commitment that are building important foundation skills for later in life. What was preached by our own parents seems obvious now, but when you’re young and free from the realities of the world, the bigger picture just isn’t clear.

As a parent, my role isn’t written down as a set of neatly printed instructions – to be honest there is a heavy dose of creativity required. It’s a bit like buying a new flat-pack unit where the instructions are in chinese – just when you think you’ve figured it out, you’ve actually put the shelf on backwards. Then when you feel like giving up because it’s taken so much time already, you just push through and make the pieces fit, because it’s the end result that matters.

Choosing Your Direction

Throughout your career, every now and again you will come to an important junction, a point at which you need to make a choice, which depending on which path you progress, will fundamentally shape the rest of your working life. Sometimes this can be for the better, sometimes for worse, but in any event, such a decision should never be taken lightly – sometimes, you may not even know that the choice exists, or the extent of the implications of what the choice will mean.

Thinking back across my own 15 year career in Information Technology, I would say I’ve reached such a point on three separate occasions, the most recent has occurred only just within the past couple of weeks, and I can guarantee that there will be more to come.

Assumed Choice:

I wasn’t someone who took the university route, or someone who really took to school in any serious way – it was somewhere I was ‘encouraged’ to learn subjects, in which I saw no value. This is probably a common situation, and as a parent I’ll face my own battles in the years to come, but from the young persons perspective it isn’t made clear how the taught subjects have relevance beyond scoring well on an exam paper. Perhaps times have changed, but so frequently it hits the media with increasing numbers of successful exam grades, increasing numbers of the unemployed young, but businesses are still crying out for relevant ‘business skills’ to be taught in education. I am not one to dig too deep into the topic, but I do recommend taking ten minutes to watch a presentation by Sir Ken Robinson, adapted from a talk given at The RSA, who talks about Changing Education Paradigms.

Computers and technology – that was something that interested me, and has naturally driven me into my career as it is now. Coming out of Sixth Form and looking at my options, my first career choice was not that which seemed assumed. I applied for and accepted my first full time position within a small software company, and for me served to feed my interests while paying a full time wage. While my friends were away at University following their own ambitions, I had money in my pocket and food for my brain. More importantly, it was in my view that experience, especially within such a fast growing, ever changing industry would hold more value than a college or university qualification.

But what if I had gone to University instead?

Inadvertent Choice:

Some years later, working another position for a big blue chip, I was well into growing my experience, my technical skill set and my business knowledge, though there was something new I was keen to try – contracting. At the time, it was something I had thought about, and had talked to a number of colleagues who had already made the ‘jump’. It actually fell into view practically overnight – when I was looking for a new position, a call came through on an opportunity to start the following day – and so I became a contractor, which was easier than I thought. My advice to anyone wanting to do the same, is that making the move is the hardest step – beyond that, self motivation is the key skill that drives success.

Having made the move, it enabled us financially to buy our first home, then shortly after, to marry, and then start a family. They say money can’t buy happiness – I don’t believe this, up to a point, as many of the financial commitments that we made together would previously have not been possible. Career progression in a permanent position, working up the corporate ladder simply does not provide the same renumeration that’s possible when freelancing within a growing market – there are benefits to both of course, with different elements of risk, and it does not always suit every type of personality.

But what if I had taken on a permanent role instead?

Considerate Choice:

Much more recently, with finances under control, the possibility and desire of driving deep into a role had became a niggling thought at the back of my mind – this is the downside to contracting. You’ll always be good at doing the same function for ten years, but being able to move into a different area of IT is much more difficult.

As an example, you could be the best Oracle Database Specialist in the world, able to build and support systems with your eyes closed – being put forward and offered contracts as a DBA is a frequent event. However, you’ve an interest in moving into Computer Security – you spend much of your spare time ‘getting up to speed’, taking on and adding value in your existing contracts by carrying out forensic investigations, or incident response following a computer intrusion, and even earning various related certifications. Trying to then get a contract, not as a DBA, but in an Information Security context is much more difficult. Contractors are taken on a temporary basis because they already have the required skill set and experience to do the job, from day one – there is no scope to train the contractor, or help grow their experience in a different area.

On the other hand, in a permanent role, there is an obligation and incentive for an organisation to grow their staff, alongside the growth of the business itself. After all, an organisation is only as successful as it’s people. Not only that, responsibilities are generally assigned and owned, where as contractors, because of their temporary status, are generally task driven based on specific deliverables.

My contract was coming to an end, with no possibilities for extension – I actually felt disappointed, as the role itself was really interesting and challenging, working for a company with rapidly expanding horizons, and for the first time was not looking forward to change. Surveying the market for new contracts did not bring to light anything ‘new’, and I was at a point in my career where I wanted to grow further, take on more responsibility, and more importantly, lay a foundation within an organisation in which I felt a real sense of belonging, on top of which I could continue to build a successful career.

In the end, after following up on a few positions, I had two confirmed permanent options on the table. Both were similar roles, within similar organisations and both with huge amounts of potential – greater than any I had been offered before. When deciding between them, I had to make considerations beyond the basics of the roles themselves, something which I had not done previously. It was one of those situations where you want to make the right choice, but don’t want to turn the other one down – If either one was the only option available, I would have chosen it there and then.

Having now made the choice, the uncertainties with finding contracts no longer exist – and while the financial benefits of contracting also no longer exist, I look forward in excited anticipation on the new possibilities that now lay ahead, that were previously not even accessible.

But what if…? Hindsight is a wonderful thing, but I’ve never needed to look back. I would make the same choices again.