Here is a script I had to put together to remove all e-mail addresses containing a specific domain (“contoso.com” in this example) on all your objects in your on-prem AD.
Naturally this script can’t be altered to only include a specific subset of users, just change the “get-adobject” to match.
What sets this script apart from all the others you can find out there on the Internet is that this uses on-prem AD. Reason for this is that a lot of organizations out there are using their on-prem AD to sync their users and groups and not all of them have complete on-prem Exchange infrastructure to manage this so using this seemed a lot better. At least it did for us.
Of course this script writes to the console what it does every step of the way and it writes to date stamped transcript, making tracing and rollbacks easy-ish.
It won’t remove the e-mail address if it’s a primary e-mail since that may cause issues but it will flag it in red so pay attention to your console or search the transcript for “primary”.
$DateStamp = Get-Date -Format "yyyy-MM-dd-HH-mm-ss"
$Logfile = $LogFile = ("c:\temp\removing_mail_addresses-" + $DateStamp + ".txt")
start-transcript $logfile
$Mailboxes = Get-adobject -Filter {proxyAddresses -like "*contoso.com"} -Properties *
foreach($mailbox in $Mailboxes)
{
$DN = ""
$upn = ""
$DN = $mailbox.DistinguishedName
$upn = $mailbox.UserPrincipalName
write-host "Processing User $upn ..."
$i=0
while($i -lt $mailbox.proxyAddresses.Count)
{
$address = $mailbox.proxyAddresses[$i]
write-host -NoNewline "Processing mailadress $address..."
if ($address -clike "smtp:*" -and $address -like "contoso.com" )
{
write-host -ForegroundColor Yellow " removing address"
Set-ADObject -identity $dn -remove @{ProxyAddresses=$address}
$i--
}
elseif($address -clike "SMTP:*" -and $address -like "*contoso.com" ) { write-host -ForegroundColor Red " MAIL ADDRESS IS PRIMARY!" }
else { write-host -ForegroundColor Green " no action taken." }
$i++
}
write-host "---------------------------------------------------------------"
}
Stop-Transcript
Since I’ve worked for a charter company for a while now we’ve gone abroad to Spain, Greece or Turkey for the past decade or so. But given the current situation that wasn’t possible so we looked at around at some place to go in Sweden together with some friends. And we ended up with Daftö Resort – a camping resort on the west coast of Sweden which we hardly ever go to. So we booked a cabin from Thursday to Sunday and packed the car for a 6 hour road trip and off we went!
We made a quick stop in Mariestad to eat lunch and were happily surprised by the local restaurant “Kingfisher”, that Google suggested. Then some ice cream and off we went again.
When we arrived and tried checking in they looked at me like I was the town idiot. Which of course I was because we had booked Friday -> Monday! So there we were, outside Strömstad on the west coast with nowhere to stay! 4 calls later we had 2 rooms booked at a local “spa resort” which was a pretty damn good hotel. But they clearly didn’t care for the covid restrictions like “no hanging around the bars” so I wasn’t too comfortable there.
After spending the night there we went into Strömstad to explore what this coastal city had to offer. And there were some nice vistas and atmosphere and we even started fishing for some crabs down in the harbor. Not to eat though, just for fun!
Then back to Daftö resort and we were allowed to checking and get comfortable in our rather large 8 person cabin. Unfortunately they had some covid restrictions here that mostly applied to how many people could be in the pool area at any given time but we managed to get a spot to get some pool action but you can clearly see that people weren’t too concerned about covid here. Safe to say I wasn’t jumping in when it looked like this!
Another covid thing was that they had shut down their little ammusement park which was really sad for our kids. Even though we had told them it was closed so they didn’t mind much but we knew how much more fun they would have had with there. But I fully support them not having it open given the current situation.
What was open was their crazy golf course. And it was really, really good. Nice designs and themes, well maintained and if it wasn’t for the queuing I’d say it was perfect. I wait, we were at the front of the queue all the time! Unfortunately the restaurants at Daftö left a bit to be desired but I guess for a camping site it’s ok – we all got something to eat and it wasn’t horrible but it wasn’t up to our standards of “going out”.
We also got some time bathing in the sea and just kicking back in the cabin enjoying a few beers and I have to say that cabin was awesome – it had everything we needed and more (didn’t expect wifi!). We also tried some more restaurants in Strömstad and I can recommend the tapas restaurant “Il Toro” – but stick to the tapas. Not only was the tapas really good but the waiter even made a movie reference based off my wifes T-shirt that even I didn’t get, so kudos for that!
Then on Sunday it was time to pack up the cars and go back home. 6 hour drive back then unpacking and settling in to try to enjoy the F1 race from Silverstone. Which is hard since the Mercs are 1 sec faster than anyone else.
So that sums up our vacation for this year. Which was fun but also sad we didn’t do it more because our son really had fun without a computer!
I’m unfortunately not talking about that episode of Star Trek DS9 – it’s life in general right now. Not only is there a global pandemic that is crippling society and tanking our company and already caused my wife to lose her grandmother and her job.. now we got cancer in the family as well.
So now I’m gonna be one of those people who say – FUCK CANCER! Fuck it hard!!! Statistically it’s kind of amazing my family has been spared this long and I always assumed it would hit me because… well, not only am I the most expendable of the lot (in my opinion), I also live my life the least healthiest with all the chemical stuff I put in my body not to mention the red meat and then some lack of physical activities on top of that and sitting in front of a computer all day long, I thought it was a given! This fear is why I’m going to our local doctor so much and why I’ve have had two MRI scans in the past year. Both came back negative. But our luck had to run out.
We’re thinking of you and hoping for the best and let me know if there’s anything I can do!
So how are you handling this Corona thing?
Back in January even I thought China would manage to contain it. Neither the previous outbreaks had any major impact on life as I knew it. Even in the middle of the SARS outbreak I flew to Singapore without much hesitation because I had a job to do. But when we saw them building a hospital in what can only be described as panic I realised this would get out of hand. And boy did it ever!
But it didn’t take long for our CDC (“Folkhälsomyndigheten”, or “FOHM” for short) to tell people to work from home if possible. Which I gladly did and I’ve been to the office maybe four times in as many months. And working from home definitely has its challenges that I’ll make a separate post about. But I was still hoping it’d blow over rather quickly so we could all go back to normal but “little did I know” right…
In the middle of March my wife’s grandmother fell victim to our society’s failure to protect the elderly and she passed away which was a big shock to everyone because she was quite the iron lady and we thought that after a week of not getting worse that she’d pull through but unfortunately not.
So we, as in our family unit of 3, started doing our best to minimize the exposure. Not seeing as many people, not going out on many trips or excursions and staying home as much as possible. But the cabin fever of being home this much, especially when working from home, is rather intense and I’m so thankful that FOHM hasn’t instituted a lockdown in Sweden!
But by early April everyone realised how bad this situation was and all companies that were effected by this were allowed to give their employees a paid furlough – working from 20, 40, 60 or even 80% while retaining most of the salary. I wasn’t effected by this as I’m apparently that vital to our IT ops but most of my colleagues went down to 40% or less. And it also effected my wife’s work since people aren’t going out for coffee that much and B2B sales got hurt by everyone working from home so she went from working 5 days per week to 2. Which meant that for 3 days she was home while I was working from home which made it even more interesting.
In early June we finally got tested for antibodies which was negative so as it turns out, we just had the normal flu back in February.
Unfortunately that wasn’t sustainable and eventually there had to be downsizing. I survived the first round of layoffs but my wife didn’t so she’s soon out of a job. That sucks, not only because she’s unemployed but because this was a job where she worked with a product she loves – coffee.
But so far we’re in good health. Well more or less anyway, we’re not suffering from Covid at least. But I know 6 people that have had it and 3 of them are still suffering from fatigue, shortness of breath and heart arrhythmia, scary stuff to say the least so I’d rather sit this one out.
So right now we’re still doing the best we can in little family unit, limiting our exposure, shopping as much online as possible and trying not to go out unless we have to.
As you may know, I worked for the Nordic part of the Thomas Cook Group. I was the O365 admin for a tenant with over 30,000 user accounts. And we ran the Azure AD Connect service for the entire group and had just moved to pass-through authentication with Seamless SSO. Although it was a royal pain sometimes to work in such a large company where even a minor change could take weeks to implement and get approval for from everywhere.
As you may also know, Thomas Cook Group filed for bankruptcy in October last year. And there was no advanced warning or anything about what was going to happen next. But for our part, we realised that we would (if the company survived) most likely be moving our Nordic business to a new O365 tenant so we began planning for that. Over the next few months a lot of stuff happened. The Nordic part of the group was sold off and started their own company NLTG and the old group was shutting down all parts of their business. Except the German part because they were backed by their government so they survived (more on that later).
When we got back after the Christmas break we were given a clear order to evacuate the tenant before end of February. Since we were a separate company and legal entity we were no longer entitled to share the old tenant which, even though it makes sense, pretty much lit a torch under our asses to get this done now. And we realised it wouldn’t be a pretty or a smooth operation, as I recall saying, “this is going to take a sledgehammer, not a scalpel!”. Fortunately I’m very used to sledgehammer my way to getting results. Yeap, thinking back to that SharePoint upgrade that was all over the place!
So there we were, less than 8 weeks to pull off a migration with 3,000 users, 5,500 mailboxes, 10TB of SharePoint data, 8TB of OneDrive data and 12TB of Exchange data. And this is how it went…
Identities : The building block of any good tenant is the identities. When we first planned for the migration our plan was to have a new on-prem AD that would be fed by.. well that’s irrelevant since there was no time for that. The only way forward was to use our existing on-prem AD. But the problem was that MSFT doesn’t support syncing your on-prem identities to two tenants. Why? I have no idea – I fully get how you wouldn’t want that in a production environment (since the UPN domain can only be valid in one tenant) but for a migration like this it would have solved a lot of headaches if we were allowed to do it like that. But nope, we really wanted to have Microsoft support for this. And we also had to retain our e-mail domains since we’re heavily dependant on the brand which is almost as Swedish as Ikea, at least in Sweden. So that presented us with the first big problem – pre-populating the new tenant with 3,000 user objects so we could start copying the data and when it was time to migrate and then play around with the UPN domains so the matching would work. But the first step was creating the 3,000 users as cloud only “onmicrosoft” accounts. This was done using powershell to export as much info on the users as possible (including “usagelocation” and “preferredlanguages” since we’re an org with offices from Thailand to Mexico!) and then powershell to recreate the users as closely as possible. Another step we had to take here was setting up a filter in Azure AD Connect that would only sync users to each tenant depending on the value of an extensionattribute. That way we could make sure no user was synced to both tenants at any time, although it did take alot of tinkering to get that logic working but fortunately Microsoft have documented how to do attribute filtering, so thanks for that.
Authentication: Remember how I said we’d just gone over to PTA for the old tenant? Well this little thing meant that as long as users were logging in to the old tenant (which we knew the Germany company would) we couldn’t use PTA for our users since it’s all based around a computer object in the AD forest with a Kerberos encryption key that’s tied to the tenant! So if we set up PTA for our new tenant that would change the key on the computer object and they wouldn’t be able to login anymore! So to solve this we did a “quick and dirty” setup of a temporary AD FS for our users to use based on domain. This was a surprisingly easy thing to do in Windows Server 2019 but it was an added “gotcha!” of this entire scenario!
SharePoint : The first problem with SharePoint was to determine which sites were relevant to keep and which weren’t. Our entire SharePoint was well over 20TB so we had to make sure to only copy over sites we knew were relevant to the Nordics business. But there’s no way of determining that without going through all the underlying permissions and groups to determine if “our” users are working on the site or not. It’s not like you can ask SharePoint to “give me all the sites that any user with the UPN domain @domain.se is working on”. Or maybe there is, I just didn’t have the knowledge to write that powershell at the time. Once that was done we used ShareGate to migrate all the SharePoint data. The biggest fear was that it wouldn’t be able to match the old identities with the new ones – which it did! I’m pretty sure it went by “DisplayName” to match them but we’re just very very thankful it did because that would have been a mess to sort out. The biggest issue I had with ShareGate was how unpredictable it was when it came to doing incremental copies, which was done through powershell. We split it up on 4 different servers with about 80 sites per server. Sometimes it could complete them all in 2 hours, sometimes it took 8 hours for one server, sometimes longer. During the weekend of the actual move it took well over 12 hours to complete which caused me a bit of unnecessary stress.
OneDrive : Since we already had a pretty nice “masterlist” of users that we would be migrating it was pretty easy to setup a CSV file to map “Old OneDrive -> New OneDrive” that we then used ShareGate to copy. That went pretty nicely although there were some instances of data not being copied over so we had to sort that our after the switch and people were missing a few files. Other than that the issue was the same as above – it was very unpredictable and I had to mess around with the queues on the weekend of the switch. We had one incident of a users OneDrive being almost empty but looking back at the old OneDrive is was empty too. So our theory there is that his OneDrive client must’ve been paused so we had to send that computer to the lab for data recovery – but that’s not ShareGates fault one bit!
Exchange : Oh joy! I was in charge of the Nordics business moving from on-prem to Online 3 years ago so I wasn’t looking forward to another move at all. After doing a quick check around for what tool to use (with our extremely limited budget – our company had gone bankrupt and we were still getting back on our feets!). It ended up being CodeTwo which was by far the cheapest alternative. But as the saying goes “you get what you paid for” and in this instance we paid for a software to move data from Mailbox A in Tenant X to Mailbox A in Tenant Y. And it did that job without much of an issue. There were still a lot of things to sort out around the move (like transport rules, conference rooms) but the big issue was just moving all the data. The biggest issue I had with that software was that they didn’t have a CSV import function when moving tenant->tenant! When moving on-prem -> tenant that wasn’t an issue, but tenant -> tenant, well the only way to enter a mailbox was to actually manually enter a mailbox! So we spent days entering 5,500 mailboxes and matching them with their new mailbox. A simply CSV import would’ve saved us days of work on this. My next issue with the software was when we were up to about 800 mailboxes per server on 7 servers, that really slowed the UI down. At the end it was so slow that when you started a queue for a incremental copy the UI would stop responding and you didn’t even know it was working until it was done and it just popped alive again.
Teams: Now Teams was the most interesting bit. Because Teams is based on so many technologies it was difficult to do a proper Teams migration. No matter how far we looked we just couldn’t find a tool that would migrate Teams with the channel/chats that also took the entire underlying SharePoint site! If you had other document libraries or data on the SharePoint site, then that was lost if you migrated the Team. But if you migrated the SharePoint site you will have lost the data in Teams that wasn’t in the default document library! So we made the choice of migrating the SharePoint sites since noone should have have posted anything business critical in a chat in a channel in Teams. Fortunately ShareGate comes with the ability to recreate O365 groups so all the groups got recreated and we only had to make the ones that were Teams into Teams manually, that was it. But it was a bit of a “unexpected behaviour” for ShareGate when it came to legacy sites (that were migrated from on-prem) that now had an O365 group, it simply wouldn’t recognise them as O365 groups or O365 Group sites and created them as legacy sites in the new tenant regardless. But that was easy enough to handle afterwards.
Licenses: This was another headache but fortunately not mine! Since our old license agreement with Microsoft was tied to our old company we couldn’t use that. And since our company was brand new we had no credit score anywhere so Microsoft couldn’t just hand us 3,000 licenses and hope we’d pay. After a lot of back and forth we managed to get the licenses in place well enough to start the migration and begin copying all that data. But there was still the matter of support contract with Microsoft. There was alot of options floating around to try different support alternatives but in the end we agreed on a premiere support deal with Microsoft. Even though the paperwork got sorted and we were told on Friday January 31st that everything was done and we now had premiere support with MSFT it turns out that like a lot of things in O365, sometimes it can take a day or two for the wheels to turn and you’ll see how critical this became for us.
Additional headache: One headache we had was that we’re not only running a normal business, we’re also running an airline. And the pilots must be able to check their e-mail for any notices and warnings from the aviation authorities before takeoff. This may include stuff like “this aircraft model isn’t flight worthy so don’t fly this aircraft model” and “Iran just shot down a civilian aircraft, avoid their airspace”. Things like that is absolutely critical for the pilots to check for, so saying “e-mail will be down for a day” is completely unacceptable from that perspective. And we were supposed to retain all the e-mail domains, and a domain can only exist in one tenant at a time. So we had to figure out a way to handle this and move their accounts and e-mail domain as quickly as possible to avoid any flight delays because their e-mails isn’t working. (spoiler – their email was down for 90 minutes)
The plan: So the best plan we came up with was to start an incremental copy of all the SharePoint/OneDrive data first thing on the morning of Saturday February 1st. Then at about 18:00 CET we’d set automatic forwarding on everyone’s mailbox in the old tenant that would forward every mail to their new mailbox with the “onmicrosoft” address. That way we were guaranteed no mail would go missing in case of bad timings. Then we did an incremental copy of all mailboxes. We had done this in plenty of tests and it would only take 2 hours so we planned to start with the first most important domain for our airline at 21:00 CET, then when that was done continue with the largest domain we had (with about 800 users) and work our way through our list of about 10 domains.
The switch consisted of alof of steps since we weren’t allowed to sync an on-prem object to two tenants.
But… “no plan of operations extends with any certainty beyond the first contact with the main hostile force“.
How it played out: I woke up early on Saturday (at about 5) to start incremental copy of all the SharePoint / OneDrive data. Unfortunately Sharegate was a bit unpredictable in it’s behaviour so I had to move sites around in the queues to make it before 18:00 but make it I did. Then I ran the powershell to set the automatic forwarding and started the incremental copy of the mail. The team (4 engineers, 1 external SME/contractor and the project manager) met up at the office at about 20:00 in the evening for pizza and a last “go-no go” check for everything. And at 21:00 I started with our airline domain And by 22:30 it was all done, every user had the proper UPN, licens, login everything was good to go. And that’s when it started – the operations team in our airline said they couldn’t access their emails in the Outlook app on their phones or computers. We had ofcourse verified that it worked through the O365 portal so we knew everything worked. After troubleshooting this for about an hour we decided to log a Severity A case with Microsoft (at 23:30) and one of us would work on this case and the rest continue working with the other domains. That work with the other domains came to a halt for one of our largest domains which wasn’t removed from the ole tenant. No user had it in their UPN, no recipient used the domain, nothing. But the domain never got deleted, it was stuck in “pending”. So another severity A case to Microsoft (at about 00:30) and we proceeded with the next domain. At about 02 in the morning that domain did eventually go away by itself and we thought everything was good when our airline operations team (who’s responsibility it is to keep the planes flying, so I have the utmost respect for them and their challenges!) wanted us to do a rollback and try again at a later date. We spent about an hour with them arguing than a rollback wouldn’t solve this issue and we didn’t have time to try again next week since we had to evacuate the old tenant. Another argument was that this is a client issue and the mails are accessible through the web and we can get Microsoft to solve the client issue after. Fortunately we were able to convince them to proceed but now we’re at 03 int he morning and I had been working for 22 hours straight and I had no energy left in me so I tried sleeping for a bit. After 2 hours I woke up to cheers because now the Outlook clients in our airlines started to work so the biggest issue we had was solved and we could keep on with the remaining few domains.
At about lunch on Sunday morning we were all done with all domains and users and started to do the clean up job of on-prem systems no one knew about that had a EWS configured to the old tenant that no longer worked etc and that continued for days.
So where was Microsoft in this? As I mentioned our premier support deal with them got activated on the day before the switch. But that hadn’t replicated to all systems and instances of those systems in Microsoft so there was a big challenge even to get them to accept a SevA case from us. But we had two cases that managed to register as SevA cases with them during this switch and they weren’t helping us with either of them. The first case was regarding Outlook clients no longer being able to connect. Many blogs on many sites on the Internet says “when moving to a new tenant this may take a few hours”. In our case we were already up to hours and when creating new users we were able to connect to them immediately, but not the ones that had been switched and we didn’t see a reason why. This started to resolve itself after about 6 hours. And it wasn’t thanks to Microsoft doing anything on their side because they called me at about 5:30 on the sunday morning to say “sorry but we still haven’t been able to find any engineer to work on this case”. The other case we had with them was regarding the domain that wasn’t getting deleted. The called back on that issue also after it was resolved to ask us to verify the domain name because according to what they were seeing the domain was no longer in the old tenant so they obviously hadn’t done anything on their end in that case either.
Lessons learned: