Rackspace hosted Exchange suffered a catastrophic failure starting December 2, 2022 and is still continuous since 12:37 AM December fourth. Initially described as connectivity and login issues, the guidance was ultimately upgraded to reveal that they were dealing with a security event.
Rackspace Hosted Exchange Issues
The Rackspace system went down in the morning hours of December 2, 2022. At first there was no word from Rackspace about what the issue was, much less an ETA of when it would be resolved.
Consumers on Buy Twitter Verification reported that Rackspace was not responding to support e-mails.
This has been rather the day with #Rackspace. Every hosted exchange customer has actually been down for 14 hours or so. Support isn’t reading/responding to tickets. Updates are unhelpful.
I am concerned now that they succumbed to something bad like the ProxyNotShell PoC hack. https://t.co/jchKsAO3Z7
— Joe Sinkwitz (@CygnusSEO) December 2, 2022
A Rackspace customer privately messaged me over social networks on Friday to relate their experience:
“All hosted Exchange customers down over the past 16 hours.
Not exactly sure how many companies that is, but it’s considerable.
They’re serving a 554 long delay bounce so people emailing in aren’t knowledgeable about the bounce for numerous hours.”
The official Rackspace status page offered a running update of the failure however the initial posts had no details besides there was a blackout and it was being investigated.
The very first authorities update was on December 2nd at 2:49 AM:
“We are investigating an issue that is impacting our Hosted Exchange environments. More details will be published as they appear.”
Thirteen minutes later on Rackspace began calling it a “connection problem.”
“We are examining reports of connectivity issues to our Exchange environments.
Users might experience an error upon accessing the Outlook Web App (Webmail) and syncing their e-mail client(s).”
By 6:36 AM the Rackspace updates explained the ongoing problem as “connection and login concerns” then later on that afternoon at 1:54 PM Rackspace revealed they were still in the “investigation phase” of the blackout, still trying to determine what went wrong.
And they were still calling it “connection and login problems” in their Cloud Workplace environments at 4:51 PM that afternoon.
Rackspace Recommends Migrating to Microsoft 365
Four hours later Rackspace described the situation as a “significant failure”and began providing their customers free Microsoft Exchange Plan 1 licenses on Microsoft 365 as a workaround until they understood the problem and might bring the system back online.
The main assistance stated:
“We experienced a substantial failure in our Hosted Exchange environment. We proactively shut down the environment to avoid any additional concerns while we continue work to bring back service. As we continue to overcome the root cause of the problem, we have an alternate option that will re-activate your capability to send and receive e-mails.
At no charge to you, we will be offering you access to Microsoft Exchange Strategy 1 licenses on Microsoft 365 up until more notification.”
Rackspace Hosted Exchange Security Event
It was not till almost 24 hours later on at 1:57 AM on December 3rd that Rackspace officially revealed that their hosted Exchange service was struggling with a security event.
The announcement further revealed that the Rackspace specialists had actually powered down and detached the Exchange environment.
“After further analysis, we have identified that this is a security event.
The recognized impact is separated to a part of our Hosted Exchange platform. We are taking essential actions to examine and secure our environments.”
Twelve hours later on that afternoon they upgraded the status page with more information that their security team and outdoors specialists were still dealing with resolving the interruption.
Was Rackspace Service Impacted by a Vulnerability?
Rackspace has actually not released information of the security occasion.
A security occasion typically includes a vulnerability and there are two serious vulnerabilities currently in the wile that were patched in November 2022.
These are the 2 most existing vulnerabilities:
Microsoft Exchange Server Server-Side Request Forgery (SSRF) Vulnerability
A Server Side Request Forgery (SSRF) attack permits a hacker to check out and change data on the server.
Microsoft Exchange Server Remote Code Execution Vulnerability
A Remote Code Execution Vulnerability is one in which an aggressor has the ability to run destructive code on a server.
An advisory published in October 2022 described the effect of the vulnerabilities:
“A confirmed remote attacker can perform SSRF attacks to intensify advantages and execute arbtirary PowerShell code on vulnerable Microsoft Exchange servers.
As the attack is targeted versus Microsoft Exchange Mail box server, the attacker can possibly access to other resources through lateral movement into Exchange and Active Directory environments.”
The Rackspace outage updates have not suggested what the specific issue was, just that it was a security occurrence.
The most present status upgrade as of December fourth mentioned that the service is still down and customers are encouraged to move to the Microsoft 365 service.
Rackspace published the following on December 4, 2022 at 12:37 AM:
“We continue to make progress in resolving the event. The schedule of your service and security of your information is of high significance.
We have actually devoted substantial internal resources and engaged first-rate external knowledge in our efforts to lessen unfavorable effects to clients.”
It’s possible that the above kept in mind vulnerabilities are related to the security event impacting the Rackspace Hosted Exchange service.
There has actually been no statement of whether client info has actually been compromised. This occasion is still continuous.
Featured image by Best SMM Panel/Orn Rin