We have mitigated the issue and all emails were processed and are processing now normally.
A few cases identified with undelivered emails are being communicated directly to the customers.
A bug in the system's code hindered proper distributed processing across multiple nodes. Consequently, when one node became overloaded, it triggered a chain reaction, overloading other nodes and causing widespread processing delays.
Posted Feb 29, 2024 - 14:58 UTC
Update
Our engineering team is currently deploying a solution to address the identified issue. The problem was traced to a scenario where network instability in a single node of our infrastructure, due to interdependencies among nodes, could potentially lead to simultaneous failures across the system. We have now eliminated this dependency, thereby preventing the cascading effect we were experiencing. Our team remains vigilant and continues to actively monitor the situation.
Posted Feb 29, 2024 - 14:24 UTC
Update
We are continuing to monitor for any further issues.
Posted Feb 29, 2024 - 10:23 UTC
Monitoring
We are continuously monitoring the situation, and as of now, all processes and deliveries are functioning correctly. Following the recent network issue, where mxHero was temporarily unavailable to some external email servers such as Office365 and GApps, there is a possibility of delayed emails being sent to our server in the next few hours. This potential delay may account for any current discrepancies in the delivery timeline. We appreciate your understanding and assure you that we are actively addressing and resolving any issues that may arise during this period.
Posted Feb 29, 2024 - 10:22 UTC
Update
Following our recent network recovery, it appears that we are now receiving emails from external servers that were previously delayed. These emails are currently being sent to us for delivery. It is possible that some delays may occur as a result of processing these incoming emails. We are actively investigating the situation to gather more details and ensure a comprehensive understanding of the current status. We appreciate your patience and understanding as we work to address any potential impact on the delivery process.
Posted Feb 29, 2024 - 08:52 UTC
Investigating
We have received notifications from clients indicating that delayed emails are still being received. Our ongoing investigation suggests that these emails may be part of the recovery process for messages that were temporarily rejected outside our network. It appears that these delayed emails are now being routed to mxHero for delivery. We are actively investigating this situation to confirm the source and nature of these delayed emails. We appreciate your patience as we work to gather more details and provide further insights into the current status.
We have indications that this issue have started at Feb 28, 1:34 AM UTC.