[Mail] Hosted Exchange (SE) — incident resolved

Investigation in progress. Following last night's Exchange SE maintenance, we received multiple user reports from around 08:00 CEST this morning. Teams are verifying cluster and service health (DAG, transport, client access). We will publish the next update as soon as we have more findings.

Update — investigation continues. Our teams are still working on the Exchange SE instability reported this morning. We are actively pursuing stabilisation of the affected hosts.

Overnight we applied technical changes on several servers, including partial rollbacks of last night’s planned maintenance — intermittent issues persist for now.

Early analysis points to host-level instability (system services / hardware re-enumeration after reboot), not a defect in Exchange itself. The behaviour remains sporadic; we have not yet reproduced or isolated the exact trigger of the crashes.

Customer reassurance: when instability is detected, Exchange databases are dismounted and protected automatically — your data is safeguarded while services recover. We will keep you informed as stabilisation progresses.

Update. We have fully rolled back the changes from Saturday evening’s planned maintenance on the affected Exchange SE hosts. This has not resolved the issue: intermittent disturbances continue, and we have not yet identified the exact cause nor implemented a lasting fix.

Our teams remain actively engaged — ongoing checks, tests and troubleshooting on the underlying host-level instability. The behaviour remains sporadic and not reproducible on demand at this stage.

Exchange is impacted as a consequence of this host instability, without an Exchange-specific fault identified. When instability occurs, databases are automatically protected (dismounted) to safeguard your data. We are still working on it — further update to follow.

Update. Verification and troubleshooting on the Exchange SE hosts will resume from 21:00 CEST (19:00 UTC) this evening. Scope: host health checks, storage/access-path validation, DAG/transport/service state review, and log correlation from prior instability events.

Incident remains open — root cause not yet confirmed. Further update after this verification cycle.

Update. Over the past several hours we have applied multiple corrective measures on the affected Exchange SE hosts. Unfortunately, one Windows Server 2025 Core node crashed again — the underlying host instability persists.

In parallel, we are building a new clean environment: 12 new Exchange SE servers are currently being prepared and staged for migration. We will keep you informed as this progresses.

Current situation: the existing platform remains operational, but random intermittent disturbances may still occur at any time until stabilisation or cutover. Databases continue to be automatically protected when required.

Update — work in progress. Our teams continue active work on the Exchange SE incident: ongoing host troubleshooting, monitoring of the existing platform, and preparation of the new environment (12 servers). No definitive fix yet — intermittent disturbances may still occur. We will keep you informed.

Update — database incident (overnight). Following the ongoing host instability, several Exchange databases were impacted during the night and required manual recovery/repair after unplanned dismounts.

Our engineers have started integrity repair procedures on affected stores (e.g. DBSE04-003 — eseutil /p in progress). These operations are lengthy; completion cannot be rushed without risking further data loss. Work is still in progress.

The existing platform remains partially available where possible; service impact may vary by mailbox/database. Parallel effort continues on the new clean environment (12 Exchange SE servers).

We are deeply sorry for the disruption. Our teams have been working on this instability continuously for several days, including overnight, to restore service as safely as possible. We will keep you informed.

Update — DBSE recovery & storage layer. Multiple repair/replay cycles have been executed on affected DBSE stores (soft recovery / eseutil paths as applicable). At this stage, approximately 75–80% of databases have been brought back online successfully; remaining stores are still under active recovery.

Storage pressure: backend SAN I/O is extremely elevated across the Exchange footprint, extending repair windows. In parallel, a physical disk failure was detected on SAN12 (GEN01); impacted ZFS pools are rebuilding, which is materially slowing both ongoing DBSE repairs and general host/storage responsiveness.

Incident remains open. We continue DB-level recovery while monitoring pool resilver/rebuild progress. Next update once the next recovery milestone is reached.

Update — storage & DBSE recovery. SAN12 (GEN01): ZFS resilver/rebuild completed — pool health back within expected parameters; we continue enhanced monitoring of physical disks and host I/O.

On the Exchange side, we successfully restored one additional DBSE database, mounted the store, and re-enabled mailboxes hosted on that database. Next database recovery cycle is starting — executed step-by-step with conservative checks between each phase.

Overall recovery remains ~75–80% of stores online; balance in queue. Incident open — cautious progression while watching hosts, SAN paths, and disk telemetry.

Update. Our teams continue active work on DBSE database restoration — recovery cycles proceed step-by-step with ongoing monitoring of hosts, SAN paths, and disk health. Incident remains open. Thank you for your patience.

Update — still in progress. While repairing DBSE04-006, the host suffered another hard Windows crash, interrupting the recovery window for that store.

To reduce risk to databases already online, we are relocating the EDB for this recovery to another member in the cluster and will resume repair/mount operations from that node under close host & storage monitoring.

Incident remains open. DBSE restoration continues on remaining stores — step-by-step. Thank you for your patience.

Update — day 2 of intensive recovery. After two days of sustained DBSE restoration — in parallel with recurring Windows host instability and very high SAN I/O — all other databases have been brought back online.

DBSE04-006 remains the outstanding case: integrity checks indicate corruption on the primary copy and, at this stage, on both additional copies as well. Our engineers have not stopped — we are continuing recovery attempts and still aim to salvage at least one of the three copies, using every safe option available.

This is a demanding situation for our teams; we remain fully engaged on your behalf. Incident open. Thank you for your patience — next update as soon as we have material progress.

Update — major recovery milestone. Thanks to a valid ZFS snapshot, our teams recovered a clean copy of DBSE04-006; all Exchange databases are now mounted and online.

We cannot yet confirm that Windows Server hosts will not experience further crashes — enhanced monitoring remains in place. The Saturday-evening maintenance/update appears to have severely destabilised the affected Windows Server 2025 Core footprint; we regret the impact this has caused.

Forward plan: 9 new Exchange SE servers are being prepared; we will provision fresh, healthy database copies and migrate mailboxes one-by-one with meticulous, cautious cutover steps. Work continues — incident remains open until stabilisation and migration milestones are met. Thank you for your patience.

Update. Our teams continue active work on the new Exchange SE environment — provisioning, configuration validation, and pre-migration testing are in progress on the 9 replacement servers. All databases remain online on the existing platform; enhanced host/storage monitoring is maintained.

Mailbox one-by-one migration will follow once the new footprint passes our readiness checks. Incident remains open. Thank you for your patience.

Update. Our teams worked through the entire weekend on the Exchange SE stabilisation programme. The new clean environment is now ready for production use.

Mailbox migrations are in progress — cutovers proceed one-by-one, with pre/post checks on each batch. The legacy platform remains monitored; enhanced host & storage surveillance continues in parallel.

Incident remains open until migration milestones are completed. Thank you for your patience — next update as batches are validated.

Update — stabilisation milestone. The Exchange SE platform has been stable since today (10 June 2026) following recovery and host work over the past days.

Mailbox migrations to the new clean environment continue — cutovers proceed one-by-one with pre/post validation. Legacy hosts remain under enhanced monitoring during the transition.

Update — migration complete. All mailbox transfers to the new Exchange SE environment were completed today (30 June 2026).

The new footprint is in production; legacy platform removed from the active mail path. Routine monitoring and post-migration validation continue.

CLOSED — service restored. Following the instability after the Saturday-evening maintenance, recovery and migration work is complete.

Platform stabilised since 10 June 2026. Mailbox transfers to the new clean environment completed on 30 June 2026. All DBSE stores are online; legacy hosts have been decommissioned from the active mail path.

We continue routine monitoring on the new footprint. Thank you for your patience throughout this extended incident.