Incident Report

Background

Gmail's delivery system is responsible for routing, classifying, and enhancing emails. Emails are placed in a queue and are processed by a set of backend services (e.g. spam classification). If everything goes well, the mail delivery system sends the emails. However, when a failure occurs, the system keeps the emails in the queue and retries delivering them later.

Summary

On 10 December 2022 at 05:29 PT, some Gmail users observed email delivery delays of 60 minutes on average, as well as email bounce backs (i.e. senders received a delivery failure notification).

Root Cause

An email campaign from a single domain triggered a latent bug in one of the dependencies of the email delivery service, which crashed and then became overloaded. These emails contained dynamic mail content triggering service instance failures. Due to this failure and a latent bug validation feature configuration, 80% of tasks crashed, which overloaded the remaining tasks.

Remediation and Prevention

On 10 December 2022 at 05:29 PT, Google Engineering was alerted to an issue via internal monitoring that Gmail delivery was impacted. Google Engineering began investigating and identified the following actions to mitigate the issue:

• Increase the memory of the affected dependency.
• Disable the feature that triggered the issue.

The backlog of emails were then able to to be processed without any additional issues.

Google is committed preventing a repeat of this issue in the future and is completing the following actions:

• Audit Gmail delivery dependencies.
• Optimize the email validation feature configuration.
• Implement automatic threshold protection against backlog overload.

Mini-Incident Report

We apologize for the inconvenience this service disruption may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support or to Google Workspace Support using help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Incident Start: 10 December 2022 05:29

Incident End: 10 December 2022 09:52

Duration: 4 hours and 23 minutes

Affected Services and Features: Gmail

Regions/Zones: Global

Description:

Gmail users experienced delays in email delivery and bounce back of emails globally for a period of 4 hours and 23 minutes. From preliminary analysis, the root cause of the issue was a latent bug in the message delivery logic.

Customer Impact:

Customers may have experienced delays in delivery or bounce back of their emails during the duration of the impact.

The issue with gmail is now fully mitigated. All the backlog of undelivered messages have been cleared and mail services are back to normal.

Thanks for your patience while we resolve this issue.

Mitigation is currently underway and email delivery is no longer failing. However, the Google Engineering team is now working through the backlog of undelivered messages and expect all messages to be delivered in the next few hours.

We will provide an update by Saturday, 2022-12-10 10:30 US/Pacific with current details.

Mitigation is currently underway and email delivery is no longer failing. However, the Google Engineering team is now working through the backlog of undelivered messages and expect all messages to be delivered in the next few hours.

We will provide more information by Saturday, 2022-12-10 10:00 PT

We are experiencing an issue with Gmail. Users may experience delays in email delivery. Our engineering team continues to investigate the issue. We will provide an update by Saturday, 2022-12-10 08:30 US/Pacific with current details.