Incident Report

Summary

On Thursday, 29 June 2023 at 05:37 US/Pacific, Google Meet group calling users experienced degraded audio quality for a duration of 7 days, 1 hour, 5 minutes. During this time, customers were unable to understand the audio.

To our Google Workspace customers who were impacted during this disruption, we sincerely apologize. This is not the level of quality and reliability we strive to offer you. We have conducted an internal investigation and are taking immediate steps to improve the resiliency of our service.

Root Cause

Google Meet leverages legacy Google Duo backend services (AKA Duo) for 1:1 calling and group calling functions. During their operations, these services determine the duration of audio packets. When the backend service detects two audio packets overlapping in duration, the second one is dropped.

The trigger for this incident was a change on those backends that overlooked an experimental configuration for Duo, causing the duration of audio packets for Duo to be determined incorrectly. This resulted in ‘random’ audio durations and audio packets that looked like they overlapped in time. Overlapping packets were dropped, and the client had to reconstruct audio from only a few packets it received.

Remediation and Prevention

Google engineers were alerted to the regression in a non production environment via internal monitoring on Saturday, 24 June. However, due to an issue in the monitoring system the alert did not trigger. Once the alerting issue was fixed, engineering began to triage the regression, however the change began rolling out to production on Thursday, 29 June at 05:37 US/Pacific. On Friday, 30 June at 09:00, Google engineers identified the root cause of the regression previously discovered in the non production environment and began creating a fix. At 14:06, user reports of audio quality issues on Google Meet group calls increased. The fix began rolling out to production on Tuesday, 04 July at 09:44 US/Pacific and the issue was escalated on Thursday, 06 July at 03:11 US/Pacific. The incident was fully mitigated on Thursday, 06 July at 06:42, once engineers fully rolled out the fix.

Google is committed to preventing a repeat of this issue in the future and is completing the following actions:

• Improve monitoring and alerting to detect audio quality failures in Google Meet and reduce mitigation time for issues of this type.
• Enhance testing in the staging environment to identify and prevent regressions before rollout progresses.

Detailed Description of Impact

From Thursday, 29 June 2023 at 05:37 to Thursday, 06 July 2023 at 06:42 US/Pacific, Google Meet group calling users experienced degraded audio quality globally.

Google Meet

Google Meet group calling users may have heard metallic noise in the background, resulting in the inability to understand the audio.

• Impact was observed across all platforms (iOS, Android, Web).
• There was no impact to customers using 1:1 calls.

Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Workspace Support using the help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Incident Start: 29 June 2023 at 05:37

Incident End: 06 July 2023 at 06:42

Duration: 7 days, 1 hour, 5 minutes

Affected Services and Features:

Google Meet - Group Calls

Regions/Zones: Global

Description:

All Google Meet group calling users experienced degraded audio quality for a duration of 7 days, 1 hour and 5 minutes. During this time, customers were unable to understand the audio.

From preliminary analysis, the root cause of the issue was an incorrect configuration change that handles parsing of audio from the source. Google engineers mitigated the issue by fixing the incorrect configuration.

Google will complete a full incident report in the following days that will provide a full root cause.

Customer Impact:

Google Meet group calling users may have heard metallic noise in the background, resulting in the inability to understand the audio. Impact was observed across all platforms (iOS, Android, Web). There was no impact to customers using 1:1 calls.

Additional Details:

Wherever possible, customers were advised to use Google Meet as an alternate workaround.