Junior Developer's Mistake Causes WhatsApp Meta AI Downtime

Date: September 18, 2024

The WhatsApp Meta AI chatbot revealed every detail of the downtime, from reason to resolution, when asked by a user after it came back online.

WhatsApp Meta AI experienced an outage for over 30 minutes on September 17, 2024, due to multiple technical issues, one of which was triggered by its junior developer. The exclusive news and every detail of the downtime was revealed when a WhatsApp user asked Meta AI about it. 

The user carried the conversation in Hindi, which may have caused Meta AI to reveal more than it should have, as the AI chatbot is no longer revealing the details to other users enquiring in English. Statusgator, an online tool continuously monitoring Meta Service Health, captured the downtime from 6 PM to 6:30 PM, confirming that Meta AI’s revelations were not hypothetical.

The user who regularly interacts with WhatsApp Meta AI began receiving an error response from the AI chatbot, which said, “There was a problem generating a response. Please try again later.” After the chatbot came back online and began responding to the user’s queries, he asked why it was down, and the AI chatbot revealed multiple details about the matter.

WhatsApp Meta AI shared the technical reasons for its outage:

• Server overload: A network congestion was triggered by too many users at the same time.
• Software bug: A bug appeared on the server while a new feature was being implemented.
• Database corruption: A database table inconsistency caused the corruption.

Meta AI also shared specific details that escalated the issue:

• Cache Memory Leak: Caused by over-accumulation of data in the cache memory.
• API Rate Limiting: One third-party API’s rate limit exceeded the threshold.

Meta AI also conducted a root cause analysis, which revealed that these events were triggered by a mistake by a junior developer. It also explained that lack of timely review and resolution guidance from a senior developer led to further escalation. The system's complexity created a domino effect, converting a minor mistake into a major reason for the complete outage.

However, the user did not stop there and asked for more details about the overall allocation to resolve the issue. Surprisingly, Meta AI shared a detailed response to this query, as shown below.

According to the response, over 10,000 hours of CPU time was utilized, and 20+ specialized engineers were allocated to resolve the issue across the network infrastructure of multiple data centers. Meta AI also shared a detailed rough estimate of the resolution cost, which was nearly $13,000, but clarified that the actual costs would depend on the internal processes and infrastructure.

Meta AI told the user that hundreds of thousands of users worldwide were impacted by the outage, but the issue has been resolved, and it is ready to serve him with full capability.

The user tried to ask for the name of the human resource who made the mistake, but Meta AI was smart enough not to reveal any confidential information about its internal team. This interaction between a general user and WhatsApp Meta AI demonstrates how helpful the AI chatbots can be while strictly adhering to its data privacy policies and the safety of its makers.