The 187 Million Dollars Gmail Bug


Discovery

The scene takes place at a multi-billion dollars company, running some of the most well known e-commerce sites in the world.

It’s a cold day of winter, as every last Tuesday of the month, it’s the on-boarding for new employees.

Teams and leaders are giving various presentations about the business, the teams, their products, the partners, the competitions and many more topics. Some talks are actually quite interesting.

One of the last presenter requests participants to use the service, find something we want and go through the purchase as if we were intending to buy it.

The point of this exercise is obviously to get some opinions and feedbacks…

I’ll spare you the full narrative of the session. Except, one insignificant details toward the end, one person in the back of the room saying she cannot login to the site.

If a decade of experience has taught me anything, it’s that for every user who complains, there can be another million who experience the same issue, making your product plain unusable for all of them. Always follow on feedback, no matter how stupid or irrelevant they seem, they can be the tree that hide the forest.

What can be so special that it doesn’t work with her? Let’s find out.

Debugging

She cannot login, because she forgot her password. She did the “reset password” many times and it does NOT work.

Why would this not work? Let’s have her try and reset it one more time.

Forgot Password => Reset => “Instructions have been sent to …”

The email is allegedly sent… and it is received by her gmail.

screenshot gmail subject
New message in the inbox!

She opens it and click the reset link and it doesn’t work.

Why does this not work? The page says “invalid or expired link, try again to be sent a new email“.

So, let’s click again. Except this time, it’s my turn.

We open the gmail, it shows that there is a new unread message again. Great!

Looking carefully, there is something unusual on the bottom of the email conversation…

screenshot gmail hidden
New messages are not shown. They need to be manually expanded. (Desktop Client View)

The email is here. It’s just hidden!

It turns out that gmail is hiding new messages when they look too similar to the previous one.

All the password reset messages are there, hidden on the bottom, each with it’s own unique link. We try the latest one and it does work! Password resetted.

it's not a bug, it's a feature
Debugging complete

She’s been reading the same old email every single time. She never noticed the hidden messages on the bottom (note that they are quite difficult to spot on mobile).

Reset links are unique and invalidated when a new one is made, hence the errors about invalid links.

Let’s remember of that as a lesson in user accessibility. What may be noticed by one user may be missed by another.

Impact

This password reset procedure is for a billion dollar e-commerce site, used by millions of customers, in most of the countries in the world.

I should say that users buy things sparingly. They buy something once, go on with their lives, maybe come back one day far away. It’s reasonable to expect a sizeable user base to forget their password as often as “every single time“.

The issue is impacting all users who forgot their password (read: already registered on the site), use gmail (not sure about other clients) and don’t notice the hidden messages on the bottom.

 

Let’s see with the data team what’s the impact of this. Incidentally, they just introduced themselves during one of the presentations.

Assuming some percentages of some percentages of some statistics of our sales. (Sorry, private numbers ^^).

The direct impact of this bug is a direct loss of revenues of $187M dollars per year, simply accounting for people who are unable to login and place any order.

 

I should add that, as shown time and again during demonstrations, participants will switch to competitors within a few minutes of frustration, especially the ones who are already familiar with them. I don’t know if they are giving up faster or slower than regular users, either way that’s food for thought.

The impact accounting for direct losses, plus indirect losses, plus recurring losses, plus reputation loss, plus word of mouth loss, plus competitors stealing our business, well, recurrently stealing our business, etc, is hard to put an estimate on. It should be within some multipliers.

 

Last but not least. This is on a single business and we’ve got more than just one.

What else is impacted has yet to be determined. That will vary by how the password resets are done.

Fix

We need to force gmail to NOT collapse emails.

Having different subjects should do the trick.

Let’s append the current time to the subject. That is minor change in the line of code that generates the subject.

i dont often write code but when i do its 3m dollars per chara
What to say next time you’re asked to code in an interview

Conclusion

All the internet is potentially affected. You should check whether your business is.

pic - if google fixes google the internet could go up by 1percent of 1percent
There is a percentage of truth behind every meme.

 

Advertisements

One thought on “The 187 Million Dollars Gmail Bug

  1. This is also why, for many sites, it makes so much sense to allow authentication via OAuth via sites likely to be popular amongst your users (Google, Facebook, Twitter, GitHub, whatever). Most such sites will offer two-factor authentication, increasing security if the user uses it, and the user’s much less likely to forget or lose the password of a site he or she uses regularly. Even for people who use password managers, an OAuth login is often easier.

    Like

Post a Comment

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s