Disclaimer: The events depicted in this story are purely factual and described to the best of my memory.

Prelude

Many years ago, in the City of London…

The Expedia empire reigns. Having conquered the world of travel.

Its two heads, Expedia and Hotels.com share the wealth in roughly equal measure, a few tens of billions each every year, for holding a tight grip and a large commission on all planes and hotels ever used in the world.

It is a cold day of winter in London, where the Hotels headquarters is located. A perfectly regular business meeting is about to take place, as every last Tuesday of month.

Chapter 1 – Free buffet

Teams and leaders will give numerous presentations about the business, products, partners, competitions and many more topics. Attendance is mandatory for all new employees and contractors.

Some talks prove to be interesting. One would learn why are there constant harassment popups like “Over 9000 people have booked this hotel in the last 24 hours.” and whether there actually were 9000 people who booked rooms.

One of the presenters gather people and improvise a usability test at some point. Think of a place you’d like to go for holidays, search it and go through the booking, as if you were intending to go.

Unbeknownst to all, one of the most trivial yet impactful bug in history is about to be discovered.

Chapter 2 – What Could Go Wrong?

I’ll spare you the full narrative of the session. Except, one insignificant detail toward the end, one person in the back of the room saying she cannot login to the site.

If a decade of experience has taught me anything, it’s that for every user who complains of an issue, there could be another million who don’t, in spite of the issue being very real and painful.

Always investigate feedback, no matter how stupid or irrelevant it could be, it could be the tree that hides the forest.

Chapter 3 – Debugging

Editor’s Note: The scene was originally happening on mobile. Screenshots were captured on desktop for illustration.

What can be so special that it doesn’t work for her? Let’s find out.

She cannot login because she forgot her password. She did the “reset password” many times and it does NOT work.

Why would this not work? Let’s have her try and reset it one more time.

Forgot Password => Reset => “Instructions have been sent to …”

The email is allegedly sent… and it is received by her Gmail.

She opens it and click the reset link and it doesn’t work.

Why does this not work? The page says “invalid or expired link, try again to be sent a new email“.

So, let’s try again. Except this time, it’s my turn.

Reset the password again. Open Gmail open, it shows that there is a new unread message again. Great!

Open it. Looking carefully, there is something unusual at the bottom of it…

The latest message is not shown upon opening. It’s showing an old message instead (in the middle of the page that was truncated for the screenshot). The newer messages are collapsed at the bottom. They have to be opened manually.

It turns out that Gmail hides new messages that look similar to previous messages.

The password reset messages are there, hidden at the bottom, each with its own unique link. We open the latest email manually and it works! The password is reset.

Debugging complete. It’s not a bug, it’s a feature.

She’s been opening the same old reset email every single time, with an obsolete link. That’s why reset didn’t work for her.

Gmail displays an old obsolete email when opening the discussion. No matter how many newer emails are received, they’re all systematically ignored and collapsed. She never noticed the hidden messages at the bottom.

Every reset message comes with a unique link, that gets invalidated when a new reset request is done, hence the errors about using an invalid link. She would never be able to reset her password without being able to view the latest email.

The collapsed messages are fairly hard to spot on desktop and almost impossible to spot on mobile. She doesn’t stand a chance.

Chapter 4 – Estimation of Impact

The password reset procedure is failing for a multi billion dollar website with hundreds of millions of customers.

Customers buy accommodation when they travel, more or less frequently. It’s reasonable to expect a sizeable user base to forget their password as often as every single time they return, especially if they travel sparingly.

The issue is impacting all users who forgot their password, use Gmail (not sure about other clients) and don’t notice the hidden messages being at the bottom (it’s really hard to spot).

Let’s estimate the impact. Incidentally, all the required metrics were distilled through today’s presentations and the data team just introduced themselves.

Assuming some percentages of some percentages of some statistics from users and sales. (read: private numbers).

The direct impact of this bug is a direct loss of revenues of $187M dollars per year, simply accounting for people who are unable to login and place any order. Existing users cannot use the service, new users can register but not return.

Then there is the indirect loss that is very hard to measure. If you were in the room, you would notice participants switch to the competition (booking.com) within a minute of frustration, especially if already familiar with it. It is brutal from a business perspective. Not sure if users might be giving up any faster or slower because of the artificial circumstances.

The full impact accounting for direct losses, plus indirect losses, plus recurring losses, plus reputation loss, plus competitors stealing our business, well, forever stealing our business because the site is unusable and lost users won’t switch back, etc… is hard to put an estimate on. It’s easily a multiple of the number above.

Chapter 5 – Fix

How to force Gmail to stop hiding similar emails.

Well, let’s make the email dissimilar. Adding the current time to the subject line ought to be enough.

The software is in Java, a verbose language. It takes 60 characters to format a date. An easy patch, easier done than said.

Conclusion

Actually, the bug is still active. I never had the opportunity to fix it, didn’t stay long.

During the hallway testing, I noticed at least 2 other people who couldn’t login and talked about it out loud. That’s a minimum of 3 out of 30 people impacted, proving with certainty that a major percentage of users were affected. Some guys actually said that there was a known issue with password reset and it was being worked on.

Long story short. It was not worked on at all. The bug is still there and it’s been there all along. I got it recently when I booked a travel.

It’s been going on for many years. It must be well over a billion dollar loss by now.

I’ve seen the email change a couple of times. Pretty sure it was rewritten from scratch, possibly more than once, yet the issue persists. I genuinely wonder if anyone ever investigated, let alone understood the root cause.

It is truly a global Gmail bug. I’ve noticed a handful of collapsed emails from various sources since then. The impact is not limited to Expedia.

It would be nice if Google could fix it.