Disclaimer: The events depicted in this story are purely factual and described to the best of my memory.
Many years ago, in the City of London…
The Expedia empire reigns. Having conquered the world of travel.
Its two heads, Expedia and Hotels.com share the wealth in roughly equal measure, a few tens of billions each every year, for holding a tight grip and a large commission on all planes and hotels ever used in the world.
It is a cold day of winter in London, where the Hotels headquarters is located. A perfectly regular business meeting is about to take place, as every last Tuesday of month.
Chapter 1 – Free buffet
Teams and leaders will give numerous presentations about the business, products, partners, competitions and many more topics. Attendance is mandatory for all new employees and contractors.
Some talks prove to be interesting. One would learn why are there constant harassment popups like “Over 9000 people have booked this hotel in the last 24 hours.” and whether there actually were 9000 people who booked rooms.
One of the presenters gather people and improvise a usability test at some point. Think of a place you’d like to go for holidays, search it and go through the booking, as if you were intending to go.
Unbeknownst to all, one of the most trivial yet impactful bug in history is about to be discovered.
Chapter 2 – What Could Go Wrong?
I’ll spare you the full narrative of the session. Except, one insignificant detail toward the end, one person in the back of the room saying she cannot login to the site.
If a decade of experience has taught me anything, it’s that for every user who complains of an issue, there could be another million who don’t, in spite of the issue being very real and painful.
Always investigate feedback, no matter how stupid or irrelevant it could be, it could be the tree that hides the forest.
Chapter 3 – Debugging
Editor’s Note: The scene was originally happening on mobile. Screenshots were captured on desktop for illustration.
What can be so special that it doesn’t work for her? Let’s find out.
She cannot login because she forgot her password. She did the “reset password” many times and it does NOT work.
Why would this not work? Let’s have her try and reset it one more time.
Forgot Password => Reset => “Instructions have been sent to …”
The email is allegedly sent… and it is received by her Gmail.
She opens it and click the reset link and it doesn’t work.
Why does this not work? The page says “invalid or expired link, try again to be sent a new email“.
So, let’s try again. Except this time, it’s my turn.
Reset the password again. Open Gmail open, it shows that there is a new unread message again. Great!
Open it. Looking carefully, there is something unusual at the bottom of it…
The latest message is not shown upon opening. It’s showing an old message instead (in the middle of the page that was truncated for the screenshot). The newer messages are collapsed at the bottom. They have to be opened manually.
It turns out that Gmail hides new messages that look similar to previous messages.
The password reset messages are there, hidden at the bottom, each with its own unique link. We open the latest email manually and it works! The password is reset.
Debugging complete. It’s not a bug, it’s a feature.
She’s been opening the same old reset email every single time, with an obsolete link. That’s why reset didn’t work for her.
Gmail displays an old obsolete email when opening the discussion. No matter how many newer emails are received, they’re all systematically ignored and collapsed. She never noticed the hidden messages at the bottom.
Every reset message comes with a unique link, that gets invalidated when a new reset request is done, hence the errors about using an invalid link. She would never be able to reset her password without being able to view the latest email.
The collapsed messages are fairly hard to spot on desktop and almost impossible to spot on mobile. She doesn’t stand a chance.
Chapter 4 – Estimation of Impact
The password reset procedure is failing for a multi billion dollar website with hundreds of millions of customers.
Customers buy accommodation when they travel, more or less frequently. It’s reasonable to expect a sizeable user base to forget their password as often as every single time they return, especially if they travel sparingly.
The issue is impacting all users who forgot their password, use Gmail (not sure about other clients) and don’t notice the hidden messages being at the bottom (it’s really hard to spot).
Let’s estimate the impact. Incidentally, all the required metrics were distilled through today’s presentations and the data team just introduced themselves.
Assuming some percentages of some percentages of some statistics from users and sales. (read: private numbers).
The direct impact of this bug is a direct loss of revenues of $187M dollars per year, simply accounting for people who are unable to login and place any order. Existing users cannot use the service, new users can register but not return.
Then there is the indirect loss that is very hard to measure. If you were in the room, you would notice participants switch to the competition (booking.com) within a minute of frustration, especially if already familiar with it. It is brutal from a business perspective. Not sure if users might be giving up any faster or slower because of the artificial circumstances.
The full impact accounting for direct losses, plus indirect losses, plus recurring losses, plus reputation loss, plus competitors stealing our business, well, forever stealing our business because the site is unusable and lost users won’t switch back, etc… is hard to put an estimate on. It’s easily a multiple of the number above.
Chapter 5 – Fix
How to force Gmail to stop hiding similar emails.
Well, let’s make the email dissimilar. Adding the current time to the subject line ought to be enough.
The software is in Java, a verbose language. It takes 60 characters to format a date. An easy patch, easier done than said.
Actually, the bug is still active. I never had the opportunity to fix it, didn’t stay long.
During the hallway testing, I noticed at least 2 other people who couldn’t login and talked about it out loud. That’s a minimum of 3 out of 30 people impacted, proving with certainty that a major percentage of users were affected. Some guys actually said that there was a known issue with password reset and it was being worked on.
Long story short. It was not worked on at all. The bug is still there and it’s been there all along. I got it recently when I booked a travel.
It’s been going on for many years. It must be well over a billion dollar loss by now.
I’ve seen the email change a couple of times. Pretty sure it was rewritten from scratch, possibly more than once, yet the issue persists. I genuinely wonder if anyone ever investigated, let alone understood the root cause.
It is truly a global Gmail bug. I’ve noticed a handful of collapsed emails from various sources since then. The impact is not limited to Expedia.
It would be nice if Google could fix it.
12 thoughts on “The Gmail bug that’s been stealing $187M a year from Expedia”
Gmail always puts messages in the order they were received – that’s how you read them, from first to last. The optimal behavior would be to collapse previously read emails and expand unread emails – which is exactly the behavior I observe in Gmail.
Also, the person who “fixes” (breaks) this isn’t adding value to the internet; they are shifting it. People are still booking their flights or hotels, just maybe not where Expedia et al want them to.
This same logic supplies to your comment about there being a billion dollar loss. That would imply that only Expedia is affected, when in fact they also benefit from the behavior affecting other vendors. It could, in fact, be a wash.
BTW, I’m not a Googler or related, so I have no skin in the game.
Conversation view needs to be off. Google recently allowed it in their Android Gmail client after like 10 years of campaigning.
Minor typo: “each with it’s own unique link” -> “each with its own unique link”
Fixed. Thank you for the notice.
Same error here: “It’s two heads, Expedia and Hotels.com…”
At that scale, standard practice is to measure and track account recovery funnels. The customer service department would also flag the outreach by people that were impacted by this so-called bug.
I think this is a case of assuming that people booking trips online still don’t know how to use Gmail 2019 (you are wrong; they do) and attributing that to a random dollar figure. You have zero data to support any of these imaginary scenarios.
Doing a user testing session with 30 people and having 3 of them unable to login is strong data. Having one of them repeat the procedure 3 times before my eyes to demonstrate that it’s completely broken is undeniable evidence of an UX issue. As for the rest of the numbers, that’s internal.
Don’t ignore the pain of users just because there are some who can find their way through. Don’t think that people will call support to reset their account, they don’t, the support number is itself fairly well hidden 😀
We’ve encountered this a lot and when sending emails to organizations that use Google Suite or Gmail.
When we explicitly don’t want the messages to be grouped our team has began making the subject of the email unique to the request.
e.g instead of:
Subject:”Password Reset Notification”
Subject:”Website Support Request”
Subject:”Your Password Reset Request – March 14th 10:19am”
Subject:”Website Support Request – Jimmy Davis, Failed Login”
This would be the best way of combatting this because you can bet your life Google won’t do anything about this. Mind you I have to implement this on my own sites!
Seriously? Doesn’t anyone see the problem root cause? Expedia and Hotels.com are loosing money because of their poor design of password recovery process. Not because of Gmail.
Making email titles unique is just a workaround, but not a solution.
Problem is that password reset link that is emailed to the user is re-created each and every time user clicks “Forgot password” button on the website. Even if previously sent link was not clicked, nor expired.
Indeed, Gmail has this feature that stacks unread emails into a thread, starting from oldest message, and keeps next emails collapsed. However, this should not impact user journey.
Correct user journey should be:
1) user clicks “Forgot password” button on the website
2) unique link is sent to him, that is valid for next 24h, or until it clicked.
3) if user clicks the link, he is immidiately logged into the website, that preserves his previous sessions (page, selected dates or product or whatsoever he was doing before he decided to log in); login link gets invalidated; also old password gets invalidated; user is shown a non-modal hint to set new password. He can do it at any time, even after he finishes his activity and reaches goal he had in mind when came to the website.
4) alternatively, if user attempts to click “Forgot password”, system checks whether password reset link exists, not used and not expired – in this case just sends same link again. No need to create new one.
Yes, even now, Sometimes “Gmail displays an old obsolete email when opening the discussion. No matter how many newer emails are received, they’re all systematically ignored and collapsed. So, you never noticed the hidden messages at the bottom.”
I could not call it a feature, its a hack!
Hahaha. I just stumbled over this article and boy do I know that problem.
Many years back I was writing a tool that sends out invoices for a little non-profit ISP. I tested it with my gmail account to see how the new HTML MIME thingy looked on gmail and all seemed well.
Sent another test-mail, can’t see anything. Tried again. Checked the Spam Folder. Then I looked and found the same as you did.
My solution was to ensure that the top of the mail (and it needs to be the top) or the subject is unique. I solved it by having a $random_string on top of the mail.
Worked great. Except it stopped working some time ago. Noticed it when doing some changes to the HTML template and tested on gmail again… Whoot? Grouping again?
https://gsuiteupdates.googleblog.com/2019/03/threading-changes-in-gmail-conversation-view.html was the announcement, that things changed again. In other words, Google _FINALLY_ fixed the “bug” or better: They changed their behavior.
The easiest solution now is to send Mail with a a References: Header that contains it’s own Message-ID. Problem solved. No more grouping.