Back
Scarcity test
Context:
As part of the strategic goals for the booking funnel, analysis showed a 75% fallout rate, with only 25% of users progressing from the Book & Pay stage to Passenger Details.
To address this, I defined several Strategic Bets and corresponding Promise to Market features, a number of which focused on the strategic use of scarcity to encourage progression.
Hypothesis
By introducing an honest and transparent form of scarcity during the Book & Pay stage, we can incentivise users to secure their booking sooner, thereby increasing conversion to the next stage
Hypothesis
We believe there are three appropriate ways to introduce scarcity on the Book & Pay page. In line with our values of delivering an exceptional, open, and truthful user experience, all scarcity signals must be accurate and genuine:
- Inform users when only a few passenger seats remain
- Inform users when only a few rooms remain
- Inform users that they have 20 minutes to secure their quote
V0 Control
- No scarcity messaging
- 25% of Visits.
V1 Seats scarcity
- API-driven message showing remaining passenger seats
- 25% of Visits
V2 Room scarcity
- API-driven message showing remaining rooms
- 25% of visits
V3 Quote time scarcity
- Countdown timer (no API required)
- 25% of visits
QA
During QA, we identified an issue with the Seat Scarcity variant:
- In only 7 out of 55 cases were there fewer than 20 seats remaining
- In 43 cases (75%+), there were plenty of seats available
- As a result, scarcity messaging would rarely be shown
Based on this, we decided to remove the Seat Scarcity variant and redistribute traffic to the more reliable variants that consistently displayed scarcity signals.
Experiment
Time range: 17:00, 19th Dec 2023 - 17:00 5th Jan 2024
Variants tested:
- Control (no scarcity)
- Room Scarcity
- Countdown Timer
Pages:
- Book & Pay page
- Passenger Details page
Note: The Book & Pay page precedes the Passenger Details page in the booking flow.
Measures of Success
The primary success metric was overall conversion (Orders / Visits), split by variant.
To assess statistical significance, I proposed using the ratio of Passenger Details page views out of Book & Pay page views, analysed via a p-test or chi-square test, to determine a valid winner.
Initial Results (04 Jan 2024)
- Countdown variant showed a statistically insignificant lift in Orders, with 38% confidence
- Countdown variant showed an 18% lift in Visits, with 83% confidence
- A minimum of 95% confidence was required for statistical significance
Due to the low confidence level, the results were considered inconclusive, and no rollout decision was made.
Countdown A/B Test Update (05 Feb 2024)
Between 24th January and noon on 5th February 2024, the Countdown variant demonstrated:
- A 15% uplift in Unique Visitors progressing from Book & Pay to the Confirmation page compared to Control
- Confidence level of approximately 85%, meaning the result is directionally positive but not yet statistically significant
Further data would be required to reach the 95% confidence threshold and confirm the outcome conclusively.
Due to the low confidence level, the results were considered inconclusive, and no rollout decision was made.
