Incident: Car Outage
Rental Cars intermittently could not be shopped for or booked.
Summary
TST suffered an outage of the Rental Car product due to a massive traffic spike overwhelming the application.
Customer impact
Both consumers and agents would have had intermittent issues shopping for and booking Rental Cars.
Timeline (ET)
- 14:43 - TST Car application begins receiving traffic, quickly spiking to more than 100x a normal daily peak.
- 14:47 - TST Car application replicas become fully saturated, alerts triggered.
- 15:04 - Manual measures are taken to considerably increase Car application replica count.
Root cause
No specific source of the traffic has been identified at this time, and the investigation is continuing. A large number of IP addresses were involved, but all showed signs of being from the same source, attempting to scrape pricing data out of the Car Search.
Mitigation and resolution
- Resolution: Manual intervention to increase Car application replicas and resources beyond its standard limit.
Prevention
- Additional replica scaling allowed for Car Application while the investigation is ongoing
- Investigating the source of queries and options to block the traffic.