The Challenge
Over the years, Kalio has seen an increasing amount of ecommerce website traffic directly attributed to bots. While initially manageable, it has started to become a serious performance issue for many sites. Case in point: our team recently received an alert indicating that a server was experiencing performance degradation. Upon investigation, it was discovered that a substantial volume of traffic originated from Google bots accessing filtered pages. This unexpected behavior posed a significant threat to the customer’s site speed and ultimately, uptime.
The Solution
Swift Response and Collaborative Solution
Recognizing the urgency of the situation, a Platform developer, a Cloud Management Services admin, and a Professional Services consultant worked together to address the issue. Through collaborative efforts with Kalio’s Solution Architect, a two-pronged approach was devised:
- Immediate Mitigation: A new Web Application Firewall (WAF) rule was implemented to temporarily block the excessive bot traffic. This immediate action provided a quick fix to alleviate the server load and prevent further performance degradation.
- Long-Term Solution: A more permanent solution involved modifying each website’s structure to effectively hide the filtered links. By making these pages inaccessible to bots, we could prevent future instances of excessive bot traffic.
Root Cause Analysis
Historically, Kalio had implemented measures to discourage bots from indexing these filtered pages. However, the recent introduction of new bots (likely AI bots) no longer respects these measures. These new bots are crawling each and every filtered combination leading to a dramatic increase in site traffic. This unexpected surge in bot activity resulted in slower response times and, if left unattended, could have led to service disruptions.
Kalio Observations and Future Implications
In summary, this case study offers valuable insights into the potential challenges posed by bot traffic. This particular incident was identified by Kalio and highlighted the importance of our proactive monitoring and quick triage management. The key elements to the resolution of this issue included:
- Continuous Monitoring: Regular monitoring of server performance (i.e., 90th percentile) and traffic patterns is essential to identifying and addressing potential issues early on.
- Adaptability: Staying updated on the latest technological trends and website traffic patterns allows us to adjust our strategies proactively.
- Collaboration: Effective teamwork and communication are crucial for swiftly and efficiently responding to unexpected challenges.
- Customer Focus: Kalio’s ability to focus on solving the customer’s problem regardless of where the issue resides is simply part of our DNA.
By adhering to our unified approach and leveraging the insights gained from this experience, Kalio can help our customers maintain high-performing and reliable websites that deliver best-in-class user experiences.
