Welcome!

Big Data Journal Authors: Pat Romanski, Elizabeth White, Adrian Bridgwater, Liz McMillan, Kevin Benedict

Related Topics: SOA & WOA, Java, Linux, Web 2.0, Security, Big Data Journal

SOA & WOA: Article

Twitter’s Story: How Homegrown Load Testing Tools Can Misfire

If Twitter built their own tool to perform their own load test, why did the selfie cause their site to crash?

The selfie that changed the world, or at least Twitter, has been in the news for the past month. On March 2, 2014, the infamous Oscar selfie of Ellen and her celebrity friends broke a record of 2 million retweets before midnight the same night. That record was previously set by President Barack Obama, hugging first lady Michelle Obama after his 2012 re-election.

The selfie caused Twitter to crash for more than 20 minutes, also breaking the record for the longest crash of the social media site. Twitter was infamous for crashing in its early days (anyone remember "Fail Whale?"), so it's no wonder the social media giant worked extra hard to completely prepare their website infrastructure before going public in November 2013. This included building their own load testing tool, Iago, in 2012.

If they built their own tool to perform their own load test, why did the selfie cause their site to crash? The Oscar selfie crash is a perfect example of what companies can easily overlook. Twitter did not test their users properly and their homegrown tool clearly doesn't solve all of their problems... their servers still crash.

What is Iago and why did Twitter decide to make it? And what does that have to do with your decision to use homegrown tools versus vendor load testing tools? Don't worry, we will tell you.

Twitter's Homegrown Load Testing Tool: Iago
Iago was created in June of 2012 by Twitter's internal engineering team. According to Twitter, Iago is a load generator created to help the social media site test services before they encounter production traffic. Chris Aniszczyk, Head of Open Source at Twitter, said, "There are many load generators available in the open source and commercial software worlds, but Iago provided us with capabilities that are uniquely suited for Twitter's environment and the precise degree to which we need to test our services."

Basically their homegrown tool was completely customized for their platform alone - a very attractive aspect of developing your own tool.

The three attributes Twitter focused on in creating Iago were:

  1. High performance: Iago was designed to generate traffic in a precise and predictable way, to minimize variance between test runs and allow comparisons to be made between development iterations.
  2. Multi-protocol: Modeling a system as complex as Twitter can be difficult, but it's made easier by decomposing it into component services. Once decomposed, each piece can be tested in isolation; which requires the load generator to speak each service's protocol. Twitter has in excess of 100 such services, and Iago tests most of them using built-in support for the internal protocols Twitter uses.
  3. Extensible: Iago is designed for engineers. It assumes the person building the system will also be interested in validating performance. As such, the tool is designed from the ground up to be extensible - making it easy to generate new traffic types, over new protocols and with individualized traffic sources.

Why Twitter Couldn't Handle Ellen's Selfie
If we were to do the math, Iago was up and running for nearly two years before the Oscar selfie. What happened to their load testing tool?

There were two main reasons why Twitter crashed. First, the tweet Ellen posted was a picture. On Twitter, a tweet accounts for only 260 bytes of data while a picture on Twitter accounts for 33KB of data, almost 130 times as much as a tweet. Second, Twitter's distributed server system was already at max capacity so the load taken on by the website couldn't be distributed to any nearby servers.

Twitter made one major mistake contributing to the crash back in March: they didn't anticipate and replicate real user activity. Most likely Iago wasn't instructed to generate a load based on a picture being retweeted millions of times, thus Twitter didn't know what to expect when the Oscars rolled around.

Homegrown Tool vs. Vendor Load Testing Solutions
Twitter was looking for a DIY homegrown solution because of their unique platform, and while most load testing tools seek to accomplish the same goal, there are always differences between tools. Here are some of the differences we see between homegrown and vendor-provided load testing tools.

Homegrown

Vendor

1. You can customize it - A homegrown tool, created completely from scratch, means you can build exactly what you think you need. But you don't get the benefit of lots of other people's experiences - so what you think you need may not turn out to be what you actually need.

2. Homegrown tools are free like a puppy is free - We have mentioned this analogy before in another post, and it needs to be brought up again. Homegrown tools aren't really free. You have to pay a team to keep them up and running.

3. What if someone leaves? - A couple of developers are experts on the tool they built, but what happens when they leave? Who is going to be your expert? If the new person isn't properly trained and if the code isn't well documented, then modifications to the code become extremely difficult.

1. Your vendor makes it extensible - A vendor that supports lots of users is constantly adding features and capabilities to support needs you may not have realized you have yet. With extensible APIs and SDKs, you still have the ability to add integrations and the protocol support your application demands.

2. You pay for something that works - With homegrown tools there is no guarantee that it will work 100% of the time. By paying a vendor to use their tool, you have assurances that your load testing tool won't let you down when you need it most.

3. Support Team - Vendors dedicate people to work on any issues and keep you worry-free. Their team of experts is available to make sure all of your questions and concerns are addressed.

Homegrown Tools Are Not for Everyone
Twitter's tool has done well to help them improve the performance of the service, but continued service disruptions show that it isn't perfect. A homegrown tool isn't always the best solution, and most of the time, companies do not have the resources handy to completely build their load testing tool from scratch. If you are interested in learning more about vendor options, give us a call. We're more than happy to help.

More Stories By Tim Hinds

Tim Hinds is the Product Marketing Manager for NeoLoad at Neotys. He has a background in Agile software development, Scrum, Kanban, Continuous Integration, Continuous Delivery, and Continuous Testing practices.

Previously, Tim was Product Marketing Manager at AccuRev, a company acquired by Micro Focus, where he worked with software configuration management, issue tracking, Agile project management, continuous integration, workflow automation, and distributed version control systems.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


Latest Stories from Big Data Journal
Enthusiasm for the Internet of Things has reached an all-time high. In 2013 alone, venture capitalists spent more than $1 billion dollars investing in the IoT space. With “smart” appliances and devices, IoT covers wearable smart devices, cloud services to hardware companies. Nest, a Google company, detects temperatures inside homes and automatically adjusts it by tracking its user’s habit. These technologies are quickly developing and with it come challenges such as bridging infrastructure gaps,...
Predicted by Gartner to add $1.9 trillion to the global economy by 2020, the Internet of Everything (IoE) is based on the idea that devices, systems and services will connect in simple, transparent ways, enabling seamless interactions among devices across brands and sectors. As this vision unfolds, it is clear that no single company can accomplish the level of interoperability required to support the horizontal aspects of the IoE. The AllSeen Alliance, announced in December 2013, was formed wi...
Goodness there is a lot of talk about cloud computing. This ‘talk and chatter’ is part of the problem, i.e., we look at it, we prod it and we might even test it out – but do we get down to practical implementation, deployment and (if you happen to be a fan of the term) actual cloud ‘rollout’ today? Cloud offers the promise of a new era they say – and a new style of IT at that. But this again is the problem and we know that cloud can only deliver on the promises it makes if it is part of a well...
There’s Big Data, then there’s really Big Data from the Internet of Things. IoT is evolving to include many data possibilities like new types of event, log and network data. The volumes are enormous, generating tens of billions of logs per day, which raise data challenges. Early IoT deployments are relying heavily on both the cloud and managed service providers to navigate these challenges. In her session at 6th Big Data Expo®, Hannah Smalltree, Director at Treasure Data, to discuss how IoT, B...
SYS-CON Events announced today that Connected Data, the creator of Transporter, the world’s first peer-to-peer private cloud storage device, will exhibit at SYS-CON's 15th International Cloud Expo®, which will take place on November 4–6, 2014, at the Santa Clara Convention Center in Santa Clara, CA. Connected Data is the creator of Transporter, the world’s first peer-to-peer private cloud storage device. Connected Data is focused on providing elegantly designed solutions for consumers, professi...
Cisco on Wedesday announced its intent to acquire privately held Metacloud. Based in Pasadena, Calif., Metacloud deploys and operates private clouds for global organizations with a unique OpenStack-as-a-Service model that delivers and remotely operates production-ready private clouds in a customer's data center. Metacloud's OpenStack-based cloud platform will accelerate Cisco's strategy to build the world's largest global Intercloud, a network of clouds, together with key partners to address cu...
Technology is enabling a new approach to collecting and using data. This approach, commonly referred to as the “Internet of Things” (IoT), enables businesses to use real-time data from all sorts of things including machines, devices and sensors to make better decisions, improve customer service, and lower the risk in the creation of new revenue opportunities. In his session at Internet of @ThingsExpo, Dave Wagstaff, Vice President and Chief Architect at BSQUARE Corporation, will discuss the real...
I write and study often on the subject of digital transformation - the digital transformation of industries, markets, products, business models, etc. In brief, digital transformation is about the impact that collected and analyzed data can have when used to enhance business processes and workflows. If Amazon knows your preferences for particular books and films based upon captured data, then they can apply analytics to predict related books and films that you may like. This improves sales. T...
IoT is still a vague buzzword for many people. In his session at Internet of @ThingsExpo, Mike Kavis, Vice President & Principal Cloud Architect at Cloud Technology Partners, will discuss the business value of IoT that goes far beyond the general public's perception that IoT is all about wearables and home consumer services. The presentation will also discuss how IoT is perceived by investors and how venture capitalist access this space. Other topics to discuss are barriers to success, what is n...
When one expects instantaneous response from video generated on the internet, lots of invisible problems have to be overcome. In his session at 6th Big Data Expo®, Tom Paquin, EVP and Chief Technology Officer at OnLive, to discuss how to overcome these problems. A Silicon Valley veteran, Tom Paquin provides vision, expertise and leadership to the technology research and development effort at OnLive as EVP and Chief Technology Officer. With more than 20 years of management experience at lead...