Welcome!

Big Data Journal Authors: Liz McMillan, Kira Makagon, Elizabeth White, Jason Bloomberg, Carmen Gonzalez

Related Topics: SOA & WOA, Java, Linux, Web 2.0, Security, Big Data Journal

SOA & WOA: Article

Twitter’s Story: How Homegrown Load Testing Tools Can Misfire

If Twitter built their own tool to perform their own load test, why did the selfie cause their site to crash?

The selfie that changed the world, or at least Twitter, has been in the news for the past month. On March 2, 2014, the infamous Oscar selfie of Ellen and her celebrity friends broke a record of 2 million retweets before midnight the same night. That record was previously set by President Barack Obama, hugging first lady Michelle Obama after his 2012 re-election.

The selfie caused Twitter to crash for more than 20 minutes, also breaking the record for the longest crash of the social media site. Twitter was infamous for crashing in its early days (anyone remember "Fail Whale?"), so it's no wonder the social media giant worked extra hard to completely prepare their website infrastructure before going public in November 2013. This included building their own load testing tool, Iago, in 2012.

If they built their own tool to perform their own load test, why did the selfie cause their site to crash? The Oscar selfie crash is a perfect example of what companies can easily overlook. Twitter did not test their users properly and their homegrown tool clearly doesn't solve all of their problems... their servers still crash.

What is Iago and why did Twitter decide to make it? And what does that have to do with your decision to use homegrown tools versus vendor load testing tools? Don't worry, we will tell you.

Twitter's Homegrown Load Testing Tool: Iago
Iago was created in June of 2012 by Twitter's internal engineering team. According to Twitter, Iago is a load generator created to help the social media site test services before they encounter production traffic. Chris Aniszczyk, Head of Open Source at Twitter, said, "There are many load generators available in the open source and commercial software worlds, but Iago provided us with capabilities that are uniquely suited for Twitter's environment and the precise degree to which we need to test our services."

Basically their homegrown tool was completely customized for their platform alone - a very attractive aspect of developing your own tool.

The three attributes Twitter focused on in creating Iago were:

  1. High performance: Iago was designed to generate traffic in a precise and predictable way, to minimize variance between test runs and allow comparisons to be made between development iterations.
  2. Multi-protocol: Modeling a system as complex as Twitter can be difficult, but it's made easier by decomposing it into component services. Once decomposed, each piece can be tested in isolation; which requires the load generator to speak each service's protocol. Twitter has in excess of 100 such services, and Iago tests most of them using built-in support for the internal protocols Twitter uses.
  3. Extensible: Iago is designed for engineers. It assumes the person building the system will also be interested in validating performance. As such, the tool is designed from the ground up to be extensible - making it easy to generate new traffic types, over new protocols and with individualized traffic sources.

Why Twitter Couldn't Handle Ellen's Selfie
If we were to do the math, Iago was up and running for nearly two years before the Oscar selfie. What happened to their load testing tool?

There were two main reasons why Twitter crashed. First, the tweet Ellen posted was a picture. On Twitter, a tweet accounts for only 260 bytes of data while a picture on Twitter accounts for 33KB of data, almost 130 times as much as a tweet. Second, Twitter's distributed server system was already at max capacity so the load taken on by the website couldn't be distributed to any nearby servers.

Twitter made one major mistake contributing to the crash back in March: they didn't anticipate and replicate real user activity. Most likely Iago wasn't instructed to generate a load based on a picture being retweeted millions of times, thus Twitter didn't know what to expect when the Oscars rolled around.

Homegrown Tool vs. Vendor Load Testing Solutions
Twitter was looking for a DIY homegrown solution because of their unique platform, and while most load testing tools seek to accomplish the same goal, there are always differences between tools. Here are some of the differences we see between homegrown and vendor-provided load testing tools.

Homegrown

Vendor

1. You can customize it - A homegrown tool, created completely from scratch, means you can build exactly what you think you need. But you don't get the benefit of lots of other people's experiences - so what you think you need may not turn out to be what you actually need.

2. Homegrown tools are free like a puppy is free - We have mentioned this analogy before in another post, and it needs to be brought up again. Homegrown tools aren't really free. You have to pay a team to keep them up and running.

3. What if someone leaves? - A couple of developers are experts on the tool they built, but what happens when they leave? Who is going to be your expert? If the new person isn't properly trained and if the code isn't well documented, then modifications to the code become extremely difficult.

1. Your vendor makes it extensible - A vendor that supports lots of users is constantly adding features and capabilities to support needs you may not have realized you have yet. With extensible APIs and SDKs, you still have the ability to add integrations and the protocol support your application demands.

2. You pay for something that works - With homegrown tools there is no guarantee that it will work 100% of the time. By paying a vendor to use their tool, you have assurances that your load testing tool won't let you down when you need it most.

3. Support Team - Vendors dedicate people to work on any issues and keep you worry-free. Their team of experts is available to make sure all of your questions and concerns are addressed.

Homegrown Tools Are Not for Everyone
Twitter's tool has done well to help them improve the performance of the service, but continued service disruptions show that it isn't perfect. A homegrown tool isn't always the best solution, and most of the time, companies do not have the resources handy to completely build their load testing tool from scratch. If you are interested in learning more about vendor options, give us a call. We're more than happy to help.

More Stories By Tim Hinds

Tim Hinds is the Product Marketing Manager for NeoLoad at Neotys. He has a background in Agile software development, Scrum, Kanban, Continuous Integration, Continuous Delivery, and Continuous Testing practices.

Previously, Tim was Product Marketing Manager at AccuRev, a company acquired by Micro Focus, where he worked with software configuration management, issue tracking, Agile project management, continuous integration, workflow automation, and distributed version control systems.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


@BigDataExpo Stories
There is no doubt that Big Data is here and getting bigger every day. Building a Big Data infrastructure today is no easy task. There are an enormous number of choices for database engines and technologies. To make things even more challenging, requirements are getting more sophisticated, and the standard paradigm of supporting historical analytics queries is often just one facet of what is needed. As Big Data growth continues, organizations are demanding real-time access to data, allowing immed...
The Internet of Things will greatly expand the opportunities for data collection and new business models driven off of that data. In her session at @ThingsExpo, Esmeralda Swartz, CMO of MetraTech, discussed how for this to be effective you not only need to have infrastructure and operational models capable of utilizing this new phenomenon, but increasingly service providers will need to convince a skeptical public to participate. Get ready to show them the money!
The Industrial Internet revolution is now underway, enabled by connected machines and billions of devices that communicate and collaborate. The massive amounts of Big Data requiring real-time analysis is flooding legacy IT systems and giving way to cloud environments that can handle the unpredictable workloads. Yet many barriers remain until we can fully realize the opportunities and benefits from the convergence of machines and devices with Big Data and the cloud, including interoperability, ...
The 3rd International Internet of @ThingsExpo, co-located with the 16th International Cloud Expo - to be held June 9-11, 2015, at the Javits Center in New York City, NY - announces that its Call for Papers is now open. The Internet of Things (IoT) is the biggest idea since the creation of the Worldwide Web more than 20 years ago.
The 4th International DevOps Summit, co-located with16th International Cloud Expo – being held June 9-11, 2015, at the Javits Center in New York City, NY – announces that its Call for Papers is now open. Born out of proven success in agile development, cloud computing, and process automation, DevOps is a macro trend you cannot afford to miss. From showcase success stories from early adopters and web-scale businesses, DevOps is expanding to organizations of all sizes, including the world's large...
Cloud Expo 2014 TV commercials will feature @ThingsExpo, which was launched in June, 2014 at New York City's Javits Center as the largest 'Internet of Things' event in the world.
“We help people build clusters, in the classical sense of the cluster. We help people put a full stack on top of every single one of those machines. We do the full bare metal install," explained Greg Bruno, Vice President of Engineering and co-founder of StackIQ, in this SYS-CON.tv interview at 15th Cloud Expo, held Nov 4–6, 2014, at the Santa Clara Convention Center in Santa Clara, CA.
"People are a lot more knowledgeable about APIs now. There are two types of people who work with APIs - IT people who want to use APIs for something internal and the product managers who want to do something outside APIs for people to connect to them," explained Roberto Medrano, Executive Vice President at SOA Software, in this SYS-CON.tv interview at Cloud Expo, held Nov 4–6, 2014, at the Santa Clara Convention Center in Santa Clara, CA.
In this demo at 15th Cloud Expo, John Meza, Product Engineer at Esri, showed how Esri products hook into Hadoop cluster to allow you to do spatial analysis on the spatial data within your cluster, and he demonstrated rendering from a data center with ArcGIS Pro, a new product that has a brand new rendering engine.
The Software Defined Data Center (SDDC), which enables organizations to seamlessly run in a hybrid cloud model (public + private cloud), is here to stay. IDC estimates that the software-defined networking market will be valued at $3.7 billion by 2016. Security is a key component and benefit of the SDDC, and offers an opportunity to build security 'from the ground up' and weave it into the environment from day one. In his session at 16th Cloud Expo, Reuven Harrison, CTO and Co-Founder of Tufin,...
SYS-CON Media announced that Splunk, a provider of the leading software platform for real-time Operational Intelligence, has launched an ad campaign on Big Data Journal. Splunk software and cloud services enable organizations to search, monitor, analyze and visualize machine-generated big data coming from websites, applications, servers, networks, sensors and mobile devices. The ads focus on delivering ROI - how improved uptime delivered $6M in annual ROI, improving customer operations by minin...
Can the spatial component of your Big Data be harnessed and visualized, adding another dimension of power and analytics to your data? In his session at Big Data Expo®, John Meza, Product Engineer and Performance Engineering Team Lead at Esri, discussed the spatial queries that can be used within the Hadoop ecosystem and their integration with GeoSpatial applications. The GIS Tools for Hadoop project was also discussed and its implementation to discover location-based patterns and relationships...
DevOps Summit 2015 New York, co-located with the 16th International Cloud Expo - to be held June 9-11, 2015, at the Javits Center in New York City, NY - announces that it is now accepting Keynote Proposals. The widespread success of cloud computing is driving the DevOps revolution in enterprise IT. Now as never before, development teams must communicate and collaborate in a dynamic, 24/7/365 environment. There is no time to wait for long development cycles that produce software that is obsolete...
Software AG and Wipro Ltd. have announced a joint solution platform for streaming analytics that provides real-time actionable intelligence for the Internet of Things (IoT) market. “The key to successfully addressing the IoT market is the ability to rapidly build and evolve apps that tap into, analyze and make smart decisions on fast, big data”, said John Bates, Global Head of Industry Solutions and CMO, Software AG. To address the huge market potential created by streaming analytics in conj...
We’re no longer looking to the future for the IoT wave. It’s no longer a distant dream but a reality that has arrived. It’s now time to make sure the industry is in alignment to meet the IoT growing pains – cooperate and collaborate as well as innovate. In his session at @ThingsExpo, Jim Hunter, Chief Scientist & Technology Evangelist at Greenwave Systems, will examine the key ingredients to IoT success and identify solutions to challenges the industry is facing. The deep industry expertise be...
DevOps means different things to different people. Qubell defines DevOps as the ability for the developer teams to do what they need to do to have this level of self-service. At DevOps Summit, Stan Klimoff, CTO of Qubell, demos the enterprise DevOps platform.
SYS-CON Events announced today that that Innodisk, the service-driven provider of industrial embedded flash and DRAM storage products and technologies, will exhibit at SYS-CON's 16th International Cloud Expo®, which will take place on June 9-11, 2015, at the Javits Center in New York City, NY. Innodisk is a service-driven provider of industrial embedded flash and DRAM storage products and technologies. With satisfied customers across the embedded, aerospace and defense, cloud storage markets an...
The 3rd International Internet of @ThingsExpo, co-located with the 16th International Cloud Expo - to be held June 9-11, 2015, at the Javits Center in New York City, NY - announces that its Call for Papers is now open. The Internet of Things (IoT) is the biggest idea since the creation of the Worldwide Web more than 20 years ago.
Eighty-five percent of companies store information in some sort of unstructured manner. In this demo at 15th Cloud Expo, Mark Fronczak, Product Manager at Solgenia, discussed their enterprise content management solution, which was created to help companies organize and take control of their digital assets.

ARMONK, N.Y., Nov. 20, 2014 /PRNewswire/ --  IBM (NYSE: IBM) today announced that it is bringing a greater level of control, security and flexibility to cloud-based application development and delivery with a single-tenant version of Bluemix, IBM's