Welcome!

@BigDataExpo Authors: Pat Romanski, Jeev Trika, Liz McMillan, Elizabeth White, Scott Allen

Related Topics: Microservices Expo, Java IoT, Linux Containers, Agile Computing, Cloud Security, @BigDataExpo

Microservices Expo: Article

Twitter’s Story: How Homegrown Load Testing Tools Can Misfire

If Twitter built their own tool to perform their own load test, why did the selfie cause their site to crash?

The selfie that changed the world, or at least Twitter, has been in the news for the past month. On March 2, 2014, the infamous Oscar selfie of Ellen and her celebrity friends broke a record of 2 million retweets before midnight the same night. That record was previously set by President Barack Obama, hugging first lady Michelle Obama after his 2012 re-election.

The selfie caused Twitter to crash for more than 20 minutes, also breaking the record for the longest crash of the social media site. Twitter was infamous for crashing in its early days (anyone remember "Fail Whale?"), so it's no wonder the social media giant worked extra hard to completely prepare their website infrastructure before going public in November 2013. This included building their own load testing tool, Iago, in 2012.

If they built their own tool to perform their own load test, why did the selfie cause their site to crash? The Oscar selfie crash is a perfect example of what companies can easily overlook. Twitter did not test their users properly and their homegrown tool clearly doesn't solve all of their problems... their servers still crash.

What is Iago and why did Twitter decide to make it? And what does that have to do with your decision to use homegrown tools versus vendor load testing tools? Don't worry, we will tell you.

Twitter's Homegrown Load Testing Tool: Iago
Iago was created in June of 2012 by Twitter's internal engineering team. According to Twitter, Iago is a load generator created to help the social media site test services before they encounter production traffic. Chris Aniszczyk, Head of Open Source at Twitter, said, "There are many load generators available in the open source and commercial software worlds, but Iago provided us with capabilities that are uniquely suited for Twitter's environment and the precise degree to which we need to test our services."

Basically their homegrown tool was completely customized for their platform alone - a very attractive aspect of developing your own tool.

The three attributes Twitter focused on in creating Iago were:

  1. High performance: Iago was designed to generate traffic in a precise and predictable way, to minimize variance between test runs and allow comparisons to be made between development iterations.
  2. Multi-protocol: Modeling a system as complex as Twitter can be difficult, but it's made easier by decomposing it into component services. Once decomposed, each piece can be tested in isolation; which requires the load generator to speak each service's protocol. Twitter has in excess of 100 such services, and Iago tests most of them using built-in support for the internal protocols Twitter uses.
  3. Extensible: Iago is designed for engineers. It assumes the person building the system will also be interested in validating performance. As such, the tool is designed from the ground up to be extensible - making it easy to generate new traffic types, over new protocols and with individualized traffic sources.

Why Twitter Couldn't Handle Ellen's Selfie
If we were to do the math, Iago was up and running for nearly two years before the Oscar selfie. What happened to their load testing tool?

There were two main reasons why Twitter crashed. First, the tweet Ellen posted was a picture. On Twitter, a tweet accounts for only 260 bytes of data while a picture on Twitter accounts for 33KB of data, almost 130 times as much as a tweet. Second, Twitter's distributed server system was already at max capacity so the load taken on by the website couldn't be distributed to any nearby servers.

Twitter made one major mistake contributing to the crash back in March: they didn't anticipate and replicate real user activity. Most likely Iago wasn't instructed to generate a load based on a picture being retweeted millions of times, thus Twitter didn't know what to expect when the Oscars rolled around.

Homegrown Tool vs. Vendor Load Testing Solutions
Twitter was looking for a DIY homegrown solution because of their unique platform, and while most load testing tools seek to accomplish the same goal, there are always differences between tools. Here are some of the differences we see between homegrown and vendor-provided load testing tools.

Homegrown

Vendor

1. You can customize it - A homegrown tool, created completely from scratch, means you can build exactly what you think you need. But you don't get the benefit of lots of other people's experiences - so what you think you need may not turn out to be what you actually need.

2. Homegrown tools are free like a puppy is free - We have mentioned this analogy before in another post, and it needs to be brought up again. Homegrown tools aren't really free. You have to pay a team to keep them up and running.

3. What if someone leaves? - A couple of developers are experts on the tool they built, but what happens when they leave? Who is going to be your expert? If the new person isn't properly trained and if the code isn't well documented, then modifications to the code become extremely difficult.

1. Your vendor makes it extensible - A vendor that supports lots of users is constantly adding features and capabilities to support needs you may not have realized you have yet. With extensible APIs and SDKs, you still have the ability to add integrations and the protocol support your application demands.

2. You pay for something that works - With homegrown tools there is no guarantee that it will work 100% of the time. By paying a vendor to use their tool, you have assurances that your load testing tool won't let you down when you need it most.

3. Support Team - Vendors dedicate people to work on any issues and keep you worry-free. Their team of experts is available to make sure all of your questions and concerns are addressed.

Homegrown Tools Are Not for Everyone
Twitter's tool has done well to help them improve the performance of the service, but continued service disruptions show that it isn't perfect. A homegrown tool isn't always the best solution, and most of the time, companies do not have the resources handy to completely build their load testing tool from scratch. If you are interested in learning more about vendor options, give us a call. We're more than happy to help.

More Stories By Tim Hinds

Tim Hinds is the Product Marketing Manager for NeoLoad at Neotys. He has a background in Agile software development, Scrum, Kanban, Continuous Integration, Continuous Delivery, and Continuous Testing practices.

Previously, Tim was Product Marketing Manager at AccuRev, a company acquired by Micro Focus, where he worked with software configuration management, issue tracking, Agile project management, continuous integration, workflow automation, and distributed version control systems.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


@BigDataExpo Stories
"We work in the area of Big Data analytics and Big Data analytics is a very crowded space - you have Hadoop, ETL, warehousing, visualization and there's a lot of effort trying to get these tools to talk to each other," explained Mukund Deshpande, head of the Analytics practice at Accelerite, in this SYS-CON.tv interview at 18th Cloud Expo, held June 7-9, 2016, at the Javits Center in New York City, NY.
Cloud Expo, Inc. has announced today that Andi Mann returns to 'DevOps at Cloud Expo 2016' as Conference Chair The @DevOpsSummit at Cloud Expo will take place on November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. "DevOps is set to be one of the most profound disruptions to hit IT in decades," said Andi Mann. "It is a natural extension of cloud computing, and I have seen both firsthand and in independent research the fantastic results DevOps delivers. So I am excited t...
SaaS companies can greatly expand revenue potential by pushing beyond their own borders. The challenge is how to do this without degrading service quality. In his session at 18th Cloud Expo, Adam Rogers, Managing Director at Anexia, discussed how IaaS providers with a global presence and both virtual and dedicated infrastructure can help companies expand their service footprint with low “go-to-market” costs.
IoT is rapidly changing the way enterprises are using data to improve business decision-making. In order to derive business value, organizations must unlock insights from the data gathered and then act on these. In their session at @ThingsExpo, Eric Hoffman, Vice President at EastBanc Technologies, and Peter Shashkin, Head of Development Department at EastBanc Technologies, discussed how one organization leveraged IoT, cloud technology and data analysis to improve customer experiences and effi...
Internet of @ThingsExpo has announced today that Chris Matthieu has been named tech chair of Internet of @ThingsExpo 2016 Silicon Valley. The 6thInternet of @ThingsExpo will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA.
"Avere Systems is a hybrid cloud solution provider. We have customers that want to use cloud storage and we have customers that want to take advantage of cloud compute," explained Rebecca Thompson, VP of Marketing at Avere Systems, in this SYS-CON.tv interview at 18th Cloud Expo, held June 7-9, 2016, at the Javits Center in New York City, NY.
Presidio has received the 2015 EMC Partner Services Quality Award from EMC Corporation for achieving outstanding service excellence and customer satisfaction as measured by the EMC Partner Services Quality (PSQ) program. Presidio was also honored as the 2015 EMC Americas Marketing Excellence Partner of the Year and 2015 Mid-Market East Partner of the Year. The EMC PSQ program is a project-specific survey program designed for partners with Service Partner designations to solicit customer feedbac...
The cloud promises new levels of agility and cost-savings for Big Data, data warehousing and analytics. But it’s challenging to understand all the options – from IaaS and PaaS to newer services like HaaS (Hadoop as a Service) and BDaaS (Big Data as a Service). In her session at @BigDataExpo at @ThingsExpo, Hannah Smalltree, a director at Cazena, provided an educational overview of emerging “as-a-service” options for Big Data in the cloud. This is critical background for IT and data profession...
Let’s face it, embracing new storage technologies, capabilities and upgrading to new hardware often adds complexity and increases costs. In his session at 18th Cloud Expo, Seth Oxenhorn, Vice President of Business Development & Alliances at FalconStor, discussed how a truly heterogeneous software-defined storage approach can add value to legacy platforms and heterogeneous environments. The result reduces complexity, significantly lowers cost, and provides IT organizations with improved efficie...
Predictive analytics tools monitor, report, and troubleshoot in order to make proactive decisions about the health, performance, and utilization of storage. Most enterprises combine cloud and on-premise storage, resulting in blended environments of physical, virtual, cloud, and other platforms, which justifies more sophisticated storage analytics. In his session at 18th Cloud Expo, Peter McCallum, Vice President of Datacenter Solutions at FalconStor, discussed using predictive analytics to mon...
What does it look like when you have access to cloud infrastructure and platform under the same roof? Let’s talk about the different layers of Technology as a Service: who cares, what runs where, and how does it all fit together. In his session at 18th Cloud Expo, Phil Jackson, Lead Technology Evangelist at SoftLayer, an IBM company, spoke about the picture being painted by IBM Cloud and how the tools being crafted can help fill the gaps in your IT infrastructure.
Most organizations prioritize data security only after their data has already been compromised. Proactive prevention is important, but how can you accomplish that on a small budget? Learn how the cloud, combined with a defense and in-depth approach, creates efficiencies by transferring and assigning risk. Security requires a multi-defense approach, and an in-house team may only be able to cherry pick from the essential components. In his session at 19th Cloud Expo, Vlad Friedman, CEO/Founder o...
Ask someone to architect an Internet of Things (IoT) solution and you are guaranteed to see a reference to the cloud. This would lead you to believe that IoT requires the cloud to exist. However, there are many IoT use cases where the cloud is not feasible or desirable. In his session at @ThingsExpo, Dave McCarthy, Director of Products at Bsquare Corporation, will discuss the strategies that exist to extend intelligence directly to IoT devices and sensors, freeing them from the constraints of ...
The pace of innovation, vendor lock-in, production sustainability, cost-effectiveness, and managing risk… In his session at 18th Cloud Expo, Dan Choquette, Founder of RackN, discussed how CIOs are challenged finding the balance of finding the right tools, technology and operational model that serves the business the best. He also discussed how clouds, open source software and infrastructure solutions have benefits but also drawbacks and how workload and operational portability between vendors ...
The initial debate is over: Any enterprise with a serious commitment to IT is migrating to the cloud. But things are not so simple. There is a complex mix of on-premises, colocated, and public-cloud deployments. In this power panel at 18th Cloud Expo, moderated by Conference Chair Roger Strukhoff, Randy De Meno, Chief Technologist - Windows Products and Microsoft Partnerships at Commvault; Dave Landa, Chief Operating Officer at kintone; William Morrish, General Manager Product Sales at Interou...
Extracting business value from Internet of Things (IoT) data doesn’t happen overnight. There are several requirements that must be satisfied, including IoT device enablement, data analysis, real-time detection of complex events and automated orchestration of actions. Unfortunately, too many companies fall short in achieving their business goals by implementing incomplete solutions or not focusing on tangible use cases. In his general session at @ThingsExpo, Dave McCarthy, Director of Products...
There are several IoTs: the Industrial Internet, Consumer Wearables, Wearables and Healthcare, Supply Chains, and the movement toward Smart Grids, Cities, Regions, and Nations. There are competing communications standards every step of the way, a bewildering array of sensors and devices, and an entire world of competing data analytics platforms. To some this appears to be chaos. In this power panel at @ThingsExpo, moderated by Conference Chair Roger Strukhoff, Bradley Holt, Developer Advocate a...
University of Colorado Athletics has selected FORTRUST, Colorado’s only Tier III Gold certified data center, as their official data center and colocation services provider, FORTRUST announced today. A nationally recognized and prominent collegiate athletics program, CU provides a high quality and comprehensive student-athlete experience. The program sponsors 17 varsity teams and in their history, the Colorado Buffaloes have collected an impressive 28 national championships. Maintaining uptime...
The Internet of Things will challenge the status quo of how IT and development organizations operate. Or will it? Certainly the fog layer of IoT requires special insights about data ontology, security and transactional integrity. But the developmental challenges are the same: People, Process and Platform and how we integrate our thinking to solve complicated problems. In his session at 19th Cloud Expo, Craig Sproule, CEO of Metavine, will demonstrate how to move beyond today's coding paradigm ...
Apixio Inc. has raised $19.3 million in Series D venture capital funding led by SSM Partners with participation from First Analysis, Bain Capital Ventures and Apixio’s largest angel investor. Apixio will dedicate the proceeds toward advancing and scaling products powered by its cognitive computing platform, further enabling insights for optimal patient care. The Series D funding comes as Apixio experiences strong momentum and increasing demand for its HCC Profiler solution, which mines unstruc...