|By Marten Terpstra||
|April 24, 2014 06:00 AM EDT||
Earlier this week I had some interesting conversations with @davehusak. Where the conversation started early in the day with a discussion on overlay networks and what network functions are performed where and in what context, later in the afternoon the discussion moved to networking solutions (and specifically Plexxi solutions) for big data applications.
It’s easy to jump to Hadoop or similarly structured cluster computing applications (Spark, Storm, or a long list of others) as the definition of a big data application. With all its simplicity for the overall distribution of work, Hadoop is a fairly tough network problem to solve if you want to do anything more than “throw bandwidth at the problem”. And when you do throw bandwidth at the problem, the extreme burstiness of the traffic will still significantly drag down the performance of the overall solution. And for many, CPU cycles to reduce the data is not the biggest challenge, storage and movement of data throughout a big data cluster is the biggest pain point. Intel has done a fine job providing compute firepower that far outpaces the evolution of network capacity.
The network plays a significant factor in several stages of a Hadoop solution cycle. It starts with chopping the to-be-analyzed data into chunks and distributing it across the datanodes. Hadoop has a notion of a rack and it has some basic intelligence when placing data and jobs that work on that data. By default the data will be replicated 3 times across at least 2 racks, if racks have been defined. The data to be distributed is easily in the 100s of Gigabytes or even Terabytes, so triple that data is being moved throughout the Hadoop cluster to the datanodes.
Once distributed, the actual Map jobs are launched against that data, these are the tasks that take the data and perform a first pass mapping into (in its most basic form) <key, value> tuples. Again here there is an attempt to have jobs work on local data, where local can be defined as local to the server that has that chunk of data or local to the rack, in an attempt to avoid as much cross rack communication as possible. This is based on the assumption that cross rack communication is much more constrained and aggregated and therefore more prone to congestion and packet loss.
Once the mapper jobs complete their task, the results of the mapping exercise is sent to reducers. Reducers take the <key, value> information and essentially tally the results. This transfer is the most taxing part of a Hadoop cycle on the network. Since most Hadoop mapping jobs run the same function on a similar sized dataset, that first set of mappers will all complete their task at about the same time and will all start sending their results to the same set of reducers, creating 1) a lot of traffic and 2) a lot of traffic to the same set of destinations. Depending on the amount of data, mappers and compute nodes, this cycle repeats (the next set of mapping jobs are fired off) and at the end of each cycle a very significant spike in network traffic appears. At the very end, all results are brought together for one last spike in traffic. Each one of these network events is a source of significant congestion.
Many variables contribute to the overall performance of the Hadoop solution. What is the relationship between the chunks of data and the amount of servers and jobs? How many reducers are used? Where are the reducers in relation to the mappers? Is the Map function compute heavy or I/O heavy? How aggressive is the speculative scheduling that allows the same data to be worked on by multiple mappers?
With that many variables that can be tuned, and with so many variables different from one analysis to the next, it is hard to imagine that a single network design or implementation provides the best supporting infrastructure. There are assumptions in Hadoop placement of data and jobs that can easily be altered. The basic concept of a rack can easily expanded into a multi layer locality definition. With the right tools in the network, the definition of a rack, or even the locality and closeness of nodes in a cluster or virtual cluster can be adjusted according to the analysis to be completed.
We tend to give our applications variables to tune its performance based on what the network provides. It is time that the network adjusts itself based on the application needs. In a clustered application like Hadoop, there is lots of knowledge and even some predictability of network traffic. Wouldn’t that make for a great opportunity to infuse the network with some of that knowledge and have it morph itself to provide the best possible service? And Hadoop is not unique, cluster compute framework almost all carefully track placement of data and compute jobs, which makes them all great candidates to share some of that information with a smart network.
There are far simpler big data needs and applications that can and should be supported by flexible networks. Storage networks are still often separated from data networks for performance reasons and fear of interference. If you could actually separate the various types of data with logically or even physically different paths in the same network, would you still?
[Today's fun fact: An MLB baseball lasts on average 7 pitches. A google search asking how many baseballs are used in a single MLB season returns several pages worth of different answers. They must all be correct, it's on the Internet afterall.]
SYS-CON Events has announced today that Roger Strukhoff has been named conference chair of Cloud Expo and @ThingsExpo 2017 New York. The 20th Cloud Expo and 7th @ThingsExpo will take place on June 6-8, 2017, at the Javits Center in New York City, NY. "The Internet of Things brings trillions of dollars of opportunity to developers and enterprise IT, no matter how you measure it," stated Roger Strukhoff. "More importantly, it leverages the power of devices and the Internet to enable us all to im...
Dec. 9, 2016 12:30 AM EST Reads: 876
Whether your IoT service is connecting cars, homes, appliances, wearable, cameras or other devices, one question hangs in the balance – how do you actually make money from this service? The ability to turn your IoT service into profit requires the ability to create a monetization strategy that is flexible, scalable and working for you in real-time. It must be a transparent, smoothly implemented strategy that all stakeholders – from customers to the board – will be able to understand and comprehe...
Dec. 9, 2016 12:00 AM EST Reads: 681
"We are the public cloud providers. We are currently providing 50% of the resources they need for doing e-commerce business in China and we are hosting about 60% of mobile gaming in China," explained Yi Zheng, CPO and VP of Engineering at CDS Global Cloud, in this SYS-CON.tv interview at 19th Cloud Expo, held November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA.
Dec. 9, 2016 12:00 AM EST Reads: 1,171
"Once customers get a year into their IoT deployments, they start to realize that they may have been shortsighted in the ways they built out their deployment and the key thing I see a lot of people looking at is - how can I take equipment data, pull it back in an IoT solution and show it in a dashboard," stated Dave McCarthy, Director of Products at Bsquare Corporation, in this SYS-CON.tv interview at @ThingsExpo, held November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA.
Dec. 8, 2016 09:45 PM EST Reads: 1,237
What happens when the different parts of a vehicle become smarter than the vehicle itself? As we move toward the era of smart everything, hundreds of entities in a vehicle that communicate with each other, the vehicle and external systems create a need for identity orchestration so that all entities work as a conglomerate. Much like an orchestra without a conductor, without the ability to secure, control, and connect the link between a vehicle’s head unit, devices, and systems and to manage the ...
Dec. 8, 2016 09:15 PM EST Reads: 959
All clouds are not equal. To succeed in a DevOps context, organizations should plan to develop/deploy apps across a choice of on-premise and public clouds simultaneously depending on the business needs. This is where the concept of the Lean Cloud comes in - resting on the idea that you often need to relocate your app modules over their life cycles for both innovation and operational efficiency in the cloud. In his session at @DevOpsSummit at19th Cloud Expo, Valentin (Val) Bercovici, CTO of Soli...
Dec. 8, 2016 08:30 PM EST Reads: 1,891
Complete Internet of Things (IoT) embedded device security is not just about the device but involves the entire product’s identity, data and control integrity, and services traversing the cloud. A device can no longer be looked at as an island; it is a part of a system. In fact, given the cross-domain interactions enabled by IoT it could be a part of many systems. Also, depending on where the device is deployed, for example, in the office building versus a factory floor or oil field, security ha...
Dec. 8, 2016 07:45 PM EST Reads: 330
Amazon has gradually rolled out parts of its IoT offerings in the last year, but these are just the tip of the iceberg. In addition to optimizing their back-end AWS offerings, Amazon is laying the ground work to be a major force in IoT – especially in the connected home and office. Amazon is extending its reach by building on its dominant Cloud IoT platform, its Dash Button strategy, recently announced Replenishment Services, the Echo/Alexa voice recognition control platform, the 6-7 strategic...
Dec. 8, 2016 07:15 PM EST Reads: 392
As data explodes in quantity, importance and from new sources, the need for managing and protecting data residing across physical, virtual, and cloud environments grow with it. Managing data includes protecting it, indexing and classifying it for true, long-term management, compliance and E-Discovery. Commvault can ensure this with a single pane of glass solution – whether in a private cloud, a Service Provider delivered public cloud or a hybrid cloud environment – across the heterogeneous enter...
Dec. 8, 2016 05:00 PM EST Reads: 1,822
Financial Technology has become a topic of intense interest throughout the cloud developer and enterprise IT communities. Accordingly, attendees at the upcoming 20th Cloud Expo at the Javits Center in New York, June 6-8, 2017, will find fresh new content in a new track called FinTech.
Dec. 8, 2016 04:45 PM EST Reads: 2,255
You have great SaaS business app ideas. You want to turn your idea quickly into a functional and engaging proof of concept. You need to be able to modify it to meet customers' needs, and you need to deliver a complete and secure SaaS application. How could you achieve all the above and yet avoid unforeseen IT requirements that add unnecessary cost and complexity? You also want your app to be responsive in any device at any time. In his session at 19th Cloud Expo, Mark Allen, General Manager of...
Dec. 8, 2016 04:45 PM EST Reads: 1,851
"We are a leader in the market space called network visibility solutions - it enables monitoring tools and Big Data analysis to access the data and be able to see the performance," explained Shay Morag, VP of Sales and Marketing at Niagara Networks, in this SYS-CON.tv interview at 19th Cloud Expo, held November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA.
Dec. 8, 2016 04:30 PM EST Reads: 406
The 20th International Cloud Expo has announced that its Call for Papers is open. Cloud Expo, to be held June 6-8, 2017, at the Javits Center in New York City, brings together Cloud Computing, Big Data, Internet of Things, DevOps, Containers, Microservices and WebRTC to one location. With cloud computing driving a higher percentage of enterprise IT budgets every year, it becomes increasingly important to plant your flag in this fast-expanding business opportunity. Submit your speaking proposal ...
Dec. 8, 2016 04:15 PM EST Reads: 2,321
Bert Loomis was a visionary. This general session will highlight how Bert Loomis and people like him inspire us to build great things with small inventions. In their general session at 19th Cloud Expo, Harold Hannon, Architect at IBM Bluemix, and Michael O'Neill, Strategic Business Development at Nvidia, discussed the accelerating pace of AI development and how IBM Cloud and NVIDIA are partnering to bring AI capabilities to "every day," on-demand. They also reviewed two "free infrastructure" pr...
Dec. 8, 2016 03:30 PM EST Reads: 1,245
Unsecured IoT devices were used to launch crippling DDOS attacks in October 2016, targeting services such as Twitter, Spotify, and GitHub. Subsequent testimony to Congress about potential attacks on office buildings, schools, and hospitals raised the possibility for the IoT to harm and even kill people. What should be done? Does the government need to intervene? This panel at @ThingExpo New York brings together leading IoT and security experts to discuss this very serious topic.
Dec. 8, 2016 03:00 PM EST Reads: 536
More and more brands have jumped on the IoT bandwagon. We have an excess of wearables – activity trackers, smartwatches, smart glasses and sneakers, and more that track seemingly endless datapoints. However, most consumers have no idea what “IoT” means. Creating more wearables that track data shouldn't be the aim of brands; delivering meaningful, tangible relevance to their users should be. We're in a period in which the IoT pendulum is still swinging. Initially, it swung toward "smart for smar...
Dec. 8, 2016 02:30 PM EST Reads: 945
CloudJumper, a Workspace as a Service (WaaS) platform innovator for agile business IT, has been recognized with the Customer Value Leadership Award for its nWorkSpace platform by Frost & Sullivan. The company was also featured in a new report(1) by the industry research firm titled, “Desktop-as-a-Service Buyer’s Guide, 2016,” which provides a comprehensive comparison of DaaS providers, including CloudJumper, Amazon, VMware, and Microsoft.
Dec. 8, 2016 02:30 PM EST Reads: 994
"Dice has been around for the last 20 years. We have been helping tech professionals find new jobs and career opportunities," explained Manish Dixit, VP of Product and Engineering at Dice, in this SYS-CON.tv interview at 19th Cloud Expo, held November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA.
Dec. 8, 2016 02:15 PM EST Reads: 1,197
SYS-CON Events announced today that Fusion, a leading provider of cloud services, will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Fusion, a leading provider of integrated cloud solutions to small, medium and large businesses, is the industry’s single source for the cloud. Fusion’s advanced, proprietary cloud service platform enables the integration of leading edge solutions in the cloud, including clou...
Dec. 8, 2016 01:00 PM EST Reads: 494
The many IoT deployments around the world are busy integrating smart devices and sensors into their enterprise IT infrastructures. Yet all of this technology – and there are an amazing number of choices – is of no use without the software to gather, communicate, and analyze the new data flows. Without software, there is no IT. In this power panel at @ThingsExpo, moderated by Conference Chair Roger Strukhoff, Dave McCarthy, Director of Products at Bsquare Corporation; Alan Williamson, Principal...
Dec. 8, 2016 12:00 PM EST Reads: 544