Welcome!

Big Data Journal Authors: Carmen Gonzalez, Liz McMillan, John Grady, Adrian Bridgwater, Pat Romanski

Related Topics: SOA & WOA, Java, Linux, AJAX & REA, Web 2.0, Big Data Journal

SOA & WOA: Article

Understanding Application Performance on the Network | Part 5

Processing Delays

In Part IV, we wrapped up our discussions on bandwidth, congestion and packet loss. In Part V, we examine the four types of processing delays visible on the network, using the request/reply paradigm we outlined in Part I.

Server Processing (Between Flows)
From the network's perspective, we allocate the time period between the end of a request flow and the beginning of the corresponding reply flow to server processing. Generally speaking, the server doesn't begin processing a request until it has received the entire flow, i.e., the last packet in the request message; similarly, the server doesn't begin sending the reply until it has finished processing the request. We sometimes refer to these delays between flows as "pure" processing delays, distinct from another type of intra-flow processing delay we call starved for data and discuss later. Server processing delays occur as a result of a request message, and therefore always occur within a thread.

Transaction Trace Illustrations
These "pure" server processing delays are generally relatively simple to detect, to understand, and to prove. Transaction Trace's Node Processing table lists all of the observed processing delays for an operation in tabular format; by splitting this table with the Bounce Diagram and highlighting a row of interest, the Bounce Diagram will display the last packet of the request flow and the first packet of the corresponding reply flow, effectively diagramming the measurement.

Use the Node Processing Table split with the Bounce Diagram to illustrate node processing delay

You may also use the Thread Analysis split with the Bounce Diagram; this will provide a view of the request and reply packet flows as well as the processing measurement.

Split the Thread View with the Bounce Diagram to illustrate the request flow, node processing and reply flow

Starved for Data (Sending Node, Within a Flow)
Sometimes, the network interface will be able to transmit data at a rate faster than the sending application can deliver to the TCP socket. For example, a busy ftp server may momentarily interrupt sending a large file because of a disk, memory or CPU bottleneck. We refer to these pauses that occur in the middle of a flow as "starved for data" conditions; there is nothing on the network (no TCP flow control constraint) preventing the request or reply flow from continuing, so the cause must be internal to the sending node. Starved for data bottlenecks occur within a flow (instead of between flows), and are related to the sending node - either the client or server.

Transaction Trace Illustration
These cases can be more difficult to visualize. Since the condition is generally not too common, it is often best to rule out other performance bottlenecks first, before checking for data starvation. When it does occur, the condition has the effect of extending the duration of a request or reply flow, and starved for data delays are included in Transaction Trace's Node Sending measurements. Sort the rate column of the Node Sending Table and split the window with the Bounce Diagram; the Bounce Diagram will illustrate the packets associated with a sending measurement. For those sending measurements where you suspect a starved for data condition, look for idle periods of time where the sending node's flow has been interrupted. Importantly, a starved for data delay will terminate with the transmission of a data packet that resumes the sender's flow, not a TCP ACK from the receiver that might suggest a TCP or application window constraint.

The Node Sending table is split with the Bounce Diagram to help illustrate Starved for Data conditions; note the pause in transmission that resumes independent of any TCP ACK.

Client Processing (Between Flows)
From the network's perspective, we allocate the time period between the end of a reply flow and the beginning of the next request flow to client processing. Generally speaking, the client cannot begin processing a reply until it has received the entire reply flow, i.e., the last packet of the reply message; similarly, the client doesn't begin sending the next (new) request until it has completed processing the reply. (This correlation generally applies to request/reply flows on the same TCP connection.)

Transaction Trace Illustrations
Similar to server processing delays, client delays are relatively simple to understand. In most cases, client delays occur between threads; in other words, after one thread has completed but before the next thread begins. For tasks with thread-level decodes, Transaction Trace's Thread Analysis Gantt chart view can illustrate these delays well.

Gaps between threads are associated with client processing delays

Discounting Client Processing
Note that we assume strict adherence to the definition of "operation" here; click to screen update. If the trace has captured multiple steps - say the user navigates through a series of operations - then the user "think time" between steps will appear as client processing delay, with corresponding gaps between threads. You may still use these multi-step tasks for analysis, remembering to discount client processing delays. Alternatively, you can save each step as a separate task by selecting a sequence of threads from the Thread table.

For more network strategies, click here for the full article.

More Stories By Gary Kaiser

Gary Kaiser is a Subject Matter Expert in Network Performance Analysis at Compuware APM. He has global field enablement responsibilities for performance monitoring and analysis solutions embracing emerging and strategic technologies, including WAN optimization, thin client infrastructures, network forensics, and a unique performance management maturity methodology. He is also a co-inventor of multiple analysis features, and continues to champion the value of software-enabled expert network analysis.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


@BigDataExpo Stories
SimpleECM is the only platform to offer a powerful combination of enterprise content management (ECM) services, capture solutions, and third-party business services providing simplified integrations and workflow development for solution providers. SimpleECM is opening the market to businesses of all sizes by reinventing the delivery of ECM services. Our APIs make the development of ECM services simple with the use of familiar technologies for a frictionless integration directly into web applicat...
The 3rd International Internet of @ThingsExpo, co-located with the 16th International Cloud Expo - to be held June 9-11, 2015, at the Javits Center in New York City, NY - announces that its Call for Papers is now open. The Internet of Things (IoT) is the biggest idea since the creation of the Worldwide Web more than 20 years ago.
The Industrial Internet revolution is now underway, enabled by connected machines and billions of devices that communicate and collaborate. The massive amounts of Big Data requiring real-time analysis is flooding legacy IT systems and giving way to cloud environments that can handle the unpredictable workloads. Yet many barriers remain until we can fully realize the opportunities and benefits from the convergence of machines and devices with Big Data and the cloud, including interoperability, da...
All major researchers estimate there will be tens of billions devices - computers, smartphones, tablets, and sensors - connected to the Internet by 2020. This number will continue to grow at a rapid pace for the next several decades. Over the summer Gartner released its much anticipated annual Hype Cycle report and the big news is that Internet of Things has now replaced Big Data as the most hyped technology. Indeed, we're hearing more and more about this fascinating new technological paradigm. ...
Cultural, regulatory, environmental, political and economic (CREPE) conditions over the past decade are creating cross-industry solution spaces that require processes and technologies from both the Internet of Things (IoT), and Data Management and Analytics (DMA). These solution spaces are evolving into Sensor Analytics Ecosystems (SAE) that represent significant new opportunities for organizations of all types. Public Utilities throughout the world, providing electricity, natural gas and water,...
Software AG helps organizations transform into Digital Enterprises, so they can differentiate from competitors and better engage customers, partners and employees. Using the Software AG Suite, companies can close the gap between business and IT to create digital systems of differentiation that drive front-line agility. We offer four on-ramps to the Digital Enterprise: alignment through collaborative process analysis; transformation through portfolio management; agility through process automation...
The Internet of Things needs an entirely new security model, or does it? Can we save some old and tested controls for the latest emerging and different technology environments? In his session at Internet of @ThingsExpo, Davi Ottenheimer, EMC Senior Director of Trust, will review hands-on lessons with IoT devices and reveal privacy options and a new risk balance you might not expect.
The information technology sphere undergoes what we like to call a paradigm shift, sea change or plain old ‘upheaval’ roughly every five years or so. Don’t ask anybody why this half decade cyclicality exists; it just has to be so. Accept that reinvention happens constantly and that major seismic shifts are tangibly felt by us human beings roughly every 1826.21 days… and we can move on.
There’s Big Data, then there’s really Big Data from the Internet of Things. IoT is evolving to include many data possibilities like new types of event, log and network data. The volumes are enormous, generating tens of billions of logs per day, which raise data challenges. Early IoT deployments are relying heavily on both the cloud and managed service providers to navigate these challenges. In her session at 6th Big Data Expo®, Hannah Smalltree, Director at Treasure Data, to discuss how IoT, B...
Cloudwick, the leading big data DevOps service and solution provider to the Fortune 1000, announced Big Loop, its multi-vendor operations platform. Cloudwick Big Loop creates greater collaboration between Fortune 1000 IT staff, developers and their database management systems as well as big data vendors. This allows customers to comprehensively manage and oversee their entire infrastructure, which leads to more successful production cluster operations, and scale-out. Cloudwick Big Loop supports ...
SYS-CON Events announced today that Objectivity, Inc., the leader in real-time, complex Big Data solutions, will exhibit at SYS-CON's 15th International Cloud Expo®, which will take place on November 4–6, 2014, at the Santa Clara Convention Center in Santa Clara, CA. Objectivity, Inc. is the Enterprise Database leader of real-time, complex Big Data solutions. Our leading edge technologies – InfiniteGraph®, The Distributed Graph Database™ and Objectivity/DB®, a distributed and scalable object ma...
In their session at DevOps Summit, Stan Klimoff, CTO of Qubell, and Mike Becker, Senior Data Engineer for RingCentral, will share the lessons learned from implementing CI/CD pipeline on AWS for a customer analytics project powered by Cloudera Hadoop, HP Vertica and Tableau. Stan Klimoff is CTO of Qubell, the enterprise DevOps platform. Stan has more than a decade of experience building distributed systems for companies such as eBay, Cisco and Seagate. Qubell is helping enterprises to become mor...
The major cloud platforms defy a simple, side-by-side analysis. Each of the major IaaS public-cloud platforms offers their own unique strengths and functionality. Options for on-site private cloud are diverse as well, and must be designed and deployed while taking existing legacy architecture and infrastructure into account. Then the reality is that most enterprises are embarking on a hybrid cloud strategy and programs. In this Power Panel at 15th Cloud Expo, moderated by Ashar Baig, Research ...
The evolution of the database is under constant upheaval, discussion, debate and (if you will excuse the expression) 'analysis.' This basic truth is now more relevant, pertinent and pressing than ever due to the prevalence of Big Data (and the need to impose analytics of insight upon it) driven by social, mobile, cloud and of course the Internet of (Every) Things. Today then, as a staple of our IT infrastructure, databases have been around for over 50 years now with first references of the ter...
Big Data means many things to many people. From November 4-6 at the Santa Clara Convention Center, thousands of people will gather at Big Data Expo to discuss what it means to them, how they are implementing it, and how Big Data plays an integral role in the maturing cloud computing world and emerging Internet of Things. Attend Big Data Expo and make your contribution. Register for Big Data Expo "FREE" with Discount Code "BigDataOCTOBER" by October 31
SYS-CON Events announced today that Cloudian, Inc., the leading provider of hybrid cloud storage solutions, has been named “Bronze Sponsor” of SYS-CON's 15th International Cloud Expo®, which will take place on November 4–6, 2014, at the Santa Clara Convention Center in Santa Clara, CA. Cloudian is a Foster City, Calif.-based software company specializing in cloud storage. Cloudian HyperStore® is an S3-compatible cloud object storage platform that enables service providers and enterprises to bui...
What process has your provider undertaken to ensure that the cloud tenant will receive predictable performance and service? What was involved in the planning? Who owns and operates the data center? What technology is being used? How is it being supported? In his session at 14th Cloud Expo, Dave Weisbrot, Cloud Business Manager for QTS, will provide the attendees a look into what it takes to stand up and stand behind a highly available certified cloud IaaS.
Things are being built upon cloud foundations to transform organizations. This CEO Power Panel at 15th Cloud Expo, moderated by Roger Strukhoff, Cloud Expo and @ThingsExpo conference chair, will address the big issues involving these technologies and, more important, the results they will achieve. How important are public, private, and hybrid cloud to the enterprise? How does one define Big Data? And how is the IoT tying all this together?
Data efficiency – the combination of technologies including data deduplication, compression, zero elimination and thin provisioning – transformed the backup storage appliance market in well under a decade. Why has it taken so long for the same changes to occur in the primary storage appliance market? The answer can be found by looking back at the early evolution of the backup appliance market, and understanding why EMC’s Data Domain continues to hold a commanding lead in that market today. The ...
Target. Home Depot. Community Health Systems. Nieman Marcus. Their names have been all in the news over the past year, though probably not in a way they would like. All have had very public data breaches affecting anywhere from 350,000 (Nieman Marcus) to 4.5 million (Community Health Systems) customers. Add the recent high-profile celebrity nude photo hacking scandal and cloud security has become the trending topic in all the news and social media. Some of the discussions reminded me of a line f...