Click here to close now.


@BigDataExpo Authors: Carmen Gonzalez, Liz McMillan, Yeshim Deniz, Esmeralda Swartz, Dana Gardner

Related Topics: @BigDataExpo, Java IoT, Microservices Expo, Linux Containers, Log Management, @CloudExpo, SDN Journal

@BigDataExpo: Article

Using Big Data for Competitive Advantage

HP HAVEn CTO Mundada on new ways for businesses to gain transformation from big data and new wave analysis

Big data capabilities and advanced business analytics have now become essential to nearly any business development activity.

The benefits that enterprises can get if they can get their hands around big data analytics and apply it to business challenges are quickly being documented -- and they come as big new profits and major market advantages. Industries around the world are rapidly seeking transformational projects using big data to gain competitive advantage.

As part of the next edition of the HP Big Data Podcast Series, BriefingsDirect sat down with two HP executives to learn how these advanced analytics seekers can best accomplish their goals. The insights gleaned include how companies worldwide are best capturing myriad knowledge, gaining ever deeper analysis, and rapidly and securely making those insights available to more people on their own terms.

So join this executive-level discussion highlighting how the latest version of HP HAVEn produces new business analytics value and strategic return with Girish Mundada, Chief Technology Officer for HP HAVEn, and Dan Wood, Worldwide Solution Marketing Lead for Big Data at HP Software. The discussion is moderated by me, Dana Gardner, Principal Analyst at Interarbor Solutions.

Here are some excerpts:

Gardner: We’re in a fascinating time because analytics and big data are now top of mind. What was once relegated to a fairly small group of data scientists and analysts as reporting tools -- and I am thinking about business intelligence (BI) -- has really now become a comprehensive capability that’s proving essential to nearly any business strategy.

What’s behind this eagerness to gain big-data capabilities and exploit analytics so broadly?

Wood: We're starting to see some very clear quantification of the value and the benefits of big data. It’s fair to say that big data is probably the hottest topic in the industry.


There’s a lot of talk across all forms of media about big data right now, but what’s happened is that credible publications like the "Harvard Business Review," for example, have started to put solid numbers around the benefits that enterprises can get if they can get their hands around big-data analytics and apply it to business challenges.

For example, Harvard Business Review is saying that, on average, data-driven organizations will be five percent more productive and six percent more profitable than their competitors.

Worth chasing after

Think about that. A six-percent distinct profitability increase would double the stock price for a lot of organizations. So there really is a prize worth chasing after.

What we’re seeing, Dana, is much more widespread interest across the organization and not just within IT. We’re seeing line-of-business leaders understanding and, in many organizations, actually starting to benefit from big-data analytics.

They’re able to analyze the call logs in a call center, better understand the clickstreams on a website, and better understand how customers are using products. All of these are ways of analyzing large amounts of data and directly tying it to specific line-of-business problems.

That’s where we are right now. Industries around the world are going through transformational projects using big data to gain competitive advantage.

Gardner: It’s interesting too, Dan, that they’re not just taking these as individual data sets and handling them individually, but increasingly businesses are combining them, and finding new relationships, and doing things that they really couldn't have done before.

Wood: Absolutely. It’s the idea of 360-degree view of their internal operations, or of their external customer trends and needs -- and it’s come from combining data sets.

This industry label of big data is perhaps not the most helpful, because it’s not just the volume of data that is the challenge and the opportunity for the business.

For example, they’re combining social media analytics on customers with the call logs into the call center, with internal systems of record around the customer relationship management (CRM) and ongoing customer transactions. It’s by combining all those insights that the real big-data opportunity reveals itself.

Gardner: And the sources for those insights and data, of course, are across almost any type of information asset. It’s not a just structured data or data that your application standard is around -- it’s getting all the data all of the time.

Wood: That’s right. In some ways, this industry label of big data is perhaps not the most helpful, because it’s not just the volume of data that is the challenge and the opportunity for the business. It’s the variety of sources, as you’ve alluded to, and also the velocity at which that data is moving.

The business needs to get hold of these multiple sources of data and immediately be able to apply the analytics, get the insights, and make the business decisions. This is why still the vast majority of that data that’s available to an enterprise remains dark.

Unused and unexploited

It’s unused and unexploited. Organizations, with their traditional analytics systems, are struggling to get the meaning and insights from all these data types that we mentioned. These include unstructured information, such as social media sentiment, voice recordings, potentially even video recordings, and the structured and semi-structured things like log files and data center data. For many organizations, getting the information quickly enough out of their CRM and enterprise resource planning (ERP) systems is a challenge as well.

Gardner: So we see that there’s a great desire to do this, and there are great returns on being able to do this well. We talked about some of the general challenges. What specifically is holding people up?

Is this an issue of cost, complexity, or skills? Why aren’t companies able to move beyond this small fraction of the available information to which they could be applying such important insight and analytics?

Wood: It’s a complexity and a skills challenge, as you mentioned. The systems they have today, Dana, typically aren’t set up to able to analyze these vast amounts of unstructured information, and also to be able to analyze the structured data at a speed needed by the organization.

Typically, the analytic systems that organizations have had aren’t able to cope with that or with that unstructured human information.

Think about the need to analyze immediately a clickstream from an online shopping application or a pay-to-use application that an organization has. That is, a rapid-scale analysis of a large amount of structured data. Typically, the analytic systems that organizations have had aren’t able to cope with that or with the unstructured human information.

This is why HP has created the HAVEn Big Data Platform, and Girish will talk in more detail about this, and how it brings together the analytics engine needed to address these issues.

Just as importantly, there’s the ecosystem around HAVEn, which includes HP experts and services and services from partners, to bring together the skills needed to turn this data collection into useful information.

And there are skills around data scientists, as well -- skills around understanding the right questions the line of business needs to be asking, and understanding actually how to visualize and represent the data.

Gardner: What were the guiding principles that you were thinking of when HAVEn was being put together?

Talking to customers

Mundada: HAVEn came together not by creating it in a dark room somewhere in the back office. It came together by talking to customers. On a regular basis, I meet with some of HP's largest customers worldwide, getting input from them. And they're telling us what their current problems are.


Let me see if I can describe the landscape in a typical organization, and we can go from there. You'll see why we created HAVEn.

Let’s visualize four different waves of data. Back in early '60s,'70s, even part of the '80s, mainframes were the primary way to process data, and we used them for operationalizing certain parts of data processing, where data was extremely high-value. If you look at the cost of the systems, it was phenomenal.

Then came the next wave in the ‘80s, where we went into what I call client-server computing, and we already know several companies that were created in this space.

I’ve lived in Silicon Valley for almost 30 years now, and a whole bunch of new companies were born in this space. I worked for a company, Postgres, which became Illustra, then became Informix, and became IBM. If you look at that entire wave of OLTP technologies, we created data-processing technologies designed to solve basic business problems.

Application software was created: CRM, supplier relationship management (SRM), you name it. Many companies that did consulting around that were created, too. That was that second wave after the mainframe.

We’re talking about volumes that are growing exponentially. In the past, they were growing linearly.

Then came the third wave, where we took this data from all these transactional systems, brought them together to find out some basic analysis, which we now call business analytics, to find out "who is my most profitable customer, what are they buying, why are they buying," and things of that nature.

We created companies for that wave, too, and many technologies. Exadata, Teradata, Netezza, and a whole bunch of companies and applications were born in that space. That wave lasted for quite a while.

What we're seeing now is that from 2003 onward, something very fundamental has happened. At least, that’s the way I've been seeing this. If you look at the three Vs that Dan has described -- volume, velocity, and variety -- we’re talking about volumes that are growing exponentially. In the past, they were growing linearly. That creates a very different kind of requirement.

More importantly, if you look at the variety that Dan mentioned, that’s really the key driver in my mind. People are now routinely bringing in machine data, human data, and your traditional structured warehouses -- all of them together.

If you visualize a bar graph, you would see that 10 percent of the data that we now can monetize is coming from traditional sources, whereas 90 percent of the data that we need to monetize is now sitting in machine data and human data.

High velocity analytics

What we're trying to do with HAVEn is create a combined platform, where you can combine these three different data types and do very high-velocity analytics.

As a simple example, if you look at Apache Web Server logs, that data is used historically by the security people to see if anybody is breaking in. That data was being used by operational people to see if machines aren’t overloaded.

More importantly the digital marketing guys now want to look at that data to see who's coming to their website, what they’re buying, what they’re not buying, why they’re buying, and which geographies they’re coming from. Then, they want to combine all these data sets with their existing structured data to make sense out of it.

Today, it's a mess in the market. When we talk to our partners and customers, they’re saying that they have point solutions for each of these things, and if you want to combine that data, it’s really hard. That’s why we had to create HAVEn.

HAVEn is the fourth wave. HAVEn is specifically about big data, the fourth wave. If you look at HP’s portfolio, we sell products and services across each of these waves, and the fastest growing wave right now is the big-data wave. It’s growing at about 35 percent a year, according to Gartner, and that's why we're excited about it.

If you look at what’s required now to process big data in its entirety, one product no longer can do it all.

Gardner: Now we know why you created it and what it’s supposed to do. Tell us a little bit more about what’s included in HAVEn and why it is that you’ve been able combine product and platform to solve this very difficult task.

Mundada: If you look at what’s required now to process big data in its entirety, one product no longer can do it all. There is a very famous paper written by some university professors titled “One size does not fit all.” It proves that different data structures are able to solve different kinds of data problems far more efficiently.

One way to think about big data is to think of it as a pile of dirt. It’s a big pile. In that pile, there’s gold, silver, platinum, iron, and other metals you don’t even know. If the cost of mining that data is high, obviously you’re going to go after only the platinum and some known objects that you care about, because that’s all you can afford.

HAVEn is about bringing that cost of processing down to a very, very low level so you can go after more metals. That means you have to bring together a set of technologies to be able to solve this. If you look at the last three years, HP has made very significant amounts of investments in the big-data space.

Best of breed

We bought companies that were best of breed to try to solve specific problems. We bought Autonomy, Vertica, ArcSight, Fortify, TippingPoint, 3PAR Data, and Knightsbridge.

Now, we have a set of technologies to be able to combine them into a unique experience. Think of it almost like Microsoft Office. Before you had Microsoft Office, you would buy a word processor from one company, a spreadsheet from another company, and presentation software from a third company.

Let’s say you wanted to create a simple table. If you had created it in a word processor or even a spreadsheet, you couldn’t mix and match that. It was impossible to mix and match very different types.

Then, Microsoft came to the table and said, “Look, here’s a simplified solution.” If you want to create a table, go ahead and create it in PowerPoint. Or if you want to create more complicated thing, put it in Excel. Then, take that Excel and put it in PowerPoint. Or, you can put the whole thing into a Word document. That was the beauty of what Microsoft did.

We’re trying to do something similar for big data, make it very easy for people to combine all these different engines and the different data types and write simple applications on it.

Today you need to combine different sets of data techniques to solve different problems, and they have to work seamlessly.

Gardner: What beyond the products and binding them together makes HAVEn unique?

Mundada: HAVEn is really two different concepts. There’s the HAVEn data platform, which we’ll talk about now, and there’s a HAVEn ecosystem, which I’ll mention in a minute.

HAVEn means Hadoop, Autonomy, Vertica, Enterprise Security, and “n” applications. That’s the acronym. So let’s look at one of these pieces, and why we need an architecture like this.

As I said, today you need to combine different sets of data techniques to solve different problems, and they have to work seamlessly. That’s what we did with HAVEn. I’ve been with HAVEn from day zero, before the project concept started, and I can tell you why and how we added these pieces and how we’re trying to integrate them better.

If you look at Hadoop as an ecosystem part of that HAVEn, our story with Hadoop at HP is that Hadoop is an integral part of HAVEn. We see a lot of our customers and partners betting on Hadoop and we think it’s a good thing to keep Hadoop open and non-proprietary.

Leading vendors

We also today work with all leading Hadoop vendors, so we have shipping appliances as well as reference architectures for both Cloudera and Hortonworks, and we’re working now with MapR to create similar infrastructure. That’s our Hadoop’s story.

We’ve also found that our customers are saying they want some flexibility in Hadoop. Today, they may want one vendor, and tomorrow, they may decide to go to another vendor for whatever business reasons they choose. They want to know if we can provide a simple management tool that works across multiple Hadoop distributions.

As an example, we had to extend our Business Service Management (BSM) portfolio, so we can manage Hadoop, Vertica, hardware, storage, and networking all from within one environment. This is simply operationalizing it. Having a standardized set of hardware that matches multiple Hadoop distributions was another thing we had to do. There are many such enterprise-class innovations that you’ll see coming from HP.

But more than that, we also found that Hadoop is really good for certain kinds of applications today, and obviously, the community will extend that. You will see more and more innovations coming from that community and ecosystem.

It’s an analytic database, and by that, I mean the underlying algorithms are completely designed from the ground up.

Today, there are several areas where there are holes in Hadoop, or maybe they’re not as strong as commercial products. One such area that you see is SQL. The SQL phase of Hadoop is going to be one of the key differentiators across the different Hadoop packaging.

In that area, we have a technology called Vertica, which is the V part of HAVEn, and you’ll see companies like Facebook, using a combination of both Hadoop and Vertica.

The classic use case we see is that people will bring all kinds of raw data, put it into Hadoop, and do some batch processing there. Hadoop is great as a file system, a batch processing environment. But then they’ll take pieces of that data and want to do deep analytics on it, like a regression analytics, and they will put it into Vertica.

Vertica is, is an analytic database platform, and I will break up those three words. It’s a database. It looks and feels like a database. It has SQL on it, open database connectivity (ODBC), and Java database connectivity (JDBC) connectivity. You can run all kinds of tools on it, the ones you are used to, Tableau, Pentaho, and Informatica. So from that perspective it’s a regular database.

What’s different is that it’s custom built for the fourth wave. It’s an analytic database, and by that, I mean the underlying algorithms are completely designed from the ground up. Michael Stonebraker who created the key products in the first wave and the second wave -- Ingres and Postgres -- also created this at MIT from the ground up.

Data today

The intuition was that if you look at the processing of data today, it’s gone from having 10 to 20 columns per row to possibly thousands of columns. A social media company, for example, might have 10,000 pieces of information on me, and while they do processing, it’s going more linear. It’s going regression-oriented in a sense. You might say “Girish, age x, lives here, and likes y. What’s the likelihood somebody else may like it?”

It’s meant for that kind of deep analytical processing, a column-oriented structure. In those kinds of applications, this database technology tends to be magnitudes faster -- tens of times faster. That’s one example of Hadoop and Vertica, and we can talk more about other pieces Autonomy and Enterprise Security with you.

Gardner: So we see that there’s a platform that you put together. There’s an ecosystem that’s supporting that. There are these binding standards that make the ecosystem and the platform more synergistic. But other people are doing the same thing. What’s making HAVEn different? What is it about HAVEn that you think is going to be a winner in the marketplace?

Mundada: There are two different answers to it. Let me talk about how we’ve taken just not the SQL piece of Hadoop, but how we extend it with other parts of HP that are unique to HAVEn. It’s the breadth of it. Let’s see how we extend this simple combination of Hadoop and Vertica.

With Vertica, we’re able to drop in other codes that are user defined and user written.

I said it’s an analytic database platform. If you look at that platform piece of it, with Vertica, we’re able to drop in other code that are user-defined and user-written. For example, you can drop in R language routines, Java, C++, or C language routines directly into the database. Now, we’re now able to combine that richness across our portfolio.

Autonomy, which is the A part of HAVEn, is a unique technology. It's one of a kind. Some of the largest governments and some of the largest organizations in the world, such as banks and financial institutions, have this in production in what it's meant for, human information processing, which is audio, video, and text.

As an example, you could take a video stream and ask simple questions. Tell me if an object is moving from point A to point B, or tell me what’s in the object. Is it a human? Is it a car? Can you read car number plates automatically?

And you could do some really sophisticated applications. Taking a car, we have cases where police cars have video cameras mounted on the side, and as they’re driving by in a parking lot, they can take photos of the number plates and compare it to stolen cars.

Crime detection

Imagine being able to take that technology and combining it automatically, through simple SQL-like or simple REST API-like commands with SQL, with your existing data and creating very sophisticated applications to understand your customer or for crime detection and things like that?

Now let’s bring in the third of part of the puzzle, the E part, which is Enterprise Security. That’s also unique. We have an entire portfolio, both for security as well as for operations management.

If you look at enterprise security and if you look at the Gartner Magic Quadrant, HP’s product set has been in the leader space for several years in a row. They are the number one vendor in that area.

Now, think about our portfolio of ArcSight, Fortify, Tipping Point, and other ESP products. Imagine being able to take the data-collection algorithms of those, bringing it into this common platform of HAVEn, combining it with other structured and unstructured data with just simple commands. That’s something we can do uniquely.

Operations management is another area where we have hundreds of these machine logs. We can collect them, break them open into modular pieces, and create new applications. You can go look at our website, Operations Analytics, where with a simple slider, you can go back and forth in time to millions of log files as if they were structured data.

With simple SQL, we can essentially write simple queries across structured and unstructured data.

We can do that uniquely, because we have that entire collection. Our BSM portfolio has been on the market for 30 years. It’s one of the leaders. This is the HP OpenView platform and this is one of the things we can do uniquely at HP, bring all these things together.

That’s the breadth of our portfolio, but it simply doesn’t stop at this platform level. Remember, I said that there are two concepts. There is a platform, and then there is the ecosystem. Let’s look at the platform level first.

We have the whole of HAVEn. We have the connectors, and we ship these 700 connectors out of the box. With simple commands, you can bring in social-media data in every language written. You can bring in machine logs and structured logs. That’s the platform.

Let’s extend it further into the ecosystem part. The next thing that people were saying was, “We want to use something very open. We have our own visualization tools. We have our own extract, transform, load (ETL) tools that we’re used to. Can you just make them work?" And we said, "Sure.”

That’s one of the things that we’re able to do now. With simple SQL, we can essentially write simple queries across structured and unstructured data. Using Tableau Software, or any other tool that you like, we can access this data through our connectors, but, more importantly, it let’s you hook in your existing ETL tools into this -- completely transparently.

Breadth and openness

So that’s the openness of the platform, the breadth and the openness of the platform. Breadth is not just about the software platform, but it’s about HP’s strength to bring together hardware, software, and services.

Even with the platform, the HAVEn components in the middle, the connectors, and being able to match them with matching hardware, our customers are asking, “Can you give us matching hardware for Hadoop, so we don’t have to spend time setting it up?” That’s one of things that HP can uniquely do, but more importantly we have appliances for Vertica, for example, which are standardized.

If you look at the other side, our customers are also saying, “We understand that HP wants to provide us all this, but we like openness and we like other partners.” So we said, “Fine, we’ll leave this entire ecosystem open.” Our software will work with HP hardware and we can optimize, but we also commit to working on everybody else’s hardware.

If you look at our visualization, we didn’t go and force a visualization technology on you. We kept it open.

Our cloud story is that we’ll work on Amazon, as well as OpenStack. For example, if you want to build a hybrid cloud, where part of your data resides on HP or your private environment using OpenStack, that’s fine. If you want to put it in Amazon or Rackspace, no problem. We’ll help you bridge all these. These are the kinds of enterprise-cloud innovations that HP is able to do, and we’re open to this.

So to answer your question very succinctly, if there were three things I would pick where HP is different, one is our breadth of our portfolio. We have very large breadth that we've brought together.

It’s the openness of the platform. HP is known to be a very open company. If you look our Hadoop story, we have an example. We didn’t create a proprietary Hadoop. We kept it open. If you look at our virtualization, we didn’t go and force a virtualization technology on you. We kept it open.

More importantly, if there is one key thing that you want to take home from what we've done with HAVEn, it's not about feeds and not about speeds. It's about business value.

The reason we created HAVEn was to create that iPhone-like environment or Android-like environment, where the vision is that you should be able to go to a website, say you have standardized on the HAVEn platform, and then, be able to point and click and download an application.

The end part of HAVEn is really the business value of it, and that’s how we see HAVEn as unique. There is nobody else, as far as we know, that has that end-vision, where you can build the applications yourself using standard tools -- SQL, ODBC, REST API, JDBC -- or you can buy ready-made software that HP Software has created.

We have packages across service, operations, and digital marketing. Or you can go with a partner. The partner could be HP Enterprise Services, Accenture, Capgemini, or any of those big partners. That’s something unique about the HP big-data ecosystem that doesn’t exist anywhere else today.


Gardner: Applications are something that take advantage of the platform, the capabilities, the breadth and depth of the data, and information.

I wonder if you could explain a little bit more about the application side of HAVEn, perhaps through examples of what people are already doing with these applications, and how they’re using them in their business setting?

Mundada: That’s actually one of the most exciting parts of my job. As I said, I meet literally 100 customers a month. I'm traveling across the continents, and the use-cases of big data that I see are truly phenomenal. It really keeps you very motivated to keep doing more.

Let's look at a very broad level of why these things matter. Big data is not just about monetary profits. It's really about what I call extended profits. It doesn’t have to be monetary. If you look at a simple example, we have medical companies using data, using our technologies, to dramatically speed up drug discovery hundreds of times more than they were able to with Hadoop.

HAVEn isn’t about speeds and feeds. It's about really creating business value in a hurry, so you get there before your competitors can.

That translates into just saving lives. At our recent Discover show in Barcelona, we saw that a very innovative organization is using our technology to look at bio-diversity and save wildlife in the Amazon.

That’s unique, but those are like edge cases. If you look at a regular enterprise, what they want to do at a very high level falls into three categories: Applications that HP itself is building, applications that partners are building, and applications that customers themselves are building.

There are three applications I’ll mention. In terms of increasing revenue, we have a product that we ship called Digital Marketing Hub, and it combines the power of Autonomy and Vertica to analyze all of your customer analytics.

You’re able to take your call center logs, your social media feeds, your emails, your phone interactions and find out what the customer is really is saying, what they want and don't want, and then, being able to optimize that interaction with the customer to create more revenue.

More precise answers

For example, when a customer calls knowing what they want, obviously you can tell them more precise things. That’s one example.

Let's look at another example, where you want to decrease your bottom line or decrease your costs. Operational Analytics is another software product we ship. We’re able to drive down costs of debugging network troubles by 80 percent by combining all these logs from machines on a very frequent basis.

We can look at this and say. "At this second, every machine was okay. A second later, machines have gone down." I can look exactly at the incremental logs that showed up, using a simple pen like a pointer, going through SQL-like data. That’s unique.

Those are the kinds of applications we’re able to create. It's not just these two. The other thing people want is improve products and services. We have something called Service Anywhere, where as you're calling or as you're typing in commands and saying you want to find information about that, the system is able to understand the meaning of what you’re saying.

Notice that this is not keyword search. This is meaning, where it's able to go through existing case reports from customers, look at existing resolutions, and then say, “Okay, this might solve your problem automatically.”

That’s the beauty of the HAVEn platform. On the same platform, you can buy HP built applications or you can build your own.

Imagine what that impacts. Your customers are happy, because the answers are quicker. We call this ticketless ID, but more important, look at some other interesting ways of how this affects a company.

For example, I was recently in Europe. I was talking to a very large telco there, and they said, “We have something like 20,000 call-center operators who are taking calls from customers. Each call volume might take six minutes and some of them are repeat calls. That’s really our problem.”

We worked out something that roughly could save them two minutes per call. That translates to about a $100 million net saving per year. That’s really phenomenal. Those are one kind of application that HP built.

Now imagine a customer wanting to build the same application themselves. That’s the beauty of the HAVEn platform. On the same platform, you can buy HP built applications or you can build your own.

Let's look at NASCAR as an example. They did something very similar for customer analytics. They are able to -- while the race is happening -- understand audio, television channels, radio, broadcast, and social media and bring that all together as if it's one unique piece of data.

Then, they’re able to use that data in really innovative ways to further their sport and to create more promotional dollars for just not themselves, but even the participants. That’s unique -- being able to analyze mass scale human data.

Looking to the future

Gardner: Well, we've learned a lot about the market, the demand, why big data makes so much sense. There is very large undertaking by HP around HAVEn, and what it’s getting in terms of openness, platforms, breadth, and these great examples of applications. But we also need to look to the future.

What's coming next in terms of HAVEn 2.0 or HAVEn 1.5? Dan, could you update us on how things are progressing, what you have in mind for the next versions of these products and, therefore, the whole increasing as sum of the parts increases?

Wood: Dana, we've just announced HAVEn 2.0. The way Girish explained HAVEn there in terms of the platform and the ecosystem and continuous innovation now is around both of those pieces. It's really important to us to be driving the ecosystem, as well as the platform. So I’ll speak to HAVEn 2.0 and one of the feature that’s the focus in driving HP forward.

In terms of the platform, there are the analytics engines that we have. Girish mentioned they were best in class at the time that HP acquired them, and we continue to invest in R and D across Autonomy, IDOL, Vertica, and the ArcSight Logger product. We recently announced new versions of all three of those, improving the analytics capability and the usability and, just as importantly, increasing the interoperability.

At the moment, on an early-access program, we’re making the IDOL engine available to developers as a cloud-based offering.

For example, we now have integration of the ArcSight Logger with the Autonomy IDOL engine for analyzing unstructured human information. A really great use case of this is Logger was previously enabling IT to understand data movements and potential threats and the risks in the organization.

For example, if I were sending 50 percent of my email to a competitor, you could combine that capability with the unstructured information analysis in Autonomy and understand by that the information layer exactly what’s in that email, 50 percent of which is going to a competitor.

Let’s start putting that together and getting a powerful view of what an individual is doing and whether it’s a risky individual in the organization, integrating those HAVEn engines and putting more effort on integrating it into the Hadoop environment as well.

For example, we have just announced integration Hadoop connectors for Autonomy. A lot of people are saying that they’re building this data lake with Hadoop and they want to have the capability of putting some analytics into the unstructured information that exists in that Hadoop data lake. Clearly, we’ve also got integration with Vertica in the Hadoop environment as well.

The other key thing within that on the engine is IDOL OnDemand. At the moment, on an early-access program, we’re making the IDOL engine available to developers as a cloud-based offering. This is to encourage the independent developer community to take components of IDOL with that social media analytics, whether it’s video or audio recognition, and start building that into their own applications.

We believe the power of HAVEn will come from the combination of HP-provided applications and also third-party applications on top.

Early-access program

We’re facilitating that with this initial early-access program on IDOL OnDemand, and also, we’re investing in developer programs to make the whole HAVEn development platform far easier for partners and independent developers to work with.

We’ve set up a HAVEn developer website, and stay tuned for some really fun events online and physical events, where we’ll be getting the developer community together.

In terms of those applications that make the whole HAVEn ecosystem come to life, Girish has mentioned some of them that we have announced over the last few weeks. So I’ll give you a quick recap on those.

We have the Operations Analytics and Service Anywhere apps, both aimed at the CIO. And we have the Digital Marketing Hub from HP aimed at marketing leaders in the organizations. These are three applications that HP has packaged on the HAVEn platform.

And along with the HAVEn 2.0 announcement, we’re really pleased that six of the leading SI partners -- Accenture, Capgemini, Deloitte, PwC, Accenture and Wipro -- themselves have put marketing applications on top of HAVEn. And those guys have gotten fascinating mixtures of very industry-specific analytics applications and more horizontal apps based on the priorities that they’re chasing after.

We’re populating our solutions and partner solutions to facilitate the whole commerce side of those applications taking off in the market.

So we’re really excited about that and expect to see many more announcements of partner applications over the next few months.

The final piece of HAVEn 2.0 to support this whole ecosystem thing is a marketplace that we’ve launched, where we’re populating our solutions and partner solutions to facilitate the whole commerce side of those applications taking off in the market.

One-stop resource

The first place to go is That’s your one-stop resource for information on this platform, all of the engines that Girish alluded to. You can get the inspiration from some amazing customer case studies we have on there -- insights from experts like Girish and other people who are talking in depth about the individual engines.

And as you rightly say, Dana, it’s finding the right on-ramp for yourself.  You can look at the case studies we have, the use cases on big data in particular industries, and take a look at what the specific pain point you have today. That’s the website, and that gives you all of that information.

You can also drill down from there, if you're a developer, and find the tools and resources that we’ve spoken about to enable you to start building apps on top of HAVEn. That’s one part.

The whole power of HP behind this HAVEn platform is in enabling, from an infrastructure and services point of view, to start building these big data analytics. A couple of key things here.

We started to build fully configured appliances around Hadoop and Vertica. So the Converged System’s team in HP has launched the ConvergedSystem 300, which enables you to have Vertica and Hadoop on a pre-configured appliance. That’s a great starting point for someone early on in the big-data analytics life cycle.

Those guys have data scientists and industry experts who can actually help customers go through the design phase for a big-data platform.

To expand on that, the Technology Services team is able to do full consulting on how to optimize the overall infrastructure from the point of view of processing, sharing, and storing this vast amount of information that all organizations are coping with today. That will then start to put in things like 3PAR storage systems and other innovations across the HP hardware business.

Another place where I see customers often needing some help to get started is in understanding exactly what the questions are that we need to be asking in terms of analytics and exactly what algorithms and analytics we need to put in place to get going. This is where the Big Data Discovery Experience Services from HP come in.

This is provided by the Enterprise Services Group (ESG). Those guys have data scientists and industry experts who can actually help customers go through the design phase for a big-data platform and than offer the HAVEn infrastructure supported by the ESG Services team.

Finally, Dana, come and see us on the road. We’ll be at HP Discover in Las Vegas June 10-12. We’re putting together several road shows and events across the main regions in Europe, the Americas, and in Asia Pacific, where we will be taking HAVEn on the road, too. Take a look at that website, and details of the events will be found on there.

Key messages

Mundada: There are two key messages: big data is really important and it’s disrupting business. Your competitors are going to do it. You have a choice to either lead and do it yourself or you will be forced to follow. It’s one of those things that are disrupting industries worldwide.

Now, when you think of big data, don’t think of pieces and don’t think of piece parts. It’s not like you need a separate solution for human information, another for machine logs, and another for structured data. You almost have to think of it holistically, because there are many kinds of newer applications that I’m seeing regularly, where you have to bring all these data types together and create joint applications.

Whichever technologies that you choose and settle on, think of that Microsoft Office-like experience. You want to combine integrated solution across the entire stack and there aren’t that many available in the market today. So whoever you work with, make sure that you’re able to handle that entire piece as one giant puzzle.

You may also be interested in:

More Stories By Dana Gardner

At Interarbor Solutions, we create the analysis and in-depth podcasts on enterprise software and cloud trends that help fuel the social media revolution. As a veteran IT analyst, Dana Gardner moderates discussions and interviews get to the meat of the hottest technology topics. We define and forecast the business productivity effects of enterprise infrastructure, SOA and cloud advances. Our social media vehicles become conversational platforms, powerfully distributed via the BriefingsDirect Network of online media partners like ZDNet and As founder and principal analyst at Interarbor Solutions, Dana Gardner created BriefingsDirect to give online readers and listeners in-depth and direct access to the brightest thought leaders on IT. Our twice-monthly BriefingsDirect Analyst Insights Edition podcasts examine the latest IT news with a panel of analysts and guests. Our sponsored discussions provide a unique, deep-dive focus on specific industry problems and the latest solutions. This podcast equivalent of an analyst briefing session -- made available as a podcast/transcript/blog to any interested viewer and search engine seeker -- breaks the mold on closed knowledge. These informational podcasts jump-start conversational evangelism, drive traffic to lead generation campaigns, and produce strong SEO returns. Interarbor Solutions provides fresh and creative thinking on IT, SOA, cloud and social media strategies based on the power of thoughtful content, made freely and easily available to proactive seekers of insights and information. As a result, marketers and branding professionals can communicate inexpensively with self-qualifiying readers/listeners in discreet market segments. BriefingsDirect podcasts hosted by Dana Gardner: Full turnkey planning, moderatiing, producing, hosting, and distribution via blogs and IT media partners of essential IT knowledge and understanding.

@BigDataExpo Stories
SYS-CON Events announced today that Super Micro Computer, Inc., a global leader in high-performance, high-efficiency server, storage technology and green computing, will exhibit at the 17th International Cloud Expo®, which will take place on November 3–5, 2015, at the Santa Clara Convention Center in Santa Clara, CA. Supermicro (NASDAQ: SMCI), the leading innovator in high-performance, high-efficiency server technology is a premier provider of advanced server Building Block Solutions® for Data ...
SYS-CON Events announced today the Containers & Microservices Bootcamp, being held November 3-4, 2015, in conjunction with 17th Cloud Expo, @ThingsExpo, and @DevOpsSummit at the Santa Clara Convention Center in Santa Clara, CA. This is your chance to get started with the latest technology in the industry. Combined with real-world scenarios and use cases, the Containers and Microservices Bootcamp, led by Janakiram MSV, a Microsoft Regional Director, will include presentations as well as hands-on...
As more intelligent IoT applications shift into gear, they’re merging into the ever-increasing traffic flow of the Internet. It won’t be long before we experience bottlenecks, as IoT traffic peaks during rush hours. Organizations that are unprepared will find themselves by the side of the road unable to cross back into the fast lane. As billions of new devices begin to communicate and exchange data – will your infrastructure be scalable enough to handle this new interconnected world?
SYS-CON Events announced today that Dyn, the worldwide leader in Internet Performance, will exhibit at SYS-CON's 17th International Cloud Expo®, which will take place on November 3-5, 2015, at the Santa Clara Convention Center in Santa Clara, CA. Dyn is a cloud-based Internet Performance company. Dyn helps companies monitor, control, and optimize online infrastructure for an exceptional end-user experience. Through a world-class network and unrivaled, objective intelligence into Internet condit...
SYS-CON Events announced today that Sandy Carter, IBM General Manager Cloud Ecosystem and Developers, and a Social Business Evangelist, will keynote at the 17th International Cloud Expo®, which will take place on November 3–5, 2015, at the Santa Clara Convention Center in Santa Clara, CA.
The Internet of Things (IoT) is growing rapidly by extending current technologies, products and networks. By 2020, Cisco estimates there will be 50 billion connected devices. Gartner has forecast revenues of over $300 billion, just to IoT suppliers. Now is the time to figure out how you’ll make money – not just create innovative products. With hundreds of new products and companies jumping into the IoT fray every month, there’s no shortage of innovation. Despite this, McKinsey/VisionMobile data...
The IoT market is on track to hit $7.1 trillion in 2020. The reality is that only a handful of companies are ready for this massive demand. There are a lot of barriers, paint points, traps, and hidden roadblocks. How can we deal with these issues and challenges? The paradigm has changed. Old-style ad-hoc trial-and-error ways will certainly lead you to the dead end. What is mandatory is an overarching and adaptive approach to effectively handle the rapid changes and exponential growth.
With major technology companies and startups seriously embracing IoT strategies, now is the perfect time to attend @ThingsExpo in Silicon Valley. Learn what is going on, contribute to the discussions, and ensure that your enterprise is as "IoT-Ready" as it can be! Internet of @ThingsExpo, taking place Nov 3-5, 2015, at the Santa Clara Convention Center in Santa Clara, CA, is co-located with 17th Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading in...
There will be 20 billion IoT devices connected to the Internet soon. What if we could control these devices with our voice, mind, or gestures? What if we could teach these devices how to talk to each other? What if these devices could learn how to interact with us (and each other) to make our lives better? What if Jarvis was real? How can I gain these super powers? In his session at 17th Cloud Expo, Chris Matthieu, co-founder and CTO of Octoblu, will show you!
The IoT is upon us, but today’s databases, built on 30-year-old math, require multiple platforms to create a single solution. Data demands of the IoT require Big Data systems that can handle ingest, transactions and analytics concurrently adapting to varied situations as they occur, with speed at scale. In his session at @ThingsExpo, Chad Jones, chief strategy officer at Deep Information Sciences, will look differently at IoT data so enterprises can fully leverage their IoT potential. He’ll sha...
The enterprise is being consumerized, and the consumer is being enterprised. Moore's Law does not matter anymore, the future belongs to business virtualization powered by invisible service architecture, powered by hyperscale and hyperconvergence, and facilitated by vertical streaming and horizontal scaling and consolidation. Both buyers and sellers want instant results, and from paperwork to paperless to mindless is the ultimate goal for any seamless transaction. The sweetest sweet spot in innov...
Today air travel is a minefield of delays, hassles and customer disappointment. Airlines struggle to revitalize the experience. GE and M2Mi will demonstrate practical examples of how IoT solutions are helping airlines bring back personalization, reduce trip time and improve reliability. In their session at @ThingsExpo, Shyam Varan Nath, Principal Architect with GE, and Dr. Sarah Cooper, M2Mi's VP Business Development and Engineering, will explore the IoT cloud-based platform technologies driv...
The Internet of Everything is re-shaping technology trends–moving away from “request/response” architecture to an “always-on” Streaming Web where data is in constant motion and secure, reliable communication is an absolute necessity. As more and more THINGS go online, the challenges that developers will need to address will only increase exponentially. In his session at @ThingsExpo, Todd Greene, Founder & CEO of PubNub, will explore the current state of IoT connectivity and review key trends an...
SYS-CON Events announced today that DataClear Inc. will exhibit at the 17th International Cloud Expo®, which will take place on November 3–5, 2015, at the Santa Clara Convention Center in Santa Clara, CA. The DataClear ‘BlackBox’ is the only solution that moves your PC, browsing and data out of the United States and away from prying (and spying) eyes. Its solution automatically builds you a clean, on-demand, virus free, new virtual cloud based PC outside of the United States, and wipes it clean...
SYS-CON Events announced today that Machkey International Company will exhibit at the 17th International Cloud Expo®, which will take place on November 3–5, 2015, at the Santa Clara Convention Center in Santa Clara, CA. Machkey provides advanced connectivity solutions for just about everyone. Businesses or individuals, Machkey is dedicated to provide high-quality and cost-effective products to meet all your needs.
Nowadays, a large number of sensors and devices are connected to the network. Leading-edge IoT technologies integrate various types of sensor data to create a new value for several business decision scenarios. The transparent cloud is a model of a new IoT emergence service platform. Many service providers store and access various types of sensor data in order to create and find out new business values by integrating such data.
Too often with compelling new technologies market participants become overly enamored with that attractiveness of the technology and neglect underlying business drivers. This tendency, what some call the “newest shiny object syndrome,” is understandable given that virtually all of us are heavily engaged in technology. But it is also mistaken. Without concrete business cases driving its deployment, IoT, like many other technologies before it, will fade into obscurity.
There are so many tools and techniques for data analytics that even for a data scientist the choices, possible systems, and even the types of data can be daunting. In his session at @ThingsExpo, Chris Harrold, Global CTO for Big Data Solutions for EMC Corporation, will show how to perform a simple, but meaningful analysis of social sentiment data using freely available tools that take only minutes to download and install. Participants will get the download information, scripts, and complete en...
Achim Weiss is Chief Executive Officer and co-founder of ProfitBricks. In 1995, he broke off his studies to co-found the web hosting company "Schlund+Partner." The company "Schlund+Partner" later became the 1&1 web hosting product line. From 1995 to 2008, he was the technical director for several important projects: the largest web hosting platform in the world, the second largest DSL platform, a video on-demand delivery network, the largest eMail backend in Europe, and a universal billing syste...
Cloud computing delivers on-demand resources that provide businesses with flexibility and cost-savings. The challenge in moving workloads to the cloud has been the cost and complexity of ensuring the initial and ongoing security and regulatory (PCI, HIPAA, FFIEC) compliance across private and public clouds. Manual security compliance is slow, prone to human error, and represents over 50% of the cost of managing cloud applications. Determining how to automate cloud security compliance is critical...

Tweets by @BigDataExpo

@BigDataExpo Blogs
This week, the team assembled in NYC for @Cloud Expo 2015 and @ThingsExpo 2015. For the past four years, this has been a must-attend event for MetraTech. We were happy to once again join industry visionaries, colleagues, customers and even competitors to share and explore the ways in which the Internet of Things (IoT) will impact our industry. Over the course of the show, we discussed the types of challenges we will collectively need to solve to capitalize on the opportunity IoT presents.
Today’s connected world is moving from devices towards things, what this means is that by using increasingly low cost sensors embedded in devices we can create many new use cases. These span across use cases in cities, vehicles, home, offices, factories, retail environments, worksites, health, logistics, and health. These use cases rely on ubiquitous connectivity and generate massive amounts of data at scale. These technologies enable new business opportunities, ways to optimize and automate, along with new ways to engage with users.
Dasher Technologies is helping to usher in the democratization of big data value to more players in less time with analytics in a cloud services model
Heat maps offer a unique way to represent data sets in a variety of settings. A common example you see most mornings is through weather reports on the news showing the movement and predictions of pressure and precipitation, intensifying across bold color schemes. Another familiar example is voting representation in specific areas of the country during election times (think red state vs. blue state). Retail stores utilize heat maps to enable managers and executives to better understand the functionality of their space, as well as the habits and preferences of the people entering the store.
Developing software for the Internet of Things (IoT) comes with its own set of challenges. Security, privacy, and unified standards are a few key issues. In addition, each IoT product is comprised of at least three separate application components: the software embedded in the device, the backend big-data service, and the mobile application for the end user's controls. Each component is developed by a different team, using different technologies and practices, and deployed to a different stack/target - this makes the integration of these separate pipelines and the coordination of software upd...
“Cloud computing is a model for enabling ubiquitous, convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications and services) that can be rapidly provisioned and released with minimal management.” While this definition is broadly accepted and has, in fact, been my adopted standard for years, it only describes technical aspects of cloud computing. The amalgamation of technologies used to deliver cloud services is not even half the story. Above all else, the successful employment requires a tight linkage to the econ...
Today air travel is a minefield of delays, hassles and customer disappointment. Airlines struggle to revitalize the experience. GE and M2Mi will demonstrate practical examples of how IoT solutions are helping airlines bring back personalization, reduce trip time and improve reliability. In their session at @ThingsExpo, Shyam Varan Nath, Principal Architect with GE, and Dr. Sarah Cooper, M2Mi's VP Business Development and Engineering, will explore the IoT cloud-based platform technologies driving this change including privacy controls, data transparency and integration of real time context w...
DevOps Summit at Cloud Expo 2014 Silicon Valley was a terrific event for us. The Qubell booth was crowded on all three days. We ran demos every 30 minutes with folks lining up to get a seat and usually standing around. It was great to meet and talk to over 500 people! My keynote was well received and so was Stan's joint presentation with RingCentral on Devops for BigData. I also participated in two Power Panels – ‘Women in Technology’ and ‘Why DevOps Is Even More Important than You Think,’ both featuring brilliant colleagues and moderators and it was a blast to be a part of.
All we need to do is have our teams self-organize, and behold! Emergent design and/or architecture springs up out of the nothingness! If only it were that easy, right? I follow in the footsteps of so many people who have long wondered at the meanings of such simple words, as though they were dogma from on high. Emerge? Self-organizing? Profound, to be sure. But what do we really make of this sentence?
The potential of big data is only limited by the creative thinking of your business stakeholders, and that may be the most important concept in the “thinking like a data scientist” process. The “thinking like a data scientist” process guides the business stakeholders into envisioning how big data can optimize their key business processes, create a more compelling customer engagement and uncover new monetization opportunities. But neither the business stakeholders, nor the data scientists, can likely do that envisioning entirely by themselves.
Today’s modern day industrial revolution is being shaped by ubiquitous connectivity, machine to machine (M2M) communications, the Internet of Things (IoT), open APIs leading to a surge in new applications and services, partnerships and eventual marketplaces. IoT has the potential to transform industry and society much like advances in steam technology, transportation, mass production and communications ushered in the industrial revolution in the 18th and 19th centuries.
Disaster recovery (DR) has traditionally been a major challenge for IT departments. Even with the advent of server virtualization and other technologies that have simplified DR implementation and some aspects of on-going management, it is still a complex and (often extremely) costly undertaking. For those applications that do not require high availability, but are still mission- and business-critical, the decision as to which [applications] to spend money on for true disaster recovery can be a struggle.
SCOPE is an acronym for Structured Computations Optimized for Parallel Execution, a declarative language for working with large-scale data. It is still under development at Microsoft. If you know SQL then working with SCOPE will be quite easy as SCOPE builds on SQL. The execution environment is different from that RDBMS oriented data. Data is still modeled as rows. Every row has typed columns and eveyr rowset has a well-defined schema. There is a SCOPe compiler that comes up with optimized execution plan and a runtime execution plan.
If you’re running Big Data applications, you’re going to want to look at some kind of distributed processing system. Hadoop is one of the best-known clustering systems, but how are you going to process all your data in a reasonable time frame? MapReduce has become a standard, perhaps the standard, for distributed file systems. While it’s a great system already, it’s really geared toward batch use, with jobs needing to queue for later output. This can severely hamper your flexibility. What if you want to explore some of your data? If it’s going to take all night, forget about it.
Too many multinational corporations delete little, if any, data even though at its creation, more than 70 percent of this data is useless for business, regulatory or legal reasons.[1] The problem is hoarding, and what businesses need is their own “Hoarders” reality show about people whose lives are driven by their stuff[2] (corporations are legally people, after all). The goal of such an intervention (and this article)? Turning hoarders into collectors.

About @BigDataExpo
Big Data focuses on how to use your own enterprise data – processed in the Cloud – most effectively to drive value for your business.