Welcome!

Big Data Journal Authors: Scott Bampton, Carmen Gonzalez, Liz McMillan, Roger Strukhoff, Pat Romanski

Blog Feed Post

CTOvision Big Data Reporting for 2012: CTOs want discipline in the language of sensemaking

By

big-data-620x400This special report provides insights from a our reporting over the last 12 months, including summaries of our Government Big Data Newsletter (sign up for this weekly report here http://ctovision.com/newsletter-subscriptions)

Among the many Big Data themes we reported on in 2012, one seemed to resonate the most with our readers– all of us with a techie bent have realized that we need more discipline in our use of the term Big Data. We revisited this need for discipline in our post of:

Big Data Defined for 2013: A definition that can help in your interaction with the IT community

In it we suggest everyone follow the lead of the TechAmerica foundation in defining Big Data. At CTOvision we will use the term this way:

Big Data: A phenomenon defined by the rapid acceleration in the expanding volume of high velocity, complex and diverse types of data. Big Data is often defined along three dimensions– volume, velocity and variety.

Big Data Solutions: Advanced techniques and technologies to enable the capture, storage, distribution, management and analysis of information.

Early in the year we provided insights for program managers that want to get a started with Big Data solutions. We gave quickstart tips on how you can stand up your own cluster in the cloud. We followed up with ways you can quickly use Whirr to automate that.

Through the year we published several pieces on topics associated with the ethics issues around Big Data. This included a series by Kord Davis who reported on topics like:

We reported extensively on new concepts for Big Data involving very large quantities of data in memory. The greatest expert in this field, Terracotta CEO Robin Gilthorpe, provided his views on Big Data Trends to watch in 2013 by a YouTube video we highlighted to our readers. His view is that requirements will drive the industry to several new highs and will include dramatic social change because of this. His five predictions for 2013 are:

  • Big Data will be fast data – Enterprises will profit from Big Data intelligence in proportion to how quickly they can act on it.
  • Rise of the hybrid cloud – It’s no longer about building your own platform; it’s more efficient to play in ecosystems.
  • CIOs and CMOs get a lot closer – Marketing spend on technology is about to eclipse IT spend on technology.
  • The Internet of things crosses the chasm – In just a few years, over 25 billion data-producing devices will be connected.
  • Social becomes part of life’s fabric – Remember e-business departments? Social will permeate in the same way.

We also wrote about new concepts for capture, storage, distribution and management of data via new concepts like dispersed compute storage. Solutions like this from Cleversafe (see Cleversafe: how does it really work?) are true game changers inserting dramatic improvements to security and functionality and doing so with a quick return on investment.

We reported on many other firms associated with the fielding of high quality Big Data solutions into the federal enterprise, including MarkLogic, Oracle, Datameer, Cloudera, Terracotta, Cleversafe, Splunk, Kapow, Sitscape, CloudFrontGroup, ClearStory, and Thetus. These firms are fielding real, working solutions for Big Data and we will be reporting more on them in 2013 we are sure.

Another clear theme in our reporting of 2012 on Big Data was the importance of mission focus. That is why we are all so excited about the new technical capabilities of Hadoop and the related technologies. It is about impact to mission. Which leads to the Government Big Data Solutions Award:

Our reporting on Big Data for 2012 included announcing the results of the Government Big Data Solutions Award. The Government Big Data Solutions Award was established to highlight innovative solutions and facilitate the exchange of best practices, lessons learned and creative ideas for addressing Big Data challenges. The Top Five Nominees of 2012 were chosen for criteria that included:

  • Focus on current solutions: The ability to make a difference in government missions in the very near term was the most important evaluation factor.
  • Focus on government teams: Industry supporting government also considered, but this is about government missions.
  • Consideration of new approaches: New business processes, techniques, tools, models for enhancing analysis are key.

Winner of the 2012 Government Big Data Solutions Award was the National Cancer Institute’s Frederick National Laboratory.

The NCI Funded Frederick National Laboratory has been using Big Data solutions in pioneering ways to support researchers working on complex challenges around the relationship between genes and cancers. In a  recent example, they have built infrastructure capable of cross-referencing the relationships between 17000 genes and five major cancer subtypes across 20 million biomedical publication abstracts.  By cross referencing TCGA gene expression data from simulated 60 million patients and miRNA expression for a simulated 900 million patients. The result: understanding additional layers of the pathways these genes operate in and the drugs that target them. This will help researchers accelerate their work in areas of importance for all humanity.  This solution, based on the Oracle Big Data Appliance with the Cloudera Distribution of Apache Hadoop (CDH), leverages capabilities available from the Big Data community today in pioneering ways that can serve a broad range of researchers. The promising approach of this solution is repeatable across many other Big Data challenges for bioinfomatics, making this approach worthy of its selection as the 2012 Government Big Data Solution Award.

We also reported on a classification framework for Big Data solutions produced by  in a very insightful post on Classifying Today’s “Big Data Innovators”.  This is an innovative approach that is easy to think through and should be repeatable for many vendors in this space, and should help enterprise technologists think through which vendors may be right for their mission needs.  In it he categorizes the 13 innovative Big Data innovators reported on by Information Week. They are:

1.  MongoDB
2.  Amazon (Redshift, EMR, DynamoDB)
3.  Cloudera (CDH, Impala)
4.  Couchbase
5.  Datameer
6.  Datastax
7.  Hadapt
8.  Hortonworks
9.  Karmasphere
10.  MapR
11.  Neo Technology
12.  Platfora
13.  Splunk

He classifies them into:

1.  Operational data stores that allow flexible schemas
2.  Hadoop distributions
3.  Real-time Hadoop-based analytical platforms
4.  Hadoop-based BI solutions

We will likely return to this classification for reporting in 2013.

What does our reporting over the last 12 months signal for the next 12 months? We believe we will see a continued expansion of the user end of big data solutions. It is probably an oversimplification to say it this way, but one way to look at is is that we have an approach to the backend infrastructure, and that is primarily one built on the Apache Hadoop framework of software over commodity IT integrated into existing but modern enterprise solutions. Their is room for innovation here of course but in general the path of the backend is set and will continue. The dynamic change to expect now is in the user-facing applications. Brace yourself! Changes there will be dynamic.

For reports on Big Data throughout 2013 please sign up for our Government Big Data Newsletter. Find the weekly report at:  http://ctovision.com/newsletter-subscriptions/

Read the original blog entry...

More Stories By Bob Gourley

Bob Gourley, former CTO of the Defense Intelligence Agency (DIA), is Founder and CTO of Crucial Point LLC, a technology research and advisory firm providing fact based technology reviews in support of venture capital, private equity and emerging technology firms. He has extensive industry experience in intelligence and security and was awarded an intelligence community meritorious achievement award by AFCEA in 2008, and has also been recognized as an Infoworld Top 25 CTO and as one of the most fascinating communicators in Government IT by GovFresh.

@BigDataExpo Stories
SYS-CON Events announced today that O'Reilly Media has been named “Media Sponsor” of SYS-CON's 15th International Cloud Expo®, which will take place on November 4–6, 2014, at the Santa Clara Convention Center in Santa Clara, CA. O'Reilly Media spreads the knowledge of innovators through its books, online services, magazines, and conferences. Since 1978, O'Reilly Media has been a chronicler and catalyst of cutting-edge development, homing in on the technology trends that really matter and spurri...
Samsung promises to be one of the 800-pound gorillas of the IoT, if its success in recent years with Android devices and other consumer electronics is any guide. Showing its willingness to be a big IoT player, the company recently acquired SmartThings, a recent startup that's developed an open smarthome appliation that currently supports 1,000 devices and 8,000 apps. SmartThings will now work under the auspices of Samsung's Open Innovation Center (OIC). SmartThings Founder and CEO Alex Hawkinson...
The Internet of Things (IoT) is going to require a new way of thinking and of developing software for speed, security and innovation. This requires IT leaders to balance business as usual while anticipating for the next market and technology trends. Cloud provides the right IT asset portfolio to help today’s IT leaders manage the old and prepare for the new. Today the cloud conversation is evolving from private and public to hybrid. This session will provide use cases and insights to reinforce t...
What process has your provider undertaken to ensure that the cloud tenant will receive predictable performance and service? What was involved in the planning? Who owns and operates the data center? What technology is being used? How is it being supported? In his session at 14th Cloud Expo, Dave Weisbrot, Cloud Business Manager for QTS, will provide the attendees a look into what it takes to stand up and stand behind a highly available certified cloud IaaS.
I'll be hosting an SAP HANA Cloud webinar at 11am eastern time, Wednesday, October 29. You can sign up now. Featured speakers will be Allan Adler, Managing Partner, Channel Cloud Consulting, and Thorsten Leiduck, VP ISVs & Digital Commerce, SAP. Attendees will learn about • Cloud economics, hybrid cloud strategy, market size and opportunity • Introduction to SAP HANA Cloud Platform and how to: - Build new next-generation applications - Extend on-premise solutions non-disruptively throu...
SYS-CON Events announced today that Gigaom Research has been named "Media Sponsor" of SYS-CON's 15th International Cloud Expo®, which will take place on November 4-6, 2014, at the Santa Clara Convention Center in Santa Clara, CA. Ashar Baig, Research Director, Cloud, at Gigaom Research, will also lead a Power Panel on the topic "Choosing the Right Cloud Option." Gigaom Research provides timely, in-depth analysis of emerging technologies for individual and corporate subscribers. Gigaom Research'...
Join both SAP and Channel Cloud Consulting for our webcast and uncover how you can extend your reach to capture a piece of the US$17 billion cloud application services market with SAP. Learn about SAPs market-leading SAP HANA Cloud Platform and an exciting opportunity to join SAPs growing ecosystem of Application Development partners. When: October 29, 11:00am EST Speakers: Allan Adler, Managing Partner, Channel Cloud Consulting Thorsten Leiduck, Vice President ISVs & Digital Commerce, SAP
Application Performance Management (APM) has been bred with all the right elements to give us the insights we need to see the health of our applications. Similar to your most trusted watch dog, it alerts us to anomalies when events occur, providing awareness to the environment that only they can observe. As enterprises embrace the DevOps philosophy, and the coalescence of the Development and Operations continues, I foresee the conditions ripening to foster innovative methods of making applicati...
SYS-CON Events announced today that IBM is holding a Bluemix Developer Playground on November 5, 10:30 am to 5:30 pm at 15th Cloud Expo. 15th Cloud Expo, co-located with @ThingsExpo, Big Data Expo, and DevOps Summit is taking place Nov. 4–6, 2014, at the Santa Clara Convention Center in Santa Clara, CA. The labs, for developers of all levels, will highlight the ease of use of Bluemix, its services and functionality and provide short-term introductory projects that developers can complete betw...
The Industrial Internet revolution is now underway, enabled by connected machines and billions of devices that communicate and collaborate. The massive amounts of Big Data requiring real-time analysis is flooding legacy IT systems and giving way to cloud environments that can handle the unpredictable workloads. Yet many barriers remain until we can fully realize the opportunities and benefits from the convergence of machines and devices with Big Data and the cloud, including interoperability, da...
Software AG helps organizations transform into Digital Enterprises, so they can differentiate from competitors and better engage customers, partners and employees. Using the Software AG Suite, companies can close the gap between business and IT to create digital systems of differentiation that drive front-line agility. We offer four on-ramps to the Digital Enterprise: alignment through collaborative process analysis; transformation through portfolio management; agility through process automation...
How do you know when a technology has become mainstream? A good clue may be when politicians start talking about it on the campaign trail and with mainstream media. David Cameron, the UK prime minister, was the latest, indicating that the world was now on “fast-forward” with the Internet of Things (IoT) ushering in the new industrial revolution. No mention of IoT targeted at the masses would be complete without the clichéd example of the communicating fridge. While it is easy to get caught up in...
In my recent article, “Software Quality Metrics for your Continuous Delivery Pipeline – Part III – Logging,” I wrote about the good parts and the not-so-good parts of logging and concluded that logging usually fails to deliver what it is so often mistakenly used for: as a mechanism for analyzing application failures in production. In response to the heated debates on reddit.com/r/devops and reddit.com/r/programing, I want to demonstrate the wealth of out-of-the-box insights you could obtain from...
The Internet of Things will greatly expand the opportunities for data collection and new business models driven off of that data. In her session at Internet of @ThingsExpo, Esmeralda Swartz, CMO of MetraTech, will discuss how for this to be effective you not only need to have infrastructure and operational models capable of utilizing this new phenomenon, but increasingly service providers will need to convince a skeptical public to participate. Get ready to show them the money! Speaker Bio: ...
This year like last year, XebiaLabs polled Fortune 1000 companies in banking, manufacturing, healthcare, government and IT, interviewing DevOps teams and everyone from QA to C-level suites. More than 1,000 people were asked to share their perspectives on software delivery trends. Last year the survey found that application deployments fail up to 30% of the time and that 75% of managers believe their deployment process deserves a failing grade. This year, the survey revealed little change in at...
Can a postmortem review help foster a curiosity for innovative possibilities to make application performance better? Blue-sky thinkers may not want to deal with the myriad of details on how to manage the events being generated operationally, but could learn something from this exercise. Consider the major system failures in your organization over the last 12 to 18 months. What if you had a system or process in place to capture those failures and mitigate them from a proactive standpoint prevent...
Machine-to-machine (M2M) technology and the resulting Internet of Things are leading us inexorably toward everything-as-a-service (XaaS). As more things get connected, the range of service opportunities expands. And as those services are presented online, they become available for use, re-use and re-purposing. At first thought, the idea of more connected devices suggests simply that there will be more devices around, and as such, more products for manufacturers to make and sell. That’s true, bu...
General Electric (GE) has been a household name for more than a century, thanks in large part to its role in making households easier to run. Starting with the light bulb invented by its founder, Thomas Edison, GE has been selling devices (“things”) to consumers throughout its 122-year history. Last week, GE announced that it is officially leaving that job to others. While the lighting division will stay, GE will now turn its attention to selling industrial machinery and analytics as a service t...
If you work in technology, you’d have to have been under a rock to have not heard about Docker. In a nutshell, Docker provides a lightweight container for code that can be installed onto a Linux system, providing both an execution environment for applications and partitioning to securely segregate sets of application code from one another. While this high-level description doesn’t sound that exciting, Docker addresses three key issues confronting application developers: One of the problems conf...
In her General Session at 15th Cloud Expo, Anne Plese, Senior Consultant, Cloud Product Marketing, at Verizon Enterprise, will focus on finding the right mix of renting vs. buying Oracle capacity to scale to meet business demands, and offer validated Oracle database TCO models for Oracle development and testing environments. Anne Plese is a marketing and technology enthusiast/realist with over 19+ years in high tech. At Verizon Enterprise, she focuses on driving growth for the Verizon Cloud pla...