Click here to close now.


@BigDataExpo Authors: Yeshim Deniz, Liz McMillan, Esmeralda Swartz, Carmen Gonzalez, Dana Gardner

Blog Feed Post

Summary from the 30 July Analyst Forum


Editor’s note: On Wednesday 30 July 2014 analysts, executives and technologists gathered at our Analyst Forum, an event established by a partnership between AnalystOne and the United States Geospatial Intelligence Foundation (USGIF) to help share lessons learned across multiple sectors of the economy, including finance, healthcare, law enforcement, emergency response, scientific research, ecommerce, IT, intelligence, and the military.

We had analysts at the event capturing content that we will summarize here over the next few weeks. And we are surveying 280 people who engaged with us to ensure we are capturing the most important results of the event as well as gaps this community believes we should be collectively tackling to improve the state of analysis. All of this will be shaping our reporting here.

In the post below Katie Kennedy provides a high level overview of the event. Stand by for more details.

Bob Gourley
Publisher, Analyst One

 30 July Analyst Forum: The First Report/Summary

Analytics 2014: Insights for Mission impact began with opening remarks from Keith Masback, Chief Executive Officer, United States Geospatial Intelligence Foundation (USGIF). Keith welcomed all and helped bring focus to some of the opportunities the event could help the community achieve.  Keith has been an analyst and a leader of analysts and is now a key champion of mission focused capabilities including those supporting the analytical community. He brings knowledge of technology and methodology and critical important missions to this topic. His key message in introducing the event was the criticality of the thoughts that would be exchanged today.

Bob Gourley welcomed attendees. Early on he underscored the gratitude the event organizers had for the sponsors. These firms sponsored the event because of their respect for the mission and understanding of the importance of this effort to the community and they are very much appreciated. Sponsors of the event are: Leidos,  BAE Systems,  Carahsoft,  Cloudera,  Digital ReasoningDigital Globe,  IBM  and  ICG.

Gourley also provided a heartfelt thanks to all attendees for coming, and spent time giving attendees a feel for who else is in the room. Attendees came from the finance sector, media outlets, ecommerce companies, retail companies, the heathcare sector, the scientific research community, law enforcement, plus multiple government agencies including DHS, DoD (Army, Navy, Air Force, Marines ), NRO, DIA, NGA, NIH, NCI, DoT, Treasury, State, VA, HHS, and many others. Commercial firms present included multiple participants from systems integrators with analytical capabilities, high tech analytical tool companies like Recorded Future,WayInPentahoOptensity and Hexis Cyber. Bob also took time to single out the company Cognitio, since that is the parent firm that sponsors and and employs Bob.

Speakers at the event are all interesting with great backgrounds in analysis. Speakers included: Tom LashJeff JonasIlkay AltintasDave WarnerDavid BrayKirk BornePhil BourneBob GrossmanJohn HarerKelly McCueDavid RobertsAbe UsherBill WallCarmen MedinaEd MornstonMark AshwellWilliam NolteMatt DevostAdam ElkusNed MoranErin SimpsonDave GauthierBob JimenezJason Thomas, and Scott Sorensen,

Tom Lash of Leidos, a key community thought leader on topics of analytics provided a welcome and an introduction to Jeff Jonas, our first morning discussant. Jeff is widely known in the analytical community for his ability to generate thoughts of direct and impactful relevance to helping people get their missions done. He is also highly regarded as a creator of both concepts and technology that add real value to organizations. Jonas has directed the design and development of multiple innovative systems, including solutions which have countered fraud in the gaming industry (See Jeff’s bio here).

Jonas prepared visuals that paired incredibly well with the messages in his presentation. It was a great help to expanding our thoughts to get Jeff’s perspectives on topics like detection of asteroids and then further turning observations into predictions of asteroid route and event predictions of future observable events like asteroid collisions. This important topic was used to underscore the importance of context when doing analysis. This theme of context was critically important to Jeff’s presentation and was also a recurring theme hit upon many other times throughout the day.

Jeff’s approach is to seek not just a little data, but as much data as possible. And of course to seek and leverage context. There needs to be more data to make better predictions, as well as filling in the gaps where there is not enough data. Jonas used the example of twins. If you have two people who look identical in every way and every feature determines them as one entity, how can you distinguish them? Therein lies the fact that two identical things cannot occupy the same space at the same time. To distinguish the two, there has to be data that supports the case that they are indeed two separate people. The more data gathered, the easier it would be to distinguish the twins and find they are two separate and distinct individuals.

By the way, Jeff received thunderous applause. It was great having him interacting with us all throughout  the day to help underscore the lessons he left us with.

Bob introduced the next speaker,  Dr. Ilkay Altintas, the Director for the Center of Excellence in Workflows for Data Science at the San Diego Supercomputer Center (SDSC), UCSD. Altintas addressed the question of “why data science workflows?” She answered,  “workflows contain a lot of small steps that become programmable and reproducible scalability.” A project the SDSC has been successfully working on is entitled “WIFIRE”, a scalable data-driven monitoring, dynamic prediction and resilience cyberinfrastructure for predicting and monitoring wildfires. Wildfires are difficult to predict, therefore WIFIRE supports an integrated process that analyzes wildfires, incorporating observations with real-time data.

Bob asked Dr. Altintas to discuss “The Scientific Method.”  Here is the idea we pondered: almost all of us grew up learning “The Scientific Method”.  The scientific method of observation, hypothesis, prediction, experimenting is still critically important.  But because of new technologies available to researchers and the incredible amount of data available, it is no longer the only workflow available to construct workable models of the world or to advance science and understanding. Dr. Altintas provided context on this topic that underscored the importance of considering workflows in analytical processes at any sector.

Bob then introduced Dr, Dave Warner, M.D., Ph.D., Medical Neuroscientist and the Director of Medical Intelligence at MindTel. Dr. Warner opened with “Computers are rocks that do math and they should worship us”. His idea that “dots are stupid” led to his highly innovative creation of hyper dimensional dots, where an observer can see thousands of different information on the dot, while taking advantage of abstract information. The dots are geolocated allowing the viewer of the dot to hover over an area and instantly have access to densely populated information on the area observed. Creating the models of information preserves relationships and transformation. Each 3-dimensional dot instantly presented multiple facets of data collected in a way that was easily understood by the vast majority of people.

After a short morning break for direct networking, everyone was ushered back into the hall where the next speaker was poised and ready to address the audience. Dr. David A. Bray, Chief Information Officer, FCC, prompted the question of “how to tackle the changing world?” Points he drove home through example after example were that analysts needed enhanced context to ensure optimal assessments. Among the many considerations he helped us think through were the fact that we all need to recognize that you can’t just build higher walls for Internet security. Computers need to alert humans when a threat arises. The FCC is making an application that will send pings during a crisis, constantly updating individuals while simultaneously sending feedback to responders. Innovation is needed more than ever; can public service change quickly enough? It is a technology as well as a people issue. The future includes algorithms working alongside public service workers:  when will having software make an unbiased recommendation better than a human? Humans are biased, so maybe algorithms are better for decision-making in some cases. In all cases, context is important.

The next segment included Cross-Sector Analytical Lessons Learned. The moderator was Dr. Kirk Borne, Data Scientist and Professor, George Mason University. Other segment participants included Dr. Philip E. Bourne, Associate Director for Data Science, National Institutes of Health; Dr. Robert Grossman, Director, Center for Data Intensive Science, University of Chicago; and Partner, Open Data Group; and Dr. John Harer, Professor of Mathematics and Computer Science, Duke University. Each individual had their own unique lessons they had learned. First was Dr. Borne, he addressed his lesson with the idea that no matter what field you are in, you can talk with people in other fields when you are speaking the trans-disciplinary language of data science. Second was Dr. Bourne, reasoning that answers to big data problems can come from anywhere either cofounded in a journal, a pandemic modeling article, or a 15 year old who was published in a leading journal. Third was Dr. Grossman, addressing that all men can see tactics, but who can see strategy? Tools are understood, but strategy is not as thought-through. Lastly, Dr. Harer prompted that successful collaborations have understandings of shared work, learning others’ languages and fields, and working with someone who needs you and who wants you to need them. Each panelist emphasized that there were many lessons learned and many more to come.

The next panel was law enforcement, with moderator: Dr. Colleen “Kelly” McCue, Senior Director, Social Science & Quantitative Methods, DigitalGlobe Analytics. The panelists were David J. Roberts, Senior Program Manager, IACP Technology Center, International Association of Chiefs of Police; Abe Usher, Chief Technology Officer, HumanGeo; and Bill Wall, Vice President, Praescient Analytics.  Each individual on this panel presented around a central theme that positively correlated better law enforcement and increased data usage. Usher presented on the lessons learned from the London Olympics. He said that as law enforcers, there should be proper identification of observation space and the creation of simple systems for recognizing normal and abnormal behavior. If there is a baseline, then it would be simple to find the abnormal. Next Wall addressed the London riots of 2011, relative to geospatial technology. Wall addressed that there should be geospatial data to build a database of individuals and incidents. Geospatial aspects of big data give us special capabilities to contextualize events. For the London riots, using the geospatial technology, law enforcers were able to find the rioters prior to their insurrection. Next, McCue presented on the Northern Virginia shooting incident that was solved using geospatial predictive analytics providing the high probability target areas that led to the suspect’s arrest. Lastly, Roberts brought to the audiences’ attention the immense impact the poor economy has had on the police workforce. Optimizing resources is extremely important and information-sharing capabilities are essential. Research has emerged looking at crime analysis data to identify systemic problems. Having data can create a mosaic, where many pixels create an image that can be used to solve crimes. The increased gathering of data has led to significant breakthroughs in solving crime.

After a lunch reception and exhibits, the next panel eagerly anticipated the audience’s return to their seats. The next panel was the Education of the Analyst, with Carmen Medina, Specialist Leader, Deloitte Consulting; Ed Mornston, Director, Human Development Directorate, National Geospatial-Intelligence Agency; and Dr. William Nolte, Program Director, Intelligence Center of Academic Excellence, University of Maryland School of Public Policy, and Mr. Mark Ashwell, internationally known developer of analytical strategies who has served as director of intelligence at the Royal Air Force and other UK MoD positions.  Medina addressed cognitive diversity and the future of intelligence work, meaning that not all thinkers are the same and there are inherent cognitive differences. She shared the formula calculating what you know as reality – observation error – bias = what you know. Mornston followed with the idea of creating a culture of learning where there is a thirst for innovation, while addressing challenges from declining resources. He urged that agility is key and needs must be anticipated. Nolte pointed out that there are now technically illiterate students and technologists with little worldliness. Analysts must think analytically, not tactically. The 21st century intelligence will be about information, not secrets, like the 20th century. Ashwell stated that the main issue is the challenge of using open source and classified information; analysts must be able to use both to not fall behind. Analysts are not prepared for using big data, we must move from the word to visualization.

In the question and answer session, Dr. Ilkay Altinas asked about the new age of data around individuals and the need for individuals to have data analysis skills and requested panelist thoughts on that topic. This powerful question generated great discussion and is a topic we will continue to help think and write about.

The following panel addressed the analytics of cyber conflict with moderator Matt Devost, President & CEO, FusionX and panelists Adam Elkus, Senior Analyst at Analyst One, Strategic Planner and Ph.D. Student, Computational Social Science, George Mason University; and Ned Moran, Professor, Analyst, Cyber Practitioner. Elkus started the discussion with the notion that cyber conflict takes place in a human made environment and within natural laws. It is a tangled mess of man and machines. What you intend a machine to do is not what it actually does. A lot of cyber issues come down to adaptation and speed. Moran stated that cyber threat intelligence means having better informed defensive decisions. There is a need to get ahead of the attacker through analysis. Moran addressed how we defend against attackers. The landscape is changing, but still the defenders are the ones always responding to the attackers. Moran said that “humans always make mistakes and attackers take advantage”. Cyber conflict is a defenders game and will be for a while.

The final panel was Lessons From and for the National Security Community, with moderator: Dr. Erin M. Simpson, Chief Executive Officer, Caerus Associates and panelists Dave Gauthier, Activity Based Intelligence Portfolio Lead, Analysis Directorate, National Geospatial-Intelligence Agency; Bob Jimenez, Chief Technology Officer, National Reconnaissance Office; and Jason Thomas, Manager of Innovation for the Government, Thomson Reuters. This panel addressed the challenges with data in national security. A few challenges include how data correlates, how to engage an NRO user, and how to recruit people to solve problems. Thomas said that “we have to adapt to what we think is coming, which is not a lateral movement”. In some cases we store more data than necessary and in others, we can’t get our hands on data we need.

The final speaker was Scott Sorensen, Chief Technology Officer,  His presentation started with a fantastic way to grab the attention of everyone in the room. We all know and the overview and insights of the technological approaches there (and impact on solutions for customers) really riveted everyone’s attention.  “At, we tell stories from data; we fill in the gaps of family history. With the use of AncestryDNA, we have collected sampled from 500,000 people and found over 10,000,000 fourth cousin matches. We provide context to family history by working with government agencies and churches.” Handwriting recognition locates words on a handwritten document, extracts the words and makes the documents easier to place in family trees. The exciting stories lead directly to very relevant conclusions for any organization seeking to enhance analysis, in any sector. We will provide details on these lessons in coming posts, Scott has agreed to share his graphics with us and they flow directly to those lessons, so stand by for more.  The short version, however, is this: How you organize you analytical efforts, including the people working them and even where they sit, is of critical importance. Getting groups to work together is a challenge especially with scientists and software engineers.

The event ended on a high note, and everyone, excited from the day’s events, filed into the adjacent room to mingle and discuss the information learned. The end of day networking allowed for attendees to more deeply connect and interact with each other and speakers.

We will be publishing more on this event and its conclusions will drive our lessons learned throughout the year. To stay in touch with us and to continue to interact with the community, sign up for the AnalystOne report and other newsletters here.  Also follow us on Twitter at @AnalystReport.

We would like to conclude this overview with a huge thank you to the team at USGIF who worked tirelessly to ensure this event would come off without a hitch. Their attention to detail and focus on getting the right things done went above and beyond the call of duty

Read the original blog entry...

More Stories By Bob Gourley

Bob Gourley, former CTO of the Defense Intelligence Agency (DIA), is Founder and CTO of Crucial Point LLC, a technology research and advisory firm providing fact based technology reviews in support of venture capital, private equity and emerging technology firms. He has extensive industry experience in intelligence and security and was awarded an intelligence community meritorious achievement award by AFCEA in 2008, and has also been recognized as an Infoworld Top 25 CTO and as one of the most fascinating communicators in Government IT by GovFresh.

@BigDataExpo Stories
As more intelligent IoT applications shift into gear, they’re merging into the ever-increasing traffic flow of the Internet. It won’t be long before we experience bottlenecks, as IoT traffic peaks during rush hours. Organizations that are unprepared will find themselves by the side of the road unable to cross back into the fast lane. As billions of new devices begin to communicate and exchange data – will your infrastructure be scalable enough to handle this new interconnected world?
SYS-CON Events announced today that Super Micro Computer, Inc., a global leader in high-performance, high-efficiency server, storage technology and green computing, will exhibit at the 17th International Cloud Expo®, which will take place on November 3–5, 2015, at the Santa Clara Convention Center in Santa Clara, CA. Supermicro (NASDAQ: SMCI), the leading innovator in high-performance, high-efficiency server technology is a premier provider of advanced server Building Block Solutions® for Data ...
SYS-CON Events announced today the Containers & Microservices Bootcamp, being held November 3-4, 2015, in conjunction with 17th Cloud Expo, @ThingsExpo, and @DevOpsSummit at the Santa Clara Convention Center in Santa Clara, CA. This is your chance to get started with the latest technology in the industry. Combined with real-world scenarios and use cases, the Containers and Microservices Bootcamp, led by Janakiram MSV, a Microsoft Regional Director, will include presentations as well as hands-on...
SYS-CON Events announced today that Dyn, the worldwide leader in Internet Performance, will exhibit at SYS-CON's 17th International Cloud Expo®, which will take place on November 3-5, 2015, at the Santa Clara Convention Center in Santa Clara, CA. Dyn is a cloud-based Internet Performance company. Dyn helps companies monitor, control, and optimize online infrastructure for an exceptional end-user experience. Through a world-class network and unrivaled, objective intelligence into Internet condit...
SYS-CON Events announced today that Sandy Carter, IBM General Manager Cloud Ecosystem and Developers, and a Social Business Evangelist, will keynote at the 17th International Cloud Expo®, which will take place on November 3–5, 2015, at the Santa Clara Convention Center in Santa Clara, CA.
The Internet of Things (IoT) is growing rapidly by extending current technologies, products and networks. By 2020, Cisco estimates there will be 50 billion connected devices. Gartner has forecast revenues of over $300 billion, just to IoT suppliers. Now is the time to figure out how you’ll make money – not just create innovative products. With hundreds of new products and companies jumping into the IoT fray every month, there’s no shortage of innovation. Despite this, McKinsey/VisionMobile data...
The IoT market is on track to hit $7.1 trillion in 2020. The reality is that only a handful of companies are ready for this massive demand. There are a lot of barriers, paint points, traps, and hidden roadblocks. How can we deal with these issues and challenges? The paradigm has changed. Old-style ad-hoc trial-and-error ways will certainly lead you to the dead end. What is mandatory is an overarching and adaptive approach to effectively handle the rapid changes and exponential growth.
With major technology companies and startups seriously embracing IoT strategies, now is the perfect time to attend @ThingsExpo in Silicon Valley. Learn what is going on, contribute to the discussions, and ensure that your enterprise is as "IoT-Ready" as it can be! Internet of @ThingsExpo, taking place Nov 3-5, 2015, at the Santa Clara Convention Center in Santa Clara, CA, is co-located with 17th Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading in...
There will be 20 billion IoT devices connected to the Internet soon. What if we could control these devices with our voice, mind, or gestures? What if we could teach these devices how to talk to each other? What if these devices could learn how to interact with us (and each other) to make our lives better? What if Jarvis was real? How can I gain these super powers? In his session at 17th Cloud Expo, Chris Matthieu, co-founder and CTO of Octoblu, will show you!
The IoT is upon us, but today’s databases, built on 30-year-old math, require multiple platforms to create a single solution. Data demands of the IoT require Big Data systems that can handle ingest, transactions and analytics concurrently adapting to varied situations as they occur, with speed at scale. In his session at @ThingsExpo, Chad Jones, chief strategy officer at Deep Information Sciences, will look differently at IoT data so enterprises can fully leverage their IoT potential. He’ll sha...
The enterprise is being consumerized, and the consumer is being enterprised. Moore's Law does not matter anymore, the future belongs to business virtualization powered by invisible service architecture, powered by hyperscale and hyperconvergence, and facilitated by vertical streaming and horizontal scaling and consolidation. Both buyers and sellers want instant results, and from paperwork to paperless to mindless is the ultimate goal for any seamless transaction. The sweetest sweet spot in innov...
Today air travel is a minefield of delays, hassles and customer disappointment. Airlines struggle to revitalize the experience. GE and M2Mi will demonstrate practical examples of how IoT solutions are helping airlines bring back personalization, reduce trip time and improve reliability. In their session at @ThingsExpo, Shyam Varan Nath, Principal Architect with GE, and Dr. Sarah Cooper, M2Mi's VP Business Development and Engineering, will explore the IoT cloud-based platform technologies driv...
The Internet of Everything is re-shaping technology trends–moving away from “request/response” architecture to an “always-on” Streaming Web where data is in constant motion and secure, reliable communication is an absolute necessity. As more and more THINGS go online, the challenges that developers will need to address will only increase exponentially. In his session at @ThingsExpo, Todd Greene, Founder & CEO of PubNub, will explore the current state of IoT connectivity and review key trends an...
SYS-CON Events announced today that DataClear Inc. will exhibit at the 17th International Cloud Expo®, which will take place on November 3–5, 2015, at the Santa Clara Convention Center in Santa Clara, CA. The DataClear ‘BlackBox’ is the only solution that moves your PC, browsing and data out of the United States and away from prying (and spying) eyes. Its solution automatically builds you a clean, on-demand, virus free, new virtual cloud based PC outside of the United States, and wipes it clean...
SYS-CON Events announced today that Machkey International Company will exhibit at the 17th International Cloud Expo®, which will take place on November 3–5, 2015, at the Santa Clara Convention Center in Santa Clara, CA. Machkey provides advanced connectivity solutions for just about everyone. Businesses or individuals, Machkey is dedicated to provide high-quality and cost-effective products to meet all your needs.
Nowadays, a large number of sensors and devices are connected to the network. Leading-edge IoT technologies integrate various types of sensor data to create a new value for several business decision scenarios. The transparent cloud is a model of a new IoT emergence service platform. Many service providers store and access various types of sensor data in order to create and find out new business values by integrating such data.
There are so many tools and techniques for data analytics that even for a data scientist the choices, possible systems, and even the types of data can be daunting. In his session at @ThingsExpo, Chris Harrold, Global CTO for Big Data Solutions for EMC Corporation, will show how to perform a simple, but meaningful analysis of social sentiment data using freely available tools that take only minutes to download and install. Participants will get the download information, scripts, and complete en...
Too often with compelling new technologies market participants become overly enamored with that attractiveness of the technology and neglect underlying business drivers. This tendency, what some call the “newest shiny object syndrome,” is understandable given that virtually all of us are heavily engaged in technology. But it is also mistaken. Without concrete business cases driving its deployment, IoT, like many other technologies before it, will fade into obscurity.
Achim Weiss is Chief Executive Officer and co-founder of ProfitBricks. In 1995, he broke off his studies to co-found the web hosting company "Schlund+Partner." The company "Schlund+Partner" later became the 1&1 web hosting product line. From 1995 to 2008, he was the technical director for several important projects: the largest web hosting platform in the world, the second largest DSL platform, a video on-demand delivery network, the largest eMail backend in Europe, and a universal billing syste...
Cloud computing delivers on-demand resources that provide businesses with flexibility and cost-savings. The challenge in moving workloads to the cloud has been the cost and complexity of ensuring the initial and ongoing security and regulatory (PCI, HIPAA, FFIEC) compliance across private and public clouds. Manual security compliance is slow, prone to human error, and represents over 50% of the cost of managing cloud applications. Determining how to automate cloud security compliance is critical...

Tweets by @BigDataExpo

@BigDataExpo Blogs
This week, the team assembled in NYC for @Cloud Expo 2015 and @ThingsExpo 2015. For the past four years, this has been a must-attend event for MetraTech. We were happy to once again join industry visionaries, colleagues, customers and even competitors to share and explore the ways in which the Internet of Things (IoT) will impact our industry. Over the course of the show, we discussed the types of challenges we will collectively need to solve to capitalize on the opportunity IoT presents.
Today’s connected world is moving from devices towards things, what this means is that by using increasingly low cost sensors embedded in devices we can create many new use cases. These span across use cases in cities, vehicles, home, offices, factories, retail environments, worksites, health, logistics, and health. These use cases rely on ubiquitous connectivity and generate massive amounts of data at scale. These technologies enable new business opportunities, ways to optimize and automate, along with new ways to engage with users.
Dasher Technologies is helping to usher in the democratization of big data value to more players in less time with analytics in a cloud services model
Heat maps offer a unique way to represent data sets in a variety of settings. A common example you see most mornings is through weather reports on the news showing the movement and predictions of pressure and precipitation, intensifying across bold color schemes. Another familiar example is voting representation in specific areas of the country during election times (think red state vs. blue state). Retail stores utilize heat maps to enable managers and executives to better understand the functionality of their space, as well as the habits and preferences of the people entering the store.
Developing software for the Internet of Things (IoT) comes with its own set of challenges. Security, privacy, and unified standards are a few key issues. In addition, each IoT product is comprised of at least three separate application components: the software embedded in the device, the backend big-data service, and the mobile application for the end user's controls. Each component is developed by a different team, using different technologies and practices, and deployed to a different stack/target - this makes the integration of these separate pipelines and the coordination of software upd...
“Cloud computing is a model for enabling ubiquitous, convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications and services) that can be rapidly provisioned and released with minimal management.” While this definition is broadly accepted and has, in fact, been my adopted standard for years, it only describes technical aspects of cloud computing. The amalgamation of technologies used to deliver cloud services is not even half the story. Above all else, the successful employment requires a tight linkage to the econ...
Today air travel is a minefield of delays, hassles and customer disappointment. Airlines struggle to revitalize the experience. GE and M2Mi will demonstrate practical examples of how IoT solutions are helping airlines bring back personalization, reduce trip time and improve reliability. In their session at @ThingsExpo, Shyam Varan Nath, Principal Architect with GE, and Dr. Sarah Cooper, M2Mi's VP Business Development and Engineering, will explore the IoT cloud-based platform technologies driving this change including privacy controls, data transparency and integration of real time context w...
DevOps Summit at Cloud Expo 2014 Silicon Valley was a terrific event for us. The Qubell booth was crowded on all three days. We ran demos every 30 minutes with folks lining up to get a seat and usually standing around. It was great to meet and talk to over 500 people! My keynote was well received and so was Stan's joint presentation with RingCentral on Devops for BigData. I also participated in two Power Panels – ‘Women in Technology’ and ‘Why DevOps Is Even More Important than You Think,’ both featuring brilliant colleagues and moderators and it was a blast to be a part of.
All we need to do is have our teams self-organize, and behold! Emergent design and/or architecture springs up out of the nothingness! If only it were that easy, right? I follow in the footsteps of so many people who have long wondered at the meanings of such simple words, as though they were dogma from on high. Emerge? Self-organizing? Profound, to be sure. But what do we really make of this sentence?
Today’s modern day industrial revolution is being shaped by ubiquitous connectivity, machine to machine (M2M) communications, the Internet of Things (IoT), open APIs leading to a surge in new applications and services, partnerships and eventual marketplaces. IoT has the potential to transform industry and society much like advances in steam technology, transportation, mass production and communications ushered in the industrial revolution in the 18th and 19th centuries.
The potential of big data is only limited by the creative thinking of your business stakeholders, and that may be the most important concept in the “thinking like a data scientist” process. The “thinking like a data scientist” process guides the business stakeholders into envisioning how big data can optimize their key business processes, create a more compelling customer engagement and uncover new monetization opportunities. But neither the business stakeholders, nor the data scientists, can likely do that envisioning entirely by themselves.
Disaster recovery (DR) has traditionally been a major challenge for IT departments. Even with the advent of server virtualization and other technologies that have simplified DR implementation and some aspects of on-going management, it is still a complex and (often extremely) costly undertaking. For those applications that do not require high availability, but are still mission- and business-critical, the decision as to which [applications] to spend money on for true disaster recovery can be a struggle.
SCOPE is an acronym for Structured Computations Optimized for Parallel Execution, a declarative language for working with large-scale data. It is still under development at Microsoft. If you know SQL then working with SCOPE will be quite easy as SCOPE builds on SQL. The execution environment is different from that RDBMS oriented data. Data is still modeled as rows. Every row has typed columns and eveyr rowset has a well-defined schema. There is a SCOPe compiler that comes up with optimized execution plan and a runtime execution plan.
If you’re running Big Data applications, you’re going to want to look at some kind of distributed processing system. Hadoop is one of the best-known clustering systems, but how are you going to process all your data in a reasonable time frame? MapReduce has become a standard, perhaps the standard, for distributed file systems. While it’s a great system already, it’s really geared toward batch use, with jobs needing to queue for later output. This can severely hamper your flexibility. What if you want to explore some of your data? If it’s going to take all night, forget about it.
Too many multinational corporations delete little, if any, data even though at its creation, more than 70 percent of this data is useless for business, regulatory or legal reasons.[1] The problem is hoarding, and what businesses need is their own “Hoarders” reality show about people whose lives are driven by their stuff[2] (corporations are legally people, after all). The goal of such an intervention (and this article)? Turning hoarders into collectors.

About @BigDataExpo
Big Data focuses on how to use your own enterprise data – processed in the Cloud – most effectively to drive value for your business.