Welcome!

@BigDataExpo Authors: John Rauser, Liz McMillan, Elizabeth White, Pat Romanski, Amy Eager

Related Topics: @BigDataExpo, Java IoT, @CloudExpo

@BigDataExpo: Article

Data Lake Plumbers | @BigDataExpo @Schmarzo #BigData #IoT #AI #ML #DL

The data lake is ideal for your data science team as it liberates them from the constraints & limitations of the data warehouse

Many of my blogs promote the business benefits of the data lake, both from a “save me more money” as well as the “make me more money” perspectives. But I fear that I’m making this thing called the data lake sound like a “silver bullet" [1] – just drop the data into the data lake and everything magically works. But much like in the world of data warehousing, there is significant work that needs to be done under the covers – in areas such as metadata management, data governance and security – to ensure that the data lake will perform for a business in a production environment. Many of the processes and techniques we learned in the data warehouse will benefit us here, though there are many new tools to be aware of that can help us in the operationalization task.

I’ve asked an industry expert in metadata management and data governance, Joe DosSantos (follow Joe on twitter: @JoeDosSantos) to co-author this blog with me. Well, to be honest, this mostly reflects Joe’s experience and thinking; I just wanted to get credit for being smart enough to know when to bring someone smarter than me into the conversation!

Data Lake Benefits
You know from previous blogs that there are many benefits to the data lake including:

  • Capture data from wide range of traditional (operational, transactional) and new sources (structured and unstructured) as-is
  • Store all your data in one environment for cross-functional business analysis
  • Support the analytics and data science to uncover new customer, product, and operational insights
  • Empower front-line employees and managers, and drive a more profitable customer engagement leveraging customer, product and operational insights
  • Integrate analytic insights into operational (Finance, Manufacturing, Marketing, Sales Force, Procurement, Logistics) and management systems (Business Intelligence reports and dashboards)

The data lake is ideal for your data science team in that it liberates them from the constraints and limitations of the data warehouse, enabling the data science team to quickly ingest, test and determine if there is any value to different data sets and analytic techniques without having to go through the rigorous operational procedures that govern the data warehouse.

However, this liberty can be quite terrifying in highly regulated environments. Companies have spent years developing governance and stewardship organizations specifically to control patient information, personal contact information, account balances, and other sensitive information. The description above seems to undo all of this work by creating free and easy access to data that should be locked down.

This is why the controls of a data lake need to be very clear. Data that is onboarded into a lake must go through a rigorous set of operational procedures to manage and govern that data set to make sure that it is appropriately tagged and protected, and then provisioned only to people who have the proper authorization. Modern data tools allow for this kind of governance capability to balance the quick and easy access to data that a data scientist needs with the security that good practices (and often the government) demand.

Operationalizing the Data Lake
Operationalizing the data lake requires several non-obvious disciplines, many of which we learned from our data warehouse experiences. These disciplines include data ingestion, indexing, cataloging, metadata management, data governance and security [2].

  1. As with a data warehouse, you will need a method to bring data into your environment. As batch windows became longer and longer in the data warehouse world and business users clamored for increasingly up-to-date information, practitioners began shifting from conventional data ETL (Extract, Transform, Load) to lower latency streaming and micro-batch. This trend was extended in the big data universe with tools like Kafka, a streaming message bus, and with Spring and Sqoop to accelerate data ingest. In the big data world, you might also want to ingest unstructured data sets as well, introducing new tools like Flume. Finally, you may want to trigger complex events based on this data stream and you might do so via Spark, Gemfire, or other in-memory grids. And just to make it more complex, you will likely use several of these tools in combination depending on your data feed needs. Keep in mind that in the world of ELT (Extract, Load, Transform) (note that the order differs from E-T-L), most of these data movements are data dumps. At this point, you have simply collected lots of raw data. It’s now time to make sense of it.
  2. Next, it is useful to tag files that you have ingested. What kind of file is this? What would be useful to know about it so that I could search for it later? Zaloni Bedrock is an example of a tool to apply metadata tags to the files, which is useful for both structured and unstructured data sets.
  3. We mentioned above that one of the key requirements of our data lake is having control over who can have access to specific data sources. Generally speaking, the data loaded in steps 1 and 2 is what we call “Bronze” data, meaning that it is good enough for the business process that created it. Data in these sets will likely be sensitive and your security should reflect it. However, we need to determine rules for how the data should be modified, obfuscated, or deleted in order to make it consumable for broader audience, or what we might call “Silver” status. You need to create business rules to manage data (e.g. birthdays should be masked and social security numbers should be stored as only the last 4). Collibra is an example of a tool for this rules definition and management. It allows data rules to be set up based on logical business entities by business people rather than technologists.
  4. For those people who are familiar with governance concepts, you will recognize the difference between a policy and a control. A policy is like a speed limit sign along the highway. The control is the police officer that pulls you over if you are driving over that speed limit. Data works the same way. While Collibra establishes the policy, you need to create a method for enforcing that policy. To do this, you need to find the logical entities buried in the data (i.e. “oh look, I found a social security number!”). Examples of such products include Global IDs for scanning structured data sets with the operational systems and Waterline for scanning data inside of Hadoop.
  5. Once you have found the data that you want, you want to implement the rules. For this, there is an open source tool called Atlas that contains an orchestration capability called Falcon that helps implement the rules.
    1. Apache Atlas is a scalable and extensible set of core foundational governance services that enables enterprises to effectively and efficiently meet their compliance requirements within Hadoop and allows integration with the complete enterprise data ecosystem.
    2. Apache Falcon is a data governance engine that defines, schedules, and monitors data management policies. Falcon allows Hadoop administrators to centrally define their data pipelines, and then Falcon uses those definitions to auto-generate workflows in Apache Oozie
  6. Now that the data is loaded, you will want to enforce security through your LDAP capability or possibly through Kerberos. There are also tools like Blue Talon that simplify the ability to authorize, provision, protect, enforce and audit data security policies across your data lake.
  7. Finally, audit controls are critical. Cloudera introduced Navigator specifically to allow simple transparency to data history and lineage. Hortonworks will rely on Atlas to provide this capability.

Data that has gone through the above processes creates a view and accessibility of the data that can be made available to a wide set of users – both business analysts and data science teams.

Summary
When you build a house, the vast majority of the creative work is in the features and curbside appeal. That’s the fun part. But without the underlying plumbing, the house would quickly degrade into a money pit.

Consider the metaphor of a retail store: stocking the shelves vs. purchasing goods. When you go to the store, you don’t care about how the goods got there, but the rules for accessing the goods are everywhere. Cigarettes are behind the front desk. Pharmaceuticals must be dispensed with a prescription. Razor blades are under lock and key (for some strange reason). There are policies and enforcements on stocking the shelves so that the shopping experience is clear and easy.

To successfully operationalize the data lake, organizations need to address all of the plumbing requirements outlined in this blog that enable the business users and data science teams to have confidence in the wealth of data that the organization is amassing. The data lake plumbing processes may not be very glamorous, but without them, you’ll end up with a stinky data dump instead of a glorious data lake.

References:

  1. A “silver bullet” is a simple and seemingly magical solution to a complicated problem.
  2. While I mention several tools, this blog is not meant to be an endorsement of these tools nor is this intended to be a comprehensive list of such tools. However, many of these tools are the same tools that we use in our data lake implementations at EMC.

Data Lake Plumbers: Operationalizing the Data Lake
Bill Schmarzo

More Stories By William Schmarzo

Bill Schmarzo, author of “Big Data: Understanding How Data Powers Big Business”, is responsible for setting the strategy and defining the Big Data service line offerings and capabilities for the EMC Global Services organization. As part of Bill’s CTO charter, he is responsible for working with organizations to help them identify where and how to start their big data journeys. He’s written several white papers, avid blogger and is a frequent speaker on the use of Big Data and advanced analytics to power organization’s key business initiatives. He also teaches the “Big Data MBA” at the University of San Francisco School of Management.

Bill has nearly three decades of experience in data warehousing, BI and analytics. Bill authored EMC’s Vision Workshop methodology that links an organization’s strategic business initiatives with their supporting data and analytic requirements, and co-authored with Ralph Kimball a series of articles on analytic applications. Bill has served on The Data Warehouse Institute’s faculty as the head of the analytic applications curriculum.

Previously, Bill was the Vice President of Advertiser Analytics at Yahoo and the Vice President of Analytic Applications at Business Objects.

@BigDataExpo Stories
IoT solutions exploit operational data generated by Internet-connected smart “things” for the purpose of gaining operational insight and producing “better outcomes” (for example, create new business models, eliminate unscheduled maintenance, etc.). The explosive proliferation of IoT solutions will result in an exponential growth in the volume of IoT data, precipitating significant Information Governance issues: who owns the IoT data, what are the rights/duties of IoT solutions adopters towards t...
With the introduction of IoT and Smart Living in every aspect of our lives, one question has become relevant: What are the security implications? To answer this, first we have to look and explore the security models of the technologies that IoT is founded upon. In his session at @ThingsExpo, Nevi Kaja, a Research Engineer at Ford Motor Company, discussed some of the security challenges of the IoT infrastructure and related how these aspects impact Smart Living. The material was delivered interac...
The current age of digital transformation means that IT organizations must adapt their toolset to cover all digital experiences, beyond just the end users’. Today’s businesses can no longer focus solely on the digital interactions they manage with employees or customers; they must now contend with non-traditional factors. Whether it's the power of brand to make or break a company, the need to monitor across all locations 24/7, or the ability to proactively resolve issues, companies must adapt to...
With major technology companies and startups seriously embracing Cloud strategies, now is the perfect time to attend 21st Cloud Expo October 31 - November 2, 2017, at the Santa Clara Convention Center, CA, and June 12-14, 2018, at the Javits Center in New York City, NY, and learn what is going on, contribute to the discussions, and ensure that your enterprise is on the right path to Digital Transformation.
The taxi industry never saw Uber coming. Startups are a threat to incumbents like never before, and a major enabler for startups is that they are instantly “cloud ready.” If innovation moves at the pace of IT, then your company is in trouble. Why? Because your data center will not keep up with frenetic pace AWS, Microsoft and Google are rolling out new capabilities. In his session at 20th Cloud Expo, Don Browning, VP of Cloud Architecture at Turner, posited that disruption is inevitable for comp...
No hype cycles or predictions of zillions of things here. IoT is big. You get it. You know your business and have great ideas for a business transformation strategy. What comes next? Time to make it happen. In his session at @ThingsExpo, Jay Mason, Associate Partner at M&S Consulting, presented a step-by-step plan to develop your technology implementation strategy. He discussed the evaluation of communication standards and IoT messaging protocols, data analytics considerations, edge-to-cloud tec...
When growing capacity and power in the data center, the architectural trade-offs between server scale-up vs. scale-out continue to be debated. Both approaches are valid: scale-out adds multiple, smaller servers running in a distributed computing model, while scale-up adds fewer, more powerful servers that are capable of running larger workloads. It’s worth noting that there are additional, unique advantages that scale-up architectures offer. One big advantage is large memory and compute capacity...
New competitors, disruptive technologies, and growing expectations are pushing every business to both adopt and deliver new digital services. This ‘Digital Transformation’ demands rapid delivery and continuous iteration of new competitive services via multiple channels, which in turn demands new service delivery techniques – including DevOps. In this power panel at @DevOpsSummit 20th Cloud Expo, moderated by DevOps Conference Co-Chair Andi Mann, panelists examined how DevOps helps to meet the de...
Cloud applications are seeing a deluge of requests to support the exploding advanced analytics market. “Open analytics” is the emerging strategy to deliver that data through an open data access layer, in the cloud, to be directly consumed by external analytics tools and popular programming languages. An increasing number of data engineers and data scientists use a variety of platforms and advanced analytics languages such as SAS, R, Python and Java, as well as frameworks such as Hadoop and Spark...
"When we talk about cloud without compromise what we're talking about is that when people think about 'I need the flexibility of the cloud' - it's the ability to create applications and run them in a cloud environment that's far more flexible,” explained Matthew Finnie, CTO of Interoute, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
Join us at Cloud Expo June 6-8 to find out how to securely connect your cloud app to any cloud or on-premises data source – without complex firewall changes. More users are demanding access to on-premises data from their cloud applications. It’s no longer a “nice-to-have” but an important differentiator that drives competitive advantages. It’s the new “must have” in the hybrid era. Users want capabilities that give them a unified view of the data to get closer to customers and grow business. The...
"Loom is applying artificial intelligence and machine learning into the entire log analysis process, from start to finish and at the end you will get a human touch,” explained Sabo Taylor Diab, Vice President, Marketing at Loom Systems, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
You know you need the cloud, but you’re hesitant to simply dump everything at Amazon since you know that not all workloads are suitable for cloud. You know that you want the kind of ease of use and scalability that you get with public cloud, but your applications are architected in a way that makes the public cloud a non-starter. You’re looking at private cloud solutions based on hyperconverged infrastructure, but you’re concerned with the limits inherent in those technologies.
"We focus on composable infrastructure. Composable infrastructure has been named by companies like Gartner as the evolution of the IT infrastructure where everything is now driven by software," explained Bruno Andrade, CEO and Founder of HTBase, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
Artificial intelligence, machine learning, neural networks. We’re in the midst of a wave of excitement around AI such as hasn’t been seen for a few decades. But those previous periods of inflated expectations led to troughs of disappointment. Will this time be different? Most likely. Applications of AI such as predictive analytics are already decreasing costs and improving reliability of industrial machinery. Furthermore, the funding and research going into AI now comes from a wide range of com...
In this presentation, Striim CTO and founder Steve Wilkes will discuss practical strategies for counteracting fraud and cyberattacks by leveraging real-time streaming analytics. In his session at @ThingsExpo, Steve Wilkes, Founder and Chief Technology Officer at Striim, will provide a detailed look into leveraging streaming data management to correlate events in real time, and identify potential breaches across IoT and non-IoT systems throughout the enterprise. Strategies for processing massive ...
SYS-CON Events announced today that GrapeUp, the leading provider of rapid product development at the speed of business, will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place October 31-November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Grape Up is a software company, specialized in cloud native application development and professional services related to Cloud Foundry PaaS. With five expert teams that operate in various sectors of the market acr...
SYS-CON Events announced today that Ayehu will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on October 31 - November 2, 2017 at the Santa Clara Convention Center in Santa Clara California. Ayehu provides IT Process Automation & Orchestration solutions for IT and Security professionals to identify and resolve critical incidents and enable rapid containment, eradication, and recovery from cyber security breaches. Ayehu provides customers greater control over IT infras...
Internet of @ThingsExpo, taking place October 31 - November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA, is co-located with 21st Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world. The Internet of Things (IoT) is the most profound change in personal and enterprise IT since the creation of the Worldwide Web more than 20 years ago. All major researchers estimate there will be tens of billions devic...
SYS-CON Events announced today that MobiDev, a client-oriented software development company, will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place October 31-November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. MobiDev is a software company that develops and delivers turn-key mobile apps, websites, web services, and complex software systems for startups and enterprises. Since 2009 it has grown from a small group of passionate engineers and business...