Welcome!

@DXWorldExpo Authors: Yeshim Deniz, Pat Romanski, Liz McMillan, Elizabeth White, Zakia Bouachraoui

Related Topics: @DXWorldExpo, Java IoT, Linux Containers, Open Source Cloud, SDN Journal

@DXWorldExpo: Article

Planet @Hadoop By @ABridgwater | @BigDataExpo [#BigData]

Hadoop has been called a foundational technology, rather than ‘just’ a database by some commentators

What Kinds of People Live on Planet Hadoop?

Labor market analytics firm Wanted Analytics recently assessed the market for technology professionals and found that demand for people with proficient levels of Hadoop expertise had skyrocketed by around 33% since last year - it is true, Hadoop is hard technology to master and the labor market is not exactly flooded with an over-abundance of skilled practitioners.

Hadoop has been called a foundational technology, rather than ‘just' a database by some commentators - this almost pushes it towards being an ‘environment' rather than it being a single software product... and this all goes towards making Hadoop even harder to master, many will agree.

Interestingly, we can essentially define Hadoop as a free Java-based ‘programming framework' that supports the processing of large data sets in a distributed computing environment with robust data processing and storage power.

Hadoop confusion
Is Hadoop a foundational technology, a data environment or programming framework? It is all of these things of course, but this may be where some of the challenge (and, specifically, the skills challenge) lies.

What kinds of people live on Planet Hadoop? Well, quite apart from being able to breath a noxious mixture of data, storage and Java-flavored oxygen, Hadoop native beings (we'll call them Hadoopers) are deeply innovative.

Hadoopers are the kind of people that are driven by an inherent desire to push the ‘analytics envelope' so far forward that it becomes an operational business process that in itself drives innovation - and therefore greater profits.

These are the sort of people who are prepared to help ‘industrialize the analytical process' and make it part of the way a firm operates from first principles. As Adrian Jones of SAS has put it, the analytics factory provides an approach that offers flexibility for analysts and a structured framework for governance.

"Your data preparation process should make it easy to quickly move between the two states. This step requires the creativity of the business combined with the process and operational efficiency of IT. Taking a factory approach to data preparation will remove this gray area from many organizations, reducing the conflict that comes through duplications and inefficiencies," writes Jones.

Hard-wired into Hadoop
As operational processes throughout the business start to get hard-wired into Hadoop, we see analytics deployed in a production state where it becomes part of the core workflow of all operations across the business - the truly analytical company starts to flourish from this point onward.

This Hadoop planet (or company, or environment, or biosphere) is populated by people with ability and aptitude of course; Hadoop is no place for dummies unfortunately... but what other traits to Hadoopers exhibit?

"Resourcefully creative with an ability to think on their feet," - would be a good way of describing these folk, i.e., they are capable of surveying the analytics landscape and knowing instinctively which datasets to combine, splice, analyze and focus on - but more than anything, they can perceive the kind of outcome that ‘might' be useful without actually knowing how it should initially be quantified and qualified.

A steely stoicism
Being able to convert the analytic process into an operational process requires an aptitude for practicality and a steely stoicism with the patience to wait and look for the right outcomes.

Really successful Hadoopers are corporate beasts with an appreciation for their company's regulatory processes and systems, i.e., there's not much point in producing great Hadoop analytics if it remains untamed analytics. Good corporate Hadoop citizens are capable of making Hadoop insights ‘universally accessible' so that every stakeholder in the organization gets the appropriate level of access.

As Fiona McNeill, Product Marketing, SAS, puts it, "The difference with analytically mature, innovative organizations, is that insights are universally accessible - whether you work in HR, finance, sales, logistics, marketing or services. The data is recognized as a corporate asset and analytical methods become intellectual property."

Nimble feet, fingers and foreheads
This discussion could go on and on. Good Hadoop pros exhibit many, many attributes, but one that is appropriate to finish with is nimbleness. Although this term ‘nimble' has become hackneyed, overused and overtired across the IT industry, we will bid to use it this one last time.

Hadoop is all about trial and error in terms of finding out what works best in relation to any particular analytics job. Nimble Hadoopers are unyielding in their ability to keep trying out analytics scenarios until they get what they need - or, perhaps more important, until they get what they want for the business.

These are some of the many elements currently populating the prosperous parts of Hadoop, whether we survive in this brave new world or not is dependent on our ability to be this kind of Hadooper - welcome to the new world.

This post is brought to you by SAS.

SAS is a leader in business analytics software and services and the largest independent vendor in the business intelligence market.

More Stories By Adrian Bridgwater

Adrian Bridgwater is a freelance journalist and corporate content creation specialist focusing on cross platform software application development as well as all related aspects software engineering, project management and technology as a whole.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


DXWorldEXPO Digital Transformation Stories
Today, we have more data to manage than ever. We also have better algorithms that help us access our data faster. Cloud is the driving force behind many of the data warehouse advancements we have enjoyed in recent years. But what are the best practices for storing data in the cloud for machine learning and data science applications?
DevOpsSummit New York 2018, colocated with CloudEXPO | DXWorldEXPO New York 2018 will be held November 11-13, 2018, in New York City. Digital Transformation (DX) is a major focus with the introduction of DXWorldEXPO within the program. Successful transformation requires a laser focus on being data-driven and on using all the tools available that enable transformation if they plan to survive over the long term.
Bill Schmarzo, author of "Big Data: Understanding How Data Powers Big Business" and "Big Data MBA: Driving Business Strategies with Data Science" is responsible for guiding the technology strategy within Hitachi Vantara for IoT and Analytics. Bill brings a balanced business-technology approach that focuses on business outcomes to drive data, analytics and technology decisions that underpin an organization's digital transformation strategy.
@DevOpsSummit at Cloud Expo, taking place November 12-13 in New York City, NY, is co-located with 22nd international CloudEXPO | first international DXWorldEXPO and will feature technical sessions from a rock star conference faculty and the leading industry players in the world. The widespread success of cloud computing is driving the DevOps revolution in enterprise IT. Now as never before, development teams must communicate and collaborate in a dynamic, 24/7/365 environment. There is no time t...
Headquartered in Plainsboro, NJ, Synametrics Technologies has provided IT professionals and computer systems developers since 1997. Based on the success of their initial product offerings (WinSQL and DeltaCopy), the company continues to create and hone innovative products that help its customers get more from their computer applications, databases and infrastructure. To date, over one million users around the world have chosen Synametrics solutions to help power their accelerated business or per...
DXWordEXPO New York 2018, colocated with CloudEXPO New York 2018 will be held November 11-13, 2018, in New York City and will bring together Cloud Computing, FinTech and Blockchain, Digital Transformation, Big Data, Internet of Things, DevOps, AI, Machine Learning and WebRTC to one location.
When talking IoT we often focus on the devices, the sensors, the hardware itself. The new smart appliances, the new smart or self-driving cars (which are amalgamations of many ‘things'). When we are looking at the world of IoT, we should take a step back, look at the big picture. What value are these devices providing. IoT is not about the devices, its about the data consumed and generated. The devices are tools, mechanisms, conduits. This paper discusses the considerations when dealing with the...
Charles Araujo is an industry analyst, internationally recognized authority on the Digital Enterprise and author of The Quantum Age of IT: Why Everything You Know About IT is About to Change. As Principal Analyst with Intellyx, he writes, speaks and advises organizations on how to navigate through this time of disruption. He is also the founder of The Institute for Digital Transformation and a sought after keynote speaker. He has been a regular contributor to both InformationWeek and CIO Insight...
For years the world's most security-focused and distributed organizations - banks, military/defense agencies, global enterprises - have sought to adopt cloud technologies that can reduce costs, future-proof against data growth, and improve user productivity. The challenges of cloud transformation for these kinds of secure organizations have centered around data security, migration from legacy systems, and performance. In our presentation, we will discuss the notion that cloud computing, properl...
CloudEXPO New York 2018, colocated with DXWorldEXPO New York 2018 will be held November 11-13, 2018, in New York City and will bring together Cloud Computing, FinTech and Blockchain, Digital Transformation, Big Data, Internet of Things, DevOps, AI, Machine Learning and WebRTC to one location.