Welcome!

@DXWorldExpo Authors: Elizabeth White, Yeshim Deniz, Liz McMillan, Pat Romanski, Kevin Benedict

Related Topics: @DXWorldExpo, Machine Learning , @ThingsExpo

@DXWorldExpo: Blog Feed Post

Natural Language Processing | @BigDataExpo #BigData #Analytics #DataScience

Bridging the Gap Between Big Data and Big Information

Apophenia is the propensity to see patterns in random data.”  We encounter it all the time in the real world. Examples include gamblers who see patterns in how the cards are being dealt or investors who imagine patterns in the movement of certain stocks, or basketball fans who believe that their favorite player has the “hot hand.” But apophenia has no place in the world of data science, especially when data science is trying to help us make better decisions about critical things such as the quality of healthcare, where to allocate police resources, ensuring that our airplanes operate effectively or making investment decisions that determine our retirement readiness.

Understanding the differences between epiphany (a sudden, intuitive perception of or insight into the reality) and apophenia (the perception of or belief in connectedness among unrelated phenomena) is critical as data scientists build analytic models to quantify cause and effect. Regression Modeling is a great tool in helping to quantify cause and effect, but one still needs to leverage industry insights and sometimes just plain common sense to make sure that we are not trying to quantify “spurious relationships.” In the world of data science, one cannot automate out the importance of common sense. I think Craig Wilkey nails this distinction in this blog he wrote:

Apophenia is the propensity to see patterns in random data. It was first coined in 1958 by Klaus Conrad – a German neurologist and psychiatrist who, perhaps a little ironically, was attempting to identify early indicators of psychosis.

An apophany (an instance of apophenia) can best be defined in contrast to an epiphany. An epiphany is a moment of sudden and striking realization that leads a person to a greater degree of clarity in the nature of reality – a discovery of a truism, often hidden in plain sight. An apophany is having the experience of an epiphany, but you’re just plain wrong.

We’ve all heard some version of the old adage that correlation does not imply causation. It can be clearly demonstrated that in neighborhoods where there is an increase in ice cream consumption, there is a roughly equivalent spike in aggravated assault incidents. We’d be foolish to assume that eating ice cream makes people irrationally violent, but that doesn’t mean there’s nothing valuable to learn from this. When we broaden the lens a bit and include other variables, the connections become clearer.

In overpopulated urban environments, where there is a greater concentration of disenfranchised people – people who are statistically more likely to commit poverty crimes, and statistically less likely to have air-conditioned homes – heat waves usher in higher levels of frustration, lower levels of tolerance, and more people eating ice cream. We can also sharpen the focus by throwing in aggravation over public transit failures, brown-outs and black-outs, lower productivity, and countless other factors.

So, while enjoying tasty dairy products does not necessarily incite violence, the correlation between ice cream consumption and violence is not meaningless. Ice cream consumption analysis may indeed provide value as a leading indicator, or bellwether, of the potential for violent acts trending upward in a given community. If not a bellwether, it certainly is a valid correlation – as opposed to a simple coincidence.

The purpose of regression analysis is to identify those variables (referred to as independent variables) that help reveal valid correlations in the phenomena one is attempting to predict (the dependent variable).

Regression analysis is a tricky beast to harness. When the whole point is to find hidden correlations that may even defy intuitive understanding, it can be tempting to throw in the entire kitchen sink and see what comes out. The greatest perceived risk in that arises from patterns that may align, but are nevertheless invalid. These coincidences are referred to as ‘spurious relationships’.

If the patterns of some spurious relationship(s) happen to align with the patterns of other independent variables in a regression analysis model, the accuracy of the model will be impacted, and could be dramatically impacted.

It would be foolish to place any faith in all those quirky coincidences we always hear about with sports teams, for example. There is no reasonably conceivable way the first initial of the middle name of the first child born in some small town after the start of a sport’s season could predict the outcome of a team’s playoff standings – but I’d be genuinely surprised if there weren’t some spurious relationship to be found there.

On the other hand, we do have a valid argument for getting rid of the dramatic orchestra strike that foreshadows violent crime in movies, and replacing it with the sound of an ice cream truck.

How do we strike the balance between the desire to uncover hidden variables that provide valuable insight into trends, and the fear of creating an apophenic, potentially psychotic, regression analysis model?

My nearly two and a half decades of experience in IT have led me to the conclusion that the field suffers from rampant apopheniphobia: The irrational fear of finding ostensibly meaningful patterns in random data. (Yes, I did just make that word up. © Craig Wilkey, 2016)

Almost invariably, we simply do not push far enough. Should stock market analysis include things like weather patterns, celebrity news stories and grade school holidays? Absolutely! Classical stock market analysis techniques don’t work as well as they used to. Why? Frankly, we have a greater number of ignorant people playing the market. The proliferation of ‘Day Traders’ has crippled the old market truisms, because so many people who are affecting the market dynamics don’t have any classical training. The things that affect the moods and daily lives of ‘normal people’ need to be considered, because ‘normal people’ are far more active in the markets than they used to be. If they don’t play by the rules, then some of those rules simply cease to apply.

Apopheniphobia is fueled by fears of falling prey to spurious relationships. Who wants to be known as the person who unleashed a dangerously psychotic algorithm into the world? People think about the many statistical oddities they’ve come across, and it stunts their creative growth. For example, did you know there is a direct correlation between the per capita consumption of margarine and the divorce rate in Maine? Cheese consumption is far more dangerous than margarine consumption – it correlates with the number of people who die by becoming tangled in their bed sheets. (And you thought lactose intolerance was bad?) The number of people who drowned by falling into a pool also correlates with the number of films Nicholas Cage appeared in from 1999 through 2009. (Source)

In IT, we have a tendency to drive toward ‘proving’ clear, unambiguous relationships that quantify efforts, justify means and, more often than not, clearly align to our own preconceived notions. We want to be able to show clear lines of progression and indisputably direct relationships – we tend to believe anything less will not be trusted by those who hold the purse strings.

Our hyper-rational modes of thinking have a tendency to overshadow our creative imaginations – which, almost inevitably, leads to hampered understanding. Perhaps the greatest value of regression analysis is that it allows us to challenge our preconceived notions and learn something new. The greatest challenge with it is rarely throwing too much data at our models – it’s not having enough.

Yes, I know… We’re IT. We’re awash with data. We’re swimming in lakes of data and constantly inhaling the fumes of endless data exhaust. What we’re missing is the meaningful data extracted from unstructured information sources – in other words, the extraordinarily valuable information that’s locked away in language that has historically been inaccessible to machines – human language.

Estimates have been telling us for a decade or more that 80% of all information in a given organization is in the form of unstructured, human-readable text. I think there is nowhere that rings more true or significant than in trying to understand customer experience. I’d also argue that the majority of the most important service information is within that 80%.

Customer Experience Personalization absolutely depends on translating that human-readable text to machine-actionable data. When it comes to understanding and deriving value from actionable insights within our customer interactions, we must extract as much understanding from that unstructured text as possible and add it all to the other data in our regression models. Apopheniphobia be damned!

While it’s, admittedly, an oversimplification, it’s convenient to talk about two general approaches to extracting data from text. Text Analytics/Mining breaks the textual input into digestible chunks of string variables and uses statistical modeling techniques to find patterns in those variables. The ideal of Natural Language Processing is to develop a translation engine between human language and machine language. It uses some of the same statistical modeling approaches as Text Analytics, but goes much further by applying semantic and syntactic analysis to extract meaning, intention, sentiment and key concepts (among other things) included in the text.

Our best opportunity to achieve our vision of industry-leading Customer Experience Personalization is to take advantage of Natural Language Processing. This barely scratches the surface of what’s possible. Natural Language Processing will enable us to step aggressively toward extracting real meaning from the vast amount of otherwise machine-invisible, extraordinarily valuable content we have. Using that extracted meaning, in conjunction with our structured data points, will allow us to build truly valuable regression analysis models to understand our customers like never before. Keep pushing until the model breaks, then dial it back a scosche. That is the path to progress.

Apopheniphobia is the enemy of personalization and Customer Relationship Management. This is why I’ve decided to launch the Apopheniphobia Awareness Campaign. Please spread the word! I need to come up with a design for the lapel pin… Maybe a ribbon with as many digits of pi I can squeeze on it – with all the prime digits bolded? Maybe we should schedule a charity walk… We can follow the streets in alphabetical order.

The post Natural Language Processing: Bridging the Gap Between Big Data and Big Information appeared first on InFocus Blog | Dell EMC Services.

Read the original blog entry...

More Stories By William Schmarzo

Bill Schmarzo, author of “Big Data: Understanding How Data Powers Big Business” and “Big Data MBA: Driving Business Strategies with Data Science”, is responsible for setting strategy and defining the Big Data service offerings for Dell EMC’s Big Data Practice.

As a CTO within Dell EMC’s 2,000+ person consulting organization, he works with organizations to identify where and how to start their big data journeys. He’s written white papers, is an avid blogger and is a frequent speaker on the use of Big Data and data science to power an organization’s key business initiatives. He is a University of San Francisco School of Management (SOM) Executive Fellow where he teaches the “Big Data MBA” course. Bill also just completed a research paper on “Determining The Economic Value of Data”. Onalytica recently ranked Bill as #4 Big Data Influencer worldwide.

Bill has over three decades of experience in data warehousing, BI and analytics. Bill authored the Vision Workshop methodology that links an organization’s strategic business initiatives with their supporting data and analytic requirements. Bill serves on the City of San Jose’s Technology Innovation Board, and on the faculties of The Data Warehouse Institute and Strata.

Previously, Bill was vice president of Analytics at Yahoo where he was responsible for the development of Yahoo’s Advertiser and Website analytics products, including the delivery of “actionable insights” through a holistic user experience. Before that, Bill oversaw the Analytic Applications business unit at Business Objects, including the development, marketing and sales of their industry-defining analytic applications.

Bill holds a Masters Business Administration from University of Iowa and a Bachelor of Science degree in Mathematics, Computer Science and Business Administration from Coe College.

@BigDataExpo Stories
As organizations shift towards IT-as-a-service models, the need for managing and protecting data residing across physical, virtual, and now cloud environments grows with it. Commvault can ensure protection, access and E-Discovery of your data – whether in a private cloud, a Service Provider delivered public cloud, or a hybrid cloud environment – across the heterogeneous enterprise. In his general session at 18th Cloud Expo, Randy De Meno, Chief Technologist - Windows Products and Microsoft Part...
Andi Mann, Chief Technology Advocate at Splunk, is an accomplished digital business executive with extensive global expertise as a strategist, technologist, innovator, marketer, and communicator. For over 30 years across five continents, he has built success with Fortune 500 corporations, vendors, governments, and as a leading research analyst and consultant.
To Really Work for Enterprises, MultiCloud Adoption Requires Far Better and Inclusive Cloud Monitoring and Cost Management … But How? Overwhelmingly, even as enterprises have adopted cloud computing and are expanding to multi-cloud computing, IT leaders remain concerned about how to monitor, manage and control costs across hybrid and multi-cloud deployments. It’s clear that traditional IT monitoring and management approaches, designed after all for on-premises data centers, are falling short in ...
Organizations planning enterprise data center consolidation and modernization projects are faced with a challenging, costly reality. Requirements to deploy modern, cloud-native applications simultaneously with traditional client/server applications are almost impossible to achieve with hardware-centric enterprise infrastructure. Compute and network infrastructure are fast moving down a software-defined path, but storage has been a laggard. Until now.
DXWorldEXPO LLC announced today that the upcoming DXWorldEXPO | CloudEXPO New York event will feature 10 companies from Poland to participate at the "Poland Digital Transformation Pavilion" on November 12-13, 2018.
Digital Transformation is much more than a buzzword. The radical shift to digital mechanisms for almost every process is evident across all industries and verticals. This is often especially true in financial services, where the legacy environment is many times unable to keep up with the rapidly shifting demands of the consumer. The constant pressure to provide complete, omnichannel delivery of customer-facing solutions to meet both regulatory and customer demands is putting enormous pressure on...
The best way to leverage your CloudEXPO | DXWorldEXPO presence as a sponsor and exhibitor is to plan your news announcements around our events. The press covering CloudEXPO | DXWorldEXPO will have access to these releases and will amplify your news announcements. More than two dozen Cloud companies either set deals at our shows or have announced their mergers and acquisitions at CloudEXPO. Product announcements during our show provide your company with the most reach through our targeted audienc...
With 10 simultaneous tracks, keynotes, general sessions and targeted breakout classes, @CloudEXPO and DXWorldEXPO are two of the most important technology events of the year. Since its launch over eight years ago, @CloudEXPO and DXWorldEXPO have presented a rock star faculty as well as showcased hundreds of sponsors and exhibitors!
In an era of historic innovation fueled by unprecedented access to data and technology, the low cost and risk of entering new markets has leveled the playing field for business. Today, any ambitious innovator can easily introduce a new application or product that can reinvent business models and transform the client experience. In their Day 2 Keynote at 19th Cloud Expo, Mercer Rowe, IBM Vice President of Strategic Alliances, and Raejeanne Skillern, Intel Vice President of Data Center Group and ...
More and more brands have jumped on the IoT bandwagon. We have an excess of wearables – activity trackers, smartwatches, smart glasses and sneakers, and more that track seemingly endless datapoints. However, most consumers have no idea what “IoT” means. Creating more wearables that track data shouldn't be the aim of brands; delivering meaningful, tangible relevance to their users should be. We're in a period in which the IoT pendulum is still swinging. Initially, it swung toward "smart for smart...
@DevOpsSummit at Cloud Expo, taking place November 12-13 in New York City, NY, is co-located with 22nd international CloudEXPO | first international DXWorldEXPO and will feature technical sessions from a rock star conference faculty and the leading industry players in the world.
DXWorldEXPO LLC announced today that All in Mobile, a mobile app development company from Poland, will exhibit at the 22nd International CloudEXPO | DXWorldEXPO. All In Mobile is a mobile app development company from Poland. Since 2014, they maintain passion for developing mobile applications for enterprises and startups worldwide.
In his keynote at 19th Cloud Expo, Sheng Liang, co-founder and CEO of Rancher Labs, discussed the technological advances and new business opportunities created by the rapid adoption of containers. With the success of Amazon Web Services (AWS) and various open source technologies used to build private clouds, cloud computing has become an essential component of IT strategy. However, users continue to face challenges in implementing clouds, as older technologies evolve and newer ones like Docker c...
DXWorldEXPO LLC announced today that ICC-USA, a computer systems integrator and server manufacturing company focused on developing products and product appliances, will exhibit at the 22nd International CloudEXPO | DXWorldEXPO. DXWordEXPO New York 2018, colocated with CloudEXPO New York 2018 will be held November 11-13, 2018, in New York City. ICC is a computer systems integrator and server manufacturing company focused on developing products and product appliances to meet a wide range of ...
Coca-Cola’s Google powered digital signage system lays the groundwork for a more valuable connection between Coke and its customers. Digital signs pair software with high-resolution displays so that a message can be changed instantly based on what the operator wants to communicate or sell. In their Day 3 Keynote at 21st Cloud Expo, Greg Chambers, Global Group Director, Digital Innovation, Coca-Cola, and Vidya Nagarajan, a Senior Product Manager at Google, discussed how from store operations and ...
We are seeing a major migration of enterprises applications to the cloud. As cloud and business use of real time applications accelerate, legacy networks are no longer able to architecturally support cloud adoption and deliver the performance and security required by highly distributed enterprises. These outdated solutions have become more costly and complicated to implement, install, manage, and maintain.SD-WAN offers unlimited capabilities for accessing the benefits of the cloud and Internet. ...
Headquartered in Plainsboro, NJ, Synametrics Technologies has provided IT professionals and computer systems developers since 1997. Based on the success of their initial product offerings (WinSQL and DeltaCopy), the company continues to create and hone innovative products that help its customers get more from their computer applications, databases and infrastructure. To date, over one million users around the world have chosen Synametrics solutions to help power their accelerated business or per...
As Cybric's Chief Technology Officer, Mike D. Kail is responsible for the strategic vision and technical direction of the platform. Prior to founding Cybric, Mike was Yahoo's CIO and SVP of Infrastructure, where he led the IT and Data Center functions for the company. He has more than 24 years of IT Operations experience with a focus on highly-scalable architectures.
Dion Hinchcliffe is an internationally recognized digital expert, bestselling book author, frequent keynote speaker, analyst, futurist, and transformation expert based in Washington, DC. He is currently Chief Strategy Officer at the industry-leading digital strategy and online community solutions firm, 7Summits.
Bill Schmarzo, author of "Big Data: Understanding How Data Powers Big Business" and "Big Data MBA: Driving Business Strategies with Data Science," is responsible for setting the strategy and defining the Big Data service offerings and capabilities for EMC Global Services Big Data Practice. As the CTO for the Big Data Practice, he is responsible for working with organizations to help them identify where and how to start their big data journeys. He's written several white papers, is an avid blogge...