Click here to close now.


@BigDataExpo Authors: Yeshim Deniz, Elizabeth White, Marc Crespi, Carmen Gonzalez, Liz McMillan

Related Topics: @BigDataExpo, Cloud Security, SDN Journal

@BigDataExpo: Article

The Role of Big Data in Threat Detection

The promise of Big Data lies in the ability to derive insight, reduce risk, and improve future security performance

Arriving at Actionable Insight: The Role of Big Data in Threat Detection

According to Gartner, Big Data refers to "high volume, high velocity, and/or high variety information assets" - and, this is the key - "that require new forms of processing to enable enhanced decision making, insight discovery and process optimization." While Big Data may seem like an invaluable tool that all security teams should try to leverage, it is not practical for everyone to attempt to harness it on their own. Finding insight from data is rarely as simple as it seems. We are still in the early stages of the Big Data revolution, with people only now beginning to understand what is possible, and what it takes to get there. Simply investing in tools and development is not enough. The fact is security teams are still struggling to identify and respond to incidents in an effective way. The Verizon Data Breach Investigations Report of 2013 noted that outside parties, whether it be a telecom provider, credit card issuer, third-party vendor or the FBI, were responsible for 70% of data breach notification, demonstrating that security teams are still missing the signs of detrimental threats that face organizations each and every day.

There is clearly promise in Big Data, but how do organizations get there? First, there is a need for human talent and expertise, as tools alone are not enough. Beyond creating a security operations center (SOC) to coordinate a cyber security strategy, it is critical for organizations to employ a data scientist or someone who is capable of consuming and analyzing the information to create effective models for identifying threats. Unfortunately, there is a vast talent gap in the field of data science, particularly at the intersection of data and security. There are also technical hurdles preventing development. Integrating high volumes and varieties of data sources and formats, both internal and external, into a security framework requires both technical expertise and resources. In many ways, these programs are for select organizations. Enterprises must facilitate financial resources, technical know-how, and data science expertise to execute a holistic and effective Big Data program.

The promise of Big Data lies not in the collection of millions of records, but in the ability to derive insight, reduce risk, and improve future security performance. So, if the barriers to an internal Big Data program are high, but the potential benefits are great, how do we arrive at the insight needed to reduce risk and improve future security performance? Fortunately, there are options available for organizations to gain these advantages without having to make the commitment to a full-scale Big Data program.

One emerging option is the possibility of Big Data as a Service (BDaaS). Through the perimeterless nature of the Internet, vendors can access, analyze, and provide actionable insight into potential - or even future - threats. For example, card issuers often turn to outside vendors for Common Point of Purchase (CPP) analysis to detect potential fraud associated with theft or breach. By outsourcing the collection and analysis, businesses can streamline their path to insight.

Organizations of all sizes face challenges of data collection and analysis on a daily basis. In order to gain insight from data, companies must invest in the tools, strategies, and staff needed to make sense of accessible information. Once the appropriate protocol is in place, insight from Big Data may function as a way of reducing risk, protecting enterprises from our hostile threat landscape.

More Stories By Stephen Boyer

Stephen Boyer cofounded BitSight in 2011 and serves as Chief Technology Officer. Prior to founding BitSight, he was President and Cofounder of Saperix, a company spun out of the MIT Lincoln Laboratory focused on vulnerability and network topology risk analysis. Saperix was acquired by FireMon in 2011.

While at the MIT Lincoln Laboratory, Stephen was a member of the Cyber Systems and Technology Group where he led R&D programs solving large-scale national cybersecurity problems. His work at the MIT Lincoln Laboratory included research, development, and evaluation of next generation intrusion detection correlation architectures, attack graph vulnerability analysis, large-scale cyber situational awareness, security risk measurement, and cyber simulation and testing.

Prior to joining the MIT Lincoln Laboratory, Stephen designed, developed, and tested products at one of the earliest Linux startup companies, Caldera Systems.

Stephen holds a Bachelors degree in Computer Science from Brigham Young University and Master of Science in Engineering and Management from the Massachusetts Institute of Technology.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.

@BigDataExpo Stories
There are so many tools and techniques for data analytics that even for a data scientist the choices, possible systems, and even the types of data can be daunting. In his session at @ThingsExpo, Chris Harrold, Global CTO for Big Data Solutions for EMC Corporation, will show how to perform a simple, but meaningful analysis of social sentiment data using freely available tools that take only minutes to download and install. Participants will get the download information, scripts, and complete en...
Too often with compelling new technologies market participants become overly enamored with that attractiveness of the technology and neglect underlying business drivers. This tendency, what some call the “newest shiny object syndrome,” is understandable given that virtually all of us are heavily engaged in technology. But it is also mistaken. Without concrete business cases driving its deployment, IoT, like many other technologies before it, will fade into obscurity.
SYS-CON Events announced today that Super Micro Computer, Inc., a global leader in high-performance, high-efficiency server, storage technology and green computing, will exhibit at the 17th International Cloud Expo®, which will take place on November 3–5, 2015, at the Santa Clara Convention Center in Santa Clara, CA. Supermicro (NASDAQ: SMCI), the leading innovator in high-performance, high-efficiency server technology is a premier provider of advanced server Building Block Solutions® for Data ...
SYS-CON Events announced today the Containers & Microservices Bootcamp, being held November 3-4, 2015, in conjunction with 17th Cloud Expo, @ThingsExpo, and @DevOpsSummit at the Santa Clara Convention Center in Santa Clara, CA. This is your chance to get started with the latest technology in the industry. Combined with real-world scenarios and use cases, the Containers and Microservices Bootcamp, led by Janakiram MSV, a Microsoft Regional Director, will include presentations as well as hands-on...
Achim Weiss is Chief Executive Officer and co-founder of ProfitBricks. In 1995, he broke off his studies to co-found the web hosting company "Schlund+Partner." The company "Schlund+Partner" later became the 1&1 web hosting product line. From 1995 to 2008, he was the technical director for several important projects: the largest web hosting platform in the world, the second largest DSL platform, a video on-demand delivery network, the largest eMail backend in Europe, and a universal billing syste...
Cloud computing delivers on-demand resources that provide businesses with flexibility and cost-savings. The challenge in moving workloads to the cloud has been the cost and complexity of ensuring the initial and ongoing security and regulatory (PCI, HIPAA, FFIEC) compliance across private and public clouds. Manual security compliance is slow, prone to human error, and represents over 50% of the cost of managing cloud applications. Determining how to automate cloud security compliance is critical...
Electric power utilities face relentless pressure on their financial performance, and reducing distribution grid losses is one of the last untapped opportunities to meet their business goals. Combining IoT-enabled sensors and cloud-based data analytics, utilities now are able to find, quantify and reduce losses faster – and with a smaller IT footprint. Solutions exist using Internet-enabled sensors deployed temporarily at strategic locations within the distribution grid to measure actual line lo...
The Internet of Everything is re-shaping technology trends–moving away from “request/response” architecture to an “always-on” Streaming Web where data is in constant motion and secure, reliable communication is an absolute necessity. As more and more THINGS go online, the challenges that developers will need to address will only increase exponentially. In his session at @ThingsExpo, Todd Greene, Founder & CEO of PubNub, will explore the current state of IoT connectivity and review key trends an...
There will be 20 billion IoT devices connected to the Internet soon. What if we could control these devices with our voice, mind, or gestures? What if we could teach these devices how to talk to each other? What if these devices could learn how to interact with us (and each other) to make our lives better? What if Jarvis was real? How can I gain these super powers? In his session at 17th Cloud Expo, Chris Matthieu, co-founder and CTO of Octoblu, will show you!
With major technology companies and startups seriously embracing IoT strategies, now is the perfect time to attend @ThingsExpo in Silicon Valley. Learn what is going on, contribute to the discussions, and ensure that your enterprise is as "IoT-Ready" as it can be! Internet of @ThingsExpo, taking place Nov 3-5, 2015, at the Santa Clara Convention Center in Santa Clara, CA, is co-located with 17th Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading in...
SYS-CON Events announced today that Sandy Carter, IBM General Manager Cloud Ecosystem and Developers, and a Social Business Evangelist, will keynote at the 17th International Cloud Expo®, which will take place on November 3–5, 2015, at the Santa Clara Convention Center in Santa Clara, CA.
The Internet of Things (IoT) is growing rapidly by extending current technologies, products and networks. By 2020, Cisco estimates there will be 50 billion connected devices. Gartner has forecast revenues of over $300 billion, just to IoT suppliers. Now is the time to figure out how you’ll make money – not just create innovative products. With hundreds of new products and companies jumping into the IoT fray every month, there’s no shortage of innovation. Despite this, McKinsey/VisionMobile data...
The IoT market is on track to hit $7.1 trillion in 2020. The reality is that only a handful of companies are ready for this massive demand. There are a lot of barriers, paint points, traps, and hidden roadblocks. How can we deal with these issues and challenges? The paradigm has changed. Old-style ad-hoc trial-and-error ways will certainly lead you to the dead end. What is mandatory is an overarching and adaptive approach to effectively handle the rapid changes and exponential growth.
Today air travel is a minefield of delays, hassles and customer disappointment. Airlines struggle to revitalize the experience. GE and M2Mi will demonstrate practical examples of how IoT solutions are helping airlines bring back personalization, reduce trip time and improve reliability. In their session at @ThingsExpo, Shyam Varan Nath, Principal Architect with GE, and Dr. Sarah Cooper, M2Mi's VP Business Development and Engineering, will explore the IoT cloud-based platform technologies driv...
The IoT is upon us, but today’s databases, built on 30-year-old math, require multiple platforms to create a single solution. Data demands of the IoT require Big Data systems that can handle ingest, transactions and analytics concurrently adapting to varied situations as they occur, with speed at scale. In his session at @ThingsExpo, Chad Jones, chief strategy officer at Deep Information Sciences, will look differently at IoT data so enterprises can fully leverage their IoT potential. He’ll sha...
Redis is not only the fastest database, but it has become the most popular among the new wave of applications running in containers. Redis speeds up just about every data interaction between your users or operational systems. In his session at 17th Cloud Expo, Dave Nielsen, Developer Relations at Redis Labs, will share the functions and data structures used to solve everyday use cases that are driving Redis' popularity
The enterprise is being consumerized, and the consumer is being enterprised. Moore's Law does not matter anymore, the future belongs to business virtualization powered by invisible service architecture, powered by hyperscale and hyperconvergence, and facilitated by vertical streaming and horizontal scaling and consolidation. Both buyers and sellers want instant results, and from paperwork to paperless to mindless is the ultimate goal for any seamless transaction. The sweetest sweet spot in innov...
SYS-CON Events announced today that DataClear Inc. will exhibit at the 17th International Cloud Expo®, which will take place on November 3–5, 2015, at the Santa Clara Convention Center in Santa Clara, CA. The DataClear ‘BlackBox’ is the only solution that moves your PC, browsing and data out of the United States and away from prying (and spying) eyes. Its solution automatically builds you a clean, on-demand, virus free, new virtual cloud based PC outside of the United States, and wipes it clean...
SYS-CON Events announced today that Machkey International Company will exhibit at the 17th International Cloud Expo®, which will take place on November 3–5, 2015, at the Santa Clara Convention Center in Santa Clara, CA. Machkey provides advanced connectivity solutions for just about everyone. Businesses or individuals, Machkey is dedicated to provide high-quality and cost-effective products to meet all your needs.
SYS-CON Events announced today that Dyn, the worldwide leader in Internet Performance, will exhibit at SYS-CON's 17th International Cloud Expo®, which will take place on November 3-5, 2015, at the Santa Clara Convention Center in Santa Clara, CA. Dyn is a cloud-based Internet Performance company. Dyn helps companies monitor, control, and optimize online infrastructure for an exceptional end-user experience. Through a world-class network and unrivaled, objective intelligence into Internet condit...

Tweets by @BigDataExpo

@BigDataExpo Blogs
Disaster recovery (DR) has traditionally been a major challenge for IT departments. Even with the advent of server virtualization and other technologies that have simplified DR implementation and some aspects of on-going management, it is still a complex and (often extremely) costly undertaking. For those applications that do not require high availability, but are still mission- and business-critical, the decision as to which [applications] to spend money on for true disaster recovery can be a struggle.
SCOPE is an acronym for Structured Computations Optimized for Parallel Execution, a declarative language for working with large-scale data. It is still under development at Microsoft. If you know SQL then working with SCOPE will be quite easy as SCOPE builds on SQL. The execution environment is different from that RDBMS oriented data. Data is still modeled as rows. Every row has typed columns and eveyr rowset has a well-defined schema. There is a SCOPe compiler that comes up with optimized execution plan and a runtime execution plan.
If you’re running Big Data applications, you’re going to want to look at some kind of distributed processing system. Hadoop is one of the best-known clustering systems, but how are you going to process all your data in a reasonable time frame? MapReduce has become a standard, perhaps the standard, for distributed file systems. While it’s a great system already, it’s really geared toward batch use, with jobs needing to queue for later output. This can severely hamper your flexibility. What if you want to explore some of your data? If it’s going to take all night, forget about it.
Too many multinational corporations delete little, if any, data even though at its creation, more than 70 percent of this data is useless for business, regulatory or legal reasons.[1] The problem is hoarding, and what businesses need is their own “Hoarders” reality show about people whose lives are driven by their stuff[2] (corporations are legally people, after all). The goal of such an intervention (and this article)? Turning hoarders into collectors.
Today’s modern day industrial revolution is being shaped by ubiquitous connectivity, machine to machine (M2M) communications, the Internet of Things (IoT), open APIs leading to a surge in new applications and services, partnerships and eventual marketplaces. IoT has the potential to transform industry and society much like advances in steam technology, transportation, mass production and communications ushered in the industrial revolution in the 18th and 19th centuries.
Today’s connected world is moving from devices towards things, what this means is that by using increasingly low cost sensors embedded in devices we can create many new use cases. These span across use cases in cities, vehicles, home, offices, factories, retail environments, worksites, health, logistics, and health. These use cases rely on ubiquitous connectivity and generate massive amounts of data at scale. These technologies enable new business opportunities, ways to optimize and automate, along with new ways to engage with users.
“Cloud computing is a model for enabling ubiquitous, convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications and services) that can be rapidly provisioned and released with minimal management.” While this definition is broadly accepted and has, in fact, been my adopted standard for years, it only describes technical aspects of cloud computing. The amalgamation of technologies used to deliver cloud services is not even half the story. Above all else, the successful employment requires a tight linkage to the econ...
DevOps Summit at Cloud Expo 2014 Silicon Valley was a terrific event for us. The Qubell booth was crowded on all three days. We ran demos every 30 minutes with folks lining up to get a seat and usually standing around. It was great to meet and talk to over 500 people! My keynote was well received and so was Stan's joint presentation with RingCentral on Devops for BigData. I also participated in two Power Panels – ‘Women in Technology’ and ‘Why DevOps Is Even More Important than You Think,’ both featuring brilliant colleagues and moderators and it was a blast to be a part of.
Today air travel is a minefield of delays, hassles and customer disappointment. Airlines struggle to revitalize the experience. GE and M2Mi will demonstrate practical examples of how IoT solutions are helping airlines bring back personalization, reduce trip time and improve reliability. In their session at @ThingsExpo, Shyam Varan Nath, Principal Architect with GE, and Dr. Sarah Cooper, M2Mi's VP Business Development and Engineering, will explore the IoT cloud-based platform technologies driving this change including privacy controls, data transparency and integration of real time context w...
Developing software for the Internet of Things (IoT) comes with its own set of challenges. Security, privacy, and unified standards are a few key issues. In addition, each IoT product is comprised of at least three separate application components: the software embedded in the device, the backend big-data service, and the mobile application for the end user's controls. Each component is developed by a different team, using different technologies and practices, and deployed to a different stack/target - this makes the integration of these separate pipelines and the coordination of software upd...
All we need to do is have our teams self-organize, and behold! Emergent design and/or architecture springs up out of the nothingness! If only it were that easy, right? I follow in the footsteps of so many people who have long wondered at the meanings of such simple words, as though they were dogma from on high. Emerge? Self-organizing? Profound, to be sure. But what do we really make of this sentence?
It’s not hard to find technology trade press commentary on the subject of Big Data. Variously defined (in non-technical terms) as the cluttered old shoebox of all data – and again (in more technical terms) as that amount of data that does not comfortably fit into a standard relational database for storage, processing and analytics within the normal constraints of processing, memory and data transport technologies – we can say that Big Data is an oft mentioned and sometimes misunderstood subject.
Recently announced Azure Data Lake addresses the big data 3V challenges; volume, velocity and variety. It is one more storage feature in addition to blobs and SQL Azure database. Azure Data Lake (should have been Azure Data Ocean IMHO) is really omnipotent. Just look at the key capabilities of Azure Data Lake:
I was recently watching one of my favorite science fiction TV shows (I’ll confess, ‘Dr. Who’). In classic dystopian fashion, there was a scene in which a young boy is running for his life across some barren ground in a war-ravaged world. One of his compatriots calls out to him to freeze, not to move another inch. The compatriot warns the young boy that he’s in a field of hand mines (no, that is not a typo, he did say hand mines). Slowly, dull gray hands with eyes in the palm start emerging from the ground around the boy and the compatriot. Suddenly, one of the hands grabs the compatriot and pu...
Organizations already struggle with the simple collection of data resulting from the proliferation of IoT, lacking the right infrastructure to manage it. They can't only rely on the cloud to collect and utilize this data because many applications still require dedicated infrastructure for security, redundancy, performance, etc. In his session at 17th Cloud Expo, Emil Sayegh, CEO of Codero Hosting, will discuss how in order to resolve the inherent issues, companies need to combine dedicated and cloud solutions through hybrid hosting – a sustainable solution for the data required to manage I...

About @BigDataExpo
Big Data focuses on how to use your own enterprise data – processed in the Cloud – most effectively to drive value for your business.