Welcome!

@DXWorldExpo Authors: Zakia Bouachraoui, Pat Romanski, Yeshim Deniz, Elizabeth White, Liz McMillan

Related Topics: @CloudExpo, Microservices Expo, Containers Expo Blog, Cloud Security, @DXWorldExpo, SDN Journal

@CloudExpo: Blog Post

The Paradox of Ephemeral Cloud Storage | @CloudExpo [#Cloud]

The moral of the story here is simple: if you put anything beyond your base OS on ephemeral storage, you are at great risk

The very name is kind of ridiculous, don't you think? The word "ephemeral" means it can go away. It's temporary. Fleeting, even. So why would I want to depend on storing something in a medium that can disappear without warning? And why am I forced to buy more of it when all I want is more CPUs or RAM?

Welcome to the paradox of ephemeral storage from cloud computing providers.

Origins and Explanations
Ephemeral storage exists only because of how first-generation cloud providers chunk up servers. The business model is simple: they buy a physical server and try to sell as many virtual machines (VMs) as possible on top of that physical server. Since the VMs are trapped on physical machines in this approach, first-generation providers dictate cookie-cutter sizes that make that stacking game easier for themselves.

In the process, though, these providers can't do anything to improve the redundancy of the disk on the physical servers, and are thus unable to offer guarantees on its availability. Instead they tell you not to trust it. It can evaporate. "Code around it instead" is what we are told.

If I can't trust it, how come I'm forced to buy more of it when I want bigger VM dimensions in other places, seeing as I probably only need 10GB for my operating system anyway? Consider the sizing chart below from PlanForCloud:

Take a look at that largest size. Who wants a 1.6 TB cloud storage liability?

Google Compute Engine and ProfitBricks Bring Sanity
One of the great features of Google Compute Engine is its approach to ephemeral storage. Google refers to this as Scratch Storage and in many cases limits each machine to 10 GB of it. That's just enough to build a base operating system upon, and that's obviously on purpose. Kudos to them.

ProfitBricks takes this a step further by not offering ephemeral storage at all. Instead, the physical servers housing the CPU cores and the RAM are on a separate pool of resources from the disk array that provides the block storage. Good IOPS is maintained by connecting the two with an 80 Gbps InfiniBand network. In the ProfitBricks model, all storage is akin to highly-available redundant block storage.

What You Really Want Is Block Storage
One of the things that public cloud noobs have a hard time getting their heads around at first is the difference between ephemeral storage and block storage. The latter, which every IaaS vendor offers, has some level of redundancy built into it and is where data should really be stored. Below are examples of how several vendors approach that redundancy, with better resulting availability:

Vendor

Block Volume Redundancy

Max Volume Size

AWS

"multiple servers in an Availability Zone"

1 TB

Azure

Offer both locally redundant and geographically redundant

1 TB

GCE

"replicated for additional redundancy"

10 TB

ProfitBricks

Double redundant RAID 10 across two Availability Zones

16 TB

Lessons Learned
The moral of the story here is simple: if you put anything beyond your base OS on ephemeral storage, you are at great risk. That data could go away at any time. You can't depend on it, so don't use it unless you add in an additional form of redundancy at your own engineering expense. Data you care about belongs on block storage: it has built-in redundancy and improved availability, which ensure that the data you care about will be there when you need it.

More Stories By Pete Johnson

Pete Johnson is senior director of product marketing at CliQr Technologies, where he focuses on the support of applications running on OpenStack based clouds. He is interested in the long-term management of applications in public and private clouds, and avoiding vendor lock-in. Prior to joining CliQr, Pete was senior director of platform evangelism at ProfitBricks after spending 19 years with HP as a heads-down developer, technical lead and chief architect.

Comments (1)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


DXWorldEXPO Digital Transformation Stories
Dynatrace is an application performance management software company with products for the information technology departments and digital business owners of medium and large businesses. Building the Future of Monitoring with Artificial Intelligence. Today we can collect lots and lots of performance data. We build beautiful dashboards and even have fancy query languages to access and transform the data. Still performance data is a secret language only a couple of people understand. The more busine...
Nicolas Fierro is CEO of MIMIR Blockchain Solutions. He is a programmer, technologist, and operations dev who has worked with Ethereum and blockchain since 2014. His knowledge in blockchain dates to when he performed dev ops services to the Ethereum Foundation as one the privileged few developers to work with the original core team in Switzerland.
René Bostic is the Technical VP of the IBM Cloud Unit in North America. Enjoying her career with IBM during the modern millennial technological era, she is an expert in cloud computing, DevOps and emerging cloud technologies such as Blockchain. Her strengths and core competencies include a proven record of accomplishments in consensus building at all levels to assess, plan, and implement enterprise and cloud computing solutions. René is a member of the Society of Women Engineers (SWE) and a m...
ICC is a computer systems integrator and server manufacturing company focused on developing products and product appliances to meet a wide range of computational needs for many industries. Their solutions provide benefits across many environments, such as datacenter deployment, HPC, workstations, storage networks and standalone server installations. ICC has been in business for over 23 years and their phenomenal range of clients include multinational corporations, universities, and small busines...
DXWorldEXPO LLC announced today that Nutanix has been named "Platinum Sponsor" of CloudEXPO | DevOpsSUMMIT | DXWorldEXPO New York, which will take place November 12-13, 2018 in New York City. Nutanix makes infrastructure invisible, elevating IT to focus on the applications and services that power their business. The Nutanix Enterprise Cloud Platform blends web-scale engineering and consumer-grade design to natively converge server, storage, virtualization and networking into a resilient, softwar...
Daniel Jones is CTO of EngineerBetter, helping enterprises deliver value faster. Previously he was an IT consultant, indie video games developer, head of web development in the finance sector, and an award-winning martial artist. Continuous Delivery makes it possible to exploit findings of cognitive psychology and neuroscience to increase the productivity and happiness of our teams.
Serveless Architectures brings the ability to independently scale, deploy and heal based on workloads and move away from monolithic designs. From the front-end, middle-ware and back-end layers, serverless workloads potentially have a larger security risk surface due to the many moving pieces. This talk will focus on key areas to consider for securing end to end, from dev to prod. We will discuss patterns for end to end TLS, session management, scaling to absorb attacks and mitigation techniques.
Andrew Keys is Co-Founder of ConsenSys Enterprise. He comes to ConsenSys Enterprise with capital markets, technology and entrepreneurial experience. Previously, he worked for UBS investment bank in equities analysis. Later, he was responsible for the creation and distribution of life settlement products to hedge funds and investment banks. After, he co-founded a revenue cycle management company where he learned about Bitcoin and eventually Ethereal. Andrew's role at ConsenSys Enterprise is a mul...
When building large, cloud-based applications that operate at a high scale, it’s important to maintain a high availability and resilience to failures. In order to do that, you must be tolerant of failures, even in light of failures in other areas of your application. “Fly two mistakes high” is an old adage in the radio control airplane hobby. It means, fly high enough so that if you make a mistake, you can continue flying with room to still make mistakes. In his session at 18th Cloud Expo, Lee A...
Whenever a new technology hits the high points of hype, everyone starts talking about it like it will solve all their business problems. Blockchain is one of those technologies. According to Gartner's latest report on the hype cycle of emerging technologies, blockchain has just passed the peak of their hype cycle curve. If you read the news articles about it, one would think it has taken over the technology world. No disruptive technology is without its challenges and potential impediments t...