Welcome!

@DXWorldExpo Authors: Zakia Bouachraoui, Yeshim Deniz, Carmen Gonzalez, Liz McMillan, Pat Romanski

Related Topics: @CloudExpo, Microservices Expo, Containers Expo Blog, Cloud Security, @DXWorldExpo, SDN Journal

@CloudExpo: Blog Post

The Paradox of Ephemeral Cloud Storage | @CloudExpo [#Cloud]

The moral of the story here is simple: if you put anything beyond your base OS on ephemeral storage, you are at great risk

The very name is kind of ridiculous, don't you think? The word "ephemeral" means it can go away. It's temporary. Fleeting, even. So why would I want to depend on storing something in a medium that can disappear without warning? And why am I forced to buy more of it when all I want is more CPUs or RAM?

Welcome to the paradox of ephemeral storage from cloud computing providers.

Origins and Explanations
Ephemeral storage exists only because of how first-generation cloud providers chunk up servers. The business model is simple: they buy a physical server and try to sell as many virtual machines (VMs) as possible on top of that physical server. Since the VMs are trapped on physical machines in this approach, first-generation providers dictate cookie-cutter sizes that make that stacking game easier for themselves.

In the process, though, these providers can't do anything to improve the redundancy of the disk on the physical servers, and are thus unable to offer guarantees on its availability. Instead they tell you not to trust it. It can evaporate. "Code around it instead" is what we are told.

If I can't trust it, how come I'm forced to buy more of it when I want bigger VM dimensions in other places, seeing as I probably only need 10GB for my operating system anyway? Consider the sizing chart below from PlanForCloud:

Take a look at that largest size. Who wants a 1.6 TB cloud storage liability?

Google Compute Engine and ProfitBricks Bring Sanity
One of the great features of Google Compute Engine is its approach to ephemeral storage. Google refers to this as Scratch Storage and in many cases limits each machine to 10 GB of it. That's just enough to build a base operating system upon, and that's obviously on purpose. Kudos to them.

ProfitBricks takes this a step further by not offering ephemeral storage at all. Instead, the physical servers housing the CPU cores and the RAM are on a separate pool of resources from the disk array that provides the block storage. Good IOPS is maintained by connecting the two with an 80 Gbps InfiniBand network. In the ProfitBricks model, all storage is akin to highly-available redundant block storage.

What You Really Want Is Block Storage
One of the things that public cloud noobs have a hard time getting their heads around at first is the difference between ephemeral storage and block storage. The latter, which every IaaS vendor offers, has some level of redundancy built into it and is where data should really be stored. Below are examples of how several vendors approach that redundancy, with better resulting availability:

Vendor

Block Volume Redundancy

Max Volume Size

AWS

"multiple servers in an Availability Zone"

1 TB

Azure

Offer both locally redundant and geographically redundant

1 TB

GCE

"replicated for additional redundancy"

10 TB

ProfitBricks

Double redundant RAID 10 across two Availability Zones

16 TB

Lessons Learned
The moral of the story here is simple: if you put anything beyond your base OS on ephemeral storage, you are at great risk. That data could go away at any time. You can't depend on it, so don't use it unless you add in an additional form of redundancy at your own engineering expense. Data you care about belongs on block storage: it has built-in redundancy and improved availability, which ensure that the data you care about will be there when you need it.

More Stories By Pete Johnson

Pete Johnson is senior director of product marketing at CliQr Technologies, where he focuses on the support of applications running on OpenStack based clouds. He is interested in the long-term management of applications in public and private clouds, and avoiding vendor lock-in. Prior to joining CliQr, Pete was senior director of platform evangelism at ProfitBricks after spending 19 years with HP as a heads-down developer, technical lead and chief architect.

Comments (1)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


DXWorldEXPO Digital Transformation Stories
At CloudEXPO Silicon Valley, June 24-26, 2019, Digital Transformation (DX) is a major focus with expanded DevOpsSUMMIT and FinTechEXPO programs within the DXWorldEXPO agenda. Successful transformation requires a laser focus on being data-driven and on using all the tools available that enable transformation if they plan to survive over the long term. A total of 88% of Fortune 500 companies from a generation ago are now out of business. Only 12% still survive. Similar percentages are found throug...
Every organization is facing their own Digital Transformation as they attempt to stay ahead of the competition, or worse, just keep up. Each new opportunity, whether embracing machine learning, IoT, or a cloud migration, seems to bring new development, deployment, and management models. The results are more diverse and federated computing models than any time in our history.
Data center, on-premise, public-cloud, private-cloud, multi-cloud, hybrid-cloud, IoT, AI, edge, SaaS, PaaS... it's an availability, security, performance and integration nightmare even for the best of the best IT experts. Organizations realize the tremendous benefits of everything the digital transformation has to offer. Cloud adoption rates are increasing significantly, and IT budgets are morphing to follow suit. But distributing applications and infrastructure around increases risk, introdu...
Today's workforce is trading their cubicles and corporate desktops in favor of an any-location, any-device work style. And as digital natives make up more and more of the modern workforce, the appetite for user-friendly, cloud-based services grows. The center of work is shifting to the user and to the cloud. But managing a proliferation of SaaS, web, and mobile apps running on any number of clouds and devices is unwieldy and increases security risks. PJ Hough, Citrix Executive Vice President and...
Atmosera delivers modern cloud services that maximize the advantages of cloud-based infrastructures. Offering private, hybrid, and public cloud solutions, Atmosera works closely with customers to engineer, deploy, and operate cloud architectures with advanced services that deliver strategic business outcomes. Atmosera's expertise simplifies the process of cloud transformation and our 20+ years of experience managing complex IT environments provides our customers with the confidence and trust tha...
Is your enterprise growing the right skills to fight the digital transformation (DX) battles? With 69% of enterprises describing the DX skill drought as being soft skills, rather than technology skills, are you ready to survive against disrupters? The next wave of business disruption is already crashing on your enterprise as AI, Blockchain and IoT change the nature and location of business. Now is the time to prepare. Drawing on experiences with large and midsize enterprises, Marco Coulter t...
Atmosera delivers modern cloud services that maximize the advantages of cloud-based infrastructures. Offering private, hybrid, and public cloud solutions, Atmosera works closely with customers to engineer, deploy, and operate cloud architectures with advanced services that deliver strategic business outcomes. Atmosera's expertise simplifies the process of cloud transformation and our 20+ years of experience managing complex IT environments provides our customers with the confidence and trust tha...
Where many organizations get into trouble, however, is that they try to have a broad and deep knowledge in each of these areas. This is a huge blow to an organization's productivity. By automating or outsourcing some of these pieces, such as databases, infrastructure, and networks, your team can instead focus on development, testing, and deployment. Further, organizations that focus their attention on these areas can eventually move to a test-driven development structure that condenses several l...
The graph represents a network of 1,329 Twitter users whose recent tweets contained "#DevOps", or who were replied to or mentioned in those tweets, taken from a data set limited to a maximum of 18,000 tweets. The network was obtained from Twitter on Thursday, 10 January 2019 at 23:50 UTC. The tweets in the network were tweeted over the 7-hour, 6-minute period from Thursday, 10 January 2019 at 16:29 UTC to Thursday, 10 January 2019 at 23:36 UTC. Additional tweets that were mentioned in this...
Over the course of two days, in addition to insightful conversations and presentations delving into the industry's current pressing challenges, there was considerable buzz about digital transformation and how it is enabling global enterprises to accelerate business growth. Blockchain has been a term that people hear but don't quite understand. The most common myths about blockchain include the assumption that it is private, or that there is only one blockchain, and the idea that blockchain is...