Welcome!

Big Data Journal Authors: Pat Romanski, Elizabeth White, Yeshim Deniz, Roger Strukhoff, Kevin Benedict

Blog Feed Post

Well Engineered use of AWS by Recovery and Transparency Board (RATB)

By

After speaking with Shawn Kingsberry in preparations for our 4 April Government Big Data Forum I realized their use of Amazon Web Services (AWS) may be of very high interest to our readers and went about looking for more info online to see what was publicly available. I was ecstatic to see a well written use case for much of it is on the AWS website. That write-up includes a nice graphic that is helpful to understanding how things were done.

Since this is provided by AWS as a way of articulating their special contributions it does not go into the many other services and components required to make this work. But many of those components are probably modular and exchangable with other capabilities. So this overview is probably a great way to get a baseline on what the RATB architecture is.

With that, the following is from: http://aws.amazon.com/solutions/case-studies/ratb/

AWS Case Study: Recovery.gov and AWS Bring Transparency to the Cloud

The Recovery Accountability and Transparency Board (RATB) was established when Congress passed the American Recovery and Reinvestment Act (ARRA) in February, 2009. To ensure against waste, fraud, and abuse, the RATB was tasked with developing a Website which met the following goals:

  • Provide easily accessible information to the public on Recovery spending and results
  • Promote official data in public debate
  • Provide fair and open access to Recovery opportunities
  • Enable public accountability for Recovery spending
  • Promote an understanding of the local impact of Recovery spending

The resulting Website is Recovery.gov.

The RATB originally intended to use Amazon Web Services (AWS) only for development, testing, and as failover, but, says Jim Warren, RATB Chief Information Officer, “When AWS outperformed our on-premises solution at a fraction of the cost, the prime contractor Smartronix and its lead sub-contractor Synteractive, provided a compelling justification for the RATB to host Recovery.gov on AWS’s platform.”

According to Mr. Warren, Smartronix selected AWS because of the flexibility provided by AWS’s Infrastructure as a Service (IaaS) model; track record of providing infrastructure for large-scale commercial projects; focus on cost-effectiveness and a pay-as-you-go-model that allowed Smartronix to control costs; commitment to security and reliability; and its FISMA Low certification.

The RATB now uses the following AWS services: Amazon Elastic Compute Cloud (Amazon EC2), Amazon Simple Storage Service (Amazon S3), Amazon Elastic Block Storage (Amazon EBS), Elastic Load Balancing (ELB), and Amazon CloudWatch. The solution also combined multiple pieces of software.

The following diagrams illustrate their topology:

 

ratb arch diagram Well Engineered use of AWS by Recovery and Transparency Board (RATB)

Recovery Accountability and Transparency Board

Business Intelligence and Data Warehousing
The website uses Microsoft’s SharePoint as it content management system and all data is aggregated into a global dimensional data warehouse to facilitate time-based analysis and reporting. The solution leverages SAP BusinessObjects and Microsoft SQL Server for reporting services that show how and where the money is being spent. The BI tools enable ad hoc reporting and are instrumental in Data Quality and Data Integrity score-carding.

Advanced Geospatial Analysis and Mapping
The Geospatial tools, based on ESRI software, allow up to 5,000 concurrent users and enables them to go directly to go to their communities of interest at the state, zip, congressional district, or county level. Hundreds of thousands of addresses are geo-coded and aggregated to display total value for each area of interest. Thematic maps and multiple view selections were incorporated to help the user better visualize the data. These thematics include funding heat maps, unemployment heat maps, and diversity maps.

Mr. Warren notes that testing and development enclaves were procured and ready on Amazon EC2 within two days of the contract award. He says, “Our migration to the cloud took only 22 days from feasibility study to production.” The RATB has also enjoyed improved computer security, including greater protection against network attacks and real-time detection of system tampering. Mr. Warren says, “In essence, the security system of AWS’s platform has been added to our existing security systems. We now have a security posture consistent with that of a multi-billion dollar company.” Additional benefits include lower costs and ability to add capacity on demand. The RATB expects to save around $750K during their current budget cycle.

The success of Recovery.gov is being noticed outside of the RATB as well: Andre Romano of Newsweek wrote, “The current incarnation of Recovery.gov…is perhaps the clearest, richest interactive database ever produced by the American bureaucracy.” The site has been given the 2009 Merit award, the 2010 Gold Addy award for Website design, InformationWeek Government IT Innovator 2010 Award, an Award of Distinction during the 16th Annual Communicator Awards, and a second place Gold Screen Award from the National Association of Government Communicators. Recovery.gov is also an official Honoree for the Financial Services category in the 14th Annual Webby Awards.

To learn more see http://recovery.gov

 Well Engineered use of AWS by Recovery and Transparency Board (RATB)

Read the original blog entry...

More Stories By Bob Gourley

Bob Gourley, former CTO of the Defense Intelligence Agency (DIA), is Founder and CTO of Crucial Point LLC, a technology research and advisory firm providing fact based technology reviews in support of venture capital, private equity and emerging technology firms. He has extensive industry experience in intelligence and security and was awarded an intelligence community meritorious achievement award by AFCEA in 2008, and has also been recognized as an Infoworld Top 25 CTO and as one of the most fascinating communicators in Government IT by GovFresh.

Cloud Expo Latest Stories
14th International Cloud Expo, held on June 10–12, 2014 at the Javits Center in New York City, featured three content-packed days with a rich array of sessions about the business and technical value of cloud computing, Internet of Things, Big Data, and DevOps led by exceptional speakers from every sector of the IT ecosystem. The Cloud Expo series is the fastest-growing Enterprise IT event in the past 10 years, devoted to every aspect of delivering massively scalable enterprise IT as a service.
Hardware will never be more valuable than on the day it hits your loading dock. Each day new servers are not deployed to production the business is losing money. While Moore’s Law is typically cited to explain the exponential density growth of chips, a critical consequence of this is rapid depreciation of servers. The hardware for clustered systems (e.g., Hadoop, OpenStack) tends to be significant capital expenses. In his session at 15th Cloud Expo, Mason Katz, CTO and co-founder of StackIQ, to discuss how infrastructure teams should be aware of the capitalization and depreciation model of these expenses to fully understand when and where automation is critical.
Over the last few years the healthcare ecosystem has revolved around innovations in Electronic Health Record (HER) based systems. This evolution has helped us achieve much desired interoperability. Now the focus is shifting to other equally important aspects – scalability and performance. While applying cloud computing environments to the EHR systems, a special consideration needs to be given to the cloud enablement of Veterans Health Information Systems and Technology Architecture (VistA), i.e., the largest single medical system in the United States.
In his session at 15th Cloud Expo, Mark Hinkle, Senior Director, Open Source Solutions at Citrix Systems Inc., will provide overview of the open source software that can be used to deploy and manage a cloud computing environment. He will include information on storage, networking(e.g., OpenDaylight) and compute virtualization (Xen, KVM, LXC) and the orchestration(Apache CloudStack, OpenStack) of the three to build their own cloud services. Speaker Bio: Mark Hinkle is the Senior Director, Open Source Solutions, at Citrix Systems Inc. He joined Citrix as a result of their July 2011 acquisition of Cloud.com where he was their Vice President of Community. He is currently responsible for Citrix open source efforts around the open source cloud computing platform, Apache CloudStack and the Xen Hypervisor. Previously he was the VP of Community at Zenoss Inc., a producer of the open source application, server, and network management software, where he grew the Zenoss Core project to over 10...
Most of today’s hardware manufacturers are building servers with at least one SATA Port, but not every systems engineer utilizes them. This is considered a loss in the game of maximizing potential storage space in a fixed unit. The SATADOM Series was created by Innodisk as a high-performance, small form factor boot drive with low power consumption to be plugged into the unused SATA port on your server board as an alternative to hard drive or USB boot-up. Built for 1U systems, this powerful device is smaller than a one dollar coin, and frees up otherwise dead space on your motherboard. To meet the requirements of tomorrow’s cloud hardware, Innodisk invested internal R&D resources to develop our SATA III series of products. The SATA III SATADOM boasts 500/180MBs R/W Speeds respectively, or double R/W Speed of SATA II products.
As more applications and services move "to the cloud" (public or on-premise) cloud environments are increasingly adopting and building out traditional enterprise features. This in turn is enabling and encouraging cloud adoption from enterprise users. In many ways the definition is blurring as features like continuous operation, geo-distribution or on-demand capacity become the norm. NuoDB is involved in both building enterprise software and using enterprise cloud capabilities. In his session at 15th Cloud Expo, Seth Proctor, CTO at NuoDB, Inc., will discuss the experiences from building, deploying and using enterprise services and suggest some ways to approach moving enterprise applications into a cloud model.
Until recently, many organizations required specialized departments to perform mapping and geospatial analysis, and they used Esri on-premise solutions for that work. In his session at 15th Cloud Expo, Dave Peters, author of the Esri Press book Building a GIS, System Architecture Design Strategies for Managers, will discuss how Esri has successfully included the cloud as a fully integrated SaaS expansion of the ArcGIS mapping platform. Organizations that have incorporated Esri cloud-based applications and content within their business models are reaping huge benefits by directly leveraging cloud-based mapping and analysis capabilities within their existing enterprise investments. The ArcGIS mapping platform includes cloud-based content management and information resources to more widely, efficiently, and affordably deliver real-time actionable information and analysis capabilities to your organization.
Almost everyone sees the potential of Internet of Things but how can businesses truly unlock that potential. The key will be in the ability to discover business insight in the midst of an ocean of Big Data generated from billions of embedded devices via Systems of Discover. Businesses will also need to ensure that they can sustain that insight by leveraging the cloud for global reach, scale and elasticity. In his session at Internet of @ThingsExpo, Mac Devine, Distinguished Engineer at IBM, will discuss bringing these three elements together via Systems of Discover.
Cloud and Big Data present unique dilemmas: embracing the benefits of these new technologies while maintaining the security of your organization’s assets. When an outside party owns, controls and manages your infrastructure and computational resources, how can you be assured that sensitive data remains private and secure? How do you best protect data in mixed use cloud and big data infrastructure sets? Can you still satisfy the full range of reporting, compliance and regulatory requirements? In his session at 15th Cloud Expo, Derek Tumulak, Vice President of Product Management at Vormetric, will discuss how to address data security in cloud and Big Data environments so that your organization isn’t next week’s data breach headline.
The cloud is everywhere and growing, and with it SaaS has become an accepted means for software delivery. SaaS is more than just a technology, it is a thriving business model estimated to be worth around $53 billion dollars by 2015, according to IDC. The question is – how do you build and scale a profitable SaaS business model? In his session at 15th Cloud Expo, Jason Cumberland, Vice President, SaaS Solutions at Dimension Data, will give the audience an understanding of common mistakes businesses make when transitioning to SaaS; how to avoid them; and how to build a profitable and scalable SaaS business.
SYS-CON Events announced today that Gridstore™, the leader in software-defined storage (SDS) purpose-built for Windows Servers and Hyper-V, will exhibit at SYS-CON's 15th International Cloud Expo®, which will take place on November 4–6, 2014, at the Santa Clara Convention Center in Santa Clara, CA. Gridstore™ is the leader in software-defined storage purpose built for virtualization that is designed to accelerate applications in virtualized environments. Using its patented Server-Side Virtual Controller™ Technology (SVCT) to eliminate the I/O blender effect and accelerate applications Gridstore delivers vmOptimized™ Storage that self-optimizes to each application or VM across both virtual and physical environments. Leveraging a grid architecture, Gridstore delivers the first end-to-end storage QoS to ensure the most important App or VM performance is never compromised. The storage grid, that uses Gridstore’s performance optimized nodes or capacity optimized nodes, starts with as few a...
SYS-CON Events announced today that Solgenia, the global market leader in Cloud Collaboration and Cloud Infrastructure software solutions, will exhibit at SYS-CON's 15th International Cloud Expo®, which will take place on November 4–6, 2014, at the Santa Clara Convention Center in Santa Clara, CA. Solgenia is the global market leader in Cloud Collaboration and Cloud Infrastructure software solutions. Designed to “Bridge the Gap” between personal and professional social, mobile and cloud user experiences, our solutions help large and medium-sized organizations dramatically improve productivity, reduce collaboration costs, and increase the overall enterprise value by bringing collaboration and infrastructure solutions to the cloud.
Cloud computing started a technology revolution; now DevOps is driving that revolution forward. By enabling new approaches to service delivery, cloud and DevOps together are delivering even greater speed, agility, and efficiency. No wonder leading innovators are adopting DevOps and cloud together! In his session at DevOps Summit, Andi Mann, Vice President of Strategic Solutions at CA Technologies, will explore the synergies in these two approaches, with practical tips, techniques, research data, war stories, case studies, and recommendations.
Enterprises require the performance, agility and on-demand access of the public cloud, and the management, security and compatibility of the private cloud. The solution? In his session at 15th Cloud Expo, Simone Brunozzi, VP and Chief Technologist(global role) for VMware, will explore how to unlock the power of the hybrid cloud and the steps to get there. He'll discuss the challenges that conventional approaches to both public and private cloud computing, and outline the tough decisions that must be made to accelerate the journey to the hybrid cloud. As part of the transition, an Infrastructure-as-a-Service model will enable enterprise IT to build services beyond their data center while owning what gets moved, when to move it, and for how long. IT can then move forward on what matters most to the organization that it supports – availability, agility and efficiency.
Every healthy ecosystem is diverse. This is especially true in cloud ecosystems, where portability and interoperability are more important than old enterprise models of proprietary ownership. In his session at 15th Cloud Expo, Mark Baker, Server Product Manager at Canonical/Ubuntu, will discuss how single vendors used to take the lead in creating and delivering technology, but in a cloud economy, where users want tools of their preference, when and where they need them, it makes no sense.