David Flynn is Data Chief Technology Officer and Co-founder of Primary Data.
New storage technologies like NVMe flash and cloud storage are helping enterprises keep up with explosive data growth and new ways to use old data, including business analytics and other intelligence applications. The trouble is that despite the diverse capabilities of storage systems across performance, price and protection, the rigidity of traditional storage and compute architectures mean that there has been no way to make sure the right resource is serving the right data at the right time. Each new type of storage becomes a silo that traps data and increases data center costs and complexity.
To overcome this inefficiency, storage resources need to be intelligently connected to automate the placement and movement of data and maximize performance while saving significantly on costs. This can be done now by adding a metadata engine into your architecture to abstract data from underlying storage hardware to virtualize it, and then unifying storage resources and capabilities through a single global data space. Data can then be moved automatically to the ideal storage for data requirements, without application interruption. Let’s examine how this can improve application service levels, increase storage utilization to slow storage sprawl, and reduce storage overprovisioning to reduce costs.
Improve Service Levels by Using the Right Storage for the Right Job
Traditionally, due to the cost and complexity of data migration, most data typically stays where it is first written until it is retired. As a result, IT typically purchases storage based on the highest projected estimates for application service requirements. While this approach enables IT to ensure it will meet Service Level Agreements (SLAs), it creates significant waste in the data center, and this is straining IT budgets. Even with this excessive over purchasing of storage, estimates can be wrong, and many admins have stories about nightmare data migration fire drills.
Automating the placement and movement of data with a metadata engine improves service levels, as follows:
- Data-aware, objective-based management. Traditionally, IT takes a bottom up approach to meeting application service level objectives, assigning storage resources based on expected application needs. Intelligent data management enables IT to focus on data needs, aligning data with the capabilities that a specific storage device can provide. A metadata engine can automatically provision data to the ideal storage for its performance and protection requirements, and move data dynamically and non-disruptively if the environment changes to ensure that service levels are always met. For example, if a noisy neighbor starts consuming storage resources, workloads can be rebalanced without manual intervention, and without disrupting application access to data.
- Global visibility into client file and storage performance. An enterprise metadata engine can gather telemetry on clients, enabling IT to see which applications are generating the workload and the performance each client receives. It can also gather performance information across all storage resources integrated into the global dataspace. This enables smart, real-time decisions to be made so data can be placed to meet SLOs. In addition, workloads can be charted historically, helping IT implement more effective policies, as well as proactively moving data to storage, as needed. For example, software might identify that financial data sees higher activity at the end of the quarter and proactively move the associated data to faster storage, and then back to more cost-effective storage once quarterly reporting is completed.
- Simpler, faster architecture. Once data is freed from individual storage containers, it becomes possible to move control (management) operations to dedicated management servers, accelerating management operations, while freeing the data path from the congestion of metadata activity. In addition, managing data through a metadata engine enables applications to access data in parallel across multiple storage systems.
- Easily integrate new storage technologies. To help enterprises overcome vendor lock-in and save on costs, any solution to automate the placement and movement of data should ideally be vendor and protocol agnostic. This makes it possible for companies to easily integrate NVMe in servers, cloud providers, such as Swift and S3, and the next advance in storage without the need to rip, replace, and upgrade storage.
Increase Storage Utilization to Reduce Storage Sprawl and Costs
To avoid the cost and complexity of data migrations, IT over-provisions storage for every application in the data center. This leads to and expensive waste. Automating the placement and movement of data across storage resources with a metadata engine enables enterprises to greatly increase storage utilization, to significantly slow storage sprawl and reduce costs, as follows:
- Global visibility into resource consumption. With a metadata engine that virtualizes data, admins can see available performance and capacity, in aggregate and by individual storage system. This ensures they know exactly when they need to purchase more storage, and of what type they need to purchase to meet data’s needs.
- Automatically place data on the ideal storage. A metadata engine can automatically move infrequently accessed data to more cost-effective storage, including the cloud. By reclaiming capacity on more expensive storage resources, this significantly extends the life of existing investments and greatly slow storage sprawl.
- Scale out any storage type, as needed. A metadata engine that is vendor and protocol agnostic views storage in the global namespace as resource pools with specific attributes across performance, price and protection. Workloads can then be automatically provisioned and rebalanced workloads as new resources are added. This makes adding performance and capacity of any storage type a simple deployment process, instead of a complex, planned migration or upgrade. As a result, enterprises can defer new storage purchases until they really need it, and purchase exactly the storage they need. This has the additional benefit of making it possible to easily introduce the latest storage technologies into the datacenter for business agility.
Integrating a metadata engine into enterprise architectures improves service levels with a data-aware approach to management, smarter and automated data placement decisions, and simpler, faster architectures. It also enables enterprises to use storage much more efficiently to reduce storage hardware, software, and maintenance costs. With these advanced capabilities, enterprises are able to cut over-provisioning costs by up to 50 percent, and those savings easily run into the millions when you are managing petabytes of data.
Opinions expressed in the article above do not necessarily reflect the opinions of Data Center Knowledge and Penton.
