Neil Barton is CTO of WhereScape.
In 1997, we saw the first mention of the buzz phrase “big data” in a research paper for the NASA Ames Research Center. Scientists are no longer the only ones discussing the promise of big data; rather it has become nomenclature for the entire technology industry at large.
According to Cisco, by 2021 the annual global IP traffic will reach 3.3 zettabytes per year, skyrocketing past previous years’ data growth. Nearly 20 years after big data made its debut, we are still mulling over what to do with the all-encompassing buzzword.
The industry has seen a significant bump in investments. Just this year alone, $57 billion will be invested to bring the realities of big data to companies and individuals. Whether we consider technologies that allow us to maximize computation, or algorithmic accuracy, analysis or even deeper knowledge, the question remains – how can we leverage big data?
Here are five steps to maximize your big data investment and ensure a positive ROI:
Think Before Jumping on the Big Data Bandwagon
As with any exciting trend, people are jumping quickly on the big data bandwagon. Unfortunately, most organizations invest in big data without first identifying their actual business needs.
To be successful, organizations must sit down and determine a problem and goals before pursuing a big data initiative. It sounds obvious, yet as the concept of big data grows, so too do the problems. Determining the underlying issue will catapult any organization down the right road to a solution and success.
Understand the Catch
Much of the technology available to manage the ever-swelling pool of data is free or open source. And just like most free things in life, there is always a catch. Just because the software itself doesn’t require a payment does not make it cheap or easy to install and operate. While the tools themselves may be free, the skills required to implement, configure, debug, manage and develop are hard to find and can be expensive.
Open source big data components tend to lack the breadth and depth of operations and maintenance capabilities that most traditional platforms have had for decades. This puts additional burdens on IT resources to manage and monitor.
Organizations and their customers do not care that it is open-source. They care that the data is governed and secured.
While free may sound tempting, it’s important to compare the total costs and benefits of open-source versus an enterprise-grade solution. You may find the enterprise solution is the most cost effective and painless way to value in the long run.
Mix and Match for the Best Combined Solution
The proliferation of big data tools and products are most valuable when they are properly matched, and all have a significant role to play in the modern data ecosystem. However, understanding that they are not a cure-all is critical. Rather, such tools are just parts of a larger and more complex ecosystem that is now becoming the new blueprint for enterprises.
It’s important to research whether applications can work together to produce your desired result before embarking on a new tool or product, or you may find yourself in the middle of a data junkyard. If you’re working in an open-sourced environment, consult with others for their experiences and best practices. Proactively seek out conflicts and potential exposures before they occur by learning from the mistakes of others.
Unfortunately, this means spending a good amount of time researching each component of your big data solution. But it’s best to research and prevent a problem rather than wasting time and money correcting an issue and repairing the trust and support of your management and customers.
Carefully Plan the Transition
While solutions like Spark and Hadoop are appealing on the surface level and relatively straightforward to implement in a development sandbox or another test environment, transitioning to production entails a level of governance and DevOps capabilities that these tools tend to lack.
This final hurdle can be a difficult step for organizations, especially large enterprises that still have governance, control, and auditing-related requirements, regardless of the underlying technologies being used.
The shift into production requires careful planning, the right skills, and the understanding to ensure success. Devote the time to develop your strategy for this part of the project; it will be a worthwhile investment.
Avoid the Resource Crunch
Traditional approaches to analyzing and deriving value from data cannot usually be applied to many of the new types of data being ingested. As a result, organizations need to adjust their culture and processes to accommodate the available new paradigms. But this can be tough when IT departments have downsized and eliminated their integration and architectural expertise.
Asking people to do new things when they are already stretched is often a recipe for disaster. For example, the notion of hand-coding and updating and maintaining the code as the underlying APIs change, or new/better components come to market, puts incredibly high demands on internal resources.
The truth is, most organizations are simply not equipped to handle this rate-of-change in the underlying technology and consequently fail.
Organizations on tight budgets and resources might struggle with the maintenance costs and maintenance of big data projects. Look to innovative approaches, such as metadata-driven solutions that can mitigate time, cost and risk to this scenario.
As big data investments continue to rise, understanding the complexities before jumping into the deep end will be critical for the high ROI organizations are seeking. Remember to look before you leap.
Opinions expressed in the article above do not necessarily reflect the opinions of Data Center Knowledge and Informa.
Industry Perspectives is a content channel at Data Center Knowledge highlighting thought leadership in the data center arena. See our guidelines and submission process for information on participating. View previously published Industry Perspectives in our Knowledge Library.
