Delivering ROI on Big Data: 3 Ways to Empower Your Data Team
In the 1960s, a public service announcement frequently aired in the evenings by U.S. television broadcasters was, “It’s 10 pm. Do you know where your children are?” It was a culturally transformative time, not unlike the technologically transformative time we’re facing in data analytics today. Today, a question that faces executives who want to achieve a significant return on investment from their Big Data and analytics initiatives is:
“It’s 2021, and it’s the era of data-driven organizations. Do you know where your optimization opportunities are?”
Never before have organizations been blessed to have so much rich data at their command. You’ve heard the mantra, “data is the new oil,” and it’s true: never before have we had such a plethora of innovative technologies helping us mine that data for insights. Indeed, this is the era of data-driven organizations, and so far we’ve only scratched the surface of what is possible.
Many organizations that have taken the lead in pioneering the use of groundbreaking data analytics platforms, such as the cloud-based data warehouse platform Snowflake, are now experiencing a downside that has a potential to become critical: runaway costs.
That’s because most analytics platforms are built to optimize speed, not cost-efficiency. In fact, many have only one “control knob,” one that increases volume and speed of queries as long as you keep pouring more money into the system.
Even enterprises that bring their burgeoning data operations in house, with Presto for example, in an effort to control costs experience significant challenges in balancing multiple priorities. Data teams find themselves under enormous pressure to do many things at once:
- Manage the requirements of different business users, including response time, concurrency, types of queries they want to run, the data formats they require, etc.
- Manage complex data applications as they scale.
- Manage resource allocation, including prioritizing and optimizing workloads (sets of queries that serve a business need) and controlling costs.
How can data teams optimize results when faced with multiple and often conflicting responsibilities? CIOs must recognize the tough task this is. If the company is to maximize ROI on Big Data and analytics, CIOs must ensure that the data teams are equipped with the support, tools and knowledge needed to optimize data operations on every dimension.
For starters, here are three ways to support your data team:
1. Provide data teams with a clear understanding of business priorities.
Your data team needs to know what matters most so they can prioritize workloads accordingly, not only to deliver the most important information first, but also to manage costs. Let’s look at two examples.
First, as mentioned above, many analytics platforms are built to scale rapidly and infinitely, as long as you are willing to pay the price for additional compute resources. But scaling just because you can is like writing a blank check. Scaling should correlate to revenue, and your data team should continually ask, are we scaling for the right reasons? Will scaling correlate to increased revenue? Your data team needs to have awareness of how their efforts track to business results.
Here’s another example. In theory, if given the time, experienced data engineers could optimize every single user query to control costs, but in reality, no engineer has that much time and not all queries are equally important. In many instances the answer isn’t to speed up a query but rather to slow it down so that a higher priority workload can use more of the shared resources and finish on time. To make these kinds of wise decisions about analytics workloads, your data team needs to understand business priorities. As CIO, it’s your job to make sure they see the big picture.
2. Give your data teams tools that provide workload observability.
The key to effectively tune analytics performance at large scale is to have the right level of observability into workloads. The ultimate query prioritization process should map back to business priorities. Your data team needs to determine which workloads and queries are required by the business teams and allocate resources to those workloads.
Some of this information comes from the query statistics provided by most query engines. More sophisticated solutions allow you to group queries by user to identify overall workload behavior. By mapping out the resource utilization of workloads from a business perspective, your data team can start to focus on where to optimize queries and where to allocate your fixed resource pool. For example, your team should be able to show you all of the resources and costs associated with providing analytics for customer reports and compare that to the resources and costs associated with other workloads that may have a higher or lower business priority.
Make sure your data team has the tools needed to optimize enterprise-wide analytics usage, keep users happy, and keep costs down.
3. Help your data team focus on what matters and automate the rest.
The purpose of data analytics is to help your organization “work smarter” by gaining insights from data. In turn, you can help your data analytics team “work smarter” by applying automation to their systems and processes. Many of the workload prioritization and optimization tasks described above can be automated, reducing the continuous whack-a-mole that can quickly burn out data teams. Automation is an optimization “no brainer.” Make sure your data team is taking full advantage of automation technology (hint: it should be built into their analytics tools and platforms) to help your organization achieve maximum ROI on its data analytics initiatives.
It’s a new era for wringing business value and competitive advantage out of data. New tools are emerging to help organizations manage the operational and economic impacts of query acceleration. On your journey from the data warehouse to the data lake, keep in mind these three tips on how to support your data ops team, so your organization can achieve the results you’re looking for.
