The good, the bad and the ugly: The world of data is changing

It has never been more exciting to work in the data world. Twenty years ago, data was relegated to a back-office function. In 2023, it lies at the heart of an organization’s competitive advantage. Digitalization has accelerated the need for IT leaders to pay special attention to their data, AI and analytics estate.  

Beyond needing to meet companies’ goals to create more compelling customer experiences and optimize operations, technology leaders will see data play an increasingly integral role in their career evolutions in new and interesting ways: According to Gartner, 25% of traditional large enterprise CIOs will be held accountable for digital business operational results — effectively becoming “COO by proxy” by next year.

To succeed, technology and data leaders will need to take stock of the good, bad and ugly of the fast-evolving data space.

The good: The data organization is now a value organization

Here is the great news: 83% of companies report that they have appointed an executive to drive their data strategy. This represents an approximately 700% growth in 10 years (in 2012, only 12% of companies had Chief Data Officers (CDOs). 70% of these data leaders report to the company’s president, CEO, COO or CIO, allowing them to focus on what creates business value rather than activities that look like a cost center.

Additionally, technology executives are now structuring their teams to support the building of data products. According to Harvard Business Review, this can reduce the time it takes to implement data in new use cases by as much as 90%, decrease total ownership costs by up to 30% and reduce risk and data governance burden.  

Consequently, nearly 40% of data leaders report adopting a product management orientation to their data strategy, hiring data product managers to ensure that members of a data product team don’t just create algorithms, but instead collaborate in deploying entire business-critical applications. 

The bad: Data leaders are misunderstood

While 92% of firms say they are seeing returns from data and AI investments, only 40% of companies said the CDO role is currently successful within their organization.  

Data chiefs sound pretty depressed, too: 62% report that they feel their role is poorly understood. They point to the typical issues of nascent organizations: overinflated expectations, unclear charters and difficulty to influence.

This tends to frustrate everyone involved: To MIT, Fortune 1000 companies claim that only half of their data leaders can drive innovation using data, and 25% say they have no single point of accountability for data within their organizations.  

The result: Close to 75% of organizations have failed to create a data-driven organization.  

This indicates the clear need for data leaders to structure their organizations in a way that adds visible value to their employers — and quickly.

The “ugly”

What’s worse is that the average tenure of data leaders is less than 950 days. This compares to 7 years for the typical CEO and just over 4.5 years for the average CIO.  

When data leaders don’t get the time to create the structure their organization needs to win with data, everyone loses. Best practices are lost; the credibility of data engineers, analysts and scientists is affected; and business counterparts lose confidence in their leadership’s ability to build the data-driven organization they’ve committed to.

Now what?

There is hope: According to recent research — and despite the possible looming macroeconomic crisis — more than 2 in 3 data leaders (68%) are looking to increase data management investments in 2023.

On average, as our internal report shows, CDOs and CIOs have managed budgets of $90 million, with about 50% going toward personnel, 40% toward third-party software and 10% on corporate overhead expenses.  

It will be interesting to see how they decide to manage their investments this year. A recent report indicates that 52% of data leaders will focus on improving governance over data and processes first, culture and literacy second (46%), and third, gaining a holistic view of customers (45%).

Data leaders also need to adapt their team’s structure as the industry shifts away from centralized data teams creating data pipelines and static dashboards towards a data mesh model. This is where data practitioners sit within the business domains and own their own data, developing dynamic data products and applications.

The data mesh model brings data and analytics projects closer to the line of business, driving tangible ROI for business users. About 60% of survey respondents indicated that they plan to shift to a data mesh model in the next five years. There are at least four key roles CDOs should count on to build this new model: the data product manager, the program manager, and UX leader, and the data engineer.

While some of these roles have existed for a while, the data product manager is a new and emerging career opportunity for aspiring data professionals.

Three critical changes to make now

From a technology standpoint, there will be three key changes that data leaders will need to make:

1. Shifting from data warehouses to data lakehouses to cost-effectively support the rising volume, variety, and velocity of data and reduce time-consuming and expensive data movement.
2. Transitioning from siloed business intelligence dashboards to data products that work at enterprise-grade (globally available, highly-reliable and optimized for high data volume) and live up to consumer-grade scenarios (fast and responsive, optimized for high concurrency and work in real-time, all the time).
3. Increasing focus on real-time and AI operationalization. Providing compelling customer experiences requires that an organization’s data and analytics infrastructure be optimized for decisions in real-time. Unfortunately, there is just too much data and too much input for data teams to provide the support they need. According to our recent CDO survey, 55% of organizations report managing 1,000 or more sources of data. Data fragmentation and complexity is the number one barrier to digital transformation. Leaders will have to find ways to build a center of competency to deploy intelligent services on top of their unified data platform.  

To summarize, data and analytics have become increasingly important to businesses and have attracted significant investments from enterprise leaders. To maximize ROI, however, enterprise data leaders should adapt to their organizational structure, the strategies they are pursuing and the types of technologies they are purchasing to drive measurable and tangible business value.

Derek Zanutto is general partner at CapitalG.