How the Future of Data Looks Like

Career AdviceWorking Life

By preparing for trends now, you will be able to make more informed decisions about future products.

In early September, researchers, data executives, founders, and practitioners gathered at the Future Data conference to discuss what’s next in the realm of data, from arising architecture designs to innovative research concerning visualization and large-scale data systems.

The conference is a turning point for data, analytics and decision-making. Data-driven decision-making is still lacking in organizations, despite the abundance of data we have.

A Tale of Two Cities is often seen as the story of two different times. The best of times and the worst of times. However, it is also the story of two different groups of people. The wise and the foolish.

The challenge of getting fast, useful answers from data, for instance, was a consistent complaint heard across the conference - despite the fact that data availability has massively accelerated. "It's important to have a data-driven culture in order to make decisions quickly," said Ben Horowitz on the second day.

However, do not mistake the frustration for futility. If teams take clear steps now, they can accelerate the pace of decision-making, navigate change more confidently and start putting their first-party data to work in order to better inform future product decisions.

SCALING UP YOUR CLOUD-NATIVE DATA SYSTEMS

The shift towards a cloud-native data architecture is first. Companies are getting rid of data lakes from last decade and moving towards more structured, cloud-based data warehouses. Although the initial justification for moving to a cloud-based infrastructure is typically cost-savings and increased storage scalability, companies often find that the true ROI lies in accelerated access to data that informs daily decision-making.

This is a great opportunity for new companies to establish a sustainable competitive advantage against any incumbents by making use of data pipelines to easily deliver real-time updates to the warehouse. Analytics engineering and transformation tools that can reformat data on the fly, as well as data observability and quality monitoring platforms that are emerging to improve the reliability of fragile SaaS and web data, are becoming increasingly popular.

Most teams already capture the kind of data necessary to drive these decisions, and the talks at Future Data did sketch out a blueprint for building a flexible data foundation:

• You should start piping your SaaS and customer data into a single, cloud-based data warehouse as soon as possible.
• It is necessary to build analytics-friendly views directly in the warehouse; this might involve breaking some old business intelligence habits, but having a granular view of customers, transactions and user sessions is key to better analytics.
• With tools that automate the analysis process within the warehouse, you can go beyond the dashboard and enable your team to ask deeper “why” questions about changing metrics.

Automating data analysis can help you work more efficiently and generate insights faster.

The negative aspect of all this data we're collecting is that, while we have more information than ever about our businesses, it's becoming more and more challenging to use it all effectively. As Herbert Simon said in his 1971 paper "Designing Organizations for an Information-Rich World," "a wealth of information creates a poverty of attention." This problem still exists today, and researchers and innovative founders are trying to find ways to help us work with complex data more efficiently and effectively.

As you try to diagnose the reasons why customer acquisition costs are increasing, average order values remaining flat, and retention rates dropping, there are simply too many potential causes to explore. The advantage of new augmented analytics platforms is that they can take the benefits of machine learning and statistical testing, then incorporate them into large-scale data warehouses in order to test hypotheses thoroughly and "give priority to the attention" of analysts who work with this information.

To be successful, data teams should use technology — including speed, processing power and iteration — as well as the key skills of context, navigating ambiguity and domain expertise that analysts bring to the table.

Data can help you make more informed decisions.

Analytics tools need to be accessible across the business and analysts need to be integrated into individual departments in order to improve decision-making.

In conclusion, I predict a future operating model where companies will transition away from a centralized, shared service model for analytics to one where expert analysts are more embedded directly into business units. We are returning to older, IT-centric models, but the cloud-native data platforms of today are better equipped to handle the needs of distributed teams and collaborative work.

By centralizing data and pushing analysis back out to decision-makers, companies can accelerate decision-making without losing visibility into the impacts of those decisions.

Companies that adopt a more cloud-native architecture can finally track every dimension of their business with great accuracy.

Having scalable data systems and processes in place to generate insights can help companies sustain an advantage, especially when trying to enter a new market or compete with more established competitors.