Why should Product Managers learn and use Data Exploration Analysis (DAE)

on

Photo by Luke Chesser on Unsplash

Why should Product Managers learn and use Data Exploration Analysis (DAE)

Before you start reading this article, I want to clarify that the opinions expressed in this post are my own and do not represent the views or opinions of the company I currently work for.  

Having clarified everything around opinions, let’s begin! 

The first time I heard about Data Exploration Analysis (DAE) was while doing Udacity course on Data Product Management.

While I had not heard the term before, I realised I had already done DAE before, through I never called it this way.

My particular DAE was the process of bootstrapping the data available from different data sources in products to answer business questions that were relevant to the growth of the products I’ve managed in the past. 

What Udacity nanodegree enabled is making this process more organised and it also helped me to improve the way the insights are presented to different business stakeholders.

This article will review the following:

  • What exactly is DAE
  • The most important step regarding DAE
  • Working with a Data Scientist
  • How to introduce DAE insights to stakeholders
  • Questions that DAE cannot answer

What exactly is DAE

DAE is the process of getting into the data your digital products generate and exploring that data in order to get answers to business questions that cannot be obtained with the current data instrumentation your product has. 

This exploratory data analysis brings insights that could generate new pathways to business, which need to be proved through product experimentation. 

Once the experiments are run, and if successful, the next step would be moving from data exploration stage to data and metric monitoring, which would need an integration of data instrumentation and visualization through product analytics tools. 

The most important step regarding DAE

The very first step of DAE is to clearly define the business objectives we are trying to achieve and asking the right questions in order to explore the answers in the data we already have. 

Sometimes the questions we ask can answer some questions based on the data we are currently collecting, and sometimes the data collected is not enough, which brings us to an opportunity to explore new possibilities to extract and process data from products to get the answers we are trying to answer. 

Let’s say you are working as PM in a digital product that only operated in the UK, and you would like to expand the business you are running from your current location to other countries. 

Some questions you would like to get answered from your data could be the following:

  • Which other countries come to consume content from our side?
  • From all the paying customers, which ones are located in other countries that are not the one you operate from?
  • Are people in other locations having specific challenges based on their location that you might not be aware of?
  • Is there a difference in recurring revenue based on the location of users in the UK vs. users of other countries? Why could this be happening?

The first step to DAE is asking the right questions

Data exploration is extremely time consuming, so you really need to have clarity around the question you want to answer, and the importance of such exploration.  

Said in other words, and as I mentioned before, the first question you need to ask is around your business objectives, and the following questions are around how your data exploration could impact your business bottom line. 

Working with a Data Scientist 

This question brings another important question to the equation.  Can the company you work for afford to pay a data analyst to help you to perform DAE?

Data experts are expensive resources and sometimes it’s not possible to cover the cost of having this role in the product team.  

  • If you have such a role, go ahead and ask for help to extract and process the data to answer your business questions.  
  • If you don’t have this luxury right now, I highly recommend exploring and extracting data as if walking through a minefield.    

What are the risks of not having a data expert to do DAE?

The reality is that human beings are biased.   

If you don’t believe humans are biased, please grab a copy of one of the following books:

  • “Thinking, Fast and Slow”, by Daniel Kahnemann
  • “Predictably Irrational” by Dan Ariely

So, having clarified this point, let me point out the most dangerous bias a PM could fall into if not supported by a data experte: confirmation bias and framing bias. 

  • Confirmation bias is the tendency to search for, interpret, favor, and recall information in a way that confirms or supports one’s prior beliefs or values.
  • Framing bias: Sometimes we frame data in a way proves the point we are trying to get through even when some of the data is hidden or removed, be it consciously or unconsciously

The two biases explained are some of the many that could happen when analysing data. We will see some more examples in this post. 

You can check a list of most known cognitive biases in this wikipedia article.

How can a data expert support a PM on DAE?

In order to escape from the confirmation bias trap, data experts need to know where and how to extract data to answer the questions the PM wants to answer.  

Data experts also need to help product managers use the data to generate graphs that can easily communicate the insights obtained from DAE. 

On one hand data experts need to know how data is being modelled, the origins of data where information can be extracted, and also analyse data distribution to remove data outliers that could distort DAE insights. 

This exploration could reveal that new data needs to be extracted from data sources, and the next step is to clear the data from outlines and see the quality of the data. 

Once data has been collected and extracted, the data expert can help the product manager generate graphs that clearly show the insights that have been extracted from the data. 

There is no data scientist in your team

I believe that PMs should be able to perform DAE even if they don’t have the right tools or knowledge.  It will surely take more time and there could be inconsistencies in the data, but that should not discourage PMs to do a DYI DAE. 

The point here is that you ask the correct questions and get the data to answer those questions.  

Maybe the data is not as clean as it should be and maybe there are inconsistencies in the data points. 

Maybe the way in which the data is presented is not as clear as you might have thought it should be, but the goal should always be to present the data and insights in order for the business to understand the insights and collectively generate hypotheses and experiments that act upon the information obtained. 

This situation also puts pressure on the product manager to push forward her data analysis skills so that the DYI DAE does not bring the biases we spoke about in the previous section of this post. 

Last but not least, the point of DAE is to get insights that bring experiments that try to improve product performance based on the correctness of the insight. 

How to introduce DAE insights to stakeholders

There is a lot of magic when you find insights you would have never guessed without performing DAE! 

Instinct draws you to run to stakeholders and start sharing what you just found out, and that is again a bias we all have as human beings. 

My strong recommendation (talking from a place of having it wrong many times) is to hold your horses before sharing your insights. 

The reason is that your DAE has been a journey around asking questions and answering the answers through data might have been a long and exciting journey. 

Your stakeholders have not gone through this journey with you, so you really need to step back, review the business questions and goals, and correctly frame your DAE insights based on this framing exercise. 

The first step is to remind the business of the high level goals and vision for the business.  Afterwards the specific business questions that were being asked to the data need to be explicitly presented to stakeholders. 

After this first high level introduction, there should be slides where each question is asked, and a graph with the answer along with a high level explanation of the insight encountered should be presented.  

This enables stakeholders to actually live the DAE journey you had to walk through to get the insights you are trying to get through to them. 

After presenting the insights, a responsible product manager should also present the questions that could not be answered and a plan to get the missing pieces of the puzzle up and running on next product iterations. 

DAE cannot answer the following questions

DAE can go deep into showing what is going on in your product based on the usage customers how users use your product. 

But what cannot be found from DAE is WHY users are doing what they are doing, and the alternatives they would like to have available to solve the problems they are facing on a faster or easier way. 

To get the whole story together we need to understand what users really want and that cannot be found with DAE. 

The best way to solve this problem is to talk to the customer segment that is facing the problems you found during DAE that cannot be inferred from the data analysis done. 

Conclusions

DAE is a very important tool in the product manager toolkit.  

Correctly used, DAE can bring insights to the business, which can significantly improve business performance, improve customer satisfaction and enable the organization to reach their business goals faster. 

Although a powerful tool, DAE only shows product managers what is happening in the products they managed, but it cannot clearly explain why this is happening or what alternatives would the customers find more effective to solve the problems they are facing.

Leave a Reply

Your email address will not be published. Required fields are marked *