Exploratory Data Analysis

In most EDA processes the very first step is getting familiar with the data set. What are the column names, how is it indexed, what is the unit of measurement (UoM) for each variable? All of these are valid questions that the analyst will need to understand.

Once a very basic understanding is formed it’s time to jump into the data. This often consists of:

These examples use The Data Mine’s olympics data set, but the concepts should apply to almost any data.

In this hypothetical case let’s say that our goal is to predict if the athlete is going to win a gold medal. That would make it a classification problem. We want to predict 1 if gold and 0 for everything else.

We can start by printing out the data:

For any of the numeric columns we can easily find missing values by using the pandas describe() function. This is also one of the first times we can check for outliers.

First thing we can see that Age, Height, and Weight are all missing values. Interestingly we can also see a potential outlier in the Age column with a max value of 97. These are all things that we’ll check into, but for now we can also check the text variables.

We can double check this if we want by using the unique() function.

The lack of NaN or Null in the output above confirms that our code to check for missing values in the text variables worked as expected.

Now that we have an idea of the variables that are missing we can go back to see if we can better understand why they may be missing. In this example we’ll look at Age, but you should check any variable that may be of interest.

My first thought is that maybe it’s due to when the records were recorded. Maybe before a certain year they didn’t record Age. We should be able to check that pretty quickly.

It looks like records don’t have an age from the 1800’s all the way through 2008 so it’s likely not due to how old they are.

For the purposes of this demonstration I’m not going to dig further into the Age column, but this illustrates one of the core components of EDA. Ask questions and make hypotheses of your data and then use the data to prove or disprove them. This helps you learn and think about how you may use the data.

After you’ve built a basic understanding of each variable, you’ll often start to dig in to how they are related to each other or how they change over time. The example below shows an example of how you may be interested to see differences in gold medals by different variables.

For this example, I cut down the data to compare the medal counts by sport for the United States and Canada. This graph helps to illustrate lots of different information. The Sport variable seems to really be differentiated by Team. These may be valuable variables to include in the model.

In the same way that we look at groups of data it’s often helpful to look at data over time. This can show relationships that we may not have been aware of. These types of variables can also be included in a model.

For example, going back to the Olympic data that we are working with we could hypothesize that the number of medals in the last Olympics may be a good predictor of medals in this Olympics. If we wanted to check this quickly in a graph, we could plot the medals over time for a few different nations. If we see a ton of variation (very spiky plot) then maybe it isn’t a great predictor. However, if we see that countries tend to keep similar trends then maybe it’s worth including in the model.

In this case we see that there is some variance over time with respect to the country. The United States has a large spike at the very beginning and then a steady upward trend. This shows that the last Olympics' medals alone may not be the best predictor. However, we could potentially create a variable that shows the historical trend for the country. That could be very helpful. These are all items that you can test with your modeling process!

Correlation and covariance are two very common terms in the world of analytics. At their core they both show how two variables relate to one another. A few great resources for understanding correlation and covariance are included below:

Rather than try to explain a topic that’s already well-covered online, our goal is to guide you through code to test the concept yourself.

We can start by generating a bit of random data. To do this we can actually use a covariance matrix.

The covariance matrix will show the relationship between the different variables. On the diagonal you’ll find the variance for each measurement. The off-diagonal elements show the covariance values. In this case the larger the value (positive or negative) the more the variables appear to move in the same pattern.

In this example the table shows us the variance of Variable 1 is 150, Variable 2 is 100, and Variable 3 is 110. If you’re not familiar with variance it’s always good to go back and review.

We can use the same matrix in the code below to generate a number of data points based on the variance and covariance in our matrix.

We can plot these new variables with histograms to see how they are distributed. In this case we’ll see that they match with the mean of 0 that we passed in with mean_to_use. You’ll also notice that if you go back and change the variance (diagonal elements in the matrix) the width of the distribution will change.

If we generate scatter plots we’d expect the visual trends to match our covariance matrix input. In this case Variable 1 and Variable 2 should have a strong positive trend (360). Variable 1 and Variable 3 should have a negative trend (-200). Variable 2 and Variable 3 should have little to no trend (0).

We see the positive, negative, and no trend graphs that we’d expect! Although as the articles mention this can be a bit hard to interpret. For example, how does 360 compare to -200. Correlation helps to fix this by standardizing the data. This puts all of the values on the same scale between -1 and 1. Pandas has an easy built-in function to check the correlation of our values.

We can see that this follows the same positive, negative, and no-trend patterns as a covariance matrix, but it’s a little easier to understand on this scale.

Now it’s your turn! Play with the code above to see how the different values change as you input different variables.

Imputation is a very common topic when working with data. Often, the data is missing a large number of samples and we have to decide what to do with them.

If there are a small number of samples missing you may just remove those rows. Although, it should be noted that it is important to understand why the samples are missing in the first place.

If there is a large number of samples missing or you are trying to add data to the data set, you may end up imputing values.

Just like with correlation and covariance there is a ton of great content already written around these topics online. Usually you’ll find a few simple techniques, such as mean imputation, and a few more complicated techniques, such as SMOTE.

We encourage you to do your own digging to learn more about imputation. In this section we’ll focus on helping to highlight some of the risks of imputation as well as how we can use interactive learning to understand the different imputation techniques.

To illustrate one of the risks of imputation we can go back to our Olympics dataset and check the Weight variable.

We know from the EDA that we did above that the variable has missing values. Many times with imputation it’s tempting to say we can just add the average. After all, it’s the average for everyone at the Olympics so it shouldn’t be too bad to use that value right?

In this case we can start to see the risk of imputation. There are so many missing values that imputing the mean causes a spike in the middle of the data. If we were to feed this variable to a model it would think that the majority of Olympic athletes are around 70 inches tall. This could be true, but this could also be a majorly misleading variable for the model.

Hopefully, this example shows the importance of understanding the impact of your modeling choices and the effect they will have on the outcome. If you’re ever unsure we recommend talking to your teammates. Getting a different opinion on a process can be very helpful!

There are lots of great resources to learn more about imputation. We included a few examples below for your reference:

Feature engineering is the idea of creating new variables, or features, that can be added to a model to help with predictive power. New features are often very impactful but may be difficult to craft. This is an area where a subject matter expert’s (SME) knowledge can come in very handy.

When building new features, it’s important to have a way to compare them. Usually this involves building a baseline model and then seeing if a feature improves its accuracy. There is also a ton of articles written around the idea of feature selection. However, for this example we are going to walk through the ideation phase of feature engineering.

Going back to our Olympics data we can see that we have the columns below to work with:

If our modeling goal is predicting the number of medals for specific countries, we would ideate on other factors that could help the model learn these trends. These could be built off data that we have in the dataset or imported from other datasets. A few potential feature examples are included below:

These are all random ideas. Your job as an analyst would be to come up with these hypotheses and then prove or disprove them with data. This also shows why a SME can be so helpful. Instead of me just guessing at these important factors imagine if you could talk to an Olympic athlete or coach. Their input could make a world of difference!