Correlation in Python

Correlation values range between -1 and 1.

There are two key components of a correlation value:

magnitude – The larger the magnitude (closer to 1 or -1), the stronger the correlation
sign – If negative, there is an inverse correlation. If positive, there is a regular correlation.

Positive Correlation

Let’s take a look at a positive correlation. Numpy implements a corrcoef() function that returns a matrix of correlations of x with x, x with y, y with x and y with y. We’re interested in the values of correlation of x with y (so position (1, 0) or (0, 1)).

In [1]:

import numpy as np

np.random.seed(1)

# 1000 random integers between 0 and 50
x = np.random.randint(0, 50, 1000)

# Positive Correlation with some noise
y = x + np.random.normal(0, 10, 1000)

np.corrcoef(x, y)

Out[1]:

array([[ 1.        ,  0.81543901],
       [ 0.81543901,  1.        ]])

This correlation is 0.815, a strong positive correlation, let’s take a look at a scatter chart.

In [2]:

import matplotlib
import matplotlib.pyplot as plt
%matplotlib inline
matplotlib.style.use('ggplot')

plt.scatter(x, y)
plt.show()

Negative Correlation

What happens to our correlation figure if we invert the correlation such that an increase in x results in a decrease in y?

In [3]:

# 1000 random integers between 0 and 50
x = np.random.randint(0, 50, 1000)

# Negative Correlation with some noise
y = 100 - x + np.random.normal(0, 5, 1000)

np.corrcoef(x, y)

Out[3]:

array([[ 1.        , -0.94957116],
       [-0.94957116,  1.        ]])

Our correlation is now negative and close to 1. Let’s take a look at what this looks like graphically:

In [4]:

plt.scatter(x, y)
plt.show()

No/Weak Correlatio

What if there is no correlation between x and y?

In [5]:

x = np.random.randint(0, 50, 1000)
y = np.random.randint(0, 50, 1000)

np.corrcoef(x, y)

Out[5]:

array([[ 1.        , -0.00554681],
       [-0.00554681,  1.        ]])

Here we see a very small value for the correlation between x and y, indicating no correlation.

Again, let’s plot this and take a look, we see there is no correlation between x and y:

In [6]:

plt.scatter(x, y)
plt.show()

Correlation Matrix

If we’re using pandas we can create a correlation matrix to view the correlations between different variables in a dataframe:

In [7]:

import pandas as pd

df = pd.DataFrame({'a': np.random.randint(0, 50, 1000)})
df['b'] = df['a'] + np.random.normal(0, 10, 1000) # positively correlated with 'a'
df['c'] = 100 - df['a'] + np.random.normal(0, 5, 1000) # negatively correlated with 'a'
df['d'] = np.random.randint(0, 50, 1000) # not correlated with 'a'

df.corr()

Out[7]:

	a	b	c	d
a	1.000000	0.825361	-0.948845	0.009802
b	0.825361	1.000000	-0.789391	0.011852
c	-0.948845	-0.789391	1.000000	-0.003228
d	0.009802	0.011852	-0.003228	1.000000

We can also view these correlations graphically as a scatter matrix:

In [8]:

pd.scatter_matrix(df, figsize=(6, 6))
plt.show()

Or we can directly plot a correlation matrix plot:

In [9]:

plt.matshow(df.corr())
plt.xticks(range(len(df.columns)), df.columns)
plt.yticks(range(len(df.columns)), df.columns)
plt.colorbar()
plt.show()