Making Predictions With Simple Linear Regression Models
One of the things that really got me interested in Machine Learning algorithms and Neural Networks was their ability to make pretty good predictions. The best way to appreciate how these technologies work is to find some real world data and come up with a project that will be interesting, and hopefully, reveal some pretty useful results.
I also find it rather interesting how all of the mathematics I learned years ago in school makes more sense when you apply it to real world situations. If nothing else, Machine Learning makes you appreciate the beauty of math and statistics. Math becomes more than just a set of rules and formulas with obscure meaning and instead presents itself as a valuable tool in the realm of Artificial Intelligence.
For Example, let's take a look at linear regression...
Linear Regression Formula
y = β0 + β1 * x
This formula was rammed down our throats in school. We were told that it is how you can estimate a line and to make sure we memorized it for the test. Like most, I jammed this into memory and held on to it not really understanding the true significance of this amazing formula and how it can be used to make interesting predictions.
Let's examine this formula starting with the obvious. We know 'x' and 'y' are variables. We know that a variable can represent anything we want it to, but what is interesting about this formula is not the variables themselves, but their relationship to one another. Statisticians call this relationship dependent vs independent. This makes sense because in order to know what 'y' is one must solve for it. In order to solve for 'y', we will need a ton of extra information in order to accomplish this goal. This is the reason 'y' can be thought of as the dependent variable. On the other hand, you have 'x' which is going to be whatever it is regardless of what happens to 'y'. In fact, 'y' cannot have its meaning unless 'x' itself has one, that is why 'x' is known as the independent variable. 'x' exists regardless of whether you have 'y' or not.
What does this formula have to do with a line, and more importantly how is this useful in the real world and automation?
After taking another look at the formula we can see the two coefficients B0 & B1. One can think of the 'B0' as the origin (intercept) of the line or the constant. All lines must have a starting point and 'B0' represents this. 'B1' simply tracks the slope of the line. The cool thing about this formula is it can be used to find the best fitting line for any dataset that has a linear relationship.
Linear relationships are all around us. Many times there are variables we assumed are correlated only to find out those variables are not. One can't always comfortably assume that all variables have a correlation. The assumption of correlation must always be put to the test.
For example, suppose someone said that in Football the more points you score per game the more games you will win. This sounds like common sense. In fact, it's so common one should not feel like there is a need to prove a statement like that with mathematics. But what if we had to prove it. What if we could take that assumption and have real data to back it up?! Even better, what if we could even use that data to make further predictions? It turns out we can.
Let's look at some data from the real world. Below is a chart of some statistics from 2016's NFL regular season.
We see a list of teams and two variables. The dependent variable, in this case, is the number of wins in a given season. The independent variable is the average points per game. What's of interest to us is to find out whether or not the more points a team averages a game leads to more wins in the season. That's the fundamental assumption. In addition, we would like to know if there is a correlation between the two, and is it possible to predict how many games a team will win based on a number of points a team averages a game. If we were to plot this data using average points per game as 'x' on the graph, and the number of wins in a season as 'y' we could visually see how all these data points stack up. Here is a graphical representation of this dataset.
Right off the bat, we notice that it is extremely difficult to determine whether or not a high average of points per game leads to more wins. In fact, this graph can be used to argue that the majority of the team averages exists in the center of the graph. Notice how we have a team that averaged more than 32 points per game and still could only manage to get about 11 wins on the season. Another point to see is a team that has achieved 14 wins on the season but averaged around 28 points a game. Where do we go from here?
Before we lose all hope and consider our initial assumption trash we can get our answer to the question of the relationship between our two variables by drawing a line through our data. This is where our linear regression formula comes in.
Our goal is to teach our machine to find the relationship in this data set using the Linear Regression algorithm within the R ecosystem. Once training is complete we will test our system to see if it has learned this relationship. Finally, we will give our algorithm datasets from NFL seasons it has never seen before, and determine how it handles it.
After importing our data and selecting only rows 2 & 3 we get the following output...
Next, we need to split our data into a set used for training, and also a set used for testing. They say a good rule of thumb is 80% of the data should go to training and 20% should go to testing the system. We'll do that next.
Now we will use the training set we have to train our brand new data model.
Now we are ready to make predictions! Luckily, R has a built-in prediction function we can take advantage of ...
Next, we are ready to see how our model performed on the training set. For this part, we will need to take our model and make predictions. Once the predictions are made we will then need to graph them accordingly. This will give us a good idea of how our system did on our training set.
Now let's see how our system performed on the training set.
Not too bad. We can see our system has found pretty good coefficients on our training data and was able to generate a clear trend line for us to see. Let's get a quick summary of our newly trained data model. In R, this is pretty easy if we use the summary function.
This is only half the battle though. We will need to see how it does on different datasets to see how it performs there. Initially, we reserved about 20% of the data for testing purposes. Let's see how we did on that last 20% of the test data.
Looks like our model did pretty well with this dataset as well. Notice how it found the best fitting line for all the data points. That's pretty Impressive! Lets really try and trip it up by showing our model data from 2015's NFL season. Let's see how it does.
Above you can see how it performed on the 2015 Regular season data set. We are still getting our consistent line even though the data is completely different! Let's see what happens when we give our system 2014 data.
Different data, same trend line.
Our model has seen data from 3 separate NFL seasons, and now that these observations have been made we can start seeing some consistencies. The first thing we know is that there is a consistent relationship between a team's average points per game and their number of wins in a season. This is probably a no-brainer for any offensive coordinator, but this does serve as a mathematical explanation. Notice the line has an upwards slope in every single data set.
We can also make some predictions based on this data as well. For example, teams that average up to, or more than 30pts a game can expect to win more than 12 games a season. This prediction is based on the data we have over the past 3 seasons utilizing the two variables average points per game and number of wins in the season. On the opposite side of the spectrum, teams should not expect to win more than 4 games if they can't average more than 15 points per game. If I fed this model all the data from every NFL season that ever existed it would likely find the exact stat lines it would take for a team to win the exact number of games it is shooting for! This is the beauty of Machine Learning and Linear Regression.
The Machine Learning Age is here. You are likely to hear some breakthroughs happening at breakneck speed over the next few months and years due to so many of these powerful algorithms being utilized in all kinds of fields. There is so much left to be discovered in all the data that is around us. All it takes is knowing these algorithms well and applying them to the right situations. It's very important that we start putting these techniques to good use as it will likely be the difference in one company's operating efficiency and survival as opposed to another. An organization not making good use of their data doesn't stand a chance against an organization that is gathering insights and leveraging algorithms to accomplish their goals.
*NFL data source: Pro Football Statistics