Machine Learning Class Notes

Subsection 5.1 Introduction

Suppose we want to apply regression or classification techniques to data sets as in Figure 5.1.

Subsection 5.2 Implementation

In the previous section we saw that the idea for implementing polynomial regression is to

1. Add polynomial features to our data X
2. Perform Linear Regression (or Logistic Regression) on the enhanced data set X.

We already know how to implement Linear and Logistic Regression, so we just need to know how to add polynomial features to our data. We will implement this with the PolynomialFeatures() tool from sklearn.

 
from sklearn.preprocessing import PolynomialFeatures
poly_features = PolynomialFeatures(degree=3, include_bias=False)
X_poly = poly_features.fit_transform(X)

In this case, \(X\) is our original data set, degree=3 specifies the degree of the polynomial model, and include_bias=True will add the column of ones, include_bias=False does not. (Usually we do not need to add the column of ones because the ML algorithms in sklearn will do this for us.)

Let's return to one of the data sets from the introduction.

We now have multiple preprocessing steps that we need to apply to transform our data, first on training data and then on testing data. It can get complicated with so many steps, so we are going to use a tool from sklearn called Pipeline. Pipeline allows us to sequentially compose a list of transformations and conclude with an estimator. For us the transformations will be the data processing steps and the estimator will be the machine learning algorithm. Pipeline will call .fit_tranform on the list of transformations in the order they are listed, and will call .fit on the last estimator. The main parameter for Pipeline(steps, memory=None, verbose=False) is steps which is a list of (name,transform) tuples. To apply the steps

1. Add polynomial features to data set
2. Apply linear regression

we would create a pipeline with

from sklearn.pipeline import Pipeline
poly_model  = Pipeline([
    ('poly_feat', PolynomialFeatures(degree=3, include_bias=False)), 
    ('lin_reg', LinearRegression()),
    ])

We interact with poly_model in the same way we would interact with LinearRegression(), or whatever the last item in our steps is. That is, we can create the model with poly_model.fit(X,y), apply the model with poly_model.predict(X) and score the model with poly_model.score(X,y).

We can still access of the individual steps if we want, but it takes a bit more work. We method uses the named_steps attribute of the Pipeline. For example, poly_model.named_steps.lin_reg.intercept_ will give the intercept from the Linear Regression model. We can also access this by with poly_model['lin_reg'].intercept_.

Let's repeat our previous code using a Pipeline! Not we still haven't split into training and testing data yet, or applied feature scaling.

In general, the process we want for a real data set would be

1. Take a look at the data and make sure you understand it.
2. SPLIT the data (both features and labels) into Training and Testing Sets. (Eventually swap steps 1 and 2.)
3. ADD relevant features to data set. (For example, add polynomial features).
4. SCALE the FEATURES of the TRAINING data (fit_transform).
5. TRAIN the model (fit) using the TRAINING data. (For example, apply linear regression.)
6. TEST the model using the TESTING data.
7. Experiment - How did your model do? Can you do better?

The last step is really important. Experimenting with the model is always part of your job as a data scientist!

from sklearn.pipeline import Pipeline
poly_model  = Pipeline([
    ('poly_feat', PolynomialFeatures(degree=3, include_bias=False)), 
    ('std_scaler', StandardScaler()),
    ('lin_reg', LinearRegression()),
    ])

Subsection 5.3 Underfitting and Overfitting

Suppose we have the following data set

A BIG IDEA - You want to get a model that fits your data well AND generalizes well to new data sets.

Underfitting A model is underfitting if it is not matching the shape of the data, you assumed the wrong shape for \(F(x)\text{,}\) aka wrong model. High error when on the training data so the model just doesn't work, it won't be usefull on new data so high error on the testing data. You can see this by poor scores on both the training and testing data.

Overfitting A model is overfitting if it is matching the shape of the training data really well (too well!) It may go through a lot of points on the training data and will have low error on the training data. BUT, you have effectively memorized your data, so this model will not generalize well to new data so high error on the testing data. Scores are good on training data, but poor on testing data.

One way to visualize underfitting versus overfitting is to Plot Learning Curves. We will examine the score on the model as we increase the percentage of training data. Note this is a different use of the term Learning Curves than the Learning Curves from Gradient Descent. There "training" means how many steps of gradient descent we took. We were testing the optimization not the model. Now, we don't want to test how well the optimization is working. Rather, we want to know how our model reacts as it is given more training data.

What should a learning curve look like for a good model? In all of these, we see that as we train with more data the model gets better at predicting the training data (red), but eventually reaches a plateau.

1. We want the scores to be good on both training and testing. Remember good depends on the score we use. In this case, the R^2 score, 1 indicates a perfect fit, so we want the scores to be close to 1. For the cost function, or mean squared error we would want the scores to be close to 0.
2. We expect the score to be better for the training data than the testing data. But we would like the score on the testing data to be close to the training data.

Jupyter notebook for implementation details of this type of learning curves and an example of using polynomial regression with logistic regression.

There are methods for searching the parameter space to find the best polynomial degree. See the next chapter. For now, experiment and get a feel for what these parameters do so that when you use the search features you understand what they are doing. There are also different measures of how well the model is doing. Start with a model, choose a small set of features, question every assumption that you made in your model test each assumption to see if you can approve take good notes for parameters