Machine Learning Class Notes

Subsection 6.1 Regularization

There are three different types of error that can occur in a machine learning model: Bias, Variance, and Irreducible Error.

Irreducible Error: This is error that is inherent in the data itself. We may be able to clean up the data in the preprocessing stage, but usually there is always still some error that is present in the data.

Bias: The bias of an estimator is its average error for different training sets. If there is high bias then the model isn't doing a very good job of making predictions. Note: This is not related to the bias term in linear regression.

Variance: The variance of an estimator indicates how sensitive it is to varying training sets. If there is high variance then the model will do very well on the training set, but not very well on the testing set.

Generally we are always trying to find a balance in our machine learning techniques between high bias and high variance. For example, in polynomial regression if the degree is too low we will have high bias and if the degree is too high we will have high variance.

Regularization is one tool to help with overfitting (high variance.) We are going to add a penalty to the cost function to constrain the size of the \(\Theta\) values. There are three different types of regularization based on the type of penalty we add to the cost function.

The first type is called Ridge Regression or \(l2\) penalty. For this technique we change the cost function to

where \(MSE(\Theta)\) stands for the mean squared error cost function for Linear Regression, but would be replaced by the appropriate cost function for the given algorithm. For example, log-loss, or cross-entropy loss for logistic regression. For Ridge Regression we penalize the \(l2\) norm in the cost function. In otherwords we try to minimize the cost function with respect to \(\theta\) values, but we add a term that penalizes the cost when the \(\theta\) values are large. Here the penalty is the \(l2\) norm or Euclidean distance from orgin. (You may also see this as \(l^2\) norm, but sklearn favors \(l2\text{.}\))

Graphs of different values of the regularization coefficient \(\alpha\) is given in Figure 6.7.

There are different ways to access this in sklearn models using either ridge or l2 penalty terminology. But there are other ways to penalize the \(\theta\) values.

The second type of regularization is called Lasso Regression or \(l1\) penalty. For this technique we change the cost function to

In Lasso Regression we penalize the \(l1\) norm in the cost function. Note that again the parameter \(\alpha\) changes the amount of reglarization to do.

Lasso Regression tends to eliminate weights of the least important features. It is especially useful if have more features than data points. However, it can cause trouble with optimization because the gradient descent may bounce around. That is, there may be convergence issues.

Graphs of different values of the regularization coefficient \(\alpha\) is given in Figure 6.10.

Lasso Regression will be referenced in sklearn models using either lasso and l1 penalty terminology.

The third type of regularization is called ElasticNet Regression or \(l1\)-ratio. For this technique we change the cost function to

Elastic Net Regression is a mixture of Ridge and Lasso. We now have two parameters, \(\alpha\) which is the amount of regularization and \(r\) which is the mix ratio (also called the l1-ratio). The mix ratio impacts how much of ridge vs lasso you want to do. In general, \(0\lt r\lt 1\text{.}\) If \(r=1\text{,}\) then it is just Lasso. If \(r=0\text{,}\) then it is just ridge

In general, Ridge Regression is a good default (\(r=0\)). Lasso Regression is good if you suspect that you only have a few good features (\(r=1\)). Elastic Net is prefered over Lasso when several features are strongly correlated or the number of features is greater than the number of training instances.

Early stopping is another way we can try to reduce overfitting. We train the data until the testing score starts getting worse. The steps for this method are

1. Train data in epochs.
2. Track training and testing errors.
3. Stop once you start doing worse on testing.

For example, in Figure 6.12 we would stop training after approximately 240 epochs.

This is usually one of the parameters you can choose in many of the machine learning models. That is, you can often apply this by choosing early_stopping=True.

See Jupyter notebook for details on implementation for these techniques.

Subsection 6.2 Parameter Searching

We now have many parameters for our models that we want to explore and we would like a way to find the best parameter or combination of parameters. Generally this requires separating testing data and measuring the performance of our model on the testing data and comparing its performance across parameters. We are going to talk about ways to do that more efficiently.

The first issue we have seen with this technique is that the way the data is split between training and testing can really impact performance and accuracy of a model. However, sometimes we have small data sets, so it can be hard to do this well. One technique we can apply to help with this is called \(k\)-fold cross validation. In this technique we break the data into \(k\) sets, use each set as test data, and the remaining data as training data. We create the model \(k\) different times, (on \(k\) different training set), examine all the scores and the average score.

Of course, sklearn has a built-in for this its cross_val_score.

from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import cross_val_score
knn = KNeighborsClassifier(n_neighbors = 3)
cv_scores = cross_val_score(knn, X, y, cv=3)   

The full list of parameters are here:

cross_val_score(estimator, X, y=None, groups=None, scoring=None, 
cv=None, n_jobs=None, verbose=0, fit_params=None, 
pre_dispatch='2*n_jobs', error_score=nan) 

The main ones we will use are

1. The estimator is the machine learning technique (kNN, PCA, LinearRegression (with or without polynomial features), LogisticRegression (with or without polynomial features), etc) that we want to apply to the data.
2. \(X\) is the set of features for our data (should be scaled!).
3. \(y\) is the labeled data if a supervised tecnique is being used. No \(y\) is entered if using an unsupervised technique.
4. scoring=None means the estimators default scoring method is used.
5. cv is the number of folds. (The default, cv=None using 5 folds.) If the estimator is a classification technique then the data is split into folds using a stratified method. A random split is used for regression techniques.

Remember that when feature scaling is applied to data, it should only be done on the training data and not the full data set. Otherwise it leaks data about the testing set. For cross validation it must be done on each training set individually. (The PIPELINE feature will really help here!)

We will return to the fruit data set from exam 1 for an example of cross validation.

We are going to revisit kNN as our classification technique and use all four features. Since the scales of the features are quite different, it is very important to use feature scaling now. We will create a Pipeline to combine ScandardScaler and kNeighborsClassifier.

Nearly all of the models have multiple parameters to tune. How do we find the best combination of parameters? For example, in the previous fruit data set example, what is the best value of \(k\) to use in \(k\) nearest neighbors? For a single parameter, we can use a validation curve. A validation_curve will produce an \(m \times n \) array of scores for training and testing where \(m= \) the number of parameters specified in the range and \(n= \) the number of folds for cross validation. We can then graph an average score across folds to see how the score varies as we change the single parameter.

from sklearn.model_selection import validation_curve
param_range = range(1,8,2)
train_scores, test_scores = validation_curve(KNeighborsClassifier(), X, y,
            param_name='n_neighbors', param_range=param_range, cv=cvnum)

The full list of parameters is

validation_curve(estimator, X, y, param_name, param_range, 
	groups=None, cv=None, scoring=None, n_jobs=None, 
	pre_dispatch='all',verbose=0, error_score=nan).
 

Most of these are the same as cross_val_score():

1. The estimator is the machine learning technique (kNN, PCA, LinearRegression (with or without polynomial features), LogisticRegression (with or without polynomial features), etc) that we want to apply to the data.
2. \(X\) is the set of features for our data
3. \(y\) is the labeled data if a supervised tecnique is being used. No \(y\) is entered if using an unsupervised technique.
4. scoring=None means the estimators default scoring method is used.
5. cv is the number of folds. (The default, cv=None using 5 folds.) If the estimator is a classification technique then the data is split into folds using a stratified method. A random split is used for regression techniques.

The new ones are param_name and param_range where we specify the parameter we want to vary.

 
train_scores, test_scores = validation_curve(KNeighborsClassifier(), X, y,
param_name='n_neighbors',param_range=[1,3,5,7], cv=3). 

Let's try it!

Now that we have all the scores, it is easiest to analyze them with a plot. We will plot the average across folds as the number of neighbors increases.

Note that we have to call the parameter name differently now. Its the name we used in the Pipeline for KNeighborsClassifer() and double underscore to reference the parameter.

We can search over multiple parameters with a parameter search. The main techniques combine cross validation with either an exhaustive search over all combinations of parameters in GridSearchCV or random search over combinations of parameters in RandomizedSearchCV. We will begin with GridSearchCV. This will perform an exhaustive search over all combinations of parameters entered, each combination calculated over \(cv\) folds. The full parameter list is

 from sklearn.model_selection import GridSearchCV
 GridSearchCV(estimator, param_grid, scoring=None, n_jobs=None,
    iid='deprecated', refit=True, cv=None, verbose=0,
    pre_dispatch='2*n_jobs', error_score=nan, return_train_score=False)

Most of these should look familiar. We specify the parameters we want to examine with a dictionary for param_grid. We can access the parameters that return the best score on the testing set with grid_knn_acc.best_params_ and their corresponding scores in grid_knn_acc.best_score_. Note that the default setting will not return the scores on the training data to reduce the number of calculations made. It is still useful to compare training and testing scores, but it may be better to go back and do that on just the best model rather than storing all the scores on the training data. A full set of results cana be obtained with grid_knn_acc.cv_results_.

Example without a Pipeline and with only one parameter.

Example with a Pipeline and two parameters.

Depending on the size of our model, an exhaustive grid search, might not be the way to go. Another option is RandomizedSearchCV. A fixed number of parameter values are examined here (specified by n_iter=10.) Parameters are specified with a dictionary or list of dictionaries in param_distributions. Ranges may be specified as a list (recommended for discrete parameters) or a distribution (recommended for continuous parameters). We'll mostly stick to lists of ranges for parameters. The full list of parameters is

 RandomizedSearchCV(estimator, param_distributions, n_iter=10,
    scoring=None, n_jobs=None, iid='deprecated', refit=True,
    cv=None,verbose=0, pre_dispatch='2*n_jobs', random_state=None,
    error_score=nan, return_train_score=False)

Let's examine a different classification technique. Let's apply LogisticRegression to the fruit data set and examine the effects of regularization. Note that the regularization parameter for Logistic Regression is C where \(C=\frac{1]{\alpha}\text{.}\) So large values of C indicate small amounts of regularization. We usually want to vary C by powers of ten. How do you think Logistic Regression will do compared to k Nearest Neighbors?

Subsection 6.3 Evaluating Classification Models

As we are adding models to our collection, how do we determine which is the best one for our data? What does best even mean? For classification models, examining the accuracy of the model is always a good score to examine, but it may not help us determine the best model for our situation.

Some other model evaluation techniques that are often used include Confusion Matrices, Precision vs Recall graphs, and ROC curves.

A Confusion Matrix records how accurate our model is in each class relative to each other class. For two classes, there are four categories,

1. True Negative (TN): The data has class 0 and the algorithm predicts class 0.
2. False Negative (FN): The data has class 1 and the algorithm predicts class 0.
3. False Positive (FP): The data has class 0 and the algorithm predicts class 1.
4. True Positive (TP): The data has class 1 and the algorithm predicts class 1.

and we can visualize this as a \(2 \times 2\) matrix.

For \(k\) classes, we can visualize this as a \(k \times k\) matrix. Note TN,FP,FN,TP no longer make sense in this case as there are many more than four cases.

In the last example for logistic regression, even our best model didn't do very well. Why not? For simplicity we are looking at the confusion matrix on the full data set, but really we should only be looking at it on the testing data.

The trade-off between Precision and Recall is usually a function of our parameter tuning. We often want to analyze this to choose the best model based on how we are going to be using the data.

The term Recall (also called True Positive Rate) is defined as \(\frac{\text{TP}}{TP+FN}\) for two classes. In general, this involves all entries in the row for class 1 (orange). What fraction of positive instances are correctly predicted as positive?

The term Precision (also known as Positive Predictive Value) is defined to be \(\frac{\text{TP}}{TP+FP}\) for two classes. In general, this involves all entries in the column for predicting class 1. What fraction of positive predictions are correct?

In some cases precision might be more important than recall and vice versa.

Ideally, we would like both precision and recall to be 100%, but generally as we vary the parameters, some parameters will have better precision at the expense of recall and vice-versa. Thus, we can examine precision versus recall scores to determine the parameters that give us the best model based on the desired use of the model/data.

from sklearn.metrics import precision_recall_curve

A sample precision versus recall curve from your textbook is below.

The term True Positive Rate = TPR (same as Recall) is defined as \(\frac{\text{TP}}{TP+FN}\) for two classes. In general, this involves all entries in the row for class 1. What fraction of class 1 is correctly predicted to be class 1?

The term False Positive Rate =FPR (also known as the probability of false alarm) = \(\frac{\text{FP}}{FP+TN}\) In general, this involves all entries in the row for class 0. What fraction of class 0 is incorrectly predicted to be class 1?

A ROC Curve (stands for Receiver Operator Characteristic Curve) is created by plotting the True Positive Rate as a function of False Positive Rate as various parameters are varied. (FPR on \(x\)-axis, and TPR on \(y\)-axis.) It was originally developed for operators of military radar receivers. In this case we are trying to find a balance between correctly detecting an event and detecting false alarms. Ideally, we want the TPR to be 100% and FPR to be 0%. However, normally as we vary parameters we can only detect all the true positives if we allow some false alarms, and we can only have no false alarms with a very poor ability to detect the desired event. Thus, we want to identify a model that corresponds to an appropriate trade-off between detection and false alarms that is determined by the given situation.

Image created by Martin Thoma, taken from Wikipedia

ROC curves are generally better when observations are balanced across classes. Precision-Recall Curves are generally better for imbalanced data sets.