[This is just a short post on regularization to show how it can help improve the generalization of a model.]

Regularization and Ridge Regression

Continuing with the polynomial regression example from PRML 1.1, we now look at adding a penalty term to the error function. This will discourage the parameters from reaching large values during the optimization. Our old loss function for linear regression and logistic regression was:

Now adding the penalty term, it becomes:

Notice again that the loss function is identical for linear regression and logistic regression; what differs is the hypothesis function . [Note: If you are following along with PRML, then will notice that Bishop refers to this as the error function and parameters are labeled instead of .]

This particular form of regularization, using a quadratic penalty term, is known as ridge regression.

We can minimize the loss function as before using gradient descent, or using an explicitly solution from linear algebra. I have implemented these solutions but not posted them for the time being because the performance of the gradient descent solution is appalling. The closed form solution is already implemented in R in the MASS package, in the lm.ridge function. This function does not have a prediction function, so I have implemented this here.

In the last post, we saw before how increasing the model complexity resulted in a poor fit on the out-of-sample data. The more complex model is overfit to the training dataset. Here we can see the same diagram using ridge regression. At high model complexity, the fit still remains roughly constant because these additional terms are penalized.

I won't expand on regularization at this stage, although I will commit the gradient descent solution to the github project. I will expand further on these topics (looking at other regularization models such as Lasso) in later posts when I continue with ESL. For now, we will start moving onto neural networks in the next post.

The next set of lectures in Stanford CS229a (ml-class.org) covers regularization, a technique that is employed to avoid overfitting. This is tied to the concepts of parsimony, model selection, degrees of freedom, and the bias/variance trade-off. I consider this one of the most fundamental concepts in machine learning, so I want to spend a little time covering it before specifically looking at regularization techniques.

This material is covered through-out the machine learning textbooks, but is especially covered in Chapter 7 of ESL and in 3.1.4 and 3.2 of PRML.

The generalization performance of a learning method relates to its prediction capability on independent test data. Assessment of this performance is extremely important in practice, since it guides the choice of learning method or model, and gives us a measure of the quality of the ultimately chosen model. (ESL 7.1)

Underfitting/Overfitting

In some of the earlier lectures, we saw how a simple linear model could be used to fit potentially complex data.

For this section, I will be reproducing the analysis in PRML 1.1, which is very similar to the material covered by Professor Ng. Suppose that we have a process which generates data in the form of a sine wave + some noise :

We want to fit a linear model to the data, but don't know what the underlying function is (in other words, we have 10 data points, but don't know that they were generated by a sine function). We might start with a simple linear model:

And progressively add more polynomial terms:

These additional terms will improve the fit to the training data, but in the process they reduce the generalization of the model.

The real function is in red and the model is in red. We can see that adding more polynomial variables improves the fit. The 9th polynomial passes directly through every data point. But it is nothing like the underlying function. So we can tell immediately that this function has been overfit to the data and won't generalize to other datasets from the same distribution.

How can we tell which parameters to leave in the model (known as "model selection")? How can we avoid overfitting?

There are several ways to solve this problem:

1. Get more data (typically impossible)
2. Choose the model which best fits the data without overfitting (very difficult)
3. Reduce the opportunity for overfitting through regularization/shrinkage

Let's first look at how getting more data would solve the problem. In the case of the 9th polynomial, having more data ensures that it fits closer to the actual distribution.

We can see that adding more data reduces the extreme values in the prediction, and the high-order polynomial starts to look more and more like the underlying sine function. This is an important lesson: the size of the dataset is a critical ingredient, especially for a model with many parameters.

Bias/Variance Tradeoff

The bias/variance trade-off is one of the most important concepts to understand in statistical learning theory. This is covered explicitly in CS229 notes 4. Bias is a measure of how well the model fits the data. Variance characterizes how much the prediction varies around its average. In our sine wave example above, the linear model has high bias (fits very poorly) and low variance (the predictions are consistent, regardless of the specific dataset). On the other hand, the 9th polynomial has low bias on the training data (fits the training data extremely well) and high variance (the predictions vary widely and this won't fit well to other data).

However with too much fitting, the model adapts itself too closely to the training data, and will not generalize well (i.e., have large test error). In that case the predictions will have large variance...In contrast, if the model is not complex enough, it will underfit and may have large bias, again resulting in poor generalization. (ESL 2.9)

We can decompose the mean-squared error (MSE) into bias and variance terms:

There are many different ways to characterize the performance of the model on in-sample (training) and out-of-sample (test and validation) datasets.

PMRL 1.1 makes use of the root-mean-square (RMS) error function (updated for our loss function convention):

To see how this trade-off operates, I divide the data into two sections: test and training. Using our original polynomial model, I progressively increase the model complexity by adding more parameters and see how the error function works.

What we see is that the lower-order polynomials (low model complexity) have high bias and low variance. In this case, the model fits poorly consistently. On the other hand, the higher-order polynomials (high model complexity) fit the training data extremely well and the test data extremely poorly. These have low bias on the training data, but very high variance. In reality, we would want to choose a model somewhere in between, that can generalize well but also fits the data reasonably well.

We will conclude this topic as part of the Stanford Machine Learning series in the next post by looking at dimension reduction techniques and the effective degrees of freedom.

The first classification model introduced in known as logistic regression (even though it is not technically a regression model since it is used for classification), which is a generialized linear model (GLM) used for binomial regression (two possible values, such as TRUE/FALSE, YES/NO). Logistic regression is covered in ESL 4.4 and PRML 4.3.2. It's also covered in Chapter 5 of my favorite regression book. "Data Analysis Using Regression and Multilevel/Hierarchical Models".

Logistic Regression

Logistic regression is covered in CS229 notes 1, although that goes into far more detail (especially on GLM's) than in CS229a. For classification, we need our function to be constrained to several discrete values. In the case when we have two groups (e.g. true/false, on/off) then we want to constrain our hypothesis to two values:

This is expressed through the sigmoid (or logistic) function.

This looks like an "S" shape, moving between 0 and 1.

Here we are expression our belief in the hypothesis as a probability, where we might choose a threshold (e.g. the hypothesis = 1 if it is greater than 0.5).

I'm going to use the South Africa Heart Data from ESL. The SA Heart data is used in several places in ESL:

A retrospective sample of males in a heart-disease high-risk region of the Western Cape, South Africa. There are roughly two controls per case of CHD. Many of the CHD positive men have undergone blood pressure reduction treatment and other programs to reduce their risk factors after their CHD event. In some cases the measurements were made after these treatments. These data are taken from a larger dataset, described in Rousseauw et al, 1983, South African Medical Journal.

As discussed in the past, assuming your dataset isn't too large, a scatterplot matrix is a really useful way to quickly look at data:

This reproduces Figure 4.12 from ESL.

Cost function and Gradient Descent

Gradient Descent works in much the same way with logistic regression as with linear regression. First, we can define the cost function in the same way as before, except that now our hypothesis is different (is a function of the sigmoid function):

As before, it is considerably easier to scale the features before applying gradient descent.

Multiple classes

Classification can also be applied in the case of multiple classes (or groups). One extension of logistic regression is known as multinomial logistic regression. The most famous dataset for this kind of analysis is Fisher's iris dataset (which is already in the R's datasets base package), from his "The use of multiple measurements in taxonomic problems." (1936). From R's help file on the data (help(iris)):

This famous (Fisher's or Anderson's) iris data set gives the measurements in centimeters of the variables sepal length and width and petal length and width, respectively, for 50 flowers from each of 3 species of iris. The species are Iris setosa, versicolor, and virginica.

Here I show how to apply Linear Discriminant Analysis and Multinomial Logistic Regression to this three-class problem.

Typically we would assess the performance of these models by dividing the data into training and test samples, and possibly choosing the parameters through cross-validation. I expect to touch on these issues in later posts as I continue this series on Stanford's open machine learning class.

As part of this, Professor Ng also provides more guidance on how to use Gradient Descent, and introduces the most widely used analytic solution to linear regression: the normal equation.

[Note: I have now committed all this code to github as an R package, which I'm currently calling stanford.ml. Currently the code is mostly contained in demo files, so you can load the package and then call the particular demo (for instance, this post could be run with demo("multivariate.regression")). My plan for the package is to build generic functions into the package, have demo files to walk through everything step-by-step, and then have a vignette to give a full description of everything. I may post this to CRAN once it's sufficiently well developed (at this stage it fails R CMD check because of lack of documentation, etc.). More details to follow. As always, feel free to fork the project and contribute!]

Multivariate Regression

It is a simple extension from univariate linear regression to multivariate regression. We can simply add more variables:

Or more concisely, if we set then we can write this in matrix notation as:

For these examples, I will continue to use the housing dataset from the UCI Machine Learning Repository. I will just use four of the available variables -- CRIM: per capita crime rate by town, RM: average number of rooms per dwelling, PTRATIO: pupil-teacher ratio by town, and LSTAT: % lower status of the population -- to predict MEDV: Median value of owner-occupied homes in $1000's:

Before looking at the data, we would expect all of the variables to have an influence. CRIM, PTRATIO, and LSTAT should have a negative coefficient (higher values would result in a lower property value) while RM should have a positive coefficient (more rooms would result in a higher property value). This is our null hypothesis.

If we plot these variables in R as a scatterplot matrix, we can see some clear relationships, in line with our expectations.

We can fit a linear model in R and look at the resulting statistics. All variables are significant (in terms of t-stats) and have values in line with what we might expect.

Optimizing with Gradient Descent

In the last post, we introduced Gradient Descent as an optimization method to find the minimum of the loss function. The loss function and gradient descent now have multiple variables, but all the other details remain the same.

Here I show the optimization path for the raw dataset (unscaled) given different values for the learning rate (). We can see that the algorithm converges on the right answer with very small values of alpha. When it doesn't converge, the values blow out to infinity. This happens because the steps taken along the loss function gradient are too large, and the optimization keeps missing the minimum value by larger and larger amounts.

Scaling the features before running gradient descent makes it easier to find the appropriate learning rate because the features are on the same scale.

The Normal Equation

Linear regression can actually be solved analytically using a little linear algebra. This is not true for most other machine learning models. This follows from the Gauss-Newton Theorem, which is itself a modification of Newton's method. One important result is the Gauss-Markov Theorem (covered in ESL 3.3.2), which finds that the best linear unbiased estimator (BLUE) of the coefficients is given by the ordinary least squares estimator.

There are many ways to derive the normal equations (wikipedia has a nice article on the subject), so I won't go through the derivation here. The normal equation is usually written as:

The hat notation means that this is an estimate. Using the normal equation is typically much faster than gradient descent, although it can be slower on very large data sets where taking the inverse matrix can be difficult.

I might expand on this subject more over time, but for now I would just highlight a few things:

1. I used Gilbert Strang's text when I was first learning the subject in school, and it was honestly one of my favorite textbooks. I recommend both Introduction to Linear Algebra and Linear Algebra and Its Applications. Strang is a true teacher: he loves the subject, and is committed to making complicated ideas understandable. And all the video lectures for his "Linear Algebra" and "Computational Science and Engineering" classes at MIT are available on OpenCourseWare.
2. You can find an introduction related to CS229 in Python with Numpy on Codebright's Blog.
3. The best R introduction to Linear Algebra that I could find is "Linear algebra in R" by Søren Højsgaard. This covers all the material required for CS229.

Basic Linear Algebra in R

Here are some of the basic ideas covered in the CS229a lectures.

For now, I won't spend any more time on linear algebra because I presume most readers are already familiar and I'd rather commit that time to exploring the next topics: multivariate and logistic regression, and regularization.

How can we find the best-fit line? This leads us into a question of optimization. Optimization typically involves finding the maximum or minimum value over a domain of possible values: in this case, we want to minimize the loss function.

...an optimization problem consists of maximizing or minimizing a real function by systematically choosing input values from within an allowed set and computing the value of the function. The generalization of optimization theory and techniques to other formulations comprises a large area of applied mathematics. More generally, optimization includes finding "best available" values of some objective function given a defined domain, including a variety of different types of objective functions and different types of domains.

Optimization

Optimization is a very big field encompassing many different kinds of techniques, ranging from linear programming, metaheuristics, particle swarm optimization, and genetic optimization. R has an entire view dedicated to the subject.

Much of Stanford Machine Learning will be concerned with optimization problems, so I won't go into too much detail on it now. But for all the autodidactic reader I would point out a few very good classes on optimization from Stephen Boyd at Stanford, including video lectures: EE263: Introduction to Linear Dynamical Systems, EE364a: Convex Optimization I, and EE364b: Convex Optimization II. In addition, Professor Boyd made his textbook -- "Convex Optimization" -- and all the related matlab code available for free.

Gradient Descent

Professor Ng introduces the first way of minimizing the cost function: gradient descent. In particular, CS229 covers batch gradient descent and stochastic gradient descent.

Gradient descent is discussed in ESL 11.4, PRML 5.2.4, and extensively through Marsland, all in the context of optimizing neural networks.

To form an intuition, we can refer back to our 3D plot of the least mean squares loss function. The loss function always forms a bowl shape in linear regression. When you think about trying to find the minimum value of a bowl, we are really looking for the place where the slope is zero (i.e. the derivative of the loss function = 0), but we want to find this point in the least number of steps. Imagine picking a random point on the surface, placing a ball there, and letting the ball roll down the surface. This is the essential idea behind gradient descent: pick a random point (i.e. set of parameter values) and try to iteratively find the steepest path down along the loss function surface.

More formally, for batch gradient descent, we want to update the parameter at each step as:

This derivative is easy to solve given our loss function and we end up with:

One unfortunate aspect of this algorithm is that we need to review every data point at each step. We repeat this updating process we reach convergence (at which point no longer updates). This is partly a function of our selection of a value for which dictates the size of our downward step at each point (also known as the learning rate.

If the dataset is very large, then this process can take a very long time to complete in which case we can use stochastic gradient descent: this simply means that at each step we only look at one data point rather than all. As a result, the process tends to wander around for a while rather than moving straight down the surface, but it can be much more efficient. So we simply modify the algorithm slightly to perform this iteratively rather than as a sum:

for i=1 to m:

There are many examples of gradient descent optimization in R already. In fact, Alexandre Martin has some very nice posts covering covering similar material based on exercise 3 of Professor Ng's OpenClassroom course.

You can also look at Chuck Anderson's lectures for CS545: Machine Learning at Colorado

Yihui Xie's animation package provides a very nice visualization of this process. This can be accessed from the grad.desc function (documented here). He has documented this fairly extensively. You can watch the optimization on a contour plot.

For python, there's a nice tutorial on this in the deeplearning documentation (including stochastic gradient descent, which is covered later in CS229). Stochastic gradient descent is also available in scikits.learn.

There are also several simpler examples of gradient descent in Python.

• "How to create a simple Gradient Descent algorithm" (on metaoptimize)

Gradient Descent Example in R

For this example, I will use the actual housing data from CS 229. I found this on the open classroom site.

On thing that isn't discussed much in the initial lectures is the fact that Gradient Descent is sensitive to feature scaling, so it is highly recommended to scale your data. We can see this easily in this example: the unscaled data explodes out to infinity (because alpha is too large), while the scaled values approach the values that result the normal equation (that is typically used to find the least-squared estimate).

Here is a plot of the optimization:

Machine Learning

What is machine learning? Ng quotes Arthur Samuel (1959) who is famous for his computer checkers:

Field of study that gives computers the ability to learn without being explicitly programmed.

He then goes on to give a more formal definition from Tom Mitchell:

A computer program is said to learn from experience E with respect to some task T and some performance measure P, if its performance on T, as measured by P, improves with experience E.

The key difference between machine learning and traditional AI is that machine learning provides intelligent behavior without any explicit programming: by learning from data. The field of "Deep Learning" is trying to move a step closer from this to AI.

Univariate Linear Regression

The first model introduced is linear regression with "one variable" (known as "univariate" in statistics, as opposed to multivariate covering more than one variable). A regression problem is generally contrasted with a classification problem: regression covers continuous variables while classification covers discrete variables (e.g. binary, groups). Linear regression is the most widely used model in statistics. This is partially true because it is easy to interpret the results; many machine learning algorithms are considered "black box" in that you can't easily interpret their meaning.

CS229 starts with a very simple example: the relationship between house square footage and price in Portland, OR. Before even looking at data for something like this, you would expect there to be a strong linear relationship: more square footage ~ higher price. I was unable to find this exact dataset online, so I'm using a similar housing dataset from the UCI Machine Learning repository.

Typically linear regression is denoted using the standard equation for a straight line:

CS229 uses a slightly different notation. The prediction is based on a hypothesis, so the model equation is defined as:

In this case, is the intercept (where the line hits the y-axis if x = 0) and is the slope of the line.

We find with our housing data that the equation for the line ends up being:

The predicted value is in $1000's, so setting x = 0 would imply that the value of a house with no rooms is -$34k. This is meaningless, given that a house cannot have zero rooms, but it's important to review the meaning of our fit model. We could adjust the model to give more meaningful values. Beyond the intercept, we see that each additional room is worth $9k. I'm not very familiar with this data, but this seems like a very small number, so I would want to investigate further before accepting this value.

Here is the plot (from the above code):

We can see a strong linear relationship (as expected), and the suggests that 48% of the predicted value is explained by the input variables.

Loss Function

Once we have a reasonable intuition for what it means to fit a straight line to data, the next question is: how does we do it? Ng introduces the cost function (more often called a loss function, or also an error or objective function) (introduced in PRML 1.5.2 and ESL 7.1). This is a function that we will aim to minimize (i.e. to minimize the loss from our model).

The problem with linear regression can be defined as minimizing the sum of the squared errors. CS229 defines the loss function as . This is a function of the parameters . In this case, the errors will be the difference between the prediction and the actual values .

This works out to be:

One of the great things about of Stanford CS229a is that it is partly intended to build an intuition about machine learning models. Having a deeper, intuitive understanding is invaluable when it comes to data analysis. In order to achieve this with the loss function, Ng tries varying the parameters over different values in and showing the behavior of the loss function . We can do this without formally optimizing to find a solution.

Given our housing data, what would it look like if we chose a number of different values for (the intercept) and (the slope)? We can test this easily by taking our original picture and plotting various other lines. We can also see what the values for the loss function as we do this. We can think of the loss function as characterizing the vertical distance between our line and every data point (since our line represents our model (or prediction), and we are given the horizontal data.

The loss function has two parameters, so it forms a 3D surface (intercept, slope, and loss).

I also show this as a contour plot with ggplot2.

I would love to demonstrate the loss function further. Please feel free to add to this if you have any good ways of showing its behavior.

The rest of this general introduction gives an example of optimization with gradient descent. I will cover that in the next post.

I will be blogging alongside the class to show how to implement some of the primary algorithms in code, mostly with R (and also possibly in Python, depending on time). I'm choosing R because it is presently the most well-developed open language for data analysis. The class itself uses Matlab and Octave.

Materials

Professor Ng provides very extensive notes for the course, and thus doesn't require any textbooks, although he does provide the following list as optional reading:

• PRML: Christopher M. Bishop Pattern Recognition and Machine Learning (Information Science and Statistics) Springer; 1st ed. 2006. Corr. 2nd printing edition (October 1, 2007); book website
• PC: Richard Duda, Peter Hart and David Stork, Pattern Classification (2nd Edition), John Wiley & Sons, 2001.
• ML: Tom Mitchell, Machine Learning. McGraw-Hill, 1997.
• ESL: Trevor Hastie, Robert Tibshirani, and Jerome Friedman The Elements of Statistical Learning: Data Mining, Inference, and Prediction, Second Edition (Springer Series in Statistics) Springer; 2nd ed. 2009. Corr. 3rd printing edition (February 9, 2009); book website (includes full text as PDF)
• RL: Richard Sutton and Andrew Barto, Reinforcement Learning: An Introduction (Adaptive Computation and Machine Learning). MIT Press, 1998

I have used all of these books in the past, except the last one. And I can recommend them all highly. Readers of this blog will know that I have been gradually working to reproduce some of the material in "The Elements of Statistical Learning", and I intend to continue that series as time permits. I would add one more to the list:

• MLAP: Stephen Marsland Machine Learning: An Algorithmic Perspective (Chapman & Hall/Crc Machine Learning & Pattern Recognition), 2009. Book website (includes Python code).

Marsland gives a very good introduction (with more of an emphasis on neural networks) and provides clear code examples in Python to accompany the discussion. Anyone looking for a practical guide will find this invaluable.

Throughout the blog posts, I will refer to sections of these texts as a given topic is covered, an will use the abbreviated form given in the above list (e.g. ESL for "Elements of Statistical Learning").

Blog Series

Any blogging that I do related to this Stanford class will focus on implementation of lecture material. The lectures themselves do an excellent job of explaining the concepts, so I don't see any point in being overly redundant. That said, I will try to highlight:

• Citations from ESL and PRML so a curious reader can put things more deeply in context.
• Any clear differences between machine learning and statistics that might confuse someone coming from the latter field.

It would go against the Stanford honor code for me to post solutions to homework problems.

All code will be posted on github, and encourage anyone who is interested to fork the project and contribute! (I noticed that jandot is providing code from this in clojure...let me know if you see anyone doing else doing this?)

I first heard about the AI class in the NY Times, and was excited because it is being co-taught by Peter Norvig. The machine learning class (CS229) is being taught by Andrew Ng, who taught the extremely popular machine learning class that was released by Stanford several years ago (videos are available on YouTube and on iTunes). I watched Ng's class in the past and really appreciate his extensive notes.

I am planning on going through the Machine Learning class again. I would love to blog about the material in either Python or R, although I suspect that I will have limited free time for it. Would anyone want to collaborate on this? If so, leave me a comment or send me a message (twitter: @statalgo).

There are currently a few different native ways to pull data into pandas, mostly contained in pandas.io (will be documented here).

• pandas.io.parsers contains functions for getting data from text files, csv, and Excel
• pandas.io.sql has functions for pulling data over SQL
• pandas.io.pytables allows for dealing with HDF5
• pandas.io.data now has functions to pull data from Yahoo! finance, the St.Louis FED (FRED), and Kenneth French's data library [NOTE: This is currently only available off git, so you will need to build it from source]

The inspiration for this is the getSymbols function in Jeff Ryan's quantmod R package, although this will eventually include non-financial functions as well.

Introducing pandas.io.data

Currently pandas.io.data contains one class: DataReader. This requires a symbol/dataset name and a data source (currently, either "yahoo", "fred", or "famafrench"). You can optionally provide as start and end date, which should be of type datetime. This returns a DataFrame for Yahoo! and FRED, and a dict of DataFrames from Fama/French.

DataReader("symbol name", "data source")

The Fama/French datasets are complex and require some investigation to use them. Pulling down a dataset will return a dict where each element is a separate DataFrame (sometimes with different indexes such as daily, monthly, or yearly factors). As an example, to get the original Fama/French factors from Fama and French, 1993, "Common Risk Factors in the Returns on Stocks and Bonds," Journal of Financial Economics:

ff = DataReader("F-F_Research_Data_Factors", "famafrench")

A quick example of how to use this with pandas. I run a simple univariate linear regression looking at standardized changes in GDP (not demeaned) regressed on the S&P 500 index:

I used the "adjusted close" price for the S&P500 returns. The regression is run on the full sample.

from pandas import ols, DataFrame
from pandas.stats.moments import rolling_std
from pandas.io.data import DataReader
import datetime

gdp_on_sp = ols(y=sp500_returns, x=DataFrame({"gdp": gdp_standard}))

Which will produce an OLS object.

-------------------------Summary of Regression Analysis-------------------------

-----------------------Summary of Estimated Coefficients------------------------
Variable Coef Std Err t-stat p-value CI 2.5% CI 97.5%
--------------------------------------------------------------------------------
gdp 0.0311 0.0162 1.92 0.0632 -0.0007 0.0629
intercept 0.0097 0.0546 0.18 0.8598 -0.0973 0.1168
---------------------------------End of Summary---------------------------------

You can also plot these time series easily with matlibplot (made easy if you're using iPython!):

sp500.plot()
gdp.plot()