The Least Squares approach to linear regression was discovered by Gauss and first published by Legendre, although there has been some historical controvesy over this point (a translation of Legendre's original from 1804). It has grown to become the workhorse of most statistical analysis: most common estimators can be cast into this framework, it is very mathematically tractable, and has been used for a long period of time.

From a machine learning standpoint, this is an example of supervised learning since we have data for the dependent (predicted) variable and the independent variable(s) (the predictors). This means that we will be training the model on some known data, and trying to find the best set of parameters to minimize the difference between our model and the observations.

The notation will vary by field, but it is common in Statistics to denote the observations of the dependent variable by the vector , observations of the independent variables by a matrix , and either or more often as the model with a set of parameters . As such, our least squares minimization problem can be defined as:

We can simplify the notation further by adding a vector of ones to the matrix X to denote the intercept term, at which point we will want to minimize this term:

Those coming from a more numerical background will find this same problem defined as .

The Least Squares solution is found my minimizing the sum of squares of the residuals (i.e. the difference between the prediction and the observations . Minimizing the Euclidean norm of the residuals, in the case of a classical linear model with i.i.d. residuals, is the best linear unbiased estimator (BLUE, from the Gauss-Markov Theorem). This is a common simplifying assumption in statistical models, and there are many manipulations to variables that can enable this to hold.

We will now start to focus on several direct methods for solving this problem, followed subsequently by iterative methods. R and Matlab have built-in methods for doing this, and they both take advantage of different characteristics of the input data (e.g. sparcity); as such, I would never advise using what I provide below in place of these functions, but only to get a better understanding of different methodologies.

A Simple Motivating Example

We start with a very simple example: 4 x and y data points:

X = matrix(1:4)
y = c(2, 1, 1, 1)
lm(y ~ X)

Similarly, the simple solution in Julia is available through the linreg() function which solves the problem efficiently with LAPACK:

X = [1:4]; y = [2, 1, 1, 1];
linreg(X, y)

Julia's linreg (which uses LAPACK) is much less rich than the infrastructure that supports lm in R. The only returned values are the coefficients. R's function accepts a formula, following on the design from the famous "White Book", "Statistical Models in S" from Chambers and Hastie. And it returns an lm.fit object, which provides a wealth of available data and features.

I want to look into several different methods for solving the least squares problem. R's lm functions use C and LINPACK under the hood, so this shouldn't be considered a fair comparison of their statistical functions, but more as a comparison of the languages themselves. I also don't go into mathematical derivations but simply provide the Julia code.

Normal Equations

[Note: By convention, I will be putting semi-colon's ; after expressions even though it isn't critical because it prevents Julia from printing out the return value.]

The normal equations result from defining the problem in terms of and finding the first order condition for a minimum.

Note that I am adding a column of 1's to the input matrix, as this will represent the intercept in the regression equation.

X = [1 1; 1 2; 1 3; 1 4]; y = [2 1 1 1]';
(m, n) = size(X);
C = X' * X; c = X' * y; yy = y' * y;
Cbar = [C c; c' yy];
Gbar = chol(Cbar)';
G = Gbar[1:2, 1:2]; z = Gbar[3, 1:2]'; rho = Gbar[3, 3];
beta = G' \ z

This makes use of the Cholesky decomposition (you can also find an implementation of a parallel cholesky decomposition in Julia from Omar Mysore as part of a project for MIT 18.337 "Parallel Computing").

[Note: This code is almost exactly valid in Matlab/Octave, with the only exception being that the returned values on line 2 should be in square parentheses and the matrix slicing on line 6 should be in round parentheses.]

QR factorization

The QR factorization solution breaks the matrix A into a matrix Q and an upper right triangle R. For this it can be convenient to break the Q matrix into two parts: Q1 corresponding to the part that intersects with R, and Q2 which corresponds to the zero part of R.

X = [1 1; 1 2; 1 3; 1 4]; y = [2 1 1 1]';
(m, n) = size(X);
(Q, R) = qr(X);
R1 = R[1:n, 1:n]; Q1 = Q[:, 1:n];
beta = R1 \ (Q1' * y)

SVD factorization

The SVD approach can handle situations in which the columns of matrix A do not have full rank.

X = [1 1; 1 2; 1 3; 1 4]; y = [2 1 1 1]';
(m, n) = size(X);
(U, S, V) = svd(X);
c = U' * y; c1 = c[1:n]; c2 = c[n+1:m];
z1 = c1 ./ S;
beta = V' * z1

You can also find a nice discussion of this with Matlab in Jim Peterson's "Numerical Analysis: Adventures in Frustration".

SVD provides a more precise solution than QR, but it is considerably more expensive to compute. As a result, modern software uses QR most of the time, unless there are specific cases in which that methodology underperforms or doesn't hold.

End notes

We will look at iterative methods in the next post. Then spend some time looking at how we can characterize the "goodness of fit" and statistical significance in the traditional sense. And finally look at ways to avoid overfitting (e.g. regularization) and models other than OLS (including weighted-least squares and logistic regression).

Some useful references on this subject:

* Much of what I showed above was based on sections from "Numerical Methods and Optimization in Finance" by Gilli, Maringer, and Schumann. I highly recommend this text to anyone involved with optimization in finance as it provides a strong overview of most major areas, including computational considerations.
* Another nice free resource is Steven E. Pav's "Numerical Methods Course Notes", which includes examples of direct and iterative methods for least squares using Octave.

[Note: The relevant Julia manual section lists all the relevant functions and should be considered the primary source.]

Linear algebra provides a mechanism for efficiently solving systems of equations. A single equation may be expressed as a row vector, such as:

As:

Similarly, systems of equations can be combined into a matrix:

Which can then be solved more efficiently using various different theorems.

I recommend Gilbert Strang's MIT course on Linear Algebra as a reference (as I did in an extremely short introduction to Linear Algebra with R).

Julia, like R, uses LAPACK (which makes use of BLAS) for all the related linear algebra functionality. LAPACK (Linear Algebra PACKage) is a software library for numerical linear algebra, written in Fortran 90. Dirk Eddelbuettel wrote an excellent paper (highly recommended) and a related R package (gcbd)benchmarking different BLAS implementations. Dirk shows four benchmarks in his paper: matrix crossproducts, SVD decomposition, QR decomposition, and LU decomposition. I give examples of the latter three of these below as these are core linear algebra algorithms.

Before starting, I wanted point out some recent news with the language:

• There is a now a public web repl hosted by Forio. This makes it easy to test out the language.
• For R developers, John Myles White published a quick language comparison.

Vector/Matrix Creation

I showed how to construct an Array in the last post on basic Julia syntax.

A matrix in Julia is nothing more than a 2-D array. There are many ways of constructing these. Some examples:

# Construction of arrays
A = randn(10) # an array of size 10
A = [1:10]

# Some special matrices
ID = eye(10) # The identity matrix

We can get various different parts of matrices with simple commands:

triu(M) # Upper triangle of matrix M
tril(M) # Lower triangle of matrix M
diag(M) # The diagonal vector from matrix M

And do basic matrix math as expected

M1 + M2
M1 - M2
M1 * M2
M1 \ M2 # Matrix division using a polyalgorithm (i.e. optimized for the type of matrix).
inv(M) # Inverse of matrix M, equivalent to 1/M

Vector/Matrix Maths

There are several important matrix decomposition (or factorization) methods which are worth exploring further and are available through LAPACK. These all seek to break a matrix into other important canonical forms.

LU Decomposition

The first method is LU decomposition, which was introduced by Alan Turing in 1948 in his paper "Rounding-off errors in matrix processes".

LU decomposition is a key step in several fundamental numerical algorithms, including as one method for solving a system of linear equations (as we shall see later), inverting a matrix (an extremely expensive operation), or computing the determinant of a matrix. This can be expressed in matrix notation as:

In Julia, we can compute the simple example on wikipedia:

With the lu function.
A = [4 3; 6 3]
A = [1 -2 3; 2 -5 12; 0 2 -10]
(L, U, p) = lu(A)
L
U

You will see that this matches the same output from R using the lu() function from Matrix:

library(Matrix)
M = matrix(c(4, 6, 3, 3), nrow=2)
expand(lu(M))

QR Decomposition

QR decomposition breaks a matrix into an orthogonal matrix and an upper triangular matrix can also be used to solve linear least squares. This is usually represented in matrix notation as:

Wikipedia shows several methods for solving for this simple matrix:

Which we can solve in Julia using the qr() function:

A = [12 -51 4; 6 167 -68; -4 24 -41]
(Q, R, p) = qr(A)
Q
R

Similarly, in R we can use the base function qr() (note that R uses LINPACK by default):

x = t(matrix(c(12, -51, 4, 6, 167, -68, -4, 24, -41), nrow=3))
qr.R(qr(x, LAPACK=TRUE))
qr.Q(qr(x, LAPACK=TRUE))

Singular Value Decomposition (SVD)

One of the most common routines is a singular value decomposition; this is also used in fitting a least squares model.

where U is a unitary matrix, the matrix Σ is a diagonal matrix with nonnegative real numbers on the diagonal, and the unitary matrix V* denotes the conjugate transpose of V.

Once again, we can solve for the example given on the wikipedia article as:

Which solves into:

This can be solved using the svd() function in Julia:

A = [1 0 0 0 2; 0 0 3 0 0; 0 0 0 0 0; 0 4 0 0 0]
(U, S, V) = svd(A)
U
S
V

Cholesky Decomposition

Lastly, we will need to use Cholesky factorization (or Choleski, depending on your background). This is a special case of LU factorization (symmetric), but it requires half the memory and half the number operations of an LU decomposition. This can be expressed in matrix notation as:

And can be solve in Julia with the chol() function:

A = [5 1.2 0.3 -0.6; 1.2 6 -0.4 0.9; 0.3 -0.4 8 1.7; -0.6 0.9 1.7 10];
R = chol(A)
R

It should be noted that I am showing these functions because they are important to solving statistical methods. But these functions are not relevant in so far as we are interested in Julia for the performance of the language, since they rely on LAPACK rather than native methods.

Now that we have briefly covered the notation for several matrix decompositions, we are ready to look at a least squares problem in the next post.

The Julia manual is already very detailed, so that should be considered the first source; I am here only going to scratch the surface, and put things in perspective (relative to R and Python/Pandas). Julia's syntax mostly resembles Matlab, so users of that language will be immediately comfortable.

But first, I would also be remiss if I didn't highlight two exciting recent developments:

• Doug Bates implemented GLM. Doug is the creator of important LME4 package in R, so he's really an expert on this subject.
• John Myles White has been falling in love with the language, and created a Julia version of Simulated Annealing.

Installation

The installation instructions are documented on the github page, but different builds are available for download here, including a windows build which was just made available thanks to Keno Fischer and Jameson Nash (available for download here). .

Julia is most readily available on Linux or Mac OS X. I am running Julia on Ubuntu. If you want full control over the language (i.e. use the source, Luke), then it may be easier to switch off Windows to Ubuntu (with Wubi) or Debian (with http://goodbye-microsoft.com/). Or there's always the virtual machine approach using VMWARE and a Bagside application.

Data Types

Julia is "typeless" like other dynamic languages, but it comes equiped with a really powerful type system. This means that you don't have to declare a variable type, but that you can do so and can easily create your own types. Julia is dynamically typed, but it can achieve such good performance through type inference with JIT from LLVM.

Just like R and Python, simply entering a number into the Julia REPL results in its immediate type inference without explicit declaration.

julia> typeof(1)
Int64

julia> typeof("hello world")
ASCIIString
Methods for generic function ASCIIString
ASCIIString(Array{Uint8,1},)

Julia also has the all-import unknown datatypes:

julia> 1/0
Inf

julia> NaN
NaN

And these might themselves be considered numeric types:

julia> NaN + 1
NaN

julia> NaN + "a"
+(Float64, ASCIIString)
no method +(Float64,ASCIIString)

The mathematical focus of Julia is also immediately apparent by the ability to specify mathematical formulas without excess notation.

julia> 1.5x^2 - .5x + 1
13.0

Also, similar to some Lisps, Julia supports imaginary and rational numbers (using the // operator). This is big deal, because beyond everything else, it means that you can avoid some floating point errors.

julia> 2//3 + 1//3 + 2 == 3
true

Arrays

Julia comes with a flexible array type, which can hold any number of dimensions.

Most simple arithmetic/logic functions can be used in a vectorized form by affixing a ".".

julia> x = randn(10)
[0.120646, 0.857561, 0.819921 ... -1.80995, -0.466323, -0.111218]

julia> x .* x
[0.0145555, 0.735411, 0.67227, 0.026993 ... 3.27594, 0.217457, 0.0123695]

Another very useful feature is comprehensions. This is based on the set notation in mathematics, and it basically defines and function and then iterates over values within that function.

julia> [ x^2 | x=1:10 ]
{1, 4, 9, 16, 25, 36, 49, 64, 81, 100}

Python users will find this very similar to list comprehensions.

As in NumPy, a Matrix in Julia is just a 2-D array:

julia> Matrix
Array{T,2}

Functions

Functions in Julia are very similar to functions in R and Python. They can be declared in long form or on one line:

function f(x,y)
x + y
end

julia> f(3, 4)
14

You can also return values with the return command, particularly if there are multiple routes through a function.

Julia also supports anonymous functions (equivalent to a lambda function in Python).

julia> map(x -> x/2.0, [1,3,-1])
[0.5, 1.5, -0.5]

Functions also support multiple return values and varargs (using the ellipsis ...). Functions currently do not support default parameter values, although that's in the works.

Julia also include methods which allow for multiple dispatch:

Although it seems a simple concept, multiple dispatch on the types of values is perhaps the single most powerful and central feature of the Julia language.

This allows for different behavior depending on the parameters passed to the function. This can readily be see by typing + at the console.

same_type_numeric{T<:Number}(x::T, y::T) = true
same_type_numeric(x::Number, y::Number) = false

julia> same_type_numeric(1, 2.0)
false

I have really glossed over many details (e.g. flow control and loops) as well as advanced language features (such as metaprogramming and parallel computing). But this should give a taste. All else is readily available in the manual.

Next up, I will give a short review of some linear algebra in Julia, before starting to look at basic statistical analysis in the language.

We want a language that’s open source, with a liberal license. We want the speed of C with the dynamism of Ruby. We want a language that’s homoiconic, with true macros like Lisp, but with obvious, familiar mathematical notation like Matlab. We want something as usable for general programming as Python, as easy for statistics as R, as natural for string processing as Perl, as powerful for linear algebra as Matlab, as good at gluing programs together as the shell. Something that is dirt simple to learn, yet keeps the most serious hackers happy. We want it interactive and we want it compiled.

This is music to my ears. This is what I want too! The thing that's really exciting is that it actually looks like the language may deliver on these things. And after a very short period of time, it is gaining a significant amount of traction on the Julia developers list (an important indicator for whether a language will succeed). [I was also interested to see that one of the language creators, Stefan Karpinski, was a high school classmate.]

I was recently reading the "Steve Jobs" biography and Jobs discussed one major realization after selling the Apply 1: to go beyond the geeks, it would be necessary to include the full package, such as a monitor, keyboard, and power supply. Julia is still at this early stage: it must be built off github, and only supports Linux and OS X. But the documentation is already extensive.

I use R and Python for all my research (with Rcpp or Cython as needed), but I would rather avoid writing in C or C++ if I can avoid it. R is a wonderful language, in large part because of the incredible community of users. It was created by statisticians, which means that data analysis lies at the very heart of the language; I consider this to be a major feature of the language and a big reason why it won't get replaced any time soon. Python is generally a better overall language, especially when you consider its blend of functional programming with object orientation. Combined with Scipy/Numpy, Pandas, and statsmodels, this provides a powerful combination. But Python is still lacking a serious community of statisticians/mathematicians.

There are always other languages to consider. I've tried OCaml, Haskell, J, K, Q, along with Matlab and Mathematica. These are all great languages and platforms. But they are generally lacking something, by either being expensive and closed source or simply lacking features and community support. It wasn't too long ago when people were considering Clojure with Incanter as an alternative. But while clojure is a nice language (i.e. Lisp is a nice language), Incanter is not a serious option for replacing R. For starters: it's performance was worse for very basic operations. And it doesn't have anywhere near the amount of libraries for analysis.

Julia and R

My interest has continued to grow with the active involvement of Douglas Bates and Harlan Harris on the Julia discussion list. Bates also wrote a nice blog post showing a performance comparison vs. R and Rcpp. Some of the discussion has been taking place on the Julia developers list:

• "R and statistical programming" from Harlan on Feb 25
• "RFC: data frame proposal"

The addition of a real data frame, and appropriate handling of NA/NaN values, will be a serious addition to Julia.

There has also been some discussion taking place on the R developers list.

The question remains: Is Julia a viable option for statistics and machine learning at this stage? I'm going to start a short blog series exploring some simple analysis with the language over the next few weeks to try and explore the language a little further. My hope is to learn a little about the language and draw some attention to interesting new developments.

[Note: I should also draw attention to Vince Buffalo's post on the same topic.]

I regretfully won't have time to fully explore the offerings over the next few months. I am signed up for Natural Language Processing, Probabilistic Graphical Models, Game Theory, Model Thinking, and Information Theory. Of these, Information Theory interests me the most since I have a background in the subject having studied Electrical Engineering (and signal processing). I started watching the lectures on the other classes and thus far continue to be extremely impressed.

While I won't be posting R code to accompany all of these lectures (I'm close to finishing a few more in the Stanford Machine Learning series, on Neural Networks, and that's a higher priority for me), I did want to point out a few relevant resources:

• For Natural Language Processing, there are two excellent books that are available for free online: "An Introduction to Information Retrieval" by Manning, Raghavan, and Schütze (2009) (book website: http://nlp.stanford.edu/IR-book/) and "Natural Language Processing with Python: Analyzing Text with the Natural Language Toolkit" by Bird, Klein, and Loper (2009). The course itself primarily uses "Speech and Language Processing" by Jurafsky and Martin. Another good book that I've reviewed in the past is "Foundations of Statistical Natural Language Processing" by Manning and Schütze (1999) (book website: http://nlp.stanford.edu/fsnlp/). R users can look at the NLP task view for further materials.
• For Probabilistic Graphical Models, the most widely used book is Probabilistic Graphical Models: Principles and Techniques by Friedman and Koller (2009). This is the also the primary (optional) text for the course as it is being taught by none other than Daphne Koller. R has a wealth of resources on this subject available in the graphical models task view.

As always, let me know if you come across other interesting material.

[This is just a short post on regularization to show how it can help improve the generalization of a model.]

Regularization and Ridge Regression

Continuing with the polynomial regression example from PRML 1.1, we now look at adding a penalty term to the error function. This will discourage the parameters from reaching large values during the optimization. Our old loss function for linear regression and logistic regression was:

Now adding the penalty term, it becomes:

Notice again that the loss function is identical for linear regression and logistic regression; what differs is the hypothesis function . [Note: If you are following along with PRML, then will notice that Bishop refers to this as the error function and parameters are labeled instead of .]

This particular form of regularization, using a quadratic penalty term, is known as ridge regression.

We can minimize the loss function as before using gradient descent, or using an explicitly solution from linear algebra. I have implemented these solutions but not posted them for the time being because the performance of the gradient descent solution is appalling. The closed form solution is already implemented in R in the MASS package, in the lm.ridge function. This function does not have a prediction function, so I have implemented this here.

In the last post, we saw before how increasing the model complexity resulted in a poor fit on the out-of-sample data. The more complex model is overfit to the training dataset. Here we can see the same diagram using ridge regression. At high model complexity, the fit still remains roughly constant because these additional terms are penalized.

I won't expand on regularization at this stage, although I will commit the gradient descent solution to the github project. I will expand further on these topics (looking at other regularization models such as Lasso) in later posts when I continue with ESL. For now, we will start moving onto neural networks in the next post.

The next set of lectures in Stanford CS229a (ml-class.org) covers regularization, a technique that is employed to avoid overfitting. This is tied to the concepts of parsimony, model selection, degrees of freedom, and the bias/variance trade-off. I consider this one of the most fundamental concepts in machine learning, so I want to spend a little time covering it before specifically looking at regularization techniques.

This material is covered through-out the machine learning textbooks, but is especially covered in Chapter 7 of ESL and in 3.1.4 and 3.2 of PRML.

The generalization performance of a learning method relates to its prediction capability on independent test data. Assessment of this performance is extremely important in practice, since it guides the choice of learning method or model, and gives us a measure of the quality of the ultimately chosen model. (ESL 7.1)

Underfitting/Overfitting

In some of the earlier lectures, we saw how a simple linear model could be used to fit potentially complex data.

For this section, I will be reproducing the analysis in PRML 1.1, which is very similar to the material covered by Professor Ng. Suppose that we have a process which generates data in the form of a sine wave + some noise :

We want to fit a linear model to the data, but don't know what the underlying function is (in other words, we have 10 data points, but don't know that they were generated by a sine function). We might start with a simple linear model:

And progressively add more polynomial terms:

These additional terms will improve the fit to the training data, but in the process they reduce the generalization of the model.

The real function is in red and the model is in red. We can see that adding more polynomial variables improves the fit. The 9th polynomial passes directly through every data point. But it is nothing like the underlying function. So we can tell immediately that this function has been overfit to the data and won't generalize to other datasets from the same distribution.

How can we tell which parameters to leave in the model (known as "model selection")? How can we avoid overfitting?

There are several ways to solve this problem:

1. Get more data (typically impossible)
2. Choose the model which best fits the data without overfitting (very difficult)
3. Reduce the opportunity for overfitting through regularization/shrinkage

Let's first look at how getting more data would solve the problem. In the case of the 9th polynomial, having more data ensures that it fits closer to the actual distribution.

We can see that adding more data reduces the extreme values in the prediction, and the high-order polynomial starts to look more and more like the underlying sine function. This is an important lesson: the size of the dataset is a critical ingredient, especially for a model with many parameters.

Bias/Variance Tradeoff

The bias/variance trade-off is one of the most important concepts to understand in statistical learning theory. This is covered explicitly in CS229 notes 4. Bias is a measure of how well the model fits the data. Variance characterizes how much the prediction varies around its average. In our sine wave example above, the linear model has high bias (fits very poorly) and low variance (the predictions are consistent, regardless of the specific dataset). On the other hand, the 9th polynomial has low bias on the training data (fits the training data extremely well) and high variance (the predictions vary widely and this won't fit well to other data).

However with too much fitting, the model adapts itself too closely to the training data, and will not generalize well (i.e., have large test error). In that case the predictions will have large variance...In contrast, if the model is not complex enough, it will underfit and may have large bias, again resulting in poor generalization. (ESL 2.9)

We can decompose the mean-squared error (MSE) into bias and variance terms:

There are many different ways to characterize the performance of the model on in-sample (training) and out-of-sample (test and validation) datasets.

PMRL 1.1 makes use of the root-mean-square (RMS) error function (updated for our loss function convention):

To see how this trade-off operates, I divide the data into two sections: test and training. Using our original polynomial model, I progressively increase the model complexity by adding more parameters and see how the error function works.

What we see is that the lower-order polynomials (low model complexity) have high bias and low variance. In this case, the model fits poorly consistently. On the other hand, the higher-order polynomials (high model complexity) fit the training data extremely well and the test data extremely poorly. These have low bias on the training data, but very high variance. In reality, we would want to choose a model somewhere in between, that can generalize well but also fits the data reasonably well.

We will conclude this topic as part of the Stanford Machine Learning series in the next post by looking at dimension reduction techniques and the effective degrees of freedom.

The first classification model introduced in known as logistic regression (even though it is not technically a regression model since it is used for classification), which is a generialized linear model (GLM) used for binomial regression (two possible values, such as TRUE/FALSE, YES/NO). Logistic regression is covered in ESL 4.4 and PRML 4.3.2. It's also covered in Chapter 5 of my favorite regression book. "Data Analysis Using Regression and Multilevel/Hierarchical Models".

Logistic Regression

Logistic regression is covered in CS229 notes 1, although that goes into far more detail (especially on GLM's) than in CS229a. For classification, we need our function to be constrained to several discrete values. In the case when we have two groups (e.g. true/false, on/off) then we want to constrain our hypothesis to two values:

This is expressed through the sigmoid (or logistic) function.

This looks like an "S" shape, moving between 0 and 1.

Here we are expression our belief in the hypothesis as a probability, where we might choose a threshold (e.g. the hypothesis = 1 if it is greater than 0.5).

I'm going to use the South Africa Heart Data from ESL. The SA Heart data is used in several places in ESL:

A retrospective sample of males in a heart-disease high-risk region of the Western Cape, South Africa. There are roughly two controls per case of CHD. Many of the CHD positive men have undergone blood pressure reduction treatment and other programs to reduce their risk factors after their CHD event. In some cases the measurements were made after these treatments. These data are taken from a larger dataset, described in Rousseauw et al, 1983, South African Medical Journal.

As discussed in the past, assuming your dataset isn't too large, a scatterplot matrix is a really useful way to quickly look at data:

This reproduces Figure 4.12 from ESL.

Cost function and Gradient Descent

Gradient Descent works in much the same way with logistic regression as with linear regression. First, we can define the cost function in the same way as before, except that now our hypothesis is different (is a function of the sigmoid function):

As before, it is considerably easier to scale the features before applying gradient descent.

Multiple classes

Classification can also be applied in the case of multiple classes (or groups). One extension of logistic regression is known as multinomial logistic regression. The most famous dataset for this kind of analysis is Fisher's iris dataset (which is already in the R's datasets base package), from his "The use of multiple measurements in taxonomic problems." (1936). From R's help file on the data (help(iris)):

This famous (Fisher's or Anderson's) iris data set gives the measurements in centimeters of the variables sepal length and width and petal length and width, respectively, for 50 flowers from each of 3 species of iris. The species are Iris setosa, versicolor, and virginica.

Here I show how to apply Linear Discriminant Analysis and Multinomial Logistic Regression to this three-class problem.

Typically we would assess the performance of these models by dividing the data into training and test samples, and possibly choosing the parameters through cross-validation. I expect to touch on these issues in later posts as I continue this series on Stanford's open machine learning class.

As part of this, Professor Ng also provides more guidance on how to use Gradient Descent, and introduces the most widely used analytic solution to linear regression: the normal equation.

[Note: I have now committed all this code to github as an R package, which I'm currently calling stanford.ml. Currently the code is mostly contained in demo files, so you can load the package and then call the particular demo (for instance, this post could be run with demo("multivariate.regression")). My plan for the package is to build generic functions into the package, have demo files to walk through everything step-by-step, and then have a vignette to give a full description of everything. I may post this to CRAN once it's sufficiently well developed (at this stage it fails R CMD check because of lack of documentation, etc.). More details to follow. As always, feel free to fork the project and contribute!]

Multivariate Regression

It is a simple extension from univariate linear regression to multivariate regression. We can simply add more variables:

Or more concisely, if we set then we can write this in matrix notation as:

For these examples, I will continue to use the housing dataset from the UCI Machine Learning Repository. I will just use four of the available variables -- CRIM: per capita crime rate by town, RM: average number of rooms per dwelling, PTRATIO: pupil-teacher ratio by town, and LSTAT: % lower status of the population -- to predict MEDV: Median value of owner-occupied homes in $1000's:

Before looking at the data, we would expect all of the variables to have an influence. CRIM, PTRATIO, and LSTAT should have a negative coefficient (higher values would result in a lower property value) while RM should have a positive coefficient (more rooms would result in a higher property value). This is our null hypothesis.

If we plot these variables in R as a scatterplot matrix, we can see some clear relationships, in line with our expectations.

We can fit a linear model in R and look at the resulting statistics. All variables are significant (in terms of t-stats) and have values in line with what we might expect.

Optimizing with Gradient Descent

In the last post, we introduced Gradient Descent as an optimization method to find the minimum of the loss function. The loss function and gradient descent now have multiple variables, but all the other details remain the same.

Here I show the optimization path for the raw dataset (unscaled) given different values for the learning rate (). We can see that the algorithm converges on the right answer with very small values of alpha. When it doesn't converge, the values blow out to infinity. This happens because the steps taken along the loss function gradient are too large, and the optimization keeps missing the minimum value by larger and larger amounts.

Scaling the features before running gradient descent makes it easier to find the appropriate learning rate because the features are on the same scale.

The Normal Equation

Linear regression can actually be solved analytically using a little linear algebra. This is not true for most other machine learning models. This follows from the Gauss-Newton Theorem, which is itself a modification of Newton's method. One important result is the Gauss-Markov Theorem (covered in ESL 3.3.2), which finds that the best linear unbiased estimator (BLUE) of the coefficients is given by the ordinary least squares estimator.

There are many ways to derive the normal equations (wikipedia has a nice article on the subject), so I won't go through the derivation here. The normal equation is usually written as:

The hat notation means that this is an estimate. Using the normal equation is typically much faster than gradient descent, although it can be slower on very large data sets where taking the inverse matrix can be difficult.

I might expand on this subject more over time, but for now I would just highlight a few things:

1. I used Gilbert Strang's text when I was first learning the subject in school, and it was honestly one of my favorite textbooks. I recommend both Introduction to Linear Algebra and Linear Algebra and Its Applications. Strang is a true teacher: he loves the subject, and is committed to making complicated ideas understandable. And all the video lectures for his "Linear Algebra" and "Computational Science and Engineering" classes at MIT are available on OpenCourseWare.
2. You can find an introduction related to CS229 in Python with Numpy on Codebright's Blog.
3. The best R introduction to Linear Algebra that I could find is "Linear algebra in R" by Søren Højsgaard. This covers all the material required for CS229.

Basic Linear Algebra in R

Here are some of the basic ideas covered in the CS229a lectures.

For now, I won't spend any more time on linear algebra because I presume most readers are already familiar and I'd rather commit that time to exploring the next topics: multivariate and logistic regression, and regularization.