Machine Learning

What is machine learning? Ng quotes Arthur Samuel (1959) who is famous for his computer checkers:

Field of study that gives computers the ability to learn without being explicitly programmed.

He then goes on to give a more formal definition from Tom Mitchell:

A computer program is said to learn from experience E with respect to some task T and some performance measure P, if its performance on T, as measured by P, improves with experience E.

The key difference between machine learning and traditional AI is that machine learning provides intelligent behavior without any explicit programming: by learning from data. The field of "Deep Learning" is trying to move a step closer from this to AI.

Univariate Linear Regression

The first model introduced is linear regression with "one variable" (known as "univariate" in statistics, as opposed to multivariate covering more than one variable). A regression problem is generally contrasted with a classification problem: regression covers continuous variables while classification covers discrete variables (e.g. binary, groups). Linear regression is the most widely used model in statistics. This is partially true because it is easy to interpret the results; many machine learning algorithms are considered "black box" in that you can't easily interpret their meaning.

CS229 starts with a very simple example: the relationship between house square footage and price in Portland, OR. Before even looking at data for something like this, you would expect there to be a strong linear relationship: more square footage ~ higher price. I was unable to find this exact dataset online, so I'm using a similar housing dataset from the UCI Machine Learning repository.

Typically linear regression is denoted using the standard equation for a straight line:

CS229 uses a slightly different notation. The prediction is based on a hypothesis, so the model equation is defined as:

In this case, is the intercept (where the line hits the y-axis if x = 0) and is the slope of the line.

We find with our housing data that the equation for the line ends up being:

The predicted value is in $1000's, so setting x = 0 would imply that the value of a house with no rooms is -$34k. This is meaningless, given that a house cannot have zero rooms, but it's important to review the meaning of our fit model. We could adjust the model to give more meaningful values. Beyond the intercept, we see that each additional room is worth $9k. I'm not very familiar with this data, but this seems like a very small number, so I would want to investigate further before accepting this value.

Here is the plot (from the above code):

We can see a strong linear relationship (as expected), and the suggests that 48% of the predicted value is explained by the input variables.

Loss Function

Once we have a reasonable intuition for what it means to fit a straight line to data, the next question is: how does we do it? Ng introduces the cost function (more often called a loss function, or also an error or objective function) (introduced in PRML 1.5.2 and ESL 7.1). This is a function that we will aim to minimize (i.e. to minimize the loss from our model).

The problem with linear regression can be defined as minimizing the sum of the squared errors. CS229 defines the loss function as . This is a function of the parameters . In this case, the errors will be the difference between the prediction and the actual values .

This works out to be:

One of the great things about of Stanford CS229a is that it is partly intended to build an intuition about machine learning models. Having a deeper, intuitive understanding is invaluable when it comes to data analysis. In order to achieve this with the loss function, Ng tries varying the parameters over different values in and showing the behavior of the loss function . We can do this without formally optimizing to find a solution.

Given our housing data, what would it look like if we chose a number of different values for (the intercept) and (the slope)? We can test this easily by taking our original picture and plotting various other lines. We can also see what the values for the loss function as we do this. We can think of the loss function as characterizing the vertical distance between our line and every data point (since our line represents our model (or prediction), and we are given the horizontal data.

The loss function has two parameters, so it forms a 3D surface (intercept, slope, and loss).

I also show this as a contour plot with ggplot2.

I would love to demonstrate the loss function further. Please feel free to add to this if you have any good ways of showing its behavior.

The rest of this general introduction gives an example of optimization with gradient descent. I will cover that in the next post.

[If you want a more complete introduction to Numpy, I recommend Timothy Oliphant's Numpy Guide, Python Scientific lecture notes by Emmanuelle Gouillart and Gaël Varoquaux, or the Tentative Numpy Tutorial. BioStatMatt also has a post comparing some of the key elements of Numpy arrays and R.]

On Dimensions

A dimension in mathematics typically refers to the number of variables (or features). One well-established result is the curse of dimensionality which shows that additional dimensions can actually lead to problems being insoluable.

Throughout this post I am using dimension to denote the number of spacial dimensions allowed in a dataset. In actual fact, a 2-dimensional dataset can hold an number of dimensions in a feature space. For instance, a time series is a very common data structure, where each row is denoted by a point in time. But each column could represent any number of other variables, such as position, temperature, size. In that example, we would have four feature space dimensions in a 2-dimensional dataset.

Installing Numpy

All the relevant R data structures come along with the core installation. To take advantage of Numpy requires a little more work. You can find installation instructions on the Scipy website.

My advice: the easiest way to go about this is to use iPython. And if you want to go a step farther, you can use pythonxy, which is what I use more of the time and includes everything but the kitchen sink.

Arrays in Numpy

You may recall from my prior post that lists in Python can mix types, while vectors in R are always of one type. So Python lists were somewhere between lists and vectors in R, while R lists are really more like dictionaries in Python. Moreover, Python lists are not "vectorized" in the same sense as R data structures. Some simple examples:

First, I take a simple vector containing five numbers and multiply it by 2. In R, this multiplies each element in the vector by 2:


> a = 1:5
> a
[1] 1 2 3 4 5
> a * 2
[1] 2 4 6 8 10


The same set of operations on a python list:


>>> a = range(1, 6)
>>> a
[1, 2, 3, 4, 5]
>>> a * 2
[1, 2, 3, 4, 5, 1, 2, 3, 4, 5]

Multiplying a list in Python is equivalent to the rep() function in R. To multiply each element, we would have to use something like a list comprehension:

>>> [b * 2 for b in a]
[2, 4, 6, 8, 10]

This is an where Numpy can be useful. Rather than using a list, we can use a Numpy array:

>>> from numpy import *
>>> b = array(a) * 2
>>> b
array([ 2, 4, 6, 8, 10])

Arrays are much more like vectors in R. They have associated types and perform operations on the entire object:

>>> b.dtype
dtype('int32')
>>> a.append("a")
>>> a
[1, 2, 3, 4, 5, 'a']
>>> array(a)
array(['1', '2', '3', '4', '5', 'a'], dtype='|S1')

So you can't mix types in a Numpy array.

Matrices

Numpy also has a 2-dimensional data type, similar to the R matrix. Although in actual fact, the Numpy array can also handle multidimensional data, and the difference between a numpy array and numpy matrix is subtle:

This is simply a transparent wrapper around arrays that forces arrays to be at least two-dimensional, and that overloads the multiplication and exponentiation operations. Multiplication becomes matrix multiplication, and exponentiation becomes matrix exponentiation.

It is entirely possible to skip numpy matrices in favor of arrays, in which case one would simply use a slightly different syntax when doing operations such as matrix multiplication.

In R, we can create a matrix by passing a vector and specifying the number of rows or columns:

> matrix(1:4, nrow=2)
[,1] [,2]
[1,] 1 3
[2,] 2 4

An R matrix is explicity 2-dimensional. For higher dimensions, one would need to use an R array(), which can hold any number of dimensions. An R matrix is actually just a vector with a superstructure that defines the length of a row or column. This has some interesting implications compared to a data.frame (which is truely a two dimensional object).

For instance, you can treat a matrix like a vector. Ordinarily a matrix is indexed with square brackets like [row, column], and a vector only has one index. A matrix can work with both:

> a <- matrix(1:4, nrow=2)
> a[1, 2]
[1] 3
> a[3]
[1] 3

This has implications elsewhere when using an R matrix, so it's useful to understand.

In Python, we can create a matrix and an array in the same way:

>>> array([[1, 2], [3, 4]])
array([[1, 2],
[3, 4]])
>>> matrix([[1, 2], [3, 4]])
matrix([[1, 2],
[3, 4]])

The indexing works similarly in most respects with square brackets [row, column], but again with some important differences:

>>> a = array([[1, 2], [3, 4]])

>>> a[1, 1]
4

Notice that in this case, the index starts at zero rather than at one (as in R). So to get the first element:

>>> a[0, 0]
1

We can alternatively work with indexes the same way we would with a python list.

>>> a[1][1]
4
>>> a[:2][:2]
array([[1, 2],
[3, 4]])

Where the first set of parentheses denotes rows, and the seconds operates on columns. Other common matrix operations (such as a transpose or matrix multiplication) are straight forward once we have an array or matrix.

Lastly, to round out this basic discussion of mutlidimensional data structures in R and Python, it is worth noting that R also has an array data type:

An array in R can have one, two or more dimensions. It is simply a vector which is stored with additional attributes giving the dimensions (attribute "dim") and optionally names for those dimensions (attribute "dimnames"). A two-dimensional array is the same thing as a matrix.

It is my experience that R arrays are used infrequently (certainly by comparison with vectors and matrices). For one thing, looking at more than 2-dimensions of data is difficult, especially when you are limited to numbers in a table or on a console. Visualization can help (e.g. by using colors, size, shape or video to display more dimensions), but additional dimensions inevitably add to the complexity of an analysis.

The fact is that most available datasets are by design 2-dimensional, or tabular, with variables as columns and observations as rows.

Which inevitably leads me into one of R's most useful data structures -- the data.frame -- and various Python equivalents in my next post. To state the problem: arrays and matrices (in both R and Python) can only handle 1 data type at a time. This is a big restriction. Of course, one way around this is to encode any non-numeric value as a numeric, but this requires an extra step. We shall explore various ways of dealing with this problem.

N.B. This post has been a long time coming due to a career change. I wasn't sure about the best way to present this, since it's clearly shaped into a very long post. Hopefully I can roll out the next few a little more quickly. I would certainly welcome any suggestions about the best way to present this in the future (e.g. should I post the code separately somewhere else, rather than cluttering up the post with it?).

Data Simulation

One of R's strengths is its collection of distribution functions which enable you to easily simulate complex datasets. Base R comes with most of the common probability distributions, and there are many more available through CRAN. There are many good overviews for these functions available, and it is also covered in the Introduction to R.

The most common distribution in the Normal (Gaussian) Distribution. We can easily generate random data from a normal distribution using the rnorm() function.

The data simulation used for this section is a Gaussian Mixture Model (GMM). A GMM is a very popular model in a number of different fields, as it allows for the creation of very complex density functions by combining several gaussians.

I start by creating two separate data sets centered at different points, which will be labeled "0" and "1" in the output variable Y.


library(ggplot2)
library(nnet)
library(MASS)
library(class)

# outcomes for the test and training sets
training.y <- c(rep(0, training.size), rep(1, training.size))
test.y <- c(rep(0, test.size), rep(1, test.size))

We now have our test and training data sets, for both the x and y variables. These will be used in both the linear regression and KNN models. I just add a few additional items to be used in the graphics.

# colors related to the outcomes
training.cols <- mycols[training.y + 1] # add 1 since R indexes start at 1 instead of zero
test.cols <- mycols[test.y + 1]

# make continuous values to cover the grid of points defining the model predictions
x.vals <- seq(min(c(training[,1], test[,1])), max(c(training[,1], test[,1])), len=grid.size)
y.vals <- seq(min(c(training[,2], test[,2])), max(c(training[,2], test[,2])), len=grid.size)
data.grid <- data.frame(expand.grid(x.vals, y.vals))
colnames(data.grid) <- c("x1", "x2")

Linear Models

The most common model used for statistical analysis is the linear regression fit by least squares. This relates inputs to an output value .

This can easily be converted into the vector form:

This can easily be converted into the vector form:

We can solve this with:

Given our test and training datasets derived earlier, we now want to fit a linear model.

lm.fit.training <- lm(y ~ x1 + x2, data=training)

##prediction on test
lm.yhat.test <- predict(lm.fit.training, newdata=test)
lm.yhat.test <- as.numeric(lm.yhat.test > 0.5)
print(paste("Linear regression prediction error in test:", 1-mean(lm.yhat.test == test$y), sep=" "))


This produces a training error of 26%, and a test error of 27.04%. It is nice to see that the training and test datasets don't perform drastically differently, which implies that we haven't overfit the data.

Now I plot the data itself, which is roughly similar to figure 2.1 in the text:

p <- ggplot(data=training)
p <- p + geom_point(aes(x1, x2, colour=color)) + geom_abline(intercept = b, slope = m)
print(p + geom_point(data=data.grid, aes(x1, x2, colour=col.grid), alpha=0.3) + opts(legend.position = "none"))

We can run the same for the test data, but I won't repeat that here. We can see that most of the blue points are in the blue region, and vice versa for the red points, so it appears to be doing a reasonable job classifying points correctly on either side of the decision boundary.

K-Nearest Neighbors

K-Nearest Neighbors is a very different model from linear regression, in particular since it does not assume any initial structure in the data. We can represent this with:

Where represents the neighborhood of defined by the k closest points. So we are really just averaging the values closest to x based on other observations in the same neighborhood (where the neighborhood size is defined by k).

We start off with k = 1, which means that each region will be decided based on the 1 nearest neighbor.

k <- 1

# same model on the test data
knn.yhat.test <- knn(training[,1:2], test[,1:2], training[,3], k=k)
print(paste("KNN prediction error in test:",1-mean((as.numeric(knn.yhat.test)-1) == test$y), sep=" "))


For k=1, we end up with a training error of 0%, and a test error of 34.31%. So clearly k=1 is fitting very closely to training dataset, at the cost of out of sample performance.

Now we plot the results, to create figure 2.3 in the text.

d <- transform(melt(matrix(knn.z.grid, grid.size)), x=x.vals[X1], y=y.vals[X2])
p <- ggplot(data=training)
p <- p + geom_point(aes(x1, x2, label=c("x1", "x2"), legend=FALSE, colour=color))
p <- p + geom_point(data=data.grid, aes(x1, x2, colour=col.grid), alpha=0.3, legend=FALSE)
print(p + geom_contour(data=d, aes(x, y, z=value), bins=0.5, color="#000000") + opts(legend.position = "none"))

We repeat the same for k=15 to generate figure 2.2 in the text.

k <- 15

# same model on the test data
knn.15.yhat.test <- knn(training[,1:2], test[,1:2], training[,3], k=k)
print(paste("KNN prediction error in test:",1-mean((as.numeric(knn.15.yhat.test)-1) == test$y), sep=" "))


This produces a training error of 24.5% and test error of 27.74%.

And plot the results:

d <- transform(melt(matrix(knn.15.z.grid, grid.size)), x=x.vals[X1], y=y.vals[X2])
p <- ggplot(data=training)
p <- p + geom_point(aes(x1, x2, label=c("x1", "x2"), legend=FALSE, colour=color))
p <- p + geom_point(data=data.grid, aes(x1, x2, colour=col.grid), alpha=0.3, legend=FALSE)
print(p + geom_contour(data=d, aes(x, y, z=value), bins=0.5, color="#000000") + opts(legend.position = "none"))

Conclusion

I have thus far demonstrated the usage of two very different models. It remains to be seen which models performs better on the data, and why. It is clear that the linear model is very rigid, while KNN is extremely flexible.

My next post will cover much of the rest of Chapter 2 in ESL, on statistical decision theory and the bias/variance tradeoff, which will help guide our model selection.

I will not be providing a complete syntax comparison; for that, you will need to go elsewhere (but let me know if you find any good ones!).

Some Key Points

Before getting into any real data structures, I would just note a few important things:

1. Python has a scalar data type, but everything is a vector in R. That means that vectorized operations in R are trivial, while with Python it requires an extra step, although there is a memory tradeoff. You can see this easily, by typing length(1) into R and len(1) into Python: notice that Python complains. The equivalent in Python would be len([1]) where we explicitly create a list of length 1.
2. Python is inherently object oriented, in a way that R is not. I may get into this in more detail later, but for now it suffices to note that a dot . has very special meaning in Python, which is the opposite of its behavior in R. It is an R convention that variable names will have dots interspersed instead of underscores (e.g. a variable could be called my.variable). Here the dot has no special meaning other than to make the name easier to read, just as you might use camelcase or an underscore in another language. In Python, a dot is used as it used in other object oriented languages: to denote a subobject. The easiest way to see this is to type the name of a class and add a dot, then hit tab. This will list everything contained in this class.
3. Both R and Python can to be used interactively. Python code is compiled at run time. There are different flavors of Python, the most popular of which is cPython, which compiles into Python byte code in pyc files. There are also Python "compilers" to distribute programs that won't require Python to be installed in order to run. R is getting closer to this kind of design with R 2.13 introducing Luke Tierney's compiler, which should be standardized in R 2.14.
4. Assignment is made in Python using =, while in R this can be handled with = or <- and -> (or the assign() function, if you want to give it a name).
5. R has a wonderful concept of a workspace, driven by using "environments" (which are similar to lists). You can search the R environment with the search() function, and view the contents of an environment with the ls() function. Python doesn't have the same construct. The closest thing that I have found is the dir() function. If you type this without any arguments, you will get a list of everything that is currently imported or assigned. Alternatively, you can pass it an object or module name (e.g. import numpy; dir(numpy) and it will list everything contained in the object or module
6. Getting help: In R, there are many ways to get help. To view the contents of a function, simply type the name without parentheses. To see the help file, type either help(function.name) or ?function.name. Similarly, in Python you can use help(functionName). I mostly use iPython, which also has the convenience functionName? and functionName?? (the extra question mark for viewing the source code).

Lastly, one last important difference: how assignments behave.

In R, when you assign something, it makes a deep copy of the object:

R> x = 1:2
R> y = x
R> x = c(x, 3)
R> x
[1] 1 2 3
R> y
[1] 1 2

So a change made to x had no impact on y in this example.

When you assign an object (does not apply to scalars) in Python, it creates a link between the objects, not a copy:

>>> x = [1, 2]
>>> y = x
>>> x.append(3)
>>> x
[1, 2, 3]
>>> y
[1, 2, 3]

In order to avoid this kind of behavior, you need to use the copy() function.

Arrays

Both R and Python have good 1-dimensional data structures for storing lists of objects (whether of the same type, or not).

Python has two basic array types: tuples and lists. R has one: vectors. There is a critical differences between the two langauges: in R, a vector has a specific type (will comment later on the R list). In Python, you can mix types.

The simplest way to create a new vector in R with the c() function:

R> a = c(1, 2, 3, 4)
R> a
[1] 1 2 3 4
R> class(a)
[1] "numeric"

Creating a list in python can be done with either the list() function or square brackets:
>>> a = [1, 2, 3, 4]
>>> a
[1, 2, 3, 4]
>>> type(a)
<type 'list'>

Notice that when we check the type of the Python list, it is a "list", while in R the type is "numeric". We can mix types in Python lists:

>>> a = [1, 2, 3, 4, "monty"]
>>> a
[1, 2, 3, 4, 'monty']

This is not the case in R, where everything gets cast into the same type:

R> a = c(1, 2, 3, 4, "monty")
R> a
[1] "1" "2" "3" "4" "monty"
R> class(a)
[1] "character"

Lists and Dictionaries

R's most powerful and flexible core data structure is the list, which forms the basis for the data frame.

R> x <- list('a' = 1, 'b' = 2)
R> x['a']
$a
[1] 1

Elements of a list can also be accessed by name with the $ operator.

R> x$a
[1] 1

The equivalent data structure in Python is the dictionary.

>>> x = {'a': 1, 'b': 2} # or equivalently:
>>> x = dict(a = 1, b = 2)
>>> x['a']
1

In both cases, these data structures can hold any kind of object, which makes them inherently very flexible.

In my next post, I'll cover multidimensional data structures (like R's matrix and data.frame), then move on to a rough statistical function comparison, and possibly end with a quick review of some of the data visualization tools. Some other topics that I could cover: functional aspects of the languages, iteration, vectorization, performance/HPC, time series analysis, and financial modelling. Let me know if any of these would be of interest to people, and I'll try to tailor the series a little!

I'm not one to start using something casually, so I am going for a deep dive into Python. I'm hoping to contribute to open source projects within the community as soon as I can; I tend to find this is the best way to really learn the language and to get to know the community. As with anyone in the Python data analysis world, I've been heavily relying on numpy and scipy. I have also started to venture into the world of scikits.statsmodels, which has an active community behind it including stats faculty from various universities (including Stanford), and I'm hoping that I can contribute something here before too long. And I'm really looking forward to trying scikit.learn.

I've been reading a few Python books on my train rides in the morning. Many of the most popular Python books are also available for free online. So far, I found "Dive Into Python" to be an excellent resource for an experienced programmer. I also found "Think Stats: Probability and Statistics for Programmers" to be a quick and enjoyable read (although don't expect to learn any statistics...). There's a recent list of free Python books here.

I thought that it might be good to blog some thoughts on this transition as it progresses. I became very attached to R over the last 5 years or so. I love the syntax, especially the vectorization of expressions. I love the language design around formulas and statistical models. The data visualization (especially when you include things like ggplot) is very difficult to beat. As are all the functional language features, along with data munging packages like plyr. The time series packages are very well thought out, and I made heavy usage of zoo and xts. Lastly, in my experience, the package system and CRAN far outstripped other language attempts at collaboration. Community really matters. And R's community is mostly comprised of scholars. It afforded an amazing opportunity to meet interesting people.

Thus far, my nicest surprise with Python is its object orientation. R, which is at heart a "non-structured" programming language (like APL), has never done a great job with providing true object oriented programming, whether you consider S3, S4, or R5. If you're coming from a background in Java or C++, this can be an occasional source of frustration. Python handles these concepts beautifully. In fact, it provides the best, most natural, and most extensive framework for object oriented programming that I have experienced.

I will post further comparing some of the core data structures in my next post on this subject, and then I'll move on to discussing some of the statistics and time series functions. At the end of the day, I won't be shocked if I end up using RPy2 occasionally.

[Note: For anyone following along with my ESL series, that should start up again soon; just blogging on some subjects that I have to come to terms with in the meantime.]

Look at your data

I would make one observation related to "example 2: prostate cancer": namely, that the scatterplot matrix can be a great tool for visualizing a new dataset (depending on the size; this doesn't work when there are too many dimensions). Moreover, it is glorious how R makes this visualization a trivial activity.

This dataset is available on the book website and also in the ElemStatLearn package on CRAN. We will revisit this dataset later in Chapter 3 when we discuss shrinkage methods, so you might as well install the package now. Now to create the scatterplot matrix we simply need to use the plot() function.

library(ElemStatLearn)
plot(prostate)

Which creates:

What does this tell us, even without knowing anything about the data? A few pretty obvious things:

• Two of the variables (svi and train) only have two possible values, so these may be something like true/false responses. Moreover, it looks like the gleeson variable only has only a few possible values. This kind of data is considered categorical (a factor in R).
• There are some clear relationships between some of the quantitative variables, as for instance between lcavol ~ lweight and lcavol ~ lpsa.
• Some of the other relationships are less clear, but it looks in a few cases like a transformation of the variables might lead to a relationship as there are many values clumped against on of the axes while the rest start to spread out.

This first post will reproduce one of the most famous early graphics. My goal is to use these posts to elaborate some important graphical concepts while also experimenting with and enhancing the webvis package. I invite others to reproduce these using ggplot.

Playfair's Wheat

William Playfair is often considered the founder of statistical graphics. His plot of wheat prices vs. wages and monarchies was originally posted in 1822 in "Letter on our agricultural distresses, their causes and remedies; accompanied with tables and copper-plate charts shewing and comparing the prices of wheat, bread and labour, from 1565 to 1821", addressed to the Lords and Commons, London (the entire original letter is available on Google books).

Playfair intended to demonstrate that “never at any former period was wheat so cheap, in proportion to mechanical labour, as it is at the present time.”

Playfair's Wheat in R

The data is available in the HistData package, as well as the webvis package itself. This is a relatively complicated graphic since it has multiple layers. The simplest way to walk through this visualization is to use the webvis demo, which follows the related Protovis example.

To run this, install webvis:

install.packages("webvis")
library(webvis)

Then run the demo:

demo("playfairs.wheat")

It can help to compare the code to the original Protovis example. The final result is parsed in a browser (doesn't work in old versions of IE, but will work in IE9).

This graphic uses three Protovis "marks": area, line, and bar.

Open source is a development method for software that harnesses the power of distributed peer review and transparency of process. The promise of open source is better quality, higher reliability, more flexibility, lower cost, and an end to predatory vendor lock-in.

- Open Source Initiative

I frequently see complaints about the performance of R. Most recently, this started with a series of blog posts from Radford Neal and followed by responses from many others including Christian Robert, Dirk Eddelbuettel, and Andrew Gelman.

I'm not going to reiterate what has already been said more ably by others who are far more intelligent and qualified, but I did want to make a few casual observations about why I feel that some of these authors are approaching this from the wrong direction:

All that said, I was really disappointed in Andrew Gelman's blog post most of all, and he seems more interested in the fact that he thinks that "the culture of R has some problems" rather than focusing on its strengths. Professor Gelman doesn't think that CRAN is "all that"; he could take or leave most of it if someone would only reprogram the main functions more elegantly in another language.

There are plenty of things about R that can be improved; performance is one of them. Is every package on CRAN perfectly crafted, or even useful? No. But CRAN is a remarkable gift to the world, full of things from the basic and useful to the esoteric and innovative models for data analysis. We should not overlook what we have in R: a language designed for data analysis that is constantly evolving through a huge, global effort of experts. And while it's hard to think about something after the fact, I suspect that what is happening in R couldn't have happened in another language. Community matters.

R comes with one object called ts() which is useful for regularly spaced time series, such as daily, monthly, or yearly data (see help(ts) for more details). See "Time Series Analysis with R" for an example of how to work with this.

This is frequently insufficient for our purposes. As such, I will primarily use the zoo and xts packages on this blog. The other options are timeSeries (which is part of Rmetrics), its, or fts (from Whit Armstrong). I will touch on some of the differences along the way. You can find more about the time series package on the CRAN view.

zoo was created originally by Achim Zeileis in 2005, and it stands for "Zeileis's ordered observations", with many subsequent contributions from Gabor Grothendieck. One of the nice things about zoo is that it is an S3 class in R, and it works with most of the standard R matrix functions (such as summary, cbind, merge, and aggregate). Hence it has a relatively small learning curve and the authors put a lot of thought into making it just work as expected.

Here's a quick example creating a dummy multivariate time series, getting a summary of the output, and plotting it:

> x1 <- zoo(matrix(rnorm(12), nrow = 6), as.Date("2008-08-01") + 0:10)
> colnames (x1) <- c ("A", "B")
> summary(x1)
Index A B
Min. :2008-08-01 Min. :-1.6231 Min. :-1.3363
1st Qu.:2008-08-03 1st Qu.:-0.9867 1st Qu.:-0.7071
Median :2008-08-06 Median :-0.5078 Median :-0.5753
Mean :2008-08-06 Mean :-0.1310 Mean :-0.1270
3rd Qu.:2008-08-08 3rd Qu.: 0.6633 3rd Qu.: 0.6533
Max. :2008-08-11 Max. : 1.8866 Max. : 1.0704
> plot(x1)

Read the zoo vignette for more details.

It looks to like a multivariate time plot where the area above the lines is filled. My only thought is to use a mosaic chart (as in this post on the Learning R blog), but this was the best I could do with a little bit of effort.  I think that using geom_ribbon would be better but I couldn't get the colors to work.

Here's the code. Is there an easier way to do this? How can I make the axes more like the original? What about the white lines between boxes and the gradual change between years? The sort order is also different.

    library(XML)
    library(plyr)
    library(ggplot2)
    theurl <- "http://en.wikipedia.org/wiki/List_of_Crayola_crayon_colors"
    tables <- readHTMLTable(theurl)
    n.rows <- unlist(lapply(tables, function(t) dim(t)[1]))
    crayola <- tables[[which.max(n.rows)]]
    x <- crayola[,c("Hex Code", "Issued", "Retired")]
    colnames(x) <- c("color", "issued", "retired")
    for (i in 1:ncol(x)) x[, i] <- type.convert(as.character(x[, i]))
    x[is.na(x[,"retired"]), "retired"] <- 2010
    x$color <- as.character(x$color)

    years <- min(x$issued):max(x$retired, na.rm=T)
    x2 <- na.omit(ldply(years, function(yr, x) {
      idx <- x$issued <= yr & x$retired >= yr
      x2 <- data.frame(year=yr, color=x[idx,"color"], size=(1/length(which(idx))))
      x2 <- x2[order(x2$color, decreasing=TRUE),]
      x2[,"xmin"] <- rep(0, nrow(x2))
      x2[,"xmax"] <- rep(1, nrow(x2))
      x2[-1,"xmin"] <- cumsum(x2$size[-1])
      x2[-nrow(x2),"xmax"] <- cumsum(x2$size[-nrow(x2)])
      x2
    }, x=x))

    p <- ggplot(x2, aes(xmin = year, xmax = year+1, ymin = xmin, ymax = xmax, fill=color))
    p <- p + theme_bw() + opts(legend.position = "none", panel.grid.major = theme_line(colour = NA),
                panel.grid.minor = theme_line(colour = NA))
    p.rect <- p + geom_rect() + scale_fill_identity()
    p.rect

Further improvements

Well, the R community never ceases to amaze. I posted this and within hours a vastly improved version was created by the Learning R blog (with some help from Baptiste on the color sorting). All the code is posted on that site. A suggestion was also made by Tobias to smooth the image with Cairo. Great work!

One crucial difference in his version (besides the vastly cleaner code) is his use of geom_area instead of the geom_rect in my version. That also allows you to set a white border above the image.

I would go so far as to say that (with the exception of things like better fonts and other touch ups) this R version is actually better than the original because it is more accurate. As I said previously, there were no color changes early in the timeline, despite that implication in the original chart.