N.B. This post has been a long time coming due to a career change. I wasn't sure about the best way to present this, since it's clearly shaped into a very long post. Hopefully I can roll out the next few a little more quickly. I would certainly welcome any suggestions about the best way to present this in the future (e.g. should I post the code separately somewhere else, rather than cluttering up the post with it?).

Data Simulation

One of R's strengths is its collection of distribution functions which enable you to easily simulate complex datasets. Base R comes with most of the common probability distributions, and there are many more available through CRAN. There are many good overviews for these functions available, and it is also covered in the Introduction to R.

The most common distribution in the Normal (Gaussian) Distribution. We can easily generate random data from a normal distribution using the rnorm() function.

The data simulation used for this section is a Gaussian Mixture Model (GMM). A GMM is a very popular model in a number of different fields, as it allows for the creation of very complex density functions by combining several gaussians.

I start by creating two separate data sets centered at different points, which will be labeled "0" and "1" in the output variable Y.


library(ggplot2)
library(nnet)
library(MASS)
library(class)

# outcomes for the test and training sets
training.y <- c(rep(0, training.size), rep(1, training.size))
test.y <- c(rep(0, test.size), rep(1, test.size))

We now have our test and training data sets, for both the x and y variables. These will be used in both the linear regression and KNN models. I just add a few additional items to be used in the graphics.

# colors related to the outcomes
training.cols <- mycols[training.y + 1] # add 1 since R indexes start at 1 instead of zero
test.cols <- mycols[test.y + 1]

# make continuous values to cover the grid of points defining the model predictions
x.vals <- seq(min(c(training[,1], test[,1])), max(c(training[,1], test[,1])), len=grid.size)
y.vals <- seq(min(c(training[,2], test[,2])), max(c(training[,2], test[,2])), len=grid.size)
data.grid <- data.frame(expand.grid(x.vals, y.vals))
colnames(data.grid) <- c("x1", "x2")

Linear Models

The most common model used for statistical analysis is the linear regression fit by least squares. This relates inputs to an output value .

This can easily be converted into the vector form:

This can easily be converted into the vector form:

We can solve this with:

Given our test and training datasets derived earlier, we now want to fit a linear model.

lm.fit.training <- lm(y ~ x1 + x2, data=training)

##prediction on test
lm.yhat.test <- predict(lm.fit.training, newdata=test)
lm.yhat.test <- as.numeric(lm.yhat.test > 0.5)
print(paste("Linear regression prediction error in test:", 1-mean(lm.yhat.test == test$y), sep=" "))


This produces a training error of 26%, and a test error of 27.04%. It is nice to see that the training and test datasets don't perform drastically differently, which implies that we haven't overfit the data.

Now I plot the data itself, which is roughly similar to figure 2.1 in the text:

p <- ggplot(data=training)
p <- p + geom_point(aes(x1, x2, colour=color)) + geom_abline(intercept = b, slope = m)
print(p + geom_point(data=data.grid, aes(x1, x2, colour=col.grid), alpha=0.3) + opts(legend.position = "none"))

We can run the same for the test data, but I won't repeat that here. We can see that most of the blue points are in the blue region, and vice versa for the red points, so it appears to be doing a reasonable job classifying points correctly on either side of the decision boundary.

K-Nearest Neighbors

K-Nearest Neighbors is a very different model from linear regression, in particular since it does not assume any initial structure in the data. We can represent this with:

Where represents the neighborhood of defined by the k closest points. So we are really just averaging the values closest to x based on other observations in the same neighborhood (where the neighborhood size is defined by k).

We start off with k = 1, which means that each region will be decided based on the 1 nearest neighbor.

k <- 1

# same model on the test data
knn.yhat.test <- knn(training[,1:2], test[,1:2], training[,3], k=k)
print(paste("KNN prediction error in test:",1-mean((as.numeric(knn.yhat.test)-1) == test$y), sep=" "))


For k=1, we end up with a training error of 0%, and a test error of 34.31%. So clearly k=1 is fitting very closely to training dataset, at the cost of out of sample performance.

Now we plot the results, to create figure 2.3 in the text.

d <- transform(melt(matrix(knn.z.grid, grid.size)), x=x.vals[X1], y=y.vals[X2])
p <- ggplot(data=training)
p <- p + geom_point(aes(x1, x2, label=c("x1", "x2"), legend=FALSE, colour=color))
p <- p + geom_point(data=data.grid, aes(x1, x2, colour=col.grid), alpha=0.3, legend=FALSE)
print(p + geom_contour(data=d, aes(x, y, z=value), bins=0.5, color="#000000") + opts(legend.position = "none"))

We repeat the same for k=15 to generate figure 2.2 in the text.

k <- 15

# same model on the test data
knn.15.yhat.test <- knn(training[,1:2], test[,1:2], training[,3], k=k)
print(paste("KNN prediction error in test:",1-mean((as.numeric(knn.15.yhat.test)-1) == test$y), sep=" "))


This produces a training error of 24.5% and test error of 27.74%.

And plot the results:

d <- transform(melt(matrix(knn.15.z.grid, grid.size)), x=x.vals[X1], y=y.vals[X2])
p <- ggplot(data=training)
p <- p + geom_point(aes(x1, x2, label=c("x1", "x2"), legend=FALSE, colour=color))
p <- p + geom_point(data=data.grid, aes(x1, x2, colour=col.grid), alpha=0.3, legend=FALSE)
print(p + geom_contour(data=d, aes(x, y, z=value), bins=0.5, color="#000000") + opts(legend.position = "none"))

Conclusion

I have thus far demonstrated the usage of two very different models. It remains to be seen which models performs better on the data, and why. It is clear that the linear model is very rigid, while KNN is extremely flexible.

My next post will cover much of the rest of Chapter 2 in ESL, on statistical decision theory and the bias/variance tradeoff, which will help guide our model selection.

I will not be providing a complete syntax comparison; for that, you will need to go elsewhere (but let me know if you find any good ones!).

Some Key Points

Before getting into any real data structures, I would just note a few important things:

1. Python has a scalar data type, but everything is a vector in R. That means that vectorized operations in R are trivial, while with Python it requires an extra step, although there is a memory tradeoff. You can see this easily, by typing length(1) into R and len(1) into Python: notice that Python complains. The equivalent in Python would be len([1]) where we explicitly create a list of length 1.
2. Python is inherently object oriented, in a way that R is not. I may get into this in more detail later, but for now it suffices to note that a dot . has very special meaning in Python, which is the opposite of its behavior in R. It is an R convention that variable names will have dots interspersed instead of underscores (e.g. a variable could be called my.variable). Here the dot has no special meaning other than to make the name easier to read, just as you might use camelcase or an underscore in another language. In Python, a dot is used as it used in other object oriented languages: to denote a subobject. The easiest way to see this is to type the name of a class and add a dot, then hit tab. This will list everything contained in this class.
3. Both R and Python can to be used interactively. Python code is compiled at run time. There are different flavors of Python, the most popular of which is cPython, which compiles into Python byte code in pyc files. There are also Python "compilers" to distribute programs that won't require Python to be installed in order to run. R is getting closer to this kind of design with R 2.13 introducing Luke Tierney's compiler, which should be standardized in R 2.14.
4. Assignment is made in Python using =, while in R this can be handled with = or <- and -> (or the assign() function, if you want to give it a name).
5. R has a wonderful concept of a workspace, driven by using "environments" (which are similar to lists). You can search the R environment with the search() function, and view the contents of an environment with the ls() function. Python doesn't have the same construct. The closest thing that I have found is the dir() function. If you type this without any arguments, you will get a list of everything that is currently imported or assigned. Alternatively, you can pass it an object or module name (e.g. import numpy; dir(numpy) and it will list everything contained in the object or module
6. Getting help: In R, there are many ways to get help. To view the contents of a function, simply type the name without parentheses. To see the help file, type either help(function.name) or ?function.name. Similarly, in Python you can use help(functionName). I mostly use iPython, which also has the convenience functionName? and functionName?? (the extra question mark for viewing the source code).

Lastly, one last important difference: how assignments behave.

In R, when you assign something, it makes a deep copy of the object:

R> x = 1:2
R> y = x
R> x = c(x, 3)
R> x
[1] 1 2 3
R> y
[1] 1 2

So a change made to x had no impact on y in this example.

When you assign an object (does not apply to scalars) in Python, it creates a link between the objects, not a copy:

>>> x = [1, 2]
>>> y = x
>>> x.append(3)
>>> x
[1, 2, 3]
>>> y
[1, 2, 3]

In order to avoid this kind of behavior, you need to use the copy() function.

Arrays

Both R and Python have good 1-dimensional data structures for storing lists of objects (whether of the same type, or not).

Python has two basic array types: tuples and lists. R has one: vectors. There is a critical differences between the two langauges: in R, a vector has a specific type (will comment later on the R list). In Python, you can mix types.

The simplest way to create a new vector in R with the c() function:

R> a = c(1, 2, 3, 4)
R> a
[1] 1 2 3 4
R> class(a)
[1] "numeric"

Creating a list in python can be done with either the list() function or square brackets:
>>> a = [1, 2, 3, 4]
>>> a
[1, 2, 3, 4]
>>> type(a)
<type 'list'>

Notice that when we check the type of the Python list, it is a "list", while in R the type is "numeric". We can mix types in Python lists:

>>> a = [1, 2, 3, 4, "monty"]
>>> a
[1, 2, 3, 4, 'monty']

This is not the case in R, where everything gets cast into the same type:

R> a = c(1, 2, 3, 4, "monty")
R> a
[1] "1" "2" "3" "4" "monty"
R> class(a)
[1] "character"

Lists and Dictionaries

R's most powerful and flexible core data structure is the list, which forms the basis for the data frame.

R> x <- list('a' = 1, 'b' = 2)
R> x['a']
$a
[1] 1

Elements of a list can also be accessed by name with the $ operator.

R> x$a
[1] 1

The equivalent data structure in Python is the dictionary.

>>> x = {'a': 1, 'b': 2} # or equivalently:
>>> x = dict(a = 1, b = 2)
>>> x['a']
1

In both cases, these data structures can hold any kind of object, which makes them inherently very flexible.

In my next post, I'll cover multidimensional data structures (like R's matrix and data.frame), then move on to a rough statistical function comparison, and possibly end with a quick review of some of the data visualization tools. Some other topics that I could cover: functional aspects of the languages, iteration, vectorization, performance/HPC, time series analysis, and financial modelling. Let me know if any of these would be of interest to people, and I'll try to tailor the series a little!

I'm not one to start using something casually, so I am going for a deep dive into Python. I'm hoping to contribute to open source projects within the community as soon as I can; I tend to find this is the best way to really learn the language and to get to know the community. As with anyone in the Python data analysis world, I've been heavily relying on numpy and scipy. I have also started to venture into the world of scikits.statsmodels, which has an active community behind it including stats faculty from various universities (including Stanford), and I'm hoping that I can contribute something here before too long. And I'm really looking forward to trying scikit.learn.

I've been reading a few Python books on my train rides in the morning. Many of the most popular Python books are also available for free online. So far, I found "Dive Into Python" to be an excellent resource for an experienced programmer. I also found "Think Stats: Probability and Statistics for Programmers" to be a quick and enjoyable read (although don't expect to learn any statistics...). There's a recent list of free Python books here.

I thought that it might be good to blog some thoughts on this transition as it progresses. I became very attached to R over the last 5 years or so. I love the syntax, especially the vectorization of expressions. I love the language design around formulas and statistical models. The data visualization (especially when you include things like ggplot) is very difficult to beat. As are all the functional language features, along with data munging packages like plyr. The time series packages are very well thought out, and I made heavy usage of zoo and xts. Lastly, in my experience, the package system and CRAN far outstripped other language attempts at collaboration. Community really matters. And R's community is mostly comprised of scholars. It afforded an amazing opportunity to meet interesting people.

Thus far, my nicest surprise with Python is its object orientation. R, which is at heart a "non-structured" programming language (like APL), has never done a great job with providing true object oriented programming, whether you consider S3, S4, or R5. If you're coming from a background in Java or C++, this can be an occasional source of frustration. Python handles these concepts beautifully. In fact, it provides the best, most natural, and most extensive framework for object oriented programming that I have experienced.

I will post further comparing some of the core data structures in my next post on this subject, and then I'll move on to discussing some of the statistics and time series functions. At the end of the day, I won't be shocked if I end up using RPy2 occasionally.

[Note: For anyone following along with my ESL series, that should start up again soon; just blogging on some subjects that I have to come to terms with in the meantime.]

Look at your data

I would make one observation related to "example 2: prostate cancer": namely, that the scatterplot matrix can be a great tool for visualizing a new dataset (depending on the size; this doesn't work when there are too many dimensions). Moreover, it is glorious how R makes this visualization a trivial activity.

This dataset is available on the book website and also in the ElemStatLearn package on CRAN. We will revisit this dataset later in Chapter 3 when we discuss shrinkage methods, so you might as well install the package now. Now to create the scatterplot matrix we simply need to use the plot() function.

library(ElemStatLearn)
plot(prostate)

Which creates:

What does this tell us, even without knowing anything about the data? A few pretty obvious things:

• Two of the variables (svi and train) only have two possible values, so these may be something like true/false responses. Moreover, it looks like the gleeson variable only has only a few possible values. This kind of data is considered categorical (a factor in R).
• There are some clear relationships between some of the quantitative variables, as for instance between lcavol ~ lweight and lcavol ~ lpsa.
• Some of the other relationships are less clear, but it looks in a few cases like a transformation of the variables might lead to a relationship as there are many values clumped against on of the axes while the rest start to spread out.

This first post will reproduce one of the most famous early graphics. My goal is to use these posts to elaborate some important graphical concepts while also experimenting with and enhancing the webvis package. I invite others to reproduce these using ggplot.

Playfair's Wheat

William Playfair is often considered the founder of statistical graphics. His plot of wheat prices vs. wages and monarchies was originally posted in 1822 in "Letter on our agricultural distresses, their causes and remedies; accompanied with tables and copper-plate charts shewing and comparing the prices of wheat, bread and labour, from 1565 to 1821", addressed to the Lords and Commons, London (the entire original letter is available on Google books).

Playfair intended to demonstrate that “never at any former period was wheat so cheap, in proportion to mechanical labour, as it is at the present time.”

Playfair's Wheat in R

The data is available in the HistData package, as well as the webvis package itself. This is a relatively complicated graphic since it has multiple layers. The simplest way to walk through this visualization is to use the webvis demo, which follows the related Protovis example.

To run this, install webvis:

install.packages("webvis")
library(webvis)

Then run the demo:

demo("playfairs.wheat")

It can help to compare the code to the original Protovis example. The final result is parsed in a browser (doesn't work in old versions of IE, but will work in IE9).

This graphic uses three Protovis "marks": area, line, and bar.

R comes with one object called ts() which is useful for regularly spaced time series, such as daily, monthly, or yearly data (see help(ts) for more details). See "Time Series Analysis with R" for an example of how to work with this.

This is frequently insufficient for our purposes. As such, I will primarily use the zoo and xts packages on this blog. The other options are timeSeries (which is part of Rmetrics), its, or fts (from Whit Armstrong). I will touch on some of the differences along the way. You can find more about the time series package on the CRAN view.

zoo was created originally by Achim Zeileis in 2005, and it stands for "Zeileis's ordered observations", with many subsequent contributions from Gabor Grothendieck. One of the nice things about zoo is that it is an S3 class in R, and it works with most of the standard R matrix functions (such as summary, cbind, merge, and aggregate). Hence it has a relatively small learning curve and the authors put a lot of thought into making it just work as expected.

Here's a quick example creating a dummy multivariate time series, getting a summary of the output, and plotting it:

> x1 <- zoo(matrix(rnorm(12), nrow = 6), as.Date("2008-08-01") + 0:10)
> colnames (x1) <- c ("A", "B")
> summary(x1)
Index A B
Min. :2008-08-01 Min. :-1.6231 Min. :-1.3363
1st Qu.:2008-08-03 1st Qu.:-0.9867 1st Qu.:-0.7071
Median :2008-08-06 Median :-0.5078 Median :-0.5753
Mean :2008-08-06 Mean :-0.1310 Mean :-0.1270
3rd Qu.:2008-08-08 3rd Qu.: 0.6633 3rd Qu.: 0.6533
Max. :2008-08-11 Max. : 1.8866 Max. : 1.0704
> plot(x1)

Read the zoo vignette for more details.