Last March I went to SXSWi 2009. I posted the notes from two of the days, but conference exhaustion got in the way of me posting the notes from the last three days. After much delay, here they are.

---

Sunday Keynote: Nate Silver interviewed by Stephen Baker

Stephen Baker The Numerati, BusinessWeek

Nate Silver Founder and President, fivethirtyeight.com

Nate Silver:

I was using this to procrastinate about the other stuff I was supposed to be doing. It was born out of frustration with with Fox News. Polls were too much a part of the narrative. I’m NOT a big fan of polls. There is room for some common sense in this.

The narrative was too simplistic. They weren’t looking at the big picture.

From a data geek’s perspective, it was the greatest experience that I could have had. The election is in slow motion when they are going one state at a time.

People are sometimes fooled by randomness.

Race is complicated. We shouldn’t lump any racial group together. People are too quick to assume race is a problem.

Stephen Baker:

The demographics people are trying to organize people into tribes.

Nate Silver:

We don’t have groups of voters. We have 175 million voters and they are all different. There are an infinite number of topologies. Micro-targeted groups have been overused.

The decision of the voter is complicated. I don’t think they get enough credit. Sometimes “the issues” are overblown. People are looking for leadership and personality.

It would be fascinating to talk to the people who went against the stereotypes.

I didn’t go into this on a whim. I’ve learned about politics for a long time. I believe in being meticulous.

Be meticulous. Baseball and politics are LONG seasons. The puzzle comes together a little bit at a time.

I was urging patience at all times. The reaction I got was, “Thank you, you kept me calm.” Keep your eye on the big picture.

We found an Appalachian variable. TN KY WV. There is a strong feeling of ancestry. They consider themselves American (instead of Irish, etc.). A poor voter in KY is different than a poor voter in WI. Work with your intuition as well as the numbers. Look at the assumptions.

Lots of baseball talk that I don’t care about or even understand…

It’s kind of hard to know what to do with this economic crash. People are fearful in a way. It’s a 9-11 type of thing. You have people who are very pessimistic.

When will you start to blame Obama for the economy. There is a grace period. After 18 months, people will blame him just as much as Bush. People expect this recession will last REALLY long. The public is really pessimistic. If he pulls it up soon, he could really beat expectations.

Intra-country migration is the lowest right now. Unemployment is the key thing to look at, but it takes a while to catch up.

People focus on recentcy and people ignore the past. The volitility has decreased in the economy. The highs and lows aren’t as big. The Black Swan.

None of us should be that shocked that there are problems. It was a worry in the back of our heads. We’re very good for making up explanations for things when essentially what we’re seeing is random noise.

A lot of the time, it has more to do where the candidate is NOT, not where they ARE. Who went home?

You’re never going to be account for everything. Everything we try to do, we put probabilities around.

There are human factors in play here. Circumstances change.

I’m skeptical about evaluating job performance with numbers. A poorly executed objective evaluation is worse than a well executed subjective evaluation.

In some fields, a masters doesn’t help you. If I could be a good programmer, that would be more helpful than more a masters in economics.

I’m a relatively private person. I’m surprised at how much people are willing to share on the Internet right now. I’m in the email generation. The average 15 year old girl sends 1500 text messages a month. I have a compulsive personality, so if I got into Facebook, I wouldn’t see the light of day.

Should we get VC or should we boot strap it? It’s hard to know what to do. It would be nice to be able to hire a programmer.

We tried to do Oscar Predictions. They assumed that we could predict anything. Sometimes it’s okay to get things wrong. The difference between a zero and a missing result. I designed the model badly. Never come out with a forecast that you KNOW to be wrong. If you keep getting the wrong prediction, work on the fucking model!

In baseball, you have a PERFECT data set. If you screw up, there is no cost. The real world is not like that. It’s not like looking for a needle in a haystack, it’s like looking for one particular needle in a stack of needles. No one has any great answers. Collecting data itself is a challenge.

Perfect data set? BASEBALL It spoils you in a way, despite the intangibles. In baseball, everyone has to take their turn.

How did you collect the data for the election? We stole the polling sites. I had some reader emailing me results. Polling companies started sending me the info. There is a ton of data at the US Census Bureau.

What do you read? Fivethirtyeight is the number of electoral votes in the college. I love reading and I read a lot. I’m trying to educate myself right now. Irrational Exuberance, Nixonland, I love to read. I read a lot of books about half way through and then don’t finish them. When I write a book, you’re perfectly welcome to buy it an only read two chapters.

There is a point when you have enough good information, the bad information doesn’t matter. You take the best data that you have.

What is the Netflix Challenge?

If McCain had won, I would think it would be more screwed up. I think it’s more interesting to think about what if Hilliary had won? I think Obama is a little inexperienced and it might have been better if we had someone who had been in the White House before.

If you have someone who is winning, they must be doing something right. There is something like PURE political talent.

I don’t think business data is all that perfect because companies are editing their balancing their sheets. There are flaws in a lot of our data, despite the quantity of it.

Stephen Baker:

One of our challenges is how to be smart about data.

Nate Silver:

I’m not a huge fan of prediciton, actually. There are huge fluctuations. The systems are more chaotic than we give it credit for it. Sometimes crowds can be VERY wrong and you don’t have a way to look at them independently. How do people form their opinions about things? I’m suspicious of prediction markets.

Sometimes the favorite doesn’t always win. Journalists don’t really win the small print.

This turned out to be an easy election to forecast. Everyone knew that Obama was going to win.

Your data is only as good as the data that your recording. How can the data