Blayne Sucks

On Friday, December 11th, my MacBook Pro stopped working properly. I couldn’t get video regardless of what I did. I took it to the Apple store the next day, where I learned that my graphics logic board was the victim of the infamous NVIDIA recall. I was told that it would take up to 10 days to get it repaired. Just as I was starting to recover from the shock of being without my computer for 10 full days, the Apple employee who examined my laptop said they would need my username and password to complete the repairs.

There is no valid reason Apple needs a username and password to repair a graphics logic board. This is a basic principle of computer security: Do not give anyone your username and password. I asked why they wanted it, and I was told that they needed to be able to log into the machine to verify that it works. This is simply false, and I’m disappointed that Apple would claim it was true. Graphics can be tested in a variety of ways without using an existing username and password. First, they could have used the guest account on the machine. Second, they could have booted into an operating system on a CD/DVD such as Knoppix. Third, they could use a bootable USB drive. Fourth, they could boot from an external hard drive. These options are even documented on their website. Needless to say, I refused to give them my username and password. They refused to send the computer off to be fixed. I asked if there was anywhere else I could get it fixed. To their credit, the Apple store employees were prepared to give me a recommendation to Ten Plus Systems.

I knew almost immediately after walking into their store that Ten Plus Systems was a quality computer repair shop. First, I saw one of the technicians talking with the receptionist about a repair. They were clearly organized, and my gut told me immediately that the technician was a genuine computer geek. Second, they were selling an original, fully restored 1984 Macintosh. It was absolutely beautiful. It looked almost new, and a great deal of care clearly went into restoring this machine. I strongly believe that people who are experts in their field have an intuitive sense that allows them to identify other experts rapidly. (Read Blink by Malcolm Gladwell if you are interested in exploring this concept.) As a computer science PhD student who has built at least a dozen computers from parts, I consider myself an expert in this field. I could tell this store was run by experts.

I arrived Monday morning and my computer was fixed 26 hours later. It was basically a one day turn around on a repair that Apple said would probably take 10 days. They didn’t need my username or password. They didn’t even ask. Ten Plus Systems is an Apple-certified repair store, which means that any machine covered by AppleCare can be repaired there. They also repair Apple and PC machines not covered by AppleCare, and they recycle old computer parts for their customers. If you are near Raleigh and need computer repair work done, I would strongly recommend Ten Plus Systems based on my experiences with them.

Disclosure #1: According to the relatively new FTC rules for bloggers, I should disclose my connection with the companies I’m endorsing. I haven’t been paid for this post. I haven’t been given any gift of any kind for this post. I haven’t had an out-of-body experience in which I was in any way compensated for this post. (At least, not yet…) I’m just a genuinely satisfied customer.

Disclosure #2: I agree with Adam Thierer: the relatively new FTC rules for bloggers are almost completely unenforceable.

Last week Bruce Schneier commented on a story about a prison that let an inmate convicted of credit card fraud reprogram a prison computer. Schneier believes this sort of thing should be an “obvious” no-no, and I agree. However, it isn’t obvious to a lot of intelligent and well-intentioned people. In fact there’s consistently been debate on whether or not criminals should be hired for computer security positions. There are people who fervently believe the myth that being an excellent criminal carries over into being an excellent law enforcement officer or security adviser.

Unfortunately, pop culture continues to prop this myth up with TV shows like the USA Network’s upcoming White Collar. The show is about an FBI agent who teams up with his nemesis-turned-good-guy to solve crimes that no one else could solve. Another TV series, called Dexter, which appears on Showtime, portrays a forensics expert who secretly murders the criminals he finds through his work. Both of these shows operate on the premise that experience committing crimes is useful in preventing them.

In reality, committing crimes and preventing crime are fundamentally different activities not because of the skill sets but because of the motivation and interests involved. In fact, the skill sets may be strikingly similar in a lot of ways. Some pirates are excellent sailors, some outlaws can shoot extremely well, and some hackers know a lot about computers. Don’t focus on asking whether the skill sets overlap. Instead, focus on questions like these: Are they dependable? Can they work well with other people in your particular work environment? How do you know they are actually interested in helping your organization? How do you know they are truly reformed?

After focusing on these questions, the truth comes to light: it is very rare that an excellent criminal history translates to an excellent crime-prevention future. There is a reason that police departments do a criminal background check before hiring someone. There is a reason that day care providers don’t hire convicted child molesters. There is a reason that banks don’t hire convicted felons to do security. Why wouldn’t the same rationale carry over to information or computer-based crimes?

Now, there are instances of convicts making amends and turning their lives around. Frank Abagnale is perhaps the most famous of these reformed con men. Hollywood capitalized on his story with the highly successful movie Catch Me If You Can. I know several people who have heard him speak at security conferences, and they have told me that he continues to apologize for his life of crime at the beginning of his talks, decades after they occurred. In fact, he may be a good model of how to lead a life of contrite contribution to law enforcement after being an extremely skilled criminal. He worked long and hard to earn the trust of banks and the FBI. He was initially paid only for positive results, and used the money he earned as a security consultant to pay back his debts.

Still, as a general rule, it should be obvious that hiring anyone convicted of computer fraud to do computer security work is a bad idea. Why take the risk? There are a lot of extraordinarily talented computer security experts who do not have the baggage of a criminal record. If you find, after searching for a non-felon, that you need the particular skills or expertise of a convicted computer fraudster, then don’t put them in a position of power. Don’t trust them without oversight. Don’t get caught up in the Hollywood story. The Frank Abagnales of the world are exceedingly rare; hiring a felon to do computer security almost never ends well.

Recent cryptography news serves as a microcosm of the development of computer security technologies. The discovery of fully homomorphic encryption by Craig Gentry, a Stanford PhD student working at IBM this summer, is by far the biggest headline in cryptography theory this week, month, year, and (probably) decade. Essentially, fully homomorphic encryption can perform arbitrary computations on encrypted data while preserving the encryption. For example, a spam filter could be used to identify encrypted emails containing spam, or an audit logging system could append an entry into an encrypted log file without decrypting it and then re-encrypting it.

Now, nothing is perfect right out of the gate, and there are caveats to this discovery. For the scheme to work, one must know in advance the maximum number of computations that can be performed on an encrypted file. It’s not practical; the discovery shows only that it is possible. Last but not least, we’ve already developed schemes that allow some limited operations, such as search, on encrypted data. These have been around for years, and some have even been reported on technical news sites. But even taking these concerns into account, the discovery is legitimately headline news.

The media loves to report juicy computer security stories, particularly relating to the discovery of new cryptographic techniques. Unfortunately, these headlines distract from the primary concern of the average computer security professional: We are just not using the tools we have! Consider last summer when a flaw in the DNS protocol became huge news. It was a problem that could have been completely avoided using existing cryptography. We just weren’t using it. In fact, despite Dan Kaminsky’s recent efforts, we still aren’t using it. Here’s a great quote from Dan:

DNS is the world’s largest PKI without the ‘K.’All DNSSEC does is add keys.

Why haven’t we “added the ‘K’” yet? DNSSEC has been sitting in a drawer, and even after last summer, it doesn’t appear to be a priority. It is designed with security in mind from the start; it is real, practical, and can be implemented without another breakthrough in cryptography. Only, we aren’t using it. And this has been the pattern of cryptography technologies for the last few decades:

1. Some smart people create something like public key encryption and/or fight against ludicrous export controls on cryptography tools.
2. The story becomes headline news for a day or two, and we all walk around feeling great about how we ‘solved’ the security problem and we’re all going to be ‘safe’ soon.
3. A few weeks pass and we find that no one is actually using the inventions that were just created and/or saved from oppressive regulation.
4. Eventually, we start all over from Step 1 with a new miracle discovery in computer security. That’s what happened this week.

Consider email encryption. Gmail (and most other webmail providers) still doesn’t support GPG. Gmail also doesn’t use persistent SSL connections by default, which means that your emails are delivered to your web browser in plain text when there’s a cheap and effective form of encryption that could easily be enabled. This was old news when I blogged about it here nearly two years ago, but Google is “looking into whether it would make sense” only recently, perhaps because of a letter organized earlier this month by Chris Soghoian and signed by numerous computer security experts.

I’m not saying that fully homomorphic encryption isn’t important, or that solving this longstanding, open academic question isn’t an achievement. It is important, exciting, and a huge achievement. All I’m saying is that fully homomorphic encryption, or any security technology, won’t solve computer security and privacy problems unless we start using the tools we have.

Edited to add: Here’s a nice piece by Brian Krebs that talks more about the letter sent to Google about ecrypting by default. In particular, I love this quote:

“What we’re saying in this letter is that as an iconic service, and one that professes to be concerned about user safety, Google could set a good example and set the right defaults, and if users want to switch back to something less secure, then they can.”

Dr. Eugene Spafford

(Full Disclosure: I am working with Dr. Spafford this summer at CERIAS on campus at Purdue University.)

Recently Google made a particularly big announcement concerning the Google App Engine: it will support Java. This announcement comes shortly after announced support for Groovy. Google has been making a serious effort to “win” the cloud computing wars, but this is the first time I’ve really seen a path to victory.

If you have never heard of the Google App Engine, it is basically a way for developers to build web applications that will be hosted in a distributed “cloud computing” fashion by Google. There are quotas and application limits, but it is possible to try things out for free. If your application becomes a big hit, then you can scale using Google’s infrastructure. You can find out more here.

Google App Engine is a big deal to developers because the worst part about developing web applications is that deployment, maintenance, and installation are the job of the developers rather than the users. End users like web applications because they are available everywhere they can find a computer and Internet access, but also because they no longer have to do any installation or software maintenance like patching or upgrading. Google App Engine makes all of this dead simple for developers as well.

Perhaps the most interesting thing about this announcement are its impacts on “cloud computing.” Cloud computing is the worst-defined buzzword since network neutrality, so perhaps that’s why they are staying away from it. I’m using it to refer to any web application where the vast majority of the data needed to run the application is stored by the server. The infrastructure and hardware that store this data and run the web application are a complete unknown to the end users. Furthermore, this infrastructure can scale dynamically as it is needed and without end user knowledge. There are all kinds of things that would fall into this category: Gmail (or really any web-based email), Apple’s MobileMe, Dropbox, or Mozy. Truly, the list could go on and on.

The only “real” competition that Google has in this space is Amazon Web Services. (I’m sure that others will emerge, but developers can use the Google and Amazon offerings now.) Developers can use AWS API to manage data and payment services, but they still have to install, host, and maintain the hardware for their application on their own.  There are some platform-specific hosting providers built on top of Amazon Web Services that can help developers on that front.  For example, a Ruby on Rails developer could host an application on Heroku, which uses Amazon Web Services. By default though, there is no “platform” built into Amazon Web Services as there is with Google App Engine. Google App Engine is the only wholly-owned development platform native to the “cloud,” and it just got a lot better for developers.

And that is how Google could win the cloud computing wars. There’s actually precedence for this if you take a look at Java. Java “won” the language wars of the 1990s not because of its beautiful syntax (blech!) or because of its amazing GUI frameworks (hah!) but because of its universal JVM. This is why so many dynamic languages are being built or ported to the JVM. Look at JRuby or Groovy for examples. It’s really only a matter of time before Ruby, Python, Groovy, and JavaScript are all as fast or faster on the JVM than they are in their native environments. Other dynamic languages, like Scala, are being built specifically for the JVM. In short, the platform was everything in the language wars of the 1990s, so why would we expect it to be any different in the cloud?

Below is a (hilarious) video about executing ideas that I saw thanks to Merlin Mann’s posting of it at the beginning of the year. Warning: this video is possibly not safe for work watching due to some language.

[Side note: If you've never heard of Ze Frank before, then I would recommend Ze Frank's TED talk.]

One of the things I would like to focus on is a quote from Ze Frank that Merlin highlighted as well. This quote from the middle of the video:

And the longer they wait, the more they convince themselves of how perfectly that idea should executed…But the bummer is most ideas kind of suck when you do them.

I love this quote and really the whole section in the video where Ze talks about ideas. There’s something both true and subtle in what he says. Think about everything you’ve ever seen, read, heard, or come across that made you think, “Wow, that’s clever.” You would never have felt that way without someone else executing their idea. Here’s the subtle part: How many ideas are just as clever, but were not executed upon by their thinker?

Good economists recognize the possible value in unrealized potential. Bastiat may have been the first to write about what is seen and what is not seen. Essentially, his argument boils down to this: Fixing a broken window may appear to be productive, but if that were really the case, then we should all break every window we can find to help improve the economy. In reality, the money spent on fixing the window could have been spent on something else that would have improved the world before the window was broken.

Although Bastiat was talking about the allocation of resources generally across industries, I think his argument applies equally well at the personal level. We need to allocate our resources on things that are actually productive and not just on things that appear to be productive. We need to stop convincing ourselves that our ideas are inherently valuable when they are actually not. If you convince yourself that you should hold off in executing on your idea until you’ve completely thought it through, then you will never realize the potential of the idea. It’s not enough to stop being actively unproductive; we have to force ourselves to continually produce.

Paul Graham has an excellent essay on ideas for startups that also touches on the value of an idea without execution. The hardest part of founding a successful startup is not generating the idea, it is executing the idea. In other words, there’s no such thing as a million dollar idea. Google was not a million dollar idea. Facebook was not a million dollar idea. Graham’s proof of this is dead simple:

Actually, startup ideas are not million dollar ideas, and here’s an experiment you can try to prove it: just try to sell one. Nothing evolves faster than markets. The fact that there’s no market for startup ideas suggests there’s no demand. Which means, in the narrow sense of the word, that startup ideas are worthless.

In other words, Google and Facebook are examples of million dollar execution, and I believe this concept is just as important at a personal level. Executing ideas is much harder than not executing them. There are all kinds of blogs out there that are devoted exclusively to dispensing advice on how to be more productive. It is easy to feel productive by reading them. It is easy to feel like you’re working on stuff. We humans are extraordinarily good at distracting ourselves or, as Ze Frank pointed out, convincing ourselves not to act, which is probably why executing ideas is so valuable.

The Cult of Done is the only example I can find that might (maybe) take executing ideas a step too far. They take an extreme position on doing things rather than thinking of things to do. (Here’s a good analysis on the Cult of Done.) We certainly need to emphasize actual execution of ideas since most people fall so far on the side of thinking and not even close to the side of doing. Perhaps adopting the spirit of The Cult of Done wouldn’t be a bad thing. After all, Ze’s right: most ideas really do suck when you do them, and the only way to find out is through execution.

Our government requires transparency to operate as a functional participatory democracy. It’s not optional. If we do not have an informed citizenry, then we don’t have a participatory democracy. The Sunlight Foundation considers improving transparency to be a key reason for their existence. Larry Lessig wants to improve transparency in congressional funding to Change Congress. Hugo Teufel, the former Chief Privacy Officer at the Department of Homeland Security, considers transparency to be the most important principle when it comes to privacy in the war on terror.

The first step in transparency is access, so let me ask a simple question: Are U.S. Government documents copyrighted? More specifically, Are State or Federal Laws protected as copyrighted works in and of themselves? Oregon decided to use copyright law to protect their statutes from being posted online. In this case, Carl Malamud of Public Resource fought back. If your intuition is telling you that laws are in the public domain and not subject to copyright, then you are correct for the most part, but the situation is really quite a bit more complicated than that.

To investigate a bit of this complication, consider the way the government publishes the law. It’s done in a piecemeal fashion, one law at a time. Yes, there is some structure to the publishing process, but you don’t have to read much of a law to realize that there are a zillion cross references to other laws. This doesn’t even begin to include case law, which clarifies the interpretation of a legal text. Ed Felten poses the scenario thusly:

Suppose I gave you a big stack of paper containing all of the laws ever passed by Congress (and signed by the President). This wouldn’t be very useful, if what you wanted was to know whether some action you were contemplating would violate the law. How would you find the laws bearing on that action? And if you did find such a law, how would you determine whether it had been repealed or amended later, or how courts had interpreted it?

Companies like Thompson West have made billions of dollars publishing information to guide lawyers that must answer questions like this. Typically, these summaries and indices are protected under copyright law as extra-value content. Thus, for virtually all practical purposes, U.S. Government Documents are copyrighted. If you are interested in more information on this, I would highly recommend James Grimmelman’s primer on the subject.

Of course, it’s not just legal texts that are technically in the public domain, but remain plagued by access problems. Court records are also public documents. Once again, this is critical to the very structure of our government. The words “secret government trial” should send chills down the spine of any American. However, as Joe Lieberman recently argued, they are still “behind a paid firewall.” (This firewall apparently comes complete with a government website from 1999.) Carl Malamud of Public Record is once again on the case.

Side note: Carl Malamud is running a campaign to be nominated by the Obama administration as the Public Printer of the United States. He’s garnered the support of Larry Lessig, Tim O’Reilly, and the EFF. (Not to mention Canadians like Cory Doctorow and Tim Bray!) You can read more about his campaign at Yes We Scan.

Technology is changing the landscape of transparency in government documents rapidly. Many of the problems are already solved. The cost of maintaining a website “containing all of the laws ever passed by Congress (and signed by the President)” is orders of magnitude cheaper than managing all this information in print. Ed Felten’s group at Princeton has argued extensively for an open government model that would just give us the data.

Yesterday the Obama administration announced that Vivek Kundra will be the “Chief Information Officer” for the United States. If you are interested in more information on Kundra, I would strongly recommend this excellent podcast on transparency in the Obama administration from the Technology Liberation Front. They discuss their thoughts on the position, his experience as the CTO for the city government of Washington DC, and several other aspects of transparency in the Obama administration.

The future is quite interesting for the development of transparency in government documents. Personally, I think dramatic improvements in public access of U.S. government documents are inevitable, whether they come from individuals like Carl Malamud, corporations like Thompson West, or from the government itself. The only question left is when…

[Update: The Technology Liberation Front posted another podcast today about PACER and accessing online court records. It features Tim Lee, James Grimmelmann, and Steve Schultze. I highly recommend it!]

Currently, Twitter is the Internet’s dominant micro-blogging service. It has shown that micro-blogging is a distinctly different form of communication deserving of it’s own niche, and it has done so well with its own micro-blogging service that micro-blogging itself is perhaps better known as Twittering.

Of course, there is one small problem. Twitter is a closed platform. As Tim Bray put it:

The basic problem is that Twitter is centralized; that’s not how the Internet works.

A quick look at history tells us that open communication protocols win in the long run. When you call someone on the phone, you aren’t limited to people using the same telephone service provider. When you email someone, you aren’t limited to people who are using the same Internet service provider. Even actual blogging has standardized norms (RSS and Atom) that allow people using blogger, WordPress, LiveJournal, or any other blogging mechanism to easily follow blogs on other platforms. (Though, cross-blog commenting is still a bit of a problem.)

Although I could talk about the Network Effect or Metcalfe’s Law, for the purposes of this post, I will focus on the key security design problem facing Twitter. This is not to say that the Network Effect and Metcalfe’s Law aren’t important. They are. I’m just talking about another, unrelated reason that supports the need for diversity in the micro-blogging industry.

A recent incident is an exemplar of the real problems caused by a centralized protocol like Twitter. An attacker was able to hijack several high-profile Twitter feeds, including Barack Obama’s campaign feed and the official Fox News feed. How did this happen? Well, it turns out that there was a security design flaw on the Twitter site that allowed rapid login attempts. This allowed an attacker to use a dictionary attack against the Twitter account of a member of Twitter’s support staff. Once the password was guessed, the attacker was able to get access to any feed in all of Twitter-dom.

The key security flaw in any centralized protocol is that such protocols are monocultures. Bananas are a great example of the danger of monocultures. Bananas are an extremely important crop worldwide, but the vast majority of bananas grown are of the Cavendish variety. Why? Because the tastier Gros Michel bananas were wiped out by a disease. They were all essentially genetically identical. There was almost no diversity in the banana ecosystem. As a result, they were unable to adapt to the disease, and since the same problem exists with the Cavendish, we’re still one bad disease away from a worldwide shortage of bananas.

The same problem exists for micro-blogging. If you want to micro-blog, you effectively need a Twitter account. Twitter is so dominant that almost all micro-bloggers are using Twitter, which makes it a monoculture. Because Twitter is a monoculture for micro-blogging, the micro-blogging itself is one bad security incident away from obliteration. Also, if Twitter were to go belly up (which is not, as Tim Bray discussed, outside the realm of possibility for an Internet-based company), then, effectively, the entire micro-blogging industry would be eliminated.

At this point you might say, “Wait! Twitter has an open API!” This is not the same as open source, and it does not eliminate the threats posed by monocultures. It does mean that it is very easy to add functionality to the Twitter protocol, but it does not mean that you can participate freely without a Twitter account.

Micro-blogging needs a viable open source alternative to create a federated micro-blogging protocol. Tim Bray proffered Laconica and one of the commenters in his thread mentioned the soon-to-be open source Jaiku, which was recently shutdown by Google.

Whatever happens, a federated micro-blogging protocol would be far more robust than the current Twitter monoculture. If I were to add a single gutsy prediction to the list over at Freedom to Tinker, it would be that a major security incident at Twitter allows an open source alternative to gain a foothold in micro-blogging. It may not happen this year, but I think it’s inevitable with any monoculture.

Last month, Jim Harper started an interesting discussion on regulation around the holidays. Although I was hiding from my computer in an attempt to take a vacation at the time I have finally caught up with the discussion. If you happen to have missed this as well, I think it’s well worth reading.

Jim started the discussion with an excellent blog post over on Cato@Liberty about ‘real’ regulation. If you haven’t read it, here’s a juicy tidbit:

What Burnett meant when she called for a “real” regulator, of course, was “the regulator I can imagine.” The regulators people imagine are foresighted, interested only in the public good, they’re resistant to lobbying, and they run efficient organizations. But these characteristics are simply imaginary.

Tim Lee followed this up with a post on the Technology Liberation Front extending the discussion to regulation of technology. Again, if you haven’t read it, here’s a juicy tidbit:

Too many advocates of regulation seem to have never considered the possibility that the FCC bureaucrats in charge of making these decisions at any point in time might be lazy, incompetent, technically confused, or biased in favor of industry incumbents. That’s often what “real regulators” are like, and it’s important that when policy makers are crafting regulatory scheme, they assume that some of the people administering the law will have these kinds of flaws, rather than imagining that the rules they right will be applied by infallible philosopher-kings.

The FCC is designed to ensure (theoretically) that these bureaucrats are independent, but if you take a look at what is actually happening, then it becomes clear that independence is in many ways imaginary. The FCC’s website describes the organization of the Commissioners as follows:

The FCC is directed by five Commissioners appointed by the President and confirmed by the Senate for 5-year terms, except when filling an unexpired term. The President designates one of the Commissioners to serve as Chairperson. Only three Commissioners may be members of the same political party. None of them can have a financial interest in any Commission-related business.

News today that FCC Chairman Kevin Martin will resign on Inauguration Day makes the FCC an even more interesting topic for discussion. With this resignation, Obama will have nominated all five of the serving FCC Commissioners by this summer. Also, it goes without saying that the FCC will be regulating some important aspects of our society, including the Digital TV transition that’s slated for February.

Obama’s choice for FCC Chairman is Julius Genachowski. Wikipedia describes him as “an American business executive with experience in telecommunication and technology issues.” This ideal of not having a financial interest in Commission-related business isn’t starting off well. Of course, that’s Wikipedia, so maybe it’s not trustworthy. Let’s look at Reuters’ description:

Genachowski was chief counsel for Reed Hundt, an FCC chairman under former President Bill Clinton. He also held various positions at Internet search and media company IAC/InterActiveCorp (IACI.O) and several firms investing in technology, including Rock Creek Ventures and LaunchBox Digital.

Again, it sorta feels like this is a man with financial interests in technology. Business Week even lauds his “business sense” as a key benefit that he brings to the table. But let’s take a step back from this individual appointment. I really don’t know much about Genachowski other than what’s been reported in the news, and I certainly don’t want to pick on him as an individual that’s emblematic of the larger problem with “real” regulation.

My point is simply to consider this: Is it really possible to find anyone who has the knowledge needed to help run a regulatory organization like the FCC that doesn’t have a financial interest in Commission-related business? If it isn’t outright provably impossible, then at the very least I think there’s a strong argument to be made that it is impossible. Sure we may wish that it weren’t so, but if wishes were fishes, we’d all be casting our nets. Of course, I’m open to your thoughts in the comments.

Malcom Gladwell’s recent New Yorker article compares “the quarterback problem” to the challenge of finding a good teacher. It’s an interesting article, but it is, perhaps, too narrow in its focus.

For those who don’t know, the quarterback problem is defined as the extremely difficult task of selecting a quarterback to play in the NFL from the pool of college football quarterbacks. It’s deceptively challenging to do this because there’s so much data available and so many ways to rank college football quarterbacks. However, the college football game is so different from the NFL game that success at the college level seems to have very little correlation, or perhaps no correlation, to success at the professional level.

One of the best examples of this is the comparison of Peyton Manning and Ryan Leaf, who were both extremely successful college quarterbacks drafted first and second in the 1998 NFL draft. Most experts thought it was a toss-up as to which of these two would have a better career. Of course, it’s clear to everyone now that Peyton Manning is a lock for the Hall of Fame while Ryan Leaf is famous for being a complete bust in the NFL.

The key element of the quarterback problem is that past results simply aren’t useful in predicting future success. Gladwell argues that this is also true of selecting good teachers. He claims that the usual metrics used to measure hiring and promotions for teachers, such as master’s degrees, teaching certifications, and other cognitive standards, are just as useless in attempting to determine a good teacher as college football statistics are in trying to determine a successful NFL quarterback.

Another important element of the quarterback problem is that the difference between ‘good’ and ‘bad’ is extremely large, potentially several orders of magnitude. There are very few ‘good’ quarterbacks in the NFL. There aren’t even very many quarterbacks decent enough to serve as an emergency backup. The quarterback position in the NFL may be the single toughest position to play in all of professional sports. Gladwell argues that the same is true of teachers:

Suppose that Mrs. Brown and Mr. Smith both teach a classroom of third graders who score at the fiftieth percentile on math and reading tests on the first day of school, in September. When the students are retested, in June, Mrs. Brown’s class scores at the seventieth percentile, while Mr. Smith’s students have fallen to the fortieth percentile. That change in the students’ rankings, value-added theory says, is a meaningful indicator of how much more effective Mrs. Brown is as a teacher than Mr. Smith.

It’s only a crude measure, of course. A teacher is not solely responsible for how much is learned in a classroom, and not everything of value that a teacher imparts to his or her students can be captured on a standardized test. Nonetheless, if you follow Brown and Smith for three or four years, their effect on their students’ test scores starts to become predictable: with enough data, it is possible to identify who the very good teachers are and who the very poor teachers are. What’s more—and this is the finding that has galvanized the educational world—the difference between good teachers and poor teachers turns out to be vast.

It follows that if you want a school system filled with good teachers, then you have to be willing to identify the poor teachers and get rid of them. This is the only solution to the quarterback problem. It’s a brutal process for both the teachers and the administration. Time Magazine recently had a cover story on Michelle Rhee’s unusual approach to improving schools in the nation’s capital, which is attempting to implement this brutal process.

Rhee wants to solve the quarterback problem the only way possible: by mitigating its effects. Simply put, if you can’t identify good teachers without seeing how they perform in the classroom, then you have to hire a bunch of teachers, watch their classroom performance, identify those that are succeeding, and reward them. Similarly, you have to identify teachers that are failing and eliminate them. Time’s article does a good job explaining why doing these two things is extraordinarily complicated in the teaching industry.

Of course, this is exactly how they solve the quarterback problem in the NFL. On-field performance is everything. Many of the best quarterbacks were identified as such by their play in real NFL games as backups for injured quarterbacks. Matt Cassel is a great example. In college, he never started a game and served as a backup for Carson Palmer and Matt Leinart. He was drafted into the NFL and played as a backup for Tom Brady, who suffered a season-ending injury in the first game of the 2008 season. Brady’s injury made Matt Cassel a starting quarterback for the first time since high school, which would undoubtedly determine his future in the NFL. If he played well, he would likely be rewarded with a starting role for another team during the off season. If he played poorly, he would fall into the nameless abyss of all the other failed NFL quarterbacks.

After reading Gladwell’s article, I had to wonder, how many other professions are like that? Surely the quarterback problem isn’t just limited to teachers and NFL quarterbacks. The first thing that came to my mind was a Paul Graham essay about great programmers, which is really a must-read for anyone in the software industry. In it, Graham talks about the nature of great programmers, and summarizes the problem of identifying them by saying, “The problem is, if you’re not a hacker, you can’t tell who the good hackers are.” Fred Brooks also talks about the vast difference between a great programmer and an average programmer in The Mytical Man-Month. Here’s Fred Brooks on great software designers:

The differences are not minor – it is rather like Salieri and Mozart. Study after study shows that the very best designers produce structures that are faster, smaller, simpler, cleaner, and produced with less effort. The differences between the great and the average approach an order of magnitude.

Clearly, selecting a software engineers fits the definition of the quarterback problem. It would be very interesting to study how the extreme challenge of creating a start-up company performs as a system for identifying great programmers.

Apparently, some people believe there’s a quarterback problem in selecting good lawyers. Although, I have no particular experience with this, I think the environment in which lawyers at big law firms operate is strikingly similar to the ideal solution to the quarterback problem. There’s an incredibly small percentages of lawyers who end up making partner at a big law firm, which indicates to me that there’s a quarterback problem in trying to hire a big law firm partner.

I’ve also seen the suggestion that selecting a mate is a version of the quarterback problem, but I personally think that’s taking things too far. I don’t think that people really have an objective idea of what a good mate is, let alone what metrics to use in measuring potential mates. Furthermore, the role of being a “mate” really isn’t the same thing as having a job.

I believe the quarterback problem is potentially much more prevalent than people currently recognize. I also think that the solution to the quarterback problem is clearly defined. The two important lessons to learn and apply from the quarterback problem:

• Don’t be afraid to give people a chance. They might surprise you.
• Don’t be afraid to make a change when things aren’t working out.

These two steps are the best known solution to the quarterback problem. What other fields could benefit from implementing them? If you have any suggestions for other areas where this problem seems to occur, please mention them in the comments.

As a technologist with a strong interest in computer security, privacy, and public policy, I am naturally drawn to the topic of electronic voting. I have written about electronic voting several times before, including this piece on Ed Felten’s work. Recently, I have seen lists of things things could have gone wrong and some lists of things that actually did go wrong. I have even seen a hilarious account of the worst case scenario, but the most interesting accounts that I’ve seen have been personal accounts of computer science professors who volunteered to operate the polls as election workers.

Avi Rubin, a Professor of Computer Science at Johns Hopkins and director of the ACCURATE Voting center, wrote a post describing his experience working the polls and posted it only minutes before most news outlets announced that Barack Obama will be the 44th President of the United States. Professor Rubin is the author of the book Brave New Ballot, an excellent book on the dangers of electronic voting machines that I have reviewed here. His experience at the polls in Maryland describes the very practical and non-technical aspects of just what a poll worker does during the day.

Steven Bellovin, a Professor of Computer Science at Columbia, also wrote about his experience as an election official. Professor Bellovin is another well-respected authority on computer security whose post focuses on the non-technical details of the responsibilities of poll workers in New Jersey. Andrew Appel, a Professor of Computer Science at Princeton, also wrote about the use of voting machines in New Jersey.

Both New Jersey and Maryland used Direct-record electronic voting machines, which have a myriad of security concerns that have been detailed extensively elsewhere. Essentially, DREs store the official record of an election in an electronic form rather than a paper form. If you are interested in some of the problems with DREs and proposed solutions to those problems, then you should check out the USACM’s page on electronic voting.

You may be asking yourself: Why would a computer science professor volunteer to work a poll as an election official? It’s not like there’s anything technical going on there. Well, any computer security expert will tell you that the first line of defense must be physical access. This means that you can have all technology you want, all the cryptography you want, and spend all the money you have and still not be secure without common sense. There was a great video on No-Tech Hacking at DefCon in 2007 which covers what I’m talking about.

Physical access is one of the key problems with DREs: thousands of people must have physical access to the machines themselves to cast their vote. The environment is filled with opportunities for absolutely simple no-tech hacking. Even if these systems weren’t notoriously bad in terms of the technology used, the physical access alone makes these devices difficult to secure.

The challenges of physical access and the stakes of a Presidential election are both great reasons that computer science professors are interested. It’s a unique opportunity to see how these machines are actually used, and some of their observations are excellent. Their posts are worth reading if you’re interested in electronic voting or computer security: Avi Rubin’s post; Steven Bellovin’s post.