May 31, 2003

Scientific suffering

The major change in endurance training over the past two decades has been the wide availability of technology allowing you to finely calibrate how much you're suffering. For years, everyone used more or less the same tools: stopwatches and bicycle computers for measuring speed. The problem with these tools is that unless you are on a totally flat surface, speed doesn't correlate very well with how hard you're actually working. Since most endurance training is directed towards training your cardiovascular system, that means you don't necessarily get the optimal training load.

Heart Rate Monitors
The first piece of consumer performance technology to appear was the heart rate monitor (HRM). A HRM typically consists of a sensor built into a chest strap. It communicates via radio with a receiver built into a watch. The watch displays your current heart rate. With an HRM, you can target specific heart rates instead of specific speeds. Thus, no matter what the course you're working out on, you can get the exact desired training effort.

What is this good for? Three things. The first use is obvious: It's really easy to fool yourself into thinking you're working hard when you're really not. The HRM tells the true story. Thus, as you get better over time the HRM automatically makes you go faster to stay at a specific effort level. Second, the HRM keeps you in check. Your long distance days should be really really easy, but people feel guilty and often go fast. If you never exceed your target easy HR, you know you're going easy enough.

The final use for an HRM is for tracking day to day training status. When you're sick or overtrained your resting heart rate goes up. So, if you take your HR every morning and you notice an elevation, it's probably time to take it easy for a couple days.

CompuTrainer
The CompuTrainer is a computerized bicycle trainer. Essentially, it's a stand you put your bicycle on and it provides a programmable load at the back wheel. Bicycle trainers have been around for a long time, but they used to just generate resistance mechanically, which made them hard to control. The CompuTrainer brings two things to the party:

  • The load is generated electronically so it's tightly controllable in resolutions of 10 watts.
  • There is a computer interface allowing you to integrate with specialized training software.

The CompuTrainer is perfect for doing interval training on, because it allows you to set your load precisely. In combination with the HRM, you can hit exactly the desired training load. Best of all, you can use your own bike, so the ergonomics are the same as they would be in a real workout.

Power meters
The big problem with the CompuTrainer is that it's fixed in place. This makes it really boring and you don't get to train in different terrain. It's also fiendishly hot in the summer since there's no airflow unless you have a very good fan blowing directly on you. In the past 5 years or so, you've been able to buy power meters that you can attach directly to your bicycle. This lets you measure your power output in field conditions. Power meters are big with the pros and lately have been moving into the amateur ranks.

So what?
So, what's all this technology buying you? At least theoretically, it's helping you improve faster. There's some modest amount of research showing that training at the right effort levels leads to faster improvement. More importantly, it's making it easier to train yourself. If you don't have a coach--most people don't--it can be very hard to select your own training loads based on time because people's strengths and weaknesses are very individual. However, the appropriate effort levels are much more standardized. You can take a few relatively simple measurements and know exactly what heart rates you need to work at. So, amateurs can get better training and flail less.

Really, though, what it's buying you is a lot of pain. When you train alone, it's easy to tell yourself that you're working hard enough. The technology keeps you honest, which means you suffer more. But in a good way, or at least that's what we tell ourselves.

Posted by ekr at 10:24 AM | Comments (12) | TrackBack

May 30, 2003

What Matrix Reloaded should have been

This really is a lot better than Matrix Reloaded. Between this and Troops I'm starting to wonder if the world wouldn't be better if directors stopped making sequels and just let the fans do it.
Posted by ekr at 10:17 AM | Comments (5) | TrackBack

NIH isn't a compliment

Nullsoft (the people behind Winamp and Gnutella) have designed a new package called WASTE (something seems to be wrong with this URL. Try the mirror). for encrypted communications within small groups. For some unknown reason they have decided to invent their own security protocol rather than using the standard tool for this job--SSL. Here's what they say:
Note: It might be worth implementing WASTE using a subset of SSL, to avoid any concern of flaws in this protocol. Feedback is gladly accepted on any potential weaknesses of the negotiation. We have spent a decent amount of time analyzing this, and although we have found a few things that are not ideal (i.e. if you know public keys from a network, you can sniff some traffic and do an offline dictionary attack on the network name/ID), but overall it seems decent. The current implementation probably needs work, too.

Huh? SSL would have done the job, but we decide not to use it, for no particular reason.

Worse yet, they don't have a protocol spec, just annotated source code and a sort of overview doc. So far, I've been too lazy to read the code, but what's in their description doesn't look really promising.

WASTE secures the links of the WASTE network by using RSA to exchange session keys and authenticate the other end of the connection. Once the hosts have authenticated each other and both have the correct session keys, the connection is encrypted using Blowfish in PCBC mode (using different IVs for each direction of the connection). The oversimplified process for bringing a link up is (see comments in the code and the code itself for a more in depth view):
  • Both sides exchange public key hashes, and verify that they know that hash
  • Both sides exchange session keys and challenge-response tokens encrypted with each others public keys.
  • Both sides decrypt and verify the challenge-response tokens, and begin encrypted communication (a stream of messages, each message is verified using an MD5).

There's a lot more to it than that, but that's the basic idea. The reality of it is that there is also a "Network ID/Name" feature that allows you to easily keep networks from colliding, as well as efforts to obfuscate the whole process (to make WASTE connections difficult to detect). Another unique feature is the way session keys are exchanged and combined so that in order to decrypt past (recorded) traffic, both private keys of a connection need to be recovered.

This property may be unique, but it's not really an improvement over SSL. SSL includes modes which offer Perfect Forward Secrecy, in which even knowing both private keys isn't sufficient to recover recorded traffic.

So far, I don't see any way in which WASTE is better than SSL, and some design choices that look questionable. Why do people insist on designing their own thing when the standard tools will do the job perfectly well? This is even a worse idea than usual when designing communications security protocols, which are hard to get right and often have subtle bugs. The fantasy that you have to have all your own stuff is called NIH (Not Invented Here) and it's not a good thing.

Extra for security guys:
Why do people who design their own protocols always seem to use Blowfish? There's a reason we have AES. For bonus points, the geniuses at Nullsoft use PCBC mode. That's not exactly a sure sign of massive crypto expertise.

Posted by ekr at 08:22 AM | Comments (10) | TrackBack

And on the other hand...

Matthew Yglesias complains that journalists often present subjects where there actually is a right answer as just a "he said, she said" story:
Quite so. I'll be the first to conclude that the facts of the matter regarding economics are often non-obvious. If there's something unclear about the facts, however, then that is what a reporter ought to report, just as when it was unclear whether or not Saddam Hussein had been killed in the "decapitation" strike reporters reported that it was unclear whether or not Saddam had been killed. The whole purpose of reporters, after all, is to try and figure out what's going on and then convey that information to the reading public. When you just quote two people at random the reader still has to do all the work himself if he wants to know what the truth is.

In Matthew's comments section, Ikraem Saaed gets pretty close to the what I suspect the real answer is:

In the field of economics, as in many other fiedls, they are clueless. Many are innumerate. They attempt to turn factual matters into horseraces. Economic issues should be covered like science issues are, but instead they are covered like political issues are.

I think this is definitely part of the equation. Lots of people (myself included) have had the experience of reading media coverage in their field and seeing the writer totally botch it, even when the subject is totally non-controversial. So, it's pretty clear that there are lots of situations where the media really doesn't understand the field.

However, I think that calling reporters stupid isn't necessarily fair. I think of it more like boundedly rational. In most social situations, the best way of discovering objective truth--even when their objective truth to be had--isn't by taking measurements and doing statistics but rather by asking around and trying to figure out who's telling the truth. The techniques of science are very powerful but also heavyweight to employ and have a limited scope. They also take a long time to get into the habit of using. So, it's not surprising that reporters try to employ social techniques to figure out what's going on, even when they're inappropriate.

To make things worse, figuring out what objective truth is to be rather a messy business. Take something like evolution, which I consider to be about as much of a scientific fact as we have. Nevertheless, you can definitely find someone who will make scientific-sounding explanations as to why it's not true. If you give me two months and are willing to learn a lot of physics, biology, and chemistry, I can show you that all of those objections are false, but reporters don't have that kind of time. How then, are reporters supposed to distinguish such issues from ones where there is genuine controversy? Easier not even to try.

Posted by ekr at 07:55 AM | Comments (41) | TrackBack

May 29, 2003

New paradigm, huh?

A company called Titan Key has just started talking about their anti-spam technology, which they say is a "new paradigm" and "superior to EVERY other anti-spam product". This sort of hype reeks of snake oil and close inspection seems to bear out that impression.

Titan Key's "new paradigm" technology is an amalgam of a number of previous anti-spam ideas:

  • A filtering proxy.
  • Challenge response.
  • Bounces.
  • Custom-built addresses.

The Titan Key challenge response system differs from most other challenge response systems in that it's intended to be implemented in the mail server instead of by the user. If a message arrives which was not sent by a known sender, the system generates an SMTP bounce and then initiates the challenge response handshake. The reason that this is supposedly cool is that the spammer processes the bounce and then takes you off their list. In practice, however, that's not the case--spammers generally ignore these messages.

The downside of this strategy, of course, is that instead of just getting a challenge, anyone who tries to send mail to you for the first time gets both a challenge anda a rather disconcerting bounce.

The other thing that the Titan Key guys do is generate custom addresses so you can hand them out for mailing lists. They also use sender filtering to supposedly protect you from those custom addresses being leaked to spammers. Unfortunately, spammers often forge their "From" addresses, in which case they might be able to use custom addresses, making this trick not very attractive.

The bottom line here is that the "new paradigm" looks a lot like the old paradigm. I don't expect to buy a Titan Key any time soon.

Posted by ekr at 04:40 PM | Comments (63) | TrackBack

The beginning of the end for Mozilla?

News.com is reporting that AOL is going to license Internet Explorer. As part of the agreement, Microsoft will pay AOL $750 million (pocket money for MS) to settle their antitrust case and AOL will get a 7 year royalty-free license for IE.

Recall that AOL bought Netscape a while back and has been continuing to maintain the open source browser Mozilla as well as ship versions of Netscape Communicator based on Mozilla. If AOL now has an IE license it's hard to understand why they would want to continue to pay to develop Mozilla. As an outsider, it never seemed to me that Mozilla became self-sustaining the way that say Linux was, so I'm not sure how long it can survive under these circumstances.

Posted by ekr at 02:35 PM | Comments (12) | TrackBack

May 28, 2003

Knowing your place

J. of the Silver Rights blog takes me to task for my post about Annika Sorenstam:
I think it's reasonable to say that you're outclassed when it's really clear that you are and that you have no chance of making up the difference. Now, it's no doubt easier to say that if you just won, but it's certainly arguable that that makes you more of a wimp, not less, since you've proved that you can compete, at least at some level, if not the highest possible level. That's not my view, however. It's critical to know your limitations.

But, would he apply the same reasoning to a man? It seems to me that women are too often told to know their limitations and what those limitations are by men who want to limit competition -- for education, for jobs and for the PGA tour.

Not only would I apply the same reasoning to a man, I did apply the same reasoning to a man a few paragraphs before, where I said:

It seems to me that a higher weight class is more or less analogous to playing in the men's tour instead of the women. Bigger guys are just stronger and will pretty much always beat smaller guys of equivalent skill level. Actually, it understates the difference, since bigger guys are typically slower than smaller guys and men are not slower than women.

I've got absolutely no problem with women competing against men in any environment. In fact, I've had my ass handed to me by women in any number of triathlons. However, it's also fairly clear that in pretty much any athletic endeavor, the top men have a more or less insurmountable advantage over the top women. There's nothing shameful about women admitting that and demanding that they have a protected (i.e. female-only) environment in which to compete. If women would prefer to have no gender divisions in sports--with the likely result that women almost never win anything--I'm fine with that too.

Posted by ekr at 09:40 PM | Comments (37) | TrackBack

More threatening noises from SCO

This Marketwatch story on the SCO/Linux kerfuffle has an interesting bit at the end:
McBride added that unless more companies start licensing SCO's property, he may also sue Linus Torvalds, who is credited with inventing the Linux operating system, for patent infringement.

Huh? Remember, folks, Unix was invented in 1969, BSD 4.3 dates from 1986, and System V dates from 1987. Pretty much all those patents are expired now or will be in a year or two. Of course, they could punish Linux for his past infringement, but what would that buy them? It would be interesting to know what patents they claim he's infringing.

Posted by ekr at 09:08 PM | Comments (13) | TrackBack

How much should you pay for software?

Last night I discussed my first area of disagreement with Dave Winer's postings on software economics. This post discusses my second and third areas of disagreement.

First, Dave's fundamental premise--that people aren't willing to pay for software--seems to me to be basically wrong. Dave writes:

When I say there's no money for software, that's not a literal statement, of course. Sure there is some money. When you buy a new computer you probably pay a hundred dollars for software, most of it going to Microsoft. So they've figured out how to get money to flow.

This understates the case a bit. The total revenues of the US software industry in 2001 (the last year for which we have data) were $91 billion. That's a lot of money, more than the size of the movie and sound recording industries combined and more than 50% more than the size of the professional art and sports industries. So, people certainly seem pretty willing to pay for software to me.

Now, it's possible that Dave just thinks that people aren't willing to pay enough for software. However, making that evaluation requires deciding what "enough" means, which is always a tricky proposition. Based on this article, it seems to me that Dave is making several arguments that we're not willing to pay enough. I'll take them in turn.

The labor theory of value
Dave's first argument is basically the labor theory of value:

A professional software organization for a well-supported product has 10-20 people, maybe as many as 30 to 40. So when you hear yourself complaining about software quality, think about how much money the developer of the product has to fully support it. Could you run a car in the Indy 500 with no money? You could try, and that's what a lot of software developers do, to no avail. Sooner or later you have to pay the bills. It costs money to live. That's as true of software as it is of people.

The refutation of the labor theory of value is pretty much the same now as it was when it was first proposed. So what if you've spent thousands of dollars digging a hole in the ground? It's still just a hole. The value of something is determined by what people are willing to pay for it, not by what it costs to produce. That doesn't mean that the cost is irrelevant, of course, since it represents the lowest cost at which a commodity can be sold (ignore the difference between fixed cost and marginal cost for the moment). If the cost of producing something is more than people value it at, then it shouldn't be produced. That's the market at work.

Reverse labor theory of value
Dave's second argument seems to be that the amount of time using something should control how much you pay:

Let's say you spend 100 hours a year using a piece of software and assume your time is worth $50 per hour. So that's $5000 of your time flowing through the software. How much self-respect is there in paying nothing for software that leverages so much of your time?

Maybe I'm missing something important, but I don't understand this argument at all. What does the amount of my time I spend using a product have to do with how much I ought to value that product? I've probably spent hundreds of hours wearing the free t-shirt I got from Wired magazine, but I don't feel at all guilty for not paying for it. By contrast, I've got friends who go to restaurants and pay $50 for an after-dinner glass of Port, which takes them 5 minutes to drink.

The relevant question when deciding how much something is worth to you is how much you would be willing to pay to do without it. Since I have hundreds of t-shirts, and get a couple of free t-shirts every time I attend a conference, the marginal value of the Wired t-shirt is pretty much zero. Similarly, the relevant question when I value a software product isn't how much I use it but rather how much better having it has made my situation. So, if this hypothetical piece of software had saved me hundreds of hours, then we'd be talking some significant value. As it is, we have no idea.

You'll be sorry later
Finally, Dave argues that if we don't pay for software we'll get bad software and therefore we ought to:

If you don't pay, the bottom-line is that you lose. It may look like you're not losing, but you are. If you paid nothing for health care, you'd likely die sooner. If you pay nothing for software, you probably won't die from it, but you may lose data, you're virtually certain to waste time, and at some point, money.

This, at least, is a recognizable modern-looking economic argument, however, I don't think it's a correct one. There are four possibilities here:

  1. People are correctly estimating the value of software and it's less than the price. Thus, they're choosing not to buy it. Obviously, this is distressing if your business is software and it's your software that's not being bought, but otherwise is nothing to be alarmed about
  2. People are incorrectly estimating the value of software and need Dave to tell them. This is certainly possible, but ordinarily in economics we assume that people know their own preferences.
  3. Software is somehow a public good and so people are (rationally) not paying their fair share. This is, of course, possible, but I don't think particularly likely. There certainly are software problems that are public goods (or rather, public bads) such as vulnerability to worms, but I suspect that the majority of problems people experience affect them alone.
  4. Customers can't tell whether or not software is good and so they're not willing to pay for it. This is what's known as a "lemons market".

Only options (1) and (4) are particularly plausible. In the case of option (1), the market is working correctly, so there's nothing to worry about. The possibility that software is a lemons market, however, is more interesting.

The market for bad software
The term "lemons market" comes from the George Akerlof's papepr "The Market for Lemons" that introduced the concept in the context of cars. Any given software manufacturer has a choice of making good software or bad software. Customers are willing to pay more for good software than bad. However, bad software is cheaper to produce. If customers have no way of telling whether a given piece of software is good or bad, then manufacturers have an incentive to sell only bad software. Even if one manufacturer is honest, some dishonest manufacturer will undercut him and drive him out of business. Thus, everyone makes bad software. Even though the customer would be willing to pay for good software, and the manufacturer would be willing to produce it, the market has no way to provide it.

The standard solution to this problem, of course, is to have some third party review the product and vouch for it's quality. Unfortunately, it turns out to be very hard to really do a solid job of this on software because a lot of the problems that people encounter are unpredictable interactions with other parts of their environment, and it's precisely such effects that are hard to test for.

The Bottom Line
Dave's basic argument seems to be that if customers just realized what the true value of software is, they would pony up and all would be well. That argument just doesn't stand up under analysis. It's simply not at all clear that the value of software is in fact high enough to justify a price higher than we in fact pay now. That may be bad news for software manufacturers, who would like to extract more money from consumers, but it's not at all clear it's bad for consumers. Even if we in fact do have a lemons market (which, based on my experience seems quite likely), the problem isn't that consumers are cheap but rather that they insist on getting value for their money, a position I consider an eminently reasonable one.

Posted by ekr at 03:15 PM | Comments (33) | TrackBack

Wait, who owns Unix?

Oh, this is fun. Now Novell claims that they retained the intellectual property rights to Unix and merely licensed them to SCO. Moreover, they claim that SCO knows this:
"To Novell's knowledge, the 1995 agreement governing SCO's purchase of Unix from Novell does not convey to SCO the associated copyrights," Novell Chief Executive Jack Messman said in the letter to SCO Chief Executive Darl McBride. He said that SCO evidently realizes this because "over the last few months you have repeatedly asked Novell to transfer the copyrights to SCO, requests that Novell has rejected."

Moreover, Novell is making noises about suing SCO.

SCO's response is here.

CO owns the contract rights to the UNIX® operating system. SCO has the contractual right to prevent improper donations of UNIX code, methods or concepts into Linux by any UNIX vendor.

Copyrights and patents are protection against strangers. Contracts are what you use against parties you have relationships with. From a legal standpoint, contracts end up being far stronger than anything you could do with copyrights.

SCO's lawsuit against IBM does not involve patents or copyrights. SCO's complaint specifically alleges breach of contract, and SCO intends to protect and enforce all of the contracts that the company has with more than 6,000 licensees.

We formed SCOsource in January 2003 to enforce our UNIX rights and we intend to aggressively continue in this successful path of operation.

This is now pretty messy and I'm not lawyer enough to work out who's right. From a game theoretic perspective, it's hard to believe that SCO doesn't have some leg to stand on here, since surely they knew that this information would come out eventually and likely before IBM settled.

Posted by ekr at 09:27 AM | Comments (47) | TrackBack

May 27, 2003

Computer programming goes commodity

Dave Winer has a couple of posts complaining about people not being willing to pay for software. I disagree with these posts in a number of ways, but I'll just tackle one for now. Dave writes:

In the NY Times on Thursday, a stirring op-ed piece by Ellen Ullman, about what we've lost in software. In the 90s it was common for two or three generations of software developers to work in the same organization. There was a handing-down of ideas, practices, tradition -- the verbal history of how things came to be as they are, Ullman says. After the dotcom bust software is becoming a detail, again, something that workmen do, not artists.

This strikes me as a perfectly natural turn of events, not something to be bemoaned. Software has simply become commoditized. This is what happens to all industries as they mature. In the beginning, only wizards can make anything work, but over the years the technology gets standardized and eventually anyone can do it. That's why the United States needed the smartest people from (at least) 6 countries to build the first atomic bomb and six months ago people were worried about Saddam Hussein building one in his basement.

Consider the following developments:

  • The first FORTRAN compiler, built in the 1950s, took 3 years to design and implement. Today, CS undergrads write compilers as class projects.
  • Ken Thompson won the Turing award for designing UNIX in 1969. Today, lots of people use an operating system designed by a Finnish college student.
  • Fifteen years ago, getting an Internet connection was a nightmare of configuration and organization. Today, AOL will send you CDs giving you hundreds of hours of Internet service for free.

Now, don't get me wrong, it's a lot more fun to work in an environment where your skills are valued and you can't be replaced by some guy in Bangalore or Hyderabad, but that's life. I'm sure that the people who build cars for General Motors don't much like having their jobs outsourced to Mexico, either. It's silly to act as if it's somehow some great and unique tragedy that the computer hacker culture is turning into an industry like any other. There will always be room for smart technicians in some industry. At the moment, bioinformatics and nanotech are looking like pretty good bets.

What's especially weird about Dave's nostalgia for the supposed golden days of software development is that, at least for the 10+ years I've been in the field, simple programming has always been viewed as an essentially mechanical activity, mostly suitable for junior personnel. This is exactly why so many people aspired to be architects--a title which as far as I could tell was intended to signify that the holder did more thinking than programming. Rather than complaining, we should rejoice that we've managed to farm off the mundane tasks to people who will do them cheaply, leaving us to do the high-level big picture thinking while avoiding the messy details.

Posted by ekr at 09:02 PM | Comments (10) | TrackBack

Another run at the college student filesharing wall?

Looks like the RIAA and MPAA are taking another run at the wall of stopping college students from sharing files. A friend at Stanford just received the following letter and reports that one of his friends from Davis received a similar letter. I'll post that if I get it. If other EG readers have received letters of this type, please forward them to me and I'll put them up (with your name removed, if you wish).

Update 20030527 12:34:
I have just received a copy of the Davis letter, dated a week ago.

Posted by ekr at 08:25 AM | Comments (30) | TrackBack

The ephedra witch hunt

So, Illinois is the first state to ban ephedra. Ephedra is an herbal "supplement" containing ephedrine, which is a mild stimulant (stronger than caffeine but much weaker than amphetamines).

From the press coverage, you might get the idea that people are dying left and right from ephedra, but the truth is a little different. There have been a fair number of adverse reactions reported, but it's not at all clear how many of them are actually related to ephedra/ephedrine and how many were just coincidence is very hard to determine. After all, healthy appearing people do occasionally die without warning, it's just unlikely. On the other hand, the numbers we're talking about are very small. Here's the relevant section from the summary of the RAND report commissioned by the FDA:

Evidence from controlled trials was sufficient to conclude that the use of ephedrine and/or the use of ephedra-containing dietary supplements or ephedrine plus caffeine is associated with two to three times the risk of nausea, vomiting, psychiatric symptoms such as anxiety and change in mood, autonomic hyperactivity, and palpitations.

The majority of case reports are insufficiently documented to make an informed judgment about a relationship between the use of ephedrine or ephedra-containing dietary supplements and the adverse event in question. For prior consumption of ephedra-containing products, we identified two deaths, three myocardial infarctions, nine cerebrovascular accidents, three seizures, and five psychiatric cases as sentinel events; for prior consumption of ephedrine, we identified three deaths, two myocardial infarctions, two cerebrovascular accidents, one seizure, and three psychiatric cases as sentinel events. We identified 43 additional cases as possible sentinel events with prior ephedra consumption and seven additional cases as possible sentinel events for prior ephedrine consumption. About half the sentinel events occurred in persons aged 30 years or younger. Classification as a sentinel event does not imply a proven cause and effect relationship.

In other words, the total number of people who we might reasonably suspect to have died as a result of ephedra is less than 50. And that includes the "possibles". There is some evidence that the lack of standardization of ephedra products is a source of risk because it makes it hard to know how much you're taking. But of course, that could be easily fixed by requiring standardization, not a total ban.

So, why all the fuss? Attribute it to the usual human reaction to find a scapegoat. In Illinois, the law was passed after a 16-year old football player died. Sure, he was taking ephedra, but people do just occasionally die and there's not always someone to blame. That's life.

Posted by ekr at 07:50 AM | Comments (116) | TrackBack

May 26, 2003

Outlook viruses, etc.

Steve over at PM Style argues that the market doesn't want Microsoft to do what would be necessary to stop e-mail viruses:
So a friend of mine and I were talking, and he says to me, "Isn't it ironic that Microsoft wants to stop spam and they can't. However, Microsoft has the power to stop Outlook viruses and they won't. What's up with that?" My only answer -- the market doesn't want to stop Outlook mail viruses. Why would I make such a claim? Because Microsoft and experts in the field have known how to stop Outlook email viruses and worms for a long time and Microsoft hasn't done it. Even though their customers lose a lot of money fighting malware outbreaks, Microsoft references market pressures for failing to change the default behavior of Outlook such that the software would prevent the propagation of viruses and worms.

So who is really to blame for Outlook mail viruses? You! You now know the truth and what are you going to do about it? Probably nothing. You'll keep buying Windows and Office. You'll keep using the existing software on your PC. You'll keep supporting the Microsoft products which don't take care of the problems you think are important. Or you would rather keep HTML mail and executable attachments than protect yourself from Outlook malware. Whatever your reasons, just realize that every day you make a choice to fail to prevent Outlook malware. So don't whine about it.

One might be led to think from this description that the problem was a simple feature/security issue. That's not so. It's true that the most recent viruses (Klez, W32.Yaha, W32.Nimda) use executable attachments to spread, but they're not ordinary attachments. [0] Rather, they're attachments with special MIME types designed to exploit a bug in IE. Instead of prompting you like it's supposed to, IE just executes the attachment. Not good. Once Microsoft fixed this problem, the vulnerability went away with no loss in functionality. The reason that this vulnerability still exists in the wild is that people haven't deployed the fix. But then, there's no reason to believe they'd be any more diligent with some hypothetical no-HTML fix. On the contrary, they'd probably be less so since it removed functionality.

Don't get me wrong--executable content is a problem and stupid users do occasionally click on .EXEs they get in e-mail. But that's not what's going on here. It just so happens that there's a bug in IE that lets attackers convince it to execute executable content. So, it's true that Microsoft could control e-mail viruses to some extent by choking off their HTML and executable content support, but that's just because the particular piece of buggy software in question happens to be in that component. There's no guarantee that the next piece of malware would be in that component, so it doesn't really make any sense to suggest that that's some sort of general fix for viruses--unless, of course, that component is especially badly written, in which case MS should rewrite it.

None of this is to say that it's not true that customers are responsible for the sorry state of security. If they had held Microsoft's feet to the fire, Microsoft would have been forced to actually put out non-buggy code. It's even somewhat fair to characterize it as a feature/security tradeoff since presumably what MS was doing with the resources it didn't spend on QA was adding features. But, it's also not the case that if Microsoft had a fix, customers wouldn't take it--albeit in the usual slow way they take all new software. The problem isn't one specific feature but rather Microsoft's general sloppiness.

[0] BTW, this isn't the only kind of MS Outlook virus. For instance, BubbleBoy exploited a hole in ActiveX. Now, ActiveX and similar scripting-type things are notoriously susceptible to bugs, but it's also possible to get that kind of stuff right if you're exceedingly careful, which MS has not been.

Posted by ekr at 04:57 PM | Comments (52) | TrackBack

How to issue Top Level Domains

Karl Manheim and Lawrence Solum have a new article analyzing the economics of issuing domain names. I've certainly read other articles on this topic before, and although Manheim and Solum cover a lot of the same territory, this is the most complete exposition I've seen.

Manheim and Solum enumerate five alternatives for managing the top level domain (TLD) space:

  1. Static root--no more new TLDs are created
  2. First come first served.
  3. Case-by-Case Public Interest Assessment--this is what ICANN does now and the authors rightly label it a "beauty contest."
  4. Lotteries
  5. Auctions

After a bunch of analysis, the authors conclude that an auction is the best approach. They propose an interesting format where people bid not only on individual domain names but also on which domain names will be allocated. Essentially, bidders could bid on any number of domain names and the top N domains would get issued and allocated to the winners. One nice feature of this format is that it avoids the need for ICANN (or a successor) to decide which domain names will be created. The auction could be held periodically to allow the introduction of new domains at a controlled rate.

Posted by ekr at 09:18 AM | Comments (10) | TrackBack

May 25, 2003

Book Recommendations: Alastair Reynolds

Alastair Reynolds is responsible for my more or less complete failure to work Friday and Saturday. Revelation Space, Chasm City, and Redemption Ark (UK only, I got my copy from Kevin Dick) are set 500 years or so in the future. Humanity has settled most of the local star systems, but due to the impassable light speed barrier has broken up into a large number of fragmented factions, each with their own distinct culture and aims. I haven't seen this richly rendered an SF universe since Hyperion.

Reynolds takes the physics seriously so it never feels like he's just making up the rules as he goes along. However, unlike so many books where the science is right but the writing is terrible, the plot is tremendously gripping. Highly recommended if you like this sort of thing.

Posted by ekr at 08:07 PM | Comments (41) | TrackBack

I'm going to change my name to Eric Securityguy

Since Terence is falling down on his poker commentary duties, I'm going to have to pick up the slack. The guy who just won the World Series of Poker is named "Chris Moneymaker".
Posted by ekr at 07:43 AM | Comments (28) | TrackBack

Annika Sorenstam's response to losing

Dan Simon points out something interesting about Annika Sorenstam's response to missing the cut at the masters:
"It was a great week but I've got to go back to my tour, where I belong," Sorenstam said. "I'm glad I did it, but this is way over my head"

and

"I wasn't as tough as I thought I was," she said. "I was so nervous."

Dan's response to this is:

I simply can't imagine any top-level male professional athlete responding this way after failing in his first attempt to break into a new, higher tier of competition. More likely, he'd declare that he'd learned a lot from his defeat, and was looking forward to doing much better in his next try. If he announced that he was way out of his depth and planned in the future just to stick to his proper level, where he "belonged", he'd be derided as a wimp, a quitter, a loser.

My initial reaction was to agree with Dan, but now I'm not so sure. Here's a passage from the latest edition of Full Contact Fighter:

A the end of the 10 minute fight, Milton [Vieira] was declared the winner by unanimous decision and evened up the game again (2-2). "I'll never fight in this categority again. I won because my technique was superior, but I felt he was stronger than me all the time. Definitely my category is 75 kg," said Vieira.

It seems to me that a higher weight class is more or less analogous to playing in the men's tour instead of the women. Bigger guys are just stronger and will pretty much always beat smaller guys of equivalent skill level. Actually, it understates the difference, since bigger guys are typically slower than smaller guys and men are not slower than women.

I think it's reasonable to say that you're outclassed when it's really clear that you are and that you have no chance of making up the difference. Now, it's no doubt easier to say that if you just won, but it's certainly arguable that that makes you more of a wimp, not less, since you've proved that you can compete, at least at some level, if not the highest possible level. That's not my view, however. It's critical to know your limitations.

Posted by ekr at 07:29 AM | Comments (46) | TrackBack

May 24, 2003

More on the Ultimatum Game

Thomas Schelling famously observed that the way to win at chicken was to throw your steering wheel out the window. When your opponent sees you do that, he knows you can't possibly swerve and so he has to. The point here is that committing yourself to behaving irrationally can be a rational strategy.

In a similar game theoretic vein, we have the Ultimatum Game. Bob McGrew over at the Cardinal Collective has anticipated my next post on the ultimatum game.

To a game theorist, this looks like "repeated game effects" - if the game were played over and over again, you'd expect the Responder to reject anything much less than 50/50 to train the Proposer to over him as close to half as possible. But this can't be repeated game effects because the game is played only once, right? Well, maybe not.

There's two possibilities for why people might play a one-shot game like this as a repeated game. The first is bounded rationality - the Ultimatum Game is actually a good model for a lot of human interaction, and so people have developed strategies for dealing with them. These strategies often involve (as Eric points out) a notion of "fairness." Whether this is genetic or cultural is of course impossible to determine, but in either case, people may just play the strategies that are familiar to them instead of wasting time thinking carefully about just how much they should value that $1.

Another, perhaps more persuasive, reason for repeated game effects is a criticism of experimental economics in general. That is, in real-life, there are no one-shot games. After you play the Ultimatum Game with the attractive woman across from you, you might run into her around campus and talk to her. You wouldn't want her to think that you are selfish - it'll make things harder to you in the future. In the real world, every action someone takes affects his reputation, and it's very difficult to design experiments respecting total secrecy. Participants see each other later and talk about the experiment, usually despite the best efforts of the experimenter.

I suspect that the answer is some combination of Bob's answers above. trns out that if you do evolutionary modelling of the Ultimatum Game as an iterated game, fairer strategies tend to dominate. Interestingly, there's no requirement that the same players re-encounter each other, merely that there be a reputation effect.

What reputation allows you to do is to advertise to other players that you're not going to accept their low offer--by not accepting other people's low offers. If they are rational they will then offer you more. The problem, as with the chicken game, is that you might not follow through on your implied threat and instead decide to do the rational thing on your next game. Again, you want to commit yourself to behaving irrationally in the short term because it's rational to do so in the long term. If one is of the evolutionary psychology bent, it's popular to speculate that the reason you get annoyed at perceived unfairness is that it causes you to reject an unfair offer even though it would be good for you in the short term. Now, of course, this mechanism serves you poorly in this specific instance, but then again, the environment in which it evolved wasn't full of behavioral economists but rather of other primates trying to cheat you.

Posted by ekr at 03:23 PM | Comments (34) | TrackBack

The Ultimatum Game

One of the classic demonstrations of human irrationality is the Ultimatum Game. It works like this: There are two players, the Proposer and the Responder. The Proposer is offered a sum of money (conventionally $1) by the experimentor. He has to offer the Responder a fraction of that. If the Responder accepts, they each get their agreed-upon share. Otherwise, neither gets anything. That's it. Most importantly, there is no negotiation. Either the Responder takes the deal or he doesn't.

The rational analysis of the Ultimatum Game is simple. The Responder has the choice of whatever the Proposer gave him or $0. Clearly, anything at all is better than $0. Thus, we would expect the Responder to take whatever the Proposer offers him. Knowing this, we would expect the Proposer to propose low values.

But in real life, this isn't what happens. When people play the Ultimatum Game in the lab, Proposers generally offer around 40-50% of the total amount and Responders will frequently reject numbers less than 30%. Now, it's not irrational for Proposers to offer higher amounts. After all, if they know that Responders will reject lower amounts, that's very rational. However, as shown above, it is irrational for the Responder to reject the proposed division. So, why do they do it?

The general consensus is that Responders reject the division because they think it's "unfair". Now, don't ask me to define what fair and unfair is in this situation, but I guess people think they know it when they see it. Bottom line, though, is that people will behave in ways that hurt them in order to avoid feeling like they've been taken advantage of. Any practical use of economic theory has to take this kind of effect into account.

Posted by ekr at 10:29 AM | Comments (112) | TrackBack

Everybody's a winner!

Everybody likes to win stuff. Unfortunately, the whole point of competions like races, tournaments, playoffs, etc. is to narrow down the field until you have exactly one winner, which, of course, leaves everyone else less satisfied. This is probably fine when it's the NCAA tournament, but when you're running a local road race and people are paying you to participate, you're probably a little more worried about customer satisfaction and so less interested in branding everyone else a loser.

Luckily, you don't have to. After all, you want people who can't walk to compete but there's no way that runners can compete with people on wheelchairs (Yes, you heard that right, the wheelchair division in marathons is much faster than even the elite running division. Rolling is better than running). So, you need a separate wheelchair division. Similarly, women just aren't as fast as men, so you should probably have a separate women's division.

So far so good, but you've probably got a thousand or so people in your race and still only about 30 people who think they're in the running to win anything. The next clever innovation is to say that old people can't run as fast as young people. I know, we need age groups! Generally these are about 5 years wide, which is a little silly because--let's face it--people in 25-29 aren't really much faster or slower as a group than 30-34 year olds. Also, this leads to the funny phenomenon that triathletes call "aging up". Once you get into the older age brackets, each age group is significantly slower than the next younger group. So, for instance, when one of the top guys in 40-44 finally turns 45, he's quite likely to dominate the 45-49 group--until, of course, the best guy in 40-44 ages up as well.

But wait, fat people are slower too! So, let's have a division of fat people. I'm not joking here. The men's division for heavy people is called Clydesdale and the women's is Athena.

You know what the end game for all this is: everyone's a winner. In fact, I've heard that said at races more than once and there are lots of races that give out "finisher's medals" to anyone who crosses the line.

Don't get me wrong here: It's a real achievement to get to the point where you can do a triathlon or a half-marathon (not so much for a 10k). I'm not knocking anyone who does that, but isn't the satisfaction of completing enough? Does it really make people feel better to get that medal?

Posted by ekr at 07:51 AM | Comments (59) | TrackBack

May 23, 2003

Detecting gender from writing style

Was just leafing through an old (April 25) issue of Science and noticed a little article about detecting gender differences from writing. It turns out that by analyzing word frequency, you can gain a fair amount of information about whether a document was written by a man or a woman. Shlomo Argamon and his team report an 80% identification rate on a corpus of various British works.

Based on the article in science it appears that they're using some sort of learning-based classifier. Of course, such classifiers just produce a model and don't tell you how to interpret it. These guys seem to be computer scientists, so it's not too surprising that they come up with a pretty amateurish explanation:

Women use words such as "for," "with," and "and" more often then men, signifying their more commual tendencies. Men are more quantitative and use more determiners," such as "an," "a," and "no."

Luckily, it's not a requirement to understand how your classifier works in order to use one. In fact, that's sort of the point.

Still, it's definitely an interesting result, if it can be replicated. I don't have access to the original article, so I do worry a bit about the statistical significance of the result. Certainly, machine learning classifiers are a powerful technology, but it can be easy to fool yourself that your classifier works when it doesn't unless you're careful to design your experiment right.

Posted by ekr at 09:14 AM | Comments (87) | TrackBack

Why would the PGA want to ban women?

Ben Maller reports a rumor that in the wake of Annika Sorenstam, the PGA is considering banning women from the tour entirely. Other than sheer sexism, I can't imagine why they would want to do such a thing. As far as I can tell, the PGA has nothing to lose from a woman on the tour, provided that she can make the cuts on her own and isn't being given special treatment.

If there's anyone who would want to ban women from the PGA tour it should be the LPGA. The whole premise of the LPGA is that it's the best women golfers. If women regularly compete in the men's division, eventually the very best women will compete only in the PGA, and LPGA will be seen as more of a farm team. That would not be good for them.

Posted by ekr at 08:44 AM | Comments (61) | TrackBack

May 22, 2003

God, that sucked

A bunch of friends and I just saw Matrix Reloaded. It seems to be de rigeur to give your opinion, so here's mine: a debacle.

At the end, Hovav and I had this conversation:

Me: that was appalling.
Hovav: I was going to say "ass".
Me: I can live with that assessment.
Hovav: I don't ever think I've seen that boring a fight scene.

The basic problem is that it was completely soulless. Sure, the effects are impressive in some technical sense but they're completely uninspired. I was already bored with slo-mo bullet-cam 5 minutes in. Combine that with the inane freshman philosophy musing and the pompous "You disobeyed a direct order" dialogue and you've got something more like an episode of Deep Space Nine than a reasonable movie. There aren't many movies that would be improved by adding Jerry Bruckheimer to the mix but I suspect this was one of them.

Oh, and then I sat through 10 minutes of credits to watch a boring trailer for Matrix Revolution.

Posted by ekr at 05:36 PM | Comments (16) | TrackBack

Interpreting the New Atkins Diet Studies

Todays papers have lots of coverage of the two Atkins Diet studies published in this week's New England Journal of Medicine. The headlines in the mainstream press mostly say things like "Atkins Diet Works" but what these studies show is a little more nuanced.

There were two studies, one by Samaha et al. and one by Foster et al. Both studies were controlled comparisons of the Atkins diet to conventional low-calorie diets. The Samaha study lasted for six months and the Foster study for a year. Here are the high points:

  • Dropout rates are very high, in the area of 40% for six months. This makes it hard to analyze the data. Dropout rates aren't significantly different between the two diets, however, so there's no reason to believe it's easier to stay on one than the other.
  • Both studies showed the Atkins group to have lost significantly more weight after 6 months, but the longer study showed no significant difference at 12 months. It's not clear if there really is no difference or just if dropout rates were so high that the remaining group had insufficient statistical power.
  • At best, the differences are pretty small, on the order of 10 lbs after 6 months.
  • Triglycleride levels were better in the low carbohydrate group. This is good news for people worried that the low carbohydrate diet (and therefore probably high fat) would lead to heart disease.
While it looks like low-carb diets are somewhat useful, the data certainly doesn't suggest that they're a magic bullet.
Posted by ekr at 07:48 AM | Comments (66) | TrackBack

May 21, 2003

Statistical screening for bad lawsuits

A lot of the problem in large class-action medical lawsuits is figuring out whether or not people were really harmed. The issue, as I indicated earlier, is that a lot of people have bad outcomes in the normal course of events and so working out whether this is just bad luck or the fault of the defendant is the kind of thing that keeps statisticians and epidemiologists up nights.

My friend Kevin Dick suggests that when people bring this kind of lawsuit, they should first have to go before a panel of statisticians. If the statisticians rule it bogus than the suit is automatically dismissed and the judge has an opportunity to fine the plaintiff and/or lawyers if there's really no evidence.

Obviously, this kind of approach circumvents the jury system, but I think of this as the judge basically dismissing the case. On the other hand, the jury has no real ability to weigh the statistical evidence and people have a natural bias to believe confident people, which is pretty much the one thing that careful statisticians are not. Still, I think we'd want to set the bar fairly low so that the jury got to hear the questionable cases. The idea isn't to have experts decide every case but just to provide an initial screen to deter clearly frivolous suits. It would also incentivize drug companies to do a broader range of studies in order to provide themselves with statistical cover.

Posted by ekr at 09:02 PM | Comments (10) | TrackBack

Could SD-DVDs fail cleanly?

Ed Felten argues that Self-Destructng DVDs will fail uncleanly:
Worse yet (and despite a claim to the contrary in FlexPlay's press release), the nature of a chemical process like oxidation seems to imply that the disk's decay will be gradual. Since DVDs use error correction, FlexPlay's engineers can make the disk reliable for any desired period; but after that there will be an inevitable period of intermittent glitches as the disk gets worse and worse, until it becomes unusable. Seeing the decay, even if it lasts only for a short time, will only make consumers angrier.

This may be the case in practice, but it need not necessarily be so. I'm no expert on DVD formatting, but consider what happens if there's some section on the disk that absolutely must be read error free in order for the disk to be readable (maybe a table of contents?). If you arranged that that was the section of the disk that got destroyed by the chemical process (or at least destroyed first) then the disk would either work or not work depending on the level of the destruction. You could put a checksum over the critical section to ensure that if it had errors that the disk wouldn't play rather than generating glitches.

Of course, you would want to use error correction to make the disk reliable against small amounts of damage, so in the early stages of the process the disk would play fine, but eventually enough damage would accumulate that the critical section would be unreadable and then the disk would stop playing. Again, this may not be the way that SD-DVDs actually behave--Flexplay is pretty cagy about this--but it's not theoretically impossible to design a system that worked that way.

Posted by ekr at 09:09 AM | Comments (12) | TrackBack

What's a fair price for a DVD?

Ed Felten makes a really important psychological point about pricing of information goods:
The underlying problem is that because SD-DVDs will be sold for less than ordinary DVDs, they will draw consumers' attention to the fact that ordinary DVDs are priced well above the marginal cost of producing them. That seems unfair to many consumers.

At this point, readers who are armchair economists (or real ones, for that matter) are raising their hands and bouncing in their seats, eager to point out that marginal-cost pricing isn't sustainable in the movie business, given the high fixed cost of making a movie and the very low marginal cost of distributing a copy of it. That's true, but I think consumers' sense of fairness is based on a different kind of market in which variable costs of production dominate fixed costs.

As long as it seemed inherently expensive to manufacture and distribute a copy of a recorded movie, consumers tended not to notice that the copy was priced above marginal cost. As marginal cost approaches zero, the gap between marginal cost and price becomes much more apparent, and consumers increasingly conclude that the studios are ripping them off.

In fact, one of the major effects of Napster and other music sharing services has been to hammer home to customers just how low the marginal cost of content really is. And as Ed says, this pisses them off.

Self-Destructing DVDs (SD-DVDs) are actually an interesting example of just how differently economists and the rest of the population thinks. To an economist, this seems like a great idea, since it allows people who want to watch the movie only once to pay less. To a consumer, it seems like a total ripoff.

The problem here is that we're rapidly homing in on the lower bound. When the marginal cost of production is moderately high, then content producers don't need market segmentation to survive (though of course it's nice to make a little extra cash) and so end up selling closer to marginal cost than they really want to in order to avoid offending consumers. However, they still made enough to survive and so the market worked. However, if the marginal cost of production is zero, then economic realities start to kick in and it is no longer possible to sell close to marginal cost and customers will just have to get used to that.

However, customer's acceptance of above-marginal cost pricing is contingent on not feeling cheated. And the fact that the content industry is so obviously working hard to extract every last dollar from them doesn't exactly lead to that trusting feeling.

Posted by ekr at 09:01 AM | Comments (47) | TrackBack

Incentives for drug side effects

The New York Times reports that some attorneys are suing drug companies claiming that patients have been injured by side effects of their drugs. Medpundit argues that this is a bad thing:
That's one sure way of stifling innovation. Sue the pharmaceutical companies into bankruptcy or into paralysing fear. God save us from the lawyers.
and I'm sympathetic to that argument, but I think it's a little more complicated.

It seems to me that we need to ask ourselves two questions:

  1. Who do we want to bear the risk of side effects from drugs?
  2. Are lawsuits the right way to assign incentives?
Who bears the risk?
So, you take some drug, which (like all drugs) has a risk of side effects. Now, we can assign the risk one of two basic ways.
  1. It's your lookout, in which case if you suffer the side effects, too bad.
  2. It's the drug company's problem, so they pay you for the negative outcome.

Both assignments have problems. The problem with the first is moral hazard. The drug company is always going to know more about side effects than you do. Since the drug company doesn't pay when people have bad side effects, they're incentivized to shade the public information a bit to make their drugs look more attractive than they really are.

We can fix the moral hazard problem by making the drug company pay when you suffer side effects. However, most drugs have a fair number of side effects. For example, here's the side effect list for Clarinex, which is pretty innocuous as these things go.

Clarinex Placebo
Adverse Experience (n= 659) (n= 661)
Pharyngitis 5% 2%
Dry Mouth 4% 2%
Somnolence 3% 2%
Fatigue 3% 2%
Influenza- Like Symptoms 2% 1%
Myalgia Nausea Dizziness 2% 2% 2% <1% 1% 1%
Dry Throat 2% 1%

So, probably the drug company would be paying out claims to about 10-20% of people. This is a lot of damages to be paying. Worse yet, as you can see from the table, the incidence of "side effects" isn't that much higher than placebo. As a consequence, the drug company is likely paying a lot of people who really haven't been harmed by the drug at all. All this insurance has to get built into the price of the drug, which makes drugs inefficiently expensive (because part of the cost is headache insurance, which you would never buy if it weren't bundled into the price of the drug!).

We have sort of a weird hybrid system:

  1. Drug companies have a duty to disclose all the side effects they know about, but only to look for certain kinds of side effects. (One of the complaints in the lawsuit is that the Bayer deliberately avoided looking for a specific kind of side effect.)
  2. The FDA decides whether the side effects are too high.
  3. Patients are responsible for bearing the risk of side effects.
  4. Because most patients have medical insurance, any serious risks are mostly borne by the insurance company
  5. If people think that a drug company withheld material information, they can sue them (which is what's happening now). The FDA can also punish the drug company.

This system is naturally rife with incentive problems. For instance, there's still a big moral hazard problem with the drug companies. However, as we've seen, it's not easy to design a system without incentive problems. (This is the same kind of problem we saw with malpractice insurance).

Are lawsuits the right way to assign incentives?
As we just saw, the system we currently have has a big moral hazard problem. The drug companies are incentivized not too look to hard for side effects. One way to deal with this is to punish drug companies for not finding serious side effects that they probably should have. This could be done either by the FDA or by private lawsuit. In our system, it's a little of both, but let's focus on the lawsuit issue.

In principle, lawsuits aren't a crazy way to allocate responsibility. Certainly, the fear of being sued can incentivize drug companies to behave responsibly, since making mistakes can be very expensive. However, there are two serious practical problems.

First, lawsuits are a very expensive way to assign responsibility. A lot of money gets paid to lawyers in order to attempt to increase the probability that your side will prevail. This sort of expenditure is just wasted and can end up being a substantial fraction of the liability. Essentially, suing a drug company is a form of rent seeking, and so we get the usual deadweight loss as the drug companies and the customers compete for the money.

Second, it's not clear that the responsibility gets assigned correctly that often. Because it's so expensive to defend a lawsuit--and one of the strategies that attorneys use is to try to increase the cost to the other side in hopes of forcing a settlement--companies often incur substantial expenses regardless of the merit of the lawsuit, even if they ultimately prevail! (Dow-Corning, for instance, was forced into bankruptcy due to breast implant litigation even though there's no real evidence that breast implants made people sick.) This provides a perverse incentive to the company not to develop drugs at all, as well the desired incentive not to cheat. Either drug prices will go up to compensate or companies will just stop developing marginal drugs. Either outcome is inefficient.

What to do?
It's not clear to me exactly how to solve this problem. Clearly, we need to have some way of dealing with the situation where some drug has really serious side effects and the drug company withholds that information. Lawsuits seem like a reasonable way of dealing with that problem. However, to the extent to which drug companies are being forced to expend large amounts of money defending lawsuits where noone has actually been harmed, that's obviously a bad thing. Remember that the base rate of side effects when people take placebos is high enough that it's very hard to distinguish these two cases. I haven't seen anyone propose a system that would clearly allow most of the first type of litigation but disincentivize most of the second.

Posted by ekr at 07:14 AM | Comments (10) | TrackBack

Can Palladium stop zombies?

Steve over at PM Style points out (sorry, broken Blogger permalink) that you almost never see cell phones being turned into zombies for DDoS and spam and argues that this is because cell phones more tightly control what software you can run:
Simply put, cell phones are not general purpose computers which allow users to make substantial decisions about the software which runs on them. The general purpose computer market (including the home market) does allow users to make the choice. And that is the fundamental weakness.

When Microsoft talks about their new Palladium platform, they dance around the real shift in the computing market. Palladium is about two things: (1) protecting Microsoft's monopoly position and (2) limiting the choices of the software which can install and run on the system. It's only #2 that can make a real dent in the number of zombies.

Actually, I think this misses the mark fairly widely:
First, even if cell phones were designed to run arbitrary user software, they would be harder to zombify:

  1. They're not a monoculture. Although there are a lot of cell phones, they're made by a lot of different manufacturers and use a lot of different versions of the firmware. If you have to individually write attack scripts for a whole bunch of different platforms it makes an environment less attractive as a whole for viruses, worms, etc. (these are generically called malware). That's why you see a lot more malware for Windows than for *BSD and Linux.
  2. More central control makes it easier to stamp out zombies. The cell network is more centralized than the Internet, so if a DDoS is in progress it's easier for the provider to detect what's going on and stop it.
  3. Their data channels are more tightly restricted. Since the kinds of data a cell phone is expected to receive (phone calls and SMS messages mainly) are more stylized, the implementations can be less complicated and are probably harder to remotely compromise. Expect this to change some with the trend towards IP-capable cell devices.

Second, it's not at all clear that Palladium or something like it will solve the problem. This argument for why Palladium or something like it is a good thing gets made fairly often by the Trusted Computing guys, but I'm not convinced. Limiting the choices of packages that people will run could potentially improve the state of security in two ways:

  1. Stop them from installing software with known holes.
  2. Stop malware from installing itself.

The problem for the first theory is that most of the vulnerabilities that have lead to serious malware are vulnerabilities in standard Microsoft programs such as IIS and Outlook. This shouldn't be surprising since Microsoft controls so much of the software market and is therefore an attractive target for attack. So, limiting people's choices to the software Microsoft wants them to run would improve the situation only very slightly.

The second theory is a little more convincing, but only a litle. It's true that malware often installs itself as a separate program on your computer, but that's only because that's the most convenient way to do things. It's quite practical to have the malware take over an existing program, at which point limiting the software you can run helps not at all.

The key fact about Palladium and Trusted Computing in general is that it takes away the user's control of his own machine. It allows a third party to restrict the space of things that the user can do. If all you want to do is protect against malware, this is mostly unnecessary. All the same security techniques that you would use to protect against malware can be deployed to separate user-authorized from user-unauthorized software, while still giving the user the full range of choices. Controlling what kinds of software the user can run is neither necessary nor sufficient.

Posted by ekr at 06:00 AM | Comments (65) | TrackBack

May 20, 2003

Security and Distributed Computing

Most computers are idle most of the time. By and large, when you're sitting at your computer, it's waiting for you to type something, move the mouse, or something like that. It seems kind of silly to have all that computing power just sitting around doing nothing, and in the past 20 years or so, a lot of effort has gone into figuring out how to somehow tap those resources.

The general idea is that there would have sort of a big distributed computation network. if you had some large computational task you needed done you could break it into small pieces and have it executed on the various computers (cycle servers) of the network. This sort of distributed computing has seen a fair amount of public interest in the last 5 years or so, including such free projects as SETI@home (processing radio signals for evidence of extraterrestrial intelligence), Distributed.net (cryptographic key cracking) and Folding@Home (protein folding simulation). The Global Grid Forum is trying to design protocols that would allow generic applications of this type.

There are three major obstacles to building this kind of system:

  1. Many computational tasks cannot be parallelized. I.e., they consist of a sequence of operations where operation N+1 depends on operation N. Such tasks pretty much need to be done on one fast computer rather than a bunch of slower ones. This is basically a pure computer science problem and there's been some forward progress on designing parallelized versions of previously serial tasks. Nevertheless, many tasks are still inherently serial.

  2. Security needs to be provided for the cycle server. So, if you're executing some computational task for someone else, you need to be sure that that task doesn't interfere with your use of the computer or have access to your private data. Some of these problems have been solved via virtual machine-type technology, but arranging for payment and ensuring that only authorized sources dole out the work are still somewhat open problems.

  3. Security needs to be provided to the person who's computation is being performed. There are actually two problems here. The first is that they needs to be sure that the cycle server isn't lying. For instance, if the customer is paying the cycle server to do work, the cycle server might try to cheat by not doing the work and generating garbage results.

    The second problem is that the cycle server gets access to the raw data. This currently limits the kinds of tasks to those where the customer doesn't mind their data being publicly exposed. There's a work being done on this problem but it's still unsolved except for a few special cases.

Interestingly, there's one special case distributed computing application where none of these problems exist: DDoS and spam zombies. Spam forwarding and DDoS are eminently parallelizable. Because the attacker has broken into and controls your machine, they're probably not worried that you're going to cheat them, and they're certainly not worried about cheating you, since that's the whole purpose of the exercise. It's kind of depressing to realize that after all these years we finally have distributed computing but it's useful primarily for criminals.

Posted by ekr at 05:47 PM | Comments (10) | TrackBack

How to fight zombies

Today's New York Times has an article about how spammers use other people's computers to forward their messages. The spammer compromises a group of victim computers and installs a piece of software on them. That software works for the spammer and will relay whatever e-mail messages he wants. The compromised computers are known as zombies.

This kind of attack is getting more common. Almost exactly the same thing happens with Distributed Denial of Service (DDoS) attacks. The attacker's motivation in both cases is twofold:

  • To leverage his resources. An attacker can use a zombie network to generate much more spam or attack traffic than he could use on his own.
  • To hide his tracks. The attacker can send a single message to each zombie to kick things off (or even have a single zombie which controls all the other zombies). Since almost no traffic is coming from the attacker he is very hard to track down.

The reason this attack is so easy to perform is that many computers have known security holes that allow an attacker to break in. Generally, fixes are available for these problems, but users often haven't installed them. For instance, the Code-Red worm exploited a 6-month old vulnerability in the Internet Information Server and yet was able to compromise over 300,000 machines. The situation is similar for other vulnerabilities. I'm presenting a paper at USENIX Security 2003 that discusses a specific incident in 2002 (the OpenSSL buffer overflows). I found that only about 1/3 of the hosts surveyed had been fixed after a month and even worse, only about 2/3 had been fixed even after a worm that exploited the bug was released. (See here for the preprint version of this paper).

So, can we do to get people to apply fixes? One suggestion, made by my friend Kevin Dick, is that a consortium of large vendors who get a lot of spam (e.g. AOL, Hotmail, etc.) should pay people to upgrade their machines when bugs come out. The way this would work would be that the consortium would periodically scan your machine to see if you were up to date and pay you if you were. Ordinarily, this kind of approach would be really susceptible to the Free Rider problem, but the number of really big e-mail providers is small enough that they could probably manage to collaborate.

Another possibility, of course, is for ISPs to cut people who haven't upgraded off the network. However, that seems like a much harder selling proposition since it's going to lead to some really unhappy customers.

Posted by ekr at 02:49 PM | Comments (16) | TrackBack

Another reason why Challenge/Response won't work for SPAM

Ed Felten makes an excellent point about Challenge/Response approaches to spam. Since the challenges are manually processed, then a spammer can disguise their spam as a "challenge", thus forcing someone to receive their message.

There are, of course, countermeasures to this attack, but none of them are very good. One thing you might think is to have the user's software respond to the challenge automatically. This obviates the whole point of CR, which is to force a human to enter the loop. If the human's software can process it, then so can spammer's software. Make the challenge stylized so it can't carry spam--but Felten points out that this makes it easy for spammers to make challenge recognizers.

The only solution I've thought of that actually works is for the sender to remember all the mails they've sent out and screen out challenges that don't match them. Unfortunately, that means that CR can't work unless everyone changes their mail clients to accomodate it, which seems extremely undesirable from a deployment perspective.

Posted by ekr at 09:04 AM | Comments (16) | TrackBack

And now Bruce Bartlett chimes in

It's one thing when Warren Buffett, who seems like he might be a bit of a liberal, doesn't like your tax cut, but when Bruce Bartlett (via Matthew Yglesias) says it's nuts, it may be time to rethink the whole thing.
Posted by ekr at 08:39 AM | Comments (21) | TrackBack

Warren Buffett says the tax cut is stupid

In an opinion piece in the Washington Post, Buffett says that the tax cut is a stupid idea in theory, but even worse in practice. But the most important message, the one that people really need to hear, is the one at the end:
When you listen to tax-cut rhetoric, remember that giving one class of taxpayer a "break" requires -- now or down the line -- that an equivalent burden be imposed on other parties. In other words, if I get a break, someone else pays. Government can't deliver a free lunch to the country as a whole. It can, ho