Educated Guesswork

EG3K

August has been the best month ever in terms of traffic for Educated Guesswork, with over 3000 visits. For the first time ever, we have averaged over 100 visits a day. I know that's not much compared to the big boys like Instapundit, (who averages 69000 visits per day), but as Terence points out, in any other time or place, if there were 100 people a day who listened to your rantings, we'd call that a cult.

Anyway, thanks for tuning in, and if you like the blog, tell your friends.

Palm expense report software?

I recently started using a Sony Clie PDA--replacing my previous organizing system of memory, post-it notes, and crumpled receipts stuck into my pockets. So far so good, but the Clie doesn't seem to come with any built-in expense report software, so I've been using the memo pad. Do any EG readers use Palm-based expense report software and can recommend it?

Censorship or something

Does anyone else see the irony in broadcasting The People vs. Larry Flynt with big black bars over all of the potentially naughty bits? It's one thing to cover breasts and genitalia, I suppose, to pacify the FCC, but someone smoking a joint? A middle finger? Maybe it would be better just not to show this movie at all...

Displaying the ten commandments

I'm glad to see that Judge Moore's Ten Commandments monument has been removed from that Alabama courthouse. I must say, I'm pretty baffled as to how anyone could claim that it didn't constitute an "establishment of religion". What exactly do they think an establishment of religion would be if a big stone monument saying "thou shalt have no god before me" isn't one?

Why don't I have a SPID???

Well, the Pac Bell tech just called. He claims that he can see my router but that I don't have any SPIDs (some sort of station ID, I think) configured into it. Now, I haven't reconfigured that router in 2 years so I'm not sure how that could have happened. Moreover, when I look at the router configuration it looks fine. We're now in the ISDN customer service dead zone. The Pac Bell line techs only care about the line working. And when I call Netgear, as I'm about to do, I bet they tell me that the thing is configured correctly and that they can't help me.

That's assuming they'll even talk to me rather than trying to charge me for support, that is...

ISDN Update

Well, I got out the old voltage meter and put it on the ISDN line. Nothing. Same story out at the external box. Just called SBC, to report it. They're supposed to have a tester look at it and get back to me. Oh joy.

Sort of back on the air...b

While I was gone in DC my ISDN line went comatose, hence no blogging. In the morning I'll haul out my test tools and try to figure out where the problem is--don't want to haul PacBell out here unless the problem is on their end. Until then I'm on the emergency backup modem line. Slow. Very slow, especially since all the mail that didn't get delivered while my line was down is now clogging the modem line.

Why is wireless more expensive?

So, I'm staying in the Marriott in Gaithersburg and therce's both wireless and wired Internet access in the rooms. The wired access is $10/day. The wireless access is $2.95 for the first 15 minutes plus $.25/additional minute. The same company seems to run both services and it's not like wireless is any more expensive to provide, so I don't understand why they're priced differently. Sure, I like wireless access but it's not worththat large a price premium to not have to stretch a wire across the room.

Update:
Apparently, Marriott's service isn't solely operated by the same people but is affiliated with Boingo. That may explain the different price structure.

Bad signage

In the past two days I've flown out of both Toronto (YYZ) and Oakland (OAK) airports and had an opportunity to compare them. Both have terrible signage but where Oakland's is merely confusing, Toronto's manages to be infuriating. The problem, you see, is that Oakland appears to have once had reasonable signage which has simply fallen into disrepair. By contrast, Toronto has what appears to be new signage which is nevertheless baffling.

When entering Toronto airport you're confronted with a sign that shows three squares with the numbers 1, 2, and 3 on them, plus arrows to show which way to go for each each of them. Now, pretty clearly these are the terminal numbers but how are you supposed to know which terminal to go to? There isn't any sign that says "Air Canada here". Worse yet, we had to return our rental car. Only when we'd committed to terminal 2 (after already going by terminal 1) did I find out where to drop my car off. Amazing.

More on CO2 cylinders

Here's another angle on the CO2 cylinder issue. I said that the ideal gas law I gave was for constant temperature. If we include temperature, we get:

PV = NRT

Where R is again some constant that we don't need to worry about and T is the temperature in Kelvins (degrees above absolute zero). Room temperature is about 20 degrees Celsius or 293 degrees Kelvins. To effect a 1.5% change in pressure we would need to change the temperature by 1.5% (about 5 Kelvins, 9 degrees Fahrenheit) less than the difference between typical daytime and nighttime temperatures. Again, we're lead to the conclusion that it's very unlikely that the tiny pressure differential between sea level and the hold of a plane constitutes a threat to CO2 canister.

Doing the math on flying with CO2 cylinders

Serious cyclists often use portable carbon dioxide inflation systems to let them fix flats fast. They're faster than pumps and lighter too. Unfortunately, the FAA won't officially let you fly with the canisters--though it used to be easy to sneak them past security. The concern seems to be that they'll explode when exposed to low pressure. Is this a problem? Let's find out, using some back of the envelope math.

The first question we need to answer is what the pressure in the CO2 cylinders is. We could probably look it up but it's more fun--and more instructive--to work it out for ourselves. We start with a simple fact about gases. The pressure in a volume of gas at constant temperature is proportional to the number of gas molecules and inversely proportional to the volume:

Pressure * Volume = C * Number_Molecules
or
PV = CN

Where C is some constant we don't need to worry about. This is a simplified version of the ideal gas law

A racing bicycle tire is a torus about 70 cm in diameter and 2 cm in cross-section. It's easiest to think of this as a 2cm cross-section tube 70*pi cm long. Such a tube has a volume of about 210 cm^3. Such tires are typically pressurized to about 140 pounds per square inch (psi). Using the equation above we can find the number of

A CO2 cylinder is a tube about 10 cm long and 2 cm in cross-section. Such a tube has a volume of about 30 cm^3. We don't know the pressure but we can find it using the ideal gas law. Since we're going to be inflating the tire using the cylinder, they must contain the same number of molecules. Thus, CN must be the same in the tire and the canister and we can substitute to get:

Ptire*Vtire=Pcanister*Vcanister
or
Pcanister=Ptire*Vtire/Vcanister

We have Ptire, Vtire, and Vcanister so we get:

Pcanister = 140 psi * (210 cm^3/30 cm^3) = 980 psi

For convenience, then, say that the canister is pressurized to 1000 psi. Ordinary outside air pressure is about 15 psi, so the walls of the canister have to be able to withstand about 985 psi. On the other hand, if we put the canister in vacuum, the pressure on the walls of the canister would be 1000 psi, 15 psi greater, or 1.5% difference. It's highly implausible that CO2 cylinders would be made with this small a margin of safety--any manufacturing or mis-filling error would lead to catastrophic failures. Moreover, in practice the air pressure in planes is much higher than vacuum. Accordingly, we needn't worry about the small effect of pressure change.

Hydrogen cars

Been using my recent plane trips to catch up on my journal reading. The July 17th issue of Science has an interesting article on whether or not hydrogen cars are a good idea. The authors conclude basically not. Some key paragraphs:

Hydrogen could essentially eliminate vehicular emissions, but the cost of reducing NOx emissions [HN5] (for example) with hydrogen will be on the order of $1 million per tonne NO2 (5). In contrast, meeting the EPA's new Tier 2 standards [HN6] will reduce emissions for about $2000 per tonne, and inspection and maintenance programs will cost about $4000 per tonne and scrappage programs (voluntary programs offering bounties for old vehicles), less than $10,000 per tonne (6-8). The cost of reducing NOx emissions from electricity production is in the same range. Similar comparisons can be made for other important air pollutants.

It is comparatively expensive to reduce pollutant emissions by using hydrogen because regulation-driven technological innovation has reduced emissions from gasoline-powered cars to the point where they have very low emissions per-unit-energy compared with other sectors and other transportation modes (see table, below). This trend will continue, reducing the benefit of zero-emission hydrogen vehicles, particularly because many technologies (e.g., electric drive) can be used on both platforms.

Hydrogen could largely eliminate the problem of "high emitters"--the few poorly designed or maintained cars that account for most automobile emission--because hydrogen cars do not have high-emission failure modes. Nevertheless, the approaches listed above, possibly in conjunction with roadside emission monitoring and other advanced techniques, provide far more cost-effective solutions (9).

and

Global CO2 emissions must decline by about an order of magnitude in order to stabilize atmospheric concentrations, so major emission reductions will eventually be required from cars. Cost-effective climate policy, however, starts with low-cost emissions reductions and proceeds at a measured pace. Analysis of optimal climate policy typically shows that to stabilize concentrations below a doubling of preindustrial levels, overall emissions do not need to be reduced by more than 30% below business-as-usual until after 2040 (17). When emission mitigation opportunities across the economy are ordered by their cost (to form a supply curve), deep reductions in automobile emissions are not in the cheapest 30%. All else equal, it is therefore wasteful to devote substantial resources to achieving deep reductions in auto emissions until after 2040 (18). Only then will radical new technologies likely be needed. Hydrogen cars should be seen as one of several long-run options, but they make no sense any time soon.

If we were certain that hydrogen fuel was the only long-run solution to eliminating CO2 emissions from cars, then it might make sense to focus R&D now, even though widespread deployment is decades away. If, however, we accept that there is considerable uncertainty about the optimum long-run solution, then early commitment to hydrogen fuel is unwise because it risks technological lock-in.

I've never really understood the enviro-enthusiasm for hydrogen cars. I think part of the problem is that many enthusiasts don't realize that hydrogen is more an energy storage medium not a fuel. It has to be manufactured from some other energy source. As the authors of this article point out, generating hydrogen necessarily produces CO2 so we need some way to mitigate that. However, if you don't know this, then hydrogen seems like a way to get out of that nasty oil business.

My suspicion is that the second source of enthusiasm is a form of absolutism. Like electric vehicles, hydrogen is a zero emission vehicle, at least at the tailpipe. If you're a purist, this sounds great, but if you do the math it's not so important. that sounds like a fantastic innovation, but really modern gasoline powered cars have such low emissions anyway that replacing them with zero emission vehicles doesn't add much value. It's a lot more useful, though less morally satisfying, to replace the "high emitters" with low emission vehicles.

Update: 20030826
Adam Roach points out that generating hydrogen doesn't necessarily produce CO2. If you have a zero-emission energy generation technology like nuclear then you can produce hydrogen by electrolysis. Of course, if you use a fossil fuel energy source it does generate CO2. I'm not sure what I was thinking when I wrote this.

Easy checkin

I'm back from the wedding but I'm flying out to DC tomorrow morning. This time I'm trying United's new Easycheck in, where you print out your boarding pass and just go to the gate. Will keep EG readers posted on how it works.

Back on the air!

I managed to find a hot spot in Toronto so I'm sort of back on the air. Amazingly my Internet works better than my cell phone, which seems to have decided it doesn't like analog mode. I've been saying for a while that I was looking forward to a day when the Internet was as ubiquitous as cellular telephony, but I may have to rethink that goal.

Involuntary vacation from EG

I'm about to fly out of town for Lisa's sister's wedding. However, since it's somewhere in rural Ontario they won't have 802.11 or cell access and it's not clear if I really even get a phone. So, blogging is likely to be suspended till I get back on Monday.

Empty viruses

As I said, I've been receiving lots of virus-infected e-mail. My friend Perry Metzger offered me a Postfix mail filter to reject such mail before it got to my inbox. This is a big improvement but revealed something interesting: there's still lots of virus sourced mail out there that my filter doesn't screen out. Like this one...

From: <fty@mediapulse.com>
To: <ekr@rtfm.com>
Subject: Re: Approved
Date: Thu, 21 Aug 2003 8:17:41 --0500
X-MailScanner: Found to be clean
Importance: Normal
X-Mailer: Microsoft Outlook Express 6.00.2600.0000
X-MSMail-Priority: Normal
X-Priority: 3 (Normal)
MIME-Version: 1.0
Message-Id: <20030821132434.51A1A7150@sierra.rtfm.com>
X-Spam-Status: No, hits=0.6 required=5.0 tests=NO_REAL_NAME version=2.20
X-Spam-Level: 

[1  <text/plain; iso-8859-1 (7bit)>]
Please see the attached file for details.

This is kind of weird. It's a virus mail but without the actual virus payload. My guess is that the payload somehow got stripped out by some intermediate filter.

However, the fact that there's no actual virus means that my antivirus screens don't work. I've had to resort to some crude content filtering to get rid of these. They all seem to use some variant of "Please see the attached file for details" in the message body so I just filter that out. So don't send me any attached files with details.

Stop sending me virus notifications!

For reasons that are currently unclear, the SoBig virus seems to be undergoing a substantial recurrence. This means that I'm getting all sorts of virus-infected e-mail. This isn't a big problem for me since FreeBSD is immune to Windows viruses and I can easily filter out the mail anyway. What is a problem is that a large number of virus scanners seem to be configured to notify the senders of viruses via e-mail.

So, I've been getting lots of messages of this nature:

To:  
From: IWPSMTP2/CheckFree@checkfree.com 
Subject: NAV detected a virus in a document you authored. 
X-Priority: 3 (Normal) 
Date: Wed, 20 Aug 2003 07:34:43 -0400 
Message-ID:  
X-MIMETrack: Serialize by Router on IWPSMTP2/CheckFree(Release 5.0.12  |Februar\
y 13, 2003) at 
 08/20/2003 07:34:44 AM 
MIME-Version: 1.0 
X-Spam-Status: No, hits=0.6 required=5.0 tests=NO_REAL_NAME version=2.20 
X-Spam-Level:  
 
Please contact your local system administrator.  Please do not reply to 
this email. 
 
 
The infected component in the scanned document was deleted. 
 
 
Virus Information: 
The attachment your_details.pif contained the virus Bloodhound.Unknown and 
was deleted. 

Which would be fine except that I'm not infected. I'm not just saying that because I'm careful--though I am--but because these are Windows viruses and I run FreeBSD! My Windows machine isn't even on! So, this message is totally bogus. The problem, of course, is that lots of viruses forge e-mail from addresses. So, even though I'm not infected, I'm getting lots of totally spurious notifications from people's virus scanners which just clutter up my mailbox. Attention scanner manufacturers: don't send automated replies to viruses that are known to forge from addresses!.

More on softwood lumber

Colby Cosh has a nice piece in the National Post about the Canadian softwood lumber industry and the US tariff. Cosh's argument is that the way that the Canadian government sets stumpage fees indeed is a subsidy and thus the US is quite right to impose a tariff.

B.C. could vapourize the tariff overnight by subjecting its forest industry to real market discipline. Even though it refuses, the prevailing view inside and outside B.C. is not that B.C.'s stubbornness has created a problem for the federal government, but that the federal government has failed to defend B.C.'s interests vigorously enough. Pierre Pettigrew might be justified in turning around and saying to British Columbians, "Hey, you made your softwood bed -- lie in it." But you won't last 30 seconds as a federal Cabinet minister pulling stunts like that.

Unfortunately, the effect is to toss another log on the fire of anti-U.S. rage. How many times have you heard somebody opine that "the U.S. won't play by the rules" when it comes to free trade and softwood lumber? Talk about sour grapes! Hidden government subsidies to private industry are exactly what the NAFTA "rules" are intended to prevent, and a countervailing duty is the precise response the "rules" mandate.

Of course, as I argued before, just because the US is entitled to impose a tariff doesn't mean we have to. And American consumers would be better off if we didn't--and I wouldn't have to hear complaining about softwood when up in Canada this weekend.

And you really don't want "good" worms

In responses to MSBlast, someone has written and released a "good" worm that cleans up the security hole exploited by the MSBlast worm. This idea has been floating around for a while but now someone has actually done it. As expected, it's not looking like that great a plan.

Matt Drudge is reporting that the repair worm for MSBlast is what brought down Air Canada yesterday. I'm not at my email right now so I can't see what the word on the security response lists it, but this doesn't isn't at all surprising. The basic problem is that the "good" worm does damage itself. Aside from the usual problems with automatic update, there's the problem that its a worm. As I've mentioned before, most of the problem with worms isn't that they directly damage your computer but rather that their propagation slows down your machine and network. All of these problems exist just as much with a repair worm as with an ordinary worm.

And then of course there are all the usual problems associated with automatic update.

Do you want automatic update?

If you've been paying attention you know that there's a new worm called MSBlast infecting machines around the network. This isn't the first or worst such worm but for some reason it seems to have pushed people over the edge into "something must be done" mode. The current best candidate for "something" is automatic update of affected software to remove vulnerabilities.

The problem, we have now, is that people aren't very good about installing patches. Thus, even long after a vulnerability is fixed there is still a lot of exposure and thus worms spread. Yesterday Microsoft said that they are considering having Windows require automatic updates unless the user explicitly turns the feature off. It's not too surprising that Microsoft wants this feature: supporting downrev software has always been a nightmare. What's surprising is that people who you wouldn't expect are in favor of it:

"I have always been a fierce enemy of the Microsoft update feature, because I just don't like the idea of someone else -- particularly Microsoft -- controlling my system," said Bruce Schneier, co-founder of Counterpane Internet Security Inc. "Now, I think it's great, because it gets the updates out to the non-technically savvy masses, and that's the majority of Internet users. Security is a trade-off, to be sure, but this is one trade-off that's worthwhile."

I'm not sure I agree. The basic problem is that it's very hard to make an update that doesn't break things. I know lots of people who have installed Windows Service Packs which thoroughly hosed their systems. It's not just Microsoft, of course. I've had problems with FreeBSD too, but Microsoft's updates certainly are often quite brittle. It's not necessarily their fault. There are just too many different ways in which people can configure their systems to be able to test a patch against all of them--and the first law of software development is that if you haven't tested it it doesn't work

Before we deploy a feature like this we really need to do the cost-benefit analysis. The MSBlast worm is annoying but it doesn't actually damage your machine. An automatic update which caused real unscheduled downtime for 10% of the machines that it was installed on would likely do a lot more aggregate damage than any of the current worms--though of course future worms could be worse. It's certainly true that there's a tradeoff here, but I'm not sure I agree that auto-update is on the right side of it.

Great, just great

Apparently some virus has pretty much brought Air Canada's operations down. Did I mention I'm flying to Toronto on Air Canada on Thursday?

palmOne?!?!?!

Maybe you haven't heard--I wish I hadn't--but now that Palm has bought Handspring and is splitting itself into a hardware company and a software company, it's going to rename the hardware company "palmOne" and the software company will be "PalmSource". Now, it's bad enough that the "One" names have been horribly overdone and that "palmOne" sounds like the name of a conference instead of a company, but check out the new logo (from CNET):

I don't know whether to laugh or cry. They've managed to combine BiCapitalization, multicolors, and the use of numbers to replace letters--all hallmarks of bad logo design--into a single logo. Did Palm have B1FF design their new logo or did they just get hAxXoReD?

Confessions of an MCS Sufferer

Like many men, I suffer from MCS. Not Multiple Chemical Sensitivity but rather Male Competitiveness Syndrome. The basic symptom of MCS is not letting people beat you on any workout--ever! I was out running on the Sawyer Camp Trail today when I saw a woman behind me and coming up fairly fast. Pretty clearly she was going to pass me the way things were going. This left me with three choices:

1. Stay at the same pace and get passed.
2. Speed up and keep her behind me.
3. Slow down and run it in with her.

Clearly my MCS ruled out option 1. Option 2 would be a clear advertisement to the world that I had MCS, so that was out of the question as well; denial is one of the other major symptoms of MCS. This left option 3, since then I could tell myself--and others--that I was "just slowing down so I had someone to pace with." Obviously, this was the only viable choice. Of course, it didn't hurt that she was pretty cute and wearing the quite attractive jogbra/shorts combination that is pretty much de rigeur for women runners these days.

Unfortunately, my attempt at deception seems to have failed. Terence and Wendy were behind me and tell me that when she went by them they immediately said "Eric isn't going to let her pass him."

Anyway, good thing that we only had a mile to go, since I stayed with her until right before the end, when she started to sprint and got ahead by a couple of yards. Me, I turned around and went back to catch Terence and Wendy, thus proving that I wasn't beaten, even if she finished first. Yeah, that's it.

Presentation style for long posts

In the previous post I'm trying out a new presentation style for long posts. Instead of just posting them as a single long post I'm using the Movable Type "Extended Entry" feature to post just the beginning and then have a link to the longer version. Do EG readers prefer this style or would you rather have everything on the main page? Comments welcome.

How to interpret margin of error

Last night I complained about the characterization of a 25-22 lead with a 5 point margin of error as "Bustamante ahead". The issue is actually a little more complicated than that, but seeing why requires understanding how polls and sampling works.

Sampling
To keep things simple, let's say that we've got a population of a million and only two candidates who we'll call Schwarzenegger and Bustamante. Say that fraction Fs people support Schwarzenegger and Fb support Bustamante. We're interested in who's going to win the election so we want to know Fb and Fs. Ideally we'd ask all of them and then directly know the answer but this is as heavyweight as actually holding the election. Obviously, that's not practical.

Instead, we sample. We pick some relatively small number of people N, at random, and ask them. When we do this, Ns support Schwarzenegger and Nb support Bustamante. This lets us estimate Fb and Fs in a very simple way:

Fs* = Ns/N
Fb* = Nb/N

We mark our estimates with * to indicate that they are estimates of the true value. The problem here is that not every sample will be the same. Imagine that three quarters of the people support Schwarzenegger and a quarter support Bustamante

Fs = .75
Fb = .25

If we just ask one person, they will either say Schwarzenegger or Bustamante. Thus, we will either estimate:

Fs* = 0/1 = 0
Fb* = 1/1 = 1

or

Fs* = 1/1 = 1
Fb* = 0/1 = 0

Now, we know that neither of these estimates is right--and in principle there's no way we could get a correct answer with a sample size of 1, since not all the people support either candidate. Now, imagine that we ask two people. There are 4 possibilities.

Schwarzenegger	Bustamante	Fs	Fb	Probability
2	0	1	0	.5625
1	1	.5	.5	.375
0	2	0	1	.0625

There are three things to notice here: First, even though there is only one true set of values for Fs and Fb, there are four different possible estimates. Which estimate we get depends on exactly which people we happen to ask. Second, our chances of making each estimate aren't equal. Since more people support Schwarzenegger, the most probable outcome is that we'll estimate that everyone will vote for him. The probabilities are shown in the far right column. Finally, none of the estimates is correct. We know the true values to be Fs = .75, Fb = .25 but none of the estimates match that.

The problem here, of course, is that we don't have a big enough sample. If we asked more people, we'd get a more accurate estimate. (In the limit, of course, if we asked everyone we'd get a completely accurate estimate.) However, as long as the sample is smaller than the total population, there's always more than one possible estimate and so we can never know we have exactly the right one. The purpose of margin of error is to estimate how far off we might be.

Estimating Error
Suppose we choose a more realistic sample size of 500. Now, in principle, we might estimate Fs = 1, Fb = 0 (everyone wants Schwarzenegger) but in practice it's vanishingly unlikely (less than 1 chance in 10^63). [0] The chance that we'd estimate that everyone wants Bustamante is even lower. Figure 1 shows the probability of getting each estimate of Fs.

Figure 1: Distribution of Fs* values (Fs = .75)

As you can see, although the most likely estimate is the true value, the probability that we'll get exactly the right value is only .04 (about one in 20).

Figure 1 lets us estimate the chance that our estimate will be the true value--which, as we've seen is pretty low--and how likely it is that it will be off by a given factor. However, this isn't quite the piece of information we want. If we already know what the true value is, we wouldn't need an estimate. The situation we're usually in is the one where we're interested in estimating the true value. We've sampled and we've got an estimate Fs and we want to know how far off we might be.

However, remember that in principle we could be arbitrarily far off. So, if the question we're asking is "how wrong could we be", the answer is "totally". That's not really satisfactory. First, it's not useful. Second, it's not accurate: even though it's theoretically possible that our estimate is way off, in practice it's most likely that we're fairly close and increasingly likely as the sample size gets larger. So, we need a more sophisticated way to describe error.

Figure 1 shows us how likely it is that we'll get estimate Fs* given that the true value is Fs = .75. We can draw a similar figure for any value of Fs. But what we really want to do is turn the question around: Given that our estimate Fs* is X, what are the likely true values Fs? So, for instance, if Fs* = .75, we get something like Figure 2.

Figure 2: Distribution of Fs values (Fs* = .75)

Margin of Error
So, given Figure 2 (or its equivalent for whatever Fs* value we measure) we can answer the question: How likely are various true values Fs>. However, presenting pictures like Figure 2 isn't very useful when you're just writing text. It would be nice if we could summarize Figure 2 with a single number. This is relatively straightforward.

Looking at Figure 2, we can say qualitatively that Fs is most likely between .71 and .79. What we want to do is formalize this notion. In particular, given that Fs* = .75 there is a .95 probability that Fs is between .71 and .79. The region .71 to .79 is called the 95% confidence interval. For any population and sample size, it's always possible to compute a 95% confidence interval. There's nothing special about 95%, of course. One could just as well use 90% or 99%. With a 95% confidence interval, the true value will fall within the confidence bounds 19 out of 20 times.

So, what's a margin of error? Essentially another name for a confidence interval. If we have a margin of error of .04 at the 95% confidence interval, that means that we can be 95% confident that Fs lies between Fs* - .04 and Fs* + .04.

Who's going to win?
In practice, of course, we're not really interested in what the exact numbers are, but who's going to win the election. For the purposes of that, we're only interested in asking the question "Is Fs > Fb?". Now, in a two-candidate election where people need to vote for one or the other this question is the same as "Is Fs > .5?" However, since there is more than one candidate in the election, the question is a lot more complicated. However, the principle is the same: we want to know if Fs > Fb. [1]

Unfortunately, as should be clear from the previous discussion, we can't always answer that question definitely. All we can answer is the question "How likely is it that Fs > Fb?" Now, if that probability is very high (close to 1) then we can feel comfortable saying "Schwarzenneger will win" or if it's close to 0 then we can say "Bustamante will win." However, if it's somewhere in the middle we can't really say anything useful other than "I don't know". Since the difference between the estimates for Schwarzenneger and Bustamante is less than the margin of error, as far as I can tell, that's the situation we're in now.

 

[0] As the sample size gets close to the population size, the degree of error starts to be limited. For instance, if the population size is 1000, a sample of 999 can obviously only be off by one vote.

[1] There are other complications as well. It's very hard to get a really random sample and that can result in estimates which are far more likely to be wrong than the margin of error would suggest. I'm describing what's pretty much the ideal case.

What do you think margin of error means?

Salon is carrying a AP-sourced story with the headline "Poll: Bustamante leads Schwarzenegger". However, when you read the text you see a different story:

Bustamante, a Democrat, had the backing of 25 percent of those questioned, compared to 22 percent for Schwarzenegger, a Republican.

...

The survey was conducted over a four-day period that ended Wednesday. It had a margin of error of plus or minus 5 percentage points.

Attention headline writers: when the difference between support for two candidates is less than the margin of error, that means we don't know who's ahead! Is that really such a hard concept to grasp?

Poker humor

Caught some of the final table of the World Series of Poker earlier. Normally the commentary is relatively calm but sometimes it can bne pretty funny. On one hand today, Amir Vahedi is heads up with Sam Farha. We're past the river and Vahedi has like 10 high but he doesn't want to go out. The commentators have been ragging on him the whole hand, and finally they just can't take any more: "What's this? Vahedi's going for his chips?!?!. The poker police should come arrest him. This is like a poker DUI."

How many blackouts should we have?

In the wake of the East Coast Blackout of 2003, there's been the usual complaining about how brittle power networks seem to be. Check out, for instance, Jon Mandle's post at Crooked Timber:

After all, in this case, it's the power companies that are not living up to their end of the bargain, not the consumers. It's not that they are now making a windfall profit from the blackout - they've already done that by sticking with an "old and antiquated" infrastructure and not investing in the necessary upgrades that would have prevented this in the first place.

Instead of admonishing consumers to modify their behavior, why not force the power companies to adopt a market-based solution? When the power companies are unable to meet demand, force them to offer consumers an incentive to conserve - say, a voucher for each kilowatt hour they use below their average that can be redeemed for free power when the crisis is over. Otherwise, appeals to one's civic duty smack of being just another marketing ploy.

However, it's important to remember that the problem here was more the grid itself than the supply of actual energy and the grid operator is more of a regulated monopoly than it is an ordinary competitive market. So, the real question, as Andrew Odlyzko points out the real question is whether we have a grid of efficient quality. That means that there are roughly four ways in which the grid could be inefficiently bad.

1. We're not spending enough money on power overall to to prevent this kind of blackout.
2. We're spending enough money but the regulation isn't working and so the grid operator is making excessive profits. (This is Mandle's argument).
3. We're spending enough money but it's not being used in the best way.

Odlyzko argues that it's probably not case 1:

Let me add yet another $0.02 worth, weighing in on the side defending the electric power industry. Let's take a very high level economic point of view. Should oodles of money be spent improving the power generation and transmission grid? Suppose that the current system were judged likely to produce blackouts such as this past week's about once every 10 years. How much does that cost the economy? To be extremely conservative, suppose that an entire day's production is completely lost. Well, in a $10 trillion economy with about 250 working days in a year, that comes to a loss of $40 billion. But if that happens just once every 10 years, the annual cost is only $4 billion. Hence before calling for giant new construction programs, make sure they will not cost more than $4 billion per year.

This is a really important point. The ideal grid would almost certainly have some large blackouts. Overengineering to eliminate all of them is very likely to be inefficient, particularly if the cost of blackouts is relatively low. Of course, that doesn't mean that that's the grid we actually have, since we may still not actually be paying enough, or, alternately, spending it in the wrong place.

If we're in case 2, then the grid operators should be raking in money hand over fist. I haven't really looked into this, but that's not my general impression. At least in California, the grid operator is the California Independent System Operator, which is a not-for-profit.

I have no idea whether or not case 3 applies. On general principles, one might expect a grid operator without a profit incentive to not spend its money very efficiently, but I don't have any evidence for this.

The bottom line here, I think, is that power engineering is quite tricky and it's not really clear whether we have a grid of the appropriate quality or not.

Refinancing

Lisa and I have spent a fair amount of the day with mortgage refinancing. Being self-employed, this is a major hassle since it means assembling all of my tax returns in order to document my income. I suppose if I were more organized it would be easier, but I'm not and it's not. I'll almost be glad when rates go way back up again and I can be sure that I won't be able to refinance.

On the other hand, I've recently signed on to Paytrust and it sure looks like it's going to make my life significantly simpler. Once you've got it going, all your bills are automatically scanned and sent to you electronically. In fact, you can even arrange for it to pay them automatically. Since I'm the kind of guy who dreads going throught that big stack of bills, this seems like a major improvement.

Ooh.... West Nile Virus...

Henry I. Miller uses West Nile Virus as a springboard for an argument that DDT shouldn't have been banned. Now, I tend to agree that the ban on DDT was a major overreaction, but West Nile doesn't make much of an argument for that. Miller writes:

The four-year old U.S. outbreak of West Nile virus is a significant threat to public health. With the peak season just beginning, the mosquito-borne virus has been found in animal hosts (primarily birds and horses) in 40 states, and has caused 367 serious infections and nine deaths in humans in 20 states.

Last year, there were more than four thousand cases and almost 300 deaths. We may be on the verge of a major epidemic, but there is no treatment and a vaccine is at least a decade away.

There are a number of problems with this argument. First, 300 deaths is a miniscule number in the United states. More people die of appendicitis than West Nile. Heck, in 1999 ofver 4000 people died of peptic ulcers. I didn't even know that people even could die of peptic ulcers! Second, it doesn't look to me like vaccines are 10 years away. On the contrary, there is a West Nile vaccine in clinical trials now and more work in the pipeline.

It's quite possible that there's some good argument for bringing back DDT (malaria is the usual one), but hysteria over West Nile isn't it.

Saved by incompetence

CNET is reporting that there have been somewhere on the order of a hundred thousand computers infected with the MSBlast worm. Given that pretty much any version of Windows is vulnerable and there are a lot of Windows machines out there, this is actually a pretty nice low number. Why? The worm isn't very well written.. Having bad enemies is a lot more important than being good.

Where did I put that MD5 again?

Slashdot is reporting that the Free Software Foundation's FTP site has been hacked. As a consequence, they've been forced to take down a bunch of their packages. The concern is that someone has replaced the package with a new version containing some malware--conventionally called a "Trojan Horse".

This sort of thing happens pretty frequently, actually. Distributions of a number well known packages have been trojaned by hackers. The good news is that good security practices are making it a lot easier to detect this kind of thing. One of these is the FreeBSD ports system. FreeBSD comes with a series of "skeletons" which you can use to download and build various applications for FreeBSD. These skeletons come with an MD5 message digest [0] for the distribution source code. This lets FreeBSD users detect if a package has been trojaned. Not only does this protect FreeBSD users, it also protects the rest of the net since it increases the chance that some FreeBSD user will notice the change and report it.

It's worth noting that you need to store the MD5 separately from the package. If the MD5 is on the web site with the package, an attacker can just replace the MD5 when they replace the binary and noone will be any wiser. With the FreeBSD ports system, of course, the FreeBSD team maintains their own MD5 list and so you would need to break both the original distribution sites and the FreeBSD site to successfully deploy a trojan.

As a developer, I do two things to prevent this kind of attack. First, I PGP sign my code so that even if my web site is compromised noone can release trojans. My PGP key isn't on my Web server so an attacker would need to break my home machine to deploy a trojan. Second, I keep a paper notebook with digests of my software releases. Even if someone managed to steal my PGP key they wouldn't be able to change that notebook. Unfortunately, what appears to have happened here is that the FSF people don't have MD5s for all of their packages and thus they can't verify the correctness of the data on their site.

[0] A message digest is basically a large number that's characteristic of a certain string of characters. If the package has been trojaned the digest will be different and so this can't be detected.

Update 15:43:
Here's a pointer to the FSF statement on this incident. Apparently the machine was cracked in March! They're posting a list of good checksums but what we really need is a list of packages that were compromised.

Yeah, whatever

I read that the Free Software Foundation (FSF) has asked the GCC [0] maintainers to add a "README.SCO" file to GCC containing their position on the SCO/Linux thing, including an implicit threat to remove SCO support:

We have been urged to drop support for SCO Unix from this release of GCC, as a protest against this irresponsible aggression against free software and GNU/Linux. However, the direct effect of this action would fall on users of GCC rather than on SCO. For the moment, we have decided not to take that action. The Free Software Foundation's overriding goal is to protect the freedom of the free software community, including developers and users, but we also want to serve users. Protecting the community from an attack sometimes requires steps that will inconvenience some in the community.

On the list of things that SCO has to worry about... being sued by Red Hat, running out of money, having nobody buy their product, I suspect that the possibility that some future version of GCC won't support SCO is somewhere behind runing out of Sprite in the lobby soda machine.

[0] GNU C Compiler--the more or less standard C compiler on Linux, BSD, and many UNIX systems.

More on social DoS attacks

Incidentally, the DoS attack is already very common as a tactic in legal proceedings. It appears to be fairly easy to drag out any proceeding almost indefinitely long, thus forcing your adversary to spend more and more money. Big corporations of course use this to protect themselves by outspending their opponents, but it can work for the little guy too. It's essentially the threat of this sort of DoS attack that makes malpractice and product liability lawsuits so effective.

Social denial of service attacks

Internet security types have recently become a lot more interested in denial of service (DoS) attacks. In a DoS attack, the attacker sets up some set of conditions that prevents some service provider (typically a server) from being able to do its job. There are a number of different ways to mount DoS attacks, but the general procedure is to set up some set of inputs that the designers of the system didn't anticipate and that the system handles badly.

Watching the California recall process, it occurs to me that it provides a bunch of opportunities for DoS attacks.

• The total number of signatures needed for a recall was around a million. Apparently Darrel Issa spent on the order of a dollar per signature. It's not hard to believe that someone willing to spend 10 million or so could get the requisite number of signatures even against a rather more popular governor than Davis is. I rather suspect that the recall process wasn't designed to handle committed attackers who just wanted to jam up the works.
• There are over 100 candidates certified on the current ballot. That's already way more than we can realistically handle. However, it would be easy to completely clog the ballot and the electoral process by arranging for hundreds of candidates at a mere $3500 a pop.
• I seem to remember hearing somewhere that there was a relatively limited pool of signature gatherers and that it was possible to hire a lot of them and thereby impede other people getting signatures. This would only work a limited number of times since eventually more gatherers would enter the market, but it could be very effective once or twice.

Of course, these aren't the only kinds of social DoS attack you can mount. There are lots of others:

• Freedom Of Information Act requests are easy to file, cheap, and require quite a bit of activity by the handling agency. Moreover, if you're a news agency, you only pay duplication fees.
• It's really easy to block various kinds of government actions by filing repeated protests: requirements for hearings, lawsuits, etc. This technique is used by environmentalists and others to block various kinds of development projects.
• Airport security is a particularly juicy target. All sorts of essentially innocuous acts on your part can trigger extensive searches of your person, thus tying up security resources.

The basic principle behind all these techniques is the same: leverage. Find something that doesn't cost you much but costs your victim a lot. The bad news is that there are an enormous number of such attacks. A fair amount of work has been invested in trying to prevent DoS attacks on computer networks--a much more limited domain--with only very modest success. Preventing social DoS attacks is likely to be even harder.

What do you mean I can't print?

Last night I downloaded a copy of the LA Times list of all the candidates for California governer. IT's a big PDF file and I don't like reading on screen so I went to print it out.... Denied! Apparently there's some bit in PDF that tells Acrobat not to print the file. Caveman digital rights management.

Most likely what what we've got here is a prime example of the DRM default effect. There was no real reason to stop me from printing this file, but whenever there's a question about whether to add some restriction or not, it's easier just to add it than think about whether it's necessary or not. Nobody ever got fired for adding too much DRM.

Incidentally, I did finally get the file to print. I hand-edited to remove some errors, ran it through pdf2ps and then ps2pdf and finally used Acrobat to print the new un-DRMed file.

Why bare your teeth

Perhaps you were wondering why baring your teeth was an aggressive act. Humans have small teeth and don't usually fight with them as much as their fists. Check out this image of a mandrill doing a "threat yawn".

Being able to do a threat yawn like that would be pretty useful in business negotiations. I sure wouldn't want to argue with any animal that had teeth that size.

Men still at work

Every couple of weeks since June, the Palo Alto paving department drops a bunch of work flashers on my street announcing that they're going to be digging my street up in a week or so and I have to keep my car off the street or I'll be towed. About half the time the target window passes and nothing happens. The other half the time they come by and do a little bit of work and then leave for a week or two. One week about two months ago they paved a 20x50 patch of road. Last week they dug up the edges of the street closest to the sidewalk.

Is there some reason they can't just repave the whole thing in one shot? I go out of town a lot and keeping track of when I can or can't park on the street is a real pain. Is this really standard procedure? I live on a small cul de sac, so I would imagine that if they just decided to do it all in one time it could be executed in a week. For that matter, I don't even know what the objective here is. They've never told me and the street was in pretty good shape before they started digging.

Why politicians shouldn't get angry

Was watching The American President last night. Fairly ghastly, of course, but it got me thinking about something. You essentially never see politicians get angry in public? Why?

Primate anger responses look stupid. When you get mad your blood pressure goes up, you bare your teeth, your face flushes and contorts, etc. Now, when you're right in front of an angry primate this is relatively scary because your own limbic system is engaged. But on television, in an isolated snippet or in freeze frame, an angry human just looks silly and out of control, as below.

Consider what happens when you see people angry in movies or TV programs. You've generally been watching the program for a while and if it's been at all skillfully done, you're already engaged in the program and so your limbic system has had a chance to respond. Thus, you find the simulated anger affecting rather than silly. By contrast, any politician who gets angry is likely to have that moment excerpted repeatedly, and seen by people who lack the context. To such people, he just looks ridiculous and so he has to avoid it. Instead, he has to settle for "determined".

The Internet Is Too Secure Already

The slides from my USENIX Security Invited Talk "The Internet is Too Secure Already" are now up on my main web site in PDF form. I'm hoping to write up the argument in some detail when I get a chance, but it's easy to summarize.

As someone who works in Internet security, I find it striking that although we have really an immense amount of security technology, in practice the Internet isn't really that safe a place. This is especially noticeable in communication security (COMSEC). Out of the fairly large number of communication security protocols that have been designed and implemented, only a very few have seen wide usage. Interestingly, the problem isn't that the they're not in customer's hands. For instance, if you're running Windows 2000, you probably have an implementation of SSL, IPsec, S/MIME, and PKIX, but the only one of these that people actually use on anything like a daily basis is SSL--and that only for e-commerce. Why? The basic problem is that the other three protocols are massively inconvenient to use.

It's my feeling that the problem is that the security types who design these protocols have got the wrong threat model. When we design protocols, we typically ask "what's the worst possible thing that the attacker could do?" and then try to design a system that can resist it. That's a pretty good strategy if you have only one attacker who will do anything to damage you, but not such a good one in a world where there are lots of attackers and potential victims but noone bears you any particular malice.

The problem is that when you design a system to protect against all possible threats it generally becomes much more heavyweight and inconvenient to use. Users generally aren't really that concerned with security and so naturally resist using systems that inconvenience them. As a consequence, we have all these systems that are theoretically secure but provide almost no security in practice since they're unusable. If we were just willing to design systems with a more realistic threat model, we might actually come out with something that was deployable. Of course, this would inevitably involve systems that had known security flaws, which is something that the incentives that drive security people tend to discourage. However, we would probably end up with a world that was more secure overall.

Hanging with the podiatrists

The American Podiatric Medical Association is holding their conference here in the same hotel we are. Unlike USENIX, they seem to have at least some of their papers being given via posters that are just set up in the hallway. This strikes me as a pretty nice approach. You can't say that much on a poster so you have to really think. This makes it much easier for a reader to figure out what's going on and then he can go read the paper if he wants to. Heck, even I wanted to check out the posters.

Of course, these particular posters have lots of surgical pictures, which is a bit off-putting, but we wouldn't have this problem in CS.

Why listen to talks?

Another take on the "why present at conferences" thing. A lot of the stuff here at USENIX is the "referreed paper" track. The way that this works is that people submit papers, just like they would for journals. The authors of the accepted papers get to present them at the conference. This generally means a 25 minute PowerPoint talk.

This has always struck me as a rather strange practice. The papers are generally available well ahead of the conference so people have read them if they care. Moreover, a PowerPoint talk really isn't that good a way of conveying this kind of work. 25 minutes is too long to have a single central point and too short to really cover everything in your paper. Finally, in most researchers aren't particularly good public speakers so the talks are often quite boring--people reading from their slides.

I appreciate the need for conferences as a schmoozing opportunity, but is there some way to make them more interesting? Do other fields do a better job?

The low cost bidder

I'm at USENIX Security now. Flew in on the red-eye on JetBlue. Now I know why JetBlue is so cheap. I don't know what the seat pitch was but I can tell you that I felt incredibly wedged in. I arrived at 7 AM EST and at 8:00 EST I'm just starting to feel like my legs are stretched out again.

Another outsourcing option

Via Wendy McKibben comes a link to a new option for outsourcing programming effort. From the FAQ:

What about software testing?

Great apes (hominids) do not have tails, while monkeys do. Research indicates that great apes are very productive in the areas of software maintenance and report writing, while most monkeys will struggle. Monkeys however are great at software testing. So the rule of thumb is, if you don't have a tail, you can probably program.

We train our baboons specifically for software testing, using various industry-standard testing tools.

If you need programming done, you might as well hire some real code monkeys.

Is it just kernel 2.4?

One interesting thing to notice is that SCO only says that the problem is with the operating system kernel and only with kernels 2.4 and 2.5. Does this mean 2.2 is in the clear or they just haven't looked? If kernel 2.2 is clear then the easiest solution may be to move back to it. This could be done by downgrading to an older version of Linux or by one of the vendors making a Linux with a new userland and an older kernel. [0] That's probably a modest amount of work but should be quite doable.

[0] Note for non Linux-heads. Linux consists of a lot of different pieces of software all distributed together. The kernel is just the main program in the operating system--the one that's responsible for talking to the hardware. Userland is all the utilities that surround it, including the graphics system, editors, mail servers, etc. The userland programs all depend on the kernel and run on top of it, but as long as the programming interface presented by the kernel is constant, they can be relatively insensitive to kernel version.

Technically speaking, Linux is just the kernel itself. Most of the userland programs were written independently, many before the kernel even existed. A lot of the userland was originally written by the Free Software Foundation for their GNU project, which is why you'll sometimes see people talk about GNU/Linux to give the FSF credit. A follow common practice and use Linux to refer both to the kernel and to joint distributions of kernel and userland such as Red Hat or SuSE. Where distinctions are necessary I'll say "kernel" specifically.

$699 for Linux seems a little pricy

So, SCO has finally put their cards on the table. They claim that if you're running Linux kernel 2.4 or 2.5 you need to pay them.

"We have identified numerous files of unlicensed UNIX System V code and UNIX System V derivative code in the Linux 2.4 and 2.5 kernels," said Chris Sontag, senior vice president and general manager of SCOsource, the intellectual property licensing division of SCO. "We believe it is necessary for Linux customers to properly license SCO's IP if they are running Linux 2.4 kernel and later versions for commercial purposes. The license insures that customers can continue their use of binary deployments of Linux without violating SCO's intellectual property rights."

The fee will be $699/machine until October 15th and then reportedly $1399/machine afterwards. That seems a bit pricy. For comparison, Windows 2000 Professional is only $259 and Windows XP is only $279. And that comes with support--or whatever passes for support at Microsoft. As far as I can tell, SCO wants $699 just to promise not to sue you.

Even if Linux is actually worth $699 a CPU, it's not clear that SCO is the lowest cost vendor. Once the stuff that they claim is copied is public knowledge, someone else can build a new Linux without that code. Your cost for that is just the temporary exposure and the cost of converting.

There are other options as well. One could convert to FreeBSD which doesn't have this potential IPR problem. Finally, this seems like a great opportunity for Red Hat or IBM to offer "SCO insurance". I bet they could do it for less than $699 a CPU. (Actually, anyone could do this, but since Red Hat and IBM have already assumed some liability, it's probably easier for them).

What I wish PowerPoint would do

I'm preparing my talk for USENIX Security 2003 (Shameless plug: come see it. It's called "The Internet Is Too Secure Already"). As usual, when I use PowerPoint, I really wish that the following feature existed:

• Display my slides on the projector.
• Display my slides and notes on the laptop screen.

Then I could talk and use the notes to remind myself of the points I want to make. At the moment, I'm reduced to printing out the notes pages and going through them as I talk.

Now, I know that this may not technically be possible in software. I'm not that familiar with how laptops video systems work, so it may be the case that the stuff that comes out the VGA port must be the same as the stuff that displays on the LCD screen. (Actually, it probably depends on your laptop). However, if this is possible, it sure would be convenient. And if it's already hidden in PowerPoint somewhere, someone please let me know.

The market speaks: Charles Taylor is out

I just received the following spam:

First, may I solicit your confidentiality in this transaction, this by virtue of its nature.

I am Tomson Taylor, a cousin to the president Charles Taylor of Liberia.