People are all the time telling me, "You can prove anything you want with statistics!" Well, no, you can't. At least, not if you're careful about how you use them.

You can't prove that smoking cigarettes is good for your health, no matter how hard tobacco companies have tried.

But in that case you'd have reason to be suspicious about motives. I'd like to tell you about an example where people using the same data but different methods came to opposite conclusions, both with the best of intentions, as far as I can tell.

In February, Hope Online Learning Academy Co-op sent out a news release noting "Dramatically Improved CSAPs" in students' first year at Hope, comparing the change in Colorado Student Assessment scores from 2004 to 2005, before Hope opened, with the change from 2005 to 2006, when they were enrolled there.

We even published an editorial about it, calling the results "hopeful" and saying, "The results look as if they'll hold up." But we had some questions about the method - we said it "still needs improvement" - although not of a kind to lead us to think the overall conclusion was wrong.

For instance, I noticed that if a student who was near the top of a score range one year and gained just a few extra points over what is expected in a year to move up a whole score range - from partially proficient to proficient, say, which is the difference between failing and passing - Hope's method would actually show a decline. That's peculiar, to say the least, but it would work the other way, too, and there was no easy way to check whether it actually happened often enough to matter. (It was also sufficiently hard to explain that we left it out of the editorial.)

The Denver Post asked Alex Medler, vice president for research and analysis at the Colorado Children's Campaign, to look at the same data. Their analysis, using a different method, showed that Hope students lost ground in their first year there compared with their gain in the previous year. Fewer than a quarter of students made a year's worth of progress while at Hope, though in the previous year about half of them did.

Hope asked Charles Brown, James Jacobs and Phyllis Resnick to do an impartial review of the two analyses. They concluded that there were specific methodological weaknesses in the Hope evaluation that led to the conflicting results.

One of the things the reviewers noticed was the oddity I described above, which they say inflated the assessment of overall performance. Hope should have used a fixed comparison point, rather than a variable one.

A second problem was that they treated missing scores from 2005 as zero, and there were a lot more missing scores from 2005 than from 2006, so that there was a systematic upward bias. I had asked Heather O'Mara, Hope's executive director, about that in February, but I didn't realize then how large the number of missing scores was.

The reviewers found no evidence that either analysis was intended "to skew results in favor of a specific conclusion." And they also warn, as Medler did, that one year's data isn't sufficient for definitive results, and also that other factors might both influence students' decision to attend Hope and affect their performance there.

This should be a warning to anyone planning to release a statistical analysis - have it reviewed by an expert before you make it public. Statistics is not a field for amateurs.

Linda Seebach is an editorial writer for the Rocky. She can be reached by telephone at 303-954-2519 or by e-mail at seebach@RockyMountainNews.com.