The Statistics of Homeschooling

Since 1999, the famous Rudner study has been touted as finally answering the academic question about homeschooling once and for all. Meanwhile, the original critique of that study has been largely ignored.

Subsequent critiques by homeschoolers themselves have come to the fore as well:

1. "Embarrassing and Dangerous" (This series was written by a homeschooling leader in Wisconsin; part two is an excellent dissection of the testing methods--taken from the Home Education Magazine).
2. "The Fraser Study: Puffing Up Homeschooling and Selling Our Freedoms" (This series was also in the Home Education Magazine, with multiple links to other articles critiquing NHERI and other research leaders).

This means that homeschoolers are not dominating in academics--or at the very best we do not statistical know how they are doing.

This may be quite shocking to some readers. I can understand; I was amazed as well. But facts are facts. I have no ax to grind since I will soon be a homeschooling dad.

What the Numbers Reveal

The Rudner study showed that homeschoolers in the Bob Jones University (BJU) locale who took the test did very well on that standardized test.

That's it.

What the Numbers Do Not Reveal

The fundamental problem with the Rudner study is a problem admitted by the author himself: "...it should be noted that it was not possible within the parameters of this study to evaluate whether this sample is truly representative of the entire population of home school students."

If his study is not "truly representative" in any conclusive sense, then why bother with it at all? Admittedly, researchers are about knowing. But after the knowing is accomplished and the caveats written, such a study reveals very little. Another answer may be that the study was funded by an organization already in favor of homeschooling. Many homeschooling groups touted the study, even to this day, without admitting any of the caveats the author himself wrote (at least the full report is there). Too often conservative Christians blame the liberal left for "tweaking" data in their favor--whose to say such a thing cannot happen amongst conservatives?

In fact, a few years later in an interview, the author of the study, Rudner, complained: "The biggest annoyance was a large number of reporters that had read previous articles and never went to the original source and read the caveat." And what and where was that caveat? Well, right in the original research itself!

"This was not a controlled experiment,'' and it "does not demonstrate that home schooling is superior to public or private schools, and the results must be interpreted with caution" are some serious caveats. And these carefully chosen words were in the opening paragraph of the report. The fact that they were not touted as loudly as the finding itself points to sloppy research or self-interest--either is not something commendable.

Through all the newspapers, journals, tv reports, lectures, postings, advertisements and word-of-mouth I encountered I never once heard these caveats.

Lone Ranger?

Perhaps some distraught readers out there think that this is a non-issue or a smokescreen brought up by an unskilled one-man army. Besides the fact that I have an extensive math background (B.S. Electrical Engineering with antenna design), scholars themselves have pointed out the limitations of the studies and even some homeschooling leaders themselves are uncomfortable with this loose usage of statistics.

One interesting read is from Professor Milton Gaither who has written a virtually definitive book on the history of modern homeschooling in America.

What About Those Experts?

Shorty after Rudner published his finely nuanced report, a peer review was published analyzing the usage and methods empoloyed--a not untypical activity in this field--by Welner & Welner.

Another expert writes in Education and Urban Society. Similar concerns were discovered in other education magazines.

And most interesting of all, Bruce Ray of the NHERI (homeschooling advocate group) notes modest academic numbers for homeschoolers. The latest work by Brian Ray and Bruce Eagleson, State Regulation of Homeschooling and Homeschoolers’ SAT Scores, notes in the introductory background information, that there are mixed results on the testing advantages of homeschoolers (SAT and ACT) (two studies show virtually no statistical advantage and two more show some advantage). As for college exams: "The few studies done on home-educated students’ performance on college-admissions tests suggest they score about as well as do those who are not homeschooled."

The Devil is in the Details...

The first analysis of Rudner's first study was accomplished by the team Welner and Welner:

"Some researchers, in fact, would say that the test scores have nothing to do with how the children were schooled and simply show the results expected for children that come from this demographic group—households that are overwhelmingly white, well educated, two-parent, and middle class (see Coleman et al., 1966; Ogbu, 1987). This is not to say that these parents did not do a good job teaching their children, it is only to say that a comparable sample within the public or private schools may have scored just as well." (Welner & Welner)

The SAT and ACT representatives have a similar take:

"Home schoolers who take the entrance tests are disproportionately white and from families with above-average income and education. This demographic group tends to score higher than the average anyway."

Furthermore, this study does not take into account that testing is not required for a number of homeschoolers. And of those states that require it, most allow the parents to use a private professional to evaluate the child, the typical public school standardized testing being completely bypassed.

One obvious reason for this severely limiting aspect of the study is that it occurred amongst a culturally narrow group: those taking the tests at BJU. What about those families from less waspish environs?

To date, a Barna study strongly suggested what some thoughtful statisticians have pointed out elsewhere: homeschooling is not a middle-class, white, Evangelical monolithic entity (Welner & Welner).

Another problem not addressed by this study is the number of unschoolers and the percentage of mixed schooling (both homeschooling and private/public) that occurs among homeschoolers. As of this date, since investigating this issue, such numbers are greatly underreported in many homeschooling circles.

Another statistician's study highlights the limitations of the Rudner study as well:

"The recent achievement studies by Rudner (1999) and Ray (2000) are notable given their large sample sizes. However, Rudner’s study has been criticized (Welner&Welner, 1999) as having a biased sample. Given that his sampling frame originated from a conservative, religious institution, it is unlikely that the diversity of the homeschooling movement is represented in his data. The response rate was also low. Ray’s (2000) response rate was problematically low. Moreover, because he obtained student data from only 38% of the 29% of families that responded to the survey, the likelihood of bias increases. Although the sample underlying the research reported here also has limitations, the data and analysis make a unique contribution to this growing research literature. (p.314 Education and Urban Society / May 2005)

Note how neither critique ignores Rudner's study; they only point out its severe limitations. It helped prompt more studies and suggest areas of research. It did not demonstrate scientifically that homeschoolers outperformed their public or private school peers.

To date, I have not found any such study.

Playing the Telephone Game

When a child is excited about a new find--a toy--he tells all his friends how amazing the toy is. His friend then tells his other friend, ad infinitum. In the process, the toy turns from a mundane matchbox car to a remote-controlled tank. Each child heard elements of the original report he liked and did not hear other parts. Sit in a circle and play the telephone game by whispering into the next ear and similar results appear.

When an adult does the same thing, it is less appealing. And more dangerous. Homeschooling leaders that I am aware of have complained about burnout. And about low retention rates. This is what naturally follows when evidence is hyped out of proportion. The leaders certainly did not make up the numbers they heard; they probably heard it from a trusted friend, who heard it from another friend, ad infinitum.

With this posting, the telephone chain ends. There is no remote-controlled tank with a magical statistical bullet to silence the nay-sayers. Now the record is set straight. The numbers are not in and may never be discovered given the decentralized nature of homechooling.

Does this mean one should not homeschool? No, just realize that it takes work, hard organizational work. Then again, whatever schooling is used, proper parental nurture is hard work. Changing one type of schooling for another does not automatically make one's child an academic success--that it a combination of God-given talent, hard work and mercy from the Lord.

SDG

[more on Christian education and homeschooling here]