ON RESEARCH AND STATISTICS For most of human existence, much of what we believed was based on folklore and myth. As science progressed, mankind gradually learned to apply empirical knowledge toward our understanding of reality. We are now at the stage where humans are capable of using research and statistics to establish what is fact and what is fiction. This has not changed the fact that a great deal of what the majority believes is still folklore, since having the capacity for analytical study does not necessarily equate with practising sound scientific methodology.
We are living during an age where studies are released almost daily, and the media generally reports conclusions without providing background data. We are therefore presented with the information as if it were universally applicable, while lacking details that would allow us to judge the validity of the research.
Certain criteria must be followed when testing a theory or collecting statistics, otherwise there is a risk that researchers will be perpetuating a falsehood under the guise of science. There are four primary questions that need to be asked when determining the value of a research project: - Is it biased? - Is the data consistent? - Are the results valid? - Are the findings significant?
*** Bias is sometimes a problem. Research is usually funded by entities with a vested interest in the area chosen. Although most scientists, and institutes that conduct surveys, try to maintain a relationship independent of the people who pay the bills, there are those who succumb to pressure, and provide what the client wishes to see. There can be a positive aspect to this potential problem, for sometimes research yields results that are completely opposite to what one might expect from a particular organisation, and hence it becomes apparent that all the data must have been thoroughly double-checked prior to release. Of course, more often than not, a report that contradicts a specific mindset is simply buried.
Ample funding can also prevent partiality. Money not only allows for a larger sampling and adequate control groups, but grants others the ability to monitor the work. The personal prejudices of a given scientist can influence results, which is less likely to occur if someone is watching how their money is being spent.
Intentional bias is often evident in the wording of surveys. A religious organisation can ask a random sampling of people the question “are you in favour of killing babies?”, and use the results to report that the vast majority are against abortion. The same organisation could ask an honest question, yet only survey members of their congregation, and proclaim the same statistics. The first instance allows phrasing to prejudice results, which then permits the researchers to interpret the data from an obviously suspect perspective. The latter example creates a misrepresentation of public opinion, by choosing participants who are more likely to provide the desired response.
A study may be inadvertently skewed by a failure to include a broad and diverse sampling. Even research based on millions of test subjects will be biased if a disproportionate number are of one age group, race, culture, gender, or social class. Inferring that any test carried out on a million tonnes of sedimentary rock holds true for all minerals is likely a mistake. Measuring solar activity on the Sun may tell us more about that particular class of star, but not necessarily a great deal about the other stars in our galaxy.
*** Consistency in research requires that a study yield the same results each time it is repeated. To do this, it is crucial to perform every test using the same guidelines. If the procedures are sound, then the results should be consistent. Any source data that serves as a foundation for a study must also be verifiable, otherwise it can taint the larger experiment, and cause discrepancies from one trial to the next.
*** Valid research shows more than a connection or pattern, it proves cause and effect, and is not subject to a variety of conflicting interpretations. Would a study that reports, for instance, that people who eat crickets are more intelligent due to their eating habits, perhaps actually show that intelligent people tend to eat crickets?
The validity of results depend on how well we’ve structured the test. Are we comparing “apples to apples”? Have we set up a control group; that is, equal circumstances that differ only in the thing or situation we are analysing? Are our criteria or conclusions a matter of fact, or purely anecdotal?
*** A research project can be designed perfectly, and still be misinterpreted by people who do not understand scientific methodology. Results are evaluated using the term ‘statistically significant’. Variations of chance play the primary role in this determination, and a statistically insignificant conclusion means we have shown the status quo holds true, that there is no effect.
Random chance must be taken into account. Say you are monitoring a cross section of twenty thousand members of society; ten thousand who sing in the shower, and ten thousand in your control group who do not. Over your ten year project, one person dies in the control group, and two pass away in the other group. Are you justified in concluding that singing in the shower doubles your risk of death? Obviously not, for the different fatality rates between the two groups may very well be random.
Every study includes a margin of error. The larger the sample, the smaller this margin will be. If we determined that most people prefer the colour green by asking every human being what their favourite colour is, then there can be no mistaking the outcome of our research. This is a logistical impossibility, so we draw conclusions based on a representative sample. We cannot test every automobile for crash worthiness, nor look at every snowflake to determine that each one is unique. Almost all broad research projects essentially provide a likely conclusion, the probability of any given conclusion being true is relative to the size of the study, and every project less than all-inclusive still contains a margin of error, for it is always possible that the sample may not contain a genuine cross-section.
A great deal of effort has gone into creating guidelines that govern how chance affects statistics. Parameters are also in place to calculate the significance of results that indicate a variation that is greater than random chance, but less than what is deemed a meaningful or acceptable level. People die every day from a normal dose of Aspirin®, but no restrictions are placed on its sale because, compared to the large number of people who safely ingest the product, these hyper-sensitive individuals account for a statistically insignificant percentage. There are actually few substances in the world that are not potentially harmful to abnormally sensitive people, and practically no circumstance that has not killed the unlucky or unwise. However, such people are fundamentally victims of natural selection, rather than of dangerous substances and situations.
*** There exists a great many studies that have been used to mislead the public, as well as numerous ones that are partially or completely false due to flawed methodology. Often research that has long been abandoned by the scientific community lives on as “fact” in the minds of the public. It is fairly easy to find instances where valid studies have been misinterpreted, or imperfect ones perpetuated.
Recently, Dr. Simo Nyh released a statistical analysis of traffic accident fatalities in Finland. The data indicates that women have a sixty-three percent higher risk of dying on Friday the thirteenth, than on any other Friday. Males show a relatively minor two percent variance. Nyh’s study is founded on firm statistics, so provided the data pertaining to accidents was properly compiled by the government, the results are quite significant. He concludes that due to a highly superstitious nature, anxiety and fear lead to an impairment of mental and motor functions, which subsequently causes serious driving errors.
Nyh followed sound guidelines while preparing his report. The control group was drawn from fatality statistics on other Fridays, rather than all days of the week. A Friday creates a unique traffic environment, for drivers are often distracted by their eagerness to get home and start the weekend. His conclusion, albeit based on inductive rather than deductive logic, is the most reasonable explanation.
The only problem with this study was caused entirely by the media outlets that tried to apply the results too broadly. Until the research is repeated in other countries, it can only be said that women in Finland are obsessively superstitious. Cultural differences can be striking, even between neighbouring nations.
An interesting anomaly is evident in Nyh’s statistics. If women are crashing their automobiles in such a way as to cause fatal injuries, one would expect to see a more pronounced rise in the male death rate. It is rather unlikely that female drivers selectively hit other female motorists or stationary objects. Don’t they also collide with men? We can infer an answer to this quandary by considering other research that doesn’t immediately come to mind. It is a fact that in automobile accidents, the airbag is sometimes the sole cause of death, especially among women and children. A collision that has enough force to trigger the deployment of an airbag is far more likely to kill a woman than a man. A detailed analysis of the automobile accidents, rather than just fatalities, would confirm or deny this hypothesis.
*** Some research is virtually impossible to conduct flawlessly. An inability to completely control variables, or ethical restrictions placed by society, can render some studies merely educated opinions. Statistics concerning seatbelts are a good example. The materials used to construct automobiles are mass produced, and not subject to the rigorous examination of, for instance, aircraft parts. Hence, microscopic flaws in the structural components lead to unique failure sequences. This means that it is extremely unlikely that two test collisions, even with identical cars, will produce precisely the same damage.
Originally, crash tests were performed by professional drivers, and there were surprisingly few serious injuries. However, early automobiles were steel behemoths, and it would be far too dangerous to test the flimsy vehicles of today using human subjects. Therefore, modern trials are done with crash-test dummies, and although they have become increasingly sophisticated over time, they are not the same as humans, and provide data that requires subjective interpretation. As well, people react in some way, such as bracing themselves or raising their arms, while dummies are completely passive.
Some “real world” statistics have shown an increase in injuries from seatbelt use. The raw data source includes variables that are not tested in the lab. Many modern vehicles “insist” by design that you wear the seatbelt, older vehicles do not. People driving contemporary automobiles are therefore more likely to be restrained than those in larger earlier models. It is simply a matter of physics, even excluding the quality of materials, that bigger, more solid objects will fare better in a collision with their smaller, flimsier counterparts. Research has shown that automobiles are no exception.
The result is that we have both studies showing a decrease in injuries when drivers buckle up, and studies that show an increased risk of injury. Here is where we can see a bias in the way conflicting research is handled by the government. Being that the installation of seatbelts was imposed upon the auto industry without any sound evidence indicating that there was an advantage to doing so, administration is loath to admit the possibility of a mistake. Authorities responded to the studies, demonstrating seatbelts increased injuries, by stating that it was due to a reckless attitude motorist adopted when buckling up: drivers felt indestructible. Aside from the fact there is no proof this assumption is true, it is contrary to logic. “Reckless” individuals are more likely to be the people who do not wear seatbelts, and the vast majority of those who do so are motivated only by a need to conform to laws or convention.
Advocates of vehicle restraint systems sometimes point to the success of those used in professional racing, but this is an invalid comparison. Competition vehicles utilise a more complex harness to restrain the driver within an almost indestructible protective cage, and with the top racing categories, the automobile itself is designed, at considerable cost, to preserve the integrity of the cage in the event of an accident.
It may be that seatbelts protect you. It may be that seatbelts harm you. It is entirely possible that they have no significant effect either way. Because there is no consistency between studies, we can only conclude that there is no valid evidence that would lead us to support one viewpoint over the other.
*** Research pertaining to the effect of specific chemicals on humans is dependent upon animal trials. This is done partly because it is unethical to risk human lives, and primarily for expediency. Our life-span is too long to allow for continuous monitoring over multiple generations. Tests evaluating direct exposure can be done using higher primates, but determining long term risk usually involves rodents. Obviously, a creature such as a rat is far different from a human being.
Animal trials do not necessarily translate well to humans, and assumptions based on exact parallels may very well be wrong. Such a mistake was dramatically evident with the tests done for the drug Thalidomide®. Although animals appeared to suffer no ill effects from even extreme levels of exposure, Thalidomide® caused horrific birth defects in humans. Babies were born with “flippers” rather than limbs.
The critical error made during the Thalidomide® studies was a failure to seriously consider the implications of why the drug had no effect whatsoever on any animal subjects; there were no negative nor positive reactions. As it turned out, humans uniquely metabolise Thalidomide®, and all biological comparisons were subsequently invalid.
*** The propaganda disseminated regarding “second-hand smoke” demonstrates an unusual phenomenon, where what the public is told to believe is actually what scientifically valid studies have proven to be false.
The EPA report, an analysis of then-existing studies, is the most common source used by media and special-interest groups; a report not only dismissed by the scientific community and U.S. congress as invalid, but legally declared as such by the courts.[1] How can research go so wrong? The problem began when the EPA announced the conclusion they intended to reach, prior to completing the work. When the results ultimately proved their assumptions false, it became necessary to manipulate the data. Obviously, determining what you wish the facts to establish, rather than allowing the evidence to control the conclusion, is not science, but politics.
The EPA seemed doomed to fail from the start. Of thirty-three available studies, they chose thirty. Of these, only six indicated a small, yet statistically significant risk from second-hand smoke. Eighteen established that there is no danger, and six found exposure to be beneficial. Deciding to compile their data from only eleven selected studies that might support the EPA’s beliefs, the results indicated that when correct scientific methodology was applied, the final conclusion proved that there is no statistically significant effect.
Once again the EPA data had proven their presumptions wrong, so for the final report, they simply changed the standard scientific guidelines to create a result that, even then, suggested a barely significant correlation between tobacco smoke and health.
The World Health Organisation had embarked on a lengthy, tightly controlled, research project[2] prior to the release of the EPA report, meant to counter the preponderance of evidence indicating that second-hand smoke was harmless. In 1998 they had their answer: no statistically significant negative impact, and in fact children raised by parents who smoke are twenty-two percent less likely to develop lung cancer[3]. It is to WHO’s credit that they only tried to hide the fact that the study ever took place, rather than falsify the data; but too many people had been aware of such an investment of time and money. WHO was eventually forced to put out a press release, which is where they took the opportunity to lie about the findings. The integrity of the participating scientists, however, could not be compromised, so ultimately the truth did come out.
Why did the WHO study, and others, indicate that second-hand smoke has a beneficial effect? It may relate to recent theories pertaining to health problems specific to Western nations, and particularly the United States. For instance, although adult allergies appear to be primarily psychosomatic, there is a suspicion that a physiological component exists as well. Because of Western cultural paranoia over germs and environmental chemicals, our obsession with “cleanliness” may actually be damaging our children. Just as childhood exposure to Mumps prevents potentially serious problems later in life, a developing immune system may benefit from early encounters with minor forms of bacteria, viruses, and airborne particulates. Without the resistance acquired in youth, the human immune system may fail to cope later in life.
*** No data is more volatile than that which addresses human sexual behaviour. Although the mechanics of mating, and much of the courtship ritual, are innate, social influences play a major role in people’s attitudes and actions. It is important to differentiate between instinctive, and therefore common, behaviour; and that which is specific to particular social environments. Frequently, the media assumes that a survey of the sexual habits of one group should apply to everyone. Sometimes researchers are eager to interpret their data too liberally.
Until relatively recently, social attitudes made it difficult to conduct studies on matters relating to human sexuality. Christianity in particular did not wish to see the evidence that humans are far more promiscuous, and obsessed with things sexual, than doctrine indicated. Therefore early researchers were hampered by a lack of funding, and frequently could not afford to ensure adequate control procedures were put in place. As well, without organisations serving as benefactors, there was nobody overseeing the work of lone scientists, and information influenced by personal bias or agenda was sometimes disseminated as being scientifically valid.
The Kinsey study is the most famous of these early projects. Conducted using methodology that is laughable by any standard, corrupted by Kinsey’s own personal sexual preferences, and strangely plagued by media reports quoting statistics that appear nowhere in the study; it stands as an example of how “junk science” can have a lingering influence. Even now, there are those who parrot the erroneous statistics that were attributed to the Kinsey report, while being completely unaware of the source.
Much further back, renowned psychologist John Watson sacrificed his credibility over a “project” he was working on. Apparently collecting data on female responses to sexual intercourse, he served as half of the test group, his female assistant being the other half. Whether or not this was an actual study seems to have been in question. At the divorce proceedings, Mr. Watson insisted it was, while his soon-to-be ex-wife and the judge both felt otherwise. Regardless of how much data was actually collected, this “research” primarily represents John Watson’s particular skills, and the sexual responses of the woman who turned out to be his next wife.
Much of the data used in contemporary surveys on sexual behaviour comes from companies with a financial interest in sex, such as condom manufacturers and magazines dealing with pornography or “lifestyle”. According to these sources, a large percentage of the population is engaging in frequent, exotic sexual intercourse with a wide variety of partners, and having paraphiliac fantasies.
In truth, surveys conducted by the sex industry reflect the activities and values of their customers, and only those who proactively participate by responding to questionnaires. Condom manufacturers, for example, have a clientele that encompasses the people who engage in risky sexual activity. Prostitutes account for a considerable portion of the prophylactic market. People who do not normally purchase the product, the majority of the adult population who are in stable relationships, are left out of the statistics.
Polls done by the sex industry may give us a somewhat distorted indication of the behaviour of consumers of certain products, but the data does not represent the lifestyle of your typical individual. These statistics would normally be of secondary importance, but the media has created a problem via their method of reporting. Often the source of a survey is noted, but never is the story prefaced with phrases like “twenty percent of brand-loyal condom users are...” or “fifty percent of Playboy® readers practice...”. It is often inferred by this omission of a qualifying statement that the figures apply universally. The average person desperately tries to conform to the herd, and many get the impression that there is something wrong with their sex life, when they see or hear erroneous generalisations in the media. Such carelessness by the media can create depression and self-doubt in individuals who already have an abundance of sexual insecurities about how they measure up to others. Adolescents may acquire an unrealistic perception of what “normal” sexual thoughts and actions are, and mimic behaviour that, in reality, represents only a fraction of the population.
*** Science has been of considerable benefit to humanity, and gradually we have come to understand a great deal about ourselves and the world around us. However, science is not, and never has been, a steady progression toward omniscience. Rather, it entails frequent misdirection, and at many times throughout recorded history, we have built our beliefs upon a foundation of falsehoods. This is not to suggest that contemporary research is wrong very often, but because knowledge is so frequently cumulative, where earlier data contributes to each successive stage of research, one error can skew the work of many.
Our modern technology provides us with the opportunity to evaluate studies as never before. Unlike the past, even the common citizen has access to details on how specific statistics were compiled, and what methodology was used to conduct experiments. No longer are the “peasants” limited to faith alone, forced to blindly accept the conclusions of our “superiors”. Of course, verifying the truth requires that we make the effort. Most people will not, but then, most people do not contemplate anything beyond their simplest needs and desires. The fact that the opportunity exists is enough; for those who are destined to believe as they are told will never question, regardless of what knowledge is available, while those who need to seek the truth will take advantage of such opportunities.
The accessibility of information is a two-edged sword: meaning that technology enables us to see for ourselves that the reasoning of others is sound, yet at the same time it enables misinformation to spread rapidly throughout the world. We must accept the fact that the majority will sometimes believe in the truth, and sometimes trust in a lie. Each of us chooses how we wish to perceive the world. Most are content to rely on faith, but those who cannot are the ones who create that which the masses believe in. Without the people who require that things be proven true, and who are willing to think contrary to the beliefs of the herd, knowledge would rarely have advanced. The greatest scientific minds that have ever existed belonged to those who did not trust in “common” knowledge. We owe everything to the individuals who questioned what was held to be right. Even when they were the lone voice in the crowd, their commitment to truth prevailed, and mankind moved forward.
[1] Copies of the EPA report are available to U.S. citizens for free. Phone (800) 438-4318 - document EPA/600/6-90/006F - "Respiratory Health Effects of Passive Smoking: Lung Cancer and Other Disorders"
The Congressional Research Service's criticism of the EPA's methodology can be read here.
Legal decision dismissing the EPA report as null and void can be read here.[3] Using standard scientific criteria, the beneficial effect from second-hand smoke reported in the WHO study could range from a low of 4 percent of children being less likely to get cancer, to a high of 36 percent less likely.
