Inductive Reasoning 1: Generalizations

Introduction:

Suppose you want to figure out what percentage of US college students commute to school. One way would be to go to each university in the US and ask every individual university student if they commute. This will give you an accurate answer but by the time you get your final answer, 5 years will have passed and you will have spent a lot of money. There must be a better way…but how?

As you may have figured out, the best way to get our number is to ask a sample of US students if they commute. Then we will generalize from the sample to a conclusion about the total US student population.

We can think of a generalization as a two premise inductive argument. Supposing we use my critical thinking students as my sample, the argument will look like this:

Example:

(P1). The students in this class are a representative sample of BGSU undergraduates.
(P2). 30% of the students in this class commute to campus.
(C). Therefore, 30% of BGSU undergrads commute to campus.

All generalizations can be put into a standard form which looks like this:

(P1). S is a representative sample of population/group P.
(P2). Proportion 1 of S are Y (Y=the trait/attribute we're interested in).
(C). Proportion 2 of P are Y.

Before we learn to evaluate generalizations we need to learn some important terminology.

Key Terms
A sampling frame precisely defines the total population and the trait you want to study. The total population is called the target group/population. In my example, I want to know how many students in Ohio commute to school. Before I even begin to study my sample I need to specify my terms. In fancy talk, when you make your define your target population and property we say you operationalize your terms. It is essential to operationalize your terms in order to eliminate vagueness.

First, I'd need to operationalize "student." Do I include elementary, high school, and college? Do graduate students count? Until I specify, the term "student" is left vague. Once I specify, I have identified my target population. I also need to operationalize the property in question (sometimes called relevant property). This is the attribute or quality that the target population has. In our example, the property in question is "commutes to school." Does this include students driven by parents? Does it include students who ride bikes or skateboards? What about students who live walking distance. Do I count them as commuters? Once I specify, I have operationalized the trait that I want to study in the population. Depending on what I want to learn, I will have to make choices about how I define my sampling frame otherwise vagueness will make my results not very useful.

The sample is the subset of individuals from the total population you want to study. A sample should be representative of the population you wish to study. That is, you want to ensure that the relevant traits or variables in the target population are also represented proportionally in the sample. For example, suppose I want to study college students in Ohio. As my sample I use BGSU students to infer what proportion commute by car to school. Suppose it's 30%. Can I now reasonably infer that 30% of Ohio college students commute to school?

Probably not. Many BGSU students live on campus. That might not be true of other Ohio universities. Also, Bowling Green is a small town and most students who live off campus live within walking distance. These patterns may not be true of university students who live and go to school in Cleveland or Columbus. If I base my generalization of Ohio college students only on BGSU students, I will have a biased sample. A biased sample is one that doesn't have the same distribution of variables as the target population. A biased sample isn't representative of the target population and will therefore weaken the strength of my generalization.

In order to avoid having a biased sample, I use a random sample. A random sample uses a selection method to avoid bias: it ensures that every member of the total population has an equal chance of being included in the sample. The probability that a sample will be biased is called the sample margin of error. To minimize the margin of error, larger samples are preferred to smaller samples. Also — especially when studying humans — we want to use stratified random sampling. This involves identification of relevant population clusters and making sure that they are represented in the sample in the same proportion that they exist in the population. In short, we want diversity in our total population to be represented in our sample. For example, if I'm studying all Ohio college students I'll want to stratify across small town colleges and big city colleges as well as perhaps across various social groups and income levels.

Evaluating Generalizations

Basic Method: Essentially we're going to
(a) evaluate each premise (premise acceptability) as well as
(b) evaluate the strength of the logical force between the premises and the conclusion.

Let's use the sample argument from earlier to learn how to evaluate a generalization argument:

(P1). The students in this class are a representative sample of BGSU undergraduates.
(P2). 30% of the students in this class commute to campus.
(C). Therefore, 30% of BGSU undergrads commute to campus.

I. Evaluating Premise 1:

Premise 1 should always be evaluated in terms of two criteria:

A. Sample size: Is the sample large enough to both capture all the relevant diversity in the target group?

Part of our answer will have to do with what trait we're interested in studying. If we're interested in what proportion of students live on campus, asking my critical thinking class is too small a sample to capture all the diversity in the BGSU undergraduate population. Critical thinking is a 100 level class and so freshman will be over-represented. BGSU has a policy that requires freshmen to live on campus. If I take my sample only from freshman, I won't get information about other sub-groups with the student population, like seniors who probably are more likely to live off campus. So, even if I select my sample randomly, if it's too small it might not include all the relevant diversity in a population. The more diverse a target group, the larger you'll want a sample to be in order to ensure that the sample captures the target group's diversity. Generally, you don't need a sample larger than around 1200 — even for national-level polls.

When a sample is too small it commits a fallacy called hasty generalization. Hence, we cannot accept premise 1.

B. Representativeness: Does the sample group have the same relevant characteristics and distribution of relevant characteristics as the target group? Suppose to correct the fact that my sample is too small I randomly select ten intro level classes for my sample. Now my sample is about 1 000 students. Now, no one can accuse me of using too small a sample. Unfortunately my sample still won't allow me to make a good generalization since it isn't representative of the relevant diversity in the student population. Again, all the students in the sample are freshmen. First year students at BGSU are mostly required to live on-campus. Seniors, for example, are not. Since seniors aren't represented in our sample in the same proportion as they are in the target population we have a biased sample. Hence, we cannot accept premise 1.

A famous case of biased sample: For most of the history of academic psychology and social psychology, experiments are done on undergrads then generalized to all humans. This is especially true for studies about moral intuitions. For example, social psychologists and economists use something called the Ultimatum Game to test people's intuitions of fairness and willingness to punish those who don't abide by fairness norms.

The game works like this. There are two participants in the study. The researcher gives participant A something of value, perhaps $100. Participant A can choose to give any amount to participant B, from 0$ to all. Why give any at all? Here's the catch, if participant B rejects your offer neither of you get anything. In other words, participant A has an incentive to give an amount to B that isn't too low, otherwise they risk getting nothing themselves. It turns out that most proposers will off 30% to 50% and people in B's position will usually reject anything below around 30%. This study has been replicated many times all over the US. So, for decades economists and social psychologists inferred that these numbers represent basic human intuitions about fairness. Can you spot the problem?

It turns out that if you conduct Ultimatum Game experiments in non-Western countries you get completely different results! In some cultures people reject money if you offer too much! And some are happy to get nothing! The mistake social scientists made was that their samples — although large — were taken from only one culture. And within the context of the world population, Western culture is a minority! In fact, the results you get in Western culture are abnormal compared to the rest of the world. Social scientists and economists had built up large interconnected theories about human behavior that relied on an outlier sample. All those theories are now suspect and must be reevaluated in light of new internationally collected data. Many will be over-turned. To learn about some of the theories that are in trouble and differences between Western and other intuitions, here's the main article. .

Anecdotes: Anecdotes are personal experiences that people usually generalize from.
anecdotal%20evidence%20not%20valid.jpganecdotal%20onion%20.jpg

Key point: Generalizations that come from anecdotal evidence (personal experience) commit both errors. Can you think of why? Critical thinking lesson: Generalizations based on anecdotal evidence are bad arguments and should be rejected. They violate both the sample size the representativeness criteria required for good generalizations.

There are other ways that premise 1 can be unacceptable which we'll examine in the lesson on polling. For now, focus on sample size and representativeness since they are the most common and most important.

II. Evaluating Premise 2:

There are several ways Premise 2 can fail to be acceptable. Most of them we'll examine in the lesson on polling. For now, here are a few to look out for. I should add that evaluating Premise 2 is a much more difficult skill that evaluating Premise 1, and often requires background knowledge of the topic the generalization is about. All that means is that we need to start with baby steps!

C. Measurement Problems: Measurement problems occur in various ways. Here are two of the most common.

1. Measurement problem: Sometimes the way data is collected affects whether I'm actually measuring what I think I'm measuring (i.e., the property in question). Suppose the university prohibits researchers from interacting with students. I'm not allowed to simply conduct interviews so I'll have to collect my data indirectly by observing behavior. I collected my data in Premise 2 by tagging students' ears with a tracking device and tracking their movements from 9am-5pm.

My data shows that 50% of them went to on campus residences in this time window. From this I conclude that 50% actually live there. It turns out that 15% of them were just visiting their friends but didn't in fact live there. The way I measured the trait I'm interested (where students live) didn't exactly track the trait I'm interested in. As we will see with polling, measurement errors happen A LOT. It also occurs with causal generalizations which we'll study in detail next week.

2. False attribution: This is a type of measurement error. Sometimes two properties are closely correlated causing a researcher to confuse one for the other. For example, in education research it's fairly well-established that children from wealthy homes do better in school than students who come from families that are poor. So, I might conclude that wealthy students are better students than poor students because they're wealthy.

It turns out that wealthy parents tend to have higher levels of education themselves and also typically have more time to help their children. So, it's not that wealthy students are better than poor students, it's that students with parents that have the education and time to help their children do better. It just so happens the wealth, education, and time are closely correlated. The correct conclusion is that children of parents who have a high level of education and time to help do better than students who don't have either — not that having wealth makes a child a better student.

Bonus measurement problem: Attrition Often with human trials, participants will drop out over the course of the study. When you read a study it will say they had x number of participants. It might sound like decent study size. The problem is that people drop out. This not only affects the size of the sample but it might also affect representativeness, and consequentially whether we can trust the result. For example, in weight loss studies, a lot of people often drop out. This skews the results because it's not random who drops out. People who lost weight usually stay in while those for whom it didn't work, drop out. This makes the effect size seem even larger than it it really is.

For an example of the problem of attrition, watch this discussion of trying to compare different weight loss diets from 8:20.

III. Evaluating the Inference from Premise 2 to the Conclusion:

There are two main ways for the inference from Premise 2 to the conclusion can be weak. We'll only look at one here. The first is fairly obvious.

D. The proportion in the conclusion is greater than the proportion in the sample: If 50% of the students in my sample live on campus using that information only I can't infer that 60% of all BGSU students live on campus. In other words, the proportion in my conclusion can't be higher than it is in my sample (without additional justification). This may seem like a fairly obvious point and it doesn't usually happen within scientific studies but it almost always happens when scientific studies get reported by the media.

Rule of thumb: When you read of a scientific study that's reported online, assume the results of study has been misinterpreted and exaggerated.

Practice:

Put the arguments into the correct standard form then evaluate Premise 1 for sample size and representativeness. If the sample size is too small, identify it as a hasty generalization.

A
Informal Presentation: 20% of Ami's students own trucks therefore around 20% of university students own trucks.

Formal Structure
(P1) The students in Ami's class are representative of all university students.
(P2) 20% of the students in Ami's class are truck owners.
(C) Therefore around 20% of university students must own trucks.

B
75% of my friends at school have student loans therefore 75% of students at BGSU have student loans.

C
All my friends have happier and more interesting lives than mine. Every time I check my facebook feed, they're posting about doing something interesting or fun. My life sucks.

D
Conrad Hilton started out dirt poor and became super-rich, therefore anyone can do it.

E
We asked anyone who was motivated to lose weight to try our new magic diet of eating only natural organic pine tree bark. Over 80% of participants lost weight. 80% of people who try our new magic diet will lose weight.

E
Fox just did a call-in telephone poll of over 10 000 people and 80% of them agreed that Obama is doing a terrible job. That shows that around 80% of Americans think Obama's doing a horrible job.

F
If you're sick you should use this homeopathic remedy. It worked for me last time I was sick.

H
1/3 of students in two-year programs at Washington State community colleges graduate within 3 years. Therefore, about 30% of people in 2 year programs graduate within 3 years

G
Trolly Dilemma (Evaluate Premise 2)

(from 2:00) and fMRI: People who make the utilitarian choice are better moral reasoners. +What is the fMRI measuring?

Some Interesting Generalizations

Homelessness stats
Tipping

What property are we measuring?
Race and private prisons

Key Concepts

Structure of Generalizations
(P1) S is a sample of Xs.
(P2) Proportion 1 of Xs in S are Y.
(C) Proportion 2 of Xs are Y.

Unless otherwise stated, the content of this page is licensed under Creative Commons Attribution-ShareAlike 3.0 License