Unit 3 Lesson 5

Sources of Bias in Experiments and Sampling

Unit 3, Topic 3.8: Sources of Bias in Experiments and Sampling

Overview

This lesson examines sources of bias in experiments, such as selection bias (unequal group assignment that favors one outcome) and placebo effect (perceived improvements from inactive treatments due to expectation), as well as sampling biases revisited from earlier (undercoverage, nonresponse, voluntary response, and response bias), which distort how well results represent the population. Bias creeps in when designs overlook fairness, like assigning healthier patients to a drug group, leading to overstated effects. Example: In a memory supplement trial, selection bias occurs if only sharp-minded volunteers are chosen for the treatment group, inflating success unlike random assignment.

Evaluating bias impact reveals how it warps conclusions and limits generalizability, for instance, nonresponse from skeptical participants skewing poll results toward optimists, making findings unreliable for the full group. Understanding this helps suggest fixes like better recruitment to restore balance. Example: Voluntary response in a job satisfaction survey from company email biases toward vocal employees, narrowing generalizability to quiet workers and weakening claims about overall morale.

Ethical considerations, like obtaining informed consent and ensuring equitable access, are crucial in data collection to protect participants and maintain trust, preventing harm from undisclosed risks in trials. Example: In a fitness app study tracking heart rates, skipping consent violates privacy ethics, potentially exposing users to data breaches and eroding confidence in health research.

Assignment:

Part 1: Guided Practice Activity

Consider the example of a clinical trial for a new pain reliever, testing it on 100 patients with chronic back pain divided into drug and placebo groups. Follow the tasks below to identify biases and evaluate their effects.

Example Scenario: Patients self-report pain levels before and after 4 weeks, but recruitment favors those with milder symptoms.

Tasks:

• Identifying Sources of Bias:
  • Identify and describe two biases in experiments (e.g., selection bias, placebo effect) and two from sampling (e.g., undercoverage, nonresponse), with ties to the trial. Example: Selection bias happens when healthier patients are chosen for the drug group, favoring better outcomes; placebo effect occurs when patients improve from believing in the fake pill, mimicking real drug benefits. Undercoverage misses severe pain sufferers if only clinic patients are recruited; nonresponse from discouraged dropouts skews toward success stories.
  • Extra Practice: For a school lunch preference survey, identify two experiment biases and two sampling biases.
• Evaluating Impact:
  • Evaluate how one experiment bias and one sampling bias affect conclusions and generalizability in the trial. Example: Selection bias overstates the drug's effectiveness by 20-30% (milder cases recover faster), limiting generalizability to chronic sufferers; nonresponse underrepresents failures (dropouts have worse pain), biasing conclusions toward success and poor application to all patients.
  • Extra Practice: For your lunch survey, evaluate impacts of two biases.

Case Study Analysis:

• Analyze the flawed trial case (e.g., selection led to 80% "success," but only mild cases included) and suggest 2-3 improvements. Example: The 80% success seems high but flaws from selection bias ignore severe cases; improvements include random assignment from all pain levels and follow-up calls to reduce nonresponse, plus consent forms for ethics.
• Write 2-3 sentences discussing ethical considerations (e.g., informed consent). Example: Informed consent ensures patients know risks like side effects, building trust; without it, the trial risks harm and invalid results, especially for vulnerable chronic pain groups.

Part 2: Independent Practice

Examine a case study on a weight loss app trial with 200 users, where app users (treatment) report 10% loss vs. controls at 2%, but recruitment was via fitness forums (voluntary response).

Tasks:

• Identify two experiment biases (e.g., placebo effect) and two sampling biases (e.g., voluntary response), explaining their sources.
• Evaluate how these biases impact conclusions (e.g., overstated loss) and generalizability (e.g., only fit users).
• Suggest 2-3 improvements to the design.
• Write 2-3 sentences on ethical considerations, like consent and equity.
• Extra Activity: Invent a biased study (e.g., social media mood survey). Identify biases, evaluate impacts, and suggest fixes.