Overview

  • The Methods section describes how the study was conducted so that it can be replicated or reproduced in future studies
  • Includes study design (see “Types of research studies” section below)
  • Details how exposures and outcomes were defined and measured, such as tools used to measure the results
  • Provides information about how participants were recruited (see “Participant recruitment” section below)
  • Explains statistical analysis used (see “Results” section)

Types of research studies

  • Often, the goal of clinical research studies is to assess associations, especially cause and effect
    • Does X cause Y? Can drug X treat disease Y?
    • X is the independent variable or exposure
    • Y is the dependent variable or outcome (Y depends on X)
  • To truly determine cause and effect, an experiment is necessary
    • One group of people with the exposure and another group of people without the exposure (control group) are compared to determine the effect on an outcome
  • However, it is often difficult to evaluate cause and effect in clinical research due to a variety of obstacles (for example, cost, time intensive, ethics)
  • Therefore, there are different types of studies with different levels of difficulty to conduct and consequently different levels of quality for assessing cause and effect

Hierarchy of evidence for clinical research studies. Generally, bottom is lowest quality and top is highest quality.

  • Causal inference
    • Hill’s criteria for causation are a set of guidelines for determining the strength of cause and effect in research studies
      • Strength: large association/effect size
      • Consistency: reproducible results across different people and places
      • Specificity: specific connection between cause and effect, with no other likely explanations
      • Temporality: cause came before effect
      • Biological gradient: dose-response relationship; changing the dose of exposure changes the effect
      • Plausibility: plausible mechanism explaining the cause and effect
      • Coherence: coherence between laboratory and epidemiological findings
      • Experiment: presence of experimental evidence
      • Analogy: similar/analogous associations are found
      • Reversibility: removing the cause should remove the effect
    • Studies meeting few of the criteria have weak support for causality and can only state that there is an “association” (which is a more general term than “causality”)
  • Two main types of studies: observational and experimental
    • In general, observational studies are lower quality than experimental studies
  • Observational studies: the independent variable/exposure is not under the control of the researcher; also called “quasi-experiments”
    • General strengths
      • Often less expensive than true experiments
      • Relatively easy to acquire data (for example, review of electronic medical record, aka “chart review”)
      • May be the only feasible or ethical type of study (for example, exposure to disease, surgical intervention, too much time between exposure and outcome)
    • General weaknesses
      • More vulnerable to biases, which are systematic errors that lead to an incorrect estimate of the association between exposure and outcome
        • Selection bias: sample of participants studied is not representative of the target population studied
        • Recall bias: some participants may be more likely to remember an event/exposure correctly than others
      • More vulnerable to confounding, which is a third factor/variable that explains the association between exposure and outcome; in other words, the association between the exposure and outcome is actually due to a third variable
        • Confounding can be controlled at different study stages
          • Study design: randomization (see “Experimental studies” section below); restriction (see “Participant recruitment” section below)
          • Study analysis: stratification; propensity matching; regression (see “Results” section)
      • Some observational studies do not have a comparison group
    • Within observational studies, the order from lowest to highest quality of evidence is generally: case report/series, cross-sectional study, case-control study, and cohort study
    • Qualitative study: nonnumerical, such as interviews and descriptions of subjective experiences
    • Case report or case series: follows few individuals who had a similar exposure, usually unique or interesting cases
    • Ecological study: compares populations, not individuals
      • Weakness: ecological fallacy – conclusions about populations cannot be assumed to apply to individuals
    • Cross-sectional study: assesses exposure and outcome of individuals at one point in time (a snapshot)
      • Weakness: don’t know whether exposure or outcome came first due to lack of time between them (did X cause Y? Or Y cause X?)
Example of a fictional cross-sectional study.
  • Case-control study: groups of people with and without an outcome are assessed for prior exposure status (a previous point in time; always retrospective in design)
    • Strengths
      • Useful for rare outcomes
      • Can assess multiple exposures
    • Weaknesses
      • Can only assess one outcome
      • Results may be affected by how exposure and outcome are defined, which may introduce selection bias
      • Vulnerable to recall bias
Example of a fictional case-control study.
  • Cohort study: longitudinal study that follows groups of people with and without the exposure to an outcome over time
    • Can be “retrospective” (review medical record over period of time in the past) or “prospective” (observe people over time starting now)
      • “Retrospective” only means “in the past” in this sense and is different from case-control studies; cohort studies are always prospective in design
    • Strengths
      • Useful for rare exposures
      • Can assess multiple outcomes
      • Appropriate temporal relationship; exposure before outcome
      • Exposures and outcomes can be clearly assessed
    • Weaknesses
      • Can only assess one exposure
      • Results may be affected by how exposure and outcome are defined, which may introduce selection bias
      • Loss to participant follow-up over time, which may also introduce selection bias
      • May need to wait a very long time for an outcome to develop
Example of a fictional cohort study.
  • Experimental studies (clinical trials): test whether an exposure (specifically, treatment or intervention) is effective in improving an outcome (for example, a disease)
    • Gold standard is the prospective, double-blind, placebo-controlled, randomized controlled trial (RCT)
    • Prospective: timing of cause before effect
    • Double-blind: neither participant nor researcher knows who is receiving treatment or not; preserves integrity of outcome
      • Single-blind: when the researcher, but not the participant, knows who is receiving treatment or not
    • Placebo-controlled: placebo is a treatment with no therapeutic value, used as a control/comparison group
      • Purpose of placebo is to determine that therapeutic value is truly due to the treatment
      • Not ethical to use placebo if there is already a known effective treatment for a disease; in this case, the control group will use the standard, currently approved treatment
    • Randomized: randomly assign to treatment or control group to reduce bias and confounding
      • Allocation concealment: hides which group (treatment or control) participants are assigned to; preserves randomization
      • Intention to treat (ITT) analysis: analyzes results based on which group participants were initially randomized/assigned to, not on actual treatment received; more conservative results
        • If a participant was assigned to the placebo group but later began taking the treatment, then in ITT, the participant will continue to be analyzed as part of the placebo group
        • Simulates real-world more closely; people sometimes stop taking or change medications
      • Per protocol (PP) analysis: only analyzes participants who completed the treatment based on the group they were initially assigned to; less conservative results (ideal world or best-case scenario)
        • If a participant was assigned to the placebo group but later began taking the treatment, then in PP, the participant will be omitted from the analysis
    • Strengths
      • High internal validity: strong cause and effect relationship within the study
      • Minimizes biases and confounding
    • Weaknesses
      • Relatively expensive
      • Not always ethical to conduct and may not be approved by the Institutional Review Board
      • Hard to determine endpoint (when to end study); endpoint should be decided before starting the study to minimize bias
      • Not necessarily generalizable to other studies or general population (external validity); in other words, results from the study might not be applicable in other contexts with participants from other backgrounds
    • Beware of subgroup or post hoc analyses on RCTs: these analyses are more vulnerable to biases because they were decided after the study was designed/performed
    • Many clinical trials are not RCTs: for example, some clinical trials are not randomized and some only have a single arm (no control group); these are weaker studies
    • Phases of clinical trials
      • Phase 1
        • Small sample size (20-100)
        • Goal is to evaluate the drug’s safety, side effects, optimal dosing, and formulation
      • Phase 2
        • Medium sample size (50-300)
        • Goal is to evaluate the drug’s biological activity or effect
      • Phase 3
        • Large sample size (300-3000)
        • Goal is to evaluate the drug’s efficacy
      • Phase 4
        • Post market surveillance; after FDA approval
        • Goal is to evaluate the drug’s long-term safety and efficacy
Example of a fictional clinical trial.
  • Other common studies
    • Systematic reviews and meta-analyses
      • They are research about existing research
      • Systematic review: a review of the literature that uses a well-defined, reproducible method of searching for published articles
        • Specific search terms in large public databases (for example, PubMed) are used to collect all the relevant published articles on a topic
        • The relevant articles are analyzed and summarized to provide more robust conclusions
        • Not all review articles are systematic reviews
          • Narrative reviews (often just called “review”) are articles that summarize a topic; they do not use a systematic approach to search for published articles
      • Meta-analysis: basically a quantitative version of the systematic review
        • Numerical results are extracted from multiple research studies and combined using statistical techniques to reach stronger conclusions
        • The larger sample size and greater diversity from combining studies are advantages
      • Pay attention to when the systematic review/meta-analysis was published
        • New research studies are published all the time, which may make the systematic review/meta-analysis outdated (although it may still be useful)

Participant recruitment

  • How many participants were recruited?
    • Larger sample sizes have more statistical power, which is the ability to correctly reject the null hypothesis (see “Results” section)
  • How were participants recruited into the study?
    • Recruitment strategy may introduce selection bias (for example, recruiting people who are healthier or sicker than the general population)
  • Which participants were included or excluded from entering the study?
    • This is the inclusion and exclusion criteria, which restricts the study sample and can minimize confounding
    • However, at the expense of minimizing confounding, the restricted sample may no longer be as generalizable to the general population

Previous page: Abstract and Introduction

Next page: Results