How researchers decide what counts as evidence before analysis begins

1. Intuitive Introduction

One of the most common design errors in research does not happen in the analysis. It happens earlier, when the researcher decides what evidence to collect. A study may begin with a sensible question, a plausible design, and even a competent method section, yet still weaken itself by building the project around whatever participants, cases, sites, records, or datasets are easiest to access. Booth et al. emphasize that good research moves from problem to claim through disciplined reasoning, not through opportunistic accumulation of whatever material happens to be nearby.

That is why this blog post joins three topics that are often taught separately: sampling, case selection, and evidence planning. On the surface, they can look like different issues. Sampling is usually discussed in quantitative work. Case selection is often treated as a qualitative or case-study concern. Evidence planning sounds broader and more abstract. But all three belong together because they answer the same prior question: what kind of evidence would count as a credible answer to this research question? Creswell and Creswell’s Research Design places design coherence at the center of empirical inquiry, which is exactly the principle at stake here.

2. Why These Three Issues Belong Together

Sampling is about how units are selected from a broader population or frame. Case selection is about why these cases, sites, events, or contexts were chosen rather than others. Evidence planning is the wider design activity that asks what participants, documents, sites, measures, time periods, or records the study actually needs before any collection begins.

These are not independent tasks. A researcher who plans evidence well will usually choose a sampling strategy or case-selection strategy that fits the question’s purpose. A researcher who does not plan evidence well often starts from access instead: the nearby school, the available administrative dataset, the easy-to-recruit students, the one site already known to the researcher, or the records that happen to be open rather than the records the question truly requires. Bornstein, Jager, and Putnick argue that sampling decisions have far-reaching consequences for what a study can legitimately infer, while Seawright and Gerring show that case selection is itself a strategic design choice rather than an afterthought.

3. The Main Design Logic of This Post

The dominant logic here is D > RQ > M. Data come first because the most visible failure usually appears in the evidence actually collected. Research question comes second because weak evidence planning often means the evidence no longer matches the scope or purpose of the question. Methodology comes third because the method may still be competently executed, but it is now operating on a weak evidentiary foundation.

This ranking matters because it shifts attention away from technical performance alone. A polished interview protocol, a careful questionnaire, or a competent statistical model cannot compensate for evidence selected without design logic. Bornstein et al. make this point for sampling, and Palinkas et al. make it for purposive selection in qualitative and mixed-method research: selection decisions must be aligned with study aims rather than left vague, habitual, or implicit.

4. Sampling: What It Is Really For

In quantitative work, sampling is often discussed in technical terms, but the deeper issue is inferential fit. A sample is not simply a set of respondents. It is the bridge between the collected evidence and the claims the researcher hopes to make. If the study aims at broader population-level inference, the sampling logic needs to reflect that aim. Bornstein et al. review several sampling strategies and show clearly that different strategies support different levels and kinds of generalization.

This does not mean every study must use probability sampling. It means that the inferential scope of the study must match the logic of selection. A convenience sample may still support exploratory, local, or bounded claims, but it should not quietly be treated as though it supports broader population conclusions. That is where many design problems begin.

5. Case Selection: More Than “Why These Cases?”

Case selection is often most visible in qualitative, historical, comparative, and case-study research, but its logic matters across designs. The key question is not simply whether the chosen case is interesting. It is why this case is analytically useful for the research question. Seawright and Gerring describe several case-selection options, including typical, diverse, extreme, deviant, influential, most-similar, and most-different cases, each linked to different analytic goals.

This is important because case selection is frequently rationalized after the fact. A researcher gains access to one site or one organization and only later constructs an argument for why that site “represents” something larger. Good design works in the opposite direction. The researcher first decides what kind of case would be informative, then explains why the selected case fits that role.

6. Evidence Planning: The Larger Design Task

Evidence planning is the umbrella activity that comes before recruitment, fieldwork, or extraction of data. It asks questions such as: What kind of evidence would count as a credible answer? How broad does the evidentiary base need to be? Is the study aiming at breadth, depth, contrast, mechanism, transferability, or contextual understanding? Which units matter: people, sites, records, interactions, organizations, events, or time periods?

This broader perspective matters because not every study is primarily a sampling problem. Some are evidence-planning problems in a wider sense. A study may not need a representative sample, but it still needs a defensible rationale for selecting records, cases, periods, settings, or respondents. Creswell and Creswell emphasize that research design requires coherence from question to evidence, and Maxwell’s Qualitative Research Design similarly treats design as an interactive set of decisions rather than a sequence of isolated techniques.

7. Common Design Failures

The first common failure is sampling without inferential discipline. The researcher uses a narrow convenience sample but writes as if the study supports claims about a far wider population. Bornstein et al. show why such mismatches matter: the selection logic constrains the scope of what can responsibly be concluded.

The second common failure is case selection without analytic justification. A site, event, or organization is chosen because it is available, already familiar, or easy to access, and only later described as strategically informative. Seawright and Gerring’s framework is useful precisely because it forces researchers to say what kind of case they are choosing and why.

The third common failure is convenience evidence disguised as design. Etikan, Musa, and Alkassim compare convenience and purposive sampling and note the practical appeal of convenience strategies, but practical ease is not the same as methodological justification. Convenience is sometimes unavoidable. It is not, by itself, an evidentiary argument.

8. How to Build a Strong Evidence Strategy

A strong design usually begins with a sequence like this:

research question → inferential goal → evidence needed → selection logic

First, clarify the question. Second, define what kind of inference the study is aiming at: broad generalization, bounded comparison, mechanism, contextual understanding, transferability, or something else. Third, identify what kind of evidence would make such a claim credible. Fourth, choose a sampling or case-selection strategy that fits that evidentiary need.

Palinkas et al. are especially useful here because they show that purposive selection should be matched to study aims rather than treated as a vague qualitative default. Their discussion of purposeful sampling in mixed-method implementation research is broader than its title might suggest: it offers a language for linking selection strategy to research purpose.

9. What Can Be Fixed Later, and What Usually Cannot

Weak evidence planning is sometimes partly repairable after data collection, but usually only by narrowing the claim. A convenience sample may still support a transparent exploratory paper. A narrow site selection may still support a bounded case analysis. A weakly justified sample may still yield useful local findings if the researcher stops pretending the evidence reaches further than it does. What usually cannot be repaired is the gap between broad original claims and weak original evidence planning. No later methodological sophistication can make a convenience sample representative in retrospect, or turn an arbitrary case choice into a strategically chosen case after the fact. That is why evidence planning belongs early in design thinking, not late in discussion writing.

Mini-Checklist

Before collecting data, ask:

• What exactly is my evidence supposed to stand for?
• Am I selecting participants, sites, or records because they fit the research purpose, or because they are easy to access?
• If access is shaping the design, have I narrowed the claim accordingly?
• Can I explain in plain language why these units are the right evidence?
• Does my planned conclusion match the scope of my evidence rather than the ambition of my topic?

Good research is not only about collecting data. It is about collecting the right kind of evidence for the claim the study wants to make.

References

Booth, W. C., Colomb, G. G., Williams, J. M., Bizup, J., & FitzGerald, W. T. (2024). The craft of research (5th ed.). University of Chicago Press. https://doi.org/10.7208/chicago/9780226826660.001.0001

Bornstein, M. H., Jager, J., & Putnick, D. L. (2013). Sampling in developmental science: Situations, shortcomings, solutions, and standards. Developmental Review, 33(4), 357–370. https://doi.org/10.1016/j.dr.2013.08.003

Creswell, J. W., & Creswell, J. D. (2023). Research design: Qualitative, quantitative, and mixed methods approaches (6th ed.). SAGE.

Etikan, I., Musa, S. A., & Alkassim, R. S. (2016). Comparison of convenience sampling and purposive sampling. American Journal of Theoretical and Applied Statistics, 5(1), 1–4. https://doi.org/10.11648/j.ajtas.20160501.11

Maxwell, J. A. (2013). Qualitative research design: An interactive approach (3rd ed.). SAGE.

Palinkas, L. A., Horwitz, S. M., Green, C. A., Wisdom, J. P., Duan, N., & Hoagwood, K. (2015). Purposeful sampling for qualitative data collection and analysis in mixed method implementation research. Administration and Policy in Mental Health and Mental Health Services Research, 42(5), 533–544. https://doi.org/10.1007/s10488-013-0528-y

Seawright, J., & Gerring, J. (2008). Case selection techniques in case study research: A menu of qualitative and quantitative options. Political Research Quarterly, 61(2), 294–308. https://doi.org/10.1177/1065912907313077