1 - Hulley et al. (2013) H3 Choosing the study subjects: specification, sampling, and recruitment
• A good choice of study subjects serves the vital purpose of ensuring that the findings in the study accurately represent
what is going on in the population of interest. The protocol must specify a sample of subjects that can be studied at an
acceptable cost in time and money (i.e. modest in size and convenient to access), yet large enough to control random error
and representative enough to allow generalizing study findings to populations of interest.
- Generalizability is a complex qualitative judgment that depends on the investigator’s choice of population and of
sampling design.
Basic terms and concepts
Populations and samples
• A population is a complete set of people with specified
characteristics, and a sample is a subset of the population.
- Clinical and demographic characteristics define the
target population, the large set of people throughout
the world to which the results may be generalized.
- The accessible population is a geographically and
temporally defined subset of the target population that is
available for study.
- The intended study sample is the subset of the accessible
population that the investigator seeks to include in the
study.
- The actual study sample is the group of subjects that does
participate in the study.
Generalizing the study findings
• The classic Framingham Study was an early approach to
scientifically designing a study to allow inferences from findings
observed in a sample to be applied to a population.
• Respondents are often healthier than nonrespondents, especially if
they are volunteers, the characteristics of the actual sample are
therefore undoubtedly different from those of the intended sample.
• Every sample has some errors and the issue is how much damage has
been done.
Steps in designing the protocol for acquiring study subjects
• An investigator begins by specifying the clinical and demographic
characteristics of the target population that will serve the research
question well. She then uses geographic and temporal criteria to specify
a study sample that is representative and practical.
Selection criteria
• A researcher begins by creating selection criteria that define the population to be studied.
Establishing selection criteria
• Inclusion criteria define the main characteristics of the target population that pertain to the research question.
● Demographic characteristics → age, gender, race
● Clinical characteristics →good general health, sexual partner
● Geographic characteristics →Patients attend clinic at hospital
● Temporal characteristics →specified period when study is conducted
• Effect modification = An effect in one factor that is different from that in other factors, also known as “an interaction”
, ● A large sample is needed, and most studies are not powered to detect effect modification.
• Exclusion criteria specify subsets of the population that will not be studied because of:
● A high likelihood of being lost to follow-up (e.g. alcoholic, plans to move)
● An inability to provide good data (e.g. language barrier)
● Being at high risk of possible adverse effects (e.g. history of stroke)
• Clinical trials differ from observational studies in being more likely to have exclusions mandated by concern for the safety
of an intervention in certain patients; e.g. the use of drugs in pregnant women.
● A good general rule that keeps things simple and preserves the number of potential study subjects is to have as few
as exclusion criteria as possible.
Sampling
Nonprobability samples
• In clinical research the study sample is often made up of people who meet the entry criteria and are easily accessible to
the investigator. This is termed a convenience sample. It has obvious advantages in cost and logistics.
• A consecutive sample can minimize volunteerism and other selection biases by consecutively selecting subjects who meet
the entry criteria. This approach is especially desirable, for example, when it amounts to taking the entire accessible
population over a long enough period to include seasonal variations or other temporal changes that are important to the
research question.
Probability samples
• Probability sampling, the gold standard for ensuring generalizability, uses a random process to guarantee that each unit of
the population has a specified chance of being included in the sample.
• A simple random sample is drawn by listing all the people in the population from which the sample will be drawn, and
selecting a subset at random.
● The most common use of this approach in clinical research is when the investigator wishes to select a
representative subset from a population that is larger than she needs.
• A systematic sample resembles a simple random sample in the first step, enumerating the population, but differs in that
the sample is selected by a preordained periodic process (e.g., the Framingham approach of taking the first two out of every
three families from a list of town families ordered by address).
● Systematic sampling is susceptible to errors caused by natural periodicities in the population, and it allows the
investigator to predict and perhaps manipulate those who will be in the sample. It offers no logistic advantages over
simple random sampling, and in clinical research it is rarely a better choice.
• A stratified random sample begins by dividing the population into subgroups according to characteristics such as sex or
race, and taking a random sample from each of these “strata.” The Stratified subsamples can be weighted to draw
disproportionately from subgroups that are less common in the population but of special interest to the investigator.
• A cluster sample is a random sample of natural groupings (clusters) of individuals in the population. Cluster sampling is
useful when the population is widely dispersed and it is impractical to list and sample from all its elements (hospital
example during lecture).
● A disadvantage of cluster sampling is the fact that naturally occurring groups are often more homogeneous for the
variables of interest than the population. This means that the effective sample size (after adjusting for
within-cluster uniformity) will be somewhat smaller than the number of subjects, and that statistical analysis must
take the clustering into account.
Summarizing the sampling design options
• The use of descriptive statistics and tests of statistical significance to draw inferences about the population from
observations in the study sample is based on the assumption that a probability sample has been used. But in clinical
research a random sample of the whole target population is almost never possible. Convenience sampling, preferably with a
consecutive design, is a practical approach that is often suitable.
,Recruitment
The goals of recruitment
• There are two main goals:
(1) to recruit a sample that adequately represents the target population, minimizing the prospect of getting the wrong
answer to the research question due to systematic error (bias); and
(2) to recruit a sufficient sample size to minimize the prospect of getting the wrong answer due to random error (chance).
Achieving a representative sample
• A particular concern, especially for descriptive studies, is the problem of nonresponse. The proportion of subjects
selected for the study who consent to be enrolled (the response rate) influences the validity of inferring that the enrolled
sample represents the population. People who are difficult to reach and those who refuse to participate once they are
contacted tend to be different from people who do enroll.
Recruiting sufficient numbers of subjects
• Falling short in the rate of recruitment is one of the commonest problems in clinical research.
● The approaches to this problem are to estimate the magnitude of the recruitment problem empirically with a
pretest, to plan the study with an accessible population that is larger than believed necessary, and to make
contingency plans should the need arise for additional subjects.
Summary
1. Most clinical research is based, philosophically and practically, on the use of a sample to represent a population.
2. The advantage of sampling is efficiency: It allows the investigator to draw inferences about a large population by
examining a subset at relatively small cost in time and effort. The disadvantage is the sources of error it introduces: If the
sample is not sufficiently representative for the research question at hand the findings may not generalize well to the target
population, and if it is not large enough the findings may not sufficiently minimize the role of chance.
3. In designing a sample, the investigator begins by conceptualizing the target population with a specific set of inclusion
criteria that establish demographic and clinical characteristics of subjects well suited to the research question.
4. She then selects an appropriate accessible population that is geographically and temporally convenient, and defines a
parsimonious set of exclusion criteria that eliminate subjects who are unethical or inappropriate to study.
5. The next step is to design an approach to sampling the population. A convenience sample may be adequate, especially
for initial studies of some questions, and a consecutive sample is often a good choice. Simple random sampling can be used
to reduce the size of the sample if necessary, and other probability sampling strategies (stratified and cluster) are useful in
certain situations.
6. Finally, the investigator must design and implement strategies for recruiting a sample of subjects that is sufficiently
representative of the target population to control systematic sources of error, and large enough to control random sources
of error.
, 1 - Hulley et al. (2013) H4 Planning the measurements: precision, accuracy, and validity
• Measures that are precise are free of random error. Measures that are accurate are free of systematic error.
• Validity = How well the variables designed for the study represent the phenomena of interest.
Measurement scales
Numeric variables: continuous and discrete
• Numeric variables can be quantified with a number that expresses how much or how many.
• Continuous variables quantify how much on an infinite scale.
● E.g. The number of possible values of body weight is limited only by the sensitivity of the machine that is used to
measure it.
● Continuous variables are rich in information.
• Discrete numeric variables quantify how many on a scale with fixed units, usually integers, such as the number of times a
woman has been pregnant.
● Discrete variables that have a considerable number of possible values can resemble continuous variables in
statistical analyses and be equivalent for the purpose of designing measurements.
Categorical variables: dichotomous, nominal, and ordinal
• Categorical variables with two possible values (e.g. dead or alive) are termed dichotomous.
• When there are more than two categories (polychotomous) you can further characterize them according to type of
interformation they contain:
● Nominal variables have categories that are not ordered; e.g. blood type
○ Nominal variables tend to have an absolute qualitative character that makes them straightforward to
measure.
● Ordinal variables do have an order; e.g. degree of pain.
○ The additional information is an advantage over nominal variables, but because ordinal variables do not
specify a numerical or uniform difference between one category and the next, the information content is
less than that of discrete or continuous numeric variables.
Choosing a measurement scale
• A good general rule is to prefer continuous over categorical variables when there is a choice, because the additional
information they contain improves statistical efficiency. The result is a study with more power and/or a smaller sample size.
• When there is the option of designing the number of response categories in an ordinal scale, as in a question about food
preferences, it is often useful to provide a half-dozen categories that range from “strongly dislike” to “extremely fond of.”
The results can later be collapsed into a dichotomy (dislike and like), but not vice versa.
• A good choice of study subjects serves the vital purpose of ensuring that the findings in the study accurately represent
what is going on in the population of interest. The protocol must specify a sample of subjects that can be studied at an
acceptable cost in time and money (i.e. modest in size and convenient to access), yet large enough to control random error
and representative enough to allow generalizing study findings to populations of interest.
- Generalizability is a complex qualitative judgment that depends on the investigator’s choice of population and of
sampling design.
Basic terms and concepts
Populations and samples
• A population is a complete set of people with specified
characteristics, and a sample is a subset of the population.
- Clinical and demographic characteristics define the
target population, the large set of people throughout
the world to which the results may be generalized.
- The accessible population is a geographically and
temporally defined subset of the target population that is
available for study.
- The intended study sample is the subset of the accessible
population that the investigator seeks to include in the
study.
- The actual study sample is the group of subjects that does
participate in the study.
Generalizing the study findings
• The classic Framingham Study was an early approach to
scientifically designing a study to allow inferences from findings
observed in a sample to be applied to a population.
• Respondents are often healthier than nonrespondents, especially if
they are volunteers, the characteristics of the actual sample are
therefore undoubtedly different from those of the intended sample.
• Every sample has some errors and the issue is how much damage has
been done.
Steps in designing the protocol for acquiring study subjects
• An investigator begins by specifying the clinical and demographic
characteristics of the target population that will serve the research
question well. She then uses geographic and temporal criteria to specify
a study sample that is representative and practical.
Selection criteria
• A researcher begins by creating selection criteria that define the population to be studied.
Establishing selection criteria
• Inclusion criteria define the main characteristics of the target population that pertain to the research question.
● Demographic characteristics → age, gender, race
● Clinical characteristics →good general health, sexual partner
● Geographic characteristics →Patients attend clinic at hospital
● Temporal characteristics →specified period when study is conducted
• Effect modification = An effect in one factor that is different from that in other factors, also known as “an interaction”
, ● A large sample is needed, and most studies are not powered to detect effect modification.
• Exclusion criteria specify subsets of the population that will not be studied because of:
● A high likelihood of being lost to follow-up (e.g. alcoholic, plans to move)
● An inability to provide good data (e.g. language barrier)
● Being at high risk of possible adverse effects (e.g. history of stroke)
• Clinical trials differ from observational studies in being more likely to have exclusions mandated by concern for the safety
of an intervention in certain patients; e.g. the use of drugs in pregnant women.
● A good general rule that keeps things simple and preserves the number of potential study subjects is to have as few
as exclusion criteria as possible.
Sampling
Nonprobability samples
• In clinical research the study sample is often made up of people who meet the entry criteria and are easily accessible to
the investigator. This is termed a convenience sample. It has obvious advantages in cost and logistics.
• A consecutive sample can minimize volunteerism and other selection biases by consecutively selecting subjects who meet
the entry criteria. This approach is especially desirable, for example, when it amounts to taking the entire accessible
population over a long enough period to include seasonal variations or other temporal changes that are important to the
research question.
Probability samples
• Probability sampling, the gold standard for ensuring generalizability, uses a random process to guarantee that each unit of
the population has a specified chance of being included in the sample.
• A simple random sample is drawn by listing all the people in the population from which the sample will be drawn, and
selecting a subset at random.
● The most common use of this approach in clinical research is when the investigator wishes to select a
representative subset from a population that is larger than she needs.
• A systematic sample resembles a simple random sample in the first step, enumerating the population, but differs in that
the sample is selected by a preordained periodic process (e.g., the Framingham approach of taking the first two out of every
three families from a list of town families ordered by address).
● Systematic sampling is susceptible to errors caused by natural periodicities in the population, and it allows the
investigator to predict and perhaps manipulate those who will be in the sample. It offers no logistic advantages over
simple random sampling, and in clinical research it is rarely a better choice.
• A stratified random sample begins by dividing the population into subgroups according to characteristics such as sex or
race, and taking a random sample from each of these “strata.” The Stratified subsamples can be weighted to draw
disproportionately from subgroups that are less common in the population but of special interest to the investigator.
• A cluster sample is a random sample of natural groupings (clusters) of individuals in the population. Cluster sampling is
useful when the population is widely dispersed and it is impractical to list and sample from all its elements (hospital
example during lecture).
● A disadvantage of cluster sampling is the fact that naturally occurring groups are often more homogeneous for the
variables of interest than the population. This means that the effective sample size (after adjusting for
within-cluster uniformity) will be somewhat smaller than the number of subjects, and that statistical analysis must
take the clustering into account.
Summarizing the sampling design options
• The use of descriptive statistics and tests of statistical significance to draw inferences about the population from
observations in the study sample is based on the assumption that a probability sample has been used. But in clinical
research a random sample of the whole target population is almost never possible. Convenience sampling, preferably with a
consecutive design, is a practical approach that is often suitable.
,Recruitment
The goals of recruitment
• There are two main goals:
(1) to recruit a sample that adequately represents the target population, minimizing the prospect of getting the wrong
answer to the research question due to systematic error (bias); and
(2) to recruit a sufficient sample size to minimize the prospect of getting the wrong answer due to random error (chance).
Achieving a representative sample
• A particular concern, especially for descriptive studies, is the problem of nonresponse. The proportion of subjects
selected for the study who consent to be enrolled (the response rate) influences the validity of inferring that the enrolled
sample represents the population. People who are difficult to reach and those who refuse to participate once they are
contacted tend to be different from people who do enroll.
Recruiting sufficient numbers of subjects
• Falling short in the rate of recruitment is one of the commonest problems in clinical research.
● The approaches to this problem are to estimate the magnitude of the recruitment problem empirically with a
pretest, to plan the study with an accessible population that is larger than believed necessary, and to make
contingency plans should the need arise for additional subjects.
Summary
1. Most clinical research is based, philosophically and practically, on the use of a sample to represent a population.
2. The advantage of sampling is efficiency: It allows the investigator to draw inferences about a large population by
examining a subset at relatively small cost in time and effort. The disadvantage is the sources of error it introduces: If the
sample is not sufficiently representative for the research question at hand the findings may not generalize well to the target
population, and if it is not large enough the findings may not sufficiently minimize the role of chance.
3. In designing a sample, the investigator begins by conceptualizing the target population with a specific set of inclusion
criteria that establish demographic and clinical characteristics of subjects well suited to the research question.
4. She then selects an appropriate accessible population that is geographically and temporally convenient, and defines a
parsimonious set of exclusion criteria that eliminate subjects who are unethical or inappropriate to study.
5. The next step is to design an approach to sampling the population. A convenience sample may be adequate, especially
for initial studies of some questions, and a consecutive sample is often a good choice. Simple random sampling can be used
to reduce the size of the sample if necessary, and other probability sampling strategies (stratified and cluster) are useful in
certain situations.
6. Finally, the investigator must design and implement strategies for recruiting a sample of subjects that is sufficiently
representative of the target population to control systematic sources of error, and large enough to control random sources
of error.
, 1 - Hulley et al. (2013) H4 Planning the measurements: precision, accuracy, and validity
• Measures that are precise are free of random error. Measures that are accurate are free of systematic error.
• Validity = How well the variables designed for the study represent the phenomena of interest.
Measurement scales
Numeric variables: continuous and discrete
• Numeric variables can be quantified with a number that expresses how much or how many.
• Continuous variables quantify how much on an infinite scale.
● E.g. The number of possible values of body weight is limited only by the sensitivity of the machine that is used to
measure it.
● Continuous variables are rich in information.
• Discrete numeric variables quantify how many on a scale with fixed units, usually integers, such as the number of times a
woman has been pregnant.
● Discrete variables that have a considerable number of possible values can resemble continuous variables in
statistical analyses and be equivalent for the purpose of designing measurements.
Categorical variables: dichotomous, nominal, and ordinal
• Categorical variables with two possible values (e.g. dead or alive) are termed dichotomous.
• When there are more than two categories (polychotomous) you can further characterize them according to type of
interformation they contain:
● Nominal variables have categories that are not ordered; e.g. blood type
○ Nominal variables tend to have an absolute qualitative character that makes them straightforward to
measure.
● Ordinal variables do have an order; e.g. degree of pain.
○ The additional information is an advantage over nominal variables, but because ordinal variables do not
specify a numerical or uniform difference between one category and the next, the information content is
less than that of discrete or continuous numeric variables.
Choosing a measurement scale
• A good general rule is to prefer continuous over categorical variables when there is a choice, because the additional
information they contain improves statistical efficiency. The result is a study with more power and/or a smaller sample size.
• When there is the option of designing the number of response categories in an ordinal scale, as in a question about food
preferences, it is often useful to provide a half-dozen categories that range from “strongly dislike” to “extremely fond of.”
The results can later be collapsed into a dichotomy (dislike and like), but not vice versa.
