Research Methods Quantitative
Session 1 - Working with Quantitative Data
Bettis, R. A. (2012). The search for asterisks: Compromised statistical tests and flawed
theories. Strategic Management Journal, 33(1), 108-113.
- The asterisks (*) is commonly used to represent 10% or better significance levels on
variables.
- A common rationale of some scholars is to play a game to get published. A core issue
here is repeated statistical tests, or more precisely the reporting of grossly
inappropriate significance levels without considering all the tests that have been run.
- Repeated tests and the resultant reporting of statistical significance when it is actually
not present are both errors that interact with professional norms such as biases against
both replication studies and non-results.
- Three broad categories of repeated testing:
1. Multiple researchers with similar interests will be using the same databases at any
time and also across time (i.e. unconnected research projects).
→ Repeated testing in unconnected research projects is unpreventable and largely
unconscious as researchers are working independently.
2. The tuning of an ex ante model with some additional tests after the planned analysis.
→ Ex post tuning of ex ante models to make them look better, while certainly not
unconscious, seems difficult for researchers to resist and is quite common.
3. Indefensible ex ante data snooping described as searching for asterisks.
→ Searching for asterisks is the most damaging form of repeated testing, since the aim
is to reject null hypotheses while consciously ignoring the many models and tests that
have been conducted and, thus, reporting greatly exaggerated levels of significance.
- A little bit of data-snooping (i.e. searching for asterisks) goes a long way: t-statistics with
[t] > 2 are the rule, not the exception. If an intercept is added to the model, the F-test
will give off-scale P-values (Freedman, 2009).
- The flawed theory results not just from repeated tests themselves, but from their
interaction with some institutional features of the strategic management field, including:
1. A bias against the publication of replication studies.
2. No reporting of so-called non-results - i.e. results that are so low quality that they are
uninformative, either due to poor experimental design or errors in data collection.
3. The belief by many scholars, referees, and journal editors that all journal articles
should both develop and test theory.
Page 1 of 20
, Research Methods Quantitative
- It is an established norm to consider a significant coefficient conclusive in proving the
theory embodied in the particular hypothesis. This is inconsistent with the probabilistic
and evidential nature of statistical testing. Researchers need to make the facts of
statistical testing correspond with how test results are reported and interpreted in the
profession.
Schilling, M. A. (2009). Understanding the alliance data. Strategic Management
Journal, 30(3), 233-260.
- Each database has its own unique set of advantages and disadvantages that make it
better suited to some types of research than others. Researchers need to understand
these advantages and disadvantages when selecting a database and creating a
research design.
- A large body of research utilises large alliance databases (e.g., SDC, MERIT-CATI, CORE,
RECAP, and BIOSCAN) to study inter-organizational relationships.
- This study analyses five alliance databases, where three are multi-sector databases
(SDC, MERIT-CATI, and CORE) and two databases are specific to the biotechnology
sector (RECAP and Bioscan). This in an attempt to examine the databases’ consistency
of coverage and completeness, and assess whether different databases yield the same
patterns in sectoral composition, temporal trends, and geographic patterns in alliance
activity.
- Firms are not required to report their alliances to any governing body, and while many
do announce their alliances publicly, whether and how these announcements are
reported in new sources is highly variable.
- For a small sample of firms, the researcher can resort to reading all of the news
coverage of those firms and coding the alliances manually, but this is impractical for all
but the most constrained samples. Luckily, researchers can get some sense of the
completeness of the datasets by comparing them to each other.
- The findings reveal that:
• Databases only report a fraction of formally announced alliances, which could have
detrimental consequences for some types of research.
• However, the databases exhibit strong symmetries in patterns of sectoral composition,
alliance activity over time, and geographic participation.
• Yet, consistency of coverage between the alliance databases is rather poor,
suggesting that none of the databases includes the population of formally announced
Page 2 of 20
, Research Methods Quantitative
alliances — they are all samples. Even though the databases only capture a sample of
alliance activity, they may yield reliable results for many research purposes.
Shortell, S. M., & Zajac, E. J. (1990). Perceptual and archival measures of Miles and
Snow's strategic types: A comprehensive assessment of reliability and validity.
Academy of Management Journal, 33(4), 817-832.
- An important issue in organisational research is the extent to which a top
management's assessment of its organisation's strategy is a reliable and valid measure.
Establishing the reliability and validity of such a self-typing measure is important
because archival data may be unavailable, incomplete, or unusable. This study attempts
to address this need by assessing the reliability and validity of measures of Miles
and Snow's (1978) typology of strategies, which divides organisations into
prospector, defender, analyser, and reactor types.
- Miles and Snow (1978) suggested that an organisation's strategies have three domains:
1. The entrepreneurial: relating to how the organisation orients itself to the
marketplace.
2. The administrative: embracing how the organisation attempt to coordinate and
implement its strategies.
3. The technical: the technology and processes used to produce the organisation's
product and service.
- Miles and Snow's four strategies:
• A firm following a prospector strategy frequently adds to and changes its products
and services, consistently attempting to be the first in the market. Such a firm tends to
stress innovation and flexibility to respond quickly to market changes.
• An analyser's strategy is to maintain a relatively stable base of products and services
while selectively moving into new areas with demonstrated promise. An analyser tends
to emphasise formal planning processes and tries to balance cost containment and
efficiency with risk taking and innovation.
• A defender's strategy is to offer a relatively stable set of services to defined markets,
concentrating on doing the best job possible in its area of expertise. It emphases tight
control and continually looks for operating efficiencies to lower costs.
• A reactor essentially lacks a consistent strategy. Its strategy has characteristics of
each of the other type's strategies at different times and thus is difficult to categorise
clearly.
Page 3 of 20
Session 1 - Working with Quantitative Data
Bettis, R. A. (2012). The search for asterisks: Compromised statistical tests and flawed
theories. Strategic Management Journal, 33(1), 108-113.
- The asterisks (*) is commonly used to represent 10% or better significance levels on
variables.
- A common rationale of some scholars is to play a game to get published. A core issue
here is repeated statistical tests, or more precisely the reporting of grossly
inappropriate significance levels without considering all the tests that have been run.
- Repeated tests and the resultant reporting of statistical significance when it is actually
not present are both errors that interact with professional norms such as biases against
both replication studies and non-results.
- Three broad categories of repeated testing:
1. Multiple researchers with similar interests will be using the same databases at any
time and also across time (i.e. unconnected research projects).
→ Repeated testing in unconnected research projects is unpreventable and largely
unconscious as researchers are working independently.
2. The tuning of an ex ante model with some additional tests after the planned analysis.
→ Ex post tuning of ex ante models to make them look better, while certainly not
unconscious, seems difficult for researchers to resist and is quite common.
3. Indefensible ex ante data snooping described as searching for asterisks.
→ Searching for asterisks is the most damaging form of repeated testing, since the aim
is to reject null hypotheses while consciously ignoring the many models and tests that
have been conducted and, thus, reporting greatly exaggerated levels of significance.
- A little bit of data-snooping (i.e. searching for asterisks) goes a long way: t-statistics with
[t] > 2 are the rule, not the exception. If an intercept is added to the model, the F-test
will give off-scale P-values (Freedman, 2009).
- The flawed theory results not just from repeated tests themselves, but from their
interaction with some institutional features of the strategic management field, including:
1. A bias against the publication of replication studies.
2. No reporting of so-called non-results - i.e. results that are so low quality that they are
uninformative, either due to poor experimental design or errors in data collection.
3. The belief by many scholars, referees, and journal editors that all journal articles
should both develop and test theory.
Page 1 of 20
, Research Methods Quantitative
- It is an established norm to consider a significant coefficient conclusive in proving the
theory embodied in the particular hypothesis. This is inconsistent with the probabilistic
and evidential nature of statistical testing. Researchers need to make the facts of
statistical testing correspond with how test results are reported and interpreted in the
profession.
Schilling, M. A. (2009). Understanding the alliance data. Strategic Management
Journal, 30(3), 233-260.
- Each database has its own unique set of advantages and disadvantages that make it
better suited to some types of research than others. Researchers need to understand
these advantages and disadvantages when selecting a database and creating a
research design.
- A large body of research utilises large alliance databases (e.g., SDC, MERIT-CATI, CORE,
RECAP, and BIOSCAN) to study inter-organizational relationships.
- This study analyses five alliance databases, where three are multi-sector databases
(SDC, MERIT-CATI, and CORE) and two databases are specific to the biotechnology
sector (RECAP and Bioscan). This in an attempt to examine the databases’ consistency
of coverage and completeness, and assess whether different databases yield the same
patterns in sectoral composition, temporal trends, and geographic patterns in alliance
activity.
- Firms are not required to report their alliances to any governing body, and while many
do announce their alliances publicly, whether and how these announcements are
reported in new sources is highly variable.
- For a small sample of firms, the researcher can resort to reading all of the news
coverage of those firms and coding the alliances manually, but this is impractical for all
but the most constrained samples. Luckily, researchers can get some sense of the
completeness of the datasets by comparing them to each other.
- The findings reveal that:
• Databases only report a fraction of formally announced alliances, which could have
detrimental consequences for some types of research.
• However, the databases exhibit strong symmetries in patterns of sectoral composition,
alliance activity over time, and geographic participation.
• Yet, consistency of coverage between the alliance databases is rather poor,
suggesting that none of the databases includes the population of formally announced
Page 2 of 20
, Research Methods Quantitative
alliances — they are all samples. Even though the databases only capture a sample of
alliance activity, they may yield reliable results for many research purposes.
Shortell, S. M., & Zajac, E. J. (1990). Perceptual and archival measures of Miles and
Snow's strategic types: A comprehensive assessment of reliability and validity.
Academy of Management Journal, 33(4), 817-832.
- An important issue in organisational research is the extent to which a top
management's assessment of its organisation's strategy is a reliable and valid measure.
Establishing the reliability and validity of such a self-typing measure is important
because archival data may be unavailable, incomplete, or unusable. This study attempts
to address this need by assessing the reliability and validity of measures of Miles
and Snow's (1978) typology of strategies, which divides organisations into
prospector, defender, analyser, and reactor types.
- Miles and Snow (1978) suggested that an organisation's strategies have three domains:
1. The entrepreneurial: relating to how the organisation orients itself to the
marketplace.
2. The administrative: embracing how the organisation attempt to coordinate and
implement its strategies.
3. The technical: the technology and processes used to produce the organisation's
product and service.
- Miles and Snow's four strategies:
• A firm following a prospector strategy frequently adds to and changes its products
and services, consistently attempting to be the first in the market. Such a firm tends to
stress innovation and flexibility to respond quickly to market changes.
• An analyser's strategy is to maintain a relatively stable base of products and services
while selectively moving into new areas with demonstrated promise. An analyser tends
to emphasise formal planning processes and tries to balance cost containment and
efficiency with risk taking and innovation.
• A defender's strategy is to offer a relatively stable set of services to defined markets,
concentrating on doing the best job possible in its area of expertise. It emphases tight
control and continually looks for operating efficiencies to lower costs.
• A reactor essentially lacks a consistent strategy. Its strategy has characteristics of
each of the other type's strategies at different times and thus is difficult to categorise
clearly.
Page 3 of 20
