Article: David J.Ketchen, Jr. R. Duane Ireland, and Lakami T. Baker (2013). The use of Archival Proxies in
Strategic Management Studies: Castles Made of Sand?
An archival proxy is a quantitative measure that is used to represent a theoretical construct that is
relevant to the design and completion of a research study. In most cases, a measure that is used as an
archival proxy was collected for some other purpose and then is discovered to be a potentially useful
research tool.
A key issue is the construct validity of an archival proxy. Construct validity is defined as how close of an
association actually exists between a measure and the theoretical construct that the measure is meant
to represent or appropriately capture. Good proxies offer a very high degree of conceptual overlap with
their corresponding constructs (strong construct validity). When a measure’s construct validity is low, the
credibility of empirical findings involving the measure is also low.
Concerns are expressed about ‘the fact that multiple researchers with similar interests will be using the
same databases at any time and also across time. This is problematic because the nature of hypothesis
testing within strategic management virtually ensures that relationships are being falsely identified as
significant when they in fact are not significant. Building on this concern, different researchers are using
the same measure to represent different constructs. The degree of match is arguable.
Several studies have relied on patent counts to represent the quantity of a firm’s innovations or
inventions. One troublesome issue is that applying for and/or receiving a patent does not demonstrate
that a firm has the capacity to develop the technology or put it to commercial use.
Adopting a broader perspective suggests that even if a proxy appears to be reasonable within the
context of an individual study, its use among different studies to represent different constructs creates
serious problems. It seems perhaps that strategic management has engaged in a process that Popper
referred to as conventionalism: the tendency of scientists to agree on issues rather than debate them
and then act as if what they have agreed upon is a scientific fact (Ferguson and Ketchen, 1999).
A second possible reason why questionable proxies have been tolerated in the past has its roots in the
time-tested mantra that “all research is flawed”. Because all research designs have limitations, editors
and reviewers have been encouraged to consider a study’s weaknesses within the context of the quality
of the entire research design.
Three actionable suggestions for improving the use of archival proxies within strategic management
research:
1. Scholars using an archival measure as a proxy for a construct should identify whether the
measure has been used in the past as a proxy for another construct and if so, demonstrate which
usage is correct.
2. Gatekeepers should actively break free of institutionalized norms by requiring authors to provide
sound logic for empirical validation of their archival proxies. Providing sound logic to support a
proxy involves explaining why the author believes a very significant conceptual overlap exists
between the proxy and the construct that is purported to assess.
3. Gatekeepers should treat poor proxies as potentially fatal flaws. Similarly, if a measure does not
effectively capture the construct that it is purported to capture, any statistical tests involving the
measure cannot be considered valid.
, Article: Kohavi, R. & Thomke, S. (2012). The surprising power of online experiments.
This article focusses on the simplest kind of controlled experiment and the A/B test. The authors spent
more than 35 years studying and practicing experiments and advising companies in a wide range of
industries about them.
IN BRIEF
THE NEED When building websites and applications, too many companies make decisions —on
everything from new product features, to look and feel, to marketing campaigns— using subjective
opinions rather than hard data.
THE SOLUTION Companies should conduct online controlled experiments to evaluate their ideas.
Potential improvements should be rigorously tested, because large investments can fail to deliver, and
some tiny changes can be surprisingly detrimental while others have big payoffs.
IMPLEMENTATION Leaders should understand how to properly design and execute A/B tests and other
controlled experiments, ensure their integrity, interpret their results, and avoid pitfalls.
Controlled experiments can transform decision making into a scientific, evidence-driven process—rather
than an intuitive reaction.
A: APPRECIATE THE VALUE OF A/B TESTS
In an A/B test the experimenter sets up two experiences:
• “A,” the control, is usually the current system and considered the “champion,”
• “B,” the treatment, is a modification that attempts to improve something—the “challenger.”
Here’s what managers need to understand:
1. Tiny changes can have a big impact.
Clearly, small investments can yield big returns. Large investments, however, may have little or
no payoff.
2. Experiments can guide investment decisions
Online tests can help managers figure out how much investment in a potential improvement is
optimal.
B: BUILD A LARGE-SCALE CAPABILITY
A company’s experimentation personnel can be organized in three ways:
1) Centralized model: a team of data scientists serve the entire company.
Advantage:
o The advantage is that they can focus on long-term projects (e.g. building better
experimentation tools and developing more-advanced statistical algorithms)
Disadvantages:
o One major drawback is that the business units using the group may have different priorities,
which could lead to conflicts over the allocation of resources and costs.
Strategic Management Studies: Castles Made of Sand?
An archival proxy is a quantitative measure that is used to represent a theoretical construct that is
relevant to the design and completion of a research study. In most cases, a measure that is used as an
archival proxy was collected for some other purpose and then is discovered to be a potentially useful
research tool.
A key issue is the construct validity of an archival proxy. Construct validity is defined as how close of an
association actually exists between a measure and the theoretical construct that the measure is meant
to represent or appropriately capture. Good proxies offer a very high degree of conceptual overlap with
their corresponding constructs (strong construct validity). When a measure’s construct validity is low, the
credibility of empirical findings involving the measure is also low.
Concerns are expressed about ‘the fact that multiple researchers with similar interests will be using the
same databases at any time and also across time. This is problematic because the nature of hypothesis
testing within strategic management virtually ensures that relationships are being falsely identified as
significant when they in fact are not significant. Building on this concern, different researchers are using
the same measure to represent different constructs. The degree of match is arguable.
Several studies have relied on patent counts to represent the quantity of a firm’s innovations or
inventions. One troublesome issue is that applying for and/or receiving a patent does not demonstrate
that a firm has the capacity to develop the technology or put it to commercial use.
Adopting a broader perspective suggests that even if a proxy appears to be reasonable within the
context of an individual study, its use among different studies to represent different constructs creates
serious problems. It seems perhaps that strategic management has engaged in a process that Popper
referred to as conventionalism: the tendency of scientists to agree on issues rather than debate them
and then act as if what they have agreed upon is a scientific fact (Ferguson and Ketchen, 1999).
A second possible reason why questionable proxies have been tolerated in the past has its roots in the
time-tested mantra that “all research is flawed”. Because all research designs have limitations, editors
and reviewers have been encouraged to consider a study’s weaknesses within the context of the quality
of the entire research design.
Three actionable suggestions for improving the use of archival proxies within strategic management
research:
1. Scholars using an archival measure as a proxy for a construct should identify whether the
measure has been used in the past as a proxy for another construct and if so, demonstrate which
usage is correct.
2. Gatekeepers should actively break free of institutionalized norms by requiring authors to provide
sound logic for empirical validation of their archival proxies. Providing sound logic to support a
proxy involves explaining why the author believes a very significant conceptual overlap exists
between the proxy and the construct that is purported to assess.
3. Gatekeepers should treat poor proxies as potentially fatal flaws. Similarly, if a measure does not
effectively capture the construct that it is purported to capture, any statistical tests involving the
measure cannot be considered valid.
, Article: Kohavi, R. & Thomke, S. (2012). The surprising power of online experiments.
This article focusses on the simplest kind of controlled experiment and the A/B test. The authors spent
more than 35 years studying and practicing experiments and advising companies in a wide range of
industries about them.
IN BRIEF
THE NEED When building websites and applications, too many companies make decisions —on
everything from new product features, to look and feel, to marketing campaigns— using subjective
opinions rather than hard data.
THE SOLUTION Companies should conduct online controlled experiments to evaluate their ideas.
Potential improvements should be rigorously tested, because large investments can fail to deliver, and
some tiny changes can be surprisingly detrimental while others have big payoffs.
IMPLEMENTATION Leaders should understand how to properly design and execute A/B tests and other
controlled experiments, ensure their integrity, interpret their results, and avoid pitfalls.
Controlled experiments can transform decision making into a scientific, evidence-driven process—rather
than an intuitive reaction.
A: APPRECIATE THE VALUE OF A/B TESTS
In an A/B test the experimenter sets up two experiences:
• “A,” the control, is usually the current system and considered the “champion,”
• “B,” the treatment, is a modification that attempts to improve something—the “challenger.”
Here’s what managers need to understand:
1. Tiny changes can have a big impact.
Clearly, small investments can yield big returns. Large investments, however, may have little or
no payoff.
2. Experiments can guide investment decisions
Online tests can help managers figure out how much investment in a potential improvement is
optimal.
B: BUILD A LARGE-SCALE CAPABILITY
A company’s experimentation personnel can be organized in three ways:
1) Centralized model: a team of data scientists serve the entire company.
Advantage:
o The advantage is that they can focus on long-term projects (e.g. building better
experimentation tools and developing more-advanced statistical algorithms)
Disadvantages:
o One major drawback is that the business units using the group may have different priorities,
which could lead to conflicts over the allocation of resources and costs.
