ISYE 6501 Homework 3 Questions and Answers
Question 5.1
Using crime data f rom the f ile uscrime.txt (http://www.statsci.org/data/general/uscrime.txt, description at
http://www.statsci.org/data/general/uscrime.html), test to see whether there are any outliers in the last
column (number of crimes per 100,000 people).
Use the grubbs.test f unction in the outliers package in R.
ANSWER
First, I unpacked the data and looked at only the crime column. Then I chose to visualize the data with a
histogram to check f or a normal distribution, and to visualize outliers (Citation: I used Copilot to assist with
creating the syntax to make a histogram in R - Code is presented in Appendix).
That histogram is visualized here
knitr::include_graphics("crimedistribution.png")
We can see that f rom the values 300 to 1300, there isn’t exactly a normal distribution, as some crime rates are
missing or occur more f requently than others. However, we can see the bell curve shape centered around 800
to 850, with rates between 500 and 550, and 1200 and 1250 occurring more f requently than the rates
1
,around them, and at a rate comparable to the bell curve’s center rate of 800 to 850.The values beyond
1300, visually, to me, may be outliers.
Next let’s visualize this in the f orm of a box and whisker plot to better visualize and put statistics around
the distribution of the data. With his graph, we can visualize potential outliers based on the data’s distri -
bution..(Citation: I used Copilot to assist with creating the syntax to make a box and whisker plot in R -
Code is presented in Appendix).
knitr::include_graphics("crimebox.png")
By def ault, the box plot f unction plots the quartiles.The median value of the crime rate data is 831. 50 %
of the data f alls in the range between 658.5 and 1057.5. Everything beyond the whisker marks of 342 and
1635 are considered outliers in the data’s distribution of values.
While earlier when viewing the histogram I said it appears to me that values beyond 1300 may be
considered outliers, when we visualize the data around quartile distributions via box and whisker, we only
have three potential outlier values: 1674, 1969, and 1993.
Now let’s see what the grubbs test will identif y as outliers.
library(outliers)
crimedata <- read.table("uscrime.txt",
header = TRUE)
#head(crimedata)
crimecolumn <- crimedata$Crime
grubb10 <- grubbs.test(crimecolumn,type = 10)
grubb11 <- grubbs.test(crimecolumn,type = 11)
grubb10
##
## Grubbs test f or one outlier
##
## data: crimecolumn
2
, ## G = 2.81287, U = 0.82426, p-value = 0.07887
## alternative hypothesis: highest value 1993 is an outlier
grubb11
##
## Grubbs test f or two opposite outliers
##
## data: crimecolumn
## G = 4.26877, U = 0.78103, p-value = 1
## alternative hypothesis: 342 and 1993 are outliers
From the grubb’s test, when we look at both tails of the data’s value distribution, and check the minimum
and maximum value (grubb11), the p -value is 1. Meaning, the minimum value of 342 and the maximum
value of 1993 are not statistically dif f erent enough in value to be considered outliers f rom the rest of the
crime rates in the data set. I say this, because typically a p -value less than .05 is used to declare if
something is statistically signif icant. In our case, the p -value would need to be less than 0.05 f or us to say
that the value of 1993 is a statistically signif icant outlier f rom the rest of the crime rate values.
When we look at the largest outlier f rom the data set via grub10, that value is 1993 - the maximum value
we visualized earlier in the box and whisker plot.
Grubb10 shows a p-value of 0.07887. As stated bef ore, because the p -value is not less than 0.05, we
might not consider the value of 1993 as a true outlier in the range of crime rate values. However, we set
the signif icance threshold. The standard is usually 0.05, but if we could justif y a higher signif icance
threshold above the p-value of 0.07887, then we would need to consider this value as a statistically
signif icant outlier, and understand its context to determine the appropriate strategy f or dealing with it
(omission, imputing a value, etc).
So, by distribution (histogram and box and whisker), there appear to be outliers in this data set. However,
via a grubb’s statistical analysis, the maximum and minimum values are not statistically dif f erent enough
(p-value is not less than 0.05) in value to be considered outliers.
Question 6.1 Describe a situation or problem f rom your job, everyday lif e, current events, etc., f or which
a Change Detection model would be appropriate. Applying the CUSUM technique, how would you choose
the critical value and the threshold?
ANSWER:
From current events, I think a good use of a Change Detection model like CUSUM would be tracking
polling values f or political candidates. I think polling values f rom multiple sources could be aggregated
and plotted via CUSUM.
For example, assume the polls are asking “Will you vote f or Candidate A or Candidate B?”.
Assume the polls are aggregated on a state level, with values like: The Badger News - 46 % Candidate A,
54 % Candidate B f or Alaska.
Then the critical value should be set to 50 % (or 0.50 as a decimal) to track which candidate is in the lead.
From here the threshold could be set to 5 %. Or, maybe the threshold could be based on historical data
f or each state’s particular trends. For example, maybe there is a trend of greater error in polling data in
one state, so they would have a unique threshold modif ier to take this into account.
Then state’s could be broken into “Likely Candidate A” or “Likely Candidate B” groups based on when
they exceed the CUSUM threshold f or one candidate or another.
And, the threshold could be increased to 10 % to designate a state as “Very Likely A/B”. Or, the threshold
could be lowered to 2 % to designate a state as “Slight A/B”.
3
Question 5.1
Using crime data f rom the f ile uscrime.txt (http://www.statsci.org/data/general/uscrime.txt, description at
http://www.statsci.org/data/general/uscrime.html), test to see whether there are any outliers in the last
column (number of crimes per 100,000 people).
Use the grubbs.test f unction in the outliers package in R.
ANSWER
First, I unpacked the data and looked at only the crime column. Then I chose to visualize the data with a
histogram to check f or a normal distribution, and to visualize outliers (Citation: I used Copilot to assist with
creating the syntax to make a histogram in R - Code is presented in Appendix).
That histogram is visualized here
knitr::include_graphics("crimedistribution.png")
We can see that f rom the values 300 to 1300, there isn’t exactly a normal distribution, as some crime rates are
missing or occur more f requently than others. However, we can see the bell curve shape centered around 800
to 850, with rates between 500 and 550, and 1200 and 1250 occurring more f requently than the rates
1
,around them, and at a rate comparable to the bell curve’s center rate of 800 to 850.The values beyond
1300, visually, to me, may be outliers.
Next let’s visualize this in the f orm of a box and whisker plot to better visualize and put statistics around
the distribution of the data. With his graph, we can visualize potential outliers based on the data’s distri -
bution..(Citation: I used Copilot to assist with creating the syntax to make a box and whisker plot in R -
Code is presented in Appendix).
knitr::include_graphics("crimebox.png")
By def ault, the box plot f unction plots the quartiles.The median value of the crime rate data is 831. 50 %
of the data f alls in the range between 658.5 and 1057.5. Everything beyond the whisker marks of 342 and
1635 are considered outliers in the data’s distribution of values.
While earlier when viewing the histogram I said it appears to me that values beyond 1300 may be
considered outliers, when we visualize the data around quartile distributions via box and whisker, we only
have three potential outlier values: 1674, 1969, and 1993.
Now let’s see what the grubbs test will identif y as outliers.
library(outliers)
crimedata <- read.table("uscrime.txt",
header = TRUE)
#head(crimedata)
crimecolumn <- crimedata$Crime
grubb10 <- grubbs.test(crimecolumn,type = 10)
grubb11 <- grubbs.test(crimecolumn,type = 11)
grubb10
##
## Grubbs test f or one outlier
##
## data: crimecolumn
2
, ## G = 2.81287, U = 0.82426, p-value = 0.07887
## alternative hypothesis: highest value 1993 is an outlier
grubb11
##
## Grubbs test f or two opposite outliers
##
## data: crimecolumn
## G = 4.26877, U = 0.78103, p-value = 1
## alternative hypothesis: 342 and 1993 are outliers
From the grubb’s test, when we look at both tails of the data’s value distribution, and check the minimum
and maximum value (grubb11), the p -value is 1. Meaning, the minimum value of 342 and the maximum
value of 1993 are not statistically dif f erent enough in value to be considered outliers f rom the rest of the
crime rates in the data set. I say this, because typically a p -value less than .05 is used to declare if
something is statistically signif icant. In our case, the p -value would need to be less than 0.05 f or us to say
that the value of 1993 is a statistically signif icant outlier f rom the rest of the crime rate values.
When we look at the largest outlier f rom the data set via grub10, that value is 1993 - the maximum value
we visualized earlier in the box and whisker plot.
Grubb10 shows a p-value of 0.07887. As stated bef ore, because the p -value is not less than 0.05, we
might not consider the value of 1993 as a true outlier in the range of crime rate values. However, we set
the signif icance threshold. The standard is usually 0.05, but if we could justif y a higher signif icance
threshold above the p-value of 0.07887, then we would need to consider this value as a statistically
signif icant outlier, and understand its context to determine the appropriate strategy f or dealing with it
(omission, imputing a value, etc).
So, by distribution (histogram and box and whisker), there appear to be outliers in this data set. However,
via a grubb’s statistical analysis, the maximum and minimum values are not statistically dif f erent enough
(p-value is not less than 0.05) in value to be considered outliers.
Question 6.1 Describe a situation or problem f rom your job, everyday lif e, current events, etc., f or which
a Change Detection model would be appropriate. Applying the CUSUM technique, how would you choose
the critical value and the threshold?
ANSWER:
From current events, I think a good use of a Change Detection model like CUSUM would be tracking
polling values f or political candidates. I think polling values f rom multiple sources could be aggregated
and plotted via CUSUM.
For example, assume the polls are asking “Will you vote f or Candidate A or Candidate B?”.
Assume the polls are aggregated on a state level, with values like: The Badger News - 46 % Candidate A,
54 % Candidate B f or Alaska.
Then the critical value should be set to 50 % (or 0.50 as a decimal) to track which candidate is in the lead.
From here the threshold could be set to 5 %. Or, maybe the threshold could be based on historical data
f or each state’s particular trends. For example, maybe there is a trend of greater error in polling data in
one state, so they would have a unique threshold modif ier to take this into account.
Then state’s could be broken into “Likely Candidate A” or “Likely Candidate B” groups based on when
they exceed the CUSUM threshold f or one candidate or another.
And, the threshold could be increased to 10 % to designate a state as “Very Likely A/B”. Or, the threshold
could be lowered to 2 % to designate a state as “Slight A/B”.
3
