ISYE 6501 : Midterm Final Quiz Summary
Module 11: Limiting the number of factors in model (note: Factor-based models: classification, clustering , Regression)
Reason: 1) Overfitting 2) Simplicity
Variable Selection methods: - No good rule of thumb
1) Forward selection (start: no factors)->find the best new factor
2) Backward Elimination (start: All factors-> find worst current factor-> if p>0.15 & bad-> remove -> still too many?
3) Stepwise Regression:(combination of forward and backward) Criteria: p-value, R^2 Akaike, Bayesian information
(most common, Greedy Algorithm): At each step take one looking best, Further options are not considered
4) LASSO: min Σ (𝑦𝑖 − 𝑦̂𝑖 )2 𝑤𝑖𝑡ℎ 𝚺|𝒂𝒋 | ≤ 𝑻, (slower but better prediction) (when the fewest variables): Do scale data
5) Elastic: min Σ (𝑦𝑖 − 𝑦 ̂𝑖 )2 𝑤𝑖𝑡ℎ 𝜆 𝚺|𝒂𝒋 | + (𝟏 − 𝝀)𝚺𝒂𝟐𝒋 ≤ 𝑻, (slower but better prediction) (when the middle-range variables):
Do scale data (advantage: variable section of LASSO, Predictive benefits of Ridge, Disadvantage: Arbitrarily rules,
underestimate coef) (If two predictors are highly correlated, Ridge regression have non-zero coefs. But Lasso will choose only one for model)
6) Ridge: min Σ (𝑦𝑖 − 𝑦̂𝑖 ) 𝑤𝑖𝑡ℎ 𝚺 𝒂𝟐𝒋 ≤ 𝑻, (better prediction) (when the most variables like Linear model)
2
- No Variable selection – effective for overfitting
Bias-Variance Tradeoff
Module 12:: Design of Experiment (DOE) (ex: two banner ads in GT)
Why need: 1) Comparison and control on factor test 2) Blocking (ex: Create variation – sports car vs family car)
- A/B Testing (Two alternatives): quick data collection – must be representative, small amount of data
Factorial Design: (determine effects of factors, Use before collecting data)
1) Full factorial design – test every combination like ANOVA (–to determine importance of end factor). If too many, test subset
of combination while keep balancing 2) Partial Factorial design – compare some combinations to estimate all effects
Exploration and Exploitation: Balance
- Multi-arm bandit model – No simple rule – As we get more sure of the best answer, we’re more likely to choose to use it
Module 13: Probability-Based Models
• Sometime simple approaches work better ex) season ticket holder- Use a simple prob dist to answer the company’s question
Probability Distribution:
1) Bernoulli – single event
2) Binomial – n events of binary 𝑃(𝑋 = 𝑥) = ( )𝑝 𝑥 (1 − 𝑝)𝑛−𝑥 : as 𝑛 → ∞, it becomes normal distribution
3) Geometric- unsuccessful flips before 1st success 𝑃(𝑋 = 𝑥) = (1 − 𝑝)𝑥 𝑝 ex) hit until bat break, how many interviews/goods
before, Airport security screen line up
𝜆𝑥 𝑒 −𝜆
4) Poisson = random arrival with time period : 𝑓(𝑥) = Ex) count arrivals at airport security line (λ=avg # of arrivals/time period)
𝑥!
−𝜆
5) Exponential: Time between successive arrival 𝑓(𝑥) = 𝜆𝑒 - ex) security line
𝑥 𝑘
𝑘 𝑥 𝑘−1
6) Weibull: Amount of time to fail. Time between failure ex) lightbulb - 𝑓(𝑥) = ( ) 𝑒 −(𝜆) where λ = scale, k = shape
𝜆 𝜆
(if k<1, failure rate decrease i.e. “worst fail first: if k>1, things that wear out (ex. Tires): if k = 1, failure rate constant, exponential)
• Visualization – QQ plot –1) compare two data sets 2) Test whether a single data set is good fit to a probability
distribution (x-axis: data, y-axis: Theoretical values of percentiles)
Queuing: Poisson Queue Employee :
Ex) Arrival rates (calls) = 𝜆, Service Rate (Calls) = 𝜇 (> 𝜆)
Transition Eq (>= calls in queue), P(next is arrived) = 𝜆/(𝜆 + 𝜇)
P(next is finished) = 𝜇/(𝜆 + 𝜇)
Memoryless – Poisson in the arrival time are exponentially distributed Ex) Law Firm (Tire Fail is not memoryless)
• Simulation { Deterministic simulation
Stochastice simulation − has randomness
-Simulation type {Continuous − time simulation ex) chemical process: what-if question: Prediction: the number of runs of simulation: Using average value
Discrete event simulation − call center
in not good enough, Run multiple time
• Validation is hard if simulation does not exist
Markov Chain Model (Memoryless) – based on stats of system: Instead of “steady state” 𝜋 ∗ 𝑝 = 𝜋 ∗ → 𝑠𝑜𝑙𝑣𝑒 𝜋 ∗ → 𝑁𝑜𝑡 𝑎𝑙𝑤𝑎𝑦𝑠 𝑒𝑥𝑖𝑠𝑡, 𝑁𝑜 𝑐𝑦𝑐𝑙𝑖𝑐,
Key Assumption: Memoryless – 1) Only depends on most recent state 𝐸𝑣𝑒𝑦 𝑠𝑡𝑎𝑡𝑒 𝑚𝑢𝑠𝑡 𝑏𝑒 𝑟𝑒𝑎𝑐ℎ𝑎𝑏𝑙𝑒
Module 11: Limiting the number of factors in model (note: Factor-based models: classification, clustering , Regression)
Reason: 1) Overfitting 2) Simplicity
Variable Selection methods: - No good rule of thumb
1) Forward selection (start: no factors)->find the best new factor
2) Backward Elimination (start: All factors-> find worst current factor-> if p>0.15 & bad-> remove -> still too many?
3) Stepwise Regression:(combination of forward and backward) Criteria: p-value, R^2 Akaike, Bayesian information
(most common, Greedy Algorithm): At each step take one looking best, Further options are not considered
4) LASSO: min Σ (𝑦𝑖 − 𝑦̂𝑖 )2 𝑤𝑖𝑡ℎ 𝚺|𝒂𝒋 | ≤ 𝑻, (slower but better prediction) (when the fewest variables): Do scale data
5) Elastic: min Σ (𝑦𝑖 − 𝑦 ̂𝑖 )2 𝑤𝑖𝑡ℎ 𝜆 𝚺|𝒂𝒋 | + (𝟏 − 𝝀)𝚺𝒂𝟐𝒋 ≤ 𝑻, (slower but better prediction) (when the middle-range variables):
Do scale data (advantage: variable section of LASSO, Predictive benefits of Ridge, Disadvantage: Arbitrarily rules,
underestimate coef) (If two predictors are highly correlated, Ridge regression have non-zero coefs. But Lasso will choose only one for model)
6) Ridge: min Σ (𝑦𝑖 − 𝑦̂𝑖 ) 𝑤𝑖𝑡ℎ 𝚺 𝒂𝟐𝒋 ≤ 𝑻, (better prediction) (when the most variables like Linear model)
2
- No Variable selection – effective for overfitting
Bias-Variance Tradeoff
Module 12:: Design of Experiment (DOE) (ex: two banner ads in GT)
Why need: 1) Comparison and control on factor test 2) Blocking (ex: Create variation – sports car vs family car)
- A/B Testing (Two alternatives): quick data collection – must be representative, small amount of data
Factorial Design: (determine effects of factors, Use before collecting data)
1) Full factorial design – test every combination like ANOVA (–to determine importance of end factor). If too many, test subset
of combination while keep balancing 2) Partial Factorial design – compare some combinations to estimate all effects
Exploration and Exploitation: Balance
- Multi-arm bandit model – No simple rule – As we get more sure of the best answer, we’re more likely to choose to use it
Module 13: Probability-Based Models
• Sometime simple approaches work better ex) season ticket holder- Use a simple prob dist to answer the company’s question
Probability Distribution:
1) Bernoulli – single event
2) Binomial – n events of binary 𝑃(𝑋 = 𝑥) = ( )𝑝 𝑥 (1 − 𝑝)𝑛−𝑥 : as 𝑛 → ∞, it becomes normal distribution
3) Geometric- unsuccessful flips before 1st success 𝑃(𝑋 = 𝑥) = (1 − 𝑝)𝑥 𝑝 ex) hit until bat break, how many interviews/goods
before, Airport security screen line up
𝜆𝑥 𝑒 −𝜆
4) Poisson = random arrival with time period : 𝑓(𝑥) = Ex) count arrivals at airport security line (λ=avg # of arrivals/time period)
𝑥!
−𝜆
5) Exponential: Time between successive arrival 𝑓(𝑥) = 𝜆𝑒 - ex) security line
𝑥 𝑘
𝑘 𝑥 𝑘−1
6) Weibull: Amount of time to fail. Time between failure ex) lightbulb - 𝑓(𝑥) = ( ) 𝑒 −(𝜆) where λ = scale, k = shape
𝜆 𝜆
(if k<1, failure rate decrease i.e. “worst fail first: if k>1, things that wear out (ex. Tires): if k = 1, failure rate constant, exponential)
• Visualization – QQ plot –1) compare two data sets 2) Test whether a single data set is good fit to a probability
distribution (x-axis: data, y-axis: Theoretical values of percentiles)
Queuing: Poisson Queue Employee :
Ex) Arrival rates (calls) = 𝜆, Service Rate (Calls) = 𝜇 (> 𝜆)
Transition Eq (>= calls in queue), P(next is arrived) = 𝜆/(𝜆 + 𝜇)
P(next is finished) = 𝜇/(𝜆 + 𝜇)
Memoryless – Poisson in the arrival time are exponentially distributed Ex) Law Firm (Tire Fail is not memoryless)
• Simulation { Deterministic simulation
Stochastice simulation − has randomness
-Simulation type {Continuous − time simulation ex) chemical process: what-if question: Prediction: the number of runs of simulation: Using average value
Discrete event simulation − call center
in not good enough, Run multiple time
• Validation is hard if simulation does not exist
Markov Chain Model (Memoryless) – based on stats of system: Instead of “steady state” 𝜋 ∗ 𝑝 = 𝜋 ∗ → 𝑠𝑜𝑙𝑣𝑒 𝜋 ∗ → 𝑁𝑜𝑡 𝑎𝑙𝑤𝑎𝑦𝑠 𝑒𝑥𝑖𝑠𝑡, 𝑁𝑜 𝑐𝑦𝑐𝑙𝑖𝑐,
Key Assumption: Memoryless – 1) Only depends on most recent state 𝐸𝑣𝑒𝑦 𝑠𝑡𝑎𝑡𝑒 𝑚𝑢𝑠𝑡 𝑏𝑒 𝑟𝑒𝑎𝑐ℎ𝑎𝑏𝑙𝑒
