lOMoAR cPSD| 70487227
lOMoAR cPSD| 70487227
Local Smoothing Methods
Comparison Summary Study guide
INTRODUCTION
The purpose of this report is to investigate three local smoothing methods: LOESS, Nadaraya-
Watson (NW) Kernel Smoothing, and Spline Smoothing. We will test each estimation method with an
equidistant and non-equidistant dataset and then compare empirical bias, variance, and MSE for each
estimator across 101 different datapoints and 1000 runs. The goal is to develop a better understanding
of the statistical properties and computational challenges of each estimator.
DATASETS
For the first part of the assignment, we will be generating our own equidistant dataset with
n=101 datapoints in the range of -2𝜋 ≤ x ≤ 2𝜋. For the second part of the assignment, we are provided
with a random dataset of 101 non-equidistant points, still in the range of -2𝜋 ≤ x ≤ 2𝜋. The three models
will be trained to predict the response variable of the famous Mexican Hat function which is defined as
follows:
Yi = f(xi) + ɛi
f(x) = (1-x2) exp(-0.5x2), -2𝜋 ≤ x ≤ 2𝜋
METHODS
ESTIMATORS
Since an ordinary linear regression model of the form, is too restrictive to predict the results of
the Mexican Hat function, we will instead investigate three local smoothing methods which are more
appropriate for a nonlinear function.
, lOMoAR cPSD| 70487227
LOESS
DATA: (𝑌i, 𝑥i) for 𝑖 = 1,2,⋯𝑛
MODEL: 𝑌i = 𝑓(𝑥I) + 𝜖i for some nonlinear function 𝑓(⋅) GOAL:
find an estimate of 𝑓(𝑥) at a given 𝑥.
The first estimator we will use is LOESS, which stands for locally estimated or weighted scatter
plot smoothing. This model is a combination of weighted least squares and k-nearest neighbor (KNN).
For the KNN portion, the model will find the k training data points that are closest to x. For the weighted
linear regression portion, the k data points will be used to fit low-degree polynomials based on a locally
weighted least squares. The closer an xi is to x, the more weight it will get. For this assignment, we will
use a span of 0.75 for the equidistant dataset and a span of 0.3365 for the non-equidistant dataset. This
means that 75% and roughly 34% of the data will be used to fit the polynomial regression at each of the
k data points. The larger span will lead to a smoother fit but can lose important features of the data
while the smaller span will capture the variations in data better but at the risk of overfitting.
Advantages of LOESS include that is non-parametric and so is flexible enough to fit many forms.
However, the drawback is that it does not fit a closed-form function and so can be difficult to explain.
Additional disadvantages include that is computationally expensive, requires a lot of data for a good fit,
and is not intended for high-dimensional data.
NADARAYA-WATSON KERNEL SMOOTHING:
Data: (𝑌i, 𝑥i) for 𝑖 = 1,2,⋯𝑛
Model: 𝑌i = 𝑓(𝑥I) + 𝜖i , i = 1,…𝑛 where 𝑋1,⋯, 𝑋n have common density 𝑝(x) and the 𝜖1,⋯,
𝜖n are i.i.d. 𝑁(0, 𝜎2) , independent of 𝑋1,⋯, 𝑋n
Goal: Estimate the regression function 𝑓(𝑥) = 𝔼(𝑌| 𝑋 )= 𝑥 from the data.
The second estimator is a type of kernel smoothing used for regression, Nadaraya-Watson.
Similar to LOESS, kernel smoothing is a non-parametric model which helps to smooth out noise,
particularly for curved datapoints. This is achieved by more heavily weighting datapoints which are
closer to the target point. An important feature of kernel smoothing is the bandwidth which determines
the amount of data which will be used for smoothing. A small bandwidth is more sensitive to noise and
so leads to less smoothing while a large bandwidth is less sensitive to noise and leads to more
smoothing, but potentially at the risk of smoothing out important details. For the purpose of this
assignment, we will use a bandwidth of 0.2. The kernel we will use is a Gaussian Kernel, K:
lOMoAR cPSD| 70487227
Local Smoothing Methods
Comparison Summary Study guide
INTRODUCTION
The purpose of this report is to investigate three local smoothing methods: LOESS, Nadaraya-
Watson (NW) Kernel Smoothing, and Spline Smoothing. We will test each estimation method with an
equidistant and non-equidistant dataset and then compare empirical bias, variance, and MSE for each
estimator across 101 different datapoints and 1000 runs. The goal is to develop a better understanding
of the statistical properties and computational challenges of each estimator.
DATASETS
For the first part of the assignment, we will be generating our own equidistant dataset with
n=101 datapoints in the range of -2𝜋 ≤ x ≤ 2𝜋. For the second part of the assignment, we are provided
with a random dataset of 101 non-equidistant points, still in the range of -2𝜋 ≤ x ≤ 2𝜋. The three models
will be trained to predict the response variable of the famous Mexican Hat function which is defined as
follows:
Yi = f(xi) + ɛi
f(x) = (1-x2) exp(-0.5x2), -2𝜋 ≤ x ≤ 2𝜋
METHODS
ESTIMATORS
Since an ordinary linear regression model of the form, is too restrictive to predict the results of
the Mexican Hat function, we will instead investigate three local smoothing methods which are more
appropriate for a nonlinear function.
, lOMoAR cPSD| 70487227
LOESS
DATA: (𝑌i, 𝑥i) for 𝑖 = 1,2,⋯𝑛
MODEL: 𝑌i = 𝑓(𝑥I) + 𝜖i for some nonlinear function 𝑓(⋅) GOAL:
find an estimate of 𝑓(𝑥) at a given 𝑥.
The first estimator we will use is LOESS, which stands for locally estimated or weighted scatter
plot smoothing. This model is a combination of weighted least squares and k-nearest neighbor (KNN).
For the KNN portion, the model will find the k training data points that are closest to x. For the weighted
linear regression portion, the k data points will be used to fit low-degree polynomials based on a locally
weighted least squares. The closer an xi is to x, the more weight it will get. For this assignment, we will
use a span of 0.75 for the equidistant dataset and a span of 0.3365 for the non-equidistant dataset. This
means that 75% and roughly 34% of the data will be used to fit the polynomial regression at each of the
k data points. The larger span will lead to a smoother fit but can lose important features of the data
while the smaller span will capture the variations in data better but at the risk of overfitting.
Advantages of LOESS include that is non-parametric and so is flexible enough to fit many forms.
However, the drawback is that it does not fit a closed-form function and so can be difficult to explain.
Additional disadvantages include that is computationally expensive, requires a lot of data for a good fit,
and is not intended for high-dimensional data.
NADARAYA-WATSON KERNEL SMOOTHING:
Data: (𝑌i, 𝑥i) for 𝑖 = 1,2,⋯𝑛
Model: 𝑌i = 𝑓(𝑥I) + 𝜖i , i = 1,…𝑛 where 𝑋1,⋯, 𝑋n have common density 𝑝(x) and the 𝜖1,⋯,
𝜖n are i.i.d. 𝑁(0, 𝜎2) , independent of 𝑋1,⋯, 𝑋n
Goal: Estimate the regression function 𝑓(𝑥) = 𝔼(𝑌| 𝑋 )= 𝑥 from the data.
The second estimator is a type of kernel smoothing used for regression, Nadaraya-Watson.
Similar to LOESS, kernel smoothing is a non-parametric model which helps to smooth out noise,
particularly for curved datapoints. This is achieved by more heavily weighting datapoints which are
closer to the target point. An important feature of kernel smoothing is the bandwidth which determines
the amount of data which will be used for smoothing. A small bandwidth is more sensitive to noise and
so leads to less smoothing while a large bandwidth is less sensitive to noise and leads to more
smoothing, but potentially at the risk of smoothing out important details. For the purpose of this
assignment, we will use a bandwidth of 0.2. The kernel we will use is a Gaussian Kernel, K:
