Lecture 1 – Simulation
Throwing a dice a lot is suppose to result into a (kinda) uniform histogram
Computer generate random numbers by all kinds of mechanism
- It is hard for computers to generate true random numbers
- It is called pseudo random numbers numbers that look like random numbers but they are
generated by an algorithm that starts with an initial value, the seed. These generated
numbers behave/look like real random numbers, that is why they are called pseudo.
o In python to set the seed: np.random.seed(13091905)
- True randomness can be achieved by inserting a physical property (like temperature)
In python taking random numbers from vectors: x = np.random.choice(np.arange(1,7), 3):
- np.arage(1,7) gives a vector from 0 till 6
- np.random.choice takes random number from the vector, this happens 3 times
- x is the 3 numbers that are chosen
- if you want a sequence of numbers that are the same (so 1,1,1) then you can get it through
an if statement:
o if np.std(x) == 0 standard deviation is 0 when the numbers are the same
- increase the number of throws to get a more specific probability
t-test and regression in python
uniform distribution = hist where all the columns are even with specific domain
binominal distribution = stats.binom.rvs(n=1, p=.5, size=100)
- in this code there are 100 random draws from the normal distribution where the probability
of success is 0.5 (equal chance on win or lose) and n (number of repeats) = 1
normal distribution = stats.norm.rvs(loc=0, scale=1, size=100)
- in this code there are 100 random draws from the normal distribution where the mean is
zero and the standard deviation is 1
probability is given by cdf stats.norm.cdf
Z-waarde:
- geeft aan hoeveel standaardafwijkingen de t-waarde van het gemiddelde van de steekproef is
verwijderd van het gemiddelde van de populatie, gecorrigeerd voor de steekproefgrootte.
Dus eigenlijk zegt het hoe representatief de steekproef is ten opzichte van de populatie.
- Dicht bij 0 = vergelijkbaar, dus representatief
- Positief = steekproefgemiddelde ligt hoger dan populatiegemiddelde
- Negatief = steekproefgemiddelde ligt lager
- Two sided t-test –> regards positive and negative, uses absolute value
How to compute = hoe kom je tot deze …
Assignment 2:
- The Z value for 95% confidence in a two sided test is Z=1.96. How to compute in python?
- z_value = stats.norm.ppf(0.975,0,1) 0.975 because two sided (1-0.025 = 0.975)
,export data python:
- np.savetxt(‘name_file.txt’, x, header = ‘x’)
- you first name the file, then you put the data you want to export (x), then you name this data
in the file (header)
null hypothesis testing
- assume there is no effect/difference
- collect data
- compare assumption with the data (via normal distribution or whatever)
- to do this in simulation in python:
o np.random.normal(loc=0.0, scale = 1.0, size=1000) more expected behaviour
o or what also works = stats.norm.rvs(loc=0.0, scale = 1.0, size=1000)
p-value logic; check type 1 error:
- the p-value is the probability of obtaining test results at least as extreme as the results
actually observed, under the assumption that the null hypothesis is correct
- type 1 error: null hypothesis is rejected when it is actually true. In other words, an effect is
measured that is not really there.
- in python:
o alpha = 0.05
o x= stats.norm.rvs(loc=0.0, scale = 1.0, size=1000)
o p_value = pg.ttest(x,0)[‘p-val’][0]
o reject_null = p_value < alpha true in about alpha = 5% of cases
o print(reject_null)
o this give 1 or 0, 1 = true and 0 = false
type 2 error:
- no effects measured, but effect does exist
Power: probability of finding true effect
- high power means that the chance of making a type 2 error is low, so detecting effects is
effective
- use power to determine the sample size
- effect size ground truth
- first simulate data with norm.rvs, and get the p value with ttest, then calculate power with
power.t.test
o pg.ttest(x, 0)[‘p-val’][0] can also be wilcox.test or binom.test
o pg.power_ttest(effect, sample size, alpha, contrast = ‘one-sample’)
simulating versus fitting models
- models describe data and could reflect a true underlying (cognitive) process
- a parameter of a model a variable that can take a range of values that describe the data
o slope parameters, mean and standard deviation, drift-rates / non-decision times
- we choose parameters to simulate models to generate simulated data
- if we assume a model for data, a parameter can also be estimated from real data (and also
simulated data)
, linear regression
- models: ANOVA and regression
Bayesian model comparison
- a method to compare different models to determine which model fits the data best.
- Bayes-factor (BF)
o BFij smaller than 1 there is more evidence for model j than for model i
o BFij is equal to 1 no difference in evidence for the models
o BFij larger than 1 evidence for model i
o Evidence:
BF 1-3 = weak evidence
BF 3-20 = somewhat evidence
BF 20-150 = strong evidence
BF >150 = very strong evidence
Polynomial regression analysis
- Here we expect that one variable would be the predictive variable in two types of ways:
o In a linear way
o Quadratic way
- To simulate random noise example:
o Always use the normal distribution to simulate the noice
o x=np.linspace(0,2,N)
o y1=3-.2*x+.5*x^2 + stats.norm.rvs(0, 0.4, N)
the normal distribution simulates random noise along the line of
3-.2*x+.5*x^2
in python ^ is **
parameters are:
intercept parameter = 3
linear term = -.2
squared term = .5
ground truth = -.2 for linear model
ground truth = .5 for squared model
look in the results at coefficient of x1 and x2, compare these to the ground
truths to see if the model fits
o fitting by .fit()
overfitting and model comparison
- Occam’s razor: when faced with two opposing explanations for the same set of evidence our
minds will naturally prefer the explanation that makes the fewest assumptions
- R squared explains the variance:
o Between 0 and 1
o 0 means no variance is explained
o < 0.3 weak
o 0.3 – 0.5 moderate
o > 0.7 strong
o 1 means everything is explained, this is suspicious, most likely overfitting
Throwing a dice a lot is suppose to result into a (kinda) uniform histogram
Computer generate random numbers by all kinds of mechanism
- It is hard for computers to generate true random numbers
- It is called pseudo random numbers numbers that look like random numbers but they are
generated by an algorithm that starts with an initial value, the seed. These generated
numbers behave/look like real random numbers, that is why they are called pseudo.
o In python to set the seed: np.random.seed(13091905)
- True randomness can be achieved by inserting a physical property (like temperature)
In python taking random numbers from vectors: x = np.random.choice(np.arange(1,7), 3):
- np.arage(1,7) gives a vector from 0 till 6
- np.random.choice takes random number from the vector, this happens 3 times
- x is the 3 numbers that are chosen
- if you want a sequence of numbers that are the same (so 1,1,1) then you can get it through
an if statement:
o if np.std(x) == 0 standard deviation is 0 when the numbers are the same
- increase the number of throws to get a more specific probability
t-test and regression in python
uniform distribution = hist where all the columns are even with specific domain
binominal distribution = stats.binom.rvs(n=1, p=.5, size=100)
- in this code there are 100 random draws from the normal distribution where the probability
of success is 0.5 (equal chance on win or lose) and n (number of repeats) = 1
normal distribution = stats.norm.rvs(loc=0, scale=1, size=100)
- in this code there are 100 random draws from the normal distribution where the mean is
zero and the standard deviation is 1
probability is given by cdf stats.norm.cdf
Z-waarde:
- geeft aan hoeveel standaardafwijkingen de t-waarde van het gemiddelde van de steekproef is
verwijderd van het gemiddelde van de populatie, gecorrigeerd voor de steekproefgrootte.
Dus eigenlijk zegt het hoe representatief de steekproef is ten opzichte van de populatie.
- Dicht bij 0 = vergelijkbaar, dus representatief
- Positief = steekproefgemiddelde ligt hoger dan populatiegemiddelde
- Negatief = steekproefgemiddelde ligt lager
- Two sided t-test –> regards positive and negative, uses absolute value
How to compute = hoe kom je tot deze …
Assignment 2:
- The Z value for 95% confidence in a two sided test is Z=1.96. How to compute in python?
- z_value = stats.norm.ppf(0.975,0,1) 0.975 because two sided (1-0.025 = 0.975)
,export data python:
- np.savetxt(‘name_file.txt’, x, header = ‘x’)
- you first name the file, then you put the data you want to export (x), then you name this data
in the file (header)
null hypothesis testing
- assume there is no effect/difference
- collect data
- compare assumption with the data (via normal distribution or whatever)
- to do this in simulation in python:
o np.random.normal(loc=0.0, scale = 1.0, size=1000) more expected behaviour
o or what also works = stats.norm.rvs(loc=0.0, scale = 1.0, size=1000)
p-value logic; check type 1 error:
- the p-value is the probability of obtaining test results at least as extreme as the results
actually observed, under the assumption that the null hypothesis is correct
- type 1 error: null hypothesis is rejected when it is actually true. In other words, an effect is
measured that is not really there.
- in python:
o alpha = 0.05
o x= stats.norm.rvs(loc=0.0, scale = 1.0, size=1000)
o p_value = pg.ttest(x,0)[‘p-val’][0]
o reject_null = p_value < alpha true in about alpha = 5% of cases
o print(reject_null)
o this give 1 or 0, 1 = true and 0 = false
type 2 error:
- no effects measured, but effect does exist
Power: probability of finding true effect
- high power means that the chance of making a type 2 error is low, so detecting effects is
effective
- use power to determine the sample size
- effect size ground truth
- first simulate data with norm.rvs, and get the p value with ttest, then calculate power with
power.t.test
o pg.ttest(x, 0)[‘p-val’][0] can also be wilcox.test or binom.test
o pg.power_ttest(effect, sample size, alpha, contrast = ‘one-sample’)
simulating versus fitting models
- models describe data and could reflect a true underlying (cognitive) process
- a parameter of a model a variable that can take a range of values that describe the data
o slope parameters, mean and standard deviation, drift-rates / non-decision times
- we choose parameters to simulate models to generate simulated data
- if we assume a model for data, a parameter can also be estimated from real data (and also
simulated data)
, linear regression
- models: ANOVA and regression
Bayesian model comparison
- a method to compare different models to determine which model fits the data best.
- Bayes-factor (BF)
o BFij smaller than 1 there is more evidence for model j than for model i
o BFij is equal to 1 no difference in evidence for the models
o BFij larger than 1 evidence for model i
o Evidence:
BF 1-3 = weak evidence
BF 3-20 = somewhat evidence
BF 20-150 = strong evidence
BF >150 = very strong evidence
Polynomial regression analysis
- Here we expect that one variable would be the predictive variable in two types of ways:
o In a linear way
o Quadratic way
- To simulate random noise example:
o Always use the normal distribution to simulate the noice
o x=np.linspace(0,2,N)
o y1=3-.2*x+.5*x^2 + stats.norm.rvs(0, 0.4, N)
the normal distribution simulates random noise along the line of
3-.2*x+.5*x^2
in python ^ is **
parameters are:
intercept parameter = 3
linear term = -.2
squared term = .5
ground truth = -.2 for linear model
ground truth = .5 for squared model
look in the results at coefficient of x1 and x2, compare these to the ground
truths to see if the model fits
o fitting by .fit()
overfitting and model comparison
- Occam’s razor: when faced with two opposing explanations for the same set of evidence our
minds will naturally prefer the explanation that makes the fewest assumptions
- R squared explains the variance:
o Between 0 and 1
o 0 means no variance is explained
o < 0.3 weak
o 0.3 – 0.5 moderate
o > 0.7 strong
o 1 means everything is explained, this is suspicious, most likely overfitting
