Data C8 Class notes

-1 to isolate growth rate, not growth multiplier

+1 when compounding




growth rates given start and end, want annual

100 * abs(difference between the values,
percent differences (w/no baseline)
divided by their average.)


second argument is base

log--natural log (base e)




second one

table methods




round round(#,ndigits = #)


Data types
"loud".upper()
string methods
'hitchhiker'.replace('hi', 'ma') arguments: txt to be replaced, replacement)




comparisons


Boolean




sum: sum of collection
len: length-- number of values in the array

When an array is combined with a single
arrays can be used in arithmetic expressions number, that number is combined with each
element of the array



ex: two arrays


Hard way: .item()




This is for computing the range (difference)
compute two arrays -->0 would be the first item in each array

Easy way: highs - lows
arrays

arithmetic operators, when acting on arrays of final result is an array
same size, preformed on each corresponding
pair of elements in the two arrays


**high is name of a array

highs.mean()
array methods
highs.sum()

highs.size



make_array()

places several values into array

Sequences functions ab array import numpy as np

computes the difference between each
np.diff(array_name)
adjacent pair of elements in an array.
numpy package--manipulate arrays




array of numbers in increasing or decreasing
order, separated by a regular interval

either 1,2,or 3 arguments

(start, end, 'step')
np.arange() function
Ranges if only 1 argument becomes end value, assume start=0, step=1

two argument give start & end, assume step=1

np.arrange(5) array([0,1,2,3,4])

range includes the start value (inclusive), does
not include end value (Exclusive)




need to use datascience module: from
datascience import *

table() empty table

makes new table with additional labeled
columns

to add even more columns, provide name and
array. (all columns must have same length)



Table().with_columns('name', make_array())




with_columns creates new table each time
called. original not affected



remember " " over file name!!!
Table.read_table(prob from csv)
first read data, then use show() method to show
the data into table

columns indexed 0,1,2....

table.column(4) to select 4th column
.column()
acess data in column becomes ARRAY

acess items in the column minard.column(4).item(0) item: acess specific element in array

table.relabeled('original','new_name')

methods plt_name.show() to display show(3) to show 3 rows

plt_name.num_columns returns the number of columns in that table

plt_name.num_rows returns the number of rows in that table

tbl.row(n) to obtain the nth row



row.item("column_name") select the element that corresponds to
column_name in a particular row.


plt_name.labels list of column names/labels

set_format('what u wanna change', Percent
for percents
formatter)
change formats
also CurrencyFormatter DateFormatter


table_name.select('column') more than one cones.select('Flavor', 'Price')

could also select(0,1) (indexing)

choosing sets of columns drop columns cones.drop('Color')

leaves original table unchanged

selecting selects TABLES, not arrays

sort(column_or_label, descending=False,
ex: table_name.sort('column')
Tables distinct=False)

creates a new table by arranging the rows of
Sorting rows
the original table in ascending order

for sorting in descending order cones.sort('Price', descending=True)

table.take(0) indexing!

indexing,
so 3->4th row

get 4,5,6 row actual is +1
Take specified rows

rows
combine! first sort, then take

creates new table consisting only of those rows

only select rows where the flavor is chocolate

label of column, value looking for
cones.where('Flavor', are.equal_to('chocolate'))
are.equal_to is optional

selecting rows

select multiple features: use "where" repeatedly

Two arguments: (find the row, criteria that
the row must meet)

Predicate: are.equal_to

range: includes left, exclude right

are.containing(s) : contains the string 's'
save typing




are.contained_in(array)


Table.read_table(data) read_table can read data directly from URL


0: total 1: male 2: female

for age, 100 rep those 100 + older 999--> sum of total pop(all age.)


Working with large tables of data
PROPORTIONS




remember to add array! when doing contained
preticate



after generating, need to add:
at here, year is x axis, pter is y axis
ex: pter_over_time.plot('Year', 'PTER')
Plots line plots

age is x axis


table_name.scatter('x', 'y)
scatterplots
use when looking for association + have non
sequential data

table_name.plot('x','y')
line plots
clear sequential order of x values, meaningul y
values

argument: ('column label/index', 'optional: specify bins u
want')




can use np.arange() to specify bins



.bin() same as group
bins=4 give 4 equally spaced bins
Numerical distribution
if dont specify any bins, default: 10 equally
wide bins between min & max of data

return two column table that contains number
of rows (counts) in each bin
binning


the last bin is just a end marker, it is exclusive
both sides. unless the value is exactly that


unless exactly 2000.


unequal bins


table_name.hist('column', bins= optional,optional
Histogram unit=' ')

both axis are numerical! can preform
arithmetic on them
graphing
area of each bar = percent of data values in
units of height: percent per 'unit on the x axis'
corresponding bin

same endpoint conversion as the bin method

distribution of quantitative variables not single value

height: area(or percent)/width

if only two column, one of category and one of
barh('label of category','label of frequencies') frequency... only need to specify label of
category: table.barh('flavour')

Subtopic 7

for single value!!!

unlike scatter plot & line plot (2 quant.
bar chart variable), bar chart has categories on one
axia & numerical quant on other

could also be plotting counts of one or two
categorical variables

Visualization display distribution of categorical variables

easier to interpret chart: first sort in decreasing
order, then, draw the bar chart
categorical distribution
count how frequently a variable appears in
table by calling the variable category and
collecting all rows in such category

.group() takes the label of column that contains
the category as argument




grouping method-> .group()
returns a table of counts of rows in each
category




lists categories in ascending order (a,b,c..1,2,3).
column of counts is called count

scatter and plot one column as the common horizontal axis

barh one column as common axis (set of category)


one common axis.




Overlaid scatterplots


each point rep row in table


sons_heights.scatter('son')
-> sons height become common x axis
overlaid graphs




Overlaid Line Plots




children.plot('AGE')




():specify the common axis of categories
usa_ca.barh('Ethnicity/Race')
Overlaid Bar Graphs




could first select (simplify table), generate
more simply graphs to better see comparison


argument to function can be any expression as
long as the value fits (even arrays)

add arguments thru ','
-remember ":"
-when calling the function: double(argument)
docstrings defined by ''' ''' --description of
-must be indented, lines after def
function
-function body ends at any unindented line

defining a function



use np.round bc arg is array not number

when using, ex: double(10)

The column values will be passed into the
corresponding arguments of the function.




apply a function that takes in 1 or multiple
arguments


apply function to column plt.apply()



apply the function to each value in said column
in the table.
table_name.apply(defined function, 'column')
returns an array of elements after function
is applied




1.defined a function that takes in 4 arg for ex.

when single argument-> counts # of rows for
each category in a column

names the function thats used to aggregate
values in other columns (fr all there rows)

(optional) second argument




this finds total price!

group creates array, then preform function
Functions and tables how it works
need to first clean the table

pass list of labels as argument to group




.group()

"single" argument


-> one row for every unique combination of
values

same as before, just that first argument is list of
two variables in []
two variables
two argument




group can go up to 3 or more categorical
variables! Just include them in list that is the
first argument

returns a table

values: column of values that will replace the
counts in each cell of the grid
-these values aren't displayed
.pivot(x, y, values, collect)
collect: indicates how to collect all values into
one aggregated value to be displayed in the
cell.

gives count of all rows of org. table that share
.pivot() if just .pivot(x,y)
combinations of column & row value




pivots, grouping, and joins
example




.join()
generate table w shared columns of tblA
and tblB, and other columns in the two
tables to form a new, merged, table.
-colA: name of column in tblA w/ values to join
on only contain info about items that appear in
-colB is name of shared column/used if column BOTH TABLES
name is different

can draw map by using Marker.map_table()

this function operates on tables whose column
are (in order): latitude, longitude, (optional)
identifier, optional 'colors', optional 'areas.




map_table
colour the points




'blue' means the colour is blue

add area to each point




sample from array

n: number of times to repeat process--return n
np.random.choice(array name, n) different random choice (number/array, NOT
table)
np.random.choice(array)
default: sample with replacement + returns an
array of items

if SRS, need to add, replace = False

sample from table

By default, it draws at random with
replacement from the rows of a tabl
Table.sample(sample size, with_replacement =False) vice
versa Sample takes in the sample size as its argument
and returns a table with only the rows that
were selected.

Differs from np.random.choice, which takes an
array and outputs a random value from the
array.

sampling from categorical distribution

samples at random with replacement from
a categorical distribution and returns the
proportions of sampled elements in each
category.

returns array containing sample proportions in
diff categories

Each time you run it you’ll get slightly different
proportions that fluctuate around the true
sample_proportions(sample size, population [0.25,0.5,0.25], illustrating the sampling
distribution (array/list)) variability.

sampling variability: Because each draw is
random, those observed proportions will
fluctuate/wiggle around the true values

mechanism: draws n random independent
samples

example

answer ONLY questions about proportions of
sampled individuals in different categories

simple random sample drawn without replacement

counts the number of non-zero values that
appear in an array
np.count_nonzero()
when array of booleon passed through, counts
number of true (True =1, False=0)




multi-line statement that allows Python to
choose among different alternatives based on
the truth value of an expression.

-remember the ':' after if
Randomness




conditional statements
last elif could be replaced by 'else', whose
body will be executed only if all previous
comparisons are false



example!

structure: begin w/ if header, followed by
indented body

repeat process multiple times


for < name of item> in <array>: followed by
indented lines of code that are repeated for
each element of the array.
For loops

indented body of for statement is executed
Iteration once for each item in that sequence

For loops to repeat lines of code multiple times


Simulations


use sequence np.arange(n) in for statement this repeats the process n times
Simulations and For Loops




np.append(array_name, value)

evaluates to a new array of ⬆️ agumented tgt

in for loops, usually change array while
.append() augmenting. -> done by assigning augmented
array to same name as original.




my_table.append(new_row) adds the new
row to the bottom of my_table.
monty_hall_game here is function that gives the
three values

Procedure:
··· Introduction to python Data 8
Simulation

Data science/Python
example
··· Introduction to statistics in Python



randomly choosing a position early in list &
systematic sample
evenly spaced positions after that

In the long run, after repeated sampling, prop
that event occurs gets closwer to the
theoretical probability of the event

*independently + identical conditions

the empirical histogram of the statistic is likely
law of average the empirical distribution of a large random to resemble the probability histogram of the
sample will resemble the distribution of the statistic, if the sample size is large and if you
population from which the sample was drawn. repeat the random sampling process numerous
times.
Sampling & Empirical Distribution
justify idea of using large random samples in
statistical inference

at median of x, 50% of populatian had ___ of x
np.median(array)
or fewer




Simulating a statistic




sum(np.abs(distribution 1 - distribution 2)) / 2




total variation distance




combine-> simulation




can use TVD (total variation distance) for only two categories
sample statistic -more than two categories

observed sample statistic is one number,
remember to sum before /2
Testing Hypothesis




FINDING P VALUE

two sided test (not equal) need abs for test
statistic

error probability type one error: reject when null is actually true error probability is same as the percent cuttoff)

1. find function that simulated one value of test
2. use for loops to simulate multiple values
statistic




choosing test statistic

if there is no difference between two
distribution in underlying population, then
labels should make no difference to statistic.

idea: shuffle all labels randomly

If the null hypothesis is correct, the two
groups (A and B) come from the same
underlying distribution. Therefore,
reordering (shuffling) the labels should not
systematically affect the differences in
shuffling ensures count of each label not group means. We shuffle labels in an A/B
change, but is important for comparability of test to simulate the null hypothesis,
simulated differences creating randomness that represents the
scenario in which the assignment to groups
is purely by chance, allowing us to measure
how likely our observed difference could
random permutation arise by random chance alone.


after shuffle, create new simulated value of should be close to 0 because randomization ->
test statistic under null hypothesis roughly equal distribution

dont need sample size, default: all rows
Use table method:
tbl.sample(with_replacement = False)
then add the column of shuffled label to table

after for loop of permutationed test statistic,
create histogram of prediction under null. then
use point of observed statistic to determine
significance

compare the two distributions (shuffled test
statistic and original)--> use p value and cutoff np.count_nonzero(differences <= observed_difference) / num_rep
determine conclusion

whether a change (A or B) causes difference in
a measurable outcome

primarily about inferring causality-- determine goes beyond detecting difference by using
whether something causes a difference in randomization (permutations) to control
outcome (not JUST whether smt is significant or coonfounding variables. -> attribute observed
diff) effect to change itself
Comparing Two samples A/B testing
randomly assigning to A or B -> groups are
similar for treatment -> allow causation




permutation test

Because of unbiased assignment of individuals
to treatment and control groups, differences
in outcomes of two groups can conclude
causation




when deeling with 1's and 0's. Proportion is the
average of the 0's and 1's
causality




random assign,ent ensures no confounding
variables

because in A/B trials are randomized
if not randomly assigned, still point to
(permutation), so it is evidence that the
association
treatment caused the difference

only with randomization can causation be called

i am __ % confident that the interval ( , )
captures the true population ___ (in context)

percentile(rank between 0-100, array) return a corresponding percentile of the array

generate new random samples through
resampling: drawing random samples from
original sample

important to resample same number of times as
original sample size (bc variability depends on
size of sample)

but, if draw w/o replacement, just get same
sample back, so we draw with replacement


bootstrap bootstrap works bc by law of average, dis of
original sample likley to resemble the
population, so dis of resamples also likley to
sample method default with replaement, and
resemble orig sample-> resemble pop as well
w/o specifying sample size default is all rows in
table

use for loops to generate one value of the
bootstrapped median based on original sample



bootstrap empirical distribution
first define function that returns one
bootstrapped median




for loop for distribution

interval of estimates 95% confident interval

A 95% confidence level means that if we were
to repeat the process of taking random
samples many times, and compute confidence
confidence level intervals from each sample using the same
method, about 95% of those intervals would
contain the true (population) parameter.
Estimation



95 confidence interval: bootstrap sample 5000
times-> 5000 estimates of median. and 95%
conf interval will be the middle 95% of all
estimates




confidence intervals

if you want a 80 percent conf interval then just
take 10% dis from each side of tail




in table, its True/False -> 0 and 1
Proportion is average of 0 and 1

mean of 0's and 1's are also proportion. ->
np.mean(____ <#) -> in brackets, make it into
true's/false aka 0,1

goal is estimating max or min/ parameters
influenced by extreme/rare values

Bootstrap percentile method is NOT probaility distribution of statistic is not roughly
expected in bell shaped

original sample is very small (law of avg, not
rep of pop)

5 % cutoff = 95% confidence level

use this to reject or retain null, if out of interval,
reject null

comparing baseline and after of one subject ->
scores are paired.


using confidence intervals




The 99% confidence interval for the average
drop in the population goes from about 17 to
about 40. The interval doesn’t contain 0. So we
reject the null hypothesis.

here, CI is wide because confidence level99% is
high & sample size rel. small

sum of deviations from average is always 0 so we square the deviation

variability variance is mean of squared deviations deviation: value - mean

square root variance to get SD

root mean square of deviations from average

np.std(array of values)




aka z score
standard deviation deviation/sd : xx SD above/below average
standardization, z scores are standard units




Center and spread



Chebychev's bounds-- these values are
"atleast"-- bouds

SD is a statistic, estimator: so sample sd
becomes more consistent with population
SD with increasing sample size when same w/ sample mean
working on raw data




SE is for sampling distribution, SD is for
The SD of the sample means is inversely population!! SE is sd over root of n! Use this
proportional to the square root of sample logic to answer questions
size.
carefull. This is only for sd (at here, se) of
sample mean/proportion (sampling
distribution), NOT raw data (raw data sd will
only become closer to pop sd)

point of inflection, also where z=1 (which is aka
how to spot sd on bell shaped curve
one sd above average)




point of inflection is 1 and -1
standard normal curve
symmetric, mean and median both 0


mean is 0, sd is 1

input: one number, return area under curve to
stats.norm.cdf(#)
left of the number

1-stats.norm.cdf then this would be area to the right

Normal Distribution function for area under normal curve




this rule is from normal distribution
68-95-99.7 rule
approximation




when you draw a large random sample with
replacement from the population.

Expected sample mean (average of sample
centeral limit theorum (CLT) distribution of means) = exactly the population
mean 𝜇

increasing number of resamples-> sd stays
about the same
Central Limit theorum
curve is bell shaped/normal if sample size big
enough

makes it possible to make inferences ab
population with little information when we have
a large random sample

Because formula SE equal sd over sqrt n, the
variability of the sample mean decreases as the
sample size increase (accuracy inc)

to increase accuracy by factor of 10, need to n is in sqrt, sample size increase by factor of 4,
multiply sample size by 100. SE only go down by 2 (sqrt4)

CI tells us how much we need to times
Variability of Sample Mean know that in + - 2 SD contains 95% of data
2 SD from both sides, so x4




Choosing sample size
in this example, use + - 2sd, and width of
interval should be less/equal 1percent

takes positive integer as argument, returns array
np.ones
consisting of that many 1's

the SD of population of 1's and 0's is at most 0.5 when 50% of pop is 1, other 0

measure of how tightly clustered a scatter
linear association
diagram is about. straight line

measures the strength of linear relationship
between 2 variables--how clustered scatter
diagram is around straight line

r is based on standard units
denote by r
not affected by changing units on either axis or switching axia does not change amount of
switching the axes clustering or sign of association

correlation coefficient r is a number between -1 and 1

measures extent to which scatter plot
custers around straight line

Correlation r=1, scatter plot is a perfect straight line sloping
when r=0, formeless...uncorrelated
upwards. r = -1, same but downwards

takes value of r as argument and simulates a
r_scatter()
scatterplot with correlation very close to r

correlation does not equal causation

correlation only measures linear variables with perfect quadratic relation may
association.!!! have correlation of 0

Correlation is affected by outliers




t= column, xy is axis
calculate correlation
correlation is the slope of the regression
line when the data are put in standard units.

takes r as argument
regression_line(r)
returns regression line of that corresponding r
value




regression to the mean

we predict that the child will be somewhat
closer to average than the parents were.

individuals who are away from average on one
variable are expected to be not quite as far
away from average on the other.

regression effect is a statement about
regression effect
averages

says that if you take all children whose
midparent height is 1.5 standard units, then
the average height of these children is doesn't state about all the children, but the avg
The regression line
somewhat less than 1.5 standard units.



slope of regression line in original units units are basically y/x




use value of x to predict value of y intercept is basically predicted y when x=0

original units: the unit is units of y

In standard units, the intercept for the
regression line is 0.

models predictions for each value
fitted values
draw the fit_line through fit_line = True in table
method scatter

for standard units there are no units bc z score
has no units

error: actual - predicted value

(np.mean(error**2))**0.5
root mean squared error (RMSE)
units: same as units of variable being predicted
(y)

Least squares 'better' lines have smaller errors

The regression line is a unique straight line that
minimizes RMSE

to find arguments of a function for which the
function returns its minimum value
Minimizing RMSE minimize()
argument of minimize() is a function itself that
takes numerical arguments and returns a
numerical value


same minimizers as RMSE


residual = observed - predicted




The residual plot of a good regression
shows no pattern. The residuals look about
the same, above and below the horizontal
line at 0, across the range of the predictor
variable.




When a residual plot shows a pattern, there
may be a non-linear relation between the
variables.




If the residual plot shows uneven variation
about the horizontal line at 0, the regression
Residuals and Diagnostics
estimates are not equally accurate across
the range of the predictor variable.




No matter what the shape of the scatter
average of residual
diagram, the average of the residuals is 0.

For every linear regression, whether good or
bad, the residual plot shows no trend. Overall, it
is flat. In other words, the residuals and the
predictor variable are uncorrelated.

If you plotted residuals against
residual plots show no trend 𝑥 and fit a straight line through them, its slope
would be zero.

if r = 0, regression line would be a flat line
at average of y




No matter what the shape of the scatter plot,
the SD of the residuals is a fraction of the SD of
the response variable.

SD of residuals
average of residual is 0, so the smaller the sd
of residuals, the closer the residuals are to 0

aka, if sd of residual is small, the overall size
of errors in regression also small

predictors nearer to center of distribution are
less variable. Likewise, predictions at predictors
further from center of dis is more variable
Regression Inference
do so by drawing scatter diagram and
first step must first decide whether regression determine
model even holds for data
then run diagnostics thru residual plot

make predictions from past examples

each individual or situation where we liked to
make a prediction
observation
ex: an order

observation has multiple attributes ex: total value of order/voters anual salary

answer to question we care about (voting fr u
or not, fraudulent or not)
each observation has class
0 or 1, ex: 0 means not fraudulent

Observe observations attributes, and try to
Classification
predict class using those attributes

to predict observation thats not in
scatterplot/data set, we find the observation in
training set thats closest/nearest to our
observation, and use that observations class as
our prediction for our observation.

intuition: if two points are near each other in the
sctterplot, then the corresponding
measurements are similar, so we expect them to
have similar class




Nearest Neighbor Classifier




example

to predict, rather than looking at one data point
closest to observation, look at 3(multiple)
points closest. And use class of the 3 data
points to predict class (use majority value)
K nearest neighbor classifier
often, pick number k. predicted class will be
based on the k data points in training set
closest to observation

tbl.row(index)

table method: row row has their own data type

can use item to acess specific elements of a row use a label in .item()

convert rows to arrays np.array converts any sequential object into array

between observation point and data point

calculate distance between two points




Implementing Classifiers k nearest neighbors




distance formula in 3d space




full example




accuracy of classifers


accuracy example

a numerical output is predicted from numerical
input attributes by multiplying each attribute
value by a different slope, then summing the
results.



Predicting the sale price involves multiplying
each attribute by the slope and summing the
result.
Screenshot 2025-08-10 at 17.50.28.png

if function we want to minimize takes in array,
must pass array = True argument to minimize

minimize also requires an initial guess of the
use minimize function to find slopes with lowest
Multiple linear regression slopes so that it knows the dimension of the
RMSE. Find best slope
input array.

add smooth=True




The RMSE of around $30,000 means that our
best linear prediction of the sale price based on
all of the attributes is off by around $30,000 on
the training set, on average

nearest neigbor prediction for price, just average price of nn

Chebyshev's Inequality (for any distribution)
estimation
Only use 68.95,99.7 rule for normal distributions

Getting additional info! ex: round? 1


it removes a specified character from the start
.strip()
or end of a string

add info 5.325e+07 is basically 5.325 * 10**7

53 to be in millions just *10**6