CS 7643 QUIZ 3 EXAM WITH
ACCURATE SOLUTIONS
Modeling bError b- bcorrect banswers b-Given ba bparticular bNN barchitecture, bthe bactual
bmodel bthat brepresents bthe breal bworld bmay bnot bbe bin bthat bspace.
When bmodel bcomplexity bincreases, bmodeling berror breduces, bbut boptimization
berror bincreases.
Estimation bError b- bcorrect banswers b-Even bif bfinding bthe bbest bhypothesis, bweights,
band bparameters bthat bminimize btraining berror, bmay bnot bgeneralize bto btest bset
Optimization bError b- bcorrect banswers b-Even bif byour bNN bcan bperfectly bmodel bthe
bworld, byour balgo bmay bnot bfind bgood bweights bthat bmodel bthe bfunction.
When bmodel bcomplexity bincreases, bmodeling berror breduces, bbut boptimization
berror bincreases.
, Effectiveness bof btransfer blearning bunder bcertain bconditions b- bcorrect banswers b-
Remove blast bFC blayer bof bCNN band binitialize bit brandomly, bthen brun bnew bdata
bthrough bnetwork bto btrain bonly bthat blayer
In border bto btrain bthe bNN bfor btransfer blearning b-freeze bthe bCNN blayers bor bearly
blayers band blearn bparameters bin bthe bFC blayers.
Performs bvery bwell bon bvery bsmall bamount bof btraining, bif bsimilar bto bthe boriginal
bdata
Does bnot bwork bvery bwell bif bthe btarget btask's bdataset bis bvery bdifferent
If byou bhave benough bdata bin bthe btarget bdomain, band bis bdifferent bthan bthe
bsource, bbetter bto bjust btrain bon bthe bnew bdata
Transfer blearning b= breuse bfeatures bwe blearn bon ba bvery blarge bdataset bon ba
bcompletely bnew bthing
Steps:
Train bon bvery blarge bdataset
Take bcustom bdataset band binitialize bnetwork bwith bweights btrained bin bStep b1
b(replace blast bfully bconnected blayer bsince bclasses bin bnew bnetwork bwill bbe
bdifferent)
Final bstep b-> bcontinue btraining bon bnew bdataset
Can beither bretrain ball bweights b("finetune") bor bfreeze b(ie: bnot bupdate) bweights bin
bcertain blayers b(freezing breduces bnumber bof bparameters bthat byou bneed bto blearn)
AlexNet b- bcorrect banswers b-
2x(CONV=>MAXPOOL=>NORM)=>3xCONV=>MAXPOOL=>3xFC
ReLU, bspecialized bnormalization blayers, bPCA-based bdata baugmentation, bDropout,
bEnsembling b(used b7 bNN bwith bdifferent brandom bweights)
Critical bdevelopment: bMore bdepth band bReLU
VGGNet b- bcorrect banswers b-2x(2xCONV=>POOL)=>3x(3xCONV=>POOL)=>3xFC
Repeated bApplication bof b3x3 bConv b(stride bof b1, bpadding bof) b& b2x2 bMax bPooling
b(stride b2) bblocks
Very blarge bnumber bof bparameters b(most bin bFC) blayers, bmost bmemory bin bConv
bLayers b(you bare bstoring bactivation bproduced bin bforward bpass)
Critical bDevelopment: bBlocks bof brepeated bstructures
Inception bNet b- bcorrect banswers b-Deeper band bmore bcomplex bthan bVGGNet
Average bPooling bbefore bFC bLayer
Repeated bblocks bthat bare brepeated bover bagain bto bform bNN
Blocks bare bmade bof bsimple blayers, bFC, bConv, bMaxPool, band bsoftmax
Parallel bfilters bof bdifferent bsizes bto bget bfeatures bat bmultiple bscales
Critical bDevelopment: bBlocks bof bparallel bpaths
Uses bNetwork bIn bNetwork bconcept bi.e b1x1 bConvolution b-sort bof bDimensionality
breduction bsee bslide
Negative bthings: bIncreased bComputational bWork
ACCURATE SOLUTIONS
Modeling bError b- bcorrect banswers b-Given ba bparticular bNN barchitecture, bthe bactual
bmodel bthat brepresents bthe breal bworld bmay bnot bbe bin bthat bspace.
When bmodel bcomplexity bincreases, bmodeling berror breduces, bbut boptimization
berror bincreases.
Estimation bError b- bcorrect banswers b-Even bif bfinding bthe bbest bhypothesis, bweights,
band bparameters bthat bminimize btraining berror, bmay bnot bgeneralize bto btest bset
Optimization bError b- bcorrect banswers b-Even bif byour bNN bcan bperfectly bmodel bthe
bworld, byour balgo bmay bnot bfind bgood bweights bthat bmodel bthe bfunction.
When bmodel bcomplexity bincreases, bmodeling berror breduces, bbut boptimization
berror bincreases.
, Effectiveness bof btransfer blearning bunder bcertain bconditions b- bcorrect banswers b-
Remove blast bFC blayer bof bCNN band binitialize bit brandomly, bthen brun bnew bdata
bthrough bnetwork bto btrain bonly bthat blayer
In border bto btrain bthe bNN bfor btransfer blearning b-freeze bthe bCNN blayers bor bearly
blayers band blearn bparameters bin bthe bFC blayers.
Performs bvery bwell bon bvery bsmall bamount bof btraining, bif bsimilar bto bthe boriginal
bdata
Does bnot bwork bvery bwell bif bthe btarget btask's bdataset bis bvery bdifferent
If byou bhave benough bdata bin bthe btarget bdomain, band bis bdifferent bthan bthe
bsource, bbetter bto bjust btrain bon bthe bnew bdata
Transfer blearning b= breuse bfeatures bwe blearn bon ba bvery blarge bdataset bon ba
bcompletely bnew bthing
Steps:
Train bon bvery blarge bdataset
Take bcustom bdataset band binitialize bnetwork bwith bweights btrained bin bStep b1
b(replace blast bfully bconnected blayer bsince bclasses bin bnew bnetwork bwill bbe
bdifferent)
Final bstep b-> bcontinue btraining bon bnew bdataset
Can beither bretrain ball bweights b("finetune") bor bfreeze b(ie: bnot bupdate) bweights bin
bcertain blayers b(freezing breduces bnumber bof bparameters bthat byou bneed bto blearn)
AlexNet b- bcorrect banswers b-
2x(CONV=>MAXPOOL=>NORM)=>3xCONV=>MAXPOOL=>3xFC
ReLU, bspecialized bnormalization blayers, bPCA-based bdata baugmentation, bDropout,
bEnsembling b(used b7 bNN bwith bdifferent brandom bweights)
Critical bdevelopment: bMore bdepth band bReLU
VGGNet b- bcorrect banswers b-2x(2xCONV=>POOL)=>3x(3xCONV=>POOL)=>3xFC
Repeated bApplication bof b3x3 bConv b(stride bof b1, bpadding bof) b& b2x2 bMax bPooling
b(stride b2) bblocks
Very blarge bnumber bof bparameters b(most bin bFC) blayers, bmost bmemory bin bConv
bLayers b(you bare bstoring bactivation bproduced bin bforward bpass)
Critical bDevelopment: bBlocks bof brepeated bstructures
Inception bNet b- bcorrect banswers b-Deeper band bmore bcomplex bthan bVGGNet
Average bPooling bbefore bFC bLayer
Repeated bblocks bthat bare brepeated bover bagain bto bform bNN
Blocks bare bmade bof bsimple blayers, bFC, bConv, bMaxPool, band bsoftmax
Parallel bfilters bof bdifferent bsizes bto bget bfeatures bat bmultiple bscales
Critical bDevelopment: bBlocks bof bparallel bpaths
Uses bNetwork bIn bNetwork bconcept bi.e b1x1 bConvolution b-sort bof bDimensionality
breduction bsee bslide
Negative bthings: bIncreased bComputational bWork
