CS7641 Problem Set Fal
i i i
l i
Instructions
Thisi problemi seti isi noti ai parti ofii youri finali gradei buti ratheri ai meansi toi helpi youi withi thei finali exam.i Ifii alli ofitheip
roblemsiareiattemptediandiyourigradeiisiarounditheicutoff,iweiwilliroundiupitoitheihigheriletterigrade.i Youi willi needi t
oi attempti eachi problemi andi submiti youri solutionsi oni Canvas.i Wei willi verifyi worki isi submittediatitheiendiofitheiter
m.i Afteritheideadline,iweiwilliprovideisolutionsiforiyouitoicompareiyourianswers.i WeiplanitoiholditwoiOfficeiHours,i
oneiforieachipartiofitheiproblemisetibeforeitheifinal.
Parti One
1. Whereiweiareidoingisupervisedilearning,iweihaveimostlyiassumediaideterministicifunction.i Imagineiinsteadiai
worldiwhereiweiareitryingitoicaptureiainon-
deterministicifunction.i Inithisicase,iweimightiseeitrainingipairsiwhereitheixivalueiappearsiseveralitimes,ibutiwit
hidifferentiyivalues.i Foriexample,iweimightiusei attributesi ofii humansi toi thei probabilityi thati theyi havei hadi chi
ckeni pox.i Ini thati case,i wei mighti seeitheisameikindiofipersonimanyitimesibutionlyisometimesitheyimayihaveih
adichickenipox.i Weiwouldilikei toi buildi ai learningi algorithmi thati willi computei thei probabilityi thati ai personi
hasi chickeni pox.i So,i giveniaisetiofitrainingidataiwhereieachiinstanceiisimappeditoi1iforitrueiori0iforifalse:
(a) DeriveitheiproperierrorifunctionitoiuseiforifindingitheiMLihypothesisiusingiBayes’iRule.i Youishouldi goit
hroughiaisimilariprocessiasitheioneiuseditoideriveileastisquaredierroriinitheilessons.
(b) Compareiandicontrastiyouriresultitoitheiruleiweiderivediforiaideterministicifunctioniperturbedibyizero-
meanigaussianinoise.i Whatiwouldiainormalineuralinetworkiusingisumiofisquaredierrorsidoi withitheseidat
a?i Whatiifitheidataiconsistediofix,iyipairsiwhereiyiwasianiestimateiofitheiprobabilityiinsteadiofi0siandi1s?
2. Designiaitwo-inputiperceptronithatiimplementsitheibooleanifunctioniAi∧i¬B.i Designiaitwo-
layerinetworkiofiperceptronsithatiimplementsiAi⊕iBi (wherei⊕iisiXOR).
3. Deriveitheiperceptronitrainingiruleiandigradientidescentitrainingiruleiforiaisingleiunitiwithioutputio,iwherei
oi=iw0i+iw1x1i+iw1x2i+i.i.i.i+iwnxni+iwnx2i.i Whati arei thei advantagesi ofii usingi gradienti descent
1 n
trainingi rulei fori trainingi neurali networksi overi thei perceptroni trainingi rule?
4. Explaini howi onei cani usei Decisioni Treesi toi performi regression?
i Showi thati wheni thei errori functioniisisquaredierrorithatitheiexpectedivalueiatianyileafiisitheimean.i Takeithei
BostoniHousingidataseti(http://lib.stat.cmu.edu/datasets/
boston)i andi usei Decisioni Treesi toi performi regression.
5. SuggestiailazyiversioniofitheieageridecisionitreeilearningialgorithmiID3.i Whatiareitheiadvantagesiandidisadvan
tagesiofiyourilazyialgorithmicompareditoitheioriginalieagerialgorithm?
6. Imagineiyouihadiailearningiproblemiwithianiinstanceispaceiofipointsionitheiplaneiandiaitargetifunctionithatiyoui
knewitookitheiformiofiailineionitheiplaneiwhereiallipointsionioneisideiofitheilineiareipositiveiandiallithoseionithei
otheriareinegative.i Ifiyouiwereiconstraineditoionlyiuseidecisionitreeiorinearest-
neighborilearning,iwhichiwouldiyouiuse?i Why?
7. Givei thei VCi dimensioni ofii thesei hypothesisi spaces,i brieflyi explainingi youri answers:
(a) Ani origin-centeredi circlei (2D)
(b) Aniorigin-centeredispherei(3D)
i
, 8. Youihaveitoicommunicateiaisignaliiniailanguageithatihasi3isymbolsiA,iBiandiC.iTheiprobabilityiofiobservi
ngiAiisi50%iwhileithatiofiiobservingiBiandiCiisi25%ieach.i Designianiappropriateiencodingiforithisilanguag
e.i Whatiisitheientropyiofithisisignaliinibits?
9. ShowithatitheiK-
meansiprocedureicanibeiviewediasiaispecialicaseiofitheiEMialgorithmiapplieditoianiappropriateimixtureiofiGaus
sianidensitiesimodel.
10. Ploti thei directioni ofii thei firsti andi secondi PCAi componentsi ini thei figuresi given.
11. Whichi clusteringi method(s)i isi mosti likelyi toi producei thei followingi resultsi ati ki=i2?
i Choosei thei mostilikelyimethod(s)iandibrieflyiexplainiwhyiit/
theyiwilliworkibetteriwhereiothersiwillinotiiniatimosti3isentences.
• Hierarchicali clusteringi withi singlei link
• Hierarchicali clusteringi withi completei link
• Hierarchicali clusteringi withi averagei link
• K-means
• EM
(a) (b) (c)
2
i
i i i
l i
Instructions
Thisi problemi seti isi noti ai parti ofii youri finali gradei buti ratheri ai meansi toi helpi youi withi thei finali exam.i Ifii alli ofitheip
roblemsiareiattemptediandiyourigradeiisiarounditheicutoff,iweiwilliroundiupitoitheihigheriletterigrade.i Youi willi needi t
oi attempti eachi problemi andi submiti youri solutionsi oni Canvas.i Wei willi verifyi worki isi submittediatitheiendiofitheiter
m.i Afteritheideadline,iweiwilliprovideisolutionsiforiyouitoicompareiyourianswers.i WeiplanitoiholditwoiOfficeiHours,i
oneiforieachipartiofitheiproblemisetibeforeitheifinal.
Parti One
1. Whereiweiareidoingisupervisedilearning,iweihaveimostlyiassumediaideterministicifunction.i Imagineiinsteadiai
worldiwhereiweiareitryingitoicaptureiainon-
deterministicifunction.i Inithisicase,iweimightiseeitrainingipairsiwhereitheixivalueiappearsiseveralitimes,ibutiwit
hidifferentiyivalues.i Foriexample,iweimightiusei attributesi ofii humansi toi thei probabilityi thati theyi havei hadi chi
ckeni pox.i Ini thati case,i wei mighti seeitheisameikindiofipersonimanyitimesibutionlyisometimesitheyimayihaveih
adichickenipox.i Weiwouldilikei toi buildi ai learningi algorithmi thati willi computei thei probabilityi thati ai personi
hasi chickeni pox.i So,i giveniaisetiofitrainingidataiwhereieachiinstanceiisimappeditoi1iforitrueiori0iforifalse:
(a) DeriveitheiproperierrorifunctionitoiuseiforifindingitheiMLihypothesisiusingiBayes’iRule.i Youishouldi goit
hroughiaisimilariprocessiasitheioneiuseditoideriveileastisquaredierroriinitheilessons.
(b) Compareiandicontrastiyouriresultitoitheiruleiweiderivediforiaideterministicifunctioniperturbedibyizero-
meanigaussianinoise.i Whatiwouldiainormalineuralinetworkiusingisumiofisquaredierrorsidoi withitheseidat
a?i Whatiifitheidataiconsistediofix,iyipairsiwhereiyiwasianiestimateiofitheiprobabilityiinsteadiofi0siandi1s?
2. Designiaitwo-inputiperceptronithatiimplementsitheibooleanifunctioniAi∧i¬B.i Designiaitwo-
layerinetworkiofiperceptronsithatiimplementsiAi⊕iBi (wherei⊕iisiXOR).
3. Deriveitheiperceptronitrainingiruleiandigradientidescentitrainingiruleiforiaisingleiunitiwithioutputio,iwherei
oi=iw0i+iw1x1i+iw1x2i+i.i.i.i+iwnxni+iwnx2i.i Whati arei thei advantagesi ofii usingi gradienti descent
1 n
trainingi rulei fori trainingi neurali networksi overi thei perceptroni trainingi rule?
4. Explaini howi onei cani usei Decisioni Treesi toi performi regression?
i Showi thati wheni thei errori functioniisisquaredierrorithatitheiexpectedivalueiatianyileafiisitheimean.i Takeithei
BostoniHousingidataseti(http://lib.stat.cmu.edu/datasets/
boston)i andi usei Decisioni Treesi toi performi regression.
5. SuggestiailazyiversioniofitheieageridecisionitreeilearningialgorithmiID3.i Whatiareitheiadvantagesiandidisadvan
tagesiofiyourilazyialgorithmicompareditoitheioriginalieagerialgorithm?
6. Imagineiyouihadiailearningiproblemiwithianiinstanceispaceiofipointsionitheiplaneiandiaitargetifunctionithatiyoui
knewitookitheiformiofiailineionitheiplaneiwhereiallipointsionioneisideiofitheilineiareipositiveiandiallithoseionithei
otheriareinegative.i Ifiyouiwereiconstraineditoionlyiuseidecisionitreeiorinearest-
neighborilearning,iwhichiwouldiyouiuse?i Why?
7. Givei thei VCi dimensioni ofii thesei hypothesisi spaces,i brieflyi explainingi youri answers:
(a) Ani origin-centeredi circlei (2D)
(b) Aniorigin-centeredispherei(3D)
i
, 8. Youihaveitoicommunicateiaisignaliiniailanguageithatihasi3isymbolsiA,iBiandiC.iTheiprobabilityiofiobservi
ngiAiisi50%iwhileithatiofiiobservingiBiandiCiisi25%ieach.i Designianiappropriateiencodingiforithisilanguag
e.i Whatiisitheientropyiofithisisignaliinibits?
9. ShowithatitheiK-
meansiprocedureicanibeiviewediasiaispecialicaseiofitheiEMialgorithmiapplieditoianiappropriateimixtureiofiGaus
sianidensitiesimodel.
10. Ploti thei directioni ofii thei firsti andi secondi PCAi componentsi ini thei figuresi given.
11. Whichi clusteringi method(s)i isi mosti likelyi toi producei thei followingi resultsi ati ki=i2?
i Choosei thei mostilikelyimethod(s)iandibrieflyiexplainiwhyiit/
theyiwilliworkibetteriwhereiothersiwillinotiiniatimosti3isentences.
• Hierarchicali clusteringi withi singlei link
• Hierarchicali clusteringi withi completei link
• Hierarchicali clusteringi withi averagei link
• K-means
• EM
(a) (b) (c)
2
i
