SUCCES MET BIG
DATA
Semester 4
Julia Schouten
, INHOUDSOPGAVE
1 Inleiding ...................................................................................................................................................... 4
1.1 Waarom? ...................................................................................................................................................... 4
1.2 Waarover? .................................................................................................................................................... 4
1.3 Voor wie? ...................................................................................................................................................... 4
1.4 Hoe?.............................................................................................................................................................. 4
2 Big data ....................................................................................................................................................... 4
2.1 Inleiding ........................................................................................................................................................ 4
2.2 Geschiedenis ................................................................................................................................................. 5
2.3 beschrijving ................................................................................................................................................... 5
2.4 relaties .......................................................................................................................................................... 6
2.5 toepassingen ................................................................................................................................................ 6
2.5.1 algemeen............................................................................................................................................... 6
2.5.2 Google ................................................................................................................................................... 6
2.5.3 CERN ...................................................................................................................................................... 6
2.5.4 NASA ..................................................................................................................................................... 6
2.5.5 Facebook ............................................................................................................................................... 7
2.5.6 Game-company ..................................................................................................................................... 7
3 Oplsag ......................................................................................................................................................... 7
3.1 Inleiding ........................................................................................................................................................ 7
3.2 Dataontwerp ................................................................................................................................................ 7
3.2.1 voorwaarden ......................................................................................................................................... 7
3.2.2 Beschrijvingen ....................................................................................................................................... 7
3.3 Dataoplsag ................................................................................................................................................... 8
3.3.1 ontstaan ................................................................................................................................................ 8
3.3.2 Database ............................................................................................................................................... 8
3.4 van data naar informatie.............................................................................................................................. 8
3.5 opslag van informatie............................................................................................................................ 9
3.5.1 soorten opslag ....................................................................................................................................... 9
3.5.2 Datawarehouse en datamart .............................................................................................................. 10
3.5.3 Datalake en datareservoir ................................................................................................................... 10
4 proces ....................................................................................................................................................... 10
4.1 inleiding ...................................................................................................................................................... 10
4.2 Data-analyse............................................................................................................................................... 10
4.2.1 Methodiek ........................................................................................................................................... 10
4.2.2 Reporting............................................................................................................................................. 11
4.2.3 Online analytical processing (OLAP) .................................................................................................... 11
4.2.4 zoekopdrachten .................................................................................................................................. 11
4.3 Datamining ................................................................................................................................................. 11
4.4 Big data-proces ........................................................................................................................................... 11
4.4.1 algemeen............................................................................................................................................. 11
4.4.2 Parallel verwerken .............................................................................................................................. 12
4.4.3 Key value ............................................................................................................................................. 12
Pagina 1 van 36
, 4.4.4 Map reduce ......................................................................................................................................... 12
4.4.5 Verwerken Big Data............................................................................................................................. 12
4.5 Predictive analytics ..................................................................................................................................... 13
4.6 CAP-theorema ............................................................................................................................................ 13
5 beslisboom ................................................................................................................................................ 13
5.1 Inleiding ...................................................................................................................................................... 13
5.2 het probleem .............................................................................................................................................. 13
5.3 de oplossing ................................................................................................................................................ 13
5.4 de beperkingen ........................................................................................................................................... 14
6 Neutrale netwerken .................................................................................................................................. 14
6.1 inleiding ...................................................................................................................................................... 14
6.3 de oplossing ................................................................................................................................................ 16
6.4 de beperkingen ........................................................................................................................................... 17
7 clusteren ................................................................................................................................................... 17
7.1 inleiding ...................................................................................................................................................... 17
7.2 het probleem .............................................................................................................................................. 18
7.3 de oplossing ................................................................................................................................................ 18
7.3.1 Algemeen ............................................................................................................................................ 18
7.3.2 Hiërarchisch clusteren......................................................................................................................... 18
7.3.3 puntsgewijs clusteren ......................................................................................................................... 18
7.3.4 dichtheidsgewijs clusteren .................................................................................................................. 19
7.3.5 validatie ............................................................................................................................................... 19
7.4 de beperkingen ........................................................................................................................................... 20
8 lineaire regressie ....................................................................................................................................... 20
8.1 Inleiding ...................................................................................................................................................... 20
8.2 het probleem .............................................................................................................................................. 21
8.3 de oplossing ................................................................................................................................................ 21
8.4 de beperkingen ........................................................................................................................................... 22
9 naaste buur ............................................................................................................................................... 23
9.1 inleiding ...................................................................................................................................................... 23
9.2 het probleem .............................................................................................................................................. 23
9.3 de oplossing ................................................................................................................................................ 24
9.4 de beperkingen ........................................................................................................................................... 24
10 Regels afleiden ........................................................................................................................................ 26
10.1 De inleiding ............................................................................................................................................... 26
10.2 Het probleem ............................................................................................................................................ 26
10.3 de oplossing .............................................................................................................................................. 27
10.3.1 Werken met bedrijfsregels ................................................................................................................ 27
10.3.2 Bestaande business rules .................................................................................................................. 28
Pagina 2 van 36
, 10.3.3 Nieuwe bedrijfsregels ....................................................................................................................... 28
10.4 de beperkingen ......................................................................................................................................... 30
10.4.1 Toepassingen .................................................................................................................................... 30
10.4.2 Beslissingsboom versus afleiden van regels ...................................................................................... 30
11 zin en onzin ............................................................................................................................................. 31
11.1 inleiding .................................................................................................................................................... 31
11.2 Kwaliteit.................................................................................................................................................... 31
11.2.1 algemeen........................................................................................................................................... 31
11.2.2 kwaliteit van data .............................................................................................................................. 32
11.2.3 kwaliteit van informatie .................................................................................................................... 32
11.2.4 kwaliteit van testen ........................................................................................................................... 32
11.2.5 kwaliteit van het ontwerp ................................................................................................................. 33
11.2.6 kwaliteit van bedrijfsplan .................................................................................................................. 33
11.3 presentatie................................................................................................................................................ 33
11.4 overdenkingen .......................................................................................................................................... 34
Ethiek en big data......................................................................................................................................... 34
12.1 inleiding .................................................................................................................................................... 34
12.2 ethiek ........................................................................................................................................................ 34
12.3 het probleem ............................................................................................................................................ 34
12.4 oplossing? ................................................................................................................................................. 35
12.5 reflectie ..................................................................................................................................................... 35
13 uitleiding ................................................................................................................................................. 36
13.1 waarom (niet)? ......................................................................................................................................... 36
13.2 Waarover niet? ......................................................................................................................................... 36
13.3 voor wie (niet)? ......................................................................................................................................... 36
13.4 Hoe niet?................................................................................................................................................... 36
Pagina 3 van 36
DATA
Semester 4
Julia Schouten
, INHOUDSOPGAVE
1 Inleiding ...................................................................................................................................................... 4
1.1 Waarom? ...................................................................................................................................................... 4
1.2 Waarover? .................................................................................................................................................... 4
1.3 Voor wie? ...................................................................................................................................................... 4
1.4 Hoe?.............................................................................................................................................................. 4
2 Big data ....................................................................................................................................................... 4
2.1 Inleiding ........................................................................................................................................................ 4
2.2 Geschiedenis ................................................................................................................................................. 5
2.3 beschrijving ................................................................................................................................................... 5
2.4 relaties .......................................................................................................................................................... 6
2.5 toepassingen ................................................................................................................................................ 6
2.5.1 algemeen............................................................................................................................................... 6
2.5.2 Google ................................................................................................................................................... 6
2.5.3 CERN ...................................................................................................................................................... 6
2.5.4 NASA ..................................................................................................................................................... 6
2.5.5 Facebook ............................................................................................................................................... 7
2.5.6 Game-company ..................................................................................................................................... 7
3 Oplsag ......................................................................................................................................................... 7
3.1 Inleiding ........................................................................................................................................................ 7
3.2 Dataontwerp ................................................................................................................................................ 7
3.2.1 voorwaarden ......................................................................................................................................... 7
3.2.2 Beschrijvingen ....................................................................................................................................... 7
3.3 Dataoplsag ................................................................................................................................................... 8
3.3.1 ontstaan ................................................................................................................................................ 8
3.3.2 Database ............................................................................................................................................... 8
3.4 van data naar informatie.............................................................................................................................. 8
3.5 opslag van informatie............................................................................................................................ 9
3.5.1 soorten opslag ....................................................................................................................................... 9
3.5.2 Datawarehouse en datamart .............................................................................................................. 10
3.5.3 Datalake en datareservoir ................................................................................................................... 10
4 proces ....................................................................................................................................................... 10
4.1 inleiding ...................................................................................................................................................... 10
4.2 Data-analyse............................................................................................................................................... 10
4.2.1 Methodiek ........................................................................................................................................... 10
4.2.2 Reporting............................................................................................................................................. 11
4.2.3 Online analytical processing (OLAP) .................................................................................................... 11
4.2.4 zoekopdrachten .................................................................................................................................. 11
4.3 Datamining ................................................................................................................................................. 11
4.4 Big data-proces ........................................................................................................................................... 11
4.4.1 algemeen............................................................................................................................................. 11
4.4.2 Parallel verwerken .............................................................................................................................. 12
4.4.3 Key value ............................................................................................................................................. 12
Pagina 1 van 36
, 4.4.4 Map reduce ......................................................................................................................................... 12
4.4.5 Verwerken Big Data............................................................................................................................. 12
4.5 Predictive analytics ..................................................................................................................................... 13
4.6 CAP-theorema ............................................................................................................................................ 13
5 beslisboom ................................................................................................................................................ 13
5.1 Inleiding ...................................................................................................................................................... 13
5.2 het probleem .............................................................................................................................................. 13
5.3 de oplossing ................................................................................................................................................ 13
5.4 de beperkingen ........................................................................................................................................... 14
6 Neutrale netwerken .................................................................................................................................. 14
6.1 inleiding ...................................................................................................................................................... 14
6.3 de oplossing ................................................................................................................................................ 16
6.4 de beperkingen ........................................................................................................................................... 17
7 clusteren ................................................................................................................................................... 17
7.1 inleiding ...................................................................................................................................................... 17
7.2 het probleem .............................................................................................................................................. 18
7.3 de oplossing ................................................................................................................................................ 18
7.3.1 Algemeen ............................................................................................................................................ 18
7.3.2 Hiërarchisch clusteren......................................................................................................................... 18
7.3.3 puntsgewijs clusteren ......................................................................................................................... 18
7.3.4 dichtheidsgewijs clusteren .................................................................................................................. 19
7.3.5 validatie ............................................................................................................................................... 19
7.4 de beperkingen ........................................................................................................................................... 20
8 lineaire regressie ....................................................................................................................................... 20
8.1 Inleiding ...................................................................................................................................................... 20
8.2 het probleem .............................................................................................................................................. 21
8.3 de oplossing ................................................................................................................................................ 21
8.4 de beperkingen ........................................................................................................................................... 22
9 naaste buur ............................................................................................................................................... 23
9.1 inleiding ...................................................................................................................................................... 23
9.2 het probleem .............................................................................................................................................. 23
9.3 de oplossing ................................................................................................................................................ 24
9.4 de beperkingen ........................................................................................................................................... 24
10 Regels afleiden ........................................................................................................................................ 26
10.1 De inleiding ............................................................................................................................................... 26
10.2 Het probleem ............................................................................................................................................ 26
10.3 de oplossing .............................................................................................................................................. 27
10.3.1 Werken met bedrijfsregels ................................................................................................................ 27
10.3.2 Bestaande business rules .................................................................................................................. 28
Pagina 2 van 36
, 10.3.3 Nieuwe bedrijfsregels ....................................................................................................................... 28
10.4 de beperkingen ......................................................................................................................................... 30
10.4.1 Toepassingen .................................................................................................................................... 30
10.4.2 Beslissingsboom versus afleiden van regels ...................................................................................... 30
11 zin en onzin ............................................................................................................................................. 31
11.1 inleiding .................................................................................................................................................... 31
11.2 Kwaliteit.................................................................................................................................................... 31
11.2.1 algemeen........................................................................................................................................... 31
11.2.2 kwaliteit van data .............................................................................................................................. 32
11.2.3 kwaliteit van informatie .................................................................................................................... 32
11.2.4 kwaliteit van testen ........................................................................................................................... 32
11.2.5 kwaliteit van het ontwerp ................................................................................................................. 33
11.2.6 kwaliteit van bedrijfsplan .................................................................................................................. 33
11.3 presentatie................................................................................................................................................ 33
11.4 overdenkingen .......................................................................................................................................... 34
Ethiek en big data......................................................................................................................................... 34
12.1 inleiding .................................................................................................................................................... 34
12.2 ethiek ........................................................................................................................................................ 34
12.3 het probleem ............................................................................................................................................ 34
12.4 oplossing? ................................................................................................................................................. 35
12.5 reflectie ..................................................................................................................................................... 35
13 uitleiding ................................................................................................................................................. 36
13.1 waarom (niet)? ......................................................................................................................................... 36
13.2 Waarover niet? ......................................................................................................................................... 36
13.3 voor wie (niet)? ......................................................................................................................................... 36
13.4 Hoe niet?................................................................................................................................................... 36
Pagina 3 van 36
