Data Mining
Introduction
Data mining is an integral part of knowledge
discovery in databases (KDD), which is the overall
process of converting raw data into useful
information. The purpose of pre-processing is to
transform the raw input data into an appropriate
format for subsequent analysis. Post-processing
ensures that only the valid and useful results are incorporated into the decision support
system.
Specific challenges that motivated the development of data mining:
- Scalability: if data mining algorithms are to handle massive data sets, then they must
be scalable.
- High dimensionality: data sets with temporal or spatial components also tend to have
high dimensionality. For some data analysis algorithms, the computational complexity
increases rapidly as the dimensionality (number of features) increases.
- Heterogeneous and complex data: the role of data mining has grown, so it has the
need for techniques that can handle heterogeneous attributes. Recent year have also
seen the emergence of more complex data objects.
- Data ownership and distribution: how to reduce the amount of communication
needed to perform the distributed computation? How to effectively consolidate the
data mining results obtained from multiple sources? How to address data security
issues?
- Non-traditional analysis: the data sets analysed in data mining are typically not the
result of a carefully designed experiment and often represent opportunistic samples
of the data, rather than random samples. The data sets also frequently involve non-
traditional types of data and data distributions.
The origins of Data Mining:
Draws ideas from machine learning/AI, pattern recognition, statistics, and database systems.
In particular, data mining draws upon ideas, such as (1) sampling, estimation, and hypothesis
testing from statistics and (2) search algorithms, modelling techniques, and learning theories
from AI, pattern recognition and machine learning. Data mining has also been quick to adopt
ideas from other areas, including optimization, evolutionary computing, information theory,
signal processing, visualization, and information retrieval.
Data Mining Tasks
Predictive tasks: to predict the value of a particular attribute (target or dependent variable)
based on the values of other attributes (explanatory or independent variables).
Introduction
Data mining is an integral part of knowledge
discovery in databases (KDD), which is the overall
process of converting raw data into useful
information. The purpose of pre-processing is to
transform the raw input data into an appropriate
format for subsequent analysis. Post-processing
ensures that only the valid and useful results are incorporated into the decision support
system.
Specific challenges that motivated the development of data mining:
- Scalability: if data mining algorithms are to handle massive data sets, then they must
be scalable.
- High dimensionality: data sets with temporal or spatial components also tend to have
high dimensionality. For some data analysis algorithms, the computational complexity
increases rapidly as the dimensionality (number of features) increases.
- Heterogeneous and complex data: the role of data mining has grown, so it has the
need for techniques that can handle heterogeneous attributes. Recent year have also
seen the emergence of more complex data objects.
- Data ownership and distribution: how to reduce the amount of communication
needed to perform the distributed computation? How to effectively consolidate the
data mining results obtained from multiple sources? How to address data security
issues?
- Non-traditional analysis: the data sets analysed in data mining are typically not the
result of a carefully designed experiment and often represent opportunistic samples
of the data, rather than random samples. The data sets also frequently involve non-
traditional types of data and data distributions.
The origins of Data Mining:
Draws ideas from machine learning/AI, pattern recognition, statistics, and database systems.
In particular, data mining draws upon ideas, such as (1) sampling, estimation, and hypothesis
testing from statistics and (2) search algorithms, modelling techniques, and learning theories
from AI, pattern recognition and machine learning. Data mining has also been quick to adopt
ideas from other areas, including optimization, evolutionary computing, information theory,
signal processing, visualization, and information retrieval.
Data Mining Tasks
Predictive tasks: to predict the value of a particular attribute (target or dependent variable)
based on the values of other attributes (explanatory or independent variables).
