Bridging the Gap Between Knowledge
and Data Driven Techniques for Time
Series Forecasting .
TU Kaiserslautern, Kaiserslautern, Germany
1
2 German Research Center for Artificial Intelligence (DFKI) GmbH, Kaiserslautern
muhammad ,
3 School of Electrical Engineering and Computer Science (SEECS), National University of
Sciences and Technology (NUST), Islamabad, Pakistan
4 Deep Learning Laboratory, National Center of Artificial Intelligence, Islamabad, Pakistan
Abstract. Artificial Intelligence (AI) can roughly be categorized into two
streams, knowledge driven and data driven both of which have their own
advantages. Incorporating knowledge into Deep Neural Networks (DNN),
that are purely data driven, can potentially improve the overall performance
of the system. This paper presents such a fusion scheme, DeepEX, that
combines these seemingly parallel streams of AI, for multistep time-series
forecasting problems. DeepEX achieves this in a way that merges best of
both worlds along with a reduction in the amount of data required to train
these modelsnow. This direction has been explored in the past for single step
forecasting by opting for a residual learning scheme. We analyze the
shortcomings of this simple residual learning scheme and enable DeepEX to
not only avoid these shortcomings but also scale to multi-step prediction
problems. DeepEX is tested on two commonly used time series forecasting
datasets, CIF2016 and NN5, where it achieves competitive results even
when trained on a reduced set of training examples. Incorporating external
knowledge to reduce network’s reliance on large amount of accurately
labeled data will prove to be extremely effective in training of neural
networks for real-world applications where the dataset sizes are small and
labeling is expensive.
Keywords: Deep Neural Networks · Knowledge Incorporation · TimeSeries ·
Residual Learning.
1 Introduction
Recent advances in computational hardware have made it possible to achieve state-of-
the-art performance in various domains, by utilizing DNNs, ranging from image
classification [21], playing board games [15], natural language processing [6] to speech
recognition [8]. As a result, there is heightened interest, both academically and
industrially, in DNNs with deep learning being listed at the top of Gartner hype cycle
for emerging technologies [5]. This increased interest coupled with advances in
, 2 Chattha et al.
hardware has paved the way for the development of more sophisticated DNN
algorithms, which may contain millions of parameters to train and optimize. Version of
NASNet-A [21] model, for example, with highest accuracy on ImageNet dataset contains
around 88.9M parameters. Optimizing such a huge number of parameters is a challenge
itself and requires equivalently bigger training dataset that allows the model to extract
enough features to train its parameters. As a result, these models perform exceptionally
well in domains where ample data is available but in data scarce domains, these model
suffer as they can easily overfit. This is even more so true for time-series domain, where
scantiness of data is further compounded by the fact that time-series often do not have
enough features for deep networks to work with. Leveraging information present in
the form of knowledge can be particularly useful here. These techniques, especially the
statistical ones, have shown considerable success in time-series domain which is
evident from results of forecasting competitions, like M3 [11], M4 [12] and NN5 [16],
which were dominated by statistical based techniques.
In contrast to DNNs, humans tend to rely on their knowledge while solving
problems. This knowledge is acquired not only from problem specific examples
but also from other sources, like education and experiences [10]. However, the
very notion of ”knowledge” is tricky to explain and equivalently difficult to collect
and store in a form that is understandable or transferable to a computing program.
Knowledge-Based Systems (KBS) aims to store such knowledge expressed in the
form of logic rules or some other declarative language which can then be used to
find solution to complex problems [19]. Similarly, there are statistical methods
that are based on strong logical reasoning, like Auto-Regressive Integrated Moving
Average (ARIMA), that do perform exceptionally well in their respective domains
and are used by many experts to aid them in decision-making process. . .
In fact, complementing DNNs with expert knowledge or some form of extra
knowledge has been actively researched upon [20,3,9]. Most of the work in the
literature, although improves performance of the DNNs but adds extra
dependency on the network on quality of expert information used [17,9]. The focus
of this work is to combine knowledge driven and data driven streams in a way that
retains advantages of both while suppressing their individual disadvantages.
Specifically, we aim to reduce the dependency of DNNs on the data by leveraging
information contained in the knowledge stream. Finding state-of-theart
knowledge-based system or DNN model is not the focus here, but instead, the goal
is to devise a knowledge incorporation scheme that bridges the gap between data
and knowledge driven approaches and combines their strengths. Chattha et al.
(2019) [4] recently introduced Knowledge Integrated Neural Network (KINN), a
residual framework that combined information contained in the knowledge
stream with the data stream in order to reduce the dependence of DNN on large
amount of accurately labeled data. However, KINN [4] failed to produce acceptable
results on benchmark time-series datasets. KINN particularly suffered when
dealing with time-series that encapsulated significant trend variation. This
resulted in poor performance on more sophisticated time-series datasets. In this
paper, we present DeepEX, that not only addresses the shortcomings of KINN [4],
but also strengthens the network allowing information in the two streams to
and Data Driven Techniques for Time
Series Forecasting .
TU Kaiserslautern, Kaiserslautern, Germany
1
2 German Research Center for Artificial Intelligence (DFKI) GmbH, Kaiserslautern
muhammad ,
3 School of Electrical Engineering and Computer Science (SEECS), National University of
Sciences and Technology (NUST), Islamabad, Pakistan
4 Deep Learning Laboratory, National Center of Artificial Intelligence, Islamabad, Pakistan
Abstract. Artificial Intelligence (AI) can roughly be categorized into two
streams, knowledge driven and data driven both of which have their own
advantages. Incorporating knowledge into Deep Neural Networks (DNN),
that are purely data driven, can potentially improve the overall performance
of the system. This paper presents such a fusion scheme, DeepEX, that
combines these seemingly parallel streams of AI, for multistep time-series
forecasting problems. DeepEX achieves this in a way that merges best of
both worlds along with a reduction in the amount of data required to train
these modelsnow. This direction has been explored in the past for single step
forecasting by opting for a residual learning scheme. We analyze the
shortcomings of this simple residual learning scheme and enable DeepEX to
not only avoid these shortcomings but also scale to multi-step prediction
problems. DeepEX is tested on two commonly used time series forecasting
datasets, CIF2016 and NN5, where it achieves competitive results even
when trained on a reduced set of training examples. Incorporating external
knowledge to reduce network’s reliance on large amount of accurately
labeled data will prove to be extremely effective in training of neural
networks for real-world applications where the dataset sizes are small and
labeling is expensive.
Keywords: Deep Neural Networks · Knowledge Incorporation · TimeSeries ·
Residual Learning.
1 Introduction
Recent advances in computational hardware have made it possible to achieve state-of-
the-art performance in various domains, by utilizing DNNs, ranging from image
classification [21], playing board games [15], natural language processing [6] to speech
recognition [8]. As a result, there is heightened interest, both academically and
industrially, in DNNs with deep learning being listed at the top of Gartner hype cycle
for emerging technologies [5]. This increased interest coupled with advances in
, 2 Chattha et al.
hardware has paved the way for the development of more sophisticated DNN
algorithms, which may contain millions of parameters to train and optimize. Version of
NASNet-A [21] model, for example, with highest accuracy on ImageNet dataset contains
around 88.9M parameters. Optimizing such a huge number of parameters is a challenge
itself and requires equivalently bigger training dataset that allows the model to extract
enough features to train its parameters. As a result, these models perform exceptionally
well in domains where ample data is available but in data scarce domains, these model
suffer as they can easily overfit. This is even more so true for time-series domain, where
scantiness of data is further compounded by the fact that time-series often do not have
enough features for deep networks to work with. Leveraging information present in
the form of knowledge can be particularly useful here. These techniques, especially the
statistical ones, have shown considerable success in time-series domain which is
evident from results of forecasting competitions, like M3 [11], M4 [12] and NN5 [16],
which were dominated by statistical based techniques.
In contrast to DNNs, humans tend to rely on their knowledge while solving
problems. This knowledge is acquired not only from problem specific examples
but also from other sources, like education and experiences [10]. However, the
very notion of ”knowledge” is tricky to explain and equivalently difficult to collect
and store in a form that is understandable or transferable to a computing program.
Knowledge-Based Systems (KBS) aims to store such knowledge expressed in the
form of logic rules or some other declarative language which can then be used to
find solution to complex problems [19]. Similarly, there are statistical methods
that are based on strong logical reasoning, like Auto-Regressive Integrated Moving
Average (ARIMA), that do perform exceptionally well in their respective domains
and are used by many experts to aid them in decision-making process. . .
In fact, complementing DNNs with expert knowledge or some form of extra
knowledge has been actively researched upon [20,3,9]. Most of the work in the
literature, although improves performance of the DNNs but adds extra
dependency on the network on quality of expert information used [17,9]. The focus
of this work is to combine knowledge driven and data driven streams in a way that
retains advantages of both while suppressing their individual disadvantages.
Specifically, we aim to reduce the dependency of DNNs on the data by leveraging
information contained in the knowledge stream. Finding state-of-theart
knowledge-based system or DNN model is not the focus here, but instead, the goal
is to devise a knowledge incorporation scheme that bridges the gap between data
and knowledge driven approaches and combines their strengths. Chattha et al.
(2019) [4] recently introduced Knowledge Integrated Neural Network (KINN), a
residual framework that combined information contained in the knowledge
stream with the data stream in order to reduce the dependence of DNN on large
amount of accurately labeled data. However, KINN [4] failed to produce acceptable
results on benchmark time-series datasets. KINN particularly suffered when
dealing with time-series that encapsulated significant trend variation. This
resulted in poor performance on more sophisticated time-series datasets. In this
paper, we present DeepEX, that not only addresses the shortcomings of KINN [4],
but also strengthens the network allowing information in the two streams to
