H.761 Support of a New <concept> Element and a New
"recognition" Node-Event to Enable Deep Learning-based
Analyses for Media-Nodes
Antonio Busson Alan L. V. Guedes Sergio Colcher
TeleMídia - PUC-Rio TeleMídia - PUC-Rio Informatics Departament - PUC-Rio
ABSTRACT that are applied to input data to classify/recognize the media con-
Machine Learning field, methods based on Deep Learning (e.g. CNN, tent. In computer vision scenario, for example, given image pixels,
RNN) becomes the state-of-the-art in several problems of the mul- the series of layers of the network can learn to extract visual fea-
timedia domain, especially in audio-visual tasks. Typically, the tures from it, the shallow layers can extract lower-level features
training of Deep Learning Methods is done in a supervised manner, (e.g. edges, corner, contours), while the deeper combine these fea-
and it is trained on datasets containing thousands/millions of media tures to produce higher-level features (e.g. textures, part of objects).
examples and several related concepts/classes. During training, the These representative features can be clustered into groups, each
Deep Learning Methods learn a hierarchy of filters that are applied one representing a specific concept.
to input data to classify/recognize the media content. In computer H.761 NCL [2] currently lacks support for Deep Learning Meth-
vision scenario, for example, given image pixels, the series of lay- ods inside their application specification. Because those languages
ers of the network can learn to extract visual features from it, the still focus on presentations tasks such as capture, streaming, and
shallow layers can extract lower-level features (e.g. edges, corner, presentation. They do not consider programmers to describe the
contours), while the deeper combine these features to produce semantic understanding of the used media and handle recognition
higher-level features (e.g. textures, part of objects). These represen- of such under-standing.
tative features can be clustered into groups, each one representing
a specific concept. H.761 NCL currently lacks support for Deep 2 PROPOSAL
Learning Methods inside their application specification. Because
ś (1) describe learning-based concepts on structured multime-
those languages still focus on presentations tasks such as capture,
dia datasets using the new <concept> element;
streaming, and presentation. They do not consider programmers to
ś (2) recognize content semantics of the media elements in pre-
describe the semantic understanding of the used media and handle
sentation time using a virtual anchor, called SemanticAnchor,
recognition of such under-standing. In this proposal, we aim at
with an associated new "recognition" event.
extending NCL to provide such support. More precisely, our pro-
posal able NCL application support: (1) describe learning-based on The overview of the proposal is illustrated in the next Figure
structured multimedia datasets; (2) recognize content semantics of The <concept> element aiming at group and associate media
the media elements in presentation time. To achieve such goals, datasets. It consists of an NCM Composite Node. The Composite
we propose, an extension that includes: (a) the new <knowledge> Node elements (e.g. <body> and <context>) are useful to define
element describe concepts based on multimedia datasets; (b) <area> compositions of multimedia data. That way, we <concept> to rep-
anchor with an associated "recognition" event that describes when resent whole media datasets for specific concepts, as well to specify
a concept occurrences in multimedia content. the associations between them. All <concept>s elements in a doc-
ument are grouped in the <knowledged> at <head>. We define two
KEYWORDS types of associations among <concept>s:
NCL, Ginga
ś The hierarchy association defines a parenthood relation
where is applied the rule: (c 1 ⇒ c 2 ) , the concept node c2
1 BACKGROUND has media features from concept node c1, where c1 and c2
Machine Learning field, methods based on Deep Learning (e.g. CNN, are called parent and child concepts, respectively.
RNN) becomes the state-of-the-art in several problems of the mul- ś The mereology association defines a parthood relation,
timedia domain, especially in audio-visual tasks. Typically, the where is applied the rule: (c 1 ⊢ c 2 ), the concept node c 2 is
training of Deep Learning Methods is done in a supervised man- part of concept node c 1 , this association indicates that media
ner, and it is trained on datasets containing thousands/millions from c 2 are parts of media of c 1 .
of media examples and several related concepts/classes. During
training, the Deep Learning Methods learn a hierarchy of filters The following <knowledge> code illustrate the definition of
the <concept>s elements and their media sets. In particular, the
In: Future of Interactive Television Workshop (V WTVDI), Rio de Janeiro, Brasil. Anais "tony_face" and "jony_face" <concept>s have hierarchy association
Estendidos do Simpósio Brasileiro de Sistemas Multimídia e Web (WebMedia). Porto "face" <concept>, consequently, "person" <concept>. Moreover,
Alegre: Sociedade Brasileira de Computação, 2019.
ISSN 2596-1683 "face_p" <port>" define a mereology. In other words, "face" is a part
of "person".
211
"recognition" Node-Event to Enable Deep Learning-based
Analyses for Media-Nodes
Antonio Busson Alan L. V. Guedes Sergio Colcher
TeleMídia - PUC-Rio TeleMídia - PUC-Rio Informatics Departament - PUC-Rio
ABSTRACT that are applied to input data to classify/recognize the media con-
Machine Learning field, methods based on Deep Learning (e.g. CNN, tent. In computer vision scenario, for example, given image pixels,
RNN) becomes the state-of-the-art in several problems of the mul- the series of layers of the network can learn to extract visual fea-
timedia domain, especially in audio-visual tasks. Typically, the tures from it, the shallow layers can extract lower-level features
training of Deep Learning Methods is done in a supervised manner, (e.g. edges, corner, contours), while the deeper combine these fea-
and it is trained on datasets containing thousands/millions of media tures to produce higher-level features (e.g. textures, part of objects).
examples and several related concepts/classes. During training, the These representative features can be clustered into groups, each
Deep Learning Methods learn a hierarchy of filters that are applied one representing a specific concept.
to input data to classify/recognize the media content. In computer H.761 NCL [2] currently lacks support for Deep Learning Meth-
vision scenario, for example, given image pixels, the series of lay- ods inside their application specification. Because those languages
ers of the network can learn to extract visual features from it, the still focus on presentations tasks such as capture, streaming, and
shallow layers can extract lower-level features (e.g. edges, corner, presentation. They do not consider programmers to describe the
contours), while the deeper combine these features to produce semantic understanding of the used media and handle recognition
higher-level features (e.g. textures, part of objects). These represen- of such under-standing.
tative features can be clustered into groups, each one representing
a specific concept. H.761 NCL currently lacks support for Deep 2 PROPOSAL
Learning Methods inside their application specification. Because
ś (1) describe learning-based concepts on structured multime-
those languages still focus on presentations tasks such as capture,
dia datasets using the new <concept> element;
streaming, and presentation. They do not consider programmers to
ś (2) recognize content semantics of the media elements in pre-
describe the semantic understanding of the used media and handle
sentation time using a virtual anchor, called SemanticAnchor,
recognition of such under-standing. In this proposal, we aim at
with an associated new "recognition" event.
extending NCL to provide such support. More precisely, our pro-
posal able NCL application support: (1) describe learning-based on The overview of the proposal is illustrated in the next Figure
structured multimedia datasets; (2) recognize content semantics of The <concept> element aiming at group and associate media
the media elements in presentation time. To achieve such goals, datasets. It consists of an NCM Composite Node. The Composite
we propose, an extension that includes: (a) the new <knowledge> Node elements (e.g. <body> and <context>) are useful to define
element describe concepts based on multimedia datasets; (b) <area> compositions of multimedia data. That way, we <concept> to rep-
anchor with an associated "recognition" event that describes when resent whole media datasets for specific concepts, as well to specify
a concept occurrences in multimedia content. the associations between them. All <concept>s elements in a doc-
ument are grouped in the <knowledged> at <head>. We define two
KEYWORDS types of associations among <concept>s:
NCL, Ginga
ś The hierarchy association defines a parenthood relation
where is applied the rule: (c 1 ⇒ c 2 ) , the concept node c2
1 BACKGROUND has media features from concept node c1, where c1 and c2
Machine Learning field, methods based on Deep Learning (e.g. CNN, are called parent and child concepts, respectively.
RNN) becomes the state-of-the-art in several problems of the mul- ś The mereology association defines a parthood relation,
timedia domain, especially in audio-visual tasks. Typically, the where is applied the rule: (c 1 ⊢ c 2 ), the concept node c 2 is
training of Deep Learning Methods is done in a supervised man- part of concept node c 1 , this association indicates that media
ner, and it is trained on datasets containing thousands/millions from c 2 are parts of media of c 1 .
of media examples and several related concepts/classes. During
training, the Deep Learning Methods learn a hierarchy of filters The following <knowledge> code illustrate the definition of
the <concept>s elements and their media sets. In particular, the
In: Future of Interactive Television Workshop (V WTVDI), Rio de Janeiro, Brasil. Anais "tony_face" and "jony_face" <concept>s have hierarchy association
Estendidos do Simpósio Brasileiro de Sistemas Multimídia e Web (WebMedia). Porto "face" <concept>, consequently, "person" <concept>. Moreover,
Alegre: Sociedade Brasileira de Computação, 2019.
ISSN 2596-1683 "face_p" <port>" define a mereology. In other words, "face" is a part
of "person".
211
