End-to-end Object Detection
with Transformers
Abstract
This approach removes the need for many hand-designed components like
non-maximum suppression procedure or anchor generation DETR doesn’t need
that! The main ingredients of the new framework, called DEtection TRansformer
or DETR are a set-based global loss that forces unique predictions via bipartite
matching and a transformer encoder-decoder architecture.
Prior methods: Current object detection pipelines include hand-crafted
components like spatial anchor generation and non-max suppression (NMS).
Each of these components is tuned specifically for a given task. For example,
NMS is threshold-based and requires an IOU (intersection over union) and
confidence threshold tuning to be able to effectively discard the overlapping
bounding boxes.
Introduction
Modern detectors address this set prediction task in an indirect way, by
defining surrogate regression and classification problems on a large set of
proposals, anchors or window centers. Their performances are significantly
influenced by postprocessing steps to collapse near-duplicate predictions.
DETR directly predicts (in parallel) the final set of detections by combining a common CNN
with a transformer architecture. During training, bipartite matching uniquely assigns
predictions with ground truth boxes.
Our DEtection TRansformer predicts all objects at once, and is trained end-to-
end with a set loss function which performs bipartite matching between
predicted and ground truth objects.
End-to-end Object Detection with Transformers 1
, Compared to most previous work on direct set prediction, the main features of
DETR are the conjunction of the bipartite matching loss and transformers with
(non-autoregressive) parallel decoding.
Related Work
Set Prediction
A task where a model predicts multiple elements whose ordering is not relevant
for correctness. (Essentially predicting multiple objects in an image).
The way this is solved now however is by introducing relationship or pre
defined knowledge into the model. For instance, the predicted bounding boxes
should not overlap significantly and should cover all detected objects.
Avoiding Near-Duplicates: In object classification sometimes there are the
same bounding boxes for the same predicition, this is solved by using NMS
however set prediction is set to resolve that.
Transformers and Parallel Decoding
Transformers introduced self-attention layers, which, similarly to Non-Local
Neural Networks, scan through each element of a sequence and update it by
aggregating information from the whole sequence.
Object Detection
Set-based loss: Several object detectors used the bipartite matching loss.
Recurrent detectors: Closest to our approach are end-to-end set predictions
for object detection and instance segmentation. Similarly to us, they use
bipartite-matching losses with encoder-decoder architectures based on CNN
activation to directly produce a set of bounding boxes. These approaches,
however, were only evaluated on small datasets and not against modern
baselines. In particular, they are based on autoregressive models (more
precisely RNNs), so they do not leverage the recent transformers with parallel
decoding.
The DETR model
Object Detection set prediction loss
End-to-end Object Detection with Transformers 2
with Transformers
Abstract
This approach removes the need for many hand-designed components like
non-maximum suppression procedure or anchor generation DETR doesn’t need
that! The main ingredients of the new framework, called DEtection TRansformer
or DETR are a set-based global loss that forces unique predictions via bipartite
matching and a transformer encoder-decoder architecture.
Prior methods: Current object detection pipelines include hand-crafted
components like spatial anchor generation and non-max suppression (NMS).
Each of these components is tuned specifically for a given task. For example,
NMS is threshold-based and requires an IOU (intersection over union) and
confidence threshold tuning to be able to effectively discard the overlapping
bounding boxes.
Introduction
Modern detectors address this set prediction task in an indirect way, by
defining surrogate regression and classification problems on a large set of
proposals, anchors or window centers. Their performances are significantly
influenced by postprocessing steps to collapse near-duplicate predictions.
DETR directly predicts (in parallel) the final set of detections by combining a common CNN
with a transformer architecture. During training, bipartite matching uniquely assigns
predictions with ground truth boxes.
Our DEtection TRansformer predicts all objects at once, and is trained end-to-
end with a set loss function which performs bipartite matching between
predicted and ground truth objects.
End-to-end Object Detection with Transformers 1
, Compared to most previous work on direct set prediction, the main features of
DETR are the conjunction of the bipartite matching loss and transformers with
(non-autoregressive) parallel decoding.
Related Work
Set Prediction
A task where a model predicts multiple elements whose ordering is not relevant
for correctness. (Essentially predicting multiple objects in an image).
The way this is solved now however is by introducing relationship or pre
defined knowledge into the model. For instance, the predicted bounding boxes
should not overlap significantly and should cover all detected objects.
Avoiding Near-Duplicates: In object classification sometimes there are the
same bounding boxes for the same predicition, this is solved by using NMS
however set prediction is set to resolve that.
Transformers and Parallel Decoding
Transformers introduced self-attention layers, which, similarly to Non-Local
Neural Networks, scan through each element of a sequence and update it by
aggregating information from the whole sequence.
Object Detection
Set-based loss: Several object detectors used the bipartite matching loss.
Recurrent detectors: Closest to our approach are end-to-end set predictions
for object detection and instance segmentation. Similarly to us, they use
bipartite-matching losses with encoder-decoder architectures based on CNN
activation to directly produce a set of bounding boxes. These approaches,
however, were only evaluated on small datasets and not against modern
baselines. In particular, they are based on autoregressive models (more
precisely RNNs), so they do not leverage the recent transformers with parallel
decoding.
The DETR model
Object Detection set prediction loss
End-to-end Object Detection with Transformers 2
