File(s) not publicly available

Semantic Bottlenecks: Quantifying and Improving Inspectability of Deep Representations

conference contribution
posted on 2023-11-29, 18:15 authored by Max Losch, Mario FritzMario Fritz, Bernt Schiele
Today’s deep learning systems deliver high performance based on end-to-end training but are notoriously hard to inspect. We argue that there are at least two reasons making inspectability challenging: (i) representations are distributed across hundreds of channels and (ii) a unifying metric quantifying inspectability is lacking. In this paper, we address both issues by proposing Semantic Bottlenecks (SB), integrated into pretrained networks, to align channel outputs with individual visual concepts and introduce the model agnostic AUiC metric to measure the alignment. We present a case study on semantic segmentation to demonstrate that SBs improve the AUiC up to four-fold over regular network outputs. We explore two types of SB-layers in this work: while concept-supervised SB-layers (SSB) offer the greatest inspectability, we show that the second type, unsupervised SBs (USB), can match the SSBs by producing one-hot encodings. Importantly, for both SB types, we can recover state of the art segmentation performance despite a drastic dimensionality reduction from 1000s of non aligned channels to 10s of semantics-aligned channels that all downstream results are based on.


Preferred Citation

Max Losch, Mario Fritz and Bernt Schiele. Semantic Bottlenecks: Quantifying and Improving Inspectability of Deep Representations. In: German Conference on Pattern Recognition (GCPR). 2020.

Primary Research Area

  • Trustworthy Information Processing

Name of Conference

German Conference on Pattern Recognition (GCPR)

Legacy Posted Date


Open Access Type

  • Green


@inproceedings{cispa_all_3367, title = "Semantic Bottlenecks: Quantifying and Improving Inspectability of Deep Representations", author = "Losch, Max and Fritz, Mario and Schiele, Bernt", booktitle="{German Conference on Pattern Recognition (GCPR)}", year="2020", }

Usage metrics


    No categories selected


    Ref. manager