Data Compression With Deep Probabilistic Models
Course by Prof. Robert Bamler at University of
At a Glance
- Mondays 16:15-17:45 and Tuesdays 12:15-13:45 on zoom.
- First lecture: Monday, 19 April; after that, lectures will be on Tuesdays, see
detailed tentative schedule below.
- 6 ECTS with grade based on group project (you may skip the group project if you don't need the
- To encourage interactivity, neither the lectures nor the tutorials will be recorded. However, I'm providing
supplementary video material that repeats important concepts discussed in the lectures in a dedicated YouTube
Tentative Schedule & Course Materials
Details of the following schedule are still subject to change.
Find out what you will learn in this course: from the information theoretical foundations of
concrete practical implementations of deep learning based compressors.
Lossless Compression I: Symbol Codes
How can we compress a sequence of symbols and what do we have to know about the data source?
Theoretical Bounds for Lossless Compression
What is the theoretical bound for lossless compression? We'll encounter a term
Optimality of Huffman Coding
We prove that the Huffman Coding algorithm produces an optimal symbol code.
In tutorial: form a team for your group project & discuss suggested topics.
Random Variables and Autoregressive Models
Our first example of a deep-learning based data compression method after a crash-course on important
concepts from probability theory.
Bits-Back Coding With Latent Variable Models
Learn how to “short sell” bits (no, not on the stock exchange).
Deadline for finalizing team members and topic of your group project.
Stream Codes I: Asymmetric Numeral Systems (ANS)
We've already learned how to create an optimal symbol code, but can we do better than symbol codes? Yes, if
we're willing to think in fractional bits.
Stream Codes II: Arithmetic Coding and Range Coding
A "bonus" video on a famous stream code with queue semantics while the problem set discusses the last
missing piece of the ANS algorithm.
Deep Latent Variable Models and Variational Autoencoders
Minimizing the bitrate directly leads us to amortized variational expectation maximization. Let's just call
“variational autoencoders”, though.
Channel Coding Theorem and Theory of Lossy Compression
Two final gems of information theory before we venture into more applied issues starting with the next
Deadline for status report on project.
Practical Machine Learning Based Lossy Compression
What are some proven model architectures for ML-based lossy compression?
Instead of tutorial: individual appointments to discuss your group projects.
Recent Advances in
What are the latest trends and open research questions?
Instead of tutorial: continuing with the group appointments.
Two pioneers of neural compression:
Tue, 13 July: Dr. Christopher Schroers from DisneyResearch|Studios in Zurich
Mon, 19 July: Prof. Dr. Stephan Mandt from University of California at Irvine
13 & 19 July
Tips for Giving Presentations and for Scientific Writing
Presenting your group project next week and finalizing the written report should be a joy, not a pain! These
tips might help you.
Group Project Presentations
The stage is yours! Let's celebrate your achievements in the group projects with a round of presentations and
demos, and with plenty of time for your questions and your feedback.