• LinkedIn
  • Facebook
  • Twitter

This course is a graduate and senior-undergraduate-level introduction to information theory. Information theory is probably the most elegant mathematical theories, with the most direct and significant engineering impacts to our life in the information age. Starting from the first paper by C. E. Shannon in 1948, information theory has found its applications in many areas, including statistics, computer science, biology, economics, etc. Even though the course will attempt to cover as many aspects of information theory as possible, the focus will be on the direct applications of information theory in digital communications.

Arguably the most important part of learning information theory is to learn a new way of thinking about engineering problems. In this sense, this course is beneficial not only to communication majored students, but also to students in other engineering disciplines.

As an advanced course, our lectures proceed with a combination of intuitive thinking and rigorous mathematical treatments. It is hoped that the methodology taught in this course will be helpful in the research of all attendees.

Prerequisite: As an advanced course, the assumption is that the students have sufficient training in rigorous mathematical reasoning. Basic knowledge of the probability theory, at the level of ENG 200, is an absolute must. You are recommended to review and evaluate yourself on your fluency of probability and linear algebra using the following text book.
D. Bertsekas and J. Tsitsiklis, Introduction to Probability
Lecture Hours: Tuesdays 13:40-16:30, Room 207 and Thursdays 13:40-16:30, Room 207
Office Hours: Tuesdays 10:00-12:00 and Thursdays 10:00-12:00
Elements of Information Theory, Second Edition, Thomas M. Cover & Joy A. Thomas,Wiley 2006.
Reference Texts::
1. Information Theory and Reliable Communication, Robert G. Gallager, Wiley, 1968.
2. Information Theory, Inference and Learning Algorithms, David J.C. McKay, Cambridge University Press, 2003.

Tentative Calendar
Lecture 1. Introduction, Entropy, Mutual Information
Lecture 2. Jensen’s Inequality, Log-Sum Inequality, Data Processing Inequality, Fano’s Inequality
Lecture 3. Markov Chains
Lecture 5. Entropy Rate, Hidden Markov Models
Lecture 6. Asymptotic Equipartition Property
Lecture 7. Data Compression, Kraft Inequality, Optimal Codes
Lecture 8. Huffman Codes, Shannon-Fano-Elias Coding, Arithmetic Coding
Lecture 9. Channel Capacity,Symmetric Channels
Lecture 10. Channel Coding Theorem
Lecture 11. Hamming Codes, Feedback Capacity, Joint Source Channel Coding Theorem
Lecture 12. Differential Entropy
Lecture 13. Gaussian Channel, Band-Limited Channels
Lecture 14. Parallel Gaussian Channels, Channels with Colored Gaussian Noise
Lecture 15. Gaussian Channels with Feedback
Lecture 16. Maximum Entropy Distributions, Spectrum Estimation, Entropy Rates of a Gaussian Process, Burg’s Theorem
Lecture 17. Quantization, Rate Distortion Function
Lecture 18. Rate Distortion Theorem
Lecture 19. Gaussian Multiple User Channels, Multiple Access Channel
Lecture 20. Encoding of Correlated Sources, Slepian-Wolf Encoding and Multiple Access Channels
Lecture 21. Broadcast Channel
Lecture 22. Relay Channel
Lecture 23. Source Coding and Rate Distortion with Side Information
Lecture 24. General Multi-Terminal Networks
Lecture 25. Law of Large Numbers, Universal Source Coding
Lecture 26. Large Deviation Theory, Sanov’s Theorem, Conditional Limit Theorem
Lecture 27. Hypothesis Testing, Stein’s Lemma, Chernoff Bound
Lecture 28. Lempel-Ziv Coding, Cramer-Rao Inequality