Information Theory: A Tutorial Introduction

Very easy to read introduction to information theory. Plenty of concrete examples make this book rather intuitive. Concepts are introduced are a slower pace than most books on the topic, enabling the reader to comfortably assimilate. No issues with the print quality.

This is my second book by this author. I highly recommend buying this book. Information theory is often something that is not given much detail in machine learning training. They just talk about entropy without developing the subject. Jim gives us a nice storytelling on the discoveries of the brilliant Claude Shannon. I plan to gradually buy the whole collection and hope for more books by the same author in the future.

tudo certo obrigado

Information Theory: A tutorial Introduction is a highly readable first account of Shannon's mathematical theory of communication, now known as information theory. It assumes little prior knowledge and discusses both information with respect to discrete and continuous random variables. It is highly readable and a great gentle introduction to the theory. It is intuitive rather than rigorous and gives very clear examples of how to use information and coding theory to frame certain problems. The book starts out by defining the units of information, bits and then how to quantify information so that it has the properties that we would like it to have namely additive, continuous, maximal with respect to certain probability measures and symmetry. He goes through some of the various examples to give a feel of how information can be considered and how many means of communication include much redundancy. The author first tackles discrete random variables and their entropy and information. Through basic ideas like the less likely a piece of information is the more information it has leads to the construction of a measure of information which fits with Shannon's conditions. From that construction of entropy the author then goes through and explores its repercussions with things like dice examples. The explanations are clear and its hard not to feel like you are really making some progress on what is supposed to be a very difficult subject. The author moves from information to coding theory, which is the practical application of the subject and introduces ideas like channel capacity, how much information can be transmitted in a noiseless channel, conditional expectations and coding schemes that can deliver results arbitrarily close to the channel capacity. The author spends some time discussing redundancy in English and coding schemes for how to create efficient blocks for the language which is really interesting. One learns very quickly through the tutorials that trying to have blocks of code which are equiprobable is the way to get to the best coding efficiency and maximize information transmission. The author then moves on to noisy channels, which is far more practical and further evolves the readers intuition. One learns how to add redundancy to codes to make them resilient to noise contamination. The author discusses continuous random variables and their information. The information of a continuous variable, which is a more difficult subject, can be considered infinite given the events possible are uncountable, but with noise this is avoided and the author explains the ideas well so that the content remains clear. The author discusses mutual information in the continuous case giving the reader a strong understanding of joint and marginal probability distributions. With these tools the author goes back to channel capacity in a noisy channel for continuous variables and re-discusses the ideas from discrete variables in the new light. The author then moves on from communication related information theory to entropy and physics. These chapters were to give a feel of the similarity and topics like thermodynamics and quantum information are lightly touched. Information Theory is a highly readable account of what is usually a very technical subject. The reader will come through it with an intuitive feel for information and how transmission of information across various channels and coding schemes to do this effectively. The only thing is there are no exercises so the confidence will be to some extent somewhat false as reading and implementing are very different things. That being said as an introductory text before tackling more mathematical works I think this is very helpful. I have both this as well as the author's work on Bayes and I prefer the content of this. Definitely recommended.

At last a book which is pragmatic, and properly so. So many other books are written by people whose entire life is about the subject at hand, they obviously find it easy, and run through the basics at 100mph. Here is a book where, clearly the author is still a professional, but understands completely that "baby steps", appealing first to intuition, is the initial way forward. This has the perfectly sloped learning curve for the person who is not a post-grad doctoral candidate in maths or physics (read: I'm a computer scientist and greek letter overload is just not my thing), yet it develops the concepts into rigour as one goes along. Would that all scientific textbooks were written this way.