Understanding Entropy

I find the concept of entropy interesting – as I came across it in a new light while studying information theory. In physics entropy is defined as the state of disorder of some thermodynamic system (i.e. a system where heat is being exchanged). So objects will have different entropy, depending upon on their temperature (this is an over simplfication). Usually people describe the Universe having a tendency towards higher entropy – ice melting is an example, since ice is more ordered (on a molecular level), than a liquid is. Nature always seems to resist ordering...Remember: Organization = low entropy, Chaos = high entropy.

We can also apply entropy to information in general. In this case we now define entropy as the predictability of an incoming message – (predictability = randomness, this I’ve already touched on in a previous post)

So in order to clarify this point we can look at the two extremes: Entropy of 0 (no entropy) and Entropy of 1 (highest entropy). All of this stuff was figured out by an interesting man named Claude Shannon back in the 40’s and 50’s.

I’ll use a phone call to demonstrate this concept, since context is very important (I will avoid the discussion of context for simplification):

Entropy of 0: If I were to call you and begin counting from 0 to infinity, you would eventually catch on to this pattern and begin predicting my next word. In fact you could probably just count along with me on the phone, get board and hang up at around 22….or maybe right away.

Entropy of 1: If I were to call you and say “I’m rolling a die and I’m going to tell you the number it lands on each time for fun”. Assuming the die was fair (not weighted), you would have no ability to predict the next word coming out of my mouth. 1,2,3,4,5 and 6 would all be equally likely, you would never reach a point where you would be able to count along with me.

Now that I’ve defined the two bounds, you can begin to understand what lower vs. higher entropy would imply. Why might it be important to understand entropy of information? Think about the worldwide telecomunications network – billions of megabytes of information shooting around the planet every second. This costs money, and this means the next logical step is to think about how to save. So in order to understand how to save (save=compress) information, we need to understand it’s entropy.

That’s why you download files in zip format, watch compressed video on youtube, and never seem to get the same reception on a cell phone as you would over a phone…All that stuff is compressed, we are trying to push it’s entropy towards 1! Because higher entropy means less redundancy = less data required to relay some form of information.