Entropy

  Originally derived as a quantity which permits to express formally the second law of thermodynamics (Clausius); the entropy S (of a closed system) changes by , where is (heat) energy transferred to the system at temperature T; S can only increase with time or stay the same. The second law is characteristic for irreversible processes, which tend to evolve towards equilibrium; as such entropy is also at the centre of debates on causality (which in many ways contradicts time reversibility) and consciousness.

In general terms, entropy is a measure of ``disorder'' and can be seen as depending directly on probability: , where k and k₀ are constants and P is the probability of a state.

Entropy is also a concept used in information theory; if N states are possible, each characterized by a probability p_i, with , then is the entropy, the lowest bound on the number of bits needed to describe all parts of the system; it corresponds to the information content of the system (see [Jain89]). This is used in data compression: entropy encoding makes use of the non-uniform occurrence of bit patterns in some quantized scheme. An efficient entropy encoding technique is Huffman coding.