Explain the relationship of entropy and disorder in the context of Shannon's formula.

In information theory, entropy is a measure of uncertainty in a random variable. In this context, the term refers to Shannon's entropy, which quantifies the expected value of the information contained in a message.

Explanation:

The entropy defined by Shannon, referring to the theory of information, refers to the average amount of information contained in a random (psychological) variable or, in particular, a binary transmission source. The information that provides a certain value, xi, of a discrete random variable X, is defined as:

I (xi) = log2 (1 / p (xi))

whose unit is the bit if the logarithm is used in base 2 (for example, when the natural or natural logarithm is used, we speak of nats).

The entropy or average information of the discrete random variable, X, is determined as the average information of the set of discrete values that can be adopted (also measured in bits):

H (x) = Σip (xi) • log2 (1 / p (xi))

In addition to its definition and study, Shannon demonstrated analytically that entropy is the maximum limit to which a source can be compressed without any loss of information.