Differences

This shows you the differences between two versions of the page.

--- tanszek:oktatas:techcomm:information [2025/10/06 20:47] – [Example of Entropy calculation] knehez
+++ tanszek:oktatas:techcomm:information [2026/10/06 07:01] (current) – [Example: three coin tosses] knehez
@@ Line 11: / Line 11: @@
 $$ I_E = \log_2 \frac{1}{p_E} = -\log_2( p_E ) [bit] $$
-Shannon used the logarithm to measure information because only the logarithmic function makes the information of independent events additive. If two independent events 𝐴 𝐵 occur, their joint probability is: \( p(A,B) = p(A) \cdot p(B) \).
+Shannon used the logarithm to measure information because only the logarithmic function makes the information of independent events additive. If two independent events //A// //B// occur, their joint probability is: \( p(A,B) = p(A) \cdot p(B) \).
 We expect that the total information should add up:
@@ Line 17: / Line 17: @@
 $$ I(A,B) = I(A) + I(B) $$
-Only the logarithm satisfies this property:
+The logarithm satisfies this property:
 $$ I(p) = -\log p \quad \Rightarrow \quad I(A,B) = -\log(p(A)p(B)) = I(A) + I(B) $$
-If we used \( I(p) = 1/p \), the values would multiply, not add.
+If we used \( I(p) = 1/p \), the values would multiply rather than add.
-The properties of a logarithm function play an important role in modeling the quantitative properties of a given information.
+==== Example: three coin tosses ====
+Consider three independent tosses of a fair coin. The probability of heads on each toss is \(1/2\). The probability of getting heads on all three tosses is:
+$$ p(H,H,H) = \frac{1}{2} \cdot \frac{1}{2} \cdot \frac{1}{2} = \frac{1}{8} $$
+If we used \(I(p) = 1/p\), learning that one toss resulted in heads would give an information value of:
+$$ I(H) = \frac{1}{1/2} = 2 $$
+Adding the information values of the three individual outcomes would give \(2+2+2=6\). However, applying the same formula to the combined outcome gives:
+$$ I(H,H,H) = \frac{1}{1/8} = 8 $$
+Therefore, this formula does not make information additive: \(8 \neq 2+2+2\). Instead, the values multiply: \(8 = 2 \cdot 2 \cdot 2\).
+With the logarithmic formula, learning the result of each coin toss provides **1 bit** of information:
+$$ I(H) = -\log_2(1/2) = 1 \text{ bit} $$
+Learning that all three tosses resulted in heads provides:
+$$ I(H,H,H) = -\log_2(1/8) = 3 \text{ bits} $$
+This is exactly the sum of the information from the three individual outcomes: **1 + 1 + 1 = 3 bits**. We obtain the same total information whether we learn the results one by one or all at once.
+----
+The properties of the logarithm function play an important role in modeling the quantitative properties of information.
 If an event space consist of two equal-probability event \(p(E_1) = p(E_2) = 0.5 \) then,
-$$ I_{E_1} = I_{E_2} = \log_2 \frac{1}{0.5} = - \log_2 0.5 = 1 [bit] $$
+$$ I_{E_1} = I_{E_2} = \log_2 \frac{1}{0.5} = - \log_2 2 = 1 [bit] $$
 So the unit of the information means the news value which is connected to the simple, less likely, same probability choice.
@@ Line 45: / Line 73: @@
 The average information content of the set of messages is called the //entropy// of the message set.
-$$ H_E = \sum_{i=1}^n p_i \cdot I_{E_i} = \sum_{i=1}^n p_i \cdot \log_2 \frac{1}{p_i} = - \sum_{i=1}^n p_i \cdot \log_2 p_i$$
+$$ H_E = \sum_{i=1}^n p_i \cdot I_{E_i} = \sum_{i=1}^n p_i \cdot \log_2 \frac{1}{p_i} = - \sum_{i=1}^n p_i \cdot \log_2 p_i  [bit]$$
 **Example**: Given an event space consisting of two events: \( E = \{E_1, E_2\} \), and further \( p = \{p_1, p_2\} \) with \( p_2 = 1 - p_1 \), then the average information content is:
@@ Line 55: / Line 83: @@
 {{:tanszek:oktatas:techcomm:pasted:20240827-130729.png}}
-We can see that entropy is highest when the two events are equally likely. In general, in this model, entropy is low when our event system includes events with low probabilities.
+For two possible outcomes, entropy is highest when both have probability 0.5. In this case, we are most uncertain about which outcome will occur.
+Entropy decreases as one outcome becomes more likely and the other becomes less likely. If one outcome is certain, entropy is zero: observing the result provides no new information.
-Entropy can also be viewed as a measure of the information "richness" of a message. In communication systems, higher entropy implies a greater potential for the message to carry a variety of content, whereas lower entropy suggests that the message is more predictable or redundant.
+Unlikely outcomes do not necessarily mean entropy is low. For example, if there are 256 equally likely outcomes, each has a probability of only 1/256, but the entropy is 8 bits. What matters is the whole probability distribution, not just the probability of one outcome.
 Example:
@@ Line 64: / Line 94: @@
   * If every letter occurs with equal probability (e.g., random characters), the entropy is maximal → the source is rich in information.
-This concept is crucial in various fields, including //data compression//, //cryptography//, and //machine learning//, where understanding and managing entropy can lead to more efficient algorithms and systems. For example, in data compression, reducing redundancy (and thus reducing entropy) can lead to more compact data representations. Similarly, in cryptography, managing entropy ensures that keys and encrypted messages are less predictable and more secure.
+This concept is crucial in various fields, including //data compression//, //cryptography//, and //machine learning//, where understanding and managing entropy can lead to more efficient algorithms and systems.
 ==== Redundancy ====