Differences

This shows you the differences between two versions of the page.

--- tanszek:oktatas:techcomm:information [2026/10/05 16:59] – [Information] knehez
+++ tanszek:oktatas:techcomm:information [2026/10/06 07:01] (current) – [Example: three coin tosses] knehez
@@ Line 55: / Line 55: @@
 If an event space consist of two equal-probability event \(p(E_1) = p(E_2) = 0.5 \) then,
-$$ I_{E_1} = I_{E_2} = \log_2 \frac{1}{0.5} = - \log_2 0.5 = 1 [bit] $$
+$$ I_{E_1} = I_{E_2} = \log_2 \frac{1}{0.5} = - \log_2 2 = 1 [bit] $$
 So the unit of the information means the news value which is connected to the simple, less likely, same probability choice.
@@ Line 83: / Line 83: @@
 {{:tanszek:oktatas:techcomm:pasted:20240827-130729.png}}
-We can see that entropy is highest when the two events are equally likely. In general, in this model, entropy is low when our event system includes events with low probabilities.
+For two possible outcomes, entropy is highest when both have probability 0.5. In this case, we are most uncertain about which outcome will occur.
-Entropy can also be viewed as a measure of the information "richness" of a message. In communication systems, higher entropy implies a greater potential for the message to carry a variety of content, whereas lower entropy suggests that the message is more predictable or redundant.
+Entropy decreases as one outcome becomes more likely and the other becomes less likely. If one outcome is certain, entropy is zero: observing the result provides no new information.
+Unlikely outcomes do not necessarily mean entropy is low. For example, if there are 256 equally likely outcomes, each has a probability of only 1/256, but the entropy is 8 bits. What matters is the whole probability distribution, not just the probability of one outcome.
 Example:
@@ Line 92: / Line 94: @@
   * If every letter occurs with equal probability (e.g., random characters), the entropy is maximal → the source is rich in information.
-This concept is crucial in various fields, including //data compression//, //cryptography//, and //machine learning//, where understanding and managing entropy can lead to more efficient algorithms and systems. For example, in data compression, reducing redundancy (and thus reducing entropy) can lead to more compact data representations. Similarly, in cryptography, managing entropy ensures that keys and encrypted messages are less predictable and more secure.
+This concept is crucial in various fields, including //data compression//, //cryptography//, and //machine learning//, where understanding and managing entropy can lead to more efficient algorithms and systems.
 ==== Redundancy ====