Differences
This shows you the differences between two versions of the page.
| Both sides previous revisionPrevious revision | |
| tanszek:oktatas:techcomm:formulas_for_mathematical_exercises [2024/09/09 11:16] – [Table] kissa | tanszek:oktatas:techcomm:formulas_for_mathematical_exercises [2024/10/15 18:47] (current) – [Information Theory] kissa |
|---|
| | $$H(X)$$ | Entropy, which measures the average amount of information (or uncertainty) in a random variable X. | $$H(X) = -\sum_{x \in X} P(x) \log_2 P(x) = \sum_{x \in X} P(x) \log_2 \frac{1}{P(x)} \text{ [bits]}$$ | | | $$H(X)$$ | Entropy, which measures the average amount of information (or uncertainty) in a random variable X. | $$H(X) = -\sum_{x \in X} P(x) \log_2 P(x) = \sum_{x \in X} P(x) \log_2 \frac{1}{P(x)} \text{ [bits]}$$ | |
| | $$H_{max}$$ | Maximum possible entropy (when all outcomes are equally likely). | $$H_{\text{max}} = \log_2 |\mathcal{X}|$$ $$|\mathcal{X}| \text{ is the number of possible outcomes in the set } \mathcal{X}$$ | | | $$H_{max}$$ | Maximum possible entropy (when all outcomes are equally likely). | $$H_{\text{max}} = \log_2 |\mathcal{X}|$$ $$|\mathcal{X}| \text{ is the number of possible outcomes in the set } \mathcal{X}$$ | |
| | $$R(X)$$ | Redundancy, which measures the portion of duplicative information within a message. | $$R(X) = 1 - \frac{H(X)}{\log_2 |X|}$$ In terms of maximum entropy: $$R = \frac{H_{\text{max}} - H}{H_{\text{max}}}$$ | | | $$R(X)$$ | Redundancy, which measures the portion of duplicative information within a message. | $$R = \frac{H_{\text{max}} - H}{H_{\text{max}}}$$ | |
| ==== Combinatorics ==== | ==== Combinatorics ==== |
| |
