Topic: Communication Systems
A binary source in which 0's occurs 3 times as often as 1's. Then its entropy in bits/symbol will be
- A. 0.75 bits/symbol
- B. 0.25 bits/symbol
- C. 0.81 bits/symbol
- D. 0.85 bits/symbol
Answer: C. 0.81 bits/symbol
Explanation: Let $p(1) = x$, then $p(0) = 3x$. Since probabilities sum to 1, $x + 3x = 1 \Rightarrow 4x = 1 \Rightarrow x = 0.25$.
$p(1) = 0.25 = \frac{1}{4}$ and $p(0) = 0.75 = \frac{3}{4}$.
Entropy $H(X) = p(0)\log_{2}\left(\frac{1}{p(0)}\right) + p(1)\log_{2}\left(\frac{1}{p(1)}\right)$
$= \frac{3}{4}\log_{2}\left(\frac{4}{3}\right) + \frac{1}{4}\log_{2}(4) = \frac{3}{4}(2 - 1.585) + \frac{1}{4}(2) = 0.311 + 0.5 = 0.811~\text{bits/symbol}$.