VTU CBCS E&C 5th sem Information theory and coding(15EC54) Module -3 notes

1. INFORMATION THEORY AND
CODING
5th SEM E&C
JAYANTHDWIJESH H P M.tech (DECS)
Assistant Professor – Dept of E&C
B.G.S INSTITUTE OF TECHNOLOGY (B.G.S.I.T)
B.G Nagara, Nagamangala Tq, Mandya District- 571448

2. FORMULAS FOR REFERENCE
MODULE –3 ( INFORMATION CHANNELS)
 The entropy of input symbol is given by
H (A) = 𝑷(𝒂𝒊)𝐥𝐨𝐠(
𝟏
𝑷 (𝒂 𝒊)
𝒓
𝒊=𝟏 ) bits/ message-symbol
 The entropy of output symbol is given by
H (B) = 𝑷(𝒃𝒋)𝐥𝐨𝐠(
𝟏
𝑷(𝒃 𝒋)
𝒔
𝒋=𝟏 ) bits/ message-symbol
 The joint entropy is given by
H (A, B) = 𝑷( 𝒂𝒊, 𝒃𝒋)𝐥𝐨𝐠(
𝟏
𝑷( 𝒂 𝒊,𝒃 𝒋)
)𝒔
𝒋=𝟏
𝒓
𝒊=𝟏 bits/ message-symbol
 The equivocation H(A/B) is given by
H (A / B) = 𝑷( 𝒂𝒊, 𝒃𝒋)𝐥𝐨𝐠(
𝟏
𝑷 ( 𝒂 𝒊/𝒃 𝒋)
)𝒔
𝒋=𝟏
𝒓
𝒊=𝟏 bits/ message-symbol
Since P( 𝒂𝒊/𝒃𝒋) =
𝑷( 𝒂 𝒊,𝒃 𝒋)
𝑷 (𝒃 𝒋
)
And another formula
H (A/B) = H (A, B) – H (B) bits/ message-symbol
 The equivocation H(B/A) is given by
H (B / A) = 𝑷( 𝒂𝒊, 𝒃𝒋)𝐥𝐨𝐠(
𝟏
𝑷(𝒃 𝒋/ 𝒂 𝒊)
)𝒔
𝒋=𝟏
𝒓
𝒊=𝟏 bits/ message-symbol
Since P(𝒃𝒋/ 𝒂𝒊) =
𝑷( 𝒂 𝒊,𝒃 𝒋)
𝑷 (𝒂 𝒊)
And another formula
H (B/A) = H (A, B) – H (A) bits/ message-symbol
 The mutual information is given by
I (A, B) = H (A) – H (A/B) bits/ message-symbol
 The channel capacity is given by
C = (log s – h) 𝒓 𝒔 bits/ sec
Since h= 𝐇
𝐁
𝐀
= 𝑷𝒋 𝐥𝐨𝐠(
𝟏
𝑷 𝒋
𝒔
𝒋=𝟏 ) bits/ message-symbol
 The capacity of the channel is given by
C=B 𝒍𝒐𝒈 𝟐 [1 + S/N]
 Source efficiency 𝜼 𝑺 is given by
𝜼 𝑺 =
𝑯(𝑿)
𝑯 𝑿 𝒎𝒂𝒙
x 100%

3.  Source efficiency 𝑹 𝜼 𝑺
is given by
𝑹 𝜼 𝑺
= 𝟏 − 𝜼 𝑺
 Channel efficiency is given by
𝜼 𝒄𝒉 =
𝐈(𝐗,𝐘)
𝐂
x 100%
 Channel redundancy is given by
Channel redundancy = (1- 𝜼 𝒄𝒉) x 100%
 Estimation of channel capacity by MUROGA’S method