1. ITW2003, Paris, France, March 31 – April 4, 2003
Performance estimation for concatenated coding schemes
Simon Huettinger and Johannes Huber
Chair of Information Transmission, University Erlangen-N¨ rnberg, Germany
u
e-mail: huettinger,huber @LNT.de
Abstract — Asymptotical analysis of concatenated codes U [i], which is the a–posteriori probability taking the received
with EXIT charts [tB99] or the AMCA [HH02b] is proven vector Y and code constraints into account, i.e.,
to be a powerful tool for the design of power–efficient com-
def
munication systems. But, usually the result of the asymp- V [i] = Pr U [i] = 0 Y . (1)
totical analysis is a binary decision, whether convergence
of iterative decoding is possible at the chosen signal–to– ˆ
Estimated symbols U [i] can be obtained from the vector of
noise ratio, or not. soft–output values V .
In this paper it is shown how to obtain the Information
Processing Characteristic (IPC) introduced in [HHJF01] II. I NFORMATION P ROCESSING C HARACTERISTICS
for concatenated coding schemes. If asymptotical anal- The Information Processing Characteristic [HHJF01] for
ysis is performed under the assumption of infinite inter- symbol–by–symbol decoding and Interleaving
leaving and infinitely many iterations, this IPC will be a
K
lower bound. Furthermore, it also is possible to estimate def ¯ def 1
the performance of realistic coding schemes by restricting IPCI (C) = I(U ; V ) = I(U [i]; V [i]) (2)
K i=1
the number of iterations.
Finally, the IPC can be used to estimate the resulting characterizes a coding scheme w.r.t. soft–output, i.e. IPCI is
bit error ratio for the concatenated coding scheme. As an the capacity of the memoryless end–to–end channel from U to
upper and a lower bound on the bit error ratio for a given soft–output V .
IPC exist, we are able to lower bound the performance of In [HHFJ02] we proved by information theoretic bounding
any concatenated coding scheme and give an achievability that the IPC of any coding scheme can be upper bounded by:
bound, i.e. it is possible to determine a performance that
can surely be achieved if sufficiently many iterations are IPCI (C) min (C/R, 1) . (3)
performed and a large interleaver is used.
A coding scheme fulfilling (3) with equality is called ideal
I. S YSTEM M ODEL coding scheme.
In the following we analyze the properties of a digital com- The IPCI is important for two reasons. Firstly, the charac-
munications system consisting of a binary Bernoulli source, terization w.r.t. soft–output is very helpful for the analysis and
a channel coder, a channel, a decoder and a sink. Without comparison of coding schemes, which will be used as com-
loss of generality we assume, that the source emits a block ponents of concatenated codes [HHJF01]. Secondly, the IPCI
of K binary information symbols U [i], i ¾ 1, 2, ¡ ¡ ¡ , K . can be a result of a convergence analysis performed with EXIT
The encoder maps the information vector U to a codeword charts [tB99] or the AMCA [HH02b].
X which consists of N symbols X[n], n ¾ 1, 2, ¡ ¡ ¡ , N . In the following we will show how the IPCI can be obtained
The rate of the code, which is supposed to be time–invariant, for concatenated coding schemes and a relationship between
is R = K/N measured in bit per channel symbol. The code- ˆ
the IPCI and the bit error ratio of the hard–output U [i] will be
word X is transmitted over a memoryless channel that corrupts derived.
the message by substitution errors, e.g., the binary symmetric
channel (BSC) or the additive white Gaussian noise channel III. I NFORMATION P ROCESSING C HARACTERISTIC AS
(AWGN Channel). Modulator and demodulator are consid- R ESULT OF A SYMPTOTICAL A NALYSIS
ered as being part of the channel. To obtain the Information Processing Characteristic for
Additionally we introduce an (theoretically infinite) inter- symbol–by–symbol decoding and interleaving IPCI (C),
leaver π½ before encoding that converts the end–to–end chan- firstly we have to determine the mutual information between
nel between U and V to a memoryless channel. the source symbols U and the post-decoding soft–output V of
the decoder using EXIT charts or the AMCA. It is possible
to obtain both, a lower bound achieved by infinite interleav-
ing and infinitely many iterations as well as estimations of the
mutual information after an arbitrary number of iterations. As
long as thereby the number of iterations is restricted such that
Figure 1: System model. the cycles in the graph of the code do not dominate the de-
coding performance the result will be close to bit error per-
The corrupted received sequence Y is processed by the de- formance that can be measured if the whole coding scheme is
coder. The decoder output is the soft–output w.r.t. symbol simulated.

2. Results or intermediate results of EXIT charts and the infinitely many iterations, the decoding process gets stuck ex-
AMCA are the mutual information between the source sym- actly at these points. Hence, from the abscissa and ordinate
bols U in parallel concatenation or the encoded symbols of values of these points a lower bound on the IPCI (C) can be
the outer encoder X in serial concatenation and the respective obtained. This asymptotical IPCI (C) is shown in Fig. 4.
extrinsic soft–output at the decoder side Z resp. Q. The post–
decoding information V , which is the final result at the output 1
−1.5 dB
of an iterative decoder, is created by maximum ratio combin-
0.9
ing [Bre59] of the extrinsic informations of all consituent de-
coders on a symbol basis. This can be modelled statistically 0.8
by information combining [HH02c].
For serial concatenation we also have to assume systematic 0.7
encoding of the outer code, to ensure that the post–decoding
IPCI (C) [bit per source symbol]
mutual information w.r.t. info bits U is the same as the post– 0.6
me
decoding mutual information w.r.t. code bits X of the outer
che
0.5
gs
encoder.
din
l co
Exemplary, IPCI (C) for the serial concatenation of Fig. 2 0.4
a
ide
will be determined in the following.
0.3
−2.5 dB
0.2
10log (E /N )=−3 dB
10 s 0
0.1
Ser. Concat.
ν=8 CC
Figure 2: Encoder for serial concatenation of a rate–1/2 MFD 0
convolutional code of memory ν = 1 with a ν = 2 scrambler 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
(Gr = 07, G = 01). C [bit per channel symbol]
As the concatenation is of extremely low complexity, it can Figure 4: IPCI (C) for the concatenation of Fig. 2, obtained
be assumed that even in practical implementations the limit by EXIT charts. For comparison also the IPCI (C) of a ν = 8
of infinite number of iterations will be closely approximated. convolutional code is given.
Hence, we firstly determine the intersection point of the trans- For a realistic coding scheme 25 iterations are sufficient to
fer characteristics within EXIT charts for the range of signal– closely approximate this behavior. In every iteration the outer
to–noise ratios. Then the post–decoding mutual information is decoder decodes a 2–state trellis of length K, and the inner
calculated using information combining. one visits 4 states in every of the 2K trellis segments. Hence,
the decoding complexity then is 25 ¡ (2 + 2 ¡ 4) = 250 visited
1
states per decoded bit, which is approximately the complexity
of decoding a memory ν = 8 convolutional code. For compar-
0.9
ison the IPCI (C) of a ν = 8 convolutional code also is plotted
0.8 into Fig. 4 This IPCI (C) is directly obtained via Monte Carlo
simulation.
0.7 Obviously, there is a substantial difference in the behav-
I(U ; E) = I(X; Y ) [bit per symbol]
5
dB ior of the two coding schemes. The concatenation shows the
0.6
−1.
)= turbo–cliff, which is typical for iteratively decoded concatena-
/N 0
0.5 g
(E
10
s
tion, whereas the convolutional code shows a more constant
0lo
1 improvement of output if channel capacity is raised. Hence, if
0.4 high mutual information (e.g., I(U ; V ) > 0.999) is required,
which corresponds to a low bit error ratio, the concatenation
0.3
outperforms the convolutional code with equal complexity.
B
5d
0.2 −2.
IV. B OUNDING B IT E RROR P ROBABILITY BY C APACITY
−3 dB Although there is no direct relationship between probabil-
0.1
ity of bit error and the capacity of a channel, an upper and
0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
a lower bound can be given. Furthermore, as channels are
I(U ; Z) = I(X; Q) [bit per symbol]
known, which satisfy these bound with equality, they are tight.
We consider memoryless symmetric channels with binary
input U ¾ 0; 1 and discrete or continuous output alphabet
Figure 3: EXIT charts for the concatenation of Fig. 2.
Î . The capacity
Fig. 3 shows some EXIT charts used to determine the I(U ; V ) = H(U ) H(U V ) (4)
IPCI (C) for the concatenation of Fig. 2. Circles mark the
intersection points of the transfer characteristics. Assuming is achieved by equiprobable signaling, i.e. H(U ) = 1.

3. 0.5
With Fano’s inequality [Fan61] which reads 1/2 H(p)
min[p,1−p]
0.45
e2 (BER) H(U V ) = 1 I(U ; V ) (5) 0.4
we have a lower bound on the probability of error. Here, e2 (¡)
0.35
0.3
denotes the binary entropy function 0.25
e2 (x) := x log2 (x) (1 x) log2 (1 x) , (6) 0.2
¾ (0, 1), e2 1 is its inverse for x ¾ (0, 1/2).
0.15
x 0.1
This minimum bit error ratio is achieved by a Binary Sym- 0.05
metric Channel (BSC). Hence, all channels with a larger out- 0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
put alphabet have no lower probability of error at the same
capacity.
p
To derive a lower bound on the capacity [HR70] we need Figure 5: Graphs of min[p, 1 p] and 1 e2 (p).
the a–posteriori probability of U = 0 having received V = v: 2
p = Pr(U = 0 V = v) (7) For any given channel output V = v the entropy of the
binary variable U is given by
and the hard–decision
e2 (p) = H(U V = v), (12)
1 for p 0.5
ˆ
U= . (8)
0 for p > 0.5 as U = 0 occurs with probability p and U = 1 occurs with
There is an equivalent binary symmetric channel from U to probability 1 p. Hence, the average entropy of U given V
ˆ
U . The crossover probability of this channel is equal to the can be expresses as the expectation over the binary entropy
a–posteriori probability p, as uniform signaling is assumed. function of the a–posteriori probability p:
Depending on the actual received V = v and the actual
H(U V ) = E H(U V = v) . (13)
hard decision, which deterministically depends on v, a bit error
occurs with probability Inserting into (4) yields
Pr(U = 0 V = v) for U = 1 = 1 H(U V )
ˆ
Pb = I(U ; V )
ˆ
Pr(U = 1 V = v) for U = 0
= 1 E H(U V = v)
p ˆ
for U = 1
= 1 E e2 (p)
1 p for U = 0
= ˆ
1 2E min[p, 1 p]
p for p 0.5
=
1 p for p > 0.5 = 1 2 ¡ BER. (14)
= min[p, 1 p]. (9) Hence, the bit error ratio can be upper bounded by
The channels bit error ratio BER is given by the expecta-
(1 I(U ; V ))
1
tion over the bit error probability Pb of the actual channels: BER (15)
2
BER = E Pb = E min[p, 1 p] . (10) or equivalently
(10) is a fundamental result for simulations. If, instead of I(U ; V ) 1 2 ¡ BER (16)
counting the really occurred error events during a simulation
of transmission, which is the classical method to determine the (16) is satisfied with equality for a Binary Erasure Chan-
bit error probability of a coding scheme or a channel, (10) is nel (BEC). On average an erasure results in half a bit error
evaluated for every transmitted symbol the variance of the esti- ER/2 = BER. Hence, the BEC is the worst case channel, as
mated bit error ratio is significantly smaller [HLS00]. Further- it maximizes the probability of error for a given capacity.
more, (10) can be used as a test to determine, whether the re- Both bounds are shown in Fig. 6. As they directly corre-
liability estimation of an algorithm yields the true a–posteriori spond to a BSC resp. a BEC they are tight in the whole range
probability [HR90]. If the BER determined by the two differ- of I(U ; V ) ¾ [0; 1].
ent methods does not coincide, the soft–output of the investi-
gated algorithm is a suboptimum reliability measure. V. E STIMATION OF B IT E RROR P ROBABILITY OF
(10) describes two line segments. For any p ¾ [0; 1] the C ONCATENATED C ODING S CHEMES
expression min[p, 1 p] can be upper bounded by 1 e2 (p), 2 After having determined the IPCI (C) for the concatena-
as e2 (p) is a strictly convex function and the line segments tions, we are able to lower bound the achievable bit error ratio
touch 1 e2 (p) at p = 0, 1 , 1 and 1 e2 (p) = 0, 1 , 0, see Fig. 5.
2 2 2 2 using (5). Furthermore, it is possible to give an achievability
bound by (15). If sufficiently large interleavers are used and
min[p, 1 p]
1
e2 (p) (11) sufficiently many iterations are performed, it can be expected,
2

4. interleaving. A block length of K = 100000 is not sufficient
0
10
to be below the acievability bound for all signal–to–noise ra-
tios.
−1
10
VI. C ONCLUSIONS
The information processing characteristic IPCI (C) of con-
catenated coding schemes can be directly obtained from
−2
10
asymptotical analysis. Without simulation of the iterative de-
BER
coding process, which is quite complex, it is possible to en-
−3
tirely characterize the behavior of a coding scheme.
10
As also the bit error ratio, which is the important perfor-
mance measure for applications, can approximately be deter-
Lower Bound mined from the IPCI (C), this analysis is sufficient to decide,
Upper Bound
10
−4
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
whether a coding scheme is appropriate for the intended appli-
I(U ; V ) [bit per symbol] cation.
For theoretical considerations the IPCI (C) has further ad-
Figure 6: Upper and lower bound on the bit error probability vantages. As it characterizes a coding scheme w.r.t. soft–
of a binary input channel. output and has a scaling that magnifies differences between
coding schemes operated below capacity, resulting in bit error
that the simulation results for the concatenation is between the ratios close to 50%, it gives much more insight than a bit error
bounds. ratio curve.
Fig. 7 shows a comparison of bit error performance simula-
R EFERENCES
tion results with the bounds for the investigated concatenation
of Fig. 2. A block length of K = 100000 resp. N = 200000 [Bre59] D. G. Brennan. Linear diversity combining techniques. In Proceed-
ings of the IRE, vol. 47, pp. 1075–1102, Jun. 1959.
has been chosen. 25 iterations are sufficient. More iterations
would not significantly improve the bit error performance. [tB99] S. ten Brink. Convergence of iterative decoding. IEE Electronics
Letters, Vol.35, No.10:pp. 806–808, May 1999.
0
10 [tB00b] S. ten Brink. Iterative Decoding Trajectories of Parallel Concate-
nated Codes. Proc. of 3rd ITG Conference on Source and Channel Cod-
ing, pp. 75–80, Munich, Germany, Jan. 2000.
−1
[Fan61] R. M. Fano. Transmission of Information: A Statistical Theory of
10
Communication. John Wiley & Sons, Inc., New York, 1961.
[HR70] M. E. Hellman and J. Raviv. Probability of Error, Equivocation, and
the Chernoff Bound. IEEE Transactions on Information Theory, vol.16,
no.4:pp.368–372, Jul. 1970.
−2
10
[HLS00] P. Hoeher, I. Land and U. Sorger. Log–Likelihood Values ans
Monte Carlo Simulation – Some Fundamental Results. In Proceedings of
BER
the International Symposium on Turbo Codes, pp. 43–46, Brest, France,
−3
10
Sept. 2000.
[HR90] J. B. Huber and A. Rueppel. Zuverl¨ ssigkeitssch¨ tzung f¨ r die Aus-
a a u
¨
gangssymbole von Trellis–Decodern [in German]. AEU Int. J. Electron.
upper bound
simulation Commun., No.1, pp. 8–21, Jan. 1990.
lower bound
−4
10
0 0.5 1 1.5 [HHJF01] S. Huettinger, J. B. Huber, R. Johannesson and R. Fischer. In-
10 log10 (Eb /Æ0 ) [dB] formation Processing in Soft–Output Decoding. In Proceedings of
39rd Allerton Conference on Communications, Control and Computing,
Oct. 2001.
Figure 7: Comparison of BER obtained via simulation and
[HHFJ02] S. Huettinger, J. Huber, R. Fischer and R. Johannesson. Soft-
estimation from IPC for the concatenation of Fig. 2. Output-Decoding: Some Aspects From Information Theory. In Proceed-
ings of 4. ITG Conference Source and Channel Coding, pp. 81–89, Berlin,
There are two main observations. Fistly, the prediction Jan. 2002.
of the bit error ratio via asymptotical analysis, IPCI (C) and
[HH02b] S. Huettinger and J. Huber Design of “Multiple–Turbo–Codes”
bounds on the bit error probability of memoryless channels with Transfer Characteristics of Component Codes In Proceedings of
are very close to the bit error ratios observed in simulations, Conference on Information Sciences and Systems (CISS ’2002), Prince-
which are much more complex to perform. Secondly, the more ton, Mar. 2002.
the turbo–cliff is pronounced by the concatenation, the closer [HH02c] S. Huettinger and J. Huber. Extrinsic and Intrinsic Information in
the bounds become, and hence the technique becomes more Systematic Coding. In Proceedings of International Symposium on In-
formation Theory 2002, Lausanne, Jul. 2002.
valueable for concatenations that are difficult to simulate.
But, Fig. 7 also shows, that even for constituent codes of [WFH99] U. Wachsmann, R. Fischer and J. B. Huber. Multilevel codes:
Theoretical concepts and practical design rules. IEEE Transactions on
small memory, which have a relatively small decoding hori- Information Theory, IT-45: pp. 1361–1391, Jul. 1999.
zon, quite large interleavers are needed to approximate infinite