Network Assurance and Testing During the Migration to VoIP
1. Analyze Assure Accelerate
ST-09 Network Assurance
and Testing During the
Migration to VoIP
TMC Developers Conference
San Francisco
Aug 03rd, 2005
Andy Huckridge
Spirent Communications.
Chair, Interop WG, MSF
2. Agenda
• Spirent overview
• Key implementation issues
• What is Triple Play / Converged networks?
• Specifics on testing SIP
• Network Impairments and Parameters that Voice
and Video Affect Quality
• Metrics for Measuring Voice and Video Quality and
Performance
• Good test methodology
Analyze Assure Accelerate
3. Spirent Communications
• Spirent is the test solution leader
– 1,800 employees in 14 countries
– More than 1,500 customers
– Sales and service capabilities in
30 countries
Analyze Assure Accelerate
6. Key Implementation Issues
• Circuit to packet migration
• Scalability and Performance
• Voice quality
• Interoperability and conformance
• Budget pressures
Analyze Assure Accelerate
7. Before you deploy!
• Network Equipment Manufacturers (Chips,
IP-PBX, Gateways, MSs & SSs)
– Characterize your system before trial
– Validate system scalability
– Identify capacity limits
– Measure call performance
• Service Providers
(NSPs, SPs, ITSPs)
– Define criteria for vendor selection
– Identify performance ceilings
– Accurately plan for your capacity needs
– End-to-end service assurance testing
Analyze Assure Accelerate
8. Data Transmission
“Non-Real-Time” Applications
Telenet
Web Name
HTTP Resolving
DNS
Email and
Messaging
POP
Data Base SMPT
MS SQL Exchange
Oracle
File Transfer
FTP Music Downloading
Home control
Data Examples: Internet access, Email, File Transfer, Portals, Database Applications,
Gaming, Government Services, Online Commerce
Analyze Assure Accelerate
9. Voice and Video
“Real-Time Applications”
IP Music/Audio/Radio Gaming
VoD VoIP, IP Telephony, Video Telephony Single / Multiplayer
G.711, G.729, G.728, G.726, G.723
H.261, H.263,
SIP, SIP-T, H323, Skinny, MGCP,
MEGACO/H.248
Real-Time Online
Communications
Instant Messenger
Webex IPTV Services
Multi-Media Netmeeting Broadcast, On-Demand,
RTP SIP Bi-directional / Interactive
H.264,Microsoft AVI, QuickTime (.mov) H.323 MPEG1, MPEG2,
Windows Media (.wmv, .asf), RealMedia (.rm), MPEG4, VC1, H264
Voice Applications: Phone service integrated with video
Video Applications: Broadcast TV, video on demand, distance learning
Analyze Assure Accelerate
10. Converged Triple Play: Data, Voice and Video
With Network Impairments
Data
Video
Dialed
On
Ring Configure
Configure
Notify
Digits#
Hook
Off Hook
Dial
Good
Back
Hello
Bye
Connect Voice Good
On
Off Hook
Connect Ring Conversation
Disconnect
Hello
Bye
Hook
Analyze Assure Accelerate
Impairments
Signaling Path can be heard in the voice conversation
12. Testing SIP Conformance
• ETSI TS 102-027-1 v2.12,Tiphon:
– RFC 3261 user agent, proxy and redirect server
compliance
• Graphical SDL and TTCN tools
– Create, edit, compile and execute simulation scripts
and conformance tests
• Additional SIP messages beyond RFC 3261
– Included in torture tests
• Additional tests as defined by the SIP Forum
Analyze Assure Accelerate
13. Testing with Configurable SIP
Configurable SIP call setup and call teardown
• Configurable call flows and messages
• Incoming message filter
– Adaptive signaling syntax for SIP
– Improves interoperability with new drafts and
non-conformant proprietary implementations
Analyze Assure Accelerate
14. Testing with Configurable SIP
• Configurable messages:
Invite, ACK, bye, register
Responses: 1xx, 2xx, 3xx, 4xx, 5xx, 6xx
• Configurable timers, message intervals
• Enable and disable optional messages:
Re-invite, cancel, options, message, info, notify, subscribe,
unsubscribe, update, refer, Prack
Fix erroneous incoming messages “on the fly” with the
“search and replace” method
Allows interoperability with SIP devices (including drafts,
non conformant, prototype)
Analyze Assure Accelerate
21. Testing SIP Robustness
• Robustness testing
– Passed: does not crash, stable, or acceptable results
– Failed: crashes, unstable, or unacceptable results
• Security testing
– It is crucial to identify SIP security holes
• SIP testing tool
– Tests SIP robustness and security
– Comprehensive negative test suites for SIP
Analyze Assure Accelerate
22. Real Signaling with real RTP
• Capability to do signaling with audio
• Capability to perform real time measurements
• Capability of using signaling without audio
• Problems of not using real signaling
• Problems of not using real RTP streams
• Real time objective metrics
Analyze Assure Accelerate
24. SIP-T Performance Testing Suites
Performance testing
– Validate and stress-test SS7 ISUP and SIP
interworking with optional media, over thousands of
emulated user agents
• SIT-T testing
– Configurable SIP-T calls with intelligent protocols
• QoS and CoS testing
– Optional TOS/Diffserv and VLAN options in SIP-T
media calls, used to measure QoS with PESQ and e-
model
• Feature testing
– Automated and configurable SIP call set-up,
teardown, flows, messages
Analyze Assure Accelerate
25. Network Impairments and Parameters
that Affect Voice and Video Quality
• Network Architecture • Timing Drift
• Types of Access Links • Route Flapping
• QoS controlled Edge Routing • Signaling protocol mismatches
• MTU Size • Network faults
• Packet Loss (Frame Loss) • Link Failures
• Out of order packets • Voice Only Impairments
• One Way Delay (Latency) – Echo
• Variable Delays (Jitter)
– Voice coding algorithms
– A/D and D/A Conversion
• Background Traffic
(Congestion, Bandwidth, – Noise – Circuit and External
Utilization, Network Load,
Load Sharing) • Video Only Impairments
– Video coding algorithms
– Fixed vs Variable Frame Rate
Analyze Assure Accelerate
26. IP Network Architecture
Local Access Local Access
Source A B Destination
Device A
LAN A Core IP Network LAN B Device B
64 kbit/s 64 kbit/s
*128 kbit/s *128 kbit/s
256 kbit/s 256 kbit/s
1000BaseX *384 kbit/s *384 kbit/s 1000BaseX
* 100BaseT Switch 512 kbit/s Route flapping 512 kbit/s * 100BaseT Switch
100BaseT Hub *768 kbit/s One-way delay *768 kbit/s 100BaseT Hub
10BaseT *T1 (1.536 kbit/s) Jitter *T1 (1.536 kbit/s) 10BaseT
* WLAN (~4 Mbit/s) E1 (1.920 kbit/s) Packet loss E1 (1.920 kbit/s) * WLAN (~4 Mbit/s)
---------------------- E3 (34 Mbit/s) E3 (34 Mbit/s) ----------------------
Occupancy level *T3 (44 Mbit/s) *T3 (44 Mbit/s) Occupancy level
Packet loss ADSL (~256 kbit/s) ADSL (~2 Mbit/s) Packet loss
*Cable (~256 kbit/s) *Cable (~3 Mbit/s)
Fiber (1-10 Gbit/s) Fiber (1-10 Gbit/s)
-------------------- --------------------
Occupancy level Occupancy level
QoS edge router QoS edge router
* Case used in impairment tables
Affects
Data, Voice and Video Quality
Analyze Assure Accelerate
27. Network Operating With Constant Delay
Affects
Voice and Video Quality
Analyze Assure Accelerate
28. End to End Delay Sources
Core
Originating Network Terminating
LAN LAN
Originating Edge Edge Terminating
Gateway Router Core Router Gateway
Network
Routers
Fixed Fixed
• Serialization Fixed • Serialization
Fixed WAN WAN
Fixed • Switching Fixed Fixed
• Look ahead
• Switching •Propagation • Switching • Decoding
• Encoding
•Serialization
• Buffer Variable Variable Variable
• VAD
• Voice contention
Variable • Voice contention • De-jitter buffer
• Packetizing • Voice contention
• Data Contention • Data Contention • Packet loss
• Video Contention • Data Contention • Video Contention
Concealment
•Video Contention
• Algorithmic delay
Affects • Serialization delay
Voice and Video Quality
• Propagation delay
• Component delay
Analyze Assure Accelerate
29. Echo Impairment on Converged network
Delay in IP Network
makes Echo sound worse Tail Circuit
TELR
IP
Phone MG
T1
Link 2 Wire
IP Network V
PBX POTS
Phone
ERLE ERL
IP
Phone Echo Canceller in MG
reduces Echo Level
ERLE – Echo Return Loss Enhancement Analog Analog
ERL – Echo Return Loss 4-Wire Link 2-Wire Link
TELR – Talker Echo Loudness Rating
RX RX
T1 Link E&M
Affects TX
TX Hybrid POTS
Voice Quality
Transformer Phone
Impedance
PBX Mismatch
Analyze Assure Accelerate
30. Echo Impairment on Converged network
Electrical Coupling
Affects • Impedance Mismatch (Hybrid)
Voice Quality Acoustical Coupling
• Speakerphone
Converged
Network
Path A to B
Path B to A
Echo Path Side A B (250ms)
Echo Path Side (250ms)
Echo is caused by impedance mismatches in hybrid circuits (2w to
4w) and feedback between the telephone mouth piece and ear piece
Analyze Assure Accelerate
31. Effect of Delay on Voice Quality
Voice Quality
> 25ms Echo Cancellation Required
<150 ms (with echo
cancellation): acceptable
150-400 ms:
acceptable if
delay expected
> 400 ms
unacceptable for
most applications
Analyze Assure Accelerate
32. Effect of Echo Level on Voice Quality
Less Echo Less Echo
More Echo More Echo
Affects
TELR – Talker Echo Loudness Rating
(Signal to Echo Ratio) Voice Quality
Analyze Assure Accelerate
33. Network with Variable Delays (Jitter)
• Variable processing delay
– A busy router or switch will take longer to look up the routing (address)
table
• Queuing delay
– Network congestion Affects
Voice and Video Quality
Delay (ms)
Time (s)
Analyze Assure Accelerate
34. Jitter Characteristics
Delay (ms) Delay (ms)
Good
Bad
Delay (ms)
Severe
Time (s)
Affects
Voice and Video Quality
Analyze Assure Accelerate
35. Packet Loss
Example: Queue Management
Threshold
Affects
Voice and Video Quality
Bit Bucket
RED (Random Early Discard)
Analyze Assure Accelerate
37. VAD – Voice Activity Detection
Timing may be different
No VAD
Affects
Voice Quality
VAD
Data is intentionally not sent during times of Silence
Analyze Assure Accelerate
38. Impact Of Packet Size
Affects
Data, Voice and Video Quality
10 Bytes
= 10 ms Speech
20 Bytes Normal size for VoIP applications
= 20 ms Speech
40 Bytes
= 40 ms Speech
80 Bytes
= 80 ms Speech
• Typically Packets are kept small for best results
• Many equipment manufacturers use dynamic packet size
to optimize for network conditions
Analyze Assure Accelerate
39. Mechanisms for Assuring QOS
QoS Class Applications (Examples) Node Mechanisms Network Techniques
Data (Y.1541)
0 Real-Time, loss sensitive, Constrained Routing and
Well Jitter sensitive, high Strict QoS. Guaranteed no Distance
Managed interaction (VoIP, VTC, over subscription on links.
IPTV
1 Real-Time, Jitter sensitive, Separate Queue with Less constrained Routing
Best interactive (VoIP, VTC). preferential servicing, and Distances
Voice Effort Traffic grooming
2 Transaction Data, Highly Constrained Routing and
Interactive, (Signaling) Separate Queue, Drop Distance
3 Transaction Data, priority Less constrained Routing
Interactive and Distances
4 Low Loss Only (Short Long Queue, Drop priority Any route/path
Video Transactions, Bulk
Data, Video Streaming)
5 Traditional Applications of Separate Queue (lowest Any route/path
Internet Default IP Networks priority)
Triple Play
• Class of Service (COS) ITU-T Y.1541 defines the 5 classes of service and their application
• Type of Services (TOS)
Affects
• TOS and COS are both elements with in an IP Packet Data, Voice and Video Quality
• DIFSER and RSVP provide mechanisms to improve QOS
Analyze Assure Accelerate
40. TIA-921 and ITU-T G.NIMM
Test Profiles Based on QoS (Y.1541) Classes
Different test profiles for different Service Level Agreements (SLAs)
Impairment Type Units Range Impairment Type Units Range Impairment Type Units Range
Jitter ms +/- 50 Jitter ms +/- 75 Jitter ms 0 to +/-
250
One Way Latency ms 50 to 100 One Way Latency ms 50 to 200
One Way Latency ms 50 to 400
Sequential Packet Loss #sequential Random Sequential Packet Loss #sequential 2 to 5
packets loss only packets Sequential Packet Loss #sequential 2 to 500
packets
Rate of Sequential Loss sec-1 Rate of Sequential Loss sec-1 0 to 2
Rate of Sequential Loss sec-1 < 10-1
Random Packet Loss % 0 to 0.05 Random Packet Loss % 0 to 2 Random Packet Loss % 0 to 20
Out of Sequence Packets % 0 to 0.001 Out of Sequence Packets % 0 to 0.1 Out of Sequence Packets % 0 to 20
Profile A Profile B Profile C
Well Managed Network Best Effort Managed Network Un-Managed Network
Table 2 Table 3 Table 4
Analyze Assure Accelerate
42. Voice Quality Testing
• Active (Intrusive) Testing
– Sends, Receives and compares Wave Files to measure voice quality
– MOS (Mean Opinion Score)
– PSQM, PSQM+ (Perceptual Speech Quality Measurement)
– PESQ (Perceptual Evaluation of Speech Quality)
– R-Value and J-MOS derived from PESQ
• Passive Testing
– R-Value – ITU-T P.VTQ
• Measures Voice Quality on RTP Packets
• Based on E-model
• Japan – J-MOS
• Similar Techniques can be used to measure Video Quality
– P.563 (ITU-T recommendation) 3SQM, P-Stream
• Measures Voice Quality of Voice traffic based on Audio Siginal
• Provides an estimate of PSQM
Analyze Assure Accelerate
43. Active (Intrusive)
Voice Quality Testing
MOS, PSQM, PSQM+, PESQ, R-Factor (PESQ Derived)
Send Wave Files
DUT Receive Wave Files
Example: (ITU-T Female
Nice File with Pilot Tone)
Measures Voice Quality by Comparing Sent and Received Wave files
Sent (Green)
and Received
(Orange)
wave files
Expanded Sent
(Green) and
Received
(Orange) wave PESQ Score vs Number of PESQ Measurements
files
Values are different for Male, Female, different
Wave Files and different Languages
Analyze Assure Accelerate
44. Passive Voice and Video Quality Testing
R-Factor/Emodel Measure:
Video Quality
MOS-LQ
MOS-CQ
MOS-PQ
J-MOS
Network R
User R
Burst statistics
Diagnostic data
IP
PSTN Network
E1/T1/E3/T3 RTP
/PRI/GR303,
V5,SLC96 Trunking
Gateway RTP
Measure:
Video Quality
ITU-T P.VTQ MOS-LQ
MOS-CQ
MOS-PQ
J-MOS
Network R
User R IP Telephone
Burst statistics
Diagnostic data
Analyze Assure Accelerate
45. Passive Voice Quality Testing
P.563 (P-Stream, 3SQM)
RTP
DUT Receive Audio
Estimates Voice Quality based on 3
Characteristic of Received Audio
Analyze Assure Accelerate
47. Sample Voice Quality Test Results
MOS
PSQM
PESQ
Created by inducing
G.711 packets lost
G.723.1 (6300 bps)
Comparison of Scores for G.711 and G.723.1 (6300 bps)
Analyze Assure Accelerate
48. Video Quality Measurement
• Video Compression
– Video Compress schemes affect the video quality
– H.261, H.263, H.264, VC1, MPEG-1, MPEG-2, MPEG-4,
Microsoft AVI, Windows Media (.wmv, .asf), RealMedia (.rm),
QuickTime (.mov)
• Interactive real-time applications (e.g., video
conferencing, voice over IP) are sensitive to latency and
Frame Rate
• Typical Video Quality Metrics
– Objective MOS – SNR
– Blockiness – Edge Noise
– Blur – Jerkiness
– PSNR – Error Blocks
– Spatial Resolution – Object Retention
– Temporal Resolution – Color Reproduction
Accuracy
Analyze Assure Accelerate
53. Video Quality Measurement
Spatial (Pixel) Resolution
128X128 32X32 8X8
Spatial Resolution
Department of Computer Science
University of Canterbury
Analyze Assure Accelerate
http://www.cosc.canterbury.ac.nz/people/mukundan/covn/Imgresl.htm
54. Video Quality Measurement
Temporal (Motion) Resolution
Vertical-Width (v)
Horizontal-
Width (h)
Temporal-Width (t)
Fk Fk+1 Fk+2 Fk+3 Fk+4 Fk+5
Video Frames
Analyze Assure Accelerate
55. Video Quality Measurement
Models
• Video Quality Metrics (VQM)
– ITU-T SG9 and VQEG are working on standard
• TV Model - optimized for higher bit-rate digital television
systems with no frame dropping (e.g., MPEG-2)
• Videoconferencing Model - optimized for lower bit-rate
videoconferencing systems that drop frames (e.g., H.261,
H.263).
• General Model - optimized for a wide range of video
quality (videoconferencing, TV)
• Developer Model - optimized for a wide range of video
quality (videoconferencing, TV) with the added constraint
of fast computation.
• PSNR Model - based on the traditional peak signal-to-
noise-ratio (PSNR) calculation.
Analyze Assure Accelerate
56. Video Quality Measurement
Techniques
• Full Reference (ITU-T J.144R and BT.1683)
– Video quality is calculated by comparing the received video with
the complete original video
• Reduced Reference (ANSI T1.801.03-2003 and ITU-T
J.143)
– Spatial and temporal information are calculated from original
video and transmitted to the receiving end
– Video quality is calculated by comparing the received video with
the reduced reference
• No Reference – Passive
– Video quality on based only on the received information (picture
content)
– Video quality is derived from RTP packet information similar to R-
Factor (E-Model) for voice quality
Analyze Assure Accelerate
57. Video Quality Measurement
Full Reference
Original Video Received Video
Transmitted Video Signal
Processor Processor
Original Video
Video Quality Score
•MOS
Full Reference
•Blockiness
• Blur
• PSNR
Analyze Assure Accelerate
58. Video Quality Measurement
Reduced Reference
Transmitted Video Received Video
Processor Processor
Reduced-Reference Signal
Low Rate
Video Quality Score
•MOS
Reduced-Reference •Blockiness
•Spatial (Pixel) and Temporal (Motion) Information • Blur
• PSNR
Analyze Assure Accelerate
59. Video Quality Measurement
No-Reference
Transmitted Video Received Video
Processor Processor
No-Reference
Picture Content
or Video Quality Score
Passive monitoring of RTP •MOS
•Blockiness
• Blur
• PSNR
Analyze Assure Accelerate
60. Types of Testing
Type of tests
Test these DUTs
• Video Quality Payload types
• IP PBX
• Voice quality • Video
• Gateways
• Functional • Voice
• IP Phone
• Scalability • Data
• Servers
• Troubleshooting • Fax
• Firewalls
• Conformance • Modem
• IAD
• Interoperability
• Triple Play (Data, With these interfaces
Voice and Video) • GigE 1000Base-SX,
With these protocols 1000Base-LX
• IP • PSTN • 10/100/1000Base-T
– SIP – CAS • Analog
– H.323 – PRI • T1/E1
– MGCP – SS7 • T3/E3
– Megaco/H.248 – NFAS
– Skinny – V5
Analyze Assure Accelerate – GR303
61. Types of Testing
•Call Establishment •Speech Quality Measurements
–Start Dial Signal Delay –PESQ
–Post Dial Delay –PSQM
–Call Duration –MOS
–Ring Duration –R Factor
•Call Disconnect –Echo Delay
–Connection Disconnect Delay –Round Trip Delay
–Release on Request –Echo Return Loss
•Call Statistics –Signal Pass Noise
–Connection set-up failures –Noise Level
–Connection premature disconnect •Video Quality Measurements
–Call completion percentages –MOS
–Blockiness
•Transport Layer Measurements
– Blur
–One-way Transmission Time – PSNR
–Roundtrip Transmission Time
–Jitter
–Packets out of order
–Packet Loss
Analyze Assure Accelerate
62. Video Telephony Testing
Distributed Testing • Isolate Network Problems
Poor
Call Quality Summary • Results Over Time
2% Bad
Fair 1%
Call Quality Summary
24%
Nearly All Users
Dissatisfied Call Completion Rate by Day Fair
Not Recommended
Good • Results by Group
2% 1%
Many Users Poor
Dissatisfied Bad
10% Very Satisfied
One Way Delay by Call Group
20%
Some Users
Dissatisfied Good
180
14%
100 73%
Packet Loss by Call Group
160
95
% Complete
140
90 1.40
120
85 Satisfied
53%
Delay (ms)
80 100 1.20 Call Setup Time by Call Group
% Complete
75 80
Mon Tue Wed Thu Fri Sat Sun
1.00
% Complete 98 97 99 99 96 99 99 Jitter by Call Group
60 350
40 0.80
% Loss
300 80
20
0.60 Jitter by Hour
0
250 70 CHI-to-DAL
Call Setup Time (ms)
NYC - DAL NYC - CHI NYC - SJ DAL - CHI DAL - SJ CHI - SJ
Delay (ms) 122 98 106 173 132 112
0.40 200 60
150 Call Quality Summary by Hour
0.20 50
CHI-to-DAL
Jitter (ms)
100 40 100
0.00
NYC - DAL NYC - CHI NYC - SJ DAL - CHI DAL - SJ CHI - SJ
80
% Loss 0.10 0.43
50 0.40 1.23 0.35 0.64
30
Jitter (ms)
60
0 5.00
20
NYC - DAL 40
NYC - CHI NYC - SJ DAL - CHI DAL - SJ CHI - SJ
Call Setup Time (ms) 150 175 130 313 110 105
20 4.00
10
Jitter (ms)
0
MOS
12 1 2 3 43.00 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11
0 A A A A A A A A A A A A P P P P P P P P P P P P
NYC - DAL NYC - CHI NYC - SJ DAL - CHI DAL - SJ CHI - SJ
M M M M M M M M M M M M M M M M M M M M M M M M
Jitter (ms) 43 51 2.00 41 73 54 45
Jitter (ms) 77 69 67 65 68 68 69 70 73 75 76 93 92 100 82 83 81 80 79 79 79 79 79 78
1.00 MOS
12 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11
AM AM AM AM AM AM AM AM AM AM AM AM PM PM PM PM PM PM PM PM PM PM PM PM
MOS 3.5 3.7 3.6 3.4 3.8 3.4 3.6 3.8 3.5 2.9 2.8 2.3 1.9 1.8 2.9 2.9 3.1 2.9 3.4 3.5 3.5 3.6 3.6 3.7
Analyze Assure Accelerate
63. Good test methodology
Implementation, Validation & Observation
– Conformance testing
• IETF 3261 & new SIP RFC’s
– Stress testing
• Scriptable call flow
• Bulk signaling with real RTP
– Robustness testing
• SIPPING Torture Test
• PROTOS / ETSI TIPHON
– Visual protocol analysis
• Application & content decoding
Analyze Assure Accelerate
