The free ebook is about troubleshooting Skype for Business and Lync.
A complex solution in unified communication marking people's life more simpler, connecting to others at any point of time, staying in contact with fellow friends and family members. Developing a set of skills, supporting and analyzing issues in this environment is an advanced task. I describe the troubleshooting work flow from general understanding of Skype for Business and Lync Server and Services.
In the Troubleshooting Guide the following areas are covered:
- General Approach to troubleshooting
- Logging, Tracing and CLS
- TCP and SIP Protocol
- SIP Session Establishment
- Lync/ Skype for Business Call Setup (entire process)
- Troubleshooting IM
- Troubleshooting Call with A/V
- Diagnostic Headers
- Monitoring
- Troubleshooting Voice
- Troubleshooting Conferencing
- Troubleshooting Web Services
- Troubleshooting Edge (external/ remote)
- Health Monitoring
- Troubleshooting Exchange Integration
(Autodiscover, Exchange Web Service EWS, IM Integration in OWA, Unified Contact Store UCS, Unified Messaging)
- Troubleshooting Mobility Services
- Troubleshooting Mobile Clients
- Troubleshooting Office Web App Server (OWA)
Troubleshooting Enterprise Voice will be released during a future update of this document (Version 2.0)
3. Contents
Skype for Business and Lync troubleshooting guide............................................................................... 2
Preamble and about the author.............................................................................................................. 6
Lync and Skype for Business Troubleshooting approach........................................................................ 7
Environmental components................................................................................................................ 8
Edge Server.......................................................................................................................................... 9
Conferencing Flow............................................................................................................................. 10
Voice Call Processing......................................................................................................................... 11
Support and troubleshooting tools................................................................................................... 12
Client Tracing Log-File location: .................................................................................................... 12
Server Tracing Log-File location: ................................................................................................... 12
Converting Tracing Log-File location:............................................................................................ 12
Service Site Logging (Central Logging Service):............................................................................. 14
General information on TCP and SIP protocol ...................................................................................... 20
IP protocol ......................................................................................................................................... 20
TCP/IP protocol.............................................................................................................................. 20
UDP over IP protocol..................................................................................................................... 22
TLS/ MTLS.......................................................................................................................................... 22
SIP protocol ....................................................................................................................................... 23
SIP protocol session setup............................................................................................................. 23
SIP Commands:.............................................................................................................................. 24
SIP Message Fields:........................................................................................................................ 26
Simple SIP Call Setup ..................................................................................................................... 28
Session establishment and differences between IM, A/V and Conferencing....................................... 30
Authentication internal and remote ................................................................................................. 30
Presence Query ................................................................................................................................. 30
IM Sessions........................................................................................................................................ 31
Audio/Video Session (Desktop/ Application - Sharing)..................................................................... 32
Conferencing ..................................................................................................................................... 33
When a call is escalated into a conference................................................................................... 35
Lync Call Setup....................................................................................................................................... 36
Call Setup over EDGE Server (General) ............................................................................................. 36
Analyzing real world call setup.......................................................................................................... 37
INVITE the USER (OUTGOING)....................................................................................................... 37
TRYING (INCOMIG) ........................................................................................................................ 40
SESSION PROGRESS (INCOMING).................................................................................................. 40
4. PROGRESS REPORT (INCOMING) -2 times (identically send) ........................................................ 41
RINGING (INCOMING) – 4 times.................................................................................................... 41
PROGRESS REPORT (INCOMING)................................................................................................... 42
PRACK (OUTGOING) ...................................................................................................................... 44
OK (INCOMING)............................................................................................................................. 44
SESSION PROGRESS (INCOMING).................................................................................................. 45
PRACK (OUTGOING) ...................................................................................................................... 46
OK (INCOMING)............................................................................................................................. 47
OK (INCOMING)............................................................................................................................. 48
ACK (OUTGOING)........................................................................................................................... 50
INVITE (OUTGOING) ...................................................................................................................... 51
TRYING (INCOMING)...................................................................................................................... 52
OK (INCOMING)............................................................................................................................. 53
ACK (OUTGOING)........................................................................................................................... 54
UPDATE (OUTGOING).................................................................................................................... 55
OK (INCOMING)............................................................................................................................. 56
BYE (INCOMING)............................................................................................................................ 57
OK (OUTGOING)............................................................................................................................. 58
Troubleshooting IM, Calls with A/V....................................................................................................... 59
AV Address Exchange, negotiation of candidates............................................................................. 59
Audio Video Call failed with ms-client-diagnostics (one client is external): ................................. 62
Audio Video Call failed with ms-client-diagnostics: (both client are external):............................ 65
Diagnostic headers............................................................................................................................ 67
MS-DIAGNOSTICS .......................................................................................................................... 67
MS-CLIENT-DIAGNOSTICS.............................................................................................................. 70
Monitoring Reports and Call Quality Issues...................................................................................... 73
Example: Submitting Metrics after Conference call...................................................................... 76
Software Defined Networking (SDN)............................................................................................. 79
Preventing Configuration and other Issues (Testing Commands)..................................................... 80
IM................................................................................................................................................... 80
Voice.............................................................................................................................................. 81
Conferencing ................................................................................................................................. 83
WEB Services ................................................................................................................................. 84
EDGE (external/ remote)............................................................................................................... 85
Health Monitoring Test User......................................................................................................... 85
Troubleshooting Exchange Integration................................................................................................. 86
5. Verify Exchange AutoDiscover setup................................................................................................. 87
Exchange Unified Contact Store Integration..................................................................................... 88
Exchange IM integration on Outlook Web Apps............................................................................... 91
Exchange Web Service Integration.................................................................................................... 92
Exchange Unified Messaging Integration.......................................................................................... 93
Two more important troubleshooting task have to be validate................................................... 94
Troubleshooting conferences................................................................................................................ 96
Persistent Shared Object Model (PSOM) protocol............................................................................ 97
External FQDN with single IP address: .............................................................................................. 98
External FQDN with multiple IP addresses:....................................................................................... 99
Conference INVITE and ACCESS....................................................................................................... 100
Call flow explanation to the illustration above ........................................................................... 101
Why not Single IP on EDGE Port 444 Problem….............................................................................. 102
Client doesn’t open Lync when meeting link is clicked................................................................... 108
Validating Conference Settings and Expiration............................................................................... 109
Activation and Deactivation ............................................................................................................ 110
Resetting a default Conferencing ID................................................................................................ 112
Troubleshooting Lync and Skype for Business Web Services.............................................................. 115
Internal and External Web Services IIS............................................................................................ 115
Mobility Services (for mobile clients).............................................................................................. 118
Scenario 1 (internal mobile/internal full client):............................................................................. 119
Scenario 2 (internal mobile behind internal firewall/internal full client): ...................................... 120
Scenario 3 (internal mobile/external full client): ............................................................................ 120
Having a look into the discovery and logon process:...................................................................... 121
Lync 2010 Mobile App:................................................................................................................ 122
Lync 2013/ Skype for Business Mobile App (Windows, iPhone, iPad and Android)................... 122
Address Book Web Services for Mobile Devices......................................................................... 123
Troubleshooting Office Web App Server............................................................................................. 125
Enterprise Voice .................................................................................................................................. 128
Voice Route and Trunk parameter.................................................................................................. 128
References........................................................................................................................................... 130
6. Preamble and about the author
First I have to say thanks to my wonderful wife supporting me during the writing and to my actual
company I’m with. I had to spend some time writing on this free eBook, which consumed quite an
amount of my time with my family.
This eBook is about troubleshooting Skype for Business and Lync. A complex solution in unified
communication making people’s life more simpler, connecting to other at any point of time, staying
in contact with fellow friends and family members.
Planning and build UC solution is only on site of the coin. Understanding how this technology works is
the other side. Developing a set of skill supporting and analyzing issues in this environment is even
more advanced. Therefore I decided, after I receive many inquiries, supporting my fellow blog reads
in troubleshooting. Writing a guide not only focusing on troubleshooting procedures, instead I
explain the complexity in this area. It is essential for troubleshooting to understand where, or at
which point within a communication path the issue might have occurred.
Thomas Poett (Author and Microsoft MVP for Lync)
Professional, consistent, and experienced expert who is technically savvy with over 20 years of
experience in IT, telecommunication and software development. Additional extensive experience in
business and market development. Specialized in intercultural and business relationship in Asia.
Successful in providing leadership on new topics and complex global projects that require interfacing
with internal/external teams and ecosystems. Early adaptor of visionary technologies. He is awarded
as a Microsoft MVP for more than 3 years, sharing Lync knowledge and guidance for planning
processes. I achieved the Premier Support for Lync Partners (PSLP) certification and support the
teams for Lync 2010/ 2013 3rd
level troubleshooting.
Special thanks to:
Jeff Schertz (Polycom), Richard Brynteson (MVP), Thomas Binder (Microsoft) and Johann Deutinger
(Ferrari electronics AG), my Allgeier workmates for their support and information provided
personally or via their blogs.
7. Lync and Skype for Business Troubleshooting approach
Seeing troubleshooting from all perspectives, we need a matrix where we are enabled analyzing the
area which possibly can causes any issues.
As identified, we see 4 major and a common configuration area.
The areas are (Quality issues):
Network
Core Performance
Gateway
Devices
The area of configuration (environment setup):
Voice Setup (from Dial Plans until Normalization and Routes)
Gateway configuration
Exchange Unified Messaging integration
Making your troubleshooting approach faster. Here is a short approach of the most common issues.
NOTE:
If you are facing an issue with AV not working externally check the following
1- PORTS (This is normally the issue)
2- DNS Records
3- Certificates and trusts
8. Environmental components
Since I spoke about the network, let’s see what else can be identified:
NETWORK SERVER CLIENT
WAN
Router / Switches
Bandwidth
Firewalls
Type of network (wired/
wireless)
MIS-Configuration
(Enterprise Voice/ DNS)
Server Resources
Application Settings
(Client CU s)
Configuration
Connectivity
Seeing here the three essential areas of involved components. Mostly, after you had reviewed the
involved server, which goes along with the configuration, you see the issues related to your network.
This is why we highly emphasize the importance of a network assessment and the implementation of
SDN, respective the implementation of network monitoring.
Coming once back to the configuration. Not only is the Enterprise Voice afflicted with configuration
issues, so to DNS and Exchange integrations. That’s why it is important, you have a proper
environment planning done upfront. Configuration issues can be therefore identified during a
conceptual review.
Network is what matters most. Therefore you need to understand the reliability of networks. Your
LAN is more reliable than your WAN, while the Internet is the most unreliable network. During
troubleshooting, you have to identify this location where the issue occurred.
9. Edge Server
Two important aspects are understanding the flow and processing of conferencing, voice and once
more the Edge server:
INTERNET DMZ CORPORATE
HTTPS (443)
HTTP (80)
XMPP (TCP:5269)
SIP/TLS (TCP:443)
SIP/MTLS(TCP:5061)
DNS(UDP/TCP:53)
PSOM/TLS(TCP:443)
STUN(UDP:3478)
STUN(TCP:443)
RTP(UDP/TCP:50.000-59.999)
HTTPS(4443)
HTTP (8080)
CLS/MTLS(TCP:50001)
CLS/MTLS(TCP:50002)
CLS/MTLS(TCP:50003)
XMPP/MTLS(TCP:23456)
SIP/MTLS(TCP:5061)
PSOM/MTLS(TCP:8057)
SIP/MTLS(TCP:5062)
STUN(UDP:3478)
STUN(TCP:443)
HTTPS(TCP:4443)
HTTP (Public CA CRL Check)
Reverse
Proxy
Lync Edge
Ext. Proxy Service
External IP
Access Edge Service
External IP
XMPP Proxy Service
WebCon Edge Service
External IP
AV Edge service
External IP
Edge A/V
Authentication
Service
Edge
Internal IP
Int. Proxy Service
Internal IP
Lync
External Web
Services
Internal Web
Services
Lync Services
OAuth Service
In case if I repeat this statement, please do not use Edge server with a single IP address. Segregate
the three different service by an individual IP address.
Please understand the TCP/ UDP port openings and the related packet direction. This is what matters
during your firewall setup.
10. Conferencing Flow
Microsoft implemented their own conferencing protocol, call CCCP or C3P (Conference Control
Channel Protocol/ Centralized Conference Control Protocol), based on Framework for Conferencing
with the Session Initiation Protocol. [RFC4353]
Reference:
http://msdn.microsoft.com/en-us/library/cc431498(v=office.12).aspx
Within a conference, 4 different MCU are existing. During troubleshooting you might need to trace
some or all MCU’s with OCSLogger/ CLS. This provides you the internal view what is going on inside.
Tracing the protocol, since the clients are joining a conference via SIP, you trace as usual. In the
prospective of a client joining a conference, you will see the C3P over SIP only.
11. Voice Call Processing
With Enterprise Voice topics it is difficult to start with.
Nevertheless, the first always is the understanding of the call process and it’s related flow with all
involved components. While a voice call initiated with a SIP URI is immediately processed, the call
using a dialed number follows an entire different flow.
As we can see in the call processing flow, the second decision is made where the call is identified as
an E.164 call, a call starting with a “+” sign. If not the number will be normalized. Again here, please
make sure it is always E.164. Once the normalization is done, the important Reverse Number Lookup
takes place, where the number is search for a matching user, either AD or Outlook. This enables the
client to display the user name instead of a phone number. If the user is identified as an internal user
enabled for UC, the call will be processed via it’s SIP URI.
Only form here the call processing in direction to external will start. It involves the checkup for invalid
numbers and Call Park Orbits, processed to now by the Voice Policies with their PSTN Usage Records
and finally the Routes with all their configurations.
After this processing the call is passed over to the Mediation Server and the related TRUNK
configuration.
Initiated LyncCall
E-9-1-1? Global?
Dial Plan
Normalization Rule
Normalization Rule
Normalization Rule
Call Park Orbit Range
404: No
matchingrule
Reverse Number Lookup
Location Policy Routes 3.Voice Policy
403: No Route
found
Mediation Server and Trunk
Configuration
Route
Route
Route
Route
PSTN Usage
PSTN Usage
PSTN Usage
1. Vacant Number Range
2. Call Park Orbit
Announcement or
Call Park Application
Gateway / IP-PBX / SIP Trunk
External Endpoint Receives CallLync Endpoint Receives Call
Inbound Routing
NO NO
YESYES
MATCH
SIP URI User=phone
PSTN Usage
NO MATCH
Dialing
behavior
Routing &
Authorization
If this process/ flow is correctly configured and ran along, the next troubleshooting step should occur
on the Trunks, Gateway and other involved SBC components.
I have dedicated an entire chapter regarding Enterprise Voice troubleshooting.
12. Support and troubleshooting tools
Which tools can be used for analyzing?
Most important is SNOOPER, you need to trace the SIP messages. NETMON is even important if you
want to look in front of SIP, into TCP. Here you most best analyze the impact of firewalls or routers.
Other tools are:
ICE Warning Flag Decoder
PreCall Diagnostic
(Central Logging Service)
(Audio Test Service)
In CSCP: Voice Routing Test Case
Client Tracing Log-File location:
The Lync 2010 client tracing logs are located at: %userprofile%tracing.
The Lync 2013 client tracing logs located at:
%userprofile%appdataLocalMicrosoftOffice15.0LyncTracing
Server Tracing Log-File location:
Both, for Lync Server 2010, Lync 2013 and Skype for Business the log-file location is
c:windowstracing
Once he log file is converted and visible in Snooper, the temporarily generate log fine (txt format) will
be in the user profile
%userprofile%appdataLocalTempOCSLogger_xxxx_xx_xx_xx…
Converting Tracing Log-File location:
The CLS and OCSLogger use event log format (*.etl) those files cannot be opened. They are
compressed in ratio 1:5. You need to convert those etl files to txt. You only can do this manually from
the command line. The exact file location must be provided after the “package for PSS” (/pss) option.
C:Program FilesMicrosoft Lync Server 2013Debugging
Tools>OCSLogger.exe "/pss:c:windowstracing"
This will open OCSLogger, where you need to choose the “etl” files by pressing the “Analyze Log
Files” button.
13. In the next windows, the possible (found) log files will be displayed.
Where after you can view the log in Snooper for further analyses.
NOTE:
The OCSLogger depends on the server version. You need to use OCSLogger in the command line from
the correct server source. Else the file will not be converted properly.
A mismatch can be identified, if the result in Snooper look like the following illustration.
14. Service Site Logging (Central Logging Service):
In lager deployments or even if you run the servers in a pool setup, the logging can become a hassle
in case of finding the correct server, where the troubleshooting needs to be started. If you have
multiple servers in a pool, you can’t identify the individual server, where a e.g. failing call is
processed. We need to make sure we can trace all activities across those member servers.
Another example is, if you have multiple pool, like several front end pool and mediation pools, you
might also want to trace the path a session is running along.
If we now start using the OCSLogger on all those machines, we have problems consolidating all so log
and as well we do have difficulties starting multiple OCSLogger session nearly simultaneously.
The solution here is: Centralized Logging Service. A service for controlled collection of data, with a
broad or narrow scope.
The service is setup with two components, the CLS Service Agent (listening on incoming command on
TCP port 50001, 50002, 50003) and the CLS Service Controller.
NOTE:
You should entirely learn about CLS. I will provide a generic overview helping you to make use of CLS.
Elements of Central Logging Service
Similar to OCSLogger, we will find those setting for CLS.
In this chapter I have used the Microsoft Technet Examples making the understandings of CLS more
transparent.
15. There are three kinds of CLS elements:
Providers
are the COMPONENTS in OCSLogger
Logging levels
OCSLogger provided the option to choose a number of levels of detail for the data collected.
All
of type fatal, error, warning, and info
Fatal
messages that indicate a failure
Error
messages that indicate an error, plus fatal messages.
Warning
messages that indicate a warning for the defined provider, plus fatal and error
messages.
Info
messages that indicate an informational message for the defined provider, plus fatal,
error, and warning messages.
Verbose
messages of type fatal, error, warning and info for the defined provider.
Flags
defined what type of information could retrieve
TF_Connection
information about connections established to and from a particular component
TF_Security
events/log entries related to security.
For example, for SipStack, these are security events such as domain validation
failure, and client authentication/authorization failures.
TF_Diag
diagnostics events like DNS warnings/errors.
TF_Protocol
protocol messages like SIP and Combined Community Codec Pack messages.
TF_Component
components specified
All
Sets all available flags available for the provider.
Scenario for Central Logging Service
A scenario include the aforementioned elements and define the scope of logging. The scope can be
either a computer, a pool, sites or global. However you can only maximum two different scenarios
for any given scope at any given time.
In Lync or Skype for Business management shell, you must provide an identity addressing
configurations. This identity defines the scope in CLS.
16. e.g. –Identity “site:Europe/LyssServiceScenario” –Provider
$LyssProvider or –Name “LyssServiceScenario” – Parent “site:Europe”
–Provider $LyssProvider
As we have seen, the Provide is defined as a string, this is because of the provider has to be
configured the following way too:
$LyssProvider = New-CsClsProvider -Name "Lyss" -Type "WPP" -Level "Debug" -
Flags "TF_Connection, TF_Diag"
The process of working with an CLS Scenario follows the principle of:
New-CsClsScenario -Name "SIPStack" -Parent "site:Europe" -Provider
$SIPStackProvider
After creating a scenario, can further modify is:
Set-CsClsScenario -Identity <name of scope and scenario> -Provider
@{Replace=<providers to replace existing provider set>}
If you need to remove a scenario, this will be done by: Remove-CsClsScenario -Identity
<name of scope and scenario>
Removing or adding a provider to existing scenario uses the Edit-CsClsScenario:
Remove:
Edit-CsClsScenario -ScenarioName <scenario to edit> -ProviderName
<provider to remove> -Remove
Add:
Edit-CsClsScenario -ScenarioName <scenario to edit> -ProviderName <
provider to add> -Level <type level> -Flags <type flags>
Having a look into the preconfigured scenarios, which are likely to be sufficient for the beginning:
Get-CsClsScenario | fl *ident*
Additionally, we should have a look into the provider, which provides the information about the
component its level and flags.
17. Get-CsClsScenario | Where-Object {$_.identity -like
"Global/AlwaysOn"} | Select-Object provider | Select-Object -
ExpandProperty provider
Configuration Settings for Central Logging Service
Before we can start, stop, flush or search results from CLS, we need to have the configuration for CLS
defined. As in the same way of scenarios, we can choose to define different setting over different
scopes (Global or Site).
The illustration below, should help you define those settings:
Command in management shell are:
Set-CsClsConfiguration
Remove-CsClsConfiguration
New-CsClsConfiguration
Get-CsClsConfiguration
18. Start and Stopping Scenarios for Central Logging Service
Starting and Stopping must be configured from the management shell. It is recommended in
troubleshooting using advance paramters.
Start-CsClsLogging -Scenario UserReplicator -Duration 8:00 -Pools
"pool01.contoso.net"
The default scenario is ALWAYSON, logging all relevant level of information and cycles the log files. If
you have this scenario started, at any given point of time you are enabled having a look into the log
files and extract what is need (see next chapter about searching)
Start-CsClsLogging -Scenario AlwaysOn
Stopping the CLS is available with Stop-CsClsLogging -Scenario AlwaysOn
Analyzing which scenario is running simple type: Show-CsClsLogging
Searching in Central Logging Service
Searching the log file is most crucial for troubleshooting. Whenever you need information make sure
you know what your are looking for.
The Search-CsClsLogging is the powerful command helping you extracting not only
information from a single computer. It is more extracting information about an entire path or even
the entire environment.
Not enough, if can also filter based on IP-Addresses or URI, components Sip Contents and more.
Example:
Search-CsClsLogging -pool "sykpe-pool.contoso.com" -IP
"192.168.0.242" -Uri "sip:hans@contoso.com" -MatchAny
19. Here is a table of all parameters you can include in your search.
Parameter Description
CallId Call identifier for specific call.
Components list of components.
Computers list of the computers
ConferenceId Conference ID
CorrelationIds list of correlation IDs to search
EndTime Specify local time zone. Defaults to 5 minutes after current time if no
StartTime specified, otherwise defaults to 30 minutes after StartTime
-StartTime "8/31/2012 8:00AM"
IP IP address
LogLevel minimum type of log entry
MatchAll all the included criteria must be matched.
MatchAny only one of the included criteria must be matched. This is the default
setting, similar to a OR command
OutputFilePath Defines the log file search result as text file to the specified location
and name. Otherwise they are written to the console.
Phone Phone number to be searched for. It must match E.164 format!.
Pools Comma-separated list of the pools
SipContents Arbitrary text to search for within the body of a SIP message.
SkipNetworkLogs instructs the Search-CsClsLogging cmdlet to avoid searching network
logs.
StartTime Beginning date and time for the log entries to be searched. Specified in
local time zone. Defaults to 30 minutes before EndTime.
Uri Uri to be searched for.
Note:
The best possibility for an end-to-end trace of SIP session is provided if you use the Centralized
Logging Service. You are entitled drawing an end-to-end session flow chat. This helps you verifying a
SIP session and other relevant data helpful troubleshooting Lync and Skype for Business.
20. General information on TCP and SIP protocol
Before you start troubleshooting or build your skill for troubleshooting, the basic understanding how
the underlying protocols are working is essential.
First we start with the IP protocol, while a TLS/ MTLS inside view will be discussed. Finally the SIP
protocol is the most essential for troubleshooting.
NOTE:
In troubleshooting the entire knowledge about the 7 layer ISO model is required. You need to
identify where possible issues are to locate. ISDN has the same layer approach, therefore identifying
e.g. if it is a connection or configuration issue, you need the understandings of all this dependencies.
IP protocol
In Lync/ Skype for Business, we make use of two ISO layer of IP, TCP (Transmission Control Protocol)
and UDP (User Datagram Protocol).
TCP/IP protocol
Generic in Lync and Skype for Business, all communication runs of TCP. This includes the internal and
external IP traffic. TCP is also always a fallback path for Audio and Video data if the UDP path has
issues or fails.
FIN WAIT 1
SYN
RECEIVE
SYN
SENT
ESTABLISHED
CLOSING CLOSE WAIT
FIN WAIT 2 TIMED WAIT LAST ACK
CLOSED
CLOSE
LISTEN
Non expected event
Path client/ receiver
Path server/ sender
Session
Start
LISTEN/- CLOSE/-
CONNECT/SYN (Step 1 of 3-way-handshake)
CLOSE/-
SYN/SYN+ACK (Step2 of 3-way-handshake
RST/- SEND/SYN
ACK/- SYN+ACK/ACK
(Step 3 of 3-way-handshake)
State of transmission
CLOSE/FIN FIN/ACK
Active session closing Passive session closing
ACK/-
FIN+ACK/ACK
FIN/ACK
FIN/ACK
ACK/-
TIMEOUT
CLOSE/FIN
ACK/-
SYN/SYN+ACK (simultaneously open)
21. The drawing gives the entire overview of the process how TCP transmission will occur. If you are
using Microsoft NETMON, you can entirely trace the TCP transmission between the sender and
receiver.
During troubleshooting an identified often issues on firewalls, where entire or only a single direction
was blocked. You need the approach to identify, which path drops packets or also which packet got
lost. The path a packet is running is relevant too, especially if you have firewalls in place.
Trace always that the sending and receiving stream is running the EXACT same path. Sometime in
routed environments you will find that source and destination network is routed differently from the
prospective of sender and receiver.
I give you an example in the next illustration, where the path is routed differently as you can see. This
is a very common mistake. Not only inside the LAN, especially in the DMZ/INTERNET setup.
10.10.10.254
10.10.20.1
10.10.20.254
10.10.10.1
ROUTING:
10.10.20.0/24 GW 10.10.10.254
ROUTING:
10.10.10.0/24 GW10.10.20.254
TCP SYN
WRONG PATH
TCP ACK
CORRECT PATH
TCP ACK
The Sender Receiver Setup follows the first initial TCP setup as illustrated in the next table:
SYN-SENT → <SEQ=100><CTL=SYN> → SYN-RECEIVED
SYN/ACK-RECEIVED ← <SEQ=300><ACK=101><CTL=SYN,ACK> ← SYN/ACK-SENT
ACK-SENT → <SEQ=101><ACK=301><CTL=ACK> → ESTABLISHED
You can also see the SEQ (Sequence Number). This is where the packet order comes into the place.
22. UDP over IP protocol
The UPD transmission is quite identically with TCP, beside it is not RELIABLE. Meaning, the sender do
not “care” if the client received the send packet. It just keeps streaming. This explains why Audio and
Video data is best to be UDP.
As we learned from our CD/ DVD players, this data can be “error corrected”. It doesn’t really matter
if we have some streaming information lost. Same applies to Lync/ Skype for Business. We can
transmit AV data over unreliable networks, e.g. internet or Wi-Fi. When the packet lost reach certain
level, first we drop the call quality, if the lost rate is still increasing, the connection might be
terminated.
Another subject of matter in UDP is, the order in which the packets are flowing in at the receiver side
do not matter, since there is no control in place and process bringing them into order.
Lync and Skype for Business with all their codecs do never start a UDP communication if the sender
and receiver didn’t agree of using UDP. The AV session establishment will always be TCP first. Why?
We need to negotiate a lot of upfront setting, e.g. the chosen protocol/ path, the codec and more.
Only after the negotiation, the UDP dataflow starts.
TLS/ MTLS
Most common mistake during encryption. You can’t verify this often enough.
Lync and Skype for Business is “SECURED by DESIGN!”, no communication ever goes unencrypted.
Both server and client must just certificated based encryption. Authentication also relay on
certificates, after initial NTLM/ KERBEROS authentication took place at the very first connect.
The TLS-DSK technology, where a Lync/ Skype for Business server act’s as a certificate authority,
handling the clients personal, per user base certificate over and the client stores this certificate in its
local store.
This is also the only certificate NOT having any trusted root authorities required. Since the Lync/
Skype for Business authentication service can himself identity the certificate. Make sure during
troubleshooting, that this certs are present on a client site and valid (date). The period can be set per
Server.
All other communication internally, as well externally relays on privat and public certificate
authorities (CA). Where the certificate need the correct CN/ SN, the root CA must be in the Trusted
Root Authority store. Please refer here to my blog about certificates used with Lync.
Explaining the difference between TLS and MTLS can be consolidated into:
While a TLS connection is session oriented, the MTLS secured connection can handle multiple session
in parallel.
23. SIP protocol
With SIP protocol we came finally into the first real troubleshooting aspects for Lync and Skype for
Business.
Once we had verified that TCP/IP and UDP is working correctly, we must have a look into the
communication itself. Since we understand, the entire traffic is encrypted, we cannot use NETMON
anymore. We would only see TLS communication flying around and don’t really see anything related
to the SIP communication. Well, we can identify the destination ports and can assume now it SIP or it
CCCP.
Here the CLS (Centralized Logging Service), OCSLogger and Snooper is our tool of choice. Only about
the snooper and all it parameter we could make an entire technical reference. Since we want to
focus on troubleshooting and the main issues, as well as the approach of troubleshooting. The core
components are SIPStack and S4. Just if we analyze SIP, the SIPStack is our favorite. (in 3rd
level
support you need Snooper and CLS for very detailed analyzes even down to the Server components.)
If you utilize Enterprise Voice, please be aware under all circumstance make use of E.164 number
format. Learn everything about how to plan, setup and implement Enterprise Voice. Regarding this
topic please ready my Demystify Enterprise Voice article. In Enterprise Voice you not only
troubleshoot SIP, more like you have to troubleshoot the number format, e.g. why Exchange UM is
not doing reverse number look up or why a GW destination isn’t reached.
SIP protocol session setup
Before we are having a look into a SIP communication, you need to understand SIP communication.
Similar with a TCP session establishment and handling, SIP follows this approach too.
Additionally to the login and register workflow, the SIP communication basics should help you understanding
voice based solutions on Lync and Skype for Business. If you initiate an IM Session the flow is with a call setup.
The provided workflow can be seen as identically if the entire call setup runs over multiple systems involved,
e.g. Client, Server, and Gateway, hopping through all parties. Part of those workflows are the understandings of
SDP, ICE and Early-Media. Here the path can be tracked in the VIA or the ROUTE header.
The illustration below shows a successful established call between the CALLER and CALLEE. The Caller
initiates the call be sending an INVITE to the Callee, who then returns the information of RINGING and OK.
After the receiver of those both commands, the Caller must acknowledge this action by sending an ACK. This
than after the Callee has hook off the call the RTP media starts to be transmitted, as voice session is
established. The site now hanging up will send the BYE command, which has to be acknowledged by 200 OK.
24. SIP Commands:
SIP commands describe the session setup information. They are the core in SIP communication.
While in the next chapter, the message fields are send along with the SIP command, provide more
detailed and necessary information. If you use SNOOPER, SNOPER has the capability displaying the
CALL FLOW based on the SIP Commands.
INVITE
(https://datatracker.ietf.org/doc/rfc4235/?include_text=1)
Command that is starting all dialogs, Calls, Presence and IM. Dialogs can be theoretically created by
many different methods, although RFC 3261 defines only one: the INVITE method.
RINGING
(https://datatracker.ietf.org/doc/rfc3960/?include_text=1)
An acknowledgement send informing about the target is ringing. Also related to Early-Media.
SESSION PROGRESS
(http://www.ietf.org/proceedings/46/I-D/draft-ietf-sip-183-00.txt)
Addressed to the RINGING and the related SDP Message. Enhancing the RINGING with further
information.
25. OK
Simple protocol related acknowledgement on any command needed to be accomplished.
PRACK
(https://www.ietf.org/rfc/rfc3262.txt)
A similar command like the BYE, but not acknowledging. A provisional response on the INVITE. It will
be marked by a RSeq, referring to the related command send earlier (e.g. INVITE).
A sub command within the PRACK is RAck, it response to support reliability of provisional responses
ACK
Command acknowledging the progress made. Related to the SIP protocol RFC.
BYE
Termination command for ending a SIP session.
26. SIP Message Fields:
The SIP messages fields are your gateway for identifying what will and what is going on and represent
how the communication flow will be established. This short description is supported by the reference
to the originated RFC. It provides you the most simple and fastest understanding of SIP protocol
troubleshooting
CALLER-ID:
Unique identifier for each call (best for grouping calls in Snooper)
AUTHENTICATION-INFO:
This field provides information about the possible and choose authentication method, e.g.
NTLS.KERBEROS or TLS-DSK.
VIA:
The path the SIP message run along, providing the path from the source to target
Record-Route
This field is similar to the VIA field, but contains information about the FQDN.
FROM:
“display name”<SIP Address> and tags + identifier
A SIP address either start with SIP: for a sip call or TEL: for phone call
TO:
Target e.g. user, phone or application
P-ASSERTED-IDENTITY:
https://www.ietf.org/rfc/rfc3325.txt
The PAI header provides a way to verify the identity of the caller. Regarding those settings, you need
to understand the SIP Trunk configuration in Lync/ Skype for Business:
http://technet.microsoft.com/en-us/library/jj688104.aspx
ALLOW:
This lists the “allowed” SIP commands usable with in this session.
27. CSeq/ RSeq/ RAck:
An increasing number starting with the first command, mostly INVITE, the CSeq rever also back in
other command, which work as a response to the initial command. RSeq and RAck, are similar to
CSeq, but act with in sequence as a “sub-counter”.
User-Agent:
Identifies the client type, e.g. Lync client, a phone edition or even the Server Application itself.
UCCAPI/15.0.4667.1000 OC/15.0.4667.1000 (Microsoft Lync)
ms-diagnostics/ ms-client-diagnostic:
The most important message, client and IP/ Port information are provided with in the statement. You
can analyze why this action in sequence was chosen, e.g Call terminated by a user, or other network
related causes
Ms-user-logon-data:
e.g. RemoteUser, identifies, from where the user is logged in
ms-media-location-type
Within the SIP message is identified, which network the client is in. Supporting the choice for
matching candiates.
a=candidate
Every client can be position in different networks, either in LAN (corporate or private home), could
be in the Internet. The candidate show the possible connection method, if TCP or UDP and which IP
addresses are involved, LAN or behind NAT.
This is the information clients need knowing the best possible path connection media data.
SERVER:
An information field, if a Skype for Business Server or client is sending this SIP message. Possible are
also Applications. Most interesting is the client version, either Lync client, or mobile device. This
helps us identifying the talking/ sending device of the related message.
There are many more options included in SIP messages, but fact is for 90% of all troubleshooting cases the
documented fields and commands are sufficient enough.
28. Simple SIP Call Setup
In one of the chapters aforementioned, the session establishment and closing was described for the
TCP/IP protocol. If we compare those establishments, we will find similarities valid for SIP session
establishment too. Within the next two paragraphs, we are digging into a Call setup, explain at which
point in a session, media data streams are establishment and what early-media is.
Without Early-Media
This “normal call setup” relates to the basic session establishment,
Caller Callee
| |
| INVITE |
|------------------------------>|
| |
| 100 Trying. |
|<------------------------------|
| 101 Session Progress | (Report)
|<------------------------------|
| 180 Ringing |
|<------------------------------|
| 200 OK |
|<------------------------------|
| ACK |
|------------------------------>|
| Both Way RTP Media | most likely UDP Data if possible
|<=============================>|
| BYE |
|<------------------------------|
| 200 OK |
|------------------------------>|
| |
Lync and Skype for Business have different scenarios, where the path of AV data is different. So
please keep in mind, if two clients are in a same or directly routed network segment (without NAT),
the media data stream is established always between the both clients, regardless if you are within
the same Topology or Federated Partner. IM session indeed have a server involved, IM cannot be
transferred directly between clients. Same is valid if we have a multi-party call, where the Server
MCUs handle the AV distribution.
NOTE:
Making a troubleshooting approach successful, first you should stress what is expected. Meaning
here from where to where the call flows, which components (Servers) are involved.
If a user is busy and can’t accept a call. A message of 486 Busy Here instead of 180 Ringing is
presented to Caller. The Caller send a BYE to the Callee and the session is aborted.
29. With Early-Media in Lync
Modern SIP environments support a faster call setup. This requires the both client starting data
exchange earlier, before a final IP path was negotiated. This is called early-media. Where the audio/
video session is established before the called party takes the call.
Early Media and Ringtone generation is described in the RFC 3960.
http://www.ietf.org/rfc/rfc3960.txt
Allowing early media in a SIP call, we must have an INVITE and 183 Session Progress
command being send and this command contains the SDP data (Session Description Protocol).
Caller Callee
| |
| INVITE | (contains SDP information - caller)
|------------------------------>|
| |
| 100 Trying. |
|<------------------------------|
| 180 Ringing |
|<------------------------------|
| 183 Session Progress | (contains SDP information - callee)
|<------------------------------|
| 101 Progress Report |
|<------------------------------|
| PRACK |
|------------------------------>|
| Both Way RTP Media | EARLY MEDIA starts flowing A/V
|<=============================>|
| 200 OK |
|<------------------------------|
| ACK |
|------------------------------>|
| Both Way RTP Media | most likely UDP Data if possible
|<=============================>|
| BYE |
|<------------------------------|
| 200 OK |
|------------------------------>|
| |
Different compared with the first session initiation is the early-media involvement. Instead of a 101
Session Progress replay, we need to include more information from the called target. That’s why it
sends an enhanced 183 Session Progress, which contains the Session Description Protocol (SDP) in
the 101 Progress Report message. This SDP is used to establish a media connection that carries those
network tones and messages. Immediately after the call was taken (you hock off the phone/ call) the
media data can be transferred without any delay.
The acknowledgement will not be a ACK, instead it replies with PRACK and the media (audio) can
start flowing even if the session is not fully established.
The rest of the session follow the identical flow like the first illustration above. Starting with the 200
OK.
30. Session establishment and differences between IM, A/V and
Conferencing
For troubleshooting it is recommended that you fully understand the different types of session
establishments. The behavior for example between an IM Session and an A/V call is quite different.
In case you need to support issues, it is essential to know where to identify and where to start with
your support approach.
In general we differentiate between server involved session, either in one-way or two-way, as well
MCU (Multipoint Connection Unit) or peer-to-peer connections Therefore we have a look into the
different types of communications.
Authentication internal and remote
Authentication
1. After DNS resolution,
Client contacts the Lync
Edge Access Server.
5. Client authenticates
4. Edge presents
certificate to Client
3. Server presents the
certificate to Edge Server
7. Trusted and encrypted
connection established
2. Edge Access Server
connect to Director
Server (Next Hop)
6. Authentication is
processed
Presence Query
Presence is a one-way query, meaning here that the user who want to add presence to a contact
either in his contact list, or when he was opening a communication window, send a query, the
SUBSCRIBE out to the referred target. This message contains an EVENT called presence (“yellow”)
and SUPPORTED of “ms-benotify”. As well as a XML batch is sent containing the query
inbetween the “action name”
One Way problem of Presence and IM… One user can the other not….
SUBSCRIBE sip:alischka@technologies.com SIP/2.0
Via: SIP/2.0/TLS 192.168.0.16:61813
Max-Forwards: 70
From: <sip:thomas.poett@domain-a.com>;tag=a2ed804245;epid=0639570a7f
To: <sip:alischka@kemptechnologies.com>
Call-ID: f7bb816122e24b68b352d07413f063e8
CSeq: 1 SUBSCRIBE
Contact: <sip:thomas.poett@domain-a.com;opaque=user:epid:2b02BQuhtlW-2_O2vbNPYAAA;gruu>
User-Agent: UCCAPI/15.0.4675.1000 OC/15.0.4675.1000 (Microsoft Lync)
Event: presence
Accept: application/msrtc-event-categories+xml, application/xpidf+xml, text/xml+msrtc.pidf, application/pidf+xml,
application/rlmi+xml, multipart/related
Supported: com.microsoft.autoextend
Supported: ms-benotify
Supported: ms-piggyback-first-notify
Proxy-Require: ms-benotify
Proxy-Authorization: TLS-DSK qop="auth", realm="SIP Communications Service", opaque="6B133F97",
31. targetname="xsrvlync7.domain-a.local", crand="ae97593f", cnum="17",
response="43890d90c8fb20d9d4776370dd874f34f71c845c"
Content-Type: application/msrtc-adrl-categorylist+xml
Content-Length: 478
- <batchSub xmlns="http://schemas.microsoft.com/2006/01/sip/batch-subscribe"
uri="sip:thomas.poett@domain-a.com"
name="">
- <action name="subscribe"
id="1784768368">
- <adhocList>
<resource uri="sip:alischka@technologies.com"/>
- </adhocList>
- <categoryList xmlns="http://schemas.microsoft.com/2006/09/sip/categorylist">
<category name="state"/>
<category name="services"/>
<category name="note"/>
<category name="contactCard"/>
<category name="calendarData"/>
- </categoryList>
- </action>
In the message trace of SNOOPER, we see the clients action:
CUccSubscriptionInfo::SetOpStatusForPresentity - Updating status for presentity sip:alaskowski@technologies.com
to 0x80ef012d
From here the client receive, if available the presence update.
IM Sessions
Interestingly, the IM can be seen similar with an email communication. Where a user is sending the
IM via all involved servers. This explains, even if connection to a server is broken, audio/ video
session (a peer-to-peer connection) stay established, but IM will be unavailable. The IM message is a
one-way directed TLS connection from the sending client (in this illustration below) to his Edge
server, via the Director, to Lync server and, the Lync server know the location of the target
participant.
1. IM sent in SIP
connection secured with
TLS
IM Traffic (SIP)
2. Edge forwards IM to
Director Pool Server (SIP/
MTLS)
3. Director Pool forwards
IM to Frontend Pool (SIP/
MTLS)
4. IM is send to client
(SIP/ TLS)
5. IM replies in the
opposite direction
This explains why we sometime see an issue in communication, where user complain they can see
the presence and be able singing IM to a target, but the target can’t reply. This is subject to an issue
in the returning way. Either be port blockings or other network related issues.
32. Audio/Video Session (Desktop/ Application - Sharing)
In Lync and Skype for business, all audio and video related data will be exchanged in a peer-to-peer
manner. But this is not valid for the session establishment. First the client send an INVITE (as we will
see later in chapter: Analyzing real world call setup) from where the a/v path will be established
directly between the two participants.
In troubleshooting you have to analyze the both paths, first the session and afterwards the a/v path.
The session establishment follows the path as described with IM.
IM Traffic (SIP)
1. Initiate IM Session via
Home Pool (SIP/ TLS)
SRTP (SIP)
2.IM Session is forward
to second Pool (SIP/
MTLS)
Lync Pool A Lync Pool B
3. IM Session is send to
Client (SIP/ TLS)
Bidirectional Channel
4. Client add A/V to the
IM Session (Signaling) via
SIP/TLS/MTLS)
5. Signaling is forwarded
to second Pool (SIP/
MTLS)
6. Signaling is send to
Client (SIP/ TLS)
7. A/V session is established via P2P connection, secured with SRTP protocol
Other P2P connections are: Desktop Sharing and File Transfer, both secured with SRTP protocol
The path for a/v depends on the exchange of candidate pairs, you first have to analyze which
candidates were send from both site and figure out the final candidates, the client want to establish
along. (Also this process will be explained later in more detail)
33. Conferencing
A Conference is very similar to a normal SIP call, the main difference here is, that the user contact a
MCU (Multipoint connection Unit) the conferencing server, which will handle all incoming and
outgoing media streams.
Here is a trace of an ad-hoc conference (“Meet now option”):
Let us have a look into the process. We send an invite to the MCU service, which will be normally
acknowledged and established as every other SIP call. The difference here is the conference ID which
is submitted.
The user setting up the conference will include the following application data:
The conference key is the identifier for this conference (“yellow”) and we can see the C3P (CCCP)
Microsoft conference protocol is used. Additionally we submit multiple other information, like the
participant rule in this conference, here (“ATTENDEE”). This is normal, because if a user joint initially
the conference, he should have the lowest rights.
Content-Type: application/cccp+xml
Content-Length: 964
- <request xmlns="urn:ietf:params:xml:ns:cccp"
mscp="http://schemas.microsoft.com/rtc/2005/08/cccpextensions"
C3PVersion="1"
to="sip:thomas.poett@domain-a.com;gruu;opaque=app:conf:focus:id:PTRL3DQ4"
from="sip:thomas.poett@ domain-a.com"
requestId="504251766368">
- <addUser>
<conferenceKeys confEntity="sip:thomas.poett@ domain-
34. a.com;gruu;opaque=app:conf:focus:id:PTRL3DQ4"/>
- <user ci="urn:ietf:params:xml:ns:conference-info"
entity="sip:thomas.poett@ domain-a.com">
- <roles>
<entry>attendee</entry>
- </roles>
- <endpoint entity="{F0228E9E-3B8C-445B-A2AC-343A9FE7735B}"
msci="http://schemas.microsoft.com/rtc/2005/08/confinfoextensions">
- <clientInfo>
<separator cis="urn:ietf:params:xml:ns:conference-info-separator"/>
<lobby-capable
msci2="http://schemas.microsoft.com/rtc/2008/12/confinfoextensions">true</lobby-capable>
- </clientInfo>
- </endpoint>
- </user>
- </addUser>
Later in the SIP/200 INVITE DIALOG CREATED, we see the escalation to the (“PRESENTER”)
rule.
- <user entity="sip:thomas.poett@domain-a.com">
- <roles>
<entry>presenter</entry>
- </roles>
The most interesting message is the last INFO message from 8:57:08.501, here the conferencing
service acknowledged all important information regarding web conferencing to the client. This is a
huge XML message included. The RULE ENTRY is where those information are provided.
NOTE:
For troubleshooting it has two aspects, the client side as show in the illustration above and the server
side. The conference here took place in between 08:57:08 and 08:57:35. On the client side no further
information are provided what was happened in this conference. There you need to start MCU
logging on the Lync/ Skype for Business server itself.
35. When a call is escalated into a conference
It its necessary to explain further more about the peep-o-peer call and a conference. As we
remember, the audio/ video is always peer-to-peer, this include also desktop and application sharing,
because of those data is similar to video. But what is with other services as POOLS, WHITE BOARD, or
Q&A? This are service subject to conferences!
Remember:
Every time a user in a call using one of those services, the call will be first escalated into a
conference!
That’s just not all. There is one more very specific conference service, the Power Point presentation.
If we are going to setup a webcast, or you want to share the presentation upfront with conference
participants, you can upload files into a conference. The Power Point is here special and will be
uploaded to the conference directory, like all other files too. But from here it will be rendered during
a presentation into HTML 5.0 document.
The Lync Client as well the Web Conferencing are HTML 5.0 aware and can present the Power Point
data directly from the Office Web Application Server (WAS or WAC).
Troubleshooting this server component is a chapter on his own.
36. Lync Call Setup
Call Setup over EDGE Server (General)
Next we want to analyze a complex flow from two different Lync. One client sitting inside its LAN and
the other remote (Internet). They need to communicate over the Edge Server.
USER A (Internet) Edge Server Pool USER B (LAN)
SIP INVITE SIP INVITE SIP INVITE
SIP 100 TRYINGSIP 100 TRYINGSIP 100 TRYING
SIP 180 RINGINGSIP 180 RINGINGSIP 180 RINGING
PRACK PRACK PRACK
SIP 183 SESSION PROGRESSSIP 183 SESSION PROGRESSSIP 183 SESSION PROGRESS
SIP 200 OK (PRACK)SIP 200 OK (PRACK)SIP 200 OK (PRACK)
SIP 200 OK (INVITE)SIP 200 OK (INVITE)SIP 200 OK (INVITE)
SIP ACK SIP ACK SIP ACK
SIP INVITE SIP INVITE SIP INVITE
SIP 100 TRYINGSIP 100 TRYINGSIP 100 TRYING
SIP 200 OKSIP 200 OKSIP 200 OK
SIP ACK SIP ACK SIP ACK
Media Session Media Session
Candidate testing Candidate testing
SESSION IS ESTABLISHED
A/V Edge
service
While the client continue negotiating their best possible IP path, the media is establish during the
negotiation process already.
In the next chapter, we are going to have look into a real-world call establishment, where two
federated partners setting up a call. Therefore also two Edge server are involved. In comparison to
the call flow diagram from above, we will simply have one more additional hub in this scenario.
37. Analyzing real world call setup
I provide an example, where one client is external (remote) belonging to domain-a.com and the
second client is internal (LAN) belonging to domain-b.com. This is the setup of an Audio call.
Unnecessary line are removed. From the second communication extract onwards.
usera@domain-a.com INVITE’s userb@domain-b.com. The direction provided is seen from the
initiating client, meaning “OUTGOING” the client is sending a SIP command. “INCOMING” the client
receives a SIP command.
INVITE the USER (OUTGOING)
Starting with the INVITE, where User A invites User B into a voice call.
INVITE sip:userb@domain-b.com SIP/2.0 (whom to invite)
Via: SIP/2.0/TLS 192.168.0.16:54763 (from where, the client IP address)
Max-Forwards: 70
From: <sip:usera@domain-a.com>;tag=c013887c44;epid=0639570a7f (the user initiating the
invite)
To: <sip:userb@domain-b.com> (the whom to invite as target)
Call-ID: ab5a007ca2124e95a227f1c82f58cff9 (our call identifier, if you search for a
dedicated session search based on this ID in SNOOPER)
38. CSeq: 1 INVITE (the SIP message sequence: 1st Invite)
Contact: <sip:usera@domain-a.com;opaque=user:epid:2b02BQuhtlW-2_O2vbNPYAAA;gruu>
User-Agent: UCCAPI/15.0.4667.1000 OC/15.0.4667.1000 (Microsoft Lync) (which client is in
used)
Supported: ms-dialog-route-set-update (this section describes the supported features within this call,
e.g if early-media is possible)
Supported: timer
Supported: histinfo
Supported: ms-safe-transfer
Supported: ms-sender
Supported: ms-early-media
Supported: 100rel
Supported: replaces
Supported: ms-conf-invite
Ms-Conversation-ID: AdAQcNCXWtgHtgCCTziAoXiNeIlHgg==
ms-keep-alive: UAC;hop-hop=yes
Allow: INVITE, BYE, ACK, CANCEL, INFO, UPDATE, REFER, NOTIFY, BENOTIFY, OPTIONS (which SIP
commands are available in this session)
ms-subnet: 192.168.0.0 (the inviting client is within the IP network)
Accept-Language: en-US
ms-endpoint-location-data: NetworkScope;ms-media-location-type=Internet (the client is
connecting via the internet, non-internal client)
P-Preferred-Identity: <sip:usera@domain-a.com>, <tel:+4989zzyy75xx> (the identity submits
information which could be used from the target site)
Proxy-Authorization: TLS-DSK qop="auth", realm="SIP Communications Service",
opaque="9731FB93", targetname="internalFQDN.domain-a.internal", crand="fe334d12",
cnum="1224", response="c262f61fccd9b7e7e915e9a4f5f8b0fb31bdcdd2" (Authentication realm)
Content-Type: multipart/alternative;boundary="----=_NextPart_000_0171_01D0107A.BB7313C0"
Content-Length: 5434
------=_NextPart_000_0171_01D0107A.BB7313C0
Content-Type: application/sdp
Content-Transfer-Encoding: 7bit
Content-ID: <34459f6b4e9af885c99f55d2f63eb545@domain-a.com>
Content-Disposition: session; handling=optional; ms-proxy-2007fallback
v=0
o=- 0 0 IN IP4 195.145.140.92
s=session
c=IN IP4 195.145.140.92
b=CT:99980
t=0 0
m=audio 54712 RTP/AVP 114 104 9 112 111 0 8 116 115 103 97 13 118 101
(This “blue” section is referring to the connection possibilities, first for
compatibility reasons, the “older” version for OCS)
a=candidate:Q4qp+2B3Rejg21tBRcAGfwXRQxwFbPH03U7efzcCp9M 1 75+hnK4QFySwmPMqbWWhJg UDP 0.830 192.168.0.16 21722
a=candidate:Q4qp+2B3Rejg21tBRcAGfwXRQxwFbPH03U7efzcCp9M 2 75+hnK4QFySwmPMqbWWhJg UDP 0.830 192.168.0.16 21723
a=candidate:hxajgyQ3ey1c0Dwrnogo1as2lNIZdsPbHDVH7UJy7mw 1 Uqk/hGPBtIDASJArq1AR2Q TCP 0.110 195.145.140.92 52196
a=candidate:hxajgyQ3ey1c0Dwrnogo1as2lNIZdsPbHDVH7UJy7mw 2 Uqk/hGPBtIDASJArq1AR2Q TCP 0.110 195.145.140.92 52196
a=candidate:Y+hVJGzBEjj4wtencMChvT7yEI8r6nHIPnQXB9Zchyw 1 Zfwzh/lMlJ/3YmOnpiJRdQ UDP 0.410 195.145.140.92 54712
a=candidate:Y+hVJGzBEjj4wtencMChvT7yEI8r6nHIPnQXB9Zchyw 2 Zfwzh/lMlJ/3YmOnpiJRdQ UDP 0.410 195.145.140.92 53613
a=candidate:M8BU29mhcNhmUej1BczG4sPn3yIb+3Rd5zc+bk0uWpY 1 k4fI8epJYEGk6WOVy/xfjA TCP 0.250 178.26.121.167 22616
a=candidate:M8BU29mhcNhmUej1BczG4sPn3yIb+3Rd5zc+bk0uWpY 2 k4fI8epJYEGk6WOVy/xfjA TCP 0.250 178.26.121.167 22616
a=candidate:8+mHOK0Z2Ks8kPymJH6FYE5RuFGbKNB/5ktLSfSF3Ns 1 gnqS5G1KOT/WlJbrw5d/uA UDP 0.550 178.26.121.167 23804
a=candidate:8+mHOK0Z2Ks8kPymJH6FYE5RuFGbKNB/5ktLSfSF3Ns 2 gnqS5G1KOT/WlJbrw5d/uA UDP 0.550 178.26.121.167 23805
a=cryptoscale:1 client AES_CM_128_HMAC_SHA1_80 inline:qk2+Fe5CWZcKttfJVk8wShtvC0ixVWQvVx1tYgYg|2^31|1:1
a=crypto:2 AES_CM_128_HMAC_SHA1_80 inline:eanuVGjZ1PGHP9Eh17H9u5uAr36WW2pjXzrH0s7W|2^31|1:1
a=crypto:3 AES_CM_128_HMAC_SHA1_80 inline:39IvYCDOPzx/MwuZNKuN5RHY+RSA0xz2LWGkBok3|2^31
a=maxptime:200
a=rtcp:53613
39. a=rtpmap:114 x-msrta/16000
a=fmtp:114 bitrate=29000
a=rtpmap:104 SILK/16000 (SKYPE CODEC)
a=fmtp:104 useinbandfec=1; usedtx=0
a=rtpmap:9 G722/8000
a=rtpmap:112 G7221/16000
a=fmtp:112 bitrate=24000
a=rtpmap:111 SIREN/16000
a=fmtp:111 bitrate=16000
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:116 AAL2-G726-32/8000
a=rtpmap:115 x-msrta/8000
a=fmtp:115 bitrate=11800
a=rtpmap:103 SILK/8000
a=fmtp:103 useinbandfec=1; usedtx=0
a=rtpmap:97 RED/8000
a=rtpmap:13 CN/8000
a=rtpmap:118 CN/16000
a=rtpmap:101 telephone-event/8000
a=fmtp:101 0-16
a=ptime:20
(This “red” section is referring to the possible codes with can be utilized)
------=_NextPart_000_0171_01D0107A.BB7313C0
Content-Type: application/sdp
Content-Transfer-Encoding: 7bit
Content-ID: <813134c085fe0bc411c93b200cb11c3e@domain-a.com>
Content-Disposition: session; handling=optional
v=0
o=- 0 1 IN IP4 195.145.140.92
s=session
c=IN IP4 195.145.140.92
b=CT:99980
t=0 0
a=x-devicecaps:audio:send,recv;video:send,recv
m=audio 57962 RTP/AVP 114 104 9 112 111 0 8 116 115 103 97 13 118 101
a=x-ssrc-range:862104576-862104576
a=rtcp-fb:* x-message app send:dsh recv:dsh
a=rtcp-rsize
a=label:main-audio
a=x-source:main-audio
a=ice-ufrag:ccwh
a=ice-pwd:8X3nfjTthYjuA8vhf+z2FYqM
(This “blue” section is referring to the connection possibilities, now for Lync and
Skype for Business)
a=candidate:1 1 UDP 2130706431 192.168.0.16 10668 typ host
a=candidate:1 2 UDP 2130705918 192.168.0.16 10669 typ host
a=x-candidate-ipv6:2 1 UDP 2130705919 fd00:26:5bcb:fb92:d812:2961:203d:afd3 15206 typ host
a=x-candidate-ipv6:2 2 UDP 2130705406 fd00:26:5bcb:fb92:d812:2961:203d:afd3 15207 typ host
a=x-candidate-ipv6:3 1 UDP 33553407 2001:0:5ef5:79fd:34ae:19fd:4de5:8658 23084 typ host
a=x-candidate-ipv6:3 2 UDP 33552894 2001:0:5ef5:79fd:34ae:19fd:4de5:8658 23085 typ host
a=candidate:4 1 TCP-PASS 174455295 195.145.140.92 54427 typ relay raddr 178.26.121.167 rport 11603
a=candidate:4 2 TCP-PASS 174454782 195.145.140.92 54427 typ relay raddr 178.26.121.167 rport 11603
a=candidate:5 1 UDP 184547327 195.145.140.92 57962 typ relay raddr 178.26.121.167 rport 6098
a=candidate:5 2 UDP 184546814 195.145.140.92 51825 typ relay raddr 178.26.121.167 rport 6099
a=candidate:6 1 UDP 1694234111 178.26.121.167 6098 typ srflx raddr 192.168.0.16 rport 6098
a=candidate:6 2 UDP 1694233598 178.26.121.167 6099 typ srflx raddr 192.168.0.16 rport 6099
a=candidate:7 1 TCP-ACT 174846975 195.145.140.92 54427 typ relay raddr 178.26.121.167 rport 11603
a=candidate:7 2 TCP-ACT 174846462 195.145.140.92 54427 typ relay raddr 178.26.121.167 rport 11603
a=candidate:8 1 TCP-ACT 1684795903 178.26.121.167 11603 typ srflx raddr 192.168.0.16 rport 11603
a=candidate:8 2 TCP-ACT 1684795390 178.26.121.167 11603 typ srflx raddr 192.168.0.16 rport 11603
a=cryptoscale:1 client AES_CM_128_HMAC_SHA1_80 inline:qk2+Fe5CWZcKttfJVk8wShtvC0ixVWQvVx1tYgYg|2^31|1:1
a=crypto:2 AES_CM_128_HMAC_SHA1_80 inline:eanuVGjZ1PGHP9Eh17H9u5uAr36WW2pjXzrH0s7W|2^31|1:1
a=crypto:3 AES_CM_128_HMAC_SHA1_80 inline:39IvYCDOPzx/MwuZNKuN5RHY+RSA0xz2LWGkBok3|2^31
a=maxptime:200
a=rtcp:51825
40. a=rtpmap:114 x-msrta/16000
a=fmtp:114 bitrate=29000
a=rtpmap:104 SILK/16000
a=fmtp:104 useinbandfec=1; usedtx=0
a=rtpmap:9 G722/8000
a=rtpmap:112 G7221/16000
a=fmtp:112 bitrate=24000
a=rtpmap:111 SIREN/16000
a=fmtp:111 bitrate=16000
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:116 AAL2-G726-32/8000
a=rtpmap:115 x-msrta/8000
a=fmtp:115 bitrate=11800
a=rtpmap:103 SILK/8000
a=fmtp:103 useinbandfec=1; usedtx=0
a=rtpmap:97 RED/8000
a=rtpmap:13 CN/8000
a=rtpmap:118 CN/16000
a=rtpmap:101 telephone-event/8000
a=fmtp:101 0-16
(This “red” section is referring to the possible codes with can be utilized)
a=ptime:20
TRYING (INCOMIG)
SIP/2.0 100 Trying (Response from the target that its processing the Invite)
ms-user-logon-data: RemoteUser
Authentication-Info: TLS-DSK qop="auth", opaque="9731FB93", srand="7A0933C0", snum="1229",
rspauth="1ae94e2225388db0f72729407866c59710b3d463", targetname="internalFQDN.domain-
a.internal", realm="SIP Communications Service", version=4
From: <sip:usera@domain-a.com>;tag=c013887c44;epid=0639570a7f
To: <sip:userb@domain-b.com>
Call-ID: ab5a007ca2124e95a227f1c82f58cff9
CSeq: 1 INVITE (message response from the 1st, initial invite)
Via: SIP/2.0/TLS 192.168.0.16:54763;received=178.26.121.167;ms-received-port=54763;ms-received-
cid=86B100 (Information about the path seeing the internal sender IP and the NAT IP
on the Internet Connection)
Server: http%3A%2F%2Fwww.microsoft.com%2FLCS%2FUdcAgent
Content-Length: 0
SESSION PROGRESS (INCOMING)
Here we are informed that the session is in progress. The target system is processing the session and
will send more about the process soon. The CSeq is still indicating the dependency on the first
INVITE.
SIP/2.0 183 Session Progress
ms-user-logon-data: RemoteUser
Authentication-Info: TLS-DSK qop="auth", opaque="9731FB93", srand="F7BE52E1", snum="1231",
rspauth="6d8fca262a42dc48169ef0142a2a2b910db30ba5", targetname="internalFQDN.domain-
a.internal", realm="SIP Communications Service", version=4
Content-Length: 0
41. Via: SIP/2.0/TLS 192.168.0.16:54763;received=178.26.121.167;ms-received-port=54763;ms-received-
cid=86B100 (
From: "Thomas Poett"<sip:usera@domain-a.com>;tag=c013887c44;epid=0639570a7f
To: <sip:userb@domain-b.com>
Call-ID: ab5a007ca2124e95a227f1c82f58cff9
CSeq: 1 INVITE
Ms-Forking: Active
Server: http%3A%2F%2Fwww.microsoft.com%2FLCS%2FDefaultRouting(Microsoft Lync Server 2013
5.0.8308.726)
ms-edge-proxy-message-trust: ms-source-type=DirectPartner;ms-ep-
fqdn=lyncedgepool.internFQDN.local;ms-source-network=federation;ms-source-verified-
user=verified (interestingly we see the message is running through the Edge server)
PROGRESS REPORT (INCOMING) -2 times (identically send)
Still processing (send has to wait for next message)
SIP/2.0 101 Progress Report
ms-user-logon-data: RemoteUser
Authentication-Info: TLS-DSK qop="auth", opaque="9731FB93", srand="1E5DE761", snum="1232",
rspauth="e415d70ab0d015011336756cb385f8c0aa35e55e", targetname="internalFQDN.domain-
a.internal", realm="SIP Communications Service", version=4
Content-Length: 0
Via: SIP/2.0/TLS 192.168.0.16:54763;received=178.26.121.167;ms-received-port=54763;ms-received-
cid=86B100
From: "Thomas Poett"<sip:usera@domain-a.com>;tag=c013887c44;epid=0639570a7f
To: <sip:userb@domain-b.com>
Call-ID: ab5a007ca2124e95a227f1c82f58cff9
CSeq: 1 INVITE
Server: InboundRouting/5.0.0.0 (send by target server)
ms-edge-proxy-message-trust: ms-source-type=DirectPartner;ms-ep-
fqdn=lyncedgepool.internFQDN.local;ms-source-network=federation;ms-source-verified-
user=verified
RINGING (INCOMING) – 4 times
This is repeated until the user pickup, rejects or don’t answer (timeout). The Ringing response is
given for each device connected on the target site.
SIP/2.0 180 Ringing
ms-user-logon-data: RemoteUser
Via: SIP/2.0/TLS 192.168.0.16:54763;received=178.26.121.167;ms-received-port=54763;ms-received-
cid=86B100
Authentication-Info: TLS-DSK qop="auth", opaque="9731FB93", srand="FA1B1315", snum="1234",
rspauth="c74126037c499380d505ad2902f868320c3a6a95", targetname="internalFQDN.domain-
a.internal", realm="SIP Communications Service", version=4
FROM: "Thomas Poett"<sip:usera@domain-a.com>;tag=c013887c44;epid=0639570a7f
42. TO: "USER B"<sip:userb@domain-b.com>;tag=32653fb637;epid=212448855504
CSEQ: 1 INVITE
CALL-ID: ab5a007ca2124e95a227f1c82f58cff9
CONTACT: <sip:userb@domain-
b.com;opaque=user:epid:OWi6ihJpQlqfhlrZfAmZywAA;gruu>;text;audio;video;image;applicationshari
ng
CONTENT-LENGTH: 0
SUPPORTED: gruu-10
ALLOW: CANCEL (The ALLOW show the possible commands based on the Ringing, meaning
how to terminate and more…)
ALLOW: BYE
ALLOW: UPDATE
ALLOW: PRACK
P-ASSERTED-IDENTITY: "USER B"<sip:userb@domain-b.com>
SERVER: RTCC/5.0.0.0 UCWA/5.0.0.0 AndroidLync/4.0.0000.0000 (Nexus 7 Android 4.4.4)
[As we saw in the initial screen shot, 4 RINGING response were received, this are
the green marked devices
User-Agent: UCCAPI/15.0.4667.1000 OC/15.0.4667.1000 (Microsoft Lync)
SERVER: RTCC/5.0.0.0 UCWA/5.0.0.0 AndroidLync/5.4.1106.0 (GXV3275 Android 4.2.2)
User-Agent: CPE/4.0.7577.4451 OCPhone/4.0.7577.4451 (Microsoft Lync Phone Edition)]
ms-edge-proxy-message-trust: ms-source-type=DirectPartner;ms-ep-
fqdn=lyncedgepool.internFQDN.local;ms-source-network=federation;ms-source-verified-
user=verified
PROGRESS REPORT (INCOMING)
(The call was now take by USER B (callee), we need another progress report here stating this). We
also need to receive the SDP here that early-media can be established with this candiates).
This is send from the device taking the call to connect with audio. In this trace from a Lync native
client.
SIP/2.0 183 Session Progress
ms-user-logon-data: RemoteUser
Authentication-Info: TLS-DSK qop="auth", opaque="9731FB93", srand="CD84738D", snum="1238",
rspauth="57062b086f1b35c4b848bc42fb28b33897f6963e", targetname="internalFQDN.domain-
a.internal", realm="SIP Communications Service", version=4
Via: SIP/2.0/TLS 192.168.0.16:54763;received=178.26.121.167;ms-received-port=54763;ms-received-
cid=86B100
From: "Thomas Poett"<sip:usera@domain-a.com>;tag=c013887c44;epid=0639570a7f
To: <sip:userb@domain-b.com>;epid=a606e73c89;tag=01c33dbedc
Call-ID: ab5a007ca2124e95a227f1c82f58cff9
CSeq: 1 INVITE (we are still in progress receiving information based on the first
INVITE)
[we first identify the target sites route and involved servers and its CANDIDATES
HERE establishing early-media]
Record-Route: <sip:lync2013.hq.domain-b.com:5061;transport=tls;opaque=state:T;lr>
Record-Route: <sip:ocs.domain-b.com:5061;transport=tls;epid=a606e73c89;lr;ms-key-info=.......;ms-
route-sig=bkNIsAXKnsiRmbGYKjO4lps9o6nw6ySSsyRfgI3kTNTO9ouKskkfZA1AAA>
Record-Route: <sip:lyncedgepool.internFQDN.local:5061;transport=tls;lr>
44. PRACK (OUTGOING)
Reliability of “Provisional Responses” in the Session Initiation Protocol (PRACK RFC3262). Two
possible responses exit: provisional and final. PRACK is the provisional response. We signalize: We
are ready to connect.
PRACK sip:userb@domain-b.com;opaque=user:epid:LLsGlnCjYlGGrdmzIoEFUwAA;gruu SIP/2.0
Via: SIP/2.0/TLS 192.168.0.16:54763
Max-Forwards: 70
From: <sip:usera@domain-a.com>;tag=c013887c44;epid=0639570a7f
To: <sip:userb@domain-b.com>;epid=a606e73c89;tag=01c33dbedc
Call-ID: ab5a007ca2124e95a227f1c82f58cff9
CSeq: 2 PRACK (CSEQ is increased to 2, we continue with the next process establishing
the call)
Route: <sip:sip.domain-a.com:5061;transport=tls;opaque=state:Ci.R86b100;lr;ms-route-
sig=fgFbc2J6H1DsQORB0QtvkWygilYrA-0Ccj13Tr9fH6LkbccFlf6mOs6wAA>
Route: <sip:lyncpool1.internFQDN.local:5061;transport=tls;ms-fe=internalFQDN.domain-
a.internal;opaque=state:F;lr;received=10.35.3.27;ms-received-cid=78E300>
Route: <sip:lyncedgepool.internFQDN.local:5061;transport=tls;lr>
Route: <sip:ocs.domain-b.com:5061;transport=tls;epid=a606e73c89;lr;ms-key-
info=AAEAAcUcKOgG4q8UZBDQAVcU1Pt2m_6ekgNlaaYrCW5qBdhWPxt97pJbLWjJqjzzJFfUkk6MJ2eS
yIm77AmUE76vtBbPqEm5zKxc8R9DxeBx2FZfM49Y16vsDldXQl4uDazo3BWXJDggmtpWiOWkYwhmW
adsT813jJ9JhAIxcPTzltiNbNZ9b0_iffEi6oI6_HlftQCLqKb-n8tq1gIsAtSYw8-o2vGxrswJ8HeIFn6z0TFE-
Sd_F76aV8DYmi-NkRPcwknKnBMwDhNKQ2pl2vZ61B9oFlcy9xFQOdLZujQUyZ27T06xfFqzz3H-
bPyd92DSPFmvjfcvGx6WGqQHI4FIkz2pe9FIKXi8ZVVJhS6EE65YtcKxvu5LfCxSN5P4ZePtClNSdlyV2FLyOt
WYXS72c1xHNM5AT53D2KJjwzRURJ1v--1CLkVuiFfCPe8GSPVbqTLmdNLlmpqtF05UiEa-
9coZDWDdYx41iGUNxfhykgSt2iFoTyjdShLxTg6BbAS5zuf5BY_VUuJwKjx9b0-
ecWhYzTKK_JaQDim4WGPFqRx8PAcx4GZtv6OQMAwYA8A5ZeXLQWa0DL_PFq4BVF5HfR_jfyQciEGjV
Gsc5BXT0CdprYmXNjSyYLzEG4WzCm4jpdX_SnInhJhZ1SZ8kv9n91oV7O7MVwGyBN6ALBbUVCUugkj5
Nfn0qzv_UKDM;ms-route-
sig=bkNIsAXKnsiRmbGYKjO4lps9o6nw6ySSsyRfgI3kTNTO9ouKskkfZA1AAA>
Route: <sip:lync2013.hq.domain-b.com:5061;transport=tls;opaque=state:T;lr>
User-Agent: UCCAPI/15.0.4667.1000
RAck: 1 1 INVITE (a PAck based on our INVITE)
Proxy-Authorization: TLS-DSK qop="auth", realm="SIP Communications Service",
opaque="9731FB93", targetname="internalFQDN.domain-a.internal", crand="ae68c543",
cnum="1226", response="8913674f898d1f2c357350d9e8e9c348d7abc36b"
Content-Length: 0
OK (INCOMING)
The PRACK was acknowledged from the
SIP/2.0 200 OK
ms-user-logon-data: RemoteUser
45. Authentication-Info: TLS-DSK qop="auth", opaque="9731FB93", srand="0B8D6EC0", snum="1239",
rspauth="971556595d08e9e1977484380009bfb0457afeb3", targetname="internalFQDN.domain-
a.internal", realm="SIP Communications Service", version=4
Via: SIP/2.0/TLS 192.168.0.16:54763;received=178.26.121.167;ms-received-port=54763;ms-received-
cid=86B100
From: <sip:usera@domain-a.com>;tag=c013887c44;epid=0639570a7f
To: <sip:userb@domain-b.com>;epid=a606e73c89;tag=01c33dbedc
Call-ID: ab5a007ca2124e95a227f1c82f58cff9
CSeq: 2 PRACK
User-Agent: UCCAPI/15.0.4667.1000 OC/15.0.4667.1000 (Microsoft Lync)
Content-Length: 0
ms-edge-proxy-message-trust: ms-source-type=DirectPartner;ms-ep-
fqdn=lyncedgepool.internFQDN.local;ms-source-network=federation;ms-source-verified-
user=verified
SESSION PROGRESS (INCOMING)
The target is signalizing it is processing the still on the first INVITE. Even it has received the PRACK
and answered for provisional progress. It was also repeating the candidates once more for later use.
This command 183 Session Progress is the responsible trigger for CALLEE’S device signalizing the
incoming call on his/ her device!
SIP/2.0 183 Session Progress
ms-user-logon-data: RemoteUser
Authentication-Info: TLS-DSK qop="auth", opaque="9731FB93", srand="2F18494A", snum="1240",
rspauth="a62e2a1927663ce245b8992e1283c26aaa1716ea", targetname="internalFQDN.domain-
a.internal", realm="SIP Communications Service", version=4
Via: SIP/2.0/TLS 192.168.0.16:54763;received=178.26.121.167;ms-received-port=54763;ms-received-
cid=86B100
From: "Thomas Poett"<sip:usera@domain-a.com>;tag=c013887c44;epid=0639570a7f
To: <sip:userb@domain-b.com>;epid=5385571cf9;tag=1ce13477ae
Call-ID: ab5a007ca2124e95a227f1c82f58cff9
CSeq: 1 INVITE
Record-Route: <sip:lync2013.hq.domain-b.com:5061;transport=tls;opaque=state:T;lr>
Record-Route: <sip:ocs.domain-b.com:5061;transport=tls;epid=5385571cf9;lr;ms-key-
info=AAEAAcUcKOgG4q8UZBDQAVcU1Pt2m_6ekgNlaaYrCW5qBdhWPxt97pJbLWjJqjzzJFfUkk6MJ2eS
yIm77AmUE76vtBbPqEm5zKxc8R9DxeBx2FZfM49Y16vsDldXQl4uDazo3BWXJDggmtpWiOWkYwhmW
adsT813jJ9JhAIxcPTzltiNbNZ9b0_iffEi6oI6_HlftQCLqKb-n8tq1gIsAtSYw8-o2vGxrswJ8HeIFn6z0TFE-
Sd_F76aV8DYmi-NkRPcwknKnBMwDhNKQ2pl2vZ61B9oFlcy9xFQOdLZujQUyZ27T06xfFqzz3H-
bPyd92DSPFmvjfcvGx6WGqQHI4FIkz2pe9FIKXi8ZVVJhS6EE65YtcKxvu5LfCxSN5P4ZePtClNSdlyV2FLyOt
WYXS72c1xHNM5AT53D2KJjwzRURJ1v--1CLkVuiFfCPe8GSPVbqTLmdNLlmpqtF05UiEa-
9coZDWDdYx41iGUNxfhykgSt2iFoTyjdShLxTg6BbAS5zuf5BY_VUuJwKjx9b0-
ecWhYzTKK_JaQDim4WGPFqRx8PAcx4GZtv6OQMAwYA8A5ZeXLQWa0DL_PFq4BVF5HfR_jfyQciEGjV
Gsc5BXT0CdprYmXNjSyYLzEG4WzCm4jpdX_SnInhJhZ1SZ8kv9n91oV7O7MVwGyBN6ALBbUVCUugkj5
Nfn0qzv_UKDM;ms-route-
sig=bk1thRQ_ihhCNLmtXzZOeqNXyFx22uhJJkDX_9qM1XZNBouKskkfZA1AAA>
Record-Route: <sip:lyncedgepool.internFQDN.local:5061;transport=tls;lr>