Presented @ 55th International Instrumentation Symposium League City, Texas, 1–5 June 2009 Ethernet is being used by a wider variety of industrial devices and applications. Industrial applications and systems require deterministic operations that traditional Ethernet and Transport Control Protocol / Internet Protocol (TCP/IP) suites were not originally designed to support. A standardized way to describe and test industrial devices is needed in order to aid users to characterize the performance of their software and hardware applications. The Manufacturing Engineering Laboratory (MEL) of the National Institute of Standards & Technology (NIST) has been working to develop a set of standardized network performance metrics, tests, and tools since 2002. NIST has cooperated with standards organizations and other groups during that time. NIST is presently working on developing an open-source test tool, called Industrial Ethernet Network Performance (IENetP), to aid vendors in characterizing the performance of their devices. The IENetP test tool will be capable of conducting a full series of performance tests and reporting the results to the user. The current version of the software is capable of analyzing network traffic and producing statistics and graphs showing the network performance of a device.

1. Test Tool for Industrial
Ethernet Network
Performance
Jim Gilsinn & Freemon Johnson
Standards
Certification
Education & Training
Publishing
Conferences & Exhibits
National Institute of Standards & Technology (NIST)
Manufacturing Engineering Laboratory (MEL)

2. Author Bio: Jim Gilsinn
• Over 16 years of experience at NIST
• Variety of manufacturing projects, including intelligent
unmanned vehicles, welding standards, smart sensors,
and nano-manufacturing
• Current research focus is industrial network reliability,
including performance and security
• Masters in Electrical Engineering from Johns Hopkins
University, specialty in Control Systems; Bachelors in
Electrical Engineering from Drexel University, specialties
in Robotics and Control Systems
2

3. Author Bio: Freemon Johnson
• Over 10 years of experience; previously worked at NASA
Goddard Space Flight Center and Motorola
• Expertise in communication protocol security,
architecture, and embedded real-time systems
• Masters in Computer Science from Johns Hopkins
University; Masters in Communication and Information
Studies from Rutgers University; Bachelors in Electrical
Engineering from New Jersey Institute of Technology
3

4. Overview
•
•
•
•
•
Background
Performance Testing Methodology
Data Analysis
Test Tool
Future Plans
4

5. Background
• The Problem
– Ethernet & TCP/IP inherently non-deterministic
– Users have many questions about industrial Ethernet devices
– Automobile Example
• Why NIST?
– Aid U.S. businesses and industries, specifically manufacturing
– Many companies are global
– Industrial Ethernet standards are available worldwide
5

6. Performance Testing Methodology:
Performance Metrics
• Publish/subscribe or peer-to-peer communications
• Main performance metric: Cyclic frequency
variability/jitter
• Real-time EtherNet/IP uses publish/subscribe
– Requested/Accepted Packet Interval (RPI/API)
– Measured Packet Interval (MPI)
6

7. Performance Testing Methodology:
Performance Metrics
• Command/response or master/slave communications
• Main performance metric: Latency
7

8. Performance Testing Methodology:
Testing Methodology
• Overall methodology for testing performance:
1. Begin recording network traffic
2. Establish a connection with the device under test (DUT)
3. Begin transmitting background network traffic, based on the
particular test conditions
4. Wait for a given amount of time
5. Stop transmitting background network traffic
6. Close the connection with the DUT
7. Stop recording network traffic
8. Analyze the network traffic capture and report the results
• Current test tool only for data analysis & reporting
8

9. Performance Testing Methodology:
Test System
9

10. Data Analysis: Overview
• Current test tool focuses on cyclic frequency
variability/jitter of the MPI
• Capable of analyzing multiple network packet streams
from different devices during a single test run
• Conduct a distribution analysis on the MPI results
– Determine minimum, maximum, mean, standard deviation,
skewness, and kurtosis of data set
• Produce graphs of MPI results
10

11. Data Analysis: Example Jitter Graph
Measured Packet Interval (s)
This Example
Test Duration ≈ 61 s
Mean ≈ 2 ms
Minimum ≈ 1.2 ms
Maximum ≈ 2.9 ms
Test Time (s)
11

12. Data Analysis: Observations
12

13. Data Analysis: Observations
13

14. Test Tool: Design
• Geared towards vendors & users
– Vendors: develop better products during design process
– Users: determine how well devices operate in a real system
• Multi-phase design approach
– Analysis → Testing
• Meaningful results
– Mathematical foundation
– User selectable complexity for results
14

15. Test Tool: Technical Details
• Free software package
– Microsoft Windows-based (Windows 2000 or later)
• Public domain source-code
– Visual C# modular design
– Commercial graphing package
• Open-source installed package
– No special software or hardware needed to run
– Integrated with Wireshark for packet analysis
15

16. Future Plans
• Planned versions of IENetP Test Tool
– Data Analysis Tool
– Version 1.x – Basic functionality & test engine
– Version 2.x – Additional data analysis capability & metrics
– Version 3.x – Additional industrial networks
– Active Testing Tool
– Version 4.x – PC-based test tool
– Version ?.? – Specialized hardware test tool
16

17. Summary
• NIST is working with industry partners to develop
industrial Ethernet performance metrics, tests, and tools
– Determined metrics
– Developed basic methodology
• NIST is currently working on a test tool
– Designed as a development tool for industrial Ethernet vendors
– Focused on data analysis presently
• Some patterns have emerged while analyzing data
• Additional functionalities are planned for the test tool
17

18. Contact Information
• Jim Gilsinn, 301-975-3865, james.gilsinn@nist.gov
• Freemon Johnson, 301-975-3429, freemon.johnson@nist.gov
• Mailing Address
– National Institute of Standards & Technology (NIST)
Manufacturing Engineering Laboratory (MEL)
100 Bureau Drive, Mailstop 8230
Gaithersburg, MD 20899-8230
• Websites
– Project: http://ienetp.sourceforge.net
– Division: http://www.isd.mel.nist.gov
18