SpringOne 2021 Session Title: Full-Stack Observability for IoT Event Stream Data Processing at Penske Speakers: Krishna Gogineni, Advisory Platform Architect at VMware; Shruti Modi, Director Data Platform at Penske Transportation Solution

1. Full-Stack Observability for IoT Stream
Data Processing at Penske
Shruti Modi - Director, Penske
Krishna Gogineni - Advisory Platform Architect,
VMware
September, 2021

2. Copyright © 3/24/2020 Penske. All rights Reserved. Page 2
How to Connect with your entire fleet of trucks in real time to React and Respond to data
being collected and fix problems before they occur.
Also, how do you monetize the capability?
Connected Fleet at Penske

3. Copyright © 3/24/2020 Penske. All rights Reserved. Page 3
• Observability outcomes for Connected Fleet
• Connected Fleet Solution Architecture
• Implementation - Application & Platform Observability
• Challenges we faced & How we addressed them?
• Application & Platform Observability Highlights
• Summary
Agenda

4. Copyright © 3/24/2020 Penske. All rights Reserved. Page 4
Connected Fleet Observability Outcomes
Reduce MTTR with full stack observability
🚦 Device & Truck monitoring without writing sophisticated custom algorithms
󰟜 Make metrics-driven Platform & Application scaling decisions
🛠 Tools to triage Application & Platform in case of failures
📋 Consolidate Business, Application & Platform metrics in a single place
🏛 Ensure compliance with all laws and government regulation

5. Copyright © 3/24/2020 Penske. All rights Reserved. Page 5
Observability Solution Architecture tenets
Application
• Support for Spring Boot OOTB integration
• Microservices patterns support
• Ability to scale to large data ingestion
• Vendor neutral implementation
• Long term metrics retention to identify
patterns
Platform
• Developer Self-service capability
• Maximize Memory & Compute utilization
through Metrics visualization
• Ability to tie App issues to Platform
components
• Vendor neutral implementation
• Comply with Penske Security and Audit needs

6. Copyright © 3/24/2020 Penske. All rights Reserved. Page 6 6
Connected Fleet Architecture
Tanzu Application Service
Poller Apps SCDF Streams Microservices
Business Users
Real Time
Analytics
VMware Tanzu
Postgres
Observability as a Service
VMware Tanzu
Gemfire
Full Fleet of
Trucks
70M Pings/Day
Tanzu MySQL
1150 App instances with 85 Streams ingesting 700GB Data per day

7. Copyright © 3/24/2020 Penske. All rights Reserved. Page 7 7
Sample flow
Streaming Event Processing
Provider Topic
Truck
Sources
(API)
Event Mapper
Provider agnostic
Topic
Process
&
Normalize
Data Routes
Domain
Topic
Normalized
Data Topic
Domain
Topic
Domain
Topic
Domain
Topic

8. Copyright © 3/24/2020 Penske. All rights Reserved. Page 8
App Observability - Challenges & Solutions
Problem
• Do not want to write custom code to get metrics
collected for Observability
Solution
• Wavefront OOTB Spring Boot starter, able to
collect all metrics exposed through micrometer,
and stored for 18 months.
• Want to join the metrics collected by individual
SCDF apps to derive meaningful insights
• End to End data processing traceability with Spring
Boot, SCDF, Kafka and multiple DBs
• OOTB SCDF integration from Wavefront has made
it a breeze to derive meaningful insights to act on.
• Leveraged OOTB Distributed Tracing capability
provided to trace data processing at Customer,
Truck and Device Level.

9. Copyright © 3/24/2020 Penske. All rights Reserved. Page 9
Platform Observability - Challenges & Solutions
Problem
• Large scale streaming apps logging is causing log
data loss at the Platform Loggregator level
Solution
• With OOTB Doppler and Loggregator monitoring,
able to find out the data transfer rates and adjust
instance counts
• A dead message in Kafka broker causing Streaming
App run in a loop causing App crashes and in turn
Diego cell crashes
• Not able to predict Platform scaling accurately due
to the change in data volumes from Devices &
Trucks
• With OOTB App instance crash integration, able to
identify the App instance and associated messages
causing issues
• With OOTB integration for TAS, able to monitor
Platform components metrics and thresholds for
when to scale each component

10. Copyright © 3/24/2020 Penske. All rights Reserved. Page 10
App Observability Highlights
Polling cycles insights
● Total Polling requests
● Total successful polls
● Total skipped polls
Poll Cycle Insights
● Avg poll cycle
● Avg response time for a poll
Customer poll insights
● Polls spread over customers

11. Copyright © 3/24/2020 Penske. All rights Reserved. Page 11
App Observability Highlights

12. Copyright © 3/24/2020 Penske. All rights Reserved. Page 12
Platform Observability Highlights
Benefits:
● Ability to visualize platform
health & metrics in a single
pane of glass.
● Ability to customize the view
for App, Platform, DevOps &
Business personas.
● Insights of real-time platform
capacity & utilization.
● Historical views of platform
trends for application
adoption and capacity
growth for forecast planning.
● Proactive alerts based on
thresholds to minimize
platform incidents

13. Copyright © 3/24/2020 Penske. All rights Reserved. Page 13
Platform Observability Highlights
Total Available Memory %:
● Total % of memory available in
foundation
Memory Available %:
● Percentage memory available for
TAS, Isolation Segments (SCDF &
API)
Memory Available (GBs):
● Actual memory available for TAS,
Isolation Segments (SCDF & API)
Total Available Disk %:
● Total % of disk available in foundation
Disk Available %:
● Percentage disk available for TAS,
Isolation Segments (SCDF & API)
Disk Available (GBs):
● Actual disk available for TAS, Isolation
Segments (SCDF & API)

14. Copyright © 3/24/2020 Penske. All rights Reserved. Page 14
Platform Foundation
Core building blocks to ensure Applications, Services and Data are on a Secure & Reliable Platform
Platform, App Health & Performance Monitoring:
• Metrics ingestion, Dashboards & Alerts
Observability
Platform, App Logging:
• ELK Stack
Applications Distributed Tracing:
• Spring Cloud Sleuth, Open Tracing

15. Copyright © 3/24/2020 Penske. All rights Reserved. Page 15
Summary
• MTTR reduced by 75% as result of full stack observability with Single pane of
Glass
• Able to monitor the end to end fleet performance with the Metrics
• Ongoing Scaling and Capacity Planning with the help of Metrics collected
• Developers and Platform Engineers are able to get to the root causes quicker
with the Granularity of Metrics
• Consolidated Business, App & Platform Metrics in a single place
• Anomaly detection and fixing issues has saved hours and dollars for Connected
Fleet program.

16. Copyright © 3/24/2020 Penske. All rights Reserved. Page 16
We are Hiring!!
https://penske.jobs/information-technology-jobs/new-jobs/