Presentation from Silicon Valley Code Camp 2013. Related code on github: * https://github.com/hoserdude/mvcmusicstore-instrumented * https://github.com/hoserdude/spring-petclinic-instrumented * https://github.com/hoserdude/nodecellar-instrumented

1. Developing highly
instrumented applications
with minimal effort
Tim Hobson
Principal Engineer, Chief Caffeine Officer - Intuit

2. About Me
2

3. Agenda
Concepts
• Quality Data
• Application
Instrumentation
• Application
Logging
Theory
• Patterns
• Supporting
Frameworks
• Supporting
Components
Practice
• .Net Example
• Java Example
• Node.js Example
GOAL: You should be able to apply this to your application today
3

4. Concepts

5. If You Do Nothing…
OR
5

6. There’s No Free Lunch
Garbage
In, Garbage
Out
Insight!
6

7. Quality In, Insight Out
Most apps start with only the framework or
app server logging (or nothing!)
None of the above is interesting to the
business or the developer.
You can’t get if you don’t give – there are
many ways to give, and many classes of
data to provide.
7

8. Classes of System Output
System Instrumentation
• JMX/WMI/SNMP monitoring
• Apache/IIS/nginx access logs
App Logging
App Instrumentation
•
•
•
•
•
•
•
•
Intentional
Business Transactions
Overtly triggers alerts
Source of business
metrics
• Aids in troubleshooting
failures, bugs
Cross-cutting (free)
App Activity
Passively triggers alerts
Source of performance
data
• Source of usage data
8

9. Theory and
Best
Practices

10. Best Practices (© splunk>)
Create human readable events
Clearly timestamp events
Use key-value pairs
Be aware of multi-value fields
Log unique identifiers
10

11. Best Practices (© Tim)
Global timestamps (UTC – 2013-08-21
22:43:31,990)
Context setting (who/what/where/when/how)
Categories/taxonomy (what tier, what
component)
Timing (time everything!)
Security (never log sensitive data: password=***)
Consistency in naming
– (action=purchase; sale=oct13; productId=123123)
– (action=buy; promo=oct13; sku=123123)
11

12. Example Output
Context
2013-08-21 22:55:36,504; LogLevel=INFO;
sid=q3prv41kt511vzojytnx1d42; rid=6500583;
userLogin=(null); ipAddress=0.0.0.0; thread=249;
category=Web.Controllers.BaseWebController;
msg=RequestInfo; server=ws001prod;
url=https://myapp.com/account/logon; method=GET;
languages=en-US,en;q=0.8;
referrer=https://myapp.com/members/dashboard;
userAgent=Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/28.0.1500.95 Safari/537.36; userId=123456;
controller=Account; action=Logon;
Metrics
2013-08-21 22:55:36,519; LogLevel=INFO;
sid=q3prv41kt511vzojytnx1d42; rid=6500583;
userLogin=(null); ipAddress=0.0.0.0; thread=249;
category=TraceInterceptor;
timeTracing=Service.Users.GetUser, time=3;
12

13. Patterns
Pipeline
– Filter
– Interceptor
Dependency injection
– Proxy
Pointcuts/aspects
13

14. HTTP Pipelines
var app =
express();
app.use(func1);
app.use(func2);
app.use(func3);
app.use(func4);
app.use(func5);
app.use(func6);
app.use(func7);
app.use(func8);
app.listen(8080);
Java (from Oracle
Docs)
ASP.Net (from MSFT
Docs)
14
express.js

15. Dependency Injection
•
Enforces interface-based programming
•
Forbids circular references
•
Lends itself to testability
•
Flexibility in changing implementations
•
For our purposes: cross-cutting
capabilities
15

16. Intercepting Calls With Dynamic Proxies
Interceptor
Dependency
A
Implementation
DI Container
16
Interceptor
Interceptor
Proxy
Interceptor
Interceptor
Interceptor
Caller
Interceptor
Proxy
Interceptor
Interceptor
Proxy
Dependency
B

17. Supporting Frameworks
The Front
Door
The Inner
Sanctum
Annotations
& Attributes
• ASP.Net
Modules/ActionFi
lters
• Java Servlet
Filters/Spring
Interceptors
• Node.js
interceptors
• Unity Dynamic
Proxies
• Spring
@AspectJ
Pointcuts
• JavaScript
Mixins
• @Instrumentable
• [Instrumentable]
17

18. Supporting Components
.Net
• Log4Net
• NLog
Java
node.js
• LogBack
• SLF4J
• Log4J
18
• SenchaLabs
Connect
• Winston

19. Intervention!

20. Sample App Intervention
3 Platforms
3 Sample
Apps
20
3 Interventions

21. Goals
Leverage an HTTP pipeline for context
Leverage aspects for interception
Apply best practices
Minimize impact on existing code
21

22. MVC Music Store Intervention (Demo)
Missing dependency injection/interception framework: Unity
Missing logging library: Log4Net, Buche
1.
2.
3.
4.
5.
6.
7.
8.
Add DI framework (Unity)
Configure aspects (Interception)
Configure controller factory, container locator
Set up logger (Log4Net)
Configure log pattern and targets (Log4Net.xml)
Wire up logging interceptor
Wire up LogActionFilter and BaseLoggingController
Deploy!
22

23. Spring Pet Clinic Intervention (Demo)
1. Set up logger
a. Configure log pattern (to support context data)
b. Configure targets (rotating file)
2. Create LogAspect.java and @Instrumentable attribute
a. Capture calling context
b. Start/stop timing
3. Create LogInterceptor.java
a. Capture request metadata
b. Set MDC
c. Start/stop timing
4. Configure aspect bean
1. Configure interceptor bean
1. Annotate the methods we care about
23

24. NodeCellar Intervention (Demo)
Missing Interception Framework: Scarlet
Missing Logging Library: Winston
1.
2.
3.
4.
5.
6.
Set up Scarlet
Configure log pattern
Configure method interception
Create logging interceptor
Create LoggingFilter
Deploy!
24

25. Closing
Thoughts

26. Got Log? Now Get Intimate With Your App
Ops
Dashboards
Business
Dashboards
Quality
Assurance
Pro-Active Service
Degradation Alerting
SLA
Tracking
Security
Alerting
2
6
Customer
Support
Performance
Metrics
Pre-Release
Sanity
Testing

27. Key Takeaways
•
It is YOUR responsibility as a developer to
provide useful operational and business data.
•
It is not hard, and most of it is for free once you
have the patterns in place.
•
The same patterns and practices can be applied
to practically any platform, and any type of
application or service.
•
When you provide consistent and predictable
data others can build on your greatness
27

28. Get the Code
@hoserdude
Spring Pet Clinic Intervention:
https://github.com/hoserdude/spring-petclinic-instrumented
MVC Music Store Intervention:
https://github.com/hoserdude/mvcmusicstore-instrumented
NodeCellar Intervention:
https://github.com/hoserdude/nodecellar-instrumented
28

29. Intuit Speakers @ Silicon Code Camp 2013:
SATURDAY
9:45 a.m. - Ramakrishna Kollipara – “Complete Automation of Performance Testing”
1:45 p.m. - Joe Wells - “QBO: Journey From legacy Java app to a Client-side HTML5
app”
3:30 p.m. - Naga Addagadde & Sangeeta Narang – “Intuit APIs for Financial Transaction
Aggregation”
5:00 p.m. Ted Drake –“Hitting the Accessibility High Notes with ARIA”
SUNDAY
9:15 a.m. - Eugene Krivopaltsev –“Building Native Mobile Apps with Custom Views”
1:15 p.m. - Tim Hobson – “Developing Highly Instrumental Applications with Minimal
Effort”
For more information about joining our organization visit our booth or
connect with our onsite recruiter:
Chriscox_recruiter@intuit.com
You don't want to miss out on a chance to win this cool headset.
Stop by our booth to enter!

30. THANK YOU
Want to talk more?
I’ll be at the Intuit booth today 3-5 PM.
30