Collecting Logs For Trouble-shooting & Analysis

1. Cisco Confidential© 2011 Cisco and/or its affiliates. All rights reserved. 1
CLP
Central Logging Platform
Collecting Logs For Trouble-shooting & Analysis
Shay Asher
SW Engineer, Headend Infrastructure, SPVTG, Cisco

2. © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 2
• High performance scalable distributed persistent logging
• Quick system-wide web real-time trouble-shooting
• Quickly scan through logs with a time-based slider
• System wide online log search and filtering based on:
• Component
• Level
• Time
• Tags
• Flow analysis
• Define & filter log patterns online
• Monitor log patterns via M+ integration

3. © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 3

4. © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 4
• logview.components
• logview.tags
• logview_<year>_<mo>_<day>_<hr>-<hr>.logs

5. © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 5
• Define log search patterns in web UI
• Patterns are held in tags DB and periodically cached in loggers
• Logs are efficiently scanned for patterns via boyer-moore
• Matched logs are added with “tags” field. “tags”: [tag1, tag2,…]
• “tags” fields are indexed with a sparse index
• Users can filter logs matching pre-defined patterns
• Monitor system can query log patterns and issue alerts
• Pluggable BI system

6. © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 6
• Minimize RAM / Disk usage
Field names: shortened conventions (ex. “t” = “time”)
Data types: component names (hashed) and log levels persisted as int32
• Minimize Number of Indexes
“_id” is used as flow context id to avoid an additional index
• Ensure Even Load distribution From Start
Log collections are created, indexed and pre-splited via daily JS cron
• Avoid Deletes / TTL & Memory Fragmentations
Old expired DBs are dropped at once via daily JS cron job
• Keep Indexes in RAM
Old existing log DBs indexes are maintained in RAM via “touch”
• Minimize Global Locking, Enhanced Concurrency
Logs are partitioned to different 4 hr DBs

7. © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 7
• More filters = more indexes = more RAM
• Minimal production startup costs (2 high memory shards + app server)
• Scale out fast with minimal IO (cron new pre-split on new DB)
• 20k logs/sec = 1.7B logs a day, index size may reach +50G/host
• Tested to uphold 100 components logging against 2 shards
• Tested to sustain a load of ~250 bytes average logs at 20k writes/sec
• <10ms average latency, shouldn’t block logging applications
• Logs are kept for 3-7 days back
• No Redundancy, No HA
• Logs are written to MongoDB at lowest durability settings (conn ACK)

8. © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 8

9. © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 9

10. © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 10

11. © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 11