Logging rails application behavior to MongoDB

1. Logging rails application behavior to MongoDB
Vasiliev Alexey Yuriyovich
leopard.in.ua
2011
Vasiliev A.Y. Logging rails application behavior to MongoDB

2. MongoDB
Introduction
MongoDB is an open source document-oriented database.
Features
Consistent UTF-8 encoding;
Cross-platform support;
Efficient storage for large binary data, for example photos and
video;
Replication and failover;
MapReduce.
Vasiliev A.Y. Logging rails application behavior to MongoDB

3. Why application should log to DB?
Centralized application
logging;
Easy remote access,
compared to files;
Can be fast and scalable if
you go with database
partitioning and/or multi-log
databases.
Vasiliev A.Y. Logging rails application behavior to MongoDB

4. What are the advantages appear logging in MongoDB?
Flexible schema;
Asynchronous inserts;
Capped collection;
Fast enough compared to
files;
Indexed and searched;
Analyze data in-place.
Vasiliev A.Y. Logging rails application behavior to MongoDB

5. Gem for Rails
MongoDB as Rails logger
Central Logger (https://github.com/
customink/central_logger). You should do
only several steps:
gem ’central_logger’;
include CentralLogger::Filter;
Config the connection to mongodb;
That’s all!
Vasiliev A.Y. Logging rails application behavior to MongoDB

6. Rails log structure
MongoDB as Rails logger
{
’ action ’ : action_name ,
’ application_name ’ : application_name ( r a i l s root ) ,
’ controller ’ : controller_name ,
’ ip ’ : ip_address ,
’ messages ’ : {
’ info ’ : [ ],
’ debug ’ : [ ] ,
’ error ’ : [ ] ,
’ warn ’ : [ ],
’ fatal ’ : [ ]
},
’ params ’ : { },
’ path ’ : path ,
’ request_time ’ : date_of_request ,
’ runtime ’ : execution_time_in_milliseconds ,
’ url ’ : full_url
}
Vasiliev A.Y. Logging rails application behavior to MongoDB

7. Add data to log
MongoDB as Rails logger
Add additional data to log
# make s u r e we ’ r e u s i n g t h e C e n t r a l L o g g e r i n t h i s e n v i r o n m e n t
i f R a i l s . l o g g e r . r e s p o n d _ t o ? ( : add_metadata )
R a i l s . l o g g e r . add_metadata ( : u s e r _ g u i d => @ u s e r . i d )
end
Vasiliev A.Y. Logging rails application behavior to MongoDB

8. Querying via the Rails console
MongoDB as Rails logger
Handle on the MongoDB
>> db = R a i l s . l o g g e r . mongo_connection
=> #&l t ; Mongo : : DB: 0 x 1 0 2 f 1 9 a c 0 @ s l a v e _ o k= n i l ...
>> c o l l e c t i o n = db [ R a i l s . l o g g e r . m o n g o _ c o l l e c t i o n _ n a m e ]
=> #&l t ; Mongo : : C o l l e c t i o n : 0 x 1 0 3 1 b 3 e e 8 . . .
Find all requests for a specific user (with guid)
>> c u r s o r = c o l l e c t i o n . f i n d ( : u s e r _ g u i d => 1 2 3 5 5 )
=> #&l t ; Mongo : : C u r s o r : 0 x 1 0 3 1 a 3 e 3 0 . . . &g t ;
>> c u r s o r . count
=> 255
Vasiliev A.Y. Logging rails application behavior to MongoDB

9. Querying via the Rails console
MongoDB as Rails logger
Find all requests that took more that two second to complete
>> c o l l e c t i o n . f i n d ( { : r u n t i m e => { ’ $gt ’ => 2 0 0 0 } } ) . c o u n t
=> 3
Find all requests with an exception
>> c o l l e c t i o n . f i n d ( { " m e s s a g e s . e r r o r " => / R o u t i n g E r r o r / } )
Vasiliev A.Y. Logging rails application behavior to MongoDB

10. Querying via the Rails console
MongoDB as Rails logger
Find by params values
>> c o l l e c t i o n . f i n d (
" params . a c t i v i t y " => { ’ $ e x i s t s ’ => t r u e } ,
" params . a c t i v i t y . s t o r i e s " => { ’ $ e x i s t s ’ => t r u e } ,
" r e q u e s t _ t i m e " => {
’ $ g t e ’ => s t a r t _ d a t e . u t c ,
’ $ l t e ’ => end_date . u t c
}
) . s o r t ( " params . a c t i v i t y . o c c u r r e d _ a t " , 1 ) . c o u n t
=> 542
Vasiliev A.Y. Logging rails application behavior to MongoDB

11. Querying via the Rails console
MongoDB as Rails logger
Find current status of story using logs
>> c o l l e c t i o n . f i n d (
" params . a c t i v i t y . s t o r i e s . i d " => s t o r y _ i d . to_i ,
" params . a c t i v i t y . s t o r i e s . c u r r e n t _ s t a t e " =>
{ ’ $ e x i s t s ’ => t r u e }
) . s o r t ( " params . a c t i v i t y . o c c u r r e d _ a t " , −1). f i r s t
=> {" _id"=>BSON : : O b j e c t I d ( ’ 4 e 7 c 6 2 e d 7 6 1 2 6 8 0 4 d 4 0 0 0 0 0 f ’ ) ,
" a c t i o n "=>"hook " ,
" c o n t r o l l e r "=>" s e r v i c e s " , " p a t h "=>"/ a p i / v1 / hook " ,
" params"=>{" a c t i v i t y "=>{ . . . ,
" s t o r i e s " = > [ { . . . , " c u r r e n t _ s t a t e "=>" f i n i s h e d " } ] } }
Vasiliev A.Y. Logging rails application behavior to MongoDB

12. Querying via the Rails console
MongoDB as Rails logger
Tailable Cursors
v a r c o l l = db . some . c a p p e d . c o l l e c t i o n ;
var l a s t V a l = c o l l . f i n d ( ) . s o r t ({ ’ $ n a t u r a l ’ : 1 })
. l i m i t ( 1 ) . next ( ) [ ’ increasing ’ ];
while (1){
c u r s o r = c o l l . f i n d ({ ’ i n c r e a s i n g ’ : {
’ $gte ’ : l a s t V a l
} });
// t a i l a b l e
c u r s o r . addOption ( 2 ) ;
// a w a i t d a t a
c u r s o r . a d d O p t i o n ( 32 ) ;
// W a i t s s e v e r a l s e c f o r more d a t a
w h i l e ( c u r s o r . hasNext ( ) ){
v a r doc = c u r s o r . n e x t ( ) ;
l a s t V a l = doc [ ’ i n c r e a s i n g ’ ] ;
p r i n t j s o n ( doc ) ;
}
} Vasiliev A.Y. Logging rails application behavior to MongoDB

13. Using MapReduce
MongoDB as Rails logger
Find average time of page loading
>> c o l l e c t i o n . g r o u p ( " f u n c t i o n ( x ) {
r e t u r n { month : x . r e q u e s t _ t i m e . getMonth ( ) ,
year : x . request_time . g e t F u l l Y e a r ( ) } ; }" ,
{ : r e q u e s t _ t i m e => {
’ $ g t e ’ => Time . u t c ( 2 0 1 1 , 0 1 , 0 1 , 0 0 , 0 0 , 0 0 ) ,
" $ l t " => Time . u t c ( 2 0 1 1 , 8 , 0 1 , 0 0 , 0 0 , 0 0 ) }
} , { : c o u n t => 0 , : t o t a l _ t i m e => 0 } ,
" f u n c t i o n ( doc , o u t ) {
o u t . c o u n t ++; o u t . t o t a l _ t i m e+=doc . r u n t i m e } " ,
" f u n c t i o n ( out ){
o u t . avg_time = o u t . t o t a l _ t i m e / o u t . c o u n t } " )
=> [
{" month "= >4.0 ,
" y e a r "= >2011.0 ,
" c o u n t "= >5385.0 ,
" t o t a l _ t i m e "= >1027015.0 ,
" avg_time "= >190.717734447539} ,
...
]
Vasiliev A.Y. Logging rails application behavior to MongoDB

14. Restrictions
MongoDB as Rails logger
The database doesn’t allow update the
existing objects in the collection (but
there is possibility).
The database doesn’t allow deleting
objects from a capped collection.
Capped collection are not shardable.
More indexes - slower inserts.
Vasiliev A.Y. Logging rails application behavior to MongoDB

15. Q&A
Thank you for attention!
Vasiliev A.Y. Logging rails application behavior to MongoDB