Weitere ähnliche Inhalte
Ähnlich wie A DevOps Perspective: MongoDB & MMF (20)
A DevOps Perspective: MongoDB & MMF
- 2. Introduction
!
MapMyFitness!was!founded!in!2007
!
Offices!in!Denver,!CO!&!AusRn,!TX
(w/!associates!in!SF,!Boston,!New!York,!LA,!and!Chicago)
!
Over!13!million!registered!users
!
~80!million!geoadata!routes!
(runs,!rides,!walks,!hikes,!etc)
!
Core!sites,!mobile!apps,!API,!whitealabel
(MapMyRun,!MapMyRide,!MapMyFitness)
- 8. Replica Set Expansion
• MongoDB!is!“replicaRon!made!elegant”
• Ridiculously!simple!to!add!addiRonal!members
• Be!sure!to!run!IniRalSync!from!a!secondary!
rs.add(!“host”!:!“livetrack_db09”,!“ini8alSync”!:!{!“state”!:!2!}!)
• Both!rs.add()!and!rs.remove()!can!be!scripted!and!connected!to!
Monitoring!systems!for!autoscaling
- 9. Monitoring and Introspection
•!MMS,!10gen's!cloudabased!monitoring!service!(best!available)
•!Supported!by!Zabbix,!Nagios,!Munin,!Server!Density,!etc
•!mongostat,!mongotop,!REST!interface,!database!profiler
•!Monitoring!system!triggers!can!iniRate!node!addiRons,
!!removals,!service!restarts,!etc
•!In!addiRon!to!servicealevel!monitoring,!use!more!advanced
!!tests!to!check!for!and!alert!on!query!latency!spikes
- 15. Network Security Automation
## Puppet Pattern for Mongodb network security
class iptables::public {
iptables::add_rule { '001 MongoDB established':
rule => '-A RH-Firewall-1-INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT'
}
iptables::add_rule { '002 MongoDB':
rule => '-A RH-Firewall-1-INPUT -i eth1 -p tcp -m tcp --dport 27017 -j ACCEPT'
}
iptables::add_rule { '003 MongoDB MMF Phase II Network':
rule => '-A RH-Firewall-1-INPUT -i eth0 -s 172.16.16.0/20 -p tcp -m tcp --dport 27017 -j ACCEPT'
}
iptables::add_rule { '004 MongoDB MMF Cloud Network':
rule => '-A RH-Firewall-1-INPUT -i eth0 -s 10.178.52.0/24 -p tcp -m tcp --dport 27017 -j ACCEPT'
}
}
- 18. Indexing Patterns or “Know Your App”
• Proper!indexing!criRcal!to!performance!at!scale
(monitor!slow!queries!to!catch!nonaperformant!requests)
• MongoDB!is!ulRmately!flexible,!being!schemaless
(mongo!gives!you!enough!rope!to!hang!yourself,!choose!wisely)
• Avoid!unaindexed!queries!at!all!costs!
(it's!quickest!way!to!crater!your!app...!consider!aanotablescan)
• Onus!on!DevOps!to!match!applicaRon!to!indexes
(know!your!query!profile,!never!assume)
• Shoot!for!'covered!queries'!wherever!possible
(answer!can!be!obtained!from!indexes!only)
- 19. Capped Collections
• Use!standard!capped!collecRons!for!retaining!a!fixed!amount!
of!data.!!Uses!a!FIFO!strategy!for!pruning.
(based!on!data!size,!not!number!of!rows)
• TTL!CollecRons!(2.2)!age!out!data!based!on!a!retenRon!Rme!
configuraRon.!!
(great!for!data!retenRon!requirements!of!all!types)
Gotcha!
Explicitly!create!the!capped!collecRon!before!any!data!is!put!
into!the!system!to!avoid!autoacreaRon!of!collecRon
- 20. Lessons Learned
•!Mongo!2.2!upgrade!containing!a!capped!collecRon!created!in!1.8.4.!!This!severely!impacted!
replicaRon!(RC:!no!"_id"!index,!!FIX:!add!"_id"!index)!
•!Never!start!mongo!when!a!mount!point!is!missing!or!incorrectly!configured.!Mongo!may!
decide!to!take!maSers!into!it's!own!hands!and!resync!itself!with!the!replica!set.!!Make!
sure!your!devops!and!your!hos2ng!provider!admins!are!aware!of!this
•!Some!drivers!that!use!connecRon!pooling!can!freak!the!freaky!freak!when!the!primary!
member!changes!(older!pymongo).!!Kicking!the!applicaRon!can!fix,!also:!upgrade!drivers
•!High!locked!%!is!a!big!redaflag,!and!can!be!caused!by!a!large!number!of!simultaneous!dml!
acRons!(high!insert!rate,!high!update!rate).!Consider!this!in!the!design!phase.
•!Be!wary!of!automaRon!that!can!change!the!state!of!a!node!during!maintenance!mode.!!
Disable!automaRon!agents!for!reduced!risk!during!criRcal!administraRve!operaRons!
(filesystem!maint,!etc)
