1. Laine Campbell,CEO
Jay Edwards, CTO
Charlie Killian, VP of Professional Services
Kevin Bowman, VP of Operations
An Introduction to Palomino

2. Who we are
Palomino staff have been managing large production
systems since before the dot com era began.
Technorati, Twitter, Mozilla Firefox, Digg, Travelocity,
Friendster, Chegg, Obama for America, Activision and
StumbleUpon are only part of our pedigree. We've
scaled companies through years of explosive growth.
*

3. Our Partners
Palomino builds partnerships that make sense for our clients and staff,
allowing us to augment our expertise.
● Amazon Web Services
● SkySQL
● MariaDB
● Basho
● Couchbase
● 10Gen
We also build key relationships with products we believe are useful and
appropriate to our clients, such as Scalearc, Continuent and TokuTek.
Palomino never resells, or takes commission on any products they
recommend.
*

4. Professional Services
● Choosing your DBMS
● High availability and disaster recovery
● Search, caching, queuing and security
● BI, ETL and real-time analytics
● Large distributed database systems (aka Big Data)
● Configuration management via Chef, Puppet and Ansible
● Scripting and automation
● Cloud Solutions, AWS and Joyent
● Operational tools development and implementation
● Backup, recovery and data management
*

5. Operational Support
● Tier 1 Oncall, integration with customer monitoring
● Tier 2 Escalation and problem solving.
● Team integration, including email, chat room and IM collaboration, as
well as regular stand-ups and project mgmt meetings.
● Proactive health checks, recovery tests, capacity and security reviews.
● Project level work as needed.
● Ownership of release management and configuration management.
○ Query Reviews.
○ DDL and DML Reviews
○ Safe production change management
● Ongoing improvements of incident and problem management.
*

6. Remote Engineering Services
● Pre-deployment
● Data modeling
● Data migration
● Cluster sizing
● Load testing and QA
● Application profiling
● Performance tuning
*

7. How we can help
● 24x7 operational support - Chegg.com
● MySQL revamping and support - Echosign
● Operations team needed - SoHalo
● Verifying systems will scale under load - Contest Factory
● Lousy application performance - Zeitbyte
● Tools development - Zendesk and Slideshare
● Proactive services - SendGrid
● Release management and tool development - Zendesk
● DBMS tuning and performance - Slideshare and Houzz
● Search solutions - Zenlok and Discover Books
*

8. 24x7 operational support needed -
Chegg.com
● 48 MySQL Nodes in 10 separate clusters
● Tier 1 support for web, search, application, middleware and RDBMS
layers
● Tier 1 and 2 support for MySQL, Cassandra and MongoDB
● 24x7 operational support with 30 minutes SLA
● Amazon EC2 tooling and architectural management
● Manage backup and recovery, database release, incident and
configuration management processes.
● Onsite support during Amazon EC2 region migrations
● Advanced DBMS tuning and management
● Upgrades of tooling and diagnostics for alerting and trending
*

9. MySQL revamping and support -
Echosign
● Monolithic MySQL with MMM to start.
● New version upgrades.
● New hardware and capacity improvements.
● Functional partitioning and data management.
● Migration to Tungsten for HA and DR.
● BI/ETL to reduce DB load, support business needs and reduce load
times for key processes.
● Aggregation and denormalization for performance improvements.
*

10. Operations team needed - SoHalo
● Startup with no operations team
● Provided DBA and techops resources
● Architected environment
● Using Amazon MySQL RDS as datastore
● Fulfilled the entire operations function
● Running both their US and EU sites
*

11. Verifying systems will scale under load -
Contest Factory
● Configuration review
● Hardware and O/S review
● Metrics and logs review
● AWS architecture review
● Implementation of recommended configuration changes
*

12. Tools development - SlideShare and
Zendesk
● Replication aware rolling DDL implementations
● Shard migration and monitoring utilities
● Chef recipes for configuration management of configurations
● MHA and HAProxy for availability
● Cacti for MySQL Trending
*

13. Lousy application performance -
Zeitbyte
● Customer blamed the DBMS (MongoDB)
● Configuration review
● Application profiling (Django)
● Identify problem as code, not DBMS
● Fix code
● Integrate with ongoing monthly support post - tuning
○ New backups
○ New HA
*

14. Proactive services - SendGrid
● System health checks
● Backup and recovery
● Upgrades to monitoring and trending.
● Chef recipes and support.
● Query reviews, indexing, benchmarking
● Upgrades
● Support of Sharding and NoSQL evaluation
○ Sharding for Short URLs.
○ HBase for Counters
*

15. Release management and tool
development - Zendesk
● Own the weekly migration process
● Identify and manage risk, push changes throughout sharded
infrastructure
● Publish views to production
● Built shard rebalancing utilities
● Manage regular shard rebalancing and capacity planning
● Slow data deleter tool
*

16. DBMS tuning - Slideshare and Houzz
● Improved slideshare site load times by factor of 3
● Benchmarked InnoDB/XtraDB conversions for both clients
● Tuned InnoDB and XtraDB to manage background IO, concurrency
issues and optimal memory management
● Eliminated locking and replication lag for Houzz.com
*

17. Search solutions - Zenlok and
Discover Books
● Installed, configured and maintain Solr and Sphinx clusters
● Gathered search requirements
● Created and optimized index and delta queries
● Created custom ranking formulas
● Improved morphology and relevance
● Reviewed and improve indexing speed by over 10x
● Reduced query response over 4x by splitting and distributing indexes
across the cluster
● Configure HA and failover
*

18. Engagement process
● Palomino is a boutique offering bespoke solutions for our customers.
How we interact is thus hard to predict.
● We make ourselves available via email, IM, phone and chat.
● Your primary DBA can spend their working hours available in your chat
rooms.
● Daily standups for busy times, and weekly project mgmt meetings are
all part of the service.
● How else can we help you?
*

19. Who we are working with
*

20. Zendesk & Palomino
“Palomino got up to speed very quickly. Palomino works
successfully on complex [data migration] issues that
have stumped other DBAs.”
*

21. SoHalo & Palomino
“Palomino has been helping us build our Development and
Production systems from the beginning. Early on, they built the dev
servers, set up our release management system, migrated us to
MySQL and Couchbase, and architected our Amazon Web
Services cluster. Now, day to day, they help us run both our US and
EU sites, and continue to support us in projects, such as a recent
German/EU Data Privacy compliance effort. We couldn't have
done it all without Palomino.”
*

22. Slideshare & Palomino
“Palomino was instrumental in helping SlideShare
develop an architecture that supported arbitrary schema
updates without downtime.”
*

23. Rent the Runway
& Palomino
“Palomino has shown persistence, talent and a great
breadth of experience as they continue to bring us to
new levels of stability and performance.”
*

24. Technorati & Palomino
“Palomino helped us to rethink our database
architecture and to automate processes that were
formally manual, if done it all. Their deep experience in
designing large database systems for scale and stability
and their ability to work side-by-side with operations
staff and technology management, made them a huge
asset.”
*

25. Chegg & Palomino
“Palomino are hands-down the best DBAs I have
worked with. Their knowledge, experience, and tenacity
make a huge difference. I would highly recommend
them.”
*

26. WorldReader & Palomino
Palomino’s partnership with WorldReader has allowed
us to focus on our core efforts. Finding talented, reliable
partners is crucial for a non-profit, and we are quite
grateful for Palomino.
*

27. Palomino Leadership
Laine Campbell Jay Edwards Charlie Killian
Chief Executive Chief Technical VP, Professional
Officer Officer Services
*

28. Palomino Leadership
Kevin Bowman Kaichi Sung Tim Ellis
VP of Operations Account and Project Principal Big Data
Management Lead Architect
*

29. Laine is the Owner/CEO and
Principal Consultant at Palomino. She
is a proven database expert with a
history of scaling high-performance
and high-availability web
applications.
Laine Campbell Laine leverages extensive
Owner/CEO operational experience and best
practices with extensive technical
knowledge, troubleshooting skills and
a passion for results to ease you
through the life-cycles of your
company.
*

30. Charlie has created the back-bone
technology for multiple million dollar
companies. An expert in web
applications, Charlie has built web
application frameworks, setup and
integrated numerous disparate
systems, built long term client
relationships, and contributed to open-
Charlie Killian source projects.
VP of Professional Services
Key to Charlie's success has been a
love of cyberspace, hardware,
software, an unwavering patience and
an unstoppable urge to figure out how
things work.
*

31. Jay was the first dedicated DBA at
Twitter. He later served as the lead
database engineer at Obama for
America during the 2012 election. In
his role as CTO for Palomino, he
Jay Edwards focuses on technology strategy,
Chief Technology Officer
evangelism and mentoring/training
staff and customers.
*

32. Kevin began his career at Garmin
offering desktop and network/server
support and was soon serving as IT
Manager, supervising areas including
service desk, Internet application
development, systems administration,
network engineering, DBA,
architecture and devops. He also
worked on delivering internet-enabled
Kevin Bowman
VP of Operations
products. He contributed significantly
on application and system architecture
and devops for the backend systems
that provide data services to products
as well as for Garmin Connect.
*

33. Kaichi occupies the dual role of
Account Manager and Scrum Master
at Palomino. During her time in
Biotech, she focused on business
analysis and project mgmt, supporting
process and product improvements.
Kaichi Sung She rolled out Agile Scrum and is
Account Manager and certified as both a Scrum Master and
Scrum Master
Product Owner. A multi tasker and
planner, she loves coordinating
projects and helping teams deliver on
customer requirements.
*

34. Tim has been working with database
clusters that serve billions of database
operations daily since the year 2000 and
has been involved in the NoSQL
community from the earliest stages.
He has a broad background in
relational database systems. He was a
Tim Ellis core operations component to the Digg
Principal Big Data Architect NoSQL rollout, and an active,
participating operations member of the
NoSQL community at StumbleUpon,
Mozilla, and Riot Games.
*

35. Contact Info
Laine Campbell, laine@Palomino.com
Charlie Killian, charlie@Palomino.com
Jay Edwards, jedwards@palominodb.com
Kevin Bowman, kbowman@palominodb.com
Kaichi Sung, kaichi@Palomino.com
Tim Ellis, time@Palomino.com
www.Palominodb.com
@Palominodb on Twitter
*