This presentation was made at LSPE event in Bangalore (India) held at Walmart labs on 10-03-2017.

1. Ansible is Our Wishbone
Python, Yaml, Jinja2
Presenter
Manosh Malai
Senior Devops & NoSQL Consultant
www.mydbops.com info@mydbops.com
1

2. Manosh Malai
Senior Devops and NoSQL Consultant @ Mydbops
mmalai@mydbops.com
@ManoshMalai
ME
2

3. About Mydbops
● Founded in 2015, HQ in Bangalore India with 150+ customer base across the globe.
● Mydbops is on Database Consulting with core specialization on MySQL and MongoDB Administration and
Support.
● We have expert team with 20+ certified DBA’s providing full time support and currently managing 300+
servers on cloud.
● Mydbops was created with a motto of developing a Devops model for Database administration offering
24*7 expert remote DBA support.
● We help organisations to architect and scale systems in MySQL/Mongo by implementing the advanced
technologies in industry which are completely open source.
● We are leading solution provider in the market for all sort of cloud based deployments and management.
3

4. Our Support Ecosystem
Percona Server
MySQL
MariaDB MariaDB Galera
Cluster
Galera Cluster for
MySQL
MongoDBProxySQL
Amazon RDS for
MySQL/Aurora
Percona XtraDB
Cluster
MariaDB MaxScale
MySQL Router
4

5. In This Presentation
How we spend our days as a DBA ?
What Next...
What Finally we got
How Ansible Helped us
5

6. Agenda
● How we spend our days as a DBA ?
● Findings our boring Tasks.
● Discovering our Wishbone.
● How we design our Wishbone ?
● What we got finally ?
● Where are we focusing now ?
6

7. Analysis of Boring Task
7

8. Our Days
How we really spend our days as a DBA
8

9. Finding Our Boring Task
9

10. Time = Money
10

11. 9.3% + 9.3% + 17.4% + 5.8% = 42%Repetitive Task
11

12. Details of 42% Task
● Install new version of DB server and its tools and any other tools that access the DB’s.
● Creating or Pushing optimized and standard config to server.
● Implements and enforce security for all of the DB server.
● Deploying Cluster, Load Balancer and Provisioning new node in a cluster(PXC, Maxscale and ProxySQL).
● Perform ongoing tuning of the database instances.
● Deploying Monitoring Infra and Monitoring Add-ons into a new or existing DB Infra(PMM,
NagiOS,MySQL-Utilities, Percona Toolkit, Sysstat and etc...).
12

13. 42% Repetitive Task and its Other Problem
Repeating same task on number of device is not only the problem, Other things too.
● Need to work with different Cloud platform(AWS, Google Cloud, Azure and etc).
● Need to work with different SAAS(Aurora, RDS).
● Need to work with different packaging system(apt, yum, npm, pip and etc).
● Each OS distribution have different configuration path for the same packages(sysstat, MySQL and etc).
● Susceptible to human error*
*80% of all downtime due to people/process error
13

14. Discovering our Wishbone
14

15. Ansible is a radically simple IT Automation Engine
Our Wishbone
15

16. BASIC FEATURE
SIMPLE
Human-readable YAML Language
No special coding skill required
Task Execute in Order
AGENTLESS
OpenSSH
Secure
Scalable
Efficient
POWERFUL
Deployment
Orchestration
Provisioning
16

17. VS
Asking Myself?
● Which method is most likely to end up in source control?
● Which method can be run multiple times safely with confidence?
● Which method can easily be run against multiple servers?
● Which method actually verifies (tests) your server for correctness?
● Which method can target certain servers easily (web, db, etc)?
● Which method supports easily templating your configuration files?
● Which method will grow to easily support your whole stack?
17

18. Ansible Architecture
18

19. Ansible Core Concept
● Inventories
● Ad-hoc Commands
● Playbooks
● Playbook >>Tasks(Handlers, Template, Variable)
● Playbook >>Roles
19

20. Inventories
● Ansible is used for managing multiple servers in the Infrastructure. The collection of hosts is known as
“Ansible Inventory”.
● Groups and Hosts are stored in inventory file
● An inventory file is expressed in a simple INI format
● Default location: /etc/ansible/hosts
● ansible or ansible-playbook -i <custom inventories path>
20

21. Inventories
21

22. Example Inventories
[all:vars]
ansible_user=vagrant
ansible_ssh_pass=vagrant
ansible_become=yes
[common]
v11 ansible_host=192.168.33.11
v12 ansible_host=192.168.33.12
[Common:vars]
percona_repo_url:
"https://www.percona.com/redir/downloads/percona-release/redhat/percona-release-0.1-4.noarch.rpm"
percona_deb_url: "https://repo.percona.com/apt/percona-release_0.1-4.{{ ansible_distribution_release
}}_all.deb"
All Groups: Variables
Group Hosts
Hosts
Groups Vars
22

23. Ad-hoc
● We don't have to create an elaborate set of tasks just to perform simple operations on your nodes.
● Ad-hoc commands allow you to execute simple tasks at the command line against one or all of your hosts.
Syntax:
ansible <host-pattern> options
23

24. Example Ad-hoc
ansible -i /etc/ansible/hosts common -m ping -f 10
ansible -i /etc/ansible/hosts common -m yum -a “name=vim state=present” -f 10
ansible all -m command -a "docker info" -u mydbops --become-user root --ask-pass --ask-become-pass
Module
Module Argument
User Becoming Sudo
Asking user password
Asking SUDO password
24

25. Playbook
● Playbooks we specify what and how our server will be configured and provisioned.
● Consist of variables, task, handlers, files, templates and roles.
● Are expressed in YAML format.
- name: Oracle Mysql Server Installation-5.7.
hosts: mysql
vars:
mysql_version: 5.7
mysql_version_repo: mysql{{ mysql_version|string |replace(".", "")}}-community
roles:
- common
- linux-tuning
- mysql
25

26. Task(Handlers, Template and Variable)
Task:
Tasks are nothing but modules called on hosts. That run on remote machines defined in inventory file.
- name: Installing Percona Server For RedHat Family.
yum:
name={{ item }}
state=present
with_items: "{{ mysql_percona_packages }}"
when: ansible_os_family == "RedHat" and 'percona' in group_names
- name: Installing Percona Server For Debian Family.
apt:
name={{ item }}
state=present"
with_items: "{{ mysql_percona_packages }}"
environment:
DEBIAN_FRONTEND: noninteractive
when: ansible_os_family == "Debian" and 'percona' in group_names
26

27. Handlers
Handlers:
● They will only run if a ‘notify’ directive is declared.
● It is useful in scenarios where a configuration file is changed and time it requires a service to be restarted.
● Handler will make sure that the service is restarted.
- name: restart mysql
service:
name={{ mysql_daemon }}
state=started
enabled={{ mysql_enabled_on_startup }}
register: mysql_service_configuration
- name: Copy my.cnf global MySQL configuration.
template:
src: mysql_conf.j2
dest: "{{ mysql_config_file }}"
owner: root
group: root
force: "{{ overwrite_global_mycnf }}"
mode: 0644
notify: restart mysql
27

28. Template
Template:
Templates is a file which contains all your configuration parameters, but the dynamic values are given as
variables.
[mysqld]
port = {{ mysql_port }}
bind-address = {{ mysql_bind_address }}
datadir = {{ mysql_datadir }}
socket = {{ mysql_socket }}
pid-file = {{ mysql_pid_file }}
{% if mysql_skip_name_resolve %}
skip-name-resolve
{% endif %}
{% if mysql_sql_mode %}
sql_mode = {{ mysql_sql_mode }}
{% endif %}
28

29. CONT...
# Memory settings.
key_buffer_size = {{ mysql_key_buffer_size }}
table_open_cache = {{ mysql_table_open_cache }}
sort_buffer_size = {{ mysql_sort_buffer_size }}
read_buffer_size = {{ mysql_read_buffer_size }}
thread_cache_size = {{ mysql_thread_cache_size }}
max_connections = {{ mysql_max_connections }}
tmp_table_size = {{ mysql_tmp_table_size }}
# InnoDB settings.
{% if mysql_supports_innodb_large_prefix %}
innodb_large_prefix = {{ mysql_innodb_large_prefix }}
innodb_file_format = {{ mysql_innodb_file_format }}
{% endif %}
innodb_file_per_table = {{ mysql_innodb_file_per_table }}
innodb_buffer_pool_size = {{ mysql_innodb_buffer_pool_size }}
innodb_log_file_size = {{ mysql_innodb_log_file_size }}
innodb_log_buffer_size = {{ mysql_innodb_log_buffer_size }}
29

30. Variable
Variable:
Variables as a way to deal with things that differ between one system and the next.
# InnoDB settings.
mysql_innodb_file_per_table: "1"
# Set .._buffer_pool_size up to 70% of RAM but beware of setting too high.
mysql_innodb_buffer_pool_size: "{{ (ansible_memtotal_mb * 0.7) | int }}M"
mysql_innodb_log_file_size: "1G"
mysql_innodb_log_buffer_size: "10M"
mysql_innodb_flush_log_at_trx_commit: "1"
mysql_innodb_lock_wait_timeout: "50"
30

31. Variable Precedence
In 2.x, we have made the order of precedence more specific (Low -> High Priority)
1. role defaults
2. inventory INI or script group vars
3. inventory group_vars/all
4. playbook group_vars/all
5. inventory group_vars/*
6. playbook group_vars/*
7. inventory INI or script host vars
8. inventory host_vars/*
9. playbook host_vars/*
10. host facts
11. play vars
12. play vars_prompt
13. play vars_files
14. role vars (defined in role/vars/main.yml)
15. block vars (only for tasks in block)
16. task vars (only for the task)
17. role (and include_role) params
18. include params
19. include_vars
20. set_facts / registered vars
21. extra vars (always win precedence)
31

32. Roles
● Ansible roles are consists of many playbooks, which is similar to modules in puppet and cook books in chef.
We term the same in ansible as roles.
● Roles are a way to group multiple tasks together into one container to do the automation in very effective
manner with clean directory structures.
● Roles are set of tasks and additional files for a certain role which allow you to break up the configurations.
● It can be easily reuse the codes by anyone if the role is suitable to someone.
● It can be easily modify and will reduce the syntax errors.
32

33. Role Directory Structure
33

34. Directory Structure
tasks - contains the main list of tasks to be executed by the role.
handlers - contains handlers, which may be used by this role or even anywhere outside this role.
defaults - default variables for the role.
vars - other variables for the role. Vars has the higher priority than defaults.
files - contains files required to transfer or deployed to the target machines via this role.
templates - contains templates which can be deployed via this role.
meta - defines some data / information about this role (author, dependency, versions, examples, etc,.)
34

35. How we designed Our Wishbone
35

36. Execution Process
36

37. Best Practice we follow for Our Wishbone
Inventory:
● Define meaningful inventory hostname, it doesn't want to resolvable.
● Inventory hostname help to view the result of play very meaningful.
● Dynamic Inventory using SSH config.
● Group all role related host into Role related groups(So we could run certain playbook against certain group).
37

38. Task & Playbook
Task & Playbook
● Use Human readable name for the Tasks and Plays.
● Keep plays and playbooks focused. don't create monolithic playbooks.
● Try to stick with ansible module try to avoid command or shell.
● last but not the least command module is safer than shell module(Because command module dont evaluate |
symbol, stringing two commands together and expanding shell variables or file globs and etc..) .
38

39. Templates
Templates
● Jinja2 is robust, try to use particularly.
● Don't overplay it, In some ways it comes down to documentation, a mixing of languages (YAML, Python,
Jinja2), and variables.
● Jinja2 can be a pain, but ultimately a very powerful tool.
● Should be:
○ Variable substitution
○ Conditional
○ Simple control structure and iteration
○ Advance filters(eg: math, ip address and etc)
39

40. What we got finally
● As we saw above, We are in heterogeneous environment. Using our Wishbone we are managing all
Obstacles in our way.
● Single Click Infra Provisioning and Managing.
● Help to keep all environment config in same way.
● Ensure no change unless things change.
● Improve Person/Team Performance and liability
● Stop everything do by hand.
● Get More sleep.
40

41. What’s Next...
● Consistency in Ansible Role development like PXC Cluster upgradation, Mysql 8.0 and etc.
● In DB Management if we config the parameter properly, we only need to focus on quieres tuning. So we are
currently developing Automate Slow query tuner.
● We are seriously planning to develop auto healing database management system.
41

42. 42

43. Happy Customers
43

44. Thank you !!!
Manosh Malai
mmalai@mydbops.com
44