6. Data Collection Overview
All Data Collections defined on each
Agent installation locally
Defaults are within the
“./agent/share/mysql-proxy” directory
SQL and Lua based collections
supported
SQL statements defined in XML format
QUAN based collections are all Lua via
the Proxy
7. Internal Data Collection Format
Defined in XML files in the Advisor bundles
Here's our definition of the
innodb_buffer_pool_size variable:
<itemList>
<attribName>innodb_buffer_pool_size</attribName>
<nameSpace>mysql</nameSpace>
<attribType>INTEGER</attribType>
<className>variables</className>
<categoryName>Memory</categoryName>
<subCategoryName>Buffers</subCategoryName>
<isCounter>false</isCounter>
</itemList>
8. Internal Data Collection Format
Types: nameSpace::className::attribName
i.e: mysql::variables::innodb_buffer_pool_size or
os::cpu::cpu_user
attribType – INTEGER | VARCHAR | DECIMAL
isCounter – Whether the variable is a counter
(true) or point in time value (false)
Counter based variables are always evaluated
using the delta between “now” and “now – 1”
collections
Multiple instances of attributes supported
internally (for things such as CPUs, disks, etc.)
Single instance collections use a “local” instance
9. SQL Based Data Collections
Defaults in:
./agent/share/mysql-proxy/items/items-mysql-monitor.xml
<class>
<namespace>mysql</namespace>
<classname>classTitle</classname>
<query><![CDATA[SELECT ...]]></query>
</class>
SELECT must return one row (others ignored
with warnings in the log file)
Can return multiple columns in 2.x
● Assign <classname> in column order
Note: <isCounter> is not available until 2.1 here
10. SQL Based Data Collections
Collections can be added to the default, or a new file
New files should be added to the agent-item-files
variable in the mysql-monitor-agent.ini file
Let's try an example with the new
PERFORMANCE_SCHEMA!
<class>
<namespace>mysql</namespace>
<classname>wait_sum_open_mutex</classname>
<query><![CDATA[
SELECT sum_timer_wait
FROM PERFORMANCE_SCHEMA.EVENTS_WAITS_SUMMARY_BY_EVENT_NAME
WHERE EVENT_NAME = 'wait/synch/mutex/sql/LOCK_open'
]]></query>
</class>
11. Lua Based Data Collections
Lua is a powerful embedded language – many
possibilities exist to collect data (running
external commands, gathering external data by
reading files, etc.)
Lua collections also go to:
./agent/share/mysql-proxy/items/
They should also be added to the agent-items-
files variable in the mysql-monitor-agent.ini file
We build up the items, attributes and instances
as associative arrays within Lua
12. Basic Lua Collections
if not items then
items = { }
end
if not items.oldag then
items[quot;oldagquot;] = { }
end
items[quot;oldagquot;][quot;childrenquot;] = {
instances = { { name = quot;ethanquot; } },
attributes = {
[quot;couch-divesquot;] = {
returns = quot;intquot;
}
},
collector = function() return { [quot;couch-divesquot;] = math.random(100) + 120 } end
}
13. Matching Lua Arrays to Types
The namespace is the top level of the items array
● “oldag”
The classname / type name comes next
● “children”
We then define the instances of the
namespace::classname by name
● “ethan”
And the attributes and their return types
● “couchdives” returning an int
Finally we define a collector function that pushes the
values we want to collect by attribute
14. Data Collections in the Dashboard
Data collection metadata is stored in the
“inventory_*” tables
inventory_namespaces (namespace info)
inventory_types (maps to “classname”
information)
inventory_attributes (attributes for
namespace::type)
inventory_instances (the instance names of
each attribute)
inventory_instance_attributes (pivot like table for
attributes to instances mappings)
15. Data Collections in the Dashboard
A useful SQL statement to get DC information:
SELECT DISTINCT
namespace, type_name, attribute_name, instance_name
FROM inventory_namespaces iin,
inventory_types iit,
inventory_instance_attributes iia,
inventory_instances ii,
inventory_attributes ia
WHERE iia.instance_id = ii.instance_id
AND ii.type_id = iit.type_id
AND iin.namespace_id = iit.namespace_id
AND iia.attribute_id = ia.attribute_id;
*************************** 1. row ***************************
namespace: os
type_name: fs
attribute_name: fs_avail
instance_name: mac:{0025003d17370000}./
16. Data Collections in the Dashboard
Actual data is stored within the “dc_ng_*” tables
The collected data is stored in it's raw form, whether it
is a counter based variable or not
Currently data is only purged, the infrastructure is
ready for aggregation and aging for historical data in
2.x (we want to test more!)
dc_ng_long_now (integer based data)
dc_ng_double_now (doubles)
dc_ng_string_now (strings)
dc_ng_(long|double)_age(0|1|2) (the rollup tables)
String data obviously can not be aggregated
17. Data Collections in the Dashboard
A useful SQL statement to get the collected data:
SELECT namespace, type_name, attribute_name, instance_name,
value, from_unixtime(end_time/1000)
FROM inventory_namespaces iin,
inventory_types iit,
dc_ng_long_now dcn,
inventory_instance_attributes iia,
inventory_instances ii,
inventory_attributes ia
WHERE dcn.instance_attribute_id = iia.instance_attribute_id
AND iia.instance_id = ii.instance_id
AND ii.type_id = iit.type_id
AND iin.namespace_id = iit.namespace_id
AND iia.attribute_id = ia.attribute_id
AND namespace = '...'
AND type_name = '...'
AND attribute_name = '...';
19. Creating Custom Graphs
Graphs also use XML for their structure
Graphs can deal with both counter and non-counter
variables
2.0 now allows you to import your own graphs:
20. Graph XML Template
<com_mysql_merlin_server_graph_Design>
<version>...</version> /* Internal graph version numbering */
<uuid>...</uuid> /* Internal graph unique identifier */
<tag>...</tag> /* Tags for the graph (will be exposed soon) */
<name>...</name> /* Visible graph name */
<rangeLabel>...</rangeLabel> /* Visible Y axis range label */
<series>
<label>...</label> /* Visible Series name / label */
<expression>...</expression> /* The expression used for the series */
</series>
<variables>
<name>...</name> /* Arbitrary name for variable in expression */
<dcItem>
<nameSpace>...</nameSpace>
<className>...</className>
<attribName>...</attribName>
</dcItem>
<instance>...</instance> /* The instance of the dc item to be used */
</variables>
</com_mysql_merlin_server_graph_Design>
21. Building a Graph - Metadata
So let's build one up! Who wants to monitor disks?
Start with the basic info
<com_mysql_merlin_server_graph_Design>
<version>1.0</version>
<uuid>a57c2bba-ea9b-102b-b396-94aca32bee28</uuid>
<name>filesystem usage - /</name>
<rangeLabel>MB</rangeLabel>
<frequency>00:05:00</frequency>
Note <frequency>, this is not required, and defaults to
1 minute
We use 5 minutes here to share with current collection
22. Building a Graph - Variables
Next, skip to the end, and define the variables
<variables>
<name>used_fs</name>
<dcItem>
<nameSpace>os</nameSpace>
<className>fs</className>
<attribName>fs_used</attribName>
</dcItem>
<instance>/</instance>
</variables>
<variables>
<name>total_fs</name>
<dcItem>
<nameSpace>os</nameSpace>
<className>fs</className>
<attribName>fs_total</attribName>
</dcItem>
<instance>/</instance>
</variables>
</com_mysql_merlin_server_graph_Design>
23. Building a Graph - Series
Now finally define each series
<series>
<label>used</label>
<expression>used_fs/1024/1024</expression>
</series>
<series>
<label>total size</label>
<expression>total_fs/1024/1024</expression>
</series>
27. Graphs – Current Caveats
Currently only line graphs are supported
Graphs are shared across all servers – so
they will show up even if the
instances/collections are not available on a
specific server
Graphs sort alphabetically – if you want
custom graphs to sort together, keep that in
mind
There is no “Delete” for graphs
29. Modifying Current Rules
All of the current rules are conservative – they
should certainly be modified to your environment
You can edit current rues on the “Manage Rules”
page within the Advisors tab
Editing only allows you to modify the
THRESHOLD and frequency values for the rule
Copying allows you to alter the expression, and
advice or problem descriptions etc., as well as
add new variables
31. Creating New Rules
Also done on the Manage Rules page
Any data collection can be used in a rule, with no limit
on the number of data collections that can be used
We still only support the default 3 THRESHOLD values
– Info / Warning / Critical
Wrapping the variable names in %percents% allows us
to do variable value substitution within the Advice and
Recommended Action sections
We can also format numerical output nicely with
'{'formatNumber:%variable_name%'}'
You can make parts bold with two underscores on
either side of a section of the advice etc. too