Watch full webinar here: https://buff.ly/2MwDyhq The use of Data Virtualization as a global delivery layer means that Denodo is a critical component of the data architecture. It cannot fail, needs to be fault tolerant and perform as designed. In this context, enterprise level-monitoring is key to make sure the virtual layer is in good health and proactively detect potential issues. Fortunately, Denodo provides a full suite of monitoring capabilities and integrates with leading monitoring tools like Splunk, Elastic and CloudWatch. Attend this session to learn: - How to configure the key global parameters of the Denodo server - How to integrate Denodo with enterprise monitoring solutions like Splunk and Cloudwatch - Key metrics to monitor

1. DATA VIRTUALIZATION PACKED LUNCH
WEBINAR SERIES
Sessions Covering Key Data Integration Challenges
Solved with Data Virtualization

2. Why Advanced Monitoring is Key for Healthy
Enterprise Deployments
Pablo Alvarez-Yanez
Director of Product Management, Denodo

3. Introduction
3

4. 4
Q: Do I Need Monitoring?
A: YES

5. 5
But Why?
• Get/Provide better support
• Detailed information to simplify resolution of issues and support cases
• Improved security
• Audit trails and cyber-attack alerts
• Keep users happy
• Proactive detection of potential issues: e.g. sources are down
• Be prepared for the future
• Base information for capacity planning, detect infrastructure bottlenecks
• Reduce infrastructure costs
• Scale down based on access patterns
Setting up, using and maintaining a proper monitoring infrastructure require some effort.
However, the benefits derived from a better understanding of the platform usage are
unquestionable:

6. 6
And How?
Monitoring can be see from many different angles and usually there is no "one tools fits all"
approach. For example, we can think of monitoring as:
• Real time monitoring
• Detailed insights on the current status of the server
• Diagnostics of issues
• Go “back-in-time” to understand why an
• Auditing access
• Examine log in attempts and queries based on different criteria
• Historical analysis
• Identification of trends and patterns that can only be seen with analysis over a longer period of
time
• Alerts
• Notifications when certain situations are problematic

7. Real Time Monitoring
7

8. 8
Query Execution
▪ Details of the query execution can be reviewed during execution time, both from the design
studio and the monitoring tool
▪ These execution traces can be saved and loaded in the design studio for detailed
examination.
▪ In addition, these query traces can be also generated and stored for all queries run on the
server.
▪ Although this is costly, it’s sometimes useful to debug some issue.
▪ Do not have this option always enabled in production!
▪ This capability can be enabled with the following command:
▪ SET 'com.denodo.vdb.interpreter.execution.saveTrace' = 'ALL_FILE'
▪ To log only erroneous queries, use the value ‘ERROR_FILE’ instead. Use ‘DISABLED’ to go back to
normal logging settings

9. 9
Denodo Monitoring Tool
Web-based tool that provides detailed information on the current status of
the Denodo server. In addition, with the right privileges, allows to act on
the system. For example, cancel queries and sessions.
Its main capabilities are:
▪ Distributed with the Denodo Platform
▪ Integrated with the system authentication, easy to use
▪ Access to very granular information about the current status of the system
▪ Users and sessions (possibility to close them from the tool)
▪ Queries (possibility to see the execution plan, and cancel queries)
▪ Cache processes
▪ Status of data sources (status, ping, connection pools)
▪ Detailed thread information

10. Log File Analysis
10

11. 11
Server Error Logs
▪ The different components of the Denodo platform log errors
in the <denodo>/logs folders
▪ These files are the most valuable resource to understand
server errors, as they contain the error trace generated
during execution
▪ The logs are generated with log4j with a rolling policy:
▪ Current log file plus an archive of 7 files of 10MB each
▪ The rolling settings, output files and format of the output can
be changed in the logj2.xml configuration files located in
<denodo>/conf/<component>/logj2.xml

12. 12
Monitor Logs (CPU, queries, sessions, cache, etc.)
Denodo can automatically log a lot of detailed information using a
monitoring agent, called “Denodo Monitor”.
▪ The information is retrieved via a JMX subscription to the server.
▪ Since the agent can run externally (normally in the Solution Manager
server), it adds minimal overhead to the actual Denodo server.
▪ It can be launched graphically from the Solution Manager, or with
shell scripts.
▪ You can find more details in the documentation
▪ It offers multiple storage options:
▪ Log to CSV files (default)
▪ Log to database (Oracle, SQLServer, MySQL, Postgres)
▪ Log to S3

13. 13
Data Source Status Logs
▪ Denodo can also ping data sources in time intervals, to make
sure that the sources are up and running
▪ Since end users interact directly with Denodo, understanding
when errors actually come from non responsive data sources is
fundamental to understand many issues
▪ These settings can be configured in the configuration file of the
Denodo monitor in <denodo>/tools/monitor/denodo-
monitor/conf or <solution-manager>/conf/solution-manager-
monitor/denodo-monitor/
/conf/solution-manager/denodo-monitor

14. Diagnostics of Individual Issues
14

15. 15
Denodo Monitoring and Diagnostics Tool
In addition to real time monitoring, Denodo’s monitoring tool can load Denodo logs for a
particular timeframe to “go back in time” to help identify and diagnose an individual issue.
Some of it’s capabilities are:
▪ Possibility to "zoom in" into smaller time frames to narrow down the scope
▪ Access to very granular information about the status of the system during a particular time frame
▪ Integration with VDP errors stack trace
▪ Users and sessions
▪ Queries
▪ Cache processes
▪ Status of data sources (status, ping, connection pools)
▪ Detailed thread information
This tool load the information from the logs in the cache database. It is intended for specific
timeframes (e.g. 1 hour, 1 day). It is not intended for historic monitoring of larger timeframes

16. Auditing
16

17. 17
Monitor Logs – Auditing details
Logging attempts, data accessed by a particular user, or sessions
opened from a particular attempt, are typical examples of access
audits.
The information required to address those questions is available in
the monitor logs mentioned in the previous section
However, if these questions are frequently asked, it’s recommended
to log directly into a relational database
This way, any reporting tool using standard SQL can be used to
easily report on those metrics

18. Historical Monitoring
18

19. 19
Historical Monitoring - Needs
Monitoring a solution over a long period of time is important to draw
conclusions on average usage, peaks and capacity planning
However, the volume of data generated by logs can grow out of control
very easily. For example:
• We see around 5-6 concurrent queries running in the server. They take from
0.5 to 1 second
• This is: 5 x 60 secs x 60 min x 24 hours = ~450k queries per day
• ~160 million queries per year
As we can see, logging relatively small loads will produce a lot of data to
manage and report on
Fortunately, there are enterprise log management products specialized on
this type of scenarios

20. 20
Enterprise Monitoring Tools
Enterprise monitoring tools normally provide the following
capabilities
1. A log gathering agent, that runs in each one of the servers
2. A processing pipeline to parse and format the data in the logs
3. A time-series storage and processing engine, capable of storing
large volumes and efficiently run queries based on time intervals
4. A dashboarding / reporting tool focused on monitoring
There are many tools in the market that cover this segment, for
example Splunk, the ELK stack (Elastic, Logstash, Kibana),
CloudWatch in AWS, etc. Some other products specialize on the
dashboarding part only, for example Grafana

21. 21
Enterprise Monitoring Tools and Denodo
Integrating these tools with Denodo is quite straightforward
1. If necessary, install the agent
2. Depending on the log you are monitoring, specify the corresponding pattern
1. Many are just CSV files, others will need a regular expression
3. Ingest and store the logs
4. Build a dashboard to show the metrics you are interested on
In the Denodo community site you can already find detailed information for Splunk
and CloudWatch

22. Alerts
22

23. 23
Alerts
Another interesting feature of enterprise monitoring tools is the
generation of alerts
Based on the incoming logs, and a set of pre-configured parameters,
these tools can generate an alert that can trigger some custom
action, normally sending an email to an administrator
For example, alerts can be useful for:
• System is not responding
• Data sources are down
• Number of concurrent requests is higher than a safety limit
In the articles for Splunk and CloudWatch in the Denodo Community
you can also find details on how to configure alerts

24. Summary
24

25. 25
Summary
▪ Good monitoring is the foundation of a healthy
deployment and sets the path for trust and growth
▪ Denodo offers multiple logs and tools that provides a
detailed view of the status of the server
▪ Enterprise monitoring tools can integrate easily to
provide additional capabilities like historical
monitoring and alerts
▪ Together, this ecosystem simplifies resolution of
issues, prevents them from happening, and paves the
path for new projects

26. Q&A

27. 27
Next Steps
Access Denodo Platform in the Cloud!
Take a Test Drive today!
www.denodo.com/TestDrive
GET STARTED TODAY

28. Thank you!
© Copyright Denodo Technologies. All rights reserved
Unless otherwise specified, no part of this PDF file may be reproduced or utilized in any for or by any means, electronic or mechanical, including photocopying and
microfilm, without prior the written authorization from Denodo Technologies.