IDERA Slides: Managing Complex Data Environments

© 2019 IDERA, Inc. All rights reserved.
MANAGING COMPLEX DATA ENVIRONMENTS
Lisa Waugh, Senior Product Manager, Aqua Data Studio

© 2016 IDERA, Inc. All rights reserved. Proprietary and confidential.© 2019 IDERA, Inc. All rights reserved. 2
COMPANIES ARE STRIVING TO STORE AND MANAGE
MORE AND MORE DATA
▪ It is rare to have a company that uses a single platform anymore, so
companies and IT teams must adapt
▪ The days of different disciplines for different roles are gone
▪ Multiple disciplines must work together
▪ The lack of the appropriate skill sets can slow the process down

CLOUD EXAMPLE OF INFRASTRUCTURE

HYBRID INFRASTRUCTURE EXAMPLE

CONSIDERATIONS FOR THE DATABASE
▪ Scalability
• SQL
• Users have to scale a relational database on powerful servers.
• The data has to be distributed, handling tables across different servers.
• NoSQL
• Automatically spreads your data onto multiple servers.
• Servers can be added or removed from the data layer.
▪ Complexity
• SQL
• The design of the data needs to fit into tables.
• The database structure can be complex and again, difficult to handle in large
volumes.
• NoSQL
• Automatically spreads your data onto multiple servers without requiring a defined
structure.
• Can cache data in system memory, therefore making it faster to retrieve. This is in
contrast to SQL databases where this has to be done using a separate
infrastructure.

RELATIONAL VS NOSQL PLATFORMS
▪ Relational databases like MySQL, PostgreSQL, and SQLite represent
and store data in tables and rows. They're based on a branch of algebraic
set theory known as relational algebra.
▪ Meanwhile, non-relational (NoSQL) databases like MongoDB represent
data in collections of JSON documents.

DISTRIBUTIONS - THAT INTEGRATE WITH HADOOP
▪ Apache Hive is a data warehouse software project build on top of Apache
Hadoop for providing data query and analysis.
▪ Hive gives a SQL like interface to query data stored in a various databases and
file systems that integrate with Hadoop with different distributions to add to the
complexity.
▪ Aqua Data Studio includes Hive, Impala, Spark connectors within these
distributions:
• Amazon EMR
• Apache
• Cloudera
• Hortonworks
• MapR

COMPLEXITY OF NOSQL AND DISTRIBUTIONS
Clients Hive Services Hive Storage and Compute
Impala JDBC
Spark JDBC
Hive JDBC
Impala Server
Spark Server
Hive Server
Meta Store
File System
Job Client
Hive
Meta Data
Store
Base
Hadoop
Cluster
Apache Hive Architecture
Driver

THERE ARE 8 BUSINESS PROCESSES
1. Determine how a hybrid strategy fits the requirements of the business.
2. Consider the implications of data governance
3. Enhance business continuity.
4. Don’t outsource security.
5. Ensure the flexibility of data movement.
6. Embrace the range of tools.
7. Expand the scope of service-level agreements.
8. Prepare for new skill sets.
Download the PDF: 8 Rules for Managing Hybrid Cloud Environments by Database Trends

IN SOME OF THESE COMPLEX ENVIRONMENTS
THERE ARE NO MORE WELL-DEFINED ROLES
Interesting article on role of DBAs changing but NOT going away:
http://www.dbta.com/Editorial/Trends-and-Applications/How-DBAs-Can-Survive-and-Thrive-in-a-World-of-Agile-Development-129894.aspx

AQUA DATA STUDIO OVERVIEW
▪ Aqua Data Studio is a universal database IDE and visual analytics tool for
IT Professionals working with data and databases
▪ Aqua Data Studio allows you to develop and manage your databases, and
access, import, export, and visually analyze data
▪ Aqua Data Studio supports over 30 database platforms, and works on all
major operating systems (Windows, macOS, Linux)
▪ Aqua Data Studio supports Relational, NoSQL, and Cloud databases,
making your data easily and quickly accessible

30+ DATABASE PLATFORMS SUPPORTED
▪ Amazon Redshift
▪ Apache Cassandra
▪ Apache Derby
▪ Db2 iSeries
▪ Db2 LUW
▪ Db2 z/OS
▪ Excel
▪ Generic JDBC
▪ Generic ODBC
▪ Google BigQuery
▪ Greenplum
▪ Hive, Impala, and Spark connectors
within these distributions
• Amazon EMR, Cloudera,
HortonWorks, Impala, MapR
▪ Informix
▪ InterBase
▪ MariaDB
▪ Microsoft Azure SQL Database
▪ Microsoft SQL Server
▪ MongoDB
▪ MySQL
• Amazon Aurora MySQL
• Google SQL Cloud MySQL
▪ Netezza
▪ Oracle
▪ ParAccel
▪ PostgreSQL
• Amazon Aurora PostgreSQL
• Google SQL Cloud PostgreSQL
▪ SAP HANA
▪ Snowflake
▪ SQLite
▪ Sybase Anywhere
▪ Sybase ASE
▪ Sybase IQ
▪ Teradata Aster
▪ Teradata
▪ Vertica
▪ VoltDB

QUERIES, QUERIES, AND MORE QUERIES
▪ Different Kinds of Queries: There are a few differences between the two
dialects, including syntax, functions, data types
• Legacy i.e. Google BigQuery
- COUNT(DISTINCT x) returns an approximate count
• Standard
- It returns an exact count
▪ Single Query
▪ Multi-Database Instance Query
▪ Scripts – Multiple queries to be saved, scheduled, etc.

RELATIONAL VS NOSQL PLATFORMS
SQL Term/Concepts
database
table
row
column
index
table joins
primary key
NoSQL/MongoDB Terms/Concepts
database
collection
document of BSON (binary JSON) document
field
index
$lookup, embedded documents
primary key
Note: Specify a unique column or column combination as a primary
Key.
The primary key is automatically set to the _id field.
SQL to Aggregation Mapping Chart

LANGUAGES OF SQL EXAMPLES
▪ PL/SQL is a proprietary procedural language used by Oracle.
▪ TSQL is a proprietary procedural language used by Microsoft in SQL Server.
▪ PL/PGSQL is a procedural language used by PostgreSQL.
Again, syntax is different; see example PL/SQL,TSQL and PL/PGSQL typical syntax:
Create table in PL/SQL
▪ CREATE TABLE table_name ( column1 datatype [ NULL | NOT NULL ], column2 datatype [
NULL | NOT NULL ], ... column_n datatype [ NULL | NOT NULL ] );
Create table in TSQL
▪ CREATE TABLE [ database_name . [ schema_name ] . | schema_name . ] table_name ( {
<column_definition> } [ ,...n ] ) [ ; ]
Create table in PL/PGSQL
▪ CREATE TABLE table_name (
column_name TYPE column_constraint,
table_constraint table_constraint
) INHERITS existing_table_name;

AQUA DATA STUDIO PRIMARY FEATURES
▪ Register Servers/Databases
▪ Database Navigator/Explorer
▪ Query Analyzer (SQL/JSON Editor)
▪ Query Builder
▪ Table Data Editor
▪ Visual Analytics
▪ Import/Export Data and DDL
▪ Schema/Data/File Compare
▪ ER Modeler
▪ Database Administration

GET TO ALL THESE SERVERS WITH ONE INTERFACE
17

AQUA DATA STUDIO SERVER REGISTRATION/PROPERTIES

AQUA DATA STUDIO SERVER REGISTRATION/PROPERTIES CONT.

MY WORKSPACE - SERVERS, WORK AREA, FILES

OPTIONS ALLOWS FOR DIFFERENT SYNTAX CHANGES

OPTIONS ALLOW FOR DIFFERENT SYNTAX CHANGES

HOW DO WE QUERY THE DATA?

AUTOMATE COMMANDS IN THE QUERY ANALYZER

QUERY THE DATA WITH AQUA DATA STUDIO

RESULTS SET CLIENT STATS

QUERY MONGODB WITH MONGOJS

MULTI-SERVER QUERY – MEANING SAME VENDOR QUERY

SCRIPTS AND MORE SCRIPTS

ENTITY RELATIONSHIP DIAGRAM

USING THE PROJECTS TAB IN AQUA DATA STUDIO
▪ Projects are comprised of AquaScripts, Server Connections, and User Files.
▪ They provide a self-contained programming unit that allows users to create a set of
AquaScripts that work on a specific set of database servers that you identify with the
ability to store and access configuration files.
▪ Once a project is completed, it may be exported and shared with other users.
See blog: Using Projects for Multi-Environments Scripting

WHAT APPEARS IN THE QUERY ANALYZER - TEMPLATE
▪ //*************************************************
******************
// Create and Email Excel File
//
// Creates an Excel file from a user-specified
SQL query and
// emails the Excel file to the specified
recipients.
//
// For more information about Aqua Open
API visit link:
http://docs.aquafold.com/ads/19.5/openapi/
//*************************************************
******************
//*************************************************
******************
// Input parameters.
//*************************************************
******************
var server ="Lisa SQL Server 2016";
var database = "Healthcare";
var schema = "dbo";
var sqlQuery = "select * from dbo.doctors";
var filePathtoStore = "C:/Users/Lisa
Waugh/build/create_and_email_excel_file.xl
s";
var toRecipient =
"Lisa_Waugh@company.net";
var from = "Lisa.Waugh@company.com";
var subject = "Test Excel email";
var message = "This is a test of the create
email Excel file in project";
▪ try {
var conn =
aqua.project.getServerConnection(server
);
conn.connect();
conn.changeDatabase(database);
if (schema !== null || schema != "") {
conn.changeSchema(schema);
}
var resultset =
conn.executeQuery(sqlQuery);
var sheet =
aqua.data.excelSpreadsheet(filePathtoSt
ore);
sheet.clear();
sheet.insert(resultset.columnStructure);
while (resultset.next()) {
sheet.insert(resultset.getRow())
}
sheet.save();
sendEmail();
conn.disconnect();
aqua.console.println("E-mail sent!");
}
catch(err) {
aqua.console.println(err.lineNumber +
": " + err);
}
▪ //*******************************************
************************
// This function is used to send the
email.
//*******************************************
************************
function sendEmail() {
/**
Use the default SMTP information
configured in Aqua Data Studio.
In Aqua Data Studio, go to File--
>Options-->Email to configure email
settings.
**/
var smtpServer =
aqua.mail.getDefaultSmtpServer();
var msg = aqua.mail.newMessage();
msg.addToRecipient(toRecipient);
msg.setFrom(from);
msg.setSubject(subject);
msg.setText(message);
msg.addFileAttachment(filePathtoStor
e);
smtpServer.sendMail(msg);
}

DIRECTORY STRUCTURE/PROJECT CREATED

SOURCE CONTROL: SVN, CVS, GIT, AND PERFORCE

TOOLS TO HELP MANAGE ALL THESE ENVIRONMENTS

SYNCHRONIZE SCHEMA

THE BUSINESS ANALYTICS CHALLENGES SOLUTION
Support the citizen analyst’s data storytelling and intelligent
decisioning with a tool that addresses the key challenges:
Aqua Data Studio
▪ Broaden access to a wide variety of data sources
▪ Simplify query and analysis tasks
▪ Provide a robust palette of graphs and charts for effective visualizations
▪ Provide an environment for development, collaboration, and integration

BUSINESS INTELLIGENCE VS ANALYTICS

BUSINESS ANALYTICS: HOW DO YOU TELL A STORY?
▪ Provide the datasheet as a package with the data and the worksheet, therefore, linking parameters
across different data sources
▪ Export as an image

VISUAL ANALYTICS

SUMMARY
▪ Today’s database environments are often more complicated
• Not just a single type of data store
• Relational, NoSQL, cloud
▪ Database professionals must perform multiple tasks across them all
• Managing different platforms
• Running queries
• Creating data analytics
▪ Aqua Data Studio makes it easier
• 14-day free trial
• Learn more at aquafold.com

THANKS!
Any questions?

IDERA Slides: Managing Complex Data Environments

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie IDERA Slides: Managing Complex Data Environments

Ähnlich wie IDERA Slides: Managing Complex Data Environments (20)

Mehr von DATAVERSITY

Mehr von DATAVERSITY (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

IDERA Slides: Managing Complex Data Environments