NoSQL databases have grown in popularity in recent years due to the flexibility of data modeling and scaling up capabilities. NoSQL databases also have been used in the big data landscape. The demo rich session will elaborate the difference between SQL and NoSQL. And data moving capabilities from NoSQL database MongoDB to Azure Data Lake by using Azure data factory.
2024: Domino Containers - The Next Step. News from the Domino Container commu...
Sql vs no sql and azure data factory glasgow data UG
1. SQL vs NoSQL and moving data from
MongoDB to Azure data lake by
using Azure Data Factory
Diponkar Paul
2. Father and
Husband
Blogger &
Speaker
Profession:
Data Engineer
Working with
BI, data
warehouse 12
years
Diverse
background:
South Asia,
Nordic
region, North
America
Community:
Lead Toronto
Data
Professionals
Community
Twitter: @Paulswengrr
Blog: www.allaboutdata.ca
3. What we
cover
Refresh our memory with traditional SQL
Know about NoSQL (MongoDB)
Demo: No SQL
Comparison
Azure data factory: Copy data from MongoDB
Demo: MongoDB with ADF
4. SQL Syntax
SELECT Id, Product, Price
From Product
Where ProductCategory=’Bikes’
Join, Insert, Update, Delete
5. Well defined
Schema
CREATE TABLE [Production].[Product](
[ProductID] [int] IDENTITY(1,1) NOT NULL,
[Name] [nvarchar](100) NOT NULL,
[ProductNumber] [nvarchar](25) NOT NULL,
[MakeFlag] [dbo].[Flag] NOT NULL,
[FinishedGoodsFlag] [dbo].[Flag] NOT NULL,
[Color] [nvarchar](15) NULL,
[SafetyStockLevel] [smallint] NOT NULL,
[StandardCost] [money] NOT NULL,
[ListPrice] [money] NOT NULL,
[Size] [nvarchar](5) NULL)
6. Relationship/Normalization
Customer Bridge table (Order)
Product
Id Name Price Description
1 “Mountain Bike “ 2500 “Bike for mountain trek”
2 “City Bike” 1000 “Best fit to roam around city”
Id Customer_ID Product_ID
1 2 1
2 2 2
3 1 1
Id Name Email
1 Morten
Sorenson
m.s@outlook.com
2 Andersen Lu al@yahoo.com
3 Derek Paul dp@outlook.com
16. SQL vs NoSQL
SQL NoSQL
Data uses Schema Schema-less (Schema Agnostic)
Maintain Relationship No relations– though you can design relationship
Data distributed in multiple tables Data in one table (embedded)
18. Azure Data Lake
Azure Data Lake is a scalable data storage
and analytics service
-Fully HDFS compliance file system
-Azure AD integrated
-Microsoft’s PAAS service big data solution
20. Pre-requisite
• azure account
• Azure data factory resource
• Linked services (Source and
target connection)
• Integration run time
Integration
Runtime
Linked Service