Building Confidence in Big Data - IBM Smarter Business 2013
Overall System Architecture of Big Data of Wind Power Based on IoT_20161...
1. 1
Overall System Architecture of Big Data of Wind Power
Based on IoT
saberteeth@sina.cn 2016.10
Wind Power Big Data and IoT Forum
Huang Yuan
2. 2
Summary
Overall Architecture of
Big Data Platform
Based on IoT
Agenda
Industry Challenges
and
Technology Trends
Project “Big Data Plat
form of Wind Power”
2
1 3
4
3. Industry Challenges and Technology Trends
- Distributed Energy Assets Life Cycle Management & System
• Industry Challenges
• Distributed Energy Assets System is the
foundation to face the revolution of Wind
Power Industry Assets Life Cycle
Management.
• Powerful and accurate data acquisition &
analysis capability based on IoT & Big Data
is the core to establish Distributed Energy
Assets System.
• Data completeness, consistency, timeliness
and validity are the prime challenges in all
information systems, especially for the
Distributed System.
• Technology Trends
• IoT technology could solve the issues of full
sources and reliable data acquisition &
efficient and secure transmission.
• Big Data technology provides the ability to
process and analyze massive data (TB/PB
level, structured/non-structured data).
3
Goals
System
Technology
Assets
Life
Cycle
Management
Distributed Energy
Management System
Big Data
IoT
Data
4. Project “Big Data Platform of Wind Power”
- To Face the Challenges of Distributed Energy Assets Life Cycle Management
4
Project Goals
} Industrial standard – unified data model for wind
turbine I/O, status and fault to >20 Manufacturers
and >90 Types of Wind Turbine.
} Reference architecture and validation for Wind
Power big data platform based on IoT, big data
and cloud computing technologies.
} Big data analytics models for accurate prediction,
diagnosis, maintenance to achieve effective
decision, better efficiency and lower cost.
} Pilot in Longyuan to prove the business value of
the platform: better profitability and management
maturity.
Project Duration
} Two Years, 2016-2017.
Project Background
} The 1st state-funded big data platform for
Wind Power, invested by National Energy
Administration, China Guodian Group
and China Longyuan Power Group.
5. Project “Big Data Platform of Wind Power”
- China Longyuan Power Group, the world‘s Largest Wind Farm Operator
5
15GW+ Capacity
of Wind Power
11,000+ Wind Turbines
160+ Wind Plants in China,
Canada and South Africa
Accumulated Wind Power
Generation
126 Billion KWH
Accumulated 130 Million
Tones of CO2 Saving
Business and Citizenship
World-wide Partnership
The world’s largest wind farm operator, funded in 1993
5
6. Overall Architecture of Big Data Platform
Information Standardization • Data Acquisition • Transmission • Big Data Platform/Application • Security Architecture
Requirements
Design
Principles
• Standardization
• Safety &
Compliance
• Reliability
• Manageability
• Self-adaption
• …
Big Data Platform
# of sensor points：10M or more
# of data increase：400TB/year or more
# of data for analytics: 1PByte or more
# of prediction models deployed: 100+
Big Data Application
# Support for world-wide wind plant
management and users
# of Applications: 64 or more
failure prediction/operation optimization/asset
management/visualization analysis
Data Acquisition
# of manufacturers: 20 or more
# of Wind Turbine types: 90 or more
# of Wind Turbine: 10K+
# of sensors per Turbine: x00-1K+
# of data collection frequency: 1Hz or higher
Data Transmission
# High security, High availability
and Support for Hybrid cloud
6
7. Design
Principles
• Standardization
• Safety &
Compliance
• Reliability
• Manageability
• Self-adaption
• …
Overall Architecture of Big Data Platform
- Information Standardization
Information Standardization
Data Acquisition
Data Transmission
Big Data Platform
Big Data Application
7
Information
standardizati
on is the
foundation
of
information
system
Standard name
and definition of
wind turbine states
Standard name
and definition of
wind turbine
sensor points
Standard name
and definition of
wind turbine fault
types
SecurityArchitecture
InformationStandardization
8. Overall Architecture of Big Data Platform
- Full Data Acquisition
Big Data Acquisition
8
• Wind turbine
• Transformer substation
• Anemometer tower
• Wind Power prediction
• Vibration monitoring
• AVC/AGC
• Smart grid and meter
• Voice/Video …
Real Time Device/System Data
• Macro political policy
• Macro economic policy
• Social Network
• …
External Data
• Finance
• Infrastructure projects
• HR
• …
ERP System Data
Design
Principles
• Standardization
• Safety &
Compliance
• Reliability
• Manageability
• Self-adaption
• …
Data Acquisition
Data Transmission
Big Data Platform
Big Data Application
SecurityArchitecture
InformationStandardization
9. Overall Architecture of Big Data Platform
- Data Transmission --- Function Architecture
Data Transmission
9
HA/Efficient/Secure
Network
Resume broken transfer
Compression/Encryption/…
Uniform mobile device
access
Completed
Transmission
Solution
Design
Principles
• Standardization
• Safety &
Compliance
• Reliability
• Manageability
• Self-adaption
• …
Data Acquisition
Data Transmission
Big Data Platform
Big Data Application
SecurityArchitecture
InformationStandardization
10. Wind Plant Network
Regional Network
Global Center Network
Office IT Network
Overall Architecture of Big Data Platform
- Data Transmission --- Technology Architecture
Data Acquisition
Data Transmission
Big Data Platform
Big Data Application
SecurityArchitecture
InformationStandardization
10
11. Overall Architecture of Big Data Platform
- Big Data Platform --- Function Architecture
Big Data Platform
11
Data ETL
Extract-Transform-Load
Data Access
RTDB/Kafka/Flume/…
Data Management
HDFS/RDB/Meta Data/…
Data Processing
MapReduce/SPARK/Storm/…
Data Interface/Tools
ODBC/JDBC/R/SPSS/…
Design
Principles
• Standardization
• Safety &
Compliance
• Reliability
• Manageability
• Self-adaption
• …
Data Acquisition
Data Transmission
Big Data Platform
Big Data Application
SecurityArchitecture
InformationStandardization
13. Overall Architecture of Big Data Platform
- Big Data Application
Big Data Application
13
BI
ERP
OA
…
EBA/
Failure
Prediction/
Operation
Optimization/
…
Mobile
APPs/
Visualization
Apps/
…
Asset
Life Cycle
Management
Design
Principles
• Standardization
• Safety &
Compliance
• Reliability
• Manageability
• Self-adaption
• …
Data Acquisition
Data Transmission
Big Data Platform
Big Data Application
SecurityArchitecture
InformationStandardization
14. 14
Big Data
System
Source
Database
Big Data Analytic Platform
Longyuan Power Group
Big Data Analytic
Platform
Customer Analytic
Model
Longyuan
Power
Group
Business
Personnel
Customer
Business
Personnel
Third
Party
Business
Personnel
Longyuan Power Group
Internal
Application System
PaaS
ApplicationAccess
Gateway
Development&
TestPlatform
BusinessServiceBus
API
IaaS
Application
Server
(Middleware)
Cloud Management Platform
Compute
Store
Network/
Message
Manageme
nt
Resource
Management
User
Management
System
Monitoring
Service
Management
Security
Objective
Storage
Block
Storage
Image
Service
Third
Party Big
Data
Analytic
Platform
1
2
3
1
4
5
6
Application Boundary
Security
2 User Identity and Access
Control
3 Big Data Platform Data and
Management Security
5 Storage Encryption Machine
and Access Control
6 Transmission Encryption
4 Memory and Image Isolation
and Erasure
Overall Architecture of Big Data Platform
- Security Architecture
SecurityArchitecture
15. 15
Cloud Monitoring
Application
vulnerability
Application
vulnerability
Security Event
Monitoring
Cloud Auditing
Database Maintenance
Cloud Service
Emergency
Response
API Interface
Monitoring &
Warning
Defense
Log Collection
Reinforcement
Cloud
Resource
Platform
Cloud Security
Operation Platform
User Control
Authorization
Management
Platform Security
Management View
Big Data Analysis
Business Security
Management View
Cloud Defense
VFW
Defense DDOS
VPN
Network Server Application
Cloud
Fortress
Machine
Management
Cloud
Control
Platform
Cloud Resource Scheduling
Cloud Platform Management
CloudCompliance Availability
Database
vulnerability
Knowledge
Database
Antivirus
WAF
Anti-
Tamper
Web Log
Measurement
Service
Security
Reinforcement
Cloud
Compliance
Overall Architecture of Big Data Platform
- Security Architecture
SecurityArchitecture
16. Field
Gateway
Device
Connectivity
Analytics &
Operationalized Insights
Presentation &
Business Connectivity
Overall Architecture of Big Data Platform
- Infrastructure Based on IoT --- Self Hosted Mode
Batch Analytics
Hadoop Spark, Mapreduce, R
Hot Path Analytics
Hadoop Storm, Spark Stream
Presentation &
Business Applications
Mobile BI, ERP, SCM
Front-End Cluster
and ETL
Field
Gateway
Field
Gateway
Management
Dedicated
Network,
Self-hosted
Dedicated
Data Center
Strength Weakness
q Higher CAPEX for
infrastructure
q High OPEX for infrastructure
and dedication operation team
q Construction and deployment
Time
q High security requirements for
dedicated infrastructure
q Real-time monitoring and
remote control
q All of the data and systems in
the user's local
Data Acquisition
Intelligent Gateway/
RTDB/…
Data Transmission
SDH/MSTP/
MPLS-VPN/…
Data Center
Big Data Platform: Hadoop/RDB/ERP/BI/…
Big Data Application: EBA/Prediction/Optimization/…
16
17. Field
Gateway
Device
Connectivity
Analytics &
Operationalized Insights
Presentation &
Business Connectivity
Batch Analytics & Visualizations
Azure Hdinsight, Azure Machine Learning
Hot Path Analytics
Azure Stream Analytics, Azure HDInsight Storm
Presentation &
Business Connectivity
App Service, Websites
Hot Path Business Logic
Micro-services(Docker and Service Fabric)IOT Hub
Field
Gateway
Management
Platform-
as-a-Service
in Public
Cloud
Strength Weakness
q Weak for higher security
requirements for dedicated
infrastructure(network/data
center)
q Weak for real-time control
q All of the data and systems at
the far end
q Lower CAPEX for infrastructure
q Lower OPEX for infrastructure
q Pay-as-go and less deployment
time
Overall Architecture of Big Data Platform
- Infrastructure Based on IoT --- Cloud Mode
Data Acquisition
Intelligent Gateway/
IoT Edge /…
Data Transmission
Internet /4G/…
LPWA(NBIoT/LoRa/SigFox…)
Data Center
IoT Hub + Cloud Service (IaaS + PaaS)
17
18. Summary
18
1
2
3
IoT technology helps us to improve the capability to build deeper and more accurate digital modeling for
the real world and achieve more closely interconnection between people, things and services.
Big Data technology provides the capability of analysis and data mining for massive data generated by IoT.
It helps us reveal the nature and connections of things and will have great support for digitization of
Energy Management System, even for Intelligence Transformation.
Project “Big Data Platform of Wind Power” is under building, technical exchanging with experts and
cooperations from parties who have interested in it are both very welcomed.
