Database ingest with Apache NiFi and MiNiFi

•

3 gefällt mir•765 views

This document discusses data ingestion using Apache NiFi and MiNiFi. Apache NiFi is a dataflow system that allows for reliable and secure transfer of data between systems. It is used for ingesting data from sources into analytic platforms and preparing data through actions like format conversion and parsing. Apache MiNiFi is a subproject that deploys smaller agents to collect data at the edge and push it to NiFi in the data center. The document provides an overview of NiFi's capabilities like guaranteed delivery, buffering, security, and clustering support.

Technologie

Data Ingestion with
Apache NiFi and MiNiFi
Lucian Neghina
Big Data Architect
by Developer for Developers

➔ Periodically batch ingestion
➔ Incremental ingestion
➔ Over 100 servers
➔ No access to the servers
➔ Push access only
➔ Networking failure
➔ Secure communication
Challenges

Apache NiFi
NiFi is based on Flow Based Programming
➔ Data agnostic
➔ Guaranteed data delivery
➔ Data Buffering / Back-Pressure
➔ Prioritized queuing
➔ Secure (SSL, HTTPS, encrypted content...)
➔ Data provenance
➔ Support push and pull models
➔ Designed for extension
➔ Clustering

Apache NiFi
What is NiFi used for?
➔ Reliable and secure transfer of data between systems
➔ Delivery of data from sources to analytic platforms
➔ Enrichment and preparation of data:
◆ Conversion between formats
◆ Extraction /Parsing
◆ Routing decisions
What is NiFi NOT used for?
➔ Distributed computation
➔ Complex Event Processing
➔ Long term data storage

Apache MiNiFi
Subproject of Apache NiFi
➔ Agent model to collect data at the edge
➔ Smaller footprint
➔ Design & Deploy
➔ Warm re-deploys
NiFi lives in the data center
MiNiFi lives as close to where data is born

System Architecture
Physical Store
...
Data Center
Server Cluster

Thank you
● NiFi Template
https://github.com/nflucian/nifi/raw/master/DemoNiFi.xml
● twitter: @nflucian
● github: github.com/nflucian

Empfohlen

Data ingestion and distribution with apache NiFiLev Brailovskiy

Nifi workshopYifeng Jiang

Apache NiFi Meetup - Introduction to NiFi RegistryBryan Bende

Joe witt may2015_kafka_nyc_apachenifi-overviewJoseph Witt

NifiJulio Castro

Apache Nifi - Custom Processor thotasrinath

Joe Witt presentation on Apache NiFiMark Kerzner

What will be new in Apache NiFi 1.2.0Koji Kawamura

Empfohlen

Data ingestion and distribution with apache NiFiLev Brailovskiy

Nifi workshopYifeng Jiang

Apache NiFi Meetup - Introduction to NiFi RegistryBryan Bende

Joe witt may2015_kafka_nyc_apachenifi-overviewJoseph Witt

NifiJulio Castro

Apache Nifi - Custom Processor thotasrinath

Joe Witt presentation on Apache NiFiMark Kerzner

What will be new in Apache NiFi 1.2.0Koji Kawamura

Big Data Day LA 2016/ Big Data Track - Building scalable enterprise data flow...Data Con LA

Integrating Apache NiFi and Apache Apex Apache Apex

Taking DataFlow Management to the Edge with Apache NiFi/MiNiFiBryan Bende

HDF Powered by Apache NiFi IntroductionMilind Pandit

Hortonworks Data in Motion Webinar Series Part 7 Apache Kafka Nifi Better Tog...Hortonworks

Introduction to data flow management using apache nifiAnshuman Ghosh

Running Apache NiFi with Apache Spark : Integration OptionsTimothy Spann

Flink and NiFi, Two Stars in the Apache Big Data ConstellationMatthew Ring

AWS User Group Meetup Berlin - Kay Lerch on Apache NiFi (2016-04-19)Kay Lerch

Beyond Messaging Enterprise Dataflow powered by Apache NiFiIsheeta Sanghi

Setting Up InfluxDB for IoT by David G SimmonsInfluxData

Integrating Apache Spark and NiFi for Data LakesDataWorks Summit/Hadoop Summit

Log Analytics OptimizationIsheeta Sanghi

IOT, Streaming Analytics and Machine Learning DataWorks Summit/Hadoop Summit

Dataflow with Apache NiFi - Apache NiFi Meetup - 2016 Hadoop Summit - San JoseAldrin Piri

InfluxDB 2.0 Client Libraries by Noah CrowleyInfluxData

Integrating NiFi and FlinkBryan Bende

Confluent kafka meetupseattle jan2017Nitin Kumar

You Can't Search Without DataBryan Bende

Extending The Yahoo Streaming Benchmark to Apache ApexApache Apex

AIDevWorldApacheNiFi101Timothy Spann

Integração de Dados com Apache NIFI - Marco Garcia CetaxMarco Garcia

Weitere ähnliche Inhalte

Was ist angesagt?

Big Data Day LA 2016/ Big Data Track - Building scalable enterprise data flow...Data Con LA

Integrating Apache NiFi and Apache Apex Apache Apex

Taking DataFlow Management to the Edge with Apache NiFi/MiNiFiBryan Bende

HDF Powered by Apache NiFi IntroductionMilind Pandit

Hortonworks Data in Motion Webinar Series Part 7 Apache Kafka Nifi Better Tog...Hortonworks

Introduction to data flow management using apache nifiAnshuman Ghosh

Running Apache NiFi with Apache Spark : Integration OptionsTimothy Spann

Flink and NiFi, Two Stars in the Apache Big Data ConstellationMatthew Ring

AWS User Group Meetup Berlin - Kay Lerch on Apache NiFi (2016-04-19)Kay Lerch

Beyond Messaging Enterprise Dataflow powered by Apache NiFiIsheeta Sanghi

Setting Up InfluxDB for IoT by David G SimmonsInfluxData

Integrating Apache Spark and NiFi for Data LakesDataWorks Summit/Hadoop Summit

Log Analytics OptimizationIsheeta Sanghi

IOT, Streaming Analytics and Machine Learning DataWorks Summit/Hadoop Summit

Dataflow with Apache NiFi - Apache NiFi Meetup - 2016 Hadoop Summit - San JoseAldrin Piri

InfluxDB 2.0 Client Libraries by Noah CrowleyInfluxData

Integrating NiFi and FlinkBryan Bende

Confluent kafka meetupseattle jan2017Nitin Kumar

You Can't Search Without DataBryan Bende

Extending The Yahoo Streaming Benchmark to Apache ApexApache Apex

Was ist angesagt? (20)

Big Data Day LA 2016/ Big Data Track - Building scalable enterprise data flow...

Integrating Apache NiFi and Apache Apex

Taking DataFlow Management to the Edge with Apache NiFi/MiNiFi

HDF Powered by Apache NiFi Introduction

Hortonworks Data in Motion Webinar Series Part 7 Apache Kafka Nifi Better Tog...

Introduction to data flow management using apache nifi

Running Apache NiFi with Apache Spark : Integration Options

Flink and NiFi, Two Stars in the Apache Big Data Constellation

AWS User Group Meetup Berlin - Kay Lerch on Apache NiFi (2016-04-19)

Beyond Messaging Enterprise Dataflow powered by Apache NiFi

Setting Up InfluxDB for IoT by David G Simmons

Integrating Apache Spark and NiFi for Data Lakes

Log Analytics Optimization

IOT, Streaming Analytics and Machine Learning

Dataflow with Apache NiFi - Apache NiFi Meetup - 2016 Hadoop Summit - San Jose

InfluxDB 2.0 Client Libraries by Noah Crowley

Integrating NiFi and Flink

Confluent kafka meetupseattle jan2017

You Can't Search Without Data

Extending The Yahoo Streaming Benchmark to Apache Apex

Ähnlich wie Database ingest with Apache NiFi and MiNiFi

AIDevWorldApacheNiFi101Timothy Spann

Integração de Dados com Apache NIFI - Marco Garcia CetaxMarco Garcia

Automate your data flows with Apache NIFIAdam Doyle

Tracking Crime as It Occurs with Apache Phoenix, Apache HBase and Apache NiFiDataWorks Summit

Introduction to Apache NiFi dws19 DWS - DC 2019Timothy Spann

Using Apache NiFi with Apache Pulsar for Fast Data On-RampTimothy Spann

OSSNA Building Modern Data Streaming AppsTimothy Spann

Timothy Spann [StreamNative] | Using FLaNK with InfluxDB for EdgeAI IoT at Sc...InfluxData

Using FLiP with influxdb for EdgeAI IoT at ScaleTimothy Spann

Moving to hyper-converged? Don’t forget about data protection.Veritas Technologies LLC

Budapest Data/ML - Building Modern Data Streaming Apps with NiFi, Flink and K...Timothy Spann

OpenSource and the Cloud ApacheCon.pptxlohitvijayarenu

Big Data Day LA 2015 - Always-on Ingestion for Data at Scale by Arvind Prabha...Data Con LA

Conf42Python -Using Apache NiFi, Apache Kafka, RisingWave, and Apache Iceberg...Timothy Spann

Lightning Fast Analytics with Hive LLAP and DruidDataWorks Summit

xGem Data Stream ProcessingJorge Hirtz

Devfest uk & ireland using apache nifi with apache pulsar for fast data on-r...Timothy Spann

Unconference Round Table NotesTimothy Spann

Machine Data 101Splunk

Big Data for Security - DNS AnalyticsMarco Casassa Mont

Ähnlich wie Database ingest with Apache NiFi and MiNiFi (20)

AIDevWorldApacheNiFi101

Integração de Dados com Apache NIFI - Marco Garcia Cetax

Automate your data flows with Apache NIFI

Tracking Crime as It Occurs with Apache Phoenix, Apache HBase and Apache NiFi

Introduction to Apache NiFi dws19 DWS - DC 2019

Using Apache NiFi with Apache Pulsar for Fast Data On-Ramp

OSSNA Building Modern Data Streaming Apps

Timothy Spann [StreamNative] | Using FLaNK with InfluxDB for EdgeAI IoT at Sc...

Using FLiP with influxdb for EdgeAI IoT at Scale

Moving to hyper-converged? Don’t forget about data protection.

Budapest Data/ML - Building Modern Data Streaming Apps with NiFi, Flink and K...

OpenSource and the Cloud ApacheCon.pptx

Big Data Day LA 2015 - Always-on Ingestion for Data at Scale by Arvind Prabha...

Conf42Python -Using Apache NiFi, Apache Kafka, RisingWave, and Apache Iceberg...

Lightning Fast Analytics with Hive LLAP and Druid

xGem Data Stream Processing

Devfest uk & ireland using apache nifi with apache pulsar for fast data on-r...

Unconference Round Table Notes

Machine Data 101

Big Data for Security - DNS Analytics

Kürzlich hochgeladen

Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...Igalia

From Event to Action: Accelerate Your Decision Making with Real-Time AutomationSafe Software

EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptxEarley Information Science

TrustArc Webinar - Stay Ahead of US State Data Privacy Law DevelopmentsTrustArc

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProduct Anonymous

presentation ICT roal in 21st century educationjfdjdjcjdnsjd

🐬 The future of MySQL is Postgres 🐘RTylerCroy

IAC 2024 - IA Fast Track to Search Focused AI SolutionsEnterprise Knowledge

Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...apidays

04-2024-HHUG-Sales-and-Marketing-Alignment.pptxHampshireHUG

GenCyber Cyber Security Day PresentationMichael W. Hawkins

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoffsammart93

Data Cloud, More than a CDP by Matt RobisonAnna Loughnan Colquhoun

Powerful Google developer tools for immediate impact! (2023-24 C)wesley chun

Handwritten Text Recognition for manuscripts and early printed textsMaria Levchenko

CNv6 Instructor Chapter 6 Quality of Servicegiselly40

08448380779 Call Girls In Greater Kailash - I Women Seeking MenDelhi Call girls

GenAI Risks & Security Meetup 01052024.pdflior mazor

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Drew Madelung

The 7 Things I Know About Cyber Security After 25 Years | April 2024Rafal Los

Kürzlich hochgeladen (20)

Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...

From Event to Action: Accelerate Your Decision Making with Real-Time Automation

EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

presentation ICT roal in 21st century education

🐬 The future of MySQL is Postgres 🐘

IAC 2024 - IA Fast Track to Search Focused AI Solutions

Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...

04-2024-HHUG-Sales-and-Marketing-Alignment.pptx

GenCyber Cyber Security Day Presentation

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

Data Cloud, More than a CDP by Matt Robison

Powerful Google developer tools for immediate impact! (2023-24 C)

Handwritten Text Recognition for manuscripts and early printed texts

CNv6 Instructor Chapter 6 Quality of Service

08448380779 Call Girls In Greater Kailash - I Women Seeking Men

GenAI Risks & Security Meetup 01052024.pdf

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

The 7 Things I Know About Cyber Security After 25 Years | April 2024

Database ingest with Apache NiFi and MiNiFi

1. Data Ingestion with Apache NiFi and MiNiFi Lucian Neghina Big Data Architect by Developer for Developers

2. ➔ Periodically batch ingestion ➔ Incremental ingestion ➔ Over 100 servers ➔ No access to the servers ➔ Push access only ➔ Networking failure ➔ Secure communication Challenges

3. Data Flow

4. Simplistic view of Data Flow

5. Realistic view of Data Flow

6. Apache NiFi NiFi is based on Flow Based Programming ➔ Data agnostic ➔ Guaranteed data delivery ➔ Data Buffering / Back-Pressure ➔ Prioritized queuing ➔ Secure (SSL, HTTPS, encrypted content...) ➔ Data provenance ➔ Support push and pull models ➔ Designed for extension ➔ Clustering

7. Apache NiFi What is NiFi used for? ➔ Reliable and secure transfer of data between systems ➔ Delivery of data from sources to analytic platforms ➔ Enrichment and preparation of data: ◆ Conversion between formats ◆ Extraction /Parsing ◆ Routing decisions What is NiFi NOT used for? ➔ Distributed computation ➔ Complex Event Processing ➔ Long term data storage

8. Apache MiNiFi Subproject of Apache NiFi ➔ Agent model to collect data at the edge ➔ Smaller footprint ➔ Design & Deploy ➔ Warm re-deploys NiFi lives in the data center MiNiFi lives as close to where data is born

9. System Architecture Physical Store ... Data Center Server Cluster

10. Data Flow - Server-side (NiFi)

11. Data Flow - Agent-side (MiNiFi)

12. Thank you ● NiFi Template https://github.com/nflucian/nifi/raw/master/DemoNiFi.xml ● twitter: @nflucian ● github: github.com/nflucian