The document discusses the flow development lifecycle (FDLC) challenges in developing and maintaining data flows with Apache NiFi. It introduces Apache NiFi Registry, which allows flows to be versioned, promoted between environments, and synchronized across instances. A demo is shown of exporting a flow to the registry, modifying and committing changes, and updating another NiFi instance with the latest flow version.

1. © Hortonworks Inc. 2011–2018. All rights reserved1
Forget Duplicating Local Changes: Apache NiFi
and the Flow Development Lifecycle (FDLC)
Andy LoPresto | @yolopey
Sr. Member of Technical Staff at Hortonworks, Apache NiFi PMC & Committer
19 April 2018 Dataworks Summit Berlin

2. © Hortonworks Inc. 2011–2018. All rights reserved2
Gauging Audience Familiarity With NiFi
“What’s a NeeFee?”
No experience with dataflow
No experience with NiFi
“I can pick this up pretty quickly”
Some experience with dataflow
Some experience with NiFi
“I refactored the Ambari
integration endpoint to allow
for mutual authentication
TLS during my coffee break”
Forgotten more about NiFi
than most of us will ever
know

3. © Hortonworks Inc. 2011–2018. All rights reserved3
Agenda
• Flow development lifecycle
• Challenges in FDLC
• Apache NiFi Registry
• Roadmap
• All slides provided online, so no need to transcribe

4. © Hortonworks Inc. 2011–2018. All rights reserved4
Flows Definitely Love Change

5. © Hortonworks Inc. 2011–2018. All rights reserved5
Flow Development Lifecycle (FDLC)
• Origins of NiFi
• Operator Experience
• MC data, don’t drop, mitigate
temporarily

6. © Hortonworks Inc. 2011–2018. All rights reserved6
Operator Experience

7. © Hortonworks Inc. 2011–2018. All rights reserved6
Operator Experience

8. © Hortonworks Inc. 2011–2018. All rights reserved6
Operator Experience

9. © Hortonworks Inc. 2011–2018. All rights reserved7
• Shows previous values (user,
time changed)
• Sensitive values are always
encrypted at rest and never
returned via the API
Component Property History

10. © Hortonworks Inc. 2011–2018. All rights reserved8
Exporting Flows
• XML templates
• Copying flow.xml.gz
between systems

11. © Hortonworks Inc. 2011–2018. All rights reserved9
Flow Developers Lose Consciousness

12. © Hortonworks Inc. 2011–2018. All rights reserved10
Challenges
• Templates
• Updates/replacement
• Sensitive property replacement
• Flow.xml.gz migration
• Key synchronization
• Environment promotion
• Approval processes
• Verifiability

13. © Hortonworks Inc. 2011–2018. All rights reserved11
Template Replacement
• Export a new version of template
• Transfer (somehow)
• Verify?
• Import onto canvas side-by-side existing
flow
• Stop processors
• Empty queues
• Reconnect queues
• Start
• Pray?

14. © Hortonworks Inc. 2011–2018. All rights reserved12
Template Replacement

15. © Hortonworks Inc. 2011–2018. All rights reserved13
Finally, Dataflow Lessons Conquer

16. © Hortonworks Inc. 2011–2018. All rights reserved14
• Previously, flows were exported via
XML templates
• Didn’t contain sensitive values
• Couldn’t be updated in-place
• No tracking system
• NiFi Registry brings asset management
as first-class citizen to NiFi
• Flows can be versioned
• Flows can be promoted between
environments
Introducing Apache NiFi Registry 0.1.0
NiFi Registry for Dataflows

17. © Hortonworks Inc. 2011–2018. All rights reserved15
First, Demos Look Cool

18. © Hortonworks Inc. 2011–2018. All rights reserved16
• Install nifi-registry
• $ mvn clean install
• $ ./bin/nifi-registry.sh
start
• Browse to http://localhost:18080
Create Registry

19. © Hortonworks Inc. 2011–2018. All rights reserved17
Create Bucket

20. © Hortonworks Inc. 2011–2018. All rights reserved18
Connect to NiFi

21. © Hortonworks Inc. 2011–2018. All rights reserved18
Connect to NiFi

22. © Hortonworks Inc. 2011–2018. All rights reserved19
Create Process Group

23. © Hortonworks Inc. 2011–2018. All rights reserved19
Create Process Group

24. © Hortonworks Inc. 2011–2018. All rights reserved20
Commit Version

25. © Hortonworks Inc. 2011–2018. All rights reserved20
Commit Version

26. © Hortonworks Inc. 2011–2018. All rights reserved20
Commit Version

27. © Hortonworks Inc. 2011–2018. All rights reserved21
View flow in Registry

28. © Hortonworks Inc. 2011–2018. All rights reserved22
Import new instance into NiFi

29. © Hortonworks Inc. 2011–2018. All rights reserved22
Import new instance into NiFi

30. © Hortonworks Inc. 2011–2018. All rights reserved22
Import new instance into NiFi

31. © Hortonworks Inc. 2011–2018. All rights reserved23
Modify the original flow

32. © Hortonworks Inc. 2011–2018. All rights reserved23
Modify the original flow

33. © Hortonworks Inc. 2011–2018. All rights reserved23
Modify the original flow

34. © Hortonworks Inc. 2011–2018. All rights reserved24
See local changes before committing

35. © Hortonworks Inc. 2011–2018. All rights reserved24
See local changes before committing

36. © Hortonworks Inc. 2011–2018. All rights reserved24
See local changes before committing

37. © Hortonworks Inc. 2011–2018. All rights reserved25
Commit

38. © Hortonworks Inc. 2011–2018. All rights reserved25
Commit

39. © Hortonworks Inc. 2011–2018. All rights reserved26
Update new instance from Registry

40. © Hortonworks Inc. 2011–2018. All rights reserved26
Update new instance from Registry

41. © Hortonworks Inc. 2011–2018. All rights reserved26
Update new instance from Registry

42. © Hortonworks Inc. 2011–2018. All rights reserved27
Future Direction and Likely Concerns

43. © Hortonworks Inc. 2011–2018. All rights reserved28
• User access control
• Toolkit
• CLI
• Multiple (* - *) registries to NiFis
Other Features

44. © Hortonworks Inc. 2011–2018. All rights reserved29
• Portability
• Share with colleagues
• Isolate responsibilities
• Versioning
• Synchronize
• Track changes
• Ensure flow accuracy
• Promotion between environments
• Collaboration
• Reusable flow “portion” components
Key Takeaways

45. © Hortonworks Inc. 2011–2018. All rights reserved30
• git-backed persistence provider [NIFIREG-162 | PR 112]
• Commit hooks [NIFIREG-160 | PR 110]
• Preview incoming changes [NIFIREG-166]
• Your idea here?
Future Enhancements

46. © Hortonworks Inc. 2011–2018. All rights reserved31
Thank you
alopresto@hortonworks.com | alopresto@apache.org | @yolopey
github.com/alopresto/slides

47. © Hortonworks Inc. 2011–2018. All rights reserved32
• NiFi 1.6.0 — 08 April 2018
• MongoDB, InfluxDB, Druid, HBase
components
• Granular @Restricted components
• MiNiFi C++ 0.4.0 — 27 January 2018
• MiNiFi Java 0.4.0 — 22 January 2018
• NiFi Registry 0.1.0 — 1 January 2018
Introducing Apache NiFi Registry
New Announcements

48. © Hortonworks Inc. 2011–2018. All rights reserved33
Community Health

49. © Hortonworks Inc. 2011–2018. All rights reserved34
Apache NiFi site
https://nifi.apache.org
Subproject MiNiFi site
https://nifi.apache.org/minifi/
Subscribe to and collaborate at
dev@nifi.apache.org
users@nifi.apache.org
Submit Ideas or Issues
https://issues.apache.org/jira/browse/NIFI
Follow us on Twitter
@apachenifi
Learn more and join us

50. © Hortonworks Inc. 2011–2018. All rights reserved35
More NiFi This Week…
Title Room Time Speaker(s)
Apache NiFi Crash Course Hall I - D 1115 - 1345 Andy LoPresto, Tim Spann
IoT with Apache MXNet and Apache NiFi and
MiNiFi
Hall I - C 1150 - 1230 Tim Spann
Best practices and lessons learnt from Running
Apache NiFi at Renault
Europe 1650 - 1730 Adel Gacem, Abdelkrim Hadjidj
From an experiment to a real production
environment
Room V 1650 - 1730 Jeroen Wolffensperger, Martijn Groen
IoT, Streaming, and Dataflow Birds of a Feather Room I 1740 - 1855 George Vetticaden, Davor Bonaci, Andy
LoPresto, Stephan Ewen
Intelligently Collecting Data at the Edge —
Intro to Apache MiNiFi
Room II 1100 - 1140 Andy LoPresto
The Power of Intelligent Flows: Realtime IoT
Botnet Classification with Apache NiFi
Hall I - C 1400 - 1440 Andy LoPresto
Forget Duplicating Local Changes: Apache NiFi
and the Flow Development Lifecycle (FDLC)
Room II 1600 - 1640 Andy LoPresto