While developing on Apache NiFi MiNiFI C++, my focus has been writing architectural details from Apache NiFi in C++. During this time I've learned of the myriad of applications in which IoT exists. In this presentation we'll discuss key lessons I've learned along with ways that we can help you navigate the IoT realm from the perspective of technical challenges that you will face. We will begin with an overview of Apache NiFi MiNiFi C++ followed by discussions of system awareness. In doing so we'll focus on system monitoring for battery and network status. We'll look at how MiNIFi can be configured to monitor battery and network status to throttle itself and be a good steward of device management. In the same vein we'll discuss the idiosyncrasies that you must be aware of when dealing with common IoT use cases, such as connected cars and embedded devices on the move. Finally, we'll look at lessons we've learned in helping other developers write software in the IoT realm. MARC PARISI, Principal SW Engineer, Hortonworks

1. 1 © Hortonworks Inc. 2011–2018. All rights reserved
Navigating Idiosyncrasies
of IoT Development
Marc Parisi
Principal Software Engineer @ Hortonworks

2. 2 © Hortonworks Inc. 2011–2018. All rights reserved
BIO
marc(int*):
push rbp
mov rbp, rsp
mov QWORD PTR [rbp-8], rdi
mov QWORD PTR [rbp-8], 0
mov rax, QWORD PTR [rbp-8]
mov eax, DWORD PTR [rax]
pop rbp
ret

3. 3 © Hortonworks Inc. 2011–2018. All rights reserved
Agenda
• What is MiNiFI C++ and its use cases
• Idiosyncracies ( Use case derived)
• Lessons learned

4. 4 © Hortonworks Inc. 2011–2018. All rights reserved
Apache NiFi MiNiFi C++
• “...[C]omplimentary data collection approach that supplements the core tenets of NiFi in
dataflow management, focusing on the collection of data at the source of its creation”
• Deliver data across a common platform of extensions and components
• NiFi empowers infrastructure

5. 5 © Hortonworks Inc. 2011–2018. All rights reserved
The NiFi Ecosystem

6. 6 © Hortonworks Inc. 2011–2018. All rights reserved
How is MiNiFi used
• Agent
• Full integration with NiFI
• API Library
• Developers choose pieces

7. 7 © Hortonworks Inc. 2011–2018. All rights reserved
IoT Versus Edge
• IoT
• Devices that share and collect data
• Edge
• Data processing at edge of one’s network ( or the internet ?)
• One in the same?

8. 8 © Hortonworks Inc. 2011–2018. All rights reserved
Explosion of Devices

9. 9 © Hortonworks Inc. 2011–2018. All rights reserved
https://pixabay.com/en/network-iot-internet-of-things-782707/
IoT Use Cases
• Connected car
• Cell phone collection
• Smart watches
• Sensor readers
• ...oh my

10. 10 © Hortonworks Inc. 2011–2018. All rights reserved
From the Device to the Warehouse...
• Devices → Warehouse/Regional
• Minimal grooming
• Apache NiFi regional and core
• May not have compute capacity
• ‘Cause a toaster’s gonna toast, toast, toast, toast [a]

11. 11 © Hortonworks Inc. 2011–2018. All rights reserved
https://pixabay.com/en/network-iot-internet-of-things-782707/
Edge Use Cases
• Connected car
• Cell phone collection
• Smart watches
• Sensor readers
• ...OR
• Custom devices
• Robotics
• …oh my?

12. 12 © Hortonworks Inc. 2011–2018. All rights reserved
Living on the Edge
• Edge processing increases capacity
• Edge devices can be ..
• Raspberry Pi
• Computer
• Phone/Watch
• Have different idiosyncrasies

13. 13 © Hortonworks Inc. 2011–2018. All rights reserved
Myriad of use cases
• Agent/Library
• Data collector/Sender
• Edge/Analytic Processor

14. 14 © Hortonworks Inc. 2011–2018. All rights reserved
Idiosyncrasies of building
• Many caveats and ways to…
• Run
• Build
• Use…
• Must account for all of this
• Custom builds

15. 15 © Hortonworks Inc. 2011–2018. All rights reserved
The struggle is real
• Software Versions
• Operating system variances
• Dependency management
• Custom hardware
• Custom software

16. 16 © Hortonworks Inc. 2011–2018. All rights reserved
IoT/Edge Runtime Nuances [1]
• Limited computing capability
• Limited power/network
• Restricted software library/platform availability
• Not frequently updated
• Competing standards/protocols
• Scalability
• Privacy & Security

17. 17 © Hortonworks Inc. 2011–2018. All rights reserved
Resource Awareness
• Battery Aware resource manager [3]
• Monitor consumption
• Adjust as needed
• Network manager
• Durable repositories

18. 18 © Hortonworks Inc. 2011–2018. All rights reserved
Command and Control
• Command and control design [6,7]
• Provide updates
• Need full control
• Full disks
• Disk failure

19. 19 © Hortonworks Inc. 2011–2018. All rights reserved
Connected car -- Data collector
IoT/Collector Examples

20. 20 © Hortonworks Inc. 2011–2018. All rights reserved
Connected Car concerns
• Limited Processing
• Power is transient
• May not have ability to shutdown gracefully

21. 21 © Hortonworks Inc. 2011–2018. All rights reserved
Edge Example
• Robotic controller
• Not typical use case
• OpenCV detects red in image *
• ListenHTTP receives movement commands
• Computation determines motor speed through GPIO
• Computation for servo control

22. 22 © Hortonworks Inc. 2011–2018. All rights reserved
Concerns faced here
• Device has processing, but must be battery aware
• Throttle if necessary
• Storage must be durable
• Power loss
• Device Failure
• Persistence can’t be a bottleneck

23. 23 © Hortonworks Inc. 2011–2018. All rights reserved
What we’ve learned
• Not all use cases created equally

24. 24 © Hortonworks Inc. 2011–2018. All rights reserved
That boy sure is a runnin’ fool
• Failure is always an option
• Upgrades to flow/binary must be small
• Mo’ features, Mo’ problems

25. 25 © Hortonworks Inc. 2011–2018. All rights reserved
Break it down simple for me
• Build a bear is the best approach
• Control data and resources
• Expect failure

26. 26 © Hortonworks Inc. 2011–2018. All rights reserved
Thank you
phrocker@apache.org // @MrTheSegfault // https://www.parisi.io

27. 27 © Hortonworks Inc. 2011–2018. All rights reserved
[1] https://www.statista.com/statistics/471264/iot-number-of-connected-devices-
worldwide/
[2] https://www.slideshare.net/Hadoop_Summit/apache-nifi-crash-course
[3] https://www.youtube.com/watch?v=1TMArQS7w40
[4] https://github.com/apache/nifi-minifi-cpp/blob/master/PROCESSORS.md
[5] https://blog.sasken.com/the-connected-car/
[6] https://cwiki.apache.org/confluence/display/MINIFI/MiNiFi+Command+and+Control
[7] https://cwiki.apache.org/confluence/display/MINIFI/C2+Design+Proposal
[8] https://github.com/apache/nifi-minifi-cpp/#managing-minfi-c-through-the-minifi-
controller
[9] https://youtu.be/a9EoCRYu39o
[a] https://www.youtube.com/watch?v=nfWlot6h_JM
Bibliography