Presentation by Bin Cheng Senior Researcher, NEC Labs Europe FIWARE Global Summit 8-9 May, 2018 Porto, Portugal

1. FIWARE at the edge:
FogFlow, a new GE for IoT edge computing
Bin Cheng (bin.cheng@neclab.eu),
Ernö Kovacs (ernoe.kovacs@neclab.eu)
NEC Labs Europe

2. Agenda
§ Overview of FogFlow
• motivation, value proposition, major design, key feature
§ How to deploy FogFlow
• Set up your own FogFlow system
§ How to use the programming models in FogFlow
• Service topology: launching dynamic data flows on demand, development process and use case
(anomaly detection)
• Fog function: serverless edge computing, development process and use case (smart parking)
§ Integration with other systems
• AWS IoT, Orion Context Broker
§ References
1

3. 2
Part I: Overview of FogFlow

4. Motivation of FogFlow
3
Lower Complexity
Lower Cost
Higher Efficiency
Smart
Cities
Public
Safety
Smart
Factories
IoT services
difficult to program
Infrastructure
difficult to manage
Edges
IoT devices
not able to be smart
After using NEC FogFlow
Edge Computing Framework
Lack of model
High complexity from
heterogeneity and
dynamics
No fast, closed control
loop
Development + Operation
Development + Operation
Faster response time,
better scalability

5. Value Proposition of FogFlow
4
IoT Service
providers
Infrastructure
Providers
Device
Providers
Edge
programming
model
Fast control
loop
Optimized
orchestration
Low
learning cost
Low
OPEX
Fast
time-to-market
Improved
QoS

6. FogFlow: Context-driven Cloud-Edge Orchestration
§ FogFlow is a cloud-edge orchestrator to orchestrate dynamic NGSI-based data
processing flows on-demand between producers and consumers for providing timely
results to make fast actions, based on context (system context and data context)
Producers
(sensors)
Consumers
(actuators)
cloud
edge edge edge
raw context information
timely results fast actions
FogFlow dynamic
processing flows
Data context
System context

7. How It Works
6
Template of your IoT Service Execution Plan Deployment Plan
cloud
edge1
edge2
Task
generation
Task
specification
Task
deployment
Cloud-Edge
Programming model Cloud-Edge Orchestration
Graphical
editor
Automation &
optimizations
Cloud-Edge
Environment IoT Devices

8. Key Features (1): Agile Edge Programming
7
Reusable
building blocks
Compositing them with declarative hints
Submit
Select
Your service is
ready in minutes
Programming Cloud and Edges easily and fast

9. Key Features (1): Automated and Optimized Orchestration for
Cloud-Edge
8
Connected
Device
Backend
Cloud(s)
Other
Edges
Sensing
Notify
Orchestrating
Collaborative
deployment
Closed control loop
Dynamic processing flows
Reacting
Nearby
Edge
Collaborative
deployment
Orchestrating and managing dynamic processing flows over cloud
and edges in an efficient and optimized manner

10. 9
Part I: Deployment of FogFlow

11. System Architecture of FogFlow
10

12. System Components in FogFlow
11

13. Set Up The Cloud Part:
§ Step 1: download the following compressed file
• wget https://github.com/smartfog/fogflow/raw/master/deployment/core/fogflow-core.tar.gz
§ Step 2: extract deployment scripts from the downloaded file
• tar xvf fogflow-core.tar.gz
§ Step3: change the configuration files for each fogflow components, including
• discovery, broker, master, worker, designer
§ Step4: use docker-compose to launch all components
• docker-compose up or docker-compose start
12

14. How configuration files should be changed accordingly?
discovery-config.json
broker-config.json
worker-config.json
master-config.json designer-config.json
Nothing to change
"my_ip": "155.54.239.141",
"host": "155.54.239.141",
"my_ip": "155.54.239.141",
"discoveryURL": "http://155.54.239.141:8071/ngsi9",
"brokerURL": "http://155.54.239.141:8070/ngsi10",
"agentIP": "155.54.239.141",
HOST_IP = 155.54.239.141

15. Set Up The Edge Part:
14
§ Step 1: download the following compressed file
• wget https://github.com/smartfog/fogflow/raw/master/deployment/edge/fogflow-edge.tar.gz
§ Step 2: extract deployment scripts from the downloaded file
• tar xvf fogflow-edge.tar.gz
§ Step3: change the configuration files for each fogflow components, including
• broker, worker
§ Step4: use docker-compose to launch all components
• docker-compose up or docker-compose start

16. How configuration files should be changed accordingly?
15
broker-config.json
worker-config.json
"host": "155.54.239.141",
"discoveryURL": "http://155.54.239.141:8071/ngsi9",
"my_ip": "155.54.239.141",
"message_bus": "amqp://admin:mypass@155.54.239.141:5672/",
"iot_discovery_url": "http://155.54.239.141:8071/ngsi9",
HOST_IP = 155.54.239.141

17. Verify Your Deployment via FogFlow Dashboard
§ Check if the FogFlow dashboard can be accessed from your browser
16
HOST_IP = 155.54.239.141
http://HOST_IP:8080/

18. Other possible changes to the configuration files
§ Port numbers used by each component
• Must pay attention to the dependency between different components
§ Geo-location of each component
17
"physical_location":{
"longitude": 139.463377,
"latitude": 36.509163,
"section": "01",
"district": "02",
"city": "Tokyo"
},

19. 18
Part II: Programming Models in FogFlow

20. Two Types of Programming Models in FogFlow
§ Service topology: on-demand data processing flows
• Triggered by a requirement on demand
• Output-driven
§ Fog function: serverless edge computing, function triggered by context availability
• Triggered by context availability automatically
• Input-driven
19
requirement
Fog
function
availability of input data
Linked automatically

21. Orchestration of Service Topology
20
Service Topology
Execution Plan
Deployment Plan
cloud
edge1edge2
q Expected output
q Scope
q scheduler
locality aware deployment
dynamic execution graph

22. Context-driven Cloud-Edge Orchestration: granularity, scope
21
Data Context and
availability
(metadata, availability)
System Context
(locality, mobility, capacity,
security, …)
Programming model
with graphical editor
scope
granularity

23. Development Process
22

24. Use Case Example
23

25. Designing a IoT Service
24

26. Developing Dockerized Operators
25
Templates for Python/Javascript are available in github
fogflow/application/template

27. Defining Your Service Topology
26

28. Triggering Your IoT Service
27
Service Topology
Execution Plan
Deployment Plan
cloud
edge1edge2
q Expected output
q Scope
q scheduler
locality aware deployment
dynamic execution graph
Orchestration requirement

29. Fog Function in FogFlow: serverless edge computing
§ Fog function: just do one thing for a single functionality
• Triggered by the availability(metadata) of its inputs, light-weight
28
fog function
input
function
instances
trigger
ff2
Twin ATwin BTwin C
ff1
Digital Twins
Fog Function
Instances
A network of twins, linked via fog
functions

30. How Fog Function Works in FogFlow
29
Input Annotation
Docker image
Output
Annotation
IoT Discovery Broker(s)
FogFlow
Orchestrator
Function
instance(s)
FogFlow Worker(s)
(cloud or edges)
IoT Device(s)
fog function
Twins
1) Submit
Fog function
2)SUBSCRIBE,
NGSI9
3) UPDATE, NGSI10
4) UPDATE,
NGSI9
5)NOTIFY,
NGSI9
5) Create and deploy fog function instances
6) Processing data and create/update
digital twin

31. Editor to Annotate a Fog Function in FogFlow
30
Python or Javascript
Code or
annotation
Implementation image
docker
images

32. Register a basic operator
31

33. Implement an operator
§ template is available at github
• https://github.com/smartfog/fogflow/tree/master/application/fogfunction/javascript
32
exports.handler = function(contextEntity, publish, query, subscribe)
Example of how to publish the generate result
Example of how to query something
Example of how to subscribe something

34. Use Case: Smart Parking
33
Connected
car
private
site
Public
site
Real-time
estimation
Prediction
Arrival time
Recommender
Utilizing NGSI API to
access digital twins
Digital twins: Public parking sites,
private parking sites, connected cars

35. 34
Part III: Integration with Other FIWARE GEs

36. Integration with Other GEs
35
Distributed Context
Management System
OrionProcessing
tasks
FogFlow subscription
APPS
Other
GE(s)
sensors actuators
FogFlow
Dashboard
notify
e.g. Cygnus
non-NGSI
devices
Adapter(s)
(e.g., openMTC, IoT
Agent)

37. Reference and Acknowledgement
§ The code repository of FogFlow is available at github,
https://github.com/smartfog/fogflow
§ Online tutorial is available at http://fogflow.readthedocs.io/
§ This work has been partially funded by the European Union’s Horizon 2020 research
and innovation program within the CPaaS.io project under Grant Agreement No.
723076
36

38. Thank you!
http://fiware.org
Follow @FIWARE on Twitter