Evolve13 presentation from OSGi Event Admin, Sling Discovery, Sling Job Handling to Granite Offloading and AEM Dam Ingestion

1. FROM EVENTING TO JOB PROCESSING
Carsten Ziegeler | Adobe Research Switzerland
1

2. 2
• RnD Team at Adobe Research Switzerland
• Co-founder Adobe Granite
• OSGi Core Platform and Enterprise Expert Groups
• Member of the ASF
• Current PMC Chair of Apache Sling
• Apache Sling, Felix, ACE
• Conference Speaker
• Technical Reviewer
• Article/Book Author
cziegele@adobe.com
@cziegeler
ABOUT

3. 3
OVERVIEW
Where to fit in the stack
Discovery
Offloading Workflow Distribution
DAM Ingestion
Sling
Granite
AEM
Job Distribution
Felix OSGi Event Admin
FromEventingtoJobProcessing

4. 4
OVERVIEW
Where to fit in the stack
Discovery
Offloading Workflow Distribution
DAM Ingestion
Sling
Granite
AEM
Job Distribution
Felix OSGi Event Admin
FromEventingtoJobProcessing

5. 5
OSGI EVENT ADMIN
Publish / Subscribe Model
OSGi Event Admin
Component
A
publish
deliver
Component
X
Component
Y

6. 6
• OSGi event is a data object with
• Topic (hierarchical namespace)
• Properties (key-value-pairs)
• Page Event
• Topic: com/day/cq/wcm/core/page
• Properties: path, change type (add/remove/edit) etc.
OSGI EVENT ADMIN
Publish / Subscribe Model

7. 7
• Publisher creates event object
• Sends event through EventAdmin service
• Either sync or async delivery
• Subscriber is an OSGi service (EventHandler)
• Service registration properties
• Interested topic(s)
• com/day/cq/wcm/core/*
• Additional filters (optional)
• (type="add“)
OSGI EVENT ADMIN
Publish / Subscribe Model

8. 8
• Immediate delivery to available subscribers
• No guarantee of delivery
• No distributed delivery
OSGI EVENT ADMIN
Publish / Subscribe Model

9. 9
• Immediate delivery to available subscribers
• No guarantee of delivery
• No distributed delivery
OSGI EVENT ADMIN
Publish / Subscribe Model
DiscoverySling Job Distribution

10. 10
OVERVIEW
Where to fit in the stack
Discovery
Offloading Workflow Distribution
DAM Ingestion
Sling
Granite
AEM
Job Distribution

11. 11
OVERVIEW
Where to fit in the stack
Discovery
Offloading Workflow Distribution
DAM Ingestion
Sling
Granite
AEM
Job Distribution
New in
5.6.1

12. 12
INSTALLATION SCENARIOS
Clustered CRX
Single
Instance
CRX
Instance
1
Instance
2
Instance
3

13. 13
Instance: Unique Id (Sling ID)
TOPOLOGIES I
Apache Sling Discovery
Clustered CRXCRX
ID : A ID : X ID : 42ID : 1
Single
Instance
Instance
1
Instance
2
Instance
3

14. 14
• Instance: Unique Id (Sling ID)
• Cluster: Unique Id and leader
TOPOLOGIES I
Apache Sling Discovery
Cluster 99Cluster 35
Clustered CRXCRX
ID : A ID : X ID : 42ID : 1
Single
Instance
Instance
1
Instance
2
Instance
3
Leader Leader

15. 15
• Topology: Set of clusters
TOPOLOGIES I
Apache Sling Discovery
Cluster 99Cluster 35
Clustered CRXCRX
ID : A ID : X ID : 42ID : 1
Single
Instance
Instance
1
Instance
2
Instance
3
Leader Leader
TopologyTopology
• Instance: Unique Id (Sling ID)
• Cluster: Unique Id and leader

16. 16
TOPOLOGIES I
Apache Sling Discovery
• Instance: Unique Id (Sling ID)
• Cluster: Unique Id and leader
• Topology: Set of clusters
Cluster 99
Clustered CRXCRX
ID : A ID : X ID : 42ID : 1
Single
Instance
Instance
1
Instance
2
Instance
3
Leader
Topology
Cluster 35
Leader

17. Cluster 99
17
• Instance
• Sling ID
• Optional: Name and description
• Belongs to a cluster
• Might be the cluster leader
• Additional properties which are distributed
• Extensible through own services (PropertyProvider)
• E.g. data center, region or enabled job topics
• Cluster
• Elects (stable) leader
• Stable instance ordering
• TopologyEventListener: receives events on topology changes
TOPOLOGIES II
Apache Sling Discovery
ID : 42
Instance
3
Topology
Leader

18. 18
OVERVIEW
Where to fit in the stack
Discovery
Offloading Workflow Distribution
DAM Ingestion
Sling
Granite
AEM
Job Distribution

19. 19
• Job : Guaranteed processing, exactly once
• Exactly one job consumer
• Started by client code, e.g. for replication, workflow...
• Job topic
• Payload is a serializable map
JOB HANDLING I
Apache Sling Job Handling

20. 20
• Sling Job Manager handles and distributes jobs
• Delivers job to a job consumer…
• …and waits for response
• Retry and failover
• Notification listeners (fail, retry, success)
JOB HANDLING I
Apache Sling Job Handling

21. 21
STARTING / PROCESSING A JOB I
Apache Sling Job Handling
public interface JobConsumer {
String PROPERTY_TOPICS = "job.topics";
enum JobResult {
OK,
FAILED,
CANCEL,
ASYNC
}
JobResult process(Job job);
}
public interface JobManager {
Job addJob(String topic, String optionalName, Map<String, Object> properties);
…
}
Starting a job
Processing a job
Note: Starting/processing of jobs through Event Admin is deprecated but still supported
New in
5.6.1

22. 22
@Component
@Service(value={JobConsumer.class})
@Property(name=JobConsumer.PROPERTY_TOPICS, value="org/apache/sling/jobs/backup")
public class BackupJobConsumer
implements JobConsumer {
@Override
public JobResult process(final Job job) {
// do backup
return JobResult.OK;
}
}
STARTING / PROCESSING A JOB II
Apache Sling Job Handling New in
5.6.1

23. 23
• New jobs are immediately persisted (resource tree / repository)
• Jobs are “pushed” to the processing instance
• Processing instances use different queues
• Associated with job topic(s)
• Main queue
• 0..n custom queues
• For example: replication agent queue or workflow queue
JOB HANDLING II
Apache Sling Job Handling

24. 24
• Queue is configurable
• Queue is started on demand in own thread
• And stopped if unused for some time
• Types
• Ordered queue (eg replication)
• Parallel queues: Plain and Topic Round Robin (eg workflow)
• Limit for parallel threads per queue
• Number of retries (-1 = endless)
• Retry delay
• Thread priority
JOB QUEUE
Apache Sling Job Handling

25. 25
• Job Manager Configuration = Main Queue Configuration
• Maximum parallel jobs (15)
• Retries (10)
• Retry Delay
• Eventing Thread Pool Configuration
• Used by all queues
• Pool size (35) = Maximum parallel jobs for a single instance
ADDITIONAL CONFIGURATIONS
Apache Sling Job Handling

26. 26
MONITORING – WEB CONSOLE
Apache Sling Job Handling

27. 27
MONITORING – WEB CONSOLE
Apache Sling Job Handling

28. 28
• Each instance determines enabled job topics
• Derived from Job Consumers (new API required)
• Can be whitelisted/blacklisted (in Job Consumer Manager)
• Announced through Topology
• Job Distribution depends on enabled job topics and queue type
• Potential set of instances derived from topology (enabled job topics)
• Ordered : processing on leader only, one job after the other
• Parallel: Round robin distribution on all potential instances
• Local cluster instances have preference
• Failover
• Instance crash: leader redistributes jobs to available instances
• Leader change taken into account
• On enabled job topics changes: potential redistribution
JOB DISTRIBUTION
Apache Sling Job Handling

29. 29
• Scalability in AEM:
• DAM Ingestion
• Non-Clustered installation requirement
• The term Offloading:
• In AEM used for all things job distribution and topology in clustered and non-
clustered installations, e.g. ‘Offloading Browser’
• More technically it’s ‘only’ a little add-on in Granite to Sling Job Distribution
for handling non-clustered installations
WHY OFFLOADING

30. 30
OVERVIEW
Where to fit in the stack
Discovery
Offloading Workflow Distribution
DAM Ingestion
Sling
Granite
AEM
Job Distribution

31. 31
OVERVIEW
Where to fit in the stack
Discovery
Offloading Workflow Distribution
DAM Ingestion
Sling
Granite
AEM
Job Distribution

32. 32
SLING JOB DISTRIBUTION
Topology
Instance
Sling ID: 1
Job Manager
Instance
Sling ID: 2
Job Manager
Instance
Sling ID: 3
Job Manager
Instance
Sling ID: 4
Job Manager
Job Consumer
Topic: A
Job Consumer
Topic: B
Job Consumer
Topic: C

33. 33
SLING JOB DISTRIBUTION
Instance
Sling ID: 1
Job Manager
Instance
Sling ID: 2
Job Manager
Instance
Sling ID: 3
Job Manager
Instance
Sling ID: 4
Job ManagerA
Job Consumer
Topic: A
Job Consumer
Topic: B
A:2
Job Consumer
Topic: CJob
Topology

34. 34
SLING JOB DISTRIBUTION
Instance
Sling ID: 1
Job Manager
Instance
Sling ID: 2
Job Manager
Instance
Sling ID: 3
Job Manager
Instance
Sling ID: 4
Job ManagerB
Job Consumer
Topic: A
Job Consumer
Topic: B
B:3
Job Consumer
Topic: CJob
Topology

35. 35
SLING JOB DISTRIBUTION
Instance
Sling ID: 1
Job Manager
Instance
Sling ID: 2
Job Manager
Instance
Sling ID: 3
Job Manager
Instance
Sling ID: 4
Job ManagerC
Job Consumer
Topic: A
Job Consumer
Topic: B
C:4
Job Consumer
Topic: CJob
?
Topology

36. 36
SLING JOB DISTRIBUTION
Instance
Sling ID: 1
Job Manager
Instance
Sling ID: 2
Job Manager
Instance
Sling ID: 3
Job Manager
Instance
Sling ID: 4
Job ManagerC
Job Consumer
Topic: A
Job Consumer
Topic: B
C:4
Job Consumer
Topic: CJob
Offloading
Framework
Topology

37. 37
• Detects offloading jobs
• Transport of job and job
payload between origin
and target instance
• Uses replication for the
transport
• No distribution of jobs
• No execution of jobs
OFFLOADING FRAMEWORK
Overview
Instance
Sling ID: 1
Job Manager
Offloading
Job Manager
C:4
Instance
Sling ID: 4
Job Manager
Job Consumer
Topic: C
Offloading
Job Manager
C:4
C
Job
Replication
1
2
3 5
4
6
7
Topology

38. 38
• Replication agents are created automatically
• Uses naming convention
• Needs manual adjustments (replication user)
OFFLOADING FRAMEWORK
Replication
Worker
Sling ID: 4
Author
Sling ID: 1
offloading_4
(outgoing agent)
offloading_reverse_4
(reverse agent)
offloading_outbox
(outbox agent)

39. 39
• Additional properties required
• Offloading job input
• Property: OffloadingJobProperties. INPUT_PAYLOAD (offloading.input.payload)
• Offloading job output
• Property: OffloadingJobProperties. OUTPUT_PAYLOAD
(offloading.output.payload)
• Takes comma separated list of paths
• Used to build the replication package
• Job sender needs to set these properties for offloading to work!
OFFLOADING FRAMEWORK
Job Payload

40. 40
• Configures Job Consumers
• Configures the topic white/black listing properties of each instance
• What jobs to execute on what instance
• Configures the distribution
• Configuration applies for both, clustered and non-clustered installations
OFFLOADING BROWSER
UI

41. 41
OFFLOADING BROWSER
UI

42. 42
OFFLOADING BROWSER
UI

43. 43
OFFLOADING BROWSER
UI

44. 44
OVERVIEW
Where to fit in the stack
Discovery
Offloading Workflow Distribution
DAM Ingestion
Sling
Granite
AEM
Job Distribution

45. 45
• New JobConsumer
• Class: WorkflowOffloadingJobConsumer
• Topic: com/adobe/granite/workflow/offloading
• Can launch new workflows
• Expects the workflow model on the job payload
• Expects the workflow payload on the job payload
• For use with clustered and non-clustered installations
WORKFLOW DISTRIBUTION

46. 46
@Component
@Service
@Property(name = JobConsumer.PROPERTY_TOPICS, value = WorkflowOffloadingJobConsumer.TOPIC)
public class WorkflowOffloadingJobConsumer implements JobConsumer {
public static final String TOPIC = "com/adobe/granite/workflow/offloading";
public static final String WORKFLOW_OFFLOADING_MODEL = "offloading.workflow.model";
public static final String WORKFLOW_OFFLOADING_PAYLOAD = "offloading.workflow.payload”;
public JobResult process(Job job) {
// read workflow model and payload from job payload
String modelPath= job.getProperty(WORKFLOW_OFFLOADING_MODEL , "");
String payloadPath= job.getProperty(WORKFLOW_OFFLOADING_PAYLOAD , "");
// get/create WorkflowSession, WorkflowModel and WorkflowData objects
WorkflowSession wfSession = ..;
WorkflowModel wfModel = ..;
WorkflowData wfData = ..;
// start the workflow
wfSession.startWorkflow(wfModel, wfData, metaData);
// all good
return JobResutl.OK;
}
}
WORKFLOW DISTRIBUTION
Job Consumer (Simplified)

47. 47
@Component
@Service
@Property(name = JobConsumer.PROPERTY_TOPICS, value = WorkflowOffloadingJobConsumer.TOPIC)
public class WorkflowOffloadingJobConsumer implements JobConsumer {
public static final String TOPIC = "com/adobe/granite/workflow/offloading";
public static final String WORKFLOW_OFFLOADING_MODEL = "offloading.workflow.model";
public static final String WORKFLOW_OFFLOADING_PAYLOAD = "offloading.workflow.payload”;
public JobResult process(Job job) {
// read workflow model and payload from job payload
String modelPath= job.getProperty(WORKFLOW_OFFLOADING_MODEL , "");
String payloadPath= job.getProperty(WORKFLOW_OFFLOADING_PAYLOAD , "");
// get/create WorkflowSession, WorkflowModel and WorkflowData objects
WorkflowSession wfSession = ..;
WorkflowModel wfModel = ..;
WorkflowData wfData = ..;
// start the workflow
wfSession.startWorkflow(wfModel, wfData, metaData);
// all good
return JobResutl.OK;
}
}
WORKFLOW DISTRIBUTION
Job Consumer (Simplified)
• Define the job topic

48. 48
@Component
@Service
@Property(name = JobConsumer.PROPERTY_TOPICS, value = WorkflowOffloadingJobConsumer.TOPIC)
public class WorkflowOffloadingJobConsumer implements JobConsumer {
public static final String TOPIC = "com/adobe/granite/workflow/offloading";
public static final String WORKFLOW_OFFLOADING_MODEL = "offloading.workflow.model";
public static final String WORKFLOW_OFFLOADING_PAYLOAD = "offloading.workflow.payload”;
public JobResult process(Job job) {
// read workflow model and payload from job payload
String modelPath= job.getProperty(WORKFLOW_OFFLOADING_MODEL , "");
String payloadPath= job.getProperty(WORKFLOW_OFFLOADING_PAYLOAD , "");
// get/create WorkflowSession, WorkflowModel and WorkflowData objects
WorkflowSession wfSession = ..;
WorkflowModel wfModel = ..;
WorkflowData wfData = ..;
// start the workflow
wfSession.startWorkflow(wfModel, wfData, metaData);
// all good
return JobResutl.OK;
}
}
WORKFLOW DISTRIBUTION
Job Consumer (Simplified)
• Create service component
• Must register with topic
• Implement new JobConsumer
interface

49. 49
@Component
@Service
@Property(name = JobConsumer.PROPERTY_TOPICS, value = WorkflowOffloadingJobConsumer.TOPIC)
public class WorkflowOffloadingJobConsumer implements JobConsumer {
public static final String TOPIC = "com/adobe/granite/workflow/offloading";
public static final String WORKFLOW_OFFLOADING_MODEL = "offloading.workflow.model";
public static final String WORKFLOW_OFFLOADING_PAYLOAD = "offloading.workflow.payload”;
public JobResult process(Job job) {
// read workflow model and payload from job payload
String modelPath= job.getProperty(WORKFLOW_OFFLOADING_MODEL , "");
String payloadPath= job.getProperty(WORKFLOW_OFFLOADING_PAYLOAD , "");
// get/create WorkflowSession, WorkflowModel and WorkflowData objects
WorkflowSession wfSession = ..;
WorkflowModel wfModel = ..;
WorkflowData wfData = ..;
// start the workflow
wfSession.startWorkflow(wfModel, wfData, metaData);
// all good
return JobResutl.OK;
}
}
WORKFLOW DISTRIBUTION
Job Consumer (Simplified)
• Access job properties (payload)
• Read workflow model and
payload from job properties

50. 50
@Component
@Service
@Property(name = JobConsumer.PROPERTY_TOPICS, value = WorkflowOffloadingJobConsumer.TOPIC)
public class WorkflowOffloadingJobConsumer implements JobConsumer {
public static final String TOPIC = "com/adobe/granite/workflow/offloading";
public static final String WORKFLOW_OFFLOADING_MODEL = "offloading.workflow.model";
public static final String WORKFLOW_OFFLOADING_PAYLOAD = "offloading.workflow.payload”;
public JobResult process(Job job) {
// read workflow model and payload from job payload
String modelPath= job.getProperty(WORKFLOW_OFFLOADING_MODEL , "");
String payloadPath= job.getProperty(WORKFLOW_OFFLOADING_PAYLOAD , "");
// get/create WorkflowSession, WorkflowModel and WorkflowData objects
WorkflowSession wfSession = ..;
WorkflowModel wfModel = ..;
WorkflowData wfData = ..;
// start the workflow
wfSession.startWorkflow(wfModel, wfData, metaData);
// all good
return JobResutl.OK;
}
}
WORKFLOW DISTRIBUTION
Job Consumer (Simplified)
• Workflow specific
• Use workflow API to start
workflow for the given model
and payload

51. 51
@Component
@Service
@Property(name = JobConsumer.PROPERTY_TOPICS, value = WorkflowOffloadingJobConsumer.TOPIC)
public class WorkflowOffloadingJobConsumer implements JobConsumer {
public static final String TOPIC = "com/adobe/granite/workflow/offloading";
public static final String WORKFLOW_OFFLOADING_MODEL = "offloading.workflow.model";
public static final String WORKFLOW_OFFLOADING_PAYLOAD = "offloading.workflow.payload”;
public JobResult process(Job job) {
// read workflow model and payload from job payload
String modelPath= job.getProperty(WORKFLOW_OFFLOADING_MODEL , "");
String payloadPath= job.getProperty(WORKFLOW_OFFLOADING_PAYLOAD , "");
// get/create WorkflowSession, WorkflowModel and WorkflowData objects
WorkflowSession wfSession = ..;
WorkflowModel wfModel = ..;
WorkflowData wfData = ..;
// start the workflow
wfSession.startWorkflow(wfModel, wfData, metaData);
// all good
return JobResutl.OK;
}
}
WORKFLOW DISTRIBUTION
Job Consumer (Simplified)
• Use JobResult enumeration to
report back the job status

52. 52
OVERVIEW
Where to fit in the stack
Discovery
Offloading Workflow Distribution
DAM Ingestion
Sling
Granite
AEM
Job Distribution

53. 53
• Default ingestion workflow: “DAM Update Asset”
• Load is put on the instance where the workflow is started, usually the author
• New ingestion workflow: “DAM Update Asset Offloading”
• Needs to be manually enabled by changing the workflow launcher
• New workflow model with a single step: AssetOffloadingProcess
• Uses WorkflowExternalProcess API
• Creates a new job on topic: com/adobe/granite/workflow/offloading
• Allows distributing the default ingestion workflow
• Load is put on the instance where the job is distributed to
• Can be used to distribute in clustered and non-clustered installations
DAM INGESTION

54. 54
@Component
@Service
public class AssetOffloadingProcess implements WorkflowExternalProcess {
@Reference
private JobManager jobManager;
private static final String TOPIC = "com/adobe/granite/workflow/offloading";
public Serializable execute(WorkItem workItem, WorkflowSession workflowSession, MetaDataMap metaDataMap){
Asset asset = ..;
String workflowModel = “/etc/workflow/models/dam/update_asset/jcr:content/model”;
String workflowPayload = “/content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg”;
ValueMap jobProperties = new ValueMapDecorator(new HashMap<String, Object>());
jobProperties.put(WORKFLOW_OFFLOADING_MODEL, workflowModel);
jobProperties.put(WORKFLOW_OFFLOADING_PAYLOAD, workflowPayload);
String offloadingInput = “/etc/workflow/models/dam/update_asset/jcr:content/model,
/content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg” ;
String offloadingOutput = “/etc/workflow/models/dam/update_asset/jcr:content/model,
/content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg” ;
jobProperties.put(OffloadingJobProperties.INPUT_PAYLOAD.propertyName(), offloadingInput);
jobProperties.put(OffloadingJobProperties.OUTPUT_PAYLOAD.propertyName(), offloadingOutput);
Job offloadingJob = jobManager.addJob(TOPIC, null, jobProperties);
return offloadingJob.getId();
}
DAM INGESTION
Create Job (from workflow step)

55. 55
@Component
@Service
public class AssetOffloadingProcess implements WorkflowExternalProcess {
@Reference
private JobManager jobManager;
private static final String TOPIC = "com/adobe/granite/workflow/offloading";
public Serializable execute(WorkItem workItem, WorkflowSession workflowSession, MetaDataMap metaDataMap){
Asset asset = ..;
String workflowModel = “/etc/workflow/models/dam/update_asset/jcr:content/model”;
String workflowPayload = “/content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg”;
ValueMap jobProperties = new ValueMapDecorator(new HashMap<String, Object>());
jobProperties.put(WORKFLOW_OFFLOADING_MODEL, workflowModel);
jobProperties.put(WORKFLOW_OFFLOADING_PAYLOAD, workflowPayload);
String offloadingInput = “/etc/workflow/models/dam/update_asset/jcr:content/model,
/content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg” ;
String offloadingOutput = “/etc/workflow/models/dam/update_asset/jcr:content/model,
/content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg” ;
jobProperties.put(OffloadingJobProperties.INPUT_PAYLOAD.propertyName(), offloadingInput);
jobProperties.put(OffloadingJobProperties.OUTPUT_PAYLOAD.propertyName(), offloadingOutput);
Job offloadingJob = jobManager.addJob(TOPIC, null, jobProperties);
return offloadingJob.getId();
}
DAM INGESTION
Create Job (from workflow step)
• Create service component
• Implement
WorkflowExternalProcess
interface
• Reference JobManager service

56. 56
@Component
@Service
public class AssetOffloadingProcess implements WorkflowExternalProcess {
@Reference
private JobManager jobManager;
private static final String TOPIC = "com/adobe/granite/workflow/offloading";
public Serializable execute(WorkItem workItem, WorkflowSession workflowSession, MetaDataMap metaDataMap){
Asset asset = ..;
String workflowModel = “/etc/workflow/models/dam/update_asset/jcr:content/model”;
String workflowPayload = “/content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg”;
ValueMap jobProperties = new ValueMapDecorator(new HashMap<String, Object>());
jobProperties.put(WORKFLOW_OFFLOADING_MODEL, workflowModel);
jobProperties.put(WORKFLOW_OFFLOADING_PAYLOAD, workflowPayload);
String offloadingInput = “/etc/workflow/models/dam/update_asset/jcr:content/model,
/content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg” ;
String offloadingOutput = “/etc/workflow/models/dam/update_asset/jcr:content/model,
/content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg” ;
jobProperties.put(OffloadingJobProperties.INPUT_PAYLOAD.propertyName(), offloadingInput);
jobProperties.put(OffloadingJobProperties.OUTPUT_PAYLOAD.propertyName(), offloadingOutput);
Job offloadingJob = jobManager.addJob(TOPIC, null, jobProperties);
return offloadingJob.getId();
}
DAM INGESTION
Create Job (from workflow step)
• DAM and Workflow specific
• Resolve to Asset
• Read model from meta data
• Read workflow payload from
Asset path

57. 57
public Serializable execute(WorkItem workItem, WorkflowSession workflowSession, MetaDataMap metaDataMap){
Asset asset = ..;
String workflowModel = “/etc/workflow/models/dam/update_asset/jcr:content/model”;
String workflowPayload = “/content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg”;
ValueMap jobProperties = new ValueMapDecorator(new HashMap<String, Object>());
jobProperties.put(WORKFLOW_OFFLOADING_MODEL, workflowModel);
jobProperties.put(WORKFLOW_OFFLOADING_PAYLOAD, workflowPayload);
String offloadingInput = “/etc/workflow/models/dam/update_asset/jcr:content/model,
/content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg” ;
String offloadingOutput = “/etc/workflow/models/dam/update_asset/jcr:content/model,
/content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg” ;
jobProperties.put(OffloadingJobProperties.INPUT_PAYLOAD.propertyName(), offloadingInput);
jobProperties.put(OffloadingJobProperties.OUTPUT_PAYLOAD.propertyName(), offloadingOutput);
Job offloadingJob = jobManager.addJob(TOPIC, null, jobProperties);
return offloadingJob.getId();
}
DAM INGESTION
Create Job (from workflow step)
• ValueMap for job properties
• Put model and payload on job
properties
• Used by the JobConsumer

58. 58
public Serializable execute(WorkItem workItem, WorkflowSession workflowSession, MetaDataMap metaDataMap){
Asset asset = ..;
String workflowModel = “/etc/workflow/models/dam/update_asset/jcr:content/model”;
String workflowPayload = “/content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg”;
ValueMap jobProperties = new ValueMapDecorator(new HashMap<String, Object>());
jobProperties.put(WORKFLOW_OFFLOADING_MODEL, workflowModel);
jobProperties.put(WORKFLOW_OFFLOADING_PAYLOAD, workflowPayload);
String offloadingInput = “/etc/workflow/models/dam/update_asset/jcr:content/model,
/content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg” ;
String offloadingOutput = “/etc/workflow/models/dam/update_asset/jcr:content/model,
/content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg” ;
jobProperties.put(OffloadingJobProperties.INPUT_PAYLOAD.propertyName(), offloadingInput);
jobProperties.put(OffloadingJobProperties.OUTPUT_PAYLOAD.propertyName(), offloadingOutput);
Job offloadingJob = jobManager.addJob(TOPIC, null, jobProperties);
return offloadingJob.getId();
}
DAM INGESTION
Create Job (from workflow step)
• Build offloading payload properties
• Comma separated list of paths
• Put them on the job payload as
well
• Only used for non-clustered
distribution

59. 59
public Serializable execute(WorkItem workItem, WorkflowSession workflowSession, MetaDataMap metaDataMap){
Asset asset = ..;
String workflowModel = “/etc/workflow/models/dam/update_asset/jcr:content/model”;
String workflowPayload = “/content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg”;
ValueMap jobProperties = new ValueMapDecorator(new HashMap<String, Object>());
jobProperties.put(WORKFLOW_OFFLOADING_MODEL, workflowModel);
jobProperties.put(WORKFLOW_OFFLOADING_PAYLOAD, workflowPayload);
String offloadingInput = “/etc/workflow/models/dam/update_asset/jcr:content/model,
/content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg” ;
String offloadingOutput = “/etc/workflow/models/dam/update_asset/jcr:content/model,
/content/dam/geometrixx-outdoors/articles/downhill-ski-conditioning.jpg” ;
jobProperties.put(OffloadingJobProperties.INPUT_PAYLOAD.propertyName(), offloadingInput);
jobProperties.put(OffloadingJobProperties.OUTPUT_PAYLOAD.propertyName(), offloadingOutput);
Job offloadingJob = jobManager.addJob(TOPIC, null, jobProperties);
return offloadingJob.getId();
}
DAM INGESTION
Create Job (from workflow step)
• Create job using JobManager
service
• Use topic from job consumer
• Put job payload properties
• Return the jobId as the workflow
process id (workflow specific)

60. 60
@Component
@Service
public class AssetOffloadingProcess implements WorkflowExternalProcess {
@Reference
private JobManager jobManager;
private static final String TOPIC = "com/adobe/granite/workflow/offloading";
public Serializable execute(WorkItem workItem, WorkflowSession workflowSession, MetaDataMap metaDataMap){
….
}
public boolean hasFinished(Serializable externalProcessId, ..){
// returns null, if job is finished
Job offloadingJob = jobManager.getJobById((String) externalProcessId);
return offloadingJob == null;
}
}
DAM INGESTION
Create Job (from workflow step)
• Workflow API specific callback
• Process id = jobId, from
execute()
• Query job by jobId
• Workflow step finished when job
is finished

61. 61
• Choose a job topic
• Create JobConsumer component and register with topic chosen
• To create a new job use new JobManager.addJob() API with the topic chosen
and the job payload
• Add offloading payload to job payload
• Bundle and deploy JobConsumer on topology instances
• Enable/Disable the new topic on the instances, using Offloading Browser
DEVELOPMENT - RECIPE

62. 62
TAKE AWAY
Discovery
Offloading Workflow Distribution
DAM Ingestion
Sling
Granite
AEM
Job Distribution

63. 63
TAKE AWAY
Discovery
Offloading Workflow Distribution
DAM Ingestion
Sling
Granite
AEM
Job Distribution
org.apache.sling.discovery.api
• Discovers topology
• Cross-Cluster detection
• Foundation for job distribution

64. 64
TAKE AWAY
Discovery
Offloading Workflow Distribution
DAM Ingestion
Sling
Granite
AEM
Job Distribution
org.apache.sling.event
• Uses Sling Discovery
• New JobConsumer API and job topics
• New JobManager API for creating new distributed jobs
• Distributes jobs based on available job topics and job queue
configuration.
• Distributes in the whole topology, including clustered and non-
clustered instances
• Can execute cluster local jobs only

65. 65
TAKE AWAY
Discovery
Offloading Workflow Distribution
DAM Ingestion
Sling
Granite
AEM
Job Distribution
com.adobe.granite.offloading.core
• Builds on top of Sling Distributed Jobs
• Does not perform distribution
• Detects jobs distributed to non-clustered instances
• Transports the jobs and payload to non-clustered
instances
• Uses replication for transport
• Does not execute jobs

66. 66
TAKE AWAY
Discovery
Offloading Workflow Distribution
DAM Ingestion
Sling
Granite
AEM
Job Distribution
com.adobe.granite.workflow.core
• Defines a job consumer to distribute the execution
of whole workflows
• Defines topic com/adobe/granite/workflow/
offloading
• Implements WorkflowOffloadingJobConsumer
component
• Supports clustered and non clustered installations

67. 67
TAKE AWAY
Discovery
Offloading Workflow Distribution
DAM Ingestion
Sling
Granite
AEM
Job Distribution
cq-dam-core
• Makes use of com/adobe/granite/workflow/
offloading topic from Workflow Distribution
• New workflow step (external step) that creates a
new job on topic com/adobe/granite/workflow/
offloading
• New “DAM Update Asset Offloading” workflow
• Supports clustered and non clustered
configurations

68. 68
TAKE AWAY
Potential Future
Discovery
Offloading Workflow Distribution
DAM Ingestion
Sling
Granite
AEM
Job Distribution
Felix OSGi Event Admin
FromEventingtoJobProcessing

69. 69
TAKE AWAY
Potential Future
Discovery
Offloading Workflow Distribution
DAM Ingestion
Sling
Granite
AEM
Job Distribution
Felix OSGi Event Admin
FromEventingtoJobProcessing
New OSGi Specification
• Distributed Eventing
• Cloud Computing

70. 70
TAKE AWAY
Potential Future
Discovery
Offloading Workflow Distribution
DAM Ingestion
Sling
Granite
AEM
Job Distribution
Felix OSGi Event Admin
FromEventingtoJobProcessing
Job Distribution
• Improved load balancing
• Pull based distribution

71. 71
TAKE AWAY
Potential Future
Discovery
Offloading Workflow Distribution
DAM Ingestion
Sling
Granite
AEM
Job Distribution
Felix OSGi Event Admin
FromEventingtoJobProcessing
Offloading
• As part of job distribution
• Even simpler setup

72. 72
TAKE AWAY
Questions?
Discovery
Offloading Workflow Distribution
DAM Ingestion
Sling
Granite
AEM
Job Distribution
Felix OSGi Event Admin
FromEventingtoJobProcessing

73. 73