Talk at Java User Group Switzerland in Zürich

1. Spring Batch
Lessons Learned out of a real life banking system.
Michael Beer
Senior Consultant
Raffael Schmid
Consultant
BASEL
1
BERN
BRUGG
LAUSANNE
ZUERICH
DUESSELDORF
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013
FRANKFURT A.M.
FREIBURG I.BR.
HAMBURG
MUNICH
STUTTGART
VIENNA

2. About us
Michael Beer
Senior Consultant
 Trivadis since 2001
 Trivadis since 2010
 design and development of
web based applications
 design and development of
web based applications
 part of the Trivadis APM
team
2
Raffael Schmid
Consultant
 interested in performance
related things on the JVM
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013

3. AGENDA
Initial
position
3
Requirements
and Topics
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013
Lessons
learned
Conclusion

4. Initial position
Initial
position
4
Requirements
and Topics
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013
Lessons
learned
Conclusion

5. Initial and target system context
Web Interface
 Automatic load out of thirdparty
systems
Rule Engine
 multiple times per day
 export / import as CSV file
 Manual load by user
common usage
 at irregular time intervals
 Migration load due to thirdparty
system decommissioning
Thirdparty system
File Interface
Thirdparty system
5
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013
 run once
 different volumes

6. Why we used Spring Batch?
 It was already part of the technology stack within the customers
environment.
 There were no real (free) alternatives at that time (or at least we didn‘t
know any).
 Exceptional permit was needed for Spring Integration because it was
shipped with Spring Batch Admin.
6
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013

7. Terminology of Spring Batch jobs
*
Job
Step
*
Job Instance
*
Job Parameter
*
*
Job Execution
*
Job Execution
Context
7
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013
Step Execution
Step Execution
Context
*
Tasklet

8. Requirements and Topics
Initial
position
8
Requirements
and Topics
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013
Lessons
learned
Conclusion

9. Requirement: Performance
Performance
Reprocess
failed items
Summary mail
Gather detailed
job information
Trigger jobs
Control load
Deactivate jobs
Inter-job
dependencies
 parallelized on multiple threads
 order of execution matters
 number of threads can be changed before or after job execution
9
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013

10. Requirement: Reprocess failed items
Performance
Reprocess
failed items
Summary mail
Gather detailed
job information
Trigger jobs
Control load
Deactivate jobs
Inter-job
dependencies
 rerunning a job processes failed items (only)
 process state of item therefore needs to be maintained
10
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013

11. Requirement: Summary mail
Performance
Reprocess
failed items
Summary mail
Gather detailed
job information
Trigger jobs
Control load
Deactivate jobs
Inter-job
dependencies
 write a detailed execution log that contains
 Exceptions
 messages out of the Rule Engine
 aggregate to summary
 Number of errors, warnings, etc.
 worst status level
 send summary to submitter and operator
11
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013

12. Requirement: Gather detailed job information
Performance
Reprocess
failed items
Summary mail
Gather detailed
job information
Trigger jobs
Control load
Deactivate jobs
Inter-job
dependencies
 rerun failed single item in trace mode
 collect diagnostic information
 e.g. out of the Rule Engine
12
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013

13. Requirement: Trigger jobs
Performance
Reprocess
failed items
Summary mail
Gather detailed
job information
Trigger jobs
Control load
Deactivate jobs
Inter-job
dependencies
 periodically
 either fixed delay or fixed rate
 on event
 e.g. new data arrived in database
13
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013

14. Requirement: Control load
Performance
Reprocess
failed items
Summary mail
Gather detailed
job information
Trigger jobs
Control load
Deactivate jobs
Inter-job
dependencies
 prevent too many jobs in parallel
 conditions might prevent job execution
 requests should prevent system shutdown
14
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013

15. Requirement: Deactivate jobs
Performance
Reprocess
failed items
Summary mail
Gather detailed
job information
Trigger jobs
Control load
Deactivate jobs
Inter-job
dependencies
 deactivate job execution
 set job execution on hold
15
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013

16. Requirement: Inter-job dependencies
Performance
Reprocess
failed items
Summary mail
Gather detailed
job information
Trigger jobs
Control load
Deactivate jobs
Inter-job
dependencies
 finished jobs might trigger dependencies
16
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013

17. Requirements grouped into five different topics
Performance
Partitioning
Error
Reprocess
failed items
Handling
Trigger jobs
Control load
Summary mail
Monitoring
Job Control
Deactivate jobs
Sourcing
17
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013
Gather detailed
& Tracing
job information
Inter-job
dependencies

18. Lessons learned
Initial
position
18
Requirements
and Topics
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013
Lessons
learned
Conclusion

19. Topics we will cover
Monitoring
& Tracing
depends on
Partitioning
depends on
supervising
Sourcing
firing trigger
depends on
Error
Handling
19
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013
Job Control

20. Sourcing
Monitoring
& Tracing
depends on
Partitioning
depends on
supervising
Sourcing
firing trigger
depends on
Error
Handling
20
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013
Job Control

21. Sourcing
Load data into staging table
CAT_ID
310
310
311
|
|
|
|
PERSON_ID
2908
1608
1608
|
|
|
|
AMOUNT
200000
100000
100000
|
|
|
|
CURRENCY
USD
CHF
CHF
|
|
|
|
VALID_FORM
03.12.2013
03.12.2013
03.12.2013
|
|
|
|
LAST_CHANGE|
02.02.2013 |
02.01.2013 |
02.01.2013 |
PARTITION
LINE
TYPE
STATE
FIELD01
FIELD02
FIELD03
FIELD04
PT_000122
171
D
02
441
1804
400000
EUR
PT_000122
172
D
02
441
1002
221000
EUR
PT_000123
1
H
01
CAT_ID
PERSON_ID
AMOUNT
CURRENCY
PT_000123
2
D
01
310
2908
200000
USD
PT_000123
3
D
01
310
1608
100000
CHF
PT_000123
4
D
01
311
1608
100000
CHF
21
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013

22. Allows partitioning, single record execution
Sourcing
PARTITION
LINE
TYPE
STATE
FIELD01
FIELD02
FIELD03
FIELD04
PT_000123
1
H
01
CAT_ID
PERSON_ID
AMOUNT
CURRENCY
PT_000123
2
D
01
310
2908
200000
USD
PT_000123
3
D
01
310
1410
100000
CHF
 Partitioning out of the box
SELECT * FROM LOAD
WHERE PARTITION =‘PT_000123’
AND LINE BETWEEN 0 AND 99;
 Reexcution of failed records
SELECT * FROM LOAD
WHERE PARTITION =‘PT_000123’
AND STATE = ‘01’
 Execution of single record
SELECT * FROM LOAD
WHERE PARTITION =‘PT_000123’
AND LINE = 8
22
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013

23. Job Control
Monitoring
& Tracing
depends on
Partitioning
depends on
supervising
Sourcing
firing trigger
depends on
Error
Handling
23
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013
Job Control

24. Job Control
Simplest way of triggering a job
0 0/5 * * * ?
Trigger
job-requests
Spring Batch
(SimpleJobLauncher)
Request Factory
 Define polling channel adapter (CronTrigger)
<int:inbound-channel-adapter method="create" channel="job-requests"
ref="requestFactory">
<int:poller cron="0 0/5 * * * ?" />
</int:inbound-channel-adapter>
 Or even simpler (PeriodicTrigger)
<int:inbound-channel-adapter method="create" channel="job-requests"
ref="requestFactory">
<int:poller fixed-rate="300000"/>
</int:inbound-channel-adapter>
24
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013

25. Job Control
Persist job launch requests into database
Data import job
(database)
job queue
Periodic jobs
*JobQueueMessageHandler
0 0/5 * * * ?
persist
*JobLaunchingMessageHandler
job launch
request
jobrequests
Guard
verification
update status
save job dependencies
25
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013
*JobLauncher

26. Partitioning
Monitoring
& Tracing
depends on
Partitioning
depends on
supervising
Sourcing
firing trigger
depends on
Error
Handling
26
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013
Job Control

27. Partitioning
Performance
Type
Description
Multi-threaded Step
Local
A step is multithreaded (TaskExecutor)
Parallel Steps
Local
Executes steps in parallel using multithreading
Partitioning Step
Local
Remote
Partitions data and splits up processing
Remote Chunking
27
Local/Remote
Remote
Distributed chunk processing to remote nodes
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013

28. Partitioning
Partitioning overview
Job
Step
Master
Step
Slave
Slave
Slave
Slave
Slave
28
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013

29. Partitioning
Partitioning detail
StepScopes
Execution
Context [0]
Execution
Context [1]
29
Slave [2]
Execution
Context [3]
Slave [3]
Execution
Context [4]
rowId % 5
Slave [1]
Execution
Context [2]
Master
Slave [0]
Slave [4]
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013

30. Partitioning
Characteristics of data
There might be dependencies between records. The order of execution
matters at some point.
CAT_ID
310
310
311
312
313
310
390
310
|
|
|
|
|
|
|
|
|
PERSON_ID
2908
1608
1608
1608
1608
1410
1108
2908
|
|
|
|
|
|
|
|
|
AMOUNT
200000
100000
100000
100000
100000
100000
100000
500000
| CURRENCY
| Insert
USD
| CHF
| CHF
| CHF
| CHF
| CHF
| CHF
|Update
USD
Read
30
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013
|
|
|
|
|
|
|
|
|
VALID_FORM
03.12.2013
03.12.2013
03.12.2013
03.12.2013
03.12.2013
03.12.2013
03.12.2013
04.12.2013
Write
Process / Write
|
|
|
|
|
|
|
|
|
LAST_CHANGE|
02.02.2013 |
02.01.2013 |
02.01.2013 |
02.01.2013 |
02.01.2013 |
02.01.2013 |
02.01.2013 |
03.02.2013 |

31. Partitioning detail – Spring Batch Admin
31
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013
Partitioning

32. Performance – Read / Write / Process
Partitioning
Begin Step
Open
Transaction
For each item
For each item
ItemReader
Item read()
ItemProcessor
Process(Item)
RuntimeException
RuntimeException
ItemWriter
Write(List<Item>)
Commit
Transaction
RuntimeException
Rollback
Transaction
Finish Step
32
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013

33. Partitioning
Performance
No error
Error on each item
• ~ 22 sec
• 0 Rollbacks
•
•
•
•
33
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013
~ 50 sec
1‘100 Rollbacks
5‘500 Filter
1‘000 Write Skips

34. Error Handling
Monitoring
& Tracing
depends on
Partitioning
depends on
supervising
Sourcing
firing trigger
depends on
Error
Handling
34
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013
Job Control

35. Error
Handling
Restartability
regular start
restart
Job 1
Instance 1
Execution 1
35
read
new/error
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013
read
new/error
Job 1
Instance 1
Execution 2

36. Error
Handling
Restartability configuration
initial (failed) execution
Job
restarted execution
serialized
context
Step
initialization
Step
process
Step
initialization
Execution
Context
Step
finalize
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013
Step
process
Step
finalize
cleanup
36
Job

37. Error
Handling
Item based error handling
Feature
What?
Where?
Skip
For nonfatal
exceptions
Keeps processing for an
incorrect item
Chunk-oriented step
Retry
37
When?
For transient
exceptions
Makes new attemps on an
operation
Chunk-oriented step,
application code
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013

38. Monitoring
Monitoring
& Tracing
depends on
Partitioning
depends on
supervising
Sourcing
firing trigger
depends on
Error
Handling
38
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013
Job Control

39. Monitoring – Spring Batch Admin
39
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013
Monitoring &
Tracing

40. Monitoring – logging to a database
Step
Monitoring &
Tracing
Listener
Before
Read
OnSkip
[WARN]
Process
LoggerService
@Transactional(propagation =
Propagation.REQUIRES_NEW)
OnError
[ERROR]
Write
After
[INFO]
2 calls on error
40
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013

41. Monitoring &
Tracing
Monitoring - Mail
Job
Step
ReadProcessWrite
FAILED
OK Mail
41
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013
Error Mail

42. Monitoring &
Tracing
Tracing – Explain plan
explicit job start
Operator
Step
ReadProcessWrite
Partitioner
Read
Process
Writer
1 Record
1 Record
Debug mode
does nothing
Step
…
42
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013

43. Monitoring &
Tracing
Monitoring / Tracing summary
Mail
OK/ERROR
Mail
ERROR
UI
Log
UI
Explain plan
User
Spring Batch
Admin
43
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013
Operator

44. Conclusion
Initial
position
44
Requirements
and Topics
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013
Lessons
learned
Conclusion

45. Challenges – Error Handling
 Fire and forget
 Commit until first error
- Skip
- Retry
 Depending on the characteristic of data
 Rollback
- Whole job
- Partition
- Chunk
 Job flow
 Deciders
 Operator takes care about the errors
45
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013

46. Why we still use Spring Batch
Out of 155 SE tests in the JSR-352 TCK, Spring Batch 3.0 (milestone 1)
passes 70.
The terminology stays more or less the same: Job, Step, Chunk, Item,
ItemProcessor, JobInstance, JobExecution.
Spring Batch
JSR 352
Tasklet
Batchlet
ItemReader / ItemStream
ItemReader
JSR-352‘s ItemReader includes Spring Batchs
ItemStream capabilities
ItemWriter / ItemStream
ItemWriter
JSR-352‘s ItemReader includes Spring Batchs
ItemStream capabilities
JobExecutionListener
JobListener
StepExecutionListener
StepListener
46
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013
Comments

47. Suitable for different applications
Spring Batch supports implementing reliable batch jobs with small effort.
In few cases even no Java code is necessary.
It’s very easy to customize existing parts at your needs. Different level of
customization is possible:
configuration
47
partial
implementation
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013
job control

48. Questions and answers ...
Michael Beer
michael.beer@trivadis.com
+41 58 459 51 90
Raffael Schmid
raffael.schmid@trivadis.com
+41 58 459 52 34
BASEL
48
BERN
BRUGG
LAUSANNE
ZUERICH
DUESSELDORF
2013 © Trivadis
Lessons Learned out of a real life banking system
December 3rd, 2013
FRANKFURT A.M.
FREIBURG I.BR.
HAMBURG
MUNICH
STUTTGART
VIENNA