YouTube: https://www.youtube.com/watch?v=Ev3pssBlVI4&index=6&list=PLnKL6-WWWE_VtIMfNLW3N3RGuCUcQkDMl JBoss EAP is a heavy-duty Java application server. It's a perfect match for huge banking applications, but also perfect impedance mismatch with DevOps processes. This is the story how we persuaded the Behemoth to work with Chef automation. The goal was to manage multitude of our JBoss clusters and be able to rebuild the infrastructure on-demand.

1. The Song of JBoss and ChefThe Song of JBoss and Chef
DevOpsDays Warsaw 2016
Tomasz Torcz
<tomasz.torcz@alior.pl>

2. Agenda
- who am I?
- what is JBoss? What is Chef?
- why is it complicated?
- pro & cons

3. Who am I?

4. Banks. Banks are special

5. JBoss?
The JBoss Enterprise Application Platform is a
Java EE-based application server runtime
platform used for building, deploying, and hosting
highly-transactional Java applications and services.

6. Chef
Configuration management tool written in Ruby
and Erlang. It uses a pure-Ruby, domain-specific
language for writing system configuration
"recipes".

7. The Song ofThe Song of
JBoss and ChefJBoss and Chef

8. The Song ofThe Song of
JBoss and ChefJBoss and Chef

9. Why did we care?
- already automated lower levels of stack
Paweł Bartusch @ https://youtu.be/U50w_ZoQygA
- needed for full CI/CD pipeline
Waldemar Piszczewiat @ https://youtu.be/9-X8qytNznA
- crucial to have accounting
git blame

10. Architecture

11. Architecture
vroom!
vrooom!

12. Workflow
- multiple teams, separation of duties
- internal developers
- external vendors

13. How do you configure JBoss?

14. 1. Web Console

15. How do you configure JBoss?
1. Web Console
2. Command Line Interface

16. How do you configure JBoss?
1. Web Console
2. Command Line Interface
3. XML files (domain.xml, host*.xml, etc.)

17. How do you configure JBoss?
1. Web Console
2. Command Line Interface
3. XML files (domain.xml, host*.xml, etc.)

18. JBoss CLI
"logging": {
…
"root-logger": {
"handlers": "["File_rotating"]",
"level": "INFO"
~/bin/jboss-cli.sh --controller=10.14.x.y:9999
--connect --commands='/profile=full-
ha/subsystem=logging/root-logger=ROOT:write-
attribute(name=handlers, value=["File_rotating"])'

19. Reading the JBoss output
{
"outcome" => "failed",
"failure-description" => {"host-failure-
descriptions" => {"vhost02" => "JBAS014749:
Operation handler failed: null"}},
"rolled-back" => true
}

20. Reading the JBoss output
eval(JBoss’ output) … success!
=> {"outcome"=>"failed",
"failure-description"=>{
"host-failure-descriptions"=>{
"vhost02"=>"JBAS014749: Operation handler
failed: null"
}
},
"rolled-back"=>true}

21. Reading the JBoss output
"extension" => {
"org.jboss.as.webservices" => undefined, …
NameError: undefined local variable or
method `undefined' for main:Object

22. Reading the JBoss output
"extension" => {
"org.jboss.as.webservices" => undefined, …
NameError: undefined local variable or
method `undefined' for main:Object
sed 's/undefined/{}/g' … success!

23. Reading the JBoss output
"content" => [{"hash" => bytes { 0x87, 0x17,
0x0a, 0xd7, 0xff, 0x72 }}]
SyntaxError: (eval):1: syntax error,
unexpected =>, expecting end-of-input
"content" => [{"hash" => bytes { 0x87, 0x...
^
sed 's/{"hash" => bytes {//g' … success!

24. JBoss output massaging function
c.gsub!('=> undefined', '=> {}')
c.gsub!(/([[:blank:]]*)(-?b[0-9]+)Lb(,?)/,
'123') # 1234L -> "1234"
c.gsub!('expression "', '"expression ')
c.gsub!('{"hash" => bytes {', '') # _ 'hash =>
bytes' to array
c.gsub!(/([ t]+)}}(],?$)/,'12') # /
c.gsub!(/'/, '' + "'") # don't ask :/
c.gsub!(/([[:blank:]]*()(".
+"[[:blank:]]=>[[:blank:]]".+"))(,?)/, '23')
c.gsub!(/(.+[)((.+))(].+)/,'123')
c.gsub!(/"/, ''')

25. JBoss output massaging function
commit 3b54e51a0fc9149330e32c42fd784ad3b20850ad
Author: Bartosz Trudnowski <bartosz.trudnowski@ge.com>
Date: Tue Feb 23 10:00:58 2016 +0100
- c.gsub!(/([[:blank:]]()(".+"[[:blank:]]=>[[:blank:]]".+"))(,?)/, '23')
+ c.gsub!(/([[:blank:]]*()(".+"[[:blank:]]=>[[:blank:]]".+"))(,?)/, '23')

26. JBoss output format

27. Other JBoss annoyances
- config keywords incoherence
- punctuation frenzy

28. The good and the bad
What lessons did we learn?

29. PROD cluster

30. PROD cluster 1 hour later

31. Handover
bph_jboss_srv bph_jboss_ml bph_jboss_app

32. Handover – number of commits
bph_jboss_srv bph_jboss_ml bph_jboss_app
80 Tomasz Torcz 254 Tomasz Torcz 50 Tomasz Torcz
9 Waldemar Piszczewiat 67 Bartosz Trudnowski 10 Bartosz Trudnowski
4 Krzysztof Kowalczyk 17 Waldemar Piszczewiat 7 Waldemar Piszczewiat
4 Bartosz Trudnowski 4 Bartosz Dempc 6 Krzysztof Kowalczyk
3 Waldemar Buczak 1 Waldemar Buczak
1 Grzegorz Prusik 1 Grzegorz Krysik

33. Data bags – number of commits
dev test prod
218 Waldemar Piszczewiat 273 Waldemar Piszczewiat 40 Bartosz Dempc
53 Tomasz Torcz 194 Bartosz Dempc 17 Paweł Rodziewicz
8 Bartosz Trudnowski 107 Grzegorz Prusik 13 Grzegorz Prusik
5 Technical CICD account 49 Rafał Markowski 6 Michał Szor
3 Waldemar Buczak 43 Grzegorz Krysik 6 Waldemar Piszczewiat
2 Bartosz Dempc 24 Grzegorz Słaby 5 Grzegorz Słaby
… …

34. Reading the Chef output
* execute[Updating SG01 property
spring.profiles.active to
cbpm,sit,mockReport,localStorage] action run
(skipped due to not_if)

35. Managable config
- many similar, but maybe different clusters?
- json-diff
- testing, moving configuration
- copy&paste
- easy to revert

36. Diverging templates
cicd-dev-apps/dev/ … /GsbTytan.properties.erb
cicd-uat-apps/dk1/ … /GsbTytan.properties.erb
cicd-uat-apps/dk2/ … /GsbTytan.properties.erb
cicd-uat-apps/tg/ … /GsbTytan.properties.erb
cicd-sit-apps/sit/ … /GsbTytan.properties.erb
cicd-sit-apps/sitstab/ … /GsbTytan.properties.erb
cicd-prod-apps/prod/ … /GsbTytan.properties.erb
- moved to external git repos

37. The Song of JBoss and ChefThe Song of JBoss and Chef
DevOpsDays Warsaw 2016
Tomasz Torcz
<tomasz.torcz@alior.pl>