Weitere ähnliche Inhalte Ähnlich wie DevOps / Agile Tools Seminar 2013 (20) Kürzlich hochgeladen (20) DevOps / Agile Tools Seminar 20131. © All rights reserved
Want to use slides from
this presentation?
Ask for permission:
ethanram@gmail.com
Agile Tools Seminar
- Technical Introduction Ethan Ram / 16 Oct. 2013
v1.2
2. Agenda: Agile Tools Seminar
• Session presents the automation model
developed in Videobet, the tools used in the
R&D, QA and operations:
–
–
–
–
–
–
–
Issue mgmt.: JIRA/Greenhopper
Build system and repository: Maven & Nexus
Build server: QuickBuild
Code quality: Sonar
Continuous Integration: Selenium Grid
Crash dump analysis: Socorro
Database versioning: Flyway DB
• Open discussion
• Refreshments & tour in Videobet office
Videobet, Paldiski Mnt. 29, Tallinn
iPad Mini – Draw on Exit
10/22/13
© All rights reserved
2
3. Videobet
• A Playtech division
• Innovative Gaming
Platform Provider
– Established 9 years ago
– 150 employees in Tallinn
and Kiev
– Operate in 9 countries
– 60% of UK market
• R&D
– Front-end teams (C++)
– Server teams (Java)
– Game development teams
10/22/13
© All rights reserved
3
4. Ethan Ram - Host
• Chief architect of Videobet
• Leading Videobet’s R&D and OPs
teams Agile Injection effort
• As VP R&D of GameGround
transformed the R&D to run in
Kanban in a Continuous
Deployment environment
• Hacker in soul and a Windows
Internals hobbyist
Find me here:
•ethnram@gmail.com
•LinkedIn
•The Null Terminator blog
10/22/13
© All rights reserved
4
5. “We forgot to implement this feature…”
“Didn’t you get that urgent build request email?!”
“But there was a much more urgent issue to work on”
“We lost a day in QA because the build was total crap”
“It will take me a half a day to switch to that branch”
10/22/13
© All rights reserved
5
6. Automate it!
Streamlined
Operations
“We forgot to implement this feature…”
“Didn’t you get that urgent build request email?!”
“But there was a much more urgent issue to work on”
“We lost a day in QA because the build was total crap”
“It will take me a half a day to switch to that branch”
10/22/13
© All rights reserved
Transparency in all operations
Prioritize often
Enable teams collaboration
Simplify dev. environment
Reduce QA cost
Improve product quality
Go Agile; go green!
6
9. Agile Issue Tracking and Mgmt: JIRA
Tips
•JIRA Agile is good!
•Latest versions are fast and
intuitive
– No more than 1-2 hours a
week to manage it all
•A few plugins used for
missing functionality
•Keep all R&D in one JIRA
project
– Team backlog based on a
team label
•If you cannot model your
process into JIRA better ask:
“What do I need to change in
my process”
•Missing a good
resource/project timeline tool
10/22/13
© All rights reserved
9
10. JIRA Agile – Scrum Board
10/22/13
© All rights reserved
10
11. Scrum Board – Sprint Planning
xxx
aBC
aBsddlfhasldjhlshC
10/22/13
© All rights reserved
11
16. Continuous Integration - Maven
• Mavenize it all
– Product: Java, C++, scripts, game resource
compilations etc.
– Embedded Operating System images
– Customer configurations and packages
• Nexus artifact repository
– Build output is here! Snapshots, releases
– Installer downloads directly from Nexus repo.
– New artifacts pushed to 2nd Nexus in remote site
• Build promotions
– Snapshot builds avail to Scrum team immediately
(Auto-deleted after a week)
– Promote builds to QA and customer (“gold”)
10/22/13
© All rights reserved
16
17. Continuous Integration – Build Server
•
•
•
•
•
Build on every commit – fail build on errors
Run unit testing – fail build on assertions
Run integration tests – alert testers
Code quality metrics – produced weekly
Distributed build agents – some in remote
sites
• Daily snapshot installed and run in QA
– Nightly scripts run on 60+ terminals
• Integration testing – run every hour
– Deploy and upgrade servers and database
– Run Selenium script on management console
– Run clients simulators
10/22/13
© All rights reserved
17
18. CI Server - QuickBuild
Build and test queue
Latest builds
promoted to QA
Alert when Continuous
Integration fails
10/22/13
© All rights reserved
18
19. Code Quality Metrics - Sonar
•
•
•
•
•
Percent of code covered by unit tests, per module
Common developer mistakes: leaks, uninitialized vars, unsafe error handling etc.
Code complexity measures based on static code analysis
Drill-down to line of code
Supports continuous improvement by developers: Java and C++
10/22/13
© All rights reserved
19
21. Integration Testing – The basics
• Initial goal: main data flows should
remain working at all times to allow
QA to run full regression test suite
• Long term goal: replace most
regression testing
• Tests are run hourly on active
branches - snapshot build
• Deployment on a set of virtual
machines
10/22/13
© All rights reserved
“Keeping product
stable is more
important than
developing new
features”
21
22. Integration Testing – DEMO
• Using Selenium scripts to test admin console and check
that a test-setup of servers and clients is working
10/22/13
© All rights reserved
22
23. Here’s a New Bug CI Found Today!
Email: CI Alerts on failed test
Tester validates it’s a bug
Bug was resolved 2 hrs after it was introduced
CI is back to “green”
10/22/13
© All rights reserved
24. Integration Testing – Solution
• Infrastructure
–
–
–
–
–
Virtual Machines deployment scripts
DB deployment scripts
Server install
Clients install
Client simulators (initially created used
for load-testing sessions)
– Various libs allowing triggering of
hardware events and reading screen
output – with activated over http
• Tests
– Selenium grid – for web-testing
• nUnit tests running on build server
10/22/13
© All rights reserved
24
25. Integration Testing – Tips
• Find the natural topic owners: the Scrum testers
– Lots a scripting is required
– They are busy – but this can be their baby
• Plan ahead but start small
– Get one test running first
• The hard part is to set the infra. You have to
combine knowledge from
–
–
–
–
–
Devs
Testers
Build
Deployment
Infra-teams
• Education is a must
– This is a radical change that ppl do not understand
• Keep tests short
• Celebrate the first bugs found
10/22/13
© All rights reserved
25
26. Crashes In a Large-Scale Install-base
• A major problem on all platforms
• Intermittent crash is very hard to
find in QA
– Hard to reproduce
– May-be overlooked by tester
– Happens only when combined with
“other” software and hardware
• Crash statistics gives clarity
– Scale of problems: focus for
urgency
– Direct access to dump for devs
10/22/13
© All rights reserved
26
27. Crash Stats Server - Socorro
Select Product
or Content and
version
Filter by time
and platform
Sort by first time
this crash was
reported
Trending!
Crash dumps are
grouped by crash location
in code;
Click to see graph
• Open source server by Mozilla provides large scale dump analysis
• All licensee’s terminals upload dumps of platform and games (“content”)
• Dumps redirected to a single server as Videobet data center
10/22/13
© All rights reserved
27
28. Crash Drill-Down- Socorro
Product build
number and
client ID
Stack trace
with symbols
directly from
build server
Create
bug
Click to go
directly to
source code
• 105 bugs opened in QA night-runs (12/’12 – 8/’13)
10/22/13
© All rights reserved
28
29. Database Versioning
Database
Code
•
•
•
•
• Branch
• Build version
Schema
Config data
Operational data / OLTP
Reporting data / OLAP
We need to find an answer to these:
•What state is the database in on this machine?
– Has this script already been applied or not?
– Has the quick fix in production been applied in test afterwards?
•Can this DB correspond to my code version?
•How do I roll-back this DB to previous release version?
•How do you set up a new database instance?
10/22/13
© All rights reserved
29
33. Flyway DB is Agile!
• Migrations kept with source code
– Applying migrations added between v1 to v2 is
easy
– Roll-back of migrations supported built-in
• Merging team-branch migrations is like (even
simpler than) merging code
• Each dev can have its own DB and manage
it’s data in a simple way
– Integrative with Maven/Ant
– Command-line or JAVA API
• Data for unit-testing can be added and
removed on the fly
• Excellent for Continues Deployment
10/22/13
© All rights reserved
33
34. Goal: Continuous Deployment
At SlideShare, instead of big launches, we
We have been doing continuous deployment code all the time... it reduces risk.
release
at Plex since our inception in 1995. We we are putting out small changes at a
Since
formalized the process with our own “deploy” breaks we know what it is and can
time, if it
button in 2003, and hit the 25k deployment
roll back.
mark less than two years thereafter. Flickr "10+ Deploys Per Day"
We
Rashmi
Steve Conover
currently average between 50-100
At bixbe.com, we auto-deploy the latest
deployments per day as soon as anything is
checked in code to our massive online
ERP system.
installed, so no safety net at all. Using this
One of the exciting things, from a
Jerry Foster
method, we’re doing 10-30 deploys per day.One of the exciting things, from a
development perspective, about Digg is
development
It’s a totally productive way to move quickly.
WordPress.com - we’ve averaged about 16 perspective, about Digg is
continuous deployment - when developers fix
We still maintain the conceptevery continuous deployment - when developers fix
product releases a day, of ‘beta’ for the last
day
aabug or add aanew feature, there's no need
features… four and bits ofyears… Everyone in our
optional a half code that canbug or add new feature, there's no need
be
to wait for aascheduled release. Instead, the
tobutton that
turned on or off by user account. This letswait for scheduled release. Instead, the
company has access to a deploy us
change can go live right away. This is great change
roll out new capabilities to checked in code to about live right away. This is great releases the latest limited exposure can go
the turnaround time for a change drops
the
and avoid production the corein ourflow. tier in less
400 disrupting servers user web turnaround time for a change drops
dramatically.
dramatically.
than 30 mtrifiro (across 3 data centers).
seconds
Andrew Bayer
Andrew Bayer
Toni Schneider
10/22/13
© All rights reserved
34
Hinweis der Redaktion Expand project mode in VB slide; expand on Playtech
A mix-n-match from Scrum and Kanban
Continuous sprint
Backlog ordered daily
Daily Scrums
Focus on minimizing Work in Progress
Report is based on
Monthly retrospectives to improve throughput