5. Typical ALM challenges
“We don’t have good visibility into project
status”
“Our teams are not communicating
effectively”
“Requirements are not sufficiently defined or
tracked”
“We need lightweight, agile development
processes”
“Software is not adequately tested”
“Cost of maintaining and operating the
solution exceeds the business benefit”
7. THE WAY WE
WORK
Must change … ;)
And also our tolos … ;)
8. ALM Tools … must have
Development
Architecture Tool Test
Tool Tool
Design
Documents
Source Test Cases
Code
Requirements Versions Project
Stats
Project
Requirements Management
Tool Tool
Shared
Server
9.
10. Agile, Scrum, Kanban, … etc
• If
you aren’t using some form of agile development, see what
you can use from Scrum, Kanban or XP
• Projects become more predictable and repeatable
• Project costs will be lowered
• Teams will grow in capability
• Teams become more responsive to change
• Scope is managed
• Customers benefit, being delivered a product that is closer to
their needs
11. TFD: Test First Development
TDD:Test Driven Development
• Proven
to create better solution designs, with less bad coupling =
more maintainable code
• Provides safeguards when refactoring
• 10% more effort now (with TFD/TDD)
or 50% more effort later (without TDD)
• A suite
of tests built up during the life of the project provide valuable
feedback when performance testing
• Tests
may replace most traditional documentation, and provide “living”
documentation beside the code
12. ATDD and BDD
• Acceptance Test Driven Development (ATDD) or Behaviour
Driven Development (BDD) will sift out the poorly spec’d out
user stories, allowing less scope shift and faster development
• BDD tests can be automated providing fast response to the
team and to users, faster turnaround of work
• BDD tests create a common language that everyone can
“speak” – users, developers, testers, managers
• BDD tests provide living project documentation
• BDD can be applied to smaller or larger units of work
13. Source Control/Version Control
• Must be fast and reliable
• Must be simple to use
• Handy if it is also inexpensive
• Wehave adopted GIT – using local repositories as well as a
central repository
• Git
has offline capabilities and works well with peer-to-peer
push/pull
• eg., desktop local repo <-> USB stick repo <-> laptop
14. GIT
Can your source control do this?
If not, get Git.
Better yet, use GIT with TFS2012.
15. The Friction:
• If done manually:
- manual testing gets boring and quality drops off
- manual builds can become too much work
- source control: varying quality code, 5pm code oops’s
- synchronising code libraries: everyone for themselves
• Developers cannot easily see how to change if they are too
busy fighting fires
• Sometimes too close to issues, cannot see that change is
needed, or that change is even possible
• Managers push for the quick result, usually less tests
16. The Fix: Continuous Integration
• Automate everything you can:
- testing, builds, and deployment where possible
• Manyways to do this from DIY, open source, to commercial
packages.
• Any improvement will help
• Find the easiest or cheapest, and start there
• We have adopted JetBrains TeamCity (free version)
• TeamCity has 3 models: free, paid, enterprise
• Supports many different build, test and deploy configurations
for Java, and .Net
17. TeamCity integration
• Integration with just about everything:
• CVS,Subversion, ClearCase, GIT, Perforce, TFS, Mercurial, S
ourceSafe, StarTeam …
• Ant, Maven 2, IntelliJ IDEA, Nant, MSBuild …
• Email, RSS, Jabber, IDE, Windows Tray …
• Eclipse, IntelliJ IDEA, Visual Studio …
• Junit, testNG, EMMA, Cobertura …
• Rake, NuGet, Powershell, Command Line, FxCop, Gradle …
21. Local Version Control
• Old-school version control like RCS
Image stolen from http://progit.org/book/ which is really good, go read it.
22. Centralized Version Control
• This is how Subversion works
Image stolen from http://progit.org/book/ which is really good, go read it.
23. Distributed Version Control
• This is how Git works
Image stolen from http://progit.org/book/ which is really good, go read it.
24. Why Do I Care?
• Because it has been mandated that you care about version
control by Prof. Laurendeau
• Because you will use version control at every single coding job
you get in the real world
• If your company doesn’t have version control in place, that’s a huge
warning sign
• Because it will change the way you think about programming,
for the better
• Because it stops you from completely screwing yourselves the
day before an iteration is due
25. Installing Git
• Let’s install Git and try some things out.
• Linux: use your favourite package manager
• Ubuntu: aptitude install git
• Use the MSysGit installer
• http://code.google.com/p/msysgit/
26. Learning Git
• Read Pro Git, it’s excellent and free
• http://progit.org/book/
• Git is extraordinarily powerful and you’ll be a better
programmer if you take the time to understand it
• If in doubt, always use the built in manual
• Every git command has a --help flag
• e.g. git cherry-pick --help
Seriously go read http://progit.org/book/.
27. Getting Started
• Let’s tell Git our name and email address
• It attaches these to any commits we make so people know who to
kill for breaking the build an hour before an iteration is due
• git config --global user.name “Your Name”
• git config --global user.email “you@place.tld”
• --global sets the variable for your OS user
• --system sets the variable for all users on your machine
• Omit the --global and --system flags to set a git config variable for
just this repository
28. Creating a Repository
• git init
• Creates a local, empty git repository in the folder where the
command is run
• Creates a .git folder for the guts of the repository
• Only one .git folder, at the root of your repository
• This is way nicer than creating a new .svn folder for every single subfolder
in your repository
• We’ll
go over how to work with other people on a remote
repository soon
29. Anatomy of a Repository
Image stolen from http://progit.org/book/ which is really good, go read it.
30. Anatomy of a Repository
• Crazy important! Memorize this!
• The working directory consists of the actual files and folders on your
machine
• The staging area lets you build commits out of snapshots taken of
the files and folders in your working directory
• The repository maintains a collection and complete hierarchical
history of all commits
31. Oh God, What’s Happening?
• git status
• Shows you the state of your staging area and of your working
directory
• git log
• Shows a commit history of your repository
• git diff
• Shows changes between commits, your working directory, the
staging area, etc.
• git diff --help to learn more
Or you could just go read http://progit.org/book/ and you’ll be an expert.
32. Adding & Removing Files
• git add file
• Adds a snapshot of file to your staging area
• You can change file and the snapshot will remain as it is
• git rm file
• Removes the file snapshot from your staging area and deletes file
from your working directory
• git rm --cached file
• Removes the file snapshot from your staging area while keeping file
in your working directory intact
33. .gitignore
• Tells Git which files to ignore outright
• Uses glob syntax for pattern matching
• There’s a decent summary of glob syntax at
linux.about.com/library/cmd/blcmdl7_glob.htm
• Git adds re-inclusion rules with !pattern
• “Include any files matched by pattern, even if they’ve been ignored
by a previous pattern.”
• There’s a sample .gitignore on the website
• Stack Overflow is a good source of .gitignore files
34. Making Commits
• git commit -m “your message”
• Creates a commit from the contents of your staging area and adds
it to the repository
• -m “your message” sets the commit message
• Ifwe keep adding commits we get a linked list that represents
the history of our repository
• gitk --all gives a graphical history of all branches
• Leaving out the --all shows just this branch
And the book at http://progit.org/book/ has a whole bunch more information. Read the damn book!
36. Branching & Why It Is Awesome
• Branching allows us to create a tree of commits instead of a
linked list
• Merging will let us turn this into a DAG
• Easy,painless branching is the most important and powerful
feature of Git
• Give each new feature its own branch, which can be merged back
into the main (master) branch after it’s been completed and is
stable
38. Creating Branches
• Gitmaintains a pointer to the current checked out branch,
called HEAD
• git branch newbranch
• Creates a new branch starting at the current commit, but does not
move the HEAD pointer
• git checkout newbranch
• Changes HEAD to point to newbranch
• Rungit branch then git checkout to create and start working
from a new branch
• Shortcut: git checkout -b newbranch
39. Branching Example
$ git branch testing
$ git checkout testing
Images again stolen from http://progit.org/book/ which you should read because it’s good.
40. Branching Example
$ git commit
$ git checkout master
Images stolen from http://progit.org/book/
41. Branching Example
$ git commit
• At this point, the branch history has diverged
• We want to branch for new features and merge them back into
the master branch
• This makes your life infinitely easier
Images once again stolen from http://progit.org/book/
42. Merge Tools
• git config --global merge.tool toolname
• Sets the merge tool for your OS user
• A mergetool allows you to fix any conflicts that arise from
merging two branches
• Google will give you a list of merge tools
• I use p4merge, it’s hard to go wrong just choosing a random merge
tool and using it
• git mergetool
• Run this command if anything goes wrong with a merge, it’ll allow
you to fix things
43. Merging
• git merge branchname
• Allows you to take another branch and merge its changes into the
currently checked out branch
• Git
has hyper-intelligent algorithms that track your content, not
your files
• If you move or rename a file and make changes to it, it will still be
detected as the same content
46. Branching & Merging in SVN
• It’s
relevant to see how the approach taken by Git differs from
that of terrible legacy systems like SVN
• Subversion doesn’t have real branches
• A branch is just a copy of your repository in a named folder
• SVN has no concept of branch history and therefore cannot
determine common ancestry to help with merging
• Subversion doesn’t have real merging
• The svn merge command should be called svn diff-and-apply-patch
because that’s all it does
• “If a merge fails, run svn revert and do it by hand.”
• There is no way to tell whether a given set of changes were the
result of a merge or were just straight edits
47. Remote Repositories
• git remote add reponame url
• Adds an external Git repository called name
• git fetch reponame
• Fetches updated branches from reponame including all updated
data
• A remote branch shows up as reponame/branchname
• Your
local information about remote repositories isn’t updated
automatically
• You need to run git fetch periodically on your remotes to get new
branch/commit information
49. Pushing
• git push reponame branchname
• Adds a local branch to a remote Git repository
• You need to have write access to the remote repository
• Alternatively, merges your local branch into a remote branch of the
same name
• git push reponame localbranch:remotebranch
• Explicit syntax for merging your changes from localbranch into
remotebranch
• Always fetch and merge before you push
• Save yourself grief and error messages
50. Pulling
• git pull reponame branchname
• Fetches from reponame then merges the remote branchname into
your local branchname
• git pull reponame localbranch:remotebranch
• Fetches from reponame then merges reponame/remotebranch into
localbranch
• Syntactic sugar for fetching and merging
• Your workflow should be “git pull; git push”
51. Tracking Branches
• git checkout -b branch remote/otherbranch
• Gives you a tracking branch that you can work on, that starts where
remote/otherbranch is
• Trackingbranches are local branches that have a direct
relationship to a remote branch
• You can just call git push/pull with no arguments and it know which
remote branch to change
52. Cloning a Repository
• git clone url
• Creates a new local git repository, creates new tracking branches
for each branch in the cloned repository, creates and checks out an
initial branch that is forked from the remote’s active branch
• Watch out if the remote repository doesn’t have any commits
yet (really common thing)
• Cloning will fail, you need to do git init, git remote add origin url,
create an initial commit with some content, then run git push origin
master
54. CI – Project Practices
Run tests and inspections as part of build
•
• All tests and inspections must pass
• Build automatically upon code commit
• Keep the build fast
• Want fast feedback; minutes, not hours
• Design builds to fail fast
• Run tests/inspections that are most likely to fail first
• Want to find out about failures as soon as possible
• Provide continuous visibility and feedback
• Project status dashboard
• Continuous feedback devices
55. CI - Developer Practices
• Commit code frequently (at least daily)
• Reduces magnitude of changes to be integrated with each build
• Reduces likelihood of stale (out-of-sync) code changes
• Reduces effort of troubleshooting failures – less has changed
• Run private builds before check-in
• Execute same set of tests and inspections that CI will
• IDE may help here (e.g. PMD or Checkstyle plugin)
• Fix broken builds immediately
• Don’t commit broken code
• Broken code should fail the private build
56. CI - Benefits
• Reduced risk
• Reduced manual, repetitive processes
• Easier debugging / troubleshooting
• Generate deployable software at any time
• Better project visibility
• Greater confidence in the software
59. Parallel Build Example
Build 1 on Machine 1 Build 2 on Machine 2
Compilation / Fast Compilation / Fast
Introspection Introspection
Unit Tests Unit Tests
Integration / System Integration / System
Tests Tests
Functional Tests Functional Tests
Overall Pass/Fail Overall Pass/Fail
61. Continuous Database Integration
• Rebuild your database and test data for every build.
• Store database artifacts in version control
• DDL, DML, config files, etc.
• Generate the database from build scripts
• Scripts for database creation, configuration, population
• Apply continuous testing / inspection against your database(s)
• LiquiBase, Rails Migrations
62. Continuous Integration vs Build Management
“Build Management, essentially, aims to produce a set of
artifacts that correspond to a specified snapshot of source
code, and to be able to hand off those artifacts to third parties
outside of development”
• UrbanCode - http://www.cmcrossroads.com/pdf/DrawingtheLine.pdf
63. CI – BM Comparisons
Continuous Integration Build Management
What is the purpose? To integrate the latest To provide a definitive set
changes and determine of usable artifacts based on
current code quality a given code slice in time
Who is the audience? Internal External (non-developer)
What are the products? Pass/Fail, Metrics, Code Deployable artifact
quality
What is the code base? The latest changes Fixed time reference
Who benefits the most? Developers Infrastructure / Support
64. Questions to Ponder
• Who should own the governance for CI? How about
for build management?
• Shouldsimilar, different, or exactly the same build
processes be used for CI and BM?
• Whatis needed to verify the quality of the artifact
generated by the BM process?
65. CI Developer Anti-Patterns
• Infrequent check-ins
• Premature check-ins (failing code)
• Ignoring broken builds
• Turning off tests to correct builds
• Waiting until end of day to commit changes
• Inconsistent developer build environments (works on my
machine)
66. CI Tooling Anti-Patterns
• Insufficient feedback
• Too much (spam) feedback
• Underpowered (slow) build machine
• Bloated build
• Weak builds that never fail (Continuous Ignorance)
• UsingScheduled Builds versus building software
with every code change
• Failing to clean out old artifacts before each build
http://www.ibm.com/developerworks/java/library/j-ap11297/index.html
67. CI - Challenges
“CI is not just a technical implementation; it is also an
organizational and cultural implementation”
– Paul Duvall, Continuous Integration
68. Challenges – Cultural
• Development culture
• Not familiar with continuous integration
• Ignore failing tests or broken builds
• Not checking in code frequently (daily)
• Not running tests before check-in
• Lack of encouragement for CI practices
69. Challenges –
Personnel/Experience
• Lack of expertise
• Continuous Integration
• CruiseControl / Hudson
• Deployment (e.g., WebSphere)
• No dedicated resources for CI (anymore)
• Made great progress when we did have that.
70. Challenges – Infrastructure
• Obtaining system accounts, disk space, etc.
to run CI
• Insufficient and constrained environments
• Not enough, not always available
• Test environments, app servers, databases, etc.
71. Challenges – Technical
Implementation
• Stuck on outdated technologies
• Java 1.4, Tru64, PVCS, VB, VisualC++
• Technology stacks
• Grails, Java, Struts, etc – all with differing folder structures
• Database changes in separate source repo
• Database changes in PVCS; main code in CVS
• Disparate systems for CI integration
• CVS, PVCS, PICCT, QualityCenter, VersionOne, Fortify
• Cruise Control / Hudson – learning curve
72. Challenges – Running Builds
• Brittle tests (e.g., data-dependency)
• Slow tests or lots of time-consuming tests
• Build “noise”
• CI
build fails for reasons outside developer control (e.g., bug in
Hudson or file access issues)
• Interactions between builds
• “Access denied” and “Cannot delete …” errors on Hudson
• Shortcomings of Windows build server
75. What is NuGet?
• .NET Package Manager
• Original OSS Project Nu now NuGet
• Ships with ASP.NET MVC3
• OS Managed by Outercurve Foundation hosted on
nuget.codeplex.com
• 32,000+ Packages
• Current version 1.6
• Extendable and is a protocol
76. What is Package Management?
• Tools to simplify discovering, obtaining, installing and updating
project references.
• Eliminate dependency hell.
• Similar to Ruby Gems and OpenWrap – http://openwrap.org
• Not like a system package manager like, Linux apt-get.
• Windows System Package Managers
• Chocolatey – http://chocolatey.org
• CoApp – http://coapp.org
77. What is the goal of NuGet?
• Promotion of projects and libraries to ensure great software is
readily available and usable.
• Eliminate discovering, obtaining, and complex configuration as a
barrier to entry for leveraging powerful software.
• Make Open Source Software ( OSS ) easy as “Add Reference”.
• Eliminate dependency hell.
• Improve developer experience and productivity.
• Leverage Conventions over configuration
• Tight integration with Visual Studio IDE and leverage power of
PowerShell.
78. How do I get started?
• Recommend installing NuGet Visual Studio Extension, NuGet
Command Line and NuGet Explorer
http://nuget.codeplex.com
• Visit http://nuget.org
• Do a one over the NuGet Documentation -
http://docs.nuget.org/
• Create a new VS 2010 solution or open an existing solution
then hack away on the Open Package Manager Console
consuming packages
• Start small, replacing a few references with NuGet Package
References.
• Watch Scott Hansleman’s screen casts on NuGet for in depth
knowledge ( references section )
79. NuGet Package Gallery
• Official NuGet Package Gallery - http://nuget.org
• Provides official community package library.
• Publish Packages with a FREE account
• Main NuGet Feed for consuming packages
• Package Stats – http://stats.nuget.org
• NuGet Team Blog - http://blog.nuget.org/
• 32,000+ packages
80.
81. Why Create Own Packages?
• Distribute .NET assemblies and dependencies.
• Eliminate dependencies from VCS.
• Perform complex configuration and setup to ease adoption.
• Distribute
• JavaScript Libraries
• Samples
• Scripts
• Support Extensions
• Extend Package Mgr Console
• Extend NuGet Command Line
• Extend NuGet Explorer
82. What makes up a NuGet
Package?
• Meta Data
• Dependencies
• Content
• Transforms
• Templates
• Libs
• dlls
• Tools
• Init.ps1
• Install.ps1
• Uninstall.ps1
83.
84. Web Start-Up Code
• Simple way to enable start-up of your library in Web Apps
• WebActivator NuGet library
• Replacements of the .NET 4 attributes
• [PreApplicationStartMethod]
• [PostApplicationStartMethod]
• Content
• App_Start
• Examples
• IoC libraries like Ninject and StructureMap
• Commmon
• $rootnamespace$
85. Creating NuGet Packages?
• Create Manifest
• Manual
• Command line
• Assembly
• Visual Studio Project
• Create Packages
• Manually using command line
• NuGet Explorer
• Visual Studio Project
• Package Manager Console using New-Package
• TeamCity
86. Package Source
• NuGet Visual Studio Extension supports multiple Package
Sources in hierarchy order.
• NuGet Source Options
• Nuget.org
• Own Local or Network Directories
• Local Cache
• Own Read Only Server
• Own NuGet Gallery
• Own NuGet Orchard Gallery
• .NET Community Source Options
• MyGet.org
• NuGetFeed.org
• TeamCity
87. Why host internal NuGet
Gallery?
• Open Source - https://github.com/NuGet/NuGetGallery
• Provides full ownership
• Publish proprietary internal packages
• Publish backups of external packages
• Browse able gallery
• Manage internal/external packages
• Extend or integrate with own requirements
• Extending NuGet Gallery
• Baseline Feed
• Client/Project Feeds
• Community Feeds
• Enterprise Feeds
• Personal Feeds
88. What about Continuous
Integration?
• No Commit Strategy
• Tracking Package Metadata
• Package Restoration
• TeamCity Pull, Package and Publish Support
• Automated Deployment with Octopus
• Convention Based
• Shipping Application in NuGet Package
• Deploying Applications to servers
89. What is the value to the team,
.NET COP, enterprise, client?
• Reduce project dependency management. Simplify setup and
onboarding. Improving version management.
• Promote reuse, standardization, discovery, collaboration
regarding packages across all levels of the organization.
• Removal of dependencies from VCS saving valuable disk
space.
• Improves continuous integration process and opens the
opportunity or automated deployments with products like
Octopus.
• Offers extension points with Chocolatey to improve developer
experience regarding machine setup and overall maintenance.
• Integration with commercial vendor NuGet feeds like
DevExpress
92. Referencias
• Download and tutorials:
• Official GIT site
http://msysgit.github.com/
• Git for Windows
http://code.google.com/p/msysgit/downloads/list?q=full+installer+official+git
• GitHub for Windows
http://windows.github.com/
• JetBrains - Team City
http://www.jetbrains.com/teamcity/
• Download TeamCity
http://www.jetbrains.com/teamcity/download/index.html
• Presentations:
• Git from Erick Lawless
http://bit.ly/VOJgXo
• NuGet – FrozenBlocks
http://bit.ly/ZfZBfk
• Scaling up and bottoming up
http://bit.ly/V2sFOz
Tools used for different purposes—working with requirements, specifying architecture, developing code, testing code, and project management—should all be able to work with a common set of interconnected artifacts stored in a common place. As Figure 2 shows, those artifacts can include requirements, design documents, various versions of source code, test cases, statistics about this development project, and more. This kind of integration allows all sorts of useful things: automatically recording code check-ins, associating tests with source code versions, generating historical reports of bug counts, and more. More important, it allows optimizing the entire process. The flow of work is now apparent, and so making that flow as smooth as possible gets easier.When our industry converges on an architecture, it means there’s broad consensus about the best way to do something.For the most part, the first generation of team development tools didn’t take this approach. The vendors have learned from their experience, however, and team-based tools today can provide this broad integration. Microsoft’s Visual Studio Team System, for example, includes specialized tools for architecture, development, testing, and more, all of which rely on Team Foundation Server (TFS) to store the artifacts they work with.
Step 1 – clean up the development process, introduce as many agile concepts as possible. Increases team communications, team capability, project repeatability and lowers costs.
Step 2 - Improve quality at the beginning, not during or after. Build in quality. You may have success now, and it might work on smaller projects, but as projects grow and become more complex, or change and have additions over time – then tests become crucial to code quality.
3 – Handle requirements in a clever way - at the same time reduce scope creep, and create living documentation. BDD tests in “Cucumber” language (Given / When / Then) format mean that you reduce the chances of miscommunicating a requirement – Cucumber is a DSL – a Domain Specific Language – one that works between users and developers. We use StoryQ internally (see http://storyq.codeplex.com ) though there are many others.
3. As we primarily produce software to run under windows, our developers use GitExtensions http://code.google.com/p/gitextensions/We simply use shared folders and ordinary file access to repositories, rather than implement MSysGit or other more complete full git:// implementations. This works well for us, and means we don’t need to fiddle with keys, Cygwin or Linux.
Image source: http://bit.ly/9bUvdC
Real life: do things manually and they fall off, automate as much as possible … leads into the next slide
Many options, we’ve settled for TeamCity from Jetbrains (makers of Resharper). See: http://www.jetbrains.com/teamcity/Alternatives CruiseControl, TFS, Hudson/Jenkins etc. For a good list to start from, go to: http://en.wikipedia.org/wiki/Continuous_Integration
For more info see: http://www.jetbrains.com/teamcity/features/index.html
XP guideline of a 10 minute build is reasonable and achievable for most projects. It s worth the effort to streamline and speed up the build – affects developers every time they commit.Provide continuous visibility and feedback: Targeted feedback. Send the right information to the right people at the right time in the right way.
Frequent broken builds suggest developers are not running private builds. Development culture and “people issues” are often more difficult to solve than technical issues. But you need to keep the build (relatively) fast so that developers are willing to run private builds. May require trade-off: allow for occasional broken builds in order to avoid expecting developers to run time-consuming or cumbersome build processes.
Reduced risk; you eliminate the blind spot of deferred integration Makes bugs dramatically easier to find and remove; If you introduce a bug and detect it quickly it's far easier to fix. Plus, bugs are also cumulative. The more bugs you have, the harder it is to remove each one.Generate deployable software at any time – Removes one of the biggest barriers to frequent deployment
Staged build(or build pipeline) – multiple builds done in sequence. First build is triggered by a commit – should be quick (smoke tests/unit tests). Second build runs slower tests/inspections – integration tests, etc.; takes much longer.Parallel builds – can also work well with staged builds. For instance, can split up the secondary build across multiple build processes or machines.Continuous database integration – Automate the database integration. Rebuild your database and test data every time you run a CI build. Database integration artifacts should be stored in the version control system: DDL, DML, config files, etc. Generate the database from build scripts. Automated scripts for database creation, configuration, population Apply continuous testing and continuous inspection against your database(s)
Martin Fowler: When I've described this practice to people, I commonly find two reactions: "it can't work (here)" and "doing it won't make much difference".