Ball Of Mud Yapc 2008

YAPC::NA 2008 - June 16th, 2008 - Handling the Ball of Mud

Handling the Ball of Mud
Or, what to do when your organically grown Perl
application grows too big for its (and your) own good

phred@redhotpenguin.com

Fred Moyer - Red Hot Penguin Consulting LLC - http://www.redhotpenguin.com/talks/yapc/2008
Monday, June 16, 2008 1


Who the heck am I?

• Hacking Perl for about 8 years, avid mod_perl hacker
• Mechanical and Computer Engineering background
• Worked for startups as well as big corporations
• I've created balls of mud
• I've worked on balls of mud



What is a Ball of Mud?

• A system that has no distinguishable architecture
• http://en.wikipedia.org/wiki/Big_ball_of_mud
• Common in large, rapidly developed applications
• Cost, Quality, Time - pick two (or Fast, Good, and Cheap)
• Cost (scalar(@engineers)) is ﬁxed
• Time is usually ﬁxed (needs to be done right now!)
• So Quality must give
• Systems are complex in terms of business logic, not
necessarily in LoC



The Startup

• Startups are great places for creating balls of mud
• Your development team (two to three engineers if you are
lucky) can try out new ideas fast
• You have to adapt to rapidly changing business requirements
• If you don't adapt, your competitors will destroy you, or
you will run out of money
• Once ideas start sticking, the application starts to generate
revenue, great for the business



The Tools Group of an
Organization

• Usually one experienced Perl developer and a number of other
technical specialists
• You're the expert in Perl
• But you're not the only one writing the code
• When there's a problem though, you're the one that gets called
• You spend a lot of time maintaining code



Evolution of the
Ball of Mud

• Development is driven by organizational needs
• Not always enough information or time to
make perfect decisions
• Everything has a purpose at some point in the
codebase lifetime
• The Perfect is the Enemy of the Good Enough
• When a piece of code is written that gets the
job done, it is usually shipped to production



Rapid, Scalable
Prototyping

• Perl allows product management, a technical specialist, or a
programmer with an idea to write a low risk proof of concept
• Rarely do you have time to architect a perfect solution
• One of Perl's greatest strengths is that you can generate
prototypes very quickly
• Another strength is those prototypes can be optimized to
perform without needing to be replaced by a C program



Hey look, it
works great!

• So you've got a large codebase
• It’s ugly, but it works, and it's Mission Critical
• And forward development is expected to continue at a
steady pace, or increase
• "Why do we need to do this refactoring thing, everything
works just ﬁne?"
• An analogy here is that your code needs maintenance like
cars need oil changes, or airplanes need regular maintenance


Five Alarm
Codebase
• The maintenance cost of the app is a visible
concern (software ﬁre drills)
• When something breaks it needs to be ﬁxed fast
• The test suite doesn't pass completely anymore
(if one exists)
• And you have to also focus on scaling the system
since you are experiencing success
• You look at the code you wrote six months ago
and shake your head



High
Maintenance
• Congratulations, you now have a ball of mud!
• The good news is that it generates revenue,
usually a lot of revenue
• Or it makes your team much needed by the
organization
• The bad news is that it's higher maintenance
than Paris Hilton
• The other bad news is that when new
developers touch it, they break it (you break it
occasionally too)


Here Comes
Downtime!
• How much does downtime cost you?
• You have dozens or hundreds of servers running this code
• You push out a code change, and something goes wrong
• Assuming a 100 person company with operational costs
roughly one million dollars per month
• So one hour of downtime costs 1.000.000/30/8, about
$4.200
• The real cost is much higher though in terms of morale and
lasting impact


Getting Your
Hands Dirty

• Now YOU have to work on this thing
• Test the code before you touch it
• No test for the use case? Write one!
• "It's too hard to test!" (everyone has a unique part
of their app that makes testing difﬁcult)
• Just step up and make an attempt at testing it


Stay within
the Lines

• Don't re-architect yet
• Follow existing design patterns
• Hard to test user facing functionality, easy to test a
module API
• Big problems are hard to solve, small problems are easy,
so look at your system as lots of small problems instead of
one big one
• Write lots of small tests as opposed to a few big tests


Frameworks are
not Silver Bullets...

• "Let's port everything to a framework!”
• Be careful of this approach, it is more difﬁcult than you think
• Business logic can run deeper than is obvious in the code
• Popular approach with managers who don’t see the technical depth
• Only a good approach when you have unbelievable test coverage
• Ensure that framework ﬁts your application design patterns



Refactoring is
your Friend

• Paying off technical debt
• Refactoring can be your single biggest ally
• Management asks "What business value do we gain here?"
• The value proposition is not so much what you gain but what
you lose - maintenance costs drop



The N-thousand
line program
• awstats.pl
• Very difﬁcult to ﬁgure out if it is doing
the right thing
• Hard to automate, but not impossible
• You can't just move all the subroutines
into modules without breaking
something
• So setup scaffolding around the
program



The N-thousand line program
use Test::More tests => 3;

my $class = 'program';
require_ok("$class.pl");
import $class qw( addition should_die );

no strict 'refs';
${"main::var"} = 5; # override globals
my $method = 'addition';
cmp_ok($method->(1,1), '==', 2, '1+1 returns 2');

$method = 'should_die';
eval { $method->() };
like($@, qr/oops/i, 'should_die() died');


The N-thousand line program

• With a test in place you can port the subroutines into modules,
but watch out for globals!
• program.pl should refactor down to a command line interface
that wraps a module
• Look at qpsmtpd for a good example (http://wiki.qpsmtpd.org)
use strict;
use Getopts::Long;
# process arguments into %args
use My::Module;
my $runner = My::Module->new(%args);
$runner->run();



The test suite
that runs forever

• As the codebase grows fast, you try to keep up with testing
• More tests equals more time to run the test suite before checkins
• At some point, you hit a piece of code that's hard to test so you
write a bad test
• Or you make a small change and don't run the entire test suite
• The test suite is a problem now, instead of a solution



Testing large
systems is difﬁcult

• The core problem is too big to handle well
• Break it up into small manageable pieces
• CPAN is a great example of this approach
• Trying to ﬁx a really big testing suite while
continuing forward feature development is tough
• The CPAN approach has proven scalable
( more than 10k modules at last count)



MyPAN

• Identify parts of your code that are loosely coupled
• My::Log and My::Conﬁg are good starting points
• 'h2xs -X' is your friend, start simple
• If you can split your one big app into a few medium sized
apps, the test suites for each become much more manageable
• Identify entrenched design patterns in your code and
create modules from those



MyPAN

Most large applications contain the following components in
some form:
• One or more web based applications (mod_perl, etc.)
• Standalone job processing daemons
• A database abstraction layer
• Libraries which are extensions to C code
In the early days of the application, hard to see the lines
between different sections because you are focused on features



MyPAN

• Use your OS's packaging system to deploy code releases
• ppm, rpm, ports, ebuilds, etc...
• Check out Ovid on CPAN for rpm generation
• Creating your spec ﬁle, ebuild, etc. is part of development
• Most shops use a version control system for deployment
• That works for a few servers, but not for a dozens or hundreds
• No need to push out a new release with 100k lines of code for a
change to one job processing daemon



My own Ball of Mud

• Started out with one codebase named SL
1. Web application
2. Special purpose standalone app
3. Reporting daemons
4. cron based utilities
5. external program interfaces
• Quickly became too big to manage, too tightly coupled
• Intertwined tests made deployment and development difﬁcult



My own Ball of Mud
1. SL => SL && SL::Conﬁg
2. && SL::Model
3. && SL::App
4. && SL::Cache
5. && RHP::Timer
6. && SL::RateLimit
• Currently 14 modules
• New modules from scratch or emerge from SL
• Unit tests are easy to maintain and always pass before release
• More maintenance work, but quality is worth it
• Happy customers don’t experience broken software


Code Compliance

• Code reviews don't always work
• You are the Perl expert, so be humble
• What's easy for you might be hard for others
• Focus on stabilizing the code base through
test writing and you'll win people over
• Bring Perl::Tidy and Perl::Critic into the
codebase slowly, starting with your own code



Code Narcissism
• Narcissus
• (när-sĭs'əs) pronunciation
• n. Greek Mythology.
• A young man who pined away in love for his own image in a
pool of water and was transformed into the ﬂower that bears
his name.
• “The Narcissism of Small Code Differences”
• http://weblog.raganwald.com/2008/05/narcissism-of-small-
code-differences.html
• Be detail oriented, but don’t anthropomorphize your code



Tools for Success

• Version control - use a system that preserves code
change history across ﬁle moves (CVS is not one of them)
• Persistence testing environment - the worst failures
occur going from QA to production, not dev to QA
• Coffee, beer, food, cigarettes - foster good working
relationships with other team members


Tools::Bugs

• Use one and only one bug tracking system
• Use a system designed for software development
• Watch out for non-technical managers offering substitutes
• Watch out for managers requiring multiple systems for the
same issues ( doubles your workload )



Tools::Bugs::Bugzilla

• Mature and time tested
• Ugly, but can be skinned
• Kitchen sink UI approach, but workﬂow built for bug tracking
• Third party tools for source code management integration
• Designed for software development, not general issue tracking
• Written in Perl



Tools::Bugs::Trac

• Another great bug tracker
• Less mature than Bugzilla, but widely adopted
• Not near as ugly as Bugzilla
• SCM integration
• Popular with developers



Tools::Bugs::RT

• A helpdesk and issue tracking system
• Not a software bug tracking system
• Can be used as a bug tracking system with customization
• Used by CPAN
• Has been known to be slow and hard to use
• Good command line interfaces
• Written in Perl



Tools::Bugs::Basecamp

• Project management hosted application
• Not a software bug tracking system
• Managers love it! Programmers aren’t so fond
• Easy to deploy
• Use a real bug tracking system instead
• Avoid “doubling up” with a bug tracker, twice as much work



Tools::Bugs::FogBugz

• A bug tracking / project management system
• SCM integration
• Don’t look at the source or you will go blind
• Goes ofﬂine when the licensing server goes down
• Not extensible
• Managers love it - but you should avoid it



Tools::Testing::Smolder

• Must have for multiple developer systems
• Uses Buildbot to ﬁnd bugs before you do
• Continuous testing ensures that the tests pass
when you svn update
• Written in Perl - see it in action at the Smolder
talk



Tools::Testing::Smolder

• Small agile teams need sharp tools
• Small teams should run their own infrastructure
• Managers may ask “Why can’t we use the
smoker the 100 dev Java team is using?”
• “Boss, would you make sushi with a butter
knife?”
• “You mean the one that crashes twice a day?”
• “Because we need it setup before next year”



Emotional
Attachments

• Programming teams are complex social dynamic systems
• It's natural for code authors to become emotionally
attached to their code
• How do you explain to your coworker who has a Ph.D. in
Astrophysics that you had to add 'use strict;' to his code
when you ﬁxed a bug the week he was out?
• Better yet, how to you get him to add strictures to his
code in the future?


Egos

• Promote ego-less programming
practices
• Lead by example, not by persuasion
• If someone writes some code that looks
dodgy, write a test for it rather than
touching their code right away
• When you ﬁnd a bug, send the list an
email with a diff, and an example of the
test passing under your change



No-Win
Scenarios
1. You add a feature and it works great
2. It exposes a previous bug
3. Guess who gets the blame for it?

1. You save the world when the system melts down on New
Years Eve at 11 pm
2. Your S.O. is not pleased that you got pulled away to work
3. When you get back into work, your solution that solved the
emergency is criticized, and you are told to be more professional
in such situations


Co-Workers

• Authors of the ball of mud
• Usually smart and good coders (but don’t always use strict;)
• “Don’t touch that piece of code!”
• Sometimes they are right, sometimes they are wrong, just
like everybody else
• Approach with respect, but don’t tolerate being bullied



Managers
• If they have never been a coder, be conscious of that
• Usually have their friends they want to hire
• Always have an agenda of their own (as should you)
• Remember this person is your boss
• Maintain a professional relationship



Technical
Managers
• Have been coders, but got promoted into management
• “Here is this problem I need you to solve. You should solve
it this way”
• Now you have two problems to solve
• Can be a great resource in many situations
• Good ones can make a world class organization


Conclusion:
Don’t Worry

• Don’t try to change the world in a day
• Don’t get ﬁred over whitespace
• Do be a positive contributor
• Lose some battles, win the war
• Admit to your mistakes
• use strict;



Credits
• YAPC::NA 2008 Staff
• Those who had to suffer through the balls of mud I created
• Those who created balls of mud which I suffered through and
gained wisdom
• The great programmers I've had a chance to work with and who
have shown me what writing good code is all about
• Red Hot Penguin Consulting LLC and Silver Lining Networks
Inc. for sponsorship of this talk



Shameless Plug

Need mod_perl / Perl / PostgreSQL consulting?

fred@redhotpenguin.com

just another mod_perl hacker


Ball Of Mud Yapc 2008

Recommended

Recommended

More Related Content

Similar to Ball Of Mud Yapc 2008

Similar to Ball Of Mud Yapc 2008 (20)

More from Fred Moyer

More from Fred Moyer (20)

Ball Of Mud Yapc 2008