AppNeta TraceView is distributed tracing software as a service. Our Ruby instrumentation is a gem that hooks into popular frameworks, clients, and drivers to capture performance data from a running application – and if you think storing data in the cloud is a hard sell, just wait until you mention that you're modifying executing code to capture it! Bad instrumentation could have all kinds of adverse impacts on your application, from crashes to slowdowns to subtle corruptions, so our guiding principle is "first, do no harm". Plenty of customers have found our gem to be fast and stable under real-world conditions, but until recently, they had to take us at our word that we weren't shipping backdoors, ticking time bombs, or dark magic.
About a year ago, we decided that it was time to open source our gem. Starting off proprietary meant that we'd never planned for the world to read our code, so we didn't yet understand what risks we were exposing ourselves and our customers to. It was also internal-focused, so we weren't even sure whether anyone who wasn't an AppNeta engineer would get anything out of it. But we wanted to make it happen, and for it to happen to more code in the future, so we developed an internal process for open-sourcing code sustainably. Before launch day, we got buy-in from involved departments, vetted the license with our lawyers, figured out how our support team would support it, audited code and removed information that needed to stay private, wrote tests that anyone could run, turned internal documentation into "TFM", figured out our internal workflow, and developed a framework for community contributions. It took a few months to get everything in order, but the day finally came when we switched over our RubyGems build to the new repo and opened it to the world. (I wrote the announcement blog post – yay!)
Our gem has been on GitHub at appneta/oboe_ruby for over six months now, and we're very pleased with the results we've seen. We've had no problem continuing to sell our product, our Ruby customers have been much more engaged, and we've even been able to release new instrumentation based on their contributed code. We're glad we open sourced our trade secrets – what's stopping you?
3. A brief history of APM
• Closed source
monitoring for closed
source applications
• Oriented around
mainframes and J2EE
• Incredibly high cost, not
production-ready
• Today: rapid value, easy
deployment, low cost
6. Our maintainer
• Peter Lombardo came
on a few years ago
• Exclusively focused on
the oboe gem
• Works remote from
Italy, or else he’d be
giving this talk!
7. oboe
• Installed like a normal
gem (with C extension)
• No configuration required
• Provides:
– Application performance
management
– Software as a service
– Distributed tracing
– Bytecode instrumentation
– Trace API
9. Software as a service
• Don’t do any processing
locally
• Report data to the
agent (UDP)
• Report data to collector
(SSL, on Heroku)
10. Distributed tracing
• Follow calls from one
application to another
– Different servers
– Multiple languages
– Asynchronous
• Common
request/response
protocols:
– HTTP
– Thrift
– EJB (for JRuby)
11. Bytecode instrumentation
• Support common
components:
– Frameworks
– Libraries
– Drivers/clients
• Without code
modification
– Ruby, Python: monkey
patching
– Java: classloader
– PHP:
12. Trace API
• Same API that we use
when adding layers
• Our backend is built
around supporting it
• Duck typing for events
• Report anything!*
*please don’t report credit card numbers, thanks
18. Why did we open source oboe?
• We <3 open source
• Larger customers want
to audit our code
• Increased community
contributions
• Better capture of
reported issues
• Good publicity
19. Why couldn’t we “flip the switch”?
• The code was not
appropriately
documented
• The test suite needed
work
• There was no
contribution guide
• It wasn’t licensed
• Not branded properly!
• Issues contained
customer-specific
information
21. Tests
• “Before opensourcing, we had to
move away from manual testing
via nosetests (30 something stack
variations, manual boot and
manually run nosetests, rinse and
repeat for the next hour).
We went with Minitest. Today
with Minitest, we now test 7
Ruby versions: each with 202
tests, 1979 assertions - biggest
win beyond coverage is that it’s
automated with Travis for each
git commit.”
– Peter Lombardo
23. Branding
• We were already
AppNeta TraceView
• But we still were
Tracelytics in our code
• Not as simple as a
s/foo/bar/; lots of
unexpected places
(support URLs!)
24. Licensing
• Open source != free
software
• Competitive advantage
• Still an unfathomable
change from where we
were 10 years ago
25. Issues with issues
• We didn’t want to lose
historical issues
• New issues would also
need to reference
specific customers and
post about their data
26. The solution
• “The key was to use two repos (1
private and 1 public) using two git
remotes. Now our default is that
all issues are filed in the public
repository unless it contains
customer specific data. If a public
issue, turns out to need a pointer
to some internal data or
resource, we file a corresponding
internal issue that points to the
public issue. We now always
prefer the public repo over the
private and only use the private
when absolutely necessary.”
– Peter Lombardo
28. Marketing helped too!
• Pre-launch
– Implications of licensing
– How to reward and
publicize contributors
– Info page for repo
• Post-launch
– Launch blog post
– Social media
– Existing customers!
29. Launch day
• Set up the public repo
as a private one
• Switch our build
process (rubygems) to
use the right repo
• Post the new build as a
GitHub Release too
• Publish the blog post!
31. appneta / oboe-ruby
• Not many visitors
• But they were
incredibly engaged:
more than 25% of them
cloned the repo.
• And about a third of
them contributed code!
32. More contributions
• 80%: post an issue
• 15%: post a fix/snippet
• 5%: an entire feature!
– Grape API: Todd Lunter
at Swipely
– EventMachine: Diogo
Benica at Abril Midia
33. Happier maintainer
• “Not sure if this part is going to make
sense but working on an opensource
project is like having a weight lifted
off of your shoulders. When it’s
private, proprietary code, the burden
is completely on you to assure quality
and to not screw up. When the code
is opensource, there is a slightly
calming feel because you know that
your code is public and is available to
be reviewed and improved upon by
anyone. We love the contributions
we’ve gotten so far and are always
looking for more.”
– Peter Lombardo
34.
35. GitHub issues
• Work being done lives
in GitHub issues on the
public repo
• Anyone can post there!
36. Supporting contributors
• Most people use the
trace API incorrectly
• We only notice when
they file support tickets
• Post custom code on
GH, and we can help
while you write it!
37. Thank you!
Come work for us! We’re a mostly
Python shop
You can meet us at:
• Sep 15-17: Velocity New York
• Sep 18: WebPerfDays New
York
• Oct 21: TechBreakfast New
York
• October 24: Surge (DC)
• Nov 11-14: AWS re:Invent
(Vegas!)
Or just try us out:
http://www.appneta.com/products/traceview/
