The Reluctant SysAdmin : 360|iDev Austin 2010

The Reluctant
SysAdmin
Managing the Server side of a Client-
Server iPhone App
Jen Harvey, Voxilate
@jen_h
360|iDev Austin
Nov 10 2010
Wednesday, November 17, 2010

• Me: Network security background,
OSS & Linux fangirl
• Currently: Co-founder of Voxilate with
Steven Hugg
• Last year: Traveling the country while
bootstrapping the company, building
iPhone apps on the road
A little background...

HeyTell
• HeyTell Voice Messenger allows users
to share short voice messages &
location
• Released February 2010
• Have been building, managing,
deploying, re-deploying, updating,
expanding, scaling on the road ever
since...

• Map of travels, 360iDev San Jose!
360|iDev San Jose!

Current Objectives
• Keep over 1 million users happy &
using our app
• Maintain respectable uptime &
performance while adding new
features & expanding our reach
• Get a little sleep at night
• Share what we’ve learned so that
others who embark on similar
journeys can also sleep!

Agenda
• Why a Server?
• Choose Your Poison
• Build It Out
• Lock It Down
• Maintain & Monitor

So...why would you
want to run a server
component?

Metrics!
• What metrics are valuable to you?
• Number of total users
• Number of active users per day/
month/year
• Number of whatever-it-is-you-do all
day (for us, submitted messages)
• Number of customers vs. users
• Busiest times of day/week/month?
grep is
awesome

Track app usage &
errors
•Speed
customer
support
•Understand
how users
really use
your app
•Be alerted
when errors
occur
•Really useful
for beta
testing to
determine app
viability

Provide value-added
content
• Virtual goods or in-app purchase
goodies
• User-to-User or User-to-Public content
sharing
• Run your own analytics or ad servers

Basic Web Server
• Informational site for game
• Customer service site
• FAQ hosting
• Note: This is not what we’re focusing
on in this talk, but the info here is
pretty general purpose! :)

Control your own Push
Notiﬁcations
• Don’t need an external service (free)
• Can be a little painful to set up, but
resources & libraries exist on web for
PHP, Java, Python, Ruby...
• Additional insight when users run
into Push Notiﬁcation issues

Your
systems
Apple’s
systems
iPhone
Client
App Store Receipt
checking
• Verify user is a
customer before
enabling feature
• To gather real-time
statistics: piracy
trends, conversion
rates for freemium
apps

#1 reason to use a
server component?

Your server is your app’s
engine
Image courtesy of Richard Smith/gocarts on flickr: http://flickr.com/gocarts

Choose Your Poison

We’re lucky! So many
hosting options!

Cloud: Infrastructure as
a Service
• Pay-as-you-go systems deployment
• Amazon Web Services (EC2, S3,
RDS, ELB, ...)
• Microsoft Azure
• VMWare vCloud
• Rackspace Cloud (formerly Mosso)
• ...

Cloud: Platform as a Service
• Write your app for the platform,
interact via API, provider handles
scaling and administrative tasks:
• Heroku (for Ruby enthusiasts, built
on EC2)
• Google App Engine (Java, Python,
JRuby...)
• Engine Yard (Ruby)
• ...

Virtual Private Servers
(VPS)
• You pay for a dedicated server,
sometimes a VM, sometimes
hardware
• Rackspace
• Slicehost
• Linode
• ...

Your Mom’s Basement
• Or your ofﬁce.
• You don’t ﬁnd sleep essential, do you?
• (No, really, this is fantastic if you have
a large team & money to build
out...but as an indie, you are likely to
have neither)

Considerations
• What’s your preferred language & OS?
Write and work with what you know!
• How much responsibility/ﬂexibility/
portability do you want/need to have?
• What’s your budget? GAE & AWS have
free tiers to give you a taste & likely
have enough horsepower to start with.

My advice:
Go with what you
know & feel
comfortable with

We chose Amazon Web
Services
• Quick & ﬂexible & full of building
blocks:
• Load balancers
• Hosted MySQL & SimpleDB
• Multiple availability zones
• Lots of h/w & memory conﬁgs
• S3 redundant storage

And...
• Great APIs: Command line tools & lots
of libraries
• Can script anything or integrate w/
web app
• Can do some management tasks
from phone
• Huge user community - many ways
to obtain support

Also...
• Quick & simple to prototype system
architecture
• Easy to bring up identical-to-
production test beds with same
conﬁguration as production - but with
discrete & separate security grouping
• Published Service Level Agreement
and Security Practices documentation

Cons
• Handle scaling (& everything else)
yourself - just because your app is “in
the cloud,” doesn’t mean it
automatically scales
• Harder to set up, pre-built machine
images available, but still need to
customize/secure
• Instances are ephemeral (but I like this because
of the way it forces you to architect)

Build It Out

A note on scaling early
• Be prepared to do it
• Know it’s coming if you’re successful
and architect/code with the
understanding that you’re the guy/gal
who’s going to have to make it work
when it comes
• Don’t overarchitect early on
• Slow, hypeless ramp-up & predictable
viral growth can help here

Cool! We have a Enterprise-Grade(TM)
horizontal webscale scaling solution!
Uh, it’s getting
corrupted every
12 hours.
SHUT
DOWN
EVERYTHING

Build with security in
mind
• Develop & build your custom software
with security in mind
• You know what anomalous behavior
is/can be
• Put on the adversary’s hat - what
could they do? What’s the worst
outcome? Is it worth building in
protection for certain scenarios?

The Voltron
Principle
Individual
components join
to build the
ultimate
defender of
the universe

• Single Linux-based
machine image we
use to build
everything on top of
• Document changes
for future migrations
(I ♥script)s
• On deployment, bolt-
on the pieces we need
& conﬁg changes
• If a host goes down,
we can bring up an
identical host in
known state in
minutes, swap out
their IPs and run the
post-mortem once
we’ve normalized
Voltron Core

• Essential logs & configuration files
periodically stored on S3
• Rotate logs frequently, especially as
you grow
• Don’t store passwords or keys in
configs, populate these on deploy (I
abuse sed, you may use something
more elegant)

Load balancer
Security
Infrastructure
Database
Persistent
storage Cache
Base AMI
Application
Core
Notiﬁcation
server

But some days...

Assume everything
will fail

Ready to setup our new domain name?
Hey, do CNAMEs
have a “.” at the
end?
D’OH!
Let’s wait 2
hours for it to
expire...

Find your possible points of failure
(rusty robot joints)
• DNS - if your hostname doesn’t
resolve, your app can’t get home
• Are backups working?
• Storage and/or database - what
happens when/if they go away?
• DDoS (intentional or not...)

• Deal with small amounts of failure
gracefully (cache, limited
functionality)
• Don’t put your web server &
application server components on the
same *anything*

But you will, without a doubt, run into a
‘ﬂesh wound’ issue

How you handle it is
pivotal

The database is bogged down. I think this
one feature is causing it.
Does anyone even
know we have that
feature?
That feature’s
GONE!

• Twitter
• Facebook
• Respond to customer support emails
(have cut & pastable friendly
response - small team has no time for
personal emails in crisis)
• You may feel like it’s the end of the
world, but this, too, shall pass
Customer
Communication == Key

Hey, guys, Justin Bieber just announced
he’s using us on Twitter!
Cool. Who’s that?
Gah! Server’s
melted! Users
revolt!

Helpful tip for high-
traffic systems
• If you’re looking to max out
connections on a single Linux-based
system, think about:
• Memory & file handles (see also:
ulimit tweaking)
• Connection tracking as relates to
memory (look up netfilter/tcp stack
tweaking)

Lock it Down

Yes, security is your
problem
• If you are storing users personal
information, you are subject to laws
and regulations in the US, speciﬁc
states, and foreign countries
• Many jurisdictions deﬁne personal
information differently
• Most regulations require a written
policy and best practices for security

So what’s best practices?
• Secure your perimeter
• Secure your services
• Detect, alert on, and block suspicious
activity
• Protect your users and encrypt user
information in transit and at rest
• Have written policies and plans

Secure Your Perimeter
• AWS has (at least) two walls
• One is its “security group” context
• One is your image’s local ﬁrewall
• Block everything by default, open only
the ports you need
• No root login
• Passwordless login only (use key
pairs)

Secure your services
• Services should not run as root (for
ex., www-data for apache2)
• Service usernames should not have
shell login access
• Monitor for security vulnerabilities &
upgrade when needed
• Build security into custom software

For host-based intrusion detection - I
love OSSEC:
• Quick & easy, lightweight, Open
Source, free
• Alerts on logs - extensive default
ruleset but can customize alerting
for your speciﬁc app
• Daily Tripwire & rootkit checks
• Active response: can block IPs on
suspicious behavior
Detect & Alert

• If you need to store user
information, encrypt in transit and
at rest
• If you need data from your systems
locally, use encryption end-to-end --
down to encrypting your drive
• Use SSL in the great wide world, it’s
not that hard!
Protect Your Users

Why use SSL?
• Protects your users from sending
personal data over the Internet in the
clear
• Protects you from neophyte reverse
engineers

On Using SSL
• EC2 Load Balancer now allows SSL
termination - https to the LB, http
inside data center
• Small & bootstrapped like us? Use
StartSSL - free certs. Go to someone
like DigiCert for nifty wildcard certs
once you’ve got the resources.

The CCATS Issue
• These guys rule: http://www.zetetic.net/
blog/2009/08/03/mass-market-encryption-
commodity-classiﬁcation-for-iphone-
applications-in-8-easy-steps/
• Can deploy to US & Canada
immediately, then expand reach after
approval
• Took us just over a month to obtain
• Check w/Apple ﬁrst; may not be
required anymore.

User Passwords
• Many users will use the same
password for everything--banks &
FourSquare.
• There’s nothing you can do about it.
• Databases full of email addresses and
passwords are attractive targets for
this reason

Don’t be an attractive target...don’t
make personal information necessary
to use the service, if at all possible

Allow mechanisms for users to update
or clear their information at any time
without your intervention

Whenever possible, educate users about
protecting their privacy (this leads to
all good - more educated users, fewer
complaints, more trust, more goodwill,
more/happier users!)

• Have a policy for purging data/
accounts/etc. that you don’t need and
follow it
• Automate this or build it into the app
if you can
• Have a written policy for data
breaches and intrusions
• Write down instructions for yourself--
this’ll keep you sane if you ever have a
real breach or a false alarm

• Keep a list of the services you use, one
quick & dirty thing to do is scrape
vulnerability feeds like feed://
nvd.nist.gov/download/nvd-rss.xml
for your service names
• When security issues are reported
and new versions released, patch out
of band, test, replace (pretty easy to
do with EC2!)

Your users & piece of
mind are totally
worth it!
This will save you
time & sanity in the
long run

Maintain & Monitor

Planning Maintenance
• If you can, use a load balancer and
switch out backend servers
• Have backup systems in working state
for fall-back
• Track usage statistics throughout
your app’s lifetime - schedule
maintenance for “slowest” time period

00 02 04 06 08 10 12 14 16 18 20 22
Sunday
Monday
Tuesday
Wednesday
Thursday
Friday
Saturday
The User Rollercoaster
(# connections/hour, GMT)
Sunday, 11:00
GMT it is, then.

Keep a Calendar
• Keep a calendar of important dates:
• Developer certificate expirations
• SSL certificate expiration
• APNS certificate expiration
• Domain name registry expiration

Monitor Uptime
• Check out Pingdom - set thresholds to
be alerted when servers are slow or
inaccessible
• Conﬁgure OSSEC to alert on
conditions that precipitate an “issue”
• Set alerts or automated account
recharges for *everything* that could
block app functionality
• Make sure someone’s always
accessible

Hey, the server’s down. Where are you
guys?
I’m on a BOAT!
I’m on a
PLANE, yo!

HeyTell Systems Uptime
97
97.75
98.5
99.25
100
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov

Downtime, Minutes
0
175
350
525
700
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov
99.2% uptime == 350
minutes == 5.8 hours!!

Total Downtime:
1.5 days
Uptime: 99.48%
(uptime sounds sexy; downtime...not so much)

Managing on the Run
• Phone SSH client (CommandBot on
Droid, iSSH on iPhone)
• EC2 Management client (Decaf on
Droid, iAWSManager on iPhone)
• Separate Support Account email setup
on phone
• Notepad app with customer support
FAQ answers

Other lifesavers on the
run
• Reliable 3G service
• Mobile broadband card and/or
tethering setup
• Netbook or small laptop

Summary
• On hosting: Go with what you know
• Architect with failure & future scaling
issues in mind
• Lock it down: Keep your data & your
users safe
• Monitoring & maintenance: Make
your systems work for you
• Good luck! You can do it!

References & Links
OSSEC: http://ossec.net
Pingdom uptime cheatsheet: http://royal.pingdom.com/2009/03/24/
a-handy-uptime-and-downtime-conversion-cheat-sheet/
AWS Free Tier info: http://aws.amazon.com/free/
AWS Security Doc: http://awsmedia.s3.amazonaws.com/pdf/
AWS_Security_Whitepaper.pdf
TCP stack tweakage: http://www.metabrew.com/article/a-million-
user-comet-application-with-mochiweb-part-1

Thanks!
email: jen@voxilate.com
twitter: @jen_h

The Reluctant SysAdmin : 360|iDev Austin 2010

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Andere mochten auch

Andere mochten auch (6)

Ähnlich wie The Reluctant SysAdmin : 360|iDev Austin 2010

Ähnlich wie The Reluctant SysAdmin : 360|iDev Austin 2010 (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

The Reluctant SysAdmin : 360|iDev Austin 2010