Tradeshift delivers electronic invoicing and social networking to the browser of businesses around the world. To do so, we employ a highly dynamic cloud-based infrastructure that scales in near real-time. In this talk we present Tradeshift's e-invoicing product, how we manage the dynamics of the infrastructure and how we ensure a continuous product development- and delivery process.
Anders Nickelsen has a background in networks and distributed system and has a PhD in computer networks from Aalborg University. At Tradeshift, he works as a quality assurance engineer with special focus on continuity of infrastructure management and quality of software releases.
Mikkel Hippe Brun is co-founder of Tradeshift and holds a Masters degree in Computer Science from DIKU. Mikkel has worked for many years with e-invoicing and large scale infrastructures for exchange of business documents. Mikkel is chair of the OASIS Business Document Exchange Technical Committee.
How cloud computing enables Tradeshift to deliver continuous and global e-invoicing and more
1. Head in the cloud and feet
on the ground
How cloud computing enables Tradeshift to deliver
continuous and global e-invoicing and more
Mikkel Hippe Brun Anders Nickelsen
mhb@tradeshift.com PhD, Quality Assurance Engineer
Twitter: @hippebrun ani@tradeshift.com
Cell: +45 3118 9102 Twitter: @anickelsen
Cell: +45 3177 6511
April 2011
2.
3. Our industry
Characteristic
Practices have not changed much for 20 years
Expensive infrastructure
Traditional business models
Costly for customers
On-boarding of a customer is a project
4. The problem with E-invoicing
Mo6vated
to
move
to
e-‐invoicing
Advanced
business
so.ware
(ERP)
Office
packages
Accoun6ng
Paper
+
email
/
PDF
systems
5. Connecting the long tail
Un6l
now:
No
good
solu6ons
for
this
segment
Advanced
business
so.ware
(ERP)
Office
packages
Accoun6ng
Paper
+
email
/
PDF
systems
6. Cloud computing allows us to
Establish an extremely scalable
infrastructure
Scales with our demand (up and down)
Offer our services for a fraction of the cost
of our competitors
No CapEX
OpEx scales with customers (and gets cheaper with volume)
Think differently
Is there a better business model?
How do we provide real value?
Network
Real time
Sharing
7. Our advantage
Blank slate
No existing customers
No legacy systems that must be maintained
= No migration
Agile
Pirate culture
Flat organization
Product development owned by teams
Emergence of Cloud technologies
We could start cheap!
10. What We Learned from Building These
Infrastructures..
Cloud + Open Standards + Open Source
Software
Can be used to realize extremely low-cost, secure,
reliable high-volume B2B infrastructure
Lowering cost significantly & opening the
infrastructure
Can bring small business into E-invoicing relationships
Capturing the long tail of small
businesses
Can be achieved by including them into a network and
making them discoverable
11. Operations
Cost per Unit
Unit
Today: ~1.2$ / Unit
Sept. 11: ~ 0.12$ / Unit
Sept. 12: ~0.04$ / Unit
13. The cloud in Tradeshift
Software-as-a-Service
Self-organizing infrastructure
Platform-as-a-Service
Infrastructure-as-a-Service
Continuous deployment into self-
organizing infrastructure
14. Software-as-a-Service
go.tradeshift.com
Companies have isolated storage on a shared cloud-based
platform
No installations at customer sites
Public APIs
Integration points to 3rd party software
and enterprise customers
15. Platform-as-a-Service
Servers provided by Amazon EC2 + Rackspace
Storage provided by Amazon S3 + Rackspace
We are (almost) independent of platform provider
Regular Ubuntu 10.04 installations (LTS)
Virtual private servers with full access
New instances can be spun up in a matter of minutes
Different instance types:
Memory, processing, HPC (CPU+network)
Enable modular infrastructure (every service on small dedicated
instances)
16. Infrastructure-as-a-Service
3-tier software architecture
Frontend, Backend, Database
State-less tiers
(support: session sharing service)
Security groups for firewalling
Tier-based load balancing
Modular infrastructure allows for
efficient scaling and load balacing
17. Infrastructure-as-a-Service
Infrastructure as code
Programmable
Testable
Deployable
Reliability issue – full recovery from:
Repository (code+infrastructure description)
Application state backup (database)
’Bare metal resources’
Automated deployment and test
of servers for infrastructure
Infrastructure connections programmed
into deployment process
Entities register automatically with our
monitoring tool for complete overview
18. Infrastructure-as-a-Service
Deployable infrastructure + cloud = rapid replication
Infrastructure replicated into different environments
Environments may be used short or long term
Manual triggering of automated environment deployment
Produc6on
Sandbox
Staging
1
Staging
2
Staging
n
22. Self-organizing infrastructure
Can be seen as a traditional control problem
Zabbix monitors load on each tier (sensor)
ControlTier automatically spawns new servers
when needed (actuator)
Load-balancing / fail-over
Elastic load balancer (Amazon service) distributes load to pool of
servers (Frontend)
Own customized application-based request distribution schema
(backend, database, proxies)
Automatically removes servers when not needed
23. Continuous deployment (integration/delivery)
Integration and release are frightening
tasks
Huge, complex, long lasting tasks that are difficult to
estimate in time and risk
Our continuous processes
Automated build on each commit (fast) (Hudson)
Automated test of each build (fast)
Tested in freshly spawned staging environments
unit-tests, Selenium
Automated test of deployment process after each build
Automated delivery of each successful build
Release early, release often
24. Continuous delivery into a self-organizing infrastructure
Classes of changes
Purely code (bugfixes)
Direct into production
Database schema changes (features)
Into new production environment redirection
Environment changes (major features)
Environment specification changed
Into new production environment redirection
Monitoring of product KPIs
Ex. If signup rate or invoice rate decrease, something is wrong with the
latest deploy
Roll-back mechanisms for all scenarios
Old references and environments are preserved
Feature bits – everything is in the code, either on or off
Segmented roll-out, A/B testing