This document discusses delivering developer tools at scale for Oracle Bare Metal Cloud Services. It outlines the challenges of supporting many programming languages, tools, services, features and rapid innovation with a small team. The solutions discussed are using Swagger to declaratively describe APIs, open sourcing tools to engage the community, and maintaining API consistency. It also addresses handling multiple release scopes by using custom fields in the Swagger specification.

1. Delivering Developer
Tools at Scale
Joe Levy (@jodoglevy) – Principal Technical Program Manager
Mike Ross – Senior Software Engineer
Oracle Bare Metal Cloud Services, Developer Experience

2. Oracle Bare Metal Cloud Services
• Infrastructure as a service (IaaS)
 Compute (physical and virtual machines)
 Block Storage
 Virtual Networking
 Identity
 Load Balancing
 Object Storage
 Audit
 Database
• Built by ex-cloud folks
• More info:
https://cloud.oracle.com/en_US/bare-
metal

3. Demo
Bare Metal Cloud Services

4. March 2015
Developer Experience:
Support devs where they are (languages)

5. Developer Experience:
Support devs where they are (tools)
https://www.linux.com/news/top-open-source-cloud-projects-2014

6. Developer Experience (aka fill all the boxes)
Compute Block
Storage
Virtual
Networking
Load
Balancing
Object
Storage
Audit Database
Java
SDK
Python
SDK
Ruby
SDK
CLI
HDFS
Connector
Chef Knife
Plugin
Terraform
Provider

7. Developer Experience: Progress to date
Compute Block
Storage
Virtual
Networking
Load
Balancing
Object
Storage
Audit Database
Java
SDK
In preview
Python
SDK
N/A
Ruby
SDK
In preview N/A
CLI N/A In preview
HDFS
Connector
N/A N/A N/A N/A N/A N/A
Chef Knife
Plugin
N/A N/A N/A N/A
Terraform
Provider
N/A

8. Developer Experience: Progress to date
Boxes: 7 dev tools X 7 services = 49
Team: 6 devs

9. Developer Experience: The future
Compute Block
Storage
Virtual
Networking
Load
Balancing
Object
Storage
Audit Database
Java
SDK
In preview
Python
SDK
N/A
Ruby
SDK
In preview N/A
CLI N/A In preview
HDFS
Connector
N/A N/A N/A N/A N/A N/A
Chef Knife
Plugin
N/A N/A N/A N/A
Terraform
Provider
N/A
Fortran
SDK
Logo
SDK
…

10. Developer Experience: The future
Boxes: N dev tools X 7 services = 7N
Team: 6 devs

11. Developer Experience: The future (for real)
Compute Block
Storage
Virtual
Networking
Load
Balancing
Object
Storage
Audit Database Puppies
as a
service
Haircuts
as a
service
…
Java
SDK
In preview
Python
SDK
N/A
Ruby
SDK
In
preview
N/A
CLI N/A In preview
HDFS
Connector
N/A N/A N/A N/A N/A N/A
Chef Knife
Plugin
N/A N/A N/A N/A
Terraform
Provider
N/A
Fortran
SDK
Logo
SDK
…

12. Developer Experience: The future (for real)
Boxes: N dev tools X M services
Team: 6 devs

13. Developer Experience: The future
(for real, for real)
• Every service is actually a collection of many resources
• More resources added to each service over time, all needing dev tools support
• VCN service resources:
• Cpe
• CrossConnect
• CrossConnectGroup
• CrossConnectMapping
• CrossConnectLocation
• CrossConnectPortShapeSpeed
• CrossConnectStatus
• DhcpOption
• Drg
• DrgAttachment
• EgressSecurityRule
• FastConnectProviderService
• IcmpOptions
• IngressSecurityRule
• InternetGateway
• IpSecConnection
• PortRange
• RouteRule
• RouteTable
• SecurityList
• Subnet
• TcpOptions
• TunnelConfig
• TunnelStatus
• UdpOptions
• Vcn
• VirtualCircuit
• VirtualCircuitBandwithShape
• Vnic
• VnicAttachment

14. Developer Experience: The future
(for real, for real)
Compute Block
Storage
Virtual
Networking
Load
Balancing
Object
Storage
Audit Database Puppies
as a
service
Haircuts
as a
service
…
Java
SDK
In preview
Python
SDK
N/A
Ruby
SDK
In
preview
N/A
CLI N/A In preview
HDFS
Connector
N/A N/A N/A N/A N/A N/A
Chef Knife
Plugin
N/A N/A N/A N/A
Terraform
Provider
N/A
Fortran
SDK
Logo
SDK
…

15. Developer Experience: The future
(for real, for real)
Boxes: N dev tools X M services X Z features
Team: 6 devs

16. Developer Experience: The future
(for real, for real, for real)
• Pace of innovation rapidly increasing
• 1 year ago org size: 300
• 1 year ago Developer Experience team size: 4 devs
• Today org size: 850 (+185%)
• Today Developer Experience team size: 6 devs (+50%)
• 1 year from now org size: 1,500 (+80%)
• 1 year from now Developer Experience team size: 9 devs (+50%)
• We must maintain feature parity, despite team size not scaling at same pace as org at large

17. Developer Experience: The future
(for real, for real, for real)
Boxes: N dev tools X M services X Z features X Y pace of innovation
Team: 6 devs

18. Developer Experience: The future
(for real, for real, for real, for real)
• Different release scopes:
• Internal-only
• Our services call each other
• For their own infrastructure management
• To enable some cross-service customer use cases
• Some of the APIs / parameters they use are not meant for public consumption
• Internal-preview
• Let Oracle and special customers try features before we’re ready to publicize them
• Public-preview
• Let public customers try out new features before we certify them as production-ready
• Public
• Release a feature as production-ready, to the public

19. Developer Experience: The future
(for real, for real, for real, for real)
Boxes: N dev tools X M services X Z features X Y pace of innovation X W release scopes
Team: 6 devs

20. The problem:
How to keep up with work of this scale!?
Boxes: N dev tools X M services X Z features X Y pace of innovation X W release scopes
Team: 6 devs

21. The solution
• Swagger
• Open Sourcing
• API Consistency

22. Swagger
• https://swagger.io
• Declaratively describe REST APIs in a machine-readable way
• Manage your services’ interface specifications as you would your code
• Open Source
• Useful for:
 Generating REST API documentation
 Code reviewing REST APIs (to ensure good REST use, and consistency across BMCS APIs)
 Automated & manual
 Creating automated pipelines around service specifications
 Service (scaffolding) generation
 Client generation! (open source generators exist for many languages)
 We generate:
 Java SDK
 Ruby SDK
 Python SDK
 CLI
 Generated Terraform provider (work in progress)

23. Swagger spec: Launch instance

24. Results when generated
• REST API docs
• Java SDK
• Python SDK
• Ruby SDK
• CLI: bmcs compute instance launch

25. Demo
Swagger and Bare Metal Cloud Services

26. Swagger: Cons & Gotchas
• Making modifications / polish to generated code
• Keeping backwards compatibility as the Swagger spec changes
• Contributions require understanding of metaprogramming and Swagger codegen, not
just language the dev tool was “written” in
• Non-language-related build tooling needed to contribute
 Ex: Maven (Java) required to build Python SDK

27. Open Sourcing
• Get the developer community involved
 Finding and fixing issues
 Adding new features
 Answering questions
 Improving documentation
• Create advocates for your products
• Find potential hires who are passionate about the area you work in
• Be a good developer citizen -- let others use your code as a basis for their projects / learning
• Other useful parts of GitHub:
 Archived releases
 Subscribe-able release notification channel
 Metrics
• Open sourced so far:
 Java SDK: https://github.com/oracle/bmcs-java-sdk
 Python SDK: https://github.com/oracle/bmcs-python-sdk
 Terraform Provider: https://github.com/oracle/terraform-provider-baremetal
 Chef BMCS Knife Plugin: https://github.com/oracle/knife-bmcs
 CLI (soon!)

28. Open Sourcing: Cons & Gotchas
• Syncing changes between GitHub repository and internal repository
• Keeping private things, private
• Accepting pull requests on generated code (since we codegen our code)

29. API Consistency
• Useful to the user
• Can build higher-level functionality once per dev tool (instead of once per resource), since
resources do things same way
 Ex: pagination
 BMCS list APIs return opc-next-page header, and take in that header’s value via query string: ?page=<token>
 Ex: waiters
 BMCS resources have a lifecycleState field to indicate creation / updating / deleting status
• Reduces need to special case generated code
 Ex: serializing OR filter semantics into an API call
 /cars?make=subaru&make=toyota
• Makes testing easier
 When resources are managed similarly, test cases are similar too
 Less investigation / writing of new code, more porting existing tests for a new resource type
• Swagger makes API consistency much easier!
 Swagger = service interface specifications as code
 Diffs
 Code review tools
 Automated review

30. One problem area left:
Multiple release scopes
N dev tools X M services X Z features X Y pace of innovation X W release scopes
• Internal-only
• Our services call each other
• For their own infrastructure management
• To enable some cross-service customer use cases
• Some of the APIs / parameters they use are not meant for public consumption
• Internal-preview
• Let Oracle and special customers try features before we’re ready to publicize them
• Public-preview
• Let public customers try out new features before we certify them as production-ready
• Public
• Release a feature as production-ready, to the public

31. Addressing the Problem:
Multiple release scopes
release
preview
some-feature-to-go-public
internal
some-never-public-feature
Swagger spec branching

32. Addressing the Problem:
Multiple release scopes
Swagger spec custom fields
CreatePuppyDetails:
properties:
name:
maxLength: 255
minLength: 1
type: string
ownerId:
maxLength: 255
minLength: 1
type: string
isPottyTrained:
type: boolean
x-obmcs-preview-only: true
dogType:
type: string
canBreatheFire:
type: boolean
x-obmcs-internal-only: true
• x-obmcs-preview-only
• Filtered out in public dev tool
generation
• Gets a preview disclaimer in its
description in preview dev tools
• x-obmcs-internal-only
• Filtered out in preview and public dev
tool generation

33. Q/A
P.S. – we’re hiring! Interested in this kind of thing?
Email me at joe.levy@oracle.com or come find me afterwards