The Cloud Foundry APIs provide a rich set of information for anyone trying to gain insight into the inner workings of the platform. This presentation explores some of the APIs, and provides some practical techniques for non-coder ways of interacting with those APIs.
1. EXPLORINGTHE CLOUD FOUNDRY API
FOR FUN AND OPS
Josh Kruck
professional doer of things
github.com/krujos
@krujos
2. WHO AM I?
• Today
• pivotal customer success
• cf cli plugin author
• service broker author
• lots of little utilities for getting shit done
• Past
• a lot of storage and application management
• Why
• use software to make work more fun and be happier
8. SOME IDEAS
• apps (dashboard, provisioner, automation, crypt keeper)
• cli plugins
• scripts (why aren’t you writing a cli plugin?)
• A one time thing? (jq will change your life)
11. Service Brokers(more of a spec really)
Lots of other people talk about service brokers and how to author them, i won’t be covering them.
12. UAA(User Account and Authentication Server)
This is where we go to login. If you’re writing CLI plugins and scripts, it’s a non issue, but for any kind of app we’ll have to deal with it.
16. but wait!
Router(eventually)
A feature that’s coming down the pipe is “router services”. While I don’t have anything to say on this today it is something to be aware of. It will allow us to do some inline
things at the router that we previously had to insert between our LB’s and routers.
18. UAA(User Account and Authentication Server)
This is where we go to login. If you’re writing CLI plugins and scripts, it’s a non issue, but for any kind of app we’ll have to deal with it.
19. WHEN/WHY DO WE INTERACT WITH
UAA?
• When we’re writing an application and DON’T want to inherit the
credentials of the user
• Dashboards, system automation and on boarding
System automation examples:
Clean up orphaned services
Check for rouge users
Reports (new URL’s for nessus, apps pushed in the last day, crash summaries… whatevs).
20. WHEN/WHY DO WE INTERACT WITH
UAA?
• When we’re writing an (web) application and we DO want to inherit
the credentials of the user
• cli plugins and scripts can usually get this for free
System automation examples:
Onboarding, cleanup, CI stuff (slackbots)
21. UN HUMANS
• For apps that do not act on behalf of a human we use
client_credentials
• See the OAuth 2.0 spec for more information
• https://tools.ietf.org/html/rfc6749#section-1.3.4
• http://tools.ietf.org/html/rfc6749#section-4.4
If you google, you will find people who have translated this into somewhat more complicated english than the spec provides.
23. auth = requests.auth.HTTPBasicAuth(client_id, client_secret)
r = requests.post(
url=uaa_uri,
headers={'accept': 'application/json'},
params={'grant_type': ‘client_credentials'},
auth=client_auth)
PYTHON
both of those examples assume certificate validation, see the docs for skipping it.
24. ➜ curl “https://user:pass@uaa.mycf.xip.io/oauth/token" -d
grant_type=client_credentials -X POST | jq “.”
{
"access_token": “eyJhbG…C6Fvww”,
"token_type": "bearer",
"expires_in": 43199,
"scope": "doppler.firehose",
"jti": "2d56398d-ba65-4dbd-9f43-4d1a302709a1"
}
STOP WITHYOURTRICKERY AND GIVE
ME A SHELL MAN
25. THE IMPORTANTTHINGS
1. UAA exposes an API protected by HTTP Basic Auth that gives us a token
http(s)://uaa.example.com/oauth/token
2. We use a client_id & client_secret for auth
3. We pass it the grant type as a parameter
grant_type=client_credentials
4. It gives us back a token and time to live
note that you can indeed send your credentials in clear text. that’s a bad idea.
you can, and should prevent http requests from getting to uaa as your traffic can be sniffed.
26. UAA IS FOR PEOPLETOO?
• CLI Plugins you’ve already assumed the identity of the user
• Scripts should use & assume the identity of the logged in user via cf
curl
• password_grant otherwise (but srsly, why do it the hard way?)
• Web apps use the regular OAuth flow
For applications that act on behalf of humans we have a few options.
For scripts cf curl is a good option as it means you can avoid (hopefully) managing usernames and passwords in the scripts.
27. require 'bundler'
require 'sinatra'
require 'omniauth'
require 'omniauth-uaa-oauth2'
use Rack::Session::Cookie, :secret => ‘its a secret'
use OmniAuth::Builder do
provider :cloudfoundry, 'app', 'appclientsecret',
{:auth_server_url => “https://login.run.cf.io", :token_server_url => “https://
uaa.run.cf.io"}
end
class App < Sinatra::Base
get '/' do
<ul><li><a href='/auth/cloudfoundry'>Sign in with Cloud Foundry</a></li></ul>
end
get '/auth/cloudfoundry/callback' do
content_type 'application/json'
request.env['omniauth.auth'].to_hash.to_json rescue "No Data"
end
get '/auth/failure' do
content_type 'text/plain'
request.env['omniauth.auth'].to_hash.inspect rescue "No Data"
end
end
run App.new
This one doesn’t do a redirect like I normally do, use has to click the auth link.
29. WHAT ISTHE FIREHOSE?
• The firehose is where we go to listen
• Combined stream of logs from all apps, plus metrics data from CF
components
• Exposed as a web socket endpoint
• Accessed via doppler.firehose scope
The scope means it’s unlikely your user accounts have access to it.
Login w/ admin
Or create user in uaac.
30. WHATS INTHE FIREHOSE
• A lot, but not everything
• https://github.com/cloudfoundry/loggregator/wiki/Firehose-Metric-Catalog
• https://docs.google.com/spreadsheets/d/176yIaJChXEmvm-
CjybmwopdRGQfDGrSzo3J_Mx8mMnk/edit#gid=472741134
• heartbeats are being removed (rejoice!)
• https://lists.cloudfoundry.org/archives/list/cf-dev@lists.cloudfoundry.org/message/
EPTBPAJLBV4LAG72GOSK3KYO7MKZPL7B/
I find the most interesting things in the firehose to be http related. I can tell the status of every request in and out of the system, as well as it’s response time. This helps
me paint a picture of “normal”. It also helps me figure out “lights are on but no one’s home” situations. CF health check just make a connection on a socket, they don’t
check http response codes. So the firehose http metrics give me a good view of what my users are seeing.
38. WHATS INTERESTING
timestamp:1439756656563113538 httpStartStop:<startTimestamp:1439756656523306795
stopTimestamp:1439756656563113538 peerType:Client method:POST uri:"api.10.244.0.34.xip.io/
v2/apps" remoteAddress:"10.0.2.15:43117" userAgent:"go-cli 6.11.2-2a26d55 / darwin"
statusCode:201 instanceId:"10.244.0.134:9022" >
timestamp:1439756656712256742 httpStartStop:<startTimestamp:1439756656666492905
stopTimestamp:1439756656712256742 peerType:Client method:PUT uri:"api.10.244.0.34.xip.io/
v2/apps/d2cd4770-64e0-44fe-a997-f2e7e6dc5c79/routes/15879527-b813-4165-9b04-8dc3f5af2a31"
remoteAddress:"10.0.2.15:43129" userAgent:"go-cli 6.11.2-2a26d55 / darwin" statusCode:201
instanceId:"10.244.0.134:9022" >
timestamp:1439756656786930549 httpStartStop:<startTimestamp:1439756656765375916
stopTimestamp:1439756656786930549 peerType:Client method:PUT uri:"api.10.244.0.34.xip.io/
v2/apps/d2cd4770-64e0-44fe-a997-f2e7e6dc5c79/bits" remoteAddress:"10.0.2.15:43133"
userAgent:"go-cli 6.11.2-2a26d55 / darwin" statusCode:201 instanceId:"10.244.0.134:9022" >
timestamp:1439756663113749547 httpStartStop:<startTimestamp:1439756661872770559
stopTimestamp:1439756663113749547 peerType:Client method:PUT uri:"api.10.244.0.34.xip.io/
v2/apps/d2cd4770-64e0-44fe-a997-f2e7e6dc5c79" remoteAddress:"10.0.2.15:43167"
userAgent:"go-cli 6.11.2-2a26d55 / darwin" statusCode:201 instanceId:"10.244.0.134:9022" >
Lets make sure we’re tracing one user’s experience through the system.
note the port changed? Why? I suspect the CLI had to open a new connection..
39. WHATS INTERESTING
timestamp:1439756656563113538 httpStartStop:<startTimestamp:1439756656523306795
stopTimestamp:1439756656563113538 peerType:Client method:POST uri:"api.10.244.0.34.xip.io/
v2/apps" remoteAddress:"10.0.2.15:43117" userAgent:"go-cli 6.11.2-2a26d55 / darwin"
statusCode:201 instanceId:"10.244.0.134:9022" >
timestamp:1439756656712256742 httpStartStop:<startTimestamp:1439756656666492905
stopTimestamp:1439756656712256742 peerType:Client method:PUT uri:"api.10.244.0.34.xip.io/
v2/apps/d2cd4770-64e0-44fe-a997-f2e7e6dc5c79/routes/15879527-b813-4165-9b04-8dc3f5af2a31"
remoteAddress:"10.0.2.15:43129" userAgent:"go-cli 6.11.2-2a26d55 / darwin" statusCode:201
instanceId:"10.244.0.134:9022" >
timestamp:1439756656786930549 httpStartStop:<startTimestamp:1439756656765375916
stopTimestamp:1439756656786930549 peerType:Client method:PUT uri:"api.10.244.0.34.xip.io/
v2/apps/d2cd4770-64e0-44fe-a997-f2e7e6dc5c79/bits" remoteAddress:"10.0.2.15:43133"
userAgent:"go-cli 6.11.2-2a26d55 / darwin" statusCode:201 instanceId:"10.244.0.134:9022" >
timestamp:1439756663113749547 httpStartStop:<startTimestamp:1439756661872770559
stopTimestamp:1439756663113749547 peerType:Client method:PUT uri:"api.10.244.0.34.xip.io/
v2/apps/d2cd4770-64e0-44fe-a997-f2e7e6dc5c79" remoteAddress:"10.0.2.15:43167"
userAgent:"go-cli 6.11.2-2a26d55 / darwin" statusCode:201 instanceId:"10.244.0.134:9022" >
lets distill further, to just client operations, we can trace one user’s experience through the system.
How long did their operation take? (6.5 seconds), firehose timestamps are nano seconds.
40. WHATS INTERESTING
timestamp:1439756656563113538 httpStartStop:<startTimestamp:1439756656523306795
stopTimestamp:1439756656563113538 peerType:Client method:POST uri:"api.10.244.0.34.xip.io/
v2/apps" remoteAddress:"10.0.2.15:43117" userAgent:"go-cli 6.11.2-2a26d55 / darwin"
statusCode:201 instanceId:"10.244.0.134:9022" >
timestamp:1439756656712256742 httpStartStop:<startTimestamp:1439756656666492905
stopTimestamp:1439756656712256742 peerType:Client method:PUT uri:"api.10.244.0.34.xip.io/
v2/apps/d2cd4770-64e0-44fe-a997-f2e7e6dc5c79/routes/15879527-b813-4165-9b04-8dc3f5af2a31"
remoteAddress:"10.0.2.15:43129" userAgent:"go-cli 6.11.2-2a26d55 / darwin" statusCode:201
instanceId:"10.244.0.134:9022" >
timestamp:1439756656786930549 httpStartStop:<startTimestamp:1439756656765375916
stopTimestamp:1439756656786930549 peerType:Client method:PUT uri:"api.10.244.0.34.xip.io/
v2/apps/d2cd4770-64e0-44fe-a997-f2e7e6dc5c79/bits" remoteAddress:"10.0.2.15:43133"
userAgent:"go-cli 6.11.2-2a26d55 / darwin" statusCode:201 instanceId:"10.244.0.134:9022" >
timestamp:1439756663113749547 httpStartStop:<startTimestamp:1439756661872770559
stopTimestamp:1439756663113749547 peerType:Client method:PUT uri:"api.10.244.0.34.xip.io/
v2/apps/d2cd4770-64e0-44fe-a997-f2e7e6dc5c79" remoteAddress:"10.0.2.15:43167"
userAgent:"go-cli 6.11.2-2a26d55 / darwin" statusCode:201 instanceId:"10.244.0.134:9022" >
What did they do what was the output?
First one crates the app.
second one uploads the bits
third one does a state change to start apps.
41. WHY ISTHAT INTERESTING?
• If I can learn what normal interaction patterns are, I can identify
abnormal ones and investigate
• Are apps failing to start after a push?
• For all teams?
to me anyway.
this lets us identify and view actions across the entirety of the user base.
Could you do this with platform logs? Probably. The beauty of the firehose is it gives you all of it, I omitted the logs from my example for readability and dramatic effect.
43. WHAT ISTHE COLLECTOR
• The collector is the “legacy” aggregator of CF metrics
• No log messages
• Grabbed all the /varz and /healthz data
• The collector still has a lot of relevant runtime information in it
• available_stagers, healthy, system.cpu.wait
Most of the collector usage is for dashboards.
44. WHAT ISTHE COLLECTOR FOR
• It’s also how we export metrics in open source via historian plugins
• https://github.com/cloudfoundry/collector/tree/master/lib/collector/
historian
• Vendors (well Pivotal at least) have their own spin
• Pivotal Ops Metrics (JMX)
The historian plugins change the way data is presented. They are usually monitoring solution specific.
45. COLLECTOR DOWNSIDE
• Vendors don’t install all the plugins in their distribution
• Consumption can be irritating, lots of guesswork
• You may end up pretending to be a historian plugin, or consuming
ones output.
47. CLOUD CONTROLLER
• The CC API is what we talk to the most
• api.run.pivotal.io
• The CC API is where we go to act and report
48. THE CC API IS UNSURPRISING
• Which is to say, the api has all of the things you would expect
• /v2/apps
• /v3/apps (experimental)
• /v2/organizations
• /v2/events
• /v2/spaces
or kinda boring
49. API DETAILS
• the v3 api is totes HATEOAS compliant!!!
• the v2 api has some links conventions
50. THE APITEAM COMMITTEDTO
MAINTAINING BACKWARDS COMPATIBILITY
• v3 api’s will have v2 compatibility layers
• I was there, they said it at cf summit, it’s like they wrote it in their
own blood
51. API DETAILS
• The v2 api is very, very, very verbose
• The v3 api is more targeted
52. API DETAILS
• v3 api is fun to talk about, but as of today much of it doesn’t exist
outside of documentation.
• we will see some examples later.
61. SO HOW DO I BUILD ATHING?
• it starts with cf curl and jq
• or the firehose and grep
62. • lets see what the firehose has to say about started apps?
export DOPPLER_ADDR=wss://doppler.10.244.0.34.xip.io:443
CF_ACCESS_TOKEN=`cf oauth-token | grep bearer` ./firehose_sample | grep -i started
cf stop hello-node
cf start hello-node
63. • how about orphaned user provided services?
cf cups my-svc -p '{"foo":"bar"}'
cf curl /v2/user_provided_service_instances
cf curl `cf curl /v2/user_provided_service_instances | jq -r '.resources[0].entity.service_bindings_url'` | jq ‘.total_results'
Think about what we would have had to do to report on that with the CLI?
64. • lets look at something more complex.
• how many ai’s are you running?
• How does the usage-report plugin work?
• https://github.com/krujos/usagereport-plugin
demo the plugin
sometimes its about mashing things up
Why is how many ai’s your running a hard question to answer?
Why is how many ai’s your running an important question to answer?
As someone who’s paying for IT charge back, am I paying too much?
How much ram am I consuming, are my quotas right?
make note of the lack of auth, plugin assumes current user, works for everyone based on their cf access.
magic starts here: https://github.com/krujos/usagereport-plugin/blob/master/usagereport.go#L57
https://github.com/krujos/usagereport-plugin/blob/master/apihelper/apihelper.go