SlideShare ist ein Scribd-Unternehmen logo
1 von 21
Downloaden Sie, um offline zu lesen
Ambari
API's and SPI's of Ambari
Tom Beerbower @ Hortonworks

April 02, 2013




© Hortonworks Inc. 2013       Page 1
Agenda
• API Overview
• Monitoring
• API Constructs
• Management
• Error Handling
• SPI Overview
• SPI Usage




     © Hortonworks Inc. 2013   Page 2
API Overview - Features
• REST
• Monitoring and Management of Hadoop Cluster
• Partial Response
• Query Predicates




     © Hortonworks Inc. 2013                    Page 3
Monitoring
• Read state of Hadoop resources.
Example:
api/v1/clusters/cluster1/services/HDFS/components/NAMENODE
{
    "href" : "http://ec2…1/services/HDFS/components/NAMENODE",
    "metrics" : {
      "boottime" : 1.364912931E9,
      "process" : {
        "proc_total" : 752.333333333,
        "proc_run" : 1.64444444444
      },
      "rpc" : {
        "rpcAuthorizationSuccesses" : 141,
        "SentBytes" : 251758,
        "rpcAuthorizationFailures" : 0,…



        © Hortonworks Inc. 2013                              Page 4
API Overview – Resource Types
• Collection Resource: This resource type doesn’t refer
  to any specific resource; rather it refers to a collection
  of resources.
  For example: api/v1/clusters/cluster1/services
  Returns a collection of services


• Instance Resource: This resource type refers to a
  single specific resource.
  For example: api/v1/clusters/cluster1/services/HDFS
  Refers to the service resource identified by the id “HDFS”.




      © Hortonworks Inc. 2013                                   Page 5
API Overview - Response
• JSON
• Collection resource
   • Always includes request href.
   • Inlcludes “items” of the collection.
   • Each item includes href and primary id fields.
• Instance resource
   •   Always includes request href.
   •   Always includes primary id fields.
   •   May include metrics and properties for instance.
   •   May include href and primary id fields for sub-resources.




         © Hortonworks Inc. 2013                                   Page 6
API Overview - Response
Collection Resource Response Example:
api/v1/clusters/cluster1/services/
{
    "href" : "http://ec2…api/v1/clusters/cluster1/services/",
    "items" : [
      {
        "href" : "http://ec2…/clusters/cluster1/services/HDFS",
        "ServiceInfo" : {
          "cluster_name" : "cluster1",
          "service_name" : "HDFS"
        }
      },
      {
        "href" : "http://ec2…/cluster1/services/MAPREDUCE",…



        © Hortonworks Inc. 2013                              Page 7
API Overview - Response
Instance Resource Response Example:
api/v1/clusters/cluster1
{
  "href" : "http://ec2...com:8080/api/v1/clusters/cluster1",
  "Clusters" : {
     "cluster_name" : "cluster1",
     "cluster_id" : 1,
     "version" : "HDP-1.2.1"
  },
  "services" : [
     {
       "href" :
"http://ec2...com:8080/api/v1/clusters/cluster1/services/MAPR
EDUCE",
       "ServiceInfo" : {
         "cluster_name" : "cluster1",
         "service_name" : "MAPREDUCE"
       }
     }, …

       © Hortonworks Inc. 2013                             Page 8
API Constructs – Partial Response
• Used to control which fields are returned by a query.
  – restrict which fields are returned.
  – reach down and return data from sub-resources.
• Properties, categories and sub-resources can be
  specified.
• The wildcard ‘*’ can be used to show all…
  – categories, fields and sub-resources.
  – provides „expand‟ functionality for sub-components.
• Primary id fields of a resource are always shown
  regardless of the specifies partial response.




     © Hortonworks Inc. 2013                              Page 9
API Constructs – Partial Response
Query Specific Field Example:
api/v1/clusters/cluster1/services/MAPREDUCE/components/JOBTRACKE
R?fields=metrics/rpc/SentBytes

{
  "href" :
"http://…/api/v1/clusters/cluster1/services/MAPREDUCE/components/
JOBTRACKER?fields=metrics/rpc/SentBytes",
  "metrics" : {
     "rpc" : {
       "SentBytes" : 2763921
     }
  },
  "ServiceComponentInfo" : {
     "cluster_name" : "cluster1",
     "component_name" : "JOBTRACKER",
     "service_name" : "MAPREDUCE"
  }
}

      © Hortonworks Inc. 2013                                 Page 10
API Constructs – Partial Response
Expand Sub-Resource Example:
api/v1/clusters/cluster1/services/MAPREDUCE?fields=components/met
rics/jvm/gcCount
{
  "href" :
"http://…/services/MAPREDUCE?fields=components/metrics/jvm/gc
Count",
  …
  },
  "components" : [
     {
       "href" : "http://…/MAPREDUCE/components/JOBTRACKER",
       "metrics" : {
         "jvm" : {
           "gcCount" : 47
         }
       },…


      © Hortonworks Inc. 2013                                Page 11
API Constructs – Query Predicates
• Limits the set of resources returned by a query.
• Consists of at least one relational expression.
• Can only be applied to collection resources.
• Relational operators ( =, !=, <, >, <=, >= )
• Logical operators ( |, &, !)
• Functions (in(), isEmpty())
• Brackets can be used to provide explicit grouping of
  expressions. Expressions within brackets have the
  highest precedence.




     © Hortonworks Inc. 2013                         Page 12
API Constructs – Query Predicates
Query For Started Services Example:
api/v1/clusters/cluster1/services?ServiceInfo/state=STARTED
{
  "href" :
"http://…/clusters/cluster1/services?ServiceInfo/state=STARTE
D",
  "items" : [
    {
      "href" : "http://…/clusters/cluster1/services/NAGIOS",
      "ServiceInfo" : {
        "cluster_name" : "cluster1",
        "state" : "STARTED",
        "service_name" : "NAGIOS"
      }
    },…



      © Hortonworks Inc. 2013                                 Page 13
Management - Create
• Create a resource.
Example:
Create a cluster named „c1‟ with the property
„Clusters/version‟ = „HDP-1.2.0‟

POST http://…:8080/api/v1/clusters/c1
{
  "Clusters": {
    "version" : "HDP-1.2.0”
  }
}




      © Hortonworks Inc. 2013                   Page 14
Management - Update
• Update a resource.
Example:
Update the state of all „INSTALLED‟ services to be
„STARTED‟
PUT
http://…:8080/api/v1/clusters/c1/services?ServiceInf
o/state=INSTALLED
{
  "ServiceInfo": {
    "state" : "STARTED”
  }
}



     © Hortonworks Inc. 2013                         Page 15
Management - Delete
• Delete a resource.
Example:
Delete the cluster named „c1‟

DELETE http://…:8080/api/v1/clusters/c1




     © Hortonworks Inc. 2013              Page 16
Error Handling

 HTTP Code                       Description
 200                             OK
 400                             Bad Request
 401                             Unauthorized
 403                             Forbidden
 404                             Not Found
 500                             Internal Server Error




       © Hortonworks Inc. 2013                           Page 17
Error Handling - Examples
api/v1/clusters/BAD_CLUSTER_NAME

404 Not Found
{
    "status" : 404,
    "message" : "The requested resource doesn't exist:
    Cluster not found,clusterName=BAD_CLUSTER_NAME"
}

api/v1/clusters/cluster1?fields=BAD_FIELD

400 Bad Request
{
    "status" : 400,
    "message" : "The properties [BAD_FIELD] specified in the
    request or predicate are not supported for the resource
    type Cluster."
}


      © Hortonworks Inc. 2013                             Page 18
SPI Overview
• Service framework for handling requests for the
  various resources that it manages and monitors.
  /api/v1/clusters/c1/hosts/host1/host_components




             Handler




                    getResources(Resource.Type.HostComponent,
                                 request, predicate)
                                                                                     Cluster
                                                                                ResourceProvider
                                                                             Service
                                                                         ResourceProvider
                                                                        Host
       ClusterController
                                                                  ResourceProvider
                                                             Component
                                                           ResourceProvider
     getResources(request, predicate)
                                                     HostComponent
                                                    ResourceProvider




        © Hortonworks Inc. 2013                                                                    Page 19
SPI Use Case
• Pluggable resource providers allow API usage with a
  GSInstaller installed cluster.

                                                API




                                                SPI

                                              GSInstaller
                                      Default ResourceProviders
                                         ResourceProviders




                               Ambari DB                      GSInstaller Artifacts




     © Hortonworks Inc. 2013                                                          Page 20
Q&A




© Hortonworks Inc. 2013   Page 21

Weitere ähnliche Inhalte

Was ist angesagt?

Hortonworks SmartSense
Hortonworks SmartSenseHortonworks SmartSense
Hortonworks SmartSenseArtem Ervits
 
Ambari Views - Overview
Ambari Views - OverviewAmbari Views - Overview
Ambari Views - OverviewHortonworks
 
Apache Ambari - What's New in 1.4.3
Apache Ambari - What's New in 1.4.3Apache Ambari - What's New in 1.4.3
Apache Ambari - What's New in 1.4.3Hortonworks
 
zData BI & Advanced Analytics Platform + 8 Week Pilot Programs
zData BI & Advanced Analytics Platform + 8 Week Pilot ProgramszData BI & Advanced Analytics Platform + 8 Week Pilot Programs
zData BI & Advanced Analytics Platform + 8 Week Pilot ProgramszData Inc.
 
Apache Ambari - What's New in 1.7.0
Apache Ambari - What's New in 1.7.0Apache Ambari - What's New in 1.7.0
Apache Ambari - What's New in 1.7.0Hortonworks
 
Apache Ambari - What's New in 1.4.2
Apache Ambari - What's New in 1.4.2Apache Ambari - What's New in 1.4.2
Apache Ambari - What's New in 1.4.2Hortonworks
 
Apache Ambari - What's New in 1.5.0
Apache Ambari - What's New in 1.5.0Apache Ambari - What's New in 1.5.0
Apache Ambari - What's New in 1.5.0Hortonworks
 
Deploying and Managing Hadoop Clusters with AMBARI
Deploying and Managing Hadoop Clusters with AMBARIDeploying and Managing Hadoop Clusters with AMBARI
Deploying and Managing Hadoop Clusters with AMBARIDataWorks Summit
 
Apache Ambari: Past, Present, Future
Apache Ambari: Past, Present, FutureApache Ambari: Past, Present, Future
Apache Ambari: Past, Present, FutureHortonworks
 
Apache Ambari Stack Extensibility
Apache Ambari Stack ExtensibilityApache Ambari Stack Extensibility
Apache Ambari Stack ExtensibilityJayush Luniya
 
Hortonworks Technical Workshop: Apache Ambari
Hortonworks Technical Workshop:   Apache AmbariHortonworks Technical Workshop:   Apache Ambari
Hortonworks Technical Workshop: Apache AmbariHortonworks
 
2015 zData Inc. - Apache Ambari Overview
2015 zData Inc. - Apache Ambari Overview2015 zData Inc. - Apache Ambari Overview
2015 zData Inc. - Apache Ambari OverviewzData Inc.
 
Hortonworks Technical Workshop: Interactive Query with Apache Hive
Hortonworks Technical Workshop: Interactive Query with Apache Hive Hortonworks Technical Workshop: Interactive Query with Apache Hive
Hortonworks Technical Workshop: Interactive Query with Apache Hive Hortonworks
 
Apache Ambari: Simplified Hadoop Cluster Operation & Troubleshooting
Apache Ambari: Simplified Hadoop Cluster Operation & TroubleshootingApache Ambari: Simplified Hadoop Cluster Operation & Troubleshooting
Apache Ambari: Simplified Hadoop Cluster Operation & TroubleshootingJayush Luniya
 
Ambari: Agent Registration Flow
Ambari: Agent Registration FlowAmbari: Agent Registration Flow
Ambari: Agent Registration Flowjsposetti
 
Past, Present and Future of Apache Ambari
Past, Present and Future of Apache AmbariPast, Present and Future of Apache Ambari
Past, Present and Future of Apache AmbariArtem Ervits
 
Managing Enterprise Hadoop Clusters with Apache Ambari
Managing Enterprise Hadoop Clusters with Apache AmbariManaging Enterprise Hadoop Clusters with Apache Ambari
Managing Enterprise Hadoop Clusters with Apache AmbariJayush Luniya
 
Implementing SharePoint on Azure, Lessons Learnt from a Real World Project
Implementing SharePoint on Azure, Lessons Learnt from a Real World ProjectImplementing SharePoint on Azure, Lessons Learnt from a Real World Project
Implementing SharePoint on Azure, Lessons Learnt from a Real World ProjectK.Mohamed Faizal
 

Was ist angesagt? (20)

Hortonworks SmartSense
Hortonworks SmartSenseHortonworks SmartSense
Hortonworks SmartSense
 
Manage Hadoop Cluster with Ambari
Manage Hadoop Cluster with AmbariManage Hadoop Cluster with Ambari
Manage Hadoop Cluster with Ambari
 
Ambari Views - Overview
Ambari Views - OverviewAmbari Views - Overview
Ambari Views - Overview
 
An Overview of Ambari
An Overview of AmbariAn Overview of Ambari
An Overview of Ambari
 
Apache Ambari - What's New in 1.4.3
Apache Ambari - What's New in 1.4.3Apache Ambari - What's New in 1.4.3
Apache Ambari - What's New in 1.4.3
 
zData BI & Advanced Analytics Platform + 8 Week Pilot Programs
zData BI & Advanced Analytics Platform + 8 Week Pilot ProgramszData BI & Advanced Analytics Platform + 8 Week Pilot Programs
zData BI & Advanced Analytics Platform + 8 Week Pilot Programs
 
Apache Ambari - What's New in 1.7.0
Apache Ambari - What's New in 1.7.0Apache Ambari - What's New in 1.7.0
Apache Ambari - What's New in 1.7.0
 
Apache Ambari - What's New in 1.4.2
Apache Ambari - What's New in 1.4.2Apache Ambari - What's New in 1.4.2
Apache Ambari - What's New in 1.4.2
 
Apache Ambari - What's New in 1.5.0
Apache Ambari - What's New in 1.5.0Apache Ambari - What's New in 1.5.0
Apache Ambari - What's New in 1.5.0
 
Deploying and Managing Hadoop Clusters with AMBARI
Deploying and Managing Hadoop Clusters with AMBARIDeploying and Managing Hadoop Clusters with AMBARI
Deploying and Managing Hadoop Clusters with AMBARI
 
Apache Ambari: Past, Present, Future
Apache Ambari: Past, Present, FutureApache Ambari: Past, Present, Future
Apache Ambari: Past, Present, Future
 
Apache Ambari Stack Extensibility
Apache Ambari Stack ExtensibilityApache Ambari Stack Extensibility
Apache Ambari Stack Extensibility
 
Hortonworks Technical Workshop: Apache Ambari
Hortonworks Technical Workshop:   Apache AmbariHortonworks Technical Workshop:   Apache Ambari
Hortonworks Technical Workshop: Apache Ambari
 
2015 zData Inc. - Apache Ambari Overview
2015 zData Inc. - Apache Ambari Overview2015 zData Inc. - Apache Ambari Overview
2015 zData Inc. - Apache Ambari Overview
 
Hortonworks Technical Workshop: Interactive Query with Apache Hive
Hortonworks Technical Workshop: Interactive Query with Apache Hive Hortonworks Technical Workshop: Interactive Query with Apache Hive
Hortonworks Technical Workshop: Interactive Query with Apache Hive
 
Apache Ambari: Simplified Hadoop Cluster Operation & Troubleshooting
Apache Ambari: Simplified Hadoop Cluster Operation & TroubleshootingApache Ambari: Simplified Hadoop Cluster Operation & Troubleshooting
Apache Ambari: Simplified Hadoop Cluster Operation & Troubleshooting
 
Ambari: Agent Registration Flow
Ambari: Agent Registration FlowAmbari: Agent Registration Flow
Ambari: Agent Registration Flow
 
Past, Present and Future of Apache Ambari
Past, Present and Future of Apache AmbariPast, Present and Future of Apache Ambari
Past, Present and Future of Apache Ambari
 
Managing Enterprise Hadoop Clusters with Apache Ambari
Managing Enterprise Hadoop Clusters with Apache AmbariManaging Enterprise Hadoop Clusters with Apache Ambari
Managing Enterprise Hadoop Clusters with Apache Ambari
 
Implementing SharePoint on Azure, Lessons Learnt from a Real World Project
Implementing SharePoint on Azure, Lessons Learnt from a Real World ProjectImplementing SharePoint on Azure, Lessons Learnt from a Real World Project
Implementing SharePoint on Azure, Lessons Learnt from a Real World Project
 

Ähnlich wie Ambari Meetup: APIs and SPIs of Ambari

Apache Ambari - What's New in 1.6.0
Apache Ambari - What's New in 1.6.0Apache Ambari - What's New in 1.6.0
Apache Ambari - What's New in 1.6.0Hortonworks
 
Build a Node.js Client for Your REST+JSON API
Build a Node.js Client for Your REST+JSON APIBuild a Node.js Client for Your REST+JSON API
Build a Node.js Client for Your REST+JSON APIStormpath
 
REST API Best Practices & Implementing in Codeigniter
REST API Best Practices & Implementing in CodeigniterREST API Best Practices & Implementing in Codeigniter
REST API Best Practices & Implementing in CodeigniterSachin G Kulkarni
 
Meetup 12-12-2017 - Application Isolation on Kubernetes
Meetup 12-12-2017 - Application Isolation on KubernetesMeetup 12-12-2017 - Application Isolation on Kubernetes
Meetup 12-12-2017 - Application Isolation on Kubernetesdtoledo67
 
Simple REST with Dropwizard
Simple REST with DropwizardSimple REST with Dropwizard
Simple REST with DropwizardAndrei Savu
 
Slider: Applications on YARN
Slider: Applications on YARNSlider: Applications on YARN
Slider: Applications on YARNSteve Loughran
 
Innovate2014 Better Integrations Through Open Interfaces
Innovate2014 Better Integrations Through Open InterfacesInnovate2014 Better Integrations Through Open Interfaces
Innovate2014 Better Integrations Through Open InterfacesSteve Speicher
 
Getting started with Apollo Client and GraphQL
Getting started with Apollo Client and GraphQLGetting started with Apollo Client and GraphQL
Getting started with Apollo Client and GraphQLMorgan Dedmon
 
Build AWS CloudFormation Custom Resources (DEV417-R2) - AWS re:Invent 2018
Build AWS CloudFormation Custom Resources (DEV417-R2) - AWS re:Invent 2018Build AWS CloudFormation Custom Resources (DEV417-R2) - AWS re:Invent 2018
Build AWS CloudFormation Custom Resources (DEV417-R2) - AWS re:Invent 2018Amazon Web Services
 
From System Engineer to Gopher
From System Engineer to GopherFrom System Engineer to Gopher
From System Engineer to GopherI-Fan Wang
 
Introduction to CloudStack API
Introduction to CloudStack APIIntroduction to CloudStack API
Introduction to CloudStack APIKrunal Jain
 
Spring 3 - Der dritte Frühling
Spring 3 - Der dritte FrühlingSpring 3 - Der dritte Frühling
Spring 3 - Der dritte FrühlingThorsten Kamann
 
RESTful API-centric Universe
RESTful API-centric UniverseRESTful API-centric Universe
RESTful API-centric UniverseTihomir Opačić
 
AWS CloudFormation macros: Coding best practices - MAD201 - New York AWS Summit
AWS CloudFormation macros: Coding best practices - MAD201 - New York AWS SummitAWS CloudFormation macros: Coding best practices - MAD201 - New York AWS Summit
AWS CloudFormation macros: Coding best practices - MAD201 - New York AWS SummitAmazon Web Services
 
Kubernetes Architecture - beyond a black box - Part 1
Kubernetes Architecture - beyond a black box - Part 1Kubernetes Architecture - beyond a black box - Part 1
Kubernetes Architecture - beyond a black box - Part 1Hao H. Zhang
 
Building APIs in an easy way using API Platform
Building APIs in an easy way using API PlatformBuilding APIs in an easy way using API Platform
Building APIs in an easy way using API PlatformAntonio Peric-Mazar
 

Ähnlich wie Ambari Meetup: APIs and SPIs of Ambari (20)

Apache Ambari - What's New in 1.6.0
Apache Ambari - What's New in 1.6.0Apache Ambari - What's New in 1.6.0
Apache Ambari - What's New in 1.6.0
 
Build a Node.js Client for Your REST+JSON API
Build a Node.js Client for Your REST+JSON APIBuild a Node.js Client for Your REST+JSON API
Build a Node.js Client for Your REST+JSON API
 
REST APIs
REST APIsREST APIs
REST APIs
 
REST API Best Practices & Implementing in Codeigniter
REST API Best Practices & Implementing in CodeigniterREST API Best Practices & Implementing in Codeigniter
REST API Best Practices & Implementing in Codeigniter
 
Meetup 12-12-2017 - Application Isolation on Kubernetes
Meetup 12-12-2017 - Application Isolation on KubernetesMeetup 12-12-2017 - Application Isolation on Kubernetes
Meetup 12-12-2017 - Application Isolation on Kubernetes
 
Simple REST with Dropwizard
Simple REST with DropwizardSimple REST with Dropwizard
Simple REST with Dropwizard
 
Slider: Applications on YARN
Slider: Applications on YARNSlider: Applications on YARN
Slider: Applications on YARN
 
Azure cosmosdb
Azure cosmosdbAzure cosmosdb
Azure cosmosdb
 
Andrei shakirin rest_cxf
Andrei shakirin rest_cxfAndrei shakirin rest_cxf
Andrei shakirin rest_cxf
 
Innovate2014 Better Integrations Through Open Interfaces
Innovate2014 Better Integrations Through Open InterfacesInnovate2014 Better Integrations Through Open Interfaces
Innovate2014 Better Integrations Through Open Interfaces
 
Getting started with Apollo Client and GraphQL
Getting started with Apollo Client and GraphQLGetting started with Apollo Client and GraphQL
Getting started with Apollo Client and GraphQL
 
Build AWS CloudFormation Custom Resources (DEV417-R2) - AWS re:Invent 2018
Build AWS CloudFormation Custom Resources (DEV417-R2) - AWS re:Invent 2018Build AWS CloudFormation Custom Resources (DEV417-R2) - AWS re:Invent 2018
Build AWS CloudFormation Custom Resources (DEV417-R2) - AWS re:Invent 2018
 
From System Engineer to Gopher
From System Engineer to GopherFrom System Engineer to Gopher
From System Engineer to Gopher
 
Introduction to CloudStack API
Introduction to CloudStack APIIntroduction to CloudStack API
Introduction to CloudStack API
 
Spring 3 - Der dritte Frühling
Spring 3 - Der dritte FrühlingSpring 3 - Der dritte Frühling
Spring 3 - Der dritte Frühling
 
RESTful API-centric Universe
RESTful API-centric UniverseRESTful API-centric Universe
RESTful API-centric Universe
 
AWS CloudFormation macros: Coding best practices - MAD201 - New York AWS Summit
AWS CloudFormation macros: Coding best practices - MAD201 - New York AWS SummitAWS CloudFormation macros: Coding best practices - MAD201 - New York AWS Summit
AWS CloudFormation macros: Coding best practices - MAD201 - New York AWS Summit
 
AMIS Oracle ADF 12c Launch event 06 Steven Davelaar future REST features
AMIS Oracle ADF 12c Launch event 06 Steven Davelaar future REST features AMIS Oracle ADF 12c Launch event 06 Steven Davelaar future REST features
AMIS Oracle ADF 12c Launch event 06 Steven Davelaar future REST features
 
Kubernetes Architecture - beyond a black box - Part 1
Kubernetes Architecture - beyond a black box - Part 1Kubernetes Architecture - beyond a black box - Part 1
Kubernetes Architecture - beyond a black box - Part 1
 
Building APIs in an easy way using API Platform
Building APIs in an easy way using API PlatformBuilding APIs in an easy way using API Platform
Building APIs in an easy way using API Platform
 

Mehr von Hortonworks

Hortonworks DataFlow (HDF) 3.3 - Taking Stream Processing to the Next Level
Hortonworks DataFlow (HDF) 3.3 - Taking Stream Processing to the Next LevelHortonworks DataFlow (HDF) 3.3 - Taking Stream Processing to the Next Level
Hortonworks DataFlow (HDF) 3.3 - Taking Stream Processing to the Next LevelHortonworks
 
IoT Predictions for 2019 and Beyond: Data at the Heart of Your IoT Strategy
IoT Predictions for 2019 and Beyond: Data at the Heart of Your IoT StrategyIoT Predictions for 2019 and Beyond: Data at the Heart of Your IoT Strategy
IoT Predictions for 2019 and Beyond: Data at the Heart of Your IoT StrategyHortonworks
 
Getting the Most Out of Your Data in the Cloud with Cloudbreak
Getting the Most Out of Your Data in the Cloud with CloudbreakGetting the Most Out of Your Data in the Cloud with Cloudbreak
Getting the Most Out of Your Data in the Cloud with CloudbreakHortonworks
 
Johns Hopkins - Using Hadoop to Secure Access Log Events
Johns Hopkins - Using Hadoop to Secure Access Log EventsJohns Hopkins - Using Hadoop to Secure Access Log Events
Johns Hopkins - Using Hadoop to Secure Access Log EventsHortonworks
 
Catch a Hacker in Real-Time: Live Visuals of Bots and Bad Guys
Catch a Hacker in Real-Time: Live Visuals of Bots and Bad GuysCatch a Hacker in Real-Time: Live Visuals of Bots and Bad Guys
Catch a Hacker in Real-Time: Live Visuals of Bots and Bad GuysHortonworks
 
HDF 3.2 - What's New
HDF 3.2 - What's NewHDF 3.2 - What's New
HDF 3.2 - What's NewHortonworks
 
Curing Kafka Blindness with Hortonworks Streams Messaging Manager
Curing Kafka Blindness with Hortonworks Streams Messaging ManagerCuring Kafka Blindness with Hortonworks Streams Messaging Manager
Curing Kafka Blindness with Hortonworks Streams Messaging ManagerHortonworks
 
Interpretation Tool for Genomic Sequencing Data in Clinical Environments
Interpretation Tool for Genomic Sequencing Data in Clinical EnvironmentsInterpretation Tool for Genomic Sequencing Data in Clinical Environments
Interpretation Tool for Genomic Sequencing Data in Clinical EnvironmentsHortonworks
 
IBM+Hortonworks = Transformation of the Big Data Landscape
IBM+Hortonworks = Transformation of the Big Data LandscapeIBM+Hortonworks = Transformation of the Big Data Landscape
IBM+Hortonworks = Transformation of the Big Data LandscapeHortonworks
 
Premier Inside-Out: Apache Druid
Premier Inside-Out: Apache DruidPremier Inside-Out: Apache Druid
Premier Inside-Out: Apache DruidHortonworks
 
Accelerating Data Science and Real Time Analytics at Scale
Accelerating Data Science and Real Time Analytics at ScaleAccelerating Data Science and Real Time Analytics at Scale
Accelerating Data Science and Real Time Analytics at ScaleHortonworks
 
TIME SERIES: APPLYING ADVANCED ANALYTICS TO INDUSTRIAL PROCESS DATA
TIME SERIES: APPLYING ADVANCED ANALYTICS TO INDUSTRIAL PROCESS DATATIME SERIES: APPLYING ADVANCED ANALYTICS TO INDUSTRIAL PROCESS DATA
TIME SERIES: APPLYING ADVANCED ANALYTICS TO INDUSTRIAL PROCESS DATAHortonworks
 
Blockchain with Machine Learning Powered by Big Data: Trimble Transportation ...
Blockchain with Machine Learning Powered by Big Data: Trimble Transportation ...Blockchain with Machine Learning Powered by Big Data: Trimble Transportation ...
Blockchain with Machine Learning Powered by Big Data: Trimble Transportation ...Hortonworks
 
Delivering Real-Time Streaming Data for Healthcare Customers: Clearsense
Delivering Real-Time Streaming Data for Healthcare Customers: ClearsenseDelivering Real-Time Streaming Data for Healthcare Customers: Clearsense
Delivering Real-Time Streaming Data for Healthcare Customers: ClearsenseHortonworks
 
Making Enterprise Big Data Small with Ease
Making Enterprise Big Data Small with EaseMaking Enterprise Big Data Small with Ease
Making Enterprise Big Data Small with EaseHortonworks
 
Webinewbie to Webinerd in 30 Days - Webinar World Presentation
Webinewbie to Webinerd in 30 Days - Webinar World PresentationWebinewbie to Webinerd in 30 Days - Webinar World Presentation
Webinewbie to Webinerd in 30 Days - Webinar World PresentationHortonworks
 
Driving Digital Transformation Through Global Data Management
Driving Digital Transformation Through Global Data ManagementDriving Digital Transformation Through Global Data Management
Driving Digital Transformation Through Global Data ManagementHortonworks
 
HDF 3.1 pt. 2: A Technical Deep-Dive on New Streaming Features
HDF 3.1 pt. 2: A Technical Deep-Dive on New Streaming FeaturesHDF 3.1 pt. 2: A Technical Deep-Dive on New Streaming Features
HDF 3.1 pt. 2: A Technical Deep-Dive on New Streaming FeaturesHortonworks
 
Hortonworks DataFlow (HDF) 3.1 - Redefining Data-In-Motion with Modern Data A...
Hortonworks DataFlow (HDF) 3.1 - Redefining Data-In-Motion with Modern Data A...Hortonworks DataFlow (HDF) 3.1 - Redefining Data-In-Motion with Modern Data A...
Hortonworks DataFlow (HDF) 3.1 - Redefining Data-In-Motion with Modern Data A...Hortonworks
 
Unlock Value from Big Data with Apache NiFi and Streaming CDC
Unlock Value from Big Data with Apache NiFi and Streaming CDCUnlock Value from Big Data with Apache NiFi and Streaming CDC
Unlock Value from Big Data with Apache NiFi and Streaming CDCHortonworks
 

Mehr von Hortonworks (20)

Hortonworks DataFlow (HDF) 3.3 - Taking Stream Processing to the Next Level
Hortonworks DataFlow (HDF) 3.3 - Taking Stream Processing to the Next LevelHortonworks DataFlow (HDF) 3.3 - Taking Stream Processing to the Next Level
Hortonworks DataFlow (HDF) 3.3 - Taking Stream Processing to the Next Level
 
IoT Predictions for 2019 and Beyond: Data at the Heart of Your IoT Strategy
IoT Predictions for 2019 and Beyond: Data at the Heart of Your IoT StrategyIoT Predictions for 2019 and Beyond: Data at the Heart of Your IoT Strategy
IoT Predictions for 2019 and Beyond: Data at the Heart of Your IoT Strategy
 
Getting the Most Out of Your Data in the Cloud with Cloudbreak
Getting the Most Out of Your Data in the Cloud with CloudbreakGetting the Most Out of Your Data in the Cloud with Cloudbreak
Getting the Most Out of Your Data in the Cloud with Cloudbreak
 
Johns Hopkins - Using Hadoop to Secure Access Log Events
Johns Hopkins - Using Hadoop to Secure Access Log EventsJohns Hopkins - Using Hadoop to Secure Access Log Events
Johns Hopkins - Using Hadoop to Secure Access Log Events
 
Catch a Hacker in Real-Time: Live Visuals of Bots and Bad Guys
Catch a Hacker in Real-Time: Live Visuals of Bots and Bad GuysCatch a Hacker in Real-Time: Live Visuals of Bots and Bad Guys
Catch a Hacker in Real-Time: Live Visuals of Bots and Bad Guys
 
HDF 3.2 - What's New
HDF 3.2 - What's NewHDF 3.2 - What's New
HDF 3.2 - What's New
 
Curing Kafka Blindness with Hortonworks Streams Messaging Manager
Curing Kafka Blindness with Hortonworks Streams Messaging ManagerCuring Kafka Blindness with Hortonworks Streams Messaging Manager
Curing Kafka Blindness with Hortonworks Streams Messaging Manager
 
Interpretation Tool for Genomic Sequencing Data in Clinical Environments
Interpretation Tool for Genomic Sequencing Data in Clinical EnvironmentsInterpretation Tool for Genomic Sequencing Data in Clinical Environments
Interpretation Tool for Genomic Sequencing Data in Clinical Environments
 
IBM+Hortonworks = Transformation of the Big Data Landscape
IBM+Hortonworks = Transformation of the Big Data LandscapeIBM+Hortonworks = Transformation of the Big Data Landscape
IBM+Hortonworks = Transformation of the Big Data Landscape
 
Premier Inside-Out: Apache Druid
Premier Inside-Out: Apache DruidPremier Inside-Out: Apache Druid
Premier Inside-Out: Apache Druid
 
Accelerating Data Science and Real Time Analytics at Scale
Accelerating Data Science and Real Time Analytics at ScaleAccelerating Data Science and Real Time Analytics at Scale
Accelerating Data Science and Real Time Analytics at Scale
 
TIME SERIES: APPLYING ADVANCED ANALYTICS TO INDUSTRIAL PROCESS DATA
TIME SERIES: APPLYING ADVANCED ANALYTICS TO INDUSTRIAL PROCESS DATATIME SERIES: APPLYING ADVANCED ANALYTICS TO INDUSTRIAL PROCESS DATA
TIME SERIES: APPLYING ADVANCED ANALYTICS TO INDUSTRIAL PROCESS DATA
 
Blockchain with Machine Learning Powered by Big Data: Trimble Transportation ...
Blockchain with Machine Learning Powered by Big Data: Trimble Transportation ...Blockchain with Machine Learning Powered by Big Data: Trimble Transportation ...
Blockchain with Machine Learning Powered by Big Data: Trimble Transportation ...
 
Delivering Real-Time Streaming Data for Healthcare Customers: Clearsense
Delivering Real-Time Streaming Data for Healthcare Customers: ClearsenseDelivering Real-Time Streaming Data for Healthcare Customers: Clearsense
Delivering Real-Time Streaming Data for Healthcare Customers: Clearsense
 
Making Enterprise Big Data Small with Ease
Making Enterprise Big Data Small with EaseMaking Enterprise Big Data Small with Ease
Making Enterprise Big Data Small with Ease
 
Webinewbie to Webinerd in 30 Days - Webinar World Presentation
Webinewbie to Webinerd in 30 Days - Webinar World PresentationWebinewbie to Webinerd in 30 Days - Webinar World Presentation
Webinewbie to Webinerd in 30 Days - Webinar World Presentation
 
Driving Digital Transformation Through Global Data Management
Driving Digital Transformation Through Global Data ManagementDriving Digital Transformation Through Global Data Management
Driving Digital Transformation Through Global Data Management
 
HDF 3.1 pt. 2: A Technical Deep-Dive on New Streaming Features
HDF 3.1 pt. 2: A Technical Deep-Dive on New Streaming FeaturesHDF 3.1 pt. 2: A Technical Deep-Dive on New Streaming Features
HDF 3.1 pt. 2: A Technical Deep-Dive on New Streaming Features
 
Hortonworks DataFlow (HDF) 3.1 - Redefining Data-In-Motion with Modern Data A...
Hortonworks DataFlow (HDF) 3.1 - Redefining Data-In-Motion with Modern Data A...Hortonworks DataFlow (HDF) 3.1 - Redefining Data-In-Motion with Modern Data A...
Hortonworks DataFlow (HDF) 3.1 - Redefining Data-In-Motion with Modern Data A...
 
Unlock Value from Big Data with Apache NiFi and Streaming CDC
Unlock Value from Big Data with Apache NiFi and Streaming CDCUnlock Value from Big Data with Apache NiFi and Streaming CDC
Unlock Value from Big Data with Apache NiFi and Streaming CDC
 

Ambari Meetup: APIs and SPIs of Ambari

  • 1. Ambari API's and SPI's of Ambari Tom Beerbower @ Hortonworks April 02, 2013 © Hortonworks Inc. 2013 Page 1
  • 2. Agenda • API Overview • Monitoring • API Constructs • Management • Error Handling • SPI Overview • SPI Usage © Hortonworks Inc. 2013 Page 2
  • 3. API Overview - Features • REST • Monitoring and Management of Hadoop Cluster • Partial Response • Query Predicates © Hortonworks Inc. 2013 Page 3
  • 4. Monitoring • Read state of Hadoop resources. Example: api/v1/clusters/cluster1/services/HDFS/components/NAMENODE { "href" : "http://ec2…1/services/HDFS/components/NAMENODE", "metrics" : { "boottime" : 1.364912931E9, "process" : { "proc_total" : 752.333333333, "proc_run" : 1.64444444444 }, "rpc" : { "rpcAuthorizationSuccesses" : 141, "SentBytes" : 251758, "rpcAuthorizationFailures" : 0,… © Hortonworks Inc. 2013 Page 4
  • 5. API Overview – Resource Types • Collection Resource: This resource type doesn’t refer to any specific resource; rather it refers to a collection of resources. For example: api/v1/clusters/cluster1/services Returns a collection of services • Instance Resource: This resource type refers to a single specific resource. For example: api/v1/clusters/cluster1/services/HDFS Refers to the service resource identified by the id “HDFS”. © Hortonworks Inc. 2013 Page 5
  • 6. API Overview - Response • JSON • Collection resource • Always includes request href. • Inlcludes “items” of the collection. • Each item includes href and primary id fields. • Instance resource • Always includes request href. • Always includes primary id fields. • May include metrics and properties for instance. • May include href and primary id fields for sub-resources. © Hortonworks Inc. 2013 Page 6
  • 7. API Overview - Response Collection Resource Response Example: api/v1/clusters/cluster1/services/ { "href" : "http://ec2…api/v1/clusters/cluster1/services/", "items" : [ { "href" : "http://ec2…/clusters/cluster1/services/HDFS", "ServiceInfo" : { "cluster_name" : "cluster1", "service_name" : "HDFS" } }, { "href" : "http://ec2…/cluster1/services/MAPREDUCE",… © Hortonworks Inc. 2013 Page 7
  • 8. API Overview - Response Instance Resource Response Example: api/v1/clusters/cluster1 { "href" : "http://ec2...com:8080/api/v1/clusters/cluster1", "Clusters" : { "cluster_name" : "cluster1", "cluster_id" : 1, "version" : "HDP-1.2.1" }, "services" : [ { "href" : "http://ec2...com:8080/api/v1/clusters/cluster1/services/MAPR EDUCE", "ServiceInfo" : { "cluster_name" : "cluster1", "service_name" : "MAPREDUCE" } }, … © Hortonworks Inc. 2013 Page 8
  • 9. API Constructs – Partial Response • Used to control which fields are returned by a query. – restrict which fields are returned. – reach down and return data from sub-resources. • Properties, categories and sub-resources can be specified. • The wildcard ‘*’ can be used to show all… – categories, fields and sub-resources. – provides „expand‟ functionality for sub-components. • Primary id fields of a resource are always shown regardless of the specifies partial response. © Hortonworks Inc. 2013 Page 9
  • 10. API Constructs – Partial Response Query Specific Field Example: api/v1/clusters/cluster1/services/MAPREDUCE/components/JOBTRACKE R?fields=metrics/rpc/SentBytes { "href" : "http://…/api/v1/clusters/cluster1/services/MAPREDUCE/components/ JOBTRACKER?fields=metrics/rpc/SentBytes", "metrics" : { "rpc" : { "SentBytes" : 2763921 } }, "ServiceComponentInfo" : { "cluster_name" : "cluster1", "component_name" : "JOBTRACKER", "service_name" : "MAPREDUCE" } } © Hortonworks Inc. 2013 Page 10
  • 11. API Constructs – Partial Response Expand Sub-Resource Example: api/v1/clusters/cluster1/services/MAPREDUCE?fields=components/met rics/jvm/gcCount { "href" : "http://…/services/MAPREDUCE?fields=components/metrics/jvm/gc Count", … }, "components" : [ { "href" : "http://…/MAPREDUCE/components/JOBTRACKER", "metrics" : { "jvm" : { "gcCount" : 47 } },… © Hortonworks Inc. 2013 Page 11
  • 12. API Constructs – Query Predicates • Limits the set of resources returned by a query. • Consists of at least one relational expression. • Can only be applied to collection resources. • Relational operators ( =, !=, <, >, <=, >= ) • Logical operators ( |, &, !) • Functions (in(), isEmpty()) • Brackets can be used to provide explicit grouping of expressions. Expressions within brackets have the highest precedence. © Hortonworks Inc. 2013 Page 12
  • 13. API Constructs – Query Predicates Query For Started Services Example: api/v1/clusters/cluster1/services?ServiceInfo/state=STARTED { "href" : "http://…/clusters/cluster1/services?ServiceInfo/state=STARTE D", "items" : [ { "href" : "http://…/clusters/cluster1/services/NAGIOS", "ServiceInfo" : { "cluster_name" : "cluster1", "state" : "STARTED", "service_name" : "NAGIOS" } },… © Hortonworks Inc. 2013 Page 13
  • 14. Management - Create • Create a resource. Example: Create a cluster named „c1‟ with the property „Clusters/version‟ = „HDP-1.2.0‟ POST http://…:8080/api/v1/clusters/c1 { "Clusters": { "version" : "HDP-1.2.0” } } © Hortonworks Inc. 2013 Page 14
  • 15. Management - Update • Update a resource. Example: Update the state of all „INSTALLED‟ services to be „STARTED‟ PUT http://…:8080/api/v1/clusters/c1/services?ServiceInf o/state=INSTALLED { "ServiceInfo": { "state" : "STARTED” } } © Hortonworks Inc. 2013 Page 15
  • 16. Management - Delete • Delete a resource. Example: Delete the cluster named „c1‟ DELETE http://…:8080/api/v1/clusters/c1 © Hortonworks Inc. 2013 Page 16
  • 17. Error Handling HTTP Code Description 200 OK 400 Bad Request 401 Unauthorized 403 Forbidden 404 Not Found 500 Internal Server Error © Hortonworks Inc. 2013 Page 17
  • 18. Error Handling - Examples api/v1/clusters/BAD_CLUSTER_NAME 404 Not Found { "status" : 404, "message" : "The requested resource doesn't exist: Cluster not found,clusterName=BAD_CLUSTER_NAME" } api/v1/clusters/cluster1?fields=BAD_FIELD 400 Bad Request { "status" : 400, "message" : "The properties [BAD_FIELD] specified in the request or predicate are not supported for the resource type Cluster." } © Hortonworks Inc. 2013 Page 18
  • 19. SPI Overview • Service framework for handling requests for the various resources that it manages and monitors. /api/v1/clusters/c1/hosts/host1/host_components Handler getResources(Resource.Type.HostComponent, request, predicate) Cluster ResourceProvider Service ResourceProvider Host ClusterController ResourceProvider Component ResourceProvider getResources(request, predicate) HostComponent ResourceProvider © Hortonworks Inc. 2013 Page 19
  • 20. SPI Use Case • Pluggable resource providers allow API usage with a GSInstaller installed cluster. API SPI GSInstaller Default ResourceProviders ResourceProviders Ambari DB GSInstaller Artifacts © Hortonworks Inc. 2013 Page 20
  • 21. Q&A © Hortonworks Inc. 2013 Page 21