Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

Apache Ambari - What's New in 2.4

2.098 Aufrufe

Veröffentlicht am

Learn about the new features of Ambari 2.4

http://ambari.apache.org/

Veröffentlicht in: Technologie
  • DOWNLOAD FULL BOOKS, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Antworten 
    Sind Sie sicher, dass Sie …  Ja  Nein
    Ihre Nachricht erscheint hier
  • DOWNLOAD FULL BOOKS, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Antworten 
    Sind Sie sicher, dass Sie …  Ja  Nein
    Ihre Nachricht erscheint hier

Apache Ambari - What's New in 2.4

  1. 1. 1 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Ambari 2.4.0 What’s New August 2016
  2. 2. 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved What is Apache Ambari? A completely open source management platform for provisioning, managing, monitoring and securing Apache Hadoop clusters. Apache Ambari takes the guesswork out of operating Hadoop.
  3. 3. 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved What Ambari Does Simplified Installation, Configuration and Management Centralized Security Setup Full Visibility into Cluster Health Highly Extensible and Customizable • Wizard-driven and automated cluster provisioning • Smart Configurations and Cluster Recommendations • Automated Rolling and Express cluster upgrades • Reduce complexity to administer security across the platform • Automate setup Kerberos • Simplify the configuration of Apache Ranger • Predefined alerts based on operational best practices • Advanced metrics visualization with Grafana • Seamlessly fit into your enterprise environment • Bring custom Services under management via Ambari Stacks • Customize the UI with Ambari Views
  4. 4. 4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved What’s New in Ambari 2.4  Alerts: Customizable SCRIPT Parameters (AMBARI-14898)  Alerts: Retry Check Counts (AMBARI-15686)  Alerts: New HDFS Alerts (AMBARI-14800)  New Host Page Filtering (AMBARI-15210)  Remove Service (AMBARI-14759)  Support for SLES 12 Technical Preview (AMBARI-16007)  Stability: Database Consistency Checking (AMBARI-16258)  Customizable Ambari Log + PID Dirs (AMBARI-15300)  New Version Registration Experience (AMBARI-15724)  Log Search Technical Preview (AMBARI-14927)  Operational Audit Logging (AMBARI-15241)  Role-Based Access Control (AMBARI-13977)  Automated Setup of Ambari Kerberos (AMBARI-15561)  Automated Setup of Ambari Proxy User (AMBARI-15561)  Customizable Host Reg. SSH Port (AMBARI-13450) Core Features Security Features  View URLs (AMBARI-15821), View Refresh (AMBARI-15682)  Inherit Cluster Permissions (AMBARI-16177)  Remote Cluster Registration (AMBARI-16274) Views Framework Features
  5. 5. 5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Agenda What’s New in Ambari 2.4.0 Feature Highlights: Alert Retry Check Counts
  6. 6. 6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Alert Check Counts  Customize the number of times an alert is checked before dispatching a notification  Avoid dispatching an alert notification (email, snmp) in case of transient issues
  7. 7. 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Configuring the Check Count  Set globally for all alerts, or override for a specific alert Global Setting Alert Override
  8. 8. 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved State Change Types  SOFT state changes do not perform a dispatch  HARD state changes (to non-OK) perform dispatch  Regardless of change: – The Ambari Web UI will show the current state (OK/WARN/CRIT) – The state change is written to ambari-alerts.log 2016-05-31 13:20:52,294 [CRITICAL] [SOFT] [AMBARI_METRICS] [grafana_webui] (Grafana Web UI) Connection failed to http://c6401.ambari.apache.org:3000 (<urlopen error [Errno 111] Connection refused>) 2016-05-31 13:22:52,290 [CRITICAL] [HARD] [AMBARI_METRICS] [grafana_webui] (Grafana Web UI) Connection failed to http://c6401.ambari.apache.org:3000 (<urlopen error [Errno 111] Connection refused>) Note: check counts are not configurable for AGGREGATE alert types. All state changes are considered HARD.
  9. 9. 9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Example: Check Count = 3 Check 1/3 State: OK Change: n/a Check 1/3 State: OK Change: n/a Check 1/3 State: CRIT Change: SOFT Check 2/3 State: CRIT Change: n/a Check 3/3 State: CRIT Change: HARD Check 1/3 State: OK Change: HARD DISPATCH Check Interval Check Interval Check Interval Check Interval Check Interval no state change state changes to CRIT performing multiple checks back to OK
  10. 10. 10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Agenda What’s New in Ambari 2.4.0 Feature Highlights: Alert Customizable Params
  11. 11. 11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Alert Types and Thresholds  Ability to customize Thresholds for SCRIPT and SERVER alerts  Ability to customize Connection Timeout for METRIC alerts Alert Type Description Thresholds (units) WEB Connects to a Web URL. Alert status is based on the HTTP response code. Response Code (n/a) Connection Timeout (seconds) PORT Connects to a port. Alert status is based on response time. Response (seconds) METRIC Checks the value of a service metric. Units vary, based on the metric being checked. Metric Value (units vary) Connection Timeout (seconds) AGGREGATE Aggregates the status for another alert. % Affected (percentage) SCRIPT Executes a script to handle the alert check. Varies SERVER Executes a server-side runnable class to handle the alert check. Varies
  12. 12. 12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Alerts: Customizable METRIC Connection Timeout  Ability to set Connection Timeout threshold via Ambari Web UI NEW!
  13. 13. 13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Alerts: Customizable SCRIPT Thresholds  Ability to set various thresholds via Ambari Web UI
  14. 14. 14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Alerts: NEW!!! Ambari Server Performance Alert  Measures the Ambari Server REST API and Backend Database response
  15. 15. 15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Agenda What’s New in Ambari 2.4.0 Feature Highlights: New HDFS Alerts
  16. 16. 16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved New HDFS Alerts Watch Trends  NameNode Client RPC Queue Latency (Hourly/Daily)  NameNode Client RPC Processing Latency (Hourly/Daily)  NameNode Service RPC Queue Latency (Hourly/Daily)  NameNode Service RPC Processing Latency (Hourly/Daily)  NameNode Heap Usage (Daily/Weekly)  HDFS Storage Capacity Usage (Daily/Weekly)
  17. 17. 17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Agenda What’s New in Ambari 2.4.0 Feature Highlights: New Host Filtering
  18. 18. 18 © Hortonworks Inc. 2011 – 2016. All Rights Reserved New Host Filtering Control in Ambari Web  Ability to perform complex host filtering from Ambari Web  Make it easier to find hosts NEW!
  19. 19. 19 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Search by Host Attribute, Service or Component
  20. 20. 20 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Host Attribute Filtering  Host Name  IP  Host Status  Cores  RAM  Stack Version + Version State  Rack
  21. 21. 21 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Service Filtering
  22. 22. 22 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Component Filtering
  23. 23. 23 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Host Filter: Examples
  24. 24. 24 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Agenda What’s New in Ambari 2.4.0 Feature Highlights: Remove Service
  25. 25. 25 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Remove Service  Ability to perform Remove Service from Ambari Web  Eliminates need to use Ambari REST API  Checks for Service dependencies  Service must be stopped  All configuration information and history is also removed  This operation is not reversible NEW!
  26. 26. 26 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Agenda What’s New in Ambari 2.4.0 Feature Highlights: Other Items
  27. 27. 27 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Customizable Ambari Log + PID Dirs (AMBARI-15300)  Ambari Server and Agents write log activity output to log files and use a PID-file that contains the process identification number (PID) for their running process. Log Location PID Location Ambari Server /var/log/ambari-server/ambari-server.log /var/run/ambari-server/ambari-server.pid Ambari Agent /var/log/ambari-agent/ambari-agent.log /var/run/ambari-agent/ambari-agent.pid
  28. 28. 28 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Customize Ambari Server Log + PID vi /etc/ambari-server/conf/ambari.properties pid.dir=/var/run/ambari-server vi /etc/ambari-server/conf/log4j.properties ambari.log.dir=${ambari.root.dir}/var/log/ambari- server Ambari Server PID Ambari Server Log 1. Stop Ambari Server prior to modifying log or pid directories. 2. You must manually create the new directories and be sure to set the directory ownership + permissions to allow the Ambari Server process access.
  29. 29. 29 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Customize Ambari Agent Log + PID vi /etc/ambari-agent/conf/ambari-agent.ini [agent] logdir=/var/log/ambari-agent piddir=/var/run/ambari-agent 1. Stop Ambari Agent prior to modifying log or pid directories. 2. You must manually create the new directories and be sure to set the directory ownership + permissions to allow the Ambari Agent process access.
  30. 30. 30 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Customizable Host Registration SSH Port  Customize SSH Port when performing Host Registration automatically NEW!
  31. 31. 31 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Stability: Database Consistency Checking  On Ambari Server start, Ambari runs a database consistency check looking for issues.  If any issues are found, Ambari Server start will abort and a message will be printed to console “DB configs consistency check failed.”  Check Ambari Server log file for more details: /var/log/ambari-server/ambari-server-check-database.log  Ability to “skip” check and force Ambari Server start ambari-server start --skip-database-check Important: if you “skip” the check to force Ambari Server start, do not make any changes to your cluster topology or perform a cluster upgrade until you correct the database consistency issues.
  32. 32. 32 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Agenda What’s New in Ambari 2.4.0 Feature Highlights: View Framework Enhancements
  33. 33. 33 © Hortonworks Inc. 2011 – 2016. All Rights Reserved View URLs (AMBARI-15821)  Ability to create a “short URL” or “vanity URL” for view instances  Provide users with a non-version or instance specific URL to a view /#/main/views/{viewName}/{viewVersion}/{viewInstanceName}/#/main/view/{viewName}/{shortURL}
  34. 34. 34 © Hortonworks Inc. 2011 – 2016. All Rights Reserved View Refresh (AMBARI-15682)  Automatically deploy new views into Ambari Server w/o a restart 1. Copy view archive to: /var/lib/ambari-server/resources/views/ 2. Ambari Server detects the new view, automatically extracts + deploys 3. View is available for creating instances 4. Click “Refresh” in Views UI
  35. 35. 35 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Remote Cluster Configuration AMBARI-16274
  36. 36. 36 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Background: View <-> Cluster Communication  Deployed Views “talk” with cluster using REST APIs (as applicable) CLUSTER ATS RM Ambari Server Tez UI View Tez UI View talks with cluster using REST APIs to ATS and ResourceManager Ambari DB LDAP AuthN
  37. 37. 37 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Background: Operational vs. Standalone Ambari Server Ambari Agent Host Ambari Agent Host Ambari Agent Host Standalone Ambari Server One or More Ambari Server Instances No Agents, no requirement to operate the cluster Operational Ambari One Ambari Server Instance Talking with Agents, Managing the cluster Ambari Server Ambari DB LDAP AuthN Ambari Server Ambari DB LDAP AuthN
  38. 38. 38 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Background: Local Cluster vs. Non-Local Ambari Server Ambari DB LDAP AuthN Ambari Server Ambari DB LDAP AuthN Standalone Ambari Server One or More Ambari Server Instances No Agents, no requirement to operate the cluster Operational Ambari One Ambari Server Instance Talking with Agents, Managing the cluster LOCAL CLUSTER NON- LOCAL CLUSTER
  39. 39. 39 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Introducing Remote Cluster Configuration (AMBARI-16274) Option Description Local Cluster When you select this Local Cluster option, Ambari will automatically determine the cluster configuration properties needed for the view instance. Criteria: • Ambari Server running the views is also managing the cluster Remote Cluster When you select Remote Cluster option, Ambari will automatically determine the cluster configuration properties needed for the view instance. Criteria: • The cluster is not local to the Ambari Server running the views (i.e. Standalone) • Cluster is being managed by Ambari Custom When you select Custom option, you must enter all configuration information, and are responsible for updating if the cluster configuration changes. Criteria: • The cluster running the view is not local to the Ambari Server • The cluster is not being managed by Ambari NEW!
  40. 40. 40 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Local vs Remote View Configuration Ambari Server Views Cluster Ambari Server Views ClusterAmbari Server LOCAL CLUSTER REMOTE CLUSTER Operational Ambari Manages cluster Standalone Ambari Manages cluster Talks to cluster Obtains view config Obtains view config Talks to cluster Operational Ambari
  41. 41. 41 © Hortonworks Inc. 2011 – 2016. All Rights Reserved View Configuration: Minimizing Need for Custom Cluster Config Ambari Server Cluster Mgmt Ambari 2.2 or Earlier Ambari 2.4 No HA, No Kerberos Operational Ambari Local Local HA or Kerberos Operational Ambari Custom Local No HA, No Kerberos Standalone Ambari Custom Remote HA or Kerberos Standalone Ambari Custom Remote No HA, No Kerberos Standalone Non-Ambari Custom Custom HA or Kerberos Standalone Non-Ambari Custom Custom
  42. 42. 42 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Inherit Cluster Permissions AMBARI-16177
  43. 43. 43 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Inherit Cluster Permissions (AMBARI-16177)  Ability to automatically grant View “Use” permission based on Cluster role  Note: Option is only available when using a Local Cluster Configuration Explicitly grant users and groups Use permission Automatically grant users and groups Use permission based on Cluster roles
  44. 44. 44 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Agenda What’s New in Ambari 2.4.0 Feature Highlights: Log Search TECH PREVIEW
  45. 45. 45 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Log Search Solr A M B A R I Log Search Search Cluster Component Logs from within Ambari Goal: When issues arise, be able to quickly find issues across all components ⬢ Capabilities – Rapid Search of all cluster component logs – Search across time ranges, log levels, and for keywords ⬢ Core Technologies: – Apache Ambari – Apache Solr – Apache Ambari Log Search
  46. 46. 46 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Log Search Architecture A M B A R I L O G F E E D E R L O G F E E D E R L O G F E E D E R L O G F E E D E R L O G F E E D E R L O G F E E D E R WO R K E R N O D E WO R K E R N O D E WO R K E R N O D E WO R K E R N O D E WO R K E R N O D E WO R K E R N O D E Solr LO G S E A R C H U I
  47. 47. 47 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Log Search Details WO R K E R N O D E L O G F E E D E R Solr LO G S E A R C H U I Solr Solr A M B A R I Java Process Multi-output Support Grok Solr Cloud Local Disk Storage TTL
  48. 48. 48 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Considerations  Log Feeders are CPU intensive, consider 1 dedicated core  Solr instances should use dedicated hardware with at least 32GB of RAM dedicated to the Solr instance  By default, logs will age out after 7 days
  49. 49. 49 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Agenda What’s New in Ambari 2.4.0 Feature Highlights: RBAC
  50. 50. 50 © Hortonworks Inc. 2011 – 2016. All Rights Reserved New Role Based Access Control  Introducing new “roles” for more granular division of control for cluster operations Old Permission New Role Notable Permissions Operator Cluster Administrator Full operational control, including upgrades. Ambari Admins are implicitly granted this Role. Cluster Operator Adding and removing hosts. Service Administrator Manage configurations, move components. Service Operator Service stop and start and service-specific operations such as HDFS Rebalance. Read-Only Cluster User View cluster service and host information. Note: Users flagged as “Ambari Administrators / Ambari Admins” are implicitly granted Cluster Administrator permission.
  51. 51. 51 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Managing Cluster Roles Assign roles to users or groups Manage roles in Block or List View layouts
  52. 52. 52 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Managing Cluster Roles View users or groups Change current role assignment
  53. 53. 53 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Agenda What’s New in Ambari 2.4.0 Feature Highlights: Security Enhancements
  54. 54. 54 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Summary of Security Enhancements  Automatic Setup of Ambari Server as a Proxyuser  Automatic Setup of Ambari Server for Kerberos
  55. 55. 55 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Automatic Setup of Ambari Server as a Proxyuser
  56. 56. 56 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Background: Proxyusers  HDFS and WebHCat (as part of Hive) support the concept of a Proxyuser  Proxyuser allows UserA to access the service on behalf of UserB (i.e. the proxyuser is allowed to impersonate other users)  Proxyuser is a commonly used capability of Hadoop HDFS “UserA” is setup as a proxyuser UserA can access HDFS as “UserA” on behalf of “UserB” HDFS ops performed are as “UserB”
  57. 57. 57 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Background: HDFS Proxyuser Setup  A proxyuser needs to be configured in core-site.xml configuration: hadoop.proxyuser.{proxyuser-name}.hosts hadoop.proxyuser.{proxyuser-name}.groups  If these settings are not present, impersonation will not be allowed and connection to the service via proxyuser will fail
  58. 58. 58 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Background: Ambari + Proxyuser  Ambari Views use proxyuser to access the cluster (such as Hive View and Pig View)  Ambari Server needs to access a service on behalf of an authenticated user Ambari Server HDFS (running as user “ambari”) “joe” authenticates to Ambari (setup for proxyuser “ambari”) hadoop.proxyuser.ambari.hosts=* hadoop.proxyuser.ambari.groups=* Ambari Server can talk to HDFS as “ambari” proxyuser on behalf of “joe” Configuration of proxyuser is commonly “missed” when setting up Ambari Views
  59. 59. 59 © Hortonworks Inc. 2011 – 2016. All Rights Reserved New: Automatic Ambari Server Proxyuser Setup  Proxyuser configurations are automatically added for HDFS and WebHCat  For example: Ambari Server as running as “ambari”, the following configurations are added during HDFS service install hadoop.proxyuser.ambari.hosts hadoop.proxyuser.ambari.groups
  60. 60. 60 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Automatic Setup of Ambari Server for Kerberos
  61. 61. 61 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Background: Hadoop + Kerberos  Strongly authenticating and establishing a user’s identity is the basis for secure access in Hadoop. Users need to be able to reliably “identify” themselves and then have that identity propagated throughout the Hadoop cluster.  Once this is done, those users can access resources (such as files or directories) or interact with the cluster (like running MapReduce jobs).  Besides users, Hadoop cluster resources themselves (such as Hosts and Services) need to authenticate with each other to avoid potential malicious systems or daemon’s “posing as” trusted components of the cluster to gain access to data.
  62. 62. 62 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Background: Hadoop + Kerberos Service Component A Service Component B Hadoop Cluster KDC keytabkeytab Service Component C keytab Service Component D keytab Service Component X Service Component X keytabkeytab Service Component X keytab Service Component X keytab Kerberos is used to secure the Components in the cluster. Kerberos identities are managed via “keytabs” on the Component hosts. Principals for the cluster are managed in the KDC.
  63. 63. 63 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Background: Automated Kerberos Setup with Ambari
  64. 64. 64 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Background: Principal and Keytab Generation & Distribution 1. User provides KDC Admin Account credentials to Ambari 2. Ambari connects to KDC, creates principals (Service and Ambari) needed for cluster 3. Ambari generates keytabs for the principals 4. Ambari distributes keytabs to Ambari Server and cluster hosts 5. Ambari discards the KDC Admin Account credentials (optional) Ambari Server KDC 1 2 4 3 5 Cluster
  65. 65. 65 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Background: Ambari + Hadoop + Kerberos  Ambari Server communicates with the cluster to retrieve information (such as metrics)  Especially important for Ambari Views (e.g. Files, Hive, Pig)  Therefore: Ambari Server ALSO needs to be “setup for Kerberos” Ambari Server Cluster Kerberos enabled
  66. 66. 66 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Background: Manual Setup of Ambari Server for Kerberos  Manual setup of Ambari Server for Kerberos (outside of “Enable Kerberos” wizard): 1. Create principal for Ambari Server 2. Generate keytab for Ambari Server 3. Place keytab on Ambari Server host 4. Run “ambari-server setup-security” on Ambari Server 5. Restart Ambari Server Configuration of Ambari Server for Kerberos is commonly “missed” when setting up Ambari Views
  67. 67. 67 © Hortonworks Inc. 2011 – 2016. All Rights Reserved New: Automatic Setup of Ambari Server for Kerberos  When enabling Kerberos and choosing an automated option (MIT or AD), Ambari Server will be setup for Kerberos automatically: 1. A principal will be created for Ambari Server 2. A keytab will be generated and placed on Ambari Server 3. Ambari Server is setup for Kerberos Note: you will still need to perform the Ambari Server restart for the Kerberos identity to get picked-up by Ambari.
  68. 68. 68 © Hortonworks Inc. 2011 – 2016. All Rights Reserved What about Proxyuser + Kerberos?
  69. 69. 69 © Hortonworks Inc. 2011 – 2016. All Rights Reserved New: Automatic Proxyuser Setup with Kerberos  When a cluster has Kerberos enabled, the proxyuser needs to be configured based on the primary part of the Kerberos principal name hadoop.proxyuser.{principal-name-primary}.hosts hadoop.proxyuser.{principal-name-primary}.groups  Ambari will adjust proxyuser configurations during Kerberos setup Ambari Server HDFS (running as user “ambari”) (setup with principal “ambari-server@EXAMPLE.COM” “joe” authenticates to Ambari (setup for proxyuser “ambari-server”) hadoop.proxyuser.ambari-server.hosts=* hadoop.proxyuser.ambari-server.groups=* Ambari Server can talk to HDFS as “ambari-server” proxyuser on behalf of “joe”
  70. 70. 70 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Agenda What’s New in Ambari 2.4.0 Feature Highlights: Ops Audit Logging
  71. 71. 71 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Operational Audit Logging  Ambari will create entries in an audit log as Ambari + Cluster operations are performed  Using the audit log, you can determine who performed the operation and when the operation was performed as well as other operation-specific information  The Ambari Audit log can be found at: /var/log/ambari-server/ambari-audit.log
  72. 72. 72 © Hortonworks Inc. 2011 – 2016. All Rights Reserved List of Operations  Stop/Start Service  Stop all Services  Add Service  Move Component  Turn On/Off Maintenance Mode  Download Client Configurations  Blueprint Export  Update Configuration **  Login (success/failed) / Logout  Create User, Group  Delete User, Group  Change Group Membership  Change User Status, Admin  Change User Password  Grant/Revoke User, Group Cluster Roles Service Operations User Operations ** Note: When a Service Configuration change is made, an entry is also written to a specific log file ambari-config-changes.log for configuration changes that provides even more detail on the change.
  73. 73. 73 © Hortonworks Inc. 2011 – 2016. All Rights Reserved List of Operations (continued)  Add/Remove Host  Enable/Disable/Edit Alert  Add/Update/Delete Alert Group  Add/Upgrade/Delete Notification  Enable/Disable Kerberos  Regenerate Kerberos Keytabs  Rename Cluster  Add/Remove Remote Clusters  Register/Deregister Version  Cluster Upgrade Cluster Operations Upgrade Operations  Create/Delete View Instance  Edit View Instance  Grant/Revoke View Permissions  Create/Delete View URLs View Operations
  74. 74. 74 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Example: Change Group Membership  Add/Remove group members creates a “Membership change” audit entry 2016-06-02T23:12:09.930Z, User(admin), RemoteIp(192.168.64.1), Operation(Membership change), RequestType(PUT), url(http://c6401.ambari.apache.org:8080/api/v1/groups/customgroup/members), ResultStatus(200 OK), Group(customgroup), Members(joeuser) 2016-06-02T23:12:34.700Z, User(admin), RemoteIp(192.168.64.1), Operation(Membership change), RequestType(PUT), url(http://c6401.ambari.apache.org:8080/api/v1/groups/customgroup/members), ResultStatus(200 OK), Group(customgroup), Members(joeuser, mike)
  75. 75. 75 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Example: Stop ZooKeeper  A single operation (like “Stop ZooKeeper”) might generate multiple audit entries  Relate entries via RequestId() 2016-06-02T23:14:35.206Z, User(admin), RemoteIp(192.168.64.1), Operation(INSTALLED: ZOOKEEPER_SERVER/ZOOKEEPER on c6401.ambari.apache.org (MyCluster)), Host name(c6401.ambari.apache.org), RequestId(7), Status(Successfully queued) 2016-06-02T00:31:56.016Z, User(admin), Operation(Stop ZooKeeper Server), Status(IN_PROGRESS), RequestId(7) 2016-06-02T00:31:56.025Z, User(admin), Operation(STOP ZOOKEEPER_SERVER), Status(QUEUED), RequestId(7), TaskId(52), Hostname(c6401.ambari.apache.org) 2016-06-02T00:31:57.370Z, User(admin), Operation(Stop ZooKeeper Server), Status(COMPLETED), RequestId(7) 2016-06-02T00:31:57.370Z, User(admin), Operation(STOP ZOOKEEPER_SERVER), Status(COMPLETED), RequestId(7), TaskId(52), Hostname(c6401.ambari.apache.org)
  76. 76. 76 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Agenda What’s New in Ambari 2.4.0 Feature Highlights: Version Registration Experience
  77. 77. 77 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Introducing the Version Definition File (VDF)  This is a meta file describing which Services are included and at which version
  78. 78. 78 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Ambari will “discover” Available Versions Tabs for list of available Stacks List of discovered Versions List of Services w/version #
  79. 79. 79 © Hortonworks Inc. 2011 – 2016. All Rights Reserved “Default Version Definition” for Backwards Compat Ambari provides a “default” Version Definition.
  80. 80. 80 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Add New Version via File Upload or URL
  81. 81. 81 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Changes in Install / Version Registration Flow Scenario Ambari 2.4 Change Internet Access / Public Repositories No change. No Internet Access / Local repositories - Upload a VDF for the Local Repository you created - Set the Local Repository URLs OR - Choose the Default Version Definition - Set the Local Repository URLs
  82. 82. 82 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Other UX Changes: Local vs. Public Repository Radio Explicit Choice
  83. 83. 83 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Other UX Changes: Local vs. Public Repository Radio Choose Local Must enter Base URLs
  84. 84. 84 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Other UX Changes: OS Add/Remove
  85. 85. 85 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Other UX Changes: RedHat Satellite/Spacewalk Explicit Choice - Ambari will not write the .repo files - User must register the repositories channels via Satellite
  86. 86. 86 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Other UX Changes: Viewing, Install and Upgrade
  87. 87. 87 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Other UX Changes: Managing Versions
  88. 88. 88 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Thank You

×