Salt Air 19 - Intro to SaltStack RAET (reliable asyncronous event transport)

•

2 gefällt mir•3,970 views

Thomas Hatch, SaltStack CTO, and Sam Smith, SaltStack director of product development, introduce SaltStack RAET as a new alternative transport medium developed specifically with SaltStack infrastructure automation and configuration management in mind. SaltStack built RAET for customers needing substantial speed and scale to automate management of massive data center infrastructure environments. SaltStack RAET is primarily an async communication layer over truly async connections, defaulting to UDP. The SaltStack RAET system uses CurveCP encryption by default. SaltStack users can now leverage substantial flexibility via either Salt SSH, ØMQ, or RAET to best address numerous use cases.

Software Technologie

2
Salt Air 19!
!
SaltStack RAET!
!
Thomas Hatch, SaltStack CTO!
&!
Sam Smith, SaltStack director product development

The need for speed
• From server closet to Web scale, speed and flexibility
matters;!
• SaltStack built for speed and scale at any level;!
• SaltStack supports:!
• SSH!
– Good for limited use cases!
– 50x faster than other other SSH-based tools!
• ØMQ!
– Super fast!
– Good for many use cases, but…!
• RAET (Reliable asynchronous event transport)!
– Even faster!
– From tens of thousands to hundreds of thousands!
– Driven by customer demand (massive infrastructures)!
– More control and infrastructure visibility

ØMQ background and overview
• ØMQ!
• The fastest of the integrated queuing services!
• Integrates PubSub Queueing into custom TCP Sockets!
• Difficult to observe and manage problems under load!
• TCP socket delays, reconnect problems!
• Suited for multi-process architectures, context
switching overhead, blocking I/O!
• TCP connection limits, presence, latency

SaltStack RAET protocol
• Bringing established automation technology to Salt;!
• Built on a precedent of successful automation
concepts;!
• Support for UDP sockets in addition to TCP;!
• Transactions for reliability;!
• More async;!
• Built in crypto (CurveCP);!
• Built to run on Python2, Python3, PyPy.
5

SaltStack RAET layering
• Micro-threaded architecture with non-blocking I/O;!
• Separates event Pub / Sub queue from socket-based transport
layer;!
• Optimize queueing and transport independently;!
• More modularity;!
• Better observability of issues and management of performance
under load;!
• More fine-grained management of events;!
• More maintainable code.

SaltStack RAET architecture
• Modular and nested architecture!
• Inter-host communication is transactioned UDP!
• Inter-process communication is paged UXD!
• Unified event-message routing!
• Fully signed and encrypted UDP comms

SaltStack RAET
• Built for massive scale to support SaltStack
Enterprise customer environments;!
• Go faster;!
• Driven by customer and user requirements;!
• More flexibility (SSH, ØMQ, now RAET);!
• Salt on RAET appears the same to the user as Salt
on ØMQ.
8

Thank You
9
www.saltstack.com
@saltstackinc
freenode IRC channel: #salt
https://github.com/saltstack/salt
For more information about SaltStack RAET, please visit:

https://github.com/saltstack/raet

Weitere ähnliche Inhalte

Was ist angesagt?

Inside the Chef Push Jobs Service - ChefConf 2015

Chef

Openstack portal-bestpractices-campbell mcneill

Campbell McNeill

Matt Bruzek - Monitor Public Cloud Use Nagios to monitor your public cloud. - No debian installer for Nagios 4? No problem! Deploy your public cloud with Juju and you can connect Nagios core services to your Ubuntu instances in the cloud. In this session, Matt will quickly go over the basic concepts of Juju and spend the rest of the time walking through examples of deploying Nagios monitoring solutions

Matt Bruzek - Monitoring Your Public Cloud With Nagios

Nagios

How DreamHost builds a Public Cloud with OpenStack

Carl Perry

OpenNebula is a great cloud orchestration and management tool, characterised by its flexibility, durability and focused feature matrix. A feature matrix that is driven largely by real world problem scenarios and real world feedback : something that has been the focus of CentOS project as well. From large scale deployment automation to patch management and state control, the CentOS Project aims to solve real world problems faced by the people who run infrastructure : The Sysadmins administrators and operations teams. During this talk, I will share why OpenNebula and CentOS Linux are a perfect match and go into some user stories that demonstrate this relationships success in real world scenarios.

CentOS and OpenNebula, a Perfect Match

NETWAYS

Service Discovery 101

Stefan Achtsnit

Continuous Delivery and Infrastructure as Code

Sascha Möllering

Configuration Management Evolution at CERN

Gavin McCance

You’re ready to make your applications more responsive, scalable, fast and secure. Then it’s time to get started with NGINX. In this webinar, you will learn how to install NGINX from a package or from source onto a Linux host. We’ll then look at some common operating system tunings you could make to ensure your NGINX install is ready for prime time. View full webinar on demand at http://nginx.com/resources/webinars/installing-tuning-nginx/

NGINX Installation and Tuning

NGINX, Inc.

Muli Ben-Yehuda, Stratoscale - The Road to a Hyper-Converged OpenStack, OpenS...

Cloud Native Day Tel Aviv

Encrypt your volumes with barbican open stack 2018

Duncan Wannamaker

DoxLon | Life with kube, containers and microservices

Richard Marshall

Developing the Stratoscale System at Scale - Muli Ben-Yehuda, Stratoscale - D...

DevOpsDays Tel Aviv

"To provide exceptional customer experiences at scale, the data pipelines that can move data reliably across the systems and applications in real-time should be seamlessly scalable. For the past several years, we relied on Message Queue based data pipelines to facilitate the transfer of data across the applications. However, as the number of use cases that require real-time data transfer increased rapidly, it became difficult to scale the messaging platform. Moving to Kafka helped us to resolve the data pipeline scaling issues and reduce the Publisher/Subscriber on-boarding time from several weeks to a few days. To support the on-demand scaling of Kafka clusters, we run them on RedHat OpenShift, an Enterprise Kubernetes. While managing Kafka that handles critical financial events, we have learned some lessons and developed efficient strategies to manage production-grade Kafka clusters on OpenShift. In this talk, we will present: 1. Some of the challenges that we faced with Kafka on OpenShift and how we evolved our infrastructure to overcome them. 2. Share our experiences from operating Kafka clusters at Scale in Production. 3. Our strategy for performing automated Kafka deployment and rollback in OpenShift. 4. Explain our fail-over strategy using Confluent’s Replicator to ensure service availability during cluster failures."

Discover Kafka on OpenShift: Processing Real-Time Financial Events at Scale (...

confluent

A database replication topology is never written in stone - it evolves along with your application and data. Changes are usually needed to help scale out, to distribute your database across multiple regions or data centers, or to perform software/hardware maintenance operations. The initial setup of a replication topology is simple, but as soon as you start changing it, things can quickly get complex. How do we fail-over our replication masters and slaves without affecting the availability and consistency of our data? In this webinar, we discuss how to perform replication topology changes in MySQL / MariaDB, and what the failover process may look like. We also discuss some external tools you may find useful when dealing with those operations. AGENDA MySQL Replication topology changes using GTID using regular replication Failover process Using MaxScale for automatic re-routing of queries Other external tools useful when dealing with failover

Webinar slides: Replication Topology Changes for MySQL and MariaDB

Severalnines

Mutating Admission Webhook creation

Victor Morales

Automating Monitoring with Puppet

Christian Mague

Microservices chat

YongHyuk Lee

Nginx in production

Graham Weldon

Watch on-demand: www.nginx.com/resources/webinars/amplify-launched-as-public-beta Gain greater visibility and control for your applications and infrastructure. NGINX Amplify is a new diagnostic tool designed specifically to monitor and troubleshoot NGINX and the applications it delivers. This webinar will introduce you to this new product and show you how to use it to make immediate improvements to your applications. See NGINX Amplify’s powerful features, including a customizable analytics dashboard, diagnostics of your app performance and NGINX configurations, and a configurable alerting system. With NGINX Amplify you get deep diagnostics and actionable insights to improve the performance and security of your apps. Join this webinar to learn how you can use NGINX Amplify to: * Begin understanding and improving application performance in as little as five minutes * Learn now to analyze your NGINX configuration and improve performance and security * See how NGINX Amplify lets you visualize key metrics from multiple NGINX instances * Configure smart alerts that notify you when parts of your system need attention * Augment your existing APM tools with dedicated and much more comprehensive NGINX analytics

Improve App Performance & Reliability with NGINX Amplify

NGINX, Inc.

Was ist angesagt? (20)

Inside the Chef Push Jobs Service - ChefConf 2015

Openstack portal-bestpractices-campbell mcneill

Matt Bruzek - Monitoring Your Public Cloud With Nagios

How DreamHost builds a Public Cloud with OpenStack

CentOS and OpenNebula, a Perfect Match

Service Discovery 101

Continuous Delivery and Infrastructure as Code

Configuration Management Evolution at CERN

NGINX Installation and Tuning

Muli Ben-Yehuda, Stratoscale - The Road to a Hyper-Converged OpenStack, OpenS...

Encrypt your volumes with barbican open stack 2018

DoxLon | Life with kube, containers and microservices

Developing the Stratoscale System at Scale - Muli Ben-Yehuda, Stratoscale - D...

Discover Kafka on OpenShift: Processing Real-Time Financial Events at Scale (...

Webinar slides: Replication Topology Changes for MySQL and MariaDB

Mutating Admission Webhook creation

Automating Monitoring with Puppet

Microservices chat

Nginx in production

Improve App Performance & Reliability with NGINX Amplify

Ähnlich wie Salt Air 19 - Intro to SaltStack RAET (reliable asyncronous event transport)

.NET Conf 2022 - Networking in .NET 7

Karel Zikmund

HiveMQ Webinar: Lightweight and scalable IoT Messaging with MQTT

Dominik Obermaier

From Device to Data Center to Insights: Architectural Considerations for the ...

P. Taylor Goetz

ClickHouse Paris Meetup. Pragma Analytics Software Suite w/ClickHouse, by Mat...

Altinity Ltd

MQTT is the de-facto protocol for the Internet of Things (IoT). This webinar covers everything you need to know about scalable pub/sub communication with MQTT for up to millions of devices and shows the available software options in the (open source) ecosystem. About the Speaker. Dominik Obermaier is CTO and co-founder of HiveMQ. He is a member of the OASIS Technical Committee and is part of the standardization committee for MQTT 3.1.1 and MQTT 5. He is the co-author of the book 'The Technical Foundations of IoT' and a frequent speaker on IoT, MQTT, and messaging. To watch the webinar recording: https://www.hivemq.com/webinars/lightweight-and-scalable-iot-messaging-with-mqtt/

Lightweight and Scalable IoT Messaging with MQTT

HiveMQ

From Device to Data Center to Insights

DataWorks Summit/Hadoop Summit

Azure Event Hubs - Behind the Scenes With Kasun Indrasiri | Current 2022 Azure Event Hubs is a hyperscale PaaS event stream broker with protocol support for HTTP, AMQP, and Apache Kafka RPC that accepts and forwards several trillion (!) events per day and is available in all global Azure regions. This session is a look behind the curtain where we dive deep into the architecture of Event Hubs and look at the Event Hubs cluster model, resource isolation, and storage strategies and also review some performance figures.

Azure Event Hubs - Behind the Scenes With Kasun Indrasiri | Current 2022

HostedbyConfluent

(Kai Waehner, Confluent) Kafka Summit SF 2018 This session discusses end-to-end use cases such as connected cars, smart home or healthcare sensors, where you integrate Internet of Things (IoT) devices with enterprise IT using open source technologies and standards. MQTT is a lightweight messaging protocol for IoT. However, MQTT is not built for high scalability, longer storage or easy integration to legacy systems. Apache Kafka is a highly scalable distributed streaming platform, which ingests, stores, processes and forwards high volumes of data from thousands of IoT devices. This session discusses the Apache Kafka open source ecosystem as a streaming platform to process IoT data. See a live demo of how MQTT brokers like Mosquitto or RabbitMQ integrate with Kafka, and how you can even integrate MQTT clients to Kafka without MQTT Broker. Learn how to analyze the IoT data either natively on Kafka with Kafka Streams/KSQL or on an external big data cluster like Spark, Flink or Elasticsearch leveraging Kafka Connect.

Processing IoT Data from End to End with MQTT and Apache Kafka

confluent

This month we focus in on WebSockets in HTML5 with Brad Drysdale. You're probably aware of the various chat systems around the web (facebook chat for example) but recently there has been more of a desire for other types of interaction using these technologies. Many have requested talks on Node.js, Comet long-polling, or similar technologies. WebSockets are the evolution of these technologies, and a developing standard being progressed by the IETF. The HTML 5 standard specifies new APIs for storage, drawing, drag-and-drop, and other areas that have made web programming painful. Browsers have already begun incorporating parts of HTML 5 (canvas, for example) even though the specification is far from complete. The HTML 5 Communication section includes two additional connectivity features: Server-Sent Events, a standardization of HTTP push, and Web Sockets, a cross-domain safe, full-duplex connection. Server-Sent Events will make real-time updates and notifications easy, and Web Sockets provide the functionality necessary to build chat for the web without the previously required hackery. Come and hear the latest on what this great technology allows! Brad Drysdale is the Technical Director in EMEA for Kaazing. Brad has also worked at Openwave and Netscape Communications in various pre-sales and technical evangelist roles.

Html5 web sockets - Brad Drysdale - London Web 2011-10-20

Nathan O'Hanlon

IOT and System Platform From Concepts to Code

Andy Robinson

Advanced use cases and approaches with stratos paa s

WSO2

Software defined networking: Primer

Muhammad Moinur Rahman

Messaging For the Cloud and Microservices

Rob Davies

MQTT is by far the most popular Internet of Things protocol used in the largest professional IoT deployments worldwide. The protocol is so simple and versatile, that it can be used for private home automation projects as well as ultra-secure and highly scalable enterprise installations. This talk will show how a pure Java and open source technology stack can be used for IoT devices as well as backend applications for building next-generation IoT projects.

A pure Java MQTT Stack for IoT

Dominik Obermaier

Picking a message queue

Vladislav Kirshtein

Ambitious Internet of Things applications have one thing in common: They produce massive amounts of data. But how to design the next-generation IoT backend that is able to meet the business requirements and doesn’t explode as soon as the traffic increases? This talk will cover how to use MQTT to connect millions of devices with commodity servers and process huge amounts of data. Learn all the common design patterns and see the technologies that actually scale. Explore when to use Cassandra, Kafka, Spark, Docker, and other tools and when to stick with your good ol’ SQL database or Enterprise Message Queue.

Lightweight and scalable IoT Architectures with MQTT

Dominik Obermaier

MQTT is the most popular IoT protocol for connecting devices at scale and a modern alternative for lightweight backend (microservice) communication. This session covers everything you need to know about scalable pub/sub communication with MQTT for up to millions of devices and shows the available software options in the (open source) ecosystem. We also address the brand new features of MQTT 5 as well as advanced microservice integration topics.

Lightweight and scalable IoT Messaging with MQTT

Dominik Obermaier

Micro Services Architecture

Ranjan Baisak

Peng Kang, Software Engineer, Dropbox + Richi Gupta, Engineering Manager, Dropbox As a scalable and reliable data streaming solution with a rich ecosystem, Kafka is widely adopted in Dropbox infrastructure in various scenarios. It is part of Dropbox’s analytics data pipeline, stream processing platform and more mission critical systems. Jetstream is the team that provides Kafka as a service in Dropbox infrastructure. We manage the clusters, develop tooling, and enforce policies, so that our users can enjoy a highly available and reliable service. In this talk, we will share our experiences and learnings running Kafka clusters, pipelines that enable high durability (direct writes to kafka) and availability (goscribe), the policies we enforce for high reliability, the tooling we have for maintenance and stress testing, and finally an overview of Dropbox’s next generation queueing service built on top Kafka. https://www.meetup.com/KafkaBayArea/events/266327152/

Apache Kafka® at Dropbox

confluent

Janus RTP forwarders @ FOSDEM 2020

Lorenzo Miniero

Ähnlich wie Salt Air 19 - Intro to SaltStack RAET (reliable asyncronous event transport) (20)

.NET Conf 2022 - Networking in .NET 7

HiveMQ Webinar: Lightweight and scalable IoT Messaging with MQTT

From Device to Data Center to Insights: Architectural Considerations for the ...

ClickHouse Paris Meetup. Pragma Analytics Software Suite w/ClickHouse, by Mat...

Lightweight and Scalable IoT Messaging with MQTT

From Device to Data Center to Insights

Azure Event Hubs - Behind the Scenes With Kasun Indrasiri | Current 2022

Processing IoT Data from End to End with MQTT and Apache Kafka

Html5 web sockets - Brad Drysdale - London Web 2011-10-20

IOT and System Platform From Concepts to Code

Advanced use cases and approaches with stratos paa s

Software defined networking: Primer

Messaging For the Cloud and Microservices

A pure Java MQTT Stack for IoT

Picking a message queue

Lightweight and scalable IoT Architectures with MQTT

Lightweight and scalable IoT Messaging with MQTT

Micro Services Architecture

Apache Kafka® at Dropbox

Janus RTP forwarders @ FOSDEM 2020

Mehr von SaltStack

A SaltConf16 use case talk by Steven Braverman of Dun & Bradstreet. Testing configuration changes for multiple server roles can be time consuming when real instances or legacy container systems are used. Applying configuration changes to each role in parallel can be difficult. So what's the best way to test configuration changes efficiently, quickly, and securely prior to applying them? See how an integrated test setup using AWS EC2 Container Service (ECS), AWS AutoScaling Group, and SaltStack simplifies the application of configuration changes and allows you to test configuration changes in parallel to reduce the time spent testing.

Integration testing for salt states using aws ec2 container service

SaltStack

Spot Trading - A case study in continuous delivery for mission critical finan...

SaltStack

Arnold Bechtoldt, Inovex GmbH Linux systems engineer - Configuration Manageme...

SaltStack

SaltConf14 - Matthew Williams, Flowroute - Salt Virt for Linux contatiners an...

SaltStack

This presentation was originally given by Joseph Hall, SaltStack senior engineer, at the combined Montreal Python and DevOps Montreal meet up on April 14, 2014. Here is the talk abstract: In ye olde days of web, a company might manage a handful of servers, each manually and frequently tuned and re-tuned to the company's needs. Those days are gone. Server farms now dominate, and it is no longer reasonable to manage individual servers by hand. Various configuration management tools have stepped in to help the modern engineer, but which to choose? It is not an easy question, and canned pitches from sales people are unlikely to take into account all of your variables. This talk will attempt to discuss The Big Four objectively, and from what angles they approach the task at hand.

Configuration Management - Finding the tool to fit your needs

SaltStack

SaltConf14 - Ben Cane - Using SaltStack in High Availability Environments

SaltStack

SaltConf14 - Oz Akan, Rackspace - Deploying OpenStack Marconi with SaltStack

SaltStack

OpenStack is an open source implementation of cloud computing, potentially at very large scale. However, it has many moving parts and is complex to operate. SaltStack appears to provide scalable and secure orchestration for OpenStack. But like all powerful solutions to complex problems, a great deal of the useful know-how has to be discovered by actual practice and hard-won experience. This session will share the inside knowledge gained through practical experience.

SaltConf14 - Yazz Atlas, HP Cloud - Installing OpenStack using SaltStack

SaltStack

This talk will highlight how the OpenStack Infrastructure team uses SaltStack for event-driven orchestration of its various cloud infrastructure components. The speakers will review the flexibility of Salt in a complex automation environment. Salt plays very well with other tools, including Puppet, which is especially critical in the OpenStack Infrastructure environment which requires the event-driven orchestration functions of Salt to synchronize workflow timing of OpenStack Infrastructure components and events. To learn when and where the next SaltConf will be, subscribe to our newsletter here: http://www.saltstack.com/salt-ink-newsletter or follow us on Twitter: http://www.twitter.com/saltstackinc

SaltConf14 - Anita Kuno, HP & OpenStack - Using SaltStack for event-driven or...

SaltStack

Using SaltStack to automate enterprise IT operations and support capabilities is not as well documented as the more traditional SaltStack use cases. This session will show how the HP Cloud team runs a secure and reliable SaltStack automation environment by writing Salt states and modules to simplify day-to-day operations and support while extending SaltStack capabilities through dynamic states and modules. The talk will also show how to protect sensitive information and safe guard against user errors.

SaltConf14 - Saurabh Surana, HP Cloud - Automating operations and support wit...

SaltStack

Deployment is often omitted from configuration management and orchestration, which typically focuses on the infrastructure and not the core application that the infrastructure supports. This presentation will give an overview of a deployment system I wrote using SaltStack to provide immediately consistent deployment for software. The deployment system has the ability for manual deploys or for continuous deployments using a Heroku-like workflow. It has support for multiple deployment schemes, with a default of Git deployment. The deployment system uses a combination of Salt peer runner calls, runners, modules, returners, pillars and grains as well as a deployment host running Apache, Sartoris (a git-deploy frontend), and Redis (for reporting).

SaltConf14 - Ryan Lane, Wikimedia - Immediate consistency with Trebuchet Depl...

SaltStack

This session will expand your knowledge of cutting-edge techniques for creating Salt states and formulas. Users will obtain a thorough understanding of how states interact with pillars, as well as map.jinja files. We'll discuss how to make formulas OS agnostic and show how the usage of external pillars combined with a map file can result in formulas that are easy to explain, easy to learn, and easy to update.

SaltConf14 - Forrest Alvarez, Choice Hotels - Salt Formulas and States

SaltStack

As a leading developer of highly scalable, large-scale Web services, Google was forced early on to develop systems to support the deployment and management of diverse workloads at an immense scale. As the broader developer community embraces cloud technologies we see significant parallels between the internal management infrastructure which Google has built over the last decade, and open source management technologies of today. This talk will describe Google's experience in managing large-scale compute services, draw parallels to open source efforts underway today, and sketch out how our past experience shapes our future development of the Google Cloud Platform.

SaltConf14 - Brendan Burns, Google - Management at Google Scale

SaltStack

Let's set aside the buzzwords for a moment and have an honest discussion about DevOps. There is the idea of putting more Dev into Ops, but just as crucial (if not more crucial) is getting your Devs to think more like Ops. Most developers have little to no experience dealing with production environments, and helping them add value to DevOps efforts can be difficult. This talk will cover practical ways of mentoring Devs into more DevOps skills and responsibilities. Ultimately, the goal is to help your Devs gain the skills leading to better production health, application performance and uptime. Of course, we'll also consider how SaltStack can help.

SaltConf14 - Justin Carmony, Deseret Digital Media - Teaching Devs About DevOps

SaltStack

As infrastructure scales, simple tasks become increasingly difficult. For large infrastructures to be manageable, we use automation. But automation, like any power tool, comes with its own set of risks and challenges. Automation should be handled like production code, and great care should be exercised with power tools. This talk will cover how SaltStack is used at LinkedIn and offer tips and tricks for automating management with SaltStack at massive scale including a look at LinkedIn-inspired Salt features such as blacklist and prereq states. It will also cover Salt master and minion instrumentation and a compilation of how not to use Salt.

SaltConf14 - Thomas Jackson, LinkedIn - Safety with Power Tools

SaltStack

Google is making the power of its datacenter, network, and technology innovations available to the world through its Cloud services. This presentation will provide an overview of the Google Cloud Platform and a deeper dive on Google Compute Engine. Google recently made an open source contribution to SaltStack and now you can now use Salt Cloud to manage your Compute Engine resources (IaaS virtual machine services). Come find out more about Google's Cloud Platform and how you can leverage Google scale with SaltStack.

SaltConf14 - Eric johnson, Google - Orchestrating Google Compute Engine with ...

SaltStack

Real-time Cloud Management with SaltStack

SaltStack

A user's perspective on SaltStack and other configuration management tools

SaltStack

Adding to your Python Armory - OpenWest 2013

SaltStack

Real-time Infrastructure Management with SaltStack - OpenWest 2013

SaltStack

Mehr von SaltStack (20)