データの民主化のために StackStorm を活用した事例

•

4 gefällt mir•3,808 views

Yoshiyasu SAEKI

Tech Night @ Shiodome #8 https://techsio.connpass.com/event/83273/ 発表資料

Technologie

StackStorm
/ @laclefyoshi / ysaeki@r.recruit.co.jp

: / @laclefyoshi
• 2011/04
• 2015/09
•
• Spark Streaming (Spark Meetup December; 2015)
• Kafka AWS Kinesis (Apache Kafka Meetup Japan #1; 2016)
• (FutureOfData; 2016)
• Queryable State for Kafka Streams (Apache Kafka Meetup Japan #2; 2016)
• Apache Spark ( Geek Night #11; 2016)
• (BigData-JAWS #6; 2017)
• Apache Kafka 0.11 Exactly Once Semantics (Apache Kafka Meetup Japan #3; 2017)
• Apache Kafka  
(Apache Kafka Meetup Japan #4; 2018)
3

StackStorm
• https://stackstorm.com/
• OSS  
+ LDAP Auth plugin https://git.io/vxbbJ
•
• Shell Script
• Crontab
5

• JP1:
• Digdag:
• Jenkins: CI
• Airﬂow: GCP Cloud Composer
• Oozie: Hadoop
6

StackStorm
•
• High Availability
• Crontab Scheduler
•
•
•
• Runner Action Workﬂow
•
• REST API
7

$StackStorm API 1. test.test Action       2. API pack: test name: test runner_type: remote-shell-cmd parameters: p: type: string required: true curl -k -X POST -H "Content-type: application/json" -H "X-Auth-Token: $TOKEN" --data-binary '{"action": “test.test",   “parameters": {"p": "..."}}' "https://${ST2_ADDR}/api/v1/executions" 8$

StackStorm
• Design UI Commercial
• GUI
• OSS
• Commercial Role Based Access Control
• OSS
• Action Workﬂow
• Jinja/YAQL
9

st2XXX, mistral-XXX
replication
cluster
mistral
(PostgreSQL 9.6)
Runner Action
Workﬂow
Rule
10

•
• 1
• Timer
• st2rulesengine Timer HA
• https://docs.stackstorm.com/reference/ha.html#st2rulesengine
• MongoDB Primary
st2rulesengine Timer
• Concurrency policy Zookeeper
Timer
11

GCP?
• BigQuery  
• BigQuery
• SQL
• Tableau
• Pandas R
• StackStorm
BigQuery
12

StackStorm
1. StackStorm
2.  
3. StackStorm
4. StackStorm
5.
6.
17

StackStorm
• Amazon Data Value Chain  
Subscribe Deliver
https://www.slideshare.net/AmazonWebServices/a-look-under-the-hood-how-
amazoncom-uses-aws-services-for-analytics-at-massive-scale-abd329-reinvent-2017
18

Action
• Action
•
•
•
• LDAP
• Stackdriver
• Action
• SQL BigQuery
• BigQuery CSV
•
19

• StackStorm Action
• Action Workﬂow  
• YAML
•  
• YAML  
• JP1 GUI
20

API
• YAML
• GUI API
• Action Workﬂow
• StackStorm Action Workﬂow API
• API
•
22

1/2
23
2. StackStorm Web UI  
StackStorm API
1. Action Workﬂow
API

2/2
24
JP1
2. JP1 1
StackStorm API
1. Action Workﬂow
API

• StackStorm
• API
• StackStorm
•
• Rule
• JP1
• Workﬂow
•
25

26
• StackStorm
• Action Workﬂow YAML
•
•
Action Workﬂow

Weitere ähnliche Inhalte

Was ist angesagt?

My talk at Strata Data Conference, London, May 2017. https://conferences.oreilly.com/strata/strata-eu/public/schedule/detail/57619 Abstract: Modern businesses have data at their core, but this data is changing continuously. How can you harness this torrent of information in real time? The answer: stream processing. The core platform for streaming data is Apache Kafka, and thousands of companies are using Kafka to transform and reshape their industries, including Netflix, Uber, PayPal, Airbnb, Goldman Sachs, Cisco, and Oracle. Unfortunately, today’s common architectures for real-time data processing at scale suffer from complexity: to succeed, many technologies need to be stitched and operated together, and each individual technology is often complex by itself. This has led to a strong discrepancy between how we engineers would like to work and how we actually end up working in practice. Michael Noll explains how Apache Kafka helps you radically simplify your data processing architectures by building normal applications to serve your real-time processing needs rather than building clusters or similar special-purpose infrastructure—while still benefiting from properties typically associated exclusively with cluster technologies, like high scalability, distributed computing, and fault tolerance. Michael also covers Kafka’s Streams API, its abstractions for streams and tables, and its recently introduced interactive queries functionality. Along the way, Michael shares common use cases that demonstrate that stream processing in practice often requires database-like functionality and how Kafka allows you to bridge the worlds of streams and databases when implementing your own core business applications (for example, in the form of event-driven, containerized microservices). As you’ll see, Kafka makes such architectures equally viable for small-, medium-, and large-scale use cases.

Rethinking Stream Processing with Apache Kafka: Applications vs. Clusters, St...

Michael Noll

Interacting with Apache Kafka seems straightforward at first, you “just” push and pull messages. Yet it can quickly become a source of frustration as the user encounters timeouts, vague error descriptions and disappearing messages. Experience helps a lot and I’m here to share what I know. In this talk you will learn the tips & tricks I wish I had known at the beginning of my Apache Kafka journey. We’ll discuss topics like producer acknowledgments, server and consumer parameters (auto_offset_reset anyone?) that are commonly overlooked causing lots of developer’s pain. I’ll share with you how to generate code that works as expected on the first run, making your first integration painless. These tips will kickstart your Apache Kafka experience in Python and save you hours of debugging.

Practical tips and tricks for Apache Kafka messages integration | Francesco T...

HostedbyConfluent

大規模環境でRailsと4年間付き合ってきて@ クックパッド * 食べログ合同勉強会

Takayuki Kyowa

Real time stock processing with apache nifi, apache flink and apache kafka

Timothy Spann

The field of astronomy is rapidly changing away from the traditional notion of a lone astronomer pointing a telescope at a single object in a static sky. Initiatives such as the Sloan Digital Sky Survey have ushered in a collaborative big data era of wide-field sky surveys, in which telescopes collect observations continuously while sweeping across the visible night sky. This method of data collection enables not only very deep imaging of far and faint objects but is also optimal for searching for objects that might be changing or moving. By analyzing the differences in astronomical image data from one night to the next, astronomers can detect "transient" objects, such as variable stars, supernova, and near Earth asteroids. New sky surveys provide a wealth of scientific value for astronomers but not without technical challenges. Survey data need to be automatically processed and the results immediately distributed to the scientific community in order to enable rapid follow-up observations as transient astronomy can be highly time sensitive. Detection alert data distribution mechanisms need to be robust and reliable to maintain scientific integrity without data loss. Additionally, alerting systems need to be scalable to support a data volume unprecedented in astronomy, as transient detection rates have increased to exceed all historical data in a single night. A streaming architecture is an ideal architecture for automated distribution and processing of transient data in real time as it is being collected. In this talk, we will discuss how Kafka and Avro are being used in wide-field astronomical sky survey pipelines to serialize and distribute transient data, the design choices behind this system, and how this alert stream system has been successfully deployed in production to distribute transient detection alerts to the scientific research community in excess of 1 million events per night.

Building a newsfeed from the Universe: Data streams in astronomy (Maria Patte...

confluent

Apache Kafka lessons learned @PAYBACK

Maxim Shelest

Apache Pulsar Community-Jennifer

StreamNative

Apache Spark is a gift to the big data community, which adds tons of new features on every release. However, it’s difficult to manage petabyte-scale Hadoop clusters with hundreds of edge nodes, multiple Spark releases and demonstrate operational efficiencies and standardization. In order to address these challenges, Paypal has developed and deployed a REST0based Spark platform: Spark Compute as a Service (SCaaS),which provides improved application development, execution, logging, security, workload management and tuning. This session will walk through the top challenges faced by PayPal administrators, developers and operations and describe how Paypal’s SCaaS platform overcomes them by leveraging open source tools and technologies, like Livy, Jupyter, SparkMagic, Zeppelin, SQL Tools, Kafka and Elastic. You’ll also hear about the improvements PayPal has added, which enable it to run greater than 10,000 Spark applications in production effectively.

Spark Compute as a Service at Paypal with Prabhu Kasinathan

Databricks

ELK-Stack is world’s most popular log management platform. These open-source products are most commonly used in log analysis in IT environments. Logstash collects and parses logs, Elasticsearch indexes and stores the information. Kibana then presents the data in visualizations that provide actionable insights into one’s environment/software. Ashwin is going to brief about ELK-stack and show how this popular log management platform can be used with BizTalk servers. Including installing ELK stack in Windows and demo on how BizTalk data can be logged and analyzed in ELK-Stack. And he is going to discuss about some of the uses cases you can use ELK-stack with BizTalk and Azure.

Using ELK-Stack (Elasticsearch, Logstash and Kibana) with BizTalk Server

BizTalk360

Introducing Apache Kafka's Streams API - Kafka meetup Munich, Jan 25 2017

Michael Noll

Log analysis using Logstash,ElasticSearch and Kibana

Avinash Ramineni

E2E Data Pipeline - Apache Spark/Airflow/Livy

Rikin Tanna

Since Pac-Man was originally released in the '80s, it has been a beacon of fun and joy for people of all ages. What few people know is that this game can also be used to inspire developers on how to build event streaming applications. In this near-zero-slides talk, attendees will get to play the game to generate events. As they play, the presenter will write from scratch a scoreboard using ksqlDB -- an open-source event streaming database built for Apache Kafka. After building the scoreboard, it will be discussed the different strategies to make the data available elsewhere so any interested service could leverage it with ease. Examples of these services will be provided to monitor in near real-time the scoreboard, revealing whoever is the most proficient Pac-Man player in the room.

Building Event Streaming Applications with Pac-Man (Ricardo Ferreira, Conflue...

HostedbyConfluent

Apache Kafka 0.8 basic training (120 slides) covering: 1. Introducing Kafka: history, Kafka at LinkedIn, Kafka adoption in the industry, why Kafka 2. Kafka core concepts: topics, partitions, replicas, producers, consumers, brokers 3. Operating Kafka: architecture, hardware specs, deploying, monitoring, P&S tuning 4. Developing Kafka apps: writing to Kafka, reading from Kafka, testing, serialization, compression, example apps 5. Playing with Kafka using Wirbelsturm Audience: developers, operations, architects Created by Michael G. Noll, Data Architect, Verisign, https://www.verisigninc.com/ Verisign is a global leader in domain names and internet security. Tools mentioned: - Wirbelsturm (https://github.com/miguno/wirbelsturm) - kafka-storm-starter (https://github.com/miguno/kafka-storm-starter) Blog post at: http://www.michael-noll.com/blog/2014/08/18/apache-kafka-training-deck-and-tutorial/ Many thanks to the LinkedIn Engineering team (the creators of Kafka) and the Apache Kafka open source community!

Apache Kafka 0.8 basic training - Verisign

Michael Noll

Kafka Summit NYC 2017 - The Source of Truth: Why the New York Times Stores Ev...

confluent

Speaker: Cedrick Lunven, Developer Advocate, DataStax Speaker Bio: Cedrick is a Developer Advocate at DataStax where he finds opportunities to share his passions by speaking about developing distributed architectures and implementing reference applications for developers. In 2013, he created FF4j, an open source framework for Feature Toggle which he still actively maintains. He is now contributor in JHipster team. Talk Synopsis: We have all introduced more or less functional programming and asynchronous operations into our applications in order to speed up and distribute treatments (e.g., multi-threading, future, completableFuture, etc.). To build truly non-blocking components, optimize resource usage, and avoid "callback hell" you have to think reactive—everything is an event. From the frontend UI to database communications, it’s now possible to develop Java applications as fully reactive with frameworks like Spring WebFlux and Reactor. With high throughput and tunable consistency, applications built on top of Apache Cassandra™ fit perfectly within this pattern. DataStax has been developing Apache Cassandra drivers for years, and in the latest version of the enterprise driver we introduced reactive programming. During this session we will migrate, step by step, a vanilla CRUD Java service (SpringBoot / SpringMVC) into reactive with both code review and live coding. Bring home a working project! Filmed at Skills Matter/Code Node London on 9th May 2019 as part of the Big Data LDN Meetup Blueprint Series. Meetup sponsored by DataStax.

Speed Up Your Apache Cassandra™ Applications: A Practical Guide to Reactive P...

Matt Stubbs

Apache Pulsar: A borderless community

StreamNative

Scala at foursquare

jorgeortiz85

Livy is a new open source Spark REST Server for submitting and interacting with your Spark jobs from anywhere. Livy is conceptually based on the incredibly popular IPython/Jupyter, but implemented to better integrate into the Hadoop ecosystem with multi users. Spark can now be offered as a service to anyone in a simple way: Spark shells in Python or Scala can be ran by Livy in the cluster while the end user is manipulating them at his own convenience through a REST api. Regular non-interactive applications can also be submitted. The output of the jobs can be introspected and returned in a tabular format, which makes it visualizable in charts. Livy can point to a unique Spark cluster and create several contexts by users. With YARN impersonation, jobs will be executed with the actual permissions of the users submitting them. Livy also enables the development of Spark Notebook applications. Those are ideal for quickly doing interactive Spark visualizations and collaboration from a Web browser! This talk is technical and details the architecture and design decisions taken for developing this server, as well as its internals. It also describes the alternatives we tried and the challenges that were faced. The capabilities of Livy will then be lived demo in Hue’s Notebook Application through a real life scenario. https://spark-summit.org/eu-2015/events/building-a-rest-job-server-for-interactive-spark-as-a-service/

Spark Summit Europe: Building a REST Job Server for interactive Spark as a se...

gethue

Robin is a Developer Advocate at Confluent, the company founded by the creators of Apache Kafka, as well as an Oracle Groundbreaker Ambassador. His career has always involved data, from the old worlds of COBOL and DB2, through the worlds of Oracle and Hadoop, and into the current world with Kafka. His particular interests are analytics, systems architecture, performance testing and optimization. He blogs at http://cnfl.io/rmoff and http://rmoff.net/ and can be found tweeting grumpy geek thoughts as @rmoff. Outside of work he enjoys drinking good beer and eating fried breakfasts, although generally not at the same time.

Building Stream Processing Applications with Apache Kafka Using KSQL (Robin M...

confluent

Was ist angesagt? (20)

Rethinking Stream Processing with Apache Kafka: Applications vs. Clusters, St...

Practical tips and tricks for Apache Kafka messages integration | Francesco T...

大規模環境でRailsと4年間付き合ってきて@ クックパッド * 食べログ合同勉強会

Real time stock processing with apache nifi, apache flink and apache kafka

Building a newsfeed from the Universe: Data streams in astronomy (Maria Patte...

Apache Kafka lessons learned @PAYBACK

Apache Pulsar Community-Jennifer

Spark Compute as a Service at Paypal with Prabhu Kasinathan

Using ELK-Stack (Elasticsearch, Logstash and Kibana) with BizTalk Server

Introducing Apache Kafka's Streams API - Kafka meetup Munich, Jan 25 2017

Log analysis using Logstash,ElasticSearch and Kibana

E2E Data Pipeline - Apache Spark/Airflow/Livy

Building Event Streaming Applications with Pac-Man (Ricardo Ferreira, Conflue...

Apache Kafka 0.8 basic training - Verisign

Kafka Summit NYC 2017 - The Source of Truth: Why the New York Times Stores Ev...

Speed Up Your Apache Cassandra™ Applications: A Practical Guide to Reactive P...

Apache Pulsar: A borderless community

Scala at foursquare

Spark Summit Europe: Building a REST Job Server for interactive Spark as a se...

Building Stream Processing Applications with Apache Kafka Using KSQL (Robin M...

Ähnlich wie データの民主化のために StackStorm を活用した事例

KafkaとAWS Kinesisの比較

Yoshiyasu SAEKI

Since mid-2016, Spark-as-a-Service has been available to researchers in Sweden from the Rise SICS ICE Data Center at www.hops.site. In this session, Dowling will discuss the challenges in building multi-tenant Spark structured streaming applications on YARN that are metered and easy-to-debug. The platform, called Hopsworks, is in an entirely UI-driven environment built with only open-source software. Learn how they use the ELK stack (Elasticsearch, Logstash and Kibana) for logging and debugging running Spark streaming applications; how they use Grafana and InfluxDB for monitoring Spark streaming applications; and, finally, how Apache Zeppelin can provide interactive visualizations and charts to end-users. This session will also show how Spark applications are run within a ‘project’ on a YARN cluster with the novel property that Spark applications are metered and charged to projects. Projects are securely isolated from each other and include support for project-specific Kafka topics. That is, Kafka topics are protected from access by users that are not members of the project. In addition, hear about the experiences of their users (over 150 users as of early 2017): how they manage their Kafka topics and quotas, patterns for how users share topics between projects, and the novel solutions for helping researchers debug and optimize Spark applications.hear about the experiences of their users (over 150 users as of early 2017): how they manage their Kafka topics and quotas, patterns for how users share topics between projects, and the novel solutions for helping researchers debug and optimize Spark applications.afka topics are protected from access by users that are not members of the project. We will also discuss the experiences of our users (over 150 users as of early 2017): how they manage their Kafka topics and quotas, patterns for how users share topics between projects, and our novel solutions for helping researchers debug and optimize Spark applications.

Structured-Streaming-as-a-Service with Kafka, YARN, and Tooling with Jim Dowling

Databricks

Kafka Summit SF Apr 26 2016 - Generating Real-time Recommendations with NiFi,...

Chris Fregly

ストリーム処理を支えるキューイングシステムの選び方

Yoshiyasu SAEKI

Apereo CAS: State of the Project 2018

Misagh Moayyed

In a recent survey, 90% of over 2400 developers reported having at least some real-time functionality in their systems. Enterprises are realizing that the ability to extract value from streaming data in near real-time is the new competitive advantage. Two technologies–Akka Streams and Kafka Streams–have emerged as popular tools to use with Apache Kafka for addressing the shared requirements of availability, scalability, and resilience for both streaming microservices and Fast Data. So which one should you use for specific use cases?

Akka Streams And Kafka Streams: Where Microservices Meet Fast Data

Lightbend

Kinesis and Spark Streaming - Advanced AWS Meetup - August 2014

Chris Fregly

DataMass Summit 2019 Edition --> http://summit.datamass.io There is quite a bit to learn about any stream processing engine. But at a reasonably high level they actually are very similar and have lots in common. Not only do all have to offer a high-level stream processing API to describe distributed streaming dataflows, but also a low-level API for more sophisticated streaming topologies. The engines translate the dataflow description into their internal runtime representation. That’s where the differences are and where we’ll be looking at. This talk compares two modern stream processing engines — Kafka Streams and Spark Structured Streaming. We’ll be talking about their internals and how the engines manage stateless and stateful streams. You will learn about their similarities and differences that should shed more light on the question when to use which engine.

Kafka Streams VS Spark Structured Streaming - Modern Stream Processing Engin...

Jacek Laskowski

Minami Aoyama Night #1 で発表したスライドです。ザッパラスで行っているリーンな開発におけるインフラ周りの話をしました。また、タイミングよく @i2key さんのブログポスト「新規事業のグロース期を支えるエンジニアリングについて」が投稿されましたが、その文脈においての一部が ElasticBeanstalk を使ったインフラストラクチャなのではないかと感じています。

ElasticBeanstalk で新規事業を爆速ローンチする

Ryo Shibayama

Given at Data Day Seattle 2015. Bitly generates over 9 billion clicks on shortened links a month, as well as over 100 million unique link shortens. Analyzing data of this scale is not without its challenges. At Bitly, we have started adopting Apache Spark as a way to process our data. In this talk, I’ll elaborate on how I use Spark as part of my data science workflow. I’ll cover how Spark fits into our existing architecture, the kind of problems I’m solving with Spark, and the benefits and challenges of using Spark for large-scale data science.

Data Science at Scale: Using Apache Spark for Data Science at Bitly

Sarah Guido

Java-Jersey 到 Python-Flask 服務不中斷重構之旅

Max Lai

Perl ウェブ開発の中世〜CGI と Plack の間〜

鉄次尾形

Flink September 2015 Community Update

Robert Metzger

Jug - ecosystem

Florent Ramiere

Streaming Big Data with Spark, Kafka, Cassandra, Akka & Scala (from webinar)

Helena Edelson

Spark Uber Development Kit

DataWorks Summit/Hadoop Summit

Queryable State for Kafka Streamsを使ってみた

Yoshiyasu SAEKI

Chti jug - 2018-06-26

Florent Ramiere

Apache spark

Sameer Mahajan

DevNation Live 2020 - What's new with Apache Camel 3

Claus Ibsen

Ähnlich wie データの民主化のために StackStorm を活用した事例 (20)

KafkaとAWS Kinesisの比較

Structured-Streaming-as-a-Service with Kafka, YARN, and Tooling with Jim Dowling

Kafka Summit SF Apr 26 2016 - Generating Real-time Recommendations with NiFi,...

ストリーム処理を支えるキューイングシステムの選び方

Apereo CAS: State of the Project 2018

Akka Streams And Kafka Streams: Where Microservices Meet Fast Data

Kinesis and Spark Streaming - Advanced AWS Meetup - August 2014

Kafka Streams VS Spark Structured Streaming - Modern Stream Processing Engin...

ElasticBeanstalk で新規事業を爆速ローンチする

Data Science at Scale: Using Apache Spark for Data Science at Bitly

Java-Jersey 到 Python-Flask 服務不中斷重構之旅

Perl ウェブ開発の中世〜CGI と Plack の間〜

Flink September 2015 Community Update

Jug - ecosystem

Streaming Big Data with Spark, Kafka, Cassandra, Akka & Scala (from webinar)

Spark Uber Development Kit

Queryable State for Kafka Streamsを使ってみた

Chti jug - 2018-06-26

Apache spark

DevNation Live 2020 - What's new with Apache Camel 3

Kürzlich hochgeladen

Strategies for Landing an Oracle DBA Job as a Fresher

Remote DBA Services

Effective data discovery is crucial for maintaining compliance and mitigating risks in today's rapidly evolving privacy landscape. However, traditional manual approaches often struggle to keep pace with the growing volume and complexity of data. Join us for an insightful webinar where industry leaders from TrustArc and Privya will share their expertise on leveraging AI-powered solutions to revolutionize data discovery. You'll learn how to: - Effortlessly maintain a comprehensive, up-to-date data inventory - Harness code scanning insights to gain complete visibility into data flows leveraging the advantages of code scanning over DB scanning - Simplify compliance by leveraging Privya's integration with TrustArc - Implement proven strategies to mitigate third-party risks Our panel of experts will discuss real-world case studies and share practical strategies for overcoming common data discovery challenges. They'll also explore the latest trends and innovations in AI-driven data management, and how these technologies can help organizations stay ahead of the curve in an ever-changing privacy landscape.

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

TrustArc

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

Product Anonymous

EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER

MadyBayot

The Good, the Bad and the Governed - Why is governance a dirty word? David O'Neill, Chief Operating Officer - APIContext Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...

apidays

presentation ICT roal in 21st century education

jfdjdjcjdnsjd

ICT role in 21st century education and its challenges

rafiqahmad00786416

Accelerating FinTech Innovation: Unleashing API Economy and GenAI Vasa Krishnan, Chief Technology Officer - FinResults Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...

apidays

FWD Group - Insurer Innovation Award 2024

The Digital Insurer

The value of a flexible API Management solution for Open Banking Steve Melan, Manager for IT Innovation and Architecture - State's and Saving's Bank of Luxembourg Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - The value of a flexible API Management solution for O...

apidays

"I see eyes in my soup": How Delivery Hero implemented the safety system for ...

Zilliz

Manulife - Insurer Transformation Award 2024

The Digital Insurer

DBX First Quarter 2024 Investor Presentation

Dropbox

Dubai, often portrayed as a shimmering oasis in the desert, faces its own set of challenges, including the occasional threat of flooding. Despite its reputation for opulence and modernity, the emirate is not immune to the forces of nature. In recent years, Dubai has experienced sporadic but significant floods, testing the resilience of its infrastructure and communities. Among the critical lifelines in this bustling metropolis is the Dubai International Airport, a bustling hub that connects the city to the world. This article explores the intersection of Dubai flood events and the resilience demonstrated by the Dubai International Airport in the face of such challenges.

Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...

Orbitshub

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

Juan lago vázquez

AWS Community Day CPH - Three problems of Terraform

Andrey Devyatkin

Following the popularity of "Cloud Revolution: Exploring the New Wave of Serverless Spatial Data," we're thrilled to announce this much-anticipated encore webinar. In this sequel, we'll dive deeper into the Cloud-Native realm by uncovering practical applications and FME support for these new formats, including COGs, COPC, FlatGeoBuf, GeoParquet, STAC, and ZARR. Building on the foundation laid by industry leaders Michelle Roby of Radiant Earth and Chris Holmes of Planet in the first webinar, this second part offers an in-depth look at the real-world application and behind-the-scenes dynamics of these cutting-edge formats. We will spotlight specific use-cases and workflows, showcasing their efficiency and relevance in practical scenarios. Discover the vast possibilities each format holds, highlighted through detailed discussions and demonstrations. Our expert speakers will dissect the key aspects and provide critical takeaways for effective use, ensuring attendees leave with a thorough understanding of how to apply these formats in their own projects. Elevate your understanding of how FME supports these cutting-edge technologies, enhancing your ability to manage, share, and analyze spatial data. Whether you're building on knowledge from our initial session or are new to the serverless spatial data landscape, this webinar is your gateway to mastering cloud-native formats in your workflows.

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Safe Software

Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...

Zilliz

MINDCTI Revenue Release Quarter One 2024

MIND CTI

Architecting Cloud Native Applications

WSO2

Kürzlich hochgeladen (20)

Strategies for Landing an Oracle DBA Job as a Fresher

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER

Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...

presentation ICT roal in 21st century education

ICT role in 21st century education and its challenges

Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...

FWD Group - Insurer Innovation Award 2024

Apidays New York 2024 - The value of a flexible API Management solution for O...

"I see eyes in my soup": How Delivery Hero implemented the safety system for ...

Manulife - Insurer Transformation Award 2024

DBX First Quarter 2024 Investor Presentation

Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

AWS Community Day CPH - Three problems of Terraform

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...

MINDCTI Revenue Release Quarter One 2024

Architecting Cloud Native Applications

データの民主化のために StackStorm を活用した事例

1. StackStorm / @laclefyoshi / ysaeki@r.recruit.co.jp

2. • StackStorm • • • 2

3. : / @laclefyoshi • 2011/04 • 2015/09 • • Spark Streaming (Spark Meetup December; 2015) • Kafka AWS Kinesis (Apache Kafka Meetup Japan #1; 2016) • (FutureOfData; 2016) • Queryable State for Kafka Streams (Apache Kafka Meetup Japan #2; 2016) • Apache Spark ( Geek Night #11; 2016) • (BigData-JAWS #6; 2017) • Apache Kafka 0.11 Exactly Once Semantics (Apache Kafka Meetup Japan #3; 2017) • Apache Kafka   (Apache Kafka Meetup Japan #4; 2018) 3

4. StackStorm

5. StackStorm • https://stackstorm.com/ • OSS   + LDAP Auth plugin https://git.io/vxbbJ • • Shell Script • Crontab 5

6. • JP1: • Digdag: • Jenkins: CI • Airﬂow: GCP Cloud Composer • Oozie: Hadoop 6

7. StackStorm • • High Availability • Crontab Scheduler • • • • Runner Action Workﬂow • • REST API 7

8. StackStorm API 1. test.test Action       2. API pack: test name: test runner_type: remote-shell-cmd parameters: p: type: string required: true curl -k -X POST -H "Content-type: application/json" -H "X-Auth-Token: $TOKEN" --data-binary '{"action": “test.test",   “parameters": {"p": "..."}}' "https://${ST2_ADDR}/api/v1/executions" 8

9. StackStorm • Design UI Commercial • GUI • OSS • Commercial Role Based Access Control • OSS • Action Workﬂow • Jinja/YAQL 9

10. st2XXX, mistral-XXX replication cluster mistral (PostgreSQL 9.6) Runner Action Workﬂow Rule 10

11. • • 1 • Timer • st2rulesengine Timer HA • https://docs.stackstorm.com/reference/ha.html#st2rulesengine • MongoDB Primary st2rulesengine Timer • Concurrency policy Zookeeper Timer 11

12. GCP? • BigQuery   • BigQuery • SQL • Tableau • Pandas R • StackStorm BigQuery 12

13.

14. • • DB • • • →   14

15. 1/2 • 15

16. 2/2 • StackStorm 16

17. StackStorm 1. StackStorm 2.   3. StackStorm 4. StackStorm 5. 6. 17

18. StackStorm • Amazon Data Value Chain   Subscribe Deliver https://www.slideshare.net/AmazonWebServices/a-look-under-the-hood-how- amazoncom-uses-aws-services-for-analytics-at-massive-scale-abd329-reinvent-2017 18

19. Action • Action • • • • LDAP • Stackdriver • Action • SQL BigQuery • BigQuery CSV • 19

20. • StackStorm Action • Action Workﬂow   • YAML •   • YAML   • JP1 GUI 20

21. 21

22. API • YAML • GUI API • Action Workﬂow • StackStorm Action Workﬂow API • API • 22

23. 1/2 23 2. StackStorm Web UI   StackStorm API 1. Action Workﬂow API

24. 2/2 24 JP1 2. JP1 1 StackStorm API 1. Action Workﬂow API

25. • StackStorm • API • StackStorm • • Rule • JP1 • Workﬂow • 25

26. 26 • StackStorm • Action Workﬂow YAML • • Action Workﬂow

データの民主化のために StackStorm を活用した事例

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie データの民主化のために StackStorm を活用した事例

Ähnlich wie データの民主化のために StackStorm を活用した事例 (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

データの民主化のために StackStorm を活用した事例