Scaling a backend for a big data and blockchain environment by Rafael Ríos at Big Data Spain 2017

•

0 likes•1,559 views

This document discusses scaling the backend of a financial platform for big data and blockchain. It describes challenges integrating big data using Apache Spark and Cassandra for tasks like predictive modeling, recommendations, and credit scoring. It also covers using a microservices architecture with Spring Cloud, Docker, and Kubernetes for deployment. Blockchain integration involves a private Ethereum network on Kubernetes for tokenization and a connection to the public Ethereum mainnet using Infura for payments and transfers.

Technology

Scaling a backend for big data
and blockchain environment

1. Introduction (The companies, project and me)
2. Backend challenge
3. Big Data Integration
4. Blockchain Integration
·············· P. 3
···················································
················ P. 7
···················································
·············· P. 17
···················································
········ P. 21

@ganchix
Head of Blockchain / Backend
Me
ganchix

Fecha
The project
• Financial platform
• Marketplace
• Tokenization
• Operate with cryptocurrency
• Liquidity predictive models
• Credit scoring
• Product recommendation

Backend challenge - The evolution
Dapp
Smart Contract
Ethereum

Backend challenge - The evolution
Dapps problem:
• Noncryptocurrency users
• Problem with some integrations
• Legal problem

Fecha
Backend challenge - Why microservices?
• Migration of Dapp easier
• Easy to scale
• Polyglot Database and Languages

Why don't use exclusively blockchain
with a database?

1. Spring Cloud
Netflix and Kubernetes
• Easy to learn.
• Nice integrations
• Spring 5 reactive
2. Docker
• Most adopted vendor technology for containers
• Well supported
3. Kubernetes
• Multi-cloud provider and on-premises data centers
• Self-repair and health check capabilities
• Auto-scale
Backend challenge - Microservice Architecture Stack

Backend challenge - Microservice Architecture Stack

• Tasks are hard, needs:
• Time
• Resources
• Not Real Time is needed.
• Event Driven Architecture.
Big Data Integration - Events

Big Data Integration - RabbitMQ vs Kafka
KafkaRabbitMq
• RabbitMQ is designed as a general
purpose message broker
• Support existing protocols like
AMQP, STOMP, MQTT.
• Finer-grained consistency
control/guarantees on a peer-
message.
• Complex routing.
• Apache Kafka is designed for high
volume publish-subscribe
messages and streams, meant to
be durable, fast, and scalable.
• Event Sourcing
• Your application needs access to
stream history.
• No complex routing.
https://content.pivotal.io/blog/understanding-when-to-use-rabbitmq-or-apache-kafka

• Deployed in Kubernetes.
• Only accessible by NodeJS API.
• All keys are stored in secrets vaults.
• Used for:
• Tokenization
• Transactions of users
Blockchain Integration - Private Ethereum

Blockchain Integration - Private Ethereum

Blockchain Integration - Ethereum Main Net

Blockchain Integration - Ethereum Main Net
• We are the owner of the wallets
• We use Infura to connect blockchain
• Used for:
• Payment
• Transfers

@ganchix
@2getherbank
@bigeeksoftwa
re
Thanks!
ICO is coming!

What's hot

You cannot operate what you cannot measure. In this talk, I am going to present the built-in metrics framework of Kafka Streams that supports monitoring Kafka Streams applications. You will learn how to setup monitoring of metrics for your Kafka Streams applications and you will hear about the following recent improvements to the metrics framework that aim to extend and simplify monitoring. KIP-444 aims to simplify and extend the built-in metrics framework. The RocksDB metrics introduced in KIP-471 and KIP-607 allow you to look directly into the built-in persistent state stores of your Kafka Streams applications. Finally, KIP-613 specifies metrics that measure end-to-end latencies in your applications. This talk will help you collect intel about the behavior of your Kafka Streams applications, and will allow you to reason about the deployment. In the end, you will be able to better understand your applications and run them in a more robust manner.

Mind the App: How to Monitor Your Kafka Streams Applications | Bruno Cadonna,...

HostedbyConfluent

Legacy migration is a journey. Mainframes cannot be replaced in a single project. A big bang will fail. This has to be planned long-term. Mainframe offloading and replacement with Apache Kafka and its ecosystem can be used to keep a more modern data store in real-time sync with the mainframe, while at the same time persisting the event data on the bus to enable microservices, and deliver the data to other systems such as data warehouses and search indexes. This session walks through the different steps some companies are already gone through. Technical options like Change Data Capture (CDC), MQ, and third-party tools for mainframe integration, offloading and replacement are explored.

Mainframe Integration, Offloading and Replacement with Apache Kafka | Kai Wae...

HostedbyConfluent

In this webinar, you’ll also be introduced to DataStax Apache Kafka Connector, and get a brief demonstration of this groundbreaking technology. You’ll directly experience how this tool can help you stream data from Kafka topics into DataStax Enterprise versions of Cassandra. The future of your organization won’t wait. Register now to reserve your spot in this exciting new webinar. Youtube: https://youtu.be/HmkNb8twUNk

Webinar | Better Together: Apache Cassandra and Apache Kafka

DataStax

Making Kafka Cloud Native | Jay Kreps, Co-Founder & CEO, Confluent

HostedbyConfluent

Presentation from South Bay.NET meetup on 3/30. Speaker: Matt Howlett, Software Engineer at Confluent Apache Kafka is a scalable streaming platform that forms a key part of the infrastructure at many companies including Uber, Netflix, Walmart, Airbnb, Goldman Sachs and LinkedIn. In this talk Matt will give a technical overview of Kafka, discuss some typical use cases (from surge pricing to fraud detection to web analytics) and show you how to use Kafka from within your C#/.NET applications.

Stream Processing with Apache Kafka and .NET

confluent

The adoption and popularity of the microservices architecture continues to grow across a spectrum of enterprises in every industry. Although a consensus on an implementation standard has yet to be reached, advanced design patterns and lessons learned about the complexities and pitfalls of deploying microservices at scale have been established by thought leaders and the development community. With Redis and Kafka becoming de facto standards across most microservices architectures, we will discuss how their combination can be used to simplify the implementation of event-driven design patterns that will provide real-time performance, scalability, resiliency, traceability to ensure compliance, observability, reduced technology sprawl, and scale to thousands of services. In this discussion, we will decompose a real-time event-driven payment-processing microservices workflow to explore capturing telemetry data, event sourcing, CQRS, orchestrated SAGA workflows, inter-service communication, state machines, and more.

Redis and Kafka - Advanced Microservices Design Patterns Simplified

Allen Terleto

Presented at Kafka Summit 2016 Operating out of multiple datacenters is a large part of most disaster recovery plans, but it brings extra complications to our data pipelines. Instead of having a straight path from front to back, it now has forks and dead ends and odd little use cases that don’t match up with a perfect view of the world. This talk will focus on how to best utilize Apache Kafka in this world, including basic architectures for multi-datacenter and multi-tier clusters. We will also touch on how to assure messages make it from producer to consumer, and how to monitor the entire ecosystem.

More Datacenters, More Problems

Todd Palino

Building Streaming Data Pipelines with Google Cloud Dataflow and Confluent Cl...

HostedbyConfluent

APAC Kafka Summit - Best Of

confluent

Presented at Orange County Advanced Analytics and Big Data Meetup, June 21 2019. Apache Kafka has fast become the dominant messaging technology for the enterprise; if you're a data scientist or data engineer and you have not yet worked with Kafka, that situation will likely change soon! In this session, Pat Patterson, director of evangelism at StreamSets, explains what Kafka is, why it has disrupted the previous generation of messaging products, and how you can use open source products to build dataflow pipelines with Kafka, without writing code.

Data Integration with Apache Kafka: What, Why, How

Pat Patterson

Saxo Bank is on a growth journey and Kafka is a critical component to that success. Securing our financial event streams is a top priority for us and initially we started with an on-prem Kafka cluster secured with (the de-facto) Kerberos. However, as we modernize and scale, the demands of hybrid cloud, multiple domains, polyglot computing and Data Mesh require us to also modernize our approach to security. In this talk, we will describe how we took the default (non-production ready) Kafka OAuth implementation and productionized it to work with Kafka in Azure Cloud, including the Kafka stack and clients. By enabling both Kerberos and OAuth running on-prem and in the cloud, we now plan to gracefully retire Kerberos from our estate.

How we eased out security journey with OAuth (Goodbye Kerberos!) | Paul Makka...

HostedbyConfluent

For many industries the need to group together related events based on a period of activity or inactivity is key. Advertising businesses, content producers are just a few examples of where session windows can be used to better understand user behavior. While such sessionization has been possible in Apache Kafka up to this point, implementing it has been rather complex and required leveraging low-level APIs. In the most recent release of Kafka, however, new capabilities have been added making session windows much easier to implement. In this online talk, we’ll introduce the concept of a session window, talk about common use cases, and walk through how Apache Kafka can be used for session-oriented use cases.

user Behavior Analysis with Session Windows and Apache Kafka's Streams API

confluent

Kafka has fast become the center of streaming analytics applications in the modern digital enterprise. Kafka operates in the context of a broad ecosystem of data lifecycle components which need a consistent platform of security, monitoring, management and governance. This problem becomes paramount when your streaming architectures go hybrid by spanning from on-premises to the cloud. Throw in the reality of a multi-cloud setup that a lot of enterprises are facing and now, you have a complex streaming architecture that is difficult to operationally manage, monitor, secure or govern. Cloudera remains committed to an open community driven approach and increasing the ease of use and visibility for Kafka based solutions. Attend this session to understand more about how streaming architectures can be extended easily to the hybrid cloud and multi-cloud. Also, learn about our plans for further community contributions.

Kafka in Context, Cloud, & Community (Simon Elliston Ball, Cloudera) Kafka Su...

HostedbyConfluent

Apache Kafka is taking the world by storm and is rapidly becoming the de-facto event bus for event-driven and streaming applications that respond to events and data in real time. OpenShift Streams for Apache Kafka is Red Hat's fully hosted and managed Apache Kafka service targeting development teams that want to incorporate streaming data and scalable messaging in their applications, without the burden of setting up and maintaining a Kafka cluster infrastructure. In this session you will discover how Apache Kafka can be used in an IoT scenario to ingest data from devices and make them available in real-time to other applications. More specifically you will learn how to: Simulate devices that send MQTT messages to a MQTT broker Use Apache Camel and Camel-K to bridge MQTT with Apache Kafka Use Kafka Streams in a Quarkus application to process the device messages Query the state of the devices using GraphQ

Kafka at the Edge: an IoT scenario with OpenShift Streams for Apache Kafka | ...

Red Hat Developers

Using Kafka to stream data into TigerGraph, a distributed graph database, is a common pattern in our customers’ data architecture. In the TigerGraph database, Kafka Connect framework was used to build the native S3 data loader. In TigerGraph Cloud, we will be building native integration with many data sources such as Azure Blob Storage and Google Cloud Storage using Kafka as an integrated component for the Cloud Portal. In this session, we will be discussing both architectures: 1. built-in Kafka Connect framework within TigerGraph database; 2. using Kafka cluster for cloud native integration with other popular data sources. Demo will be provided for both data streaming processes.

How a distributed graph analytics platform uses Apache Kafka for data ingesti...

HostedbyConfluent

Guru Sattanathan, Confluent, Senior Solutions Engineer Enterprise Integration technologies (aka Middleware) are the key enablers when it comes to Real-time data flows or Event Driven Architecture. Starting from real-time payments, e-commerce, travel booking systems, etc, everything is powered by a middleware underneath. It did transform a lot of things but with caveats! Are ESB’s & MQ’s enough for today’s integration needs? Do you know their technical debts? If you are someone looking at integrating your applications or an Integration Architect this session is for you. It's time to refresh yourself and see how organizations are building integrations today. In this session, we will go in this order: -Recap on Enterprise Integration technologies -What are the key flaws & What needs improvement? -What is Apache Kafka? -Rethinking Integration using Apache Kafka https://www.meetup.com/KafkaMelbourne/events/280590162/

Death of the dumb pipes: Using Apache Kafka® for Integration projects

HostedbyConfluent

Modern businesses have data at their core, and this data is changing continuously. How can we harness this torrent of information in real-time? The answer is stream processing, and the technology that has since become the core platform for streaming data is Apache Kafka. Among the thousands of companies that use Kafka to transform and reshape their industries are the likes of Netflix, Uber, PayPal, and AirBnB, but also established players such as Goldman Sachs, Cisco, and Oracle. Unfortunately, today’s common architectures for real-time data processing at scale suffer from complexity: there are many technologies that need to be stitched and operated together, and each individual technology is often complex by itself. This has led to a strong discrepancy between how we, as engineers, would like to work vs. how we actually end up working in practice. In this session we talk about how Apache Kafka helps you to radically simplify your data processing architectures. We cover how you can now build normal applications to serve your real-time processing needs — rather than building clusters or similar special-purpose infrastructure — and still benefit from properties such as high scalability, distributed computing, and fault-tolerance, which are typically associated exclusively with cluster technologies. Notably, we introduce Kafka’s Streams API, its abstractions for streams and tables, and its recently introduced Interactive Queries functionality. As we will see, Kafka makes such architectures equally viable for small, medium, and large scale use cases.

Introducing Kafka's Streams API

confluent

A Look into the Mirror: Patterns and Best Practices for MirrorMaker2 | Cliff ...

HostedbyConfluent

Are you looking for a cloud-based architecture that includes the best of breed streaming and database technologies? In this session you will learn how to setup and configure the Confluent Cloud with MongoDB Atlas. We'll start the journey learning about the basic connectivity between the two cloud services and end with a brief discovery of what you can do with data once it is in MongoDB Atlas. By the end of this session you will know how to securely setup and configure the MongoDB Atlas connectors in the Confluent Cloud in both a source and sink configuration.

Streaming Data in the Cloud with Confluent and MongoDB Atlas | Robert Walters...

HostedbyConfluent

Scaling an Event-Driven Architecture with IBM and Confluent | Antony Amanse a...

HostedbyConfluent

What's hot (20)

Mind the App: How to Monitor Your Kafka Streams Applications | Bruno Cadonna,...

Mainframe Integration, Offloading and Replacement with Apache Kafka | Kai Wae...

Webinar | Better Together: Apache Cassandra and Apache Kafka

Making Kafka Cloud Native | Jay Kreps, Co-Founder & CEO, Confluent

Stream Processing with Apache Kafka and .NET

Redis and Kafka - Advanced Microservices Design Patterns Simplified

More Datacenters, More Problems

Building Streaming Data Pipelines with Google Cloud Dataflow and Confluent Cl...

APAC Kafka Summit - Best Of

Data Integration with Apache Kafka: What, Why, How

How we eased out security journey with OAuth (Goodbye Kerberos!) | Paul Makka...

user Behavior Analysis with Session Windows and Apache Kafka's Streams API

Kafka in Context, Cloud, & Community (Simon Elliston Ball, Cloudera) Kafka Su...

Kafka at the Edge: an IoT scenario with OpenShift Streams for Apache Kafka | ...

How a distributed graph analytics platform uses Apache Kafka for data ingesti...

Death of the dumb pipes: Using Apache Kafka® for Integration projects

Introducing Kafka's Streams API

A Look into the Mirror: Patterns and Best Practices for MirrorMaker2 | Cliff ...

Streaming Data in the Cloud with Confluent and MongoDB Atlas | Robert Walters...

Scaling an Event-Driven Architecture with IBM and Confluent | Antony Amanse a...

Similar to Scaling a backend for a big data and blockchain environment by Rafael Ríos at Big Data Spain 2017

FIWARE Wednesday Webinars - Short Term History within Smart Systems

FIWARE

OSACon 2023_ Unlocking Financial Data with Real-Time Pipelines Unlocking Financial Data with Real-Time Pipelines Financial institutions thrive on accurate and timely data to drive critical decision-making processes, risk assessments, and regulatory compliance. However, managing and processing vast amounts of financial data in real-time can be a daunting task. To overcome this challenge, modern data engineering solutions have emerged, combining powerful technologies like Apache Flink, Apache NiFi, Apache Kafka, and Iceberg to create efficient and reliable real-time data pipelines. In this talk, we will explore how this technology stack can unlock the full potential of financial data, enabling organizations to make data-driven decisions swiftly and with confidence. Introduction: Financial institutions operate in a fast-paced environment where real-time access to accurate and reliable data is crucial. Traditional batch processing falls short when it comes to handling rapidly changing financial markets and responding to customer demands promptly. In this talk, we will delve into the power of real-time data pipelines, utilizing the strengths of Apache Flink, Apache NiFi, Apache Kafka, and Iceberg, to unlock the potential of financial data. Key Points to be Covered: Introduction to Real-Time Data Pipelines: a. The limitations of traditional batch processing in the financial domain. b. Understanding the need for real-time data processing. Apache Flink: Powering Real-Time Stream Processing: a. Overview of Apache Flink and its role in real-time stream processing. b. Use cases for Apache Flink in the financial industry. c. How Flink enables fast, scalable, and fault-tolerant processing of streaming financial data. Apache Kafka: Building Resilient Event Streaming Platforms: a. Introduction to Apache Kafka and its role as a distributed streaming platform. b. Kafka's capabilities in handling high-throughput, fault-tolerant, and real-time data streaming. c. Integration of Kafka with financial data sources and consumers. Apache NiFi: Data Ingestion and Flow Management: a. Overview of Apache NiFi and its role in data ingestion and flow management. b. Data integration and transformation capabilities of NiFi for financial data. c. Utilizing NiFi to collect and process financial data from diverse sources. Iceberg: Efficient Data Lake Management: a. Understanding Iceberg and its role in managing large-scale data lakes. b. Iceberg's schema evolution and table-level metadata capabilities. c. How Iceberg simplifies data lake management in financial institutions. Real-World Use Cases: a. Real-time fraud detection using Flink, Kafka, and NiFi. b. Portfolio risk analysis with Iceberg and Flink. c. Streamlined regulatory reporting leveraging all four technologies. Best Practices and Considerations: a. Architectural considerations when building real-time financial data pipelines. b. Ensuring data integrity, security, and compliance in real-time pipelines. c. Scalability an

OSACon 2023_ Unlocking Financial Data with Real-Time Pipelines

Timothy Spann

From Kafka to BigQuery - Strata Singapore

Ofir Sharony

Geek Nights Hong Kong

Rahul Gupta

IBM Blockchain Platform - Architectural Good Practices v1.0

Matt Lucas

Wwc developing hyperledger applications v4

LennartF

Time series denver an introduction to prometheus

Bob Cotton

Citi Tech Talk: Hybrid Cloud

confluent

Apache Kafka is an open source event streaming platform. It is often used to complement or even replace existing middleware to integrate applications and build microservice architectures. Apache Kafka is already used in various projects in almost every bigger company today. Understood, battled-tested, highly scalable, reliable, real-time. Blockchain is a different story. This technology is a lot in the news, especially related to cryptocurrencies like Bitcoin. But what is the added value for software architectures? Is blockchain just hype and adds complexity? Or will it be used by everybody in the future, like a web browser or mobile app today? And how is it related to an integration architecture and event streaming platform? This session explores use cases for blockchains and discusses different alternatives such as Hyperledger, Ethereum and a Kafka-native tamper-proof blockchain implementation. Different architectures are discussed to understand when blockchain really adds value and how it can be combined with the Apache Kafka ecosystem to integrate blockchain with the rest of the enterprise architecture to build a highly scalable and reliable event streaming infrastructure. Speakers: Kai Waehner, Technology Evangelist, Confluent Stephen Reed, CTO, Co-Founder, AiB

Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka ...

confluent

Benefits of an Agile Data Fabric for Business Intelligence

DataWorks Summit/Hadoop Summit

Read this blog for an overview of Kalix: https://www.kalix.io/blog/kalix-move-to-the-cloud-extend-to-the-edge-go-beyond Abstract: What will the future of the Cloud and Edge look like for us as developers? We have great infrastructure nowadays, but that only solves half of the problem. The Serverless developer experience shows the way, but it’s clear that FaaS is not the final answer. What we need is a programming model and developer UX that takes full advantage of new Cloud and Edge infrastructure, allowing us to build general-purpose applications, without needless complexity. What if you only had to think about your business logic, public API, and how your domain data is structured, not worry about how to store and manage it? What if you could not only be serverless but become “databaseless” and forget about databases, storage APIs, and message brokers? Instead, what if your data just existed wherever it needed to be, co-located with the service and its user, at the edge, in the cloud, or in your own private network—always there and available, always correct and consistent? Where the data is injected into your services on an as-needed basis, automatically, timely, efficiently, and intelligently. Services, powered with this “data plane” of application state—attached to and available throughout the network—can run anywhere in the world: from the public Cloud to 10,000s of PoPs out at the Edge of the network, in close physical approximation to its users, where the co-location of state, processing, and end-user, ensures ultra-low latency and high throughput. Sounds exciting? Let me show you how we are making this vision a reality building a distributed real-time Data Plane PaaS using technologies like Akka, Kubernetes, gRPC, Linkerd, and more.

Kalix: Tackling the The Cloud to Edge Continuum

Jonas Bonér

Coherence RoadMap 2018

harvraja

Meetup: Streaming Data Pipeline Development In this interactive session, Tim will lead participants through how to best build streaming data pipelines. He will cover how to build applications from some common use cases and highlight tips, tricks, best practices and patterns. He will show how to build the easy way and then dive deep into the underlying open source technologies including Apache NiFi, Apache Flink, Apache Kafka and Apache Iceberg. If you wish to follow along, please download open source projects beforehand. You can also download this helpful streaming platform: https://docs.cloudera.com/csp-ce/latest/installation/topics/csp-ce-installing-ce.html All source code and slides will be shared for those interested in building their own FLaNK Apps. https://www.flankstack.dev/ You can join the meeting virtually here: https://cloudera.zoom.us/j/91603330726 Speaker - Tim Spann Tim Spann is a Principal Developer Advocate in Data In Motion for Cloudera. He works with Apache NiFi, Apache Pulsar, Apache Kafka, Apache Flink, Flink SQL, Apache Pinot, Trino, Apache Iceberg, DeltaLake, Apache Spark, Big Data, IoT, Cloud, AI/DL, machine learning, and deep learning. Tim has over ten years of experience with the IoT, big data, distributed computing, messaging, streaming technologies, and Java programming. Previously, he was a Developer Advocate at StreamNative, Principal DataFlow Field Engineer at Cloudera, a Senior Solutions Engineer at Hortonworks, a Senior Solutions Architect at AirisData, a Senior Field Engineer at Pivotal and a Team Leader at HPE. He blogs for DZone, where he is the Big Data Zone leader, and runs a popular meetup in Princeton & NYC on Big Data, Cloud, IoT, deep learning, streaming, NiFi, the blockchain, and Spark. Tim is a frequent speaker at conferences such as ApacheCon, DeveloperWeek, Pulsar Summit and many more. He holds a BS and MS in computer science.

Meetup: Streaming Data Pipeline Development

Timothy Spann

Big Data on Cloud Native Platform

Sunil Govindan

Big Data on Cloud Native Platform

Sunil Govindan

PayPal Data Lake Journey | 2017-Oct | San Diego | Teradata Edge of Next Gimel [http://www.gimel.io] is a Big Data Processing Library, open sourced by PayPal. https://www.youtube.com/watch?v=52PdNno_9cU&t=3s Gimel empowers analysts, scientists, data engineers alike to access a variety of Big Data / Traditional Data Stores - with just SQL or a single line of code (Unified Data API). This is possible via the Catalog of Technical properties abstracted from users, along with a rich collection of Data Store Connectors available in Gimel Library. A Catalog provider can be Hive or User Supplied (runtime) or UDC. In addition, PayPal recently open sourced UDC [Unified Data Catalog], which can host and serve the Technical Metatada of the Data Stores & Objects. Visit http://www.unifieddatacatalog.io to experience first hand.

PayPal datalake journey | teradata - edge of next | san diego | 2017 october ...

Deepak Chandramouli

To remain competitive, organizations need to democratize access to fast analytics, not only to gain real-time insights on their business but also to power smart apps that need to react in the moment. In this session, you will learn how Kafka and SingleStore enable modern, yet simple data architecture to analyze both fast paced incoming data as well as large historical datasets. In particular, you will understand why SingleStore is well suited process data streams coming from Kafka.

SingleStore & Kafka: Better Together to Power Modern Real-Time Data Architect...

HostedbyConfluent

Architecture patterns for distributed, hybrid, edge and global Apache Kafka deployments Multi-cluster and cross-data center deployments of Apache Kafka have become the norm rather than an exception. This session gives an overview of several scenarios that may require multi-cluster solutions and discusses real-world examples with their specific requirements and trade-offs, including disaster recovery, aggregation for analytics, cloud migration, mission-critical stretched deployments and global Kafka. Key takeaways: In many scenarios, one Kafka cluster is not enough. Understand different architectures and alternatives for multi-cluster deployments. Zero data loss and high availability are two key requirements. Understand how to realize this, including trade-offs. Learn about features and limitations of Kafka for multi cluster deployments Global Kafka and mission-critical multi-cluster deployments with zero data loss and high availability became the normal, not an exception.

Architecture patterns for distributed, hybrid, edge and global Apache Kafka d...

Kai Wähner

DataStax Enterprise & Apache Cassandra – Essentials for Financial Services – ...

Daniel Cohen

Introduction to Apache NiFi 1.11.4

Timothy Spann

Similar to Scaling a backend for a big data and blockchain environment by Rafael Ríos at Big Data Spain 2017 (20)

FIWARE Wednesday Webinars - Short Term History within Smart Systems

OSACon 2023_ Unlocking Financial Data with Real-Time Pipelines

From Kafka to BigQuery - Strata Singapore

Geek Nights Hong Kong

IBM Blockchain Platform - Architectural Good Practices v1.0

Wwc developing hyperledger applications v4

Time series denver an introduction to prometheus

Citi Tech Talk: Hybrid Cloud

Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka ...

Benefits of an Agile Data Fabric for Business Intelligence

Kalix: Tackling the The Cloud to Edge Continuum

Coherence RoadMap 2018

Meetup: Streaming Data Pipeline Development

Big Data on Cloud Native Platform

PayPal datalake journey | teradata - edge of next | san diego | 2017 october ...

SingleStore & Kafka: Better Together to Power Modern Real-Time Data Architect...

Architecture patterns for distributed, hybrid, edge and global Apache Kafka d...

DataStax Enterprise & Apache Cassandra – Essentials for Financial Services – ...

Introduction to Apache NiFi 1.11.4

Recently uploaded

This presentations targets students or working professionals. You may know Google for search, YouTube, Android, Chrome, and Gmail, but did you know Google has many developer tools, platforms & APIs? This comprehensive yet still high-level overview outlines the most impactful tools for where to run your code, store & analyze your data. It will also inspire you as to what's possible. This talk is 50 minutes in length.

Powerful Google developer tools for immediate impact! (2023-24 C)

wesley chun

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

Neo4j

Discord is a free app offering voice, video, and text chat functionalities, primarily catering to the gaming community. It serves as a hub for users to create and join servers tailored to their interests. Discord’s ecosystem comprises servers, each functioning as a distinct online community with its own channels dedicated to specific topics or activities. Users can engage in text-based discussions, voice calls, or video chats within these channels. Understanding Discord Servers Discord servers are virtual spaces where users congregate to interact, share content, and build communities. Servers may revolve around gaming, hobbies, interests, or fandoms, providing a platform for like-minded individuals to connect. Communication Features Discord offers a range of communication tools, including text channels for messaging, voice channels for real-time audio conversations, and video channels for face-to-face interactions. These features facilitate seamless communication and collaboration. What Does NSFW Mean? The acronym NSFW stands for “Not Safe For Work,” indicating content that may be inappropriate for professional or public settings. NSFW Content NSFW content encompasses material that is sexually explicit, violent, or otherwise graphic in nature. It often includes nudity, profanity, or depictions of sensitive topics.

Understanding Discord NSFW Servers A Guide for Responsible Users.pdf

UK Journal

Strategies for Landing an Oracle DBA Job as a Fresher

Remote DBA Services

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

The Digital Insurer

As privacy and data protection regulations evolve rapidly, organizations operating in multiple jurisdictions face mounting challenges to ensure compliance and safeguard customer data. With state-specific privacy laws coming up in multiple states this year, it is essential to understand what their unique data protection regulations will require clearly. How will data privacy evolve in the US in 2024? How to stay compliant? Our panellists will guide you through the intricacies of these states' specific data privacy laws, clarifying complex legal frameworks and compliance requirements. This webinar will review: - The essential aspects of each state's privacy landscape and the latest updates - Common compliance challenges faced by organizations operating in multiple states and best practices to achieve regulatory adherence - Valuable insights into potential changes to existing regulations and prepare your organization for the evolving landscape

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

TrustArc

Three things you will take away from the session: • How to run an effective tenant-to-tenant migration • Best practices for before, during, and after migration • Tips for using migration as a springboard to prepare for Copilot in Microsoft 365 Main ideas: Migration Overview: The presentation covers the current reality of cross-tenant migrations, the triggers, phases, best practices, and benefits of a successful tenant migration Considerations: When considering a migration, it is important to consider the migration scope, performance, customization, flexibility, user-friendly interface, automation, monitoring, support, training, scalability, data integrity, data security, cost, and licensing structure Next Wave: The next wave of change includes the launch of Copilot, which requires businesses to be prepared for upcoming changes related to Copilot and the cloud, and to consolidate data and tighten governance ShareGate: ShareGate can help with pre-migration analysis, configurable migration tool, and automated, end-user driven collaborative governance

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

sammart93

Scaling API-first – The story of a global engineering organization Ian Reasor, Senior Computer Scientist - Adobe Radu Cotescu, Senior Computer Scientist - Adobe Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe

apidays

Partners Life - Insurer Innovation Award 2024

The Digital Insurer

Finology Group – Insurtech Innovation Award 2024

The Digital Insurer

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

Product Anonymous

presentation ICT roal in 21st century education

jfdjdjcjdnsjd

Automating Google Workspace (GWS) & more with Apps Script

wesley chun

Building Digital Trust in a Digital Economy Veronica Tan, Director - Cyber Security Agency of Singapore Apidays Singapore 2024: Connecting Customers, Business and Technology (April 17 & 18, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...

apidays

With more memory available, system performance of three Dell devices increased, which can translate to a better user experience Conclusion When your system has plenty of RAM to meet your needs, you can efficiently access the applications and data you need to finish projects and to-do lists without sacrificing time and focus. Our test results show that with more memory available, three Dell PCs delivered better performance and took less time to complete the Procyon Office Productivity benchmark. These advantages translate to users being able to complete workflows more quickly and multitask more easily. Whether you need the mobility of the Latitude 5440, the creative capabilities of the Precision 3470, or the high performance of the OptiPlex Tower Plus 7010, configuring your system with more RAM can help keep processes running smoothly, enabling you to do more without compromising performance.

Boost PC performance: How more available memory can improve productivity

Principled Technologies

Developing An App To Navigate The Roads of Brazil

V3cube

Handwritten Text Recognition for manuscripts and early printed texts

Maria Levchenko

GenCyber Cyber Security Day Presentation

Michael W. Hawkins

The presentation explores the development and application of artificial intelligence (AI) from its inception to its current status in the modern world. The term "artificial intelligence" was first coined by John McCarthy in 1956 to describe efforts to develop computer programs capable of performing tasks that typically require human intelligence. This concept was first introduced at a conference held at Dartmouth College, where programs demonstrated capabilities such as playing chess, proving theorems, and interpreting texts. In the early stages, Alan Turing contributed to the field by defining intelligence as the ability of a being to respond to certain questions intelligently, proposing what is now known as the Turing Test to evaluate the presence of intelligent behavior in machines. As the decades progressed, AI evolved significantly. The 1980s focused on machine learning, teaching computers to learn from data, leading to the development of models that could improve their performance based on their experiences. The 1990s and 2000s saw further advances in algorithms and computational power, which allowed for more sophisticated data analysis techniques, including data mining. By the 2010s, the proliferation of big data and the refinement of deep learning techniques enabled AI to become mainstream. Notable milestones included the success of Google's AlphaGo and advancements in autonomous vehicles by companies like Tesla and Waymo. A major theme of the presentation is the application of generative AI, which has been used for tasks such as natural language text generation, translation, and question answering. Generative AI uses large datasets to train models that can then produce new, coherent pieces of text or other media. The presentation also discusses the ethical implications and the need for regulation in AI, highlighting issues such as privacy, bias, and the potential for misuse. These concerns have prompted calls for comprehensive regulations to ensure the safe and equitable use of AI technologies. Artificial intelligence has also played a significant role in healthcare, particularly highlighted during the COVID-19 pandemic, where it was used in drug discovery, vaccine development, and analyzing the spread of the virus. The capabilities of AI in healthcare are vast, ranging from medical diagnostics to personalized medicine, demonstrating the technology's potential to revolutionize fields beyond just technical or consumer applications. In conclusion, AI continues to be a rapidly evolving field with significant implications for various aspects of society. The development from theoretical concepts to real-world applications illustrates both the potential benefits and the challenges that come with integrating advanced technologies into everyday life. The ongoing discussion about AI ethics and regulation underscores the importance of managing these technologies responsibly to maximize their their benefits while minimizing potential harms.

Artificial Intelligence: Facts and Myths

Joaquim Jorge

What are drone anti-jamming systems? The drone anti-jamming systems and anti-spoof technology protect against interference, jamming, and spoofing of the UAVs. To protect their security, countries are beginning to research drone anti-jamming systems, also known as drone strike weapons. The anti-jam and anti-spoof technology protects against interference, jamming and spoofing. A drone strike weapon is a drone attack weapon that can attack and destroy enemy drones. So what is so unique about this amazing system?

What Are The Drone Anti-jamming Systems Technology?

Antenna Manufacturer Coco

Recently uploaded (20)

Powerful Google developer tools for immediate impact! (2023-24 C)

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

Understanding Discord NSFW Servers A Guide for Responsible Users.pdf

Strategies for Landing an Oracle DBA Job as a Fresher

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe

Partners Life - Insurer Innovation Award 2024

Finology Group – Insurtech Innovation Award 2024

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

presentation ICT roal in 21st century education

Automating Google Workspace (GWS) & more with Apps Script

Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...

Boost PC performance: How more available memory can improve productivity

Developing An App To Navigate The Roads of Brazil

Handwritten Text Recognition for manuscripts and early printed texts

GenCyber Cyber Security Day Presentation

Artificial Intelligence: Facts and Myths

What Are The Drone Anti-jamming Systems Technology?

Scaling a backend for a big data and blockchain environment by Rafael Ríos at Big Data Spain 2017

2. Scaling a backend for big data and blockchain environment

3. 1. Introduction (The companies, project and me) 2. Backend challenge 3. Big Data Integration 4. Blockchain Integration ·············· P. 3 ··················································· ················ P. 7 ··················································· ·············· P. 17 ··················································· ········ P. 21

4. @ganchix Head of Blockchain / Backend Me ganchix

5. The companies

7. Fecha The project • Financial platform • Marketplace • Tokenization • Operate with cryptocurrency • Liquidity predictive models • Credit scoring • Product recommendation

8. Backend challenge - The evolution Dapp Smart Contract Ethereum

9. Backend challenge - The evolution

10. Backend challenge - The evolution Dapps problem: • Noncryptocurrency users • Problem with some integrations • Legal problem

11. Backend challenge - The evolution

12. Backend challenge - The evolution

13. Fecha Backend challenge - Why microservices? • Migration of Dapp easier • Easy to scale • Polyglot Database and Languages

14. Why don't use exclusively blockchain with a database?

15. 1. Spring Cloud Netflix and Kubernetes • Easy to learn. • Nice integrations • Spring 5 reactive 2. Docker • Most adopted vendor technology for containers • Well supported 3. Kubernetes • Multi-cloud provider and on-premises data centers • Self-repair and health check capabilities • Auto-scale Backend challenge - Microservice Architecture Stack

16. Backend challenge - Microservice Architecture Stack

17. Backend challenge - Deployment

18. PFM values generation from user data. Apache Spark + Cassandra Forecast prediction and regeneration of this models Apache Spark + Cassandra Product recommendations based on the economic profile of the user and his real needs. Apache Spark + Cassandra + Neo4j Credit scoring calculation Apache Spark + Cassandra Big Data Integration - Tasks

19. • Tasks are hard, needs: • Time • Resources • Not Real Time is needed. • Event Driven Architecture. Big Data Integration - Events

20. Big Data Integration - How?

21. Big Data Integration - RabbitMQ vs Kafka KafkaRabbitMq • RabbitMQ is designed as a general purpose message broker • Support existing protocols like AMQP, STOMP, MQTT. • Finer-grained consistency control/guarantees on a peer- message. • Complex routing. • Apache Kafka is designed for high volume publish-subscribe messages and streams, meant to be durable, fast, and scalable. • Event Sourcing • Your application needs access to stream history. • No complex routing. https://content.pivotal.io/blog/understanding-when-to-use-rabbitmq-or-apache-kafka

22. • Deployed in Kubernetes. • Only accessible by NodeJS API. • All keys are stored in secrets vaults. • Used for: • Tokenization • Transactions of users Blockchain Integration - Private Ethereum

23. Blockchain Integration - Private Ethereum

24. Blockchain Integration - Ethereum Main Net

25. Blockchain Integration - Ethereum Main Net

26. Blockchain Integration - Ethereum Main Net • We are the owner of the wallets • We use Infura to connect blockchain • Used for: • Payment • Transfers

27. @ganchix @2getherbank @bigeeksoftwa re Thanks! ICO is coming!

Scaling a backend for a big data and blockchain environment by Rafael Ríos at Big Data Spain 2017

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Scaling a backend for a big data and blockchain environment by Rafael Ríos at Big Data Spain 2017

Similar to Scaling a backend for a big data and blockchain environment by Rafael Ríos at Big Data Spain 2017 (20)

More from Big Data Spain

More from Big Data Spain (20)

Recently uploaded

Recently uploaded (20)

Scaling a backend for a big data and blockchain environment by Rafael Ríos at Big Data Spain 2017