Usually, companies have a clear separation between testing environments and development environments. But what if we take a different approach and combine everything into one environment and still survive? The speaker will discuss ways to organize cloud environments for testing and development, compare them, and discuss the pros and cons of the combined approach.
A deck from the first CDIsrael meetup, presenting our CD flow at Snyk, focusing on our testing framework. A day in a life of a developer - code, test, publish, deploy, monitor.
The document discusses MockServer-driven testing (MSDT), a methodology for developing automated UI tests using a mock server. The main ideas are to create a mock server that replaces the real backend and cover features supported by the mock server with UI tests. Key benefits include faster, independent tests and the ability to test more scenarios. Potential downsides include costs of maintaining the mock server and possibility of bugs. Examples of use cases and implementing a proof of concept mock server for chat messaging are provided.
Join this info-packed and hands-on workshop where we will cover:
Introduction to Kubernetes & GitOps talk:
We'll cover the most popular path that has brought success to many users already - GitOps as a natural evolution of Kubernetes. We'll give an overview of how you can benefit from Kubernetes and GitOps: greater security, reliability, velocity and more. Importantly, we cover definitions and principles standardized by the CNCF's OpenGitOps group and what it means for you.
Get Started with GitOps:
You'll have GitOps up and running in about 30 mins using our free and open source tools! We'll give a brief vision of where you want to be with those security, reliability, and velocity benefits, and then we'll support you while go through the getting started steps. During the workshop, you'll also experience in action and see demos for:
* an opinionated repo structure to minimize decision fatigue
* disaster recovery using GitOps
* Helm charts example
* Multi-cluster example
* all with free and open source tools mostly in the CNCF (eg. Flux and Helm).
If you have questions before or after the workshop, talk to us at #weave-gitops http://bit.ly/WeaveGitOpsSlack (If you need to invite yourself to the Slack, visit https://slack.weave.works/)
Vskills certification for The Grinder Testing Professional assesses the candidate as per the company’s need for load testing web applications. The certification tests the candidates on various areas in agents, workers, properties file, logging, console, TCPProxy, scripts, Jython, Clojure, instrumentation, script gallery, plug-ins, statistics, SSL and garbage collection.
Pluggable Infrastructure with CI/CD and DockerBob Killen
The docker cluster ecosystem is still young, and highly modular. This presentation covers some of the challenges we faced deciding on what infrastructure to deploy, and a few tips and tricks in making both applications and infrastructure easily adaptable.
The journey to Native Cloud Architecture & Microservices, tracing the footste...Mek Srunyu Stittri
The document discusses Netflix's adoption of microservices and continuous delivery to improve speed and agility. Key points include:
1) Netflix moved to microservices and continuous delivery on the cloud to dramatically speed up product development and deployment.
2) This allowed independent teams to deploy code frequently without coordination, with automated testing and deployment replacing handoffs and long release cycles.
3) Netflix's approach involved building stateless, independently deployable microservices; continuous monitoring; and other techniques to enable developers to deploy code safely and rapidly.
This document discusses automating development operations and introduces seven capabilities for effective DevOps: isolation, portability, automated configuration, simplified code reviews, continuous integration and testing, atomic deployments, and automated monitoring. For each capability, it outlines challenges and lists common automation options and tools to address those challenges, noting pros and cons of different approaches. It emphasizes that automation should adapt to existing workflows, scale gradually, and not reduce developer productivity.
Orchestrating Automated Tests in Different Systems - Please Download to watch...Kelvin Silva
This talk was presented in DevOpsDays Sao Paulo (June/2018) and in Daitan DevWeek (July/2018).
What should you do if your product must be tested on multiple operating systems before delivering? And what if it must be tested on different browsers? Now it is a lot of combinations. And how to do this in an agile way?
Here is a talk about how we can orchestrate automated tests on different systems, on different browsers. Assuring the coverage on all operating systems and browsers supported by a product is one of the toughest tasks commonly faced when transitioning to a DevOps setup.
Due to the high quantity of environment combinations, the complexity of organizing and executing tests increases and can lead to an ineffective pipeline. In this talk, we will evaluate an automated orchestration of complex test suites that delivers the effectiveness needed for proper DevOps.
A deck from the first CDIsrael meetup, presenting our CD flow at Snyk, focusing on our testing framework. A day in a life of a developer - code, test, publish, deploy, monitor.
The document discusses MockServer-driven testing (MSDT), a methodology for developing automated UI tests using a mock server. The main ideas are to create a mock server that replaces the real backend and cover features supported by the mock server with UI tests. Key benefits include faster, independent tests and the ability to test more scenarios. Potential downsides include costs of maintaining the mock server and possibility of bugs. Examples of use cases and implementing a proof of concept mock server for chat messaging are provided.
Join this info-packed and hands-on workshop where we will cover:
Introduction to Kubernetes & GitOps talk:
We'll cover the most popular path that has brought success to many users already - GitOps as a natural evolution of Kubernetes. We'll give an overview of how you can benefit from Kubernetes and GitOps: greater security, reliability, velocity and more. Importantly, we cover definitions and principles standardized by the CNCF's OpenGitOps group and what it means for you.
Get Started with GitOps:
You'll have GitOps up and running in about 30 mins using our free and open source tools! We'll give a brief vision of where you want to be with those security, reliability, and velocity benefits, and then we'll support you while go through the getting started steps. During the workshop, you'll also experience in action and see demos for:
* an opinionated repo structure to minimize decision fatigue
* disaster recovery using GitOps
* Helm charts example
* Multi-cluster example
* all with free and open source tools mostly in the CNCF (eg. Flux and Helm).
If you have questions before or after the workshop, talk to us at #weave-gitops http://bit.ly/WeaveGitOpsSlack (If you need to invite yourself to the Slack, visit https://slack.weave.works/)
Vskills certification for The Grinder Testing Professional assesses the candidate as per the company’s need for load testing web applications. The certification tests the candidates on various areas in agents, workers, properties file, logging, console, TCPProxy, scripts, Jython, Clojure, instrumentation, script gallery, plug-ins, statistics, SSL and garbage collection.
Pluggable Infrastructure with CI/CD and DockerBob Killen
The docker cluster ecosystem is still young, and highly modular. This presentation covers some of the challenges we faced deciding on what infrastructure to deploy, and a few tips and tricks in making both applications and infrastructure easily adaptable.
The journey to Native Cloud Architecture & Microservices, tracing the footste...Mek Srunyu Stittri
The document discusses Netflix's adoption of microservices and continuous delivery to improve speed and agility. Key points include:
1) Netflix moved to microservices and continuous delivery on the cloud to dramatically speed up product development and deployment.
2) This allowed independent teams to deploy code frequently without coordination, with automated testing and deployment replacing handoffs and long release cycles.
3) Netflix's approach involved building stateless, independently deployable microservices; continuous monitoring; and other techniques to enable developers to deploy code safely and rapidly.
This document discusses automating development operations and introduces seven capabilities for effective DevOps: isolation, portability, automated configuration, simplified code reviews, continuous integration and testing, atomic deployments, and automated monitoring. For each capability, it outlines challenges and lists common automation options and tools to address those challenges, noting pros and cons of different approaches. It emphasizes that automation should adapt to existing workflows, scale gradually, and not reduce developer productivity.
Orchestrating Automated Tests in Different Systems - Please Download to watch...Kelvin Silva
This talk was presented in DevOpsDays Sao Paulo (June/2018) and in Daitan DevWeek (July/2018).
What should you do if your product must be tested on multiple operating systems before delivering? And what if it must be tested on different browsers? Now it is a lot of combinations. And how to do this in an agile way?
Here is a talk about how we can orchestrate automated tests on different systems, on different browsers. Assuring the coverage on all operating systems and browsers supported by a product is one of the toughest tasks commonly faced when transitioning to a DevOps setup.
Due to the high quantity of environment combinations, the complexity of organizing and executing tests increases and can lead to an ineffective pipeline. In this talk, we will evaluate an automated orchestration of complex test suites that delivers the effectiveness needed for proper DevOps.
The document discusses options for testing software including manual testing with Vagrant and Robot Framework as well as automated testing. It recommends using OpenShift for automated testing as it provides a reliable solution with Kubernetes that is scalable and controllable while Docker provides speed and high density. Open vSwitch enables network connectivity in OpenShift. Workflow and reference details are also included.
Intro to GitOps with Weave GitOps, Flagger and LinkerdWeaveworks
This document provides an overview of GitOps, service meshes, Linkerd, Flux, Weave GitOps and progressive delivery. It introduces the speakers and outlines the agenda which includes explanations of GitOps, service meshes, Linkerd and Weave GitOps. It then demonstrates how Weave GitOps and Linkerd can be used together for progressive delivery and provides a Q&A section at the end.
Free GitOps Workshop (with Intro to Kubernetes & GitOps)Weaveworks
View this video on Youtube here: https://youtu.be/tK4S8y3j5TA
In this info-packed and hands-on workshop we covered:
Introduction to Kubernetes & GitOps talk:
We covered the most popular path that has brought success to many users already - GitOps as a natural evolution of Kubernetes. We'll give an overview of how you can benefit from Kubernetes and GitOps: greater security, reliability, velocity and more. Importantly, we cover definitions and principles standardized by the CNCF's OpenGitOps group and what it means for you.
Get Started with GitOps:
You'll have GitOps up and running in about 30 mins using our free and open source tools! We'll give a brief vision of where you want to be with those security, reliability, and velocity benefits, and then we'll support you while go through the getting started steps. During the workshop, you'll also experience in action and see demos for:
- an opinionated repo structure to minimize decision fatigue
- disaster recovery using GitOps
- Helm charts example
- Multi-cluster example
- all with free and open source tools mostly in the CNCF (eg. Flux and Helm).
If you have questions before or after the workshop, talk to us at #weave-gitops http://bit.ly/WeaveGitOpsSlack (If you need to invite yourself to the Slack, visit https://slack.weave.works/)
The DevOps methodology integrates development and operations so that system changes can get rolled out quickly without causing unplanned downtime. Industrial organizations that successfully implement DevOps will have a strong advantage, but knowing how to get started can be a real challenge.
The presentation which I was using during my talk at EPAM Lviv JS community about offline-first applications. Contains high-level review of tools and web platform to submerge folks in a world of offline-first thinking.
apidays LIVE New York - Navigating the Sea of Javascript Tools to Discover Sc...apidays
apidays LIVE New York - API for Legacy Industries: Banking, Insurance, Healthcare and Retail
Navigating the Sea of Javascript Tools to Discover Scalable Tools for Continuous Delivery
Menelaos Kotsollaris, Senior Software Engineer
Viki Green, Senior Software Developer at Trulioo
1. The document discusses building a small data center network for Dyn, focusing on lessons learned from two design iterations.
2. The first design used MPLS VPNs but had issues with routing and IPv6 support. The second design used virtual routing and forwarding with multiple routing tables to separate traffic and improve service mobility.
3. Key lessons included validating designs before deploying, automating network operations, and moving security policies to instances rather than the network to improve agility and isolate impacts.
Montreal Kubernetes Meetup: Developer-first workflows (for microservices) on ...Ambassador Labs
1. The document discusses developer-first workflows for building and operating microservices on Kubernetes.
2. It recommends creating self-sufficient, autonomous teams and using Kubernetes, Docker, and Envoy to provide the basic infrastructure primitives needed for distributed workflows.
3. The strategies suggested depend on the service maturity level and include using similar development and production environments for prototyping, implementing software redundancy for production services, and defining service level objectives and network observability for internal dependencies.
Dive deep into specific OSS packages to examine the top issues in the enterprise with two of our most qualified OSS architects, Bill Crowell and Vince Cox walkthrough: Their day-to-day work in OSS packages; ways to fix reported issues; why you can’t expect in-house developers to handle issues in OSS packages.
Things You MUST Know Before Deploying OpenStack: Bruno Lago, Catalyst ITOpenStack
Audience: Advanced
About: Real world lessons and war stories about Catalyst IT’s experience in rolling out an OpenStack based public cloud in New Zealand.
This presentation will provide tips and advice that may save you a lot of time, money and nights of sleep if you are planning to run OpenStack in the future. It may also bring some insights to people that are already running OpenStack in production.
Topics covered will include: selection of hardware for optimal costs, techniques that drive quality and service levels up, common deployment mistakes, in place upgrades, how to identify the maturity level of each project and decide what is ready for production, and much more!
Speaker Bio: Bruno Lago – Entrepreneur, Catalyst IT Limited
Bruno Lago is a solutions architect that has been involved with the Catalyst Cloud (New Zealand’s first public cloud based on OpenStack) from its inception. He is passionate about open source software, cloud computing and disruptive technologies.
OpenStack Australia Day - Sydney 2016
https://events.aptira.com/openstack-australia-day-sydney-2016/
Antifragility and testing for distributed systems failureDiUS
Failure is inevitable. In our modern world filled with continuously delivered and increasingly complex distributed architectures (looking at you micro-services), it is important to be able to test and improve our systems under a range of failure conditions.
In this talk, Matt discusses these complexities and the forces they exert on development teams, presenting some simple strategies and practical advice to deal with them.
Do you have a healthy CI/CD pipeline? Do releases simply flow through? CI, CD, PRs, Pipelines, Releases, Deployments and all that jazz.
Whether you're new to Continuous Delivery or a hardened traveller down that road, this session has something for you. We’ll start with an exploration of branching strategy (releaseflow.org) before walking through a healthy continuous delivery configuration.
We’ll watch a code change make it's way through a pipeline to production and discuss how we can apply such practices to our everyday work.
Weave GitOps 2022.09 Release: A Fast & Reliable Path to Production with Progr...Weaveworks
Weave GitOps 2022.09 Features Launch Event
The latest release of Weave GitOps introduces new features enabling progressive delivery, policy as code, and accelerated application onboarding.
Weave GitOps is the leading full-stack GitOps platform to automate trusted application delivery and secure infrastructure operations on premise, in the cloud and at the edge. Trusted by Customers, including Deutsche Telekom and The Department of Defense, Platform and Application Teams, Weave GitOps unlocks the benefits of increased efficiency and compliance, while boosting deployment velocity and confidence.
Join us where we’ll do a live demo of Weave GitOps showcasing:
- Advanced Deployment Patterns—Progressive Delivery has never been easier
- Multi-tenancy and Application Portability—More collaboration and control
- Strengthened GitOps Security—If you can code it, you can secure it.
The monolith to cloud-native, microservices evolution has driven a shift from monitoring to observability. OpenTelemetry, a merger of the OpenTracing and OpenCensus projects, is enabling Observability 2.0. This talk gives an overview of the OpenTelemetry project and then outlines some production-proven architectures for improving the observability of your applications and systems.
Learnings from the Field. Lessons from Working with Dozens of Small & Large D...HostedbyConfluent
- Upgrades should be done often to get bug fixes and improvements, following the upgrade guide carefully. Start with a healthy cluster and upgrade components outward from Zookeeper to Kafka brokers to clients. Don't rush the process or have any unresolved partition reassignments.
- Collect JMX metrics to monitor the cluster as outages can be prolonged without visibility. The Kafka defaults are suitable for single node deployments but replication factor, threads, and broker configuration should be tuned for larger clusters.
- Quotas like replication throttling and bandwidth/request limits per client or topic should be used to protect the cluster and clients. Log files should separate each component and be retained for a few days. Consider multiple clusters by SLA
This document summarizes Netflix's use of Kafka in their data pipeline. It discusses how Netflix evolved from using S3 and EMR to introducing Kafka and Kafka producers and consumers to handle 400 billion events per day. It covers challenges of scaling Kafka clusters and tuning Kafka clients and brokers. Finally, it outlines Netflix's roadmap which includes contributing to open source projects like Kafka and testing failure resilience.
This document summarizes Netflix's use of Kafka in their data pipeline. It discusses the evolution of Netflix's data pipeline to incorporate Kafka to handle 400 billion events per day. It describes how Netflix uses Kafka clusters with different priorities and configurations. It also outlines some of the challenges of using Kafka at Netflix's scale, such as Zookeeper client issues and cluster scaling, and the solutions Netflix developed to address these challenges.
There is a problem of finding the best instance of a service in distributed systems with dynamic configuration. Nowadays, there are many products for the configuration storage and service discovery. It should be mentioned at least Netflix Eureka, Consul, etc or good old Zookeeper. These products can keep and give configuration, manage service instances lifecycle and some of them even can be as dynamic DNS service. But main question is not about what instance may be called at the certain time. It is about what instance is better for call? This means that smart load balancing top on service discovery is required. Spring Cloud project allows to integrate these products to your project and provides powerful solutions for typical problems, that make cloud native services developing easier. This talk will review the internal structure of SpringCloud implementation of Client-Side Service Discovery and Client Load Balancing patterns. It also will include specific details of concrete implementations with examples from official libraries and the author’s own library.
Процесс разработки не начинается и не заканчивается на написании кода программного продукта. Мы пишем документацию, придумываем, как это всё оттестировать, и заботимся о том, чтобы доступность приложения была на высоком уровне.
Мы все делаем привычные вещи привычным для нас способом. Порой выполняя много ручной и неэффективной работы. Но что, если есть другой, радикальный подход. Можно ли формализовать свою деятельность и переложить её в код? Какие практики и инструменты для этого использовать?
В докладе будет представлен личный опыт автора по автоматизации различных элементов разработки ПО.
Weitere ähnliche Inhalte
Ähnlich wie Cocktail of Environments. How to Mix Test and Development Environments and Stay Alive
The document discusses options for testing software including manual testing with Vagrant and Robot Framework as well as automated testing. It recommends using OpenShift for automated testing as it provides a reliable solution with Kubernetes that is scalable and controllable while Docker provides speed and high density. Open vSwitch enables network connectivity in OpenShift. Workflow and reference details are also included.
Intro to GitOps with Weave GitOps, Flagger and LinkerdWeaveworks
This document provides an overview of GitOps, service meshes, Linkerd, Flux, Weave GitOps and progressive delivery. It introduces the speakers and outlines the agenda which includes explanations of GitOps, service meshes, Linkerd and Weave GitOps. It then demonstrates how Weave GitOps and Linkerd can be used together for progressive delivery and provides a Q&A section at the end.
Free GitOps Workshop (with Intro to Kubernetes & GitOps)Weaveworks
View this video on Youtube here: https://youtu.be/tK4S8y3j5TA
In this info-packed and hands-on workshop we covered:
Introduction to Kubernetes & GitOps talk:
We covered the most popular path that has brought success to many users already - GitOps as a natural evolution of Kubernetes. We'll give an overview of how you can benefit from Kubernetes and GitOps: greater security, reliability, velocity and more. Importantly, we cover definitions and principles standardized by the CNCF's OpenGitOps group and what it means for you.
Get Started with GitOps:
You'll have GitOps up and running in about 30 mins using our free and open source tools! We'll give a brief vision of where you want to be with those security, reliability, and velocity benefits, and then we'll support you while go through the getting started steps. During the workshop, you'll also experience in action and see demos for:
- an opinionated repo structure to minimize decision fatigue
- disaster recovery using GitOps
- Helm charts example
- Multi-cluster example
- all with free and open source tools mostly in the CNCF (eg. Flux and Helm).
If you have questions before or after the workshop, talk to us at #weave-gitops http://bit.ly/WeaveGitOpsSlack (If you need to invite yourself to the Slack, visit https://slack.weave.works/)
The DevOps methodology integrates development and operations so that system changes can get rolled out quickly without causing unplanned downtime. Industrial organizations that successfully implement DevOps will have a strong advantage, but knowing how to get started can be a real challenge.
The presentation which I was using during my talk at EPAM Lviv JS community about offline-first applications. Contains high-level review of tools and web platform to submerge folks in a world of offline-first thinking.
apidays LIVE New York - Navigating the Sea of Javascript Tools to Discover Sc...apidays
apidays LIVE New York - API for Legacy Industries: Banking, Insurance, Healthcare and Retail
Navigating the Sea of Javascript Tools to Discover Scalable Tools for Continuous Delivery
Menelaos Kotsollaris, Senior Software Engineer
Viki Green, Senior Software Developer at Trulioo
1. The document discusses building a small data center network for Dyn, focusing on lessons learned from two design iterations.
2. The first design used MPLS VPNs but had issues with routing and IPv6 support. The second design used virtual routing and forwarding with multiple routing tables to separate traffic and improve service mobility.
3. Key lessons included validating designs before deploying, automating network operations, and moving security policies to instances rather than the network to improve agility and isolate impacts.
Montreal Kubernetes Meetup: Developer-first workflows (for microservices) on ...Ambassador Labs
1. The document discusses developer-first workflows for building and operating microservices on Kubernetes.
2. It recommends creating self-sufficient, autonomous teams and using Kubernetes, Docker, and Envoy to provide the basic infrastructure primitives needed for distributed workflows.
3. The strategies suggested depend on the service maturity level and include using similar development and production environments for prototyping, implementing software redundancy for production services, and defining service level objectives and network observability for internal dependencies.
Dive deep into specific OSS packages to examine the top issues in the enterprise with two of our most qualified OSS architects, Bill Crowell and Vince Cox walkthrough: Their day-to-day work in OSS packages; ways to fix reported issues; why you can’t expect in-house developers to handle issues in OSS packages.
Things You MUST Know Before Deploying OpenStack: Bruno Lago, Catalyst ITOpenStack
Audience: Advanced
About: Real world lessons and war stories about Catalyst IT’s experience in rolling out an OpenStack based public cloud in New Zealand.
This presentation will provide tips and advice that may save you a lot of time, money and nights of sleep if you are planning to run OpenStack in the future. It may also bring some insights to people that are already running OpenStack in production.
Topics covered will include: selection of hardware for optimal costs, techniques that drive quality and service levels up, common deployment mistakes, in place upgrades, how to identify the maturity level of each project and decide what is ready for production, and much more!
Speaker Bio: Bruno Lago – Entrepreneur, Catalyst IT Limited
Bruno Lago is a solutions architect that has been involved with the Catalyst Cloud (New Zealand’s first public cloud based on OpenStack) from its inception. He is passionate about open source software, cloud computing and disruptive technologies.
OpenStack Australia Day - Sydney 2016
https://events.aptira.com/openstack-australia-day-sydney-2016/
Antifragility and testing for distributed systems failureDiUS
Failure is inevitable. In our modern world filled with continuously delivered and increasingly complex distributed architectures (looking at you micro-services), it is important to be able to test and improve our systems under a range of failure conditions.
In this talk, Matt discusses these complexities and the forces they exert on development teams, presenting some simple strategies and practical advice to deal with them.
Do you have a healthy CI/CD pipeline? Do releases simply flow through? CI, CD, PRs, Pipelines, Releases, Deployments and all that jazz.
Whether you're new to Continuous Delivery or a hardened traveller down that road, this session has something for you. We’ll start with an exploration of branching strategy (releaseflow.org) before walking through a healthy continuous delivery configuration.
We’ll watch a code change make it's way through a pipeline to production and discuss how we can apply such practices to our everyday work.
Weave GitOps 2022.09 Release: A Fast & Reliable Path to Production with Progr...Weaveworks
Weave GitOps 2022.09 Features Launch Event
The latest release of Weave GitOps introduces new features enabling progressive delivery, policy as code, and accelerated application onboarding.
Weave GitOps is the leading full-stack GitOps platform to automate trusted application delivery and secure infrastructure operations on premise, in the cloud and at the edge. Trusted by Customers, including Deutsche Telekom and The Department of Defense, Platform and Application Teams, Weave GitOps unlocks the benefits of increased efficiency and compliance, while boosting deployment velocity and confidence.
Join us where we’ll do a live demo of Weave GitOps showcasing:
- Advanced Deployment Patterns—Progressive Delivery has never been easier
- Multi-tenancy and Application Portability—More collaboration and control
- Strengthened GitOps Security—If you can code it, you can secure it.
The monolith to cloud-native, microservices evolution has driven a shift from monitoring to observability. OpenTelemetry, a merger of the OpenTracing and OpenCensus projects, is enabling Observability 2.0. This talk gives an overview of the OpenTelemetry project and then outlines some production-proven architectures for improving the observability of your applications and systems.
Learnings from the Field. Lessons from Working with Dozens of Small & Large D...HostedbyConfluent
- Upgrades should be done often to get bug fixes and improvements, following the upgrade guide carefully. Start with a healthy cluster and upgrade components outward from Zookeeper to Kafka brokers to clients. Don't rush the process or have any unresolved partition reassignments.
- Collect JMX metrics to monitor the cluster as outages can be prolonged without visibility. The Kafka defaults are suitable for single node deployments but replication factor, threads, and broker configuration should be tuned for larger clusters.
- Quotas like replication throttling and bandwidth/request limits per client or topic should be used to protect the cluster and clients. Log files should separate each component and be retained for a few days. Consider multiple clusters by SLA
This document summarizes Netflix's use of Kafka in their data pipeline. It discusses how Netflix evolved from using S3 and EMR to introducing Kafka and Kafka producers and consumers to handle 400 billion events per day. It covers challenges of scaling Kafka clusters and tuning Kafka clients and brokers. Finally, it outlines Netflix's roadmap which includes contributing to open source projects like Kafka and testing failure resilience.
This document summarizes Netflix's use of Kafka in their data pipeline. It discusses the evolution of Netflix's data pipeline to incorporate Kafka to handle 400 billion events per day. It describes how Netflix uses Kafka clusters with different priorities and configurations. It also outlines some of the challenges of using Kafka at Netflix's scale, such as Zookeeper client issues and cluster scaling, and the solutions Netflix developed to address these challenges.
Ähnlich wie Cocktail of Environments. How to Mix Test and Development Environments and Stay Alive (20)
There is a problem of finding the best instance of a service in distributed systems with dynamic configuration. Nowadays, there are many products for the configuration storage and service discovery. It should be mentioned at least Netflix Eureka, Consul, etc or good old Zookeeper. These products can keep and give configuration, manage service instances lifecycle and some of them even can be as dynamic DNS service. But main question is not about what instance may be called at the certain time. It is about what instance is better for call? This means that smart load balancing top on service discovery is required. Spring Cloud project allows to integrate these products to your project and provides powerful solutions for typical problems, that make cloud native services developing easier. This talk will review the internal structure of SpringCloud implementation of Client-Side Service Discovery and Client Load Balancing patterns. It also will include specific details of concrete implementations with examples from official libraries and the author’s own library.
Процесс разработки не начинается и не заканчивается на написании кода программного продукта. Мы пишем документацию, придумываем, как это всё оттестировать, и заботимся о том, чтобы доступность приложения была на высоком уровне.
Мы все делаем привычные вещи привычным для нас способом. Порой выполняя много ручной и неэффективной работы. Но что, если есть другой, радикальный подход. Можно ли формализовать свою деятельность и переложить её в код? Какие практики и инструменты для этого использовать?
В докладе будет представлен личный опыт автора по автоматизации различных элементов разработки ПО.
There is a problem of finding the correct operating services In a distributed systems with dynamic configuration. Currently, there are designed one and more solutions for the ever-changing storage configuration. It should be mentioned at least Netflix Eureka, Consul, etcd or good old Zookeeper. Spring Cloud project allows to integrate some of these solutions to your project and provides powerful solutions for typical problems. However, on the way to unicorns not the most obvious subtleties of implementation and associated problems of use in real projects wait for a developer.
This talk will review the internal structure SpringCloud, implementation of Client-Side Service Discovery pattern, and specific details of concrete implementations on the example of the official libraries and the author's own library.
Сейчас только ленивый не говорит про DevOps, краеугольным камнем которого является организация потока непрерывной доставки ценности клиенту. Continuous Delivery перестаёт быть опцией и становится обязательным требованием.
В докладе будут рассмотрены:
- общие подходы к организации Continuous Delivery на базе Jenkins-а в совсем не тепличных условиях
- практики и подходы, которые позволяют быстро настраивать и собирать десятки микросервисов
- подводные камни, с которыми пришлось столкнуться, и способы борьбы с ними
Everyone has heard about microservices. Someone tries to implement them in practice. And only the bravests of us have already used them in production environment. But then why so many people hype around microservices, if the idea is not quite new? What are microservices? What is the difference between it and good old SOA approach? How can developers create modern enterprise applications in Java easily with the help of this approach?
Our guests should try to answer these questions and share their own experience related to live coding of imaginary startup ‘Hippos’.
The following topics will be covered:
What are microservices? Where is the theory, bro?
What kind of technologies should we choose? What have we chosen and why?
Why RPC is still competetive in REST-domination era?
How to pack and distribute microservices? How can SpringBoot and Docker help us to solve our problems?
Why is service discovery considered as one of the crucial components? How to cook Spring Cloud and what problems you may face in real life?
Security of microservices, API gateway and other.
Speakers will also cover other topics related to distributed system development.
Everyone has heard about microservices. Someone tries to implement them in practice. And only the bravests of us have already used them in production environment. But then why so many people hype around microservices, if the idea is not quite new? What are microservices? What is the difference between it and good old SOA approach? How can developers create modern enterprise applications in Java easily with the help of this approach?
Our guests should try to answer these questions and share their own experience related to live coding of imaginary startup 'Hippos'. The following topics will be covered:
What are microservices? Where is the theory, bro?
What kind of technologies should we choose? What have we chosen and why?
Why RPC is still competetive in REST-domination era?
How to pack and distribute microservices? How can SpringBoot and Docker help us to solve our problems?
Why is service discovery considered as one of the crucial components? How to cook Spring Cloud and what problems you may face in real life?
Security of microservices, API gateway and other.
Speakers will also cover other topics related to distributed system development.
Сколько раз мы собираем ПО каждый день? Сколько раз за день код компилируется, тестируется и превращается в артефакты? Какие разнообразные сценарии возможны в казалось бы простой цепочке compile-test-assemble? Билд-системы из узкоспециализованных утилит превратились в мощный инструмент сборки, валидации, дистрибуции и автоматизации наших программных продуктов.
Это будет тематический доклад о том как настроить инструмент сборки для себя, а не против себя. Мы рассмотрим основные принципы работы Gradle DSL и возможности его расширения. Как и на чём их писать, как тестировать и как делиться результатами своего труда со своими коллегами.
Вместе с вами мы напишем автоматизированный сценарий сборки для проекта и покажем как ловко можно вставлять костыли всю силу и мощь Gradle 2.x.
Любите ли вы велосипеды? Все разработчики любят свои ненаколеночныерешения велосипеды! И мы не исключение. В нашем докладе мы покажем как собирать, сколачивать, вылепливать собственный велосипед так, чтобы на нем потом могла ездить без слёз вся команда, компания, или может весь мир.
Что в докладе будет:
- много Spring Boot-а;
- live coding;
- создание собственного Spring Boot Starter-а;
- Apache Thrift в качестве подопытного кролика.
Чего не будет:
- бенчмарков и сравнений Thrift vs REST vs gRPC vs XXX.
Микросервисы, кто-то только слышал о них, кто-то пытался делать, кто-то уже использует в продакшене. Идеи, заложенные в концепцию микросервисов, не новы и основные постулаты уже звучали раньше. Так почему же в последнее время мы всё чаще слышим о микросервисах? Что такое микросервисы для нас и чем они отличаются от старого доброго подхода SOA? Как теперь разрабатывать enterprise-приложения с микросервисным подходом на нашем любимом языке программирования Java?
На эти и некоторые другие вопросы постараемся ответить во время встречи. Наши гости, Кирилл Толкачёв и Александр Тарасов, в режиме live coding попытаются создать небольшой стартап, попутно использовав новомодные подходы и инструменты.
На пути к релизу стартапа будут затронуты основные проблемы выбранных подходов в целом и технологий в частности:
Микросервис — что это, для чего и как с этим дальше жить. Где теория брат? ;)
На чём писать API: REST или RPC, и почему Thrift имеет право на жизнь в эпоху тотального распространения JSON-а. Упрощай и превозмогай с помощью Spring boot starter;
Какой стек выбрать для разработки, что выбрали мы и почему. Небольшое сравнение легковесных и не очень java фреймворков а так же сопутствующих инструментов;
Способы упаковки, дистрибуции и разворачивания микросервисов, как Spring Boot и Docker помогают нам в решении этих непростых для разработчика проблемах;
Как микросервисам найти друг друга, как готовить Spring Cloud и как обойти существующие проблемы и ограничения. Не доверяйте технологиям, доверяйте только себе;
API Gateway. Предохраняй и сохраняй свои микросервисы.
Так же речь пойдет о других распространенных проблемах распределенных систем и их решениях.
Docker on a local machine and Docker in production — are two big differences. It's easy to play with technology but it's hard to do something real for many customers.
Half a year ago inside of AlfaLab (division of Alfa-Bank) we've started building architecture with microservices approach for few our pilot projects. We've almost completely changed a stack of the used technologies on a frontend and significantly changed it on a middle layer. For package and distribution we have choosen Docker. Two months ago we've deployed project to production and have opened service for clients.
In the report the following topics will be covered:
- reasons of a choice Docker;
- why Docker without other tools is not enough for a production;
- what stack of technologies we used in our solution;
- what advantages we've got;
- what problems have been faced and how we've solved them.
Docker on a local machine and Docker in production — are two big differences. It's easy to play with technology but it's hard to do something real for many customers.
Half a year ago inside of Alpha Laboratory (division of Alfa-Bank) we've started building new microservices architecture for one of our pilot projects. We've almost completely changed a stack of the used technologies on a frontend and significantly changed it on a middle layer. For package and distribution we have choosen Docker. Two months ago we've deployed project to production and have opened service for clients.
In the report the following topics will be covered:
- reasons of a choice Docker;
- why Docker without other tools is not enough for a production;
- what stack of technologies we used in our solution;
- what advantages we've got;
- what problems have been faced and how we've solved them.
Best 20 SEO Techniques To Improve Website Visibility In SERPPixlogix Infotech
Boost your website's visibility with proven SEO techniques! Our latest blog dives into essential strategies to enhance your online presence, increase traffic, and rank higher on search engines. From keyword optimization to quality content creation, learn how to make your site stand out in the crowded digital landscape. Discover actionable tips and expert insights to elevate your SEO game.
Ocean lotus Threat actors project by John Sitima 2024 (1).pptxSitimaJohn
Ocean Lotus cyber threat actors represent a sophisticated, persistent, and politically motivated group that poses a significant risk to organizations and individuals in the Southeast Asian region. Their continuous evolution and adaptability underscore the need for robust cybersecurity measures and international cooperation to identify and mitigate the threats posed by such advanced persistent threat groups.
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...Jeffrey Haguewood
Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows.
We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases.
This video focuses on integration of Salesforce with Bonterra Impact Management.
Interested in deploying an integration with Salesforce for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.
HCL Notes and Domino License Cost Reduction in the World of DLAUpanagenda
Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-and-domino-license-cost-reduction-in-the-world-of-dlau/
The introduction of DLAU and the CCB & CCX licensing model caused quite a stir in the HCL community. As a Notes and Domino customer, you may have faced challenges with unexpected user counts and license costs. You probably have questions on how this new licensing approach works and how to benefit from it. Most importantly, you likely have budget constraints and want to save money where possible. Don’t worry, we can help with all of this!
We’ll show you how to fix common misconfigurations that cause higher-than-expected user counts, and how to identify accounts which you can deactivate to save money. There are also frequent patterns that can cause unnecessary cost, like using a person document instead of a mail-in for shared mailboxes. We’ll provide examples and solutions for those as well. And naturally we’ll explain the new licensing model.
Join HCL Ambassador Marc Thomas in this webinar with a special guest appearance from Franz Walder. It will give you the tools and know-how to stay on top of what is going on with Domino licensing. You will be able lower your cost through an optimized configuration and keep it low going forward.
These topics will be covered
- Reducing license cost by finding and fixing misconfigurations and superfluous accounts
- How do CCB and CCX licenses really work?
- Understanding the DLAU tool and how to best utilize it
- Tips for common problem areas, like team mailboxes, functional/test users, etc
- Practical examples and best practices to implement right away
In the rapidly evolving landscape of technologies, XML continues to play a vital role in structuring, storing, and transporting data across diverse systems. The recent advancements in artificial intelligence (AI) present new methodologies for enhancing XML development workflows, introducing efficiency, automation, and intelligent capabilities. This presentation will outline the scope and perspective of utilizing AI in XML development. The potential benefits and the possible pitfalls will be highlighted, providing a balanced view of the subject.
We will explore the capabilities of AI in understanding XML markup languages and autonomously creating structured XML content. Additionally, we will examine the capacity of AI to enrich plain text with appropriate XML markup. Practical examples and methodological guidelines will be provided to elucidate how AI can be effectively prompted to interpret and generate accurate XML markup.
Further emphasis will be placed on the role of AI in developing XSLT, or schemas such as XSD and Schematron. We will address the techniques and strategies adopted to create prompts for generating code, explaining code, or refactoring the code, and the results achieved.
The discussion will extend to how AI can be used to transform XML content. In particular, the focus will be on the use of AI XPath extension functions in XSLT, Schematron, Schematron Quick Fixes, or for XML content refactoring.
The presentation aims to deliver a comprehensive overview of AI usage in XML development, providing attendees with the necessary knowledge to make informed decisions. Whether you’re at the early stages of adopting AI or considering integrating it in advanced XML development, this presentation will cover all levels of expertise.
By highlighting the potential advantages and challenges of integrating AI with XML development tools and languages, the presentation seeks to inspire thoughtful conversation around the future of XML development. We’ll not only delve into the technical aspects of AI-powered XML development but also discuss practical implications and possible future directions.
Have you ever been confused by the myriad of choices offered by AWS for hosting a website or an API?
Lambda, Elastic Beanstalk, Lightsail, Amplify, S3 (and more!) can each host websites + APIs. But which one should we choose?
Which one is cheapest? Which one is fastest? Which one will scale to meet our needs?
Join me in this session as we dive into each AWS hosting service to determine which one is best for your scenario and explain why!
Main news related to the CCS TSI 2023 (2023/1695)Jakub Marek
An English 🇬🇧 translation of a presentation to the speech I gave about the main changes brought by CCS TSI 2023 at the biggest Czech conference on Communications and signalling systems on Railways, which was held in Clarion Hotel Olomouc from 7th to 9th November 2023 (konferenceszt.cz). Attended by around 500 participants and 200 on-line followers.
The original Czech 🇨🇿 version of the presentation can be found here: https://www.slideshare.net/slideshow/hlavni-novinky-souvisejici-s-ccs-tsi-2023-2023-1695/269688092 .
The videorecording (in Czech) from the presentation is available here: https://youtu.be/WzjJWm4IyPk?si=SImb06tuXGb30BEH .
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAUpanagenda
Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-und-domino-lizenzkostenreduzierung-in-der-welt-von-dlau/
DLAU und die Lizenzen nach dem CCB- und CCX-Modell sind für viele in der HCL-Community seit letztem Jahr ein heißes Thema. Als Notes- oder Domino-Kunde haben Sie vielleicht mit unerwartet hohen Benutzerzahlen und Lizenzgebühren zu kämpfen. Sie fragen sich vielleicht, wie diese neue Art der Lizenzierung funktioniert und welchen Nutzen sie Ihnen bringt. Vor allem wollen Sie sicherlich Ihr Budget einhalten und Kosten sparen, wo immer möglich. Das verstehen wir und wir möchten Ihnen dabei helfen!
Wir erklären Ihnen, wie Sie häufige Konfigurationsprobleme lösen können, die dazu führen können, dass mehr Benutzer gezählt werden als nötig, und wie Sie überflüssige oder ungenutzte Konten identifizieren und entfernen können, um Geld zu sparen. Es gibt auch einige Ansätze, die zu unnötigen Ausgaben führen können, z. B. wenn ein Personendokument anstelle eines Mail-Ins für geteilte Mailboxen verwendet wird. Wir zeigen Ihnen solche Fälle und deren Lösungen. Und natürlich erklären wir Ihnen das neue Lizenzmodell.
Nehmen Sie an diesem Webinar teil, bei dem HCL-Ambassador Marc Thomas und Gastredner Franz Walder Ihnen diese neue Welt näherbringen. Es vermittelt Ihnen die Tools und das Know-how, um den Überblick zu bewahren. Sie werden in der Lage sein, Ihre Kosten durch eine optimierte Domino-Konfiguration zu reduzieren und auch in Zukunft gering zu halten.
Diese Themen werden behandelt
- Reduzierung der Lizenzkosten durch Auffinden und Beheben von Fehlkonfigurationen und überflüssigen Konten
- Wie funktionieren CCB- und CCX-Lizenzen wirklich?
- Verstehen des DLAU-Tools und wie man es am besten nutzt
- Tipps für häufige Problembereiche, wie z. B. Team-Postfächer, Funktions-/Testbenutzer usw.
- Praxisbeispiele und Best Practices zum sofortigen Umsetzen
Webinar: Designing a schema for a Data WarehouseFederico Razzoli
Are you new to data warehouses (DWH)? Do you need to check whether your data warehouse follows the best practices for a good design? In both cases, this webinar is for you.
A data warehouse is a central relational database that contains all measurements about a business or an organisation. This data comes from a variety of heterogeneous data sources, which includes databases of any type that back the applications used by the company, data files exported by some applications, or APIs provided by internal or external services.
But designing a data warehouse correctly is a hard task, which requires gathering information about the business processes that need to be analysed in the first place. These processes must be translated into so-called star schemas, which means, denormalised databases where each table represents a dimension or facts.
We will discuss these topics:
- How to gather information about a business;
- Understanding dictionaries and how to identify business entities;
- Dimensions and facts;
- Setting a table granularity;
- Types of facts;
- Types of dimensions;
- Snowflakes and how to avoid them;
- Expanding existing dimensions and facts.
5th LF Energy Power Grid Model Meet-up SlidesDanBrown980551
5th Power Grid Model Meet-up
It is with great pleasure that we extend to you an invitation to the 5th Power Grid Model Meet-up, scheduled for 6th June 2024. This event will adopt a hybrid format, allowing participants to join us either through an online Mircosoft Teams session or in person at TU/e located at Den Dolech 2, Eindhoven, Netherlands. The meet-up will be hosted by Eindhoven University of Technology (TU/e), a research university specializing in engineering science & technology.
Power Grid Model
The global energy transition is placing new and unprecedented demands on Distribution System Operators (DSOs). Alongside upgrades to grid capacity, processes such as digitization, capacity optimization, and congestion management are becoming vital for delivering reliable services.
Power Grid Model is an open source project from Linux Foundation Energy and provides a calculation engine that is increasingly essential for DSOs. It offers a standards-based foundation enabling real-time power systems analysis, simulations of electrical power grids, and sophisticated what-if analysis. In addition, it enables in-depth studies and analysis of the electrical power grid’s behavior and performance. This comprehensive model incorporates essential factors such as power generation capacity, electrical losses, voltage levels, power flows, and system stability.
Power Grid Model is currently being applied in a wide variety of use cases, including grid planning, expansion, reliability, and congestion studies. It can also help in analyzing the impact of renewable energy integration, assessing the effects of disturbances or faults, and developing strategies for grid control and optimization.
What to expect
For the upcoming meetup we are organizing, we have an exciting lineup of activities planned:
-Insightful presentations covering two practical applications of the Power Grid Model.
-An update on the latest advancements in Power Grid -Model technology during the first and second quarters of 2024.
-An interactive brainstorming session to discuss and propose new feature requests.
-An opportunity to connect with fellow Power Grid Model enthusiasts and users.
21. Benefits
● Fast feedback loop
● Good for simple usecases
Drawbacks
● Hard to test complex scenarios
● Hard to collaborate
○ QA
○ other developers
Local Only
21
25. 25
Full Copy for each Developer
Own logical DB
Own topics and
subscriptions
26. Issues to
address
● Tool to establish “Vasya” env
○ 10-15 minutes max
● Handle the load
○ 10k workloads and much more
● Support separated infra components
○ dbs in containers
○ emulators for queues
26
27. Issues to
address
● Tool to establish “Vasya” env
○ 10-15 minutes max
● Handle the load
○ 10k workloads and much more
● Support separated infra components
○ dbs in containers
○ emulators for queues
27
28. Issues to
address
● Tool to establish “Vasya” env
○ 10-15 minutes max
● Handle the load
○ 10k workloads and much more
● Support separated infra components
○ dbs in containers
○ emulators for queues
28
29. Optimizations
● Part copy instead of full
○ core services first
○ specific services on-demand
● Shutdown every night
● Env per squad not developer
● Get rid of X
○ do not use service mesh
○ do not keep logs
29
30. Optimizations
● Part copy instead of full
○ core services first
○ specific services on-demand
● Shutdown every night
● Env per squad not developer
● Get rid of X
○ do not use service mesh
○ do not keep logs
30
31. Optimizations
● Part copy instead of full
○ core services first
○ specific services on-demand
● Shutdown every night
● Env per squad, not developer
● Get rid of X
○ do not use service mesh
○ do not keep logs
31
32. Optimizations
● Part copy instead of full
○ core services first
○ specific services on-demand
● Shutdown every night
● Env per squad, not developer
● Get rid of X
○ do not use service mesh
○ do not keep logs
32
33. Benefits
● Full isolation
● The cognitive load is low
Drawbacks
● Custom configuration
● High resources consumption
● You own - you troubleshooting
Full Copy for each Developer
33
57. What if not?
● No dogfooding
○ A silo between QA and developers
○ Developers are pushed to fix the stage
● We should keep stable two environments instead of one
○ Staging should be stable by design
○ The development environment should be stable too
■ If the authorization service doesn’t work -
developer cannot test their branch
57
58. What if not?
● No dogfooding
○ A silo between QA and developers
○ Developers are pushed to fix the stage
● We should keep stable two environments instead of one
○ Staging should be stable by design
○ The development environment should be stable too
■ If the authorization service doesn’t work -
developer cannot test their branch
58
129. Types of
Ephemeral
Environments
● One service branch
● Several services branches
○ under one x-source-route
○ Jira-based
● Custom environments
○ for squad (payment-dev)
○ for domain (warehouse)
129
130. Types of
Ephemeral
Environments
● One service branch
● Several services branches
○ under one x-source-route
○ Jira-based
● Custom environments
○ for squad (payment-dev)
○ for domain (warehouse)
130