Mayur Thakur, Managing Director, Goldman Sachs, at MLconf NYC 2017

•

4 gefällt mir•999 views

This document describes a proposed surveillance platform for bank compliance. It discusses the large scale of financial data that must be analyzed, including over 1 billion pieces of text and trading events per year. It outlines the technical challenges of working with diverse data formats and sources that can change over time. The proposed architecture preprocesses and flattens data from various sources into a common format before running multiple surveillance algorithms to detect issues like spoofing and generate alerts. Examples of real-world spoofing cases are provided to illustrate the types of patterns the system aims to identify.

Technologie

Surveillance Platform for Bank Compliance
Mayur Thakur, Goldman Sachs

2
• “Banks pay out £166bn over six years: a history of banking
misdeeds and fines” – The Guardian
• “Banks 'pay 60%' of profits in fines and customer payments” – BBC
News
• “Deutsche Bank to Pay $2.5 Billion to Settle Libor Investigation” –
The Wall Street Journal
• “$1.2 Billion Fine for Hedge Fund SAC Capital in Insider Case” – NY
Times
Stakes Are High

Key Technical Challenges
 Diverse data sets and formats (sql, flatfiles, proprietary, etc)
 Size of data, updated frequently
• ~1B* pieces of text per year
• ~1B edges in a graph
• 100s of millions of trading events in a day
 Data from past can change (e.g., manual trade correction)
• Causes a cascade of changes
 Surveillance decisions need to be debuggable
• Why was trade X on Oct 25, 2015 not flagged?
 Not real time; often need time guarantees (say, T+1)
* All numbers are “orders of magnitudes”
3

Surveillance Architechture
4
SQL 1
Surv. 1
SQL n
Flatfile 1
Flatfile m
Prop 1
Prop k
HDFS 1
HDFS q
Flattened
1
Flattened
2
Flattened
n
Preprocessing
pipeline 1
Preprocessing
pipeline 2
Preprocessing
pipeline n
Alerts…
Bookkeeping

A Real World Spoofing Case
Navinder Singh Sarao was accused of spoofing
 …and even contributing to the flash crash of 2010
Sarao pled guilty to spoofing in Nov 2016
He allegedly made $40M in illegal profit over years.
6

Review of Regulatory Cases
7
– Analyzed six regulatory enforcement cases for related to spoofing
– Identified common factors indicative of spoofing behavior
• Creating false impression of demand by placing spoof orders on opposite
side to trigger a price movement (“order imbalance”)
• Cancellation of spoof orders within short time after pivot execution (“time
to cancel post execution”)
Case
Factors
Order Imbalance Time to Cancel Post Execution
( > 2.5 times ) ( < 1 sec )
Sarao / Flash Crash a a
Hold Brothers a a
Coscia/ Panther a a
Visionary Trading NA a
Swift a 5 secs
3 Red a a

Transactions Data Pipeline
 Spoofing implementation has 2 parts: data preprocessing and surveillance logic
 Data preprocessing pipeline is reused for multiple surveillances
 ~100M orders, 1B mkt data points, 100K products, multiple order mgmt. system
8
Order 1
Related
Transactions
Spoofing
Orders n
Exec. 1
Exec m
Market 1
Market k
Account
Product
Flattened
Order
Flattened
Exec
Flattened
Market
Order Processing
Pipeline
Exec. Processing
Pipeline
Mkt. Processing
Pipeline
Front
running
Surv. n…
Alerts Alerts Alerts…

Related Transactions Table
9
Related
Transactions
Pivot Exec Orders Execs/Cancels MktData
216.8, 216.9, …
One row of the related transactions table contains information about one pivot
execution and all the activity around the time of that execution.
X X X X

Search Problem
10
Given a semi-structured corpus of about a 1B documents in a
hadoop cluster, design a search engine over YARN that is
fast and satisfies the investigative needs of a variety of users.
Unique Challenges
 Cannot move data outside of an already existing hadoop cluster
 Support deep scoring algorithms specifically for GS-specific signals
(colloquial language, trades, etc)
 Unstructured and structured signals

Search Workflow
11
Search
Master
Ranker
Fast Index
Servers
Slow Index
Servers
HBase
Web
Client
Yarn containers
HDFS
• Implemented as YARN apps
• Auth enabled
• Slow index Servers can scale as much as HBase
# indexed documents > 1Billion
# indexed tokens > 500 billion
Current Index Size Runs in several TBs (Memory
and Disk)

Weitere ähnliche Inhalte

Andere mochten auch

Alexandra Johnson, Software Engineer, SigOpt Alexandra works on everything from infrastructure to product features to blog posts. Previously, she worked on growth, APIs, and recommender systems at Polyvore (acquired by Yahoo). She majored in computer science at Carnegie Mellon University with a minor in discrete mathematics and logic, and during the summers she A/B tested recommendations at internships with Facebook and Rent the Runway. Abstract Summary: Common Problems In Hyperparameter Optimization: All large machine learning pipelines have tunable parameters, commonly referred to as hyperparameters. Hyperparameter optimization is the process by which we find the values for these parameters that cause our system to perform the best. SigOpt provides a Bayesian optimization platform that is commonly used for hyperparameter optimization, and I’m going to share some of the common problems we’ve seen when integrating into machine learning pipelines.

Alexandra Johnson, Software Engineer, SigOpt, at MLconf NYC 2017

MLconf

Hanie Sedghi is a Research Scientist at Allen Institute for Artificial Intelligence (AI2). Her research interests include large-scale machine learning, high-dimensional statistics and probabilistic models. More recently, she has been working on inference and learning in latent variable models. She has received her Ph.D. from University of Southern California with a minor in Mathematics in 2015. She was also a visiting researcher at University of California, Irvine working with professor Anandkumar during her Ph.D. She received her B.Sc. and M.Sc. degree from Sharif University of Technology, Tehran, Iran. Abstract summary Beating Perils of Non-convexity:Guaranteed Training of Neural Networks using Tensor Methods: Neural networks have revolutionized performance across multiple domains such as computer vision and speech recognition. However, training a neural network is a highly non-convex problem and the conventional stochastic gradient descent can get stuck in spurious local optima. We propose a computationally efficient method for training neural networks that also has guaranteed risk bounds. It is based on tensor decomposition which is guaranteed to converge to the globally optimal solution under mild conditions. We explain how this framework can be leveraged to train feedforward and recurrent neural networks.

Hanie Sedghi, Research Scientist at Allen Institute for Artificial Intelligen...

MLconf

Layla El Asri is a research Scientist at Maluuba. Her work explores artificial intelligence in the context of language understanding, dialogue and human-machine interaction. Layla leads a team seeking to build artificial intelligence systems that are knowledgeable and can exchange information with users to help users accomplish tasks or gain knowledge. Layla completed her PhD at Université de Lorraine in France. Abstract Summary: Teaching AI To Make Decisions and Communicate: Many advances have been made in the area of artificial intelligence, with the goal of building agents that understand how they can interact with their environments, reason and solve complex tasks, and communicate their findings to humans. In this talk, I will focus on efficient decision-making and communication. For decision-making, I will present some work on building an efficient representation of the environment and breaking down tasks into generalizable subtasks. For communication, I will focus on dialogue through natural language and present some of our work in this area.

Layla El Asri, Research Scientist, Maluuba

MLconf

Machine Learning with TensorFlow: TensorFlow has enabled cutting-edge machine learning research at the top AI labs in the world. At the same time it has made the technology accessible to a large audience leading to some amazing uses. TensorFlow is used for classification, recommendation, text parsing, sentiment analysis and more. This talk will go over the design that makes it fast, flexible, and easy to use, and describe how we continue to make it better.

Rajat Monga, Engineering Director, TensorFlow, Google at MLconf 2016

MLconf

Jeff Bradshaw is the founder of Adaptris and Group CTO of Adaptris/F4F/DBT within Reed Business Information. He has spent his career integrating data wherever it resides and in-flight across a number of industries including Agriculture, Airlines, Telecommunications, Healthcare, Government and Finance. Jeff has worked with and contributed to a number of international standards bodies and continues to work with large enterprises to help them extract value from their data silos and share data seamlessly with their trading partners to achieve business benefit. For the last few years Jeff has been focusing on Big Data and how to gather that across a wide range of sources to help gain insight into the agri-food supply chain. Abstract Summary: Precision agriculture – Predicting outcomes for farmers using machine learning to help feed the world: Agricultural data is vast, often unstructured and includes many challenges when working with legacy farm systems on premise in rural areas. For instance, traditional farm equipment such as tractors, sprayers, and combines aren’t often from the same vendor, and it’s complex moving data between them. This is further complicated with the vast array of other systems used by our farmers. Furthermore, the number of sensors in agriculture is astonishing, whether it is sensors that measure the gait of the cow walking into the dairy parlor, or chickens that are pecking. All this data needs to turn into usable information on a global scale to improve the yields farmers get and provide greater visibility into what’s going on both in and out of the farm. In this session, a case study will be shared on how data was collected, normalized and analyzed leveraging the open source HPCC Systems platform from remote Farm Management Systems (used by farmers to manage their farms), and when merged with weather data, soil data and actual machinery data, the analyzed predictions is used to feed Agronomists and Crop Protection/Seed Manufacturers to get recommendations back. The goal is to deliver a precision agriculture solution, helping farmers increase their yield, which then helps feed the growing population of the world.

Jeff Bradshaw, Founder, Adaptris

MLconf

Andrew recently joined Lucidworks to head up their Advisory practice, and is a Committer and PMC member on the Apache Mahout project. Abstract summary Apache Mahout: Distributed Matrix Math for Machine Learning: Machine learning and statistics tools like R and Scikit-learn are declarative, flexible, and extensible, but they scale poorly. “Big Data” tools such as Apache Spark, Apache Flink, and H2O distribute well, but have rudimentary functionality for machine learning and are not easily extensible. In this talk we present Apache Mahout, which provides a Scala-based, R-like DSL for doing linear algebra on distributed systems, letting practitioners quickly implement algorithms on distributed matrices. We will highlight new features in version 0.13 including the hybrid CPU/GPU-optimized engine, and a new framework for user-contributed methods and algorithms similar to R’s CRAN. We will cover some history of Mahout, introduce the R-Like Scala DSL, provide an overview of how Mahout is able to operate on matrices distributed across multiple computers, and how it takes advantage of GPUs on each computer in a cluster creating a hybrid distributed/GPU-accelerated environment; then demonstrate the kinds of normally complex or unfeasible problems users can easily solve with Mahout; show an integration which allows Mahout to leverage the visualization packages of projects such as R, Python, and D3; and lastly explain how to develop algorithms and submit them to the Mahout project for other users to use.

Andrew Musselman, Committer and PMC Member, Apache Mahout, at MLconf Seattle ...

MLconf

Corinna Cortes is a Danish computer scientist known for her contributions to machine learning. She is currently the Head of Google Research, New York. Cortes is a recipient of the Paris Kanellakis Theory and Practice Award for her work on theoretical foundations of support vector machines. Cortes received her M.S. degree in physics from Copenhagen University in 1989. In the same year she joined AT&T Bell Labs as a researcher and remained there for about ten years. She received her Ph.D. in computer science from the University of Rochester in 1993. Cortes currently serves as the Head of Google Research, New York. She is an Editorial Board member of the journal Machine Learning. Cortes’ research covers a wide range of topics in machine learning, including support vector machines and data mining. In 2008, she jointly with Vladimir Vapnik received the Paris Kanellakis Theory and Practice Award for the development of a highly effective algorithm for supervised learning known as support vector machines (SVM). Today, SVM is one of the most frequently used algorithms in machine learning, which is used in many practical applications, including medical diagnosis and weather forecasting. Abstract Summary: Harnessing Neural Networks: Deep learning has demonstrated impressive performance gain in many machine learning applications. However, unveiling and realizing these performance gains is not always straightforward. Discovering the right network architecture is critical for accuracy and often requires a human in the loop. Some network architectures occasionally produce spurious outputs, and the outputs have to be restricted to meet the needs of an application. Finally, realizing the performance gain in a production system can be difficult because of extensive inference times. In this talk we discuss methods for making neural networks efficient in production systems. We also discuss an efficient method for automatically learning the network architecture, called AdaNet. We provide theoretical arguments for the algorithm and present experimental evidence for its effectiveness.

Corinna Cortes, Head of Research, Google, at MLconf NYC 2017

MLconf

Aaron Roth is an Associate Professor of Computer and Information Sciences at the University of Pennsylvania, affiliated with the Warren Center for Network and Data Science, and co-director of the Networked and Social Systems Engineering (NETS) program. Previously, he received his PhD from Carnegie Mellon University and spent a year as a postdoctoral researcher at Microsoft Research New England. He is the recipient of a Presidential Early Career Award for Scientists and Engineers (PECASE) awarded by President Obama in 2016, an Alfred P. Sloan Research Fellowship, an NSF CAREER award, and a Yahoo! ACE award. His research focuses on the algorithmic foundations of data privacy, algorithmic fairness, game theory and mechanism design, learning theory, and the intersections of these topics. Together with Cynthia Dwork, he is the author of the book “The Algorithmic Foundations of Differential Privacy.” Abstract Summary: Differential Privacy and Machine Learning: In this talk, we will give a friendly introduction to Differential Privacy, a rigorous methodology for analyzing data subject to provable privacy guarantees, that has recently been widely deployed in several settings. The talk will specifically focus on the relationship between differential privacy and machine learning, which is surprisingly rich. This includes both the ability to do machine learning subject to differential privacy, and tools arising from differential privacy that can be used to make learning more reliable and robust (even when privacy is not a concern).

Aaron Roth, Associate Professor, University of Pennsylvania, at MLconf NYC 2017

MLconf

Yi Wang is the tech lead of AI Platform at Baidu. The team is a primary contributor of PaddlePaddle, the open source deep learning platform originally developed in Baidu. Before Baidu, he was a founding member of ScaledInference, a Palo Alto-based AI startup company. Before that, he was a senior staff at LinkedIn, engineering director of advertising system at Tencent, and researcher at Google. Abstract Summary: Fault-tolerable Deep Learning on General-purpose Clusters: Researchers have been used to running deep learning jobs on clusters. In industrial applications, AI is built on top of big data and deep learning is only one stage of the data pipeline. That is where MPI-based clusters are not enough, and general-purpose cluster management systems are necessary to run Web servers like Nginx, log collectors like fluentd and Kafka, data processors on top of Hadoop, Spark, and Storm, and deep learning, which improves the Web service quality. This talk explains how we integrate PaddlePaddle and Kubernetes to provide an open source fault-tolerable large-scale deep learning platform.

Yi Wang, Tech Lead of AI Platform, Baidu, at MLconf 2017

MLconf

DL4J and DataVec for Enterprise Deep Learning Workflows: Applications in NLP, sensor processing (IoT), image processing, and audio processing have all emerged as prime deep learning applications. In this session we will take a look at a practical review of building practical and secure Deep Learning workflows in the enterprise. We’ll see how DL4J’s DataVec tool enables scalable ETL and vectorization pipelines to be created for a single machine or scale out to Spark on Hadoop. We’ll also see how Deep Networks such as Recurrent Neural Networks are able to leverage DataVec to more quickly process data for modeling.

Josh Patterson, Advisor, Skymind – Deep learning for Industry at MLconf ATL 2016

MLconf

Sanjeev Satheesh, leads the Deep Speech team at Baidu’s Silicon valley AI lab. Baidu SVAIL is focused on developing hard AI technologies to impact hundreds of millions of people. The Story of End to End Models in Deep Learning The past few years have seen the explosive entrance of end to end deep learning models - in computer vision, speech recognition, machine translation, text to speech and others. In this talk, we look at this trend to identify what has worked well, and try to make some predictions for the future based on the next set of unsolved problems.

Sanjeev Satheesj, Research Scientist, Baidu at The AI Conference 2017

MLconf

Local Search Optimization for Hyper-Parameter Tuning: Many machine learning algorithms are sensitive to their hyper-parameter settings, lacking good universal rule-of-thumb defaults. In this talk we discuss the use of black-box local search optimization (LSO) for machine learning hyper-parameter tuning. Viewed as a black-box objective function of hyper-parameters, machine learning algorithms create a difficult class of optimization problems. The corresponding objective functions involved tend to be nonsmooth, discontinuous, unpredictably computationally expensive, requiring support for both continuous, categorical, and integer variables. Further evaluations can fail for a variety of reasons such as early exits due to node failure or hitting max time. Additionally, not all hyper-parameter combinations are compatible (creating so called “hidden constraints”). In this context, we apply a parallel hybrid derivative-free optimization algorithm that can make progress despite these difficulties providing significantly improved results over default settings with minimal user interaction. Further, we will address efficient parallel paradigms for different types of machine learning problems, while exploring the importance of validation to avoid overfitting and emphasizing that even for small data problems, the need to perform cross validations can create computationally intense functions that benefit from a distributed/threaded environment.

Funda Gunes, Senior Research Statistician Developer & Patrick Koch, Principal...

MLconf

Xin Luna Dong is a Principal Scientist at Amazon, leading the efforts of constructing Amazon Product Graph. She was one of the major contributors to the Knowledge Vault project, and has led the Knowledge-based Trust project, which is called the “Google Truth Machine” by Washington’s Post. She has won the VLDB Early Career Research Contribution Award for “advancing the state of the art of knowledge fusion”, and the Best Demo award in Sigmod 2005. She has co-authored book “Big Data Integration”, published 65+ papers in top conferences and journals, and given 20+ keynotes/invited-talks/tutorials. She is the PC co-chair for Sigmod 2018 and WAIM 2015, and serves as an area chair for Sigmod 2017, CIKM 2017, Sigmod 2015, ICDE 2013, and CIKM 2011. Abstract summary Leave No Valuable Data Behind: the Crazy Ideas and the Business: With the mission “leave no valuable data behind”, we developed techniques for knowledge fusion to guarantee the correctness of the knowledge. This talk starts with describing a few crazy ideas we have tested. The first, known as “Knowledge Vault”, used 15 extractors to automatically extract knowledge from 1B+ Webpages, obtaining 3B+ distinct (subject, predicate, object) knowledge triples and predicting well-calibrated probabilities for extracted triples. The second, known as “Knowledge-Based Trust”, estimated the trustworthiness of 119M webpages and 5.6M websites based on the correctness of their factual information. We then present how we bring the ideas to business in filling the gap between the knowledge at existing knowledge bases and the knowledge in the world.

Luna Dong, Principal Scientist, Amazon at MLconf Seattle 2017

MLconf

Scott is co-founder and CEO of SigOpt, a YC and a16z backed “Optimization as a Service” startup in San Francisco. Scott has been applying optimal learning techniques in industry and academia for years, from bioinformatics to production advertising systems. Before SigOpt, Scott worked on the Ad Targeting team at Yelp leading the charge on academic research and outreach with projects like the Yelp Dataset Challenge and open sourcing MOE. Scott holds a PhD in Applied Mathematics and an MS in Computer Science from Cornell University and BS degrees in Mathematics, Physics, and Computational Physics from Oregon State University. Scott was chosen as one of Forbes’ 30 under 30 in 2016. Abstract summary Bayesian Global Optimization: Using Optimal Learning to Deep Learning Models: In this talk we introduce Bayesian Optimization as an efficient way to optimize machine learning model parameters, especially when evaluating different configurations is time-consuming or expensive. Deep learning pipelines are notoriously expensive to train and often have many tunable parameters including hyperparameters, the architecture, feature transformations that can have a large impact on the efficacy of the model. We will motivate the problem by giving several example applications using open source deep learning frameworks and open datasets. We’ll compare the results of Bayesian Optimization to standard techniques like grid search, random search, and expert tuning.

Scott Clark, CEO, SigOpt, at MLconf Seattle 2017

MLconf

Ben Lau is a quantitative researcher in a macro hedge fund in Hong Kong and he looks to apply mathematical models and signal processing techniques to study the financial market. Prior joining the financial industry, he specialized in using his mathematical modelling skills to discover the mysteries of the universe whilst working at Stanford Linear Accelerator Centre, a national accelerator laboratory where he studied the asymmetry between matter and antimatter by analysing tens of billions of collision events created by the particle accelerators. Ben was awarded his Ph.D. in Particle Physics from Princeton University and his undergraduate degree (with First Class Honours) at the Chinese University of Hong Kong. Abstract Summary: Deep Reinforcement Learning: Developing a robotic car with the ability to form long term driving strategies is the key for enabling fully autonomous driving in the future. Reinforcement learning has been considered a strong AI paradigm which can be used to teach machines through interaction with the environment and by learning from their mistakes. In this talk, we will discuss how to apply deep reinforcement learning technique to train a self-driving car under an open source racing car simulator called TORCS. I am going to share how this is implemented and will discuss various challenges in this project.

Ben Lau, Quantitative Researcher, Hobbyist, at MLconf NYC 2017

MLconf

Smart Reply: Learning a Model of Conversation from Data: Smart Reply is a text assistance feature that was recently introduced to Inbox by Gmail. Given an incoming email message, the Smartreply system analyzes its contents and suggests complete responses that the recipient can send with just one tap. This talk will cover how we built Smartreply using a combination of deep learning and semantic clustering, as well as what we learned along the way and why we think it shows promise for the future of dialogue models.

Anjuli Kannan, Software Engineer, Google at MLconf SF 2016

MLconf

A Friendly Introduction To Causality: Causality has been studied under several frameworks in statistics and artificial intelligence. We will briefly survey Pearl’s Structural Equation model and explain how interventions can be used to discover causality. We will also present a novel information theoretic framework for discovering causal directions from observational data when interventions are not possible. The starting point is conditional independence in joint probability distributions and no prior knowledge on causal inference is required.

Alex Dimakis, Associate Professor, Dept. of Electrical and Computer Engineeri...

MLconf

Ross Goodwin, Technologist – Creater, Sunspring Ross Goodwin is a creative technologist, artist, hacker, data scientist, and former White House ghostwriter. Ross helped conceive Sunspring, a 2016 experimental science fiction short film entirely written by an artificial intelligence bot using neural networks. He employs machine learning, natural language processing, and other computational tools to realize new forms and interfaces for written language. Abstract Summary: Narrated Reality: Can machine intelligence enable new forms and interfaces for written language, or does it merely reveal an “uncanny valley” of text? Join Ross Goodwin as he discusses his work with neural networks for creative applications, including expressive image captioning, narration devices for your home and car, and a film (Sunspring) created from a computer generated screenplay.

Ross Goodwin, Technologist, Sunspring, MLconf NYC 2017

MLconf

A General Framework for Communication-Efficient Distributed Optimization: Communication remains the most significant bottleneck in the performance of distributed optimization algorithms for large-scale machine learning. In light of this, we propose a general framework, CoCoA, that uses local computation in a primal-dual setting to dramatically reduce the amount of necessary communication. Our framework enjoys strong convergence guarantees and exhibits state-of-the-art empirical performance in the distributed setting. We demonstrate this performance with extensive experiments in Apache Spark, achieving speedups of up to 50x compared to leading distributed methods on common machine learning objectives.

Virginia Smith, Researcher, UC Berkeley at MLconf SF 2016

MLconf

Andere mochten auch (19)

Alexandra Johnson, Software Engineer, SigOpt, at MLconf NYC 2017

Hanie Sedghi, Research Scientist at Allen Institute for Artificial Intelligen...

Layla El Asri, Research Scientist, Maluuba

Rajat Monga, Engineering Director, TensorFlow, Google at MLconf 2016

Jeff Bradshaw, Founder, Adaptris

Andrew Musselman, Committer and PMC Member, Apache Mahout, at MLconf Seattle ...

Corinna Cortes, Head of Research, Google, at MLconf NYC 2017

Aaron Roth, Associate Professor, University of Pennsylvania, at MLconf NYC 2017

Yi Wang, Tech Lead of AI Platform, Baidu, at MLconf 2017

Josh Patterson, Advisor, Skymind – Deep learning for Industry at MLconf ATL 2016

Sanjeev Satheesj, Research Scientist, Baidu at The AI Conference 2017

Funda Gunes, Senior Research Statistician Developer & Patrick Koch, Principal...

Luna Dong, Principal Scientist, Amazon at MLconf Seattle 2017

Scott Clark, CEO, SigOpt, at MLconf Seattle 2017

Ben Lau, Quantitative Researcher, Hobbyist, at MLconf NYC 2017

Anjuli Kannan, Software Engineer, Google at MLconf SF 2016

Alex Dimakis, Associate Professor, Dept. of Electrical and Computer Engineeri...

Ross Goodwin, Technologist, Sunspring, MLconf NYC 2017

Virginia Smith, Researcher, UC Berkeley at MLconf SF 2016

Ähnlich wie Mayur Thakur, Managing Director, Goldman Sachs, at MLconf NYC 2017

Quant congressusa2011algotradinglast

Tomasz Waszczyk

Algorithmic and high-frequency_trading 2011

jy Torres

Learn how the combination of Apache Kafka and Apache Flink is making stateful stream processing even more expressive and flexible to support applications in streaming that were previously not considered streamable. The new world of applications and fast data architectures has broken up the database: Raw data persistence comes in the form of event logs, and the state of the world is computed by a stream processor. Apache Kafka provides a strong solution for the event log, while Apache Flink forms a powerful foundation for the computation over the event streams. In this talk we discuss how Flink’s abstraction and management of application state have evolved over time and how Flink’s snapshot persistence model and Kafka’s log work together to form a base to build ‘versioned applications’. We will also show how end-to-end exactly-once processing works through a smart integration of Kafka’s transactions and Flink’s checkpointing mechanism.

2018-04 Kafka Summit London: Stephan Ewen - "Apache Flink and Apache Kafka fo...

Ververica

Speaker: Neil Avery, Technologist, Office of the CTO, Confluent Stream processing is now at the forefront of many company strategies. Over the last couple of years we have seen streaming use cases explode and now proliferate the landscape of any modern business. Use cases including digital transformation, IoT, real-time risk, payments microservices and machine learning are all built on the fundamental that they need fast data and they need it at scale. Apache Kafka® has long been the streaming platform of choice, its origins of being dumb pipes for big data have long since been left behind and now it is the goto-streaming platform of choice. Stream processing beckons as being the vehicle for driving those streams, and along with it brings a world of real-time semantics surrounding windowing, joining, correctness, elasticity, and accessibility. The ‘current state of stream processing’ walks through the origins of stream processing, applicable use cases and then dives into the challenges currently facing the world of stream processing as it drives the next data revolution. Neil is a Technologist in the Office of the CTO at Confluent, the company founded by the creators of Apache Kafka. He has over 20 years of expertise of working on distributed computing, messaging and stream processing. He has built or redesigned commercial messaging platforms, distributed caching products as well as developed large scale bespoke systems for tier-1 banks. After a period at ThoughtWorks, he went on to build some of the first distributed risk engines in financial services. In 2008 he launched a startup that specialised in distributed data analytics and visualization. Prior to joining Confluent he was the CTO at a fintech consultancy. Watch the recording: https://videos.confluent.io/watch/rmU6GHrd4EKFaZrRhdTE3s?.

The State of Stream Processing

confluent

Kafka Streams State Stores Being Persistent

confluent

The new stringent MiFID II regulatory technical standards (RTS) is top of mind for Banks, venues, and participants. One of the areas of major concern and confusion are the new regulatory technical standards (RTS) for trade transaction record management and reporting for trades made to venues that support high speed trading i.e., almost all electronic trades. The new MiFID II RTS standards require all parties involved in a trade to capture, record, maintain and report all information related to the transaction life-cycle with precise time-stamps accurately synchronised to the world time standard - UTC (Coordinated Universal Time). Existing systems cannot do this. The reason the regulator wants to do this is to be able to retrospectively recreate the sequence of market actions that led to a particular market condition (possibly abusive behaviour) and forensically verify who did what, how markets reacted, and who is to blame. It is very difficult to meet these new MiFID II regulatory technical standards (RTS) and may cost many millions of dollars to replace or retrofit the existing trading systems with this new capability. In addition, they will incur the ongoing costs of remaining compliant amid changes in their technology and amid future regulatory requirements. The banks and venues are concerned that even if they spend this money, their implementations may not meet the requirements and intention of (MiFID II) the new regulations, and failure to be compliant may result in significant fines. Corvil is introducing a novel new solution that addresses this problem - it is able to meet these new MIFID II requirements for UTC-synchronised trade transaction record management without requiring a retrofit or re-architecture of trading platform components.

MiFID II Compliance Solution from Corvil

Corvil

Apache Flink for IoT: How Event-Time Processing Enables Easy and Accurate Ana...

Big Data Spain

The Case for Disaggregation of Compute in the Data Center

Juniper Networks

7 Predictive Analytics, Spark , Streaming use cases

DataWorks Summit/Hadoop Summit

In this tutorial we walk through state-of-the-art streaming systems, algorithms, and deployment architectures and cover the typical challenges in modern real-time big data platforms and offering insights on how to address them. We also discuss how advances in technology might impact the streaming architectures and applications of the future. Along the way, we explore the interplay between storage and stream processing and discuss future developments.

Modern real-time streaming architectures

Arun Kejariwal

Block chain as a graph

DZee Solutions

Лекция в СГТУ: Информационные системы и технологическая инфраструктура биржев...

Iosif Itkin

Circonus: Design failures - A Case Study

Heinrich Hartmann

As more and more organizations and individual users turn to Apache Flink for their streaming workloads, there is a bigger demand for additional functionality out-of-the-box. On one hand, there is demand for more low-level APIs that allow for more control, while on the other, users ask for more high-level additions that make the common cases easier to express. This talk will present the new concepts added to the Datastream API in Flink-1.2 and for the upcoming Flink-1.3 release that tried to consolidate the aforementioned goals. We will talk, among others, about the ProcessFunction, a new low level stream processing primitive that gives the user full control over how each event is processed and can register and react to timers, changes in the windowing logic that allow for more flexible windowing strategies, side outputs, and new features concerning the Flink connectors.

Flink Forward SF 2017: Konstantinos Kloudas - Extending Flink’s Streaming APIs

Flink Forward

Real- Time Analytics – Process Automation

BSP Media Group

SI_050212_Olsen_Alknecht_PGE_CBMproject

Jerry Olson

Oh, now what? Another technobabble name - megaservice! What is a megaservice? A megaservice is a microservice that is capable of processing up to one million logical operations per second. In this context, the service prefix indicates the expected maximum sustained per second logical operation throughput. In this talk, we will look at the more common deca-service (tens of LOPS), hecto-service (hundreds of LOPS), and kilo-service (thousands of LOPS) implementation techniques. We examine some of the common performance bottlenecks and look at methods to push services to higher throughput levels. I'll share our experiences with a demo microservice application that we are using to push the cloud service providers to their limits. The demo application is composed of two Akka CQRS microservices. The app provides an interactive world map UI that visualizes IoT devices' distribution spread across the planet. With the click of the mouse, we generate thousands, tens of thousands, even millions of IoT devices and observe how various databases and cloud systems react when push to the limit. All of the application code is built using OSS such as Akka, Java, JavaScript, and Kubernetes. The demo app is available for you to use as a learning tool for building your megaservices.

How to build megaservices mind7 2021 June 29

Hugh McKee

Putting the Micro into Microservices with Stateful Stream Processing

confluent

Event Driven Services Part 3: Putting the Micro into Microservices with State...

Ben Stopford

7. transaction mang

khoahuy82

Ähnlich wie Mayur Thakur, Managing Director, Goldman Sachs, at MLconf NYC 2017 (20)

Quant congressusa2011algotradinglast

Algorithmic and high-frequency_trading 2011

2018-04 Kafka Summit London: Stephan Ewen - "Apache Flink and Apache Kafka fo...

The State of Stream Processing

Kafka Streams State Stores Being Persistent

MiFID II Compliance Solution from Corvil

Apache Flink for IoT: How Event-Time Processing Enables Easy and Accurate Ana...

The Case for Disaggregation of Compute in the Data Center

7 Predictive Analytics, Spark , Streaming use cases

Modern real-time streaming architectures

Block chain as a graph

Лекция в СГТУ: Информационные системы и технологическая инфраструктура биржев...

Circonus: Design failures - A Case Study

Flink Forward SF 2017: Konstantinos Kloudas - Extending Flink’s Streaming APIs

Real- Time Analytics – Process Automation

SI_050212_Olsen_Alknecht_PGE_CBMproject

How to build megaservices mind7 2021 June 29

Putting the Micro into Microservices with Stateful Stream Processing

Event Driven Services Part 3: Putting the Micro into Microservices with State...

7. transaction mang

Mehr von MLconf

Understanding Human Impact: Social and Equity Assessments for AI Technologies Social and Equity Impact Assessments have broad applications but can be a useful tool to explore and mitigate for Machine Learning fairness issues and can be applied to product specific questions as a way to generate insights and learnings about users, as well as impacts on society broadly as a result of the deployment of new and emerging technologies. In this presentation, my goal is to advocate for and highlight the need to consult community and external stakeholder engagement to develop a new knowledge base and understanding of the human and social consequences of algorithmic decision making and to introduce principles, methods and process for these types of impact assessments.

Jamila Smith-Loud - Understanding Human Impact: Social and Equity Assessments...

MLconf

The Brain’s Guide to Dealing with Context in Language Understanding Like the visual cortex, the regions of the brain involved in understanding language represent information hierarchically. But whereas the visual cortex organizes things into a spatial hierarchy, the language regions encode information into a hierarchy of timescale. This organization is key to our uniquely human ability to integrate semantic information across narratives. More and more, deep learning-based approaches to natural language understanding embrace models that incorporate contextual information at varying timescales. This has not only led to state-of-the art performance on many difficult natural language tasks, but also to breakthroughs in our understanding of brain activity. In this talk, we will discuss the important connection between language understanding and context at different timescales. We will explore how different deep learning architectures capture timescales in language and how closely their encodings mimic the brain. Along the way, we will uncover some surprising discoveries about what depth does and doesn’t buy you in deep recurrent neural networks. And we’ll describe a new, more flexible way to think about these architectures and ease design space exploration. Finally, we’ll discuss some of the exciting applications made possible by these breakthroughs.

Ted Willke - The Brain’s Guide to Dealing with Context in Language Understanding

MLconf

Applying Computer Vision to Reduce Contamination in the Recycling Stream With China’s recent refusal of most foreign recyclables, North American waste haulers are scrambling to figure out how to make on-shore recycling cost-effective in order to continue providing recycling services. Recyclables that were once being shipped to China for manual sorting are now primarily being redirected to landfills or incinerators. Without a solution, a nearly $5 billion annual recycling market could come to a halt. Purity in the recycling stream is key to this effort as contaminants in the stream can increase the cost of operations, damage equipment and reduce the ability to create pure commodities suitable for creating recycled goods. This market disruption as a result of China’s new regulations, however, provides us the chance to re-examine and improve our current disposal & collection habits with modern monitoring & artificial intelligence technology. Using images from our in-dumpster cameras, Compology has developed an ML-based process that helps identify, measure and alert for contaminants in recycling containers before they are picked-up, helping keep the recycling stream clean. Our convolutional neural network flags potential instances of contamination inside a dumpster, enabling garbage haulers to know which containers have the wrong type of material inside. This allows them to provide targeted, timely education, and when appropriate, assess fines, to improve recycling compliance at the businesses and residences they serve, helping keep recycling services financially viable. In this presentation, we will walk through our ML-based contamination measurement and scoring process by showing how Waste Management, a national waste hauler, has experienced 57% contamination reduction in nearly 2,000 containers over six months, This progress shows significant strides towards financially viable recycling services.

Justin Armstrong - Applying Computer Vision to Reduce Contamination in the Re...

MLconf

Quantum Computing: a Treasure Hunt, not a Gold Rush Quantum computers promise a significant step up in computational power over conventional computers, but also suffer a number of counterintuitive limitations --- both in their computational model and in leading lab implementations. In this talk, we review how quantum computers compete with conventional computers and how conventional computers try to hold their ground. Then we outline what stands in the way of successful quantum ML applications.

Igor Markov - Quantum Computing: a Treasure Hunt, not a Gold Rush

MLconf

Data Labeling as Religious Experience One of the most common places to deploy a production machine learning systems is as a replacement for a legacy rules-based system that is having a hard time keeping up with new edge cases and requirements. I'll be walking through the process and tooling we used to help us design, train, and deploy a model to replace a set of static rules we had for handling invite spam at Slack, talk about what we learned, and discuss some problems to solve in order to make these migrations easier for everyone.

Josh Wills - Data Labeling as Religious Experience

MLconf

Project GaitNet: Ushering in the ImageNet moment for human Gait kinematics The emergence of the upright human bipedal gait can be traced back 4 to 2.8 million years ago, to the now extinct hominin Australopithecus afarensis. Fine grained analysis of gait using the modern MEMS sensors found on all smartphones not just reveals a lot about the person’s orthopedic and neuromuscular health status, but also has enough idiosyncratic clues that it can be harnessed as a passive biometric. While there were many siloed attempts made by the machine learning community to model Bipedal Gait sensor data, these were done with small datasets oft collected in restricted academic environs. In this talk, we will introduce the ImageNet moment for human gait analysis by presenting 'Project GaitNet', the largest ever planet-sized motion sensor based human bipedal gait dataset ever curated. We’ll also present the associated state-of-the-art results in classifying humans harnessing novel deep neural architectures and the related success stories we have enjoyed in transfer-learning into disparate domains of human kinematics analysis.

Vinay Prabhu - Project GaitNet: Ushering in the ImageNet moment for human Gai...

MLconf

Machine Learning Methods in Detecting Alzheimer’s Disease from Speech and Language Alzheimer's disease affects millions of people worldwide, and it is important to predict the disease as early and as accurate as possible. In this talk, I will discuss development of novel ML models that help classifying healthy people from those who develop Alzheimer's, using short samples of human speech. As an input to the model, features of different modalities are extracted from speech audio samples and transcriptions: (1) syntactic measures, such as e.g. production rules extracted from syntactic parse trees, (2) lexical measures, such as e.g. features of lexical richness and complexity and lexical norms, and (3) acoustic measures, such as e.g. standard Mel-frequency cepstral coefficients. I will present the ML model that detects cognitive impairment by reaching agreement among modalities. The resulting model is able to achieve state of the art performance in both supervised and semi-supervised manner, using manual transcripts of human speech. Additionally, I will discuss potential limitations of any fully-automated speech-based Alzheimer's disease detection model, focusing mostly on the analysis of the impact of a not-so-accurate automatic speech recognition (ASR) on the classification performance. To illustrate this, I will present the experiments with controlled amounts of artificially generated ASR errors and explain how the deletion errors affect Alzheimer's detection performance the most, due to their impact on the features of syntactic and lexical complexity.

Jekaterina Novikova - Machine Learning Methods in Detecting Alzheimer’s Disea...

MLconf

Optimized Image Classification on the Cheap In this talk, we anchor on building an image classifier trained on the Stanford Cars dataset to evaluate two approaches to transfer learning -fine tuning and feature extraction- and the impact of hyperparameter optimization on these techniques. Once we define the most performant transfer learning technique for Stanford Cars, we will double the size of the dataset through image augmentation to boost the classifier’s performance. We will use Bayesian optimization to learn the hyperparameters associated with image transformations using the downstream image classifier’s performance as the guide. In conjunction with model performance, we will also focus on the features of these augmented images and the downstream implications for our image classifier. To both maximize model performance on a budget and explore the impact of optimization on these methods, we apply a particularly efficient implementation of Bayesian optimization to each of these architectures in this comparison. Our goal is to draw on a rigorous set of experimental results that can help us answer the question: how can resource-constrained teams make trade-offs between efficiency and effectiveness using pre-trained models?

Meghana Ravikumar - Optimized Image Classification on the Cheap

MLconf

The Importance of Modeling Data Collection Data sets used in machine learning are often collected in a systematically biased way - certain data points are more likely to be collected than others. We call this "observation bias". For example, in health care, we are more likely to see lab tests when the patient is feeling unwell than otherwise. Failing to account for observation bias can, of course, result in poor predictions on new data. By contrast, properly accounting for this bias allows us to make better use of the data we do have. In this presentation, we discuss practical and theoretical approaches to dealing with observation bias. When the nature of the bias is known, there are simple adjustments we can make to nonparametric function estimation techniques, such as Gaussian Process models. We also discuss the scenario where the data collection model is unknown. In this case, there are steps we can take to estimate it from observed data. Finally, we demonstrate that having a small subset of data points that are known to be collected at random - that is, in an unbiased way - can vastly improve our ability to account for observation bias in the rest of the data set. My hope is that attendees of this presentation will be aware of the perils of observation bias in their own work, and be equipped with tools to address it.

Noam Finkelstein - The Importance of Modeling Data Collection

MLconf

The Uncanny Valley of ML Every so often, the conundrum of the Uncanny Valley re-emerges as advanced technologies evolve from clearly experimental products to refined accepted technologies. We have seen its effects in robotics, computer graphics, and page load times. The debate of how to handle the new technology detracts from its benefits. When machine learning is added to human decision systems a similar effect can be measured in increased response time and decreased accuracy. These systems include radiology, judicial assignments, bus schedules, housing prices, power grids and a growing variety of applications. Unfortunately, the Uncanny Valley of ML can be hard to detect in these systems and can lead to degraded system performance when ML is introduced, at great expense. Here, we'll introduce key design principles for introducing ML into human decision systems to navigate around the Uncanny Valley and avoid its pitfalls.

June Andrews - The Uncanny Valley of ML

MLconf

Deep Learning Architectures for Semantic Relation Detection Tasks Recognizing and distinguishing specific semantic relations from other types of semantic relations is an essential part of language understanding systems. Identifying expressions with similar and contrasting meanings is valuable for NLP systems which go beyond recognizing semantic relatedness and require to identify specific semantic relations. In this talk, I will first present novel techniques for creating labelled datasets required for training deep learning models for classifying semantic relations between phrases. I will further present various neural network architectures that integrate morphological features into integrated path-based and distributional relation detection algorithms and demonstrate that this model outperforms state-of-the-art models in distinguishing semantic relations and is capable of efficiently handling multi-word expressions.

Sneha Rajana - Deep Learning Architectures for Semantic Relation Detection Tasks

MLconf

Building an Incrementally Trained, Local Taste Aware, Global Deep Learned Recommender System Model At Netflix, our main goal is to maximize our members’ enjoyment of the selected show by minimizing the amount of time it takes for them to find it. We try to achieve this goal by personalizing almost all the aspects of our product -- from what shows to recommend, to how to present these shows and construct their home-pages to what images to select per show, among many other things. Everything is recommendations for us and as an applied Machine Learning group, we spend our time building models for personalization that will eventually increase the joy and satisfaction of our members. In this talk we will primarily focus our attention on a) making a global deep learned recommender model that is regional tastes and popularity aware and b) adapting this model to changing taste preferences as well as dynamic catalog availability. We will first go through some standard recommender system models that use Matrix Factorization and Topic Models and then compare and contrast them with more powerful and higher capacity deep learning based models such as sequence models that use recurrent neural networks. We will show what it entails to build a global model that is aware of regional taste preferences and catalog availability. We will show how models that are built on simple Maximum Likelihood principle fail to do that. We will then describe one solution that we have employed in order to enable the global deep learned models to focus their attention on capturing regional taste preferences and changing catalog.In the latter half of the talk, we will discuss how we do incremental learning of deep learned recommender system models. Why do we need to do that ? Everything changes with time. Users’ tastes change with time. What’s available on Netflix and what’s popular also change over time. Therefore, updating or improving recommendation systems over time is necessary to bring more joy to users. In addition to how we apply incremental learning, we will discuss some of the challenges we face involving large-scale data preparation, infrastructure setup for incremental model training as well as pipeline scheduling. The incremental training enables us to serve fresher models trained on fresher and larger amounts of data. This helps our recommender system to nicely and quickly adapt to catalog and users’ taste changes, and improve overall performance.

Anoop Deoras - Building an Incrementally Trained, Local Taste Aware, Global D...

MLconf

Vito Ostuni - The Voice: New Challenges in a Zero UI World The adoption of voice-enabled devices has seen an explosive growth in the last few years and music consumption is among the most popular use cases. Music personalization and recommendation plays a major role at Pandora in providing a daily delightful listening experience for millions of users. In turn, providing the same perfectly tailored listening experience through these novel voice interfaces brings new interesting challenges and exciting opportunities. In this talk we will describe how we apply personalization and recommendation techniques in three common voice scenarios which can be defined in terms of request types: known-item, thematic, and broad open-ended. We will describe how we use deep learning slot filling techniques and query classification to interpret the user intent and identify the main concepts in the query. We will also present the differences and challenges regarding evaluation of voice powered recommendation systems. Since pure voice interfaces do not contain visual UI elements, relevance labels need to be inferred through implicit actions such as play time, query reformulations or other types of session level information. Another difference is that while the typical recommendation task corresponds to recommending a ranked list of items, a voice play request translates into a single item play action. Thus, some considerations about closed feedback loops need to be made. In summary, improving the quality of voice interactions in music services is a relatively new challenge and many exciting opportunities for breakthroughs still remain. There are many new aspects of recommendation system interfaces to address to bring a delightful and effortless experience for voice users. We will share a few open challenges to solve for the future.

Vito Ostuni - The Voice: New Challenges in a Zero UI World

MLconf

Anna choromanska - Data-driven Challenges in AI: Scale, Information Selection...

MLconf

Janani Kalyanam - Machine Learning to Detect Illegal Online Sales of Prescrip...

MLconf

Esperanza Lopez Aguilera - Using a Bayesian Neural Network in the Detection o...

MLconf

Neel Sundaresan - Teaching a machine to code

MLconf

Rishabh Mehrotra - Recommendations in a Marketplace: Personalizing Explainabl...

MLconf

Soumith Chintala - Increasing the Impact of AI Through Better Software

MLconf

Roy Lowrance - Predicting Bond Prices: Regime Changes

MLconf

Mehr von MLconf (20)

Jamila Smith-Loud - Understanding Human Impact: Social and Equity Assessments...

Ted Willke - The Brain’s Guide to Dealing with Context in Language Understanding

Justin Armstrong - Applying Computer Vision to Reduce Contamination in the Re...

Igor Markov - Quantum Computing: a Treasure Hunt, not a Gold Rush

Josh Wills - Data Labeling as Religious Experience

Vinay Prabhu - Project GaitNet: Ushering in the ImageNet moment for human Gai...

Jekaterina Novikova - Machine Learning Methods in Detecting Alzheimer’s Disea...

Meghana Ravikumar - Optimized Image Classification on the Cheap

Noam Finkelstein - The Importance of Modeling Data Collection

June Andrews - The Uncanny Valley of ML

Sneha Rajana - Deep Learning Architectures for Semantic Relation Detection Tasks

Anoop Deoras - Building an Incrementally Trained, Local Taste Aware, Global D...

Vito Ostuni - The Voice: New Challenges in a Zero UI World

Anna choromanska - Data-driven Challenges in AI: Scale, Information Selection...

Janani Kalyanam - Machine Learning to Detect Illegal Online Sales of Prescrip...

Esperanza Lopez Aguilera - Using a Bayesian Neural Network in the Detection o...

Neel Sundaresan - Teaching a machine to code

Rishabh Mehrotra - Recommendations in a Marketplace: Personalizing Explainabl...

Soumith Chintala - Increasing the Impact of AI Through Better Software

Roy Lowrance - Predicting Bond Prices: Regime Changes

Kürzlich hochgeladen

Strategies for Landing an Oracle DBA Job as a Fresher

Remote DBA Services

The presentation explores the development and application of artificial intelligence (AI) from its inception to its current status in the modern world. The term "artificial intelligence" was first coined by John McCarthy in 1956 to describe efforts to develop computer programs capable of performing tasks that typically require human intelligence. This concept was first introduced at a conference held at Dartmouth College, where programs demonstrated capabilities such as playing chess, proving theorems, and interpreting texts. In the early stages, Alan Turing contributed to the field by defining intelligence as the ability of a being to respond to certain questions intelligently, proposing what is now known as the Turing Test to evaluate the presence of intelligent behavior in machines. As the decades progressed, AI evolved significantly. The 1980s focused on machine learning, teaching computers to learn from data, leading to the development of models that could improve their performance based on their experiences. The 1990s and 2000s saw further advances in algorithms and computational power, which allowed for more sophisticated data analysis techniques, including data mining. By the 2010s, the proliferation of big data and the refinement of deep learning techniques enabled AI to become mainstream. Notable milestones included the success of Google's AlphaGo and advancements in autonomous vehicles by companies like Tesla and Waymo. A major theme of the presentation is the application of generative AI, which has been used for tasks such as natural language text generation, translation, and question answering. Generative AI uses large datasets to train models that can then produce new, coherent pieces of text or other media. The presentation also discusses the ethical implications and the need for regulation in AI, highlighting issues such as privacy, bias, and the potential for misuse. These concerns have prompted calls for comprehensive regulations to ensure the safe and equitable use of AI technologies. Artificial intelligence has also played a significant role in healthcare, particularly highlighted during the COVID-19 pandemic, where it was used in drug discovery, vaccine development, and analyzing the spread of the virus. The capabilities of AI in healthcare are vast, ranging from medical diagnostics to personalized medicine, demonstrating the technology's potential to revolutionize fields beyond just technical or consumer applications. In conclusion, AI continues to be a rapidly evolving field with significant implications for various aspects of society. The development from theoretical concepts to real-world applications illustrates both the potential benefits and the challenges that come with integrating advanced technologies into everyday life. The ongoing discussion about AI ethics and regulation underscores the importance of managing these technologies responsibly to maximize their their benefits while minimizing potential harms.

Artificial Intelligence: Facts and Myths

Joaquim Jorge

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

The Digital Insurer

Real Time Object Detection Using Open CV

Khem

The 7 Things I Know About Cyber Security After 25 Years | April 2024

Rafal Los

presentation ICT roal in 21st century education

jfdjdjcjdnsjd

Discord is a free app offering voice, video, and text chat functionalities, primarily catering to the gaming community. It serves as a hub for users to create and join servers tailored to their interests. Discord’s ecosystem comprises servers, each functioning as a distinct online community with its own channels dedicated to specific topics or activities. Users can engage in text-based discussions, voice calls, or video chats within these channels. Understanding Discord Servers Discord servers are virtual spaces where users congregate to interact, share content, and build communities. Servers may revolve around gaming, hobbies, interests, or fandoms, providing a platform for like-minded individuals to connect. Communication Features Discord offers a range of communication tools, including text channels for messaging, voice channels for real-time audio conversations, and video channels for face-to-face interactions. These features facilitate seamless communication and collaboration. What Does NSFW Mean? The acronym NSFW stands for “Not Safe For Work,” indicating content that may be inappropriate for professional or public settings. NSFW Content NSFW content encompasses material that is sexually explicit, violent, or otherwise graphic in nature. It often includes nudity, profanity, or depictions of sensitive topics.

Understanding Discord NSFW Servers A Guide for Responsible Users.pdf

UK Journal

Automating Google Workspace (GWS) & more with Apps Script

wesley chun

Join our latest Connector Corner webinar to discover how UiPath Integration Service revolutionizes API-centric automation in a 'Quote to Cash' process—and how that automation empowers businesses to accelerate revenue generation. A comprehensive demo will explore connecting systems, GenAI, and people, through powerful pre-built connectors designed to speed process cycle times. Speakers: James Dickson, Senior Software Engineer Charlie Greenberg, Host, Product Marketing Manager

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

DianaGray10

Manulife - Insurer Innovation Award 2024

The Digital Insurer

With more memory available, system performance of three Dell devices increased, which can translate to a better user experience Conclusion When your system has plenty of RAM to meet your needs, you can efficiently access the applications and data you need to finish projects and to-do lists without sacrificing time and focus. Our test results show that with more memory available, three Dell PCs delivered better performance and took less time to complete the Procyon Office Productivity benchmark. These advantages translate to users being able to complete workflows more quickly and multitask more easily. Whether you need the mobility of the Latitude 5440, the creative capabilities of the Precision 3470, or the high performance of the OptiPlex Tower Plus 7010, configuring your system with more RAM can help keep processes running smoothly, enabling you to do more without compromising performance.

Boost PC performance: How more available memory can improve productivity

Principled Technologies

🐬 The future of MySQL is Postgres 🐘

RTylerCroy

The value of a flexible API Management solution for Open Banking Steve Melan, Manager for IT Innovation and Architecture - State's and Saving's Bank of Luxembourg Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - The value of a flexible API Management solution for O...

apidays

MySQL Webinar, presented on the 25th of April, 2024. Summary: MySQL solutions enable the deployment of diverse Database Architectures tailored to specific needs, including High Availability, Disaster Recovery, and Read Scale-Out. With MySQL Shell's AdminAPI, administrators can seamlessly set up, manage, and monitor these solutions, ensuring efficiency and ease of use in their administration. MySQL Router, on the other hand, provides transparent routing from the application traffic to the backend servers in the architectures, requiring minimal configuration. Completely built in-house and supported by Oracle, these solutions have been adopted by enterprises of all sizes for their business-critical applications. In this presentation, we'll delve into various database architecture solutions to help you choose the right one based on your business requirements. Focusing on technical details and the latest features to maximize the potential of these solutions.

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...

Miguel Araújo

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Edi Saputra

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

The Digital Insurer

Following the popularity of "Cloud Revolution: Exploring the New Wave of Serverless Spatial Data," we're thrilled to announce this much-anticipated encore webinar. In this sequel, we'll dive deeper into the Cloud-Native realm by uncovering practical applications and FME support for these new formats, including COGs, COPC, FlatGeoBuf, GeoParquet, STAC, and ZARR. Building on the foundation laid by industry leaders Michelle Roby of Radiant Earth and Chris Holmes of Planet in the first webinar, this second part offers an in-depth look at the real-world application and behind-the-scenes dynamics of these cutting-edge formats. We will spotlight specific use-cases and workflows, showcasing their efficiency and relevance in practical scenarios. Discover the vast possibilities each format holds, highlighted through detailed discussions and demonstrations. Our expert speakers will dissect the key aspects and provide critical takeaways for effective use, ensuring attendees leave with a thorough understanding of how to apply these formats in their own projects. Elevate your understanding of how FME supports these cutting-edge technologies, enhancing your ability to manage, share, and analyze spatial data. Whether you're building on knowledge from our initial session or are new to the serverless spatial data landscape, this webinar is your gateway to mastering cloud-native formats in your workflows.

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Safe Software

This presentation explores the impact of HTML injection attacks on web applications, detailing how attackers exploit vulnerabilities to inject malicious code into web pages. Learn about the potential consequences of such attacks and discover effective mitigation strategies to protect your web applications from HTML injection vulnerabilities. for more information visit https://bostoninstituteofanalytics.org/category/cyber-security-ethical-hacking/

HTML Injection Attacks: Impact and Mitigation Strategies

Boston Institute of Analytics

MINDCTI Revenue Release Quarter One 2024

MIND CTI

Scaling API-first – The story of a global engineering organization Ian Reasor, Senior Computer Scientist - Adobe Radu Cotescu, Senior Computer Scientist - Adobe Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe

apidays

Kürzlich hochgeladen (20)

Strategies for Landing an Oracle DBA Job as a Fresher

Artificial Intelligence: Facts and Myths

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

Real Time Object Detection Using Open CV

The 7 Things I Know About Cyber Security After 25 Years | April 2024

presentation ICT roal in 21st century education

Understanding Discord NSFW Servers A Guide for Responsible Users.pdf

Automating Google Workspace (GWS) & more with Apps Script

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

Manulife - Insurer Innovation Award 2024

Boost PC performance: How more available memory can improve productivity

🐬 The future of MySQL is Postgres 🐘

Apidays New York 2024 - The value of a flexible API Management solution for O...

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

HTML Injection Attacks: Impact and Mitigation Strategies

MINDCTI Revenue Release Quarter One 2024

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe

Mayur Thakur, Managing Director, Goldman Sachs, at MLconf NYC 2017

1. Surveillance Platform for Bank Compliance Mayur Thakur, Goldman Sachs

2. 2 • “Banks pay out £166bn over six years: a history of banking misdeeds and fines” – The Guardian • “Banks 'pay 60%' of profits in fines and customer payments” – BBC News • “Deutsche Bank to Pay $2.5 Billion to Settle Libor Investigation” – The Wall Street Journal • “$1.2 Billion Fine for Hedge Fund SAC Capital in Insider Case” – NY Times Stakes Are High

3. Key Technical Challenges  Diverse data sets and formats (sql, flatfiles, proprietary, etc)  Size of data, updated frequently • ~1B* pieces of text per year • ~1B edges in a graph • 100s of millions of trading events in a day  Data from past can change (e.g., manual trade correction) • Causes a cascade of changes  Surveillance decisions need to be debuggable • Why was trade X on Oct 25, 2015 not flagged?  Not real time; often need time guarantees (say, T+1) * All numbers are “orders of magnitudes” 3

4. Surveillance Architechture 4 SQL 1 Surv. 1 SQL n Flatfile 1 Flatfile m Prop 1 Prop k HDFS 1 HDFS q Flattened 1 Flattened 2 Flattened n Preprocessing pipeline 1 Preprocessing pipeline 2 Preprocessing pipeline n Alerts… Bookkeeping

5. Spoofing Illustration 5

6. A Real World Spoofing Case Navinder Singh Sarao was accused of spoofing  …and even contributing to the flash crash of 2010 Sarao pled guilty to spoofing in Nov 2016 He allegedly made $40M in illegal profit over years. 6

7. Review of Regulatory Cases 7 – Analyzed six regulatory enforcement cases for related to spoofing – Identified common factors indicative of spoofing behavior • Creating false impression of demand by placing spoof orders on opposite side to trigger a price movement (“order imbalance”) • Cancellation of spoof orders within short time after pivot execution (“time to cancel post execution”) Case Factors Order Imbalance Time to Cancel Post Execution ( > 2.5 times ) ( < 1 sec ) Sarao / Flash Crash a a Hold Brothers a a Coscia/ Panther a a Visionary Trading NA a Swift a 5 secs 3 Red a a

8. Transactions Data Pipeline  Spoofing implementation has 2 parts: data preprocessing and surveillance logic  Data preprocessing pipeline is reused for multiple surveillances  ~100M orders, 1B mkt data points, 100K products, multiple order mgmt. system 8 Order 1 Related Transactions Spoofing Orders n Exec. 1 Exec m Market 1 Market k Account Product Flattened Order Flattened Exec Flattened Market Order Processing Pipeline Exec. Processing Pipeline Mkt. Processing Pipeline Front running Surv. n… Alerts Alerts Alerts…

9. Related Transactions Table 9 Related Transactions Pivot Exec Orders Execs/Cancels MktData 216.8, 216.9, … One row of the related transactions table contains information about one pivot execution and all the activity around the time of that execution. X X X X

10. Search Problem 10 Given a semi-structured corpus of about a 1B documents in a hadoop cluster, design a search engine over YARN that is fast and satisfies the investigative needs of a variety of users. Unique Challenges  Cannot move data outside of an already existing hadoop cluster  Support deep scoring algorithms specifically for GS-specific signals (colloquial language, trades, etc)  Unstructured and structured signals

11. Search Workflow 11 Search Master Ranker Fast Index Servers Slow Index Servers HBase Web Client Yarn containers HDFS • Implemented as YARN apps • Auth enabled • Slow index Servers can scale as much as HBase # indexed documents > 1Billion # indexed tokens > 500 billion Current Index Size Runs in several TBs (Memory and Disk)

Mayur Thakur, Managing Director, Goldman Sachs, at MLconf NYC 2017

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Andere mochten auch

Andere mochten auch (19)

Ähnlich wie Mayur Thakur, Managing Director, Goldman Sachs, at MLconf NYC 2017

Ähnlich wie Mayur Thakur, Managing Director, Goldman Sachs, at MLconf NYC 2017 (20)

Mehr von MLconf

Mehr von MLconf (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Mayur Thakur, Managing Director, Goldman Sachs, at MLconf NYC 2017