Я хочу рассказать о нашем опыте создания высоконагруженного REST API, который имеет следующие параметры:
* Интуитивно понятный
* Масштабируемый
* Отказоустойчивой
* Защита от атаки
* Высокая пропускная способность
Также я расскажу о том:
* как мы делали нагрузочное тестирование
* как мы пережили множество DDoS-атак
* как мы сделали Blue-Green deploy
Использовался следующий список технологий: .NET Core 2, ASP.NET Core 2, Consul, Fabio, Orleans.NET, Kafka, RabbitMQ, Serilog, xMetrics, Grafana, ELK
6. Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019
15+ years of
experience
Many
different
products
Like new
technologies
Launched
several APIs
About myself
12. Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019
Login API
Authenticate Fast response High throughput
Resistance to
DDoS and Brute
Force attacks
High Availability
(99.99 %)
13. Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019
Standard building blocks
DATABASE CACHE SEARCH INDEXES QUEUE
14. Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019
Application with stateless services
16. Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019
Stateless architecture: Pros
Services are easy to scale
No state in services
When service server is dead – a new one
could be up without any crucial affect
17. Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019
Stateless architecture: Cons
Add latency
Non efficient access to data
Limits scalability
• Master / slave replication
• Sharding
Concurrency
18. Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019
Stateless architecture with Cache: Pros
Add latency
Non efficient access to data
Limits scalability
•Master / slave replication
•Sharding
Concurrency
Cache invalidation
19. Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019
What we want
DATA LOCALITY EASY SCALING DEFENSE
20. Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019
Defence
Fast analyze of
input requests
In-Memory
statistics of
success/failed
requests
Calculation on the
fly
Fast and
serializable access
to the cache
21. Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019
Data locality
Moving compute to data is
typically faster than moving
data to compute
For low latency flow Data intensive service
23. Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019
Actors as the universal primitives of concurrent
computation.
Actor can:
• Receive messages
• Make local decisions
• Create more actors
• Send more messages
• Determine how to respond to the next message
received
Actor model
24. Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019
• .NET Framework used to create Scalable,
Distributed, .NET Applications
• Focused on low response latency and high
concurrency
• Usable in any .NET application (but frequently used
with WebAPI Applications)
• Based on a system of VirtualActors
Orleans
25. Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019
Virtual actors
Actor instances always exist, virtually
• Application neither creates nor deletes them. They never fail.
• Code can always call methods on an actor
Activations are created on-demand
• If there is no existing activation, a message sent to it triggers instantiation
• Transparent recovery from server failures
• Lifecycle is managed by the runtime
• Runtime can create multiple activations of stateless actors (for performance)
Location transparency
• Actors can pass around references to one another or persist them
• These are logical (virtual) references, always valid, not tied to a specific activation
26. Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019
• Grains are C# classes
• Derived from GrainBase
• Implement an Interface (e.g. IPlayer, ISession)
• Messages passing = Calling Interface methods
• Ex: AddLoginSession (int playerId, Guid sessionId)
• Can be Stateless or Stateful
Orleans: Grain = Virtual actor
27. Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019
Orleans: Grains
Grains: Individually
isolated objects that
are messaging
28. Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019
Orleans: Concurrency model for Grains
Orleans uses a
cooperative
multithreading
scheduler
Scheduler schedules
only one message at a
time for a grain
A message is processed
completely before
another message is
scheduled
A message is processed
as a sequence of one or
more turns
(continuations)
29. Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019
Transparent
scalability by
default
New Silos can
be added at any
time
Location
transparency
Multiplexed
communication
Efficient
scheduling
Orleans cluster
Grains
Silo
31. Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019
Smart cache
Most popular pattern
Use cases
Read-only, write-though or write behind cache
State usually is backed by persistence storage
Orleans solution
Actor per data item
Time-based and/or explicit validation
32. Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019
An actor which keeps track
of a set of other actors
• Does not intervene in iteration with the
individual actors
• Its own state is just a list of references
to other actors
Registry
33. Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019
Orleans grains
Session Grain
Player Grain
Online Grain
Operator Grain IP Grain
Login Grain
Cache layer Statistics layerOnline layer
Grain Cache Service
Operator Cache Grain
Player Cache
Grain
34. Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019
Cache pre-population on startup
Update cache on a regular basis (Kafka loader)
In-memory statistics of all login attempts
• Regulation specific data is being stored in DB
All logged in players are in memory
Time regulation features are based on Timer
Orleans usage
37. Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019
Protection & Security
HOW TO IDENTIFY
POSSIBLE FRAUD?
HOW TO PROTECT? WHAT TO ANALYZE? WHERE TO STORE DATA
FOR ANALYZING?
39. Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019
Cookie challenge: If the client supports cookies, we respond to an
HTTP request with a cookie. Web browsers typically will store and
resend this cookie. Most bots do not support cookies and therefore will
not respond.
JS cookie challenge: After receiving an HTTP request, we respond
with a JS cookie, instructing the browser to perform an action. Web
browsers typically will execute the JavaScript instructions, on the other
hand most bots do not support a JS engine and therefore will not
respond
Incapsula
40. Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019
Captcha
Send a CAPTCHA challenge,
expecting a human
response to the challenge
LoginAPI generates captcha
value for current session
and stores it inside Orleans
Use a non-white
background with
interspersed or roughened
41. Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019
Fraud detection
Analyze
IP addresses
User-Agent
Amount of failed login attempts per minute
Amount of success login attempts per
minute
42. Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019
Block players from the login base on the analysis of failed and success attempts per period
Blocks
43. Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019
Communication schema
Client
Login API
JWT Secret
Credentials
Anonymous JWT
Credentials
Anonymous JWT
OKAuth JWT