For large financial institutions, it can be extremely hard to predict when your architecture may need to scale to process millions of financial transactions per day. HSBC addressed this challenge by integrating its on-premises mainframe with AWS services such as AWS Lambda, Amazon Kinesis, and Amazon DynamoDB. This integration enables the bank to engage in real time with millions of retail banking customers in a more personal, dynamic, and useful way. The bank applies business logic to its transaction data, and it harnesses the information it gleans to communicate directly with customers through a messaging platform that runs on AWS. In this session, we share an architecture pattern that demonstrates how retail banks can add value by investing in their legacy system when integrating streaming data from on-premises systems to an event-driven, serverless architecture at scale.

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How HSBC Uses Serverless to Process
Millions of Transactions in Real Time
Mainframe to Mobile In Near Real-Time using Serverless Technologies
Srimanth Rudraraju
Lead Digital Solutions Architect
HSBC
S e s s i o n I D : F S V 3 0 5
Santiago Freitas
Principal Solutions Architect
AWS Global Financial Services

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Objective
• Provide a reference architecture to build a distributed system leveraging
serverless services
• Share the lessons learned so you don’t make the same mistakes we did….
Instead you make new ones 

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Agenda
• HSBC Overview
• Customer Centric Communications
• Realtime Serverless Event Processing
• Design Considerations and Lessons Learned
• Monitoring and Alerting at Scale
• Key Takeaways

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
HSBC - The Worlds
Leading International
Bank

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The Diversity of our Business makes Technology
Complex
Multiple
banking
platforms
Geographically
dispersed
people and
systems
Highly
regulated
operating
environment
Rapidly
evolving
customer
needs and
expectations

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
HSBC Digital
• Simplification
• Innovation
• Better Customer Experiences
delivery
velocity value and
insights
more quickly
engaging
experiences

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Reasons to communicate
We’re here to make customers’ lives simple, so they can focus on what
matters

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Solution Overview
HSBC UK
Mainframes
Mapper
EMR
Spark
Kinesis
StreamsDirect
Connect
Customer Preferences
DynamoDB Lambda API Gateway
Data Service
AuroraEMRDynamoDBAPI GatewayKinesis
Streams
Event Engine
Kinesis
Streams
Lambda
Push Notifications
Notification Service
API GatewayKinesis
Streams
Lambda
Message Service
API GatewayDynamoDBKinesis
Streams
Lambda
JSON
ASCII
Dead Letter Queues
SNSSQSVPC CloudWatch KMS
Common Services
EU-West-1
AVRO
EBCDIC
Kafka
AVRO
EBCDIC

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Kinesis Data Streams (KDS) key concepts
• Shard: base throughput unit of a stream.
Contains an ordered sequence of records
ordered by arrival time.
• Data Stream: logical grouping of shards.
• Partition Key: a identifier specified by
data producer used to route data records
to different shards.
• Producer: sends records to a stream and
assign partition keys to records.
• Data Record: composed of a sequence
number, partition key, and data blob.
• Consumer: retrieves data from a shard in
a stream.
Lambda functions
Lambda functions

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Lambda with KDS - Scaling Behavior
• Number of Kinesis Data Streams shards is the unit
of concurrency
• E.g. 10 shards = 10 concurrent executions
• FIFO behaviour is per shard
• If there are no records in a shard, the respective
execution environment may go cold

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Lambda with Kinesis Data Streams
• Subscribe Lambda functions to automatically read batches of records off your
Kinesis Data Streams stream. Lambda polls the stream.
• When (not if) exception occurs, the shard is blocked but records from the other
shards do not throw errors and will proceed as normal
• No default Dead Letter Queues (DLQ) – you need to build one
Any Endpoint
Remote Call

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
We need a repeatable unique correlation ID
Infosphere CDC Kafka
Avro
Payload =
ID +
Transaction Data
NiFi
Avro
Payload =
ID +
Transaction Data
“Throughout the system we use the ID generated at source to track a transaction end to end”
JSON
Kinesis Event
Lambda
Processor
JSON
put-record
Kinesis Data
Streams (KDS)
{
"Data": ID + Transaction Data,
"PartitionKey": "ID",
"StreamName": "name"
}

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Processing records at most once and at least once
Kinesis Data
Streams (KDS)
Function Invocation
“ID”
Service State
Table
{
id: “a8098c1a-f86e-11da-bd1a”,
id_state: “processed”,
processed_tstamp: “1538747486”
}
Batch
Kinesis Data
Streams (KDS)
Shards Shards
SNSSQS
Error
Replay
Same pattern with “Service State Table”
repeated within each service

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Lambda and Kinesis Data Streams Lessons Learned
• Increasing number of Kinesis Data Streams shards may not increase system
performance, batch size matters. Perform load test.
• Consider the impact of language and VPC usage on Lambda startup time vs. Lambda
execution time
• Java-based functions start slower vs. Python/Node but executes faster
• 3GB memory isn’t always fastest for VPC attached Lambdas. Most optimum mem
allocation for Java-based functions was 1GB. Consider ENI-reuse.
• Consider pre-warming VPC attached functions to achieve your latency SLA

20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Providing internet access in a highly controlled way
Internet
Proxy Fleet
Network Load
Balancer
EU-West-1-A
Internet
Proxy Fleet
EU-West-1-B EU-West-1-C
Internet
Proxy Fleet
Auto Scaling / Endpoint Service
VPC
Endpoint
Application
Lambda
security group
Internet
gateway
• Lambda can optionally be attached to a VPC. At HSBC all Lambdas are within a VPC
• Internet access is controlled by an immutable sidecar proxy fleet with a restricted whitelist
• Design limits blast radius if any malicious code is deployed to a function
• Removed the requirement for VPC peering, simplifying routing tables and increasing isolation between VPCs
Notification
Service VPC

21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Loosely coupled network architecture to enable scale
EU-West-1-A EU-West-1-B EU-West-1-C
Platform VPC
Bank bound VPC
Endpoint
VPC subnet
1 X
…
VPC subnet
1 X
…
VPC subnet
1 X
…
• As VPC attached Lambda function scales, subnets must have available IP addresses to
match the number of ENIs = large CIDR block required to your VPC
• Access to on-premise provided via VPC endpoint which encapsulates a set of proxy servers
located on a VPC with Direct Connect = small CIDR used on VPC connected to on-premise
Large CIDR block
Bank Bound
Proxy Fleet
EU-West-1-A
Bank Bound
Proxy Fleet
EU-West-1-B EU-West-1-C
Bank Bound
Proxy Fleet
Platform DX VPC
Endpoint service
Network
Load
Balancer
Small CIDR block
HSBC UK
Direct
Connect

22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Networking lessons learned
• Have a network resource as part your cross-functional team, will save you time
• Consider isolating Lambda functions within a VPC with a large CIDR to ensure
that if you are scaling, you don’t run out of IP addresses
• Lambda functions attached to a VPC support Security Group and VPC Flow Logs,
your security team may require them
• Lambda Invoke API is not available as a VPC endpoint, caused functions
previously working outside a VPC to fail. Alternatively use API GW or Kinesis
Data Streams

23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Monitoring and Alerting
• Key Metrics
• Throughput
• Latency
• Errors
• System Health
CloudWatch
Custom Metrics
Out of the box
Metrics
Poll Data
• Key Challenges
• Calculating Latency across
serverless components
• AWS Tagging strategy to make
sense of metrics in DataDog
Dashboards
Alerts

25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Dashboards
• Automatically deployed via
pipeline alongside app code
• Logically grouped by app and
infra services
• Leverages anomaly detection
for smart detection and
interpretation

26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Key Takeaways
• Follow the principle of "extract data once and reuse multiple times” to power new
customer experiences
• Generating a repeatable correlation ID from source is critical in a distributed system
• Perform load tests to fine tune your system and identify choke points
• Know the AWS services soft and hard limits
• Plan your network architecture to provide service isolation and to support production scale
• Consider how to unify your existing and cloud operation model – logging, monitoring and
alerting

28. Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.