Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Â
Email architecture using SES
1. Email
Architecture
using
Amazon SES
1
Deliverability and Scalability
2. Introduction
⢠Email is a method of exchanging
digital messages from an author to
one or more recipients
⢠Email is one of the most critical
component in an application for
communication between the
application and its users
⢠Sending effective emails help
business to bring in more people to
use their sites and to share
information that helps the end
users 2
3. How Email is sent?
⢠Internet was built on
trust and email also
works in the same
fashion
⢠No authentication
exists between ISP to
send emails
⢠SPAM abuse is high as
this trust is misused
3
4. What ISPâs are doing?
⢠ISPâs nowadays actively
block IP ranges that
they believe generate
SPAM
⢠ISPâs employ a range of
software to categorize
email as JUNK, SPAM,
Phishing, etcâŚ
⢠Countries are actively
legislating to punish
offenders 4
5. What companies are worried about?
⢠Email deliverability
⢠Ensure email reaches
the Inbox of the
recipients
⢠Handle ISP reports
⢠Complaints
⢠Bounces
⢠Statistics
⢠Keep delivery costs low
5
6. Deliverability
⢠Refers to the likelihood of the email reaching the intended
destination
⢠Majority of global email traffic is either unsolicited or undesired
bulk email
⢠88 to 92% of email traffic in year 2010 were spam
⢠Due to the security measures of ISP, even genuine emails are
marked as SPAM causing the companyâs reputation to be
downgraded from a users perspective
⢠Users are nowadays extremely cautious about clicking any links in
email that are marked as SPAM 6
7. ISP reports
⢠ISPâs respond to origin with reports on emails that were
processed by the systems
⢠Bounces â Messages sent to non existent email address
⢠Rejections â Size limitations and attachment types
⢠Complaints â Complaints about email content, etcâŚ
⢠Email systems need to handle these responses and need to
proactively handle the issue
⢠Ignoring reports can force an ISP to block the IP address of origin 7
server causing email outage
8. Statistics
⢠Email Statistics are an essential component of a business
application to understand the reach of emails
⢠Successful deliveries
⢠Click tracking
⢠Failure reports
⢠The above statistics are vital to understand the success of email
marketing and campaigns
⢠This data adds business benefit to ensure that the customer can
reach your system in case of emergency
⢠Unable to deliver newly generated passwords since the users email 8
box is closed
10. Amazon Simple Email Service
Features Benefits
⢠Simple Setup ⢠Simple
⢠Zero Infrastructure ⢠Inexpensive
⢠Bulk & transactional email-
sending service ⢠Reliable
⢠Standard SMTP service ⢠Scalable
⢠DKIM support ⢠No upfront costs
⢠API availability ⢠Workflow Integration
10
11. About SES
What is it What its not
⢠Its an email delivery service ⢠Not a replacement for your
for bulk and transactional exchange servers. It doesnât
system receive emails
⢠Not a marketing tool for
⢠Supports both SMTP and API sending bulk messages and
based access track status. Youâve to build
the system on top of SES
⢠Offers support for setting up
DKIM and signing messages ⢠Not a guaranteed email
service that handles
everything for you. It helps
⢠Offers endpoint to receive you to build a guaranteed
information on email delivery delivery system.
11
12. How SES helps business?
⢠Offers easy interface to help with complex setups like DKIM, etc..
⢠Offers a scalable, highly available service for sending emails
⢠Offers feedback loops to handle bounces and complaints
⢠Offers a no CAPEX model for your email delivery setup
⢠Automatically filters your content to prevent SPAM messages
from being sent out if your infrastructure is compromised
⢠Requires no changes to existing applications since SES is 12
compliant with SMTP protocol
13. DKIM Signing
⢠DomainKeys Identified Mail (DKIM) is a standard that allows
senders to sign their email messages and ISPs to use those
signatures to verify that those messages are legitimate and have
not been modified by a third party in transit.
⢠DKIM signing helps in deliveryability of email messages as ISPâs
can verify the senderâs domain for authenticity.
⢠DKIM Setup in SES can be done in 2 ways
⢠Easy DKIM using the SES console
⢠Manual DKIM signing by using the SES REST API
13
14. Easy DKIM Signing
1
⢠Verify a new Domain
⢠Generate DKIM Settings
⢠Add the DNS records with
your registrar
⢠1 TXT record 2
⢠3 CNAME records
3
14
15. Error Handling
⢠SES offers endpoints to handle the following errors:-
⢠Bounces
⢠Complaints
⢠SES can be configured to handle errors in the following
ways:-
⢠SNS topics (HTTP endpoints)
⢠Email Forwarding
15
16. Send Rates
⢠Depending on the needs of the application, SES throttles the
maximum messages that you can send in a second
⢠Adherence to this rate is essential to ensure SES service
availability
⢠You can request the SES team to increase the rates if your
application needs are more
⢠Send rates improve based on email quality and decreases based
on JUNK/SPAM/Bounces
16
19. Delivery Workflow
⢠Application logs a queue message for sending email
⢠Email service polls the queue to get the message
⢠Locks the queue message as being processed
⢠Parses the message to identify the message and recipient
⢠Calls SES to send the email message to the recipient
⢠Deletes the message from queue on success
19
⢠Marks the message as unread if failure in SMTP connectivity
20. Email Service
⢠Independent Tier to handle Email delivery
⢠Can be multi threaded / multi instance for scale
⢠Distributed architecture for handling failures
⢠Has logic to handle the following
⢠Log bounces to prevent resending to an non existent address
⢠Control send rates based on configuration
⢠Failover in case SES is down to switch to a different SMTP service
⢠Can be independently tested
20
⢠Can offload logic to generate email messages based on a email
event â template pattern
21. Bounce Handling
⢠SES receives the bounce message from ISP
⢠SES sends the message to the SNS topic configured for bounces
⢠SNS sends the message to the subscriptions
⢠Can be an email address to manually handle the bounces
⢠Can be an HTTP endpoint for an application to process the message
immediately
⢠Can be an SQS queue for distributed processing (Current Architecture)
⢠Email Service polls the SQS bounce queue to receive the bounces
⢠Takes appropriate actions based on the setup
⢠Mark the email address as non functional in database
⢠Maintain its own store of failed email address (Validates against this
every time when an email needs to be sent out)
⢠Sends an alert to the userâs alternate email address about the 21
applications inability to reach the user
22. Load Testing your service
⢠The email service can be independently tested as it is not directly
linked to the main application
⢠Email service capabilities like bounce handling can be tested by
using the new SES mailbox simulator
⢠You can read more about the mailbox simulator here
⢠http://aws.amazon.com/about-aws/whats-
new/2012/10/03/amazon-simple-email-service-announces-
mailbox-simulator/
22
⢠Using the mailbox simulator will not affect your SES reputation
23. SES safety measures
⢠SES is a cost-effective email delivery service
⢠Being cost-effective makes it attractive for spammers
⢠Bounces are unavoidable in an internet accessible application that
allows registrations.
⢠SES evaluates your reputation on the following basis:-
⢠Bounce Percentage
⢠Complaints received
⢠Quality of emails
⢠SES takes these parameters seriously and can block production access
when SES service is abused
23
⢠As long as the bounce percentage and the quality of emails are good,
SES is a cost-effective service for your emailing needs