This document summarizes a presentation about how Rent-A-Center deployed a scalable SAP Hybris e-commerce cluster using Docker on Amazon ECS. It discusses Rent-A-Center's evolution to DevOps practices over time, including moving infrastructure to code and using tools like Ansible. It then details the architecture designed for the SAP Hybris platform, including using ECS, Aurora, Lambda, S3, CloudFront and other AWS services. It also covers some of the challenges faced like Hybris node discovery and how they were addressed. Finally, it discusses the business and technical outcomes like increased availability, innovation through infrastructure as code, and achieving PCI compliance.
2. Today’s Presenter
Sr. Director, DevOps
Rent-A-Center owns 3,000 rent-to-own
retail stores for name-brand furniture,
electronics, appliances, and computers
across the U.S.
http://www.rentacenter.com
4. Team Members
Troy Washburn
James Lucas
Xiaolin Liu
Junhong Liu
Tyson Malik
Samprita Hedge
Ashay Chitnis
Nitin Ayyagari
Juan Mesa
Artem Kobrin
Ali Hussain
5. Outline
Evolution of DevOps at RAC
The e-commerce platform
○Business case
○Architecture
○Challenges and Lessons Learned
The outcomes
6. DevOps Timeline
2015 2015 2016 2016 Q4Q1 Q4 Q1
DevOps
Organization
at RAC
VAN Project on
AWS
Infrastructure as
Code/ELK Stack
eCommerce
project
launch eCommerce
Go-Live
Serverless
Computing
Oracle RDS
Migration
7. Business Case for VAN Project
• Secure B2B portal for our Acceptance Now business unit
which enables our partners to help grow their business
by increasing sales and expanding their customer base.
• PII data and PCI compliance requirements
8. First Success
Security: No last-minute surprises before go-live;
Least Privilege; RDS patching,
Centralized Logging, Threat protection,
Encryption at-rest and in-motion.
Availability: HA with multi-AZ solution; Auto-Scaling
Innovation: Infrastructure as Code, Agility and
Flexibility, Ansible playbooks as build
docs
9. Evolution: E-commerce Platform
Digital transformation:
Give our customers the
ability to rent online
Unified view of
customer
Self-service account
management
SAP Hybris selected
as the eCommerce
platform
10. Goals
Setup an SAP Hybris
ecommerce platform to
scale to 2 million users a
month
Ability to support
Black Friday traffic
Secure for PCI
Compliance
Stateless infrastructure -
HA across all components
including DR
Create an agile developer
workflow for rapid
execution
No downtime
deployment
Performance Scalability Security
High Availability Agility CI/CD
11. Outline
Evolution of DevOps at RAC
The e-commerce platform
○Architecture
○Challenges and Lessons Learned
The outcomes
12. Process
Phase 2: Attune
Phase 3:
Knowledge Transfer
Phase 1: Assess
Run the 2-week sprints
Transfer the knowledge at the end of each sprint
Understand the requirements and the current state, architect the desired
state, and create a punch list
18. Deploying Aurora DB with Hybris
Performance
Scaling
Low management
overhead
Use of AWS Aurora
DB instead of Oracle
or MySQL
Hybris supports
MySQL, Aurora
worked out of the box
Why? What? How?
19. Using AWS WAF (OWASP Top 10)
PCI-ready AWS WAF used to filter
traffic per rules
-CloudFront logs written to
S3
-S3 triggered Lambda
-Offending IPs were
blocked
Why? How?
To S3 and
ELB
Trigger
Lambda
Configure
rules
20. ECS Auto-scaling
Servicing seasonal
traffic patterns at high
performance and low
cost
ECS auto-scaling to scale individual services
Lambda function to auto-scale underlying ECS
nodes:
-Read stats from ECS
-Decide when to scale up/down -Trigger the
operation
Why? How?
21. ECS Autoscaling (Cont’d)
Read current
state of ECS and
ASG
Trigger Lambda
every 5 mins
let 0 … n be the running ECS services
let dck be the desired number of containers of service k
Let desiredCnt be the current desired number of instance in ASG
Let minCnt be the minimum number of instances needed in ASG
Let maxCnt be the maximum number of instances allowed in ASG
max ← MAX(dc0, .., dcn)
instanceCnt ← max + extraCapacity
If instanceCnt ≠ desiredCnt AND instanceCnt <= maxCnt AND
instanceCnt >= minCnt:
Update ASG desiredCnt to instanceCnt
Update Auto-Scaling Groups with new
desired instancesOur blog: https://aws.amazon.com/blogs/compute/amazon-
ecs-service-auto-scaling-enables-rent-a-center-sap-hybris-
solution/
22. Hybris Node Discovery
- Hybris nodes needs to be aware of each other
- Standard method (multi-cast) doesn’t work in VPCs
- Solution: Each Hybris process registers its IP:Port to
the DB
But, how does the process know its IP?
What?
23. Hybris Node Discovery (Cont’d)
Problem: Hybris can get the IP of the container it’s running in
but container IP is irrelevant. Need host IP.
Interim Solution: Wrote a startup script to get host IP using
EC2 metadata and passed on the IP to Hybris as a config
Better solution: Network Overlay (feature request to ECS
team)
24. Outline
Evolution of DevOps at RAC
The e-commerce platform
○Architecture
○Challenges and Lessons Learned
The outcomes
25. Outcomes
Business: Growth-driver, 360 degree customer view
Security: PCI Compliant ready, immutable infrastructure
Availability: HA with multi-AZ solution; Auto-Scaling
Innovation:
Infrastructure as Code
Agile and Flexible infrastructure
Automated delivery of infrastructure, code,
containers, and security rules
26. PCI Compliance
What? How?
The infrastructure is
expected to undergo a PCI
audit
Several Best Practices Applied:
Separate AWS accounts for Prod
SSO for AWS Console
IAM Roles for AWS Credentials
AWS account activity logged using CloudTrail
No VMs in DMZ (aka. Public subnets)
Multi-VPC, DirectConnect to on-premise
Immutable Docker containers with no human logins
DB credentials remain encrypted in S3 using KMS and
injected into app container via env on demand
All data encrypted at rest using EBS encryption
Encrypt web traffic using SSL from AWS Cert Mngr.
AWS WAF to block suspicious web traffic
Ansible/Docker to automate patch management
27. Summary
AWS evolution from EC2 instances, ECS Docker containers to
Serverless architecture
DevOps journey: X-As-a-Service, Infrastructure as Code, Micro-
Services, CI/CD
DevOps business drivers: lower TCO, faster release cycles
Digital transformation has enabled business to be more agile: speed to
market, greater stability and increased reliability