Most folks ask - how easy is it to migrate to Cloud ? What are the downsides ? What expertise do i need ?
I did a fun mind-project of trying to migrate an existing monolithic software stack to Cloud. And, i realized pretty soon how easy it is to migrate to Cloud !
I chose to come up with a migration strategy for an imaginary image-sharing application running in a conventional data-center. And, came up with some requirements as to its up-time availability, world-wide access and future growth anticipations.
Disclaimer: Now, this is just a fun mind exercise, and none of the migration strategy discussed here are have either been endorsed or checked by Oracle or any of its affiliates.
2. Disclaimer : This article and its contents are neither endorsed by
Oracle, nor validated from a technical team. These are my own
personal views
Although i have used Cloud as a target migration
platform, none of the views here are endorsed by
Oracle , or any of its associates
3. Use Case
An existing application in Perl, running on BSD linux with SQL DB is used for
hosting an image sharing site
Is this a good candidate to migrate to Cloud ?
Users to upload and share photos with each other - using mobile or desktop
application, and web browsers
Photo processing such as resizing, cropping and thumbnail generation
required
Photo commenting an omnipresent feature
4. Constraints
Current app built on Perl , running on linux
Millions of active users (and growing)
99.997% uptime SLA
Global customer base
End users to be served in less than quarter of a second , globally
System has High Availability, and is scalable for future growth
5. Tasks - technical requirements
Image uploading
Quick : Fast network accessing backend storage
Secure : User authenticatication application (signed URLs, APIs)
Hash tables (metadata) of images persistently stored to track successful uploads, edition of
images
Storage
Inexhaustible storage: global, reliable
Processing
Scalable computing for image processing, transcoding
Serving
Fast and secure downloads around the globe
Scalable
Media Applications
Assimilation and integration of metadata with application domain data
Image management (e.g deletion, dedupe, sharing on social media)
Assumption : Current App (in Perl) will be used for media processing
Fast : Load Balancing, Content Delivery Caching
Reliable : Cluster, Load Balancing
6. Tasks: Mapping Cloud Platform Solutions
Tasks
Image upload
Storage
Processing
Serving
Media Application
Cloud Building Blocks
Application Layer:
Signed URL authentication for browsers,
cloud endpoint APIs for uploads from apps
Manages workflow code for upload, processing &
download of images
● Cloud Storage
○ Uses cloud network , fast and secure
○ Cloud CDN for edge caching
● Cloud SQL
7. Cloud Storage
Cloud SQL
Compute machine
Media Processing
(Perl on Linux)
App machine
Web Application
Authentication
Application WorkFlow
HTTP/S
request from
browser
Application
(Mobile and
Desk)
HTTP response
RESTful API response
Embedded Signed
URL
Image
upload
request
Cloud Storage
Access Info
Image upload via
signed URL
Network
Cloud Load Balancing
Network
Cloud Load Balancing
Metadata
Push
Metadata
Stored
Task
Queue
Images pulled in for
processing
Images pushed back
after processing
Image upload and Processing Schema
8. Images Upload and Processing Schema - Details
● Clients start uploading images in two ways : 1) HTTP from browser 2) Applications .
● App Machine (PaaS) authenticates these in two ways :
○ For Browser, give back a signed upload URL to Cloud Storage , embedded in HTTP/POST
○ For Apps, return Cloud Storage access information as protected Cloud Storage RESTful APIs via
Cloud Storage Endpoint Authentication
● The App machine is merely authenticating , and providing access to Cloud Storage. All images are
uploaded directly to Cloud Storage via Network in a fast, secure manner
○ Cloud Storage acknowledges upload via a response back (HTTP/API response)
● For successful uploads, image metadata is pushed to App machine application, which is then persistently
stored in Cloud SQL
○ App machine can receive metadata from browsers & Apps, or
○ Cloud Storage can notify App machine
● For processing media, the App machine creates a task on the task queue
○ Current Perl based Application, running on Linux based Compute machine(s) picks up the task from
task queue
○ The Compute Machines pulls in and pushes out media for processing as required from Storage
9. Points to consider - Image Upload and Processing
● Application data and image metadata can be stored individually, or jointly on either Cloud Datastore (NoSQL) or Cloud
SQL.
○ Depends on whether the its relational data or not
○ NoSQL provides better scaling for non-relational data (can be combined with Redis)
○ Either DB solution might benefit from memcache
○ Current technical resourcing and expertise to be taken into account
● Metadata for successful image uploads can be pushed to App machine via Cloud Storage (as shown, by using Object
Change Notification) or via individual browsers and end-point applications
○ If reliable internet connection at high speed is expected at end clients, latter might be more optimized due to
distributed metadata communication points
● For Image processing (such as thumbnail generation) , App machine can host Apps such as Images Go
● Fast image upload and access: App machine application plays a critical role in handling incoming traffic and download of
media. This will influence user experience greatly.
● Scaling :
○ Auto-scaling can be used for front end web-servers for elastic scaling by using Cloud Load Balancing , which
includes Single Cast IP address
○ For elastic scaling of Compute machine (hosting Media processing Perl application), App machine can be used to
monitor load on Compute machine, and spin up more Compute machines as necessary
■ Assumption : Perl code for Media Processing app is a stateless server
■ If above is not true, clusters of Media Processing servers can be provisioned from start and monitored for
load utilization or App machine apps can be considered as alternate
10. Cloud Storage
Cloud SQL
App machine
Web Application
Authentication
Application WorkFlow
HTTP/S
request from
browser
Application
(Mobile and
Desk)
Image
download
request
sent
Check rules
against
Metadata
Image download and sharing
Embedded Signed
URL
OAuth via
RESTful API
Network
with
Edge
Cache
App machine
Image Service App
Image resizing,
cropping
Image being served via cache or
from Cloud Storage
11. Image download and service schema - Details
● Clients start download and sharing requests via contacting App machine.
○ Mobile and desktop apps have RESTful API provided by the App machine application using Cloud
Endpoints.
○ Web interfaces have a Web UI provided by web-server frontend
● App machine looks up access information on Cloud Storage
○ For Web UIs, an embedded signed URL is provided to clients for secure and fast downloads
○ For Mobile or Desktop Apps , OAuth response provided via RESTful APIs
● App machine checks content sharing rules based on media metadata in Cloud SQL (or Datastore)
● Images are served fast via Cloud Network Content Delivery Network , which has caching capabilities near
the edge point
○ If image is not cached, its retrieved via Cloud Storage (and populated in Cache)
○ If image has to be resized or cropped, App machine Image Service App can be used
12. Migration overview
● Internal Teams generate maps of application dependencies, network ports, DNS requirements, and
network traffic
○ Load broken down by geographic regions and time pattern
● Generate initial specs based on compute power requirements (CPU, memory, database storage)
● Package up machine images using variety of resources available here
○ If Linux version(s) on Compute Machines needs to be updated as part of migration, new VMs
can be built and application code can be loaded in either standard or flexible Perl environments
● Configure Network rules to match internal systems
● Test :
○ Test minimum viable cloud for POC setup for functionality testing
○ (If applicable) Test against external and internal security audit standards
● Run load tests against the setup
○ Adjust and optimize as necessary
● Go Live !
○ Sync latest data via incremental backups-recovery or replication, divert network traffic, change
DNS
13. Next Steps
● Review functional spec and design
○ Current design is supposed to meet SLAs for service-time
■ More modifications such as utilizing memcache, SSDs , etc possible during testing phase
○ Review design consideration such as using Cloud SQL vs Datastore
○ Review architecture for separation of services like Image Services (which provides on the fly image
resizing, cropping, thumbnail generation) to App machines
■ Other option is to continue using them from current Perl Application
● Identify and document and additional software requirements (e.g, security, internal ticketing, paging
integration, etc)
● Start discovery phase of current application, as described in previous (Migration Overview) slide
○ partners available to assist in discovery, planning and migration
● Review SLAs provided by Cloud Platform here , identify any concerns
○ Clustering can help improve SLAs for uptime
● Review intrinsic full-stack monitoring and reporting mechanism by Cloud, and identify any further needs