David Chen presents techniques for optimizing performance on Google App Engine. He discusses analyzing logs to identify inefficient requests and using AppStat to profile RPC calls. Key optimizations include batching and async RPCs, caching results, optimizing datastore usage by reducing indexes and entity size, and hosting dynamic content statically. Server-side caching and tasks are also recommended to improve control flow efficiency.

1. Google App Engine performance tuning
David Chen from Tagtoo
Saturday, 1 June, 13

2. About me
• Tagtoo Cofounder & Senior Engineer
• NCTU Computer Science M.S.
• UW. Computational Finance M.S.
• Interested in:
• Python
• Google Cloud Platform
• Ski
• Taipei.py ROCK!
Saturday, 1 June, 13

3. Google App Engine & Google Cloud Platform
Saturday, 1 June, 13

4. Introduction
• Google App Engine in 2007
• PaaS (Platform as a Service)
• Total Solution
• Scalable, Easy to use (really?)
• Automatic scaling and loading balance
Saturday, 1 June, 13

5. Introduction
• Google App Engine in 2007
• PaaS (Platform as a Service)
• Total Solution
• Scalable, Easy to use (really?)
• Automatic scaling and loading balance
Hint:我要打十個
Saturday, 1 June, 13

6. Google Cloud Platform Family
https://cloud.google.com/
Saturday, 1 June, 13

7. Google Cloud Platform Family
• App Engine becomes gateway to
• Google Computing Engine (as EC2)
• CloudSQL (as MySQL)
• Big Query (SQL for terabyte)
• Cloud Storage
• 2013 Google I/O
• Datastore Service
• Php Support ...
Saturday, 1 June, 13

8. Why We Use App Engine
• Really Easy to configure
• Really Easy to scale
• Never Few Failed
• Fully Python environment
• Real Reason:
• Too Lazy to learn a complex platform
• Cheap / Powerful / Easy if use it carefully
Saturday, 1 June, 13

9. Why We Use App Engine
• Really Easy to configure
• Really Easy to scale
• Never Few Failed
• Fully Python environment
• Real Reason:
• Too Lazy to learn a complex platform
• Cheap / Powerful / Easy if use it carefully
Saturday, 1 June, 13

10. Performance Tuning
Saturday, 1 June, 13

11. Environment Configuration
• Python 2.7
• Thread: Safe
• High Replication Datastore (HRD)
• webapp2 + jinja2
https://developers.google.com/appengine/docs/adminconsole/performancesettings
Saturday, 1 June, 13

12. Setting
• Setting Frontend Instance Class
• Configure the Scheduler
• Idle Instance
• Pending Latency
• Task queue control
• X-AppEngine-FailFast
• Backend Instance
• Discount Instance Hour (40% off)
Super Easy to Scale
Saturday, 1 June, 13

13. Performance Tuning #1: Log Analysis, find out
which kind of request need to be optimize
Saturday, 1 June, 13

14. Log Analysis
Saturday, 1 June, 13

15. Log Analysis
Saturday, 1 June, 13

16. Log Analysis
Saturday, 1 June, 13

17. Log Analysis
Saturday, 1 June, 13

18. Log Analysis
Saturday, 1 June, 13

19. Analysis Log in Details
• Export logs
• appcfg.py request_logs myapp/ mylogs.txt
• options:
--num_days
--end_date
--serverity=[0-4]
• appcfg.py --num_days=1 --end_date=2013-05-22 request_logs myapp/
mylogs.txt
Saturday, 1 June, 13

20. Log Parser
• Sqlite:
http://google-app-engine-samples.googlecode.com/svn/trunk/logparser/
logparser.py
• BigQuery:
https://code.google.com/p/log2bq/
MapReduce -> CloudStorage -> BigQuery
• MySQL:
http://github/lucemia/log2sql.git
Saturday, 1 June, 13

21. Query
• select path, count(*) as freq, avg(cpm_usd) as avg_cost from log group by
path order by avg_cost desc limit 10
• select path, avg(ms), count(*) from log where group by path order by avg(ms)
desc limit 10
Saturday, 1 June, 13

22. Query
• select path, count(*) as freq, avg(cpm_usd) as avg_cost from log group by
path order by avg_cost desc limit 10
• select path, avg(ms), count(*) from log where group by path order by avg(ms)
desc limit 10
Saturday, 1 June, 13

23. Performance Tuning #2: AppStat, find out how to
optimize request
Saturday, 1 June, 13

24. AppStat
• Profiling the RPC (remote procedure call) performance of your application.
• Datastore
• Memcache
• Url Fetch
• Mail
• ...
• AppEngine Charge for RPC
https://developers.google.com/appengine/
docs/python/tools/appstats
Saturday, 1 June, 13

25. AppStat
• Profiling the RPC (remote procedure call) performance of your application.
• Datastore
• Memcache
• Url Fetch
• Mail
• ...
• AppEngine Charge for RPC
https://developers.google.com/appengine/
docs/python/tools/appstats
Saturday, 1 June, 13

26. Install App Stats
App.yaml
appengine_config.py
Saturday, 1 June, 13

27. Demo App Stat
Saturday, 1 June, 13

28. What AppStat can tell
• Is your application making unnecessary RPC calls?
• Should it be caching data instead of making repeated RPC calls to get the
same data?
• Will your application perform better if multiple requests are executed in
parallel rather than serially?
Saturday, 1 June, 13

29. Performance Tuning #3: optimize rpc
Saturday, 1 June, 13

30. Method 1: Batch RPC
Batch Version
Saturday, 1 June, 13

31. Example: Batch RPC
Hint: Batch as much as you can (less than 500)
Saturday, 1 June, 13

32. Example: Batch RPC
Hint: Some function support batch mode
Saturday, 1 June, 13

33. Method 2: Async RPC
Saturday, 1 June, 13

34. Example: Async RPC
Hint: Read async need to control the flow carefully. fire the rpc as soon as
possible, get the result as late as possible.
Saturday, 1 June, 13

35. Example: Async RPC
Hint: For write, always try async, or try batch + async
Saturday, 1 June, 13

36. Method 3: Cache RPC result
RPC
RPC
Cached Version
Write to CacheRead Cache
Return if cache exists
Saturday, 1 June, 13

37. Example: Cache RPC
Hint: cache repeat rpc, check the hit ratio at Memcache Viewer
Saturday, 1 June, 13

38. Example: Cache RPC
Hint: memcache accept both async and batch operation
Saturday, 1 June, 13

39. Performance Tuning #4: optimize datastore
Saturday, 1 June, 13

40. Google App Engine Datastore
• Schemaless (NoSql) for large scale
• Need to manually configure index
• Support limited SQL command, but have different behavior
• offset
• No inequality on more than one property
• ...
• Support MapReduce (but expansive..)
Saturday, 1 June, 13

41. Datastore Index
• 太長了.. 有機會再講
Saturday, 1 June, 13

42. Datastore Pricing
• Write: 50000 free, $0.1 / 100k
Read: 50000 free, $0.07 / 100k
Small: 50000 free, $0.01 / 100k
• Looks Fine?
• In fact:
• Write could be much more expansive than you expected
Saturday, 1 June, 13

43. Datastore Pricing
Saturday, 1 June, 13

44. Datastore Pricing
Saturday, 1 June, 13

45. Datastore Pricing
Saturday, 1 June, 13

46. SAMPLE
• Get:
1 read ops = $0.07 / 100k
• Insert:
2 + 2 x 12 = 26 write ops = $2.6 / 100k
Hint: Each index property cost money
Saturday, 1 June, 13

47. SAMPLE
• Get:
1 read ops = $0.07 / 100k
• Insert:
2 + 2 x 12 = 26 write ops = $2.6 / 100k 37x
Hint: Each index property cost money
Saturday, 1 June, 13

48. How to optimize it?
Saturday, 1 June, 13

49. Redefined Model with “indexed=False”
• Get:
1 read ops = $0.07 / 100k
• Insert:
2 = 2 write ops = $0.2 / 100k
• Cannot query with property
with indexed=False
2x
Hint: know how you wanna query your data
before define the model
Saturday, 1 June, 13

50. ListProperty
Hint: ListProperty is useful, but could also be dangerous
Saturday, 1 June, 13

51. ListProperty + MapReduce
Hint: MapReduce + ListProperty could be more than dangerous...
Saturday, 1 June, 13

52. Entity Size
Hint: Entity size won’t affect the cost (and won’t affect the performance)
Saturday, 1 June, 13

53. Datastore Hints
• Make Table BIG!
• but only index if it is necessary
• Find Alternative Solution:
• CloudSQL + Cache
• query index and tree in memory
Saturday, 1 June, 13

54. Build index
https://code.google.com/p/google-app-engine-ranklist/
Hint: Binary Tree, Trie, etc.. Build complex data structure
Saturday, 1 June, 13

55. More Design Principle
• Denormalize is better than normalize
• Think about real user case
• MxM or Mxn or nxn?
• More read or more write?
• immutable data?
• relation or duplicate?
• Get or Query?
• ...
Zen of datastore
Saturday, 1 June, 13

56. Google App Engine - NDB
• Automatic caching
• In-Context Cache
• write through - Memcache Cache
• The StructuredProperty class, which allows entities to have nested structure
• Asynchronous APIs which allow concurrent actions (and "synchronous" APIs
if you don't need that)
• Watch Out: Different Async Behavior
Hint: use ndb
Saturday, 1 June, 13

57. Performance Tuning #5: optimize control
Datastore RenderLogic
Saturday, 1 June, 13

58. Use server side cache
Datastore RenderLogic
Update Cache
Return if cache exists
Saturday, 1 June, 13

59. Use server side cache
Datastore RenderLogic
Update Cache
Return if cache exists
Saturday, 1 June, 13

60. Use server side cache
Datastore RenderLogic
Update Cache
Return if cache exists
Saturday, 1 June, 13

61. Use Tasks for non-request bound functionality
• Offline Update
•
Datastore RenderLogic
Saturday, 1 June, 13

62. Host dynamic content as static
• Last-Modified, ETag
• Expires, max-age
• Expires
• Edge-Control
Saturday, 1 June, 13