1. PHP & Continuous Data Processing Michael Peacock, October, 2011

2. No. Not milk floats (anymore) All Electric, Commercial Vehicles. Photo courtesy of kenjonbro: http://www.flickr.com/photos/kenjonbro/4037649210/in/set-72157623026469013

4. Web Systems Developer

5. Telemetry Team – Smith Electric Vehicles US Corp

6. Author

7. PHP 5 Social Networking, PHP 5 E-Commerce Development, Drupal Social Networking (6 & 7), Selling online with Drupal e-Commerce, Building Websites with TYPO3

8. PHPNE Volunteer

9. Occasional technical speaker

11. System Architecture

12. System Architecture

14. Un-predictable amounts of data at any one time

15. More vehicles rolling off the production line with telemetry enabled

17. If we are processing at full capacity, surplus builds in in the message queue

19. Secure: All data is encrypted and sent over SSL

20. Reliable: Huge investment in server infrastructure

21. Hosted: Backed up with an SLA

23. Data is rapidly converted into batch SQL files, which are imported to MySQL via “LOAD DATA INFILE”

24. Results in high number of inserts per second (20,000 – 80,000)

26. Pushing the capabilities of our SAN, tweaking configs where no DBA has gone before

27. www.samlambert.com

28. http://www.samlambert.com/2011/07/how-to-push-your-san-with-open-iscsi_13.html

29. http://www.samlambert.com/2011/07/diagnosing-and-fixing-mysql-io.html

31. We shard the data based on the truck it came from, each truck has its own database

33. Real Time Status and Tracking

36. Involved the use of Flash based “widgets” which called separate PHP scripts to query the data

37. Pages loaded relatively quickly

38. Data points took a little time to load

40. Memcachewas used when processing the data from the message queue, keeping a copy of the most recent of each data point for each truck

42. Give the registry or container details of memcache

45. Data doesn’t change every second, e.g.

46. Battery state of charge may take several minutes to loose a percentage point

47. Fault flags only change to 1 when there is a fault

48. Make an assumption.

49. We compare the data to the last known value…if it’s the same we don’t insert, instead we assume it was the same

51. Used key indicators and time differences to work out if/when the truck was off, and extrapolation should stop

54. The move from MySQL to Memcache reduces database load, but large number of requests still add strain to web server

56. Large number of concurrent requests per screenSession Locking Completely locks out a users session, as PHP hasn’t closed the session

57. Race Conditions: PHP & Sessions session_write_close() Added after each write to the $_SESSION array. Closes the current session. (requires a call to session_start immediately before any further reads or writes)

59. You can’t re-initialise your session once output has been sent

61. Certain exports, actions and processes take longer than 30 seconds, so the default execution time is longer

62. Initially the project lacked a single entry point, and execution flow was muddled

64. Motor RPM x Various vehicle specific constants

65. Calculated for every RPM value held during drive process

66. How much energy did the vehicle use

67. Battery Current x Battery Voltage x Time

68. For every current and voltage value combination held during the driving process

69. How well was the vehicle driven

70. Analysis of idle time

71. Harshness of accelerator and brake pedal usage

72. Inappropriate duration of AC / Heater on time?

75. Calculations involve holding this data in memory

77. Save it as a compiled report

79. Re-wrote to maintain only one connection per server, and switch the database used

80. Toggles to instruct the export to only apply for 1 of the servers at a time

82. Run PHP as a daemon

83. http://kevin.vanzonneveld.net/techblog/article/create_daemons_in_php/

84. Continually monitor for specific changes to memcache variables

85. Node.js

86. Light weight and fast

87. Give PHP another friend

89. Based on time – keep the individual tables smaller

90. NoSQL?

91. Currently investigating NoSQL solutions as alternatives

92. Rationalisation

93. Do we need as much data as we collect?

94. Abstraction

95. We need to continually abstract concepts and ideas to make on-going maintenance and expansion easier; especially in terms of mapping code to database shards

96. More hardware

97. Expand our DB cluster, more RAM, R&D

98. Design

100. Where data collection is critical, use a message queue, ideally hosted or “cloud based”

101. Hire a genius DBA to push your database engine

102. Make use of data caching systems to reduce strain on the database

103. Calculations and post-processing should be done during dead time and automated

104. Add more tools to your toolbox – PHP needs lots of friends in these situations

105. Watch out for Session race conditions: where they can’t be avoided, use session_write_close, a template engine and a single entry point