Talk I gave on infrastructure as development at the 2009 Red Hat Summit. Slides are rapid-fire. It was only a 45 minute talk.

1. Infrastructure is Development Michael Stahnke (stahnma) 02-SEP-2009

2. The ideas presented today are not representative of my employer, business success, past jobs and do not offer endorsement to any particular products brands or companies. Heck, a lot of this stuff isn't even my idea to begin with. http://www.flickr.com/photos/sea-turtle/3049443478/ http://www.flickr.com/photos/jjze/726682393/

4. Who Am I?

5. Who Am I?

6. Who Am I?

7. Who Am I?

8. Who Am I?

9. Who Am I? <Insert Large Company Name Here>

11. http://www.stahnkage.com

12. @stahnma on Twitter

13. @stahnma on identi.ca

14. stahnma on github

15. [email_address]

16. stahnma on Freenode IRC

25. The Problem

26. System Administration isn't a science

27. You can't get a degree in System Administration

28. Heterogeneous Systems

29. No Clear Job Description

30. Expectations not Clear

31. System Admins are @ holes

32. Can't keep up with demand

33. Admins fight management http://www.flickr.com/photos/jjze/726682393/

34. Solutions

35. Are delivered Solutions

36. Through Ideas Solutions

37. Put together by Good People Solutions

38. Using some awesome technologies Solutions

39. Topic 2 Infrastructure Practices

40. Why is your infrastructure special?

41. Steal Ideas http://www.flickr.com/photos/purplemattfish/3508761485/

42. Open your infrastructure

43. Opening your infrastructure includes your issues

44. Open your infrastructure and your solutions

45. Well, what do you mean?

48. LDAP Directory

49. Hardware Management Tools

50. Power Management Tools

51. Monitoring Tools

52. Provisioning Tools

53. Storage Management Tools

54. Backup Tools

55. Policy Engines

56. Patch Tools

57. Security Scanning Tools

58. Virtualization Management Tools

59. Log Management Tools

61. LDAP Directory – RHDS, 389, SunOne, AD, OpenLDAP

62. Hardware Management Tools – IBM Director, DRAC

63. Power Management Tools – APC

64. Monitoring Tools – Nagios, Tivoli, OpenView

65. Provisioning Tools – Cobbler, Vmware,

66. Storage Management Tools – IBM Whatever, Some Custom Stuff

67. Backup Tools – Netbackup, Tivoli, Networker, Tar, Gzip, Rsync, Cron

68. Policy Engines – Cfengine, Puppet, Scripts

69. Patch Tools – RHN, NIM, Custom Repos

70. Security Scanning Tools – Lots of stuff

71. Virtualization Management Tools – vCenter, Virt-Manager, Spacewalk, RHN

72. Log Management Tools – Syslog Server

73. DNS – Bind

80. Infrastructure is Development

81. The Infrastructure is the Application

82. Application == Infrastructure

85. Make the infrastructure an investment, not a cost

88. Automation

90. Automation

91. The right mix of people

93. Automation

94. The right mix of people

95. The right decision processes

97. Automation

98. The right mix of people

99. The right decision processes

100. Vision

102. Automation

103. The right mix of people

104. The right decision processes

105. Vision

106. Known State

107. Where do we start?

109. Your Infrastructure isn't a secret

110. Your Infrastructure isn't the secret sauce

111. Your Infrastructure isn't differentiating

113. Somebody has probably solved this problem

115. IRC

116. Sourceforge

117. Ohloh.net

127. I have/need a purchased proprietary solution for Problem X

129. Software/Tool Selection

133. Performance

135. Performance

136. Functionality

138. Performance

139. Functionality

140. But really....

141. Meatcloud Manifesto

142. Cloud Computing is all the rage

148. Choose your technologies wisely

149. Very wisely

152. The provided API must have all functionality that the application provides.

154. The provided API must have all functionality that the application provides.

155. The provided API must be tailored to more than one language and platform.

157. The provided API must have all functionality that the application provides.

158. The provided API must be tailored to more than one language and platform.

159. Source code counts as an API, and may be restricted to one language or platform.

161. The provided API must have all functionality that the application provides.

162. The provided API must be tailored to more than one language and platform.

163. Source code counts as an API, and may be restricted to one language or platform.

164. The API must include functional examples and not require someone to be an expert on the product to use.

167. All data stored in the product must be machine readable and writable by applications other than the product itself.

169. All data stored in the product must be machine readable and writable by applications other than the product itself.

170. Writing work-arounds to cover the deficiencies in a product should be less work than writing the product’s designed functionality.

172. All data stored in the product must be machine readable and writable by applications other than the product itself.

173. Writing work-arounds to cover the deficiencies in a product should be less work than writing the product’s designed functionality.

174. But I don't have automation specialists

175. That's a problem.

176. Do you have developers?

178. What's the difference between development and automation?

179. Process?

180. Requirements Gathering?

181. Cost?

182. Testing?

183. Egos?

184. Funky Lava Lamps?

185. Design Patterns?

186. Code portability?

187. What's the difference between development and automation?

190. People make the Infrastructure Agile

191. Attract Talent People

192. Retain Talent People

199. You want rock stars remember?

212. Big Picture thinkers.

214. Big Picture thinkers.

215. Visionaries.

217. Big Picture thinkers.

218. Visionaries.

219. Holistic View.

222. Tune like it's going out of style.

224. Tune like it's going out of style.

225. Maximum ROI.

227. Tune like it's going out of style.

228. Maximum ROI.

229. Deep understanding of the technology.

233. How many people is the right amount?

234. 50:1?

235. 100:1?

236. 200:1?

237. 2:1?

238. Gartner Says:

239. Gartner Says: “Yes”

240. There is one rule for Server/Admin Ratio:

243. We've Covered Technology Selection

244. We've Covered People Decisions

245. Let's cover Community

247. Code for the generic case

248. Stop reinventing the wheel, srsly.

260. Portability of solutions http://www.flickr.com:80/photos/thomas-merton/255204957

262. Portability of solutions http://www.flickr.com:80/photos/thomas-merton/255204957

264. Portability of solutions

265. Ability to hire knowledgeable individuals http://www.flickr.com:80/photos/thomas-merton/255204957

267. Portability of solutions

268. Ability to hire knowledgeable individuals

269. Able to retain rock stars http://www.flickr.com:80/photos/thomas-merton/255204957

271. Portability of solutions

272. Ability to hire knowledgeable individuals

273. Able to retain rock stars

274. Community built around tools and support http://www.flickr.com:80/photos/thomas-merton/255204957

277. Your Time is Valuable

278. Waste less of it

279. Your team's time is valuable

280. Manage it

282. Account Management

283. Ad-Hoc File Transfer type activity

284. Patches http://www.flickr.com:80/photos/denaldo/3326738675/

286. Parties/Showers

287. Volunteer Stuff

289. See Also: People

291. Focus on one thing only

292. Seriously, this will help

293. You're ready to automate.

295. OK, maybe you're not ready to automate.

296. Don't automate bad process. http://www.flickr.com:80/photos/houseofcards/94935538/in/photostream/

297. You get bad results.....faster.

298. Bad processes deliver bad results.

299. You don't need a panflute.

301. It starts with requirements

303. Do you need a tool?

304. Do you need it automated?

305. Do you need a pan-flute?

332. Requirements http://www.flickr.com/photos/usonian/106005435/

335. 2 CPUs

336. 4 GB RAM

337. 100G Disk Space

338. RHEL 5 http://www.flickr.com:80/photos/jimfrazier/1480809458/

341. An application team should be able to see the status of their requests.

343. An application team should be able to see the status of their requests.

344. A capacity planner should approve the request

346. An application team should be able to see the status of their requests.

347. A capacity planner should approve the request

348. A system should have multipath from RHEL 5.3 installed

350. An application team should be able to see the status of their requests.

351. A capacity planner should approve the request

352. A system should have multipath from RHEL 5.3 installed

353. A physical system should have a bonded network connection

367. You will iterate again.

374. Testing

384. Specifications for Testing http://www.flickr.com/photos/fireflythegreat/2845637227/

390. Can I script this?

399. Test Case #!/bin/bash . config_vars.sh rc=0 # First, if we a VM exit all good if ( lspci | grep -i vmware &> /dev/null ) ; then rc=0 else if ( ! ifconfig -a | grep -i bond0 &> /dev/null ) ; then echo &quot;FAIL: Network interfaces not bonded.&quot; rc=1 fi fi exit $rc

400. Test Case #!/bin/bash . config_vars.sh rc=0 # First, if we a VM exit all good if ( lspci | grep -i vmware &> /dev/null ) ; then rc=0 else if ( ! ifconfig -a | grep -i bond0 &> /dev/null ) ; then echo &quot;FAIL: Network interfaces not bonded.&quot; rc=1 fi fi exit $rc Site Specific Data

401. Test Case #!/bin/bash . config_vars.sh rc=0 # First, if we a VM exit all good if ( lspci | grep -i vmware &> /dev/null ) ; then rc=0 else if ( ! ifconfig -a | grep -i bond0 &> /dev/null ) ; then echo &quot;FAIL: Network interfaces not bonded.&quot; rc=1 fi fi exit $rc Site Specific Data Decoupled Logic

402. Test Case I have results, where do I put them?

403. Pick One

406. Asking for/updating configuration info

407. Authentication Modules

420. Host has the correct Version of Multipath

421. Host should have 4 GB RAM

422. Host should have 2 CPUs

423. Host must have a network bond if it's physica

437. Self-test the Build

438. Commit early, commit often

439. Have a test environment

440. Report Progress for everybody (laconica?)

441. Automate Deployment – (usage packages)

443. Plan change-sets

444. Plan your work-life balance

445. Work with setting expectations for consumers of your infrastructure

446. Work on hot/rolling releases

447. Communicate like a politician, except tell the truth

459. Go From This:

460. To This:

464. http://www.boingboing.net/2007/08/22/panflute-flowchart.html

465. http://gilesbowkett.blogspot.com/2007_05_01_archive.html

466. http://www.verber.com/mark/sysadm/how-many-admins.html

468. http://gist.github.com/161265

469. https://fedoraproject.org/wiki/Infrastructure

470. https://fedorahosted.org/csi/

471. Ship It – Pragmatic Press

472. Automating Unix and Linux Administration – Apress

473. Pro Unix Administration – Apress

474. Pulling Strings with Puppet - Apress

475. Release IT – Pragmatic Press

477. http://www.melconway.com/research/committees.html

478. http://status.net/?source=laconica

479. http://reductivelabs.com/trac/puppet/

480. https://fedorahosted.org/cobbler/

481. http://www.mockobjects.com/book/

482. Infrastructure is Development