In this deck I describe twenty best practices for using external DSLs to develop software (also know as Model-Driven Development). The best practices are based on my personal experience, but they are also rated based on feedback I received from a number of colleagues to express the level of confidence experts have in the particular best practice. The slides cover three main aspects: language definition, model processing as well as process and organizational issues. Examples include the importance of notations, viewpoints and partitioning, the role of constraints, model transformations and testing as well as some thoughts on documentation, reviews and project roles. Putting twenty best practices into 5.000 words of course results in each best practice being quite brief. However, it also means each is highly condensed and to the point. I believe the slides remind developers of most of the critical aspects of using DSLs, and it can serve as a checklist when starting an MD* project.

1. MD* MD* Best Practices MarkusVoelter MD* Independent/itemis AG www.voelter.devoelter@acm.org

3. About Markus Voelter Independent Consultant

4. About Markus Voelter Independent Consultant Software Architecture DSLs & MDSD ProductLines

5. About Markus Voelter Independent Consultant http://www.voelter.de voelter@acm.org skype: schogglad

7. Read the Paper! http://www.voelter.de/data/articles/DSLBestPractices-Website.pdf

9.  DesigningLanguage  the

11. Sourcesforthe Language

12. Technical DSLs extractedfrom … framework … library … pattern (language) … architecture Language Sources

13. Business DSLs extractedfrom Language Sources

14. Domain Analysis Language Sources

15. Domain Analysis Language Sources Not just software!

16. Limit Expressiveness

17. Limit Expressiveness

18. Configuration select …fromconfigspace Limit Expressiveness … Properties … Feature Models

19. Customization compose … creatively Limit Expressiveness … Vocabulary … Sentences … Box and Line

20. Precision preciselywhat Limit Expressiveness … facts … declarative … domainexpertscan!

21. AlgorithmicCompleteness formallyhow Limit Expressiveness … automation … execution … in model processors … developers can!

22. Limit Expressiveness Use a 3GLifnecessary Generate APIs, Hooks

23. Do you want users to build their own Limit Expressiveness abstractions with the language?

24. DSL is a compromise: … Domain Abstractions … Reuse, Modularization, … Limit Expressiveness … All Data for Generation … DSL Tool influences Viewpoints?

25. Notation, Notation, Notation

26. Domain Users caredeeply Notation, … aboutnotation! „UI“ forthelanguage

27. Textual Graphical Form-Based Spreadsheet Notation, …

28. Parts Notation, …

29. Convertible Notation, …

30. Embedded Notation, …

31. Notation, … Tool Specific!

32. Graphical vs. Textual

33. Graphical Sequence/Flow Grapical vs. Textual

34. Graphical Grapical vs. Textual Relationships

35. Graphical Grapical vs. Textual Timing

36. Textual In all othercases! Grapical vs. Textual

37. Textual Real simple… …CVS/SVN Integration … Diff/Merge … Buildautomation … Model Migration Grapical vs. Textual

38. Text + Visualization Grapical vs. Textual

39. Text + Visualization … problem-specific … anwers specific questions … highlight specific aspects … several different Grapical vs. Textual visualizations

40. DSL Semantics

41. ? Whatdoesit all mean? DSL Semantics

42. DSL Semantics

43. DSL Semantics

44. DSL Semantics

45. DSL Semantics Documentation

46. Viewpoints

47. Viewpoints

48. Viewpoints

49. Well-defined DependenciesandConnectionPoints Viewpoints

50. Try toavoid overlapandtheneedforsynchronization. Viewpoints

51. Viewpoints Tool Specific!

52. Partitioning

53. scale MD* tools do not Partitioning arbitrarily!

54. Partitions Partitioning … separate resources … != logicalstructure … unitsof check in/out … processable separately … unit of IP

55. X cross Partition references Partitioning … lazy … byproxy … byname w/ linker

56. Partition Partitioning … not transparent … partoflanguage design … referencableelements … „includepath“

57. Partitioning Tool Specific!

58. Evolution

59. Whatto do withexisting models Evolution ifthe language changes?

60. MightRequire… Evolution … configmanagement … version tag in models … changetracking … migration M2M

61. Think about… Evolution … backwardcompatibility … deprecation … instrumentation … viewpoints + partitions

62. Evolution Tool Specific!

63. The Fallacyof GenericLanguages

64. FallacyofGeneric Lang.

65. Youcan model everything with FallacyofGeneric Lang.

66. Youcan model everything with FallacyofGeneric Lang. somehow!

67. Problem Shoehorningdomainabstractionsintothe genericlanguage FallacyofGeneric Lang.

68. Problem Sidetrackedbyexistingabstractionsandnotations FallacyofGeneric Lang.

69. FallacyofGeneric Lang. Very LimitedTool Support! Notations/ Abstractionsextensiblevia Profiles

70. Meta Model Complexity! FallacyofGeneric Lang.

71. Wherearestandardsuseful? FallacyofGeneric Lang.

72. Peoplehavetolearnunderlying concepts anyway. FallacyofGeneric Lang.

73. Is UML witha profilestill a standardlanguage? FallacyofGeneric Lang.

74. On whichmetaleveldo I wanttostandardize? M2 (UML), M3 (MOF)? FallacyofGeneric Lang.

75. Isn‘t a DSLbased on MOFas „standard“ as a profilebased on UML? FallacyofGeneric Lang.

76. Introduce an intermediate language FallacyofGeneric Lang.

77. separate DSL viewpoints UML + DSL DS Model FallacyofGeneric Lang.

78. FallacyofGeneric Lang. BPM is to analysts what UML is to developers.

79. Butdon‘treinventthewheeleither. FallacyofGeneric Lang.

80. Build your own languageinspiredbyexisting language FallacyofGeneric Lang.

81. Learnfrom 3GLs

82. DSL != 3GL Learnfrom 3GLs But:

83. Namespaces VisibilityEncapsulation Scoping Specialization Cohesion Coupling abstract … Wecan learn: Learnfrom 3GLs

84. Read thisBook: Learnfrom 3GLs Concepts, TechniquesandModels of Computer Programming by Peter Van Roy and SeifHaridi

85. Type Systems

86. Knownfrom3GLs Type Systems Classes & Objects

87. Examples: Components & Instances Type Systems Data Types & Instances Config Table Defs & ConfigTables

88. Viewpoint B (Types) Type Systems instanceof Viewpoint A (Instances)

89. Viewpoint B (Types) Type Systems instanceof Viewpoint A (Instances)

90. MetaMeta Model (M3) instanceof Viewpoint B (Types) MetaModel (M2) Type Systems instanceof Viewpoint A (Instances) instanceof Model (M1)

91. Support for ReuseandVariations

92. There‘smoreto reuse Reuse andVariations than partitions

93. The language must provide explicit Reuse andVariations support! (wecanlearnthatfrom OO)

94. Specialization overriding, overwriting Leaving Holes Reuse andVariations for variant tofill in InjectStuff severalplacesat a time?

95. Specialization Inheritance Leaving Holes Reuse andVariations Template Method Pattern InjectStuff Aspect Orientation

96. Reuse andVariations Makesureyou delineate the API!

97. Reuse andVariations Domain Users might not understand this!

98. Who are 1st Class Citizens?

99. Big Language Who are 1st Class Citizens withmanyconcepts!

100. Small Language Who are 1st Class Citizens withfew, but powefulconcepts!

101. Big Language Easierfor Business Users Concepts easy to find COBOL style Who are 1st Class Citizens

102. Small Language Technical DSLs Conceptsharderto find More expressive Lisp Style Who are 1st Class Citizens

103. Do not mix Reuse andVariations thetwostyles!

104. Teamwork Support

105. Versioning Lock Check in/out Diff/compare Merge Branch Tag Teamwork Support

106. Versioning Lock Check in/out Diff/compare Merge Branch Tag Teamwork Support On the Level ofthe Concrete Syntax!

107. Versioning Lock Check in/out Diff/compare Merge Branch Tag Teamwork Support Together withmanually written Source!

108. Repository vs. Teamwork Support File-Based

109. Repository … Element-Specific … Real-Time … Oftengoodfor Business DSLs Teamwork Support

110. File-Based … like SCMs … integrates wellwithmanuallywrittencode … Technical DSLs Teamwork Support

111. File-Based … integratesvery well with (real) textual DSLs Teamwork Support

112. Teamwork Support Tool Specific!

113. ToolingMatters!

114. The Language is not ToolingMatters! enough!

115. Teamwork Navigation Overviews Searching Quick-find Find-references Show usage Refactoring Debugging Code Completion Syntax Highlighting The Language is not ToolingMatters! enough!

116. The Language is not ToolingMatters! enough! Thisis also trueforthe Meta Developers!

117. Users shouldbeable toworkwithandstore wrong or ToolingMatters! incomplete models!

118. Users shouldbeable toworkwithandstore wrong or ToolingMatters! incomplete models! Temporarily. Noprocessing!

119. ToolingMatters! Nightly Build!

121.  ProcessingModels  the

123. Interpretation Generation vs.

124. Interpretation Interpretation vs. Generation

125. Generation Interpretation vs. Generation

126. Generation resultingcode Interpretation vs. Generation canbeeasily inspected

127. Generation resultingcode Interpretation vs. Generation canbeeasily debugged

128. Generation resultingcode canbe Interpretation vs. Generation optimized andmore efficient

129. Generation Templates canbe Interpretation vs. Generation derived fromexistingcode

130. Generation workaround Interpretation vs. Generation limitations oftargetlanguage

131. Generation nochanges Interpretation vs. Generation totargetenvironment (leavesnotrace)

132. Generation reuse Interpretation vs. Generation runtime infrastructure (garbage collection, monitoring…)

133. Interpretation faster turnaround Interpretation vs. Generation noregeneration test build deploy

134. Interpretation for platform indepenence an interpreter might be Interpretation vs. Generation less porting effort

135. Combinations Interpretation vs. Generation

136. Rich Domain SpecificPlatform

137. Rich Platform

138. Rich Platform Grownwiththe DSL!

139. Extreme Case Rich Platform populates

140. Checks Firstand Separate

141. Language Structure is not enough. Checks firstand separate Youneedconstraints. Boolean expressionsthatvalidate the model beyondstructure.

142. Model G Checks firstand separate … complex Constraints Code

143. Model Checks firstand separate G G‘ … complex … duplication Constraints Constraints Code Code‘

144. separate phase Model firstclasscitizens muchbetter. Constraints Checks firstand separate G G‘ check asmany aspossible. makeittight. Code Code‘

145. different constraints atdifferenttimes orfor Checks firstand separate different partitions ofthe model partition-local: editor, on-save global: batch, on-request

146. check early. Model moresemantics. Constraints bettermessages. T Model‘ Checks firstand separate T Model‘ G Code

147. Model Constraints 1 constraints 1 ok T implies Model‘ constraints 2 ok Checks firstand separate Constraints 2 implies T constraints 3 ok Model‘ Constraints 3 G Code

148. ERROR WARNING Checks firstand separate INFO

149. IntegratingGeneratedandManuallyWritten Code

150. Ifat all possible, Do not modify Integrated Gen/Man Code Generated Code!

151. ProtectedRegions are a badideabecause generatedcode Integrated Gen/Man Code is not a throwaway anymore.

152. ProtectedRegions … needto check in … sedimentofgeneratedcode Integrated Gen/Man Code

153. Better Approach: Hooks Integrated Gen/Man Code in thegeneratedcode

154. Better Approach: Hooks Integrated Gen/Man Code in thegeneratedcode extension points, base class, abstract methods & subclassing, empty callback methods, delegate, implement interfaces, #include, reflection, AOP, design patterns, partial classes

155. Integrated Gen/Man Code

156. ControlManuallyWritten Code

157. After codegeneration how do youmakesure developersfollow Control Manual Code all therequired procedures?

158. procedures? … subclass … overwrite … naming conventions Control Manual Code

159. Compiler Errors are not enough. Control Manual Code wrong abstraction level!

160. generatechecks againstthecodebase Control Manual Code evaluatedbytheIDE

161. if (false) { GeneratedBaseClass x = new ManualSubclass(); } Control Manual Code

162. Care aboutgeneratedcode Andrew Vargas

163. Generated Code a Throwaway Product? Care aboutGenerated Code

164. Generated Code a Throwaway Product? Care aboutGenerated Code Yes. Can beregenerated.

165. But: must be… … written (templates) … understood … debugged Care aboutGenerated Code

166. But: must be… … written (templates) … understood … debugged … extended … programmedagainst Care aboutGenerated Code ifyoudon‘tgenerate100%:

167. Care! … indent … usegoodnames … document … modularize Care aboutGenerated Code

168. Make CodeTrue to Model

169. Analyses on the model canverify all kindsof properties aboutthe Code truetothe Model system.

170. Analyses on the model canverify all kindsof properties aboutthe Code truetothe Model system. Iffthecodeis true tothe model

171. Use a clever programming model Code truetothe Model thatdoes not allow violations.

172. generatethe configuration for architectureanalysis tools. SoftwareTomographySonarJ Structure 101 … andthelike Code truetothe Model

173. Viewpoint-AwareProcessing

174. whenandhow do you and validate process Viewpoint-Aware Processing eachviewpoint?

175. whenandhow do you and validate process Viewpoint-Aware Processing eachviewpoint? Roles? Process?

176. Generate in phases: type developer implementmanualcode Viewpoint-Aware Processing deployment integrator run on targetsystem behaviour runtime interpretstatemachine

177. Also consider partitions Viewpoint-Aware Processing in thiscontext!

178. Overall ConfigurationViewpoint

179. Teamwork Partitions Viewpoints Multiple Targets Overall ConfigurationViewpt. Multiple Configurations

180. Teamwork Partitions Viewpoints Multiple Targets Overall ConfigurationViewpt. Multiple Configurations large ormany models

181. Overall ConfigurationViewpt.

182. Overall ConfigurationViewpt.

183. Putthisoverall configurationinto Overall ConfigurationViewpt. a model: configuration viewpoint

184. Care About Templates

185. Templates … importantasset Care about Templates … containplatform knowledge … tendtogrow morecomplex

186. Care! … modularize Care about Templates … functions … naming … polymorphism … aspects … refactor

187. Indentforthetemplates andthenuse a beautifier forthe Care about Templates generatedcode.

188. Indentforthetemplates andthenuse a beautifier forthe Care about Templates generatedcode. exceptifyouuse a langauage withsemanticwhitespace!

189. Care about Templates GoodPlatform. Fewer Templates. Less Care.

190. Model-2-Model ToSimplify Generators

191. Reducingtemplate complexity isimportant. M2M toSimplify Generators

192. Reducingtempalte complexity isimportant. M2M toSimplify Generators Separation ofConcerns isthewaytogo.

193. Insteadofputting complexlogic intothetemplates M2M toSimplify Generators putitinto an M2M thatruns beforecode gen.

194. M2M toSimplify Generators

195. M2M toSimplify Generators

196. In-language reduction rules: M2M toSimplify Generators

197. Model-2-Model For Simulation andProof

198. Many usefulformalisms alreadyexist. M2M for Simulation andProof

199. Many usefulformalisms alreadyexist. M2M for Simulation andProof Simulation Proofs Properties

200. Use an M2M forthisifpossible. M2M for Simulation andProof OftentheinputisXML so youactually „generatecode“

201. Cascading

202. ? PIM? Cascading PSM? PSSM?

203. BottomUp Cascading Works Better!

204. Cascading M2T Code + othertargetplatformartifacts

205. Cascading M2M M2M M2T M2T Code + othertargetplatformartifacts

206. M2M Cascading M2M M2M M2T M2T Code + othertargetplatformartifacts

207. MODELS READ ONLY! M2M Cascading M2M M2M M2T M2T Code + othertargetplatformartifacts

208. Allowfor Adaptations

209. EconomiesofScale Reuse! AllowforAdaptations

210. EconomiesofScale Reuse! AllowforAdaptations But: Adaptations!

211. EconomiesofScale Reuse! AllowforAdaptations But: Adaptations! Unexpected

212. Annotations on Models n/v pairs AllowforAdaptations tobeused in generators Adaptations on Templates Template-AO Factories/Polymorphism

213. AllowforAdaptations Becarefulto delineate the API!

214. Annotation Models

215. M2M Annotation Models M2M M2T M2T Code + othertargetplatformartifacts

216. M2M Annotation Models M2M M2M‘ M2T M2T Code + othertargetplatformartifacts

217. Annotation Model references elements in base model. Annotation Models Transformation takes additional informationinto account.

218. Makesurethe annotation model onlycaptures Annotation Models Exceptions fromthedefault in thetemplates.

219. ClassifyBehaviour

220. ? Action Semantics ClassifyBehaviour Languages useful?

221. ? Action Semantics ClassifyBehaviour Languages useful! But…

222. Classify! … statebased … businessrules ClassifyBehaviour … mathematics

223. Classify! … statebased … businessrules ClassifyBehaviour … mathematics … or a specific DSL

224. Classify! … statebased … businessrules ClassifyBehaviour … mathematics … or a specific DSL … 3GL code

225. Don‘tforget Testing

226. Don‘t Forget Testing Limited Expressiveness. Reduced Need For Tests.

227. Don‘t Forget Testing Constraint Checks. A Form of Test.

228. Better Testing Generator Example Models Code Don‘t Forget Testing Based On Binary Test Cases (hand written) Tests

229. Better Testing Generator Example Models Code Don‘t Forget Testing Based On Models and Language serve as meaningful „spec“ for what to test Binary Test Cases (hand written) Tests

230. Testing Generators Generator Reference Model Code Don‘t Forget Testing

231. Testing Generators Generator Reference Model Code Don‘t Forget Testing Based On Binary Reference Test Cases Tests

232. TestingTransformations M2M ResultModel Reference Model Don‘t Forget Testing

233. TestingTransformations M2M ResultModel Reference Model Don‘t Forget Testing Based On Reference Constraints Tests

234. TestingMetware Reference Model … maintaned! Don‘t Forget Testing … bymetaware Reference Test Cases developers Reference Constraints

235. This tests only the generators Generator Model Code Don‘t Forget Testing Tests Generator Test Code

236. This tests not the model: self fulfilling! Generator Model Code Don‘t Forget Testing Tests Generator Test Code

237. Separate test models and generated test code Model Generator Code Don‘t Forget Testing based on tests Test Model (Test Language) Generator Test Code

238. Separate test models and generate Mocks Model Generator Code Mocks Don‘t Forget Testing based on tests Generator Test Model (Test Language) Generator Test Code

239. Auto-Derived Test Models work sometimes. Model Generator Code Don‘t Forget Testing automatically derived tests Test Model (Test Language) Generator Test Code

241.  ProcessOrganization  the

243. Iterate!

244. Waterfallisbad! WithorWithout MD* Iterate!

245. Iterate! Concepts

246. Iterate! Concepts Language

247. Iterate! Examples Concepts Language

248. Generator+Tests Iterate! Examples Concepts Language

249. Generator+Tests Editor Beautification Iterate! Examples Concepts Language

250. Co-Evolve Language andConcepts

251. Co-Evolve Language & Domain

252. Co-Evolve Language & Domain

253. Building a language requires Formalization Co-Evolve Language & Domain

254. Building a language requires Formalization Co-Evolve Language & Domain requiresyouto thinkanddecide aboutthedomain.

255. Co-Evolve Language & Domain requiresfrequent Evolution!

256. Co-Evolve Language & Domain and flexible, agile Tooling!

257. Documentationis still necessary

258. The DSL and the „programs“ are documentation. Documentation still necessary

259. The DSL and the „programs“ are documentation. Not Quite! Documentation still necessary

260. Language Definition is not a Teaching Tool! Documentation still necessary

261. Tutorials … Concepts … Howtouse Language Documentation still necessary … Howtointegrate manualcode Example-Driven!

262. Language Definition captures the WHAT but not the WHY Documentation still necessary

263. Rationales … whytheconcepts? … whywegenerate Documentation still necessary whatwegenerate … targetplatform de- cisionsandidioms Grammar as Reference!

264. Different Media Documentation still necessary

265. Reviews

266. Reviews In mostcases, peoplecan still makemistakes.

267. DSL programs aremoreconcise, so Reviews reviewsare moreefficient.

268. Repeated „Mistakes“? Add constraint check. Reviews

269. Repeated „Mistakes“? Add constraint check. Reviews Ormaybethelanguage iswronganditis not a mistake?

270. Letpeople do what they‘regoodat

271. MD* hasseveral clearlydefined Let‘em do whatthey‘regoodat Roles

272. Tech Experts … evaluatetechnologies … digdeep Let‘em do whatthey‘regoodat … tune … createtemplates … spreadknowledge

273. Language Designer … works w/ domain expert abstractions, notations Let‘em do whatthey‘regoodat … adds modularization, in- heritance, „abstract“, etc. … works w/ architect generators, interpreters … requires „metapeople“

274. App Developer … caresaboutappdomain … uses DSLs + metaware Let‘em do whatthey‘regoodat … isisolatedfrom technology, does not have to care (that much)

275. Flip side: Let‘em do whatthey‘regoodat Youactuallyneedpeople whoaregoodatthis!

276. Domain Users Programming?

277. Precision != Programming Domain Users Programming?

278. Precision != Programming Domain Users Programming? DomainUsers Programmers Programming = Precision + X

279. Precision … Scientists … Insurance Mathematicians Domain Users Programming? … Logisticians … Medical Doctors

280. Domain Users Programming?

281. If Domain Users don‘tgetit Domain Users Programming? itmighthintat a problemwith thelanguage! … orthedocumentation!

282. Things learned from 3GLs Domain Users Programming? often not intuitively understandable for domain people.

283. Domain Users vs. Experts

284. Creatingthe Language vs. Domain Users vs. Experts Usingthe Language

285. Creating: Domain Expert Domain Users vs. Experts Using: Domain User

286. Creating: Domain Expert … senior … complete Domain Users vs. Experts … bigpicture … deep … precise, formal … guru

287. Using: Domain User … not senior … narrower Domain Users vs. Experts … shallow

288. Can Domain Users understand whattheExperts putinto Domain Users vs. Experts thelanguage? Verifyearlyandoften!

289. MetawareasProduct

290. Product: … releaseschedule … incrementaldevelopmt MetawareasProduct … requirementsmgt. … issuetracking … documentation … supportstaff

291. Exhange People … makemetadevelopers develop real apps MetawareasProduct … letappdevelopers developmetaware

292. Compatible Organization

293. MD* requires cross-project CompatibleOrganization work.

294. MD* requires cross-project CompatibleOrganization work. A strictproject-focused organizationdoes not work

295. Make room & budget forcross-cutting CompatibleOrganization work. Open Source?

296. Forget PublishedCase Studies

297. How do youknow ifitworks foryoursituation? Forget Published Case Studies

298. How do youknow ifitworks foryoursituation? Forget Published Case Studies Do not judgeby published casestudies!

299. How do youknow ifitworks foryoursituation? Forget Published Case Studies Do not judgeby published casestudies! (yes, theyareworthreading but don‘tdecidebased on them)

300. Instead: Prototype! … meaningful Forget Published Case Studies … 4-8 personweeks … incremental … externalhelp?

302.   Challenges

303. Mixing Notations Challenges

304. Mixing Notations Language Modularity Challenges

305. Mixing Notations Language Modularity MetawareRefactoring Challenges

306. Mixing Notations Language Modularity MetawareRefactoring Model/Code Refactoring Challenges

307. Mixing Notations Language Modularity MetawareRefactoring Model/Code Refactoring Challenges Automatic Model Migration

308. Mixing Notations Language Modularity MetawareRefactoring Model/Code Refactoring Challenges Automatic Model Migration Model Debugging

309. Mixing Notations Language Modularity MetawareRefactoring Model/Code Refactoring Challenges Automatic Model Migration Model Debugging Interpretation -> Code Gen

310. Mixing Notations Language Modularity MetawareRefactoring Model/Code Refactoring Challenges Automatic Model Migration Model Debugging Interpretation -> Code Gen Cartridges

312. THE END. Thankyou. Questions?