2. Has this happened to you? Email to users results in 50+ undeliverable Need to verify the users in Active Directory Then “deactivate” former employees in Crowd 750 mouse clicks later, you’re done! 2 http://www.flickr.com/photos/left-hand/4231405740/

3. 3 Automate That!

4. Agenda Use cases for scripting Atlassian APIs available for scripting The awesome power and simplicity of python Examples 4

5. When is scripting useful? Automate time consuming tasks Perform data analysis Cross-reference data from multiple systems 5

6. Some specific use cases Crowd – Deactivate Users and remove from all groups Bamboo – Disable all plans in a project JIRA – Release Notes Subversion – custom commit acceptance Custom build processes – pull code linked to a specific issue into a patch archive 6

8. Installing new plugins can require a restart

9. Prefer to minimize ad hoc changes on the server

10. Need to correlate information from several systems

12. More APIs for scripting(the ones we prefer to use) RESTful Remote APIs (now deprecated) High level interface Supports a handful of actions Now emerging: “real” REST interfaces High level interface Supports a handful of actions http://confluence.atlassian.com/display/REST/Guidelines+for+Atlassian+REST+API+Design 9

13. Why Python? Powerful standard libraries Http(s) with cookie handling XML and JSON Unicode Third Party Libraries SOAP REST Templates Subversion Portable, cross-platform 10

14. Python Versions 2.x Ships with most linux distributions Lots of third-party packages available 3.x Latest version Deliberately incompatible with 2.x Not as many third-party libraries 11

15. HTTP(s) with Python Python 2 httplib – low level, all HTTP verbs urllib – GET and POST, utilities urllib2 – GET and POST using Request class, easier manipulation of headers, handlers for cookies, proxies, etc. Python 3 http.client – low level, all HTTP verbs http.parse - utilities urllib.request – similar to urllib2 Third-Party httplib2 – high-level interface with all HTTP verbs, plus caching, compression, etc. 12

16. Example 1JIRA Issue Query & Retrieval 13

17. 14 Discovering URLs for XML

18. Simple Issue Retrieval 15 import urllib, httplib import xml.etree.ElementTree as etree jira_serverurl = 'http://jira.atlassian.com' jira_userid = 'myuserid' jira_password = 'mypassword' detailsURL = jira_serverurl + br /> "/si/jira.issueviews:issue-xml/JRA-9/JRA-9.xml" + br /> "?os_username=" + jira_userid + "&os_password=" + jira_password f = urllib.urlopen(detailsURL) tree=etree.parse(f) f.close() Construct a URL that looks like the one in the UI, with extra parms for our user auth Open the URL with one line! Parse the XML with one line!

19. Find details in XML 16 Find based on tag name or path to element details = tree.getroot() print "Issue: " + details.find("channel/item/key").text print "Status: " + details.find("channel/item/status").text print "Summary: " + details.find("channel/item/summary").text print "Description: " + details.find("channel/item/description").text Issue: JRA-9 Status: Open Summary: User Preference: User Time Zones Description: <p>Add time zones to user profile. That way the dates displayed to a user are always contiguous with their local time zone, rather than the server's time zone.</p>

20. Behind the scenes…cookies! 17 Turn on debugging and see exactly what’s happening httplib.HTTPConnection.debuglevel= 1 f = urllib.urlopen(detailsURL) send: 'GET /si/jira.issueviews:issue-xml/JRA-9/JRA-9.xml?os_username=myuserid&os_password=mypassword HTTP/1.0Host: jira.atlassian.comUser-Agent: Python-urllib/1.17' reply: 'HTTP/1.1 200 OK' header: Date: Wed, 20 Apr 2011 12:04:37 GMT header: Server: Apache-Coyote/1.1 header: X-AREQUESTID: 424x2804517x1 header: X-Seraph-LoginReason: OK header: X-AUSERNAME: myuserid header: X-ASESSIONID: 19b3b8o header: Content-Type: text/xml;charset=UTF-8 header: Set-Cookie: JSESSIONID=A1357C4805B1345356404A65333436D3; Path=/ header: Set-Cookie: atlassian.xsrf.token=AKVY-YUFR-9LM7-97AB|e5545d754a98ea0e54f 8434fde36326fb340e8b7|lin; Path=/ header: Connection: close JSESSIONID cookie sent from JIRA

21. Authentication User credentials determine: The data returned The operations allowed Methods Available: Basic Authentication JSESSIONID Cookie Token Method 18

22. Basic Authentication Authentication credentials passed with each request Can be used with REST API 19

23. JSESSIONID Cookie Authentication credentials passed once; then cookie is used Used when scripting the user interface Can be used with REST API for JIRA, Confluence, and Bamboo 20

24. Token Method Authentication credentials passed once; then token is used Used with Fisheye/Crucible REST Used with Deprecated Bamboo Remote API 21

25. Obtaining a cookie Scripting the user interface login page Adding parameters to the user interface URL: “?os_username=myUserID&os_password=myPassword” Using the JIRA REST API 22

26. JIRA REST Authentication 23 import urllib, urllib2, cookielib, json # set up cookiejar for handling URLs cookiejar = cookielib.CookieJar() myopener= urllib2.build_opener(urllib2.HTTPCookieProcessor(cookiejar)) creds = { "username" : jira_userid, "password" : jira_password } queryurl = jira_serverurl + "/rest/auth/latest/session" req = urllib2.Request(queryurl) req.add_data(json.dumps(creds)) req.add_header("Content-type", "application/json") req.add_header("Accept", "application/json") fp= myopener.open(req) fp.close() urllib2 handles cookies automatically. We just need to give it a CookieJar Request and response are both JSON We don’t care about response, just the cookie

27. Submitting a JIRA Querywith the user interface 24 # Search using JQL queryJQL = urllib.quote("key in watchedIssues()") queryURL = jira_serverurl + br /> "/sr/jira.issueviews:searchrequest-xml/temp/SearchRequest.xml" + br /> "?tempMax=1000&jqlQuery=" + queryJQL fp = myopener.open(queryURL) # Search using an existing filter filterId = "20124" queryURL = jira_serverurl + br /> "/sr/jira.issueviews:searchrequest-xml/" + br /> "{0}/SearchRequest-{0}.xml?tempMax=1000".format(filterId) fp = myopener.open(queryURL) Pass any JQL Query Or Pass the ID of an existing shared filter

28. A JQL Query using REST 25 # Search using JQL queryJQL= "key in watchedIssues()" IssuesQuery= { "jql" : queryJQL, "startAt" : 0, "maxResults" : 1000 } queryURL = jira_serverurl + "/rest/api/latest/search" req = urllib2.Request(queryURL) req.add_data(json.dumps(IssuesQuery)) req.add_header("Content-type", "application/json") req.add_header("Accept", "application/json") fp= myopener.open(req) data = json.load(fp) fp.close() Pass any JQL Query Request and response are both JSON

29. XML returned from user interface query 26 An RSS Feed with all issues and requested fields that have values

30. JSON returnedfrom a REST query 27 {u'total': 83, u'startAt': 0, u'issues': [{u'self': u'http://jira.atlassian.com/rest/api/latest/issue/JRA-23969', u'key': u'JRA-23969'}, {u'self': u'http://jira.atlassian.com/rest/api/latest/issue/JRA-23138', u'key': u'JRA-23138'}, {u'self': u'http://jira.atlassian.com/rest/api/latest/issue/BAM-2770', u'key': u'BAM-2770'}, {u'self': u'http://jira.atlassian.com/rest/api/latest/issue/BAM-2489', u'key': u'BAM-2489'}, {u'self': u'http://jira.atlassian.com/rest/api/latest/issue/BAM-1410', u'key': u'BAM-1410'}, {u'self': u'http://jira.atlassian.com/rest/api/latest/issue/BAM-1143', u'key': u'BAM-1143'}], u'maxResults': 200} A list of the issues found, with links to retrieve more information

31. JSON issue details 28 All applicable fields are returned, even if there’s no value Expand the html property to get rendered html for description, comments

32. What’s the difference? 29 <reporter username="mlassau">Mark Lassau [Atlassian]</reporter> <customfield id="customfield_10160" key="com.atlassian.jira.toolkit:dayslastcommented"> <customfieldname>Last commented</customfieldname> <customfieldvalues> 1 week ago </customfieldvalues> </customfield> u'reporter': { u'type': u'com.opensymphony.user.User', u'name': u'reporter', u'value': { u'self': u'http://jira.atlassian.com/rest/api/latest/user?username=mlassau', u'displayName': u'MarkLassau [Atlassian]', u'name': u'mlassau'}}, u'customfield_10160': { u'type': u'com.atlassian.jira.toolkit:dayslastcommented', u'name': u'Last commented', u'value': 604800}, XML values are display strings REST values are type-dependent

34. Returns all fields

35. Values require type-specific interpretation

36. Easier to transition issues

38. XML returns only fields that contain values

39. Values always one or more display strings

41. Which build resolved my issue? Bamboo keeps track of “related issues” (based on issue IDs included in commit comments), but doesn’t know when issues are resolved. If we know the issue is resolved in JIRA, we can look to see the latest build that lists our ID as a “related issue” Not a continuous integration build? We’ll need to look in fisheye to determine the highest revision related to this issue and then look in bamboo to see if a build using this revision has completed successfully. 32

42. To Fisheye for related commits! 33 queryURL= FisheyeServer + "/rest-service-fe/changeset-v1/listChangesets" + br /> "?rep={0}&comment={1}&expand=changesets".format(FisheyeRepo, myissue) req= urllib2.Request(queryURL) auth_string = '{0}:{1}'.format(fisheye_userid,fisheye_password) base64string = base64.encodestring(auth_string)[:-1] req.add_header("Authorization", "Basic {0}".format(base64string)) response = myopener.open(req) issuecommits=etree.parse(response).getroot() response.close() Query a specific fisheye repository for a commit with our JIRA issue ID in the comments Use basic auth headers to authenticate

43. Fisheye changesets returned 34 <results expand="changesets"> <changesets> <changeset> <csid>130948</csid> <date>2011-04-29T12:35:56.150-04:00</date> <author>lc6081</author> <branch>trunk</branch> <comment>MYJIRAPROJECT-2823 Modified to add parameters</comment> <revisions size="1" /> </changeset> </changesets> </results>

44. Parsing the changesets 35 commits = [] for changeset in issuecommits.findall("changesets/changeset"): commits.append(changeset.findtext("csid")) commits.sort() print "Highest commit is: " + commits[-1] Highest commit is: 130948

45. Logging into Bamboo 36 urllib2.HTTPCookieProcessor(cookiejar)) cookiejar = cookielib.CookieJar() myopener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cookiejar)) queryURL = bambooServer + "/userlogin!default.action“ params= urllib.urlencode({ "os_username" : bambooUserid, "os_password" : bambooPassword}) response = myopener.open(queryURL, params) response.close() Using a POST to the user interface login screen to retrieve a JSESSIONID cookie

46. Querying for build results 37 # Warning: This is a very resource-intensive operation. # You should consider limiting the number of builds returned queryURL= bambooServer + "/rest/api/latest/result/MYPROJECT-MYPLAN" + br /> "?expand=results[-10:-1].result.jiraIssues" req = urllib2.Request(queryURL) req.add_header("Accept", "application/xml") response = myopener.open(req) results=etree.parse(response).getroot() response.close() Use negative indexes to return the last entries in build list, e.g. [-10:-1] returns last ten builds in list Request the related issues Ask for XML (JSON also available)

47. Example (partial) build results 38 <results expand="results"> <link href="http://mybamboo.domain.com:8080/rest/api/latest/result/MYPROJECT-MYPLAN" rel="self" /> <results expand="result" max-result="25" size="46" start-index="0"> <result expand="comments,labels,jiraIssues,stages" id="3146125" key="MYPROJECT-MYPLAN-26" lifeCycleState="Finished" number="26" state="Successful"> <link href="http://mybamboo.domain.com:8080/rest/api/latest/result/MYPROJECT-MYPLAN-26" rel="self" /> <buildStartedTime>2011-04-29T05:04:14.460-05:00</buildStartedTime> <buildCompletedTime>2011-04-29T05:34:35.687-05:00</buildCompletedTime> <buildRelativeTime>4 days ago</buildRelativeTime> <vcsRevisionKey>4483</vcsRevisionKey> <buildReason>Code has changed</buildReason> <comments max-result="0" size="0" start-index="0" /> <labels max-result="0" size="0" start-index="0" /> <jiraIssues max-result="1" size="1" start-index="0"> <issue iconUrl="http://myjira.domain.com/images/icons/bug.gif" issueType="Defect" key="MYJIRAPROJECT-1629" summary="Need to display an error message when balance is zero."> <urlhref="http://myjira.domain.com/browse/MYJIRAPROJECT-1629" rel="self" /> </issue> </jiraIssues> <stages max-result="1" size="1" start-index="0" /> </result> </results> </results> Can also expand comments, labels, and stages jiraIssues property has been expanded here

48. Walking through build results 39 for result in results.findall("results/result"): print result.get("key") + ":" print "Revision: " + result.findtext("vcsRevisionKey") issues = [issue.get("key") for issue in result.findall("jiraIssues/issue")] print "Issues: " + ", ".join(issues) MYPROJECT-MYPLAN-31: Revision: 4489 Issues: MYJIRAPROJECT-1658 MYPROJECT-MYPLAN-30: Revision: 4486 Issues: MYJIRAPROJECT-1630 MYPROJECT-MYPLAN-29: Revision: 4485 Issues: MYJIRAPROJECT-1616, MYJIRAPROJECT-1663

49. Example 3Removing a user from a Crowd group 40

50. Beyond GET and POST 41 Lower-level HTTPConnection needed connection = httplib.HTTPConnection('myCrowdServer.mydomain.com:8080') operation = 'DELETE' urlpath = "/rest/usermanagement/latest/user/group/direct" + br /> "?username={0}&groupname={1}".format(userToRemove, fromGroup) body = None auth_string = '{0}:{1}'.format(crowdAppName,crowdAppPassword) base64string = base64.encodestring(auth_string)[:-1] headers = {'Authorization' : "Basic {0}".format(base64string)} connection.request(operation, urlpath, body, headers) response = connection.getresponse() print response.status, response.reason connection.close() Authenticate as a Crowd Application 204 - group membership is successfully deleted 403 - not allowed to delete the group membership 404 - the user or group or membership could not be found

51. A few loose ends Be prepared to handle Unicode strings Error handling – not shown here, but important! Formatting output – several python libraries for handling templates are available REST Interfaces – you can write your own! http://confluence.atlassian.com/display/DEVNET/Plugin+Tutorial+-+Writing+REST+Services 42

52. Links for more information http://confluence.atlassian.com/display/JIRA/Displaying+Search+Results+in+XML http://confluence.atlassian.com/display/JIRA/JIRA+REST+API+(Alpha)+Tutorial http://confluence.atlassian.com/display/CONFDEV/Confluence+REST+APIs http://confluence.atlassian.com/display/FECRUDEV/REST+API+Guide http://confluence.atlassian.com/display/BAMBOO/Bamboo+REST+APIs http://confluence.atlassian.com/display/CROWDDEV/Crowd+REST+APIs 43