Buildout guide for creating repeatable Python applications
•Als ODP, PDF herunterladen•
1 gefällt mir•1,032 views
The document discusses how buildout is used to create repeatable Python application deployments. Buildout uses a configuration file and recipes to automate the installation of applications and their dependencies. It describes how buildout is used to deploy complex Plone and Django applications across multiple servers. Buildout solves the problems of repeatable and easy application deployment through its use of recipes, parts, and versions control of dependencies.
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (20)
Ähnlich wie Buildout guide for creating repeatable Python applications
Ähnlich wie Buildout guide for creating repeatable Python applications (20)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
Buildout guide for creating repeatable Python applications
- 1. Buildout: creating and deploying repeatable applications in python Internet Solutions www.codesyntax.com Azitain industrialdea, 3-K E-20600 Eibar Tel.: +34 943821780 Mikel Larreategi - @erralin PySS – Donostia - 2014-09-27
- 2. Who are we?
- 3. What we do? Websites! (mostly)
- 4. What we do? ● Collaborative news websites ● Community sites ● Corporate portals and intranets ● Multilingual sites / Localization ● R&D
- 5. How we do it? PYTHON!
- 6. Our background Back in 2001.....
- 7. Zope ● Official definition: Free, open-source, object-oriented web application server written in Python. ● A way to publish python objects in a web environment. ● Originally developed to replace/improve CGI-based development
- 8. Zope ● Introduced object oriented databases (ZODB) in 2001 ● Through-the-web development (don't tell anyone) ● Restricted python (stripped-down python for security reasons) ● DTML or ZPT as template language
- 9. But we also do Django ● Official definition: high-level python web framework that encourages rapid development and clean, pragmatic design. ● A framework to develop sites easily and fastly usually with SQL based data-schema and storage
- 10. Zope until .../2.5/2.6 ● Monolithic tarball ● Even included python (2.1) in earlier versions ● Add-ons developed as “Products”: folders with __init__.py and some initialisation code ● Drop add-ons on a folder and restart
- 11. Zope 2.7/2.8/2.9 ● From monolithic -> installable thing ● ./configure && make && make install ● System python or custom compiled ● Zope instance: install once, create N instances ● ZEO for database decoupling
- 12. But.... we needed more
- 13. Plone is a CMS ● Simple and powerful ● Adaptable workflow ● Security (users, groups, roles, permissions) ● Extensibility ● Usability
- 14. But the installation.... ● Plone 1.x, 2.0, 2.1, 2.5, 3.0, 3.1: ● Tarball Zope 2.6, 2.7, → 2.8, 2.9 and 2.10 ● Plone 3 (2007) and 3.1 (2008): ● Zope → Zope 3: small reusable components ● Plone 3.2 (2009): ● Tarball → eggs ● Everyone started developing “python eggs” for Plone
- 15. Buildout to the rescue ● Jim Fulton (creator of Zope), created zc.buildout ● To solve 2 problems: ● Application-centric assembly and deployment ● Repeatable assembly of programs from python eggs
- 16. Plone Architecture Apache/Nginx :80 Zope (w Plone) :8080 ZEO (w ZODB) :8081
- 17. Complex installation ● With python eggs, the installation is not to tar xzf Plone-3.2.0.tgz ● But, python setup.py install a lot of eggs: ● Plone 3.x: +150 ● Plone 4.x: +250 ● Complex setups possible
- 18. Plone Architecture Apache/Nginx :80 Zope (w Plone) :8082 ZEO (w ZODB) :8084 Zope (w Plone) :8081 Zope (w Plone) :8083
- 19. Plone Architecture Apache/Nginx :80 Zope (w Plone) :8082 ZEO (w ZODB) :8084 Zope (w Plone) :8081 Zope (w Plone) :8083 SERVER 1 SERVER 2 SERVER 3
- 20. Needs ● Easy deployment (2-3 commands) ● Repeatable: ● We need to run several times in development (each time we add new eggs) ● We need to run it in several servers (own, development, production, ...)
- 21. How solves it buildout? ● ini-format config file (buildout.cfg) ● Ordered execution of “parts” ● Each “part” runs a “recipe”: ● python egg that given an input, does something ● Nothing else
- 22. Extending buildout ● Buildout parts can be extended from other files and override options (OOP concept) ● Extension can happen over-the-net (extends= http://myserver/buildout.cfg) ● Known Good Set of eggs for proper installation ● Create your own recipe
- 23. Why repeatable? ● You can set exact versions of eggs ● You say which files you download from where ● The generated filesystem layout is always the same ● If all versions are set correctly you will always get the same installation
- 24. How you develop new eggs? ● They have no exact version: ● We can switch from given versions to source checkouts (svn, git, hg, …) easily using an extension: mr.developer ● Extensions are similar to recipes ● Enable and deactivate develop-eggs
- 25. Buildout example (minimal) [buildout] parts = installation [installation] recipe = zc.recipe.egg eggs = feedparser interpreter = custompy Creates a python interpreter (custompy) with feedparser in the path
- 26. Buildout example: Plone [buildout] extensions = mr.developer showpickedversions = true parts = instance zeo omelette zopepy extends = http://dist.plone.org/release/4.3.3/versions.cfg (show plonebuildout.cfg)
- 27. Q: You use this for django? ● A: Yes! ● Q: Why? ● A: because we came to django after Plone, and we knew how it worked and we were used to it. All our deployments are done in the same way
- 28. Q: but... ● A: yes, we could install django with pip in a virtualenv... but we were used to use a more complex solution that allowed us to create bigger things
- 29. Buildout example: django [buildout] parts = django staticfiles cron (show djangobuildout.cfg)
- 30. What recipes do? ● Plone world: ● Create a Zope instance and put all eggs in the path ● Configure port numbers, database location, blob file location, object cache sizes, … ● Create a ZEO instance to decouple the database (configure ports, paths, etc)
- 31. What recipes do? ● Django world: ● Create an interpreter with django installed without polluting the system installation ● Run static file collector ● Create WSGI scripts ● Install gunicorn and configure to run with nginx
- 32. What recipes do? ● Any world: ● Create supervisor/circusd scripts to run services ● Download packages from svn/git/hg ● Download tarballs ● Create cronjobs
- 33. What recipes do? ● Any world: ● Install Apache/nginx ● Configure load balancers (pound/haproxy) ● Update po files and upload to Google Docs for easy translation ● Create egg omelette to easily navigate on installed eggs ● ...
- 34. Real life examples ● In all of our servers we have +120 buildouts ● Everyone knows (Plone or django developer) how to run them: ● ./bin/buildout -vv ● It's predictable
- 35. Example architecture with buildout: www.bertsozale.com NGINX :80 (system package) VARNISH :8081 HAPROXY :8082 ZOPE :8083 ZOPE :8083 ZOPE :8083 ZEO :8084 All in one server
- 36. Example architecture: EEA APACHE :80 POUND :8080 ZEO :8080 ZOPE :8081 ZOPE :8082 ZOPE :8083 ZOPE :8084 ZOPE :8081 SERVER 2 ZOPE :8082 ZOPE :8083 ZOPE :8084 ZOPE :8081 SERVER 3 ZOPE :8082 ZOPE :8083 ZOPE SERVER 1 :8084 SERVER 4 SERVER 5
- 37. Example architecture: EEA ● 4 servers, everything configured using one buildout file. ● https://github.com/eea/esdrt.buildout ● EEA Sysadmins encourage buildout setup ● Repeatability ● No need of root user to install things
- 38. Buildout vs fabric? ● Use of buildout does not exclude use of fabric. ● Fabric also used for automation and repeatable environments ● All is about what is best for you: ● Simple fab file? → do it ● A lot of deploy code lines? → try buildout
- 39. Buildout & fabric ● Use of buildout doesn't exclude fabric (did I say that?) ● Use fabric to run update and run buildout: ● Update git checkout ● Run buildout ● Restart services
- 40. Buildout & fabric ● Some EEA projects use it ● We don't, we ssh & run buildout ● But there is literature about this: ● http://tinyurl.com/fab-buildout ● http://tinyurl.com/fab-buildout-so ● Fabric is a way to run commands on remote servers
- 41. Buildout: Summary ● Assemble and deploy simple and complex setups ● Repeatable → installed ecosystem is always the same ● Specially suited for python based applications (Plone, Zope, Django, Pyramid), but also for non-python things: Apache, Pound, HAProxy, Memcached, ...
- 42. Buildout (the bad part) ● If your deployments are not similar, finding and searching recipes can be hard. ● Error messages, sometimes are not helpful. ● setuptools & distribute fight in the last 6 months
- 43. Q & A ?
Hinweis der Redaktion
- We are a team of 15 people working on internet since 2001 We are not all developers: Management: 1 Content authoring, client support and management help: 4 Designers: 2 Developers: 9
- In simple language: we do websites
- But what kind of websites? We build sites for our customers so that they can manage the site through the web, without HTML, FTP or any other prior knowledge. They manage the sites using a WYSIWYG editor We also do much R&D, and many of our projects come after much time invested on R&D: we now work with real-time-twitter-stream parsing and content curation for instance.
- And we do everything in python, and we are proud of it. We love python Both for our internal scripts and also for all web development we do OK, I admit we do some WordPress...
- We come from Zope world In 2001 one of the founders discovered Zope and Python Zope has been recognized as a python killer app People say that Zope has used several concepts that later have been widely used. There are two quotes in one Zope book by Phillip J. Eby (creator of setuptools) that say: “Where Zope leads, Python follows” “Those who do not studay Zope, are condemned to reinvent it”
- The story says that Zope was invented in a plane, by Jim Fulton who was going to teach CGI development, and thought that object publishing was a saner way to do web-development
- Zope encouraged web-based-development to develop quick and powerful dynamic sites through the web Yes python scripts and templates were written in a form in a management area But, for security reasons, a stripped-down version of python was used: only some of the standard library packages could be used They invented 2 templating languages: DTML: something like PHP, mixing DTML tags and HTML tags ZPT: using XML namespaces to create XML-valid documents If you needed to do more powerful things, you could write normal python packages on the filesystem
- But not everything is Zope in CodeSyntax. Although we developed mysql based applications on top of Zope writing raw SQL queries, we finally adopted Django to do SQL-schema-driven applications Django is also great!
- First versions of Zope were monolithic tarballs, that you needed to extract and run using a custom script. The filesystem products were just dropped in a given folder on the filesystem, and everything worked quite nicely.
- With Zope 2.7, Zope brought the concept of “Zope instance”: you install Zope once, and then you can create several instances: You just have to install Zope ones You can even install products that can be used in all of your instances But you can install custom products/packages only in your instance Zope need to be compiled with a given python version and run cmmi You can also create an independent ZEO instance, with that you can install the database in a separate instance on the same server or in another one.
- But for large websites, we thought that we needed an integrated powerfull environment, and found Plone, which has many things already done (we can understand Zope as a set of tools, and Plone a product built on top of those tools that provides more features).
- As a complex product, Plone installation was each time more comples: On first versions, Plone was released as a tarball: download, untar it inside Zope and worked But people started using and developing products for Plone (extending core functionality), and at the same time, Zope started to be rewritten into Zope 3, to provide small reusable components, providing a way to upgrade independent components of Zope, and encourage reusability outside Zope. That's why with Plone 3.2 everyone started writing standard python eggs and something needed to be done with the installation, because doing python setup.py install to all products was a no-go
- So, the creator of Zope, Jim Fulton, wrote zc.buildout to install Zope 3 applications, and Plone got benefit from that to install Plone.
- This is a typical Plone setup, with a web server, the application server, in which Plone is installed, and the database server. Everything can be installed on the same server or split across several servers
- As you see, Plone's installation needed to install a lot of python eggs, and needed to create several scripts to run zope instances, configure database connection, ….
- And the installation can be splitted across several servers, so the installation needs to be easy and repeatable. You don't want to run the same application with different egg versions on the same server (all instances need to have the same version setup).
- And if you split everything on 3 (or more) servers, you need to have everywhere the same versions
- So, we (and Plone developers, and sysadmins), needed something that could be run easily and fastly (when in development several times a day) It needed to be repeatable: we want the same layout, and same set of versions on development or production servers And that's why buildout was created
- Buildout gets that defining an ordered execution of “parts”. Each part is run by a “recipe” And a recipe is a “do something” script. Each recipe knows what to do.
- You can even create your own recipes, or extend one buildout file from another. For instance: In development servers, you need to start Zope in “debug mode”, on production not; In development servers you install just one instance; on production 2; In development servers the port numbers are ones; on production others This is achieved creating your own file, saying there that you extend another one, and overriding the values. It's just and OOP thing.
- Why we say this is repeatable? Because the installation is done on all servers in the same way You say in the buildout configuration file which files or which packages and which versions to download. The generated filesystem layout is always the same If all versions are set, the installation result will be always the same.
- Is this just for deployment or also for development? You need to say buildout the versions of product you install (feedparser = 5.1.2), but what if you are developing a new product? We use an extension, called “mr.developer”, that easily plugs into buildout the so-called “develop eggs”, eggs without a fixed version that are being developed. Moreover this extension eases the way to checkout and clone packages from git/svn/mercurial/etc.
- Show plonebuildout.cfg
- And yes, we use this for django too. Perhaps django installation is easier without buildout (virtualenv & pip), but we came into django after learning buildout for Plone, and buildout worked so well that we adopted buildout also for
- Yes, OK, … we could do it with pip and virtualenv, but this way, we use everywhere the same deploying mechanism and everyone know how to run things.
- Show djangobuildout.cfg
- What can buildout recipe do? In plone environments the basic recipes: Create zope instances from zope installation Create ZOE instances to decouple data-base installation Configure paths, environment vars, database locations, paths, por numbers, object cache sizes....
- In django worlds: Install django in an isolated environment Run static file collectors Create WSGI scripts
- But you have many other recipes to do almost everything: Create supervisor or circus scripts and configurations to run the services Download packages Create cron jobs
- Install apache Configure load balancers depending on the server instance setup Upload PO files to a Google Doc to easily translate interface messages Create egg omelette to navigate in eggs... Search pypi for recipes...
- But this is used in real-life? All Plone installations since 2009 are installed using buildout. Buildout is the responsible of the most web-traffic on pypi In all of our servers we have more than 120 buildouts
- This is an example architecture we installed with buildout in one server (it was quite big server) for a customer. It installed varnish (caching server), haproxy (load balancer), several Zope instances and a database server. We also tested to put a database slave in another machine, and the master database was replicated life there to have a live backup-copy
- In another project for the European Environment Agency, we have this setup in 5 servers, everything installed using buildout One server with Apache and Pound: Three servers, each with 4 zope instances Another server with the databse.
- All this complex setup (remember, you need to deploy the exact same versions of everything on all servers), is handled by one or two buildout files, and everything is public. You can check the configuration files on github. The EEA sysadmins even encourage this buildout setup (they are not Plone experts), because all setups across their premises are handled in the same way, the users have the same permission-set and errors can be found easily.
- And what about fabric? Cannot you use fabric? Yes, the use of buildout doesn't exclude fabric (or any other tool). Fabric is a tool for automation and running remote task. Everything is about what is best for you. We feel confortable with buildout, we also think that it's easier to reuse all those existing recipes than building tasks with fabric. Anyway, we don't exclude the use of fabric, if that suits you, use it!
- You can even use it with buildout. There's literature about using fabric with buildout: not to handle the installation of all those packages, but to run buildout on the server and to update the svn/git checkouts.
- So, as a summary: Buildout helps you assembling and deploying both simple and complex applications. Those setups are repeatable: everywhere you run them, you will get the same setup It's specially suited for python applications (Zope, Plone, Django, pyramid), …. but you can also install Apache, Pound, HAProxy, Memcached, ...)