Let try to think what we need to implement for community package manager in COS

1. Approaching Package Manager
Timur Safin
Intersystems

2. Where to
find
components?
GitHub?
SourceForge?
Developers
site?
How easy to find Caché components?
There is no single location,
and there is no easy way to
install extension

3. Popular
languages all
have package
managers
Perl
Python
JavaScript
Ruby
Haskell
Importance of a package manager
Package manager is
a key in language
ecosystem
popularity

4. Historical prospective
Package managers for ages and for languages

5. Timeline
CTAN
CPAN
CRAN
FreeBSD Ports
NetBSD pkgsrc
DarwinPorts/MacPorts
PythonPI
RubyGems
Node.js NPM
MacOS HomeBrew
Apache Maven
NuGet
Chocolatey
Debian Dpkg
Red Hat RPM
Apple App Store
Android Market/Google Play
Jan-92 Jan-94 Jan-96 Jan-98 Jan-00 Jan-02 Jan-04 Jan-06 Jan-08 Jan-10 Jan-12

6. CPAN
Perl Package Manager

7. Experienced Perl programmers often comment that half of Perl's
power is in the CPAN. It has been called Perl's killer app. Though the
TeX typesetting language has an equivalent, the CTAN (and in fact the
CPAN's name is based on the CTAN), few languages have an
exhaustive central repository for libraries. The PHP language has PECL
and PEAR, Python has a PyPI (Python Package Index) repository, Ruby
has RubyGems, R has CRAN, Node.js has npm, Lua has LuaRocks,
Haskell has Hackage and an associated installer/make clone cabal but
none of these are as large as the CPAN. … Other major languages,
such as Java and C++, have nothing similar to the CPAN (though for
Java there is central Maven).
CPAN
”

8. The CPAN has grown so large and comprehensive over the years
that Perl users are known to express surprise when they start to
encounter topics for which a CPAN module doesn't exist already.
CPAN
”

9. • 90ties were years of source-based package managers
• CTAN is the 1st language distribution which established practice to install
contributed content (Tex packages and extensions) from central repository;
• Real success achieved when Perl community started to use this model in CPAN:
– since the moment of CPAN inception in the 1995 it collected “140,712 Perl
modules in 30,670 distributions, written by 11,811 authors, mirrored on 251
servers”
• Similar repository created for R (statistical language) and “surprisingly” called
CRAN;
– regardless the fact that R is "relatively rarely used" CRAN has accumulated
6000+ packages of extensions
CPAN, CRAN, CTAN

10. • Simple instruction to install a package
• This will proceed following steps:
– Query repository for package availability;
– Download latest package version;
– Build it (if necessary)
– Test it (if described how)
– And install generated/provided package files
CPAN, CRAN, CTAN
>cpan install HTTP::Proxy

11. • At the same middle 90-ies FreeBSD has introduced their “ports collection” as a way to
distribute open-source software
• BSD-derivatives (e.g. OpenBSD or NetBSD) maintained their own ports collections
– Basic mechanism is the same for all BSD:
• Sources to be installed from the selected media (CD-ROM, DVD or internet site);
• Product built using the Makefile given and compiler(s) available;
• Build targets installed according to the rules written in the Makefile or package
definition file;
• From implementation prospective Darwin Ports/MacPorts are BSD derivatives.
BSD: FreeBSD ports, NetBSD pkgsrc, and Darwin ports
cd /port/location; make install

12. • Recursive install is available
– There is an option to handle all dependencies of a given port if there was request,
so full installation for bigger package could be initiated via single command and package
manager will handle all the recursive dependencies appropriately.
e.g. for Darwin ports
BSD: FreeBSD ports, NetBSD pkgsrc, and Darwin ports
$ sudo port install apache2

13. OS Package
Managers
Source
Packages
CPAN CTAN CRAN BSD ports
Binary
Packages
Linux RPM Linux Dpkg
Windows
Nuget

14. Binary package managers
Linux, Windows, all the rest

15. • With source-code based packages we had few obvious problems:
– Not all software could be deployed in their source form, there should be
convenient way to deploy binaries of a proprietary software;
– Building of a big project source tree may took a huge chunk of time (hours)
• People needed binary package managers which could distribute already built
and ready for consumption executables and binaries;
Request for binary package managers

16. • So Debian has introduced .deb format as 1st binary package format:
– .deb format used by Debian package manager (dpkg);
– original format introduced in Debian 0.93 in the March 1993 and was just the tar.gz with magic
prefixes;
– currently .deb package is just the AR archive consisting of 3 files (debian-binary with version,
control.tar.gz with metadata and data.tar.* with the installed files)
– Usually dpkg is not used directly but rather via APT (Advanced Packaging Tool)
• The apt-get’ reach functionality has influenced all later package managers
Linux: Debian and Red Hat

17. • RPM (Red Hat Package Manager) introduced with Red Hat V2.0 the late 1995
• Red Hat quickly became the most popular Linux distribution and solid RPM features was
one of factors winning competition here.
• RPM started to be used by all RedHat-based distributions (e.g. Mandriva, ASPLinux,
SUSE, Fedora or CentOS), and even beyond Linux by Novell Netware, or IBM AIX
• Similar to APT/dpkg there is Yum wrapper for RPM packages, which provides high-level
services for dependency tracking or building/versioning.
Linux: Debian and Red Hat

18. • Apple App Store has became 1st truly popular application store for mobile OS.
• App Store and Google Play are essentially OS specific package managers:
– With some extra API for online purchases;
– And some extra support for multiple hardware architectures. E.g. for Android application it
handles equally well Arm, x64, or MIPS devices
• Hardware optimizations/adaptations could be done at the installation time (Android
Lollipop ART engine), but this part of installation process is simply considered a part of
OS services.
“Package manager” handles this cross-platform optimization transparently to the user or
programmer;
Mobile sores: iOS App Store, Android Market/Google Play

19. There were several, not very popular, repositories of Windows applications:
– Windows Store for Windows Metro applications
– NuGet package manager for .NET extensions working via Visual Studio;
– Cygwin repository for Unix applications ported to Windows via Cygwin API;
But there has never been any generic solution like apt-get in Debian Linux;
Chocolatey Nuget has fixed this:
Windows applications: Chocolatey Nuget
Chocolatey is a package manager for Windows (like apt-get or
yum but for Windows). It was designed to be decentralized
framework for quickly installing applications and tools that you need.
It is built on the NuGet infrastructure currently using PowerShell as
its focus for delivering packages from the distros to your door, err
computer.
”

20. • There are multiple factors which have led to recent success of JavaScript as a server-side
language.
• One of them is the availability of central Node.js modules repository - NPM (Node Package
Manager) . Which is bundled with Node.js distribution since version 0.6.3 (November 2011).
• NPM is modeled similarly to CPAN, i.e. script running from command-line:
– connects to central repository;
– search for requested module;
– downloads appropriate version of package;
– parses package metainfo;
– and if there are external dependencies then process them recursively ;
Node.js NPM

21. C:UsersTimurDownloads>npm install -g less
npm http GET https://registry.npmjs.org/less
npm http 304 https://registry.npmjs.org/less
npm http GET https://registry.npmjs.org/graceful-fs
npm http GET https://registry.npmjs.org/mime
npm http GET https://registry.npmjs.org/request
…
npm http 200 https://registry.npmjs.org/asn1
npm http GET https://registry.npmjs.org/asn1/-/asn1-0.1.11.tgz
npm http 200 https://registry.npmjs.org/asn1/-/asn1-0.1.11.tgz
C:UsersTimurAppDataRoamingnpmlessc -> C:UsersTimurAppDataRoamingnpmnode_moduleslessbinlessc
less@2.0.0 C:UsersTimurAppDataRoamingnpmnode_modulesless
├── mime@1.2.11
├── graceful-fs@3.0.4
├── promise@6.0.1 (asap@1.0.0)
├── source-map@0.1.40 (amdefine@0.1.0)
├── mkdirp@0.5.0 (minimist@0.0.8)
└── request@2.47.0 (caseless@0.6.0, forever-agent@0.5.2, aws-sign2@0.5.0, json-stringify-safe@5.0.0, tunnel-agent@0.4.0, stringstream@0.0.4, oauth-
sign@0.4.0, node-uuid@1.4.1, mime-types@1.0.2, qs@2.3.2, form-data @0.1.4, tough-cookie@0.12.1, hawk@1.1.1, combined-stream@0.0.7, bl@0.9.3, http-
signature@0.10.0)
Node.js NPM

22. Creating Caché Package Manager
Ok, all this good, but what about Caché ObjectScript?

23. Ok, you buy importance of package manager in COS and
want it now, so…
• What should we put to package container?
• And how all this infrastructure will work?

24. • Which container format is the suitable for our packages?
– ZIP would be ideal
– But even Studio Project XML will fit the purpose
• even now it handles all interesting file types (CLS, RTN, INC, CSP, ZEN, CSS, GIF, etc.)
• Yes, XML is very inefficient, bloated, and keeps binary data as base64
(i.e. 3/2x length increase)
• But it’s not a big deal for initial step;
Container Format
Which
Container?
ZIP
XML

25. • What information should be in metadata?
– At least dependency on other packages (name, version, url);
– Anything from Node.js package.json could be reused by us, but;
• JSON is not yet natively supported now,
so could start from XML serialization of the same information (metainfo.xml)
Metadata File
{ "dependencies" :
{ "foo" : "1.0.0 - 2.9999.9999"
, "bar" : ">=1.0.2 <2.1.2"
, "qux" : "<1.0.0 || >=2.3.1 <2.4.5 || >=2.5.2 <3.0.0"
, "asd" : "http://asdf.com/asdf.tar.gz"
, "til" : "~1.2"
, "elf" : "~1.2.3"
, "lat" : "latest"
, "dyl" : "file:../dyl"
}
}

26. TOML
Tom's Obvious, Minimal Language.
By Tom Preston-Werner.
Objectives
TOML aims to be a minimal configuration
file format that's easy to read due to obvious
semantics. TOML is designed to map
unambiguously to a hash table. TOML
should be easy to parse into data structures
in a wide variety of languages.
Metadata File – Alternate Format
# This is a TOML document. Boom.
title = "TOML Example"
[owner]
name = "Lance Uppercut"
dob = 1979-05-27T07:32:00-08:00
[database]
server = "192.168.1.1"
ports = [ 8001, 8001, 8002 ]
enabled = true
[servers]
[servers.alpha]
ip = "10.0.0.1"
dc = "eqdc10"
[servers.beta]
ip = "10.0.0.2"

27. • TOML is easy and clean
– Kind of “old-good” INI format but semantically equivalent to XML and JSON
as serialization format;
• But there is no TOML parser in COS written yet;
– So once it’s ready (probably as a separate “package”) we will add appropriate dependency and
will consume TOML as metadata format.
– Not required now in V1.0;
Metadata File – Alternate Format

28. • There is tough problem to handle – dependency on a system packages which may be
presented only on some product flavors:
– iKnow,
– Ensemble
– DeepSee,
– TrakCare,
– HealthShare,
– particular version of product
• May mark this dependency in the general dependency list as “iKnow”, “Ensemble
2013.1+”, etc.
• But it’s too early to handle in the 1st iteration - we will revisit this later.
Dependency on a system classes

29. • CPAN is successful because we could easily deploy modules which are implemented in C
(some optimized libraries, OS calls, etc.);
– Perl XS API is for such Perl/C modules;
– Cache’ callout API is very similar in their nature, but much more complicated;
– Perl provides a lot of handy utilities which simplify XS modules development, e.g.
• H2xs - preprocessor which generates XS header file using given C header file;
• Xsubpp – preprocessor to convert XS code to pure C code, etc.
• Beyond default callout mechanism there are a couple of more advanced ways fro FFI:
– Builtin FFI mechanism used, for example, by iKnow libraries;
– Or CNA (Caché Native Access) callout wrapper which uses LibFFI mechanism for easy calling
of any C-runtime function from the COS code;
[Cross-platform] binary modules

30. From the practical prospective, taking into account
multiple Caché platforms we should handle equally
well (Windows, Linux, Mac OS X, or even VMS), and
the fact that these FFI mechanisms are not yet
officially supported, we should admit that they are not
ready yet, and could not be recommended as a way
to handle deployment of mixed C/COS packages.
We are not implementing them in v1.0
[Cross-platform] binary modules

31. • Having builtin mechanism for unit-testing has impacted positively the whole stability of
CPAN ecosystem.
– Package will not be installed at the user system unless it’s passing internal testing;
• V1.0 will ignore unit-testing for a moment, but later, for binary modules deployment we
should have unit-testing working
Unit-testing

32. • User experience is a big success booster, if system is inconvenient then it may be
left unnoticed*
• So both ways to invoke package manager should be working
– To invoke package manager shell from any namespace in Cache’ terminal, .e.g
– To install COS packages from command-line, e.g.
• Eventually these operation might be wrapped in GUI. Not in V1.0
Command-line access
do ^%CPM
>install TOML-Parser
cpm install deepsee-mobile-server

33. • There used to be old-dark times when every site had to handle DDoS on their own way,
and to keep geo-spread mirror network for their purposes;
• Now all is easily handled by VM host providers or special CDN providers:
– Amazon Azure, Google Cloud, etc;
– Amazon CloudFront, MaxCDN, etc;
• Once we have static files (distributions files, static html descriptions, static images, etc) it
could be easily handled by CDN;
• But… for community site we supposed to have some dynamic part which will work with
authors and new contributions
Advices are welcome how to handle this mix easily and in most effective way
Mirroring and CDN

34. Suggested usage cases for V1.0
Package User COS System COS Binaries Good for v1.0
iKnowSocial Yes No No Good
Atom-COS-Studio Yes Yes Yes
(Atom +
extension)
Bad
WebTerminal No Yes No Good
CNA Yes No Yes
(callout module)
Bad
Coslint Yes
(deployed
OBJ code)
No No Probably
(deployed)

35. CPM: Contributors Invited!

36. • https://github.com/intersystems-ru/CPM
– Private repository made public today;
– Only 2 contributors so far;
– More is always welcome!
CPM: Contributors Invited!

37. Идём?