2. 2
Siarhei Hladkou
• Senior Project Manager, EPAM Systems
• 11+ years on technical and project lead positions at EPAM
• 20+ years of production experience
• Started programming at 1987
• Built my first Sinclair ZX analog at 1989
• Experience with building platforms:
– Z51, MCS51, Atmel, Analog Devices, PC
– ASM, C, C++, Fortran, Pascal, C#, JavaScript
– Delphi, C++ MVC, COM+, .NET, NodeJS
• Today my favorite is GOLANG
ABOUT ME
4. 4
The modern enterprise
“When we read the word enterprise we probably think of older, slow-moving bureaucracies, like IBM,
HP, or even Red Hat. But it’s been a long time since companies like those have been technical leaders.
Decades, in some cases. Now, companies like Google, Amazon, Twitter, Netflix, Facebook, Spotify, or
even SoundCloud set the tone for our industry. These modern enterprises tend to share a few
characteristics:
• Tech-oriented
• Consumer-focused
• Successful, exponential growth
• 100–1000+ engineers
• A service-oriented architecture
The last point in particular is important.”
Peter Bourgon
5. 5
A service-oriented architecture
“With notable exceptions, most major modern enterprises have adopted a service-oriented (or
microservice) architecture, after very similar patterns of evolution.
1. Start with a Ruby-on-Rails (or equivalent) monolith
2. Break out key components as services, leading to an SOA
3. Maybe, further deconstruction toward microservices
SOAs yield big advantages, but they come at a cost.Any set of network services is inherently more
complex than its monolithic equivalent, owing to the well-documented fallacies of distributed
computing. Managing that complexity usually begins with conventions and best practices, which
eventually evolve into libraries and frameworks. In the modern enterprise, de-facto standards are
beginning to emerge, often as open-source projects.”
Peter Bourgon
6. 6
A service-oriented language
“In this environment, a language like Go has the opportunity to shine. Its coherent and considered
design, developer friendly toolchain, architecture native binaries, and near C efficiency make it
incredibly attractive in a service-oriented architecture, especially when compared against the current
state of the art. Go also punches well above its age-class in terms of library and ecosystem maturity,
and in-the-wild success stories.”
Peter Bourgon
7. 7
“Any organization that designs a system (defined broadly)
will produce a design whose structure is a copy of the
organization's communication structure.”
Melvyn Conway, 1967
Modern monolithic application structure
8. 8
“The microservice approach to division is different,
splitting up into services organized around business
capability. Such services take a broad-stack
implementation of software for that business area,
including user-interface, persistant storage, and any
external collaborations. Consequently the teams are
cross-functional, including the full range of skills required
for the development: user-experience, database, and
project management.”
Martin Fowler
Service boundaries reinforced by team boundaries
9. 9
“While monolithic applications prefer a single logical
database for persistant data, enterprises often prefer a
single database across a range of applications - many of
these decisions driven through vendor's commercial
models around licensing. Microservices prefer letting each
service manage its own database, either different
instances of the same database technology, or entirely
different database systems - an approach called Polyglot
Persistence. You can use polyglot persistence in a
monolith, but it appears more frequently with
Microservices.”
Martin Fowler
Polyglot Persistence
10. 10
Microservices
“In short, the microservice architectural style is an approach to developing a single application as
a suite of small services, each running in its own process and communicating with lightweight
mechanisms, often an HTTP resource API. These services are built around business capabilities and
independently deployable by fully automated deployment machinery. There is a bare minimum of
centralized management of these services, which may be written in different programming languages
and use different data storage technologies.”
James Lewis and Martin Fowler
13. 13
“Microservices Premium - a cost we pay in reduced
productivity to learn and manage this style. As a system
becomes more complex, this premium is outweighed by the
fact that the style reduces the productivity loss that
increasing system complexity imposes on us. But it's a
price only worth paying for more complex software
endeavors.”
Martin Fowler
Microservices Premium
15. 15
The Go programming language was conceived in late 2007 as an answer to some of the problems we were seeing developing
software infrastructure at Google. The computing landscape today is almost unrelated to the environment in which the
languages being used, mostly C++, Java, and Python, had been created. The problems introduced by multicore processors,
networked systems, massive computation clusters, and the web programming model were being worked around rather than
addressed head-on. Moreover, the scale has changed: today's server programs comprise tens of millions of lines of code, are
worked on by hundreds or even thousands of programmers, and are updated literally every day. To make matters worse, build
times, even on large compilation clusters, have stretched to many minutes, even hours.
Go was designed and developed to make working in this environment more productive. Besides its better-known aspects such
as built-in concurrency and garbage collection, Go's design considerations include rigorous dependency management, the
adaptability of software architecture as systems grow, and robustness across the boundaries between components.
From keynote talk given by Rob Pike at the SPLASH 2012 conference in Tucson, Arizona, on October 25, 2012.
Go is a compiled, concurrent, garbage-collected, statically typed language developed at Google. It is an open source project:
Google imports the public repository rather than the other way around.
Go is efficient, scalable, and productive.
Go at Google
Go programming language roots
19. 19
Go does have address the issues that make large-scale software development difficult. These issues include:
• slow builds
• uncontrolled dependencies
• each programmer using a different subset of the language
• poor program understanding (code hard to read, poorly documented, and so on)
• duplication of effort
• cost of updates
• version skew
• difficulty of writing automatic tools
• cross-language builds
Go Benefits
20. 20
import "encoding/json"
The first step to making Go scale, dependency-wise, is that the language defines that unused dependencies are a compile-
time error (not a warning, an error). If the source file imports a package it does not use, the program will not compile. This
guarantees by construction that the dependency tree for any Go program is precise, that it has no extraneous edges. That, in
turn, guarantees that no extra code will be compiled when building the program, which minimizes compilation time.
Dependencies in Go
21. 21
package json
The design of Go's package system combines some of the properties of libraries, name spaces, and modules into a single
construct.
Remote packages
go get github.com/4ad/doozer // Shell command to fetch package
import "github.com/4ad/doozer" // Doozer client's import statement
var client doozer.Conn // Client's use of package
Packages in Go
22. 22
For a systems language, garbage collection can be a controversial feature, yet we spent very little time deciding that Go
would be a garbage-collected language. Go has no explicit memory-freeing operation: the only way allocated memory returns
to the pool is through the garbage collector.
It was an easy decision to make because memory management has a profound effect on the way a language works in practice.
In C and C++, too much programming effort is spent on memory allocation and freeing. The resulting designs tend to expose
details of memory management that could well be hidden; conversely memory considerations limit how they can be used. By
contrast, garbage collection makes interfaces easier to specify.
Moreover, in a concurrent object-oriented language it's almost essential to have automatic memory management because the
ownership of a piece of memory can be tricky to manage as it is passed around among concurrent executions. It's important
to separate behavior from resource management.
The language is much easier to use because of garbage collection.
Garbage collection in Go
23. 23
Go's use is growing inside Google.
Several big user-facing services use it, including youtube.com and dl.google.com (the download server that delivers Chrome,
Android and other downloads), as well as our own golang.org. And of course many small ones do, mostly built using Google
App Engine's native support for Go.
• Many other companies use Go as well; the list is very long, but a few of the better known are:
• BBC Worldwide
• Canonical
• Heroku
• Nokia
• SoundCloud
It looks like Go is meeting its goals. Still, it's too early to declare it a success. We don't have enough experience yet,
especially with big programs (millions of lines of code) to know whether the attempts to build a scalable language have paid
off. All the indicators are positive though.
Summary
25. 25
Bee tool: rapid development
“Bee tool is a project that helps
develop beego rapidly. With bee
tool we can create, auto compile
and reload, develop, test, and
deploy beego applications easily.”
Beego web site
26. 26
Bee tool: new MVC website in 3 seconds
“The new command can create a
new web project. You can create a
new beego project by typing bee
new <project name> under
$GOPATH/src. It will generate all
the project folders and files.”
Beego web site
27. 27
Bee tool: new MVC website in 3 seconds
“The new command can create a
new web project. You can create a
new beego project by typing bee
new <project name> under
$GOPATH/src. It will generate all
the project folders and files.”
Beego web site
28. 28
Bee tool: new MVC website in 3 seconds
To run new application just from
command line just use Run
command.
30. 30
Bee tool: new MVC website in 3 seconds
Go build command is used to
compress the project into a single
file. Then we can run web
application as a single file.
32. 32
Bee tool: new API website in 3 seconds
The api command can create a
new API project.
33. 33
Bee tool: new API website in 3 seconds
Creating documentation and
running.
34. 34
Bee tool: new API website in 3 seconds
Creating documentation and
running.
35. 35
Swagger
“Swagger is a simple yet powerful
representation of your RESTful
API. With a Swagger-enabled API,
you get interactive
documentation, client SDK
generation and discoverability.
Swagger helps companies like
Apigee, Getty Images, Intuit,
LivingSocial, McKesson, Microsoft,
Morningstar, and PayPal build the
best possible services with RESTful
APIs.
Now in version 2.0, Swagger is
more enabling than ever. And it's
100% open source software.”
Swagger web site
36. 36
Swagger
Its possible to review and query
web API with just a browser page.
Immediate results upon your
requests.
37. 37
Swagger
Adding data thru API? Testing
implementation? Checking API
availability? It’s everything is so
simple.
39. 39
Building an API is a multi-step process. The API first gets
designed, resources and associated actions are identified,
the request endpoints, payloads and parameters all get
defined. Once that’s done the design goes trough a
review process: will the UI another team has to build on
top have all the information it requires? will dependent
service X be able to list the resources it needs efficiently?
will dependent service Y be able to update the fields of
this other resource? After a few back and forth it’s time
to actually implement the API. And after a while it’s back
to square 1 with new requirements for APIv2.
GOA: Design-based HTTP microservice development in Go
40. 40
Code Generation
Goa relies on code generation to alleviate the need for
reflection or repetitive coding. The goagen tool analyzes
the API design and produces various artifacts including
code, documentation, tests or even your own output. The
end result is application code that is clean and terse.
Design Based
The goa language allows writing self-explanatory code
that describes the design of the API. The description
focuses on listing the resources and the actions they
support. The goa language is flexible and stays out of the
way, you get to decide how your API should work.
GOA: Design-based HTTP microservice development in Go
41. 41
Goa lets you write the specification of your API in code. It then uses that code to produce a number of outputs including HTTP
handlers that take care of validating the incoming requests. This means that the specification is translated automatically into
the implementation, what got reviewed is whatis implemented.
The final implementation, however, is very familiar looking and can plug-in to many existing HTTP processing packages. HTTP
requests are accepted by the net/http server, routed by a router (goa uses httprouter) and handled by the application code.
The only difference being that the application code is composed of two parts: the generated handler which validates the
request and creates the context object (more on that later) and the user code that provides the business logic.
GOA: Design-based HTTP microservice development in Go
43. 43
GOA: Design language
Now that we have defined our API we need to define the show bottle request endpoint.
44. 44
GOA: Design language
A resource may specify a default media type used to render OK responses. In goa the media
type describes the data structure rendered in the response body. In the example the Bottle
resource refers to the BottleMedia media type.
45. 45
GOA: The Magic - Code Generation
The purpose of specifying the API using a DSL is to make it executable. In Go the preferred
method for this is to generate code and this is the path goa takes. goa comes with the
goagen tool which is the goa code generator.
51. 51
SERVICE MONITORING: What is Prometheus?
Prometheus is an open-source systems monitoring and alerting toolkit originally built at
SoundCloud. Since its inception in 2012, many companies and organizations have adopted
Prometheus, and the project has a very active developer and user community. It is now a
standalone open source project and maintained independently of any company.
55. 55
SERVICE MONITORING: Prometheus
When does it fit?
Prometheus works well for recording any purely numeric time series. It fits both machine-centric
monitoring as well as monitoring of highly dynamic service-oriented architectures. In a world of
microservices, its support for multi-dimensional data collection and querying is a particular
strength.
Prometheus is designed for reliability, to be the system you go to during an outage to allow you
to quickly diagnose problems. Each Prometheus server is standalone, not depending on network
storage or other remote services. You can rely on it when other parts of your infrastructure are
broken, and you do not have to set up complex infrastructure to use it.
When does it not fit?
Prometheus values reliability. You can always view what statistics are available about your
system, even under failure conditions. If you need 100% accuracy, such as for per-request billing,
Prometheus is not a good choice as the collected data will likely not be detailed and complete
enough. In such a case you would be best off using some other system to collect and analyse the
data for billing, and Prometheus for the rest of your monitoring.
57. 57
SERVICE MONITORING: Influxdata
THE “T” IN THE TICK STACK
Telegraf is an open source agent written in Go for
collecting metrics and data on the system it's
running on or from other services. Telegraf then
writes the data to InfluxDB in the correct format.
58. 58
SERVICE MONITORING: Influxdata
THE “I” IN THE TICK STACK
InfluxDB is an open source database written in Go
specifically to handle time series data with high
availability and performance requirements.
InfluxDB installs in minutes without external
dependencies, yet is flexible and scalable enough
for complex deployments.
59. 59
SERVICE MONITORING: Influxdata
THE “C” IN THE TICK STACK
Chronograf is a single binary web application that
you can deploy behind your firewall to do ad hoc
exploration of your time series data in InfluxDB. It
includes a library of intelligent, pre-configured
dashboards for common data sets.
60. 60
SERVICE MONITORING: Influxdata
THE “K” IN THE TICK STACK
Kapacitor is a data processing engine. It can
process both stream (subscribe realtime) and
batch (bulk query) data from InfluxDB. Kapacitor
lets you define custom logic to process alerts with
dynamic thresholds, match metrics for patterns,
compute statistical anomalies, etc.
62. 62
CLUSTER AND BALANCING: GO-CIRCUIT
The CIRCUIT is a new way of thinking. It is
deceptively similar to existing software, while
being quite different.
Circuit is a programmable platform-as-a-service
(PaaS) and/or Infrastructure-as-a-Service (IaaS),
for management, discovery, synchronization and
orchestration of services and hosts comprising
cloud applications.
Circuit was designed to enable clear, accountable
and safe interface between the human
engineering roles in a technology enterprise,
ultimately increasing productivity. Engineering
role separation in a typical circuit-based
architecture is illustrated at the picture.
63. 63
CLUSTER AND BALANCING: GO-CIRCUIT
Users of circuit are
• Operations engineers, who sustain cloud
applications at host, process and network level
• Data scientists, who develop distributed
compute pipelines by linking together and
distributing third-party utilities
• Manufacturers of distributed software, who
wish to codify installation and maintenance
procedures in a standardized fashion instead
of communicating them through
documentation (viz. MySQL)
64. 64
CLUSTER AND BALANCING: GO-CIRCUIT
A few technical features of circuit:
• Single- and multi-datacenter out-of-the-box
• Authentication and security of system traffic
• TCP- and UDP multicast-based (similarly to mDNS) internal node discovery
• Zero-configuration/blind-restart for ease on the host provisioning side
• Global, point-of-view consistent key/value space, where keys are hierarchical paths and values
are control objects for data, processes, synchronizations, and so on.
• Consistency guarantees at ultra-high churn rates of physical hosts
• Command-line and programmatic access to the API
• Integration with Docker
65. 65
CLUSTER AND BALANCING: GO-CIRCUIT
In a typical circuit scenario:
• Provisioning engineers ensure newly provisioned machines start the zero-configuration circuit
server as a daemon on startup.
• Operations engineers program start-up as well as dynamic-response behavior via command-line
tool or language bindings.
Adoption considerations:
• Small footprint: Circuit daemons leave almost no communication and memory footprint when
left idle. This makes circuit ideal for incremental adoption alongside pre-existing architectures
• Immediate impact: Even short and simple circuit scripts save manual time going forward
• Knowledge accounting: Circuit scripts can replace textual post-mortem reports with executable
discovery, diagnosis, action and inaction recipes.
• Circuit servers log all operations in their execution orders, enabling maximum visibility during
post-factum debugging and analysis.
Programming environment:
• Circuit programs (sequences of invocations of the circuit API) are not declarative (as in Puppet,
Chef, etc.). They are effectively imperative programs in the CSP concurrency model, which
allows engineers to encode complex dynamic response behavior, spanning multiple data centers.
66. 66
Example: Make a process
The purpose of each circuit server is to host a
collection of control primitives, called elements,
on behalf of the user. On each server the hosted
elements are organized in a hierarchy (similarly to
the file system in Apache Zookeeper), whose
nodes are called anchors. Anchors (akin to file
system directories) have names and each anchor
can host one circuit element or be empty.
The hierarchies of all servers are logically unified
by a global circuit root anchor, whose children are
the individual circuit server hierarchies.
67. 67
Example: Make a docker container
Much like for the case of OS processes, the circuit
can create, manage and synchronize Docker
containers, and attach the corresponding docker
elements to a path in the anchor file system.
To allow creation of docker elements, any
individual server must be started with the -docker
switch.
69. 69
GO-KIT
“Go kit is a distributed programming toolkit for building microservices in large organizations. We
solve common problems in distributed systems, so you can focus on your business logic.
Go kit composed of multiple co-related packages, that together form an opinionated framework for
constructing large SOAs. The toolkit should be comprehensive, ticking all of the boxes that are
important to system architects. It should do that while encoding strong opinions and conventions,
informed by the operational experience of its contributors, as well as prior art like Finagle. And it
should integrate with the most common infrastructural components, to reduce deployment friction
and promote interop with existing systems.”
Peter Bourgon
76. 76
GO DATABASE DRIVERS: MONGO
mgo http://labix.org/mgo
• mgo offers automated cluster topology discovery and maintenance. Even if provided the address
to a single server in the cluster, mgo will figure out the cluster topology and communicate with
any of the servers as necessary.
• mgo will automatically failover in the event that the master server changes.
• mgo offers a synchronous interface, with a concurrent backend. Concurrent operations on the
same socket do not wait for the previous operation's roundtrip before being delivered. Documents
may also start being processed as soon as the first document is received from the network, and
will continue being received in the background so that the connection is unblocked for further
requests.
• mgo offers configurable pre-fetching of documents, so that the next batch of results are requested
automatically when an established percentage of the current batch has been processed.
• etc…
Running MongoDB Queries Concurrently With Go
77. 77
GO DATABASE DRIVERS: CASSANDRA
gocql https://github.com/gocql/gocql
Modern Cassandra client using the native transport
• Automatic type conversations between Cassandra and Go
• Support for all common types including sets, lists and maps
• Custom types can implement a Marshaler and Unmarshaler interface
• Strict type conversations without any loss of precision
• Built-In support for UUIDs (version 1 and 4)
• Support for logged, unlogged and counter batches
• Cluster management
• Automatic reconnect on connection failures with exponential falloff
• Round robin distribution of queries to different hosts
• Round robin distribution of queries to different connections on a host
• Each connection can execute up to n concurrent queries (whereby n is the limit set by the protocol version the client chooses to use)
• Optional automatic discovery of nodes
• Optional support for periodic node discovery via system.peers
• Policy based connection pool with token aware and round-robin policy implementations
• Support for password authentication
• Iteration over paged results with configurable page size
• Support for TLS/SSL
• Optional frame compression (using snappy)
• Automatic query preparation
• Support for query tracing
• Support for Cassandra 2.1+ binary protocol version 3
• Support for up to 32768 streams
• Support for tuple types
• Support for client side timestamps by default
• Support for UDTs via a custom marshaller or struct tags
• An API to access the schema metadata of a given keyspace
79. 79
marcio.io: cheap MapReduce at Go
Sometimes you don’t need overly complex
infrastructures or systems to do a job well. In this
case, we were running these exact same
aggregations over close to 20 EMR instances that
would take a few minutes to execute the entire
MapReduce job over hundreds of Gigabytes of data
each day.
When we decided to take a look at this problem
again, we rewrote this task using Go, and we now
simply run this on a single 8-core machine and the
whole daily execution takes about 10 minutes. We
cut a lot of the costs associated with maintaining
and running these EMR systems and we just schedule
this Go app to run once a day over our daily dataset.
Original Article
81. 81
Introducing Hugo, a new idea around making
website creation simple again. Hugo flexibly
works with many formats and is ideal for
blogs, docs, portfolios and much more.
Hugo’s speed fosters creativity and makes
building a website fun again.
Hugo is quite possibly the easiest to install
software you’ve ever used, simply download
and run. Hugo doesn’t depend on
administrative privileges, databases,
runtimes, interpreters or external libraries.
Sites built with Hugo can be deployed on S3,
GitHub Pages, Dropbox or any web host.
Original article
Make the Web Fun Again
82. 82
ADDITIONAL READING
GOLANG
• Case Studies
• Go language web site
• Little Go book
• Learn Go
• Go search for libraries
SQL DRIVERS
• SQL drivers for go
• Mongo driver for go
• Cassandra driver for go
MICROSERVICES & CLOUDS
• Martin Fowler
• GopherCon 2015 videos
• CoreOS
LOREM IPSUM DOLOR AMET
• Nulla nu nisi
• Risus purus id fusce
• Lobortis ipsum felis sed
ACCELERATORS
• Beego web site
• Swagger web site
• Auto generated docs
API
• Untagling microservices
• Goa design or API DSL
• MapReduce library marcio.io
UNIT TESTING & MONITORING
• Unit Testing with GoConvey
• Monitoring with Prometheus
• Time series with Influxdata
84. 84
Sometimes you don’t need overly
complex infrastructures or systems to do
a job well. In this case, we were running
these exact same aggregations over close
to 20 EMR instances that would take a
few minutes to execute the entire
MapReduce job over hundreds of
Gigabytes of data each day.
When we decided to take a look at this
problem again, we rewrote this task using
Go, and we now simply run this on a
single 8-core machine and the whole daily
execution takes about 10 minutes. We cut
a lot of the costs associated with
maintaining and running these EMR
systems and we just schedule this Go app
to run once a day over our daily dataset.
Original article
Cheap MapReduce in Go
85. 85
We had several clients building hardware
devices that constantly sent in data,
24/7, and we had to collect it and
process it into something useful.
We built the first version of IronWorker
with Ruby. Our sustained CPU usage on
our servers was approximately 50-
60%. When it increased a bit, we’d add
more servers to keep it around 50%.
After we rolled out our Go version, we
reduced our server count to two and we
really only had two for redundancy. They
were barely utilized, it was almost as if
nothing was running on them. Our CPU
utilization was less than 5% and the entire
process started up with only a few
hundred KB’s of memory (on startup) vs
our Rails apps which were ~50MB (on
startup). Compare that even to JVM
memory usage! It was night and day.
Original article
How We Went from 30 Servers to 2
86. 86
Specific things that we love about the
language, the things we learned along the
way. In no specific order, here they are:
• Performance
• Memory
• Concurrency
• Reliability
• Deployment
• Talent
Concurrency is a huge part of Go with some
high level constructs that make it a breeze
to use. I used Java for many years and got
very comfortable with the java concurrency
package, which is a pretty nice set of tools,
but it’s just not the same as Go in terms of
simplicity or the underlying
implementations.
Original article
Go After 2 Years in Production
87. 87
To see more cases of modern companies using Go in production systems please take a look to
this perfect article The Case For Go.
Want more?
89. 89
TJ Holowaychuk, a prominent module writer, made his farewells to Node.js and is turning to
Go, which he calls a “next generation,” programming language in the same company as Rust
and Julia.
He writes in a post published yesterday that Node.js does some things well but it is not
suited to his interests these days in distributed systems. Node.js stresses performance over
usability and robustness. He finds Go “performs better, is easier to maintain, and has better
test coverage since synchronous code is generally nicer and simpler to work with.”
Summarizing, he says Go is robust for its age and refactoring with types is pleasant and simple.
The tooling for profiling and debugging works and the community has “a very strong
conventions regarding documentation, formatting, benchmarking, and API design.”Go’s
primitives are more suited to distributed computing than Node.js’s generators. Node.js does
not have separate stack error handling. Without it, reporting is mediocre. There’s also Go’s
error-handling, too, which Holowaychuk says is superior.
Say Farewell to Node.js and Hello to Go
Say Farewell to Node.js and Hello to Go
90. 90
A lot of other companies are using Go now including Heroku and Google and the people all have
similar opinions. +Rob Pike, one of the creators of Go said:
“We realized that the kind of software we build at Google is not always served well by the
languages we had available,” Pike said in 2011. “Robert Griesemer, Ken Thompson, and myself
decided to make a language that would be very good for writing the kinds of programs we
write at Google.”
+Derek Collison, the founder of Apcera, said recently in a Wired article:
“The management layers and infrastructure layers of the newer technologies that provide this
cloud delivery model?” he tells Wired. “Within two years, a majority will be written in Go.”
Is Go the next gen language we’ve been waiting for? It’s a bit too early to say, but it’s certainly
off to a good start.
CONCLUSION