SlideShare ist ein Scribd-Unternehmen logo
1 von 32
Downloaden Sie, um offline zu lesen
Perfect Integration




___________________________________________________________
© Copyright 2011 – Martijn Linssen          martijnlinssen.com
2 / 32



   1. Architectural approach to Integration




IT Architecture divides its world in a few parts. Various frameworks have various
models, but I still like Capgemini's one best, since it follows people and organisation
structure. Above, my own visualisation of that

Businesses thrive on business. They do business. Without business, there is no
business - need I continue?
Business defines the business rules, the business exceptions. They make and need
business cases with which they can justify investments, change, growth. Business
events occur at a certain frequency and volume. When all that matches with other
people, business agreements can be made with other companies, allowing them to
do business across the barriers of their own company

Information is in the input for business, and the output for it. Information needs to
comply with the business rules and exceptions. People support the business here by
handling this information: users

Information systems allow the business to scale. They automate what can be
automated, store history, make life as pleasantly as possible for the end-user who
works for the business. They serve as vehicles for the information

The technical infrastructure supports and carries the systems: the iron, the
networks. They form the firm base for the information systems, they are the roads the
traffic travels on

What if two businesses want to do business?
Both parties can’t just exchange information like one would normally do in a
telephone call or in a conversation with someone else; usually the business is too
complex for that. Nor can users do this, as there usually is a high degree of
computerisation involved due to the required speed of processing information, and
the volume of information


___________________________________________________________
© Copyright 2011 – Martijn Linssen          martijnlinssen.com
3 / 32

The information systems themselves can exchange information, but not one system
in the world is exactly alike another one: they always differ in structure, size,
business they support and operating system or database used to support it: therefore
a common layout (interface) needs to be chosen to enable the technical translation
from one system to another

That interface must then be supported by the underlying infrastructure: as these
infrastructures differ from one another as well, a common interface needs to be
chosen here too

On all levels in the enterprise the common theme returns: looking for and defining
common grounds or subsets, and translating those to common interfaces.
If all four levels mentioned (Business, Information, Information Systems, and
Infrastructure) supported these subsets, integration can become a fact

It’s like establishing a common topic to talk about in a conversation with someone
else:

   •   what is the shared interest (Business)
   •   which information do you want to share (Information)
   •   which definitions are mutually exchangeable (Information Systems)
   •   how do you want to exchange ideas (Infrastructure)


In the next chapter, I'll go into detail with regards to the three themes of integration:
messaging, transportation and transformation




___________________________________________________________
© Copyright 2011 – Martijn Linssen          martijnlinssen.com
4 / 32



   2. Common subset and transformation




The common subset found on all levels in the previous chapter was:

   •   what is the shared interest (Business)
   •   which information do you want to share (Information)
   •   which definitions are mutually exchangeable (Information Systems)
   •   how do you want to exchange ideas (Infrastructure)


Information exchange form: messaging
After having agreed on mutual business and information to be exchanged,
transferring information in a certain message layout from and to systems is the
logical next step.
As different systems almost always produce different message layouts, there has to
be a conversion in between these different message layouts: this conversion is
called interface, where a message that originates from one system is converted to a
message that has another system as destination

This conversion is purely on a physical level: the logical and functional information
in the message should not be altered. Good interfacing doesn’t involve more than
simple formatting changes like periods to comma’s or vice versa in decimal fields,
and translation from and to solid standards such as ISO and EAN. There are many,
many standards on the business document level such as EDIFACT and X12, HL7,
SWIFT, ClieOp, etcetera. Normally, in mature and high-volume high-frequency
industries such as banking, automotive and logistics, these standards are easy to
find and hard ( = expensive) to deviate from.
In less mature industries or those closer to government and non-profit, one will find
that people find the leisure to play around with non-interoperable standards by
usually inventing their own XML and SOAP.

Information exchange method: transportation
Messages need to be transported from the originating system to the destination
system (and vice versa). Ideally, a communication protocol must be chosen that is
supported by all parties involved. A communicastion protocol is nothing more than a
worldwide standard: telephone, television, fax, telex, the internet: all thouse wouldn't
have been possible without simple, static, clear and unambiguous standards.
Fortunately there aren’t as many communication protocols as there are
application systems, so a choice can be made. Communication protocols are, as
opposed to information systems, highly standardised on a global scale. Therefor a
choice can be made, instead of having to convert one transportation protocol to the
other. Next to that, almost all (simple) transportation protocols are free. The ones that


___________________________________________________________
© Copyright 2011 – Martijn Linssen          martijnlinssen.com
5 / 32

are more complex and offer a higher degree of functionality and built-in reliability
usually aren’t free

However, when integration is considered to be “door-to-door”, meaning starting
at the very originating system at e.g. one company, and ending at the very
destination system of e.g. another company, there almost always is some kind of
conversion involved, as chances are slim that both systems are connected end-to-
end on the exact same common communication protocol. From a security point of
view there also are many restrictions that simply won't allow for a direct connection
between an internal system and external system

The integration topic: transformation
Information exchange is the only goal of integration.
The only purpose is doing business on a cost-effective basis in order to make or
save (more or better) money.
The integration theme is to establish a common subset, and this subset can
physically be realised via transformation.
Messaging and Transportation can be defined as the two main methods of
integration.
Transformation is the sole integration topic: business, nor information, nor
information systems, nor infrastructure are alike when integration is exercised on an
enterprise scale

Standardisation: dynamics and statics
Standardisation is key to integration. Standardisation makes matters static, and worth
the effort of translating. Not standardised or volatile things are dynamic, and
impossible or at least very costly to automate, and keep automated

Gas, electricity and water are so standard that we can actually put the plumbing in
our floors, ceilings and / or walls. Gasoline is so very standard that we can get it
everywhere, because it will always be the same.
The fact that these products are so very static as well as standard, is no
coincidence. Electricity, gas and water nor gasoline have changed over the last
decades or centuries.
Like I have argued in every single chapter so far, standardisation resides in the lower
architectural IT layers, and not in the top ones, as dynamics at the top, in the
business where people are involved, are far larger than at the bottom, the
infrastructure where machines are involved.
No one has rewritten the TCP/IP stack or the HTTP protocol. Have databases
significantly changed?

So, standardisation in IT can be achieved, and already is partially achieved.
But, it will only be achieved “down below”, and not in the business area. Business
simply is way too dynamic for that

So how do we achieve standards between dynamic businesses? That, in the
next chapter




___________________________________________________________
© Copyright 2011 – Martijn Linssen          martijnlinssen.com
6 / 32



   3. Common language: semantics first




This chapter will elaborate on messaging and transformation (part I), and
explain how information exchange works in the daily world, considering simple or
complex information exchanges. That will then be related to IT, and the basic ways of
“writing down” information in IT will be explained.

If you want to have a chat with someone else, what do you do?
First, a common theme to discuss has to be found - usually not that hard. The
favourite ones in e.g. IT usually are new technologies, cars, and the occasional
sports.
The topics then have to be found, which is a bit harder. Do we discuss hardware
technologies, or software technologies, or is it architecture or open source we want to
discuss?
If the topics have been identified, last but not least the definitions have to be
checked: are we talking about the same subjects?
The Greek philosopher Socrates filled entire conversations, and books, with
merely trying to establish that he and his counterpart were talking about the same
thing. There are numerous occasions on which he talked half a dozen times or more
to someone, only to mutually agree on the fact that they didn’t mean the same thing
while using the same word, and then end the discussion.
Nowadays this is a serious topic in integration: establishing a common mindset
and “dictionary”. Semantic Interoperability is a very interesting and worthwhile item
to closely investigate and embrace, before engaging any inter-company integration.
Even when conducting integration within a mid-sized enterprise itself, it is very
valuable to have a common dictionary that explains in detail what words mean to all
parties involved.
After all, what is the use of a service-oriented enterprise and its service repository if
the function of a certain service can be interpreted in more than one unambiguous
way?
So, for argument's sake, let's assume that you have

   •   an elaborate dictionary of all the relevant business words used in your
       enterprise
   •   and their full, unambiguous meaning

___________________________________________________________
© Copyright 2011 – Martijn Linssen          martijnlinssen.com
7 / 32

   •   complete with the structures they fit in
   •   and the order in which they do so
   •   for each and every business process they occur in


Now, on a semantical level, everything is clear: the most important part has been
successfully completed. Secondly, agreement has to be reached on another level:
the syntactical part

When setting up information exchange between two parties, life can be simple. A
common form will have to be chosen, with which both parties feel comfortable: is it
OK to converse by fax only, or is the topic so intimate that it can only be done face-
to-face? When living in entirely different time zones, it will be practical to have a way
of conversation that is not that interactive, e.g. writing old-fashioned letters or
sending emails. And when living far apart, it is a costly business (well, at least it
always used to be) to make long phone calls.
In IT, this common form is (again) called an interface. The interface chosen should
always be a message, a static representation of an event. The idea is that an
information system itself takes its dynamic, internal data, and exports that outside of
its own system in the form of a static message. That message will always reflect the
state of that small subpart of the system at that specific time.
The other system is expecting a similar message, and will import that into its own
system, using the static data to update its own dynamic data.
To bridge the gap between the two systems, the one static message will have to
be converted to the other static message, preferably outside the systems

This transformation of messages, which involves designing and creating interfaces,
also known as interfacing, can be compared to the day-to-day world of translation
of languages. People who speak different languages can’t understand each other
unless one speaks the language of the other. However, the scale on which a
language should be mastered differs considerably from that of a simple interface or
service.
On the other hand, if you just met this mysterious foreigner and only want to
exchange a few romantic words over a warm campfire, usually a 30-minute crash-
course suffices.

But, being an application or system in an enterprise IT-landscape, there regularly is a
need for far more than just mere pleasantries: it therefore is fair to compare
interfacing and servicing to the every-day art of translating and interpreting
languages




___________________________________________________________
© Copyright 2011 – Martijn Linssen          martijnlinssen.com
8 / 32



   4. Common language: direct translation




This chapter is about my favourite subject: translation. Long, long, long ago I
aspired to become an interpreter - but changed my mind. Still, I consider myself a
linguist and a good one at that, speaking 6 languages of which 3 fluent. The
existence of extreme diversity in languages of the world, and the same existence in
diversity in an average enterprise IT landscape have always hooked my attention

Just like people speak different dialects and languages, so do applications: not
one of them can perfectly understand the other without changing a thing. Even when
interfacing between SAP systems, there will be differences on the technical level that
have to be resolved.
So, how do you solve these misunderstandings at the technical level? How do you
"translate" between the different languages and dialects spoken? One of the sides
has to adapt, or maybe both? When, and how?
To answer that, let's look at the everyday world of natural languages. A completely
different world, with an identical problem - and a working solution

Learning a language can be very hard and time-consuming, and very few will
ever master a non-native language perfectly. Most of us know someone within the
same country that has an accent and just can’t seem to get rid of it; those that have
connections across the border might have heard other people speak their language
and have difficulties understanding them. Or one has heard a colleague or neighbour
speak a foreign language and, while not being able to speak that language fluently
themselves, has good reason to suspect that there will be people on the other side
having difficulties understanding that “version of the language”

In short, a language spoken by a so-called non-native speaker always has a
certain external “flavour” to it. If a Dutchman, a German and a Frenchman all have
to learn English in order to exchange information with each other, it is almost a
guarantee that that English will have elements of all those three languages in it. It will
not be British English, nor will it be American English; it will be some kind of English
that sounds a bit like English combined with Dutch, German and French. Dunglish,
Genglish and Frenglish combined: it will make for a hilarious comedy, unless it's a
serious professional occasion.
Learning a language takes time, money, and last but not least talent

In professional business, therefore, often interpreters are used: people who have
made it their profession to be fluent in at least 2 languages, and in that way can
___________________________________________________________
© Copyright 2011 – Martijn Linssen          martijnlinssen.com
9 / 32

interact between two persons or parties who only speak one of them. There no
longer is a need for individuals to master a foreign language: people can speak their
own, and the interpreter will just translate them.
Professional interpreters come at a price: it is obvious why not everyone can
afford one, and thus has to learn a foreign language himself

On the other hand, some people can’t afford the time needed to invest in a foreign
language: it is of course inconceivable that a president or CEO is sent on a language
course for several years in order to master a language: that simply is an investment
that would never render a return. Even worse, a person in that position would not
have to be fluent in only two, but perhaps a dozen languages. So, learning a
language yourself comes at a price too. Mastering a language in a perfect way is
"priceless" for most

Moving the interpreter into place provides not only a short term benefit, but also a
long-term one: if the president or CEO is ever succeeded, the successor doesn’t
have to be sent on a language course either. One interpreter: a lifetime
investment that returns every single day. And the best news? Interpreters
themselves are fairly disposable too...

So, in certain strategic positions within a company, it is a very wise investment to
make use of interpreters. In general those positions are characterised by people that
have unique, expert knowledge and skills that make them highly valuable for the
company. A lot of time and money was invested to either cultivate or buy those
specific characteristics: one can easily compare that to the careful selection of a
business-critical application or system within the IT-landscape

Translating one language to the other, however, whether done by people themselves
or with the help of an intermediate like an interpreter, is still not eliminating
dependencies: there are only exactly two languages involved, and the only
translation possible is based on the combined knowledge of those two. Here's what
happens when you do the math on that:

When e.g. a German, a Frenchman and an Englishman would participate in a
conversation, there would be a need for three interpreters: one that translates from
German to French (and vice versa), one from German to English (and vice versa),
and one from French to English (and vice versa).
If another person would participate, e.g. a Dutchman, three additional interpreters
would be needed: one that translates from German to Dutch (and vice versa), one
from English to Dutch (and vice versa), and one from French to Dutch (and vice
versa)

Two parties, one interpreter; three parties, three interpreters; four parties, six
interpreters: this is going the wrong way!
It seems like there is a formula for direct translation and parties involved: for n parties
involved, there are n/2 * (n – 1) interpreters needed: one interpreter for every two
parties (n/2), and as many interpreters as there are other languages (n – 1)

That formula exists, is widely recognised and true, and that is why in real life
the solution is only one: next stop, the European Parliament

___________________________________________________________
© Copyright 2011 – Martijn Linssen          martijnlinssen.com
10 / 32


   5. Common language: indirect translation




This chapter is about indirect translation, in contrast with the direct translation
shown in the previous chapter, which came with costly, exponential dependencies

When looking towards large-scale use of translators, e.g. the European Committee
in Brussels, it is easily observed how these dependencies can be greatly reduced: all
languages are first translated into English, French or German, after which either one
of those is translated into the final destination language

Of course this is doubling the effort involved, as everything has to be translated
twice. However, an intermediate channel is created that everyone can use: compare
that to villages and towns, each digging and building their own roads in order to travel
towards each other, or a municipality that builds a highway for all to use; in this case,
all that has to be done is just construct a small piece of road to connect to the
highway, considerably reducing the effort, time and money involved

Thus, in the European Parliament a common language is defined, viz. English,
French and German. In the specific example of the European Parliament the more
than 500 possible combinations are now reduced to 75.
For political and linguistical reasons the choice is made to support more than one
intermediary language in the European Parliament for the purpose of interpretation:
real-time translation between humans. Spanish and Italian are languages that are
very closely related, but Serbian and Chinese have nothing in common. The chance
of finding an interpreter that speaks Italian as well as Spanish is far greater than
finding one that speaks Serbian as well as Chinese

In general any language would suffice for discussing any topics: grammar and
syntax allow for carrying across messages that are complicated as well as simple.
Languages that couldn't be compared to or fit onto each other could be languages
that are far apart on an evolutionary scale, for instance languages that are only used
by a few people living in tribes in the Australian bush and only have a small
___________________________________________________________
© Copyright 2011 – Martijn Linssen          martijnlinssen.com
11 / 32

vocabulary and no noticeable grammar, in relation to an average language in the
modern world with a vocabulary of tens of thousands of words and a complicated
grammar that allows for a few different past tenses.
In IT, there are such differences as well: if you have to hand-translate in between an
old-fashioned mainframe and XML, the distance would be simply too great. You can
land a Boeing on a highway in case of emergency, but if you do so on a trail you're a
lot less likely to succeed

So what needs to be done in information exchange, after establishing the semantics,
is finding a common language: in interfaces, this common language is formed by a
common, or rather generic, intermediate form. Both parties can (and rather, should)
still speak their own language, but the translation is done "in the middle".
This interface, like the languages, needs to be able to hold the logical structure of
the desired information interchange, and the functionality desired.
When this logical and functional structure has been designed, a format needs to be
chosen in which this can reside. As with languages themselves, it doesn’t really
matter what exactly this format is, as long as it is easily accessible, commonly
available, as simple as possible and fit-for-purpose

Translating that to IT, first of all a logical and functional design has to be made to
determine the scope of the common subset, and determine which exactly is the
definition of the various (sub)topics and functional entities, as shown in chapter three.
Such constructs are generally referred to as Canonical Models, or Common Object
Models.They effectively establish a business language that is spoken within the
enterprise, making the extreme diversity of the underlying IT-landscape fully
transparent, or rather, from a business point of View, irrelevant

Canonical Models do, what IT should have done ages ago: let the business not
bother about the trivialities of IT.
Direct translation enables this in an efficient way: rather than sending specialised
applications on a language course and teach them the business language of their
freshly-joined company, the translators will do that work for them.
Indirect translation perfects this in the best cost-efficient way: professional translators
translate the diverse application mumbo-jumbo to the generic company business
language: a win-win-win situation

This concludes the first 5 chapters, dealing with the semantical level. The next two
chapters will go one level deeper, and drill down to the syntactical level. After that,
we'll resurface to the semantical level again, to deal with addressing in general

In the next chapter, a word about interface message formats, or rather: if I were
to define such an interface, what are my choices?




___________________________________________________________
© Copyright 2011 – Martijn Linssen          martijnlinssen.com
12 / 32



   6. Common language: syntaxes




In the previous chapters I explained semantics, syntax, and the fastest, cheapest and
easiest way to get from diverse IT applications to one uniform business language.
This chapter will take a deep dive into message formats such as Flat file,
EDIFACT, XML and JSON

Ever wondered about the pros and cons of XML? JSON? What it is really? What
other possible message syntaxes are?
When semantics and a logical structure has been defined, and functional groups and
fields have been identified, just about any message format can be chosen.
Choosing message formats (or rather, predefining them) is a rather pragmatic
measure that narrows down the margin for discussion when it comes to choosing a
form for your common language

Message formats can be divided into two characteristics: length and dimension.
Length can be fixed-width or delimited, and dimension can be are horizontal or
vertical. Basically, that's all there's to it

Fixed-length messages have records and fields, containing the information, that
have a fixed width. The same type of information will always be found at the same
location (position) within a record or field. Fields not fully used (the information in it is
physically shorter than the maximum length of the field allows for) will be filled with
spaces or zeroes, depending on the data type of those fields.
E.g. An empty record consisting of 5 fields, with a total length of 500 characters, will
always be 500 characters long

Delimited or variable-length messages have records and fields, containing the
information, that have a delimited width. All records and fields are delimited by so-
called delimiters, i.e. commonly defined characters that indicate the end of a record
or field.
The same type of information will always be found after the same delimiter. Fields not
fully used (the information in it is physically shorter than the maximum length of the
field allows for) will be just totally empty
E.g. an empty record consisting of 5 fields, with a total length of 500 characters, will
always be 5 characters (delimiters) long




___________________________________________________________
© Copyright 2011 – Martijn Linssen          martijnlinssen.com
13 / 32

Horizontal messages look much like traditional files: fields within a record are found
at the same line, one after another. This message format is the most common and
resembles files or database-tables

Vertical messages look much like a single-row table: fields within a record are found
each at a new line, one line after another. This format is not widely used but solves
the fixed-width problem in an easy way: each field starts on a new line, preceded by
a so-called “tag” that identifies the function of the field. The field ends when the last
character on that line is encountered

Nowadays EDIFACT is the most common language used, mainly for its global
approach, broad business document standards (over 200 different functional
messages) and the fact that its delimited type allows for minimum message size. It is
in existence since 1986, and evolved on ANSI X12 (1979). ANSI X12 is in use in the
United States, whereas EDIFACT is widely used in the rest of the world

XML was gaining grounds, but due to lack of standards, and the fact that it is an
unlucky combination of a delimited interface with fixed-width field names it will not
play a large role any time soon, unless it is pushed and supported by organisations
such as the UNECE. Evangelising Web Services as having to be encapsulated in
XML however have sped up the evolution of it.
Additional push is delivered by most new applications offering XML-support
nowadays when it comes to disclosing information.
A general pull is delivered by Google, Twitter and Facebook moving away, or having
moved away already, from the format. Facebook is on the verge of doing so, Google
never has used XML, and Twitter deprecated its use last year. Both Twitter and
Facebook have embraced JSON, a far more simple and concise message structure
that serves the same purpose

IDOC is a standard invented by SAP that is just a simple fixed-width interface,
however supported by large organisations. Needless to say, however, that as SAP-
implementations differ all over the world, so do the iDOCs

Information exchange via physical files goes back decennia. A decade ago links
started being made via COM, CORBA, DCOM etc, during which information wasn’t
exchanged between systems via physical messages but via memory objects.
This in fact meant that applications were tightly coupled via a virtual point-to-point
interface. Development and maintenance of such solutions has proven to be (much)
too expensive

There’s a lot of talk about loosely-coupled things these days: the ancient hard-
coupling (point-to-point interfacing, COM, CORBA, etc) appeared to be very time-
consuming and costly with regards to development and maintenance

Nowadays it’s architects that use the term loosely-coupled: some of them think
that applications should be loosely-coupled towards the architecture, instead of vice
versa. These never are business-architects, but IT-architects. Business architects
know that applications change at high speed, and that -fast changing- business
needs should be supported by IT, being able to move in or out applications or


___________________________________________________________
© Copyright 2011 – Martijn Linssen          martijnlinssen.com
14 / 32

systems when need be - the European Parliament approach as sketched in the
previous chapter.
Loosely-coupled means that applications and systems should plug in and out of the
enterprise almost overnight. Plugging in an application should happen as fast and
cheap as possible. Regard it as just another politician speaking in Brussels: just
another (relatively cheap) translator has to be found, after which business can
continue as usual

So, applications must always speak their native language. This is the most
simple and cost-efficient idea, leaving the application to do what it’s best at: provide
functionality. The translator will provide its own specialty, which is translating
messages from any language into any other language - the common language that
serves your company as a uniform, tool- and platform independent business
language that is relatively timeless and understandable to all

For an Oracle Siebel CRM application this usually means speaking Siebel XML, for
SAP this usually means speaking iDOC; a mainframe will feel most comfortable with
fixed or delimited flat files, etc.
If you visit China, you'll talk Chinese, in Greece it's Greek, and when wanting to
speak English you better distinguish between UK English, US English, Australian
English or, what is generally in use in social media networks: global English

This was the deep dive into messaging formats, concluding the information
exchange form: messaging. Now let's resurface and address the information
exchange method: transportation




___________________________________________________________
© Copyright 2011 – Martijn Linssen          martijnlinssen.com
15 / 32



   7. Information exchange: transporation




[Image courtesy of Ferdinand Reus]

After creating and or choosing a common or generic format to exchange the
information, there is one other field to explore: the facilitation of various
communication protocols through which this information can be transported

What applies to messages, also applies to transport: a common language is to
be advised as "main artery" for all the traffic. Besides that, each application must
speak its own language here as well, and use its own infrastructure as much as
possible. Adjusting applications for transport protocols can be even harder than doing
so for messages: transport protocols are very, very static and rely on worldwide
standards. Making your own version of it is highly unappreciated

Transportation of data is, like integration itself, a necessary evil. It doesn’t really
matter how information gets from A to B, as long as it stays the same. Information
exchange by letter, fax, phone, email, SMS, video etcetera, will always be in different
formats, but the general contents will always be the same. It will be influenced by
certain boundaries and limitations inherent to each protocol, but it won’t change in
essence - fortunately!

The average opening scene of a James Bond movie sets a good example of the
infrastructural variations one can encounter: almost all vehicles imaginable are used
by Bond while "travelling from A to B" (usually that means escaping death from his
pursuers). The same applies to a message that needs to be transported: on the
enterprise level there's nearly all transport protocols one could think of, and ideally
they should all be supported – within reason

A glance at all possible ways to get information across
The most widespread and easy to use communication protocol is via file systems.
Usually, networks and applications within a company are "close enough" to find a
___________________________________________________________
© Copyright 2011 – Martijn Linssen          martijnlinssen.com
16 / 32

mutual place where they can drop off and pick up files, just like one would at or from
a post-office.
For intra-company transportation this might suffice, but inter-company information
exchange calls for a protocol that can easily connect to and from different operating
system

In the old days, FTP (file transfer protocol) was the easiest and most widespread
protocol. It is free, widely available, runs on all platforms, and capable of being a
separate layer on top of entire applications, systems, enterprises. It is limited in its
commands itself but allows for all the desired transportation methods.
Other common protocols are SMTP (the protocol used for email), and HTTP (the
protocol used for web). JMS is quickly gaining grounds because it’s a queuing
mechanism, and free. Queuing systems offer a high degree of reliability, with
guaranteed delivery and retry-mechanisms built-in

There is a growing need for security in general, with inter-company communication
spreading "around the world". Dedicated bilateral communication lines are no longer
the default, communication is now largely based on transportation via internet.
AS1 and AS2 are secure applications of SMTP and HTTP, respectively. Both
specifications were created by EDI over the Internet (EDIINT), a working group of the
Internet Engineering Task Force (IETF) for developing secure and reliable business
communications standards.
The added value of AS1, AS2 and AS3 is the fact that they offer so-called “non-
repudiation”: on successful physical reception of a file a receipt is sent back in return,
through which it is indisputable that the message has been sent and received. These
work with certificates on both sides, so all parties can authenticate themselves in a
trusted and secure manner

As security is a hot topic when it comes down to communication, guaranteed
delivery is one when it comes to transportation.
The basic way to ascertain guaranteed delivery is to receive notifications for each
message having been delivered, from the recipient. This way there's always proof
that a message has been delivered. When e.g. an order appears to not have made it
to the warehouse it saves a lot of time when the sender has a receipt of the actual
message sent that contained the order

In the old days proof of delivery was given by keeping log files of messages sent and
received, using checksums, etc. Proof of delivery was something that could be
requested in exceptional situations, with manual intervention by people. With the use
of a.o. queueing systems such mechanisms became built-in, and nowadays
guaranteed delivery is something that has moved from being available in exceptions,
to becoming part of the rule

Again, on the transportation level: a good diversity in between them all, and a
common subset needed in order to be able to share the same means of
information exchange. Next up: how to link messaging to transportation




___________________________________________________________
© Copyright 2011 – Martijn Linssen          martijnlinssen.com
17 / 32



   8. History of the last decades
In the first seven chapters, the approaches to Integration have been shown. The
architectural top-down approach, the common subset theme, and the central
integration methods: messaging, transformation and transportation.
Now the approach to successful integration has been set out, a brief story about
integration in the real IT world, over the last decades, is in its place

Point to point interfacing

Point to point connections, is a term that describes how applications can be
connected. Point to point interfacing was commonly used to link applications together
"in the early days", and was a reasonable solution as long as the scale of integration
stayed small




Point-to-point interfacing involved applications having to speak each other's
language; the resulting IT-landscape was a tightly interwoven one, where a lot of
effort and time was spent in information exchange. Replacing an application by
another one meant that all connected applications had to change their interface to it
at the same time.
Point-to-point interfacing has proven to be complex, costly and unreliable for any
mid-sized enterprise or up.
Point-to-point interfacing lacked centralisation and standards, making it very hard to
trace and solve issues

There is a formula for point-to-point interfacing: it is n * (n - 1)

That means that for every connected / interfacing application in an enterprise, there
are n * (n - 1) connections: each application (n) needs to support an interface to all
other applications (n - 1)

___________________________________________________________
© Copyright 2011 – Martijn Linssen          martijnlinssen.com
18 / 32



However, the enterprise formula is not that interesting: the formula that is valid for
each connected application, has far more impact: 2 * (n - 1)

That means that every connected / interfacing application has to support 2 * (n - 1)
interfaces: 1 * (n - 1) to give information to all other applications connected (as an
answer to their request), and 1 * (n - 1) to do so the other way around (requesting an
answer from them)

Point-to-point connections thus posed a considerable problem to a company: the
more widespread its usage became, the harder it was to maintain and monitor.
It in fact reduced all flexibility with regards to upgrading or replacing applications to
almost zero

Moreover, point-to-point connections are hard to build, as they form a direct link
between 2 applications: the combined knowledge of both applications is needed to
support them

Point-to-point connections are hard to support, as they're usually built as
'temporary' interfaces without much documentation, although that is just a historical
note and has little to do with the (lack of) success of point-to-point interfaces
themselves.
Point-to-point connections are hard to control, as they're mostly built on top of an
existing application, meaning that a proprietary application has to bend and stretch
itself in order to comply with another application, and vice versa: and that repeated
many times

As an answer to the problems point-to-point interfacing caused, EAI was developed

Enterprise Application Integration (EAI) is a term that describes how applications
can be connected. EAI is commonly used to link applications together, and is an
answer to the disadvantages of point-to-point interfacing.
EAI introduces a centralised hub, that takes over the interfacing effort from the
surrounding (i.e. connected) applications. Not only is the translation performed by the
hub, but centralisation also manages to bring a great span of control to an enterprise.
Within the hub, there usually still is point-to-point interfacing involved.
However, EAI-tools are very well suited to do so, thus lifting the burden from
applications to the right place.
With EAI, replacing an application by another one means that all connected
applications don't have to do anything: only within the hub an effort would have to
be made.
EAI interfacing has proven to be complex but very cost-efficient. However, it didn't
solve the diversity-problem




___________________________________________________________
© Copyright 2011 – Martijn Linssen          martijnlinssen.com
19 / 32




There is a formula for EAI: it is n2

That means that for every connected / interfacing application in an enterprise, there
are n2 connections: each application (n) needs to have an interface translated by the
hub for each application (n)

The enterprise formula for EAI is thus greater than that of point-to-point interfacing;
on the other hand the formula that is valid for each connected application, has
become smaller: n

That means that every connected / interfacing application has to support n
interfaces: 1 to give information to all other applications connected (as an answer to
their request), and (n - 1) to do so the other way around (requesting an answer from
them)

EAI connections are hard to build, as they form a direct link between 2 applications:
the combined knowledge of both applications is needed to support them.
EAI connections are easy to control, as they're centralised in one tool and place.
As an answer to the problems EAI interfacing didn't solve, ESB developed
An Enterprise Service Bus (ESB) is a virtual bus through which services travel
within an enterprise.

   •   An ESB's primary goal is to offer a very high degree of standardisation
   •   An ESB encompasses services, which are usually, but not exclusively, Web
       services


Those services are wrapped in envelopes which are usually, but not exclusively,
designed according to SOAP.
There's no consensus about whether an ESB does, or doesn't, transform messages.
Some say it's not part of it, others say it is but that it should happen "outside the bus".


___________________________________________________________
© Copyright 2011 – Martijn Linssen          martijnlinssen.com
20 / 32

The strict version of an ESB is that the connected applications all comply with the
standard message and protocol used in the bus, and the ESB itself only routes the
messages - establishing a clear difference between itself and Enterprise Application
Integration (EAI)




There is a formula for ESB: it is n

That means that for every connected / interfacing application in an enterprise, there
are n connections: each application (n) needs to port their own interface to the bus.
On the application level, the formula is also n.
That means that every connected / interfacing application has to support n interfaces:
1 to give information to all other applications connected, and (n - 1) to do so the other
way around.
Formula-wise, ESB is the best of all solutions so far

ESB connections are very hard to build, as they require the ability to support highly
customized messages from just day-to-day applications. ESB connections are hard
to control, as they're mostly built on top of an existing application.
ESB services involve applications having to speak the language of the bus; with
this approach, IT is in fact back to the point-to-point interfacing of some
decades ago, although the diversity in messages exchanged has now become
slightly less. On the application level, a lot of effort and time is spent in information
exchange.
Replacing an application by another one means that the new application has to
support all the existing interfaces in the IT-landscape before it can be connected.
ESB is a very nice thought on the IT enterprise level, were not much has to be done:
on the application level however, ESB interfacing has proven to be very complex,
costly and time-consuming. Most applications, that aren't specialised in creating
highly customised interfaces, produce unstable and unreliable interfaces

From a business and ROI point of view, pure ESB's are financial disasters


___________________________________________________________
© Copyright 2011 – Martijn Linssen          martijnlinssen.com
21 / 32

In the next chapter, which will be slightly shorter, the differences between all
these approaches, and of course the pros and cons




___________________________________________________________
© Copyright 2011 – Martijn Linssen          martijnlinssen.com
22 / 32



   9. History with hindsight




In the previous chapter, the history of Integration passed: point-to-point, EAI and
ESB. For those who read and grasped chapter 1 through 7, it'll be clear why I favour
which one - but let me explain it in more detail

What are the differences between the different historical approaches?

The crucial difference is that EAI describes a hub and spoke architecture, where
proprietary messages to and from applications are translated by a central integration
broker. Yes, that is the European Parliament chapter, among others

As described, applications can plug in and out of the landscape this way, and the
necessary transformation of information is taken care of by the central hub

In an ESB, there is no hub: the hub has become a bus. Transformation of
information is done by the applications themselves, who thus custom-built their
(small) integration engine which does the job. If you want to plug into the Matrix
there, you have to follow all the courses - and that takes a lot of time and money,
unlike Neo learned combat

There are advantages and disadvantages to EAI as well as ESB

Greatest advantage of EAI is its cost effectiveness, and the fact that it can support
just about anything.
Greatest advantage of ESB is the fact that it doesn’t support just about anything, but
needs and enforces a highly standardised environment

The biggest disadvantage of EAI is the fact that scalability might become an issue
because everything goes through the central hub, but that worry was years ago. The
biggest disadvantage of an ESB is the fact that there are either numerous and
various integration brokers in the landscape, each tightly coupled to the applications
themselves, with a highly specialized knowledge required, and a lot of licensing costs
involved, or add-ons to existing applications and systems that might not be part of the
default architecture in mind.

___________________________________________________________
© Copyright 2011 – Martijn Linssen          martijnlinssen.com
23 / 32

In either case, they're mostly uncontrolled, bespoke, and hard and costly to maintain.
And basically, in this scenario, the ESB does nothing else than route messages -
now how exciting is that?

From an IT enterprise point of view, there is not much of a difference between
properly conducted EAI and ESB: applications are decoupled and accessible via
services (or standardized interfaces) and as such appear to be plug-and-playable to
the business. From a business PoV, however, cost plays a role, time-to-market, and
flexibility: and that's where pure ESB's lose hard, compared to EAI

It is somehow even strange that EAI has been “followed up” by ESB, as hardware
costs have dramatically decreased over the last few dozen years.
On the other hand, integration has become so obvious that most packages now
offer a small integration suite with the system itself. However, these are still as
monolithic as the majority of those systems themselves, and never capable of
handling enterprise-wide integration if there are more packages involved than just
their own (cf. SAP XI / PI. Getting better every year, they pale in comparison to
integration brokers 15 years ago). They are small message brokers, capable of doing
some lightweight transformation at best

In order to make ESB viable, integration suites coming along with entire systems
should be fully able to handle Enterprise Integration, and they should be included in
the system, not sold separately as an option. If it’s not part of the default package,
where is the benefit? In the 21st century, every single application should be able to
integrate with other applications by "disclosing their functionality" in the form of a
message.
Message transformation will always be necessary, as there can be no such thing
as a worldwide agreement on business documents, as the experience of the past 30
years has clearly shown. No matter which standard exists, parties will always make
their own subset, or grow away from it; compare this to the average dictionary that
describes a full language, and of which no more than very small subsets are used in
day-to-day communication

So, currently, put your money on the hub-and-spoke architecture of EAI, due to
the fact that it is still very -very- costly to realise a true Enterprise Service Bus within
most companies the pure ESB way. It is far easier, faster and cheaper to use the EAI
model and a Canonical in order to achieve a uniform language across your diverse
IT-landscape

In chapter no. 10, I'll address the magical mechanism that enables all of this: one
uniform business language across an ocean of diverse IT applications, messages
being translated from any language into an intermediate one, and everything being
seamlessly transported from A through B and back to finally Z; the most important
part of integration, usually if not always forgotten:

the envelope




___________________________________________________________
© Copyright 2011 – Martijn Linssen          martijnlinssen.com
24 / 32



   10.       The missing link: envelope




With a common language, a common transport protocol, and the need to exercise
the necessary translation and transformation on both levels in between, there is a
growing need to be able to identify all "service requests" on a generic level too

Numerous and various requests will be made, in different formats, via different
transport protocols. This certainly is not Utopia, but a pragmatic observation that is
maintained in almost an evolutionary way: within successful companies IT systems,
like people, are selected based on professional excellence (meaning specialist
instead of generic properties).
Excelling in one area usually means being moderate in a few others, and being
capable of good and generic integration is usually one of those. Intellectual
supermodels, and good-looking scientists: they are either scarce or non-existent

When all those requests are being made in different ways too, it becomes nearly
impossible, but at least very time-consuming and costly, to collect and streamline all
the information. Whatever the flexibility on the messaging and transportation level
may be, addressing all those variations must be made possible in a generic and
structured way

In the ideal world, all service requests are sent in an envelope, like in the old-
fashioned mail. Like in the old-fashioned mail, however, we already see standards on
a national level conflicting with those of another country, so even there is no uniform
envelope-definition on a global scale. However, the differences are minimal. Anyone
can deliver a letter anywhere in the world, simply by reading the envelope - how cool
is that? The wheel is there, extremely round, and has been for centuries, if not
millennia

What needs to be on an envelope? Recipient is a must, Sender would be nice if
you want to get feedback, some unique reference and a date timestamp would make
it complete. Look at an envelope, and you'll see it all there: a recipient, a sender, and
a unique address in 4 stages: country; zip; street and number. In fact, the country
indicates how the address is exactly made up: can't use a UK address notation for an
NL address, or vice versa. Just do in Rome as the Romans do...

The envelope will conceal any diversity and carry across anything. Its
architecture and design is so splendid and loosely coupled, that it even allows for

___________________________________________________________
© Copyright 2011 – Martijn Linssen          martijnlinssen.com
25 / 32

letter bombs. Of course those are lamentable, but it shows the awesome power of a
simple yet sufficient and efficient mechanism.
There's only one but: global mail is just one way of delivering a message. There's
also UPS, DHL, small couriers, troikas and rickshaws: they need "an envelope" as
well.
Then, the big problem has to be tackled: what if your envelope isn't equal to my
envelope? Yet again, the answer to that is in the FAQ

If you order a book at Amazon, they look it up. They put it in a brown carton, and
put a barcode on top - Amazon's own envelope. At the doorstep of Amazon, e.g.
UPS picks it up. They put the brown carton inside a plastic bag, and stick a
barcode on it that is also humanly legible - UPS's own envelope. Across borders,
national couriers take over. They probably just turn the plastic bag around and
slap another barcode sticker on it - their own envelope.
So, history repeats itself: each envelope is wrapped in another one. What does the
final recipient do? As he is the only one interested in the package content itself, he
unwraps all envelopes. As he has no need for them in the future, he throws them
away

Been there done that, haven't we? Let's compare that to our favourite mailman

   •   You have a letter, which you put in an envelope
   •   The envelope makes it to the mailbox
   •   When the mailman makes it to the mailbox, he takes out his own envelope
       and wraps that around all other envelopes. That envelope is what people
       usually call a mailbag. He tags the mailbag so he knows who the sender was
       (this very mailbox), thus adding Sender, unique reference and date-time
   •   At the post office or sorting center, he unwraps the envelope / empties the
       mailbag. The content gets sorted, and makes it into other envelopes: mailbags
       to carry them to the final destination
   •   Those envelopes make it to distribution centres, are once more emptied, and
       sorted again into smaller portions, fitting a street or district / quarter
   •   The mailman delivers the envelopes to their addresses, et voila, done deal


So, another mystery solved:

   1. An envelope will be the link between messaging and transportation, and
      survive transformation
   2. All attributes on an envelope are mandatory, well-defined and highly
      standardised
   3. If it's not your envelope, just wrap your own around it
   4. If you want to read its content, you'll have to unwrap all other envelopes


Next: messaging, transportation and envelope collaboration bringing you
100% BPM, SOA or EDA




___________________________________________________________
© Copyright 2011 – Martijn Linssen          martijnlinssen.com
26 / 32



   11.       Orchestration




I've compared the diversity of an IT application landscape and managing its
information exchange in a uniform way to translation, with the European Parliament
as a perfect example of translating dozens of languages via three intermediate
languages. In IT, we only need one, as languages (syntaxes) there are far less
complex than in the linguistic world

I've used a similar metaphor to explain how different transport protocols can be
handled, by referring to James Bond's opening scenes, where he takes all kinds of
transportation in order to escape death. He simply manages by getting off one kind of
transportation, and getting on another one. Clever hey?

In the previous chapter, I've explained the no. 10 of Integration: the envelope. I
explained that mechanism that needn't be explained by pointing out how the
everyday mailman handles letters, by wrapping and unwrapping envelopes over
envelopes

Together with the hub-and-spoke architecture, it is evident that an Integrated
Enterprise needs a single point of entry for all this to make real sense. An airport
has one control tower, a house has one front door, a lock has one keyhole, a
labyrinth has one centre, a process has one starting point, everything has a
beginning and an end...

And an orchestra has one orchestrator

With all interfaces and services passing through the same Enterprise Integration front
door, they can be perfectly orchestrated. There will be many points of physical
transport entry, and these will have to guarantee delivery to the single
Orchestration point of entry. Every single message can then be received,
transformed and routed to its destination point. There, business logic will be applied,
and something will be sent back in return: a technical confirmation receipt, a
functional acknowledgment, a reply to a request, an error: all that is up to the
agreements made between the sender of the message and its recipient.


___________________________________________________________
© Copyright 2011 – Martijn Linssen          martijnlinssen.com
27 / 32

They will make it past the Orchestrator again, completing the transaction, who then
has to guarantee delivery by the many points of physical exit

This way, everything going on externally and needing your business process
touch, will be handled by one single person - well it'll of course be a machine, but
stick with me please. Will that effectively give you the ability to manage those
externally offered business processes? Yes. Maybe manage is a big word, but it will
give you full traceability over them: you'll know who did what when, what and when
the outcome to that was - it will all pass the Orchestrator and you can decide to keep
the record of that forever

What if you'd also send and receive your internal service requests through the
Orchestrator? Don't mention the O-word nor the P-word (Overhead and
Performance), just think about it from a logical, conceptual point of view.
Would it be possible? I think so, why not? Would it be helpful? I think so, why not?
Would it give you Enterprise-scale BPM via a small-step approach, where you can
take one application or system at a time? Yes

Tick of the BPM box right there then, no need for massive million dollar
devouring seemingly endless big bang projects

If you don't do the BPM, but just stay there, with all externally-used and internally
used service requests routing through one and the same Orchestrator, wouldn't that
be a nimble yet full-blown SOA?
Wouldn't it? Yes

Tick of the SOA box right there then, no need for massive million dollar
devouring seemingly endless big bang projects




___________________________________________________________
© Copyright 2011 – Martijn Linssen          martijnlinssen.com
28 / 32



   12.       The dont's of Integration




This chapter is about debunking TLA's and FLA's. XML, SOAP and REST primarily,
but Webservices and other concepts whose added value is flimsy or even absent, will
get proper attention

XML is a syntax, making it part of the Integration messaging theme

Back in 2000, when I started to say that XML doesn't add anything to Enterprise
business application integration other than overhead, complexity and confusion, the
most-heard pro of XML was "it's the language of the future"
I always responded with "I can't do business with the future, can I?" and explained
how 1,000 years ago we Europeans (economically, politically) all spoke Latin in
Europe, followed by French in 1700's, and English in 1900's, without anything ever
changing to (economical, political) business itself as a result.
Was there a businesscase for speaking the new language? At some point, when a
growing minority started to speak it, yes. Has this ever been the case with XML? No,
never. And it never will:
Google never used XML, didn't even think about it. Twitter deprecated XML use in
2010, and Facebook is doing so now. Google moved to a proprietary format, Twitter
and Facebook "will go flat-file": JSON

Don't use XML for Integration. It is nice for humans to read, but in the machine-
to-machine integration world, it has failed to prove its value in the last 10 years
- because there is no value

SOAP can't be placed in any of the Integration themes (messaging, transportation,
transformation). It touches some aspects of The Envelope

SOAP was originally started in 1998 by Microsoft as an attempt to do what it says:
Simple Object Access Protocol. W3C adopted it to describe exchanging structured
and typed information between peers in a decentralized, distributed environment.
It is an exchange mechanism, but relies exclusively on a fixed message format (XML)
and a fixed transportation protocol (HTTP). Its main focus is on the SOAP envelope,
yet doesn't describe that at all: the only mandatory thing in there is the word
Envelope itself.


___________________________________________________________
© Copyright 2011 – Martijn Linssen          martijnlinssen.com
29 / 32

Everything SOAP tries to implement has already been taken care of by SMTP, the
transport protocol used for email, in a far better, standardised and successful way.
SOAP will cause you to reinvent the wheel and describe your own envelope, yet tie
you down with highly interpretable mandatory or optional elements. Will it audit or
certify your SOAP inventions? No. So what's the use?

Don't use SOAP for Integration. It restricts you to one message syntax, one
transportation protocol, and leaves it up to you to reinvent your own SOAP
thing. It has failed to prove its value in the last 10 years - because there is no
value

Web services can't be placed in any of the Integration themes (messaging,
transportation, transformation). Like SOAP, they focus on a very fixed tech solution
for unknown business problems

W3C defines a Web service as "a software system designed to support interoperable
machine-to-machine interaction over a network. It has an interface described in a
machine-processable format (specifically Web Services Description Language
WSDL). Other systems interact with the Web service in a manner prescribed by its
description using SOAP messages, typically conveyed using HTTP with an XML
serialization in conjunction with other Web-related standards".
What's the goal? The point? What are the alternatives? Why is this the best solution?

Every single format can be processed by machines, if -and only if- it can be
understood and agreed upon by humans doing business. W3C doesn't know
anything about doing business, and XML, SOAP, WSDL, UDDI and Web Services all
were invented by them

Don't use Web services for Integration. They restrict you to one message
syntax, one transportation protocol, one way of writing down your web service,
and leave it up to you to reinvent your own web service. It has failed to prove
its value in the last 9 years - because there is no value

I have struggled to write this down, as a chapter with a subject like this basically only
destroys. There is one big positive thing to mention: all these concepts were invented
by W3C. Tim Berners-Lee heads that organisation, who accidentally invented the
Internet - so that's why people put value onto W3C concepts

What these acronyms all have in common, is inventing their own way of laying
down the exact structure of How things should be said - without taking into
consideration What can be said




___________________________________________________________
© Copyright 2011 – Martijn Linssen          martijnlinssen.com
30 / 32



   13.       The do's of Integration
Final chapter, this is the summary and conclusion, to be used as some sort of
checklist if you like

When conducting enterprise business application integration, within the
enterprise IT landscape among applications and systems, or from there to others at
another company or even directed towards the customer, here are the pragmatic
rules:

Start at the business level, and write down which process it concerns. What is the
business event, which its trigger, how often does it occur, how sensitive is the data?
Why should this be automated, in other words, what is the manual alternative and the
benefits and concerns of both?
Write out the functionality it concerns. List the entities, their cardinalities, and all
attributes concerned, and describe them as complete as possible. Here's an example




There are about 20,000 employees, each one of which has 1 or 2 addresses, and
working on 1 to 5 assignments. The employee data has attributes, of which the
functional description is shown.
At this level, anyone can read and understand what is to be exchanged: business
people, tech people, inside people, outside people, suppliers, customers: everyone

Depending on the bsiuness goal, some of these will be required, and some
won't: specify that as well. When creating a new employee, First name, last name
and personal ID will be required, but not an employee number: that will be handed
out by the system. When modifying an employee, that employee number will be
required, including the last name, but perhaps not the personal ID.
So, for the intended business goal, write down which attributes are required and
when

Then, with this Information list in hand, make the transition to the systems on
both sides: do those requirements meet and match on both sides? If so, you can do
business, if not, you have an issue. This indeed means that business agreement,
functional analysis and information system (gap) analysis is part of the preliminary
phase and could result in the conclusion that business simply can't be done
___________________________________________________________
© Copyright 2011 – Martijn Linssen          martijnlinssen.com
31 / 32



If successful, both sides will add their information system requirements (these
are actually limitations) to the Information: Last name will be 30 characters in one
system, and 25 in the other - what to do? Is the business agreement still in place or
do we have to revise it?

Apart from the messaging level, there will now be limitations on the transportation
level as well: what are the existing possibilities to get messages across? Data
sensitivity as defined by the business will lead to transport security and encryption
and access restriction requirements, and these will meet limitations: again, the
common subset has to be found.
Again, is the business agreement still in place or do we have to revise it?

On both sides there will now be a full technical description of the interface,
complete with field length, datatype, format. The business frequency and the given
cardinalities will give a rough guesstimation of the resulting file size, thus the volume
and load, making it possible to calculate storage requirements and capacity planning.
Files will have to be kept online for a certain amount of time, and archived as well,
depending on compliancy requirements the business has to follow up - all that costs
space as well.
For the transformation into an intermediate language, this will have to be
repeated of course. Here the IT department will mostly determine how long these
have to be available. If used as basis for a full-blown BPM, the underlying processes
will drive these requirements too, although files should only be used for resending
(on failure) and "evidence" of what was received and sent


At this point, the issue arises: what file format and transportation protocol
should we use?


Which ever ones are most practical and pragmatic, will be the answer

For A2A (application-to-application, a TLA used to describe internal integration) the
usual pick will be flat-file for the message and shared folders for transport. A flat-file
can be a CSV, a fixed flat-file, JSON, whichever makes you feel good. Maybe it's an
XML, an iDoc: all that is up to the applications themselves. When you use an
intermediate language and a canonical model, you will have to choose a file format
yourself.
If volumes and frequencies are high, a more sturdy transport protocol might be
required; thing of queueing mechanisms. XML will definitely pose capacity and
storage problems there, ask Google, Twitter and Facebook and companies in the
traditional B2B area

If one of the A2A interfaces "makes it out there", i.e. is a candidate for exchanging
information with external parties and you engage in B2B or B2C, the question of file
format and transportation protocol will arise once again - just in a different context.
What is most appropriate then?
Again, do in Rome as the Romans do. Speak X12 in US industries, EDIFACT in
Europe, Swift in banking, HL7 in health, etcetera. Transport? FTP or FTPS for static

___________________________________________________________
© Copyright 2011 – Martijn Linssen          martijnlinssen.com
32 / 32

solutions, HTTP for more dynamic ones and AS1 or AS2 for the "bigger boys". Treat
each of these questions as a simple daily one: should I pick UPS, global or local mail
or courier to transport my package? That of course depends on what the package is,
and how fast it should get somewhere

Last but absolutely not least, you should define an envelope for each interface:
who is the sender, who is the recipient, what does it contain? This will enable you to
route its contents freely across the globe, just like SMTP (email) does, without
opening the envelopes and looking at the letters. Don't address envelopes to URLs
or other silly notations, you're doing business with an organisational entity, not a
machine: let the organisation make up where exactly they want to route their stuff
internally, but keep that transparent for the outside world

In an effort to keep this chapter short, I'll just mention that business integrity
requirements will lead to business rules and exceptions, which can be translated into
error handling and (user) feedback on a functional and technical level. These will
lead to functional and technical acknowledgments on the interface level, retry
mechanisms when a file can't be sent out, and exception handling and agreements
for that.
To really define the interface, the byte-level has to be spelled out as well: character
code (ASCII or UTF-8, EBCDIC or UCS-2, any other flavours), hexadecimal form of
delimiters, periods or commas, etcetera: the real dirty stuff

This concludes my thoughts on Perfect Integration. I hope I touched all possible
questions and answers, and helped unravel the mystery of buzzword Three Letter
Acronyms. I think I have reached a satisfactory level of detail without losing your
attention - but you'll be the judge of that




___________________________________________________________
© Copyright 2011 – Martijn Linssen          martijnlinssen.com

Weitere ähnliche Inhalte

Was ist angesagt?

The Future Of Bpm Six Trends Shaping Process Management
The Future Of Bpm Six Trends Shaping Process ManagementThe Future Of Bpm Six Trends Shaping Process Management
The Future Of Bpm Six Trends Shaping Process Management
Nathaniel Palmer
 
Vint big data research privacy technology and the law
Vint big data research privacy technology and the lawVint big data research privacy technology and the law
Vint big data research privacy technology and the law
Karlos Svoboda
 
Survey of accountability, trust, consent, tracking, security and privacy mech...
Survey of accountability, trust, consent, tracking, security and privacy mech...Survey of accountability, trust, consent, tracking, security and privacy mech...
Survey of accountability, trust, consent, tracking, security and privacy mech...
Karlos Svoboda
 
RealOps IOA Editorial for BM Mag - FINAL
RealOps IOA Editorial for BM Mag - FINALRealOps IOA Editorial for BM Mag - FINAL
RealOps IOA Editorial for BM Mag - FINAL
John Scott
 
MISO L003 network computing
MISO L003 network computingMISO L003 network computing
MISO L003 network computing
Jan Wong
 

Was ist angesagt? (20)

The Future Of Bpm Six Trends Shaping Process Management
The Future Of Bpm Six Trends Shaping Process ManagementThe Future Of Bpm Six Trends Shaping Process Management
The Future Of Bpm Six Trends Shaping Process Management
 
Vint big data research privacy technology and the law
Vint big data research privacy technology and the lawVint big data research privacy technology and the law
Vint big data research privacy technology and the law
 
Means of ict
Means of ictMeans of ict
Means of ict
 
Enterprise architecture-for-healthcare
Enterprise architecture-for-healthcareEnterprise architecture-for-healthcare
Enterprise architecture-for-healthcare
 
Survey of accountability, trust, consent, tracking, security and privacy mech...
Survey of accountability, trust, consent, tracking, security and privacy mech...Survey of accountability, trust, consent, tracking, security and privacy mech...
Survey of accountability, trust, consent, tracking, security and privacy mech...
 
Service automation and organisational structure an application example based ...
Service automation and organisational structure an application example based ...Service automation and organisational structure an application example based ...
Service automation and organisational structure an application example based ...
 
Managing the digital firm
Managing the digital firmManaging the digital firm
Managing the digital firm
 
Jammanna
JammannaJammanna
Jammanna
 
Ms 54 management information systems
Ms   54 management information systemsMs   54 management information systems
Ms 54 management information systems
 
RealOps IOA Editorial for BM Mag - FINAL
RealOps IOA Editorial for BM Mag - FINALRealOps IOA Editorial for BM Mag - FINAL
RealOps IOA Editorial for BM Mag - FINAL
 
Article 11
Article 11Article 11
Article 11
 
Dit yvol3iss8
Dit yvol3iss8Dit yvol3iss8
Dit yvol3iss8
 
Information system infrastructure
Information system infrastructureInformation system infrastructure
Information system infrastructure
 
Federated e-Identity Management across the Gulf Cooperation Council
Federated e-Identity Management across the Gulf Cooperation CouncilFederated e-Identity Management across the Gulf Cooperation Council
Federated e-Identity Management across the Gulf Cooperation Council
 
Chapter 07 pertemuan 9- donpas - keunggulan kompetitif ti dan si strategis
Chapter 07 pertemuan 9- donpas - keunggulan kompetitif ti dan si strategisChapter 07 pertemuan 9- donpas - keunggulan kompetitif ti dan si strategis
Chapter 07 pertemuan 9- donpas - keunggulan kompetitif ti dan si strategis
 
james o'brien chapter 7 electronic business system
james o'brien chapter 7 electronic business system james o'brien chapter 7 electronic business system
james o'brien chapter 7 electronic business system
 
50516
5051650516
50516
 
MISO L003 network computing
MISO L003 network computingMISO L003 network computing
MISO L003 network computing
 
Knowledge notebooks
Knowledge notebooksKnowledge notebooks
Knowledge notebooks
 
Blockchain Design and Modelling
Blockchain Design and ModellingBlockchain Design and Modelling
Blockchain Design and Modelling
 

Ähnlich wie Perfect Integration

Evaluation Of Integration Technologies For The Healthcare...
Evaluation Of Integration Technologies For The Healthcare...Evaluation Of Integration Technologies For The Healthcare...
Evaluation Of Integration Technologies For The Healthcare...
Jessica Cannella
 
Assignment 1TextbookInformation Systems for Business and Beyond.docx
Assignment 1TextbookInformation Systems for Business and Beyond.docxAssignment 1TextbookInformation Systems for Business and Beyond.docx
Assignment 1TextbookInformation Systems for Business and Beyond.docx
sherni1
 
Assignment 1TextbookInformation Systems for Business and Beyond.docx
Assignment 1TextbookInformation Systems for Business and Beyond.docxAssignment 1TextbookInformation Systems for Business and Beyond.docx
Assignment 1TextbookInformation Systems for Business and Beyond.docx
deanmtaylor1545
 
Information Technology for ManagementThis book is li.docx
Information Technology for ManagementThis book is li.docxInformation Technology for ManagementThis book is li.docx
Information Technology for ManagementThis book is li.docx
dirkrplav
 
Enabling Hybrid Cloud Today With Microsoft-technologies-v1-0
Enabling Hybrid Cloud Today With Microsoft-technologies-v1-0Enabling Hybrid Cloud Today With Microsoft-technologies-v1-0
Enabling Hybrid Cloud Today With Microsoft-technologies-v1-0
David J Rosenthal
 
Information technology (IT) is dramatically changing the business la.pdf
Information technology (IT) is dramatically changing the business la.pdfInformation technology (IT) is dramatically changing the business la.pdf
Information technology (IT) is dramatically changing the business la.pdf
akshay1213
 
Metalayer now Colayer - Part 3/3 - full Presentation
Metalayer now Colayer - Part 3/3 - full PresentationMetalayer now Colayer - Part 3/3 - full Presentation
Metalayer now Colayer - Part 3/3 - full Presentation
Markus Hegi
 
enterprise-data-everywhere
enterprise-data-everywhereenterprise-data-everywhere
enterprise-data-everywhere
Bill Peer
 
Information Technology Applications For A Team Of People
Information Technology Applications For A Team Of PeopleInformation Technology Applications For A Team Of People
Information Technology Applications For A Team Of People
Stephanie Williams
 
EMC_SurvivalKit_A5_04_Digitaal
EMC_SurvivalKit_A5_04_DigitaalEMC_SurvivalKit_A5_04_Digitaal
EMC_SurvivalKit_A5_04_Digitaal
Jacques Boschung
 

Ähnlich wie Perfect Integration (20)

The Internet of Things: Exploring revenue generating use cases
The Internet of Things: Exploring revenue generating use casesThe Internet of Things: Exploring revenue generating use cases
The Internet of Things: Exploring revenue generating use cases
 
Evaluation Of Integration Technologies For The Healthcare...
Evaluation Of Integration Technologies For The Healthcare...Evaluation Of Integration Technologies For The Healthcare...
Evaluation Of Integration Technologies For The Healthcare...
 
MIS.ppt
MIS.pptMIS.ppt
MIS.ppt
 
Assignment 1TextbookInformation Systems for Business and Beyond.docx
Assignment 1TextbookInformation Systems for Business and Beyond.docxAssignment 1TextbookInformation Systems for Business and Beyond.docx
Assignment 1TextbookInformation Systems for Business and Beyond.docx
 
Assignment 1TextbookInformation Systems for Business and Beyond.docx
Assignment 1TextbookInformation Systems for Business and Beyond.docxAssignment 1TextbookInformation Systems for Business and Beyond.docx
Assignment 1TextbookInformation Systems for Business and Beyond.docx
 
Information Technology for ManagementThis book is li.docx
Information Technology for ManagementThis book is li.docxInformation Technology for ManagementThis book is li.docx
Information Technology for ManagementThis book is li.docx
 
Enabling Hybrid Cloud Today With Microsoft-technologies-v1-0
Enabling Hybrid Cloud Today With Microsoft-technologies-v1-0Enabling Hybrid Cloud Today With Microsoft-technologies-v1-0
Enabling Hybrid Cloud Today With Microsoft-technologies-v1-0
 
Information technology (IT) is dramatically changing the business la.pdf
Information technology (IT) is dramatically changing the business la.pdfInformation technology (IT) is dramatically changing the business la.pdf
Information technology (IT) is dramatically changing the business la.pdf
 
Metalayer now Colayer - Part 3/3 - full Presentation
Metalayer now Colayer - Part 3/3 - full PresentationMetalayer now Colayer - Part 3/3 - full Presentation
Metalayer now Colayer - Part 3/3 - full Presentation
 
Systems analysis and design lecture 1
Systems analysis and design lecture 1Systems analysis and design lecture 1
Systems analysis and design lecture 1
 
Data processing in Industrial systems Course Notes 1- 3 weeks
Data processing in Industrial systems Course Notes 1- 3 weeksData processing in Industrial systems Course Notes 1- 3 weeks
Data processing in Industrial systems Course Notes 1- 3 weeks
 
Challenges of IT Implementation
Challenges of IT ImplementationChallenges of IT Implementation
Challenges of IT Implementation
 
What is the Future of Voice & Collaboration in Financial Services?
What is the Future of  Voice & Collaboration  in Financial Services?What is the Future of  Voice & Collaboration  in Financial Services?
What is the Future of Voice & Collaboration in Financial Services?
 
enterprise-data-everywhere
enterprise-data-everywhereenterprise-data-everywhere
enterprise-data-everywhere
 
Management Information Systems MGMT205
Management Information Systems MGMT205Management Information Systems MGMT205
Management Information Systems MGMT205
 
How Payment Innovations Help Accelerate Ecosystems
How Payment Innovations Help Accelerate EcosystemsHow Payment Innovations Help Accelerate Ecosystems
How Payment Innovations Help Accelerate Ecosystems
 
A Special Report on Infrastructure Futures: Keeping Pace in the Era of Big Da...
A Special Report on Infrastructure Futures: Keeping Pace in the Era of Big Da...A Special Report on Infrastructure Futures: Keeping Pace in the Era of Big Da...
A Special Report on Infrastructure Futures: Keeping Pace in the Era of Big Da...
 
Information Technology Applications For A Team Of People
Information Technology Applications For A Team Of PeopleInformation Technology Applications For A Team Of People
Information Technology Applications For A Team Of People
 
EMC_SurvivalKit_A5_04_Digitaal
EMC_SurvivalKit_A5_04_DigitaalEMC_SurvivalKit_A5_04_Digitaal
EMC_SurvivalKit_A5_04_Digitaal
 
Scei technical whitepaper-19.06.2012
Scei technical whitepaper-19.06.2012Scei technical whitepaper-19.06.2012
Scei technical whitepaper-19.06.2012
 

Kürzlich hochgeladen

IAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsIAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI Solutions
Enterprise Knowledge
 
Histor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slideHistor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slide
vu2urc
 

Kürzlich hochgeladen (20)

Handwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed textsHandwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed texts
 
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
 
08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking Men08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking Men
 
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdfThe Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
 
Evaluating the top large language models.pdf
Evaluating the top large language models.pdfEvaluating the top large language models.pdf
Evaluating the top large language models.pdf
 
Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)
 
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
 
IAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsIAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI Solutions
 
GenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day PresentationGenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day Presentation
 
🐬 The future of MySQL is Postgres 🐘
🐬  The future of MySQL is Postgres   🐘🐬  The future of MySQL is Postgres   🐘
🐬 The future of MySQL is Postgres 🐘
 
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)
 
08448380779 Call Girls In Greater Kailash - I Women Seeking Men
08448380779 Call Girls In Greater Kailash - I Women Seeking Men08448380779 Call Girls In Greater Kailash - I Women Seeking Men
08448380779 Call Girls In Greater Kailash - I Women Seeking Men
 
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
 
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdfUnderstanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
 
Scaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organizationScaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organization
 
How to convert PDF to text with Nanonets
How to convert PDF to text with NanonetsHow to convert PDF to text with Nanonets
How to convert PDF to text with Nanonets
 
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationFrom Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
 
[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf
 
presentation ICT roal in 21st century education
presentation ICT roal in 21st century educationpresentation ICT roal in 21st century education
presentation ICT roal in 21st century education
 
Histor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slideHistor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slide
 

Perfect Integration

  • 2. 2 / 32 1. Architectural approach to Integration IT Architecture divides its world in a few parts. Various frameworks have various models, but I still like Capgemini's one best, since it follows people and organisation structure. Above, my own visualisation of that Businesses thrive on business. They do business. Without business, there is no business - need I continue? Business defines the business rules, the business exceptions. They make and need business cases with which they can justify investments, change, growth. Business events occur at a certain frequency and volume. When all that matches with other people, business agreements can be made with other companies, allowing them to do business across the barriers of their own company Information is in the input for business, and the output for it. Information needs to comply with the business rules and exceptions. People support the business here by handling this information: users Information systems allow the business to scale. They automate what can be automated, store history, make life as pleasantly as possible for the end-user who works for the business. They serve as vehicles for the information The technical infrastructure supports and carries the systems: the iron, the networks. They form the firm base for the information systems, they are the roads the traffic travels on What if two businesses want to do business? Both parties can’t just exchange information like one would normally do in a telephone call or in a conversation with someone else; usually the business is too complex for that. Nor can users do this, as there usually is a high degree of computerisation involved due to the required speed of processing information, and the volume of information ___________________________________________________________ © Copyright 2011 – Martijn Linssen martijnlinssen.com
  • 3. 3 / 32 The information systems themselves can exchange information, but not one system in the world is exactly alike another one: they always differ in structure, size, business they support and operating system or database used to support it: therefore a common layout (interface) needs to be chosen to enable the technical translation from one system to another That interface must then be supported by the underlying infrastructure: as these infrastructures differ from one another as well, a common interface needs to be chosen here too On all levels in the enterprise the common theme returns: looking for and defining common grounds or subsets, and translating those to common interfaces. If all four levels mentioned (Business, Information, Information Systems, and Infrastructure) supported these subsets, integration can become a fact It’s like establishing a common topic to talk about in a conversation with someone else: • what is the shared interest (Business) • which information do you want to share (Information) • which definitions are mutually exchangeable (Information Systems) • how do you want to exchange ideas (Infrastructure) In the next chapter, I'll go into detail with regards to the three themes of integration: messaging, transportation and transformation ___________________________________________________________ © Copyright 2011 – Martijn Linssen martijnlinssen.com
  • 4. 4 / 32 2. Common subset and transformation The common subset found on all levels in the previous chapter was: • what is the shared interest (Business) • which information do you want to share (Information) • which definitions are mutually exchangeable (Information Systems) • how do you want to exchange ideas (Infrastructure) Information exchange form: messaging After having agreed on mutual business and information to be exchanged, transferring information in a certain message layout from and to systems is the logical next step. As different systems almost always produce different message layouts, there has to be a conversion in between these different message layouts: this conversion is called interface, where a message that originates from one system is converted to a message that has another system as destination This conversion is purely on a physical level: the logical and functional information in the message should not be altered. Good interfacing doesn’t involve more than simple formatting changes like periods to comma’s or vice versa in decimal fields, and translation from and to solid standards such as ISO and EAN. There are many, many standards on the business document level such as EDIFACT and X12, HL7, SWIFT, ClieOp, etcetera. Normally, in mature and high-volume high-frequency industries such as banking, automotive and logistics, these standards are easy to find and hard ( = expensive) to deviate from. In less mature industries or those closer to government and non-profit, one will find that people find the leisure to play around with non-interoperable standards by usually inventing their own XML and SOAP. Information exchange method: transportation Messages need to be transported from the originating system to the destination system (and vice versa). Ideally, a communication protocol must be chosen that is supported by all parties involved. A communicastion protocol is nothing more than a worldwide standard: telephone, television, fax, telex, the internet: all thouse wouldn't have been possible without simple, static, clear and unambiguous standards. Fortunately there aren’t as many communication protocols as there are application systems, so a choice can be made. Communication protocols are, as opposed to information systems, highly standardised on a global scale. Therefor a choice can be made, instead of having to convert one transportation protocol to the other. Next to that, almost all (simple) transportation protocols are free. The ones that ___________________________________________________________ © Copyright 2011 – Martijn Linssen martijnlinssen.com
  • 5. 5 / 32 are more complex and offer a higher degree of functionality and built-in reliability usually aren’t free However, when integration is considered to be “door-to-door”, meaning starting at the very originating system at e.g. one company, and ending at the very destination system of e.g. another company, there almost always is some kind of conversion involved, as chances are slim that both systems are connected end-to- end on the exact same common communication protocol. From a security point of view there also are many restrictions that simply won't allow for a direct connection between an internal system and external system The integration topic: transformation Information exchange is the only goal of integration. The only purpose is doing business on a cost-effective basis in order to make or save (more or better) money. The integration theme is to establish a common subset, and this subset can physically be realised via transformation. Messaging and Transportation can be defined as the two main methods of integration. Transformation is the sole integration topic: business, nor information, nor information systems, nor infrastructure are alike when integration is exercised on an enterprise scale Standardisation: dynamics and statics Standardisation is key to integration. Standardisation makes matters static, and worth the effort of translating. Not standardised or volatile things are dynamic, and impossible or at least very costly to automate, and keep automated Gas, electricity and water are so standard that we can actually put the plumbing in our floors, ceilings and / or walls. Gasoline is so very standard that we can get it everywhere, because it will always be the same. The fact that these products are so very static as well as standard, is no coincidence. Electricity, gas and water nor gasoline have changed over the last decades or centuries. Like I have argued in every single chapter so far, standardisation resides in the lower architectural IT layers, and not in the top ones, as dynamics at the top, in the business where people are involved, are far larger than at the bottom, the infrastructure where machines are involved. No one has rewritten the TCP/IP stack or the HTTP protocol. Have databases significantly changed? So, standardisation in IT can be achieved, and already is partially achieved. But, it will only be achieved “down below”, and not in the business area. Business simply is way too dynamic for that So how do we achieve standards between dynamic businesses? That, in the next chapter ___________________________________________________________ © Copyright 2011 – Martijn Linssen martijnlinssen.com
  • 6. 6 / 32 3. Common language: semantics first This chapter will elaborate on messaging and transformation (part I), and explain how information exchange works in the daily world, considering simple or complex information exchanges. That will then be related to IT, and the basic ways of “writing down” information in IT will be explained. If you want to have a chat with someone else, what do you do? First, a common theme to discuss has to be found - usually not that hard. The favourite ones in e.g. IT usually are new technologies, cars, and the occasional sports. The topics then have to be found, which is a bit harder. Do we discuss hardware technologies, or software technologies, or is it architecture or open source we want to discuss? If the topics have been identified, last but not least the definitions have to be checked: are we talking about the same subjects? The Greek philosopher Socrates filled entire conversations, and books, with merely trying to establish that he and his counterpart were talking about the same thing. There are numerous occasions on which he talked half a dozen times or more to someone, only to mutually agree on the fact that they didn’t mean the same thing while using the same word, and then end the discussion. Nowadays this is a serious topic in integration: establishing a common mindset and “dictionary”. Semantic Interoperability is a very interesting and worthwhile item to closely investigate and embrace, before engaging any inter-company integration. Even when conducting integration within a mid-sized enterprise itself, it is very valuable to have a common dictionary that explains in detail what words mean to all parties involved. After all, what is the use of a service-oriented enterprise and its service repository if the function of a certain service can be interpreted in more than one unambiguous way? So, for argument's sake, let's assume that you have • an elaborate dictionary of all the relevant business words used in your enterprise • and their full, unambiguous meaning ___________________________________________________________ © Copyright 2011 – Martijn Linssen martijnlinssen.com
  • 7. 7 / 32 • complete with the structures they fit in • and the order in which they do so • for each and every business process they occur in Now, on a semantical level, everything is clear: the most important part has been successfully completed. Secondly, agreement has to be reached on another level: the syntactical part When setting up information exchange between two parties, life can be simple. A common form will have to be chosen, with which both parties feel comfortable: is it OK to converse by fax only, or is the topic so intimate that it can only be done face- to-face? When living in entirely different time zones, it will be practical to have a way of conversation that is not that interactive, e.g. writing old-fashioned letters or sending emails. And when living far apart, it is a costly business (well, at least it always used to be) to make long phone calls. In IT, this common form is (again) called an interface. The interface chosen should always be a message, a static representation of an event. The idea is that an information system itself takes its dynamic, internal data, and exports that outside of its own system in the form of a static message. That message will always reflect the state of that small subpart of the system at that specific time. The other system is expecting a similar message, and will import that into its own system, using the static data to update its own dynamic data. To bridge the gap between the two systems, the one static message will have to be converted to the other static message, preferably outside the systems This transformation of messages, which involves designing and creating interfaces, also known as interfacing, can be compared to the day-to-day world of translation of languages. People who speak different languages can’t understand each other unless one speaks the language of the other. However, the scale on which a language should be mastered differs considerably from that of a simple interface or service. On the other hand, if you just met this mysterious foreigner and only want to exchange a few romantic words over a warm campfire, usually a 30-minute crash- course suffices. But, being an application or system in an enterprise IT-landscape, there regularly is a need for far more than just mere pleasantries: it therefore is fair to compare interfacing and servicing to the every-day art of translating and interpreting languages ___________________________________________________________ © Copyright 2011 – Martijn Linssen martijnlinssen.com
  • 8. 8 / 32 4. Common language: direct translation This chapter is about my favourite subject: translation. Long, long, long ago I aspired to become an interpreter - but changed my mind. Still, I consider myself a linguist and a good one at that, speaking 6 languages of which 3 fluent. The existence of extreme diversity in languages of the world, and the same existence in diversity in an average enterprise IT landscape have always hooked my attention Just like people speak different dialects and languages, so do applications: not one of them can perfectly understand the other without changing a thing. Even when interfacing between SAP systems, there will be differences on the technical level that have to be resolved. So, how do you solve these misunderstandings at the technical level? How do you "translate" between the different languages and dialects spoken? One of the sides has to adapt, or maybe both? When, and how? To answer that, let's look at the everyday world of natural languages. A completely different world, with an identical problem - and a working solution Learning a language can be very hard and time-consuming, and very few will ever master a non-native language perfectly. Most of us know someone within the same country that has an accent and just can’t seem to get rid of it; those that have connections across the border might have heard other people speak their language and have difficulties understanding them. Or one has heard a colleague or neighbour speak a foreign language and, while not being able to speak that language fluently themselves, has good reason to suspect that there will be people on the other side having difficulties understanding that “version of the language” In short, a language spoken by a so-called non-native speaker always has a certain external “flavour” to it. If a Dutchman, a German and a Frenchman all have to learn English in order to exchange information with each other, it is almost a guarantee that that English will have elements of all those three languages in it. It will not be British English, nor will it be American English; it will be some kind of English that sounds a bit like English combined with Dutch, German and French. Dunglish, Genglish and Frenglish combined: it will make for a hilarious comedy, unless it's a serious professional occasion. Learning a language takes time, money, and last but not least talent In professional business, therefore, often interpreters are used: people who have made it their profession to be fluent in at least 2 languages, and in that way can ___________________________________________________________ © Copyright 2011 – Martijn Linssen martijnlinssen.com
  • 9. 9 / 32 interact between two persons or parties who only speak one of them. There no longer is a need for individuals to master a foreign language: people can speak their own, and the interpreter will just translate them. Professional interpreters come at a price: it is obvious why not everyone can afford one, and thus has to learn a foreign language himself On the other hand, some people can’t afford the time needed to invest in a foreign language: it is of course inconceivable that a president or CEO is sent on a language course for several years in order to master a language: that simply is an investment that would never render a return. Even worse, a person in that position would not have to be fluent in only two, but perhaps a dozen languages. So, learning a language yourself comes at a price too. Mastering a language in a perfect way is "priceless" for most Moving the interpreter into place provides not only a short term benefit, but also a long-term one: if the president or CEO is ever succeeded, the successor doesn’t have to be sent on a language course either. One interpreter: a lifetime investment that returns every single day. And the best news? Interpreters themselves are fairly disposable too... So, in certain strategic positions within a company, it is a very wise investment to make use of interpreters. In general those positions are characterised by people that have unique, expert knowledge and skills that make them highly valuable for the company. A lot of time and money was invested to either cultivate or buy those specific characteristics: one can easily compare that to the careful selection of a business-critical application or system within the IT-landscape Translating one language to the other, however, whether done by people themselves or with the help of an intermediate like an interpreter, is still not eliminating dependencies: there are only exactly two languages involved, and the only translation possible is based on the combined knowledge of those two. Here's what happens when you do the math on that: When e.g. a German, a Frenchman and an Englishman would participate in a conversation, there would be a need for three interpreters: one that translates from German to French (and vice versa), one from German to English (and vice versa), and one from French to English (and vice versa). If another person would participate, e.g. a Dutchman, three additional interpreters would be needed: one that translates from German to Dutch (and vice versa), one from English to Dutch (and vice versa), and one from French to Dutch (and vice versa) Two parties, one interpreter; three parties, three interpreters; four parties, six interpreters: this is going the wrong way! It seems like there is a formula for direct translation and parties involved: for n parties involved, there are n/2 * (n – 1) interpreters needed: one interpreter for every two parties (n/2), and as many interpreters as there are other languages (n – 1) That formula exists, is widely recognised and true, and that is why in real life the solution is only one: next stop, the European Parliament ___________________________________________________________ © Copyright 2011 – Martijn Linssen martijnlinssen.com
  • 10. 10 / 32 5. Common language: indirect translation This chapter is about indirect translation, in contrast with the direct translation shown in the previous chapter, which came with costly, exponential dependencies When looking towards large-scale use of translators, e.g. the European Committee in Brussels, it is easily observed how these dependencies can be greatly reduced: all languages are first translated into English, French or German, after which either one of those is translated into the final destination language Of course this is doubling the effort involved, as everything has to be translated twice. However, an intermediate channel is created that everyone can use: compare that to villages and towns, each digging and building their own roads in order to travel towards each other, or a municipality that builds a highway for all to use; in this case, all that has to be done is just construct a small piece of road to connect to the highway, considerably reducing the effort, time and money involved Thus, in the European Parliament a common language is defined, viz. English, French and German. In the specific example of the European Parliament the more than 500 possible combinations are now reduced to 75. For political and linguistical reasons the choice is made to support more than one intermediary language in the European Parliament for the purpose of interpretation: real-time translation between humans. Spanish and Italian are languages that are very closely related, but Serbian and Chinese have nothing in common. The chance of finding an interpreter that speaks Italian as well as Spanish is far greater than finding one that speaks Serbian as well as Chinese In general any language would suffice for discussing any topics: grammar and syntax allow for carrying across messages that are complicated as well as simple. Languages that couldn't be compared to or fit onto each other could be languages that are far apart on an evolutionary scale, for instance languages that are only used by a few people living in tribes in the Australian bush and only have a small ___________________________________________________________ © Copyright 2011 – Martijn Linssen martijnlinssen.com
  • 11. 11 / 32 vocabulary and no noticeable grammar, in relation to an average language in the modern world with a vocabulary of tens of thousands of words and a complicated grammar that allows for a few different past tenses. In IT, there are such differences as well: if you have to hand-translate in between an old-fashioned mainframe and XML, the distance would be simply too great. You can land a Boeing on a highway in case of emergency, but if you do so on a trail you're a lot less likely to succeed So what needs to be done in information exchange, after establishing the semantics, is finding a common language: in interfaces, this common language is formed by a common, or rather generic, intermediate form. Both parties can (and rather, should) still speak their own language, but the translation is done "in the middle". This interface, like the languages, needs to be able to hold the logical structure of the desired information interchange, and the functionality desired. When this logical and functional structure has been designed, a format needs to be chosen in which this can reside. As with languages themselves, it doesn’t really matter what exactly this format is, as long as it is easily accessible, commonly available, as simple as possible and fit-for-purpose Translating that to IT, first of all a logical and functional design has to be made to determine the scope of the common subset, and determine which exactly is the definition of the various (sub)topics and functional entities, as shown in chapter three. Such constructs are generally referred to as Canonical Models, or Common Object Models.They effectively establish a business language that is spoken within the enterprise, making the extreme diversity of the underlying IT-landscape fully transparent, or rather, from a business point of View, irrelevant Canonical Models do, what IT should have done ages ago: let the business not bother about the trivialities of IT. Direct translation enables this in an efficient way: rather than sending specialised applications on a language course and teach them the business language of their freshly-joined company, the translators will do that work for them. Indirect translation perfects this in the best cost-efficient way: professional translators translate the diverse application mumbo-jumbo to the generic company business language: a win-win-win situation This concludes the first 5 chapters, dealing with the semantical level. The next two chapters will go one level deeper, and drill down to the syntactical level. After that, we'll resurface to the semantical level again, to deal with addressing in general In the next chapter, a word about interface message formats, or rather: if I were to define such an interface, what are my choices? ___________________________________________________________ © Copyright 2011 – Martijn Linssen martijnlinssen.com
  • 12. 12 / 32 6. Common language: syntaxes In the previous chapters I explained semantics, syntax, and the fastest, cheapest and easiest way to get from diverse IT applications to one uniform business language. This chapter will take a deep dive into message formats such as Flat file, EDIFACT, XML and JSON Ever wondered about the pros and cons of XML? JSON? What it is really? What other possible message syntaxes are? When semantics and a logical structure has been defined, and functional groups and fields have been identified, just about any message format can be chosen. Choosing message formats (or rather, predefining them) is a rather pragmatic measure that narrows down the margin for discussion when it comes to choosing a form for your common language Message formats can be divided into two characteristics: length and dimension. Length can be fixed-width or delimited, and dimension can be are horizontal or vertical. Basically, that's all there's to it Fixed-length messages have records and fields, containing the information, that have a fixed width. The same type of information will always be found at the same location (position) within a record or field. Fields not fully used (the information in it is physically shorter than the maximum length of the field allows for) will be filled with spaces or zeroes, depending on the data type of those fields. E.g. An empty record consisting of 5 fields, with a total length of 500 characters, will always be 500 characters long Delimited or variable-length messages have records and fields, containing the information, that have a delimited width. All records and fields are delimited by so- called delimiters, i.e. commonly defined characters that indicate the end of a record or field. The same type of information will always be found after the same delimiter. Fields not fully used (the information in it is physically shorter than the maximum length of the field allows for) will be just totally empty E.g. an empty record consisting of 5 fields, with a total length of 500 characters, will always be 5 characters (delimiters) long ___________________________________________________________ © Copyright 2011 – Martijn Linssen martijnlinssen.com
  • 13. 13 / 32 Horizontal messages look much like traditional files: fields within a record are found at the same line, one after another. This message format is the most common and resembles files or database-tables Vertical messages look much like a single-row table: fields within a record are found each at a new line, one line after another. This format is not widely used but solves the fixed-width problem in an easy way: each field starts on a new line, preceded by a so-called “tag” that identifies the function of the field. The field ends when the last character on that line is encountered Nowadays EDIFACT is the most common language used, mainly for its global approach, broad business document standards (over 200 different functional messages) and the fact that its delimited type allows for minimum message size. It is in existence since 1986, and evolved on ANSI X12 (1979). ANSI X12 is in use in the United States, whereas EDIFACT is widely used in the rest of the world XML was gaining grounds, but due to lack of standards, and the fact that it is an unlucky combination of a delimited interface with fixed-width field names it will not play a large role any time soon, unless it is pushed and supported by organisations such as the UNECE. Evangelising Web Services as having to be encapsulated in XML however have sped up the evolution of it. Additional push is delivered by most new applications offering XML-support nowadays when it comes to disclosing information. A general pull is delivered by Google, Twitter and Facebook moving away, or having moved away already, from the format. Facebook is on the verge of doing so, Google never has used XML, and Twitter deprecated its use last year. Both Twitter and Facebook have embraced JSON, a far more simple and concise message structure that serves the same purpose IDOC is a standard invented by SAP that is just a simple fixed-width interface, however supported by large organisations. Needless to say, however, that as SAP- implementations differ all over the world, so do the iDOCs Information exchange via physical files goes back decennia. A decade ago links started being made via COM, CORBA, DCOM etc, during which information wasn’t exchanged between systems via physical messages but via memory objects. This in fact meant that applications were tightly coupled via a virtual point-to-point interface. Development and maintenance of such solutions has proven to be (much) too expensive There’s a lot of talk about loosely-coupled things these days: the ancient hard- coupling (point-to-point interfacing, COM, CORBA, etc) appeared to be very time- consuming and costly with regards to development and maintenance Nowadays it’s architects that use the term loosely-coupled: some of them think that applications should be loosely-coupled towards the architecture, instead of vice versa. These never are business-architects, but IT-architects. Business architects know that applications change at high speed, and that -fast changing- business needs should be supported by IT, being able to move in or out applications or ___________________________________________________________ © Copyright 2011 – Martijn Linssen martijnlinssen.com
  • 14. 14 / 32 systems when need be - the European Parliament approach as sketched in the previous chapter. Loosely-coupled means that applications and systems should plug in and out of the enterprise almost overnight. Plugging in an application should happen as fast and cheap as possible. Regard it as just another politician speaking in Brussels: just another (relatively cheap) translator has to be found, after which business can continue as usual So, applications must always speak their native language. This is the most simple and cost-efficient idea, leaving the application to do what it’s best at: provide functionality. The translator will provide its own specialty, which is translating messages from any language into any other language - the common language that serves your company as a uniform, tool- and platform independent business language that is relatively timeless and understandable to all For an Oracle Siebel CRM application this usually means speaking Siebel XML, for SAP this usually means speaking iDOC; a mainframe will feel most comfortable with fixed or delimited flat files, etc. If you visit China, you'll talk Chinese, in Greece it's Greek, and when wanting to speak English you better distinguish between UK English, US English, Australian English or, what is generally in use in social media networks: global English This was the deep dive into messaging formats, concluding the information exchange form: messaging. Now let's resurface and address the information exchange method: transportation ___________________________________________________________ © Copyright 2011 – Martijn Linssen martijnlinssen.com
  • 15. 15 / 32 7. Information exchange: transporation [Image courtesy of Ferdinand Reus] After creating and or choosing a common or generic format to exchange the information, there is one other field to explore: the facilitation of various communication protocols through which this information can be transported What applies to messages, also applies to transport: a common language is to be advised as "main artery" for all the traffic. Besides that, each application must speak its own language here as well, and use its own infrastructure as much as possible. Adjusting applications for transport protocols can be even harder than doing so for messages: transport protocols are very, very static and rely on worldwide standards. Making your own version of it is highly unappreciated Transportation of data is, like integration itself, a necessary evil. It doesn’t really matter how information gets from A to B, as long as it stays the same. Information exchange by letter, fax, phone, email, SMS, video etcetera, will always be in different formats, but the general contents will always be the same. It will be influenced by certain boundaries and limitations inherent to each protocol, but it won’t change in essence - fortunately! The average opening scene of a James Bond movie sets a good example of the infrastructural variations one can encounter: almost all vehicles imaginable are used by Bond while "travelling from A to B" (usually that means escaping death from his pursuers). The same applies to a message that needs to be transported: on the enterprise level there's nearly all transport protocols one could think of, and ideally they should all be supported – within reason A glance at all possible ways to get information across The most widespread and easy to use communication protocol is via file systems. Usually, networks and applications within a company are "close enough" to find a ___________________________________________________________ © Copyright 2011 – Martijn Linssen martijnlinssen.com
  • 16. 16 / 32 mutual place where they can drop off and pick up files, just like one would at or from a post-office. For intra-company transportation this might suffice, but inter-company information exchange calls for a protocol that can easily connect to and from different operating system In the old days, FTP (file transfer protocol) was the easiest and most widespread protocol. It is free, widely available, runs on all platforms, and capable of being a separate layer on top of entire applications, systems, enterprises. It is limited in its commands itself but allows for all the desired transportation methods. Other common protocols are SMTP (the protocol used for email), and HTTP (the protocol used for web). JMS is quickly gaining grounds because it’s a queuing mechanism, and free. Queuing systems offer a high degree of reliability, with guaranteed delivery and retry-mechanisms built-in There is a growing need for security in general, with inter-company communication spreading "around the world". Dedicated bilateral communication lines are no longer the default, communication is now largely based on transportation via internet. AS1 and AS2 are secure applications of SMTP and HTTP, respectively. Both specifications were created by EDI over the Internet (EDIINT), a working group of the Internet Engineering Task Force (IETF) for developing secure and reliable business communications standards. The added value of AS1, AS2 and AS3 is the fact that they offer so-called “non- repudiation”: on successful physical reception of a file a receipt is sent back in return, through which it is indisputable that the message has been sent and received. These work with certificates on both sides, so all parties can authenticate themselves in a trusted and secure manner As security is a hot topic when it comes down to communication, guaranteed delivery is one when it comes to transportation. The basic way to ascertain guaranteed delivery is to receive notifications for each message having been delivered, from the recipient. This way there's always proof that a message has been delivered. When e.g. an order appears to not have made it to the warehouse it saves a lot of time when the sender has a receipt of the actual message sent that contained the order In the old days proof of delivery was given by keeping log files of messages sent and received, using checksums, etc. Proof of delivery was something that could be requested in exceptional situations, with manual intervention by people. With the use of a.o. queueing systems such mechanisms became built-in, and nowadays guaranteed delivery is something that has moved from being available in exceptions, to becoming part of the rule Again, on the transportation level: a good diversity in between them all, and a common subset needed in order to be able to share the same means of information exchange. Next up: how to link messaging to transportation ___________________________________________________________ © Copyright 2011 – Martijn Linssen martijnlinssen.com
  • 17. 17 / 32 8. History of the last decades In the first seven chapters, the approaches to Integration have been shown. The architectural top-down approach, the common subset theme, and the central integration methods: messaging, transformation and transportation. Now the approach to successful integration has been set out, a brief story about integration in the real IT world, over the last decades, is in its place Point to point interfacing Point to point connections, is a term that describes how applications can be connected. Point to point interfacing was commonly used to link applications together "in the early days", and was a reasonable solution as long as the scale of integration stayed small Point-to-point interfacing involved applications having to speak each other's language; the resulting IT-landscape was a tightly interwoven one, where a lot of effort and time was spent in information exchange. Replacing an application by another one meant that all connected applications had to change their interface to it at the same time. Point-to-point interfacing has proven to be complex, costly and unreliable for any mid-sized enterprise or up. Point-to-point interfacing lacked centralisation and standards, making it very hard to trace and solve issues There is a formula for point-to-point interfacing: it is n * (n - 1) That means that for every connected / interfacing application in an enterprise, there are n * (n - 1) connections: each application (n) needs to support an interface to all other applications (n - 1) ___________________________________________________________ © Copyright 2011 – Martijn Linssen martijnlinssen.com
  • 18. 18 / 32 However, the enterprise formula is not that interesting: the formula that is valid for each connected application, has far more impact: 2 * (n - 1) That means that every connected / interfacing application has to support 2 * (n - 1) interfaces: 1 * (n - 1) to give information to all other applications connected (as an answer to their request), and 1 * (n - 1) to do so the other way around (requesting an answer from them) Point-to-point connections thus posed a considerable problem to a company: the more widespread its usage became, the harder it was to maintain and monitor. It in fact reduced all flexibility with regards to upgrading or replacing applications to almost zero Moreover, point-to-point connections are hard to build, as they form a direct link between 2 applications: the combined knowledge of both applications is needed to support them Point-to-point connections are hard to support, as they're usually built as 'temporary' interfaces without much documentation, although that is just a historical note and has little to do with the (lack of) success of point-to-point interfaces themselves. Point-to-point connections are hard to control, as they're mostly built on top of an existing application, meaning that a proprietary application has to bend and stretch itself in order to comply with another application, and vice versa: and that repeated many times As an answer to the problems point-to-point interfacing caused, EAI was developed Enterprise Application Integration (EAI) is a term that describes how applications can be connected. EAI is commonly used to link applications together, and is an answer to the disadvantages of point-to-point interfacing. EAI introduces a centralised hub, that takes over the interfacing effort from the surrounding (i.e. connected) applications. Not only is the translation performed by the hub, but centralisation also manages to bring a great span of control to an enterprise. Within the hub, there usually still is point-to-point interfacing involved. However, EAI-tools are very well suited to do so, thus lifting the burden from applications to the right place. With EAI, replacing an application by another one means that all connected applications don't have to do anything: only within the hub an effort would have to be made. EAI interfacing has proven to be complex but very cost-efficient. However, it didn't solve the diversity-problem ___________________________________________________________ © Copyright 2011 – Martijn Linssen martijnlinssen.com
  • 19. 19 / 32 There is a formula for EAI: it is n2 That means that for every connected / interfacing application in an enterprise, there are n2 connections: each application (n) needs to have an interface translated by the hub for each application (n) The enterprise formula for EAI is thus greater than that of point-to-point interfacing; on the other hand the formula that is valid for each connected application, has become smaller: n That means that every connected / interfacing application has to support n interfaces: 1 to give information to all other applications connected (as an answer to their request), and (n - 1) to do so the other way around (requesting an answer from them) EAI connections are hard to build, as they form a direct link between 2 applications: the combined knowledge of both applications is needed to support them. EAI connections are easy to control, as they're centralised in one tool and place. As an answer to the problems EAI interfacing didn't solve, ESB developed An Enterprise Service Bus (ESB) is a virtual bus through which services travel within an enterprise. • An ESB's primary goal is to offer a very high degree of standardisation • An ESB encompasses services, which are usually, but not exclusively, Web services Those services are wrapped in envelopes which are usually, but not exclusively, designed according to SOAP. There's no consensus about whether an ESB does, or doesn't, transform messages. Some say it's not part of it, others say it is but that it should happen "outside the bus". ___________________________________________________________ © Copyright 2011 – Martijn Linssen martijnlinssen.com
  • 20. 20 / 32 The strict version of an ESB is that the connected applications all comply with the standard message and protocol used in the bus, and the ESB itself only routes the messages - establishing a clear difference between itself and Enterprise Application Integration (EAI) There is a formula for ESB: it is n That means that for every connected / interfacing application in an enterprise, there are n connections: each application (n) needs to port their own interface to the bus. On the application level, the formula is also n. That means that every connected / interfacing application has to support n interfaces: 1 to give information to all other applications connected, and (n - 1) to do so the other way around. Formula-wise, ESB is the best of all solutions so far ESB connections are very hard to build, as they require the ability to support highly customized messages from just day-to-day applications. ESB connections are hard to control, as they're mostly built on top of an existing application. ESB services involve applications having to speak the language of the bus; with this approach, IT is in fact back to the point-to-point interfacing of some decades ago, although the diversity in messages exchanged has now become slightly less. On the application level, a lot of effort and time is spent in information exchange. Replacing an application by another one means that the new application has to support all the existing interfaces in the IT-landscape before it can be connected. ESB is a very nice thought on the IT enterprise level, were not much has to be done: on the application level however, ESB interfacing has proven to be very complex, costly and time-consuming. Most applications, that aren't specialised in creating highly customised interfaces, produce unstable and unreliable interfaces From a business and ROI point of view, pure ESB's are financial disasters ___________________________________________________________ © Copyright 2011 – Martijn Linssen martijnlinssen.com
  • 21. 21 / 32 In the next chapter, which will be slightly shorter, the differences between all these approaches, and of course the pros and cons ___________________________________________________________ © Copyright 2011 – Martijn Linssen martijnlinssen.com
  • 22. 22 / 32 9. History with hindsight In the previous chapter, the history of Integration passed: point-to-point, EAI and ESB. For those who read and grasped chapter 1 through 7, it'll be clear why I favour which one - but let me explain it in more detail What are the differences between the different historical approaches? The crucial difference is that EAI describes a hub and spoke architecture, where proprietary messages to and from applications are translated by a central integration broker. Yes, that is the European Parliament chapter, among others As described, applications can plug in and out of the landscape this way, and the necessary transformation of information is taken care of by the central hub In an ESB, there is no hub: the hub has become a bus. Transformation of information is done by the applications themselves, who thus custom-built their (small) integration engine which does the job. If you want to plug into the Matrix there, you have to follow all the courses - and that takes a lot of time and money, unlike Neo learned combat There are advantages and disadvantages to EAI as well as ESB Greatest advantage of EAI is its cost effectiveness, and the fact that it can support just about anything. Greatest advantage of ESB is the fact that it doesn’t support just about anything, but needs and enforces a highly standardised environment The biggest disadvantage of EAI is the fact that scalability might become an issue because everything goes through the central hub, but that worry was years ago. The biggest disadvantage of an ESB is the fact that there are either numerous and various integration brokers in the landscape, each tightly coupled to the applications themselves, with a highly specialized knowledge required, and a lot of licensing costs involved, or add-ons to existing applications and systems that might not be part of the default architecture in mind. ___________________________________________________________ © Copyright 2011 – Martijn Linssen martijnlinssen.com
  • 23. 23 / 32 In either case, they're mostly uncontrolled, bespoke, and hard and costly to maintain. And basically, in this scenario, the ESB does nothing else than route messages - now how exciting is that? From an IT enterprise point of view, there is not much of a difference between properly conducted EAI and ESB: applications are decoupled and accessible via services (or standardized interfaces) and as such appear to be plug-and-playable to the business. From a business PoV, however, cost plays a role, time-to-market, and flexibility: and that's where pure ESB's lose hard, compared to EAI It is somehow even strange that EAI has been “followed up” by ESB, as hardware costs have dramatically decreased over the last few dozen years. On the other hand, integration has become so obvious that most packages now offer a small integration suite with the system itself. However, these are still as monolithic as the majority of those systems themselves, and never capable of handling enterprise-wide integration if there are more packages involved than just their own (cf. SAP XI / PI. Getting better every year, they pale in comparison to integration brokers 15 years ago). They are small message brokers, capable of doing some lightweight transformation at best In order to make ESB viable, integration suites coming along with entire systems should be fully able to handle Enterprise Integration, and they should be included in the system, not sold separately as an option. If it’s not part of the default package, where is the benefit? In the 21st century, every single application should be able to integrate with other applications by "disclosing their functionality" in the form of a message. Message transformation will always be necessary, as there can be no such thing as a worldwide agreement on business documents, as the experience of the past 30 years has clearly shown. No matter which standard exists, parties will always make their own subset, or grow away from it; compare this to the average dictionary that describes a full language, and of which no more than very small subsets are used in day-to-day communication So, currently, put your money on the hub-and-spoke architecture of EAI, due to the fact that it is still very -very- costly to realise a true Enterprise Service Bus within most companies the pure ESB way. It is far easier, faster and cheaper to use the EAI model and a Canonical in order to achieve a uniform language across your diverse IT-landscape In chapter no. 10, I'll address the magical mechanism that enables all of this: one uniform business language across an ocean of diverse IT applications, messages being translated from any language into an intermediate one, and everything being seamlessly transported from A through B and back to finally Z; the most important part of integration, usually if not always forgotten: the envelope ___________________________________________________________ © Copyright 2011 – Martijn Linssen martijnlinssen.com
  • 24. 24 / 32 10. The missing link: envelope With a common language, a common transport protocol, and the need to exercise the necessary translation and transformation on both levels in between, there is a growing need to be able to identify all "service requests" on a generic level too Numerous and various requests will be made, in different formats, via different transport protocols. This certainly is not Utopia, but a pragmatic observation that is maintained in almost an evolutionary way: within successful companies IT systems, like people, are selected based on professional excellence (meaning specialist instead of generic properties). Excelling in one area usually means being moderate in a few others, and being capable of good and generic integration is usually one of those. Intellectual supermodels, and good-looking scientists: they are either scarce or non-existent When all those requests are being made in different ways too, it becomes nearly impossible, but at least very time-consuming and costly, to collect and streamline all the information. Whatever the flexibility on the messaging and transportation level may be, addressing all those variations must be made possible in a generic and structured way In the ideal world, all service requests are sent in an envelope, like in the old- fashioned mail. Like in the old-fashioned mail, however, we already see standards on a national level conflicting with those of another country, so even there is no uniform envelope-definition on a global scale. However, the differences are minimal. Anyone can deliver a letter anywhere in the world, simply by reading the envelope - how cool is that? The wheel is there, extremely round, and has been for centuries, if not millennia What needs to be on an envelope? Recipient is a must, Sender would be nice if you want to get feedback, some unique reference and a date timestamp would make it complete. Look at an envelope, and you'll see it all there: a recipient, a sender, and a unique address in 4 stages: country; zip; street and number. In fact, the country indicates how the address is exactly made up: can't use a UK address notation for an NL address, or vice versa. Just do in Rome as the Romans do... The envelope will conceal any diversity and carry across anything. Its architecture and design is so splendid and loosely coupled, that it even allows for ___________________________________________________________ © Copyright 2011 – Martijn Linssen martijnlinssen.com
  • 25. 25 / 32 letter bombs. Of course those are lamentable, but it shows the awesome power of a simple yet sufficient and efficient mechanism. There's only one but: global mail is just one way of delivering a message. There's also UPS, DHL, small couriers, troikas and rickshaws: they need "an envelope" as well. Then, the big problem has to be tackled: what if your envelope isn't equal to my envelope? Yet again, the answer to that is in the FAQ If you order a book at Amazon, they look it up. They put it in a brown carton, and put a barcode on top - Amazon's own envelope. At the doorstep of Amazon, e.g. UPS picks it up. They put the brown carton inside a plastic bag, and stick a barcode on it that is also humanly legible - UPS's own envelope. Across borders, national couriers take over. They probably just turn the plastic bag around and slap another barcode sticker on it - their own envelope. So, history repeats itself: each envelope is wrapped in another one. What does the final recipient do? As he is the only one interested in the package content itself, he unwraps all envelopes. As he has no need for them in the future, he throws them away Been there done that, haven't we? Let's compare that to our favourite mailman • You have a letter, which you put in an envelope • The envelope makes it to the mailbox • When the mailman makes it to the mailbox, he takes out his own envelope and wraps that around all other envelopes. That envelope is what people usually call a mailbag. He tags the mailbag so he knows who the sender was (this very mailbox), thus adding Sender, unique reference and date-time • At the post office or sorting center, he unwraps the envelope / empties the mailbag. The content gets sorted, and makes it into other envelopes: mailbags to carry them to the final destination • Those envelopes make it to distribution centres, are once more emptied, and sorted again into smaller portions, fitting a street or district / quarter • The mailman delivers the envelopes to their addresses, et voila, done deal So, another mystery solved: 1. An envelope will be the link between messaging and transportation, and survive transformation 2. All attributes on an envelope are mandatory, well-defined and highly standardised 3. If it's not your envelope, just wrap your own around it 4. If you want to read its content, you'll have to unwrap all other envelopes Next: messaging, transportation and envelope collaboration bringing you 100% BPM, SOA or EDA ___________________________________________________________ © Copyright 2011 – Martijn Linssen martijnlinssen.com
  • 26. 26 / 32 11. Orchestration I've compared the diversity of an IT application landscape and managing its information exchange in a uniform way to translation, with the European Parliament as a perfect example of translating dozens of languages via three intermediate languages. In IT, we only need one, as languages (syntaxes) there are far less complex than in the linguistic world I've used a similar metaphor to explain how different transport protocols can be handled, by referring to James Bond's opening scenes, where he takes all kinds of transportation in order to escape death. He simply manages by getting off one kind of transportation, and getting on another one. Clever hey? In the previous chapter, I've explained the no. 10 of Integration: the envelope. I explained that mechanism that needn't be explained by pointing out how the everyday mailman handles letters, by wrapping and unwrapping envelopes over envelopes Together with the hub-and-spoke architecture, it is evident that an Integrated Enterprise needs a single point of entry for all this to make real sense. An airport has one control tower, a house has one front door, a lock has one keyhole, a labyrinth has one centre, a process has one starting point, everything has a beginning and an end... And an orchestra has one orchestrator With all interfaces and services passing through the same Enterprise Integration front door, they can be perfectly orchestrated. There will be many points of physical transport entry, and these will have to guarantee delivery to the single Orchestration point of entry. Every single message can then be received, transformed and routed to its destination point. There, business logic will be applied, and something will be sent back in return: a technical confirmation receipt, a functional acknowledgment, a reply to a request, an error: all that is up to the agreements made between the sender of the message and its recipient. ___________________________________________________________ © Copyright 2011 – Martijn Linssen martijnlinssen.com
  • 27. 27 / 32 They will make it past the Orchestrator again, completing the transaction, who then has to guarantee delivery by the many points of physical exit This way, everything going on externally and needing your business process touch, will be handled by one single person - well it'll of course be a machine, but stick with me please. Will that effectively give you the ability to manage those externally offered business processes? Yes. Maybe manage is a big word, but it will give you full traceability over them: you'll know who did what when, what and when the outcome to that was - it will all pass the Orchestrator and you can decide to keep the record of that forever What if you'd also send and receive your internal service requests through the Orchestrator? Don't mention the O-word nor the P-word (Overhead and Performance), just think about it from a logical, conceptual point of view. Would it be possible? I think so, why not? Would it be helpful? I think so, why not? Would it give you Enterprise-scale BPM via a small-step approach, where you can take one application or system at a time? Yes Tick of the BPM box right there then, no need for massive million dollar devouring seemingly endless big bang projects If you don't do the BPM, but just stay there, with all externally-used and internally used service requests routing through one and the same Orchestrator, wouldn't that be a nimble yet full-blown SOA? Wouldn't it? Yes Tick of the SOA box right there then, no need for massive million dollar devouring seemingly endless big bang projects ___________________________________________________________ © Copyright 2011 – Martijn Linssen martijnlinssen.com
  • 28. 28 / 32 12. The dont's of Integration This chapter is about debunking TLA's and FLA's. XML, SOAP and REST primarily, but Webservices and other concepts whose added value is flimsy or even absent, will get proper attention XML is a syntax, making it part of the Integration messaging theme Back in 2000, when I started to say that XML doesn't add anything to Enterprise business application integration other than overhead, complexity and confusion, the most-heard pro of XML was "it's the language of the future" I always responded with "I can't do business with the future, can I?" and explained how 1,000 years ago we Europeans (economically, politically) all spoke Latin in Europe, followed by French in 1700's, and English in 1900's, without anything ever changing to (economical, political) business itself as a result. Was there a businesscase for speaking the new language? At some point, when a growing minority started to speak it, yes. Has this ever been the case with XML? No, never. And it never will: Google never used XML, didn't even think about it. Twitter deprecated XML use in 2010, and Facebook is doing so now. Google moved to a proprietary format, Twitter and Facebook "will go flat-file": JSON Don't use XML for Integration. It is nice for humans to read, but in the machine- to-machine integration world, it has failed to prove its value in the last 10 years - because there is no value SOAP can't be placed in any of the Integration themes (messaging, transportation, transformation). It touches some aspects of The Envelope SOAP was originally started in 1998 by Microsoft as an attempt to do what it says: Simple Object Access Protocol. W3C adopted it to describe exchanging structured and typed information between peers in a decentralized, distributed environment. It is an exchange mechanism, but relies exclusively on a fixed message format (XML) and a fixed transportation protocol (HTTP). Its main focus is on the SOAP envelope, yet doesn't describe that at all: the only mandatory thing in there is the word Envelope itself. ___________________________________________________________ © Copyright 2011 – Martijn Linssen martijnlinssen.com
  • 29. 29 / 32 Everything SOAP tries to implement has already been taken care of by SMTP, the transport protocol used for email, in a far better, standardised and successful way. SOAP will cause you to reinvent the wheel and describe your own envelope, yet tie you down with highly interpretable mandatory or optional elements. Will it audit or certify your SOAP inventions? No. So what's the use? Don't use SOAP for Integration. It restricts you to one message syntax, one transportation protocol, and leaves it up to you to reinvent your own SOAP thing. It has failed to prove its value in the last 10 years - because there is no value Web services can't be placed in any of the Integration themes (messaging, transportation, transformation). Like SOAP, they focus on a very fixed tech solution for unknown business problems W3C defines a Web service as "a software system designed to support interoperable machine-to-machine interaction over a network. It has an interface described in a machine-processable format (specifically Web Services Description Language WSDL). Other systems interact with the Web service in a manner prescribed by its description using SOAP messages, typically conveyed using HTTP with an XML serialization in conjunction with other Web-related standards". What's the goal? The point? What are the alternatives? Why is this the best solution? Every single format can be processed by machines, if -and only if- it can be understood and agreed upon by humans doing business. W3C doesn't know anything about doing business, and XML, SOAP, WSDL, UDDI and Web Services all were invented by them Don't use Web services for Integration. They restrict you to one message syntax, one transportation protocol, one way of writing down your web service, and leave it up to you to reinvent your own web service. It has failed to prove its value in the last 9 years - because there is no value I have struggled to write this down, as a chapter with a subject like this basically only destroys. There is one big positive thing to mention: all these concepts were invented by W3C. Tim Berners-Lee heads that organisation, who accidentally invented the Internet - so that's why people put value onto W3C concepts What these acronyms all have in common, is inventing their own way of laying down the exact structure of How things should be said - without taking into consideration What can be said ___________________________________________________________ © Copyright 2011 – Martijn Linssen martijnlinssen.com
  • 30. 30 / 32 13. The do's of Integration Final chapter, this is the summary and conclusion, to be used as some sort of checklist if you like When conducting enterprise business application integration, within the enterprise IT landscape among applications and systems, or from there to others at another company or even directed towards the customer, here are the pragmatic rules: Start at the business level, and write down which process it concerns. What is the business event, which its trigger, how often does it occur, how sensitive is the data? Why should this be automated, in other words, what is the manual alternative and the benefits and concerns of both? Write out the functionality it concerns. List the entities, their cardinalities, and all attributes concerned, and describe them as complete as possible. Here's an example There are about 20,000 employees, each one of which has 1 or 2 addresses, and working on 1 to 5 assignments. The employee data has attributes, of which the functional description is shown. At this level, anyone can read and understand what is to be exchanged: business people, tech people, inside people, outside people, suppliers, customers: everyone Depending on the bsiuness goal, some of these will be required, and some won't: specify that as well. When creating a new employee, First name, last name and personal ID will be required, but not an employee number: that will be handed out by the system. When modifying an employee, that employee number will be required, including the last name, but perhaps not the personal ID. So, for the intended business goal, write down which attributes are required and when Then, with this Information list in hand, make the transition to the systems on both sides: do those requirements meet and match on both sides? If so, you can do business, if not, you have an issue. This indeed means that business agreement, functional analysis and information system (gap) analysis is part of the preliminary phase and could result in the conclusion that business simply can't be done ___________________________________________________________ © Copyright 2011 – Martijn Linssen martijnlinssen.com
  • 31. 31 / 32 If successful, both sides will add their information system requirements (these are actually limitations) to the Information: Last name will be 30 characters in one system, and 25 in the other - what to do? Is the business agreement still in place or do we have to revise it? Apart from the messaging level, there will now be limitations on the transportation level as well: what are the existing possibilities to get messages across? Data sensitivity as defined by the business will lead to transport security and encryption and access restriction requirements, and these will meet limitations: again, the common subset has to be found. Again, is the business agreement still in place or do we have to revise it? On both sides there will now be a full technical description of the interface, complete with field length, datatype, format. The business frequency and the given cardinalities will give a rough guesstimation of the resulting file size, thus the volume and load, making it possible to calculate storage requirements and capacity planning. Files will have to be kept online for a certain amount of time, and archived as well, depending on compliancy requirements the business has to follow up - all that costs space as well. For the transformation into an intermediate language, this will have to be repeated of course. Here the IT department will mostly determine how long these have to be available. If used as basis for a full-blown BPM, the underlying processes will drive these requirements too, although files should only be used for resending (on failure) and "evidence" of what was received and sent At this point, the issue arises: what file format and transportation protocol should we use? Which ever ones are most practical and pragmatic, will be the answer For A2A (application-to-application, a TLA used to describe internal integration) the usual pick will be flat-file for the message and shared folders for transport. A flat-file can be a CSV, a fixed flat-file, JSON, whichever makes you feel good. Maybe it's an XML, an iDoc: all that is up to the applications themselves. When you use an intermediate language and a canonical model, you will have to choose a file format yourself. If volumes and frequencies are high, a more sturdy transport protocol might be required; thing of queueing mechanisms. XML will definitely pose capacity and storage problems there, ask Google, Twitter and Facebook and companies in the traditional B2B area If one of the A2A interfaces "makes it out there", i.e. is a candidate for exchanging information with external parties and you engage in B2B or B2C, the question of file format and transportation protocol will arise once again - just in a different context. What is most appropriate then? Again, do in Rome as the Romans do. Speak X12 in US industries, EDIFACT in Europe, Swift in banking, HL7 in health, etcetera. Transport? FTP or FTPS for static ___________________________________________________________ © Copyright 2011 – Martijn Linssen martijnlinssen.com
  • 32. 32 / 32 solutions, HTTP for more dynamic ones and AS1 or AS2 for the "bigger boys". Treat each of these questions as a simple daily one: should I pick UPS, global or local mail or courier to transport my package? That of course depends on what the package is, and how fast it should get somewhere Last but absolutely not least, you should define an envelope for each interface: who is the sender, who is the recipient, what does it contain? This will enable you to route its contents freely across the globe, just like SMTP (email) does, without opening the envelopes and looking at the letters. Don't address envelopes to URLs or other silly notations, you're doing business with an organisational entity, not a machine: let the organisation make up where exactly they want to route their stuff internally, but keep that transparent for the outside world In an effort to keep this chapter short, I'll just mention that business integrity requirements will lead to business rules and exceptions, which can be translated into error handling and (user) feedback on a functional and technical level. These will lead to functional and technical acknowledgments on the interface level, retry mechanisms when a file can't be sent out, and exception handling and agreements for that. To really define the interface, the byte-level has to be spelled out as well: character code (ASCII or UTF-8, EBCDIC or UCS-2, any other flavours), hexadecimal form of delimiters, periods or commas, etcetera: the real dirty stuff This concludes my thoughts on Perfect Integration. I hope I touched all possible questions and answers, and helped unravel the mystery of buzzword Three Letter Acronyms. I think I have reached a satisfactory level of detail without losing your attention - but you'll be the judge of that ___________________________________________________________ © Copyright 2011 – Martijn Linssen martijnlinssen.com