Apache Camel - Stéphane Kay - April 2011

1. Apache Camel
Stephane Kay
Ville de Lausanne

2. Scope
 IS
Integration
 EIP
 Routing
 Mediation
 Camel Container
 Unit testing
 Hands-on
 Real Use Case
JMS
/ Active MQ
JBI / NMR
OSGI / ServiceMix
Scala

3. Table of contents
IS Integration heuristics
I.
◦
◦
Common issues
Paradigms
Camel’s bleat
II.
◦
◦
Concepts responding to integration needs
Getting started
What’s under the humps?
III.
◦
◦
Camel architecture
Distribution
Camel in Action
IV.
◦
◦
Persistence, transactions, concurrency, failover
Aggregation
A Real Use-Case
V.
◦
Description, issues & achievement

4. I. IS Integration Heuristics

5. I. IS Integration heuristics
Just do it !

Just coding simple things might be just not so simple …
public class FileCopier {
public static void main(String args[]) throws Exception {
File inboxDirectory = new File("data/inbox");
File outboxDirectory = new File("data/outbox");
outboxDirectory.mkdir();
File[] files = inboxDirectory.listFiles();
for (File source : files) {
if (source.isFile()) {
File dest = new File(
outboxDirectory.getPath()
+ File.separator
+ source.getName());
copyFile(source, dest);
}
}
}
private static void copyFile(File source, File dest)
throws IOException {
OutputStream out = new FileOutputStream(dest);
byte[] buffer = new byte[(int) source.length()];
FileInputStream in = new FileInputStream(source);
in.read(buffer);
try {
out.write(buffer);
} finally {
out.close();
in.close();
}
}
}

6. I. IS Integration heuristics
A Framework?

The bad :
◦ Learning effort consumes time.
◦ Adds anarchical dependencies.
◦ Risk of coming with a new need / not
implemented

The good :
◦ Gain value after a training period
◦ Leverage your code to a higher abstraction
◦ Potentially a future standard ?
 Indicator : community commits & posts curve.

The plus-value
◦ Not obvious

7. I. IS Integration heuristics
Separation of concerns

IS Integration involves
1. Endpoints
 A common endpoint interface abstraction
2. Data Processing
 Using Message / exchange abstraction and
conversion tool
3. Routing
 A common vocabulary / pattern as DSL
foundation.

8. I. IS Integration | endpoint
Message Endpoint pattern


Routes should refer to a common abstract
Message Endpoint rather than directly using their
technical interface
A good Endpoint abstraction is a URI description

9. II. Camel’s bleat | data
Uniformed Exchange

Message’s container during routing

10. I. IS Integration | data
Data Transformation

Data Format Transformation
◦ Message body is transformed from one
format to another.
 CSV -> XML

Data Type Transformation
◦ Java Type of message body is
transformed/casted into another.
 java.lang.String -> java.jms.TextMessage
 java.io.File -> java.lang.String

11. I. IS Integration | routing
Use case : Filtering Route
◦ How can a component avoid receiving
uninteresting messages ?
◦ …coming from a different system

12. I. IS Integration | routing
EIP?

A Catalog of Design patterns for common IS
integration issues

http://camel.apache.org/enterprise-integration-patterns.html

13. I. IS Integration | routing
EIPs Samples…

Filter
◦

Splitter
◦

How can a component avoid receiving uninteresting messages?
How can we process a message if it contains multiple elements, each of which may have to be
processed in a different way?
Router
◦ How can you decouple individual processing steps so that messages can be
passed to different filters depending on a set of conditions?

Aggregator
◦ How do we combine the results of individual, but related messages so that
they can be processed as a whole?

14. I. IS Integration heuristics
Putting all needs together

Route definition attributes :
◦ Endpoints
 Producer / Consumer(s)
◦ Processor(s)
 Uniformed exchange processing interface
◦ Predicate(s)

A route language should provide those
key abstractions
Endpoint A = endpoint(“activemq:queue:quote”);
Endpoint B = endpoint(“mq:quote”);
Predicate isWidget = xpath(“/quote/product =
„widget‟”);
from(A).filter(isWidget).to(B);

15. I. IS Integration heuristics
I. Review

IS Integration paradigms
◦ Endpoint abstraction -> URIs?
◦ Data Tx -> Functions // Java beans ?
◦ Routing -> EIPs

16. II. Camel’s bleat
Apache Camel is a Powerful Open
Source Integration Framework based
on known Enterprise Integration
Patterns
*
 Mission Statement :
Making integration easier and more
accessible to developers

17. II. Camel’s bleat | endpoint
Endpoint URI

A camel endpoint URI consists of
three parts :
ftp://riders.com/orders?username=rider&password=secret
Scheme
Context path
Options

18. II. Camel’s bleat | routing
Java DSL : Builder pattern

The Builder Pattern (GOF) is central to the Camel Java DSL
public class FileCopierWithCamel {
public static void main(String args[]) throws Exception {
// create CamelContext
CamelContext context = new DefaultCamelContext();
// add our route to the CamelContext
context.addRoutes(new RouteBuilder() {
public void configure() {
from("file:data/inbox?noop=true").to("file:data/outbox");
}
});
// start the route and let it do its work
context.start();
Thread.sleep(10000);
// stop the CamelContext
context.stop();
}
}

19. II. Camel’s bleat | data
Data Format transformations

Marshal / unmarshal
from("activemq:QueueWithJavaObjects”)
.marshal().jaxb()
.to("mq:QueueWithXmlMessages");
from("activemq:queue:MY_QUEUE").unmarshal().zip().proce
ss(new UnZippedMessageProcessor());
http://camel.apache.org/data-format.html

20. II. Camel’s bleat | data
Message Translator EIP

Aka Adapter (GOF)

Using a Processor
from("direct:start").process(new Processor() {
public void process(Exchange exchange) {
exchange.getIn().setBody(in.getBody(String.class) + " World!"); }
}).to("mock:result");

Using Bean binding
from("direct:start") .bean(MyBean.class, "hello(String,String)") .to("mock:result");

Using transform() DSL
from("direct:start").transform(body().regexReplaceAll(“n”,”<br/>”))
.to("mock:result");

21. II. Camel’s bleat | first ride
Getting started : IDE
Eclipse
 Maven integration (m2eclipse)

◦ http://m2eclipse.sonatype.org/sites/m2e
◦ Check your eclipse.ini : locate your JDK
-vm C:/dev/java/jdk1.6.0_18/bin

Optional : Fuse Camel plugin for
Eclipse
◦ http://repo.fusesource.com/beta/eclipse/u
pdate/

22. II. Camel’s bleat | first ride
Getting started : Maven

Basic Maven configuration

23. II. Camel’s bleat | first ride
First Ride
Copy contents from one folder to
another, control final name / dir
 JUnit

◦
◦
◦
◦

Startup context
Route definition : File component
Processor / Exchange
Simple Language (File Component)
Delegate JUnit test to Main class

24. II. Camel’s bleat | first ride
Second Ride

Add some features to JUnit Class
◦
◦
◦
◦

Log
Add failure if file already exist
Predicate : Simple Language
Error handling
Router EIP (Choice)

25. II. Camel’s bleat | spring
Spring XML : content-based router
<camelContext>
<route>
<from uri="activemq:NewOrders"/>
<choice>
<when>
<xpath>/order/product = 'widget'</xpath>
<to uri="activemq:Orders.Widgets"/>
</when>
<otherwise>
<to uri="activemq:Orders.Gadgets"/>
</otherwise>
</choice>
</route>
</camelContext>

26. II. Camel’s bleat
II. Review
Endpoint / Component URIs
 Java DSL
 How to start / process exchanges
 Log, error handling / routing


27. III. What’s under the humps?

High-level architecture
◦ Container & Services
Toolkit provided out-of-the box
 Distribution


28. III. What’s under the humps?
High-level view

Container responsibilities :
◦ Create and activate the necessary Endpoint instances using the available
Component implementations

29. III. What’s under the humps?
Camel Context Services

Components
◦

Endpoints
◦

Contains the loaded data formats.
Registry
◦

Contains the loaded type converters. Camel has a mechanism that allows you to manually
or automatically convert from one type to another.
Data formats
◦

Contains the routes that have been added.
Type converters
◦

Contains the endpoints that have been created.
Routes
◦

Loads components on the fly either by autodiscovery on the classpath or when a new
bundle is activated in an OSGi container.
Contains a registry that allows you to look up beans. By default, this will be a JNDI registry.
If you’re using Camel from Spring, this will be the Spring ApplicationContext. It can also be
an OSGi registry if you use Camel in an OSGi container.
Languages
◦
Contains the loaded languages. Camel defines 3 different DSLs (Java, XML/Spring, and
Scala) and allows you to use many different languages to create expressions.

30. III. What’s under the humps?
Out of the box : EIPs

50+ Routing Patterns based on EIPs

http://camel.apache.org/enterprise-integration-patterns.html

31. III. What’s under the humps?
Out of the box : Components
70+ Endpoint impl. aka « Components »
 All available via URI/scheme paradigm
 http://camel.apache.org/components.html


32. III. What’s under the humps?
Out of the box : Data Format

18 Data Formats - uniformed usage :
from(“file://input").marshal().zip().to("activemq:queue:MY_QUEUE");

http://camel.apache.org/data-format.html

33. III. What’s under the humps?
Out of the box : Languages

Predicates & Expressions

http://camel.apache.org/languages.html

34. Camel distribution
http://camel.apache.org/download.html
Components sources
 Examples
 Maven Central (current v2.7)

◦ Snapshot Repo
<repository>
<id>apache.snapshots</id>
<name>Apache Development Snapshot Repository</name>
<url>https://repository.apache.org/content/repositories/snapshots/</url>
<releases> <enabled>false</enabled> </releases>
<snapshots> <enabled>true</enabled> </snapshots>
</repository>

35. III. Review
Camel is an integration framework
 Based on Enterprise Integration Patterns
 Routing and mediation
 Easy to use DSL to define routes
 No heavy specification
 No container dependency
 Payload agnostic
 Connectivity to a great wealth of
transports


36. IV. Camel in Action
Test support
 Transactions
 Exceptions
 Concurrency
 Persistence
 Aggregation
 Runtime


37. IV. Camel in Action| test
Camel Test Support

Provides context/registry support

MockEndpoint assertions
resultEndpoint.expectedBodiesReceived(expectedBody);
template.sendBodyAndHeader(expectedBody, "foo",
"bar");
resultEndpoint.assertIsSatisfied();

38. IV. Camel in Action| test
Using test support
CamelTestSupport Junit
 @EndpointInject
 @Produce
 Filter Pattern


39. IV. Camel in Action | tx
Transactions

Component level
◦ Polling strategy
public interface PollingConsumerPollStrategy {
boolean begin(Consumer consumer, Endpoint endpoint);
void commit(Consumer consumer, Endpoint endpoint, int polledMessages);
boolean rollback(Consumer consumer, Endpoint endpoint, int retryCounter, Exception cause) throws
Exception;
}

Routing level
◦ Transactional Client EIP
 http://camel.apache.org/transactional-client.html
◦ Transacted DSL
from("jetty:http://localhost/myservice/order").transacted()
.to("bean:validateOrder").to("jms:queue:order");

40. IV. Camel in Action | tx
Propagation

Define transaction behaviour
<bean id="PROPAGATION_REQUIRED"
class="org.apache.camel.spring.spi.SpringTransactionPoli
cy">
<property name="transactionManager" ref="txManager"/>
<property name="propagationBehaviorName"
value="PROPAGATION_REQUIRED"/>
</bean>
<bean id="PROPAGATION_REQUIRES_NEW"
class="org.apache.camel.spring.spi.SpringTransactionPolicy">
<property name="transactionManager" ref="txManager"/> <property
name="propagationBehaviorName" value="PROPAGATION_REQUIRES_NEW"/>
</bean>

Use transacted DSL
from("direct:mixed")
.transacted("PROPAGATION_REQUIRED“)
.setBody(constant("Tiger in Action"))
.beanRef("bookService")
.setBody(constant("Elephant in Action"))
.beanRef("bookService“).to("direct:mixed2");

41. IV. Camel in Action | errors
Exceptions

onException clause
new RouteBuilder() {
public void configure() {
onException(IOException.class).maximumRedeliveries(3);
from("direct:start") .process("processor1")
.process("processor2") // <--- throws a ConnectException
.to("mock:theEnd");
}
◦ Note : By default, a producer will not redeliver

Handle
onException(ValidationException.class)
.handled(true).transform(body(constant("INVALID ORDER")));
from("jetty:http://localhost/myservice/order").transacted()
.to("bean:validateOrder").to("jms:queue:order");

42. IV. Camel in Action | errors
Dead Letter EIP
The Error Handler propagates error back to the client.
 Dead Letter Channel will always moved failed
exchanges to this queue.


useOriginalMessage is used for routing the original
input message instead of the current message that
potentially is modified during routing.
from("jms:queue:order:input").to("bean:validateOrder")
.to("bean:transformOrder").to("bean:handleOrder");
errorHandler(deadLetterChannel("jms:queue:dead")
.useOriginalMessage().mamimumRedeliveries(5)
.redeliverDelay(5000);

43. IV. Camel in Action | toolkit
Concurency

Dealing with data flows means mastering
concurency
◦ Seda Component :
from("seda:bar?concurrentConsumers=5").to("file://output");
◦ Thread DSL
 How do we avoid the messages that does not
expect a reply to block ?
from("jms:queue:order") // do some sanity check validation
.to("bean:validateOrder") .to("mock:validate").threads(20)
// do some CPU heavy processing of the message (we simulate and delay)
.unmarshal(mySecureDataFormat).delay(1000)
.to("bean:handleOrder").to("mock:order");

44. IV. Camel in Action | toolkit
Sync to Async

toAsync Processor
from("direct:start").to("mock:a").toAsync("direct:bar", 5).to("mock:result");
from("direct:bar").to("mock:b").transform(constant("Bye World"));

How does this work?
void process(Exchange exchange, AsyncCallback callback) throws Exception;
◦ With AsyncCallback interface :
void onTaskCompleted(Exchange exchange);

Http Async Producer
public class JettyHttpProducer extends DefaultProducer implements AsyncProcessor
◦ Then invoke the callback when reply is ready on the Exchange.
◦ Routing with Spring DSL
<route> <!-- we route from a direct endpoint -->
<from uri="direct:start"/> <!-- log the request -->
<to uri="log:+++ request +++"/> <!-- then doing a non blocking async request/reply to the http server with a pool of 10 threads
-->
<to uri="jetty://http://0.0.0.0:9123/myapp" async="true" poolSize="10"/>
<!-- the reply from the server is logged -->
<to uri="log:+++ reply +++"/> <!-- and to our mock so we can assert we got all responses -->
<to ref="result"/>
</route>

45. IV. Camel in Action | toolkit
Persistence


SQL, JDBC and JPA components provided out-of the box
File idempotent consumer with pluggable repository
from("file://inbox?idempotent=true").to("...");
<bean id="fileStore"
class="org.apache.camel.processor.idempotent.FileIdempotentRepository">
<!-- the filename for the store -->
<property name="fileStore" value="target/fileidempotent/.filestore.dat"/>
<!-- the max filesize in bytes for the file. Camel will trunk and flush the
cache if the file gets bigger -->
<property name="maxFileStoreSize" value="512000"/>
<!-- the number of elements in our store -->
<property name="cacheSize" value="250"/> </bean>
<camelContext xmlns="http://camel.apache.org/schema/spring">
<route>
<from
uri="file://target/fileidempotent/?idempotent=true&amp;idempotentRepository=
#fileStore&amp;move=done/${file:name}"/>
<to uri="mock:result"/>
</route>

Stateful EIP : Aggregator

46. IV. Camel in Action | toolkit
Aggregator EIP

Combine a number of messages together into a
single message.
◦ http://camel.apache.org/aggregator2.html

Completion predicates
from("direct:start“).
.aggregate(header("id"), new BodyInAggregatingStrategy()).completionTimeout(3000)
.to("mock:aggregated");
from("direct:start“).aggregate(header("id"), new
BodyInAggregatingStrategy()).completionSize(3) .to("mock:aggregated");
from("direct:start“).aggregate(header("id"), new BodyInAggregatingStrategy())
.eagerCheckCompletion().completionPredicate(body().isEqualTo("END"))
.to("mock:aggregated");
from("direct:start“).aggregate(header("id"), new BodyInAggregatingStrategy()).
completionSize(header("mySize")) .to("mock:aggregated");

47. IV. Camel in Action | toolkit
Aggregator Strategy & Fail Over

Strategy
public class NumberAggregationStrategy implements AggregationStrategy {
public Exchange aggregate(Exchange oldExchange, Exchange newExchange) {
if (oldExchange == null) { return newExchange; }
Integer num1 = oldExchange.getIn().getBody(Integer.class);
Integer num2 = newExchange.getIn().getBody(Integer.class);
Integer num3 = (num1 != null ? num1 : 0) + (num2 != null ? num2 : 0);
oldExchange.getIn().setBody(num3); return oldExchange;
}
}

http://camel.apache.org/aggregate-example.html

48. IV. Camel in Action | runtime
Running Camel

Known Deployment Options
◦
◦
◦
◦
◦
◦
◦
◦
Standalone Java Application
Java Web Start
Spring Application
Web Application
Google App Engine
J2EE Application
JBI
OSGi

49. IV. Riding Camel | runtime
Running inside App Server
Servlet component
camel-example-servlet-tomcat
 CXF Component
camel-example-cxf-tomcat
 Camel Web console
actually « Web Component »


50. IV. Review
Camel is not a server
 Camel is lightweight and embeddable
 Camel supports Enterprise Integration
paradigms from the JEE/Spring world.
 Camel is not an ESB though it
implements EIPs.
 Camel doesn’t need but supports any
messaging infrastructure (JMS / MQ)
 Camel supports messaging at JBI
abstraction level


51. V. A real Use-Case

Multiple source FTP import to a centralized repository
◦ Collect files (0-5 x 2-20 Megs) from N different FTP endpoints, every
hours on a 24/7 basis
◦ End-user configuration
◦ Logs : technical + business
◦ Monitor failures
FTP
SAN
OPCON

52. Constraints








Big binary files : maybe JMS would not fit
Implementation should fit some common
corporate approach to entreprise intergation.
Currently only SAN @ client. Do not need App
Server yet.
Client is technical-oriented : would like hands-on.
Keep intact current logs / traces, but must be
possible to connect on JDBC.
Would be interested in indexing meta –infos
within Alfresco (JSR-170).
Central schedule/alerting infrastructure must be
connected.
Nagios could be an option at mid-term.

53. Real issues

Multiple source endpoints pattern :
◦ not found (!!!).
FTP component is an active poller; no
easy way to stop polling after having
consuming all files.
 No easy way to put into configuration
the context path of Endpoint URIs


54. Other applications…
Camel as an ETL : Excel sources
 PeopleSoft to ActiveDirectory
 Bidirectional Federal <-> City
Entreprises DB official data
 Reconcilliation of mutations streams


55. Conclusions

Open-Source
◦ Help yourself through debug mode on sources

Maven-based development
◦ « Real » modularity

Real Standards
◦ EIPs
◦ Maven, Spring, Java6.
◦ JBI (Glassfish)

Gateway to the future
◦ OSGI (Fuse ESB)

Camel. What else?

56. More information needed?

Camel website:
◦ http://camel.apache.org

Camel article:
◦ http://architects.dzone.com/articles/apachecamel-integration

FuseSource website:
◦ http://fusesource.com

Camel in Action book:
◦ http://manning.com/ibsen

Camel in Action Source code:
◦ http://code.google.com/p/camelinaction/

57. Questions