This talk was presented at the Kafka Meetup London meeting on 20 January 2016. You can find more information about Message Hub here: http://ibm.biz/message-hub-bluemix-catalog

1. 1 © 2016 IBM Corporation
IBM Message Hub service in Bluemix
Apache Kafka in a public cloud
Andrew Schofield
IBM Hursley Park

2. 2 © 2016 IBM Corporation
A scalable, distributed, high throughput message bus for your cloud-native applications
Connects services inside Bluemix
with your systems beyond
Connects microservices,
using open standard protocols
Streams data to analytics
services for real-time insights
Built on Apache
What is Message Hub?

3. 3 © 2016 IBM Corporation
Bluemix is an open-standards, cloud-based platform for building, running, and
managing applications
Build your apps, your way
Use the most prominent
compute technologies to power
your app: Cloud Foundry,
Docker, OpenStack.
Extend apps with services
A catalog of IBM, third party,
and open source services allow
the developer to stitch an
application together quickly.
Scale more than just instances
Development, monitoring,
deployment, and logging tools
allow the developer to run and
manage the entire application.
Layered security
IBM secures the platform and
infrastructure and provides you
with the tools to secure your
apps.
Deploy and manage hybrid apps
seamlessly
Get a seamless dev and
management experience across a
number of hybrid
implementations options.
Flexible pricing
Try compute options and services
for free and, when you’re ready,
pay only for what you use. Pay as
you go and subscription models
offer choice and flexibility.
What is IBM Bluemix?

4. 4 © 2016 IBM Corporation
Start with a boilerplate

5. 5 © 2016 IBM Corporation
Or a runtime for building your app from scratch

6. 6 © 2016 IBM Corporation
Choose from a wide range of services to use in your app

7. 7 © 2016 IBM Corporation
Including Message Hub

8. 8 © 2016 IBM Corporation
Message Hub is Kafka in Bluemix

9. 9 © 2016 IBM Corporation
Bind to the service to get your credentials

10. 10 © 2016 IBM Corporation
Why are we using Kafka?
Kafka has many valuable characteristics
• Fast
• High throughput to handle whatever you throw at it
• Scalable
• Easily add more brokers to increase throughput
• Durable
• Consumers can be stopped for maintenance and catch
up without impacting the service
• Distributed
• Replication and failure characteristics make managing
the service easy

11. 11 © 2016 IBM Corporation
Kafka in Bluemix
• Some services in Bluemix integrate well with Kafka’s
streaming data-flow model
• For example, analytics such as Spark
• Exposing, not hiding, the Kafka interface gives an event-
driven programming interface for application code
• Some internal services also use Kafka as plumbing
• Some compound services make use of Kafka for
internal communication
• Think SaaS built using a PaaS
• Message Hub is intended to support all of these
scenarios

12. 12 © 2016 IBM Corporation
Why might you use Kafka on the cloud?
• The benefits of Kafka are applicable to services in
the cloud
• For example, feeding data into real-time analytics
• Serious use of any software takes investment
• Either grow your own skills
• Or pay someone else to support you
• Pay-as-you-go pricing makes it a cost-effective
platform for experimentation
• For example, try some analytics on telemetry data from
your mobile app

13. 13 © 2016 IBM Corporation
Kafka use cases which fit with Message Hub nicely
• Messaging
• Replacement for a traditional message broker
• Website activity tracking
• High-volume feeds of site activity such as views,
searches
• Metrics
• Aggregating operational monitoring data
• Log aggregation
• Copying physical log files from servers to central point
• Stream processing
• Data flowing on the topics between processing stages

14. 14 © 2016 IBM Corporation
Kafka use cases which don’t fit so well
• Event sourcing
• State changes are logged as time-ordered sequence,
rather than storing the current state
• Commit log
• Assists with resynchronisation of failed nodes
• Change data capture
• Take a feed of database changes and store them as a
compact change history in Kafka, and feed data into
multiple consumers from Kafka
The difference is that log data needs to be retained
and perhaps compacted

15. 15 © 2016 IBM Corporation
In the public cloud, you’re sharing resources
• Two instances of Message Hub available now
• Dallas and London
• Each is a 5-node cluster, shared among the tenants
• Cluster-wide configuration is common
• Replication factor: 3
• Clean-up policy: delete
• Log retention: 24 hours
• Maximum message size: 1 MB
• If you need more performance or isolation
• Bluemix Dedicated
• Dedicated hardware in IBM datacenters, IBM ops team
• Bluemix Local
• Customer hardware, IBM ops team

16. 16 © 2016 IBM Corporation
Pricing

17. 17 © 2016 IBM Corporation
Pricing example
• A workload of 10 msg/second published to a topic
with 1 partition
• 26,784,000 messages in a month
• About $25

18. 18 © 2016 IBM Corporation
Modifications to Kafka for use in the cloud
• Client connections must provide credentials
• Username or API key tell us which tenant is connecting
• Client connections are always encrypted
• Connections are over the public internet
• Real topic names have a tenant-specific prefix
• Kafka topics are not hierarchical
• Number of partitions is capped per tenant
• Per-tenant metering data used to generate bills

19. 19 © 2016 IBM Corporation
Topic management
• Message Hub topics do not auto-create
• Auto-create can only give a default topic config anyway
• Kafka users can use kafka-create-topic.sh
• You can’t access zookeeper or a shell
• Two options for Message Hub:
• Admin REST API for create/delete/list topics
• Bluemix UI

20. 20 © 2016 IBM Corporation
Topic management in the UI

21. 21 © 2016 IBM Corporation
Topic configuration
Property Can set in MH? Default Description
cleanup.policy No “delete” “compact” would enable log
compaction use cases, but
would consume more
storage
delete.retention.ms No 24 hours Applies to log compaction
retention.bytes No 1 GB Maximum log size
retention.ms Not yet 24 hours How long a log segment is
retained
partitions Yes 1
replicas No 3
Defaults are taken from the cluster config, and that’s common for all tenants

22. 22 © 2016 IBM Corporation
Message Hub deployment architecture
Red zone (internet)
Yellow zone (DMZ)
Green zone (internal)
FIREWALL
FIREWALL
App using
Kafka client
HA proxy
HA proxy
Kafka ZK
Bare-metal
Bluemix services: Logging, billing, authentication
Metering
Alerting
Cloudant DB
TLS
TLS
Bare-metal

23. 23 © 2016 IBM Corporation
Continuous delivery and microservices
• Message Hub is developed using a continuous delivery
process and a microservice architecture
• It’s built out of small pieces that we can enhance rapidly
• Multiple deployment environments with promotion
from dev/test towards production
• We can deploy changes into production as often as required,
even several times as day if we had to
• When there’s a critical patch for Kafka, we…
• Build the patch as deploy to dev/test
• Test it thoroughly
• Evaluate suitability for promotion to staging…
• And repeat until the patch reaches production

24. 24 © 2016 IBM Corporation
Example – multiple partitions
• The first delivery of Message Hub actually only
supports one partition per topic
• Deliver quickly and iterate often
• We are about to add multiple partitions
• Poised in our staging environment awaiting promotion
• Includes:
• Admin API
• UI
• Metering

25. 25 © 2016 IBM Corporation
Performance
• Throughput of the cluster is about 300,000 msg/s
• 100-byte messages
• Secure connection from public network
• Disk encryption of Kafka log
• Secure connections for all user data, even within DMZ
• Latency ranges from 20ms to 100ms, average 50ms

26. 26 © 2016 IBM Corporation
Involvement in the community
• Contributed several patches, particularly as release
of 0.9 approached
• Primarily related to SSL/TLS and SASL
• Planning to propose Kafka Improvement Proposals
shortly to augment SASL capabilities
• Have submitted sessions for the Kafka Summit in
April 2016

27. 27 © 2016 IBM Corporation
A scalable, distributed, high throughput message bus for your cloud-native applications
Connects services inside Bluemix
with your systems beyond
Connects microservices,
using open standard protocols
Streams data to analytics
services for real-time insights
Built on Apache
What is Message Hub?

28. 28 © 2016 IBM Corporation
Questions?
Twitter:
@IBMmessaging
DeveloperWorks:
https://developer.ibm.com/messaging/message-hub/
Bluemix blogs:
https://developer.ibm.com/bluemix/blog/
LinkedIn:
ibm.biz/ibmmessaging
Message Hub service:
https://console.ng.bluemix.net/catalog/services/message-hub