An Introduction to Fluentd Plugins for Couchdb,Amazon SQS/SNS. at "Fluentd Meetup in Tokyo." http://www.zusaar.com/event/193104 #fluentd

1. An Introduction to ...
Fluentd plugins for
CouchDB,
Amazon SQS,
Amazon SNS
@ixixi

2. $ whoami
@ixixi (Yudai Odagiri)
Engineer of Future Architect, Inc.
Hadoop
Ustreamer
CORSS, MongoTokyo2012, ,
GraphDB , Twitter , MongoDB ......
Ustream

3. Agenda
CouchDB?
fluent-plugin-couch
demonstration
Amazon SQS?
fluent-plugin-sqs
Amazon SNS?
fluent-plugin-sns
next features...

4. CouchDB?
Relax!

5. CouchDB?
Document-oriented database
MVCC (MultiVersion Cuncurrency Control)
Auto versioning (No “update”)
RESTfull HTTP JSON APIs

6. CouchDB?
Small footprint
run on mobile devices ( android / iOS )
Multi-master Replication
Replication (via HTTP)

7. CouchDB?
Written in Erlang
Not only DB Server, but also Application &
Web Server.
Built-in web GUI
“Futon”

8. MapReduce on CouchDB

9. CouchDB MapReduce
User-defined query structured as M/R
MapReduce View (index)
view index is built lazily
View Definition (MapReduce)
Result

10. CouchDB MapReduce
Incremental view index update
“possible to use M/R to query huge partitioned clusters in
realtime, instead of having to wait for a whole M/R job to
complete or having stale, occasionally updated indexes.”
~Damien Katz
http://damienkatz.net/2008/02/incremental_map.html
implementation details
{ omitted } this is Lightning Talk :-P
see : http://horicky.blogspot.com/2008/10/couchdb-
implementation.html

11. fluent-plugin-couch

12. fluent-plugin-couch
fluentd (buffered) output plugin
store fluent-event as CouchDB Document to CouchDB
install
$ gem install fluent-plugin-couch
<match couch.**>
configuration type couch
database fluent
host ixixi.cloudant.com
port 80
# for Basic Authentication (optional)
user ixixi
password **********
# Other buffer configurations....
</match>

13. Demo
count tweets in public timeline (use Streaming API)
group by timezone and language in real-time.
M/R view update
(incremental)
bulk insert
Streaming API fluentd flush interval => 5sec.
buffer

14. fluent-plugin-sqs

15. why need queue?
to stop and update receiver application
to consider the state of receiver servers
Receiver servers ( on which heavy processes are running)
may not be able to deal with messages.
fluentd

16. Amazon SQS
Amazon Simple Queue Service (SQS)
Distributed Queue Service
since 2006 ( the oldest service of AWS )
Automatic Replication in data centers
High Availability
High Scalability
Secure

17. why SQS?
Queue Server requires high availability
Generally, high availability queue server is
very expensive
Amazon SQS is inexpensive
$0.000001( per request )
free tier (~100,000 requests/month)
$~0.201 /GB (Data transfer out)

18. fluent-plugin-sqs
fluentd (buffered) output plugin
store fluent-event to Amazon SQS
install
$ gem install fluent-plugin-sqs
configuration
<match sqs.**>
type sqs
aws_key_id {AWS_KEY_ID}
aws_sec_key {AWS_SEC_KEY}
queue_name fluent-queue
sqs_endpoint sqs.ap-northeast-1.amazonaws.com
# Other buffer configurations....
</match>

19. fluent-plugin-sns

20. Amazon SNS
Amazon Simple Notification Service (SNS)
multi-protocol
e-mail,email(json),http(s),SMS,SQS(!!)
Installation not required
push-base architecture
high availability
very inexpensive

21. Amazon SNS
Manager,Publisher,Subscriber Model
manager makes “topic” and
set subscribers to “topic(s)”
Publisher sends message to SNS topic,
then SNS pushes message to subscribers

22. Amazon SNS
Subscriber
Publisher e-mail
SNS
fluentd SMS
Subscriber
Topic A
send
HTTP Subscriber
Topic B
Manager
Subscriber

23. Amazon SNS
Publisher(->fluentd) is not Manager.
Should be written many subscriber
information in “fluent.conf” ? ---- No.

24. fluent-plugin-sns
fluentd output plugin
send fluent-event to Amazon SQS
install
$ gem install fluent-plugin-sns
configuration
<match sns.**>
type sns
aws_key_id {AWS_KEY_ID}
aws_sec_key {AWS_SEC_KEY}
sns_topic_name = fluentd-topic
sqs_endpoint sqs.ap-northeast-1.amazonaws.com
sns_subject_key title
</match>

25. next features...?

26. next feature...?
fluent-plugin-couch
“update document”
e.g.) To handle events such as updating the status of the server.
can refer to previous documents.
∵ CouchDB is MVCC based , automatic versioning.
fluent-plugin-sqs
input-plugin
fluent-plugin-sns
customize formatting & multi formatting

27. thank you!