1. BigQuery,
Fluentd,
and tagomoris
gcp ja night #28
2014/09/16
TAGOMORI Satoshi(tagomoris)

2. Satoshi Tagomori (@tagomoris)
LINE Corporation
Analytics Platform Team

3. スーパードゥラアア
アアァァァァァァア
アアアアアアアァァ
イイイイイイイイイイ
イイイイイイィィィィ
ィイイイイイイイイィ
イィイイイイィイィ

4. 極度乾燥
(しなさい)。

5. BigQuery
What all of us love :)

6. Fluentd
It is that with cute logo :)

7. Fluentd
Readable configuration
Flexible buffer system
Various input/output plugins
Very simple/easy plugin system
High performance for many uses

8. fluent-plugin-bigquery
Insert events from Fluentd into BigQuery
over streaming inserts
Table Sharding inserts

9. versions
v0.0.x ～ v0.1.x
by @tagomoris
many patches from @yugui
v0.2.x
KAIZEN platform Inc (@naoya_ito)

10. tagomoris:
I want new maintainer for fluent-plugin-bigquery,
who uses BQ actually....

11. naoya_ito:
OK, KAIZEN platform will do!

12. tagomoris:
Great! I’ll transfer my repository to KAIZEN’s account...
Can I have commit bit for Fluentd related fixes?

13. naoya_ito:
Sure!

15. ( д) ﾟ ﾟ

16. Disclosure:
I’m an employee of LINE now!

17. performance
Use Table Sharding inserts
tables table1,table2,table3,table4
Use many threads for concurrent insertion
num_threads 4
(Same with tables)

18. Features
Authentication
auth_method compute_engine
auth_method private_key
Schema
specs per fields
schema_path
fetch_schema

19. API Quota
number of records over streaming inserts
10,000 rows per sec per table
10MB per sec per table
Use Table Sharding
Max row size: 20KB
Max data size per insert: 1MB
Max rows per request: 500

20. Hobby programming &
Cloud services
For hobby programming
To setup environments is very troublesome...
Cloud services are easy to use!
Cloud service specific limitations/restrictions are funny
to play with!

21. Enjoy!