- The document summarizes a presentation about ClickHouse, an open source column-oriented database management system. - It discusses how ClickHouse stores and indexes data to enable fast queries, how it scales horizontally across servers, and how different engines like MergeTree and ReplicatedMergeTree allow for high performance and fault tolerance. - Examples are provided showing how ClickHouse can quickly analyze large datasets with SQL and optimize queries using its features like distributed processing, partitioning, and specialized functions.

1. Become dangerous on ClickHouse
in 30 minutes
Robert Hodges
SF ClickHouse Meetup - 19 February 2019

2. Presenter Background
● CEO of Altinity, a ClickHouse commercial provider
● Experience in databases:
○ M204...
○ Followed by Sybase, Cloudscape, Oracle, MySQL, …, ClickHouse
● Qualifications for working in data management:
○ A sick interest in math
○ Insatiable curiosity

3. So...What is this ClickHouse?
Understands SQL
Portable from bare metal to cloud
Stores data in columns
Parallelizes query execution
Scales to many petabytes
Is open source (Apache 2.0)
Is WAY fast!
Id a b c d
Id a b c d
Id a b c d
Id a b c d

4. What does “WAY fast” mean?
SELECT Dest d, count(*) c, avg(DepDelayMinutes) dd
FROM ontime GROUP BY d HAVING c > 100000
ORDER BY c DESC limit 5
┌─d───┬───────c─┬─────────────────dd─┐
│ ATL │ 8281604 │ 10.328921909330608 │
│ ORD │ 8040484 │ 11.893720825761235 │
│ DFW │ 6928489 │ 8.954579418398442 │
│ LAX │ 5164096 │ 10.109661594207388 │
│ DEN │ 4641382 │ 9.664600112638865 │
└─────┴─────────┴────────────────────┘
5 rows in set. Elapsed: 0.938 sec. Processed
153.53 million rows, 2.46 GB (163.67 million
rows/s., 2.62 GB/s.)
TEST BED
40€/month
Intel Core
i7-6700
32GB
Seagate 4TB
7200RPM
128MB cache

5. Getting started is easy
$ docker run -d --name ch-s yandex/clickhouse-server
$ docker exec -it ch-s clickhouse client
...
11e99303c78e :) select version()
SELECT version()
┌─version()─┐
│ 19.3.3 │
└───────────┘
1 rows in set. Elapsed: 0.001 sec.

6. Let’s do some technical due diligence
CREATE TABLE sdata (
DevId Int32,
Type String,
MDate Date,
MDatetime DateTime,
Value Float64
) ENGINE = MergeTree() PARTITION BY toYYYYMM(MDate)
ORDER BY (DevId, MDatetime)
INSERT INTO sdata VALUES
(15, 'TEMP', '2018-01-01', '2018-01-01 23:29:55', 18.0),
(15, 'TEMP', '2018-01-01', '2018-01-01 23:30:56', 18.7)
INSERT INTO sdata VALUES
(15, 'TEMP', '2018-01-01', '2018-01-01 23:31:53', 18.1),
(2, 'TEMP', '2018-01-01', '2018-01-01 23:31:55', 7.9)

7. Select results can be surprising!
SELECT *
FROM sdata ┌─DevId─┬─Type─┬──────MDate─┬───────────MDatetime─┬─Value─┐
│ 15 │ TEMP │ 2018-01-01 │ 2018-01-01 23:29:55 │ 18 │
│ 15 │ TEMP │ 2018-01-01 │ 2018-01-01 23:30:56 │ 18.7 │
└───────┴──────┴────────────┴─────────────────────┴───────┘
┌─DevId─┬─Type─┬──────MDate─┬───────────MDatetime─┬─Value─┐
│ 2 │ TEMP │ 2018-01-01 │ 2018-01-01 23:31:55 │ 7.9 │
│ 15 │ TEMP │ 2018-01-01 │ 2018-01-01 23:31:53 │ 18.1 │
└───────┴──────┴────────────┴─────────────────────┴───────┘
┌─DevId─┬─Type─┬──────MDate─┬───────────MDatetime─┬─Value─┐
│ 2 │ TEMP │ 2018-01-01 │ 2018-01-01 23:31:55 │ 7.9 │
│ 15 │ TEMP │ 2018-01-01 │ 2018-01-01 23:29:55 │ 18 │
│ 15 │ TEMP │ 2018-01-01 │ 2018-01-01 23:30:56 │ 18.7 │
│ 15 │ TEMP │ 2018-01-01 │ 2018-01-01 23:31:53 │ 18.1 │
└───────┴──────┴────────────┴─────────────────────┴───────┘
Result right after INSERT:
Result somewhat later:

8. Time for some research into table engines
CREATE TABLE sdata (
DevId Int32,
Type String,
MDate Date,
MDatetime DateTime,
Value Float64
) ENGINE = MergeTree()
PARTITION BY toYYYYMM(MDate)
ORDER BY (DevId, MDatetime)
How to manage data
and handle queries
How to break table
into parts
How to index and
sort data in each part

9. MergeTree writes parts quickly and merges them offline
/var/lib/clickhouse/data/default/sdata
201801_1_1_0/
201801_2_2_0/
Multiple parts after initial
insertion ( => very fast writes)
201801_1_2_1/
Single part after merge
( => very fast reads)

10. Rows are indexed and sorted inside each part
/var/lib/clickhouse/data/default/sdata
... ...
956 2018-01-01 15:22:37
575 2018-01-01 23:31:53
1300 2018-01-02 05:14:47
... ...
primary.idx
||||
.mrk .bin
||||
.mrk .bin
||||
.mrk .bin
||||
.mrk .bin
201802_1_1_0/
(DevId, MDateTime) DevId Type MDate MDatetime...
primary.idx .mrk .bin .mrk .bin .mrk .bin .mrk .bin
201801_1_2_1/
(DevId, MDateTime) DevId Type MDate MDatetime...

11. ClickHouse
Now we can follow how query works on a single server
SELECT DevId, Type, avg(Value)
FROM sdata
WHERE MDate = '2018-01-01'
GROUP BY DevId, Type
Identify parts to search
Query in parallel
Aggregate results
Result Set

12. What if you have a lot of data?
CREATE TABLE
sense.sdata_dist AS sense.sdata
ENGINE = Distributed(scluster, sense, sdata, DevId)
Engine for distributed
tables
Cluster, database,
and table names
Distribute data
on this value
Copy this
schema

13. What’s a Cluster? (Hint: it’s a file.)
<yandex>
<remote_servers>
<scluster>
<shard>
<internal_replication>true</internal_replication>
<replica>
<host>ch-d7dc980i1-0.d7dc980i1s</host>
<port>9000</port>
</replica>
</shard>
<shard>
<internal_replication>true</internal_replication>
<replica>
<host>ch-d7dc980i2-0.d7dc980i2s</host>
<port>9000</port>
</replica>
</shard>
...
</scluster>
</remote_servers>
</yandex>

14. Distributed engine spreads queries across shards
SELECT ...
FROM
sdata_dist
ClickHouse
sdata_dist
(Distributed)
sdata
(MergeTable)
ClickHouse
sdata_dist sdata
ClickHouse
sdata_dist sdata
Result Set

15. What if you have a lot of data and you don’t want to lose it?
CREATE TABLE sense.sdata (
DevId Int32, Type String,
MDate Date,
MDatetime DateTime,
Value Float64
) engine=ReplicatedMergeTree(
'/clickhouse/{installation}/{scluster}/tables/{scluster-
shard}/sense/sdata',
'{replica}', MDate, (DevId, MDatetime), 8192);
Engine for replicated
tables
MergeTree parameters
Cluster identification
Replica #

16. Now we distribute across shards and replicas
ClickHouse
sdata_dist
sdata
ReplicatedMergeTree
Engine
ClickHouse
sdata_dist
sdata
ClickHouse
sdata_dist
sdata
ClickHouse
sdata_dist
sdata
ClickHouse
sdata_dist
sdata
ClickHouse
sdata_dist
sdata
SELECT ...
FROM
sdata_dist
Result Set
Zookeeper
Zookeeper
Zookeeper

17. SELECT Dest, count(*) c, avg(DepDelayMinutes)
FROM ontime
GROUP BY Dest HAVING c > 100000
ORDER BY c DESC limit 5
SELECT Dest, count(*) c, avg(DepDelayMinutes)
FROM ontime
WHERE toYear(FlightDate) =
toYear(toDate('2016-01-01'))
GROUP BY Dest HAVING c > 100000
ORDER BY c DESC limit 5
With basic engine knowledge you can now tune queries
Scans 355 table parts
in parallel; does not
use index
Scans 12 parts (3%
of data) because
FlightDate is
partition key
Hint: clickhouse-server.log has the query plan
Faster

18. SELECT
Dest d, Name n, count(*) c, avg(ArrDelayMinutes)
FROM ontime
JOIN airports ON (airports.IATA = ontime.Dest)
GROUP BY d, n HAVING c > 100000 ORDER BY ad DESC
SELECT dest, Name n, c AS flights, ad FROM (
SELECT Dest dest, count(*) c, avg(ArrDelayMinutes) ad
FROM ontime
GROUP BY dest HAVING c > 100000
ORDER BY ad DESC
) LEFT JOIN airports ON airports.IATA = dest
You can also optimize joins
Subquery
minimizes data
scanned in
parallel; joins on
GROUP BY results
Joins on data
before GROUP BY,
increased amount
to scan
Faster

19. ClickHouse has a wealth of features to help queries go fast
Dictionaries
Materialized Views
Arrays
Specialized functions and SQL
extensions
Lots more table engines

20. Parting Question: What’s ClickHouse good for?
● Fast, scalable data warehouse for online services (SaaS
and in-house apps
● Built-in data warehouse for installed analytic applications
● Exploration -- throw in a bunch of data and go crazy!

21. Thank you!
Comments to:
rhodges at altinity.com
Visit us at:
https://www.altinity.com