A comparism of various Data Serialization Frameworks and their Java APIs

1. Data Serialization Frameworks for Java and the IoT
Manfred Dreese, codecentric AG
1

2. AGENDA
2
Shortcomings and no-frills
What Data-Serialization Frameworks do
Architectural considerations
API, Speed and Size Battle
Back to binary encoded messages?

3. Why size matters
(again)
3

4. Standard Interface
Definitions languages
or hand-crafted binary
Protocols
4
History of binary serialization frameworks
80s
String-encoded Interfaces,
String encoded verification
String-encoded Protocols
90s
Renaissance of binary
protocols as backend-
turbocharger.
Just with more frills.
2000
Binary Serialization to
Turbocharge embedded
Devices
2010

5. Standard Interface
Definitions languages
or hand-crafted binary
Protocols
5
History of binary serialization frameworks
80s
String-encoded Interfaces,
String encoded verification
String-encoded Protocols
90s
Renaissance of binary
protocols as backend-
turbocharger.
Just with more frills.
2000
Binary Serialization to
Turbocharge embedded
Devices
2010>
ASN-1
Thrift
Protobuf
JSON
Avro
CBOR
lots_of
{custom_protocol}

6. Message length considerations for IoT appliances
6
Cloud Application IoT Solution
Powerful, scalable Servers Finite number of constrained devices
High amount of influence on technology stack Mixed technology stacks, processor architectures
and languages
High connections speeds Serial links, Industrial bus connections, Cellular
networks.
No restrictions on data transfer Low bandwidth and duty cycle regulations
Nodes available all the time Power Saving, Mobile Data Saving, possible bad
signal level
Data transfer is (almost) free of charge (Literally) expensive data transfer when mobile data
is involved.
Processing power for data transfer is neglectible Sophisticated String processing or data transfer in
resource competition with actual functionality

7. What a serialization
Framework does
7

8. Code generator
8
How data serialization frameworks work
Schema-driven Frameworks
Interface Definition
Java code
Golang code
Javascript code
{any} code

9. IDL/Schema Definition
Code generation
Message Composition and Serialization
9
Example
1
2
3
Message Transfer
4
Message Deserialization
5

10. IDL/Schema Definition
Code generation
Message Composition and Serialization
10
Example
syntax = "proto3";
import "Common.proto";
import "google/protobuf/timestamp.proto";
package jcon.telemetry;
option java_package = "de.m9d.telemetry.engine";
option go_package = "m9d.de/telemetry/engine“;
message Telegram {
string engineId = 1;
google.protobuf.Timestamp timeFrom = 2;
google.protobuf.Timestamp timeTo = 3;
double totalWork = 4;
Curve engineSpeed = 5;
Curve temperature = 6;
Curve oilPressure = 7;
Curve fuelConsumption = 8;
}
1
2
3
Define interface using a meta-language.
Message Transfer
4
Message Deserialization
5

11. 11
Example
<plugin>
<groupId>com.github.os72</groupId>
<artifactId>protoc-jar-maven-plugin</artifactId>
<version>3.5.0</version>
<executions>
<execution>
<phase>generate-sources</phase>
<goals>
<goal>run</goal>
</goals>
<configuration>
<protocVersion>3.5.0</protocVersion>
<includeStdTypes>true</includeStdTypes>
<includeDirectories>
</includeDirectories>
<inputDirectories>
<include>../idl</include>
</inputDirectories>
</configuration>
</execution>
</executions>
</plugin>
<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java</artifactId>
<version>3.5.1</version>
</dependency>
IDL/Schema Definition
Code generation
Message Composition and Serialization
1
2
3
Use code-generator (CLI / build plugin)
to build producer/consumer code for your platform
Message Transfer
4
Message Deserialization
5 protoc
-I=../idl
—cpp_out=api/
—wrap=*.proto

12. 12
Example
Common.Curve.Builder sampleCurve = Common.Curve.newBuilder();
for (int x = 0 ; x < 32 ; x++) {
sampleCurve.addPoints(
Common.Point.newBuilder()
.setX(x)
.setY(x)
.build());
}
EngineTelemetry.Telegram message= EngineTelemetry.Telegram
.newBuilder()
.setTimeFrom(Timestamp.newBuilder()
.setSeconds(fromTimestamp)
.setTimeTo(Timestamp.newBuilder()
.setSeconds(toTimestamp)
.setTotalWork(42.23)
.setEngineId(engineUid.toString())
.setFuelConsumption(sampleCurve)
.setOilPressure(sampleCurve)
.build();
byte[] proto = message.toByteArray();
IDL/Schema Definition
Code generation
Message Composition and Serialization
1
2
3
Message Transfer
4
Message Deserialization
5
Use generated code or generic API to compose a message.

13. 13
0 |24 30 37 37 64 33 39 36 64 2D 38 37 31 32 2D 34 |$077d396 d-8712-4
10 |61 62 34 2D 62 62 34 31 2D 35 62 66 65 62 31 65 |ab4-bb41 -5bfeb1e
20 |32 33 34 66 63 12 6 8 D6 DF E9 DD 5 1A 6 8 |234fc
30 |C6 C3 E9 DD 5 21 3D A D7 A3 70 1D 45 40 3A F6 | != p E@:
40 | 2 A 0 A A D 0 0 80 3F 15 0 0 80 3F A | ? ?
50 | A D 0 0 0 40 15 0 0 0 40 A A D 0 0 | @ @
60 |40 40 15 0 0 40 40 A A D 0 0 80 40 15 0 |@@ @@ @
70 | 0 80 40 A A D 0 0 A0 40 15 0 0 A0 40 A | @ @ @
80 | A D 0 0 C0 40 15 0 0 C0 40 A A D 0 0 | @ @
90 |E0 40 15 0 0 E0 40 A A D 0 0 0 41 15 0 | @ @ A
A0 | 0 0 41 A A D 0 0 10 41 15 0 0 10 41 A | A A A
B0 | A D 0 0 20 41 15 0 0 20 41 A A D 0 0 | A A
C0 |30 41 15 0 0 30 41 A A D 0 0 40 41 15 0 |0A 0A @A
D0 | 0 40 41 A A D 0 0 50 41 15 0 0 50 41 A | @A PA PA
E0 | A D 0 0 60 41 15 0 0 60 41 A A D 0 0 | `A `A

14. 14
0 |22 65 6E 67 69 6E 65 49 64 22 3A 20 22 30 37 37 |"engineI d": "077
10 |64 33 39 36 64 2D 38 37 31 32 2D 34 61 62 34 2D |d396d-87 12-4ab4-
20 |62 62 34 31 2D 35 62 66 65 62 31 65 32 33 34 66 |bb41-5bf eb1e234f
30 |63 22 2C 22 74 69 6D 65 46 72 6F 6D 22 3A 20 7B |c","time From": {
40 |22 73 65 63 6F 6E 64 73 22 3A 20 31 35 33 38 39 |"seconds ": 15389
50 |34 34 39 38 32 7D 2C 22 74 69 6D 65 54 6F 22 3A |44982}," timeTo":
60 |20 7B 22 73 65 63 6F 6E 64 73 22 3A 20 31 35 33 | {"secon ds": 153
70 |38 39 34 31 33 38 32 7D 2C 22 74 6F 74 61 6C 57 |8941382} ,"totalW
80 |6F 72 6B 22 3A 20 34 32 2E 32 33 2C 22 6F 69 6C |ork": 42 .23,"oil
90 |50 72 65 73 73 75 72 65 22 3A 20 7B 22 70 6F 69 |Pressure ": {"poi
A0 |6E 74 73 22 3A 20 5B 7B 7D 2C 7B 22 78 22 3A 20 |nts": [{ },{"x":
B0 |31 2E 30 2C 22 79 22 3A 20 31 2E 30 7D 2C 7B 22 |1.0,"y": 1.0},{"
C0 |78 22 3A 20 32 2E 30 2C 22 79 22 3A 20 32 2E 30 |x": 2.0, "y": 2.0
D0 |7D 2C 7B 22 78 22 3A 20 33 2E 30 2C 22 79 22 3A |},{"x": 3.0,"y":
E0 |20 33 2E 30 7D 2C 7B 22 78 22 3A 20 34 2E 30 2C | 3.0},{" x": 4.0,

15. 15
Example
EngineTelemetry.Telegram deserialized =
EngineTelemetry.Telegram.parseFrom(wire);
IDL/Schema Definition
Code generation
Message Composition and Serialization
1
2
3
Message Transfer
4
Message Deserialization
5 The same way back

16. Generated Builder
API
Builder API or types
serializers/deserializers
based on Interface
definition
Generic Builder
API
Field-By-Field API
Abstracted-away
Workhorse
Serving as a Serializer
Option for i.E. Jackson
16
API types

17. 17
..it is not
A uint32 is a uint32…
Message with low int values telegram.
.setMaximumTorque(32)
.setTotalWork(32)
.build();
telegram.
.setMaximumTorque(65535)
.setTotalWork(65535)
.build();
2 8 20 21 0 0 0 0 0 0 40 40
(13 bytes)
4 8 FF FF 3 21 0 0 0 0 E0 FF EF 40
(15 bytes)
Message with high int values

18. 18
40 | 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 61 | a
50 |34 30 35 66 31 65 32 2D 32 63 62 61 2D 34 37 39 |405f1e2- 2cba-479
60 |32 2D 39 37 66 38 2D 37 36 38 36 66 39 30 34 63 |2-97f8-7 686f904c
70 |63 31 64 0 0 0 0 1 0 0 0 7 1 0 0 80 |c1d
80 | 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 |
90 | 0 80 3F 0 0 80 3F 0 0 0 40 0 0 0 40 0 | ? ? @ @
A0 | 0 40 40 0 0 40 40 0 0 80 40 0 0 80 40 0 | @@ @@ @ @
B0 | 0 A0 40 0 0 A0 40 0 0 C0 40 0 0 C0 40 0 | @ @ @ @
C0 | 0 E0 40 0 0 E0 40 0 0 0 41 0 0 0 41 0 | @ @ A A
D0 | 0 10 41 0 0 10 41 0 0 20 41 0 0 20 41 0 | A A A A
Random Access
With fixed length fields
-Predictable offset of any item
-Decode just a subset of the message
-Just point reading routines to raw data, no need to allocate additional memory

19. Shortcomings
19

20. 20
No error correction
Encode message
Meet the bad guys
EngineTelemetry.Telegram telemetry= EngineTelemetry.Telegram.newBuilder()
.setTimeFrom(Timestamp.now())
.setTotalWork(65535)
.build();
wire[10] = 0x40;
Decoding works fine! EngineTelemetry.Telegram deserialized =
EngineTelemetry.Telegram.parseFrom(wire);
deserialized:
{
„timeFrom": {"seconds": 65535},
"totalWork": 65535.0078125
}
Well, fine enough for another
Diesel joke…
4 8 FF FF 3 21 0 0 0 0 E0 FF EF 40
4 8 FF FF 3 21 0 0 0 0 40 FF EF 40

21. 21
40 | 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 61 | a
50 |34 30 35 66 31 65 32 2D 32 63 62 61 2D 34 37 39 |405f1e2- 2cba-479
60 |32 2D 39 37 66 38 2D 37 36 38 36 66 39 30 34 63 |2-97f8-7 686f904c
70 |63 31 64 0 0 0 0 1 0 0 0 7 1 0 0 80 |c1d
80 | 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 |
90 | 0 80 3F 0 0 80 3F 0 0 0 40 0 0 0 40 0 | ? ? @ @
A0 | 0 40 40 0 0 40 40 0 0 80 40 0 0 80 40 0 | @@ @@ @ @
B0 | 0 A0 40 0 0 A0 40 0 0 C0 40 0 0 C0 40 0 | @ @ @ @
C0 | 0 E0 40 0 0 E0 40 0 0 0 41 0 0 0 41 0 | @ @ A A
D0 | 0 10 41 0 0 10 41 0 0 20 41 0 0 20 41 0 | A A A A
// CAFE, BABE? could be anything!
No frills, no type announcement
-Type awareness has to be established on decoration or transport

22. 22
Non-opiniated with regards to transport
-Just an array of bytes
-No encryption or compression by default
-No transport or remote method methods by default
-Except Add-Ons (gRPC, AVRO RPC)
-Integrateable in existing transport infrastructure
-Raw sockets
-MQTT, AMQP, Kafka, …
-Machine bus protocols
-XMPP
-Local memory

23. Serialization Frameworks
and
Microservices
23

24. 24
Use-case: Binary encoded content
Message Broker
(MQTT/AMQP/…)
IoT Device Consumers
Submit

25. 25
Use-case: API generation
Code generator
Service A
API description
V1.23.42
Service C
API code
Service B
API code
Checkout
Generate
Generate

26. 26
Use-case: Backend Payload
Backend Service B Backend Service CBackend Service A
Message Broker
Pub/Sub
Pub/Sub
Pub/Sub

27. 27
Best Usecases
-High message sizes and frequencies
-Structure data
-Lists, Arrays, Time Series
-Cross-Platform and Languages

28. Java APIs comparism
(and BATTLE!)
28

29. Serialize a engine telemetry data item
as fast as possible
With as little data transfer as possible
Only no-brainer optimizations are
allowed
Given situation
29
A fictional telemetry use-case
EngineID Uuid
(String encoding if no native type)
Observation Interval Unix timestamp from, to
Total amount of work Double
Fuel consumption curve 32 float tupels (0,0)..(32,32)
Oil Pressure curve 32 float tupels (0,0)..(32,32)
Serialization in Java

30. JSON
30
0 |22 65 6E 67 69 6E 65 49 64 22 3A 20 22 30 37 37 |"engineI d": "077
10 |64 33 39 36 64 2D 38 37 31 32 2D 34 61 62 34 2D |d396d-87 12-4ab4-
20 |62 62 34 31 2D 35 62 66 65 62 31 65 32 33 34 66 |bb41-5bf eb1e234f
30 |63 22 2C 22 74 69 6D 65 46 72 6F 6D 22 3A 20 7B |c","time From": {
40 |22 73 65 63 6F 6E 64 73 22 3A 20 31 35 33 38 39 |"seconds ": 15389
50 |34 34 39 38 32 7D 2C 22 74 69 6D 65 54 6F 22 3A |44982}," timeTo":
60 |20 7B 22 73 65 63 6F 6E 64 73 22 3A 20 31 35 33 | {"secon ds": 153
70 |38 39 34 31 33 38 32 7D 2C 22 74 6F 74 61 6C 57 |8941382} ,"totalW
80 |6F 72 6B 22 3A 20 34 32 2E 32 33 2C 22 6F 69 6C |ork": 42 .23,"oil
90 |50 72 65 73 73 75 72 65 22 3A 20 7B 22 70 6F 69 |Pressure ": {"poi
A0 |6E 74 73 22 3A 20 5B 7B 7D 2C 7B 22 78 22 3A 20 |nts": [{ },{"x":
B0 |31 2E 30 2C 22 79 22 3A 20 31 2E 30 7D 2C 7B 22 |1.0,"y": 1.0},{"
C0 |78 22 3A 20 32 2E 30 2C 22 79 22 3A 20 32 2E 30 |x": 2.0, "y": 2.0
D0 |7D 2C 7B 22 78 22 3A 20 33 2E 30 2C 22 79 22 3A |},{"x": 3.0,"y":
E0 |20 33 2E 30 7D 2C 7B 22 78 22 3A 20 34 2E 30 2C | 3.0},{" x": 4.0,
F0 |22 79 22 3A 20 34 2E 30 7D 2C 7B 22 78 22 3A 20 |"y": 4.0 },{"x":
100 |35 2E 30 2C 22 79 22 3A 20 35 2E 30 7D 2C 7B 22 |5.0,"y": 5.0},{"
110 |78 22 3A 20 36 2E 30 2C 22 79 22 3A 20 36 2E 30 |x": 6.0, "y": 6.0
120 |7D 2C 7B 22 78 22 3A 20 37 2E 30 2C 22 79 22 3A |},{"x": 7.0,"y":
130 |20 37 2E 30 7D 2C 7B 22 78 22 3A 20 38 2E 30 2C | 7.0},{" x": 8.0,
140 |22 79 22 3A 20 38 2E 30 7D 2C 7B 22 78 22 3A 20 |"y": 8.0 },{"x":
150 |39 2E 30 2C 22 79 22 3A 20 39 2E 30 7D 2C 7B 22 |9.0,"y": 9.0},{"
160 |78 22 3A 20 31 30 2E 30 2C 22 79 22 3A 20 31 30 |x": 10.0 ,"y": 10
170 |2E 30 7D 2C 7B 22 78 22 3A 20 31 31 2E 30 2C 22 |.0},{"x" : 11.0,"
180 |79 22 3A 20 31 31 2E 30 7D 2C 7B 22 78 22 3A 20 |y": 11.0 },{"x":
190 |31 32 2E 30 2C 22 79 22 3A 20 31 32 2E 30 7D 2C |12.0,"y" : 12.0},
1A0 |7B 22 78 22 3A 20 31 33 2E 30 2C 22 79 22 3A 20 |{"x": 13 .0,"y":
1B0 |31 33 2E 30 7D 2C 7B 22 78 22 3A 20 31 34 2E 30 |13.0},{" x": 14.0
1C0 |2C 22 79 22 3A 20 31 34 2E 30 7D 2C 7B 22 78 22 |,"y": 14 .0},{"x"
1D0 |3A 20 31 35 2E 30 2C 22 79 22 3A 20 31 35 2E 30 |: 15.0," y": 15.0
1E0 |7D 2C 7B 22 78 22 3A 20 31 36 2E 30 2C 22 79 22 |},{"x": 16.0,"y"
1F0 |3A 20 31 36 2E 30 7D 2C 7B 22 78 22 3A 20 31 37 |: 16.0}, {"x": 17
200 |2E 30 2C 22 79 22 3A 20 31 37 2E 30 7D 2C 7B 22 |.0,"y": 17.0},{"
210 |78 22 3A 20 31 38 2E 30 2C 22 79 22 3A 20 31 38 |x": 18.0 ,"y": 18
220 |2E 30 7D 2C 7B 22 78 22 3A 20 31 39 2E 30 2C 22 |.0},{"x" : 19.0,"
230 |79 22 3A 20 31 39 2E 30 7D 2C 7B 22 78 22 3A 20 |y": 19.0 },{"x":
240 |32 30 2E 30 2C 22 79 22 3A 20 32 30 2E 30 7D 2C |20.0,"y" : 20.0},
250 |7B 22 78 22 3A 20 32 31 2E 30 2C 22 79 22 3A 20 |{"x": 21 .0,"y":
260 |32 31 2E 30 7D 2C 7B 22 78 22 3A 20 32 32 2E 30 |21.0},{" x": 22.0
270 |2C 22 79 22 3A 20 32 32 2E 30 7D 2C 7B 22 78 22 |,"y": 22 .0},{"x"
// (cut)
60000ns 1537 bytes

31. 31
Google Protocol Buffers
syntax = "proto3";
import "Common.proto";
import "google/protobuf/timestamp.proto";
package jcon.telemetry;
option java_package = "de.m9d.telemetry.engine";
option go_package = "de.m9d/telemetry/engine";
message Telegram {
string engineId = 1;
google.protobuf.Timestamp timeFrom = 2;
google.protobuf.Timestamp timeTo = 3;
double totalWork = 4;
Curve engineSpeed = 5;
Curve temperature = 6;
Curve oilPressure = 7;
Curve fuelConsumption = 8;
}
syntax = "proto3";
package jcon.telemetry;
option java_package = "de.m9d.telemetry.engine";
option go_package = "de.m9d/telemetry/engine";
message Point {
float x = 1;
float y = 2;
}
message Curve {
repeated Point points = 1;
}

32. 32
public void serializeProtobuf() {
Common.Curve.Builder sampleCurve = Common.Curve.newBuilder();
for (int x = 0 ; x < 32 ; x++) {
sampleCurve.addPoints(Common.Point.newBuilder().setX(x).setY(x).build());
}
EngineTelemetry.Telegram message= EngineTelemetry.Telegram.newBuilder()
.setTimeFrom(Timestamp.newBuilder().setSeconds(Instant.now().getEpochSecond()).build())
.setTimeTo(Timestamp.newBuilder().setSeconds(Instant.now().getEpochSecond()-3600).build())
.setTotalWork(42.23)
.setEngineId(UUID.randomUUID().toString())
.setFuelConsumption(sampleCurve)
.setOilPressure(sampleCurve)
.build();
byte[] proto = message.toByteArray();
}
Google Protocol Buffers

33. Google Protocol Buffers
33
0 |24 30 37 37 64 33 39 36 64 2D 38 37 31 32 2D 34 |$077d396 d-8712-4
10 |61 62 34 2D 62 62 34 31 2D 35 62 66 65 62 31 65 |ab4-bb41 -5bfeb1e
20 |32 33 34 66 63 12 6 8 D6 DF E9 DD 5 1A 6 8 |234fc
30 |C6 C3 E9 DD 5 21 3D A D7 A3 70 1D 45 40 3A F6 | != p E@:
40 | 2 A 0 A A D 0 0 80 3F 15 0 0 80 3F A | ? ?
50 | A D 0 0 0 40 15 0 0 0 40 A A D 0 0 | @ @
60 |40 40 15 0 0 40 40 A A D 0 0 80 40 15 0 |@@ @@ @
70 | 0 80 40 A A D 0 0 A0 40 15 0 0 A0 40 A | @ @ @
80 | A D 0 0 C0 40 15 0 0 C0 40 A A D 0 0 | @ @
90 |E0 40 15 0 0 E0 40 A A D 0 0 0 41 15 0 | @ @ A
A0 | 0 0 41 A A D 0 0 10 41 15 0 0 10 41 A | A A A
B0 | A D 0 0 20 41 15 0 0 20 41 A A D 0 0 | A A
C0 |30 41 15 0 0 30 41 A A D 0 0 40 41 15 0 |0A 0A @A
D0 | 0 40 41 A A D 0 0 50 41 15 0 0 50 41 A | @A PA PA
E0 | A D 0 0 60 41 15 0 0 60 41 A A D 0 0 | `A `A
F0 |70 41 15 0 0 70 41 A A D 0 0 80 41 15 0 |pA pA A
100 | 0 80 41 A A D 0 0 88 41 15 0 0 88 41 A | A A A
110 | A D 0 0 90 41 15 0 0 90 41 A A D 0 0 | A A
120 |98 41 15 0 0 98 41 A A D 0 0 A0 41 15 0 | A A A
130 | 0 A0 41 A A D 0 0 A8 41 15 0 0 A8 41 A | A A A
140 | A D 0 0 B0 41 15 0 0 B0 41 A A D 0 0 | A A
150 |B8 41 15 0 0 B8 41 A A D 0 0 C0 41 15 0 | A A A
160 | 0 C0 41 A A D 0 0 C8 41 15 0 0 C8 41 A | A A A
170 | A D 0 0 D0 41 15 0 0 D0 41 A A D 0 0 | A A
180 |D8 41 15 0 0 D8 41 A A D 0 0 E0 41 15 0 | A A A
190 | 0 E0 41 A A D 0 0 E8 41 15 0 0 E8 41 A | A A A
1A0 | A D 0 0 F0 41 15 0 0 F0 41 A A D 0 0 | A A
1B0 |F8 41 15 0 0 F8 41 42 F6 2 A 0 A A D 0 | A AB
1C0 | 0 80 3F 15 0 0 80 3F A A D 0 0 0 40 15 | ? ? @
1D0 | 0 0 0 40 A A D 0 0 40 40 15 0 0 40 40 | @ @@ @@
1E0 | A A D 0 0 80 40 15 0 0 80 40 A A D 0 | @ @
1F0 | 0 A0 40 15 0 0 A0 40 A A D 0 0 C0 40 15 | @ @ @
200 | 0 0 C0 40 A A D 0 0 E0 40 15 0 0 E0 40 | @ @ @
210 | A A D 0 0 0 41 15 0 0 0 41 A A D 0 | A A
220 | 0 10 41 15 0 0 10 41 A A D 0 0 20 41 15 | A A A
230 | 0 0 20 41 A A D 0 0 30 41 15 0 0 30 41 | A 0A 0A
240 | A A D 0 0 40 41 15 0 0 40 41 A A D 0 | @A @A
250 | 0 50 41 15 0 0 50 41 A A D 0 0 60 41 15 | PA PA `A
260 | 0 0 60 41 A A D 0 0 70 41 15 0 0 70 41 | `A pA pA
270 | A A D 0 0 80 41 15 0 0 80 41 A A D 0 | A A
280 | 0 88 41 15 0 0 88 41 A A D 0 0 90 41 15 | A A A
290 | 0 0 90 41 A A D 0 0 98 41 15 0 0 98 41 | A A A
// (cut)
2700ns 817 bytes

34. 34
Captain Proto
@0xcfe6e6668c89c78f;
using Java = import "/java.capnp";
$Java.package("de.m9d.telemetry.engine");
$Java.outerClassname("Telemetry");
struct Telegram
{
code @0 :Text;
engineId @1 :Text;
timeFrom @2 :UInt32;
timeTo @3 :UInt32;
totalWork @4 :Float64;
engineSpeed @5: Curve;
temperature @6: Curve;
oilPressure @7: Curve;
fuelConsumption @8: Curve;
}
struct Curve
{
points @0: List(Point);
}
struct Point
{
x @0: Float32;
y @1: Float32;
}

35. 35
Captain Proto
public void serializeCaptainProto() throws IOException {
String engineId = UUID.randomUUID().toString();
org.capnproto.MessageBuilder message =
new org.capnproto.MessageBuilder();
de.m9d.telemetry.engine.Telemetry.Telegram.Builder builder = message.initRoot(Telemetry.Telegram.factory);
builder.setEngineId(engineId);
builder.setTimeFrom((int) Instant.now().getEpochSecond());
builder.setTimeTo((int) Instant.now().getEpochSecond());
builder.setTotalWork((float)42.561);
builder.initEngineSpeed();
de.m9d.telemetry.engine.Telemetry.Curve.Builder engineSpeedBuilder = builder.getEngineSpeed();
engineSpeedBuilder.initPoints(32);
for (int idx = 0 ; idx < 32 ; idx ++) {
engineSpeedBuilder.getPoints().get(idx).setX(idx);
engineSpeedBuilder.getPoints().get(idx).setY(idx);
}
// Same for oil pressure
ByteArrayOutputStream bos = new ByteArrayOutputStream();
WritableByteChannel wbc = Channels.newChannel(bos);
org.capnproto.Serialize.write(wbc, message);
byte[] proto = bos.toByteArray();
}

36. Captain Proto
36
0 | 0 0 0 52 0 0 0 0 0 0 0 2 0 6 0 DC | R
10 |BD BB 5B DC BD BB 5B 77 3E 2A 42 0 0 0 0 0 | [ [w >*B
20 | 0 0 0 0 0 0 0 11 0 0 0 2A 1 0 0 20 | *
30 | 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 A0 |
40 | 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 61 | a
50 |34 30 35 66 31 65 32 2D 32 63 62 61 2D 34 37 39 |405f1e2- 2cba-479
60 |32 2D 39 37 66 38 2D 37 36 38 36 66 39 30 34 63 |2-97f8-7 686f904c
70 |63 31 64 0 0 0 0 1 0 0 0 7 1 0 0 80 |c1d
80 | 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 |
90 | 0 80 3F 0 0 80 3F 0 0 0 40 0 0 0 40 0 | ? ? @ @
A0 | 0 40 40 0 0 40 40 0 0 80 40 0 0 80 40 0 | @@ @@ @ @
B0 | 0 A0 40 0 0 A0 40 0 0 C0 40 0 0 C0 40 0 | @ @ @ @
C0 | 0 E0 40 0 0 E0 40 0 0 0 41 0 0 0 41 0 | @ @ A A
D0 | 0 10 41 0 0 10 41 0 0 20 41 0 0 20 41 0 | A A A A
E0 | 0 30 41 0 0 30 41 0 0 40 41 0 0 40 41 0 | 0A 0A @A @A
F0 | 0 50 41 0 0 50 41 0 0 60 41 0 0 60 41 0 | PA PA `A `A
100 | 0 70 41 0 0 70 41 0 0 80 41 0 0 80 41 0 | pA pA A A
110 | 0 88 41 0 0 88 41 0 0 90 41 0 0 90 41 0 | A A A A
120 | 0 98 41 0 0 98 41 0 0 A0 41 0 0 A0 41 0 | A A A A
130 | 0 A8 41 0 0 A8 41 0 0 B0 41 0 0 B0 41 0 | A A A A
140 | 0 B8 41 0 0 B8 41 0 0 C0 41 0 0 C0 41 0 | A A A A
150 | 0 C8 41 0 0 C8 41 0 0 D0 41 0 0 D0 41 0 | A A A A
160 | 0 D8 41 0 0 D8 41 0 0 E0 41 0 0 E0 41 0 | A A A A
170 | 0 E8 41 0 0 E8 41 0 0 F0 41 0 0 F0 41 0 | A A A A
180 | 0 F8 41 0 0 F8 41 1 0 0 0 7 1 0 0 80 | A A
190 | 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 |
1A0 | 0 80 3F 0 0 80 3F 0 0 0 40 0 0 0 40 0 | ? ? @ @
1B0 | 0 40 40 0 0 40 40 0 0 80 40 0 0 80 40 0 | @@ @@ @ @
1C0 | 0 A0 40 0 0 A0 40 0 0 C0 40 0 0 C0 40 0 | @ @ @ @
1D0 | 0 E0 40 0 0 E0 40 0 0 0 41 0 0 0 41 0 | @ @ A A
1E0 | 0 10 41 0 0 10 41 0 0 20 41 0 0 20 41 0 | A A A A
1F0 | 0 30 41 0 0 30 41 0 0 40 41 0 0 40 41 0 | 0A 0A @A @A
200 | 0 50 41 0 0 50 41 0 0 60 41 0 0 60 41 0 | PA PA `A `A
210 | 0 70 41 0 0 70 41 0 0 80 41 0 0 80 41 0 | pA pA A A
220 | 0 88 41 0 0 88 41 0 0 90 41 0 0 90 41 0 | A A A A
230 | 0 98 41 0 0 98 41 0 0 A0 41 0 0 A0 41 0 | A A A A
240 | 0 A8 41 0 0 A8 41 0 0 B0 41 0 0 B0 41 0 | A A A A
250 | 0 B8 41 0 0 B8 41 0 0 C0 41 0 0 C0 41 0 | A A A A
260 | 0 C8 41 0 0 C8 41 0 0 D0 41 0 0 D0 41 0 | A A A A
270 | 0 D8 41 0 0 D8 41 0 0 E0 41 0 0 E0 41 0 | A A A A
280 | 0 E8 41 0 0 E8 41 0 0 F0 41 0 0 F0 41 0 | A A A A
290 | 0 F8 41 0 0 F8 41
5200ns 664 bytes

37. 37
AVRO
[
{
"namespace": "jcon.telemetryserver.domain",
"type": "record",
"name": "Curve",
"fields": [
{"name": "points", "type": {"type": "array", "items": {
"name":"Point",
"type":"record",
"fields":[
{"name":"x", "type":"float"},
{"name":"y", "type":"float"}
]
}}}
]
},
{
"namespace": "jcon.telemetryserver.domain",
"type": "record",
"name": "Telegram",
"fields": [
{"name": "engineId", "type": "string"},
{"name": "timeFrom", "type": {"type": "long", "logicalType": "timestamp-millis"}},
{"name": "timeTo", "type": {"type": "long", "logicalType": "timestamp-millis"}},
{"name": "totalWork", "type": "double"},
{"name": "engineSpeed", "type": ["null","jcon.telemetryserver.domain.Curve"], "default": null },
{"name": "temperature", "type": ["null","jcon.telemetryserver.domain.Curve"], "default": null },
{"name": "oilPressure", "type": ["null","jcon.telemetryserver.domain.Curve"], "default": null },
{"name": "fuelConsumption", "type": ["null","jcon.telemetryserver.domain.Curve"], "default": null }
]
}
]

38. 38
AVRO
public void serializeWithAvroAndSchemas() throws IOException {
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
DatumWriter<Telegram> writer = new SpecificDatumWriter<>(Telegram.getClassSchema());
BinaryEncoder encoder = EncoderFactory.get().binaryEncoder(outputStream, null);
List<Point> points = new ArrayList<>();
for (float x = 0 ; x < 32 ; x++) {
points.add(Point.newBuilder().setX(x).setY(x).build());
}
Curve curve = Curve.newBuilder()
.setPoints(points)
.build();
Telegram telegram = Telegram.newBuilder()
.setEngineId(UUID.randomUUID().toString())
.setTimeFrom(DateTime.now())
.setTimeTo(DateTime.now())
.setTotalWork(42.23)
.setEngineSpeed(curve)
.setFuelConsumption(curve)
.build();
writer.write(telegram, encoder);
encoder.flush();
byte[] wire = outputStream.toByteArray();
}

39. AVRO
39
0 |64 63 62 33 33 39 33 65 2D 35 34 63 61 2D 34 66 |dcb3393e -54ca-4f
10 |39 39 2D 38 31 35 64 2D 35 62 36 64 35 31 33 35 |99-815d- 5b6d5135
20 |37 30 31 34 84 F3 A9 85 CA 59 B0 F3 A9 85 CA 59 |7014 Y Y
30 |3D A D7 A3 70 1D 45 40 2 40 0 0 0 0 0 0 |= p E@ @
40 | 0 0 0 0 80 3F 0 0 80 3F 0 0 0 40 0 0 | ? ? @
50 | 0 40 0 0 40 40 0 0 40 40 0 0 80 40 0 0 | @ @@ @@ @
60 |80 40 0 0 A0 40 0 0 A0 40 0 0 C0 40 0 0 | @ @ @ @
70 |C0 40 0 0 E0 40 0 0 E0 40 0 0 0 41 0 0 | @ @ @ A
80 | 0 41 0 0 10 41 0 0 10 41 0 0 20 41 0 0 | A A A A
90 |20 41 0 0 30 41 0 0 30 41 0 0 40 41 0 0 | A 0A 0A @A
A0 |40 41 0 0 50 41 0 0 50 41 0 0 60 41 0 0 |@A PA PA `A
B0 |60 41 0 0 70 41 0 0 70 41 0 0 80 41 0 0 |`A pA pA A
C0 |80 41 0 0 88 41 0 0 88 41 0 0 90 41 0 0 | A A A A
D0 |90 41 0 0 98 41 0 0 98 41 0 0 A0 41 0 0 | A A A A
E0 |A0 41 0 0 A8 41 0 0 A8 41 0 0 B0 41 0 0 | A A A A
F0 |B0 41 0 0 B8 41 0 0 B8 41 0 0 C0 41 0 0 | A A A A
100 |C0 41 0 0 C8 41 0 0 C8 41 0 0 D0 41 0 0 | A A A A
110 |D0 41 0 0 D8 41 0 0 D8 41 0 0 E0 41 0 0 | A A A A
120 |E0 41 0 0 E8 41 0 0 E8 41 0 0 F0 41 0 0 | A A A A
130 |F0 41 0 0 F8 41 0 0 F8 41 0 0 0 2 40 0 | A A A @
140 | 0 0 0 0 0 0 0 0 0 80 3F 0 0 80 3F 0 | ? ?
150 | 0 0 40 0 0 0 40 0 0 40 40 0 0 40 40 0 | @ @ @@ @@
160 | 0 80 40 0 0 80 40 0 0 A0 40 0 0 A0 40 0 | @ @ @ @
170 | 0 C0 40 0 0 C0 40 0 0 E0 40 0 0 E0 40 0 | @ @ @ @
180 | 0 0 41 0 0 0 41 0 0 10 41 0 0 10 41 0 | A A A A
190 | 0 20 41 0 0 20 41 0 0 30 41 0 0 30 41 0 | A A 0A 0A
1A0 | 0 40 41 0 0 40 41 0 0 50 41 0 0 50 41 0 | @A @A PA PA
1B0 | 0 60 41 0 0 60 41 0 0 70 41 0 0 70 41 0 | `A `A pA pA
1C0 | 0 80 41 0 0 80 41 0 0 88 41 0 0 88 41 0 | A A A A
1D0 | 0 90 41 0 0 90 41 0 0 98 41 0 0 98 41 0 | A A A A
1E0 | 0 A0 41 0 0 A0 41 0 0 A8 41 0 0 A8 41 0 | A A A A
1F0 | 0 B0 41 0 0 B0 41 0 0 B8 41 0 0 B8 41 0 | A A A A
200 | 0 C0 41 0 0 C0 41 0 0 C8 41 0 0 C8 41 0 | A A A A
210 | 0 D0 41 0 0 D0 41 0 0 D8 41 0 0 D8 41 0 | A A A A
220 | 0 E0 41 0 0 E0 41 0 0 E8 41 0 0 E8 41 0 | A A A A
230 | 0 F0 41 0 0 F0 41 0 0 F8 41 0 0 F8 41 0 | A A A A
240 |
5400ns 577 bytes

40. 40
CBOR
public void serializeCbor() throws IOException, CborException {
ByteArrayOutputStream baos = new ByteArrayOutputStream();
CborBuilder di = new CborBuilder();
ArrayBuilder ab = di.addArray();
for (float x = 0 ; x < 32 ; x++) {
ab.addArray()
.add(x)
.add(x)
.end();
}
List<DataItem> curves = di.build();
new CborEncoder(baos).encode(new CborBuilder()
.add(UUID.randomUUID().toString())
.add(Instant.now().getEpochSecond())
.add(Instant.now().getEpochSecond())
.add(42.23)
.add(curves.get(0))
.add(curves.get(0))
.build());
byte[] encodedBytes = baos.toByteArray();
}

41. CBOR
41
0 |24 35 39 38 32 31 33 63 36 2D 64 65 37 37 2D 34 |$598213c 6-de77-4
10 |33 63 63 2D 62 64 38 61 2D 30 31 37 30 30 64 35 |3cc-bd8a -01700d5
20 |30 62 62 35 39 1A 5B BB C4 CC 1A 5B BB C4 CC FB |0bb59 [ [
30 |40 45 1D 70 A3 D7 A 3D 98 20 82 F9 0 0 F9 0 |@E p =
40 | 0 82 F9 3C 0 F9 3C 0 82 F9 40 0 F9 40 0 82 | < < @ @
50 |F9 42 0 F9 42 0 82 F9 44 0 F9 44 0 82 F9 45 | B B D D E
60 | 0 F9 45 0 82 F9 46 0 F9 46 0 82 F9 47 0 F9 | E F F G
70 |47 0 82 F9 48 0 F9 48 0 82 F9 48 80 F9 48 80 |G H H H H
80 |82 F9 49 0 F9 49 0 82 F9 49 80 F9 49 80 82 F9 | I I I I
90 |4A 0 F9 4A 0 82 F9 4A 80 F9 4A 80 82 F9 4B 0 |J J J J K
A0 |F9 4B 0 82 F9 4B 80 F9 4B 80 82 F9 4C 0 F9 4C | K K K L L
B0 | 0 82 F9 4C 40 F9 4C 40 82 F9 4C 80 F9 4C 80 82 | L@ L@ L L
C0 |F9 4C C0 F9 4C C0 82 F9 4D 0 F9 4D 0 82 F9 4D | L L M M M
D0 |40 F9 4D 40 82 F9 4D 80 F9 4D 80 82 F9 4D C0 F9 |@ M@ M M M
E0 |4D C0 82 F9 4E 0 F9 4E 0 82 F9 4E 40 F9 4E 40 |M N N N@ N@
F0 |82 F9 4E 80 F9 4E 80 82 F9 4E C0 F9 4E C0 82 F9 | N N N N
100 |4F 0 F9 4F 0 82 F9 4F 40 F9 4F 40 82 F9 4F 80 |O O O @ O@ O
110 |F9 4F 80 82 F9 4F C0 F9 4F C0 98 20 82 F9 0 0 | O O O
120 |F9 0 0 82 F9 3C 0 F9 3C 0 82 F9 40 0 F9 40 | < < @ @
130 | 0 82 F9 42 0 F9 42 0 82 F9 44 0 F9 44 0 82 | B B D D
140 |F9 45 0 F9 45 0 82 F9 46 0 F9 46 0 82 F9 47 | E E F F G
150 | 0 F9 47 0 82 F9 48 0 F9 48 0 82 F9 48 80 F9 | G H H H
160 |48 80 82 F9 49 0 F9 49 0 82 F9 49 80 F9 49 80 |H I I I I
170 |82 F9 4A 0 F9 4A 0 82 F9 4A 80 F9 4A 80 82 F9 | J J J J
180 |4B 0 F9 4B 0 82 F9 4B 80 F9 4B 80 82 F9 4C 0 |K K K K L
190 |F9 4C 0 82 F9 4C 40 F9 4C 40 82 F9 4C 80 F9 4C | L L@ L@ L L
1A0 |80 82 F9 4C C0 F9 4C C0 82 F9 4D 0 F9 4D 0 82 | L L M M
1B0 |F9 4D 40 F9 4D 40 82 F9 4D 80 F9 4D 80 82 F9 4D | M@ M@ M M M
1C0 |C0 F9 4D C0 82 F9 4E 0 F9 4E 0 82 F9 4E 40 F9 | M N N N@
1D0 |4E 40 82 F9 4E 80 F9 4E 80 82 F9 4E C0 F9 4E C0 |N@ N N N N
1E0 |82 F9 4F 0 F9 4F 0 82 F9 4F 40 F9 4F 40 82 F9 | O O O@ O@
1F0 |4F 80 F9 4F 80 82 F9 4F C0 F9 4F C0
14000ns 509 bytes

42. Battle Results
42
Message size Time needed Cross-
Platform
JSON 1537 60000nsYes
Java Serialization
(not cross-language)
1022 5000nsNo
Protocol Buffers 817 2100nsYes
Captain Proto 664 5200nsYes
CBOR 509 14000nsYes
AVRO 577 5700nsYes

43. 43
Thank you!

44. 44