Presentation slides of Airframe Meetup #3 https://airframe.connpass.com/event/148169/ - Airframe 19 Milestone - AirSpec: A new testing library for Scala -

1. Taro L. Saito, Ph.D.
Arm Treasure Data
October 23nd, 2019
Arm Treasure Data Tokyo Office
Airframe Meetup #3
1

2. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
About Me: Taro L. Saito (Leo)
2
● Principal Software Engineer at Arm
Treasure Data
● Building distributed query engine services
● Living in US for 4 years
● DBMS & Data Science Background
● Ph.D. of Computer Science
● OSS Projects around Scala
● sbt-sonatype: used for releasing 3000+
Scala projects
● snappy-java: a compression library used
in Spark, Parquet, etc.
● etc.

3. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
Recent Work: Designing Data-Intensive Applications
● Techniques and concepts around distributed
data processing systems
3

4. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
Today’s Agenda
● Airframe Updates
● The current version: 19.10.1
■ 25+ releases since July 2019~
● AirSpec: a new testing framework for Scala
● Lifecycle design
● airframe-http updates
■ Route filters
● Releasing OSS with GitHub Actions
● sbt-sonatype: Blazing Fast Release to Sonatype
● 3 Lightning Talks
4
Airframe

5. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
Airframe: Collection of Our Best Practices
● Gather the best practices of Scala into Airframe OSS
● Get the real experiences by operating 24/7 services
5
Knowledge
Experiences
Design Decisions
Products
24/7 Services
Business Values
Programming OSS Outcome
Airframe

6. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
AirSpec: A New Testing Framework for Scala and Scala.js
● Simple Usage
● import wvlet.airspec._
● Extend AirSpec
● Define test cases as public functions
● Low Learning Cost
● No annotation (like JUnit5) is
necessary.
● Simple assertions: assert(cond), x
shouldBe y, etc.
■ No need to learn other complex
DSLs.
6

7. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
AirSpec: Assertions
7

8. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
Example: AirSpec Assertions
8

9. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
AirSpec: Scala.js
● ScalaTest RefSpec
● Has similar syntax with AirSpec, but it doesn’t support Scala.js because of lack of
runtime reflection.
● AirSpec uses airframe-surface to collect test methods
■ Scala.js: Collect test methods (MethodSurface) at compile-time
■ Scala JVM: Uses reflection at runtime
9

10. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
Writing Specs in Natural Languages
10

11. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
AirSpec: Motivation
● Q: Why do you need a new testing framework?
● A: For maintaining Airframe
● Because of (relatively) slow ScalaTest releases, we couldn’t catch up with the
latest versions of Scala and Scala.js.
■ We usually needed to wait for several weeks or years (!) to test new Scala
versions.
■ ScalaTest still doesn’t support Scala.js 1.0.x
● A: Taking a balance between minimalist approaches and too complex DSLs
● minimalist: minitest, scala-verify
■ Simple, but using them is not so fun (e.g., no support for test filtering by
name)
● complex DSLs: ScalaTest, Specs2, uTest
■ High-learning cost
■ Need an agreement on the assertion style within the team
11

12. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
AirSpec: Motivation
● Another answer: It was surprisingly simple to implement a testing framework
12

13. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
AirSpec Features
● Production Ready
● All Airframe test cases are now using AirSpec
● sbt integration
● Handy keyword search for test methods
■ > testOnly -- (a pattern for class or method names)
● Property-based testing integrated with ScalaCheck
● Scala 2.11, 2.12, 2.13, and Scala.js (0.6.x and 1.0.x) support
● Integration with Airframe
● Automatic DI session management
● handy logging with airframe-log
13

14. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
AirSpec: Logging with airframe-log
14

15. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
AirSpec + DI
● Injecting dependency as test function
arguments
● Override AirSpec.design to
customize your designs
● Managing lifecycle of test
dependencies
● The lifecycle (e.g., start and
shutdown) of the injected services
can be managed by Airframe DI
● Internally nested Airframe sessions
will be created for individual test
cases
● Nest-free resource management
● no loan pattern
● no monad
15
Lifecycle Design (New Feature!)

16. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
Pro Tips: AirSpec + Airframe DI
● Tips
● Use objects to define
component designs
● Add objects for test modules
and override existing
bindings (if necessary)
● Use the designs inside
AirSpec
● Why?
● Your design becomes
reusable for both
production/test code
● Airframe DI uses lazy binding
by default, so having
unnecessary binding is OK
■ If necessary, use
coverage report option
to minimize the design
16

17. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
DI Coverage Stats
17

18. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
Property-Based Testing with ScalaCheck
18

19. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
Airframe DI Update: Lifecycle Design
● AutoCloseable (parent of Closeable interface) support
● If an injected instance implements AutoCloseable interface (def close()), it will be
automatically closed upon Session.shutdown
● Support defining lifecycle hooks within a design
● Useful for reusing services with a proper lifecycle management
● e.g., http client/server, database connection pool, etc.
● Testing environments can be prepared by adding necessary designs:
■ design + service1Design + service2Design + ...
19

20. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
Example: Test Design
20

21. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
airframe-http: Using Scala As An IDL
● IDL: Interface Definition Language
● e.g. ProtocolBuffers, gRPC
● Use Scala’s Standard Functionalities
● case classes
■ Define HTTP request/response
data structure
● functions
■ Define HTTP endpoints
■ Use @Endpoint annotation for
for URL to method mapping
● Router
● Router.add[X]
● filter andThen router (Added in
19.8.x)
21

22. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
Path Mapping
● :param
● Mapping to a function argument
● /user/:id
■ /user/1
■ def getUser(id:Int)
○ getUser(1)
● *param
● Mapping url path tail to a function
argument
● /resource/*path
■ /resource/my/key/path
■ def getPath(path:String)
○ getPath(my/key/path)
● case classes
● request/response body
● JSON <-> MessagePack <-> Object
22

23. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
New: airframe-http: Request Filter
● Router
● .add[X]
● .andThen[X]
● Filter
● apply(request, context)
■ returns Future[Response]
23

24. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
airframe-http-client
● Read server responses as target object types
● Supports both JSON and MessagePack
● Added requestFilter argument to manipulate HTTP headers
24

25. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
Distributed Tracing with DataDog APM
● Setup
● Just run JVM applications with -javaagent:dd-agent.jar
● Implicit logging
● No explicit instrumentation is necessary
■ ByteBuddy is used to inject instrumentation code
● Major libraries are already supported
■ https://github.com/DataDog/dd-trace-java/tree/master/dd-java-agent/instru
mentation
■ e.g., jetty, netty4, akka, okhttp, aws-sdk, servlet, spring, JDBC connections,
etc.
● Cost: $31/instance/month, $1.2 / 1M tracing events
25

26. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
DataDog APM: Request Summary
26

27. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
Airframe/AirSpec 19 Milestones
● Airframe 19 Milestone
● https://github.com/wvlet/airframe/issues/528
■ airframe-json:
○ Stream JSON -> MessagePack converter (done)
■ airframe-codec: Scala.js support
■ airframe-control: Adding circuit breaker
■ airframe-http: production ready
○ We have migrated several internal APIs into airframe-http. Advanced features
like request logging, CORS support, admin interface, JMX metrics are
internally available. Need to extract them.
■ airframe-http-client: Jetty, akka-http support
● AirSpec 19 Milestone
● https://github.com/wvlet/airframe/issues/606
■ Standalone launcher and async testing
■ Using the return values of test functions for reporting
■ IntelliJ support (?) 27

28. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
[WIP] Scala.js RPC
● Scala.js
● Compiling Scala code into JavaScript for Web Browsers
● airframe-codec: Passing model class data between Scala and Scala.js
UserInfo MessagePack UserInfo
Pack Unpack
PackUnpack
Scala
Server Side
Scala.js
Client Side
XML RPC
28
airframe-codec airframe-codec

29. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
Releasing Airframe
● Migrated from Travis CI to GitHub Actions
● Uses .github/workflows folder
● Examples:
https://github.com/wvlet/airframe/tree/master/.github/workflows
● Good integration with the pull request UI
29

30. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
GitHub Actions
● Keywords
● workflow -> job -> steps
● workflow
■ YAML file
■ define conditions to trigger jobs (PR,
push, etc.)
● job
■ A unit of execution
○ = a single docker container
■ Multiple jobs run in parallel
■ No local folder sharing between jobs
● steps
■ Can import third-party step definitions
■ Define a sequence of command lines
30

31. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
GitHub Actions: Secrets
● Secrets (e.g., password, token) can be
defined in GitHub settings
● Secrets cannot be used in PRs nor
forked repositories
● exception: GITHUB_TOKEN
■ e.g., for updating PR tags,
comments, etc.
● Uploading test artifacts (e.g., test
coverage report) in PR to other than
GitHub is not straightforward
31

32. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
GitHub Actions: Releasing to Sonatype
● Steps
● Import GPG private key
● publishSigned
■ sbt-pgp
● Upload to Sonatype
● Tips
● Needed to Install GPG1
■ TTY env of GitHub action is
broken, so gpg2-agent cannot
pass PGP_PASSPHRASE to GPG2
■ This might be a bug of sbt-pgp
plugin
● Encode GPG private key with base64
and set it as secret.PGP_SECRET
■ decrypt with
secret.PGP_PASSPHRASE
32

33. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
● Blog post:
https://medium.com/@taroleo/sbt-sonatype-f02bdafd78f1
● Airframe needs to upload 2000+ files for each
release
● When Sonatype API is slow, it could take
20 minutes ~ 1 hour
● With bundle upload, it takes less than 1
minute
● Bundle Upload
● publishTo := target/sonatype-bundle/(version)
■ Collecting artifacts to the local folder
● Upload artifacts as a single jar
sbt-sonatype: Blazing Fast Upload to Sonatype
33

34. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
Happy reports from sbt-sonatype users
34

35. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
scala-steward
● A time saver for OSS maintainers
● Create PRs for library updates
● Became the top committer for
October 2019
35

36. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
Summary
● AirSpec
● A new testing framework with DI support
● Define reusable designs
■ lifecycle of servicers can be designed now
● Airframe
● airframe-http is production-ready with filters
● Development
● Using GitHub Actions for CI and releasing Airframe
● Bundle upload with a new sbt-sonatype plugin
36

37. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
Join Arm Treasure Data Team!
● Solving Challenges In The Real World Data Processing
● Building a scalable data processing platform on the cloud
● Advanced data analytics with Presto, Hive, Spark, and workflows
■ Processing 1 million queries / day
● Providing distributed computing as a service for 450+ companies
37
Airframe

38. Confidential © Arm 2017Confidential © Arm 2017Confidential © Arm 2017
Thank You!
Danke!
Merci!
谢谢!
ありがとう!
Gracias!
Kiitos!
38

39. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
Module Mix-InPackaging
HTTP Requests and
Responses
Data
airframe-launcher
> _
airframe-log
production:
port: 10010
user: xxxx
...
airframe-config
airframe-codec
sbt-pack
airframe-fluentd
Scala
Objects
Table Data
(CSV, TSV)
JSON
airframe-jsonairframe-surface
airframe-tablet
airframe-jmx
Monitor Runtime States
Generate Mapping Codec
Metrics &
Log Data
JDBC
ResultSets
airframe-jdbc
airframe-http
airframe-http-finagle
Launch HTTP
Services
airframe DI
Debug Logs
Schema-On-Read
Mapping
Airframe

40. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
The Architecture of Arm Treasure Data
40
DataLogs
Device
Data
Batch
Data
PlazmaDB
Table Schema
Data Collection Cloud Storage Distributed Data Processing
2 million records / sec. 130 trillion records 1 billion rows processed / sec.
Jobs
Job Management
SQL Editor
Scheduler
Workflows
Machine
Learning
Treasure Data OSS
Third Party OSS