Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.
Netty
from the trenches
June 2015
@jordi9
Netty…?
Netty is an
asynchronous
event-driven
network
application
framework
for rapid
development of
maintainable high
performance
protocol servers &
clients.
Netty is a NIO
client server
framework which
enables quick and
easy development
of network
applications such
as protocol servers
and clients. It
greatly simplifies
and streamlines
network
network
programming such
as TCP and UDP
socket server.
OH, MY
The Freaking
Basics
You’ve heard of:
Async apps
You’ve heard of:
Event driven
frameworks
You’ve heard of:
non-blocking
operations
You’ve heard of:
Node is
cooler
because...
I/O
I/O is
everywhere
I/O, approx:
CPU
Memory
Device
read()
read()
//wait...
read()
//wait...
keepWorkingBro()
Our program
BLOCKS
Resources
WASTED
You know what
I’m talking
about
Webserver
Request
Database
//wait...
OH,
Parallelism!
new Thread();
Webserver
Request
Database
//wait...
Webserver
Request
Database
//wait...
Request
Database
//wait...
Webserver
Request
Database
//wait...
Request
Database
//wait...
Request
Database
//wait.
A mess
We can
do better
read()
read()
keepWorkingBro()
read()
keepWorkingBro()
//I’m
//done!
You (kinda) did this:
Listeners
You (kinda) did this:
Guava’s
EventBus
You (kinda) did this:
Javascript
callbacks
Hollywood principle
“Don’t call us,
we’ll call you”
Keep working,
instead of
waiting
Hi,
Async I/O
NIO
with Java
java.nio
since 1.4, 2002
java.nio
since 1.7: NIO2
more goodies
Hi,
Complexity
Netty
to the Rescue
Built on top of:
java.nio
Built on top of:
java.net
Dependecy-haters:
Just
JDK 1.6+
ONE
big difference
All Netty APIs
are async
It’s can be one
more tool!
What the hell
is Netty for?
Think...
HTTP
everywhere
Really
BRO?
Maybe, you’re
trying to
implement a
protocol
Finally, hello
Netty!
Netty:
Tool to
implement network
applications
Netty:
Both from server
and client side
Netty:
NIO
everywhere
Why
Netty?
Netty has been
designed carefully
with the
experiences
earned from the
implementation of
a lot of protocols
such as FTP,
SMTP, HTTP, and
various binary and
text-based legacy
protocols. As a
protocols. As a
result, Netty has
succeeded to find
a way to achieve...
Ease of
development
Performance
Stability
Flexibility
Without a
compromise
IT’S TRUE
“Implementing
a protocol”
NOT THAT HARD
Most common
protocols:
out-of-the-box
Boring Protocols:
HTTP
Boring Protocols:
SSL
Boring Protocols:
UDP
Cooler Protocols:
SPDY
Cooler Protocols:
HTTP/2
Cooler Protocols:
Protobuf
Even Protocols:
Memcache
Not only I/O
Powerful
Thread model
But, where’s
the downside?
Downers:
Constant API
changes*
Downers:
Constant API
changes*
*For the best
Downers:
Learning
curve...
Downers:
Learning
curve...
Buy Netty in Action :P
Downers:
Join the
mailing list
Downers:
Join the
mailing list
You’ll see how
STUPID you are
Downers:
Join the
mailing list
But you’ll learn ALOT ;)
Netty,
The Code
Daytime
Protocol
The
Server
class SimpleDaytimeHandler extends
ChannelInboundHandlerAdapter {
}
The Hand e
class SimpleDaytimeHandler extends
ChannelInboundHandlerAdapter {
// Handler = Business Logic
}
The Hand e
class SimpleDaytimeHandler extends
ChannelInboundHandlerAdapter {
// ChannelHandler: handles operations
// for that Channe...
class SimpleDaytimeHandler extends
ChannelInboundHandlerAdapter {
// Inbound handler: incoming traffic,
// dispatch events...
class SimpleDaytimeHandler extends
ChannelInboundHandlerAdapter {
// Outbound handler: same, but the
// other direction. Y...
class SimpleDaytimeHandler extends
ChannelInboundHandlerAdapter {
// Inbound + Outbound...
}
The Hand e
class SimpleDaytimeHandler extends
ChannelInboundHandlerAdapter {
// Inbound + Outbound..
// 5.0: DEPRECATED!
// [NOTE: In...
Hand e me hod
@Override
public void channelRead(
ChannelHandlerContext ctx, Object msg) {
// Will trigger when we receive
// some data
}...
@Override
public void channelActive(
ChannelHandlerContext ctx, Object msg) {
// Will a Channel is opened, this
// method ...
@Override
public void channelActive(ChannelHandlerContext ctx, Object msg) {
String date = DATE_TIME.print(new DateTime())...
@Override
public void channelActive(ChannelHandlerContext ctx, Object msg) {
String date = DATE_TIME.print(new DateTime())...
It’s clear,
right? :D
Network
standard way of
working?
byte[]
You said this
was fun?
Meet
ByteBuf
ByteBufUtil
@Override
public void channelActive(ChannelHandlerContext ctx, Object msg) {
String date = DATE_TIME.print(new DateTime())...
@Override
public void channelActive(ChannelHandlerContext ctx, Object msg) {
String date = DATE_TIME.print(new DateTime())...
@Override
public void channelActive(ChannelHandlerContext ctx, Object msg) {
String date = DATE_TIME.print(new DateTime())...
We have our
Handler in place
Let’s Bootstrap
the server
Ma n ass
public class DaytimeServer {
void run() throws Exception {
// fun stuff
}
public static void main(String[] args) ...
java -jar daytimeserver-fat.jar
un()
EventLoopGroup bossGroup = new NioEventLoopGroup();
EventLoopGroup workerGroup = new NioEventLoopGroup();
E en LoopG oup
Netty’s way to
handle threads
E en LoopG oup
EventLoopGroup
contains some
EventLoops
E en Loop
EventLoop
handles many
Channels
About to die
BRO?
E en LoopG oup
E en Loop
hanne
hanne
hanne
hanne
hanne hanne
hanne
hanne
hanne
hanne
hanne hanne
E en Loop
E en LoopG oup
E en Loop
hanne
hanne
hanne
hanne
hanne hanne
hanne
hanne
hanne
hanne
hanne hanne
E en Loop
This immutable
assignment is
the key
un()
EventLoopGroup bossGroup = new NioEventLoopGroup();
EventLoopGroup workerGroup = new NioEventLoopGroup();
un()
EventLoopGroup bossGroup = new NioEventLoopGroup();
EventLoopGroup workerGroup = new NioEventLoopGroup();
// Boss gro...
Se e Boo s ap
ServerBootstrap b = new ServerBootstrap();
b.group(bossGroup, workerGroup)
.channel(NioServerSocketChannel.c...
Se e Boo s ap
ServerBootstrap b = new ServerBootstrap();
b.group(bossGroup, workerGroup)
.channel(NioServerSocketChannel.c...
Se e Boo s ap
ServerBootstrap b = new ServerBootstrap();
b.group(bossGroup, workerGroup)
.channel(NioServerSocketChannel.c...
Se e Boo s ap
ServerBootstrap b = new ServerBootstrap();
b.group(bossGroup, workerGroup)
.channel(NioServerSocketChannel.c...
Se e Boo s ap
ServerBootstrap b = new ServerBootstrap();
b.group(bossGroup, workerGroup)
.channel(NioServerSocketChannel.c...
We’re
almost there!
hanne P pe ne
b.childHandler(new ChannelInitializer<SocketChannel>() {
@Override
public void initChannel(SocketChannel ch)...
hanne P pe ne
b.childHandler(new ChannelInitializer<SocketChannel>() {
@Override
public void initChannel(SocketChannel ch)...
hanne P pe ne
b.childHandler(new ChannelInitializer<SocketChannel>() {
@Override
public void initChannel(SocketChannel ch)...
hanne P pe ne
b.childHandler(new ChannelInitializer<SocketChannel>() {
@Override
public void initChannel(SocketChannel ch)...
RUN!
b.childHandler(new ChannelInitializer<SocketChannel>() {
@Override
public void initChannel(SocketChannel ch)
throws E...
.I.A.
Futures
ByteBuf API
Codecs
Transports
Zero-file-copy
We can also
create
client code
Real life
Insights
We’re using Netty for
Real-Time Bidding
Boo s ap
Tons o op ons
Don’t be afraid
of “low-level”
Ta HTTP
Explore what’s
inside Netty
Integrate things
you love. In my
case: GUICE
Gu e
@Inject
DaytimeService(Provider<DaytimeServer> daytimeProvider) {}
b.childHandler(() → {
ChannelPipeline p = ch.pipel...
But… is Netty
FAST?
2ms*
* Without network latency
2ms*
Yah but… what
volume are you
talking about?
+10k QPS
1 node*
+3k QPS
*Fou 2.79GHz In e Pen um Xeon P o esso s, 7.5GB RAM
But, is this
ALOT or not?
Parse example
http://blog.parse.com/learn/how-we-moved-our-api-from-ruby-to-go-and-saved-our-sanity/
(From Ruby to Go) A y...
We have 16 nodes!
4x Netty nodes
10x Kafka, DNS, the
usual suspects…
I KNOW IT’S
NOT FAIR,
but it’s good stuff
for your ego BRO
Lovely ecosystem
like Ratpack,
or users,
with vert.x or akka
It will change the
way you program
some of your apps
Hello, Async
problems
Forget about
max onn params
One more thing...
(It’s 5:00 in the
morning and I had
to say it)
(did you notice the
“Apple
background”?)
JUST KIDDING
It doesn’ really
matter!
Have fun!
THANKS
ProTip:
We’re hiring
Q & A
#HiddenTrovitTrac@jordi9
Netty from the trenches
Netty from the trenches
Netty from the trenches
Netty from the trenches
Nächste SlideShare
Wird geladen in …5
×

Netty from the trenches

2.074 Aufrufe

Veröffentlicht am

Netty is an asynchronous event-driven network application framework for rapid development of maintainable high performance protocol servers & clients. AND IT'S TRUE!

In this talk given at JBCNConf 2015 in Barcelona, we will see how we use Netty at Trovit since 2013, what brought to us and how it opened our minds. We will share tips that helped us to learn more about Netty, some performance tricks and all things that worked for us.

Veröffentlicht in: Technologie
  • Als Erste(r) kommentieren

Netty from the trenches

  1. 1. Netty from the trenches June 2015 @jordi9
  2. 2. Netty…?
  3. 3. Netty is an asynchronous event-driven network application framework
  4. 4. for rapid development of maintainable high performance protocol servers & clients.
  5. 5. Netty is a NIO client server framework which enables quick and easy development of network
  6. 6. applications such as protocol servers and clients. It greatly simplifies and streamlines network
  7. 7. network programming such as TCP and UDP socket server.
  8. 8. OH, MY
  9. 9. The Freaking Basics
  10. 10. You’ve heard of: Async apps
  11. 11. You’ve heard of: Event driven frameworks
  12. 12. You’ve heard of: non-blocking operations
  13. 13. You’ve heard of: Node is cooler because...
  14. 14. I/O
  15. 15. I/O is everywhere
  16. 16. I/O, approx: CPU Memory Device
  17. 17. read()
  18. 18. read() //wait...
  19. 19. read() //wait... keepWorkingBro()
  20. 20. Our program BLOCKS
  21. 21. Resources WASTED
  22. 22. You know what I’m talking about
  23. 23. Webserver Request Database //wait...
  24. 24. OH, Parallelism!
  25. 25. new Thread();
  26. 26. Webserver Request Database //wait...
  27. 27. Webserver Request Database //wait... Request Database //wait...
  28. 28. Webserver Request Database //wait... Request Database //wait... Request Database //wait.
  29. 29. A mess
  30. 30. We can do better
  31. 31. read()
  32. 32. read() keepWorkingBro()
  33. 33. read() keepWorkingBro() //I’m //done!
  34. 34. You (kinda) did this: Listeners
  35. 35. You (kinda) did this: Guava’s EventBus
  36. 36. You (kinda) did this: Javascript callbacks
  37. 37. Hollywood principle “Don’t call us, we’ll call you”
  38. 38. Keep working, instead of waiting
  39. 39. Hi, Async I/O
  40. 40. NIO with Java
  41. 41. java.nio
  42. 42. since 1.4, 2002 java.nio
  43. 43. since 1.7: NIO2 more goodies
  44. 44. Hi, Complexity
  45. 45. Netty to the Rescue
  46. 46. Built on top of: java.nio
  47. 47. Built on top of: java.net
  48. 48. Dependecy-haters: Just JDK 1.6+
  49. 49. ONE big difference
  50. 50. All Netty APIs are async
  51. 51. It’s can be one more tool!
  52. 52. What the hell is Netty for?
  53. 53. Think...
  54. 54. HTTP everywhere
  55. 55. Really BRO?
  56. 56. Maybe, you’re trying to implement a protocol
  57. 57. Finally, hello Netty!
  58. 58. Netty: Tool to implement network applications
  59. 59. Netty: Both from server and client side
  60. 60. Netty: NIO everywhere
  61. 61. Why Netty?
  62. 62. Netty has been designed carefully with the experiences earned from the implementation of
  63. 63. a lot of protocols such as FTP, SMTP, HTTP, and various binary and text-based legacy protocols. As a
  64. 64. protocols. As a result, Netty has succeeded to find a way to achieve...
  65. 65. Ease of development
  66. 66. Performance
  67. 67. Stability
  68. 68. Flexibility
  69. 69. Without a compromise
  70. 70. IT’S TRUE
  71. 71. “Implementing a protocol” NOT THAT HARD
  72. 72. Most common protocols: out-of-the-box
  73. 73. Boring Protocols: HTTP
  74. 74. Boring Protocols: SSL
  75. 75. Boring Protocols: UDP
  76. 76. Cooler Protocols: SPDY
  77. 77. Cooler Protocols: HTTP/2
  78. 78. Cooler Protocols: Protobuf
  79. 79. Even Protocols: Memcache
  80. 80. Not only I/O
  81. 81. Powerful Thread model
  82. 82. But, where’s the downside?
  83. 83. Downers: Constant API changes*
  84. 84. Downers: Constant API changes* *For the best
  85. 85. Downers: Learning curve...
  86. 86. Downers: Learning curve... Buy Netty in Action :P
  87. 87. Downers: Join the mailing list
  88. 88. Downers: Join the mailing list You’ll see how STUPID you are
  89. 89. Downers: Join the mailing list But you’ll learn ALOT ;)
  90. 90. Netty, The Code
  91. 91. Daytime Protocol
  92. 92. The Server
  93. 93. class SimpleDaytimeHandler extends ChannelInboundHandlerAdapter { } The Hand e
  94. 94. class SimpleDaytimeHandler extends ChannelInboundHandlerAdapter { // Handler = Business Logic } The Hand e
  95. 95. class SimpleDaytimeHandler extends ChannelInboundHandlerAdapter { // ChannelHandler: handles operations // for that Channel, duh } The Hand e
  96. 96. class SimpleDaytimeHandler extends ChannelInboundHandlerAdapter { // Inbound handler: incoming traffic, // dispatch events to next handler // If we have inbound... } The Hand e
  97. 97. class SimpleDaytimeHandler extends ChannelInboundHandlerAdapter { // Outbound handler: same, but the // other direction. Yeah, there’s some // flow between Handlers (eg: Pipeline) } The Hand e
  98. 98. class SimpleDaytimeHandler extends ChannelInboundHandlerAdapter { // Inbound + Outbound... } The Hand e
  99. 99. class SimpleDaytimeHandler extends ChannelInboundHandlerAdapter { // Inbound + Outbound.. // 5.0: DEPRECATED! // [NOTE: Insert a grumpy cat here] } The Hand e
  100. 100. Hand e me hod
  101. 101. @Override public void channelRead( ChannelHandlerContext ctx, Object msg) { // Will trigger when we receive // some data } Hand e me hod
  102. 102. @Override public void channelActive( ChannelHandlerContext ctx, Object msg) { // Will a Channel is opened, this // method will be called } Hand e me hod
  103. 103. @Override public void channelActive(ChannelHandlerContext ctx, Object msg) { String date = DATE_TIME.print(new DateTime()); // Get the date } Hand e wo
  104. 104. @Override public void channelActive(ChannelHandlerContext ctx, Object msg) { String date = DATE_TIME.print(new DateTime()); ctx.writeAndFlush(ByteBufUtil.encodeString( ctx.alloc(), CharBuffer.wrap(date), CharsetUtil.US_ASCII)); } Hand e wo
  105. 105. It’s clear, right? :D
  106. 106. Network standard way of working? byte[]
  107. 107. You said this was fun?
  108. 108. Meet ByteBuf ByteBufUtil
  109. 109. @Override public void channelActive(ChannelHandlerContext ctx, Object msg) { String date = DATE_TIME.print(new DateTime()); ctx.writeAndFlush(ByteBufUtil.encodeString( ctx.alloc(), CharBuffer.wrap(date), CharsetUtil.US_ASCII)); // We need to encode the String } Hand e wo
  110. 110. @Override public void channelActive(ChannelHandlerContext ctx, Object msg) { String date = DATE_TIME.print(new DateTime()); ctx.writeAndFlush(ByteBufUtil.encodeString( ctx.alloc(), CharBuffer.wrap(date), CharsetUtil.US_ASCII)); // We allocate some space // +Netty: Keeps internal pools } Hand e wo
  111. 111. @Override public void channelActive(ChannelHandlerContext ctx, Object msg) { String date = DATE_TIME.print(new DateTime()); ctx.writeAndFlush(ByteBufUtil.encodeString( ctx.alloc(), CharBuffer.wrap(date), CharsetUtil.US_ASCII)); // Write the message // Request to actually flush the data // back to the Channel } Hand e wo
  112. 112. We have our Handler in place
  113. 113. Let’s Bootstrap the server
  114. 114. Ma n ass public class DaytimeServer { void run() throws Exception { // fun stuff } public static void main(String[] args) throws Exception { DaytimeServer daytimeServer = new DaytimeServer(); daytimeServer.run(); } }
  115. 115. java -jar daytimeserver-fat.jar
  116. 116. un() EventLoopGroup bossGroup = new NioEventLoopGroup(); EventLoopGroup workerGroup = new NioEventLoopGroup();
  117. 117. E en LoopG oup Netty’s way to handle threads
  118. 118. E en LoopG oup EventLoopGroup contains some EventLoops
  119. 119. E en Loop EventLoop handles many Channels
  120. 120. About to die BRO?
  121. 121. E en LoopG oup E en Loop hanne hanne hanne hanne hanne hanne hanne hanne hanne hanne hanne hanne E en Loop
  122. 122. E en LoopG oup E en Loop hanne hanne hanne hanne hanne hanne hanne hanne hanne hanne hanne hanne E en Loop
  123. 123. This immutable assignment is the key
  124. 124. un() EventLoopGroup bossGroup = new NioEventLoopGroup(); EventLoopGroup workerGroup = new NioEventLoopGroup();
  125. 125. un() EventLoopGroup bossGroup = new NioEventLoopGroup(); EventLoopGroup workerGroup = new NioEventLoopGroup(); // Boss group accepts connections // Work group handles the work
  126. 126. Se e Boo s ap ServerBootstrap b = new ServerBootstrap(); b.group(bossGroup, workerGroup) .channel(NioServerSocketChannel.class) .localAddress(8080) .option(ChannelOption.SO_BACKLOG, 100)
  127. 127. Se e Boo s ap ServerBootstrap b = new ServerBootstrap(); b.group(bossGroup, workerGroup) .channel(NioServerSocketChannel.class) .localAddress(8080) .option(ChannelOption.SO_BACKLOG, 100) // We assign both event loops
  128. 128. Se e Boo s ap ServerBootstrap b = new ServerBootstrap(); b.group(bossGroup, workerGroup) .channel(NioServerSocketChannel.class) .localAddress(8080) .option(ChannelOption.SO_BACKLOG, 100) // We use a ServerSocketChannel // to accept TCP/IP connections // as the RFC says
  129. 129. Se e Boo s ap ServerBootstrap b = new ServerBootstrap(); b.group(bossGroup, workerGroup) .channel(NioServerSocketChannel.class) .localAddress(8080) .option(ChannelOption.SO_BACKLOG, 100) // Simply bind the local address
  130. 130. Se e Boo s ap ServerBootstrap b = new ServerBootstrap(); b.group(bossGroup, workerGroup) .channel(NioServerSocketChannel.class) .localAddress(8080) .option(ChannelOption.SO_BACKLOG, 100) // Set some Socket options... why not? // Just remember: This is not handled // by Netty or the JVM, it’s the OS
  131. 131. We’re almost there!
  132. 132. hanne P pe ne b.childHandler(new ChannelInitializer<SocketChannel>() { @Override public void initChannel(SocketChannel ch) throws Exception { ChannelPipeline p = ch.pipeline(); p.addLast(new LoggingHandler(LogLevel.INFO)); p.addLast(new SimpleDaytimeHandler()); } });
  133. 133. hanne P pe ne b.childHandler(new ChannelInitializer<SocketChannel>() { @Override public void initChannel(SocketChannel ch) throws Exception { ChannelPipeline p = ch.pipeline(); p.addLast(new LoggingHandler(LogLevel.INFO)); p.addLast(new SimpleDaytimeHandler()); } }); // ChannelPipeline to define your // application workflow
  134. 134. hanne P pe ne b.childHandler(new ChannelInitializer<SocketChannel>() { @Override public void initChannel(SocketChannel ch) throws Exception { ChannelPipeline p = ch.pipeline(); p.addLast(new LoggingHandler(LogLevel.INFO)); p.addLast(new SimpleDaytimeHandler()); } }); // Append our handlers // ProTip: use LoggingHandler to // understand Netty
  135. 135. hanne P pe ne b.childHandler(new ChannelInitializer<SocketChannel>() { @Override public void initChannel(SocketChannel ch) throws Exception { ChannelPipeline p = ch.pipeline(); p.addLast(new LoggingHandler(LogLevel.INFO)); p.addLast(new SimpleDaytimeHandler()); } }); // Finally, we add our handler
  136. 136. RUN! b.childHandler(new ChannelInitializer<SocketChannel>() { @Override public void initChannel(SocketChannel ch) throws Exception { ChannelPipeline p = ch.pipeline(); p.addLast(new LoggingHandler(LogLevel.INFO)); p.addLast(new SimpleDaytimeHandler()); } }); ChannelFuture f = b.bind().sync(); f.channel().closeFuture().sync(); // It works!
  137. 137. .I.A. Futures ByteBuf API Codecs Transports Zero-file-copy
  138. 138. We can also create client code
  139. 139. Real life Insights
  140. 140. We’re using Netty for Real-Time Bidding
  141. 141. Boo s ap
  142. 142. Tons o op ons
  143. 143. Don’t be afraid of “low-level”
  144. 144. Ta HTTP
  145. 145. Explore what’s inside Netty
  146. 146. Integrate things you love. In my case: GUICE
  147. 147. Gu e @Inject DaytimeService(Provider<DaytimeServer> daytimeProvider) {} b.childHandler(() → { ChannelPipeline p = ch.pipeline(); p.addLast(new LoggingHandler(LogLevel.INFO)); p.addLast(daytimeProvider.get()); }); // Inject a Provider<T> and get // instances
  148. 148. But… is Netty FAST?
  149. 149. 2ms*
  150. 150. * Without network latency 2ms*
  151. 151. Yah but… what volume are you talking about?
  152. 152. +10k QPS
  153. 153. 1 node* +3k QPS *Fou 2.79GHz In e Pen um Xeon P o esso s, 7.5GB RAM
  154. 154. But, is this ALOT or not?
  155. 155. Parse example http://blog.parse.com/learn/how-we-moved-our-api-from-ruby-to-go-and-saved-our-sanity/ (From Ruby to Go) A year and a half in, at the end of 2012, we had 200 API servers running on m1. xlarge instance types with 24 unicorn workers per instance. This was to serve 3000 requests per second for 60,000 mobile apps
  156. 156. We have 16 nodes! 4x Netty nodes 10x Kafka, DNS, the usual suspects…
  157. 157. I KNOW IT’S NOT FAIR, but it’s good stuff for your ego BRO
  158. 158. Lovely ecosystem like Ratpack, or users, with vert.x or akka
  159. 159. It will change the way you program some of your apps
  160. 160. Hello, Async problems
  161. 161. Forget about max onn params
  162. 162. One more thing...
  163. 163. (It’s 5:00 in the morning and I had to say it)
  164. 164. (did you notice the “Apple background”?)
  165. 165. JUST KIDDING It doesn’ really matter!
  166. 166. Have fun!
  167. 167. THANKS
  168. 168. ProTip: We’re hiring
  169. 169. Q & A #HiddenTrovitTrac@jordi9

×