Analytics with Storm and Redis with dashboard visualisation

1. RT Analytics
with Data
Streaming
Felix Crisan / Co-Founder & CTO NETOPIA
[mobilPay, web2sms, kartela.ro]

2. About Us
- Largest SMS Aggregator in RO
(+100MM/year)
- Largest mobile payments processor in RO
(own wallet and SMS)
- Second largest online card processing
gateway in RO (+2M trx / year)

3. RT Analytics @
mobilPay

4. Is it BigData?

5. What do we use it for
- Operational Insights
- Transaction Monitoring
- Health Checks
- Business Dashboards (internal for now, GA
later this year)

6. Storm one-pager
- Streaming processor, Apache Incubator
(http://storm.incubator.apache.org/)
- Zookeeper/Nimbus(+ZeroMQ/Netty)
- Scalable / No Data Loss (guaranteed)
- Fault - Tolerant
- Topologies (~ MR job) / Streams / Spouts
(data sources) / Bolts (data processors)

7. Redis one-pager
- In-Memory (disk persistent) KV Store (http:
//redis.io)
- Values can be complex types (strings,
hashes, lists, sets, sorted sets)
- Single-threaded (!) but high-performance
(500k req/s)
- Atomic operations

8. (Simplified) Data Model

9. Our Data Streaming setup

10. What Happens
- Transaction details are put in a RabbitMQ
queue (state, errors, values etc)
- A Storm Topology reads the details and
aggregates
- per Merchant / Service
- per Status / Error / Currency
- per time interval (5min, H, D, M)

11. Why did we choose this setup?

12. So what’s the
result?

13. Know Your Queries

14. Better insights

15. Clear(er) views

16. Better predictions

17. KYC

18. Summary

19. Thank you!
felix@mobilpay.com