Aleksei Udatšnõi – Crunching thousands of events per second in nearly real time
Imagine you have a product which generates up to 10 thousands events per second or around 1 billion events per day. This live stream of data need to be tracked, processed and presented to end-users in a visually appealing way. The solution needs to be integrated into a traditional web application. That is the real use case at Softonic. In this talk we will show how it was solved in Softonic. We use the stack of technologies around Big Data to process and store live stream of data and present the results to users in nearly real time. This real-life solution is built around Hadoop ecosystem and it includes Flume, Hive, Oozie and Impala. We will show how to store and query such volumes of data using NoSQL database and how to build a scalable end-user web application using nearly real time data feed.