Streaming data analytics : background, technologies and outlook

Abstract

Companies seek significant potential from the datasets they generate within their core processes. However, due to the variety and volatile nature of the data analytic landscapes, in terms of analytic models, programming environment, and software libraries, companies view analytics-related investment too risky. The aim of this dissertation was to gather empirical evidence on the differences in maturity level of the presented data analytical tools.

In this dissertation, we map the state of the art in Big Data and data analytic tools, following a top-down method using semi-structured interviews with companies. These interviews are analyzed with the help of NvivoTM and the results are compared against the state of the art.

Consequently, recommendations for principles and data analytics tool aimed at reducing the risk of investments, from company’s point of view were provided. The maturity level of the technological landscape in streaming data analytics were also highlighted. By analyzing the interview results and reflecting on previous research, among the chosen streaming data analytics tools, Spark Streaming was deemed to have slight upper hand.

Additionally, the result from this dissertation is also intended at guiding the construction of a data analytics infrastructure, aimed at encouraging industry-academia collaboration. This dissertation sheds light on how the companies are not always aware of the state-of-the-art and there is a huge gap between practicalities and research findings. A lot of work has to be done in this area so that the gap does not widen further.