Making Real Time Data Analytics Available as a Service

Select |




Print


Xu, Donna; Wu, Dongyao; Xu, Xiwei (Sherry); Zhu, Liming; Bass, Len

Xu, Donna; Wu, Dongyao; Xu, Xiwei (Sherry); Zhu, Liming; Bass, Len


2015-05-05


Conference Material


International ACM Sigsoft Conference on the Quality of Software Architectures


Montreal, Canada


73-82


Conducting (big) data analytics in an organization is not just about using a processing framework (e.g. Hadoop/Spark) to learn a model from data currently in a single file system (e.g. HDFS). We frequently need to pipeline real time data from other systems into the processing framework, and continually update the learned model. The processing frameworks need to be easily invokable for different purposes to produce different models. The model and the subsequent model updates need to be integrated with a product that may require a real time prediction using the latest trained model. All these need to be shared among different teams in the organization for different data analytics purposes. In this paper, we propose a real time data-analytics-as-service architecture that uses RESTful web services to wrap and integrate data services, dynamic model training services (supported by big data processing framework), prediction services and the product that uses the models. We discuss the challenges in wrapping big data processing frameworks as services and other architecturally significant factors that affect system reliability, real time performance and prediction accuracy. We evaluate our architecture using a log-driven system operation anomaly detection system where staleness of data used in model training, speed of model update and prediction are critical requirements.


Big Data, Real Time, Spark, Data Analytics


https://doi.org/10.1145/2737182.2737186


http://qosa.ipd.kit.edu/qosa_2015/


nicta:8636


Xu, Donna; Wu, Dongyao; Xu, Xiwei (Sherry); Zhu, Liming; Bass, Len. Making Real Time Data Analytics Available as a Service. In: International ACM Sigsoft Conference on the Quality of Software Architectures; Montreal, Canada. 2015-05-05. 73-82. https://doi.org/10.1145/2737182.2737186



Loading citation data...

Citation counts
(Requires subscription to view)