Elastic Stream Computing with Clouds

Stream computing, also known as data stream processing, has emerged as a new processing paradigm that processes incoming data streams from tremendous numbers of sensors in a real-time fashion. Data stream applications must have low latency even when the incoming data rate fluctuates wildly. This is almost impossible with a local stream computing environment because its computational resources are finite. To address this kind of problem, we have devised a method and an architecture that transfers data stream processing to a Cloud environment as required in response to the changes of the data rate in the input data stream. Since a trade-off exists between application's latency and the economic costs when using the Cloud environment, we treat it as an optimization problem that minimizes the economic cost of using the Cloud. We implemented a prototype system using Amazon EC2 and an IBM System S stream computing system to evaluate the effectiveness of our approach. Our experimental results show that our approach reduces the costs by 80% while keeping the application's response latency low.