Document Type

Other

Publication Date

3-11-2005

Abstract

Main memory is a critical resource in push-based non-blocking query processing, espe- cially for queries with stateful operators. Works in the literature apply partitioned parallel processing to alleviate the stringent memory demands. However, main memory of a dis- tributed system remains limited. Thus, there is a demand for efficient main memory usage even for partitioned parallel queries. In this work, we first investigate two adaptations, namely, disk-based adaptation and distributed adaptation, that adapt operator states when memory overflow happens for complex multi-input operators. We analyze the tradeoffs regarding the factors and polices to be used when adapting operator states to overcome memory overflow. Two approaches, namely, lazy-disk and active-disk adaptations, are pro- posed to integrate the disk-based and distributed adaptations when the aggregated main memory of a distributed system is not sufficient for the query processing. Both appproaches aim to maximize the overall throughput. Extensive experiments have been conducted on a working system. These experiments reveal various aspects of partitioned parallel processing and their adaptation strategies.

DOI

WPI-CS-TR-05-04

Share

 
COinS