Etd

CloudNotes: Annotation Management in Cloud-Based Platforms

Public

Downloadable Content

open in viewer

We present an annotation management system for cloud-based platforms, which is called “CloudNotes”. CloudNotes enables the annotation management feature in the scalable Hadoop and MapRedue platforms. In CloudNotes system, every piece of data may have one or more annotations associate with it, and these annotations will be propagated when the data is being transformed through the MapReduce jobs. Such an annotation management system is important for understanding the provenance and quality of data, especially in applications that deal with integration of scientific and biological data at unprecedented scale and complexity. We propose several extensions to the Hadoop platform that allow end-users to add and retrieve annotations seamlessly. Annotations in CloudNotes will be generated, propagated and managed in a distributed manner. We address several challenges that include attaching annotations to data at various granularities in Hadoop, annotating data in flat files with no known schema until query time, and creating and storing the annotations is a distributed fashion. We also present new storage mechanisms and novel indexing techniques that enable adding the annotations in small increments although Hadoop’s file system is optimized for large batch processing.

Creator
Contributors
Degree
Unit
Publisher
Language
  • English
Identifier
  • etd-042414-223528
Keyword
Advisor
Defense date
Year
  • 2014
Date created
  • 2014-04-24
Resource type
Rights statement

Relations

In Collection:

Items

Items

Permanent link to this page: https://digital.wpi.edu/show/vq27zn55t