Context

Data management is often a key challenge for large organizations that carry the burden of periodically processing, cleaning, curating and harmonizing datasets collected in different formats from multiple heterogenous sources. Data managers spend a tremendous amount of time doing this tedious work manually and repetitively.

Some organizations have invested in automating recurring data management tasks using traditional Extract Transform Load (ETL) tools. Unfortunately, it is not scalable to depend on software engineers to modify ETLs for each new data cleaning challenge.

Objectives

Build a secure web-based platform to:

• Increase autonomy of data managers and reduce reliance on software engineers
• Harmonize and centralize the datasets collected from multiple heterogenous sources into a single harmonized datasets
• Encourage data sharing and attribution
• Provide tools to visualize the new data
• Keep a complete revision history for all data transforms and loads