loading page

  • +2
  • Ben Cardoen ,
  • Hanene Ben Yedder ,
  • Sieun Lee ,
  • Ivan Robert Nabi ,
  • Ghassan Hamarneh
Ben Cardoen
Simon Fraser University

Corresponding Author:[email protected]

Author Profile
Hanene Ben Yedder
Author Profile
Sieun Lee
Author Profile
Ivan Robert Nabi
Author Profile
Ghassan Hamarneh
Author Profile


Large-scale processing of heterogeneous datasets in interdisciplinary research often requires time-consuming manual data curation. Ambiguity in the data layout and preprocessing conventions can easily compromise reproducibility and scientific discovery, and even when detected, it requires time and effort to be corrected by domain experts. Poor data curation can also interrupt processing jobs on large computing clusters, causing frustration and delays. We introduce DataCurator , a portable software package that verifies arbitrarily complex datasets of mixed formats, working equally well on clusters as on local systems. Human-readable TOML recipes are converted into executable machine-verifiable templates, enabling users to easily verify datasets using custom rules without writing code. Recipes can be used to transform and validate data, for pre- or post-processing, selection of data subsets, sampling, and aggregation, such as summary statistics. Processing pipelines no longer need to be burdened by laborious data validation, with data curation and validation replaced by human and machine verifiable recipes specifying rules and actions. Multithreaded execution ensures scalability on clusters, and existing Julia, R, and Python libraries can be reused. DataCurator enables efficient remote workflows, offering integration with Slack and the ability to transfer curated data to clusters using OwnCloud and SCP. Code available at: https://github.com/bencardoen/DataCurator.jl.
05 Jan 2023Published in Bioinformatics Advances volume 3 issue 1. 10.1093/bioadv/vbad068