How waldronlab/curatedmetagenomicdatacuration works
Main Components
• Core Package Logic:
• R Scripts: Include the functions provided in the “R/” directory (e.g., PubMed-Class.R, PubMed.R, checkCuration.R, get_metadata.R, makeCombinedMetadata.R, metadata_long_form.R). These represent the underlying modules that perform metadata extraction, validation, and formatting.
• Man Documentation: Represents the API help files generated from the R code, defining user interaction points.
• Data Component:
• Curated Data Files: The “inst/curated” directory contains numerous dataset-specific subdirectories holding metadata TSV files (and occasionally helper scripts in other languages such as Python). These serve as input files.
• Template/Ext Data: Files provided in “inst/extdata” (like the template.csv) define the expected structure for the metadata.
• Testing and Validation:
• The “tests/testthat” directory includes unit tests that ensure that the functions work correctly.
• User Interface and Documentation:
• Vignettes: The “vignettes/curatedMetagenomicDataCuration.Rmd” offers a walkthrough for users.
• README and Wiki: Serve as guides for both curators and developers.
• Continuous Integration / Deployment:
• GitHub Workflows (under “.github/workflows/”): Contains YAML configurations to automatically check and build the package whenever changes are pushed or pull requests are submitted.
Relationships and Interactions
• Data Flow:
• Input: A user adds or updates a metadata TSV file under “inst/curated/SomeDataset/”.
• Processing: The R package’s core functions (e.g., checkCuration, get_metadata) read, validate, and transform the input data.
• Output & Reporting: After processing, the functions may trigger reports (e.g., the detailed syntax-checking report) that can be viewed as HTML vignettes or linked on the project website.
• Testing:
• Tests run via GitHub Actions and locally to verify the results from various functions.
• CI/CD Pipeline:
• The GitHub workflows automatically run the test suite, build documentation (using pkgdown), and possibly generate badges showing build status.
• User Engagement:
• Contributors interact through pull requests that update the curated files, and GitHub Issues (with templates) are used to file bug reports or request features.
Architectural Patterns and Design Principles
• Modular Design:
• The package’s logic is split across multiple files (following the typical R package structure) to separate concerns.
• Input Validation & Data Transformation:
• Functions are dedicated to checking the integrity of metadata and converting between ‘long form’ and other representations.
• Continuous Integration:
• Integration of automated testing and documentation builds ensures the system maintains high quality.
• Package Documentation Best Practices:
• Inclusion of man pages and vignettes makes the system accessible for both end users and developers.
Relevant Technologies, Frameworks, and Libraries
• R (and Bioconductor):
• The project is built as an R package with standard tools such as DESCRIPTION and NAMESPACE.
• GitHub Actions:
• Configuration files under “.github/workflows/” manage CI/CD.
• pkgdown:
• Likely used for generating the project website from the documentation.
• Testing with testthat:
• Ensures robust unit testing.
• Python Scripts:
• Some curated dataset directories include Python helper scripts, indicating minor cross-language components.
• Markdown and R Markdown:
• Used for documentation and vignettes.