refine.bio Documentation

refine.bio is a multi-organism collection of genome-wide transcriptome or gene expression data that has been obtained from publicly available repositories and uniformly processed and normalized. refine.bio allows biologists, clinicians, and machine learning researchers to search for experiments from different source repositories all in one place and build custom data sets for their questions of interest.

refine.bio is well-suited for quickly assessing if signals are present in particular datasets, for identifying and obtaining data sets for accelerated validation of findings, and for building large compendia for training machine learning models that can adequately handle the technical noise associated with integrating multiple experiments and platforms.

refine.bio is not a substitute for experiments and processing pipelines tailored to answer specific biological questions of interest or for input from relevant experts (e.g., those with statistics expertise), but rather a repository of samples processed with standardized pipelines that have been selected based on their wide-ranging utility.

For examples of how to use refine.bio data, please see Downstream Analysis with refine.bio Examples.

Frequently asked questions

What type of data does refine.bio support?

How do you process the data?

How can I find out what versions of software/packages were used to process the data?

• Source Data
  • Types of Data
  • Sources
  • Metadata
    • refine.bio-harmonized Metadata
    • Submitter Supplied Metadata
• Processing Information
  • refine.bio processed
    • Microarray pipelines
      • Affymetrix
      • Illumina BeadArrays
    • RNA-seq pipelines
      • Salmon
      • tximport
  • Submitter processed
    • Affymetrix identifier conversion
    • Illumina identifier conversion
  • Aggregations
    • Limitations of gene identifiers when combining across platforms
  • Transformations
    • Quantile normalization
      • Reference distribution
      • Quantile normalizing samples for delivery
      • Limitations of quantile normalization across platforms with many zeroes
      • Skipping quantile normalization for RNA-seq experiments
    • Gene transformations
• Downloadable Files
  • The download folder structure for data aggregated by experiment:
  • The download folder structure for data aggregated by species:
  • Gene Expression Matrix
  • Sample Metadata
    • TSV files
    • JSON files
  • Experiment Metadata
• refine.bio Compendia
  • Normalized compendia
    • Collapsing by genus
    • Normalized Compendium Download Folder
  • RNA-Seq Sample Compendia
    • RNA-Seq Sample Compendium Download Folder
• API
• Downstream Analysis with refine.bio Examples
• Getting Started with a refine.bio dataset
  • Structure
  • Contents
  • Usage
    • Reading TSV Files
    • Reading JSON Files
• Getting Started with Normalized Compendia
  • Structure
  • Contents
  • Notes and observations
    • Methods evaluation and exploratory data analysis
  • Usage
    • Reading TSV Files
    • Reading JSON Files
• FAQ
  • What is the difference between refine.bio-processed and submitter-processed datasets?
  • How do you process the data?
  • What type of data does refine.bio support?
  • What does “corrected” metadata mean?
  • Why do the values differ a little bit if I download different datasets?
  • Why do I get a limited number of genes back when I aggregate samples from different experiments?
  • Why can’t I add certain samples to my dataset?
  • Why do the genes included in RNA-seq experiments change between experiments from the same organism?
  • How can I find out what genome build and release were used to process RNA-seq data?
  • How can I find out what versions of software/packages were used to process the data?
  • Are refine.bio datasets I download batch corrected?
  • Why are the expression values different if I regenerate a dataset?
  • What does it mean to skip quantile normalization for RNA-seq samples?
  • How do I cite refine.bio?
• License

Citing refine.bio

Please use the following:

Casey S. Greene, Dongbo Hu, Richard W. W. Jones, Stephanie Liu, David S. Mejia, Rob Patro, Stephen R. Piccolo, Ariel Rodriguez Romero, Hirak Sarkar, Candace L. Savonen, Jaclyn N. Taroni, William E. Vauclain, Deepashree Venkatesh Prasad, Kurt G. Wheeler. refine.bio: a resource of uniformly processed publicly available gene expression datasets. URL: https://www.refine.bio

Note that the contributor list is in alphabetical order as we prepare a manuscript for submission.

Questions/Feedback?

If you have a question or comment, please file an issue on GitHub or send us an email at requests@ccdatalab.org.