Lung Fidelity Benchmark Software Methodology

Purpose

This methodology describes the complete workflow behind the Lung Fidelity Benchmark software. The software compares healthy, mock, or untreated lung epithelial cell-culture RNA-seq profiles against a fixed healthy human distal lung tissue reference and reports transcriptomic proximity using PCA centroid distance, Spearman correlation, and robustness summaries across PCA preprocessing choices.

1. Executive Summary

The Lung Fidelity Benchmark software is a reference-anchored transcriptomic benchmarking workflow for lung epithelial culture models. It asks a direct biological question: which two-dimensional or three-dimensional lung epithelial culture system is transcriptomically closest to healthy human distal lung tissue?

The bundled software includes a built-in benchmark database assembled from public bulk RNA-seq datasets. The database contains a healthy human distal lung tissue reference and multiple lung epithelial culture platforms, including primary monolayers, immortalized monolayers, ALI-differentiated epithelium, bronchial organoids, and engineered airway organ tissue equivalents.

The software can be run in demo mode using the bundled database alone, or it can accept a new user dataset and compare the user model against the built-in reference and culture-model database. The analysis uses two primary metrics: PCA centroid distance and sample-wise Spearman correlation. A Dolan-Moré performance profile summarizes the stability of PCA-based rankings across 32 background PCA preprocessing choices.

2. Built-In Benchmark Database

The built-in database contains public bulk RNA-seq expression matrices from GEO-derived lung epithelial and distal lung tissue datasets. Only healthy, mock, untreated, or baseline samples are included. This restriction reduces confounding from infection, pharmacologic treatment, or other perturbations.

The database is intentionally not exhaustive. It is a reproducible reference panel that demonstrates how a fixed benchmark and consistent preprocessing can be used to score culture systems and compare new models against a declared human tissue reference.

3. Software Operating Modes

The software supports three run modes. The same analysis engine is used in all modes after the input database has been assembled.

4. Required User Data Format

User-provided expression data must be supplied as CSV files. The first column must be named exactly Gene Symbol. Each additional column must represent one RNA-seq sample. Rows represent genes. Values should be raw counts or count-like non-negative expression values.

Gene Symbol,Sample_1,Sample_2,Sample_3
TP53,120,135,141
EPCAM,500,490,510
KRT8,850,870,860

• File type must be CSV.
• The first column name must be exactly Gene Symbol.
• Each remaining column must be a sample.
• Values should be raw counts or count-like non-negative values.
• Gene identifiers should be human gene symbols.
• Already z-scored values should not be used.
• Already batch-corrected values should not be used.
• Healthy, mock, untreated, or baseline samples are recommended for fidelity benchmarking.
• At least two to three biological replicates are recommended when possible.
• If batches exist, the user should provide batch labels through metadata or the interactive prompt.

4.1 Simple User Mode Metadata

In simple mode, the user provides one expression CSV and answers prompts for the dataset name, culture-system name, culture format, and optional technical batch label. The software then creates metadata automatically and appends the user dataset to the built-in database.

4.2 Advanced User Mode Metadata

In advanced mode, users can provide multiple datasets or multiple culture-system groups. A metadata file should define the group identity, culture format, source file, and technical batch labels. This mode is intended for users who want to compare several new models or multiple experimental batches in one run.

5. End-to-End Analysis Workflow

The software workflow has two major branches. The first branch generates the main PCA analysis and PCA centroid distances. The second branch generates Spearman correlation scores. A third background branch computes 32 PCA sensitivity distance methods and the Dolan-Moré robustness summaries.

6. Gene Harmonization and Matrix Assembly

1. All selected CSV files are read from the built-in database and optional user-data folder.
2. Each file is checked for the required Gene Symbol column.
3. Expression values are converted to numeric values, and missing values are set to zero.
4. Gene symbols are cleaned for the paper-exact PCA path by removing whitespace and converting symbols to uppercase.
5. Duplicate gene symbols within each file are summed.
6. All datasets are merged on shared gene symbols.
7. Genes with zero counts across all samples are removed before count-based correction and normalization.

The shared-gene design ensures that every sample, culture model, and user-added dataset is scored in the same molecular feature space.

7. Main PCA Analysis

The main PCA figures follow the paper-exact pipeline. This branch is used for the primary PCA scatter plot and the PCA centroid-distance figure.

7.1 ComBat-seq Correction

ComBat-seq is applied to raw counts using source_file as the batch variable in the paper-exact PCA branch. This reproduces the manuscript PCA workflow and preserves the discrete count scale before downstream normalization.

7.2 TMM Normalization and logCPM Transformation

After ComBat-seq, the adjusted count matrix is normalized with edgeR TMM normalization. This accounts for differences in library size and RNA composition. Values are then transformed to log2(CPM + 1) with a prior count of 1.

7.3 PCA Scaling

Genes with zero variance across samples are removed. PCA is then calculated using centered and scaled expression values, which is equivalent to gene-wise z-scoring before PCA. This prevents high-abundance genes from dominating the PCA solely because of their expression scale.

8. PCA Centroid Distance

Each sample receives PC1 and PC2 coordinates. For each analysis group, the centroid is calculated as the mean PC1 and PC2 coordinate across that group. The GSE165192-Human centroid is treated as the reference centroid.

For every non-reference group, Euclidean distance from the group centroid to the GSE165192-Human centroid is calculated. A smaller distance indicates stronger global transcriptomic proximity to the human distal lung reference.

9. Spearman Correlation Analysis

Spearman correlation provides a complementary gene-level similarity score. It is calculated on TMM-normalized log2(CPM + 1) values without gene-wise z-scoring.

The mean GSE165192-Human reference profile is calculated across all reference samples. Each culture-model sample is then correlated with this mean profile using Spearman correlation across shared genes. Results are summarized by analysis group using mean, standard deviation, and sample count.

10. Background PCA Sensitivity Analysis

The software also calculates 32 background PCA distance methods. These methods are stored as tables and are used for robustness evaluation, but they are not used to generate the main PCA figures.

10.1 Dolan-Moré Performance Profile

The Dolan-Moré performance profile summarizes which culture model is closest to the human reference across the 32 background PCA distance methods. For each PCA method, the smallest centroid distance is treated as the best distance. Each model receives a performance ratio equal to its distance divided by the best distance for that method.

A performance ratio of 1 means that the model ranked closest to the human reference for that PCA method. A ratio of 1.1 means that the model was within 10 percent of the best model. The Dolan-Moré curve shows the fraction of PCA methods in which each model is within a given performance-ratio threshold.

11. Software Outputs

12. Interpretation of Results

A culture model is considered more similar to human distal lung tissue when it has a high Spearman correlation, a low PCA centroid distance, and stable performance across background PCA methods. The two main metrics are complementary. PCA centroid distance reflects global transcriptomic structure, while Spearman correlation reflects rank concordance across genes.

• High Spearman correlation indicates that the gene-expression ordering is similar to the human distal lung reference profile.
• Low PCA centroid distance indicates that the group occupies a transcriptomic position close to the human reference in the main PCA space.
• High Dolan-Moré performance means that the model remains near the best model across many PCA preprocessing choices.
• Lower within-group dispersion suggests stronger reproducibility within a culture model or manufacturing batch.

13. Limitations and Recommended Reporting Language

The benchmark database is based on public bulk RNA-seq datasets and should be viewed as a reproducible reference panel, not a complete atlas of all lung culture systems. Cross-study comparisons can be affected by platform, library preparation, alignment, and quantification differences. The main PCA branch follows the paper-exact source-file ComBat-seq procedure, while background PCA distances are retained to assess method sensitivity.

14. How the Software Runs

1. The user launches the interactive Python script or a PowerShell helper script.
2. The user selects demo mode, simple user mode, or advanced user mode.
3. The software copies the built-in database and optional user files into the workspace input folder.
4. The validation script checks input structure, required columns, numeric values, and metadata consistency.
5. The R preprocessing script performs gene harmonization, count preparation, ComBat-seq, TMM normalization, and paper-exact PCA score generation.
6. The Python analysis script computes Spearman scores, PCA distances, background PCA sensitivity distances, Dolan-Moré summaries, figures, and reports.
7. All outputs are written to the workspace results folder.

cd "D:\lung_fidelity_benchmark_software"
python -m pip install -r requirements.txt
Rscript -e "if (!requireNamespace('BiocManager', quietly=TRUE)) install.packages('BiocManager'); BiocManager::install(c('edgeR','sva'), ask=FALSE, update=FALSE)"
python scripts\00_run_interactive.py

15. Package Structure

lung_fidelity_benchmark_software/
  database/expression/
  database/metadata/
  user_data/place_user_dataset_here/
  scripts/
  docs/
  workspace/input/
  workspace/processed/
  workspace/results/
  requirements.txt
  README.txt
  run_interactive.ps1
  run_demo.ps1

16. Validation and Quality Control

• Input validation checks that each CSV contains the required Gene Symbol column.
• All sample columns are checked for numeric values.
• Negative values are not expected for count-based input.
• User metadata are checked for required fields in advanced mode.
• The report includes total sample number, reference group, and rankings.
• If key tables are empty, the plotting script stops with a clear error rather than writing empty figures.
• All primary plots are exported as PNG, PDF, and SVG for manuscript and presentation use.

17. Glossary