# Data and analysis outputs for the Difference Significance Profile (DSP) study

## Overview

This repository contains the particle-discrete input data and processed statistical outputs used to demonstrate the **Difference Significance Profile (DSP)** framework for comparing two particle size distributions (PSDs). DSP performs a bin-resolved comparison of two PSDs, standardises the difference in each common size bin by its estimated sampling uncertainty, and uses a parametric-bootstrap threshold to identify bins that remain significant after simultaneous calibration across the complete histogram.

The dataset covers two experimental comparisons of number-weighted particle size distributions measured during spray drying of alumina:

1. **Different operating conditions:** a run at 120 °C compared with a run at 145 °C (`T1_T2/`).
2. **Repeatability / negative-control comparison:** two runs performed at the nominally identical temperature of 120 °C (`T1_T1_repeat/`).

The repository contains both the original particle-level size measurements and the derived tables needed to reproduce the PSD overlays, difference profiles, DSP statistics, bootstrap calibration, covariance diagnostics, and summary results reported in the associated manuscript.

> **Associated manuscript**  
> Rahul Mitra, Lukas Fuchs, Ralf Ditscherlein, Volker Schmidt, and Urs Alexander Peuker, *A bin-resolved statistical framework for comparing particle size distributions*.

Detailed code-oriented instructions are provided separately in [`PYTHON_SCRIPTS.md`](PYTHON_SCRIPTS.md).

---

## Experimental devices and analysis settings

The principal experimental devices and analysis settings are listed here before the individual files are described.

### Particle-size measurement

- **System:** SOPAT **InView** in-line imaging system.
- **Probe:** SOPAT PL camera probe.
- **Measurement principle:** in-line two-dimensional dynamic imaging of individual spray-dried alumina particles.
- **Reported particle-size unit:** micrometres (µm).
- **Distribution weighting:** number-weighted; each detected particle contributes one observation.
- **Experimental conditions represented:** 120 °C, a repeat experiment at 120 °C, and 145 °C.

### Mercury intrusion porosimetry

- **Method:** mercury intrusion porosimetry (MIP).
- **Samples:** spray-dried alumina produced at 120 °C and 145 °C.
- **Recorded quantities:** intrusion and extrusion pressure and volume, pore size, specific pore volume, porosity, differential pore volume, and surface-area quantities.
- **Role in this repository:** supporting bulk pore-structure characterisation; the MIP data are not inputs to the DSP calculation.
- **Instrument identification:** PASCAL 440.

### DSP calculation settings

- **Common bins:** 75.
- **Bootstrap iterations:** 40,000.
- **Significance levels:** α = 0.01, 0.05, and 0.10.
- **Corresponding simultaneous levels:** 99%, 95%, and 90%.
- **Working model:** multinomial sampling model for number-weighted particle counts.
- **Primary analysis level:** α = 0.05, corresponding to the filename suffix `005`.

---

## File naming convention

Processed files follow this pattern:

```text
experimental_<sample1>_<sample2>_<output-type>_<alpha-code>.csv
```

Example:

```text
experimental_T120_1_T145_1_dsp_per_bin_005.csv
```

This means:

- sample 1: `T120_1`;
- sample 2: `T145_1`;
- output type: bin-resolved DSP results;
- significance level: `alpha = 0.05`.

### Significance-level suffixes

| Suffix | Significance level `alpha` | Simultaneous level `1 - alpha` | Typical interpretation |
|---:|---:|---:|---|
| `001` | 0.01 | 99% | Most conservative |
| `005` | 0.05 | 95% | Main analysis |
| `010` | 0.10 | 90% | More sensitive, less conservative |

The suffix changes the bootstrap decision threshold and, consequently, the number of bins marked significant. Files such as the PSD tables and covariance matrices are repeated for each suffix for completeness, even where their numerical content does not depend on `alpha`.
---

---

## Repository structure

```text
data/
├── SOPAT/
│   ├── experimental_T120_1.csv
│   ├── experimental_T120_11.csv
│   └── experimental_T145_1.csv
│
├── T1_T2/
│   ├── experimental_T120_porosity.csv
│   ├── experimental_T145_porosity.csv
│   ├── experimental_T120_1_T145_1_psd_bins_001.csv
│   ├── experimental_T120_1_T145_1_psd_bins_005.csv
│   ├── experimental_T120_1_T145_1_psd_bins_010.csv
│   ├── experimental_T120_1_T145_1_raw_difference_001.csv
│   ├── experimental_T120_1_T145_1_raw_difference_005.csv
│   ├── experimental_T120_1_T145_1_raw_difference_010.csv
│   ├── experimental_T120_1_T145_1_dsp_per_bin_001.csv
│   ├── experimental_T120_1_T145_1_dsp_per_bin_005.csv
│   ├── experimental_T120_1_T145_1_dsp_per_bin_010.csv
│   ├── experimental_T120_1_T145_1_summary_stats_001.csv
│   ├── experimental_T120_1_T145_1_summary_stats_005.csv
│   ├── experimental_T120_1_T145_1_summary_stats_010.csv
│   ├── experimental_T120_1_T145_1_bootstrap_M_001.csv
│   ├── experimental_T120_1_T145_1_bootstrap_M_005.csv
│   ├── experimental_T120_1_T145_1_bootstrap_M_010.csv
│   ├── experimental_T120_1_T145_1_bootstrap_running_c_001.csv
│   ├── experimental_T120_1_T145_1_bootstrap_running_c_005.csv
│   ├── experimental_T120_1_T145_1_bootstrap_running_c_010.csv
│   ├── experimental_T120_1_T145_1_cov_matrix_001.csv
│   ├── experimental_T120_1_T145_1_cov_matrix_005.csv
│   └── experimental_T120_1_T145_1_cov_matrix_010.csv
│
└── T1_T1_repeat/
    └── (same DSP outputs for the repeat comparison)
```

The repository contains **47 CSV files**:

- 3 raw SOPAT particle datasets
- 23 files in `T1_T2`
- 21 files in `T1_T1_repeat`

---

## How to navigate the repository

| If you want to... | Open... |
|-------------------|---------|
| Original particle sizes | `SOPAT/experimental_*.csv` |
| Common binned PSDs | `*_psd_bins_*.csv` |
| Raw probability differences | `*_raw_difference_*.csv` |
| Complete DSP results | `*_dsp_per_bin_*.csv` |
| Overall statistical summary | `*_summary_stats_*.csv` |
| Bootstrap distribution | `*_bootstrap_M_*.csv` |
| Bootstrap convergence | `*_bootstrap_running_c_*.csv` |
| Covariance matrix | `*_cov_matrix_*.csv` |
| Mercury intrusion porosimetry data | `experimental_T120_porosity.csv`, `experimental_T145_porosity.csv` |

---

## Experimental datasets and sample identifiers

The particle sizes were obtained from in-line two-dimensional imaging of spray-dried alumina using a SOPAT PL camera probe. Each row in the raw files represents one detected particle.

| Identifier | File | Condition | Number of particles | Size range in file |
|---|---|---:|---:|---:|
| `T120_1` | `SOPAT/experimental_T120_1.csv` | First run at 120 °C | 2,881 | 2.149–12.820 µm |
| `T120_11` | `SOPAT/experimental_T120_11.csv` | Repeat run at 120 °C | 3,015 | 2.149–13.012 µm |
| `T145_1` | `SOPAT/experimental_T145_1.csv` | Run at 145 °C | 2,658 | 2.149–10.988 µm |

The comparison order matters because all signed DSP quantities are calculated as:

```text
sample 1 - sample 2
```

Therefore:

- in `T1_T2/`, sample 1 is `T120_1` and sample 2 is `T145_1`;
- in `T1_T1_repeat/`, sample 1 is `T120_1` and sample 2 is `T120_11`.

A positive difference means that the first sample has a larger probability in that size bin. A negative difference means that the second sample has a larger probability.

---

## Detailed description of each file type

## 1. Raw particle-level files: `SOPAT/experimental_*.csv`

Each file contains one column:

| Column | Type | Unit | Description |
|---|---|---|---|
| `size` | floating point | µm | Measured size of one detected particle. Each row is one particle observation. |

Example:

```csv
size
3.474734882
4.2971833798
```

These files can be used to:

- reconstruct the empirical number-weighted PSDs;
- apply alternative binning rules;
- compute descriptive statistics or cumulative distributions;
- reproduce the common pooled binning used for DSP; or
- test other distribution-comparison methods.

Only positive, non-missing particle sizes should be used in the analysis.

---

## 2. Binned PSD files: `*_psd_bins_<alpha-code>.csv`

These files contain the two empirical PSDs evaluated using the same 75 common bins.

| Column | Type | Unit | Description |
|---|---|---|---|
| `bin_left` | floating point | µm | Left edge of the bin. |
| `bin_right` | floating point | µm | Right edge of the bin. |
| `bin_mid` | floating point | µm | Arithmetic midpoint of the bin. |
| `p1hat` | floating point | dimensionless probability | Empirical probability for sample 1 in the bin. |
| `p2hat` | floating point | dimensionless probability | Empirical probability for sample 2 in the bin. |

For each sample, the bin probabilities sum to approximately 1, subject only to floating-point precision.

Use these files to create:

- overlaid histogram/bar representations of the PSDs;
- per-bin probability comparisons; or
- cumulative distributions through `cumsum(p1hat)` and `cumsum(p2hat)`.

The binning is identical across the three alpha versions of a given comparison.

---

## 3. Raw difference files: `*_raw_difference_<alpha-code>.csv`

These compact tables contain the unstandardised difference between the two empirical PSDs.

| Column | Type | Unit | Description |
|---|---|---|---|
| `bin_mid` | floating point | µm | Midpoint of the common size bin. |
| `Delta_i` | floating point | probability difference | `p1hat - p2hat` in the bin. |

Interpretation:

- `Delta_i > 0`: sample 1 is enriched relative to sample 2;
- `Delta_i < 0`: sample 1 is depleted relative to sample 2;
- `Delta_i = 0`: equal observed probability in that bin.

Because both PSDs sum to 1, the differences sum to approximately zero:

```text
sum(Delta_i) ≈ 0
```

These files are descriptive only. Statistical significance should be taken from the corresponding `dsp_per_bin` files.

---

## 4. Main bin-resolved DSP files: `*_dsp_per_bin_<alpha-code>.csv`

These are the central analysis tables. Each row represents one common size bin.

| Column | Type | Unit | Description |
|---|---|---|---|
| `bin_left` | floating point | µm | Left bin edge. |
| `bin_right` | floating point | µm | Right bin edge. |
| `bin_mid` | floating point | µm | Bin midpoint. |
| `n1i` | integer | particle count | Number of sample-1 particles in bin `i`. |
| `n2i` | integer | particle count | Number of sample-2 particles in bin `i`. |
| `p1hat` | floating point | probability | `n1i / n1`. |
| `p2hat` | floating point | probability | `n2i / n2`. |
| `Delta` | floating point | probability difference | `p1hat - p2hat`. |
| `SE` | floating point | probability difference | Estimated standard error of `Delta` under the pooled null model. |
| `Z` | floating point | dimensionless | Standardised DSP statistic, `Delta / SE`; set to 0 when `SE = 0`. |
| `sig_flag` | integer | binary | `1` if `abs(Z) > crit_boot`; otherwise `0`. |
| `crit_boot` | floating point | dimensionless | Bootstrap-calibrated simultaneous threshold for the alpha level encoded in the filename. The same value is repeated in every row. |
| `chi2_contrib` | floating point | dimensionless | Contribution of the bin to the global Pearson chi-square statistic. |

The estimated pooled bin probability is not stored explicitly, but it can be reconstructed as:

```text
phat_i = (n1i + n2i) / (n1 + n2)
```

The standard error is:

```text
SE_i = sqrt[phat_i * (1 - phat_i) * (1/n1 + 1/n2)]
```

The local DSP statistic is:

```text
Z_i = Delta_i / SE_i
```

A bin is marked significant when:

```text
abs(Z_i) > crit_boot
```

Equivalent simultaneous bounds on the probability-difference scale are:

```text
-crit_boot * SE_i  and  +crit_boot * SE_i
```

Use these files to:

- identify significant size classes;
- inspect the direction and magnitude of local differences;
- recreate the DSP difference-band figure;
- recreate the Z-profile figure; or
- rank bins by `abs(Z)`, `abs(Delta)`, or `chi2_contrib`.

---

## 5. Comparison-level summary files: `*_summary_stats_<alpha-code>.csv`

Each summary file contains one row describing the complete comparison.

| Column | Type | Description |
|---|---|---|
| `n1` | integer | Total number of particles in sample 1. |
| `n2` | integer | Total number of particles in sample 2. |
| `num_bins` | integer | Number of common bins; 75 in this repository. |
| `chi2_pearson` | floating point | Global Pearson chi-square discrepancy. |
| `p_bootstrap` | floating point | Bootstrap p-value for the global Pearson statistic. |
| `c_bootstrap` | floating point | Simultaneous bootstrap threshold. Despite the column name, its actual level follows the filename suffix. |
| `n_sig_bins` | integer | Number of bins with `sig_flag = 1`. |
| `sig_bins_percent` | floating point | Percentage of all bins marked significant. |
| `B_boot` | integer | Number of bootstrap iterations; 40,000 here. |

A stored `p_bootstrap` value of `0.0` means that none of the 40,000 bootstrap Pearson statistics was at least as large as the observed statistic in the implemented calculation. It should be reported as an empirical value below the resolution of the simulation, rather than interpreted as a mathematically exact probability of zero. With 40,000 replicates, a cautious report is `p < 1/40000` (or according to the p-value convention used in the final analysis code).

### Summary of the supplied results

| Comparison | Alpha | Simultaneous threshold | Pearson chi-square | Bootstrap p-value | Significant bins |
|---|---:|---:|---:|---:|---:|
| `T120_1` vs `T145_1` | 0.01 | 3.5630 | 257.0438 | 0.0 in file | 6 / 75 (8.0%) |
| `T120_1` vs `T145_1` | 0.05 | 3.1389 | 257.0438 | 0.0 in file | 10 / 75 (13.33%) |
| `T120_1` vs `T145_1` | 0.10 | 2.9464 | 257.0438 | 0.0 in file | 11 / 75 (14.67%) |
| `T120_1` vs `T120_11` | 0.01 | 3.5678 | 61.5611 | 0.313425 | 0 / 75 (0%) |
| `T120_1` vs `T120_11` | 0.05 | 3.1558 | 61.5611 | 0.313425 | 0 / 75 (0%) |
| `T120_1` vs `T120_11` | 0.10 | 2.9484 | 61.5611 | 0.313425 | 0 / 75 (0%) |

These values show a pronounced difference between the two drying temperatures and no detected bin-wise difference between the repeat runs at the supplied simultaneous levels.

---

## 6. Bootstrap-maximum files: `*_bootstrap_M_<alpha-code>.csv`

Each file contains 40,000 bootstrap replicates of the maximum absolute local statistic under the fitted null model.

| Column | Type | Description |
|---|---|---|
| `iter` | integer | Bootstrap iteration index, from 1 to 40,000. |
| `M_star` | floating point | `max_i(abs(Z_i_star))` for that bootstrap replicate. |

The simultaneous threshold is the empirical `(1 - alpha)` quantile of `M_star`.

The `M_star` sequence is identical across the `001`, `005`, and `010` versions of the same comparison because the same bootstrap sample is used; only the selected quantile changes.

Use these files to:

- recalculate thresholds for other alpha levels without rerunning the multinomial bootstrap;
- plot the null distribution of the maximum statistic;
- assess Monte Carlo variability; or
- compare alternative quantile conventions.

---

## 7. Running-threshold files: `*_bootstrap_running_c_<alpha-code>.csv`

These files show how the estimated threshold evolves as progressively more bootstrap replicates are included.

| Column | Type | Description |
|---|---|---|
| `iter` | integer | Number of bootstrap replicates included in the running estimate. |
| `running_c` | floating point | Running `(1 - alpha)` quantile of the available `M_star` values. The actual alpha is given by the filename suffix. |

Use these files to reproduce convergence plots and determine whether the chosen 40,000 iterations provide a stable threshold.

The final row should agree, up to numerical precision, with the threshold reported in the corresponding summary and per-bin files.

---

## 8. Covariance matrices: `*_cov_matrix_<alpha-code>.csv`

Each covariance file is a 75 × 75 square matrix. Columns are named `bin_0` to `bin_74`; row order follows the same bin order.

The matrix represents the theoretical covariance of the sample-1 empirical proportions under the pooled multinomial null model:

```text
Cov(p1hat) = [diag(phat) - phat * phat^T] / n1
```

Interpretation:

- diagonal entries are the per-bin variances;
- off-diagonal entries are generally negative because multinomial bin probabilities are compositional and must sum to one;
- row `i`, column `j` corresponds to the covariance between bins `i` and `j`;
- map matrix index `i` to its physical size interval using row `i` of the corresponding `psd_bins` or `dsp_per_bin` file.

The covariance matrix does not depend on `alpha`; therefore, the `001`, `005`, and `010` copies within a given comparison are numerically identical.

---

## 9. Mercury intrusion porosimetry data

The two porosity files contain **mercury intrusion porosimetry** measurements of the spray-dried alumina powders produced at **120 °C** and **145 °C**.

These measurements are **not part of the DSP algorithm**. Instead, they provide additional physical characterisation of the powders and were used to support the interpretation of the statistically significant differences identified by DSP.

### Files

- `experimental_T120_porosity.csv`
- `experimental_T145_porosity.csv`

### Variables

The files contain measurements such as:

- Intrusion pressure (MPa)
- Intrusion volume (mm³/g)
- Extrusion pressure (MPa)
- Extrusion volume (mm³/g)
- Pore size (nm)
- Specific pore volume (mm³/g)
- Porosity (%)
- Differential pore volume (`dV/dD`)
- Cumulative surface area (m²/g)
- Relative surface area (m²/g)

These data can be used to:

- plot pore-size distributions,
- compare cumulative pore volume,
- compare total porosity,
- analyse surface-area distributions,
- correlate internal pore structure with the observed PSD changes.

---

## Reproducibility and analysis scripts

The Python examples, package requirements, file-loading instructions, plotting workflows, conceptual reproduction steps, and integrity checks are documented separately in [`PYTHON_SCRIPTS.md`](PYTHON_SCRIPTS.md). Keeping the code-oriented documentation separate makes this README easier to navigate while preserving detailed reproducibility guidance.

---

## Units, weighting, and interpretation

- Particle size is given in **micrometres (µm)**.
- The raw observations and derived PSDs are **number-weighted**: every detected particle contributes one observation.
- `p1hat`, `p2hat`, `Delta`, and `SE` are dimensionless probabilities or probability differences.
- `Z`, `crit_boot`, `M_star`, and Pearson chi-square quantities are dimensionless.
- The output does not represent a mass- or volume-weighted PSD.
- DSP indicates where the observed probability mass differs statistically. It does not by itself identify a unique physical mechanism responsible for the difference.
- Since PSDs are compositional, enrichment in one region must be accompanied by depletion elsewhere. Significant bins should therefore be interpreted as locations of a broader redistribution of probability mass.

---

## Software and file format

All files are plain-text comma-separated values (`.csv`) and can be opened with:

- Python/pandas;
- R;
- MATLAB;
- Julia;
- LibreOffice Calc;
- Microsoft Excel; or
- any text editor.

Technical conventions:

- delimiter: comma;
- decimal separator: period;
- header row: present;
- text encoding: UTF-8-compatible ASCII column names;
- missing values: none expected in the processed outputs;
- matrix row labels: not stored in the covariance CSVs; row order is the bin index order.

For very large files such as the 40,000-row bootstrap tables, a scripting environment is usually more convenient than spreadsheet software.

---

## Limitations and cautions

1. **Model assumption.** The supplied DSP inference is based on a multinomial working model for independent particle detections. Strong within-sample dependence or overdispersion is not represented in these output tables.
2. **Binning dependence.** Bin-resolved results apply to the supplied common binning. Alternative bin edges may change local estimates and detection power.
3. **Simultaneous, not point-wise, interpretation.** `crit_boot` is calibrated using the maximum statistic across all bins. It is not an ordinary per-bin 95% normal cutoff.
4. **No posterior probabilities.** A significant bin is not a statement that the bin differs with a 95% or 99% posterior probability.
5. **Failure to detect is not proof of equality.** `sig_flag = 0` means the observed local difference does not exceed the supplied simultaneous threshold with the available sample sizes.
6. **Repeated files across alpha levels.** Some files are duplicated across `001`, `005`, and `010`, even when their content does not depend on alpha.
7. **Global p-value resolution.** A stored bootstrap p-value of zero reflects finite simulation resolution and should not be described as an exact probability of zero.

---

## Contact

For questions concerning the dataset or DSP analysis, contact:

**Dr. Rahul Mitra**  
Institute of Mechanical Process Engineering and Mineral Processing  
Technische Universität Bergakademie Freiberg  
Agricolastr. 1, 09599 Freiberg, Germany  
Email: Rahul.Mitra@mvtat.tu-freiberg.de