Explainable AI for Spectroscopy: Why FDA Demands Interpretability

Your spectral classifier achieves 97% sensitivity and 95% specificity on held-out patient data. The ROC curve is beautiful. The confusion matrix is clean. You are ready for a regulatory submission.

Then the reviewer asks: why does your model classify this spectrum as malignant?

If your answer is "because the model said so," your submission is in trouble. Not because explainability is always legally mandated - the regulatory landscape is more nuanced than that - but because a regulator who cannot understand what your model learned cannot assess whether it learned the right thing. And a clinician who cannot understand a diagnostic result cannot safely act on it.

This article covers the four primary explainability methods for spectral classifiers - SHAP, LIME, attention mechanisms, and gradient-based methods - with working Python code. It also covers the regulatory context: what the FDA and the EU AI Act actually require, and how to structure an explainability report for a marketing submission.

We assume you have already built a spectral classification pipeline. If not, start with our end-to-end ML pipeline guide, which covers preprocessing through deployment. The preprocessing choices you made there - baseline correction, normalization, derivative features - directly affect what your explainability methods reveal. See our complete preprocessing reference for the details.

The Regulatory Landscape

FDA: Strongly Recommended, Not Mandated

The FDA does not require explainability for AI/ML-based Software as a Medical Device (SaMD). But it strongly recommends it, and the recommendation has grown louder with each guidance document since 2021.

The trajectory is clear:

1. October 2021 - Good Machine Learning Practice (GMLP) for Medical Device Development introduced 10 guiding principles developed jointly with Health Canada and the UK MHRA. Principle 6 states that models should be designed to be "as understandable as practicable" and that "efforts should be made to ensure that they are interpretable."

2. April 2023 - Marketing Submission Recommendations for a Predetermined Change Control Plan (PCCP) guidance introduced the concept of documenting how a model will change post-market. Explaining what the model currently does is a prerequisite for explaining how it will change.

3. June 2024 - Transparency for Machine Learning-Enabled Medical Devices guidance made the recommendation explicit. The FDA recommends that manufacturers provide information about the AI's "logic" or "explainability" to the extent practicable, so that users can understand the basis for the device's output. This is not a mandate - the guidance uses "recommend," not "require" - but it sets the expectation.

4. January 2025 - Artificial Intelligence-Enabled Device Software Functions: Lifecycle Management and Marketing Submission Recommendations consolidated the FDA's Total Product Lifecycle (TPLC) approach. The guidance walks through what manufacturers should include in marketing submissions for AI-enabled devices. Explainability is woven throughout: in the algorithm description, in the performance evaluation, and in the labeling recommendations.

The practical reality: a 510(k) or De Novo submission for a spectral diagnostic that includes no explainability analysis is not automatically rejected. But you will receive questions. Those questions add months to your review timeline and require data you should have generated during development. It is cheaper to build explainability into your pipeline from the start.

For a deeper dive into FDA SaMD classification for spectroscopy software, see our SaMD classification guide.

EU AI Act: Transparency Is a Legal Requirement

The European Union's AI Act takes a harder line. AI systems embedded in or acting as medical devices regulated under the Medical Device Regulation (MDR) or the In Vitro Diagnostic Regulation (IVDR) are automatically classified as high-risk AI systems. Spectroscopy-based diagnostics - an FTIR bacterial classifier, a Raman tissue analyzer - almost certainly qualify.

High-risk AI systems must meet mandatory requirements including:

• Transparency obligations (Article 13): The system must be designed to allow users to "interpret the system's output and use it appropriately." Providers must supply information about the AI system's capabilities, limitations, performance, and "when applicable, the logic involved" in producing outputs.
• Human oversight (Article 14): The system must be designed to be "effectively overseen by natural persons" during its period of use. This requires that users understand what the system is doing - which requires explainability.
• Technical documentation (Annex IV): Providers must document the "general logic of the AI system" and the "main design choices" including the system's computational methodology.

The compliance deadline for high-risk AI under the AI Act was originally August 2, 2026. However, the December 2025 EU Commission proposals may extend certain deadlines - AI-enabled medical devices under Annex I may shift to August 2028. Regardless of the exact date, the requirements are final and preparation should be underway.

For the full dual-compliance picture (IVDR + AI Act), see our EU regulatory guide for spectroscopy diagnostics.

SHAP for Spectral Features

SHAP (SHapley Additive exPlanations) is rooted in cooperative game theory. Each feature (wavenumber) receives a Shapley value representing its marginal contribution to the prediction. The sum of all Shapley values equals the difference between the model's prediction for this spectrum and the average prediction across the training set.

For spectral data, SHAP answers the question: which wavenumber regions pushed this classification toward malignant vs. benign?

KernelSHAP for Model-Agnostic Explanation

KernelSHAP works with any classifier - SVM, Random Forest, PLS-DA, neural networks. It approximates Shapley values by sampling feature coalitions:

import numpy as np
import shap
from sklearn.ensemble import RandomForestClassifier
from sklearn.decomposition import PCA
 
# Assume: spectra (n_samples, n_wavenumbers), labels, wavenumbers
# Model trained on PCA-reduced spectra
pca = PCA(n_components=20)
scores = pca.fit_transform(spectra)
 
clf = RandomForestClassifier(n_estimators=500, random_state=42)
clf.fit(scores, labels)
 
# Create SHAP explainer in PCA space
background = shap.sample(scores, 100)
explainer = shap.KernelExplainer(clf.predict_proba, background)
 
# Explain a single spectrum
sample_scores = scores[0:1]
shap_values_pca = explainer.shap_values(sample_scores)
 
# Map SHAP values back to wavenumber space
# Each PCA component contributes to each wavenumber via the loadings
shap_values_spectral = shap_values_pca[1] @ pca.components_

TreeSHAP for Ensemble Models

If your model is tree-based (Random Forest, XGBoost, LightGBM), TreeSHAP computes exact Shapley values in polynomial time - orders of magnitude faster than KernelSHAP:

import shap
import xgboost as xgb
 
model = xgb.XGBClassifier(
    n_estimators=300,
    max_depth=6,
    learning_rate=0.1,
    use_label_encoder=False,
    eval_metric="logloss"
)
model.fit(spectra, labels)
 
explainer = shap.TreeExplainer(model)
shap_values = explainer.shap_values(spectra)
 
# shap_values shape: (n_samples, n_wavenumbers)
# Each value = contribution of that wavenumber to the prediction

Visualizing SHAP Over the Spectrum

The key visualization for spectral SHAP is a plot of SHAP value vs. wavenumber - it looks like a spectrum itself, but the y-axis shows importance rather than intensity:

import matplotlib.pyplot as plt
 
def plot_spectral_shap(wavenumbers, spectrum, shap_values,
                       class_name="Malignant"):
    fig, axes = plt.subplots(2, 1, figsize=(12, 8), sharex=True)
 
    # Top: original spectrum
    axes[0].plot(wavenumbers, spectrum, color="white", linewidth=0.8)
    axes[0].set_ylabel("Absorbance")
    axes[0].set_title("Input Spectrum")
 
    # Bottom: SHAP values across wavenumbers
    colors = np.where(shap_values > 0, "#ff2d55", "#00d4ff")
    axes[1].bar(wavenumbers, shap_values, color=colors, width=2)
    axes[1].set_xlabel("Wavenumber (cm⁻¹)")
    axes[1].set_ylabel("SHAP Value")
    axes[1].set_title(f"Feature Importance for '{class_name}' Prediction")
    axes[1].axhline(y=0, color="gray", linewidth=0.5)
 
    plt.tight_layout()
    return fig

This plot is the centerpiece of your explainability report. Peaks in the SHAP plot should correspond to known molecular vibrations relevant to the diagnostic target. If the model is classifying tissue samples as malignant based on SHAP peaks at the amide I (1650 cm⁻¹) and amide II (1545 cm⁻¹) bands, that aligns with the known biochemistry - cancer alters protein secondary structure. If the SHAP peaks are in regions with no known biological relevance, the model may be learning artifacts.

Spectral Zones SHAP

Standard SHAP perturbs individual wavenumbers independently. This is physically unrealistic - neighboring wavenumbers in a spectrum are highly correlated, and perturbing one while holding adjacent points fixed creates spectra that could never exist in reality. Contreras et al. (2024, Analytical Chemistry) addressed this with Spectral Zones-based SHAP, which groups neighboring wavenumbers into contiguous zones before perturbation:

def create_spectral_zones(wavenumbers, zone_width_cm=50):
    zones = []
    zone_start = wavenumbers[0]
    current_zone = []
 
    for i, wn in enumerate(wavenumbers):
        if wn - zone_start <= zone_width_cm:
            current_zone.append(i)
        else:
            zones.append(current_zone)
            current_zone = [i]
            zone_start = wn
    if current_zone:
        zones.append(current_zone)
 
    return zones
 
def zone_shap_values(explainer, spectrum, zones,
                     n_perturbations=500):
    n_zones = len(zones)
    zone_shap = np.zeros(n_zones)
    baseline = explainer.expected_value
 
    for _ in range(n_perturbations):
        mask = np.random.binomial(1, 0.5, n_zones)
        perturbed = spectrum.copy()
 
        for z_idx, zone_indices in enumerate(zones):
            if mask[z_idx] == 0:
                perturbed[zone_indices] = 0.0
 
        pred = explainer.model.predict_proba(
            perturbed.reshape(1, -1)
        )[0, 1]
        contribution = pred - baseline
 
        for z_idx in range(n_zones):
            if mask[z_idx] == 1:
                zone_shap[z_idx] += contribution / n_perturbations
 
    return zone_shap

Zone-based perturbation produces explanations that are more chemically interpretable because each zone maps to a contiguous spectral region - often corresponding to a single molecular vibration or a group of related vibrations.

LIME for Spectral Classifiers

LIME (Local Interpretable Model-agnostic Explanations) explains individual predictions by fitting a simple interpretable model (typically linear regression) to perturbed versions of the input in the neighborhood of the sample being explained.

For spectra, LIME perturbs the input by zeroing out groups of wavenumbers and observes how the prediction changes:

import lime
import lime.lime_tabular
 
# Create LIME explainer
lime_explainer = lime.lime_tabular.LimeTabularExplainer(
    training_data=spectra,
    feature_names=[f"{wn:.0f} cm⁻¹" for wn in wavenumbers],
    class_names=["Benign", "Malignant"],
    mode="classification",
    discretize_continuous=False
)
 
# Explain a single prediction
explanation = lime_explainer.explain_instance(
    spectra[0],
    clf.predict_proba,
    num_features=20,
    num_samples=5000
)
 
# Extract feature importance
important_features = explanation.as_list()
# Returns: [('1650 cm⁻¹', 0.15), ('1545 cm⁻¹', 0.12), ...]

When to Use LIME vs. SHAP

LIME is faster for individual explanations - it fits a local linear model rather than computing Shapley values over all feature coalitions. But SHAP has stronger theoretical guarantees: the Shapley values are the only feature attribution method that satisfies local accuracy, missingness, and consistency simultaneously.

For regulatory submissions, SHAP is generally preferred because:

1. Consistency -- SHAP values always sum to the prediction difference, making them directly auditable
2. Global explanations -- Aggregating SHAP values across samples produces a global feature importance ranking, which regulators want to see alongside per-sample explanations
3. Theoretical foundation -- Shapley values have a 70-year mathematical foundation in cooperative game theory, which helps in regulatory discussions

LIME is useful as a complementary method - showing that two independent explainability approaches agree on the important spectral regions strengthens your submission.

Attention-Based Explanations

If your model uses an attention mechanism - a transformer operating on spectral patches, or a CNN with an attention module - the attention weights themselves provide explanations. Unlike post-hoc methods (SHAP, LIME), attention-based explanations are generated during inference at no additional computational cost.

Self-Attention Spectral Transformer

A transformer that processes spectral patches naturally produces attention maps showing which spectral regions the model focuses on:

import torch
import torch.nn as nn
 
class SpectralPatchTransformer(nn.Module):
    def __init__(self, spectrum_length, patch_size=16,
                 d_model=128, n_heads=4, n_layers=3,
                 n_classes=2):
        super().__init__()
        self.patch_size = patch_size
        n_patches = spectrum_length // patch_size
 
        self.patch_embed = nn.Linear(patch_size, d_model)
        self.pos_embed = nn.Parameter(
            torch.randn(1, n_patches + 1, d_model) * 0.02
        )
        self.cls_token = nn.Parameter(
            torch.randn(1, 1, d_model) * 0.02
        )
 
        encoder_layer = nn.TransformerEncoderLayer(
            d_model=d_model,
            nhead=n_heads,
            dim_feedforward=d_model * 4,
            dropout=0.1,
            batch_first=True
        )
        self.transformer = nn.TransformerEncoder(
            encoder_layer, num_layers=n_layers
        )
 
        self.head = nn.Linear(d_model, n_classes)
        self.attention_weights = None
 
    def forward(self, x):
        batch_size = x.shape[0]
        # x shape: (batch, 1, spectrum_length)
        x = x.squeeze(1)
 
        # Split spectrum into patches
        patches = x.unfold(1, self.patch_size, self.patch_size)
        # patches shape: (batch, n_patches, patch_size)
 
        tokens = self.patch_embed(patches)
        cls = self.cls_token.expand(batch_size, -1, -1)
        tokens = torch.cat([cls, tokens], dim=1)
        tokens = tokens + self.pos_embed
 
        encoded = self.transformer(tokens)
 
        self.attention_weights = self._extract_attention(tokens)
 
        cls_output = encoded[:, 0]
        return self.head(cls_output)
 
    def _extract_attention(self, tokens):
        with torch.no_grad():
            q = k = tokens
            attn = torch.matmul(q, k.transpose(-2, -1))
            attn = attn / (tokens.shape[-1] ** 0.5)
            attn = torch.softmax(attn, dim=-1)
            # Return attention from CLS token to spectral patches
            return attn[:, 0, 1:]

Extracting Attention Maps

After a forward pass, map the patch-level attention weights back to the full spectrum:

def attention_to_wavenumber_importance(attention_weights,
                                       wavenumbers, patch_size):
    n_patches = len(attention_weights)
    importance = np.zeros(len(wavenumbers))
 
    for i in range(n_patches):
        start = i * patch_size
        end = start + patch_size
        if end <= len(wavenumbers):
            importance[start:end] = attention_weights[i].item()
 
    # Normalize to [0, 1]
    importance = (importance - importance.min()) / (
        importance.max() - importance.min() + 1e-8
    )
    return importance

Caveat: Attention Is Not Always Explanation

Attention weights show where the model looks, not necessarily what it uses for classification. Research has shown that attention distributions can be rearranged without changing predictions. For regulatory purposes, attention maps should be presented as supplementary evidence alongside SHAP or LIME, not as the sole explainability method.

Gradient-Based Methods

Gradient-based explanations compute the gradient of the model output with respect to the input spectrum. Large gradients indicate that small changes in that wavenumber region would significantly change the prediction - suggesting the model is sensitive to that region.

Integrated Gradients

Vanilla gradients are noisy and can saturate. Integrated gradients address this by accumulating gradients along a straight-line path from a baseline (typically a zero spectrum) to the actual input:

import torch
 
def integrated_gradients(model, spectrum, baseline=None,
                         n_steps=200, target_class=1):
    if baseline is None:
        baseline = torch.zeros_like(spectrum)
 
    # Create interpolation path
    alphas = torch.linspace(0, 1, n_steps).view(-1, 1, 1)
    interpolated = baseline + alphas * (spectrum - baseline)
    interpolated = interpolated.to(spectrum.device)
    interpolated.requires_grad_(True)
 
    # Forward pass on all interpolated inputs
    outputs = model(interpolated)
    target_scores = outputs[:, target_class]
 
    # Backward pass
    target_scores.sum().backward()
 
    # Integrate gradients along the path
    gradients = interpolated.grad  # (n_steps, 1, spectrum_length)
    avg_gradients = gradients.mean(dim=0)
 
    # Integrated gradients = (input - baseline) * average gradient
    ig = (spectrum - baseline) * avg_gradients
 
    return ig.detach().squeeze()

Saliency Maps

Saliency maps are the simplest gradient-based method - just the absolute value of the gradient of the output with respect to the input:

def saliency_map(model, spectrum, target_class=1):
    spectrum = spectrum.clone().requires_grad_(True)
    output = model(spectrum)
    target_score = output[0, target_class]
    target_score.backward()
 
    saliency = spectrum.grad.abs().squeeze()
    return saliency.detach()

Saliency maps are fast (one forward + backward pass) and useful for quick exploration during development. For regulatory submissions, integrated gradients are preferred because they satisfy the completeness axiom - the attributions sum to the difference between the output at the input and the output at the baseline.

Method Comparison

Each explainability method has different trade-offs. The right choice depends on your model architecture, your regulatory pathway, and your computational budget.

Recommendation for regulatory submissions: Use SHAP (KernelSHAP or TreeSHAP depending on your model) as the primary method. Include integrated gradients if your model is a neural network. Present LIME results as concordance evidence. Show attention maps as supplementary visualizations if applicable.

Building the Explainability Report for Regulatory Submission

An explainability report for an FDA marketing submission or EU AI Act technical documentation should contain five sections:

1. Global Feature Importance

Aggregate SHAP values across your validation dataset to show which wavenumber regions the model uses most:

def global_feature_importance(shap_values, wavenumbers):
    mean_abs_shap = np.mean(np.abs(shap_values), axis=0)
 
    top_k = 10
    top_indices = np.argsort(mean_abs_shap)[-top_k:][::-1]
 
    print("Top spectral features (by mean |SHAP|):")
    for idx in top_indices:
        print(f"  {wavenumbers[idx]:.0f} cm⁻¹: "
              f"mean |SHAP| = {mean_abs_shap[idx]:.4f}")
 
    return mean_abs_shap

The key question a reviewer will ask: do the important wavenumber regions correspond to known molecular vibrations relevant to the diagnostic target? If your model for tissue classification relies heavily on the amide I band (1650 cm⁻¹, protein secondary structure) and the CH₂ stretching region (2850-2920 cm⁻¹, lipid membrane composition), that is biologically plausible. If it relies on a region with no known biological significance, investigate whether it is modeling an instrument artifact or a preprocessing residual.

2. Per-Class Explanation

Show that the model uses different spectral features for different classes - this demonstrates that it has learned class-specific biochemical signatures rather than a generic pattern:

def per_class_shap_analysis(shap_values, labels, wavenumbers,
                            class_names):
    for cls_idx, cls_name in enumerate(class_names):
        mask = labels == cls_idx
        cls_shap = shap_values[mask]
        mean_shap = np.mean(cls_shap, axis=0)
 
        top_positive = np.argsort(mean_shap)[-5:][::-1]
        top_negative = np.argsort(mean_shap)[:5]
 
        print(f"\n{cls_name}:")
        print("  Regions pushing TOWARD this class:")
        for idx in top_positive:
            print(f"    {wavenumbers[idx]:.0f} cm⁻¹ "
                  f"(SHAP = {mean_shap[idx]:+.4f})")
        print("  Regions pushing AWAY from this class:")
        for idx in top_negative:
            print(f"    {wavenumbers[idx]:.0f} cm⁻¹ "
                  f"(SHAP = {mean_shap[idx]:+.4f})")

3. Representative Individual Explanations

Select 3-5 representative spectra from each class and provide per-sample SHAP plots. Include:

• Correctly classified samples (typical behavior)
• Incorrectly classified samples (failure modes)
• Borderline cases (near the decision boundary)

This demonstrates the model's behavior across the decision boundary.

4. Concordance with Domain Knowledge

Map the top SHAP features to known molecular assignments. For an FTIR tissue classifier:

This table is the most important component of the explainability report. It demonstrates that the model has learned biochemically meaningful features - not measurement artifacts, not noise patterns, not patient-specific correlations that will not generalize.

5. Failure Mode Analysis

Show what happens when the model encounters out-of-distribution spectra. Compute SHAP values for spectra that the model classifies with low confidence or incorrectly. If the SHAP values for failed predictions highlight different spectral regions than successful predictions, document this - it reveals the model's failure modes and supports the confidence threshold system described in our pipeline guide.

Practical Recommendations

Start with the simplest model that meets your performance requirements. A PLS-DA or Random Forest model with PCA features is inherently more interpretable than a CNN. PCA loadings tell you which wavenumber regions drive each component. VIP (Variable Importance in Projection) scores from PLS directly rank wavenumbers by diagnostic importance. If a linear model meets your sensitivity and specificity targets, the entire explainability report writes itself from the model's own parameters. Reserve deep learning - and the added explainability burden - for problems where classical methods genuinely fall short.

Generate explainability artifacts during development, not after. If you discover during the explainability analysis that your model relies on spectral regions with no biological relevance, you need to investigate whether it has learned an instrument artifact or a confounding variable. This discovery should happen during model development, when you can fix it - not during the regulatory submission, when fixing it means retraining, revalidating, and resubmitting. The SpectraDx platform integrates explainability analysis into the model deployment workflow, generating SHAP reports automatically alongside performance metrics.

Use multiple methods and check for agreement. If SHAP says the amide I band is the most important feature and integrated gradients agree, that is strong evidence. If they disagree substantially, investigate why - the model may be doing something more complex than either method reveals alone.

Document the preprocessing pipeline in your explainability report. Explainability methods explain the model's learned function. If your preprocessing includes a second derivative (which fundamentally changes what the input "looks like"), SHAP values will reference features in the derivative spectrum, not the raw spectrum. Reviewers need to understand the preprocessing chain to interpret the explainability results. For the full preprocessing picture, see our preprocessing comparison guide.

Consider transfer learning stability. If your model will be deployed across multiple instruments, verify that the explainability profiles are consistent across instruments. If the model uses different spectral features on Instrument A vs. Instrument B, it may be learning instrument-specific artifacts rather than sample chemistry - a problem that transfer learning and domain adaptation can address.

Further Reading

• Building AI Pipelines for Spectral Classification - end-to-end ML pipeline for spectral diagnostics
• Spectral Preprocessing for Clinical ML Models - the preprocessing step that determines model performance
• Transfer Learning for Spectral Models - solving cross-instrument generalization
• SaMD Classification for Spectroscopy Software - regulatory framework for AI-based diagnostics
• EU IVDR and the AI Act - dual compliance for European market
• IEC 62304 for Spectroscopy Software - software lifecycle standard for medical devices