When Your Metrics Disagree: Evaluating Saliency Models on ASD and TD Fixation Data

I spent a few months comparing three saliency models against fixation maps recorded from two groups of viewers: autistic (ASD) participants and typically developing (TD) participants. The models were FES, a classical hand-designed feature model; SalFBNet, a feedback-recurrent convolutional network; and TranSalNet, a transformer-augmented encoder-decoder. I scored all of them with four metrics that appear together in nearly every saliency paper: AUC-Borji, correlation coefficient (CC), KL divergence, and normalized scanpath saliency (NSS).

The models did not rank the same way under all four. I expected some jitter. What I got was a stable, reproducible inversion: a model that led on one metric trailed on another, and re-running the pipeline did not fix it, because there was nothing to fix.

That is the thing worth writing down. These four metrics are not four noisy estimators of one hidden quantity called “saliency quality.” They measure different properties of a predicted map. When they disagree, at least one of them is usually telling you something true that the others are structurally unable to see.

What each metric actually measures

It helps to be precise here, because the differences are not cosmetic.

AUC-Borji is location-based. It treats the saliency map as a binary classifier over pixel locations and sweeps a threshold to trace an ROC curve. The positives are the fixated locations for the image being scored. The negatives are the distinguishing part: they are sampled from fixation locations recorded on other images, not uniformly from the image plane. That sampling scheme is deliberate. Human fixations cluster near the center of an image regardless of content, so uniformly sampled negatives would hand free credit to any model that outputs a center blob. Drawing negatives from the fixation distribution of other images gives the negatives the same center bias as the positives, and the metric then asks whether the model beats that prior.

The consequence people forget: AUC is invariant to any monotonic transform of the saliency map. Take a map, apply a strictly increasing function to every value, and the AUC is unchanged. Only the ordering of pixel values matters. The map’s calibration, its scale, its confidence, all of it is discarded. Two maps with wildly different distributions of predicted density can score identically.

This is also why AUC saturates. Once a model gets the coarse ranking of regions right, it lands in a narrow band near the ceiling, and remaining models pile into that band. Differences that look small on AUC can be large under a metric that reads magnitude.

CC is the Pearson correlation coefficient between the predicted map and the ground-truth fixation density, both treated as distributions over pixels. It is bounded in the range from -1 to 1, and it is symmetric in its two arguments: swap prediction and ground truth and you get the same number. Symmetry means it charges the same price for a false positive as for a false negative. Putting density where there are no fixations costs the same as failing to put density where there are.

That symmetry is a real design choice, not a neutral default. For some questions it is exactly right. For others it is exactly wrong.

KL divergence is also distribution-based, but it is asymmetric, and the asymmetry is the whole story. In the direction normally used for saliency, KL heavily penalizes the model for assigning near-zero density at locations where the ground truth has fixations. Missing a fixated region is expensive. Spreading extra density over unfixated regions is comparatively cheap.

# The term that dominates: ground-truth mass at locations
# where the prediction has almost nothing.
kl = np.sum(gt * np.log(eps + gt / (eps + pred)))

Look at that ratio. As pred approaches zero at a location where gt is nonzero, the term blows up. Which means KL is extraordinarily sensitive to how you regularize the map. The epsilon you pick, whether you add a small uniform floor, how much Gaussian blur you apply, all of that moves the score. A model that outputs sharp, confident, near-zero-elsewhere maps gets punished by KL for exactly the property NSS rewards.

NSS is location-based. Z-normalize the saliency map to zero mean and unit variance across the whole image, then take the mean of the normalized values at the fixated points. That normalization step means NSS is sensitive to the map’s variance, not just its peaks. A map with high response at fixations and high response everywhere else scores poorly, because normalizing shrinks the peaks relative to the elevated background. NSS punishes false positives in a way AUC does not, and rewards peaked confident maps in a way KL does not.

Why disagreement is legitimate

Line those four up and the pattern is clear. AUC ignores magnitude entirely. NSS depends on magnitude through the normalization. CC weights the two error types equally. KL weights them very unequally.

So a model can win on AUC and lose on KL without either metric being broken. Suppose it ranks regions correctly, so AUC is high, but it drives density to near-zero outside its top predictions. Any fixation cluster the model missed now sits in a near-zero region, and KL charges enormously for it. Both numbers are correct. They are answering different questions.

Two preprocessing choices move rankings more than most people report:

Blur. Convolving the map with a Gaussian shifts every score, and not in the same direction. Blurring raises KL performance by filling in near-zero regions, and it usually lowers NSS by flattening peaks relative to the map variance. There is a blur sigma that maximizes each metric, and those optima are not the same sigma. If two papers report different sigmas, or report none, their numbers are not comparable. This is quietly one of the largest sources of apparent disagreement in the saliency literature.

Center bias. Human fixations concentrate centrally. A model that has absorbed a center prior scores well on metrics that do not correct for it. AUC-Borji corrects through its negative sampling. CC, KL, and NSS do not, at least not directly. So the center-bias question is baked into the metric choice before you have looked at a single model.

Choosing a metric is choosing a definition of correctness. That sentence sounds like a platitude until you have watched a ranking flip on it.

The ASD and TD comparison compounds all of this

Now add the group dimension. Published work on autistic viewers reports differences in fixation distribution: reduced attention to faces and social cues, and different center-bias behaviour compared with typically developing viewers. Treat those as reported findings in the literature, not as something my study established.

Saliency models are trained on fixation data pooled from typically developing viewers. That is what the large public datasets contain. So a model’s learned prior encodes one group’s attention as the target. When you evaluate that model against ASD fixation maps, you are measuring transfer, and every metric-choice problem above now interacts with a group difference.

Concretely: if ASD fixation maps have a different center-bias profile, then the metrics that do not correct for center bias will shift between groups for reasons that have nothing to do with model quality. If ASD fixations land more often on regions a face-sensitive model assigns low density to, KL will punish that model hard on the ASD group while AUC barely moves. The between-group gap you measure is partly a property of the metric, and separating those contributions is not trivial.

Underneath is an assumption I think should be said out loud: a model trained on TD fixation data treats TD attention as normative. Every “error” it makes on ASD data is defined relative to that baseline. That framing is a choice. It is a defensible one for some engineering purposes and a poor one for others, but it should be visible rather than buried in the dataset.

What I would actually recommend

Report multiple metrics, and report them at the level of the individual metric rather than as an average. Averaging AUC and KL into a composite score destroys the only information the disagreement was carrying. If you need one number to sort a table, say which metric it is and why that one.

Report preprocessing in full. Blur sigma, normalization scheme, epsilon or density floor, whether maps were histogram-matched, what resolution the maps were scored at. Without those, a saliency comparison is not reproducible, and the differences between published numbers may be entirely procedural.

When metrics disagree, look at the maps. Every inversion I saw had a visible explanation once I put the predicted map next to the group fixation map. The number told me something changed; the image told me what.

Never rank models by a single scalar. If the ranking is not stable across metrics, that instability is your result, and reporting only the metric that produced your preferred ordering is a way of hiding it.

Limits

This was a model-comparison study on existing datasets. I did not build a new model, did not retrain or fine-tune anything on group-specific fixation data, and did not collect new eye-tracking data. The datasets are small, and group-level fixation maps average over substantial variance between individuals, so any difference I measured is a difference in aggregate and says nothing about a particular viewer.

It does not support any diagnostic use. Nothing in this work indicates that saliency-model performance can identify, screen for, or characterize autism in any individual, and it should not be cited or built on as if it does. The question I asked was narrow: do standard metrics agree about how these models transfer across two group-level fixation distributions? They do not. That is the whole finding.