Sentinel-2 image registration benchmark Demo

Two images of the same place never line up exactly.

Off by a few pixels, a computer comparing them sees change where nothing changed. Registration measures that offset and undoes it. Below, four methods try it on four places. Drag the divider: magenta marks the false changes, before correction on the left and after on the right.

Site
Conditions
Method

Before and after correction

Show
Zoom

One pair shows a method at work. It does not measure it: the measurements, on thousands of pairs, are in the evaluation.

When the offset changes across the image

Here the same window is warped instead of shifted, so each point moves by its own amount. An arrow shows where a point of the reference is found in the moving image, one every 16 px. The first picture is the truth, the second the method's estimate, the third the difference between the two, magnified: the shorter its arrows, the better.

Error arrows drawn at

What accuracy costs

RAFT, the neural network of this benchmark, corrects its own estimate several times in one call: these are its refinements. Fewer refinements make a call shorter and its estimate less accurate. Each point is one RAFT model with 4, 8 or 12 refinements, and the number beside it says which.

None of this was measured on the pairs above. It was measured on the 1,800 pairs of the two held-out sites, a mountain site and a desert site whose landscape types are in no training image.

Time read on

  • RAFT large
  • RAFT small
  • fine-tuned: filled, full line
  • pretrained: hollow, dashed line
  • int8 copy of RAFT large, fine-tuned: the same model computing on 8-bit integers
  • classic method

No point is marked as the one to keep. These are held-out pairs, and no choice may be made on their figures.

How these figures were read

The error is the mean distance between the estimated and the true displacement, over the pairs that did not fail. A pair fails when the method gives no estimate or is more than 1 px off on average, so a point with many failed pairs has its error read on the easier ones: the chart of failed pairs is there for that.

Time is the median time of one call on a 256 × 256 px pair. Memory is what calling the method adds to a process that does nothing else: a lower bound, and the memory of SIFT + RANSAC and of AROSICS had not settled when it was read.

The same figures as a table
Error, failed pairs, time and memory of each point, as the curve files hold them. An empty cell is a figure the files do not hold: a classic method has no refinements, and neither it nor an int8 copy was timed on the T4.
Method Refinements Error (px) Failed pairs Time on the M1 (ms) Memory added on the M1 (MiB) Time on the T4 (ms)

Read this before you quote a figure

This page shows. It measures nothing.
One pair per site says nothing of what a method does on average: a method can fail here and hold on most pairs, or the reverse. No figure on this page enters a table. Rates and means are read in the evaluation, on thousands of pairs.
A rule picks the demo pairs, not an eye.
The benchmark draws each site's pairs from a fixed seed, the same pairs in the same order on every run. For each site, the demo pair is the first of them whose true shift is over 2 px: under that, a slider has nothing to show. The rule reads the true shift only, so no method runs to pick a pair. The threshold was set after the true shifts of the first pairs were seen, and before any method's result on them was looked at.
The four sites are seen sites.
Their pixels were never trained on, but their landscape types were. Each demo pair is also a pair of the published tables, so the page shows a pair the tables already count and adds nothing to them.
The cost curve comes from other pairs.
It was measured on the 1,800 pairs of the two held-out sites, whose landscape types are in no training image. Its figures are the only held-out figures on this page.
Everything was computed ahead.
One command, satreg demo, wrote every file this page reads: the images, the changed pixels, the arrows. The curve files are copies of the published ones. Your browser draws them and runs no model.