Evaluation on the test set - CosmoSim Experiments July 2026

This document evaluates one trained machine learning model, see details below. We use both conventional metrics and comparison with a resimulated image.

We use three datafiles, which must be downloaded if this document is to be executed.

cluster2-testing.csv is ground truth for model training
pred-cluster2-testing.csv is the predicted amplitudes from machine learning.
cluster2-dataset.csv is the original lens parameters used to generate the training, testing, and validation data, i.e. it has more rows than the other two sets.

The neural network used for machine learning is almost arbitrarily chosen and has not been tuned. The architecture is one of the best performing in Nicolò’s experiments on other datasets from CosmoSim, but the hyperparameters are arbitrarily chosen. The training set used is 16000 images.

The specification of the distribution is discussed in Cluster Dataset (2x SIE) and can be downloaded (cluster2-dataset.toml).

Configuration¶

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from PIL import Image
import IPython.display as idp
import toml
import json
import CosmoSim.Image as csimg
import CosmoSim as cs
import CosmoSim.datagen as dg
import CosmoSim.roulettegen as rg
imshow = csimg.imshow

Let us check the parameters underlying this test. This gives the neural network and hyperparameters used.

cfg = toml.load( "ml.toml" )
print( "CosmoSim version is", cs.__version__ )
print( "Data from directory", cfg["output"]["directory"] )
print( "Neural network used:", cfg["settings"]["model"] )
print( json.dumps( cfg["hyperparameters"], indent=4 ) )

CosmoSim version is 3.3.0
Data from directory experiment002
Neural network used: regnet_y_32gf
{
    "learning-rate": 0.001,
    "batch-size": 32,
    "weight-decay": 0.0001
}

Let’s load the test set, both the ground truth (gt) and the predictions (df).

gt = pd.read_csv( "../cluster2-testing.csv", index_col="filename" )
df = pd.read_csv( "pred-cluster2-testing.csv", index_col="filename" )
display( gt.head() )
display( df.head() )

As we can see, we have columns for all the parameters in question, and rows for the individual objects. There is no superfluous data in the data frames.

We can also check the difference.

rawerrors = gt - df
display( rawerrors.head() )

That was straight forward. Now we come to the tricky part - making sense of the numbers. There are two questions that we want to consider.

How well does the model perform overall?
How do particular images behave, and what are the best and the worst samples?

Model evaluation¶

It is interesting to see how the model behaves on each column. To see this, we can aggregate the rows, computing the mean of absolute errors and mean of squared errors.

mse = (rawerrors**2).mean()
mae = rawerrors.abs().mean()
mse.name = "MSE"
mae.name = "MAE"
colerrors = pd.concat( [ mse, mae ], axis=1 )
display( colerrors )

In practice, relative errors may be more interesting. We can calculate that too.

colerrors["mean"] = gt.abs().mean()
colerrors["Relative MAE"] = colerrors["MAE"] / colerrors["mean"]
display( colerrors )

This looks good with mean relative errors less than 10^-8, and no column is particularly bad or good.

Object evaluation¶

To investigate individual objects, we start by calculating sum of squared errors for each one.

sse = (rawerrors**2).sum(axis=1)
display(sse)

filename
image-010001.png    8.097897e-12
image-010002.png    4.178766e-11
image-010003.png    1.062804e-10
image-010004.png    1.588092e-10
image-010005.png    5.551587e-13
                        ...     
image-013996.png    6.644094e-12
image-013997.png    2.056408e-11
image-013998.png    3.961108e-11
image-013999.png    1.588975e-11
image-014000.png    4.960676e-12
Length: 4000, dtype: float64

We can look at the three best and three worst images, thus,

sse.nlargest(3)

filename
image-012584.png    3.787224e-10
image-011983.png    3.616541e-10
image-010282.png    3.381361e-10
dtype: float64

sse.nsmallest(3)

filename
image-010628.png    4.911157e-14
image-011329.png    6.545188e-14
image-013847.png    6.834784e-14
dtype: float64

We note that the errors are small, but there is also a huge span between the best and the worst. For the purpose of this test, we do not assume that we have access to the original images, but we can resimulate them from the roulette amplitudes. First we record the filenames.

best = list(sse.nlargest(3).index)
worst = list(sse.nsmallest(3).index)
print( "Best:", best )
print( "Worst:", worst )

Best: ['image-012584.png', 'image-011983.png', 'image-010282.png']
Worst: ['image-010628.png', 'image-011329.png', 'image-013847.png']

Simulation from Roulette Amplitudes¶

Before we can simulate, we need to set up some basic parameters.

cfg = { "simulator" : { "imagesize" : 512, "cropsize" : 256, "xireference" : True }
      , "source" : { "mode" : "SersicSphere" } }
param = cs.Parameters( cfg )

Lets first consider the worst images and the ground truth. The data is extracted as

gtw = gt.loc[ worst ]
display( gtw )

To simulate, we instantiate the Resim class which is the simulator for roulette resimulation. For now, we test in just one image.

imsim = rg.Resim(gtw.iloc[0],param=param,verbose=0)
im = imsim.getImage()
imshow( im )

[getSource] src=SersicSphere, ltprf0=None, verbose=1
[SphericalSource] constructor done

We can make the same simulation from the reconstructed amplitudes and compare.

dfw = df.loc[ worst ]
imsim2 = rg.Resim(dfw.iloc[0],param=param,verbose=0)
im2 = imsim2.getImage()
csimg.imageCompare( im, im2, "Ground Truth", "Reconstructed" )

[getSource] src=SersicSphere, ltprf0=None, verbose=1
[SphericalSource] constructor done

This looks like perfect match.

We can do the same for all the top and bottom three.

for fn in worst:
    dfsim = rg.Resim(df.loc[fn],param=param,verbose=0)
    dfim = dfsim.getImage()
    gtsim = rg.Resim(gt.loc[fn],param=param,verbose=0)
    gtim = gtsim.getImage()
    csimg.imageCompare( dfim, gtim, fn, "Ground Truth", axiscross=True )

[getSource] src=SersicSphere, ltprf0=None, verbose=1
[SphericalSource] constructor done

[getSource] src=SersicSphere, ltprf0=None, verbose=1
[SphericalSource] constructor done

[getSource] src=SersicSphere, ltprf0=None, verbose=1
[SphericalSource] constructor done

[getSource] src=SersicSphere, ltprf0=None, verbose=1
[SphericalSource] constructor done

[getSource] src=SersicSphere, ltprf0=None, verbose=1
[SphericalSource] constructor done

[getSource] src=SersicSphere, ltprf0=None, verbose=1
[SphericalSource] constructor done

No visible discrepancy. We can continue with the best images, obviously expecting perfect match again.

for fn in best:
    dfsim = rg.Resim(df.loc[fn],param=param,verbose=0)
    dfim = dfsim.getImage()
    gtsim = rg.Resim(gt.loc[fn],param=param,verbose=0)
    gtim = gtsim.getImage()
    csimg.imageCompare( dfim, gtim, "Reconstructed", "Ground Truth" )

[getSource] src=SersicSphere, ltprf0=None, verbose=1
[SphericalSource] constructor done

[getSource] src=SersicSphere, ltprf0=None, verbose=1
[SphericalSource] constructor done

[getSource] src=SersicSphere, ltprf0=None, verbose=1
[SphericalSource] constructor done

[getSource] src=SersicSphere, ltprf0=None, verbose=1
[SphericalSource] constructor done

[getSource] src=SersicSphere, ltprf0=None, verbose=1
[SphericalSource] constructor done

[getSource] src=SersicSphere, ltprf0=None, verbose=1
[SphericalSource] constructor done

Simulations from Lens Parameters¶

We can also load the original lens parameters, from which the ground truth was computed.

orig = pd.read_csv( "../cluster2-dataset.csv", index_col="filename" )
display( orig.head() )

This dataset is different from the others, giving physical parameters of the lens instead of roulette amplitudes in a point of observation. Thus, we need a different simulator. On a positive note, we can use raytrace simulation, which is accurate.

The SimImage simulator is parameterised in a slightly different way. We need to add the row data from the dataset to the Parameters object instead of passing it as a separate argument. To avoid interference, we make a copy of params.

cfg["simulator"]["model"] = "Raytrace"
cfg["simulator"]["centred"] = True
cfg["lens"] = { "mode" : "SIE" }
p2 = cs.Parameters( cfg )
for fn in worst:
    dfsim = rg.Resim(df.loc[fn],param=param,verbose=0)
    dfim = dfsim.getImage()
    p2.setRow( orig.loc[fn] )
    p2["simulator"]["centred"] = True
    gtsim = dg.SimImage(param=p2,verbose=0)
    gtim = gtsim.getImage()
    csimg.imageCompare( dfim, gtim, fn, "Original raytrace simulation", axiscross=True )

[getSource] src=SersicSphere, ltprf0=None, verbose=1
[SphericalSource] constructor done

[getSource] src=SersicSphere, ltprf0=None, verbose=1
[SphericalSource] constructor done

[getSource] src=SersicSphere, ltprf0=None, verbose=1
[SphericalSource] constructor done

The match is not perfect, but it may be no more than the numeric inaccuracy due to post-processing centring the image for machine learning. as we discussed in the report from the SIE experiment.

for fn in best:
    dfsim = rg.Resim(df.loc[fn],param=param,verbose=0)
    dfim = dfsim.getImage()
    p2.setRow( orig.loc[fn] )
    gtsim = dg.SimImage(param=p2,verbose=0)
    gtim = gtsim.getImage()
    csimg.imageCompare( dfim, gtim, fn, "Original raytrace simulation", axiscross=True )

[getSource] src=SersicSphere, ltprf0=None, verbose=1
[SphericalSource] constructor done

[getSource] src=SersicSphere, ltprf0=None, verbose=1
[SphericalSource] constructor done

[getSource] src=SersicSphere, ltprf0=None, verbose=1
[SphericalSource] constructor done

Now, this looks bad.Let’s quickly check that the primary image is within the frame.

for fn in best:
    t = df.loc[fn] 
    print( fn, ( t["xiX"], t["xiY"] ) )

image-012584.png (np.float64(70.07575), np.float64(-130.56299))
image-011983.png (np.float64(-63.767246), np.float64(-12.143284))
image-010282.png (np.float64(-94.56414), np.float64(-136.30283))

The frame ranges between $\pm128$ , so the first and the last items are actually drawn outside, which may explain the error. What we can see in the image may well be secondary images, outside the convergence ring as well as spurious iamges.

The middle image is harder to explain. The image is correct in position and overall shape, but the discrepancy is a bit too much to attribute to numerical error without investigation. We remember that the resimulation matches the original roulette simulation too. It may be useful to enhance the contrast, by boosting the light intensity of the image.

fn = best[1]
dfim = rg.Resim(df.loc[fn],param=param,verbose=0).getImage()
p2.setRow( orig.loc[fn] )
gtim = dg.SimImage(param=p2,verbose=0).getImage()
csimg.imageCompare( 3*dfim, 3*gtim, fn, "Original raytrace simulation", axiscross=True )

[getSource] src=SersicSphere, ltprf0=None, verbose=1
[SphericalSource] constructor done

This shows better the effect of the spurious image in the lower right quarter, and how the primary image is elongated towards the spurious image in the roulette formalism. Taking this effect together with the numerical error from centring the image, may explain the discrepancy.

Conclusion¶

There are several problems with this experiment. The dataset creates images outside the frame, so we should make more care in designing the distribution. We also observe artifacts which we cannot fully explain. Both post-processing effects and roulette artifacts can explain a lot, but not necessarily all of it. This requires further research.