<html>
<body>
<font size=3>Hi Luis, Tom, others,<br><br>
Related to this:<br><br>
I have written a optimizer base class that handles the scaling
functionality of optimizers. By letting all itk optimizers inherit from
this class, the scales are dealt with in exactly the same manner. In
fact, the code is similar to the
itkSingleValuedVnlCostFunctionAdaptor.cxx mechanism, including the
proposed modification of Tom.<br><br>
You may have a look at the doxygen documentation:<br><br>
<a href="http://elastix.isi.uu.nl/doxygen/a00554.html" eudora="autourl">http://elastix.isi.uu.nl/doxygen/a00554.html</a><br>
which describes the ScaledCostFunction class<br><br>
<a href="http://elastix.isi.uu.nl/doxygen/a00555.html" eudora="autourl">http://elastix.isi.uu.nl/doxygen/a00555.html</a><br>
which described the ScaledOptimizer class<br><br>
I've programmed quite some itk::optimizers that inherit from this base
class. On the site mentioned above you can find links to the inheriting
optimizers, including a quasi-Newton LBFGS optimizer (which does exactly
the same as the current vnl-based-version, but is written entirely in
ITK-style) and a conjugate gradient optimizer, which implements various
types of conjugate gradient (polakribier, fletcher reeves, dai-yuan etc).
Also a derivative-free evolution-strategy-based optimizer (CMA-ES) is
implemented. <br><br>
It is part of a larger piece of software for image registration. You
could inspect the source code at
<a href="http://elastix.isi.uu.nl/download.php" eudora="autourl">http://elastix.isi.uu.nl/download.php</a>
. <br><br>
Kind regards,<br>
Stefan.<br><br>
<br>
At 12:32 24-4-2008, you wrote:<br><br>
<blockquote type=cite class=cite cite>Hi Tom,<br><br>
Thanks for pointing this out.<br><br>
You are right, the current code doesn't perform the correct<br>
mathematical operation that corresponds to scaling the argument<br>
of a cost function.<br><br>
<br>
However, that was not the intent of introducing the scaling<br>
in these classes.<br><br>
<br>
The motivation for this scaling operation was to extend the<br>
functionality of VNL optimizers to be usable with domains outside<br>
of the cannonical [-1:+1] domain.<br><br>
<br>
Since these functions are used for the purpose of optimization,<br>
we intended simply to shrink (or expand) the domain of the original<br>
problem in order to map to the [-1:+1] that the algorithmic<br>
implementation of vnl optimizers expect.<br><br>
To make sense of it, you may find interesting to look at both the<br>
ITK wrappers for VNL optimizer, and the ITK wrapper for the cost<br>
function.<br><br>
<br>
Take for example,<br>
the itkAmoebaOptimizer.cxx:<br><br>
it its call to GetValue() in line 84, it uses the scales<br>
to multiply the parameters *before* they are passes to<br>
the cost function.<br><br>
84: parameters[i] *= scales[i];<br><br>
In its call to StartOptimization(), before invoking the<br>
VNL optimizer, we scale the parameters of the transform<br><br>
218: parameters[i] *= scales[i];<br><br>
then we invoke the VNL optimizer<br><br>
227: m_VnlOptimizer->minimize( parameters );<br><br>
and then we restore the scale of the transform parameters<br><br>
242: parameters[i] /= scales[i];<br><br>
<br>
The goal, is to create a situation where the VNL optimizer<br>
"thinks" that is optimizing a function whose domain is close<br>
to [-1:+1], as VNL algorithms expect or assume.<br><br>
<br>
However, the VNL optimizer is calling the cost function<br>
in order to get values from it. That cost function will<br>
happen to be an ITK ImageMetric (most of the time), and<br>
therefore the VNL cost function adaptor must "restore"<br>
the scale of the parametric space to what the metric<br>
expects as parameters. That's why the cost function adaptor<br><br>
itkSingleValuedVnlCostFunctionAdaptor.cxx<br><br>
in its call to "f()" rescales the parameters *back* by using<br>
in line 61:<br><br>
61: parameters[i] = inparameters[i]/m_Scales[i];<br><br>
and then calling the actual ITK cost function in line<br><br>
69: m_CostFunction->GetValue( parameters ));<br><br>
<br>
In this way, ITK metrics will still work in a space where<br>
the domain of translations may be [-500mm, +500mm] and<br>
[-0.1degrees,+0.1degrees], while the VNL optimizer works<br>
in a space with a domain close to [-1:+1].<br><br>
<br>
The point is well taken in that we could have applied the<br>
scaling to the resulting values of the gradients. Something<br>
that could conveniently be done in the method:<br><br>
ConvertExternalToInternalGradient()<br><br>
in line 176 of itkSingleValuedVnlCostFunctionAdaptor.cxx<br><br>
<br>
This additional (and mathematically correct scaling) will<br>
indicate the following:<br><br>
<br>
A) if we have an image of 100x100 pixels with spacing<br>
2.0mm x 2.0mm and we are optimizing the Mean square<br>
metric against another image of similar dimensions,<br>
we see this problem happening in a domain of<br>
[200x200]mm. If the Metric changes by 35 intensity<br>
values in average when you translate the images by<br>
5 millimeters, that corresponds to a derivative of<br>
<br>
7 iv / mm<br><br>
<br>
B) The scaling conversion will be equivalent to replace<br>
the units of mm with units of "image width". That is<br>
somehow normalizing the space to the size of the image.<br><br>
In this context, the conversion<br><br>
F(x) = f(x/s)<br><br>
grad(F(x)) = grad(f(x/s))/s<br><br>
will give us the value of the gradient on the units<br>
of "image width", as:<br><br>
(7/200) iv / iw<br><br>
<br><br>
At this point, we could make this correction,<br>
but this may require to recalibrate the parameters<br>
of most of image registration examples and tests<br>
that use VNL optimizers...<br><br>
<br>
Have you run an experimental build with this change<br>
to evaluate the impact of the modification ?<br><br>
<br>
----<br><br>
<br>
The operation is not done in the same way either across optimizers.<br><br>
For example,<br>
if you look at the GradientDescent and RegularStepGradientDescent<br>
optimizers, the scales are applied only to the *resulting* gradient<br>
vector returned from the cost function, and then, the resulting vector<br>
is used during the computation of the step to be taken. In these two<br>
optimizers, the scale factor is not used for feeding the values<br>
inside the cost function.<br><br>
<br><br>
Regards,<br><br>
<br>
Luis<br><br>
<br>
-----------------------<br>
Tom Vercauteren wrote:<br>
<blockquote type=cite class=cite cite>Hi all,<br>
I think I found a bug in<br>
SingleValuedVnlCostFunctionAdaptor::ConvertExternalToInternalGradient.<br>
Here is a brief description:<br>
SingleValuedVnlCostFunctionAdaptor can be used to wrap an itk cost<br>
function into a vnl cost function.<br>
Additionally the user can specify a set of scales such that the<br>
function to be optimized is F(x) = f(x./s) instead of f(x).<br>
The gradient of F, grad(F)(x) is thus given by grad(F)(x) = grad(f)(x./s)./s<br>
Instead of that, the cost function adpator returns grad(f)(x./s)<br><br>
I have filed a bug report in the bug tracker:<br>
<a href="http://www.itk.org/Bug/view.php?id=6887" eudora="autourl">http://www.itk.org/Bug/view.php?id=6887</a><br>
Let me know if I am missing something.<br>
Best regards,<br>
Tom Vercauteren<br>
_______________________________________________<br>
Insight-users mailing list<br>
Insight-users@itk.org<br>
<a href="http://www.itk.org/mailman/listinfo/insight-users" eudora="autourl">http://www.itk.org/mailman/listinfo/insight-users</a><br>
</blockquote>_______________________________________________<br>
Insight-users mailing list<br>
Insight-users@itk.org<br>
<a href="http://www.itk.org/mailman/listinfo/insight-users" eudora="autourl">http://www.itk.org/mailman/listinfo/insight-users</a></font></blockquote></body>
<br>
</html>