[Insight-users] Regular Step Gradient Descent optimizer

[Luis Ibanez]

Hi Paul,


Well, it seems that there is enough evidence out there for justifying
a modification of the RegularStepGradientDescent optimizer, and offering
the option of replacing the "change of direction" criteria with a
"change of function value" criteria.


In order maintain backward compatibility we will have to let the current
behavior be the default one, and add the new behavior as an option that
the user will have to turn on.

I still agree with Kevin, that a gradient descent "has the right" of
increasing the value of the function, but only at one iteration. This
will be the natural occurrence of "passing over" the minimum and
starting to go up-hill at the other side of the minimum.  After that
the optimizer should descend again...

If the optimizer going up-hill for more than one iteration, then it is
definitely in a region of the cost-function that doesn't match our
simplistic assumptions about the heuristic of relaxing the step lenght
based on changes of direction.

Ming: could you give it a try to use the modified version of
       the optimizer that Paul and Tolga posted to the list ?

       and let us know if this behaves better in your context ?


We can then add this behavior as an option to the optimizer.

A secon alternative is to put a new optimizer altogether... but
it seems that there is enough common code here that an extra
option in the optimizer sounds like a better idea than a full
new optimizers.



    Thanks


        Luis




=========================
Paul Koshevoy wrote:
> Kevin H. Hobbs wrote:
> 
>>On Wed, 2006-05-03 at 10:34 -0700, Ming Chao wrote:
>>  
>>
>>>Hi Luis,
>>> 
>>>Actually my original question was quite simple. I didn't intend to
>>>compare A with B to decide which one is better. What concerned me is
>>>why when I used itkRegularStepGradientDescentOptimizer the metric
>>>values first decreased and then increased (isn't this abnormal?).
>>>While using LBFGSBOptimizer the metric values kept decreasing and
>>>stopped when the optimizer thought it reached mimimum. Unless I missed
>>>something, I haven't understood why.
>>> 
>>>Cheers,
>>>Ming
>>> 
>>>
>>>    
>>
>>I don't think this is so odd. Try making the observer print the size of
>>the optimizer, if it has one.  I think what you're seeing is that the
>>optimizer finds a minimum and then looks around that minimum. The points
>>around the minimum will be worse than the minimum, duh, so you'll see an
>>increase. The size of the steps will be shrinking though. The last value
>>that the observer spits out is not necessarily the best one.  In fact I
>>think it must not be...
>>
>>At least I've seen this before...
>>  
> 
> I've seen itk::RegularStepGradientDescentOptimizer veer  away wildly
> from a minima on many occasions, and not at the last iteration either,
> which is what prompted Tolga to investigate the relaxation criteria.
> 
> The way it is implemented now, it is possible to jump around various
> local minima basins -- so long as the derivatives point in the same
> direction, the optimizer assumes it is going down the same hill, which
> is not necessarily true. There may be two hills near by, and if the step
> size is big enough you could jump from the slope of one hill to the
> slope of the next hill and the derivative direction may remain the same,
> but the function value may have worsened. This is a scenario where the
> current optimizer fails. A robust optimizer would detect the change in
> function value instead of the change in derivative direction.
> 
>     Paul.
> 
> 
>