[Insight-developers] problm with vnl optimizers and scales

[Zachary Pincus]

Hello all,

I think I've found a minor problem with how the VNL optimizer wrappers 
deal with user-provided scales. Specifically, they do not adjust the 
starting parameter values to take account of the scale factors, but 
they adjust everything else. This causes the starting parameter values 
to become distorted, which poses problems in cases where the starting 
values are important. (Say, when using large scales to hold a parameter 
near it's starting value, or when using hill-climbing.)

Here's how things work, and what's wrong (as far as I can tell):

(1) VNL wrapper optimizers use a "cost function adapter" which scales 
parameters by dividing them by the scale. So if the vnl optimizer asks 
for the value at (10, 10), and the scales are (10, 2), then the cost 
function adapter returns the value at (1, 5). So if the initial 
position is at (0, 0), the real optimum is at (2, 1), and the scales 
are set to (1000, 1000), the vnl optimizer would have to move from 
(0,0) to (2000, 1000) to find that optimum. This is how setting a large 
scale in one axis can slow the optimizer's progress in that axis.

(2) The VNL wrapper optimizers divide the final parameter values they 
get from the optimizer by the scales before returning them to the user. 
This takes the (2000, 1000) from the previous example and sets it back 
to the expected (2, 1).

(3) Here is the problem: The VNL wrapper optimizers DO NOT scale up the 
initial parameters. Suppose that in the previous example, the initial 
position is set precisely to the maximum (2, 1). So the optimizer 
shouldn't have to move at all! However, currently, the initial position 
is passed un-scaled to the vnl optimizer. So the VNL optimizer starts 
at (2, 1) and has to traverse to (2000, 1000). The right thing to do is 
quite clearly to multiply the initial parameters by the scale.

Do people agree that this is a problem? If so, I'll fix the vnl 
optimizer wrappers to deal with it.

Zach