[Insight-developers] REPRODUCIBILITY: IEEE CVPR 2010, Moved to the Bright Side !!!

[David Doria]

On Sun, Nov 22, 2009 at 3:21 PM, Torsten Rohlfing
<torsten at synapse.sri.com> wrote:
> Luis:
>
> Slight disagreement here again ;)
>
> Reproducibility alone does not cure cancer. If we all simply keep repeating
> and re-confirming each others results, there is no innovation.
>
> As far as papers are concerned, I would say we need to distinguish between
> "reproducible" (which is good) and "reproduction" (which, unless the paper
> is concerned with something very hard to reproduce, is not).
>
> I agree, though, that "originality" alone is also not an important measure
> for a paper. Claiming that I can cure cancer by sticking a carrot up the
> patient's nose and blowing in his ear is mighty original, but hardly
> effective. Probably not reproducible, either.
>
> In the end, I'd say it all comes down to "significance." Something long
> known, although easily reproducible, is simply not significant. Likewise,
> something very new and surprising is not significant if it cannot be
> reproduced. Both novelty and reproducibility are required to make a work of
> science significant.
>
> Have a nice weekend!
>  TR


Torsten,

My main complaint about lack of reproducibility is not from such a
philosophical standpoint, but more of a practical one. If you write a
paper about a method to cure cancer - assuming we believe it works
(which we should now be able to verify ourselves since it is required
for it to be reproducible), my task is now to develop something that
either cures it faster, or with less side effects, etc. If I can use
your work directly as a building block, I can spend 100% of my time on
the new task at hand. If you do not provide the means to easily and
quickly reproduce your results, then I have to spend 80% of my time
re-doing what you have done (what I would call "wasting time") and
then only 20% of my time gets directed to the new, original problem.

So I agree that both novelty and reproducibility are required, but my
claim is that reproducibility (in fact, the means to actual
reproduction) directly enables time and effort to be spent on solving
new/different problems rather than everyone continuously reinventing
the wheel.

Thanks,

David