<div dir="ltr">Matt,<div>The tree map visualization looks great! A great tool to pick out long running tests (if the fonts of the label could be scaled accordingly, that would be even better) !</div><div><br><div>Looks like the code coverage loss is not much. </div>
<div> +1 on your patches.</div><div><br></div><div>-Xiaoxiao<br><div><br><div><br><div><br></div></div></div></div></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Sep 24, 2013 at 10:15 PM, Matt McCormick <span dir="ltr"><<a href="mailto:matt.mccormick@kitware.com" target="_blank">matt.mccormick@kitware.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Brad,<br>
<div class="im"><br>
> My ITK valgrind it taking too long. Just some test are just too long. How would I disable these tests on a dashboard script?<br>
<br>
</div>These tests could be disabled with<br>
<br>
set(CTEST_TEST_ARGS EXCLUDE_LABEL RUNS_LONG)<br>
<div class="im"><br>
> Also any idea what the change is coverage is?<br>
<br>
</div>Yes, the decrease is about 0.26 %. Here are my local results:<br>
<br>
Default run 1:<br>
Covered LOC: 123682<br>
Not covered LOC: 22557<br>
Total LOC: 146239<br>
Percentage Coverage: 84.58%<br>
<br>
Default run 2:<br>
Covered LOC: 123671<br>
Not covered LOC: 22570<br>
Total LOC: 146241<br>
Percentage Coverage: 84.57%<br>
<br>
-LE RUNS_LONG:<br>
Covered LOC: 123306<br>
Not covered LOC: 22942<br>
Total LOC: 146248<br>
Percentage Coverage: 84.31%<br>
<br>
Thanks,<br>
Matt<br>
<div class="HOEnZb"><div class="h5"><br>
<br>
> On Sep 20, 2013, at 6:40 PM, Matt McCormick <<a href="mailto:matt.mccormick@kitware.com">matt.mccormick@kitware.com</a>> wrote:<br>
><br>
>> I did a visualization on test times [1] and it turns out that less<br>
>> than 1% of the tests (0.97%) take of half (48.7%) of the test times!<br>
>><br>
>> It would be good to add a "RUNS_LONG" label [2] to these tests so that<br>
>> developers can run the test suite relatively quickly with<br>
>><br>
>> ctest -LE RUNS_LONG<br>
>><br>
>> and the Visual Studio Debug Nightly builds can exclude them with<br>
>><br>
>> set(CTEST_TEST_ARGS EXCLUDE_LABEL RUNS_LONG)<br>
>><br>
>> These tests are mostly integration tests that test complicated<br>
>> algorithms on semi-realistic data. They have value, but leaving them<br>
>> out in these situations should not do much harm relative to their time<br>
>> overhead.<br>
>><br>
>> Thanks,<br>
>> Matt<br>
>><br>
>> [1] <a href="http://review.source.kitware.com/#/c/12733/" target="_blank">http://review.source.kitware.com/#/c/12733/</a><br>
>> [2] <a href="http://review.source.kitware.com/#/c/12734/" target="_blank">http://review.source.kitware.com/#/c/12734/</a><br>
>><br>
>><br>
>> On Fri, Aug 2, 2013 at 11:13 AM, Bill Lorensen <<a href="mailto:bill.lorensen@gmail.com">bill.lorensen@gmail.com</a>> wrote:<br>
>>> In general, we do want to make sure the tests produce "correct" results and<br>
>>> make sure there are no regressions. By having the Release build run a proper<br>
>>> set of iterations, the developer can verify the correctness.<br>
>>><br>
>>> That said, using a lower resolution input dataset could probably help most<br>
>>> the these tests and the release/debug times would both be reduced.<br>
>>><br>
>>><br>
>>><br>
>>> On Fri, Aug 2, 2013 at 10:49 AM, David Cole <<a href="mailto:dlrdave@aol.com">dlrdave@aol.com</a>> wrote:<br>
>>>><br>
>>>><br>
>>>>> I think it is OK to drop the iterations in the Debug tests. We are still<br>
>>>>> exercising the code. If we<br>
>>>>> need alternate baselines, we will still catch exceptions. This will also<br>
>>>>> allow more valgrind tests<br>
>>>>> to run to completion.<br>
>>>><br>
>>>><br>
>>>> Ha ha. Ironic. I just noticed the slowest 3 tests have the word “Fast” in<br>
>>>> the test name... Perhaps we should at the very least consider a rename. 😊<br>
>>>><br>
>>>> If it’s ok to drop the iterations in the Debug tests, then why not drop<br>
>>>> the iterations in the Release tests, too? There is something to be said for<br>
>>>> the “same” code being run for the test in both Debug and Release. If it’s<br>
>>>> not the same, then I should definitely be able to tell just by glancing at<br>
>>>> the source code for a test that there are Debug/non-Debug differences...<br>
>>>><br>
>>>> I would say each individual test should be able to run in *seconds*, not<br>
>>>> minutes or hours. Ideally, less than a second, but I realize that’s overly<br>
>>>> ambitious with some ITK code.<br>
>>>><br>
>>>> However, for the main purpose I have in mind here, namely reducing the<br>
>>>> time burden on my volunteer machine, I would be absolutely *ecstatic* if all<br>
>>>> of the 20 tests I listed in the original email were able to run in under 4<br>
>>>> or 5 minutes each.<br>
>>>><br>
>>>> Let me know if I can help out in some additional way (beyond just<br>
>>>> continuing to submit the build to CDash...)<br>
>>>><br>
>>>><br>
>>>> Thanks for the discussion, I appreciate it<br>
>>>> D<br>
>>>><br>
>>><br>
>>><br>
>>><br>
>>><br>
>>> --<br>
>>> Unpaid intern in BillsBasement at noware dot com<br>
>>><br>
>>> _______________________________________________<br>
>>> Powered by <a href="http://www.kitware.com" target="_blank">www.kitware.com</a><br>
>>><br>
>>> Visit other Kitware open-source projects at<br>
>>> <a href="http://www.kitware.com/opensource/opensource.html" target="_blank">http://www.kitware.com/opensource/opensource.html</a><br>
>>><br>
>>> Kitware offers ITK Training Courses, for more information visit:<br>
>>> <a href="http://kitware.com/products/protraining.php" target="_blank">http://kitware.com/products/protraining.php</a><br>
>>><br>
>>> Please keep messages on-topic and check the ITK FAQ at:<br>
>>> <a href="http://www.itk.org/Wiki/ITK_FAQ" target="_blank">http://www.itk.org/Wiki/ITK_FAQ</a><br>
>>><br>
>>> Follow this link to subscribe/unsubscribe:<br>
>>> <a href="http://www.itk.org/mailman/listinfo/insight-developers" target="_blank">http://www.itk.org/mailman/listinfo/insight-developers</a><br>
>>><br>
>> _______________________________________________<br>
>> Powered by <a href="http://www.kitware.com" target="_blank">www.kitware.com</a><br>
>><br>
>> Visit other Kitware open-source projects at<br>
>> <a href="http://www.kitware.com/opensource/opensource.html" target="_blank">http://www.kitware.com/opensource/opensource.html</a><br>
>><br>
>> Kitware offers ITK Training Courses, for more information visit:<br>
>> <a href="http://kitware.com/products/protraining.php" target="_blank">http://kitware.com/products/protraining.php</a><br>
>><br>
>> Please keep messages on-topic and check the ITK FAQ at:<br>
>> <a href="http://www.itk.org/Wiki/ITK_FAQ" target="_blank">http://www.itk.org/Wiki/ITK_FAQ</a><br>
>><br>
>> Follow this link to subscribe/unsubscribe:<br>
>> <a href="http://www.itk.org/mailman/listinfo/insight-developers" target="_blank">http://www.itk.org/mailman/listinfo/insight-developers</a><br>
><br>
_______________________________________________<br>
Powered by <a href="http://www.kitware.com" target="_blank">www.kitware.com</a><br>
<br>
Visit other Kitware open-source projects at<br>
<a href="http://www.kitware.com/opensource/opensource.html" target="_blank">http://www.kitware.com/opensource/opensource.html</a><br>
<br>
Kitware offers ITK Training Courses, for more information visit:<br>
<a href="http://kitware.com/products/protraining.php" target="_blank">http://kitware.com/products/protraining.php</a><br>
<br>
Please keep messages on-topic and check the ITK FAQ at:<br>
<a href="http://www.itk.org/Wiki/ITK_FAQ" target="_blank">http://www.itk.org/Wiki/ITK_FAQ</a><br>
<br>
Follow this link to subscribe/unsubscribe:<br>
<a href="http://www.itk.org/mailman/listinfo/insight-developers" target="_blank">http://www.itk.org/mailman/listinfo/insight-developers</a><br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div><br></div><div><br></div>---------------------------------------------<br><b>Xiaoxiao Liu</b>, Ph.D.<div><div><div><font face="arial, sans-serif"><span style="border-collapse:collapse">R & D Engineer</span></font></div>
<div><div><a href="http://www.kitware.com/" target="_blank">Kitware Inc</a>.</div></div><div><span style="border-collapse:collapse;font-family:arial,sans-serif;font-size:13px">Clifton Park, NY</span><div><div>Phone: (518) 881-4924 or <span style="border-collapse:collapse;color:rgb(136,136,136);font-family:arial,sans-serif;font-size:13px">(518) 371-3971 x124</span></div>
<div><font face="Verdana, Arial, Helvetica, sans-serif"><span style="line-height:31px;font-size:x-small"><br></span></font></div></div></div></div></div>
</div>