[Insight-developers] FW: [Dart] Named measurements (customized content)
Miller, James V (CRD)
millerjv@crd.ge.com
Tue, 4 Sep 2001 07:32:55 -0400
Should have sent this message here as well. This capability is part of the regression testing
subsystem.
Jim
-----Original Message-----
From: Miller, James V (CRD)
Sent: Tuesday, September 04, 2001 7:30 AM
To: 'dart@public.kitware.com'
Subject: [Dart] Named measurements (customized content)
We are in the process of adding "named measurements" to Dart. "Named measurements" allow a test to
output information in its stdout streamed that gets presented differently from the remainder of the
stdout of a test. Specifically, named measurements take the form of
<DartMeasurement name="foo" type="numeric/double" encoding="none" compression="none">
data
</DartMeasurement>
<DartMeasurementFile name="foo" type="image/jpeg">
filename
</DartMeasrementFile>
The "type" can be one of the following:
text/plain
text/string
text/html
text/xml
link/url
link/image
numeric/integer
numeric/float
numeric/double
numeric/boolean
image/png
image/jpeg
The "encoding" can be:
none (default, does not have to be specified)
base64
The "compression" can be:
none (default, does not have to be specified)
gzip
The "encoding" and "compression" attributes specify how the data within the tag is encoded or
compressed. In most cases, the encoding and compression will be "none".
The <DartMeasurementFile></DartMeasurementFile> tags are provided for convenience. This tag will
pick the appropriate encoding/compression schemes and convert the tag to a <DartMeasurement> tag.
Stdout of the test will be scanned for these tags and the information extracted into Test.xml. This
data will be presented in a table on the Test result HTML page. The row headings will be the "name"
of the measurements. The table entries will provide the measurement values for text/string,
link/url, link/image, numeric/*, image/png and image/jpeg. The table entries will be links for
text/xml, text/html and text/plain.
After the table of values, the "rest" (everything that was not within a Dart tag) of the test's
stdout will be displayed.
An example of named measurements is the following link. The images, cpu time, wall time, image error,
and baseline image were "named measurements". The images were reported using DartMeasurementFile tags
while the remainer used DartMeasurement tags.
http://public.kitware.com/vtkhtml/Testing/HTML/TestingResults/Sites/amber1.kitware/A_Mandrake7.2-1-Ni
ghtly/20010903-Nightly/Results/__Rendering_Testing_Tcl_volTM2DCropRegions-image.html
Currently, the numeric/*, text/string, link/*, and image/* named measurements have been implemented
and are in the current repository. The named measurements text/plain, text/html and text/xml have not
been implemented yet. Furthermore, the combinations of encodings and compressions have not been
completely flushed out. But given these restrictions, named measurements are currently very useful.
How does this all work? (In case you are interested...)
Test.tcl parses the output of a test looking for DartMeasurement and DartMeasurementFile tags. When
it finds a tagset, it converts the tag into a NamedMeasurement tag in the Test.xml file. If the tag
was a DartMeasurementFile tag, the data from the file is read and converted to base64 and placed
within the NamedMeasurement tag in the Test.xml file. Any part of the test's stdout that is not
within DartMeasurements tags is passed as the measurement for a test (just like before).
Test.xml is shipped to the Dart server. When Test.xsl converts Test.xml to Test.html, it creates a
table labeling the rows with the "name"'s of the measurements and filling in the entries with the
specific measurement. If the measurement was an image, Test.xsl extracts the base64 data for that
image from Test.xml into a separate file. An img tagset is placed in the measurement table,
referencing an image whose name is constructed from the "measurement name" (i.e. the value of the
"name" attribute in the tag). DashboardManager.tcl then converts all base64 files into the unencoded
files.
-----Original Message-----
From: Miller, James V (CRD)
Sent: Tuesday, August 28, 2001 11:29 AM
To: 'dart@public.kitware.com'
Subject: RE: [Dart] Question about customizing HTML and/or XML form
conten t
Dan and I discussed this further this morning and this is what we came up with.
A test can output a snippet of XML that Dart will recognize and process/display appropriately. Two
tagsets are proposed:
<DartMeasurement name="foo" type="numeric/double" encoding="none" compression="none">
data
</DartMeasurement>
<DartMeasurementFile name="foo" type="text/plain">
filename
</DartMeasrementFile>
The "type" can be one of the following:
text/plain
text/string
text/html
text/xml
text/url
numeric/integer
numeric/float
numeric/double
numeric/boolean
image/png
The "encoding" can be:
none (default)
base64
The "compression" can be:
none (default)
gzip
The <DartMeasurementFile></DartMeasurementFile> tags are provided for convenience. This tag will
pick the appropriate encoding/compression schemes and convert the tag to a <DartMeasurement> tag.
Stdout of the test will be scanned for these tags and the information extracted into Test.xml.
This data will be presented in a table on the Test result HTML page. The column headings will be the
"name" of the measurements. The table entries will provide the measurement values for text/string,
text/url, numeric/*, and image/png. The table entries will be links for text/xml, text/html.
After the table of values, the "rest" (everything that was not within a Dart tag) of the test's
stdout will be displayed.
Jim/Dan
-----Original Message-----
From: Miller, James V (CRD)
Sent: Tuesday, August 28, 2001 8:35 AM
To: 'ebmiddlecamp@imation.com'; Blezek, Daniel J (CRD);
'dart@public.kitware.com'
Subject: RE: [Dart] Question about customizing HTML and/or XML form
conten t
Eric,
I think you are on the right track. The only caveat is that for one of our software systems (VTK) we
have 200 Tcl tests and 10 C++ tests. My point is that people will use Dart in ways that we would
have never imagined.
One goal of Dart is to noninvasive to the software it is testing. I think your suggestion fills that
bill. One idea I would like to play with is to allow people to output data from their tests and have
that data called out by Dart Dashboards. Currently, Dart just gathers all the output from stdout and
sends it to the Dashboard in XML as
<Test>
<Results>
<Measurement>
<Value>
stdout information
</Value>
</Measurement>
</Results>
</Test>
One naive approach would be just to have a test output an XML snippet that got naively wrapped by the
<Value></Value> pairs. This could then be picked up by the XSL systems. It could look something
like:
<Test>
<Results>
<Measurement>
<Value>
<WallTime>1.24</WallTime>
<CPUTime>0.47</CPUTime>
</Value>
</Measurement>
</Results>
</Test>
This, however, is pretty much a hack. It force fits extra information inside the <Value></Value>
tags.
The XML should really look something like:
<Test>
<Results>
<Measurement>
<WallTime>
1.24
</WallTime>
<CPUTime>
0.47
</CPUTime>
<Value>
rest of stdout
</Value>
</Measurement>
</Results>
</Test>
or like
<Test>
<Results>
<WallTime>
1.24
</WallTime>
<CPUTime>
0.47
</CPUTime>
<Measurement>
<Value>
rest of stdout
</Value>
</Measurement>
</Results>
</Test>
Either of these two methods require the Dart tcl scripts to parse the output of a test for tags that
it may understand.
Another option is for a test to callout items for Dart explictly, The test could output:
standard_output_stuff
<DartMeasurement>
<WallTime>
1.24
</WallTime>
</DartMeasurement>
<DartMeasurement>
<CPUTime>
1.24
</CPUTime>
</DartMeasurement>
more standard_output_stuff
Or maybe the syntax should be
<DartMeasurement type="WallTime">
1.24
</DartMeasurement>
We'll bat around a couple of ideas and see what fits cleanest into the XML/XSL framework.
-----Original Message-----
From: ebmiddlecamp@imation.com [mailto:ebmiddlecamp@imation.com]
Sent: Monday, August 27, 2001 3:28 PM
To: Miller, James V (CRD)
Cc: dart@public.kitware.com
Subject: RE: [Dart] Question about customizing HTML and/or XML form
conten t
James,
I've thought some more about how "cpu timing" might work; if we grant that
"cpu timing" is an optional feature (per test), we can gather the
information by allowing the test output itself to include the wall time and
the cpu time. For instance, the last two lines of test output could be
"Wall time- hh:mm:ss.xx" and "CPU time- hh:mm:ss.xx". Since most tests are
written in C++ anyway, gathering these times should be a snap. All the Tcl
scripts would have to do is look for these strings in the test output, and
include the times again somewhere in the submitted XML..
If that sounds acceptable, we can move on to thinking about items #2 and
#3, below..?
Regards,
Eric Middlecamp
Imation Corp.
"Miller, James V (CRD)" <millerjv@crd.ge.com> on 07/02/2001 11:43:15 AM
To: "'ebmiddlecamp@imation.com'" <ebmiddlecamp@imation.com>,
dart@public.kitware.com
cc: "Blezek, Daniel J (CRD)" <blezek@crd.ge.com>
Subject: RE: [Dart] Question about customizing HTML and/or XML form conten
t
Yes,
Test timing will be added to Dart. We just have to figure out:
1) How to measure the CPU time a given test takes from when run from a Tcl
exec call.
2) How to structure the "search" backwards in time for historical
information
3) How we want to render this information. I would suggest that we use one
of the standard packages
(python, perl, tcl) that wraps the GDI graphic utility.
We are not using any special java code. Dart comes with one jar file that
rolls up the Xalan parser
and XSLT engines together. These are separate open-source projects.
The server side is really driven by the XSLT files (XML Style sheet
language translator). Given the
standard xalan.jar file and an xsl file, a given xml file is converted into
HTML. So all the "magic"
would happen in Test.xsl which is run on the server. If we use approach
#3, above, this might mean
simply extracting some information into a separate XML file and putting a
link in an HTML page that
will run the cgi-bin script.
-----Original Message-----
From: ebmiddlecamp@imation.com [mailto:ebmiddlecamp@imation.com]
Sent: Monday, July 02, 2001 12:10 PM
To: dart@public.kitware.com
Subject: [Dart] Question about customizing HTML and/or XML form content
Hi all,
This is my first post to the list, so forgive me if this question has been
asked already.. (I couldn't find the archives, if there are any.)
Here is my question: Are there any plans to add on "CPU Timing" information
to Dart, like the VTK dashboard has? I would attempt to do this myself, but
the Java source for HTML generation is currently not available. I imagine
that the modifications at the client would be fairly trivial (modify
test.tcl, modify test.xsl), but the modifications at the server are an
unknown (well, for me, at least.)
Does this sound feasible? If so, would it be possible to get a copy of the
Java source code?
Regards,
Eric Middlecamp
Imation Corp
_______________________________________________
Dart mailing list
Dart@public.kitware.com
http://public.kitware.com/mailman/listinfo/dart
_______________________________________________
Dart mailing list
Dart@public.kitware.com
http://public.kitware.com/mailman/listinfo/dart
_______________________________________________
Dart mailing list
Dart@public.kitware.com
http://public.kitware.com/mailman/listinfo/dart
_______________________________________________
Dart mailing list
Dart@public.kitware.com
http://public.kitware.com/mailman/listinfo/dart