readme.html
1<!doctype html public "-//w3c//dtd html 4.0 transitional//en">
2<html>
3 <head>
4 <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
5 <title>org.eclipse.test.performance.ui readme</title>
6 </head>
7
8<p><b><font size="+2">README</font></b><br> </p><p>last updated: October 26, 2005<br>
9</p><p><b>Introduction</b><br> <br> This readme describes how to create performance
10charts and tables similar to those available on the eclipse platform download
11pages using the stand-alone java program <b>org.eclipse.test.performance.ui.Main</b>
12stored in this project. Although the program is currently geared to generating
13results for the Eclipse Platform builds, the program may work for you given the
14following conditions:</p><ul> <li>The "eclipse.perf.config" system property
15used to run tests used three keys config, build and jvm <ul> <li>for example,
16"eclipse.perf.config=config=relengbuildwin2;build=I20050219-1500;jvm=sun1.4.2_06".
17<br> <br> </li></ul></li><li>Values for the "build" key begin with either an 'I'
18or 'N' with the exception of the reference builds (for example, 3.0). <ul> <li>The
19line graphs plot the reference build, all builds starting with 'I' and the seven
20'N' builds preceding the current build.</li><li>This can be overriden with the use of the "-current.prefix" parameter where you can specify a comma-separated list of prefixes for build ids to include in the main line graph.<br> <br> </li></ul></li><li>Scenario names
21all contain a prefix "<component>.test" <ul> <li>For example,
22it is assumed that the scenario "org.eclipse.help.tests.performance.appserver.TestAppserver#testAppserverCycle()"
23belongs to the "org.eclipse.help" component. </li><li>Unique scenario
24prefixes are derived from all the scenarios in the database and are used to group
25results.</li></ul></li></ul><p><br> For purposes of this document, the term "<a href="images/descriptions.html#fp">fingerprint</a>"
26will be used to refer to a bar graph which represents the performance summary
27at a global or component level for each test configuration.<br> <br> The term
28"<a href="images/descriptions.html#sst">scenario status table</a>" will
29be used to refer to a table of scenarios with green checks and red x indicators
30below a component fingerprint. This table gives an overall view of the status
31for all scenarios for a component for all configurations.<br> </p><p>The term
32"<a href="images/descriptions.html#sr">scenario results</a>" will be
33used to refer to a web page comparing current performance measurements to the
34baseline measurements for one scenario on one configuration. The web page displays
35the raw data for the current and baseline measurements and also displays line
36graphs showing the measured values over the course of builds between the current
37and baseline.</p><p><br> <b>Requirements</b> </p><ul> <li>Windows or Linux (x86)
38operating system</li><li>Cloudscape or Derby plug-in</li><li>org.eclipse.releng.basebuilder</li><li>1.4
39level jvm installed and on path</li></ul><p><b>Setup</b><br> </p><ul> <li>Checkout
40org.eclipse.releng.basebuilder from cvs repository dev.eclipse.org:/home/eclipse
41(HEAD should be fine).</li><li>Create a Cloudscape or derby plug-in as described
42<A HREF="http://dev.eclipse.org/viewcvs/index.cgi/*checkout*/org.eclipse.test.performance/doc/Performance%20Tests%20HowTo.html?rev=HEAD&content-type=text/html#cloudscape">here</A>.</li><li>Set
43the environment variable "dbloc" to the location of the database containing
44the performance data<br> For example:<br> <ul> <li> export dbloc=net://minsky.ottawa.ibm.com
45(Linux)</li><li>set dbloc==net://minsky.ottawa.ibm.com (Windows)<br> </li></ul></li></ul><p><b>Running
46the program</b></p><p>From a terminal or dos prompt,<br> </p><OL><LI>Set the environment
47variable "dbloc" to the location of the database containing the performance
48data<BR> For example:<UL><LI>export dbloc=net://minsky.ottawa.ibm.com (Linux)</LI><LI>set
49dbloc==net://minsky.ottawa.ibm.com (Windows)<BR><BR></LI></UL></LI><LI>cd
50to org.eclipse.releng.basebuilder\plugins\org.eclipse.test.performance.ui\scripts<BR></LI><LI>If
51running on Linux, execute "chmod 755 genresults"</LI><LI>Execute the
52following command:<b><br> <br> genresults -baseline <baseline build id>
53-current <current build id> -config <comma-separated list of configs>
54-jvm <jvm name> -output <output dir> [-config.properties <properties>]
55[highlight <buildid patterns>] [scenario.pattern <scenario prefix
56patterns>] [-fingerprints][-scenarioresults][-current.prefix <comma-separated list of build id prefixes>][-baseline.prefix <comma-separated list of build id prefixes>]</b></LI></OL><br> Parameter descriptions:
57<blockquote> <p>-baseline <baseline build id. A previous value used in the
58"build" key in the eclipse.perf.config system property or the eclipse.perf.assertAgainst
59property. > <br> -current <current build id. The value used in the "build"
60key in the eclipse.perf.config system property.><br> -config <a comma-separated
61list of names of configurations for which to produce results. The values used
62should be the same as values specified for the "config" key in the eclipse.perf.config
63system property.><br> -jvm <jvm description. The value used in the "jvm"
64key in the eclipse.perf.config system property.><br> -output <path to an
65output directory for fingerprints and scenarios for each configuration specified
66in -config parameter. Fingerprint gifs and html files are generated in the directory
67specified, configuration specific subdirectories are created to store scenario
68results and line graphs.></p></blockquote><p><br> Optional arguments:</p><blockquote>
69<p>-config.properties <semi-colon separated list of: config, alternate config
70description. The value should be quoted if there are spaces in the value.><BR>of builds which match any prefix in this list. When omitted, a magenta line is draw on the graph which represents the baseline value specified in the -baseline parameter.><br> -highlight <Comma-separated
71list of build Id prefixes(excluding 'I' and 'N'), which are used to add and highlight
72the most recent matching buildId on line graphs."><br> -scenario.pattern
73<Scenario prefix pattern used to query performance results database. Can be
74used to generate results for a single component or subset of scenarios.><br>
75-fingerprints or -scenarioresults <use one or the other to generate fingerprints
76or scenario results only. Not specifying either will execute both.><BR>-baseline.prefix <semi-colon separated list of: build id prefixes used in eclipse.perf.assertAgainst property or eclipse.perf.config system property. Values used to generate a second line graph representing changes in repeated baseline test runs>.<BR>-current.prefix <semi-colon separated list of: build id prefixes used in eclipse.perf.config system property. Values here used to override default of "N,I". Used to select build id's to include in main line graph.></p></blockquote>
77</html>