1Copyright (c) 2014-2015, Intel Corporation
2All rights reserved.
3
4Redistribution and use in source and binary forms, with or without modification,
5are permitted provided that the following conditions are met:
6
71. Redistributions of source code must retain the above copyright notice, this
8list of conditions and the following disclaimer.
9
102. Redistributions in binary form must reproduce the above copyright notice,
11this list of conditions and the following disclaimer in the documentation and/or
12other materials provided with the distribution.
13
143. Neither the name of the copyright holder nor the names of its contributors
15may be used to endorse or promote products derived from this software without
16specific prior written permission.
17
18THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
19ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
20WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
21DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR
22ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
23(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
24LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
25ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
26(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
27SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
28
29# Client simulator
30
31## Project
32
33This project allows users to test Parameter Framework configurations. It
34allows writing scenario, mixing criterion changes and arbitrary script
35execution. The goal of such tests are to recreate, under a test harness, the
36real use-cases that will be executed by a Parameter Framework.
37
38In short: it eases the development of automated Parameter Framework
39configuration functional tests.
40
41## How to run tests
42
43You can run tests using pfClientSimulator.py. `test-platform` and
44`remote-process` need to be in the PATH (e.g. by installing the Parameter
45Framework - see the main README file).
46
47You have to run the script with at least the test directory argument:
48
49* `test_directory`: path to the test directory
50
51Different options are available:
52
53* `-s`, `--scenario` : precise the scenario number to launch
54* `-i`, `--interactive` : run in interactive mode to test new vectors
55* `-v`, `--verbose` : make the script talk on stdin
56* `-c`, `--coverage` : generate the html coverage report at the end of the script
57* `-h`, `--help` : show options recap
58
59To see available scenarios in the test directory, you can run:
60
61 ./pfClientSimulator.py path/to/tests/
62
63The script will wait for a choice and the desired scenario will be
64launched. You can also directly select a scenario with -`s` option.
65
66Regarding coverage, you might want to read
67[this README](https://github.com/01org/parameter-framework/blob/master/tools/coverage/README.md)
68to learn what it is about.
69
70## How to create a test environment
71
72### Test Directory
73
74A test directory should contains a `conf.json` file containing:
75
76- The desired command prefix (e.g. `adb shell` in order to execute tests on an
77 android board or empty to execute locally).
78- The port on which the test-platform should be started.
79- The criterion file path (see
80 [this README](https://github.com/01org/parameter-framework/tree/master/tools/xmlGenerator#domaingeneratorpy)).
81- The path to the Parameter Framework toplevel configuration file.
82- The path to the directory containing the scenario files.
83- The path to the scripts definitions file (optional) (see below).
84- The path to the actions definitions (aka "ActionGatherer") file (optional)
85 (see below).
86- The path to the log output file (optional but needed for coverage).
87- The path to the directory containing the coverage generation tool
88 (optional; for coverage only).
89- The path to the html coverage output file (optional; for coverage only).
90- The path to the Parameter Framework domain configuration file (optional; for
91 coverage only).
92
93Relative paths in `conf.json` will be evaluated *relative to the test
94directory*.
95
96## Example Client Simulator configuration file
97
98```{.json}
99{
100 "PrefixCommand" : "adb shell",
101 "TestPlatformPort" : "5001",
102
103 "CriterionFile" : "MyCriteria.txt",
104 "PfwConfFile" : "/home/user/tests/TopLevel.xml",
105
106 "ScenariosDirectory" : "test-vectors",
107 "ScriptsFile" : "my-test-scripts.json",
108 "ActionGathererFile" : "my-test-actions.json",
109
110 "LogFile" : "tests.log",
111
112 "CoverageDir" : "/home/user/parameter-framework/tools/coverage",
113 "CoverageFile" : "coverage.html",
114 "PfwDomainConfFile" : "MyConfigurableDomains.xml"
115}
116```
117
118### Scenario
119
120All scenario files need to be put in the directory mentioned by the
121`ScenariosDirectory` configuration item.
122
123A scenario file contains all the actions you want to take. *Note*: it is an
124ordered list. There are two possible kind of actions: `setCriterion` and
125`script`.
126For example:
127
128```{.json}
129[
130 {"setCriterion" :
131 {
132 "Mood" : "mad"
133 }
134 },
135 {"script" : "myFirstScript"},
136 {"setCriterion" :
137 {
138 "Mood" : "glad",
139 "SmokeGenerator" : "On",
140 "Colors" : "red blue"
141 }
142 },
143 {"script" : "mySecondScript"}
144]
145```
146
147This scenario file sets a criterion, then runs a script, then sets three
148criteria and finally runs another script.
149
150The `setCriterion` type allows user to offer a dictionary describing changing
151criteria (configurations are automatically applied after using this action
152type). Other criteria keep there old values.
153
154The other type, `script` allows users to launch a script when he wants.
155
156### Action definitions (aka ActionGatherer)
157
158You can also define your own types based on the system ones, defined ones. You
159have to edit the `ActionGathererFile` specified in the `conf.json` file to do
160that. This file has this pattern :
161
162```{.json}
163{
164 "getMad":
165 {"setCriterion" :
166 {
167 "Mood" : "mad"
168 }
169 },
170 "party" :
171 {"setCriterion" :
172 {
173 "SmokeGenerator" : "On",
174 "Colors":"red blue"
175 }
176 }
177}
178```
179
180Here we define five new types based on `setCriterion`. When writing a scenario,
181we can now use those types as basis and add some criteria to set in the same
182time.
183
184**For now, defining `script` actions are not supported**; only `setCriterion`
185action definitions are supported.
186
187Here is the example scenario, rewritten using the above definitions:
188
189```{.json}
190[
191 {"getMad" : {}},
192 {"script" : "myFirstScript"},
193 {"party":
194 {
195 "Mood":"glad",
196 }
197 },
198 {"script" : "mySecondScript"},
199]
200```
201
202During the `party` step, the `SmokeGenerator` and `Colors` criteria are set as
203defined in the `ActionGathererFile` but also the `Mood` criteria.
204
205### Scripts
206
207Scripts are defined in the `ScriptsFile` specified in the `conf.json` file *or*
208they can be an inline shell script.
209
210This `ScriptsFile` file should look like this :
211
212```{.json}
213{
214 "myFirstScript" : ["test-scripts/first.sh","asynchronous"]
215 "mySecondScript" : ["test-scripts/second.sh","synchronous"],
216}
217```
218
219This dictionary is composed as such:
220
221```{.json}
222name : ["path/to/script",synchronousness]
223```
224
225The path to the script is relative *to the path of the `ScriptsFile`*.
226
227The synchronousness can be either `"synchronous"` or `"asynchronous"` and
228determines whether the Client Simulator waits for the script to return before
229executing the next scenario step. To be clear:
230
231* asynchronous : The script will run concurrently to the execution of the
232 Client Simulator;
233* synchronous : The Client Simulator will wait the end of the script before
234 resuming execution and thus, the rest of the scenario won't be executed until
235 the script returns.
236