| fc3da5bbdcd63840cd224282812c050c5520db11 |
|
06-Jan-2010 |
mbligh <mbligh@592f7852-d20e-0410-864c-8624ca9c26a4> |
More refactoring of the job state code to try and reduce duplication.
In the process this also refactors a bunch of job.run_test duplication,
specifically the handling of job tags (there's still lots more
duplicate code left).
The main state refactorings are:
* Storing client job steps in __steps and special-casing its removal
in autotest.py is gone. Instead the client job code stores it
directly in the client.steps state, and the autotest.py generically
removes the entire client.* namespace.
* Moving this out of the publicly-visible state and into a private
namespace allows us to drop has_state and discard_state, which
were really only needed so that we could make sure __steps was
not persisted across jobs.
* All of this client-job state managment was moved out of autotest.py
and into server_job.py. All the data this code was getting and
setting was in the job object, so it seemed like a better place
to put it. This also let us drop load_state and save_state, which
were really only there to make it easier to push and pull data
out of the server job object.
* Eliminate get_run_number and set_run_number in favour of a generic
use_sequence_number boolean property (that works on both client
and server automatically). The get_ and set_ code was only being
used to enable and disable an auto-incrementing sequence number to
be tagged to each test run, so we can just make the public
interface a boolean and handle the sequencing under the covers.
* Drop all the test_tag and test_tag_prefix code in favour of a
persistent automatic_test_tag property. Also drops the
show_kernel_in_test_tag flag in favour of just implementing this
in the control file generator using automatic_test_tag.
* Refactor all the test name tagging code into base_job, unifying
the tagging portion of the job.run_test code.
Signed-off-by: John Admanski <jadmanski@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@4075 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/control_file.py
|
| fbf73aecdd357094ae05e7d1e4ea99b1ecf93ee4 |
|
19-Dec-2009 |
mbligh <mbligh@592f7852-d20e-0410-864c-8624ca9c26a4> |
Refactor all of the persistent state managing code in the client job
class into the common_lib base job class. The major change is that all
the details of the job state implementation are pulled into a class
base_job.job_state. This class handles all the internal details of the
state managment that job.get_state and job.set_state did, and also
provides a mechanism for creating job properties which are automatically
backed by job state.
This also includes a major change to the semantics of state passed
between autoserv and any Autotest clients it ran. Previously the code
was a bit of a hack, with code in server/autotest.py that would
pre-generate a state file with some pre-set values, and code that would
pull back the state file after the client job and extract some values
from it, as well as some more code that would prepend a bunch of
state initialization calls at the start of the client control file.
Instead of all these hacks, autotest.py now just pushes all of the
state out to the client at the start, and pulls it all back at the end
(with a special exception for __steps that's difficult to eliminate at
this time). So any state set on the server is automatically available
on the client, and any state set on the client is automatically
available on the server.
There are a couple of outstanding issues:
* More of the set_state/get_state code needs to be refactored
to either use the automatic properties, or the job-internal state
API (so that all the internal state is not leaked into the "public"
state visible through get_state and set_state).
* The job_state class is perhaps somewhat overengineered. It's really
just a big, automatically persistect dict-of-dicts. But most of it
is actually code that was being used before, it was just scattered
all over server/autotest.py and client/bin/job.py so it was less
obvious how overgrownt he state managment is.
Risk: High
Visibility: Replaces a bunch of get_*+set_*+default_* methods with
persistent properties, refactors all of the state managment
into a single implementation consistently used in both the
server and client.
Signed-off-by: John Admanski <jadmanski@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@4022 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/control_file.py
|
| 232b7aecb0df153af9e6dca62bb6194acbfd709c |
|
10-Nov-2009 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Added a upload_kernel_config (default False) keyword parameter to
generate_control_file RPC that produces server side code to download and
transfer on the client the kernel config (if it's an URL) and point the
client to the new location. This allows to have kernel config URLs point
to servers that are reachable on the server side but not on the client
side.
Signed-off-by: Mihai Rusu <dizzy@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@3920 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/control_file.py
|
| 39c843ba954b8862f7da24a4533de2069eed48fc |
|
10-Nov-2009 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Do not overwrite the per kernel config_file value if it has been
provided by the RPC user.
Signed-off-by: Mihai Rusu <dizzy@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@3919 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/control_file.py
|
| 0d0f67d6abcf6599ad626042d3236020e093558f |
|
06-Nov-2009 |
mbligh <mbligh@592f7852-d20e-0410-864c-8624ca9c26a4> |
Create a base_job class in the common library as a location to
refactor common code from client/bin/job.py and server/server_job.py
into.
Initially this just creates the base class and attempts to refactor
some of the __init__ code into it. It also tries to refactor the
code somewhat so that it's easier to test without mocking out large
amounts of external dependencies.
It also tries to standardize on what all the official publicly usable
attributes are, mostly on the basis of looking at what the existing
code actually does. In general my approach was:
* If used outside of the job class, assume it's public. If it makes
sense make it a requirement of the base class, but in some cases
(e.g. serverdir) it really only makes sense for one of the
concrete classes. In that case I define it as "OPTIONAL", which
means it can be None (but must still be defined).
* If only used within the job class, make it "private", which means
it gets a name starting with _. Code outside of the job modules
shouldn't be touching these. This includes the "constants" that
we define with all-caps in the class.
* I added unit tests which try to actually validate this, by
checking that all the correct properties are defined after
__init__. The test cases are still limited (all the test_init
cases) because of the pain of stubbing out __init__, but I will
be adding more in the future as I refactor to reduce the amount
of mocking/stubbing needed.
Overall, I don't know that this change by itself makes the code
simpler or easier to understand, but I don't think it makes things
any worse in that regard, while adding a lot more testing and
finally giving us a well-defined interface. I have more changes
planned, but as new improvements kept creeping into this change I
decided to mail it out for some review instead of letting it grow
even bigger. Some ideas of what I still intend to do in followup
patches:
* Refactor more of the directory management code into job_directory, to
get the rest of the mkdir/makedirs code into there. That way all
of the directory management is concentrated in one place.
* Refactor all the of get_state/set_state code into something that
automatically handles this with properties without a lot of
boilerplate getter and setter methods.
* Refactor the job.record related code that's duplicated into the
base class.
Risk: High
Visibility: Mostly code changes to standardize the job class interface.
A lot of the visible changes just involve privatizing attributes
that aren't used outside of job anyway, or adding in attributes
that were only defined on the client or server.
Signed-off-by: John Admanski <jadmanski@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@3916 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/control_file.py
|
| eaa75e594adc241b99522e57354e9efc8ba30808 |
|
06-Nov-2009 |
mbligh <mbligh@592f7852-d20e-0410-864c-8624ca9c26a4> |
Fixed support for source kernels in generated control files (using the
generate_control_file() RPC) by adding a build() method call.
Implemented a dummy do nothing build() version in rpm_kernel (also
inherited by suse_rpm_kernel). Updated unittests.
Signed-off-by: Mihai Rusu <dizzy@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@3909 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/control_file.py
|
| 3c0ea96b0dbd2115fa2057c21635173eaf461f31 |
|
06-Nov-2009 |
mbligh <mbligh@592f7852-d20e-0410-864c-8624ca9c26a4> |
Added common_lib.hosts.Host.cleanup_kernels and some functions it's using.
Implemented a fast local version of those functions in LocalHost to not
go through the run() abstraction. Added call to cleanup_kernels() from
generated control file code so it will cleanup unused kernels (not
referenced by the bootloader) each time it's installing a new kernel.
Signed-off-by: Mihai Rusu <dizzy@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@3907 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/control_file.py
|
| 91f8510c0fabcdbc3108034ce4d573a3f68fe567 |
|
12-Oct-2009 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
NOTE: This depends on the (not yet committed) patches mailed out to add default_profile_only support to the job classes.
Add a new parameter to the control file generation RPCs to generate
control files that make use of the new set_default_profile_only
API.
Risk: Low
Visibility: Adds a new optional parameter to generate_control_file
Signed-off-by: John Admanski <jadmanski@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@3829 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/control_file.py
|
| a3c58d20be69a292bdfafbea735c7503f4995b4c |
|
25-Aug-2009 |
mbligh <mbligh@592f7852-d20e-0410-864c-8624ca9c26a4> |
Added support to specify cmdline for the booted kernels for both the CLI
and the frontend. For the CLI there is a new "--kernel-cmdline" option
that will be used for all the kernels configured to be booted with the
"-k" parameter. For the frontend there's a new textbox under the kernel
version one where cmdline can be provided.
Signed-off-by: Mihai Rusu <dizzy@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@3599 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/control_file.py
|
| 7eacbc2a36d4efc50b67a429bf7b2f0cc0155b9a |
|
29-Jul-2009 |
mbligh <mbligh@592f7852-d20e-0410-864c-8624ca9c26a4> |
Fixed multiple kernel control file generation for server side tests for
the case where the original test control file gives a "tag" argument to
run_test() by changing the server_job default test_tag attribute to be
instead a tag prefix.
Signed-off-by: Mihai Rusu <dizzy@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@3465 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/control_file.py
|
| a5512cdc604c8b9091e9a75bed7da05811e9575a |
|
30-Jun-2009 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Fix missing import error in kernel control file install stanza. It was
about to raise an error anyways so as it is, it still bails out but the
error message isn't the desired nice one.
Signed-off-by: Gregory Smith <gps@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@3330 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/control_file.py
|
| 415dc21b70d44616803cb5c0245248514b73adca |
|
15-Jun-2009 |
mbligh <mbligh@592f7852-d20e-0410-864c-8624ca9c26a4> |
Allow a couple of machines failing to install a kernel in the server
test kernel install stanza when >4 machines are supplied. This allows
for large multi-machine tests to proceed even if some of the hosts die
while failing to install or boot the kernel.
This is not the ideal implementation. We should really have some way
to get the max(sync_count) required by a control file's step_test()
before calling the function. Ideas welcome.
Signed-off-by: Gregory Smith <gps@google.com>
(this patch breaks a unittest. the fix is the patch i am mailing immediately after this, sorry. 2 mails is easier than merging them into one patch before mailing for me)
git-svn-id: http://test.kernel.org/svn/autotest/trunk@3284 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/control_file.py
|
| d262415ddedd11391e7625b248cb44ab0fc13380 |
|
29-Apr-2009 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Fixed indenting multiline quoted content that uses triple quotes (" or
') in the control file generation RPC. Added unittest case to catch this
issue in the future.
Signed-off-by: Mihai Rusu <dizzy@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@3065 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/control_file.py
|
| c86113b66e4f06eeecf0f3268d6abcf2e4ab4c3e |
|
28-Apr-2009 |
mbligh <mbligh@592f7852-d20e-0410-864c-8624ca9c26a4> |
Added server_job.set_test_tag() function (similar to the one in
client.bin.job) to specify a default tag to use for run_test() runs.
Changed the way we generate server control files from the frontend to be
very close to the client control files (using step_init(), stepX()
functions for each phase of the test). Changed generation of server
kernel control file to properly deal with the case of multiple kernels
given.
Risk: High
Signed-off-by: Mihai Rusu <dizzy@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@3051 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/control_file.py
|
| 120351e0c9a9b3e7c7a4aa51477a5ae2366214c4 |
|
24-Jan-2009 |
mbligh <mbligh@592f7852-d20e-0410-864c-8624ca9c26a4> |
Allow control file body text to be supplied as an alternative to a list
of test names for the generate_control_file AFE rpc.
This allows the command line client to let people specify a kernel along with
their own client-side control file.
It adds the kernel installation steps to the supplied control file before
scheduling it (using the same code as the GUI when a kernel is specified
there).
Signed-off-by: Gregory Smith <gps@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@2684 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/control_file.py
|
| 8ae7d3a68b34bb2791f4206d3b41b1fcc50a0d00 |
|
03-Dec-2008 |
jadmanski <jadmanski@592f7852-d20e-0410-864c-8624ca9c26a4> |
Add profiler support to the server-side control file generation.
Risk: Low
Visibility: You can use the frontend control file generation for
profilers even with server-side jobs.
Signed-off-by: John Admanski <jadmanski@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@2539 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/control_file.py
|
| e39c3c0dbe16caed55f4538af78dbf057deb7fad |
|
18-Nov-2008 |
mbligh <mbligh@592f7852-d20e-0410-864c-8624ca9c26a4> |
Adds the ability for the client.bin.base_job class to keep track of run
numbers and include them in the output directory / test tag.
Adds the ability for the client.bin.base_job class to insert the
running kernel version (uname -r) into the test tag at run_test() time.
Enabled both of the above in the generated control file when multiple
kernels are given via the AFE.
Signed-off-by: Gregory Smith <gps@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@2433 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/control_file.py
|
| 15fe995a5e73b0ed8da3a9450a9a42a48bcdf475 |
|
15-Nov-2008 |
mbligh <mbligh@592f7852-d20e-0410-864c-8624ca9c26a4> |
Fix server side control files generated by the autotest front end. kernel is no longer defined, kernel_list is.
Signed-off-by: Gregory Smith <gps@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@2424 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/control_file.py
|
| 1429e7f683cefd9849bef084e6fbb3d0577fa7b2 |
|
14-Nov-2008 |
mbligh <mbligh@592f7852-d20e-0410-864c-8624ca9c26a4> |
Don't include a kernel version tag prefix when only one kernel was given.
Signed-off-by: Gregory Smith <gps@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@2418 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/control_file.py
|
| 6d2a6f937d417f45f64cefcae7a5f888ba1fcbcb |
|
13-Nov-2008 |
mbligh <mbligh@592f7852-d20e-0410-864c-8624ca9c26a4> |
The kernel global needs to be set properly for many existing control files to
run.
Signed-off-by: Gregory Smith <gps@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@2411 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/control_file.py
|
| 12eafffaf869570a62025f4dc1f3d0908285998c |
|
06-Nov-2008 |
mbligh <mbligh@592f7852-d20e-0410-864c-8624ca9c26a4> |
Add support for running a test over a sequence of multiple kernels.
The kernel version is appended to the test name before the tag when
making directories for test output.
Signed-off-by: Gregory Smith <gps@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@2382 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/control_file.py
|
| 8d631c9d894ee1f438733530fb8b966f7af07df1 |
|
18-Aug-2008 |
jadmanski <jadmanski@592f7852-d20e-0410-864c-8624ca9c26a4> |
Convert all our uses of directoy SSHHost construction over to use
hosts.create_host instead. Also, fix up a bunch of test references
that use SSHHost when they really just need to run against the
abstract RemoteHost class, and fix some tests using the MockSSHHost
class to just use the mock framework instead.
Risk: Low
Visibility: Gets rid of almost all direct references to SSHHost.
Signed-off-by: John Admanski <jadmanski@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@2010 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/control_file.py
|
| c5ddfd1f71caef9ec0c84c53ef7db42fcdc33e1c |
|
04-Aug-2008 |
mbligh <mbligh@592f7852-d20e-0410-864c-8624ca9c26a4> |
I have attached the patch for the packaging system.
The documentation is available at : http://test.kernel.org/autotest/PackagingSystem
Signed-off-by: Ashwin Ganti <aganti@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@1953 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/control_file.py
|
| f5fdfab3cfb3078b2c068f05c79cb7d60fcde1c8 |
|
17-Jun-2008 |
mbligh <mbligh@592f7852-d20e-0410-864c-8624ca9c26a4> |
This changes the automatically generated control files to use spaces
instead of tabs.
Signed-off-by: Jeremy Orlow <jorlow@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@1709 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/control_file.py
|
| 2b9a88bcd336233fa483490892338e29f0a5fa67 |
|
13-Jun-2008 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
-allow users to select profilers in the create job view
-create profilers table + model
-create RPCs for managing profilers
-modify control file generation code to accept profilers
-modify create job view
-some refactoring to create job view to allow code sharing and do things more cleanly with some sweet generics
-get rid of rpc_utils.sorted. who knew there was a builtin?
git-svn-id: http://test.kernel.org/svn/autotest/trunk@1703 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/control_file.py
|
| 0afbb6369aa5aa9a75ea67dd9e95ec4b21c0c181 |
|
06-Jun-2008 |
jadmanski <jadmanski@592f7852-d20e-0410-864c-8624ca9c26a4> |
Convert all python code to use four-space indents instead of eight-space tabs.
Signed-off-by: John Admanski <jadmanski@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@1658 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/control_file.py
|
| 9ca52701a301ca2315e21d6ab876329decf53158 |
|
02-Jun-2008 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
The autotest frontend should run every test in its own step so that, if one test reboots the machine, the rest keep running.
git-svn-id: http://test.kernel.org/svn/autotest/trunk@1584 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/control_file.py
|
| 6437ff5bb28a0cc97582766826f866a39affae2a |
|
17-Apr-2008 |
mbligh <mbligh@592f7852-d20e-0410-864c-8624ca9c26a4> |
Use the new parser library directly inside of autoserv, instead of
repeatedly invoking the parser as a separate process. In order to
actually invoke this capability you need to pass in a -P option, with
the job name (tag).
In order to properly support the parsing of server-side synchronous
jobs you need to make use of job.parallel_simple, a new wrapper around
the standard parallel_simple that adds support for parsing the results
of all of the parallel jobs individually. Unfortunately, complex
server-side synch jobs that use parallel instead of parallel_simple
will be unable to take advantage of this.
Signed-off-by: John Admanski <jadmanski@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@1446 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/control_file.py
|
| 1d445e9d9bc140b685b10a9fdd53cb5e92c0883e |
|
12-Mar-2008 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Support kernel install for server-side control files
git-svn-id: http://test.kernel.org/svn/autotest/trunk@1338 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/control_file.py
|
| e8819cdf80ca0e0602d22551a50f970aa68e108d |
|
15-Feb-2008 |
mbligh <mbligh@592f7852-d20e-0410-864c-8624ca9c26a4> |
Add frontend and scheduler for Autotest
The attached tarball includes the new Autotest web frontend for creating
and monitoring jobs and the new scheduler for executing jobs. We've
been working hard to get these complete and stabilized, and although
they still have a long, long way to go in terms of features, we believe
they are now stable and powerful enough to be useful.
The frontend consists of two parts, the server and the client. The
server is implemented in Python using the Django web framework, and is
contained under frontend/ and frontend/afe. The client is implemented
using Google Web Toolkit and is contained under frontend/client. We
tried to use Dojo initially, as was generally agreed on, but it proved
to be too young and poorly documented of a framework, and developing in
it was just too inefficient.
The scheduler is contained entirely in the scheduler/monitor_db Python
file. It looks at the database used by the web frontend and spawns
autoserv processes to run jobs, monitoring them and recording status.
The scheduler was developed by Paul Turner, who will be sending out some
detailed documentation of it soon.
I've also included the DB migrations code for which I recently submitted
a patch to the mailing list. I've included this code because it is
necessary to create the frontend DB, but it will (hopefully) soon be
part of the SVN repository.
Lastly, there is an apache directory containing configuration files for
running the web server through Apache.
I've put instructions for installing a full Autotest server, including
the web frontend, the scheduler, and the existing "tko" results backend,
at http://test.kernel.org/autotest/AutotestServerInstall. I've also
created a brief guide to using the web frontend, with plenty of
screenshots, at http://test.kernel.org/autotest/WebFrontendHowTo.
Please let me know if you find any problems with either of these pages.
Take a look, try it out, send me comments, complaints, and bugs, and I
hope you find it useful!
Steve Howard, and the rest of the Google Autotest team
From: Steve Howard <showard@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@1242 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/control_file.py
|