| d53e1492e481c3e9d07ba790577914ba20d7631a |
|
15-Dec-2015 |
MK Ryu <mkryu@google.com> |
[autotest] Consistent lock_time for a host
In the old code, 'lock_time' field for a host is recorded
when django saves the modified host record. It can sometimes
generate a slight lag (a couple seconds) for lock_time
between the master and a shard DB. This CL makes it always
consistent.
BUG=chromium:535651
TEST=Puppylab. Try locking a sharded host with a test code that
delays the message (modify_host_local) forwarding to a shard.
Change-Id: I33885f659c7511ae136493228380b459dae001d5
Reviewed-on: https://chromium-review.googlesource.com/318387
Commit-Ready: Mungyung Ryu <mkryu@google.com>
Tested-by: Mungyung Ryu <mkryu@google.com>
Reviewed-by: Mungyung Ryu <mkryu@google.com>
/external/autotest/frontend/afe/models.py
|
| 5c7ef30e9d5ad3fd49a57b3ca08520875b2cce6d |
|
11-Jul-2015 |
MK Ryu <mkryu@google.com> |
[autotest] Forward the job created by specifying host label to shard
On AFE, we can create a job by specifying a host label.
Such jobs are not currently forwarded to a shard.
By creating a record in afe_jobs_dependency_labels table for the job,
we can make it forwarded.
BUG=chromium:508770
TEST=Puppylab. Create a job with a meta_host from AFE and check if
the job is forwarded to the shard.
DEPLOY=apache
Change-Id: I86388abb7f685f8c6950364ebed95e6b4081f143
Reviewed-on: https://chromium-review.googlesource.com/284924
Reviewed-by: Mungyung Ryu <mkryu@google.com>
Commit-Queue: Mungyung Ryu <mkryu@google.com>
Tested-by: Mungyung Ryu <mkryu@google.com>
Reviewed-by: Fang Deng <fdeng@chromium.org>
/external/autotest/frontend/afe/models.py
|
| 2e3e6059f397ba791a2f9c836ed38bb0e3dfe34b |
|
10-Jul-2015 |
MK Ryu <mkryu@google.com> |
[autotest] Frontend job is not forwared to its shard
frontend.afe.models.Job.assign_to_shard() had a bug.
BUG=chromium:508697
TEST=puppylab. Create a frontend job to a shard host.
Change-Id: I5218af68dff3531202c25b1f960766b8f810970b
Reviewed-on: https://chromium-review.googlesource.com/284674
Trybot-Ready: Mungyung Ryu <mkryu@google.com>
Tested-by: Mungyung Ryu <mkryu@google.com>
Reviewed-by: Fang Deng <fdeng@chromium.org>
Commit-Queue: Mungyung Ryu <mkryu@google.com>
/external/autotest/frontend/afe/models.py
|
| 2b79fd7627875033b2650c302e385712976aea7f |
|
21-May-2015 |
Fang Deng <fdeng@chromium.org> |
[autotest] Optimize shard queries
The django filters are slow as it translates
to a sql that searches through entire afe_host_queue_entries
table in order to exclude completed jobs, non-aborted known jobs
and active jobs.
This CL optimizes the quries.
DEPLOY=apache
BUG=chromium:490485
TEST=1. Test in puppy lab. Shard sync jobs and frontend jobs correctly.
Aborting jobs works.
2. manually execute the resulting SQL and verify return value.
3. python
>>> shard =
models.Shard.objects.filter(hostname='chromeos-server14.mtv.corp.google.com')[0]
>>> q = models.Job.objects.filter(dependency_labels=shard.labels.all(),
hostqueueentry__isnull=False)
>>> q = models.Job._add_filters_for_shard_assignment(q, [333])
The above query takes 30 secs in prod.
4. unittest
With the raw sql, it takes 5 secs.
Change-Id: I95297f26a7f81b48e1ad18f7247b0e3d19751a9d
Reviewed-on: https://chromium-review.googlesource.com/272516
Tested-by: Fang Deng <fdeng@chromium.org>
Reviewed-by: Dan Shi <dshi@chromium.org>
Commit-Queue: Fang Deng <fdeng@chromium.org>
/external/autotest/frontend/afe/models.py
|
| 8f8cdb425db94473f50579e2123f61831fac7322 |
|
12-May-2015 |
MK Ryu <mkryu@google.com> |
[autotest] Getting mixed HQEs and special tasks fail on shard AFE
BUG=chromium:486926
TEST=Prepare shard testing cluster. Run a dummy suite on a DUT
that is sharded, and check if I can get mixed HQEs and special tasks
on the shard AFE.
Change-Id: I66449e07d6f5ae4c5e39ee87d4608b8eae9777f3
Reviewed-on: https://chromium-review.googlesource.com/270342
Reviewed-by: Dan Shi <dshi@chromium.org>
Tested-by: Mungyung Ryu <mkryu@google.com>
Commit-Queue: Mungyung Ryu <mkryu@google.com>
/external/autotest/frontend/afe/models.py
|
| 638a6cf6935329fd726bade7a87a45a180218f29 |
|
08-May-2015 |
MK Ryu <mkryu@google.com> |
[autotest] Shard supports multi board labels
A single chromeos board can have more than one board label.
For example, stumpy DUTs can have either "board:stumpy" or
"board:stumpy_freon".
DUTs with "board:stumpy" and those with "board:stumpy_freon" can
be swapped each other by DUT management scripts.
Current sharding design requires those DUTs be in the same shard.
Therefore, one shard should support multi board labels.
BUG=chromium:485817
TEST=Prepare DUTs with "board:stumpy" and DUTs with "board:stumpy_freon".
Create a shard that has both board labels.
Run dummy suite and bvt-cq suite in puppylab and in testing machines
using real DUTs.
Change-Id: I6a7d04b1afd9b23fa4ea4e21358d50d31e82388f
Reviewed-on: https://chromium-review.googlesource.com/270114
Tested-by: Mungyung Ryu <mkryu@google.com>
Reviewed-by: Fang Deng <fdeng@chromium.org>
Commit-Queue: Mungyung Ryu <mkryu@google.com>
/external/autotest/frontend/afe/models.py
|
| 0c1a37dd9b1237fe8d43c7f911ce601104806339 |
|
30-Apr-2015 |
MK Ryu <mkryu@google.com> |
[autotest] Support sharding for getting HQEs and special tasks
Following 2 RPCs that are used by AFE java client are improved
to support sharding.
get_host_queue_entries_and_special_tasks
get_num_host_queue_entries_and_special_tasks
BUG=chromium:462819
DEPLOY=afe,apache
TEST=puppylab. Get HQEs and special tasks for a lumpy host
which is sharded from the master AFE.
Change-Id: I68c0d1a70fb6b61034c755d3a2b7f26475994bb0
Reviewed-on: https://chromium-review.googlesource.com/268523
Reviewed-by: Mungyung Ryu <mkryu@google.com>
Commit-Queue: Mungyung Ryu <mkryu@google.com>
Tested-by: Mungyung Ryu <mkryu@google.com>
/external/autotest/frontend/afe/models.py
|
| 6818633834ad52c3de153235639ea9299a6e9a6d |
|
28-Apr-2015 |
Matthew Sartori <msartori@chromium.org> |
[autotest] Require lock reason to lock device
When locking a device, a locking reason now must be provided.
This applies to both adding a new device, and modifying an
existing device from both the web frontend and the atest
command-line tool.
BUG=chromium:336805
DEPLOY=migrate
TEST=Tested adding locked/unlocked devices and locking/unlocking devices
from both the web frontend and using the 'atest host ...' command-line tools.
Change-Id: I3a8cd8891a2999f026dd709ae8a79e2b8cbc251a
Reviewed-on: https://chromium-review.googlesource.com/267595
Tested-by: Matthew Sartori <msartori@chromium.org>
Reviewed-by: Dan Shi <dshi@chromium.org>
Commit-Queue: Matthew Sartori <msartori@chromium.org>
/external/autotest/frontend/afe/models.py
|
| 06a4b52b21f663027da8a2b64c552b63224896d0 |
|
25-Apr-2015 |
MK Ryu <mkryu@google.com> |
[autotest] Exclude jobs that don't have HQE in heartbeat response
For a heartbeat response, a master included jobs
that don't have HQEs. HQE records can be accidently removed.
This CL makes autotest tolerable to such an accident.
BUG=chromium:479766
DEPLOY=apache
TEST=puppylab
Change-Id: Id1c75c9cd869e91552749f8fc7ea3bb14b810bb1
Reviewed-on: https://chromium-review.googlesource.com/267290
Reviewed-by: Fang Deng <fdeng@chromium.org>
Commit-Queue: Mungyung Ryu <mkryu@google.com>
Tested-by: Mungyung Ryu <mkryu@google.com>
/external/autotest/frontend/afe/models.py
|
| c9e1714424621786322d0e6ea48ac167bf35ce16 |
|
19-Feb-2015 |
Dan Shi <dshi@chromium.org> |
[autotest] Add a new field require_ssp in afe_jobs table
When a job is created, data in control file is parsed and saved in afe_jobs.
scheduler picks up the database row and tries to compile the commandline for
autoserv to run. require_ssp (require server-side packaging) is a new field
will be added to control file to force to use/do not use ssp when running
a server side test.
If the value is True or None(default), server-side packaging will be applied
to run server side test, if global config AUTOSERV/enable_ssp_container is
set to True.
Since the attribute can only be retrieved when parsing control file, it needs
to be stored in afe_jobs table. Therefore, the new field is added to the table.
require_ssp is nullable and has no default value, so the sql command should
only take several minutes to finish in production database.
BUG=chromium:453624
TEST=local run migrate.py sync and migrate.py sync 97
run scheduler locally and run a dummy suite, confirm scheduler and other
components work without issue (refer to bug 459523).
DEPLOY=migrate, db must be migrated after this CL is pushed to prod. Otherwise
scheduler will fail.
Change-Id: I7c65150b7ee3309310fdbed42b5079e564845a65
Reviewed-on: https://chromium-review.googlesource.com/251340
Trybot-Ready: Dan Shi <dshi@chromium.org>
Tested-by: Dan Shi <dshi@chromium.org>
Reviewed-by: Fang Deng <fdeng@chromium.org>
Commit-Queue: Dan Shi <dshi@chromium.org>
/external/autotest/frontend/afe/models.py
|
| ff36159d40fc7e7b5124f4950b6a18103aebd922 |
|
03-Feb-2015 |
Fang Deng <fdeng@chromium.org> |
[autotest] Sync based on key instead of id for attribute-like models
HostAttribute and JobKeyval are updated on both shard and
master. Both table should be synced based on "key" instead
auto-incremental id.
BUG=chromium:453122
DEPLOY=shard_cient, apache
TEST=python
>>import common
>>from autotest_lib.frontend import setup_django_environment
>>from autotest_lib.frontend.afe import models
>>m = models.Host.objects.get(id=20)
>>s = m.serialize()
>>s['hostattribute_set'] = A LIST OF NEW ATTRIBUTE RECORDS
>>m = models.Host.deserialize(s)
>>m.hostattribute_set.all()[0].__dict__
>>j = models.Job.objects.get(id=35)
>>s = j.serialize()
>>s['jobkeyval_set'] = A LIST OF NEW JOBKEYVAL RECORDS
>>m = models.Job.deserialize(s)
>>m.jobkeyval_set.all()[0].__dict__
TEST=Run a shard, schedule a dummy suite, ensure everything still
works.
Change-Id: I9c6450f93f06510477312bd02b119fd96b70857c
Reviewed-on: https://chromium-review.googlesource.com/245560
Reviewed-by: Fang Deng <fdeng@chromium.org>
Tested-by: Fang Deng <fdeng@chromium.org>
Commit-Queue: Fang Deng <fdeng@chromium.org>
/external/autotest/frontend/afe/models.py
|
| b72f4fbcf1583da27f09f4abb9d8162530bf4559 |
|
21-Jan-2015 |
Gabe Black <gabeblack@chromium.org> |
graphite: Reorganize the elastic search code so we can put it in chromite.
This change reorganizes the elastic search integration code so that it's
separate from the code that, for instance, reads config information from the
autotest global config. That way, it can be moved from chromite without
breaking any dependencies.
BUG=chromium:446291
TEST=Ran stats_es_functionaltest.py. Ran unit tests. Ran a butterfly-paladin
tryjob with --hwtest.
Change-Id: I0dbf135c4f1732d633e5fc9d5edb9e1f4f7199d5
Reviewed-on: https://chromium-review.googlesource.com/242701
Reviewed-by: Dan Shi <dshi@chromium.org>
Tested-by: Gabe Black <gabeblack@chromium.org>
Commit-Queue: Gabe Black <gabeblack@chromium.org>
/external/autotest/frontend/afe/models.py
|
| 8c98ac10beaa08bfb975c412b0b3bda23178763a |
|
23-Dec-2014 |
Prashanth Balasubramanian <beeps@google.com> |
[autotest] Send frontend jobs to shards.
Frontend jobs on hosts that are on the shard are disallowed
currently, because the host-scheduler on master currently
ignore jobs based on meta-host, but frontend jobs have no
meta-host. This CL have the following changes:
- Make host-scheduler ignore frontend jobs that are supposed
to be picked by shard.
- Send such frontend jobs in heartbeat.
- Allow creation of frontend jobs in rpc.
TEST=Test the follows:
- Create a job on a host on shard from AFE frontend.
Observe it runs on shards and completes on master.
- Create a job on two hosts (one host on shard, the other on master)
from AFE frontend. Make sure exception is railed with correct
message.
- Run a normal dummy suite on shard, make sure normal flow still
works. Heartbeat contains the right information.
- Run a normal dummy suite on master, make sure it works.
BUG=chromium:444790
DEPLOY=apache, host-scheduler
Change-Id: Ibca3d36cb59fed695233ffdc89506364c402cc37
Reviewed-on: https://chromium-review.googlesource.com/240396
Reviewed-by: Mungyung Ryu <mkryu@google.com>
Reviewed-by: Dan Shi <dshi@chromium.org>
Commit-Queue: Fang Deng <fdeng@chromium.org>
Tested-by: Fang Deng <fdeng@chromium.org>
/external/autotest/frontend/afe/models.py
|
| 86248502517d42d6c036a66a57e3a24d4a6dc3b8 |
|
19-Dec-2014 |
Fang Deng <fdeng@chromium.org> |
[autotest] Serialize foreign key we don't want to follow
In some case, we don't want to follow a relationship when
serialize and de-serialize an object. However, we may
want to serialize the value of the foreign key value.
For example, we follow Host.hostattribute_set relationship to
serialize HostAttribute objects associated with a host. We
don't want to follow HostAttribute.host_id to go back to Host.
This CL allows a model to claim foreign keys that will be
serialized but will not be followed.
BUG=None
TEST=In python
>>> import common
>>> from autotest_lib.frontend import setup_django_environment
>>> from autotest_lib.frontend.afe import models
>>> models.HostAttribute.objects.filter(host_id=20)[0].serialize()
{'host_id': 20L, u'id': 203L, 'value': u'2271', 'attribute':
u'need_crash_log'}
>>> models.HostAttribute.deserialize({'host_id': 20L, u'id': 251L,
'value': u'2271', 'attribute': u'need_crash_log'})
<HostAttribute: HostAttribute object>
TEST=Test shard heartbeat works when host attributes present.
DEPLOY=apache, shard_client
Change-Id: I8e0fbb1b7eb037f0968c1599334446301ddbbcb0
Reviewed-on: https://chromium-review.googlesource.com/236708
Reviewed-by: Fang Deng <fdeng@chromium.org>
Tested-by: Fang Deng <fdeng@chromium.org>
Reviewed-by: Prashanth B <beeps@chromium.org>
Tested-by: Prashanth B <beeps@chromium.org>
Commit-Queue: Prashanth B <beeps@chromium.org>
Trybot-Ready: Prashanth B <beeps@chromium.org>
/external/autotest/frontend/afe/models.py
|
| 6964fa53798525f39cd490e26481e8d923815406 |
|
18-Dec-2014 |
Dan Shi <dshi@chromium.org> |
[autotest] Use stable version from afe_stable_versions table
Add a util to get stable version from afe_stable_versions table. If the table
is not configured, return the global config value CROS.stable_cros_version.
Add an RPC get_stable_version(board) to get stable version for given board.
If board is not specified, default stable version will be returned.
Update calls in moblab_RunSuite and cros_host to get stable version through
the new RPC.
BUG=chromium:436656
DEPLOY=apache
CQ-DEPEND=CL:236571
TEST=local rpc test, run repair, confirm repair uses stable version from
afe_stable_versions table.
Change-Id: I6a22a4b471060297403cc1ae7fcbf96bec6563d9
Reviewed-on: https://chromium-review.googlesource.com/236582
Reviewed-by: Dan Shi <dshi@chromium.org>
Commit-Queue: Dan Shi <dshi@chromium.org>
Trybot-Ready: Dan Shi <dshi@chromium.org>
Tested-by: Dan Shi <dshi@chromium.org>
/external/autotest/frontend/afe/models.py
|
| af5166418be945356238768cfc8226f3c2aa7138 |
|
13-Dec-2014 |
Prashanth Balasubramanian <beeps@google.com> |
[autotest] Delete shard user if it differs from master.
For most models the shard starts off with an empty database
and copies over records from the master. There is one exception,
afe_users, this is because users are proactively created when
someone navigates to the frontend. So even though we start
off with a clean afe_users table the `debug_user` will be
inserted the first time anyone visits the shard frontend. The id
of this record can be different from the id on the master.
To avoid this race, before saving any user record for the first
time check if a matching user already exists (matching id or login)
and simply delete that user before saving the new one.
TEST=Ran suites on shard.
BUG=chromium:423225
DEPLOY=apache
Change-Id: Ibe1d452bbba037c88e6525c749f27a2f1f5ca2c3
Reviewed-on: https://chromium-review.googlesource.com/235533
Tested-by: Prashanth B <beeps@chromium.org>
Reviewed-by: Dan Shi <dshi@chromium.org>
Commit-Queue: Prashanth B <beeps@chromium.org>
Trybot-Ready: Prashanth B <beeps@chromium.org>
/external/autotest/frontend/afe/models.py
|
| 6edaaf9f5d10cda21d1c7f2401eb1d0bef2192fa |
|
25-Nov-2014 |
Prashanth Balasubramanian <beeps@google.com> |
[autotest] Disallow frontend jobs on hosts given to shards.
TEST=Ran jobs and checked exceptions.
Ran jobs on non-shard hosts.
BUG=chromium:431789
DEPLOY=apache
Change-Id: Ide385ed1db135a7e98ab0385df1f9a64d97bd631
Reviewed-on: https://chromium-review.googlesource.com/231735
Tested-by: Prashanth B <beeps@chromium.org>
Reviewed-by: Dan Shi <dshi@chromium.org>
Reviewed-by: Fang Deng <fdeng@chromium.org>
Commit-Queue: Prashanth B <beeps@chromium.org>
/external/autotest/frontend/afe/models.py
|
| 5949b4af7a872aeb58e7ad29090812d648725ed5 |
|
23-Nov-2014 |
Prashanth Balasubramanian <beeps@google.com> |
[Autotest] Allow the syncing of labels added/removed.
This allows us to perform label add/remove actions
on the master and have it pass down to the shards those hosts
are on by introducing a generic decorator.
The cl also fixes some logging issues with the shard-client,
and a bug with propogating the invalid bit of a host.
TEST=Added/removed labels.
BUG=chromium:431786
DEPLOY=apache
Change-Id: Ic176054e04a5508bb3e967ecc902422628446d32
Reviewed-on: https://chromium-review.googlesource.com/231558
Reviewed-by: Prashanth B <beeps@chromium.org>
Tested-by: Prashanth B <beeps@chromium.org>
Commit-Queue: Prashanth B <beeps@chromium.org>
/external/autotest/frontend/afe/models.py
|
| de87dea6a5a82b4ce1ddc5ce881ddbc70b54fdda |
|
10-Nov-2014 |
Prashanth Balasubramanian <beeps@google.com> |
[autotest] Include the time created as a uid in task results.
Since tasks will run directly on shards there is some chance
that a shard will failover and its successor clobbers the
special task logs of the dead shard in google storage.
To avoid this scenario, include the creation time of the
task as a uid in the path to the logs. Unless one of the
shards is living in the past these timestamps should never
repeat. Moreover, we take measures against clock drift (ntpd)
and if a shard has the wrong system time configuration all its
jobs will timeout anyway, since each job has a TTL based off
time_created which is set on the master.
TEST=Ran suites.
BUG=chromium:423225,chromium:425347
DEPLOY=scheduler,apache
Change-Id: Ia23ac8fd721f53fbb9b475c8eb9f8d25e4fd1c2f
Reviewed-on: https://chromium-review.googlesource.com/228781
Tested-by: Prashanth B <beeps@chromium.org>
Reviewed-by: Dan Shi <dshi@chromium.org>
Commit-Queue: Prashanth B <beeps@chromium.org>
/external/autotest/frontend/afe/models.py
|
| 02e6129187328bd3edd96dae8062e6a19ab7936c |
|
17-Oct-2014 |
Jakob Juelich <jakobjuelich@chromium.org> |
[autotest] Allow job updates from shard without shard_id.
In the following scenario currently an exception is raised:
A job was synced from the master to a shard. It was then aborted
on the master, while the shard was executing it. The abortion
sets the shard_id of the job back to null (that happens always
when a job is set to a completed status).
The there was no heartbeat between the abortion and the completion
of the job on the shard.
The next heartbeat will find the completed job on the shard and will
try uploading it to the master. The sanity check if the job is a
assigned to the shard it was uploaded from will fail, as the shard
id is set to null on the master.
This changes this behavior: The sanity check will not fail if
the shard doesn't have a shard_id set.
BUG=None
DEPLOY=afe
TEST=Ran Suites.
Change-Id: Ifd8eafd3898e7fadab6e52e48bc033acd5ae1dd5
Reviewed-on: https://chromium-review.googlesource.com/224110
Tested-by: Jakob Jülich <jakobjuelich@chromium.org>
Reviewed-by: Dan Shi <dshi@chromium.org>
Commit-Queue: Jakob Jülich <jakobjuelich@chromium.org>
/external/autotest/frontend/afe/models.py
|
| 1b52574752be108a743d3b33561c34324f8538e7 |
|
30-Sep-2014 |
Jakob Juelich <jakobjuelich@chromium.org> |
[autotest] Retransmit hosts and jobs if heartbeat failed on client.
If the client fails while performing a heartbeat, the master will
have already set the shard_id and therefore won't ever send jobs or
hosts again.
This changes this behavior: The shard client sends ids of hosts and
incomplete jobs it already has in the heartbeat.
Objects with id's that weren't sent, will be returned by the
heartbeat regardless of already having a shard_id set.
BUG=None
DEPLOY=apache
TEST=Ran suites
Change-Id: I46bbb13a81886476ec48c6f879f123290769b659
Reviewed-on: https://chromium-review.googlesource.com/220692
Tested-by: Jakob Jülich <jakobjuelich@chromium.org>
Reviewed-by: Prashanth B <beeps@chromium.org>
Commit-Queue: Jakob Jülich <jakobjuelich@chromium.org>
/external/autotest/frontend/afe/models.py
|
| 852ec0d282735523b140e9d42c2ef3e36d1baf24 |
|
16-Sep-2014 |
Jakob Juelich <jakobjuelich@chromium.org> |
[autotest] Re-sync aborted jobs to shards
If a job is aborted on the master, the shards need to know, so
they can abort the job.
With this commit jobs that are already synced but have been aborted
in the meantime will be re-synced.
BUG=None
TEST=Ran suites
Change-Id: I077400047ba51b19151035f0f58c585959ceba7f
Reviewed-on: https://chromium-review.googlesource.com/218390
Tested-by: Jakob Jülich <jakobjuelich@chromium.org>
Reviewed-by: Fang Deng <fdeng@chromium.org>
Commit-Queue: Jakob Jülich <jakobjuelich@chromium.org>
/external/autotest/frontend/afe/models.py
|
| a0acfbcb08a96fbe13eeb68203e119f71f1ce56c |
|
15-Oct-2014 |
Dan Shi <dshi@chromium.org> |
[autotest] log the user that lock/unlock a dut to metaDB
BUG=None
TEST=local setup, lock and unlock couple times, then check metaDB to see the
data.
http://172.25.61.45:9200/_plugin/elastic-hammer/
with query:
{"query": {"bool": {"minimum_should_match": 1,
"should": [{"term": {"_type": "lock_history"}}]}},
"size": 100,
"sort": [{"time_recorded": "desc"}]}
Confirm response has entries like:
changed_by: "debug_user"
hostname: "172.27.213.193"
locked: true
time_recorded: 1413327364.848333
Change-Id: I643d5aefa37494170000729e2083dad799e1b8f5
Reviewed-on: https://chromium-review.googlesource.com/223371
Commit-Queue: Dan Shi <dshi@chromium.org>
Tested-by: Dan Shi <dshi@chromium.org>
Reviewed-by: Fang Deng <fdeng@chromium.org>
/external/autotest/frontend/afe/models.py
|
| acf359283e28219cc42ff3fe82b8a53f6a2d90f3 |
|
03-Oct-2014 |
MK Ryu <mkryu@google.com> |
[autotest] Add a new RPC interface, get_host_attribute.
Retrieve a host's attribute from afe_host_attributes table.
When you need to get only attribute info of a host, this RPC call is
lighter than using get_hosts RPC call.
DEPLOY=apache
BUG=chromium:215160
TEST=AUTOTEST_ROOT$ python
>>import common
>>from autotest_lib.server.cros.dynamic_suite import tools, frontend_wrappers
>>afe = frontend_wrappers.RetryingAFE(timeout_min=5, delay_sec=10)
>>afe.get_host_attribute('need_crash_logs', hostname=<host ip>)
Change-Id: I86bc17091c078823c270206d80bd572f2fbbeb95
Reviewed-on: https://chromium-review.googlesource.com/221510
Reviewed-by: Fang Deng <fdeng@chromium.org>
Commit-Queue: Mungyung Ryu <mkryu@google.com>
Tested-by: Mungyung Ryu <mkryu@google.com>
/external/autotest/frontend/afe/models.py
|
| f865d33cc3b0584f10b2f6f4de414a6d7f9e3000 |
|
29-Sep-2014 |
Jakob Juelich <jakobjuelich@chromium.org> |
[autotest] Allow updating selected model attributes
Right now, objects that have been persisted on the client once,
will never be overwritten again with new data, that comes from
the master.
In order to sync i.e. aborted bits from the master to the slave,
this needs to change.
This introduces a new variable SERIALIZATION_LOCAL_LINKS_TO_UPDATE,
which objects can overwrite with a set of names of attributes,
that should be overwritten with data that's sent from the master.
BUG=None
DEPLOY=shard_client
TEST=Ran suites.
Change-Id: I6b461475116db0acd2de7b8fe628aed791e7c418
Reviewed-on: https://chromium-review.googlesource.com/220331
Tested-by: Jakob Jülich <jakobjuelich@chromium.org>
Reviewed-by: Fang Deng <fdeng@chromium.org>
Commit-Queue: Jakob Jülich <jakobjuelich@chromium.org>
/external/autotest/frontend/afe/models.py
|
| 8421d5905ab0aed8689c2eea6be8d9c4042ce618 |
|
18-Sep-2014 |
Jakob Juelich <jakobjuelich@chromium.org> |
[autotest] Sync completed jobs back to master
Records of jobs that are completed should be send back to the master
upon a heartbeat.
This changes the heartbeat to include these things.
BUG=None
CQ-DEPEND=CL:212725
DEPLOY=apache,afe
TEST=Ran suites, tried out manually
Change-Id: I25daed310dc1bad064bd0c4819a43d18ba04606d
Reviewed-on: https://chromium-review.googlesource.com/218730
Tested-by: Jakob Jülich <jakobjuelich@chromium.org>
Reviewed-by: Fang Deng <fdeng@chromium.org>
Commit-Queue: Jakob Jülich <jakobjuelich@chromium.org>
/external/autotest/frontend/afe/models.py
|
| a94efe60bc94c9aa10ecfe40bddf97518985c7c2 |
|
19-Sep-2014 |
Jakob Juelich <jakobjuelich@chromium.org> |
[autotest] Endpoints for syncing jobs and hqes back to master
The status of jobs that were completed on a shard should be visible
in the master AFE, so a human can easily see which jobs have been run in one
central place.
This adds the functionality to the AFE to retrieve newer versions of
jobs and hqes from shards.
This is a newer version of CL:213901 which was reverted because of
http://crbug.com/418022.
BUG=418022
DEPLOY=apache, afe
TEST=Ran Suites
Change-Id: I3c9a35c78585a43d193874decbb7b16216b2c69a
Reviewed-on: https://chromium-review.googlesource.com/220141
Reviewed-by: Fang Deng <fdeng@chromium.org>
Tested-by: Jakob Jülich <jakobjuelich@chromium.org>
Commit-Queue: Jakob Jülich <jakobjuelich@chromium.org>
/external/autotest/frontend/afe/models.py
|
| e5482e5614e47a1eda238902eaa83e16a0f5475d |
|
26-Sep-2014 |
Owen Lin <owenlin@chromium.org> |
Revert "[autotest] Endpoints for syncing jobs and hqes back to master"
This reverts commit 1b22ff2921c65b45e3727a8aaa852325f0f0cfb4.
BUG=418022
Change-Id: I30473d21d16911f620ee7f06917c13e6017f20fb
Reviewed-on: https://chromium-review.googlesource.com/219918
Reviewed-by: Owen Lin <owenlin@chromium.org>
Commit-Queue: Owen Lin <owenlin@chromium.org>
Tested-by: Owen Lin <owenlin@chromium.org>
/external/autotest/frontend/afe/models.py
|
| 1b22ff2921c65b45e3727a8aaa852325f0f0cfb4 |
|
19-Sep-2014 |
Jakob Juelich <jakobjuelich@chromium.org> |
[autotest] Endpoints for syncing jobs and hqes back to master
The status of jobs that were completed on a shard should be visible
in the master AFE, so a human can easily see which jobs have been run in one
central place.
This adds the functionality to the AFE to retrieve newer versions of
jobs and hqes from shards.
BUG=None
DEPLOY=apache, afe
TEST=Ran Suites
Change-Id: I04c3401921a3f54bae6070639af805904952bf6a
Reviewed-on: https://chromium-review.googlesource.com/213901
Reviewed-by: Fang Deng <fdeng@chromium.org>
Commit-Queue: Jakob Jülich <jakobjuelich@chromium.org>
Tested-by: Jakob Jülich <jakobjuelich@chromium.org>
/external/autotest/frontend/afe/models.py
|
| 116ff0fc2674082fe475afd64ce4dec998ed71b7 |
|
18-Sep-2014 |
Jakob Juelich <jakobjuelich@chromium.org> |
[autotest] Deserializer: Allow updating objects
Currently the state of an object is never changed by the
deserializer.
After a shard executed a job, the master should be informed
about it's result so one can see if a job has been executed or not
in one central place for all jobs.
To update a job's status on the master, this is necessary.
Doing this, a bug of the previous implementation became apparent:
Deserializing a host object while no AclGroups were present yet
would create an Everyone-AclGroup in it's save() method. This is
dangerous because this way the states of master and shards might
diverge, specifically AclGroups with same names and different ids
might occur, which will then, upon dumping the Everyone-AclGroup from
the master, will cause exceptions to be raised.
Therefore just not running them prevents errors.
BUG=None
DEPLOY=scheduler,apache,host_scheduler
TEST=Ran suites, tested with real data on shard
Change-Id: Ie8ccd3205ac0efd5bbba4402da5de4c2615b495c
Reviewed-on: https://chromium-review.googlesource.com/218806
Reviewed-by: Prashanth B <beeps@chromium.org>
Commit-Queue: Jakob Jülich <jakobjuelich@chromium.org>
Tested-by: Jakob Jülich <jakobjuelich@chromium.org>
/external/autotest/frontend/afe/models.py
|
| 59cfe5497d7f3be2bc4ade144d649a131e88448e |
|
03-Sep-2014 |
Jakob Juelich <jakobjuelich@chromium.org> |
[autotest] Add heartbeat AFE endpoint to shard Autotest
To improve the workload the autotest setup can handle, jobs should
be executed by multiple shards. They are sharded by type of board.
This a HTTP accessible endpoint to AFE. This assigns jobs and hosts
to moblabs and returns them as records to insert into a shard's
local database.
TEST=Ran suites and tried manually retrieving jobs
DEPLOY=scheduler,apache,host-scheduler
Change-Id: I20ac41364e4c67e17883729181af798e18d8093e
Reviewed-on: https://chromium-review.googlesource.com/217968
Reviewed-by: Jakob Jülich <jakobjuelich@chromium.org>
Reviewed-by: Fang Deng <fdeng@chromium.org>
Commit-Queue: Jakob Jülich <jakobjuelich@chromium.org>
Tested-by: Jakob Jülich <jakobjuelich@chromium.org>
/external/autotest/frontend/afe/models.py
|
| f88fa938050d0e2b662a2b32ef7e931e01a4d8fe |
|
04-Sep-2014 |
Jakob Juelich <jakobjuelich@chromium.org> |
[autotest] Deserialize records and persist them
To send records from the master to shards it's necessary to serialize
them. This adds the deserialization functionality.
TEST=Ran suites.
DEPLOY=apache
Change-Id: I2806b0cfe1e4fbfba5d89e6d422800c7637ed4e9
Reviewed-on: https://chromium-review.googlesource.com/216355
Reviewed-by: Prashanth B <beeps@chromium.org>
Tested-by: Jakob Jülich <jakobjuelich@chromium.org>
Commit-Queue: Jakob Jülich <jakobjuelich@chromium.org>
/external/autotest/frontend/afe/models.py
|
| 23b2895d2c3819a0191deab727327bcb894a0f99 |
|
13-Sep-2014 |
Fang Deng <fdeng@chromium.org> |
This is suspect to break our afe "debug" link. I am going to revert it for now.
Revert "[autotest] Add heartbeat AFE endpoint to shard Autotest"
This reverts commit 1e10d745c65ecafa79bd4f0b4f0b743bd5f1eff3.
Change-Id: I81fd554a910c8c5b7537b34ec13301fcf15be3fd
Reviewed-on: https://chromium-review.googlesource.com/217980
Reviewed-by: Fang Deng <fdeng@chromium.org>
Commit-Queue: Fang Deng <fdeng@chromium.org>
Tested-by: Fang Deng <fdeng@chromium.org>
/external/autotest/frontend/afe/models.py
|
| 1e10d745c65ecafa79bd4f0b4f0b743bd5f1eff3 |
|
03-Sep-2014 |
Jakob Juelich <jakobjuelich@chromium.org> |
[autotest] Add heartbeat AFE endpoint to shard Autotest
To improve the workload the autotest setup can handle, jobs should
be executed by multiple shards. They are sharded by type of board.
This a HTTP accessible endpoint to AFE. This assigns jobs and hosts
to moblabs and returns them as records to insert into a shard's
local database.
TEST=Ran suites and tried manually retrieving jobs
DEPLOY=scheduler,apache,host-scheduler
Change-Id: Ie1dfa7bb295f685c56bb1354675d83f82d933c65
Reviewed-on: https://chromium-review.googlesource.com/211214
Reviewed-by: Alex Miller <milleral@chromium.org>
Tested-by: Jakob Jülich <jakobjuelich@chromium.org>
Commit-Queue: Jakob Jülich <jakobjuelich@chromium.org>
/external/autotest/frontend/afe/models.py
|
| 3bb7c8095d7c66d955462673dd3df5565c1b4a96 |
|
03-Sep-2014 |
Jakob Juelich <jakobjuelich@chromium.org> |
[autotest] Add serialization functionality including dependencies
In order to send hosts and jobs to shards it's necessary to
serialize them including their dependencies.
This changelist adds functionality to find an object's dependencies
and to serialize it into lists and dicts, so it can be converted
to json.
TEST=Ran suites.
DEPLOY=apache
Change-Id: Icc158de1f23c7483ebc22b6ca3482b7ffccdd5c4
Reviewed-on: https://chromium-review.googlesource.com/215998
Tested-by: Jakob Jülich <jakobjuelich@chromium.org>
Reviewed-by: Alex Miller <milleral@chromium.org>
Commit-Queue: Jakob Jülich <jakobjuelich@chromium.org>
/external/autotest/frontend/afe/models.py
|
| de2b9a93674cb762230a83b1c41cc373073e4fdb |
|
03-Sep-2014 |
Jakob Juelich <jakobjuelich@chromium.org> |
[autotest] Add shard_id to hosts
To assign hosts to shards and avoid reassigning them again and again
all the time, somehow it must be stored if a host was already synced
to a shard.
This changelist adds a column in afe_hosts to reference the shard
it has been synced to.
TEST=Ran suites and manually tried db migrations and running dummy.
DEPLOY=scheduler,apache,host-scheduler,migrate
Change-Id: Ide181a221f363fe96130611da7732baec9680a4f
Reviewed-on: https://chromium-review.googlesource.com/215997
Reviewed-by: Alex Miller <milleral@chromium.org>
Commit-Queue: Jakob Jülich <jakobjuelich@chromium.org>
Tested-by: Jakob Jülich <jakobjuelich@chromium.org>
/external/autotest/frontend/afe/models.py
|
| 92c06334ea5e9baab4289ad70135e3f17ed9f872 |
|
25-Aug-2014 |
Jakob Jülich <jakobjuelich@chromium.org> |
Revert "Revert "[autotest] Add shard tables and models to Autotest""
This reverts commit c1a9d91f65d8fa5225ab673266d165e759c645ab.
Change-Id: Ic6e01cbf06279e19b87926a076548731677024fa
Reviewed-on: https://chromium-review.googlesource.com/214027
Reviewed-by: Alex Miller <milleral@chromium.org>
Tested-by: Jakob Jülich <jakobjuelich@chromium.org>
Commit-Queue: Jakob Jülich <jakobjuelich@chromium.org>
/external/autotest/frontend/afe/models.py
|
| c1a9d91f65d8fa5225ab673266d165e759c645ab |
|
22-Aug-2014 |
Prashanth B <beeps@chromium.org> |
Revert "[autotest] Add shard tables and models to Autotest"
Afe jobs has 7 million rows and adding a column might take upwards of 20 minutes. I would like to avoid the downtime
during the day, and we need a push asap for a servo fix
that's affecting cq. Since this isn't critical, I'm moving
it out of the way till we have space for the migrate.
This reverts commit 962e3413dce0a6ea2fb0bed87323655f1641e57c.
Change-Id: I053c59cdab314aa7bab5db37b3a4d7cada1f1c7d
Reviewed-on: https://chromium-review.googlesource.com/213791
Reviewed-by: Prashanth B <beeps@chromium.org>
Commit-Queue: Prashanth B <beeps@chromium.org>
Tested-by: Prashanth B <beeps@chromium.org>
/external/autotest/frontend/afe/models.py
|
| 962e3413dce0a6ea2fb0bed87323655f1641e57c |
|
15-Aug-2014 |
Jakob Juelich <jakobjuelich@chromium.org> |
[autotest] Add shard tables and models to Autotest
To improve the workload the autotest setup can handle, jobs should
be executed by multiple shards. They are sharded by type of board.
This is to enable autotest to manage shard entities.
TEST=Ran suites and dummy suite. Tried db migrations up and down.
DEPLOY=afe,apache,tko,scheduler
Change-Id: I9a62f98b5b8eaf111bb53a4f3e3499c4e5a647df
Reviewed-on: https://chromium-review.googlesource.com/212662
Reviewed-by: Alex Miller <milleral@chromium.org>
Tested-by: Jakob Jülich <jakobjuelich@chromium.org>
Commit-Queue: Jakob Jülich <jakobjuelich@chromium.org>
/external/autotest/frontend/afe/models.py
|
| d85d611e7560f28baf622c0f76b992e0a7fb2789 |
|
14-Jul-2014 |
Dan Shi <dshi@chromium.org> |
[autotest] Do not record time_finished if a special task is not started yet
BUG=chromium:387918
TEST=local setup
Change-Id: I4cd64792f16305e375981f01019b5485f43f7756
Reviewed-on: https://chromium-review.googlesource.com/207810
Tested-by: Dan Shi <dshi@chromium.org>
Reviewed-by: Fang Deng <fdeng@chromium.org>
Commit-Queue: Dan Shi <dshi@chromium.org>
/external/autotest/frontend/afe/models.py
|
| 51599038f08395067097dc265127cfbcf77c427d |
|
24-Jun-2014 |
Fang Deng <fdeng@chromium.org> |
[autotest] Record ending time of HostQueueEntry
Add finished_on to afe_host_queue_entries table.
Update the models.
Set the value of finished_on when the hqe is set as completed.
TEST=Run a dummy suite, observe the value is properly set in database.
BUG=chromium:387917
DEPLOY=scheduler, migrate, apache
Change-Id: I462cc0ea508dac6bdc6960b8169199ceb1091086
Reviewed-on: https://chromium-review.googlesource.com/205287
Reviewed-by: Fang Deng <fdeng@chromium.org>
Commit-Queue: Fang Deng <fdeng@chromium.org>
Tested-by: Fang Deng <fdeng@chromium.org>
/external/autotest/frontend/afe/models.py
|
| d07255493d6301c5fd58762073d70e2b8a520671 |
|
24-Jun-2014 |
Dan Shi <dshi@chromium.org> |
[autotest] Record ending time for special tasks in afe_special_tasks
Add a new field time_finished in afe_special_tasks table. The new field records
when a special task is finished.
In SpecialTask.finish method, set the value of time_finished.
BUG=chromium:387918
DEPLOY=migrate,apache
TEST=run a verify job in local setup, confirm the new field has a value of the
verify task.
Change-Id: Ic570bc8ec1c594587c8ec6ee51c31ec4ad140a6a
Reviewed-on: https://chromium-review.googlesource.com/205291
Tested-by: Dan Shi <dshi@chromium.org>
Reviewed-by: Fang Deng <fdeng@chromium.org>
Commit-Queue: Dan Shi <dshi@chromium.org>
/external/autotest/frontend/afe/models.py
|
| 8d89b64d7edb2ea7d7f2d9c59ded2e2f09ed3ea8 |
|
03-May-2014 |
Simran Basi <sbasi@chromium.org> |
[autotest] models.Job set default synch_count to 0.
Currently when creating a suite job we pass in a synch_count of
None, pass synch_count validation, then create a Job entry in
the database. However the default value for synch_count is 1 and
that is what gets recorded in the database.
This breaks the ability to create a recurring job for a suite job
as the original job's synch_count does not match what was used to
create it. In order to address this, make the default value for
synch_count be 0.
BUG=chromium:369726
TEST=./database/migrate sync and ensured afe_jobs now shows not null
for synch_count. Created a regular job from the afe, kicked off
a suite via run_suite, created a recurring suite job and a recurring
regular job. frontend_unittest.py
DEPLOY=scheduler,apache,migrate
Change-Id: I31728f9f876c917a3acaa390c7b9ffc33042bbc2
Reviewed-on: https://chromium-review.googlesource.com/198102
Reviewed-by: Simran Basi <sbasi@chromium.org>
Commit-Queue: Simran Basi <sbasi@chromium.org>
Tested-by: Simran Basi <sbasi@chromium.org>
/external/autotest/frontend/afe/models.py
|
| 489b91d72cd225e902081dbd3f9e47448fe867f6 |
|
15-Mar-2014 |
Prashanth B <beeps@google.com> |
[autotest] Establish a common interface for host representation.
This cl has the work needed to ensure that schema changes made on
the server trickled down into the client. If the same changes don't
reflect on the client, creating or saving a client host wrapper for
a given host will fail deterministically on the client side until
modules using the rdb_host are modified to reflect the changes.
1. rdb_hosts: A module containing the host heirarchy needed to
establish a dependence between the creation of the RDBServerHostWrapper
(which is serialized and returned to the client, which converts it
into an RDBClientHostWrapper) and the saving of the RDBClientHostWrapper
through and rdb update request.
2. rdb_requests: Contains the requests/request managers that were in
rdb_utils, because I plan to expand them in subsequent cls.
3. rdb_model_extensions: Contains model classes common
to both server and client that help in establishing
the common host model interface.
4. rdb integration tests.
TEST=Ran suites, unittests
BUG=chromium: 348176
DEPLOY=scheduler
Change-Id: I0bbab1dd184e505b1130ee73714e45ceb7bf4189
Reviewed-on: https://chromium-review.googlesource.com/191357
Commit-Queue: Prashanth B <beeps@chromium.org>
Tested-by: Prashanth B <beeps@chromium.org>
Reviewed-by: Dan Shi <dshi@chromium.org>
/external/autotest/frontend/afe/models.py
|
| ec21225c11b0983b563c24bb71828f5bb6bbcaee |
|
01-Mar-2014 |
Alex Miller <milleral@chromium.org> |
[autotest] Feed job labels from the scheduler to autoserv.
All job labels now get passed into reset, verify, cleanup, provision,
and repair. Everything bug provision can run standalone, and thus
passing the job labels needs to be optional for those.
BUG=chromium:334418
DEPLOY=scheduler
TEST=special tasks run fine, scheduler shows autoserv being invoked
correctly
Change-Id: I524cc8441e654685e50c935fb1d55cf87e8a2f73
Reviewed-on: https://chromium-review.googlesource.com/188455
Reviewed-by: Alex Miller <milleral@chromium.org>
Tested-by: Alex Miller <milleral@chromium.org>
Commit-Queue: Alex Miller <milleral@chromium.org>
/external/autotest/frontend/afe/models.py
|
| cc9fc70587d37775673e47b3dcb4d6ded0c6dcb4 |
|
02-Dec-2013 |
beeps <beeps@chromium.org> |
[autotest] RDB Refactor II + Request/Response API.
Scheduler Refactor:
1. Batched processing of jobs.
2. Rdb hits the database instead of going through host_scheduler.
3. Migration to add a leased column.The scheduler released hosts
every tick, back to the rdb.
4. Client rdb host that queue_entries use to track a host, instead
of a database model.
Establishes a basic request/response api for the rdb:
rdb_utils:
1. Requests: Assert the format and fields of some basic request types.
2. Helper client/server modules to communicate with the rdb.
rdb_lib:
1. Request managers for rdb methods:
a. Match request-response
b. Abstract the batching of requests.
2. JobQueryManager: Regulates database access for job information.
rdb:
1. QueryManagers: Regulate database access
2. RequestHandlers: Use query managers to get things done.
3. Dispatchers: Send incoming requests to the appropriate handlers.
Ignores wire formats.
TEST=unittests, functional verification.
BUG=chromium:314081, chromium:314083, chromium:314084
DEPLOY=scheduler, migrate
Change-Id: Id174c663c6e78295d365142751053eae4023116d
Reviewed-on: https://chromium-review.googlesource.com/183385
Reviewed-by: Prashanth B <beeps@chromium.org>
Commit-Queue: Prashanth B <beeps@chromium.org>
Tested-by: Prashanth B <beeps@chromium.org>
/external/autotest/frontend/afe/models.py
|
| 94dc003dcce29c296a071add2f397013045c240c |
|
12-Nov-2013 |
Simran Basi <sbasi@chromium.org> |
[Autotest] Update Database timeout to timeout_mins.
This change adds a timeout_mins column to the afe_jobs table and updates
exisiting jobs to include this entry.
Also updates the django models that reflect afe_jobs.
BUG=chromium:221264
DEPLOY=migrate, scheduler
TEST=Ran migration, checked column is in db, scheduler_models_unittest.py,
ran smoke suite on local AFE.
Change-Id: I09f8b6023a31dcef37e98c298dc0d63b37bc0a9e
Reviewed-on: https://chromium-review.googlesource.com/173916
Reviewed-by: Simran Basi <sbasi@chromium.org>
Tested-by: Simran Basi <sbasi@chromium.org>
Commit-Queue: Simran Basi <sbasi@chromium.org>
/external/autotest/frontend/afe/models.py
|
| 7d658cf6bade565c1098fd7b47075e96e7b542ca |
|
05-Sep-2013 |
Alex Miller <milleral@chromium.org> |
[autotest] Thread priority from create_suite_job through to create_job.
This schedules a suite job itself at the same priority level of its
tests, and also injects the priority value into the suite control files.
The change to thread priority through from dynamic_suite into create_job
needs to happen concurrently, because the type of `priority` is changing
from a string to an int. This also means everything that passes in a
priority gets fixed up in this CL.
suite_scheduler has also been tweaked to schedule all tests at the
PostBuild priority (for now...).
This also does all of the work to get priority to show up by name
instead of by integral value. In the create_job pane, we disallow
anyone to schedule a job at a priority that would preempt major build
processes (anything coming from a waterfall). However, all old jobs are
shown at the integral value, because showing two seperate priority
scales at the same time would be incredibly confusing. (Especially
since URGENT would be incredibly low priority)
BUG=chromium:250583
DEPLOY=afe, apache, suite_scheduler
TEST=unit, run_suite with a priority level
Change-Id: I9ecf5ceed5c58bd8ee0815c6d15f4aba300082fe
Reviewed-on: https://chromium-review.googlesource.com/168143
Reviewed-by: Alex Miller <milleral@chromium.org>
Tested-by: Alex Miller <milleral@chromium.org>
Commit-Queue: Alex Miller <milleral@chromium.org>
/external/autotest/frontend/afe/models.py
|
| 8bb1f7d8766d202d220879642a7d5c40e2775e52 |
|
05-Aug-2013 |
beeps <beeps@chromium.org> |
[autotest] Teach the scheduler to abort SpecialTasks.
1. Database migration that adds a column to afe_special_tasks.
2. An rpc that modifies the db with the abort bit.
3. A scheduler method that polls the database for aborted jobs
and handles the agent.
TEST=Scheduled a job and aborted it's special tasks. Checked that
subsequent special tasks were scheduled.
Aborted Repair->host goes into repair fail.
Aborted (Reset, Verify, Cleanup)->Repair queued.
Checked that the Hqe is requeued through PreJobTask epilog.
Aborted SpecialTasks without hqes.
Aborted jobs normally through the frontend.
BUG=chromium:234223
DEPLOY=migrate, apache, scheduler
Change-Id: I1a47bc2d801486a8abdffb44091c59a8f5bdbefc
Reviewed-on: https://gerrit.chromium.org/gerrit/64753
Commit-Queue: Prashanth Balasubramanian <beeps@chromium.org>
Reviewed-by: Prashanth Balasubramanian <beeps@chromium.org>
Tested-by: Prashanth Balasubramanian <beeps@chromium.org>
/external/autotest/frontend/afe/models.py
|
| dfff2fdc8477be3ba89fd915fde2afe8d3716624 |
|
28-May-2013 |
Alex Miller <milleral@chromium.org> |
[autotest] Add a provision special task.
We now insert a special task which calls |autoserv --provision| with the
host that the HQE is about to run on to provision the machine correctly
before the test runs. If the provisioning fails, the HQE will also be
marked as failed. No provisioning special task will be queued if no
provisioning needs to be done to the host before the job can/will run.
With *just* this CL, no provisioning tasks should actually get
scheduled, because the part of the scheduler that maps HQEs to hosts
hasn't been taught about provisioning yet. That will come in a later
CL.
Once this CL goes in, it should not be reverted. The scheduler will
become very unhappy if it sees special tasks in its database, but can't
find a corresponding AgentTask definition for them. One would need to
do manual database cleanup to revert this CL. However, since one can
disable provisioning by reverting the (future) scheduling change CL,
this shouldn't be an issue.
BUG=chromium:249437
DEPLOY=scheduler
TEST=lots:
* Ran a job on a host with a non-matching cros-version:* label, and
a provision special task was correctly created. It ran after Reset,
and correctly kicked off the HQE after it finished.
* Ran a job on a host with a matching cros-version:* label, and no
provision special task was created.
* Ran a job on a host with a non-matching cros-version:* label, and
modified Reset so that it would fail. When reset failed, it canceled
the provision task, and the HQE was still rescheduled.
* Ran a job on a host with a non-matching cros-version:* label, and
modified the cros-version provisioning test to throw an exception.
The provision special task aborted the HQE with the desired semantics
(see comments in the ProvisionTask class in monitor_db), and scheduled
a repair to run after its failure.
The provision failures were all deduped against each other when bug
filing was enabled. See
https://code.google.com/p/autotest-bug-filing-test/issues/detail?id=1678
* Successfully debugged an autoupdate/devserver issue from provision
logs, thus proving that sufficient information is collected for debug.
Change-Id: I96dbfc7b001b90e7dc09e1196c0901adf35ba4d8
Reviewed-on: https://gerrit.chromium.org/gerrit/58385
Reviewed-by: Prashanth Balasubramanian <beeps@chromium.org>
Tested-by: Alex Miller <milleral@chromium.org>
Commit-Queue: Prashanth Balasubramanian <beeps@chromium.org>
/external/autotest/frontend/afe/models.py
|
| 07e09aff0baf871b33e5479e337e5e3e0523b729 |
|
12-Apr-2013 |
Dan Shi <dshi@chromium.org> |
[Autotest] merge cleanup and verify
The objective of this CL is to merge cleanup and verify into a single job to
reduce run time of each test. In existing design, by default, a cleanup job is
scheduled after a test is finished, and a verify job is scheduled before a
test is started. By merging these two jobs together, we are seeing the total
run time of these two jobs is reduced from about 47s to 37s, around 10s saving.
That does not include the saving on scheduler to schedule two jobs, which may
take another 5-10s.
The design is to create a new special task, reset, which runs at the beginning
of a job by default. Verify task is changed to not to run by default before a
job starts. Cleanup job will only be run if a job is scheduled to reboot and
any test failed in that job.
BUG=chromium:220679
TEST=tested with run_suite in local machine
DEPLOY=afe,apache,scheduler,change all users' preference on reboot_after to
Never, sql: |update chromeos_autotest_db.afe_users set reboot_after=0|
Change-Id: Ia38baf6b73897b7e09fdf635eadedc752b5eba2f
Reviewed-on: https://gerrit.chromium.org/gerrit/48685
Commit-Queue: Dan Shi <dshi@chromium.org>
Reviewed-by: Dan Shi <dshi@chromium.org>
Tested-by: Dan Shi <dshi@chromium.org>
/external/autotest/frontend/afe/models.py
|
| 97582a2097bfb27a41514e9eaa071de50d1f7be6 |
|
27-Jun-2013 |
Simran Basi <sbasi@chromium.org> |
Fix bad reference in models.py
Left behind a bug in models.py referencing an non-existant module.
Fixing to point to the correct functions.
BUG=chromium:255130
TEST=unittests
Change-Id: I4034669b74a72cbde9f4437c0ca4f7fed9bfefd7
Reviewed-on: https://gerrit.chromium.org/gerrit/60253
Reviewed-by: Aviv Keshet <akeshet@chromium.org>
Tested-by: Simran Basi <sbasi@chromium.org>
Commit-Queue: Simran Basi <sbasi@chromium.org>
/external/autotest/frontend/afe/models.py
|
| c1b26769845fe59168f2d9b7d6b1d37d93ea4bf1 |
|
26-Jun-2013 |
Simran Basi <sbasi@chromium.org> |
Fix duplicate entry Integrity Error when aborting from AFE
Currently aborting a suite job and its sub jobs from the afe causes
the AFE to spit out duplicate entry errors.
This is do the suite job aborting before a child job does and aborts
the child job. However the model object for that child job has not
been updated so we try to abort it as well and create a new
aborted host queue entry for it but it already exists.
Refactored the logic from abort to a class method that takes in a list
of HQE's to abort. This can be shared between abort and the rpc_interface
code.
BUG=chromium:242040
TEST=Started a suite, did a bulk abort and the error no longer occured.
Change-Id: I0d82506234bac5fea9307b5cd20c836cdf7ccb9c
Reviewed-on: https://gerrit.chromium.org/gerrit/60138
Commit-Queue: Simran Basi <sbasi@chromium.org>
Reviewed-by: Simran Basi <sbasi@chromium.org>
Tested-by: Simran Basi <sbasi@chromium.org>
/external/autotest/frontend/afe/models.py
|
| fa1990038c71dd4abed91a145d7a186dd076a70f |
|
09-May-2013 |
Aviv Keshet <akeshet@chromium.org> |
[autotest] Clarify django ImproperlyConfigured exception.
If a developer imports a django models or django models dependent module
before importing one of the django setup modules, they get a slightly
cryptic ImproperlyConfigured exception from Django which doesn't point
them in the direction of our existing django setup modules. This CL
catches the exception and rethrows it as a more descriptive one.
BUG=None
TEST=From interactive python session, import common, then import
direct_afe without first importing one of the the db setup modules. See
clearer exception message.
Change-Id: I397bbd1d3762b9cba0f4f6203c6b73a5d8de9212
Reviewed-on: https://gerrit.chromium.org/gerrit/50901
Commit-Queue: Aviv Keshet <akeshet@chromium.org>
Reviewed-by: Aviv Keshet <akeshet@chromium.org>
Tested-by: Aviv Keshet <akeshet@chromium.org>
/external/autotest/frontend/afe/models.py
|
| 3dd8beb386f7298ffe84d7410d00cce26973e170 |
|
14-May-2013 |
Aviv Keshet <akeshet@chromium.org> |
[autotest] make a consistent CONTROL_TYPE enum across the codebase
Prior to this CL, there were a multitude of duplicate defitions of the
two control files types (Client or Server), incluiding a difference
between the afe Job model (1 = Server) and the afe Test model (1 =
Client). This CL introduces enums CONTROL_TYPE and CONTROL_TYPE_NAMES to
control_data, to act as the central and consistent defition across the
codebase. In order to avoid needing to mangle the running Jobs table, we
have adopted the existing Job model convention (1 = Server); the Test
table will be re-written with the new consistent convention during test
import.
BUG=chromium:240643
TEST=All existing unit tests pass;
In local autotest without this patch applied, started a suite. Halfway
through suite, applied this patch, ran test importer, restarted apache,
and restarted scheduler. Suite finished successfully.
Verified manually that Client/Server type control files show up
correctly in afe Create Job view.
DEPLOY=scheduler
DEPLOY=apache
DEPLOY=test_importer
Change-Id: Ia5b2573e1d08d96b3826f2837903ef407dcae303
Reviewed-on: https://gerrit.chromium.org/gerrit/51191
Commit-Queue: Aviv Keshet <akeshet@chromium.org>
Reviewed-by: Aviv Keshet <akeshet@chromium.org>
Tested-by: Aviv Keshet <akeshet@chromium.org>
/external/autotest/frontend/afe/models.py
|
| dea67041f8d99fa3a385d7be919e4477cf642a34 |
|
23-Apr-2013 |
Alex Miller <milleral@chromium.org> |
[autotest] Reland Aborting a job aborts all child jobs.
Django has been upgraded to 1.5, so this is now safe to land.
When a HQE is aborted, we find all HQEs associated with a parent_id of
job that is being aborted, and abort those also.
This was implemented as "requesting a job to abort also requests all
child jobs to abort". All calls to abort a host queue entry end up in
the afe model's abort(). If one directly modifies the database, one can
still abort a singular job without also aborting its children, in case
anyone ever has a need to do so.
BUG=chromium:226826
DEPLOY=apache, scheduler
TEST=Kicked off two suites. Aborted first one immediately, and the
other suite was unaffected. Aborted the second suite after the first
job finished, and the completed job was unaffected. All other tests
(and suite job) were aborted. Confirmed in database that the host queue
entries rows were set correctly, and the appropriate rows in the
|afe_aborted_host_queue_entries| were still being inserted.
Modified a suite after it was made so that it was its own parent.
Confirmed that aborting the suite ran correctly.
Change-Id: I48303dfe5264d24183c66fa7fd2158a6b838cd51
Reviewed-on: https://gerrit.chromium.org/gerrit/48864
Tested-by: Alex Miller <milleral@chromium.org>
Reviewed-by: Simran Basi <sbasi@chromium.org>
Commit-Queue: Alex Miller <milleral@chromium.org>
/external/autotest/frontend/afe/models.py
|
| 2304143dd6f6324c3057e8895341cc4fe56142f4 |
|
17-Apr-2013 |
Scott Zawalski <scottz@chromium.org> |
Revert "[autotest] Aborting a job aborts all child jobs."
https://code.google.com/p/chromium/issues/detail?id=232388
bulk_create unknown attribute.
This reverts commit f87c46e524d64bbe8b28058856faa2fb4acc7367
Change-Id: I1a4fcaefaeb2b9e27eea31494b29def1fe8f13d1
Reviewed-on: https://gerrit.chromium.org/gerrit/48363
Reviewed-by: Scott Zawalski <scottz@chromium.org>
Tested-by: Scott Zawalski <scottz@chromium.org>
/external/autotest/frontend/afe/models.py
|
| f87c46e524d64bbe8b28058856faa2fb4acc7367 |
|
09-Apr-2013 |
Alex Miller <milleral@chromium.org> |
[autotest] Aborting a job aborts all child jobs.
When a HQE is aborted, we find all HQEs associated with a parent_id of
job that is being aborted, and abort those also.
This was implemented as "requesting a job to abort also requests all
child jobs to abort". All calls to abort a host queue entry end up in
the afe model's abort(). If one directly modifies the database, one can
still abort a singular job without also aborting its children, in case
anyone ever has a need to do so.
BUG=chromium:226826
DEPLOY=scheduler
TEST=Kicked off two suites. Aborted first one immediately, and the
other suite was unaffected. Aborted the second suite after the first
job finished, and the completed job was unaffected. All other tests
(and suite job) were aborted. Confirmed in database that the host queue
entries rows were set correctly, and the appropriate rows in the
|afe_aborted_host_queue_entries| were still being inserted.
Modified a suite after it was made so that it was its own parent.
Confirmed that aborting the suite ran correctly.
Change-Id: I7cea75b3917974210b2d975c9b36870129966dcc
Reviewed-on: https://gerrit.chromium.org/gerrit/47695
Commit-Queue: Alex Miller <milleral@chromium.org>
Reviewed-by: Alex Miller <milleral@chromium.org>
Tested-by: Alex Miller <milleral@chromium.org>
/external/autotest/frontend/afe/models.py
|
| 9af96d3b0c1d3fc7e137a3535835391b9019b810 |
|
05-Mar-2013 |
Aviv Keshet <akeshet@chromium.org> |
[autotest] Add test_retry field to afe_autotests table
BUG=chromium-os:37158
TEST=database/migrate.py sync. Verified that new column is created, and
defaults to value 0 for all existing entries.
Change-Id: Iac5af51aa66edc00c5a00df035626e6470ae7d0c
Reviewed-on: https://gerrit.chromium.org/gerrit/44701
Reviewed-by: Aviv Keshet <akeshet@chromium.org>
Tested-by: Aviv Keshet <akeshet@chromium.org>
Commit-Queue: Scott Zawalski <scottz@chromium.org>
Reviewed-by: Scott Zawalski <scottz@chromium.org>
Tested-by: Scott Zawalski <scottz@chromium.org>
/external/autotest/frontend/afe/models.py
|
| cd1ff9b72bd73c7308145750f07908b26c5d08da |
|
01-Mar-2013 |
Aviv Keshet <akeshet@chromium.org> |
[autotest] Add test_retry field to afe_jobs table
Does what it says on the tin.
CQ-DEPEND=If8134fe263bb33ee5d52bc92e78faee05388b239
BUG=chromium-os:37158
TEST=database/migrate.py sync, ensure that jobs can still be viewed in
afe. Ensure that new column exists and is zeroed out for all jobs
already in db. Kick off a smoke suite and ensure that it runs.
Change-Id: I0790d60d7c86d9278507b46ae1579a7b72b20c5a
Reviewed-on: https://gerrit.chromium.org/gerrit/44427
Tested-by: Aviv Keshet <akeshet@chromium.org>
Reviewed-by: Scott Zawalski <scottz@chromium.org>
Commit-Queue: Scott Zawalski <scottz@chromium.org>
Tested-by: Scott Zawalski <scottz@chromium.org>
/external/autotest/frontend/afe/models.py
|
| 18308922b8086ecc9aed084f02ddf831ae252bf7 |
|
20-Feb-2013 |
Aviv Keshet <akeshet@chromium.org> |
[autotest] Create dynamic_suite child jobs with parent_job_id
Child jobs of a dynamic_suite call are now created with appropriate
parent_job_id pointing back at the suite job.
BUG=chromium-os:38253
TEST=Kick of a suite with run_suite.py. Child jobs have correct parent
job id, as seen in job detail view.
Change-Id: Ie45aac1b741b1282ddf792812480bc1f4d35278a
Reviewed-on: https://gerrit.chromium.org/gerrit/43583
Reviewed-by: Alex Miller <milleral@chromium.org>
Tested-by: Aviv Keshet <akeshet@chromium.org>
Reviewed-by: Aviv Keshet <akeshet@chromium.org>
Commit-Queue: Aviv Keshet <akeshet@chromium.org>
/external/autotest/frontend/afe/models.py
|
| 0b9cfc93f17ec81535de54e3bae4f5f0e8d8595b |
|
05-Feb-2013 |
Aviv Keshet <akeshet@chromium.org> |
[autotest] Add parent_job field to database and models
Adds a parent_job field to dbmodels, and a parent_job_id column to the
afe jobs table, allowing jobs to have some notion of referring to parent
jobs that may have kicked them off (for instance, in a dynamic suite --
to be implemented in future commit).
BUG=chromium-os:38253
TEST=Ran database/migrate.py sync . Ensuree that previous jobs can still
be viewed in the afe, and than a new parent_job_id column has been
created in afe jobs table (via mysql client). Ran a smoke suite
successfully.
CQ-DEPEND=CL:Ie45aac1b741b1282ddf792812480bc1f4d35278a
Change-Id: Iadfedc2a9e8f603ab223c87ed7c1510a46622216
Reviewed-on: https://gerrit.chromium.org/gerrit/42673
Commit-Queue: Aviv Keshet <akeshet@chromium.org>
Reviewed-by: Aviv Keshet <akeshet@chromium.org>
Tested-by: Aviv Keshet <akeshet@chromium.org>
/external/autotest/frontend/afe/models.py
|
| 7db38bab49bb85027c216bdf89e289d0f24dba16 |
|
13-Feb-2013 |
Dennis Jeffrey <dennisjeffrey@chromium.org> |
Fix presubmit warnings in autotest's frontend/afe/models.py.
BUG=chromium-os:38655
TEST=None
Change-Id: I7adfd21e3fc1265fc74bdfa141b515160d31d16e
Reviewed-on: https://gerrit.chromium.org/gerrit/43192
Reviewed-by: Scott Zawalski <scottz@chromium.org>
Commit-Queue: Dennis Jeffrey <dennisjeffrey@chromium.org>
Tested-by: Dennis Jeffrey <dennisjeffrey@chromium.org>
/external/autotest/frontend/afe/models.py
|
| 34217022229b755bc1ee52f83665acba76bd5044 |
|
06-Nov-2012 |
Simran Basi <sbasi@chromium.org> |
Autotest: Increase run timeout granularity
This CL modifies all references to max_runtime_hrs to instead be
max_runtime_mins. This includes the django models, rpc interfaces, the
cleanup timeout code, and the frontend java views.
The frontend java code will need to be recompiled once this commits to
prevent the frontend from breaking.
The cleanup timeout pathway has been adjusted to find all timedout jobs
by minute, and has been changed to run every 5 minutes vs every hour as
before.
BUG=chromium-os:36067
TEST=Ran on my local afe, ensure that jobs can still be created correctly,
and jobs with short timeouts do indeed get aborted when expected.
Change-Id: Idfdeb3f1d4947d6b2e6b48127a31db535704e972
Reviewed-on: https://gerrit.chromium.org/gerrit/37827
Tested-by: Simran Basi <sbasi@chromium.org>
Reviewed-by: Scott Zawalski <scottz@chromium.org>
Reviewed-by: Simran Basi <sbasi@chromium.org>
Commit-Ready: Simran Basi <sbasi@chromium.org>
/external/autotest/frontend/afe/models.py
|
| a8a0d75c2591ab872f695d7b8f22f2ca10493dab |
|
08-Nov-2012 |
Scott Zawalski <scottz@chromium.org> |
Revert "Autotest: Increase run timeout granularity"
This causes autoupdate problems, see bug crosbug.com/36183
This reverts commit daffa57a6738c2a80caee1f311a527313ece1207
Change-Id: Iad65be9bf1b0ea329866bfdc764b0089c4f9b2a5
Reviewed-on: https://gerrit.chromium.org/gerrit/37657
Commit-Ready: Scott Zawalski <scottz@chromium.org>
Reviewed-by: Scott Zawalski <scottz@chromium.org>
Tested-by: Scott Zawalski <scottz@chromium.org>
/external/autotest/frontend/afe/models.py
|
| daffa57a6738c2a80caee1f311a527313ece1207 |
|
06-Nov-2012 |
Simran Basi <sbasi@chromium.org> |
Autotest: Increase run timeout granularity
This CL modifies all references to max_runtime_hrs to instead be
max_runtime_mins. This includes the django models, rpc interfaces, the
cleanup timeout code, and the frontend java views.
The frontend java code will need to be recompiled once this commits to
prevent the frontend from breaking.
The cleanup timeout pathway has been adjusted to find all timedout jobs
by minute, and has been changed to run every 5 minutes vs every hour as
before.
BUG=chromium-os:36067
TEST=Ran on my local afe, ensure that jobs can still be created correctly,
and jobs with short timeouts do indeed get aborted when expected.
Change-Id: Id7668bbd05a9b02c22e7c549fac232fae02fc728
Reviewed-on: https://gerrit.chromium.org/gerrit/37479
Reviewed-by: Scott Zawalski <scottz@chromium.org>
Commit-Ready: Simran Basi <sbasi@chromium.org>
Tested-by: Simran Basi <sbasi@chromium.org>
/external/autotest/frontend/afe/models.py
|
| 0a99391317b5caca3d373d0bba931bcb62302566 |
|
17-May-2011 |
Eric Li <ericli@chromium.org> |
Merge autotest upstream from @5337 ~ @5360
Change-Id: Iad59314186db370f9219c3fb294b851f4e214e2e
Reviewed-on: http://gerrit.chromium.org/gerrit/1045
Reviewed-by: Eric Li <ericli@chromium.org>
Tested-by: Eric Li <ericli@chromium.org>
/external/autotest/frontend/afe/models.py
|
| 6c0b70ba5fa83b57fc0bc2f6cf347b546c1bc9e8 |
|
17-May-2011 |
Eric Li <ericli@chromium.org> |
Revert "Merge autotest upstream from @5337 ~ @5360"
This reverts commit ce1b0620fddfb6efa33cc9441d9c91ce06b30de3.
Change-Id: Iab10c235ce76bfd45e7a2036ea2413729e34ae15
Reviewed-on: http://gerrit.chromium.org/gerrit/1033
Reviewed-by: Eric Li <ericli@chromium.org>
Tested-by: Eric Li <ericli@chromium.org>
/external/autotest/frontend/afe/models.py
|
| ce1b0620fddfb6efa33cc9441d9c91ce06b30de3 |
|
16-May-2011 |
Eric Li <ericli@chromium.org> |
Merge autotest upstream from @5337 ~ @5360
Change-Id: I2052380f1fb1ff952977914cb6f516a31c076aba
Reviewed-on: http://gerrit.chromium.org/gerrit/1031
Tested-by: Eric Li <ericli@chromium.org>
Reviewed-by: Eric Li <ericli@chromium.org>
/external/autotest/frontend/afe/models.py
|
| 5a8c6ad7e67d86090ed6c4ef216b82e45c0f9283 |
|
01-Feb-2011 |
Paul Pendlebury <pauldean@chromium.org> |
Add support for an --image flag to atest.
This enables creating jobs from the CLI that use autoserv to install a new OS image before running the requested test.
This was not added to the database like the reboot before/reboot after flags to maintain backward compatibility and avoid any changes to the database schema.
Also it is not working as a pure parameterized job as the current implementation is not 100% complete and would require more work to finish. And since mixed jobs are not allowed it would also mean moving existing control file jobs to parameterized jobs.
So the implementation of adding a parameterized id to control jobs and using a known test to hold the OS image path is the most straight forward of the options.
Change-Id: I77cdda0c50c222a4c594da2626a71fa55f5957cb
BUG=chromium-os:11486
TEST=Manual testing using atest cli to create jobs with --image parameters and verifying the value is passed to autoserv.
Review URL: http://codereview.chromium.org/6181003
/external/autotest/frontend/afe/models.py
|
| 4a41e0145c2ede3605d19cdfacb779ca6aca1ccc |
|
17-Jul-2010 |
jamesren <jamesren@592f7852-d20e-0410-864c-8624ca9c26a4> |
Begin adding support for parameterized jobs.
This will allow test developers to specify certain parameters that a test
control file may take, so that users can then easily set those parameters
on job create. Enabling this feature removes the ability to edit the
control file directly on job creation.
Feature is currently INCOMPLETE. Do not attempt to use. This feature will
be committed in small pieces for the sake of having smaller code reviews.
Signed-off-by: James Ren <jamesren@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@4720 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| 76fcf19ec42d5c7580d2e7891e4610e5fe725286 |
|
21-Apr-2010 |
jamesren <jamesren@592f7852-d20e-0410-864c-8624ca9c26a4> |
Add ability to associate drone sets with jobs. This restricts a job to
running on a specified set of drones.
Signed-off-by: James Ren <jamesren@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@4439 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| cd7a81a58c7a1a16105dcd9dca5e9f1680eff65b |
|
21-Apr-2010 |
jamesren <jamesren@592f7852-d20e-0410-864c-8624ca9c26a4> |
Various changes to support further high-level automation efforts.
* added a RESTful interface for TKO. right now there's only a single, simple resource for accessing test attributes.
* extended the REST server library in a few ways, most notably to support
* querying on keyvals, with something like ?has_keyval=mykey=myvalue&...
* operators, delimited by a colon, like ?hostname:in=host1,host2,host3
* loading relationships over many items efficiently (see InstanceEntry.prepare_for_full_representation()). this is used to fill in keyvals when requesting a job listing, but it can (and should) be used in other places, such as listing labels for a host collection.
* loading a collection with inlined full representations, by passing full_representations=true
* added various features to the AFE RESTful interface as necessary.
* various fixes to the rest_client library, most notably
* changed HTTP client in rest_client.py to use DI rather than singleton, easing testability. the same should be done for _get_request_headers(), to be honest.
* better support for query params, including accepting a MultiValueDict and supporting URIs that already have query args
* basic support for redirects
* builtin support for requesting a full collection (get_full()), when clients explicitly expect the result not to be paged. i'm still considering alternative approaches to this -- it may make sense to have something like this be the default, and have clients set a default page size limit rather than passing it every time.
* minor change to mock.py to provide better debugging output.
Signed-off-by: Steve Howard <showard@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@4438 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| d8b6e17c8edb2845949e39c75337e4389a630c63 |
|
17-Apr-2010 |
jamesren <jamesren@592f7852-d20e-0410-864c-8624ca9c26a4> |
Handle the case where keyvals may not be specified on job create. This
also fixes a bug where recurring jobs are created incorrectly.
Signed-off-by: James Ren <jamesren@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@4423 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| 121eee6338595777668e669c61521ceeceeeddf7 |
|
13-Apr-2010 |
jamesren <jamesren@592f7852-d20e-0410-864c-8624ca9c26a4> |
Removing references to the 'Dead' host status
Signed-off-by: James Ren <jamesren@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@4409 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| dd855244f44b65d0508345c6fef74846652c8c26 |
|
02-Mar-2010 |
jamesren <jamesren@592f7852-d20e-0410-864c-8624ca9c26a4> |
Abstract out common models used in the frontend's models.py so that django is not required to interact with non Django portions of the code.
This includes the enums RebootBefore, RebootAfter and Test.Type
git-svn-id: http://test.kernel.org/svn/autotest/trunk@4280 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| e36562323bec14d14b7c583db94788a725514ff2 |
|
02-Mar-2010 |
jamesren <jamesren@592f7852-d20e-0410-864c-8624ca9c26a4> |
Fix Django's QuerySet.delete() behavior for models that use "invalid"
Signed-off-by: James Ren <jamesren@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@4276 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| cae88c6d6f2bd3c8b3ee46628e7230b3fcc444bc |
|
19-Feb-2010 |
jamesren <jamesren@592f7852-d20e-0410-864c-8624ca9c26a4> |
Have the description column in AFE admin interface for tests actually
say "Description"
Signed-off-by: James Ren <jamesren@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@4251 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| 35a70227e69e03d0492fe47608a6370dc339f287 |
|
16-Feb-2010 |
jamesren <jamesren@592f7852-d20e-0410-864c-8624ca9c26a4> |
Fix styling of AFE admin interface.
The change_list.html for tests used to be handled as a customized
template. The template failed to extend the original change_list.html,
so it didn't keep up to date. Additionally, the "white-space:pre" style
was applied to all <td> elements instead of just the description.
This restricts that style change to the description only, and also
maintains most of the default Django admin interface.
Signed-off-by: James Ren <jamesren@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@4247 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| 4d23375ff9b95e95e9f1d8443d20122f7270c1f4 |
|
20-Jan-2010 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Move logic for job reboot before/after defaults from RPC method to job model. Also adding a reference to the host protections enum in the Host model as a convenience for some other code I'm working on.
Signed-off-by: Steve Howard <showard@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@4157 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| c1a98d1e146080bd3e4f034cb13d740dfb1535f4 |
|
15-Jan-2010 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Support for job keyvals
* can be passed as an argument to create_job, stored in AFE DB
* scheduler reads them from the AFE DB and writes them to the job-level keyval file before the job starts
* parser reads them from the keyval file and writes them to the TKO DB in a new table
Since the field name "key" happens to be a MySQL keyword, I went ahead and made db.py support proper quoting of field names. Evetually it'd be really nice to deprecate db.py and use Django models exclusively, but that is a far-off dream.
Still lacking support in the AFE and TKO web clients and CLIs, at least the TKO part will be coming soon
Signed-off-by: Steve Howard <showard@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@4123 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| c510344e16543a6f810be1608e1a879c3d3bbcde |
|
15-Jan-2010 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Slight change to SpecialTask scheduling code. This is to support my work on a new AFE interface.
Signed-off-by: Steve Howard <showard@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@4118 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| cfcdd80cb19eb28f593a79f9ba2e56496c3f9f3a |
|
15-Jan-2010 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Fix creation of the autotest_system user
Signed-off-by: James Ren <jamesren@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@4116 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| 64a9595406f2884fb3ece241190b10aa054439a9 |
|
13-Jan-2010 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
When using Django models from a script, make the current user default to an actual database user named "autotest_system". This allows for simpler, more consistent code.
Signed-off-by: Steve Howard <showard@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@4114 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| eab66ce582bfe05076ff096c3a044d8f0497bbca |
|
23-Dec-2009 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Rename the tables in the databases, by prefixing the app name. This is
in preparation for merging the two databases and the two Django
projects into one.
Note that this renames *all* standard Autotest DB tables in both the
autotest_web and tko databases. If you have scripts written directly
against these databases, *they will break*. If your scripts access
the RPC interfaces, they should continue to work.
Another patch will be along within the next few weeks to actually move
the TKO tables into the autotest_web database.
From: James Ren <jamesren@google.com>
Signed-off-by: Steve Howard <showard@google.com>
Rename the tables in the databases, by prefixing the app name. This is
in preparation for merging the two databases and the two Django projects
into one.
Note that this renames *all* standard Autotest DB tables in both the autotest_web and tko databases. If you have scripts written directly against these databases, *they will break*. If your scripts access the RPC interfaces, they should continue to work.
From: James Ren <jamesren@google.com>
Signed-off-by: Steve Howard <showard@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@4040 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| a9545c0ab3d8f3e36efadaefdcf37393708666d9 |
|
18-Dec-2009 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
backend support for hostless jobs
* support in rpc_interface.create_job() and models for creating a hostless job -- a job with one queue entry with no host, meta_host or atomic_group
* support in scheduler for recognizing and executing such a job. the bulk of the work was in extracting an AbstractQueueTask class from QueueTask, containing all the logic not pertaining to hosts. I then added a simple HostlessQueueTask class also inheriting from it.
Also got rid of HostQueueEntry.get_host() and added an extra log line when AgentTasks finish (used to be for QueueTasks only).
Signed-off-by: Steve Howard <showard@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@4018 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| d11956572cb7a5c8e9c588c9a6b4a0892de00384 |
|
08-Dec-2009 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Make drone_manager track running processes counts using only the information passed in from the scheduler. Currently it also uses process counts derived from "ps", but that is an unreliable source of information. This improves accuracy and consistency and gives us full control over the process.
This involves a few primary changes:
* made the drone_manager track process counts with each PidfileId
* added method declare_process_count() for the scheduler to indicate the process count of a pidfile ID during recovery (in other cases, the DroneManager gets that info in execute_process())
Doing this involved some extensive refactorings. Because the scheduler now needs to declare process counts during recovery, and because the AgentTasks are the entities that know about process counts, it made sense to move the bulk of the recovery process to the AgentTasks. Changes for this include:
* converted a bunch of AgentTask instance variables to abstract methods, and added overriding implementations in subclasses as necessary
* added methods register_necessary_pidfiles() and recover() to AgentTasks, allowing them to perform recovery for themselves. got rid of the recover_run_monitor() argument to AgentTasks as a result.
* changed recovery code to delegate most of the work to the AgentTasks. The flow now looks like this: create all AgentTasks, call them to register pidfiles, call DroneManager to refresh pidfile contents, call AgentTasks to recover themselves, perform extra cleanup and error checking. This simplified the Dispatcher somewhat, in my opinion, though there's room for more simplification.
Other changes include:
* removed DroneManager.get_process_for(), which was unused, as well as related code (include the DroneManager._processes structure)
* moved logic from HostQueueEntry.handle_host_failure to SpecialAgentTask._fail_queue_entry. That was the only call site.
And some other bug fixes:
* eliminated some extra state from QueueTask
* fixed models.HostQueueEntry.execution_path(). It was returning the wrong value, but it was never used.
* eliminated some big chunks from monitor_db_unittest. These broke from the refactorings described above and I deemed it not worthwhile to fix them up for the new code. I checked and the total coverage was unaffected by deleting these chunks.
Signed-off-by: Steve Howard <showard@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@4007 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| 9bb960b90d5102cce1c8a15314900035c6c4e69a |
|
19-Nov-2009 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Support restricting access to drones by user. Administrators can put lines like
<hostname>_users: showard,scottz
in the global config, where <hostname> is a drone hostname. That drone will then be limited to use by those users (that is, by jobs launched by those users, and tasks launched due to those jobs). This required numerous changes:
* added a requested_by field to SpecialTask (with corresponding migration). For tasks with queue_entries, we can infer this from the job, but for those without, we need this information explicitly declared. Note this can be null if the task was created by the system, not in response to any user action. The only place this occurs now is in scheduler recovery (Dispatcher._recover_hosts_where()), but there may be an upcoming feature to periodically reverify hosts, which would be another (much more common) case.
* modified all SpecialTask creation sites to pass requested_by if necessary.
* modified AgentTask to keep a username attribute, and modified its run() method to pass that to PidfileRunMonitor.run(), which passes it along to DroneManager.execute_command().
* modified Agent to always keep self.task around, there's no reason to throw it away and now that we're looking at it from other classes, it's problematic if it disappears.
* modified Dispatcher throttling code to pass the username when requesting max runnable processes.
* added an allowed_users property to _AbstractDrone, and made DroneManager load it from the global config.
* made DroneManager's max_runnable_processes() and _choose_drone_for_execution() methods accept the username and obey user restrictions.
* added extensive tests for everything. the modiications required to monitor_db_unittest were annoying but not too bad. but parts of that file may need to be removed as they'll be obsoleted by monitor_db_functional_test and they'll become increasingly annoying to maintain.
couple other related changes:
* got rid of CleanupHostsMixin. it was only acutally needed by GatherLogsTasks (since we made the change to have GatherLogsTask always run), so I inlined it there and simplified code accordingly.
* changed a bunch of places in the scheduler that were constructing new instances of Django models for existing rows. they would do something like "models.Host(id=<id of existing host>)". that's correct for scheduler DBModels, but not for Django models. For Django models, you only instantiate new instances when you want to create a new row. for fetching existing rows you always use a manager -- Model.objects.get() or Model.objects.filter() etc. this was an existing bug but wasn't exposed until I made some of the changes involved in this feature.
Signed-off-by: Steve Howard <showard@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@3961 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| e60e44ece1445d97977a77cb79f0896989b869d7 |
|
13-Nov-2009 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Special tasks show "Failed" as their status instead of "Completed" if
they failed
Signed-off-by: James Ren <jamesren@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@3946 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| d3771cc3c770639a816a6a3065aa5a1ef53a24a1 |
|
07-Oct-2009 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
tiny change necessary to keep frontend/afe/rpc_interface_unittest happy
Signed-off-by: Steve Howard <showard@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@3803 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| afd97de523a8c8c4e6657fa2db6214fda68d8086 |
|
01-Oct-2009 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
When a one-time host is added as a normal host while it's running a job, the status would get reset, messing up the scheduler. Fix that.
Signed-off-by: Steve Howard <showard@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@3778 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| eaa408e59e932ce0540aa5d50bf966af5080f640 |
|
11-Sep-2009 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Add atomic group job support to why_isnt_my_job_running to be able to
tackle the frequently asked questions regarding forever queued jobs.
Signed-off-by: Gregory Smith <gps@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@3707 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| 8cbaf1e2984c7c5f7df877ac6a759ccef13aa63a |
|
08-Sep-2009 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Disallow modification of the Everyone ACL from the CLI. Also removed
Everyone from the Django admin interface.
Signed-off-by: James Ren <jamesren@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@3666 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| 8cc058f50a46976e0a446aa3054f7f2349d6291a |
|
08-Sep-2009 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Make scheduler more stateless. Agents are now scheduled only by the
dispatcher, and agents no longer use in-memory state to remember
multiple tasks. All state is managed by the database.
Risk: high (large scheduler change)
Visibility: medium (scheduler restarts are now more stable)
Signed-off-by: James Ren <jamesren@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@3664 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| 474d136e9a71f288f294167112dda909f9273434 |
|
21-Aug-2009 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Sanity check: prevent the creation of a special task for a host that already
has an active special task. This should only ever happen due to a scheduler
bug. Dying to allow human cleanup is desirable today.
Signed-off-by: Gregory Smith <gps@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@3571 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| f2f84cd56026a86fe0c4ec7acae558c1ece26016 |
|
03-Aug-2009 |
mbligh <mbligh@592f7852-d20e-0410-864c-8624ca9c26a4> |
Do not blindly set Host entries to 'Ready' in the database when adding a one
time host. If the entry already exists and has a verify/repair/cleanup task
running on it, this will circumvent that and allow things to run before that
task is finished. Newly created one time hosts are status='Ready' by
default. Existing ones will reach 'Ready' or 'Repair-Failed' once
their current task finishes.
This means that a one time host that enters 'Repair-Failed' -after- a
new job was scheduled on it will stuck until the user Clones and
Resubmits the job. Annoying and a bug of its own, but much better than
triggering the scheduler's "one task per host" assertion or allowing
multiple things to run at once on a host.
Signed-off-by: Gregory Smith <gps@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@3488 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| 7890e797d3549b3de18b79448817ffc80a79ee42 |
|
28-Jul-2009 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Fix models.host_or_metahost_name to not die when called on a HostQueueEntry
with no host or meta_host because it is a queued atomic_group job.
Signed-off-by: Gregory Smith <gps@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@3461 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| a5288b4bb2b09aafe914d0b7d5aab79a7e433eaf |
|
28-Jul-2009 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Upgrade from Django 0.96 to Django 1.0.2.
Risk: high (framework change)
Visibility: medium
Signed-off-by: James Ren <jamesren@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@3457 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| 97446887819594f1e2a329dcff289ee8e934b626 |
|
21-Jul-2009 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Log changes to SpecialTask active/complete state.
Signed-off-by: Steve Howard <showard@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@3426 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| c0ac3a79cceec87873257a8b6d41d19c9fb02ec6 |
|
08-Jul-2009 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
add the option to include verify/repair/cleanups in the job table on the host detail page.
* added RPC get_host_queue_entries_and_special_tasks, which returns both HQEs and SpecialTasks for a host in a carefully interleaved manner.
* added appropriate frontend unit tests
* added support to HostDetailView to include these entries, but have them not be selectable. this required changes to SelectionManager. I also added a utility method to Utils for opening a new window, and changed all sites that do that to call the new method.
Signed-off-by: Steve Howard <showard@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@3385 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| ed2afea4ca6e23a82d20d1f2ee1067d0c25a8cc2 |
|
07-Jul-2009 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
make SpecialTasks recoverable. this involves quite a few changes.
* run tasks in determined dirs instead of temp dirs. the dir paths look like hosts//-, for example, hosts/myhost/4-verify. the ID comes from the SpecialTask DB row. this allows us to find the pidfile when we go looking for it during recovery, and it makes it simple to find the logs for any given special task, much like for HostQueueEntries. added SpecialTask.execution_path() for this purpose, and added models_test to test it.
* added execution_path() to HostQueueEntry to match the interface of SpecialTask, allowing for more polymorphism, and changed most call sites to use it.
* since we're running in these dirs, copy the full results back in these dirs, instead of just copying a single log file.
* move process recovery code up into AgentTask, so that all AgentTasks can share the same generic process recovery code.
* change SpecialTask recovery code to do process recovery.
* change VerifyTask handling of multiple pending verify requests for a machine. instead of updating all the requests, just delete all other tasks. they're not specially tracked in any way so it's simplest to just delete them.
* made special tasks get marked is_active=False when they complete, to be consistent with HQEs
other changes:
* added null=True to SpecialTask.time_started definition
* made EmailManager.enqueue_notify_email always log the message, and removed explicit logging calls from call sites
* added feature to DroneManager.execute_command() to automatically substitute the working directory into the command. this avoids some duplicate information being passed around and simplifies the unit test.
Signed-off-by: Steve Howard <showard@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@3380 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| 2fe3f1df42f5fd1dc6296219df289851dcf77025 |
|
06-Jul-2009 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Enter all Verify/Cleanup/Repair tasks into the special_tasks table. Also
keep track of which Host Queue Entry (if any) each Verify/Cleanup/Repair
task belongs to. Additionally, implement recovery for jobs in
Verify/Cleanup/Repair (i.e., do not simply reverify the host and requeue
the job).
Risk: medium (scheduler changes)
Visibility: medium (functionality change)
Signed-off-by: James Ren <jamesren@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@3372 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| a984ad1d988fe5035df315f58cbe89e0bab77071 |
|
02-Jul-2009 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Remove duplicate "sync count" line from Django admin interface's edit
page for Tests.
Risk: low
Visibility: low
Signed-off-by: James Ren <jamesren@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@3352 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| e9450c934d860f154175abfee19389bf1a114305 |
|
30-Jun-2009 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Change the default for the max_number_of_machines in atomicgroups to be infinite*. (* see definition of INFINITE_MACHINES).
The common use for rack tests is to want all machines in the group to be used.
Signed-off-by: Gregory Smith <gps@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@3328 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| 98ead171ab61b324414dd0316478c5ce72bbd4b1 |
|
22-Jun-2009 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
exempt one-time-hosts from ACL checking in the frontend as well, and update the unit test to accurately test this
Signed-off-by: Steve Howard <showard@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@3303 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| 2924b0ac9e0ca35e2cd45a23b60ecfc204360c44 |
|
19-Jun-2009 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Ensure one-time-hosts aren't in the Everyone ACL, and make the scheduler ignore this.
Signed-off-by: Steve Howard <showard@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@3299 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| f1175bb2edbe3215f4bfa117c6b93d2e090d1d07 |
|
17-Jun-2009 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
get rid of some now-obsolete code
* client.common_lib.debug
* abstract_ssh.LoggerFile
Changed {SSHHost,ParamikoHost}.run() default args for std{out,err}_tee to a constant instead of None. This preserves the existing behavior (teeing to the logging module by default), but allows callers to explicitly avoid this teeing if desired, by passing stdout_tee=None.
Signed-off-by: Steve Howard <showard@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@3293 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| 6d7b2ff05b2232b1b225a4cb3521d76c0152cad9 |
|
10-Jun-2009 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Redesign the reverify hosts feature. Host status is no longer changed
from the frontend; scheduler deals with all status changes.
Risk: medium (scheduler behavior change)
Visibility: low
Signed-off-by: James Ren <jamesren@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@3238 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| 25aaf3f6c159bc00233d9180e4d724e53cbfc53c |
|
09-Jun-2009 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Minor change to get_status_counts() to ensure all jobs have status counts. This only makes a difference when there's a job with no HQEs, which is an invalid case anyway, but we might as well handle it as smoothly as possible.
Signed-off-by: Steve Howard <showard@google.com>
Autotest@test.kernel.org
http://test.kernel.org/cgi-bin/mailman/listinfo/autotest
git-svn-id: http://test.kernel.org/svn/autotest/trunk@3225 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| 01a5167f13d9788c9f359ebba31358e329c98ebc |
|
29-May-2009 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Have the scheduler check for and sometimes cleanup various DB inconsistencies.
* in periodic cleanup, check for relationships to invalidated objects, and remove them (and send notification email)
* in 24hr cleanup, check for hosts with != 1 platform, and send notification email
Also changed AFE models to have labels remove associations with tests (as dependencies) when deleted (invalidated).
Signed-off-by: Steve Howard <showard@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@3185 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| cafd16e33f658f63fcbe09b0f232bbeb354ae3c2 |
|
29-May-2009 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Check for multiple platforms on a machine when modifying labels.
Signed-off-by: Steve Howard <showard@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@3179 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| b6d1662e18d756483d5fd81f4057cae4ef62152c |
|
26-May-2009 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
fix JobManager.get_status_counts, which was returning incorrect counts in some cases when jobs were aborted. the problem was that it's possible for a complete entry to have aborted set or not and have the same full status, which was violating an assumption of the method.
to test it, instead of adding stuff to the doctests (which would be messy in this particular case, since we need to reach in and mess with HQE stauses), i instead started a new rpc_interface_unittest, which seems to be the way of the future. since it shared a bunch of logic with the scheduler unit test (which also depends on setting up a fake AFE database), i extracted common logic into frontend/afe/frontend_test_utils.py. i also fixed up some of the logic extracted from monitor_db_unittest for reusing an initial DB between tests.
Signed-off-by: Steve Howard <showard@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@3177 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| 1ff7b2e88ae7a382f85ab76e786a471134e8a6a0 |
|
16-May-2009 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Add ability to reverify a host from the Host List.
Risk: low
Visilibity: medium (UI change)
Signed-off-by: James Ren <jamesren@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@3143 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| 12f3e3212795a539d95973f893ac570e669e3a22 |
|
13-May-2009 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Add job maximum runtime, a new per-job timeout that counts time since the job actually started.
* added started_on field to host_queue_entries, so that we could actually compute this timeout
* added max_runtime_hrs to jobs, with default in global config, and added option to create_job() RPC
* added the usual controls to AFE and the CLI for the new job option
* added new max runtime timeout method to
* added migration to add new fields and set a safe default max runtime for existing jobs
Signed-off-by: Steve Howard <showard@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@3132 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| a1e74b3e9d68792fae0c926f89b6de1736b1fe21 |
|
12-May-2009 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Add job option for whether or not to parse failed repair results as part of a job, with a default value in global_config. Since the number of options associated with a job is getting out of hand, I packaged them up into a dict in the RPC entry point and passed them around that way from then on.
Signed-off-by: Steve Howard <showard@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@3110 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| f8b19046c4496f570c776b65288382108a633ab4 |
|
12-May-2009 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Add the ability for users to add test attributes. Non-user-created attributes (added by the parser) are still immutable.
* add boolean column user_created to TestAttribute to keep track of which are user created, so we can preserve immutability
* add id primary key field to test attributes. Django requires this and we've been squeezing by without it, but we can't anymore.
* declare some PK fields AutoFields in TKO models, as they should be. this didn't matter before but does now that we have a TKO unit test, since these models determine how the test DB gets created.
* add set_test_attribute RPC to set/delete attributes
* modify get_detailed_test_views() to use the new populate_relationships() method to gather attributes and labels much more efficiently
* add rpc_interface_unittest, a unit test for the TKO rpc interface. TKO was previously completely untested, so this is the first unit test of any kind for it. since the doctests on AFE turned out to be quite unpopular, I'm using the unittest framework this time. this required some changes to AFE testing code.
* various fixes to model_logic to account for assumptions we were previously making that aren't true in TKO (mostly about PK fields being named "id").
Note that the migration may be slow, as it's adding two columns to test_attributes, a potentially large table.
Signed-off-by: Steve Howard <showard@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@3109 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| 0957a848a17b726836aaf9b5532bbfac691d983d |
|
11-May-2009 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Add host attributes to AFE. Nothing actually uses them; they're purely for users and scripts to use. Initial implementation is minimal - new migration, new model, RPCs to set and delete, and inclusion of attributes in get_hosts() results. No CLI or frontend support.
The change to get_hosts() is of primary interest here, since it involves a general problems that occurs many places in Autotest -- gathering relationships over many items. This has been solved in two ways previously:
* inefficiently in most places, by doing a query for each object (i.e. get_hosts()).
* efficiently in one place (JobManager.populate_dependencies()), by doing a single query to get a pivot table and then postprocessing to gather relationships for each object.
This time, I went ahead and implemented a general solution, a generalized version of JobManager.populate_dependencies(). I removed populate_dependencies() and made get_jobs() use the new general version instead, and I made get_hosts() use the new version for all its relationships, including the new attributes relationship. This should speed up get_hosts() considerably. In a future change I'll apply this solution to get_test_views() over in TKO land, since that would also greatly benefit in performance.
Signed-off-by: Steve Howard <showard@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@3102 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| 29f7cd27e51add2648fb62ab2a0c588f9acb1ec4 |
|
29-Apr-2009 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Here is a patch, which extends the autotest system with recurring job
executions. When you create a new recurring job, you can specify:
- start time (on server)
- loop count (0 means infinite): how many times it will executed
- loop period: how many time will wait between two execution
Added features:
- Create job: can create Template job.
- View job: added "Create recurring job"
- New tab "Recurring job"
- list of recurring jobs
- can click on it to view the executed job
- selection support
- Action -> remove
- creation panel (can be accessible thru "Create recurring job")
- submit/create
- reset
- data validity check
From: Zoltan Sogor <weth@inf.u-szeged.hu>
Signed-off-by: Steve Howard <showard@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@3064 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| d3dc199703bfb8784a2f8f072d0514532c86c0a9 |
|
22-Apr-2009 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Add support to the scheduler to run autoserv --collect_crashinfo after a job finishes or is aborted.
* added new state "Gathering" for when we're running collect_crashinfo and copying logs to the results repository
* added new GatherLogsTask to the scheduler to perform these two tasks, and made it get run either after a job finishes or after a job is aborted. this task shares a lot with FinalReparseTask, so extracted common code into a new PostJobTask.
* made changes to scheduler/drone code to support generic monitoring and recovery of processes via pidfiles, since we need to be able to recover the collect_crashinfo processes too. this will also made the scheduler recover parse processes instead of just killing them as it does now, which is nice.
* changed abort logic significantly. since we now need to put aborted jobs through the gathering and parsing stages, but then know to put them into "aborted" afterwards, we can't depend on the old path of abort -> aborting -> aborted statuses. instead, we need to add an "aborted" flag to the HQE DB table and use that. this actually makes things generally cleaner in my opinion -- for one, we can get rid of the "Abort" and "Aborting" statuses altogether. added a migration to add this flag, edited model and relevant logic appropriately, including changing how job statuses are reported for aborted entries.
Signed-off-by: Steve Howard <showard@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@3031 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| 946a7af939c07a64a627234acb094688125c4683 |
|
15-Apr-2009 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Ensure one-time-hosts get unlocked when created.
Signed-off-by: Steve Howard <showard@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@2989 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| c92da83b2dd5b7992a58081c6005b3dfb3dc5f3a |
|
07-Apr-2009 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Add support for atomic groups to the frontend RPC interface.
Signed-off-by: Gregory Smith <gps@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@2967 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| 205fd60f9c9d2f64ec2773f295de1cf5cfd3bc77 |
|
21-Mar-2009 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Fix the AtomicGroup name display in the admin interface.
Adds an invalid bool column and use the existing invalid model to avoid
problems when deleting from the django admin interface.
Signed-off-by: Gregory Smith <gps@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@2918 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| 89f84dbadf071ba430244356b57af395c79486e4 |
|
12-Mar-2009 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Add the concept of an Atomic Group to the scheduler and database.
Scheduling a job on an atomic group means that all of the Ready machines
(up to a maximum specified in the atomic group) in a single label associated
with that atomic group will be used to run the job.
The job synch_count becomes a minimum when scheduling on an atomic group.
Both HostQueueEntrys and Labels may have an AtomicGroup associated with
them:
* A HostQueueEntry with an AtomicGroup acts to schedule a job on
all Ready machines of a single Label associated with that AtomicGroup.
* A Label with an AtomicGroup means that any Hosts bearing that Label
may only be scheduled together as a group with other hosts of that Label
to satisify a Job's HostQueueEntry bearing the same AtomicGroup.
Such Hosts will never be scheduled as normal metahosts.
Future patches are coming that will add the ability to schedule jobs using
this feature to the RPC interface, CLI and GUI.
Signed-off-by: Gregory Smith <gps@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@2878 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| 25cbdbd2da6242296abe6b1342521b29993993f2 |
|
17-Feb-2009 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Get rid of the host_queue_entries.priority field in the autotest_web DB. It's just duplicating information from the jobs table.
Signed-off-by: Steve Howard <showard@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@2813 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| d9ac445a60d6d11537f566503164344e09527917 |
|
07-Feb-2009 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Remove the old acl_groups_users and acl_groups_hosts many2many pivot
table column name hack.
Renames acl_group_id -> aclgroup_id.
Adds a migration script and updates the one piece of code that actually
depended upon the old name.
This is needed for my upcoming change that ports autotest to Django
1.0.x but seems worth cleaning up as a change of its own.
Signed-off-by: Gregory Smith <gps@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@2764 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| b5b7b5dc37012db192e9e47cd8e6fe45faf48dd2 |
|
03-Feb-2009 |
mbligh <mbligh@592f7852-d20e-0410-864c-8624ca9c26a4> |
Improved error message to the user when entering a host as a one time
host that is already in the autotest database.
Signed-off-by: Gregory Smith <gps@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@2731 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| 65c267df570f3e827c8a700f3746c26fcab0da77 |
|
21-Jan-2009 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Check synch_count in RPC method, before job gets created. Also remove a call to save() which was redundant (it gets called in add_object()). It was misleading and was probably what led to the erroneously placed check on synch_count.
Signed-off-by: Steve Howard <showard@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@2661 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| 67831ae7a70ab35b48f98fad7f14d285b69d9bd9 |
|
16-Jan-2009 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Make sure the debug messages from models get printed while the scheduler is running. The log level got changed during the scalability work and this was disabling output of these important status change messages,
Signed-off-by: Steve Howard <showard@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@2657 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| 5a198b91a2d3ca37943d02628886aac19ee04283 |
|
11-Dec-2008 |
mbligh <mbligh@592f7852-d20e-0410-864c-8624ca9c26a4> |
Add 'Verifying' to class Job so that View Job allows users to view machines
that are 'stuck' in verify or taking a long time in cleaning
Signed-off-by: Scott Zawalski <scottz@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@2560 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| c85c21bec23af3f805397f1f7e289b9a4b0bfd05 |
|
24-Nov-2008 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
* allow scheduler email "from" address to be specified in global config
* allow global config to specify statuses which should trigger immediate emails (in addition to email upon job completion)
* make "Parsing" an active=complete=0 status, and modify Job.num_complete() appropriately
* restructuring of scheduler email notification code to only email at the right time and to include a more informative message
git-svn-id: http://test.kernel.org/svn/autotest/trunk@2506 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| a58594a616ab5364c72aa824b7ccde7cdd7b10bf |
|
18-Nov-2008 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Don't reset active bit when setting status to "Abort". The scheduler needs that information to know if the queue entry was active before being aborted.
git-svn-id: http://test.kernel.org/svn/autotest/trunk@2436 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| 3f15eedfedab4b1b9eaff67713e411eada0c84df |
|
14-Nov-2008 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Fix bug in generation of error message for unauthorized abort that occurs when aborting an unassigned metahost entry.
git-svn-id: http://test.kernel.org/svn/autotest/trunk@2419 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| 97db5ba6cdee21d706dff3d1a85c354c8bfae6f9 |
|
12-Nov-2008 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Add a user preference for showing experimental tests, disabled by default, and add code to obey it in TestSelector.
git-svn-id: http://test.kernel.org/svn/autotest/trunk@2406 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| dc8175153fc559d1e7ae605b15ceef66e25e7cde |
|
12-Nov-2008 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
allow users to abort other users' jobs, but only for machines that are both ACL-accessible and *not* in the "Everyone" ACL.
git-svn-id: http://test.kernel.org/svn/autotest/trunk@2404 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| 2bab8f45adedeacbf2d62d37b90255581adc3c7d |
|
12-Nov-2008 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Implement sync_count. The primary change here is replacing the job.synch_type field with a synch_count field. There is no longer just a distinction between synchronous and asynchronous jobs. Instead, every job as a synch_count, with synch_count = 1 corresponding to the old concept of synchronous jobs. This required:
-changes to the job creation RPC and corresponding client code in AFE and the CLI
-massive changes to the scheduler to schedule all jobs in groups based on synch_count (this unified the old synch and async code paths)
-changed results directory structure to accomodate synchronous groups, as documented at http://autotest.kernel.org/wiki/SchedulerSpecification, including widespread changes to monitor_db and a change in AFE
-changes to AFE abort code to handle synchronous groups instead of just synchronous jobs
-also got rid of the "synchronizing" field in the jobs table, since I was changing the table anyway and it seems very likely now that that field will never be used
other changes included:
-add some logging to afe/models.py to match what the scheduler code does, since the scheduler is starting to use the models more
-added checks for aborts of synchronous groups to abort_host_queue_entries RPC
git-svn-id: http://test.kernel.org/svn/autotest/trunk@2402 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| 45ae819b2d2a67d0882edafaf1a8f7b95c3fb9d2 |
|
05-Nov-2008 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Add a formal cleanup phase to the scheduler flow.
-add a --cleanup or -C option to autoserv, which runs a new control segment, cleanup. this option and control segment obsolete the old -b option and reboot_segment control segment.
-change the RebootTask in the scheduler into a more generic CleanupTask, which calls autoserv --cleanup.
-change the host status "Rebooting" to "Cleaning"
git-svn-id: http://test.kernel.org/svn/autotest/trunk@2377 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| 97aed504f709270614ccbcef299e394333a76598 |
|
04-Nov-2008 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Rewrite final reparse code in scheduler. the final reparse is now handled by a separate AgentTask, and there's a "Parsing" status for queue entries. This is a cleaner implementation that allows us to still implement parse throttling with ease and get proper recovery of reparses after a system crash fairly easily.
git-svn-id: http://test.kernel.org/svn/autotest/trunk@2367 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| a3ab0d56117c0d55f768d5817284c2c2a0b0305d |
|
03-Nov-2008 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
-change AFE abort code to always set to "Abort" status and never skip straight to "Aborted". Doing so is prone to a race condition with the scheduler. The scheduler handles a non-active "Abort" entries perfectly already, setting them immediately to "Aborted" without trying to kill anything.
-change scheduler timeout code to use AFE models abort code instead of it's own SQL version. unfortunately we need need a fragment of SQL to do the timeout computation, which means no testability under SQLite. I manually tested it.
-also extracted the scheduler periodic "cleanup" code into its own method.
git-svn-id: http://test.kernel.org/svn/autotest/trunk@2365 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| 9886397ceb7c752db78a6acd9737992db891015b |
|
29-Oct-2008 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Add job start timeout for synchronous jobs. This timeout applies to synchronous jobs that are holding a public pool machine (i.e. in the Everyone ACL) as "Pending". This includes a new global config option, scheduler code to enforce the timeout and a unit test.
Note that the new scheduler code uses the Django models instead of making DB queries directly. This is a first example of how the scheduler can use the models to simplify DB interaction and reuse code from the frontend. I'd like to move in this direction from now on, although I certainly won't be making any sweeping changes to rewrite existing code.
git-svn-id: http://test.kernel.org/svn/autotest/trunk@2358 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| 0fc3830f17d644bab74bfe38556299f5e58bc0fa |
|
23-Oct-2008 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Add user preferences for reboot options, including simple user preferences tab which could later be expanded to include more options.
git-svn-id: http://test.kernel.org/svn/autotest/trunk@2330 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| 21baa459ea14f96e06212f1f35fcddab9442b3fc |
|
21-Oct-2008 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Add options to control reboots before and after a job.
-add reboot_before and reboot_after fields to Job, along with enums for each
-add options to create_job RPC for reboot_before and reboot_after
-add options to job create CLI for these fields, and made job stat -v display them
-add widgets to job create page in AFE for these fields and made job detail view display them
-add dirty field to Hosts, defaulting to True, and set to True when a host is locked
-made scheduler set this field when a job runs and clear it when a host is rebooted
-updated scheduler's PidfileRunMonitor to read a new three-line .autoserv_execute format, where the third line contains the number of tests that failed
-made scheduler Job.run() include a RebootTask before the verify task according to the reboot_before option
-made QueueTask.epilog() launch a RebootTask for each host according to the reboot_after option
-updated autoserv to write out a third line to .autoserv_execute containing the number of failed tests.
Other changes:
-added support for displaying Job.run_verify in the CLI (job stat -v) and job detail page on AFE
-updated ModelExtensions to convert BooleanField values to actual booleans. The MySQL Django backend just leaves them as ints (as they are represented in the DB), and it's stupid and annoying (Yes, bool is a subclass of int, so it's often not a problem. But yes, it can be.).
-get rid of use of Job.synch_count since we don't actually support it. I think this was meant for inclusion in a previous change and got left out.
-made the scheduler use the new setup_django_environment stuff to import and use the django models. It doesn't *really* use the models yet -- it just uses the Job.Reboot{Before,After} enum objects -- but this shows we could easily start using the models, and that's definitely the direction I want to go long term.
-refactored PidfileRunMonitor generally and made it a bit more robust by having it email errors for corrupt pidfiles and continue gracefully, instead of just crashing the scheduler
-changed the way Agent.tick() works. now, it basically runs through as much work as it can in a single call. for example, if there's a RebootTask and a VerifyTask, and the RebootTask has just finished, in a single call it will finish up the RebootTask and start the VerifyTask. this used to take two cycles and that was problematic for cases like this one -- the RebootTask would like to set host.status=Ready, but then the host could get snatched up on the next scheduling round, before the VerifyTask got started. This was sort of solved previously by keeping the HostQueueEntry active, and we could apply that approach here by making a new status for HostQueueEntries like "Rebooting". But I prefer this approach as I think it's more efficient, more powerful and easier to work with.
Risk: extremely high
Visibility: new reboot options for jobs, skip verify now displayed in AFE + CLI
Signed-off-by: Steve Howard <showard@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@2308 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| cfd66a35e8c939998ff354d9740bdf15c9bc3fda |
|
15-Oct-2008 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Make scheduler set host status to "Pending" when there's a pending queue entry against the host.
git-svn-id: http://test.kernel.org/svn/autotest/trunk@2292 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| 9976ce9873867a397e448d358543a9dc1d33aa77 |
|
15-Oct-2008 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
-make monitor_db implement "skip verify" properly, and add unit tests for it
-change order of a couple fields in AFE models to match DB order
git-svn-id: http://test.kernel.org/svn/autotest/trunk@2291 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| 9dbdcda5104991cbf344ea5cba1aa58e1af444f3 |
|
14-Oct-2008 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Add feature to abort individual host queue entries the job detail and host detail pages. Performed a few other cleanups along the way.
-refactor TableActionsPanel, extract separate TableSelectionPanel
-make TableDecorator support a SelectionManager and a TableActionsPanel (or selection panel or other such control) just above the paginator. this makes it really easy to add selection capability, selection checkboxes, and an actions menu to any table.
-refactor TableDecorator in general since it kinda sucked
-change all existing code that does table selection to use the new TableDecorator support. this include job list view, the create job host selector (which now uses the common selection links instead of its old buttons), and tko.TableView.
-add selection support + a bulk abort action to the tables in job detail view and host detail view. this itself was easy given the above refactorings.
git-svn-id: http://test.kernel.org/svn/autotest/trunk@2283 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| 68c7aa02743042196b1f4ce3d41cbe03d1152ec0 |
|
09-Oct-2008 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Don't rely on auto_now_add in Django, since it is apparently not
reliable.
Risk: Low
Visibility: No more supposedly auto-add date fields set to zero in
the database instead of now.
Signed-off-by: John Admanski <jadmanski@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@2258 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| b1e5187f9aa303c4fc914f07312286d302b46a0e |
|
07-Oct-2008 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Get the scheduler unittest to run against SQLite!
* get rid of monitor_db.DatabaseConn, and make monitor_db use the new DatabaseConnection
* modify some queries in monitor_db that weren't SQLite-compatible (SQLite doesn't support TRUE and FALSE literals)
* add frontend/django_test_utils.py, which contains utilities to
* setup a django environment (something manage.py normally does for you)
* replace the configured DB with a SQLite one, either in-memory or on disk
* run syncdb on the test DB
* backup and restore the test DB, handy because then we can syncdb once, save the fresh DB, and quickly restore it between unittests without having to run syncdb again (syncdb is terribly slow for whatever reason)
* modify monitor_db_unittest to use these methods to set up a temporary SQLite DB, run syncdb on it, and test against it
* replace much of the data modification code in monitor_db_unittest with use of the django models. The INSERTs were very problematic with SQLite because syncdb doesn't set database defaults, but using the models solves that (django inserts the defaults itself). using the models is much cleaner anyway as you can see. it was just difficult to do before, but now that we've got the infrastructure to setup the environment anyway, it's easy. this is a good model for how we can make the scheduler use the django models eventually.
* reorder fields of Label model to match actual DB ordering; this is necessary since monitor_db depends on field ordering
* add defaults to some fields in AFE models that should've had them
* make DatabaseConnection.get_test_database support SQLite in files, which gives us persistence that is necessary and handy in the scheduler unittest
* add a fix to _SqliteBackend for pysqlite2 crappiness
The following are extras that weren't strictly necessary to get things working:
* add a debug feature to DatabaseConnection to print all queries
* add an execute_script method to DatabaseConnection (it was duplicated in migrate and monitor_db_unittest)
* rename "arguments" to "parameters" in _GenericBackend.execute, to match the DB-API names
* get rid of some debug code that was left in monitor_db, and one unnecessary statement
Signed-off-by: Steve Howard <showard@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@2252 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| 56e9377c6d618ba478bc2ca92679830ece8dff52 |
|
06-Oct-2008 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Get the frontend unittest to run against SQLite. This required scattered changes:
* change frontend_unittest to tell django to use an in-memory SQLite database. ideally this would've been the only change necessary, but it wasn't.
* change long to ints in the doctests, as that's what pysqlite2 returns
* change several imports to use autotest_lib; that's the way everything should be, and the other way breaks things when trying to run stuff without using manage.py
* make readonly_connection better support testing by changing its connection attribute into a method, and adding the capability to disable the module completely during testing
* get rid of use of GROUP_CONCAT in models.py; SQLite doesn't support it (not in our old version anyhow), and it;s really not necessary anyway
git-svn-id: http://test.kernel.org/svn/autotest/trunk@2244 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| 9a1f2e1062efdef97e3d229dbdc941b61545e7b2 |
|
02-Oct-2008 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Two bug fixes:
* When aborting host queue entries upon job deletion, need to pass the current user, since abort() now accepts an aborted_by param.
* Fixed bug in JobManager.populate_dependencies(), where it would construct an invalid SQL query because it assumed the input list wasn't empty.
git-svn-id: http://test.kernel.org/svn/autotest/trunk@2221 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| 989f25dcbb6361218f0f84d1c8404761b4c39d96 |
|
01-Oct-2008 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
two new major features:
(1) added test and job dependencies
-added M2M relationship between tests and labels and between jobs and labels, for tracking the labels on which a test/job depends
-modified test_importer to read the DEPENDENCIES field and create the right M2M relationships
-modified generate_control_file() RPC to compute and return the union of test dependencies. since generate_control_file now returns four pieces of information, i converted its return type from tuple to dict, and changed clients accordingly.
-modified job creation clients (GWT and CLI) to pass this dependency list to the create_job() RPC
-modified the create_job() RPC to check that hosts satisfy job dependencies, and to create M2M relationships
-modified the scheduler to check dependencies when scheduling jobs
-modified JobDetailView to show a job's dependencies
(2) added "only_if_needed" bit to labels; if true, a machine with this label can only be used if the label is requested (either by job dependencies or by the metahost label)
-added boolean field to Labels
-modified CLI label creation/viewing to support this new field
-made create_job() RPC and scheduler check for hosts with such a label that was not requested, and reject such hosts
also did some slight refactoring of other code in create_job() to simplify it while I was changing things there.
a couple notes:
-an only_if_needed label can be used if either the job depends on the label or it's a metahost for that label. we assume that if the user specifically requests the label in a metahost, then it's OK, even if the job doesn't depend on that label.
-one-time-hosts are assumed to satisfy job dependencies.
Signed-off-by: Steve Howard <showard@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@2215 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| 4c119049441b360505464f31cfd113b5dc90b241 |
|
29-Sep-2008 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Add support to the frontend for logging a user and timestamp when a
job is aborted. Doesn't actually expose this information anywhere
(yet).
Risk: Medium
Visibility: A new table with extra log info about aborted host queue
entries will be written to when jobs are aborted.
Signed-off-by: John Admanski <jadmanski@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@2201 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| 542e840486b02b5025d26da16f98fed97898a601 |
|
19-Sep-2008 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Added email_list field to front end. On job completion emails on this
list will be notified of the completion. Also the function send_email
in manage_db.py adds the ability to put a delimited email list in
global_settings.py rather than just a single email.
Signed-off-by: Bryce Boe <bboe@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@2173 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| d5afc2f14dad583057f1eaffbc0366eb389f70d6 |
|
12-Aug-2008 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Fixed the protection field in the Host class, so that a default of "No
protection" can be set correctly. Also fixed the RPC interface unit
tests.
Risk: low
Visibility: low
Signed-off-by: James Ren <jamesren@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@1976 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| a8411aff779fc965537d8470268107b5a87e0346 |
|
08-Aug-2008 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Validate one-time hosts to avoid empty hostnames.
git-svn-id: http://test.kernel.org/svn/autotest/trunk@1960 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| 1ab512bb148ff7706cde7d74eee9de08c47118e0 |
|
31-Jul-2008 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
One-time hosts now have their protection levels set to "Do not repair".
Risk: low
Visbility: medium (host creation behavior change)
Signed-off-by: James Ren <jamesren@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@1932 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| e65d2af7d45b147e899f9158f39aeeb204e8f5a9 |
|
31-Jul-2008 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Creating a new job with a one-time host that has been previously used
now works as expected. The job will be successfully created.
Risk: low
Visibility: medium (UI bug fix)
Signed-off-by: James Ren <jamesren@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@1930 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| 04f2cd81b1776c12085d9314ec27c3a7f755a843 |
|
25-Jul-2008 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Automatically add the current user to new ACL groups. Also add the user
to the group if all users are removed from the ACL group.
Risk: low
Visibility: medium (changing behavior)
Signed-off-by: James Ren <jamesren@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@1904 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| 517f205cb5dfbe5d52a4ab7a3811c93a97af4c46 |
|
25-Jul-2008 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Remove requeue_job from the RPC interface. Neither the CLI nor the web
frontend uses it anymore. Also updated the frontend unit test.
Risk: low
Visibility: low
Signed-off-by: James Ren <jamesren@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@1903 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| d823b3648911ef831e75c9d645bba5cfbfed4dc4 |
|
24-Jul-2008 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Added handling in the frontend for the 'Starting' job status.
Risk: low
Visibility: medium (UI bug fix)
Signed-off-by: James Ren <jamesren@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@1896 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| 31c570e61cdbe7edb5318525871162819d52fd96 |
|
23-Jul-2008 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Fix adding new ACL groups. Adding a check that the original_object
attribute exists in AclGroup.Manipulator before accessing it.
Risk: low
Visibility: medium (UI bug fix)
Signed-off-by: James Ren <jamesren@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@1881 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| fb2a7fa621ddc91634dde6c56a47a1c8df2610ef |
|
17-Jul-2008 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Adding new columns "locked_by_id" and "lock_time" to the hosts table, to
indicate who locked a host and when.
Risk: low
Visibility: low
Signed-off-by: James Ren <jamesren@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@1864 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| 909c7a661e6739c110690e70e2f4421ffc4e5433 |
|
15-Jul-2008 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Initial release of test auto importer
Update models.py to reflect database changes
Add the following columns to autotests table:
* author
* dependencies
* experimental
* run_verify
* test_time
* test_category
* sync_count
Add run_verify to jobs table
Update scheduler to assert with run_verify
Risk: Medium
Visibility: High, people addings tests will now see more fields via the admin frontend
Signed-off-by: Scott Zawalski <scottz@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@1837 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| 3dd47c247147ca6920b222d43a8eb6ee54209c8f |
|
10-Jul-2008 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Added better ACL group support. More checks are now performed to ensure
that users cannot create jobs against hosts that are not visible, and
that users cannot abort other users' jobs.
Note: If applying this patch, you must also apply the "move logged-in user
tracking to separate thread_local module" patch.
Risk: medium (disallowing actions may have unintended side effects)
Visibility: medium (new errors will occur if disallowed actions are
attempted)
Signed-off-by: James Ren <jamesren@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@1797 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| df062569a6407ec084c4ee05b9390f8a0183d37b |
|
03-Jul-2008 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Adding protection levels to hosts. Allows the user to specify how much
the repair operation is allowed to do on the host (e.g., do not repair,
repair filesystem only, allow reimaging).
Risk: low
Visbility: medium (adding a new input field)
git-svn-id: http://test.kernel.org/svn/autotest/trunk@1771 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| b8471e33ed22512001ec4cec8c33fdf01f32eb62 |
|
03-Jul-2008 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Added a new input that allows used to specify a one-time host when
creating a job. The job will be run against that host once, and the
host will not appear in the "Available hosts" selector.
Risk: medium (deleting records from database)
Visibility: medium (adding an input field)
git-svn-id: http://test.kernel.org/svn/autotest/trunk@1768 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| 3bb499f74c04acec1f802a531cdfcba8f5ac0164 |
|
03-Jul-2008 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Adding a timeout field to the "Create Job" tab, modified the create_job
RPC to handle a "timeout" argument, and added a "timeout" column to the
AUTOTEST_WEB database. Sets how long a job should run until it is
automatically aborted.
git-svn-id: http://test.kernel.org/svn/autotest/trunk@1765 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| 08f981bddc5f1a199d789d177cbf583dee9f6c17 |
|
24-Jun-2008 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
-every time an ACL group is edited, automatically
-add hosts to everyone if they aren't a member of any group
-remove hosts from everyone if they're a member of any other group
-simplify job block logic in models.py back to the way it used to be.
-get rid of some dead code in model_logic. we can resurrect it later if we need it later.
git-svn-id: http://test.kernel.org/svn/autotest/trunk@1738 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| 2b9a88bcd336233fa483490892338e29f0a5fa67 |
|
13-Jun-2008 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
-allow users to select profilers in the create job view
-create profilers table + model
-create RPCs for managing profilers
-modify control file generation code to accept profilers
-modify create job view
-some refactoring to create job view to allow code sharing and do things more cleanly with some sweet generics
-get rid of rpc_utils.sorted. who knew there was a builtin?
git-svn-id: http://test.kernel.org/svn/autotest/trunk@1703 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| 0afbb6369aa5aa9a75ea67dd9e95ec4b21c0c181 |
|
06-Jun-2008 |
jadmanski <jadmanski@592f7852-d20e-0410-864c-8624ca9c26a4> |
Convert all python code to use four-space indents instead of eight-space tabs.
Signed-off-by: John Admanski <jadmanski@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@1658 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| 20f47064be6855277e251cee7611d8336bcc9149 |
|
05-Jun-2008 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
-check ACLs directly in the scheduler (bypassing ineligible_host_queues)
-rewrite scheduler queries to avoid all subqueries. they are just bad in mysql.
-rip out all that code related to using ineligible_host_queues to enforce ACLs. good riddance!
-update scheduler unit test to reflect this new policy (no ineligible_host_queue blocks for ACLs)
-minor bugfixes to scheduler unit test. this sucks, but i did go back and ensure the old scheduler passed the fixed up unit test suite as well.
-remove a blanket except: block from the scheduler. it wasn't necessary, it was inconsistent, and it was interfering with unit testing.
git-svn-id: http://test.kernel.org/svn/autotest/trunk@1608 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| 7c7852819d0611b4d0e8e69b3011b79e4016a770 |
|
29-May-2008 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Restructure code in preparation for sharing with new TKO.
-Change structure of RPC handler. New rpc_handler.py is independent of AFE and will eventually be moved to a common dir, along with the JSON and JSON-RPC stuff. rpc_handler.py now has a class structure that allows multiple apps to instantiate their own RPC handlers with different interfaces.
-Pull out generic model extensions from models.py into model_logic.py. This too will be moved to a common dir.
git-svn-id: http://test.kernel.org/svn/autotest/trunk@1566 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| 5875c151c05a62346afd214ec35122018849ec99 |
|
28-May-2008 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Add commits for the two methods in which we do manual queries that modify data. Turns out Django disables MySQL autocommit but has its own autocommit mechanism. This works fine until you start doing queries manually, in which case you need to commit manually.
git-svn-id: http://test.kernel.org/svn/autotest/trunk@1560 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| 3cbce7cce58bffe9b77ed5d06b9f118311aa5460 |
|
22-May-2008 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Add one more SELECT DISTINCT that should've been there.
git-svn-id: http://test.kernel.org/svn/autotest/trunk@1545 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| 9a179061f5e5c7c412d5fd4768c1417f47c0dee9 |
|
20-May-2008 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Change filter_[not_]in_subquery to use explicit joins instead of IN. Some versions of MySQL consider all subqueries dependent when used with IN, resulting in terrible performance. Explicit joins are properly optimized. This should be at least 50x faster.
git-svn-id: http://test.kernel.org/svn/autotest/trunk@1532 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| 21b4a1b7bffd11ee0472a7cc30a6b535f97f0fcf |
|
15-May-2008 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Need to call on_change() on the ACL object, not the manipulator.
git-svn-id: http://test.kernel.org/svn/autotest/trunk@1520 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| e01a9ab0fe7e310c70f4df1d9159a4411ea4d468 |
|
13-May-2008 |
mbligh <mbligh@592f7852-d20e-0410-864c-8624ca9c26a4> |
Add the ability to view all machines rebooting on the server.
I have been asked a few times about being able to view this and have also
run into some areas where this would have been helpful
Signed-off-by: Scott Zawalski <scottz@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@1508 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| 2a9378b200ace9a96afb1166105a57da6e862731 |
|
09-May-2008 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Add docstrings I forgot to add previously.
git-svn-id: http://test.kernel.org/svn/autotest/trunk@1500 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| c2992ba854878443b7b7ca0f424edc3c06f1cfd8 |
|
06-May-2008 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Speed up ineligible host queue computations, to speed up ACL
operations. This involved:
-extending filter_in_subquery() to be a bit more flexible
-added create_in_bulk() and delete_in_bulk() to create/delete many
rows with one SQL query. I don't think Django has a built-in way to
create in bulk, and it had become a major performance issue. The bulk
delete capabilities of Django were unnecessarily inefficient for the
present purposes, so I added that as well since it was pretty simple,
although it's a less significant performance increase.
-make job block recomputation upon ACL change a bit more focused (only
recompute for possibly affected jobs)
Also added a new doctest for block computation since it is
particularly error-prone, but it's kind of behind-the-scenes so I
didn't want it cluttering the main rpc_test.txt doctest.
git-svn-id: http://test.kernel.org/svn/autotest/trunk@1492 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| b8d3424d02cfdc6c3eeda788e3527c5ec50ad8f9 |
|
25-Apr-2008 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
-make get_host_queue_entries return full info on related objects (host + job)
-display said info in job detail + host detail pages
git-svn-id: http://test.kernel.org/svn/autotest/trunk@1469 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| 34dc5fa61c425389f9b931d3970c7556baf44dca |
|
24-Apr-2008 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
-instead of that stupid job_status RPC, just provide get_host_queue_entries directly.
-improve reporting of metahost entries in web frontend
-fix bugs in models code which didn't surface until I started querying host queue entries
-fix bug with job detail page refreshing
git-svn-id: http://test.kernel.org/svn/autotest/trunk@1464 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| 5244cbb54ca3d426b35561c4853c356f3f51f5fd |
|
24-Apr-2008 |
mbligh <mbligh@592f7852-d20e-0410-864c-8624ca9c26a4> |
Never delete hosts or labels. Instead, mark them as invalid.
Signed-off-by: Steve Howard <showard@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@1462 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| eb3be4d9b4c1bb292723ea1d52612aec5967ca0e |
|
21-Apr-2008 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
-include acl-inaccessible hosts in ineligible_host_queues blocks. This should make metahosts obey ACLs.
-recompute blocks for all non-complete jobs after any acl changes
-make the scheduler clear out blocks for completed jobs
This is messier than I would've liked because it required two things Django is not very good at - subqueries and notification of every time a many-to-many relationship changes. I used a custom manager and a custom manipulator to get around the two, respectively, in as clean a way I could find.
git-svn-id: http://test.kernel.org/svn/autotest/trunk@1459 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| 3c43dc6bef3b2c551228027a43948208c1403024 |
|
15-Apr-2008 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Need to use distinct() since we're sometimes querying over multiple tables. This was causing a bug in the host table.
git-svn-id: http://test.kernel.org/svn/autotest/trunk@1428 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| 6f85a1f0e9f3dc342d376a9245075106eeedc011 |
|
10-Apr-2008 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
To hide fields from the admin interface, use editable=False instead of the Admin fields specifier. This is the right way to do it and lets us hide a field (i.e. host status) without causing admin errors or weirdness.
git-svn-id: http://test.kernel.org/svn/autotest/trunk@1423 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| 8e3aa5e027e28d8fdcb12083e0b30a80767a35cf |
|
08-Apr-2008 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
-support filtering on multiple labels
-fix problem mostly related to history handling that was causing unnecessary RPCs to be executed
git-svn-id: http://test.kernel.org/svn/autotest/trunk@1414 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| 1c8c2215e525de8813c375e796354f8ffb811a08 |
|
03-Apr-2008 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Change all tables to be server-side. This required a lot of work, but I think I ended up cleaning up the frontend code a lot and providing room for more powerful features to be added in the future. It is a huge change though.
git-svn-id: http://test.kernel.org/svn/autotest/trunk@1396 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| 31a244a2cc1c2947ce0f6672a9e71182a3737911 |
|
25-Mar-2008 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Be sure to always trim hostnames.
git-svn-id: http://test.kernel.org/svn/autotest/trunk@1360 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| 1385b161fa9391242a2121027878e4943ce3c81f |
|
13-Mar-2008 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
On the frontend, only show hosts to which this user has ACL access.
git-svn-id: http://test.kernel.org/svn/autotest/trunk@1343 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| 8fd5824df1310058af6b8b503b800d575ddd3787 |
|
10-Mar-2008 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
-associate a synch type with each test in the DB (defaults to asynchronous for all)
-make frontend default to synch type for selected tests, and disable selecting tests of different synch types
-allow synchronous client-side tests
git-svn-id: http://test.kernel.org/svn/autotest/trunk@1322 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| 99229e489bcda356c67582890124172307541fe9 |
|
06-Mar-2008 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Get rid of job.status and job.submitted_on usage in models.py. This field was deprecated but these (harmless) statements were left lying around.
git-svn-id: http://test.kernel.org/svn/autotest/trunk@1308 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| 3cab4a71760ef833f9549778f09197d1ffbe724c |
|
06-Mar-2008 |
mbligh <mbligh@592f7852-d20e-0410-864c-8624ca9c26a4> |
Requeue support for the frontend. Requeue creates and enqueues a copy of the job. Added support to Job model, new RPC method, and new button to job detail page.
Also fixing a little bug involving the is_dead() method.
Signed-off-by: Steve Howard <showard@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@1305 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| d5c95802c1045ca9fa84da18758bd9d44a7e04bf |
|
05-Mar-2008 |
mbligh <mbligh@592f7852-d20e-0410-864c-8624ca9c26a4> |
Implemented abort functionality in scheduler.
From: Svitlana Tumanova <stumanova@google.com>
Signed-off-by: Steve Howard <showard@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@1293 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|
| e8819cdf80ca0e0602d22551a50f970aa68e108d |
|
15-Feb-2008 |
mbligh <mbligh@592f7852-d20e-0410-864c-8624ca9c26a4> |
Add frontend and scheduler for Autotest
The attached tarball includes the new Autotest web frontend for creating
and monitoring jobs and the new scheduler for executing jobs. We've
been working hard to get these complete and stabilized, and although
they still have a long, long way to go in terms of features, we believe
they are now stable and powerful enough to be useful.
The frontend consists of two parts, the server and the client. The
server is implemented in Python using the Django web framework, and is
contained under frontend/ and frontend/afe. The client is implemented
using Google Web Toolkit and is contained under frontend/client. We
tried to use Dojo initially, as was generally agreed on, but it proved
to be too young and poorly documented of a framework, and developing in
it was just too inefficient.
The scheduler is contained entirely in the scheduler/monitor_db Python
file. It looks at the database used by the web frontend and spawns
autoserv processes to run jobs, monitoring them and recording status.
The scheduler was developed by Paul Turner, who will be sending out some
detailed documentation of it soon.
I've also included the DB migrations code for which I recently submitted
a patch to the mailing list. I've included this code because it is
necessary to create the frontend DB, but it will (hopefully) soon be
part of the SVN repository.
Lastly, there is an apache directory containing configuration files for
running the web server through Apache.
I've put instructions for installing a full Autotest server, including
the web frontend, the scheduler, and the existing "tko" results backend,
at http://test.kernel.org/autotest/AutotestServerInstall. I've also
created a brief guide to using the web frontend, with plenty of
screenshots, at http://test.kernel.org/autotest/WebFrontendHowTo.
Please let me know if you find any problems with either of these pages.
Take a look, try it out, send me comments, complaints, and bugs, and I
hope you find it useful!
Steve Howard, and the rest of the Google Autotest team
From: Steve Howard <showard@google.com>
git-svn-id: http://test.kernel.org/svn/autotest/trunk@1242 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models.py
|