History log of /external/autotest/frontend/afe/models_test.py
Revision Date Author Comments (<<< Hide modified files) (Show modified files >>>)
7c8f23e9ac9b8895dbdb70b4fa6325ddb1ef04c5 15-Oct-2015 Ilja H. Friedel <ihf@chromium.org> Delete graphics labels.

Graphics tests will not rely on labels anymore. The code is brittle,
autotest codebase is not available on DUT during label generation (407951).
DEPENDENCIES have not enough features (348189). Instead graphics tests
will run everywhere and decide how much testing they are going to do on
that particular DUT.

BUG=chromium:348189, chromium:407951
TEST=grep.

Change-Id: I97d5561ba9314b733fec53195b981e3e43b7b0bd
Reviewed-on: https://chromium-review.googlesource.com/305841
Commit-Ready: Ilja Friedel <ihf@chromium.org>
Tested-by: Ilja Friedel <ihf@chromium.org>
Reviewed-by: Stéphane Marchesin <marcheu@chromium.org>
Reviewed-by: Haixia Shi <hshi@chromium.org>
/external/autotest/frontend/afe/models_test.py
6818633834ad52c3de153235639ea9299a6e9a6d 28-Apr-2015 Matthew Sartori <msartori@chromium.org> [autotest] Require lock reason to lock device

When locking a device, a locking reason now must be provided.
This applies to both adding a new device, and modifying an
existing device from both the web frontend and the atest
command-line tool.

BUG=chromium:336805
DEPLOY=migrate
TEST=Tested adding locked/unlocked devices and locking/unlocking devices
from both the web frontend and using the 'atest host ...' command-line tools.

Change-Id: I3a8cd8891a2999f026dd709ae8a79e2b8cbc251a
Reviewed-on: https://chromium-review.googlesource.com/267595
Tested-by: Matthew Sartori <msartori@chromium.org>
Reviewed-by: Dan Shi <dshi@chromium.org>
Commit-Queue: Matthew Sartori <msartori@chromium.org>
/external/autotest/frontend/afe/models_test.py
c9e1714424621786322d0e6ea48ac167bf35ce16 19-Feb-2015 Dan Shi <dshi@chromium.org> [autotest] Add a new field require_ssp in afe_jobs table

When a job is created, data in control file is parsed and saved in afe_jobs.
scheduler picks up the database row and tries to compile the commandline for
autoserv to run. require_ssp (require server-side packaging) is a new field
will be added to control file to force to use/do not use ssp when running
a server side test.

If the value is True or None(default), server-side packaging will be applied
to run server side test, if global config AUTOSERV/enable_ssp_container is
set to True.

Since the attribute can only be retrieved when parsing control file, it needs
to be stored in afe_jobs table. Therefore, the new field is added to the table.

require_ssp is nullable and has no default value, so the sql command should
only take several minutes to finish in production database.

BUG=chromium:453624
TEST=local run migrate.py sync and migrate.py sync 97
run scheduler locally and run a dummy suite, confirm scheduler and other
components work without issue (refer to bug 459523).

DEPLOY=migrate, db must be migrated after this CL is pushed to prod. Otherwise
scheduler will fail.

Change-Id: I7c65150b7ee3309310fdbed42b5079e564845a65
Reviewed-on: https://chromium-review.googlesource.com/251340
Trybot-Ready: Dan Shi <dshi@chromium.org>
Tested-by: Dan Shi <dshi@chromium.org>
Reviewed-by: Fang Deng <fdeng@chromium.org>
Commit-Queue: Dan Shi <dshi@chromium.org>
/external/autotest/frontend/afe/models_test.py
ff36159d40fc7e7b5124f4950b6a18103aebd922 03-Feb-2015 Fang Deng <fdeng@chromium.org> [autotest] Sync based on key instead of id for attribute-like models

HostAttribute and JobKeyval are updated on both shard and
master. Both table should be synced based on "key" instead
auto-incremental id.

BUG=chromium:453122
DEPLOY=shard_cient, apache
TEST=python
>>import common
>>from autotest_lib.frontend import setup_django_environment
>>from autotest_lib.frontend.afe import models
>>m = models.Host.objects.get(id=20)
>>s = m.serialize()
>>s['hostattribute_set'] = A LIST OF NEW ATTRIBUTE RECORDS
>>m = models.Host.deserialize(s)
>>m.hostattribute_set.all()[0].__dict__

>>j = models.Job.objects.get(id=35)
>>s = j.serialize()
>>s['jobkeyval_set'] = A LIST OF NEW JOBKEYVAL RECORDS
>>m = models.Job.deserialize(s)
>>m.jobkeyval_set.all()[0].__dict__

TEST=Run a shard, schedule a dummy suite, ensure everything still
works.

Change-Id: I9c6450f93f06510477312bd02b119fd96b70857c
Reviewed-on: https://chromium-review.googlesource.com/245560
Reviewed-by: Fang Deng <fdeng@chromium.org>
Tested-by: Fang Deng <fdeng@chromium.org>
Commit-Queue: Fang Deng <fdeng@chromium.org>
/external/autotest/frontend/afe/models_test.py
8c98ac10beaa08bfb975c412b0b3bda23178763a 23-Dec-2014 Prashanth Balasubramanian <beeps@google.com> [autotest] Send frontend jobs to shards.

Frontend jobs on hosts that are on the shard are disallowed
currently, because the host-scheduler on master currently
ignore jobs based on meta-host, but frontend jobs have no
meta-host. This CL have the following changes:
- Make host-scheduler ignore frontend jobs that are supposed
to be picked by shard.
- Send such frontend jobs in heartbeat.
- Allow creation of frontend jobs in rpc.

TEST=Test the follows:
- Create a job on a host on shard from AFE frontend.
Observe it runs on shards and completes on master.
- Create a job on two hosts (one host on shard, the other on master)
from AFE frontend. Make sure exception is railed with correct
message.
- Run a normal dummy suite on shard, make sure normal flow still
works. Heartbeat contains the right information.
- Run a normal dummy suite on master, make sure it works.
BUG=chromium:444790
DEPLOY=apache, host-scheduler

Change-Id: Ibca3d36cb59fed695233ffdc89506364c402cc37
Reviewed-on: https://chromium-review.googlesource.com/240396
Reviewed-by: Mungyung Ryu <mkryu@google.com>
Reviewed-by: Dan Shi <dshi@chromium.org>
Commit-Queue: Fang Deng <fdeng@chromium.org>
Tested-by: Fang Deng <fdeng@chromium.org>
/external/autotest/frontend/afe/models_test.py
f865d33cc3b0584f10b2f6f4de414a6d7f9e3000 29-Sep-2014 Jakob Juelich <jakobjuelich@chromium.org> [autotest] Allow updating selected model attributes

Right now, objects that have been persisted on the client once,
will never be overwritten again with new data, that comes from
the master.

In order to sync i.e. aborted bits from the master to the slave,
this needs to change.

This introduces a new variable SERIALIZATION_LOCAL_LINKS_TO_UPDATE,
which objects can overwrite with a set of names of attributes,
that should be overwritten with data that's sent from the master.

BUG=None
DEPLOY=shard_client
TEST=Ran suites.

Change-Id: I6b461475116db0acd2de7b8fe628aed791e7c418
Reviewed-on: https://chromium-review.googlesource.com/220331
Tested-by: Jakob Jülich <jakobjuelich@chromium.org>
Reviewed-by: Fang Deng <fdeng@chromium.org>
Commit-Queue: Jakob Jülich <jakobjuelich@chromium.org>
/external/autotest/frontend/afe/models_test.py
116ff0fc2674082fe475afd64ce4dec998ed71b7 18-Sep-2014 Jakob Juelich <jakobjuelich@chromium.org> [autotest] Deserializer: Allow updating objects

Currently the state of an object is never changed by the
deserializer.

After a shard executed a job, the master should be informed
about it's result so one can see if a job has been executed or not
in one central place for all jobs.

To update a job's status on the master, this is necessary.

Doing this, a bug of the previous implementation became apparent:
Deserializing a host object while no AclGroups were present yet
would create an Everyone-AclGroup in it's save() method. This is
dangerous because this way the states of master and shards might
diverge, specifically AclGroups with same names and different ids
might occur, which will then, upon dumping the Everyone-AclGroup from
the master, will cause exceptions to be raised.

Therefore just not running them prevents errors.

BUG=None
DEPLOY=scheduler,apache,host_scheduler
TEST=Ran suites, tested with real data on shard

Change-Id: Ie8ccd3205ac0efd5bbba4402da5de4c2615b495c
Reviewed-on: https://chromium-review.googlesource.com/218806
Reviewed-by: Prashanth B <beeps@chromium.org>
Commit-Queue: Jakob Jülich <jakobjuelich@chromium.org>
Tested-by: Jakob Jülich <jakobjuelich@chromium.org>
/external/autotest/frontend/afe/models_test.py
f88fa938050d0e2b662a2b32ef7e931e01a4d8fe 04-Sep-2014 Jakob Juelich <jakobjuelich@chromium.org> [autotest] Deserialize records and persist them

To send records from the master to shards it's necessary to serialize
them. This adds the deserialization functionality.

TEST=Ran suites.
DEPLOY=apache

Change-Id: I2806b0cfe1e4fbfba5d89e6d422800c7637ed4e9
Reviewed-on: https://chromium-review.googlesource.com/216355
Reviewed-by: Prashanth B <beeps@chromium.org>
Tested-by: Jakob Jülich <jakobjuelich@chromium.org>
Commit-Queue: Jakob Jülich <jakobjuelich@chromium.org>
/external/autotest/frontend/afe/models_test.py
3bb7c8095d7c66d955462673dd3df5565c1b4a96 03-Sep-2014 Jakob Juelich <jakobjuelich@chromium.org> [autotest] Add serialization functionality including dependencies

In order to send hosts and jobs to shards it's necessary to
serialize them including their dependencies.

This changelist adds functionality to find an object's dependencies
and to serialize it into lists and dicts, so it can be converted
to json.

TEST=Ran suites.
DEPLOY=apache

Change-Id: Icc158de1f23c7483ebc22b6ca3482b7ffccdd5c4
Reviewed-on: https://chromium-review.googlesource.com/215998
Tested-by: Jakob Jülich <jakobjuelich@chromium.org>
Reviewed-by: Alex Miller <milleral@chromium.org>
Commit-Queue: Jakob Jülich <jakobjuelich@chromium.org>
/external/autotest/frontend/afe/models_test.py
3dd8beb386f7298ffe84d7410d00cce26973e170 14-May-2013 Aviv Keshet <akeshet@chromium.org> [autotest] make a consistent CONTROL_TYPE enum across the codebase

Prior to this CL, there were a multitude of duplicate defitions of the
two control files types (Client or Server), incluiding a difference
between the afe Job model (1 = Server) and the afe Test model (1 =
Client). This CL introduces enums CONTROL_TYPE and CONTROL_TYPE_NAMES to
control_data, to act as the central and consistent defition across the
codebase. In order to avoid needing to mangle the running Jobs table, we
have adopted the existing Job model convention (1 = Server); the Test
table will be re-written with the new consistent convention during test
import.

BUG=chromium:240643
TEST=All existing unit tests pass;
In local autotest without this patch applied, started a suite. Halfway
through suite, applied this patch, ran test importer, restarted apache,
and restarted scheduler. Suite finished successfully.
Verified manually that Client/Server type control files show up
correctly in afe Create Job view.
DEPLOY=scheduler
DEPLOY=apache
DEPLOY=test_importer

Change-Id: Ia5b2573e1d08d96b3826f2837903ef407dcae303
Reviewed-on: https://gerrit.chromium.org/gerrit/51191
Commit-Queue: Aviv Keshet <akeshet@chromium.org>
Reviewed-by: Aviv Keshet <akeshet@chromium.org>
Tested-by: Aviv Keshet <akeshet@chromium.org>
/external/autotest/frontend/afe/models_test.py
4a41e0145c2ede3605d19cdfacb779ca6aca1ccc 17-Jul-2010 jamesren <jamesren@592f7852-d20e-0410-864c-8624ca9c26a4> Begin adding support for parameterized jobs.

This will allow test developers to specify certain parameters that a test
control file may take, so that users can then easily set those parameters
on job create. Enabling this feature removes the ability to edit the
control file directly on job creation.

Feature is currently INCOMPLETE. Do not attempt to use. This feature will
be committed in small pieces for the sake of having smaller code reviews.

Signed-off-by: James Ren <jamesren@google.com>



git-svn-id: http://test.kernel.org/svn/autotest/trunk@4720 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models_test.py
76fcf19ec42d5c7580d2e7891e4610e5fe725286 21-Apr-2010 jamesren <jamesren@592f7852-d20e-0410-864c-8624ca9c26a4> Add ability to associate drone sets with jobs. This restricts a job to
running on a specified set of drones.

Signed-off-by: James Ren <jamesren@google.com>


git-svn-id: http://test.kernel.org/svn/autotest/trunk@4439 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models_test.py
e36562323bec14d14b7c583db94788a725514ff2 02-Mar-2010 jamesren <jamesren@592f7852-d20e-0410-864c-8624ca9c26a4> Fix Django's QuerySet.delete() behavior for models that use "invalid"

Signed-off-by: James Ren <jamesren@google.com>


git-svn-id: http://test.kernel.org/svn/autotest/trunk@4276 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models_test.py
fd8b89f0117366c9aeaad9b600a43238a84b4ab9 20-Jan-2010 showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> don't set the current user to my_user in frontend_test_utils. let it default to the new autotest_system user.

Signed-off-by: Steve Howard <showard@google.com>


git-svn-id: http://test.kernel.org/svn/autotest/trunk@4156 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models_test.py
d11956572cb7a5c8e9c588c9a6b4a0892de00384 08-Dec-2009 showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> Make drone_manager track running processes counts using only the information passed in from the scheduler. Currently it also uses process counts derived from "ps", but that is an unreliable source of information. This improves accuracy and consistency and gives us full control over the process.

This involves a few primary changes:
* made the drone_manager track process counts with each PidfileId
* added method declare_process_count() for the scheduler to indicate the process count of a pidfile ID during recovery (in other cases, the DroneManager gets that info in execute_process())

Doing this involved some extensive refactorings. Because the scheduler now needs to declare process counts during recovery, and because the AgentTasks are the entities that know about process counts, it made sense to move the bulk of the recovery process to the AgentTasks. Changes for this include:
* converted a bunch of AgentTask instance variables to abstract methods, and added overriding implementations in subclasses as necessary
* added methods register_necessary_pidfiles() and recover() to AgentTasks, allowing them to perform recovery for themselves. got rid of the recover_run_monitor() argument to AgentTasks as a result.
* changed recovery code to delegate most of the work to the AgentTasks. The flow now looks like this: create all AgentTasks, call them to register pidfiles, call DroneManager to refresh pidfile contents, call AgentTasks to recover themselves, perform extra cleanup and error checking. This simplified the Dispatcher somewhat, in my opinion, though there's room for more simplification.

Other changes include:
* removed DroneManager.get_process_for(), which was unused, as well as related code (include the DroneManager._processes structure)
* moved logic from HostQueueEntry.handle_host_failure to SpecialAgentTask._fail_queue_entry. That was the only call site.
And some other bug fixes:
* eliminated some extra state from QueueTask
* fixed models.HostQueueEntry.execution_path(). It was returning the wrong value, but it was never used.
* eliminated some big chunks from monitor_db_unittest. These broke from the refactorings described above and I deemed it not worthwhile to fix them up for the new code. I checked and the total coverage was unaffected by deleting these chunks.

Signed-off-by: Steve Howard <showard@google.com>


git-svn-id: http://test.kernel.org/svn/autotest/trunk@4007 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models_test.py
9bb960b90d5102cce1c8a15314900035c6c4e69a 19-Nov-2009 showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> Support restricting access to drones by user. Administrators can put lines like

<hostname>_users: showard,scottz

in the global config, where <hostname> is a drone hostname. That drone will then be limited to use by those users (that is, by jobs launched by those users, and tasks launched due to those jobs). This required numerous changes:
* added a requested_by field to SpecialTask (with corresponding migration). For tasks with queue_entries, we can infer this from the job, but for those without, we need this information explicitly declared. Note this can be null if the task was created by the system, not in response to any user action. The only place this occurs now is in scheduler recovery (Dispatcher._recover_hosts_where()), but there may be an upcoming feature to periodically reverify hosts, which would be another (much more common) case.
* modified all SpecialTask creation sites to pass requested_by if necessary.
* modified AgentTask to keep a username attribute, and modified its run() method to pass that to PidfileRunMonitor.run(), which passes it along to DroneManager.execute_command().
* modified Agent to always keep self.task around, there's no reason to throw it away and now that we're looking at it from other classes, it's problematic if it disappears.
* modified Dispatcher throttling code to pass the username when requesting max runnable processes.
* added an allowed_users property to _AbstractDrone, and made DroneManager load it from the global config.
* made DroneManager's max_runnable_processes() and _choose_drone_for_execution() methods accept the username and obey user restrictions.
* added extensive tests for everything. the modiications required to monitor_db_unittest were annoying but not too bad. but parts of that file may need to be removed as they'll be obsoleted by monitor_db_functional_test and they'll become increasingly annoying to maintain.

couple other related changes:
* got rid of CleanupHostsMixin. it was only acutally needed by GatherLogsTasks (since we made the change to have GatherLogsTask always run), so I inlined it there and simplified code accordingly.
* changed a bunch of places in the scheduler that were constructing new instances of Django models for existing rows. they would do something like "models.Host(id=<id of existing host>)". that's correct for scheduler DBModels, but not for Django models. For Django models, you only instantiate new instances when you want to create a new row. for fetching existing rows you always use a manager -- Model.objects.get() or Model.objects.filter() etc. this was an existing bug but wasn't exposed until I made some of the changes involved in this feature.

Signed-off-by: Steve Howard <showard@google.com>


git-svn-id: http://test.kernel.org/svn/autotest/trunk@3961 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models_test.py
e60e44ece1445d97977a77cb79f0896989b869d7 13-Nov-2009 showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> Special tasks show "Failed" as their status instead of "Completed" if
they failed

Signed-off-by: James Ren <jamesren@google.com>


git-svn-id: http://test.kernel.org/svn/autotest/trunk@3946 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models_test.py
afd97de523a8c8c4e6657fa2db6214fda68d8086 01-Oct-2009 showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> When a one-time host is added as a normal host while it's running a job, the status would get reset, messing up the scheduler. Fix that.

Signed-off-by: Steve Howard <showard@google.com>


git-svn-id: http://test.kernel.org/svn/autotest/trunk@3778 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models_test.py
8cc058f50a46976e0a446aa3054f7f2349d6291a 08-Sep-2009 showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> Make scheduler more stateless. Agents are now scheduled only by the
dispatcher, and agents no longer use in-memory state to remember
multiple tasks. All state is managed by the database.

Risk: high (large scheduler change)
Visibility: medium (scheduler restarts are now more stable)

Signed-off-by: James Ren <jamesren@google.com>


git-svn-id: http://test.kernel.org/svn/autotest/trunk@3664 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models_test.py
6631273af8b88842cbd6202cc4615daf050cc957 27-Aug-2009 mbligh <mbligh@592f7852-d20e-0410-864c-8624ca9c26a4> Make a bunch of stuff executable



git-svn-id: http://test.kernel.org/svn/autotest/trunk@3621 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models_test.py
474d136e9a71f288f294167112dda909f9273434 21-Aug-2009 showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> Sanity check: prevent the creation of a special task for a host that already
has an active special task. This should only ever happen due to a scheduler
bug. Dying to allow human cleanup is desirable today.

Signed-off-by: Gregory Smith <gps@google.com>


git-svn-id: http://test.kernel.org/svn/autotest/trunk@3571 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models_test.py
a5288b4bb2b09aafe914d0b7d5aab79a7e433eaf 28-Jul-2009 showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> Upgrade from Django 0.96 to Django 1.0.2.

Risk: high (framework change)
Visibility: medium

Signed-off-by: James Ren <jamesren@google.com>


git-svn-id: http://test.kernel.org/svn/autotest/trunk@3457 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models_test.py
97446887819594f1e2a329dcff289ee8e934b626 21-Jul-2009 showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> Log changes to SpecialTask active/complete state.

Signed-off-by: Steve Howard <showard@google.com>


git-svn-id: http://test.kernel.org/svn/autotest/trunk@3426 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models_test.py
c0ac3a79cceec87873257a8b6d41d19c9fb02ec6 08-Jul-2009 showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> add the option to include verify/repair/cleanups in the job table on the host detail page.
* added RPC get_host_queue_entries_and_special_tasks, which returns both HQEs and SpecialTasks for a host in a carefully interleaved manner.
* added appropriate frontend unit tests
* added support to HostDetailView to include these entries, but have them not be selectable. this required changes to SelectionManager. I also added a utility method to Utils for opening a new window, and changed all sites that do that to call the new method.

Signed-off-by: Steve Howard <showard@google.com>


git-svn-id: http://test.kernel.org/svn/autotest/trunk@3385 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models_test.py
ed2afea4ca6e23a82d20d1f2ee1067d0c25a8cc2 07-Jul-2009 showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> make SpecialTasks recoverable. this involves quite a few changes.
* run tasks in determined dirs instead of temp dirs. the dir paths look like hosts//-, for example, hosts/myhost/4-verify. the ID comes from the SpecialTask DB row. this allows us to find the pidfile when we go looking for it during recovery, and it makes it simple to find the logs for any given special task, much like for HostQueueEntries. added SpecialTask.execution_path() for this purpose, and added models_test to test it.
* added execution_path() to HostQueueEntry to match the interface of SpecialTask, allowing for more polymorphism, and changed most call sites to use it.
* since we're running in these dirs, copy the full results back in these dirs, instead of just copying a single log file.
* move process recovery code up into AgentTask, so that all AgentTasks can share the same generic process recovery code.
* change SpecialTask recovery code to do process recovery.
* change VerifyTask handling of multiple pending verify requests for a machine. instead of updating all the requests, just delete all other tasks. they're not specially tracked in any way so it's simplest to just delete them.
* made special tasks get marked is_active=False when they complete, to be consistent with HQEs

other changes:
* added null=True to SpecialTask.time_started definition
* made EmailManager.enqueue_notify_email always log the message, and removed explicit logging calls from call sites
* added feature to DroneManager.execute_command() to automatically substitute the working directory into the command. this avoids some duplicate information being passed around and simplifies the unit test.

Signed-off-by: Steve Howard <showard@google.com>


git-svn-id: http://test.kernel.org/svn/autotest/trunk@3380 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models_test.py