7c8f23e9ac9b8895dbdb70b4fa6325ddb1ef04c5 |
|
15-Oct-2015 |
Ilja H. Friedel <ihf@chromium.org> |
Delete graphics labels. Graphics tests will not rely on labels anymore. The code is brittle, autotest codebase is not available on DUT during label generation (407951). DEPENDENCIES have not enough features (348189). Instead graphics tests will run everywhere and decide how much testing they are going to do on that particular DUT. BUG=chromium:348189, chromium:407951 TEST=grep. Change-Id: I97d5561ba9314b733fec53195b981e3e43b7b0bd Reviewed-on: https://chromium-review.googlesource.com/305841 Commit-Ready: Ilja Friedel <ihf@chromium.org> Tested-by: Ilja Friedel <ihf@chromium.org> Reviewed-by: Stéphane Marchesin <marcheu@chromium.org> Reviewed-by: Haixia Shi <hshi@chromium.org>
/external/autotest/frontend/afe/models_test.py
|
6818633834ad52c3de153235639ea9299a6e9a6d |
|
28-Apr-2015 |
Matthew Sartori <msartori@chromium.org> |
[autotest] Require lock reason to lock device When locking a device, a locking reason now must be provided. This applies to both adding a new device, and modifying an existing device from both the web frontend and the atest command-line tool. BUG=chromium:336805 DEPLOY=migrate TEST=Tested adding locked/unlocked devices and locking/unlocking devices from both the web frontend and using the 'atest host ...' command-line tools. Change-Id: I3a8cd8891a2999f026dd709ae8a79e2b8cbc251a Reviewed-on: https://chromium-review.googlesource.com/267595 Tested-by: Matthew Sartori <msartori@chromium.org> Reviewed-by: Dan Shi <dshi@chromium.org> Commit-Queue: Matthew Sartori <msartori@chromium.org>
/external/autotest/frontend/afe/models_test.py
|
c9e1714424621786322d0e6ea48ac167bf35ce16 |
|
19-Feb-2015 |
Dan Shi <dshi@chromium.org> |
[autotest] Add a new field require_ssp in afe_jobs table When a job is created, data in control file is parsed and saved in afe_jobs. scheduler picks up the database row and tries to compile the commandline for autoserv to run. require_ssp (require server-side packaging) is a new field will be added to control file to force to use/do not use ssp when running a server side test. If the value is True or None(default), server-side packaging will be applied to run server side test, if global config AUTOSERV/enable_ssp_container is set to True. Since the attribute can only be retrieved when parsing control file, it needs to be stored in afe_jobs table. Therefore, the new field is added to the table. require_ssp is nullable and has no default value, so the sql command should only take several minutes to finish in production database. BUG=chromium:453624 TEST=local run migrate.py sync and migrate.py sync 97 run scheduler locally and run a dummy suite, confirm scheduler and other components work without issue (refer to bug 459523). DEPLOY=migrate, db must be migrated after this CL is pushed to prod. Otherwise scheduler will fail. Change-Id: I7c65150b7ee3309310fdbed42b5079e564845a65 Reviewed-on: https://chromium-review.googlesource.com/251340 Trybot-Ready: Dan Shi <dshi@chromium.org> Tested-by: Dan Shi <dshi@chromium.org> Reviewed-by: Fang Deng <fdeng@chromium.org> Commit-Queue: Dan Shi <dshi@chromium.org>
/external/autotest/frontend/afe/models_test.py
|
ff36159d40fc7e7b5124f4950b6a18103aebd922 |
|
03-Feb-2015 |
Fang Deng <fdeng@chromium.org> |
[autotest] Sync based on key instead of id for attribute-like models HostAttribute and JobKeyval are updated on both shard and master. Both table should be synced based on "key" instead auto-incremental id. BUG=chromium:453122 DEPLOY=shard_cient, apache TEST=python >>import common >>from autotest_lib.frontend import setup_django_environment >>from autotest_lib.frontend.afe import models >>m = models.Host.objects.get(id=20) >>s = m.serialize() >>s['hostattribute_set'] = A LIST OF NEW ATTRIBUTE RECORDS >>m = models.Host.deserialize(s) >>m.hostattribute_set.all()[0].__dict__ >>j = models.Job.objects.get(id=35) >>s = j.serialize() >>s['jobkeyval_set'] = A LIST OF NEW JOBKEYVAL RECORDS >>m = models.Job.deserialize(s) >>m.jobkeyval_set.all()[0].__dict__ TEST=Run a shard, schedule a dummy suite, ensure everything still works. Change-Id: I9c6450f93f06510477312bd02b119fd96b70857c Reviewed-on: https://chromium-review.googlesource.com/245560 Reviewed-by: Fang Deng <fdeng@chromium.org> Tested-by: Fang Deng <fdeng@chromium.org> Commit-Queue: Fang Deng <fdeng@chromium.org>
/external/autotest/frontend/afe/models_test.py
|
8c98ac10beaa08bfb975c412b0b3bda23178763a |
|
23-Dec-2014 |
Prashanth Balasubramanian <beeps@google.com> |
[autotest] Send frontend jobs to shards. Frontend jobs on hosts that are on the shard are disallowed currently, because the host-scheduler on master currently ignore jobs based on meta-host, but frontend jobs have no meta-host. This CL have the following changes: - Make host-scheduler ignore frontend jobs that are supposed to be picked by shard. - Send such frontend jobs in heartbeat. - Allow creation of frontend jobs in rpc. TEST=Test the follows: - Create a job on a host on shard from AFE frontend. Observe it runs on shards and completes on master. - Create a job on two hosts (one host on shard, the other on master) from AFE frontend. Make sure exception is railed with correct message. - Run a normal dummy suite on shard, make sure normal flow still works. Heartbeat contains the right information. - Run a normal dummy suite on master, make sure it works. BUG=chromium:444790 DEPLOY=apache, host-scheduler Change-Id: Ibca3d36cb59fed695233ffdc89506364c402cc37 Reviewed-on: https://chromium-review.googlesource.com/240396 Reviewed-by: Mungyung Ryu <mkryu@google.com> Reviewed-by: Dan Shi <dshi@chromium.org> Commit-Queue: Fang Deng <fdeng@chromium.org> Tested-by: Fang Deng <fdeng@chromium.org>
/external/autotest/frontend/afe/models_test.py
|
f865d33cc3b0584f10b2f6f4de414a6d7f9e3000 |
|
29-Sep-2014 |
Jakob Juelich <jakobjuelich@chromium.org> |
[autotest] Allow updating selected model attributes Right now, objects that have been persisted on the client once, will never be overwritten again with new data, that comes from the master. In order to sync i.e. aborted bits from the master to the slave, this needs to change. This introduces a new variable SERIALIZATION_LOCAL_LINKS_TO_UPDATE, which objects can overwrite with a set of names of attributes, that should be overwritten with data that's sent from the master. BUG=None DEPLOY=shard_client TEST=Ran suites. Change-Id: I6b461475116db0acd2de7b8fe628aed791e7c418 Reviewed-on: https://chromium-review.googlesource.com/220331 Tested-by: Jakob Jülich <jakobjuelich@chromium.org> Reviewed-by: Fang Deng <fdeng@chromium.org> Commit-Queue: Jakob Jülich <jakobjuelich@chromium.org>
/external/autotest/frontend/afe/models_test.py
|
116ff0fc2674082fe475afd64ce4dec998ed71b7 |
|
18-Sep-2014 |
Jakob Juelich <jakobjuelich@chromium.org> |
[autotest] Deserializer: Allow updating objects Currently the state of an object is never changed by the deserializer. After a shard executed a job, the master should be informed about it's result so one can see if a job has been executed or not in one central place for all jobs. To update a job's status on the master, this is necessary. Doing this, a bug of the previous implementation became apparent: Deserializing a host object while no AclGroups were present yet would create an Everyone-AclGroup in it's save() method. This is dangerous because this way the states of master and shards might diverge, specifically AclGroups with same names and different ids might occur, which will then, upon dumping the Everyone-AclGroup from the master, will cause exceptions to be raised. Therefore just not running them prevents errors. BUG=None DEPLOY=scheduler,apache,host_scheduler TEST=Ran suites, tested with real data on shard Change-Id: Ie8ccd3205ac0efd5bbba4402da5de4c2615b495c Reviewed-on: https://chromium-review.googlesource.com/218806 Reviewed-by: Prashanth B <beeps@chromium.org> Commit-Queue: Jakob Jülich <jakobjuelich@chromium.org> Tested-by: Jakob Jülich <jakobjuelich@chromium.org>
/external/autotest/frontend/afe/models_test.py
|
f88fa938050d0e2b662a2b32ef7e931e01a4d8fe |
|
04-Sep-2014 |
Jakob Juelich <jakobjuelich@chromium.org> |
[autotest] Deserialize records and persist them To send records from the master to shards it's necessary to serialize them. This adds the deserialization functionality. TEST=Ran suites. DEPLOY=apache Change-Id: I2806b0cfe1e4fbfba5d89e6d422800c7637ed4e9 Reviewed-on: https://chromium-review.googlesource.com/216355 Reviewed-by: Prashanth B <beeps@chromium.org> Tested-by: Jakob Jülich <jakobjuelich@chromium.org> Commit-Queue: Jakob Jülich <jakobjuelich@chromium.org>
/external/autotest/frontend/afe/models_test.py
|
3bb7c8095d7c66d955462673dd3df5565c1b4a96 |
|
03-Sep-2014 |
Jakob Juelich <jakobjuelich@chromium.org> |
[autotest] Add serialization functionality including dependencies In order to send hosts and jobs to shards it's necessary to serialize them including their dependencies. This changelist adds functionality to find an object's dependencies and to serialize it into lists and dicts, so it can be converted to json. TEST=Ran suites. DEPLOY=apache Change-Id: Icc158de1f23c7483ebc22b6ca3482b7ffccdd5c4 Reviewed-on: https://chromium-review.googlesource.com/215998 Tested-by: Jakob Jülich <jakobjuelich@chromium.org> Reviewed-by: Alex Miller <milleral@chromium.org> Commit-Queue: Jakob Jülich <jakobjuelich@chromium.org>
/external/autotest/frontend/afe/models_test.py
|
3dd8beb386f7298ffe84d7410d00cce26973e170 |
|
14-May-2013 |
Aviv Keshet <akeshet@chromium.org> |
[autotest] make a consistent CONTROL_TYPE enum across the codebase Prior to this CL, there were a multitude of duplicate defitions of the two control files types (Client or Server), incluiding a difference between the afe Job model (1 = Server) and the afe Test model (1 = Client). This CL introduces enums CONTROL_TYPE and CONTROL_TYPE_NAMES to control_data, to act as the central and consistent defition across the codebase. In order to avoid needing to mangle the running Jobs table, we have adopted the existing Job model convention (1 = Server); the Test table will be re-written with the new consistent convention during test import. BUG=chromium:240643 TEST=All existing unit tests pass; In local autotest without this patch applied, started a suite. Halfway through suite, applied this patch, ran test importer, restarted apache, and restarted scheduler. Suite finished successfully. Verified manually that Client/Server type control files show up correctly in afe Create Job view. DEPLOY=scheduler DEPLOY=apache DEPLOY=test_importer Change-Id: Ia5b2573e1d08d96b3826f2837903ef407dcae303 Reviewed-on: https://gerrit.chromium.org/gerrit/51191 Commit-Queue: Aviv Keshet <akeshet@chromium.org> Reviewed-by: Aviv Keshet <akeshet@chromium.org> Tested-by: Aviv Keshet <akeshet@chromium.org>
/external/autotest/frontend/afe/models_test.py
|
4a41e0145c2ede3605d19cdfacb779ca6aca1ccc |
|
17-Jul-2010 |
jamesren <jamesren@592f7852-d20e-0410-864c-8624ca9c26a4> |
Begin adding support for parameterized jobs. This will allow test developers to specify certain parameters that a test control file may take, so that users can then easily set those parameters on job create. Enabling this feature removes the ability to edit the control file directly on job creation. Feature is currently INCOMPLETE. Do not attempt to use. This feature will be committed in small pieces for the sake of having smaller code reviews. Signed-off-by: James Ren <jamesren@google.com> git-svn-id: http://test.kernel.org/svn/autotest/trunk@4720 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models_test.py
|
76fcf19ec42d5c7580d2e7891e4610e5fe725286 |
|
21-Apr-2010 |
jamesren <jamesren@592f7852-d20e-0410-864c-8624ca9c26a4> |
Add ability to associate drone sets with jobs. This restricts a job to running on a specified set of drones. Signed-off-by: James Ren <jamesren@google.com> git-svn-id: http://test.kernel.org/svn/autotest/trunk@4439 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models_test.py
|
e36562323bec14d14b7c583db94788a725514ff2 |
|
02-Mar-2010 |
jamesren <jamesren@592f7852-d20e-0410-864c-8624ca9c26a4> |
Fix Django's QuerySet.delete() behavior for models that use "invalid" Signed-off-by: James Ren <jamesren@google.com> git-svn-id: http://test.kernel.org/svn/autotest/trunk@4276 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models_test.py
|
fd8b89f0117366c9aeaad9b600a43238a84b4ab9 |
|
20-Jan-2010 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
don't set the current user to my_user in frontend_test_utils. let it default to the new autotest_system user. Signed-off-by: Steve Howard <showard@google.com> git-svn-id: http://test.kernel.org/svn/autotest/trunk@4156 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models_test.py
|
d11956572cb7a5c8e9c588c9a6b4a0892de00384 |
|
08-Dec-2009 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Make drone_manager track running processes counts using only the information passed in from the scheduler. Currently it also uses process counts derived from "ps", but that is an unreliable source of information. This improves accuracy and consistency and gives us full control over the process. This involves a few primary changes: * made the drone_manager track process counts with each PidfileId * added method declare_process_count() for the scheduler to indicate the process count of a pidfile ID during recovery (in other cases, the DroneManager gets that info in execute_process()) Doing this involved some extensive refactorings. Because the scheduler now needs to declare process counts during recovery, and because the AgentTasks are the entities that know about process counts, it made sense to move the bulk of the recovery process to the AgentTasks. Changes for this include: * converted a bunch of AgentTask instance variables to abstract methods, and added overriding implementations in subclasses as necessary * added methods register_necessary_pidfiles() and recover() to AgentTasks, allowing them to perform recovery for themselves. got rid of the recover_run_monitor() argument to AgentTasks as a result. * changed recovery code to delegate most of the work to the AgentTasks. The flow now looks like this: create all AgentTasks, call them to register pidfiles, call DroneManager to refresh pidfile contents, call AgentTasks to recover themselves, perform extra cleanup and error checking. This simplified the Dispatcher somewhat, in my opinion, though there's room for more simplification. Other changes include: * removed DroneManager.get_process_for(), which was unused, as well as related code (include the DroneManager._processes structure) * moved logic from HostQueueEntry.handle_host_failure to SpecialAgentTask._fail_queue_entry. That was the only call site. And some other bug fixes: * eliminated some extra state from QueueTask * fixed models.HostQueueEntry.execution_path(). It was returning the wrong value, but it was never used. * eliminated some big chunks from monitor_db_unittest. These broke from the refactorings described above and I deemed it not worthwhile to fix them up for the new code. I checked and the total coverage was unaffected by deleting these chunks. Signed-off-by: Steve Howard <showard@google.com> git-svn-id: http://test.kernel.org/svn/autotest/trunk@4007 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models_test.py
|
9bb960b90d5102cce1c8a15314900035c6c4e69a |
|
19-Nov-2009 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Support restricting access to drones by user. Administrators can put lines like <hostname>_users: showard,scottz in the global config, where <hostname> is a drone hostname. That drone will then be limited to use by those users (that is, by jobs launched by those users, and tasks launched due to those jobs). This required numerous changes: * added a requested_by field to SpecialTask (with corresponding migration). For tasks with queue_entries, we can infer this from the job, but for those without, we need this information explicitly declared. Note this can be null if the task was created by the system, not in response to any user action. The only place this occurs now is in scheduler recovery (Dispatcher._recover_hosts_where()), but there may be an upcoming feature to periodically reverify hosts, which would be another (much more common) case. * modified all SpecialTask creation sites to pass requested_by if necessary. * modified AgentTask to keep a username attribute, and modified its run() method to pass that to PidfileRunMonitor.run(), which passes it along to DroneManager.execute_command(). * modified Agent to always keep self.task around, there's no reason to throw it away and now that we're looking at it from other classes, it's problematic if it disappears. * modified Dispatcher throttling code to pass the username when requesting max runnable processes. * added an allowed_users property to _AbstractDrone, and made DroneManager load it from the global config. * made DroneManager's max_runnable_processes() and _choose_drone_for_execution() methods accept the username and obey user restrictions. * added extensive tests for everything. the modiications required to monitor_db_unittest were annoying but not too bad. but parts of that file may need to be removed as they'll be obsoleted by monitor_db_functional_test and they'll become increasingly annoying to maintain. couple other related changes: * got rid of CleanupHostsMixin. it was only acutally needed by GatherLogsTasks (since we made the change to have GatherLogsTask always run), so I inlined it there and simplified code accordingly. * changed a bunch of places in the scheduler that were constructing new instances of Django models for existing rows. they would do something like "models.Host(id=<id of existing host>)". that's correct for scheduler DBModels, but not for Django models. For Django models, you only instantiate new instances when you want to create a new row. for fetching existing rows you always use a manager -- Model.objects.get() or Model.objects.filter() etc. this was an existing bug but wasn't exposed until I made some of the changes involved in this feature. Signed-off-by: Steve Howard <showard@google.com> git-svn-id: http://test.kernel.org/svn/autotest/trunk@3961 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models_test.py
|
e60e44ece1445d97977a77cb79f0896989b869d7 |
|
13-Nov-2009 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Special tasks show "Failed" as their status instead of "Completed" if they failed Signed-off-by: James Ren <jamesren@google.com> git-svn-id: http://test.kernel.org/svn/autotest/trunk@3946 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models_test.py
|
afd97de523a8c8c4e6657fa2db6214fda68d8086 |
|
01-Oct-2009 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
When a one-time host is added as a normal host while it's running a job, the status would get reset, messing up the scheduler. Fix that. Signed-off-by: Steve Howard <showard@google.com> git-svn-id: http://test.kernel.org/svn/autotest/trunk@3778 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models_test.py
|
8cc058f50a46976e0a446aa3054f7f2349d6291a |
|
08-Sep-2009 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Make scheduler more stateless. Agents are now scheduled only by the dispatcher, and agents no longer use in-memory state to remember multiple tasks. All state is managed by the database. Risk: high (large scheduler change) Visibility: medium (scheduler restarts are now more stable) Signed-off-by: James Ren <jamesren@google.com> git-svn-id: http://test.kernel.org/svn/autotest/trunk@3664 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models_test.py
|
6631273af8b88842cbd6202cc4615daf050cc957 |
|
27-Aug-2009 |
mbligh <mbligh@592f7852-d20e-0410-864c-8624ca9c26a4> |
Make a bunch of stuff executable git-svn-id: http://test.kernel.org/svn/autotest/trunk@3621 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models_test.py
|
474d136e9a71f288f294167112dda909f9273434 |
|
21-Aug-2009 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Sanity check: prevent the creation of a special task for a host that already has an active special task. This should only ever happen due to a scheduler bug. Dying to allow human cleanup is desirable today. Signed-off-by: Gregory Smith <gps@google.com> git-svn-id: http://test.kernel.org/svn/autotest/trunk@3571 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models_test.py
|
a5288b4bb2b09aafe914d0b7d5aab79a7e433eaf |
|
28-Jul-2009 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Upgrade from Django 0.96 to Django 1.0.2. Risk: high (framework change) Visibility: medium Signed-off-by: James Ren <jamesren@google.com> git-svn-id: http://test.kernel.org/svn/autotest/trunk@3457 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models_test.py
|
97446887819594f1e2a329dcff289ee8e934b626 |
|
21-Jul-2009 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
Log changes to SpecialTask active/complete state. Signed-off-by: Steve Howard <showard@google.com> git-svn-id: http://test.kernel.org/svn/autotest/trunk@3426 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models_test.py
|
c0ac3a79cceec87873257a8b6d41d19c9fb02ec6 |
|
08-Jul-2009 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
add the option to include verify/repair/cleanups in the job table on the host detail page. * added RPC get_host_queue_entries_and_special_tasks, which returns both HQEs and SpecialTasks for a host in a carefully interleaved manner. * added appropriate frontend unit tests * added support to HostDetailView to include these entries, but have them not be selectable. this required changes to SelectionManager. I also added a utility method to Utils for opening a new window, and changed all sites that do that to call the new method. Signed-off-by: Steve Howard <showard@google.com> git-svn-id: http://test.kernel.org/svn/autotest/trunk@3385 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models_test.py
|
ed2afea4ca6e23a82d20d1f2ee1067d0c25a8cc2 |
|
07-Jul-2009 |
showard <showard@592f7852-d20e-0410-864c-8624ca9c26a4> |
make SpecialTasks recoverable. this involves quite a few changes. * run tasks in determined dirs instead of temp dirs. the dir paths look like hosts//-, for example, hosts/myhost/4-verify. the ID comes from the SpecialTask DB row. this allows us to find the pidfile when we go looking for it during recovery, and it makes it simple to find the logs for any given special task, much like for HostQueueEntries. added SpecialTask.execution_path() for this purpose, and added models_test to test it. * added execution_path() to HostQueueEntry to match the interface of SpecialTask, allowing for more polymorphism, and changed most call sites to use it. * since we're running in these dirs, copy the full results back in these dirs, instead of just copying a single log file. * move process recovery code up into AgentTask, so that all AgentTasks can share the same generic process recovery code. * change SpecialTask recovery code to do process recovery. * change VerifyTask handling of multiple pending verify requests for a machine. instead of updating all the requests, just delete all other tasks. they're not specially tracked in any way so it's simplest to just delete them. * made special tasks get marked is_active=False when they complete, to be consistent with HQEs other changes: * added null=True to SpecialTask.time_started definition * made EmailManager.enqueue_notify_email always log the message, and removed explicit logging calls from call sites * added feature to DroneManager.execute_command() to automatically substitute the working directory into the command. this avoids some duplicate information being passed around and simplifies the unit test. Signed-off-by: Steve Howard <showard@google.com> git-svn-id: http://test.kernel.org/svn/autotest/trunk@3380 592f7852-d20e-0410-864c-8624ca9c26a4
/external/autotest/frontend/afe/models_test.py
|