agent_task.py revision ec21225c11b0983b563c24bb71828f5bb6bbcaee
13b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich#pylint: disable-msg=C0111 23b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich 33b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich""" This is the module for everything related to the AgentTask. 43b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich 53b27dbc2358aef655e050a92510ff8e9e080bf81Jakob JuelichThe BaseAgentTask imposes an interface through which the scheduler can monitor 63b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelicha processes; Examples of such processes include Verify, Cleanup and the Queue 73b27dbc2358aef655e050a92510ff8e9e080bf81Jakob JuelichTasks that run the tests. The scheduler itself only understands Agents. 83b27dbc2358aef655e050a92510ff8e9e080bf81Jakob JuelichAgents: 93b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich The Agent is the bridge between the scheduler and the AgentTask. The 103b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich schedulers tick has a method called handle_agents, which calls the 113b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich tick of each agent in the Dispatchers queue. This leads to the Agent 12f960d89a7b197fe3b3bd28546c6c89c2331b9f14Jakob Juelich polling its AgentTask. The scheduler will keep polling a task through 133b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich the associated Agent till the Agent is removed from the dispatcher. 143b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich 153b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich At a high level: 163b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich agents finished = tasks done 173b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich agent polls till finished 183b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich task polls till done 191e1c41b1b4a1b97c0b7086b8430856ed45e064d3Gabe Black task sets done 2022dd226625255110c079e979113dcda1f4fa5ea8Prashanth Balasubramanian agent is removed from dispatcher 213b27dbc2358aef655e050a92510ff8e9e080bf81Jakob JuelichAgentTasks: 225949b4af7a872aeb58e7ad29090812d648725ed5Prashanth Balasubramanian Basic AgentTasks are created when an hqe changes state. Examples of these 238421d5905ab0aed8689c2eea6be8d9c4042ce618Jakob Juelich are the QueueTask, which is created when a hqe goes into the Starting state 2475be1d3f881ef4f4f9cffe0c38fc3139338d8f84Prashanth Balasubramanian and the FinalReparseTask, which is created when the hqe goes into parsing. 253b27dbc2358aef655e050a92510ff8e9e080bf81Jakob JuelichSpecialAgentTasks: 263b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich Unlink AgentTasks, SpecialAgentTasks are only created when a row is inserted 273b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich in the afe_special_tasks table. All PrejobTasks are SpecialAgentTasks. 283b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich 293b27dbc2358aef655e050a92510ff8e9e080bf81Jakob JuelichMonitor_db.get_agent_task_for_special_task/get_agent_task_for_queue_entry maps 303b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelichan AgentTask to an Agent, which the scheduler understands. From this point 313b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelichonward, the scheduler manages the task through the Agents interface,as follows: 323b27dbc2358aef655e050a92510ff8e9e080bf81Jakob JuelichAt a high level: 333b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich task poll 343b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich start 353b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich prolog 363b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich tick till we get an exit code 373b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich finished(exit==0) 383b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich done=True 393b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich epilog 403b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich cleanup 413b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich set is_active, is_complete, success (checked in scheduler) 423b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich 433b27dbc2358aef655e050a92510ff8e9e080bf81Jakob JuelichThe first special task for an HQE is usually Reset. 443b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich-poll: The first poll will start the task, polls thereafter will call the tasks 453b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich tick method. A started task will have the started bit set. 461b52574752be108a743d3b33561c34324f8538e7Jakob Juelich- start: Call prolog, run the process and set the start bit. 471b52574752be108a743d3b33561c34324f8538e7Jakob Juelich - prolog: Usually where one puts any model state changes that happen before 481b52574752be108a743d3b33561c34324f8538e7Jakob Juelich the actual task. Different per Task. Examples of things that might 491b52574752be108a743d3b33561c34324f8538e7Jakob Juelich happen in a prolog: 501b52574752be108a743d3b33561c34324f8538e7Jakob Juelich - state of Host, HQE (to something like Resetting) 511b52574752be108a743d3b33561c34324f8538e7Jakob Juelich - delete any unwanted queued special tasks 523b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich - register a pidfile 533b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich - set the is_active bit on the special task 543b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich - run: 553b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich - create a PidfileRunMonitor 561b52574752be108a743d3b33561c34324f8538e7Jakob Juelich - pass the autoserv command, working directory etc to drone manager. 571b52574752be108a743d3b33561c34324f8538e7Jakob Juelich This will start the actual autoserv process. 581b52574752be108a743d3b33561c34324f8538e7Jakob Juelich - set the start bit: so subsequent polls do not 'start' again 593b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich 603b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich- tick: For as long as a started tasks done bit is not set, a poll will lead 613b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich to a tick. The tick monitors the pid file of the autoserv process 623b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich running on the drone through the PidfileRunMonitor created in prolog. 633b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich If the autoserv process has finished we call finished with true/false 643b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich depending on autoserv exit code. 653b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich 663b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich - finished: sets the done and success values, then calls epilog. The 673b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich done bit is important because the Agent polls this bit to 683b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich measure the success or failure of its task. 693b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich 703b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich - epilog: Is generally where we set status of the Host/HQE again, 713b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich requeue any other task that needs to run after this one 723b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich and perform cleanup. Just like the prolog, this step is 733b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich different per task. 743b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich 753b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich - cleanup: Sets the is_active and is_complete and success 761b52574752be108a743d3b33561c34324f8538e7Jakob Juelich states on the tasks model. Also uses the 771b52574752be108a743d3b33561c34324f8538e7Jakob Juelich drone_manager to: 781b52574752be108a743d3b33561c34324f8538e7Jakob Juelich unregister the pidfile 791b52574752be108a743d3b33561c34324f8538e7Jakob Juelich copy results of the task 803b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich (Note this is not to be confused with the 813b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich special task called cleanup). 823b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich 833b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich The actions we take in the epilog are based on the 84f960d89a7b197fe3b3bd28546c6c89c2331b9f14Jakob Juelich success/failure of the autoserv process set in cleanup, 858421d5905ab0aed8689c2eea6be8d9c4042ce618Jakob Juelich eg: if reset failed we will enqueue a repair, but if all 868421d5905ab0aed8689c2eea6be8d9c4042ce618Jakob Juelich is well the epilog will just return. Prejob task epilogs 878421d5905ab0aed8689c2eea6be8d9c4042ce618Jakob Juelich also have an on_pending method that change the status of 888421d5905ab0aed8689c2eea6be8d9c4042ce618Jakob Juelich the HQE to pending/starting, which gets picked up in the 891e1c41b1b4a1b97c0b7086b8430856ed45e064d3Gabe Black scheduler. 9022dd226625255110c079e979113dcda1f4fa5ea8Prashanth BalasubramanianBy this point the is_done flag is set, which results in the Agent noticing that 913b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelichthe task has finished and unregistering it from the dispatcher.Class hierarchy: 923b27dbc2358aef655e050a92510ff8e9e080bf81Jakob JuelichBaseAgentTask 933b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich |--->SpecialAgentTask (prejob_task.py) 943b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich |--->RepairTask 953b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich |--->PreJobTask 968421d5905ab0aed8689c2eea6be8d9c4042ce618Jakob Juelich |--->Verify, Cleanup, Reset, Provision 973b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich 983b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich |--->AbstractQueueTask (monitor_db.py) 993b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich |--->QueueTask 1003b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich |--->HostlessQueueTask 1018421d5905ab0aed8689c2eea6be8d9c4042ce618Jakob Juelich 1028421d5905ab0aed8689c2eea6be8d9c4042ce618Jakob Juelich |--->PostJobTask (postjob_task.py) 1038421d5905ab0aed8689c2eea6be8d9c4042ce618Jakob Juelich |--->GatherLogsTask 1043b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich |--->SelfThrottledPostJobTask 1053b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich |--->FinalReparseTask 1063b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich |--->ArchiveResultsTask 1078421d5905ab0aed8689c2eea6be8d9c4042ce618Jakob Juelich 1083b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich""" 1093b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich 110f960d89a7b197fe3b3bd28546c6c89c2331b9f14Jakob Juelichimport logging 1113b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelichimport os 1123b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelichimport urllib 1133b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelichimport time 1143b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich 1153b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelichfrom autotest_lib.frontend.afe import models 1163b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelichfrom autotest_lib.scheduler import drone_manager, pidfile_monitor 1173b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelichfrom autotest_lib.client.common_lib import utils 1183b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelichfrom autotest_lib.scheduler import email_manager, host_scheduler 1193b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelichfrom autotest_lib.scheduler import rdb_lib 1203b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelichfrom autotest_lib.scheduler import scheduler_models 1213b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelichfrom autotest_lib.server import autoserv_utils 1223b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich 1233b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich 1241e1c41b1b4a1b97c0b7086b8430856ed45e064d3Gabe Black_drone_manager = drone_manager.instance() 125f960d89a7b197fe3b3bd28546c6c89c2331b9f14Jakob Juelich 1261e1c41b1b4a1b97c0b7086b8430856ed45e064d3Gabe BlackAUTOSERV_NICE_LEVEL = 10 127f960d89a7b197fe3b3bd28546c6c89c2331b9f14Jakob Juelich 128f960d89a7b197fe3b3bd28546c6c89c2331b9f14Jakob Juelich 1293b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelichclass BaseAgentTask(object): 13075be1d3f881ef4f4f9cffe0c38fc3139338d8f84Prashanth Balasubramanian class _NullMonitor(object): 13175be1d3f881ef4f4f9cffe0c38fc3139338d8f84Prashanth Balasubramanian pidfile_id = None 1323b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich 13375be1d3f881ef4f4f9cffe0c38fc3139338d8f84Prashanth Balasubramanian def has_process(self): 13475be1d3f881ef4f4f9cffe0c38fc3139338d8f84Prashanth Balasubramanian return True 1353b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich 13622dd226625255110c079e979113dcda1f4fa5ea8Prashanth Balasubramanian 13722dd226625255110c079e979113dcda1f4fa5ea8Prashanth Balasubramanian def __init__(self, log_file_name=None): 13822dd226625255110c079e979113dcda1f4fa5ea8Prashanth Balasubramanian """ 13922dd226625255110c079e979113dcda1f4fa5ea8Prashanth Balasubramanian @param log_file_name: (optional) name of file to log command output to 14022dd226625255110c079e979113dcda1f4fa5ea8Prashanth Balasubramanian """ 14122dd226625255110c079e979113dcda1f4fa5ea8Prashanth Balasubramanian self.done = False 14222dd226625255110c079e979113dcda1f4fa5ea8Prashanth Balasubramanian self.started = False 14322dd226625255110c079e979113dcda1f4fa5ea8Prashanth Balasubramanian self.success = None 14422dd226625255110c079e979113dcda1f4fa5ea8Prashanth Balasubramanian self.aborted = False 14522dd226625255110c079e979113dcda1f4fa5ea8Prashanth Balasubramanian self.monitor = None 14622dd226625255110c079e979113dcda1f4fa5ea8Prashanth Balasubramanian self.queue_entry_ids = [] 14722dd226625255110c079e979113dcda1f4fa5ea8Prashanth Balasubramanian self.host_ids = [] 1483b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich self._log_file_name = log_file_name 1498421d5905ab0aed8689c2eea6be8d9c4042ce618Jakob Juelich 1508421d5905ab0aed8689c2eea6be8d9c4042ce618Jakob Juelich 1518421d5905ab0aed8689c2eea6be8d9c4042ce618Jakob Juelich def _set_ids(self, host=None, queue_entries=None): 1528421d5905ab0aed8689c2eea6be8d9c4042ce618Jakob Juelich if queue_entries and queue_entries != [None]: 1538421d5905ab0aed8689c2eea6be8d9c4042ce618Jakob Juelich self.host_ids = [entry.host.id for entry in queue_entries] 1548421d5905ab0aed8689c2eea6be8d9c4042ce618Jakob Juelich self.queue_entry_ids = [entry.id for entry in queue_entries] 1558421d5905ab0aed8689c2eea6be8d9c4042ce618Jakob Juelich else: 1568421d5905ab0aed8689c2eea6be8d9c4042ce618Jakob Juelich assert host 1578421d5905ab0aed8689c2eea6be8d9c4042ce618Jakob Juelich self.host_ids = [host.id] 1588421d5905ab0aed8689c2eea6be8d9c4042ce618Jakob Juelich 1598421d5905ab0aed8689c2eea6be8d9c4042ce618Jakob Juelich 1608421d5905ab0aed8689c2eea6be8d9c4042ce618Jakob Juelich def poll(self): 1618421d5905ab0aed8689c2eea6be8d9c4042ce618Jakob Juelich if not self.started: 1628421d5905ab0aed8689c2eea6be8d9c4042ce618Jakob Juelich self.start() 1638421d5905ab0aed8689c2eea6be8d9c4042ce618Jakob Juelich if not self.done: 1648421d5905ab0aed8689c2eea6be8d9c4042ce618Jakob Juelich self.tick() 1658421d5905ab0aed8689c2eea6be8d9c4042ce618Jakob Juelich 1668421d5905ab0aed8689c2eea6be8d9c4042ce618Jakob Juelich 1678421d5905ab0aed8689c2eea6be8d9c4042ce618Jakob Juelich def tick(self): 1688421d5905ab0aed8689c2eea6be8d9c4042ce618Jakob Juelich assert self.monitor 1698421d5905ab0aed8689c2eea6be8d9c4042ce618Jakob Juelich exit_code = self.monitor.exit_code() 1708421d5905ab0aed8689c2eea6be8d9c4042ce618Jakob Juelich if exit_code is None: 1718421d5905ab0aed8689c2eea6be8d9c4042ce618Jakob Juelich return 17222dd226625255110c079e979113dcda1f4fa5ea8Prashanth Balasubramanian 17322dd226625255110c079e979113dcda1f4fa5ea8Prashanth Balasubramanian success = (exit_code == 0) 17422dd226625255110c079e979113dcda1f4fa5ea8Prashanth Balasubramanian self.finished(success) 1758421d5905ab0aed8689c2eea6be8d9c4042ce618Jakob Juelich 17622dd226625255110c079e979113dcda1f4fa5ea8Prashanth Balasubramanian 17722dd226625255110c079e979113dcda1f4fa5ea8Prashanth Balasubramanian def is_done(self): 1788421d5905ab0aed8689c2eea6be8d9c4042ce618Jakob Juelich return self.done 1798421d5905ab0aed8689c2eea6be8d9c4042ce618Jakob Juelich 1808421d5905ab0aed8689c2eea6be8d9c4042ce618Jakob Juelich 1818421d5905ab0aed8689c2eea6be8d9c4042ce618Jakob Juelich def finished(self, success): 1828421d5905ab0aed8689c2eea6be8d9c4042ce618Jakob Juelich if self.done: 1838421d5905ab0aed8689c2eea6be8d9c4042ce618Jakob Juelich assert self.started 18422dd226625255110c079e979113dcda1f4fa5ea8Prashanth Balasubramanian return 1858421d5905ab0aed8689c2eea6be8d9c4042ce618Jakob Juelich self.started = True 1868421d5905ab0aed8689c2eea6be8d9c4042ce618Jakob Juelich self.done = True 1878421d5905ab0aed8689c2eea6be8d9c4042ce618Jakob Juelich self.success = success 1888421d5905ab0aed8689c2eea6be8d9c4042ce618Jakob Juelich self.epilog() 1898421d5905ab0aed8689c2eea6be8d9c4042ce618Jakob Juelich 1908421d5905ab0aed8689c2eea6be8d9c4042ce618Jakob Juelich 1918421d5905ab0aed8689c2eea6be8d9c4042ce618Jakob Juelich def prolog(self): 1928421d5905ab0aed8689c2eea6be8d9c4042ce618Jakob Juelich """ 1938421d5905ab0aed8689c2eea6be8d9c4042ce618Jakob Juelich To be overridden. 1948421d5905ab0aed8689c2eea6be8d9c4042ce618Jakob Juelich """ 1958421d5905ab0aed8689c2eea6be8d9c4042ce618Jakob Juelich assert not self.monitor 1968421d5905ab0aed8689c2eea6be8d9c4042ce618Jakob Juelich self.register_necessary_pidfiles() 1978421d5905ab0aed8689c2eea6be8d9c4042ce618Jakob Juelich 1981b52574752be108a743d3b33561c34324f8538e7Jakob Juelich 1991b52574752be108a743d3b33561c34324f8538e7Jakob Juelich def _log_file(self): 2001b52574752be108a743d3b33561c34324f8538e7Jakob Juelich if not self._log_file_name: 2011b52574752be108a743d3b33561c34324f8538e7Jakob Juelich return None 2021b52574752be108a743d3b33561c34324f8538e7Jakob Juelich return os.path.join(self._working_directory(), self._log_file_name) 2031b52574752be108a743d3b33561c34324f8538e7Jakob Juelich 2041b52574752be108a743d3b33561c34324f8538e7Jakob Juelich 2051b52574752be108a743d3b33561c34324f8538e7Jakob Juelich def cleanup(self): 2061b52574752be108a743d3b33561c34324f8538e7Jakob Juelich log_file = self._log_file() 2071b52574752be108a743d3b33561c34324f8538e7Jakob Juelich if self.monitor and log_file: 2081b52574752be108a743d3b33561c34324f8538e7Jakob Juelich self.monitor.try_copy_to_results_repository(log_file) 2091b52574752be108a743d3b33561c34324f8538e7Jakob Juelich 2101b52574752be108a743d3b33561c34324f8538e7Jakob Juelich 2111b52574752be108a743d3b33561c34324f8538e7Jakob Juelich def epilog(self): 2121b52574752be108a743d3b33561c34324f8538e7Jakob Juelich """ 2135949b4af7a872aeb58e7ad29090812d648725ed5Prashanth Balasubramanian To be overridden. 2145949b4af7a872aeb58e7ad29090812d648725ed5Prashanth Balasubramanian """ 2151b52574752be108a743d3b33561c34324f8538e7Jakob Juelich self.cleanup() 2161b52574752be108a743d3b33561c34324f8538e7Jakob Juelich logging.info("%s finished with success=%s", type(self).__name__, 2171b52574752be108a743d3b33561c34324f8538e7Jakob Juelich self.success) 21822dd226625255110c079e979113dcda1f4fa5ea8Prashanth Balasubramanian 21922dd226625255110c079e979113dcda1f4fa5ea8Prashanth Balasubramanian 22022dd226625255110c079e979113dcda1f4fa5ea8Prashanth Balasubramanian def start(self): 22122dd226625255110c079e979113dcda1f4fa5ea8Prashanth Balasubramanian if not self.started: 22222dd226625255110c079e979113dcda1f4fa5ea8Prashanth Balasubramanian self.prolog() 22322dd226625255110c079e979113dcda1f4fa5ea8Prashanth Balasubramanian self.run() 22422dd226625255110c079e979113dcda1f4fa5ea8Prashanth Balasubramanian 22522dd226625255110c079e979113dcda1f4fa5ea8Prashanth Balasubramanian self.started = True 22622dd226625255110c079e979113dcda1f4fa5ea8Prashanth Balasubramanian 22722dd226625255110c079e979113dcda1f4fa5ea8Prashanth Balasubramanian 22822dd226625255110c079e979113dcda1f4fa5ea8Prashanth Balasubramanian def abort(self): 22922dd226625255110c079e979113dcda1f4fa5ea8Prashanth Balasubramanian if self.monitor: 23022dd226625255110c079e979113dcda1f4fa5ea8Prashanth Balasubramanian self.monitor.kill() 23122dd226625255110c079e979113dcda1f4fa5ea8Prashanth Balasubramanian self.done = True 23222dd226625255110c079e979113dcda1f4fa5ea8Prashanth Balasubramanian self.aborted = True 23322dd226625255110c079e979113dcda1f4fa5ea8Prashanth Balasubramanian self.cleanup() 23422dd226625255110c079e979113dcda1f4fa5ea8Prashanth Balasubramanian 23522dd226625255110c079e979113dcda1f4fa5ea8Prashanth Balasubramanian 23622dd226625255110c079e979113dcda1f4fa5ea8Prashanth Balasubramanian def _get_consistent_execution_path(self, execution_entries): 23722dd226625255110c079e979113dcda1f4fa5ea8Prashanth Balasubramanian first_execution_path = execution_entries[0].execution_path() 23822dd226625255110c079e979113dcda1f4fa5ea8Prashanth Balasubramanian for execution_entry in execution_entries[1:]: 23922dd226625255110c079e979113dcda1f4fa5ea8Prashanth Balasubramanian assert execution_entry.execution_path() == first_execution_path, ( 240f960d89a7b197fe3b3bd28546c6c89c2331b9f14Jakob Juelich '%s (%s) != %s (%s)' % (execution_entry.execution_path(), 2413b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich execution_entry, 2423b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich first_execution_path, 2433b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich execution_entries[0])) 2443b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich return first_execution_path 2453b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich 2463b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich 2473b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich def _copy_results(self, execution_entries, use_monitor=None): 2483b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich """ 2498421d5905ab0aed8689c2eea6be8d9c4042ce618Jakob Juelich @param execution_entries: list of objects with execution_path() method 25022dd226625255110c079e979113dcda1f4fa5ea8Prashanth Balasubramanian """ 2511e1c41b1b4a1b97c0b7086b8430856ed45e064d3Gabe Black if use_monitor is not None and not use_monitor.has_process(): 25222dd226625255110c079e979113dcda1f4fa5ea8Prashanth Balasubramanian return 25322dd226625255110c079e979113dcda1f4fa5ea8Prashanth Balasubramanian 2541e1c41b1b4a1b97c0b7086b8430856ed45e064d3Gabe Black assert len(execution_entries) > 0 25522dd226625255110c079e979113dcda1f4fa5ea8Prashanth Balasubramanian if use_monitor is None: 2568421d5905ab0aed8689c2eea6be8d9c4042ce618Jakob Juelich assert self.monitor 25722dd226625255110c079e979113dcda1f4fa5ea8Prashanth Balasubramanian use_monitor = self.monitor 2583b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich assert use_monitor.has_process() 2593b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich execution_path = self._get_consistent_execution_path(execution_entries) 2603b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich results_path = execution_path + '/' 2613b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich use_monitor.try_copy_to_results_repository(results_path) 2623b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich 2633b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich 2643b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich def _parse_results(self, queue_entries): 2653b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich for queue_entry in queue_entries: 2663b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich queue_entry.set_status(models.HostQueueEntry.Status.PARSING) 2673b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich 2683b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich 2693b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich def _archive_results(self, queue_entries): 2703b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich for queue_entry in queue_entries: 2713b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich queue_entry.set_status(models.HostQueueEntry.Status.ARCHIVING) 2723b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich 2733b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich 2743b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich def _command_line(self): 2753b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich """ 2763b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich Return the command line to run. Must be overridden. 2773b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich """ 2783b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich raise NotImplementedError 2793b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich 2803b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich 2813b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich @property 2823b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich def num_processes(self): 2833b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich """ 2843b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich Return the number of processes forked by this BaseAgentTask's process. 2853b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich It may only be approximate. To be overridden if necessary. 2863b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich """ 2873b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich return 1 2883b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich 2893b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich 2903b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich def _paired_with_monitor(self): 2918421d5905ab0aed8689c2eea6be8d9c4042ce618Jakob Juelich """ 2928421d5905ab0aed8689c2eea6be8d9c4042ce618Jakob Juelich If this BaseAgentTask's process must run on the same machine as some 2938421d5905ab0aed8689c2eea6be8d9c4042ce618Jakob Juelich previous process, this method should be overridden to return a 2948421d5905ab0aed8689c2eea6be8d9c4042ce618Jakob Juelich PidfileRunMonitor for that process. 2958421d5905ab0aed8689c2eea6be8d9c4042ce618Jakob Juelich """ 2968421d5905ab0aed8689c2eea6be8d9c4042ce618Jakob Juelich return self._NullMonitor() 2978421d5905ab0aed8689c2eea6be8d9c4042ce618Jakob Juelich 2988421d5905ab0aed8689c2eea6be8d9c4042ce618Jakob Juelich 2998421d5905ab0aed8689c2eea6be8d9c4042ce618Jakob Juelich @property 3008421d5905ab0aed8689c2eea6be8d9c4042ce618Jakob Juelich def owner_username(self): 3018421d5905ab0aed8689c2eea6be8d9c4042ce618Jakob Juelich """ 3028421d5905ab0aed8689c2eea6be8d9c4042ce618Jakob Juelich Return login of user responsible for this task. May be None. Must be 3038421d5905ab0aed8689c2eea6be8d9c4042ce618Jakob Juelich overridden. 3048421d5905ab0aed8689c2eea6be8d9c4042ce618Jakob Juelich """ 3058421d5905ab0aed8689c2eea6be8d9c4042ce618Jakob Juelich raise NotImplementedError 3063b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich 3073b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich 3083b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich def _working_directory(self): 3093b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich """ 3103b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich Return the directory where this BaseAgentTask's process executes. 3113b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich Must be overridden. 3123b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich """ 3133b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich raise NotImplementedError 3143b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich 3153b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich 3163b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich def _pidfile_name(self): 3173b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich """ 3183b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich Return the name of the pidfile this BaseAgentTask's process uses. To be 3193b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich overridden if necessary. 3203b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich """ 3213b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich return drone_manager.AUTOSERV_PID_FILE 3228421d5905ab0aed8689c2eea6be8d9c4042ce618Jakob Juelich 3233b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich 3243b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich def _check_paired_results_exist(self): 3253b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich if not self._paired_with_monitor().has_process(): 3263b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich email_manager.manager.enqueue_notify_email( 3273b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich 'No paired results in task', 3283b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich 'No paired results in task %s at %s' 3291e1c41b1b4a1b97c0b7086b8430856ed45e064d3Gabe Black % (self, self._paired_with_monitor().pidfile_id)) 3303b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich self.finished(False) 3313b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich return False 3323b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich return True 3333b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich 3343b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich 3351e1c41b1b4a1b97c0b7086b8430856ed45e064d3Gabe Black def _create_monitor(self): 3363b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich assert not self.monitor 3373b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich self.monitor = pidfile_monitor.PidfileRunMonitor() 3383b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich 3393b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich 3403b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich def run(self): 3413b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich if not self._check_paired_results_exist(): 3423b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich return 3433b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich 3443b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich self._create_monitor() 3455949b4af7a872aeb58e7ad29090812d648725ed5Prashanth Balasubramanian self.monitor.run( 3465949b4af7a872aeb58e7ad29090812d648725ed5Prashanth Balasubramanian self._command_line(), self._working_directory(), 3475949b4af7a872aeb58e7ad29090812d648725ed5Prashanth Balasubramanian num_processes=self.num_processes, 3483b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich nice_level=AUTOSERV_NICE_LEVEL, log_file=self._log_file(), 3493b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich pidfile_name=self._pidfile_name(), 3503b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich paired_with_pidfile=self._paired_with_monitor().pidfile_id, 3513b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich username=self.owner_username, 3523b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich drone_hostnames_allowed=self.get_drone_hostnames_allowed()) 3533b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich 3543b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich 3553b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich def get_drone_hostnames_allowed(self): 3563b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich if not models.DroneSet.drone_sets_enabled(): 3573b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich return None 3583b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich 3593b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich hqes = models.HostQueueEntry.objects.filter(id__in=self.queue_entry_ids) 3603b27dbc2358aef655e050a92510ff8e9e080bf81Jakob Juelich if not hqes: 361 # Only special tasks could be missing host queue entries 362 assert isinstance(self, SpecialAgentTask) 363 return self._user_or_global_default_drone_set( 364 self.task, self.task.requested_by) 365 366 job_ids = hqes.values_list('job', flat=True).distinct() 367 assert job_ids.count() == 1, ("BaseAgentTask's queue entries " 368 "span multiple jobs") 369 370 job = models.Job.objects.get(id=job_ids[0]) 371 drone_set = job.drone_set 372 if not drone_set: 373 return self._user_or_global_default_drone_set(job, job.user()) 374 375 return drone_set.get_drone_hostnames() 376 377 378 def _user_or_global_default_drone_set(self, obj_with_owner, user): 379 """ 380 Returns the user's default drone set, if present. 381 382 Otherwise, returns the global default drone set. 383 """ 384 default_hostnames = models.DroneSet.get_default().get_drone_hostnames() 385 if not user: 386 logging.warn('%s had no owner; using default drone set', 387 obj_with_owner) 388 return default_hostnames 389 if not user.drone_set: 390 logging.warn('User %s has no default drone set, using global ' 391 'default', user.login) 392 return default_hostnames 393 return user.drone_set.get_drone_hostnames() 394 395 396 def register_necessary_pidfiles(self): 397 pidfile_id = _drone_manager.get_pidfile_id_from( 398 self._working_directory(), self._pidfile_name()) 399 _drone_manager.register_pidfile(pidfile_id) 400 401 paired_pidfile_id = self._paired_with_monitor().pidfile_id 402 if paired_pidfile_id: 403 _drone_manager.register_pidfile(paired_pidfile_id) 404 405 406 def recover(self): 407 if not self._check_paired_results_exist(): 408 return 409 410 self._create_monitor() 411 self.monitor.attach_to_existing_process( 412 self._working_directory(), pidfile_name=self._pidfile_name(), 413 num_processes=self.num_processes) 414 if not self.monitor.has_process(): 415 # no process to recover; wait to be started normally 416 self.monitor = None 417 return 418 419 self.started = True 420 logging.info('Recovering process %s for %s at %s', 421 self.monitor.get_process(), type(self).__name__, 422 self._working_directory()) 423 424 425 def _check_queue_entry_statuses(self, queue_entries, allowed_hqe_statuses, 426 allowed_host_statuses=None): 427 class_name = self.__class__.__name__ 428 for entry in queue_entries: 429 if entry.status not in allowed_hqe_statuses: 430 raise host_scheduler.SchedulerError( 431 '%s attempting to start entry with invalid status %s: ' 432 '%s' % (class_name, entry.status, entry)) 433 invalid_host_status = ( 434 allowed_host_statuses is not None 435 and entry.host.status not in allowed_host_statuses) 436 if invalid_host_status: 437 raise host_scheduler.SchedulerError( 438 '%s attempting to start on queue entry with invalid ' 439 'host status %s: %s' 440 % (class_name, entry.host.status, entry)) 441 442 443SiteAgentTask = utils.import_site_class( 444 __file__, 'autotest_lib.scheduler.site_monitor_db', 445 'SiteAgentTask', BaseAgentTask) 446 447class AgentTask(SiteAgentTask): 448 pass 449 450 451class TaskWithJobKeyvals(object): 452 """AgentTask mixin providing functionality to help with job keyval files.""" 453 _KEYVAL_FILE = 'keyval' 454 def _format_keyval(self, key, value): 455 return '%s=%s' % (key, value) 456 457 458 def _keyval_path(self): 459 """Subclasses must override this""" 460 raise NotImplementedError 461 462 463 def _write_keyval_after_job(self, field, value): 464 assert self.monitor 465 if not self.monitor.has_process(): 466 return 467 _drone_manager.write_lines_to_file( 468 self._keyval_path(), [self._format_keyval(field, value)], 469 paired_with_process=self.monitor.get_process()) 470 471 472 def _job_queued_keyval(self, job): 473 return 'job_queued', int(time.mktime(job.created_on.timetuple())) 474 475 476 def _write_job_finished(self): 477 self._write_keyval_after_job("job_finished", int(time.time())) 478 479 480 def _write_keyvals_before_job_helper(self, keyval_dict, keyval_path): 481 keyval_contents = '\n'.join(self._format_keyval(key, value) 482 for key, value in keyval_dict.iteritems()) 483 # always end with a newline to allow additional keyvals to be written 484 keyval_contents += '\n' 485 _drone_manager.attach_file_to_execution(self._working_directory(), 486 keyval_contents, 487 file_path=keyval_path) 488 489 490 def _write_keyvals_before_job(self, keyval_dict): 491 self._write_keyvals_before_job_helper(keyval_dict, self._keyval_path()) 492 493 494 def _write_host_keyvals(self, host): 495 keyval_path = os.path.join(self._working_directory(), 'host_keyvals', 496 host.hostname) 497 platform, all_labels = host.platform_and_labels() 498 all_labels = [ urllib.quote(label) for label in all_labels ] 499 keyval_dict = dict(platform=platform, labels=','.join(all_labels)) 500 self._write_keyvals_before_job_helper(keyval_dict, keyval_path) 501 502 503class SpecialAgentTask(AgentTask, TaskWithJobKeyvals): 504 """ 505 Subclass for AgentTasks that correspond to a SpecialTask entry in the DB. 506 """ 507 508 TASK_TYPE = None 509 host = None 510 queue_entry = None 511 512 def __init__(self, task, extra_command_args): 513 super(SpecialAgentTask, self).__init__() 514 515 assert self.TASK_TYPE is not None, 'self.TASK_TYPE must be overridden' 516 517 self.host = rdb_lib.get_hosts([task.host.id])[0] 518 self.host.dbg_str = 'Task: %s' % str(task) 519 self.queue_entry = None 520 if task.queue_entry: 521 self.queue_entry = scheduler_models.HostQueueEntry( 522 id=task.queue_entry.id) 523 self.host.dbg_str += self.queue_entry.get_dbg_str() 524 525 self.task = task 526 self._extra_command_args = extra_command_args 527 528 529 def _keyval_path(self): 530 return os.path.join(self._working_directory(), self._KEYVAL_FILE) 531 532 533 def _command_line(self): 534 return autoserv_utils._autoserv_command_line(self.host.hostname, 535 self._extra_command_args, 536 queue_entry=self.queue_entry) 537 538 539 def _working_directory(self): 540 return self.task.execution_path() 541 542 543 @property 544 def owner_username(self): 545 if self.task.requested_by: 546 return self.task.requested_by.login 547 return None 548 549 550 def prolog(self): 551 super(SpecialAgentTask, self).prolog() 552 self.task.activate() 553 self._write_host_keyvals(self.host) 554 555 556 def _fail_queue_entry(self): 557 assert self.queue_entry 558 559 if self.queue_entry.meta_host: 560 return # don't fail metahost entries, they'll be reassigned 561 562 self.queue_entry.update_from_database() 563 if self.queue_entry.status != models.HostQueueEntry.Status.QUEUED: 564 return # entry has been aborted 565 566 self._actually_fail_queue_entry() 567 568 569 # TODO(milleral): http://crbug.com/268607 570 # All this used to be a part of _fail_queue_entry. The 571 # exact semantics of when one should and should not be failing a queue 572 # entry need to be worked out, because provisioning has placed us in a 573 # case where we want to fail a queue entry that could be requeued, 574 # which makes us fail the two above if statements, and thus 575 # _fail_queue_entry() would exit early and have no effect. 576 # What's left here with _actually_fail_queue_entry is a hack to be able to 577 # bypass the checks and unconditionally execute the code. 578 def _actually_fail_queue_entry(self): 579 self.queue_entry.set_execution_subdir() 580 queued_key, queued_time = self._job_queued_keyval( 581 self.queue_entry.job) 582 self._write_keyval_after_job(queued_key, queued_time) 583 self._write_job_finished() 584 585 # copy results logs into the normal place for job results 586 self.monitor.try_copy_results_on_drone( 587 source_path=self._working_directory() + '/', 588 destination_path=self.queue_entry.execution_path() + '/') 589 590 pidfile_id = _drone_manager.get_pidfile_id_from( 591 self.queue_entry.execution_path(), 592 pidfile_name=drone_manager.AUTOSERV_PID_FILE) 593 _drone_manager.register_pidfile(pidfile_id) 594 595 if self.queue_entry.job.parse_failed_repair: 596 self._parse_results([self.queue_entry]) 597 else: 598 self._archive_results([self.queue_entry]) 599 600 # Also fail all other special tasks that have not yet run for this HQE 601 pending_tasks = models.SpecialTask.objects.filter( 602 queue_entry__id=self.queue_entry.id, 603 is_complete=0) 604 for task in pending_tasks: 605 task.finish(False) 606 607 608 def cleanup(self): 609 super(SpecialAgentTask, self).cleanup() 610 611 # We will consider an aborted task to be "Failed" 612 self.task.finish(bool(self.success)) 613 614 if self.monitor: 615 if self.monitor.has_process(): 616 self._copy_results([self.task]) 617 if self.monitor.pidfile_id is not None: 618 _drone_manager.unregister_pidfile(self.monitor.pidfile_id) 619 620 621 def remove_special_tasks(self, special_task_to_remove, keep_last_one=False): 622 """Remove a type of special task in all tasks, keep last one if needed. 623 624 @param special_task_to_remove: type of special task to be removed, e.g., 625 models.SpecialTask.Task.VERIFY. 626 @param keep_last_one: True to keep the last special task if its type is 627 the same as of special_task_to_remove. 628 629 """ 630 queued_special_tasks = models.SpecialTask.objects.filter( 631 host__id=self.host.id, 632 task=special_task_to_remove, 633 is_active=False, is_complete=False, queue_entry=None) 634 if keep_last_one: 635 queued_special_tasks = queued_special_tasks.exclude(id=self.task.id) 636 queued_special_tasks.delete() 637 638 639 def _generate_autoserv_label_args(self, task): 640 """ 641 @param task: An instance of afe model's SpecialTask. 642 @returns: The list of arguments to pass to autoserv to tell it what the 643 labels of a job are. 644 645 """ 646 labels = {x.name for x in task.queue_entry.job.labels} 647 return ['--job-labels', ','.join(labels)] 648