| c1a81e55c53729df443180f1763423218dbb5706 |
|
02-Oct-2015 |
Dan Shi <dshi@chromium.org> |
[autotest] Remove unnecessary time.sleep.
Also increase the thread count to 16 for better performance.
BUG=None
TEST=local run
Change-Id: Ib47af433f2dd170e945f7f5b7520dfdc90140d64
Reviewed-on: https://chromium-review.googlesource.com/303536
Commit-Ready: Dan Shi <dshi@chromium.org>
Tested-by: Dan Shi <dshi@chromium.org>
Reviewed-by: Mungyung Ryu <mkryu@google.com>
/external/autotest/site_utils/host_history_utils.py
|
| 7cde53d430e208139638ee65116c0cd1ae7a7a36 |
|
27-May-2015 |
Dan Shi <dshi@chromium.org> |
[autotest] performance tuning for host history collection
1. Increase query timeout to 30s (default is 3)
2. Update query generation code to use msut, rather than should.
3. Add multithreading to host history collection.
The runtime is reduced to 25% for query like
./site_utils/host_history.py --board falco --pool suites
BUG=None
TEST=./site_utils/host_history.py --board falco --pool suites
Change-Id: I0f35c358c99aed07e60d75470f76309489235429
Reviewed-on: https://chromium-review.googlesource.com/273392
Trybot-Ready: Dan Shi <dshi@chromium.org>
Tested-by: Dan Shi <dshi@chromium.org>
Reviewed-by: Keith Haddow <haddowk@chromium.org>
Commit-Queue: Dan Shi <dshi@chromium.org>
/external/autotest/site_utils/host_history_utils.py
|
| 5d53f70f710af3d552fe1f97720471973a98cbc9 |
|
13-May-2015 |
Dan Shi <dshi@chromium.org> |
[autotest] Fix couple issues in host history from metadb
BUG=None
TEST=test against production db, unittest
./site_utils/host_history.py -l 24 --hosts chromeos4-row8-rack5-host17
Change-Id: Ib173671c4033e1a757009f5290c283eb4ea24643
Reviewed-on: https://chromium-review.googlesource.com/270790
Trybot-Ready: Dan Shi <dshi@chromium.org>
Tested-by: Dan Shi <dshi@chromium.org>
Reviewed-by: Fang Deng <fdeng@chromium.org>
Commit-Queue: Dan Shi <dshi@chromium.org>
/external/autotest/site_utils/host_history_utils.py
|
| f242c20641311794fbddd022d8dda076eb3e8ea6 |
|
20-Feb-2015 |
Dan Shi <dshi@chromium.org> |
[autotest] Fix couple issues in host history/label collection.
Issue was introduced in CL 242701. There is no unittest to catch such issue
because the data needs to be retrieved from a live ES instance. Some manual
test could have helped though.
BUG=None
TEST=edit local shadow config to target to production metadb
CROS/ES_HOST and CLIENT/metadata_index
run some commands:
./collect_host_stats.py --span 24
./host_history.py -l 24 --hosts chromeos1-row2-rack4-host2 -v
Change-Id: If5c6409c0b4479c7e6cfd8575099c2e704d37c28
Reviewed-on: https://chromium-review.googlesource.com/251570
Commit-Queue: Dan Shi <dshi@chromium.org>
Trybot-Ready: Dan Shi <dshi@chromium.org>
Tested-by: Dan Shi <dshi@chromium.org>
Reviewed-by: Gabe Black <gabeblack@chromium.org>
/external/autotest/site_utils/host_history_utils.py
|
| b72f4fbcf1583da27f09f4abb9d8162530bf4559 |
|
21-Jan-2015 |
Gabe Black <gabeblack@chromium.org> |
graphite: Reorganize the elastic search code so we can put it in chromite.
This change reorganizes the elastic search integration code so that it's
separate from the code that, for instance, reads config information from the
autotest global config. That way, it can be moved from chromite without
breaking any dependencies.
BUG=chromium:446291
TEST=Ran stats_es_functionaltest.py. Ran unit tests. Ran a butterfly-paladin
tryjob with --hwtest.
Change-Id: I0dbf135c4f1732d633e5fc9d5edb9e1f4f7199d5
Reviewed-on: https://chromium-review.googlesource.com/242701
Reviewed-by: Dan Shi <dshi@chromium.org>
Tested-by: Gabe Black <gabeblack@chromium.org>
Commit-Queue: Gabe Black <gabeblack@chromium.org>
/external/autotest/site_utils/host_history_utils.py
|
| 55bfe14a6c2cc2710593ecf4d461af64181915c0 |
|
05-Jan-2015 |
Gabe Black <gabeblack@chromium.org> |
graphite: Factor time_utils out of the es_utils module.
We're planning to move the graphite support code into chromite so it can be
used by autotest and by other infrastructure bits. To do that, we have to
remove dependencies on other bits of autotest, specifically time_utils and the
global configuration.
The time_utils module is used in essentially one spot where it converts a
function parameter into a particular format. To remove that use, the conversion
is pushed out to the callers. That's a bit clunky and cumbersome, but once the
code has been moved we can add a little wrapper function which will centralize
that again. I don't want to add the wrapper now because I'd want to put it
where the other code is currently.
BUG=chromium:446291
TEST=Ran a trybot for butterfly-paladin with --hwtest.
Change-Id: Iaf2ee2e664ff8040f4e143dfd32119c471d0bc43
Reviewed-on: https://chromium-review.googlesource.com/238526
Reviewed-by: Gabe Black <gabeblack@chromium.org>
Commit-Queue: Gabe Black <gabeblack@chromium.org>
Trybot-Ready: Gabe Black <gabeblack@chromium.org>
Tested-by: Gabe Black <gabeblack@chromium.org>
/external/autotest/site_utils/host_history_utils.py
|
| f867da40b1e52ab6ea80b93bf6e3e875d50e0d51 |
|
13-Nov-2014 |
Dan Shi <dshi@chromium.org> |
[autotest] Bug fix on how lock is counted in host history.
The logic to count lock status in host history was not correct. This CL changes
the behavior to only apply lock to Ready status, since statuses other than
`Repair Failed` and Repairing indicate a dut is working on something, and
a dut with status of `Repair Failed` and Repairing is not 'available' any way.
BUG=None
TEST=unittest, run against a dut in lab
./host_history.py --hosts chromeos2-row4-rack6-host7 -l 24 -v
Change-Id: I513210f1d106fdd70214aa58330e0beed3c43f23
Reviewed-on: https://chromium-review.googlesource.com/229432
Tested-by: Dan Shi <dshi@chromium.org>
Reviewed-by: Fang Deng <fdeng@chromium.org>
Commit-Queue: Dan Shi <dshi@chromium.org>
/external/autotest/site_utils/host_history_utils.py
|
| 1c3b0d1836ec77e92851d59a0fd24cdefdac44f3 |
|
27-Sep-2014 |
Dan Shi <dshi@chromium.org> |
[autotest] report -1 for machine stats that match no host
graphite keeps the last value of a gauge if no new data is reported. In the
case of a board disconued from a pool, the stats will go stale for the last
value. Set the value to -1 to indicate the stats is not applicable for
analysis.
Also, use host_history script to get host history directly, without making
any RPC through AFE.
CQ-DEPEND=CL:221743
BUG=chromium:419043
TEST=run collect_host_stats.py locally
Change-Id: I5b581873687c8daa5a871bcaa32be3eecb03b266
Reviewed-on: https://chromium-review.googlesource.com/220210
Tested-by: Dan Shi <dshi@chromium.org>
Reviewed-by: Fang Deng <fdeng@chromium.org>
Commit-Queue: Dan Shi <dshi@chromium.org>
/external/autotest/site_utils/host_history_utils.py
|
| 17ecbbf63319dabd5514827a34f04b5a0b724352 |
|
06-Oct-2014 |
Dan Shi <dshi@chromium.org> |
[autotest] Redesign how host history is queried to avoid any DB access
The host list is retrieved from metaDB for given board and label.
BUG=chromium:419043
TEST=local run
host_history.py --board peppy -l 240
host_history.py -l 240 --hosts 172.27.213.193
Change-Id: I63595534f167b1676f62e03e0c1643479e1e1f12
Reviewed-on: https://chromium-review.googlesource.com/221743
Tested-by: Dan Shi <dshi@chromium.org>
Reviewed-by: Fang Deng <fdeng@chromium.org>
Commit-Queue: Dan Shi <dshi@chromium.org>
/external/autotest/site_utils/host_history_utils.py
|
| 1ccb65208c26615096985b5f8b52368f7342a077 |
|
20-Sep-2014 |
Dan Shi <dshi@chromium.org> |
[autotest] add a utility to calculate machine stats and report to graphite
reference doc about machine stats:
https://docs.google.com/a/google.com/document/d/1N8Fk5Ey08y7OdYctmAxTQTyc1KjIpy14Y7WUpI-anUI/edit#heading=h.2vvien6shony
A cron job will be created in one of the lab server to collect the stats daily.
BUG=chromium:394451
TEST=local run site_utils/collect_host_stats.py
Change-Id: I935bb3431bf5064898a9e54a3204827a63e05701
Reviewed-on: https://chromium-review.googlesource.com/219167
Tested-by: Dan Shi <dshi@chromium.org>
Reviewed-by: Prashanth B <beeps@chromium.org>
Commit-Queue: Dan Shi <dshi@chromium.org>
/external/autotest/site_utils/host_history_utils.py
|
| dfea368e5c830b1d7950ced5ee7b191e3b141ca3 |
|
11-Aug-2014 |
Dan Shi <dshi@chromium.org> |
[autotest] Add RPC get_host_history to get host history.
The RPC takes in arguments like board, pool or a list of hosts, and return the
history of specified hosts. The history is a dictionary of each host and its
history of status. For example:
{'172.22.33.51': [{'status': 'Resetting'
'start_time': '2014-08-07 10:02:16',
'end_time': '2014-08-07 10:03:16',
'log_url': 'http://autotest/reset-546546/debug',
'task_id': 19441991},
{'status': 'Running'
'start_time': '2014-08-07 10:03:18',
'end_time': '2014-08-07 10:13:00',
'log_url': 'http://autotest/reset-546546/debug',
'job_id': 14995562}
]
}
Add a client util module time_utils to handle common task to do time string
and epoch time conversion.
BUG=chromium:394451
TEST=local setup
from autotest_lib.client.common_lib import global_config
from autotest_lib.server.cros.dynamic_suite import frontend_wrappers
instance_server = global_config.global_config.get_config_value(
'SERVER', 'hostname', type=str)
afe = frontend_wrappers.RetryingAFE(
server=instance_server, timeout_min=60, delay_sec=0)
result = afe.run('get_host_history', board='lumpy', pool='bvt',
start_time='2014-08-05 14:00:00',
end_time='2014-08-06 14:00:00')
import pprint
pprint.pprint(result)
rebuild local afe, test run_suite and create job from afe.
Change-Id: I66ecc5bb69a4a1fdb33437520288ed08e562ed00
Reviewed-on: https://chromium-review.googlesource.com/211772
Tested-by: Dan Shi <dshi@chromium.org>
Reviewed-by: Richard Barnette <jrbarnette@chromium.org>
Reviewed-by: Prashanth B <beeps@chromium.org>
Commit-Queue: Dan Shi <dshi@chromium.org>
/external/autotest/site_utils/host_history_utils.py
|
| 7cf3d84fda609f6402543bb7e0bf3e3b7f93d539 |
|
13-Aug-2014 |
Dan Shi <dshi@chromium.org> |
[autotest] Record more metadata to include more job related information.
For host history rpc to return more info like job name/owner, we added a new
attribute metadata_info to host object. metadata_info is a dictionary
containing information such as task_id, task_name, job_id, job_name,
parent_job_id.
When host status is changed, metadata_info will be reported to metaDB.
Examples of metadata_info are:
{"hostname": "192.96.48.88", "task_name": "Verify", "task_id": 6551}
{"job_name": "dummy_pass", "hostname": "192.96.48.88", "task_name": "Reset", "task_id": 6552, "job_id": 4133, "parent_job_id": 4132}
{'owner': 'debug_user', 'parent_job_id': null, 'job_id': 4140, 'job_name': 'dummy_pass'}
BUG=chromium:394451
TEST=local setup,
site_utils/host_history.py --hosts 100.96.48.196
test output with visiting page: http://172.25.61.45:9200/_plugin/elastic-hammer/
Enter following in filter, then click search to see results.
{"query": {"bool": {"minimum_should_match": 3,
"should": [{"term": {"_type": "host_history"}},
{"term": {"hostname": "100.96.48.196"}},
{"range": {"time_recorded": {"gte": 1408121317,
"lte": 1409007215}}}]}},
"size": 10000,
"sort": [{"time_recorded": "asc"}]}
Change-Id: Icddc27fb39529924d0030dbec97176a2d8b683fc
Reviewed-on: https://chromium-review.googlesource.com/212304
Reviewed-by: Dan Shi <dshi@chromium.org>
Tested-by: Dan Shi <dshi@chromium.org>
Commit-Queue: Dan Shi <dshi@chromium.org>
/external/autotest/site_utils/host_history_utils.py
|
| 3cea1699de56f06fda83d053a87710e3ed5ceee2 |
|
13-Aug-2014 |
Michael Liang <michaelliang@chromium.org> |
[Autotest] Fix bug in lock interval function
Previously there was a key error. entry['fields']['status'] does not exist.
Instead, it should be entry['fields']['locked'].
BUG=None
TEST=python host_history.py --index=cautotest --start="2014-08-12 00:00:00" \
--end="2014-08-13 00:00:00" --board=falco
Change-Id: Ib9da1dde83494af27a842590abd7fc958e3916eb
Reviewed-on: https://chromium-review.googlesource.com/212302
Tested-by: Michael Liang <michaelliang@chromium.org>
Reviewed-by: Dan Shi <dshi@chromium.org>
Commit-Queue: Michael Liang <michaelliang@chromium.org>
/external/autotest/site_utils/host_history_utils.py
|
| 9e0be9d048fecdf06a525bd4c31d272715494653 |
|
07-Aug-2014 |
Michael Liang <michaelliang@chromium.org> |
[autotest] Migrate host history to site_utils, fix bugs
Move from contrib/ to site_utils/ host_history{_utils}.py
Fixed a bug where value of --index is not passed into
query to elasticsearch.
BUG=None
TEST=python host_history.py --index=cautotest -n 10000 \
-l 24 --board=daisy
Change-Id: If1f4133335dae6f4416eb3ab31df94c1f70a0bcc
Reviewed-on: https://chromium-review.googlesource.com/211336
Commit-Queue: Michael Liang <michaelliang@chromium.org>
Tested-by: Michael Liang <michaelliang@chromium.org>
Reviewed-by: Dan Shi <dshi@chromium.org>
/external/autotest/site_utils/host_history_utils.py
|