1<!-- Copyright 2015 The Chromium Authors. All rights reserved.
2     Use of this source code is governed by a BSD-style license that can be
3     found in the LICENSE file.
4-->
5
6# Telemetry
7
8Telemetry is the performance testing framework used by Chrome.  It allows you
9to perform arbitrary actions on a set of web pages (or any android application!)
10and report metrics about it.  The framework abstracts:
11
12*   Launching a browser with arbitrary flags on any platform.
13*   Opening a tab and navigating to the page under test.
14*   Launching an Android application with intents through ADB.
15*   Fetching data via the Inspector timeline and traces.
16*   Using [Web Page Replay](https://github.com/chromium/web-page-replay) to
17    cache real-world websites so they don’t change when used in benchmarks.
18
19## Design Principles
20
21*   Write one performance test that runs on major platforms - Windows, Mac,
22    Linux, Chrome OS, and Android for both Chrome and ContentShell.
23*   Run on browser binaries, without a full Chromium checkout, and without
24    having to build the browser yourself.
25*   Use Web Page Replay to get repeatable test results.
26*   Clean architecture for writing benchmarks that keeps measurements and use
27    cases separate.
28
29**Telemetry is designed for measuring performance rather than checking
30  correctness. If you want to check for correctness,
31  [browser tests](http://www.chromium.org/developers/testing/browser-tests) are
32  your friend.**
33
34**If you are a Chromium developer looking to add a new Telemetry benchmark to
35[`src/tools/perf/`](https://code.google.com/p/chromium/codesearch#chromium/src/tools/perf/),
36please make sure to read our
37[Benchmark Policy](https://docs.google.com/document/d/1bBKyYCW3VlUUPDpQE4xvrMFdA6tovQMZoqO9KCcmqqQ/preview)
38first.**
39
40## Code Concepts
41
42Telemetry provides two major functionality groups: those that provide test
43automation, and those that provide the capability to collect data.
44
45### Test Automation
46
47The test automation facilities of Telemetry provide Python wrappers for a number
48of different system concepts.
49
50*   _Platforms_ use a variety of libraries & tools to abstract away the OS
51    specific logic.
52*   _Browser_ wraps Chrome's
53    [DevTools Remote Debugging Protocol](https://developer.chrome.com/devtools/docs/remote-debugging)
54    to perform actions and extract information from the browser.
55*   _Android App_ is a Python wrapper around
56    [`adb shell`](http://developer.android.com/tools/help/adb.html).
57
58The Telemetry framework lives in
59[`src/third_party/catapult/telemetry/`](https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/)
60and performance benchmarks that use Telemetry live in
61[`src/tools/perf/`](https://code.google.com/p/chromium/codesearch#chromium/src/tools/perf/).
62
63### Data Collection
64
65Telemetry offers a framework for collecting metrics that quantify the
66performance of automated actions in terms of benchmarks, measurements, and story
67sets.
68
69*   A
70    [_benchmark_](https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/telemetry/benchmark.py)
71    combines a _measurement_ together with a _story set_, and optionally a set
72    of browser options.
73    *   We strongly discourage benchmark authors from using command-line flags
74        to specify the behavior of benchmarks, since benchmarks should be
75        cross-platform.
76    *   Benchmarks are discovered and run by the
77        [benchmark runner](https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/telemetry/benchmark_runner.py),
78        which is wrapped by scripts like
79        [`run_benchmark`](https://code.google.com/p/chromium/codesearch#chromium/src/tools/perf/run_benchmark)
80        in `tools/perf`.
81*   A _measurement_ (called
82    [`StoryTest`](https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/telemetry/web_perf/story_test.py)
83    in the code) is responsible for setting up and tearing down the testing
84    platform, and for collecting _metrics_ that quantify the application
85    scenario under test.
86    *   Measurements need to work with all story sets, to provide consistency
87        and prevent benchmark rot.
88    *   You probably don't need to override `StoryTest` (see "Timeline Based
89        Measurement" below). If you think you do, please talk to us.
90*   A
91    [_story set_](https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/telemetry/story/story_set.py)
92    is a set of _stories_ together with a
93    [_shared state_](https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/telemetry/story/shared_state.py)
94    that describes application-level configuration options.
95*   A
96    [_story_](https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/telemetry/story/story.py)
97    is an application scenario and a set of actions to run in that scenario. In
98    the typical Chromium use case, this will be a web page together with actions
99    like scrolling, clicking, or executing JavaScript.
100*   A _metric_ describes how to collect data about the story run and compute
101    results.
102    *   New metrics should generally be
103        [timeline-based](https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/telemetry/web_perf/metrics/timeline_based_metric.py).
104    *   Metrics can specify many different types of results, including numbers,
105        histograms, traces, and failures.
106*   _Timeline Based Measurement_ is a built-in `StoryTest` that runs all
107    available timeline-based metrics, and benchmarks that use it can filter
108    relevant results.
109
110## Next Steps
111
112*   [Run Telemetry benchmarks locally](/telemetry/docs/run_benchmarks_locally.md)
113*   [Record a story set](https://sites.google.com/a/chromium.org/dev/developers/telemetry/record_a_page_set)
114    with Web Page Replay
115*   [Add a measurement](https://sites.google.com/a/chromium.org/dev/developers/telemetry/add_a_measurement)
116*   [Feature guidelines](https://sites.google.com/a/chromium.org/dev/developers/telemetry/telemetry-feature-guidelines)
117*   [Profiling with Telemetry](https://sites.google.com/a/chromium.org/dev/developers/telemetry/profiling)
118*   [Profile generation](https://sites.google.com/a/chromium.org/dev/developers/telemetry/telemetry-profile-generation)
119*   [Telemetry unittests](https://sites.google.com/a/chromium.org/dev/developers/telemetry/telemetry-unittests)
120
121## Contact Us or Follow Along
122
123If you have questions, please email telemetry@chromium.org.
124
125You can keep up with Telemetry related discussions by joining the
126[telemetry group](https://groups.google.com/a/chromium.org/forum/#!forum/telemetry).
127
128[For Googlers](http://go/telemetry)
129
130## Frequently Asked Questions
131
132### I get an error when I try to use recorded story sets.
133
134The recordings are not included in the Chromium source tree. If you are a Google
135partner, run `gsutil config` to authenticate, then try running the test again.
136If you don't have `gsutil` installed on your machine, you can find it in
137`build/third_party/gsutil/gsutil`.
138
139If you are not a Google partner, you can run on live sites with
140--use-live-sites` or
141[record your own](http://dev.chromium.org/developers/telemetry/record_a_page_set)
142story set archive.
143
144### I get mysterious errors about device\_forwarder failing.
145
146Your forwarder binary may be outdated. If you have built the forwarder in
147src/out that one will be used. if there isn't anything there Telemetry will
148default to downloading a pre-built binary. Try re-building the forwarder, or
149alternatively wiping the contents of `src/out/` and running `run_benchmark`,
150which should download the latest binary.
151
152### I'm having problems with keychain prompts on Mac.
153
154Make sure that your keychain is
155[correctly configured](https://sites.google.com/a/chromium.org/dev/developers/telemetry/telemetry-mac-keychain-setup).
156