README.md
1<!-- Copyright 2015 The Chromium Authors. All rights reserved.
2 Use of this source code is governed by a BSD-style license that can be
3 found in the LICENSE file.
4-->
5
6# Telemetry
7
8Telemetry is the performance testing framework used by Chrome. It allows you
9to perform arbitrary actions on a set of web pages (or any android application!)
10and report metrics about it. The framework abstracts:
11
12* Launching a browser with arbitrary flags on any platform.
13* Opening a tab and navigating to the page under test.
14* Launching an Android application with intents through ADB.
15* Fetching data via the Inspector timeline and traces.
16* Using [Web Page Replay](https://github.com/chromium/web-page-replay) to
17 cache real-world websites so they don’t change when used in benchmarks.
18
19## Design Principles
20
21* Write one performance test that runs on major platforms - Windows, Mac,
22 Linux, Chrome OS, and Android for both Chrome and ContentShell.
23* Run on browser binaries, without a full Chromium checkout, and without
24 having to build the browser yourself.
25* Use Web Page Replay to get repeatable test results.
26* Clean architecture for writing benchmarks that keeps measurements and use
27 cases separate.
28
29**Telemetry is designed for measuring performance rather than checking
30 correctness. If you want to check for correctness,
31 [browser tests](http://www.chromium.org/developers/testing/browser-tests) are
32 your friend.**
33
34**If you are a Chromium developer looking to add a new Telemetry benchmark to
35[`src/tools/perf/`](https://code.google.com/p/chromium/codesearch#chromium/src/tools/perf/),
36please make sure to read our
37[Benchmark Policy](https://docs.google.com/document/d/1bBKyYCW3VlUUPDpQE4xvrMFdA6tovQMZoqO9KCcmqqQ/preview)
38first.**
39
40## Code Concepts
41
42Telemetry provides two major functionality groups: those that provide test
43automation, and those that provide the capability to collect data.
44
45### Test Automation
46
47The test automation facilities of Telemetry provide Python wrappers for a number
48of different system concepts.
49
50* _Platforms_ use a variety of libraries & tools to abstract away the OS
51 specific logic.
52* _Browser_ wraps Chrome's
53 [DevTools Remote Debugging Protocol](https://developer.chrome.com/devtools/docs/remote-debugging)
54 to perform actions and extract information from the browser.
55* _Android App_ is a Python wrapper around
56 [`adb shell`](http://developer.android.com/tools/help/adb.html).
57
58The Telemetry framework lives in
59[`src/tools/telemetry/`](https://code.google.com/p/chromium/codesearch#chromium/src/tools/telemetry/)
60and performance benchmarks that use Telemetry live in
61[`src/tools/perf/`](https://code.google.com/p/chromium/codesearch#chromium/src/tools/perf/).
62
63### Data Collection
64
65Telemetry offers a framework for collecting metrics that quantify the
66performance of automated actions in terms of benchmarks, measurements, and story
67sets.
68
69* A
70 [_benchmark_](https://code.google.com/p/chromium/codesearch#chromium/src/tools/telemetry/telemetry/benchmark.py)
71 combines a _measurement_ together with a _story set_, and optionally a set
72 of browser options.
73 * We strongly discourage benchmark authors from using command-line flags
74 to specify the behavior of benchmarks, since benchmarks should be
75 cross-platform.
76 * Benchmarks are discovered and run by the
77 [benchmark runner](https://code.google.com/p/chromium/codesearch#chromium/src/tools/telemetry/telemetry/benchmark_runner.py),
78 which is wrapped by scripts like
79 [`run_benchmark`](https://code.google.com/p/chromium/codesearch#chromium/src/tools/perf/run_benchmark)
80 in `tools/perf`.
81* A _measurement_ (called
82 [`StoryTest`](https://code.google.com/p/chromium/codesearch#chromium/src/tools/telemetry/telemetry/web_perf/story_test.py)
83 in the code) is responsible for setting up and tearing down the testing
84 platform, and for collecting _metrics_ that quantify the application
85 scenario under test.
86 * Measurements need to work with all story sets, to provide consistency
87 and prevent benchmark rot.
88 * You probably don't need to override `StoryTest` (see "Timeline Based
89 Measurement" below). If you think you do, please talk to us.
90* A
91 [_story set_](https://code.google.com/p/chromium/codesearch#chromium/src/tools/telemetry/telemetry/story/story_set.py)
92 is a set of _stories_ together with a
93 [_shared state_](https://code.google.com/p/chromium/codesearch#chromium/src/tools/telemetry/telemetry/story/shared_state.py)
94 that describes application-level configuration options.
95* A
96 [_story_](https://code.google.com/p/chromium/codesearch#chromium/src/tools/telemetry/telemetry/story/story.py)
97 is an application scenario and a set of actions to run in that scenario. In
98 the typical Chromium use case, this will be a web page together with actions
99 like scrolling, clicking, or executing JavaScript.
100* A _metric_ describes how to collect data about the story run and compute
101 results.
102 * New metrics should generally be
103 [timeline-based](https://code.google.com/p/chromium/codesearch#chromium/src/tools/telemetry/telemetry/web_perf/metrics/timeline_based_metric.py).
104 * Metrics can specify many different types of results, including numbers,
105 histograms, traces, and failures.
106* _Timeline Based Measurement_ is a built-in `StoryTest` that runs all
107 available timeline-based metrics, and benchmarks that use it can filter
108 relevant results.
109
110## Next Steps
111
112* [Run Telemetry benchmarks locally](/telemetry/docs/run_benchmarks_locally.md)
113* [Record a story set](https://sites.google.com/a/chromium.org/dev/developers/telemetry/record_a_page_set)
114 with Web Page Replay
115* [Add a measurement](https://sites.google.com/a/chromium.org/dev/developers/telemetry/add_a_measurement)
116* [Feature guidelines](https://sites.google.com/a/chromium.org/dev/developers/telemetry/telemetry-feature-guidelines)
117* [Profiling with Telemetry](https://sites.google.com/a/chromium.org/dev/developers/telemetry/profiling)
118* [Profile generation](https://sites.google.com/a/chromium.org/dev/developers/telemetry/telemetry-profile-generation)
119* [Telemetry unittests](https://sites.google.com/a/chromium.org/dev/developers/telemetry/telemetry-unittests)
120
121## Contact Us or Follow Along
122
123If you have questions, please email telemetry@chromium.org.
124
125You can keep up with Telemetry related discussions by joining the
126[telemetry group](https://groups.google.com/a/chromium.org/forum/#!forum/telemetry).
127
128[For Googlers](http://go/telemetry)
129
130## Frequently Asked Questions
131
132### I get an error when I try to use recorded story sets.
133
134The recordings are not included in the Chromium source tree. If you are a Google
135partner, run `gsutil config` to authenticate, then try running the test again.
136If you don't have `gsutil` installed on your machine, you can find it in
137`src/tools/telemetry/third_party/gsutil/gsutil`.
138
139If you are not a Google partner, you can run on live sites with
140--use-live-sites` or
141[record your own](http://dev.chromium.org/developers/telemetry/record_a_page_set)
142story set archive.
143
144### I get mysterious errors about device\_forwarder failing.
145
146Your forwarder binary may be outdated. If you have built the forwarder in
147src/out that one will be used. if there isn't anything there Telemetry will
148default to downloading a pre-built binary. Try re-building the forwarder, or
149alternatively wiping the contents of `src/out/` and running `run_benchmark`,
150which should download the latest binary.
151
152### I'm having problems with keychain prompts on Mac.
153
154Make sure that your keychain is
155[correctly configured](https://sites.google.com/a/chromium.org/dev/developers/telemetry/telemetry-mac-keychain-setup).
156