History log of /external/trappy/tests/trappy/ftrace.py
Revision Date Author Comments (<<< Hide modified files) (Show modified files >>>)
9ea1d01da31d97ec83db0f4e6594d0f3a673e4bb 25-Oct-2017 Kevin DuBois <kevindubois@google.com> Merge branch 'master' of ssh://github.com/ARM-software/trappy into goog-master

Change-Id: I1c291fba04d70b5a5613a495a404bc38de76e1fd
957826a0f5991de2c9b31b32274361837aa4406e 17-Aug-2017 Brendan Jackman <brendan.jackman@arm.com> ftrace: Disable caching by default

There are some issues with caching at the moment which I haven't been
able to debug but that are confirmed as affecting multiple
people. Until we can find the time to fix the issue, let's disable
caching by default.
/external/trappy/trappy/ftrace.py
96f1148d408ef63a47db70eb97739189dce27f9f 09-Jul-2017 Joel Fernandes <joelaf@google.com> cache: Fix issue where single missing CSV deletes all

There's a bug in trappy caching where corruption or removal of single CSVs
causes all CSVs to get removed. Fix this and add a test-case to cover it.
Also test that only missing items are regenerated.

Change-Id: Iea41c8e0a005515790b580e7f78f06bdd60141a3
Signed-off-by: Joel Fernandes <joelaf@google.com>
Reviewed-by: KP Singh <kpsingh@google.com>
/external/trappy/trappy/ftrace.py
49f4c42b6750e96c668840a7d54e2cc2f322ab09 11-Jul-2017 Joel Fernandes <joelaf@google.com> trappy/utils: Make apply_callbacks more generic for single DF use

Change-Id: I3e8ff4d9d11220cddeac955e9a949fc3464ecc36
Signed-off-by: Joel Fernandes <joelaf@google.com>
/external/trappy/trappy/ftrace.py
ff807880438eb0c480bda11c961764e155b49eb2 09-Jul-2017 Joel Fernandes <joelaf@google.com> cache: Fix issue where single missing CSV deletes all

There's a bug in trappy caching where corruption or removal of single CSVs
causes all CSVs to get removed. Fix this and add a test-case to cover it.
Also test that only missing items are regenerated.

Change-Id: Iea41c8e0a005515790b580e7f78f06bdd60141a3
Signed-off-by: Joel Fernandes <joelaf@google.com>
/external/trappy/trappy/ftrace.py
a951463b87e0a17229d5fcd426b0b05a4ffb0764 30-Jun-2017 Joel Fernandes <joelaf@google.com> trappy/caching: Couple of fixes

- json params loading represents keys as unicode, use dumps to fix it
- finalize_objects need to be run always for cases where some events not
cached, fix it by moving it up and also handle bjackman's concern that we run
it twice by skipping finalize for cached events.
- fix time normalization breakage as a result of the above changes

Change-Id: I2011de0ae8112e937ee61baee8d53a63d0bbe85a
Signed-off-by: Joel Fernandes <joelaf@google.com>
/external/trappy/trappy/ftrace.py
cd7f2e045fa07bb14433a6506ab8756e94cd0f19 30-Jun-2017 Joel Fernandes <joelaf@google.com> trappy/caching: Couple of fixes

- json params loading represents keys as unicode, use dumps to fix it
- finalize_objects need to be run always for cases where some events not
cached, fix it by moving it up and also handle bjackman's concern that we run
it twice by skipping finalize for cached events.
- fix time normalization breakage as a result of the above changes

Change-Id: I2011de0ae8112e937ee61baee8d53a63d0bbe85a
Signed-off-by: Joel Fernandes <joelaf@google.com>
/external/trappy/trappy/ftrace.py
7f0b67931d5035205915a27cf6808707cf5b2030 28-Jun-2017 Joel Fernandes <joelaf@google.com> trappy: add support to parse TGID in systrace

Change-Id: Ie79698d90e0406cc11c52d364144ec08c33dfac4
Signed-off-by: Joel Fernandes <joelaf@google.com>
/external/trappy/trappy/ftrace.py
2a58727813c91d5497ce5452b5bff1bc1a467dc7 27-Jun-2017 Joel Fernandes <joelaf@google.com> trappy/ftrace: refactor getting class for unique word

Lets move this out of the main parsing loop so that the code is easier to read.

Suggested-by: Patrick Bellasi <patrick.bellasi@arm.com>
Signed-off-by: Joel Fernandes <joelaf@google.com>
Reviewed-by: KP Singh <kpsingh@google.com>
/external/trappy/trappy/ftrace.py
9bb52fae4f0bb3f81da19cc2f0387260d7e21c0d 25-Jun-2017 Joel Fernandes <joelaf@google.com> trappy: skip empty array regex sub for common case

I found that trappy spends a lot of time looking for the array pattern "{}".
Vast majority of trace lines don't have them. Instead of running the regex
every time, check for the ={} pattern using the 'in' operator which is much
faster than the regex for the common case, and then do the regex sub if needed.

The speed up is huge and in my test run, I saw parsing stage on a 100MB trace
speed up from 13s -> 10.5s !!

Change-Id: I7ae68efc99eaf8844624871be2a4d66a7820a9b0
Signed-off-by: Joel Fernandes <joelaf@google.com>
Reviewed-by: KP Singh <kpsingh@google.com>
/external/trappy/trappy/ftrace.py
677e6e5471b4d20270b8a7a839b6c0a961f7fc3e 25-Jun-2017 Joel Fernandes <joelaf@google.com> trappy: remove double check for unique word

Its not necessary to check for unique word twice. Reusing the 'class detection'
path for this purpose is sufficient. This improve performance by 3-6% of the
parsing stage.

Change-Id: Iff8ebd0086d2ccc10bec14e3d40a63f3c04aaf1a
Signed-off-by: Joel Fernandes <joelaf@google.com>
Reviewed-by: KP Singh <kpsingh@google.com>
/external/trappy/trappy/ftrace.py
c291667cc5ec1a71aee51a197f0cd0b45b9b20c6 19-Jun-2017 Connor O'Brien <connoro@google.com> Add support for trace event processing using callbacks

Some trace analysis tasks are most easily expressed by iterating over
events in chronological order. Add an apply_callbacks() method to the
GenericFtrace class that will run user-defined functions on trace
events in the order they occurred.

Signed-off-by: Connor O'Brien <connoro@google.com>
Reviewed-by: KP Singh <kpsingh@google.com>
/external/trappy/trappy/ftrace.py
3fdf6872b0d59743ff05ca4f7ec7f429a3cc81be 14-Jun-2017 Brendan Jackman <brendan.jackman@arm.com> ftrace: Store and retrieve window params in trace cache

Store a JSON file with the parameters used to parse a trace in the cache. If, on
retrieval, we find a mismatch then simply invalidate the cache.

A richer implementation might detect whether we can still use the cache (e.g. if
the requested window is a subset of the cached one), but for now I don't think
that's worth the effort/complexity.

Reviewed-by: KP Singh <kpsingh@google.com>
/external/trappy/trappy/ftrace.py
a19ff2508e37fce5e4909e0b717c3a4d2f0be679 15-May-2017 Joel Fernandes <joelaf@google.com> trappy: Speed up trappy by caching trace parsing

Pandas is extremely fast at parsing csv to data frames. Astonishingly it takes
< 1s to serialize/deserialize a 100MB work of traces with 430000 events to/from
csv. We leverage this and write out a data frames into a csv file when they are
created for the first time. Next time we read it out if it exists. To make
sure, the cache isn't stale, we take the md5sum of the trace file and also
ensure all CSVs exist before reading from the cache. I get a speed up of 16s to
1s when parsing a 100MB trace.

Co-developed-by: Brendan Jackman <brendan.jackman@arm.com>
Signed-off-by: Joel Fernandes <joelaf@google.com>
Reviewed-by: KP Singh <kpsingh@google.com>
/external/trappy/trappy/ftrace.py
ef596e5739b2e0946cbdfea09da09a314879dca8 17-Jun-2017 Chris Redpath <chris.redpath@arm.com> trappy/parsing: remove unnecessary raw flags from API and fix comments

Signed-off-by: Chris Redpath <chris.redpath@arm.com>
Reviewed-by: KP Singh <kpsingh@google.com>
/external/trappy/trappy/ftrace.py
12c3570032a60f8b3321026383d80eb318e3120f 22-Jun-2017 Chris Redpath <chris.redpath@arm.com> trappy/ftrace: Raise an exception if attempting to parse .raw.txt files

Since we no longer generate these, we should let the user either remove
their text files manually or merge the files if they no longer have the
.dat file. The exception describes how to deal with this format change
and which events should be moved from the raw to the formatted text file
if the user no longer has the .dat file.

Signed-off-by: Chris Redpath <chris.redpath@arm.com>
Reviewed-by: KP Singh <kpsingh@google.com>
/external/trappy/trappy/ftrace.py
c10c721b4821a050afaad471d19f5f9f66a93c78 17-Jun-2017 Chris Redpath <chris.redpath@arm.com> trappy/ftrace: Stop generating trace.raw.txt

Now that we have events constructed before generating trace files, we can
scan them and determine which events need to be formatted and which should
be unformatted (raw). trace-cmd report lets us specify events we want to
print raw. Use this to produce a single trace.txt which has all the
necessary formatted & raw events instead of calling trace-cmd report
twice.

Signed-off-by: Chris Redpath <chris.redpath@arm.com>
Reviewed-by: KP Singh <kpsingh@google.com>
/external/trappy/trappy/ftrace.py
b01c848f3928dfcdf2f3d7dae1343bbe677847ef 17-Jun-2017 Chris Redpath <chris.redpath@arm.com> trappy/Parsing: Split trace class init into two phases

This patch set is to modify the parsing step so that we don't need to
build a raw and a formatted trace file. To do that, we need to know
which events should have raw output and which should use their default
formatting. The events indicate which are which, but currently we
generate the trace files before we populate the events.

Splitting the initialisation into two parts means that we can populate
the events so that a later patch can create the text trace with each
event either formatted or raw as required.

Signed-off-by: Chris Redpath <chris.redpath@arm.com>
Reviewed-by: KP Singh <kpsingh@google.com>
/external/trappy/trappy/ftrace.py
8dcd96197ec48f37615d3ae7c6465e41f4ca6e3f 20-Jun-2017 Patrick Bellasi <patrick.bellasi@arm.com> trappy/ftrace: add support for boot clock timestamp

FTrace can be configured to report events timestamp using different
clock sources which can be selected via the trace_clock sysfs attribute
as described in:
https://www.kernel.org/doc/Documentation/trace/ftrace.txt

The global clock source reports time in [s] with a [us] resolution.
Other sources instead, like for example the boot clock, uses [ns].
Thus, in these last cases we do not have decimals in the timestamp.

Let's update the special fields regexp to match both [s] and [ns]
formatted times and do the required pre-processing to ensure that
DataFrames are alwasy expressed using a [s].[decimals] format.
This also update the base test to add a set of trace events which are
expressed in [ns] resolution.

Signed-off-by: Patrick Bellasi <patrick.bellasi@arm.com>
Reviewed-by: KP Singh <kpsingh@google.com>
/external/trappy/trappy/ftrace.py
658d4cc50a81cc0234bd4cdb3fa87f38782223b1 14-Jun-2017 Joel Fernandes <joelaf@google.com> trappy/ftrace: add a line number column

Useful for joining DataFrames that have timestamp collisions or for
iterating through a group of DataFrames in line order.

Signed-off-by: Joel Fernandes <joelaf@google.com>
Reviewed-by: KP Singh <kpsingh@google.com>
/external/trappy/trappy/ftrace.py
750f7187a8d0ad2d72a0261261bc9a7d81fbae93 14-Jun-2017 Patrick Bellasi <patrick.bellasi@arm.com> trappy/ftrace: add support to account line numbers

Knowing the line number of a trace event into the original trace file can
be super useful for two main reasons:
1. it can be used as a truly unique ID of each DF entry, since it is not
subject to the possible index collisions due to the timestamp resoluton
2. it can be super-useful to correlate DF entries with the original trace

If this ID is also tracked the same way tools like kernelshark does, than
we have a definitive and easy tool to match DF records with kernelshark.

Since kernelshark uses a 0-based indexing for trace events, let's account
for proper trace events line numbers starting from 0. This number is available
as a class property, in case we are interested to know to real total number
of events which are available in the original trace. Moreover, it will be
used by a following patch to add a unique ID to each DF entry.

Signed-off-by: Patrick Bellasi <patrick.bellasi@arm.com>
Reviewed-by: KP Singh <kpsingh@google.com>
/external/trappy/trappy/ftrace.py
663f1042f35a3dad09f8610bf60d4ecb30efec24 14-Jun-2017 Patrick Bellasi <patrick.bellasi@arm.com> trappy/ftrace: use special fileds RE to drive trace_hasnt_started

Currently we open an FTrace two times: one to parse metadata and another
to actually parse trace events. The second time we open the trace we don't
skip the metadata since they are filtered out by the attention words match.

Since trappy is now empowered by the trace_hasnt_started() iterator filter,
let's use this to skip the metadata before entering the parsing loop.
This change is preparatory for a proper accounting of trace events from
the parsing loop.

While we are at that, let's be eco-friendly and recycle the special field
regexp also for the exit condition of __populate_metadata().

Signed-off-by: Patrick Bellasi <patrick.bellasi@arm.com>
Reviewed-by: KP Singh <kpsingh@google.com>
/external/trappy/trappy/ftrace.py
071bbff272362e8f0eeb027ba7b148070b465403 14-Jun-2017 Joel Fernandes <joelaf@google.com> trappy: Handle the case for special events injected with tracing_mark_write

Certain fake events can be injected using tracing_mark_write and should be
classified into their own event classes than tracing_mark_write. One way to
handle this is to use tracing_mark_write only as a backup. For this reason lets
introduced the concept of backup event classes and classify TracingMarkWrite as
that.

This case normally doesn't happen because dynamic events are inserted into the
beginning of the event list from what I've seen. Since tracing_mark_write is a
built-in event, and not dynamic, its always at after and so is less preferred.
However we should protect for the future case where there might be a built-in
which happens to be after TrackingMarkWrite class in the list, so lets handle
it now.

Signed-off-by: Joel Fernandes <joelaf@google.com>
Reviewed-by: KP Singh <kpsingh@google.com>
/external/trappy/trappy/ftrace.py
f375f3f00bd6f6f2ebc681d6184c38d2222c593b 12-Jun-2017 Joel Fernandes <joelaf@google.com> trappy/ftrace: Parse data string along with special fields

This is needed so that we're not dependent anymore on making sure data starts
at something with '=' and will allow to parse custom systrace
tracing_mark_write events which don't have an '=' in them. The new regex will
also work for lines that do already have an '=' and doesn't change their
behavior.

The last line of the new regex in this patch works like this:
r"(?P<timestamp>[0-9]+\.[0-9]+): (\w+:\s+)+(?P<data>.+)"

We use skip over '<string>:<whitespace> ' and the next thing we find is
considered the start of the data. This works well to find the start of a line
for existing traces and is needed for systraces which don't have '='.

Signed-off-by: Joel Fernandes <joelaf@google.com>
Reviewed-by: KP Singh <kpsingh@google.com>
/external/trappy/trappy/ftrace.py
b1b1c9ec6deb6cb879c683f7ddaebb98d2f016ac 03-May-2017 Brendan Jackman <brendan.jackman@arm.com> ftrace: Improve error messages when failing to parse trace
/external/trappy/trappy/ftrace.py
e81fdcb135d0325e3bc22fae0583555d20aae280 04-Jan-2017 Brendan Jackman <brendan.jackman@arm.com> Update ARM Ltd. copyright to 2017
/external/trappy/trappy/ftrace.py
5e97a10f18e26d77c131e0454bf807427d02dfe9 07-Oct-2016 Brendan Jackman <brendan.jackman@arm.com> ftrace: Note in documentation that `window` is inclusive
/external/trappy/trappy/ftrace.py
e10e0879d72ed0c46864bead8ff70918b9033da7 01-Apr-2016 Michele Di Giorgio <michele.digiorgio@arm.com> ftrace: fix default name for traces

Having "." as default name for traces was preventing TRAPpy from
assigning a meaningful name, like "Trace 0", to traces for which the
user does not explicitly set a name. As a consequence, every plot with
such traces was showing a dot in the legend and in the title (where
trace name is relevant).

close #168
/external/trappy/trappy/ftrace.py
8a3fdcd2ea43b4e49330f21af433f5be214b7b5a 28-Mar-2016 Patrick Bellasi <patrick.bellasi@arm.com> ftrace: improve special fields matching

612384c (ftrace: match comm names that have a '[' as part of their name)
fixed parsing of special fields to match also names with contains a '['
by forcing a task name to match up to 16 chars.
This regexp seems to not be robust enough for systrace collected traces.
Indeed, not only names can contains the '[' char but, in general, they can
also be longer than 16 chars.

What happen is that, while the linux kernel truncates a task name (comm) to
16 chars, the systrace tool tries its best to "fix" that by fixing truncated
names. Specifically, this is done by the "atrace" agent, e.g.
https://github.com/catapult-project/catapult/blob/0d71be6/systrace/systrace/agents/atrace_agent.py#L583

Thus, in traces collected via systrace/atrace, not only names can contain
the '[' character but they can also be longer than 16 chars.

This patch uses a more generic reg-exp, which is a refinemend of the one
proposed in PR #90, which finally should allow to match task names of any
length and set of used characters.
A meaningless but challenging task name has been added in the test to
proof the patch and ensure a more robust coverage of this "sensible" code.

Signed-off-by: Patrick Bellasi <patrick.bellasi@arm.com>
/external/trappy/trappy/ftrace.py
46458d66cd5118b6d9cac7cb455f8dae1286b9b2 22-Mar-2016 Javi Merino <javi.merino@arm.com> systrace: add a class to parse trace coming from SysTrace

The SysTrace class has the same interface as FTrace. SysTrace doesn't
support raw traces. The only special thing that we have to do is skip
all the HTML in the file and parse the actual trace.
/external/trappy/trappy/ftrace.py
ce68bd04807c4833ebf27db3602c735633646b5b 22-Mar-2016 Javi Merino <javi.merino@arm.com> ftrace: provide a method to drop the beginning/end of a file

Subclasses of GenericFTrace that want to skip the beginning or end of
the file can now do so by providing trace_hasnt_started() and
trace_hasnt_finished() functions.
/external/trappy/trappy/ftrace.py
7f0458aed5b14da13882dbb9467f2f94659af5d1 22-Mar-2016 Patrick Bellasi <patrick.bellasi@arm.com> ftrace: update the special fields regexp to match also systrace embedded traces

catapult/systrace.py collects a textual trace from the target which is
embedded in the generated TraceView HTML format report file. These
events come straight from /sys/kernel/debug/tracing/trace and have a
slightly different format with respect to the ones normally generated by
"trace-cmd report".

This patch updates the regexp used to match special files of each event
to match both the trace-cmd and the systrace ones.

Signed-off-by: Patrick Bellasi <patrick.bellasi@arm.com>
/external/trappy/trappy/ftrace.py
299725afaf586e70777d4c9d2b0f00a3952cf6b2 22-Mar-2016 Javi Merino <javi.merino@arm.com> ftrace: abstract parsing of ftrace output to a generic class

We currently have the FTrace class to parse output from trace-cmd but
FTrace output can come from other sources. Abstract it so that the
generic parsing can be reused for parsers of other sources.
/external/trappy/trappy/ftrace.py
094e742d3504d692e74913ed85ec3d4b176f0469 22-Mar-2016 Javi Merino <javi.merino@arm.com> dynamic: use register_ftrace_parser() internally

Most registration of base events in trappy were being done with
FTrace.register_parser() which introduces unnecessary noise when
refactoring. Use register_ftrace_parser() instead as it is what we
recommend for external classes and it simplifies refactoring.
/external/trappy/trappy/ftrace.py
612384c42c3f00725f555265b5ba2f92de2c1e99 07-Mar-2016 Javi Merino <javi.merino@arm.com> ftrace: match comm names that have a '[' as part of their name

990f925224107 ("run: update special fields regexp to parse task names with a
space") fixed parsing task names that had spaces by assuming that no task
would have a "[" as part of their name. The day has come that we have found
a task with a "[".

Set the regexp to match anything up to 16 characters. The kernel seems to
limit the task name to 16 characters so this should be safe from now on (until
we find another task that breaks the regexp, that is).
/external/trappy/trappy/ftrace.py
ab94dac062ed58f03b1728a2857fa1563ed1e900 05-Jan-2016 Javi Merino <javi.merino@arm.com> trappy: partially revert 8da875103915

In 8da875103915 ("ftrace: fix error with only one actor") we made
ftrace.plot_allfreqs() treat the case when there's only one actor a
special case that ignored the requirement of the "ax" parameter being an
array. The documentation of plot_allfreqs() is correct: if ax is not
None, it must be an array of the same size as map_label.

Revert the change to ftrace.py and fix the test to match what the
documentation says.
/external/trappy/trappy/ftrace.py
4ec4aee55dbd4045cfb6a2fe099615a569ce7ff7 05-Jan-2016 Javi Merino <javi.merino@arm.com> trappy: update copyright to 2016
/external/trappy/trappy/ftrace.py
3fb959cce49adea597c588e89679b32cdb649fb6 05-Jan-2016 Kapileshwar Singh <kapileshwarsingh@gmail.com> Merge pull request #119 from JaviMerino/unregister_events

Unregister events
8da87510391505576e2b19331f236eab06844316 25-Dec-2015 Leo Yan <leo.yan@linaro.org> ftrace: fix error with only one actor

When system has only one actor and call method trappy.summary_plots, it
will report failure "zip argument #1 must support iteration".

This failure is cause by parsing allfreqs plots. Python always pass by
value, so when there have only one actor the axis will pass by value and
it's not a array type anymore. So finally zip will report it cannot
support iteration. So in this case convert axis to array type.

Signed-off-by: Leo Yan <leo.yan@linaro.org>
/external/trappy/trappy/ftrace.py
b280b4fd175789648650f66ed5fb3b4f42fb3de2 22-Dec-2015 Javi Merino <javi.merino@arm.com> ftrace: learn to unregister events

We have register_dynamic_ftrace() and register_ftrace_parser() but we
don't have a way to delete those events. Once an event has been
registered, all subsequent calls to FTrace will parse the event (unless
using scope="custom"). Add unregister_dynamic_ftrace() and
unregister_ftrace_parser() to remove events if they're not needed.
/external/trappy/trappy/ftrace.py
fecb2676b883265a98e37deff1e4764337647c16 21-Dec-2015 Javi Merino <javi.merino@arm.com> dynamic: ensure dynamic methods refer to ftrace

Now that we can have traces coming from sources other than ftrace,
rename register_dynamic() and register_class() to
register_dynamic_ftrace() and register_ftrace_parser() to make it clear
that they are only available for traces coming from ftrace.
/external/trappy/trappy/ftrace.py
08f3c34d0a24e7ffeff543e302efc911306e7bea 21-Dec-2015 Javi Merino <javi.merino@arm.com> bare_trace: abstract some of the basic FTrace functionality to a generic BareTrace class

Part of the functionality in FTrace is generic. In order to make the
rest of trappy less dependent on FTrace, move the generic parts to a new
BareTrace class and inherit from it.
/external/trappy/trappy/ftrace.py
4c0e5505be736c393a47880583ea053addf035d8 21-Dec-2015 Javi Merino <javi.merino@arm.com> ftrace: Learn again to normalize the time against an arbitrary basetime

In the past ftrace.normalize_time() accepted an arbitrary basetime
parameter. This was lost in c79eccd5370e ("run: drop get_basetime()
method") but the documentation of normalize_time() still talks about the
basetime parameter. Recover it.
/external/trappy/trappy/ftrace.py
886c03e14e88478cb199cef643bd56059ffdc1ca 21-Dec-2015 Javi Merino <javi.merino@arm.com> ftrace: append the events added by add_parsed_event() to trace_classes

get_duration() relies on all the classes being present in
trace_classes. Add events from add_parsed_event() to trace_classes so
that get_duration() takes them into account.
/external/trappy/trappy/ftrace.py
8ac68cb1a848f094f7ba990c4a295c92383277e4 21-Dec-2015 Javi Merino <javi.merino@arm.com> ftrace: set normalized_time if anybody calls normalize_time()

normalized_time is set to true when the time in the trace events are
normalized. Set it when normalize_time() is called to behave properly
even if FTrace() was called with normalize_time=0 and then we called
.normalize_time() on the object.
/external/trappy/trappy/ftrace.py
c26a323210533d4ed3a8b4e62c33744236e3beda 11-Dec-2015 Javi Merino <javi.merino@arm.com> ftrace: rename Run to FTrace

Run is a very generic and inappropriate name for an object that parses
the result of ftrace into a form consumable by python. Rename it to
FTrace in preparation for its generalization.
/external/trappy/trappy/ftrace.py