History log of /system/core/logd/LogStatistics.h
Revision Date Author Comments (<<< Hide modified files) (Show modified files >>>)
501c373916e292764400dbae735f44b33378400f 10-Mar-2017 Mark Salyzyn <salyzyn@google.com> logd: specify clang format

Switch _all_ file's coding style to match to ease all future changes.

SideEffects: None
Test: compile
Bug: 35373582
Change-Id: I470cb17f64fa48f14aafc02f574e296bffe3a3f3
/system/core/logd/LogStatistics.h
61e9ce6709a12cf39a9471476da3d50339efe466 12-Sep-2016 Mark Salyzyn <salyzyn@google.com> logd: add getEventTag command and service

Will register a new event tag by name and format, and return an
event-log-tags format response with the newly allocated tag.
If format is not specified, then nothing will be recorded, but
a pre-existing named entry will be listed. If name and format are
not specified, list all dynamic entries. If name=* list all
event log tag entries.

Stickiness through logd crash will be managed with the tmpfs file
/dev/event-log-tags and through a reboot with add_tag entries in
the pmsg last logcat event log. On debug builds we retain a
/data/misc/logd/event-log-tags file that aids stickiness and that
can be picked up by the bugreport.

If we detect truncation damage to /dev/event-log-tags, or to
/data/misc/logd/event-log-tags, rebuild file with a new first line
signature incorporating the time so mmap'd readers of the file can
detect the possible change in shape and order.

Manual testing:

Make sure nc (netcat) is built for the target platform on the host:
$ m nc

Then the following can be used to issue a request on the platform:
$ echo -n 'getEventTag name=<name> format="<format>"\0EXIT\0' |
> nc -U /dev/socket/logd

Test: gTest logd-unit-test --gtest_filter=getEventTag*
Bug: 31456426
Change-Id: I5dacc5f84a24d52dae09cca5ee1a3a9f9207f06d
/system/core/logd/LogStatistics.h
2ead17196c10f40af35e954dd7b42a2c48304b1e 20-Dec-2016 Treehugger Robot <treehugger-gerrit@google.com> Merge "logd: trailing spaces in log statistics (part deux)"
b545e1c9e3d5989e2e889e65436524fdcfed930e 19-Dec-2016 Mark Salyzyn <salyzyn@google.com> logd: trailing spaces in log statistics (part deux)

Test: manual
Bug: 30118730
Change-Id: Iafda6f95e1b1377649e01868c81c0056c116e01b
/system/core/logd/LogStatistics.h
3296291cffd13af13ea9e60a8ac1138101cf8e4c 12-Sep-2016 Mark Salyzyn <salyzyn@google.com> logd: add android::sizesTotal() function

Test: gTest liblog-unit-tests, logd-unit-tests and logcat-unit-tests
Bug: 31456426
Change-Id: I6fb6cd589227fc43d22f9336e01f1bf7e3feb24c
/system/core/logd/LogStatistics.h
a2c022257c5bed56fbc47de25c5d909bbe880f7b 13-Dec-2016 Mark Salyzyn <salyzyn@google.com> logd: record multiple duplicate messages as chatty

If a series of messages arrive from a single source with identical
message content payload, then suppress them and generate a chatty
report. The checking is done on a per log id basis.

This alters the assumption that chatty messages are always at the
oldest entries, they now show up in the middle too. To address this
change in behavior we print the first line, a chatty reference
which internally takes little space, then the last line in the series.

This does not conserve processing time in logd, and certainly has no
impact on the long path of formatting and submitting log messages from
from the source, but it may contribute to memory space and signal to
noise savings under heavy spammy loads.

Test: gTest liblog-unit-tests, logd-unit-tests & logcat-unit-tests
Bug: 33535908
Change-Id: I3160c36d4f4e2f8216f528605a1b3993173f4dec
/system/core/logd/LogStatistics.h
0eeb06b932f185e10377e4494475d2cdd6adfa1b 02-Dec-2016 Mark Salyzyn <salyzyn@google.com> utils: Add FastStrcmp.h

Move existing fast<str*cmp> templates for general use, pulled from
the implementation used in logd that dealt with speed through cache
locality and subroutine call mitigation. Rename to fastcmp.

Test: logd-benchmarks and based on manual profiling from the past
Bug: 31456426
Change-Id: Ic62f4a437fc3e06ffdeaae73a6f34e197957a6b0
/system/core/logd/LogStatistics.h
b0672290e320113f4b6d8fc6bb2d1e0afe493c5c 06-Oct-2016 Mark Salyzyn <salyzyn@google.com> logd: report last prune memory overhead

An estimate based on chatty impact for all known pids, uids and tags
and per log id if applicable, calculate the maximum last pruned
watermark iterator map usage and add to the Total Overhead.

Test: Confirm that the Total Overhead change is negligable.
Bug: 31942525
Change-Id: Icd2e9bc0747c3376ca0e9c90aa110c103529d98f
/system/core/logd/LogStatistics.h
6d981af12023bcaf22bfa0612a85ae4580ecb971 06-Oct-2016 Mark Salyzyn <salyzyn@google.com> logd: report statistics memory overhead

Add in to the Total Overhead the amount of storage we are
using to hold on to the statistics.

Test: see that the Total Overhead accounts for about 100K
Bug: 31942525
Change-Id: Ibe241c0bccc5a9df52395802338c8a7fc3b64104
/system/core/logd/LogStatistics.h
807e40ecc9786755e2f74a7a6a9b20c812588119 22-Sep-2016 Mark Salyzyn <salyzyn@google.com> liblog: logd: Add android_lookupEventTag_len()

Allows us to mitigate the impact of MAP_PRIVATE and copy on write by
calling android_lookupEventTag_len instead of android_lookupEventTag,
and delaying the copy on write impact to the later. We return a
string length in a supplied location along with the string pointer
with android_lookupEventTag_len(const EventTagMap* map, size_t* len,
int tag). The string is not guaranteed to be nul terminated. Since
android_lookupEventTag() called even once can cause the memory
impact, we will mark it as deprecated, but we currently have no
timeframe for removal since this is a very old interface.

Add an API for __android_log_is_loggable_len() that accepts the non
null terminated content and fixup callers that would gain because the
length is known prior to the call either in the compiler or at
runtime. Tackle transition to android_lookupEventTag_len() and
fixup callers.

On any application that performs logging (eg: com.android.phone)

/proc/<pid>/smaps before:

xxxxxxxxxx-xxxxxxxxxx rw-p 00000000 fd:00 463 /system/etc/event-log-tags
Size: 20 kB
Rss: 20 kB
Pss: 1 kB
Shared_Clean: 0 kB
Shared_Dirty: 20 kB
Private_Clean: 0 kB
Private_Dirty: 0 kB
Referenced: 0 kB
Anonymous: 20 kB
AnonHugePages: 0 kB
Swap: 0 kB
SwapPss: 0 kB
KernelPageSize: 4 kB
MMUPageSize: 4 kB
Locked: 0 kB
VmFlags: rd wr mr mw me ac

/proc/<pid>/smaps after:

xxxxxxxxxx-xxxxxxxxxx rw-p 00000000 fd:00 1773 /system/etc/event-log-tags
Size: 20 kB
Rss: 20 kB
Pss: 1 kB
Shared_Clean: 20 kB (was 0kB)
Shared_Dirty: 0 kB (was 20kB)
Private_Clean: 0 kB
Private_Dirty: 0 kB
Referenced: 20 kB (was 0kB)
Anonymous: 0 kB (was 20kB)
AnonHugePages: 0 kB
Swap: 0 kB
SwapPss: 0 kB
KernelPageSize: 4 kB
MMUPageSize: 4 kB
Locked: 0 kB
VmFlags: rd wr mr mw me ac

Added liblog-unit-tests --gtest_filter=liblog.event_log_tags to
check for Shared_Clean: to not be 0 and Anonymous: to be 0 for
all processes referencing event-log-tags. Which can include multiple
references to /system/etc/event-log-tags and future possible refs to
/data/misc/logd/event-log-tags and /dev/event-log-tags. We want
failure messages to help point to errant code using the deprecated
interface.

This change saves 1/4MB of memory or more on a typical system.

Test: gTest liblog-unit-tests
Bug: 31456426
Change-Id: I9e08e44d9092bd96fe704b5709242e7195281d33
/system/core/logd/LogStatistics.h
0dd4431072cce3c62876b728cb20aa5b77b11a8d 29-Sep-2016 Mark Salyzyn <salyzyn@google.com> logd: logcat: Replace log/log.h with android/log.h

Should use android/log.h instead of log/log.h as a good example
to all others.

Test: Compile
Bug: 26552300
Bug: 31289077
Change-Id: If4c9711eb57267d4707b03d54a932af9de241b13
/system/core/logd/LogStatistics.h
9e70ba6352ad1ee4fdb3f70a8945c2985638de77 02-Aug-2016 Chih-Hung Hsieh <chh@google.com> Fix google-explicit-constructor warnings in logd.

Bug: 28341362
Change-Id: I5b419c40593982925034e9b7217b0eb53c51adcb
Test: build with WITH_TIDY=1
/system/core/logd/LogStatistics.h
6a06694a610d103afdf424b0bb69dc8f7f2b8e5a 15-Jul-2016 Mark Salyzyn <salyzyn@google.com> logd: Worst Tag filter enabled for events buffer

- Add drop logistics to TagTable
- replace uid references to a key reference since it
is an UID for most buffers, but a TAG for the
events and security buffer
- template the find worst entry mechanics into LogFindWorst class

Bug: 30118730
Change-Id: Ibea4be2c50d6ff4b39039e371365fed2453f17a2
/system/core/logd/LogStatistics.h
0f858179851ae24e1fe27b4eb3968218b5510ca2 15-Jul-2016 Mark Salyzyn <salyzyn@google.com> logd: rename sort() for pids to sortPids()

Bug: 30118730
Change-Id: If992b53b52a7470427fbe82abb93c95c1b2dc57a
/system/core/logd/LogStatistics.h
f0f94fe49b04c2d2bf0a0a44a7afb9118607b351 19-May-2016 Chih-Hung Hsieh <chh@google.com> Fix misc-macro-parentheses warnings in liblog.

Bug: 28705665
Change-Id: Ib272a83cb93b09d986c19a40e23a2679b9e17558
/system/core/logd/LogStatistics.h
bec3c3def945576d59d3344c16e149e6d9154e15 28-Aug-2015 Mark Salyzyn <salyzyn@google.com> logd: Add worst pid of system filter

- Add a new statistic that reports per pid and log_id for AID_SYSTEM
- Add a new pruning filter ~1000/! boolean
- Use this new statistic to prune on worst pid within AID_SYSTEM

Bug: 26029733
Bug: 21615139
Bug: 22855208
Change-Id: Iab5dd28f807dcf03d276372853883f3b6afa8294
/system/core/logd/LogStatistics.h
ee3b838e13dc2140ac2051c1012d471effd0fd5f 17-Dec-2015 Mark Salyzyn <salyzyn@google.com> logd: statistics per-pid filter

Primarily gives access to the Chattiest TIDs and TAGs
associated with a pid.

Has a secondary effect of allowing us to pull out the
command line, comm and in some cases the associated
PACKAGE for a specific pid while the logs are still
present even if the executable is gone.

Bug: 26029733
Bug: 21615139
Change-Id: I1ea63165a680a9318360579b70b1512078ed5682
/system/core/logd/LogStatistics.h
083b037c0740ca00f72429e4457bfdd4b4d4dfa7 04-Dec-2015 Mark Salyzyn <salyzyn@google.com> logd: liblog: logcat: Add LOG_ID_SECURITY

- Largish commit, buffer and access controls done together
- Add LOG_ID_SECURITY binary content log
- Add "default" meta buffer
- allow LOG_ID_SECURITY only from AID_SYSTEM and AID_ROOT UID & GID
- Use __android_log_security() to gate logging
- Add __android_log_security_bwrite() native access to security
logging.
- Add liblog.__security_buffer end-to-end gTest

Bug: 26029733
Change-Id: Ibcf5b4660c17c1aa6902c0d93f8ffd29c93d9a93
/system/core/logd/LogStatistics.h
4f71319df011d796a60a43fc1bc68e16fbf7d321 05-Dec-2015 Elliott Hughes <enh@google.com> Track rename of base/ to android-base/.

Change-Id: Idf9444fece4aa89c93e15640de59a91f6e758ccf
/system/core/logd/LogStatistics.h
c723df805a1a3199577cac947cebf2ab53abdb34 24-Aug-2015 Mark Salyzyn <salyzyn@google.com> logd: statistics report chatty effective percentage

Report the ESTIMATED instantaneous percentage decrease or increase
that an UID has to the logs as a result of the chatty filtration.

Bug: 22855208
Change-Id: If1e77afb81a2739a72b39bc7c57071763c1d64d8
/system/core/logd/LogStatistics.h
58b8be8906f903ac3d83c41bcb0fb9c7841945f0 30-Sep-2015 Mark Salyzyn <salyzyn@google.com> logd: correct for number of elements in prune

Chatty logs would distort the average log size by elevating the
elements, but not the size. Add statistical collection for the
number of elements that report chatty, and subtract that from
the number of elements to improve the pruning estimate. Pick
minElements as 1% rather than 10% of the total with this more
accurate number of elements, to a minumum of 4.

Bug: 24511000
Change-Id: I3f36558138aa0b2a50e4fac6440c3a8505d95276
/system/core/logd/LogStatistics.h
9b3a2784b94d88492906384ab3f3c9863f6b5eea 13-Oct-2015 Mark Salyzyn <salyzyn@google.com> Merge "logd: use coalesce instead of merge (cleanup)"
aaad42f47c7363d68ddfb9ef8f1b51972c4d429d 30-Sep-2015 Mark Salyzyn <salyzyn@google.com> logd: use coalesce instead of merge (cleanup)

- switch to coalesce instead of merge in naming of functions
and variables. Confusing since we also to merge-sorts and
other activities in the logger.
- define maxPrune rather than using a number in the code path.

Bug: 24511000
/system/core/logd/LogStatistics.h
758058ffd8820df71c27db7675c50a90a5fa02b4 22-Aug-2015 Mark Salyzyn <salyzyn@google.com> logd: object layer format statistics

Simplify table generation by placing the line and header formatting
into each type's (UID, PID, TID) object. Switch to const return
values for the ownership passing functions (*ToName() functions
and methods). Use longer variable names to reduce confusion.

Switch from LINES To NUM for pruned column as that more accurately
reflects what is dropped since one entry can contain several lines.

Bug: 22855208
Change-Id: Ib110dce98a68cf5f844eb30f8a192a1f691eeba2
/system/core/logd/LogStatistics.h
ddda212faa81d62f637926680cd8163345120f71 02-Oct-2015 Mark Salyzyn <salyzyn@google.com> logd: optimize code hotspots

Discovered that we had a few libc hotspots. Adjust code to generally
reduce or nullify the number of calls to malloc, free, strlen,
strcmp, strncmp, memcmp & strncasecmp. Total gain looks to be about
3% of logd's processing time. malloc still contributes to 3%, but all
others are now total 0.5%.

Bug: 23685592
Change-Id: Ife721121667969260cdb8b055524ae90f5911278
/system/core/logd/LogStatistics.h
5ac5c6b19364b5b3061a59db796b2357c95c3b64 28-Aug-2015 Mark Salyzyn <salyzyn@google.com> logd: Add LogUtils.h

Move prototypes from LogBufferElement.h to LogUtils.h

Change-Id: I55e42e17e6c997e35b2b78b87fd2f84f8f393282
/system/core/logd/LogStatistics.h
73160acc5cb5236b30327569e6b51dbfe73e4a0f 20-Aug-2015 Mark Salyzyn <salyzyn@google.com> logd: switch asprintf to std::string

Bug: 23350706
Change-Id: I715cdd4563a09de3680081947a3439f0cac623be
/system/core/logd/LogStatistics.h
ed777e9eece54bf899f1a77a83f8b702970de686 25-Jun-2015 Mark Salyzyn <salyzyn@google.com> logd: serialize accesses to stats helpers

Quick low-risk to resolve possible hash table corruption.
Resolved an unlikely path memory leak.

ToDo: replace lock with nested lock so no lock
helpers are required.

Bug: 22068332
Change-Id: I303ab06608502c7d61d42f111a9c43366f184d0c
/system/core/logd/LogStatistics.h
511338dd575572d567c04d69eaea60627b6c3452 19-May-2015 Mark Salyzyn <salyzyn@google.com> logd: switch to unordered_map from BasicHashtable

BasicHashtable is relatively untested, move over to
a C++ template library that has more bake time.

Bug: 20419786
Bug: 21590652
Bug: 20500228
Change-Id: I926aaecdc8345eca75c08fdd561b0473504c5d95
/system/core/logd/LogStatistics.h
17ed6797df722464eb5cc6dfc3e1e32aec284b70 20-Apr-2015 Mark Salyzyn <salyzyn@google.com> logd: Add TID statistics

Bug: 19608965
Change-Id: Ifbf0b00c48ef12b5970b9f9f217bd1dd8f587f2c
/system/core/logd/LogStatistics.h
344bff4391dd434dda501e812f18f524290c5a7c 13-Apr-2015 Mark Salyzyn <salyzyn@google.com> logd: Add Tag statistics

- Optional class of statistics for events Tags
- export tagToName from LogBuffer (located in main.cp to address
https://android-review.googlesource.com/#/c/110204)
- Can not handle dropped because getTag() can not work,
will need to be fixed if we start filtering.

Bug: 19608965
Change-Id: I7b90607ca588bf37bab71f19b1570a290e772776
/system/core/logd/LogStatistics.h
aa43ae2268076227e8b72ea095f8aeac439b3168 20-Apr-2015 Mark Salyzyn <salyzyn@google.com> logd: pidToName deficiency

any logging in zygote permanently sets the pidToName

Change-Id: I0b8fdc96fb5965a04d4dc4293c39815beef5eb8e
/system/core/logd/LogStatistics.h
81b3eabc49736b89c4f99940f79785074955eaa5 13-Apr-2015 Mark Salyzyn <salyzyn@google.com> logd: class hierarcy for Uid and Pid statistics.

Add EntryBase and EntryBaseDropped base classes for statistical
entries to inherit from. Abstract add(), subtract() and drop()
methods to common LogBufferElement in each for entry policy decisions.
Some move of details out of LogStatistics.cpp and place them into
LogStatistics.h. Add statistical add(), subtract() and
drop() methods to hash table to call entries for policy.

Bug: 19608965
Change-Id: Ib8a33a8fe28871ef165d1632c6546a5c606231e8
/system/core/logd/LogStatistics.h
ab0dcf682867bd7e1fdebfd8d8f9fafaccfad7f6 16-Mar-2015 Mark Salyzyn <salyzyn@google.com> logd: annotate worst-UID pruned entries

- internal dropped entries are associated by prune by worst UID
and are applied by UID and by PID
- track dropped entries by rewriting them in place
- merge similar dropped entries together for same UID(implied),
PID and TID so that blame can more clearly be placed
- allow aging of dropped entries by the general backgound pruning
- report individual dropped entries formatted to reader
- add statistics to track dropped entries by UID, the combination
of statistics and dropped logging can track over-the-top Chattiest
clients.

Bug: 19608965
Change-Id: Ibc68480df0c69c55703270cd70c6b26aea165853
/system/core/logd/LogStatistics.h
720f6d1d55d936d98cc9752e96f479e03e6d5009 16-Mar-2015 Mark Salyzyn <salyzyn@google.com> logd: Add Pid statistics

- Optional class of statistics for PID
- Enhance pidToName
- Enhanced uidToName
- Enhance pidToUid
- template sort and iteration

Bug: 19608965
Change-Id: I04a1f02e9851b62987f9b176908134e455f22d1d
/system/core/logd/LogStatistics.h
97c1c2beee2cbd8c67c1cd507367e5b084d853c8 10-Mar-2015 Mark Salyzyn <salyzyn@google.com> logd: optimize statistics

- Go back to basic requirements
- Simplify
- use hash tables to minimize memory impact

Bug: 19608965
Change-Id: If7becb34354d6415e5c387ecea7d4109a15259c8
/system/core/logd/LogStatistics.h
4ed16b4381e6723670575ff8da3cfe12fe74998a 03-Mar-2015 Mark Salyzyn <salyzyn@google.com> Revert "logd: Add minimum time bucket statistics"

This forward port reverts
commit e457b74ce6ee6d799812dc2ec5e4b8b18bcd3e91

No longer as necessary once we add
liblog: Instrument logging of logd write drops
Although this provided an indication of how close statistically we
were to overloading logd it is simpler to understand fails thus to
hunt and peck a corrected value for /proc/sys/net/unix/max_dgram_qlen

Change-Id: I2b30e0fc30625a48fd11a12c2d2cc6a41f26226f
/system/core/logd/LogStatistics.h
f5fc50958978fc039bc3207cbfcb140a277442eb 21-Sep-2014 Mark Salyzyn <salyzyn@google.com> logd: Add control statistics enable/disable.

- ro.build.type=user turn off statistics
- ro.config.low_ram=true turn off statistics
- logd.statistics override

Bug: 17526159
Bug: 17526187
Change-Id: I74796043ac34753c6dd10018719ebc0bcd94e00f
/system/core/logd/LogStatistics.h
e72c6e43668c8c6e1af77e2e5038557581cbf148 21-Sep-2014 Mark Salyzyn <salyzyn@google.com> logd: cleanup

- simplify access and control exposure to class list
- indent
- compile warning
- Follow standard naming convention for variables and methods
- merge common fragments
- Side Effects: none

Change-Id: I74796043ac34753c6dd10018719ebc0bcd94e010
/system/core/logd/LogStatistics.h
fdabe7287609af29793bb6ca15712582ba082521 21-Apr-2014 Mark Salyzyn <salyzyn@google.com> logd: oneline statistics does not report PID gone

- oneline, add check for gone
- Add pidGone() method, which caches the gone status

Change-Id: I8f03daeafb58583c2b08ec6b540486c05da0872d
/system/core/logd/LogStatistics.h
c8a576c637ae00577273b778498019dd609fcd15 05-Apr-2014 Mark Salyzyn <salyzyn@google.com> logd: Statistics improvements

- Drop in-place sorting.
- Add fast-track for uid sizes and elements.
- Add sort() for per-uid and per-pid statistics.

Change-Id: Ib8655d4cc0b3bd4e87534522987bcc7fc7a5251a
/system/core/logd/LogStatistics.h
4ba0387af5acc52e43e4004ccb4c46b8bc700349 07-Apr-2014 Mark Salyzyn <salyzyn@google.com> logd: Add pidToUid helper

Change-Id: I23ebae1957c027bff6cbc2573a227bf0c44c08a2
/system/core/logd/LogStatistics.h
9a03863e88da99ba010342c874252089dd771f7f 07-Apr-2014 Mark Salyzyn <salyzyn@google.com> logd: Add pidToName helper

Change-Id: Idd8e804ab65feb8dc432150ae701464de1ad5302
/system/core/logd/LogStatistics.h
e457b74ce6ee6d799812dc2ec5e4b8b18bcd3e91 20-Feb-2014 Mark Salyzyn <salyzyn@google.com> logd: Add minimum time bucket statistics

* Only enabled for dev builds because halves performance.
- Used to establish if /proc/sys/net/unix/max_dgram_qlen
is appropriate for the platform
(see logd/LogStatistics.cpp comments)
- enabled if logd.dgram_qlen.statistics is not zero/empty

Change-Id: Ib8ecfeb75b6f0f954c099a6b1032b8afb5efcbd4
/system/core/logd/LogStatistics.h
dfa7a07f5be656cfafdb4e75916bc9dcd9e592e7 11-Feb-2014 Mark Salyzyn <salyzyn@google.com> logd: liblog: logcat: Add LogWhiteBlackList

- liblog android_logger_get_log_size and android_logger_get_readable_size
adjusted to return long instead of int because of -G flag extending range

NB: ifdef'd only for userdebug and eng builds

- liblog Add android_logger_[sg]et_prune_list and android_logger_set_log_size
- logcat Add -P, -p and -G flags
- logd Add LogWhiteBlackList and configurable log size

(cherry picked from commit 18a5432158ad43b8faefe4950b30e760200ce0b4)

Change-Id: I1572338c1b34bd968ad7867857ef708156ec3b6a
/system/core/logd/LogStatistics.h
34facab86b0fe7ec613de92b46b637f864fb0682 06-Feb-2014 Mark Salyzyn <salyzyn@google.com> logd: liblog: logcat: Add Statistics

- logd add statistical collection and formatting
- liblog add android_logger_get_statistics call
- logcat add -S flag
- logcat add -b all

(cherry picked from commit 51a29c8dc445e4fb89860561933e54a231e6ffb4)

Change-Id: I521753b1969ecd4590c956aeeb1557d101059d67
/system/core/logd/LogStatistics.h