History log of /frameworks/native/cmds/installd/InstalldNativeService.cpp
Revision Date Author Comments (<<< Hide modified files) (Show modified files >>>)
4ca8ff98e40023c32e0f05473e1fe906b6537417 07-Jun-2017 Jeff Sharkey <jsharkey@android.com> Don't double-count external cached app data.

Cached app data is already counted in extStats.dataSize, so we don't
need to count it twice.

Test: builds, boots, verified Settings UI before/after
Bug: 62343684
Change-Id: Ic02cbe3f5938949730f9bc8d911ed398436f84d7
/frameworks/native/cmds/installd/InstalldNativeService.cpp
c523409f0fd03aa498cedc486f85e9a4b7257f3a 01-Jun-2017 Andreas Gampe <agampe@google.com> Installd: deleteOdex's outputPath can be null

Add @nullable, as the outputPath can be null for system apps.

Bug: 62236497
Test: m
Test: Fill up /data, apply OTA
Change-Id: I61d86721b485f85195b153d76ce25412dcb68be6
/frameworks/native/cmds/installd/InstalldNativeService.cpp
60f8a5330ca921d936ae306bf6d17596b82b518c 30-May-2017 Jeff Sharkey <jsharkey@android.com> Consistent "low storage" behavior.

When freeing cached data, the caller can now provide a "reserved"
size which we won't clear cached data from.

Bug: 38008706
Test: cts-tradefed run commandAndExit cts-dev -m CtsJobSchedulerTestCases -t android.jobscheduler.cts.StorageConstraintTest
Test: cts-tradefed run commandAndExit cts-dev -m CtsAppSecurityHostTestCases -t android.appsecurity.cts.StorageHostTest
Change-Id: Ieb91b3e5345a950d4785fd7915f520f0a68a567a
/frameworks/native/cmds/installd/InstalldNativeService.cpp
5cb903e0a976a143cd70441dcbd0fcb5a9630daa 17-May-2017 Jaekyun Seok <jaekyun@google.com> Newly create idmap only when it is stale

For now, OverlayManagerService calls Installd.idmap() whenever a user
is changed, and then a idmap is re-generated even though there are no
changes on its target apk and overlay apk.
This CL is to avoid such unnecessary re-generation of idmap. Instead
only a group id will be updated when the idmap isn't outdated.

To correctly check staleness of idmap, "--verify" function of idmap is
invented as well.

Test: building succeeded and tested on sailfish.
Bug: 37179531
Change-Id: I874be9765d37bfa6c562c3f39a395040dc6a7d1f
/frameworks/native/cmds/installd/InstalldNativeService.cpp
b57d3d222a1fe73a06e2a07416bfec1585e358a5 17-May-2017 Adam Lesinski <adamlesinski@google.com> Merge "Revert "Newly create idmap only when it is outdated"" into oc-dev
b6182b3080be69f7983ad6ff8e3d23ecabf4d266 17-May-2017 Adam Lesinski <adamlesinski@google.com> Revert "Newly create idmap only when it is outdated"

This reverts commit 1d091465334784adeded4ef182cc6f732eb6f99e.

Reason for revert: Causes b/37992057

Change-Id: I7811483016b28a88d6a50bd445dd5d5e8d2e48d6
/frameworks/native/cmds/installd/InstalldNativeService.cpp
d6ca10b76d2d00d2b60bb186f71b125f917825b0 09-May-2017 TreeHugger Robot <treehugger-gerrit@google.com> Merge "Do not return an error if the dalvik-cache odex is missing" into oc-dev
249f2d3e018bd98eba74c3184b80bc91b04281cd 04-May-2017 Calin Juravle <calin@google.com> Do not return an error if the dalvik-cache odex is missing

Test: adb shell /data/nativetest/installd_service_test/installd_service_test
Tried uninstalling an updated version of prebuilt app.
Didn't find the error message in the logcat.
Bug: 35804241
Change-Id: Iec045fa9c9ac8f036785fe5cbd0f7e43f2b680c1
/frameworks/native/cmds/installd/InstalldNativeService.cpp
d712f0ccc120357e1267a04ac6de9dd732b27e76 03-May-2017 Jeff Sharkey <jsharkey@android.com> Clear cached files on external storage.

When clearing cached files belonging to an app, include any cached
files on external storage. Since we need to keep sdcardfs in the
loop about any file deletions, we always mutate by going through the
sdcardfs layer instead of behind its back.

Test: cts-tradefed run commandAndExit cts-dev -m CtsAppSecurityHostTestCases -t android.appsecurity.cts.StorageHostTest
Bug: 37486230, 37566983, 37913442, 37914374
Change-Id: If174bf7eaf682da83cf0ab1b4938fe9a5956d464
/frameworks/native/cmds/installd/InstalldNativeService.cpp
bb54a7df3a42d4beeb5fd99e01369415a7d1c24c 26-Apr-2017 TreeHugger Robot <treehugger-gerrit@google.com> Merge "Newly create idmap only when it is outdated" into oc-dev
1d091465334784adeded4ef182cc6f732eb6f99e 19-Apr-2017 Jaekyun Seok <jaekyun@google.com> Newly create idmap only when it is outdated

For now, OverlayManagerService calls Installd.idmap() whenever a user
is changed, and then a idmap is re-generated even though there are no
changes on its target apk and overlay apk.
This CL is to avoid such unnecessary re-generation of idmap. Instead
only a group id will be updated when the idmap isn't outdated.

Test: building succeeded and tested on sailfish.
Bug: 37179531
Change-Id: Id726cbbc621bd64f2a329798b2ab5436f0c28d70
/frameworks/native/cmds/installd/InstalldNativeService.cpp
a084fcdcd052d71e8205a2eea0955911b99160b9 18-Apr-2017 Jeff Sharkey <jsharkey@android.com> Fix external cache and secondary user bugs.

External cache files now have their own GID range, so measure using
those quotas. Fix secondary user code measurement, since it always
lives under user 0. Fix secondary user data measurement in manual
mode; we need to match based on appId instead of pure UID.

Test: cts-tradefed run commandAndExit cts-dev -m CtsAppSecurityHostTestCases -t android.appsecurity.cts.StorageHostTest
Bug: 35812899, 35844919, 37193650
Change-Id: Ic3b153798164c33ea10e8a4dabc65edff26c56ca
/frameworks/native/cmds/installd/InstalldNativeService.cpp
6e03e0e348a09154c35058e12d26383efdbc57a1 16-Apr-2017 TreeHugger Robot <treehugger-gerrit@google.com> Merge "Fix use-after-free issue caught by ASAN." into oc-dev
c729b027424807e58db4f8fa82b2296624024f2b 16-Apr-2017 TreeHugger Robot <treehugger-gerrit@google.com> Merge "Fix profile disk space accounting bugs." into oc-dev
7b6c840c4281aeb2cc42ed022d1708fec0cda2ec 15-Apr-2017 Jeff Sharkey <jsharkey@android.com> Hopefully fix funky SELinux labels.

We've seen reports of cases where DE storage cache directories are
missing the SELinux category information, so explicitly offer to
restorecon those directories to fix the labels.

This matches what we're already doing for CE storage.

Test: builds, boots
Bug: 35991539
Change-Id: I0eb6df801e933fbb005e3f8fdfe57a4e5ce3313d
/frameworks/native/cmds/installd/InstalldNativeService.cpp
06a6184da6c2d80ea8a30f7cf827ecdfe191622d 15-Apr-2017 Jeff Sharkey <jsharkey@android.com> Fix profile disk space accounting bugs.

Profile data is only stored on internal storage (it doesn't move to
adoptable devices), so only count it when asking about internal
storage (which has a null UUID).

Also fix bug where "current" profile data was being double-counted
for apps.

Test: runtest -x frameworks/base/services/tests/servicestests/src/com/android/server/pm/InstallerTest.java
Bug: 36102172, 36101738
Change-Id: I22b82e8cfb3bf1be1155fb827a538c21f09d88eb
/frameworks/native/cmds/installd/InstalldNativeService.cpp
d9ff77fa4b934b54e287ba259cb02a0eee0d9ce8 15-Apr-2017 Jeff Sharkey <jsharkey@android.com> Fix use-after-free issue caught by ASAN.

Keep strong references to std::string objects around while we're
actively using them in fts_open().

Test: builds, boots
Bug: 36975037
Change-Id: Icd7cba5852a01f0a1015e7d0d7dcd3087fa44ae8
/frameworks/native/cmds/installd/InstalldNativeService.cpp
ec1be62c18b216901d738be0ccc8a05aab8edef0 10-Apr-2017 Jeff Sharkey <jsharkey@android.com> Get a hack in place for broken quotas.

Sadly we mixed together both UID and GID quotas in such a way that
we couldn't detangle the actual data-vs-cache usage in all cases.

Temporarily patch over this to get CTS passing until we can circle
back and try again.

Test: cts-tradefed run commandAndExit cts-dev -m CtsAppSecurityHostTestCases -t android.appsecurity.cts.StorageHostTest
Bug: 36731175, 37193650
Change-Id: I5870b4e9c22398762fc47672e4d5d78c06bef79f
/frameworks/native/cmds/installd/InstalldNativeService.cpp
cb556e340b50d16e8fac41b0c69d8d35bd7915f9 05-Apr-2017 Calin Juravle <calin@google.com> Restore selinux labels for secondary dex oat directory

Bug: 36896515
Test: adb shell cmd package compile -r bg-dexopt --secondary-dex com.google.android.gms
adb shell ls
/data/user_de/0/com.google.android.gms/app_chimera/m/0000000c/oat/arm64/
-Z
youtube loads gms modules without extracting from apk

Change-Id: I4e12a6f532a1442a840e3ed8d01b98dd9a328eb6
/frameworks/native/cmds/installd/InstalldNativeService.cpp
e12d5964a8d14abe7f2eb6e57469cbe7f7391a19 04-Apr-2017 Jeff Sharkey <jsharkey@android.com> Offer to "fixup" GIDs used for app data.

We recently started tracking cached app data using a per-app GID for
the "cache" and "code_cache" directories and their contents. For
upgraded devices, we ideally want to "fixup" the GIDs of any existing
data while the device is still showing the boot animation, instead of
blocking the user when they unlock the device.

Since all the information we need is available in metadata, we can
update GIDs before the user has unlocked data. We're pretty paranoid
and we only pivot between the normal app GID and the cache GID; any
other GID values are ignored.

This "fixup" method can also be used in the future to ensure
consistency of the files on disk. Also fix bug by always using
"fts_path" instead of "fts_accpath" which is based on racy chdir().

Test: /data/nativetest/installd_service_test/installd_service_test
Bug: 34201111, 35084485
Change-Id: Ia52694f3763cba09926082c08f0766477e03e39c
/frameworks/native/cmds/installd/InstalldNativeService.cpp
e59c85cc0e78bfcc8fec6acc8e37e6a472ffc07f 03-Apr-2017 Jeff Sharkey <jsharkey@android.com> Define upper-bound disk quotas for all apps.

Abusive or broken apps can go crazy and try allocating all of the
disk space on the device. To mitigate the impact on system health,
set hard limits to block any given app from using more than 90% of
disk blocks, or 50% of disk inodes.

Also define the hard limit for AID_MEDIA_RW to avoid filling up the
device via the SD card.

Kick QUOTAON when scanning devices, since ext4 doesn't toggle
DQUOT_LIMITS_ENABLED during initial mount.

Test: cts-tradefed run commandAndExit cts-dev -m CtsAppSecurityHostTestCases -t android.appsecurity.cts.StorageHostTest
Test: cts-tradefed run commandAndExit cts-dev -m CtsOsTestCases -t android.os.cts.EnvironmentTest#testSaneInodes
Bug: 36450358
Change-Id: Iaa2bc6a2f0bc9047ee54c1d1a49bbda92142457a
/frameworks/native/cmds/installd/InstalldNativeService.cpp
1cb4aafc17daae520a5c3a991f59605ea3ee4ee5 28-Mar-2017 Jeff Sharkey <jsharkey@android.com> Remove legacy cache clearing logic.

It was riddled with funky memory access, and it's been fully replaced
by the new V2 logic.

Test: builds, boots
Bug: 27948817, 36655947
Change-Id: Ic5f0636c414ccf5ccf56eb50fea321b6688a83a0
/frameworks/native/cmds/installd/InstalldNativeService.cpp
ed909ae8db2f44ce7fe7003c6fee457f13669702 23-Mar-2017 Jeff Sharkey <jsharkey@android.com> Follow "atomic" to "group" refactoring.

Remove noisy logging about UIDs that are relying on default cache
quota of 64MiB.

Move away from yucky old statfs() and use statvfs() instead.

Test: /data/nativetest/installd_cache_test/installd_cache_test
Bug: 35812899, 35684969, 36482620
Change-Id: I3d68da97eac2ebcda489bdf9d27061cac5b3f7cc
/frameworks/native/cmds/installd/InstalldNativeService.cpp
8348fa3545e0ad4e178b5f3faf37a3b68da727d6 17-Mar-2017 Calin Juravle <calin@google.com> Merge "Support profile guided compilation for secondary dex files" am: cb2e477f14 am: f01cd5f89f
am: 95c5f4b85d

Change-Id: I432fd0a8e7056b5d41e6beb9a71d8fd74c3a34b8
114f08107be384a3f9cc954bdec2b6b7619354ae 09-Mar-2017 Calin Juravle <calin@google.com> Support profile guided compilation for secondary dex files

Test: adb shell cmd package bg-dexopt-job works for sercondary dex files
Bug: 26719109
Change-Id: I4011a780f65378c23a4631c07c37cdb4f5fee41e
/frameworks/native/cmds/installd/InstalldNativeService.cpp
444ad1ee82a549c96bf9e5087b9041d6264c93ec 12-Mar-2017 Jeff Sharkey <jsharkey@android.com> Fix external manual calculation bug.

Ignore file extensions anywhere under "Android" since those files
belong to specific apps. Don't depend on GID being set for those
directories.

Bug: 35684969, 35441006, 34945234
Test: cts-tradefed run commandAndExit cts-dev -m CtsAppSecurityHostTestCases -t android.appsecurity.cts.StorageHostTest
Change-Id: Ifeb8d314cc2b803ac0d3595974daf91f51862061
/frameworks/native/cmds/installd/InstalldNativeService.cpp
2c5d1a85898f9178b0a1130b7f36ea3031361133 12-Mar-2017 TreeHugger Robot <treehugger-gerrit@google.com> Merge "Finer-grained locking for size operations."
b26786d647b624498c11405075e5223d1853f30a 12-Mar-2017 Jeff Sharkey <jsharkey@android.com> Finer-grained locking for size operations.

Disk space measurements are read-only and don't perform mutations,
so other installd operations shouldn't block them.

If there's an ongoing parallel operation (such as a dexopt) that
could race and skew the results, that's no different than an actively
running app changing it's disk usage during the measurement.

This change also allows measurements to happen in parallel, so we can
no longer rely on getcwd() being stable, which means all fts(3) users
now need to use FTS_NOCHDIR.

Bug: 36032444, 35706513
Test: runtest -x frameworks/base/services/tests/servicestests/src/com/android/server/pm/InstallerTest.java
Change-Id: I67d303d3ecce148052d41444cef67381b1d34ab0
/frameworks/native/cmds/installd/InstalldNativeService.cpp
47ec28647af5ce9d7079ee2ab782cf42e6299cad 11-Mar-2017 Calin Juravle <calin@google.com> Merge "Use std::string for profile operations instead of const char" am: 7c7d888d94 am: ccaa1f3f78
am: 0045d606d1

Change-Id: I6ad63a777811dbc5e5d74870441d8c76dc3273af
76268c56febde9a77183387fbd4baabe6694e6b5 09-Mar-2017 Calin Juravle <calin@google.com> Use std::string for profile operations instead of const char

Will make things cleaner when adding secondary dex profile support.

Also, add tests to verify profile path creation.

Bug: 26719109
Test: /data/nativetest64/installd_utils_test/installd_utils_test
Change-Id: I8cb1a11cefee21f5001d2b729110696d52fc8323
/frameworks/native/cmds/installd/InstalldNativeService.cpp
c9e55782b05366bfcda975c2f5e4b80632b8c884 07-Mar-2017 Calin Juravle <calin@google.com> Merge "Installd: Clean up code related to foreign dex use" am: 62b34b57c4 am: 9d4819ff2e
am: 9181bdb57f

Change-Id: I2257654b3b73e74a92a179e1068f771e2b6af186
35c8465e31381dcc9a59655f63a91849a50294a6 04-Mar-2017 Calin Juravle <calin@google.com> Installd: Clean up code related to foreign dex use

We simplified the way we track whether or not a dex file is used by
other apps. DexManager in the framework keeps track of the data and we
no longer need file markers on disk.

Also, fix the getUserSize by considering the size of the actual profiles
not of the set of empty markers.

Test: device boots, foreign dex markers are not created anymore

Bug: 32871170
Change-Id: I9250e816a1bfa95cb10155e513ae28ebfda7ce4b
/frameworks/native/cmds/installd/InstalldNativeService.cpp
c9eab382fa4d71ade0077e0547969a47f39ecf05 25-Jan-2017 Calin Juravle <calin@google.com> [Installd] Clean up logic for secondary dex oat files

Add a new method to installd, reconcileSecondaryOdex, which checks if
the given dex files still exist and if not, deletes the oat files that
were generated for it.

Test: devices bots
adb shell cmd package reconcile-secondary-dex
com.android.google.gms (after artificially/temporarily renaming some
dex files)

Bug: 32871170

(cherry picked from commit bd9683607d391a29b1422a50f8972267e9bddc47)

Change-Id: Icb5c71b43a0e531d5be5d900149e707d0fe0a8de

Merged-In: I8465a7be9fd4e44e191ad40f7bd0f41c8b2d6f73
/frameworks/native/cmds/installd/InstalldNativeService.cpp
80a2125b85eaa0916daea7e90b016fe4f1b59924 17-Jan-2017 Calin Juravle <calin@google.com> Add installd logic for compiling secondary dex files

Secondary dex compilation takes almost the same path as primary apk
compilation.

The main difference is in the fact that for secondary dex files we
create the oat dir on the fly and execute dexoptanalyzer (the equivalent
of GetDexOptNeeded) to check if we really need to perform the
compilation.

Test: adb shell cmd package compile -f -m speed --secondary-dex com.google.android.gms
Bug: 32871170

(cherry picked from commit 42451c029b0e87990e5833daea2286bb12c21df5)

Change-Id: Ie5efe6eccc6b8c91ca7bd7c9e680aa7288d79ae8

Merged-In: I2c56d57322899968a338ccabffca575d66f8ee08
/frameworks/native/cmds/installd/InstalldNativeService.cpp
871a8f236ef2a055b9955b47a342b2c44c020ef7 22-Feb-2017 Jeff Sharkey <jsharkey@android.com> Logic for atmoic/tombstone behavior; split mode.

Flesh out logic for cache directories that request new atomic and/or
tombstone clearing behaviors. Atomic directories are considered for
deletion as a single all-or-nothing unit, and tombstone directories
truncate any removed files instead of unlinking them.

Since these behaviors can be mixed together, add local tests that
quickly verify several different permutations.

Reduce memory footprint of CacheItem objects by only storing name
and pointer to parent (instead of full path). Fix ordering bug by
switching to std::stable_sort.

Add "V2_DEFY_QUOTA" flag so we can split clearing into two distinct
phases: clearing data for apps above their quotas, and then pushing
deeper by clearing data for apps below their quotas.

Test: adb shell /data/nativetest64/installd_cache_test/installd_cache_test
Bug: 34692014, 33811826
Change-Id: I156897de1d1d1c371b2b837128b2e286bf33d40d
/frameworks/native/cmds/installd/InstalldNativeService.cpp
325b8c94a5ae47b7903ea5fe386a78466d6cb3b7 21-Feb-2017 Jeff Sharkey <jsharkey@android.com> Method to test for quota support.

Framework logic needs to change its behavior depending on how quickly
disk usage calculations can be performed.

Test: builds, boots
Bug: 34690590
Change-Id: I6139d1ea6a6084b1fd42062e2032ec9f515f6d98
/frameworks/native/cmds/installd/InstalldNativeService.cpp
5e5ad9c139c92670219b5701ff49f29e47426073 30-Jan-2017 Calin Juravle <calin@google.com> Merge "[Installd] Clean up logic for secondary dex oat files"
20e0fd800ffa9eaa9e95f3c9b84e1610068827ee 30-Jan-2017 Calin Juravle <calin@google.com> Merge "Add installd logic for compiling secondary dex files"
bd9683607d391a29b1422a50f8972267e9bddc47 25-Jan-2017 Calin Juravle <calin@google.com> [Installd] Clean up logic for secondary dex oat files

Add a new method to installd, reconcileSecondaryOdex, which checks if
the given dex files still exist and if not, deletes the oat files that
were generated for it.

Test: devices bots
adb shell cmd package reconcile-secondary-dex
com.android.google.gms (after artificially/temporarily renaming some
dex files)

Bug: 32871170

Change-Id: I8465a7be9fd4e44e191ad40f7bd0f41c8b2d6f73
/frameworks/native/cmds/installd/InstalldNativeService.cpp
42451c029b0e87990e5833daea2286bb12c21df5 17-Jan-2017 Calin Juravle <calin@google.com> Add installd logic for compiling secondary dex files

Secondary dex compilation takes almost the same path as primary apk
compilation.

The main difference is in the fact that for secondary dex files we
create the oat dir on the fly and execute dexoptanalyzer (the equivalent
of GetDexOptNeeded) to check if we really need to perform the
compilation.

Test: adb shell cmd package compile -f -m speed --secondary-dex com.google.android.gms
Bug: 32871170

Change-Id: I2c56d57322899968a338ccabffca575d66f8ee08
/frameworks/native/cmds/installd/InstalldNativeService.cpp
1b705dc20b346ccda5f5ce02287c28aaa50491e2 26-Jan-2017 Jeff Sharkey <jsharkey@android.com> Merge "Fix two bugs in external storage accounting." am: d1361ceb0c am: e4dc5d4ec5 am: 0f08cf28bd
am: 8f1b3ed54f

Change-Id: I3324121cd4f987c58cd0065200d21fbdfc98065b
64f07c5fdccbe59468f318512591a316d8eec909 26-Jan-2017 TreeHugger Robot <treehugger-gerrit@google.com> Merge "Remove from preloads/file_cache when clearing the cache"
76ddaeb2017e422de5a8d686ab62844d23d17624 26-Jan-2017 Jeff Sharkey <jsharkey@android.com> Fix two bugs in external storage accounting.

When doing a manual tree walk, we always want to inherit fts_number
from the parent fts node.

For create_data_media_package_path(), two of the arguments had been
swapped.

Test: cts-tradefed run commandAndExit cts-dev -m CtsAppSecurityHostTestCases -t android.appsecurity.cts.StorageHostTest
Bug: 27948817, 30927649
Change-Id: Ic3cbf387cf119aa82af6e380bde9d9b069f6bfb0
/frameworks/native/cmds/installd/InstalldNativeService.cpp
f3124d8fab2377d6b5ec2c7cbc775e2ab20df3ef 07-Sep-2016 Fyodor Kupolov <fkupolov@google.com> Remove from preloads/file_cache when clearing the cache

Test: pm trim-caches 50G internal
Bug: 31008665
Change-Id: If74f4d3c2d838ecb924aba84ad76293af34089bc
/frameworks/native/cmds/installd/InstalldNativeService.cpp
5c6944c35211accd783d2584fac6421dd0af5bdd 15-Dec-2015 MÃ¥rten Kongstad <marten.kongstad@sonymobile.com> installd: add command 'removeIdmap'

Add an installd command to remove an idmap file. This is the inverse of
the 'idmap' command and is intended for clean-up once an idmap file is
no longer needed because an APK was removed, etc.

This commit depends on a corresponding commit in frameworks/base (with
the same Change-Id).

Bug: 31052947
Test: run tests from 'OMS: tests for OverlayManagerService'
Change-Id: Iae19a519803f0c172b02a32faa283ef36f43863c
/frameworks/native/cmds/installd/InstalldNativeService.cpp
ea3a2b8f848a3a06667577e97ae9b0d5fb72c217 19-Jan-2017 Jeff Sharkey <jsharkey@android.com> Merge "Measure external storage using new GIDs." am: f5a21ba425 am: 973d5aeb17 am: 9715247a54
am: 3801d6668c

Change-Id: Iced41d9d4d5ccc146e044de68046b8ee48d2fc52
65344c8c20faf7544ad8323b3361108bd8d4a9d1 19-Jan-2017 Jeff Sharkey <jsharkey@android.com> Merge "First pass at updated cache clearing logic." am: 45270e18cc am: fe40750da2 am: 19c96b49b5
am: 4aaed0fc6d

Change-Id: If99a870507c953ff92ac56653fe26cd7f29597d6
7bf5b9532e192e846be3a31f28171cdba0f3c830 19-Jan-2017 Jeff Sharkey <jsharkey@android.com> Measure external storage using new GIDs.

We just defined a new range of GIDs that will be used to mark per-app
data files stored on external storage, so start measuring them.

Also measure all OBB files using another new GID that was defined.

Note that we're relying on the per-app cache GIDs to track cache
usage on *both* internal and external storage, which means that stats
and extStats won't always add up. (The framework already combines
these values together, and we didn't want to waste precious GID
space on explicitly tracking cache files on external storage.)

Test: builds, boots
Bug: 27948817, 34263266
Change-Id: Ife087df299ff8ee1a75fce1e39b4b737cf9375d5
/frameworks/native/cmds/installd/InstalldNativeService.cpp
88ddd94834dae9c5862a07a1e4432b171b2f5d9f 18-Jan-2017 Jeff Sharkey <jsharkey@android.com> First pass at updated cache clearing logic.

The old clearing algorithm is very naive and it sorts all cached files
globally by modified time. This sadly lets apps gamify the system by
setting their modified times far in the future, and it's very
expensive because it requires a global filesystem traversal to free
up even the smallest amount of data.

Instead, this CL introduces a much more fair cache clearing algorithm
that deletes files from specific UIDs based on how much cache space
that UID is using proportional to the space allocated to them. This
new design has several nice properties:

-- It uses the recently added quotactl() feature to rapidly target
the apps that are using the most space.
-- We only need to traverse the filesystem for UIDs that actively
enter the crosshairs of the clearing algorithm.
-- Disciplined apps who stay under their allocated quota will be
the last to have their cached data cleared.
-- This design can be easily adapted to support additional features
such as atomic purging and tombstones.

In summary, the new algorithm is extremely efficient when freeing up
the typical small-to-medium amounts of disk space, and is only
moderately less efficient than the old algorithm when forced to clear
all cached data.

Test: builds, boots, clearing strategy looks sane
Bug: 33965858
Change-Id: I66f95089cb33f1add3f31fcf1082ab2469870fda
/frameworks/native/cmds/installd/InstalldNativeService.cpp
7d9483a508029826b69b3bac437754d7a9d993f6 18-Jan-2017 Jeff Sharkey <jsharkey@android.com> Merge "Add systrace details to storage measurements." am: a693382d94 am: b0858a472e am: 6248cde3b3
am: 64a2155e1d

Change-Id: I93d3eba418b98081eca6a2b0cabda58db5649057
466459b68da52edf20962c3c80e2fe0325569c18 17-Jan-2017 Jeff Sharkey <jsharkey@android.com> Add systrace details to storage measurements.

Can be used to find bottlenecks, and confirm that quota stuff is
being used when expected.

Test: builds, boots
Bug: 34341727
Change-Id: Ifefc155e5d3b0c74245e0d287e84df699ce09d16
/frameworks/native/cmds/installd/InstalldNativeService.cpp
28702bb83a5d300d7ca949b49587a3300da60a3f 17-Jan-2017 Jeff Sharkey <jsharkey@android.com> Merge "Handle devices without quota, speed up lookup." am: 36ba03a248 am: 411ae16a91 am: 9eeb77f0dd
am: bf8bc22859

Change-Id: I94e86de78b8d84f0d3a23ff627b0808812a47ded
d972d4206db6e42cc574586a1af2f94264a04519 17-Jan-2017 Jeff Sharkey <jsharkey@android.com> Wrap all installd operations in lock. am: 7a9059e8ed am: b940bdae84 am: 0c4cc3ebf1
am: 53c09d3b56

Change-Id: I97bb0b683c0033815009e6d9376aa18d1f503ebc
eca1bff0ce7709830816418107c8f444afdd33f5 17-Jan-2017 Jeff Sharkey <jsharkey@android.com> Mask st_mode before comparing it. am: 24ef15bd00 am: 0e636867b5 am: fc7bb04d26
am: d7025b423a

Change-Id: I19192f9d88fc022a4c252035ee6360e7a897b05e
66b1a12e7a120f85042669bb58f1db864616b506 17-Jan-2017 Jeff Sharkey <jsharkey@android.com> Handle devices without quota, speed up lookup.

Start tracking which block devices have quota support, and gracefully
clear FLAG_USE_QUOTA when no support is present.

Also build a cached map of mounted volumes that support quota, which
halves the average quota calculation speed from 0.70ms to 0.35ms,
since we're no longer parsing procfs every time.

Test: builds, boots, common operations work
Bug: 34249218
Change-Id: Ie791df7801b67495331f3eea256c018860c9b4f6
/frameworks/native/cmds/installd/InstalldNativeService.cpp
7a9059e8ed297a8e025c843b632c7c8571682b46 17-Jan-2017 Jeff Sharkey <jsharkey@android.com> Wrap all installd operations in lock.

Historically installd has a single lock up in the framework, but
we're starting to call in from multiple locations, so installd needs
to protect itself.

This specifically applies to any disk measurement requests, which
use the optimized fts(3) mode which uses chdir() internally.

Test: builds, boots, common operations work
Bug: 34330051
Change-Id: Ic746ea890ebcc7b2bf923404874b04dbc19d42ad
/frameworks/native/cmds/installd/InstalldNativeService.cpp
24ef15bd009204b73ebf0e94ba3d868177a5f44c 13-Jan-2017 Jeff Sharkey <jsharkey@android.com> Mask st_mode before comparing it.

Otherwise it will never match the requested mode_t.

Test: builds, boots, skips when matching
Bug: 34201111
Change-Id: Iebaa8063a72eb6eaf2e773c1d35204617d0f16b8
/frameworks/native/cmds/installd/InstalldNativeService.cpp
e0ea99cc4ba89452b9b6b1baa33bddcb11f3351a 12-Jan-2017 Mark Salyzyn <salyzyn@google.com> resolve merge conflicts of a2c2d11f9 to master

Test: compile
Bug: 26552300
Bug: 31289077
Change-Id: I84d0ffa430df3beca0dcc6ed205ce220d7be1f87
7823e124e00576e20e47ec717cbe8bc89f0f2bf2 29-Sep-2016 Mark Salyzyn <salyzyn@google.com> Replace cutils/log.h and rationalize log/log.h and android/log.h

- cutils/log.h treat as deprecated and lead by example
- android/log.h to be used instead of log/log.h if possible
- add system includes that are assumed as side effects
- define LOG_TAG first

Test: compile
Bug: 31289077
Bug: 30465923
Change-Id: I8d99b24c333578c9b5aa9f2a01324bd0bba268dd
/frameworks/native/cmds/installd/InstalldNativeService.cpp
78ef6e13966b830d111f18702948d77cfa2634a7 10-Jan-2017 Jeff Sharkey <jsharkey@android.com> Merge "Methods to calculate user and external disk usage." am: 4d9746d7a3 am: 15e80e0436 am: e5281b7f9b
am: 4e4e8ab3a4

Change-Id: I954e87563f1a8222be418221657e730ffda51814
df2d754b38796e0c49c70e0a67f7d383e3079ff2 07-Jan-2017 Jeff Sharkey <jsharkey@android.com> Methods to calculate user and external disk usage.

Add method to calculate user disk usage, which will be faster than
making a Binder call for every single appId under a user. Add method
to calculate external disk usage, which uses file extensions to track
usage with "audio", "video", and "images" categories.

Add script to generate optimized file extension matcher logic.

Start measuring internal and external storage space separately; new
GIDs are coming in a future CL. Pass down all package names, inodes,
and code paths, since shared UIDs host more than one.

Test: builds, boots, stats are consistent
Bug: 27948817, 32206268
Change-Id: Icb9843ac5159e0e5f1503d9b64e0bcae407e1a5d
/frameworks/native/cmds/installd/InstalldNativeService.cpp
2fc82a9b9319753c72acbbd7cbc5ce535e1a173b 04-Jan-2017 Jeff Sharkey <jsharkey@android.com> Merge "Reference profiles use shared GID for user 0." am: 226f77d87a am: ac437f476f am: 93d4073bc9
am: cdeb04edca

Change-Id: I876e3cbb360117aac41ab0f6601e65888d8b4e9f
adddd9876e145879169369913b5afe469b17c02d 03-Jan-2017 Jeff Sharkey <jsharkey@android.com> Reference profiles use shared GID for user 0.

Test: builds, boots
Bug: 33977235
Change-Id: I4de3852b3f230a6dc874a630e3c1b9fff2d83169
/frameworks/native/cmds/installd/InstalldNativeService.cpp
018343e4f29e809db3bc1b45682797c9568c315d 03-Jan-2017 Jeff Sharkey <jsharkey@android.com> Merge "Offer to measure disk stats using quotas." am: 8aed997c14 am: 5fcfd07918 am: c7cd1acccd
am: 92e6bc636a

Change-Id: Ifb9a2248fe048d0be91499169d3fd201dbb4e4f7
3dfae0c008576c873c4039bb4c2e54a6adf3720a 13-Dec-2016 Jeff Sharkey <jsharkey@android.com> Offer to measure disk stats using quotas.

Now we're getting somewhere! This CL starts using quotactl() to read
UID/GID quota statistics when called with FLAG_USE_QUOTA, otherwise
it continues using the old heavy-weight traversal for calculation.

We now set the recently defined per-app GID used for identifying
cached data, and we use the sticky GID bit on cache directories to
ensure that newly created data inherits the GID for tracking
purposes. For any existing apps during an upgrade, an initial
bootstrapping case will recursively set this new GID.

This change also shuffles around a bunch of the tedious manual
accounting logic so that we exactly match the new quota statistics.

Test: builds, boots, quota stats match manual stats
Bug: 27948817
Change-Id: I8512c3193ce698f197a3f446e625d6a1c74e7649
/frameworks/native/cmds/installd/InstalldNativeService.cpp
74b37b98cd39e6592953da8a3aad2700e23a872f 21-Dec-2016 Jeff Sharkey <jsharkey@android.com> Merge "Only log restorecon changes for existing dirs." am: ee080ea229 am: ed8915cc2c am: 4500fde965
am: 1e6f01483a

Change-Id: I11a036ab1c2e6d7e854474205da8fced743d4cb1
35b83dff3de2b5fe781a060566ee09266b3c2617 21-Dec-2016 Jeff Sharkey <jsharkey@android.com> Only log restorecon changes for existing dirs.

If we just created an app private data directory, we're always going
to be relabeling it, so don't log in that case. Continue logging if
something caused us to relabel an existing app data directory.

Test: builds, boots, only logs for existing dirs
Bug: 31958779
Change-Id: Id03f7c18aed191a4c7fb9dcf9380136c93798a4b
/frameworks/native/cmds/installd/InstalldNativeService.cpp
f4237c11d2daa4fdce728dc607b70563bdb8d7bb 20-Dec-2016 Jeff Sharkey <jsharkey@android.com> Merge "Remove ~1/3 of the installd calls at boot." am: ed04f3c7e5 am: 3f79c64976 am: 32aaf887ef
am: e5b37140f6

Change-Id: I4fda5d7b7f2d8f19d4b3ec6ea4a6ab31708f1c16
36a900a0db526d63e899ed5d5b0a463ad7f9236e 20-Dec-2016 Jeff Sharkey <jsharkey@android.com> Remove ~1/3 of the installd calls at boot.

When preparing CE storage for an app, we always perform a second call
to extract any newly created CE directory inode. Let's simplify this
and just return the inode number from the createAppData() call.

Test: builds, boots, reads CE inodes after wipe
Bug: 33463450
Change-Id: I68485e4b9b49e9da7b3ac7c66f50c6abc750b7e3
/frameworks/native/cmds/installd/InstalldNativeService.cpp
451c17d82077cca61d0d04097961e99d84f54146 16-Dec-2016 Jeff Sharkey <jsharkey@android.com> Merge "Handle invalid shared GIDs." am: 788bdeae78 am: 477601a9ff am: 4be16dccfe
am: 2d9bf64d9f

Change-Id: I3fb92764c2a0271483e31646c63fbabd1f1c58c6
2e6dc9baaaf5d50b42ce36435e13c2559808b837 15-Dec-2016 Jeff Sharkey <jsharkey@android.com> Handle invalid shared GIDs.

Shared GIDs are only available inside a range from 50000-59999, so
system UIDs (outside the normal app range) don't have a valid GID to
use. The best we can do is skip the profile setup in those cases.

Test: builds
Bug: 27948817
Change-Id: I8c38a0e3631eafd07e96da3e37e641b3cce195a6
/frameworks/native/cmds/installd/InstalldNativeService.cpp
0ac72cff4dd86cc7f4e4ef0d30f1dc7651ac9dbb 13-Dec-2016 Jeff Sharkey <jsharkey@android.com> resolve merge conflicts of d77e675 to master

Change-Id: I211db442e0081ee51f889b1fe788dad08889c2ec
7da5fc78535e3fd1581b3594b694a454a5badfe0 12-Dec-2016 Jeff Sharkey <jsharkey@android.com> Merge "Minor shuffling of installd error reporting." am: 991707a167 am: 405dbf06b5 am: 18148ac756
am: bbb06f3b6a

Change-Id: Ifd2d7c59b6698d2c6dfb35e1a99aa51df4bce9df
90aff26f0135379db19432ae90c40c0831ba5954 12-Dec-2016 Jeff Sharkey <jsharkey@android.com> Split all dexopt-related code into dexopt.cpp.

Only a few minor changes to return bool instead of binder::Status.

Test: builds, boots, apps install/uninstall fine
Change-Id: I538024c0e9da8ca2c0983a6e779c34bd40752053
/frameworks/native/cmds/installd/InstalldNativeService.cpp
7be5ead81aa4d3b2536556c762bab03445a98155 12-Dec-2016 Jeff Sharkey <jsharkey@android.com> Minor shuffling of installd error reporting.

We actually want to early-return when creating user data fails, and
we want to return an exception when getAppDataInode() is called with
unsupported flags.

Test: builds, boots, apps install/uninstall fine
Bug: 13758960, 30944031
Change-Id: I69816275a0be7a73a8ae7f3d90f2cdea80d8ebfc
/frameworks/native/cmds/installd/InstalldNativeService.cpp
ae01df2851d0f188f2459eccefca926cf5ccf9d0 12-Dec-2016 Jeff Sharkey <jsharkey@android.com> Merge "Return real error strings from installd." am: 8f3ce2cd66 am: 6bd17e8e69 am: 7231e852a2
am: ee0d2ed9fa

Change-Id: Ia9f0bf02967d66e3bd39dc0e51cffd234a97c108
423e746ac7d4b3a3d772dd0e01bdb9fd6029d439 10-Dec-2016 Jeff Sharkey <jsharkey@android.com> Return real error strings from installd.

Now that we've moved installd to Binder, we can return nice detailed
error strings explaining why a call failed. This is particularly
valuable when we record the error message into the PackageManager
persistent log, because up until now those errors were limited to
an unhelpful "installd returned -1" message.

Also perform uniform enforcement of all incoming package name and
UUID arguments.

Test: builds, boots, apps install/uninstall fine
Bug: 13758960, 30944031
Change-Id: Ic1f65ce8c10b1329e01d6a49d72cafa879c4d8bc
/frameworks/native/cmds/installd/InstalldNativeService.cpp
0f70df22c1b950aeacb6edd3d2396a6439674643 11-Dec-2016 Jeff Sharkey <jsharkey@android.com> Merge "Filename refactoring, remove DEXOPT_OTA flag." am: 68603971f6 am: 4f1d7392b8 am: 00ee711d09
am: 671ccdfc37

Change-Id: I9b16e7c1dfe6650b2a63ec0e45e8818b3af228be
f3e30b936ef103dc0f3d8697e0f86ba82b49609e 10-Dec-2016 Jeff Sharkey <jsharkey@android.com> Filename refactoring, remove DEXOPT_OTA flag.

Test: builds, boots
Bug: 13758960, 30944031
Change-Id: Ib60899fc70b12f76fad9dfad596b513777752703
/frameworks/native/cmds/installd/InstalldNativeService.cpp
0274c977e3b10b52936fd5b2eb464857f0ca6358 06-Dec-2016 Jeff Sharkey <jsharkey@android.com> Shuffle installd Binder implementation around.

Instead of writing wrapper methods in InstalldNativeService.cpp,
this change shifts the Binder implementation over into the existing
commands.cpp file. This will let us migrate all methods over, and
then perform one final file rename to InstalldNativeService.cpp.

The downside of this approach is that we no longer have a giant kill
switch to quickly switch back.

Moves the moveCompleteApp() method over to Binder.

Test: builds, boots, apps install fine
Bug: 13758960, 30944031
Change-Id: I53550e05bc2b65155c3de18424f67b1a95450a6f
/frameworks/native/cmds/installd/InstalldNativeService.cpp
9087400f3c82b97aa17a74329c7e65c0a5ff4062 05-Dec-2016 Jeff Sharkey <jsharkey@android.com> Initial Binder interface for installd.

Define an AIDL for installd, starting with a single method. Publish
that interface for the system to start using. We'll circle back and
migrate more methods in future CLs.

Migrating installd to Binder will allow us to get rid of one of
the few lingering text-based command protocols, improving system
maintainability and security.

Test: builds, boots
Bug: 13758960, 30944031
Change-Id: Icdb5320082567e0355d8f76f413f01cfacf0fb99
/frameworks/native/cmds/installd/InstalldNativeService.cpp