| bf9b0180f3e75b710d1f419fff343885ee1b3880 |
|
08-Dec-2016 |
Christopher Ferris <cferris@google.com> |
Update to jemalloc 4.4.0.
Merge remote-tracking branch 'aosp/upstream-new' into upgrade
Includes regenerating the necessary files.
Bug: 33321361
Test: Built for x86_64/arm64. Built the no tcache and tcache enabled
Test: configs (nexus 9/nexus 6p). Compared the before after running
Test: all dumps through the memory replay and verified no unexpected
Test: increases. Ran bionic unit tests in both configs, 32 bit and 64 bit
Test: variants. Ran the jemalloc unit tests in both configs.
Change-Id: I2e8f3305cd1717c7efced69718fff90797f21068
|
| 5c77af98b16a0f5b15bc807f2b323a91fe2a048b |
|
15-Nov-2016 |
Jason Evans <jasone@canonware.com> |
Add extent serial numbers.
Add extent serial numbers and use them where appropriate as a sort key
that is higher priority than address, so that the allocation policy
prefers older extents.
This resolves #147.
/external/jemalloc/src/huge.c
|
| fb1f094f163a3bf15d8958ba845e83559c0e6dfe |
|
08-Nov-2016 |
Christopher Ferris <cferris@google.com> |
Merge remote-tracking branch 'aosp/upstream-new' into fix
Included in this change are all of the updated generated files.
Bug: 32673024
Test: Built the angler build (normal config) and the volantis build
Test: (svelte config). Ran memory_replay 32 bit and 64 bit on both
Test: platforms before and after and verified results are similar.
Test: Ran bionic unit tests and jemalloc unit tests.
Test: Verified that two jemalloc unit test failures are due to
Test: Android extension that puts all large chunks on arena 0.
Change-Id: I12428bdbe15f51383489c9a1d72d687499fff01b
|
| 962a2979e353f876f3725417179f201e671d9dbb |
|
21-Oct-2016 |
Jason Evans <jasone@canonware.com> |
Do not (recursively) allocate within tsd_fetch().
Refactor tsd so that tsdn_fetch() does not trigger allocation, since
allocation could cause infinite recursion.
This resolves #458.
/external/jemalloc/src/huge.c
|
| e2bcf037d445a84a71c7997670819ebd0a893b4a |
|
13-Oct-2016 |
Jason Evans <jasone@canonware.com> |
Make dss operations lockless.
Rather than protecting dss operations with a mutex, use atomic
operations. This has negligible impact on synchronization overhead
during typical dss allocation, but is a substantial improvement for
chunk_in_dss() and the newly added chunk_dss_mergeable(), which can be
called multiple times during chunk deallocations.
This change also has the advantage of avoiding tsd in deallocation paths
associated with purging, which resolves potential deadlocks during
thread exit due to attempted tsd resurrection.
This resolves #425.
/external/jemalloc/src/huge.c
|
| 5ab62bd1b55ef0804dbea305f8697bf9d247f3a4 |
|
14-Jun-2016 |
Christopher Ferris <cferris@google.com> |
Merge remote-tracking branch 'aosp/upstream-release' into fix
Bug: 28860984
Change-Id: I9eaf67f53f9872177d068d660d1e051cecdc82a0
|
| a7fdcc8b09d197d64d1772d1e8025add44f48f0a |
|
18-May-2016 |
Jason Evans <jasone@canonware.com> |
Fix opt_zero-triggered in-place huge reallocation zeroing.
Fix huge_ralloc_no_move_expand() to update the extent's zeroed attribute
based on the intersection of the previous value and that of the newly
merged trailing extent.
/external/jemalloc/src/huge.c
|
| 1c35f63797d63a1d08507ea724ec5d8898e8d76d |
|
12-May-2016 |
Jason Evans <jasone@canonware.com> |
Guard tsdn_tsd() call with tsdn_null() check.
/external/jemalloc/src/huge.c
|
| 7790a0ba403b02bcb8804534c8120d605b4dc5f4 |
|
11-May-2016 |
Jason Evans <jasone@canonware.com> |
Fix chunk accounting related to triggering gdump profiles.
Fix in place huge reallocation to update the chunk counters that are
used for triggering gdump profiles.
/external/jemalloc/src/huge.c
|
| c1e00ef2a6442d1d047950247c757821560db329 |
|
11-May-2016 |
Jason Evans <jasone@canonware.com> |
Resolve bootstrapping issues when embedded in FreeBSD libc.
b2c0d6322d2307458ae2b28545f8a5c9903d7ef5 (Add witness, a simple online
locking validator.) caused a broad propagation of tsd throughout the
internal API, but tsd_fetch() was designed to fail prior to tsd
bootstrapping. Fix this by splitting tsd_t into non-nullable tsd_t and
nullable tsdn_t, and modifying all internal APIs that do not critically
rely on tsd to take nullable pointers. Furthermore, add the
tsd_booted_get() function so that tsdn_fetch() can probe whether tsd
bootstrapping is complete and return NULL if not. All dangerous
conversions of nullable pointers are tsdn_tsd() calls that assert-fail
on invalid conversion.
/external/jemalloc/src/huge.c
|
| 3ef51d7f733ac6432e80fa902a779ab5b98d74f6 |
|
06-May-2016 |
Jason Evans <jasone@canonware.com> |
Optimize the fast paths of calloc() and [m,d,sd]allocx().
This is a broader application of optimizations to malloc() and free() in
f4a0f32d340985de477bbe329ecdaecd69ed1055 (Fast-path improvement:
reduce # of branches and unnecessary operations.).
This resolves #321.
/external/jemalloc/src/huge.c
|
| 90827a3f3ef2099dcd480d542aacc9f44a0787e8 |
|
04-May-2016 |
Jason Evans <jasone@canonware.com> |
Fix huge_palloc() regression.
Split arena_choose() into arena_[i]choose() and use arena_ichoose() for
arena lookup during internal allocation. This fixes huge_palloc() so
that it always succeeds during extent node allocation.
This regression was introduced by
66cd953514a18477eb49732e40d5c2ab5f1b12c5 (Do not allocate metadata via
non-auto arenas, nor tcaches.).
/external/jemalloc/src/huge.c
|
| 66cd953514a18477eb49732e40d5c2ab5f1b12c5 |
|
22-Apr-2016 |
Jason Evans <jasone@canonware.com> |
Do not allocate metadata via non-auto arenas, nor tcaches.
This assures that all internally allocated metadata come from the
first opt_narenas arenas, i.e. the automatically multiplexed arenas.
/external/jemalloc/src/huge.c
|
| b2c0d6322d2307458ae2b28545f8a5c9903d7ef5 |
|
14-Apr-2016 |
Jason Evans <jasone@canonware.com> |
Add witness, a simple online locking validator.
This resolves #358.
/external/jemalloc/src/huge.c
|
| a82070ef5fc3aa81fda43086cdcc22bfa826b894 |
|
28-Mar-2016 |
Chris Peterson <cpeterson@mozilla.com> |
Add JEMALLOC_ALLOC_JUNK and JEMALLOC_FREE_JUNK macros
Replace hardcoded 0xa5 and 0x5a junk values with JEMALLOC_ALLOC_JUNK and
JEMALLOC_FREE_JUNK macros, respectively.
/external/jemalloc/src/huge.c
|
| 473a1853074261bb265060e04deeb8efb0fe6fac |
|
03-Mar-2016 |
Christopher Ferris <cferris@google.com> |
Update generated files for 4.1.0 update.
Removed the hack to do an always purge. Instead use the new decay
purging mechanism, but set the decay timeout to 0 so it always purges
without the need to change the code.
Added back the a0get function to use for huge chunk allocation patch
we use for android.
Bug: 27408522
Change-Id: Ic75c4a03b362fd0f9654f0803fb9624ef82d417e
/external/jemalloc/src/huge.c
|
| fb9c9c8d5230956caa48501dad4fde4b90e00319 |
|
02-Mar-2016 |
Christopher Ferris <cferris@google.com> |
Merge remote-tracking branch 'aosp/upstream-dev' into merge
Bug: 26807329
Change-Id: I25b32740420ffdfd62e32ddcd3fdc54a5010e4cf
|
| 0c516a00c4cb28cff55ce0995f756b5aae074c9e |
|
26-Feb-2016 |
Jason Evans <je@fb.com> |
Make *allocx() size class overflow behavior defined.
Limit supported size and alignment to HUGE_MAXCLASS, which in turn is
now limited to be less than PTRDIFF_MAX.
This resolves #278 and #295.
/external/jemalloc/src/huge.c
|
| 243f7a0508bb014c2a7bf592c466a923911db234 |
|
20-Feb-2016 |
Jason Evans <jasone@canonware.com> |
Implement decay-based unused dirty page purging.
This is an alternative to the existing ratio-based unused dirty page
purging, and is intended to eventually become the sole purging
mechanism.
Add mallctls:
- opt.purge
- opt.decay_time
- arena.<i>.decay
- arena.<i>.decay_time
- arenas.decay_time
- stats.arenas.<i>.decay_time
This resolves #325.
/external/jemalloc/src/huge.c
|
| 6be84e0ca9e04c8ebe7fc10c5e8eef7f4ac15a12 |
|
02-Feb-2016 |
Christopher Ferris <cferris@google.com> |
Merge remote-tracking branch 'aosp/upstream-dev' into merge
Bug: 24264290
|
| f4a0f32d340985de477bbe329ecdaecd69ed1055 |
|
27-Oct-2015 |
Qi Wang <interwq@gwu.edu> |
Fast-path improvement: reduce # of branches and unnecessary operations.
- Combine multiple runtime branches into a single malloc_slow check.
- Avoid calling arena_choose / size2index / index2size on fast path.
- A few micro optimizations.
/external/jemalloc/src/huge.c
|
| d260f442ce693de4351229027b37b3293fcbfd7d |
|
25-Sep-2015 |
Jason Evans <jasone@canonware.com> |
Fix xallocx(..., MALLOCX_ZERO) bugs.
Zero all trailing bytes of large allocations when
--enable-cache-oblivious configure option is enabled. This regression
was introduced by 8a03cf039cd06f9fa6972711195055d865673966 (Implement
cache index randomization for large allocations.).
Zero trailing bytes of huge allocations when resizing from/to a size
class that is not a multiple of the chunk size.
/external/jemalloc/src/huge.c
|
| 708ed79834fc3b8e5b14dbb0128a0ebfce63a38f |
|
15-Sep-2015 |
Jason Evans <jasone@canonware.com> |
Resolve an unsupported special case in arena_prof_tctx_set().
Add arena_prof_tctx_reset() and use it instead of arena_prof_tctx_set()
when resetting the tctx pointer during reallocation, which happens
whenever an originally sampled reallocated object is not sampled during
reallocation.
This regression was introduced by
594c759f37c301d0245dc2accf4d4aaf9d202819 (Optimize
arena_prof_tctx_set().)
/external/jemalloc/src/huge.c
|
| 560a4e1e01d3733c2f107cdb3cc3580f3ed84442 |
|
12-Sep-2015 |
Jason Evans <jasone@canonware.com> |
Fix xallocx() bugs.
Fix xallocx() bugs related to the 'extra' parameter when specified as
non-zero.
/external/jemalloc/src/huge.c
|
| 36629d29adb4c02683a890d7590ac38ce5c29d92 |
|
09-Sep-2015 |
Christopher Ferris <cferris@google.com> |
Merge remote-tracking branch 'aosp/upstream-dev' into merge
Change-Id: I7d6244c2eec08c8aac45d3e24b6d50627f05b0a9
|
| 4a2a3c9a6e5f8fda3536f48095d68ddaa13cd977 |
|
28-Aug-2015 |
Mike Hommey <mh@glandium.org> |
Don't purge junk filled chunks when shrinking huge allocations
When junk filling is enabled, shrinking an allocation fills the bytes
that were previously allocated but now aren't. Purging the chunk before
doing that is just a waste of time.
This resolves #260.
/external/jemalloc/src/huge.c
|
| 6d8075f1e6e72ee274832f1164c164f713788e34 |
|
28-Aug-2015 |
Mike Hommey <mh@glandium.org> |
Fix chunk purge hook calls for in-place huge shrinking reallocation.
Fix chunk purge hook calls for in-place huge shrinking reallocation to
specify the old chunk size rather than the new chunk size. This bug
caused no correctness issues for the default chunk purge function, but
was visible to custom functions set via the "arena.<i>.chunk_hooks"
mallctl.
This resolves #264.
/external/jemalloc/src/huge.c
|
| 6ff2aaf17f95465cde748dda006e9e17e04a3fde |
|
10-Aug-2015 |
Christopher Ferris <cferris@google.com> |
On 32 bit systems, force huge allocs on arena 0.
The new version of jemalloc has moved all huge allocations from
a single cache, to a per arena cache. This can result in virtual
address space exhaustion when someone has a pattern where they
allocate huge allocations from new threads all of the time.
Bug: 22172059
(cherry picked from commit dcfc29b2aa1ee53fe4b40368d02f80ca4bddeabc)
Change-Id: I0f380c1295a49eb76390cf5f3c9285e685122706
/external/jemalloc/src/huge.c
|
| 8fadb1a8c2d0219aded566bc5fac7d29cff9bb67 |
|
04-Aug-2015 |
Jason Evans <jasone@canonware.com> |
Implement chunk hook support for page run commit/decommit.
Cascade from decommit to purge when purging unused dirty pages, so that
it is possible to decommit cleaned memory rather than just purging. For
non-Windows debug builds, decommit runs rather than purging them, since
this causes access of deallocated runs to segfault.
This resolves #251.
/external/jemalloc/src/huge.c
|
| b49a334a645b854dbb1649f15c38d646fee66738 |
|
28-Jul-2015 |
Jason Evans <je@fb.com> |
Generalize chunk management hooks.
Add the "arena.<i>.chunk_hooks" mallctl, which replaces and expands on
the "arena.<i>.chunk.{alloc,dalloc,purge}" mallctls. The chunk hooks
allow control over chunk allocation/deallocation, decommit/commit,
purging, and splitting/merging, such that the application can rely on
jemalloc's internal chunk caching and retaining functionality, yet
implement a variety of chunk management mechanisms and policies.
Merge the chunks_[sz]ad_{mmap,dss} red-black trees into
chunks_[sz]ad_retained. This slightly reduces how hard jemalloc tries
to honor the dss precedence setting; prior to this change the precedence
setting was also consulted when recycling chunks.
Fix chunk purging. Don't purge chunks in arena_purge_stashed(); instead
deallocate them in arena_unstash_purged(), so that the dirty memory
linkage remains valid until after the last time it is used.
This resolves #176 and #201.
/external/jemalloc/src/huge.c
|
| d059b9d6a1ac3e7f834260ba001bf0d1599fb0bf |
|
25-Jul-2015 |
Jason Evans <jasone@canonware.com> |
Implement support for non-coalescing maps on MinGW.
- Do not reallocate huge objects in place if the number of backing
chunks would change.
- Do not cache multi-chunk mappings.
This resolves #213.
/external/jemalloc/src/huge.c
|
| 40cbd30d508b0d4e6462f5c36ffdbf6c1f29da22 |
|
25-Jul-2015 |
Jason Evans <jasone@canonware.com> |
Fix huge_ralloc_no_move() to succeed more often.
Fix huge_ralloc_no_move() to succeed if an allocation request results in
the same usable size as the existing allocation, even if the request
size is smaller than the usable size. This bug did not cause
correctness issues, but it could cause unnecessary moves during
reallocation.
/external/jemalloc/src/huge.c
|
| 87ccb5554769d915338b9a980d36359a5c6ec3fa |
|
24-Jul-2015 |
Jason Evans <jasone@canonware.com> |
Fix huge_palloc() to handle size rather than usize input.
huge_ralloc() passes a size that may not be precisely a size class, so
make huge_palloc() handle the more general case of a size input rather
than usize.
This regression appears to have been introduced by the addition of
in-place huge reallocation; as such it was never incorporated into a
release.
/external/jemalloc/src/huge.c
|
| 831d585c1dad2e6e39804db2a223064d295cb4fe |
|
16-May-2015 |
Jason Evans <jasone@canonware.com> |
Avoid atomic operations for dependent rtree reads.
Bug: 21326736
(cherry picked from commit fd5f9e43c35b39740e218fececbb70d929546bb0)
Change-Id: Ie3b3d15a7bd18cb48b05b71f2df2b238517f019c
/external/jemalloc/src/huge.c
|
| fd5f9e43c35b39740e218fececbb70d929546bb0 |
|
16-May-2015 |
Jason Evans <jasone@canonware.com> |
Avoid atomic operations for dependent rtree reads.
/external/jemalloc/src/huge.c
|
| 83e5767ee9a8c68150cca06ae0d27a13ba4fcaf8 |
|
22-Apr-2015 |
Christopher Ferris <cferris@google.com> |
Revert "Revert "Merge remote-tracking branch 'aosp/upstream-dev' into merge""
This reverts commit 75929a97332565c3b987986f35652b6d5d275d3c.
The original failure this was reverted for seems to have been a bug somewhere else.
Change-Id: Ib29ba03b1b967f940dc19eceac2aa1d2923be1eb
/external/jemalloc/src/huge.c
|
| 75929a97332565c3b987986f35652b6d5d275d3c |
|
16-Apr-2015 |
Nicolas Geoffray <ngeoffray@google.com> |
Revert "Merge remote-tracking branch 'aosp/upstream-dev' into merge"
Revert due to random ART crashes seen.
This reverts commit 5b5d97b42e84c2ac417271c3fab6fc282496a335.
Change-Id: I62a784301fded7ee853b182d172be46bb32bded7
/external/jemalloc/src/huge.c
|
| 65db63cf3f0c5dd5126a1b3786756486eaf931ba |
|
26-Mar-2015 |
Jason Evans <je@fb.com> |
Fix in-place shrinking huge reallocation purging bugs.
Fix the shrinking case of huge_ralloc_no_move_similar() to purge the
correct number of pages, at the correct offset. This regression was
introduced by 8d6a3e8321a7767cb2ca0930b85d5d488a8cc659 (Implement
dynamic per arena control over dirty page purging.).
Fix huge_ralloc_no_move_shrink() to purge the correct number of pages.
This bug was introduced by 9673983443a0782d975fbcb5d8457cfd411b8b56
(Purge/zero sub-chunk huge allocations as necessary.).
/external/jemalloc/src/huge.c
|
| 8d6a3e8321a7767cb2ca0930b85d5d488a8cc659 |
|
19-Mar-2015 |
Jason Evans <je@fb.com> |
Implement dynamic per arena control over dirty page purging.
Add mallctls:
- arenas.lg_dirty_mult is initialized via opt.lg_dirty_mult, and can be
modified to change the initial lg_dirty_mult setting for newly created
arenas.
- arena.<i>.lg_dirty_mult controls an individual arena's dirty page
purging threshold, and synchronously triggers any purging that may be
necessary to maintain the constraint.
- arena.<i>.chunk.purge allows the per arena dirty page purging function
to be replaced.
This resolves #93.
/external/jemalloc/src/huge.c
|
| a4e1888d1a12d864f42350f2859e33eb3a0033f2 |
|
18-Feb-2015 |
Jason Evans <jasone@canonware.com> |
Simplify extent_node_t and add extent_node_init().
/external/jemalloc/src/huge.c
|
| ee41ad409a43d12900a5a3108f6c14f84e4eb0eb |
|
16-Feb-2015 |
Jason Evans <jasone@canonware.com> |
Integrate whole chunks into unused dirty page purging machinery.
Extend per arena unused dirty page purging to manage unused dirty chunks
in aaddtion to unused dirty runs. Rather than immediately unmapping
deallocated chunks (or purging them in the --disable-munmap case), store
them in a separate set of trees, chunks_[sz]ad_dirty. Preferrentially
allocate dirty chunks. When excessive unused dirty pages accumulate,
purge runs and chunks in ingegrated LRU order (and unmap chunks in the
--enable-munmap case).
Refactor extent_node_t to provide accessor functions.
/external/jemalloc/src/huge.c
|
| 2195ba4e1f8f262b7e6586106d90f4dc0aea7630 |
|
16-Feb-2015 |
Jason Evans <jasone@canonware.com> |
Normalize *_link and link_* fields to all be *_link.
/external/jemalloc/src/huge.c
|
| cbf3a6d70371d2390b8b0e76814e04cc6088002c |
|
11-Feb-2015 |
Jason Evans <jasone@canonware.com> |
Move centralized chunk management into arenas.
Migrate all centralized data structures related to huge allocations and
recyclable chunks into arena_t, so that each arena can manage huge
allocations and recyclable virtual memory completely independently of
other arenas.
Add chunk node caching to arenas, in order to avoid contention on the
base allocator.
Use chunks_rtree to look up huge allocations rather than a red-black
tree. Maintain a per arena unsorted list of huge allocations (which
will be needed to enumerate huge allocations during arena reset).
Remove the --enable-ivsalloc option, make ivsalloc() always available,
and use it for size queries if --enable-debug is enabled. The only
practical implications to this removal are that 1) ivsalloc() is now
always available during live debugging (and the underlying radix tree is
available during core-based debugging), and 2) size query validation can
no longer be enabled independent of --enable-debug.
Remove the stats.chunks.{current,total,high} mallctls, and replace their
underlying statistics with simpler atomically updated counters used
exclusively for gdump triggering. These statistics are no longer very
useful because each arena manages chunks independently, and per arena
statistics provide similar information.
Simplify chunk synchronization code, now that base chunk allocation
cannot cause recursive lock acquisition.
/external/jemalloc/src/huge.c
|
| 1cb181ed632e7573fb4eab194e4d216867222d27 |
|
30-Jan-2015 |
Jason Evans <je@fb.com> |
Implement explicit tcache support.
Add the MALLOCX_TCACHE() and MALLOCX_TCACHE_NONE macros, which can be
used in conjunction with the *allocx() API.
Add the tcache.create, tcache.flush, and tcache.destroy mallctls.
This resolves #145.
/external/jemalloc/src/huge.c
|
| eee27b2a38d6bb741d9de5e028d5b23e2f4ec4cf |
|
25-Jan-2015 |
Sébastien Marie <semarie@users.noreply.github.com> |
huge_node_locked don't have to unlock huge_mtx
in src/huge.c, after each call of huge_node_locked(), huge_mtx is
already unlocked. don't unlock it twice (it is a undefined behaviour).
/external/jemalloc/src/huge.c
|
| 4581b97809e7e545c38b996870a4e7284a620bc5 |
|
27-Nov-2014 |
Jason Evans <je@fb.com> |
Implement metadata statistics.
There are three categories of metadata:
- Base allocations are used for bootstrap-sensitive internal allocator
data structures.
- Arena chunk headers comprise pages which track the states of the
non-metadata pages.
- Internal allocations differ from application-originated allocations
in that they are for internal use, and that they are omitted from heap
profiles.
The metadata statistics comprise the metadata categories as follows:
- stats.metadata: All metadata -- base + arena chunk headers + internal
allocations.
- stats.arenas.<i>.metadata.mapped: Arena chunk headers.
- stats.arenas.<i>.metadata.allocated: Internal allocations. This is
reported separately from the other metadata statistics because it
overlaps with the allocated and active statistics, whereas the other
metadata statistics do not.
Base allocations are not reported separately, though their magnitude can
be computed by subtracting the arena-specific metadata.
This resolves #163.
/external/jemalloc/src/huge.c
|
| 2c5cb613dfbdf58f88152321b63e60c58cd23972 |
|
08-Dec-2014 |
Guilherme Goncalves <guilherme.p.gonc@gmail.com> |
Introduce two new modes of junk filling: "alloc" and "free".
In addition to true/false, opt.junk can now be either "alloc" or "free",
giving applications the possibility of junking memory only on allocation
or deallocation.
This resolves #172.
/external/jemalloc/src/huge.c
|
| 1036ddbf11b7e9ec566b92b3dd50e105fc5f6932 |
|
05-Dec-2014 |
Jason Evans <je@fb.com> |
Fix OOM cleanup in huge_palloc().
Fix OOM cleanup in huge_palloc() to call idalloct() rather than
base_node_dalloc(). This bug is a result of incomplete refactoring, and
has no impact other than leaking memory during OOM.
/external/jemalloc/src/huge.c
|
| 2012d5a5601c787ce464fac0cbd2b16e3754cfa2 |
|
17-Nov-2014 |
Jason Evans <je@fb.com> |
Fix pointer arithmetic undefined behavior.
Reported by Denis Denisov.
/external/jemalloc/src/huge.c
|
| a9ea10d27c320926cab2e59c66ebcd25c49df24c |
|
16-Oct-2014 |
Daniel Micay <danielmicay@gmail.com> |
use sized deallocation internally for ralloc
The size of the source allocation is known at this point, so reading the
chunk header can be avoided for the small size class fast path. This is
not very useful right now, but it provides a significant performance
boost with an alternate ralloc entry point taking the old size.
/external/jemalloc/src/huge.c
|
| 9673983443a0782d975fbcb5d8457cfd411b8b56 |
|
16-Oct-2014 |
Jason Evans <jasone@canonware.com> |
Purge/zero sub-chunk huge allocations as necessary.
Purge trailing pages during shrinking huge reallocation when resulting
size is not a multiple of the chunk size. Similarly, zero pages if
necessary during growing huge reallocation when the resulting size is
not a multiple of the chunk size.
/external/jemalloc/src/huge.c
|
| 9b41ac909facf4f09bb1b637b78ba647348e572e |
|
15-Oct-2014 |
Jason Evans <je@fb.com> |
Fix huge allocation statistics.
/external/jemalloc/src/huge.c
|
| 3c4d92e82a31f652a7c77ca937a02d0185085b06 |
|
13-Oct-2014 |
Jason Evans <jasone@canonware.com> |
Add per size class huge allocation statistics.
Add per size class huge allocation statistics, and normalize various
stats:
- Change the arenas.nlruns type from size_t to unsigned.
- Add the arenas.nhchunks and arenas.hchunks.<i>.size mallctl's.
- Replace the stats.arenas.<i>.bins.<j>.allocated mallctl with
stats.arenas.<i>.bins.<j>.curregs .
- Add the stats.arenas.<i>.hchunks.<j>.nmalloc,
stats.arenas.<i>.hchunks.<j>.ndalloc,
stats.arenas.<i>.hchunks.<j>.nrequests, and
stats.arenas.<i>.hchunks.<j>.curhchunks mallctl's.
/external/jemalloc/src/huge.c
|
| fc0b3b7383373d66cfed2cd4e2faa272a6868d32 |
|
10-Oct-2014 |
Jason Evans <jasone@canonware.com> |
Add configure options.
Add:
--with-lg-page
--with-lg-page-sizes
--with-lg-size-class-group
--with-lg-quantum
Get rid of STATIC_PAGE_SHIFT, in favor of directly setting LG_PAGE.
Fix various edge conditions exposed by the configure options.
/external/jemalloc/src/huge.c
|
| f22214a29ddd3bed005cbcc8f2aff7c61ef4940b |
|
06-Oct-2014 |
Daniel Micay <danielmicay@gmail.com> |
Use regular arena allocation for huge tree nodes.
This avoids grabbing the base mutex, as a step towards fine-grained
locking for huge allocations. The thread cache also provides a tiny
(~3%) improvement for serial huge allocations.
/external/jemalloc/src/huge.c
|
| 8bb3198f72fc7587dc93527f9f19fb5be52fa553 |
|
08-Oct-2014 |
Jason Evans <jasone@canonware.com> |
Refactor/fix arenas manipulation.
Abstract arenas access to use arena_get() (or a0get() where appropriate)
rather than directly reading e.g. arenas[ind]. Prior to the addition of
the arenas.extend mallctl, the worst possible outcome of directly
accessing arenas was a stale read, but arenas.extend may allocate and
assign a new array to arenas.
Add a tsd-based arenas_cache, which amortizes arenas reads. This
introduces some subtle bootstrapping issues, with tsd_boot() now being
split into tsd_boot[01]() to support tsd wrapper allocation
bootstrapping, as well as an arenas_cache_bypass tsd variable which
dynamically terminates allocation of arenas_cache itself.
Promote a0malloc(), a0calloc(), and a0free() to be generally useful for
internal allocation, and use them in several places (more may be
appropriate).
Abstract arena->nthreads management and fix a missing decrement during
thread destruction (recent tsd refactoring left arenas_cleanup()
unused).
Change arena_choose() to propagate OOM, and handle OOM in all callers.
This is important for providing consistent allocation behavior when the
MALLOCX_ARENA() flag is being used. Prior to this fix, it was possible
for an OOM to result in allocation silently allocating from a different
arena than the one specified.
/external/jemalloc/src/huge.c
|
| 155bfa7da18cab0d21d87aa2dce4554166836f5d |
|
06-Oct-2014 |
Jason Evans <jasone@canonware.com> |
Normalize size classes.
Normalize size classes to use the same number of size classes per size
doubling (currently hard coded to 4), across the intire range of size
classes. Small size classes already used this spacing, but in order to
support this change, additional small size classes now fill [4 KiB .. 16
KiB). Large size classes range from [16 KiB .. 4 MiB). Huge size
classes now support non-multiples of the chunk size in order to fill (4
MiB .. 16 MiB).
/external/jemalloc/src/huge.c
|
| a95018ee819abf897562d9d1f3bc31d4dd725a8d |
|
04-Oct-2014 |
Daniel Micay <danielmicay@gmail.com> |
Attempt to expand huge allocations in-place.
This adds support for expanding huge allocations in-place by requesting
memory at a specific address from the chunk allocator.
It's currently only implemented for the chunk recycling path, although
in theory it could also be done by optimistically allocating new chunks.
On Linux, it could attempt an in-place mremap. However, that won't work
in practice since the heap is grown downwards and memory is not unmapped
(in a normal build, at least).
Repeated vector reallocation micro-benchmark:
#include <string.h>
#include <stdlib.h>
int main(void) {
for (size_t i = 0; i < 100; i++) {
void *ptr = NULL;
size_t old_size = 0;
for (size_t size = 4; size < (1 << 30); size *= 2) {
ptr = realloc(ptr, size);
if (!ptr) return 1;
memset(ptr + old_size, 0xff, size - old_size);
old_size = size;
}
free(ptr);
}
}
The glibc allocator fails to do any in-place reallocations on this
benchmark once it passes the M_MMAP_THRESHOLD (default 128k) but it
elides the cost of copies via mremap, which is currently not something
that jemalloc can use.
With this improvement, jemalloc still fails to do any in-place huge
reallocations for the first outer loop, but then succeeds 100% of the
time for the remaining 99 iterations. The time spent doing allocations
and copies drops down to under 5%, with nearly all of it spent doing
purging + faulting (when huge pages are disabled) and the array memset.
An improved mremap API (MREMAP_RETAIN - #138) would be far more general
but this is a portable optimization and would still be useful on Linux
for xallocx.
Numbers with transparent huge pages enabled:
glibc (copies elided via MREMAP_MAYMOVE): 8.471s
jemalloc: 17.816s
jemalloc + no-op madvise: 13.236s
jemalloc + this commit: 6.787s
jemalloc + this commit + no-op madvise: 6.144s
Numbers with transparent huge pages disabled:
glibc (copies elided via MREMAP_MAYMOVE): 15.403s
jemalloc: 39.456s
jemalloc + no-op madvise: 12.768s
jemalloc + this commit: 15.534s
jemalloc + this commit + no-op madvise: 6.354s
Closes #137
/external/jemalloc/src/huge.c
|
| 551ebc43647521bdd0bc78558b106762b3388928 |
|
03-Oct-2014 |
Jason Evans <jasone@canonware.com> |
Convert to uniform style: cond == false --> !cond
/external/jemalloc/src/huge.c
|
| f8034540a16a6f4fc7948e4783747ca1e9055823 |
|
30-Sep-2014 |
Daniel Micay <danielmicay@gmail.com> |
Implement in-place huge allocation shrinking.
Trivial example:
#include <stdlib.h>
int main(void) {
void *ptr = malloc(1024 * 1024 * 8);
if (!ptr) return 1;
ptr = realloc(ptr, 1024 * 1024 * 4);
if (!ptr) return 1;
}
Before:
mmap(NULL, 8388608, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fcfff000000
mmap(NULL, 4194304, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fcffec00000
madvise(0x7fcfff000000, 8388608, MADV_DONTNEED) = 0
After:
mmap(NULL, 8388608, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1934800000
madvise(0x7f1934c00000, 4194304, MADV_DONTNEED) = 0
Closes #134
/external/jemalloc/src/huge.c
|
| 5460aa6f6676c7f253bfcb75c028dfd38cae8aaf |
|
23-Sep-2014 |
Jason Evans <jasone@canonware.com> |
Convert all tsd variables to reside in a single tsd structure.
/external/jemalloc/src/huge.c
|
| 9c640bfdd4e2f25180a32ed3704ce8e4c4cc21f1 |
|
12-Sep-2014 |
Jason Evans <jasone@canonware.com> |
Apply likely()/unlikely() to allocation/deallocation fast paths.
/external/jemalloc/src/huge.c
|
| b718cf77e9917f6ae1995c2e2b219ff4219c9f46 |
|
07-Sep-2014 |
Jason Evans <je@fb.com> |
Optimize [nmd]alloc() fast paths.
Optimize [nmd]alloc() fast paths such that the (flags == 0) case is
streamlined, flags decoding only happens to the minimum degree
necessary, and no conditionals are repeated.
/external/jemalloc/src/huge.c
|
| 602c8e0971160e4b85b08b16cf8a2375aa24bc04 |
|
19-Aug-2014 |
Jason Evans <jasone@canonware.com> |
Implement per thread heap profiling.
Rename data structures (prof_thr_cnt_t-->prof_tctx_t,
prof_ctx_t-->prof_gctx_t), and convert to storing a prof_tctx_t for
sampled objects.
Convert PROF_ALLOC_PREP() to prof_alloc_prep(), since precise backtrace
depth within jemalloc functions is no longer an issue (pprof prunes
irrelevant frames).
Implement mallctl's:
- prof.reset implements full sample data reset, and optional change of
sample interval.
- prof.lg_sample reads the current sample interval (opt.lg_prof_sample
was the permanent source of truth prior to prof.reset).
- thread.prof.name provides naming capability for threads within heap
profile dumps.
- thread.prof.active makes it possible to activate/deactivate heap
profiling for individual threads.
Modify the heap dump files to contain per thread heap profile data.
This change is incompatible with the existing pprof, which will require
enhancements to read and process the enriched data.
/external/jemalloc/src/huge.c
|
| e2deab7a751c8080c2b2cdcfd7b11887332be1bb |
|
16-May-2014 |
Jason Evans <je@fb.com> |
Refactor huge allocation to be managed by arenas.
Refactor huge allocation to be managed by arenas (though the global
red-black tree of huge allocations remains for lookup during
deallocation). This is the logical conclusion of recent changes that 1)
made per arena dss precedence apply to huge allocation, and 2) made it
possible to replace the per arena chunk allocation/deallocation
functions.
Remove the top level huge stats, and replace them with per arena huge
stats.
Normalize function names and types to *dalloc* (some were *dealloc*).
Remove the --enable-mremap option. As jemalloc currently operates, this
is a performace regression for some applications, but planned work to
logarithmically space huge size classes should provide similar amortized
performance. The motivation for this change was that mremap-based huge
reallocation forced leaky abstractions that prevented refactoring.
/external/jemalloc/src/huge.c
|
| fb7fe50a88ca9bde74e9a401ae17ad3b15bbae28 |
|
06-May-2014 |
aravind <aravind@fb.com> |
Add support for user-specified chunk allocators/deallocators.
Add new mallctl endpoints "arena<i>.chunk.alloc" and
"arena<i>.chunk.dealloc" to allow userspace to configure
jemalloc's chunk allocator and deallocator on a per-arena
basis.
/external/jemalloc/src/huge.c
|
| 4d434adb146375ad17f0d5e994ed5728d2942e3f |
|
15-Apr-2014 |
Jason Evans <je@fb.com> |
Make dss non-optional, and fix an "arena.<i>.dss" mallctl bug.
Make dss non-optional on all platforms which support sbrk(2).
Fix the "arena.<i>.dss" mallctl to return an error if "primary" or
"secondary" precedence is specified, but sbrk(2) is not supported.
/external/jemalloc/src/huge.c
|
| fbb31029a5c2f556f39e04a8781340d4ee4cf16c |
|
27-Mar-2014 |
Max Wang <mwang@fb.com> |
Use arena dss prec instead of default for huge allocs.
Pass a dss_prec_t parameter to huge_{m,p,r}alloc instead of defaulting
to the chunk dss prec.
/external/jemalloc/src/huge.c
|
| 940fdfd5eef45f5425f9124e250fddde5c5c48bf |
|
25-Feb-2014 |
Jason Evans <jasone@canonware.com> |
Fix junk filling for mremap(2)-based huge reallocation.
If mremap(2) is used for huge reallocation, physical pages are mapped to
new virtual addresses rather than data being copied to new pages. This
bypasses the normal junk filling that would happen during allocation, so
add junk filling that is specific to this case.
/external/jemalloc/src/huge.c
|
| b2c31660be917ea6d59cd54e6f650b06b5e812ed |
|
13-Jan-2014 |
Jason Evans <je@fb.com> |
Extract profiling code from [re]allocation functions.
Extract profiling code from malloc(), imemalign(), calloc(), realloc(),
mallocx(), rallocx(), and xallocx(). This slightly reduces the amount
of code compiled into the fast paths, but the primary benefit is the
combinatorial complexity reduction.
Simplify iralloc[t]() by creating a separate ixalloc() that handles the
no-move cases.
Further simplify [mrxn]allocx() (and by implication [mrn]allocm()) to
make request size overflows due to size class and/or alignment
constraints trigger undefined behavior (detected by debug-only
assertions).
Report ENOMEM rather than EINVAL if an OOM occurs during heap profiling
backtrace creation in imemalign(). This bug impacted posix_memalign()
and aligned_alloc().
/external/jemalloc/src/huge.c
|
| 6b694c4d47278cddfaaedeb7ee49fa5757e35ed5 |
|
08-Jan-2014 |
Jason Evans <je@fb.com> |
Add junk/zero filling unit tests, and fix discovered bugs.
Fix growing large reallocation to junk fill new space.
Fix huge deallocation to junk fill when munmap is disabled.
/external/jemalloc/src/huge.c
|
| 6e62984ef6ca4312cf0a2e49ea2cc38feb94175b |
|
16-Dec-2013 |
Jason Evans <jasone@canonware.com> |
Don't junk-fill reallocations unless usize changes.
Don't junk fill reallocations for which the request size is less than
the current usable size, but not enough smaller to cause a size class
change. Unlike malloc()/calloc()/realloc(), *allocx() contractually
treats the full usize as the allocation, so a caller can ask for zeroed
memory via mallocx() and a series of rallocx() calls that all specify
MALLOCX_ZERO, and be assured that all newly allocated bytes will be
zeroed and made available to the application without danger of allocator
mutation until the size class decreases enough to cause usize reduction.
/external/jemalloc/src/huge.c
|
| d82a5e6a34f20698ab9368bb2b4953b81d175552 |
|
13-Dec-2013 |
Jason Evans <jasone@canonware.com> |
Implement the *allocx() API.
Implement the *allocx() API, which is a successor to the *allocm() API.
The *allocx() functions are slightly simpler to use because they have
fewer parameters, they directly return the results of primary interest,
and mallocx()/rallocx() avoid the strict aliasing pitfall that
allocm()/rallocx() share with posix_memalign(). The following code
violates strict aliasing rules:
foo_t *foo;
allocm((void **)&foo, NULL, 42, 0);
whereas the following is safe:
foo_t *foo;
void *p;
allocm(&p, NULL, 42, 0);
foo = (foo_t *)p;
mallocx() does not have this problem:
foo_t *foo = (foo_t *)mallocx(42, 0);
/external/jemalloc/src/huge.c
|
| 2a83ed0284e92c7ba4bd4efe9df149ac724b2f26 |
|
09-Dec-2013 |
Jason Evans <jasone@canonware.com> |
Refactor tests.
Refactor tests to use explicit testing assertions, rather than diff'ing
test output. This makes the test code a bit shorter, more explicitly
encodes testing intent, and makes test failure diagnosis more
straightforward.
/external/jemalloc/src/huge.c
|
| 609ae595f0358157b19311b0f9f9591db7cee705 |
|
11-Oct-2012 |
Jason Evans <je@fb.com> |
Add arena-specific and selective dss allocation.
Add the "arenas.extend" mallctl, so that it is possible to create new
arenas that are outside the set that jemalloc automatically multiplexes
threads onto.
Add the ALLOCM_ARENA() flag for {,r,d}allocm(), so that it is possible
to explicitly allocate from a particular arena.
Add the "opt.dss" mallctl, which controls the default precedence of dss
allocation relative to mmap allocation.
Add the "arena.<i>.dss" mallctl, which makes it possible to set the
default dss precedence on a per arena or global basis.
Add the "arena.<i>.purge" mallctl, which obsoletes "arenas.purge".
Add the "stats.arenas.<i>.dss" mallctl.
/external/jemalloc/src/huge.c
|
| 2e671ffbadc02fc7de8cbafdd1031e3b0ad73c5b |
|
10-May-2012 |
Jason Evans <je@fb.com> |
Add the --enable-mremap option.
Add the --enable-mremap option, and disable the use of mremap(2) by
default, for the same reason that freeing chunks via munmap(2) is
disabled by default on Linux: semi-permanent VM map fragmentation.
/external/jemalloc/src/huge.c
|
| a14bce85e885f83c96116cc5438ae52d740f3727 |
|
30-Apr-2012 |
Mike Hommey <mh@glandium.org> |
Use Get/SetLastError on Win32
Using errno on win32 doesn't quite work, because the value set in a shared
library can't be read from e.g. an executable calling the function setting
errno.
At the same time, since buferror always uses errno/GetLastError, don't pass
it.
/external/jemalloc/src/huge.c
|
| 7ad54c1c30e0805e0758690115875f982de46cf2 |
|
22-Apr-2012 |
Jason Evans <jasone@canonware.com> |
Fix chunk allocation/deallocation bugs.
Fix chunk_alloc_dss() to zero memory when requested.
Fix chunk_dealloc() to avoid chunk_dealloc_mmap() for dss-allocated
memory.
Fix huge_palloc() to always junk fill when requested.
Improve chunk_recycle() to report that memory is zeroed as a side effect
of pages_purge().
/external/jemalloc/src/huge.c
|
| 122449b073bcbaa504c4f592ea2d733503c272d2 |
|
06-Apr-2012 |
Jason Evans <je@fb.com> |
Implement Valgrind support, redzones, and quarantine.
Implement Valgrind support, as well as the redzone and quarantine
features, which help Valgrind detect memory errors. Redzones are only
implemented for small objects because the changes necessary to support
redzones around large and huge objects are complicated by in-place
reallocation, to the point that it isn't clear that the maintenance
burden is worth the incremental improvement to Valgrind support.
Merge arena_salloc() and arena_salloc_demote().
Refactor i[v]salloc() to expose the 'demote' option.
/external/jemalloc/src/huge.c
|
| eae269036c9f702d9fa9be497a1a2aa1be13a29e |
|
10-Apr-2012 |
Mike Hommey <mh@glandium.org> |
Add alignment support to chunk_alloc().
/external/jemalloc/src/huge.c
|
| 4e2e3dd9cf19ed5991938a708a8b50611aa5bbf8 |
|
14-Mar-2012 |
Jason Evans <je@fb.com> |
Fix fork-related bugs.
Acquire/release arena bin locks as part of the prefork/postfork. This
bug made deadlock in the child between fork and exec a possibility.
Split jemalloc_postfork() into jemalloc_postfork_{parent,child}() so
that the child can reinitialize mutexes rather than unlocking them. In
practice, this bug tended not to cause problems.
/external/jemalloc/src/huge.c
|
| d81e4bdd5c991bd5642c8b859ef1f752b51cd9be |
|
06-Mar-2012 |
Jason Evans <je@fb.com> |
Implement malloc_vsnprintf().
Implement malloc_vsnprintf() (a subset of vsnprintf(3)) as well as
several other printing functions based on it, so that formatted printing
can be relied upon without concern for inducing a dependency on floating
point runtime support. Replace malloc_write() calls with
malloc_*printf() where doing so simplifies the code.
Add name mangling for library-private symbols in the data and BSS
sections. Adjust CONF_HANDLE_*() macros in malloc_conf_init() to expose
all opt_* variable use to cpp so that proper mangling occurs.
/external/jemalloc/src/huge.c
|
| 4162627757889ea999264c2ddbc3c354768774e2 |
|
13-Feb-2012 |
Jason Evans <je@fb.com> |
Remove the swap feature.
Remove the swap feature, which enabled per application swap files. In
practice this feature has not proven itself useful to users.
/external/jemalloc/src/huge.c
|
| 7372b15a31c63ac5cb9ed8aeabc2a0a3c005e8bf |
|
11-Feb-2012 |
Jason Evans <je@fb.com> |
Reduce cpp conditional logic complexity.
Convert configuration-related cpp conditional logic to use static
constant variables, e.g.:
#ifdef JEMALLOC_DEBUG
[...]
#endif
becomes:
if (config_debug) {
[...]
}
The advantage is clearer, more concise code. The main disadvantage is
that data structures no longer have conditionally defined fields, so
they pay the cost of all fields regardless of whether they are used. In
practice, this is only a minor concern; config_stats will go away in an
upcoming change, and config_prof is the only other major feature that
depends on more than a few special-purpose fields.
/external/jemalloc/src/huge.c
|
| 12a488782681cbd740a5f54e0b7e74ea84858e21 |
|
11-Nov-2011 |
Jason Evans <je@fb.com> |
Fix huge_ralloc to maintain chunk statistics.
Fix huge_ralloc() to properly maintain chunk statistics when using
mremap(2).
/external/jemalloc/src/huge.c
|
| fa351d9fdcbbbfe7455279311fdf3d65751a4e75 |
|
09-Nov-2011 |
Jason Evans <je@fb.com> |
Fix huge_ralloc() race when using mremap(2).
Fix huge_ralloc() to remove the old memory region from tree of huge
allocations *before* calling mremap(2), in order to make sure that no
other thread acquires the old memory region via mmap() and encounters
stale metadata in the tree.
Reported by: Rich Prohaska
/external/jemalloc/src/huge.c
|
| 7427525c28d58c423a68930160e3b0fe577fe953 |
|
01-Apr-2011 |
Jason Evans <jasone@canonware.com> |
Move repo contents in jemalloc/ to top level.
/external/jemalloc/src/huge.c
|