15821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML//EN"> 25821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<HTML> 35821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 45821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<HEAD> 55821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) <link rel="stylesheet" href="designstyle.css"> 65821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) <title>Gperftools Heap Profiler</title> 75821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)</HEAD> 85821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 95821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<BODY> 105821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 115821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<p align=right> 125821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) <i>Last modified 135821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) <script type=text/javascript> 145821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) var lm = new Date(document.lastModified); 155821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) document.write(lm.toDateString()); 165821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) </script></i> 175821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)</p> 185821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 195821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<p>This is the heap profiler we use at Google, to explore how C++ 205821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)programs manage memory. This facility can be useful for</p> 215821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<ul> 225821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) <li> Figuring out what is in the program heap at any given time 235821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) <li> Locating memory leaks 245821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) <li> Finding places that do a lot of allocation 255821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)</ul> 265821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 275821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<p>The profiling system instruments all allocations and frees. It 285821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)keeps track of various pieces of information per allocation site. An 295821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)allocation site is defined as the active stack trace at the call to 305821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<code>malloc</code>, <code>calloc</code>, <code>realloc</code>, or, 315821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<code>new</code>.</p> 325821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 335821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<p>There are three parts to using it: linking the library into an 345821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)application, running the code, and analyzing the output.</p> 355821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 365821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 375821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<h1>Linking in the Library</h1> 385821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 395821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<p>To install the heap profiler into your executable, add 405821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<code>-ltcmalloc</code> to the link-time step for your executable. 415821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)Also, while we don't necessarily recommend this form of usage, it's 425821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)possible to add in the profiler at run-time using 435821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<code>LD_PRELOAD</code>: 445821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<pre>% env LD_PRELOAD="/usr/lib/libtcmalloc.so" <binary></pre> 455821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 465821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<p>This does <i>not</i> turn on heap profiling; it just inserts the 475821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)code. For that reason, it's practical to just always link 485821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<code>-ltcmalloc</code> into a binary while developing; that's what we 495821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)do at Google. (However, since any user can turn on the profiler by 505821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)setting an environment variable, it's not necessarily recommended to 515821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)install profiler-linked binaries into a production, running 525821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)system.) Note that if you wish to use the heap profiler, you must 535821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)also use the tcmalloc memory-allocation library. There is no way 545821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)currently to use the heap profiler separate from tcmalloc.</p> 555821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 565821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 575821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<h1>Running the Code</h1> 585821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 595821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<p>There are several alternatives to actually turn on heap profiling 605821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)for a given run of an executable:</p> 615821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 625821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<ol> 635821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) <li> <p>Define the environment variable HEAPPROFILE to the filename 645821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) to dump the profile to. For instance, to profile 655821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) <code>/usr/local/bin/my_binary_compiled_with_tcmalloc</code>:</p> 665821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) <pre>% env HEAPPROFILE=/tmp/mybin.hprof /usr/local/bin/my_binary_compiled_with_tcmalloc</pre> 675821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) <li> <p>In your code, bracket the code you want profiled in calls to 685821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) <code>HeapProfilerStart()</code> and <code>HeapProfilerStop()</code>. 695821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) (These functions are declared in <code><gperftools/heap-profiler.h></code>.) 705821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) <code>HeapProfilerStart()</code> will take the 715821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) profile-filename-prefix as an argument. Then, as often as 725821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) you'd like before calling <code>HeapProfilerStop()</code>, you 735821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) can use <code>HeapProfilerDump()</code> or 745821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) <code>GetHeapProfile()</code> to examine the profile. In case 755821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) it's useful, <code>IsHeapProfilerRunning()</code> will tell you 765821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) whether you've already called HeapProfilerStart() or not.</p> 775821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)</ol> 785821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 795821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 805821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<p>For security reasons, heap profiling will not write to a file -- 815821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)and is thus not usable -- for setuid programs.</p> 825821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 835821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<H2>Modifying Runtime Behavior</H2> 845821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 855821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<p>You can more finely control the behavior of the heap profiler via 865821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)environment variables.</p> 875821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 885821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<table frame=box rules=sides cellpadding=5 width=100%> 895821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 905821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<tr valign=top> 915821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) <td><code>HEAP_PROFILE_ALLOCATION_INTERVAL</code></td> 925821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) <td>default: 1073741824 (1 Gb)</td> 935821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) <td> 945821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) Dump heap profiling information once every specified number of 955821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) bytes has been allocated by the program. 965821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) </td> 975821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)</tr> 985821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 995821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<tr valign=top> 1005821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) <td><code>HEAP_PROFILE_INUSE_INTERVAL</code></td> 1015821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) <td>default: 104857600 (100 Mb)</td> 1025821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) <td> 1035821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) Dump heap profiling information whenever the high-water memory 1045821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) usage mark increases by the specified number of bytes. 1055821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) </td> 1065821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)</tr> 1075821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 1085821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<tr valign=top> 1095821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) <td><code>HEAP_PROFILE_MMAP</code></td> 1105821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) <td>default: false</td> 1115821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) <td> 1125821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) Profile <code>mmap</code>, <code>mremap</code> and <code>sbrk</code> 1135821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) calls in addition 1145821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) to <code>malloc</code>, <code>calloc</code>, <code>realloc</code>, 1155821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) and <code>new</code>. <b>NOTE:</b> this causes the profiler to 1165821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) profile calls internal to tcmalloc, since tcmalloc and friends use 1175821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) mmap and sbrk internally for allocations. One partial solution is 1185821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) to filter these allocations out when running <code>pprof</code>, 1195821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) with something like 1205821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) <code>pprof --ignore='DoAllocWithArena|SbrkSysAllocator::Alloc|MmapSysAllocator::Alloc</code>. 1215821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) </td> 1225821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)</tr> 1235821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 1245821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<tr valign=top> 1255821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) <td><code>HEAP_PROFILE_MMAP_ONLY</code></td> 1265821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) <td>default: false</td> 1275821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) <td> 1285821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) Only profile <code>mmap</code>, <code>mremap</code>, and <code>sbrk</code> 1295821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) calls; do not profile 1305821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) <code>malloc</code>, <code>calloc</code>, <code>realloc</code>, 1315821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) or <code>new</code>. 1325821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) </td> 1335821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)</tr> 1345821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 1355821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<tr valign=top> 1365821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) <td><code>HEAP_PROFILE_MMAP_LOG</code></td> 1375821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) <td>default: false</td> 1385821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) <td> 1395821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) Log <code>mmap</code>/<code>munmap</code> calls. 1405821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) </td> 1415821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)</tr> 1425821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 1435821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)</table> 1445821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 1455821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<H2>Checking for Leaks</H2> 1465821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 1475821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<p>You can use the heap profiler to manually check for leaks, for 1485821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)instance by reading the profiler output and looking for large 1495821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)allocations. However, for that task, it's easier to use the <A 1505821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)HREF="heap_checker.html">automatic heap-checking facility</A> built 1515821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)into tcmalloc.</p> 1525821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 1535821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 1545821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<h1><a name="pprof">Analyzing the Output</a></h1> 1555821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 1565821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<p>If heap-profiling is turned on in a program, the program will 1575821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)periodically write profiles to the filesystem. The sequence of 1585821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)profiles will be named:</p> 1595821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<pre> 1605821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) <prefix>.0000.heap 1615821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) <prefix>.0001.heap 1625821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) <prefix>.0002.heap 1635821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ... 1645821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)</pre> 1655821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<p>where <code><prefix></code> is the filename-prefix supplied 1665821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)when running the code (e.g. via the <code>HEAPPROFILE</code> 1675821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)environment variable). Note that if the supplied prefix 1685821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)does not start with a <code>/</code>, the profile files will be 1695821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)written to the program's working directory.</p> 1705821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 1715821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<p>The profile output can be viewed by passing it to the 1725821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<code>pprof</code> tool -- the same tool that's used to analyze <A 1735821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)HREF="cpuprofile.html">CPU profiles</A>. 1745821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 1755821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<p>Here are some examples. These examples assume the binary is named 1765821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<code>gfs_master</code>, and a sequence of heap profile files can be 1775821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)found in files named:</p> 1785821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<pre> 1795821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /tmp/profile.0001.heap 1805821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /tmp/profile.0002.heap 1815821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ... 1825821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) /tmp/profile.0100.heap 1835821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)</pre> 1845821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 1855821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<h3>Why is a process so big</h3> 1865821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 1875821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<pre> 1885821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) % pprof --gv gfs_master /tmp/profile.0100.heap 1895821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)</pre> 1905821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 1915821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<p>This command will pop-up a <code>gv</code> window that displays 1925821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)the profile information as a directed graph. Here is a portion 1935821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)of the resulting output:</p> 1945821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 1955821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<p><center> 1965821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<img src="heap-example1.png"> 1975821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)</center></p> 1985821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 1995821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)A few explanations: 2005821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<ul> 2015821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<li> <code>GFS_MasterChunk::AddServer</code> accounts for 255.6 MB 2025821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) of the live memory, which is 25% of the total live memory. 2035821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<li> <code>GFS_MasterChunkTable::UpdateState</code> is directly 2045821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) accountable for 176.2 MB of the live memory (i.e., it directly 2055821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) allocated 176.2 MB that has not been freed yet). Furthermore, 2065821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) it and its callees are responsible for 729.9 MB. The 2075821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) labels on the outgoing edges give a good indication of the 2085821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) amount allocated by each callee. 2095821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)</ul> 2105821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 2115821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<h3>Comparing Profiles</h3> 2125821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 2135821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<p>You often want to skip allocations during the initialization phase 2145821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)of a program so you can find gradual memory leaks. One simple way to 2155821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)do this is to compare two profiles -- both collected after the program 2165821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)has been running for a while. Specify the name of the first profile 2175821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)using the <code>--base</code> option. For example:</p> 2185821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<pre> 2195821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) % pprof --base=/tmp/profile.0004.heap gfs_master /tmp/profile.0100.heap 2205821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)</pre> 2215821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 2225821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<p>The memory-usage in <code>/tmp/profile.0004.heap</code> will be 2235821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)subtracted from the memory-usage in 2245821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<code>/tmp/profile.0100.heap</code> and the result will be 2255821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)displayed.</p> 2265821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 2275821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<h3>Text display</h3> 2285821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 2295821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<pre> 2305821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)% pprof --text gfs_master /tmp/profile.0100.heap 2315821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 255.6 24.7% 24.7% 255.6 24.7% GFS_MasterChunk::AddServer 2325821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 184.6 17.8% 42.5% 298.8 28.8% GFS_MasterChunkTable::Create 2335821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 176.2 17.0% 59.5% 729.9 70.5% GFS_MasterChunkTable::UpdateState 2345821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 169.8 16.4% 75.9% 169.8 16.4% PendingClone::PendingClone 2355821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 76.3 7.4% 83.3% 76.3 7.4% __default_alloc_template::_S_chunk_alloc 2365821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 49.5 4.8% 88.0% 49.5 4.8% hashtable::resize 2375821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) ... 2385821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)</pre> 2395821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 2405821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<p> 2415821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<ul> 2425821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) <li> The first column contains the direct memory use in MB. 2435821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) <li> The fourth column contains memory use by the procedure 2445821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) and all of its callees. 2455821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) <li> The second and fifth columns are just percentage 2465821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) representations of the numbers in the first and fourth columns. 2475821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) <li> The third column is a cumulative sum of the second column 2485821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) (i.e., the <code>k</code>th entry in the third column is the 2495821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) sum of the first <code>k</code> entries in the second column.) 2505821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)</ul> 2515821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 2525821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<h3>Ignoring or focusing on specific regions</h3> 2535821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 2545821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<p>The following command will give a graphical display of a subset of 2555821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)the call-graph. Only paths in the call-graph that match the regular 2565821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)expression <code>DataBuffer</code> are included:</p> 2575821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<pre> 2585821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)% pprof --gv --focus=DataBuffer gfs_master /tmp/profile.0100.heap 2595821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)</pre> 2605821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 2615821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<p>Similarly, the following command will omit all paths subset of the 2625821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)call-graph. All paths in the call-graph that match the regular 2635821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)expression <code>DataBuffer</code> are discarded:</p> 2645821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<pre> 2655821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)% pprof --gv --ignore=DataBuffer gfs_master /tmp/profile.0100.heap 2665821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)</pre> 2675821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 2685821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<h3>Total allocations + object-level information</h3> 2695821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 2705821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<p>All of the previous examples have displayed the amount of in-use 2715821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)space. I.e., the number of bytes that have been allocated but not 2725821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)freed. You can also get other types of information by supplying a 2735821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)flag to <code>pprof</code>:</p> 2745821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 2755821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<center> 2765821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<table frame=box rules=sides cellpadding=5 width=100%> 2775821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 2785821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<tr valign=top> 2795821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) <td><code>--inuse_space</code></td> 2805821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) <td> 2815821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) Display the number of in-use megabytes (i.e. space that has 2825821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) been allocated but not freed). This is the default. 2835821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) </td> 2845821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)</tr> 2855821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 2865821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<tr valign=top> 2875821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) <td><code>--inuse_objects</code></td> 2885821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) <td> 2895821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) Display the number of in-use objects (i.e. number of 2905821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) objects that have been allocated but not freed). 2915821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) </td> 2925821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)</tr> 2935821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 2945821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<tr valign=top> 2955821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) <td><code>--alloc_space</code></td> 2965821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) <td> 2975821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) Display the number of allocated megabytes. This includes 2985821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) the space that has since been de-allocated. Use this 2995821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if you want to find the main allocation sites in the 3005821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) program. 3015821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) </td> 3025821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)</tr> 3035821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 3045821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<tr valign=top> 3055821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) <td><code>--alloc_objects</code></td> 3065821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) <td> 3075821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) Display the number of allocated objects. This includes 3085821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) the objects that have since been de-allocated. Use this 3095821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) if you want to find the main allocation sites in the 3105821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) program. 3115821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) </td> 3125821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 3135821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)</table> 3145821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)</center> 3155821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 3165821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 3175821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<h3>Interactive mode</a></h3> 3185821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 3195821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<p>By default -- if you don't specify any flags to the contrary -- 3205821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)pprof runs in interactive mode. At the <code>(pprof)</code> prompt, 3215821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)you can run many of the commands described above. You can type 3225821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<code>help</code> for a list of what commands are available in 3235821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)interactive mode.</p> 3245821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 3255821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 3265821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<h1>Caveats</h1> 3275821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 3285821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<ul> 3295821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) <li> Heap profiling requires the use of libtcmalloc. This 3305821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) requirement may be removed in a future version of the heap 3315821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) profiler, and the heap profiler separated out into its own 3325821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) library. 3335821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 3345821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) <li> If the program linked in a library that was not compiled 3355821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) with enough symbolic information, all samples associated 3365821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) with the library may be charged to the last symbol found 3375821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) in the program before the libary. This will artificially 3385821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) inflate the count for that symbol. 3395821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 3405821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) <li> If you run the program on one machine, and profile it on 3415821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) another, and the shared libraries are different on the two 3425821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) machines, the profiling output may be confusing: samples that 3435821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) fall within the shared libaries may be assigned to arbitrary 3445821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) procedures. 3455821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 3465821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) <li> Several libraries, such as some STL implementations, do their 3475821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) own memory management. This may cause strange profiling 3485821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) results. We have code in libtcmalloc to cause STL to use 3495821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) tcmalloc for memory management (which in our tests is better 3505821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) than STL's internal management), though it only works for some 3515821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) STL implementations. 3525821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 3535821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) <li> If your program forks, the children will also be profiled 3545821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) (since they inherit the same HEAPPROFILE setting). Each 3555821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) process is profiled separately; to distinguish the child 3565821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) profiles from the parent profile and from each other, all 3575821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) children will have their process-id attached to the HEAPPROFILE 3585821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) name. 3595821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 3605821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) <li> Due to a hack we make to work around a possible gcc bug, your 3615821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) profiles may end up named strangely if the first character of 3625821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) your HEAPPROFILE variable has ascii value greater than 127. 3635821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) This should be exceedingly rare, but if you need to use such a 3645821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) name, just set prepend <code>./</code> to your filename: 3655821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) <code>HEAPPROFILE=./Ägypten</code>. 3665821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)</ul> 3675821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles) 3685821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<hr> 3695821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<address>Sanjay Ghemawat 3705821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)<!-- Created: Tue Dec 19 10:43:14 PST 2000 --> 3715821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)</address> 3725821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)</body> 3735821806d5e7f356e8fa4b058a389a808ea183019Torne (Richard Coles)</html> 374
