1e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\documentclass{article} 2e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 3e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat% 4e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat% Copyright (C) 2005, 2006 Alan D. Brunelle <Alan.Brunelle@hp.com> 5e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat% 6e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat% This program is free software; you can redistribute it and/or modify 7e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat% it under the terms of the GNU General Public License as published by 8e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat% the Free Software Foundation; either version 2 of the License, or 9e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat% (at your option) any later version. 10e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat% 11e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat% This program is distributed in the hope that it will be useful, 12e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat% but WITHOUT ANY WARRANTY; without even the implied warranty of 13e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat% MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 14e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat% GNU General Public License for more details. 15e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat% 16e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat% You should have received a copy of the GNU General Public License 17e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat% along with this program; if not, write to the Free Software 18e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat% Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA 19e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat% 20e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 21e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\title{blktrace User Guide} 22e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\author{blktrace: Jens Axboe (jens.axboe@oracle.com)\\ 23e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat User Guide: Alan D. Brunelle (Alan.Brunelle@hp.com)} 24e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\date{27 May 2008} 25e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 26e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\begin{document} 27e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\maketitle 28e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat%--------------------- 29e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\section{\label{sec:intro}Introduction} 30e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 31e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehatblktrace is a block layer IO tracing mechanism which provides detailed 32e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehatinformation about request queue operations up to user space. There are 33e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehatthree major components that are provided: 34e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 35e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\begin{description} 36e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat \item[Kernel patch] A patch to the Linux kernel which includes the 37e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat kernel event logging interfaces, and patches to areas within the block 38e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat layer to emit event traces. If you run a 2.6.17-rc1 or newer kernel, 39e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat you don't need to patch blktrace support as it is already included. 40e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 41e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat \item[blktrace] A utility which transfers event traces from the kernel 42e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat into either long-term on-disk storage, or provides direct formatted 43e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat output (via blkparse). 44e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 45e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat \item[blkparse] A utility which formats events stored in files, or when 46e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat run in \emph{live} mode directly outputs data collected by blktrace. 47e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\end{description} 48e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 49e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\subsection{blktrace Download Area} 50e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 51e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatThe blktrace and blkparse utilities and associated kernel patch are provided 52e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehatas part of the following git repository: 53e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 54e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehatgit://git.kernel.org/pub/scm/linux/kernel/git/axboe/blktrace.git bt 55e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 56e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat%-------------------------- 57e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\newpage\section{\label{sec:quick-start}Quick Start Guide} 58e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 59e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatThe following sections outline some quick steps towards utilizing 60e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehatblktrace. Some of the specific instructions below may need to be tailored 61e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehatto your environment. 62e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 63e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\subsection{\label{sec:get-blktrace}Retrieving blktrace} 64e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 65e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatAs noted above, the kernel patch along with the blktrace and blkparse utilities are stored in a git repository. One simple way to get going would be: 66e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 67e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\begin{verbatim} 68e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat% git clone git://git.kernel.org/pub/scm/linux/kernel/git/axboe/blktrace.git bt 69e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat% cd bt 70e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat% git checkout 71e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\end{verbatim} 72e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 73e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\subsection{\label{sec:patching}Patching and configuring the Linux kernel} 74e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 75e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatA patch for a \emph{specific Linux kernel} is provided in bt/kernel (where 76e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\emph{bt} is the name of the directory from the above git sequence). The 77e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehatdetailed actual patching instructions for a Linux kernel is outside the 78e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehatscope of this document, but the following may be used as a sample template. 79e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatNote that you may skip this step, if you kernel is at least 2.6.17-rc1. 80e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 81e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatAs an example, bt/kernel contains blk-trace-2.6.14-rc1-git-G2, download 82e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehatlinux-2.6.13.tar.bz2 and patch-2.6.14-rc1.bz2 83e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 84e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\begin{verbatim} 85e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat% tar xjf linux-2.6.13.tar.bz2 86e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat% mv linux-2.6.13 linux-2.6.14-rc1 87e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat% cd linux-2.6.14-rc1 88e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat% bunzip2 -c ../patch-2.6.14-rc1.bz2 | patch -p1 89e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\end{verbatim} 90e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 91e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatAt this point you may (optionally) remove linux-2.6.13.tar.bz2 and 92e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehatpatch-2.6.14-rc1.bz2. 93e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 94e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatAt this point you should configure the Linux kernel for your specific 95e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehatsystem -- again, outside the scope of this document -- and then enable 96e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\emph{Support for tracing block io actions.} To do this, run 97e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 98e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\begin{verbatim} 99e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat% make menuconfig or make xconfig, or edit .config, or ... 100e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\end{verbatim} 101e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 102e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehatand navigate through \emph{Device Drivers} and \emph{Block devices} 103e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehatand then down to \emph{Support for tracing block io actions} and hit Y. 104e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 105e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatInstall the new kernel (and modules\ldots) and reboot. 106e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 107e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\subsection{\label{sec:mount}Mounting the debugfs file system} 108e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 109e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehatblktrace utilizes files under the debug file system, and thus must have 110e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehatthe mount point set up -- mounted on the directory /sys/kernel/debug. 111e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatTo do this one may do either of the following: 112e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 113e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\begin{enumerate} 114e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat \item Manually mount after each boot: 115e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\begin{verbatim} 116e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat% mount -t debugfs debugfs /sys/kernel/debug 117e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\end{verbatim} 118e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 119e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat \item Add an entry into /etc/fstab, and have it done automatically at 120e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat each boot\footnote{Note: after adding the entry to /etc/fstab, you 121e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat could then mount the directory this time only by doing: \% mount debug}: 122e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\begin{verbatim} 123e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehatdebug /sys/kernel/debug debugfs default 0 0 124e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\end{verbatim} 125e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\end{enumerate} 126e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 127e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\subsection{\label{sec:build}Build the tools} 128e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 129e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatTo build and install the tools, execute the following sequence (as root): 130e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 131e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\begin{verbatim} 132e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat% cd bt 133e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat% make && make install 134e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\end{verbatim} 135e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 136e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\subsection{\label{sec:live-blktrace}blktrace -- live} 137e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 138e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatNow to simply watch what is going on for a specific disk (to stop the 139e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehattrace, hit control-C): 140e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 141e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\begin{verbatim} 142e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat% blktrace -d /dev/sda -o - | blkparse -i - 143e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 8,0 3 1 0.000000000 697 G W 223490 + 8 [kjournald] 144e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 8,0 3 2 0.000001829 697 P R [kjournald] 145e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 8,0 3 3 0.000002197 697 Q W 223490 + 8 [kjournald] 146e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 8,0 3 4 0.000005533 697 M W 223498 + 8 [kjournald] 147e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 8,0 3 5 0.000008607 697 M W 223506 + 8 [kjournald] 148e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 8,0 3 6 0.000011569 697 M W 223514 + 8 [kjournald] 149e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 8,0 3 7 0.000014407 697 M W 223522 + 8 [kjournald] 150e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 8,0 3 8 0.000017367 697 M W 223530 + 8 [kjournald] 151e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 8,0 3 9 0.000020161 697 M W 223538 + 8 [kjournald] 152e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 8,0 3 10 0.000024062 697 D W 223490 + 56 [kjournald] 153e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 8,0 1 11 0.009507758 0 C W 223490 + 56 [0] 154e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 8,0 1 12 0.009538995 697 G W 223546 + 8 [kjournald] 155e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 8,0 1 13 0.009540033 697 P R [kjournald] 156e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 8,0 1 14 0.009540313 697 Q W 223546 + 8 [kjournald] 157e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 8,0 1 15 0.009542980 697 D W 223546 + 8 [kjournald] 158e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 8,0 1 16 0.013542170 0 C W 223546 + 8 [0] 159e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat... 160e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat^C 161e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat... 162e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatCPU1 (8,0): 163e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat Reads Queued: 0, 0KiB Writes Queued: 7, 128KiB 164e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat Read Dispatches: 0, 0KiB Write Dispatches: 7, 128KiB 165e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat Reads Completed: 0, 0KiB Writes Completed: 11, 168KiB 166e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat Read Merges: 0 Write Merges: 25 167e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat IO unplugs: 0 Timer unplugs: 0 168e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat... 169e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatCPU3 (8,0): 170e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat Reads Queued: 0, 0KiB Writes Queued: 1, 28KiB 171e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat Read Dispatches: 0, 0KiB Write Dispatches: 1, 28KiB 172e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat Reads Completed: 0, 0KiB Writes Completed: 0, 0KiB 173e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat Read Merges: 0 Write Merges: 6 174e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat IO unplugs: 0 Timer unplugs: 0 175e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 176e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatTotal (8,0): 177e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat Reads Queued: 0, 0KiB Writes Queued: 11, 168KiB 178e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat Read Dispatches: 0, 0KiB Write Dispatches: 11, 168KiB 179e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat Reads Completed: 0, 0KiB Writes Completed: 11, 168KiB 180e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat Read Merges: 0 Write Merges: 31 181e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat IO unplugs: 0 Timer unplugs: 3 182e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 183e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatEvents (8,0): 89 entries, 0 skips 184e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\end{verbatim} 185e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 186e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatA \emph{btrace} script is included in the distribution to ease live 187e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehattracing of devices. The above could also be accomplished by issuing: 188e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 189e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\begin{verbatim} 190e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat% btrace /dev/sda 191e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\end{verbatim} 192e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 193e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatBy default, \emph{btrace} runs the trace in quiet mode so it will not 194e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehatinclude statistics when you break the run. Add the \emph{-S} option to 195e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehatget that dumped as well. 196e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 197e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\subsection{\label{sec:pc-blktrace}blktrace -- SCSI commands} 198e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 199e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatThe previous section showed typical file system io actions, but blktrace 200e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehatcan also show SCSI commands going in and out of the queue as submitted 201e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehatby applications using the SCSI Generic (\emph{sg}) interface. 202e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 203e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\begin{verbatim} 204e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat% btrace /dev/cdrom 205e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat[...] 206e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 3,0 0 25 0.004884107 13528 G R 0 + 0 [inquiry] 207e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 3,0 0 26 0.004890361 13528 I R 56 (12 00 00 00 38 ..) [inquiry] 208e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 3,0 0 27 0.004891223 13528 P R [inquiry] 209e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 3,0 0 28 0.004893250 13528 D R 56 (12 00 00 00 38 ..) [inquiry] 210e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 3,0 0 29 0.005344910 0 C R (12 00 00 00 38 ..) [0] 211e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\end{verbatim} 212e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 213e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatHere we see a program issuing an INQUIRY command to the CDROM device. 214e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatThe program requested a read of 56 bytes of data, the CDB is included 215e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehatin parenthesis after the data length. The completion event shows shows 216e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehatthat the command completed successfully. Tracing SCSI commands can be 217e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehatvery useful for debugging problems with programs talking directly to the 218e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehatdevice. An example of that would be \emph{cdrecord} burning. 219e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 220e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\subsection{\label{sec:blktrace-post}blktrace -- post-processing} 221e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 222e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatAnother way to run blktrace is to have blktrace save data away for later 223e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehatformatting by blkparse. This would be useful if you want to get 224e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehatmeasurements while running specific loads. 225e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 226e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatTo do this, one would specify the device (or devices) to be watched. Then 227e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehatgo run you test cases. Stop the trace, and at your leisure utilize 228e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehatblkparse to see the results. 229e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 230e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatIn this example, devices /dev/sdaa, /dev/sdc and /dev/sdo are used in an 231e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatLVM volume called adb3/vol. 232e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 233e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\begin{verbatim} 234e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat% blktrace /dev/sdaa /dev/sdc /dev/sdo & 235e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat[1] 9713 236e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat% 237e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat% mkfs -t ext3 /dev/adb3/vol 238e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehatmke2fs 1.35 (28-Feb-2004) 239e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatFilesystem label= 240e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatOS type: Linux 241e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatBlock size=4096 (log=2) 242e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatFragment size=4096 (log=2) 243e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat16793600 inodes, 33555456 blocks 244e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat1677772 blocks (5.00%) reserved for the super user 245e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatFirst data block=0 246e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatMaximum filesystem blocks=4294967296 247e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat1025 block groups 248e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat32768 blocks per group, 32768 fragments per group 249e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat16384 inodes per group 250e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatSuperblock backups stored on blocks: 251e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 252e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 4096000, 7962624, 11239424, 20480000, 23887872 253e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 254e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatWriting inode tables: done 255e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatCreating journal (8192 blocks): done 256e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatWriting superblocks and filesystem accounting information: done 257e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 258e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatThis filesystem will be automatically checked every 27 mounts or 259e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat180 days, whichever comes first. Use tune2fs -c or -i to override. 260e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat% 261e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat% kill -15 9713 262e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\end{verbatim} 263e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 264e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatThen you could process the events later: 265e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 266e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\begin{verbatim} 267e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat% 268e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat% blkparse sdaa sdc sdo > events 269e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat% less events 270e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 8,32 1 1 0.000000000 9728 G R 384 + 32 [mkfs.ext3] 271e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 8,32 1 2 0.000001959 9728 P R [mkfs.ext3] 272e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 8,32 1 3 0.000002446 9728 Q R 384 + 32 [mkfs.ext3] 273e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 8,32 1 4 0.000005110 9728 D R 384 + 32 [mkfs.ext3] 274e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 8,32 3 5 0.000200570 0 C R 384 + 32 [0] 275e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 8,224 3 1 0.021658989 9728 G R 384 + 32 [mkfs.ext3] 276e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat... 277e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 65,160 3 163392 41.117070504 0 C W 87469088 + 1376 [0] 278e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 8,32 3 163374 41.122683668 0 C W 88168160 + 1376 [0] 279e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 65,160 3 163393 41.129952433 0 C W 87905984 + 1376 [0] 280e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 65,160 3 163394 41.130049431 0 D W 89129344 + 1376 [swapper] 281e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 65,160 3 163395 41.130067135 0 D W 89216704 + 1376 [swapper] 282e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 65,160 3 163396 41.130083785 0 D W 89304096 + 1376 [swapper] 283e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 65,160 3 163397 41.130099455 0 D W 89391488 + 1376 [swapper] 284e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 65,160 3 163398 41.130114732 0 D W 89478848 + 1376 [swapper] 285e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 65,160 3 163399 41.130128885 0 D W 89481536 + 64 [swapper] 286e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 8,32 3 163375 41.134758196 0 C W 86333152 + 1376 [0] 287e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 65,160 3 163400 41.142229726 0 C W 89129344 + 1376 [0] 288e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 65,160 3 163401 41.144952314 0 C W 89481536 + 64 [0] 289e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 8,32 3 163376 41.147441930 0 C W 88342912 + 1376 [0] 290e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 65,160 3 163402 41.155869604 0 C W 89478848 + 1376 [0] 291e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 8,32 3 163377 41.159466082 0 C W 86245760 + 1376 [0] 292e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 65,160 3 163403 41.166944976 0 C W 89216704 + 1376 [0] 293e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 65,160 3 163404 41.178968252 0 C W 89304096 + 1376 [0] 294e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 65,160 3 163405 41.191860173 0 C W 89391488 + 1376 [0] 295e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat... 296e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatEvents (sdo): 0 entries, 0 skips 297e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 298e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatCPU0 (65,160): 299e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat Reads Queued: 0, 0KiB Writes Queued: 9, 5,520KiB 300e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat Read Dispatches: 0, 0KiB Write Dispatches: 0, 0KiB 301e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat Reads Completed: 0, 0KiB Writes Completed: 0, 0KiB 302e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat Read Merges: 0 Write Merges: 336 303e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat IO unplugs: 0 Timer unplugs: 0 304e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatCPU1 (65,160): 305e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat Reads Queued: 2,411, 38,576KiB Writes Queued: 769, 425,408KiB 306e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat Read Dispatches: 2,407, 38,512KiB Write Dispatches: 118, 61,680KiB 307e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat Reads Completed: 0, 0KiB Writes Completed: 0, 0KiB 308e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat Read Merges: 0 Write Merges: 25,819 309e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat IO unplugs: 0 Timer unplugs: 4 310e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatCPU2 (65,160): 311e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat Reads Queued: 2, 32KiB Writes Queued: 18, 10,528KiB 312e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat Read Dispatches: 2, 32KiB Write Dispatches: 3, 1,344KiB 313e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat Reads Completed: 0, 0KiB Writes Completed: 0, 0KiB 314e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat Read Merges: 0 Write Merges: 640 315e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat IO unplugs: 0 Timer unplugs: 0 316e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatCPU3 (65,160): 317e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat Reads Queued: 20,572, 329,152KiB Writes Queued: 594, 279,712KiB 318e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat Read Dispatches: 20,576, 329,216KiB Write Dispatches: 1,474, 740,720KiB 319e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat Reads Completed: 22,985, 367,760KiB Writes Completed: 1,390, 721,168KiB 320e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat Read Merges: 0 Write Merges: 16,888 321e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat IO unplugs: 0 Timer unplugs: 0 322e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 323e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatTotal (65,160): 324e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat Reads Queued: 22,985, 367,760KiB Writes Queued: 1,390, 721,168KiB 325e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat Read Dispatches: 22,985, 367,760KiB Write Dispatches: 1,595, 803,744KiB 326e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat Reads Completed: 22,985, 367,760KiB Writes Completed: 1,390, 721,168KiB 327e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat Read Merges: 0 Write Merges: 43,683 328e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat IO unplugs: 0 Timer unplugs: 4 329e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat... 330e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\end{verbatim} 331e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 332e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat%---------------------------- 333e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\newpage\section{\label{sec:blktrace-ug}blktrace User Guide} 334e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 335e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatThe \emph{blktrace} utility extracts event traces from the kernel (via 336e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehatthe relaying through the debug file system). Some background details 337e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehatconcerning the run-time behaviour of blktrace will help to understand some 338e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehatof the more arcane command line options: 339e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 340e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\begin{itemize} 341e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat \item blktrace receives data from the kernel in buffers passed up 342e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat through the debug file system (relay). Each device being traced has 343e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat a file created in the mounted directory for the debugfs, which defaults 344e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat to \emph{/sys/kernel/debug} -- this can be overridden with the \emph{-r} 345e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat command line argument. 346e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 347e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat \item blktrace defaults to collecting \emph{all} events that can be 348e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat traced. To limit the events being captured, you can specify one or 349e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat more filter masks via the \emph{-a} option. 350e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 351e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat Alternatively, one may specify the entire mask utilizing a hexadecimal 352e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat value that is version-specific. (Requires understanding of the internal 353e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat representation of the filter mask.) 354e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 355e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat \item As noted above, the events are passed up via a series of buffers 356e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat stored into debugfs files. The size and number of buffers can be 357e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat specified via the \emph{-b} and \emph{-n} arguments respectively. 358e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 359e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat \item blktrace stores the extracted data into files stored in the 360e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat \emph{local} directory. The format of the file names is (by default) 361e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat \emph{device}.blktrace.\emph{cpu}, where \emph{device} is the base 362e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat device name (e.g, if we are tracing /dev/sda, the base device name would 363e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat be \emph{sda}); and \emph{cpu} identifies a CPU for the event stream. 364e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 365e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat The \emph{device} portion of the event file name can be changed via 366e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat the \emph{-o} option. 367e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 368e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat \item blktrace may also be run concurrently with blkparse to produce 369e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat \emph{live} output -- to do this specify \emph{-o -} for blktrace. 370e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 371e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat \item The default behaviour for blktrace is to run forever until explicitly killed by the user (via a control-C, or \emph{kill} utility invocation). There are two ways to modify this: 372e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 373e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat \begin{enumerate} 374e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat \item You may utilize the blktrace utility itself to \emph{kill} 375e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat a running trace -- via the \emph{-k} option. 376e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 377e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat \item You can specify a run-time duration for blktrace via the 378e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat \emph{-w} option -- then blktrace will run for the specified number 379e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat of seconds, and then halt. 380e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat \end{enumerate} 381e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\end{itemize} 382e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 383e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\subsection{\label{sec:blktrace-args}Command line arguments} 384e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\begin{tabular}{|l|l|l|}\hline 385e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatShort & Long & Description \\ \hline\hline 386e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat-A \emph{hex-mask} & --set-mask=\emph{hex-mask} & Set filter mask to \emph{hex-mask} \\ \hline 387e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat-a \emph{mask} & --act-mask=\emph{mask} & Add \emph{mask} to current filter (see below for masks) \\ \hline 388e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat-b \emph{size} & --buffer-size=\emph{size} & Specifies buffer size for event extraction (scaled by $2^{10}$) \\ \hline 389e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat-d \emph{dev} & --dev=\emph{dev} & Adds \emph{dev} as a device to trace \\ \hline 390e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat-k & --kill & Kill on-going trace \\ \hline 391e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat-n \emph{num-sub} & --num-sub=\emph{num-sub} & Specifies number of buffers to use \\ \hline 392e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat-o \emph{file} & --output=\emph{file} & Prepend \emph{file} to output file name(s) \\ \hline 393e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat-r \emph{rel-path} & --relay=\emph{rel-path} & Specifies debugfs mount point \\ \hline 394e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat-V & --version & Outputs version \\ \hline 395e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat-w \emph{seconds} & --stopwatch=\emph{seconds} & Sets run time to the number of seconds specified \\ \hline 396e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat-I \emph{devs file}& --input-devs=\emph{devs file}& Adds devices found in \emph{devs file} to list of devices to trace. \\ 397e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat & & (One device per line.) \\ \hline 398e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\end{tabular} 399e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 400e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\subsubsection{\label{sec:filter-mask}Filter Masks} 401e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatThe following masks may be passed with the \emph{-a} command line 402e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehatoption, multiple filters may be combined via multiple \emph{-a} command 403e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehatline options.\smallskip 404e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 405e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\begin{tabular}{|l|l|}\hline 406e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehatbarrier & \emph{barrier} attribute \\ \hline 407e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehatcomplete & \emph{completed} by driver \\ \hline 408e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehatfs & \emph{FS} requests \\ \hline 409e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehatissue & \emph{issued} to driver \\ \hline 410e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehatpc & \emph{packet command} events \\ \hline 411e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehatqueue & \emph{queue} operations \\ \hline 412e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehatread & \emph{read} traces \\ \hline 413e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehatrequeue & \emph{requeue} operations \\ \hline 414e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehatsync & \emph{synchronous} attribute \\ \hline 415e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehatwrite & \emph{write} traces \\ \hline 416e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehatnotify & \emph{notify} trace messages \\ \hline 417e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\end{tabular} 418e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 419e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\subsubsection{\label{sec:request-types}Request types} 420e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehatblktrace disguingishes between two types of block layer requests, 421e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehatfile system and scsi commands. The former are dubbed \emph{fs} 422e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehatrequests, the latter \emph{pc} requests. File system requests are 423e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehatnormal read/write operations, ie any type of read or write from a 424e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehatspecific disk location at a given size. These requests typically 425e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehatoriginate from a user process, but they may also be initiated by 426e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehatthe vm flushing dirty data to disk or the file system syncing 427e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehata super or journal block to disk. \emph{pc} requests are SCSI 428e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehatcommands. blktrace sends the command data block as a payload 429e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehatso that blkparse can decode it. 430e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 431e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat%---------------------------- 432e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\newpage\section{\label{sec:blkparse-ug}blkparse User Guide} 433e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 434e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatThe \emph{blkparse} utility will attempt to combine streams of events 435e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehatfor various devices on various CPUs, and produce a formatted output of 436e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehatthe event information. As with blktrace, some details concerning blkparse 437e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehatwill help in understanding the command line options presented below. 438e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 439e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\begin{itemize} 440e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat \item By default, blkparse expects to run in a post-processing mode 441e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat -- one where the trace events have been saved by a previous run 442e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat of blktrace, and blkparse is combining event streams and dumping 443e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat formatted data. 444e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 445e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat blkparse \emph{may} be run in a \emph{live} manner concurrently with 446e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat blktrace by specifying \emph{-i -} to blkparse, and combining it with 447e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat the live option for blktrace. An example would be: 448e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 449e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat \begin{verbatim} 450e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat % blktrace -d /dev/sda -o - | blkparse -i - 451e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat \end{verbatim} 452e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 453e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat \item You can set how many blkparse batches event reads via the 454e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat \emph{-b} option, the default is to handle events in batches of 512. 455e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 456e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat \item If you have saved event traces in blktrace with different output 457e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat names (via the \emph{-o} option to blktrace), you must specify the 458e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat same \emph{input} name via the \emph{-i} option. 459e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 460e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat \item The format of the output data can be controlled via the \emph{-f} 461e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat or \emph{-F} options -- see section~\ref{sec:blkparse-format} for details. 462e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 463e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat By default, blkparse sends formatted data to standard output. This may 464e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat be changed via the \emph{-o} option, or text output can be disabled 465e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat via the\emph{-O} option. A merged binary stream can be produced using 466e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat the \emph{-d} option. 467e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 468e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\end{itemize} 469e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 470e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\newpage\subsection{\label{sec:blkparse-args}Command line arguments} 471e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\begin{tabular}{|l|l|l|}\hline 472e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatShort & Long & Description \\ \hline\hline 473e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat-b \emph{batch} & --batch={batch} & Standard input read batching \\ \hline 474e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 475e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat-i \emph{file} & --input=\emph{file} & Specifies base name for input files -- default is \emph{device}.blktrace.\emph{cpu}. \\ 476e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat & & As noted above, specifying \emph{-i -} runs in \emph{live} mode with blktrace \\ 477e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat & & (reading data from standard in). \\ \hline 478e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 479e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat-F \emph{typ,fmt} & --format=\emph{typ,fmt} & Sets output format \\ 480e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat-f \emph{fmt} & --format-spec=\emph{fmt} & (See section~\ref{sec:blkparse-format} for details.) \\ 481e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat & & \\ 482e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat & & The -f form specifies a format for all events \\ 483e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat & & \\ 484e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat & & The -F form allows one to specify a format for a specific \\ 485e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat & & event type. The single-character \emph{typ} field is one of the \\ 486e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat & & action specifiers in section~\ref{sec:act-table} \\ \hline 487e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 488e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 489e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat-m & --missing & Print missing entries\\ \hline 490e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 491e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat-h & --hash-by-name & Hash processes by name, not by PID\\ \hline 492e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 493e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat-o \emph{file} & --output=\emph{file} & Output file \\ \hline 494e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat-O & --no-text-output & Do \emph{not} produce text output, used for binary (-d) only \\ \hline 495e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 496e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat-d \emph{file} & --dump-binary=\emph{file} & Binary output file \\ \hline 497e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 498e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat-q & --quiet & Quite mode \\ \hline 499e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 500e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat-s & --per-program-stats & Displays data sorted by program \\ \hline 501e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 502e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat-t & --track-ios & Display time deltas per IO \\ \hline 503e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 504e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat-w \emph{span} & --stopwatch=\emph{span} & Display traces for the \emph{span} specified -- where span can be: \\ 505e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat & & \emph{end-time} -- Display traces from time 0 through \emph{end-time} (in ns) \\ 506e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat & & or \\ 507e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat & & \emph{start:end-time} -- Display traces from time \emph{start} \\ 508e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat & & through {end-time} (in ns). \\ \hline 509e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 510e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat-M & --no-msgs & Do not add messages to binary output file \\\hline 511e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat-v & --verbose & More verbose marginal on marginal errors \\ \hline 512e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat-V & --version & Display version \\ \hline 513e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 514e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\end{tabular} 515e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 516e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\newpage 517e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\subsection{\label{sec:blkparse-actions}Trace actions} 518e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 519e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\begin{description} 520e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat \item[C -- complete] A previously issued request has been completed. 521e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat The output will detail the sector and size of that request, as well 522e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat as the success or failure of it. 523e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 524e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat \item[D -- issued] A request that previously resided on the block layer 525e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat queue or in the io scheduler has been sent to the driver. 526e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 527e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat \item[I -- inserted] A request is being sent to the io scheduler for 528e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat addition to the internal queue and later service by the driver. The 529e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat request is fully formed at this time. 530e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 531e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat \item[Q -- queued] This notes intent to queue io at the given location. 532e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat No real requests exists yet. 533e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 534e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat \item[B -- bounced] The data pages attached to this \emph{bio} are 535e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat not reachable by the hardware and must be bounced to a lower memory 536e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat location. This causes a big slowdown in io performance, since the data 537e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat must be copied to/from kernel buffers. Usually this can be fixed with 538e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat using better hardware - either a better io controller, or a platform 539e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat with an IOMMU. 540e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 541e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat \item[m -- message] Text message generated via kernel call to 542e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat \texttt{blk\_add\_trace\_msg}. 543e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 544e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat \item[M -- back merge] A previously inserted request exists that ends 545e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat on the boundary of where this io begins, so the io scheduler can merge 546e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat them together. 547e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 548e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat \item[F -- front merge] Same as the back merge, except this io ends 549e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat where a previously inserted requests starts. 550e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 551e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat \item[G -- get request] To send any type of request to a block device, 552e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat a \emph{struct request} container must be allocated first. 553e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 554e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat \item[S -- sleep] No available request structures were available, so 555e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat the issuer has to wait for one to be freed. 556e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 557e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat \item[P -- plug] When io is queued to a previously empty block device 558e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat queue, Linux will plug the queue in anticipation of future ios being 559e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat added before this data is needed. 560e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 561e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat \item[U -- unplug] Some request data already queued in the device, 562e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat start sending requests to the driver. This may happen automatically 563e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat if a timeout period has passed (see next entry) or if a number of 564e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat requests have been added to the queue. 565e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 566e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat \item[T -- unplug due to timer] If nobody requests the io that was queued 567e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat after plugging the queue, Linux will automatically unplug it after a 568e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat defined period has passed. 569e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 570e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat \item[X -- split] On raid or device mapper setups, an incoming io may 571e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat straddle a device or internal zone and needs to be chopped up into 572e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat smaller pieces for service. This may indicate a performance problem due 573e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat to a bad setup of that raid/dm device, but may also just be part of 574e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat normal boundary conditions. dm is notably bad at this and will clone 575e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat lots of io. 576e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 577e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat \item[A -- remap] For stacked devices, incoming io is remapped to device 578e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat below it in the io stack. The remap action details what exactly is 579e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat being remapped to what. 580e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 581e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\end{description} 582e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 583e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\subsection{\label{sec:blkparse-format}Output Description and Formatting} 584e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 585e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatThe output from blkparse can be tailored for specific use - in particular, 586e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehatto ease parsing of output, and/or limit output fields to those the user 587e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehatwants to see. The data for fields which can be output include: 588e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 589e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\smallskip 590e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\begin{tabular}{|l|l|}\hline 591e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatField & Description \\ 592e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatSpecifier & \\ \hline\hline 593e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\emph{a} & Action, a (small) string (1 or 2 characters) -- see table below for more details \\ \hline 594e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\emph{c} & CPU id \\ \hline 595e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\emph{C} & Command \\ \hline 596e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\emph{d} & RWBS field, a (small) string (1-3 characters) -- see section below for more details \\ \hline 597e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\emph{D} & 7-character string containing the major and minor numbers of 598e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehatthe event's device \\ 599e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat & (separated by a comma). \\ \hline 600e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\emph{e} & Error value \\ \hline 601e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\emph{m} & Minor number of event's device. \\ \hline 602e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\emph{M} & Major number of event's device. \\ \hline 603e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\emph{n} & Number of blocks \\ \hline 604e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\emph{N} & Number of bytes \\ \hline 605e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\emph{p} & Process ID \\ \hline 606e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\emph{P} & Display packet data -- series of hexadecimal values\\ \hline 607e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\emph{s} & Sequence numbers \\ \hline 608e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\emph{S} & Sector number \\ \hline 609e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\emph{t} & Time stamp (nanoseconds) \\ \hline 610e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\emph{T} & Time stamp (seconds) \\ \hline 611e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\emph{u} & Elapsed value in microseconds (\emph{-t} command line option) \\ \hline 612e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\emph{U} & Payload unsigned integer \\ \hline 613e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\end{tabular} 614e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 615e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatNote that the user can optionally specify field display width, and 616e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehatoptionally a left-aligned specifier. These precede field specifiers, 617e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehatwith a '\%' character, followed by the optional left-alignment specifer 618e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat(-) followed by the width (a decimal number) and then the field. 619e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 620e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatThus, to specify the command in a 12-character field that is left aligned: 621e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 622e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\begin{verbatim} 623e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat-f "%-12C" 624e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\end{verbatim} 625e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 626e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\newpage 627e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\subsubsection{\label{sec:act-table}Action Table} 628e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatThe following table shows the various actions which may be output. 629e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 630e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\begin{tabular}{|l|l|}\hline 631e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatAct & Description \\ \hline\hline 632e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatA & IO was remapped to a different device \\ \hline 633e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatB & IO bounced \\ \hline 634e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatC & IO completion \\ \hline 635e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatD & IO issued to driver \\ \hline 636e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatF & IO front merged with request on queue \\ \hline 637e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatG & Get request \\ \hline 638e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatI & IO inserted onto request queue \\ \hline 639e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatM & IO back merged with request on queue \\ \hline 640e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatP & Plug request \\ \hline 641e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatQ & IO handled by request queue code \\ \hline 642e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatS & Sleep request \\ \hline 643e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatT & Unplug due to timeout \\ \hline 644e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatU & Unplug request \\ \hline 645e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatX & Split \\ \hline 646e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\end{tabular} 647e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 648e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\subsubsection{\label{sec:act-table}RWBS Description} 649e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatThis is a small string containing at least one character ('R' for read, 650e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat'W' for write, or 'D' for block discard operation), and optionally either 651e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehata 'B' (for barrier operations) or 'S' (for synchronous operations). 652e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 653e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\subsubsection{\label{sec:default-output}Default output} 654e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 655e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatThe standard \emph{header} (or initial fields displayed) include: 656e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 657e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\begin{verbatim} 658e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat"%D %2c %8s %5T.%9t %5p %2a %3d " 659e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\end{verbatim} 660e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 661e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatBreaking this down: 662e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 663e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\begin{description} 664e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat \item[\%D] Displays the event's device major/minor as: \%3d,\%-3d. 665e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat \item[\%2c] CPU ID (2-character field). 666e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat \item[\%8s] Sequence number 667e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat \item[\%5T.\%9t] 5-charcter field for the seconds portion of the 668e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat time stamp and a 9-character field for the nanoseconds in the time stamp. 669e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat \item[\%5p] 5-character field for the process ID. 670e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat \item[\%2a] 2-character field for one of the actions. 671e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat \item[\%3d] 3-character field for the RWBS data. 672e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\end{description} 673e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 674e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatSeeing this in action: 675e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 676e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\begin{verbatim} 677e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 8,0 3 1 0.000000000 697 G W 223490 + 8 [kjournald] 678e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\end{verbatim} 679e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 680e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatThe header is the data in this line up to the 223490 (starting block). 681e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 682e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatThe default output for all event types includes this header. 683e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 684e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\paragraph{Default output per action} 685e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 686e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\begin{description} 687e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat \item[C -- complete] If a payload is present, this is presented between 688e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat parenthesis following the header, followed by the error value. 689e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 690e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat If no payload is present, the sector and number of blocks are presented 691e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat (with an intervening plus (+) character). If the \emph{-t} option 692e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat was specified, then the elapsed time is presented. In either case, 693e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat it is followed by the error value for the completion. 694e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 695e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat \item[D -- issued] 696e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat \item[I -- inserted] 697e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat \item[Q -- queued] 698e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat \item[B -- bounced] If a payload is present, the number of payload bytes 699e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat is output, followed by the payload in hexadecimal between parenthesis. 700e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 701e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat If no payload is present, the sector and number of blocks are presented 702e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat (with an intervening plus (+) character). If the \emph{-t} option was 703e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat specified, then the elapsed time is presented (in parenthesis). In 704e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat either case, it is followed by the command associated with the event 705e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat (surrounded by square brackets). 706e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 707e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat \item[M -- back merge] 708e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat \item[F -- front merge] 709e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat \item[G -- get request] 710e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat \item[S -- sleep] The starting sector and number of blocks is output 711e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat (with an intervening plus (+) character), followed by the command 712e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat associated with the event (surrounded by square brackets). 713e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 714e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat \item[P -- plug] The command associated with the event (surrounded by 715e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat square brackets) is output. 716e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 717e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat \item[U -- unplug] 718e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat \item[T -- unplug due to timer] The command associated with the event 719e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat (surrounded by square brackets) is output, followed by the number of 720e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat requests outstanding. 721e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 722e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat \item[X -- split] The original starting sector followed by the new 723e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat sector (separated by a slash (/) is output, followed by the command 724e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat associated with the event (surrounded by square brackets). 725e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 726e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat \item[A -- remap] Sector and length is output, along with the original 727e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat device and sector offset. 728e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 729e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat \item[m -- message] The supplied message is appended to the end of 730e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat the standard header. 731e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 732e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\end{description} 733e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 734e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat%------------------------------ 735e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\newpage 736e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\newpage\section*{\label{sec:blktrace-kg}Appendix: blktrace Kernel Guide} 737e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 738e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatThe blktrace facility provides an efficient event transfer mechanism which 739e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehatsupplies block IO layer state transition data via the relay 740e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehatfilesystem. This section provides some details as to the interfaces 741e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehatblktrace utilizes in the kernel to effect this. It is good background data 742e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehatto help understand some of the outputs and command-line options above. 743e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 744e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\subsection{blktrace.h Definitions} 745e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatFiles which include $<linux/blktrace.h>$ are supplied with the following 746e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehatdefinitions: 747e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 748e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\subsubsection{Trace Action Specifiers} 749e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\begin{tabular}{|l|l|}\hline 750e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat BLK\_TA\_QUEUE & (RQ) Command queued to request\_queue. \\ 751e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat & (BIO) Command queued by elevator. \\ \hline 752e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat BLK\_TA\_BACKMERGE & Back merging elevator operation \\ \hline 753e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat BLK\_TA\_FRONTMERGE & Front merging elevator operation \\ \hline 754e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat BLK\_TA\_GETRQ & Free request retrieved. \\ \hline 755e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat BLK\_TA\_SLEEPRQ & No requests available, device unplugged. \\ \hline 756e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat BLK\_TA\_REQUEUE & Request requeued. \\ \hline 757e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat BLK\_TA\_ISSUE & Command set to driver for request\_queue. \\ \hline 758e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat BLK\_TA\_COMPLETE & Command completed by driver. \\ \hline 759e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat BLK\_TA\_PLUG & Device is plugged \\ \hline 760e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat BLK\_TA\_UNPLUG\_IO & Unplug device as IO is made available. \\ \hline 761e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat BLK\_TA\_UNPLUG\_TIMER & Unplug device after timer expired. \\ \hline 762e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat BLK\_TA\_INSERT & Insert request into queue. \\ \hline 763e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat BLK\_TA\_SPLIT & BIO split into 2 or more requests. \\ \hline 764e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat BLK\_TA\_BOUNCE & BIO was bounced \\ \hline 765e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat BLK\_TA\_REMAP & BIO was remapped \\ \hline 766e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\end{tabular} 767e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 768e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat%.......................................... 769e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\subsection{blktrace.h Routines} 770e20e1347b9914aa05e30548c15d7cd5e412cc0e2San MehatFiles which include $<linux/blktrace.h>$ are supplied with the following 771e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehatkernel routine invocable interfaces: 772e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 773e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\begin{description} 774e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat \item[blk\_add\_trace\_rq(struct request\_queue *q, struct request\_queue 775e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat *rq, u32 what)] 776e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat Adds a trace event describing the state change of the passed in 777e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat request\_queue. The \emph{what} parameter describes the change in 778e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat the request\_queue state, and is one of the request queue action 779e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat specifiers -- BLK\_TA\_QUEUE, BLK\_TA\_REQUEUE, BLK\_TA\_ISSUE, 780e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat or BLK\_TA\_COMPLETE. 781e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 782e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat \item[blk\_add\_trace\_bio(struct request\_queue *q, struct bio *bio, 783e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat u32 what)] 784e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat Adds a trace event for the BIO passed in. The \emph{what} parameter 785e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat describes the action being performed on the BIO, and is one of 786e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat BLK\_TA\_BACKMERGE, BLK\_TA\_FRONTMERGE, or BLK\_TA\_QUEUE. 787e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 788e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat \item[blk\_add\_trace\_generic(struct request\_queue *q, struct bio *bio, 789e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat int rw, u32 what)] 790e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat Adds a \emph{generic} trace event -- not one of the request queue 791e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat or BIO traces. The \emph{what} parameter describes the action being 792e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat performed on the BIO (if bio is non-NULL), and is one of 793e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat BLK\_TA\_PLUG, BLK\_TA\_GETRQ or BLK\_TA\_SLEEPRQ. 794e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 795e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat \item[blk\_add\_trace\_pdu\_int(struct request\_queue *q, u32 what, 796e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat u32 pdu)] 797e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat Adds a trace with some payload data -- in this case, an unsigned 798e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 32-bit entity (the \emph{pdu} parameter). The \emph{what} parameter 799e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat describes the nature of the payload, and is one of 800e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat BLK\_TA\_UNPLUG\_IO or BLK\_TA\_UNPLUG\_TIMER. 801e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 802e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat \item[blk\_add\_trace\_remap(struct request\_queue *q, struct bio *bio, 803e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat dev\_t dev, sector\_t sector)] 804e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat Adds a trace with a remap event. \emph{dev} and \emph{sector} denote 805e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat the original device this \emph{bio} was mapped from. 806e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 807e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat \item[blk\_add\_trace\_msg(struct request\_queue *q, char *fmt, ...)] 808e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat Adds a formatted message to the output stream. The total message 809e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat size can not exceed BLK\_TN\_MSG\_MSG characters (currently 810e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 1024). Standard format conversions are supported (as supplied 811e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat by \texttt{vscnprintf}. 812e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat 813e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\end{description} 814e20e1347b9914aa05e30548c15d7cd5e412cc0e2San Mehat\end{document} 815