History log of /external/fio/backend.c
Revision Date Author Comments (<<< Hide modified files) (Show modified files >>>)
842d2e2a85cb0f1ef11409a4f3b09d8a2e7a68cc 24-Feb-2015 Jens Axboe <axboe@fb.com> Make 'loops' honor full write+verify loops

Signed-off-by: Jens Axboe <axboe@fb.com>
/external/fio/backend.c
1b35740d5492c4a0a163ec49996265a8cd5e0100 29-Jan-2015 Justin Eno <jeno@micron.com> Allow random overwrite workloads to write io_limit

Limiting the write phase of (random) overwrite workloads to 'size'
before the verify phase doesn't make sense, so allow them to write
up to io_limit. This better matches verify_only behavior, and
allows the [verify-job] below to succeed:

[global]
thread=1
direct=1
ioengine=sync
verify=meta
bs=1k
offset=0
size=5M
io_limit=6M
norandommap=1
verify_fatal=1
randseed=42

[write-job]
filename=datafile.tmp
rw=randwrite
do_verify=0

[verify-job]
stonewall
create_serialize=0
filename=datafile.tmp
rw=randwrite
do_verify=1
verify_only=1

Signed-off-by: Justin Eno <jeno@micron.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
/external/fio/backend.c
7318762572051fc71be15d75dd3fb05e88f70717 09-Feb-2015 Gwendal Grignou <gwendal@chromium.org> verify: Fix latency log for verify commands.

When commands when requeued for the verify operation,
their start time was not reset, resulting in bogus latency graphs.

Signed-off-by: Gwendal Grignou <gwendal@chromium.org>
Signed-off-by: Jens Axboe <axboe@fb.com>
/external/fio/backend.c
1b7ab803d8914d355bbb61e81e3d745592c04fce 28-Jan-2015 Justin Eno <jeno@micron.com> Fix for verify_only (do_dry_run()) broken by 74d6277f

Previous commit to backend.c tracks io_limit more closely by
counting submitted (in-flight) I/O instead of completed I/O.
do_dry_run() does not submit I/O, so its I/O is not counted,
and it loops forever.
Signed-off-by: Justin Eno <jeno@micron.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
/external/fio/backend.c
5d61d116ccd97167a78348557cca653fc8c73ade 16-Jan-2015 Jens Axboe <axboe@fb.com> Improve precision of the io_limit setting

For async engines, we look only at completions. But we could have
a bunch inflight with a high queue depth, making us go higher than
we should.

Signed-off-by: Jens Axboe <axboe@fb.com>
/external/fio/backend.c
13e34359a8bbce50b65a531d5caf294fac3c2746 07-Jan-2015 Jens Axboe <axboe@fb.com> Fix disk utils being updated too often

Fixes: f893b76d5745
Signed-off-by: Jens Axboe <axboe@fb.com>
/external/fio/backend.c
42ec5758abe080ab5e6d0e3cbe379308534bd133 16-Dec-2014 Jens Axboe <axboe@fb.com> backend: use monotonic clock for ETA, if we have it

Signed-off-by: Jens Axboe <axboe@fb.com>
/external/fio/backend.c
581b5df3b18dd95d77efad330e7f4c811682c76b 15-Dec-2014 Jens Axboe <axboe@fb.com> backend: fix off-by-one in nsec calculation

If it's equal to 10^9 we need to bump seconds and subtract 10^9
from nsec, not only if it's larger than.

Signed-off-by: Jens Axboe <axboe@fb.com>
/external/fio/backend.c
b2c52da8436eac7a09a22613f600b76c45c8d0e2 12-Dec-2014 Jens Axboe <axboe@fb.com> backend: fix should_check_rate() typo in wait_for_completions()

Signed-off-by: Jens Axboe <axboe@fb.com>
/external/fio/backend.c
24528e7db9b66f7fa77304f467589844ef78b635 12-Dec-2014 Jens Axboe <axboe@fb.com> backend: abstract out waiting for completions

We do it both in do_verify() and in do_io(), so it's duplicated code.
And we want to potentially add a completor thread, so lets move this
to a helper.

Signed-off-by: Jens Axboe <axboe@fb.com>
/external/fio/backend.c
1006b2d6b019b03ef34b76f6a804d0c690f0b917 09-Dec-2014 Jens Axboe <axboe@fb.com> Use fio_option_is_set() for ioprio setting

Signed-off-by: Jens Axboe <axboe@fb.com>
/external/fio/backend.c
94b360f41976a2bd3c4d3476b5e26a96e1ea9b12 09-Dec-2014 Jens Axboe <axboe@fb.com> Get rid of _set variables

We can just check the option set mask now.

Signed-off-by: Jens Axboe <axboe@fb.com>
/external/fio/backend.c
07f512cd4b6c988c9130eabdd76ffc8df6045123 19-Nov-2014 Jens Axboe <axboe@fb.com> trigger: always send trigger, not just when remote command is set

Signed-off-by: Jens Axboe <axboe@fb.com>
/external/fio/backend.c
7eebf50b44c46d73411608887057efc47882a3d2 19-Nov-2014 Jens Axboe <axboe@fb.com> client: defer local trigger execute until after state is received

Don't run it immediately, we have to wait until we receive the
data from the server.

Signed-off-by: Jens Axboe <axboe@fb.com>
/external/fio/backend.c
a4432079c42dbccbc1e5fb460454e58bbe9f7e17 19-Nov-2014 Jens Axboe <axboe@fb.com> trigger: enable separate remote and local trigger

Change the options so that we have the trigger file and command
separate, and remote and local trigger command separate as well.

Signed-off-by: Jens Axboe <axboe@fb.com>
/external/fio/backend.c
de54cfd8b8e93d2a32a02961f1587b83f0763aa8 11-Nov-2014 Jens Axboe <axboe@fb.com> Add support for verify triggers and verify state saving

This allows you to (for instance) instantly terminate a verify
write workload, and then later that everything was written
correctly up to that very point. This can be useful for testing
powercut scenarios, which is often problematic on storage
devices.

The trigger part is a file based notification scheme, similar
to what is provided for the status dumps. When triggered,
fio will exit immediately and write the verify state safely
to disk. A trigger can be accompanied by a trigger command.
Say you wanted to test powercut safety, the trigger could be
something that immediately cut power to the machine.

The verify state is either saved locally (if run locally),
or saved over the network if run in client/server mode.

Signed-off-by: Jens Axboe <axboe@fb.com>
/external/fio/backend.c
b89f89e64592907195d5642abd33e696dbd716a4 10-Nov-2014 Jens Axboe <axboe@fb.com> Fix crash on threads being reaped before they are created

If we do that, then ->rusage_sem is never allocated.

Signed-off-by: Jens Axboe <axboe@fb.com>
/external/fio/backend.c
c38e38f16f77567a423f75c26e1929cbe4902e24 10-Nov-2014 Vasily Tarasov <tarasov@vasily.name> Fix for a race when fio prints I/O statistics periodically

Below is the demonstration for the latest code in git:

Job-file:

[job]
filename=/dev/sda
rw=randread
blocksize=4096
direct=1
time_based=1
runtime=300
numjobs=16

Command line:

fio --output-format=json --status-interval=10 /tmp/job.file | tee /tmp/fio.out

After parsing json and printing per-job "read->runtime" and "read->iops" in two
columns, one can see this:

runtime iops
9854 4762
19861 4724
29867 4711
39873 4705
49880 4701
59886 4698
69894 4696
79900 4694
89907 4693
99913 4692
109919 4692
119926 4691
129933 4691
139940 4691
149946 4691
159953 4690
169959 4690
179966 4690
189972 4690
199978 4690
209984 4690
219990 4690
229996 4690
240002 4690
250009 4690
260015 4690
270021 4690
280027 4690
290033 4690
600043 2344 <--- You can see a spurious jump in runtime here
300001 4690

The problem is two-fold, IMHO:

1) td->ts.runtime[DDIR_READ/DDIR_WRITE/DDIR_TRIM] are set in microseconds
instead of milliseconds in thread_main() function. The values are later
converted to milliseconds (after exiting from the loop in thread_main()),
but it is too late for the out-of-band access by the helper_thread
(the one that prints periodic statistics).

2) The helper_thread in __show_running_run_stats() temporarily adds to
td->ts.runtime[DDIR_READ/DDIR_WRITE/DDIR_TRIM]
the time that has passed since the beginning of the loop (td->start).
However, the same addition happens in thread_main() by the job
thread and td->start is not reset atomically with this addition.
As a result, the runtime can be added twice. (That's why in the output
we see 600043, which is approximately twice larger than the
previous reading).

My proposal is to make the job threads updating both td->ts.runtime and
td->start atomically by using an already existing stat_mutex.
__show_running_run_stats() already takes this mutex where needed.

Signed-off-by: Jens Axboe <axboe@fb.com>
/external/fio/backend.c
0f7f9a918590b45589224568245f75e2dbff9e14 06-Nov-2014 Jens Axboe <axboe@fb.com> Make fio -Wshadow clean

Found a few issues, actually.

Signed-off-by: Jens Axboe <axboe@fb.com>
/external/fio/backend.c
f678701783f1c200ce574215feeb7ff801c4ff72 06-Nov-2014 Jens Axboe <axboe@fb.com> Get rid of __ prefix for internal frand state

It made (some) sense when we had both OS and frand in anon unions,
but now lets just kill it.

Signed-off-by: Jens Axboe <axboe@fb.com>
/external/fio/backend.c
1266d3fe08ef1414e92fbb7825234ee3679f8a89 24-Oct-2014 Jens Axboe <axboe@fb.com> diskutil: get rid of disk_util_start_exit()

No need to wrap this in a function, just set the exit variable.

Signed-off-by: Jens Axboe <axboe@fb.com>
/external/fio/backend.c
7fe36313273ef051670d16aa27953699fd5cdf06 24-Oct-2014 Jens Axboe <axboe@fb.com> Consolidate disk util, eta, and status check thread

We don't need two, we can just have one do everything.

Signed-off-by: Jens Axboe <axboe@fb.com>
/external/fio/backend.c
04247864f004c0a20ceb9fb0c0ce3c3a3c512326 24-Oct-2014 Jens Axboe <axboe@fb.com> stat: avoid hang with race in stat thread exit and rusage update

Signed-off-by: Jens Axboe <axboe@fb.com>
/external/fio/backend.c
fdb0da8028e156c0da43aca18e1423d1b300bdad 23-Oct-2014 Jens Axboe <axboe@fb.com> Start and stop status interval (or file) thread separately

This gets rid of the need to create threads on the fly for this,
and we can manage it and stop it before we do real stats at
the end.

Signed-off-by: Jens Axboe <axboe@fb.com>
/external/fio/backend.c
a2e694def8d85ee79e91f014cd2a5d5fc7dd9022 23-Oct-2014 Jens Axboe <axboe@fb.com> backend: remove debug du thread exit notice

Signed-off-by: Jens Axboe <axboe@fb.com>
/external/fio/backend.c
925296a6991eda56081454d96af2bbd73d0f47af 23-Oct-2014 Jens Axboe <axboe@fb.com> Get rid of (now unused) disk_util_mutex

Signed-off-by: Jens Axboe <axboe@fb.com>
/external/fio/backend.c
62244c9dbfa64f54a61b26af8f15a722362ac41a 23-Oct-2014 Jens Axboe <axboe@fb.com> Fix exit being slowed by disk util stat time

Add a cv/lock to handle exit of the disk util thread. This is a
lot cleaner than the adhoc method that was used before, and it
means that we can signal the exit immediately instead of waiting
for DISKUTIL_MSEC to pass.

Signed-off-by: Jens Axboe <axboe@fb.com>
/external/fio/backend.c
bc2e69cd37641b8ddc64885fce859449cb5b8035 13-Oct-2014 Jens Axboe <axboe@fb.com> backend: use log_info_flush(), not fflush()

f_out may not be stdout, so use the appropriate wrapper
that flushes the appropriate FILE (or nothing, if nothing
needs to be done).

Signed-off-by: Jens Axboe <axboe@fb.com>
/external/fio/backend.c
2e627243f7877cefb3606ef99ec24debe32ac98c 25-Jul-2014 Jens Axboe <axboe@fb.com> Don't grab stat mutex for final stat output

We should not need it, and there's a bug with getting stuck on it
that hasn't been fully debugged/understood yet. So for now, lets
just forget about the mutex for final output.

Signed-off-by: Jens Axboe <axboe@fb.com>
/external/fio/backend.c
3b44e8bb2a96110472c3b5132d5115f903e354ed 09-Jul-2014 Jens Axboe <axboe@fb.com> Fix cases where td->terminate is set, but terminate clock not marked

Signed-off-by: Jens Axboe <axboe@fb.com>
/external/fio/backend.c
958b891fdbdc4f0558e0907d7c2e3a1ed4964bfa 09-Jul-2014 Jens Axboe <axboe@fb.com> Allow threads 60 seconds to exit before being forceful

Right now the fio status thread can sit forever waiting for jobs
to exit, when ctrl-c (or another signal) is sent. Be a bit more
brutal and force quit jobs if they haven't exited on their own
in 60 seconds. That should be long enough to ensure that they
are stuck in some way.

Signed-off-by: Jens Axboe <axboe@fb.com>
/external/fio/backend.c
65f93caca2ca0dd8d8150831f16854b3a8c3845a 03-Jul-2014 Jens Axboe <axboe@fb.com> Move tp.[ch] to lib/

It's just helper code, not fio core.

Signed-off-by: Jens Axboe <axboe@fb.com>
/external/fio/backend.c
38a812d7fa79a673855efb6324c8fc68b92b89c1 03-Jul-2014 Jens Axboe <axboe@fb.com> Add support for runtime log compression

If log_compression=64M is specified, fio will "bite" off chunks of
the IO logs and runtime compress them with zlib. This can greatly
reduce the amount of memory required to do iops/bw/lat logging of
a run, at the cost of using some background CPU for the compression.

Signed-off-by: Jens Axboe <axboe@fb.com>
/external/fio/backend.c
633d825ccf450073e4489ee8ab66a44303b1a0d7 02-Jul-2014 Jens Axboe <axboe@fb.com> log: fix use-after-free

Introduced in commit cb7e0ace739c.

Signed-off-by: Jens Axboe <axboe@fb.com>
/external/fio/backend.c
987c4f49f15092a3793c3375be9116c052b5b364 02-Jul-2014 Jens Axboe <axboe@fb.com> Cleanup logging

If the log knows the filename it is supposed to log to, then we
don't need to have part of that on the init side and the other
part on the flush side. So clean it up so that the caller of
setup_log() provides everything we need. This gets rid of
various ways of ending the log, and opens us up to doing
log flushing mid-run.

Signed-off-by: Jens Axboe <axboe@fb.com>
/external/fio/backend.c
ccefd5fea1de87a83db9c372985b69f4fbbe922c 01-Jul-2014 Jens Axboe <axboe@fb.com> Add option for including byte offset for each log entry

When iops/latency/bw logs are used in the job, we can note the offset
at which the specific data entry took place.

Based on an initial patch from Abutalib Aghayev <agayev@gmail.com>.
Extended by me to:

- Not do this by default, as not to add 8 bytes to each log entry
stored.

- Add support for the client/server protocol.

- Add documentation.

We should bump the server protocol rev again, but that was just done
in the previous commit, so...

Signed-off-by: Jens Axboe <axboe@fb.com>
/external/fio/backend.c
ef7035a9b2c7af1e745b7cd2448685527ef7eeb0 19-Jun-2014 Jens Axboe <axboe@fb.com> Add support for compiling for ESX

With contributions from Ryan Haynes <rhaynes@fusionio.com>

Signed-off-by: Jens Axboe <axboe@fb.com>
/external/fio/backend.c
386626f66c67c16c4ff0032b8ceacd4b8a7eb8fa 09-Jun-2014 Jens Axboe <axboe@fb.com> Fix regression introduced by commit 334185e

We need to run fio_time_init() before the threads are invoked,
but we should do it in global context so that all threads share
the same clock and init settings.

Signed-off-by: Jens Axboe <axboe@fb.com>
/external/fio/backend.c
890b665611751661aac1030603175bb7bbde34bc 07-May-2014 Jens Axboe <axboe@fb.com> verify: fix verification of uncompleted or errored ios

Stoo Davies reports:

------

I'm doing some powerfail recovery testing on a storage array over iSCSI.
Host is RHEL 6.4 kernel 2.6.32-358.el6.x86_64.

With fio 2.1.2 -> 2.1.4 the job file below rides through the disks going
away, and continues I/O after they come back, without reporting any
errors. With fio 2.1.5 -> 2.1.8 when the disks come back fio
immediately reports a meta verification error.

I captured a trace with an finisar analyzer, and can see that after the
disks come back and the host logs back in, a read is issued for an lba
which was never written to. Since I don't see verification errors
outside of the powerfail testing, I suspect fio isn't correctly handling
failed writes during the time the disks are unavailable.

------

The issue is caused by a change between 2.1.4 and 2.1.5 that started
logging IO for verification at issue time, instead of at completion
time. If the IO only partially completes or completes in error, then
we need to adjust the verification unit so that we verify only what
fio actually wrote.

Signed-off-by: Jens Axboe <axboe@fb.com>
/external/fio/backend.c
435228488ffd062f4eac710aaa862e04cd20dfee 01-May-2014 Daniel Gollub <daniel.gollub@gmail.com> Avoid buildenv conditional in thread_option struct

Managed to run into issues with an external ioengine
which got build with CONFIG_LIBNUMA not defined. Fio
itself got build with CONFIG_LIBNUMA this resulted
in different struct members offsets in the two different
ELF objects. Causing crashes due to invalidate offsets
inside the thread_data structure (e.g. td->io_ops->data).

Ideally all structs which might be used by external
ioengines should be independent of buildenv conditionals
like CONFIG_LIBNUMA or others.

Removed the CONFIG_LIBNUMA in thread_options.h and replaced
the libnuma specific "struct bitmask" members with strings
which hold the option's input value. This should also make
the marshaling/demarshaling in cconv.c easier.
(Note: the NUMA bits are not handled in cconv.c at the
moment. And not part of the thread_options_packed struct)

Signed-off-by: Daniel Gollub <daniel.gollub@gmail.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
/external/fio/backend.c
77731b29c35286a43ca6109ebf13389169ec20e9 28-Apr-2014 Jens Axboe <axboe@fb.com> Add option for io_limit

'size' denotes both the size of the region for IO, as well as the
amount of IO to transfer. Sometimes it's useful to be able to set
them separately instead. With this patch, you can do:

size=100G
io_limit=10G

and have fio do 10G of IO between 0..100G of the address space.

Signed-off-by: Jens Axboe <axboe@fb.com>
/external/fio/backend.c
4c707a3b3ac73eb5ef00e1664a902fa0a6b69cd0 15-Apr-2014 Jens Axboe <axboe@fb.com> backend: fix potential division-by-zero

Signed-off-by: Jens Axboe <axboe@fb.com>
/external/fio/backend.c
49c6f33d1739f96c16a0ab54cde3dc4ef64b90e9 14-Apr-2014 Jens Axboe <axboe@fb.com> backend: better NULL termination fix for IO scheduler switching

Signed-off-by: Jens Axboe <axboe@fb.com>
/external/fio/backend.c
49cba9b3e5fa6b30f0fdff52eaf83427ec1baa3f 14-Apr-2014 Jens Axboe <axboe@fb.com> backend: fix potential divide-by-zero

Signed-off-by: Jens Axboe <axboe@fb.com>
/external/fio/backend.c
e0c8b741108cd59ce4435082b5afd5c69a6f713e 11-Apr-2014 Jens Axboe <axboe@fb.com> Ensure that fread() string is always NULL terminated

Signed-off-by: Jens Axboe <axboe@fb.com>
/external/fio/backend.c
905e3d4ff8d01a4402186e1ad1f5f1fbbb6b7ef4 03-Apr-2014 Jens Axboe <axboe@fb.com> Cleanup the parallellized log writing

And move it all to iolog.c, this means we can un-export
some of the log functions, too.

Signed-off-by: Jens Axboe <axboe@fb.com>
/external/fio/backend.c
243bfe190245a10e9d0981bf2a7c722edc4c43d4 02-Apr-2014 Jens Axboe <axboe@fb.com> iolog: don't serialize the writing of all logs

Serialize based on filename, so that independent logs can be
written at the same time. If the logs are big, this can take
a while to flush to disk. It's silly to have all jobs waiting
on each other, when they could write their own logs independently.

Signed-off-by: Jens Axboe <axboe@fb.com>
/external/fio/backend.c
3939fe853fd146378811a05b78731d7f20d0152f 12-Mar-2014 Jens Axboe <axboe@fb.com> Add exceeds_number_ios() helper

Commit 26251d8d open-coded the logic in two places, collapse
that into one function.

Signed-off-by: Jens Axboe <axboe@fb.com>
/external/fio/backend.c
26251d8d635bc9b9d31de2427095684b0c87b836 11-Mar-2014 Jens Axboe <axboe@fb.com> Try a little harder to honor number_ios more accurately

Previously, fio checked this on completion, which means we could
almost spill over by the iodepth setting. Check this before
issue instead, taking the in flight IOs into account as well.

Reported-by: Robert Elliott <Elliott@hp.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
/external/fio/backend.c
2e63e96b79c5153a32bec2d442f1c9a5790c17f5 04-Mar-2014 Puthikorn Voravootivat <puthik@chromium.org> Fix verify_only option not working properly

In commit c4b6117, we changed the place to call log_io_piece but
the new place won't be reached when using verify_only option so
that when we run with the verify_only option, fio won't verify
anything at all.

This patch fix that by also call the log_io_piece in the do_dry_run
function for the verify_only code path.

Signed-off-by: Puthikorn Voravootivat <puthik@chromium.org>
Reviewed-by: Gwendal Grignou <gwendal@chromium.org>
Signed-off-by: Jens Axboe <axboe@fb.com>
/external/fio/backend.c
30cb4c650a0a4c7ca79d4336ddb1fe3f988a4b2a 28-Feb-2014 Jens Axboe <axboe@fb.com> Fix off-by-one bug in CPU split mask handling

The thread number is not zero based, it starts from 1.

Signed-off-by: Jens Axboe <axboe@fb.com>
/external/fio/backend.c
c2acfbac7e9eb8ac5407ac024c0fd000614c77e2 28-Feb-2014 Jens Axboe <axboe@fb.com> Add support for cpus_allowed_policy

Two policies are supported:

shared cpumask is shared between all threads. This is the original
behavior.

split cpumask is split between threads, each thread gets a unique
CPU.

Signed-off-by: Jens Axboe <axboe@fb.com>
/external/fio/backend.c
0de5b26f6e177aacac0683306c47e0cbaf58b0b6 22-Feb-2014 Jens Axboe <axboe@fb.com> Unify the time handling

Some options are in seconds, if no units are given. These include
runtime/timeout, startdelay, and ramp_time. Handle this
appropriately. Internally it should all be microseconds now, it's
just the conversion factor.

Signed-off-by: Jens Axboe <axboe@fb.com>
/external/fio/backend.c
9900706854ce4919afb064ef6be42447f698d581 20-Feb-2014 Peter Oberparleiter <oberpar@linux.vnet.ibm.com> fio: flush log files on test end

Ensure proper flushing of all logs at the end of tests.

Signed-off-by: Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
/external/fio/backend.c
74454ce40f1a5e1e682da0a8acb824a7f6910270 20-Feb-2014 Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com> fio: allow milliseconds on all time specifiers

This patch allows all time specifiers to be specified down to
milliseconds. Default will stay seconds for compatibility with old
configs.

It also adds documentation to the existing time units day, hour and
minute.

Signed-off-by: Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
/external/fio/backend.c
002fe73409d1e3d5e7dfe2885f75885bfaf506bc 11-Feb-2014 Jens Axboe <axboe@fb.com> Fix issue with openfiles= and file sizing

Fill the given size as well as we can, given the size and nr
of files.

Fix an issue where we don't properly honor how many files to
keep open at any point in time.

Signed-off-by: Jens Axboe <axboe@fb.com>
/external/fio/backend.c
ea66e04fe1a803f6a9ddf31cb999641d4396d67c 10-Feb-2014 Jens Axboe <axboe@fb.com> Fix issue with td->mutex being used-after-free

Depending on how the OS schedules the threads, it's not necessarily
safe to immediately free a mutex after we have been woken up.
Remove the td->mutex freeing to normal cleanup after the job is done.

Reported-by: Sitsofe Wheeler <sitsofe@yahoo.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
/external/fio/backend.c
f940128526dbe468a1951cce10c2fe5dbd23875f 06-Feb-2014 Jens Axboe <axboe@fb.com> verify: always log IO in the order they are issued

We currently log verify_backlog verifies when they complete,
which means the sequence of verify and issue might be different.

Change this to log in one spot, prior to issue, and track the
completion state of the logged unit instead. This unifies the
handling of verifies.

Signed-off-by: Jens Axboe <axboe@fb.com>
/external/fio/backend.c
78a6469cfc9094763320e61c60f9aaef0ece9689 05-Feb-2014 Jens Axboe <axboe@fb.com> More precise fix for verify_backlog verification failure

Commit c2703bf3 could be a bit more precise - we only need to
add to the byte count if verify_backlog is enabled, since that
is the case where do_io() will run both the IO and verify IO.

Signed-off-by: Jens Axboe <axboe@fb.com>
/external/fio/backend.c
9a50c5c5650fdd9f3117e3745f16cc63c5b267b7 05-Feb-2014 Puthikorn Voravootivat <puthik@chromium.org> Fix verify_backlog start verification before finish writing

Due to recent fix for checking rand_seed in verify phase. In write phase,
the IO transaction will be logged before completing the actual data write.
If verify_backlog is enabled, verification may start before all data has
been written. This patch disable that fix if the verify_backlog is enabled.

Signed-off-by: Puthikorn Voravootivat <puthik@chromium.org>
Signed-off-by: Jens Axboe <axboe@fb.com>
/external/fio/backend.c
c2703bf35e146872d12f85c9e6490008b5fabb34 05-Feb-2014 Jens Axboe <axboe@fb.com> Fix for prematurely stopping on verify

Commit 20876c53 added a total byte check, but forgot to
account for potential verification bytes. Fix that.

Signed-off-by: Jens Axboe <axboe@fb.com>
/external/fio/backend.c
c4b6117b236370b174f3e3d2e299bf8b3733be97 05-Feb-2014 Puthikorn Voravootivat <puthik@chromium.org> Fix rand_seed mismatches in verify phase

In verify phase, the rand_seed generated on replay does not match
the written rand_seed.

Multiple problems are causing this:
1. In verify phase fio does not set io_u->rand_seed to compare with
hdr->rand_seed
2. In randrw scenario, fio log is stored in red-black tree in "sorted by LBA"
order. Thus, it is imposible to replay the written order, or rather
generate the seeds again in the same order.
3. In write phase, the code currently will generate rand_seed, write data
and log rand_seed. When queuedepth > 1, it's possible the writes complete
in a different order than rand_seed was generated. Thus when replaying
the log, the generated rand_seed might not match what was written.
4. verify_backlog option will start verification before all the data has been
written and it make rand_seed replay code broken with current design.

Proposed fixes:
1. Use of existing verify_state to generate verify header.
(and assumes this was the original intention of verify_state). And also
adds code to replay rand_seed in verify phase.
2. If verifysort option is not enabled, store the write log in a list instead
of the red-black tree. Otherwise, don't attempt to verify the rand_seed
in the header.
3. In write phase, generate rand_seed, log rand_seed, write data. I.e. log
IO transactions in the order generated, not completed.
4. Don't verify rand_seed when verify_backlog is enabled.

Signed-off-by: Puthikorn Voravootivat <puthik@chromium.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
621677626f2551bedfdc4a5fc3b3e5f8492b94fa 17-Sep-2013 Juan Casse <jcasse@chromium.org> Adds verify_only option

When this option is set, a dry run (no actual io is performed) of the
workload will be done in order to compute the numberio for each block
header without overwriting the data on disk. Then, do_verify() will be
effectively verifying data that was written in a previous fio run.
In the case that "loops" is set to more than 1, do_verify() will delay
the verification of numberio to the last iteration when the same
numberio state that would have been written to disk in a previous
fio run has been reached.

Signed-off-by: Juan Casse <jcasse@chromium.org>
Reviewed-by: Grant Grundler <grundler@chromium.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
ce35b1ece06ce27ebff111e47c2a6610c311a92b 14-Jan-2014 Jens Axboe <axboe@kernel.dk> Add option for specifically setting buffer contents

Fio can use zeroes, slightly scrambled data, full random data,
or specifically compressible data. With this option, the user
can now fully control the pattern written, similarly to how
verify_pattern works for verify=meta.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
3e260a46ea9a8de224c3d0a29a608da3440f284a 09-Dec-2013 Jens Axboe <axboe@kernel.dk> Add options to have fio latency profile a device

This adds three new options:

- latency_target. This defines a specific latency target, in usec.
- latency_window. This defines the period over which fio samples.
- latency_percentile. This defines the percentage of IOs that must
meet the criteria specified by latency_target/latency_window.

With these options set, fio will run the described workload and
vary the queue depth between 1 and iodepth= to find the best
performing spot that meets the criteria specified by the three
options.

A sample job file is also added to demonstrate how to use this.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
d01612f3ae2515eb035d0c4ce954d8cb167a0a61 15-Nov-2013 Chris Mason <chris.mason@fusionio.com> Add support for O_ATOMIC

O_ATOMIC makes writes atomic, meaning that they are fully stable
on media (in the event of a power cut) when acknowledged by the
device and OS.

This only truly works on Linux with the pending patches to
add O_ATOMIC.

Updated by Jens to:

- Add man page and HOWTO description of the option
- Make O_ATOMIC imply O_DIRECT, so that it actually works if you
don't set O_DIRECT manually.
- Add the option to the conversion list so it works for
client/server.
- Error handling so that if atomic=1 is set and the OS does not
support it, error out instead of just pretending it works.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
334185e9e18c36b13c713ccdbb65e7ebd84795db 01-Nov-2013 Bruce Cran <bcran@fusionio.com> server: ensure that fio_time_init() is called before option parsing

When run in server mode, jobs are processed before fio_time_init()
is called, which causes pthread_setspecific() to be called before
pthread_key_create(). Work around this by calling it in thread_main()
too.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
20876c53b5d32f2da9049af5e7fb102133946981 16-Sep-2013 Juan Casse <jcasse@chromium.org> Add condition to stop issuing io in do_io().

Problem:
When specifying iodepth > 1 while doing random async io
without a random map (and using the default random generator), the
main while loop in do_io() will issue extra io in the amount of io
allowed to accumulate in the queue. For example, a job with size=24k
and bs=8k will perform 4 io instead of the expected 3.

Reason:
The above behavior occurs because the while loop in do_io()
will continue until the amount of "completed io" >= size, but io
in the queue are not accounted for because they have not been
completed by io_u_queued_complete().

Exceptions:
The above behavior does not manifest itself when:
using random_generator=lfsr bacause lfsr knows when to stop
generating, and get_io_u() inside the while loop returns NULL,
breaking out of the loop.
using random map because random map ensures that all blocks
are hit, and after they are all hit, no more are generated.

Proposed Solution:
Stop while loop based on bytes issued instead of bytes completed.

Signed-off-by: Juan Casse <jcasse@chromium.org>
Reviewed-by: Grant Grundler <grundler@chromium.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
ce4864950d2ced54e0a1ccb8ef9a1e62135f1d2d 17-Jul-2013 Erwan Velu <erwan@enovance.com> backend: Logging exec_{pre|post}run

When using the exec_{pre|post}run option in fio jobs, it's very
important being able to read the output of thoses commands later.

As the output could be very verbose, let's log all the output of the
command in separate file.

As an example, when running two jobs called "write" and "read" the
exec_prerun option is used. During fio's execution, the following
message will be shown :
write : Saving output of prerun in write.prerun.txt
read : Saving output of prerun in read.prerun.txt

The output filename is so composed of the job name appended with the pre
or post run name and a final .txt extension.

That's pretty simple and shall do the job.
/external/fio/backend.c
8d271b66d2bc1025f760d3c5a976dac42262a178 17-Jul-2013 Erwan Velu <erwan@enovance.com> backend: Get rid of sh -c for exec_string

Running commands in a 'sh -c' does kill the existing environment values.
i.e that does kill ssh-agent.

It sounds more 'direct' to execute the command in the same environment
as fio.
/external/fio/backend.c
2ae0b204743d6b4048c6fffd46c6280a70f2ecd1 28-May-2013 Jens Axboe <axboe@kernel.dk> Replace list based free/busy/requeue list with FIFO + ring

Cache friendliness of the list is pretty low. This has
provably lower overhead.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
4a88752ad8e0b67ef9499262998d221559d6dac1 23-May-2013 Jens Axboe <axboe@kernel.dk> act: basic reporting

Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
002e7183cb86d6100efef690b6fa90bf0988b084 17-May-2013 Jens Axboe <axboe@kernel.dk> Ensure that we have no IO pending when sleeping

For rate limiting or thinktime handling, ensure that we have
no busy IO when we do spin or sleep. Otherwise we potentially
skew the latencies a lot, since events could have been reaped.

Already working for rate iops, just abstracted it out and
ensure that we do the same for thinktime. Only a problem for
certain workloads, those with queuedepth > 1.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
cef9175e52e8a240117b9f45b228fdaa0f1c0572 27-Apr-2013 Jens Axboe <axboe@kernel.dk> Ensure that stat outputs are serialized

Two issues:

- We don't want potentially two stat outputs running at the same time.
This could potentially happen if USR1 or interval stats race with
the thread exit.

- We could potentially crash if the interval or USR1 signaled stat
output trigger when fio is tearing down. Let the main thread wait
for any in-progress output in that case.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
06464907159baf7a1eeac654a023743e33f28d86 25-Apr-2013 Jens Axboe <axboe@kernel.dk> Add support for other ways of triggering intermediate result outputs

Fio has support for using the USR1 signal to dump the current
results while continuing to run. Add a --status-interval parameter
to allow this to be configured to automatically happen every X
seconds.

There's also support for signaling fio through the file system. To
do that, simply touch /tmp/fio-dump-status. When fio sees this file,
it will unlink and dump the current results.

A small tweak is needed for the file approach to work in Windows.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
fd8a09b8f4a7e0da67fe64cfb1d023bfb500d7eb 25-Apr-2013 peter chang <dpf@google.com> fix memory allocation in the cases we may need to align

when using the sg backend we crash w/ a SEGV in fio_sgio_getevents()
because of memory corruption from the IO. when we allocate the td's
backing store there's a possible post-allocation adjustment to page
align the buffers. however, the extra memory that this alignment may
need is not taken into account when allocating the buffer.

Tested:
- the problematic fio file looks like
fdha347:~# cat foo.fio
[global]
bs=64k
thread
rw=read
size=100g
time_based=1
runtime=60
iodepth=1

[sdc]
ioengine=sg
filename=/dev/sg2
numjobs=1

Signed-off-by: peter chang <dpf@google.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
4e0a8fa2593006505b7f4e18931a201d221b49e9 15-Apr-2013 Jens Axboe <axboe@kernel.dk> Fixup bad logging types

Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
8f91cc3f7835cdca84944d456e1da77d44a7c354 11-Apr-2013 Jens Axboe <axboe@kernel.dk> Merge branch 'gfio' into gfio-int

Conflicts:
backend.c
fio.c
fio.h
init.c
libfio.c
options.c
thread_options.h

Signed-off-by: Jens Axboe <axboe@kernel.dk>
5a812f9b9d987d1cacbb6f17a65835cbe0d66361 14-Mar-2012 Jens Axboe <axboe@kernel.dk> iolog: note type of log

Signed-off-by: Jens Axboe <axboe@kernel.dk>

Conflicts:
backend.c
init.c
iolog.h

Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
3a5f6bde1d14faa9e60cde1c8b45c101bc462f34 11-Apr-2013 Jens Axboe <axboe@kernel.dk> Make nr_thread/nr_process private to backend

No need export that knowledge. Merge from gfio.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
4896473eb0c69d870c41a6a679765d2d5f03e199 11-Apr-2013 Jens Axboe <axboe@kernel.dk> backend: merge in td->o to o-> thread_main() change

Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
9a3f110001f68968def40797bed5632c2406cdfa 28-Mar-2012 Jens Axboe <axboe@kernel.dk> Make lockmem a per job option

We need to get rid of per job options that fiddle with global
state. It's confusing, and it breaks remote option handling.

Signed-off-by: Jens Axboe <axboe@kernel.dk>

Conflicts:
backend.c
fio.h
init.c
options.c

Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
dc5bfbb227f1af83adf0b686d147b7ae2107c144 10-Apr-2013 Jens Axboe <axboe@kernel.dk> Merge in crc32c-intel probe

Then we can kill the option callback.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
62232d0618e3f5aa4805b4be34e872895ba1beb2 05-Apr-2013 Jens Axboe <axboe@kernel.dk> Merge branch 'master' into gfio

Conflicts:
Makefile

Signed-off-by: Jens Axboe <axboe@kernel.dk>
c97f1ad6d2a2fb4fe9f3b15e40158aac21e5699e 28-Mar-2013 Jens Axboe <axboe@kernel.dk> Fix usr/sys/ctx/majf/minf for -USR1 usage

Let the threads themselves update their stats.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
74f4b020b3c8053ff319c3da900a5cf07e51c8eb 23-Mar-2013 Jens Axboe <axboe@kernel.dk> Consider the maximum block size difference the minimum for loop exit

For mixed block sizes, we can hit the condition where we decided
to stop even if we could have done a small block size. Don't do
another loop for those cases.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
d7213923067aa49922962a469a691c3ec951064d 21-Mar-2013 Jens Axboe <axboe@kernel.dk> Merge branch 'master' into gfio

Conflicts:
Makefile
fio.h
options.c

Signed-off-by: Jens Axboe <axboe@kernel.dk>
ee98e23416e8543bf48b734802cd1e49ed894f9e 19-Mar-2013 Jens Axboe <axboe@kernel.dk> Fix re-run issue when job size is set to "unknown"

Assert that td->o.size is valid before comparing the IO amount
we did to it. This prevents re-running a job because we think
there's more work to do. But if we haven't set ->o.size to a
valid value, then that must not determine whether to re-run
the loop or not.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
a5e0ee11ec34cfc70a78aee4f7a2eeb01d9cf41f 12-Mar-2013 Oleg <olegm@olegm-vmware.(none)> Conditionally include <sys/shm.h> based on OS support. Define missing MAP_HUGETLB for Android.

Signed-off-by: Oleg Matcovschi <olegmatcovsky@gmail.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
6a5c4d92ce70a05d2fee981b6f133373c0ef62f8 20-Feb-2013 Jens Axboe <axboe@kernel.dk> Merge branch 'master' into gfio

Conflicts:
configure
fio.c
init.c

Signed-off-by: Jens Axboe <axboe@kernel.dk>
61ee0f86fc428db9403f0c1123e4c2f125152466 11-Feb-2013 Jens Axboe <axboe@kernel.dk> Handle end_fsync if a file is closed

Should not happen, but better be careful since it could have
an impact on data integrity.

Additionally, add a log_err() error if we do fail it.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
90b7a96d9573a14f20470af2aa0dd4e60611a477 04-Feb-2013 Jens Axboe <axboe@kernel.dk> Merge branch 'master' into gfio

Conflicts:
Makefile
fio.h
parse.c

Signed-off-by: Jens Axboe <axboe@kernel.dk>
5bd5f71a3e54950fb4bee64c7ad5edc67e40b8c8 03-Feb-2013 Jens Axboe <axboe@kernel.dk> Fix failure to exit IO loop on some IO sizes

If the size of a file isn't a multiple of the block size being
used, it can cause fio to exit the IO loop, check bytes done,
and then decide to do one more loop since we didn't do quite
as much IO as we wanted to. This happens because the minimum
block size is larger than the remainder. So check for that,
and stop if we need to.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
95820b6e6c92025df8d89c0bf39b174e53137c41 31-Jan-2013 Jens Axboe <axboe@kernel.dk> Merge branch 'master' into gfio

Conflicts:
Makefile
client.c
configure
fio.c
fio.h
server.c
server.h

Signed-off-by: Jens Axboe <axboe@kernel.dk>
f2a2ce0eedb44eaa8689e4cbfa77c79b1751b216 30-Jan-2013 Huadong Liu <HLiu@fusionio.com> Add support for idletime profiling

Idletime profiling allows a benchmark to run while filling the
idle cycles on the server, hence giving you some metric of how
much pressure the system is under. This is useful to be able
to profile and compare changes in a setup or driver, for instance.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
135be493d843d4cae2966a35cbd22a3058ec8e4b 29-Jan-2013 Jens Axboe <axboe@kernel.dk> Merge branch 'master' into gfio

Conflicts:
Makefile
configure
options.c

Signed-off-by: Jens Axboe <axboe@kernel.dk>
44404c5a7f90aa42c3228b56b6f686f15a50fb29 24-Jan-2013 Jens Axboe <axboe@kernel.dk> Add lib/getrusage.c

Abstract out the SELF/THREAD complication.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
7732a09b0a5331f20b7fdf42f811a3f2805dcaae 24-Jan-2013 Huadong Liu <HLiu@fusionio.com> Use RUSAGE_THREAD, if available

Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
836fcc0fceb233ebcc41ee63b4ea5cae20b678a4 24-Jan-2013 Jens Axboe <axboe@kernel.dk> Merge branch 'master' into gfio

Conflicts:
Makefile
backend.c
client.c
fio.c
fio.h
ioengine.h
options.c
os/os-linux.h
server.c

Signed-off-by: Jens Axboe <axboe@kernel.dk>
100f49f105126a26de6f1069679c20fd75194188 23-Jan-2013 Jens Axboe <axboe@kernel.dk> verify: stop on actual number of bytes needed to be verified

If we don't use LFSR or a random map, we don't get told when
to stop by the random generator or map. So track it on the
side, using the same mechanism as do_io().

Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
bcd5abfa9f230bbe4365dad1289fdea1f5509f74 23-Jan-2013 Jens Axboe <axboe@kernel.dk> Make experimental_verify=1 handle all cases properly

- Don't track written bytes, just replay the workload by resetting
all the random generators. This should work for any mixture of IO.

- Handle trims for verify.

- Ensure that rwmix is replayed properly for verify.

- Fixup logging for replay.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
44cbc6daaae3674ea5d0a113b66596ca24f372e0 21-Jan-2013 Jens Axboe <axboe@kernel.dk> verify: add new experimental mode that requires no meta data

Should work fine, but we need to account and track trims to know
which blocks NOT to verify (or verify as zero).

Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
1ae83d45ed853cd73b95b89ae14cacac5b97187e 12-Jan-2013 Jens Axboe <axboe@kernel.dk> Pre-load and sort random blocks for pure read verify workloads

If fio is run with a write phase before a read phase and the
IO type is random, the end read verify phase will get sorted
blocks to read back. But if fio is asked to verify something
that was previously randomly written, it will generate the
same random offsets in random order and verify those. This is
usually much slower than a sorted read back.

So add a verifysort_nr option that allows the user to specify
a number of random offsets to pre-generate and sort, before
reading them and verifying the contents. This greatly speeds
up pure read verify workloads.

Default to 1024, and put a max of 64K entries on the option.
We do a merge list sort on the entries, so we don't want a
huge amount of backlog.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
67bf982340d95ca98098ea050b54b4c7adb116c0 10-Jan-2013 Jens Axboe <axboe@kernel.dk> Add configure script

Get rid of all the fragile guessing and checking of features,
and roll a configure script instead.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
92a0daf8163eaa8d117d1e6e4fdc11a8dddac2d2 09-Jan-2013 Jens Axboe <axboe@kernel.dk> Merge branch 'master' into gfio

Conflicts:
client.c
server.h

Signed-off-by: Jens Axboe <axboe@kernel.dk>
fbccf46c939c9fee9f738da31c2e22791be5eab7 08-Jan-2013 Jens Axboe <axboe@kernel.dk> Fix potential null pointer dereference on verify and requeue events

We clear io_u to NULL, but later we derefence io_u to check
the data direction. Do as we do in the main IO loop - store
the data direction in a local variable, and use that after
the main switch.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
5d879392c0e4025e508cce8118f6fc13ee57cc0e 18-Dec-2012 Jens Axboe <axboe@kernel.dk> gettime: use pthread_{set,get}specific() for TLS

__thread doesn't work on OSX at least (thanks...), so resort to
using the pthread functions to maintaining a per-thread clock
check.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
2a988d8bcb447eb098fc382835cc507587c6ba66 11-Dec-2012 Jens Axboe <axboe@kernel.dk> Merge branch 'master' into gfio

Conflicts:
fio.c

Signed-off-by: Jens Axboe <axboe@kernel.dk>
32dbca2c33c37dcd62bc118dbea33f3c0a488c82 06-Dec-2012 Bruce Cran <BCran@fusionio.com> Free io_u related structures before killing IO engine

Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
c73ed24623ad94a5254e8302298797aba2eb1e9a 06-Dec-2012 Jens Axboe <axboe@kernel.dk> windowsaio: initialize and map windowsaio IO structure to io_u

Instead of searching for a free entry everytime we enter
queue, do this when the io_u is setup. It's cleaner and
faster.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
49758e11f3658686ccd1c61724a5eba142f3ee4f 03-Dec-2012 Jens Axboe <axboe@kernel.dk> Merge branch 'master' into gfio

Conflicts:
Makefile
fio.h
io_u.c
ioengine.h
memory.c

Signed-off-by: Jens Axboe <axboe@kernel.dk>
36d80bc7c7f7fbc2612941b7dd7ceaf645798c7f 30-Nov-2012 Jens Axboe <axboe@kernel.dk> Wire up SIGUSR2 to kill blocking threads

Get rid of io engine ops SIGTERM flag, it didn't really
work.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
ec5c6b125c1eab992882602158bab54957aa733d 21-Nov-2012 Aaron Carroll <aaronc@cse.unsw.edu.au> Add support for Android

Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
1e5324e723116a5faf9da686993cc79c14d62d4b 14-Nov-2012 Jens Axboe <axboe@kernel.dk> Merge branch 'master' into gfio

Conflicts:
Makefile
backend.c
client.c
fio.h
options.c
parse.c
parse.h
server.c
server.h

Signed-off-by: Jens Axboe <axboe@kernel.dk>
108fea772db5f1dd91e2fb67737e3e0d36827b76 14-Nov-2012 Jens Axboe <axboe@kernel.dk> client/server: fixup "All clients" reporting

We didn't properly account for no group_reporting and multiple
jobs on multiple connections.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
7d7803fa331bec4b53edf7bfa3c70ccd7f2920d9 23-Oct-2012 Yufei Ren <renyufei83@gmail.com> rdma ioengine improvement

1) Use fio shipped reentrant and thread-safe rand to replace buggy rand().
2) Add a pause time period before client start committing IOs.
In SEND/RECV test, it's a good practice to setup the iodepth of
of the RECV side deeper than that of the SEND side to
avoid RNR (receiver not ready) error. The
SEND side may send so many unsolicited message before
RECV side commits sufficient recv buffers into recv queue.
This may lead to RNR error. Here, SEND side pauses for a while
during which RECV side commits sufficient recv buffers.
3) Fix server thread hanging bug.
For RDMA WRITE/READ test, No IO bytes are accumulated in server side
during test. Server thread indicates its task completion by changing
its state into `done' as an alternitive. Add `td->done' checking in
do_io().
4) Some comments revision.
export EXTFLAGS and EXTLIBS with '+=".
explanation on exchanging steps of RDMA ioengine control messages.

All these changes are tested in 40Gbps RoCE and 56Gbps InfiniBand testbeds.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
d5abee06d57595679bf8f5fbe70c532a3d8cfdba 22-Oct-2012 Jens Axboe <axboe@kernel.dk> backend: only check bytes_done for engines without FIO_NOIO

This basically reverts the functional part of commit 2557f7e6.
Instead of having to fiddle with the io_u manually, have
the backend check whether we expect data to have been transerred
or not.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
d0b937ed8841484c2d651919943e8e4c70dfd460 20-Oct-2012 Yufei Ren <renyufei83@gmail.com> Fine-grained job level numa control

Two new options, numa_cpu_nodes and numa_mem_policy, are created
for a fine-grained job level numa control. Please refer HOWTO and
README for detailed description.
A example job, examples/numa, is added as well.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
f681d0bac9c42e5d6bcb3601d2f3cfaa1c2cefb4 11-Oct-2012 Jens Axboe <axboe@kernel.dk> Merge branch 'master' into gfio
2f694507513e44e59139f57f28e08c5d0772880e 10-Oct-2012 Bruce Cran <bruce@cran.org.uk> Windows: handle SIGBREAK.

The only way to send a POSIX-style signal from another application on Windows
is to use GenerateConsoleCtrlEvent with either CTRL_C_EVENT or
CTRL_BREAK_EVENT. CTRL_BREAK_EVENT is the only one which gets through to
applications, so add a signal handler for it to allow fio to quit cleanly.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
3d7174e3c4b8aa4daa36261a8e638b702fac5aba 27-Sep-2012 Jens Axboe <axboe@kernel.dk> Merge branch 'master' into gfio
27357187b0899380bae5c6748f2b2d034502e162 27-Sep-2012 Jens Axboe <axboe@kernel.dk> Fix segfault race on exit for platforms that don't have disk util support

We still need to quisce the disk util thread, as it also functions
as the runtime ETA status printer.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
de26d51f3e909997c9a500bf9f15a1bd5c18bb49 25-Sep-2012 Jens Axboe <axboe@kernel.dk> Merge branch 'master' into gfio
88045e04fdd81dbb584d70f9faeb1f6498a85159 25-Sep-2012 Dmitry Monakhov <dmonakhov@openvz.org> backend: fix data direction flow check

This patch fix regression caused SIGSEGV in case of writeonly
or readonly mode, caused by commit 59d8d0f5.

Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
f982d35f5ae88f728c6853aaf57448db8f5fcf98 24-Sep-2012 Jens Axboe <axboe@kernel.dk> Merge branch 'master' into gfio
59d8d0f5fb939ef23e4aa495dbe8fabd59d5bf25 24-Sep-2012 Jens Axboe <axboe@kernel.dk> backend: don't allocate memory buffers for non-data workloads

A trim workload doesn't transfer any actual data, so don't any
backing memory for the IO buffers.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
e65a950ab73d7d91f267ad7f7b084d94e266fcb0 24-Sep-2012 Jens Axboe <axboe@kernel.dk> Merge branch 'master' into gfio

Conflicts:
backend.c
fio.h
io_u.c

Signed-off-by: Jens Axboe <axboe@kernel.dk>
d79db1222039e906dd49ae290daa59701f4e2385 24-Sep-2012 Jens Axboe <axboe@kernel.dk> Merge branch 'master' into gfio

Conflicts:
Makefile
backend.c
client.c
fio.h
init.c
io_ddir.h
options.c
server.h
stat.c
stat.h

Signed-off-by: Jens Axboe <axboe@kernel.dk>
8b28bd41375930664a0ff9ff9b101a88ac416ac5 23-Sep-2012 Dmitry Monakhov <dmonakhov@openvz.org> backend: Add configurable non fatal error list

Sometimes it is reasonable to perform test nearly system limits where
errors are possible and expected, in that case one may provide non
fatal error list which will be ignored during execution.
This patch add two options:
ignore_error: List of non fatal error
error_dump: Whenever ignored list should be dumped to log.

Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
f3afa57e36550288340f1b6c694f354ae72654b9 17-Sep-2012 Jens Axboe <axboe@kernel.dk> Add --output-format command line option

Right now we have normal, terse, and json output. Lets add a parameter
that controls how to output results, instead of having options for
both terse and json outputs. If we are going to add a 4th output
type in the future, it's only going to get more messy.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
342f4be43b2e74cf2f9a2102afdad11c8951b73f 14-Sep-2012 Jens Axboe <axboe@kernel.dk> Add ddir_rw_sum()

We sum up the three data direction primitives a lot, this is
more readable.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
6eaf09d6e9ca1f8accb057cdb18620b7e53ae33f 14-Sep-2012 Shaohua Li <shli@fusionio.com> Add support for trim as a workload type

This only works on Linux so far, and it's always sync given what
the interface to the kernel looks like. Also restricted to pure
block devices.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
c2e9cc4d20c1db1c81276fdaadb56b8b0085d0d8 21-Aug-2012 Jens Axboe <axboe@kernel.dk> Merge branch 'master' into gfio

Conflicts:
eta.c

Signed-off-by: Jens Axboe <axboe@kernel.dk>
521da527743088a9bd2ab882f8b64799d49d5848 02-Aug-2012 Jens Axboe <axboe@kernel.dk> mutex: make 0/1 FIO_MUTEX_LOCKED and FIO_MUTEX_UNLOCKED

Makes the API look cleaner.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
9ec7779f89d4dde9fe1e23d68abaefda54b35de7 02-Aug-2012 Jens Axboe <axboe@kernel.dk> diskutil: ensure that we lock around disk_list access

And also ensure that before we free the disk util structures,
we wait for the disk util thread to exit.

This is v2 of the patch. Commit feb41855 had a bug where it
would deadlock on the disk_util_mutex, when handling drives
that had slaves (lvm/md).

Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
99614958d743d0fdea9b1875d3a41a5a51e9769d 01-Aug-2012 Jens Axboe <axboe@kernel.dk> Revert "diskutil: ensure that we lock around disk_list access"

This reverts commit feb418556a236aa041a625b083d6b99e55d23d74.

It causes hangs, needs further testing.
/external/fio/backend.c
80666bece308842c36132245e4cbbe9eb0bc506b 01-Aug-2012 Jens Axboe <axboe@kernel.dk> Merge branch 'master' into gfio

Conflicts:
iolog.h
stat.c

Signed-off-by: Jens Axboe <axboe@kernel.dk>
feb418556a236aa041a625b083d6b99e55d23d74 01-Aug-2012 Jens Axboe <axboe@kernel.dk> diskutil: ensure that we lock around disk_list access

And also ensure that before we free the disk util structures,
we wait for the disk util thread to exit.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
fb390e242cac8868e01cdc2046c63e04e3d53d70 07-May-2012 Jens Axboe <axboe@kernel.dk> Merge branch 'master' into gfio

Conflicts:
Makefile
backend.c
client.c
fio.h
os/windows/install.wxs
server.c
server.h

Signed-off-by: Jens Axboe <axboe@kernel.dk>
5d89ff793b002e2bd8c1d490030bf35b6f9ee2db 26-Apr-2012 Tao Ma <boyu.mt@taobao.com> fio: Add to cgroup if parameter "cgroup" is enabled.

Currently, we only add the tasks to cgroup if cgroup_weight is set by
the user. But if we want to test other blkcg parameters like block
throttle, the fio tasks isn't added to the corresponding cgroup and
it doesn't make sense.

So add the task to the corresponding cgroup if the cgroup name is given.

Signed-off-by: Tao Ma <boyu.mt@taobao.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
4c6d91e8808fd11f3cee663e42db813b0934c8bb 30-Mar-2012 Jens Axboe <axboe@kernel.dk> Define SIGUSR1 to inform fio to dump run stats while continuing to run

For long running jobs, it may be interesting to see the stats without
having to terminate the job. Set up SIGUSR1 so that it does that,
gives you the stats at that point in time.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
a462baefd211d1847cefbaa500c8aaad2128374b 30-Mar-2012 Jens Axboe <axboe@kernel.dk> Free disk util on exit, not on stats display

Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
b0df7176176935b4d39fd196ad35f046bbe10333 02-Apr-2012 Jens Axboe <axboe@kernel.dk> Merge branch 'master' into gfio
a5e371a61c9e37a2389f7473d65e35b29dec1ccd 02-Apr-2012 Jens Axboe <axboe@kernel.dk> Add indication of whether a job got killed

We have 'X' for exited with error, add 'K' for killed by
signal as well.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
b852e7cf097cec1c7fb92305f9c3c6b394bb39ad 30-Mar-2012 Jens Axboe <axboe@kernel.dk> Define SIGUSR1 to inform fio to dump run stats while continuing to run

For long running jobs, it may be interesting to see the stats without
having to terminate the job. Set up SIGUSR1 so that it does that,
gives you the stats at that point in time.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
d44256b0094fc2e0516adfaf5126d2de9dc951e7 30-Mar-2012 Jens Axboe <axboe@kernel.dk> Free disk util on exit, not on stats display

Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
28727df79348d4ebbd8fd4b0dd89bbca035f68d0 29-Mar-2012 Jens Axboe <axboe@kernel.dk> options: move ioprio/ioprio_class into thread_options space

Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
1b79a070d9307ad57cea5922292241bd3851a039 28-Mar-2012 Jens Axboe <axboe@kernel.dk> Make lockmem a per job option

We need to get rid of per job options that fiddle with global
state. It's confusing, and it breaks remote option handling.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
83ea422a62b8a42b14c362db61c2e7bf53862e80 28-Mar-2012 Jens Axboe <axboe@kernel.dk> Start the process of making options more gfio friendly

We either need to eliminate the option callbacks, or ensure that
they don't have side effects outside of touching td->o. We will
need to use &td->o as the data passed in for the callbacks, not
the full td.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
3c3ed070502bbfec387ded2c43d5e4559ca24a63 27-Mar-2012 Jens Axboe <axboe@kernel.dk> White space spring cleaning

Lets squash them all, with a bit of tooling help.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
812409e9486a37ba2f51e6e23c97e6ae8e26ff9a 19-Mar-2012 Jens Axboe <axboe@kernel.dk> backend: privatize nr_thread/nr_process

We need to start moving init items out of add_job(), to allow
for updating of job options. The number of threads and processes
can be calculated before running jobs, so move them there.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
122c772599f1b0a3148a5790775698d3fa92cc10 19-Mar-2012 Jens Axboe <axboe@kernel.dk> server: make client connections fork off

Instead of handling connections inline from the server, fork
them off before running them. This makes the able to handle
the idle loop command send/reply directly, without being
invoked from the idle loop.

New connections are forked off automatically as well, leaving
the server free to accept new connections. Not sure we want that,
or at least it should be an option to return BUSY on attempted
new connections.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
9746fdbbd89cb202ecf2ee402b79ef215390eaea 17-Mar-2012 Jens Axboe <axboe@kernel.dk> Merge branch 'master' into gfio
c04e4661e4da3b6079f415897e4507cf8e610c54 16-Mar-2012 Daniel Ehrenberg <dehrenberg@google.com> time_based: Avoid restarting main I/O loop

Previously, when fio had written a volume of I/O equal to the size
argument, it restarted the main do_io loop.

This patch allows time_based tests to be run for longer than one
cycle in the do_io main loop. This has a couple of advantages:
* The random number generator is not reset on each iteration
of the loop, so running longer will reach different locations.
* There is not a throughput-reducing point where all operations
must be reaped before new operations are submitted.

The implementation consists of two minor changes:
* In the do_io loop, a time_based test will not exit the loop for
reading or writing too much data.
* When reading or writing sequentially, the operations wrap around
to the beginning after reading the end within the
get_next_seq_block function.

Modified by Jens to properly restart when using a randommap
as well.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
6a4cf74fc2521b601cb1cc9478526cc441830a5c 16-Mar-2012 Jens Axboe <axboe@kernel.dk> Merge branch 'master' into gfio

Conflicts:
fio.h

Signed-off-by: Jens Axboe <axboe@kernel.dk>
8a74b56d7aef44ea75d7abf910ad75378b438fe3 16-Mar-2012 Jens Axboe <axboe@kernel.dk> Fix problem with iodepth_batch_complete=0 and accounting

If the queue is full, we always need to reap at least one event.
Otherwise we end up failing get_io_u() and restarting the file.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
ea51b956a15dd4aee047233be892bb3607f3adbd 14-Mar-2012 Jens Axboe <axboe@kernel.dk> iolog: note type of log

Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
82af2a7ca1a543b41c003de69d5e3c36860f47d5 13-Mar-2012 Jens Axboe <axboe@kernel.dk> Fix failure to verify in mixed read/write workload with backlog

If you run a workload like this:

fio --rw=randrw --bs=4k --direct=1 --ioengine=libaio --iodepth=32
--verify=meta --verify_backlog=1024 --verify_fatal=1 --name=ver-test
--filename=foo --size=1G --verify_pattern=0xaaa

Fio ends up never actually verifying the written blocks. This happens
because as we generate an entry to be verified, the backend checks
whether this is a read/write mixed workload. It then thinks that the
READ is just that, a normal READ, when in fact it could be coming
from our verify list.

Mark such a read as coming from our verify list, so that we know if
it's a "normal" read or one generated to verify previously written
data.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
f7078f7b517ace62bdf22967c18525b5b6c68b85 07-Mar-2012 Jens Axboe <axboe@kernel.dk> Fix bug that causes early termination of fio with verify_backlog

Running this job:

[global]
bs=4k
ioengine=sync
size=100m
direct=1
filename=xxx

[write]
verify=md5
verify_backlog=32
rw=write

causes fio to exit when reads + writes have exceeded 100m, where
it should only exit when writes have exceeded 100m. Fix this by
checking the appropriate counters based on what type of job it is.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
9c42684e32325da26e862280388798343c5f1305 02-Mar-2012 Jens Axboe <axboe@kernel.dk> Add buffer_compress_percentage

The option is pending testing, so not documented yet.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
9e684a4976b7934f5ce011ea281dfef3352e5738 20-Feb-2012 Dan Ehrenberg <dehrenberg@google.com> Token-based flow control

This patch allows two fio jobs to be kept to a certain
proportion of each other using token-based flow control.
There are three new parameters: flow, flow_watermark, and
flow_sleep, documented in the fio options. An example of an fio
job using these parameters is below:

[global]
norandommap
thread
time_based
runtime=30
direct=1
ioengine=libaio
iodepth=256
size=100g
bs=8k
filename=/tmp/testfile
flow_watermark=100
flow_sleep=1000

[job2]
numjobs=1
rw=write
flow=-8

[job1]
numjobs=1
rw=randread
flow=1

The motivating application of this patch was to allow random reads
and sequential writes at a particular given proportion.

This initial version is only correct when run with 'thread', as shared
state is represented with a global variable. It also only allows two
jobs to be synchronized properly. A future version might do more, but
no more functionality was needed for my application.

Tested: Ran a few fio jobs with this flow control, observing
the proportion of IOPS to match what was intended by the job file.
Varied the flow_watermark and flow_sleep parameters and observed
the effect on throughput.

Signed-off-by: Dan Ehrenberg <dehrenberg@google.com>

Modified by me to support flow_id, so an arbitrary number of flows can
be used. This means it no longer relies on global context, so it can be
used from a thread or process alike. Also added man page documentation.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
e43606c2b13ad7fc1af2bbe4a61cf8480ee3a532 20-Feb-2012 Bruce Cran <bruce@cran.org.uk> Fix assumption that pointers fits in a 'long'

Windows uses LLP64 model so [u]intptr_t is more correct.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
649c10c59b016ae8586e54746d3761bc6df33c9b 20-Feb-2012 Bruce Cran <bruce@cran.org.uk> nice() error handling

I've attached a patch which fixes a potential issue I noticed while
reading the POSIX specs: nice() can succeed and return -1, so it's
recommended to set errno to 0 and check it afterwards:

"As -1 is a permissible return value in a successful situation, an
application wishing to check for error situations should set errno to 0,
then call nice(), and if it returns -1, check to see whether errno is
non-zero."

Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
4e87c37a06b96c4ed1f941eb91ec25e839c68d21 15-Feb-2012 Jens Axboe <axboe@kernel.dk> Fix grammatical error

Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
a3efc919b0dcc27d65a6e84edca209b91cee173d 09-Feb-2012 Jens Axboe <axboe@kernel.dk> Move variables to backend, fix up line spaces

Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c
2e1df07d1ea30e0304cc65370f3ed161a6f22cd4 09-Feb-2012 Jens Axboe <axboe@kernel.dk> Move some code around to better separate front/backend

Signed-off-by: Jens Axboe <axboe@kernel.dk>
/external/fio/backend.c