summaryrefslogtreecommitdiffstats
AgeCommit message (Collapse)AuthorFilesLines
2014-05-10blk-mq: bitmap tag: select random tag betweet 0 and (depth - 1)Ming Lei1-4/+1
The selected tag should be selected at random between 0 and (depth - 1) with probability 1/depth, instead between 0 and (depth - 2) with probability 1/(depth - 1). Signed-off-by: Ming Lei <tom.leiming@gmail.com> Signed-off-by: Jens Axboe <axboe@fb.com>
2014-05-10blk-mq: bitmap tag: remove barrier in bt_clear_tag()Ming Lei1-1/+0
The barrier isn't necessary because both atomic_dec_and_test() and wake_up() implicate one barrier. Signed-off-by: Ming Lei <tom.leiming@gmail.com> Signed-off-by: Jens Axboe <axboe@fb.com>
2014-05-10blk-mq: bitmap tag: use clear_bit_unlock in bt_clear_tag()Ming Lei1-1/+5
The unlock memory barrier need to order access to req in free path and clearing tag bit, otherwise either request free path may see a allocated request, or initialized request in allocate path might be modified by the ongoing free path. Signed-off-by: Ming Lei <tom.leiming@gmail.com> Signed-off-by: Jens Axboe <axboe@fb.com>
2014-05-09blk-mq: use sparser tag layout for lower queue depthJens Axboe2-16/+36
For best performance, spreading tags over multiple cachelines makes the tagging more efficient on multicore systems. But since we have 8 * sizeof(unsigned long) tags per cacheline, we don't always get a nice spread. Attempt to spread the tags over at least 4 cachelines, using fewer number of bits per unsigned long if we have to. This improves tagging performance in setups with 32-128 tags. For higher depths, the spread is the same as before (BITS_PER_LONG tags per cacheline). Signed-off-by: Jens Axboe <axboe@fb.com>
2014-05-09blk-mq: implement new and more efficient tagging schemeJens Axboe5-99/+391
blk-mq currently uses percpu_ida for tag allocation. But that only works well if the ratio between tag space and number of CPUs is sufficiently high. For most devices and systems, that is not the case. The end result if that we either only utilize the tag space partially, or we end up attempting to fully exhaust it and run into lots of lock contention with stealing between CPUs. This is not optimal. This new tagging scheme is a hybrid bitmap allocator. It uses two tricks to both be SMP friendly and allow full exhaustion of the space: 1) We cache the last allocated (or freed) tag on a per blk-mq software context basis. This allows us to limit the space we have to search. The key element here is not caching it in the shared tag structure, otherwise we end up dirtying more shared cache lines on each allocate/free operation. 2) The tag space is split into cache line sized groups, and each context will start off randomly in that space. Even up to full utilization of the space, this divides the tag users efficiently into cache line groups, avoiding dirtying the same one both between allocators and between allocator and freeer. This scheme shows drastically better behaviour, both on small tag spaces but on large ones as well. It has been tested extensively to show better performance for all the cases blk-mq cares about. Signed-off-by: Jens Axboe <axboe@fb.com>
2014-05-09blk-mq: initialize struct request fields individuallyChristoph Hellwig2-5/+49
This allows us to avoid a non-atomic memset over ->atomic_flags as well as killing lots of duplicate initializations. Signed-off-by: Christoph Hellwig <hch@lst.de> Signed-off-by: Jens Axboe <axboe@fb.com>
2014-05-09blk-mq: update a hotplug comment for grammarJens Axboe1-4/+4
Signed-off-by: Jens Axboe <axboe@fb.com>
2014-05-07blk-mq: add basic round-robin of what CPU to queue workqueue work onJens Axboe2-14/+35
Right now we just pick the first CPU in the mask, but that can easily overload that one. Add some basic batching and round-robin all the entries in the mask instead. Signed-off-by: Jens Axboe <axboe@fb.com>
2014-05-02block/blk-throttle.c: fix return of 0/1 with return type boolFabian Frederick1-4/+4
Fix 4 coccinelle warnings. Cc: Jens Axboe <axboe@kernel.dk> Cc: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Fabian Frederick <fabf@skynet.be> Signed-off-by: Jens Axboe <axboe@fb.com>
2014-05-02block/blk-iopoll.c: use iop instead of iopollFabian Frederick1-2/+2
All blk_iopoll functions use iop for parent iopoll structure except blk_iopoll_complete.This also fixes one kernel-doc warning. Cc: Jens Axboe <axboe@kernel.dk> Cc: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Fabian Frederick <fabf@skynet.be> Signed-off-by: Jens Axboe <axboe@fb.com>
2014-05-02blk-mq: remove extra requeue traceJens Axboe1-4/+0
We already issue a blktrace requeue event in __blk_mq_requeue_request(), don't do it from the original caller as well. Signed-off-by: Jens Axboe <axboe@fb.com>
2014-04-30block: Fix format string mismatch in cfq-iosched.cMasanari Iida1-1/+1
Fix format string mismatch in cfq_var_show() Signed-off-by: Masanari Iida <standby24x7@gmail.com> Signed-off-by: Jens Axboe <axboe@fb.com>
2014-04-30blk-mq: refactor request insertion/mergingJens Axboe1-7/+15
Refactor the logic around adding a new bio to a software queue, so we nest the ctx->lock where we really need it (merge and insertion) and don't hold it when we don't (init and IO start accounting). Signed-off-by: Jens Axboe <axboe@fb.com>
2014-04-30blk-mq remove debug BUG_ON() when draining software queuesJens Axboe1-1/+0
It's never been of any use, lets get rid of it. Signed-off-by: Jens Axboe <axboe@fb.com>
2014-04-29blk-mq: fix waiting for reserved tagsJens Axboe3-4/+4
blk_mq_wait_for_tags() is only able to wait for "normal" tags, not reserved tags. Pass in which one we should attempt to get a tag for, so that waiting for reserved tags will work. Reserved tags are used for internal commands, which are usually serialized. Hence no waiting generally takes place, but we should ensure that it actually works if users need that functionality. Signed-off-by: Jens Axboe <axboe@fb.com>
2014-04-28random: export add_disk_randomnessChristoph Hellwig1-0/+1
This will be needed for pending changes to the scsi midlayer that now calls lower level block APIs, as well as any blk-mq driver that wants to contribute to the random pool. Signed-off-by: Christoph Hellwig <hch@lst.de> Acked-by: "Theodore Ts'o" <tytso@mit.edu> Signed-off-by: Jens Axboe <axboe@fb.com>
2014-04-25block: fold __blk_add_timer into blk_add_timerChristoph Hellwig1-23/+11
Signed-off-by: Christoph Hellwig <hch@lst.de> Signed-off-by: Jens Axboe <axboe@fb.com>
2014-04-25blk-mq: respect rq_affinityChristoph Hellwig4-46/+6
The blk-mq code is using it's own version of the I/O completion affinity tunables, which causes a few issues: - the rq_affinity sysfs file doesn't work for blk-mq devices, even if it still is present, thus breaking existing tuning setups. - the rq_affinity = 1 mode, which is the defauly for legacy request based drivers isn't implemented at all. - blk-mq drivers don't implement any completion affinity with the default flag settings. This patches removes the blk-mq ipi_redirect flag and sysfs file, as well as the internal BLK_MQ_F_SHOULD_IPI flag and replaces it with code that respects the queue-wide rq_affinity flags and also implements the rq_affinity = 1 mode. This means I/O completion affinity can now only be tuned block-queue wide instead of per context, which seems more sensible to me anyway. Signed-off-by: Christoph Hellwig <hch@lst.de> Signed-off-by: Jens Axboe <axboe@fb.com>
2014-04-24blk-mq: fix race with timeouts and requeue eventsJens Axboe4-18/+42
If a requeue event races with a timeout, we can get into the situation where we attempt to complete a request from the timeout handler when it's not start anymore. This causes a crash. So have the timeout handler check that REQ_ATOM_STARTED is still set on the request - if not, we ignore the event. If this happens, the request has now been marked as complete. As a consequence, we need to ensure to clear REQ_ATOM_COMPLETE in blk_mq_start_request(), as to maintain proper request state. Signed-off-by: Jens Axboe <axboe@fb.com>
2014-04-24Revert "blk-mq: initialize req->q in allocation"Jens Axboe1-3/+1
This reverts commit 6a3c8a3ac0e68dcfc2a01f4aa1ca0edd1a1701eb. We need selective clearing of the request to make the init-at-free time completely safe. Otherwise we end up stomping on rq->atomic_flags, which we don't want to do.
2014-04-23blk-mq: fix leak of set->tagsMing Lei1-0/+1
set->tags should be freed in blk_mq_free_tag_set(). Signed-off-by: Ming Lei <tom.leiming@gmail.com> Signed-off-by: Jens Axboe <axboe@fb.com>
2014-04-22fs/bio.c: remove nr_segs (unused function parameter)Fabian Frederick1-3/+2
nr_segs is no longer used in bio_alloc_map_data since c8db444820a1e3 ("block: Don't save/copy bvec array anymore") Signed-off-by: Fabian Frederick <fabf@skynet.be> Cc: Jens Axboe <axboe@kernel.dk> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Jens Axboe <axboe@fb.com>
2014-04-22fs/bio: remove bs paramater in biovec_create_poolFabian Frederick3-4/+4
bs is no longer used in biovec_create_pool since 9f060e2231ca96 ("block: Convert integrity to bvec_alloc_bs()") Signed-off-by: Fabian Frederick <fabf@skynet.be> Cc: Jens Axboe <axboe@kernel.dk> Signed-off-by: Jens Axboe <axboe@fb.com>
2014-04-21block/blk-throttle.c: add static to blk_throtl_dispatch_work_fnFabian Frederick1-1/+1
blk_throtl_dispatch_work_fn is only used in blk-throttle.c Cc: Jens Axboe <axboe@kernel.dk> Cc: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Fabian Frederick <fabf@skynet.be> Signed-off-by: Jens Axboe <axboe@fb.com>
2014-04-21fs: fix new kernel-doc warnings in fs/bio.cRandy Dunlap1-0/+2
Fix new kernel-doc warnings in fs/bio.c: Warning(fs/bio.c:316): No description found for parameter 'bio' Warning(fs/bio.c:316): No description found for parameter 'parent' Signed-off-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Jens Axboe <axboe@fb.com>
2014-04-21blk-mq: initialize req->q in allocationMing Lei1-1/+3
The patch basically reverts the patch of(blk-mq: initialize request on allocation) in Jens's tree(already in -next), and only initialize req->q in allocation for two reasons: - presumed cache hotness on completion - blk_rq_tagged(rq) depends on reset of req->mq_ctx Signed-off-by: Ming Lei <tom.leiming@gmail.com> Signed-off-by: Jens Axboe <axboe@fb.com>
2014-04-21blk-mq: user (1 << order) to implement order_to_size()Ming Lei1-6/+1
Cc: Jörg-Volker Peetz <jvpeetz@web.de> Cc: Max Filippov <jcmvbkbc@gmail.com> Signed-off-by: Ming Lei <tom.leiming@gmail.com> Signed-off-by: Jens Axboe <axboe@fb.com>
2014-04-21blk-mq: fix allocation of set->tagsMing Lei1-1/+2
type of set->tags is struct blk_mq_tags **. Reviewed-by: Christoph Hellwig <hch@lst.de> Signed-off-by: Ming Lei <tom.leiming@gmail.com> Signed-off-by: Jens Axboe <axboe@fb.com>
2014-04-21blk-mq: free hctx->ctx_map when init failedMing Lei1-0/+1
Avoid memory leak in the failure path. Reviewed-by: Christoph Hellwig <hch@lst.de> Signed-off-by: Ming Lei <tom.leiming@gmail.com> Signed-off-by: Jens Axboe <axboe@fb.com>
2014-04-16sd/skd: stuff discard page in request->completion_dataJens Axboe2-6/+5
Store the pointer to the page there, so we can always safely reference it from end_io context where ->bio may have been cleared. Signed-off-by: Jens Axboe <axboe@fb.com>
2014-04-16jsflash: missed conversion from rq->buffer to bio_data(rq->bio)Jens Axboe1-1/+1
Signed-off-by: Jens Axboe <axboe@fb.com>
2014-04-16block: relax when to modify the timeout timerJens Axboe1-2/+13
Since we are now, by default, applying timer slack to expiry times, the logic for when to modify a timer in the block code is suboptimal. The block layer keeps a forward rolling timer per queue for all requests, and modifies this timer if a request has a shorter timeout than what the current expiry time is. However, this breaks down when our rounded timer values get applied slack. Then each new request ends up modifying the timer, since we're still a little in front of the timer + slack. Fix this by allowing a tolerance of HZ / 2, the timeout handling doesn't need to be very precise. This drastically cuts down the number of timer modifications we have to make. Signed-off-by: Jens Axboe <axboe@fb.com>
2014-04-16block: export blk_finish_requestChristoph Hellwig2-1/+3
This allows to mirror the blk-mq code flow for more a more readable I/O completion handler in SCSI. Signed-off-by: Christoph Hellwig <hch@lst.de> Signed-off-by: Jens Axboe <axboe@fb.com>
2014-04-16blk-mq: rename mq_flush_work struct request memberChristoph Hellwig2-4/+4
We will use this work_struct to requeue scsi commands from the completion handler as well, so give it a more generic name. Signed-off-by: Christoph Hellwig <hch@lst.de> Signed-off-by: Jens Axboe <axboe@fb.com>
2014-04-16blk-mq: add blk_mq_requeue_requestChristoph Hellwig2-2/+18
This allows to requeue a request that has been accepted by ->queue_rq earlier. This is needed by the SCSI layer in various error conditions. The existing internal blk_mq_requeue_request is renamed to __blk_mq_requeue_request as it is a lower level building block for this funtionality. Signed-off-by: Christoph Hellwig <hch@lst.de> Signed-off-by: Jens Axboe <axboe@fb.com>
2014-04-16blk-mq: add blk_mq_start_hw_queuesChristoph Hellwig2-0/+12
Add a helper to unconditionally kick contexts of a queue. This will be needed by the SCSI layer to provide fair queueing between multiple devices on a single host. Signed-off-by: Christoph Hellwig <hch@lst.de> Signed-off-by: Jens Axboe <axboe@fb.com>
2014-04-16blk-mq: add blk_mq_delay_queueChristoph Hellwig3-9/+46
Add a blk-mq equivalent to blk_delay_queue so that the scsi layer can ask to be kicked again after a delay. Signed-off-by: Christoph Hellwig <hch@lst.de> Modified by me to kill the unnecessary preempt disable/enable in the delayed workqueue handler. Signed-off-by: Jens Axboe <axboe@fb.com>
2014-04-16blk-mq: add async parameter to blk_mq_start_stopped_hw_queuesChristoph Hellwig3-5/+5
Signed-off-by: Christoph Hellwig <hch@lst.de> Signed-off-by: Jens Axboe <axboe@fb.com>
2014-04-16blk-mq: bidi supportChristoph Hellwig1-2/+7
Add two unlinkely branches to make sure the resid is initialized correctly for bidi request pairs, and the second request gets properly freed. Signed-off-by: Christoph Hellwig <hch@lst.de> Signed-off-by: Jens Axboe <axboe@fb.com>
2014-04-16blk-mq: allow drivers to hook into I/O completionChristoph Hellwig2-13/+12
Split out the bottom half of blk_mq_end_io so that drivers can perform work when they know a request has been completed, but before it has been freed. This also obsoletes blk_mq_end_io_partial as drivers can now pass any value to blk_update_request directly. Signed-off-by: Christoph Hellwig <hch@lst.de> Signed-off-by: Jens Axboe <axboe@fb.com>
2014-04-16blk-mq: kill preempt disable/enable in blk_mq_work_fn()Jens Axboe1-2/+0
blk_mq_work_fn() is always invoked off the bounded workqueues, so it can happily preempt among the queues in that set without causing any issues for blk-mq. Signed-off-by: Jens Axboe <axboe@fb.com>
2014-04-16blk-mq: don't use preempt_count() to check for right CPUJens Axboe1-1/+1
UP or CONFIG_PREEMPT_NONE will return 0, and what we really want to check is whether or not we are on the right CPU. So don't make PREEMPT part of this, just test the CPU in the mask directly. Signed-off-by: Jens Axboe <axboe@fb.com>
2014-04-16gdrom: missed conversion from req->bufferJens Axboe1-1/+1
The friendly Intel kbuild test robot reported: drivers/cdrom/gdrom.c: In function 'gdrom_readdisk_dma': drivers/cdrom/gdrom.c:605:3: error: 'struct request' has no member named 'buffer' Convert that from req->buffer to bio_data(rq->bio). Apparently my grep missed this one, and I don't build for Sega Dreamcast enough. Signed-off-by: Jens Axboe <axboe@fb.com>
2014-04-15block: all blk-mq requests are taggedChristoph Hellwig1-1/+2
Instead of setting the REQ_QUEUED flag on each of them just take it into account in the only macro checking it. Signed-off-by: Christoph Hellwig <hch@lst.de> Signed-off-by: Jens Axboe <axboe@fb.com>
2014-04-15blk-mq: split out tag initialization, support shared tagsChristoph Hellwig8-200/+262
Add a new blk_mq_tag_set structure that gets set up before we initialize the queue. A single blk_mq_tag_set structure can be shared by multiple queues. Signed-off-by: Christoph Hellwig <hch@lst.de> Modular export of blk_mq_{alloc,free}_tagset added by me. Signed-off-by: Jens Axboe <axboe@fb.com>
2014-04-15blk-mq: initialize request on allocationChristoph Hellwig1-3/+1
If we want to share tag and request allocation between queues we cannot initialize the request at init/free time, but need to initialize it at allocation time as it might get used for different queues over its lifetime. Signed-off-by: Christoph Hellwig <hch@lst.de> Signed-off-by: Jens Axboe <axboe@fb.com>
2014-04-15blk-mq: add ->init_request and ->exit_request methodsChristoph Hellwig3-87/+55
The current blk_mq_init_commands/blk_mq_free_commands interface has a two problems: 1) Because only the constructor is passed to blk_mq_init_commands there is no easy way to clean up when a comman initialization failed. The current code simply leaks the allocations done in the constructor. 2) There is no good place to call blk_mq_free_commands: before blk_cleanup_queue there is no guarantee that all outstanding commands have completed, so we can't free them yet. After blk_cleanup_queue the queue has usually been freed. This can be worked around by grabbing an unconditional reference before calling blk_cleanup_queue and dropping it after blk_mq_free_commands is done, although that's not exatly pretty and driver writers are guaranteed to get it wrong sooner or later. Both issues are easily fixed by making the request constructor and destructor normal blk_mq_ops methods. Signed-off-by: Christoph Hellwig <hch@lst.de> Signed-off-by: Jens Axboe <axboe@fb.com>
2014-04-15blk-mq: make ->flush_rq fully transparent to driversChristoph Hellwig3-10/+24
Drivers shouldn't have to care about the block layer setting aside a request to implement the flush state machine. We already override the mq context and tag to make it more transparent, but so far haven't deal with the driver private data in the request. Make sure to override this as well, and while we're at it add a proper helper sitting in blk-mq.c that implements the full impersonation. Signed-off-by: Christoph Hellwig <hch@lst.de> Signed-off-by: Jens Axboe <axboe@fb.com>
2014-04-15blk-mq: do not initialize req->specialChristoph Hellwig5-27/+9
Drivers can reach their private data easily using the blk_mq_rq_to_pdu helper and don't need req->special. By not initializing it code can be simplified nicely, and we also shave off a few more instructions from the I/O path. Signed-off-by: Christoph Hellwig <hch@lst.de> Signed-off-by: Jens Axboe <axboe@fb.com>
2014-04-15blk-mq: initialize resid_lenChristoph Hellwig1-0/+2
Signed-off-by: Christoph Hellwig <hch@lst.de> Signed-off-by: Jens Axboe <axboe@fb.com>
generated by cgit v1.2.3 (git 2.39.1) at 2024-09-19 12:20:45 +0000