diff options
1172 files changed, 44511 insertions, 23361 deletions
diff --git a/Documentation/ABI/stable/sysfs-devices-system-cpu b/Documentation/ABI/stable/sysfs-devices-system-cpu new file mode 100644 index 000000000000..33c133e2a631 --- /dev/null +++ b/Documentation/ABI/stable/sysfs-devices-system-cpu @@ -0,0 +1,25 @@ +What: /sys/devices/system/cpu/dscr_default +Date: 13-May-2014 +KernelVersion: v3.15.0 +Contact: +Description: Writes are equivalent to writing to + /sys/devices/system/cpu/cpuN/dscr on all CPUs. + Reads return the last written value or 0. + This value is not a global default: it is a way to set + all per-CPU defaults at the same time. +Values: 64 bit unsigned integer (bit field) + +What: /sys/devices/system/cpu/cpu[0-9]+/dscr +Date: 13-May-2014 +KernelVersion: v3.15.0 +Contact: +Description: Default value for the Data Stream Control Register (DSCR) on + a CPU. + This default value is used when the kernel is executing and + for any process that has not set the DSCR itself. + If a process ever sets the DSCR (via direct access to the + SPR) that value will be persisted for that process and used + on any CPU where it executes (overriding the value described + here). + If set by a process it will be inherited by child processes. +Values: 64 bit unsigned integer (bit field) diff --git a/Documentation/ABI/testing/ima_policy b/Documentation/ABI/testing/ima_policy index f1c5cc9d17a8..4c3efe434806 100644 --- a/Documentation/ABI/testing/ima_policy +++ b/Documentation/ABI/testing/ima_policy @@ -23,7 +23,7 @@ Description: [fowner]] lsm: [[subj_user=] [subj_role=] [subj_type=] [obj_user=] [obj_role=] [obj_type=]] - option: [[appraise_type=]] + option: [[appraise_type=]] [permit_directio] base: func:= [BPRM_CHECK][MMAP_CHECK][FILE_CHECK][MODULE_CHECK] mask:= [MAY_READ] [MAY_WRITE] [MAY_APPEND] [MAY_EXEC] diff --git a/Documentation/cgroups/memory.txt b/Documentation/cgroups/memory.txt index b3429aec444c..02ab997a1ed2 100644 --- a/Documentation/cgroups/memory.txt +++ b/Documentation/cgroups/memory.txt @@ -458,15 +458,11 @@ About use_hierarchy, see Section 6. 5.1 force_empty memory.force_empty interface is provided to make cgroup's memory usage empty. - You can use this interface only when the cgroup has no tasks. When writing anything to this # echo 0 > memory.force_empty - Almost all pages tracked by this memory cgroup will be unmapped and freed. - Some pages cannot be freed because they are locked or in-use. Such pages are - moved to parent (if use_hierarchy==1) or root (if use_hierarchy==0) and this - cgroup will be empty. + the cgroup will be reclaimed and as many pages reclaimed as possible. The typical use case for this interface is before calling rmdir(). Because rmdir() moves all pages to parent, some out-of-use page caches can be diff --git a/Documentation/cgroups/unified-hierarchy.txt b/Documentation/cgroups/unified-hierarchy.txt new file mode 100644 index 000000000000..324b182e6000 --- /dev/null +++ b/Documentation/cgroups/unified-hierarchy.txt @@ -0,0 +1,359 @@ + +Cgroup unified hierarchy + +April, 2014 Tejun Heo <tj@kernel.org> + +This document describes the changes made by unified hierarchy and +their rationales. It will eventually be merged into the main cgroup +documentation. + +CONTENTS + +1. Background +2. Basic Operation + 2-1. Mounting + 2-2. cgroup.subtree_control + 2-3. cgroup.controllers +3. Structural Constraints + 3-1. Top-down + 3-2. No internal tasks +4. Other Changes + 4-1. [Un]populated Notification + 4-2. Other Core Changes + 4-3. Per-Controller Changes + 4-3-1. blkio + 4-3-2. cpuset + 4-3-3. memory +5. Planned Changes + 5-1. CAP for resource control + + +1. Background + +cgroup allows an arbitrary number of hierarchies and each hierarchy +can host any number of controllers. While this seems to provide a +high level of flexibility, it isn't quite useful in practice. + +For example, as there is only one instance of each controller, utility +type controllers such as freezer which can be useful in all +hierarchies can only be used in one. The issue is exacerbated by the +fact that controllers can't be moved around once hierarchies are +populated. Another issue is that all controllers bound to a hierarchy +are forced to have exactly the same view of the hierarchy. It isn't +possible to vary the granularity depending on the specific controller. + +In practice, these issues heavily limit which controllers can be put +on the same hierarchy and most configurations resort to putting each +controller on its own hierarchy. Only closely related ones, such as +the cpu and cpuacct controllers, make sense to put on the same +hierarchy. This often means that userland ends up managing multiple +similar hierarchies repeating the same steps on each hierarchy +whenever a hierarchy management operation is necessary. + +Unfortunately, support for multiple hierarchies comes at a steep cost. +Internal implementation in cgroup core proper is dazzlingly +complicated but more importantly the support for multiple hierarchies +restricts how cgroup is used in general and what controllers can do. + +There's no limit on how many hierarchies there may be, which means +that a task's cgroup membership can't be described in finite length. +The key may contain any varying number of entries and is unlimited in +length, which makes it highly awkward to handle and leads to addition +of controllers which exist only to identify membership, which in turn +exacerbates the original problem. + +Also, as a controller can't have any expectation regarding what shape +of hierarchies other controllers would be on, each controller has to +assume that all other controllers are operating on completely +orthogonal hierarchies. This makes it impossible, or at least very +cumbersome, for controllers to cooperate with each other. + +In most use cases, putting controllers on hierarchies which are +completely orthogonal to each other isn't necessary. What usually is +called for is the ability to have differing levels of granularity +depending on the specific controller. In other words, hierarchy may +be collapsed from leaf towards root when viewed from specific +controllers. For example, a given configuration might not care about +how memory is distributed beyond a certain level while still wanting +to control how CPU cycles are distributed. + +Unified hierarchy is the next version of cgroup interface. It aims to +address the aforementioned issues by having more structure while +retaining enough flexibility for most use cases. Various other +general and controller-specific interface issues are also addressed in +the process. + + +2. Basic Operation + +2-1. Mounting + +Currently, unified hierarchy can be mounted with the following mount +command. Note that this is still under development and scheduled to +change soon. + + mount -t cgroup -o __DEVEL__sane_behavior cgroup $MOUNT_POINT + +All controllers which are not bound to other hierarchies are +automatically bound to unified hierarchy and show up at the root of +it. Controllers which are enabled only in the root of unified +hierarchy can be bound to other hierarchies at any time. This allows +mixing unified hierarchy with the traditional multiple hierarchies in +a fully backward compatible way. + + +2-2. cgroup.subtree_control + +All cgroups on unified hierarchy have a "cgroup.subtree_control" file +which governs which controllers are enabled on the children of the +cgroup. Let's assume a hierarchy like the following. + + root - A - B - C + \ D + +root's "cgroup.subtree_control" file determines which controllers are +enabled on A. A's on B. B's on C and D. This coincides with the +fact that controllers on the immediate sub-level are used to +distribute the resources of the parent. In fact, it's natural to +assume that resource control knobs of a child belong to its parent. +Enabling a controller in a "cgroup.subtree_control" file declares that +distribution of the respective resources of the cgroup will be +controlled. Note that this means that controller enable states are +shared among siblings. + +When read, the file contains a space-separated list of currently +enabled controllers. A write to the file should contain a +space-separated list of controllers with '+' or '-' prefixed (without +the quotes). Controllers prefixed with '+' are enabled and '-' +disabled. If a controller is listed multiple times, the last entry +wins. The specific operations are executed atomically - either all +succeed or fail. + + +2-3. cgroup.controllers + +Read-only "cgroup.controllers" file contains a space-separated list of +controllers which can be enabled in the cgroup's +"cgroup.subtree_control" file. + +In the root cgroup, this lists controllers which are not bound to +other hierarchies and the content changes as controllers are bound to +and unbound from other hierarchies. + +In non-root cgroups, the content of this file equals that of the +parent's "cgroup.subtree_control" file as only controllers enabled +from the parent can be used in its children. + + +3. Structural Constraints + +3-1. Top-down + +As it doesn't make sense to nest control of an uncontrolled resource, +all non-root "cgroup.subtree_control" files can only contain +controllers which are enabled in the parent's "cgroup.subtree_control" +file. A controller can be enabled only if the parent has the +controller enabled and a controller can't be disabled if one or more +children have it enabled. + + +3-2. No internal tasks + +One long-standing issue that cgroup faces is the competition between +tasks belonging to the parent cgroup and its children cgroups. This +is inherently nasty as two different types of entities compete and +there is no agreed-upon obvious way to handle it. Different +controllers are doing different things. + +The cpu controller considers tasks and cgroups as equivalents and maps +nice levels to cgroup weights. This works for some cases but falls +flat when children should be allocated specific ratios of CPU cycles +and the number of internal tasks fluctuates - the ratios constantly +change as the number of competing entities fluctuates. There also are +other issues. The mapping from nice level to weight isn't obvious or +universal, and there are various other knobs which simply aren't +available for tasks. + +The blkio controller implicitly creates a hidden leaf node for each +cgroup to host the tasks. The hidden leaf has its own copies of all +the knobs with "leaf_" prefixed. While this allows equivalent control +over internal tasks, it's with serious drawbacks. It always adds an +extra layer of nesting which may not be necessary, makes the interface +messy and significantly complicates the implementation. + +The memory controller currently doesn't have a way to control what +happens between internal tasks and child cgroups and the behavior is +not clearly defined. There have been attempts to add ad-hoc behaviors +and knobs to tailor the behavior to specific workloads. Continuing +this direction will lead to problems which will be extremely difficult +to resolve in the long term. + +Multiple controllers struggle with internal tasks and came up with +different ways to deal with it; unfortunately, all the approaches in +use now are severely flawed and, furthermore, the widely different +behaviors make cgroup as whole highly inconsistent. + +It is clear that this is something which needs to be addressed from +cgroup core proper in a uniform way so that controllers don't need to +worry about it and cgroup as a whole shows a consistent and logical +behavior. To achieve that, unified hierarchy enforces the following +structural constraint: + + Except for the root, only cgroups which don't contain any task may + have controllers enabled in their "cgroup.subtree_control" files. + +Combined with other properties, this guarantees that, when a +controller is looking at the part of the hierarchy which has it +enabled, tasks are always only on the leaves. This rules out +situations where child cgroups compete against internal tasks of the +parent. + +There are two things to note. Firstly, the root cgroup is exempt from +the restriction. Root contains tasks and anonymous resource +consumption which can't be associated with any other cgroup and +requires special treatment from most controllers. How resource +consumption in the root cgroup is governed is up to each controller. + +Secondly, the restriction doesn't take effect if there is no enabled +controller in the cgroup's "cgroup.subtree_control" file. This is +important as otherwise it wouldn't be possible to create children of a +populated cgroup. To control resource distribution of a cgroup, the +cgroup must create children and transfer all its tasks to the children +before enabling controllers in its "cgroup.subtree_control" file. + + +4. Other Changes + +4-1. [Un]populated Notification + +cgroup users often need a way to determine when a cgroup's +subhierarchy becomes empty so that it can be cleaned up. cgroup +currently provides release_agent for it; unfortunately, this mechanism +is riddled with issues. + +- It delivers events by forking and execing a userland binary + specified as the release_agent. This is a long deprecated method of + notification delivery. It's extremely heavy, slow and cumbersome to + integrate with larger infrastructure. + +- There is single monitoring point at the root. There's no way to + delegate management of a subtree. + +- The event isn't recursive. It triggers when a cgroup doesn't have + any tasks or child cgroups. Events for internal nodes trigger only + after all children are removed. This again makes it impossible to + delegate management of a subtree. + +- Events are filtered from the kernel side. A "notify_on_release" + file is used to subscribe to or suppress release events. This is + unnecessarily complicated and probably done this way because event + delivery itself was expensive. + +Unified hierarchy implements an interface file "cgroup.populated" +which can be used to monitor whether the cgroup's subhierarchy has +tasks in it or not. Its value is 0 if there is no task in the cgroup +and its descendants; otherwise, 1. poll and [id]notify events are +triggered when the value changes. + +This is significantly lighter and simpler and trivially allows +delegating management of subhierarchy - subhierarchy monitoring can +block further propagation simply by putting itself or another process +in the subhierarchy and monitor events that it's interested in from +there without interfering with monitoring higher in the tree. + +In unified hierarchy, the release_agent mechanism is no longer +supported and the interface files "release_agent" and +"notify_on_release" do not exist. + + +4-2. Other Core Changes + +- None of the mount options is allowed. + +- remount is disallowed. + +- rename(2) is disallowed. + +- The "tasks" file is removed. Everything should at process + granularity. Use the "cgroup.procs" file instead. + +- The "cgroup.procs" file is not sorted. pids will be unique unless + they got recycled in-between reads. + +- The "cgroup.clone_children" file is removed. + + +4-3. Per-Controller Changes + +4-3-1. blkio + +- blk-throttle becomes properly hierarchical. + + +4-3-2. cpuset + +- Tasks are kept in empty cpusets after hotplug and take on the masks + of the nearest non-empty ancestor, instead of being moved to it. + +- A task can be moved into an empty cpuset, and again it takes on the + masks of the nearest non-empty ancestor. + + +4-3-3. memory + +- use_hierarchy is on by default and the cgroup file for the flag is + not created. + + +5. Planned Changes + +5-1. CAP for resource control + +Unified hierarchy will require one of the capabilities(7), which is +yet to be decided, for all resource control related knobs. Process +organization operations - creation of sub-cgroups and migration of +processes in sub-hierarchies may be delegated by changing the +ownership and/or permissions on the cgroup directory and +"cgroup.procs" interface file; however, all operations which affect +resource control - writes to a "cgroup.subtree_control" file or any +controller-specific knobs - will require an explicit CAP privilege. + +This, in part, is to prevent the cgroup interface from being +inadvertently promoted to programmable API used by non-privileged +binaries. cgroup exposes various aspects of the system in ways which +aren't properly abstracted for direct consumption by regular programs. +This is an administration interface much closer to sysctl knobs than +system calls. Even the basic access model, being filesystem path +based, isn't suitable for direct consumption. There's no way to +access "my cgroup" in a race-free way or make multiple operations +atomic against migration to another cgroup. + +Another aspect is that, for better or for worse, the cgroup interface +goes through far less scrutiny than regular interfaces for +unprivileged userland. The upside is that cgroup is able to expose +useful features which may not be suitable for general consumption in a +reasonable time frame. It provides a relatively short path between +internal details and userland-visible interface. Of course, this +shortcut comes with high risk. We go through what we go through for +general kernel APIs for good reasons. It may end up leaking internal +details in a way which can exert significant pain by locking the +kernel into a contract that can't be maintained in a reasonable +manner. + +Also, due to the specific nature, cgroup and its controllers don't +tend to attract attention from a wide scope of developers. cgroup's +short history is already fraught with severely mis-designed +interfaces, unnecessary commitments to and exposing of internal +details, broken and dangerous implementations of various features. + +Keeping cgroup as an administration interface is both advantageous for +its role and imperative given its nature. Some of the cgroup features +may make sense for unprivileged access. If deemed justified, those +must be further abstracted and implemented as a different interface, +be it a system call or process-private filesystem, and survive through +the scrutiny that any interface for general consumption is required to +go through. + +Requiring CAP is not a complete solution but should serve as a +significant deterrent against spraying cgroup usages in non-privileged +programs. diff --git a/Documentation/devicetree/bindings/ata/ahci-platform.txt b/Documentation/devicetree/bindings/ata/ahci-platform.txt index 48b285ffa3a6..c96d8dcf98fd 100644 --- a/Documentation/devicetree/bindings/ata/ahci-platform.txt +++ b/Documentation/devicetree/bindings/ata/ahci-platform.txt @@ -4,10 +4,16 @@ SATA nodes are defined to describe on-chip Serial ATA controllers. Each SATA controller should have its own node. Required properties: -- compatible : compatible list, one of "snps,spear-ahci", - "snps,exynos5440-ahci", "ibm,476gtr-ahci", - "allwinner,sun4i-a10-ahci", "fsl,imx53-ahci" - "fsl,imx6q-ahci" or "snps,dwc-ahci" +- compatible : compatible string, one of: + - "allwinner,sun4i-a10-ahci" + - "fsl,imx53-ahci" + - "fsl,imx6q-ahci" + - "hisilicon,hisi-ahci" + - "ibm,476gtr-ahci" + - "marvell,armada-380-ahci" + - "snps,dwc-ahci" + - "snps,exynos5440-ahci" + - "snps,spear-ahci" - interrupts : <interrupt mapping for SATA IRQ> - reg : <registers mapping> diff --git a/Documentation/devicetree/bindings/clock/corenet-clock.txt b/Documentation/devicetree/bindings/clock/qoriq-clock.txt index 24711af48e30..5666812fc42b 100644 --- a/Documentation/devicetree/bindings/clock/corenet-clock.txt +++ b/Documentation/devicetree/bindings/clock/qoriq-clock.txt @@ -7,6 +7,14 @@ which can then be passed to a variety of internal logic, including cores and peripheral IP blocks. Please refer to the Reference Manual for details. +All references to "1.0" and "2.0" refer to the QorIQ chassis version to +which the chip complies. + +Chassis Version Example Chips +--------------- ------------- +1.0 p4080, p5020, p5040 +2.0 t4240, b4860, t1040 + 1. Clock Block Binding Required properties: @@ -85,7 +93,7 @@ Example for clock block and clock provider: #clock-cells = <0>; compatible = "fsl,qoriq-sysclk-1.0"; clock-output-names = "sysclk"; - } + }; pll0: pll0@800 { #clock-cells = <1>; diff --git a/Documentation/devicetree/bindings/dma/mmp-dma.txt b/Documentation/devicetree/bindings/dma/mmp-dma.txt index a4fa4efa1d83..7a802f64e5bd 100644 --- a/Documentation/devicetree/bindings/dma/mmp-dma.txt +++ b/Documentation/devicetree/bindings/dma/mmp-dma.txt @@ -1,17 +1,20 @@ * MARVELL MMP DMA controller Marvell Peripheral DMA Controller -Used platfroms: pxa688, pxa910, pxa3xx, etc +Used platforms: pxa688, pxa910, pxa3xx, etc Required properties: - compatible: Should be "marvell,pdma-1.0" - reg: Should contain DMA registers location and length. - interrupts: Either contain all of the per-channel DMA interrupts or one irq for pdma device -- #dma-channels: Number of DMA channels supported by the controller. + +Optional properties: +- #dma-channels: Number of DMA channels supported by the controller (defaults + to 32 when not specified) "marvell,pdma-1.0" -Used platfroms: pxa25x, pxa27x, pxa3xx, pxa93x, pxa168, pxa910, pxa688. +Used platforms: pxa25x, pxa27x, pxa3xx, pxa93x, pxa168, pxa910, pxa688. Examples: @@ -45,7 +48,7 @@ pdma: dma-controller@d4000000 { Marvell Two Channel DMA Controller used specifically for audio -Used platfroms: pxa688, pxa910 +Used platforms: pxa688, pxa910 Required properties: - compatible: Should be "marvell,adma-1.0" or "marvell,pxa910-squ" diff --git a/Documentation/devicetree/bindings/dma/xilinx/xilinx_vdma.txt b/Documentation/devicetree/bindings/dma/xilinx/xilinx_vdma.txt new file mode 100644 index 000000000000..1405ed071bb4 --- /dev/null +++ b/Documentation/devicetree/bindings/dma/xilinx/xilinx_vdma.txt @@ -0,0 +1,75 @@ +Xilinx AXI VDMA engine, it does transfers between memory and video devices. +It can be configured to have one channel or two channels. If configured +as two channels, one is to transmit to the video device and another is +to receive from the video device. + +Required properties: +- compatible: Should be "xlnx,axi-vdma-1.00.a" +- #dma-cells: Should be <1>, see "dmas" property below +- reg: Should contain VDMA registers location and length. +- xlnx,num-fstores: Should be the number of framebuffers as configured in h/w. +- dma-channel child node: Should have at least one channel and can have up to + two channels per device. This node specifies the properties of each + DMA channel (see child node properties below). + +Optional properties: +- xlnx,include-sg: Tells configured for Scatter-mode in + the hardware. +- xlnx,flush-fsync: Tells which channel to Flush on Frame sync. + It takes following values: + {1}, flush both channels + {2}, flush mm2s channel + {3}, flush s2mm channel + +Required child node properties: +- compatible: It should be either "xlnx,axi-vdma-mm2s-channel" or + "xlnx,axi-vdma-s2mm-channel". +- interrupts: Should contain per channel VDMA interrupts. +- xlnx,data-width: Should contain the stream data width, take values + {32,64...1024}. + +Optional child node properties: +- xlnx,include-dre: Tells hardware is configured for Data + Realignment Engine. +- xlnx,genlock-mode: Tells Genlock synchronization is + enabled/disabled in hardware. + +Example: +++++++++ + +axi_vdma_0: axivdma@40030000 { + compatible = "xlnx,axi-vdma-1.00.a"; + #dma_cells = <1>; + reg = < 0x40030000 0x10000 >; + xlnx,num-fstores = <0x8>; + xlnx,flush-fsync = <0x1>; + dma-channel@40030000 { + compatible = "xlnx,axi-vdma-mm2s-channel"; + interrupts = < 0 54 4 >; + xlnx,datawidth = <0x40>; + } ; + dma-channel@40030030 { + compatible = "xlnx,axi-vdma-s2mm-channel"; + interrupts = < 0 53 4 >; + xlnx,datawidth = <0x40>; + } ; +} ; + + +* DMA client + +Required properties: +- dmas: a list of <[Video DMA device phandle] [Channel ID]> pairs, + where Channel ID is '0' for write/tx and '1' for read/rx + channel. +- dma-names: a list of DMA channel names, one per "dmas" entry + +Example: +++++++++ + +vdmatest_0: vdmatest@0 { + compatible ="xlnx,axi-vdma-test-1.00.a"; + dmas = <&axi_vdma_0 0 + &axi_vdma_0 1>; + dma-names = "vdma0", "vdma1"; +} ; diff --git a/Documentation/devicetree/bindings/gpio/gpio_keys.txt b/Documentation/devicetree/bindings/input/gpio-keys.txt index 5c2c02140a62..5c2c02140a62 100644 --- a/Documentation/devicetree/bindings/gpio/gpio_keys.txt +++ b/Documentation/devicetree/bindings/input/gpio-keys.txt diff --git a/Documentation/devicetree/bindings/input/st-keyscan.txt b/Documentation/devicetree/bindings/input/st-keyscan.txt new file mode 100644 index 000000000000..51eb428e5c85 --- /dev/null +++ b/Documentation/devicetree/bindings/input/st-keyscan.txt @@ -0,0 +1,60 @@ +* ST Keyscan controller Device Tree bindings + +The ST keyscan controller Device Tree binding is based on the +matrix-keymap. + +Required properties: +- compatible: "st,sti-keyscan" + +- reg: Register base address and size of st-keyscan controller. + +- interrupts: Interrupt number for the st-keyscan controller. + +- clocks: Must contain one entry, for the module clock. + See ../clocks/clock-bindings.txt for details. + +- pinctrl: Should specify pin control groups used for this controller. + See ../pinctrl/pinctrl-bindings.txt for details. + +- linux,keymap: The keymap for keys as described in the binding document + devicetree/bindings/input/matrix-keymap.txt. + +- keypad,num-rows: Number of row lines connected to the keypad controller. + +- keypad,num-columns: Number of column lines connected to the keypad + controller. + +Optional property: +- st,debounce_us: Debouncing interval time in microseconds + +Example: + +keyscan: keyscan@fe4b0000 { + compatible = "st,sti-keyscan"; + reg = <0xfe4b0000 0x2000>; + interrupts = <GIC_SPI 212 IRQ_TYPE_NONE>; + clocks = <&CLK_SYSIN>; + pinctrl-names = "default"; + pinctrl-0 = <&pinctrl_keyscan>; + + keypad,num-rows = <4>; + keypad,num-columns = <4>; + st,debounce_us = <5000>; + + linux,keymap = < MATRIX_KEY(0x00, 0x00, KEY_F13) + MATRIX_KEY(0x00, 0x01, KEY_F9) + MATRIX_KEY(0x00, 0x02, KEY_F5) + MATRIX_KEY(0x00, 0x03, KEY_F1) + MATRIX_KEY(0x01, 0x00, KEY_F14) + MATRIX_KEY(0x01, 0x01, KEY_F10) + MATRIX_KEY(0x01, 0x02, KEY_F6) + MATRIX_KEY(0x01, 0x03, KEY_F2) + MATRIX_KEY(0x02, 0x00, KEY_F15) + MATRIX_KEY(0x02, 0x01, KEY_F11) + MATRIX_KEY(0x02, 0x02, KEY_F7) + MATRIX_KEY(0x02, 0x03, KEY_F3) + MATRIX_KEY(0x03, 0x00, KEY_F16) + MATRIX_KEY(0x03, 0x01, KEY_F12) + MATRIX_KEY(0x03, 0x02, KEY_F8) + MATRIX_KEY(0x03, 0x03, KEY_F4) >; + }; diff --git a/Documentation/devicetree/bindings/input/touchscreen/sun4i.txt b/Documentation/devicetree/bindings/input/touchscreen/sun4i.txt new file mode 100644 index 000000000000..aef57791f40b --- /dev/null +++ b/Documentation/devicetree/bindings/input/touchscreen/sun4i.txt @@ -0,0 +1,20 @@ +sun4i resistive touchscreen controller +-------------------------------------- + +Required properties: + - compatible: "allwinner,sun4i-a10-ts" + - reg: mmio address range of the chip + - interrupts: interrupt to which the chip is connected + +Optional properties: + - allwinner,ts-attached: boolean indicating that an actual touchscreen is + attached to the controller + +Example: + + rtp: rtp@01c25000 { + compatible = "allwinner,sun4i-a10-ts"; + reg = <0x01c25000 0x100>; + interrupts = <29>; + allwinner,ts-attached; + }; diff --git a/Documentation/devicetree/bindings/input/touchscreen/touchscreen.txt b/Documentation/devicetree/bindings/input/touchscreen/touchscreen.txt new file mode 100644 index 000000000000..d8e06163c54e --- /dev/null +++ b/Documentation/devicetree/bindings/input/touchscreen/touchscreen.txt @@ -0,0 +1,27 @@ +General Touchscreen Properties: + +Optional properties for Touchscreens: + - touchscreen-size-x : horizontal resolution of touchscreen + (in pixels) + - touchscreen-size-y : vertical resolution of touchscreen + (in pixels) + - touchscreen-max-pressure : maximum reported pressure (arbitrary range + dependent on the controller) + - touchscreen-fuzz-x : horizontal noise value of the absolute input + device (in pixels) + - touchscreen-fuzz-y : vertical noise value of the absolute input + device (in pixels) + - touchscreen-fuzz-pressure : pressure noise value of the absolute input + device (arbitrary range dependent on the + controller) + - touchscreen-inverted-x : X axis is inverted (boolean) + - touchscreen-inverted-y : Y axis is inverted (boolean) + +Deprecated properties for Touchscreens: + - x-size : deprecated name for touchscreen-size-x + - y-size : deprecated name for touchscreen-size-y + - moving-threshold : deprecated name for a combination of + touchscreen-fuzz-x and touchscreen-fuzz-y + - contact-threshold : deprecated name for touchscreen-fuzz-pressure + - x-invert : deprecated name for touchscreen-inverted-x + - y-invert : deprecated name for touchscreen-inverted-y diff --git a/Documentation/devicetree/bindings/input/touchscreen/tsc2005.txt b/Documentation/devicetree/bindings/input/touchscreen/tsc2005.txt new file mode 100644 index 000000000000..4b641c7bf1c2 --- /dev/null +++ b/Documentation/devicetree/bindings/input/touchscreen/tsc2005.txt @@ -0,0 +1,42 @@ +* Texas Instruments tsc2005 touchscreen controller + +Required properties: + - compatible : "ti,tsc2005" + - reg : SPI device address + - spi-max-frequency : Maximal SPI speed + - interrupts : IRQ specifier + - reset-gpios : GPIO specifier + - vio-supply : Regulator specifier + +Optional properties: + - ti,x-plate-ohms : integer, resistance of the touchscreen's X plates + in ohm (defaults to 280) + - ti,esd-recovery-timeout-ms : integer, if the touchscreen does not respond after + the configured time (in milli seconds), the driver + will reset it. This is disabled by default. + - properties defined in touchscreen.txt + +Example: + +&mcspi1 { + tsc2005@0 { + compatible = "ti,tsc2005"; + spi-max-frequency = <6000000>; + reg = <0>; + + vio-supply = <&vio>; + + reset-gpios = <&gpio4 8 GPIO_ACTIVE_HIGH>; /* 104 */ + interrupts-extended = <&gpio4 4 IRQ_TYPE_EDGE_RISING>; /* 100 */ + + touchscreen-fuzz-x = <4>; + touchscreen-fuzz-y = <7>; + touchscreen-fuzz-pressure = <2>; + touchscreen-max-x = <4096>; + touchscreen-max-y = <4096>; + touchscreen-max-pressure = <2048>; + + ti,x-plate-ohms = <280>; + ti,esd-recovery-timeout-ms = <8000>; + }; +} diff --git a/Documentation/devicetree/bindings/mfd/bfticu.txt b/Documentation/devicetree/bindings/mfd/bfticu.txt new file mode 100644 index 000000000000..65c90776c620 --- /dev/null +++ b/Documentation/devicetree/bindings/mfd/bfticu.txt @@ -0,0 +1,25 @@ +KEYMILE bfticu Chassis Management FPGA + +The bfticu is a multifunction device that manages the whole chassis. +Its main functionality is to collect IRQs from the whole chassis and signals +them to a single controller. + +Required properties: +- compatible: "keymile,bfticu" +- interrupt-controller: the bfticu FPGA is an interrupt controller +- interrupts: the main IRQ line to signal the collected IRQs +- #interrupt-cells : is 2 and their usage is compliant to the 2 cells variant + of Documentation/devicetree/bindings/interrupt-controller/interrupts.txt +- interrupt-parent: the parent IRQ ctrl the main IRQ is connected to +- reg: access on the parent local bus (chip select, offset in chip select, size) + +Example: + + chassis-mgmt@3,0 { + compatible = "keymile,bfticu"; + interrupt-controller; + #interrupt-cells = <2>; + reg = <3 0 0x100>; + interrupt-parent = <&mpic>; + interrupts = <6 1 0 0>; + }; diff --git a/Documentation/devicetree/bindings/mfd/qriox.txt b/Documentation/devicetree/bindings/mfd/qriox.txt new file mode 100644 index 000000000000..f301e2d4ce76 --- /dev/null +++ b/Documentation/devicetree/bindings/mfd/qriox.txt @@ -0,0 +1,17 @@ +KEYMILE qrio Board Control CPLD + +The qrio is a multifunction device that controls the KEYMILE boards based on +the kmp204x design. +It is consists of a reset controller, watchdog timer, LEDs, and 2 IRQ capable +GPIO blocks. + +Required properties: +- compatible: "keymile,qriox" +- reg: access on the parent local bus (chip select, offset in chip select, size) + +Example: + + board-control@1,0 { + compatible = "keymile,qriox"; + reg = <1 0 0x80>; + }; diff --git a/Documentation/devicetree/bindings/mmc/mmc.txt b/Documentation/devicetree/bindings/mmc/mmc.txt index 9dce540771fb..3c18001dfd5d 100644 --- a/Documentation/devicetree/bindings/mmc/mmc.txt +++ b/Documentation/devicetree/bindings/mmc/mmc.txt @@ -38,6 +38,8 @@ Optional properties: - mmc-highspeed-ddr-1_2v: eMMC high-speed DDR mode(1.2V I/O) is supported - mmc-hs200-1_8v: eMMC HS200 mode(1.8V I/O) is supported - mmc-hs200-1_2v: eMMC HS200 mode(1.2V I/O) is supported +- mmc-hs400-1_8v: eMMC HS400 mode(1.8V I/O) is supported +- mmc-hs400-1_2v: eMMC HS400 mode(1.2V I/O) is supported *NOTE* on CD and WP polarity. To use common for all SD/MMC host controllers line polarity properties, we have to fix the meaning of the "normal" and "inverted" diff --git a/Documentation/devicetree/bindings/mmc/moxa,moxart-mmc.txt b/Documentation/devicetree/bindings/mmc/moxa,moxart-mmc.txt new file mode 100644 index 000000000000..b63819149f22 --- /dev/null +++ b/Documentation/devicetree/bindings/mmc/moxa,moxart-mmc.txt @@ -0,0 +1,30 @@ +MOXA ART MMC Host Controller Interface + + Inherits from mmc binding[1]. + + [1] Documentation/devicetree/bindings/mmc/mmc.txt + +Required properties: + +- compatible : Must be "moxa,moxart-mmc" or "faraday,ftsdc010" +- reg : Should contain registers location and length +- interrupts : Should contain the interrupt number +- clocks : Should contain phandle for the clock feeding the MMC controller + +Optional properties: + +- dmas : Should contain two DMA channels, line request number must be 5 for + both channels +- dma-names : Must be "tx", "rx" + +Example: + + mmc: mmc@98e00000 { + compatible = "moxa,moxart-mmc"; + reg = <0x98e00000 0x5C>; + interrupts = <5 0>; + clocks = <&clk_apb>; + dmas = <&dma 5>, + <&dma 5>; + dma-names = "tx", "rx"; + }; diff --git a/Documentation/devicetree/bindings/mmc/synopsys-dw-mshc.txt b/Documentation/devicetree/bindings/mmc/synopsys-dw-mshc.txt index 8f3f13315358..2d4a7258a10d 100644 --- a/Documentation/devicetree/bindings/mmc/synopsys-dw-mshc.txt +++ b/Documentation/devicetree/bindings/mmc/synopsys-dw-mshc.txt @@ -69,10 +69,6 @@ Optional properties: * supports-highspeed: Enables support for high speed cards (up to 50MHz) -* caps2-mmc-hs200-1_8v: Supports mmc HS200 SDR 1.8V mode - -* caps2-mmc-hs200-1_2v: Supports mmc HS200 SDR 1.2V mode - * broken-cd: as documented in mmc core bindings. * vmmc-supply: The phandle to the regulator to use for vmmc. If this is @@ -103,7 +99,6 @@ board specific portions as listed below. clock-freq-min-max = <400000 200000000>; num-slots = <1>; supports-highspeed; - caps2-mmc-hs200-1_8v; broken-cd; fifo-depth = <0x80>; card-detect-delay = <200>; diff --git a/Documentation/devicetree/bindings/mmc/usdhi6rol0.txt b/Documentation/devicetree/bindings/mmc/usdhi6rol0.txt new file mode 100644 index 000000000000..8babdaa8623b --- /dev/null +++ b/Documentation/devicetree/bindings/mmc/usdhi6rol0.txt @@ -0,0 +1,33 @@ +* Renesas usdhi6rol0 SD/SDIO host controller + +Required properties: + +- compatible: must be + "renesas,usdhi6rol0" +- interrupts: 3 interrupts, named "card detect", "data" and "SDIO" must be + specified +- clocks: a clock binding for the IMCLK input + +Optional properties: + +- vmmc-supply: a phandle of a regulator, supplying Vcc to the card +- vqmmc-supply: a phandle of a regulator, supplying VccQ to the card + +Additionally any standard mmc bindings from mmc.txt can be used. + +Example: + +sd0: sd@ab000000 { + compatible = "renesas,usdhi6rol0"; + reg = <0xab000000 0x200>; + interrupts = <0 23 0x4 + 0 24 0x4 + 0 25 0x4>; + interrupt-names = "card detect", "data", "SDIO"; + bus-width = <4>; + max-frequency = <50000000>; + cap-power-off-card; + clocks = <&imclk>; + vmmc-supply = <&vcc_sd0>; + vqmmc-supply = <&vccq_sd0>; +}; diff --git a/Documentation/devicetree/bindings/powerpc/4xx/akebono.txt b/Documentation/devicetree/bindings/powerpc/4xx/akebono.txt new file mode 100644 index 000000000000..db939210e29d --- /dev/null +++ b/Documentation/devicetree/bindings/powerpc/4xx/akebono.txt @@ -0,0 +1,54 @@ + +IBM Akebono board device tree +============================= + +The IBM Akebono board is a development board for the PPC476GTR SoC. + +0) The root node + + Required properties: + + - model : "ibm,akebono". + - compatible : "ibm,akebono" , "ibm,476gtr". + +1.a) The Secure Digital Host Controller Interface (SDHCI) node + + Represent the Secure Digital Host Controller Interfaces. + + Required properties: + + - compatible : should be "ibm,476gtr-sdhci","generic-sdhci". + - reg : should contain the SDHCI registers location and length. + - interrupt-parent : a phandle for the interrupt controller. + - interrupts : should contain the SDHCI interrupt. + +1.b) The Advanced Host Controller Interface (AHCI) SATA node + + Represents the advanced host controller SATA interface. + + Required properties: + + - compatible : should be "ibm,476gtr-ahci". + - reg : should contain the AHCI registers location and length. + - interrupt-parent : a phandle for the interrupt controller. + - interrupts : should contain the AHCI interrupt. + +1.c) The FPGA node + + The Akebono board stores some board information such as the revision + number in an FPGA which is represented by this node. + + Required properties: + + - compatible : should be "ibm,akebono-fpga". + - reg : should contain the FPGA registers location and length. + +1.d) The AVR node + + The Akebono board has an Atmel AVR microprocessor attached to the I2C + bus as a power controller for the board. + + Required properties: + + - compatible : should be "ibm,akebono-avr". + - reg : should contain the I2C bus address for the AVR. diff --git a/Documentation/devicetree/bindings/powerpc/4xx/hsta.txt b/Documentation/devicetree/bindings/powerpc/4xx/hsta.txt new file mode 100644 index 000000000000..c737c8338705 --- /dev/null +++ b/Documentation/devicetree/bindings/powerpc/4xx/hsta.txt @@ -0,0 +1,19 @@ + +ppc476gtr High Speed Serial Assist (HSTA) node +============================================== + +The 476gtr SoC contains a high speed serial assist module attached +between the plb4 and plb6 system buses to provide high speed data +transfer between memory and system peripherals as well as support for +PCI message signalled interrupts. + +Currently only the MSI support is used by Linux using the following +device tree entries: + +Require properties: +- compatible : "ibm,476gtr-hsta-msi", "ibm,hsta-msi" +- reg : register mapping for the HSTA MSI space +- interrupt-parent : parent controller for mapping interrupts +- interrupts : ordered interrupt mapping for each MSI in the register + space. The first interrupt should be associated with a + register offset of 0x00, the second to 0x10, etc. diff --git a/Documentation/devicetree/bindings/powerpc/fsl/board.txt b/Documentation/devicetree/bindings/powerpc/fsl/board.txt index 380914e965e0..700dec4774fa 100644 --- a/Documentation/devicetree/bindings/powerpc/fsl/board.txt +++ b/Documentation/devicetree/bindings/powerpc/fsl/board.txt @@ -67,3 +67,20 @@ Example: gpio-controller; }; }; + +* Freescale on-board FPGA connected on I2C bus + +Some Freescale boards like BSC9132QDS have on board FPGA connected on +the i2c bus. + +Required properties: +- compatible: Should be a board-specific string followed by a string + indicating the type of FPGA. Example: + "fsl,<board>-fpga", "fsl,fpga-qixis-i2c" +- reg: Should contain the address of the FPGA + +Example: + fpga: fpga@66 { + compatible = "fsl,bsc9132qds-fpga", "fsl,fpga-qixis-i2c"; + reg = <0x66>; + }; diff --git a/Documentation/devicetree/bindings/powerpc/fsl/ccf.txt b/Documentation/devicetree/bindings/powerpc/fsl/ccf.txt new file mode 100644 index 000000000000..454da7e08acd --- /dev/null +++ b/Documentation/devicetree/bindings/powerpc/fsl/ccf.txt @@ -0,0 +1,46 @@ +Freescale CoreNet Coherency Fabric(CCF) Device Tree Binding + +DESCRIPTION + +The CoreNet coherency fabric is a fabric-oriented, connectivity infrastructure +that enables the implementation of coherent, multicore systems. + +Required properties: + +- compatible: <string list> + fsl,corenet1-cf - CoreNet coherency fabric version 1. + Example chips: T4240, B4860 + + fsl,corenet2-cf - CoreNet coherency fabric version 2. + Example chips: P5040, P5020, P4080, P3041, P2041 + + fsl,corenet-cf - Used to represent the common registers + between CCF version 1 and CCF version 2. This compatible + is retained for compatibility reasons, as it was already + used for both CCF version 1 chips and CCF version 2 + chips. It should be specified after either + "fsl,corenet1-cf" or "fsl,corenet2-cf". + +- reg: <prop-encoded-array> + A standard property. Represents the CCF registers. + +- interrupts: <prop-encoded-array> + Interrupt mapping for CCF error interrupt. + +- fsl,ccf-num-csdids: <u32> + Specifies the number of Coherency Subdomain ID Port Mapping + Registers that are supported by the CCF. + +- fsl,ccf-num-snoopids: <u32> + Specifies the number of Snoop ID Port Mapping Registers that + are supported by CCF. + +Example: + + corenet-cf@18000 { + compatible = "fsl,corenet2-cf", "fsl,corenet-cf"; + reg = <0x18000 0x1000>; + interrupts = <16 2 1 31>; + fsl,ccf-num-csdids = <32>; + fsl,ccf-num-snoopids = <32>; + }; diff --git a/Documentation/devicetree/bindings/powerpc/fsl/cpus.txt b/Documentation/devicetree/bindings/powerpc/fsl/cpus.txt index 922c30ad90d1..f8cd2397aa04 100644 --- a/Documentation/devicetree/bindings/powerpc/fsl/cpus.txt +++ b/Documentation/devicetree/bindings/powerpc/fsl/cpus.txt @@ -20,3 +20,14 @@ PROPERTIES a property named fsl,eref-[CAT], where [CAT] is the abbreviated category name with all uppercase letters converted to lowercase, indicates that the category is supported by the implementation. + + - fsl,portid-mapping + Usage: optional + Value type: <u32> + Definition: The Coherency Subdomain ID Port Mapping Registers and + Snoop ID Port Mapping registers, which are part of the CoreNet + Coherency fabric (CCF), provide a CoreNet Coherency Subdomain + ID/CoreNet Snoop ID to cpu mapping functions. Certain bits from + these registers should be set if the coresponding CPU should be + snooped. This property defines a bitmask which selects the bit + that should be set if this cpu should be snooped. diff --git a/Documentation/devicetree/bindings/powerpc/fsl/pamu.txt b/Documentation/devicetree/bindings/powerpc/fsl/pamu.txt index 1f5e329f756c..c2b2899885f2 100644 --- a/Documentation/devicetree/bindings/powerpc/fsl/pamu.txt +++ b/Documentation/devicetree/bindings/powerpc/fsl/pamu.txt @@ -34,6 +34,15 @@ Optional properties: for legacy drivers. - interrupt-parent : <phandle> Phandle to interrupt controller +- fsl,portid-mapping : <u32> + The Coherency Subdomain ID Port Mapping Registers and + Snoop ID Port Mapping registers, which are part of the + CoreNet Coherency fabric (CCF), provide a CoreNet + Coherency Subdomain ID/CoreNet Snoop ID to pamu mapping + functions. Certain bits from these registers should be + set if PAMUs should be snooped. This property defines + a bitmask which selects the bits that should be set if + PAMUs should be snooped. Child nodes: @@ -88,6 +97,7 @@ Example: compatible = "fsl,pamu-v1.0", "fsl,pamu"; reg = <0x20000 0x5000>; ranges = <0 0x20000 0x5000>; + fsl,portid-mapping = <0xf80000>; #address-cells = <1>; #size-cells = <1>; interrupts = < diff --git a/Documentation/devicetree/bindings/vendor-prefixes.txt b/Documentation/devicetree/bindings/vendor-prefixes.txt index 5261271046ce..4d7f3758d1b4 100644 --- a/Documentation/devicetree/bindings/vendor-prefixes.txt +++ b/Documentation/devicetree/bindings/vendor-prefixes.txt @@ -142,3 +142,4 @@ wm Wondermedia Technologies, Inc. xes Extreme Engineering Solutions (X-ES) xlnx Xilinx zyxel ZyXEL Communications Corp. +zarlink Zarlink Semiconductor diff --git a/Documentation/filesystems/f2fs.txt b/Documentation/filesystems/f2fs.txt index 25311e113e75..51afba17bbae 100644 --- a/Documentation/filesystems/f2fs.txt +++ b/Documentation/filesystems/f2fs.txt @@ -461,11 +461,11 @@ The number of blocks and buckets are determined by, # of blocks in level #n = | `- 4, Otherwise - ,- 2^ (n + dir_level), - | if n < MAX_DIR_HASH_DEPTH / 2, + ,- 2^(n + dir_level), + | if n + dir_level < MAX_DIR_HASH_DEPTH / 2, # of buckets in level #n = | - `- 2^((MAX_DIR_HASH_DEPTH / 2 + dir_level) - 1), - Otherwise + `- 2^((MAX_DIR_HASH_DEPTH / 2) - 1), + Otherwise When F2FS finds a file name in a directory, at first a hash value of the file name is calculated. Then, F2FS scans the hash table in level #0 to find the diff --git a/Documentation/filesystems/nfs/nfs41-server.txt b/Documentation/filesystems/nfs/nfs41-server.txt index b930ad087780..c49cd7e796e7 100644 --- a/Documentation/filesystems/nfs/nfs41-server.txt +++ b/Documentation/filesystems/nfs/nfs41-server.txt @@ -176,7 +176,5 @@ Nonstandard compound limitations: ca_maxrequestsize request and a ca_maxresponsesize reply, so we may fail to live up to the promise we made in CREATE_SESSION fore channel negotiation. -* No more than one read-like operation allowed per compound; encoding - replies that cross page boundaries (except for read data) not handled. See also http://wiki.linux-nfs.org/wiki/index.php/Server_4.0_and_4.1_issues. diff --git a/Documentation/platform/x86-laptop-drivers.txt b/Documentation/platform/x86-laptop-drivers.txt new file mode 100644 index 000000000000..01facd2590bb --- /dev/null +++ b/Documentation/platform/x86-laptop-drivers.txt @@ -0,0 +1,18 @@ +compal-laptop +============= +List of supported hardware: + +by Compal: + Compal FL90/IFL90 + Compal FL91/IFL91 + Compal FL92/JFL92 + Compal FT00/IFT00 + +by Dell: + Dell Vostro 1200 + Dell Mini 9 (Inspiron 910) + Dell Mini 10 (Inspiron 1010) + Dell Mini 10v (Inspiron 1011) + Dell Mini 1012 (Inspiron 1012) + Dell Inspiron 11z (Inspiron 1110) + Dell Mini 12 (Inspiron 1210) diff --git a/Documentation/powerpc/cpu_families.txt b/Documentation/powerpc/cpu_families.txt new file mode 100644 index 000000000000..fc08e22feb1a --- /dev/null +++ b/Documentation/powerpc/cpu_families.txt @@ -0,0 +1,221 @@ +CPU Families +============ + +This document tries to summarise some of the different cpu families that exist +and are supported by arch/powerpc. + + +Book3S (aka sPAPR) +------------------ + + - Hash MMU + - Mix of 32 & 64 bit + + +--------------+ +----------------+ + | Old POWER | --------------> | RS64 (threads) | + +--------------+ +----------------+ + | + | + v + +--------------+ +----------------+ +------+ + | 601 | --------------> | 603 | ---> | e300 | + +--------------+ +----------------+ +------+ + | | + | | + v v + +--------------+ +----------------+ +-------+ + | 604 | | 750 (G3) | ---> | 750CX | + +--------------+ +----------------+ +-------+ + | | | + | | | + v v v + +--------------+ +----------------+ +-------+ + | 620 (64 bit) | | 7400 | | 750CL | + +--------------+ +----------------+ +-------+ + | | | + | | | + v v v + +--------------+ +----------------+ +-------+ + | POWER3/630 | | 7410 | | 750FX | + +--------------+ +----------------+ +-------+ + | | + | | + v v + +--------------+ +----------------+ + | POWER3+ | | 7450 | + +--------------+ +----------------+ + | | + | | + v v + +--------------+ +----------------+ + | POWER4 | | 7455 | + +--------------+ +----------------+ + | | + | | + v v + +--------------+ +-------+ +----------------+ + | POWER4+ | --> | 970 | | 7447 | + +--------------+ +-------+ +----------------+ + | | | + | | | + v v v + +--------------+ +-------+ +----------------+ + | POWER5 | | 970FX | | 7448 | + +--------------+ +-------+ +----------------+ + | | | + | | | + v v v + +--------------+ +-------+ +----------------+ + | POWER5+ | | 970MP | | e600 | + +--------------+ +-------+ +----------------+ + | + | + v + +--------------+ + | POWER5++ | + +--------------+ + | + | + v + +--------------+ +-------+ + | POWER6 | <-?-> | Cell | + +--------------+ +-------+ + | + | + v + +--------------+ + | POWER7 | + +--------------+ + | + | + v + +--------------+ + | POWER7+ | + +--------------+ + | + | + v + +--------------+ + | POWER8 | + +--------------+ + + + +---------------+ + | PA6T (64 bit) | + +---------------+ + + +IBM BookE +--------- + + - Software loaded TLB. + - All 32 bit + + +--------------+ + | 401 | + +--------------+ + | + | + v + +--------------+ + | 403 | + +--------------+ + | + | + v + +--------------+ + | 405 | + +--------------+ + | + | + v + +--------------+ + | 440 | + +--------------+ + | + | + v + +--------------+ +----------------+ + | 450 | --> | BG/P | + +--------------+ +----------------+ + | + | + v + +--------------+ + | 460 | + +--------------+ + | + | + v + +--------------+ + | 476 | + +--------------+ + + +Motorola/Freescale 8xx +---------------------- + + - Software loaded with hardware assist. + - All 32 bit + + +-------------+ + | MPC8xx Core | + +-------------+ + + +Freescale BookE +--------------- + + - Software loaded TLB. + - e6500 adds HW loaded indirect TLB entries. + - Mix of 32 & 64 bit + + +--------------+ + | e200 | + +--------------+ + + + +--------------------------------+ + | e500 | + +--------------------------------+ + | + | + v + +--------------------------------+ + | e500v2 | + +--------------------------------+ + | + | + v + +--------------------------------+ + | e500mc (Book3e) | + +--------------------------------+ + | + | + v + +--------------------------------+ + | e5500 (64 bit) | + +--------------------------------+ + | + | + v + +--------------------------------+ + | e6500 (HW TLB) (Multithreaded) | + +--------------------------------+ + + +IBM A2 core +----------- + + - Book3E, software loaded TLB + HW loaded indirect TLB entries. + - 64 bit + + +--------------+ +----------------+ + | A2 core | --> | WSP | + +--------------+ +----------------+ + | + | + v + +--------------+ + | BG/Q | + +--------------+ diff --git a/Documentation/scsi/LICENSE.qla2xxx b/Documentation/scsi/LICENSE.qla2xxx index 5020b7b5a244..52f0b4359234 100644 --- a/Documentation/scsi/LICENSE.qla2xxx +++ b/Documentation/scsi/LICENSE.qla2xxx @@ -1,4 +1,4 @@ -Copyright (c) 2003-2013 QLogic Corporation +Copyright (c) 2003-2014 QLogic Corporation QLogic Linux FC-FCoE Driver This program includes a device driver for Linux 3.x. diff --git a/Documentation/security/Smack.txt b/Documentation/security/Smack.txt index 5ea996f21d6c..b6ef7e9dba30 100644 --- a/Documentation/security/Smack.txt +++ b/Documentation/security/Smack.txt @@ -204,6 +204,16 @@ onlycap these capabilities are effective at for processes with any label. The value is set by writing the desired label to the file or cleared by writing "-" to the file. +ptrace + This is used to define the current ptrace policy + 0 - default: this is the policy that relies on smack access rules. + For the PTRACE_READ a subject needs to have a read access on + object. For the PTRACE_ATTACH a read-write access is required. + 1 - exact: this is the policy that limits PTRACE_ATTACH. Attach is + only allowed when subject's and object's labels are equal. + PTRACE_READ is not affected. Can be overriden with CAP_SYS_PTRACE. + 2 - draconian: this policy behaves like the 'exact' above with an + exception that it can't be overriden with CAP_SYS_PTRACE. revoke-subject Writing a Smack label here sets the access to '-' for all access rules with that subject label. diff --git a/Documentation/trace/ftrace.txt b/Documentation/trace/ftrace.txt index bd365988e8d8..2479b2a0c77c 100644 --- a/Documentation/trace/ftrace.txt +++ b/Documentation/trace/ftrace.txt @@ -2003,6 +2003,32 @@ want, depending on your needs. 360.774530 | 1) 0.594 us | __phys_addr(); +The function name is always displayed after the closing bracket +for a function if the start of that function is not in the +trace buffer. + +Display of the function name after the closing bracket may be +enabled for functions whose start is in the trace buffer, +allowing easier searching with grep for function durations. +It is default disabled. + + hide: echo nofuncgraph-tail > trace_options + show: echo funcgraph-tail > trace_options + + Example with nofuncgraph-tail (default): + 0) | putname() { + 0) | kmem_cache_free() { + 0) 0.518 us | __phys_addr(); + 0) 1.757 us | } + 0) 2.861 us | } + + Example with funcgraph-tail: + 0) | putname() { + 0) | kmem_cache_free() { + 0) 0.518 us | __phys_addr(); + 0) 1.757 us | } /* kmem_cache_free() */ + 0) 2.861 us | } /* putname() */ + You can put some comments on specific functions by using trace_printk() For example, if you want to put a comment inside the __might_sleep() function, you just have to include diff --git a/Documentation/trace/tracepoints.txt b/Documentation/trace/tracepoints.txt index 6b018b53177a..a3efac621c5a 100644 --- a/Documentation/trace/tracepoints.txt +++ b/Documentation/trace/tracepoints.txt @@ -115,6 +115,30 @@ If the tracepoint has to be used in kernel modules, an EXPORT_TRACEPOINT_SYMBOL_GPL() or EXPORT_TRACEPOINT_SYMBOL() can be used to export the defined tracepoints. +If you need to do a bit of work for a tracepoint parameter, and +that work is only used for the tracepoint, that work can be encapsulated +within an if statement with the following: + + if (trace_foo_bar_enabled()) { + int i; + int tot = 0; + + for (i = 0; i < count; i++) + tot += calculate_nuggets(); + + trace_foo_bar(tot); + } + +All trace_<tracepoint>() calls have a matching trace_<tracepoint>_enabled() +function defined that returns true if the tracepoint is enabled and +false otherwise. The trace_<tracepoint>() should always be within the +block of the if (trace_<tracepoint>_enabled()) to prevent races between +the tracepoint being enabled and the check being seen. + +The advantage of using the trace_<tracepoint>_enabled() is that it uses +the static_key of the tracepoint to allow the if statement to be implemented +with jump labels and avoid conditional branches. + Note: The convenience macro TRACE_EVENT provides an alternative way to define tracepoints. Check http://lwn.net/Articles/379903, http://lwn.net/Articles/381064 and http://lwn.net/Articles/383362 diff --git a/MAINTAINERS b/MAINTAINERS index 1b22565c59ac..b4a66b9d6b4d 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -2384,16 +2384,35 @@ L: netdev@vger.kernel.org S: Maintained F: drivers/connector/ -CONTROL GROUPS (CGROUPS) +CONTROL GROUP (CGROUP) M: Tejun Heo <tj@kernel.org> M: Li Zefan <lizefan@huawei.com> -L: containers@lists.linux-foundation.org L: cgroups@vger.kernel.org T: git git://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup.git S: Maintained +F: Documentation/cgroups/ F: include/linux/cgroup* F: kernel/cgroup* -F: mm/*cgroup* + +CONTROL GROUP - CPUSET +M: Li Zefan <lizefan@huawei.com> +L: cgroups@vger.kernel.org +W: http://www.bullopensource.org/cpuset/ +W: http://oss.sgi.com/projects/cpusets/ +T: git git://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup.git +S: Maintained +F: Documentation/cgroups/cpusets.txt +F: include/linux/cpuset.h +F: kernel/cpuset.c + +CONTROL GROUP - MEMORY RESOURCE CONTROLLER (MEMCG) +M: Johannes Weiner <hannes@cmpxchg.org> +M: Michal Hocko <mhocko@suse.cz> +L: cgroups@vger.kernel.org +L: linux-mm@kvack.org +S: Maintained +F: mm/memcontrol.c +F: mm/page_cgroup.c CORETEMP HARDWARE MONITORING DRIVER M: Fenghua Yu <fenghua.yu@intel.com> @@ -2464,17 +2483,6 @@ M: Thomas Renninger <trenn@suse.de> S: Maintained F: tools/power/cpupower/ -CPUSETS -M: Li Zefan <lizefan@huawei.com> -L: cgroups@vger.kernel.org -W: http://www.bullopensource.org/cpuset/ -W: http://oss.sgi.com/projects/cpusets/ -T: git git://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup.git -S: Maintained -F: Documentation/cgroups/cpusets.txt -F: include/linux/cpuset.h -F: kernel/cpuset.c - CRAMFS FILESYSTEM W: http://sourceforge.net/projects/cramfs/ S: Orphan / Obsolete @@ -2692,6 +2700,15 @@ S: Orphan F: Documentation/networking/decnet.txt F: net/decnet/ +DECSTATION PLATFORM SUPPORT +M: "Maciej W. Rozycki" <macro@linux-mips.org> +L: linux-mips@linux-mips.org +W: http://www.linux-mips.org/wiki/DECstation +S: Maintained +F: arch/mips/dec/ +F: arch/mips/include/asm/dec/ +F: arch/mips/include/asm/mach-dec/ + DEFXX FDDI NETWORK DRIVER M: "Maciej W. Rozycki" <macro@linux-mips.org> S: Maintained @@ -3775,7 +3792,8 @@ F: fs/fscache/ F: include/linux/fscache*.h F2FS FILE SYSTEM -M: Jaegeuk Kim <jaegeuk.kim@samsung.com> +M: Jaegeuk Kim <jaegeuk@kernel.org> +M: Changman Lee <cm224.lee@samsung.com> L: linux-f2fs-devel@lists.sourceforge.net W: http://en.wikipedia.org/wiki/F2FS T: git git://git.kernel.org/pub/scm/linux/kernel/git/jaegeuk/f2fs.git @@ -4373,7 +4391,7 @@ S: Supported F: drivers/crypto/nx/ IBM Power 842 compression accelerator -M: Robert Jennings <rcj@linux.vnet.ibm.com> +M: Nathan Fontenot <nfont@linux.vnet.ibm.com> S: Supported F: drivers/crypto/nx/nx-842.c F: include/linux/nx842.h @@ -4389,12 +4407,18 @@ L: netdev@vger.kernel.org S: Supported F: drivers/net/ethernet/ibm/ibmveth.* -IBM Power Virtual SCSI/FC Device Drivers -M: Robert Jennings <rcj@linux.vnet.ibm.com> +IBM Power Virtual SCSI Device Drivers +M: Nathan Fontenot <nfont@linux.vnet.ibm.com> +L: linux-scsi@vger.kernel.org +S: Supported +F: drivers/scsi/ibmvscsi/ibmvscsi* +F: drivers/scsi/ibmvscsi/viosrp.h + +IBM Power Virtual FC Device Drivers +M: Brian King <brking@linux.vnet.ibm.com> L: linux-scsi@vger.kernel.org S: Supported -F: drivers/scsi/ibmvscsi/ -X: drivers/scsi/ibmvscsi/ibmvstgt.c +F: drivers/scsi/ibmvscsi/ibmvfc* IBM ServeRAID RAID DRIVER P: Jack Hammer @@ -5757,17 +5781,6 @@ F: include/linux/memory_hotplug.h F: include/linux/vmalloc.h F: mm/ -MEMORY RESOURCE CONTROLLER -M: Johannes Weiner <hannes@cmpxchg.org> -M: Michal Hocko <mhocko@suse.cz> -M: Balbir Singh <bsingharora@gmail.com> -M: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com> -L: cgroups@vger.kernel.org -L: linux-mm@kvack.org -S: Maintained -F: mm/memcontrol.c -F: mm/page_cgroup.c - MEMORY TECHNOLOGY DEVICES (MTD) M: David Woodhouse <dwmw2@infradead.org> M: Brian Norris <computersforpeace@gmail.com> @@ -5961,6 +5974,7 @@ M: Chris Ball <chris@printf.net> M: Ulf Hansson <ulf.hansson@linaro.org> L: linux-mmc@vger.kernel.org T: git git://git.kernel.org/pub/scm/linux/kernel/git/cjb/mmc.git +T: git git://git.linaro.org/people/ulf.hansson/mmc.git S: Maintained F: drivers/mmc/ F: include/linux/mmc/ @@ -6022,6 +6036,28 @@ M: Petr Vandrovec <petr@vandrovec.name> S: Odd Fixes F: fs/ncpfs/ +NCR 5380 SCSI DRIVERS +M: Finn Thain <fthain@telegraphics.com.au> +M: Michael Schmitz <schmitzmic@gmail.com> +L: linux-scsi@vger.kernel.org +S: Maintained +F: Documentation/scsi/g_NCR5380.txt +F: drivers/scsi/NCR5380.* +F: drivers/scsi/arm/cumana_1.c +F: drivers/scsi/arm/oak.c +F: drivers/scsi/atari_NCR5380.c +F: drivers/scsi/atari_scsi.* +F: drivers/scsi/dmx3191d.c +F: drivers/scsi/dtc.* +F: drivers/scsi/g_NCR5380.* +F: drivers/scsi/g_NCR5380_mmio.c +F: drivers/scsi/mac_scsi.* +F: drivers/scsi/pas16.* +F: drivers/scsi/sun3_NCR5380.c +F: drivers/scsi/sun3_scsi.* +F: drivers/scsi/sun3_scsi_vme.c +F: drivers/scsi/t128.* + NCR DUAL 700 SCSI DRIVER (MICROCHANNEL) M: "James E.J. Bottomley" <James.Bottomley@HansenPartnership.com> L: linux-scsi@vger.kernel.org @@ -9068,7 +9104,7 @@ F: include/linux/toshiba.h F: include/uapi/linux/toshiba.h TMIO MMC DRIVER -M: Ian Molton <ian@mnementh.co.uk> +M: Ian Molton <ian.molton@codethink.co.uk> L: linux-mmc@vger.kernel.org S: Maintained F: drivers/mmc/host/tmio_mmc* @@ -9109,7 +9145,6 @@ F: drivers/char/tpm/ TRACING M: Steven Rostedt <rostedt@goodmis.org> -M: Frederic Weisbecker <fweisbec@gmail.com> M: Ingo Molnar <mingo@redhat.com> T: git git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git perf/core S: Maintained diff --git a/arch/arc/boot/dts/angel4.dts b/arch/arc/boot/dts/angel4.dts index bcf662d21a57..5bb2fdaca02f 100644 --- a/arch/arc/boot/dts/angel4.dts +++ b/arch/arc/boot/dts/angel4.dts @@ -17,7 +17,7 @@ interrupt-parent = <&intc>; chosen { - bootargs = "console=ttyARC0,115200n8"; + bootargs = "console=ttyARC0,115200n8 earlyprintk=ttyARC0"; }; aliases { diff --git a/arch/arc/include/asm/cache.h b/arch/arc/include/asm/cache.h index 2fd3162ec4df..c1d3d2da1191 100644 --- a/arch/arc/include/asm/cache.h +++ b/arch/arc/include/asm/cache.h @@ -55,4 +55,31 @@ extern void read_decode_cache_bcr(void); #endif /* !__ASSEMBLY__ */ +/* Instruction cache related Auxiliary registers */ +#define ARC_REG_IC_BCR 0x77 /* Build Config reg */ +#define ARC_REG_IC_IVIC 0x10 +#define ARC_REG_IC_CTRL 0x11 +#define ARC_REG_IC_IVIL 0x19 +#if defined(CONFIG_ARC_MMU_V3) || defined (CONFIG_ARC_MMU_V4) +#define ARC_REG_IC_PTAG 0x1E +#endif + +/* Bit val in IC_CTRL */ +#define IC_CTRL_CACHE_DISABLE 0x1 + +/* Data cache related Auxiliary registers */ +#define ARC_REG_DC_BCR 0x72 /* Build Config reg */ +#define ARC_REG_DC_IVDC 0x47 +#define ARC_REG_DC_CTRL 0x48 +#define ARC_REG_DC_IVDL 0x4A +#define ARC_REG_DC_FLSH 0x4B +#define ARC_REG_DC_FLDL 0x4C +#if defined(CONFIG_ARC_MMU_V3) || defined (CONFIG_ARC_MMU_V4) +#define ARC_REG_DC_PTAG 0x5C +#endif + +/* Bit val in DC_CTRL */ +#define DC_CTRL_INV_MODE_FLUSH 0x40 +#define DC_CTRL_FLUSH_STATUS 0x100 + #endif /* _ASM_CACHE_H */ diff --git a/arch/arc/include/asm/irq.h b/arch/arc/include/asm/irq.h index 291a70db68b8..fb4efb648971 100644 --- a/arch/arc/include/asm/irq.h +++ b/arch/arc/include/asm/irq.h @@ -19,8 +19,6 @@ #include <asm-generic/irq.h> extern void arc_init_IRQ(void); -extern int get_hw_config_num_irq(void); - -void arc_local_timer_setup(unsigned int cpu); +void arc_local_timer_setup(void); #endif diff --git a/arch/arc/include/asm/processor.h b/arch/arc/include/asm/processor.h index 15334ab66b56..d99f9b37cd15 100644 --- a/arch/arc/include/asm/processor.h +++ b/arch/arc/include/asm/processor.h @@ -18,7 +18,6 @@ #ifndef __ASSEMBLY__ -#include <asm/arcregs.h> /* for STATUS_E1_MASK et all */ #include <asm/ptrace.h> /* Arch specific stuff which needs to be saved per task. @@ -41,15 +40,13 @@ struct thread_struct { /* Forward declaration, a strange C thing */ struct task_struct; -/* - * Return saved PC of a blocked thread. - */ +/* Return saved PC of a blocked thread */ unsigned long thread_saved_pc(struct task_struct *t); #define task_pt_regs(p) \ ((struct pt_regs *)(THREAD_SIZE + (void *)task_stack_page(p)) - 1) -/* Free all resources held by a thread. */ +/* Free all resources held by a thread */ #define release_thread(thread) do { } while (0) /* Prepare to copy thread state - unlazy all lazy status */ @@ -82,26 +79,8 @@ unsigned long thread_saved_pc(struct task_struct *t); #define KSTK_BLINK(tsk) KSTK_REG(tsk, 4) #define KSTK_FP(tsk) KSTK_REG(tsk, 0) -/* - * Do necessary setup to start up a newly executed thread. - * - * E1,E2 so that Interrupts are enabled in user mode - * L set, so Loop inhibited to begin with - * lp_start and lp_end seeded with bogus non-zero values so to easily catch - * the ARC700 sr to lp_start hardware bug - */ -#define start_thread(_regs, _pc, _usp) \ -do { \ - set_fs(USER_DS); /* reads from user space */ \ - (_regs)->ret = (_pc); \ - /* Interrupts enabled in User Mode */ \ - (_regs)->status32 = STATUS_U_MASK | STATUS_L_MASK \ - | STATUS_E1_MASK | STATUS_E2_MASK; \ - (_regs)->sp = (_usp); \ - /* bogus seed values for debugging */ \ - (_regs)->lp_start = 0x10; \ - (_regs)->lp_end = 0x80; \ -} while (0) +extern void start_thread(struct pt_regs * regs, unsigned long pc, + unsigned long usp); extern unsigned int get_wchan(struct task_struct *p); diff --git a/arch/arc/include/uapi/asm/Kbuild b/arch/arc/include/uapi/asm/Kbuild index 18fefaea73fd..f50d02df78d5 100644 --- a/arch/arc/include/uapi/asm/Kbuild +++ b/arch/arc/include/uapi/asm/Kbuild @@ -2,11 +2,4 @@ include include/uapi/asm-generic/Kbuild.asm header-y += elf.h header-y += page.h -header-y += setup.h -header-y += byteorder.h header-y += cachectl.h -header-y += ptrace.h -header-y += sigcontext.h -header-y += signal.h -header-y += swab.h -header-y += unistd.h diff --git a/arch/arc/kernel/entry.S b/arch/arc/kernel/entry.S index 29b82adbf0b4..83a046a7cd06 100644 --- a/arch/arc/kernel/entry.S +++ b/arch/arc/kernel/entry.S @@ -156,7 +156,7 @@ ARCFP_DATA int1_saved_reg int1_saved_reg: .zero 4 -/* Each Interrupt level needs it's own scratch */ +/* Each Interrupt level needs its own scratch */ #ifdef CONFIG_ARC_COMPACT_IRQ_LEVELS ARCFP_DATA int2_saved_reg @@ -473,7 +473,7 @@ trap_with_param: lr r0, [efa] mov r1, sp - ; Now that we have read EFA, its safe to do "fake" rtie + ; Now that we have read EFA, it is safe to do "fake" rtie ; and get out of CPU exception mode FAKE_RET_FROM_EXCPN r11 @@ -678,9 +678,9 @@ not_exception: brne r9, event_IRQ2, 149f ;------------------------------------------------------------------ - ; if L2 IRQ interrupted a L1 ISR, we'd disbaled preemption earlier - ; so that sched doesnt move to new task, causing L1 to be delayed - ; undeterministically. Now that we've achieved that, lets reset + ; if L2 IRQ interrupted an L1 ISR, we'd disabled preemption earlier + ; so that sched doesn't move to new task, causing L1 to be delayed + ; undeterministically. Now that we've achieved that, let's reset ; things to what they were, before returning from L2 context ;---------------------------------------------------------------- @@ -736,7 +736,7 @@ ENTRY(ret_from_fork) ; put last task in scheduler queue bl @schedule_tail - ; If kernel thread, jump to it's entry-point + ; If kernel thread, jump to its entry-point ld r9, [sp, PT_status32] brne r9, 0, 1f diff --git a/arch/arc/kernel/head.S b/arch/arc/kernel/head.S index 4ad04915dc6b..07a58f2d3077 100644 --- a/arch/arc/kernel/head.S +++ b/arch/arc/kernel/head.S @@ -12,10 +12,42 @@ * to skip certain things during boot on simulator */ +#include <linux/linkage.h> #include <asm/asm-offsets.h> #include <asm/entry.h> -#include <linux/linkage.h> #include <asm/arcregs.h> +#include <asm/cache.h> + +.macro CPU_EARLY_SETUP + + ; Setting up Vectror Table (in case exception happens in early boot + sr @_int_vec_base_lds, [AUX_INTR_VEC_BASE] + + ; Disable I-cache/D-cache if kernel so configured + lr r5, [ARC_REG_IC_BCR] + breq r5, 0, 1f ; I$ doesn't exist + lr r5, [ARC_REG_IC_CTRL] +#ifdef CONFIG_ARC_HAS_ICACHE + bclr r5, r5, 0 ; 0 - Enable, 1 is Disable +#else + bset r5, r5, 0 ; I$ exists, but is not used +#endif + sr r5, [ARC_REG_IC_CTRL] + +1: + lr r5, [ARC_REG_DC_BCR] + breq r5, 0, 1f ; D$ doesn't exist + lr r5, [ARC_REG_DC_CTRL] + bclr r5, r5, 6 ; Invalidate (discard w/o wback) +#ifdef CONFIG_ARC_HAS_DCACHE + bclr r5, r5, 0 ; Enable (+Inv) +#else + bset r5, r5, 0 ; Disable (+Inv) +#endif + sr r5, [ARC_REG_DC_CTRL] + +1: +.endm .cpu A7 @@ -27,7 +59,7 @@ stext: ; Don't clobber r0-r2 yet. It might have bootloader provided info ;------------------------------------------------------------------- - sr @_int_vec_base_lds, [AUX_INTR_VEC_BASE] + CPU_EARLY_SETUP #ifdef CONFIG_SMP ; Ensure Boot (Master) proceeds. Others wait in platform dependent way @@ -90,7 +122,7 @@ stext: first_lines_of_secondary: - sr @_int_vec_base_lds, [AUX_INTR_VEC_BASE] + CPU_EARLY_SETUP ; setup per-cpu idle task as "current" on this CPU ld r0, [@secondary_idle_tsk] diff --git a/arch/arc/kernel/irq.c b/arch/arc/kernel/irq.c index a4b141ee9a6a..7d653c0d0773 100644 --- a/arch/arc/kernel/irq.c +++ b/arch/arc/kernel/irq.c @@ -150,24 +150,6 @@ void arch_do_IRQ(unsigned int irq, struct pt_regs *regs) set_irq_regs(old_regs); } -int get_hw_config_num_irq(void) -{ - uint32_t val = read_aux_reg(ARC_REG_VECBASE_BCR); - - switch (val & 0x03) { - case 0: - return 16; - case 1: - return 32; - case 2: - return 8; - default: - return 0; - } - - return 0; -} - /* * arch_local_irq_enable - Enable interrupts. * diff --git a/arch/arc/kernel/process.c b/arch/arc/kernel/process.c index 07a3a968fe49..fdd89715d2d3 100644 --- a/arch/arc/kernel/process.c +++ b/arch/arc/kernel/process.c @@ -151,6 +151,29 @@ int copy_thread(unsigned long clone_flags, } /* + * Do necessary setup to start up a new user task + */ +void start_thread(struct pt_regs * regs, unsigned long pc, unsigned long usp) +{ + set_fs(USER_DS); /* user space */ + + regs->sp = usp; + regs->ret = pc; + + /* + * [U]ser Mode bit set + * [L] ZOL loop inhibited to begin with - cleared by a LP insn + * Interrupts enabled + */ + regs->status32 = STATUS_U_MASK | STATUS_L_MASK | + STATUS_E1_MASK | STATUS_E2_MASK; + + /* bogus seed values for debugging */ + regs->lp_start = 0x10; + regs->lp_end = 0x80; +} + +/* * Some archs flush debug and FPU info here */ void flush_thread(void) diff --git a/arch/arc/kernel/smp.c b/arch/arc/kernel/smp.c index 40859e5619f9..cf90b6f4d3e0 100644 --- a/arch/arc/kernel/smp.c +++ b/arch/arc/kernel/smp.c @@ -138,7 +138,7 @@ void start_kernel_secondary(void) if (machine_desc->init_smp) machine_desc->init_smp(smp_processor_id()); - arc_local_timer_setup(cpu); + arc_local_timer_setup(); local_irq_enable(); preempt_disable(); diff --git a/arch/arc/kernel/time.c b/arch/arc/kernel/time.c index 71c42521c77f..36c2aa99436f 100644 --- a/arch/arc/kernel/time.c +++ b/arch/arc/kernel/time.c @@ -219,12 +219,13 @@ static struct irqaction arc_timer_irq = { /* * Setup the local event timer for @cpu */ -void arc_local_timer_setup(unsigned int cpu) +void arc_local_timer_setup() { - struct clock_event_device *clk = &per_cpu(arc_clockevent_device, cpu); + struct clock_event_device *evt = this_cpu_ptr(&arc_clockevent_device); + int cpu = smp_processor_id(); - clk->cpumask = cpumask_of(cpu); - clockevents_config_and_register(clk, arc_get_core_freq(), + evt->cpumask = cpumask_of(cpu); + clockevents_config_and_register(evt, arc_get_core_freq(), 0, ARC_TIMER_MAX); /* @@ -261,7 +262,7 @@ void __init time_init(void) clocksource_register_hz(&arc_counter, arc_get_core_freq()); /* sets up the periodic event timer */ - arc_local_timer_setup(smp_processor_id()); + arc_local_timer_setup(); if (machine_desc->init_time) machine_desc->init_time(); diff --git a/arch/arc/mm/cache_arc700.c b/arch/arc/mm/cache_arc700.c index 89edf7961a2f..1f676c4794e0 100644 --- a/arch/arc/mm/cache_arc700.c +++ b/arch/arc/mm/cache_arc700.c @@ -73,33 +73,6 @@ #include <asm/cachectl.h> #include <asm/setup.h> -/* Instruction cache related Auxiliary registers */ -#define ARC_REG_IC_BCR 0x77 /* Build Config reg */ -#define ARC_REG_IC_IVIC 0x10 -#define ARC_REG_IC_CTRL 0x11 -#define ARC_REG_IC_IVIL 0x19 -#if (CONFIG_ARC_MMU_VER > 2) -#define ARC_REG_IC_PTAG 0x1E -#endif - -/* Bit val in IC_CTRL */ -#define IC_CTRL_CACHE_DISABLE 0x1 - -/* Data cache related Auxiliary registers */ -#define ARC_REG_DC_BCR 0x72 /* Build Config reg */ -#define ARC_REG_DC_IVDC 0x47 -#define ARC_REG_DC_CTRL 0x48 -#define ARC_REG_DC_IVDL 0x4A -#define ARC_REG_DC_FLSH 0x4B -#define ARC_REG_DC_FLDL 0x4C -#if (CONFIG_ARC_MMU_VER > 2) -#define ARC_REG_DC_PTAG 0x5C -#endif - -/* Bit val in DC_CTRL */ -#define DC_CTRL_INV_MODE_FLUSH 0x40 -#define DC_CTRL_FLUSH_STATUS 0x100 - char *arc_cache_mumbojumbo(int c, char *buf, int len) { int n = 0; @@ -168,72 +141,43 @@ void read_decode_cache_bcr(void) */ void arc_cache_init(void) { - unsigned int cpu = smp_processor_id(); - struct cpuinfo_arc_cache *ic = &cpuinfo_arc700[cpu].icache; - struct cpuinfo_arc_cache *dc = &cpuinfo_arc700[cpu].dcache; - unsigned int dcache_does_alias, temp; + unsigned int __maybe_unused cpu = smp_processor_id(); + struct cpuinfo_arc_cache __maybe_unused *ic, __maybe_unused *dc; char str[256]; printk(arc_cache_mumbojumbo(0, str, sizeof(str))); - if (!ic->ver) - goto chk_dc; - -#ifdef CONFIG_ARC_HAS_ICACHE - /* 1. Confirm some of I-cache params which Linux assumes */ - if (ic->line_len != L1_CACHE_BYTES) - panic("Cache H/W doesn't match kernel Config"); - - if (ic->ver != CONFIG_ARC_MMU_VER) - panic("Cache ver doesn't match MMU ver\n"); -#endif - - /* Enable/disable I-Cache */ - temp = read_aux_reg(ARC_REG_IC_CTRL); - #ifdef CONFIG_ARC_HAS_ICACHE - temp &= ~IC_CTRL_CACHE_DISABLE; -#else - temp |= IC_CTRL_CACHE_DISABLE; + ic = &cpuinfo_arc700[cpu].icache; + if (ic->ver) { + if (ic->line_len != L1_CACHE_BYTES) + panic("ICache line [%d] != kernel Config [%d]", + ic->line_len, L1_CACHE_BYTES); + + if (ic->ver != CONFIG_ARC_MMU_VER) + panic("Cache ver [%d] doesn't match MMU ver [%d]\n", + ic->ver, CONFIG_ARC_MMU_VER); + } #endif - write_aux_reg(ARC_REG_IC_CTRL, temp); - -chk_dc: - if (!dc->ver) - return; - #ifdef CONFIG_ARC_HAS_DCACHE - if (dc->line_len != L1_CACHE_BYTES) - panic("Cache H/W doesn't match kernel Config"); + dc = &cpuinfo_arc700[cpu].dcache; + if (dc->ver) { + unsigned int dcache_does_alias; - /* check for D-Cache aliasing */ - dcache_does_alias = (dc->sz / dc->assoc) > PAGE_SIZE; + if (dc->line_len != L1_CACHE_BYTES) + panic("DCache line [%d] != kernel Config [%d]", + dc->line_len, L1_CACHE_BYTES); - if (dcache_does_alias && !cache_is_vipt_aliasing()) - panic("Enable CONFIG_ARC_CACHE_VIPT_ALIASING\n"); - else if (!dcache_does_alias && cache_is_vipt_aliasing()) - panic("Don't need CONFIG_ARC_CACHE_VIPT_ALIASING\n"); -#endif - - /* Set the default Invalidate Mode to "simpy discard dirty lines" - * as this is more frequent then flush before invalidate - * Ofcourse we toggle this default behviour when desired - */ - temp = read_aux_reg(ARC_REG_DC_CTRL); - temp &= ~DC_CTRL_INV_MODE_FLUSH; + /* check for D-Cache aliasing */ + dcache_does_alias = (dc->sz / dc->assoc) > PAGE_SIZE; -#ifdef CONFIG_ARC_HAS_DCACHE - /* Enable D-Cache: Clear Bit 0 */ - write_aux_reg(ARC_REG_DC_CTRL, temp & ~IC_CTRL_CACHE_DISABLE); -#else - /* Flush D cache */ - write_aux_reg(ARC_REG_DC_FLSH, 0x1); - /* Disable D cache */ - write_aux_reg(ARC_REG_DC_CTRL, temp | IC_CTRL_CACHE_DISABLE); + if (dcache_does_alias && !cache_is_vipt_aliasing()) + panic("Enable CONFIG_ARC_CACHE_VIPT_ALIASING\n"); + else if (!dcache_does_alias && cache_is_vipt_aliasing()) + panic("Don't need CONFIG_ARC_CACHE_VIPT_ALIASING\n"); + } #endif - - return; } #define OP_INV 0x1 @@ -253,12 +197,16 @@ static inline void __cache_line_loop(unsigned long paddr, unsigned long vaddr, if (cacheop == OP_INV_IC) { aux_cmd = ARC_REG_IC_IVIL; +#if (CONFIG_ARC_MMU_VER > 2) aux_tag = ARC_REG_IC_PTAG; +#endif } else { /* d$ cmd: INV (discard or wback-n-discard) OR FLUSH (wback) */ aux_cmd = cacheop & OP_INV ? ARC_REG_DC_IVDL : ARC_REG_DC_FLDL; +#if (CONFIG_ARC_MMU_VER > 2) aux_tag = ARC_REG_DC_PTAG; +#endif } /* Ensure we properly floor/ceil the non-line aligned/sized requests diff --git a/arch/arc/plat-arcfpga/Kconfig b/arch/arc/plat-arcfpga/Kconfig index 33058aa40e77..e27bb5cc3c1e 100644 --- a/arch/arc/plat-arcfpga/Kconfig +++ b/arch/arc/plat-arcfpga/Kconfig @@ -48,36 +48,4 @@ config ARC_SERIAL_BAUD help Baud rate for the ARC UART -menuconfig ARC_HAS_BVCI_LAT_UNIT - bool "BVCI Bus Latency Unit" - depends on ARC_BOARD_ML509 || ARC_BOARD_ANGEL4 - help - IP to add artificial latency to BVCI Bus Based FPGA builds. - The default latency (even worst case) for FPGA is non-realistic - (~10 SDRAM, ~5 SSRAM). - -config BVCI_LAT_UNITS - hex "Latency Unit(s) Bitmap" - default "0x0" - depends on ARC_HAS_BVCI_LAT_UNIT - help - There are multiple Latency Units corresponding to the many - interfaces of the system bus arbiter (both CPU side as well as - the peripheral side). - To add latency to ALL memory transaction, choose Unit 0, otherwise - for finer grainer - interface wise latency, specify a bitmap (1 bit - per unit) of all units. e.g. 1,2,12 will be 0x1003 - - Unit 0 - System Arb and Mem Controller - Unit 1 - I$ and System Bus - Unit 2 - D$ and System Bus - .. - Unit 12 - IDE Disk controller and System Bus - -config BVCI_LAT_CYCLES - int "Latency Value in cycles" - range 0 63 - default "30" - depends on ARC_HAS_BVCI_LAT_UNIT - endif diff --git a/arch/arc/plat-arcfpga/Makefile b/arch/arc/plat-arcfpga/Makefile index a44e22ebc1b7..4d1bddc34b5b 100644 --- a/arch/arc/plat-arcfpga/Makefile +++ b/arch/arc/plat-arcfpga/Makefile @@ -9,4 +9,4 @@ KBUILD_CFLAGS += -Iarch/arc/plat-arcfpga/include obj-y := platform.o irq.o -obj-$(CONFIG_SMP) += smp.o +obj-$(CONFIG_ISS_SMP_EXTN) += smp.o diff --git a/arch/arc/plat-arcfpga/platform.c b/arch/arc/plat-arcfpga/platform.c index 19b76b61f44b..61c7e5997387 100644 --- a/arch/arc/plat-arcfpga/platform.c +++ b/arch/arc/plat-arcfpga/platform.c @@ -22,59 +22,6 @@ #include <plat/smp.h> #include <plat/irq.h> -/*-----------------------BVCI Latency Unit -----------------------------*/ - -#ifdef CONFIG_ARC_HAS_BVCI_LAT_UNIT - -int lat_cycles = CONFIG_BVCI_LAT_CYCLES; - -/* BVCI Bus Profiler: Latency Unit */ -static void __init setup_bvci_lat_unit(void) -{ -#define MAX_BVCI_UNITS 12 - - unsigned int i; - unsigned int *base = (unsigned int *)BVCI_LAT_UNIT_BASE; - const unsigned long units_req = CONFIG_BVCI_LAT_UNITS; - const unsigned int REG_UNIT = 21; - const unsigned int REG_VAL = 22; - - /* - * There are multiple Latency Units corresponding to the many - * interfaces of the system bus arbiter (both CPU side as well as - * the peripheral side). - * - * Unit 0 - System Arb and Mem Controller - adds latency to all - * memory trasactions - * Unit 1 - I$ and System Bus - * Unit 2 - D$ and System Bus - * .. - * Unit 12 - IDE Disk controller and System Bus - * - * The programmers model requires writing to lat_unit reg first - * and then the latency value (cycles) to lat_value reg - */ - - if (CONFIG_BVCI_LAT_UNITS == 0) { - writel(0, base + REG_UNIT); - writel(lat_cycles, base + REG_VAL); - pr_info("BVCI Latency for all Memory Transactions %d cycles\n", - lat_cycles); - } else { - for_each_set_bit(i, &units_req, MAX_BVCI_UNITS) { - writel(i + 1, base + REG_UNIT); /* loop is 0 based */ - writel(lat_cycles, base + REG_VAL); - pr_info("BVCI Latency for Unit[%d] = %d cycles\n", - (i + 1), lat_cycles); - } - } -} -#else -static void __init setup_bvci_lat_unit(void) -{ -} -#endif - /*----------------------- Platform Devices -----------------------------*/ #if IS_ENABLED(CONFIG_SERIAL_ARC) @@ -132,16 +79,11 @@ static void arc_fpga_serial_init(void) ARRAY_SIZE(fpga_early_devs)); /* - * ARC console driver registers itself as an early platform driver - * of class "earlyprintk". - * Install it here, followed by probe of devices. - * The installation here doesn't require earlyprintk in command line - * To do so however, replace the lines below with - * parse_early_param(); - * early_platform_driver_probe("earlyprintk", 1, 1); - * ^^ + * ARC console driver registers (build time) as an early platform driver + * of class "earlyprintk". However it needs explicit cmdline toggle + * "earlyprintk=ttyARC0" to be successfuly runtime registered. + * Otherwise the early probe below fails to find the driver */ - early_platform_driver_register_all("earlyprintk"); early_platform_driver_probe("earlyprintk", 1, 0); /* @@ -165,11 +107,9 @@ static void __init plat_fpga_early_init(void) { pr_info("[plat-arcfpga]: registering early dev resources\n"); - setup_bvci_lat_unit(); - arc_fpga_serial_init(); -#ifdef CONFIG_SMP +#ifdef CONFIG_ISS_SMP_EXTN iss_model_init_early_smp(); #endif } @@ -211,7 +151,7 @@ MACHINE_START(ANGEL4, "angel4") .init_early = plat_fpga_early_init, .init_machine = plat_fpga_populate_dev, .init_irq = plat_fpga_init_IRQ, -#ifdef CONFIG_SMP +#ifdef CONFIG_ISS_SMP_EXTN .init_smp = iss_model_init_smp, #endif MACHINE_END diff --git a/arch/arc/plat-arcfpga/smp.c b/arch/arc/plat-arcfpga/smp.c index 8a12741f5f7a..92bad9122077 100644 --- a/arch/arc/plat-arcfpga/smp.c +++ b/arch/arc/plat-arcfpga/smp.c @@ -42,6 +42,24 @@ static void iss_model_smp_wakeup_cpu(int cpu, unsigned long pc) } +static inline int get_hw_config_num_irq(void) +{ + uint32_t val = read_aux_reg(ARC_REG_VECBASE_BCR); + + switch (val & 0x03) { + case 0: + return 16; + case 1: + return 32; + case 2: + return 8; + default: + return 0; + } + + return 0; +} + /* * Any SMP specific init any CPU does when it comes up. * Here we setup the CPU to enable Inter-Processor-Interrupts diff --git a/arch/arm/boot/dts/am4372.dtsi b/arch/arm/boot/dts/am4372.dtsi index 794c73e5c4e4..db464d7eaca8 100644 --- a/arch/arm/boot/dts/am4372.dtsi +++ b/arch/arm/boot/dts/am4372.dtsi @@ -857,6 +857,35 @@ ti,hwmods = "hdq1w"; status = "disabled"; }; + + dss: dss@4832a000 { + compatible = "ti,omap3-dss"; + reg = <0x4832a000 0x200>; + status = "disabled"; + ti,hwmods = "dss_core"; + clocks = <&disp_clk>; + clock-names = "fck"; + #address-cells = <1>; + #size-cells = <1>; + ranges; + + dispc@4832a400 { + compatible = "ti,omap3-dispc"; + reg = <0x4832a400 0x400>; + interrupts = <GIC_SPI 127 IRQ_TYPE_LEVEL_HIGH>; + ti,hwmods = "dss_dispc"; + clocks = <&disp_clk>; + clock-names = "fck"; + }; + + rfbi: rfbi@4832a800 { + compatible = "ti,omap3-rfbi"; + reg = <0x4832a800 0x100>; + ti,hwmods = "dss_rfbi"; + clocks = <&disp_clk>; + clock-names = "fck"; + }; + }; }; }; diff --git a/arch/arm/boot/dts/am437x-gp-evm.dts b/arch/arm/boot/dts/am437x-gp-evm.dts index c25d15837ce9..003766c47bbf 100644 --- a/arch/arm/boot/dts/am437x-gp-evm.dts +++ b/arch/arm/boot/dts/am437x-gp-evm.dts @@ -19,6 +19,10 @@ model = "TI AM437x GP EVM"; compatible = "ti,am437x-gp-evm","ti,am4372","ti,am43"; + aliases { + display0 = &lcd0; + }; + vmmcsd_fixed: fixedregulator-sd { compatible = "regulator-fixed"; regulator-name = "vmmcsd_fixed"; @@ -64,6 +68,44 @@ 0x02000069 /* LEFT */ 0x0201006c>; /* DOWN */ }; + + lcd0: display { + compatible = "osddisplays,osd057T0559-34ts", "panel-dpi"; + label = "lcd"; + + pinctrl-names = "default"; + pinctrl-0 = <&lcd_pins>; + + /* + * SelLCDorHDMI, LOW to select HDMI. This is not really the + * panel's enable GPIO, but we don't have HDMI driver support nor + * support to switch between two displays, so using this gpio as + * panel's enable should be safe. + */ + enable-gpios = <&gpio5 8 GPIO_ACTIVE_HIGH>; + + panel-timing { + clock-frequency = <33000000>; + hactive = <800>; + vactive = <480>; + hfront-porch = <210>; + hback-porch = <16>; + hsync-len = <30>; + vback-porch = <10>; + vfront-porch = <22>; + vsync-len = <13>; + hsync-active = <0>; + vsync-active = <0>; + de-active = <1>; + pixelclk-active = <1>; + }; + + port { + lcd_in: endpoint { + remote-endpoint = <&dpi_out>; + }; + }; + }; }; &am43xx_pinmux { @@ -171,6 +213,47 @@ 0x9c (PIN_OUTPUT | MUX_MODE0) /* gpmc_be0n_cle.gpmc_be0n_cle */ >; }; + + dss_pins: dss_pins { + pinctrl-single,pins = < + 0x020 (PIN_OUTPUT_PULLUP | MUX_MODE1) /*gpmc ad 8 -> DSS DATA 23 */ + 0x024 (PIN_OUTPUT_PULLUP | MUX_MODE1) + 0x028 (PIN_OUTPUT_PULLUP | MUX_MODE1) + 0x02c (PIN_OUTPUT_PULLUP | MUX_MODE1) + 0x030 (PIN_OUTPUT_PULLUP | MUX_MODE1) + 0x034 (PIN_OUTPUT_PULLUP | MUX_MODE1) + 0x038 (PIN_OUTPUT_PULLUP | MUX_MODE1) + 0x03c (PIN_OUTPUT_PULLUP | MUX_MODE1) /*gpmc ad 15 -> DSS DATA 16 */ + 0x0a0 (PIN_OUTPUT_PULLUP | MUX_MODE0) /* DSS DATA 0 */ + 0x0a4 (PIN_OUTPUT_PULLUP | MUX_MODE0) + 0x0a8 (PIN_OUTPUT_PULLUP | MUX_MODE0) + 0x0ac (PIN_OUTPUT_PULLUP | MUX_MODE0) + 0x0b0 (PIN_OUTPUT_PULLUP | MUX_MODE0) + 0x0b4 (PIN_OUTPUT_PULLUP | MUX_MODE0) + 0x0b8 (PIN_OUTPUT_PULLUP | MUX_MODE0) + 0x0bc (PIN_OUTPUT_PULLUP | MUX_MODE0) + 0x0c0 (PIN_OUTPUT_PULLUP | MUX_MODE0) + 0x0c4 (PIN_OUTPUT_PULLUP | MUX_MODE0) + 0x0c8 (PIN_OUTPUT_PULLUP | MUX_MODE0) + 0x0cc (PIN_OUTPUT_PULLUP | MUX_MODE0) + 0x0d0 (PIN_OUTPUT_PULLUP | MUX_MODE0) + 0x0d4 (PIN_OUTPUT_PULLUP | MUX_MODE0) + 0x0d8 (PIN_OUTPUT_PULLUP | MUX_MODE0) + 0x0dc (PIN_OUTPUT_PULLUP | MUX_MODE0) /* DSS DATA 15 */ + 0x0e0 (PIN_OUTPUT_PULLUP | MUX_MODE0) /* DSS VSYNC */ + 0x0e4 (PIN_OUTPUT_PULLUP | MUX_MODE0) /* DSS HSYNC */ + 0x0e8 (PIN_OUTPUT_PULLUP | MUX_MODE0) /* DSS PCLK */ + 0x0ec (PIN_OUTPUT_PULLUP | MUX_MODE0) /* DSS AC BIAS EN */ + + >; + }; + + lcd_pins: lcd_pins { + pinctrl-single,pins = < + /* GPIO 5_8 to select LCD / HDMI */ + 0x238 (PIN_OUTPUT_PULLUP | MUX_MODE7) + >; + }; }; &i2c0 { @@ -359,3 +442,17 @@ }; }; }; + +&dss { + status = "ok"; + + pinctrl-names = "default"; + pinctrl-0 = <&dss_pins>; + + port { + dpi_out: endpoint@0 { + remote-endpoint = <&lcd_in>; + data-lines = <24>; + }; + }; +}; diff --git a/arch/arm/boot/dts/am43x-epos-evm.dts b/arch/arm/boot/dts/am43x-epos-evm.dts index ad362c50e32e..19f1f7e87597 100644 --- a/arch/arm/boot/dts/am43x-epos-evm.dts +++ b/arch/arm/boot/dts/am43x-epos-evm.dts @@ -19,6 +19,10 @@ model = "TI AM43x EPOS EVM"; compatible = "ti,am43x-epos-evm","ti,am4372","ti,am43"; + aliases { + display0 = &lcd0; + }; + vmmcsd_fixed: fixedregulator-sd { compatible = "regulator-fixed"; regulator-name = "vmmcsd_fixed"; @@ -27,6 +31,44 @@ enable-active-high; }; + lcd0: display { + compatible = "osddisplays,osd057T0559-34ts", "panel-dpi"; + label = "lcd"; + + pinctrl-names = "default"; + pinctrl-0 = <&lcd_pins>; + + /* + * SelLCDorHDMI, LOW to select HDMI. This is not really the + * panel's enable GPIO, but we don't have HDMI driver support nor + * support to switch between two displays, so using this gpio as + * panel's enable should be safe. + */ + enable-gpios = <&gpio2 1 GPIO_ACTIVE_HIGH>; + + panel-timing { + clock-frequency = <33000000>; + hactive = <800>; + vactive = <480>; + hfront-porch = <210>; + hback-porch = <16>; + hsync-len = <30>; + vback-porch = <10>; + vfront-porch = <22>; + vsync-len = <13>; + hsync-active = <0>; + vsync-active = <0>; + de-active = <1>; + pixelclk-active = <1>; + }; + + port { + lcd_in: endpoint { + remote-endpoint = <&dpi_out>; + }; + }; + }; + am43xx_pinmux: pinmux@44e10800 { cpsw_default: cpsw_default { pinctrl-single,pins = < @@ -161,6 +203,46 @@ 0x234 (PIN_INPUT_PULLUP | MUX_MODE1) /* cam1_wen.hdq_gpio */ >; }; + + dss_pins: dss_pins { + pinctrl-single,pins = < + 0x020 (PIN_OUTPUT_PULLUP | MUX_MODE1) /*gpmc ad 8 -> DSS DATA 23 */ + 0x024 (PIN_OUTPUT_PULLUP | MUX_MODE1) + 0x028 (PIN_OUTPUT_PULLUP | MUX_MODE1) + 0x02C (PIN_OUTPUT_PULLUP | MUX_MODE1) + 0x030 (PIN_OUTPUT_PULLUP | MUX_MODE1) + 0x034 (PIN_OUTPUT_PULLUP | MUX_MODE1) + 0x038 (PIN_OUTPUT_PULLUP | MUX_MODE1) + 0x03C (PIN_OUTPUT_PULLUP | MUX_MODE1) /*gpmc ad 15 -> DSS DATA 16 */ + 0x0A0 (PIN_OUTPUT_PULLUP | MUX_MODE0) /* DSS DATA 0 */ + 0x0A4 (PIN_OUTPUT_PULLUP | MUX_MODE0) + 0x0A8 (PIN_OUTPUT_PULLUP | MUX_MODE0) + 0x0AC (PIN_OUTPUT_PULLUP | MUX_MODE0) + 0x0B0 (PIN_OUTPUT_PULLUP | MUX_MODE0) + 0x0B4 (PIN_OUTPUT_PULLUP | MUX_MODE0) + 0x0B8 (PIN_OUTPUT_PULLUP | MUX_MODE0) + 0x0BC (PIN_OUTPUT_PULLUP | MUX_MODE0) + 0x0C0 (PIN_OUTPUT_PULLUP | MUX_MODE0) + 0x0C4 (PIN_OUTPUT_PULLUP | MUX_MODE0) + 0x0C8 (PIN_OUTPUT_PULLUP | MUX_MODE0) + 0x0CC (PIN_OUTPUT_PULLUP | MUX_MODE0) + 0x0D0 (PIN_OUTPUT_PULLUP | MUX_MODE0) + 0x0D4 (PIN_OUTPUT_PULLUP | MUX_MODE0) + 0x0D8 (PIN_OUTPUT_PULLUP | MUX_MODE0) + 0x0DC (PIN_OUTPUT_PULLUP | MUX_MODE0) /* DSS DATA 15 */ + 0x0E0 (PIN_OUTPUT_PULLUP | MUX_MODE0) /* DSS VSYNC */ + 0x0E4 (PIN_OUTPUT_PULLUP | MUX_MODE0) /* DSS HSYNC */ + 0x0E8 (PIN_OUTPUT_PULLUP | MUX_MODE0) /* DSS PCLK */ + 0x0EC (PIN_OUTPUT_PULLUP | MUX_MODE0) /* DSS AC BIAS EN */ + >; + }; + + lcd_pins: lcd_pins { + pinctrl-single,pins = < + /* GPMC CLK -> GPIO 2_1 to select LCD / HDMI */ + 0x08C (PIN_OUTPUT_PULLUP | MUX_MODE7) + >; + }; }; matrix_keypad: matrix_keypad@0 { @@ -468,3 +550,17 @@ pinctrl-names = "default"; pinctrl-0 = <&hdq_pins>; }; + +&dss { + status = "ok"; + + pinctrl-names = "default"; + pinctrl-0 = <&dss_pins>; + + port { + dpi_out: endpoint@0 { + remote-endpoint = <&lcd_in>; + data-lines = <24>; + }; + }; +}; diff --git a/arch/arm/boot/dts/omap3-evm-37xx.dts b/arch/arm/boot/dts/omap3-evm-37xx.dts index 9cba94bed7ad..a8bd4349c7d2 100644 --- a/arch/arm/boot/dts/omap3-evm-37xx.dts +++ b/arch/arm/boot/dts/omap3-evm-37xx.dts @@ -26,7 +26,44 @@ }; }; +&dss { + pinctrl-names = "default"; + pinctrl-0 = < + &dss_dpi_pins1 + &dss_dpi_pins2 + >; +}; + &omap3_pmx_core { + dss_dpi_pins1: pinmux_dss_dpi_pins2 { + pinctrl-single,pins = < + OMAP3_CORE1_IOPAD(0x20d4, PIN_OUTPUT | MUX_MODE0) /* dss_pclk.dss_pclk */ + OMAP3_CORE1_IOPAD(0x20d6, PIN_OUTPUT | MUX_MODE0) /* dss_hsync.dss_hsync */ + OMAP3_CORE1_IOPAD(0x20d8, PIN_OUTPUT | MUX_MODE0) /* dss_vsync.dss_vsync */ + OMAP3_CORE1_IOPAD(0x20da, PIN_OUTPUT | MUX_MODE0) /* dss_acbias.dss_acbias */ + + OMAP3_CORE1_IOPAD(0x20e8, PIN_OUTPUT | MUX_MODE0) /* dss_data6.dss_data6 */ + OMAP3_CORE1_IOPAD(0x20ea, PIN_OUTPUT | MUX_MODE0) /* dss_data7.dss_data7 */ + OMAP3_CORE1_IOPAD(0x20ec, PIN_OUTPUT | MUX_MODE0) /* dss_data8.dss_data8 */ + OMAP3_CORE1_IOPAD(0x20ee, PIN_OUTPUT | MUX_MODE0) /* dss_data9.dss_data9 */ + OMAP3_CORE1_IOPAD(0x20f0, PIN_OUTPUT | MUX_MODE0) /* dss_data10.dss_data10 */ + OMAP3_CORE1_IOPAD(0x20f2, PIN_OUTPUT | MUX_MODE0) /* dss_data11.dss_data11 */ + OMAP3_CORE1_IOPAD(0x20f4, PIN_OUTPUT | MUX_MODE0) /* dss_data12.dss_data12 */ + OMAP3_CORE1_IOPAD(0x20f6, PIN_OUTPUT | MUX_MODE0) /* dss_data13.dss_data13 */ + OMAP3_CORE1_IOPAD(0x20f8, PIN_OUTPUT | MUX_MODE0) /* dss_data14.dss_data14 */ + OMAP3_CORE1_IOPAD(0x20fa, PIN_OUTPUT | MUX_MODE0) /* dss_data15.dss_data15 */ + OMAP3_CORE1_IOPAD(0x20fc, PIN_OUTPUT | MUX_MODE0) /* dss_data16.dss_data16 */ + OMAP3_CORE1_IOPAD(0x20fe, PIN_OUTPUT | MUX_MODE0) /* dss_data17.dss_data17 */ + + OMAP3_CORE1_IOPAD(0x2100, PIN_OUTPUT | MUX_MODE3) /* dss_data18.dss_data0 */ + OMAP3_CORE1_IOPAD(0x2102, PIN_OUTPUT | MUX_MODE3) /* dss_data19.dss_data1 */ + OMAP3_CORE1_IOPAD(0x2104, PIN_OUTPUT | MUX_MODE3) /* dss_data20.dss_data2 */ + OMAP3_CORE1_IOPAD(0x2106, PIN_OUTPUT | MUX_MODE3) /* dss_data21.dss_data3 */ + OMAP3_CORE1_IOPAD(0x2108, PIN_OUTPUT | MUX_MODE3) /* dss_data22.dss_data4 */ + OMAP3_CORE1_IOPAD(0x210a, PIN_OUTPUT | MUX_MODE3) /* dss_data23.dss_data5 */ + >; + }; + mmc1_pins: pinmux_mmc1_pins { pinctrl-single,pins = < 0x114 (PIN_OUTPUT_PULLUP | MUX_MODE0) /* sdmmc1_clk.sdmmc1_clk */ @@ -75,6 +112,19 @@ }; }; +&omap3_pmx_wkup { + dss_dpi_pins2: pinmux_dss_dpi_pins1 { + pinctrl-single,pins = < + 0x0a (PIN_OUTPUT | MUX_MODE3) /* sys_boot0.dss_data18 */ + 0x0c (PIN_OUTPUT | MUX_MODE3) /* sys_boot1.dss_data19 */ + 0x10 (PIN_OUTPUT | MUX_MODE3) /* sys_boot3.dss_data20 */ + 0x12 (PIN_OUTPUT | MUX_MODE3) /* sys_boot4.dss_data21 */ + 0x14 (PIN_OUTPUT | MUX_MODE3) /* sys_boot5.dss_data22 */ + 0x16 (PIN_OUTPUT | MUX_MODE3) /* sys_boot6.dss_data23 */ + >; + }; +}; + &mmc1 { pinctrl-names = "default"; pinctrl-0 = <&mmc1_pins>; diff --git a/arch/arm/boot/dts/omap3-evm-common.dtsi b/arch/arm/boot/dts/omap3-evm-common.dtsi index 3007e79c9cd6..8ae8f007c8ad 100644 --- a/arch/arm/boot/dts/omap3-evm-common.dtsi +++ b/arch/arm/boot/dts/omap3-evm-common.dtsi @@ -44,6 +44,11 @@ #include "twl4030.dtsi" #include "twl4030_omap3.dtsi" +#include "omap3-panel-sharp-ls037v7dw01.dtsi" + +&backlight0 { + gpios = <&twl_gpio 18 GPIO_ACTIVE_LOW>; +}; &i2c2 { clock-frequency = <400000>; @@ -61,6 +66,27 @@ }; }; +&lcd_3v3 { + gpio = <&gpio5 25 GPIO_ACTIVE_LOW>; /* gpio153 */ + enable-active-low; +}; + +&lcd0 { + enable-gpios = <&gpio5 24 GPIO_ACTIVE_HIGH>; /* gpio152, lcd INI */ + reset-gpios = <&gpio5 27 GPIO_ACTIVE_HIGH>; /* gpio155, lcd RESB */ + mode-gpios = <&gpio5 26 GPIO_ACTIVE_HIGH /* gpio154, lcd MO */ + &gpio1 2 GPIO_ACTIVE_HIGH /* gpio2, lcd LR */ + &gpio1 3 GPIO_ACTIVE_HIGH>; /* gpio3, lcd UD */ +}; + +&mcspi1 { + tsc2046@0 { + interrupt-parent = <&gpio6>; + interrupts = <15 0>; /* gpio175 */ + pendown-gpio = <&gpio6 15 0>; + }; +}; + &mmc1 { vmmc-supply = <&vmmc1>; vmmc_aux-supply = <&vsim>; diff --git a/arch/arm/boot/dts/omap3-gta04.dts b/arch/arm/boot/dts/omap3-gta04.dts index f8ad125fa46f..021311f7964b 100644 --- a/arch/arm/boot/dts/omap3-gta04.dts +++ b/arch/arm/boot/dts/omap3-gta04.dts @@ -44,6 +44,36 @@ ti,mcbsp = <&mcbsp2>; ti,codec = <&twl_audio>; }; + + spi_lcd { + compatible = "spi-gpio"; + #address-cells = <0x1>; + #size-cells = <0x0>; + pinctrl-names = "default"; + pinctrl-0 = <&spi_gpio_pins>; + + gpio-sck = <&gpio1 12 0>; + gpio-miso = <&gpio1 18 0>; + gpio-mosi = <&gpio1 20 0>; + cs-gpios = <&gpio1 19 0>; + num-chipselects = <1>; + + /* lcd panel */ + lcd: td028ttec1@0 { + compatible = "toppoly,td028ttec1"; + reg = <0>; + spi-max-frequency = <100000>; + spi-cpol; + spi-cpha; + + label = "lcd"; + port { + lcd_in: endpoint { + remote-endpoint = <&dpi_out>; + }; + }; + }; + }; }; &omap3_pmx_core { @@ -78,6 +108,47 @@ 0x11e (PIN_INPUT_PULLUP | MUX_MODE0) /* sdmmc1_dat3.sdmmc1_dat3 */ >; }; + + dss_dpi_pins: pinmux_dss_dpi_pins { + pinctrl-single,pins = < + 0x0a4 (PIN_OUTPUT | MUX_MODE0) /* dss_pclk.dss_pclk */ + 0x0a6 (PIN_OUTPUT | MUX_MODE0) /* dss_hsync.dss_hsync */ + 0x0a8 (PIN_OUTPUT | MUX_MODE0) /* dss_vsync.dss_vsync */ + 0x0aa (PIN_OUTPUT | MUX_MODE0) /* dss_acbias.dss_acbias */ + 0x0ac (PIN_OUTPUT | MUX_MODE0) /* dss_data0.dss_data0 */ + 0x0ae (PIN_OUTPUT | MUX_MODE0) /* dss_data1.dss_data1 */ + 0x0b0 (PIN_OUTPUT | MUX_MODE0) /* dss_data2.dss_data2 */ + 0x0b2 (PIN_OUTPUT | MUX_MODE0) /* dss_data3.dss_data3 */ + 0x0b4 (PIN_OUTPUT | MUX_MODE0) /* dss_data4.dss_data4 */ + 0x0b6 (PIN_OUTPUT | MUX_MODE0) /* dss_data5.dss_data5 */ + 0x0b8 (PIN_OUTPUT | MUX_MODE0) /* dss_data6.dss_data6 */ + 0x0ba (PIN_OUTPUT | MUX_MODE0) /* dss_data7.dss_data7 */ + 0x0bc (PIN_OUTPUT | MUX_MODE0) /* dss_data8.dss_data8 */ + 0x0be (PIN_OUTPUT | MUX_MODE0) /* dss_data9.dss_data9 */ + 0x0c0 (PIN_OUTPUT | MUX_MODE0) /* dss_data10.dss_data10 */ + 0x0c2 (PIN_OUTPUT | MUX_MODE0) /* dss_data11.dss_data11 */ + 0x0c4 (PIN_OUTPUT | MUX_MODE0) /* dss_data12.dss_data12 */ + 0x0c6 (PIN_OUTPUT | MUX_MODE0) /* dss_data13.dss_data13 */ + 0x0c8 (PIN_OUTPUT | MUX_MODE0) /* dss_data14.dss_data14 */ + 0x0ca (PIN_OUTPUT | MUX_MODE0) /* dss_data15.dss_data15 */ + 0x0cc (PIN_OUTPUT | MUX_MODE0) /* dss_data16.dss_data16 */ + 0x0ce (PIN_OUTPUT | MUX_MODE0) /* dss_data17.dss_data17 */ + 0x0d0 (PIN_OUTPUT | MUX_MODE0) /* dss_data18.dss_data18 */ + 0x0d2 (PIN_OUTPUT | MUX_MODE0) /* dss_data19.dss_data19 */ + 0x0d4 (PIN_OUTPUT | MUX_MODE0) /* dss_data20.dss_data20 */ + 0x0d6 (PIN_OUTPUT | MUX_MODE0) /* dss_data21.dss_data21 */ + 0x0d8 (PIN_OUTPUT | MUX_MODE0) /* dss_data22.dss_data22 */ + 0x0da (PIN_OUTPUT | MUX_MODE0) /* dss_data23.dss_data23 */ + >; + }; + + spi_gpio_pins: spi_gpio_pinmux { + pinctrl-single,pins = <0x5a8 (PIN_OUTPUT | MUX_MODE4) /* clk */ + 0x5b6 (PIN_OUTPUT | MUX_MODE4) /* cs */ + 0x5b8 (PIN_OUTPUT | MUX_MODE4) /* tx */ + 0x5b4 (PIN_INPUT | MUX_MODE4) /* rx */ + >; + }; }; &i2c1 { @@ -219,3 +290,22 @@ regulator-min-microvolt = <2800000>; regulator-max-microvolt = <3150000>; }; + +/* Needed to power the DPI pins */ +&vpll2 { + regulator-always-on; +}; + +&dss { + pinctrl-names = "default"; + pinctrl-0 = < &dss_dpi_pins >; + + status = "okay"; + + port { + dpi_out: endpoint { + remote-endpoint = <&lcd_in>; + data-lines = <24>; + }; + }; +}; diff --git a/arch/arm/boot/dts/omap3-ldp.dts b/arch/arm/boot/dts/omap3-ldp.dts index 476ff158ddb3..af272c156e21 100644 --- a/arch/arm/boot/dts/omap3-ldp.dts +++ b/arch/arm/boot/dts/omap3-ldp.dts @@ -164,6 +164,11 @@ #include "twl4030.dtsi" #include "twl4030_omap3.dtsi" +#include "omap3-panel-sharp-ls037v7dw01.dtsi" + +&backlight0 { + gpios = <&twl_gpio 7 GPIO_ACTIVE_HIGH>; +}; &i2c2 { clock-frequency = <400000>; @@ -173,6 +178,25 @@ clock-frequency = <400000>; }; +/* tps61130rsa enabled by twl4030 regen */ +&lcd_3v3 { + regulator-always-on; +}; + +&lcd0 { + enable-gpios = <&twl_gpio 15 GPIO_ACTIVE_HIGH>; /* lcd INI */ + reset-gpios = <&gpio2 23 GPIO_ACTIVE_HIGH>; /* gpio55, lcd RESB */ + mode-gpios = <&gpio2 24 GPIO_ACTIVE_HIGH>; /* gpio56, lcd MO */ +}; + +&mcspi1 { + tsc2046@0 { + interrupt-parent = <&gpio2>; + interrupts = <22 0>; /* gpio54 */ + pendown-gpio = <&gpio2 22 0>; + }; +}; + &mmc1 { /* See 35xx errata 2.1.1.128 in SPRZ278F */ compatible = "ti,omap3-pre-es3-hsmmc"; @@ -251,8 +275,3 @@ /* Needed for ads7846 */ regulator-name = "vcc"; }; - -&vpll2 { - /* Needed for DSS */ - regulator-name = "vdds_dsi"; -}; diff --git a/arch/arm/boot/dts/omap3-n900.dts b/arch/arm/boot/dts/omap3-n900.dts index 059a8ff1e6ac..ae8ae3f4f9bf 100644 --- a/arch/arm/boot/dts/omap3-n900.dts +++ b/arch/arm/boot/dts/omap3-n900.dts @@ -651,9 +651,24 @@ * Also... order in the device tree actually matters here. */ tsc2005@0 { - compatible = "tsc2005"; + compatible = "ti,tsc2005"; spi-max-frequency = <6000000>; reg = <0>; + + vio-supply = <&vio>; + + reset-gpios = <&gpio4 8 GPIO_ACTIVE_HIGH>; /* 104 */ + interrupts-extended = <&gpio4 4 IRQ_TYPE_EDGE_RISING>; /* 100 */ + + touchscreen-fuzz-x = <4>; + touchscreen-fuzz-y = <7>; + touchscreen-fuzz-pressure = <2>; + touchscreen-max-x = <4096>; + touchscreen-max-y = <4096>; + touchscreen-max-pressure = <2048>; + + ti,x-plate-ohms = <280>; + ti,esd-recovery-timeout-ms = <8000>; }; acx565akm@2 { diff --git a/arch/arm/boot/dts/omap3-overo-alto35-common.dtsi b/arch/arm/boot/dts/omap3-overo-alto35-common.dtsi index 19d64864a109..7aae8fb82c1f 100644 --- a/arch/arm/boot/dts/omap3-overo-alto35-common.dtsi +++ b/arch/arm/boot/dts/omap3-overo-alto35-common.dtsi @@ -11,6 +11,7 @@ */ #include "omap3-overo-common-peripherals.dtsi" +#include "omap3-overo-common-lcd35.dtsi" #include <dt-bindings/input/input.h> diff --git a/arch/arm/boot/dts/omap3-overo-chestnut43-common.dtsi b/arch/arm/boot/dts/omap3-overo-chestnut43-common.dtsi index 19de6ff79686..17b82f82638a 100644 --- a/arch/arm/boot/dts/omap3-overo-chestnut43-common.dtsi +++ b/arch/arm/boot/dts/omap3-overo-chestnut43-common.dtsi @@ -11,6 +11,7 @@ */ #include "omap3-overo-common-peripherals.dtsi" +#include "omap3-overo-common-lcd43.dtsi" #include <dt-bindings/input/input.h> diff --git a/arch/arm/boot/dts/omap3-overo-common-dvi.dtsi b/arch/arm/boot/dts/omap3-overo-common-dvi.dtsi new file mode 100644 index 000000000000..802f704f67e5 --- /dev/null +++ b/arch/arm/boot/dts/omap3-overo-common-dvi.dtsi @@ -0,0 +1,111 @@ +/* + * Copyright (C) 2014 Florian Vaussard, EPFL Mobots group + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +/* + * DVI output for some Gumstix Overo boards (Tobi and Summit) + */ + +&omap3_pmx_core { + dss_dpi_pins: pinmux_dss_dpi_pins { + pinctrl-single,pins = < + OMAP3_CORE1_IOPAD(0x20d4, PIN_OUTPUT | MUX_MODE0) /* dss_pclk.dss_pclk */ + OMAP3_CORE1_IOPAD(0x20d6, PIN_OUTPUT | MUX_MODE0) /* dss_hsync.dss_hsync */ + OMAP3_CORE1_IOPAD(0x20d8, PIN_OUTPUT | MUX_MODE0) /* dss_vsync.dss_vsync */ + OMAP3_CORE1_IOPAD(0x20da, PIN_OUTPUT | MUX_MODE0) /* dss_acbias.dss_acbias */ + OMAP3_CORE1_IOPAD(0x20dc, PIN_OUTPUT | MUX_MODE0) /* dss_data0.dss_data0 */ + OMAP3_CORE1_IOPAD(0x20de, PIN_OUTPUT | MUX_MODE0) /* dss_data1.dss_data1 */ + OMAP3_CORE1_IOPAD(0x20e0, PIN_OUTPUT | MUX_MODE0) /* dss_data2.dss_data2 */ + OMAP3_CORE1_IOPAD(0x20e2, PIN_OUTPUT | MUX_MODE0) /* dss_data3.dss_data3 */ + OMAP3_CORE1_IOPAD(0x20e4, PIN_OUTPUT | MUX_MODE0) /* dss_data4.dss_data4 */ + OMAP3_CORE1_IOPAD(0x20e6, PIN_OUTPUT | MUX_MODE0) /* dss_data5.dss_data5 */ + OMAP3_CORE1_IOPAD(0x20e8, PIN_OUTPUT | MUX_MODE0) /* dss_data6.dss_data6 */ + OMAP3_CORE1_IOPAD(0x20ea, PIN_OUTPUT | MUX_MODE0) /* dss_data7.dss_data7 */ + OMAP3_CORE1_IOPAD(0x20ec, PIN_OUTPUT | MUX_MODE0) /* dss_data8.dss_data8 */ + OMAP3_CORE1_IOPAD(0x20ee, PIN_OUTPUT | MUX_MODE0) /* dss_data9.dss_data9 */ + OMAP3_CORE1_IOPAD(0x20f0, PIN_OUTPUT | MUX_MODE0) /* dss_data10.dss_data10 */ + OMAP3_CORE1_IOPAD(0x20f2, PIN_OUTPUT | MUX_MODE0) /* dss_data11.dss_data11 */ + OMAP3_CORE1_IOPAD(0x20f4, PIN_OUTPUT | MUX_MODE0) /* dss_data12.dss_data12 */ + OMAP3_CORE1_IOPAD(0x20f6, PIN_OUTPUT | MUX_MODE0) /* dss_data13.dss_data13 */ + OMAP3_CORE1_IOPAD(0x20f8, PIN_OUTPUT | MUX_MODE0) /* dss_data14.dss_data14 */ + OMAP3_CORE1_IOPAD(0x20fa, PIN_OUTPUT | MUX_MODE0) /* dss_data15.dss_data15 */ + OMAP3_CORE1_IOPAD(0x20fc, PIN_OUTPUT | MUX_MODE0) /* dss_data16.dss_data16 */ + OMAP3_CORE1_IOPAD(0x20fe, PIN_OUTPUT | MUX_MODE0) /* dss_data17.dss_data17 */ + OMAP3_CORE1_IOPAD(0x2100, PIN_OUTPUT | MUX_MODE0) /* dss_data18.dss_data18 */ + OMAP3_CORE1_IOPAD(0x2102, PIN_OUTPUT | MUX_MODE0) /* dss_data19.dss_data19 */ + OMAP3_CORE1_IOPAD(0x2104, PIN_OUTPUT | MUX_MODE0) /* dss_data20.dss_data20 */ + OMAP3_CORE1_IOPAD(0x2106, PIN_OUTPUT | MUX_MODE0) /* dss_data21.dss_data21 */ + OMAP3_CORE1_IOPAD(0x2108, PIN_OUTPUT | MUX_MODE0) /* dss_data22.dss_data22 */ + OMAP3_CORE1_IOPAD(0x210a, PIN_OUTPUT | MUX_MODE0) /* dss_data23.dss_data23 */ + >; + }; +}; + +/* Needed to power the DPI pins */ +&vpll2 { + regulator-always-on; +}; + +&dss { + status = "ok"; + + pinctrl-names = "default"; + pinctrl-0 = <&dss_dpi_pins>; + + port { + dpi_out: endpoint { + remote-endpoint = <&tfp410_in>; + data-lines = <24>; + }; + }; +}; + +/ { + aliases { + display0 = &dvi0; + }; + + tfp410: encoder@0 { + compatible = "ti,tfp410"; + + ports { + #address-cells = <1>; + #size-cells = <0>; + + port@0 { + reg = <0>; + + tfp410_in: endpoint@0 { + remote-endpoint = <&dpi_out>; + }; + }; + + port@1 { + reg = <1>; + + tfp410_out: endpoint@0 { + remote-endpoint = <&dvi_connector_in>; + }; + }; + }; + }; + + dvi0: connector@0 { + compatible = "dvi-connector"; + label = "dvi"; + + digital; + ddc-i2c-bus = <&i2c3>; + + port { + dvi_connector_in: endpoint { + remote-endpoint = <&tfp410_out>; + }; + }; + }; +}; + diff --git a/arch/arm/boot/dts/omap3-overo-common-lcd35.dtsi b/arch/arm/boot/dts/omap3-overo-common-lcd35.dtsi new file mode 100644 index 000000000000..233c69e50ae3 --- /dev/null +++ b/arch/arm/boot/dts/omap3-overo-common-lcd35.dtsi @@ -0,0 +1,165 @@ +/* + * Copyright (C) 2014 Florian Vaussard, EPFL Mobots group + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +/* + * 4.3'' LCD panel output for some Gumstix Overo boards (Gallop43, Chestnut43) + */ + +&omap3_pmx_core { + dss_dpi_pins: pinmux_dss_dpi_pins { + pinctrl-single,pins = < + OMAP3_CORE1_IOPAD(0x20d4, PIN_OUTPUT | MUX_MODE0) /* dss_pclk.dss_pclk */ + OMAP3_CORE1_IOPAD(0x20d6, PIN_OUTPUT | MUX_MODE0) /* dss_hsync.dss_hsync */ + OMAP3_CORE1_IOPAD(0x20d8, PIN_OUTPUT | MUX_MODE0) /* dss_vsync.dss_vsync */ + OMAP3_CORE1_IOPAD(0x20da, PIN_OUTPUT | MUX_MODE0) /* dss_acbias.dss_acbias */ + OMAP3_CORE1_IOPAD(0x20dc, PIN_OUTPUT | MUX_MODE0) /* dss_data0.dss_data0 */ + OMAP3_CORE1_IOPAD(0x20de, PIN_OUTPUT | MUX_MODE0) /* dss_data1.dss_data1 */ + OMAP3_CORE1_IOPAD(0x20e0, PIN_OUTPUT | MUX_MODE0) /* dss_data2.dss_data2 */ + OMAP3_CORE1_IOPAD(0x20e2, PIN_OUTPUT | MUX_MODE0) /* dss_data3.dss_data3 */ + OMAP3_CORE1_IOPAD(0x20e4, PIN_OUTPUT | MUX_MODE0) /* dss_data4.dss_data4 */ + OMAP3_CORE1_IOPAD(0x20e6, PIN_OUTPUT | MUX_MODE0) /* dss_data5.dss_data5 */ + OMAP3_CORE1_IOPAD(0x20e8, PIN_OUTPUT | MUX_MODE0) /* dss_data6.dss_data6 */ + OMAP3_CORE1_IOPAD(0x20ea, PIN_OUTPUT | MUX_MODE0) /* dss_data7.dss_data7 */ + OMAP3_CORE1_IOPAD(0x20ec, PIN_OUTPUT | MUX_MODE0) /* dss_data8.dss_data8 */ + OMAP3_CORE1_IOPAD(0x20ee, PIN_OUTPUT | MUX_MODE0) /* dss_data9.dss_data9 */ + OMAP3_CORE1_IOPAD(0x20f0, PIN_OUTPUT | MUX_MODE0) /* dss_data10.dss_data10 */ + OMAP3_CORE1_IOPAD(0x20f2, PIN_OUTPUT | MUX_MODE0) /* dss_data11.dss_data11 */ + OMAP3_CORE1_IOPAD(0x20f4, PIN_OUTPUT | MUX_MODE0) /* dss_data12.dss_data12 */ + OMAP3_CORE1_IOPAD(0x20f6, PIN_OUTPUT | MUX_MODE0) /* dss_data13.dss_data13 */ + OMAP3_CORE1_IOPAD(0x20f8, PIN_OUTPUT | MUX_MODE0) /* dss_data14.dss_data14 */ + OMAP3_CORE1_IOPAD(0x20fa, PIN_OUTPUT | MUX_MODE0) /* dss_data15.dss_data15 */ + OMAP3_CORE1_IOPAD(0x20fc, PIN_OUTPUT | MUX_MODE0) /* dss_data16.dss_data16 */ + OMAP3_CORE1_IOPAD(0x20fe, PIN_OUTPUT | MUX_MODE0) /* dss_data17.dss_data17 */ + OMAP3_CORE1_IOPAD(0x2100, PIN_OUTPUT | MUX_MODE0) /* dss_data18.dss_data18 */ + OMAP3_CORE1_IOPAD(0x2102, PIN_OUTPUT | MUX_MODE0) /* dss_data19.dss_data19 */ + OMAP3_CORE1_IOPAD(0x2104, PIN_OUTPUT | MUX_MODE0) /* dss_data20.dss_data20 */ + OMAP3_CORE1_IOPAD(0x2106, PIN_OUTPUT | MUX_MODE0) /* dss_data21.dss_data21 */ + OMAP3_CORE1_IOPAD(0x2108, PIN_OUTPUT | MUX_MODE0) /* dss_data22.dss_data22 */ + OMAP3_CORE1_IOPAD(0x210a, PIN_OUTPUT | MUX_MODE0) /* dss_data23.dss_data23 */ + >; + }; + + lb035_pins: pinmux_lb035_pins { + pinctrl-single,pins = < + OMAP3_CORE1_IOPAD(0x2174, PIN_OUTPUT | MUX_MODE4) /* uart2_cts.gpio_144 */ + >; + }; + + backlight_pins: pinmux_backlight_pins { + pinctrl-single,pins = < + OMAP3_CORE1_IOPAD(0x2176, PIN_OUTPUT | MUX_MODE4) /* uart2_rts.gpio_145 */ + >; + }; + + mcspi1_pins: pinmux_mcspi1_pins { + pinctrl-single,pins = < + OMAP3_CORE1_IOPAD(0x21c8, PIN_INPUT | MUX_MODE0) /* mcspi1_clk.mcspi1_clk */ + OMAP3_CORE1_IOPAD(0x21ca, PIN_INPUT | MUX_MODE0) /* mcspi1_simo.mcspi1_simo */ + OMAP3_CORE1_IOPAD(0x21cc, PIN_INPUT | MUX_MODE0) /* mcspi1_somi.mcspi1_somi */ + OMAP3_CORE1_IOPAD(0x21ce, PIN_INPUT | MUX_MODE0) /* mcspi1_cs0.mcspi1_cs0 */ + >; + }; + + ads7846_pins: pinmux_ads7846_pins { + pinctrl-single,pins = < + OMAP3_CORE1_IOPAD(0x2138, PIN_INPUT_PULLDOWN | MUX_MODE4) /* csi2_dx1.gpio_114 */ + >; + }; +}; + +/* Needed to power the DPI pins */ +&vpll2 { + regulator-always-on; +}; + +&dss { + status = "ok"; + + pinctrl-names = "default"; + pinctrl-0 = <&dss_dpi_pins>; + + port { + dpi_out: endpoint { + remote-endpoint = <&lcd_in>; + data-lines = <24>; + }; + }; +}; + +/ { + aliases { + display0 = &lcd0; + }; + + ads7846reg: ads7846-reg { + compatible = "regulator-fixed"; + regulator-name = "ads7846-reg"; + regulator-min-microvolt = <3300000>; + regulator-max-microvolt = <3300000>; + }; + + backlight { + compatible = "gpio-backlight"; + + pinctrl-names = "default"; + pinctrl-0 = <&backlight_pins>; + gpios = <&gpio5 17 GPIO_ACTIVE_HIGH>; /* gpio_145 */ + + default-on; + }; +}; + +&mcspi1 { + pinctrl-names = "default"; + pinctrl-0 = <&mcspi1_pins>; + + lcd0: display@0 { + compatible = "lgphilips,lb035q02"; + label = "lcd"; + + reg = <1>; /* CS1 */ + spi-max-frequency = <10000000>; + spi-cpol; + spi-cpha; + + pinctrl-names = "default"; + pinctrl-0 = <&lb035_pins>; + enable-gpios = <&gpio5 16 GPIO_ACTIVE_HIGH>; /* gpio_144 */ + + port { + lcd_in: endpoint { + remote-endpoint = <&dpi_out>; + }; + }; + }; + + /* touch controller */ + ads7846@0 { + pinctrl-names = "default"; + pinctrl-0 = <&ads7846_pins>; + + compatible = "ti,ads7846"; + vcc-supply = <&ads7846reg>; + + reg = <0>; /* CS0 */ + spi-max-frequency = <1500000>; + + interrupt-parent = <&gpio4>; + interrupts = <18 0>; /* gpio_114 */ + pendown-gpio = <&gpio4 18 0>; + + ti,x-min = /bits/ 16 <0x0>; + ti,x-max = /bits/ 16 <0x0fff>; + ti,y-min = /bits/ 16 <0x0>; + ti,y-max = /bits/ 16 <0x0fff>; + ti,x-plate-ohms = /bits/ 16 <180>; + ti,pressure-max = /bits/ 16 <255>; + + linux,wakeup; + }; +}; diff --git a/arch/arm/boot/dts/omap3-overo-common-lcd43.dtsi b/arch/arm/boot/dts/omap3-overo-common-lcd43.dtsi new file mode 100644 index 000000000000..f5395b7da912 --- /dev/null +++ b/arch/arm/boot/dts/omap3-overo-common-lcd43.dtsi @@ -0,0 +1,178 @@ +/* + * Copyright (C) 2014 Florian Vaussard, EPFL Mobots group + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +/* + * 4.3'' LCD panel output for some Gumstix Overo boards (Gallop43, Chestnut43) + */ + +&omap3_pmx_core { + dss_dpi_pins: pinmux_dss_dpi_pins { + pinctrl-single,pins = < + OMAP3_CORE1_IOPAD(0x20d4, PIN_OUTPUT | MUX_MODE0) /* dss_pclk.dss_pclk */ + OMAP3_CORE1_IOPAD(0x20d6, PIN_OUTPUT | MUX_MODE0) /* dss_hsync.dss_hsync */ + OMAP3_CORE1_IOPAD(0x20d8, PIN_OUTPUT | MUX_MODE0) /* dss_vsync.dss_vsync */ + OMAP3_CORE1_IOPAD(0x20da, PIN_OUTPUT | MUX_MODE0) /* dss_acbias.dss_acbias */ + OMAP3_CORE1_IOPAD(0x20dc, PIN_OUTPUT | MUX_MODE0) /* dss_data0.dss_data0 */ + OMAP3_CORE1_IOPAD(0x20de, PIN_OUTPUT | MUX_MODE0) /* dss_data1.dss_data1 */ + OMAP3_CORE1_IOPAD(0x20e0, PIN_OUTPUT | MUX_MODE0) /* dss_data2.dss_data2 */ + OMAP3_CORE1_IOPAD(0x20e2, PIN_OUTPUT | MUX_MODE0) /* dss_data3.dss_data3 */ + OMAP3_CORE1_IOPAD(0x20e4, PIN_OUTPUT | MUX_MODE0) /* dss_data4.dss_data4 */ + OMAP3_CORE1_IOPAD(0x20e6, PIN_OUTPUT | MUX_MODE0) /* dss_data5.dss_data5 */ + OMAP3_CORE1_IOPAD(0x20e8, PIN_OUTPUT | MUX_MODE0) /* dss_data6.dss_data6 */ + OMAP3_CORE1_IOPAD(0x20ea, PIN_OUTPUT | MUX_MODE0) /* dss_data7.dss_data7 */ + OMAP3_CORE1_IOPAD(0x20ec, PIN_OUTPUT | MUX_MODE0) /* dss_data8.dss_data8 */ + OMAP3_CORE1_IOPAD(0x20ee, PIN_OUTPUT | MUX_MODE0) /* dss_data9.dss_data9 */ + OMAP3_CORE1_IOPAD(0x20f0, PIN_OUTPUT | MUX_MODE0) /* dss_data10.dss_data10 */ + OMAP3_CORE1_IOPAD(0x20f2, PIN_OUTPUT | MUX_MODE0) /* dss_data11.dss_data11 */ + OMAP3_CORE1_IOPAD(0x20f4, PIN_OUTPUT | MUX_MODE0) /* dss_data12.dss_data12 */ + OMAP3_CORE1_IOPAD(0x20f6, PIN_OUTPUT | MUX_MODE0) /* dss_data13.dss_data13 */ + OMAP3_CORE1_IOPAD(0x20f8, PIN_OUTPUT | MUX_MODE0) /* dss_data14.dss_data14 */ + OMAP3_CORE1_IOPAD(0x20fa, PIN_OUTPUT | MUX_MODE0) /* dss_data15.dss_data15 */ + OMAP3_CORE1_IOPAD(0x20fc, PIN_OUTPUT | MUX_MODE0) /* dss_data16.dss_data16 */ + OMAP3_CORE1_IOPAD(0x20fe, PIN_OUTPUT | MUX_MODE0) /* dss_data17.dss_data17 */ + OMAP3_CORE1_IOPAD(0x2100, PIN_OUTPUT | MUX_MODE0) /* dss_data18.dss_data18 */ + OMAP3_CORE1_IOPAD(0x2102, PIN_OUTPUT | MUX_MODE0) /* dss_data19.dss_data19 */ + OMAP3_CORE1_IOPAD(0x2104, PIN_OUTPUT | MUX_MODE0) /* dss_data20.dss_data20 */ + OMAP3_CORE1_IOPAD(0x2106, PIN_OUTPUT | MUX_MODE0) /* dss_data21.dss_data21 */ + OMAP3_CORE1_IOPAD(0x2108, PIN_OUTPUT | MUX_MODE0) /* dss_data22.dss_data22 */ + OMAP3_CORE1_IOPAD(0x210a, PIN_OUTPUT | MUX_MODE0) /* dss_data23.dss_data23 */ + >; + }; + + lte430_pins: pinmux_lte430_pins { + pinctrl-single,pins = < + OMAP3_CORE1_IOPAD(0x2174, PIN_OUTPUT | MUX_MODE4) /* uart2_cts.gpio_144 */ + >; + }; + + backlight_pins: pinmux_backlight_pins { + pinctrl-single,pins = < + OMAP3_CORE1_IOPAD(0x2176, PIN_OUTPUT | MUX_MODE4) /* uart2_rts.gpio_145 */ + >; + }; + + mcspi1_pins: pinmux_mcspi1_pins { + pinctrl-single,pins = < + OMAP3_CORE1_IOPAD(0x21c8, PIN_INPUT | MUX_MODE0) /* mcspi1_clk.mcspi1_clk */ + OMAP3_CORE1_IOPAD(0x21ca, PIN_INPUT | MUX_MODE0) /* mcspi1_simo.mcspi1_simo */ + OMAP3_CORE1_IOPAD(0x21cc, PIN_INPUT | MUX_MODE0) /* mcspi1_somi.mcspi1_somi */ + OMAP3_CORE1_IOPAD(0x21ce, PIN_INPUT | MUX_MODE0) /* mcspi1_cs0.mcspi1_cs0 */ + >; + }; + + ads7846_pins: pinmux_ads7846_pins { + pinctrl-single,pins = < + OMAP3_CORE1_IOPAD(0x2138, PIN_INPUT_PULLDOWN | MUX_MODE4) /* csi2_dx1.gpio_114 */ + >; + }; +}; + +/* Needed to power the DPI pins */ +&vpll2 { + regulator-always-on; +}; + +&dss { + status = "ok"; + + pinctrl-names = "default"; + pinctrl-0 = <&dss_dpi_pins>; + + port { + dpi_out: endpoint { + remote-endpoint = <&lcd_in>; + data-lines = <24>; + }; + }; +}; + +/ { + aliases { + display0 = &lcd0; + }; + + lcd0: display@0 { + compatible = "samsung,lte430wq-f0c", "panel-dpi"; + label = "lcd"; + + pinctrl-names = "default"; + pinctrl-0 = <<e430_pins>; + enable-gpios = <&gpio5 16 GPIO_ACTIVE_HIGH>; /* gpio_144 */ + + port { + lcd_in: endpoint { + remote-endpoint = <&dpi_out>; + }; + }; + + panel-timing { + clock-frequency = <9200000>; + hactive = <480>; + vactive = <272>; + hfront-porch = <8>; + hback-porch = <4>; + hsync-len = <41>; + vback-porch = <2>; + vfront-porch = <4>; + vsync-len = <10>; + + hsync-active = <0>; + vsync-active = <0>; + de-active = <1>; + pixelclk-active = <1>; + }; + }; + + ads7846reg: ads7846-reg { + compatible = "regulator-fixed"; + regulator-name = "ads7846-reg"; + regulator-min-microvolt = <3300000>; + regulator-max-microvolt = <3300000>; + }; + + backlight { + compatible = "gpio-backlight"; + + pinctrl-names = "default"; + pinctrl-0 = <&backlight_pins>; + gpios = <&gpio5 17 GPIO_ACTIVE_HIGH>; /* gpio_145 */ + + default-on; + }; +}; + +&mcspi1 { + pinctrl-names = "default"; + pinctrl-0 = <&mcspi1_pins>; + + /* touch controller */ + ads7846@0 { + pinctrl-names = "default"; + pinctrl-0 = <&ads7846_pins>; + + compatible = "ti,ads7846"; + vcc-supply = <&ads7846reg>; + + reg = <0>; /* CS0 */ + spi-max-frequency = <1500000>; + + interrupt-parent = <&gpio4>; + interrupts = <18 0>; /* gpio_114 */ + pendown-gpio = <&gpio4 18 0>; + + ti,x-min = /bits/ 16 <0x0>; + ti,x-max = /bits/ 16 <0x0fff>; + ti,y-min = /bits/ 16 <0x0>; + ti,y-max = /bits/ 16 <0x0fff>; + ti,x-plate-ohms = /bits/ 16 <180>; + ti,pressure-max = /bits/ 16 <255>; + + linux,wakeup; + }; +}; + diff --git a/arch/arm/boot/dts/omap3-overo-gallop43-common.dtsi b/arch/arm/boot/dts/omap3-overo-gallop43-common.dtsi index 5e848c26986b..49d2254a99b0 100644 --- a/arch/arm/boot/dts/omap3-overo-gallop43-common.dtsi +++ b/arch/arm/boot/dts/omap3-overo-gallop43-common.dtsi @@ -11,6 +11,7 @@ */ #include "omap3-overo-common-peripherals.dtsi" +#include "omap3-overo-common-lcd43.dtsi" #include <dt-bindings/input/input.h> diff --git a/arch/arm/boot/dts/omap3-overo-palo43-common.dtsi b/arch/arm/boot/dts/omap3-overo-palo43-common.dtsi index abea232825b9..087aedf5b902 100644 --- a/arch/arm/boot/dts/omap3-overo-palo43-common.dtsi +++ b/arch/arm/boot/dts/omap3-overo-palo43-common.dtsi @@ -11,6 +11,7 @@ */ #include "omap3-overo-common-peripherals.dtsi" +#include "omap3-overo-common-lcd43.dtsi" #include <dt-bindings/input/input.h> diff --git a/arch/arm/boot/dts/omap3-overo-summit-common.dtsi b/arch/arm/boot/dts/omap3-overo-summit-common.dtsi index 999d1cd4a09f..0ac97ba98549 100644 --- a/arch/arm/boot/dts/omap3-overo-summit-common.dtsi +++ b/arch/arm/boot/dts/omap3-overo-summit-common.dtsi @@ -11,6 +11,7 @@ */ #include "omap3-overo-common-peripherals.dtsi" +#include "omap3-overo-common-dvi.dtsi" / { leds { diff --git a/arch/arm/boot/dts/omap3-overo-tobi-common.dtsi b/arch/arm/boot/dts/omap3-overo-tobi-common.dtsi index 13df50b39442..9e24b6a1d07b 100644 --- a/arch/arm/boot/dts/omap3-overo-tobi-common.dtsi +++ b/arch/arm/boot/dts/omap3-overo-tobi-common.dtsi @@ -11,6 +11,7 @@ */ #include "omap3-overo-common-peripherals.dtsi" +#include "omap3-overo-common-dvi.dtsi" / { leds { diff --git a/arch/arm/boot/dts/omap3-panel-sharp-ls037v7dw01.dtsi b/arch/arm/boot/dts/omap3-panel-sharp-ls037v7dw01.dtsi new file mode 100644 index 000000000000..f4b1a61853e3 --- /dev/null +++ b/arch/arm/boot/dts/omap3-panel-sharp-ls037v7dw01.dtsi @@ -0,0 +1,71 @@ +/* + * Common file for omap dpi panels with QVGA and reset pins + * + * Note that the board specifc DTS file needs to specify + * at minimum the GPIO enable-gpios for display, and + * gpios for gpio-backlight. + */ + +/ { + aliases { + display0 = &lcd0; + }; + + backlight0: backlight { + compatible = "gpio-backlight"; + default-on; + }; + + /* 3.3V GPIO controlled regulator for LCD_ENVDD */ + lcd_3v3: regulator-lcd-3v3 { + compatible = "regulator-fixed"; + regulator-name = "lcd_3v3"; + regulator-min-microvolt = <3300000>; + regulator-max-microvolt = <3300000>; + startup-delay-us = <70000>; + }; + + lcd0: display { + compatible = "sharp,ls037v7dw01"; + label = "lcd"; + power-supply = <&lcd_3v3>; + + port { + lcd_in: endpoint { + remote-endpoint = <&dpi_out>; + }; + }; + }; +}; + +/* Needed to power the DPI pins */ +&vpll2 { + regulator-always-on; +}; + +&dss { + status = "ok"; + port { + dpi_out: endpoint { + remote-endpoint = <&lcd_in>; + data-lines = <18>; + }; + }; +}; + +&mcspi1 { + tsc2046@0 { + reg = <0>; /* CS0 */ + compatible = "ti,tsc2046"; + spi-max-frequency = <1000000>; + vcc-supply = <&lcd_3v3>; + ti,x-min = /bits/ 16 <0>; + ti,x-max = /bits/ 16 <8000>; + ti,y-min = /bits/ 16 <0>; + ti,y-max = /bits/ 16 <4800>; + ti,x-plate-ohms = /bits/ 16 <40>; + ti,pressure-max = /bits/ 16 <255>; + ti,swap-xy; + linux,wakeup; + }; +}; diff --git a/arch/arm/boot/dts/omap4-duovero-parlor.dts b/arch/arm/boot/dts/omap4-duovero-parlor.dts index cd53a64d8f2e..6dc84d9f9b4c 100644 --- a/arch/arm/boot/dts/omap4-duovero-parlor.dts +++ b/arch/arm/boot/dts/omap4-duovero-parlor.dts @@ -15,6 +15,10 @@ model = "OMAP4430 Gumstix Duovero on Parlor"; compatible = "gumstix,omap4-duovero-parlor", "gumstix,omap4-duovero", "ti,omap4430", "ti,omap4"; + aliases { + display0 = &hdmi0; + }; + leds { compatible = "gpio-leds"; led0 { @@ -35,6 +39,21 @@ gpio-key,wakeup; }; }; + + hdmi0: connector@0 { + compatible = "hdmi-connector"; + label = "hdmi"; + + type = "d"; + + hpd-gpios = <&gpio2 31 GPIO_ACTIVE_HIGH>; /* gpio_63 */ + + port { + hdmi_connector_in: endpoint { + remote-endpoint = <&hdmi_out>; + }; + }; + }; }; &omap4_pmx_core { @@ -77,6 +96,15 @@ OMAP4_IOPAD(0x070, PIN_INPUT_PULLUP | MUX_MODE3) /* gpmc_a24.gpio_48: amdix enabled */ >; }; + + dss_hdmi_pins: pinmux_dss_hdmi_pins { + pinctrl-single,pins = < + OMAP4_IOPAD(0x098, PIN_INPUT | MUX_MODE3) /* hdmi_hpd.gpio_63 */ + OMAP4_IOPAD(0x09a, PIN_INPUT_PULLUP | MUX_MODE0) /* hdmi_cec.hdmi_cec */ + OMAP4_IOPAD(0x09c, PIN_INPUT_PULLUP | MUX_MODE0) /* hdmi_ddc_scl.hdmi_ddc_scl */ + OMAP4_IOPAD(0x09e, PIN_INPUT_PULLUP | MUX_MODE0) /* hdmi_ddc_sda.hdmi_ddc_sda */ + >; + }; }; &i2c2 { @@ -143,4 +171,20 @@ }; }; +&dss { + status = "ok"; +}; + +&hdmi { + status = "ok"; + + pinctrl-names = "default"; + pinctrl-0 = <&dss_hdmi_pins>; + + port { + hdmi_out: endpoint { + remote-endpoint = <&hdmi_connector_in>; + }; + }; +}; diff --git a/arch/arm/boot/dts/omap4.dtsi b/arch/arm/boot/dts/omap4.dtsi index 43a587e097d4..7e26d222bfe3 100644 --- a/arch/arm/boot/dts/omap4.dtsi +++ b/arch/arm/boot/dts/omap4.dtsi @@ -922,6 +922,8 @@ ti,hwmods = "dss_hdmi"; clocks = <&dss_48mhz_clk>, <&dss_sys_clk>; clock-names = "fck", "sys_clk"; + dmas = <&sdma 76>; + dma-names = "audio_tx"; }; }; }; diff --git a/arch/arm/boot/dts/omap5-uevm.dts b/arch/arm/boot/dts/omap5-uevm.dts index 3b99ec25b748..1e1b05768cec 100644 --- a/arch/arm/boot/dts/omap5-uevm.dts +++ b/arch/arm/boot/dts/omap5-uevm.dts @@ -20,6 +20,10 @@ reg = <0x80000000 0x7F000000>; /* 2032 MB */ }; + aliases { + display0 = &hdmi0; + }; + vmmcsd_fixed: fixedregulator-mmcsd { compatible = "regulator-fixed"; regulator-name = "vmmcsd_fixed"; @@ -51,6 +55,51 @@ default-state = "off"; }; }; + + tpd12s015: encoder@0 { + compatible = "ti,tpd12s015"; + + pinctrl-names = "default"; + pinctrl-0 = <&tpd12s015_pins>; + + gpios = <&gpio9 0 GPIO_ACTIVE_HIGH>, /* TCA6424A P01, CT CP HPD */ + <&gpio9 1 GPIO_ACTIVE_HIGH>, /* TCA6424A P00, LS OE */ + <&gpio7 1 GPIO_ACTIVE_HIGH>; /* GPIO 193, HPD */ + + ports { + #address-cells = <1>; + #size-cells = <0>; + + port@0 { + reg = <0>; + + tpd12s015_in: endpoint@0 { + remote-endpoint = <&hdmi_out>; + }; + }; + + port@1 { + reg = <1>; + + tpd12s015_out: endpoint@0 { + remote-endpoint = <&hdmi_connector_in>; + }; + }; + }; + }; + + hdmi0: connector@0 { + compatible = "hdmi-connector"; + label = "hdmi"; + + type = "b"; + + port { + hdmi_connector_in: endpoint { + remote-endpoint = <&tpd12s015_out>; + }; + }; + }; }; &omap5_pmx_core { @@ -183,6 +232,19 @@ >; }; + dss_hdmi_pins: pinmux_dss_hdmi_pins { + pinctrl-single,pins = < + 0x0fc (PIN_INPUT_PULLUP | MUX_MODE0) /* hdmi_cec.hdmi_cec */ + 0x100 (PIN_INPUT | MUX_MODE0) /* hdmi_ddc_scl.hdmi_ddc_scl */ + 0x102 (PIN_INPUT | MUX_MODE0) /* hdmi_ddc_sda.hdmi_ddc_sda */ + >; + }; + + tpd12s015_pins: pinmux_tpd12s015_pins { + pinctrl-single,pins = < + 0x0fe (PIN_INPUT_PULLDOWN | MUX_MODE6) /* hdmi_hpd.gpio7_193 */ + >; + }; }; &omap5_pmx_wkup { @@ -434,6 +496,13 @@ pinctrl-0 = <&i2c5_pins>; clock-frequency = <400000>; + + gpio9: gpio@22 { + compatible = "ti,tca6424"; + reg = <0x22>; + gpio-controller; + #gpio-cells = <2>; + }; }; &mcbsp3 { @@ -491,3 +560,21 @@ &cpu0 { cpu0-supply = <&smps123_reg>; }; + +&dss { + status = "ok"; +}; + +&hdmi { + status = "ok"; + vdda-supply = <&ldo4_reg>; + + pinctrl-names = "default"; + pinctrl-0 = <&dss_hdmi_pins>; + + port { + hdmi_out: endpoint { + remote-endpoint = <&tpd12s015_in>; + }; + }; +}; diff --git a/arch/arm/boot/dts/omap5.dtsi b/arch/arm/boot/dts/omap5.dtsi index e58be57984ab..3bfda16c8b52 100644 --- a/arch/arm/boot/dts/omap5.dtsi +++ b/arch/arm/boot/dts/omap5.dtsi @@ -924,6 +924,68 @@ ti,hwmods = "sata"; }; + dss: dss@58000000 { + compatible = "ti,omap5-dss"; + reg = <0x58000000 0x80>; + status = "disabled"; + ti,hwmods = "dss_core"; + clocks = <&dss_dss_clk>; + clock-names = "fck"; + #address-cells = <1>; + #size-cells = <1>; + ranges; + + dispc@58001000 { + compatible = "ti,omap5-dispc"; + reg = <0x58001000 0x1000>; + interrupts = <GIC_SPI 25 IRQ_TYPE_LEVEL_HIGH>; + ti,hwmods = "dss_dispc"; + clocks = <&dss_dss_clk>; + clock-names = "fck"; + }; + + dsi1: encoder@58004000 { + compatible = "ti,omap5-dsi"; + reg = <0x58004000 0x200>, + <0x58004200 0x40>, + <0x58004300 0x40>; + reg-names = "proto", "phy", "pll"; + interrupts = <GIC_SPI 53 IRQ_TYPE_LEVEL_HIGH>; + status = "disabled"; + ti,hwmods = "dss_dsi1"; + clocks = <&dss_dss_clk>, <&dss_sys_clk>; + clock-names = "fck", "sys_clk"; + }; + + dsi2: encoder@58005000 { + compatible = "ti,omap5-dsi"; + reg = <0x58009000 0x200>, + <0x58009200 0x40>, + <0x58009300 0x40>; + reg-names = "proto", "phy", "pll"; + interrupts = <GIC_SPI 55 IRQ_TYPE_LEVEL_HIGH>; + status = "disabled"; + ti,hwmods = "dss_dsi2"; + clocks = <&dss_dss_clk>, <&dss_sys_clk>; + clock-names = "fck", "sys_clk"; + }; + + hdmi: encoder@58060000 { + compatible = "ti,omap5-hdmi"; + reg = <0x58040000 0x200>, + <0x58040200 0x80>, + <0x58040300 0x80>, + <0x58060000 0x19000>; + reg-names = "wp", "pll", "phy", "core"; + interrupts = <GIC_SPI 101 IRQ_TYPE_LEVEL_HIGH>; + status = "disabled"; + ti,hwmods = "dss_hdmi"; + clocks = <&dss_48mhz_clk>, <&dss_sys_clk>; + clock-names = "fck", "sys_clk"; + dmas = <&sdma 76>; + dma-names = "audio_tx"; + }; + }; }; }; diff --git a/arch/arm/mach-omap2/devices.c b/arch/arm/mach-omap2/devices.c index e58609b312c7..592ba0a0ecf3 100644 --- a/arch/arm/mach-omap2/devices.c +++ b/arch/arm/mach-omap2/devices.c @@ -18,7 +18,6 @@ #include <linux/slab.h> #include <linux/of.h> #include <linux/pinctrl/machine.h> -#include <linux/platform_data/omap4-keypad.h> #include <linux/platform_data/mailbox-omap.h> #include <asm/mach-types.h> @@ -29,7 +28,6 @@ #include "iomap.h" #include "omap_hwmod.h" #include "omap_device.h" -#include "omap4-keypad.h" #include "soc.h" #include "common.h" @@ -255,37 +253,6 @@ static inline void omap_init_camera(void) #endif } -int __init omap4_keyboard_init(struct omap4_keypad_platform_data - *sdp4430_keypad_data, struct omap_board_data *bdata) -{ - struct platform_device *pdev; - struct omap_hwmod *oh; - struct omap4_keypad_platform_data *keypad_data; - unsigned int id = -1; - char *oh_name = "kbd"; - char *name = "omap4-keypad"; - - oh = omap_hwmod_lookup(oh_name); - if (!oh) { - pr_err("Could not look up %s\n", oh_name); - return -ENODEV; - } - - keypad_data = sdp4430_keypad_data; - - pdev = omap_device_build(name, id, oh, keypad_data, - sizeof(struct omap4_keypad_platform_data)); - - if (IS_ERR(pdev)) { - WARN(1, "Can't build omap_device for %s:%s.\n", - name, oh->name); - return PTR_ERR(pdev); - } - oh->mux = omap_hwmod_mux_init(bdata->pads, bdata->pads_cnt); - - return 0; -} - #if defined(CONFIG_OMAP2PLUS_MBOX) || defined(CONFIG_OMAP2PLUS_MBOX_MODULE) static inline void __init omap_init_mbox(void) { diff --git a/arch/arm/mach-omap2/omap4-keypad.h b/arch/arm/mach-omap2/omap4-keypad.h deleted file mode 100644 index 20de0d5a7e77..000000000000 --- a/arch/arm/mach-omap2/omap4-keypad.h +++ /dev/null @@ -1,8 +0,0 @@ -#ifndef ARCH_ARM_PLAT_OMAP4_KEYPAD_H -#define ARCH_ARM_PLAT_OMAP4_KEYPAD_H - -struct omap_board_data; - -extern int omap4_keyboard_init(struct omap4_keypad_platform_data *, - struct omap_board_data *); -#endif diff --git a/arch/arm/mach-s5pv210/mach-goni.c b/arch/arm/mach-s5pv210/mach-goni.c index 6c719eccb94e..c1ce921c4088 100644 --- a/arch/arm/mach-s5pv210/mach-goni.c +++ b/arch/arm/mach-s5pv210/mach-goni.c @@ -234,14 +234,6 @@ static void __init goni_radio_init(void) /* TSP */ static struct mxt_platform_data qt602240_platform_data = { - .x_line = 17, - .y_line = 11, - .x_size = 800, - .y_size = 480, - .blen = 0x21, - .threshold = 0x28, - .voltage = 2800000, /* 2.8V */ - .orient = MXT_DIAGONAL, .irqflags = IRQF_TRIGGER_FALLING, }; diff --git a/arch/mips/Kbuild b/arch/mips/Kbuild index d2cfe45f332b..dd295335891a 100644 --- a/arch/mips/Kbuild +++ b/arch/mips/Kbuild @@ -16,7 +16,7 @@ obj- := $(platform-) obj-y += kernel/ obj-y += mm/ -obj-y += math-emu/ +obj-y += net/ ifdef CONFIG_KVM obj-y += kvm/ diff --git a/arch/mips/Kbuild.platforms b/arch/mips/Kbuild.platforms index 6e239123d6fe..f5e18bf3275e 100644 --- a/arch/mips/Kbuild.platforms +++ b/arch/mips/Kbuild.platforms @@ -18,6 +18,7 @@ platforms += loongson1 platforms += mti-malta platforms += mti-sead3 platforms += netlogic +platforms += paravirt platforms += pmcs-msp71xx platforms += pnx833x platforms += ralink diff --git a/arch/mips/Kconfig b/arch/mips/Kconfig index 5e0014e864f3..7a469acee33c 100644 --- a/arch/mips/Kconfig +++ b/arch/mips/Kconfig @@ -12,6 +12,7 @@ config MIPS select HAVE_ARCH_KGDB select HAVE_ARCH_SECCOMP_FILTER select HAVE_ARCH_TRACEHOOK + select HAVE_BPF_JIT if !CPU_MICROMIPS select ARCH_HAVE_CUSTOM_GPIO_H select HAVE_FUNCTION_TRACER select HAVE_FUNCTION_TRACE_MCOUNT_TEST @@ -50,6 +51,8 @@ config MIPS select CLONE_BACKWARDS select HAVE_DEBUG_STACKOVERFLOW select HAVE_CC_STACKPROTECTOR + select CPU_PM if CPU_IDLE + select ARCH_HAS_TICK_BROADCAST if GENERIC_CLOCKEVENTS_BROADCAST menu "Machine selection" @@ -83,6 +86,7 @@ config AR7 select SYS_HAS_EARLY_PRINTK select SYS_SUPPORTS_32BIT_KERNEL select SYS_SUPPORTS_LITTLE_ENDIAN + select SYS_SUPPORTS_MIPS16 select SYS_SUPPORTS_ZBOOT_UART16550 select ARCH_REQUIRE_GPIOLIB select VLYNQ @@ -106,6 +110,7 @@ config ATH79 select SYS_HAS_EARLY_PRINTK select SYS_SUPPORTS_32BIT_KERNEL select SYS_SUPPORTS_BIG_ENDIAN + select SYS_SUPPORTS_MIPS16 help Support for the Atheros AR71XX/AR724X/AR913X SoCs. @@ -122,6 +127,7 @@ config BCM47XX select NO_EXCEPT_FILL select SYS_SUPPORTS_32BIT_KERNEL select SYS_SUPPORTS_LITTLE_ENDIAN + select SYS_SUPPORTS_MIPS16 select SYS_HAS_EARLY_PRINTK select USE_GENERIC_EARLY_PRINTK_8250 help @@ -168,9 +174,9 @@ config MACH_DECSTATION bool "DECstations" select BOOT_ELF32 select CEVT_DS1287 - select CEVT_R4K + select CEVT_R4K if CPU_R4X00 select CSRC_IOASIC - select CSRC_R4K + select CSRC_R4K if CPU_R4X00 select CPU_DADDI_WORKAROUNDS if 64BIT select CPU_R4000_WORKAROUNDS if 64BIT select CPU_R4400_WORKAROUNDS if 64BIT @@ -248,6 +254,7 @@ config LANTIQ select SYS_HAS_CPU_MIPS32_R2 select SYS_SUPPORTS_BIG_ENDIAN select SYS_SUPPORTS_32BIT_KERNEL + select SYS_SUPPORTS_MIPS16 select SYS_SUPPORTS_MULTITHREADING select SYS_HAS_EARLY_PRINTK select ARCH_REQUIRE_GPIOLIB @@ -330,6 +337,7 @@ config MIPS_MALTA select SYS_SUPPORTS_LITTLE_ENDIAN select SYS_SUPPORTS_MIPS_CMP select SYS_SUPPORTS_MIPS_CPS + select SYS_SUPPORTS_MIPS16 select SYS_SUPPORTS_MULTITHREADING select SYS_SUPPORTS_SMARTMIPS select SYS_SUPPORTS_ZBOOT @@ -361,6 +369,7 @@ config MIPS_SEAD3 select SYS_SUPPORTS_LITTLE_ENDIAN select SYS_SUPPORTS_SMARTMIPS select SYS_SUPPORTS_MICROMIPS + select SYS_SUPPORTS_MIPS16 select USB_EHCI_BIG_ENDIAN_DESC select USB_EHCI_BIG_ENDIAN_MMIO select USE_OF @@ -380,6 +389,7 @@ config MACH_VR41XX select CEVT_R4K select CSRC_R4K select SYS_HAS_CPU_VR41XX + select SYS_SUPPORTS_MIPS16 select ARCH_REQUIRE_GPIOLIB config NXP_STB220 @@ -407,6 +417,7 @@ config PMC_MSP select SYS_HAS_CPU_MIPS32_R2 select SYS_SUPPORTS_32BIT_KERNEL select SYS_SUPPORTS_BIG_ENDIAN + select SYS_SUPPORTS_MIPS16 select IRQ_CPU select SERIAL_8250 select SERIAL_8250_CONSOLE @@ -430,6 +441,7 @@ config RALINK select SYS_HAS_CPU_MIPS32_R2 select SYS_SUPPORTS_32BIT_KERNEL select SYS_SUPPORTS_LITTLE_ENDIAN + select SYS_SUPPORTS_MIPS16 select SYS_HAS_EARLY_PRINTK select HAVE_MACH_CLKDEV select CLKDEV_LOOKUP @@ -674,7 +686,6 @@ config SNI_RM select SYS_SUPPORTS_BIG_ENDIAN select SYS_SUPPORTS_HIGHMEM select SYS_SUPPORTS_LITTLE_ENDIAN - select USE_GENERIC_EARLY_PRINTK_8250 help The SNI RM200/300/400 are MIPS-based machines manufactured by Siemens Nixdorf Informationssysteme (SNI), parent company of Pyramid @@ -721,6 +732,11 @@ config CAVIUM_OCTEON_SOC select ZONE_DMA32 select HOLES_IN_ZONE select ARCH_REQUIRE_GPIOLIB + select LIBFDT + select USE_OF + select ARCH_SPARSEMEM_ENABLE + select SYS_SUPPORTS_SMP + select NR_CPUS_DEFAULT_16 help This option supports all of the Octeon reference boards from Cavium Networks. It builds a kernel that dynamically determines the Octeon @@ -789,6 +805,25 @@ config NLM_XLP_BOARD This board is based on Netlogic XLP Processor. Say Y here if you have a XLP based board. +config MIPS_PARAVIRT + bool "Para-Virtualized guest system" + select CEVT_R4K + select CSRC_R4K + select DMA_COHERENT + select SYS_SUPPORTS_64BIT_KERNEL + select SYS_SUPPORTS_32BIT_KERNEL + select SYS_SUPPORTS_BIG_ENDIAN + select SYS_SUPPORTS_SMP + select NR_CPUS_DEFAULT_4 + select SYS_HAS_EARLY_PRINTK + select SYS_HAS_CPU_MIPS32_R2 + select SYS_HAS_CPU_MIPS64_R2 + select SYS_HAS_CPU_CAVIUM_OCTEON + select HW_HAS_PCI + select SWAP_IO_SPACE + help + This option supports guest running under ???? + endchoice source "arch/mips/alchemy/Kconfig" @@ -809,6 +844,7 @@ source "arch/mips/cavium-octeon/Kconfig" source "arch/mips/loongson/Kconfig" source "arch/mips/loongson1/Kconfig" source "arch/mips/netlogic/Kconfig" +source "arch/mips/paravirt/Kconfig" endmenu @@ -1059,6 +1095,7 @@ config SOC_PNX833X select SYS_SUPPORTS_32BIT_KERNEL select SYS_SUPPORTS_LITTLE_ENDIAN select SYS_SUPPORTS_BIG_ENDIAN + select SYS_SUPPORTS_MIPS16 select CPU_MIPSR2_IRQ_VI config SOC_PNX8335 @@ -1398,16 +1435,11 @@ config CPU_SB1 config CPU_CAVIUM_OCTEON bool "Cavium Octeon processor" depends on SYS_HAS_CPU_CAVIUM_OCTEON - select ARCH_SPARSEMEM_ENABLE select CPU_HAS_PREFETCH select CPU_SUPPORTS_64BIT_KERNEL - select SYS_SUPPORTS_SMP - select NR_CPUS_DEFAULT_16 select WEAK_ORDERING select CPU_SUPPORTS_HIGHMEM select CPU_SUPPORTS_HUGEPAGES - select LIBFDT - select USE_OF select USB_EHCI_BIG_ENDIAN_MMIO select MIPS_L1_CACHE_SHIFT_7 help @@ -1659,6 +1691,12 @@ config SYS_HAS_CPU_XLR config SYS_HAS_CPU_XLP bool +config MIPS_MALTA_PM + depends on MIPS_MALTA + depends on PCI + bool + default y + # # CPU may reorder R->R, R->W, W->R, W->W # Reordering beyond LL and SC is handled in WEAK_REORDERING_BEYOND_LLSC @@ -1842,7 +1880,7 @@ config FORCE_MAX_ZONEORDER config CEVT_GIC bool "Use GIC global counter for clock events" - depends on IRQ_GIC && !(MIPS_SEAD3 || MIPS_MT_SMTC) + depends on IRQ_GIC && !MIPS_SEAD3 help Use the GIC global counter for the clock events. The R4K clock event driver is always present, so if the platform ends up not @@ -1895,19 +1933,8 @@ config CPU_R4K_CACHE_TLB bool default y if !(CPU_R3000 || CPU_R8000 || CPU_SB1 || CPU_TX39XX || CPU_CAVIUM_OCTEON) -choice - prompt "MIPS MT options" - -config MIPS_MT_DISABLED - bool "Disable multithreading support" - help - Use this option if your platform does not support the MT ASE - which is hardware multithreading support. On systems without - an MT-enabled processor, this will be the only option that is - available in this menu. - config MIPS_MT_SMP - bool "Use 1 TC on each available VPE for SMP" + bool "MIPS MT SMP support (1 TC on each available VPE)" depends on SYS_SUPPORTS_MULTITHREADING select CPU_MIPSR2_IRQ_VI select CPU_MIPSR2_IRQ_EI @@ -1926,26 +1953,6 @@ config MIPS_MT_SMP Intel Hyperthreading feature. For further information go to <http://www.imgtec.com/mips/mips-multithreading.asp>. -config MIPS_MT_SMTC - bool "Use all TCs on all VPEs for SMP (DEPRECATED)" - depends on CPU_MIPS32_R2 - depends on SYS_SUPPORTS_MULTITHREADING - depends on !MIPS_CPS - select CPU_MIPSR2_IRQ_VI - select CPU_MIPSR2_IRQ_EI - select MIPS_MT - select SMP - select SMP_UP - select SYS_SUPPORTS_SMP - select NR_CPUS_DEFAULT_8 - help - This is a kernel model which is known as SMTC. This is - supported on cores with the MT ASE and presents all TCs - available on all VPEs to support SMP. For further - information see <http://www.linux-mips.org/wiki/34K#SMTC>. - -endchoice - config MIPS_MT bool @@ -1967,7 +1974,7 @@ config SYS_SUPPORTS_MULTITHREADING config MIPS_MT_FPAFF bool "Dynamic FPU affinity for FP-intensive threads" default y - depends on MIPS_MT_SMP || MIPS_MT_SMTC + depends on MIPS_MT_SMP config MIPS_VPE_LOADER bool "VPE loader support." @@ -1989,29 +1996,6 @@ config MIPS_VPE_LOADER_MT default "y" depends on MIPS_VPE_LOADER && !MIPS_CMP -config MIPS_MT_SMTC_IM_BACKSTOP - bool "Use per-TC register bits as backstop for inhibited IM bits" - depends on MIPS_MT_SMTC - default n - help - To support multiple TC microthreads acting as "CPUs" within - a VPE, VPE-wide interrupt mask bits must be specially manipulated - during interrupt handling. To support legacy drivers and interrupt - controller management code, SMTC has a "backstop" to track and - if necessary restore the interrupt mask. This has some performance - impact on interrupt service overhead. - -config MIPS_MT_SMTC_IRQAFF - bool "Support IRQ affinity API" - depends on MIPS_MT_SMTC - default n - help - Enables SMP IRQ affinity API (/proc/irq/*/smp_affinity, etc.) - for SMTC Linux kernel. Requires platform support, of which - an example can be found in the MIPS kernel i8259 and Malta - platform code. Adds some overhead to interrupt dispatch, and - should be used only if you know what you are doing. - config MIPS_VPE_LOADER_TOM bool "Load VPE program into memory hidden from linux" depends on MIPS_VPE_LOADER @@ -2039,7 +2023,7 @@ config MIPS_VPE_APSP_API_MT config MIPS_CMP bool "MIPS CMP framework support (DEPRECATED)" - depends on SYS_SUPPORTS_MIPS_CMP && !MIPS_MT_SMTC + depends on SYS_SUPPORTS_MIPS_CMP select MIPS_GIC_IPI select SYNC_R4K select WEAK_ORDERING @@ -2057,9 +2041,11 @@ config MIPS_CPS depends on SYS_SUPPORTS_MIPS_CPS select MIPS_CM select MIPS_CPC + select MIPS_CPS_PM if HOTPLUG_CPU select MIPS_GIC_IPI select SMP select SYNC_R4K if (CEVT_R4K || CSRC_R4K) + select SYS_SUPPORTS_HOTPLUG_CPU select SYS_SUPPORTS_SMP select WEAK_ORDERING help @@ -2069,6 +2055,9 @@ config MIPS_CPS no external assistance. It is safe to enable this when hardware support is unavailable. +config MIPS_CPS_PM + bool + config MIPS_GIC_IPI bool @@ -2199,6 +2188,13 @@ config SYS_SUPPORTS_SMARTMIPS config SYS_SUPPORTS_MICROMIPS bool +config SYS_SUPPORTS_MIPS16 + bool + help + This option must be set if a kernel might be executed on a MIPS16- + enabled CPU even if MIPS16 is not actually being used. In other + words, it makes the kernel MIPS16-tolerant. + config CPU_SUPPORTS_MSA bool @@ -2239,7 +2235,7 @@ config NODES_SHIFT config HW_PERF_EVENTS bool "Enable hardware performance counter support for perf events" - depends on PERF_EVENTS && !MIPS_MT_SMTC && OPROFILE=n && (CPU_MIPS32 || CPU_MIPS64 || CPU_R10000 || CPU_SB1 || CPU_CAVIUM_OCTEON || CPU_XLP) + depends on PERF_EVENTS && OPROFILE=n && (CPU_MIPS32 || CPU_MIPS64 || CPU_R10000 || CPU_SB1 || CPU_CAVIUM_OCTEON || CPU_XLP) default y help Enable hardware performance counter support for perf events. If @@ -2297,8 +2293,8 @@ config NR_CPUS_DEFAULT_64 bool config NR_CPUS - int "Maximum number of CPUs (2-64)" - range 2 64 + int "Maximum number of CPUs (2-256)" + range 2 256 depends on SMP default "4" if NR_CPUS_DEFAULT_4 default "8" if NR_CPUS_DEFAULT_8 @@ -2671,12 +2667,16 @@ endmenu config MIPS_EXTERNAL_TIMER bool -if CPU_SUPPORTS_CPUFREQ && MIPS_EXTERNAL_TIMER menu "CPU Power Management" + +if CPU_SUPPORTS_CPUFREQ && MIPS_EXTERNAL_TIMER source "drivers/cpufreq/Kconfig" -endmenu endif +source "drivers/cpuidle/Kconfig" + +endmenu + source "net/Kconfig" source "drivers/Kconfig" diff --git a/arch/mips/Kconfig.debug b/arch/mips/Kconfig.debug index 25de29211d76..3a2b775e8458 100644 --- a/arch/mips/Kconfig.debug +++ b/arch/mips/Kconfig.debug @@ -79,15 +79,6 @@ config CMDLINE_OVERRIDE Normally, you will choose 'N' here. -config SMTC_IDLE_HOOK_DEBUG - bool "Enable additional debug checks before going into CPU idle loop" - depends on DEBUG_KERNEL && MIPS_MT_SMTC - help - This option enables Enable additional debug checks before going into - CPU idle loop. For details on these checks, see - arch/mips/kernel/smtc.c. This debugging option result in significant - overhead so should be disabled in production kernels. - config SB1XXX_CORELIS bool "Corelis Debugger" depends on SIBYTE_SB1xxx_SOC diff --git a/arch/mips/Makefile b/arch/mips/Makefile index 60a359cfa328..a8521de14791 100644 --- a/arch/mips/Makefile +++ b/arch/mips/Makefile @@ -120,7 +120,7 @@ cflags-$(CONFIG_SB1XXX_CORELIS) += $(call cc-option,-mno-sched-prolog) \ -fno-omit-frame-pointer ifeq ($(CONFIG_CPU_HAS_MSA),y) -toolchain-msa := $(call cc-option-yn,-mhard-float -mfp64 -mmsa) +toolchain-msa := $(call cc-option-yn,-mhard-float -mfp64 -Wa$(comma)-mmsa) cflags-$(toolchain-msa) += -DTOOLCHAIN_SUPPORTS_MSA endif @@ -251,6 +251,7 @@ OBJCOPYFLAGS += --remove-section=.reginfo head-y := arch/mips/kernel/head.o libs-y += arch/mips/lib/ +libs-y += arch/mips/math-emu/ # See arch/mips/Kbuild for content of core part of the kernel core-y += arch/mips/ diff --git a/arch/mips/alchemy/board-xxs1500.c b/arch/mips/alchemy/board-xxs1500.c index bd5513650293..3fb814be0e91 100644 --- a/arch/mips/alchemy/board-xxs1500.c +++ b/arch/mips/alchemy/board-xxs1500.c @@ -49,7 +49,7 @@ void __init prom_init(void) prom_init_cmdline(); memsize_str = prom_getenv("memsize"); - if (!memsize_str || strict_strtoul(memsize_str, 0, &memsize)) + if (!memsize_str || kstrtoul(memsize_str, 0, &memsize)) memsize = 0x04000000; add_memory_region(0, memsize, BOOT_MEM_RAM); diff --git a/arch/mips/alchemy/common/setup.c b/arch/mips/alchemy/common/setup.c index 566a1743f685..8267e3c97721 100644 --- a/arch/mips/alchemy/common/setup.c +++ b/arch/mips/alchemy/common/setup.c @@ -67,6 +67,12 @@ void __init plat_mem_setup(void) case ALCHEMY_CPU_AU1500: case ALCHEMY_CPU_AU1100: coherentio = 0; + break; + case ALCHEMY_CPU_AU1200: + /* Au1200 AB USB does not support coherent memory */ + if (0 == (read_c0_prid() & PRID_REV_MASK)) + coherentio = 0; + break; } board_setup(); /* board specific setup */ diff --git a/arch/mips/alchemy/common/usb.c b/arch/mips/alchemy/common/usb.c index 2adc7edda49c..d193dbea84a1 100644 --- a/arch/mips/alchemy/common/usb.c +++ b/arch/mips/alchemy/common/usb.c @@ -355,47 +355,25 @@ static inline void __au1200_udc_control(void __iomem *base, int enable) } } -static inline int au1200_coherency_bug(void) -{ -#if defined(CONFIG_DMA_COHERENT) - /* Au1200 AB USB does not support coherent memory */ - if (!(read_c0_prid() & PRID_REV_MASK)) { - printk(KERN_INFO "Au1200 USB: this is chip revision AB !!\n"); - printk(KERN_INFO "Au1200 USB: update your board or re-configure" - " the kernel\n"); - return -ENODEV; - } -#endif - return 0; -} - static inline int au1200_usb_control(int block, int enable) { void __iomem *base = (void __iomem *)KSEG1ADDR(AU1200_USB_CTL_PHYS_ADDR); - int ret = 0; switch (block) { case ALCHEMY_USB_OHCI0: - ret = au1200_coherency_bug(); - if (ret && enable) - goto out; __au1200_ohci_control(base, enable); break; case ALCHEMY_USB_UDC0: __au1200_udc_control(base, enable); break; case ALCHEMY_USB_EHCI0: - ret = au1200_coherency_bug(); - if (ret && enable) - goto out; __au1200_ehci_control(base, enable); break; default: - ret = -ENODEV; + return -ENODEV; } -out: - return ret; + return 0; } diff --git a/arch/mips/alchemy/devboards/pm.c b/arch/mips/alchemy/devboards/pm.c index b86bff31d1d3..61e90fe9eab1 100644 --- a/arch/mips/alchemy/devboards/pm.c +++ b/arch/mips/alchemy/devboards/pm.c @@ -158,7 +158,7 @@ static ssize_t db1x_pmattr_store(struct kobject *kobj, int tmp; if (ATTRCMP(timer_timeout)) { - tmp = strict_strtoul(instr, 0, &l); + tmp = kstrtoul(instr, 0, &l); if (tmp) return tmp; @@ -181,7 +181,7 @@ static ssize_t db1x_pmattr_store(struct kobject *kobj, } } else if (ATTRCMP(wakemsk)) { - tmp = strict_strtoul(instr, 0, &l); + tmp = kstrtoul(instr, 0, &l); if (tmp) return tmp; diff --git a/arch/mips/bcm47xx/prom.c b/arch/mips/bcm47xx/prom.c index 0af808dfd1ca..1a03a2f43496 100644 --- a/arch/mips/bcm47xx/prom.c +++ b/arch/mips/bcm47xx/prom.c @@ -69,15 +69,18 @@ static __init void prom_init_mem(void) * BCM47XX uses 128MB for addressing the ram, if the system contains * less that that amount of ram it remaps the ram more often into the * available space. - * Accessing memory after 128MB will cause an exception. - * max contains the biggest possible address supported by the platform. - * If the method wants to try something above we assume 128MB ram. */ - off = (unsigned long)prom_init; - max = off | ((128 << 20) - 1); - for (mem = (1 << 20); mem < (128 << 20); mem += (1 << 20)) { - if ((off + mem) > max) { - mem = (128 << 20); + + /* Physical address, without mapping to any kernel segment */ + off = CPHYSADDR((unsigned long)prom_init); + + /* Accessing memory after 128 MiB will cause an exception */ + max = 128 << 20; + + for (mem = 1 << 20; mem < max; mem += 1 << 20) { + /* Loop condition may be not enough, off may be over 1 MiB */ + if (off + mem >= max) { + mem = max; printk(KERN_DEBUG "assume 128MB RAM\n"); break; } diff --git a/arch/mips/cavium-octeon/Kconfig b/arch/mips/cavium-octeon/Kconfig index 227705d9d5ae..602866657938 100644 --- a/arch/mips/cavium-octeon/Kconfig +++ b/arch/mips/cavium-octeon/Kconfig @@ -10,6 +10,17 @@ config CAVIUM_CN63XXP1 non-CN63XXP1 hardware, so it is recommended to select "n" unless it is known the workarounds are needed. +config CAVIUM_OCTEON_CVMSEG_SIZE + int "Number of L1 cache lines reserved for CVMSEG memory" + range 0 54 + default 1 + help + CVMSEG LM is a segment that accesses portions of the dcache as a + local memory; the larger CVMSEG is, the smaller the cache is. + This selects the size of CVMSEG LM, which is in cache blocks. The + legally range is from zero to 54 cache blocks (i.e. CVMSEG LM is + between zero and 6192 bytes). + endif # CPU_CAVIUM_OCTEON if CAVIUM_OCTEON_SOC @@ -23,17 +34,6 @@ config CAVIUM_OCTEON_2ND_KERNEL with this option to be run at the same time as one built without this option. -config CAVIUM_OCTEON_CVMSEG_SIZE - int "Number of L1 cache lines reserved for CVMSEG memory" - range 0 54 - default 1 - help - CVMSEG LM is a segment that accesses portions of the dcache as a - local memory; the larger CVMSEG is, the smaller the cache is. - This selects the size of CVMSEG LM, which is in cache blocks. The - legally range is from zero to 54 cache blocks (i.e. CVMSEG LM is - between zero and 6192 bytes). - config CAVIUM_OCTEON_LOCK_L2 bool "Lock often used kernel code in the L2" default "y" @@ -86,7 +86,6 @@ config SWIOTLB select IOMMU_HELPER select NEED_SG_DMA_LENGTH - config OCTEON_ILM tristate "Module to measure interrupt latency using Octeon CIU Timer" help diff --git a/arch/mips/cavium-octeon/executive/cvmx-helper.c b/arch/mips/cavium-octeon/executive/cvmx-helper.c index 8553ad5c72b6..7e5cf7a5e2f3 100644 --- a/arch/mips/cavium-octeon/executive/cvmx-helper.c +++ b/arch/mips/cavium-octeon/executive/cvmx-helper.c @@ -106,6 +106,158 @@ int cvmx_helper_ports_on_interface(int interface) EXPORT_SYMBOL_GPL(cvmx_helper_ports_on_interface); /** + * @INTERNAL + * Return interface mode for CN68xx. + */ +static cvmx_helper_interface_mode_t __cvmx_get_mode_cn68xx(int interface) +{ + union cvmx_mio_qlmx_cfg qlm_cfg; + switch (interface) { + case 0: + qlm_cfg.u64 = cvmx_read_csr(CVMX_MIO_QLMX_CFG(0)); + /* QLM is disabled when QLM SPD is 15. */ + if (qlm_cfg.s.qlm_spd == 15) + return CVMX_HELPER_INTERFACE_MODE_DISABLED; + + if (qlm_cfg.s.qlm_cfg == 2) + return CVMX_HELPER_INTERFACE_MODE_SGMII; + else if (qlm_cfg.s.qlm_cfg == 3) + return CVMX_HELPER_INTERFACE_MODE_XAUI; + else + return CVMX_HELPER_INTERFACE_MODE_DISABLED; + case 2: + case 3: + case 4: + qlm_cfg.u64 = cvmx_read_csr(CVMX_MIO_QLMX_CFG(interface)); + /* QLM is disabled when QLM SPD is 15. */ + if (qlm_cfg.s.qlm_spd == 15) + return CVMX_HELPER_INTERFACE_MODE_DISABLED; + + if (qlm_cfg.s.qlm_cfg == 2) + return CVMX_HELPER_INTERFACE_MODE_SGMII; + else if (qlm_cfg.s.qlm_cfg == 3) + return CVMX_HELPER_INTERFACE_MODE_XAUI; + else + return CVMX_HELPER_INTERFACE_MODE_DISABLED; + case 7: + qlm_cfg.u64 = cvmx_read_csr(CVMX_MIO_QLMX_CFG(3)); + /* QLM is disabled when QLM SPD is 15. */ + if (qlm_cfg.s.qlm_spd == 15) { + return CVMX_HELPER_INTERFACE_MODE_DISABLED; + } else if (qlm_cfg.s.qlm_cfg != 0) { + qlm_cfg.u64 = cvmx_read_csr(CVMX_MIO_QLMX_CFG(1)); + if (qlm_cfg.s.qlm_cfg != 0) + return CVMX_HELPER_INTERFACE_MODE_DISABLED; + } + return CVMX_HELPER_INTERFACE_MODE_NPI; + case 8: + return CVMX_HELPER_INTERFACE_MODE_LOOP; + default: + return CVMX_HELPER_INTERFACE_MODE_DISABLED; + } +} + +/** + * @INTERNAL + * Return interface mode for an Octeon II + */ +static cvmx_helper_interface_mode_t __cvmx_get_mode_octeon2(int interface) +{ + union cvmx_gmxx_inf_mode mode; + + if (OCTEON_IS_MODEL(OCTEON_CN68XX)) + return __cvmx_get_mode_cn68xx(interface); + + if (interface == 2) + return CVMX_HELPER_INTERFACE_MODE_NPI; + + if (interface == 3) + return CVMX_HELPER_INTERFACE_MODE_LOOP; + + /* Only present in CN63XX & CN66XX Octeon model */ + if ((OCTEON_IS_MODEL(OCTEON_CN63XX) && + (interface == 4 || interface == 5)) || + (OCTEON_IS_MODEL(OCTEON_CN66XX) && + interface >= 4 && interface <= 7)) { + return CVMX_HELPER_INTERFACE_MODE_DISABLED; + } + + if (OCTEON_IS_MODEL(OCTEON_CN66XX)) { + union cvmx_mio_qlmx_cfg mio_qlm_cfg; + + /* QLM2 is SGMII0 and QLM1 is SGMII1 */ + if (interface == 0) + mio_qlm_cfg.u64 = cvmx_read_csr(CVMX_MIO_QLMX_CFG(2)); + else if (interface == 1) + mio_qlm_cfg.u64 = cvmx_read_csr(CVMX_MIO_QLMX_CFG(1)); + else + return CVMX_HELPER_INTERFACE_MODE_DISABLED; + + if (mio_qlm_cfg.s.qlm_spd == 15) + return CVMX_HELPER_INTERFACE_MODE_DISABLED; + + if (mio_qlm_cfg.s.qlm_cfg == 9) + return CVMX_HELPER_INTERFACE_MODE_SGMII; + else if (mio_qlm_cfg.s.qlm_cfg == 11) + return CVMX_HELPER_INTERFACE_MODE_XAUI; + else + return CVMX_HELPER_INTERFACE_MODE_DISABLED; + } else if (OCTEON_IS_MODEL(OCTEON_CN61XX)) { + union cvmx_mio_qlmx_cfg qlm_cfg; + + if (interface == 0) { + qlm_cfg.u64 = cvmx_read_csr(CVMX_MIO_QLMX_CFG(2)); + if (qlm_cfg.s.qlm_cfg == 2) + return CVMX_HELPER_INTERFACE_MODE_SGMII; + else if (qlm_cfg.s.qlm_cfg == 3) + return CVMX_HELPER_INTERFACE_MODE_XAUI; + else + return CVMX_HELPER_INTERFACE_MODE_DISABLED; + } else if (interface == 1) { + qlm_cfg.u64 = cvmx_read_csr(CVMX_MIO_QLMX_CFG(0)); + if (qlm_cfg.s.qlm_cfg == 2) + return CVMX_HELPER_INTERFACE_MODE_SGMII; + else if (qlm_cfg.s.qlm_cfg == 3) + return CVMX_HELPER_INTERFACE_MODE_XAUI; + else + return CVMX_HELPER_INTERFACE_MODE_DISABLED; + } + } else if (OCTEON_IS_MODEL(OCTEON_CNF71XX)) { + if (interface == 0) { + union cvmx_mio_qlmx_cfg qlm_cfg; + qlm_cfg.u64 = cvmx_read_csr(CVMX_MIO_QLMX_CFG(0)); + if (qlm_cfg.s.qlm_cfg == 2) + return CVMX_HELPER_INTERFACE_MODE_SGMII; + } + return CVMX_HELPER_INTERFACE_MODE_DISABLED; + } + + if (interface == 1 && OCTEON_IS_MODEL(OCTEON_CN63XX)) + return CVMX_HELPER_INTERFACE_MODE_DISABLED; + + mode.u64 = cvmx_read_csr(CVMX_GMXX_INF_MODE(interface)); + + if (OCTEON_IS_MODEL(OCTEON_CN63XX)) { + switch (mode.cn63xx.mode) { + case 0: + return CVMX_HELPER_INTERFACE_MODE_SGMII; + case 1: + return CVMX_HELPER_INTERFACE_MODE_XAUI; + default: + return CVMX_HELPER_INTERFACE_MODE_DISABLED; + } + } else { + if (!mode.s.en) + return CVMX_HELPER_INTERFACE_MODE_DISABLED; + + if (mode.s.type) + return CVMX_HELPER_INTERFACE_MODE_GMII; + else + return CVMX_HELPER_INTERFACE_MODE_RGMII; + } +} + +/** * Get the operating mode of an interface. Depending on the Octeon * chip and configuration, this function returns an enumeration * of the type of packet I/O supported by an interface. @@ -118,6 +270,20 @@ EXPORT_SYMBOL_GPL(cvmx_helper_ports_on_interface); cvmx_helper_interface_mode_t cvmx_helper_interface_get_mode(int interface) { union cvmx_gmxx_inf_mode mode; + + if (interface < 0 || + interface >= cvmx_helper_get_number_of_interfaces()) + return CVMX_HELPER_INTERFACE_MODE_DISABLED; + + /* + * Octeon II models + */ + if (OCTEON_IS_MODEL(OCTEON_CN6XXX) || OCTEON_IS_MODEL(OCTEON_CNF71XX)) + return __cvmx_get_mode_octeon2(interface); + + /* + * Octeon and Octeon Plus models + */ if (interface == 2) return CVMX_HELPER_INTERFACE_MODE_NPI; diff --git a/arch/mips/cavium-octeon/octeon-irq.c b/arch/mips/cavium-octeon/octeon-irq.c index 3aa5b46b2d40..1b82ac6921e0 100644 --- a/arch/mips/cavium-octeon/octeon-irq.c +++ b/arch/mips/cavium-octeon/octeon-irq.c @@ -1260,11 +1260,13 @@ static void __init octeon_irq_init_ciu(void) for (i = 0; i < 4; i++) octeon_irq_force_ciu_mapping(ciu_domain, i + OCTEON_IRQ_PCI_MSI0, 0, i + 40); + octeon_irq_force_ciu_mapping(ciu_domain, OCTEON_IRQ_TWSI, 0, 45); octeon_irq_force_ciu_mapping(ciu_domain, OCTEON_IRQ_RML, 0, 46); for (i = 0; i < 4; i++) octeon_irq_force_ciu_mapping(ciu_domain, i + OCTEON_IRQ_TIMER0, 0, i + 52); octeon_irq_force_ciu_mapping(ciu_domain, OCTEON_IRQ_USB0, 0, 56); + octeon_irq_force_ciu_mapping(ciu_domain, OCTEON_IRQ_TWSI2, 0, 59); /* CIU_1 */ for (i = 0; i < 16; i++) diff --git a/arch/mips/cavium-octeon/setup.c b/arch/mips/cavium-octeon/setup.c index f1bec00d5a85..008e9c8b8eac 100644 --- a/arch/mips/cavium-octeon/setup.c +++ b/arch/mips/cavium-octeon/setup.c @@ -729,17 +729,6 @@ void __init prom_init(void) octeon_write_lcd("Linux"); #endif -#ifdef CONFIG_CAVIUM_GDB - /* - * When debugging the linux kernel, force the cores to enter - * the debug exception handler to break in. - */ - if (octeon_get_boot_debug_flag()) { - cvmx_write_csr(CVMX_CIU_DINT, 1 << cvmx_get_core_num()); - cvmx_read_csr(CVMX_CIU_DINT); - } -#endif - octeon_setup_delays(); /* @@ -779,12 +768,6 @@ void __init prom_init(void) MAX_MEMORY = 32ull << 30; if (*p == '@') RESERVE_LOW_MEM = memparse(p + 1, &p); - } else if (strcmp(arg, "ecc_verbose") == 0) { -#ifdef CONFIG_CAVIUM_REPORT_SINGLE_BIT_ECC - __cvmx_interrupt_ecc_report_single_bit_errors = 1; - pr_notice("Reporting of single bit ECC errors is " - "turned on\n"); -#endif #ifdef CONFIG_KEXEC } else if (strncmp(arg, "crashkernel=", 12) == 0) { crashk_size = memparse(arg+12, &p); diff --git a/arch/mips/cavium-octeon/smp.c b/arch/mips/cavium-octeon/smp.c index 67a078ffc464..a7b3ae104d8c 100644 --- a/arch/mips/cavium-octeon/smp.c +++ b/arch/mips/cavium-octeon/smp.c @@ -218,15 +218,6 @@ void octeon_prepare_cpus(unsigned int max_cpus) */ static void octeon_smp_finish(void) { -#ifdef CONFIG_CAVIUM_GDB - unsigned long tmp; - /* Pulse MCD0 signal on Ctrl-C to stop all the cores. Also set the MCD0 - to be not masked by this core so we know the signal is received by - someone */ - asm volatile ("dmfc0 %0, $22\n" - "ori %0, %0, 0x9100\n" "dmtc0 %0, $22\n" : "=r" (tmp)); -#endif - octeon_user_io_init(); /* to generate the first CPU timer interrupt */ @@ -234,21 +225,6 @@ static void octeon_smp_finish(void) local_irq_enable(); } -/** - * Hook for after all CPUs are online - */ -static void octeon_cpus_done(void) -{ -#ifdef CONFIG_CAVIUM_GDB - unsigned long tmp; - /* Pulse MCD0 signal on Ctrl-C to stop all the cores. Also set the MCD0 - to be not masked by this core so we know the signal is received by - someone */ - asm volatile ("dmfc0 %0, $22\n" - "ori %0, %0, 0x9100\n" "dmtc0 %0, $22\n" : "=r" (tmp)); -#endif -} - #ifdef CONFIG_HOTPLUG_CPU /* State of each CPU. */ @@ -405,7 +381,6 @@ struct plat_smp_ops octeon_smp_ops = { .send_ipi_mask = octeon_send_ipi_mask, .init_secondary = octeon_init_secondary, .smp_finish = octeon_smp_finish, - .cpus_done = octeon_cpus_done, .boot_secondary = octeon_boot_secondary, .smp_setup = octeon_smp_setup, .prepare_cpus = octeon_prepare_cpus, diff --git a/arch/mips/configs/ath79_defconfig b/arch/mips/configs/ath79_defconfig index e3a3836508ec..134879c1310a 100644 --- a/arch/mips/configs/ath79_defconfig +++ b/arch/mips/configs/ath79_defconfig @@ -46,7 +46,6 @@ CONFIG_MTD=y CONFIG_MTD_REDBOOT_PARTS=y CONFIG_MTD_REDBOOT_DIRECTORY_BLOCK=-2 CONFIG_MTD_CMDLINE_PARTS=y -CONFIG_MTD_CHAR=y CONFIG_MTD_BLOCK=y CONFIG_MTD_CFI=y CONFIG_MTD_JEDECPROBE=y @@ -54,7 +53,7 @@ CONFIG_MTD_CFI_AMDSTD=y CONFIG_MTD_COMPLEX_MAPPINGS=y CONFIG_MTD_PHYSMAP=y CONFIG_MTD_M25P80=y -# CONFIG_M25PXX_USE_FAST_READ is not set +CONFIG_MTD_SPI_NOR=y CONFIG_NETDEVICES=y # CONFIG_NET_PACKET_ENGINE is not set CONFIG_ATH_COMMON=m diff --git a/arch/mips/configs/db1xxx_defconfig b/arch/mips/configs/db1xxx_defconfig index c99b6eeda90b..a64b30b96a0d 100644 --- a/arch/mips/configs/db1xxx_defconfig +++ b/arch/mips/configs/db1xxx_defconfig @@ -113,6 +113,7 @@ CONFIG_MTD_NAND=y CONFIG_MTD_NAND_ECC_BCH=y CONFIG_MTD_NAND_AU1550=y CONFIG_MTD_NAND_PLATFORM=y +CONFIG_MTD_SPI_NOR=y CONFIG_EEPROM_AT24=y CONFIG_EEPROM_AT25=y CONFIG_SCSI_TGT=y diff --git a/arch/mips/configs/maltasmtc_defconfig b/arch/mips/configs/maltasmtc_defconfig deleted file mode 100644 index eb316447588c..000000000000 --- a/arch/mips/configs/maltasmtc_defconfig +++ /dev/null @@ -1,196 +0,0 @@ -CONFIG_MIPS_MALTA=y -CONFIG_CPU_LITTLE_ENDIAN=y -CONFIG_CPU_MIPS32_R2=y -CONFIG_PAGE_SIZE_16KB=y -CONFIG_MIPS_MT_SMTC=y -# CONFIG_MIPS_MT_FPAFF is not set -CONFIG_NR_CPUS=9 -CONFIG_HZ_48=y -CONFIG_LOCALVERSION="smtc" -CONFIG_SYSVIPC=y -CONFIG_POSIX_MQUEUE=y -CONFIG_AUDIT=y -CONFIG_IKCONFIG=y -CONFIG_IKCONFIG_PROC=y -CONFIG_LOG_BUF_SHIFT=15 -CONFIG_SYSCTL_SYSCALL=y -CONFIG_EMBEDDED=y -CONFIG_SLAB=y -CONFIG_MODULES=y -CONFIG_MODULE_UNLOAD=y -CONFIG_MODVERSIONS=y -CONFIG_MODULE_SRCVERSION_ALL=y -# CONFIG_BLK_DEV_BSG is not set -CONFIG_PCI=y -# CONFIG_CORE_DUMP_DEFAULT_ELF_HEADERS is not set -CONFIG_NET=y -CONFIG_PACKET=y -CONFIG_UNIX=y -CONFIG_XFRM_USER=m -CONFIG_NET_KEY=y -CONFIG_INET=y -CONFIG_IP_MULTICAST=y -CONFIG_IP_ADVANCED_ROUTER=y -CONFIG_IP_MULTIPLE_TABLES=y -CONFIG_IP_ROUTE_MULTIPATH=y -CONFIG_IP_ROUTE_VERBOSE=y -CONFIG_IP_PNP=y -CONFIG_IP_PNP_DHCP=y -CONFIG_IP_PNP_BOOTP=y -CONFIG_NET_IPIP=m -CONFIG_IP_MROUTE=y -CONFIG_IP_PIMSM_V1=y -CONFIG_IP_PIMSM_V2=y -CONFIG_SYN_COOKIES=y -CONFIG_INET_AH=m -CONFIG_INET_ESP=m -CONFIG_INET_IPCOMP=m -# CONFIG_INET_LRO is not set -CONFIG_INET6_AH=m -CONFIG_INET6_ESP=m -CONFIG_INET6_IPCOMP=m -CONFIG_IPV6_TUNNEL=m -CONFIG_BRIDGE=m -CONFIG_VLAN_8021Q=m -CONFIG_ATALK=m -CONFIG_DEV_APPLETALK=m -CONFIG_IPDDP=m -CONFIG_IPDDP_ENCAP=y -CONFIG_NET_SCHED=y -CONFIG_NET_SCH_CBQ=m -CONFIG_NET_SCH_HTB=m -CONFIG_NET_SCH_HFSC=m -CONFIG_NET_SCH_PRIO=m -CONFIG_NET_SCH_RED=m -CONFIG_NET_SCH_SFQ=m -CONFIG_NET_SCH_TEQL=m -CONFIG_NET_SCH_TBF=m -CONFIG_NET_SCH_GRED=m -CONFIG_NET_SCH_DSMARK=m -CONFIG_NET_SCH_NETEM=m -CONFIG_NET_SCH_INGRESS=m -CONFIG_NET_CLS_BASIC=m -CONFIG_NET_CLS_TCINDEX=m -CONFIG_NET_CLS_ROUTE4=m -CONFIG_NET_CLS_FW=m -CONFIG_NET_CLS_U32=m -CONFIG_NET_CLS_RSVP=m -CONFIG_NET_CLS_RSVP6=m -CONFIG_NET_CLS_ACT=y -CONFIG_NET_ACT_POLICE=y -CONFIG_NET_CLS_IND=y -# CONFIG_WIRELESS is not set -CONFIG_DEVTMPFS=y -CONFIG_BLK_DEV_LOOP=y -CONFIG_BLK_DEV_CRYPTOLOOP=m -CONFIG_IDE=y -# CONFIG_IDE_PROC_FS is not set -# CONFIG_IDEPCI_PCIBUS_ORDER is not set -CONFIG_BLK_DEV_GENERIC=y -CONFIG_BLK_DEV_PIIX=y -CONFIG_SCSI=y -CONFIG_BLK_DEV_SD=y -CONFIG_CHR_DEV_SG=y -# CONFIG_SCSI_LOWLEVEL is not set -CONFIG_NETDEVICES=y -# CONFIG_NET_VENDOR_3COM is not set -# CONFIG_NET_VENDOR_ADAPTEC is not set -# CONFIG_NET_VENDOR_ALTEON is not set -CONFIG_PCNET32=y -# CONFIG_NET_VENDOR_ATHEROS is not set -# CONFIG_NET_VENDOR_BROADCOM is not set -# CONFIG_NET_VENDOR_BROCADE is not set -# CONFIG_NET_VENDOR_CHELSIO is not set -# CONFIG_NET_VENDOR_CISCO is not set -# CONFIG_NET_VENDOR_DEC is not set -# CONFIG_NET_VENDOR_DLINK is not set -# CONFIG_NET_VENDOR_EMULEX is not set -# CONFIG_NET_VENDOR_EXAR is not set -# CONFIG_NET_VENDOR_HP is not set -# CONFIG_NET_VENDOR_INTEL is not set -# CONFIG_NET_VENDOR_MARVELL is not set -# CONFIG_NET_VENDOR_MELLANOX is not set -# CONFIG_NET_VENDOR_MICREL is not set -# CONFIG_NET_VENDOR_MYRI is not set -# CONFIG_NET_VENDOR_NATSEMI is not set -# CONFIG_NET_VENDOR_NVIDIA is not set -# CONFIG_NET_VENDOR_OKI is not set -# CONFIG_NET_PACKET_ENGINE is not set -# CONFIG_NET_VENDOR_QLOGIC is not set -# CONFIG_NET_VENDOR_REALTEK is not set -# CONFIG_NET_VENDOR_RDC is not set -# CONFIG_NET_VENDOR_SEEQ is not set -# CONFIG_NET_VENDOR_SILAN is not set -# CONFIG_NET_VENDOR_SIS is not set -# CONFIG_NET_VENDOR_SMSC is not set -# CONFIG_NET_VENDOR_STMICRO is not set -# CONFIG_NET_VENDOR_SUN is not set -# CONFIG_NET_VENDOR_TEHUTI is not set -# CONFIG_NET_VENDOR_TI is not set -# CONFIG_NET_VENDOR_TOSHIBA is not set -# CONFIG_NET_VENDOR_VIA is not set -# CONFIG_WLAN is not set -# CONFIG_VT is not set -CONFIG_LEGACY_PTY_COUNT=16 -CONFIG_SERIAL_8250=y -CONFIG_SERIAL_8250_CONSOLE=y -CONFIG_HW_RANDOM=y -# CONFIG_HWMON is not set -CONFIG_VIDEO_OUTPUT_CONTROL=m -CONFIG_FB=y -CONFIG_FIRMWARE_EDID=y -CONFIG_FB_MATROX=y -CONFIG_FB_MATROX_G=y -CONFIG_USB=y -CONFIG_USB_EHCI_HCD=y -# CONFIG_USB_EHCI_TT_NEWSCHED is not set -CONFIG_USB_UHCI_HCD=y -CONFIG_USB_STORAGE=y -CONFIG_NEW_LEDS=y -CONFIG_LEDS_CLASS=y -CONFIG_LEDS_TRIGGERS=y -CONFIG_LEDS_TRIGGER_TIMER=y -CONFIG_LEDS_TRIGGER_IDE_DISK=y -CONFIG_LEDS_TRIGGER_HEARTBEAT=y -CONFIG_LEDS_TRIGGER_BACKLIGHT=y -CONFIG_LEDS_TRIGGER_DEFAULT_ON=y -CONFIG_RTC_CLASS=y -CONFIG_RTC_DRV_CMOS=y -CONFIG_EXT2_FS=y -CONFIG_EXT3_FS=y -# CONFIG_EXT3_DEFAULTS_TO_ORDERED is not set -CONFIG_XFS_FS=y -CONFIG_XFS_QUOTA=y -CONFIG_XFS_POSIX_ACL=y -CONFIG_QUOTA=y -CONFIG_QFMT_V2=y -CONFIG_MSDOS_FS=m -CONFIG_VFAT_FS=m -CONFIG_PROC_KCORE=y -CONFIG_TMPFS=y -CONFIG_NFS_FS=y -CONFIG_ROOT_NFS=y -CONFIG_CIFS=m -CONFIG_CIFS_WEAK_PW_HASH=y -CONFIG_CIFS_XATTR=y -CONFIG_CIFS_POSIX=y -CONFIG_NLS_CODEPAGE_437=m -CONFIG_NLS_ISO8859_1=m -# CONFIG_FTRACE is not set -CONFIG_CRYPTO_NULL=m -CONFIG_CRYPTO_PCBC=m -CONFIG_CRYPTO_HMAC=y -CONFIG_CRYPTO_MICHAEL_MIC=m -CONFIG_CRYPTO_SHA512=m -CONFIG_CRYPTO_TGR192=m -CONFIG_CRYPTO_WP512=m -CONFIG_CRYPTO_ANUBIS=m -CONFIG_CRYPTO_BLOWFISH=m -CONFIG_CRYPTO_CAST5=m -CONFIG_CRYPTO_CAST6=m -CONFIG_CRYPTO_KHAZAD=m -CONFIG_CRYPTO_SERPENT=m -CONFIG_CRYPTO_TEA=m -CONFIG_CRYPTO_TWOFISH=m -# CONFIG_CRYPTO_ANSI_CPRNG is not set -# CONFIG_CRYPTO_HW is not set diff --git a/arch/mips/configs/maltasmvp_defconfig b/arch/mips/configs/maltasmvp_defconfig index 10ef3bed5f43..f8a32315bb38 100644 --- a/arch/mips/configs/maltasmvp_defconfig +++ b/arch/mips/configs/maltasmvp_defconfig @@ -4,10 +4,9 @@ CONFIG_CPU_MIPS32_R2=y CONFIG_PAGE_SIZE_16KB=y CONFIG_MIPS_MT_SMP=y CONFIG_SCHED_SMT=y -CONFIG_MIPS_CMP=y +CONFIG_MIPS_CPS=y CONFIG_NR_CPUS=8 CONFIG_HZ_100=y -CONFIG_LOCALVERSION="cmp" CONFIG_SYSVIPC=y CONFIG_POSIX_MQUEUE=y CONFIG_AUDIT=y diff --git a/arch/mips/configs/maltasmvp_eva_defconfig b/arch/mips/configs/maltasmvp_eva_defconfig index 2d3002cba102..c83338a39917 100644 --- a/arch/mips/configs/maltasmvp_eva_defconfig +++ b/arch/mips/configs/maltasmvp_eva_defconfig @@ -5,10 +5,9 @@ CONFIG_CPU_MIPS32_3_5_FEATURES=y CONFIG_PAGE_SIZE_16KB=y CONFIG_MIPS_MT_SMP=y CONFIG_SCHED_SMT=y -CONFIG_MIPS_CMP=y +CONFIG_MIPS_CPS=y CONFIG_NR_CPUS=8 CONFIG_HZ_100=y -CONFIG_LOCALVERSION="cmp" CONFIG_SYSVIPC=y CONFIG_POSIX_MQUEUE=y CONFIG_AUDIT=y diff --git a/arch/mips/configs/mips_paravirt_defconfig b/arch/mips/configs/mips_paravirt_defconfig new file mode 100644 index 000000000000..84cfcb4bf2ea --- /dev/null +++ b/arch/mips/configs/mips_paravirt_defconfig @@ -0,0 +1,103 @@ +CONFIG_MIPS_PARAVIRT=y +CONFIG_CPU_MIPS64_R2=y +CONFIG_64BIT=y +CONFIG_TRANSPARENT_HUGEPAGE=y +CONFIG_SMP=y +CONFIG_HZ_1000=y +CONFIG_PREEMPT=y +CONFIG_SYSVIPC=y +CONFIG_BSD_PROCESS_ACCT=y +CONFIG_BSD_PROCESS_ACCT_V3=y +CONFIG_IKCONFIG=y +CONFIG_IKCONFIG_PROC=y +CONFIG_LOG_BUF_SHIFT=14 +CONFIG_RELAY=y +CONFIG_BLK_DEV_INITRD=y +CONFIG_EXPERT=y +CONFIG_SLAB=y +CONFIG_MODULES=y +CONFIG_MODULE_UNLOAD=y +# CONFIG_BLK_DEV_BSG is not set +CONFIG_PCI=y +CONFIG_MIPS32_COMPAT=y +CONFIG_MIPS32_O32=y +CONFIG_MIPS32_N32=y +CONFIG_NET=y +CONFIG_PACKET=y +CONFIG_UNIX=y +CONFIG_INET=y +CONFIG_IP_MULTICAST=y +CONFIG_IP_ADVANCED_ROUTER=y +CONFIG_IP_MULTIPLE_TABLES=y +CONFIG_IP_ROUTE_MULTIPATH=y +CONFIG_IP_ROUTE_VERBOSE=y +CONFIG_IP_PNP=y +CONFIG_IP_PNP_DHCP=y +CONFIG_IP_PNP_BOOTP=y +CONFIG_IP_PNP_RARP=y +CONFIG_IP_MROUTE=y +CONFIG_IP_PIMSM_V1=y +CONFIG_IP_PIMSM_V2=y +CONFIG_SYN_COOKIES=y +# CONFIG_INET_LRO is not set +CONFIG_IPV6=y +# CONFIG_WIRELESS is not set +CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug" +# CONFIG_FW_LOADER is not set +CONFIG_BLK_DEV_LOOP=y +CONFIG_VIRTIO_BLK=y +CONFIG_SCSI=y +CONFIG_BLK_DEV_SD=y +CONFIG_NETDEVICES=y +CONFIG_VIRTIO_NET=y +# CONFIG_NET_VENDOR_BROADCOM is not set +# CONFIG_NET_VENDOR_INTEL is not set +# CONFIG_NET_VENDOR_MARVELL is not set +# CONFIG_NET_VENDOR_MICREL is not set +# CONFIG_NET_VENDOR_NATSEMI is not set +# CONFIG_NET_VENDOR_SMSC is not set +# CONFIG_NET_VENDOR_STMICRO is not set +# CONFIG_NET_VENDOR_WIZNET is not set +CONFIG_PHYLIB=y +CONFIG_MARVELL_PHY=y +CONFIG_BROADCOM_PHY=y +CONFIG_BCM87XX_PHY=y +# CONFIG_WLAN is not set +# CONFIG_INPUT is not set +# CONFIG_SERIO is not set +# CONFIG_VT is not set +CONFIG_VIRTIO_CONSOLE=y +# CONFIG_HW_RANDOM is not set +# CONFIG_HWMON is not set +# CONFIG_USB_SUPPORT is not set +CONFIG_VIRTIO_PCI=y +CONFIG_VIRTIO_BALLOON=y +CONFIG_VIRTIO_MMIO=y +# CONFIG_IOMMU_SUPPORT is not set +CONFIG_EXT4_FS=y +CONFIG_EXT4_FS_POSIX_ACL=y +CONFIG_EXT4_FS_SECURITY=y +CONFIG_MSDOS_FS=y +CONFIG_VFAT_FS=y +CONFIG_PROC_KCORE=y +CONFIG_TMPFS=y +CONFIG_HUGETLBFS=y +# CONFIG_MISC_FILESYSTEMS is not set +CONFIG_NFS_FS=y +CONFIG_NFS_V4=y +CONFIG_NFS_V4_1=y +CONFIG_ROOT_NFS=y +CONFIG_NLS_CODEPAGE_437=y +CONFIG_NLS_ASCII=y +CONFIG_NLS_ISO8859_1=y +CONFIG_NLS_UTF8=y +CONFIG_DEBUG_INFO=y +CONFIG_DEBUG_FS=y +CONFIG_MAGIC_SYSRQ=y +# CONFIG_SCHED_DEBUG is not set +# CONFIG_FTRACE is not set +CONFIG_CRYPTO_CBC=y +CONFIG_CRYPTO_HMAC=y +CONFIG_CRYPTO_MD5=y +CONFIG_CRYPTO_DES=y +# CONFIG_CRYPTO_ANSI_CPRNG is not set diff --git a/arch/mips/configs/rt305x_defconfig b/arch/mips/configs/rt305x_defconfig index d1741bcf8949..d14ae2fa7d13 100644 --- a/arch/mips/configs/rt305x_defconfig +++ b/arch/mips/configs/rt305x_defconfig @@ -81,7 +81,6 @@ CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug" # CONFIG_FIRMWARE_IN_KERNEL is not set CONFIG_MTD=y CONFIG_MTD_CMDLINE_PARTS=y -CONFIG_MTD_CHAR=y CONFIG_MTD_BLOCK=y CONFIG_MTD_CFI=y CONFIG_MTD_CFI_AMDSTD=y @@ -89,6 +88,7 @@ CONFIG_MTD_COMPLEX_MAPPINGS=y CONFIG_MTD_PHYSMAP=y CONFIG_MTD_PHYSMAP_OF=y CONFIG_MTD_M25P80=y +CONFIG_MTD_SPI_NOR=y CONFIG_EEPROM_93CX6=m CONFIG_SCSI=y CONFIG_BLK_DEV_SD=y diff --git a/arch/mips/dec/setup.c b/arch/mips/dec/setup.c index 56e6e2c23683..41bbffd9cc0e 100644 --- a/arch/mips/dec/setup.c +++ b/arch/mips/dec/setup.c @@ -23,6 +23,7 @@ #include <asm/bootinfo.h> #include <asm/cpu.h> #include <asm/cpu-features.h> +#include <asm/cpu-type.h> #include <asm/irq.h> #include <asm/irq_cpu.h> #include <asm/mipsregs.h> @@ -748,6 +749,10 @@ void __init arch_init_irq(void) cpu_fpu_mask = 0; dec_interrupt[DEC_IRQ_FPU] = -1; } + /* Free the halt interrupt unused on R4k systems. */ + if (current_cpu_type() == CPU_R4000SC || + current_cpu_type() == CPU_R4400SC) + dec_interrupt[DEC_IRQ_HALT] = -1; /* Register board interrupts: FPU and cascade. */ if (dec_interrupt[DEC_IRQ_FPU] >= 0) diff --git a/arch/mips/include/asm/asmmacro.h b/arch/mips/include/asm/asmmacro.h index b464b8b1147a..935543f14538 100644 --- a/arch/mips/include/asm/asmmacro.h +++ b/arch/mips/include/asm/asmmacro.h @@ -17,26 +17,8 @@ #ifdef CONFIG_64BIT #include <asm/asmmacro-64.h> #endif -#ifdef CONFIG_MIPS_MT_SMTC -#include <asm/mipsmtregs.h> -#endif - -#ifdef CONFIG_MIPS_MT_SMTC - .macro local_irq_enable reg=t0 - mfc0 \reg, CP0_TCSTATUS - ori \reg, \reg, TCSTATUS_IXMT - xori \reg, \reg, TCSTATUS_IXMT - mtc0 \reg, CP0_TCSTATUS - _ehb - .endm - .macro local_irq_disable reg=t0 - mfc0 \reg, CP0_TCSTATUS - ori \reg, \reg, TCSTATUS_IXMT - mtc0 \reg, CP0_TCSTATUS - _ehb - .endm -#elif defined(CONFIG_CPU_MIPSR2) +#ifdef CONFIG_CPU_MIPSR2 .macro local_irq_enable reg=t0 ei irq_enable_hazard @@ -71,7 +53,7 @@ sw \reg, TI_PRE_COUNT($28) #endif .endm -#endif /* CONFIG_MIPS_MT_SMTC */ +#endif /* CONFIG_CPU_MIPSR2 */ .macro fpu_save_16even thread tmp=t0 cfc1 \tmp, fcr31 @@ -267,13 +249,35 @@ .set pop .endm #else + +#ifdef CONFIG_CPU_MICROMIPS +#define CFC_MSA_INSN 0x587e0056 +#define CTC_MSA_INSN 0x583e0816 +#define LDD_MSA_INSN 0x58000837 +#define STD_MSA_INSN 0x5800083f +#define COPY_UW_MSA_INSN 0x58f00056 +#define COPY_UD_MSA_INSN 0x58f80056 +#define INSERT_W_MSA_INSN 0x59300816 +#define INSERT_D_MSA_INSN 0x59380816 +#else +#define CFC_MSA_INSN 0x787e0059 +#define CTC_MSA_INSN 0x783e0819 +#define LDD_MSA_INSN 0x78000823 +#define STD_MSA_INSN 0x78000827 +#define COPY_UW_MSA_INSN 0x78f00059 +#define COPY_UD_MSA_INSN 0x78f80059 +#define INSERT_W_MSA_INSN 0x79300819 +#define INSERT_D_MSA_INSN 0x79380819 +#endif + /* * Temporary until all toolchains in use include MSA support. */ .macro cfcmsa rd, cs .set push .set noat - .word 0x787e0059 | (\cs << 11) + .insn + .word CFC_MSA_INSN | (\cs << 11) move \rd, $1 .set pop .endm @@ -282,7 +286,7 @@ .set push .set noat move $1, \rs - .word 0x783e0819 | (\cd << 6) + .word CTC_MSA_INSN | (\cd << 6) .set pop .endm @@ -290,7 +294,7 @@ .set push .set noat add $1, \base, \off - .word 0x78000823 | (\wd << 6) + .word LDD_MSA_INSN | (\wd << 6) .set pop .endm @@ -298,14 +302,15 @@ .set push .set noat add $1, \base, \off - .word 0x78000827 | (\wd << 6) + .word STD_MSA_INSN | (\wd << 6) .set pop .endm .macro copy_u_w rd, ws, n .set push .set noat - .word 0x78f00059 | (\n << 16) | (\ws << 11) + .insn + .word COPY_UW_MSA_INSN | (\n << 16) | (\ws << 11) /* move triggers an assembler bug... */ or \rd, $1, zero .set pop @@ -314,7 +319,8 @@ .macro copy_u_d rd, ws, n .set push .set noat - .word 0x78f80059 | (\n << 16) | (\ws << 11) + .insn + .word COPY_UD_MSA_INSN | (\n << 16) | (\ws << 11) /* move triggers an assembler bug... */ or \rd, $1, zero .set pop @@ -325,7 +331,7 @@ .set noat /* move triggers an assembler bug... */ or $1, \rs, zero - .word 0x79300819 | (\n << 16) | (\wd << 6) + .word INSERT_W_MSA_INSN | (\n << 16) | (\wd << 6) .set pop .endm @@ -334,7 +340,7 @@ .set noat /* move triggers an assembler bug... */ or $1, \rs, zero - .word 0x79380819 | (\n << 16) | (\wd << 6) + .word INSERT_D_MSA_INSN | (\n << 16) | (\wd << 6) .set pop .endm #endif diff --git a/arch/mips/include/asm/branch.h b/arch/mips/include/asm/branch.h index e28a3e0eb3cb..de781cf54bc7 100644 --- a/arch/mips/include/asm/branch.h +++ b/arch/mips/include/asm/branch.h @@ -8,6 +8,8 @@ #ifndef _ASM_BRANCH_H #define _ASM_BRANCH_H +#include <asm/cpu-features.h> +#include <asm/mipsregs.h> #include <asm/ptrace.h> #include <asm/inst.h> @@ -18,12 +20,40 @@ extern int __compute_return_epc_for_insn(struct pt_regs *regs, extern int __microMIPS_compute_return_epc(struct pt_regs *regs); extern int __MIPS16e_compute_return_epc(struct pt_regs *regs); +/* + * microMIPS bitfields + */ +#define MM_POOL32A_MINOR_MASK 0x3f +#define MM_POOL32A_MINOR_SHIFT 0x6 +#define MM_MIPS32_COND_FC 0x30 + +extern int __mm_isBranchInstr(struct pt_regs *regs, + struct mm_decoded_insn dec_insn, unsigned long *contpc); + +static inline int mm_isBranchInstr(struct pt_regs *regs, + struct mm_decoded_insn dec_insn, unsigned long *contpc) +{ + if (!cpu_has_mmips) + return 0; + + return __mm_isBranchInstr(regs, dec_insn, contpc); +} static inline int delay_slot(struct pt_regs *regs) { return regs->cp0_cause & CAUSEF_BD; } +static inline void clear_delay_slot(struct pt_regs *regs) +{ + regs->cp0_cause &= ~CAUSEF_BD; +} + +static inline void set_delay_slot(struct pt_regs *regs) +{ + regs->cp0_cause |= CAUSEF_BD; +} + static inline unsigned long exception_epc(struct pt_regs *regs) { if (likely(!delay_slot(regs))) diff --git a/arch/mips/include/asm/cacheflush.h b/arch/mips/include/asm/cacheflush.h index 69468ded2828..e08381a37f8b 100644 --- a/arch/mips/include/asm/cacheflush.h +++ b/arch/mips/include/asm/cacheflush.h @@ -113,6 +113,12 @@ unsigned long run_uncached(void *func); extern void *kmap_coherent(struct page *page, unsigned long addr); extern void kunmap_coherent(void); +extern void *kmap_noncoherent(struct page *page, unsigned long addr); + +static inline void kunmap_noncoherent(void) +{ + kunmap_coherent(); +} #define ARCH_HAS_FLUSH_KERNEL_DCACHE_PAGE static inline void flush_kernel_dcache_page(struct page *page) diff --git a/arch/mips/include/asm/cmp.h b/arch/mips/include/asm/cmp.h index 89a73fb93ae6..033d97303c85 100644 --- a/arch/mips/include/asm/cmp.h +++ b/arch/mips/include/asm/cmp.h @@ -10,7 +10,6 @@ extern void cmp_smp_setup(void); extern void cmp_smp_finish(void); extern void cmp_boot_secondary(int cpu, struct task_struct *t); extern void cmp_init_secondary(void); -extern void cmp_cpus_done(void); extern void cmp_prepare_cpus(unsigned int max_cpus); /* This is platform specific */ diff --git a/arch/mips/include/asm/cpu-features.h b/arch/mips/include/asm/cpu-features.h index f56cc975b92f..c7d8c997d93e 100644 --- a/arch/mips/include/asm/cpu-features.h +++ b/arch/mips/include/asm/cpu-features.h @@ -110,9 +110,15 @@ #ifndef cpu_has_smartmips #define cpu_has_smartmips (cpu_data[0].ases & MIPS_ASE_SMARTMIPS) #endif + #ifndef cpu_has_rixi -#define cpu_has_rixi (cpu_data[0].options & MIPS_CPU_RIXI) +# ifdef CONFIG_64BIT +# define cpu_has_rixi (cpu_data[0].options & MIPS_CPU_RIXI) +# else /* CONFIG_32BIT */ +# define cpu_has_rixi ((cpu_data[0].options & MIPS_CPU_RIXI) && !cpu_has_64bits) +# endif #endif + #ifndef cpu_has_mmips # ifdef CONFIG_SYS_SUPPORTS_MICROMIPS # define cpu_has_mmips (cpu_data[0].options & MIPS_CPU_MICROMIPS) @@ -120,6 +126,7 @@ # define cpu_has_mmips 0 # endif #endif + #ifndef cpu_has_vtag_icache #define cpu_has_vtag_icache (cpu_data[0].icache.flags & MIPS_CACHE_VTAG) #endif @@ -183,6 +190,17 @@ /* * Shortcuts ... */ +#define cpu_has_mips_2_3_4_5 (cpu_has_mips_2 | cpu_has_mips_3_4_5) +#define cpu_has_mips_3_4_5 (cpu_has_mips_3 | cpu_has_mips_4_5) +#define cpu_has_mips_4_5 (cpu_has_mips_4 | cpu_has_mips_5) + +#define cpu_has_mips_2_3_4_5_r (cpu_has_mips_2 | cpu_has_mips_3_4_5_r) +#define cpu_has_mips_3_4_5_r (cpu_has_mips_3 | cpu_has_mips_4_5_r) +#define cpu_has_mips_4_5_r (cpu_has_mips_4 | cpu_has_mips_5_r) +#define cpu_has_mips_5_r (cpu_has_mips_5 | cpu_has_mips_r) + +#define cpu_has_mips_4_5_r2 (cpu_has_mips_4_5 | cpu_has_mips_r2) + #define cpu_has_mips32 (cpu_has_mips32r1 | cpu_has_mips32r2) #define cpu_has_mips64 (cpu_has_mips64r1 | cpu_has_mips64r2) #define cpu_has_mips_r1 (cpu_has_mips32r1 | cpu_has_mips64r1) diff --git a/arch/mips/include/asm/cpu-info.h b/arch/mips/include/asm/cpu-info.h index ff2707ab3295..47d5967ce7ef 100644 --- a/arch/mips/include/asm/cpu-info.h +++ b/arch/mips/include/asm/cpu-info.h @@ -65,18 +65,13 @@ struct cpuinfo_mips { #ifdef CONFIG_64BIT int vmbits; /* Virtual memory size in bits */ #endif -#if defined(CONFIG_MIPS_MT_SMP) || defined(CONFIG_MIPS_MT_SMTC) +#ifdef CONFIG_MIPS_MT_SMP /* - * In the MIPS MT "SMTC" model, each TC is considered - * to be a "CPU" for the purposes of scheduling, but - * exception resources, ASID spaces, etc, are common - * to all TCs within the same VPE. + * There is not necessarily a 1:1 mapping of VPE num to CPU number + * in particular on multi-core systems. */ int vpe_id; /* Virtual Processor number */ #endif -#ifdef CONFIG_MIPS_MT_SMTC - int tc_id; /* Thread Context number */ -#endif void *data; /* Additional data */ unsigned int watch_reg_count; /* Number that exist */ unsigned int watch_reg_use_cnt; /* Usable by ptrace */ @@ -117,7 +112,7 @@ struct proc_cpuinfo_notifier_args { unsigned long n; }; -#if defined(CONFIG_MIPS_MT_SMP) || defined(CONFIG_MIPS_MT_SMTC) +#ifdef CONFIG_MIPS_MT_SMP # define cpu_vpe_id(cpuinfo) ((cpuinfo)->vpe_id) #else # define cpu_vpe_id(cpuinfo) 0 diff --git a/arch/mips/include/asm/cpu-type.h b/arch/mips/include/asm/cpu-type.h index 721906130a57..b4e2bd87df50 100644 --- a/arch/mips/include/asm/cpu-type.h +++ b/arch/mips/include/asm/cpu-type.h @@ -155,9 +155,6 @@ static inline int __pure __get_cpu_type(const int cpu_type) case CPU_RM7000: case CPU_SR71000: #endif -#ifdef CONFIG_SYS_HAS_CPU_RM9000 - case CPU_RM9000: -#endif #ifdef CONFIG_SYS_HAS_CPU_SB1 case CPU_SB1: case CPU_SB1A: @@ -166,6 +163,7 @@ static inline int __pure __get_cpu_type(const int cpu_type) case CPU_CAVIUM_OCTEON: case CPU_CAVIUM_OCTEON_PLUS: case CPU_CAVIUM_OCTEON2: + case CPU_CAVIUM_OCTEON3: #endif #if defined(CONFIG_SYS_HAS_CPU_BMIPS32_3300) || \ diff --git a/arch/mips/include/asm/cpu.h b/arch/mips/include/asm/cpu.h index 530eb8b3a68e..129d08701e91 100644 --- a/arch/mips/include/asm/cpu.h +++ b/arch/mips/include/asm/cpu.h @@ -201,6 +201,7 @@ #define PRID_IMP_NETLOGIC_XLP3XX 0x1100 #define PRID_IMP_NETLOGIC_XLP2XX 0x1200 #define PRID_IMP_NETLOGIC_XLP9XX 0x1500 +#define PRID_IMP_NETLOGIC_XLP5XX 0x1300 /* * Particular Revision values for bits 7:0 of the PRId register. @@ -281,7 +282,7 @@ enum cpu_type_enum { CPU_R4700, CPU_R5000, CPU_R5500, CPU_NEVADA, CPU_R5432, CPU_R10000, CPU_R12000, CPU_R14000, CPU_VR41XX, CPU_VR4111, CPU_VR4121, CPU_VR4122, CPU_VR4131, CPU_VR4133, CPU_VR4181, CPU_VR4181A, CPU_RM7000, - CPU_SR71000, CPU_RM9000, CPU_TX49XX, + CPU_SR71000, CPU_TX49XX, /* * R8000 class processors diff --git a/arch/mips/include/asm/dec/kn05.h b/arch/mips/include/asm/dec/kn05.h index 56d22dc8803a..8e14f677e5ef 100644 --- a/arch/mips/include/asm/dec/kn05.h +++ b/arch/mips/include/asm/dec/kn05.h @@ -49,12 +49,20 @@ #define KN4K_RES_15 (15*IOASIC_SLOT_SIZE) /* unused? */ /* + * MB ASIC interrupt bits. + */ +#define KN4K_MB_INR_MB 4 /* ??? */ +#define KN4K_MB_INR_MT 3 /* memory, I/O bus read/write errors */ +#define KN4K_MB_INR_RES_2 2 /* unused */ +#define KN4K_MB_INR_RTC 1 /* RTC */ +#define KN4K_MB_INR_TC 0 /* I/O ASIC cascade */ + +/* * Bits for the MB interrupt register. * The register appears read-only. */ -#define KN4K_MB_INT_TC (1<<0) /* TURBOchannel? */ -#define KN4K_MB_INT_RTC (1<<1) /* RTC? */ -#define KN4K_MB_INT_MT (1<<3) /* I/O ASIC cascade */ +#define KN4K_MB_INT_IRQ (0x1f<<0) /* CPU Int[4:0] status. */ +#define KN4K_MB_INT_IRQ_N(n) (1<<(n)) /* Individual status bits. */ /* * Bits for the MB control & status register. @@ -70,6 +78,7 @@ #define KN4K_MB_CSR_NC (1<<14) /* ??? */ #define KN4K_MB_CSR_EE (1<<15) /* (bus) Exception Enable? */ #define KN4K_MB_CSR_MSK (0x1f<<16) /* CPU Int[4:0] mask */ +#define KN4K_MB_CSR_MSK_N(n) (1<<((n)+16)) /* Individual mask bits. */ #define KN4K_MB_CSR_FW (1<<21) /* ??? */ #define KN4K_MB_CSR_W (1<<31) /* ??? */ diff --git a/arch/mips/include/asm/fixmap.h b/arch/mips/include/asm/fixmap.h index 8c012af2f451..6842ffafd1e7 100644 --- a/arch/mips/include/asm/fixmap.h +++ b/arch/mips/include/asm/fixmap.h @@ -48,11 +48,7 @@ enum fixed_addresses { #define FIX_N_COLOURS 8 FIX_CMAP_BEGIN, -#ifdef CONFIG_MIPS_MT_SMTC - FIX_CMAP_END = FIX_CMAP_BEGIN + (FIX_N_COLOURS * NR_CPUS * 2), -#else FIX_CMAP_END = FIX_CMAP_BEGIN + (FIX_N_COLOURS * 2), -#endif #ifdef CONFIG_HIGHMEM /* reserved pte's for temporary kernel mappings */ FIX_KMAP_BEGIN = FIX_CMAP_END + 1, diff --git a/arch/mips/include/asm/fpu.h b/arch/mips/include/asm/fpu.h index 4d86b72750c7..a939574f8293 100644 --- a/arch/mips/include/asm/fpu.h +++ b/arch/mips/include/asm/fpu.h @@ -17,6 +17,7 @@ #include <asm/mipsregs.h> #include <asm/cpu.h> #include <asm/cpu-features.h> +#include <asm/fpu_emulator.h> #include <asm/hazards.h> #include <asm/processor.h> #include <asm/current.h> @@ -28,7 +29,6 @@ struct sigcontext; struct sigcontext32; -extern void fpu_emulator_init_fpu(void); extern void _init_fpu(void); extern void _save_fp(struct task_struct *); extern void _restore_fp(struct task_struct *); @@ -156,15 +156,16 @@ static inline int init_fpu(void) int ret = 0; preempt_disable(); + if (cpu_has_fpu) { ret = __own_fpu(); if (!ret) _init_fpu(); - } else { + } else fpu_emulator_init_fpu(); - } preempt_enable(); + return ret; } diff --git a/arch/mips/include/asm/fpu_emulator.h b/arch/mips/include/asm/fpu_emulator.h index 2abb587d5ab4..0195745b4b1b 100644 --- a/arch/mips/include/asm/fpu_emulator.h +++ b/arch/mips/include/asm/fpu_emulator.h @@ -23,9 +23,12 @@ #ifndef _ASM_FPU_EMULATOR_H #define _ASM_FPU_EMULATOR_H +#include <linux/sched.h> #include <asm/break.h> +#include <asm/thread_info.h> #include <asm/inst.h> #include <asm/local.h> +#include <asm/processor.h> #ifdef CONFIG_DEBUG_FS @@ -36,6 +39,11 @@ struct mips_fpu_emulator_stats { local_t cp1ops; local_t cp1xops; local_t errors; + local_t ieee754_inexact; + local_t ieee754_underflow; + local_t ieee754_overflow; + local_t ieee754_zerodiv; + local_t ieee754_invalidop; }; DECLARE_PER_CPU(struct mips_fpu_emulator_stats, fpuemustats); @@ -71,4 +79,17 @@ int mm_isBranchInstr(struct pt_regs *regs, struct mm_decoded_insn dec_insn, */ #define BREAK_MATH (0x0000000d | (BRK_MEMU << 16)) +#define SIGNALLING_NAN 0x7ff800007ff80000LL + +static inline void fpu_emulator_init_fpu(void) +{ + struct task_struct *t = current; + int i; + + t->thread.fpu.fcr31 = 0; + + for (i = 0; i < 32; i++) + set_fpr64(&t->thread.fpu.fpr[i], 0, SIGNALLING_NAN); +} + #endif /* _ASM_FPU_EMULATOR_H */ diff --git a/arch/mips/include/asm/gic.h b/arch/mips/include/asm/gic.h index 082716690589..10f6a99f92c2 100644 --- a/arch/mips/include/asm/gic.h +++ b/arch/mips/include/asm/gic.h @@ -380,6 +380,7 @@ extern unsigned int gic_compare_int (void); extern cycle_t gic_read_count(void); extern cycle_t gic_read_compare(void); extern void gic_write_compare(cycle_t cnt); +extern void gic_write_cpu_compare(cycle_t cnt, int cpu); extern void gic_send_ipi(unsigned int intr); extern unsigned int plat_ipi_call_int_xlate(unsigned int); extern unsigned int plat_ipi_resched_int_xlate(unsigned int); diff --git a/arch/mips/include/asm/gio_device.h b/arch/mips/include/asm/gio_device.h index 0878701712f8..4be1a57cdbb0 100644 --- a/arch/mips/include/asm/gio_device.h +++ b/arch/mips/include/asm/gio_device.h @@ -50,7 +50,7 @@ static inline void gio_device_free(struct gio_device *dev) extern int gio_register_driver(struct gio_driver *); extern void gio_unregister_driver(struct gio_driver *); -#define gio_get_drvdata(_dev) drv_get_drvdata(&(_dev)->dev) -#define gio_set_drvdata(_dev, data) drv_set_drvdata(&(_dev)->dev, (data)) +#define gio_get_drvdata(_dev) dev_get_drvdata(&(_dev)->dev) +#define gio_set_drvdata(_dev, data) dev_set_drvdata(&(_dev)->dev, (data)) extern void gio_set_master(struct gio_device *); diff --git a/arch/mips/include/asm/idle.h b/arch/mips/include/asm/idle.h index d192158886b1..d9f932de80e9 100644 --- a/arch/mips/include/asm/idle.h +++ b/arch/mips/include/asm/idle.h @@ -1,6 +1,7 @@ #ifndef __ASM_IDLE_H #define __ASM_IDLE_H +#include <linux/cpuidle.h> #include <linux/linkage.h> extern void (*cpu_wait)(void); @@ -20,4 +21,17 @@ static inline int address_is_in_r4k_wait_irqoff(unsigned long addr) addr < (unsigned long)__pastwait; } +extern int mips_cpuidle_wait_enter(struct cpuidle_device *dev, + struct cpuidle_driver *drv, int index); + +#define MIPS_CPUIDLE_WAIT_STATE {\ + .enter = mips_cpuidle_wait_enter,\ + .exit_latency = 1,\ + .target_residency = 1,\ + .power_usage = UINT_MAX,\ + .flags = CPUIDLE_FLAG_TIME_VALID,\ + .name = "wait",\ + .desc = "MIPS wait",\ +} + #endif /* __ASM_IDLE_H */ diff --git a/arch/mips/include/asm/irq.h b/arch/mips/include/asm/irq.h index 7bc2cdb35057..ae1f7b24dd1a 100644 --- a/arch/mips/include/asm/irq.h +++ b/arch/mips/include/asm/irq.h @@ -26,104 +26,8 @@ static inline int irq_canonicalize(int irq) #define irq_canonicalize(irq) (irq) /* Sane hardware, sane code ... */ #endif -#ifdef CONFIG_MIPS_MT_SMTC - -struct irqaction; - -extern unsigned long irq_hwmask[]; -extern int setup_irq_smtc(unsigned int irq, struct irqaction * new, - unsigned long hwmask); - -static inline void smtc_im_ack_irq(unsigned int irq) -{ - if (irq_hwmask[irq] & ST0_IM) - set_c0_status(irq_hwmask[irq] & ST0_IM); -} - -#else - -static inline void smtc_im_ack_irq(unsigned int irq) -{ -} - -#endif /* CONFIG_MIPS_MT_SMTC */ - -#ifdef CONFIG_MIPS_MT_SMTC_IRQAFF -#include <linux/cpumask.h> - -extern int plat_set_irq_affinity(struct irq_data *d, - const struct cpumask *affinity, bool force); -extern void smtc_forward_irq(struct irq_data *d); - -/* - * IRQ affinity hook invoked at the beginning of interrupt dispatch - * if option is enabled. - * - * Up through Linux 2.6.22 (at least) cpumask operations are very - * inefficient on MIPS. Initial prototypes of SMTC IRQ affinity - * used a "fast path" per-IRQ-descriptor cache of affinity information - * to reduce latency. As there is a project afoot to optimize the - * cpumask implementations, this version is optimistically assuming - * that cpumask.h macro overhead is reasonable during interrupt dispatch. - */ -static inline int handle_on_other_cpu(unsigned int irq) -{ - struct irq_data *d = irq_get_irq_data(irq); - - if (cpumask_test_cpu(smp_processor_id(), d->affinity)) - return 0; - smtc_forward_irq(d); - return 1; -} - -#else /* Not doing SMTC affinity */ - -static inline int handle_on_other_cpu(unsigned int irq) { return 0; } - -#endif /* CONFIG_MIPS_MT_SMTC_IRQAFF */ - -#ifdef CONFIG_MIPS_MT_SMTC_IM_BACKSTOP - -static inline void smtc_im_backstop(unsigned int irq) -{ - if (irq_hwmask[irq] & 0x0000ff00) - write_c0_tccontext(read_c0_tccontext() & - ~(irq_hwmask[irq] & 0x0000ff00)); -} - -/* - * Clear interrupt mask handling "backstop" if irq_hwmask - * entry so indicates. This implies that the ack() or end() - * functions will take over re-enabling the low-level mask. - * Otherwise it will be done on return from exception. - */ -static inline int smtc_handle_on_other_cpu(unsigned int irq) -{ - int ret = handle_on_other_cpu(irq); - - if (!ret) - smtc_im_backstop(irq); - return ret; -} - -#else - -static inline void smtc_im_backstop(unsigned int irq) { } -static inline int smtc_handle_on_other_cpu(unsigned int irq) -{ - return handle_on_other_cpu(irq); -} - -#endif - extern void do_IRQ(unsigned int irq); -#ifdef CONFIG_MIPS_MT_SMTC_IRQAFF - -extern void do_IRQ_no_affinity(unsigned int irq); - -#endif /* CONFIG_MIPS_MT_SMTC_IRQAFF */ - extern void arch_init_irq(void); extern void spurious_interrupt(void); diff --git a/arch/mips/include/asm/irqflags.h b/arch/mips/include/asm/irqflags.h index 45c00951888b..0fa5fdcd1f01 100644 --- a/arch/mips/include/asm/irqflags.h +++ b/arch/mips/include/asm/irqflags.h @@ -17,7 +17,7 @@ #include <linux/stringify.h> #include <asm/hazards.h> -#if defined(CONFIG_CPU_MIPSR2) && !defined(CONFIG_MIPS_MT_SMTC) +#ifdef CONFIG_CPU_MIPSR2 static inline void arch_local_irq_disable(void) { @@ -118,30 +118,15 @@ void arch_local_irq_disable(void); unsigned long arch_local_irq_save(void); void arch_local_irq_restore(unsigned long flags); void __arch_local_irq_restore(unsigned long flags); -#endif /* if defined(CONFIG_CPU_MIPSR2) && !defined(CONFIG_MIPS_MT_SMTC) */ - - -extern void smtc_ipi_replay(void); +#endif /* CONFIG_CPU_MIPSR2 */ static inline void arch_local_irq_enable(void) { -#ifdef CONFIG_MIPS_MT_SMTC - /* - * SMTC kernel needs to do a software replay of queued - * IPIs, at the cost of call overhead on each local_irq_enable() - */ - smtc_ipi_replay(); -#endif __asm__ __volatile__( " .set push \n" " .set reorder \n" " .set noat \n" -#ifdef CONFIG_MIPS_MT_SMTC - " mfc0 $1, $2, 1 # SMTC - clear TCStatus.IXMT \n" - " ori $1, 0x400 \n" - " xori $1, 0x400 \n" - " mtc0 $1, $2, 1 \n" -#elif defined(CONFIG_CPU_MIPSR2) +#if defined(CONFIG_CPU_MIPSR2) " ei \n" #else " mfc0 $1,$12 \n" @@ -163,11 +148,7 @@ static inline unsigned long arch_local_save_flags(void) asm __volatile__( " .set push \n" " .set reorder \n" -#ifdef CONFIG_MIPS_MT_SMTC - " mfc0 %[flags], $2, 1 \n" -#else " mfc0 %[flags], $12 \n" -#endif " .set pop \n" : [flags] "=r" (flags)); @@ -177,14 +158,7 @@ static inline unsigned long arch_local_save_flags(void) static inline int arch_irqs_disabled_flags(unsigned long flags) { -#ifdef CONFIG_MIPS_MT_SMTC - /* - * SMTC model uses TCStatus.IXMT to disable interrupts for a thread/CPU - */ - return flags & 0x400; -#else return !(flags & 1); -#endif } #endif /* #ifndef __ASSEMBLY__ */ diff --git a/arch/mips/include/asm/kvm_para.h b/arch/mips/include/asm/kvm_para.h new file mode 100644 index 000000000000..5a9aa918abe6 --- /dev/null +++ b/arch/mips/include/asm/kvm_para.h @@ -0,0 +1,109 @@ +#ifndef _ASM_MIPS_KVM_PARA_H +#define _ASM_MIPS_KVM_PARA_H + +#include <uapi/asm/kvm_para.h> + +#define KVM_HYPERCALL ".word 0x42000028" + +/* + * Hypercalls for KVM. + * + * Hypercall number is passed in v0. + * Return value will be placed in v0. + * Up to 3 arguments are passed in a0, a1, and a2. + */ +static inline unsigned long kvm_hypercall0(unsigned long num) +{ + register unsigned long n asm("v0"); + register unsigned long r asm("v0"); + + n = num; + __asm__ __volatile__( + KVM_HYPERCALL + : "=r" (r) : "r" (n) : "memory" + ); + + return r; +} + +static inline unsigned long kvm_hypercall1(unsigned long num, + unsigned long arg0) +{ + register unsigned long n asm("v0"); + register unsigned long r asm("v0"); + register unsigned long a0 asm("a0"); + + n = num; + a0 = arg0; + __asm__ __volatile__( + KVM_HYPERCALL + : "=r" (r) : "r" (n), "r" (a0) : "memory" + ); + + return r; +} + +static inline unsigned long kvm_hypercall2(unsigned long num, + unsigned long arg0, unsigned long arg1) +{ + register unsigned long n asm("v0"); + register unsigned long r asm("v0"); + register unsigned long a0 asm("a0"); + register unsigned long a1 asm("a1"); + + n = num; + a0 = arg0; + a1 = arg1; + __asm__ __volatile__( + KVM_HYPERCALL + : "=r" (r) : "r" (n), "r" (a0), "r" (a1) : "memory" + ); + + return r; +} + +static inline unsigned long kvm_hypercall3(unsigned long num, + unsigned long arg0, unsigned long arg1, unsigned long arg2) +{ + register unsigned long n asm("v0"); + register unsigned long r asm("v0"); + register unsigned long a0 asm("a0"); + register unsigned long a1 asm("a1"); + register unsigned long a2 asm("a2"); + + n = num; + a0 = arg0; + a1 = arg1; + a2 = arg2; + __asm__ __volatile__( + KVM_HYPERCALL + : "=r" (r) : "r" (n), "r" (a0), "r" (a1), "r" (a2) : "memory" + ); + + return r; +} + +static inline bool kvm_check_and_clear_guest_paused(void) +{ + return false; +} + +static inline unsigned int kvm_arch_para_features(void) +{ + return 0; +} + +#ifdef CONFIG_MIPS_PARAVIRT +static inline bool kvm_para_available(void) +{ + return true; +} +#else +static inline bool kvm_para_available(void) +{ + return false; +} +#endif + + +#endif /* _ASM_MIPS_KVM_PARA_H */ diff --git a/arch/mips/include/asm/mach-cavium-octeon/cpu-feature-overrides.h b/arch/mips/include/asm/mach-cavium-octeon/cpu-feature-overrides.h index 94ed063eec92..cf8022872892 100644 --- a/arch/mips/include/asm/mach-cavium-octeon/cpu-feature-overrides.h +++ b/arch/mips/include/asm/mach-cavium-octeon/cpu-feature-overrides.h @@ -22,7 +22,6 @@ #define cpu_has_3k_cache 0 #define cpu_has_4k_cache 0 #define cpu_has_tx39_cache 0 -#define cpu_has_fpu 0 #define cpu_has_counter 1 #define cpu_has_watch 1 #define cpu_has_divec 1 diff --git a/arch/mips/include/asm/mach-cavium-octeon/irq.h b/arch/mips/include/asm/mach-cavium-octeon/irq.h index 60fc4c347c44..cceae32a0732 100644 --- a/arch/mips/include/asm/mach-cavium-octeon/irq.h +++ b/arch/mips/include/asm/mach-cavium-octeon/irq.h @@ -35,6 +35,8 @@ enum octeon_irq { OCTEON_IRQ_PCI_MSI2, OCTEON_IRQ_PCI_MSI3, + OCTEON_IRQ_TWSI, + OCTEON_IRQ_TWSI2, OCTEON_IRQ_RML, OCTEON_IRQ_TIMER0, OCTEON_IRQ_TIMER1, diff --git a/arch/mips/include/asm/mach-ip22/cpu-feature-overrides.h b/arch/mips/include/asm/mach-ip22/cpu-feature-overrides.h index 1bcb6421205e..1dfe47453ea4 100644 --- a/arch/mips/include/asm/mach-ip22/cpu-feature-overrides.h +++ b/arch/mips/include/asm/mach-ip22/cpu-feature-overrides.h @@ -39,6 +39,10 @@ #define cpu_has_nofpuex 0 #define cpu_has_64bits 1 +#define cpu_has_mips_2 1 +#define cpu_has_mips_3 1 +#define cpu_has_mips_5 0 + #define cpu_has_mips32r1 0 #define cpu_has_mips32r2 0 #define cpu_has_mips64r1 0 diff --git a/arch/mips/include/asm/mach-malta/kernel-entry-init.h b/arch/mips/include/asm/mach-malta/kernel-entry-init.h index 7c5e17a17849..77eeda77e73c 100644 --- a/arch/mips/include/asm/mach-malta/kernel-entry-init.h +++ b/arch/mips/include/asm/mach-malta/kernel-entry-init.h @@ -80,36 +80,6 @@ .endm .macro kernel_entry_setup -#ifdef CONFIG_MIPS_MT_SMTC - mfc0 t0, CP0_CONFIG - bgez t0, 9f - mfc0 t0, CP0_CONFIG, 1 - bgez t0, 9f - mfc0 t0, CP0_CONFIG, 2 - bgez t0, 9f - mfc0 t0, CP0_CONFIG, 3 - and t0, 1<<2 - bnez t0, 0f -9: - /* Assume we came from YAMON... */ - PTR_LA v0, 0x9fc00534 /* YAMON print */ - lw v0, (v0) - move a0, zero - PTR_LA a1, nonmt_processor - jal v0 - - PTR_LA v0, 0x9fc00520 /* YAMON exit */ - lw v0, (v0) - li a0, 1 - jal v0 - -1: b 1b - - __INITDATA -nonmt_processor: - .asciz "SMTC kernel requires the MT ASE to run\n" - __FINIT -#endif #ifdef CONFIG_EVA sync diff --git a/arch/mips/include/asm/mach-malta/malta-pm.h b/arch/mips/include/asm/mach-malta/malta-pm.h new file mode 100644 index 000000000000..c2c2e201013d --- /dev/null +++ b/arch/mips/include/asm/mach-malta/malta-pm.h @@ -0,0 +1,37 @@ +/* + * Copyright (C) 2014 Imagination Technologies + * Author: Paul Burton <paul.burton@imgtec.com> + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the + * Free Software Foundation; either version 2 of the License, or (at your + * option) any later version. + */ + +#ifndef __ASM_MIPS_MACH_MALTA_PM_H__ +#define __ASM_MIPS_MACH_MALTA_PM_H__ + +#include <asm/mips-boards/piix4.h> + +#ifdef CONFIG_MIPS_MALTA_PM + +/** + * mips_pm_suspend - enter a suspend state + * @state: the state to enter, one of PIIX4_FUNC3IO_PMCNTRL_SUS_TYP_* + * + * Enters a suspend state via the Malta's PIIX4. If the state to be entered + * is one which loses context (eg. SOFF) then this function will never + * return. + */ +extern int mips_pm_suspend(unsigned state); + +#else /* !CONFIG_MIPS_MALTA_PM */ + +static inline int mips_pm_suspend(unsigned state) +{ + return -EINVAL; +} + +#endif /* !CONFIG_MIPS_MALTA_PM */ + +#endif /* __ASM_MIPS_MACH_MALTA_PM_H__ */ diff --git a/arch/mips/include/asm/mach-netlogic/topology.h b/arch/mips/include/asm/mach-netlogic/topology.h index 0da99fa11c38..ceeb1f5e7129 100644 --- a/arch/mips/include/asm/mach-netlogic/topology.h +++ b/arch/mips/include/asm/mach-netlogic/topology.h @@ -10,10 +10,12 @@ #include <asm/mach-netlogic/multi-node.h> +#ifdef CONFIG_SMP #define topology_physical_package_id(cpu) cpu_to_node(cpu) #define topology_core_id(cpu) (cpu_logical_map(cpu) / NLM_THREADS_PER_CORE) #define topology_thread_cpumask(cpu) (&cpu_sibling_map[cpu]) #define topology_core_cpumask(cpu) cpumask_of_node(cpu_to_node(cpu)) +#endif #include <asm-generic/topology.h> diff --git a/arch/mips/include/asm/mach-paravirt/cpu-feature-overrides.h b/arch/mips/include/asm/mach-paravirt/cpu-feature-overrides.h new file mode 100644 index 000000000000..725e1ed83f6a --- /dev/null +++ b/arch/mips/include/asm/mach-paravirt/cpu-feature-overrides.h @@ -0,0 +1,36 @@ +/* + * This file is subject to the terms and conditions of the GNU General Public + * License. See the file "COPYING" in the main directory of this archive + * for more details. + * + * Copyright (C) 2013 Cavium, Inc. + */ +#ifndef __ASM_MACH_PARAVIRT_CPU_FEATURE_OVERRIDES_H +#define __ASM_MACH_PARAVIRT_CPU_FEATURE_OVERRIDES_H + +#define cpu_has_4kex 1 +#define cpu_has_3k_cache 0 +#define cpu_has_tx39_cache 0 +#define cpu_has_counter 1 +#define cpu_has_llsc 1 +/* + * We Disable LL/SC on non SMP systems as it is faster to disable + * interrupts for atomic access than a LL/SC. + */ +#ifdef CONFIG_SMP +# define kernel_uses_llsc 1 +#else +# define kernel_uses_llsc 0 +#endif + +#ifdef CONFIG_CPU_CAVIUM_OCTEON +#define cpu_dcache_line_size() 128 +#define cpu_icache_line_size() 128 +#define cpu_has_octeon_cache 1 +#define cpu_has_4k_cache 0 +#else +#define cpu_has_octeon_cache 0 +#define cpu_has_4k_cache 1 +#endif + +#endif /* __ASM_MACH_PARAVIRT_CPU_FEATURE_OVERRIDES_H */ diff --git a/arch/mips/include/asm/mach-paravirt/irq.h b/arch/mips/include/asm/mach-paravirt/irq.h new file mode 100644 index 000000000000..9b4d35eca977 --- /dev/null +++ b/arch/mips/include/asm/mach-paravirt/irq.h @@ -0,0 +1,19 @@ +/* + * This file is subject to the terms and conditions of the GNU General Public + * License. See the file "COPYING" in the main directory of this archive + * for more details. + * + * Copyright (C) 2013 Cavium, Inc. + */ +#ifndef __ASM_MACH_PARAVIRT_IRQ_H__ +#define __ASM_MACH_PARAVIRT_IRQ_H__ + +#define NR_IRQS 64 +#define MIPS_CPU_IRQ_BASE 1 + +#define MIPS_IRQ_PCIA (MIPS_CPU_IRQ_BASE + 8) + +#define MIPS_IRQ_MBOX0 (MIPS_CPU_IRQ_BASE + 32) +#define MIPS_IRQ_MBOX1 (MIPS_CPU_IRQ_BASE + 33) + +#endif /* __ASM_MACH_PARAVIRT_IRQ_H__ */ diff --git a/arch/mips/include/asm/mach-paravirt/kernel-entry-init.h b/arch/mips/include/asm/mach-paravirt/kernel-entry-init.h new file mode 100644 index 000000000000..2f82bfa3a773 --- /dev/null +++ b/arch/mips/include/asm/mach-paravirt/kernel-entry-init.h @@ -0,0 +1,50 @@ +/* + * This file is subject to the terms and conditions of the GNU General Public + * License. See the file "COPYING" in the main directory of this archive + * for more details. + * + * Copyright (C) 2013 Cavium, Inc + */ +#ifndef __ASM_MACH_PARAVIRT_KERNEL_ENTRY_H +#define __ASM_MACH_PARAVIRT_KERNEL_ENTRY_H + +#define CP0_EBASE $15, 1 + + .macro kernel_entry_setup + mfc0 t0, CP0_EBASE + andi t0, t0, 0x3ff # CPUNum + beqz t0, 1f + # CPUs other than zero goto smp_bootstrap + j smp_bootstrap + +1: + .endm + +/* + * Do SMP slave processor setup necessary before we can safely execute + * C code. + */ + .macro smp_slave_setup + mfc0 t0, CP0_EBASE + andi t0, t0, 0x3ff # CPUNum + slti t1, t0, NR_CPUS + bnez t1, 1f +2: + di + wait + b 2b # Unknown CPU, loop forever. +1: + PTR_LA t1, paravirt_smp_sp + PTR_SLL t0, PTR_SCALESHIFT + PTR_ADDU t1, t1, t0 +3: + PTR_L sp, 0(t1) + beqz sp, 3b # Spin until told to proceed. + + PTR_LA t1, paravirt_smp_gp + PTR_ADDU t1, t1, t0 + sync + PTR_L gp, 0(t1) + .endm + +#endif /* __ASM_MACH_PARAVIRT_KERNEL_ENTRY_H */ diff --git a/arch/mips/include/asm/mach-paravirt/war.h b/arch/mips/include/asm/mach-paravirt/war.h new file mode 100644 index 000000000000..36d3afb98451 --- /dev/null +++ b/arch/mips/include/asm/mach-paravirt/war.h @@ -0,0 +1,25 @@ +/* + * This file is subject to the terms and conditions of the GNU General Public + * License. See the file "COPYING" in the main directory of this archive + * for more details. + * + * Copyright (C) 2002, 2004, 2007 by Ralf Baechle <ralf@linux-mips.org> + * Copyright (C) 2013 Cavium Networks <support@caviumnetworks.com> + */ +#ifndef __ASM_MIPS_MACH_PARAVIRT_WAR_H +#define __ASM_MIPS_MACH_PARAVIRT_WAR_H + +#define R4600_V1_INDEX_ICACHEOP_WAR 0 +#define R4600_V1_HIT_CACHEOP_WAR 0 +#define R4600_V2_HIT_CACHEOP_WAR 0 +#define R5432_CP0_INTERRUPT_WAR 0 +#define BCM1250_M3_WAR 0 +#define SIBYTE_1956_WAR 0 +#define MIPS4K_ICACHE_REFILL_WAR 0 +#define MIPS_CACHE_SYNC_WAR 0 +#define TX49XX_ICACHE_INDEX_INV_WAR 0 +#define ICACHE_REFILLS_WORKAROUND_WAR 0 +#define R10000_LLSC_WAR 0 +#define MIPS34K_MISSED_ITLB_WAR 0 + +#endif /* __ASM_MIPS_MACH_PARAVIRT_WAR_H */ diff --git a/arch/mips/include/asm/mach-pmcs-msp71xx/msp_usb.h b/arch/mips/include/asm/mach-pmcs-msp71xx/msp_usb.h index aa45e6a07126..fe1566f2913e 100644 --- a/arch/mips/include/asm/mach-pmcs-msp71xx/msp_usb.h +++ b/arch/mips/include/asm/mach-pmcs-msp71xx/msp_usb.h @@ -25,11 +25,7 @@ #ifndef MSP_USB_H_ #define MSP_USB_H_ -#ifdef CONFIG_MSP_HAS_DUAL_USB -#define NUM_USB_DEVS 2 -#else #define NUM_USB_DEVS 1 -#endif /* Register spaces for USB host 0 */ #define MSP_USB0_MAB_START (MSP_USB0_BASE + 0x0) diff --git a/arch/mips/include/asm/mach-ralink/war.h b/arch/mips/include/asm/mach-ralink/war.h index a7b712cf2d28..c074b5dc1f82 100644 --- a/arch/mips/include/asm/mach-ralink/war.h +++ b/arch/mips/include/asm/mach-ralink/war.h @@ -17,7 +17,6 @@ #define MIPS4K_ICACHE_REFILL_WAR 0 #define MIPS_CACHE_SYNC_WAR 0 #define TX49XX_ICACHE_INDEX_INV_WAR 0 -#define RM9000_CDEX_SMP_WAR 0 #define ICACHE_REFILLS_WORKAROUND_WAR 0 #define R10000_LLSC_WAR 0 #define MIPS34K_MISSED_ITLB_WAR 0 diff --git a/arch/mips/include/asm/mach-sead3/kernel-entry-init.h b/arch/mips/include/asm/mach-sead3/kernel-entry-init.h index 3dfbd8e7947f..6cccd4d558d7 100644 --- a/arch/mips/include/asm/mach-sead3/kernel-entry-init.h +++ b/arch/mips/include/asm/mach-sead3/kernel-entry-init.h @@ -10,37 +10,6 @@ #define __ASM_MACH_MIPS_KERNEL_ENTRY_INIT_H .macro kernel_entry_setup -#ifdef CONFIG_MIPS_MT_SMTC - mfc0 t0, CP0_CONFIG - bgez t0, 9f - mfc0 t0, CP0_CONFIG, 1 - bgez t0, 9f - mfc0 t0, CP0_CONFIG, 2 - bgez t0, 9f - mfc0 t0, CP0_CONFIG, 3 - and t0, 1<<2 - bnez t0, 0f -9 : - /* Assume we came from YAMON... */ - PTR_LA v0, 0x9fc00534 /* YAMON print */ - lw v0, (v0) - move a0, zero - PTR_LA a1, nonmt_processor - jal v0 - - PTR_LA v0, 0x9fc00520 /* YAMON exit */ - lw v0, (v0) - li a0, 1 - jal v0 - -1 : b 1b - - __INITDATA -nonmt_processor : - .asciz "SMTC kernel requires the MT ASE to run\n" - __FINIT -0 : -#endif .endm /* diff --git a/arch/mips/include/asm/mips-boards/piix4.h b/arch/mips/include/asm/mips-boards/piix4.h index 9cf54041d416..9e340be52a50 100644 --- a/arch/mips/include/asm/mips-boards/piix4.h +++ b/arch/mips/include/asm/mips-boards/piix4.h @@ -55,4 +55,16 @@ #define PIIX4_FUNC3_PMREGMISC 0x80 #define PIIX4_FUNC3_PMREGMISC_EN (1 << 0) +/* Power Management IO Space */ +#define PIIX4_FUNC3IO_PMSTS 0x00 +#define PIIX4_FUNC3IO_PMSTS_PWRBTN_STS (1 << 8) +#define PIIX4_FUNC3IO_PMCNTRL 0x04 +#define PIIX4_FUNC3IO_PMCNTRL_SUS_EN (1 << 13) +#define PIIX4_FUNC3IO_PMCNTRL_SUS_TYP (0x7 << 10) +#define PIIX4_FUNC3IO_PMCNTRL_SUS_TYP_SOFF (0x0 << 10) +#define PIIX4_FUNC3IO_PMCNTRL_SUS_TYP_STR (0x1 << 10) + +/* Data for magic special PCI cycle */ +#define PIIX4_SUSPEND_MAGIC 0x00120002 + #endif /* __ASM_MIPS_BOARDS_PIIX4_H */ diff --git a/arch/mips/include/asm/mips-cpc.h b/arch/mips/include/asm/mips-cpc.h index 988507e46d42..e139a534e0fd 100644 --- a/arch/mips/include/asm/mips-cpc.h +++ b/arch/mips/include/asm/mips-cpc.h @@ -72,7 +72,12 @@ static inline bool mips_cpc_present(void) #define MIPS_CPC_COCB_OFS 0x4000 /* Macros to ease the creation of register access functions */ -#define BUILD_CPC_R_(name, off) \ +#define BUILD_CPC_R_(name, off) \ +static inline u32 *addr_cpc_##name(void) \ +{ \ + return (u32 *)(mips_cpc_base + (off)); \ +} \ + \ static inline u32 read_cpc_##name(void) \ { \ return __raw_readl(mips_cpc_base + (off)); \ @@ -147,4 +152,31 @@ BUILD_CPC_Cx_RW(other, 0x10) #define CPC_Cx_OTHER_CORENUM_SHF 16 #define CPC_Cx_OTHER_CORENUM_MSK (_ULCAST_(0xff) << 16) +#ifdef CONFIG_MIPS_CPC + +/** + * mips_cpc_lock_other - lock access to another core + * core: the other core to be accessed + * + * Call before operating upon a core via the 'other' register region in + * order to prevent the region being moved during access. Must be followed + * by a call to mips_cpc_unlock_other. + */ +extern void mips_cpc_lock_other(unsigned int core); + +/** + * mips_cpc_unlock_other - unlock access to another core + * + * Call after operating upon another core via the 'other' register region. + * Must be called after mips_cpc_lock_other. + */ +extern void mips_cpc_unlock_other(void); + +#else /* !CONFIG_MIPS_CPC */ + +static inline void mips_cpc_lock_other(unsigned int core) { } +static inline void mips_cpc_unlock_other(void) { } + +#endif /* !CONFIG_MIPS_CPC */ + #endif /* __MIPS_ASM_MIPS_CPC_H__ */ diff --git a/arch/mips/include/asm/mips_mt.h b/arch/mips/include/asm/mips_mt.h index a3df0c3faa0e..f6ba004a7711 100644 --- a/arch/mips/include/asm/mips_mt.h +++ b/arch/mips/include/asm/mips_mt.h @@ -1,7 +1,6 @@ /* - * Definitions and decalrations for MIPS MT support - * that are common between SMTC, VSMP, and/or AP/SP - * kernel models. + * Definitions and decalrations for MIPS MT support that are common between + * the VSMP, and AP/SP kernel models. */ #ifndef __ASM_MIPS_MT_H #define __ASM_MIPS_MT_H diff --git a/arch/mips/include/asm/mipsmtregs.h b/arch/mips/include/asm/mipsmtregs.h index 6efa79a27b6a..5f8052ce43bf 100644 --- a/arch/mips/include/asm/mipsmtregs.h +++ b/arch/mips/include/asm/mipsmtregs.h @@ -36,6 +36,8 @@ #define read_c0_tcbind() __read_32bit_c0_register($2, 2) +#define write_c0_tchalt(val) __write_32bit_c0_register($2, 4, val) + #define read_c0_tccontext() __read_32bit_c0_register($2, 5) #define write_c0_tccontext(val) __write_32bit_c0_register($2, 5, val) diff --git a/arch/mips/include/asm/mipsregs.h b/arch/mips/include/asm/mipsregs.h index 3e025b5311db..98e9754a4b6b 100644 --- a/arch/mips/include/asm/mipsregs.h +++ b/arch/mips/include/asm/mipsregs.h @@ -709,11 +709,18 @@ #ifndef __ASSEMBLY__ /* - * Macros for handling the ISA mode bit for microMIPS. + * Macros for handling the ISA mode bit for MIPS16 and microMIPS. */ +#if defined(CONFIG_SYS_SUPPORTS_MIPS16) || \ + defined(CONFIG_SYS_SUPPORTS_MICROMIPS) #define get_isa16_mode(x) ((x) & 0x1) #define msk_isa16_mode(x) ((x) & ~0x1) #define set_isa16_mode(x) do { (x) |= 0x1; } while(0) +#else +#define get_isa16_mode(x) 0 +#define msk_isa16_mode(x) (x) +#define set_isa16_mode(x) do { } while(0) +#endif /* * microMIPS instructions can be 16-bit or 32-bit in length. This @@ -1007,19 +1014,8 @@ do { \ #define write_c0_compare3(val) __write_32bit_c0_register($11, 7, val) #define read_c0_status() __read_32bit_c0_register($12, 0) -#ifdef CONFIG_MIPS_MT_SMTC -#define write_c0_status(val) \ -do { \ - __write_32bit_c0_register($12, 0, val); \ - __ehb(); \ -} while (0) -#else -/* - * Legacy non-SMTC code, which may be hazardous - * but which might not support EHB - */ + #define write_c0_status(val) __write_32bit_c0_register($12, 0, val) -#endif /* CONFIG_MIPS_MT_SMTC */ #define read_c0_cause() __read_32bit_c0_register($13, 0) #define write_c0_cause(val) __write_32bit_c0_register($13, 0, val) @@ -1743,11 +1739,6 @@ static inline void tlb_write_random(void) /* * Manipulate bits in a c0 register. */ -#ifndef CONFIG_MIPS_MT_SMTC -/* - * SMTC Linux requires shutting-down microthread scheduling - * during CP0 register read-modify-write sequences. - */ #define __BUILD_SET_C0(name) \ static inline unsigned int \ set_c0_##name(unsigned int set) \ @@ -1786,121 +1777,6 @@ change_c0_##name(unsigned int change, unsigned int val) \ return res; \ } -#else /* SMTC versions that manage MT scheduling */ - -#include <linux/irqflags.h> - -/* - * This is a duplicate of dmt() in mipsmtregs.h to avoid problems with - * header file recursion. - */ -static inline unsigned int __dmt(void) -{ - int res; - - __asm__ __volatile__( - " .set push \n" - " .set mips32r2 \n" - " .set noat \n" - " .word 0x41610BC1 # dmt $1 \n" - " ehb \n" - " move %0, $1 \n" - " .set pop \n" - : "=r" (res)); - - instruction_hazard(); - - return res; -} - -#define __VPECONTROL_TE_SHIFT 15 -#define __VPECONTROL_TE (1UL << __VPECONTROL_TE_SHIFT) - -#define __EMT_ENABLE __VPECONTROL_TE - -static inline void __emt(unsigned int previous) -{ - if ((previous & __EMT_ENABLE)) - __asm__ __volatile__( - " .set mips32r2 \n" - " .word 0x41600be1 # emt \n" - " ehb \n" - " .set mips0 \n"); -} - -static inline void __ehb(void) -{ - __asm__ __volatile__( - " .set mips32r2 \n" - " ehb \n" " .set mips0 \n"); -} - -/* - * Note that local_irq_save/restore affect TC-specific IXMT state, - * not Status.IE as in non-SMTC kernel. - */ - -#define __BUILD_SET_C0(name) \ -static inline unsigned int \ -set_c0_##name(unsigned int set) \ -{ \ - unsigned int res; \ - unsigned int new; \ - unsigned int omt; \ - unsigned long flags; \ - \ - local_irq_save(flags); \ - omt = __dmt(); \ - res = read_c0_##name(); \ - new = res | set; \ - write_c0_##name(new); \ - __emt(omt); \ - local_irq_restore(flags); \ - \ - return res; \ -} \ - \ -static inline unsigned int \ -clear_c0_##name(unsigned int clear) \ -{ \ - unsigned int res; \ - unsigned int new; \ - unsigned int omt; \ - unsigned long flags; \ - \ - local_irq_save(flags); \ - omt = __dmt(); \ - res = read_c0_##name(); \ - new = res & ~clear; \ - write_c0_##name(new); \ - __emt(omt); \ - local_irq_restore(flags); \ - \ - return res; \ -} \ - \ -static inline unsigned int \ -change_c0_##name(unsigned int change, unsigned int newbits) \ -{ \ - unsigned int res; \ - unsigned int new; \ - unsigned int omt; \ - unsigned long flags; \ - \ - local_irq_save(flags); \ - \ - omt = __dmt(); \ - res = read_c0_##name(); \ - new = res & ~change; \ - new |= (newbits & change); \ - write_c0_##name(new); \ - __emt(omt); \ - local_irq_restore(flags); \ - \ - return res; \ -} -#endif - __BUILD_SET_C0(status) __BUILD_SET_C0(cause) __BUILD_SET_C0(config) @@ -1916,6 +1792,15 @@ __BUILD_SET_C0(brcm_cmt_ctrl) __BUILD_SET_C0(brcm_config) __BUILD_SET_C0(brcm_mode) +/* + * Return low 10 bits of ebase. + * Note that under KVM (MIPSVZ) this returns vcpu id. + */ +static inline unsigned int get_ebase_cpunum(void) +{ + return read_c0_ebase() & 0x3ff; +} + #endif /* !__ASSEMBLY__ */ #endif /* _ASM_MIPSREGS_H */ diff --git a/arch/mips/include/asm/mmu_context.h b/arch/mips/include/asm/mmu_context.h index e277bbad2871..2e373da5f8e9 100644 --- a/arch/mips/include/asm/mmu_context.h +++ b/arch/mips/include/asm/mmu_context.h @@ -18,10 +18,6 @@ #include <asm/cacheflush.h> #include <asm/hazards.h> #include <asm/tlbflush.h> -#ifdef CONFIG_MIPS_MT_SMTC -#include <asm/mipsmtregs.h> -#include <asm/smtc.h> -#endif /* SMTC */ #include <asm-generic/mm_hooks.h> #define TLBMISS_HANDLER_SETUP_PGD(pgd) \ @@ -31,11 +27,15 @@ do { \ } while (0) #ifdef CONFIG_MIPS_PGD_C0_CONTEXT + +#define TLBMISS_HANDLER_RESTORE() \ + write_c0_xcontext((unsigned long) smp_processor_id() << \ + SMP_CPUID_REGSHIFT) + #define TLBMISS_HANDLER_SETUP() \ do { \ TLBMISS_HANDLER_SETUP_PGD(swapper_pg_dir); \ - write_c0_xcontext((unsigned long) smp_processor_id() << \ - SMP_CPUID_REGSHIFT); \ + TLBMISS_HANDLER_RESTORE(); \ } while (0) #else /* !CONFIG_MIPS_PGD_C0_CONTEXT: using pgd_current*/ @@ -47,9 +47,12 @@ do { \ */ extern unsigned long pgd_current[]; -#define TLBMISS_HANDLER_SETUP() \ +#define TLBMISS_HANDLER_RESTORE() \ write_c0_context((unsigned long) smp_processor_id() << \ - SMP_CPUID_REGSHIFT); \ + SMP_CPUID_REGSHIFT) + +#define TLBMISS_HANDLER_SETUP() \ + TLBMISS_HANDLER_RESTORE(); \ back_to_back_c0_hazard(); \ TLBMISS_HANDLER_SETUP_PGD(swapper_pg_dir) #endif /* CONFIG_MIPS_PGD_C0_CONTEXT*/ @@ -63,13 +66,6 @@ extern unsigned long pgd_current[]; #define ASID_INC 0x10 #define ASID_MASK 0xff0 -#elif defined(CONFIG_MIPS_MT_SMTC) - -#define ASID_INC 0x1 -extern unsigned long smtc_asid_mask; -#define ASID_MASK (smtc_asid_mask) -#define HW_ASID_MASK 0xff -/* End SMTC/34K debug hack */ #else /* FIXME: not correct for R6000 */ #define ASID_INC 0x1 @@ -92,7 +88,6 @@ static inline void enter_lazy_tlb(struct mm_struct *mm, struct task_struct *tsk) #define ASID_VERSION_MASK ((unsigned long)~(ASID_MASK|(ASID_MASK-1))) #define ASID_FIRST_VERSION ((unsigned long)(~ASID_VERSION_MASK) + 1) -#ifndef CONFIG_MIPS_MT_SMTC /* Normal, classic MIPS get_new_mmu_context */ static inline void get_new_mmu_context(struct mm_struct *mm, unsigned long cpu) @@ -115,12 +110,6 @@ get_new_mmu_context(struct mm_struct *mm, unsigned long cpu) cpu_context(cpu, mm) = asid_cache(cpu) = asid; } -#else /* CONFIG_MIPS_MT_SMTC */ - -#define get_new_mmu_context(mm, cpu) smtc_get_new_mmu_context((mm), (cpu)) - -#endif /* CONFIG_MIPS_MT_SMTC */ - /* * Initialize the context related info for a new mm_struct * instance. @@ -141,46 +130,12 @@ static inline void switch_mm(struct mm_struct *prev, struct mm_struct *next, { unsigned int cpu = smp_processor_id(); unsigned long flags; -#ifdef CONFIG_MIPS_MT_SMTC - unsigned long oldasid; - unsigned long mtflags; - int mytlb = (smtc_status & SMTC_TLB_SHARED) ? 0 : cpu_data[cpu].vpe_id; local_irq_save(flags); - mtflags = dvpe(); -#else /* Not SMTC */ - local_irq_save(flags); -#endif /* CONFIG_MIPS_MT_SMTC */ /* Check if our ASID is of an older version and thus invalid */ if ((cpu_context(cpu, next) ^ asid_cache(cpu)) & ASID_VERSION_MASK) get_new_mmu_context(next, cpu); -#ifdef CONFIG_MIPS_MT_SMTC - /* - * If the EntryHi ASID being replaced happens to be - * the value flagged at ASID recycling time as having - * an extended life, clear the bit showing it being - * in use by this "CPU", and if that's the last bit, - * free up the ASID value for use and flush any old - * instances of it from the TLB. - */ - oldasid = (read_c0_entryhi() & ASID_MASK); - if(smtc_live_asid[mytlb][oldasid]) { - smtc_live_asid[mytlb][oldasid] &= ~(0x1 << cpu); - if(smtc_live_asid[mytlb][oldasid] == 0) - smtc_flush_tlb_asid(oldasid); - } - /* - * Tread softly on EntryHi, and so long as we support - * having ASID_MASK smaller than the hardware maximum, - * make sure no "soft" bits become "hard"... - */ - write_c0_entryhi((read_c0_entryhi() & ~HW_ASID_MASK) | - cpu_asid(cpu, next)); - ehb(); /* Make sure it propagates to TCStatus */ - evpe(mtflags); -#else write_c0_entryhi(cpu_asid(cpu, next)); -#endif /* CONFIG_MIPS_MT_SMTC */ TLBMISS_HANDLER_SETUP_PGD(next->pgd); /* @@ -213,34 +168,12 @@ activate_mm(struct mm_struct *prev, struct mm_struct *next) unsigned long flags; unsigned int cpu = smp_processor_id(); -#ifdef CONFIG_MIPS_MT_SMTC - unsigned long oldasid; - unsigned long mtflags; - int mytlb = (smtc_status & SMTC_TLB_SHARED) ? 0 : cpu_data[cpu].vpe_id; -#endif /* CONFIG_MIPS_MT_SMTC */ - local_irq_save(flags); /* Unconditionally get a new ASID. */ get_new_mmu_context(next, cpu); -#ifdef CONFIG_MIPS_MT_SMTC - /* See comments for similar code above */ - mtflags = dvpe(); - oldasid = read_c0_entryhi() & ASID_MASK; - if(smtc_live_asid[mytlb][oldasid]) { - smtc_live_asid[mytlb][oldasid] &= ~(0x1 << cpu); - if(smtc_live_asid[mytlb][oldasid] == 0) - smtc_flush_tlb_asid(oldasid); - } - /* See comments for similar code above */ - write_c0_entryhi((read_c0_entryhi() & ~HW_ASID_MASK) | - cpu_asid(cpu, next)); - ehb(); /* Make sure it propagates to TCStatus */ - evpe(mtflags); -#else write_c0_entryhi(cpu_asid(cpu, next)); -#endif /* CONFIG_MIPS_MT_SMTC */ TLBMISS_HANDLER_SETUP_PGD(next->pgd); /* mark mmu ownership change */ @@ -258,48 +191,15 @@ static inline void drop_mmu_context(struct mm_struct *mm, unsigned cpu) { unsigned long flags; -#ifdef CONFIG_MIPS_MT_SMTC - unsigned long oldasid; - /* Can't use spinlock because called from TLB flush within DVPE */ - unsigned int prevvpe; - int mytlb = (smtc_status & SMTC_TLB_SHARED) ? 0 : cpu_data[cpu].vpe_id; -#endif /* CONFIG_MIPS_MT_SMTC */ local_irq_save(flags); if (cpumask_test_cpu(cpu, mm_cpumask(mm))) { get_new_mmu_context(mm, cpu); -#ifdef CONFIG_MIPS_MT_SMTC - /* See comments for similar code above */ - prevvpe = dvpe(); - oldasid = (read_c0_entryhi() & ASID_MASK); - if (smtc_live_asid[mytlb][oldasid]) { - smtc_live_asid[mytlb][oldasid] &= ~(0x1 << cpu); - if(smtc_live_asid[mytlb][oldasid] == 0) - smtc_flush_tlb_asid(oldasid); - } - /* See comments for similar code above */ - write_c0_entryhi((read_c0_entryhi() & ~HW_ASID_MASK) - | cpu_asid(cpu, mm)); - ehb(); /* Make sure it propagates to TCStatus */ - evpe(prevvpe); -#else /* not CONFIG_MIPS_MT_SMTC */ write_c0_entryhi(cpu_asid(cpu, mm)); -#endif /* CONFIG_MIPS_MT_SMTC */ } else { /* will get a new context next time */ -#ifndef CONFIG_MIPS_MT_SMTC cpu_context(cpu, mm) = 0; -#else /* SMTC */ - int i; - - /* SMTC shares the TLB (and ASIDs) across VPEs */ - for_each_online_cpu(i) { - if((smtc_status & SMTC_TLB_SHARED) - || (cpu_data[i].vpe_id == cpu_data[cpu].vpe_id)) - cpu_context(i, mm) = 0; - } -#endif /* CONFIG_MIPS_MT_SMTC */ } local_irq_restore(flags); } diff --git a/arch/mips/include/asm/module.h b/arch/mips/include/asm/module.h index c2edae382d5d..800fe578dc99 100644 --- a/arch/mips/include/asm/module.h +++ b/arch/mips/include/asm/module.h @@ -144,13 +144,7 @@ search_module_dbetables(unsigned long addr) #define MODULE_KERNEL_TYPE "64BIT " #endif -#ifdef CONFIG_MIPS_MT_SMTC -#define MODULE_KERNEL_SMTC "MT_SMTC " -#else -#define MODULE_KERNEL_SMTC "" -#endif - #define MODULE_ARCH_VERMAGIC \ - MODULE_PROC_FAMILY MODULE_KERNEL_TYPE MODULE_KERNEL_SMTC + MODULE_PROC_FAMILY MODULE_KERNEL_TYPE #endif /* _ASM_MODULE_H */ diff --git a/arch/mips/include/asm/msa.h b/arch/mips/include/asm/msa.h index a2aba6c3ec05..538f6d482db8 100644 --- a/arch/mips/include/asm/msa.h +++ b/arch/mips/include/asm/msa.h @@ -84,7 +84,7 @@ static inline void write_msa_##name(unsigned int val) \ __asm__ __volatile__( \ " .set push\n" \ " .set msa\n" \ - " cfcmsa $" #cs ", %0\n" \ + " ctcmsa $" #cs ", %0\n" \ " .set pop\n" \ : : "r"(val)); \ } @@ -96,6 +96,13 @@ static inline void write_msa_##name(unsigned int val) \ * allow compilation with toolchains that do not support MSA. Once all * toolchains in use support MSA these can be removed. */ +#ifdef CONFIG_CPU_MICROMIPS +#define CFC_MSA_INSN 0x587e0056 +#define CTC_MSA_INSN 0x583e0816 +#else +#define CFC_MSA_INSN 0x787e0059 +#define CTC_MSA_INSN 0x783e0819 +#endif #define __BUILD_MSA_CTL_REG(name, cs) \ static inline unsigned int read_msa_##name(void) \ @@ -104,7 +111,8 @@ static inline unsigned int read_msa_##name(void) \ __asm__ __volatile__( \ " .set push\n" \ " .set noat\n" \ - " .word 0x787e0059 | (" #cs " << 11)\n" \ + " .insn\n" \ + " .word #CFC_MSA_INSN | (" #cs " << 11)\n" \ " move %0, $1\n" \ " .set pop\n" \ : "=r"(reg)); \ @@ -117,7 +125,8 @@ static inline void write_msa_##name(unsigned int val) \ " .set push\n" \ " .set noat\n" \ " move $1, %0\n" \ - " .word 0x783e0819 | (" #cs " << 6)\n" \ + " .insn\n" \ + " .word #CTC_MSA_INSN | (" #cs " << 6)\n" \ " .set pop\n" \ : : "r"(val)); \ } diff --git a/arch/mips/include/asm/netlogic/mips-extns.h b/arch/mips/include/asm/netlogic/mips-extns.h index de9aada6f4c1..06f1f75bfa9b 100644 --- a/arch/mips/include/asm/netlogic/mips-extns.h +++ b/arch/mips/include/asm/netlogic/mips-extns.h @@ -146,9 +146,10 @@ static inline int hard_smp_processor_id(void) static inline int nlm_nodeid(void) { - uint32_t prid = read_c0_prid(); + uint32_t prid = read_c0_prid() & PRID_IMP_MASK; - if ((prid & 0xff00) == PRID_IMP_NETLOGIC_XLP9XX) + if ((prid == PRID_IMP_NETLOGIC_XLP9XX) || + (prid == PRID_IMP_NETLOGIC_XLP5XX)) return (__read_32bit_c0_register($15, 1) >> 7) & 0x7; else return (__read_32bit_c0_register($15, 1) >> 5) & 0x3; diff --git a/arch/mips/include/asm/netlogic/xlp-hal/iomap.h b/arch/mips/include/asm/netlogic/xlp-hal/iomap.h index 1f23dfaa7167..805bfd21f33e 100644 --- a/arch/mips/include/asm/netlogic/xlp-hal/iomap.h +++ b/arch/mips/include/asm/netlogic/xlp-hal/iomap.h @@ -74,6 +74,8 @@ #define XLP_IO_USB_OHCI2_OFFSET(node) XLP_HDR_OFFSET(node, 0, 2, 4) #define XLP_IO_USB_OHCI3_OFFSET(node) XLP_HDR_OFFSET(node, 0, 2, 5) +#define XLP_IO_SATA_OFFSET(node) XLP_HDR_OFFSET(node, 0, 3, 2) + /* XLP2xx has an updated USB block */ #define XLP2XX_IO_USB_OFFSET(node, i) XLP_HDR_OFFSET(node, 0, 4, i) #define XLP2XX_IO_USB_XHCI0_OFFSET(node) XLP_HDR_OFFSET(node, 0, 4, 1) @@ -103,13 +105,11 @@ #define XLP_IO_SYS_OFFSET(node) XLP_HDR_OFFSET(node, 0, 6, 5) #define XLP_IO_JTAG_OFFSET(node) XLP_HDR_OFFSET(node, 0, 6, 6) +/* Flash */ #define XLP_IO_NOR_OFFSET(node) XLP_HDR_OFFSET(node, 0, 7, 0) #define XLP_IO_NAND_OFFSET(node) XLP_HDR_OFFSET(node, 0, 7, 1) #define XLP_IO_SPI_OFFSET(node) XLP_HDR_OFFSET(node, 0, 7, 2) -/* SD flash */ -#define XLP_IO_SD_OFFSET(node) XLP_HDR_OFFSET(node, 0, 7, 3) -#define XLP_IO_MMC_OFFSET(node, slot) \ - ((XLP_IO_SD_OFFSET(node))+(slot*0x100)+XLP_IO_PCI_HDRSZ) +#define XLP_IO_MMC_OFFSET(node) XLP_HDR_OFFSET(node, 0, 7, 3) /* Things have changed drastically in XLP 9XX */ #define XLP9XX_HDR_OFFSET(n, d, f) \ @@ -120,6 +120,8 @@ #define XLP9XX_IO_UART_OFFSET(node) XLP9XX_HDR_OFFSET(node, 2, 2) #define XLP9XX_IO_SYS_OFFSET(node) XLP9XX_HDR_OFFSET(node, 6, 0) #define XLP9XX_IO_FUSE_OFFSET(node) XLP9XX_HDR_OFFSET(node, 6, 1) +#define XLP9XX_IO_CLOCK_OFFSET(node) XLP9XX_HDR_OFFSET(node, 6, 2) +#define XLP9XX_IO_POWER_OFFSET(node) XLP9XX_HDR_OFFSET(node, 6, 3) #define XLP9XX_IO_JTAG_OFFSET(node) XLP9XX_HDR_OFFSET(node, 6, 4) #define XLP9XX_IO_PCIE_OFFSET(node, i) XLP9XX_HDR_OFFSET(node, 1, i) @@ -135,11 +137,11 @@ /* XLP9XX on-chip SATA controller */ #define XLP9XX_IO_SATA_OFFSET(node) XLP9XX_HDR_OFFSET(node, 3, 2) +/* Flash */ #define XLP9XX_IO_NOR_OFFSET(node) XLP9XX_HDR_OFFSET(node, 7, 0) #define XLP9XX_IO_NAND_OFFSET(node) XLP9XX_HDR_OFFSET(node, 7, 1) #define XLP9XX_IO_SPI_OFFSET(node) XLP9XX_HDR_OFFSET(node, 7, 2) -/* SD flash */ -#define XLP9XX_IO_MMCSD_OFFSET(node) XLP9XX_HDR_OFFSET(node, 7, 3) +#define XLP9XX_IO_MMC_OFFSET(node) XLP9XX_HDR_OFFSET(node, 7, 3) /* PCI config header register id's */ #define XLP_PCI_CFGREG0 0x00 @@ -186,8 +188,10 @@ #define PCI_DEVICE_ID_NLM_NOR 0x1015 #define PCI_DEVICE_ID_NLM_NAND 0x1016 #define PCI_DEVICE_ID_NLM_MMC 0x1018 -#define PCI_DEVICE_ID_NLM_XHCI 0x101d +#define PCI_DEVICE_ID_NLM_SATA 0x101A +#define PCI_DEVICE_ID_NLM_XHCI 0x101D +#define PCI_DEVICE_ID_XLP9XX_MMC 0x9018 #define PCI_DEVICE_ID_XLP9XX_SATA 0x901A #define PCI_DEVICE_ID_XLP9XX_XHCI 0x901D diff --git a/arch/mips/include/asm/netlogic/xlp-hal/pcibus.h b/arch/mips/include/asm/netlogic/xlp-hal/pcibus.h index d4deb87ad069..91540f41e1e4 100644 --- a/arch/mips/include/asm/netlogic/xlp-hal/pcibus.h +++ b/arch/mips/include/asm/netlogic/xlp-hal/pcibus.h @@ -69,6 +69,20 @@ #define PCIE_9XX_BYTE_SWAP_IO_BASE 0x25e #define PCIE_9XX_BYTE_SWAP_IO_LIM 0x25f +#define PCIE_9XX_BRIDGE_MSIX_ADDR_BASE 0x264 +#define PCIE_9XX_BRIDGE_MSIX_ADDR_LIMIT 0x265 +#define PCIE_9XX_MSI_STATUS 0x283 +#define PCIE_9XX_MSI_EN 0x284 +/* 128 MSIX vectors available in 9xx */ +#define PCIE_9XX_MSIX_STATUS0 0x286 +#define PCIE_9XX_MSIX_STATUSX(n) (n + 0x286) +#define PCIE_9XX_MSIX_VEC 0x296 +#define PCIE_9XX_MSIX_VECX(n) (n + 0x296) +#define PCIE_9XX_INT_STATUS0 0x397 +#define PCIE_9XX_INT_STATUS1 0x398 +#define PCIE_9XX_INT_EN0 0x399 +#define PCIE_9XX_INT_EN1 0x39a + /* other */ #define PCIE_NLINKS 4 diff --git a/arch/mips/include/asm/netlogic/xlp-hal/pic.h b/arch/mips/include/asm/netlogic/xlp-hal/pic.h index f10bf3bba58f..41cefe94f0c9 100644 --- a/arch/mips/include/asm/netlogic/xlp-hal/pic.h +++ b/arch/mips/include/asm/netlogic/xlp-hal/pic.h @@ -199,6 +199,10 @@ #define PIC_IRT_PCIE_LINK_3_INDEX 81 #define PIC_IRT_PCIE_LINK_INDEX(num) ((num) + PIC_IRT_PCIE_LINK_0_INDEX) +#define PIC_9XX_IRT_PCIE_LINK_0_INDEX 191 +#define PIC_9XX_IRT_PCIE_LINK_INDEX(num) \ + ((num) + PIC_9XX_IRT_PCIE_LINK_0_INDEX) + #define PIC_CLOCK_TIMER 7 #if !defined(LOCORE) && !defined(__ASSEMBLY__) diff --git a/arch/mips/include/asm/netlogic/xlp-hal/sys.h b/arch/mips/include/asm/netlogic/xlp-hal/sys.h index d9b107ffca93..bc7bddf25be9 100644 --- a/arch/mips/include/asm/netlogic/xlp-hal/sys.h +++ b/arch/mips/include/asm/netlogic/xlp-hal/sys.h @@ -118,6 +118,10 @@ #define SYS_SCRTCH3 0x4c /* PLL registers XLP2XX */ +#define SYS_CPU_PLL_CTRL0(core) (0x1c0 + (core * 4)) +#define SYS_CPU_PLL_CTRL1(core) (0x1c1 + (core * 4)) +#define SYS_CPU_PLL_CTRL2(core) (0x1c2 + (core * 4)) +#define SYS_CPU_PLL_CTRL3(core) (0x1c3 + (core * 4)) #define SYS_PLL_CTRL0 0x240 #define SYS_PLL_CTRL1 0x241 #define SYS_PLL_CTRL2 0x242 @@ -147,6 +151,32 @@ #define SYS_SYS_PLL_MEM_REQ 0x2a3 #define SYS_PLL_MEM_STAT 0x2a4 +/* PLL registers XLP9XX */ +#define SYS_9XX_CPU_PLL_CTRL0(core) (0xc0 + (core * 4)) +#define SYS_9XX_CPU_PLL_CTRL1(core) (0xc1 + (core * 4)) +#define SYS_9XX_CPU_PLL_CTRL2(core) (0xc2 + (core * 4)) +#define SYS_9XX_CPU_PLL_CTRL3(core) (0xc3 + (core * 4)) +#define SYS_9XX_DMC_PLL_CTRL0 0x140 +#define SYS_9XX_DMC_PLL_CTRL1 0x141 +#define SYS_9XX_DMC_PLL_CTRL2 0x142 +#define SYS_9XX_DMC_PLL_CTRL3 0x143 +#define SYS_9XX_PLL_CTRL0 0x144 +#define SYS_9XX_PLL_CTRL1 0x145 +#define SYS_9XX_PLL_CTRL2 0x146 +#define SYS_9XX_PLL_CTRL3 0x147 + +#define SYS_9XX_PLL_CTRL0_DEVX(x) (0x148 + (x) * 4) +#define SYS_9XX_PLL_CTRL1_DEVX(x) (0x149 + (x) * 4) +#define SYS_9XX_PLL_CTRL2_DEVX(x) (0x14a + (x) * 4) +#define SYS_9XX_PLL_CTRL3_DEVX(x) (0x14b + (x) * 4) + +#define SYS_9XX_CPU_PLL_CHG_CTRL 0x188 +#define SYS_9XX_PLL_CHG_CTRL 0x189 +#define SYS_9XX_CLK_DEV_DIS 0x18a +#define SYS_9XX_CLK_DEV_SEL 0x18b +#define SYS_9XX_CLK_DEV_DIV 0x18d +#define SYS_9XX_CLK_DEV_CHG 0x18f + /* Registers changed on 9XX */ #define SYS_9XX_POWER_ON_RESET_CFG 0x00 #define SYS_9XX_CHIP_RESET 0x01 @@ -170,6 +200,11 @@ #define nlm_get_fuse_regbase(node) \ (nlm_get_fuse_pcibase(node) + XLP_IO_PCI_HDRSZ) +#define nlm_get_clock_pcibase(node) \ + nlm_pcicfg_base(XLP9XX_IO_CLOCK_OFFSET(node)) +#define nlm_get_clock_regbase(node) \ + (nlm_get_clock_pcibase(node) + XLP_IO_PCI_HDRSZ) + unsigned int nlm_get_pic_frequency(int node); #endif #endif diff --git a/arch/mips/include/asm/netlogic/xlp-hal/xlp.h b/arch/mips/include/asm/netlogic/xlp-hal/xlp.h index 2b0c9599ebe5..a862b93223cc 100644 --- a/arch/mips/include/asm/netlogic/xlp-hal/xlp.h +++ b/arch/mips/include/asm/netlogic/xlp-hal/xlp.h @@ -58,6 +58,10 @@ #define PIC_I2C_1_IRQ 31 #define PIC_I2C_2_IRQ 32 #define PIC_I2C_3_IRQ 33 +#define PIC_SPI_IRQ 34 +#define PIC_NAND_IRQ 37 +#define PIC_SATA_IRQ 38 +#define PIC_GPIO_IRQ 39 #define PIC_PCIE_LINK_MSI_IRQ_BASE 44 /* 44 - 47 MSI IRQ */ #define PIC_PCIE_LINK_MSI_IRQ(i) (44 + (i)) @@ -66,8 +70,9 @@ #define PIC_PCIE_MSIX_IRQ_BASE 48 /* 48 - 51 MSI-X IRQ */ #define PIC_PCIE_MSIX_IRQ(i) (48 + (i)) -#define NLM_MSIX_VEC_BASE 96 /* 96 - 127 - MSIX mapped */ -#define NLM_MSI_VEC_BASE 128 /* 128 -255 - MSI mapped */ +/* XLP9xx and XLP8xx has 128 and 32 MSIX vectors respectively */ +#define NLM_MSIX_VEC_BASE 96 /* 96 - 223 - MSIX mapped */ +#define NLM_MSI_VEC_BASE 224 /* 224 -351 - MSI mapped */ #define NLM_PIC_INDIRECT_VEC_BASE 512 #define NLM_GPIO_VEC_BASE 768 @@ -95,17 +100,19 @@ void *xlp_dt_init(void *fdtp); static inline int cpu_is_xlpii(void) { - int chip = read_c0_prid() & 0xff00; + int chip = read_c0_prid() & PRID_IMP_MASK; return chip == PRID_IMP_NETLOGIC_XLP2XX || - chip == PRID_IMP_NETLOGIC_XLP9XX; + chip == PRID_IMP_NETLOGIC_XLP9XX || + chip == PRID_IMP_NETLOGIC_XLP5XX; } static inline int cpu_is_xlp9xx(void) { - int chip = read_c0_prid() & 0xff00; + int chip = read_c0_prid() & PRID_IMP_MASK; - return chip == PRID_IMP_NETLOGIC_XLP9XX; + return chip == PRID_IMP_NETLOGIC_XLP9XX || + chip == PRID_IMP_NETLOGIC_XLP5XX; } #endif /* !__ASSEMBLY__ */ #endif /* _ASM_NLM_XLP_H */ diff --git a/arch/mips/include/asm/nile4.h b/arch/mips/include/asm/nile4.h index 2e2436d0e94e..99e97f8bfbca 100644 --- a/arch/mips/include/asm/nile4.h +++ b/arch/mips/include/asm/nile4.h @@ -1,7 +1,7 @@ /* * asm-mips/nile4.h -- NEC Vrc-5074 Nile 4 definitions * - * Copyright (C) 2000 Geert Uytterhoeven <geert@sonycom.com> + * Copyright (C) 2000 Geert Uytterhoeven <geert@linux-m68k.org> * Sony Software Development Center Europe (SDCE), Brussels * * This file is based on the following documentation: diff --git a/arch/mips/include/asm/octeon/octeon.h b/arch/mips/include/asm/octeon/octeon.h index f5d77b91537f..d781f9e66884 100644 --- a/arch/mips/include/asm/octeon/octeon.h +++ b/arch/mips/include/asm/octeon/octeon.h @@ -211,7 +211,6 @@ union octeon_cvmemctl { extern void octeon_write_lcd(const char *s); extern void octeon_check_cpu_bist(void); -extern int octeon_get_boot_debug_flag(void); extern int octeon_get_boot_uart(void); struct uart_port; diff --git a/arch/mips/include/asm/pgtable.h b/arch/mips/include/asm/pgtable.h index 008324d1c261..539ddd148bbb 100644 --- a/arch/mips/include/asm/pgtable.h +++ b/arch/mips/include/asm/pgtable.h @@ -32,6 +32,8 @@ struct vm_area_struct; _page_cachable_default) #define PAGE_KERNEL __pgprot(_PAGE_PRESENT | __READABLE | __WRITEABLE | \ _PAGE_GLOBAL | _page_cachable_default) +#define PAGE_KERNEL_NC __pgprot(_PAGE_PRESENT | __READABLE | __WRITEABLE | \ + _PAGE_GLOBAL | _CACHE_CACHABLE_NONCOHERENT) #define PAGE_USERIO __pgprot(_PAGE_PRESENT | (cpu_has_rixi ? 0 : _PAGE_READ) | _PAGE_WRITE | \ _page_cachable_default) #define PAGE_KERNEL_UNCACHED __pgprot(_PAGE_PRESENT | __READABLE | \ diff --git a/arch/mips/include/asm/pm-cps.h b/arch/mips/include/asm/pm-cps.h new file mode 100644 index 000000000000..625eda53d571 --- /dev/null +++ b/arch/mips/include/asm/pm-cps.h @@ -0,0 +1,51 @@ +/* + * Copyright (C) 2014 Imagination Technologies + * Author: Paul Burton <paul.burton@imgtec.com> + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the + * Free Software Foundation; either version 2 of the License, or (at your + * option) any later version. + */ + +#ifndef __MIPS_ASM_PM_CPS_H__ +#define __MIPS_ASM_PM_CPS_H__ + +/* + * The CM & CPC can only handle coherence & power control on a per-core basis, + * thus in an MT system the VPEs within each core are coupled and can only + * enter or exit states requiring CM or CPC assistance in unison. + */ +#ifdef CONFIG_MIPS_MT +# define coupled_coherence cpu_has_mipsmt +#else +# define coupled_coherence 0 +#endif + +/* Enumeration of possible PM states */ +enum cps_pm_state { + CPS_PM_NC_WAIT, /* MIPS wait instruction, non-coherent */ + CPS_PM_CLOCK_GATED, /* Core clock gated */ + CPS_PM_POWER_GATED, /* Core power gated */ + CPS_PM_STATE_COUNT, +}; + +/** + * cps_pm_support_state - determine whether the system supports a PM state + * @state: the state to test for support + * + * Returns true if the system supports the given state, otherwise false. + */ +extern bool cps_pm_support_state(enum cps_pm_state state); + +/** + * cps_pm_enter_state - enter a PM state + * @state: the state to enter + * + * Enter the given PM state. If coupled_coherence is non-zero then it is + * expected that this function be called at approximately the same time on + * each coupled CPU. Returns 0 on successful entry & exit, otherwise -errno. + */ +extern int cps_pm_enter_state(enum cps_pm_state state); + +#endif /* __MIPS_ASM_PM_CPS_H__ */ diff --git a/arch/mips/include/asm/pm.h b/arch/mips/include/asm/pm.h new file mode 100644 index 000000000000..7c03469e043f --- /dev/null +++ b/arch/mips/include/asm/pm.h @@ -0,0 +1,159 @@ +/* + * Copyright (C) 2014 Imagination Technologies Ltd + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the + * Free Software Foundation; either version 2 of the License, or (at your + * option) any later version. + * + * PM helper macros for CPU power off (e.g. Suspend-to-RAM). + */ + +#ifndef __ASM_PM_H +#define __ASM_PM_H + +#ifdef __ASSEMBLY__ + +#include <asm/asm-offsets.h> +#include <asm/asm.h> +#include <asm/mipsregs.h> +#include <asm/regdef.h> + +/* Save CPU state to stack for suspend to RAM */ +.macro SUSPEND_SAVE_REGS + subu sp, PT_SIZE + /* Call preserved GPRs */ + LONG_S $16, PT_R16(sp) + LONG_S $17, PT_R17(sp) + LONG_S $18, PT_R18(sp) + LONG_S $19, PT_R19(sp) + LONG_S $20, PT_R20(sp) + LONG_S $21, PT_R21(sp) + LONG_S $22, PT_R22(sp) + LONG_S $23, PT_R23(sp) + LONG_S $28, PT_R28(sp) + LONG_S $30, PT_R30(sp) + LONG_S $31, PT_R31(sp) + /* A couple of CP0 registers with space in pt_regs */ + mfc0 k0, CP0_STATUS + LONG_S k0, PT_STATUS(sp) +.endm + +/* Restore CPU state from stack after resume from RAM */ +.macro RESUME_RESTORE_REGS_RETURN + .set push + .set noreorder + /* A couple of CP0 registers with space in pt_regs */ + LONG_L k0, PT_STATUS(sp) + mtc0 k0, CP0_STATUS + /* Call preserved GPRs */ + LONG_L $16, PT_R16(sp) + LONG_L $17, PT_R17(sp) + LONG_L $18, PT_R18(sp) + LONG_L $19, PT_R19(sp) + LONG_L $20, PT_R20(sp) + LONG_L $21, PT_R21(sp) + LONG_L $22, PT_R22(sp) + LONG_L $23, PT_R23(sp) + LONG_L $28, PT_R28(sp) + LONG_L $30, PT_R30(sp) + LONG_L $31, PT_R31(sp) + /* Pop and return */ + jr ra + addiu sp, PT_SIZE + .set pop +.endm + +/* Get address of static suspend state into t1 */ +.macro LA_STATIC_SUSPEND + la t1, mips_static_suspend_state +.endm + +/* Save important CPU state for early restoration to global data */ +.macro SUSPEND_SAVE_STATIC +#ifdef CONFIG_EVA + /* + * Segment configuration is saved in global data where it can be easily + * reloaded without depending on the segment configuration. + */ + mfc0 k0, CP0_PAGEMASK, 2 /* SegCtl0 */ + LONG_S k0, SSS_SEGCTL0(t1) + mfc0 k0, CP0_PAGEMASK, 3 /* SegCtl1 */ + LONG_S k0, SSS_SEGCTL1(t1) + mfc0 k0, CP0_PAGEMASK, 4 /* SegCtl2 */ + LONG_S k0, SSS_SEGCTL2(t1) +#endif + /* save stack pointer (pointing to GPRs) */ + LONG_S sp, SSS_SP(t1) +.endm + +/* Restore important CPU state early from global data */ +.macro RESUME_RESTORE_STATIC +#ifdef CONFIG_EVA + /* + * Segment configuration must be restored prior to any access to + * allocated memory, as it may reside outside of the legacy kernel + * segments. + */ + LONG_L k0, SSS_SEGCTL0(t1) + mtc0 k0, CP0_PAGEMASK, 2 /* SegCtl0 */ + LONG_L k0, SSS_SEGCTL1(t1) + mtc0 k0, CP0_PAGEMASK, 3 /* SegCtl1 */ + LONG_L k0, SSS_SEGCTL2(t1) + mtc0 k0, CP0_PAGEMASK, 4 /* SegCtl2 */ + tlbw_use_hazard +#endif + /* restore stack pointer (pointing to GPRs) */ + LONG_L sp, SSS_SP(t1) +.endm + +/* flush caches to make sure context has reached memory */ +.macro SUSPEND_CACHE_FLUSH + .extern __wback_cache_all + .set push + .set noreorder + la t1, __wback_cache_all + LONG_L t0, 0(t1) + jalr t0 + nop + .set pop + .endm + +/* Save suspend state and flush data caches to RAM */ +.macro SUSPEND_SAVE + SUSPEND_SAVE_REGS + LA_STATIC_SUSPEND + SUSPEND_SAVE_STATIC + SUSPEND_CACHE_FLUSH +.endm + +/* Restore saved state after resume from RAM and return */ +.macro RESUME_RESTORE_RETURN + LA_STATIC_SUSPEND + RESUME_RESTORE_STATIC + RESUME_RESTORE_REGS_RETURN +.endm + +#else /* __ASSEMBLY__ */ + +/** + * struct mips_static_suspend_state - Core saved CPU state across S2R. + * @segctl: CP0 Segment control registers. + * @sp: Stack frame where GP register context is saved. + * + * This structure contains minimal CPU state that must be saved in static kernel + * data in order to be able to restore the rest of the state. This includes + * segmentation configuration in the case of EVA being enabled, as they must be + * restored prior to any kmalloc'd memory being referenced (even the stack + * pointer). + */ +struct mips_static_suspend_state { +#ifdef CONFIG_EVA + unsigned long segctl[3]; +#endif + unsigned long sp; +}; + +#endif /* !__ASSEMBLY__ */ + +#endif /* __ASM_PM_HELPERS_H */ diff --git a/arch/mips/include/asm/ptrace.h b/arch/mips/include/asm/ptrace.h index bf1ac8d35783..7e6e682aece3 100644 --- a/arch/mips/include/asm/ptrace.h +++ b/arch/mips/include/asm/ptrace.h @@ -39,9 +39,6 @@ struct pt_regs { unsigned long cp0_badvaddr; unsigned long cp0_cause; unsigned long cp0_epc; -#ifdef CONFIG_MIPS_MT_SMTC - unsigned long cp0_tcstatus; -#endif /* CONFIG_MIPS_MT_SMTC */ #ifdef CONFIG_CPU_CAVIUM_OCTEON unsigned long long mpl[3]; /* MTM{0,1,2} */ unsigned long long mtp[3]; /* MTP{0,1,2} */ diff --git a/arch/mips/include/asm/r4kcache.h b/arch/mips/include/asm/r4kcache.h index ca64cbe44493..0b8bd28a0df1 100644 --- a/arch/mips/include/asm/r4kcache.h +++ b/arch/mips/include/asm/r4kcache.h @@ -43,11 +43,10 @@ : "i" (op), "R" (*(unsigned char *)(addr))) #ifdef CONFIG_MIPS_MT + /* - * Temporary hacks for SMTC debug. Optionally force single-threaded - * execution during I-cache flushes. + * Optionally force single-threaded execution during I-cache flushes. */ - #define PROTECT_CACHE_FLUSHES 1 #ifdef PROTECT_CACHE_FLUSHES @@ -524,6 +523,8 @@ __BUILD_BLAST_CACHE(s, scache, Index_Writeback_Inv_SD, Hit_Writeback_Inv_SD, 32, __BUILD_BLAST_CACHE(d, dcache, Index_Writeback_Inv_D, Hit_Writeback_Inv_D, 64, ) __BUILD_BLAST_CACHE(i, icache, Index_Invalidate_I, Hit_Invalidate_I, 64, ) __BUILD_BLAST_CACHE(s, scache, Index_Writeback_Inv_SD, Hit_Writeback_Inv_SD, 64, ) +__BUILD_BLAST_CACHE(d, dcache, Index_Writeback_Inv_D, Hit_Writeback_Inv_D, 128, ) +__BUILD_BLAST_CACHE(i, icache, Index_Invalidate_I, Hit_Invalidate_I, 128, ) __BUILD_BLAST_CACHE(s, scache, Index_Writeback_Inv_SD, Hit_Writeback_Inv_SD, 128, ) __BUILD_BLAST_CACHE(inv_d, dcache, Index_Writeback_Inv_D, Hit_Invalidate_D, 16, ) diff --git a/arch/mips/include/asm/sgi/ip22.h b/arch/mips/include/asm/sgi/ip22.h index 8db1a3588cf2..87ec9eaa04e3 100644 --- a/arch/mips/include/asm/sgi/ip22.h +++ b/arch/mips/include/asm/sgi/ip22.h @@ -69,6 +69,8 @@ #define SGI_EISA_IRQ SGINT_LOCAL2 + 3 /* EISA interrupts */ #define SGI_KEYBD_IRQ SGINT_LOCAL2 + 4 /* keyboard */ #define SGI_SERIAL_IRQ SGINT_LOCAL2 + 5 /* onboard serial */ +#define SGI_GIOEXP0_IRQ (SGINT_LOCAL2 + 6) /* Indy GIO EXP0 */ +#define SGI_GIOEXP1_IRQ (SGINT_LOCAL2 + 7) /* Indy GIO EXP1 */ #define ip22_is_fullhouse() (sgioc->sysid & SGIOC_SYSID_FULLHOUSE) diff --git a/arch/mips/include/asm/smp-cps.h b/arch/mips/include/asm/smp-cps.h index d60d1a2180d1..a06a08a9afc6 100644 --- a/arch/mips/include/asm/smp-cps.h +++ b/arch/mips/include/asm/smp-cps.h @@ -13,17 +13,28 @@ #ifndef __ASSEMBLY__ -struct boot_config { - unsigned int core; - unsigned int vpe; +struct vpe_boot_config { unsigned long pc; unsigned long sp; unsigned long gp; }; -extern struct boot_config mips_cps_bootcfg; +struct core_boot_config { + atomic_t vpe_mask; + struct vpe_boot_config *vpe_config; +}; + +extern struct core_boot_config *mips_cps_core_bootcfg; extern void mips_cps_core_entry(void); +extern void mips_cps_core_init(void); + +extern struct vpe_boot_config *mips_cps_boot_vpes(void); + +extern bool mips_cps_smp_in_use(void); + +extern void mips_cps_pm_save(void); +extern void mips_cps_pm_restore(void); #else /* __ASSEMBLY__ */ diff --git a/arch/mips/include/asm/smp-ops.h b/arch/mips/include/asm/smp-ops.h index 73d35b18fb64..6ba1fb8b11e2 100644 --- a/arch/mips/include/asm/smp-ops.h +++ b/arch/mips/include/asm/smp-ops.h @@ -26,7 +26,6 @@ struct plat_smp_ops { void (*send_ipi_mask)(const struct cpumask *mask, unsigned int action); void (*init_secondary)(void); void (*smp_finish)(void); - void (*cpus_done)(void); void (*boot_secondary)(int cpu, struct task_struct *idle); void (*smp_setup)(void); void (*prepare_cpus)(unsigned int max_cpus); diff --git a/arch/mips/include/asm/smp.h b/arch/mips/include/asm/smp.h index efa02acd3dd5..b037334fca22 100644 --- a/arch/mips/include/asm/smp.h +++ b/arch/mips/include/asm/smp.h @@ -46,6 +46,9 @@ extern int __cpu_logical_map[NR_CPUS]; extern volatile cpumask_t cpu_callin_map; +/* Mask of CPUs which are currently definitely operating coherently */ +extern cpumask_t cpu_coherent_mask; + extern void asmlinkage smp_bootstrap(void); /* diff --git a/arch/mips/include/asm/smtc.h b/arch/mips/include/asm/smtc.h deleted file mode 100644 index e56b439b7871..000000000000 --- a/arch/mips/include/asm/smtc.h +++ /dev/null @@ -1,78 +0,0 @@ -#ifndef _ASM_SMTC_MT_H -#define _ASM_SMTC_MT_H - -/* - * Definitions for SMTC multitasking on MIPS MT cores - */ - -#include <asm/mips_mt.h> -#include <asm/smtc_ipi.h> - -/* - * System-wide SMTC status information - */ - -extern unsigned int smtc_status; - -#define SMTC_TLB_SHARED 0x00000001 -#define SMTC_MTC_ACTIVE 0x00000002 - -/* - * TLB/ASID Management information - */ - -#define MAX_SMTC_TLBS 2 -#define MAX_SMTC_ASIDS 256 -#if NR_CPUS <= 8 -typedef char asiduse; -#else -#if NR_CPUS <= 16 -typedef short asiduse; -#else -typedef long asiduse; -#endif -#endif - -/* - * VPE Management information - */ - -#define MAX_SMTC_VPES MAX_SMTC_TLBS /* FIXME: May not always be true. */ - -extern asiduse smtc_live_asid[MAX_SMTC_TLBS][MAX_SMTC_ASIDS]; - -struct mm_struct; -struct task_struct; - -void smtc_get_new_mmu_context(struct mm_struct *mm, unsigned long cpu); -void self_ipi(struct smtc_ipi *); -void smtc_flush_tlb_asid(unsigned long asid); -extern int smtc_build_cpu_map(int startslot); -extern void smtc_prepare_cpus(int cpus); -extern void smtc_smp_finish(void); -extern void smtc_boot_secondary(int cpu, struct task_struct *t); -extern void smtc_cpus_done(void); -extern void smtc_init_secondary(void); - - -/* - * Sharing the TLB between multiple VPEs means that the - * "random" index selection function is not allowed to - * select the current value of the Index register. To - * avoid additional TLB pressure, the Index registers - * are "parked" with an non-Valid value. - */ - -#define PARKED_INDEX ((unsigned int)0x80000000) - -/* - * Define low-level interrupt mask for IPIs, if necessary. - * By default, use SW interrupt 1, which requires no external - * hardware support, but which works only for single-core - * MIPS MT systems. - */ -#ifndef MIPS_CPU_IPI_IRQ -#define MIPS_CPU_IPI_IRQ 1 -#endif - -#endif /* _ASM_SMTC_MT_H */ diff --git a/arch/mips/include/asm/smtc_ipi.h b/arch/mips/include/asm/smtc_ipi.h deleted file mode 100644 index 15278dbd7e79..000000000000 --- a/arch/mips/include/asm/smtc_ipi.h +++ /dev/null @@ -1,129 +0,0 @@ -/* - * Definitions used in MIPS MT SMTC "Interprocessor Interrupt" code. - */ -#ifndef __ASM_SMTC_IPI_H -#define __ASM_SMTC_IPI_H - -#include <linux/spinlock.h> - -//#define SMTC_IPI_DEBUG - -#ifdef SMTC_IPI_DEBUG -#include <asm/mipsregs.h> -#include <asm/mipsmtregs.h> -#endif /* SMTC_IPI_DEBUG */ - -/* - * An IPI "message" - */ - -struct smtc_ipi { - struct smtc_ipi *flink; - int type; - void *arg; - int dest; -#ifdef SMTC_IPI_DEBUG - int sender; - long stamp; -#endif /* SMTC_IPI_DEBUG */ -}; - -/* - * Defined IPI Types - */ - -#define LINUX_SMP_IPI 1 -#define SMTC_CLOCK_TICK 2 -#define IRQ_AFFINITY_IPI 3 - -/* - * A queue of IPI messages - */ - -struct smtc_ipi_q { - struct smtc_ipi *head; - spinlock_t lock; - struct smtc_ipi *tail; - int depth; - int resched_flag; /* reschedule already queued */ -}; - -static inline void smtc_ipi_nq(struct smtc_ipi_q *q, struct smtc_ipi *p) -{ - unsigned long flags; - - spin_lock_irqsave(&q->lock, flags); - if (q->head == NULL) - q->head = q->tail = p; - else - q->tail->flink = p; - p->flink = NULL; - q->tail = p; - q->depth++; -#ifdef SMTC_IPI_DEBUG - p->sender = read_c0_tcbind(); - p->stamp = read_c0_count(); -#endif /* SMTC_IPI_DEBUG */ - spin_unlock_irqrestore(&q->lock, flags); -} - -static inline struct smtc_ipi *__smtc_ipi_dq(struct smtc_ipi_q *q) -{ - struct smtc_ipi *p; - - if (q->head == NULL) - p = NULL; - else { - p = q->head; - q->head = q->head->flink; - q->depth--; - /* Arguably unnecessary, but leaves queue cleaner */ - if (q->head == NULL) - q->tail = NULL; - } - - return p; -} - -static inline struct smtc_ipi *smtc_ipi_dq(struct smtc_ipi_q *q) -{ - unsigned long flags; - struct smtc_ipi *p; - - spin_lock_irqsave(&q->lock, flags); - p = __smtc_ipi_dq(q); - spin_unlock_irqrestore(&q->lock, flags); - - return p; -} - -static inline void smtc_ipi_req(struct smtc_ipi_q *q, struct smtc_ipi *p) -{ - unsigned long flags; - - spin_lock_irqsave(&q->lock, flags); - if (q->head == NULL) { - q->head = q->tail = p; - p->flink = NULL; - } else { - p->flink = q->head; - q->head = p; - } - q->depth++; - spin_unlock_irqrestore(&q->lock, flags); -} - -static inline int smtc_ipi_qdepth(struct smtc_ipi_q *q) -{ - unsigned long flags; - int retval; - - spin_lock_irqsave(&q->lock, flags); - retval = q->depth; - spin_unlock_irqrestore(&q->lock, flags); - return retval; -} - -extern void smtc_send_ipi(int cpu, int type, unsigned int action); - -#endif /* __ASM_SMTC_IPI_H */ diff --git a/arch/mips/include/asm/smtc_proc.h b/arch/mips/include/asm/smtc_proc.h deleted file mode 100644 index 25da651f1f5f..000000000000 --- a/arch/mips/include/asm/smtc_proc.h +++ /dev/null @@ -1,23 +0,0 @@ -/* - * Definitions for SMTC /proc entries - * Copyright(C) 2005 MIPS Technologies Inc. - */ -#ifndef __ASM_SMTC_PROC_H -#define __ASM_SMTC_PROC_H - -/* - * per-"CPU" statistics - */ - -struct smtc_cpu_proc { - unsigned long timerints; - unsigned long selfipis; -}; - -extern struct smtc_cpu_proc smtc_cpu_stats[NR_CPUS]; - -/* Count of number of recoveries of "stolen" FPU access rights on 34K */ - -extern atomic_t smtc_fpu_recoveries; - -#endif /* __ASM_SMTC_PROC_H */ diff --git a/arch/mips/include/asm/stackframe.h b/arch/mips/include/asm/stackframe.h index d301e108d5b8..b188c797565c 100644 --- a/arch/mips/include/asm/stackframe.h +++ b/arch/mips/include/asm/stackframe.h @@ -19,22 +19,12 @@ #include <asm/asm-offsets.h> #include <asm/thread_info.h> -/* - * For SMTC kernel, global IE should be left set, and interrupts - * controlled exclusively via IXMT. - */ -#ifdef CONFIG_MIPS_MT_SMTC -#define STATMASK 0x1e -#elif defined(CONFIG_CPU_R3000) || defined(CONFIG_CPU_TX39XX) +#if defined(CONFIG_CPU_R3000) || defined(CONFIG_CPU_TX39XX) #define STATMASK 0x3f #else #define STATMASK 0x1f #endif -#ifdef CONFIG_MIPS_MT_SMTC -#include <asm/mipsmtregs.h> -#endif /* CONFIG_MIPS_MT_SMTC */ - .macro SAVE_AT .set push .set noat @@ -186,16 +176,6 @@ mfc0 v1, CP0_STATUS LONG_S $2, PT_R2(sp) LONG_S v1, PT_STATUS(sp) -#ifdef CONFIG_MIPS_MT_SMTC - /* - * Ideally, these instructions would be shuffled in - * to cover the pipeline delay. - */ - .set mips32 - mfc0 k0, CP0_TCSTATUS - .set mips0 - LONG_S k0, PT_TCSTATUS(sp) -#endif /* CONFIG_MIPS_MT_SMTC */ LONG_S $4, PT_R4(sp) mfc0 v1, CP0_CAUSE LONG_S $5, PT_R5(sp) @@ -321,36 +301,6 @@ .set push .set reorder .set noat -#ifdef CONFIG_MIPS_MT_SMTC - .set mips32r2 - /* - * We need to make sure the read-modify-write - * of Status below isn't perturbed by an interrupt - * or cross-TC access, so we need to do at least a DMT, - * protected by an interrupt-inhibit. But setting IXMT - * also creates a few-cycle window where an IPI could - * be queued and not be detected before potentially - * returning to a WAIT or user-mode loop. It must be - * replayed. - * - * We're in the middle of a context switch, and - * we can't dispatch it directly without trashing - * some registers, so we'll try to detect this unlikely - * case and program a software interrupt in the VPE, - * as would be done for a cross-VPE IPI. To accommodate - * the handling of that case, we're doing a DVPE instead - * of just a DMT here to protect against other threads. - * This is a lot of cruft to cover a tiny window. - * If you can find a better design, implement it! - * - */ - mfc0 v0, CP0_TCSTATUS - ori v0, TCSTATUS_IXMT - mtc0 v0, CP0_TCSTATUS - _ehb - DVPE 5 # dvpe a1 - jal mips_ihb -#endif /* CONFIG_MIPS_MT_SMTC */ mfc0 a0, CP0_STATUS ori a0, STATMASK xori a0, STATMASK @@ -362,59 +312,6 @@ and v0, v1 or v0, a0 mtc0 v0, CP0_STATUS -#ifdef CONFIG_MIPS_MT_SMTC -/* - * Only after EXL/ERL have been restored to status can we - * restore TCStatus.IXMT. - */ - LONG_L v1, PT_TCSTATUS(sp) - _ehb - mfc0 a0, CP0_TCSTATUS - andi v1, TCSTATUS_IXMT - bnez v1, 0f - -/* - * We'd like to detect any IPIs queued in the tiny window - * above and request an software interrupt to service them - * when we ERET. - * - * Computing the offset into the IPIQ array of the executing - * TC's IPI queue in-line would be tedious. We use part of - * the TCContext register to hold 16 bits of offset that we - * can add in-line to find the queue head. - */ - mfc0 v0, CP0_TCCONTEXT - la a2, IPIQ - srl v0, v0, 16 - addu a2, a2, v0 - LONG_L v0, 0(a2) - beqz v0, 0f -/* - * If we have a queue, provoke dispatch within the VPE by setting C_SW1 - */ - mfc0 v0, CP0_CAUSE - ori v0, v0, C_SW1 - mtc0 v0, CP0_CAUSE -0: - /* - * This test should really never branch but - * let's be prudent here. Having atomized - * the shared register modifications, we can - * now EVPE, and must do so before interrupts - * are potentially re-enabled. - */ - andi a1, a1, MVPCONTROL_EVP - beqz a1, 1f - evpe -1: - /* We know that TCStatua.IXMT should be set from above */ - xori a0, a0, TCSTATUS_IXMT - or a0, a0, v1 - mtc0 a0, CP0_TCSTATUS - _ehb - - .set mips0 -#endif /* CONFIG_MIPS_MT_SMTC */ LONG_L v1, PT_EPC(sp) MTC0 v1, CP0_EPC LONG_L $31, PT_R31(sp) @@ -467,33 +364,11 @@ * Set cp0 enable bit as sign that we're running on the kernel stack */ .macro CLI -#if !defined(CONFIG_MIPS_MT_SMTC) mfc0 t0, CP0_STATUS li t1, ST0_CU0 | STATMASK or t0, t1 xori t0, STATMASK mtc0 t0, CP0_STATUS -#else /* CONFIG_MIPS_MT_SMTC */ - /* - * For SMTC, we need to set privilege - * and disable interrupts only for the - * current TC, using the TCStatus register. - */ - mfc0 t0, CP0_TCSTATUS - /* Fortunately CU 0 is in the same place in both registers */ - /* Set TCU0, TMX, TKSU (for later inversion) and IXMT */ - li t1, ST0_CU0 | 0x08001c00 - or t0, t1 - /* Clear TKSU, leave IXMT */ - xori t0, 0x00001800 - mtc0 t0, CP0_TCSTATUS - _ehb - /* We need to leave the global IE bit set, but clear EXL...*/ - mfc0 t0, CP0_STATUS - ori t0, ST0_EXL | ST0_ERL - xori t0, ST0_EXL | ST0_ERL - mtc0 t0, CP0_STATUS -#endif /* CONFIG_MIPS_MT_SMTC */ irq_disable_hazard .endm @@ -502,35 +377,11 @@ * Set cp0 enable bit as sign that we're running on the kernel stack */ .macro STI -#if !defined(CONFIG_MIPS_MT_SMTC) mfc0 t0, CP0_STATUS li t1, ST0_CU0 | STATMASK or t0, t1 xori t0, STATMASK & ~1 mtc0 t0, CP0_STATUS -#else /* CONFIG_MIPS_MT_SMTC */ - /* - * For SMTC, we need to set privilege - * and enable interrupts only for the - * current TC, using the TCStatus register. - */ - _ehb - mfc0 t0, CP0_TCSTATUS - /* Fortunately CU 0 is in the same place in both registers */ - /* Set TCU0, TKSU (for later inversion) and IXMT */ - li t1, ST0_CU0 | 0x08001c00 - or t0, t1 - /* Clear TKSU *and* IXMT */ - xori t0, 0x00001c00 - mtc0 t0, CP0_TCSTATUS - _ehb - /* We need to leave the global IE bit set, but clear EXL...*/ - mfc0 t0, CP0_STATUS - ori t0, ST0_EXL - xori t0, ST0_EXL - mtc0 t0, CP0_STATUS - /* irq_enable_hazard below should expand to EHB for 24K/34K cpus */ -#endif /* CONFIG_MIPS_MT_SMTC */ irq_enable_hazard .endm @@ -540,32 +391,6 @@ * Set cp0 enable bit as sign that we're running on the kernel stack */ .macro KMODE -#ifdef CONFIG_MIPS_MT_SMTC - /* - * This gets baroque in SMTC. We want to - * protect the non-atomic clearing of EXL - * with DMT/EMT, but we don't want to take - * an interrupt while DMT is still in effect. - */ - - /* KMODE gets invoked from both reorder and noreorder code */ - .set push - .set mips32r2 - .set noreorder - mfc0 v0, CP0_TCSTATUS - andi v1, v0, TCSTATUS_IXMT - ori v0, TCSTATUS_IXMT - mtc0 v0, CP0_TCSTATUS - _ehb - DMT 2 # dmt v0 - /* - * We don't know a priori if ra is "live" - */ - move t0, ra - jal mips_ihb - nop /* delay slot */ - move ra, t0 -#endif /* CONFIG_MIPS_MT_SMTC */ mfc0 t0, CP0_STATUS li t1, ST0_CU0 | (STATMASK & ~1) #if defined(CONFIG_CPU_R3000) || defined(CONFIG_CPU_TX39XX) @@ -576,25 +401,6 @@ or t0, t1 xori t0, STATMASK & ~1 mtc0 t0, CP0_STATUS -#ifdef CONFIG_MIPS_MT_SMTC - _ehb - andi v0, v0, VPECONTROL_TE - beqz v0, 2f - nop /* delay slot */ - emt -2: - mfc0 v0, CP0_TCSTATUS - /* Clear IXMT, then OR in previous value */ - ori v0, TCSTATUS_IXMT - xori v0, TCSTATUS_IXMT - or v0, v1, v0 - mtc0 v0, CP0_TCSTATUS - /* - * irq_disable_hazard below should expand to EHB - * on 24K/34K CPUS - */ - .set pop -#endif /* CONFIG_MIPS_MT_SMTC */ irq_disable_hazard .endm diff --git a/arch/mips/include/asm/thread_info.h b/arch/mips/include/asm/thread_info.h index d2d961d6cb86..7de865805deb 100644 --- a/arch/mips/include/asm/thread_info.h +++ b/arch/mips/include/asm/thread_info.h @@ -159,11 +159,7 @@ static inline struct thread_info *current_thread_info(void) * We stash processor id into a COP0 register to retrieve it fast * at kernel exception entry. */ -#if defined(CONFIG_MIPS_MT_SMTC) -#define SMP_CPUID_REG 2, 2 /* TCBIND */ -#define ASM_SMP_CPUID_REG $2, 2 -#define SMP_CPUID_PTRSHIFT 19 -#elif defined(CONFIG_MIPS_PGD_C0_CONTEXT) +#if defined(CONFIG_MIPS_PGD_C0_CONTEXT) #define SMP_CPUID_REG 20, 0 /* XCONTEXT */ #define ASM_SMP_CPUID_REG $20 #define SMP_CPUID_PTRSHIFT 48 @@ -179,13 +175,8 @@ static inline struct thread_info *current_thread_info(void) #define SMP_CPUID_REGSHIFT (SMP_CPUID_PTRSHIFT + 2) #endif -#ifdef CONFIG_MIPS_MT_SMTC -#define ASM_CPUID_MFC0 mfc0 -#define UASM_i_CPUID_MFC0 uasm_i_mfc0 -#else #define ASM_CPUID_MFC0 MFC0 #define UASM_i_CPUID_MFC0 UASM_i_MFC0 -#endif #endif /* __KERNEL__ */ #endif /* _ASM_THREAD_INFO_H */ diff --git a/arch/mips/include/asm/time.h b/arch/mips/include/asm/time.h index 24f534a7fbc3..8f3047d611ee 100644 --- a/arch/mips/include/asm/time.h +++ b/arch/mips/include/asm/time.h @@ -52,14 +52,11 @@ extern int (*perf_irq)(void); */ extern unsigned int __weak get_c0_compare_int(void); extern int r4k_clockevent_init(void); -extern int smtc_clockevent_init(void); extern int gic_clockevent_init(void); static inline int mips_clockevent_init(void) { -#ifdef CONFIG_MIPS_MT_SMTC - return smtc_clockevent_init(); -#elif defined(CONFIG_CEVT_GIC) +#if defined(CONFIG_CEVT_GIC) return (gic_clockevent_init() | r4k_clockevent_init()); #elif defined(CONFIG_CEVT_R4K) return r4k_clockevent_init(); diff --git a/arch/mips/include/asm/timex.h b/arch/mips/include/asm/timex.h index c5424757da65..b05bb70a2e46 100644 --- a/arch/mips/include/asm/timex.h +++ b/arch/mips/include/asm/timex.h @@ -4,12 +4,16 @@ * for more details. * * Copyright (C) 1998, 1999, 2003 by Ralf Baechle + * Copyright (C) 2014 by Maciej W. Rozycki */ #ifndef _ASM_TIMEX_H #define _ASM_TIMEX_H #ifdef __KERNEL__ +#include <linux/compiler.h> + +#include <asm/cpu.h> #include <asm/cpu-features.h> #include <asm/mipsregs.h> #include <asm/cpu-type.h> @@ -45,29 +49,54 @@ typedef unsigned int cycles_t; * However for now the implementaton of this function doesn't get these * fine details right. */ -static inline cycles_t get_cycles(void) +static inline int can_use_mips_counter(unsigned int prid) { - switch (boot_cpu_type()) { - case CPU_R4400PC: - case CPU_R4400SC: - case CPU_R4400MC: - if ((read_c0_prid() & 0xff) >= 0x0050) - return read_c0_count(); - break; + int comp = (prid & PRID_COMP_MASK) != PRID_COMP_LEGACY; - case CPU_R4000PC: - case CPU_R4000SC: - case CPU_R4000MC: - break; + if (__builtin_constant_p(cpu_has_counter) && !cpu_has_counter) + return 0; + else if (__builtin_constant_p(cpu_has_mips_r) && cpu_has_mips_r) + return 1; + else if (likely(!__builtin_constant_p(cpu_has_mips_r) && comp)) + return 1; + /* Make sure we don't peek at cpu_data[0].options in the fast path! */ + if (!__builtin_constant_p(cpu_has_counter)) + asm volatile("" : "=m" (cpu_data[0].options)); + if (likely(cpu_has_counter && + prid >= (PRID_IMP_R4000 | PRID_REV_ENCODE_44(5, 0)))) + return 1; + else + return 0; +} - default: - if (cpu_has_counter) - return read_c0_count(); - break; - } +static inline cycles_t get_cycles(void) +{ + if (can_use_mips_counter(read_c0_prid())) + return read_c0_count(); + else + return 0; /* no usable counter */ +} + +/* + * Like get_cycles - but where c0_count is not available we desperately + * use c0_random in an attempt to get at least a little bit of entropy. + * + * R6000 and R6000A neither have a count register nor a random register. + * That leaves no entropy source in the CPU itself. + */ +static inline unsigned long random_get_entropy(void) +{ + unsigned int prid = read_c0_prid(); + unsigned int imp = prid & PRID_IMP_MASK; - return 0; /* no usable counter */ + if (can_use_mips_counter(prid)) + return read_c0_count(); + else if (likely(imp != PRID_IMP_R6000 && imp != PRID_IMP_R6000A)) + return read_c0_random(); + else + return 0; /* no usable register */ } +#define random_get_entropy random_get_entropy #endif /* __KERNEL__ */ diff --git a/arch/mips/include/asm/uasm.h b/arch/mips/include/asm/uasm.h index c33a9564fb41..f8d63b3b40b4 100644 --- a/arch/mips/include/asm/uasm.h +++ b/arch/mips/include/asm/uasm.h @@ -55,6 +55,9 @@ void ISAOPC(op)(u32 **buf, unsigned int a, unsigned int b, unsigned int c) #define Ip_u2u1u3(op) \ void ISAOPC(op)(u32 **buf, unsigned int a, unsigned int b, unsigned int c) +#define Ip_u3u2u1(op) \ +void ISAOPC(op)(u32 **buf, unsigned int a, unsigned int b, unsigned int c) + #define Ip_u3u1u2(op) \ void ISAOPC(op)(u32 **buf, unsigned int a, unsigned int b, unsigned int c) @@ -74,6 +77,9 @@ void ISAOPC(op)(u32 **buf, unsigned int a, unsigned int b, unsigned int c, \ #define Ip_u1u2(op) \ void ISAOPC(op)(u32 **buf, unsigned int a, unsigned int b) +#define Ip_u2u1(op) \ +void ISAOPC(op)(u32 **buf, unsigned int a, unsigned int b) + #define Ip_u1s2(op) \ void ISAOPC(op)(u32 **buf, unsigned int a, signed int b) @@ -99,6 +105,7 @@ Ip_u2u1s3(_daddiu); Ip_u3u1u2(_daddu); Ip_u2u1msbu3(_dins); Ip_u2u1msbu3(_dinsm); +Ip_u1u2(_divu); Ip_u1u2u3(_dmfc0); Ip_u1u2u3(_dmtc0); Ip_u2u1u3(_drotr); @@ -114,16 +121,22 @@ Ip_u2u1msbu3(_ext); Ip_u2u1msbu3(_ins); Ip_u1(_j); Ip_u1(_jal); +Ip_u2u1(_jalr); Ip_u1(_jr); +Ip_u2s3u1(_lb); Ip_u2s3u1(_ld); Ip_u3u1u2(_ldx); +Ip_u2s3u1(_lh); Ip_u2s3u1(_ll); Ip_u2s3u1(_lld); Ip_u1s2(_lui); Ip_u2s3u1(_lw); Ip_u3u1u2(_lwx); Ip_u1u2u3(_mfc0); +Ip_u1(_mfhi); +Ip_u1(_mflo); Ip_u1u2u3(_mtc0); +Ip_u3u1u2(_mul); Ip_u3u1u2(_or); Ip_u2u1u3(_ori); Ip_u2s3u1(_pref); @@ -133,17 +146,25 @@ Ip_u2s3u1(_sc); Ip_u2s3u1(_scd); Ip_u2s3u1(_sd); Ip_u2u1u3(_sll); +Ip_u3u2u1(_sllv); +Ip_u2u1s3(_sltiu); +Ip_u3u1u2(_sltu); Ip_u2u1u3(_sra); Ip_u2u1u3(_srl); +Ip_u3u2u1(_srlv); Ip_u3u1u2(_subu); Ip_u2s3u1(_sw); +Ip_u1(_sync); Ip_u1(_syscall); Ip_0(_tlbp); Ip_0(_tlbr); Ip_0(_tlbwi); Ip_0(_tlbwr); +Ip_u1(_wait); +Ip_u2u1(_wsbh); Ip_u3u1u2(_xor); Ip_u2u1u3(_xori); +Ip_u2u1(_yield); /* Handle labels. */ @@ -264,6 +285,8 @@ void uasm_il_bbit0(u32 **p, struct uasm_reloc **r, unsigned int reg, unsigned int bit, int lid); void uasm_il_bbit1(u32 **p, struct uasm_reloc **r, unsigned int reg, unsigned int bit, int lid); +void uasm_il_beq(u32 **p, struct uasm_reloc **r, unsigned int r1, + unsigned int r2, int lid); void uasm_il_beqz(u32 **p, struct uasm_reloc **r, unsigned int reg, int lid); void uasm_il_beqzl(u32 **p, struct uasm_reloc **r, unsigned int reg, int lid); void uasm_il_bgezl(u32 **p, struct uasm_reloc **r, unsigned int reg, int lid); diff --git a/arch/mips/include/uapi/asm/Kbuild b/arch/mips/include/uapi/asm/Kbuild index be7196eacb88..96fe7395ed8d 100644 --- a/arch/mips/include/uapi/asm/Kbuild +++ b/arch/mips/include/uapi/asm/Kbuild @@ -4,6 +4,7 @@ include include/uapi/asm-generic/Kbuild.asm generic-y += auxvec.h generic-y += ipcbuf.h +header-y += bitfield.h header-y += bitsperlong.h header-y += break.h header-y += byteorder.h diff --git a/arch/mips/include/uapi/asm/bitfield.h b/arch/mips/include/uapi/asm/bitfield.h new file mode 100644 index 000000000000..ad9861359cea --- /dev/null +++ b/arch/mips/include/uapi/asm/bitfield.h @@ -0,0 +1,29 @@ +/* + * This file is subject to the terms and conditions of the GNU General Public + * License. See the file "COPYING" in the main directory of this archive + * for more details. + * + * Copyright (C) 2014 by Ralf Baechle <ralf@linux-mips.org> + */ +#ifndef __UAPI_ASM_BITFIELD_H +#define __UAPI_ASM_BITFIELD_H + +/* + * * Damn ... bitfields depend from byteorder :-( + * */ +#ifdef __MIPSEB__ +#define __BITFIELD_FIELD(field, more) \ + field; \ + more + +#elif defined(__MIPSEL__) + +#define __BITFIELD_FIELD(field, more) \ + more \ + field; + +#else /* !defined (__MIPSEB__) && !defined (__MIPSEL__) */ +#error "MIPS but neither __MIPSEL__ nor __MIPSEB__?" +#endif + +#endif /* __UAPI_ASM_BITFIELD_H */ diff --git a/arch/mips/include/uapi/asm/inst.h b/arch/mips/include/uapi/asm/inst.h index 3125797f2a88..4b7160259292 100644 --- a/arch/mips/include/uapi/asm/inst.h +++ b/arch/mips/include/uapi/asm/inst.h @@ -13,6 +13,8 @@ #ifndef _UAPI_ASM_INST_H #define _UAPI_ASM_INST_H +#include <asm/bitfield.h> + /* * Major opcodes; before MIPS IV cop1x was called cop3. */ @@ -74,16 +76,17 @@ enum spec2_op { enum spec3_op { ext_op, dextm_op, dextu_op, dext_op, ins_op, dinsm_op, dinsu_op, dins_op, - lx_op = 0x0a, lwle_op = 0x19, - lwre_op = 0x1a, cachee_op = 0x1b, - sbe_op = 0x1c, she_op = 0x1d, - sce_op = 0x1e, swe_op = 0x1f, - bshfl_op = 0x20, swle_op = 0x21, - swre_op = 0x22, prefe_op = 0x23, - dbshfl_op = 0x24, lbue_op = 0x28, - lhue_op = 0x29, lbe_op = 0x2c, - lhe_op = 0x2d, lle_op = 0x2e, - lwe_op = 0x2f, rdhwr_op = 0x3b + yield_op = 0x09, lx_op = 0x0a, + lwle_op = 0x19, lwre_op = 0x1a, + cachee_op = 0x1b, sbe_op = 0x1c, + she_op = 0x1d, sce_op = 0x1e, + swe_op = 0x1f, bshfl_op = 0x20, + swle_op = 0x21, swre_op = 0x22, + prefe_op = 0x23, dbshfl_op = 0x24, + lbue_op = 0x28, lhue_op = 0x29, + lbe_op = 0x2c, lhe_op = 0x2d, + lle_op = 0x2e, lwe_op = 0x2f, + rdhwr_op = 0x3b }; /* @@ -125,7 +128,8 @@ enum bcop_op { enum cop0_coi_func { tlbr_op = 0x01, tlbwi_op = 0x02, tlbwr_op = 0x06, tlbp_op = 0x08, - rfe_op = 0x10, eret_op = 0x18 + rfe_op = 0x10, eret_op = 0x18, + wait_op = 0x20, }; /* @@ -202,6 +206,16 @@ enum lx_func { }; /* + * BSHFL opcodes + */ +enum bshfl_func { + wsbh_op = 0x2, + dshd_op = 0x5, + seb_op = 0x10, + seh_op = 0x18, +}; + +/* * (microMIPS) Major opcodes. */ enum mm_major_op { @@ -244,17 +258,22 @@ enum mm_32i_minor_op { enum mm_32a_minor_op { mm_sll32_op = 0x000, mm_ins_op = 0x00c, + mm_sllv32_op = 0x010, mm_ext_op = 0x02c, mm_pool32axf_op = 0x03c, mm_srl32_op = 0x040, mm_sra_op = 0x080, + mm_srlv32_op = 0x090, mm_rotr_op = 0x0c0, mm_lwxs_op = 0x118, mm_addu32_op = 0x150, mm_subu32_op = 0x1d0, + mm_wsbh_op = 0x1ec, + mm_mul_op = 0x210, mm_and_op = 0x250, mm_or32_op = 0x290, mm_xor32_op = 0x310, + mm_sltu_op = 0x390, }; /* @@ -294,15 +313,20 @@ enum mm_32axf_minor_op { mm_mfc0_op = 0x003, mm_mtc0_op = 0x00b, mm_tlbp_op = 0x00d, + mm_mfhi32_op = 0x035, mm_jalr_op = 0x03c, mm_tlbr_op = 0x04d, + mm_mflo32_op = 0x075, mm_jalrhb_op = 0x07c, mm_tlbwi_op = 0x08d, mm_tlbwr_op = 0x0cd, mm_jalrs_op = 0x13c, mm_jalrshb_op = 0x17c, + mm_sync_op = 0x1ad, mm_syscall_op = 0x22d, + mm_wait_op = 0x24d, mm_eret_op = 0x3cd, + mm_divu_op = 0x5dc, }; /* @@ -480,24 +504,6 @@ enum MIPS6e_i8_func { */ #define MM_NOP16 0x0c00 -/* - * Damn ... bitfields depend from byteorder :-( - */ -#ifdef __MIPSEB__ -#define __BITFIELD_FIELD(field, more) \ - field; \ - more - -#elif defined(__MIPSEL__) - -#define __BITFIELD_FIELD(field, more) \ - more \ - field; - -#else /* !defined (__MIPSEB__) && !defined (__MIPSEL__) */ -#error "MIPS but neither __MIPSEL__ nor __MIPSEB__?" -#endif - struct j_format { __BITFIELD_FIELD(unsigned int opcode : 6, /* Jump format */ __BITFIELD_FIELD(unsigned int target : 26, diff --git a/arch/mips/include/uapi/asm/kvm_para.h b/arch/mips/include/uapi/asm/kvm_para.h index 14fab8f0b957..7e16d7c42e65 100644 --- a/arch/mips/include/uapi/asm/kvm_para.h +++ b/arch/mips/include/uapi/asm/kvm_para.h @@ -1 +1,5 @@ -#include <asm-generic/kvm_para.h> +#ifndef _UAPI_ASM_MIPS_KVM_PARA_H +#define _UAPI_ASM_MIPS_KVM_PARA_H + + +#endif /* _UAPI_ASM_MIPS_KVM_PARA_H */ diff --git a/arch/mips/include/uapi/asm/types.h b/arch/mips/include/uapi/asm/types.h index 7ac9d0baad84..f3dd9ff0cc0c 100644 --- a/arch/mips/include/uapi/asm/types.h +++ b/arch/mips/include/uapi/asm/types.h @@ -14,9 +14,12 @@ /* * We don't use int-l64.h for the kernel anymore but still use it for * userspace to avoid code changes. + * + * However, some user programs (e.g. perf) may not want this. They can + * flag __SANE_USERSPACE_TYPES__ to get int-ll64.h here. */ #ifndef __KERNEL__ -# if _MIPS_SZLONG == 64 +# if _MIPS_SZLONG == 64 && !defined(__SANE_USERSPACE_TYPES__) # include <asm-generic/int-l64.h> # else # include <asm-generic/int-ll64.h> diff --git a/arch/mips/kernel/Makefile b/arch/mips/kernel/Makefile index 277dab301cea..008a2fed0584 100644 --- a/arch/mips/kernel/Makefile +++ b/arch/mips/kernel/Makefile @@ -17,7 +17,6 @@ endif obj-$(CONFIG_CEVT_BCM1480) += cevt-bcm1480.o obj-$(CONFIG_CEVT_R4K) += cevt-r4k.o -obj-$(CONFIG_MIPS_MT_SMTC) += cevt-smtc.o obj-$(CONFIG_CEVT_DS1287) += cevt-ds1287.o obj-$(CONFIG_CEVT_GIC) += cevt-gic.o obj-$(CONFIG_CEVT_GT641XX) += cevt-gt641xx.o @@ -42,7 +41,7 @@ obj-$(CONFIG_CPU_R4K_FPU) += r4k_fpu.o r4k_switch.o obj-$(CONFIG_CPU_R3000) += r2300_fpu.o r2300_switch.o obj-$(CONFIG_CPU_R6000) += r6000_fpu.o r4k_switch.o obj-$(CONFIG_CPU_TX39XX) += r2300_fpu.o r2300_switch.o -obj-$(CONFIG_CPU_CAVIUM_OCTEON) += octeon_switch.o +obj-$(CONFIG_CPU_CAVIUM_OCTEON) += r4k_fpu.o octeon_switch.o obj-$(CONFIG_SMP) += smp.o obj-$(CONFIG_SMP_UP) += smp-up.o @@ -50,7 +49,6 @@ obj-$(CONFIG_CPU_BMIPS) += smp-bmips.o bmips_vec.o obj-$(CONFIG_MIPS_MT) += mips-mt.o obj-$(CONFIG_MIPS_MT_FPAFF) += mips-mt-fpaff.o -obj-$(CONFIG_MIPS_MT_SMTC) += smtc.o smtc-asm.o smtc-proc.o obj-$(CONFIG_MIPS_MT_SMP) += smp-mt.o obj-$(CONFIG_MIPS_CMP) += smp-cmp.o obj-$(CONFIG_MIPS_CPS) += smp-cps.o cps-vec.o @@ -107,6 +105,9 @@ obj-$(CONFIG_JUMP_LABEL) += jump_label.o obj-$(CONFIG_MIPS_CM) += mips-cm.o obj-$(CONFIG_MIPS_CPC) += mips-cpc.o +obj-$(CONFIG_CPU_PM) += pm.o +obj-$(CONFIG_MIPS_CPS_PM) += pm-cps.o + # # DSP ASE supported for MIPS32 or MIPS64 Release 2 cores only. It is not # safe to unconditionnaly use the assembler -mdsp / -mdspr2 switches diff --git a/arch/mips/kernel/asm-offsets.c b/arch/mips/kernel/asm-offsets.c index 0ea75c244b48..02f075df8f2e 100644 --- a/arch/mips/kernel/asm-offsets.c +++ b/arch/mips/kernel/asm-offsets.c @@ -14,6 +14,7 @@ #include <linux/mm.h> #include <linux/kbuild.h> #include <linux/suspend.h> +#include <asm/pm.h> #include <asm/ptrace.h> #include <asm/processor.h> #include <asm/smp-cps.h> @@ -64,9 +65,6 @@ void output_ptreg_defines(void) OFFSET(PT_BVADDR, pt_regs, cp0_badvaddr); OFFSET(PT_STATUS, pt_regs, cp0_status); OFFSET(PT_CAUSE, pt_regs, cp0_cause); -#ifdef CONFIG_MIPS_MT_SMTC - OFFSET(PT_TCSTATUS, pt_regs, cp0_tcstatus); -#endif /* CONFIG_MIPS_MT_SMTC */ #ifdef CONFIG_CPU_CAVIUM_OCTEON OFFSET(PT_MPL, pt_regs, mpl); OFFSET(PT_MTP, pt_regs, mtp); @@ -404,6 +402,20 @@ void output_pbe_defines(void) } #endif +#ifdef CONFIG_CPU_PM +void output_pm_defines(void) +{ + COMMENT(" PM offsets. "); +#ifdef CONFIG_EVA + OFFSET(SSS_SEGCTL0, mips_static_suspend_state, segctl[0]); + OFFSET(SSS_SEGCTL1, mips_static_suspend_state, segctl[1]); + OFFSET(SSS_SEGCTL2, mips_static_suspend_state, segctl[2]); +#endif + OFFSET(SSS_SP, mips_static_suspend_state, sp); + BLANK(); +} +#endif + void output_kvm_defines(void) { COMMENT(" KVM/MIPS Specfic offsets. "); @@ -472,10 +484,14 @@ void output_kvm_defines(void) void output_cps_defines(void) { COMMENT(" MIPS CPS offsets. "); - OFFSET(BOOTCFG_CORE, boot_config, core); - OFFSET(BOOTCFG_VPE, boot_config, vpe); - OFFSET(BOOTCFG_PC, boot_config, pc); - OFFSET(BOOTCFG_SP, boot_config, sp); - OFFSET(BOOTCFG_GP, boot_config, gp); + + OFFSET(COREBOOTCFG_VPEMASK, core_boot_config, vpe_mask); + OFFSET(COREBOOTCFG_VPECONFIG, core_boot_config, vpe_config); + DEFINE(COREBOOTCFG_SIZE, sizeof(struct core_boot_config)); + + OFFSET(VPEBOOTCFG_PC, vpe_boot_config, pc); + OFFSET(VPEBOOTCFG_SP, vpe_boot_config, sp); + OFFSET(VPEBOOTCFG_GP, vpe_boot_config, gp); + DEFINE(VPEBOOTCFG_SIZE, sizeof(struct vpe_boot_config)); } #endif diff --git a/arch/mips/kernel/branch.c b/arch/mips/kernel/branch.c index 76122ff5cb5e..7b2df224f041 100644 --- a/arch/mips/kernel/branch.c +++ b/arch/mips/kernel/branch.c @@ -48,6 +48,202 @@ int __isa_exception_epc(struct pt_regs *regs) return epc; } +/* (microMIPS) Convert 16-bit register encoding to 32-bit register encoding. */ +static const unsigned int reg16to32map[8] = {16, 17, 2, 3, 4, 5, 6, 7}; + +int __mm_isBranchInstr(struct pt_regs *regs, struct mm_decoded_insn dec_insn, + unsigned long *contpc) +{ + union mips_instruction insn = (union mips_instruction)dec_insn.insn; + int bc_false = 0; + unsigned int fcr31; + unsigned int bit; + + if (!cpu_has_mmips) + return 0; + + switch (insn.mm_i_format.opcode) { + case mm_pool32a_op: + if ((insn.mm_i_format.simmediate & MM_POOL32A_MINOR_MASK) == + mm_pool32axf_op) { + switch (insn.mm_i_format.simmediate >> + MM_POOL32A_MINOR_SHIFT) { + case mm_jalr_op: + case mm_jalrhb_op: + case mm_jalrs_op: + case mm_jalrshb_op: + if (insn.mm_i_format.rt != 0) /* Not mm_jr */ + regs->regs[insn.mm_i_format.rt] = + regs->cp0_epc + + dec_insn.pc_inc + + dec_insn.next_pc_inc; + *contpc = regs->regs[insn.mm_i_format.rs]; + return 1; + } + } + break; + case mm_pool32i_op: + switch (insn.mm_i_format.rt) { + case mm_bltzals_op: + case mm_bltzal_op: + regs->regs[31] = regs->cp0_epc + + dec_insn.pc_inc + + dec_insn.next_pc_inc; + /* Fall through */ + case mm_bltz_op: + if ((long)regs->regs[insn.mm_i_format.rs] < 0) + *contpc = regs->cp0_epc + + dec_insn.pc_inc + + (insn.mm_i_format.simmediate << 1); + else + *contpc = regs->cp0_epc + + dec_insn.pc_inc + + dec_insn.next_pc_inc; + return 1; + case mm_bgezals_op: + case mm_bgezal_op: + regs->regs[31] = regs->cp0_epc + + dec_insn.pc_inc + + dec_insn.next_pc_inc; + /* Fall through */ + case mm_bgez_op: + if ((long)regs->regs[insn.mm_i_format.rs] >= 0) + *contpc = regs->cp0_epc + + dec_insn.pc_inc + + (insn.mm_i_format.simmediate << 1); + else + *contpc = regs->cp0_epc + + dec_insn.pc_inc + + dec_insn.next_pc_inc; + return 1; + case mm_blez_op: + if ((long)regs->regs[insn.mm_i_format.rs] <= 0) + *contpc = regs->cp0_epc + + dec_insn.pc_inc + + (insn.mm_i_format.simmediate << 1); + else + *contpc = regs->cp0_epc + + dec_insn.pc_inc + + dec_insn.next_pc_inc; + return 1; + case mm_bgtz_op: + if ((long)regs->regs[insn.mm_i_format.rs] <= 0) + *contpc = regs->cp0_epc + + dec_insn.pc_inc + + (insn.mm_i_format.simmediate << 1); + else + *contpc = regs->cp0_epc + + dec_insn.pc_inc + + dec_insn.next_pc_inc; + return 1; + case mm_bc2f_op: + case mm_bc1f_op: + bc_false = 1; + /* Fall through */ + case mm_bc2t_op: + case mm_bc1t_op: + preempt_disable(); + if (is_fpu_owner()) + asm volatile("cfc1\t%0,$31" : "=r" (fcr31)); + else + fcr31 = current->thread.fpu.fcr31; + preempt_enable(); + + if (bc_false) + fcr31 = ~fcr31; + + bit = (insn.mm_i_format.rs >> 2); + bit += (bit != 0); + bit += 23; + if (fcr31 & (1 << bit)) + *contpc = regs->cp0_epc + + dec_insn.pc_inc + + (insn.mm_i_format.simmediate << 1); + else + *contpc = regs->cp0_epc + + dec_insn.pc_inc + dec_insn.next_pc_inc; + return 1; + } + break; + case mm_pool16c_op: + switch (insn.mm_i_format.rt) { + case mm_jalr16_op: + case mm_jalrs16_op: + regs->regs[31] = regs->cp0_epc + + dec_insn.pc_inc + dec_insn.next_pc_inc; + /* Fall through */ + case mm_jr16_op: + *contpc = regs->regs[insn.mm_i_format.rs]; + return 1; + } + break; + case mm_beqz16_op: + if ((long)regs->regs[reg16to32map[insn.mm_b1_format.rs]] == 0) + *contpc = regs->cp0_epc + + dec_insn.pc_inc + + (insn.mm_b1_format.simmediate << 1); + else + *contpc = regs->cp0_epc + + dec_insn.pc_inc + dec_insn.next_pc_inc; + return 1; + case mm_bnez16_op: + if ((long)regs->regs[reg16to32map[insn.mm_b1_format.rs]] != 0) + *contpc = regs->cp0_epc + + dec_insn.pc_inc + + (insn.mm_b1_format.simmediate << 1); + else + *contpc = regs->cp0_epc + + dec_insn.pc_inc + dec_insn.next_pc_inc; + return 1; + case mm_b16_op: + *contpc = regs->cp0_epc + dec_insn.pc_inc + + (insn.mm_b0_format.simmediate << 1); + return 1; + case mm_beq32_op: + if (regs->regs[insn.mm_i_format.rs] == + regs->regs[insn.mm_i_format.rt]) + *contpc = regs->cp0_epc + + dec_insn.pc_inc + + (insn.mm_i_format.simmediate << 1); + else + *contpc = regs->cp0_epc + + dec_insn.pc_inc + + dec_insn.next_pc_inc; + return 1; + case mm_bne32_op: + if (regs->regs[insn.mm_i_format.rs] != + regs->regs[insn.mm_i_format.rt]) + *contpc = regs->cp0_epc + + dec_insn.pc_inc + + (insn.mm_i_format.simmediate << 1); + else + *contpc = regs->cp0_epc + + dec_insn.pc_inc + dec_insn.next_pc_inc; + return 1; + case mm_jalx32_op: + regs->regs[31] = regs->cp0_epc + + dec_insn.pc_inc + dec_insn.next_pc_inc; + *contpc = regs->cp0_epc + dec_insn.pc_inc; + *contpc >>= 28; + *contpc <<= 28; + *contpc |= (insn.j_format.target << 2); + return 1; + case mm_jals32_op: + case mm_jal32_op: + regs->regs[31] = regs->cp0_epc + + dec_insn.pc_inc + dec_insn.next_pc_inc; + /* Fall through */ + case mm_j32_op: + *contpc = regs->cp0_epc + dec_insn.pc_inc; + *contpc >>= 27; + *contpc <<= 27; + *contpc |= (insn.j_format.target << 1); + set_isa16_mode(*contpc); + return 1; + } + return 0; +} + /* * Compute return address and emulate branch in microMIPS mode after an * exception only. It does not handle compact branches/jumps and cannot @@ -366,7 +562,11 @@ int __compute_return_epc_for_insn(struct pt_regs *regs, case cop1_op: preempt_disable(); if (is_fpu_owner()) - asm volatile("cfc1\t%0,$31" : "=r" (fcr31)); + asm volatile( + ".set push\n" + "\t.set mips1\n" + "\tcfc1\t%0,$31\n" + "\t.set pop" : "=r" (fcr31)); else fcr31 = current->thread.fpu.fcr31; preempt_enable(); diff --git a/arch/mips/kernel/cevt-gic.c b/arch/mips/kernel/cevt-gic.c index 594cbbf16d62..6093716980b9 100644 --- a/arch/mips/kernel/cevt-gic.c +++ b/arch/mips/kernel/cevt-gic.c @@ -26,7 +26,7 @@ static int gic_next_event(unsigned long delta, struct clock_event_device *evt) cnt = gic_read_count(); cnt += (u64)delta; - gic_write_compare(cnt); + gic_write_cpu_compare(cnt, cpumask_first(evt->cpumask)); res = ((int)(gic_read_count() - cnt) >= 0) ? -ETIME : 0; return res; } @@ -73,7 +73,8 @@ int gic_clockevent_init(void) cd = &per_cpu(gic_clockevent_device, cpu); cd->name = "MIPS GIC"; - cd->features = CLOCK_EVT_FEAT_ONESHOT; + cd->features = CLOCK_EVT_FEAT_ONESHOT | + CLOCK_EVT_FEAT_C3STOP; clockevent_set_clock(cd, gic_frequency); diff --git a/arch/mips/kernel/cevt-r4k.c b/arch/mips/kernel/cevt-r4k.c index 50d3f5a8d6bb..bc127e22fdab 100644 --- a/arch/mips/kernel/cevt-r4k.c +++ b/arch/mips/kernel/cevt-r4k.c @@ -12,17 +12,10 @@ #include <linux/smp.h> #include <linux/irq.h> -#include <asm/smtc_ipi.h> #include <asm/time.h> #include <asm/cevt-r4k.h> #include <asm/gic.h> -/* - * The SMTC Kernel for the 34K, 1004K, et. al. replaces several - * of these routines with SMTC-specific variants. - */ - -#ifndef CONFIG_MIPS_MT_SMTC static int mips_next_event(unsigned long delta, struct clock_event_device *evt) { @@ -36,8 +29,6 @@ static int mips_next_event(unsigned long delta, return res; } -#endif /* CONFIG_MIPS_MT_SMTC */ - void mips_set_clock_mode(enum clock_event_mode mode, struct clock_event_device *evt) { @@ -47,7 +38,6 @@ void mips_set_clock_mode(enum clock_event_mode mode, DEFINE_PER_CPU(struct clock_event_device, mips_clockevent_device); int cp0_timer_irq_installed; -#ifndef CONFIG_MIPS_MT_SMTC irqreturn_t c0_compare_interrupt(int irq, void *dev_id) { const int r2 = cpu_has_mips_r2; @@ -72,9 +62,6 @@ irqreturn_t c0_compare_interrupt(int irq, void *dev_id) /* Clear Count/Compare Interrupt */ write_c0_compare(read_c0_compare()); cd = &per_cpu(mips_clockevent_device, cpu); -#ifdef CONFIG_CEVT_GIC - if (!gic_present) -#endif cd->event_handler(cd); } @@ -82,8 +69,6 @@ out: return IRQ_HANDLED; } -#endif /* Not CONFIG_MIPS_MT_SMTC */ - struct irqaction c0_compare_irqaction = { .handler = c0_compare_interrupt, .flags = IRQF_PERCPU | IRQF_TIMER, @@ -170,7 +155,6 @@ int c0_compare_int_usable(void) return 1; } -#ifndef CONFIG_MIPS_MT_SMTC int r4k_clockevent_init(void) { unsigned int cpu = smp_processor_id(); @@ -195,7 +179,9 @@ int r4k_clockevent_init(void) cd = &per_cpu(mips_clockevent_device, cpu); cd->name = "MIPS"; - cd->features = CLOCK_EVT_FEAT_ONESHOT; + cd->features = CLOCK_EVT_FEAT_ONESHOT | + CLOCK_EVT_FEAT_C3STOP | + CLOCK_EVT_FEAT_PERCPU; clockevent_set_clock(cd, mips_hpt_frequency); @@ -210,9 +196,6 @@ int r4k_clockevent_init(void) cd->set_mode = mips_set_clock_mode; cd->event_handler = mips_event_handler; -#ifdef CONFIG_CEVT_GIC - if (!gic_present) -#endif clockevents_register_device(cd); if (cp0_timer_irq_installed) @@ -225,4 +208,3 @@ int r4k_clockevent_init(void) return 0; } -#endif /* Not CONFIG_MIPS_MT_SMTC */ diff --git a/arch/mips/kernel/cevt-smtc.c b/arch/mips/kernel/cevt-smtc.c deleted file mode 100644 index b6cf0a60d896..000000000000 --- a/arch/mips/kernel/cevt-smtc.c +++ /dev/null @@ -1,324 +0,0 @@ -/* - * This file is subject to the terms and conditions of the GNU General Public - * License. See the file "COPYING" in the main directory of this archive - * for more details. - * - * Copyright (C) 2007 MIPS Technologies, Inc. - * Copyright (C) 2007 Ralf Baechle <ralf@linux-mips.org> - * Copyright (C) 2008 Kevin D. Kissell, Paralogos sarl - */ -#include <linux/clockchips.h> -#include <linux/interrupt.h> -#include <linux/percpu.h> -#include <linux/smp.h> -#include <linux/irq.h> - -#include <asm/smtc_ipi.h> -#include <asm/time.h> -#include <asm/cevt-r4k.h> - -/* - * Variant clock event timer support for SMTC on MIPS 34K, 1004K - * or other MIPS MT cores. - * - * Notes on SMTC Support: - * - * SMTC has multiple microthread TCs pretending to be Linux CPUs. - * But there's only one Count/Compare pair per VPE, and Compare - * interrupts are taken opportunisitically by available TCs - * bound to the VPE with the Count register. The new timer - * framework provides for global broadcasts, but we really - * want VPE-level multicasts for best behavior. So instead - * of invoking the high-level clock-event broadcast code, - * this version of SMTC support uses the historical SMTC - * multicast mechanisms "under the hood", appearing to the - * generic clock layer as if the interrupts are per-CPU. - * - * The approach taken here is to maintain a set of NR_CPUS - * virtual timers, and track which "CPU" needs to be alerted - * at each event. - * - * It's unlikely that we'll see a MIPS MT core with more than - * 2 VPEs, but we *know* that we won't need to handle more - * VPEs than we have "CPUs". So NCPUs arrays of NCPUs elements - * is always going to be overkill, but always going to be enough. - */ - -unsigned long smtc_nexttime[NR_CPUS][NR_CPUS]; -static int smtc_nextinvpe[NR_CPUS]; - -/* - * Timestamps stored are absolute values to be programmed - * into Count register. Valid timestamps will never be zero. - * If a Zero Count value is actually calculated, it is converted - * to be a 1, which will introduce 1 or two CPU cycles of error - * roughly once every four billion events, which at 1000 HZ means - * about once every 50 days. If that's actually a problem, one - * could alternate squashing 0 to 1 and to -1. - */ - -#define MAKEVALID(x) (((x) == 0L) ? 1L : (x)) -#define ISVALID(x) ((x) != 0L) - -/* - * Time comparison is subtle, as it's really truncated - * modular arithmetic. - */ - -#define IS_SOONER(a, b, reference) \ - (((a) - (unsigned long)(reference)) < ((b) - (unsigned long)(reference))) - -/* - * CATCHUP_INCREMENT, used when the function falls behind the counter. - * Could be an increasing function instead of a constant; - */ - -#define CATCHUP_INCREMENT 64 - -static int mips_next_event(unsigned long delta, - struct clock_event_device *evt) -{ - unsigned long flags; - unsigned int mtflags; - unsigned long timestamp, reference, previous; - unsigned long nextcomp = 0L; - int vpe = current_cpu_data.vpe_id; - int cpu = smp_processor_id(); - local_irq_save(flags); - mtflags = dmt(); - - /* - * Maintain the per-TC virtual timer - * and program the per-VPE shared Count register - * as appropriate here... - */ - reference = (unsigned long)read_c0_count(); - timestamp = MAKEVALID(reference + delta); - /* - * To really model the clock, we have to catch the case - * where the current next-in-VPE timestamp is the old - * timestamp for the calling CPE, but the new value is - * in fact later. In that case, we have to do a full - * scan and discover the new next-in-VPE CPU id and - * timestamp. - */ - previous = smtc_nexttime[vpe][cpu]; - if (cpu == smtc_nextinvpe[vpe] && ISVALID(previous) - && IS_SOONER(previous, timestamp, reference)) { - int i; - int soonest = cpu; - - /* - * Update timestamp array here, so that new - * value gets considered along with those of - * other virtual CPUs on the VPE. - */ - smtc_nexttime[vpe][cpu] = timestamp; - for_each_online_cpu(i) { - if (ISVALID(smtc_nexttime[vpe][i]) - && IS_SOONER(smtc_nexttime[vpe][i], - smtc_nexttime[vpe][soonest], reference)) { - soonest = i; - } - } - smtc_nextinvpe[vpe] = soonest; - nextcomp = smtc_nexttime[vpe][soonest]; - /* - * Otherwise, we don't have to process the whole array rank, - * we just have to see if the event horizon has gotten closer. - */ - } else { - if (!ISVALID(smtc_nexttime[vpe][smtc_nextinvpe[vpe]]) || - IS_SOONER(timestamp, - smtc_nexttime[vpe][smtc_nextinvpe[vpe]], reference)) { - smtc_nextinvpe[vpe] = cpu; - nextcomp = timestamp; - } - /* - * Since next-in-VPE may me the same as the executing - * virtual CPU, we update the array *after* checking - * its value. - */ - smtc_nexttime[vpe][cpu] = timestamp; - } - - /* - * It may be that, in fact, we don't need to update Compare, - * but if we do, we want to make sure we didn't fall into - * a crack just behind Count. - */ - if (ISVALID(nextcomp)) { - write_c0_compare(nextcomp); - ehb(); - /* - * We never return an error, we just make sure - * that we trigger the handlers as quickly as - * we can if we fell behind. - */ - while ((nextcomp - (unsigned long)read_c0_count()) - > (unsigned long)LONG_MAX) { - nextcomp += CATCHUP_INCREMENT; - write_c0_compare(nextcomp); - ehb(); - } - } - emt(mtflags); - local_irq_restore(flags); - return 0; -} - - -void smtc_distribute_timer(int vpe) -{ - unsigned long flags; - unsigned int mtflags; - int cpu; - struct clock_event_device *cd; - unsigned long nextstamp; - unsigned long reference; - - -repeat: - nextstamp = 0L; - for_each_online_cpu(cpu) { - /* - * Find virtual CPUs within the current VPE who have - * unserviced timer requests whose time is now past. - */ - local_irq_save(flags); - mtflags = dmt(); - if (cpu_data[cpu].vpe_id == vpe && - ISVALID(smtc_nexttime[vpe][cpu])) { - reference = (unsigned long)read_c0_count(); - if ((smtc_nexttime[vpe][cpu] - reference) - > (unsigned long)LONG_MAX) { - smtc_nexttime[vpe][cpu] = 0L; - emt(mtflags); - local_irq_restore(flags); - /* - * We don't send IPIs to ourself. - */ - if (cpu != smp_processor_id()) { - smtc_send_ipi(cpu, SMTC_CLOCK_TICK, 0); - } else { - cd = &per_cpu(mips_clockevent_device, cpu); - cd->event_handler(cd); - } - } else { - /* Local to VPE but Valid Time not yet reached. */ - if (!ISVALID(nextstamp) || - IS_SOONER(smtc_nexttime[vpe][cpu], nextstamp, - reference)) { - smtc_nextinvpe[vpe] = cpu; - nextstamp = smtc_nexttime[vpe][cpu]; - } - emt(mtflags); - local_irq_restore(flags); - } - } else { - emt(mtflags); - local_irq_restore(flags); - - } - } - /* Reprogram for interrupt at next soonest timestamp for VPE */ - if (ISVALID(nextstamp)) { - write_c0_compare(nextstamp); - ehb(); - if ((nextstamp - (unsigned long)read_c0_count()) - > (unsigned long)LONG_MAX) - goto repeat; - } -} - - -irqreturn_t c0_compare_interrupt(int irq, void *dev_id) -{ - int cpu = smp_processor_id(); - - /* If we're running SMTC, we've got MIPS MT and therefore MIPS32R2 */ - handle_perf_irq(1); - - if (read_c0_cause() & (1 << 30)) { - /* Clear Count/Compare Interrupt */ - write_c0_compare(read_c0_compare()); - smtc_distribute_timer(cpu_data[cpu].vpe_id); - } - return IRQ_HANDLED; -} - - -int smtc_clockevent_init(void) -{ - uint64_t mips_freq = mips_hpt_frequency; - unsigned int cpu = smp_processor_id(); - struct clock_event_device *cd; - unsigned int irq; - int i; - int j; - - if (!cpu_has_counter || !mips_hpt_frequency) - return -ENXIO; - if (cpu == 0) { - for (i = 0; i < num_possible_cpus(); i++) { - smtc_nextinvpe[i] = 0; - for (j = 0; j < num_possible_cpus(); j++) - smtc_nexttime[i][j] = 0L; - } - /* - * SMTC also can't have the usablility test - * run by secondary TCs once Compare is in use. - */ - if (!c0_compare_int_usable()) - return -ENXIO; - } - - /* - * With vectored interrupts things are getting platform specific. - * get_c0_compare_int is a hook to allow a platform to return the - * interrupt number of it's liking. - */ - irq = MIPS_CPU_IRQ_BASE + cp0_compare_irq; - if (get_c0_compare_int) - irq = get_c0_compare_int(); - - cd = &per_cpu(mips_clockevent_device, cpu); - - cd->name = "MIPS"; - cd->features = CLOCK_EVT_FEAT_ONESHOT; - - /* Calculate the min / max delta */ - cd->mult = div_sc((unsigned long) mips_freq, NSEC_PER_SEC, 32); - cd->shift = 32; - cd->max_delta_ns = clockevent_delta2ns(0x7fffffff, cd); - cd->min_delta_ns = clockevent_delta2ns(0x300, cd); - - cd->rating = 300; - cd->irq = irq; - cd->cpumask = cpumask_of(cpu); - cd->set_next_event = mips_next_event; - cd->set_mode = mips_set_clock_mode; - cd->event_handler = mips_event_handler; - - clockevents_register_device(cd); - - /* - * On SMTC we only want to do the data structure - * initialization and IRQ setup once. - */ - if (cpu) - return 0; - /* - * And we need the hwmask associated with the c0_compare - * vector to be initialized. - */ - irq_hwmask[irq] = (0x100 << cp0_compare_irq); - if (cp0_timer_irq_installed) - return 0; - - cp0_timer_irq_installed = 1; - - setup_irq(irq, &c0_compare_irqaction); - - return 0; -} diff --git a/arch/mips/kernel/cps-vec.S b/arch/mips/kernel/cps-vec.S index f7a46db4b161..6f4f739dad96 100644 --- a/arch/mips/kernel/cps-vec.S +++ b/arch/mips/kernel/cps-vec.S @@ -14,19 +14,43 @@ #include <asm/asmmacro.h> #include <asm/cacheops.h> #include <asm/mipsregs.h> +#include <asm/mipsmtregs.h> +#include <asm/pm.h> -#define GCR_CL_COHERENCE_OFS 0x2008 +#define GCR_CL_COHERENCE_OFS 0x2008 +#define GCR_CL_ID_OFS 0x2028 + +.extern mips_cm_base + +.set noreorder + + /* + * Set dest to non-zero if the core supports the MT ASE, else zero. If + * MT is not supported then branch to nomt. + */ + .macro has_mt dest, nomt + mfc0 \dest, CP0_CONFIG + bgez \dest, \nomt + mfc0 \dest, CP0_CONFIG, 1 + bgez \dest, \nomt + mfc0 \dest, CP0_CONFIG, 2 + bgez \dest, \nomt + mfc0 \dest, CP0_CONFIG, 3 + andi \dest, \dest, MIPS_CONF3_MT + beqz \dest, \nomt + .endm .section .text.cps-vec .balign 0x1000 -.set noreorder LEAF(mips_cps_core_entry) /* - * These first 8 bytes will be patched by cps_smp_setup to load the - * base address of the CM GCRs into register v1. + * These first 12 bytes will be patched by cps_smp_setup to load the + * base address of the CM GCRs into register v1 and the CCA to use into + * register s0. */ .quad 0 + .word 0 /* Check whether we're here due to an NMI */ mfc0 k0, CP0_STATUS @@ -117,10 +141,11 @@ icache_done: add a0, a0, t0 dcache_done: - /* Set Kseg0 cacheable, coherent, write-back, write-allocate */ + /* Set Kseg0 CCA to that in s0 */ mfc0 t0, CP0_CONFIG ori t0, 0x7 - xori t0, 0x2 + xori t0, 0x7 + or t0, t0, s0 mtc0 t0, CP0_CONFIG ehb @@ -134,21 +159,24 @@ dcache_done: jr t0 nop -1: /* We're up, cached & coherent */ + /* + * We're up, cached & coherent. Perform any further required core-level + * initialisation. + */ +1: jal mips_cps_core_init + nop /* - * TODO: We should check the VPE number we intended to boot here, and - * if non-zero we should start that VPE and stop this one. For - * the moment this doesn't matter since CPUs are brought up - * sequentially and in order, but once hotplug is implemented - * this will need revisiting. + * Boot any other VPEs within this core that should be online, and + * deactivate this VPE if it should be offline. */ + jal mips_cps_boot_vpes + nop /* Off we go! */ - la t0, mips_cps_bootcfg - lw t1, BOOTCFG_PC(t0) - lw gp, BOOTCFG_GP(t0) - lw sp, BOOTCFG_SP(t0) + lw t1, VPEBOOTCFG_PC(v0) + lw gp, VPEBOOTCFG_GP(v0) + lw sp, VPEBOOTCFG_SP(v0) jr t1 nop END(mips_cps_core_entry) @@ -189,3 +217,271 @@ LEAF(excep_ejtag) jr k0 nop END(excep_ejtag) + +LEAF(mips_cps_core_init) +#ifdef CONFIG_MIPS_MT + /* Check that the core implements the MT ASE */ + has_mt t0, 3f + nop + + .set push + .set mt + + /* Only allow 1 TC per VPE to execute... */ + dmt + + /* ...and for the moment only 1 VPE */ + dvpe + la t1, 1f + jr.hb t1 + nop + + /* Enter VPE configuration state */ +1: mfc0 t0, CP0_MVPCONTROL + ori t0, t0, MVPCONTROL_VPC + mtc0 t0, CP0_MVPCONTROL + + /* Retrieve the number of VPEs within the core */ + mfc0 t0, CP0_MVPCONF0 + srl t0, t0, MVPCONF0_PVPE_SHIFT + andi t0, t0, (MVPCONF0_PVPE >> MVPCONF0_PVPE_SHIFT) + addi t7, t0, 1 + + /* If there's only 1, we're done */ + beqz t0, 2f + nop + + /* Loop through each VPE within this core */ + li t5, 1 + +1: /* Operate on the appropriate TC */ + mtc0 t5, CP0_VPECONTROL + ehb + + /* Bind TC to VPE (1:1 TC:VPE mapping) */ + mttc0 t5, CP0_TCBIND + + /* Set exclusive TC, non-active, master */ + li t0, VPECONF0_MVP + sll t1, t5, VPECONF0_XTC_SHIFT + or t0, t0, t1 + mttc0 t0, CP0_VPECONF0 + + /* Set TC non-active, non-allocatable */ + mttc0 zero, CP0_TCSTATUS + + /* Set TC halted */ + li t0, TCHALT_H + mttc0 t0, CP0_TCHALT + + /* Next VPE */ + addi t5, t5, 1 + slt t0, t5, t7 + bnez t0, 1b + nop + + /* Leave VPE configuration state */ +2: mfc0 t0, CP0_MVPCONTROL + xori t0, t0, MVPCONTROL_VPC + mtc0 t0, CP0_MVPCONTROL + +3: .set pop +#endif + jr ra + nop + END(mips_cps_core_init) + +LEAF(mips_cps_boot_vpes) + /* Retrieve CM base address */ + la t0, mips_cm_base + lw t0, 0(t0) + + /* Calculate a pointer to this cores struct core_boot_config */ + lw t0, GCR_CL_ID_OFS(t0) + li t1, COREBOOTCFG_SIZE + mul t0, t0, t1 + la t1, mips_cps_core_bootcfg + lw t1, 0(t1) + addu t0, t0, t1 + + /* Calculate this VPEs ID. If the core doesn't support MT use 0 */ + has_mt t6, 1f + li t9, 0 + + /* Find the number of VPEs present in the core */ + mfc0 t1, CP0_MVPCONF0 + srl t1, t1, MVPCONF0_PVPE_SHIFT + andi t1, t1, MVPCONF0_PVPE >> MVPCONF0_PVPE_SHIFT + addi t1, t1, 1 + + /* Calculate a mask for the VPE ID from EBase.CPUNum */ + clz t1, t1 + li t2, 31 + subu t1, t2, t1 + li t2, 1 + sll t1, t2, t1 + addiu t1, t1, -1 + + /* Retrieve the VPE ID from EBase.CPUNum */ + mfc0 t9, $15, 1 + and t9, t9, t1 + +1: /* Calculate a pointer to this VPEs struct vpe_boot_config */ + li t1, VPEBOOTCFG_SIZE + mul v0, t9, t1 + lw t7, COREBOOTCFG_VPECONFIG(t0) + addu v0, v0, t7 + +#ifdef CONFIG_MIPS_MT + + /* If the core doesn't support MT then return */ + bnez t6, 1f + nop + jr ra + nop + + .set push + .set mt + +1: /* Enter VPE configuration state */ + dvpe + la t1, 1f + jr.hb t1 + nop +1: mfc0 t1, CP0_MVPCONTROL + ori t1, t1, MVPCONTROL_VPC + mtc0 t1, CP0_MVPCONTROL + ehb + + /* Loop through each VPE */ + lw t6, COREBOOTCFG_VPEMASK(t0) + move t8, t6 + li t5, 0 + + /* Check whether the VPE should be running. If not, skip it */ +1: andi t0, t6, 1 + beqz t0, 2f + nop + + /* Operate on the appropriate TC */ + mfc0 t0, CP0_VPECONTROL + ori t0, t0, VPECONTROL_TARGTC + xori t0, t0, VPECONTROL_TARGTC + or t0, t0, t5 + mtc0 t0, CP0_VPECONTROL + ehb + + /* Skip the VPE if its TC is not halted */ + mftc0 t0, CP0_TCHALT + beqz t0, 2f + nop + + /* Calculate a pointer to the VPEs struct vpe_boot_config */ + li t0, VPEBOOTCFG_SIZE + mul t0, t0, t5 + addu t0, t0, t7 + + /* Set the TC restart PC */ + lw t1, VPEBOOTCFG_PC(t0) + mttc0 t1, CP0_TCRESTART + + /* Set the TC stack pointer */ + lw t1, VPEBOOTCFG_SP(t0) + mttgpr t1, sp + + /* Set the TC global pointer */ + lw t1, VPEBOOTCFG_GP(t0) + mttgpr t1, gp + + /* Copy config from this VPE */ + mfc0 t0, CP0_CONFIG + mttc0 t0, CP0_CONFIG + + /* Ensure no software interrupts are pending */ + mttc0 zero, CP0_CAUSE + mttc0 zero, CP0_STATUS + + /* Set TC active, not interrupt exempt */ + mftc0 t0, CP0_TCSTATUS + li t1, ~TCSTATUS_IXMT + and t0, t0, t1 + ori t0, t0, TCSTATUS_A + mttc0 t0, CP0_TCSTATUS + + /* Clear the TC halt bit */ + mttc0 zero, CP0_TCHALT + + /* Set VPE active */ + mftc0 t0, CP0_VPECONF0 + ori t0, t0, VPECONF0_VPA + mttc0 t0, CP0_VPECONF0 + + /* Next VPE */ +2: srl t6, t6, 1 + addi t5, t5, 1 + bnez t6, 1b + nop + + /* Leave VPE configuration state */ + mfc0 t1, CP0_MVPCONTROL + xori t1, t1, MVPCONTROL_VPC + mtc0 t1, CP0_MVPCONTROL + ehb + evpe + + /* Check whether this VPE is meant to be running */ + li t0, 1 + sll t0, t0, t9 + and t0, t0, t8 + bnez t0, 2f + nop + + /* This VPE should be offline, halt the TC */ + li t0, TCHALT_H + mtc0 t0, CP0_TCHALT + la t0, 1f +1: jr.hb t0 + nop + +2: .set pop + +#endif /* CONFIG_MIPS_MT */ + + /* Return */ + jr ra + nop + END(mips_cps_boot_vpes) + +#if defined(CONFIG_MIPS_CPS_PM) && defined(CONFIG_CPU_PM) + + /* Calculate a pointer to this CPUs struct mips_static_suspend_state */ + .macro psstate dest + .set push + .set noat + lw $1, TI_CPU(gp) + sll $1, $1, LONGLOG + la \dest, __per_cpu_offset + addu $1, $1, \dest + lw $1, 0($1) + la \dest, cps_cpu_state + addu \dest, \dest, $1 + .set pop + .endm + +LEAF(mips_cps_pm_save) + /* Save CPU state */ + SUSPEND_SAVE_REGS + psstate t1 + SUSPEND_SAVE_STATIC + jr v0 + nop + END(mips_cps_pm_save) + +LEAF(mips_cps_pm_restore) + /* Restore CPU state */ + psstate t1 + RESUME_RESTORE_STATIC + RESUME_RESTORE_REGS_RETURN + END(mips_cps_pm_restore) + +#endif /* CONFIG_MIPS_CPS_PM && CONFIG_CPU_PM */ diff --git a/arch/mips/kernel/cpu-probe.c b/arch/mips/kernel/cpu-probe.c index 6e8fb85ce7c3..d74f957c561e 100644 --- a/arch/mips/kernel/cpu-probe.c +++ b/arch/mips/kernel/cpu-probe.c @@ -62,7 +62,7 @@ static inline void check_errata(void) case CPU_34K: /* * Erratum "RPS May Cause Incorrect Instruction Execution" - * This code only handles VPE0, any SMP/SMTC/RTOS code + * This code only handles VPE0, any SMP/RTOS code * making use of VPE1 will be responsable for that VPE. */ if ((c->processor_id & PRID_REV_MASK) <= PRID_REV_34K_V1_0_2) @@ -423,7 +423,7 @@ static void decode_configs(struct cpuinfo_mips *c) #ifndef CONFIG_MIPS_CPS if (cpu_has_mips_r2) { - c->core = read_c0_ebase() & 0x3ff; + c->core = get_ebase_cpunum(); if (cpu_has_mipsmt) c->core >>= fls(core_nvpes()) - 1; } @@ -684,21 +684,6 @@ static inline void cpu_probe_legacy(struct cpuinfo_mips *c, unsigned int cpu) */ c->tlbsize = (read_c0_info() & (1 << 29)) ? 64 : 48; break; - case PRID_IMP_RM9000: - c->cputype = CPU_RM9000; - __cpu_name[cpu] = "RM9000"; - set_isa(c, MIPS_CPU_ISA_IV); - c->options = R4K_OPTS | MIPS_CPU_FPU | MIPS_CPU_32FPR | - MIPS_CPU_LLSC; - /* - * Bit 29 in the info register of the RM9000 - * indicates if the TLB has 48 or 64 entries. - * - * 29 1 => 64 entry JTLB - * 0 => 48 entry JTLB - */ - c->tlbsize = (read_c0_info() & (1 << 29)) ? 64 : 48; - break; case PRID_IMP_R8000: c->cputype = CPU_R8000; __cpu_name[cpu] = "RM8000"; @@ -1041,6 +1026,7 @@ static inline void cpu_probe_ingenic(struct cpuinfo_mips *c, unsigned int cpu) decode_configs(c); /* JZRISC does not implement the CP0 counter. */ c->options &= ~MIPS_CPU_COUNTER; + BUG_ON(!__builtin_constant_p(cpu_has_counter) || cpu_has_counter); switch (c->processor_id & PRID_IMP_MASK) { case PRID_IMP_JZRISC: c->cputype = CPU_JZRISC; @@ -1074,6 +1060,7 @@ static inline void cpu_probe_netlogic(struct cpuinfo_mips *c, int cpu) switch (c->processor_id & PRID_IMP_MASK) { case PRID_IMP_NETLOGIC_XLP2XX: case PRID_IMP_NETLOGIC_XLP9XX: + case PRID_IMP_NETLOGIC_XLP5XX: c->cputype = CPU_XLP; __cpu_name[cpu] = "Broadcom XLPII"; break; diff --git a/arch/mips/kernel/entry.S b/arch/mips/kernel/entry.S index e5786858cdb6..4353d323f017 100644 --- a/arch/mips/kernel/entry.S +++ b/arch/mips/kernel/entry.S @@ -16,9 +16,6 @@ #include <asm/isadep.h> #include <asm/thread_info.h> #include <asm/war.h> -#ifdef CONFIG_MIPS_MT_SMTC -#include <asm/mipsmtregs.h> -#endif #ifndef CONFIG_PREEMPT #define resume_kernel restore_all @@ -89,41 +86,6 @@ FEXPORT(syscall_exit) bnez t0, syscall_exit_work restore_all: # restore full frame -#ifdef CONFIG_MIPS_MT_SMTC -#ifdef CONFIG_MIPS_MT_SMTC_IM_BACKSTOP -/* Re-arm any temporarily masked interrupts not explicitly "acked" */ - mfc0 v0, CP0_TCSTATUS - ori v1, v0, TCSTATUS_IXMT - mtc0 v1, CP0_TCSTATUS - andi v0, TCSTATUS_IXMT - _ehb - mfc0 t0, CP0_TCCONTEXT - DMT 9 # dmt t1 - jal mips_ihb - mfc0 t2, CP0_STATUS - andi t3, t0, 0xff00 - or t2, t2, t3 - mtc0 t2, CP0_STATUS - _ehb - andi t1, t1, VPECONTROL_TE - beqz t1, 1f - EMT -1: - mfc0 v1, CP0_TCSTATUS - /* We set IXMT above, XOR should clear it here */ - xori v1, v1, TCSTATUS_IXMT - or v1, v0, v1 - mtc0 v1, CP0_TCSTATUS - _ehb - xor t0, t0, t3 - mtc0 t0, CP0_TCCONTEXT -#endif /* CONFIG_MIPS_MT_SMTC_IM_BACKSTOP */ -/* Detect and execute deferred IPI "interrupts" */ - LONG_L s0, TI_REGS($28) - LONG_S sp, TI_REGS($28) - jal deferred_smtc_ipi - LONG_S s0, TI_REGS($28) -#endif /* CONFIG_MIPS_MT_SMTC */ .set noat RESTORE_TEMP RESTORE_AT diff --git a/arch/mips/kernel/genex.S b/arch/mips/kernel/genex.S index a9ce3408be25..ac35e12cb1f3 100644 --- a/arch/mips/kernel/genex.S +++ b/arch/mips/kernel/genex.S @@ -21,20 +21,6 @@ #include <asm/war.h> #include <asm/thread_info.h> -#ifdef CONFIG_MIPS_MT_SMTC -#define PANIC_PIC(msg) \ - .set push; \ - .set nomicromips; \ - .set reorder; \ - PTR_LA a0,8f; \ - .set noat; \ - PTR_LA AT, panic; \ - jr AT; \ -9: b 9b; \ - .set pop; \ - TEXT(msg) -#endif - __INIT /* @@ -251,15 +237,6 @@ NESTED(except_vec_vi, 0, sp) SAVE_AT .set push .set noreorder -#ifdef CONFIG_MIPS_MT_SMTC - /* - * To keep from blindly blocking *all* interrupts - * during service by SMTC kernel, we also want to - * pass the IM value to be cleared. - */ -FEXPORT(except_vec_vi_mori) - ori a0, $0, 0 -#endif /* CONFIG_MIPS_MT_SMTC */ PTR_LA v1, except_vec_vi_handler FEXPORT(except_vec_vi_lui) lui v0, 0 /* Patched */ @@ -277,37 +254,10 @@ EXPORT(except_vec_vi_end) NESTED(except_vec_vi_handler, 0, sp) SAVE_TEMP SAVE_STATIC -#ifdef CONFIG_MIPS_MT_SMTC - /* - * SMTC has an interesting problem that interrupts are level-triggered, - * and the CLI macro will clear EXL, potentially causing a duplicate - * interrupt service invocation. So we need to clear the associated - * IM bit of Status prior to doing CLI, and restore it after the - * service routine has been invoked - we must assume that the - * service routine will have cleared the state, and any active - * level represents a new or otherwised unserviced event... - */ - mfc0 t1, CP0_STATUS - and t0, a0, t1 -#ifdef CONFIG_MIPS_MT_SMTC_IM_BACKSTOP - mfc0 t2, CP0_TCCONTEXT - or t2, t0, t2 - mtc0 t2, CP0_TCCONTEXT -#endif /* CONFIG_MIPS_MT_SMTC_IM_BACKSTOP */ - xor t1, t1, t0 - mtc0 t1, CP0_STATUS - _ehb -#endif /* CONFIG_MIPS_MT_SMTC */ CLI #ifdef CONFIG_TRACE_IRQFLAGS move s0, v0 -#ifdef CONFIG_MIPS_MT_SMTC - move s1, a0 -#endif TRACE_IRQS_OFF -#ifdef CONFIG_MIPS_MT_SMTC - move a0, s1 -#endif move v0, s0 #endif @@ -496,9 +446,6 @@ NESTED(nmi_handler, PT_SIZE, sp) .align 5 LEAF(handle_ri_rdhwr_vivt) -#ifdef CONFIG_MIPS_MT_SMTC - PANIC_PIC("handle_ri_rdhwr_vivt called") -#else .set push .set noat .set noreorder @@ -517,7 +464,6 @@ NESTED(nmi_handler, PT_SIZE, sp) .set pop bltz k1, handle_ri /* slow path */ /* fall thru */ -#endif END(handle_ri_rdhwr_vivt) LEAF(handle_ri_rdhwr) diff --git a/arch/mips/kernel/head.S b/arch/mips/kernel/head.S index e712dcf18b2d..95afd663cd45 100644 --- a/arch/mips/kernel/head.S +++ b/arch/mips/kernel/head.S @@ -35,33 +35,12 @@ */ .macro setup_c0_status set clr .set push -#ifdef CONFIG_MIPS_MT_SMTC - /* - * For SMTC, we need to set privilege and disable interrupts only for - * the current TC, using the TCStatus register. - */ - mfc0 t0, CP0_TCSTATUS - /* Fortunately CU 0 is in the same place in both registers */ - /* Set TCU0, TMX, TKSU (for later inversion) and IXMT */ - li t1, ST0_CU0 | 0x08001c00 - or t0, t1 - /* Clear TKSU, leave IXMT */ - xori t0, 0x00001800 - mtc0 t0, CP0_TCSTATUS - _ehb - /* We need to leave the global IE bit set, but clear EXL...*/ - mfc0 t0, CP0_STATUS - or t0, ST0_CU0 | ST0_EXL | ST0_ERL | \set | \clr - xor t0, ST0_EXL | ST0_ERL | \clr - mtc0 t0, CP0_STATUS -#else mfc0 t0, CP0_STATUS or t0, ST0_CU0|\set|0x1f|\clr xor t0, 0x1f|\clr mtc0 t0, CP0_STATUS .set noreorder sll zero,3 # ehb -#endif .set pop .endm @@ -115,24 +94,6 @@ NESTED(kernel_entry, 16, sp) # kernel entry point jr t0 0: -#ifdef CONFIG_MIPS_MT_SMTC - /* - * In SMTC kernel, "CLI" is thread-specific, in TCStatus. - * We still need to enable interrupts globally in Status, - * and clear EXL/ERL. - * - * TCContext is used to track interrupt levels under - * service in SMTC kernel. Clear for boot TC before - * allowing any interrupts. - */ - mtc0 zero, CP0_TCCONTEXT - - mfc0 t0, CP0_STATUS - ori t0, t0, 0xff1f - xori t0, t0, 0x001e - mtc0 t0, CP0_STATUS -#endif /* CONFIG_MIPS_MT_SMTC */ - PTR_LA t0, __bss_start # clear .bss LONG_S zero, (t0) PTR_LA t1, __bss_stop - LONGSIZE @@ -164,25 +125,8 @@ NESTED(kernel_entry, 16, sp) # kernel entry point * function after setting up the stack and gp registers. */ NESTED(smp_bootstrap, 16, sp) -#ifdef CONFIG_MIPS_MT_SMTC - /* - * Read-modify-writes of Status must be atomic, and this - * is one case where CLI is invoked without EXL being - * necessarily set. The CLI and setup_c0_status will - * in fact be redundant for all but the first TC of - * each VPE being booted. - */ - DMT 10 # dmt t2 /* t0, t1 are used by CLI and setup_c0_status() */ - jal mips_ihb -#endif /* CONFIG_MIPS_MT_SMTC */ smp_slave_setup setup_c0_status_sec -#ifdef CONFIG_MIPS_MT_SMTC - andi t2, t2, VPECONTROL_TE - beqz t2, 2f - EMT # emt -2: -#endif /* CONFIG_MIPS_MT_SMTC */ j start_secondary END(smp_bootstrap) #endif /* CONFIG_SMP */ diff --git a/arch/mips/kernel/i8259.c b/arch/mips/kernel/i8259.c index 2b91fe80c436..50b364897dda 100644 --- a/arch/mips/kernel/i8259.c +++ b/arch/mips/kernel/i8259.c @@ -42,9 +42,6 @@ static struct irq_chip i8259A_chip = { .irq_disable = disable_8259A_irq, .irq_unmask = enable_8259A_irq, .irq_mask_ack = mask_and_ack_8259A, -#ifdef CONFIG_MIPS_MT_SMTC_IRQAFF - .irq_set_affinity = plat_set_irq_affinity, -#endif /* CONFIG_MIPS_MT_SMTC_IRQAFF */ }; /* @@ -180,7 +177,6 @@ handle_real_irq: outb(cached_master_mask, PIC_MASTER_IMR); outb(0x60+irq, PIC_MASTER_CMD); /* 'Specific EOI to master */ } - smtc_im_ack_irq(irq); raw_spin_unlock_irqrestore(&i8259A_lock, flags); return; diff --git a/arch/mips/kernel/idle.c b/arch/mips/kernel/idle.c index 837ff27950bc..09ce45980758 100644 --- a/arch/mips/kernel/idle.c +++ b/arch/mips/kernel/idle.c @@ -224,29 +224,26 @@ void __init check_wait(void) cpu_wait = r4k_wait; */ break; - case CPU_RM9000: - if ((c->processor_id & 0x00ff) >= 0x40) - cpu_wait = r4k_wait; - break; default: break; } } -static void smtc_idle_hook(void) -{ -#ifdef CONFIG_MIPS_MT_SMTC - void smtc_idle_loop_hook(void); - - smtc_idle_loop_hook(); -#endif -} - void arch_cpu_idle(void) { - smtc_idle_hook(); if (cpu_wait) cpu_wait(); else local_irq_enable(); } + +#ifdef CONFIG_CPU_IDLE + +int mips_cpuidle_wait_enter(struct cpuidle_device *dev, + struct cpuidle_driver *drv, int index) +{ + arch_cpu_idle(); + return index; +} + +#endif diff --git a/arch/mips/kernel/irq-gic.c b/arch/mips/kernel/irq-gic.c index 8520dad6d4e3..88e4c323382c 100644 --- a/arch/mips/kernel/irq-gic.c +++ b/arch/mips/kernel/irq-gic.c @@ -54,6 +54,21 @@ void gic_write_compare(cycle_t cnt) (int)(cnt & 0xffffffff)); } +void gic_write_cpu_compare(cycle_t cnt, int cpu) +{ + unsigned long flags; + + local_irq_save(flags); + + GICWRITE(GIC_REG(VPE_LOCAL, GIC_VPE_OTHER_ADDR), cpu); + GICWRITE(GIC_REG(VPE_OTHER, GIC_VPE_COMPARE_HI), + (int)(cnt >> 32)); + GICWRITE(GIC_REG(VPE_OTHER, GIC_VPE_COMPARE_LO), + (int)(cnt & 0xffffffff)); + + local_irq_restore(flags); +} + cycle_t gic_read_compare(void) { unsigned int hi, lo; diff --git a/arch/mips/kernel/irq-msc01.c b/arch/mips/kernel/irq-msc01.c index fab40f7d2e03..4858642d543d 100644 --- a/arch/mips/kernel/irq-msc01.c +++ b/arch/mips/kernel/irq-msc01.c @@ -53,13 +53,9 @@ static inline void unmask_msc_irq(struct irq_data *d) */ static void level_mask_and_ack_msc_irq(struct irq_data *d) { - unsigned int irq = d->irq; - mask_msc_irq(d); if (!cpu_has_veic) MSCIC_WRITE(MSC01_IC_EOI, 0); - /* This actually needs to be a call into platform code */ - smtc_im_ack_irq(irq); } /* @@ -78,7 +74,6 @@ static void edge_mask_and_ack_msc_irq(struct irq_data *d) MSCIC_WRITE(MSC01_IC_SUP+irq*8, r | ~MSC01_IC_SUP_EDGE_BIT); MSCIC_WRITE(MSC01_IC_SUP+irq*8, r); } - smtc_im_ack_irq(irq); } /* diff --git a/arch/mips/kernel/irq.c b/arch/mips/kernel/irq.c index 1818da4dbb85..d2bfbc2e8995 100644 --- a/arch/mips/kernel/irq.c +++ b/arch/mips/kernel/irq.c @@ -73,7 +73,6 @@ void free_irqno(unsigned int irq) */ void ack_bad_irq(unsigned int irq) { - smtc_im_ack_irq(irq); printk("unexpected IRQ # %d\n", irq); } @@ -142,23 +141,7 @@ void __irq_entry do_IRQ(unsigned int irq) { irq_enter(); check_stack_overflow(); - if (!smtc_handle_on_other_cpu(irq)) - generic_handle_irq(irq); - irq_exit(); -} - -#ifdef CONFIG_MIPS_MT_SMTC_IRQAFF -/* - * To avoid inefficient and in some cases pathological re-checking of - * IRQ affinity, we have this variant that skips the affinity check. - */ - -void __irq_entry do_IRQ_no_affinity(unsigned int irq) -{ - irq_enter(); - smtc_im_backstop(irq); generic_handle_irq(irq); irq_exit(); } -#endif /* CONFIG_MIPS_MT_SMTC_IRQAFF */ diff --git a/arch/mips/kernel/mips-cpc.c b/arch/mips/kernel/mips-cpc.c index c9dc67402969..ba473608a347 100644 --- a/arch/mips/kernel/mips-cpc.c +++ b/arch/mips/kernel/mips-cpc.c @@ -9,12 +9,18 @@ */ #include <linux/errno.h> +#include <linux/percpu.h> +#include <linux/spinlock.h> #include <asm/mips-cm.h> #include <asm/mips-cpc.h> void __iomem *mips_cpc_base; +static DEFINE_PER_CPU_ALIGNED(spinlock_t, cpc_core_lock); + +static DEFINE_PER_CPU_ALIGNED(unsigned long, cpc_core_lock_flags); + phys_t __weak mips_cpc_phys_base(void) { u32 cpc_base; @@ -39,6 +45,10 @@ phys_t __weak mips_cpc_phys_base(void) int mips_cpc_probe(void) { phys_t addr; + unsigned cpu; + + for_each_possible_cpu(cpu) + spin_lock_init(&per_cpu(cpc_core_lock, cpu)); addr = mips_cpc_phys_base(); if (!addr) @@ -50,3 +60,21 @@ int mips_cpc_probe(void) return 0; } + +void mips_cpc_lock_other(unsigned int core) +{ + unsigned curr_core; + preempt_disable(); + curr_core = current_cpu_data.core; + spin_lock_irqsave(&per_cpu(cpc_core_lock, curr_core), + per_cpu(cpc_core_lock_flags, curr_core)); + write_cpc_cl_other(core << CPC_Cx_OTHER_CORENUM_SHF); +} + +void mips_cpc_unlock_other(void) +{ + unsigned curr_core = current_cpu_data.core; + spin_unlock_irqrestore(&per_cpu(cpc_core_lock, curr_core), + per_cpu(cpc_core_lock_flags, curr_core)); + preempt_enable(); +} diff --git a/arch/mips/kernel/mips-mt-fpaff.c b/arch/mips/kernel/mips-mt-fpaff.c index cb098628aee8..362bb3707e62 100644 --- a/arch/mips/kernel/mips-mt-fpaff.c +++ b/arch/mips/kernel/mips-mt-fpaff.c @@ -1,5 +1,5 @@ /* - * General MIPS MT support routines, usable in AP/SP, SMVP, or SMTC kernels + * General MIPS MT support routines, usable in AP/SP and SMVP. * Copyright (C) 2005 Mips Technologies, Inc */ #include <linux/cpu.h> diff --git a/arch/mips/kernel/mips-mt.c b/arch/mips/kernel/mips-mt.c index 6ded9bd1489c..88b1ef5f868a 100644 --- a/arch/mips/kernel/mips-mt.c +++ b/arch/mips/kernel/mips-mt.c @@ -1,5 +1,5 @@ /* - * General MIPS MT support routines, usable in AP/SP, SMVP, or SMTC kernels + * General MIPS MT support routines, usable in AP/SP and SMVP. * Copyright (C) 2005 Mips Technologies, Inc */ @@ -57,9 +57,6 @@ void mips_mt_regdump(unsigned long mvpctl) int tc; unsigned long haltval; unsigned long tcstatval; -#ifdef CONFIG_MIPS_MT_SMTC - void smtc_soft_dump(void); -#endif /* CONFIG_MIPT_MT_SMTC */ local_irq_save(flags); vpflags = dvpe(); @@ -116,9 +113,6 @@ void mips_mt_regdump(unsigned long mvpctl) if (!haltval) write_tc_c0_tchalt(0); } -#ifdef CONFIG_MIPS_MT_SMTC - smtc_soft_dump(); -#endif /* CONFIG_MIPT_MT_SMTC */ printk("===========================\n"); evpe(vpflags); local_irq_restore(flags); @@ -295,21 +289,11 @@ void mips_mt_set_cpuoptions(void) void mt_cflush_lockdown(void) { -#ifdef CONFIG_MIPS_MT_SMTC - void smtc_cflush_lockdown(void); - - smtc_cflush_lockdown(); -#endif /* CONFIG_MIPS_MT_SMTC */ /* FILL IN VSMP and AP/SP VERSIONS HERE */ } void mt_cflush_release(void) { -#ifdef CONFIG_MIPS_MT_SMTC - void smtc_cflush_release(void); - - smtc_cflush_release(); -#endif /* CONFIG_MIPS_MT_SMTC */ /* FILL IN VSMP and AP/SP VERSIONS HERE */ } diff --git a/arch/mips/kernel/octeon_switch.S b/arch/mips/kernel/octeon_switch.S index 029e002a4ea0..f6547680c81c 100644 --- a/arch/mips/kernel/octeon_switch.S +++ b/arch/mips/kernel/octeon_switch.S @@ -10,24 +10,12 @@ * Copyright (C) 2000 MIPS Technologies, Inc. * written by Carsten Langgaard, carstenl@mips.com */ -#include <asm/asm.h> -#include <asm/cachectl.h> -#include <asm/fpregdef.h> -#include <asm/mipsregs.h> -#include <asm/asm-offsets.h> -#include <asm/pgtable-bits.h> -#include <asm/regdef.h> -#include <asm/stackframe.h> -#include <asm/thread_info.h> - -#include <asm/asmmacro.h> - -/* - * Offset to the current process status flags, the first 32 bytes of the - * stack are not used. - */ -#define ST_OFF (_THREAD_SIZE - 32 - PT_SIZE + PT_STATUS) +#define USE_ALTERNATE_RESUME_IMPL 1 + .set push + .set arch=mips64r2 +#include "r4k_switch.S" + .set pop /* * task_struct *resume(task_struct *prev, task_struct *next, * struct thread_info *next_ti, int usedfpu) @@ -40,6 +28,61 @@ cpu_save_nonscratch a0 LONG_S ra, THREAD_REG31(a0) + /* + * check if we need to save FPU registers + */ + PTR_L t3, TASK_THREAD_INFO(a0) + LONG_L t0, TI_FLAGS(t3) + li t1, _TIF_USEDFPU + and t2, t0, t1 + beqz t2, 1f + nor t1, zero, t1 + + and t0, t0, t1 + LONG_S t0, TI_FLAGS(t3) + + /* + * clear saved user stack CU1 bit + */ + LONG_L t0, ST_OFF(t3) + li t1, ~ST0_CU1 + and t0, t0, t1 + LONG_S t0, ST_OFF(t3) + + .set push + .set arch=mips64r2 + fpu_save_double a0 t0 t1 # c0_status passed in t0 + # clobbers t1 + .set pop +1: + + /* check if we need to save COP2 registers */ + PTR_L t2, TASK_THREAD_INFO(a0) + LONG_L t0, ST_OFF(t2) + bbit0 t0, 30, 1f + + /* Disable COP2 in the stored process state */ + li t1, ST0_CU2 + xor t0, t1 + LONG_S t0, ST_OFF(t2) + + /* Enable COP2 so we can save it */ + mfc0 t0, CP0_STATUS + or t0, t1 + mtc0 t0, CP0_STATUS + + /* Save COP2 */ + daddu a0, THREAD_CP2 + jal octeon_cop2_save + dsubu a0, THREAD_CP2 + + /* Disable COP2 now that we are done */ + mfc0 t0, CP0_STATUS + li t1, ST0_CU2 + xor t0, t1 + mtc0 t0, CP0_STATUS + +1: #if CONFIG_CAVIUM_OCTEON_CVMSEG_SIZE > 0 /* Check if we need to store CVMSEG state */ mfc0 t0, $11,7 /* CvmMemCtl */ @@ -85,12 +128,7 @@ move $28, a2 cpu_restore_nonscratch a1 -#if (_THREAD_SIZE - 32) < 0x8000 - PTR_ADDIU t0, $28, _THREAD_SIZE - 32 -#else - PTR_LI t0, _THREAD_SIZE - 32 - PTR_ADDU t0, $28 -#endif + PTR_ADDU t0, $28, _THREAD_SIZE - 32 set_saved_sp t0, t1, t2 mfc0 t1, CP0_STATUS /* Do we really need this? */ diff --git a/arch/mips/kernel/pm-cps.c b/arch/mips/kernel/pm-cps.c new file mode 100644 index 000000000000..5aa4c6f8cf83 --- /dev/null +++ b/arch/mips/kernel/pm-cps.c @@ -0,0 +1,716 @@ +/* + * Copyright (C) 2014 Imagination Technologies + * Author: Paul Burton <paul.burton@imgtec.com> + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the + * Free Software Foundation; either version 2 of the License, or (at your + * option) any later version. + */ + +#include <linux/init.h> +#include <linux/percpu.h> +#include <linux/slab.h> + +#include <asm/asm-offsets.h> +#include <asm/cacheflush.h> +#include <asm/cacheops.h> +#include <asm/idle.h> +#include <asm/mips-cm.h> +#include <asm/mips-cpc.h> +#include <asm/mipsmtregs.h> +#include <asm/pm.h> +#include <asm/pm-cps.h> +#include <asm/smp-cps.h> +#include <asm/uasm.h> + +/* + * cps_nc_entry_fn - type of a generated non-coherent state entry function + * @online: the count of online coupled VPEs + * @nc_ready_count: pointer to a non-coherent mapping of the core ready_count + * + * The code entering & exiting non-coherent states is generated at runtime + * using uasm, in order to ensure that the compiler cannot insert a stray + * memory access at an unfortunate time and to allow the generation of optimal + * core-specific code particularly for cache routines. If coupled_coherence + * is non-zero and this is the entry function for the CPS_PM_NC_WAIT state, + * returns the number of VPEs that were in the wait state at the point this + * VPE left it. Returns garbage if coupled_coherence is zero or this is not + * the entry function for CPS_PM_NC_WAIT. + */ +typedef unsigned (*cps_nc_entry_fn)(unsigned online, u32 *nc_ready_count); + +/* + * The entry point of the generated non-coherent idle state entry/exit + * functions. Actually per-core rather than per-CPU. + */ +static DEFINE_PER_CPU_READ_MOSTLY(cps_nc_entry_fn[CPS_PM_STATE_COUNT], + nc_asm_enter); + +/* Bitmap indicating which states are supported by the system */ +DECLARE_BITMAP(state_support, CPS_PM_STATE_COUNT); + +/* + * Indicates the number of coupled VPEs ready to operate in a non-coherent + * state. Actually per-core rather than per-CPU. + */ +static DEFINE_PER_CPU_ALIGNED(u32*, ready_count); +static DEFINE_PER_CPU_ALIGNED(void*, ready_count_alloc); + +/* Indicates online CPUs coupled with the current CPU */ +static DEFINE_PER_CPU_ALIGNED(cpumask_t, online_coupled); + +/* + * Used to synchronize entry to deep idle states. Actually per-core rather + * than per-CPU. + */ +static DEFINE_PER_CPU_ALIGNED(atomic_t, pm_barrier); + +/* Saved CPU state across the CPS_PM_POWER_GATED state */ +DEFINE_PER_CPU_ALIGNED(struct mips_static_suspend_state, cps_cpu_state); + +/* A somewhat arbitrary number of labels & relocs for uasm */ +static struct uasm_label labels[32] __initdata; +static struct uasm_reloc relocs[32] __initdata; + +/* CPU dependant sync types */ +static unsigned stype_intervention; +static unsigned stype_memory; +static unsigned stype_ordering; + +enum mips_reg { + zero, at, v0, v1, a0, a1, a2, a3, + t0, t1, t2, t3, t4, t5, t6, t7, + s0, s1, s2, s3, s4, s5, s6, s7, + t8, t9, k0, k1, gp, sp, fp, ra, +}; + +bool cps_pm_support_state(enum cps_pm_state state) +{ + return test_bit(state, state_support); +} + +static void coupled_barrier(atomic_t *a, unsigned online) +{ + /* + * This function is effectively the same as + * cpuidle_coupled_parallel_barrier, which can't be used here since + * there's no cpuidle device. + */ + + if (!coupled_coherence) + return; + + smp_mb__before_atomic_inc(); + atomic_inc(a); + + while (atomic_read(a) < online) + cpu_relax(); + + if (atomic_inc_return(a) == online * 2) { + atomic_set(a, 0); + return; + } + + while (atomic_read(a) > online) + cpu_relax(); +} + +int cps_pm_enter_state(enum cps_pm_state state) +{ + unsigned cpu = smp_processor_id(); + unsigned core = current_cpu_data.core; + unsigned online, left; + cpumask_t *coupled_mask = this_cpu_ptr(&online_coupled); + u32 *core_ready_count, *nc_core_ready_count; + void *nc_addr; + cps_nc_entry_fn entry; + struct core_boot_config *core_cfg; + struct vpe_boot_config *vpe_cfg; + + /* Check that there is an entry function for this state */ + entry = per_cpu(nc_asm_enter, core)[state]; + if (!entry) + return -EINVAL; + + /* Calculate which coupled CPUs (VPEs) are online */ +#ifdef CONFIG_MIPS_MT + if (cpu_online(cpu)) { + cpumask_and(coupled_mask, cpu_online_mask, + &cpu_sibling_map[cpu]); + online = cpumask_weight(coupled_mask); + cpumask_clear_cpu(cpu, coupled_mask); + } else +#endif + { + cpumask_clear(coupled_mask); + online = 1; + } + + /* Setup the VPE to run mips_cps_pm_restore when started again */ + if (config_enabled(CONFIG_CPU_PM) && state == CPS_PM_POWER_GATED) { + core_cfg = &mips_cps_core_bootcfg[core]; + vpe_cfg = &core_cfg->vpe_config[current_cpu_data.vpe_id]; + vpe_cfg->pc = (unsigned long)mips_cps_pm_restore; + vpe_cfg->gp = (unsigned long)current_thread_info(); + vpe_cfg->sp = 0; + } + + /* Indicate that this CPU might not be coherent */ + cpumask_clear_cpu(cpu, &cpu_coherent_mask); + smp_mb__after_clear_bit(); + + /* Create a non-coherent mapping of the core ready_count */ + core_ready_count = per_cpu(ready_count, core); + nc_addr = kmap_noncoherent(virt_to_page(core_ready_count), + (unsigned long)core_ready_count); + nc_addr += ((unsigned long)core_ready_count & ~PAGE_MASK); + nc_core_ready_count = nc_addr; + + /* Ensure ready_count is zero-initialised before the assembly runs */ + ACCESS_ONCE(*nc_core_ready_count) = 0; + coupled_barrier(&per_cpu(pm_barrier, core), online); + + /* Run the generated entry code */ + left = entry(online, nc_core_ready_count); + + /* Remove the non-coherent mapping of ready_count */ + kunmap_noncoherent(); + + /* Indicate that this CPU is definitely coherent */ + cpumask_set_cpu(cpu, &cpu_coherent_mask); + + /* + * If this VPE is the first to leave the non-coherent wait state then + * it needs to wake up any coupled VPEs still running their wait + * instruction so that they return to cpuidle, which can then complete + * coordination between the coupled VPEs & provide the governor with + * a chance to reflect on the length of time the VPEs were in the + * idle state. + */ + if (coupled_coherence && (state == CPS_PM_NC_WAIT) && (left == online)) + arch_send_call_function_ipi_mask(coupled_mask); + + return 0; +} + +static void __init cps_gen_cache_routine(u32 **pp, struct uasm_label **pl, + struct uasm_reloc **pr, + const struct cache_desc *cache, + unsigned op, int lbl) +{ + unsigned cache_size = cache->ways << cache->waybit; + unsigned i; + const unsigned unroll_lines = 32; + + /* If the cache isn't present this function has it easy */ + if (cache->flags & MIPS_CACHE_NOT_PRESENT) + return; + + /* Load base address */ + UASM_i_LA(pp, t0, (long)CKSEG0); + + /* Calculate end address */ + if (cache_size < 0x8000) + uasm_i_addiu(pp, t1, t0, cache_size); + else + UASM_i_LA(pp, t1, (long)(CKSEG0 + cache_size)); + + /* Start of cache op loop */ + uasm_build_label(pl, *pp, lbl); + + /* Generate the cache ops */ + for (i = 0; i < unroll_lines; i++) + uasm_i_cache(pp, op, i * cache->linesz, t0); + + /* Update the base address */ + uasm_i_addiu(pp, t0, t0, unroll_lines * cache->linesz); + + /* Loop if we haven't reached the end address yet */ + uasm_il_bne(pp, pr, t0, t1, lbl); + uasm_i_nop(pp); +} + +static int __init cps_gen_flush_fsb(u32 **pp, struct uasm_label **pl, + struct uasm_reloc **pr, + const struct cpuinfo_mips *cpu_info, + int lbl) +{ + unsigned i, fsb_size = 8; + unsigned num_loads = (fsb_size * 3) / 2; + unsigned line_stride = 2; + unsigned line_size = cpu_info->dcache.linesz; + unsigned perf_counter, perf_event; + unsigned revision = cpu_info->processor_id & PRID_REV_MASK; + + /* + * Determine whether this CPU requires an FSB flush, and if so which + * performance counter/event reflect stalls due to a full FSB. + */ + switch (__get_cpu_type(cpu_info->cputype)) { + case CPU_INTERAPTIV: + perf_counter = 1; + perf_event = 51; + break; + + case CPU_PROAPTIV: + /* Newer proAptiv cores don't require this workaround */ + if (revision >= PRID_REV_ENCODE_332(1, 1, 0)) + return 0; + + /* On older ones it's unavailable */ + return -1; + + /* CPUs which do not require the workaround */ + case CPU_P5600: + return 0; + + default: + WARN_ONCE(1, "pm-cps: FSB flush unsupported for this CPU\n"); + return -1; + } + + /* + * Ensure that the fill/store buffer (FSB) is not holding the results + * of a prefetch, since if it is then the CPC sequencer may become + * stuck in the D3 (ClrBus) state whilst entering a low power state. + */ + + /* Preserve perf counter setup */ + uasm_i_mfc0(pp, t2, 25, (perf_counter * 2) + 0); /* PerfCtlN */ + uasm_i_mfc0(pp, t3, 25, (perf_counter * 2) + 1); /* PerfCntN */ + + /* Setup perf counter to count FSB full pipeline stalls */ + uasm_i_addiu(pp, t0, zero, (perf_event << 5) | 0xf); + uasm_i_mtc0(pp, t0, 25, (perf_counter * 2) + 0); /* PerfCtlN */ + uasm_i_ehb(pp); + uasm_i_mtc0(pp, zero, 25, (perf_counter * 2) + 1); /* PerfCntN */ + uasm_i_ehb(pp); + + /* Base address for loads */ + UASM_i_LA(pp, t0, (long)CKSEG0); + + /* Start of clear loop */ + uasm_build_label(pl, *pp, lbl); + + /* Perform some loads to fill the FSB */ + for (i = 0; i < num_loads; i++) + uasm_i_lw(pp, zero, i * line_size * line_stride, t0); + + /* + * Invalidate the new D-cache entries so that the cache will need + * refilling (via the FSB) if the loop is executed again. + */ + for (i = 0; i < num_loads; i++) { + uasm_i_cache(pp, Hit_Invalidate_D, + i * line_size * line_stride, t0); + uasm_i_cache(pp, Hit_Writeback_Inv_SD, + i * line_size * line_stride, t0); + } + + /* Completion barrier */ + uasm_i_sync(pp, stype_memory); + uasm_i_ehb(pp); + + /* Check whether the pipeline stalled due to the FSB being full */ + uasm_i_mfc0(pp, t1, 25, (perf_counter * 2) + 1); /* PerfCntN */ + + /* Loop if it didn't */ + uasm_il_beqz(pp, pr, t1, lbl); + uasm_i_nop(pp); + + /* Restore perf counter 1. The count may well now be wrong... */ + uasm_i_mtc0(pp, t2, 25, (perf_counter * 2) + 0); /* PerfCtlN */ + uasm_i_ehb(pp); + uasm_i_mtc0(pp, t3, 25, (perf_counter * 2) + 1); /* PerfCntN */ + uasm_i_ehb(pp); + + return 0; +} + +static void __init cps_gen_set_top_bit(u32 **pp, struct uasm_label **pl, + struct uasm_reloc **pr, + unsigned r_addr, int lbl) +{ + uasm_i_lui(pp, t0, uasm_rel_hi(0x80000000)); + uasm_build_label(pl, *pp, lbl); + uasm_i_ll(pp, t1, 0, r_addr); + uasm_i_or(pp, t1, t1, t0); + uasm_i_sc(pp, t1, 0, r_addr); + uasm_il_beqz(pp, pr, t1, lbl); + uasm_i_nop(pp); +} + +static void * __init cps_gen_entry_code(unsigned cpu, enum cps_pm_state state) +{ + struct uasm_label *l = labels; + struct uasm_reloc *r = relocs; + u32 *buf, *p; + const unsigned r_online = a0; + const unsigned r_nc_count = a1; + const unsigned r_pcohctl = t7; + const unsigned max_instrs = 256; + unsigned cpc_cmd; + int err; + enum { + lbl_incready = 1, + lbl_poll_cont, + lbl_secondary_hang, + lbl_disable_coherence, + lbl_flush_fsb, + lbl_invicache, + lbl_flushdcache, + lbl_hang, + lbl_set_cont, + lbl_secondary_cont, + lbl_decready, + }; + + /* Allocate a buffer to hold the generated code */ + p = buf = kcalloc(max_instrs, sizeof(u32), GFP_KERNEL); + if (!buf) + return NULL; + + /* Clear labels & relocs ready for (re)use */ + memset(labels, 0, sizeof(labels)); + memset(relocs, 0, sizeof(relocs)); + + if (config_enabled(CONFIG_CPU_PM) && state == CPS_PM_POWER_GATED) { + /* + * Save CPU state. Note the non-standard calling convention + * with the return address placed in v0 to avoid clobbering + * the ra register before it is saved. + */ + UASM_i_LA(&p, t0, (long)mips_cps_pm_save); + uasm_i_jalr(&p, v0, t0); + uasm_i_nop(&p); + } + + /* + * Load addresses of required CM & CPC registers. This is done early + * because they're needed in both the enable & disable coherence steps + * but in the coupled case the enable step will only run on one VPE. + */ + UASM_i_LA(&p, r_pcohctl, (long)addr_gcr_cl_coherence()); + + if (coupled_coherence) { + /* Increment ready_count */ + uasm_i_sync(&p, stype_ordering); + uasm_build_label(&l, p, lbl_incready); + uasm_i_ll(&p, t1, 0, r_nc_count); + uasm_i_addiu(&p, t2, t1, 1); + uasm_i_sc(&p, t2, 0, r_nc_count); + uasm_il_beqz(&p, &r, t2, lbl_incready); + uasm_i_addiu(&p, t1, t1, 1); + + /* Ordering barrier */ + uasm_i_sync(&p, stype_ordering); + + /* + * If this is the last VPE to become ready for non-coherence + * then it should branch below. + */ + uasm_il_beq(&p, &r, t1, r_online, lbl_disable_coherence); + uasm_i_nop(&p); + + if (state < CPS_PM_POWER_GATED) { + /* + * Otherwise this is not the last VPE to become ready + * for non-coherence. It needs to wait until coherence + * has been disabled before proceeding, which it will do + * by polling for the top bit of ready_count being set. + */ + uasm_i_addiu(&p, t1, zero, -1); + uasm_build_label(&l, p, lbl_poll_cont); + uasm_i_lw(&p, t0, 0, r_nc_count); + uasm_il_bltz(&p, &r, t0, lbl_secondary_cont); + uasm_i_ehb(&p); + uasm_i_yield(&p, zero, t1); + uasm_il_b(&p, &r, lbl_poll_cont); + uasm_i_nop(&p); + } else { + /* + * The core will lose power & this VPE will not continue + * so it can simply halt here. + */ + uasm_i_addiu(&p, t0, zero, TCHALT_H); + uasm_i_mtc0(&p, t0, 2, 4); + uasm_build_label(&l, p, lbl_secondary_hang); + uasm_il_b(&p, &r, lbl_secondary_hang); + uasm_i_nop(&p); + } + } + + /* + * This is the point of no return - this VPE will now proceed to + * disable coherence. At this point we *must* be sure that no other + * VPE within the core will interfere with the L1 dcache. + */ + uasm_build_label(&l, p, lbl_disable_coherence); + + /* Invalidate the L1 icache */ + cps_gen_cache_routine(&p, &l, &r, &cpu_data[cpu].icache, + Index_Invalidate_I, lbl_invicache); + + /* Writeback & invalidate the L1 dcache */ + cps_gen_cache_routine(&p, &l, &r, &cpu_data[cpu].dcache, + Index_Writeback_Inv_D, lbl_flushdcache); + + /* Completion barrier */ + uasm_i_sync(&p, stype_memory); + uasm_i_ehb(&p); + + /* + * Disable all but self interventions. The load from COHCTL is defined + * by the interAptiv & proAptiv SUMs as ensuring that the operation + * resulting from the preceeding store is complete. + */ + uasm_i_addiu(&p, t0, zero, 1 << cpu_data[cpu].core); + uasm_i_sw(&p, t0, 0, r_pcohctl); + uasm_i_lw(&p, t0, 0, r_pcohctl); + + /* Sync to ensure previous interventions are complete */ + uasm_i_sync(&p, stype_intervention); + uasm_i_ehb(&p); + + /* Disable coherence */ + uasm_i_sw(&p, zero, 0, r_pcohctl); + uasm_i_lw(&p, t0, 0, r_pcohctl); + + if (state >= CPS_PM_CLOCK_GATED) { + err = cps_gen_flush_fsb(&p, &l, &r, &cpu_data[cpu], + lbl_flush_fsb); + if (err) + goto out_err; + + /* Determine the CPC command to issue */ + switch (state) { + case CPS_PM_CLOCK_GATED: + cpc_cmd = CPC_Cx_CMD_CLOCKOFF; + break; + case CPS_PM_POWER_GATED: + cpc_cmd = CPC_Cx_CMD_PWRDOWN; + break; + default: + BUG(); + goto out_err; + } + + /* Issue the CPC command */ + UASM_i_LA(&p, t0, (long)addr_cpc_cl_cmd()); + uasm_i_addiu(&p, t1, zero, cpc_cmd); + uasm_i_sw(&p, t1, 0, t0); + + if (state == CPS_PM_POWER_GATED) { + /* If anything goes wrong just hang */ + uasm_build_label(&l, p, lbl_hang); + uasm_il_b(&p, &r, lbl_hang); + uasm_i_nop(&p); + + /* + * There's no point generating more code, the core is + * powered down & if powered back up will run from the + * reset vector not from here. + */ + goto gen_done; + } + + /* Completion barrier */ + uasm_i_sync(&p, stype_memory); + uasm_i_ehb(&p); + } + + if (state == CPS_PM_NC_WAIT) { + /* + * At this point it is safe for all VPEs to proceed with + * execution. This VPE will set the top bit of ready_count + * to indicate to the other VPEs that they may continue. + */ + if (coupled_coherence) + cps_gen_set_top_bit(&p, &l, &r, r_nc_count, + lbl_set_cont); + + /* + * VPEs which did not disable coherence will continue + * executing, after coherence has been disabled, from this + * point. + */ + uasm_build_label(&l, p, lbl_secondary_cont); + + /* Now perform our wait */ + uasm_i_wait(&p, 0); + } + + /* + * Re-enable coherence. Note that for CPS_PM_NC_WAIT all coupled VPEs + * will run this. The first will actually re-enable coherence & the + * rest will just be performing a rather unusual nop. + */ + uasm_i_addiu(&p, t0, zero, CM_GCR_Cx_COHERENCE_COHDOMAINEN_MSK); + uasm_i_sw(&p, t0, 0, r_pcohctl); + uasm_i_lw(&p, t0, 0, r_pcohctl); + + /* Completion barrier */ + uasm_i_sync(&p, stype_memory); + uasm_i_ehb(&p); + + if (coupled_coherence && (state == CPS_PM_NC_WAIT)) { + /* Decrement ready_count */ + uasm_build_label(&l, p, lbl_decready); + uasm_i_sync(&p, stype_ordering); + uasm_i_ll(&p, t1, 0, r_nc_count); + uasm_i_addiu(&p, t2, t1, -1); + uasm_i_sc(&p, t2, 0, r_nc_count); + uasm_il_beqz(&p, &r, t2, lbl_decready); + uasm_i_andi(&p, v0, t1, (1 << fls(smp_num_siblings)) - 1); + + /* Ordering barrier */ + uasm_i_sync(&p, stype_ordering); + } + + if (coupled_coherence && (state == CPS_PM_CLOCK_GATED)) { + /* + * At this point it is safe for all VPEs to proceed with + * execution. This VPE will set the top bit of ready_count + * to indicate to the other VPEs that they may continue. + */ + cps_gen_set_top_bit(&p, &l, &r, r_nc_count, lbl_set_cont); + + /* + * This core will be reliant upon another core sending a + * power-up command to the CPC in order to resume operation. + * Thus an arbitrary VPE can't trigger the core leaving the + * idle state and the one that disables coherence might as well + * be the one to re-enable it. The rest will continue from here + * after that has been done. + */ + uasm_build_label(&l, p, lbl_secondary_cont); + + /* Ordering barrier */ + uasm_i_sync(&p, stype_ordering); + } + + /* The core is coherent, time to return to C code */ + uasm_i_jr(&p, ra); + uasm_i_nop(&p); + +gen_done: + /* Ensure the code didn't exceed the resources allocated for it */ + BUG_ON((p - buf) > max_instrs); + BUG_ON((l - labels) > ARRAY_SIZE(labels)); + BUG_ON((r - relocs) > ARRAY_SIZE(relocs)); + + /* Patch branch offsets */ + uasm_resolve_relocs(relocs, labels); + + /* Flush the icache */ + local_flush_icache_range((unsigned long)buf, (unsigned long)p); + + return buf; +out_err: + kfree(buf); + return NULL; +} + +static int __init cps_gen_core_entries(unsigned cpu) +{ + enum cps_pm_state state; + unsigned core = cpu_data[cpu].core; + unsigned dlinesz = cpu_data[cpu].dcache.linesz; + void *entry_fn, *core_rc; + + for (state = CPS_PM_NC_WAIT; state < CPS_PM_STATE_COUNT; state++) { + if (per_cpu(nc_asm_enter, core)[state]) + continue; + if (!test_bit(state, state_support)) + continue; + + entry_fn = cps_gen_entry_code(cpu, state); + if (!entry_fn) { + pr_err("Failed to generate core %u state %u entry\n", + core, state); + clear_bit(state, state_support); + } + + per_cpu(nc_asm_enter, core)[state] = entry_fn; + } + + if (!per_cpu(ready_count, core)) { + core_rc = kmalloc(dlinesz * 2, GFP_KERNEL); + if (!core_rc) { + pr_err("Failed allocate core %u ready_count\n", core); + return -ENOMEM; + } + per_cpu(ready_count_alloc, core) = core_rc; + + /* Ensure ready_count is aligned to a cacheline boundary */ + core_rc += dlinesz - 1; + core_rc = (void *)((unsigned long)core_rc & ~(dlinesz - 1)); + per_cpu(ready_count, core) = core_rc; + } + + return 0; +} + +static int __init cps_pm_init(void) +{ + unsigned cpu; + int err; + + /* Detect appropriate sync types for the system */ + switch (current_cpu_data.cputype) { + case CPU_INTERAPTIV: + case CPU_PROAPTIV: + case CPU_M5150: + case CPU_P5600: + stype_intervention = 0x2; + stype_memory = 0x3; + stype_ordering = 0x10; + break; + + default: + pr_warn("Power management is using heavyweight sync 0\n"); + } + + /* A CM is required for all non-coherent states */ + if (!mips_cm_present()) { + pr_warn("pm-cps: no CM, non-coherent states unavailable\n"); + goto out; + } + + /* + * If interrupts were enabled whilst running a wait instruction on a + * non-coherent core then the VPE may end up processing interrupts + * whilst non-coherent. That would be bad. + */ + if (cpu_wait == r4k_wait_irqoff) + set_bit(CPS_PM_NC_WAIT, state_support); + else + pr_warn("pm-cps: non-coherent wait unavailable\n"); + + /* Detect whether a CPC is present */ + if (mips_cpc_present()) { + /* Detect whether clock gating is implemented */ + if (read_cpc_cl_stat_conf() & CPC_Cx_STAT_CONF_CLKGAT_IMPL_MSK) + set_bit(CPS_PM_CLOCK_GATED, state_support); + else + pr_warn("pm-cps: CPC does not support clock gating\n"); + + /* Power gating is available with CPS SMP & any CPC */ + if (mips_cps_smp_in_use()) + set_bit(CPS_PM_POWER_GATED, state_support); + else + pr_warn("pm-cps: CPS SMP not in use, power gating unavailable\n"); + } else { + pr_warn("pm-cps: no CPC, clock & power gating unavailable\n"); + } + + for_each_present_cpu(cpu) { + err = cps_gen_core_entries(cpu); + if (err) + return err; + } +out: + return 0; +} +arch_initcall(cps_pm_init); diff --git a/arch/mips/kernel/pm.c b/arch/mips/kernel/pm.c new file mode 100644 index 000000000000..fefdf39d3df3 --- /dev/null +++ b/arch/mips/kernel/pm.c @@ -0,0 +1,99 @@ +/* + * Copyright (C) 2014 Imagination Technologies Ltd. + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the + * Free Software Foundation; either version 2 of the License, or (at your + * option) any later version. + * + * CPU PM notifiers for saving/restoring general CPU state. + */ + +#include <linux/cpu_pm.h> +#include <linux/init.h> + +#include <asm/dsp.h> +#include <asm/fpu.h> +#include <asm/mmu_context.h> +#include <asm/pm.h> +#include <asm/watch.h> + +/* Used by PM helper macros in asm/pm.h */ +struct mips_static_suspend_state mips_static_suspend_state; + +/** + * mips_cpu_save() - Save general CPU state. + * Ensures that general CPU context is saved, notably FPU and DSP. + */ +static int mips_cpu_save(void) +{ + /* Save FPU state */ + lose_fpu(1); + + /* Save DSP state */ + save_dsp(current); + + return 0; +} + +/** + * mips_cpu_restore() - Restore general CPU state. + * Restores important CPU context. + */ +static void mips_cpu_restore(void) +{ + unsigned int cpu = smp_processor_id(); + + /* Restore ASID */ + if (current->mm) + write_c0_entryhi(cpu_asid(cpu, current->mm)); + + /* Restore DSP state */ + restore_dsp(current); + + /* Restore UserLocal */ + if (cpu_has_userlocal) + write_c0_userlocal(current_thread_info()->tp_value); + + /* Restore watch registers */ + __restore_watch(); +} + +/** + * mips_pm_notifier() - Notifier for preserving general CPU context. + * @self: Notifier block. + * @cmd: CPU PM event. + * @v: Private data (unused). + * + * This is called when a CPU power management event occurs, and is used to + * ensure that important CPU context is preserved across a CPU power down. + */ +static int mips_pm_notifier(struct notifier_block *self, unsigned long cmd, + void *v) +{ + int ret; + + switch (cmd) { + case CPU_PM_ENTER: + ret = mips_cpu_save(); + if (ret) + return NOTIFY_STOP; + break; + case CPU_PM_ENTER_FAILED: + case CPU_PM_EXIT: + mips_cpu_restore(); + break; + } + + return NOTIFY_OK; +} + +static struct notifier_block mips_pm_notifier_block = { + .notifier_call = mips_pm_notifier, +}; + +static int __init mips_pm_init(void) +{ + return cpu_pm_register_notifier(&mips_pm_notifier_block); +} +arch_initcall(mips_pm_init); diff --git a/arch/mips/kernel/process.c b/arch/mips/kernel/process.c index 60e39dc7f1eb..0a1ec0f3beff 100644 --- a/arch/mips/kernel/process.c +++ b/arch/mips/kernel/process.c @@ -140,13 +140,6 @@ int copy_thread(unsigned long clone_flags, unsigned long usp, */ childregs->cp0_status &= ~(ST0_CU2|ST0_CU1); -#ifdef CONFIG_MIPS_MT_SMTC - /* - * SMTC restores TCStatus after Status, and the CU bits - * are aliased there. - */ - childregs->cp0_tcstatus &= ~(ST0_CU2|ST0_CU1); -#endif clear_tsk_thread_flag(p, TIF_USEDFPU); #ifdef CONFIG_MIPS_MT_FPAFF diff --git a/arch/mips/kernel/r4k_switch.S b/arch/mips/kernel/r4k_switch.S index abacac7c33ef..81ca3f70fe29 100644 --- a/arch/mips/kernel/r4k_switch.S +++ b/arch/mips/kernel/r4k_switch.S @@ -28,6 +28,7 @@ */ #define ST_OFF (_THREAD_SIZE - 32 - PT_SIZE + PT_STATUS) +#ifndef USE_ALTERNATE_RESUME_IMPL /* * task_struct *resume(task_struct *prev, task_struct *next, * struct thread_info *next_ti, s32 fp_save) @@ -87,18 +88,6 @@ PTR_ADDU t0, $28, _THREAD_SIZE - 32 set_saved_sp t0, t1, t2 -#ifdef CONFIG_MIPS_MT_SMTC - /* Read-modify-writes of Status must be atomic on a VPE */ - mfc0 t2, CP0_TCSTATUS - ori t1, t2, TCSTATUS_IXMT - mtc0 t1, CP0_TCSTATUS - andi t2, t2, TCSTATUS_IXMT - _ehb - DMT 8 # dmt t0 - move t1,ra - jal mips_ihb - move ra,t1 -#endif /* CONFIG_MIPS_MT_SMTC */ mfc0 t1, CP0_STATUS /* Do we really need this? */ li a3, 0xff01 and t1, a3 @@ -107,22 +96,12 @@ and a2, a3 or a2, t1 mtc0 a2, CP0_STATUS -#ifdef CONFIG_MIPS_MT_SMTC - _ehb - andi t0, t0, VPECONTROL_TE - beqz t0, 1f - emt -1: - mfc0 t1, CP0_TCSTATUS - xori t1, t1, TCSTATUS_IXMT - or t1, t1, t2 - mtc0 t1, CP0_TCSTATUS - _ehb -#endif /* CONFIG_MIPS_MT_SMTC */ move v0, a0 jr ra END(resume) +#endif /* USE_ALTERNATE_RESUME_IMPL */ + /* * Save a thread's fp context. */ @@ -176,19 +155,10 @@ LEAF(_restore_msa) #define FPU_DEFAULT 0x00000000 LEAF(_init_fpu) -#ifdef CONFIG_MIPS_MT_SMTC - /* Rather than manipulate per-VPE Status, set per-TC bit in TCStatus */ - mfc0 t0, CP0_TCSTATUS - /* Bit position is the same for Status, TCStatus */ - li t1, ST0_CU1 - or t0, t1 - mtc0 t0, CP0_TCSTATUS -#else /* Normal MIPS CU1 enable */ mfc0 t0, CP0_STATUS li t1, ST0_CU1 or t0, t1 mtc0 t0, CP0_STATUS -#endif /* CONFIG_MIPS_MT_SMTC */ enable_fpu_hazard li t1, FPU_DEFAULT diff --git a/arch/mips/kernel/rtlx-mt.c b/arch/mips/kernel/rtlx-mt.c index 9c1aca00fd54..5a66b975989e 100644 --- a/arch/mips/kernel/rtlx-mt.c +++ b/arch/mips/kernel/rtlx-mt.c @@ -36,7 +36,6 @@ static irqreturn_t rtlx_interrupt(int irq, void *dev_id) unsigned long flags; int i; - /* Ought not to be strictly necessary for SMTC builds */ local_irq_save(flags); vpeflags = dvpe(); set_c0_status(0x100 << MIPS_CPU_RTLX_IRQ); diff --git a/arch/mips/kernel/smp-bmips.c b/arch/mips/kernel/smp-bmips.c index ea4c2dc31692..df9e2bd9b2c2 100644 --- a/arch/mips/kernel/smp-bmips.c +++ b/arch/mips/kernel/smp-bmips.c @@ -281,13 +281,6 @@ static void bmips_smp_finish(void) } /* - * Runs on CPU0 after all CPUs have been booted - */ -static void bmips_cpus_done(void) -{ -} - -/* * BMIPS5000 raceless IPIs * * Each CPU has two inbound SW IRQs which are independent of all other CPUs. @@ -434,7 +427,6 @@ struct plat_smp_ops bmips43xx_smp_ops = { .boot_secondary = bmips_boot_secondary, .smp_finish = bmips_smp_finish, .init_secondary = bmips_init_secondary, - .cpus_done = bmips_cpus_done, .send_ipi_single = bmips43xx_send_ipi_single, .send_ipi_mask = bmips43xx_send_ipi_mask, #ifdef CONFIG_HOTPLUG_CPU @@ -449,7 +441,6 @@ struct plat_smp_ops bmips5000_smp_ops = { .boot_secondary = bmips_boot_secondary, .smp_finish = bmips_smp_finish, .init_secondary = bmips_init_secondary, - .cpus_done = bmips_cpus_done, .send_ipi_single = bmips5000_send_ipi_single, .send_ipi_mask = bmips5000_send_ipi_mask, #ifdef CONFIG_HOTPLUG_CPU diff --git a/arch/mips/kernel/smp-cmp.c b/arch/mips/kernel/smp-cmp.c index 3ef55fb7ac03..fc8a51553426 100644 --- a/arch/mips/kernel/smp-cmp.c +++ b/arch/mips/kernel/smp-cmp.c @@ -49,14 +49,11 @@ static void cmp_init_secondary(void) /* Enable per-cpu interrupts: platform specific */ -#if defined(CONFIG_MIPS_MT_SMP) || defined(CONFIG_MIPS_MT_SMTC) +#ifdef CONFIG_MIPS_MT_SMP if (cpu_has_mipsmt) c->vpe_id = (read_c0_tcbind() >> TCBIND_CURVPE_SHIFT) & TCBIND_CURVPE; #endif -#ifdef CONFIG_MIPS_MT_SMTC - c->tc_id = (read_c0_tcbind() & TCBIND_CURTC) >> TCBIND_CURTC_SHIFT; -#endif } static void cmp_smp_finish(void) @@ -75,11 +72,6 @@ static void cmp_smp_finish(void) local_irq_enable(); } -static void cmp_cpus_done(void) -{ - pr_debug("SMPCMP: CPU%d: %s\n", smp_processor_id(), __func__); -} - /* * Setup the PC, SP, and GP of a secondary processor and start it running * smp_bootstrap is the place to resume from @@ -135,10 +127,6 @@ void __init cmp_smp_setup(void) unsigned int mvpconf0 = read_c0_mvpconf0(); nvpe = ((mvpconf0 & MVPCONF0_PVPE) >> MVPCONF0_PVPE_SHIFT) + 1; -#elif defined(CONFIG_MIPS_MT_SMTC) - unsigned int mvpconf0 = read_c0_mvpconf0(); - - nvpe = ((mvpconf0 & MVPCONF0_PTC) >> MVPCONF0_PTC_SHIFT) + 1; #endif smp_num_siblings = nvpe; } @@ -165,7 +153,6 @@ struct plat_smp_ops cmp_smp_ops = { .send_ipi_mask = gic_send_ipi_mask, .init_secondary = cmp_init_secondary, .smp_finish = cmp_smp_finish, - .cpus_done = cmp_cpus_done, .boot_secondary = cmp_boot_secondary, .smp_setup = cmp_smp_setup, .prepare_cpus = cmp_prepare_cpus, diff --git a/arch/mips/kernel/smp-cps.c b/arch/mips/kernel/smp-cps.c index 536eec0d21b6..df0598d9bfdd 100644 --- a/arch/mips/kernel/smp-cps.c +++ b/arch/mips/kernel/smp-cps.c @@ -20,104 +20,43 @@ #include <asm/mips-cpc.h> #include <asm/mips_mt.h> #include <asm/mipsregs.h> +#include <asm/pm-cps.h> #include <asm/smp-cps.h> #include <asm/time.h> #include <asm/uasm.h> static DECLARE_BITMAP(core_power, NR_CPUS); -struct boot_config mips_cps_bootcfg; +struct core_boot_config *mips_cps_core_bootcfg; -static void init_core(void) +static unsigned core_vpe_count(unsigned core) { - unsigned int nvpes, t; - u32 mvpconf0, vpeconf0, vpecontrol, tcstatus, tcbind, status; + unsigned cfg; - if (!cpu_has_mipsmt) - return; - - /* Enter VPE configuration state */ - dvpe(); - set_c0_mvpcontrol(MVPCONTROL_VPC); - - /* Retrieve the count of VPEs in this core */ - mvpconf0 = read_c0_mvpconf0(); - nvpes = ((mvpconf0 & MVPCONF0_PVPE) >> MVPCONF0_PVPE_SHIFT) + 1; - smp_num_siblings = nvpes; - - for (t = 1; t < nvpes; t++) { - /* Use a 1:1 mapping of TC index to VPE index */ - settc(t); - - /* Bind 1 TC to this VPE */ - tcbind = read_tc_c0_tcbind(); - tcbind &= ~TCBIND_CURVPE; - tcbind |= t << TCBIND_CURVPE_SHIFT; - write_tc_c0_tcbind(tcbind); - - /* Set exclusive TC, non-active, master */ - vpeconf0 = read_vpe_c0_vpeconf0(); - vpeconf0 &= ~(VPECONF0_XTC | VPECONF0_VPA); - vpeconf0 |= t << VPECONF0_XTC_SHIFT; - vpeconf0 |= VPECONF0_MVP; - write_vpe_c0_vpeconf0(vpeconf0); - - /* Declare TC non-active, non-allocatable & interrupt exempt */ - tcstatus = read_tc_c0_tcstatus(); - tcstatus &= ~(TCSTATUS_A | TCSTATUS_DA); - tcstatus |= TCSTATUS_IXMT; - write_tc_c0_tcstatus(tcstatus); - - /* Halt the TC */ - write_tc_c0_tchalt(TCHALT_H); - - /* Allow only 1 TC to execute */ - vpecontrol = read_vpe_c0_vpecontrol(); - vpecontrol &= ~VPECONTROL_TE; - write_vpe_c0_vpecontrol(vpecontrol); - - /* Copy (most of) Status from VPE 0 */ - status = read_c0_status(); - status &= ~(ST0_IM | ST0_IE | ST0_KSU); - status |= ST0_CU0; - write_vpe_c0_status(status); - - /* Copy Config from VPE 0 */ - write_vpe_c0_config(read_c0_config()); - write_vpe_c0_config7(read_c0_config7()); - - /* Ensure no software interrupts are pending */ - write_vpe_c0_cause(0); - - /* Sync Count */ - write_vpe_c0_count(read_c0_count()); - } + if (!config_enabled(CONFIG_MIPS_MT_SMP) || !cpu_has_mipsmt) + return 1; - /* Leave VPE configuration state */ - clear_c0_mvpcontrol(MVPCONTROL_VPC); + write_gcr_cl_other(core << CM_GCR_Cx_OTHER_CORENUM_SHF); + cfg = read_gcr_co_config() & CM_GCR_Cx_CONFIG_PVPE_MSK; + return (cfg >> CM_GCR_Cx_CONFIG_PVPE_SHF) + 1; } static void __init cps_smp_setup(void) { unsigned int ncores, nvpes, core_vpes; int c, v; - u32 core_cfg, *entry_code; /* Detect & record VPE topology */ ncores = mips_cm_numcores(); pr_info("VPE topology "); for (c = nvpes = 0; c < ncores; c++) { - if (cpu_has_mipsmt && config_enabled(CONFIG_MIPS_MT_SMP)) { - write_gcr_cl_other(c << CM_GCR_Cx_OTHER_CORENUM_SHF); - core_cfg = read_gcr_co_config(); - core_vpes = ((core_cfg & CM_GCR_Cx_CONFIG_PVPE_MSK) >> - CM_GCR_Cx_CONFIG_PVPE_SHF) + 1; - } else { - core_vpes = 1; - } - + core_vpes = core_vpe_count(c); pr_cont("%c%u", c ? ',' : '{', core_vpes); + /* Use the number of VPEs in core 0 for smp_num_siblings */ + if (!c) + smp_num_siblings = core_vpes; + for (v = 0; v < min_t(int, core_vpes, NR_CPUS - nvpes); v++) { cpu_data[nvpes + v].core = c; #ifdef CONFIG_MIPS_MT_SMP @@ -137,19 +76,14 @@ static void __init cps_smp_setup(void) __cpu_logical_map[v] = v; } + /* Set a coherent default CCA (CWB) */ + change_c0_config(CONF_CM_CMASK, 0x5); + /* Core 0 is powered up (we're running on it) */ bitmap_set(core_power, 0, 1); - /* Disable MT - we only want to run 1 TC per VPE */ - if (cpu_has_mipsmt) - dmt(); - /* Initialise core 0 */ - init_core(); - - /* Patch the start of mips_cps_core_entry to provide the CM base */ - entry_code = (u32 *)&mips_cps_core_entry; - UASM_i_LA(&entry_code, 3, (long)mips_cm_base); + mips_cps_core_init(); /* Make core 0 coherent with everything */ write_gcr_cl_coherence(0xff); @@ -157,15 +91,99 @@ static void __init cps_smp_setup(void) static void __init cps_prepare_cpus(unsigned int max_cpus) { + unsigned ncores, core_vpes, c, cca; + bool cca_unsuitable; + u32 *entry_code; + mips_mt_set_cpuoptions(); + + /* Detect whether the CCA is unsuited to multi-core SMP */ + cca = read_c0_config() & CONF_CM_CMASK; + switch (cca) { + case 0x4: /* CWBE */ + case 0x5: /* CWB */ + /* The CCA is coherent, multi-core is fine */ + cca_unsuitable = false; + break; + + default: + /* CCA is not coherent, multi-core is not usable */ + cca_unsuitable = true; + } + + /* Warn the user if the CCA prevents multi-core */ + ncores = mips_cm_numcores(); + if (cca_unsuitable && ncores > 1) { + pr_warn("Using only one core due to unsuitable CCA 0x%x\n", + cca); + + for_each_present_cpu(c) { + if (cpu_data[c].core) + set_cpu_present(c, false); + } + } + + /* + * Patch the start of mips_cps_core_entry to provide: + * + * v0 = CM base address + * s0 = kseg0 CCA + */ + entry_code = (u32 *)&mips_cps_core_entry; + UASM_i_LA(&entry_code, 3, (long)mips_cm_base); + uasm_i_addiu(&entry_code, 16, 0, cca); + dma_cache_wback_inv((unsigned long)&mips_cps_core_entry, + (void *)entry_code - (void *)&mips_cps_core_entry); + + /* Allocate core boot configuration structs */ + mips_cps_core_bootcfg = kcalloc(ncores, sizeof(*mips_cps_core_bootcfg), + GFP_KERNEL); + if (!mips_cps_core_bootcfg) { + pr_err("Failed to allocate boot config for %u cores\n", ncores); + goto err_out; + } + + /* Allocate VPE boot configuration structs */ + for (c = 0; c < ncores; c++) { + core_vpes = core_vpe_count(c); + mips_cps_core_bootcfg[c].vpe_config = kcalloc(core_vpes, + sizeof(*mips_cps_core_bootcfg[c].vpe_config), + GFP_KERNEL); + if (!mips_cps_core_bootcfg[c].vpe_config) { + pr_err("Failed to allocate %u VPE boot configs\n", + core_vpes); + goto err_out; + } + } + + /* Mark this CPU as booted */ + atomic_set(&mips_cps_core_bootcfg[current_cpu_data.core].vpe_mask, + 1 << cpu_vpe_id(¤t_cpu_data)); + + return; +err_out: + /* Clean up allocations */ + if (mips_cps_core_bootcfg) { + for (c = 0; c < ncores; c++) + kfree(mips_cps_core_bootcfg[c].vpe_config); + kfree(mips_cps_core_bootcfg); + mips_cps_core_bootcfg = NULL; + } + + /* Effectively disable SMP by declaring CPUs not present */ + for_each_possible_cpu(c) { + if (c == 0) + continue; + set_cpu_present(c, false); + } } -static void boot_core(struct boot_config *cfg) +static void boot_core(unsigned core) { u32 access; /* Select the appropriate core */ - write_gcr_cl_other(cfg->core << CM_GCR_Cx_OTHER_CORENUM_SHF); + write_gcr_cl_other(core << CM_GCR_Cx_OTHER_CORENUM_SHF); /* Set its reset vector */ write_gcr_co_reset_base(CKSEG1ADDR((unsigned long)mips_cps_core_entry)); @@ -175,104 +193,74 @@ static void boot_core(struct boot_config *cfg) /* Ensure the core can access the GCRs */ access = read_gcr_access(); - access |= 1 << (CM_GCR_ACCESS_ACCESSEN_SHF + cfg->core); + access |= 1 << (CM_GCR_ACCESS_ACCESSEN_SHF + core); write_gcr_access(access); - /* Copy cfg */ - mips_cps_bootcfg = *cfg; - if (mips_cpc_present()) { - /* Select the appropriate core */ - write_cpc_cl_other(cfg->core << CPC_Cx_OTHER_CORENUM_SHF); - /* Reset the core */ + mips_cpc_lock_other(core); write_cpc_co_cmd(CPC_Cx_CMD_RESET); + mips_cpc_unlock_other(); } else { /* Take the core out of reset */ write_gcr_co_reset_release(0); } /* The core is now powered up */ - bitmap_set(core_power, cfg->core, 1); + bitmap_set(core_power, core, 1); } -static void boot_vpe(void *info) +static void remote_vpe_boot(void *dummy) { - struct boot_config *cfg = info; - u32 tcstatus, vpeconf0; - - /* Enter VPE configuration state */ - dvpe(); - set_c0_mvpcontrol(MVPCONTROL_VPC); - - settc(cfg->vpe); - - /* Set the TC restart PC */ - write_tc_c0_tcrestart((unsigned long)&smp_bootstrap); - - /* Activate the TC, allow interrupts */ - tcstatus = read_tc_c0_tcstatus(); - tcstatus &= ~TCSTATUS_IXMT; - tcstatus |= TCSTATUS_A; - write_tc_c0_tcstatus(tcstatus); - - /* Clear the TC halt bit */ - write_tc_c0_tchalt(0); - - /* Activate the VPE */ - vpeconf0 = read_vpe_c0_vpeconf0(); - vpeconf0 |= VPECONF0_VPA; - write_vpe_c0_vpeconf0(vpeconf0); - - /* Set the stack & global pointer registers */ - write_tc_gpr_sp(cfg->sp); - write_tc_gpr_gp(cfg->gp); - - /* Leave VPE configuration state */ - clear_c0_mvpcontrol(MVPCONTROL_VPC); - - /* Enable other VPEs to execute */ - evpe(EVPE_ENABLE); + mips_cps_boot_vpes(); } static void cps_boot_secondary(int cpu, struct task_struct *idle) { - struct boot_config cfg; + unsigned core = cpu_data[cpu].core; + unsigned vpe_id = cpu_vpe_id(&cpu_data[cpu]); + struct core_boot_config *core_cfg = &mips_cps_core_bootcfg[core]; + struct vpe_boot_config *vpe_cfg = &core_cfg->vpe_config[vpe_id]; unsigned int remote; int err; - cfg.core = cpu_data[cpu].core; - cfg.vpe = cpu_vpe_id(&cpu_data[cpu]); - cfg.pc = (unsigned long)&smp_bootstrap; - cfg.sp = __KSTK_TOS(idle); - cfg.gp = (unsigned long)task_thread_info(idle); + vpe_cfg->pc = (unsigned long)&smp_bootstrap; + vpe_cfg->sp = __KSTK_TOS(idle); + vpe_cfg->gp = (unsigned long)task_thread_info(idle); + + atomic_or(1 << cpu_vpe_id(&cpu_data[cpu]), &core_cfg->vpe_mask); - if (!test_bit(cfg.core, core_power)) { + preempt_disable(); + + if (!test_bit(core, core_power)) { /* Boot a VPE on a powered down core */ - boot_core(&cfg); - return; + boot_core(core); + goto out; } - if (cfg.core != current_cpu_data.core) { + if (core != current_cpu_data.core) { /* Boot a VPE on another powered up core */ for (remote = 0; remote < NR_CPUS; remote++) { - if (cpu_data[remote].core != cfg.core) + if (cpu_data[remote].core != core) continue; if (cpu_online(remote)) break; } BUG_ON(remote >= NR_CPUS); - err = smp_call_function_single(remote, boot_vpe, &cfg, 1); + err = smp_call_function_single(remote, remote_vpe_boot, + NULL, 1); if (err) panic("Failed to call remote CPU\n"); - return; + goto out; } BUG_ON(!cpu_has_mipsmt); /* Boot a VPE on this core */ - boot_vpe(&cfg); + mips_cps_boot_vpes(); +out: + preempt_enable(); } static void cps_init_secondary(void) @@ -281,10 +269,6 @@ static void cps_init_secondary(void) if (cpu_has_mipsmt) dmt(); - /* TODO: revisit this assumption once hotplug is implemented */ - if (cpu_vpe_id(¤t_cpu_data) == 0) - init_core(); - change_c0_status(ST0_IM, STATUSF_IP3 | STATUSF_IP4 | STATUSF_IP6 | STATUSF_IP7); } @@ -302,10 +286,148 @@ static void cps_smp_finish(void) local_irq_enable(); } -static void cps_cpus_done(void) +#ifdef CONFIG_HOTPLUG_CPU + +static int cps_cpu_disable(void) +{ + unsigned cpu = smp_processor_id(); + struct core_boot_config *core_cfg; + + if (!cpu) + return -EBUSY; + + if (!cps_pm_support_state(CPS_PM_POWER_GATED)) + return -EINVAL; + + core_cfg = &mips_cps_core_bootcfg[current_cpu_data.core]; + atomic_sub(1 << cpu_vpe_id(¤t_cpu_data), &core_cfg->vpe_mask); + smp_mb__after_atomic_dec(); + set_cpu_online(cpu, false); + cpu_clear(cpu, cpu_callin_map); + + return 0; +} + +static DECLARE_COMPLETION(cpu_death_chosen); +static unsigned cpu_death_sibling; +static enum { + CPU_DEATH_HALT, + CPU_DEATH_POWER, +} cpu_death; + +void play_dead(void) +{ + unsigned cpu, core; + + local_irq_disable(); + idle_task_exit(); + cpu = smp_processor_id(); + cpu_death = CPU_DEATH_POWER; + + if (cpu_has_mipsmt) { + core = cpu_data[cpu].core; + + /* Look for another online VPE within the core */ + for_each_online_cpu(cpu_death_sibling) { + if (cpu_data[cpu_death_sibling].core != core) + continue; + + /* + * There is an online VPE within the core. Just halt + * this TC and leave the core alone. + */ + cpu_death = CPU_DEATH_HALT; + break; + } + } + + /* This CPU has chosen its way out */ + complete(&cpu_death_chosen); + + if (cpu_death == CPU_DEATH_HALT) { + /* Halt this TC */ + write_c0_tchalt(TCHALT_H); + instruction_hazard(); + } else { + /* Power down the core */ + cps_pm_enter_state(CPS_PM_POWER_GATED); + } + + /* This should never be reached */ + panic("Failed to offline CPU %u", cpu); +} + +static void wait_for_sibling_halt(void *ptr_cpu) { + unsigned cpu = (unsigned)ptr_cpu; + unsigned vpe_id = cpu_data[cpu].vpe_id; + unsigned halted; + unsigned long flags; + + do { + local_irq_save(flags); + settc(vpe_id); + halted = read_tc_c0_tchalt(); + local_irq_restore(flags); + } while (!(halted & TCHALT_H)); +} + +static void cps_cpu_die(unsigned int cpu) +{ + unsigned core = cpu_data[cpu].core; + unsigned stat; + int err; + + /* Wait for the cpu to choose its way out */ + if (!wait_for_completion_timeout(&cpu_death_chosen, + msecs_to_jiffies(5000))) { + pr_err("CPU%u: didn't offline\n", cpu); + return; + } + + /* + * Now wait for the CPU to actually offline. Without doing this that + * offlining may race with one or more of: + * + * - Onlining the CPU again. + * - Powering down the core if another VPE within it is offlined. + * - A sibling VPE entering a non-coherent state. + * + * In the non-MT halt case (ie. infinite loop) the CPU is doing nothing + * with which we could race, so do nothing. + */ + if (cpu_death == CPU_DEATH_POWER) { + /* + * Wait for the core to enter a powered down or clock gated + * state, the latter happening when a JTAG probe is connected + * in which case the CPC will refuse to power down the core. + */ + do { + mips_cpc_lock_other(core); + stat = read_cpc_co_stat_conf(); + stat &= CPC_Cx_STAT_CONF_SEQSTATE_MSK; + mips_cpc_unlock_other(); + } while (stat != CPC_Cx_STAT_CONF_SEQSTATE_D0 && + stat != CPC_Cx_STAT_CONF_SEQSTATE_D2 && + stat != CPC_Cx_STAT_CONF_SEQSTATE_U2); + + /* Indicate the core is powered off */ + bitmap_clear(core_power, core, 1); + } else if (cpu_has_mipsmt) { + /* + * Have a CPU with access to the offlined CPUs registers wait + * for its TC to halt. + */ + err = smp_call_function_single(cpu_death_sibling, + wait_for_sibling_halt, + (void *)cpu, 1); + if (err) + panic("Failed to call remote sibling CPU\n"); + } } +#endif /* CONFIG_HOTPLUG_CPU */ + static struct plat_smp_ops cps_smp_ops = { .smp_setup = cps_smp_setup, .prepare_cpus = cps_prepare_cpus, @@ -314,9 +436,18 @@ static struct plat_smp_ops cps_smp_ops = { .smp_finish = cps_smp_finish, .send_ipi_single = gic_send_ipi_single, .send_ipi_mask = gic_send_ipi_mask, - .cpus_done = cps_cpus_done, +#ifdef CONFIG_HOTPLUG_CPU + .cpu_disable = cps_cpu_disable, + .cpu_die = cps_cpu_die, +#endif }; +bool mips_cps_smp_in_use(void) +{ + extern struct plat_smp_ops *mp_ops; + return mp_ops == &cps_smp_ops; +} + int register_cps_smp_ops(void) { if (!mips_cm_present()) { diff --git a/arch/mips/kernel/smp-gic.c b/arch/mips/kernel/smp-gic.c index 3bb1f92ab525..3b21a96d1ccb 100644 --- a/arch/mips/kernel/smp-gic.c +++ b/arch/mips/kernel/smp-gic.c @@ -15,12 +15,14 @@ #include <linux/printk.h> #include <asm/gic.h> +#include <asm/mips-cpc.h> #include <asm/smp-ops.h> void gic_send_ipi_single(int cpu, unsigned int action) { unsigned long flags; unsigned int intr; + unsigned int core = cpu_data[cpu].core; pr_debug("CPU%d: %s cpu %d action %u status %08x\n", smp_processor_id(), __func__, cpu, action, read_c0_status()); @@ -41,6 +43,15 @@ void gic_send_ipi_single(int cpu, unsigned int action) } gic_send_ipi(intr); + + if (mips_cpc_present() && (core != current_cpu_data.core)) { + while (!cpumask_test_cpu(cpu, &cpu_coherent_mask)) { + mips_cpc_lock_other(core); + write_cpc_co_cmd(CPC_Cx_CMD_PWRUP); + mips_cpc_unlock_other(); + } + } + local_irq_restore(flags); } diff --git a/arch/mips/kernel/smp-mt.c b/arch/mips/kernel/smp-mt.c index f8e13149604d..3babf6e4f894 100644 --- a/arch/mips/kernel/smp-mt.c +++ b/arch/mips/kernel/smp-mt.c @@ -183,10 +183,6 @@ static void vsmp_smp_finish(void) local_irq_enable(); } -static void vsmp_cpus_done(void) -{ -} - /* * Setup the PC, SP, and GP of a secondary processor and start it * running! @@ -287,7 +283,6 @@ struct plat_smp_ops vsmp_smp_ops = { .send_ipi_mask = vsmp_send_ipi_mask, .init_secondary = vsmp_init_secondary, .smp_finish = vsmp_smp_finish, - .cpus_done = vsmp_cpus_done, .boot_secondary = vsmp_boot_secondary, .smp_setup = vsmp_smp_setup, .prepare_cpus = vsmp_prepare_cpus, diff --git a/arch/mips/kernel/smp-up.c b/arch/mips/kernel/smp-up.c index 7fde3e4d978f..17878d71ef2b 100644 --- a/arch/mips/kernel/smp-up.c +++ b/arch/mips/kernel/smp-up.c @@ -36,11 +36,6 @@ static void up_smp_finish(void) { } -/* Hook for after all CPUs are online */ -static void up_cpus_done(void) -{ -} - /* * Firmware CPU startup hook */ @@ -73,7 +68,6 @@ struct plat_smp_ops up_smp_ops = { .send_ipi_mask = up_send_ipi_mask, .init_secondary = up_init_secondary, .smp_finish = up_smp_finish, - .cpus_done = up_cpus_done, .boot_secondary = up_boot_secondary, .smp_setup = up_smp_setup, .prepare_cpus = up_prepare_cpus, diff --git a/arch/mips/kernel/smp.c b/arch/mips/kernel/smp.c index 0a022ee33b2a..9bad52ede903 100644 --- a/arch/mips/kernel/smp.c +++ b/arch/mips/kernel/smp.c @@ -43,10 +43,6 @@ #include <asm/time.h> #include <asm/setup.h> -#ifdef CONFIG_MIPS_MT_SMTC -#include <asm/mipsmtregs.h> -#endif /* CONFIG_MIPS_MT_SMTC */ - volatile cpumask_t cpu_callin_map; /* Bitmask of started secondaries */ int __cpu_number_map[NR_CPUS]; /* Map physical to logical */ @@ -66,6 +62,8 @@ EXPORT_SYMBOL(cpu_sibling_map); /* representing cpus for which sibling maps can be computed */ static cpumask_t cpu_sibling_setup_map; +cpumask_t cpu_coherent_mask; + static inline void set_cpu_sibling_map(int cpu) { int i; @@ -102,12 +100,6 @@ asmlinkage void start_secondary(void) { unsigned int cpu; -#ifdef CONFIG_MIPS_MT_SMTC - /* Only do cpu_probe for first TC of CPU */ - if ((read_c0_tcbind() & TCBIND_CURTC) != 0) - __cpu_name[smp_processor_id()] = __cpu_name[0]; - else -#endif /* CONFIG_MIPS_MT_SMTC */ cpu_probe(); cpu_report(); per_cpu_trap_init(false); @@ -124,6 +116,7 @@ asmlinkage void start_secondary(void) cpu = smp_processor_id(); cpu_data[cpu].udelay_val = loops_per_jiffy; + cpu_set(cpu, cpu_coherent_mask); notify_cpu_starting(cpu); set_cpu_online(cpu, true); @@ -173,7 +166,6 @@ void smp_send_stop(void) void __init smp_cpus_done(unsigned int max_cpus) { - mp_ops->cpus_done(); } /* called from main before smp_init() */ @@ -186,6 +178,7 @@ void __init smp_prepare_cpus(unsigned int max_cpus) #ifndef CONFIG_HOTPLUG_CPU init_cpu_present(cpu_possible_mask); #endif + cpumask_copy(&cpu_coherent_mask, cpu_possible_mask); } /* preload SMP state for boot cpu */ @@ -238,13 +231,10 @@ static void flush_tlb_mm_ipi(void *mm) * o collapses to normal function call on UP kernels * o collapses to normal function call on systems with a single shared * primary cache. - * o CONFIG_MIPS_MT_SMTC currently implies there is only one physical core. */ static inline void smp_on_other_tlbs(void (*func) (void *info), void *info) { -#ifndef CONFIG_MIPS_MT_SMTC smp_call_function(func, info, 1); -#endif } static inline void smp_on_each_tlb(void (*func) (void *info), void *info) @@ -404,3 +394,46 @@ void dump_send_ipi(void (*dump_ipi_callback)(void *)) } EXPORT_SYMBOL(dump_send_ipi); #endif + +#ifdef CONFIG_GENERIC_CLOCKEVENTS_BROADCAST + +static DEFINE_PER_CPU(atomic_t, tick_broadcast_count); +static DEFINE_PER_CPU(struct call_single_data, tick_broadcast_csd); + +void tick_broadcast(const struct cpumask *mask) +{ + atomic_t *count; + struct call_single_data *csd; + int cpu; + + for_each_cpu(cpu, mask) { + count = &per_cpu(tick_broadcast_count, cpu); + csd = &per_cpu(tick_broadcast_csd, cpu); + + if (atomic_inc_return(count) == 1) + smp_call_function_single_async(cpu, csd); + } +} + +static void tick_broadcast_callee(void *info) +{ + int cpu = smp_processor_id(); + tick_receive_broadcast(); + atomic_set(&per_cpu(tick_broadcast_count, cpu), 0); +} + +static int __init tick_broadcast_init(void) +{ + struct call_single_data *csd; + int cpu; + + for (cpu = 0; cpu < NR_CPUS; cpu++) { + csd = &per_cpu(tick_broadcast_csd, cpu); + csd->func = tick_broadcast_callee; + } + + return 0; +} +early_initcall(tick_broadcast_init); + +#endif /* CONFIG_GENERIC_CLOCKEVENTS_BROADCAST */ diff --git a/arch/mips/kernel/smtc-asm.S b/arch/mips/kernel/smtc-asm.S deleted file mode 100644 index 2866863a39df..000000000000 --- a/arch/mips/kernel/smtc-asm.S +++ /dev/null @@ -1,133 +0,0 @@ -/* - * Assembly Language Functions for MIPS MT SMTC support - */ - -/* - * This file should be built into the kernel only if CONFIG_MIPS_MT_SMTC is set. */ - -#include <asm/regdef.h> -#include <asm/asmmacro.h> -#include <asm/stackframe.h> -#include <asm/irqflags.h> - -/* - * "Software Interrupt" linkage. - * - * This is invoked when an "Interrupt" is sent from one TC to another, - * where the TC to be interrupted is halted, has it's Restart address - * and Status values saved by the "remote control" thread, then modified - * to cause execution to begin here, in kenel mode. This code then - * disguises the TC state as that of an exception and transfers - * control to the general exception or vectored interrupt handler. - */ - .set noreorder - -/* -The __smtc_ipi_vector would use k0 and k1 as temporaries and -1) Set EXL (this is per-VPE, so this can't be done by proxy!) -2) Restore the K/CU and IXMT bits to the pre "exception" state - (EXL means no interrupts and access to the kernel map). -3) Set EPC to be the saved value of TCRestart. -4) Jump to the exception handler entry point passed by the sender. - -CAN WE PROVE THAT WE WON'T DO THIS IF INTS DISABLED?? -*/ - -/* - * Reviled and slandered vision: Set EXL and restore K/CU/IXMT - * state of pre-halt thread, then save everything and call - * thought some function pointer to imaginary_exception, which - * will parse a register value or memory message queue to - * deliver things like interprocessor interrupts. On return - * from that function, jump to the global ret_from_irq code - * to invoke the scheduler and return as appropriate. - */ - -#define PT_PADSLOT4 (PT_R0-8) -#define PT_PADSLOT5 (PT_R0-4) - - .text - .align 5 -FEXPORT(__smtc_ipi_vector) -#ifdef CONFIG_CPU_MICROMIPS - nop -#endif - .set noat - /* Disable thread scheduling to make Status update atomic */ - DMT 27 # dmt k1 - _ehb - /* Set EXL */ - mfc0 k0,CP0_STATUS - ori k0,k0,ST0_EXL - mtc0 k0,CP0_STATUS - _ehb - /* Thread scheduling now inhibited by EXL. Restore TE state. */ - andi k1,k1,VPECONTROL_TE - beqz k1,1f - emt -1: - /* - * The IPI sender has put some information on the anticipated - * kernel stack frame. If we were in user mode, this will be - * built above the saved kernel SP. If we were already in the - * kernel, it will be built above the current CPU SP. - * - * Were we in kernel mode, as indicated by CU0? - */ - sll k1,k0,3 - .set noreorder - bltz k1,2f - move k1,sp - .set reorder - /* - * If previously in user mode, set CU0 and use kernel stack. - */ - li k1,ST0_CU0 - or k1,k1,k0 - mtc0 k1,CP0_STATUS - _ehb - get_saved_sp - /* Interrupting TC will have pre-set values in slots in the new frame */ -2: subu k1,k1,PT_SIZE - /* Load TCStatus Value */ - lw k0,PT_TCSTATUS(k1) - /* Write it to TCStatus to restore CU/KSU/IXMT state */ - mtc0 k0,$2,1 - _ehb - lw k0,PT_EPC(k1) - mtc0 k0,CP0_EPC - /* Save all will redundantly recompute the SP, but use it for now */ - SAVE_ALL - CLI - TRACE_IRQS_OFF - /* Function to be invoked passed stack pad slot 5 */ - lw t0,PT_PADSLOT5(sp) - /* Argument from sender passed in stack pad slot 4 */ - lw a0,PT_PADSLOT4(sp) - LONG_L s0, TI_REGS($28) - LONG_S sp, TI_REGS($28) - PTR_LA ra, ret_from_irq - jr t0 - -/* - * Called from idle loop to provoke processing of queued IPIs - * First IPI message in queue passed as argument. - */ - -LEAF(self_ipi) - /* Before anything else, block interrupts */ - mfc0 t0,CP0_TCSTATUS - ori t1,t0,TCSTATUS_IXMT - mtc0 t1,CP0_TCSTATUS - _ehb - /* We know we're in kernel mode, so prepare stack frame */ - subu t1,sp,PT_SIZE - sw ra,PT_EPC(t1) - sw a0,PT_PADSLOT4(t1) - la t2,ipi_decode - sw t2,PT_PADSLOT5(t1) - /* Save pre-disable value of TCStatus */ - sw t0,PT_TCSTATUS(t1) - j __smtc_ipi_vector - nop -END(self_ipi) diff --git a/arch/mips/kernel/smtc-proc.c b/arch/mips/kernel/smtc-proc.c deleted file mode 100644 index 38635a996cbf..000000000000 --- a/arch/mips/kernel/smtc-proc.c +++ /dev/null @@ -1,102 +0,0 @@ -/* - * /proc hooks for SMTC kernel - * Copyright (C) 2005 Mips Technologies, Inc - */ - -#include <linux/kernel.h> -#include <linux/sched.h> -#include <linux/cpumask.h> -#include <linux/interrupt.h> - -#include <asm/cpu.h> -#include <asm/processor.h> -#include <linux/atomic.h> -#include <asm/hardirq.h> -#include <asm/mmu_context.h> -#include <asm/mipsregs.h> -#include <asm/cacheflush.h> -#include <linux/proc_fs.h> -#include <linux/seq_file.h> - -#include <asm/smtc_proc.h> - -/* - * /proc diagnostic and statistics hooks - */ - -/* - * Statistics gathered - */ -unsigned long selfipis[NR_CPUS]; - -struct smtc_cpu_proc smtc_cpu_stats[NR_CPUS]; - -atomic_t smtc_fpu_recoveries; - -static int smtc_proc_show(struct seq_file *m, void *v) -{ - int i; - extern unsigned long ebase; - - seq_printf(m, "SMTC Status Word: 0x%08x\n", smtc_status); - seq_printf(m, "Config7: 0x%08x\n", read_c0_config7()); - seq_printf(m, "EBASE: 0x%08lx\n", ebase); - seq_printf(m, "Counter Interrupts taken per CPU (TC)\n"); - for (i=0; i < NR_CPUS; i++) - seq_printf(m, "%d: %ld\n", i, smtc_cpu_stats[i].timerints); - seq_printf(m, "Self-IPIs by CPU:\n"); - for(i = 0; i < NR_CPUS; i++) - seq_printf(m, "%d: %ld\n", i, smtc_cpu_stats[i].selfipis); - seq_printf(m, "%d Recoveries of \"stolen\" FPU\n", - atomic_read(&smtc_fpu_recoveries)); - return 0; -} - -static int smtc_proc_open(struct inode *inode, struct file *file) -{ - return single_open(file, smtc_proc_show, NULL); -} - -static const struct file_operations smtc_proc_fops = { - .open = smtc_proc_open, - .read = seq_read, - .llseek = seq_lseek, - .release = single_release, -}; - -void init_smtc_stats(void) -{ - int i; - - for (i=0; i<NR_CPUS; i++) { - smtc_cpu_stats[i].timerints = 0; - smtc_cpu_stats[i].selfipis = 0; - } - - atomic_set(&smtc_fpu_recoveries, 0); - - proc_create("smtc", 0444, NULL, &smtc_proc_fops); -} - -static int proc_cpuinfo_chain_call(struct notifier_block *nfb, - unsigned long action_unused, void *data) -{ - struct proc_cpuinfo_notifier_args *pcn = data; - struct seq_file *m = pcn->m; - unsigned long n = pcn->n; - - if (!cpu_has_mipsmt) - return NOTIFY_OK; - - seq_printf(m, "VPE\t\t\t: %d\n", cpu_data[n].vpe_id); - seq_printf(m, "TC\t\t\t: %d\n", cpu_data[n].tc_id); - - return NOTIFY_OK; -} - -static int __init proc_cpuinfo_notifier_init(void) -{ - return proc_cpuinfo_notifier(proc_cpuinfo_chain_call, 0); -} - -subsys_initcall(proc_cpuinfo_notifier_init); diff --git a/arch/mips/kernel/smtc.c b/arch/mips/kernel/smtc.c deleted file mode 100644 index c1681d65dd5c..000000000000 --- a/arch/mips/kernel/smtc.c +++ /dev/null @@ -1,1528 +0,0 @@ -/* - * This program is free software; you can redistribute it and/or - * modify it under the terms of the GNU General Public License - * as published by the Free Software Foundation; either version 2 - * of the License, or (at your option) any later version. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, write to the Free Software - * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. - * - * Copyright (C) 2004 Mips Technologies, Inc - * Copyright (C) 2008 Kevin D. Kissell - */ - -#include <linux/clockchips.h> -#include <linux/kernel.h> -#include <linux/sched.h> -#include <linux/smp.h> -#include <linux/cpumask.h> -#include <linux/interrupt.h> -#include <linux/kernel_stat.h> -#include <linux/module.h> -#include <linux/ftrace.h> -#include <linux/slab.h> - -#include <asm/cpu.h> -#include <asm/processor.h> -#include <linux/atomic.h> -#include <asm/hardirq.h> -#include <asm/hazards.h> -#include <asm/irq.h> -#include <asm/idle.h> -#include <asm/mmu_context.h> -#include <asm/mipsregs.h> -#include <asm/cacheflush.h> -#include <asm/time.h> -#include <asm/addrspace.h> -#include <asm/smtc.h> -#include <asm/smtc_proc.h> -#include <asm/setup.h> - -/* - * SMTC Kernel needs to manipulate low-level CPU interrupt mask - * in do_IRQ. These are passed in setup_irq_smtc() and stored - * in this table. - */ -unsigned long irq_hwmask[NR_IRQS]; - -#define LOCK_MT_PRA() \ - local_irq_save(flags); \ - mtflags = dmt() - -#define UNLOCK_MT_PRA() \ - emt(mtflags); \ - local_irq_restore(flags) - -#define LOCK_CORE_PRA() \ - local_irq_save(flags); \ - mtflags = dvpe() - -#define UNLOCK_CORE_PRA() \ - evpe(mtflags); \ - local_irq_restore(flags) - -/* - * Data structures purely associated with SMTC parallelism - */ - - -/* - * Table for tracking ASIDs whose lifetime is prolonged. - */ - -asiduse smtc_live_asid[MAX_SMTC_TLBS][MAX_SMTC_ASIDS]; - -/* - * Number of InterProcessor Interrupt (IPI) message buffers to allocate - */ - -#define IPIBUF_PER_CPU 4 - -struct smtc_ipi_q IPIQ[NR_CPUS]; -static struct smtc_ipi_q freeIPIq; - - -/* - * Number of FPU contexts for each VPE - */ - -static int smtc_nconf1[MAX_SMTC_VPES]; - - -/* Forward declarations */ - -void ipi_decode(struct smtc_ipi *); -static void post_direct_ipi(int cpu, struct smtc_ipi *pipi); -static void setup_cross_vpe_interrupts(unsigned int nvpe); -void init_smtc_stats(void); - -/* Global SMTC Status */ - -unsigned int smtc_status; - -/* Boot command line configuration overrides */ - -static int vpe0limit; -static int ipibuffers; -static int nostlb; -static int asidmask; -unsigned long smtc_asid_mask = 0xff; - -static int __init vpe0tcs(char *str) -{ - get_option(&str, &vpe0limit); - - return 1; -} - -static int __init ipibufs(char *str) -{ - get_option(&str, &ipibuffers); - return 1; -} - -static int __init stlb_disable(char *s) -{ - nostlb = 1; - return 1; -} - -static int __init asidmask_set(char *str) -{ - get_option(&str, &asidmask); - switch (asidmask) { - case 0x1: - case 0x3: - case 0x7: - case 0xf: - case 0x1f: - case 0x3f: - case 0x7f: - case 0xff: - smtc_asid_mask = (unsigned long)asidmask; - break; - default: - printk("ILLEGAL ASID mask 0x%x from command line\n", asidmask); - } - return 1; -} - -__setup("vpe0tcs=", vpe0tcs); -__setup("ipibufs=", ipibufs); -__setup("nostlb", stlb_disable); -__setup("asidmask=", asidmask_set); - -#ifdef CONFIG_SMTC_IDLE_HOOK_DEBUG - -static int hang_trig; - -static int __init hangtrig_enable(char *s) -{ - hang_trig = 1; - return 1; -} - - -__setup("hangtrig", hangtrig_enable); - -#define DEFAULT_BLOCKED_IPI_LIMIT 32 - -static int timerq_limit = DEFAULT_BLOCKED_IPI_LIMIT; - -static int __init tintq(char *str) -{ - get_option(&str, &timerq_limit); - return 1; -} - -__setup("tintq=", tintq); - -static int imstuckcount[MAX_SMTC_VPES][8]; -/* vpemask represents IM/IE bits of per-VPE Status registers, low-to-high */ -static int vpemask[MAX_SMTC_VPES][8] = { - {0, 0, 1, 0, 0, 0, 0, 1}, - {0, 0, 0, 0, 0, 0, 0, 1} -}; -int tcnoprog[NR_CPUS]; -static atomic_t idle_hook_initialized = ATOMIC_INIT(0); -static int clock_hang_reported[NR_CPUS]; - -#endif /* CONFIG_SMTC_IDLE_HOOK_DEBUG */ - -/* - * Configure shared TLB - VPC configuration bit must be set by caller - */ - -static void smtc_configure_tlb(void) -{ - int i, tlbsiz, vpes; - unsigned long mvpconf0; - unsigned long config1val; - - /* Set up ASID preservation table */ - for (vpes=0; vpes<MAX_SMTC_TLBS; vpes++) { - for(i = 0; i < MAX_SMTC_ASIDS; i++) { - smtc_live_asid[vpes][i] = 0; - } - } - mvpconf0 = read_c0_mvpconf0(); - - if ((vpes = ((mvpconf0 & MVPCONF0_PVPE) - >> MVPCONF0_PVPE_SHIFT) + 1) > 1) { - /* If we have multiple VPEs, try to share the TLB */ - if ((mvpconf0 & MVPCONF0_TLBS) && !nostlb) { - /* - * If TLB sizing is programmable, shared TLB - * size is the total available complement. - * Otherwise, we have to take the sum of all - * static VPE TLB entries. - */ - if ((tlbsiz = ((mvpconf0 & MVPCONF0_PTLBE) - >> MVPCONF0_PTLBE_SHIFT)) == 0) { - /* - * If there's more than one VPE, there had better - * be more than one TC, because we need one to bind - * to each VPE in turn to be able to read - * its configuration state! - */ - settc(1); - /* Stop the TC from doing anything foolish */ - write_tc_c0_tchalt(TCHALT_H); - mips_ihb(); - /* No need to un-Halt - that happens later anyway */ - for (i=0; i < vpes; i++) { - write_tc_c0_tcbind(i); - /* - * To be 100% sure we're really getting the right - * information, we exit the configuration state - * and do an IHB after each rebinding. - */ - write_c0_mvpcontrol( - read_c0_mvpcontrol() & ~ MVPCONTROL_VPC ); - mips_ihb(); - /* - * Only count if the MMU Type indicated is TLB - */ - if (((read_vpe_c0_config() & MIPS_CONF_MT) >> 7) == 1) { - config1val = read_vpe_c0_config1(); - tlbsiz += ((config1val >> 25) & 0x3f) + 1; - } - - /* Put core back in configuration state */ - write_c0_mvpcontrol( - read_c0_mvpcontrol() | MVPCONTROL_VPC ); - mips_ihb(); - } - } - write_c0_mvpcontrol(read_c0_mvpcontrol() | MVPCONTROL_STLB); - ehb(); - - /* - * Setup kernel data structures to use software total, - * rather than read the per-VPE Config1 value. The values - * for "CPU 0" gets copied to all the other CPUs as part - * of their initialization in smtc_cpu_setup(). - */ - - /* MIPS32 limits TLB indices to 64 */ - if (tlbsiz > 64) - tlbsiz = 64; - cpu_data[0].tlbsize = current_cpu_data.tlbsize = tlbsiz; - smtc_status |= SMTC_TLB_SHARED; - local_flush_tlb_all(); - - printk("TLB of %d entry pairs shared by %d VPEs\n", - tlbsiz, vpes); - } else { - printk("WARNING: TLB Not Sharable on SMTC Boot!\n"); - } - } -} - - -/* - * Incrementally build the CPU map out of constituent MIPS MT cores, - * using the specified available VPEs and TCs. Plaform code needs - * to ensure that each MIPS MT core invokes this routine on reset, - * one at a time(!). - * - * This version of the build_cpu_map and prepare_cpus routines assumes - * that *all* TCs of a MIPS MT core will be used for Linux, and that - * they will be spread across *all* available VPEs (to minimise the - * loss of efficiency due to exception service serialization). - * An improved version would pick up configuration information and - * possibly leave some TCs/VPEs as "slave" processors. - * - * Use c0_MVPConf0 to find out how many TCs are available, setting up - * cpu_possible_mask and the logical/physical mappings. - */ - -int __init smtc_build_cpu_map(int start_cpu_slot) -{ - int i, ntcs; - - /* - * The CPU map isn't actually used for anything at this point, - * so it's not clear what else we should do apart from set - * everything up so that "logical" = "physical". - */ - ntcs = ((read_c0_mvpconf0() & MVPCONF0_PTC) >> MVPCONF0_PTC_SHIFT) + 1; - for (i=start_cpu_slot; i<NR_CPUS && i<ntcs; i++) { - set_cpu_possible(i, true); - __cpu_number_map[i] = i; - __cpu_logical_map[i] = i; - } -#ifdef CONFIG_MIPS_MT_FPAFF - /* Initialize map of CPUs with FPUs */ - cpus_clear(mt_fpu_cpumask); -#endif - - /* One of those TC's is the one booting, and not a secondary... */ - printk("%i available secondary CPU TC(s)\n", i - 1); - - return i; -} - -/* - * Common setup before any secondaries are started - * Make sure all CPUs are in a sensible state before we boot any of the - * secondaries. - * - * For MIPS MT "SMTC" operation, we set up all TCs, spread as evenly - * as possible across the available VPEs. - */ - -static void smtc_tc_setup(int vpe, int tc, int cpu) -{ - static int cp1contexts[MAX_SMTC_VPES]; - - /* - * Make a local copy of the available FPU contexts in order - * to keep track of TCs that can have one. - */ - if (tc == 1) - { - /* - * FIXME: Multi-core SMTC hasn't been tested and the - * maximum number of VPEs may change. - */ - cp1contexts[0] = smtc_nconf1[0] - 1; - cp1contexts[1] = smtc_nconf1[1]; - } - - settc(tc); - write_tc_c0_tchalt(TCHALT_H); - mips_ihb(); - write_tc_c0_tcstatus((read_tc_c0_tcstatus() - & ~(TCSTATUS_TKSU | TCSTATUS_DA | TCSTATUS_IXMT)) - | TCSTATUS_A); - /* - * TCContext gets an offset from the base of the IPIQ array - * to be used in low-level code to detect the presence of - * an active IPI queue. - */ - write_tc_c0_tccontext((sizeof(struct smtc_ipi_q) * cpu) << 16); - - /* Bind TC to VPE. */ - write_tc_c0_tcbind(vpe); - - /* In general, all TCs should have the same cpu_data indications. */ - memcpy(&cpu_data[cpu], &cpu_data[0], sizeof(struct cpuinfo_mips)); - - /* Check to see if there is a FPU context available for this TC. */ - if (!cp1contexts[vpe]) - cpu_data[cpu].options &= ~MIPS_CPU_FPU; - else - cp1contexts[vpe]--; - - /* Store the TC and VPE into the cpu_data structure. */ - cpu_data[cpu].vpe_id = vpe; - cpu_data[cpu].tc_id = tc; - - /* FIXME: Multi-core SMTC hasn't been tested, but be prepared. */ - cpu_data[cpu].core = (read_vpe_c0_ebase() >> 1) & 0xff; -} - -/* - * Tweak to get Count registers synced as closely as possible. The - * value seems good for 34K-class cores. - */ - -#define CP0_SKEW 8 - -void smtc_prepare_cpus(int cpus) -{ - int i, vpe, tc, ntc, nvpe, tcpervpe[NR_CPUS], slop, cpu; - unsigned long flags; - unsigned long val; - int nipi; - struct smtc_ipi *pipi; - - /* disable interrupts so we can disable MT */ - local_irq_save(flags); - /* disable MT so we can configure */ - dvpe(); - dmt(); - - spin_lock_init(&freeIPIq.lock); - - /* - * We probably don't have as many VPEs as we do SMP "CPUs", - * but it's possible - and in any case we'll never use more! - */ - for (i=0; i<NR_CPUS; i++) { - IPIQ[i].head = IPIQ[i].tail = NULL; - spin_lock_init(&IPIQ[i].lock); - IPIQ[i].depth = 0; - IPIQ[i].resched_flag = 0; /* No reschedules queued initially */ - } - - /* cpu_data index starts at zero */ - cpu = 0; - cpu_data[cpu].vpe_id = 0; - cpu_data[cpu].tc_id = 0; - cpu_data[cpu].core = (read_c0_ebase() >> 1) & 0xff; - cpu++; - - /* Report on boot-time options */ - mips_mt_set_cpuoptions(); - if (vpelimit > 0) - printk("Limit of %d VPEs set\n", vpelimit); - if (tclimit > 0) - printk("Limit of %d TCs set\n", tclimit); - if (nostlb) { - printk("Shared TLB Use Inhibited - UNSAFE for Multi-VPE Operation\n"); - } - if (asidmask) - printk("ASID mask value override to 0x%x\n", asidmask); - - /* Temporary */ -#ifdef CONFIG_SMTC_IDLE_HOOK_DEBUG - if (hang_trig) - printk("Logic Analyser Trigger on suspected TC hang\n"); -#endif /* CONFIG_SMTC_IDLE_HOOK_DEBUG */ - - /* Put MVPE's into 'configuration state' */ - write_c0_mvpcontrol( read_c0_mvpcontrol() | MVPCONTROL_VPC ); - - val = read_c0_mvpconf0(); - nvpe = ((val & MVPCONF0_PVPE) >> MVPCONF0_PVPE_SHIFT) + 1; - if (vpelimit > 0 && nvpe > vpelimit) - nvpe = vpelimit; - ntc = ((val & MVPCONF0_PTC) >> MVPCONF0_PTC_SHIFT) + 1; - if (ntc > NR_CPUS) - ntc = NR_CPUS; - if (tclimit > 0 && ntc > tclimit) - ntc = tclimit; - slop = ntc % nvpe; - for (i = 0; i < nvpe; i++) { - tcpervpe[i] = ntc / nvpe; - if (slop) { - if((slop - i) > 0) tcpervpe[i]++; - } - } - /* Handle command line override for VPE0 */ - if (vpe0limit > ntc) vpe0limit = ntc; - if (vpe0limit > 0) { - int slopslop; - if (vpe0limit < tcpervpe[0]) { - /* Reducing TC count - distribute to others */ - slop = tcpervpe[0] - vpe0limit; - slopslop = slop % (nvpe - 1); - tcpervpe[0] = vpe0limit; - for (i = 1; i < nvpe; i++) { - tcpervpe[i] += slop / (nvpe - 1); - if(slopslop && ((slopslop - (i - 1) > 0))) - tcpervpe[i]++; - } - } else if (vpe0limit > tcpervpe[0]) { - /* Increasing TC count - steal from others */ - slop = vpe0limit - tcpervpe[0]; - slopslop = slop % (nvpe - 1); - tcpervpe[0] = vpe0limit; - for (i = 1; i < nvpe; i++) { - tcpervpe[i] -= slop / (nvpe - 1); - if(slopslop && ((slopslop - (i - 1) > 0))) - tcpervpe[i]--; - } - } - } - - /* Set up shared TLB */ - smtc_configure_tlb(); - - for (tc = 0, vpe = 0 ; (vpe < nvpe) && (tc < ntc) ; vpe++) { - /* Get number of CP1 contexts for each VPE. */ - if (tc == 0) - { - /* - * Do not call settc() for TC0 or the FPU context - * value will be incorrect. Besides, we know that - * we are TC0 anyway. - */ - smtc_nconf1[0] = ((read_vpe_c0_vpeconf1() & - VPECONF1_NCP1) >> VPECONF1_NCP1_SHIFT); - if (nvpe == 2) - { - settc(1); - smtc_nconf1[1] = ((read_vpe_c0_vpeconf1() & - VPECONF1_NCP1) >> VPECONF1_NCP1_SHIFT); - settc(0); - } - } - if (tcpervpe[vpe] == 0) - continue; - if (vpe != 0) - printk(", "); - printk("VPE %d: TC", vpe); - for (i = 0; i < tcpervpe[vpe]; i++) { - /* - * TC 0 is bound to VPE 0 at reset, - * and is presumably executing this - * code. Leave it alone! - */ - if (tc != 0) { - smtc_tc_setup(vpe, tc, cpu); - if (vpe != 0) { - /* - * Set MVP bit (possibly again). Do it - * here to catch CPUs that have no TCs - * bound to the VPE at reset. In that - * case, a TC must be bound to the VPE - * before we can set VPEControl[MVP] - */ - write_vpe_c0_vpeconf0( - read_vpe_c0_vpeconf0() | - VPECONF0_MVP); - } - cpu++; - } - printk(" %d", tc); - tc++; - } - if (vpe != 0) { - /* - * Allow this VPE to control others. - */ - write_vpe_c0_vpeconf0(read_vpe_c0_vpeconf0() | - VPECONF0_MVP); - - /* - * Clear any stale software interrupts from VPE's Cause - */ - write_vpe_c0_cause(0); - - /* - * Clear ERL/EXL of VPEs other than 0 - * and set restricted interrupt enable/mask. - */ - write_vpe_c0_status((read_vpe_c0_status() - & ~(ST0_BEV | ST0_ERL | ST0_EXL | ST0_IM)) - | (STATUSF_IP0 | STATUSF_IP1 | STATUSF_IP7 - | ST0_IE)); - /* - * set config to be the same as vpe0, - * particularly kseg0 coherency alg - */ - write_vpe_c0_config(read_c0_config()); - /* Clear any pending timer interrupt */ - write_vpe_c0_compare(0); - /* Propagate Config7 */ - write_vpe_c0_config7(read_c0_config7()); - write_vpe_c0_count(read_c0_count() + CP0_SKEW); - ehb(); - } - /* enable multi-threading within VPE */ - write_vpe_c0_vpecontrol(read_vpe_c0_vpecontrol() | VPECONTROL_TE); - /* enable the VPE */ - write_vpe_c0_vpeconf0(read_vpe_c0_vpeconf0() | VPECONF0_VPA); - } - - /* - * Pull any physically present but unused TCs out of circulation. - */ - while (tc < (((val & MVPCONF0_PTC) >> MVPCONF0_PTC_SHIFT) + 1)) { - set_cpu_possible(tc, false); - set_cpu_present(tc, false); - tc++; - } - - /* release config state */ - write_c0_mvpcontrol( read_c0_mvpcontrol() & ~ MVPCONTROL_VPC ); - - printk("\n"); - - /* Set up coprocessor affinity CPU mask(s) */ - -#ifdef CONFIG_MIPS_MT_FPAFF - for (tc = 0; tc < ntc; tc++) { - if (cpu_data[tc].options & MIPS_CPU_FPU) - cpu_set(tc, mt_fpu_cpumask); - } -#endif - - /* set up ipi interrupts... */ - - /* If we have multiple VPEs running, set up the cross-VPE interrupt */ - - setup_cross_vpe_interrupts(nvpe); - - /* Set up queue of free IPI "messages". */ - nipi = NR_CPUS * IPIBUF_PER_CPU; - if (ipibuffers > 0) - nipi = ipibuffers; - - pipi = kmalloc(nipi *sizeof(struct smtc_ipi), GFP_KERNEL); - if (pipi == NULL) - panic("kmalloc of IPI message buffers failed"); - else - printk("IPI buffer pool of %d buffers\n", nipi); - for (i = 0; i < nipi; i++) { - smtc_ipi_nq(&freeIPIq, pipi); - pipi++; - } - - /* Arm multithreading and enable other VPEs - but all TCs are Halted */ - emt(EMT_ENABLE); - evpe(EVPE_ENABLE); - local_irq_restore(flags); - /* Initialize SMTC /proc statistics/diagnostics */ - init_smtc_stats(); -} - - -/* - * Setup the PC, SP, and GP of a secondary processor and start it - * running! - * smp_bootstrap is the place to resume from - * __KSTK_TOS(idle) is apparently the stack pointer - * (unsigned long)idle->thread_info the gp - * - */ -void smtc_boot_secondary(int cpu, struct task_struct *idle) -{ - extern u32 kernelsp[NR_CPUS]; - unsigned long flags; - int mtflags; - - LOCK_MT_PRA(); - if (cpu_data[cpu].vpe_id != cpu_data[smp_processor_id()].vpe_id) { - dvpe(); - } - settc(cpu_data[cpu].tc_id); - - /* pc */ - write_tc_c0_tcrestart((unsigned long)&smp_bootstrap); - - /* stack pointer */ - kernelsp[cpu] = __KSTK_TOS(idle); - write_tc_gpr_sp(__KSTK_TOS(idle)); - - /* global pointer */ - write_tc_gpr_gp((unsigned long)task_thread_info(idle)); - - smtc_status |= SMTC_MTC_ACTIVE; - write_tc_c0_tchalt(0); - if (cpu_data[cpu].vpe_id != cpu_data[smp_processor_id()].vpe_id) { - evpe(EVPE_ENABLE); - } - UNLOCK_MT_PRA(); -} - -void smtc_init_secondary(void) -{ -} - -void smtc_smp_finish(void) -{ - int cpu = smp_processor_id(); - - /* - * Lowest-numbered CPU per VPE starts a clock tick. - * Like per_cpu_trap_init() hack, this assumes that - * SMTC init code assigns TCs consdecutively and - * in ascending order across available VPEs. - */ - if (cpu > 0 && (cpu_data[cpu].vpe_id != cpu_data[cpu - 1].vpe_id)) - write_c0_compare(read_c0_count() + mips_hpt_frequency/HZ); - - local_irq_enable(); - - printk("TC %d going on-line as CPU %d\n", - cpu_data[smp_processor_id()].tc_id, smp_processor_id()); -} - -void smtc_cpus_done(void) -{ -} - -/* - * Support for SMTC-optimized driver IRQ registration - */ - -/* - * SMTC Kernel needs to manipulate low-level CPU interrupt mask - * in do_IRQ. These are passed in setup_irq_smtc() and stored - * in this table. - */ - -int setup_irq_smtc(unsigned int irq, struct irqaction * new, - unsigned long hwmask) -{ -#ifdef CONFIG_SMTC_IDLE_HOOK_DEBUG - unsigned int vpe = current_cpu_data.vpe_id; - - vpemask[vpe][irq - MIPS_CPU_IRQ_BASE] = 1; -#endif - irq_hwmask[irq] = hwmask; - - return setup_irq(irq, new); -} - -#ifdef CONFIG_MIPS_MT_SMTC_IRQAFF -/* - * Support for IRQ affinity to TCs - */ - -void smtc_set_irq_affinity(unsigned int irq, cpumask_t affinity) -{ - /* - * If a "fast path" cache of quickly decodable affinity state - * is maintained, this is where it gets done, on a call up - * from the platform affinity code. - */ -} - -void smtc_forward_irq(struct irq_data *d) -{ - unsigned int irq = d->irq; - int target; - - /* - * OK wise guy, now figure out how to get the IRQ - * to be serviced on an authorized "CPU". - * - * Ideally, to handle the situation where an IRQ has multiple - * eligible CPUS, we would maintain state per IRQ that would - * allow a fair distribution of service requests. Since the - * expected use model is any-or-only-one, for simplicity - * and efficiency, we just pick the easiest one to find. - */ - - target = cpumask_first(d->affinity); - - /* - * We depend on the platform code to have correctly processed - * IRQ affinity change requests to ensure that the IRQ affinity - * mask has been purged of bits corresponding to nonexistent and - * offline "CPUs", and to TCs bound to VPEs other than the VPE - * connected to the physical interrupt input for the interrupt - * in question. Otherwise we have a nasty problem with interrupt - * mask management. This is best handled in non-performance-critical - * platform IRQ affinity setting code, to minimize interrupt-time - * checks. - */ - - /* If no one is eligible, service locally */ - if (target >= NR_CPUS) - do_IRQ_no_affinity(irq); - else - smtc_send_ipi(target, IRQ_AFFINITY_IPI, irq); -} - -#endif /* CONFIG_MIPS_MT_SMTC_IRQAFF */ - -/* - * IPI model for SMTC is tricky, because interrupts aren't TC-specific. - * Within a VPE one TC can interrupt another by different approaches. - * The easiest to get right would probably be to make all TCs except - * the target IXMT and set a software interrupt, but an IXMT-based - * scheme requires that a handler must run before a new IPI could - * be sent, which would break the "broadcast" loops in MIPS MT. - * A more gonzo approach within a VPE is to halt the TC, extract - * its Restart, Status, and a couple of GPRs, and program the Restart - * address to emulate an interrupt. - * - * Within a VPE, one can be confident that the target TC isn't in - * a critical EXL state when halted, since the write to the Halt - * register could not have issued on the writing thread if the - * halting thread had EXL set. So k0 and k1 of the target TC - * can be used by the injection code. Across VPEs, one can't - * be certain that the target TC isn't in a critical exception - * state. So we try a two-step process of sending a software - * interrupt to the target VPE, which either handles the event - * itself (if it was the target) or injects the event within - * the VPE. - */ - -static void smtc_ipi_qdump(void) -{ - int i; - struct smtc_ipi *temp; - - for (i = 0; i < NR_CPUS ;i++) { - pr_info("IPIQ[%d]: head = 0x%x, tail = 0x%x, depth = %d\n", - i, (unsigned)IPIQ[i].head, (unsigned)IPIQ[i].tail, - IPIQ[i].depth); - temp = IPIQ[i].head; - - while (temp != IPIQ[i].tail) { - pr_debug("%d %d %d: ", temp->type, temp->dest, - (int)temp->arg); -#ifdef SMTC_IPI_DEBUG - pr_debug("%u %lu\n", temp->sender, temp->stamp); -#else - pr_debug("\n"); -#endif - temp = temp->flink; - } - } -} - -/* - * The standard atomic.h primitives don't quite do what we want - * here: We need an atomic add-and-return-previous-value (which - * could be done with atomic_add_return and a decrement) and an - * atomic set/zero-and-return-previous-value (which can't really - * be done with the atomic.h primitives). And since this is - * MIPS MT, we can assume that we have LL/SC. - */ -static inline int atomic_postincrement(atomic_t *v) -{ - unsigned long result; - - unsigned long temp; - - __asm__ __volatile__( - "1: ll %0, %2 \n" - " addu %1, %0, 1 \n" - " sc %1, %2 \n" - " beqz %1, 1b \n" - __WEAK_LLSC_MB - : "=&r" (result), "=&r" (temp), "=m" (v->counter) - : "m" (v->counter) - : "memory"); - - return result; -} - -void smtc_send_ipi(int cpu, int type, unsigned int action) -{ - int tcstatus; - struct smtc_ipi *pipi; - unsigned long flags; - int mtflags; - unsigned long tcrestart; - int set_resched_flag = (type == LINUX_SMP_IPI && - action == SMP_RESCHEDULE_YOURSELF); - - if (cpu == smp_processor_id()) { - printk("Cannot Send IPI to self!\n"); - return; - } - if (set_resched_flag && IPIQ[cpu].resched_flag != 0) - return; /* There is a reschedule queued already */ - - /* Set up a descriptor, to be delivered either promptly or queued */ - pipi = smtc_ipi_dq(&freeIPIq); - if (pipi == NULL) { - bust_spinlocks(1); - mips_mt_regdump(dvpe()); - panic("IPI Msg. Buffers Depleted"); - } - pipi->type = type; - pipi->arg = (void *)action; - pipi->dest = cpu; - if (cpu_data[cpu].vpe_id != cpu_data[smp_processor_id()].vpe_id) { - /* If not on same VPE, enqueue and send cross-VPE interrupt */ - IPIQ[cpu].resched_flag |= set_resched_flag; - smtc_ipi_nq(&IPIQ[cpu], pipi); - LOCK_CORE_PRA(); - settc(cpu_data[cpu].tc_id); - write_vpe_c0_cause(read_vpe_c0_cause() | C_SW1); - UNLOCK_CORE_PRA(); - } else { - /* - * Not sufficient to do a LOCK_MT_PRA (dmt) here, - * since ASID shootdown on the other VPE may - * collide with this operation. - */ - LOCK_CORE_PRA(); - settc(cpu_data[cpu].tc_id); - /* Halt the targeted TC */ - write_tc_c0_tchalt(TCHALT_H); - mips_ihb(); - - /* - * Inspect TCStatus - if IXMT is set, we have to queue - * a message. Otherwise, we set up the "interrupt" - * of the other TC - */ - tcstatus = read_tc_c0_tcstatus(); - - if ((tcstatus & TCSTATUS_IXMT) != 0) { - /* - * If we're in the the irq-off version of the wait - * loop, we need to force exit from the wait and - * do a direct post of the IPI. - */ - if (cpu_wait == r4k_wait_irqoff) { - tcrestart = read_tc_c0_tcrestart(); - if (address_is_in_r4k_wait_irqoff(tcrestart)) { - write_tc_c0_tcrestart(__pastwait); - tcstatus &= ~TCSTATUS_IXMT; - write_tc_c0_tcstatus(tcstatus); - goto postdirect; - } - } - /* - * Otherwise we queue the message for the target TC - * to pick up when he does a local_irq_restore() - */ - write_tc_c0_tchalt(0); - UNLOCK_CORE_PRA(); - IPIQ[cpu].resched_flag |= set_resched_flag; - smtc_ipi_nq(&IPIQ[cpu], pipi); - } else { -postdirect: - post_direct_ipi(cpu, pipi); - write_tc_c0_tchalt(0); - UNLOCK_CORE_PRA(); - } - } -} - -/* - * Send IPI message to Halted TC, TargTC/TargVPE already having been set - */ -static void post_direct_ipi(int cpu, struct smtc_ipi *pipi) -{ - struct pt_regs *kstack; - unsigned long tcstatus; - unsigned long tcrestart; - extern u32 kernelsp[NR_CPUS]; - extern void __smtc_ipi_vector(void); -//printk("%s: on %d for %d\n", __func__, smp_processor_id(), cpu); - - /* Extract Status, EPC from halted TC */ - tcstatus = read_tc_c0_tcstatus(); - tcrestart = read_tc_c0_tcrestart(); - /* If TCRestart indicates a WAIT instruction, advance the PC */ - if ((tcrestart & 0x80000000) - && ((*(unsigned int *)tcrestart & 0xfe00003f) == 0x42000020)) { - tcrestart += 4; - } - /* - * Save on TC's future kernel stack - * - * CU bit of Status is indicator that TC was - * already running on a kernel stack... - */ - if (tcstatus & ST0_CU0) { - /* Note that this "- 1" is pointer arithmetic */ - kstack = ((struct pt_regs *)read_tc_gpr_sp()) - 1; - } else { - kstack = ((struct pt_regs *)kernelsp[cpu]) - 1; - } - - kstack->cp0_epc = (long)tcrestart; - /* Save TCStatus */ - kstack->cp0_tcstatus = tcstatus; - /* Pass token of operation to be performed kernel stack pad area */ - kstack->pad0[4] = (unsigned long)pipi; - /* Pass address of function to be called likewise */ - kstack->pad0[5] = (unsigned long)&ipi_decode; - /* Set interrupt exempt and kernel mode */ - tcstatus |= TCSTATUS_IXMT; - tcstatus &= ~TCSTATUS_TKSU; - write_tc_c0_tcstatus(tcstatus); - ehb(); - /* Set TC Restart address to be SMTC IPI vector */ - write_tc_c0_tcrestart(__smtc_ipi_vector); -} - -static void ipi_resched_interrupt(void) -{ - scheduler_ipi(); -} - -static void ipi_call_interrupt(void) -{ - /* Invoke generic function invocation code in smp.c */ - smp_call_function_interrupt(); -} - -DECLARE_PER_CPU(struct clock_event_device, mips_clockevent_device); - -static void __irq_entry smtc_clock_tick_interrupt(void) -{ - unsigned int cpu = smp_processor_id(); - struct clock_event_device *cd; - int irq = MIPS_CPU_IRQ_BASE + 1; - - irq_enter(); - kstat_incr_irq_this_cpu(irq); - cd = &per_cpu(mips_clockevent_device, cpu); - cd->event_handler(cd); - irq_exit(); -} - -void ipi_decode(struct smtc_ipi *pipi) -{ - void *arg_copy = pipi->arg; - int type_copy = pipi->type; - - smtc_ipi_nq(&freeIPIq, pipi); - - switch (type_copy) { - case SMTC_CLOCK_TICK: - smtc_clock_tick_interrupt(); - break; - - case LINUX_SMP_IPI: - switch ((int)arg_copy) { - case SMP_RESCHEDULE_YOURSELF: - ipi_resched_interrupt(); - break; - case SMP_CALL_FUNCTION: - ipi_call_interrupt(); - break; - default: - printk("Impossible SMTC IPI Argument %p\n", arg_copy); - break; - } - break; -#ifdef CONFIG_MIPS_MT_SMTC_IRQAFF - case IRQ_AFFINITY_IPI: - /* - * Accept a "forwarded" interrupt that was initially - * taken by a TC who doesn't have affinity for the IRQ. - */ - do_IRQ_no_affinity((int)arg_copy); - break; -#endif /* CONFIG_MIPS_MT_SMTC_IRQAFF */ - default: - printk("Impossible SMTC IPI Type 0x%x\n", type_copy); - break; - } -} - -/* - * Similar to smtc_ipi_replay(), but invoked from context restore, - * so it reuses the current exception frame rather than set up a - * new one with self_ipi. - */ - -void deferred_smtc_ipi(void) -{ - int cpu = smp_processor_id(); - - /* - * Test is not atomic, but much faster than a dequeue, - * and the vast majority of invocations will have a null queue. - * If irq_disabled when this was called, then any IPIs queued - * after we test last will be taken on the next irq_enable/restore. - * If interrupts were enabled, then any IPIs added after the - * last test will be taken directly. - */ - - while (IPIQ[cpu].head != NULL) { - struct smtc_ipi_q *q = &IPIQ[cpu]; - struct smtc_ipi *pipi; - unsigned long flags; - - /* - * It may be possible we'll come in with interrupts - * already enabled. - */ - local_irq_save(flags); - spin_lock(&q->lock); - pipi = __smtc_ipi_dq(q); - spin_unlock(&q->lock); - if (pipi != NULL) { - if (pipi->type == LINUX_SMP_IPI && - (int)pipi->arg == SMP_RESCHEDULE_YOURSELF) - IPIQ[cpu].resched_flag = 0; - ipi_decode(pipi); - } - /* - * The use of the __raw_local restore isn't - * as obviously necessary here as in smtc_ipi_replay(), - * but it's more efficient, given that we're already - * running down the IPI queue. - */ - __arch_local_irq_restore(flags); - } -} - -/* - * Cross-VPE interrupts in the SMTC prototype use "software interrupts" - * set via cross-VPE MTTR manipulation of the Cause register. It would be - * in some regards preferable to have external logic for "doorbell" hardware - * interrupts. - */ - -static int cpu_ipi_irq = MIPS_CPU_IRQ_BASE + MIPS_CPU_IPI_IRQ; - -static irqreturn_t ipi_interrupt(int irq, void *dev_idm) -{ - int my_vpe = cpu_data[smp_processor_id()].vpe_id; - int my_tc = cpu_data[smp_processor_id()].tc_id; - int cpu; - struct smtc_ipi *pipi; - unsigned long tcstatus; - int sent; - unsigned long flags; - unsigned int mtflags; - unsigned int vpflags; - - /* - * So long as cross-VPE interrupts are done via - * MFTR/MTTR read-modify-writes of Cause, we need - * to stop other VPEs whenever the local VPE does - * anything similar. - */ - local_irq_save(flags); - vpflags = dvpe(); - clear_c0_cause(0x100 << MIPS_CPU_IPI_IRQ); - set_c0_status(0x100 << MIPS_CPU_IPI_IRQ); - irq_enable_hazard(); - evpe(vpflags); - local_irq_restore(flags); - - /* - * Cross-VPE Interrupt handler: Try to directly deliver IPIs - * queued for TCs on this VPE other than the current one. - * Return-from-interrupt should cause us to drain the queue - * for the current TC, so we ought not to have to do it explicitly here. - */ - - for_each_online_cpu(cpu) { - if (cpu_data[cpu].vpe_id != my_vpe) - continue; - - pipi = smtc_ipi_dq(&IPIQ[cpu]); - if (pipi != NULL) { - if (cpu_data[cpu].tc_id != my_tc) { - sent = 0; - LOCK_MT_PRA(); - settc(cpu_data[cpu].tc_id); - write_tc_c0_tchalt(TCHALT_H); - mips_ihb(); - tcstatus = read_tc_c0_tcstatus(); - if ((tcstatus & TCSTATUS_IXMT) == 0) { - post_direct_ipi(cpu, pipi); - sent = 1; - } - write_tc_c0_tchalt(0); - UNLOCK_MT_PRA(); - if (!sent) { - smtc_ipi_req(&IPIQ[cpu], pipi); - } - } else { - /* - * ipi_decode() should be called - * with interrupts off - */ - local_irq_save(flags); - if (pipi->type == LINUX_SMP_IPI && - (int)pipi->arg == SMP_RESCHEDULE_YOURSELF) - IPIQ[cpu].resched_flag = 0; - ipi_decode(pipi); - local_irq_restore(flags); - } - } - } - - return IRQ_HANDLED; -} - -static void ipi_irq_dispatch(void) -{ - do_IRQ(cpu_ipi_irq); -} - -static struct irqaction irq_ipi = { - .handler = ipi_interrupt, - .flags = IRQF_PERCPU, - .name = "SMTC_IPI" -}; - -static void setup_cross_vpe_interrupts(unsigned int nvpe) -{ - if (nvpe < 1) - return; - - if (!cpu_has_vint) - panic("SMTC Kernel requires Vectored Interrupt support"); - - set_vi_handler(MIPS_CPU_IPI_IRQ, ipi_irq_dispatch); - - setup_irq_smtc(cpu_ipi_irq, &irq_ipi, (0x100 << MIPS_CPU_IPI_IRQ)); - - irq_set_handler(cpu_ipi_irq, handle_percpu_irq); -} - -/* - * SMTC-specific hacks invoked from elsewhere in the kernel. - */ - - /* - * smtc_ipi_replay is called from raw_local_irq_restore - */ - -void smtc_ipi_replay(void) -{ - unsigned int cpu = smp_processor_id(); - - /* - * To the extent that we've ever turned interrupts off, - * we may have accumulated deferred IPIs. This is subtle. - * we should be OK: If we pick up something and dispatch - * it here, that's great. If we see nothing, but concurrent - * with this operation, another TC sends us an IPI, IXMT - * is clear, and we'll handle it as a real pseudo-interrupt - * and not a pseudo-pseudo interrupt. The important thing - * is to do the last check for queued message *after* the - * re-enabling of interrupts. - */ - while (IPIQ[cpu].head != NULL) { - struct smtc_ipi_q *q = &IPIQ[cpu]; - struct smtc_ipi *pipi; - unsigned long flags; - - /* - * It's just possible we'll come in with interrupts - * already enabled. - */ - local_irq_save(flags); - - spin_lock(&q->lock); - pipi = __smtc_ipi_dq(q); - spin_unlock(&q->lock); - /* - ** But use a raw restore here to avoid recursion. - */ - __arch_local_irq_restore(flags); - - if (pipi) { - self_ipi(pipi); - smtc_cpu_stats[cpu].selfipis++; - } - } -} - -EXPORT_SYMBOL(smtc_ipi_replay); - -void smtc_idle_loop_hook(void) -{ -#ifdef CONFIG_SMTC_IDLE_HOOK_DEBUG - int im; - int flags; - int mtflags; - int bit; - int vpe; - int tc; - int hook_ntcs; - /* - * printk within DMT-protected regions can deadlock, - * so buffer diagnostic messages for later output. - */ - char *pdb_msg; - char id_ho_db_msg[768]; /* worst-case use should be less than 700 */ - - if (atomic_read(&idle_hook_initialized) == 0) { /* fast test */ - if (atomic_add_return(1, &idle_hook_initialized) == 1) { - int mvpconf0; - /* Tedious stuff to just do once */ - mvpconf0 = read_c0_mvpconf0(); - hook_ntcs = ((mvpconf0 & MVPCONF0_PTC) >> MVPCONF0_PTC_SHIFT) + 1; - if (hook_ntcs > NR_CPUS) - hook_ntcs = NR_CPUS; - for (tc = 0; tc < hook_ntcs; tc++) { - tcnoprog[tc] = 0; - clock_hang_reported[tc] = 0; - } - for (vpe = 0; vpe < 2; vpe++) - for (im = 0; im < 8; im++) - imstuckcount[vpe][im] = 0; - printk("Idle loop test hook initialized for %d TCs\n", hook_ntcs); - atomic_set(&idle_hook_initialized, 1000); - } else { - /* Someone else is initializing in parallel - let 'em finish */ - while (atomic_read(&idle_hook_initialized) < 1000) - ; - } - } - - /* Have we stupidly left IXMT set somewhere? */ - if (read_c0_tcstatus() & 0x400) { - write_c0_tcstatus(read_c0_tcstatus() & ~0x400); - ehb(); - printk("Dangling IXMT in cpu_idle()\n"); - } - - /* Have we stupidly left an IM bit turned off? */ -#define IM_LIMIT 2000 - local_irq_save(flags); - mtflags = dmt(); - pdb_msg = &id_ho_db_msg[0]; - im = read_c0_status(); - vpe = current_cpu_data.vpe_id; - for (bit = 0; bit < 8; bit++) { - /* - * In current prototype, I/O interrupts - * are masked for VPE > 0 - */ - if (vpemask[vpe][bit]) { - if (!(im & (0x100 << bit))) - imstuckcount[vpe][bit]++; - else - imstuckcount[vpe][bit] = 0; - if (imstuckcount[vpe][bit] > IM_LIMIT) { - set_c0_status(0x100 << bit); - ehb(); - imstuckcount[vpe][bit] = 0; - pdb_msg += sprintf(pdb_msg, - "Dangling IM %d fixed for VPE %d\n", bit, - vpe); - } - } - } - - emt(mtflags); - local_irq_restore(flags); - if (pdb_msg != &id_ho_db_msg[0]) - printk("CPU%d: %s", smp_processor_id(), id_ho_db_msg); -#endif /* CONFIG_SMTC_IDLE_HOOK_DEBUG */ - - smtc_ipi_replay(); -} - -void smtc_soft_dump(void) -{ - int i; - - printk("Counter Interrupts taken per CPU (TC)\n"); - for (i=0; i < NR_CPUS; i++) { - printk("%d: %ld\n", i, smtc_cpu_stats[i].timerints); - } - printk("Self-IPI invocations:\n"); - for (i=0; i < NR_CPUS; i++) { - printk("%d: %ld\n", i, smtc_cpu_stats[i].selfipis); - } - smtc_ipi_qdump(); - printk("%d Recoveries of \"stolen\" FPU\n", - atomic_read(&smtc_fpu_recoveries)); -} - - -/* - * TLB management routines special to SMTC - */ - -void smtc_get_new_mmu_context(struct mm_struct *mm, unsigned long cpu) -{ - unsigned long flags, mtflags, tcstat, prevhalt, asid; - int tlb, i; - - /* - * It would be nice to be able to use a spinlock here, - * but this is invoked from within TLB flush routines - * that protect themselves with DVPE, so if a lock is - * held by another TC, it'll never be freed. - * - * DVPE/DMT must not be done with interrupts enabled, - * so even so most callers will already have disabled - * them, let's be really careful... - */ - - local_irq_save(flags); - if (smtc_status & SMTC_TLB_SHARED) { - mtflags = dvpe(); - tlb = 0; - } else { - mtflags = dmt(); - tlb = cpu_data[cpu].vpe_id; - } - asid = asid_cache(cpu); - - do { - if (!((asid += ASID_INC) & ASID_MASK) ) { - if (cpu_has_vtag_icache) - flush_icache_all(); - /* Traverse all online CPUs (hack requires contiguous range) */ - for_each_online_cpu(i) { - /* - * We don't need to worry about our own CPU, nor those of - * CPUs who don't share our TLB. - */ - if ((i != smp_processor_id()) && - ((smtc_status & SMTC_TLB_SHARED) || - (cpu_data[i].vpe_id == cpu_data[cpu].vpe_id))) { - settc(cpu_data[i].tc_id); - prevhalt = read_tc_c0_tchalt() & TCHALT_H; - if (!prevhalt) { - write_tc_c0_tchalt(TCHALT_H); - mips_ihb(); - } - tcstat = read_tc_c0_tcstatus(); - smtc_live_asid[tlb][(tcstat & ASID_MASK)] |= (asiduse)(0x1 << i); - if (!prevhalt) - write_tc_c0_tchalt(0); - } - } - if (!asid) /* fix version if needed */ - asid = ASID_FIRST_VERSION; - local_flush_tlb_all(); /* start new asid cycle */ - } - } while (smtc_live_asid[tlb][(asid & ASID_MASK)]); - - /* - * SMTC shares the TLB within VPEs and possibly across all VPEs. - */ - for_each_online_cpu(i) { - if ((smtc_status & SMTC_TLB_SHARED) || - (cpu_data[i].vpe_id == cpu_data[cpu].vpe_id)) - cpu_context(i, mm) = asid_cache(i) = asid; - } - - if (smtc_status & SMTC_TLB_SHARED) - evpe(mtflags); - else - emt(mtflags); - local_irq_restore(flags); -} - -/* - * Invoked from macros defined in mmu_context.h - * which must already have disabled interrupts - * and done a DVPE or DMT as appropriate. - */ - -void smtc_flush_tlb_asid(unsigned long asid) -{ - int entry; - unsigned long ehi; - - entry = read_c0_wired(); - - /* Traverse all non-wired entries */ - while (entry < current_cpu_data.tlbsize) { - write_c0_index(entry); - ehb(); - tlb_read(); - ehb(); - ehi = read_c0_entryhi(); - if ((ehi & ASID_MASK) == asid) { - /* - * Invalidate only entries with specified ASID, - * makiing sure all entries differ. - */ - write_c0_entryhi(CKSEG0 + (entry << (PAGE_SHIFT + 1))); - write_c0_entrylo0(0); - write_c0_entrylo1(0); - mtc0_tlbw_hazard(); - tlb_write_indexed(); - } - entry++; - } - write_c0_index(PARKED_INDEX); - tlbw_use_hazard(); -} - -/* - * Support for single-threading cache flush operations. - */ - -static int halt_state_save[NR_CPUS]; - -/* - * To really, really be sure that nothing is being done - * by other TCs, halt them all. This code assumes that - * a DVPE has already been done, so while their Halted - * state is theoretically architecturally unstable, in - * practice, it's not going to change while we're looking - * at it. - */ - -void smtc_cflush_lockdown(void) -{ - int cpu; - - for_each_online_cpu(cpu) { - if (cpu != smp_processor_id()) { - settc(cpu_data[cpu].tc_id); - halt_state_save[cpu] = read_tc_c0_tchalt(); - write_tc_c0_tchalt(TCHALT_H); - } - } - mips_ihb(); -} - -/* It would be cheating to change the cpu_online states during a flush! */ - -void smtc_cflush_release(void) -{ - int cpu; - - /* - * Start with a hazard barrier to ensure - * that all CACHE ops have played through. - */ - mips_ihb(); - - for_each_online_cpu(cpu) { - if (cpu != smp_processor_id()) { - settc(cpu_data[cpu].tc_id); - write_tc_c0_tchalt(halt_state_save[cpu]); - } - } - mips_ihb(); -} diff --git a/arch/mips/kernel/sync-r4k.c b/arch/mips/kernel/sync-r4k.c index c24ad5f4b324..2242bdd4370e 100644 --- a/arch/mips/kernel/sync-r4k.c +++ b/arch/mips/kernel/sync-r4k.c @@ -6,8 +6,6 @@ * not have done anything significant (but they may have had interrupts * enabled briefly - prom_smp_finish() should not be responsible for enabling * interrupts...) - * - * FIXME: broken for SMTC */ #include <linux/kernel.h> @@ -33,14 +31,6 @@ void synchronise_count_master(int cpu) unsigned long flags; unsigned int initcount; -#ifdef CONFIG_MIPS_MT_SMTC - /* - * SMTC needs to synchronise per VPE, not per CPU - * ignore for now - */ - return; -#endif - printk(KERN_INFO "Synchronize counters for CPU %u: ", cpu); local_irq_save(flags); @@ -110,14 +100,6 @@ void synchronise_count_slave(int cpu) int i; unsigned int initcount; -#ifdef CONFIG_MIPS_MT_SMTC - /* - * SMTC needs to synchronise per VPE, not per CPU - * ignore for now - */ - return; -#endif - /* * Not every cpu is online at the time this gets called, * so we first wait for the master to say everyone is ready diff --git a/arch/mips/kernel/time.c b/arch/mips/kernel/time.c index dcb8e5d3bb8a..8d0170969e22 100644 --- a/arch/mips/kernel/time.c +++ b/arch/mips/kernel/time.c @@ -26,7 +26,6 @@ #include <asm/cpu-features.h> #include <asm/cpu-type.h> #include <asm/div64.h> -#include <asm/smtc_ipi.h> #include <asm/time.h> /* diff --git a/arch/mips/kernel/traps.c b/arch/mips/kernel/traps.c index 8119ac2fdfc9..51706d6dd5b0 100644 --- a/arch/mips/kernel/traps.c +++ b/arch/mips/kernel/traps.c @@ -15,6 +15,7 @@ #include <linux/bug.h> #include <linux/compiler.h> #include <linux/context_tracking.h> +#include <linux/cpu_pm.h> #include <linux/kexec.h> #include <linux/init.h> #include <linux/kernel.h> @@ -370,9 +371,6 @@ void __noreturn die(const char *str, struct pt_regs *regs) { static int die_counter; int sig = SIGSEGV; -#ifdef CONFIG_MIPS_MT_SMTC - unsigned long dvpret; -#endif /* CONFIG_MIPS_MT_SMTC */ oops_enter(); @@ -382,13 +380,7 @@ void __noreturn die(const char *str, struct pt_regs *regs) console_verbose(); raw_spin_lock_irq(&die_lock); -#ifdef CONFIG_MIPS_MT_SMTC - dvpret = dvpe(); -#endif /* CONFIG_MIPS_MT_SMTC */ bust_spinlocks(1); -#ifdef CONFIG_MIPS_MT_SMTC - mips_mt_regdump(dvpret); -#endif /* CONFIG_MIPS_MT_SMTC */ printk("%s[#%d]:\n", str, ++die_counter); show_registers(regs); @@ -712,10 +704,12 @@ int process_fpemu_return(int sig, void __user *fault_addr) si.si_addr = fault_addr; si.si_signo = sig; if (sig == SIGSEGV) { + down_read(¤t->mm->mmap_sem); if (find_vma(current->mm, (unsigned long)fault_addr)) si.si_code = SEGV_ACCERR; else si.si_code = SEGV_MAPERR; + up_read(¤t->mm->mmap_sem); } else { si.si_code = BUS_ADRERR; } @@ -1759,19 +1753,6 @@ static void *set_vi_srs_handler(int n, vi_handler_t addr, int srs) extern char rollback_except_vec_vi; char *vec_start = using_rollback_handler() ? &rollback_except_vec_vi : &except_vec_vi; -#ifdef CONFIG_MIPS_MT_SMTC - /* - * We need to provide the SMTC vectored interrupt handler - * not only with the address of the handler, but with the - * Status.IM bit to be masked before going there. - */ - extern char except_vec_vi_mori; -#if defined(CONFIG_CPU_MICROMIPS) || defined(CONFIG_CPU_BIG_ENDIAN) - const int mori_offset = &except_vec_vi_mori - vec_start + 2; -#else - const int mori_offset = &except_vec_vi_mori - vec_start; -#endif -#endif /* CONFIG_MIPS_MT_SMTC */ #if defined(CONFIG_CPU_MICROMIPS) || defined(CONFIG_CPU_BIG_ENDIAN) const int lui_offset = &except_vec_vi_lui - vec_start + 2; const int ori_offset = &except_vec_vi_ori - vec_start + 2; @@ -1795,12 +1776,6 @@ static void *set_vi_srs_handler(int n, vi_handler_t addr, int srs) #else handler_len); #endif -#ifdef CONFIG_MIPS_MT_SMTC - BUG_ON(n > 7); /* Vector index %d exceeds SMTC maximum. */ - - h = (u16 *)(b + mori_offset); - *h = (0x100 << n); -#endif /* CONFIG_MIPS_MT_SMTC */ h = (u16 *)(b + lui_offset); *h = (handler >> 16) & 0xffff; h = (u16 *)(b + ori_offset); @@ -1865,32 +1840,16 @@ static int __init ulri_disable(char *s) } __setup("noulri", ulri_disable); -void per_cpu_trap_init(bool is_boot_cpu) +/* configure STATUS register */ +static void configure_status(void) { - unsigned int cpu = smp_processor_id(); - unsigned int status_set = ST0_CU0; - unsigned int hwrena = cpu_hwrena_impl_bits; -#ifdef CONFIG_MIPS_MT_SMTC - int secondaryTC = 0; - int bootTC = (cpu == 0); - - /* - * Only do per_cpu_trap_init() for first TC of Each VPE. - * Note that this hack assumes that the SMTC init code - * assigns TCs consecutively and in ascending order. - */ - - if (((read_c0_tcbind() & TCBIND_CURTC) != 0) && - ((read_c0_tcbind() & TCBIND_CURVPE) == cpu_data[cpu - 1].vpe_id)) - secondaryTC = 1; -#endif /* CONFIG_MIPS_MT_SMTC */ - /* * Disable coprocessors and select 32-bit or 64-bit addressing * and the 16/32 or 32/32 FPR register model. Reset the BEV * flag that some firmware may have left set and the TS bit (for * IP27). Set XX for ISA IV code to work. */ + unsigned int status_set = ST0_CU0; #ifdef CONFIG_64BIT status_set |= ST0_FR|ST0_KX|ST0_SX|ST0_UX; #endif @@ -1901,6 +1860,12 @@ void per_cpu_trap_init(bool is_boot_cpu) change_c0_status(ST0_CU|ST0_MX|ST0_RE|ST0_FR|ST0_BEV|ST0_TS|ST0_KX|ST0_SX|ST0_UX, status_set); +} + +/* configure HWRENA register */ +static void configure_hwrena(void) +{ + unsigned int hwrena = cpu_hwrena_impl_bits; if (cpu_has_mips_r2) hwrena |= 0x0000000f; @@ -1910,11 +1875,10 @@ void per_cpu_trap_init(bool is_boot_cpu) if (hwrena) write_c0_hwrena(hwrena); +} -#ifdef CONFIG_MIPS_MT_SMTC - if (!secondaryTC) { -#endif /* CONFIG_MIPS_MT_SMTC */ - +static void configure_exception_vector(void) +{ if (cpu_has_veic || cpu_has_vint) { unsigned long sr = set_c0_status(ST0_BEV); write_c0_ebase(ebase); @@ -1930,6 +1894,16 @@ void per_cpu_trap_init(bool is_boot_cpu) } else set_c0_cause(CAUSEF_IV); } +} + +void per_cpu_trap_init(bool is_boot_cpu) +{ + unsigned int cpu = smp_processor_id(); + + configure_status(); + configure_hwrena(); + + configure_exception_vector(); /* * Before R2 both interrupt numbers were fixed to 7, so on R2 only: @@ -1949,10 +1923,6 @@ void per_cpu_trap_init(bool is_boot_cpu) cp0_perfcount_irq = -1; } -#ifdef CONFIG_MIPS_MT_SMTC - } -#endif /* CONFIG_MIPS_MT_SMTC */ - if (!cpu_data[cpu].asid_cache) cpu_data[cpu].asid_cache = ASID_FIRST_VERSION; @@ -1961,23 +1931,10 @@ void per_cpu_trap_init(bool is_boot_cpu) BUG_ON(current->mm); enter_lazy_tlb(&init_mm, current); -#ifdef CONFIG_MIPS_MT_SMTC - if (bootTC) { -#endif /* CONFIG_MIPS_MT_SMTC */ /* Boot CPU's cache setup in setup_arch(). */ if (!is_boot_cpu) cpu_cache_init(); tlb_init(); -#ifdef CONFIG_MIPS_MT_SMTC - } else if (!secondaryTC) { - /* - * First TC in non-boot VPE must do subset of tlb_init() - * for MMU countrol registers. - */ - write_c0_pagemask(PM_DEFAULT_MASK); - write_c0_wired(0); - } -#endif /* CONFIG_MIPS_MT_SMTC */ TLBMISS_HANDLER_SETUP(); } @@ -2185,3 +2142,32 @@ void __init trap_init(void) cu2_notifier(default_cu2_call, 0x80000000); /* Run last */ } + +static int trap_pm_notifier(struct notifier_block *self, unsigned long cmd, + void *v) +{ + switch (cmd) { + case CPU_PM_ENTER_FAILED: + case CPU_PM_EXIT: + configure_status(); + configure_hwrena(); + configure_exception_vector(); + + /* Restore register with CPU number for TLB handlers */ + TLBMISS_HANDLER_RESTORE(); + + break; + } + + return NOTIFY_OK; +} + +static struct notifier_block trap_pm_notifier_block = { + .notifier_call = trap_pm_notifier, +}; + +static int __init trap_pm_init(void) +{ + return cpu_pm_register_notifier(&trap_pm_notifier_block); +} +arch_initcall(trap_pm_init); diff --git a/arch/mips/kernel/vpe-mt.c b/arch/mips/kernel/vpe-mt.c index 949ae0e17018..2e003b11a098 100644 --- a/arch/mips/kernel/vpe-mt.c +++ b/arch/mips/kernel/vpe-mt.c @@ -127,9 +127,8 @@ int vpe_run(struct vpe *v) clear_c0_mvpcontrol(MVPCONTROL_VPC); /* - * SMTC/SMVP kernels manage VPE enable independently, - * but uniprocessor kernels need to turn it on, even - * if that wasn't the pre-dvpe() state. + * SMVP kernels manage VPE enable independently, but uniprocessor + * kernels need to turn it on, even if that wasn't the pre-dvpe() state. */ #ifdef CONFIG_SMP evpe(vpeflags); @@ -454,12 +453,11 @@ int __init vpe_module_init(void) settc(tc); - /* Any TC that is bound to VPE0 gets left as is - in - * case we are running SMTC on VPE0. A TC that is bound - * to any other VPE gets bound to VPE0, ideally I'd like - * to make it homeless but it doesn't appear to let me - * bind a TC to a non-existent VPE. Which is perfectly - * reasonable. + /* + * A TC that is bound to any other VPE gets bound to + * VPE0, ideally I'd like to make it homeless but it + * doesn't appear to let me bind a TC to a non-existent + * VPE. Which is perfectly reasonable. * * The (un)bound state is visible to an EJTAG probe so * may notify GDB... diff --git a/arch/mips/lantiq/irq.c b/arch/mips/lantiq/irq.c index 85685e1cdb89..030568a70ac4 100644 --- a/arch/mips/lantiq/irq.c +++ b/arch/mips/lantiq/irq.c @@ -61,7 +61,7 @@ /* we have a cascade of 8 irqs */ #define MIPS_CPU_IRQ_CASCADE 8 -#if defined(CONFIG_MIPS_MT_SMP) || defined(CONFIG_MIPS_MT_SMTC) +#ifdef CONFIG_MIPS_MT_SMP int gic_present; #endif @@ -440,7 +440,7 @@ int __init icu_of_init(struct device_node *node, struct device_node *parent) arch_init_ipiirq(MIPS_CPU_IRQ_BASE + MIPS_CPU_IPI_CALL_IRQ, &irq_call); #endif -#if !defined(CONFIG_MIPS_MT_SMP) && !defined(CONFIG_MIPS_MT_SMTC) +#ifndef CONFIG_MIPS_MT_SMP set_c0_status(IE_IRQ0 | IE_IRQ1 | IE_IRQ2 | IE_IRQ3 | IE_IRQ4 | IE_IRQ5); #else diff --git a/arch/mips/lib/delay.c b/arch/mips/lib/delay.c index 705cfb7c1a74..21d27c6819a2 100644 --- a/arch/mips/lib/delay.c +++ b/arch/mips/lib/delay.c @@ -11,7 +11,9 @@ #include <linux/module.h> #include <linux/param.h> #include <linux/smp.h> +#include <linux/stringify.h> +#include <asm/asm.h> #include <asm/compiler.h> #include <asm/war.h> @@ -27,11 +29,7 @@ void __delay(unsigned long loops) " .set noreorder \n" " .align 3 \n" "1: bnez %0, 1b \n" -#if BITS_PER_LONG == 32 - " subu %0, %1 \n" -#else - " dsubu %0, %1 \n" -#endif + " " __stringify(LONG_SUBU) " %0, %1 \n" " .set reorder \n" : "=r" (loops) : GCC_DADDI_IMM_ASM() (1), "0" (loops)); diff --git a/arch/mips/lib/mips-atomic.c b/arch/mips/lib/mips-atomic.c index 6807f7172eaf..57bcdaf1f1c8 100644 --- a/arch/mips/lib/mips-atomic.c +++ b/arch/mips/lib/mips-atomic.c @@ -15,7 +15,7 @@ #include <linux/export.h> #include <linux/stringify.h> -#if !defined(CONFIG_CPU_MIPSR2) || defined(CONFIG_MIPS_MT_SMTC) +#ifndef CONFIG_CPU_MIPSR2 /* * For cli() we have to insert nops to make sure that the new value @@ -42,12 +42,7 @@ notrace void arch_local_irq_disable(void) __asm__ __volatile__( " .set push \n" " .set noat \n" -#ifdef CONFIG_MIPS_MT_SMTC - " mfc0 $1, $2, 1 \n" - " ori $1, 0x400 \n" - " .set noreorder \n" - " mtc0 $1, $2, 1 \n" -#elif defined(CONFIG_CPU_MIPSR2) +#if defined(CONFIG_CPU_MIPSR2) /* see irqflags.h for inline function */ #else " mfc0 $1,$12 \n" @@ -77,13 +72,7 @@ notrace unsigned long arch_local_irq_save(void) " .set push \n" " .set reorder \n" " .set noat \n" -#ifdef CONFIG_MIPS_MT_SMTC - " mfc0 %[flags], $2, 1 \n" - " ori $1, %[flags], 0x400 \n" - " .set noreorder \n" - " mtc0 $1, $2, 1 \n" - " andi %[flags], %[flags], 0x400 \n" -#elif defined(CONFIG_CPU_MIPSR2) +#if defined(CONFIG_CPU_MIPSR2) /* see irqflags.h for inline function */ #else " mfc0 %[flags], $12 \n" @@ -108,29 +97,13 @@ notrace void arch_local_irq_restore(unsigned long flags) { unsigned long __tmp1; -#ifdef CONFIG_MIPS_MT_SMTC - /* - * SMTC kernel needs to do a software replay of queued - * IPIs, at the cost of branch and call overhead on each - * local_irq_restore() - */ - if (unlikely(!(flags & 0x0400))) - smtc_ipi_replay(); -#endif preempt_disable(); __asm__ __volatile__( " .set push \n" " .set noreorder \n" " .set noat \n" -#ifdef CONFIG_MIPS_MT_SMTC - " mfc0 $1, $2, 1 \n" - " andi %[flags], 0x400 \n" - " ori $1, 0x400 \n" - " xori $1, 0x400 \n" - " or %[flags], $1 \n" - " mtc0 %[flags], $2, 1 \n" -#elif defined(CONFIG_CPU_MIPSR2) && defined(CONFIG_IRQ_CPU) +#if defined(CONFIG_CPU_MIPSR2) && defined(CONFIG_IRQ_CPU) /* see irqflags.h for inline function */ #elif defined(CONFIG_CPU_MIPSR2) /* see irqflags.h for inline function */ @@ -163,14 +136,7 @@ notrace void __arch_local_irq_restore(unsigned long flags) " .set push \n" " .set noreorder \n" " .set noat \n" -#ifdef CONFIG_MIPS_MT_SMTC - " mfc0 $1, $2, 1 \n" - " andi %[flags], 0x400 \n" - " ori $1, 0x400 \n" - " xori $1, 0x400 \n" - " or %[flags], $1 \n" - " mtc0 %[flags], $2, 1 \n" -#elif defined(CONFIG_CPU_MIPSR2) && defined(CONFIG_IRQ_CPU) +#if defined(CONFIG_CPU_MIPSR2) && defined(CONFIG_IRQ_CPU) /* see irqflags.h for inline function */ #elif defined(CONFIG_CPU_MIPSR2) /* see irqflags.h for inline function */ @@ -192,4 +158,4 @@ notrace void __arch_local_irq_restore(unsigned long flags) } EXPORT_SYMBOL(__arch_local_irq_restore); -#endif /* !defined(CONFIG_CPU_MIPSR2) || defined(CONFIG_MIPS_MT_SMTC) */ +#endif /* !CONFIG_CPU_MIPSR2 */ diff --git a/arch/mips/loongson/Kconfig b/arch/mips/loongson/Kconfig index 603d79a95f47..e6a86ccc4421 100644 --- a/arch/mips/loongson/Kconfig +++ b/arch/mips/loongson/Kconfig @@ -95,10 +95,11 @@ config CS5536 config CS5536_MFGPT bool "CS5536 MFGPT Timer" - depends on CS5536 + depends on CS5536 && !HIGH_RES_TIMERS select MIPS_EXTERNAL_TIMER help - This option enables the mfgpt0 timer of AMD CS5536. + This option enables the mfgpt0 timer of AMD CS5536. With this timer + switched on you can not use high resolution timers. If you want to enable the Loongson2 CPUFreq Driver, Please enable this option at first, otherwise, You will get wrong system time. diff --git a/arch/mips/loongson/loongson-3/smp.c b/arch/mips/loongson/loongson-3/smp.c index c665fe16d4c9..1e8894020ea5 100644 --- a/arch/mips/loongson/loongson-3/smp.c +++ b/arch/mips/loongson/loongson-3/smp.c @@ -279,13 +279,6 @@ static void loongson3_boot_secondary(int cpu, struct task_struct *idle) loongson3_ipi_write64(startargs[0], (void *)(ipi_mailbox_buf[cpu]+0x0)); } -/* - * Final cleanup after all secondaries booted - */ -static void __init loongson3_cpus_done(void) -{ -} - #ifdef CONFIG_HOTPLUG_CPU static int loongson3_cpu_disable(void) @@ -432,7 +425,6 @@ struct plat_smp_ops loongson3_smp_ops = { .send_ipi_mask = loongson3_send_ipi_mask, .init_secondary = loongson3_init_secondary, .smp_finish = loongson3_smp_finish, - .cpus_done = loongson3_cpus_done, .boot_secondary = loongson3_boot_secondary, .smp_setup = loongson3_smp_setup, .prepare_cpus = loongson3_prepare_cpus, diff --git a/arch/mips/loongson1/Kconfig b/arch/mips/loongson1/Kconfig index fbf75f635798..e23c25d09963 100644 --- a/arch/mips/loongson1/Kconfig +++ b/arch/mips/loongson1/Kconfig @@ -14,6 +14,7 @@ config LOONGSON1_LS1B select SYS_SUPPORTS_32BIT_KERNEL select SYS_SUPPORTS_LITTLE_ENDIAN select SYS_SUPPORTS_HIGHMEM + select SYS_SUPPORTS_MIPS16 select SYS_HAS_EARLY_PRINTK select COMMON_CLK diff --git a/arch/mips/math-emu/Makefile b/arch/mips/math-emu/Makefile index 121a848a3594..619cfc1a2442 100644 --- a/arch/mips/math-emu/Makefile +++ b/arch/mips/math-emu/Makefile @@ -2,10 +2,12 @@ # Makefile for the Linux/MIPS kernel FPU emulation. # -obj-y := cp1emu.o ieee754m.o ieee754d.o ieee754dp.o ieee754sp.o ieee754.o \ - ieee754xcpt.o dp_frexp.o dp_modf.o dp_div.o dp_mul.o dp_sub.o \ - dp_add.o dp_fsp.o dp_cmp.o dp_logb.o dp_scalb.o dp_simple.o \ - dp_tint.o dp_fint.o dp_tlong.o dp_flong.o sp_frexp.o sp_modf.o \ - sp_div.o sp_mul.o sp_sub.o sp_add.o sp_fdp.o sp_cmp.o sp_logb.o \ - sp_scalb.o sp_simple.o sp_tint.o sp_fint.o sp_tlong.o sp_flong.o \ - dp_sqrt.o sp_sqrt.o kernel_linkage.o dsemul.o +obj-y += cp1emu.o ieee754dp.o ieee754sp.o ieee754.o dp_div.o dp_mul.o \ + dp_sub.o dp_add.o dp_fsp.o dp_cmp.o dp_simple.o dp_tint.o \ + dp_fint.o dp_tlong.o dp_flong.o sp_div.o sp_mul.o sp_sub.o \ + sp_add.o sp_fdp.o sp_cmp.o sp_simple.o sp_tint.o sp_fint.o \ + sp_tlong.o sp_flong.o dsemul.o + +lib-y += ieee754d.o dp_sqrt.o sp_sqrt.o + +obj-$(CONFIG_DEBUG_FS) += me-debugfs.o diff --git a/arch/mips/math-emu/cp1emu.c b/arch/mips/math-emu/cp1emu.c index 7b3c9acae689..736c17a226e9 100644 --- a/arch/mips/math-emu/cp1emu.c +++ b/arch/mips/math-emu/cp1emu.c @@ -1,5 +1,5 @@ /* - * cp1emu.c: a MIPS coprocessor 1 (fpu) instruction emulator + * cp1emu.c: a MIPS coprocessor 1 (FPU) instruction emulator * * MIPS floating point support * Copyright (C) 1994-2000 Algorithmics Ltd. @@ -18,61 +18,46 @@ * * You should have received a copy of the GNU General Public License along * with this program; if not, write to the Free Software Foundation, Inc., - * 59 Temple Place - Suite 330, Boston MA 02111-1307, USA. + * 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. * * A complete emulator for MIPS coprocessor 1 instructions. This is * required for #float(switch) or #float(trap), where it catches all * COP1 instructions via the "CoProcessor Unusable" exception. * * More surprisingly it is also required for #float(ieee), to help out - * the hardware fpu at the boundaries of the IEEE-754 representation + * the hardware FPU at the boundaries of the IEEE-754 representation * (denormalised values, infinities, underflow, etc). It is made * quite nasty because emulation of some non-COP1 instructions is * required, e.g. in branch delay slots. * - * Note if you know that you won't have an fpu, then you'll get much + * Note if you know that you won't have an FPU, then you'll get much * better performance by compiling with -msoft-float! */ #include <linux/sched.h> -#include <linux/module.h> #include <linux/debugfs.h> +#include <linux/kconfig.h> +#include <linux/percpu-defs.h> #include <linux/perf_event.h> +#include <asm/branch.h> #include <asm/inst.h> -#include <asm/bootinfo.h> -#include <asm/processor.h> #include <asm/ptrace.h> #include <asm/signal.h> -#include <asm/mipsregs.h> +#include <asm/uaccess.h> + +#include <asm/processor.h> #include <asm/fpu_emulator.h> #include <asm/fpu.h> -#include <asm/uaccess.h> -#include <asm/branch.h> #include "ieee754.h" -/* Strap kernel emulator for full MIPS IV emulation */ - -#ifdef __mips -#undef __mips -#endif -#define __mips 4 - /* Function which emulates a floating point instruction. */ static int fpu_emu(struct pt_regs *, struct mips_fpu_struct *, mips_instruction); -#if __mips >= 4 && __mips != 32 static int fpux_emu(struct pt_regs *, struct mips_fpu_struct *, mips_instruction, void *__user *); -#endif - -/* Further private data for which no space exists in mips_fpu_struct */ - -#ifdef CONFIG_DEBUG_FS -DEFINE_PER_CPU(struct mips_fpu_emulator_stats, fpuemustats); -#endif /* Control registers */ @@ -82,27 +67,6 @@ DEFINE_PER_CPU(struct mips_fpu_emulator_stats, fpuemustats); /* Determine rounding mode from the RM bits of the FCSR */ #define modeindex(v) ((v) & FPU_CSR_RM) -/* microMIPS bitfields */ -#define MM_POOL32A_MINOR_MASK 0x3f -#define MM_POOL32A_MINOR_SHIFT 0x6 -#define MM_MIPS32_COND_FC 0x30 - -/* Convert Mips rounding mode (0..3) to IEEE library modes. */ -static const unsigned char ieee_rm[4] = { - [FPU_CSR_RN] = IEEE754_RN, - [FPU_CSR_RZ] = IEEE754_RZ, - [FPU_CSR_RU] = IEEE754_RU, - [FPU_CSR_RD] = IEEE754_RD, -}; -/* Convert IEEE library modes to Mips rounding mode (0..3). */ -static const unsigned char mips_rm[4] = { - [IEEE754_RN] = FPU_CSR_RN, - [IEEE754_RZ] = FPU_CSR_RZ, - [IEEE754_RD] = FPU_CSR_RD, - [IEEE754_RU] = FPU_CSR_RU, -}; - -#if __mips >= 4 /* convert condition code register number to csr bit */ static const unsigned int fpucondbit[8] = { FPU_CSR_COND0, @@ -114,10 +78,6 @@ static const unsigned int fpucondbit[8] = { FPU_CSR_COND6, FPU_CSR_COND7 }; -#endif - -/* (microMIPS) Convert 16-bit register encoding to 32-bit register encoding. */ -static const unsigned int reg16to32map[8] = {16, 17, 2, 3, 4, 5, 6, 7}; /* (microMIPS) Convert certain microMIPS instructions to MIPS32 format. */ static const int sd_format[] = {16, 17, 0, 0, 0, 0, 0, 0}; @@ -466,199 +426,6 @@ static int microMIPS32_to_MIPS32(union mips_instruction *insn_ptr) return 0; } -int mm_isBranchInstr(struct pt_regs *regs, struct mm_decoded_insn dec_insn, - unsigned long *contpc) -{ - union mips_instruction insn = (union mips_instruction)dec_insn.insn; - int bc_false = 0; - unsigned int fcr31; - unsigned int bit; - - if (!cpu_has_mmips) - return 0; - - switch (insn.mm_i_format.opcode) { - case mm_pool32a_op: - if ((insn.mm_i_format.simmediate & MM_POOL32A_MINOR_MASK) == - mm_pool32axf_op) { - switch (insn.mm_i_format.simmediate >> - MM_POOL32A_MINOR_SHIFT) { - case mm_jalr_op: - case mm_jalrhb_op: - case mm_jalrs_op: - case mm_jalrshb_op: - if (insn.mm_i_format.rt != 0) /* Not mm_jr */ - regs->regs[insn.mm_i_format.rt] = - regs->cp0_epc + - dec_insn.pc_inc + - dec_insn.next_pc_inc; - *contpc = regs->regs[insn.mm_i_format.rs]; - return 1; - } - } - break; - case mm_pool32i_op: - switch (insn.mm_i_format.rt) { - case mm_bltzals_op: - case mm_bltzal_op: - regs->regs[31] = regs->cp0_epc + - dec_insn.pc_inc + - dec_insn.next_pc_inc; - /* Fall through */ - case mm_bltz_op: - if ((long)regs->regs[insn.mm_i_format.rs] < 0) - *contpc = regs->cp0_epc + - dec_insn.pc_inc + - (insn.mm_i_format.simmediate << 1); - else - *contpc = regs->cp0_epc + - dec_insn.pc_inc + - dec_insn.next_pc_inc; - return 1; - case mm_bgezals_op: - case mm_bgezal_op: - regs->regs[31] = regs->cp0_epc + - dec_insn.pc_inc + - dec_insn.next_pc_inc; - /* Fall through */ - case mm_bgez_op: - if ((long)regs->regs[insn.mm_i_format.rs] >= 0) - *contpc = regs->cp0_epc + - dec_insn.pc_inc + - (insn.mm_i_format.simmediate << 1); - else - *contpc = regs->cp0_epc + - dec_insn.pc_inc + - dec_insn.next_pc_inc; - return 1; - case mm_blez_op: - if ((long)regs->regs[insn.mm_i_format.rs] <= 0) - *contpc = regs->cp0_epc + - dec_insn.pc_inc + - (insn.mm_i_format.simmediate << 1); - else - *contpc = regs->cp0_epc + - dec_insn.pc_inc + - dec_insn.next_pc_inc; - return 1; - case mm_bgtz_op: - if ((long)regs->regs[insn.mm_i_format.rs] <= 0) - *contpc = regs->cp0_epc + - dec_insn.pc_inc + - (insn.mm_i_format.simmediate << 1); - else - *contpc = regs->cp0_epc + - dec_insn.pc_inc + - dec_insn.next_pc_inc; - return 1; - case mm_bc2f_op: - case mm_bc1f_op: - bc_false = 1; - /* Fall through */ - case mm_bc2t_op: - case mm_bc1t_op: - preempt_disable(); - if (is_fpu_owner()) - asm volatile("cfc1\t%0,$31" : "=r" (fcr31)); - else - fcr31 = current->thread.fpu.fcr31; - preempt_enable(); - - if (bc_false) - fcr31 = ~fcr31; - - bit = (insn.mm_i_format.rs >> 2); - bit += (bit != 0); - bit += 23; - if (fcr31 & (1 << bit)) - *contpc = regs->cp0_epc + - dec_insn.pc_inc + - (insn.mm_i_format.simmediate << 1); - else - *contpc = regs->cp0_epc + - dec_insn.pc_inc + dec_insn.next_pc_inc; - return 1; - } - break; - case mm_pool16c_op: - switch (insn.mm_i_format.rt) { - case mm_jalr16_op: - case mm_jalrs16_op: - regs->regs[31] = regs->cp0_epc + - dec_insn.pc_inc + dec_insn.next_pc_inc; - /* Fall through */ - case mm_jr16_op: - *contpc = regs->regs[insn.mm_i_format.rs]; - return 1; - } - break; - case mm_beqz16_op: - if ((long)regs->regs[reg16to32map[insn.mm_b1_format.rs]] == 0) - *contpc = regs->cp0_epc + - dec_insn.pc_inc + - (insn.mm_b1_format.simmediate << 1); - else - *contpc = regs->cp0_epc + - dec_insn.pc_inc + dec_insn.next_pc_inc; - return 1; - case mm_bnez16_op: - if ((long)regs->regs[reg16to32map[insn.mm_b1_format.rs]] != 0) - *contpc = regs->cp0_epc + - dec_insn.pc_inc + - (insn.mm_b1_format.simmediate << 1); - else - *contpc = regs->cp0_epc + - dec_insn.pc_inc + dec_insn.next_pc_inc; - return 1; - case mm_b16_op: - *contpc = regs->cp0_epc + dec_insn.pc_inc + - (insn.mm_b0_format.simmediate << 1); - return 1; - case mm_beq32_op: - if (regs->regs[insn.mm_i_format.rs] == - regs->regs[insn.mm_i_format.rt]) - *contpc = regs->cp0_epc + - dec_insn.pc_inc + - (insn.mm_i_format.simmediate << 1); - else - *contpc = regs->cp0_epc + - dec_insn.pc_inc + - dec_insn.next_pc_inc; - return 1; - case mm_bne32_op: - if (regs->regs[insn.mm_i_format.rs] != - regs->regs[insn.mm_i_format.rt]) - *contpc = regs->cp0_epc + - dec_insn.pc_inc + - (insn.mm_i_format.simmediate << 1); - else - *contpc = regs->cp0_epc + - dec_insn.pc_inc + dec_insn.next_pc_inc; - return 1; - case mm_jalx32_op: - regs->regs[31] = regs->cp0_epc + - dec_insn.pc_inc + dec_insn.next_pc_inc; - *contpc = regs->cp0_epc + dec_insn.pc_inc; - *contpc >>= 28; - *contpc <<= 28; - *contpc |= (insn.j_format.target << 2); - return 1; - case mm_jals32_op: - case mm_jal32_op: - regs->regs[31] = regs->cp0_epc + - dec_insn.pc_inc + dec_insn.next_pc_inc; - /* Fall through */ - case mm_j32_op: - *contpc = regs->cp0_epc + dec_insn.pc_inc; - *contpc >>= 27; - *contpc <<= 27; - *contpc |= (insn.j_format.target << 1); - set_isa16_mode(*contpc); - return 1; - } - return 0; -} - /* * Redundant with logic already in kernel/branch.c, * embedded in compute_return_epc. At some point, @@ -817,7 +584,11 @@ static int isBranchInstr(struct pt_regs *regs, struct mm_decoded_insn dec_insn, if (insn.i_format.rs == bc_op) { preempt_disable(); if (is_fpu_owner()) - asm volatile("cfc1\t%0,$31" : "=r" (fcr31)); + asm volatile( + ".set push\n" + "\t.set mips1\n" + "\tcfc1\t%0,$31\n" + "\t.set pop" : "=r" (fcr31)); else fcr31 = current->thread.fpu.fcr31; preempt_enable(); @@ -867,23 +638,25 @@ static int isBranchInstr(struct pt_regs *regs, struct mm_decoded_insn dec_insn, */ static inline int cop1_64bit(struct pt_regs *xcp) { -#if defined(CONFIG_64BIT) && !defined(CONFIG_MIPS32_O32) - return 1; -#elif defined(CONFIG_32BIT) && !defined(CONFIG_MIPS_O32_FP64_SUPPORT) - return 0; -#else + if (config_enabled(CONFIG_64BIT) && !config_enabled(CONFIG_MIPS32_O32)) + return 1; + else if (config_enabled(CONFIG_32BIT) && + !config_enabled(CONFIG_MIPS_O32_FP64_SUPPORT)) + return 0; + return !test_thread_flag(TIF_32BIT_FPREGS); -#endif } -#define SIFROMREG(si, x) do { \ +#define SIFROMREG(si, x) \ +do { \ if (cop1_64bit(xcp)) \ (si) = get_fpr32(&ctx->fpr[x], 0); \ else \ (si) = get_fpr32(&ctx->fpr[(x) & ~1], (x) & 1); \ } while (0) -#define SITOREG(si, x) do { \ +#define SITOREG(si, x) \ +do { \ if (cop1_64bit(xcp)) { \ unsigned i; \ set_fpr32(&ctx->fpr[x], 0, si); \ @@ -896,17 +669,19 @@ static inline int cop1_64bit(struct pt_regs *xcp) #define SIFROMHREG(si, x) ((si) = get_fpr32(&ctx->fpr[x], 1)) -#define SITOHREG(si, x) do { \ +#define SITOHREG(si, x) \ +do { \ unsigned i; \ set_fpr32(&ctx->fpr[x], 1, si); \ for (i = 2; i < ARRAY_SIZE(ctx->fpr[x].val32); i++) \ set_fpr32(&ctx->fpr[x], i, 0); \ } while (0) -#define DIFROMREG(di, x) \ +#define DIFROMREG(di, x) \ ((di) = get_fpr64(&ctx->fpr[(x) & ~(cop1_64bit(xcp) == 0)], 0)) -#define DITOREG(di, x) do { \ +#define DITOREG(di, x) \ +do { \ unsigned fpr, i; \ fpr = (x) & ~(cop1_64bit(xcp) == 0); \ set_fpr64(&ctx->fpr[fpr], 0, di); \ @@ -927,23 +702,36 @@ static inline int cop1_64bit(struct pt_regs *xcp) static int cop1Emulate(struct pt_regs *xcp, struct mips_fpu_struct *ctx, struct mm_decoded_insn dec_insn, void *__user *fault_addr) { - mips_instruction ir; unsigned long contpc = xcp->cp0_epc + dec_insn.pc_inc; - unsigned int cond; - int pc_inc; + unsigned int cond, cbit; + mips_instruction ir; + int likely, pc_inc; + u32 __user *wva; + u64 __user *dva; + u32 value; + u32 wval; + u64 dval; + int sig; + + /* + * These are giving gcc a gentle hint about what to expect in + * dec_inst in order to do better optimization. + */ + if (!cpu_has_mmips && dec_insn.micro_mips_mode) + unreachable(); /* XXX NEC Vr54xx bug workaround */ - if (xcp->cp0_cause & CAUSEF_BD) { + if (delay_slot(xcp)) { if (dec_insn.micro_mips_mode) { if (!mm_isBranchInstr(xcp, dec_insn, &contpc)) - xcp->cp0_cause &= ~CAUSEF_BD; + clear_delay_slot(xcp); } else { if (!isBranchInstr(xcp, dec_insn, &contpc)) - xcp->cp0_cause &= ~CAUSEF_BD; + clear_delay_slot(xcp); } } - if (xcp->cp0_cause & CAUSEF_BD) { + if (delay_slot(xcp)) { /* * The instruction to be emulated is in a branch delay slot * which means that we have to emulate the branch instruction @@ -985,96 +773,85 @@ static int cop1Emulate(struct pt_regs *xcp, struct mips_fpu_struct *ctx, return SIGILL; } - emul: +emul: perf_sw_event(PERF_COUNT_SW_EMULATION_FAULTS, 1, xcp, 0); MIPS_FPU_EMU_INC_STATS(emulated); switch (MIPSInst_OPCODE(ir)) { - case ldc1_op:{ - u64 __user *va = (u64 __user *) (xcp->regs[MIPSInst_RS(ir)] + - MIPSInst_SIMM(ir)); - u64 val; - + case ldc1_op: + dva = (u64 __user *) (xcp->regs[MIPSInst_RS(ir)] + + MIPSInst_SIMM(ir)); MIPS_FPU_EMU_INC_STATS(loads); - if (!access_ok(VERIFY_READ, va, sizeof(u64))) { + if (!access_ok(VERIFY_READ, dva, sizeof(u64))) { MIPS_FPU_EMU_INC_STATS(errors); - *fault_addr = va; + *fault_addr = dva; return SIGBUS; } - if (__get_user(val, va)) { + if (__get_user(dval, dva)) { MIPS_FPU_EMU_INC_STATS(errors); - *fault_addr = va; + *fault_addr = dva; return SIGSEGV; } - DITOREG(val, MIPSInst_RT(ir)); + DITOREG(dval, MIPSInst_RT(ir)); break; - } - - case sdc1_op:{ - u64 __user *va = (u64 __user *) (xcp->regs[MIPSInst_RS(ir)] + - MIPSInst_SIMM(ir)); - u64 val; + case sdc1_op: + dva = (u64 __user *) (xcp->regs[MIPSInst_RS(ir)] + + MIPSInst_SIMM(ir)); MIPS_FPU_EMU_INC_STATS(stores); - DIFROMREG(val, MIPSInst_RT(ir)); - if (!access_ok(VERIFY_WRITE, va, sizeof(u64))) { + DIFROMREG(dval, MIPSInst_RT(ir)); + if (!access_ok(VERIFY_WRITE, dva, sizeof(u64))) { MIPS_FPU_EMU_INC_STATS(errors); - *fault_addr = va; + *fault_addr = dva; return SIGBUS; } - if (__put_user(val, va)) { + if (__put_user(dval, dva)) { MIPS_FPU_EMU_INC_STATS(errors); - *fault_addr = va; + *fault_addr = dva; return SIGSEGV; } break; - } - - case lwc1_op:{ - u32 __user *va = (u32 __user *) (xcp->regs[MIPSInst_RS(ir)] + - MIPSInst_SIMM(ir)); - u32 val; + case lwc1_op: + wva = (u32 __user *) (xcp->regs[MIPSInst_RS(ir)] + + MIPSInst_SIMM(ir)); MIPS_FPU_EMU_INC_STATS(loads); - if (!access_ok(VERIFY_READ, va, sizeof(u32))) { + if (!access_ok(VERIFY_READ, wva, sizeof(u32))) { MIPS_FPU_EMU_INC_STATS(errors); - *fault_addr = va; + *fault_addr = wva; return SIGBUS; } - if (__get_user(val, va)) { + if (__get_user(wval, wva)) { MIPS_FPU_EMU_INC_STATS(errors); - *fault_addr = va; + *fault_addr = wva; return SIGSEGV; } - SITOREG(val, MIPSInst_RT(ir)); + SITOREG(wval, MIPSInst_RT(ir)); break; - } - - case swc1_op:{ - u32 __user *va = (u32 __user *) (xcp->regs[MIPSInst_RS(ir)] + - MIPSInst_SIMM(ir)); - u32 val; + case swc1_op: + wva = (u32 __user *) (xcp->regs[MIPSInst_RS(ir)] + + MIPSInst_SIMM(ir)); MIPS_FPU_EMU_INC_STATS(stores); - SIFROMREG(val, MIPSInst_RT(ir)); - if (!access_ok(VERIFY_WRITE, va, sizeof(u32))) { + SIFROMREG(wval, MIPSInst_RT(ir)); + if (!access_ok(VERIFY_WRITE, wva, sizeof(u32))) { MIPS_FPU_EMU_INC_STATS(errors); - *fault_addr = va; + *fault_addr = wva; return SIGBUS; } - if (__put_user(val, va)) { + if (__put_user(wval, wva)) { MIPS_FPU_EMU_INC_STATS(errors); - *fault_addr = va; + *fault_addr = wva; return SIGSEGV; } break; - } case cop1_op: switch (MIPSInst_RS(ir)) { - -#if defined(__mips64) case dmfc_op: + if (!cpu_has_mips_3_4_5 && !cpu_has_mips64) + return SIGILL; + /* copregister fs -> gpr[rt] */ if (MIPSInst_RT(ir) != 0) { DIFROMREG(xcp->regs[MIPSInst_RT(ir)], @@ -1083,10 +860,12 @@ static int cop1Emulate(struct pt_regs *xcp, struct mips_fpu_struct *ctx, break; case dmtc_op: + if (!cpu_has_mips_3_4_5 && !cpu_has_mips64) + return SIGILL; + /* copregister fs <- rt */ DITOREG(xcp->regs[MIPSInst_RT(ir)], MIPSInst_RD(ir)); break; -#endif case mfhc_op: if (!cpu_has_mips_r2) @@ -1120,19 +899,14 @@ static int cop1Emulate(struct pt_regs *xcp, struct mips_fpu_struct *ctx, SITOREG(xcp->regs[MIPSInst_RT(ir)], MIPSInst_RD(ir)); break; - case cfc_op:{ + case cfc_op: /* cop control register rd -> gpr[rt] */ - u32 value; - if (MIPSInst_RD(ir) == FPCREG_CSR) { value = ctx->fcr31; - value = (value & ~FPU_CSR_RM) | - mips_rm[modeindex(value)]; -#ifdef CSRTRACE - printk("%p gpr[%d]<-csr=%08x\n", - (void *) (xcp->cp0_epc), - MIPSInst_RT(ir), value); -#endif + value = (value & ~FPU_CSR_RM) | modeindex(value); + pr_debug("%p gpr[%d]<-csr=%08x\n", + (void *) (xcp->cp0_epc), + MIPSInst_RT(ir), value); } else if (MIPSInst_RD(ir) == FPCREG_RID) value = 0; @@ -1141,12 +915,9 @@ static int cop1Emulate(struct pt_regs *xcp, struct mips_fpu_struct *ctx, if (MIPSInst_RT(ir)) xcp->regs[MIPSInst_RT(ir)] = value; break; - } - case ctc_op:{ + case ctc_op: /* copregister rd <- rt */ - u32 value; - if (MIPSInst_RT(ir) == 0) value = 0; else @@ -1155,37 +926,33 @@ static int cop1Emulate(struct pt_regs *xcp, struct mips_fpu_struct *ctx, /* we only have one writable control reg */ if (MIPSInst_RD(ir) == FPCREG_CSR) { -#ifdef CSRTRACE - printk("%p gpr[%d]->csr=%08x\n", - (void *) (xcp->cp0_epc), - MIPSInst_RT(ir), value); -#endif + pr_debug("%p gpr[%d]->csr=%08x\n", + (void *) (xcp->cp0_epc), + MIPSInst_RT(ir), value); /* * Don't write reserved bits, * and convert to ieee library modes */ - ctx->fcr31 = (value & - ~(FPU_CSR_RSVD | FPU_CSR_RM)) | - ieee_rm[modeindex(value)]; + ctx->fcr31 = (value & ~(FPU_CSR_RSVD | FPU_CSR_RM)) | + modeindex(value); } if ((ctx->fcr31 >> 5) & ctx->fcr31 & FPU_CSR_ALL_E) { return SIGFPE; } break; - } - case bc_op:{ - int likely = 0; - - if (xcp->cp0_cause & CAUSEF_BD) + case bc_op: + if (delay_slot(xcp)) return SIGILL; -#if __mips >= 4 - cond = ctx->fcr31 & fpucondbit[MIPSInst_RT(ir) >> 2]; -#else - cond = ctx->fcr31 & FPU_CSR_COND; -#endif + if (cpu_has_mips_4_5_r) + cbit = fpucondbit[MIPSInst_RT(ir) >> 2]; + else + cbit = FPU_CSR_COND; + cond = ctx->fcr31 & cbit; + + likely = 0; switch (MIPSInst_RT(ir) & 3) { case bcfl_op: likely = 1; @@ -1201,10 +968,10 @@ static int cop1Emulate(struct pt_regs *xcp, struct mips_fpu_struct *ctx, return SIGILL; } - xcp->cp0_cause |= CAUSEF_BD; + set_delay_slot(xcp); if (cond) { - /* branch taken: emulate dslot - * instruction + /* + * Branch taken: emulate dslot instruction */ xcp->cp0_epc += dec_insn.pc_inc; @@ -1238,23 +1005,37 @@ static int cop1Emulate(struct pt_regs *xcp, struct mips_fpu_struct *ctx, switch (MIPSInst_OPCODE(ir)) { case lwc1_op: + goto emul; + case swc1_op: -#if (__mips >= 2 || defined(__mips64)) + goto emul; + case ldc1_op: case sdc1_op: -#endif + if (cpu_has_mips_2_3_4_5 || + cpu_has_mips64) + goto emul; + + return SIGILL; + goto emul; + case cop1_op: -#if __mips >= 4 && __mips != 32 - case cop1x_op: -#endif - /* its one of ours */ goto emul; -#if __mips >= 4 + + case cop1x_op: + if (cpu_has_mips_4_5 || cpu_has_mips64) + /* its one of ours */ + goto emul; + + return SIGILL; + case spec_op: + if (!cpu_has_mips_4_5_r) + return SIGILL; + if (MIPSInst_FUNC(ir) == movc_op) goto emul; break; -#endif } /* @@ -1262,10 +1043,7 @@ static int cop1Emulate(struct pt_regs *xcp, struct mips_fpu_struct *ctx, * instruction in the dslot */ return mips_dsemul(xcp, ir, contpc); - } - else { - /* branch not taken */ - if (likely) { + } else if (likely) { /* branch not taken */ /* * branch likely nullifies * dslot if not taken @@ -1277,34 +1055,31 @@ static int cop1Emulate(struct pt_regs *xcp, struct mips_fpu_struct *ctx, * dslot as normal insn */ } - } break; - } default: if (!(MIPSInst_RS(ir) & 0x10)) return SIGILL; - { - int sig; - /* a real fpu computation instruction */ - if ((sig = fpu_emu(xcp, ctx, ir))) - return sig; - } + /* a real fpu computation instruction */ + if ((sig = fpu_emu(xcp, ctx, ir))) + return sig; } break; -#if __mips >= 4 && __mips != 32 - case cop1x_op:{ - int sig = fpux_emu(xcp, ctx, ir, fault_addr); + case cop1x_op: + if (!cpu_has_mips_4_5 && !cpu_has_mips64) + return SIGILL; + + sig = fpux_emu(xcp, ctx, ir, fault_addr); if (sig) return sig; break; - } -#endif -#if __mips >= 4 case spec_op: + if (!cpu_has_mips_4_5_r) + return SIGILL; + if (MIPSInst_FUNC(ir) != movc_op) return SIGILL; cond = fpucondbit[MIPSInst_RT(ir) >> 2]; @@ -1312,8 +1087,6 @@ static int cop1Emulate(struct pt_regs *xcp, struct mips_fpu_struct *ctx, xcp->regs[MIPSInst_RD(ir)] = xcp->regs[MIPSInst_RS(ir)]; break; -#endif - default: sigill: return SIGILL; @@ -1321,7 +1094,7 @@ sigill: /* we did it !! */ xcp->cp0_epc = contpc; - xcp->cp0_cause &= ~CAUSEF_BD; + clear_delay_slot(xcp); return 0; } @@ -1342,44 +1115,42 @@ static const unsigned char cmptab[8] = { }; -#if __mips >= 4 && __mips != 32 - /* * Additional MIPS4 instructions */ -#define DEF3OP(name, p, f1, f2, f3) \ -static ieee754##p fpemu_##p##_##name(ieee754##p r, ieee754##p s, \ - ieee754##p t) \ -{ \ - struct _ieee754_csr ieee754_csr_save; \ - s = f1(s, t); \ - ieee754_csr_save = ieee754_csr; \ - s = f2(s, r); \ - ieee754_csr_save.cx |= ieee754_csr.cx; \ - ieee754_csr_save.sx |= ieee754_csr.sx; \ - s = f3(s); \ - ieee754_csr.cx |= ieee754_csr_save.cx; \ - ieee754_csr.sx |= ieee754_csr_save.sx; \ - return s; \ +#define DEF3OP(name, p, f1, f2, f3) \ +static union ieee754##p fpemu_##p##_##name(union ieee754##p r, \ + union ieee754##p s, union ieee754##p t) \ +{ \ + struct _ieee754_csr ieee754_csr_save; \ + s = f1(s, t); \ + ieee754_csr_save = ieee754_csr; \ + s = f2(s, r); \ + ieee754_csr_save.cx |= ieee754_csr.cx; \ + ieee754_csr_save.sx |= ieee754_csr.sx; \ + s = f3(s); \ + ieee754_csr.cx |= ieee754_csr_save.cx; \ + ieee754_csr.sx |= ieee754_csr_save.sx; \ + return s; \ } -static ieee754dp fpemu_dp_recip(ieee754dp d) +static union ieee754dp fpemu_dp_recip(union ieee754dp d) { return ieee754dp_div(ieee754dp_one(0), d); } -static ieee754dp fpemu_dp_rsqrt(ieee754dp d) +static union ieee754dp fpemu_dp_rsqrt(union ieee754dp d) { return ieee754dp_div(ieee754dp_one(0), ieee754dp_sqrt(d)); } -static ieee754sp fpemu_sp_recip(ieee754sp s) +static union ieee754sp fpemu_sp_recip(union ieee754sp s) { return ieee754sp_div(ieee754sp_one(0), s); } -static ieee754sp fpemu_sp_rsqrt(ieee754sp s) +static union ieee754sp fpemu_sp_rsqrt(union ieee754sp s) { return ieee754sp_div(ieee754sp_one(0), ieee754sp_sqrt(s)); } @@ -1403,8 +1174,8 @@ static int fpux_emu(struct pt_regs *xcp, struct mips_fpu_struct *ctx, switch (MIPSInst_FMA_FFMT(ir)) { case s_fmt:{ /* 0 */ - ieee754sp(*handler) (ieee754sp, ieee754sp, ieee754sp); - ieee754sp fd, fr, fs, ft; + union ieee754sp(*handler) (union ieee754sp, union ieee754sp, union ieee754sp); + union ieee754sp fd, fr, fs, ft; u32 __user *va; u32 val; @@ -1467,18 +1238,26 @@ static int fpux_emu(struct pt_regs *xcp, struct mips_fpu_struct *ctx, SPTOREG(fd, MIPSInst_FD(ir)); copcsr: - if (ieee754_cxtest(IEEE754_INEXACT)) + if (ieee754_cxtest(IEEE754_INEXACT)) { + MIPS_FPU_EMU_INC_STATS(ieee754_inexact); rcsr |= FPU_CSR_INE_X | FPU_CSR_INE_S; - if (ieee754_cxtest(IEEE754_UNDERFLOW)) + } + if (ieee754_cxtest(IEEE754_UNDERFLOW)) { + MIPS_FPU_EMU_INC_STATS(ieee754_underflow); rcsr |= FPU_CSR_UDF_X | FPU_CSR_UDF_S; - if (ieee754_cxtest(IEEE754_OVERFLOW)) + } + if (ieee754_cxtest(IEEE754_OVERFLOW)) { + MIPS_FPU_EMU_INC_STATS(ieee754_overflow); rcsr |= FPU_CSR_OVF_X | FPU_CSR_OVF_S; - if (ieee754_cxtest(IEEE754_INVALID_OPERATION)) + } + if (ieee754_cxtest(IEEE754_INVALID_OPERATION)) { + MIPS_FPU_EMU_INC_STATS(ieee754_invalidop); rcsr |= FPU_CSR_INV_X | FPU_CSR_INV_S; + } ctx->fcr31 = (ctx->fcr31 & ~FPU_CSR_ALL_X) | rcsr; if ((ctx->fcr31 >> 5) & ctx->fcr31 & FPU_CSR_ALL_E) { - /*printk ("SIGFPE: fpu csr = %08x\n", + /*printk ("SIGFPE: FPU csr = %08x\n", ctx->fcr31); */ return SIGFPE; } @@ -1492,8 +1271,8 @@ static int fpux_emu(struct pt_regs *xcp, struct mips_fpu_struct *ctx, } case d_fmt:{ /* 1 */ - ieee754dp(*handler) (ieee754dp, ieee754dp, ieee754dp); - ieee754dp fd, fr, fs, ft; + union ieee754dp(*handler) (union ieee754dp, union ieee754dp, union ieee754dp); + union ieee754dp fd, fr, fs, ft; u64 __user *va; u64 val; @@ -1574,7 +1353,6 @@ static int fpux_emu(struct pt_regs *xcp, struct mips_fpu_struct *ctx, return 0; } -#endif @@ -1586,23 +1364,25 @@ static int fpu_emu(struct pt_regs *xcp, struct mips_fpu_struct *ctx, { int rfmt; /* resulting format */ unsigned rcsr = 0; /* resulting csr */ + unsigned int oldrm; + unsigned int cbit; unsigned cond; union { - ieee754dp d; - ieee754sp s; + union ieee754dp d; + union ieee754sp s; int w; -#ifdef __mips64 s64 l; -#endif } rv; /* resulting value */ + u64 bits; MIPS_FPU_EMU_INC_STATS(cp1ops); switch (rfmt = (MIPSInst_FFMT(ir) & 0xf)) { - case s_fmt:{ /* 0 */ + case s_fmt: { /* 0 */ union { - ieee754sp(*b) (ieee754sp, ieee754sp); - ieee754sp(*u) (ieee754sp); + union ieee754sp(*b) (union ieee754sp, union ieee754sp); + union ieee754sp(*u) (union ieee754sp); } handler; + union ieee754sp fs, ft; switch (MIPSInst_FUNC(ir)) { /* binary ops */ @@ -1620,148 +1400,167 @@ static int fpu_emu(struct pt_regs *xcp, struct mips_fpu_struct *ctx, goto scopbop; /* unary ops */ -#if __mips >= 2 || defined(__mips64) case fsqrt_op: + if (!cpu_has_mips_4_5_r) + return SIGILL; + handler.u = ieee754sp_sqrt; goto scopuop; -#endif -#if __mips >= 4 && __mips != 32 + + /* + * Note that on some MIPS IV implementations such as the + * R5000 and R8000 the FSQRT and FRECIP instructions do not + * achieve full IEEE-754 accuracy - however this emulator does. + */ case frsqrt_op: + if (!cpu_has_mips_4_5_r2) + return SIGILL; + handler.u = fpemu_sp_rsqrt; goto scopuop; + case frecip_op: + if (!cpu_has_mips_4_5_r2) + return SIGILL; + handler.u = fpemu_sp_recip; goto scopuop; -#endif -#if __mips >= 4 + case fmovc_op: + if (!cpu_has_mips_4_5_r) + return SIGILL; + cond = fpucondbit[MIPSInst_FT(ir) >> 2]; if (((ctx->fcr31 & cond) != 0) != ((MIPSInst_FT(ir) & 1) != 0)) return 0; SPFROMREG(rv.s, MIPSInst_FS(ir)); break; + case fmovz_op: + if (!cpu_has_mips_4_5_r) + return SIGILL; + if (xcp->regs[MIPSInst_FT(ir)] != 0) return 0; SPFROMREG(rv.s, MIPSInst_FS(ir)); break; + case fmovn_op: + if (!cpu_has_mips_4_5_r) + return SIGILL; + if (xcp->regs[MIPSInst_FT(ir)] == 0) return 0; SPFROMREG(rv.s, MIPSInst_FS(ir)); break; -#endif + case fabs_op: handler.u = ieee754sp_abs; goto scopuop; + case fneg_op: handler.u = ieee754sp_neg; goto scopuop; + case fmov_op: /* an easy one */ SPFROMREG(rv.s, MIPSInst_FS(ir)); goto copcsr; /* binary op on handler */ - scopbop: - { - ieee754sp fs, ft; - - SPFROMREG(fs, MIPSInst_FS(ir)); - SPFROMREG(ft, MIPSInst_FT(ir)); - - rv.s = (*handler.b) (fs, ft); - goto copcsr; - } - scopuop: - { - ieee754sp fs; +scopbop: + SPFROMREG(fs, MIPSInst_FS(ir)); + SPFROMREG(ft, MIPSInst_FT(ir)); - SPFROMREG(fs, MIPSInst_FS(ir)); - rv.s = (*handler.u) (fs); - goto copcsr; - } - copcsr: - if (ieee754_cxtest(IEEE754_INEXACT)) + rv.s = (*handler.b) (fs, ft); + goto copcsr; +scopuop: + SPFROMREG(fs, MIPSInst_FS(ir)); + rv.s = (*handler.u) (fs); + goto copcsr; +copcsr: + if (ieee754_cxtest(IEEE754_INEXACT)) { + MIPS_FPU_EMU_INC_STATS(ieee754_inexact); rcsr |= FPU_CSR_INE_X | FPU_CSR_INE_S; - if (ieee754_cxtest(IEEE754_UNDERFLOW)) + } + if (ieee754_cxtest(IEEE754_UNDERFLOW)) { + MIPS_FPU_EMU_INC_STATS(ieee754_underflow); rcsr |= FPU_CSR_UDF_X | FPU_CSR_UDF_S; - if (ieee754_cxtest(IEEE754_OVERFLOW)) + } + if (ieee754_cxtest(IEEE754_OVERFLOW)) { + MIPS_FPU_EMU_INC_STATS(ieee754_overflow); rcsr |= FPU_CSR_OVF_X | FPU_CSR_OVF_S; - if (ieee754_cxtest(IEEE754_ZERO_DIVIDE)) + } + if (ieee754_cxtest(IEEE754_ZERO_DIVIDE)) { + MIPS_FPU_EMU_INC_STATS(ieee754_zerodiv); rcsr |= FPU_CSR_DIV_X | FPU_CSR_DIV_S; - if (ieee754_cxtest(IEEE754_INVALID_OPERATION)) + } + if (ieee754_cxtest(IEEE754_INVALID_OPERATION)) { + MIPS_FPU_EMU_INC_STATS(ieee754_invalidop); rcsr |= FPU_CSR_INV_X | FPU_CSR_INV_S; + } break; /* unary conv ops */ case fcvts_op: return SIGILL; /* not defined */ - case fcvtd_op:{ - ieee754sp fs; + case fcvtd_op: SPFROMREG(fs, MIPSInst_FS(ir)); rv.d = ieee754dp_fsp(fs); rfmt = d_fmt; goto copcsr; - } - case fcvtw_op:{ - ieee754sp fs; + case fcvtw_op: SPFROMREG(fs, MIPSInst_FS(ir)); rv.w = ieee754sp_tint(fs); rfmt = w_fmt; goto copcsr; - } -#if __mips >= 2 || defined(__mips64) case fround_op: case ftrunc_op: case fceil_op: - case ffloor_op:{ - unsigned int oldrm = ieee754_csr.rm; - ieee754sp fs; + case ffloor_op: + if (!cpu_has_mips_2_3_4_5 && !cpu_has_mips64) + return SIGILL; + oldrm = ieee754_csr.rm; SPFROMREG(fs, MIPSInst_FS(ir)); - ieee754_csr.rm = ieee_rm[modeindex(MIPSInst_FUNC(ir))]; + ieee754_csr.rm = modeindex(MIPSInst_FUNC(ir)); rv.w = ieee754sp_tint(fs); ieee754_csr.rm = oldrm; rfmt = w_fmt; goto copcsr; - } -#endif /* __mips >= 2 */ -#if defined(__mips64) - case fcvtl_op:{ - ieee754sp fs; + case fcvtl_op: + if (!cpu_has_mips_3_4_5 && !cpu_has_mips64) + return SIGILL; SPFROMREG(fs, MIPSInst_FS(ir)); rv.l = ieee754sp_tlong(fs); rfmt = l_fmt; goto copcsr; - } case froundl_op: case ftruncl_op: case fceill_op: - case ffloorl_op:{ - unsigned int oldrm = ieee754_csr.rm; - ieee754sp fs; + case ffloorl_op: + if (!cpu_has_mips_3_4_5 && !cpu_has_mips64) + return SIGILL; + oldrm = ieee754_csr.rm; SPFROMREG(fs, MIPSInst_FS(ir)); - ieee754_csr.rm = ieee_rm[modeindex(MIPSInst_FUNC(ir))]; + ieee754_csr.rm = modeindex(MIPSInst_FUNC(ir)); rv.l = ieee754sp_tlong(fs); ieee754_csr.rm = oldrm; rfmt = l_fmt; goto copcsr; - } -#endif /* defined(__mips64) */ default: if (MIPSInst_FUNC(ir) >= fcmp_op) { unsigned cmpop = MIPSInst_FUNC(ir) - fcmp_op; - ieee754sp fs, ft; + union ieee754sp fs, ft; SPFROMREG(fs, MIPSInst_FS(ir)); SPFROMREG(ft, MIPSInst_FT(ir)); @@ -1774,19 +1573,18 @@ static int fpu_emu(struct pt_regs *xcp, struct mips_fpu_struct *ctx, else goto copcsr; - } - else { + } else return SIGILL; - } break; } break; } - case d_fmt:{ + case d_fmt: { + union ieee754dp fs, ft; union { - ieee754dp(*b) (ieee754dp, ieee754dp); - ieee754dp(*u) (ieee754dp); + union ieee754dp(*b) (union ieee754dp, union ieee754dp); + union ieee754dp(*u) (union ieee754dp); } handler; switch (MIPSInst_FUNC(ir)) { @@ -1805,21 +1603,33 @@ static int fpu_emu(struct pt_regs *xcp, struct mips_fpu_struct *ctx, goto dcopbop; /* unary ops */ -#if __mips >= 2 || defined(__mips64) case fsqrt_op: + if (!cpu_has_mips_2_3_4_5_r) + return SIGILL; + handler.u = ieee754dp_sqrt; goto dcopuop; -#endif -#if __mips >= 4 && __mips != 32 + /* + * Note that on some MIPS IV implementations such as the + * R5000 and R8000 the FSQRT and FRECIP instructions do not + * achieve full IEEE-754 accuracy - however this emulator does. + */ case frsqrt_op: + if (!cpu_has_mips_4_5_r2) + return SIGILL; + handler.u = fpemu_dp_rsqrt; goto dcopuop; case frecip_op: + if (!cpu_has_mips_4_5_r2) + return SIGILL; + handler.u = fpemu_dp_recip; goto dcopuop; -#endif -#if __mips >= 4 case fmovc_op: + if (!cpu_has_mips_4_5_r) + return SIGILL; + cond = fpucondbit[MIPSInst_FT(ir) >> 2]; if (((ctx->fcr31 & cond) != 0) != ((MIPSInst_FT(ir) & 1) != 0)) @@ -1827,16 +1637,21 @@ static int fpu_emu(struct pt_regs *xcp, struct mips_fpu_struct *ctx, DPFROMREG(rv.d, MIPSInst_FS(ir)); break; case fmovz_op: + if (!cpu_has_mips_4_5_r) + return SIGILL; + if (xcp->regs[MIPSInst_FT(ir)] != 0) return 0; DPFROMREG(rv.d, MIPSInst_FS(ir)); break; case fmovn_op: + if (!cpu_has_mips_4_5_r) + return SIGILL; + if (xcp->regs[MIPSInst_FT(ir)] == 0) return 0; DPFROMREG(rv.d, MIPSInst_FS(ir)); break; -#endif case fabs_op: handler.u = ieee754dp_abs; goto dcopuop; @@ -1851,91 +1666,78 @@ static int fpu_emu(struct pt_regs *xcp, struct mips_fpu_struct *ctx, goto copcsr; /* binary op on handler */ - dcopbop:{ - ieee754dp fs, ft; - - DPFROMREG(fs, MIPSInst_FS(ir)); - DPFROMREG(ft, MIPSInst_FT(ir)); - - rv.d = (*handler.b) (fs, ft); - goto copcsr; - } - dcopuop:{ - ieee754dp fs; - - DPFROMREG(fs, MIPSInst_FS(ir)); - rv.d = (*handler.u) (fs); - goto copcsr; - } +dcopbop: + DPFROMREG(fs, MIPSInst_FS(ir)); + DPFROMREG(ft, MIPSInst_FT(ir)); - /* unary conv ops */ - case fcvts_op:{ - ieee754dp fs; + rv.d = (*handler.b) (fs, ft); + goto copcsr; +dcopuop: + DPFROMREG(fs, MIPSInst_FS(ir)); + rv.d = (*handler.u) (fs); + goto copcsr; + /* + * unary conv ops + */ + case fcvts_op: DPFROMREG(fs, MIPSInst_FS(ir)); rv.s = ieee754sp_fdp(fs); rfmt = s_fmt; goto copcsr; - } + case fcvtd_op: return SIGILL; /* not defined */ - case fcvtw_op:{ - ieee754dp fs; - + case fcvtw_op: DPFROMREG(fs, MIPSInst_FS(ir)); rv.w = ieee754dp_tint(fs); /* wrong */ rfmt = w_fmt; goto copcsr; - } -#if __mips >= 2 || defined(__mips64) case fround_op: case ftrunc_op: case fceil_op: - case ffloor_op:{ - unsigned int oldrm = ieee754_csr.rm; - ieee754dp fs; + case ffloor_op: + if (!cpu_has_mips_2_3_4_5_r) + return SIGILL; + oldrm = ieee754_csr.rm; DPFROMREG(fs, MIPSInst_FS(ir)); - ieee754_csr.rm = ieee_rm[modeindex(MIPSInst_FUNC(ir))]; + ieee754_csr.rm = modeindex(MIPSInst_FUNC(ir)); rv.w = ieee754dp_tint(fs); ieee754_csr.rm = oldrm; rfmt = w_fmt; goto copcsr; - } -#endif -#if defined(__mips64) - case fcvtl_op:{ - ieee754dp fs; + case fcvtl_op: + if (!cpu_has_mips_3_4_5 && !cpu_has_mips64) + return SIGILL; DPFROMREG(fs, MIPSInst_FS(ir)); rv.l = ieee754dp_tlong(fs); rfmt = l_fmt; goto copcsr; - } case froundl_op: case ftruncl_op: case fceill_op: - case ffloorl_op:{ - unsigned int oldrm = ieee754_csr.rm; - ieee754dp fs; + case ffloorl_op: + if (!cpu_has_mips_3_4_5 && !cpu_has_mips64) + return SIGILL; + oldrm = ieee754_csr.rm; DPFROMREG(fs, MIPSInst_FS(ir)); - ieee754_csr.rm = ieee_rm[modeindex(MIPSInst_FUNC(ir))]; + ieee754_csr.rm = modeindex(MIPSInst_FUNC(ir)); rv.l = ieee754dp_tlong(fs); ieee754_csr.rm = oldrm; rfmt = l_fmt; goto copcsr; - } -#endif /* __mips >= 3 */ default: if (MIPSInst_FUNC(ir) >= fcmp_op) { unsigned cmpop = MIPSInst_FUNC(ir) - fcmp_op; - ieee754dp fs, ft; + union ieee754dp fs, ft; DPFROMREG(fs, MIPSInst_FS(ir)); DPFROMREG(ft, MIPSInst_FT(ir)); @@ -1957,11 +1759,8 @@ static int fpu_emu(struct pt_regs *xcp, struct mips_fpu_struct *ctx, break; } break; - } - - case w_fmt:{ - ieee754sp fs; + case w_fmt: switch (MIPSInst_FUNC(ir)) { case fcvts_op: /* convert word to single precision real */ @@ -1981,9 +1780,11 @@ static int fpu_emu(struct pt_regs *xcp, struct mips_fpu_struct *ctx, break; } -#if defined(__mips64) - case l_fmt:{ - u64 bits; + case l_fmt: + + if (!cpu_has_mips_3_4_5 && !cpu_has_mips64) + return SIGILL; + DIFROMREG(bits, MIPSInst_FS(ir)); switch (MIPSInst_FUNC(ir)) { @@ -2001,8 +1802,6 @@ static int fpu_emu(struct pt_regs *xcp, struct mips_fpu_struct *ctx, return SIGILL; } break; - } -#endif default: return SIGILL; @@ -2017,7 +1816,7 @@ static int fpu_emu(struct pt_regs *xcp, struct mips_fpu_struct *ctx, */ ctx->fcr31 = (ctx->fcr31 & ~FPU_CSR_ALL_X) | rcsr; if ((ctx->fcr31 >> 5) & ctx->fcr31 & FPU_CSR_ALL_E) { - /*printk ("SIGFPE: fpu csr = %08x\n",ctx->fcr31); */ + /*printk ("SIGFPE: FPU csr = %08x\n",ctx->fcr31); */ return SIGFPE; } @@ -2025,18 +1824,18 @@ static int fpu_emu(struct pt_regs *xcp, struct mips_fpu_struct *ctx, * Now we can safely write the result back to the register file. */ switch (rfmt) { - case -1:{ -#if __mips >= 4 - cond = fpucondbit[MIPSInst_FD(ir) >> 2]; -#else - cond = FPU_CSR_COND; -#endif + case -1: + + if (cpu_has_mips_4_5_r) + cbit = fpucondbit[MIPSInst_RT(ir) >> 2]; + else + cbit = FPU_CSR_COND; if (rv.w) - ctx->fcr31 |= cond; + ctx->fcr31 |= cbit; else - ctx->fcr31 &= ~cond; + ctx->fcr31 &= ~cbit; break; - } + case d_fmt: DPTOREG(rv.d, MIPSInst_FD(ir)); break; @@ -2046,11 +1845,12 @@ static int fpu_emu(struct pt_regs *xcp, struct mips_fpu_struct *ctx, case w_fmt: SITOREG(rv.w, MIPSInst_FD(ir)); break; -#if defined(__mips64) case l_fmt: + if (!cpu_has_mips_3_4_5 && !cpu_has_mips64) + return SIGILL; + DITOREG(rv.l, MIPSInst_FD(ir)); break; -#endif default: return SIGILL; } @@ -2138,11 +1938,7 @@ int fpu_emulator_cop1Handler(struct pt_regs *xcp, struct mips_fpu_struct *ctx, * ieee754_csr. But ieee754_csr.rm is ieee * library modes. (not mips rounding mode) */ - /* convert to ieee library modes */ - ieee754_csr.rm = ieee_rm[ieee754_csr.rm]; sig = cop1Emulate(xcp, ctx, dec_insn, fault_addr); - /* revert to mips rounding mode */ - ieee754_csr.rm = mips_rm[ieee754_csr.rm]; } if (has_fpu) @@ -2155,58 +1951,8 @@ int fpu_emulator_cop1Handler(struct pt_regs *xcp, struct mips_fpu_struct *ctx, /* SIGILL indicates a non-fpu instruction */ if (sig == SIGILL && xcp->cp0_epc != oldepc) - /* but if epc has advanced, then ignore it */ + /* but if EPC has advanced, then ignore it */ sig = 0; return sig; } - -#ifdef CONFIG_DEBUG_FS - -static int fpuemu_stat_get(void *data, u64 *val) -{ - int cpu; - unsigned long sum = 0; - for_each_online_cpu(cpu) { - struct mips_fpu_emulator_stats *ps; - local_t *pv; - ps = &per_cpu(fpuemustats, cpu); - pv = (void *)ps + (unsigned long)data; - sum += local_read(pv); - } - *val = sum; - return 0; -} -DEFINE_SIMPLE_ATTRIBUTE(fops_fpuemu_stat, fpuemu_stat_get, NULL, "%llu\n"); - -extern struct dentry *mips_debugfs_dir; -static int __init debugfs_fpuemu(void) -{ - struct dentry *d, *dir; - - if (!mips_debugfs_dir) - return -ENODEV; - dir = debugfs_create_dir("fpuemustats", mips_debugfs_dir); - if (!dir) - return -ENOMEM; - -#define FPU_STAT_CREATE(M) \ - do { \ - d = debugfs_create_file(#M , S_IRUGO, dir, \ - (void *)offsetof(struct mips_fpu_emulator_stats, M), \ - &fops_fpuemu_stat); \ - if (!d) \ - return -ENOMEM; \ - } while (0) - - FPU_STAT_CREATE(emulated); - FPU_STAT_CREATE(loads); - FPU_STAT_CREATE(stores); - FPU_STAT_CREATE(cp1ops); - FPU_STAT_CREATE(cp1xops); - FPU_STAT_CREATE(errors); - - return 0; -} -__initcall(debugfs_fpuemu); -#endif diff --git a/arch/mips/math-emu/dp_add.c b/arch/mips/math-emu/dp_add.c index c57c8adc42c4..7f64577df984 100644 --- a/arch/mips/math-emu/dp_add.c +++ b/arch/mips/math-emu/dp_add.c @@ -5,8 +5,6 @@ * MIPS floating point support * Copyright (C) 1994-2000 Algorithmics Ltd. * - * ######################################################################## - * * This program is free software; you can distribute it and/or modify it * under the terms of the GNU General Public License (Version 2) as * published by the Free Software Foundation. @@ -18,24 +16,22 @@ * * You should have received a copy of the GNU General Public License along * with this program; if not, write to the Free Software Foundation, Inc., - * 59 Temple Place - Suite 330, Boston MA 02111-1307, USA. - * - * ######################################################################## - * + * 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. */ - #include "ieee754dp.h" -ieee754dp ieee754dp_add(ieee754dp x, ieee754dp y) +union ieee754dp ieee754dp_add(union ieee754dp x, union ieee754dp y) { + int s; + COMPXDP; COMPYDP; EXPLODEXDP; EXPLODEYDP; - CLEARCX; + ieee754_clearcx(); FLUSHXDP; FLUSHYDP; @@ -52,8 +48,8 @@ ieee754dp ieee754dp_add(ieee754dp x, ieee754dp y) case CLPAIR(IEEE754_CLASS_SNAN, IEEE754_CLASS_NORM): case CLPAIR(IEEE754_CLASS_SNAN, IEEE754_CLASS_DNORM): case CLPAIR(IEEE754_CLASS_SNAN, IEEE754_CLASS_INF): - SETCX(IEEE754_INVALID_OPERATION); - return ieee754dp_nanxcpt(ieee754dp_indef(), "add", x, y); + ieee754_setcx(IEEE754_INVALID_OPERATION); + return ieee754dp_nanxcpt(ieee754dp_indef()); case CLPAIR(IEEE754_CLASS_ZERO, IEEE754_CLASS_QNAN): case CLPAIR(IEEE754_CLASS_NORM, IEEE754_CLASS_QNAN): @@ -69,14 +65,14 @@ ieee754dp ieee754dp_add(ieee754dp x, ieee754dp y) return x; - /* Infinity handling - */ - + /* + * Infinity handling + */ case CLPAIR(IEEE754_CLASS_INF, IEEE754_CLASS_INF): if (xs == ys) return x; - SETCX(IEEE754_INVALID_OPERATION); - return ieee754dp_xcpt(ieee754dp_indef(), "add", x, y); + ieee754_setcx(IEEE754_INVALID_OPERATION); + return ieee754dp_indef(); case CLPAIR(IEEE754_CLASS_NORM, IEEE754_CLASS_INF): case CLPAIR(IEEE754_CLASS_ZERO, IEEE754_CLASS_INF): @@ -88,15 +84,14 @@ ieee754dp ieee754dp_add(ieee754dp x, ieee754dp y) case CLPAIR(IEEE754_CLASS_INF, IEEE754_CLASS_DNORM): return x; - /* Zero handling - */ - + /* + * Zero handling + */ case CLPAIR(IEEE754_CLASS_ZERO, IEEE754_CLASS_ZERO): if (xs == ys) return x; else - return ieee754dp_zero(ieee754_csr.rm == - IEEE754_RD); + return ieee754dp_zero(ieee754_csr.rm == FPU_CSR_RD); case CLPAIR(IEEE754_CLASS_NORM, IEEE754_CLASS_ZERO): case CLPAIR(IEEE754_CLASS_DNORM, IEEE754_CLASS_ZERO): @@ -125,20 +120,24 @@ ieee754dp ieee754dp_add(ieee754dp x, ieee754dp y) assert(xm & DP_HIDDEN_BIT); assert(ym & DP_HIDDEN_BIT); - /* provide guard,round and stick bit space */ + /* + * Provide guard,round and stick bit space. + */ xm <<= 3; ym <<= 3; if (xe > ye) { - /* have to shift y fraction right to align + /* + * Have to shift y fraction right to align. */ - int s = xe - ye; + s = xe - ye; ym = XDPSRS(ym, s); ye += s; } else if (ye > xe) { - /* have to shift x fraction right to align + /* + * Have to shift x fraction right to align. */ - int s = ye - xe; + s = ye - xe; xm = XDPSRS(xm, s); xe += s; } @@ -146,14 +145,15 @@ ieee754dp ieee754dp_add(ieee754dp x, ieee754dp y) assert(xe <= DP_EMAX); if (xs == ys) { - /* generate 28 bit result of adding two 27 bit numbers - * leaving result in xm,xs,xe + /* + * Generate 28 bit result of adding two 27 bit numbers + * leaving result in xm, xs and xe. */ xm = xm + ym; xe = xe; xs = xs; - if (xm >> (DP_MBITS + 1 + 3)) { /* carry out */ + if (xm >> (DP_FBITS + 1 + 3)) { /* carry out */ xm = XDPSRS1(xm); xe++; } @@ -168,15 +168,16 @@ ieee754dp ieee754dp_add(ieee754dp x, ieee754dp y) xs = ys; } if (xm == 0) - return ieee754dp_zero(ieee754_csr.rm == - IEEE754_RD); + return ieee754dp_zero(ieee754_csr.rm == FPU_CSR_RD); - /* normalize to rounding precision */ - while ((xm >> (DP_MBITS + 3)) == 0) { + /* + * Normalize to rounding precision. + */ + while ((xm >> (DP_FBITS + 3)) == 0) { xm <<= 1; xe--; } - } - DPNORMRET2(xs, xe, xm, "add", x, y); + + return ieee754dp_format(xs, xe, xm); } diff --git a/arch/mips/math-emu/dp_cmp.c b/arch/mips/math-emu/dp_cmp.c index 0f32486b0ed9..30f95f6e9ac4 100644 --- a/arch/mips/math-emu/dp_cmp.c +++ b/arch/mips/math-emu/dp_cmp.c @@ -5,8 +5,6 @@ * MIPS floating point support * Copyright (C) 1994-2000 Algorithmics Ltd. * - * ######################################################################## - * * This program is free software; you can distribute it and/or modify it * under the terms of the GNU General Public License (Version 2) as * published by the Free Software Foundation. @@ -18,16 +16,16 @@ * * You should have received a copy of the GNU General Public License along * with this program; if not, write to the Free Software Foundation, Inc., - * 59 Temple Place - Suite 330, Boston MA 02111-1307, USA. - * - * ######################################################################## + * 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. */ - #include "ieee754dp.h" -int ieee754dp_cmp(ieee754dp x, ieee754dp y, int cmp, int sig) +int ieee754dp_cmp(union ieee754dp x, union ieee754dp y, int cmp, int sig) { + s64 vx; + s64 vy; + COMPXDP; COMPYDP; @@ -35,21 +33,21 @@ int ieee754dp_cmp(ieee754dp x, ieee754dp y, int cmp, int sig) EXPLODEYDP; FLUSHXDP; FLUSHYDP; - CLEARCX; /* Even clear inexact flag here */ + ieee754_clearcx(); /* Even clear inexact flag here */ if (ieee754dp_isnan(x) || ieee754dp_isnan(y)) { if (sig || xc == IEEE754_CLASS_SNAN || yc == IEEE754_CLASS_SNAN) - SETCX(IEEE754_INVALID_OPERATION); + ieee754_setcx(IEEE754_INVALID_OPERATION); if (cmp & IEEE754_CUN) return 1; if (cmp & (IEEE754_CLT | IEEE754_CGT)) { - if (sig && SETANDTESTCX(IEEE754_INVALID_OPERATION)) - return ieee754si_xcpt(0, "fcmpf", x); + if (sig && ieee754_setandtestcx(IEEE754_INVALID_OPERATION)) + return 0; } return 0; } else { - s64 vx = x.bits; - s64 vy = y.bits; + vx = x.bits; + vy = y.bits; if (vx < 0) vx = -vx ^ DP_SIGN_BIT; diff --git a/arch/mips/math-emu/dp_div.c b/arch/mips/math-emu/dp_div.c index a1bce1b7c09c..bef0e55e5938 100644 --- a/arch/mips/math-emu/dp_div.c +++ b/arch/mips/math-emu/dp_div.c @@ -5,8 +5,6 @@ * MIPS floating point support * Copyright (C) 1994-2000 Algorithmics Ltd. * - * ######################################################################## - * * This program is free software; you can distribute it and/or modify it * under the terms of the GNU General Public License (Version 2) as * published by the Free Software Foundation. @@ -18,23 +16,24 @@ * * You should have received a copy of the GNU General Public License along * with this program; if not, write to the Free Software Foundation, Inc., - * 59 Temple Place - Suite 330, Boston MA 02111-1307, USA. - * - * ######################################################################## + * 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. */ - #include "ieee754dp.h" -ieee754dp ieee754dp_div(ieee754dp x, ieee754dp y) +union ieee754dp ieee754dp_div(union ieee754dp x, union ieee754dp y) { + u64 rm; + int re; + u64 bm; + COMPXDP; COMPYDP; EXPLODEXDP; EXPLODEYDP; - CLEARCX; + ieee754_clearcx(); FLUSHXDP; FLUSHYDP; @@ -51,8 +50,8 @@ ieee754dp ieee754dp_div(ieee754dp x, ieee754dp y) case CLPAIR(IEEE754_CLASS_SNAN, IEEE754_CLASS_NORM): case CLPAIR(IEEE754_CLASS_SNAN, IEEE754_CLASS_DNORM): case CLPAIR(IEEE754_CLASS_SNAN, IEEE754_CLASS_INF): - SETCX(IEEE754_INVALID_OPERATION); - return ieee754dp_nanxcpt(ieee754dp_indef(), "div", x, y); + ieee754_setcx(IEEE754_INVALID_OPERATION); + return ieee754dp_nanxcpt(ieee754dp_indef()); case CLPAIR(IEEE754_CLASS_ZERO, IEEE754_CLASS_QNAN): case CLPAIR(IEEE754_CLASS_NORM, IEEE754_CLASS_QNAN): @@ -68,12 +67,12 @@ ieee754dp ieee754dp_div(ieee754dp x, ieee754dp y) return x; - /* Infinity handling - */ - + /* + * Infinity handling + */ case CLPAIR(IEEE754_CLASS_INF, IEEE754_CLASS_INF): - SETCX(IEEE754_INVALID_OPERATION); - return ieee754dp_xcpt(ieee754dp_indef(), "div", x, y); + ieee754_setcx(IEEE754_INVALID_OPERATION); + return ieee754dp_indef(); case CLPAIR(IEEE754_CLASS_NORM, IEEE754_CLASS_INF): case CLPAIR(IEEE754_CLASS_ZERO, IEEE754_CLASS_INF): @@ -85,17 +84,17 @@ ieee754dp ieee754dp_div(ieee754dp x, ieee754dp y) case CLPAIR(IEEE754_CLASS_INF, IEEE754_CLASS_DNORM): return ieee754dp_inf(xs ^ ys); - /* Zero handling - */ - + /* + * Zero handling + */ case CLPAIR(IEEE754_CLASS_ZERO, IEEE754_CLASS_ZERO): - SETCX(IEEE754_INVALID_OPERATION); - return ieee754dp_xcpt(ieee754dp_indef(), "div", x, y); + ieee754_setcx(IEEE754_INVALID_OPERATION); + return ieee754dp_indef(); case CLPAIR(IEEE754_CLASS_NORM, IEEE754_CLASS_ZERO): case CLPAIR(IEEE754_CLASS_DNORM, IEEE754_CLASS_ZERO): - SETCX(IEEE754_ZERO_DIVIDE); - return ieee754dp_xcpt(ieee754dp_inf(xs ^ ys), "div", x, y); + ieee754_setcx(IEEE754_ZERO_DIVIDE); + return ieee754dp_inf(xs ^ ys); case CLPAIR(IEEE754_CLASS_ZERO, IEEE754_CLASS_NORM): case CLPAIR(IEEE754_CLASS_ZERO, IEEE754_CLASS_DNORM): @@ -122,35 +121,34 @@ ieee754dp ieee754dp_div(ieee754dp x, ieee754dp y) xm <<= 3; ym <<= 3; - { - /* now the dirty work */ - - u64 rm = 0; - int re = xe - ye; - u64 bm; - - for (bm = DP_MBIT(DP_MBITS + 2); bm; bm >>= 1) { - if (xm >= ym) { - xm -= ym; - rm |= bm; - if (xm == 0) - break; - } - xm <<= 1; - } - rm <<= 1; - if (xm) - rm |= 1; /* have remainder, set sticky */ + /* now the dirty work */ - assert(rm); + rm = 0; + re = xe - ye; - /* normalise rm to rounding precision ? - */ - while ((rm >> (DP_MBITS + 3)) == 0) { - rm <<= 1; - re--; + for (bm = DP_MBIT(DP_FBITS + 2); bm; bm >>= 1) { + if (xm >= ym) { + xm -= ym; + rm |= bm; + if (xm == 0) + break; } + xm <<= 1; + } + + rm <<= 1; + if (xm) + rm |= 1; /* have remainder, set sticky */ - DPNORMRET2(xs == ys ? 0 : 1, re, rm, "div", x, y); + assert(rm); + + /* + * Normalise rm to rounding precision ? + */ + while ((rm >> (DP_FBITS + 3)) == 0) { + rm <<= 1; + re--; } + + return ieee754dp_format(xs == ys ? 0 : 1, re, rm); } diff --git a/arch/mips/math-emu/dp_fint.c b/arch/mips/math-emu/dp_fint.c index 88571288c9e0..10258f0afd69 100644 --- a/arch/mips/math-emu/dp_fint.c +++ b/arch/mips/math-emu/dp_fint.c @@ -5,8 +5,6 @@ * MIPS floating point support * Copyright (C) 1994-2000 Algorithmics Ltd. * - * ######################################################################## - * * This program is free software; you can distribute it and/or modify it * under the terms of the GNU General Public License (Version 2) as * published by the Free Software Foundation. @@ -18,21 +16,18 @@ * * You should have received a copy of the GNU General Public License along * with this program; if not, write to the Free Software Foundation, Inc., - * 59 Temple Place - Suite 330, Boston MA 02111-1307, USA. - * - * ######################################################################## + * 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. */ - #include "ieee754dp.h" -ieee754dp ieee754dp_fint(int x) +union ieee754dp ieee754dp_fint(int x) { u64 xm; int xe; int xs; - CLEARCX; + ieee754_clearcx(); if (x == 0) return ieee754dp_zero(0); @@ -51,29 +46,11 @@ ieee754dp ieee754dp_fint(int x) xm = x; } -#if 1 /* normalize - result can never be inexact or overflow */ - xe = DP_MBITS; - while ((xm >> DP_MBITS) == 0) { + xe = DP_FBITS; + while ((xm >> DP_FBITS) == 0) { xm <<= 1; xe--; } return builddp(xs, xe + DP_EBIAS, xm & ~DP_HIDDEN_BIT); -#else - /* normalize */ - xe = DP_MBITS + 3; - while ((xm >> (DP_MBITS + 3)) == 0) { - xm <<= 1; - xe--; - } - DPNORMRET1(xs, xe, xm, "fint", x); -#endif -} - -ieee754dp ieee754dp_funs(unsigned int u) -{ - if ((int) u < 0) - return ieee754dp_add(ieee754dp_1e31(), - ieee754dp_fint(u & ~(1 << 31))); - return ieee754dp_fint(u); } diff --git a/arch/mips/math-emu/dp_flong.c b/arch/mips/math-emu/dp_flong.c index 14fc01ec742d..a267c2e39d78 100644 --- a/arch/mips/math-emu/dp_flong.c +++ b/arch/mips/math-emu/dp_flong.c @@ -5,8 +5,6 @@ * MIPS floating point support * Copyright (C) 1994-2000 Algorithmics Ltd. * - * ######################################################################## - * * This program is free software; you can distribute it and/or modify it * under the terms of the GNU General Public License (Version 2) as * published by the Free Software Foundation. @@ -18,21 +16,18 @@ * * You should have received a copy of the GNU General Public License along * with this program; if not, write to the Free Software Foundation, Inc., - * 59 Temple Place - Suite 330, Boston MA 02111-1307, USA. - * - * ######################################################################## + * 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. */ - #include "ieee754dp.h" -ieee754dp ieee754dp_flong(s64 x) +union ieee754dp ieee754dp_flong(s64 x) { u64 xm; int xe; int xs; - CLEARCX; + ieee754_clearcx(); if (x == 0) return ieee754dp_zero(0); @@ -52,26 +47,19 @@ ieee754dp ieee754dp_flong(s64 x) } /* normalize */ - xe = DP_MBITS + 3; - if (xm >> (DP_MBITS + 1 + 3)) { + xe = DP_FBITS + 3; + if (xm >> (DP_FBITS + 1 + 3)) { /* shunt out overflow bits */ - while (xm >> (DP_MBITS + 1 + 3)) { + while (xm >> (DP_FBITS + 1 + 3)) { XDPSRSX1(); } } else { /* normalize in grs extended double precision */ - while ((xm >> (DP_MBITS + 3)) == 0) { + while ((xm >> (DP_FBITS + 3)) == 0) { xm <<= 1; xe--; } } - DPNORMRET1(xs, xe, xm, "dp_flong", x); -} -ieee754dp ieee754dp_fulong(u64 u) -{ - if ((s64) u < 0) - return ieee754dp_add(ieee754dp_1e63(), - ieee754dp_flong(u & ~(1ULL << 63))); - return ieee754dp_flong(u); + return ieee754dp_format(xs, xe, xm); } diff --git a/arch/mips/math-emu/dp_frexp.c b/arch/mips/math-emu/dp_frexp.c deleted file mode 100644 index cb15a5eaecbb..000000000000 --- a/arch/mips/math-emu/dp_frexp.c +++ /dev/null @@ -1,52 +0,0 @@ -/* IEEE754 floating point arithmetic - * double precision: common utilities - */ -/* - * MIPS floating point support - * Copyright (C) 1994-2000 Algorithmics Ltd. - * - * ######################################################################## - * - * This program is free software; you can distribute it and/or modify it - * under the terms of the GNU General Public License (Version 2) as - * published by the Free Software Foundation. - * - * This program is distributed in the hope it will be useful, but WITHOUT - * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or - * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License - * for more details. - * - * You should have received a copy of the GNU General Public License along - * with this program; if not, write to the Free Software Foundation, Inc., - * 59 Temple Place - Suite 330, Boston MA 02111-1307, USA. - * - * ######################################################################## - */ - - -#include "ieee754dp.h" - -/* close to ieeep754dp_logb -*/ -ieee754dp ieee754dp_frexp(ieee754dp x, int *eptr) -{ - COMPXDP; - CLEARCX; - EXPLODEXDP; - - switch (xc) { - case IEEE754_CLASS_SNAN: - case IEEE754_CLASS_QNAN: - case IEEE754_CLASS_INF: - case IEEE754_CLASS_ZERO: - *eptr = 0; - return x; - case IEEE754_CLASS_DNORM: - DPDNORMX; - break; - case IEEE754_CLASS_NORM: - break; - } - *eptr = xe + 1; - return builddp(xs, -1 + DP_EBIAS, xm & ~DP_HIDDEN_BIT); -} diff --git a/arch/mips/math-emu/dp_fsp.c b/arch/mips/math-emu/dp_fsp.c index daed6834dc15..ffb69c5830b0 100644 --- a/arch/mips/math-emu/dp_fsp.c +++ b/arch/mips/math-emu/dp_fsp.c @@ -5,8 +5,6 @@ * MIPS floating point support * Copyright (C) 1994-2000 Algorithmics Ltd. * - * ######################################################################## - * * This program is free software; you can distribute it and/or modify it * under the terms of the GNU General Public License (Version 2) as * published by the Free Software Foundation. @@ -18,56 +16,58 @@ * * You should have received a copy of the GNU General Public License along * with this program; if not, write to the Free Software Foundation, Inc., - * 59 Temple Place - Suite 330, Boston MA 02111-1307, USA. - * - * ######################################################################## + * 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. */ - +#include "ieee754sp.h" #include "ieee754dp.h" -ieee754dp ieee754dp_fsp(ieee754sp x) +union ieee754dp ieee754dp_fsp(union ieee754sp x) { COMPXSP; EXPLODEXSP; - CLEARCX; + ieee754_clearcx(); FLUSHXSP; switch (xc) { case IEEE754_CLASS_SNAN: - SETCX(IEEE754_INVALID_OPERATION); - return ieee754dp_nanxcpt(ieee754dp_indef(), "fsp"); + ieee754_setcx(IEEE754_INVALID_OPERATION); + return ieee754dp_nanxcpt(ieee754dp_indef()); + case IEEE754_CLASS_QNAN: return ieee754dp_nanxcpt(builddp(xs, DP_EMAX + 1 + DP_EBIAS, ((u64) xm - << (DP_MBITS - - SP_MBITS))), "fsp", - x); + << (DP_FBITS - + SP_FBITS)))); case IEEE754_CLASS_INF: return ieee754dp_inf(xs); + case IEEE754_CLASS_ZERO: return ieee754dp_zero(xs); + case IEEE754_CLASS_DNORM: /* normalize */ - while ((xm >> SP_MBITS) == 0) { + while ((xm >> SP_FBITS) == 0) { xm <<= 1; xe--; } break; + case IEEE754_CLASS_NORM: break; } - /* CAN'T possibly overflow,underflow, or need rounding + /* + * Can't possibly overflow,underflow, or need rounding */ /* drop the hidden bit */ xm &= ~SP_HIDDEN_BIT; return builddp(xs, xe + DP_EBIAS, - (u64) xm << (DP_MBITS - SP_MBITS)); + (u64) xm << (DP_FBITS - SP_FBITS)); } diff --git a/arch/mips/math-emu/dp_logb.c b/arch/mips/math-emu/dp_logb.c deleted file mode 100644 index 151127e59f5c..000000000000 --- a/arch/mips/math-emu/dp_logb.c +++ /dev/null @@ -1,53 +0,0 @@ -/* IEEE754 floating point arithmetic - * double precision: common utilities - */ -/* - * MIPS floating point support - * Copyright (C) 1994-2000 Algorithmics Ltd. - * - * ######################################################################## - * - * This program is free software; you can distribute it and/or modify it - * under the terms of the GNU General Public License (Version 2) as - * published by the Free Software Foundation. - * - * This program is distributed in the hope it will be useful, but WITHOUT - * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or - * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License - * for more details. - * - * You should have received a copy of the GNU General Public License along - * with this program; if not, write to the Free Software Foundation, Inc., - * 59 Temple Place - Suite 330, Boston MA 02111-1307, USA. - * - * ######################################################################## - */ - - -#include "ieee754dp.h" - -ieee754dp ieee754dp_logb(ieee754dp x) -{ - COMPXDP; - - CLEARCX; - - EXPLODEXDP; - - switch (xc) { - case IEEE754_CLASS_SNAN: - return ieee754dp_nanxcpt(x, "logb", x); - case IEEE754_CLASS_QNAN: - return x; - case IEEE754_CLASS_INF: - return ieee754dp_inf(0); - case IEEE754_CLASS_ZERO: - return ieee754dp_inf(1); - case IEEE754_CLASS_DNORM: - DPDNORMX; - break; - case IEEE754_CLASS_NORM: - break; - } - return ieee754dp_fint(xe); -} diff --git a/arch/mips/math-emu/dp_modf.c b/arch/mips/math-emu/dp_modf.c deleted file mode 100644 index b01f9cf6d402..000000000000 --- a/arch/mips/math-emu/dp_modf.c +++ /dev/null @@ -1,79 +0,0 @@ -/* IEEE754 floating point arithmetic - * double precision: common utilities - */ -/* - * MIPS floating point support - * Copyright (C) 1994-2000 Algorithmics Ltd. - * - * ######################################################################## - * - * This program is free software; you can distribute it and/or modify it - * under the terms of the GNU General Public License (Version 2) as - * published by the Free Software Foundation. - * - * This program is distributed in the hope it will be useful, but WITHOUT - * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or - * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License - * for more details. - * - * You should have received a copy of the GNU General Public License along - * with this program; if not, write to the Free Software Foundation, Inc., - * 59 Temple Place - Suite 330, Boston MA 02111-1307, USA. - * - * ######################################################################## - */ - - -#include "ieee754dp.h" - -/* modf function is always exact for a finite number -*/ -ieee754dp ieee754dp_modf(ieee754dp x, ieee754dp *ip) -{ - COMPXDP; - - CLEARCX; - - EXPLODEXDP; - - switch (xc) { - case IEEE754_CLASS_SNAN: - case IEEE754_CLASS_QNAN: - case IEEE754_CLASS_INF: - case IEEE754_CLASS_ZERO: - *ip = x; - return x; - case IEEE754_CLASS_DNORM: - /* far to small */ - *ip = ieee754dp_zero(xs); - return x; - case IEEE754_CLASS_NORM: - break; - } - if (xe < 0) { - *ip = ieee754dp_zero(xs); - return x; - } - if (xe >= DP_MBITS) { - *ip = x; - return ieee754dp_zero(xs); - } - /* generate ipart mantissa by clearing bottom bits - */ - *ip = builddp(xs, xe + DP_EBIAS, - ((xm >> (DP_MBITS - xe)) << (DP_MBITS - xe)) & - ~DP_HIDDEN_BIT); - - /* generate fpart mantissa by clearing top bits - * and normalizing (must be able to normalize) - */ - xm = (xm << (64 - (DP_MBITS - xe))) >> (64 - (DP_MBITS - xe)); - if (xm == 0) - return ieee754dp_zero(xs); - - while ((xm >> DP_MBITS) == 0) { - xm <<= 1; - xe--; - } - return builddp(xs, xe + DP_EBIAS, xm & ~DP_HIDDEN_BIT); -} diff --git a/arch/mips/math-emu/dp_mul.c b/arch/mips/math-emu/dp_mul.c index 09175f461920..d3acdedb5b9d 100644 --- a/arch/mips/math-emu/dp_mul.c +++ b/arch/mips/math-emu/dp_mul.c @@ -5,8 +5,6 @@ * MIPS floating point support * Copyright (C) 1994-2000 Algorithmics Ltd. * - * ######################################################################## - * * This program is free software; you can distribute it and/or modify it * under the terms of the GNU General Public License (Version 2) as * published by the Free Software Foundation. @@ -18,23 +16,32 @@ * * You should have received a copy of the GNU General Public License along * with this program; if not, write to the Free Software Foundation, Inc., - * 59 Temple Place - Suite 330, Boston MA 02111-1307, USA. - * - * ######################################################################## + * 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. */ - #include "ieee754dp.h" -ieee754dp ieee754dp_mul(ieee754dp x, ieee754dp y) +union ieee754dp ieee754dp_mul(union ieee754dp x, union ieee754dp y) { + int re; + int rs; + u64 rm; + unsigned lxm; + unsigned hxm; + unsigned lym; + unsigned hym; + u64 lrm; + u64 hrm; + u64 t; + u64 at; + COMPXDP; COMPYDP; EXPLODEXDP; EXPLODEYDP; - CLEARCX; + ieee754_clearcx(); FLUSHXDP; FLUSHYDP; @@ -51,8 +58,8 @@ ieee754dp ieee754dp_mul(ieee754dp x, ieee754dp y) case CLPAIR(IEEE754_CLASS_SNAN, IEEE754_CLASS_NORM): case CLPAIR(IEEE754_CLASS_SNAN, IEEE754_CLASS_DNORM): case CLPAIR(IEEE754_CLASS_SNAN, IEEE754_CLASS_INF): - SETCX(IEEE754_INVALID_OPERATION); - return ieee754dp_nanxcpt(ieee754dp_indef(), "mul", x, y); + ieee754_setcx(IEEE754_INVALID_OPERATION); + return ieee754dp_nanxcpt(ieee754dp_indef()); case CLPAIR(IEEE754_CLASS_ZERO, IEEE754_CLASS_QNAN): case CLPAIR(IEEE754_CLASS_NORM, IEEE754_CLASS_QNAN): @@ -68,12 +75,13 @@ ieee754dp ieee754dp_mul(ieee754dp x, ieee754dp y) return x; - /* Infinity handling */ - + /* + * Infinity handling + */ case CLPAIR(IEEE754_CLASS_INF, IEEE754_CLASS_ZERO): case CLPAIR(IEEE754_CLASS_ZERO, IEEE754_CLASS_INF): - SETCX(IEEE754_INVALID_OPERATION); - return ieee754dp_xcpt(ieee754dp_indef(), "mul", x, y); + ieee754_setcx(IEEE754_INVALID_OPERATION); + return ieee754dp_indef(); case CLPAIR(IEEE754_CLASS_NORM, IEEE754_CLASS_INF): case CLPAIR(IEEE754_CLASS_DNORM, IEEE754_CLASS_INF): @@ -107,70 +115,59 @@ ieee754dp ieee754dp_mul(ieee754dp x, ieee754dp y) /* rm = xm * ym, re = xe+ye basically */ assert(xm & DP_HIDDEN_BIT); assert(ym & DP_HIDDEN_BIT); - { - int re = xe + ye; - int rs = xs ^ ys; - u64 rm; - /* shunt to top of word */ - xm <<= 64 - (DP_MBITS + 1); - ym <<= 64 - (DP_MBITS + 1); + re = xe + ye; + rs = xs ^ ys; + + /* shunt to top of word */ + xm <<= 64 - (DP_FBITS + 1); + ym <<= 64 - (DP_FBITS + 1); - /* multiply 32bits xm,ym to give high 32bits rm with stickness - */ + /* + * Multiply 32 bits xm, ym to give high 32 bits rm with stickness. + */ - /* 32 * 32 => 64 */ + /* 32 * 32 => 64 */ #define DPXMULT(x, y) ((u64)(x) * (u64)y) - { - unsigned lxm = xm; - unsigned hxm = xm >> 32; - unsigned lym = ym; - unsigned hym = ym >> 32; - u64 lrm; - u64 hrm; - - lrm = DPXMULT(lxm, lym); - hrm = DPXMULT(hxm, hym); - - { - u64 t = DPXMULT(lxm, hym); - { - u64 at = - lrm + (t << 32); - hrm += at < lrm; - lrm = at; - } - hrm = hrm + (t >> 32); - } - - { - u64 t = DPXMULT(hxm, lym); - { - u64 at = - lrm + (t << 32); - hrm += at < lrm; - lrm = at; - } - hrm = hrm + (t >> 32); - } - rm = hrm | (lrm != 0); - } - - /* - * sticky shift down to normal rounding precision - */ - if ((s64) rm < 0) { - rm = - (rm >> (64 - (DP_MBITS + 1 + 3))) | - ((rm << (DP_MBITS + 1 + 3)) != 0); + lxm = xm; + hxm = xm >> 32; + lym = ym; + hym = ym >> 32; + + lrm = DPXMULT(lxm, lym); + hrm = DPXMULT(hxm, hym); + + t = DPXMULT(lxm, hym); + + at = lrm + (t << 32); + hrm += at < lrm; + lrm = at; + + hrm = hrm + (t >> 32); + + t = DPXMULT(hxm, lym); + + at = lrm + (t << 32); + hrm += at < lrm; + lrm = at; + + hrm = hrm + (t >> 32); + + rm = hrm | (lrm != 0); + + /* + * Sticky shift down to normal rounding precision. + */ + if ((s64) rm < 0) { + rm = (rm >> (64 - (DP_FBITS + 1 + 3))) | + ((rm << (DP_FBITS + 1 + 3)) != 0); re++; - } else { - rm = - (rm >> (64 - (DP_MBITS + 1 + 3 + 1))) | - ((rm << (DP_MBITS + 1 + 3 + 1)) != 0); - } - assert(rm & (DP_HIDDEN_BIT << 3)); - DPNORMRET2(rs, re, rm, "mul", x, y); + } else { + rm = (rm >> (64 - (DP_FBITS + 1 + 3 + 1))) | + ((rm << (DP_FBITS + 1 + 3 + 1)) != 0); } + assert(rm & (DP_HIDDEN_BIT << 3)); + + return ieee754dp_format(rs, re, rm); } diff --git a/arch/mips/math-emu/dp_scalb.c b/arch/mips/math-emu/dp_scalb.c deleted file mode 100644 index 6f5df438dda8..000000000000 --- a/arch/mips/math-emu/dp_scalb.c +++ /dev/null @@ -1,57 +0,0 @@ -/* IEEE754 floating point arithmetic - * double precision: common utilities - */ -/* - * MIPS floating point support - * Copyright (C) 1994-2000 Algorithmics Ltd. - * - * ######################################################################## - * - * This program is free software; you can distribute it and/or modify it - * under the terms of the GNU General Public License (Version 2) as - * published by the Free Software Foundation. - * - * This program is distributed in the hope it will be useful, but WITHOUT - * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or - * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License - * for more details. - * - * You should have received a copy of the GNU General Public License along - * with this program; if not, write to the Free Software Foundation, Inc., - * 59 Temple Place - Suite 330, Boston MA 02111-1307, USA. - * - * ######################################################################## - */ - - -#include "ieee754dp.h" - -ieee754dp ieee754dp_scalb(ieee754dp x, int n) -{ - COMPXDP; - - CLEARCX; - - EXPLODEXDP; - - switch (xc) { - case IEEE754_CLASS_SNAN: - return ieee754dp_nanxcpt(x, "scalb", x, n); - case IEEE754_CLASS_QNAN: - case IEEE754_CLASS_INF: - case IEEE754_CLASS_ZERO: - return x; - case IEEE754_CLASS_DNORM: - DPDNORMX; - break; - case IEEE754_CLASS_NORM: - break; - } - DPNORMRET2(xs, xe + n, xm << 3, "scalb", x, n); -} - - -ieee754dp ieee754dp_ldexp(ieee754dp x, int n) -{ - return ieee754dp_scalb(x, n); -} diff --git a/arch/mips/math-emu/dp_simple.c b/arch/mips/math-emu/dp_simple.c index 79ce2673a714..bccbe90efceb 100644 --- a/arch/mips/math-emu/dp_simple.c +++ b/arch/mips/math-emu/dp_simple.c @@ -5,8 +5,6 @@ * MIPS floating point support * Copyright (C) 1994-2000 Algorithmics Ltd. * - * ######################################################################## - * * This program is free software; you can distribute it and/or modify it * under the terms of the GNU General Public License (Version 2) as * published by the Free Software Foundation. @@ -18,33 +16,17 @@ * * You should have received a copy of the GNU General Public License along * with this program; if not, write to the Free Software Foundation, Inc., - * 59 Temple Place - Suite 330, Boston MA 02111-1307, USA. - * - * ######################################################################## + * 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. */ - #include "ieee754dp.h" -int ieee754dp_finite(ieee754dp x) -{ - return DPBEXP(x) != DP_EMAX + 1 + DP_EBIAS; -} - -ieee754dp ieee754dp_copysign(ieee754dp x, ieee754dp y) -{ - CLEARCX; - DPSIGN(x) = DPSIGN(y); - return x; -} - - -ieee754dp ieee754dp_neg(ieee754dp x) +union ieee754dp ieee754dp_neg(union ieee754dp x) { COMPXDP; EXPLODEXDP; - CLEARCX; + ieee754_clearcx(); FLUSHXDP; /* @@ -55,30 +37,29 @@ ieee754dp ieee754dp_neg(ieee754dp x) DPSIGN(x) ^= 1; if (xc == IEEE754_CLASS_SNAN) { - ieee754dp y = ieee754dp_indef(); - SETCX(IEEE754_INVALID_OPERATION); + union ieee754dp y = ieee754dp_indef(); + ieee754_setcx(IEEE754_INVALID_OPERATION); DPSIGN(y) = DPSIGN(x); - return ieee754dp_nanxcpt(y, "neg"); + return ieee754dp_nanxcpt(y); } return x; } - -ieee754dp ieee754dp_abs(ieee754dp x) +union ieee754dp ieee754dp_abs(union ieee754dp x) { COMPXDP; EXPLODEXDP; - CLEARCX; + ieee754_clearcx(); FLUSHXDP; /* Clear sign ALWAYS, irrespective of NaN */ DPSIGN(x) = 0; if (xc == IEEE754_CLASS_SNAN) { - SETCX(IEEE754_INVALID_OPERATION); - return ieee754dp_nanxcpt(ieee754dp_indef(), "abs"); + ieee754_setcx(IEEE754_INVALID_OPERATION); + return ieee754dp_nanxcpt(ieee754dp_indef()); } return x; diff --git a/arch/mips/math-emu/dp_sqrt.c b/arch/mips/math-emu/dp_sqrt.c index b874d60a942b..041bbb6124bb 100644 --- a/arch/mips/math-emu/dp_sqrt.c +++ b/arch/mips/math-emu/dp_sqrt.c @@ -5,8 +5,6 @@ * MIPS floating point support * Copyright (C) 1994-2000 Algorithmics Ltd. * - * ######################################################################## - * * This program is free software; you can distribute it and/or modify it * under the terms of the GNU General Public License (Version 2) as * published by the Free Software Foundation. @@ -18,12 +16,9 @@ * * You should have received a copy of the GNU General Public License along * with this program; if not, write to the Free Software Foundation, Inc., - * 59 Temple Place - Suite 330, Boston MA 02111-1307, USA. - * - * ######################################################################## + * 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. */ - #include "ieee754dp.h" static const unsigned table[] = { @@ -34,44 +29,49 @@ static const unsigned table[] = { 1742, 661, 130 }; -ieee754dp ieee754dp_sqrt(ieee754dp x) +union ieee754dp ieee754dp_sqrt(union ieee754dp x) { struct _ieee754_csr oldcsr; - ieee754dp y, z, t; + union ieee754dp y, z, t; unsigned scalx, yh; COMPXDP; EXPLODEXDP; - CLEARCX; + ieee754_clearcx(); FLUSHXDP; /* x == INF or NAN? */ switch (xc) { case IEEE754_CLASS_QNAN: /* sqrt(Nan) = Nan */ - return ieee754dp_nanxcpt(x, "sqrt"); + return ieee754dp_nanxcpt(x); + case IEEE754_CLASS_SNAN: - SETCX(IEEE754_INVALID_OPERATION); - return ieee754dp_nanxcpt(ieee754dp_indef(), "sqrt"); + ieee754_setcx(IEEE754_INVALID_OPERATION); + return ieee754dp_nanxcpt(ieee754dp_indef()); + case IEEE754_CLASS_ZERO: /* sqrt(0) = 0 */ return x; + case IEEE754_CLASS_INF: if (xs) { /* sqrt(-Inf) = Nan */ - SETCX(IEEE754_INVALID_OPERATION); - return ieee754dp_nanxcpt(ieee754dp_indef(), "sqrt"); + ieee754_setcx(IEEE754_INVALID_OPERATION); + return ieee754dp_nanxcpt(ieee754dp_indef()); } /* sqrt(+Inf) = Inf */ return x; + case IEEE754_CLASS_DNORM: DPDNORMX; /* fall through */ + case IEEE754_CLASS_NORM: if (xs) { /* sqrt(-x) = Nan */ - SETCX(IEEE754_INVALID_OPERATION); - return ieee754dp_nanxcpt(ieee754dp_indef(), "sqrt"); + ieee754_setcx(IEEE754_INVALID_OPERATION); + return ieee754dp_nanxcpt(ieee754dp_indef()); } break; } @@ -80,7 +80,7 @@ ieee754dp ieee754dp_sqrt(ieee754dp x) oldcsr = ieee754_csr; ieee754_csr.mx &= ~IEEE754_INEXACT; ieee754_csr.sx &= ~IEEE754_INEXACT; - ieee754_csr.rm = IEEE754_RN; + ieee754_csr.rm = FPU_CSR_RN; /* adjust exponent to prevent overflow */ scalx = 0; @@ -110,19 +110,19 @@ ieee754dp ieee754dp_sqrt(ieee754dp x) /* triple to almost 56 sig. bits: y ~= sqrt(x) to within 1 ulp */ /* t=y*y; z=t; pt[n0]+=0x00100000; t+=z; z=(x-z)*y; */ z = t = ieee754dp_mul(y, y); - t.parts.bexp += 0x001; + t.bexp += 0x001; t = ieee754dp_add(t, z); z = ieee754dp_mul(ieee754dp_sub(x, z), y); /* t=z/(t+x) ; pt[n0]+=0x00100000; y+=t; */ t = ieee754dp_div(z, ieee754dp_add(t, x)); - t.parts.bexp += 0x001; + t.bexp += 0x001; y = ieee754dp_add(y, t); /* twiddle last bit to force y correctly rounded */ /* set RZ, clear INEX flag */ - ieee754_csr.rm = IEEE754_RZ; + ieee754_csr.rm = FPU_CSR_RZ; ieee754_csr.sx &= ~IEEE754_INEXACT; /* t=x/y; ...chopped quotient, possibly inexact */ @@ -139,10 +139,10 @@ ieee754dp ieee754dp_sqrt(ieee754dp x) oldcsr.sx |= IEEE754_INEXACT; switch (oldcsr.rm) { - case IEEE754_RP: + case FPU_CSR_RU: y.bits += 1; /* drop through */ - case IEEE754_RN: + case FPU_CSR_RN: t.bits += 1; break; } @@ -155,7 +155,7 @@ ieee754dp ieee754dp_sqrt(ieee754dp x) } /* py[n0]=py[n0]+scalx; ...scale back y */ - y.parts.bexp += scalx; + y.bexp += scalx; /* restore rounding mode, possibly set inexact */ ieee754_csr = oldcsr; diff --git a/arch/mips/math-emu/dp_sub.c b/arch/mips/math-emu/dp_sub.c index 91e0a4b5cbc7..7a174029043a 100644 --- a/arch/mips/math-emu/dp_sub.c +++ b/arch/mips/math-emu/dp_sub.c @@ -5,8 +5,6 @@ * MIPS floating point support * Copyright (C) 1994-2000 Algorithmics Ltd. * - * ######################################################################## - * * This program is free software; you can distribute it and/or modify it * under the terms of the GNU General Public License (Version 2) as * published by the Free Software Foundation. @@ -18,23 +16,22 @@ * * You should have received a copy of the GNU General Public License along * with this program; if not, write to the Free Software Foundation, Inc., - * 59 Temple Place - Suite 330, Boston MA 02111-1307, USA. - * - * ######################################################################## + * 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. */ - #include "ieee754dp.h" -ieee754dp ieee754dp_sub(ieee754dp x, ieee754dp y) +union ieee754dp ieee754dp_sub(union ieee754dp x, union ieee754dp y) { + int s; + COMPXDP; COMPYDP; EXPLODEXDP; EXPLODEYDP; - CLEARCX; + ieee754_clearcx(); FLUSHXDP; FLUSHYDP; @@ -51,8 +48,8 @@ ieee754dp ieee754dp_sub(ieee754dp x, ieee754dp y) case CLPAIR(IEEE754_CLASS_SNAN, IEEE754_CLASS_NORM): case CLPAIR(IEEE754_CLASS_SNAN, IEEE754_CLASS_DNORM): case CLPAIR(IEEE754_CLASS_SNAN, IEEE754_CLASS_INF): - SETCX(IEEE754_INVALID_OPERATION); - return ieee754dp_nanxcpt(ieee754dp_indef(), "sub", x, y); + ieee754_setcx(IEEE754_INVALID_OPERATION); + return ieee754dp_nanxcpt(ieee754dp_indef()); case CLPAIR(IEEE754_CLASS_ZERO, IEEE754_CLASS_QNAN): case CLPAIR(IEEE754_CLASS_NORM, IEEE754_CLASS_QNAN): @@ -68,14 +65,14 @@ ieee754dp ieee754dp_sub(ieee754dp x, ieee754dp y) return x; - /* Infinity handling - */ - + /* + * Infinity handling + */ case CLPAIR(IEEE754_CLASS_INF, IEEE754_CLASS_INF): if (xs != ys) return x; - SETCX(IEEE754_INVALID_OPERATION); - return ieee754dp_xcpt(ieee754dp_indef(), "sub", x, y); + ieee754_setcx(IEEE754_INVALID_OPERATION); + return ieee754dp_indef(); case CLPAIR(IEEE754_CLASS_ZERO, IEEE754_CLASS_INF): case CLPAIR(IEEE754_CLASS_DNORM, IEEE754_CLASS_INF): @@ -87,15 +84,14 @@ ieee754dp ieee754dp_sub(ieee754dp x, ieee754dp y) case CLPAIR(IEEE754_CLASS_INF, IEEE754_CLASS_DNORM): return x; - /* Zero handling - */ - + /* + * Zero handling + */ case CLPAIR(IEEE754_CLASS_ZERO, IEEE754_CLASS_ZERO): if (xs != ys) return x; else - return ieee754dp_zero(ieee754_csr.rm == - IEEE754_RD); + return ieee754dp_zero(ieee754_csr.rm == FPU_CSR_RD); case CLPAIR(IEEE754_CLASS_NORM, IEEE754_CLASS_ZERO): case CLPAIR(IEEE754_CLASS_DNORM, IEEE754_CLASS_ZERO): @@ -136,15 +132,17 @@ ieee754dp ieee754dp_sub(ieee754dp x, ieee754dp y) ym <<= 3; if (xe > ye) { - /* have to shift y fraction right to align + /* + * Have to shift y fraction right to align */ - int s = xe - ye; + s = xe - ye; ym = XDPSRS(ym, s); ye += s; } else if (ye > xe) { - /* have to shift x fraction right to align + /* + * Have to shift x fraction right to align */ - int s = ye - xe; + s = ye - xe; xm = XDPSRS(xm, s); xe += s; } @@ -158,7 +156,7 @@ ieee754dp ieee754dp_sub(ieee754dp x, ieee754dp y) xe = xe; xs = xs; - if (xm >> (DP_MBITS + 1 + 3)) { /* carry out */ + if (xm >> (DP_FBITS + 1 + 3)) { /* carry out */ xm = XDPSRS1(xm); /* shift preserving sticky */ xe++; } @@ -173,7 +171,7 @@ ieee754dp ieee754dp_sub(ieee754dp x, ieee754dp y) xs = ys; } if (xm == 0) { - if (ieee754_csr.rm == IEEE754_RD) + if (ieee754_csr.rm == FPU_CSR_RD) return ieee754dp_zero(1); /* round negative inf. => sign = -1 */ else return ieee754dp_zero(0); /* other round modes => sign = 1 */ @@ -181,10 +179,11 @@ ieee754dp ieee754dp_sub(ieee754dp x, ieee754dp y) /* normalize to rounding precision */ - while ((xm >> (DP_MBITS + 3)) == 0) { + while ((xm >> (DP_FBITS + 3)) == 0) { xm <<= 1; xe--; } } - DPNORMRET2(xs, xe, xm, "sub", x, y); + + return ieee754dp_format(xs, xe, xm); } diff --git a/arch/mips/math-emu/dp_tint.c b/arch/mips/math-emu/dp_tint.c index 0ebe8598b94a..6ffc336c530e 100644 --- a/arch/mips/math-emu/dp_tint.c +++ b/arch/mips/math-emu/dp_tint.c @@ -5,8 +5,6 @@ * MIPS floating point support * Copyright (C) 1994-2000 Algorithmics Ltd. * - * ######################################################################## - * * This program is free software; you can distribute it and/or modify it * under the terms of the GNU General Public License (Version 2) as * published by the Free Software Foundation. @@ -18,20 +16,21 @@ * * You should have received a copy of the GNU General Public License along * with this program; if not, write to the Free Software Foundation, Inc., - * 59 Temple Place - Suite 330, Boston MA 02111-1307, USA. - * - * ######################################################################## + * 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. */ - -#include <linux/kernel.h> #include "ieee754dp.h" -int ieee754dp_tint(ieee754dp x) +int ieee754dp_tint(union ieee754dp x) { + u64 residue; + int round; + int sticky; + int odd; + COMPXDP; - CLEARCX; + ieee754_clearcx(); EXPLODEXDP; FLUSHXDP; @@ -40,10 +39,12 @@ int ieee754dp_tint(ieee754dp x) case IEEE754_CLASS_SNAN: case IEEE754_CLASS_QNAN: case IEEE754_CLASS_INF: - SETCX(IEEE754_INVALID_OPERATION); - return ieee754si_xcpt(ieee754si_indef(), "dp_tint", x); + ieee754_setcx(IEEE754_INVALID_OPERATION); + return ieee754si_indef(); + case IEEE754_CLASS_ZERO: return 0; + case IEEE754_CLASS_DNORM: case IEEE754_CLASS_NORM: break; @@ -51,44 +52,39 @@ int ieee754dp_tint(ieee754dp x) if (xe > 31) { /* Set invalid. We will only use overflow for floating point overflow */ - SETCX(IEEE754_INVALID_OPERATION); - return ieee754si_xcpt(ieee754si_indef(), "dp_tint", x); + ieee754_setcx(IEEE754_INVALID_OPERATION); + return ieee754si_indef(); } /* oh gawd */ - if (xe > DP_MBITS) { - xm <<= xe - DP_MBITS; - } else if (xe < DP_MBITS) { - u64 residue; - int round; - int sticky; - int odd; - + if (xe > DP_FBITS) { + xm <<= xe - DP_FBITS; + } else if (xe < DP_FBITS) { if (xe < -1) { residue = xm; round = 0; sticky = residue != 0; xm = 0; } else { - residue = xm << (64 - DP_MBITS + xe); + residue = xm << (64 - DP_FBITS + xe); round = (residue >> 63) != 0; sticky = (residue << 1) != 0; - xm >>= DP_MBITS - xe; + xm >>= DP_FBITS - xe; } /* Note: At this point upper 32 bits of xm are guaranteed to be zero */ odd = (xm & 0x1) != 0x0; switch (ieee754_csr.rm) { - case IEEE754_RN: + case FPU_CSR_RN: if (round && (sticky || odd)) xm++; break; - case IEEE754_RZ: + case FPU_CSR_RZ: break; - case IEEE754_RU: /* toward +Infinity */ + case FPU_CSR_RU: /* toward +Infinity */ if ((round || sticky) && !xs) xm++; break; - case IEEE754_RD: /* toward -Infinity */ + case FPU_CSR_RD: /* toward -Infinity */ if ((round || sticky) && xs) xm++; break; @@ -96,27 +92,14 @@ int ieee754dp_tint(ieee754dp x) /* look for valid corner case 0x80000000 */ if ((xm >> 31) != 0 && (xs == 0 || xm != 0x80000000)) { /* This can happen after rounding */ - SETCX(IEEE754_INVALID_OPERATION); - return ieee754si_xcpt(ieee754si_indef(), "dp_tint", x); + ieee754_setcx(IEEE754_INVALID_OPERATION); + return ieee754si_indef(); } if (round || sticky) - SETCX(IEEE754_INEXACT); + ieee754_setcx(IEEE754_INEXACT); } if (xs) return -xm; else return xm; } - - -unsigned int ieee754dp_tuns(ieee754dp x) -{ - ieee754dp hb = ieee754dp_1e31(); - - /* what if x < 0 ?? */ - if (ieee754dp_lt(x, hb)) - return (unsigned) ieee754dp_tint(x); - - return (unsigned) ieee754dp_tint(ieee754dp_sub(x, hb)) | - ((unsigned) 1 << 31); -} diff --git a/arch/mips/math-emu/dp_tlong.c b/arch/mips/math-emu/dp_tlong.c index 133ce2ba0012..9cdc145b75e0 100644 --- a/arch/mips/math-emu/dp_tlong.c +++ b/arch/mips/math-emu/dp_tlong.c @@ -5,8 +5,6 @@ * MIPS floating point support * Copyright (C) 1994-2000 Algorithmics Ltd. * - * ######################################################################## - * * This program is free software; you can distribute it and/or modify it * under the terms of the GNU General Public License (Version 2) as * published by the Free Software Foundation. @@ -18,19 +16,21 @@ * * You should have received a copy of the GNU General Public License along * with this program; if not, write to the Free Software Foundation, Inc., - * 59 Temple Place - Suite 330, Boston MA 02111-1307, USA. - * - * ######################################################################## + * 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. */ - #include "ieee754dp.h" -s64 ieee754dp_tlong(ieee754dp x) +s64 ieee754dp_tlong(union ieee754dp x) { + u64 residue; + int round; + int sticky; + int odd; + COMPXDP; - CLEARCX; + ieee754_clearcx(); EXPLODEXDP; FLUSHXDP; @@ -39,10 +39,12 @@ s64 ieee754dp_tlong(ieee754dp x) case IEEE754_CLASS_SNAN: case IEEE754_CLASS_QNAN: case IEEE754_CLASS_INF: - SETCX(IEEE754_INVALID_OPERATION); - return ieee754di_xcpt(ieee754di_indef(), "dp_tlong", x); + ieee754_setcx(IEEE754_INVALID_OPERATION); + return ieee754di_indef(); + case IEEE754_CLASS_ZERO: return 0; + case IEEE754_CLASS_DNORM: case IEEE754_CLASS_NORM: break; @@ -53,18 +55,13 @@ s64 ieee754dp_tlong(ieee754dp x) return -0x8000000000000000LL; /* Set invalid. We will only use overflow for floating point overflow */ - SETCX(IEEE754_INVALID_OPERATION); - return ieee754di_xcpt(ieee754di_indef(), "dp_tlong", x); + ieee754_setcx(IEEE754_INVALID_OPERATION); + return ieee754di_indef(); } /* oh gawd */ - if (xe > DP_MBITS) { - xm <<= xe - DP_MBITS; - } else if (xe < DP_MBITS) { - u64 residue; - int round; - int sticky; - int odd; - + if (xe > DP_FBITS) { + xm <<= xe - DP_FBITS; + } else if (xe < DP_FBITS) { if (xe < -1) { residue = xm; round = 0; @@ -75,51 +72,38 @@ s64 ieee754dp_tlong(ieee754dp x) * so we do it in two steps. Be aware that xe * may be -1 */ residue = xm << (xe + 1); - residue <<= 63 - DP_MBITS; + residue <<= 63 - DP_FBITS; round = (residue >> 63) != 0; sticky = (residue << 1) != 0; - xm >>= DP_MBITS - xe; + xm >>= DP_FBITS - xe; } odd = (xm & 0x1) != 0x0; switch (ieee754_csr.rm) { - case IEEE754_RN: + case FPU_CSR_RN: if (round && (sticky || odd)) xm++; break; - case IEEE754_RZ: + case FPU_CSR_RZ: break; - case IEEE754_RU: /* toward +Infinity */ + case FPU_CSR_RU: /* toward +Infinity */ if ((round || sticky) && !xs) xm++; break; - case IEEE754_RD: /* toward -Infinity */ + case FPU_CSR_RD: /* toward -Infinity */ if ((round || sticky) && xs) xm++; break; } if ((xm >> 63) != 0) { /* This can happen after rounding */ - SETCX(IEEE754_INVALID_OPERATION); - return ieee754di_xcpt(ieee754di_indef(), "dp_tlong", x); + ieee754_setcx(IEEE754_INVALID_OPERATION); + return ieee754di_indef(); } if (round || sticky) - SETCX(IEEE754_INEXACT); + ieee754_setcx(IEEE754_INEXACT); } if (xs) return -xm; else return xm; } - - -u64 ieee754dp_tulong(ieee754dp x) -{ - ieee754dp hb = ieee754dp_1e63(); - - /* what if x < 0 ?? */ - if (ieee754dp_lt(x, hb)) - return (u64) ieee754dp_tlong(x); - - return (u64) ieee754dp_tlong(ieee754dp_sub(x, hb)) | - (1ULL << 63); -} diff --git a/arch/mips/math-emu/dsemul.c b/arch/mips/math-emu/dsemul.c index 7ea622ab8dad..4f514f3724cb 100644 --- a/arch/mips/math-emu/dsemul.c +++ b/arch/mips/math-emu/dsemul.c @@ -1,30 +1,12 @@ -#include <linux/compiler.h> -#include <linux/mm.h> -#include <linux/signal.h> -#include <linux/smp.h> - -#include <asm/asm.h> -#include <asm/bootinfo.h> -#include <asm/byteorder.h> -#include <asm/cpu.h> -#include <asm/inst.h> -#include <asm/processor.h> -#include <asm/uaccess.h> #include <asm/branch.h> -#include <asm/mipsregs.h> #include <asm/cacheflush.h> - #include <asm/fpu_emulator.h> +#include <asm/inst.h> +#include <asm/mipsregs.h> +#include <asm/uaccess.h> #include "ieee754.h" -/* Strap kernel emulator for full MIPS IV emulation */ - -#ifdef __mips -#undef __mips -#endif -#define __mips 4 - /* * Emulate the arbritrary instruction ir at xcp->cp0_epc. Required when * we have to emulate the instruction in a COP1 branch delay slot. Do @@ -59,13 +41,11 @@ int mips_dsemul(struct pt_regs *regs, mips_instruction ir, unsigned long cpc) (ir == 0)) { /* NOP is easy */ regs->cp0_epc = cpc; - regs->cp0_cause &= ~CAUSEF_BD; + clear_delay_slot(regs); return 0; } -#ifdef DSEMUL_TRACE - printk("dsemul %lx %lx\n", regs->cp0_epc, cpc); -#endif + pr_debug("dsemul %lx %lx\n", regs->cp0_epc, cpc); /* * The strategy is to push the instruction onto the user stack @@ -167,9 +147,8 @@ int do_dsemulret(struct pt_regs *xcp) * emulating the branch delay instruction. */ -#ifdef DSEMUL_TRACE - printk("dsemulret\n"); -#endif + pr_debug("dsemulret\n"); + if (__get_user(epc, &fr->epc)) { /* Saved EPC */ /* This is not a good situation to be in */ force_sig(SIGBUS, current); diff --git a/arch/mips/math-emu/ieee754.c b/arch/mips/math-emu/ieee754.c index 0015cf1989da..53f1d2287084 100644 --- a/arch/mips/math-emu/ieee754.c +++ b/arch/mips/math-emu/ieee754.c @@ -10,8 +10,6 @@ * MIPS floating point support * Copyright (C) 1994-2000 Algorithmics Ltd. * - * ######################################################################## - * * This program is free software; you can distribute it and/or modify it * under the terms of the GNU General Public License (Version 2) as * published by the Free Software Foundation. @@ -23,105 +21,69 @@ * * You should have received a copy of the GNU General Public License along * with this program; if not, write to the Free Software Foundation, Inc., - * 59 Temple Place - Suite 330, Boston MA 02111-1307, USA. - * - * ######################################################################## + * 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. */ +#include <linux/compiler.h> -#include "ieee754int.h" +#include "ieee754.h" #include "ieee754sp.h" #include "ieee754dp.h" -#define DP_EBIAS 1023 -#define DP_EMIN (-1022) -#define DP_EMAX 1023 - -#define SP_EBIAS 127 -#define SP_EMIN (-126) -#define SP_EMAX 127 - -/* special constants -*/ - - -#if (defined(BYTE_ORDER) && BYTE_ORDER == LITTLE_ENDIAN) || defined(__MIPSEL__) -#define SPSTR(s, b, m) {m, b, s} -#define DPSTR(s, b, mh, ml) {ml, mh, b, s} -#endif - -#ifdef __MIPSEB__ -#define SPSTR(s, b, m) {s, b, m} -#define DPSTR(s, b, mh, ml) {s, b, mh, ml} -#endif +/* + * Special constants + */ -const struct ieee754dp_konst __ieee754dp_spcvals[] = { - DPSTR(0, DP_EMIN - 1 + DP_EBIAS, 0, 0), /* + zero */ - DPSTR(1, DP_EMIN - 1 + DP_EBIAS, 0, 0), /* - zero */ - DPSTR(0, DP_EBIAS, 0, 0), /* + 1.0 */ - DPSTR(1, DP_EBIAS, 0, 0), /* - 1.0 */ - DPSTR(0, 3 + DP_EBIAS, 0x40000, 0), /* + 10.0 */ - DPSTR(1, 3 + DP_EBIAS, 0x40000, 0), /* - 10.0 */ - DPSTR(0, DP_EMAX + 1 + DP_EBIAS, 0, 0), /* + infinity */ - DPSTR(1, DP_EMAX + 1 + DP_EBIAS, 0, 0), /* - infinity */ - DPSTR(0, DP_EMAX+1+DP_EBIAS, 0x7FFFF, 0xFFFFFFFF), /* + indef quiet Nan */ - DPSTR(0, DP_EMAX + DP_EBIAS, 0xFFFFF, 0xFFFFFFFF), /* + max */ - DPSTR(1, DP_EMAX + DP_EBIAS, 0xFFFFF, 0xFFFFFFFF), /* - max */ - DPSTR(0, DP_EMIN + DP_EBIAS, 0, 0), /* + min normal */ - DPSTR(1, DP_EMIN + DP_EBIAS, 0, 0), /* - min normal */ - DPSTR(0, DP_EMIN - 1 + DP_EBIAS, 0, 1), /* + min denormal */ - DPSTR(1, DP_EMIN - 1 + DP_EBIAS, 0, 1), /* - min denormal */ - DPSTR(0, 31 + DP_EBIAS, 0, 0), /* + 1.0e31 */ - DPSTR(0, 63 + DP_EBIAS, 0, 0), /* + 1.0e63 */ -}; +#define DPCNST(s, b, m) \ +{ \ + .sign = (s), \ + .bexp = (b) + DP_EBIAS, \ + .mant = (m) \ +} -const struct ieee754sp_konst __ieee754sp_spcvals[] = { - SPSTR(0, SP_EMIN - 1 + SP_EBIAS, 0), /* + zero */ - SPSTR(1, SP_EMIN - 1 + SP_EBIAS, 0), /* - zero */ - SPSTR(0, SP_EBIAS, 0), /* + 1.0 */ - SPSTR(1, SP_EBIAS, 0), /* - 1.0 */ - SPSTR(0, 3 + SP_EBIAS, 0x200000), /* + 10.0 */ - SPSTR(1, 3 + SP_EBIAS, 0x200000), /* - 10.0 */ - SPSTR(0, SP_EMAX + 1 + SP_EBIAS, 0), /* + infinity */ - SPSTR(1, SP_EMAX + 1 + SP_EBIAS, 0), /* - infinity */ - SPSTR(0, SP_EMAX+1+SP_EBIAS, 0x3FFFFF), /* + indef quiet Nan */ - SPSTR(0, SP_EMAX + SP_EBIAS, 0x7FFFFF), /* + max normal */ - SPSTR(1, SP_EMAX + SP_EBIAS, 0x7FFFFF), /* - max normal */ - SPSTR(0, SP_EMIN + SP_EBIAS, 0), /* + min normal */ - SPSTR(1, SP_EMIN + SP_EBIAS, 0), /* - min normal */ - SPSTR(0, SP_EMIN - 1 + SP_EBIAS, 1), /* + min denormal */ - SPSTR(1, SP_EMIN - 1 + SP_EBIAS, 1), /* - min denormal */ - SPSTR(0, 31 + SP_EBIAS, 0), /* + 1.0e31 */ - SPSTR(0, 63 + SP_EBIAS, 0), /* + 1.0e63 */ +const union ieee754dp __ieee754dp_spcvals[] = { + DPCNST(0, DP_EMIN - 1, 0x0000000000000ULL), /* + zero */ + DPCNST(1, DP_EMIN - 1, 0x0000000000000ULL), /* - zero */ + DPCNST(0, 0, 0x0000000000000ULL), /* + 1.0 */ + DPCNST(1, 0, 0x0000000000000ULL), /* - 1.0 */ + DPCNST(0, 3, 0x4000000000000ULL), /* + 10.0 */ + DPCNST(1, 3, 0x4000000000000ULL), /* - 10.0 */ + DPCNST(0, DP_EMAX + 1, 0x0000000000000ULL), /* + infinity */ + DPCNST(1, DP_EMAX + 1, 0x0000000000000ULL), /* - infinity */ + DPCNST(0, DP_EMAX + 1, 0x7FFFFFFFFFFFFULL), /* + indef quiet Nan */ + DPCNST(0, DP_EMAX, 0xFFFFFFFFFFFFFULL), /* + max */ + DPCNST(1, DP_EMAX, 0xFFFFFFFFFFFFFULL), /* - max */ + DPCNST(0, DP_EMIN, 0x0000000000000ULL), /* + min normal */ + DPCNST(1, DP_EMIN, 0x0000000000000ULL), /* - min normal */ + DPCNST(0, DP_EMIN - 1, 0x0000000000001ULL), /* + min denormal */ + DPCNST(1, DP_EMIN - 1, 0x0000000000001ULL), /* - min denormal */ + DPCNST(0, 31, 0x0000000000000ULL), /* + 1.0e31 */ + DPCNST(0, 63, 0x0000000000000ULL), /* + 1.0e63 */ }; - -int ieee754si_xcpt(int r, const char *op, ...) -{ - struct ieee754xctx ax; - - if (!TSTX()) - return r; - ax.op = op; - ax.rt = IEEE754_RT_SI; - ax.rv.si = r; - va_start(ax.ap, op); - ieee754_xcpt(&ax); - va_end(ax.ap); - return ax.rv.si; +#define SPCNST(s, b, m) \ +{ \ + .sign = (s), \ + .bexp = (b) + SP_EBIAS, \ + .mant = (m) \ } -s64 ieee754di_xcpt(s64 r, const char *op, ...) -{ - struct ieee754xctx ax; - - if (!TSTX()) - return r; - ax.op = op; - ax.rt = IEEE754_RT_DI; - ax.rv.di = r; - va_start(ax.ap, op); - ieee754_xcpt(&ax); - va_end(ax.ap); - return ax.rv.di; -} +const union ieee754sp __ieee754sp_spcvals[] = { + SPCNST(0, SP_EMIN - 1, 0x000000), /* + zero */ + SPCNST(1, SP_EMIN - 1, 0x000000), /* - zero */ + SPCNST(0, 0, 0x000000), /* + 1.0 */ + SPCNST(1, 0, 0x000000), /* - 1.0 */ + SPCNST(0, 3, 0x200000), /* + 10.0 */ + SPCNST(1, 3, 0x200000), /* - 10.0 */ + SPCNST(0, SP_EMAX + 1, 0x000000), /* + infinity */ + SPCNST(1, SP_EMAX + 1, 0x000000), /* - infinity */ + SPCNST(0, SP_EMAX + 1, 0x3FFFFF), /* + indef quiet Nan */ + SPCNST(0, SP_EMAX, 0x7FFFFF), /* + max normal */ + SPCNST(1, SP_EMAX, 0x7FFFFF), /* - max normal */ + SPCNST(0, SP_EMIN, 0x000000), /* + min normal */ + SPCNST(1, SP_EMIN, 0x000000), /* - min normal */ + SPCNST(0, SP_EMIN - 1, 0x000001), /* + min denormal */ + SPCNST(1, SP_EMIN - 1, 0x000001), /* - min denormal */ + SPCNST(0, 31, 0x000000), /* + 1.0e31 */ + SPCNST(0, 63, 0x000000), /* + 1.0e63 */ +}; diff --git a/arch/mips/math-emu/ieee754.h b/arch/mips/math-emu/ieee754.h index 22796e012060..43c4fb522ac2 100644 --- a/arch/mips/math-emu/ieee754.h +++ b/arch/mips/math-emu/ieee754.h @@ -13,7 +13,7 @@ * * You should have received a copy of the GNU General Public License along * with this program; if not, write to the Free Software Foundation, Inc., - * 59 Temple Place - Suite 330, Boston MA 02111-1307, USA. + * 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. * * Nov 7, 2000 * Modification to allow integration with Linux kernel @@ -24,186 +24,93 @@ #ifndef __ARCH_MIPS_MATH_EMU_IEEE754_H #define __ARCH_MIPS_MATH_EMU_IEEE754_H +#include <linux/compiler.h> #include <asm/byteorder.h> +#include <linux/kernel.h> #include <linux/types.h> #include <linux/sched.h> +#include <asm/bitfield.h> -/* - * Not very pretty, but the Linux kernel's normal va_list definition - * does not allow it to be used as a structure element, as it is here. - */ -#ifndef _STDARG_H -#include <stdarg.h> -#endif - -#ifdef __LITTLE_ENDIAN -struct ieee754dp_konst { - unsigned mantlo:32; - unsigned manthi:20; - unsigned bexp:11; - unsigned sign:1; -}; -struct ieee754sp_konst { - unsigned mant:23; - unsigned bexp:8; - unsigned sign:1; -}; - -typedef union _ieee754dp { - struct ieee754dp_konst oparts; +union ieee754dp { struct { - u64 mant:52; - unsigned int bexp:11; - unsigned int sign:1; - } parts; + __BITFIELD_FIELD(unsigned int sign:1, + __BITFIELD_FIELD(unsigned int bexp:11, + __BITFIELD_FIELD(u64 mant:52, + ;))) + }; u64 bits; - double d; -} ieee754dp; - -typedef union _ieee754sp { - struct ieee754sp_konst parts; - float f; - u32 bits; -} ieee754sp; -#endif - -#ifdef __BIG_ENDIAN -struct ieee754dp_konst { - unsigned sign:1; - unsigned bexp:11; - unsigned manthi:20; - unsigned mantlo:32; }; -typedef union _ieee754dp { - struct ieee754dp_konst oparts; +union ieee754sp { struct { - unsigned int sign:1; - unsigned int bexp:11; - u64 mant:52; - } parts; - double d; - u64 bits; -} ieee754dp; - -struct ieee754sp_konst { - unsigned sign:1; - unsigned bexp:8; - unsigned mant:23; -}; - -typedef union _ieee754sp { - struct ieee754sp_konst parts; - float f; + __BITFIELD_FIELD(unsigned sign:1, + __BITFIELD_FIELD(unsigned bexp:8, + __BITFIELD_FIELD(unsigned mant:23, + ;))) + }; u32 bits; -} ieee754sp; -#endif +}; /* * single precision (often aka float) */ -int ieee754sp_finite(ieee754sp x); -int ieee754sp_class(ieee754sp x); - -ieee754sp ieee754sp_abs(ieee754sp x); -ieee754sp ieee754sp_neg(ieee754sp x); -ieee754sp ieee754sp_scalb(ieee754sp x, int); -ieee754sp ieee754sp_logb(ieee754sp x); - -/* x with sign of y */ -ieee754sp ieee754sp_copysign(ieee754sp x, ieee754sp y); - -ieee754sp ieee754sp_add(ieee754sp x, ieee754sp y); -ieee754sp ieee754sp_sub(ieee754sp x, ieee754sp y); -ieee754sp ieee754sp_mul(ieee754sp x, ieee754sp y); -ieee754sp ieee754sp_div(ieee754sp x, ieee754sp y); - -ieee754sp ieee754sp_fint(int x); -ieee754sp ieee754sp_funs(unsigned x); -ieee754sp ieee754sp_flong(s64 x); -ieee754sp ieee754sp_fulong(u64 x); -ieee754sp ieee754sp_fdp(ieee754dp x); - -int ieee754sp_tint(ieee754sp x); -unsigned int ieee754sp_tuns(ieee754sp x); -s64 ieee754sp_tlong(ieee754sp x); -u64 ieee754sp_tulong(ieee754sp x); - -int ieee754sp_cmp(ieee754sp x, ieee754sp y, int cop, int sig); -/* - * basic sp math - */ -ieee754sp ieee754sp_modf(ieee754sp x, ieee754sp * ip); -ieee754sp ieee754sp_frexp(ieee754sp x, int *exp); -ieee754sp ieee754sp_ldexp(ieee754sp x, int exp); +int ieee754sp_class(union ieee754sp x); -ieee754sp ieee754sp_ceil(ieee754sp x); -ieee754sp ieee754sp_floor(ieee754sp x); -ieee754sp ieee754sp_trunc(ieee754sp x); +union ieee754sp ieee754sp_abs(union ieee754sp x); +union ieee754sp ieee754sp_neg(union ieee754sp x); -ieee754sp ieee754sp_sqrt(ieee754sp x); +union ieee754sp ieee754sp_add(union ieee754sp x, union ieee754sp y); +union ieee754sp ieee754sp_sub(union ieee754sp x, union ieee754sp y); +union ieee754sp ieee754sp_mul(union ieee754sp x, union ieee754sp y); +union ieee754sp ieee754sp_div(union ieee754sp x, union ieee754sp y); -/* - * double precision (often aka double) -*/ -int ieee754dp_finite(ieee754dp x); -int ieee754dp_class(ieee754dp x); +union ieee754sp ieee754sp_fint(int x); +union ieee754sp ieee754sp_flong(s64 x); +union ieee754sp ieee754sp_fdp(union ieee754dp x); -/* x with sign of y */ -ieee754dp ieee754dp_copysign(ieee754dp x, ieee754dp y); +int ieee754sp_tint(union ieee754sp x); +s64 ieee754sp_tlong(union ieee754sp x); -ieee754dp ieee754dp_add(ieee754dp x, ieee754dp y); -ieee754dp ieee754dp_sub(ieee754dp x, ieee754dp y); -ieee754dp ieee754dp_mul(ieee754dp x, ieee754dp y); -ieee754dp ieee754dp_div(ieee754dp x, ieee754dp y); +int ieee754sp_cmp(union ieee754sp x, union ieee754sp y, int cop, int sig); -ieee754dp ieee754dp_abs(ieee754dp x); -ieee754dp ieee754dp_neg(ieee754dp x); -ieee754dp ieee754dp_scalb(ieee754dp x, int); +union ieee754sp ieee754sp_sqrt(union ieee754sp x); -/* return exponent as integer in floating point format - */ -ieee754dp ieee754dp_logb(ieee754dp x); +/* + * double precision (often aka double) +*/ +int ieee754dp_class(union ieee754dp x); -ieee754dp ieee754dp_fint(int x); -ieee754dp ieee754dp_funs(unsigned x); -ieee754dp ieee754dp_flong(s64 x); -ieee754dp ieee754dp_fulong(u64 x); -ieee754dp ieee754dp_fsp(ieee754sp x); +union ieee754dp ieee754dp_add(union ieee754dp x, union ieee754dp y); +union ieee754dp ieee754dp_sub(union ieee754dp x, union ieee754dp y); +union ieee754dp ieee754dp_mul(union ieee754dp x, union ieee754dp y); +union ieee754dp ieee754dp_div(union ieee754dp x, union ieee754dp y); -ieee754dp ieee754dp_ceil(ieee754dp x); -ieee754dp ieee754dp_floor(ieee754dp x); -ieee754dp ieee754dp_trunc(ieee754dp x); +union ieee754dp ieee754dp_abs(union ieee754dp x); +union ieee754dp ieee754dp_neg(union ieee754dp x); -int ieee754dp_tint(ieee754dp x); -unsigned int ieee754dp_tuns(ieee754dp x); -s64 ieee754dp_tlong(ieee754dp x); -u64 ieee754dp_tulong(ieee754dp x); +union ieee754dp ieee754dp_fint(int x); +union ieee754dp ieee754dp_flong(s64 x); +union ieee754dp ieee754dp_fsp(union ieee754sp x); -int ieee754dp_cmp(ieee754dp x, ieee754dp y, int cop, int sig); -/* - * basic sp math - */ -ieee754dp ieee754dp_modf(ieee754dp x, ieee754dp * ip); -ieee754dp ieee754dp_frexp(ieee754dp x, int *exp); -ieee754dp ieee754dp_ldexp(ieee754dp x, int exp); +int ieee754dp_tint(union ieee754dp x); +s64 ieee754dp_tlong(union ieee754dp x); -ieee754dp ieee754dp_ceil(ieee754dp x); -ieee754dp ieee754dp_floor(ieee754dp x); -ieee754dp ieee754dp_trunc(ieee754dp x); +int ieee754dp_cmp(union ieee754dp x, union ieee754dp y, int cop, int sig); -ieee754dp ieee754dp_sqrt(ieee754dp x); +union ieee754dp ieee754dp_sqrt(union ieee754dp x); /* 5 types of floating point number */ -#define IEEE754_CLASS_NORM 0x00 -#define IEEE754_CLASS_ZERO 0x01 -#define IEEE754_CLASS_DNORM 0x02 -#define IEEE754_CLASS_INF 0x03 -#define IEEE754_CLASS_SNAN 0x04 -#define IEEE754_CLASS_QNAN 0x05 +enum { + IEEE754_CLASS_NORM = 0x00, + IEEE754_CLASS_ZERO = 0x01, + IEEE754_CLASS_DNORM = 0x02, + IEEE754_CLASS_INF = 0x03, + IEEE754_CLASS_SNAN = 0x04, + IEEE754_CLASS_QNAN = 0x05, +}; /* exception numbers */ #define IEEE754_INEXACT 0x01 @@ -219,114 +126,84 @@ ieee754dp ieee754dp_sqrt(ieee754dp x); #define IEEE754_CGT 0x04 #define IEEE754_CUN 0x08 -/* rounding mode -*/ -#define IEEE754_RN 0 /* round to nearest */ -#define IEEE754_RZ 1 /* round toward zero */ -#define IEEE754_RD 2 /* round toward -Infinity */ -#define IEEE754_RU 3 /* round toward +Infinity */ - -/* other naming */ -#define IEEE754_RM IEEE754_RD -#define IEEE754_RP IEEE754_RU - /* "normal" comparisons */ -static inline int ieee754sp_eq(ieee754sp x, ieee754sp y) +static inline int ieee754sp_eq(union ieee754sp x, union ieee754sp y) { return ieee754sp_cmp(x, y, IEEE754_CEQ, 0); } -static inline int ieee754sp_ne(ieee754sp x, ieee754sp y) +static inline int ieee754sp_ne(union ieee754sp x, union ieee754sp y) { return ieee754sp_cmp(x, y, IEEE754_CLT | IEEE754_CGT | IEEE754_CUN, 0); } -static inline int ieee754sp_lt(ieee754sp x, ieee754sp y) +static inline int ieee754sp_lt(union ieee754sp x, union ieee754sp y) { return ieee754sp_cmp(x, y, IEEE754_CLT, 0); } -static inline int ieee754sp_le(ieee754sp x, ieee754sp y) +static inline int ieee754sp_le(union ieee754sp x, union ieee754sp y) { return ieee754sp_cmp(x, y, IEEE754_CLT | IEEE754_CEQ, 0); } -static inline int ieee754sp_gt(ieee754sp x, ieee754sp y) +static inline int ieee754sp_gt(union ieee754sp x, union ieee754sp y) { return ieee754sp_cmp(x, y, IEEE754_CGT, 0); } -static inline int ieee754sp_ge(ieee754sp x, ieee754sp y) +static inline int ieee754sp_ge(union ieee754sp x, union ieee754sp y) { return ieee754sp_cmp(x, y, IEEE754_CGT | IEEE754_CEQ, 0); } -static inline int ieee754dp_eq(ieee754dp x, ieee754dp y) +static inline int ieee754dp_eq(union ieee754dp x, union ieee754dp y) { return ieee754dp_cmp(x, y, IEEE754_CEQ, 0); } -static inline int ieee754dp_ne(ieee754dp x, ieee754dp y) +static inline int ieee754dp_ne(union ieee754dp x, union ieee754dp y) { return ieee754dp_cmp(x, y, IEEE754_CLT | IEEE754_CGT | IEEE754_CUN, 0); } -static inline int ieee754dp_lt(ieee754dp x, ieee754dp y) +static inline int ieee754dp_lt(union ieee754dp x, union ieee754dp y) { return ieee754dp_cmp(x, y, IEEE754_CLT, 0); } -static inline int ieee754dp_le(ieee754dp x, ieee754dp y) +static inline int ieee754dp_le(union ieee754dp x, union ieee754dp y) { return ieee754dp_cmp(x, y, IEEE754_CLT | IEEE754_CEQ, 0); } -static inline int ieee754dp_gt(ieee754dp x, ieee754dp y) +static inline int ieee754dp_gt(union ieee754dp x, union ieee754dp y) { return ieee754dp_cmp(x, y, IEEE754_CGT, 0); } -static inline int ieee754dp_ge(ieee754dp x, ieee754dp y) +static inline int ieee754dp_ge(union ieee754dp x, union ieee754dp y) { return ieee754dp_cmp(x, y, IEEE754_CGT | IEEE754_CEQ, 0); } - -/* - * Like strtod - */ -ieee754dp ieee754dp_fstr(const char *s, char **endp); -char *ieee754dp_tstr(ieee754dp x, int prec, int fmt, int af); - - /* * The control status register */ struct _ieee754_csr { -#ifdef __BIG_ENDIAN - unsigned pad0:7; - unsigned nod:1; /* set 1 for no denormalised numbers */ - unsigned c:1; /* condition */ - unsigned pad1:5; - unsigned cx:6; /* exceptions this operation */ - unsigned mx:5; /* exception enable mask */ - unsigned sx:5; /* exceptions total */ - unsigned rm:2; /* current rounding mode */ -#endif -#ifdef __LITTLE_ENDIAN - unsigned rm:2; /* current rounding mode */ - unsigned sx:5; /* exceptions total */ - unsigned mx:5; /* exception enable mask */ - unsigned cx:6; /* exceptions this operation */ - unsigned pad1:5; - unsigned c:1; /* condition */ - unsigned nod:1; /* set 1 for no denormalised numbers */ - unsigned pad0:7; -#endif + __BITFIELD_FIELD(unsigned pad0:7, + __BITFIELD_FIELD(unsigned nod:1, /* set 1 for no denormalised numbers */ + __BITFIELD_FIELD(unsigned c:1, /* condition */ + __BITFIELD_FIELD(unsigned pad1:5, + __BITFIELD_FIELD(unsigned cx:6, /* exceptions this operation */ + __BITFIELD_FIELD(unsigned mx:5, /* exception enable mask */ + __BITFIELD_FIELD(unsigned sx:5, /* exceptions total */ + __BITFIELD_FIELD(unsigned rm:2, /* current rounding mode */ + ;)))))))) }; #define ieee754_csr (*(struct _ieee754_csr *)(¤t->thread.fpu.fcr31)) @@ -377,8 +254,8 @@ static inline int ieee754_sxtest(unsigned n) } /* debugging */ -ieee754sp ieee754sp_dump(char *s, ieee754sp x); -ieee754dp ieee754dp_dump(char *s, ieee754dp x); +union ieee754sp ieee754sp_dump(char *s, union ieee754sp x); +union ieee754dp ieee754dp_dump(char *s, union ieee754dp x); #define IEEE754_SPCVAL_PZERO 0 #define IEEE754_SPCVAL_NZERO 1 @@ -398,10 +275,10 @@ ieee754dp ieee754dp_dump(char *s, ieee754dp x); #define IEEE754_SPCVAL_P1E31 15 /* + 1.0e31 */ #define IEEE754_SPCVAL_P1E63 16 /* + 1.0e63 */ -extern const struct ieee754dp_konst __ieee754dp_spcvals[]; -extern const struct ieee754sp_konst __ieee754sp_spcvals[]; -#define ieee754dp_spcvals ((const ieee754dp *)__ieee754dp_spcvals) -#define ieee754sp_spcvals ((const ieee754sp *)__ieee754sp_spcvals) +extern const union ieee754dp __ieee754dp_spcvals[]; +extern const union ieee754sp __ieee754sp_spcvals[]; +#define ieee754dp_spcvals ((const union ieee754dp *)__ieee754dp_spcvals) +#define ieee754sp_spcvals ((const union ieee754sp *)__ieee754sp_spcvals) /* * Return infinity with given sign @@ -431,28 +308,15 @@ extern const struct ieee754sp_konst __ieee754sp_spcvals[]; /* * Indefinite integer value */ -#define ieee754si_indef() INT_MAX -#ifdef LONG_LONG_MAX -#define ieee754di_indef() LONG_LONG_MAX -#else -#define ieee754di_indef() ((s64)(~0ULL>>1)) -#endif - -/* IEEE exception context, passed to handler */ -struct ieee754xctx { - const char *op; /* operation name */ - int rt; /* result type */ - union { - ieee754sp sp; /* single precision */ - ieee754dp dp; /* double precision */ -#ifdef IEEE854_XP - ieee754xp xp; /* extended precision */ -#endif - int si; /* standard signed integer (32bits) */ - s64 di; /* extended signed integer (64bits) */ - } rv; /* default result format implied by op */ - va_list ap; -}; +static inline int ieee754si_indef(void) +{ + return INT_MAX; +} + +static inline s64 ieee754di_indef(void) +{ + return S64_MAX; +} /* result types for xctx.rt */ #define IEEE754_RT_SP 0 @@ -461,8 +325,6 @@ struct ieee754xctx { #define IEEE754_RT_SI 3 #define IEEE754_RT_DI 4 -extern void ieee754_xcpt(struct ieee754xctx *xcp); - /* compat */ #define ieee754dp_fix(x) ieee754dp_tint(x) #define ieee754sp_fix(x) ieee754sp_tint(x) diff --git a/arch/mips/math-emu/ieee754d.c b/arch/mips/math-emu/ieee754d.c index 9599bdd32585..a04e8a7e5ac3 100644 --- a/arch/mips/math-emu/ieee754d.c +++ b/arch/mips/math-emu/ieee754d.c @@ -16,7 +16,7 @@ * * You should have received a copy of the GNU General Public License along * with this program; if not, write to the Free Software Foundation, Inc., - * 59 Temple Place - Suite 330, Boston MA 02111-1307, USA. + * 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. * * Nov 7, 2000 * Modified to build and operate in Linux kernel environment. @@ -25,38 +25,13 @@ * Copyright (C) 2000 MIPS Technologies, Inc. All rights reserved. */ -#include <linux/kernel.h> +#include <linux/types.h> +#include <linux/printk.h> #include "ieee754.h" +#include "ieee754sp.h" +#include "ieee754dp.h" -#define DP_EBIAS 1023 -#define DP_EMIN (-1022) -#define DP_EMAX 1023 -#define DP_FBITS 52 - -#define SP_EBIAS 127 -#define SP_EMIN (-126) -#define SP_EMAX 127 -#define SP_FBITS 23 - -#define DP_MBIT(x) ((u64)1 << (x)) -#define DP_HIDDEN_BIT DP_MBIT(DP_FBITS) -#define DP_SIGN_BIT DP_MBIT(63) - - -#define SP_MBIT(x) ((u32)1 << (x)) -#define SP_HIDDEN_BIT SP_MBIT(SP_FBITS) -#define SP_SIGN_BIT SP_MBIT(31) - - -#define SPSIGN(sp) (sp.parts.sign) -#define SPBEXP(sp) (sp.parts.bexp) -#define SPMANT(sp) (sp.parts.mant) - -#define DPSIGN(dp) (dp.parts.sign) -#define DPBEXP(dp) (dp.parts.bexp) -#define DPMANT(dp) (dp.parts.mant) - -ieee754dp ieee754dp_dump(char *m, ieee754dp x) +union ieee754dp ieee754dp_dump(char *m, union ieee754dp x) { int i; @@ -96,7 +71,7 @@ ieee754dp ieee754dp_dump(char *m, ieee754dp x) return x; } -ieee754sp ieee754sp_dump(char *m, ieee754sp x) +union ieee754sp ieee754sp_dump(char *m, union ieee754sp x) { int i; diff --git a/arch/mips/math-emu/ieee754dp.c b/arch/mips/math-emu/ieee754dp.c index 068e56be8de9..fd134675fc2e 100644 --- a/arch/mips/math-emu/ieee754dp.c +++ b/arch/mips/math-emu/ieee754dp.c @@ -5,8 +5,6 @@ * MIPS floating point support * Copyright (C) 1994-2000 Algorithmics Ltd. * - * ######################################################################## - * * This program is free software; you can distribute it and/or modify it * under the terms of the GNU General Public License (Version 2) as * published by the Free Software Foundation. @@ -18,104 +16,68 @@ * * You should have received a copy of the GNU General Public License along * with this program; if not, write to the Free Software Foundation, Inc., - * 59 Temple Place - Suite 330, Boston MA 02111-1307, USA. - * - * ######################################################################## + * 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. */ +#include <linux/compiler.h> #include "ieee754dp.h" -int ieee754dp_class(ieee754dp x) +int ieee754dp_class(union ieee754dp x) { COMPXDP; EXPLODEXDP; return xc; } -int ieee754dp_isnan(ieee754dp x) +int ieee754dp_isnan(union ieee754dp x) { return ieee754dp_class(x) >= IEEE754_CLASS_SNAN; } -int ieee754dp_issnan(ieee754dp x) +static inline int ieee754dp_issnan(union ieee754dp x) { assert(ieee754dp_isnan(x)); - return ((DPMANT(x) & DP_MBIT(DP_MBITS-1)) == DP_MBIT(DP_MBITS-1)); + return ((DPMANT(x) & DP_MBIT(DP_FBITS-1)) == DP_MBIT(DP_FBITS-1)); } -ieee754dp ieee754dp_xcpt(ieee754dp r, const char *op, ...) -{ - struct ieee754xctx ax; - if (!TSTX()) - return r; - - ax.op = op; - ax.rt = IEEE754_RT_DP; - ax.rv.dp = r; - va_start(ax.ap, op); - ieee754_xcpt(&ax); - va_end(ax.ap); - return ax.rv.dp; -} - -ieee754dp ieee754dp_nanxcpt(ieee754dp r, const char *op, ...) +union ieee754dp __cold ieee754dp_nanxcpt(union ieee754dp r) { - struct ieee754xctx ax; - assert(ieee754dp_isnan(r)); if (!ieee754dp_issnan(r)) /* QNAN does not cause invalid op !! */ return r; - if (!SETANDTESTCX(IEEE754_INVALID_OPERATION)) { + if (!ieee754_setandtestcx(IEEE754_INVALID_OPERATION)) { /* not enabled convert to a quiet NaN */ - DPMANT(r) &= (~DP_MBIT(DP_MBITS-1)); + DPMANT(r) &= (~DP_MBIT(DP_FBITS-1)); if (ieee754dp_isnan(r)) return r; else return ieee754dp_indef(); } - ax.op = op; - ax.rt = 0; - ax.rv.dp = r; - va_start(ax.ap, op); - ieee754_xcpt(&ax); - va_end(ax.ap); - return ax.rv.dp; + return r; } -ieee754dp ieee754dp_bestnan(ieee754dp x, ieee754dp y) -{ - assert(ieee754dp_isnan(x)); - assert(ieee754dp_isnan(y)); - - if (DPMANT(x) > DPMANT(y)) - return x; - else - return y; -} - - -static u64 get_rounding(int sn, u64 xm) +static u64 ieee754dp_get_rounding(int sn, u64 xm) { /* inexact must round of 3 bits */ if (xm & (DP_MBIT(3) - 1)) { switch (ieee754_csr.rm) { - case IEEE754_RZ: + case FPU_CSR_RZ: break; - case IEEE754_RN: + case FPU_CSR_RN: xm += 0x3 + ((xm >> 3) & 1); /* xm += (xm&0x8)?0x4:0x3 */ break; - case IEEE754_RU: /* toward +Infinity */ + case FPU_CSR_RU: /* toward +Infinity */ if (!sn) /* ?? */ xm += 0x8; break; - case IEEE754_RD: /* toward -Infinity */ + case FPU_CSR_RD: /* toward -Infinity */ if (sn) /* ?? */ xm += 0x8; break; @@ -130,11 +92,11 @@ static u64 get_rounding(int sn, u64 xm) * xe is an unbiased exponent * xm is 3bit extended precision value. */ -ieee754dp ieee754dp_format(int sn, int xe, u64 xm) +union ieee754dp ieee754dp_format(int sn, int xe, u64 xm) { assert(xm); /* we don't gen exact zeros (probably should) */ - assert((xm >> (DP_MBITS + 1 + 3)) == 0); /* no execess */ + assert((xm >> (DP_FBITS + 1 + 3)) == 0); /* no execess */ assert(xm & (DP_HIDDEN_BIT << 3)); if (xe < DP_EMIN) { @@ -142,32 +104,32 @@ ieee754dp ieee754dp_format(int sn, int xe, u64 xm) int es = DP_EMIN - xe; if (ieee754_csr.nod) { - SETCX(IEEE754_UNDERFLOW); - SETCX(IEEE754_INEXACT); + ieee754_setcx(IEEE754_UNDERFLOW); + ieee754_setcx(IEEE754_INEXACT); switch(ieee754_csr.rm) { - case IEEE754_RN: - case IEEE754_RZ: + case FPU_CSR_RN: + case FPU_CSR_RZ: return ieee754dp_zero(sn); - case IEEE754_RU: /* toward +Infinity */ - if(sn == 0) + case FPU_CSR_RU: /* toward +Infinity */ + if (sn == 0) return ieee754dp_min(0); else return ieee754dp_zero(1); - case IEEE754_RD: /* toward -Infinity */ - if(sn == 0) + case FPU_CSR_RD: /* toward -Infinity */ + if (sn == 0) return ieee754dp_zero(0); else return ieee754dp_min(1); } } - if (xe == DP_EMIN - 1 - && get_rounding(sn, xm) >> (DP_MBITS + 1 + 3)) + if (xe == DP_EMIN - 1 && + ieee754dp_get_rounding(sn, xm) >> (DP_FBITS + 1 + 3)) { /* Not tiny after rounding */ - SETCX(IEEE754_INEXACT); - xm = get_rounding(sn, xm); + ieee754_setcx(IEEE754_INEXACT); + xm = ieee754dp_get_rounding(sn, xm); xm >>= 1; /* Clear grs bits */ xm &= ~(DP_MBIT(3) - 1); @@ -183,17 +145,17 @@ ieee754dp ieee754dp_format(int sn, int xe, u64 xm) } } if (xm & (DP_MBIT(3) - 1)) { - SETCX(IEEE754_INEXACT); + ieee754_setcx(IEEE754_INEXACT); if ((xm & (DP_HIDDEN_BIT << 3)) == 0) { - SETCX(IEEE754_UNDERFLOW); + ieee754_setcx(IEEE754_UNDERFLOW); } /* inexact must round of 3 bits */ - xm = get_rounding(sn, xm); + xm = ieee754dp_get_rounding(sn, xm); /* adjust exponent for rounding add overflowing */ - if (xm >> (DP_MBITS + 3 + 1)) { + if (xm >> (DP_FBITS + 3 + 1)) { /* add causes mantissa overflow */ xm >>= 1; xe++; @@ -202,24 +164,24 @@ ieee754dp ieee754dp_format(int sn, int xe, u64 xm) /* strip grs bits */ xm >>= 3; - assert((xm >> (DP_MBITS + 1)) == 0); /* no execess */ + assert((xm >> (DP_FBITS + 1)) == 0); /* no execess */ assert(xe >= DP_EMIN); if (xe > DP_EMAX) { - SETCX(IEEE754_OVERFLOW); - SETCX(IEEE754_INEXACT); + ieee754_setcx(IEEE754_OVERFLOW); + ieee754_setcx(IEEE754_INEXACT); /* -O can be table indexed by (rm,sn) */ switch (ieee754_csr.rm) { - case IEEE754_RN: + case FPU_CSR_RN: return ieee754dp_inf(sn); - case IEEE754_RZ: + case FPU_CSR_RZ: return ieee754dp_max(sn); - case IEEE754_RU: /* toward +Infinity */ + case FPU_CSR_RU: /* toward +Infinity */ if (sn == 0) return ieee754dp_inf(0); else return ieee754dp_max(1); - case IEEE754_RD: /* toward -Infinity */ + case FPU_CSR_RD: /* toward -Infinity */ if (sn == 0) return ieee754dp_max(0); else @@ -232,10 +194,10 @@ ieee754dp ieee754dp_format(int sn, int xe, u64 xm) /* we underflow (tiny/zero) */ assert(xe == DP_EMIN); if (ieee754_csr.mx & IEEE754_UNDERFLOW) - SETCX(IEEE754_UNDERFLOW); + ieee754_setcx(IEEE754_UNDERFLOW); return builddp(sn, DP_EMIN - 1 + DP_EBIAS, xm); } else { - assert((xm >> (DP_MBITS + 1)) == 0); /* no execess */ + assert((xm >> (DP_FBITS + 1)) == 0); /* no execess */ assert(xm & DP_HIDDEN_BIT); return builddp(sn, xe + DP_EBIAS, xm & ~DP_HIDDEN_BIT); diff --git a/arch/mips/math-emu/ieee754dp.h b/arch/mips/math-emu/ieee754dp.h index f139c724c59a..61fd6fd31350 100644 --- a/arch/mips/math-emu/ieee754dp.h +++ b/arch/mips/math-emu/ieee754dp.h @@ -6,8 +6,6 @@ * MIPS floating point support * Copyright (C) 1994-2000 Algorithmics Ltd. * - * ######################################################################## - * * This program is free software; you can distribute it and/or modify it * under the terms of the GNU General Public License (Version 2) as * published by the Free Software Foundation. @@ -19,64 +17,66 @@ * * You should have received a copy of the GNU General Public License along * with this program; if not, write to the Free Software Foundation, Inc., - * 59 Temple Place - Suite 330, Boston MA 02111-1307, USA. - * - * ######################################################################## + * 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. */ +#include <linux/compiler.h> #include "ieee754int.h" #define assert(expr) ((void)0) +#define DP_EBIAS 1023 +#define DP_EMIN (-1022) +#define DP_EMAX 1023 +#define DP_FBITS 52 +#define DP_MBITS 52 + +#define DP_MBIT(x) ((u64)1 << (x)) +#define DP_HIDDEN_BIT DP_MBIT(DP_FBITS) +#define DP_SIGN_BIT DP_MBIT(63) + +#define DPSIGN(dp) (dp.sign) +#define DPBEXP(dp) (dp.bexp) +#define DPMANT(dp) (dp.mant) + +static inline int ieee754dp_finite(union ieee754dp x) +{ + return DPBEXP(x) != DP_EMAX + 1 + DP_EBIAS; +} + /* 3bit extended double precision sticky right shift */ #define XDPSRS(v,rs) \ - ((rs > (DP_MBITS+3))?1:((v) >> (rs)) | ((v) << (64-(rs)) != 0)) + ((rs > (DP_FBITS+3))?1:((v) >> (rs)) | ((v) << (64-(rs)) != 0)) #define XDPSRSX1() \ - (xe++, (xm = (xm >> 1) | (xm & 1))) + (xe++, (xm = (xm >> 1) | (xm & 1))) #define XDPSRS1(v) \ - (((v) >> 1) | ((v) & 1)) + (((v) >> 1) | ((v) & 1)) /* convert denormal to normalized with extended exponent */ #define DPDNORMx(m,e) \ - while( (m >> DP_MBITS) == 0) { m <<= 1; e--; } + while ((m >> DP_FBITS) == 0) { m <<= 1; e--; } #define DPDNORMX DPDNORMx(xm, xe) #define DPDNORMY DPDNORMx(ym, ye) -static inline ieee754dp builddp(int s, int bx, u64 m) +static inline union ieee754dp builddp(int s, int bx, u64 m) { - ieee754dp r; + union ieee754dp r; assert((s) == 0 || (s) == 1); assert((bx) >= DP_EMIN - 1 + DP_EBIAS && (bx) <= DP_EMAX + 1 + DP_EBIAS); - assert(((m) >> DP_MBITS) == 0); + assert(((m) >> DP_FBITS) == 0); - r.parts.sign = s; - r.parts.bexp = bx; - r.parts.mant = m; - return r; -} + r.sign = s; + r.bexp = bx; + r.mant = m; -extern int ieee754dp_isnan(ieee754dp); -extern int ieee754dp_issnan(ieee754dp); -extern int ieee754si_xcpt(int, const char *, ...); -extern s64 ieee754di_xcpt(s64, const char *, ...); -extern ieee754dp ieee754dp_xcpt(ieee754dp, const char *, ...); -extern ieee754dp ieee754dp_nanxcpt(ieee754dp, const char *, ...); -extern ieee754dp ieee754dp_bestnan(ieee754dp, ieee754dp); -extern ieee754dp ieee754dp_format(int, int, u64); - - -#define DPNORMRET2(s, e, m, name, a0, a1) \ -{ \ - ieee754dp V = ieee754dp_format(s, e, m); \ - if(TSTX()) \ - return ieee754dp_xcpt(V, name, a0, a1); \ - else \ - return V; \ + return r; } -#define DPNORMRET1(s, e, m, name, a0) DPNORMRET2(s, e, m, name, a0, a0) +extern int ieee754dp_isnan(union ieee754dp); +extern union ieee754dp __cold ieee754dp_nanxcpt(union ieee754dp); +extern union ieee754dp ieee754dp_format(int, int, u64); diff --git a/arch/mips/math-emu/ieee754int.h b/arch/mips/math-emu/ieee754int.h index 4b6c6fb35304..f0365bb86747 100644 --- a/arch/mips/math-emu/ieee754int.h +++ b/arch/mips/math-emu/ieee754int.h @@ -6,8 +6,6 @@ * MIPS floating point support * Copyright (C) 1994-2000 Algorithmics Ltd. * - * ######################################################################## - * * This program is free software; you can distribute it and/or modify it * under the terms of the GNU General Public License (Version 2) as * published by the Free Software Foundation. @@ -19,146 +17,125 @@ * * You should have received a copy of the GNU General Public License along * with this program; if not, write to the Free Software Foundation, Inc., - * 59 Temple Place - Suite 330, Boston MA 02111-1307, USA. - * - * ######################################################################## + * 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. */ - +#ifndef __IEEE754INT_H +#define __IEEE754INT_H #include "ieee754.h" -#define DP_EBIAS 1023 -#define DP_EMIN (-1022) -#define DP_EMAX 1023 -#define DP_MBITS 52 - -#define SP_EBIAS 127 -#define SP_EMIN (-126) -#define SP_EMAX 127 -#define SP_MBITS 23 - -#define DP_MBIT(x) ((u64)1 << (x)) -#define DP_HIDDEN_BIT DP_MBIT(DP_MBITS) -#define DP_SIGN_BIT DP_MBIT(63) - -#define SP_MBIT(x) ((u32)1 << (x)) -#define SP_HIDDEN_BIT SP_MBIT(SP_MBITS) -#define SP_SIGN_BIT SP_MBIT(31) - - -#define SPSIGN(sp) (sp.parts.sign) -#define SPBEXP(sp) (sp.parts.bexp) -#define SPMANT(sp) (sp.parts.mant) - -#define DPSIGN(dp) (dp.parts.sign) -#define DPBEXP(dp) (dp.parts.bexp) -#define DPMANT(dp) (dp.parts.mant) - #define CLPAIR(x, y) ((x)*6+(y)) -#define CLEARCX \ - (ieee754_csr.cx = 0) - -#define SETCX(x) \ - (ieee754_csr.cx |= (x), ieee754_csr.sx |= (x)) +static inline void ieee754_clearcx(void) +{ + ieee754_csr.cx = 0; +} -#define SETANDTESTCX(x) \ - (SETCX(x), ieee754_csr.mx & (x)) +static inline void ieee754_setcx(const unsigned int flags) +{ + ieee754_csr.cx |= flags; + ieee754_csr.sx |= flags; +} -#define TSTX() \ - (ieee754_csr.cx & ieee754_csr.mx) +static inline int ieee754_setandtestcx(const unsigned int x) +{ + ieee754_setcx(x); + return ieee754_csr.mx & x; +} #define COMPXSP \ - unsigned xm; int xe; int xs __maybe_unused; int xc + unsigned xm; int xe; int xs __maybe_unused; int xc #define COMPYSP \ - unsigned ym; int ye; int ys; int yc - -#define EXPLODESP(v, vc, vs, ve, vm) \ -{\ - vs = SPSIGN(v);\ - ve = SPBEXP(v);\ - vm = SPMANT(v);\ - if(ve == SP_EMAX+1+SP_EBIAS){\ - if(vm == 0)\ - vc = IEEE754_CLASS_INF;\ - else if(vm & SP_MBIT(SP_MBITS-1)) \ - vc = IEEE754_CLASS_SNAN;\ - else \ - vc = IEEE754_CLASS_QNAN;\ - } else if(ve == SP_EMIN-1+SP_EBIAS) {\ - if(vm) {\ - ve = SP_EMIN;\ - vc = IEEE754_CLASS_DNORM;\ - } else\ - vc = IEEE754_CLASS_ZERO;\ - } else {\ - ve -= SP_EBIAS;\ - vm |= SP_HIDDEN_BIT;\ - vc = IEEE754_CLASS_NORM;\ - }\ + unsigned ym; int ye; int ys; int yc + +#define EXPLODESP(v, vc, vs, ve, vm) \ +{ \ + vs = SPSIGN(v); \ + ve = SPBEXP(v); \ + vm = SPMANT(v); \ + if (ve == SP_EMAX+1+SP_EBIAS) { \ + if (vm == 0) \ + vc = IEEE754_CLASS_INF; \ + else if (vm & SP_MBIT(SP_FBITS-1)) \ + vc = IEEE754_CLASS_SNAN; \ + else \ + vc = IEEE754_CLASS_QNAN; \ + } else if (ve == SP_EMIN-1+SP_EBIAS) { \ + if (vm) { \ + ve = SP_EMIN; \ + vc = IEEE754_CLASS_DNORM; \ + } else \ + vc = IEEE754_CLASS_ZERO; \ + } else { \ + ve -= SP_EBIAS; \ + vm |= SP_HIDDEN_BIT; \ + vc = IEEE754_CLASS_NORM; \ + } \ } #define EXPLODEXSP EXPLODESP(x, xc, xs, xe, xm) #define EXPLODEYSP EXPLODESP(y, yc, ys, ye, ym) #define COMPXDP \ -u64 xm; int xe; int xs __maybe_unused; int xc + u64 xm; int xe; int xs __maybe_unused; int xc #define COMPYDP \ -u64 ym; int ye; int ys; int yc - -#define EXPLODEDP(v, vc, vs, ve, vm) \ -{\ - vm = DPMANT(v);\ - vs = DPSIGN(v);\ - ve = DPBEXP(v);\ - if(ve == DP_EMAX+1+DP_EBIAS){\ - if(vm == 0)\ - vc = IEEE754_CLASS_INF;\ - else if(vm & DP_MBIT(DP_MBITS-1)) \ - vc = IEEE754_CLASS_SNAN;\ - else \ - vc = IEEE754_CLASS_QNAN;\ - } else if(ve == DP_EMIN-1+DP_EBIAS) {\ - if(vm) {\ - ve = DP_EMIN;\ - vc = IEEE754_CLASS_DNORM;\ - } else\ - vc = IEEE754_CLASS_ZERO;\ - } else {\ - ve -= DP_EBIAS;\ - vm |= DP_HIDDEN_BIT;\ - vc = IEEE754_CLASS_NORM;\ - }\ + u64 ym; int ye; int ys; int yc + +#define EXPLODEDP(v, vc, vs, ve, vm) \ +{ \ + vm = DPMANT(v); \ + vs = DPSIGN(v); \ + ve = DPBEXP(v); \ + if (ve == DP_EMAX+1+DP_EBIAS) { \ + if (vm == 0) \ + vc = IEEE754_CLASS_INF; \ + else if (vm & DP_MBIT(DP_FBITS-1)) \ + vc = IEEE754_CLASS_SNAN; \ + else \ + vc = IEEE754_CLASS_QNAN; \ + } else if (ve == DP_EMIN-1+DP_EBIAS) { \ + if (vm) { \ + ve = DP_EMIN; \ + vc = IEEE754_CLASS_DNORM; \ + } else \ + vc = IEEE754_CLASS_ZERO; \ + } else { \ + ve -= DP_EBIAS; \ + vm |= DP_HIDDEN_BIT; \ + vc = IEEE754_CLASS_NORM; \ + } \ } #define EXPLODEXDP EXPLODEDP(x, xc, xs, xe, xm) #define EXPLODEYDP EXPLODEDP(y, yc, ys, ye, ym) -#define FLUSHDP(v, vc, vs, ve, vm) \ - if(vc==IEEE754_CLASS_DNORM) {\ - if(ieee754_csr.nod) {\ - SETCX(IEEE754_INEXACT);\ - vc = IEEE754_CLASS_ZERO;\ - ve = DP_EMIN-1+DP_EBIAS;\ - vm = 0;\ - v = ieee754dp_zero(vs);\ - }\ +#define FLUSHDP(v, vc, vs, ve, vm) \ + if (vc==IEEE754_CLASS_DNORM) { \ + if (ieee754_csr.nod) { \ + ieee754_setcx(IEEE754_INEXACT); \ + vc = IEEE754_CLASS_ZERO; \ + ve = DP_EMIN-1+DP_EBIAS; \ + vm = 0; \ + v = ieee754dp_zero(vs); \ + } \ } -#define FLUSHSP(v, vc, vs, ve, vm) \ - if(vc==IEEE754_CLASS_DNORM) {\ - if(ieee754_csr.nod) {\ - SETCX(IEEE754_INEXACT);\ - vc = IEEE754_CLASS_ZERO;\ - ve = SP_EMIN-1+SP_EBIAS;\ - vm = 0;\ - v = ieee754sp_zero(vs);\ - }\ +#define FLUSHSP(v, vc, vs, ve, vm) \ + if (vc==IEEE754_CLASS_DNORM) { \ + if (ieee754_csr.nod) { \ + ieee754_setcx(IEEE754_INEXACT); \ + vc = IEEE754_CLASS_ZERO; \ + ve = SP_EMIN-1+SP_EBIAS; \ + vm = 0; \ + v = ieee754sp_zero(vs); \ + } \ } #define FLUSHXDP FLUSHDP(x, xc, xs, xe, xm) #define FLUSHYDP FLUSHDP(y, yc, ys, ye, ym) #define FLUSHXSP FLUSHSP(x, xc, xs, xe, xm) #define FLUSHYSP FLUSHSP(y, yc, ys, ye, ym) + +#endif /* __IEEE754INT_H */ diff --git a/arch/mips/math-emu/ieee754m.c b/arch/mips/math-emu/ieee754m.c deleted file mode 100644 index 24190f3c9dd6..000000000000 --- a/arch/mips/math-emu/ieee754m.c +++ /dev/null @@ -1,55 +0,0 @@ -/* - * floor, trunc, ceil - */ -/* - * MIPS floating point support - * Copyright (C) 1994-2000 Algorithmics Ltd. - * - * ######################################################################## - * - * This program is free software; you can distribute it and/or modify it - * under the terms of the GNU General Public License (Version 2) as - * published by the Free Software Foundation. - * - * This program is distributed in the hope it will be useful, but WITHOUT - * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or - * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License - * for more details. - * - * You should have received a copy of the GNU General Public License along - * with this program; if not, write to the Free Software Foundation, Inc., - * 59 Temple Place - Suite 330, Boston MA 02111-1307, USA. - * - * ######################################################################## - */ - - -#include "ieee754.h" - -ieee754dp ieee754dp_floor(ieee754dp x) -{ - ieee754dp i; - - if (ieee754dp_lt(ieee754dp_modf(x, &i), ieee754dp_zero(0))) - return ieee754dp_sub(i, ieee754dp_one(0)); - else - return i; -} - -ieee754dp ieee754dp_ceil(ieee754dp x) -{ - ieee754dp i; - - if (ieee754dp_gt(ieee754dp_modf(x, &i), ieee754dp_zero(0))) - return ieee754dp_add(i, ieee754dp_one(0)); - else - return i; -} - -ieee754dp ieee754dp_trunc(ieee754dp x) -{ - ieee754dp i; - - (void) ieee754dp_modf(x, &i); - return i; -} diff --git a/arch/mips/math-emu/ieee754sp.c b/arch/mips/math-emu/ieee754sp.c index 15d1e36cfe64..d348efe91445 100644 --- a/arch/mips/math-emu/ieee754sp.c +++ b/arch/mips/math-emu/ieee754sp.c @@ -5,8 +5,6 @@ * MIPS floating point support * Copyright (C) 1994-2000 Algorithmics Ltd. * - * ######################################################################## - * * This program is free software; you can distribute it and/or modify it * under the terms of the GNU General Public License (Version 2) as * published by the Free Software Foundation. @@ -18,105 +16,68 @@ * * You should have received a copy of the GNU General Public License along * with this program; if not, write to the Free Software Foundation, Inc., - * 59 Temple Place - Suite 330, Boston MA 02111-1307, USA. - * - * ######################################################################## + * 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. */ +#include <linux/compiler.h> #include "ieee754sp.h" -int ieee754sp_class(ieee754sp x) +int ieee754sp_class(union ieee754sp x) { COMPXSP; EXPLODEXSP; return xc; } -int ieee754sp_isnan(ieee754sp x) +int ieee754sp_isnan(union ieee754sp x) { return ieee754sp_class(x) >= IEEE754_CLASS_SNAN; } -int ieee754sp_issnan(ieee754sp x) +static inline int ieee754sp_issnan(union ieee754sp x) { assert(ieee754sp_isnan(x)); - return (SPMANT(x) & SP_MBIT(SP_MBITS-1)); + return (SPMANT(x) & SP_MBIT(SP_FBITS-1)); } -ieee754sp ieee754sp_xcpt(ieee754sp r, const char *op, ...) -{ - struct ieee754xctx ax; - - if (!TSTX()) - return r; - - ax.op = op; - ax.rt = IEEE754_RT_SP; - ax.rv.sp = r; - va_start(ax.ap, op); - ieee754_xcpt(&ax); - va_end(ax.ap); - return ax.rv.sp; -} - -ieee754sp ieee754sp_nanxcpt(ieee754sp r, const char *op, ...) +union ieee754sp __cold ieee754sp_nanxcpt(union ieee754sp r) { - struct ieee754xctx ax; - assert(ieee754sp_isnan(r)); if (!ieee754sp_issnan(r)) /* QNAN does not cause invalid op !! */ return r; - if (!SETANDTESTCX(IEEE754_INVALID_OPERATION)) { + if (!ieee754_setandtestcx(IEEE754_INVALID_OPERATION)) { /* not enabled convert to a quiet NaN */ - SPMANT(r) &= (~SP_MBIT(SP_MBITS-1)); + SPMANT(r) &= (~SP_MBIT(SP_FBITS-1)); if (ieee754sp_isnan(r)) return r; else return ieee754sp_indef(); } - ax.op = op; - ax.rt = 0; - ax.rv.sp = r; - va_start(ax.ap, op); - ieee754_xcpt(&ax); - va_end(ax.ap); - return ax.rv.sp; -} - -ieee754sp ieee754sp_bestnan(ieee754sp x, ieee754sp y) -{ - assert(ieee754sp_isnan(x)); - assert(ieee754sp_isnan(y)); - - if (SPMANT(x) > SPMANT(y)) - return x; - else - return y; + return r; } - -static unsigned get_rounding(int sn, unsigned xm) +static unsigned ieee754sp_get_rounding(int sn, unsigned xm) { /* inexact must round of 3 bits */ if (xm & (SP_MBIT(3) - 1)) { switch (ieee754_csr.rm) { - case IEEE754_RZ: + case FPU_CSR_RZ: break; - case IEEE754_RN: + case FPU_CSR_RN: xm += 0x3 + ((xm >> 3) & 1); /* xm += (xm&0x8)?0x4:0x3 */ break; - case IEEE754_RU: /* toward +Infinity */ + case FPU_CSR_RU: /* toward +Infinity */ if (!sn) /* ?? */ xm += 0x8; break; - case IEEE754_RD: /* toward -Infinity */ + case FPU_CSR_RD: /* toward -Infinity */ if (sn) /* ?? */ xm += 0x8; break; @@ -131,11 +92,11 @@ static unsigned get_rounding(int sn, unsigned xm) * xe is an unbiased exponent * xm is 3bit extended precision value. */ -ieee754sp ieee754sp_format(int sn, int xe, unsigned xm) +union ieee754sp ieee754sp_format(int sn, int xe, unsigned xm) { assert(xm); /* we don't gen exact zeros (probably should) */ - assert((xm >> (SP_MBITS + 1 + 3)) == 0); /* no execess */ + assert((xm >> (SP_FBITS + 1 + 3)) == 0); /* no execess */ assert(xm & (SP_HIDDEN_BIT << 3)); if (xe < SP_EMIN) { @@ -143,38 +104,37 @@ ieee754sp ieee754sp_format(int sn, int xe, unsigned xm) int es = SP_EMIN - xe; if (ieee754_csr.nod) { - SETCX(IEEE754_UNDERFLOW); - SETCX(IEEE754_INEXACT); + ieee754_setcx(IEEE754_UNDERFLOW); + ieee754_setcx(IEEE754_INEXACT); switch(ieee754_csr.rm) { - case IEEE754_RN: - case IEEE754_RZ: + case FPU_CSR_RN: + case FPU_CSR_RZ: return ieee754sp_zero(sn); - case IEEE754_RU: /* toward +Infinity */ - if(sn == 0) + case FPU_CSR_RU: /* toward +Infinity */ + if (sn == 0) return ieee754sp_min(0); else return ieee754sp_zero(1); - case IEEE754_RD: /* toward -Infinity */ - if(sn == 0) + case FPU_CSR_RD: /* toward -Infinity */ + if (sn == 0) return ieee754sp_zero(0); else return ieee754sp_min(1); } } - if (xe == SP_EMIN - 1 - && get_rounding(sn, xm) >> (SP_MBITS + 1 + 3)) + if (xe == SP_EMIN - 1 && + ieee754sp_get_rounding(sn, xm) >> (SP_FBITS + 1 + 3)) { /* Not tiny after rounding */ - SETCX(IEEE754_INEXACT); - xm = get_rounding(sn, xm); + ieee754_setcx(IEEE754_INEXACT); + xm = ieee754sp_get_rounding(sn, xm); xm >>= 1; /* Clear grs bits */ xm &= ~(SP_MBIT(3) - 1); xe++; - } - else { + } else { /* sticky right shift es bits */ SPXSRSXn(es); @@ -183,17 +143,17 @@ ieee754sp ieee754sp_format(int sn, int xe, unsigned xm) } } if (xm & (SP_MBIT(3) - 1)) { - SETCX(IEEE754_INEXACT); + ieee754_setcx(IEEE754_INEXACT); if ((xm & (SP_HIDDEN_BIT << 3)) == 0) { - SETCX(IEEE754_UNDERFLOW); + ieee754_setcx(IEEE754_UNDERFLOW); } /* inexact must round of 3 bits */ - xm = get_rounding(sn, xm); + xm = ieee754sp_get_rounding(sn, xm); /* adjust exponent for rounding add overflowing */ - if (xm >> (SP_MBITS + 1 + 3)) { + if (xm >> (SP_FBITS + 1 + 3)) { /* add causes mantissa overflow */ xm >>= 1; xe++; @@ -202,24 +162,24 @@ ieee754sp ieee754sp_format(int sn, int xe, unsigned xm) /* strip grs bits */ xm >>= 3; - assert((xm >> (SP_MBITS + 1)) == 0); /* no execess */ + assert((xm >> (SP_FBITS + 1)) == 0); /* no execess */ assert(xe >= SP_EMIN); if (xe > SP_EMAX) { - SETCX(IEEE754_OVERFLOW); - SETCX(IEEE754_INEXACT); + ieee754_setcx(IEEE754_OVERFLOW); + ieee754_setcx(IEEE754_INEXACT); /* -O can be table indexed by (rm,sn) */ switch (ieee754_csr.rm) { - case IEEE754_RN: + case FPU_CSR_RN: return ieee754sp_inf(sn); - case IEEE754_RZ: + case FPU_CSR_RZ: return ieee754sp_max(sn); - case IEEE754_RU: /* toward +Infinity */ + case FPU_CSR_RU: /* toward +Infinity */ if (sn == 0) return ieee754sp_inf(0); else return ieee754sp_max(1); - case IEEE754_RD: /* toward -Infinity */ + case FPU_CSR_RD: /* toward -Infinity */ if (sn == 0) return ieee754sp_max(0); else @@ -232,10 +192,10 @@ ieee754sp ieee754sp_format(int sn, int xe, unsigned xm) /* we underflow (tiny/zero) */ assert(xe == SP_EMIN); if (ieee754_csr.mx & IEEE754_UNDERFLOW) - SETCX(IEEE754_UNDERFLOW); + ieee754_setcx(IEEE754_UNDERFLOW); return buildsp(sn, SP_EMIN - 1 + SP_EBIAS, xm); } else { - assert((xm >> (SP_MBITS + 1)) == 0); /* no execess */ + assert((xm >> (SP_FBITS + 1)) == 0); /* no execess */ assert(xm & SP_HIDDEN_BIT); return buildsp(sn, xe + SP_EBIAS, xm & ~SP_HIDDEN_BIT); diff --git a/arch/mips/math-emu/ieee754sp.h b/arch/mips/math-emu/ieee754sp.h index 754fd54649b5..ad268e332318 100644 --- a/arch/mips/math-emu/ieee754sp.h +++ b/arch/mips/math-emu/ieee754sp.h @@ -6,8 +6,6 @@ * MIPS floating point support * Copyright (C) 1994-2000 Algorithmics Ltd. * - * ######################################################################## - * * This program is free software; you can distribute it and/or modify it * under the terms of the GNU General Public License (Version 2) as * published by the Free Software Foundation. @@ -19,70 +17,71 @@ * * You should have received a copy of the GNU General Public License along * with this program; if not, write to the Free Software Foundation, Inc., - * 59 Temple Place - Suite 330, Boston MA 02111-1307, USA. - * - * ######################################################################## + * 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. */ +#include <linux/compiler.h> #include "ieee754int.h" #define assert(expr) ((void)0) +#define SP_EBIAS 127 +#define SP_EMIN (-126) +#define SP_EMAX 127 +#define SP_FBITS 23 +#define SP_MBITS 23 + +#define SP_MBIT(x) ((u32)1 << (x)) +#define SP_HIDDEN_BIT SP_MBIT(SP_FBITS) +#define SP_SIGN_BIT SP_MBIT(31) + +#define SPSIGN(sp) (sp.sign) +#define SPBEXP(sp) (sp.bexp) +#define SPMANT(sp) (sp.mant) + +static inline int ieee754sp_finite(union ieee754sp x) +{ + return SPBEXP(x) != SP_EMAX + 1 + SP_EBIAS; +} + /* 3bit extended single precision sticky right shift */ -#define SPXSRSXn(rs) \ - (xe += rs, \ - xm = (rs > (SP_MBITS+3))?1:((xm) >> (rs)) | ((xm) << (32-(rs)) != 0)) +#define SPXSRSXn(rs) \ + (xe += rs, \ + xm = (rs > (SP_FBITS+3))?1:((xm) >> (rs)) | ((xm) << (32-(rs)) != 0)) #define SPXSRSX1() \ - (xe++, (xm = (xm >> 1) | (xm & 1))) + (xe++, (xm = (xm >> 1) | (xm & 1))) -#define SPXSRSYn(rs) \ - (ye+=rs, \ - ym = (rs > (SP_MBITS+3))?1:((ym) >> (rs)) | ((ym) << (32-(rs)) != 0)) +#define SPXSRSYn(rs) \ + (ye+=rs, \ + ym = (rs > (SP_FBITS+3))?1:((ym) >> (rs)) | ((ym) << (32-(rs)) != 0)) #define SPXSRSY1() \ - (ye++, (ym = (ym >> 1) | (ym & 1))) + (ye++, (ym = (ym >> 1) | (ym & 1))) /* convert denormal to normalized with extended exponent */ #define SPDNORMx(m,e) \ - while( (m >> SP_MBITS) == 0) { m <<= 1; e--; } + while ((m >> SP_FBITS) == 0) { m <<= 1; e--; } #define SPDNORMX SPDNORMx(xm, xe) #define SPDNORMY SPDNORMx(ym, ye) -static inline ieee754sp buildsp(int s, int bx, unsigned m) +static inline union ieee754sp buildsp(int s, int bx, unsigned m) { - ieee754sp r; + union ieee754sp r; assert((s) == 0 || (s) == 1); assert((bx) >= SP_EMIN - 1 + SP_EBIAS && (bx) <= SP_EMAX + 1 + SP_EBIAS); - assert(((m) >> SP_MBITS) == 0); + assert(((m) >> SP_FBITS) == 0); - r.parts.sign = s; - r.parts.bexp = bx; - r.parts.mant = m; + r.sign = s; + r.bexp = bx; + r.mant = m; return r; } -extern int ieee754sp_isnan(ieee754sp); -extern int ieee754sp_issnan(ieee754sp); -extern int ieee754si_xcpt(int, const char *, ...); -extern s64 ieee754di_xcpt(s64, const char *, ...); -extern ieee754sp ieee754sp_xcpt(ieee754sp, const char *, ...); -extern ieee754sp ieee754sp_nanxcpt(ieee754sp, const char *, ...); -extern ieee754sp ieee754sp_bestnan(ieee754sp, ieee754sp); -extern ieee754sp ieee754sp_format(int, int, unsigned); - - -#define SPNORMRET2(s, e, m, name, a0, a1) \ -{ \ - ieee754sp V = ieee754sp_format(s, e, m); \ - if(TSTX()) \ - return ieee754sp_xcpt(V, name, a0, a1); \ - else \ - return V; \ -} - -#define SPNORMRET1(s, e, m, name, a0) SPNORMRET2(s, e, m, name, a0, a0) +extern int ieee754sp_isnan(union ieee754sp); +extern union ieee754sp __cold ieee754sp_nanxcpt(union ieee754sp); +extern union ieee754sp ieee754sp_format(int, int, unsigned); diff --git a/arch/mips/math-emu/ieee754xcpt.c b/arch/mips/math-emu/ieee754xcpt.c deleted file mode 100644 index 967167116ae8..000000000000 --- a/arch/mips/math-emu/ieee754xcpt.c +++ /dev/null @@ -1,47 +0,0 @@ -/* - * MIPS floating point support - * Copyright (C) 1994-2000 Algorithmics Ltd. - * - * ######################################################################## - * - * This program is free software; you can distribute it and/or modify it - * under the terms of the GNU General Public License (Version 2) as - * published by the Free Software Foundation. - * - * This program is distributed in the hope it will be useful, but WITHOUT - * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or - * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License - * for more details. - * - * You should have received a copy of the GNU General Public License along - * with this program; if not, write to the Free Software Foundation, Inc., - * 59 Temple Place - Suite 330, Boston MA 02111-1307, USA. - * - * ######################################################################## - */ - -/************************************************************************** - * Nov 7, 2000 - * Added preprocessor hacks to map to Linux kernel diagnostics. - * - * Kevin D. Kissell, kevink@mips.com and Carsten Langgaard, carstenl@mips.com - * Copyright (C) 2000 MIPS Technologies, Inc. All rights reserved. - *************************************************************************/ - -#include <linux/kernel.h> -#include "ieee754.h" - -/* - * Very naff exception handler (you can plug in your own and - * override this). - */ - -static const char *const rtnames[] = { - "sp", "dp", "xp", "si", "di" -}; - -void ieee754_xcpt(struct ieee754xctx *xcp) -{ - printk(KERN_DEBUG "floating point exception in \"%s\", type=%s\n", - xcp->op, rtnames[xcp->rt]); -} diff --git a/arch/mips/math-emu/kernel_linkage.c b/arch/mips/math-emu/kernel_linkage.c deleted file mode 100644 index eb58a85b3157..000000000000 --- a/arch/mips/math-emu/kernel_linkage.c +++ /dev/null @@ -1,45 +0,0 @@ -/* - * Kevin D. Kissell, kevink@mips and Carsten Langgaard, carstenl@mips.com - * Copyright (C) 2000 MIPS Technologies, Inc. All rights reserved. - * - * This program is free software; you can distribute it and/or modify it - * under the terms of the GNU General Public License (Version 2) as - * published by the Free Software Foundation. - * - * This program is distributed in the hope it will be useful, but WITHOUT - * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or - * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License - * for more details. - * - * You should have received a copy of the GNU General Public License along - * with this program; if not, write to the Free Software Foundation, Inc., - * 59 Temple Place - Suite 330, Boston MA 02111-1307, USA. - * - * Routines corresponding to Linux kernel FP context - * manipulation primitives for the Algorithmics MIPS - * FPU Emulator - */ -#include <linux/sched.h> -#include <asm/processor.h> -#include <asm/signal.h> -#include <asm/uaccess.h> - -#include <asm/fpu.h> -#include <asm/fpu_emulator.h> - -#define SIGNALLING_NAN 0x7ff800007ff80000LL - -void fpu_emulator_init_fpu(void) -{ - static int first = 1; - int i; - - if (first) { - first = 0; - printk("Algorithmics/MIPS FPU Emulator v1.5\n"); - } - - current->thread.fpu.fcr31 = 0; - for (i = 0; i < 32; i++) - set_fpr64(¤t->thread.fpu.fpr[i], 0, SIGNALLING_NAN); -} diff --git a/arch/mips/math-emu/me-debugfs.c b/arch/mips/math-emu/me-debugfs.c new file mode 100644 index 000000000000..becdd63e14a9 --- /dev/null +++ b/arch/mips/math-emu/me-debugfs.c @@ -0,0 +1,67 @@ +#include <linux/cpumask.h> +#include <linux/debugfs.h> +#include <linux/fs.h> +#include <linux/init.h> +#include <linux/percpu.h> +#include <linux/types.h> +#include <asm/fpu_emulator.h> +#include <asm/local.h> + +DEFINE_PER_CPU(struct mips_fpu_emulator_stats, fpuemustats); + +static int fpuemu_stat_get(void *data, u64 *val) +{ + int cpu; + unsigned long sum = 0; + + for_each_online_cpu(cpu) { + struct mips_fpu_emulator_stats *ps; + local_t *pv; + + ps = &per_cpu(fpuemustats, cpu); + pv = (void *)ps + (unsigned long)data; + sum += local_read(pv); + } + *val = sum; + return 0; +} +DEFINE_SIMPLE_ATTRIBUTE(fops_fpuemu_stat, fpuemu_stat_get, NULL, "%llu\n"); + +extern struct dentry *mips_debugfs_dir; +static int __init debugfs_fpuemu(void) +{ + struct dentry *d, *dir; + + if (!mips_debugfs_dir) + return -ENODEV; + dir = debugfs_create_dir("fpuemustats", mips_debugfs_dir); + if (!dir) + return -ENOMEM; + +#define FPU_EMU_STAT_OFFSET(m) \ + offsetof(struct mips_fpu_emulator_stats, m) + +#define FPU_STAT_CREATE(m) \ +do { \ + d = debugfs_create_file(#m , S_IRUGO, dir, \ + (void *)FPU_EMU_STAT_OFFSET(m), \ + &fops_fpuemu_stat); \ + if (!d) \ + return -ENOMEM; \ +} while (0) + + FPU_STAT_CREATE(emulated); + FPU_STAT_CREATE(loads); + FPU_STAT_CREATE(stores); + FPU_STAT_CREATE(cp1ops); + FPU_STAT_CREATE(cp1xops); + FPU_STAT_CREATE(errors); + FPU_STAT_CREATE(ieee754_inexact); + FPU_STAT_CREATE(ieee754_underflow); + FPU_STAT_CREATE(ieee754_overflow); + FPU_STAT_CREATE(ieee754_zerodiv); + FPU_STAT_CREATE(ieee754_invalidop); + + return 0; +} +__initcall(debugfs_fpuemu); diff --git a/arch/mips/math-emu/sp_add.c b/arch/mips/math-emu/sp_add.c index c446e64637e2..2d84d460cb67 100644 --- a/arch/mips/math-emu/sp_add.c +++ b/arch/mips/math-emu/sp_add.c @@ -5,8 +5,6 @@ * MIPS floating point support * Copyright (C) 1994-2000 Algorithmics Ltd. * - * ######################################################################## - * * This program is free software; you can distribute it and/or modify it * under the terms of the GNU General Public License (Version 2) as * published by the Free Software Foundation. @@ -18,23 +16,22 @@ * * You should have received a copy of the GNU General Public License along * with this program; if not, write to the Free Software Foundation, Inc., - * 59 Temple Place - Suite 330, Boston MA 02111-1307, USA. - * - * ######################################################################## + * 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. */ - #include "ieee754sp.h" -ieee754sp ieee754sp_add(ieee754sp x, ieee754sp y) +union ieee754sp ieee754sp_add(union ieee754sp x, union ieee754sp y) { + int s; + COMPXSP; COMPYSP; EXPLODEXSP; EXPLODEYSP; - CLEARCX; + ieee754_clearcx(); FLUSHXSP; FLUSHYSP; @@ -51,8 +48,8 @@ ieee754sp ieee754sp_add(ieee754sp x, ieee754sp y) case CLPAIR(IEEE754_CLASS_SNAN, IEEE754_CLASS_NORM): case CLPAIR(IEEE754_CLASS_SNAN, IEEE754_CLASS_DNORM): case CLPAIR(IEEE754_CLASS_SNAN, IEEE754_CLASS_INF): - SETCX(IEEE754_INVALID_OPERATION); - return ieee754sp_nanxcpt(ieee754sp_indef(), "add", x, y); + ieee754_setcx(IEEE754_INVALID_OPERATION); + return ieee754sp_nanxcpt(ieee754sp_indef()); case CLPAIR(IEEE754_CLASS_ZERO, IEEE754_CLASS_QNAN): case CLPAIR(IEEE754_CLASS_NORM, IEEE754_CLASS_QNAN): @@ -68,14 +65,14 @@ ieee754sp ieee754sp_add(ieee754sp x, ieee754sp y) return x; - /* Infinity handling - */ - + /* + * Infinity handling + */ case CLPAIR(IEEE754_CLASS_INF, IEEE754_CLASS_INF): if (xs == ys) return x; - SETCX(IEEE754_INVALID_OPERATION); - return ieee754sp_xcpt(ieee754sp_indef(), "add", x, y); + ieee754_setcx(IEEE754_INVALID_OPERATION); + return ieee754sp_indef(); case CLPAIR(IEEE754_CLASS_NORM, IEEE754_CLASS_INF): case CLPAIR(IEEE754_CLASS_ZERO, IEEE754_CLASS_INF): @@ -87,15 +84,14 @@ ieee754sp ieee754sp_add(ieee754sp x, ieee754sp y) case CLPAIR(IEEE754_CLASS_INF, IEEE754_CLASS_DNORM): return x; - /* Zero handling - */ - + /* + * Zero handling + */ case CLPAIR(IEEE754_CLASS_ZERO, IEEE754_CLASS_ZERO): if (xs == ys) return x; else - return ieee754sp_zero(ieee754_csr.rm == - IEEE754_RD); + return ieee754sp_zero(ieee754_csr.rm == FPU_CSR_RD); case CLPAIR(IEEE754_CLASS_NORM, IEEE754_CLASS_ZERO): case CLPAIR(IEEE754_CLASS_DNORM, IEEE754_CLASS_ZERO): @@ -108,6 +104,8 @@ ieee754sp ieee754sp_add(ieee754sp x, ieee754sp y) case CLPAIR(IEEE754_CLASS_DNORM, IEEE754_CLASS_DNORM): SPDNORMX; + /* FALL THROUGH */ + case CLPAIR(IEEE754_CLASS_NORM, IEEE754_CLASS_DNORM): SPDNORMY; break; @@ -122,33 +120,38 @@ ieee754sp ieee754sp_add(ieee754sp x, ieee754sp y) assert(xm & SP_HIDDEN_BIT); assert(ym & SP_HIDDEN_BIT); - /* provide guard,round and stick bit space */ + /* + * Provide guard, round and stick bit space. + */ xm <<= 3; ym <<= 3; if (xe > ye) { - /* have to shift y fraction right to align + /* + * Have to shift y fraction right to align. */ - int s = xe - ye; + s = xe - ye; SPXSRSYn(s); } else if (ye > xe) { - /* have to shift x fraction right to align + /* + * Have to shift x fraction right to align. */ - int s = ye - xe; + s = ye - xe; SPXSRSXn(s); } assert(xe == ye); assert(xe <= SP_EMAX); if (xs == ys) { - /* generate 28 bit result of adding two 27 bit numbers - * leaving result in xm,xs,xe + /* + * Generate 28 bit result of adding two 27 bit numbers + * leaving result in xm, xs and xe. */ xm = xm + ym; xe = xe; xs = xs; - if (xm >> (SP_MBITS + 1 + 3)) { /* carry out */ + if (xm >> (SP_FBITS + 1 + 3)) { /* carry out */ SPXSRSX1(); } } else { @@ -162,15 +165,16 @@ ieee754sp ieee754sp_add(ieee754sp x, ieee754sp y) xs = ys; } if (xm == 0) - return ieee754sp_zero(ieee754_csr.rm == - IEEE754_RD); + return ieee754sp_zero(ieee754_csr.rm == FPU_CSR_RD); - /* normalize in extended single precision */ - while ((xm >> (SP_MBITS + 3)) == 0) { + /* + * Normalize in extended single precision + */ + while ((xm >> (SP_FBITS + 3)) == 0) { xm <<= 1; xe--; } - } - SPNORMRET2(xs, xe, xm, "add", x, y); + + return ieee754sp_format(xs, xe, xm); } diff --git a/arch/mips/math-emu/sp_cmp.c b/arch/mips/math-emu/sp_cmp.c index 716cf37e2465..addbccb2f556 100644 --- a/arch/mips/math-emu/sp_cmp.c +++ b/arch/mips/math-emu/sp_cmp.c @@ -5,8 +5,6 @@ * MIPS floating point support * Copyright (C) 1994-2000 Algorithmics Ltd. * - * ######################################################################## - * * This program is free software; you can distribute it and/or modify it * under the terms of the GNU General Public License (Version 2) as * published by the Free Software Foundation. @@ -18,16 +16,16 @@ * * You should have received a copy of the GNU General Public License along * with this program; if not, write to the Free Software Foundation, Inc., - * 59 Temple Place - Suite 330, Boston MA 02111-1307, USA. - * - * ######################################################################## + * 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. */ - #include "ieee754sp.h" -int ieee754sp_cmp(ieee754sp x, ieee754sp y, int cmp, int sig) +int ieee754sp_cmp(union ieee754sp x, union ieee754sp y, int cmp, int sig) { + int vx; + int vy; + COMPXSP; COMPYSP; @@ -35,21 +33,21 @@ int ieee754sp_cmp(ieee754sp x, ieee754sp y, int cmp, int sig) EXPLODEYSP; FLUSHXSP; FLUSHYSP; - CLEARCX; /* Even clear inexact flag here */ + ieee754_clearcx(); /* Even clear inexact flag here */ if (ieee754sp_isnan(x) || ieee754sp_isnan(y)) { if (sig || xc == IEEE754_CLASS_SNAN || yc == IEEE754_CLASS_SNAN) - SETCX(IEEE754_INVALID_OPERATION); + ieee754_setcx(IEEE754_INVALID_OPERATION); if (cmp & IEEE754_CUN) return 1; if (cmp & (IEEE754_CLT | IEEE754_CGT)) { - if (sig && SETANDTESTCX(IEEE754_INVALID_OPERATION)) - return ieee754si_xcpt(0, "fcmpf", x); + if (sig && ieee754_setandtestcx(IEEE754_INVALID_OPERATION)) + return 0; } return 0; } else { - int vx = x.bits; - int vy = y.bits; + vx = x.bits; + vy = y.bits; if (vx < 0) vx = -vx ^ SP_SIGN_BIT; diff --git a/arch/mips/math-emu/sp_div.c b/arch/mips/math-emu/sp_div.c index d7747928c954..721f317aa877 100644 --- a/arch/mips/math-emu/sp_div.c +++ b/arch/mips/math-emu/sp_div.c @@ -5,8 +5,6 @@ * MIPS floating point support * Copyright (C) 1994-2000 Algorithmics Ltd. * - * ######################################################################## - * * This program is free software; you can distribute it and/or modify it * under the terms of the GNU General Public License (Version 2) as * published by the Free Software Foundation. @@ -18,23 +16,24 @@ * * You should have received a copy of the GNU General Public License along * with this program; if not, write to the Free Software Foundation, Inc., - * 59 Temple Place - Suite 330, Boston MA 02111-1307, USA. - * - * ######################################################################## + * 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. */ - #include "ieee754sp.h" -ieee754sp ieee754sp_div(ieee754sp x, ieee754sp y) +union ieee754sp ieee754sp_div(union ieee754sp x, union ieee754sp y) { + unsigned rm; + int re; + unsigned bm; + COMPXSP; COMPYSP; EXPLODEXSP; EXPLODEYSP; - CLEARCX; + ieee754_clearcx(); FLUSHXSP; FLUSHYSP; @@ -51,8 +50,8 @@ ieee754sp ieee754sp_div(ieee754sp x, ieee754sp y) case CLPAIR(IEEE754_CLASS_SNAN, IEEE754_CLASS_NORM): case CLPAIR(IEEE754_CLASS_SNAN, IEEE754_CLASS_DNORM): case CLPAIR(IEEE754_CLASS_SNAN, IEEE754_CLASS_INF): - SETCX(IEEE754_INVALID_OPERATION); - return ieee754sp_nanxcpt(ieee754sp_indef(), "div", x, y); + ieee754_setcx(IEEE754_INVALID_OPERATION); + return ieee754sp_nanxcpt(ieee754sp_indef()); case CLPAIR(IEEE754_CLASS_ZERO, IEEE754_CLASS_QNAN): case CLPAIR(IEEE754_CLASS_NORM, IEEE754_CLASS_QNAN): @@ -68,12 +67,12 @@ ieee754sp ieee754sp_div(ieee754sp x, ieee754sp y) return x; - /* Infinity handling - */ - + /* + * Infinity handling + */ case CLPAIR(IEEE754_CLASS_INF, IEEE754_CLASS_INF): - SETCX(IEEE754_INVALID_OPERATION); - return ieee754sp_xcpt(ieee754sp_indef(), "div", x, y); + ieee754_setcx(IEEE754_INVALID_OPERATION); + return ieee754sp_indef(); case CLPAIR(IEEE754_CLASS_NORM, IEEE754_CLASS_INF): case CLPAIR(IEEE754_CLASS_ZERO, IEEE754_CLASS_INF): @@ -85,17 +84,17 @@ ieee754sp ieee754sp_div(ieee754sp x, ieee754sp y) case CLPAIR(IEEE754_CLASS_INF, IEEE754_CLASS_DNORM): return ieee754sp_inf(xs ^ ys); - /* Zero handling - */ - + /* + * Zero handling + */ case CLPAIR(IEEE754_CLASS_ZERO, IEEE754_CLASS_ZERO): - SETCX(IEEE754_INVALID_OPERATION); - return ieee754sp_xcpt(ieee754sp_indef(), "div", x, y); + ieee754_setcx(IEEE754_INVALID_OPERATION); + return ieee754sp_indef(); case CLPAIR(IEEE754_CLASS_NORM, IEEE754_CLASS_ZERO): case CLPAIR(IEEE754_CLASS_DNORM, IEEE754_CLASS_ZERO): - SETCX(IEEE754_ZERO_DIVIDE); - return ieee754sp_xcpt(ieee754sp_inf(xs ^ ys), "div", x, y); + ieee754_setcx(IEEE754_ZERO_DIVIDE); + return ieee754sp_inf(xs ^ ys); case CLPAIR(IEEE754_CLASS_ZERO, IEEE754_CLASS_NORM): case CLPAIR(IEEE754_CLASS_ZERO, IEEE754_CLASS_DNORM): @@ -122,35 +121,33 @@ ieee754sp ieee754sp_div(ieee754sp x, ieee754sp y) xm <<= 3; ym <<= 3; - { - /* now the dirty work */ - - unsigned rm = 0; - int re = xe - ye; - unsigned bm; - - for (bm = SP_MBIT(SP_MBITS + 2); bm; bm >>= 1) { - if (xm >= ym) { - xm -= ym; - rm |= bm; - if (xm == 0) - break; - } - xm <<= 1; - } - rm <<= 1; - if (xm) - rm |= 1; /* have remainder, set sticky */ + /* now the dirty work */ - assert(rm); + rm = 0; + re = xe - ye; - /* normalise rm to rounding precision ? - */ - while ((rm >> (SP_MBITS + 3)) == 0) { - rm <<= 1; - re--; + for (bm = SP_MBIT(SP_FBITS + 2); bm; bm >>= 1) { + if (xm >= ym) { + xm -= ym; + rm |= bm; + if (xm == 0) + break; } + xm <<= 1; + } + + rm <<= 1; + if (xm) + rm |= 1; /* have remainder, set sticky */ - SPNORMRET2(xs == ys ? 0 : 1, re, rm, "div", x, y); + assert(rm); + + /* normalise rm to rounding precision ? + */ + while ((rm >> (SP_FBITS + 3)) == 0) { + rm <<= 1; + re--; } + + return ieee754sp_format(xs == ys ? 0 : 1, re, rm); } diff --git a/arch/mips/math-emu/sp_fdp.c b/arch/mips/math-emu/sp_fdp.c index e1515aae0166..1b266fb16973 100644 --- a/arch/mips/math-emu/sp_fdp.c +++ b/arch/mips/math-emu/sp_fdp.c @@ -5,8 +5,6 @@ * MIPS floating point support * Copyright (C) 1994-2000 Algorithmics Ltd. * - * ######################################################################## - * * This program is free software; you can distribute it and/or modify it * under the terms of the GNU General Public License (Version 2) as * published by the Free Software Foundation. @@ -18,59 +16,61 @@ * * You should have received a copy of the GNU General Public License along * with this program; if not, write to the Free Software Foundation, Inc., - * 59 Temple Place - Suite 330, Boston MA 02111-1307, USA. - * - * ######################################################################## + * 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. */ - #include "ieee754sp.h" +#include "ieee754dp.h" -ieee754sp ieee754sp_fdp(ieee754dp x) +union ieee754sp ieee754sp_fdp(union ieee754dp x) { + u32 rm; + COMPXDP; - ieee754sp nan; + union ieee754sp nan; EXPLODEXDP; - CLEARCX; + ieee754_clearcx(); FLUSHXDP; switch (xc) { case IEEE754_CLASS_SNAN: - SETCX(IEEE754_INVALID_OPERATION); - return ieee754sp_nanxcpt(ieee754sp_indef(), "fdp"); + ieee754_setcx(IEEE754_INVALID_OPERATION); + return ieee754sp_nanxcpt(ieee754sp_indef()); + case IEEE754_CLASS_QNAN: nan = buildsp(xs, SP_EMAX + 1 + SP_EBIAS, (u32) - (xm >> (DP_MBITS - SP_MBITS))); + (xm >> (DP_FBITS - SP_FBITS))); if (!ieee754sp_isnan(nan)) nan = ieee754sp_indef(); - return ieee754sp_nanxcpt(nan, "fdp", x); + return ieee754sp_nanxcpt(nan); + case IEEE754_CLASS_INF: return ieee754sp_inf(xs); + case IEEE754_CLASS_ZERO: return ieee754sp_zero(xs); + case IEEE754_CLASS_DNORM: /* can't possibly be sp representable */ - SETCX(IEEE754_UNDERFLOW); - SETCX(IEEE754_INEXACT); - if ((ieee754_csr.rm == IEEE754_RU && !xs) || - (ieee754_csr.rm == IEEE754_RD && xs)) - return ieee754sp_xcpt(ieee754sp_mind(xs), "fdp", x); - return ieee754sp_xcpt(ieee754sp_zero(xs), "fdp", x); + ieee754_setcx(IEEE754_UNDERFLOW); + ieee754_setcx(IEEE754_INEXACT); + if ((ieee754_csr.rm == FPU_CSR_RU && !xs) || + (ieee754_csr.rm == FPU_CSR_RD && xs)) + return ieee754sp_mind(xs); + return ieee754sp_zero(xs); + case IEEE754_CLASS_NORM: break; } - { - u32 rm; - - /* convert from DP_MBITS to SP_MBITS+3 with sticky right shift - */ - rm = (xm >> (DP_MBITS - (SP_MBITS + 3))) | - ((xm << (64 - (DP_MBITS - (SP_MBITS + 3)))) != 0); + /* + * Convert from DP_FBITS to SP_FBITS+3 with sticky right shift. + */ + rm = (xm >> (DP_FBITS - (SP_FBITS + 3))) | + ((xm << (64 - (DP_FBITS - (SP_FBITS + 3)))) != 0); - SPNORMRET1(xs, xe, rm, "fdp", x); - } + return ieee754sp_format(xs, xe, rm); } diff --git a/arch/mips/math-emu/sp_fint.c b/arch/mips/math-emu/sp_fint.c index 9694d6c016cb..d5d8495b2cc4 100644 --- a/arch/mips/math-emu/sp_fint.c +++ b/arch/mips/math-emu/sp_fint.c @@ -5,8 +5,6 @@ * MIPS floating point support * Copyright (C) 1994-2000 Algorithmics Ltd. * - * ######################################################################## - * * This program is free software; you can distribute it and/or modify it * under the terms of the GNU General Public License (Version 2) as * published by the Free Software Foundation. @@ -18,21 +16,18 @@ * * You should have received a copy of the GNU General Public License along * with this program; if not, write to the Free Software Foundation, Inc., - * 59 Temple Place - Suite 330, Boston MA 02111-1307, USA. - * - * ######################################################################## + * 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. */ - #include "ieee754sp.h" -ieee754sp ieee754sp_fint(int x) +union ieee754sp ieee754sp_fint(int x) { unsigned xm; int xe; int xs; - CLEARCX; + ieee754_clearcx(); if (x == 0) return ieee754sp_zero(0); @@ -50,30 +45,21 @@ ieee754sp ieee754sp_fint(int x) } else { xm = x; } - xe = SP_MBITS + 3; + xe = SP_FBITS + 3; - if (xm >> (SP_MBITS + 1 + 3)) { + if (xm >> (SP_FBITS + 1 + 3)) { /* shunt out overflow bits */ - while (xm >> (SP_MBITS + 1 + 3)) { + while (xm >> (SP_FBITS + 1 + 3)) { SPXSRSX1(); } } else { /* normalize in grs extended single precision */ - while ((xm >> (SP_MBITS + 3)) == 0) { + while ((xm >> (SP_FBITS + 3)) == 0) { xm <<= 1; xe--; } } - SPNORMRET1(xs, xe, xm, "fint", x); -} - - -ieee754sp ieee754sp_funs(unsigned int u) -{ - if ((int) u < 0) - return ieee754sp_add(ieee754sp_1e31(), - ieee754sp_fint(u & ~(1 << 31))); - return ieee754sp_fint(u); + return ieee754sp_format(xs, xe, xm); } diff --git a/arch/mips/math-emu/sp_flong.c b/arch/mips/math-emu/sp_flong.c index 16a651f29865..012e30ce7589 100644 --- a/arch/mips/math-emu/sp_flong.c +++ b/arch/mips/math-emu/sp_flong.c @@ -5,8 +5,6 @@ * MIPS floating point support * Copyright (C) 1994-2000 Algorithmics Ltd. * - * ######################################################################## - * * This program is free software; you can distribute it and/or modify it * under the terms of the GNU General Public License (Version 2) as * published by the Free Software Foundation. @@ -18,21 +16,18 @@ * * You should have received a copy of the GNU General Public License along * with this program; if not, write to the Free Software Foundation, Inc., - * 59 Temple Place - Suite 330, Boston MA 02111-1307, USA. - * - * ######################################################################## + * 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. */ - #include "ieee754sp.h" -ieee754sp ieee754sp_flong(s64 x) +union ieee754sp ieee754sp_flong(s64 x) { u64 xm; /* <--- need 64-bit mantissa temp */ int xe; int xs; - CLEARCX; + ieee754_clearcx(); if (x == 0) return ieee754sp_zero(0); @@ -50,29 +45,20 @@ ieee754sp ieee754sp_flong(s64 x) } else { xm = x; } - xe = SP_MBITS + 3; + xe = SP_FBITS + 3; - if (xm >> (SP_MBITS + 1 + 3)) { + if (xm >> (SP_FBITS + 1 + 3)) { /* shunt out overflow bits */ - while (xm >> (SP_MBITS + 1 + 3)) { + while (xm >> (SP_FBITS + 1 + 3)) { SPXSRSX1(); } } else { /* normalize in grs extended single precision */ - while ((xm >> (SP_MBITS + 3)) == 0) { + while ((xm >> (SP_FBITS + 3)) == 0) { xm <<= 1; xe--; } } - SPNORMRET1(xs, xe, xm, "sp_flong", x); -} - - -ieee754sp ieee754sp_fulong(u64 u) -{ - if ((s64) u < 0) - return ieee754sp_add(ieee754sp_1e63(), - ieee754sp_flong(u & ~(1ULL << 63))); - return ieee754sp_flong(u); + return ieee754sp_format(xs, xe, xm); } diff --git a/arch/mips/math-emu/sp_frexp.c b/arch/mips/math-emu/sp_frexp.c deleted file mode 100644 index 5bc993c30044..000000000000 --- a/arch/mips/math-emu/sp_frexp.c +++ /dev/null @@ -1,52 +0,0 @@ -/* IEEE754 floating point arithmetic - * single precision - */ -/* - * MIPS floating point support - * Copyright (C) 1994-2000 Algorithmics Ltd. - * - * ######################################################################## - * - * This program is free software; you can distribute it and/or modify it - * under the terms of the GNU General Public License (Version 2) as - * published by the Free Software Foundation. - * - * This program is distributed in the hope it will be useful, but WITHOUT - * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or - * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License - * for more details. - * - * You should have received a copy of the GNU General Public License along - * with this program; if not, write to the Free Software Foundation, Inc., - * 59 Temple Place - Suite 330, Boston MA 02111-1307, USA. - * - * ######################################################################## - */ - - -#include "ieee754sp.h" - -/* close to ieeep754sp_logb -*/ -ieee754sp ieee754sp_frexp(ieee754sp x, int *eptr) -{ - COMPXSP; - CLEARCX; - EXPLODEXSP; - - switch (xc) { - case IEEE754_CLASS_SNAN: - case IEEE754_CLASS_QNAN: - case IEEE754_CLASS_INF: - case IEEE754_CLASS_ZERO: - *eptr = 0; - return x; - case IEEE754_CLASS_DNORM: - SPDNORMX; - break; - case IEEE754_CLASS_NORM: - break; - } - *eptr = xe + 1; - return buildsp(xs, -1 + SP_EBIAS, xm & ~SP_HIDDEN_BIT); -} diff --git a/arch/mips/math-emu/sp_logb.c b/arch/mips/math-emu/sp_logb.c deleted file mode 100644 index 9c14e0c75bd2..000000000000 --- a/arch/mips/math-emu/sp_logb.c +++ /dev/null @@ -1,53 +0,0 @@ -/* IEEE754 floating point arithmetic - * single precision - */ -/* - * MIPS floating point support - * Copyright (C) 1994-2000 Algorithmics Ltd. - * - * ######################################################################## - * - * This program is free software; you can distribute it and/or modify it - * under the terms of the GNU General Public License (Version 2) as - * published by the Free Software Foundation. - * - * This program is distributed in the hope it will be useful, but WITHOUT - * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or - * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License - * for more details. - * - * You should have received a copy of the GNU General Public License along - * with this program; if not, write to the Free Software Foundation, Inc., - * 59 Temple Place - Suite 330, Boston MA 02111-1307, USA. - * - * ######################################################################## - */ - - -#include "ieee754sp.h" - -ieee754sp ieee754sp_logb(ieee754sp x) -{ - COMPXSP; - - CLEARCX; - - EXPLODEXSP; - - switch (xc) { - case IEEE754_CLASS_SNAN: - return ieee754sp_nanxcpt(x, "logb", x); - case IEEE754_CLASS_QNAN: - return x; - case IEEE754_CLASS_INF: - return ieee754sp_inf(0); - case IEEE754_CLASS_ZERO: - return ieee754sp_inf(1); - case IEEE754_CLASS_DNORM: - SPDNORMX; - break; - case IEEE754_CLASS_NORM: - break; - } - return ieee754sp_fint(xe); -} diff --git a/arch/mips/math-emu/sp_modf.c b/arch/mips/math-emu/sp_modf.c deleted file mode 100644 index 25a0fbaa0556..000000000000 --- a/arch/mips/math-emu/sp_modf.c +++ /dev/null @@ -1,79 +0,0 @@ -/* IEEE754 floating point arithmetic - * single precision - */ -/* - * MIPS floating point support - * Copyright (C) 1994-2000 Algorithmics Ltd. - * - * ######################################################################## - * - * This program is free software; you can distribute it and/or modify it - * under the terms of the GNU General Public License (Version 2) as - * published by the Free Software Foundation. - * - * This program is distributed in the hope it will be useful, but WITHOUT - * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or - * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License - * for more details. - * - * You should have received a copy of the GNU General Public License along - * with this program; if not, write to the Free Software Foundation, Inc., - * 59 Temple Place - Suite 330, Boston MA 02111-1307, USA. - * - * ######################################################################## - */ - - -#include "ieee754sp.h" - -/* modf function is always exact for a finite number -*/ -ieee754sp ieee754sp_modf(ieee754sp x, ieee754sp *ip) -{ - COMPXSP; - - CLEARCX; - - EXPLODEXSP; - - switch (xc) { - case IEEE754_CLASS_SNAN: - case IEEE754_CLASS_QNAN: - case IEEE754_CLASS_INF: - case IEEE754_CLASS_ZERO: - *ip = x; - return x; - case IEEE754_CLASS_DNORM: - /* far to small */ - *ip = ieee754sp_zero(xs); - return x; - case IEEE754_CLASS_NORM: - break; - } - if (xe < 0) { - *ip = ieee754sp_zero(xs); - return x; - } - if (xe >= SP_MBITS) { - *ip = x; - return ieee754sp_zero(xs); - } - /* generate ipart mantissa by clearing bottom bits - */ - *ip = buildsp(xs, xe + SP_EBIAS, - ((xm >> (SP_MBITS - xe)) << (SP_MBITS - xe)) & - ~SP_HIDDEN_BIT); - - /* generate fpart mantissa by clearing top bits - * and normalizing (must be able to normalize) - */ - xm = (xm << (32 - (SP_MBITS - xe))) >> (32 - (SP_MBITS - xe)); - if (xm == 0) - return ieee754sp_zero(xs); - - while ((xm >> SP_MBITS) == 0) { - xm <<= 1; - xe--; - } - return buildsp(xs, xe + SP_EBIAS, xm & ~SP_HIDDEN_BIT); -} diff --git a/arch/mips/math-emu/sp_mul.c b/arch/mips/math-emu/sp_mul.c index fa4675cf2aad..890c13a2965e 100644 --- a/arch/mips/math-emu/sp_mul.c +++ b/arch/mips/math-emu/sp_mul.c @@ -5,8 +5,6 @@ * MIPS floating point support * Copyright (C) 1994-2000 Algorithmics Ltd. * - * ######################################################################## - * * This program is free software; you can distribute it and/or modify it * under the terms of the GNU General Public License (Version 2) as * published by the Free Software Foundation. @@ -18,23 +16,32 @@ * * You should have received a copy of the GNU General Public License along * with this program; if not, write to the Free Software Foundation, Inc., - * 59 Temple Place - Suite 330, Boston MA 02111-1307, USA. - * - * ######################################################################## + * 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. */ - #include "ieee754sp.h" -ieee754sp ieee754sp_mul(ieee754sp x, ieee754sp y) +union ieee754sp ieee754sp_mul(union ieee754sp x, union ieee754sp y) { + int re; + int rs; + unsigned rm; + unsigned short lxm; + unsigned short hxm; + unsigned short lym; + unsigned short hym; + unsigned lrm; + unsigned hrm; + unsigned t; + unsigned at; + COMPXSP; COMPYSP; EXPLODEXSP; EXPLODEYSP; - CLEARCX; + ieee754_clearcx(); FLUSHXSP; FLUSHYSP; @@ -51,8 +58,8 @@ ieee754sp ieee754sp_mul(ieee754sp x, ieee754sp y) case CLPAIR(IEEE754_CLASS_SNAN, IEEE754_CLASS_NORM): case CLPAIR(IEEE754_CLASS_SNAN, IEEE754_CLASS_DNORM): case CLPAIR(IEEE754_CLASS_SNAN, IEEE754_CLASS_INF): - SETCX(IEEE754_INVALID_OPERATION); - return ieee754sp_nanxcpt(ieee754sp_indef(), "mul", x, y); + ieee754_setcx(IEEE754_INVALID_OPERATION); + return ieee754sp_nanxcpt(ieee754sp_indef()); case CLPAIR(IEEE754_CLASS_ZERO, IEEE754_CLASS_QNAN): case CLPAIR(IEEE754_CLASS_NORM, IEEE754_CLASS_QNAN): @@ -68,12 +75,13 @@ ieee754sp ieee754sp_mul(ieee754sp x, ieee754sp y) return x; - /* Infinity handling */ - + /* + * Infinity handling + */ case CLPAIR(IEEE754_CLASS_INF, IEEE754_CLASS_ZERO): case CLPAIR(IEEE754_CLASS_ZERO, IEEE754_CLASS_INF): - SETCX(IEEE754_INVALID_OPERATION); - return ieee754sp_xcpt(ieee754sp_indef(), "mul", x, y); + ieee754_setcx(IEEE754_INVALID_OPERATION); + return ieee754sp_indef(); case CLPAIR(IEEE754_CLASS_NORM, IEEE754_CLASS_INF): case CLPAIR(IEEE754_CLASS_DNORM, IEEE754_CLASS_INF): @@ -108,63 +116,50 @@ ieee754sp ieee754sp_mul(ieee754sp x, ieee754sp y) assert(xm & SP_HIDDEN_BIT); assert(ym & SP_HIDDEN_BIT); - { - int re = xe + ye; - int rs = xs ^ ys; - unsigned rm; - - /* shunt to top of word */ - xm <<= 32 - (SP_MBITS + 1); - ym <<= 32 - (SP_MBITS + 1); - - /* multiply 32bits xm,ym to give high 32bits rm with stickness - */ - { - unsigned short lxm = xm & 0xffff; - unsigned short hxm = xm >> 16; - unsigned short lym = ym & 0xffff; - unsigned short hym = ym >> 16; - unsigned lrm; - unsigned hrm; - - lrm = lxm * lym; /* 16 * 16 => 32 */ - hrm = hxm * hym; /* 16 * 16 => 32 */ - - { - unsigned t = lxm * hym; /* 16 * 16 => 32 */ - { - unsigned at = lrm + (t << 16); - hrm += at < lrm; - lrm = at; - } - hrm = hrm + (t >> 16); - } - - { - unsigned t = hxm * lym; /* 16 * 16 => 32 */ - { - unsigned at = lrm + (t << 16); - hrm += at < lrm; - lrm = at; - } - hrm = hrm + (t >> 16); - } - rm = hrm | (lrm != 0); - } - - /* - * sticky shift down to normal rounding precision - */ - if ((int) rm < 0) { - rm = (rm >> (32 - (SP_MBITS + 1 + 3))) | - ((rm << (SP_MBITS + 1 + 3)) != 0); - re++; - } else { - rm = (rm >> (32 - (SP_MBITS + 1 + 3 + 1))) | - ((rm << (SP_MBITS + 1 + 3 + 1)) != 0); - } - assert(rm & (SP_HIDDEN_BIT << 3)); - - SPNORMRET2(rs, re, rm, "mul", x, y); + re = xe + ye; + rs = xs ^ ys; + + /* shunt to top of word */ + xm <<= 32 - (SP_FBITS + 1); + ym <<= 32 - (SP_FBITS + 1); + + /* + * Multiply 32 bits xm, ym to give high 32 bits rm with stickness. + */ + lxm = xm & 0xffff; + hxm = xm >> 16; + lym = ym & 0xffff; + hym = ym >> 16; + + lrm = lxm * lym; /* 16 * 16 => 32 */ + hrm = hxm * hym; /* 16 * 16 => 32 */ + + t = lxm * hym; /* 16 * 16 => 32 */ + at = lrm + (t << 16); + hrm += at < lrm; + lrm = at; + hrm = hrm + (t >> 16); + + t = hxm * lym; /* 16 * 16 => 32 */ + at = lrm + (t << 16); + hrm += at < lrm; + lrm = at; + hrm = hrm + (t >> 16); + + rm = hrm | (lrm != 0); + + /* + * Sticky shift down to normal rounding precision. + */ + if ((int) rm < 0) { + rm = (rm >> (32 - (SP_FBITS + 1 + 3))) | + ((rm << (SP_FBITS + 1 + 3)) != 0); + re++; + } else { + rm = (rm >> (32 - (SP_FBITS + 1 + 3 + 1))) | + ((rm << (SP_FBITS + 1 + 3 + 1)) != 0); } + assert(rm & (SP_HIDDEN_BIT << 3)); + + return ieee754sp_format(rs, re, rm); } diff --git a/arch/mips/math-emu/sp_scalb.c b/arch/mips/math-emu/sp_scalb.c deleted file mode 100644 index dd76196984c8..000000000000 --- a/arch/mips/math-emu/sp_scalb.c +++ /dev/null @@ -1,57 +0,0 @@ -/* IEEE754 floating point arithmetic - * single precision - */ -/* - * MIPS floating point support - * Copyright (C) 1994-2000 Algorithmics Ltd. - * - * ######################################################################## - * - * This program is free software; you can distribute it and/or modify it - * under the terms of the GNU General Public License (Version 2) as - * published by the Free Software Foundation. - * - * This program is distributed in the hope it will be useful, but WITHOUT - * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or - * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License - * for more details. - * - * You should have received a copy of the GNU General Public License along - * with this program; if not, write to the Free Software Foundation, Inc., - * 59 Temple Place - Suite 330, Boston MA 02111-1307, USA. - * - * ######################################################################## - */ - - -#include "ieee754sp.h" - -ieee754sp ieee754sp_scalb(ieee754sp x, int n) -{ - COMPXSP; - - CLEARCX; - - EXPLODEXSP; - - switch (xc) { - case IEEE754_CLASS_SNAN: - return ieee754sp_nanxcpt(x, "scalb", x, n); - case IEEE754_CLASS_QNAN: - case IEEE754_CLASS_INF: - case IEEE754_CLASS_ZERO: - return x; - case IEEE754_CLASS_DNORM: - SPDNORMX; - break; - case IEEE754_CLASS_NORM: - break; - } - SPNORMRET2(xs, xe + n, xm << 3, "scalb", x, n); -} - - -ieee754sp ieee754sp_ldexp(ieee754sp x, int n) -{ - return ieee754sp_scalb(x, n); -} diff --git a/arch/mips/math-emu/sp_simple.c b/arch/mips/math-emu/sp_simple.c index ae4fcfafd853..f1ffaa9a17e0 100644 --- a/arch/mips/math-emu/sp_simple.c +++ b/arch/mips/math-emu/sp_simple.c @@ -5,8 +5,6 @@ * MIPS floating point support * Copyright (C) 1994-2000 Algorithmics Ltd. * - * ######################################################################## - * * This program is free software; you can distribute it and/or modify it * under the terms of the GNU General Public License (Version 2) as * published by the Free Software Foundation. @@ -18,33 +16,17 @@ * * You should have received a copy of the GNU General Public License along * with this program; if not, write to the Free Software Foundation, Inc., - * 59 Temple Place - Suite 330, Boston MA 02111-1307, USA. - * - * ######################################################################## + * 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. */ - #include "ieee754sp.h" -int ieee754sp_finite(ieee754sp x) -{ - return SPBEXP(x) != SP_EMAX + 1 + SP_EBIAS; -} - -ieee754sp ieee754sp_copysign(ieee754sp x, ieee754sp y) -{ - CLEARCX; - SPSIGN(x) = SPSIGN(y); - return x; -} - - -ieee754sp ieee754sp_neg(ieee754sp x) +union ieee754sp ieee754sp_neg(union ieee754sp x) { COMPXSP; EXPLODEXSP; - CLEARCX; + ieee754_clearcx(); FLUSHXSP; /* @@ -55,30 +37,29 @@ ieee754sp ieee754sp_neg(ieee754sp x) SPSIGN(x) ^= 1; if (xc == IEEE754_CLASS_SNAN) { - ieee754sp y = ieee754sp_indef(); - SETCX(IEEE754_INVALID_OPERATION); + union ieee754sp y = ieee754sp_indef(); + ieee754_setcx(IEEE754_INVALID_OPERATION); SPSIGN(y) = SPSIGN(x); - return ieee754sp_nanxcpt(y, "neg"); + return ieee754sp_nanxcpt(y); } return x; } - -ieee754sp ieee754sp_abs(ieee754sp x) +union ieee754sp ieee754sp_abs(union ieee754sp x) { COMPXSP; EXPLODEXSP; - CLEARCX; + ieee754_clearcx(); FLUSHXSP; /* Clear sign ALWAYS, irrespective of NaN */ SPSIGN(x) = 0; if (xc == IEEE754_CLASS_SNAN) { - SETCX(IEEE754_INVALID_OPERATION); - return ieee754sp_nanxcpt(ieee754sp_indef(), "abs"); + ieee754_setcx(IEEE754_INVALID_OPERATION); + return ieee754sp_nanxcpt(ieee754sp_indef()); } return x; diff --git a/arch/mips/math-emu/sp_sqrt.c b/arch/mips/math-emu/sp_sqrt.c index fed20175f5fb..b7c098a86f95 100644 --- a/arch/mips/math-emu/sp_sqrt.c +++ b/arch/mips/math-emu/sp_sqrt.c @@ -5,8 +5,6 @@ * MIPS floating point support * Copyright (C) 1994-2000 Algorithmics Ltd. * - * ######################################################################## - * * This program is free software; you can distribute it and/or modify it * under the terms of the GNU General Public License (Version 2) as * published by the Free Software Foundation. @@ -18,15 +16,12 @@ * * You should have received a copy of the GNU General Public License along * with this program; if not, write to the Free Software Foundation, Inc., - * 59 Temple Place - Suite 330, Boston MA 02111-1307, USA. - * - * ######################################################################## + * 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. */ - #include "ieee754sp.h" -ieee754sp ieee754sp_sqrt(ieee754sp x) +union ieee754sp ieee754sp_sqrt(union ieee754sp x) { int ix, s, q, m, t, i; unsigned int r; @@ -35,34 +30,38 @@ ieee754sp ieee754sp_sqrt(ieee754sp x) /* take care of Inf and NaN */ EXPLODEXSP; - CLEARCX; + ieee754_clearcx(); FLUSHXSP; /* x == INF or NAN? */ switch (xc) { case IEEE754_CLASS_QNAN: /* sqrt(Nan) = Nan */ - return ieee754sp_nanxcpt(x, "sqrt"); + return ieee754sp_nanxcpt(x); + case IEEE754_CLASS_SNAN: - SETCX(IEEE754_INVALID_OPERATION); - return ieee754sp_nanxcpt(ieee754sp_indef(), "sqrt"); + ieee754_setcx(IEEE754_INVALID_OPERATION); + return ieee754sp_nanxcpt(ieee754sp_indef()); + case IEEE754_CLASS_ZERO: /* sqrt(0) = 0 */ return x; + case IEEE754_CLASS_INF: if (xs) { /* sqrt(-Inf) = Nan */ - SETCX(IEEE754_INVALID_OPERATION); - return ieee754sp_nanxcpt(ieee754sp_indef(), "sqrt"); + ieee754_setcx(IEEE754_INVALID_OPERATION); + return ieee754sp_nanxcpt(ieee754sp_indef()); } /* sqrt(+Inf) = Inf */ return x; + case IEEE754_CLASS_DNORM: case IEEE754_CLASS_NORM: if (xs) { /* sqrt(-x) = Nan */ - SETCX(IEEE754_INVALID_OPERATION); - return ieee754sp_nanxcpt(ieee754sp_indef(), "sqrt"); + ieee754_setcx(IEEE754_INVALID_OPERATION); + return ieee754sp_nanxcpt(ieee754sp_indef()); } break; } @@ -99,12 +98,12 @@ ieee754sp ieee754sp_sqrt(ieee754sp x) } if (ix != 0) { - SETCX(IEEE754_INEXACT); + ieee754_setcx(IEEE754_INEXACT); switch (ieee754_csr.rm) { - case IEEE754_RP: + case FPU_CSR_RU: q += 2; break; - case IEEE754_RN: + case FPU_CSR_RN: q += (q & 1); break; } diff --git a/arch/mips/math-emu/sp_sub.c b/arch/mips/math-emu/sp_sub.c index e595c6f3d0bb..8592e49032b8 100644 --- a/arch/mips/math-emu/sp_sub.c +++ b/arch/mips/math-emu/sp_sub.c @@ -5,8 +5,6 @@ * MIPS floating point support * Copyright (C) 1994-2000 Algorithmics Ltd. * - * ######################################################################## - * * This program is free software; you can distribute it and/or modify it * under the terms of the GNU General Public License (Version 2) as * published by the Free Software Foundation. @@ -18,23 +16,22 @@ * * You should have received a copy of the GNU General Public License along * with this program; if not, write to the Free Software Foundation, Inc., - * 59 Temple Place - Suite 330, Boston MA 02111-1307, USA. - * - * ######################################################################## + * 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. */ - #include "ieee754sp.h" -ieee754sp ieee754sp_sub(ieee754sp x, ieee754sp y) +union ieee754sp ieee754sp_sub(union ieee754sp x, union ieee754sp y) { + int s; + COMPXSP; COMPYSP; EXPLODEXSP; EXPLODEYSP; - CLEARCX; + ieee754_clearcx(); FLUSHXSP; FLUSHYSP; @@ -51,8 +48,8 @@ ieee754sp ieee754sp_sub(ieee754sp x, ieee754sp y) case CLPAIR(IEEE754_CLASS_SNAN, IEEE754_CLASS_NORM): case CLPAIR(IEEE754_CLASS_SNAN, IEEE754_CLASS_DNORM): case CLPAIR(IEEE754_CLASS_SNAN, IEEE754_CLASS_INF): - SETCX(IEEE754_INVALID_OPERATION); - return ieee754sp_nanxcpt(ieee754sp_indef(), "sub", x, y); + ieee754_setcx(IEEE754_INVALID_OPERATION); + return ieee754sp_nanxcpt(ieee754sp_indef()); case CLPAIR(IEEE754_CLASS_ZERO, IEEE754_CLASS_QNAN): case CLPAIR(IEEE754_CLASS_NORM, IEEE754_CLASS_QNAN): @@ -68,14 +65,14 @@ ieee754sp ieee754sp_sub(ieee754sp x, ieee754sp y) return x; - /* Infinity handling - */ - + /* + * Infinity handling + */ case CLPAIR(IEEE754_CLASS_INF, IEEE754_CLASS_INF): if (xs != ys) return x; - SETCX(IEEE754_INVALID_OPERATION); - return ieee754sp_xcpt(ieee754sp_indef(), "sub", x, y); + ieee754_setcx(IEEE754_INVALID_OPERATION); + return ieee754sp_indef(); case CLPAIR(IEEE754_CLASS_ZERO, IEEE754_CLASS_INF): case CLPAIR(IEEE754_CLASS_DNORM, IEEE754_CLASS_INF): @@ -87,15 +84,14 @@ ieee754sp ieee754sp_sub(ieee754sp x, ieee754sp y) case CLPAIR(IEEE754_CLASS_INF, IEEE754_CLASS_DNORM): return x; - /* Zero handling - */ - + /* + * Zero handling + */ case CLPAIR(IEEE754_CLASS_ZERO, IEEE754_CLASS_ZERO): if (xs != ys) return x; else - return ieee754sp_zero(ieee754_csr.rm == - IEEE754_RD); + return ieee754sp_zero(ieee754_csr.rm == FPU_CSR_RD); case CLPAIR(IEEE754_CLASS_NORM, IEEE754_CLASS_ZERO): case CLPAIR(IEEE754_CLASS_DNORM, IEEE754_CLASS_ZERO): @@ -104,7 +100,7 @@ ieee754sp ieee754sp_sub(ieee754sp x, ieee754sp y) case CLPAIR(IEEE754_CLASS_ZERO, IEEE754_CLASS_NORM): case CLPAIR(IEEE754_CLASS_ZERO, IEEE754_CLASS_DNORM): /* quick fix up */ - DPSIGN(y) ^= 1; + SPSIGN(y) ^= 1; return y; case CLPAIR(IEEE754_CLASS_DNORM, IEEE754_CLASS_DNORM): @@ -133,14 +129,16 @@ ieee754sp ieee754sp_sub(ieee754sp x, ieee754sp y) ym <<= 3; if (xe > ye) { - /* have to shift y fraction right to align + /* + * have to shift y fraction right to align */ - int s = xe - ye; + s = xe - ye; SPXSRSYn(s); } else if (ye > xe) { - /* have to shift x fraction right to align + /* + * have to shift x fraction right to align */ - int s = ye - xe; + s = ye - xe; SPXSRSXn(s); } assert(xe == ye); @@ -153,7 +151,7 @@ ieee754sp ieee754sp_sub(ieee754sp x, ieee754sp y) xe = xe; xs = xs; - if (xm >> (SP_MBITS + 1 + 3)) { /* carry out */ + if (xm >> (SP_FBITS + 1 + 3)) { /* carry out */ SPXSRSX1(); /* shift preserving sticky */ } } else { @@ -167,17 +165,18 @@ ieee754sp ieee754sp_sub(ieee754sp x, ieee754sp y) xs = ys; } if (xm == 0) { - if (ieee754_csr.rm == IEEE754_RD) + if (ieee754_csr.rm == FPU_CSR_RD) return ieee754sp_zero(1); /* round negative inf. => sign = -1 */ else return ieee754sp_zero(0); /* other round modes => sign = 1 */ } /* normalize to rounding precision */ - while ((xm >> (SP_MBITS + 3)) == 0) { + while ((xm >> (SP_FBITS + 3)) == 0) { xm <<= 1; xe--; } } - SPNORMRET2(xs, xe, xm, "sub", x, y); + + return ieee754sp_format(xs, xe, xm); } diff --git a/arch/mips/math-emu/sp_tint.c b/arch/mips/math-emu/sp_tint.c index 0fe9acc7716e..091299a31798 100644 --- a/arch/mips/math-emu/sp_tint.c +++ b/arch/mips/math-emu/sp_tint.c @@ -5,8 +5,6 @@ * MIPS floating point support * Copyright (C) 1994-2000 Algorithmics Ltd. * - * ######################################################################## - * * This program is free software; you can distribute it and/or modify it * under the terms of the GNU General Public License (Version 2) as * published by the Free Software Foundation. @@ -18,20 +16,21 @@ * * You should have received a copy of the GNU General Public License along * with this program; if not, write to the Free Software Foundation, Inc., - * 59 Temple Place - Suite 330, Boston MA 02111-1307, USA. - * - * ######################################################################## + * 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. */ - -#include <linux/kernel.h> #include "ieee754sp.h" -int ieee754sp_tint(ieee754sp x) +int ieee754sp_tint(union ieee754sp x) { + u32 residue; + int round; + int sticky; + int odd; + COMPXSP; - CLEARCX; + ieee754_clearcx(); EXPLODEXSP; FLUSHXSP; @@ -40,10 +39,12 @@ int ieee754sp_tint(ieee754sp x) case IEEE754_CLASS_SNAN: case IEEE754_CLASS_QNAN: case IEEE754_CLASS_INF: - SETCX(IEEE754_INVALID_OPERATION); - return ieee754si_xcpt(ieee754si_indef(), "sp_tint", x); + ieee754_setcx(IEEE754_INVALID_OPERATION); + return ieee754si_indef(); + case IEEE754_CLASS_ZERO: return 0; + case IEEE754_CLASS_DNORM: case IEEE754_CLASS_NORM: break; @@ -54,18 +55,13 @@ int ieee754sp_tint(ieee754sp x) return -0x80000000; /* Set invalid. We will only use overflow for floating point overflow */ - SETCX(IEEE754_INVALID_OPERATION); - return ieee754si_xcpt(ieee754si_indef(), "sp_tint", x); + ieee754_setcx(IEEE754_INVALID_OPERATION); + return ieee754si_indef(); } /* oh gawd */ - if (xe > SP_MBITS) { - xm <<= xe - SP_MBITS; + if (xe > SP_FBITS) { + xm <<= xe - SP_FBITS; } else { - u32 residue; - int round; - int sticky; - int odd; - if (xe < -1) { residue = xm; round = 0; @@ -76,51 +72,38 @@ int ieee754sp_tint(ieee754sp x) * so we do it in two steps. Be aware that xe * may be -1 */ residue = xm << (xe + 1); - residue <<= 31 - SP_MBITS; + residue <<= 31 - SP_FBITS; round = (residue >> 31) != 0; sticky = (residue << 1) != 0; - xm >>= SP_MBITS - xe; + xm >>= SP_FBITS - xe; } odd = (xm & 0x1) != 0x0; switch (ieee754_csr.rm) { - case IEEE754_RN: + case FPU_CSR_RN: if (round && (sticky || odd)) xm++; break; - case IEEE754_RZ: + case FPU_CSR_RZ: break; - case IEEE754_RU: /* toward +Infinity */ + case FPU_CSR_RU: /* toward +Infinity */ if ((round || sticky) && !xs) xm++; break; - case IEEE754_RD: /* toward -Infinity */ + case FPU_CSR_RD: /* toward -Infinity */ if ((round || sticky) && xs) xm++; break; } if ((xm >> 31) != 0) { /* This can happen after rounding */ - SETCX(IEEE754_INVALID_OPERATION); - return ieee754si_xcpt(ieee754si_indef(), "sp_tint", x); + ieee754_setcx(IEEE754_INVALID_OPERATION); + return ieee754si_indef(); } if (round || sticky) - SETCX(IEEE754_INEXACT); + ieee754_setcx(IEEE754_INEXACT); } if (xs) return -xm; else return xm; } - - -unsigned int ieee754sp_tuns(ieee754sp x) -{ - ieee754sp hb = ieee754sp_1e31(); - - /* what if x < 0 ?? */ - if (ieee754sp_lt(x, hb)) - return (unsigned) ieee754sp_tint(x); - - return (unsigned) ieee754sp_tint(ieee754sp_sub(x, hb)) | - ((unsigned) 1 << 31); -} diff --git a/arch/mips/math-emu/sp_tlong.c b/arch/mips/math-emu/sp_tlong.c index d0ca6e22be29..9f3c742c1cea 100644 --- a/arch/mips/math-emu/sp_tlong.c +++ b/arch/mips/math-emu/sp_tlong.c @@ -5,8 +5,6 @@ * MIPS floating point support * Copyright (C) 1994-2000 Algorithmics Ltd. * - * ######################################################################## - * * This program is free software; you can distribute it and/or modify it * under the terms of the GNU General Public License (Version 2) as * published by the Free Software Foundation. @@ -18,19 +16,22 @@ * * You should have received a copy of the GNU General Public License along * with this program; if not, write to the Free Software Foundation, Inc., - * 59 Temple Place - Suite 330, Boston MA 02111-1307, USA. - * - * ######################################################################## + * 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. */ - #include "ieee754sp.h" +#include "ieee754dp.h" -s64 ieee754sp_tlong(ieee754sp x) +s64 ieee754sp_tlong(union ieee754sp x) { + u32 residue; + int round; + int sticky; + int odd; + COMPXDP; /* <-- need 64-bit mantissa tmp */ - CLEARCX; + ieee754_clearcx(); EXPLODEXSP; FLUSHXSP; @@ -39,10 +40,12 @@ s64 ieee754sp_tlong(ieee754sp x) case IEEE754_CLASS_SNAN: case IEEE754_CLASS_QNAN: case IEEE754_CLASS_INF: - SETCX(IEEE754_INVALID_OPERATION); - return ieee754di_xcpt(ieee754di_indef(), "sp_tlong", x); + ieee754_setcx(IEEE754_INVALID_OPERATION); + return ieee754di_indef(); + case IEEE754_CLASS_ZERO: return 0; + case IEEE754_CLASS_DNORM: case IEEE754_CLASS_NORM: break; @@ -53,69 +56,51 @@ s64 ieee754sp_tlong(ieee754sp x) return -0x8000000000000000LL; /* Set invalid. We will only use overflow for floating point overflow */ - SETCX(IEEE754_INVALID_OPERATION); - return ieee754di_xcpt(ieee754di_indef(), "sp_tlong", x); + ieee754_setcx(IEEE754_INVALID_OPERATION); + return ieee754di_indef(); } /* oh gawd */ - if (xe > SP_MBITS) { - xm <<= xe - SP_MBITS; - } else if (xe < SP_MBITS) { - u32 residue; - int round; - int sticky; - int odd; - + if (xe > SP_FBITS) { + xm <<= xe - SP_FBITS; + } else if (xe < SP_FBITS) { if (xe < -1) { residue = xm; round = 0; sticky = residue != 0; xm = 0; } else { - residue = xm << (32 - SP_MBITS + xe); + residue = xm << (32 - SP_FBITS + xe); round = (residue >> 31) != 0; sticky = (residue << 1) != 0; - xm >>= SP_MBITS - xe; + xm >>= SP_FBITS - xe; } odd = (xm & 0x1) != 0x0; switch (ieee754_csr.rm) { - case IEEE754_RN: + case FPU_CSR_RN: if (round && (sticky || odd)) xm++; break; - case IEEE754_RZ: + case FPU_CSR_RZ: break; - case IEEE754_RU: /* toward +Infinity */ + case FPU_CSR_RU: /* toward +Infinity */ if ((round || sticky) && !xs) xm++; break; - case IEEE754_RD: /* toward -Infinity */ + case FPU_CSR_RD: /* toward -Infinity */ if ((round || sticky) && xs) xm++; break; } if ((xm >> 63) != 0) { /* This can happen after rounding */ - SETCX(IEEE754_INVALID_OPERATION); - return ieee754di_xcpt(ieee754di_indef(), "sp_tlong", x); + ieee754_setcx(IEEE754_INVALID_OPERATION); + return ieee754di_indef(); } if (round || sticky) - SETCX(IEEE754_INEXACT); + ieee754_setcx(IEEE754_INEXACT); } if (xs) return -xm; else return xm; } - - -u64 ieee754sp_tulong(ieee754sp x) -{ - ieee754sp hb = ieee754sp_1e63(); - - /* what if x < 0 ?? */ - if (ieee754sp_lt(x, hb)) - return (u64) ieee754sp_tlong(x); - - return (u64) ieee754sp_tlong(ieee754sp_sub(x, hb)) | - (1ULL << 63); -} diff --git a/arch/mips/mm/c-octeon.c b/arch/mips/mm/c-octeon.c index f41a5c5b0865..05b1d7cf9514 100644 --- a/arch/mips/mm/c-octeon.c +++ b/arch/mips/mm/c-octeon.c @@ -137,8 +137,10 @@ static void octeon_flush_cache_sigtramp(unsigned long addr) { struct vm_area_struct *vma; + down_read(¤t->mm->mmap_sem); vma = find_vma(current->mm, addr); octeon_flush_icache_all_cores(vma); + up_read(¤t->mm->mmap_sem); } diff --git a/arch/mips/mm/c-r4k.c b/arch/mips/mm/c-r4k.c index 1c74a6ad072a..f2e8302fa70f 100644 --- a/arch/mips/mm/c-r4k.c +++ b/arch/mips/mm/c-r4k.c @@ -7,6 +7,7 @@ * Copyright (C) 1997, 1998, 1999, 2000, 2001, 2002 Ralf Baechle (ralf@gnu.org) * Copyright (C) 1999, 2000 Silicon Graphics, Inc. */ +#include <linux/cpu_pm.h> #include <linux/hardirq.h> #include <linux/init.h> #include <linux/highmem.h> @@ -50,7 +51,7 @@ static inline void r4k_on_each_cpu(void (*func) (void *info), void *info) { preempt_disable(); -#if !defined(CONFIG_MIPS_MT_SMP) && !defined(CONFIG_MIPS_MT_SMTC) +#ifndef CONFIG_MIPS_MT_SMP smp_call_function(func, info, 1); #endif func(info); @@ -105,22 +106,37 @@ static inline void r4k_blast_dcache_page_dc32(unsigned long addr) static inline void r4k_blast_dcache_page_dc64(unsigned long addr) { - R4600_HIT_CACHEOP_WAR_IMPL; blast_dcache64_page(addr); } +static inline void r4k_blast_dcache_page_dc128(unsigned long addr) +{ + blast_dcache128_page(addr); +} + static void r4k_blast_dcache_page_setup(void) { unsigned long dc_lsize = cpu_dcache_line_size(); - if (dc_lsize == 0) + switch (dc_lsize) { + case 0: r4k_blast_dcache_page = (void *)cache_noop; - else if (dc_lsize == 16) + break; + case 16: r4k_blast_dcache_page = blast_dcache16_page; - else if (dc_lsize == 32) + break; + case 32: r4k_blast_dcache_page = r4k_blast_dcache_page_dc32; - else if (dc_lsize == 64) + break; + case 64: r4k_blast_dcache_page = r4k_blast_dcache_page_dc64; + break; + case 128: + r4k_blast_dcache_page = r4k_blast_dcache_page_dc128; + break; + default: + break; + } } #ifndef CONFIG_EVA @@ -159,6 +175,8 @@ static void r4k_blast_dcache_page_indexed_setup(void) r4k_blast_dcache_page_indexed = blast_dcache32_page_indexed; else if (dc_lsize == 64) r4k_blast_dcache_page_indexed = blast_dcache64_page_indexed; + else if (dc_lsize == 128) + r4k_blast_dcache_page_indexed = blast_dcache128_page_indexed; } void (* r4k_blast_dcache)(void); @@ -176,6 +194,8 @@ static void r4k_blast_dcache_setup(void) r4k_blast_dcache = blast_dcache32; else if (dc_lsize == 64) r4k_blast_dcache = blast_dcache64; + else if (dc_lsize == 128) + r4k_blast_dcache = blast_dcache128; } /* force code alignment (used for TX49XX_ICACHE_INDEX_INV_WAR) */ @@ -265,6 +285,8 @@ static void r4k_blast_icache_page_setup(void) r4k_blast_icache_page = blast_icache32_page; else if (ic_lsize == 64) r4k_blast_icache_page = blast_icache64_page; + else if (ic_lsize == 128) + r4k_blast_icache_page = blast_icache128_page; } #ifndef CONFIG_EVA @@ -338,6 +360,8 @@ static void r4k_blast_icache_setup(void) r4k_blast_icache = blast_icache32; } else if (ic_lsize == 64) r4k_blast_icache = blast_icache64; + else if (ic_lsize == 128) + r4k_blast_icache = blast_icache128; } static void (* r4k_blast_scache_page)(unsigned long addr); @@ -428,7 +452,7 @@ static void r4k___flush_cache_all(void) static inline int has_valid_asid(const struct mm_struct *mm) { -#if defined(CONFIG_MIPS_MT_SMP) || defined(CONFIG_MIPS_MT_SMTC) +#ifdef CONFIG_MIPS_MT_SMP int i; for_each_online_cpu(i) @@ -1094,6 +1118,21 @@ static void probe_pcache(void) c->dcache.waybit = 0; break; + case CPU_CAVIUM_OCTEON3: + /* For now lie about the number of ways. */ + c->icache.linesz = 128; + c->icache.sets = 16; + c->icache.ways = 8; + c->icache.flags |= MIPS_CACHE_VTAG; + icache_size = c->icache.sets * c->icache.ways * c->icache.linesz; + + c->dcache.linesz = 128; + c->dcache.ways = 8; + c->dcache.sets = 8; + dcache_size = c->dcache.sets * c->dcache.ways * c->dcache.linesz; + c->options |= MIPS_CPU_PREFETCH; + break; + default: if (!(config & MIPS_CONF_M)) panic("Don't know how to probe P-caches on this cpu."); @@ -1414,6 +1453,7 @@ static void setup_scache(void) loongson3_sc_init(); return; + case CPU_CAVIUM_OCTEON3: case CPU_XLP: /* don't need to worry about L2, fully coherent */ return; @@ -1644,3 +1684,26 @@ void r4k_cache_init(void) coherency_setup(); board_cache_error_setup = r4k_cache_error_setup; } + +static int r4k_cache_pm_notifier(struct notifier_block *self, unsigned long cmd, + void *v) +{ + switch (cmd) { + case CPU_PM_ENTER_FAILED: + case CPU_PM_EXIT: + coherency_setup(); + break; + } + + return NOTIFY_OK; +} + +static struct notifier_block r4k_cache_pm_notifier_block = { + .notifier_call = r4k_cache_pm_notifier, +}; + +int __init r4k_cache_init_pm(void) +{ + return cpu_pm_register_notifier(&r4k_cache_pm_notifier_block); +} +arch_initcall(r4k_cache_init_pm); diff --git a/arch/mips/mm/init.c b/arch/mips/mm/init.c index 4fc74c78265a..6e4413330e36 100644 --- a/arch/mips/mm/init.c +++ b/arch/mips/mm/init.c @@ -44,27 +44,6 @@ #include <asm/tlb.h> #include <asm/fixmap.h> -/* Atomicity and interruptability */ -#ifdef CONFIG_MIPS_MT_SMTC - -#include <asm/mipsmtregs.h> - -#define ENTER_CRITICAL(flags) \ - { \ - unsigned int mvpflags; \ - local_irq_save(flags);\ - mvpflags = dvpe() -#define EXIT_CRITICAL(flags) \ - evpe(mvpflags); \ - local_irq_restore(flags); \ - } -#else - -#define ENTER_CRITICAL(flags) local_irq_save(flags) -#define EXIT_CRITICAL(flags) local_irq_restore(flags) - -#endif /* CONFIG_MIPS_MT_SMTC */ - /* * We have up to 8 empty zeroed pages so we can map one of the right colour * when needed. This is necessary only on R4000 / R4400 SC and MC versions @@ -100,21 +79,7 @@ void setup_zero_pages(void) zero_page_mask = ((PAGE_SIZE << order) - 1) & PAGE_MASK; } -#ifdef CONFIG_MIPS_MT_SMTC -static pte_t *kmap_coherent_pte; -static void __init kmap_coherent_init(void) -{ - unsigned long vaddr; - - /* cache the first coherent kmap pte */ - vaddr = __fix_to_virt(FIX_CMAP_BEGIN); - kmap_coherent_pte = kmap_get_fixmap_pte(vaddr); -} -#else -static inline void kmap_coherent_init(void) {} -#endif - -void *kmap_coherent(struct page *page, unsigned long addr) +static void *__kmap_pgprot(struct page *page, unsigned long addr, pgprot_t prot) { enum fixed_addresses idx; unsigned long vaddr, flags, entrylo; @@ -126,58 +91,48 @@ void *kmap_coherent(struct page *page, unsigned long addr) pagefault_disable(); idx = (addr >> PAGE_SHIFT) & (FIX_N_COLOURS - 1); -#ifdef CONFIG_MIPS_MT_SMTC - idx += FIX_N_COLOURS * smp_processor_id() + - (in_interrupt() ? (FIX_N_COLOURS * NR_CPUS) : 0); -#else idx += in_interrupt() ? FIX_N_COLOURS : 0; -#endif vaddr = __fix_to_virt(FIX_CMAP_END - idx); - pte = mk_pte(page, PAGE_KERNEL); + pte = mk_pte(page, prot); #if defined(CONFIG_64BIT_PHYS_ADDR) && defined(CONFIG_CPU_MIPS32) entrylo = pte.pte_high; #else entrylo = pte_to_entrylo(pte_val(pte)); #endif - ENTER_CRITICAL(flags); + local_irq_save(flags); old_ctx = read_c0_entryhi(); write_c0_entryhi(vaddr & (PAGE_MASK << 1)); write_c0_entrylo0(entrylo); write_c0_entrylo1(entrylo); -#ifdef CONFIG_MIPS_MT_SMTC - set_pte(kmap_coherent_pte - (FIX_CMAP_END - idx), pte); - /* preload TLB instead of local_flush_tlb_one() */ - mtc0_tlbw_hazard(); - tlb_probe(); - tlb_probe_hazard(); - tlbidx = read_c0_index(); - mtc0_tlbw_hazard(); - if (tlbidx < 0) - tlb_write_random(); - else - tlb_write_indexed(); -#else tlbidx = read_c0_wired(); write_c0_wired(tlbidx + 1); write_c0_index(tlbidx); mtc0_tlbw_hazard(); tlb_write_indexed(); -#endif tlbw_use_hazard(); write_c0_entryhi(old_ctx); - EXIT_CRITICAL(flags); + local_irq_restore(flags); return (void*) vaddr; } +void *kmap_coherent(struct page *page, unsigned long addr) +{ + return __kmap_pgprot(page, addr, PAGE_KERNEL); +} + +void *kmap_noncoherent(struct page *page, unsigned long addr) +{ + return __kmap_pgprot(page, addr, PAGE_KERNEL_NC); +} + void kunmap_coherent(void) { -#ifndef CONFIG_MIPS_MT_SMTC unsigned int wired; unsigned long flags, old_ctx; - ENTER_CRITICAL(flags); + local_irq_save(flags); old_ctx = read_c0_entryhi(); wired = read_c0_wired() - 1; write_c0_wired(wired); @@ -189,8 +144,7 @@ void kunmap_coherent(void) tlb_write_indexed(); tlbw_use_hazard(); write_c0_entryhi(old_ctx); - EXIT_CRITICAL(flags); -#endif + local_irq_restore(flags); pagefault_enable(); } @@ -256,7 +210,7 @@ EXPORT_SYMBOL_GPL(copy_from_user_page); void __init fixrange_init(unsigned long start, unsigned long end, pgd_t *pgd_base) { -#if defined(CONFIG_HIGHMEM) || defined(CONFIG_MIPS_MT_SMTC) +#ifdef CONFIG_HIGHMEM pgd_t *pgd; pud_t *pud; pmd_t *pmd; @@ -327,8 +281,6 @@ void __init paging_init(void) #ifdef CONFIG_HIGHMEM kmap_init(); #endif - kmap_coherent_init(); - #ifdef CONFIG_ZONE_DMA max_zone_pfns[ZONE_DMA] = MAX_DMA_PFN; #endif diff --git a/arch/mips/mm/tlb-r4k.c b/arch/mips/mm/tlb-r4k.c index eeaf50f5df2b..3914e27456f2 100644 --- a/arch/mips/mm/tlb-r4k.c +++ b/arch/mips/mm/tlb-r4k.c @@ -8,6 +8,7 @@ * Carsten Langgaard, carstenl@mips.com * Copyright (C) 2002 MIPS Technologies, Inc. All rights reserved. */ +#include <linux/cpu_pm.h> #include <linux/init.h> #include <linux/sched.h> #include <linux/smp.h> @@ -25,28 +26,6 @@ extern void build_tlb_refill_handler(void); -/* Atomicity and interruptability */ -#ifdef CONFIG_MIPS_MT_SMTC - -#include <asm/smtc.h> -#include <asm/mipsmtregs.h> - -#define ENTER_CRITICAL(flags) \ - { \ - unsigned int mvpflags; \ - local_irq_save(flags);\ - mvpflags = dvpe() -#define EXIT_CRITICAL(flags) \ - evpe(mvpflags); \ - local_irq_restore(flags); \ - } -#else - -#define ENTER_CRITICAL(flags) local_irq_save(flags) -#define EXIT_CRITICAL(flags) local_irq_restore(flags) - -#endif /* CONFIG_MIPS_MT_SMTC */ - /* * LOONGSON2/3 has a 4 entry itlb which is a subset of dtlb, * unfortunately, itlb is not totally transparent to software. @@ -75,7 +54,7 @@ void local_flush_tlb_all(void) unsigned long old_ctx; int entry, ftlbhighset; - ENTER_CRITICAL(flags); + local_irq_save(flags); /* Save old context and create impossible VPN2 value */ old_ctx = read_c0_entryhi(); write_c0_entrylo0(0); @@ -112,7 +91,7 @@ void local_flush_tlb_all(void) tlbw_use_hazard(); write_c0_entryhi(old_ctx); flush_itlb(); - EXIT_CRITICAL(flags); + local_irq_restore(flags); } EXPORT_SYMBOL(local_flush_tlb_all); @@ -142,7 +121,7 @@ void local_flush_tlb_range(struct vm_area_struct *vma, unsigned long start, if (cpu_context(cpu, mm) != 0) { unsigned long size, flags; - ENTER_CRITICAL(flags); + local_irq_save(flags); start = round_down(start, PAGE_SIZE << 1); end = round_up(end, PAGE_SIZE << 1); size = (end - start) >> (PAGE_SHIFT + 1); @@ -176,7 +155,7 @@ void local_flush_tlb_range(struct vm_area_struct *vma, unsigned long start, drop_mmu_context(mm, cpu); } flush_itlb(); - EXIT_CRITICAL(flags); + local_irq_restore(flags); } } @@ -184,7 +163,7 @@ void local_flush_tlb_kernel_range(unsigned long start, unsigned long end) { unsigned long size, flags; - ENTER_CRITICAL(flags); + local_irq_save(flags); size = (end - start + (PAGE_SIZE - 1)) >> PAGE_SHIFT; size = (size + 1) >> 1; if (size <= (current_cpu_data.tlbsizeftlbsets ? @@ -220,7 +199,7 @@ void local_flush_tlb_kernel_range(unsigned long start, unsigned long end) local_flush_tlb_all(); } flush_itlb(); - EXIT_CRITICAL(flags); + local_irq_restore(flags); } void local_flush_tlb_page(struct vm_area_struct *vma, unsigned long page) @@ -233,7 +212,7 @@ void local_flush_tlb_page(struct vm_area_struct *vma, unsigned long page) newpid = cpu_asid(cpu, vma->vm_mm); page &= (PAGE_MASK << 1); - ENTER_CRITICAL(flags); + local_irq_save(flags); oldpid = read_c0_entryhi(); write_c0_entryhi(page | newpid); mtc0_tlbw_hazard(); @@ -253,7 +232,7 @@ void local_flush_tlb_page(struct vm_area_struct *vma, unsigned long page) finish: write_c0_entryhi(oldpid); flush_itlb_vm(vma); - EXIT_CRITICAL(flags); + local_irq_restore(flags); } } @@ -266,7 +245,7 @@ void local_flush_tlb_one(unsigned long page) unsigned long flags; int oldpid, idx; - ENTER_CRITICAL(flags); + local_irq_save(flags); oldpid = read_c0_entryhi(); page &= (PAGE_MASK << 1); write_c0_entryhi(page); @@ -285,7 +264,7 @@ void local_flush_tlb_one(unsigned long page) } write_c0_entryhi(oldpid); flush_itlb(); - EXIT_CRITICAL(flags); + local_irq_restore(flags); } /* @@ -308,7 +287,7 @@ void __update_tlb(struct vm_area_struct * vma, unsigned long address, pte_t pte) if (current->active_mm != vma->vm_mm) return; - ENTER_CRITICAL(flags); + local_irq_save(flags); pid = read_c0_entryhi() & ASID_MASK; address &= (PAGE_MASK << 1); @@ -358,7 +337,7 @@ void __update_tlb(struct vm_area_struct * vma, unsigned long address, pte_t pte) } tlbw_use_hazard(); flush_itlb_vm(vma); - EXIT_CRITICAL(flags); + local_irq_restore(flags); } void add_wired_entry(unsigned long entrylo0, unsigned long entrylo1, @@ -369,7 +348,7 @@ void add_wired_entry(unsigned long entrylo0, unsigned long entrylo1, unsigned long old_pagemask; unsigned long old_ctx; - ENTER_CRITICAL(flags); + local_irq_save(flags); /* Save old context and create impossible VPN2 value */ old_ctx = read_c0_entryhi(); old_pagemask = read_c0_pagemask(); @@ -389,7 +368,7 @@ void add_wired_entry(unsigned long entrylo0, unsigned long entrylo1, tlbw_use_hazard(); /* What is the hazard here? */ write_c0_pagemask(old_pagemask); local_flush_tlb_all(); - EXIT_CRITICAL(flags); + local_irq_restore(flags); } #ifdef CONFIG_TRANSPARENT_HUGEPAGE @@ -399,13 +378,13 @@ int __init has_transparent_hugepage(void) unsigned int mask; unsigned long flags; - ENTER_CRITICAL(flags); + local_irq_save(flags); write_c0_pagemask(PM_HUGE_MASK); back_to_back_c0_hazard(); mask = read_c0_pagemask(); write_c0_pagemask(PM_DEFAULT_MASK); - EXIT_CRITICAL(flags); + local_irq_restore(flags); return mask == PM_HUGE_MASK; } @@ -421,7 +400,10 @@ static int __init set_ntlb(char *str) __setup("ntlb=", set_ntlb); -void tlb_init(void) +/* + * Configure TLB (for init or after a CPU has been powered off). + */ +static void r4k_tlb_configure(void) { /* * You should never change this register: @@ -453,6 +435,11 @@ void tlb_init(void) local_flush_tlb_all(); /* Did I tell you that ARC SUCKS? */ +} + +void tlb_init(void) +{ + r4k_tlb_configure(); if (ntlb) { if (ntlb > 1 && ntlb <= current_cpu_data.tlbsize) { @@ -466,3 +453,26 @@ void tlb_init(void) build_tlb_refill_handler(); } + +static int r4k_tlb_pm_notifier(struct notifier_block *self, unsigned long cmd, + void *v) +{ + switch (cmd) { + case CPU_PM_ENTER_FAILED: + case CPU_PM_EXIT: + r4k_tlb_configure(); + break; + } + + return NOTIFY_OK; +} + +static struct notifier_block r4k_tlb_pm_notifier_block = { + .notifier_call = r4k_tlb_pm_notifier, +}; + +static int __init r4k_tlb_init_pm(void) +{ + return cpu_pm_register_notifier(&r4k_tlb_pm_notifier_block); +} +arch_initcall(r4k_tlb_init_pm); diff --git a/arch/mips/mm/tlbex.c b/arch/mips/mm/tlbex.c index f99ec587b151..e80e10bafc83 100644 --- a/arch/mips/mm/tlbex.c +++ b/arch/mips/mm/tlbex.c @@ -1256,7 +1256,7 @@ static void build_r4000_tlb_refill_handler(void) memset(relocs, 0, sizeof(relocs)); memset(final_handler, 0, sizeof(final_handler)); - if ((scratch_reg >= 0 || scratchpad_available()) && use_bbit_insns()) { + if (IS_ENABLED(CONFIG_64BIT) && (scratch_reg >= 0 || scratchpad_available()) && use_bbit_insns()) { htlb_info = build_fast_tlb_refill_handler(&p, &l, &r, K0, K1, scratch_reg); vmalloc_mode = refill_scratch; diff --git a/arch/mips/mm/uasm-micromips.c b/arch/mips/mm/uasm-micromips.c index b8d580ca02e5..775c2800cba2 100644 --- a/arch/mips/mm/uasm-micromips.c +++ b/arch/mips/mm/uasm-micromips.c @@ -63,6 +63,7 @@ static struct insn insn_table_MM[] = { { insn_cache, M(mm_pool32b_op, 0, 0, mm_cache_func, 0, 0), RT | RS | SIMM }, { insn_daddu, 0, 0 }, { insn_daddiu, 0, 0 }, + { insn_divu, M(mm_pool32a_op, 0, 0, 0, mm_divu_op, mm_pool32axf_op), RT | RS }, { insn_dmfc0, 0, 0 }, { insn_dmtc0, 0, 0 }, { insn_dsll, 0, 0 }, @@ -78,14 +79,20 @@ static struct insn insn_table_MM[] = { { insn_ext, M(mm_pool32a_op, 0, 0, 0, 0, mm_ext_op), RT | RS | RD | RE }, { insn_j, M(mm_j32_op, 0, 0, 0, 0, 0), JIMM }, { insn_jal, M(mm_jal32_op, 0, 0, 0, 0, 0), JIMM }, + { insn_jalr, M(mm_pool32a_op, 0, 0, 0, mm_jalr_op, mm_pool32axf_op), RT | RS }, { insn_jr, M(mm_pool32a_op, 0, 0, 0, mm_jalr_op, mm_pool32axf_op), RS }, + { insn_lb, M(mm_lb32_op, 0, 0, 0, 0, 0), RT | RS | SIMM }, { insn_ld, 0, 0 }, + { insn_lh, M(mm_lh32_op, 0, 0, 0, 0, 0), RS | RS | SIMM }, { insn_ll, M(mm_pool32c_op, 0, 0, (mm_ll_func << 1), 0, 0), RS | RT | SIMM }, { insn_lld, 0, 0 }, { insn_lui, M(mm_pool32i_op, mm_lui_op, 0, 0, 0, 0), RS | SIMM }, { insn_lw, M(mm_lw32_op, 0, 0, 0, 0, 0), RT | RS | SIMM }, { insn_mfc0, M(mm_pool32a_op, 0, 0, 0, mm_mfc0_op, mm_pool32axf_op), RT | RS | RD }, + { insn_mfhi, M(mm_pool32a_op, 0, 0, 0, mm_mfhi32_op, mm_pool32axf_op), RS }, + { insn_mflo, M(mm_pool32a_op, 0, 0, 0, mm_mflo32_op, mm_pool32axf_op), RS }, { insn_mtc0, M(mm_pool32a_op, 0, 0, 0, mm_mtc0_op, mm_pool32axf_op), RT | RS | RD }, + { insn_mul, M(mm_pool32a_op, 0, 0, 0, 0, mm_mul_op), RT | RS | RD }, { insn_or, M(mm_pool32a_op, 0, 0, 0, 0, mm_or32_op), RT | RS | RD }, { insn_ori, M(mm_ori32_op, 0, 0, 0, 0, 0), RT | RS | UIMM }, { insn_pref, M(mm_pool32c_op, 0, 0, (mm_pref_func << 1), 0, 0), RT | RS | SIMM }, @@ -94,15 +101,22 @@ static struct insn insn_table_MM[] = { { insn_scd, 0, 0 }, { insn_sd, 0, 0 }, { insn_sll, M(mm_pool32a_op, 0, 0, 0, 0, mm_sll32_op), RT | RS | RD }, + { insn_sllv, M(mm_pool32a_op, 0, 0, 0, 0, mm_sllv32_op), RT | RS | RD }, + { insn_sltiu, M(mm_sltiu32_op, 0, 0, 0, 0, 0), RT | RS | SIMM }, + { insn_sltu, M(mm_pool32a_op, 0, 0, 0, 0, mm_sltu_op), RT | RS | RD }, { insn_sra, M(mm_pool32a_op, 0, 0, 0, 0, mm_sra_op), RT | RS | RD }, { insn_srl, M(mm_pool32a_op, 0, 0, 0, 0, mm_srl32_op), RT | RS | RD }, + { insn_srlv, M(mm_pool32a_op, 0, 0, 0, 0, mm_srlv32_op), RT | RS | RD }, { insn_rotr, M(mm_pool32a_op, 0, 0, 0, 0, mm_rotr_op), RT | RS | RD }, { insn_subu, M(mm_pool32a_op, 0, 0, 0, 0, mm_subu32_op), RT | RS | RD }, { insn_sw, M(mm_sw32_op, 0, 0, 0, 0, 0), RT | RS | SIMM }, + { insn_sync, M(mm_pool32a_op, 0, 0, 0, mm_sync_op, mm_pool32axf_op), RS }, { insn_tlbp, M(mm_pool32a_op, 0, 0, 0, mm_tlbp_op, mm_pool32axf_op), 0 }, { insn_tlbr, M(mm_pool32a_op, 0, 0, 0, mm_tlbr_op, mm_pool32axf_op), 0 }, { insn_tlbwi, M(mm_pool32a_op, 0, 0, 0, mm_tlbwi_op, mm_pool32axf_op), 0 }, { insn_tlbwr, M(mm_pool32a_op, 0, 0, 0, mm_tlbwr_op, mm_pool32axf_op), 0 }, + { insn_wait, M(mm_pool32a_op, 0, 0, 0, mm_wait_op, mm_pool32axf_op), SCIMM }, + { insn_wsbh, M(mm_pool32a_op, 0, 0, 0, mm_wsbh_op, mm_pool32axf_op), RT | RS }, { insn_xor, M(mm_pool32a_op, 0, 0, 0, 0, mm_xor32_op), RT | RS | RD }, { insn_xori, M(mm_xori32_op, 0, 0, 0, 0, 0), RT | RS | UIMM }, { insn_dins, 0, 0 }, diff --git a/arch/mips/mm/uasm-mips.c b/arch/mips/mm/uasm-mips.c index 3abd609518c9..38792c2364f5 100644 --- a/arch/mips/mm/uasm-mips.c +++ b/arch/mips/mm/uasm-mips.c @@ -67,6 +67,7 @@ static struct insn insn_table[] = { { insn_daddu, M(spec_op, 0, 0, 0, 0, daddu_op), RS | RT | RD }, { insn_dinsm, M(spec3_op, 0, 0, 0, 0, dinsm_op), RS | RT | RD | RE }, { insn_dins, M(spec3_op, 0, 0, 0, 0, dins_op), RS | RT | RD | RE }, + { insn_divu, M(spec_op, 0, 0, 0, 0, divu_op), RS | RT }, { insn_dmfc0, M(cop0_op, dmfc_op, 0, 0, 0, 0), RT | RD | SET}, { insn_dmtc0, M(cop0_op, dmtc_op, 0, 0, 0, 0), RT | RD | SET}, { insn_drotr32, M(spec_op, 1, 0, 0, 0, dsrl32_op), RT | RD | RE }, @@ -82,17 +83,23 @@ static struct insn insn_table[] = { { insn_ins, M(spec3_op, 0, 0, 0, 0, ins_op), RS | RT | RD | RE }, { insn_j, M(j_op, 0, 0, 0, 0, 0), JIMM }, { insn_jal, M(jal_op, 0, 0, 0, 0, 0), JIMM }, + { insn_jalr, M(spec_op, 0, 0, 0, 0, jalr_op), RS | RD }, { insn_j, M(j_op, 0, 0, 0, 0, 0), JIMM }, { insn_jr, M(spec_op, 0, 0, 0, 0, jr_op), RS }, + { insn_lb, M(lb_op, 0, 0, 0, 0, 0), RS | RT | SIMM }, { insn_ld, M(ld_op, 0, 0, 0, 0, 0), RS | RT | SIMM }, { insn_ldx, M(spec3_op, 0, 0, 0, ldx_op, lx_op), RS | RT | RD }, + { insn_lh, M(lw_op, 0, 0, 0, 0, 0), RS | RT | SIMM }, { insn_lld, M(lld_op, 0, 0, 0, 0, 0), RS | RT | SIMM }, { insn_ll, M(ll_op, 0, 0, 0, 0, 0), RS | RT | SIMM }, { insn_lui, M(lui_op, 0, 0, 0, 0, 0), RT | SIMM }, { insn_lw, M(lw_op, 0, 0, 0, 0, 0), RS | RT | SIMM }, { insn_lwx, M(spec3_op, 0, 0, 0, lwx_op, lx_op), RS | RT | RD }, { insn_mfc0, M(cop0_op, mfc_op, 0, 0, 0, 0), RT | RD | SET}, + { insn_mfhi, M(spec_op, 0, 0, 0, 0, mfhi_op), RD }, + { insn_mflo, M(spec_op, 0, 0, 0, 0, mflo_op), RD }, { insn_mtc0, M(cop0_op, mtc_op, 0, 0, 0, 0), RT | RD | SET}, + { insn_mul, M(spec2_op, 0, 0, 0, 0, mul_op), RS | RT | RD}, { insn_ori, M(ori_op, 0, 0, 0, 0, 0), RS | RT | UIMM }, { insn_or, M(spec_op, 0, 0, 0, 0, or_op), RS | RT | RD }, { insn_pref, M(pref_op, 0, 0, 0, 0, 0), RS | RT | SIMM }, @@ -102,17 +109,25 @@ static struct insn insn_table[] = { { insn_sc, M(sc_op, 0, 0, 0, 0, 0), RS | RT | SIMM }, { insn_sd, M(sd_op, 0, 0, 0, 0, 0), RS | RT | SIMM }, { insn_sll, M(spec_op, 0, 0, 0, 0, sll_op), RT | RD | RE }, + { insn_sllv, M(spec_op, 0, 0, 0, 0, sllv_op), RS | RT | RD }, + { insn_sltiu, M(sltiu_op, 0, 0, 0, 0, 0), RS | RT | SIMM }, + { insn_sltu, M(spec_op, 0, 0, 0, 0, sltu_op), RS | RT | RD }, { insn_sra, M(spec_op, 0, 0, 0, 0, sra_op), RT | RD | RE }, { insn_srl, M(spec_op, 0, 0, 0, 0, srl_op), RT | RD | RE }, + { insn_srlv, M(spec_op, 0, 0, 0, 0, srlv_op), RS | RT | RD }, { insn_subu, M(spec_op, 0, 0, 0, 0, subu_op), RS | RT | RD }, { insn_sw, M(sw_op, 0, 0, 0, 0, 0), RS | RT | SIMM }, + { insn_sync, M(spec_op, 0, 0, 0, 0, sync_op), RE }, { insn_syscall, M(spec_op, 0, 0, 0, 0, syscall_op), SCIMM}, { insn_tlbp, M(cop0_op, cop_op, 0, 0, 0, tlbp_op), 0 }, { insn_tlbr, M(cop0_op, cop_op, 0, 0, 0, tlbr_op), 0 }, { insn_tlbwi, M(cop0_op, cop_op, 0, 0, 0, tlbwi_op), 0 }, { insn_tlbwr, M(cop0_op, cop_op, 0, 0, 0, tlbwr_op), 0 }, + { insn_wait, M(cop0_op, cop_op, 0, 0, 0, wait_op), SCIMM }, + { insn_wsbh, M(spec3_op, 0, 0, 0, wsbh_op, bshfl_op), RT | RD }, { insn_xori, M(xori_op, 0, 0, 0, 0, 0), RS | RT | UIMM }, { insn_xor, M(spec_op, 0, 0, 0, 0, xor_op), RS | RT | RD }, + { insn_yield, M(spec3_op, 0, 0, 0, 0, yield_op), RS | RD }, { insn_invalid, 0, 0 } }; diff --git a/arch/mips/mm/uasm.c b/arch/mips/mm/uasm.c index b9d14b6c7f58..00515805fe41 100644 --- a/arch/mips/mm/uasm.c +++ b/arch/mips/mm/uasm.c @@ -47,14 +47,16 @@ enum opcode { insn_addiu, insn_addu, insn_and, insn_andi, insn_bbit0, insn_bbit1, insn_beq, insn_beql, insn_bgez, insn_bgezl, insn_bltz, insn_bltzl, insn_bne, insn_cache, insn_daddiu, insn_daddu, insn_dins, insn_dinsm, - insn_dmfc0, insn_dmtc0, insn_drotr, insn_drotr32, insn_dsll, + insn_divu, insn_dmfc0, insn_dmtc0, insn_drotr, insn_drotr32, insn_dsll, insn_dsll32, insn_dsra, insn_dsrl, insn_dsrl32, insn_dsubu, insn_eret, - insn_ext, insn_ins, insn_j, insn_jal, insn_jr, insn_ld, insn_ldx, - insn_ll, insn_lld, insn_lui, insn_lw, insn_lwx, insn_mfc0, insn_mtc0, + insn_ext, insn_ins, insn_j, insn_jal, insn_jalr, insn_jr, insn_lb, + insn_ld, insn_ldx, insn_lh, insn_ll, insn_lld, insn_lui, insn_lw, + insn_lwx, insn_mfc0, insn_mfhi, insn_mflo, insn_mtc0, insn_mul, insn_or, insn_ori, insn_pref, insn_rfe, insn_rotr, insn_sc, insn_scd, - insn_sd, insn_sll, insn_sra, insn_srl, insn_subu, insn_sw, - insn_syscall, insn_tlbp, insn_tlbr, insn_tlbwi, insn_tlbwr, insn_xor, - insn_xori, + insn_sd, insn_sll, insn_sllv, insn_sltiu, insn_sltu, insn_sra, + insn_srl, insn_srlv, insn_subu, insn_sw, insn_sync, insn_syscall, + insn_tlbp, insn_tlbr, insn_tlbwi, insn_tlbwr, insn_wait, insn_wsbh, + insn_xor, insn_xori, insn_yield, }; struct insn { @@ -144,6 +146,13 @@ Ip_u2u1u3(op) \ } \ UASM_EXPORT_SYMBOL(uasm_i##op); +#define I_u3u2u1(op) \ +Ip_u3u2u1(op) \ +{ \ + build_insn(buf, insn##op, c, b, a); \ +} \ +UASM_EXPORT_SYMBOL(uasm_i##op); + #define I_u3u1u2(op) \ Ip_u3u1u2(op) \ { \ @@ -200,6 +209,13 @@ Ip_u1u2(op) \ } \ UASM_EXPORT_SYMBOL(uasm_i##op); +#define I_u2u1(op) \ +Ip_u1u2(op) \ +{ \ + build_insn(buf, insn##op, b, a); \ +} \ +UASM_EXPORT_SYMBOL(uasm_i##op); + #define I_u1s2(op) \ Ip_u1s2(op) \ { \ @@ -237,6 +253,7 @@ I_u1u2u3(_dmfc0) I_u1u2u3(_dmtc0) I_u2u1s3(_daddiu) I_u3u1u2(_daddu) +I_u1u2(_divu) I_u2u1u3(_dsll) I_u2u1u3(_dsll32) I_u2u1u3(_dsra) @@ -250,14 +267,20 @@ I_u2u1msbdu3(_ext) I_u2u1msbu3(_ins) I_u1(_j) I_u1(_jal) +I_u2u1(_jalr) I_u1(_jr) +I_u2s3u1(_lb) I_u2s3u1(_ld) +I_u2s3u1(_lh) I_u2s3u1(_ll) I_u2s3u1(_lld) I_u1s2(_lui) I_u2s3u1(_lw) I_u1u2u3(_mfc0) +I_u1(_mfhi) +I_u1(_mflo) I_u1u2u3(_mtc0) +I_u3u1u2(_mul) I_u2u1u3(_ori) I_u3u1u2(_or) I_0(_rfe) @@ -265,17 +288,25 @@ I_u2s3u1(_sc) I_u2s3u1(_scd) I_u2s3u1(_sd) I_u2u1u3(_sll) +I_u3u2u1(_sllv) +I_u2u1s3(_sltiu) +I_u3u1u2(_sltu) I_u2u1u3(_sra) I_u2u1u3(_srl) +I_u3u2u1(_srlv) I_u2u1u3(_rotr) I_u3u1u2(_subu) I_u2s3u1(_sw) +I_u1(_sync) I_0(_tlbp) I_0(_tlbr) I_0(_tlbwi) I_0(_tlbwr) +I_u1(_wait); +I_u2u1(_wsbh) I_u3u1u2(_xor) I_u2u1u3(_xori) +I_u2u1(_yield) I_u2u1msbu3(_dins); I_u2u1msb32u3(_dinsm); I_u1(_syscall); @@ -469,6 +500,14 @@ void ISAFUNC(uasm_il_b)(u32 **p, struct uasm_reloc **r, int lid) } UASM_EXPORT_SYMBOL(ISAFUNC(uasm_il_b)); +void ISAFUNC(uasm_il_beq)(u32 **p, struct uasm_reloc **r, unsigned int r1, + unsigned int r2, int lid) +{ + uasm_r_mips_pc16(r, *p, lid); + ISAFUNC(uasm_i_beq)(p, r1, r2, 0); +} +UASM_EXPORT_SYMBOL(ISAFUNC(uasm_il_beq)); + void ISAFUNC(uasm_il_beqz)(u32 **p, struct uasm_reloc **r, unsigned int reg, int lid) { diff --git a/arch/mips/mti-malta/Makefile b/arch/mips/mti-malta/Makefile index eae0ba3876d9..b9510ea8db56 100644 --- a/arch/mips/mti-malta/Makefile +++ b/arch/mips/mti-malta/Makefile @@ -9,5 +9,4 @@ obj-y := malta-amon.o malta-display.o malta-init.o \ malta-int.o malta-memory.o malta-platform.o \ malta-reset.o malta-setup.o malta-time.o -# FIXME FIXME FIXME -obj-$(CONFIG_MIPS_MT_SMTC) += malta-smtc.o +obj-$(CONFIG_MIPS_MALTA_PM) += malta-pm.o diff --git a/arch/mips/mti-malta/malta-init.c b/arch/mips/mti-malta/malta-init.c index 4f9e44d358b7..0f60256d3784 100644 --- a/arch/mips/mti-malta/malta-init.c +++ b/arch/mips/mti-malta/malta-init.c @@ -116,8 +116,6 @@ phys_t mips_cpc_default_phys_base(void) return CPC_BASE_ADDR; } -extern struct plat_smp_ops msmtc_smp_ops; - void __init prom_init(void) { mips_display_message("LINUX"); @@ -304,8 +302,4 @@ mips_pci_controller: return; if (!register_vsmp_smp_ops()) return; - -#ifdef CONFIG_MIPS_MT_SMTC - register_smp_ops(&msmtc_smp_ops); -#endif } diff --git a/arch/mips/mti-malta/malta-int.c b/arch/mips/mti-malta/malta-int.c index b71ee809191a..ecc2785f7858 100644 --- a/arch/mips/mti-malta/malta-int.c +++ b/arch/mips/mti-malta/malta-int.c @@ -504,28 +504,9 @@ void __init arch_init_irq(void) } else if (cpu_has_vint) { set_vi_handler(MIPSCPU_INT_I8259A, malta_hw0_irqdispatch); set_vi_handler(MIPSCPU_INT_COREHI, corehi_irqdispatch); -#ifdef CONFIG_MIPS_MT_SMTC - setup_irq_smtc(MIPS_CPU_IRQ_BASE+MIPSCPU_INT_I8259A, &i8259irq, - (0x100 << MIPSCPU_INT_I8259A)); - setup_irq_smtc(MIPS_CPU_IRQ_BASE+MIPSCPU_INT_COREHI, - &corehi_irqaction, (0x100 << MIPSCPU_INT_COREHI)); - /* - * Temporary hack to ensure that the subsidiary device - * interrupts coing in via the i8259A, but associated - * with low IRQ numbers, will restore the Status.IM - * value associated with the i8259A. - */ - { - int i; - - for (i = 0; i < 16; i++) - irq_hwmask[i] = (0x100 << MIPSCPU_INT_I8259A); - } -#else /* Not SMTC */ setup_irq(MIPS_CPU_IRQ_BASE+MIPSCPU_INT_I8259A, &i8259irq); setup_irq(MIPS_CPU_IRQ_BASE+MIPSCPU_INT_COREHI, &corehi_irqaction); -#endif /* CONFIG_MIPS_MT_SMTC */ } else { setup_irq(MIPS_CPU_IRQ_BASE+MIPSCPU_INT_I8259A, &i8259irq); setup_irq(MIPS_CPU_IRQ_BASE+MIPSCPU_INT_COREHI, diff --git a/arch/mips/mti-malta/malta-memory.c b/arch/mips/mti-malta/malta-memory.c index f2364e419682..6d9773096750 100644 --- a/arch/mips/mti-malta/malta-memory.c +++ b/arch/mips/mti-malta/malta-memory.c @@ -26,8 +26,8 @@ unsigned long physical_memsize = 0L; fw_memblock_t * __init fw_getmdesc(int eva) { - char *memsize_str, *ememsize_str __maybe_unused = NULL, *ptr; - unsigned long memsize = 0, ememsize __maybe_unused = 0; + char *memsize_str, *ememsize_str = NULL, *ptr; + unsigned long memsize = 0, ememsize = 0; static char cmdline[COMMAND_LINE_SIZE] __initdata; int tmp; diff --git a/arch/mips/mti-malta/malta-pm.c b/arch/mips/mti-malta/malta-pm.c new file mode 100644 index 000000000000..c1e456c01a44 --- /dev/null +++ b/arch/mips/mti-malta/malta-pm.c @@ -0,0 +1,96 @@ +/* + * Copyright (C) 2014 Imagination Technologies + * Author: Paul Burton <paul.burton@imgtec.com> + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the + * Free Software Foundation; either version 2 of the License, or (at your + * option) any later version. + */ + +#include <linux/delay.h> +#include <linux/init.h> +#include <linux/io.h> +#include <linux/pci.h> + +#include <asm/mach-malta/malta-pm.h> + +static struct pci_bus *pm_pci_bus; +static resource_size_t pm_io_offset; + +int mips_pm_suspend(unsigned state) +{ + int spec_devid; + u16 sts; + + if (!pm_pci_bus || !pm_io_offset) + return -ENODEV; + + /* Ensure the power button status is clear */ + while (1) { + sts = inw(pm_io_offset + PIIX4_FUNC3IO_PMSTS); + if (!(sts & PIIX4_FUNC3IO_PMSTS_PWRBTN_STS)) + break; + outw(sts, pm_io_offset + PIIX4_FUNC3IO_PMSTS); + } + + /* Enable entry to suspend */ + outw(state | PIIX4_FUNC3IO_PMCNTRL_SUS_EN, + pm_io_offset + PIIX4_FUNC3IO_PMCNTRL); + + /* If the special cycle occurs too soon this doesn't work... */ + mdelay(10); + + /* + * The PIIX4 will enter the suspend state only after seeing a special + * cycle with the correct magic data on the PCI bus. Generate that + * cycle now. + */ + spec_devid = PCI_DEVID(0, PCI_DEVFN(0x1f, 0x7)); + pci_bus_write_config_dword(pm_pci_bus, spec_devid, 0, + PIIX4_SUSPEND_MAGIC); + + /* Give the system some time to power down */ + mdelay(1000); + + return 0; +} + +static int __init malta_pm_setup(void) +{ + struct pci_dev *dev; + int res, io_region = PCI_BRIDGE_RESOURCES; + + /* Find a reference to the PCI bus */ + pm_pci_bus = pci_find_next_bus(NULL); + if (!pm_pci_bus) { + pr_warn("malta-pm: failed to find reference to PCI bus\n"); + return -ENODEV; + } + + /* Find the PIIX4 PM device */ + dev = pci_get_subsys(PCI_VENDOR_ID_INTEL, + PCI_DEVICE_ID_INTEL_82371AB_3, PCI_ANY_ID, + PCI_ANY_ID, NULL); + if (!dev) { + pr_warn("malta-pm: failed to find PIIX4 PM\n"); + return -ENODEV; + } + + /* Request access to the PIIX4 PM IO registers */ + res = pci_request_region(dev, io_region, "PIIX4 PM IO registers"); + if (res) { + pr_warn("malta-pm: failed to request PM IO registers (%d)\n", + res); + pci_dev_put(dev); + return -ENODEV; + } + + /* Find the offset to the PIIX4 PM IO registers */ + pm_io_offset = pci_resource_start(dev, io_region); + + pci_dev_put(dev); + return 0; +} + +late_initcall(malta_pm_setup); diff --git a/arch/mips/mti-malta/malta-reset.c b/arch/mips/mti-malta/malta-reset.c index d627d4b2b47f..2fd2cc2c5034 100644 --- a/arch/mips/mti-malta/malta-reset.c +++ b/arch/mips/mti-malta/malta-reset.c @@ -10,6 +10,7 @@ #include <linux/pm.h> #include <asm/reboot.h> +#include <asm/mach-malta/malta-pm.h> #define SOFTRES_REG 0x1f000500 #define GORESET 0x42 @@ -24,17 +25,22 @@ static void mips_machine_restart(char *command) static void mips_machine_halt(void) { - unsigned int __iomem *softres_reg = - ioremap(SOFTRES_REG, sizeof(unsigned int)); + while (true); +} - __raw_writel(GORESET, softres_reg); +static void mips_machine_power_off(void) +{ + mips_pm_suspend(PIIX4_FUNC3IO_PMCNTRL_SUS_TYP_SOFF); + + pr_info("Failed to power down, resetting\n"); + mips_machine_restart(NULL); } static int __init mips_reboot_setup(void) { _machine_restart = mips_machine_restart; _machine_halt = mips_machine_halt; - pm_power_off = mips_machine_halt; + pm_power_off = mips_machine_power_off; return 0; } diff --git a/arch/mips/mti-malta/malta-setup.c b/arch/mips/mti-malta/malta-setup.c index bf621516afff..db7c9e5826a6 100644 --- a/arch/mips/mti-malta/malta-setup.c +++ b/arch/mips/mti-malta/malta-setup.c @@ -77,11 +77,7 @@ const char *get_system_type(void) return "MIPS Malta"; } -#if defined(CONFIG_MIPS_MT_SMTC) -const char display_string[] = " SMTC LINUX ON MALTA "; -#else const char display_string[] = " LINUX ON MALTA "; -#endif /* CONFIG_MIPS_MT_SMTC */ #ifdef CONFIG_BLK_DEV_FD static void __init fd_activate(void) diff --git a/arch/mips/mti-malta/malta-smtc.c b/arch/mips/mti-malta/malta-smtc.c deleted file mode 100644 index c4849904f013..000000000000 --- a/arch/mips/mti-malta/malta-smtc.c +++ /dev/null @@ -1,162 +0,0 @@ -/* - * Malta Platform-specific hooks for SMP operation - */ -#include <linux/irq.h> -#include <linux/init.h> - -#include <asm/mipsregs.h> -#include <asm/mipsmtregs.h> -#include <asm/smtc.h> -#include <asm/smtc_ipi.h> - -/* VPE/SMP Prototype implements platform interfaces directly */ - -/* - * Cause the specified action to be performed on a targeted "CPU" - */ - -static void msmtc_send_ipi_single(int cpu, unsigned int action) -{ - /* "CPU" may be TC of same VPE, VPE of same CPU, or different CPU */ - smtc_send_ipi(cpu, LINUX_SMP_IPI, action); -} - -static void msmtc_send_ipi_mask(const struct cpumask *mask, unsigned int action) -{ - unsigned int i; - - for_each_cpu(i, mask) - msmtc_send_ipi_single(i, action); -} - -/* - * Post-config but pre-boot cleanup entry point - */ -static void msmtc_init_secondary(void) -{ - int myvpe; - - /* Don't enable Malta I/O interrupts (IP2) for secondary VPEs */ - myvpe = read_c0_tcbind() & TCBIND_CURVPE; - if (myvpe != 0) { - /* Ideally, this should be done only once per VPE, but... */ - clear_c0_status(ST0_IM); - set_c0_status((0x100 << cp0_compare_irq) - | (0x100 << MIPS_CPU_IPI_IRQ)); - if (cp0_perfcount_irq >= 0) - set_c0_status(0x100 << cp0_perfcount_irq); - } - - smtc_init_secondary(); -} - -/* - * Platform "CPU" startup hook - */ -static void msmtc_boot_secondary(int cpu, struct task_struct *idle) -{ - smtc_boot_secondary(cpu, idle); -} - -/* - * SMP initialization finalization entry point - */ -static void msmtc_smp_finish(void) -{ - smtc_smp_finish(); -} - -/* - * Hook for after all CPUs are online - */ - -static void msmtc_cpus_done(void) -{ -} - -/* - * Platform SMP pre-initialization - * - * As noted above, we can assume a single CPU for now - * but it may be multithreaded. - */ - -static void __init msmtc_smp_setup(void) -{ - /* - * we won't get the definitive value until - * we've run smtc_prepare_cpus later, but - * we would appear to need an upper bound now. - */ - smp_num_siblings = smtc_build_cpu_map(0); -} - -static void __init msmtc_prepare_cpus(unsigned int max_cpus) -{ - smtc_prepare_cpus(max_cpus); -} - -struct plat_smp_ops msmtc_smp_ops = { - .send_ipi_single = msmtc_send_ipi_single, - .send_ipi_mask = msmtc_send_ipi_mask, - .init_secondary = msmtc_init_secondary, - .smp_finish = msmtc_smp_finish, - .cpus_done = msmtc_cpus_done, - .boot_secondary = msmtc_boot_secondary, - .smp_setup = msmtc_smp_setup, - .prepare_cpus = msmtc_prepare_cpus, -}; - -#ifdef CONFIG_MIPS_MT_SMTC_IRQAFF -/* - * IRQ affinity hook - */ - - -int plat_set_irq_affinity(struct irq_data *d, const struct cpumask *affinity, - bool force) -{ - cpumask_t tmask; - int cpu = 0; - void smtc_set_irq_affinity(unsigned int irq, cpumask_t aff); - - /* - * On the legacy Malta development board, all I/O interrupts - * are routed through the 8259 and combined in a single signal - * to the CPU daughterboard, and on the CoreFPGA2/3 34K models, - * that signal is brought to IP2 of both VPEs. To avoid racing - * concurrent interrupt service events, IP2 is enabled only on - * one VPE, by convention VPE0. So long as no bits are ever - * cleared in the affinity mask, there will never be any - * interrupt forwarding. But as soon as a program or operator - * sets affinity for one of the related IRQs, we need to make - * sure that we don't ever try to forward across the VPE boundary, - * at least not until we engineer a system where the interrupt - * _ack() or _end() function can somehow know that it corresponds - * to an interrupt taken on another VPE, and perform the appropriate - * restoration of Status.IM state using MFTR/MTTR instead of the - * normal local behavior. We also ensure that no attempt will - * be made to forward to an offline "CPU". - */ - - cpumask_copy(&tmask, affinity); - for_each_cpu(cpu, affinity) { - if ((cpu_data[cpu].vpe_id != 0) || !cpu_online(cpu)) - cpu_clear(cpu, tmask); - } - cpumask_copy(d->affinity, &tmask); - - if (cpus_empty(tmask)) - /* - * We could restore a default mask here, but the - * runtime code can anyway deal with the null set - */ - printk(KERN_WARNING - "IRQ affinity leaves no legal CPU for IRQ %d\n", d->irq); - - /* Do any generic SMTC IRQ affinity setup */ - smtc_set_irq_affinity(d->irq, tmask); - - return IRQ_SET_MASK_OK_NOCOPY; -} -#endif /* CONFIG_MIPS_MT_SMTC_IRQAFF */ diff --git a/arch/mips/mti-sead3/sead3-pic32-i2c-drv.c b/arch/mips/mti-sead3/sead3-pic32-i2c-drv.c index b921e5ec507c..80fe194cfa53 100644 --- a/arch/mips/mti-sead3/sead3-pic32-i2c-drv.c +++ b/arch/mips/mti-sead3/sead3-pic32-i2c-drv.c @@ -312,16 +312,13 @@ static int i2c_platform_probe(struct platform_device *pdev) pr_debug("i2c_platform_probe\n"); r = platform_get_resource(pdev, IORESOURCE_MEM, 0); - if (!r) { - ret = -ENODEV; - goto out; - } + if (!r) + return -ENODEV; - priv = kzalloc(sizeof(struct i2c_platform_data), GFP_KERNEL); - if (!priv) { - ret = -ENOMEM; - goto out; - } + priv = devm_kzalloc(&pdev->dev, sizeof(struct i2c_platform_data), + GFP_KERNEL); + if (!priv) + return -ENOMEM; /* FIXME: need to allocate resource in PIC32 space */ #if 0 @@ -330,10 +327,8 @@ static int i2c_platform_probe(struct platform_device *pdev) #else priv->base = r->start; #endif - if (!priv->base) { - ret = -EBUSY; - goto out_mem; - } + if (!priv->base) + return -EBUSY; priv->xfer_timeout = 200; priv->ack_timeout = 200; @@ -348,17 +343,13 @@ static int i2c_platform_probe(struct platform_device *pdev) i2c_platform_setup(priv); ret = i2c_add_numbered_adapter(&priv->adap); - if (ret == 0) { - platform_set_drvdata(pdev, priv); - return 0; + if (ret) { + i2c_platform_disable(priv); + return ret; } - i2c_platform_disable(priv); - -out_mem: - kfree(priv); -out: - return ret; + platform_set_drvdata(pdev, priv); + return 0; } static int i2c_platform_remove(struct platform_device *pdev) @@ -369,7 +360,6 @@ static int i2c_platform_remove(struct platform_device *pdev) platform_set_drvdata(pdev, NULL); i2c_del_adapter(&priv->adap); i2c_platform_disable(priv); - kfree(priv); return 0; } diff --git a/arch/mips/net/Makefile b/arch/mips/net/Makefile new file mode 100644 index 000000000000..ae74b3a91f5c --- /dev/null +++ b/arch/mips/net/Makefile @@ -0,0 +1,3 @@ +# MIPS networking code + +obj-$(CONFIG_BPF_JIT) += bpf_jit.o diff --git a/arch/mips/net/bpf_jit.c b/arch/mips/net/bpf_jit.c new file mode 100644 index 000000000000..a67b9753330b --- /dev/null +++ b/arch/mips/net/bpf_jit.c @@ -0,0 +1,1399 @@ +/* + * Just-In-Time compiler for BPF filters on MIPS + * + * Copyright (c) 2014 Imagination Technologies Ltd. + * Author: Markos Chandras <markos.chandras@imgtec.com> + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the + * Free Software Foundation; version 2 of the License. + */ + +#include <linux/bitops.h> +#include <linux/compiler.h> +#include <linux/errno.h> +#include <linux/filter.h> +#include <linux/if_vlan.h> +#include <linux/kconfig.h> +#include <linux/moduleloader.h> +#include <linux/netdevice.h> +#include <linux/string.h> +#include <linux/slab.h> +#include <linux/types.h> +#include <asm/bitops.h> +#include <asm/cacheflush.h> +#include <asm/cpu-features.h> +#include <asm/uasm.h> + +#include "bpf_jit.h" + +/* ABI + * + * s0 1st scratch register + * s1 2nd scratch register + * s2 offset register + * s3 BPF register A + * s4 BPF register X + * s5 *skb + * s6 *scratch memory + * + * On entry (*bpf_func)(*skb, *filter) + * a0 = MIPS_R_A0 = skb; + * a1 = MIPS_R_A1 = filter; + * + * Stack + * ... + * M[15] + * M[14] + * M[13] + * ... + * M[0] <-- r_M + * saved reg k-1 + * saved reg k-2 + * ... + * saved reg 0 <-- r_sp + * <no argument area> + * + * Packet layout + * + * <--------------------- len ------------------------> + * <--skb-len(r_skb_hl)-->< ----- skb->data_len ------> + * ---------------------------------------------------- + * | skb->data | + * ---------------------------------------------------- + */ + +#define RSIZE (sizeof(unsigned long)) +#define ptr typeof(unsigned long) + +/* ABI specific return values */ +#ifdef CONFIG_32BIT /* O32 */ +#ifdef CONFIG_CPU_LITTLE_ENDIAN +#define r_err MIPS_R_V1 +#define r_val MIPS_R_V0 +#else /* CONFIG_CPU_LITTLE_ENDIAN */ +#define r_err MIPS_R_V0 +#define r_val MIPS_R_V1 +#endif +#else /* N64 */ +#define r_err MIPS_R_V0 +#define r_val MIPS_R_V0 +#endif + +#define r_ret MIPS_R_V0 + +/* + * Use 2 scratch registers to avoid pipeline interlocks. + * There is no overhead during epilogue and prologue since + * any of the $s0-$s6 registers will only be preserved if + * they are going to actually be used. + */ +#define r_s0 MIPS_R_S0 /* scratch reg 1 */ +#define r_s1 MIPS_R_S1 /* scratch reg 2 */ +#define r_off MIPS_R_S2 +#define r_A MIPS_R_S3 +#define r_X MIPS_R_S4 +#define r_skb MIPS_R_S5 +#define r_M MIPS_R_S6 +#define r_tmp_imm MIPS_R_T6 /* No need to preserve this */ +#define r_tmp MIPS_R_T7 /* No need to preserve this */ +#define r_zero MIPS_R_ZERO +#define r_sp MIPS_R_SP +#define r_ra MIPS_R_RA + +#define SCRATCH_OFF(k) (4 * (k)) + +/* JIT flags */ +#define SEEN_CALL (1 << BPF_MEMWORDS) +#define SEEN_SREG_SFT (BPF_MEMWORDS + 1) +#define SEEN_SREG_BASE (1 << SEEN_SREG_SFT) +#define SEEN_SREG(x) (SEEN_SREG_BASE << (x)) +#define SEEN_S0 SEEN_SREG(0) +#define SEEN_S1 SEEN_SREG(1) +#define SEEN_OFF SEEN_SREG(2) +#define SEEN_A SEEN_SREG(3) +#define SEEN_X SEEN_SREG(4) +#define SEEN_SKB SEEN_SREG(5) +#define SEEN_MEM SEEN_SREG(6) + +/* Arguments used by JIT */ +#define ARGS_USED_BY_JIT 2 /* only applicable to 64-bit */ + +#define FLAG_NEED_X_RESET (1 << 0) + +#define SBIT(x) (1 << (x)) /* Signed version of BIT() */ + +/** + * struct jit_ctx - JIT context + * @skf: The sk_filter + * @prologue_bytes: Number of bytes for prologue + * @idx: Instruction index + * @flags: JIT flags + * @offsets: Instruction offsets + * @target: Memory location for the compiled filter + */ +struct jit_ctx { + const struct sk_filter *skf; + unsigned int prologue_bytes; + u32 idx; + u32 flags; + u32 *offsets; + u32 *target; +}; + + +static inline int optimize_div(u32 *k) +{ + /* power of 2 divides can be implemented with right shift */ + if (!(*k & (*k-1))) { + *k = ilog2(*k); + return 1; + } + + return 0; +} + +/* Simply emit the instruction if the JIT memory space has been allocated */ +#define emit_instr(ctx, func, ...) \ +do { \ + if ((ctx)->target != NULL) { \ + u32 *p = &(ctx)->target[ctx->idx]; \ + uasm_i_##func(&p, ##__VA_ARGS__); \ + } \ + (ctx)->idx++; \ +} while (0) + +/* Determine if immediate is within the 16-bit signed range */ +static inline bool is_range16(s32 imm) +{ + if (imm >= SBIT(15) || imm < -SBIT(15)) + return true; + return false; +} + +static inline void emit_addu(unsigned int dst, unsigned int src1, + unsigned int src2, struct jit_ctx *ctx) +{ + emit_instr(ctx, addu, dst, src1, src2); +} + +static inline void emit_nop(struct jit_ctx *ctx) +{ + emit_instr(ctx, nop); +} + +/* Load a u32 immediate to a register */ +static inline void emit_load_imm(unsigned int dst, u32 imm, struct jit_ctx *ctx) +{ + if (ctx->target != NULL) { + /* addiu can only handle s16 */ + if (is_range16(imm)) { + u32 *p = &ctx->target[ctx->idx]; + uasm_i_lui(&p, r_tmp_imm, (s32)imm >> 16); + p = &ctx->target[ctx->idx + 1]; + uasm_i_ori(&p, dst, r_tmp_imm, imm & 0xffff); + } else { + u32 *p = &ctx->target[ctx->idx]; + uasm_i_addiu(&p, dst, r_zero, imm); + } + } + ctx->idx++; + + if (is_range16(imm)) + ctx->idx++; +} + +static inline void emit_or(unsigned int dst, unsigned int src1, + unsigned int src2, struct jit_ctx *ctx) +{ + emit_instr(ctx, or, dst, src1, src2); +} + +static inline void emit_ori(unsigned int dst, unsigned src, u32 imm, + struct jit_ctx *ctx) +{ + if (imm >= BIT(16)) { + emit_load_imm(r_tmp, imm, ctx); + emit_or(dst, src, r_tmp, ctx); + } else { + emit_instr(ctx, ori, dst, src, imm); + } +} + + +static inline void emit_daddu(unsigned int dst, unsigned int src1, + unsigned int src2, struct jit_ctx *ctx) +{ + emit_instr(ctx, daddu, dst, src1, src2); +} + +static inline void emit_daddiu(unsigned int dst, unsigned int src, + int imm, struct jit_ctx *ctx) +{ + /* + * Only used for stack, so the imm is relatively small + * and it fits in 15-bits + */ + emit_instr(ctx, daddiu, dst, src, imm); +} + +static inline void emit_addiu(unsigned int dst, unsigned int src, + u32 imm, struct jit_ctx *ctx) +{ + if (is_range16(imm)) { + emit_load_imm(r_tmp, imm, ctx); + emit_addu(dst, r_tmp, src, ctx); + } else { + emit_instr(ctx, addiu, dst, src, imm); + } +} + +static inline void emit_and(unsigned int dst, unsigned int src1, + unsigned int src2, struct jit_ctx *ctx) +{ + emit_instr(ctx, and, dst, src1, src2); +} + +static inline void emit_andi(unsigned int dst, unsigned int src, + u32 imm, struct jit_ctx *ctx) +{ + /* If imm does not fit in u16 then load it to register */ + if (imm >= BIT(16)) { + emit_load_imm(r_tmp, imm, ctx); + emit_and(dst, src, r_tmp, ctx); + } else { + emit_instr(ctx, andi, dst, src, imm); + } +} + +static inline void emit_xor(unsigned int dst, unsigned int src1, + unsigned int src2, struct jit_ctx *ctx) +{ + emit_instr(ctx, xor, dst, src1, src2); +} + +static inline void emit_xori(ptr dst, ptr src, u32 imm, struct jit_ctx *ctx) +{ + /* If imm does not fit in u16 then load it to register */ + if (imm >= BIT(16)) { + emit_load_imm(r_tmp, imm, ctx); + emit_xor(dst, src, r_tmp, ctx); + } else { + emit_instr(ctx, xori, dst, src, imm); + } +} + +static inline void emit_stack_offset(int offset, struct jit_ctx *ctx) +{ + if (config_enabled(CONFIG_64BIT)) + emit_instr(ctx, daddiu, r_sp, r_sp, offset); + else + emit_instr(ctx, addiu, r_sp, r_sp, offset); + +} + +static inline void emit_subu(unsigned int dst, unsigned int src1, + unsigned int src2, struct jit_ctx *ctx) +{ + emit_instr(ctx, subu, dst, src1, src2); +} + +static inline void emit_neg(unsigned int reg, struct jit_ctx *ctx) +{ + emit_subu(reg, r_zero, reg, ctx); +} + +static inline void emit_sllv(unsigned int dst, unsigned int src, + unsigned int sa, struct jit_ctx *ctx) +{ + emit_instr(ctx, sllv, dst, src, sa); +} + +static inline void emit_sll(unsigned int dst, unsigned int src, + unsigned int sa, struct jit_ctx *ctx) +{ + /* sa is 5-bits long */ + BUG_ON(sa >= BIT(5)); + emit_instr(ctx, sll, dst, src, sa); +} + +static inline void emit_srlv(unsigned int dst, unsigned int src, + unsigned int sa, struct jit_ctx *ctx) +{ + emit_instr(ctx, srlv, dst, src, sa); +} + +static inline void emit_srl(unsigned int dst, unsigned int src, + unsigned int sa, struct jit_ctx *ctx) +{ + /* sa is 5-bits long */ + BUG_ON(sa >= BIT(5)); + emit_instr(ctx, srl, dst, src, sa); +} + +static inline void emit_sltu(unsigned int dst, unsigned int src1, + unsigned int src2, struct jit_ctx *ctx) +{ + emit_instr(ctx, sltu, dst, src1, src2); +} + +static inline void emit_sltiu(unsigned dst, unsigned int src, + unsigned int imm, struct jit_ctx *ctx) +{ + /* 16 bit immediate */ + if (is_range16((s32)imm)) { + emit_load_imm(r_tmp, imm, ctx); + emit_sltu(dst, src, r_tmp, ctx); + } else { + emit_instr(ctx, sltiu, dst, src, imm); + } + +} + +/* Store register on the stack */ +static inline void emit_store_stack_reg(ptr reg, ptr base, + unsigned int offset, + struct jit_ctx *ctx) +{ + if (config_enabled(CONFIG_64BIT)) + emit_instr(ctx, sd, reg, offset, base); + else + emit_instr(ctx, sw, reg, offset, base); +} + +static inline void emit_store(ptr reg, ptr base, unsigned int offset, + struct jit_ctx *ctx) +{ + emit_instr(ctx, sw, reg, offset, base); +} + +static inline void emit_load_stack_reg(ptr reg, ptr base, + unsigned int offset, + struct jit_ctx *ctx) +{ + if (config_enabled(CONFIG_64BIT)) + emit_instr(ctx, ld, reg, offset, base); + else + emit_instr(ctx, lw, reg, offset, base); +} + +static inline void emit_load(unsigned int reg, unsigned int base, + unsigned int offset, struct jit_ctx *ctx) +{ + emit_instr(ctx, lw, reg, offset, base); +} + +static inline void emit_load_byte(unsigned int reg, unsigned int base, + unsigned int offset, struct jit_ctx *ctx) +{ + emit_instr(ctx, lb, reg, offset, base); +} + +static inline void emit_half_load(unsigned int reg, unsigned int base, + unsigned int offset, struct jit_ctx *ctx) +{ + emit_instr(ctx, lh, reg, offset, base); +} + +static inline void emit_mul(unsigned int dst, unsigned int src1, + unsigned int src2, struct jit_ctx *ctx) +{ + emit_instr(ctx, mul, dst, src1, src2); +} + +static inline void emit_div(unsigned int dst, unsigned int src, + struct jit_ctx *ctx) +{ + if (ctx->target != NULL) { + u32 *p = &ctx->target[ctx->idx]; + uasm_i_divu(&p, dst, src); + p = &ctx->target[ctx->idx + 1]; + uasm_i_mfhi(&p, dst); + } + ctx->idx += 2; /* 2 insts */ +} + +static inline void emit_mod(unsigned int dst, unsigned int src, + struct jit_ctx *ctx) +{ + if (ctx->target != NULL) { + u32 *p = &ctx->target[ctx->idx]; + uasm_i_divu(&p, dst, src); + p = &ctx->target[ctx->idx + 1]; + uasm_i_mflo(&p, dst); + } + ctx->idx += 2; /* 2 insts */ +} + +static inline void emit_dsll(unsigned int dst, unsigned int src, + unsigned int sa, struct jit_ctx *ctx) +{ + emit_instr(ctx, dsll, dst, src, sa); +} + +static inline void emit_dsrl32(unsigned int dst, unsigned int src, + unsigned int sa, struct jit_ctx *ctx) +{ + emit_instr(ctx, dsrl32, dst, src, sa); +} + +static inline void emit_wsbh(unsigned int dst, unsigned int src, + struct jit_ctx *ctx) +{ + emit_instr(ctx, wsbh, dst, src); +} + +/* load a function pointer to register */ +static inline void emit_load_func(unsigned int reg, ptr imm, + struct jit_ctx *ctx) +{ + if (config_enabled(CONFIG_64BIT)) { + /* At this point imm is always 64-bit */ + emit_load_imm(r_tmp, (u64)imm >> 32, ctx); + emit_dsll(r_tmp_imm, r_tmp, 16, ctx); /* left shift by 16 */ + emit_ori(r_tmp, r_tmp_imm, (imm >> 16) & 0xffff, ctx); + emit_dsll(r_tmp_imm, r_tmp, 16, ctx); /* left shift by 16 */ + emit_ori(reg, r_tmp_imm, imm & 0xffff, ctx); + } else { + emit_load_imm(reg, imm, ctx); + } +} + +/* Move to real MIPS register */ +static inline void emit_reg_move(ptr dst, ptr src, struct jit_ctx *ctx) +{ + if (config_enabled(CONFIG_64BIT)) + emit_daddu(dst, src, r_zero, ctx); + else + emit_addu(dst, src, r_zero, ctx); +} + +/* Move to JIT (32-bit) register */ +static inline void emit_jit_reg_move(ptr dst, ptr src, struct jit_ctx *ctx) +{ + emit_addu(dst, src, r_zero, ctx); +} + +/* Compute the immediate value for PC-relative branches. */ +static inline u32 b_imm(unsigned int tgt, struct jit_ctx *ctx) +{ + if (ctx->target == NULL) + return 0; + + /* + * We want a pc-relative branch. We only do forward branches + * so tgt is always after pc. tgt is the instruction offset + * we want to jump to. + + * Branch on MIPS: + * I: target_offset <- sign_extend(offset) + * I+1: PC += target_offset (delay slot) + * + * ctx->idx currently points to the branch instruction + * but the offset is added to the delay slot so we need + * to subtract 4. + */ + return ctx->offsets[tgt] - + (ctx->idx * 4 - ctx->prologue_bytes) - 4; +} + +static inline void emit_bcond(int cond, unsigned int reg1, unsigned int reg2, + unsigned int imm, struct jit_ctx *ctx) +{ + if (ctx->target != NULL) { + u32 *p = &ctx->target[ctx->idx]; + + switch (cond) { + case MIPS_COND_EQ: + uasm_i_beq(&p, reg1, reg2, imm); + break; + case MIPS_COND_NE: + uasm_i_bne(&p, reg1, reg2, imm); + break; + case MIPS_COND_ALL: + uasm_i_b(&p, imm); + break; + default: + pr_warn("%s: Unhandled branch conditional: %d\n", + __func__, cond); + } + } + ctx->idx++; +} + +static inline void emit_b(unsigned int imm, struct jit_ctx *ctx) +{ + emit_bcond(MIPS_COND_ALL, r_zero, r_zero, imm, ctx); +} + +static inline void emit_jalr(unsigned int link, unsigned int reg, + struct jit_ctx *ctx) +{ + emit_instr(ctx, jalr, link, reg); +} + +static inline void emit_jr(unsigned int reg, struct jit_ctx *ctx) +{ + emit_instr(ctx, jr, reg); +} + +static inline u16 align_sp(unsigned int num) +{ + /* Double word alignment for 32-bit, quadword for 64-bit */ + unsigned int align = config_enabled(CONFIG_64BIT) ? 16 : 8; + num = (num + (align - 1)) & -align; + return num; +} + +static inline void update_on_xread(struct jit_ctx *ctx) +{ + if (!(ctx->flags & SEEN_X)) + ctx->flags |= FLAG_NEED_X_RESET; + + ctx->flags |= SEEN_X; +} + +static bool is_load_to_a(u16 inst) +{ + switch (inst) { + case BPF_S_LD_W_LEN: + case BPF_S_LD_W_ABS: + case BPF_S_LD_H_ABS: + case BPF_S_LD_B_ABS: + case BPF_S_ANC_CPU: + case BPF_S_ANC_IFINDEX: + case BPF_S_ANC_MARK: + case BPF_S_ANC_PROTOCOL: + case BPF_S_ANC_RXHASH: + case BPF_S_ANC_VLAN_TAG: + case BPF_S_ANC_VLAN_TAG_PRESENT: + case BPF_S_ANC_QUEUE: + return true; + default: + return false; + } +} + +static void save_bpf_jit_regs(struct jit_ctx *ctx, unsigned offset) +{ + int i = 0, real_off = 0; + u32 sflags, tmp_flags; + + /* Adjust the stack pointer */ + emit_stack_offset(-align_sp(offset), ctx); + + if (ctx->flags & SEEN_CALL) { + /* Argument save area */ + if (config_enabled(CONFIG_64BIT)) + /* Bottom of current frame */ + real_off = align_sp(offset) - RSIZE; + else + /* Top of previous frame */ + real_off = align_sp(offset) + RSIZE; + emit_store_stack_reg(MIPS_R_A0, r_sp, real_off, ctx); + emit_store_stack_reg(MIPS_R_A1, r_sp, real_off + RSIZE, ctx); + + real_off = 0; + } + + tmp_flags = sflags = ctx->flags >> SEEN_SREG_SFT; + /* sflags is essentially a bitmap */ + while (tmp_flags) { + if ((sflags >> i) & 0x1) { + emit_store_stack_reg(MIPS_R_S0 + i, r_sp, real_off, + ctx); + real_off += RSIZE; + } + i++; + tmp_flags >>= 1; + } + + /* save return address */ + if (ctx->flags & SEEN_CALL) { + emit_store_stack_reg(r_ra, r_sp, real_off, ctx); + real_off += RSIZE; + } + + /* Setup r_M leaving the alignment gap if necessary */ + if (ctx->flags & SEEN_MEM) { + if (real_off % (RSIZE * 2)) + real_off += RSIZE; + emit_addiu(r_M, r_sp, real_off, ctx); + } +} + +static void restore_bpf_jit_regs(struct jit_ctx *ctx, + unsigned int offset) +{ + int i, real_off = 0; + u32 sflags, tmp_flags; + + if (ctx->flags & SEEN_CALL) { + if (config_enabled(CONFIG_64BIT)) + /* Bottom of current frame */ + real_off = align_sp(offset) - RSIZE; + else + /* Top of previous frame */ + real_off = align_sp(offset) + RSIZE; + emit_load_stack_reg(MIPS_R_A0, r_sp, real_off, ctx); + emit_load_stack_reg(MIPS_R_A1, r_sp, real_off + RSIZE, ctx); + + real_off = 0; + } + + tmp_flags = sflags = ctx->flags >> SEEN_SREG_SFT; + /* sflags is a bitmap */ + i = 0; + while (tmp_flags) { + if ((sflags >> i) & 0x1) { + emit_load_stack_reg(MIPS_R_S0 + i, r_sp, real_off, + ctx); + real_off += RSIZE; + } + i++; + tmp_flags >>= 1; + } + + /* restore return address */ + if (ctx->flags & SEEN_CALL) + emit_load_stack_reg(r_ra, r_sp, real_off, ctx); + + /* Restore the sp and discard the scrach memory */ + emit_stack_offset(align_sp(offset), ctx); +} + +static unsigned int get_stack_depth(struct jit_ctx *ctx) +{ + int sp_off = 0; + + + /* How may s* regs do we need to preserved? */ + sp_off += hweight32(ctx->flags >> SEEN_SREG_SFT) * RSIZE; + + if (ctx->flags & SEEN_MEM) + sp_off += 4 * BPF_MEMWORDS; /* BPF_MEMWORDS are 32-bit */ + + if (ctx->flags & SEEN_CALL) + /* + * The JIT code make calls to external functions using 2 + * arguments. Therefore, for o32 we don't need to allocate + * space because we don't care if the argumetns are lost + * across calls. We do need however to preserve incoming + * arguments but the space is already allocated for us by + * the caller. On the other hand, for n64, we need to allocate + * this space ourselves. We need to preserve $ra as well. + */ + sp_off += config_enabled(CONFIG_64BIT) ? + (ARGS_USED_BY_JIT + 1) * RSIZE : RSIZE; + + /* + * Subtract the bytes for the last registers since we only care about + * the location on the stack pointer. + */ + return sp_off - RSIZE; +} + +static void build_prologue(struct jit_ctx *ctx) +{ + u16 first_inst = ctx->skf->insns[0].code; + int sp_off; + + /* Calculate the total offset for the stack pointer */ + sp_off = get_stack_depth(ctx); + save_bpf_jit_regs(ctx, sp_off); + + if (ctx->flags & SEEN_SKB) + emit_reg_move(r_skb, MIPS_R_A0, ctx); + + if (ctx->flags & FLAG_NEED_X_RESET) + emit_jit_reg_move(r_X, r_zero, ctx); + + /* Do not leak kernel data to userspace */ + if ((first_inst != BPF_S_RET_K) && !(is_load_to_a(first_inst))) + emit_jit_reg_move(r_A, r_zero, ctx); +} + +static void build_epilogue(struct jit_ctx *ctx) +{ + unsigned int sp_off; + + /* Calculate the total offset for the stack pointer */ + + sp_off = get_stack_depth(ctx); + restore_bpf_jit_regs(ctx, sp_off); + + /* Return */ + emit_jr(r_ra, ctx); + emit_nop(ctx); +} + +static u64 jit_get_skb_b(struct sk_buff *skb, unsigned offset) +{ + u8 ret; + int err; + + err = skb_copy_bits(skb, offset, &ret, 1); + + return (u64)err << 32 | ret; +} + +static u64 jit_get_skb_h(struct sk_buff *skb, unsigned offset) +{ + u16 ret; + int err; + + err = skb_copy_bits(skb, offset, &ret, 2); + + return (u64)err << 32 | ntohs(ret); +} + +static u64 jit_get_skb_w(struct sk_buff *skb, unsigned offset) +{ + u32 ret; + int err; + + err = skb_copy_bits(skb, offset, &ret, 4); + + return (u64)err << 32 | ntohl(ret); +} + +#define PKT_TYPE_MAX 7 +static int pkt_type_offset(void) +{ + struct sk_buff skb_probe = { + .pkt_type = ~0, + }; + char *ct = (char *)&skb_probe; + unsigned int off; + + for (off = 0; off < sizeof(struct sk_buff); off++) { + if (ct[off] == PKT_TYPE_MAX) + return off; + } + pr_err_once("Please fix pkt_type_offset(), as pkt_type couldn't be found\n"); + return -1; +} + +static int build_body(struct jit_ctx *ctx) +{ + void *load_func[] = {jit_get_skb_b, jit_get_skb_h, jit_get_skb_w}; + const struct sk_filter *prog = ctx->skf; + const struct sock_filter *inst; + unsigned int i, off, load_order, condt; + u32 k, b_off __maybe_unused; + + for (i = 0; i < prog->len; i++) { + inst = &(prog->insns[i]); + pr_debug("%s: code->0x%02x, jt->0x%x, jf->0x%x, k->0x%x\n", + __func__, inst->code, inst->jt, inst->jf, inst->k); + k = inst->k; + + if (ctx->target == NULL) + ctx->offsets[i] = ctx->idx * 4; + + switch (inst->code) { + case BPF_S_LD_IMM: + /* A <- k ==> li r_A, k */ + ctx->flags |= SEEN_A; + emit_load_imm(r_A, k, ctx); + break; + case BPF_S_LD_W_LEN: + BUILD_BUG_ON(FIELD_SIZEOF(struct sk_buff, len) != 4); + /* A <- len ==> lw r_A, offset(skb) */ + ctx->flags |= SEEN_SKB | SEEN_A; + off = offsetof(struct sk_buff, len); + emit_load(r_A, r_skb, off, ctx); + break; + case BPF_S_LD_MEM: + /* A <- M[k] ==> lw r_A, offset(M) */ + ctx->flags |= SEEN_MEM | SEEN_A; + emit_load(r_A, r_M, SCRATCH_OFF(k), ctx); + break; + case BPF_S_LD_W_ABS: + /* A <- P[k:4] */ + load_order = 2; + goto load; + case BPF_S_LD_H_ABS: + /* A <- P[k:2] */ + load_order = 1; + goto load; + case BPF_S_LD_B_ABS: + /* A <- P[k:1] */ + load_order = 0; +load: + emit_load_imm(r_off, k, ctx); +load_common: + ctx->flags |= SEEN_CALL | SEEN_OFF | SEEN_S0 | + SEEN_SKB | SEEN_A; + + emit_load_func(r_s0, (ptr)load_func[load_order], + ctx); + emit_reg_move(MIPS_R_A0, r_skb, ctx); + emit_jalr(MIPS_R_RA, r_s0, ctx); + /* Load second argument to delay slot */ + emit_reg_move(MIPS_R_A1, r_off, ctx); + /* Check the error value */ + if (config_enabled(CONFIG_64BIT)) { + /* Get error code from the top 32-bits */ + emit_dsrl32(r_s0, r_val, 0, ctx); + /* Branch to 3 instructions ahead */ + emit_bcond(MIPS_COND_NE, r_s0, r_zero, 3 << 2, + ctx); + } else { + /* Branch to 3 instructions ahead */ + emit_bcond(MIPS_COND_NE, r_err, r_zero, 3 << 2, + ctx); + } + emit_nop(ctx); + /* We are good */ + emit_b(b_imm(i + 1, ctx), ctx); + emit_jit_reg_move(r_A, r_val, ctx); + /* Return with error */ + emit_b(b_imm(prog->len, ctx), ctx); + emit_reg_move(r_ret, r_zero, ctx); + break; + case BPF_S_LD_W_IND: + /* A <- P[X + k:4] */ + load_order = 2; + goto load_ind; + case BPF_S_LD_H_IND: + /* A <- P[X + k:2] */ + load_order = 1; + goto load_ind; + case BPF_S_LD_B_IND: + /* A <- P[X + k:1] */ + load_order = 0; +load_ind: + update_on_xread(ctx); + ctx->flags |= SEEN_OFF | SEEN_X; + emit_addiu(r_off, r_X, k, ctx); + goto load_common; + case BPF_S_LDX_IMM: + /* X <- k */ + ctx->flags |= SEEN_X; + emit_load_imm(r_X, k, ctx); + break; + case BPF_S_LDX_MEM: + /* X <- M[k] */ + ctx->flags |= SEEN_X | SEEN_MEM; + emit_load(r_X, r_M, SCRATCH_OFF(k), ctx); + break; + case BPF_S_LDX_W_LEN: + /* X <- len */ + ctx->flags |= SEEN_X | SEEN_SKB; + off = offsetof(struct sk_buff, len); + emit_load(r_X, r_skb, off, ctx); + break; + case BPF_S_LDX_B_MSH: + /* X <- 4 * (P[k:1] & 0xf) */ + ctx->flags |= SEEN_X | SEEN_CALL | SEEN_S0 | SEEN_SKB; + /* Load offset to a1 */ + emit_load_func(r_s0, (ptr)jit_get_skb_b, ctx); + /* + * This may emit two instructions so it may not fit + * in the delay slot. So use a0 in the delay slot. + */ + emit_load_imm(MIPS_R_A1, k, ctx); + emit_jalr(MIPS_R_RA, r_s0, ctx); + emit_reg_move(MIPS_R_A0, r_skb, ctx); /* delay slot */ + /* Check the error value */ + if (config_enabled(CONFIG_64BIT)) { + /* Top 32-bits of $v0 on 64-bit */ + emit_dsrl32(r_s0, r_val, 0, ctx); + emit_bcond(MIPS_COND_NE, r_s0, r_zero, + 3 << 2, ctx); + } else { + emit_bcond(MIPS_COND_NE, r_err, r_zero, + 3 << 2, ctx); + } + /* No need for delay slot */ + /* We are good */ + /* X <- P[1:K] & 0xf */ + emit_andi(r_X, r_val, 0xf, ctx); + /* X << 2 */ + emit_b(b_imm(i + 1, ctx), ctx); + emit_sll(r_X, r_X, 2, ctx); /* delay slot */ + /* Return with error */ + emit_b(b_imm(prog->len, ctx), ctx); + emit_load_imm(r_ret, 0, ctx); /* delay slot */ + break; + case BPF_S_ST: + /* M[k] <- A */ + ctx->flags |= SEEN_MEM | SEEN_A; + emit_store(r_A, r_M, SCRATCH_OFF(k), ctx); + break; + case BPF_S_STX: + /* M[k] <- X */ + ctx->flags |= SEEN_MEM | SEEN_X; + emit_store(r_X, r_M, SCRATCH_OFF(k), ctx); + break; + case BPF_S_ALU_ADD_K: + /* A += K */ + ctx->flags |= SEEN_A; + emit_addiu(r_A, r_A, k, ctx); + break; + case BPF_S_ALU_ADD_X: + /* A += X */ + ctx->flags |= SEEN_A | SEEN_X; + emit_addu(r_A, r_A, r_X, ctx); + break; + case BPF_S_ALU_SUB_K: + /* A -= K */ + ctx->flags |= SEEN_A; + emit_addiu(r_A, r_A, -k, ctx); + break; + case BPF_S_ALU_SUB_X: + /* A -= X */ + ctx->flags |= SEEN_A | SEEN_X; + emit_subu(r_A, r_A, r_X, ctx); + break; + case BPF_S_ALU_MUL_K: + /* A *= K */ + /* Load K to scratch register before MUL */ + ctx->flags |= SEEN_A | SEEN_S0; + emit_load_imm(r_s0, k, ctx); + emit_mul(r_A, r_A, r_s0, ctx); + break; + case BPF_S_ALU_MUL_X: + /* A *= X */ + update_on_xread(ctx); + ctx->flags |= SEEN_A | SEEN_X; + emit_mul(r_A, r_A, r_X, ctx); + break; + case BPF_S_ALU_DIV_K: + /* A /= k */ + if (k == 1) + break; + if (optimize_div(&k)) { + ctx->flags |= SEEN_A; + emit_srl(r_A, r_A, k, ctx); + break; + } + ctx->flags |= SEEN_A | SEEN_S0; + emit_load_imm(r_s0, k, ctx); + emit_div(r_A, r_s0, ctx); + break; + case BPF_S_ALU_MOD_K: + /* A %= k */ + if (k == 1 || optimize_div(&k)) { + ctx->flags |= SEEN_A; + emit_jit_reg_move(r_A, r_zero, ctx); + } else { + ctx->flags |= SEEN_A | SEEN_S0; + emit_load_imm(r_s0, k, ctx); + emit_mod(r_A, r_s0, ctx); + } + break; + case BPF_S_ALU_DIV_X: + /* A /= X */ + update_on_xread(ctx); + ctx->flags |= SEEN_X | SEEN_A; + /* Check if r_X is zero */ + emit_bcond(MIPS_COND_EQ, r_X, r_zero, + b_imm(prog->len, ctx), ctx); + emit_load_imm(r_val, 0, ctx); /* delay slot */ + emit_div(r_A, r_X, ctx); + break; + case BPF_S_ALU_MOD_X: + /* A %= X */ + update_on_xread(ctx); + ctx->flags |= SEEN_X | SEEN_A; + /* Check if r_X is zero */ + emit_bcond(MIPS_COND_EQ, r_X, r_zero, + b_imm(prog->len, ctx), ctx); + emit_load_imm(r_val, 0, ctx); /* delay slot */ + emit_mod(r_A, r_X, ctx); + break; + case BPF_S_ALU_OR_K: + /* A |= K */ + ctx->flags |= SEEN_A; + emit_ori(r_A, r_A, k, ctx); + break; + case BPF_S_ALU_OR_X: + /* A |= X */ + update_on_xread(ctx); + ctx->flags |= SEEN_A; + emit_ori(r_A, r_A, r_X, ctx); + break; + case BPF_S_ALU_XOR_K: + /* A ^= k */ + ctx->flags |= SEEN_A; + emit_xori(r_A, r_A, k, ctx); + break; + case BPF_S_ANC_ALU_XOR_X: + case BPF_S_ALU_XOR_X: + /* A ^= X */ + update_on_xread(ctx); + ctx->flags |= SEEN_A; + emit_xor(r_A, r_A, r_X, ctx); + break; + case BPF_S_ALU_AND_K: + /* A &= K */ + ctx->flags |= SEEN_A; + emit_andi(r_A, r_A, k, ctx); + break; + case BPF_S_ALU_AND_X: + /* A &= X */ + update_on_xread(ctx); + ctx->flags |= SEEN_A | SEEN_X; + emit_and(r_A, r_A, r_X, ctx); + break; + case BPF_S_ALU_LSH_K: + /* A <<= K */ + ctx->flags |= SEEN_A; + emit_sll(r_A, r_A, k, ctx); + break; + case BPF_S_ALU_LSH_X: + /* A <<= X */ + ctx->flags |= SEEN_A | SEEN_X; + update_on_xread(ctx); + emit_sllv(r_A, r_A, r_X, ctx); + break; + case BPF_S_ALU_RSH_K: + /* A >>= K */ + ctx->flags |= SEEN_A; + emit_srl(r_A, r_A, k, ctx); + break; + case BPF_S_ALU_RSH_X: + ctx->flags |= SEEN_A | SEEN_X; + update_on_xread(ctx); + emit_srlv(r_A, r_A, r_X, ctx); + break; + case BPF_S_ALU_NEG: + /* A = -A */ + ctx->flags |= SEEN_A; + emit_neg(r_A, ctx); + break; + case BPF_S_JMP_JA: + /* pc += K */ + emit_b(b_imm(i + k + 1, ctx), ctx); + emit_nop(ctx); + break; + case BPF_S_JMP_JEQ_K: + /* pc += ( A == K ) ? pc->jt : pc->jf */ + condt = MIPS_COND_EQ | MIPS_COND_K; + goto jmp_cmp; + case BPF_S_JMP_JEQ_X: + ctx->flags |= SEEN_X; + /* pc += ( A == X ) ? pc->jt : pc->jf */ + condt = MIPS_COND_EQ | MIPS_COND_X; + goto jmp_cmp; + case BPF_S_JMP_JGE_K: + /* pc += ( A >= K ) ? pc->jt : pc->jf */ + condt = MIPS_COND_GE | MIPS_COND_K; + goto jmp_cmp; + case BPF_S_JMP_JGE_X: + ctx->flags |= SEEN_X; + /* pc += ( A >= X ) ? pc->jt : pc->jf */ + condt = MIPS_COND_GE | MIPS_COND_X; + goto jmp_cmp; + case BPF_S_JMP_JGT_K: + /* pc += ( A > K ) ? pc->jt : pc->jf */ + condt = MIPS_COND_GT | MIPS_COND_K; + goto jmp_cmp; + case BPF_S_JMP_JGT_X: + ctx->flags |= SEEN_X; + /* pc += ( A > X ) ? pc->jt : pc->jf */ + condt = MIPS_COND_GT | MIPS_COND_X; +jmp_cmp: + /* Greater or Equal */ + if ((condt & MIPS_COND_GE) || + (condt & MIPS_COND_GT)) { + if (condt & MIPS_COND_K) { /* K */ + ctx->flags |= SEEN_S0 | SEEN_A; + emit_sltiu(r_s0, r_A, k, ctx); + } else { /* X */ + ctx->flags |= SEEN_S0 | SEEN_A | + SEEN_X; + emit_sltu(r_s0, r_A, r_X, ctx); + } + /* A < (K|X) ? r_scrach = 1 */ + b_off = b_imm(i + inst->jf + 1, ctx); + emit_bcond(MIPS_COND_GT, r_s0, r_zero, b_off, + ctx); + emit_nop(ctx); + /* A > (K|X) ? scratch = 0 */ + if (condt & MIPS_COND_GT) { + /* Checking for equality */ + ctx->flags |= SEEN_S0 | SEEN_A | SEEN_X; + if (condt & MIPS_COND_K) + emit_load_imm(r_s0, k, ctx); + else + emit_jit_reg_move(r_s0, r_X, + ctx); + b_off = b_imm(i + inst->jf + 1, ctx); + emit_bcond(MIPS_COND_EQ, r_A, r_s0, + b_off, ctx); + emit_nop(ctx); + /* Finally, A > K|X */ + b_off = b_imm(i + inst->jt + 1, ctx); + emit_b(b_off, ctx); + emit_nop(ctx); + } else { + /* A >= (K|X) so jump */ + b_off = b_imm(i + inst->jt + 1, ctx); + emit_b(b_off, ctx); + emit_nop(ctx); + } + } else { + /* A == K|X */ + if (condt & MIPS_COND_K) { /* K */ + ctx->flags |= SEEN_S0 | SEEN_A; + emit_load_imm(r_s0, k, ctx); + /* jump true */ + b_off = b_imm(i + inst->jt + 1, ctx); + emit_bcond(MIPS_COND_EQ, r_A, r_s0, + b_off, ctx); + emit_nop(ctx); + /* jump false */ + b_off = b_imm(i + inst->jf + 1, + ctx); + emit_bcond(MIPS_COND_NE, r_A, r_s0, + b_off, ctx); + emit_nop(ctx); + } else { /* X */ + /* jump true */ + ctx->flags |= SEEN_A | SEEN_X; + b_off = b_imm(i + inst->jt + 1, + ctx); + emit_bcond(MIPS_COND_EQ, r_A, r_X, + b_off, ctx); + emit_nop(ctx); + /* jump false */ + b_off = b_imm(i + inst->jf + 1, ctx); + emit_bcond(MIPS_COND_NE, r_A, r_X, + b_off, ctx); + emit_nop(ctx); + } + } + break; + case BPF_S_JMP_JSET_K: + ctx->flags |= SEEN_S0 | SEEN_S1 | SEEN_A; + /* pc += (A & K) ? pc -> jt : pc -> jf */ + emit_load_imm(r_s1, k, ctx); + emit_and(r_s0, r_A, r_s1, ctx); + /* jump true */ + b_off = b_imm(i + inst->jt + 1, ctx); + emit_bcond(MIPS_COND_NE, r_s0, r_zero, b_off, ctx); + emit_nop(ctx); + /* jump false */ + b_off = b_imm(i + inst->jf + 1, ctx); + emit_b(b_off, ctx); + emit_nop(ctx); + break; + case BPF_S_JMP_JSET_X: + ctx->flags |= SEEN_S0 | SEEN_X | SEEN_A; + /* pc += (A & X) ? pc -> jt : pc -> jf */ + emit_and(r_s0, r_A, r_X, ctx); + /* jump true */ + b_off = b_imm(i + inst->jt + 1, ctx); + emit_bcond(MIPS_COND_NE, r_s0, r_zero, b_off, ctx); + emit_nop(ctx); + /* jump false */ + b_off = b_imm(i + inst->jf + 1, ctx); + emit_b(b_off, ctx); + emit_nop(ctx); + break; + case BPF_S_RET_A: + ctx->flags |= SEEN_A; + if (i != prog->len - 1) + /* + * If this is not the last instruction + * then jump to the epilogue + */ + emit_b(b_imm(prog->len, ctx), ctx); + emit_reg_move(r_ret, r_A, ctx); /* delay slot */ + break; + case BPF_S_RET_K: + /* + * It can emit two instructions so it does not fit on + * the delay slot. + */ + emit_load_imm(r_ret, k, ctx); + if (i != prog->len - 1) { + /* + * If this is not the last instruction + * then jump to the epilogue + */ + emit_b(b_imm(prog->len, ctx), ctx); + emit_nop(ctx); + } + break; + case BPF_S_MISC_TAX: + /* X = A */ + ctx->flags |= SEEN_X | SEEN_A; + emit_jit_reg_move(r_X, r_A, ctx); + break; + case BPF_S_MISC_TXA: + /* A = X */ + ctx->flags |= SEEN_A | SEEN_X; + update_on_xread(ctx); + emit_jit_reg_move(r_A, r_X, ctx); + break; + /* AUX */ + case BPF_S_ANC_PROTOCOL: + /* A = ntohs(skb->protocol */ + ctx->flags |= SEEN_SKB | SEEN_OFF | SEEN_A; + BUILD_BUG_ON(FIELD_SIZEOF(struct sk_buff, + protocol) != 2); + off = offsetof(struct sk_buff, protocol); + emit_half_load(r_A, r_skb, off, ctx); +#ifdef CONFIG_CPU_LITTLE_ENDIAN + /* This needs little endian fixup */ + if (cpu_has_mips_r2) { + /* R2 and later have the wsbh instruction */ + emit_wsbh(r_A, r_A, ctx); + } else { + /* Get first byte */ + emit_andi(r_tmp_imm, r_A, 0xff, ctx); + /* Shift it */ + emit_sll(r_tmp, r_tmp_imm, 8, ctx); + /* Get second byte */ + emit_srl(r_tmp_imm, r_A, 8, ctx); + emit_andi(r_tmp_imm, r_tmp_imm, 0xff, ctx); + /* Put everyting together in r_A */ + emit_or(r_A, r_tmp, r_tmp_imm, ctx); + } +#endif + break; + case BPF_S_ANC_CPU: + ctx->flags |= SEEN_A | SEEN_OFF; + /* A = current_thread_info()->cpu */ + BUILD_BUG_ON(FIELD_SIZEOF(struct thread_info, + cpu) != 4); + off = offsetof(struct thread_info, cpu); + /* $28/gp points to the thread_info struct */ + emit_load(r_A, 28, off, ctx); + break; + case BPF_S_ANC_IFINDEX: + /* A = skb->dev->ifindex */ + ctx->flags |= SEEN_SKB | SEEN_A | SEEN_S0; + off = offsetof(struct sk_buff, dev); + emit_load(r_s0, r_skb, off, ctx); + /* error (0) in the delay slot */ + emit_bcond(MIPS_COND_EQ, r_s0, r_zero, + b_imm(prog->len, ctx), ctx); + emit_reg_move(r_ret, r_zero, ctx); + BUILD_BUG_ON(FIELD_SIZEOF(struct net_device, + ifindex) != 4); + off = offsetof(struct net_device, ifindex); + emit_load(r_A, r_s0, off, ctx); + break; + case BPF_S_ANC_MARK: + ctx->flags |= SEEN_SKB | SEEN_A; + BUILD_BUG_ON(FIELD_SIZEOF(struct sk_buff, mark) != 4); + off = offsetof(struct sk_buff, mark); + emit_load(r_A, r_skb, off, ctx); + break; + case BPF_S_ANC_RXHASH: + ctx->flags |= SEEN_SKB | SEEN_A; + BUILD_BUG_ON(FIELD_SIZEOF(struct sk_buff, hash) != 4); + off = offsetof(struct sk_buff, hash); + emit_load(r_A, r_skb, off, ctx); + break; + case BPF_S_ANC_VLAN_TAG: + case BPF_S_ANC_VLAN_TAG_PRESENT: + ctx->flags |= SEEN_SKB | SEEN_S0 | SEEN_A; + BUILD_BUG_ON(FIELD_SIZEOF(struct sk_buff, + vlan_tci) != 2); + off = offsetof(struct sk_buff, vlan_tci); + emit_half_load(r_s0, r_skb, off, ctx); + if (inst->code == BPF_S_ANC_VLAN_TAG) + emit_and(r_A, r_s0, VLAN_VID_MASK, ctx); + else + emit_and(r_A, r_s0, VLAN_TAG_PRESENT, ctx); + break; + case BPF_S_ANC_PKTTYPE: + off = pkt_type_offset(); + + if (off < 0) + return -1; + emit_load_byte(r_tmp, r_skb, off, ctx); + /* Keep only the last 3 bits */ + emit_andi(r_A, r_tmp, PKT_TYPE_MAX, ctx); + break; + case BPF_S_ANC_QUEUE: + ctx->flags |= SEEN_SKB | SEEN_A; + BUILD_BUG_ON(FIELD_SIZEOF(struct sk_buff, + queue_mapping) != 2); + BUILD_BUG_ON(offsetof(struct sk_buff, + queue_mapping) > 0xff); + off = offsetof(struct sk_buff, queue_mapping); + emit_half_load(r_A, r_skb, off, ctx); + break; + default: + pr_warn("%s: Unhandled opcode: 0x%02x\n", __FILE__, + inst->code); + return -1; + } + } + + /* compute offsets only during the first pass */ + if (ctx->target == NULL) + ctx->offsets[i] = ctx->idx * 4; + + return 0; +} + +int bpf_jit_enable __read_mostly; + +void bpf_jit_compile(struct sk_filter *fp) +{ + struct jit_ctx ctx; + unsigned int alloc_size, tmp_idx; + + if (!bpf_jit_enable) + return; + + memset(&ctx, 0, sizeof(ctx)); + + ctx.offsets = kcalloc(fp->len, sizeof(*ctx.offsets), GFP_KERNEL); + if (ctx.offsets == NULL) + return; + + ctx.skf = fp; + + if (build_body(&ctx)) + goto out; + + tmp_idx = ctx.idx; + build_prologue(&ctx); + ctx.prologue_bytes = (ctx.idx - tmp_idx) * 4; + /* just to complete the ctx.idx count */ + build_epilogue(&ctx); + + alloc_size = 4 * ctx.idx; + ctx.target = module_alloc(alloc_size); + if (ctx.target == NULL) + goto out; + + /* Clean it */ + memset(ctx.target, 0, alloc_size); + + ctx.idx = 0; + + /* Generate the actual JIT code */ + build_prologue(&ctx); + build_body(&ctx); + build_epilogue(&ctx); + + /* Update the icache */ + flush_icache_range((ptr)ctx.target, (ptr)(ctx.target + ctx.idx)); + + if (bpf_jit_enable > 1) + /* Dump JIT code */ + bpf_jit_dump(fp->len, alloc_size, 2, ctx.target); + + fp->bpf_func = (void *)ctx.target; + fp->jited = 1; + +out: + kfree(ctx.offsets); +} + +void bpf_jit_free(struct sk_filter *fp) +{ + if (fp->jited) + module_free(NULL, fp->bpf_func); + kfree(fp); +} diff --git a/arch/mips/net/bpf_jit.h b/arch/mips/net/bpf_jit.h new file mode 100644 index 000000000000..3a5751b4335a --- /dev/null +++ b/arch/mips/net/bpf_jit.h @@ -0,0 +1,44 @@ +/* + * Just-In-Time compiler for BPF filters on MIPS + * + * Copyright (c) 2014 Imagination Technologies Ltd. + * Author: Markos Chandras <markos.chandras@imgtec.com> + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the + * Free Software Foundation; version 2 of the License. + */ + +#ifndef BPF_JIT_MIPS_OP_H +#define BPF_JIT_MIPS_OP_H + +/* Registers used by JIT */ +#define MIPS_R_ZERO 0 +#define MIPS_R_V0 2 +#define MIPS_R_V1 3 +#define MIPS_R_A0 4 +#define MIPS_R_A1 5 +#define MIPS_R_T6 14 +#define MIPS_R_T7 15 +#define MIPS_R_S0 16 +#define MIPS_R_S1 17 +#define MIPS_R_S2 18 +#define MIPS_R_S3 19 +#define MIPS_R_S4 20 +#define MIPS_R_S5 21 +#define MIPS_R_S6 22 +#define MIPS_R_S7 23 +#define MIPS_R_SP 29 +#define MIPS_R_RA 31 + +/* Conditional codes */ +#define MIPS_COND_EQ 0x1 +#define MIPS_COND_GE (0x1 << 1) +#define MIPS_COND_GT (0x1 << 2) +#define MIPS_COND_NE (0x1 << 3) +#define MIPS_COND_ALL (0x1 << 4) +/* Conditionals on X register or K immediate */ +#define MIPS_COND_X (0x1 << 5) +#define MIPS_COND_K (0x1 << 6) + +#endif /* BPF_JIT_MIPS_OP_H */ diff --git a/arch/mips/netlogic/common/irq.c b/arch/mips/netlogic/common/irq.c index 5afc4b7fce0f..c100b9afa0ab 100644 --- a/arch/mips/netlogic/common/irq.c +++ b/arch/mips/netlogic/common/irq.c @@ -203,6 +203,8 @@ void nlm_set_pic_extra_ack(int node, int irq, void (*xack)(struct irq_data *)) xirq = nlm_irq_to_xirq(node, irq); pic_data = irq_get_handler_data(xirq); + if (WARN_ON(!pic_data)) + return; pic_data->extra_ack = xack; } diff --git a/arch/mips/netlogic/common/reset.S b/arch/mips/netlogic/common/reset.S index b231fe1e7a09..701c4bcb9e47 100644 --- a/arch/mips/netlogic/common/reset.S +++ b/arch/mips/netlogic/common/reset.S @@ -35,6 +35,7 @@ #include <asm/asm.h> #include <asm/asm-offsets.h> +#include <asm/cpu.h> #include <asm/cacheops.h> #include <asm/regdef.h> #include <asm/mipsregs.h> @@ -74,13 +75,25 @@ .endm /* + * Allow access to physical mem >64G by enabling ELPA in PAGEGRAIN + * register. This is needed before going to C code since the SP can + * in this region. Called from all HW threads. + */ +.macro xlp_early_mmu_init + mfc0 t0, CP0_PAGEMASK, 1 + li t1, (1 << 29) /* ELPA bit */ + or t0, t1 + mtc0 t0, CP0_PAGEMASK, 1 +.endm + +/* * L1D cache has to be flushed before enabling threads in XLP. * On XLP8xx/XLP3xx, we do a low level flush using processor control * registers. On XLPII CPUs, usual cache instructions work. */ .macro xlp_flush_l1_dcache mfc0 t0, CP0_EBASE, 0 - andi t0, t0, 0xff00 + andi t0, t0, PRID_IMP_MASK slt t1, t0, 0x1200 beqz t1, 15f nop @@ -159,11 +172,15 @@ FEXPORT(nlm_reset_entry) 1: /* Entry point on core wakeup */ mfc0 t0, CP0_EBASE, 0 /* processor ID */ - andi t0, 0xff00 + andi t0, PRID_IMP_MASK li t1, 0x1500 /* XLP 9xx */ beq t0, t1, 2f /* does not need to set coherent */ nop + li t1, 0x1300 /* XLP 5xx */ + beq t0, t1, 2f /* does not need to set coherent */ + nop + /* set bit in SYS coherent register for the core */ mfc0 t0, CP0_EBASE, 1 mfc0 t1, CP0_EBASE, 1 @@ -197,6 +214,9 @@ FEXPORT(nlm_reset_entry) EXPORT(nlm_boot_siblings) /* core L1D flush before enable threads */ xlp_flush_l1_dcache + /* save ra and sp, will be used later (only for boot cpu) */ + dmtc0 ra, $22, 6 + dmtc0 sp, $22, 7 /* Enable hw threads by writing to MAP_THREADMODE of the core */ li t0, CKSEG1ADDR(RESET_DATA_PHYS) lw t1, BOOT_THREAD_MODE(t0) /* t1 <- thread mode */ @@ -225,6 +245,8 @@ EXPORT(nlm_boot_siblings) #endif mtc0 t1, CP0_STATUS + xlp_early_mmu_init + /* mark CPU ready */ li t3, CKSEG1ADDR(RESET_DATA_PHYS) ADDIU t1, t3, BOOT_CPU_READY @@ -238,14 +260,12 @@ EXPORT(nlm_boot_siblings) nop /* - * For the boot CPU, we have to restore registers and - * return + * For the boot CPU, we have to restore ra and sp and return, rest + * of the registers will be restored by the caller */ -4: dmfc0 t0, $4, 2 /* restore SP from UserLocal */ - li t1, 0xfadebeef - dmtc0 t1, $4, 2 /* restore SP from UserLocal */ - PTR_SUBU sp, t0, PT_SIZE - RESTORE_ALL +4: + dmfc0 ra, $22, 6 + dmfc0 sp, $22, 7 jr ra nop EXPORT(nlm_reset_entry_end) @@ -253,6 +273,7 @@ EXPORT(nlm_reset_entry_end) LEAF(nlm_init_boot_cpu) #ifdef CONFIG_CPU_XLP xlp_config_lsu + xlp_early_mmu_init #endif jr ra nop diff --git a/arch/mips/netlogic/common/smp.c b/arch/mips/netlogic/common/smp.c index 6baae15cc7b1..4fde7ac76cc9 100644 --- a/arch/mips/netlogic/common/smp.c +++ b/arch/mips/netlogic/common/smp.c @@ -135,10 +135,6 @@ void nlm_smp_finish(void) local_irq_enable(); } -void nlm_cpus_done(void) -{ -} - /* * Boot all other cpus in the system, initialize them, and bring them into * the boot function @@ -198,7 +194,7 @@ void __init nlm_smp_setup(void) cpumask_scnprintf(buf, ARRAY_SIZE(buf), cpu_possible_mask); pr_info("Possible CPU mask: %s\n", buf); - /* check with the cores we have worken up */ + /* check with the cores we have woken up */ for (ncore = 0, i = 0; i < NLM_NR_NODES; i++) ncore += hweight32(nlm_get_node(i)->coremask); @@ -213,6 +209,7 @@ static int nlm_parse_cpumask(cpumask_t *wakeup_mask) { uint32_t core0_thr_mask, core_thr_mask; int threadmode, i, j; + char buf[64]; core0_thr_mask = 0; for (i = 0; i < NLM_THREADS_PER_CORE; i++) @@ -247,8 +244,8 @@ static int nlm_parse_cpumask(cpumask_t *wakeup_mask) return threadmode; unsupp: - panic("Unsupported CPU mask %lx", - (unsigned long)cpumask_bits(wakeup_mask)[0]); + cpumask_scnprintf(buf, ARRAY_SIZE(buf), wakeup_mask); + panic("Unsupported CPU mask %s", buf); return 0; } @@ -277,7 +274,6 @@ struct plat_smp_ops nlm_smp_ops = { .send_ipi_mask = nlm_send_ipi_mask, .init_secondary = nlm_init_secondary, .smp_finish = nlm_smp_finish, - .cpus_done = nlm_cpus_done, .boot_secondary = nlm_boot_secondary, .smp_setup = nlm_smp_setup, .prepare_cpus = nlm_prepare_cpus, diff --git a/arch/mips/netlogic/common/smpboot.S b/arch/mips/netlogic/common/smpboot.S index 8597657c27fc..805355b0bd05 100644 --- a/arch/mips/netlogic/common/smpboot.S +++ b/arch/mips/netlogic/common/smpboot.S @@ -54,8 +54,9 @@ .set noat .set arch=xlr /* for mfcr/mtcr, XLR is sufficient */ -FEXPORT(xlp_boot_core0_siblings) /* "Master" cpu starts from here */ - dmtc0 sp, $4, 2 /* SP saved in UserLocal */ +/* Called by the boot cpu to wake up its sibling threads */ +NESTED(xlp_boot_core0_siblings, PT_SIZE, sp) + /* CPU register contents lost when enabling threads, save them first */ SAVE_ALL sync /* find the location to which nlm_boot_siblings was relocated */ @@ -65,9 +66,12 @@ FEXPORT(xlp_boot_core0_siblings) /* "Master" cpu starts from here */ dsubu t2, t1 daddu t2, t0 /* call it */ - jr t2 + jalr t2 nop - /* not reached */ + RESTORE_ALL + jr ra + nop +END(xlp_boot_core0_siblings) NESTED(nlm_boot_secondary_cpus, 16, sp) /* Initialize CP0 Status */ diff --git a/arch/mips/netlogic/common/time.c b/arch/mips/netlogic/common/time.c index 13391b8a6031..0c0a1a606f73 100644 --- a/arch/mips/netlogic/common/time.c +++ b/arch/mips/netlogic/common/time.c @@ -82,6 +82,7 @@ static struct clocksource csrc_pic = { static void nlm_init_pic_timer(void) { uint64_t picbase = nlm_get_node(0)->picbase; + u32 picfreq; nlm_pic_set_timer(picbase, PIC_CLOCK_TIMER, ~0ULL, 0, 0); if (current_cpu_data.cputype == CPU_XLR) { @@ -92,7 +93,9 @@ static void nlm_init_pic_timer(void) csrc_pic.read = nlm_get_pic_timer; } csrc_pic.rating = 1000; - clocksource_register_hz(&csrc_pic, pic_timer_freq()); + picfreq = pic_timer_freq(); + clocksource_register_hz(&csrc_pic, picfreq); + pr_info("PIC clock source added, frequency %d\n", picfreq); } void __init plat_time_init(void) diff --git a/arch/mips/netlogic/dts/xlp_gvp.dts b/arch/mips/netlogic/dts/xlp_gvp.dts index 047d27f54487..bb4ecd1d47fc 100644 --- a/arch/mips/netlogic/dts/xlp_gvp.dts +++ b/arch/mips/netlogic/dts/xlp_gvp.dts @@ -26,11 +26,12 @@ interrupt-parent = <&pic>; interrupts = <17>; }; - pic: pic@4000 { - interrupt-controller; + pic: pic@110000 { + compatible = "netlogic,xlp-pic"; #address-cells = <0>; #interrupt-cells = <1>; reg = <0 0x110000 0x200>; + interrupt-controller; }; nor_flash@1,0 { diff --git a/arch/mips/netlogic/xlp/Makefile b/arch/mips/netlogic/xlp/Makefile index ed9a93c04650..be358a8050c5 100644 --- a/arch/mips/netlogic/xlp/Makefile +++ b/arch/mips/netlogic/xlp/Makefile @@ -2,3 +2,5 @@ obj-y += setup.o nlm_hal.o cop2-ex.o dt.o obj-$(CONFIG_SMP) += wakeup.o obj-$(CONFIG_USB) += usb-init.o obj-$(CONFIG_USB) += usb-init-xlp2.o +obj-$(CONFIG_SATA_AHCI) += ahci-init.o +obj-$(CONFIG_SATA_AHCI) += ahci-init-xlp2.o diff --git a/arch/mips/netlogic/xlp/ahci-init-xlp2.c b/arch/mips/netlogic/xlp/ahci-init-xlp2.c new file mode 100644 index 000000000000..c83dbf3689e2 --- /dev/null +++ b/arch/mips/netlogic/xlp/ahci-init-xlp2.c @@ -0,0 +1,377 @@ +/* + * Copyright (c) 2003-2014 Broadcom Corporation + * All Rights Reserved + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the Broadcom + * license below: + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * + * THIS SOFTWARE IS PROVIDED BY BROADCOM ``AS IS'' AND ANY EXPRESS OR + * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED + * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL BROADCOM OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR + * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR + * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, + * WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE + * OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN + * IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#include <linux/dma-mapping.h> +#include <linux/kernel.h> +#include <linux/delay.h> +#include <linux/init.h> +#include <linux/pci.h> +#include <linux/irq.h> +#include <linux/bitops.h> +#include <linux/pci_ids.h> +#include <linux/nodemask.h> + +#include <asm/cpu.h> +#include <asm/mipsregs.h> + +#include <asm/netlogic/common.h> +#include <asm/netlogic/haldefs.h> +#include <asm/netlogic/mips-extns.h> +#include <asm/netlogic/xlp-hal/xlp.h> +#include <asm/netlogic/xlp-hal/iomap.h> + +#define SATA_CTL 0x0 +#define SATA_STATUS 0x1 /* Status Reg */ +#define SATA_INT 0x2 /* Interrupt Reg */ +#define SATA_INT_MASK 0x3 /* Interrupt Mask Reg */ +#define SATA_BIU_TIMEOUT 0x4 +#define AXIWRSPERRLOG 0x5 +#define AXIRDSPERRLOG 0x6 +#define BiuTimeoutLow 0x7 +#define BiuTimeoutHi 0x8 +#define BiuSlvErLow 0x9 +#define BiuSlvErHi 0xa +#define IO_CONFIG_SWAP_DIS 0xb +#define CR_REG_TIMER 0xc +#define CORE_ID 0xd +#define AXI_SLAVE_OPT1 0xe +#define PHY_MEM_ACCESS 0xf +#define PHY0_CNTRL 0x10 +#define PHY0_STAT 0x11 +#define PHY0_RX_ALIGN 0x12 +#define PHY0_RX_EQ_LO 0x13 +#define PHY0_RX_EQ_HI 0x14 +#define PHY0_BIST_LOOP 0x15 +#define PHY1_CNTRL 0x16 +#define PHY1_STAT 0x17 +#define PHY1_RX_ALIGN 0x18 +#define PHY1_RX_EQ_LO 0x19 +#define PHY1_RX_EQ_HI 0x1a +#define PHY1_BIST_LOOP 0x1b +#define RdExBase 0x1c +#define RdExLimit 0x1d +#define CacheAllocBase 0x1e +#define CacheAllocLimit 0x1f +#define BiuSlaveCmdGstNum 0x20 + +/*SATA_CTL Bits */ +#define SATA_RST_N BIT(0) /* Active low reset sata_core phy */ +#define SataCtlReserve0 BIT(1) +#define M_CSYSREQ BIT(2) /* AXI master low power, not used */ +#define S_CSYSREQ BIT(3) /* AXI slave low power, not used */ +#define P0_CP_DET BIT(8) /* Reserved, bring in from pad */ +#define P0_MP_SW BIT(9) /* Mech Switch */ +#define P0_DISABLE BIT(10) /* disable p0 */ +#define P0_ACT_LED_EN BIT(11) /* Active LED enable */ +#define P0_IRST_HARD_SYNTH BIT(12) /* PHY hard synth reset */ +#define P0_IRST_HARD_TXRX BIT(13) /* PHY lane hard reset */ +#define P0_IRST_POR BIT(14) /* PHY power on reset*/ +#define P0_IPDTXL BIT(15) /* PHY Tx lane dis/power down */ +#define P0_IPDRXL BIT(16) /* PHY Rx lane dis/power down */ +#define P0_IPDIPDMSYNTH BIT(17) /* PHY synthesizer dis/porwer down */ +#define P0_CP_POD_EN BIT(18) /* CP_POD enable */ +#define P0_AT_BYPASS BIT(19) /* P0 address translation by pass */ +#define P1_CP_DET BIT(20) /* Reserved,Cold Detect */ +#define P1_MP_SW BIT(21) /* Mech Switch */ +#define P1_DISABLE BIT(22) /* disable p1 */ +#define P1_ACT_LED_EN BIT(23) /* Active LED enable */ +#define P1_IRST_HARD_SYNTH BIT(24) /* PHY hard synth reset */ +#define P1_IRST_HARD_TXRX BIT(25) /* PHY lane hard reset */ +#define P1_IRST_POR BIT(26) /* PHY power on reset*/ +#define P1_IPDTXL BIT(27) /* PHY Tx lane dis/porwer down */ +#define P1_IPDRXL BIT(28) /* PHY Rx lane dis/porwer down */ +#define P1_IPDIPDMSYNTH BIT(29) /* PHY synthesizer dis/porwer down */ +#define P1_CP_POD_EN BIT(30) +#define P1_AT_BYPASS BIT(31) /* P1 address translation by pass */ + +/* Status register */ +#define M_CACTIVE BIT(0) /* m_cactive, not used */ +#define S_CACTIVE BIT(1) /* s_cactive, not used */ +#define P0_PHY_READY BIT(8) /* phy is ready */ +#define P0_CP_POD BIT(9) /* Cold PowerOn */ +#define P0_SLUMBER BIT(10) /* power mode slumber */ +#define P0_PATIAL BIT(11) /* power mode patial */ +#define P0_PHY_SIG_DET BIT(12) /* phy dignal detect */ +#define P0_PHY_CALI BIT(13) /* phy calibration done */ +#define P1_PHY_READY BIT(16) /* phy is ready */ +#define P1_CP_POD BIT(17) /* Cold PowerOn */ +#define P1_SLUMBER BIT(18) /* power mode slumber */ +#define P1_PATIAL BIT(19) /* power mode patial */ +#define P1_PHY_SIG_DET BIT(20) /* phy dignal detect */ +#define P1_PHY_CALI BIT(21) /* phy calibration done */ + +/* SATA CR_REG_TIMER bits */ +#define CR_TIME_SCALE (0x1000 << 0) + +/* SATA PHY specific registers start and end address */ +#define RXCDRCALFOSC0 0x0065 +#define CALDUTY 0x006e +#define RXDPIF 0x8065 +#define PPMDRIFTMAX_HI 0x80A4 + +#define nlm_read_sata_reg(b, r) nlm_read_reg(b, r) +#define nlm_write_sata_reg(b, r, v) nlm_write_reg(b, r, v) +#define nlm_get_sata_pcibase(node) \ + nlm_pcicfg_base(XLP9XX_IO_SATA_OFFSET(node)) +#define nlm_get_sata_regbase(node) \ + (nlm_get_sata_pcibase(node) + 0x100) + +/* SATA PHY config for register block 1 0x0065 .. 0x006e */ +static const u8 sata_phy_config1[] = { + 0xC9, 0xC9, 0x07, 0x07, 0x18, 0x18, 0x01, 0x01, 0x22, 0x00 +}; + +/* SATA PHY config for register block 2 0x0x8065 .. 0x0x80A4 */ +static const u8 sata_phy_config2[] = { + 0xAA, 0x00, 0x4C, 0xC9, 0xC9, 0x07, 0x07, 0x18, + 0x18, 0x05, 0x0C, 0x10, 0x00, 0x10, 0x00, 0xFF, + 0xCF, 0xF7, 0xE1, 0xF5, 0xFD, 0xFD, 0xFF, 0xFF, + 0xFF, 0xFF, 0xE3, 0xE7, 0xDB, 0xF5, 0xFD, 0xFD, + 0xF5, 0xF5, 0xFF, 0xFF, 0xE3, 0xE7, 0xDB, 0xF5, + 0xFD, 0xFD, 0xF5, 0xF5, 0xFF, 0xFF, 0xFF, 0xF5, + 0x3F, 0x00, 0x32, 0x00, 0x03, 0x01, 0x05, 0x05, + 0x04, 0x00, 0x00, 0x08, 0x04, 0x00, 0x00, 0x04, +}; + +const int sata_phy_debug = 0; /* set to verify PHY writes */ + +static void sata_clear_glue_reg(u64 regbase, u32 off, u32 bit) +{ + u32 reg_val; + + reg_val = nlm_read_sata_reg(regbase, off); + nlm_write_sata_reg(regbase, off, (reg_val & ~bit)); +} + +static void sata_set_glue_reg(u64 regbase, u32 off, u32 bit) +{ + u32 reg_val; + + reg_val = nlm_read_sata_reg(regbase, off); + nlm_write_sata_reg(regbase, off, (reg_val | bit)); +} + +static void write_phy_reg(u64 regbase, u32 addr, u32 physel, u8 data) +{ + nlm_write_sata_reg(regbase, PHY_MEM_ACCESS, + (1u << 31) | (physel << 24) | (data << 16) | addr); + udelay(850); +} + +static u8 read_phy_reg(u64 regbase, u32 addr, u32 physel) +{ + u32 val; + + nlm_write_sata_reg(regbase, PHY_MEM_ACCESS, + (0 << 31) | (physel << 24) | (0 << 16) | addr); + udelay(850); + val = nlm_read_sata_reg(regbase, PHY_MEM_ACCESS); + return (val >> 16) & 0xff; +} + +static void config_sata_phy(u64 regbase) +{ + u32 port, i, reg; + + for (port = 0; port < 2; port++) { + for (i = 0, reg = RXCDRCALFOSC0; reg <= CALDUTY; reg++, i++) + write_phy_reg(regbase, reg, port, sata_phy_config1[i]); + + for (i = 0, reg = RXDPIF; reg <= PPMDRIFTMAX_HI; reg++, i++) + write_phy_reg(regbase, reg, port, sata_phy_config2[i]); + } +} + +static void check_phy_register(u64 regbase, u32 addr, u32 physel, u8 xdata) +{ + u8 data; + + data = read_phy_reg(regbase, addr, physel); + pr_info("PHY read addr = 0x%x physel = %d data = 0x%x %s\n", + addr, physel, data, data == xdata ? "TRUE" : "FALSE"); +} + +static void verify_sata_phy_config(u64 regbase) +{ + u32 port, i, reg; + + for (port = 0; port < 2; port++) { + for (i = 0, reg = RXCDRCALFOSC0; reg <= CALDUTY; reg++, i++) + check_phy_register(regbase, reg, port, + sata_phy_config1[i]); + + for (i = 0, reg = RXDPIF; reg <= PPMDRIFTMAX_HI; reg++, i++) + check_phy_register(regbase, reg, port, + sata_phy_config2[i]); + } +} + +static void nlm_sata_firmware_init(int node) +{ + u32 reg_val; + u64 regbase; + int n; + + pr_info("Initializing XLP9XX On-chip AHCI...\n"); + regbase = nlm_get_sata_regbase(node); + + /* Reset port0 */ + sata_clear_glue_reg(regbase, SATA_CTL, P0_IRST_POR); + sata_clear_glue_reg(regbase, SATA_CTL, P0_IRST_HARD_TXRX); + sata_clear_glue_reg(regbase, SATA_CTL, P0_IRST_HARD_SYNTH); + sata_clear_glue_reg(regbase, SATA_CTL, P0_IPDTXL); + sata_clear_glue_reg(regbase, SATA_CTL, P0_IPDRXL); + sata_clear_glue_reg(regbase, SATA_CTL, P0_IPDIPDMSYNTH); + + /* port1 */ + sata_clear_glue_reg(regbase, SATA_CTL, P1_IRST_POR); + sata_clear_glue_reg(regbase, SATA_CTL, P1_IRST_HARD_TXRX); + sata_clear_glue_reg(regbase, SATA_CTL, P1_IRST_HARD_SYNTH); + sata_clear_glue_reg(regbase, SATA_CTL, P1_IPDTXL); + sata_clear_glue_reg(regbase, SATA_CTL, P1_IPDRXL); + sata_clear_glue_reg(regbase, SATA_CTL, P1_IPDIPDMSYNTH); + udelay(300); + + /* Set PHY */ + sata_set_glue_reg(regbase, SATA_CTL, P0_IPDTXL); + sata_set_glue_reg(regbase, SATA_CTL, P0_IPDRXL); + sata_set_glue_reg(regbase, SATA_CTL, P0_IPDIPDMSYNTH); + sata_set_glue_reg(regbase, SATA_CTL, P1_IPDTXL); + sata_set_glue_reg(regbase, SATA_CTL, P1_IPDRXL); + sata_set_glue_reg(regbase, SATA_CTL, P1_IPDIPDMSYNTH); + + udelay(1000); + sata_set_glue_reg(regbase, SATA_CTL, P0_IRST_POR); + udelay(1000); + sata_set_glue_reg(regbase, SATA_CTL, P1_IRST_POR); + udelay(1000); + + /* setup PHY */ + config_sata_phy(regbase); + if (sata_phy_debug) + verify_sata_phy_config(regbase); + + udelay(1000); + sata_set_glue_reg(regbase, SATA_CTL, P0_IRST_HARD_TXRX); + sata_set_glue_reg(regbase, SATA_CTL, P0_IRST_HARD_SYNTH); + sata_set_glue_reg(regbase, SATA_CTL, P1_IRST_HARD_TXRX); + sata_set_glue_reg(regbase, SATA_CTL, P1_IRST_HARD_SYNTH); + udelay(300); + + /* Override reset in serial PHY mode */ + sata_set_glue_reg(regbase, CR_REG_TIMER, CR_TIME_SCALE); + /* Set reset SATA */ + sata_set_glue_reg(regbase, SATA_CTL, SATA_RST_N); + sata_set_glue_reg(regbase, SATA_CTL, M_CSYSREQ); + sata_set_glue_reg(regbase, SATA_CTL, S_CSYSREQ); + + pr_debug("Waiting for PHYs to come up.\n"); + n = 10000; + do { + reg_val = nlm_read_sata_reg(regbase, SATA_STATUS); + if ((reg_val & P1_PHY_READY) && (reg_val & P0_PHY_READY)) + break; + udelay(10); + } while (--n > 0); + + if (reg_val & P0_PHY_READY) + pr_info("PHY0 is up.\n"); + else + pr_info("PHY0 is down.\n"); + if (reg_val & P1_PHY_READY) + pr_info("PHY1 is up.\n"); + else + pr_info("PHY1 is down.\n"); + + pr_info("XLP AHCI Init Done.\n"); +} + +static int __init nlm_ahci_init(void) +{ + int node; + + if (!cpu_is_xlp9xx()) + return 0; + for (node = 0; node < NLM_NR_NODES; node++) + if (nlm_node_present(node)) + nlm_sata_firmware_init(node); + return 0; +} + +static void nlm_sata_intr_ack(struct irq_data *data) +{ + u64 regbase; + u32 val; + int node; + + node = data->irq / NLM_IRQS_PER_NODE; + regbase = nlm_get_sata_regbase(node); + val = nlm_read_sata_reg(regbase, SATA_INT); + sata_set_glue_reg(regbase, SATA_INT, val); +} + +static void nlm_sata_fixup_bar(struct pci_dev *dev) +{ + dev->resource[5] = dev->resource[0]; + memset(&dev->resource[0], 0, sizeof(dev->resource[0])); +} + +static void nlm_sata_fixup_final(struct pci_dev *dev) +{ + u32 val; + u64 regbase; + int node; + + /* Find end bridge function to find node */ + node = xlp_socdev_to_node(dev); + regbase = nlm_get_sata_regbase(node); + + /* clear pending interrupts and then enable them */ + val = nlm_read_sata_reg(regbase, SATA_INT); + sata_set_glue_reg(regbase, SATA_INT, val); + + /* Enable only the core interrupt */ + sata_set_glue_reg(regbase, SATA_INT_MASK, 0x1); + + dev->irq = nlm_irq_to_xirq(node, PIC_SATA_IRQ); + nlm_set_pic_extra_ack(node, PIC_SATA_IRQ, nlm_sata_intr_ack); +} + +arch_initcall(nlm_ahci_init); + +DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_BROADCOM, PCI_DEVICE_ID_XLP9XX_SATA, + nlm_sata_fixup_bar); + +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_BROADCOM, PCI_DEVICE_ID_XLP9XX_SATA, + nlm_sata_fixup_final); diff --git a/arch/mips/netlogic/xlp/ahci-init.c b/arch/mips/netlogic/xlp/ahci-init.c new file mode 100644 index 000000000000..a9d0fae02103 --- /dev/null +++ b/arch/mips/netlogic/xlp/ahci-init.c @@ -0,0 +1,209 @@ +/* + * Copyright (c) 2003-2014 Broadcom Corporation + * All Rights Reserved + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the Broadcom + * license below: + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * + * THIS SOFTWARE IS PROVIDED BY BROADCOM ``AS IS'' AND ANY EXPRESS OR + * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED + * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL BROADCOM OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR + * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR + * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, + * WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE + * OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN + * IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#include <linux/dma-mapping.h> +#include <linux/kernel.h> +#include <linux/delay.h> +#include <linux/init.h> +#include <linux/pci.h> +#include <linux/irq.h> +#include <linux/bitops.h> + +#include <asm/cpu.h> +#include <asm/mipsregs.h> + +#include <asm/netlogic/haldefs.h> +#include <asm/netlogic/xlp-hal/xlp.h> +#include <asm/netlogic/common.h> +#include <asm/netlogic/xlp-hal/iomap.h> +#include <asm/netlogic/mips-extns.h> + +#define SATA_CTL 0x0 +#define SATA_STATUS 0x1 /* Status Reg */ +#define SATA_INT 0x2 /* Interrupt Reg */ +#define SATA_INT_MASK 0x3 /* Interrupt Mask Reg */ +#define SATA_CR_REG_TIMER 0x4 /* PHY Conrol Timer Reg */ +#define SATA_CORE_ID 0x5 /* Core ID Reg */ +#define SATA_AXI_SLAVE_OPT1 0x6 /* AXI Slave Options Reg */ +#define SATA_PHY_LOS_LEV 0x7 /* PHY LOS Level Reg */ +#define SATA_PHY_MULTI 0x8 /* PHY Multiplier Reg */ +#define SATA_PHY_CLK_SEL 0x9 /* Clock Select Reg */ +#define SATA_PHY_AMP1_GEN1 0xa /* PHY Transmit Amplitude Reg 1 */ +#define SATA_PHY_AMP1_GEN2 0xb /* PHY Transmit Amplitude Reg 2 */ +#define SATA_PHY_AMP1_GEN3 0xc /* PHY Transmit Amplitude Reg 3 */ +#define SATA_PHY_PRE1 0xd /* PHY Transmit Preemphasis Reg 1 */ +#define SATA_PHY_PRE2 0xe /* PHY Transmit Preemphasis Reg 2 */ +#define SATA_PHY_PRE3 0xf /* PHY Transmit Preemphasis Reg 3 */ +#define SATA_SPDMODE 0x10 /* Speed Mode Reg */ +#define SATA_REFCLK 0x11 /* Reference Clock Control Reg */ +#define SATA_BYTE_SWAP_DIS 0x12 /* byte swap disable */ + +/*SATA_CTL Bits */ +#define SATA_RST_N BIT(0) +#define PHY0_RESET_N BIT(16) +#define PHY1_RESET_N BIT(17) +#define PHY2_RESET_N BIT(18) +#define PHY3_RESET_N BIT(19) +#define M_CSYSREQ BIT(2) +#define S_CSYSREQ BIT(3) + +/*SATA_STATUS Bits */ +#define P0_PHY_READY BIT(4) +#define P1_PHY_READY BIT(5) +#define P2_PHY_READY BIT(6) +#define P3_PHY_READY BIT(7) + +#define nlm_read_sata_reg(b, r) nlm_read_reg(b, r) +#define nlm_write_sata_reg(b, r, v) nlm_write_reg(b, r, v) +#define nlm_get_sata_pcibase(node) \ + nlm_pcicfg_base(XLP_IO_SATA_OFFSET(node)) +/* SATA device specific configuration registers are starts at 0x900 offset */ +#define nlm_get_sata_regbase(node) \ + (nlm_get_sata_pcibase(node) + 0x900) + +static void sata_clear_glue_reg(uint64_t regbase, uint32_t off, uint32_t bit) +{ + uint32_t reg_val; + + reg_val = nlm_read_sata_reg(regbase, off); + nlm_write_sata_reg(regbase, off, (reg_val & ~bit)); +} + +static void sata_set_glue_reg(uint64_t regbase, uint32_t off, uint32_t bit) +{ + uint32_t reg_val; + + reg_val = nlm_read_sata_reg(regbase, off); + nlm_write_sata_reg(regbase, off, (reg_val | bit)); +} + +static void nlm_sata_firmware_init(int node) +{ + uint32_t reg_val; + uint64_t regbase; + int i; + + pr_info("XLP AHCI Initialization started.\n"); + regbase = nlm_get_sata_regbase(node); + + /* Reset SATA */ + sata_clear_glue_reg(regbase, SATA_CTL, SATA_RST_N); + /* Reset PHY */ + sata_clear_glue_reg(regbase, SATA_CTL, + (PHY3_RESET_N | PHY2_RESET_N + | PHY1_RESET_N | PHY0_RESET_N)); + + /* Set SATA */ + sata_set_glue_reg(regbase, SATA_CTL, SATA_RST_N); + /* Set PHY */ + sata_set_glue_reg(regbase, SATA_CTL, + (PHY3_RESET_N | PHY2_RESET_N + | PHY1_RESET_N | PHY0_RESET_N)); + + pr_debug("Waiting for PHYs to come up.\n"); + i = 0; + do { + reg_val = nlm_read_sata_reg(regbase, SATA_STATUS); + i++; + } while (((reg_val & 0xF0) != 0xF0) && (i < 10000)); + + for (i = 0; i < 4; i++) { + if (reg_val & (P0_PHY_READY << i)) + pr_info("PHY%d is up.\n", i); + else + pr_info("PHY%d is down.\n", i); + } + + pr_info("XLP AHCI init done.\n"); +} + +static int __init nlm_ahci_init(void) +{ + int node = 0; + int chip = read_c0_prid() & PRID_REV_MASK; + + if (chip == PRID_IMP_NETLOGIC_XLP3XX) + nlm_sata_firmware_init(node); + return 0; +} + +static void nlm_sata_intr_ack(struct irq_data *data) +{ + uint32_t val = 0; + uint64_t regbase; + + regbase = nlm_get_sata_regbase(nlm_nodeid()); + val = nlm_read_sata_reg(regbase, SATA_INT); + sata_set_glue_reg(regbase, SATA_INT, val); +} + +static void nlm_sata_fixup_bar(struct pci_dev *dev) +{ + /* + * The AHCI resource is in BAR 0, move it to + * BAR 5, where it is expected + */ + dev->resource[5] = dev->resource[0]; + memset(&dev->resource[0], 0, sizeof(dev->resource[0])); +} + +static void nlm_sata_fixup_final(struct pci_dev *dev) +{ + uint32_t val; + uint64_t regbase; + int node = 0; /* XLP3XX does not support multi-node */ + + regbase = nlm_get_sata_regbase(node); + + /* clear pending interrupts and then enable them */ + val = nlm_read_sata_reg(regbase, SATA_INT); + sata_set_glue_reg(regbase, SATA_INT, val); + + /* Mask the core interrupt. If all the interrupts + * are enabled there are spurious interrupt flow + * happening, to avoid only enable core interrupt + * mask. + */ + sata_set_glue_reg(regbase, SATA_INT_MASK, 0x1); + + dev->irq = PIC_SATA_IRQ; + nlm_set_pic_extra_ack(node, PIC_SATA_IRQ, nlm_sata_intr_ack); +} + +arch_initcall(nlm_ahci_init); + +DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_NETLOGIC, PCI_DEVICE_ID_NLM_SATA, + nlm_sata_fixup_bar); +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_NETLOGIC, PCI_DEVICE_ID_NLM_SATA, + nlm_sata_fixup_final); diff --git a/arch/mips/netlogic/xlp/dt.c b/arch/mips/netlogic/xlp/dt.c index bdde33147bce..7cc46032b28e 100644 --- a/arch/mips/netlogic/xlp/dt.c +++ b/arch/mips/netlogic/xlp/dt.c @@ -48,9 +48,10 @@ static void *xlp_fdt_blob; void __init *xlp_dt_init(void *fdtp) { if (!fdtp) { - switch (current_cpu_data.processor_id & 0xff00) { + switch (current_cpu_data.processor_id & PRID_IMP_MASK) { #ifdef CONFIG_DT_XLP_GVP case PRID_IMP_NETLOGIC_XLP9XX: + case PRID_IMP_NETLOGIC_XLP5XX: fdtp = __dtb_xlp_gvp_begin; break; #endif diff --git a/arch/mips/netlogic/xlp/nlm_hal.c b/arch/mips/netlogic/xlp/nlm_hal.c index 997cd9ee10de..bc24beb3a426 100644 --- a/arch/mips/netlogic/xlp/nlm_hal.c +++ b/arch/mips/netlogic/xlp/nlm_hal.c @@ -54,6 +54,8 @@ void nlm_node_init(int node) struct nlm_soc_info *nodep; nodep = nlm_get_node(node); + if (node == 0) + nodep->coremask = 1; /* node 0, boot cpu */ nodep->sysbase = nlm_get_sys_regbase(node); nodep->picbase = nlm_get_pic_regbase(node); nodep->ebase = read_c0_ebase() & (~((1 << 12) - 1)); @@ -64,31 +66,39 @@ void nlm_node_init(int node) spin_lock_init(&nodep->piclock); } -int nlm_irq_to_irt(int irq) +static int xlp9xx_irq_to_irt(int irq) +{ + switch (irq) { + case PIC_GPIO_IRQ: + return 12; + case PIC_9XX_XHCI_0_IRQ: + return 114; + case PIC_9XX_XHCI_1_IRQ: + return 115; + case PIC_UART_0_IRQ: + return 133; + case PIC_UART_1_IRQ: + return 134; + case PIC_SATA_IRQ: + return 143; + case PIC_SPI_IRQ: + return 152; + case PIC_MMC_IRQ: + return 153; + case PIC_PCIE_LINK_LEGACY_IRQ(0): + case PIC_PCIE_LINK_LEGACY_IRQ(1): + case PIC_PCIE_LINK_LEGACY_IRQ(2): + case PIC_PCIE_LINK_LEGACY_IRQ(3): + return 191 + irq - PIC_PCIE_LINK_LEGACY_IRQ_BASE; + } + return -1; +} + +static int xlp_irq_to_irt(int irq) { uint64_t pcibase; int devoff, irt; - /* bypass for 9xx */ - if (cpu_is_xlp9xx()) { - switch (irq) { - case PIC_9XX_XHCI_0_IRQ: - return 114; - case PIC_9XX_XHCI_1_IRQ: - return 115; - case PIC_UART_0_IRQ: - return 133; - case PIC_UART_1_IRQ: - return 134; - case PIC_PCIE_LINK_LEGACY_IRQ(0): - case PIC_PCIE_LINK_LEGACY_IRQ(1): - case PIC_PCIE_LINK_LEGACY_IRQ(2): - case PIC_PCIE_LINK_LEGACY_IRQ(3): - return 191 + irq - PIC_PCIE_LINK_LEGACY_IRQ_BASE; - } - return -1; - } - devoff = 0; switch (irq) { case PIC_UART_0_IRQ: @@ -98,7 +108,7 @@ int nlm_irq_to_irt(int irq) devoff = XLP_IO_UART1_OFFSET(0); break; case PIC_MMC_IRQ: - devoff = XLP_IO_SD_OFFSET(0); + devoff = XLP_IO_MMC_OFFSET(0); break; case PIC_I2C_0_IRQ: /* I2C will be fixed up */ case PIC_I2C_1_IRQ: @@ -109,6 +119,18 @@ int nlm_irq_to_irt(int irq) else devoff = XLP_IO_I2C0_OFFSET(0); break; + case PIC_SATA_IRQ: + devoff = XLP_IO_SATA_OFFSET(0); + break; + case PIC_GPIO_IRQ: + devoff = XLP_IO_GPIO_OFFSET(0); + break; + case PIC_NAND_IRQ: + devoff = XLP_IO_NAND_OFFSET(0); + break; + case PIC_SPI_IRQ: + devoff = XLP_IO_SPI_OFFSET(0); + break; default: if (cpu_is_xlpii()) { switch (irq) { @@ -164,61 +186,123 @@ int nlm_irq_to_irt(int irq) /* HW bug, PCI IRT entries are bad on early silicon, fix */ irt = PIC_IRT_PCIE_LINK_INDEX(irq - PIC_PCIE_LINK_LEGACY_IRQ_BASE); - } else if (irq >= PIC_PCIE_LINK_MSI_IRQ(0) && - irq <= PIC_PCIE_LINK_MSI_IRQ(3)) { - irt = -2; - } else if (irq >= PIC_PCIE_MSIX_IRQ(0) && - irq <= PIC_PCIE_MSIX_IRQ(3)) { - irt = -2; } else { irt = -1; } return irt; } -unsigned int nlm_get_core_frequency(int node, int core) +int nlm_irq_to_irt(int irq) { - unsigned int pll_divf, pll_divr, dfs_div, ext_div; - unsigned int rstval, dfsval, denom; - uint64_t num, sysbase; + /* return -2 for irqs without 1-1 mapping */ + if (irq >= PIC_PCIE_LINK_MSI_IRQ(0) && irq <= PIC_PCIE_LINK_MSI_IRQ(3)) + return -2; + if (irq >= PIC_PCIE_MSIX_IRQ(0) && irq <= PIC_PCIE_MSIX_IRQ(3)) + return -2; - sysbase = nlm_get_node(node)->sysbase; if (cpu_is_xlp9xx()) - rstval = nlm_read_sys_reg(sysbase, SYS_9XX_POWER_ON_RESET_CFG); + return xlp9xx_irq_to_irt(irq); else - rstval = nlm_read_sys_reg(sysbase, SYS_POWER_ON_RESET_CFG); - if (cpu_is_xlpii()) { - num = 1000000ULL * (400 * 3 + 100 * (rstval >> 26)); - denom = 3; + return xlp_irq_to_irt(irq); +} + +static unsigned int nlm_xlp2_get_core_frequency(int node, int core) +{ + unsigned int pll_post_div, ctrl_val0, ctrl_val1, denom; + uint64_t num, sysbase, clockbase; + + if (cpu_is_xlp9xx()) { + clockbase = nlm_get_clock_regbase(node); + ctrl_val0 = nlm_read_sys_reg(clockbase, + SYS_9XX_CPU_PLL_CTRL0(core)); + ctrl_val1 = nlm_read_sys_reg(clockbase, + SYS_9XX_CPU_PLL_CTRL1(core)); } else { - dfsval = nlm_read_sys_reg(sysbase, SYS_CORE_DFS_DIV_VALUE); - pll_divf = ((rstval >> 10) & 0x7f) + 1; - pll_divr = ((rstval >> 8) & 0x3) + 1; - ext_div = ((rstval >> 30) & 0x3) + 1; - dfs_div = ((dfsval >> (core * 4)) & 0xf) + 1; - - num = 800000000ULL * pll_divf; - denom = 3 * pll_divr * ext_div * dfs_div; + sysbase = nlm_get_node(node)->sysbase; + ctrl_val0 = nlm_read_sys_reg(sysbase, + SYS_CPU_PLL_CTRL0(core)); + ctrl_val1 = nlm_read_sys_reg(sysbase, + SYS_CPU_PLL_CTRL1(core)); + } + + /* Find PLL post divider value */ + switch ((ctrl_val0 >> 24) & 0x7) { + case 1: + pll_post_div = 2; + break; + case 3: + pll_post_div = 4; + break; + case 7: + pll_post_div = 8; + break; + case 6: + pll_post_div = 16; + break; + case 0: + default: + pll_post_div = 1; + break; } + + num = 1000000ULL * (400 * 3 + 100 * (ctrl_val1 & 0x3f)); + denom = 3 * pll_post_div; do_div(num, denom); + return (unsigned int)num; } -/* Calculate Frequency to the PIC from PLL. - * freq_out = ( ref_freq/2 * (6 + ctrl2[7:0]) + ctrl2[20:8]/2^13 ) / - * ((2^ctrl0[7:5]) * Table(ctrl0[26:24])) +static unsigned int nlm_xlp_get_core_frequency(int node, int core) +{ + unsigned int pll_divf, pll_divr, dfs_div, ext_div; + unsigned int rstval, dfsval, denom; + uint64_t num, sysbase; + + sysbase = nlm_get_node(node)->sysbase; + rstval = nlm_read_sys_reg(sysbase, SYS_POWER_ON_RESET_CFG); + dfsval = nlm_read_sys_reg(sysbase, SYS_CORE_DFS_DIV_VALUE); + pll_divf = ((rstval >> 10) & 0x7f) + 1; + pll_divr = ((rstval >> 8) & 0x3) + 1; + ext_div = ((rstval >> 30) & 0x3) + 1; + dfs_div = ((dfsval >> (core * 4)) & 0xf) + 1; + + num = 800000000ULL * pll_divf; + denom = 3 * pll_divr * ext_div * dfs_div; + do_div(num, denom); + + return (unsigned int)num; +} + +unsigned int nlm_get_core_frequency(int node, int core) +{ + if (cpu_is_xlpii()) + return nlm_xlp2_get_core_frequency(node, core); + else + return nlm_xlp_get_core_frequency(node, core); +} + +/* + * Calculate PIC frequency from PLL registers. + * freq_out = (ref_freq/2 * (6 + ctrl2[7:0]) + ctrl2[20:8]/2^13) / + * ((2^ctrl0[7:5]) * Table(ctrl0[26:24])) */ -static unsigned int nlm_2xx_get_pic_frequency(int node) +static unsigned int nlm_xlp2_get_pic_frequency(int node) { - u32 ctrl_val0, ctrl_val2, vco_post_div, pll_post_div; + u32 ctrl_val0, ctrl_val2, vco_post_div, pll_post_div, cpu_xlp9xx; u32 mdiv, fdiv, pll_out_freq_den, reg_select, ref_div, pic_div; - u64 ref_clk, sysbase, pll_out_freq_num, ref_clk_select; + u64 sysbase, pll_out_freq_num, ref_clk_select, clockbase, ref_clk; sysbase = nlm_get_node(node)->sysbase; + clockbase = nlm_get_clock_regbase(node); + cpu_xlp9xx = cpu_is_xlp9xx(); /* Find ref_clk_base */ - ref_clk_select = - (nlm_read_sys_reg(sysbase, SYS_POWER_ON_RESET_CFG) >> 18) & 0x3; + if (cpu_xlp9xx) + ref_clk_select = (nlm_read_sys_reg(sysbase, + SYS_9XX_POWER_ON_RESET_CFG) >> 18) & 0x3; + else + ref_clk_select = (nlm_read_sys_reg(sysbase, + SYS_POWER_ON_RESET_CFG) >> 18) & 0x3; switch (ref_clk_select) { case 0: ref_clk = 200000000ULL; @@ -239,30 +323,70 @@ static unsigned int nlm_2xx_get_pic_frequency(int node) } /* Find the clock source PLL device for PIC */ - reg_select = (nlm_read_sys_reg(sysbase, SYS_CLK_DEV_SEL) >> 22) & 0x3; - switch (reg_select) { - case 0: - ctrl_val0 = nlm_read_sys_reg(sysbase, SYS_PLL_CTRL0); - ctrl_val2 = nlm_read_sys_reg(sysbase, SYS_PLL_CTRL2); - break; - case 1: - ctrl_val0 = nlm_read_sys_reg(sysbase, SYS_PLL_CTRL0_DEVX(0)); - ctrl_val2 = nlm_read_sys_reg(sysbase, SYS_PLL_CTRL2_DEVX(0)); - break; - case 2: - ctrl_val0 = nlm_read_sys_reg(sysbase, SYS_PLL_CTRL0_DEVX(1)); - ctrl_val2 = nlm_read_sys_reg(sysbase, SYS_PLL_CTRL2_DEVX(1)); - break; - case 3: - ctrl_val0 = nlm_read_sys_reg(sysbase, SYS_PLL_CTRL0_DEVX(2)); - ctrl_val2 = nlm_read_sys_reg(sysbase, SYS_PLL_CTRL2_DEVX(2)); - break; + if (cpu_xlp9xx) { + reg_select = nlm_read_sys_reg(clockbase, + SYS_9XX_CLK_DEV_SEL) & 0x3; + switch (reg_select) { + case 0: + ctrl_val0 = nlm_read_sys_reg(clockbase, + SYS_9XX_PLL_CTRL0); + ctrl_val2 = nlm_read_sys_reg(clockbase, + SYS_9XX_PLL_CTRL2); + break; + case 1: + ctrl_val0 = nlm_read_sys_reg(clockbase, + SYS_9XX_PLL_CTRL0_DEVX(0)); + ctrl_val2 = nlm_read_sys_reg(clockbase, + SYS_9XX_PLL_CTRL2_DEVX(0)); + break; + case 2: + ctrl_val0 = nlm_read_sys_reg(clockbase, + SYS_9XX_PLL_CTRL0_DEVX(1)); + ctrl_val2 = nlm_read_sys_reg(clockbase, + SYS_9XX_PLL_CTRL2_DEVX(1)); + break; + case 3: + ctrl_val0 = nlm_read_sys_reg(clockbase, + SYS_9XX_PLL_CTRL0_DEVX(2)); + ctrl_val2 = nlm_read_sys_reg(clockbase, + SYS_9XX_PLL_CTRL2_DEVX(2)); + break; + } + } else { + reg_select = (nlm_read_sys_reg(sysbase, + SYS_CLK_DEV_SEL) >> 22) & 0x3; + switch (reg_select) { + case 0: + ctrl_val0 = nlm_read_sys_reg(sysbase, + SYS_PLL_CTRL0); + ctrl_val2 = nlm_read_sys_reg(sysbase, + SYS_PLL_CTRL2); + break; + case 1: + ctrl_val0 = nlm_read_sys_reg(sysbase, + SYS_PLL_CTRL0_DEVX(0)); + ctrl_val2 = nlm_read_sys_reg(sysbase, + SYS_PLL_CTRL2_DEVX(0)); + break; + case 2: + ctrl_val0 = nlm_read_sys_reg(sysbase, + SYS_PLL_CTRL0_DEVX(1)); + ctrl_val2 = nlm_read_sys_reg(sysbase, + SYS_PLL_CTRL2_DEVX(1)); + break; + case 3: + ctrl_val0 = nlm_read_sys_reg(sysbase, + SYS_PLL_CTRL0_DEVX(2)); + ctrl_val2 = nlm_read_sys_reg(sysbase, + SYS_PLL_CTRL2_DEVX(2)); + break; + } } vco_post_div = (ctrl_val0 >> 5) & 0x7; pll_post_div = (ctrl_val0 >> 24) & 0x7; mdiv = ctrl_val2 & 0xff; - fdiv = (ctrl_val2 >> 8) & 0xfff; + fdiv = (ctrl_val2 >> 8) & 0x1fff; /* Find PLL post divider value */ switch (pll_post_div) { @@ -292,7 +416,12 @@ static unsigned int nlm_2xx_get_pic_frequency(int node) do_div(pll_out_freq_num, pll_out_freq_den); /* PIC post divider, which happens after PLL */ - pic_div = (nlm_read_sys_reg(sysbase, SYS_CLK_DEV_DIV) >> 22) & 0x3; + if (cpu_xlp9xx) + pic_div = nlm_read_sys_reg(clockbase, + SYS_9XX_CLK_DEV_DIV) & 0x3; + else + pic_div = (nlm_read_sys_reg(sysbase, + SYS_CLK_DEV_DIV) >> 22) & 0x3; do_div(pll_out_freq_num, 1 << pic_div); return pll_out_freq_num; @@ -300,12 +429,8 @@ static unsigned int nlm_2xx_get_pic_frequency(int node) unsigned int nlm_get_pic_frequency(int node) { - /* TODO Has to calculate freq as like 2xx */ - if (cpu_is_xlp9xx()) - return 250000000; - if (cpu_is_xlpii()) - return nlm_2xx_get_pic_frequency(node); + return nlm_xlp2_get_pic_frequency(node); else return 133333333; } diff --git a/arch/mips/netlogic/xlp/setup.c b/arch/mips/netlogic/xlp/setup.c index 8c60a2dd9ef6..4fdd9fd29d1d 100644 --- a/arch/mips/netlogic/xlp/setup.c +++ b/arch/mips/netlogic/xlp/setup.c @@ -121,8 +121,9 @@ void __init plat_mem_setup(void) const char *get_system_type(void) { - switch (read_c0_prid() & 0xff00) { + switch (read_c0_prid() & PRID_IMP_MASK) { case PRID_IMP_NETLOGIC_XLP9XX: + case PRID_IMP_NETLOGIC_XLP5XX: case PRID_IMP_NETLOGIC_XLP2XX: return "Broadcom XLPII Series"; default: diff --git a/arch/mips/netlogic/xlp/wakeup.c b/arch/mips/netlogic/xlp/wakeup.c index 9a92617a2af5..e5f44d2605a8 100644 --- a/arch/mips/netlogic/xlp/wakeup.c +++ b/arch/mips/netlogic/xlp/wakeup.c @@ -135,11 +135,19 @@ static void xlp_enable_secondary_cores(const cpumask_t *wakeup_mask) if (cpu_is_xlp9xx()) { fusebase = nlm_get_fuse_regbase(n); fusemask = nlm_read_reg(fusebase, FUSE_9XX_DEVCFG6); - mask = 0xfffff; + switch (read_c0_prid() & PRID_IMP_MASK) { + case PRID_IMP_NETLOGIC_XLP5XX: + mask = 0xff; + break; + case PRID_IMP_NETLOGIC_XLP9XX: + default: + mask = 0xfffff; + break; + } } else { fusemask = nlm_read_sys_reg(nodep->sysbase, SYS_EFUSE_DEVICE_CFG_STATUS0); - switch (read_c0_prid() & 0xff00) { + switch (read_c0_prid() & PRID_IMP_MASK) { case PRID_IMP_NETLOGIC_XLP3XX: mask = 0xf; break; @@ -159,10 +167,6 @@ static void xlp_enable_secondary_cores(const cpumask_t *wakeup_mask) */ syscoremask = (1 << hweight32(~fusemask & mask)) - 1; - /* The boot cpu */ - if (n == 0) - nodep->coremask = 1; - pr_info("Node %d - SYS/FUSE coremask %x\n", n, syscoremask); for (core = 0; core < nlm_cores_per_node(); core++) { /* we will be on node 0 core 0 */ diff --git a/arch/mips/paravirt/Kconfig b/arch/mips/paravirt/Kconfig new file mode 100644 index 000000000000..ecae5861b601 --- /dev/null +++ b/arch/mips/paravirt/Kconfig @@ -0,0 +1,6 @@ +if MIPS_PARAVIRT + +config MIPS_PCI_VIRTIO + def_bool y + +endif # MIPS_PARAVIRT diff --git a/arch/mips/paravirt/Makefile b/arch/mips/paravirt/Makefile new file mode 100644 index 000000000000..5023af733a35 --- /dev/null +++ b/arch/mips/paravirt/Makefile @@ -0,0 +1,14 @@ +# +# Makefile for MIPS para-virtualized specific kernel interface routines +# under Linux. +# +# This file is subject to the terms and conditions of the GNU General Public +# License. See the file "COPYING" in the main directory of this archive +# for more details. +# +# Copyright (C) 2013 Cavium, Inc. +# + +obj-y := setup.o serial.o paravirt-irq.o + +obj-$(CONFIG_SMP) += paravirt-smp.o diff --git a/arch/mips/paravirt/Platform b/arch/mips/paravirt/Platform new file mode 100644 index 000000000000..7e76ef25ea17 --- /dev/null +++ b/arch/mips/paravirt/Platform @@ -0,0 +1,8 @@ +# +# Generic para-virtualized guest. +# +platform-$(CONFIG_MIPS_PARAVIRT) += paravirt/ +cflags-$(CONFIG_MIPS_PARAVIRT) += \ + -I$(srctree)/arch/mips/include/asm/mach-paravirt + +load-$(CONFIG_MIPS_PARAVIRT) = 0xffffffff80010000 diff --git a/arch/mips/paravirt/paravirt-irq.c b/arch/mips/paravirt/paravirt-irq.c new file mode 100644 index 000000000000..8987b06c9de9 --- /dev/null +++ b/arch/mips/paravirt/paravirt-irq.c @@ -0,0 +1,368 @@ +/* + * This file is subject to the terms and conditions of the GNU General Public + * License. See the file "COPYING" in the main directory of this archive + * for more details. + * + * Copyright (C) 2013 Cavium, Inc. + */ + +#include <linux/interrupt.h> +#include <linux/cpumask.h> +#include <linux/kernel.h> +#include <linux/mutex.h> + +#include <asm/io.h> + +#define MBOX_BITS_PER_CPU 2 + +static int cpunum_for_cpu(int cpu) +{ +#ifdef CONFIG_SMP + return cpu_logical_map(cpu); +#else + return get_ebase_cpunum(); +#endif +} + +struct core_chip_data { + struct mutex core_irq_mutex; + bool current_en; + bool desired_en; + u8 bit; +}; + +static struct core_chip_data irq_core_chip_data[8]; + +static void irq_core_ack(struct irq_data *data) +{ + struct core_chip_data *cd = irq_data_get_irq_chip_data(data); + unsigned int bit = cd->bit; + + /* + * We don't need to disable IRQs to make these atomic since + * they are already disabled earlier in the low level + * interrupt code. + */ + clear_c0_status(0x100 << bit); + /* The two user interrupts must be cleared manually. */ + if (bit < 2) + clear_c0_cause(0x100 << bit); +} + +static void irq_core_eoi(struct irq_data *data) +{ + struct core_chip_data *cd = irq_data_get_irq_chip_data(data); + + /* + * We don't need to disable IRQs to make these atomic since + * they are already disabled earlier in the low level + * interrupt code. + */ + set_c0_status(0x100 << cd->bit); +} + +static void irq_core_set_enable_local(void *arg) +{ + struct irq_data *data = arg; + struct core_chip_data *cd = irq_data_get_irq_chip_data(data); + unsigned int mask = 0x100 << cd->bit; + + /* + * Interrupts are already disabled, so these are atomic. + */ + if (cd->desired_en) + set_c0_status(mask); + else + clear_c0_status(mask); + +} + +static void irq_core_disable(struct irq_data *data) +{ + struct core_chip_data *cd = irq_data_get_irq_chip_data(data); + cd->desired_en = false; +} + +static void irq_core_enable(struct irq_data *data) +{ + struct core_chip_data *cd = irq_data_get_irq_chip_data(data); + cd->desired_en = true; +} + +static void irq_core_bus_lock(struct irq_data *data) +{ + struct core_chip_data *cd = irq_data_get_irq_chip_data(data); + + mutex_lock(&cd->core_irq_mutex); +} + +static void irq_core_bus_sync_unlock(struct irq_data *data) +{ + struct core_chip_data *cd = irq_data_get_irq_chip_data(data); + + if (cd->desired_en != cd->current_en) { + on_each_cpu(irq_core_set_enable_local, data, 1); + cd->current_en = cd->desired_en; + } + + mutex_unlock(&cd->core_irq_mutex); +} + +static struct irq_chip irq_chip_core = { + .name = "Core", + .irq_enable = irq_core_enable, + .irq_disable = irq_core_disable, + .irq_ack = irq_core_ack, + .irq_eoi = irq_core_eoi, + .irq_bus_lock = irq_core_bus_lock, + .irq_bus_sync_unlock = irq_core_bus_sync_unlock, + + .irq_cpu_online = irq_core_eoi, + .irq_cpu_offline = irq_core_ack, + .flags = IRQCHIP_ONOFFLINE_ENABLED, +}; + +static void __init irq_init_core(void) +{ + int i; + int irq; + struct core_chip_data *cd; + + /* Start with a clean slate */ + clear_c0_status(ST0_IM); + clear_c0_cause(CAUSEF_IP0 | CAUSEF_IP1); + + for (i = 0; i < ARRAY_SIZE(irq_core_chip_data); i++) { + cd = irq_core_chip_data + i; + cd->current_en = false; + cd->desired_en = false; + cd->bit = i; + mutex_init(&cd->core_irq_mutex); + + irq = MIPS_CPU_IRQ_BASE + i; + + switch (i) { + case 0: /* SW0 */ + case 1: /* SW1 */ + case 5: /* IP5 */ + case 6: /* IP6 */ + case 7: /* IP7 */ + irq_set_chip_data(irq, cd); + irq_set_chip_and_handler(irq, &irq_chip_core, + handle_percpu_irq); + break; + default: + break; + } + } +} + +static void __iomem *mips_irq_chip; +#define MIPS_IRQ_CHIP_NUM_BITS 0 +#define MIPS_IRQ_CHIP_REGS 8 + +static int mips_irq_cpu_stride; +static int mips_irq_chip_reg_raw; +static int mips_irq_chip_reg_src; +static int mips_irq_chip_reg_en; +static int mips_irq_chip_reg_raw_w1s; +static int mips_irq_chip_reg_raw_w1c; +static int mips_irq_chip_reg_en_w1s; +static int mips_irq_chip_reg_en_w1c; + +static void irq_pci_enable(struct irq_data *data) +{ + u32 mask = 1u << data->irq; + + __raw_writel(mask, mips_irq_chip + mips_irq_chip_reg_en_w1s); +} + +static void irq_pci_disable(struct irq_data *data) +{ + u32 mask = 1u << data->irq; + + __raw_writel(mask, mips_irq_chip + mips_irq_chip_reg_en_w1c); +} + +static void irq_pci_ack(struct irq_data *data) +{ +} + +static void irq_pci_mask(struct irq_data *data) +{ + u32 mask = 1u << data->irq; + + __raw_writel(mask, mips_irq_chip + mips_irq_chip_reg_en_w1c); +} + +static void irq_pci_unmask(struct irq_data *data) +{ + u32 mask = 1u << data->irq; + + __raw_writel(mask, mips_irq_chip + mips_irq_chip_reg_en_w1s); +} + +static struct irq_chip irq_chip_pci = { + .name = "PCI", + .irq_enable = irq_pci_enable, + .irq_disable = irq_pci_disable, + .irq_ack = irq_pci_ack, + .irq_mask = irq_pci_mask, + .irq_unmask = irq_pci_unmask, +}; + +static void irq_mbox_all(struct irq_data *data, void __iomem *base) +{ + int cpu; + unsigned int mbox = data->irq - MIPS_IRQ_MBOX0; + u32 mask; + + WARN_ON(mbox >= MBOX_BITS_PER_CPU); + + for_each_online_cpu(cpu) { + unsigned int cpuid = cpunum_for_cpu(cpu); + mask = 1 << (cpuid * MBOX_BITS_PER_CPU + mbox); + __raw_writel(mask, base + (cpuid * mips_irq_cpu_stride)); + } +} + +static void irq_mbox_enable(struct irq_data *data) +{ + irq_mbox_all(data, mips_irq_chip + mips_irq_chip_reg_en_w1s + sizeof(u32)); +} + +static void irq_mbox_disable(struct irq_data *data) +{ + irq_mbox_all(data, mips_irq_chip + mips_irq_chip_reg_en_w1c + sizeof(u32)); +} + +static void irq_mbox_ack(struct irq_data *data) +{ + u32 mask; + unsigned int mbox = data->irq - MIPS_IRQ_MBOX0; + + WARN_ON(mbox >= MBOX_BITS_PER_CPU); + + mask = 1 << (get_ebase_cpunum() * MBOX_BITS_PER_CPU + mbox); + __raw_writel(mask, mips_irq_chip + mips_irq_chip_reg_raw_w1c + sizeof(u32)); +} + +void irq_mbox_ipi(int cpu, unsigned int actions) +{ + unsigned int cpuid = cpunum_for_cpu(cpu); + u32 mask; + + WARN_ON(actions >= (1 << MBOX_BITS_PER_CPU)); + + mask = actions << (cpuid * MBOX_BITS_PER_CPU); + __raw_writel(mask, mips_irq_chip + mips_irq_chip_reg_raw_w1s + sizeof(u32)); +} + +static void irq_mbox_cpu_onoffline(struct irq_data *data, void __iomem *base) +{ + unsigned int mbox = data->irq - MIPS_IRQ_MBOX0; + unsigned int cpuid = get_ebase_cpunum(); + u32 mask; + + WARN_ON(mbox >= MBOX_BITS_PER_CPU); + + mask = 1 << (cpuid * MBOX_BITS_PER_CPU + mbox); + __raw_writel(mask, base + (cpuid * mips_irq_cpu_stride)); + +} + +static void irq_mbox_cpu_online(struct irq_data *data) +{ + irq_mbox_cpu_onoffline(data, mips_irq_chip + mips_irq_chip_reg_en_w1s + sizeof(u32)); +} + +static void irq_mbox_cpu_offline(struct irq_data *data) +{ + irq_mbox_cpu_onoffline(data, mips_irq_chip + mips_irq_chip_reg_en_w1c + sizeof(u32)); +} + +static struct irq_chip irq_chip_mbox = { + .name = "MBOX", + .irq_enable = irq_mbox_enable, + .irq_disable = irq_mbox_disable, + .irq_ack = irq_mbox_ack, + .irq_cpu_online = irq_mbox_cpu_online, + .irq_cpu_offline = irq_mbox_cpu_offline, + .flags = IRQCHIP_ONOFFLINE_ENABLED, +}; + +static void __init irq_pci_init(void) +{ + int i, stride; + u32 num_bits; + + mips_irq_chip = ioremap(0x1e010000, 4096); + + num_bits = __raw_readl(mips_irq_chip + MIPS_IRQ_CHIP_NUM_BITS); + stride = 8 * (1 + ((num_bits - 1) / 64)); + + + pr_notice("mips_irq_chip: %u bits, reg stride: %d\n", num_bits, stride); + mips_irq_chip_reg_raw = MIPS_IRQ_CHIP_REGS + 0 * stride; + mips_irq_chip_reg_raw_w1s = MIPS_IRQ_CHIP_REGS + 1 * stride; + mips_irq_chip_reg_raw_w1c = MIPS_IRQ_CHIP_REGS + 2 * stride; + mips_irq_chip_reg_src = MIPS_IRQ_CHIP_REGS + 3 * stride; + mips_irq_chip_reg_en = MIPS_IRQ_CHIP_REGS + 4 * stride; + mips_irq_chip_reg_en_w1s = MIPS_IRQ_CHIP_REGS + 5 * stride; + mips_irq_chip_reg_en_w1c = MIPS_IRQ_CHIP_REGS + 6 * stride; + mips_irq_cpu_stride = stride * 4; + + for (i = 0; i < 4; i++) + irq_set_chip_and_handler(i + MIPS_IRQ_PCIA, &irq_chip_pci, handle_level_irq); + + for (i = 0; i < 2; i++) + irq_set_chip_and_handler(i + MIPS_IRQ_MBOX0, &irq_chip_mbox, handle_percpu_irq); + + + set_c0_status(STATUSF_IP2); +} + +static void irq_pci_dispatch(void) +{ + unsigned int cpuid = get_ebase_cpunum(); + u32 en; + + en = __raw_readl(mips_irq_chip + mips_irq_chip_reg_src + + (cpuid * mips_irq_cpu_stride)); + + if (!en) { + en = __raw_readl(mips_irq_chip + mips_irq_chip_reg_src + (cpuid * mips_irq_cpu_stride) + sizeof(u32)); + en = (en >> (2 * cpuid)) & 3; + + if (!en) + spurious_interrupt(); + else + do_IRQ(__ffs(en) + MIPS_IRQ_MBOX0); /* MBOX type */ + } else { + do_IRQ(__ffs(en)); + } +} + + +void __init arch_init_irq(void) +{ + irq_init_core(); + irq_pci_init(); +} + +asmlinkage void plat_irq_dispatch(void) +{ + unsigned int pending = read_c0_cause() & read_c0_status() & ST0_IM; + int ip; + + if (unlikely(!pending)) { + spurious_interrupt(); + return; + } + + ip = ffs(pending) - 1 - STATUSB_IP0; + if (ip == 2) + irq_pci_dispatch(); + else + do_IRQ(MIPS_CPU_IRQ_BASE + ip); +} diff --git a/arch/mips/paravirt/paravirt-smp.c b/arch/mips/paravirt/paravirt-smp.c new file mode 100644 index 000000000000..0164b0c48352 --- /dev/null +++ b/arch/mips/paravirt/paravirt-smp.c @@ -0,0 +1,143 @@ +/* + * This file is subject to the terms and conditions of the GNU General Public + * License. See the file "COPYING" in the main directory of this archive + * for more details. + * + * Copyright (C) 2013 Cavium, Inc. + */ + +#include <linux/interrupt.h> +#include <linux/cpumask.h> +#include <linux/kernel.h> +#include <linux/sched.h> + +#include <asm/mipsregs.h> +#include <asm/setup.h> +#include <asm/time.h> +#include <asm/smp.h> + +/* + * Writing the sp releases the CPU, so writes must be ordered, gp + * first, then sp. + */ +unsigned long paravirt_smp_sp[NR_CPUS]; +unsigned long paravirt_smp_gp[NR_CPUS]; + +static int numcpus = 1; + +static int __init set_numcpus(char *str) +{ + int newval; + + if (get_option(&str, &newval)) { + if (newval < 1 || newval >= NR_CPUS) + goto bad; + numcpus = newval; + return 0; + } +bad: + return -EINVAL; +} +early_param("numcpus", set_numcpus); + + +static void paravirt_smp_setup(void) +{ + int id; + unsigned int cpunum = get_ebase_cpunum(); + + if (WARN_ON(cpunum >= NR_CPUS)) + return; + + /* The present CPUs are initially just the boot cpu (CPU 0). */ + for (id = 0; id < NR_CPUS; id++) { + set_cpu_possible(id, id == 0); + set_cpu_present(id, id == 0); + } + __cpu_number_map[cpunum] = 0; + __cpu_logical_map[0] = cpunum; + + for (id = 0; id < numcpus; id++) { + set_cpu_possible(id, true); + set_cpu_present(id, true); + __cpu_number_map[id] = id; + __cpu_logical_map[id] = id; + } +} + +void irq_mbox_ipi(int cpu, unsigned int actions); +static void paravirt_send_ipi_single(int cpu, unsigned int action) +{ + irq_mbox_ipi(cpu, action); +} + +static void paravirt_send_ipi_mask(const struct cpumask *mask, unsigned int action) +{ + unsigned int cpu; + + for_each_cpu_mask(cpu, *mask) + paravirt_send_ipi_single(cpu, action); +} + +static void paravirt_init_secondary(void) +{ + unsigned int sr; + + sr = set_c0_status(ST0_BEV); + write_c0_ebase((u32)ebase); + + sr |= STATUSF_IP2; /* Interrupt controller on IP2 */ + write_c0_status(sr); + + irq_cpu_online(); +} + +static void paravirt_smp_finish(void) +{ + /* to generate the first CPU timer interrupt */ + write_c0_compare(read_c0_count() + mips_hpt_frequency / HZ); + local_irq_enable(); +} + +static void paravirt_boot_secondary(int cpu, struct task_struct *idle) +{ + paravirt_smp_gp[cpu] = (unsigned long)task_thread_info(idle); + smp_wmb(); + paravirt_smp_sp[cpu] = __KSTK_TOS(idle); +} + +static irqreturn_t paravirt_reched_interrupt(int irq, void *dev_id) +{ + scheduler_ipi(); + return IRQ_HANDLED; +} + +static irqreturn_t paravirt_function_interrupt(int irq, void *dev_id) +{ + smp_call_function_interrupt(); + return IRQ_HANDLED; +} + +static void paravirt_prepare_cpus(unsigned int max_cpus) +{ + if (request_irq(MIPS_IRQ_MBOX0, paravirt_reched_interrupt, + IRQF_PERCPU | IRQF_NO_THREAD, "Scheduler", + paravirt_reched_interrupt)) { + panic("Cannot request_irq for SchedulerIPI"); + } + if (request_irq(MIPS_IRQ_MBOX1, paravirt_function_interrupt, + IRQF_PERCPU | IRQF_NO_THREAD, "SMP-Call", + paravirt_function_interrupt)) { + panic("Cannot request_irq for SMP-Call"); + } +} + +struct plat_smp_ops paravirt_smp_ops = { + .send_ipi_single = paravirt_send_ipi_single, + .send_ipi_mask = paravirt_send_ipi_mask, + .init_secondary = paravirt_init_secondary, + .smp_finish = paravirt_smp_finish, + .boot_secondary = paravirt_boot_secondary, + .smp_setup = paravirt_smp_setup, + .prepare_cpus = paravirt_prepare_cpus, +}; diff --git a/arch/mips/paravirt/serial.c b/arch/mips/paravirt/serial.c new file mode 100644 index 000000000000..02b665c02272 --- /dev/null +++ b/arch/mips/paravirt/serial.c @@ -0,0 +1,40 @@ +/* + * This file is subject to the terms and conditions of the GNU General Public + * License. See the file "COPYING" in the main directory of this archive + * for more details. + * + * Copyright (C) 2013 Cavium, Inc. + */ + +#include <linux/kernel.h> +#include <linux/virtio_console.h> +#include <linux/kvm_para.h> + +/* + * Emit one character to the boot console. + */ +int prom_putchar(char c) +{ + kvm_hypercall3(KVM_HC_MIPS_CONSOLE_OUTPUT, 0 /* port 0 */, + (unsigned long)&c, 1 /* len == 1 */); + + return 1; +} + +#ifdef CONFIG_VIRTIO_CONSOLE +static int paravirt_put_chars(u32 vtermno, const char *buf, int count) +{ + kvm_hypercall3(KVM_HC_MIPS_CONSOLE_OUTPUT, vtermno, + (unsigned long)buf, count); + + return count; +} + +static int __init paravirt_cons_init(void) +{ + virtio_cons_early_init(paravirt_put_chars); + return 0; +} +core_initcall(paravirt_cons_init); + +#endif diff --git a/arch/mips/paravirt/setup.c b/arch/mips/paravirt/setup.c new file mode 100644 index 000000000000..cb8448b373a7 --- /dev/null +++ b/arch/mips/paravirt/setup.c @@ -0,0 +1,67 @@ +/* + * This file is subject to the terms and conditions of the GNU General Public + * License. See the file "COPYING" in the main directory of this archive + * for more details. + * + * Copyright (C) 2013 Cavium, Inc. + */ + +#include <linux/kernel.h> +#include <linux/kvm_para.h> + +#include <asm/reboot.h> +#include <asm/bootinfo.h> +#include <asm/smp-ops.h> +#include <asm/time.h> + +extern struct plat_smp_ops paravirt_smp_ops; + +const char *get_system_type(void) +{ + return "MIPS Para-Virtualized Guest"; +} + +void __init plat_time_init(void) +{ + mips_hpt_frequency = kvm_hypercall0(KVM_HC_MIPS_GET_CLOCK_FREQ); + + preset_lpj = mips_hpt_frequency / (2 * HZ); +} + +static void pv_machine_halt(void) +{ + kvm_hypercall0(KVM_HC_MIPS_EXIT_VM); +} + +/* + * Early entry point for arch setup + */ +void __init prom_init(void) +{ + int i; + int argc = fw_arg0; + char **argv = (char **)fw_arg1; + +#ifdef CONFIG_32BIT + set_io_port_base(KSEG1ADDR(0x1e000000)); +#else /* CONFIG_64BIT */ + set_io_port_base(PHYS_TO_XKSEG_UNCACHED(0x1e000000)); +#endif + + for (i = 0; i < argc; i++) { + strlcat(arcs_cmdline, argv[i], COMMAND_LINE_SIZE); + if (i < argc - 1) + strlcat(arcs_cmdline, " ", COMMAND_LINE_SIZE); + } + _machine_halt = pv_machine_halt; + register_smp_ops(¶virt_smp_ops); +} + +void __init plat_mem_setup(void) +{ + /* Do nothing, the "mem=???" parser handles our memory. */ +} + +void __init prom_free_prom_memory(void) +{ +} diff --git a/arch/mips/pci/Makefile b/arch/mips/pci/Makefile index d61138a177cc..ff8a5539b363 100644 --- a/arch/mips/pci/Makefile +++ b/arch/mips/pci/Makefile @@ -21,7 +21,7 @@ obj-$(CONFIG_BCM63XX) += pci-bcm63xx.o fixup-bcm63xx.o \ obj-$(CONFIG_MIPS_ALCHEMY) += pci-alchemy.o obj-$(CONFIG_SOC_AR71XX) += pci-ar71xx.o obj-$(CONFIG_PCI_AR724X) += pci-ar724x.o - +obj-$(CONFIG_MIPS_PCI_VIRTIO) += pci-virtio-guest.o # # These are still pretty much in the old state, watch, go blind. # diff --git a/arch/mips/pci/fixup-malta.c b/arch/mips/pci/fixup-malta.c index 2f9e52a1a750..40e920c653cc 100644 --- a/arch/mips/pci/fixup-malta.c +++ b/arch/mips/pci/fixup-malta.c @@ -68,6 +68,7 @@ static void malta_piix_func0_fixup(struct pci_dev *pdev) { unsigned char reg_val; u32 reg_val32; + u16 reg_val16; /* PIIX PIRQC[A:D] irq mappings */ static int piixirqmap[PIIX4_FUNC0_PIRQRC_IRQ_ROUTING_MAX] = { 0, 0, 0, 3, @@ -107,6 +108,11 @@ static void malta_piix_func0_fixup(struct pci_dev *pdev) pci_read_config_byte(pdev, PIIX4_FUNC0_SERIRQC, ®_val); reg_val |= PIIX4_FUNC0_SERIRQC_EN | PIIX4_FUNC0_SERIRQC_CONT; pci_write_config_byte(pdev, PIIX4_FUNC0_SERIRQC, reg_val); + + /* Enable response to special cycles */ + pci_read_config_word(pdev, PCI_COMMAND, ®_val16); + pci_write_config_word(pdev, PCI_COMMAND, + reg_val16 | PCI_COMMAND_SPECIAL); } DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_82371AB_0, diff --git a/arch/mips/pci/msi-octeon.c b/arch/mips/pci/msi-octeon.c index 2b91b0e61566..ab0c5d14c6f7 100644 --- a/arch/mips/pci/msi-octeon.c +++ b/arch/mips/pci/msi-octeon.c @@ -15,6 +15,7 @@ #include <asm/octeon/cvmx-npi-defs.h> #include <asm/octeon/cvmx-pci-defs.h> #include <asm/octeon/cvmx-npei-defs.h> +#include <asm/octeon/cvmx-sli-defs.h> #include <asm/octeon/cvmx-pexp-defs.h> #include <asm/octeon/pci-octeon.h> @@ -162,6 +163,11 @@ msi_irq_allocated: msg.address_lo = (0 + CVMX_NPEI_PCIE_MSI_RCV) & 0xffffffff; msg.address_hi = (0 + CVMX_NPEI_PCIE_MSI_RCV) >> 32; break; + case OCTEON_DMA_BAR_TYPE_PCIE2: + /* When using PCIe2, Bar 0 is based at 0 */ + msg.address_lo = (0 + CVMX_SLI_PCIE_MSI_RCV) & 0xffffffff; + msg.address_hi = (0 + CVMX_SLI_PCIE_MSI_RCV) >> 32; + break; default: panic("arch_setup_msi_irq: Invalid octeon_dma_bar_type"); } diff --git a/arch/mips/pci/msi-xlp.c b/arch/mips/pci/msi-xlp.c index 3249685e03ad..fa374fe3746b 100644 --- a/arch/mips/pci/msi-xlp.c +++ b/arch/mips/pci/msi-xlp.c @@ -56,8 +56,8 @@ #include <asm/netlogic/xlp-hal/bridge.h> #define XLP_MSIVEC_PER_LINK 32 -#define XLP_MSIXVEC_TOTAL 32 -#define XLP_MSIXVEC_PER_LINK 8 +#define XLP_MSIXVEC_TOTAL (cpu_is_xlp9xx() ? 128 : 32) +#define XLP_MSIXVEC_PER_LINK (cpu_is_xlp9xx() ? 32 : 8) /* 128 MSI irqs per node, mapped starting at NLM_MSI_VEC_BASE */ static inline int nlm_link_msiirq(int link, int msivec) @@ -65,35 +65,44 @@ static inline int nlm_link_msiirq(int link, int msivec) return NLM_MSI_VEC_BASE + link * XLP_MSIVEC_PER_LINK + msivec; } +/* get the link MSI vector from irq number */ static inline int nlm_irq_msivec(int irq) { - return irq % XLP_MSIVEC_PER_LINK; + return (irq - NLM_MSI_VEC_BASE) % XLP_MSIVEC_PER_LINK; } +/* get the link from the irq number */ static inline int nlm_irq_msilink(int irq) { - return (irq % (XLP_MSIVEC_PER_LINK * PCIE_NLINKS)) / - XLP_MSIVEC_PER_LINK; + int total_msivec = XLP_MSIVEC_PER_LINK * PCIE_NLINKS; + + return ((irq - NLM_MSI_VEC_BASE) % total_msivec) / + XLP_MSIVEC_PER_LINK; } /* - * Only 32 MSI-X vectors are possible because there are only 32 PIC - * interrupts for MSI. We split them statically and use 8 MSI-X vectors - * per link - this keeps the allocation and lookup simple. + * For XLP 8xx/4xx/3xx/2xx, only 32 MSI-X vectors are possible because + * there are only 32 PIC interrupts for MSI. We split them statically + * and use 8 MSI-X vectors per link - this keeps the allocation and + * lookup simple. + * On XLP 9xx, there are 32 vectors per link, and the interrupts are + * not routed thru PIC, so we can use all 128 MSI-X vectors. */ static inline int nlm_link_msixirq(int link, int bit) { return NLM_MSIX_VEC_BASE + link * XLP_MSIXVEC_PER_LINK + bit; } +/* get the link MSI vector from irq number */ static inline int nlm_irq_msixvec(int irq) { - return irq % XLP_MSIXVEC_TOTAL; /* works when given xirq */ + return (irq - NLM_MSIX_VEC_BASE) % XLP_MSIXVEC_TOTAL; } -static inline int nlm_irq_msixlink(int irq) +/* get the link from MSIX vec */ +static inline int nlm_irq_msixlink(int msixvec) { - return nlm_irq_msixvec(irq) / XLP_MSIXVEC_PER_LINK; + return msixvec / XLP_MSIXVEC_PER_LINK; } /* @@ -129,7 +138,11 @@ static void xlp_msi_enable(struct irq_data *d) vec = nlm_irq_msivec(d->irq); spin_lock_irqsave(&md->msi_lock, flags); md->msi_enabled_mask |= 1u << vec; - nlm_write_reg(md->lnkbase, PCIE_MSI_EN, md->msi_enabled_mask); + if (cpu_is_xlp9xx()) + nlm_write_reg(md->lnkbase, PCIE_9XX_MSI_EN, + md->msi_enabled_mask); + else + nlm_write_reg(md->lnkbase, PCIE_MSI_EN, md->msi_enabled_mask); spin_unlock_irqrestore(&md->msi_lock, flags); } @@ -142,7 +155,11 @@ static void xlp_msi_disable(struct irq_data *d) vec = nlm_irq_msivec(d->irq); spin_lock_irqsave(&md->msi_lock, flags); md->msi_enabled_mask &= ~(1u << vec); - nlm_write_reg(md->lnkbase, PCIE_MSI_EN, md->msi_enabled_mask); + if (cpu_is_xlp9xx()) + nlm_write_reg(md->lnkbase, PCIE_9XX_MSI_EN, + md->msi_enabled_mask); + else + nlm_write_reg(md->lnkbase, PCIE_MSI_EN, md->msi_enabled_mask); spin_unlock_irqrestore(&md->msi_lock, flags); } @@ -156,11 +173,18 @@ static void xlp_msi_mask_ack(struct irq_data *d) xlp_msi_disable(d); /* Ack MSI on bridge */ - nlm_write_reg(md->lnkbase, PCIE_MSI_STATUS, 1u << vec); + if (cpu_is_xlp9xx()) + nlm_write_reg(md->lnkbase, PCIE_9XX_MSI_STATUS, 1u << vec); + else + nlm_write_reg(md->lnkbase, PCIE_MSI_STATUS, 1u << vec); /* Ack at eirr and PIC */ ack_c0_eirr(PIC_PCIE_LINK_MSI_IRQ(link)); - nlm_pic_ack(md->node->picbase, PIC_IRT_PCIE_LINK_INDEX(link)); + if (cpu_is_xlp9xx()) + nlm_pic_ack(md->node->picbase, + PIC_9XX_IRT_PCIE_LINK_INDEX(link)); + else + nlm_pic_ack(md->node->picbase, PIC_IRT_PCIE_LINK_INDEX(link)); } static struct irq_chip xlp_msi_chip = { @@ -172,30 +196,45 @@ static struct irq_chip xlp_msi_chip = { }; /* - * The MSI-X interrupt handling is different from MSI, there are 32 - * MSI-X interrupts generated by the PIC and each of these correspond - * to a MSI-X vector (0-31) that can be assigned. + * XLP8XX/4XX/3XX/2XX: + * The MSI-X interrupt handling is different from MSI, there are 32 MSI-X + * interrupts generated by the PIC and each of these correspond to a MSI-X + * vector (0-31) that can be assigned. * - * We divide the MSI-X vectors to 8 per link and do a per-link - * allocation + * We divide the MSI-X vectors to 8 per link and do a per-link allocation + * + * XLP9XX: + * 32 MSI-X vectors are available per link, and the interrupts are not routed + * thru the PIC. PIC ack not needed. * * Enable and disable done using standard MSI functions. */ static void xlp_msix_mask_ack(struct irq_data *d) { - struct xlp_msi_data *md = irq_data_get_irq_handler_data(d); + struct xlp_msi_data *md; int link, msixvec; + uint32_t status_reg, bit; msixvec = nlm_irq_msixvec(d->irq); - link = nlm_irq_msixlink(d->irq); + link = nlm_irq_msixlink(msixvec); mask_msi_irq(d); + md = irq_data_get_irq_handler_data(d); /* Ack MSI on bridge */ - nlm_write_reg(md->lnkbase, PCIE_MSIX_STATUS, 1u << msixvec); + if (cpu_is_xlp9xx()) { + status_reg = PCIE_9XX_MSIX_STATUSX(link); + bit = msixvec % XLP_MSIXVEC_PER_LINK; + } else { + status_reg = PCIE_MSIX_STATUS; + bit = msixvec; + } + nlm_write_reg(md->lnkbase, status_reg, 1u << bit); /* Ack at eirr and PIC */ ack_c0_eirr(PIC_PCIE_MSIX_IRQ(link)); - nlm_pic_ack(md->node->picbase, PIC_IRT_PCIE_MSIX_INDEX(msixvec)); + if (!cpu_is_xlp9xx()) + nlm_pic_ack(md->node->picbase, + PIC_IRT_PCIE_MSIX_INDEX(msixvec)); } static struct irq_chip xlp_msix_chip = { @@ -219,10 +258,18 @@ static void xlp_config_link_msi(uint64_t lnkbase, int lirq, uint64_t msiaddr) { u32 val; - val = nlm_read_reg(lnkbase, PCIE_INT_EN0); - if ((val & 0x200) == 0) { - val |= 0x200; /* MSI Interrupt enable */ - nlm_write_reg(lnkbase, PCIE_INT_EN0, val); + if (cpu_is_xlp9xx()) { + val = nlm_read_reg(lnkbase, PCIE_9XX_INT_EN0); + if ((val & 0x200) == 0) { + val |= 0x200; /* MSI Interrupt enable */ + nlm_write_reg(lnkbase, PCIE_9XX_INT_EN0, val); + } + } else { + val = nlm_read_reg(lnkbase, PCIE_INT_EN0); + if ((val & 0x200) == 0) { + val |= 0x200; + nlm_write_reg(lnkbase, PCIE_INT_EN0, val); + } } val = nlm_read_reg(lnkbase, 0x1); /* CMD */ @@ -269,9 +316,12 @@ static int xlp_setup_msi(uint64_t lnkbase, int node, int link, spin_lock_irqsave(&md->msi_lock, flags); if (md->msi_alloc_mask == 0) { - /* switch the link IRQ to MSI range */ xlp_config_link_msi(lnkbase, lirq, msiaddr); - irt = PIC_IRT_PCIE_LINK_INDEX(link); + /* switch the link IRQ to MSI range */ + if (cpu_is_xlp9xx()) + irt = PIC_9XX_IRT_PCIE_LINK_INDEX(link); + else + irt = PIC_IRT_PCIE_LINK_INDEX(link); nlm_setup_pic_irq(node, lirq, lirq, irt); nlm_pic_init_irt(nlm_get_node(node)->picbase, irt, lirq, node * nlm_threads_per_node(), 1 /*en */); @@ -311,10 +361,19 @@ static void xlp_config_link_msix(uint64_t lnkbase, int lirq, uint64_t msixaddr) val |= 0x80000000U; nlm_write_reg(lnkbase, 0x2C, val); } - val = nlm_read_reg(lnkbase, PCIE_INT_EN0); - if ((val & 0x200) == 0) { - val |= 0x200; /* MSI Interrupt enable */ - nlm_write_reg(lnkbase, PCIE_INT_EN0, val); + + if (cpu_is_xlp9xx()) { + val = nlm_read_reg(lnkbase, PCIE_9XX_INT_EN0); + if ((val & 0x200) == 0) { + val |= 0x200; /* MSI Interrupt enable */ + nlm_write_reg(lnkbase, PCIE_9XX_INT_EN0, val); + } + } else { + val = nlm_read_reg(lnkbase, PCIE_INT_EN0); + if ((val & 0x200) == 0) { + val |= 0x200; /* MSI Interrupt enable */ + nlm_write_reg(lnkbase, PCIE_INT_EN0, val); + } } val = nlm_read_reg(lnkbase, 0x1); /* CMD */ @@ -329,10 +388,19 @@ static void xlp_config_link_msix(uint64_t lnkbase, int lirq, uint64_t msixaddr) val |= (1 << 8) | lirq; nlm_write_pci_reg(lnkbase, 0xf, val); - /* MSI-X addresses */ - nlm_write_reg(lnkbase, PCIE_BRIDGE_MSIX_ADDR_BASE, msixaddr >> 8); - nlm_write_reg(lnkbase, PCIE_BRIDGE_MSIX_ADDR_LIMIT, - (msixaddr + MSI_ADDR_SZ) >> 8); + if (cpu_is_xlp9xx()) { + /* MSI-X addresses */ + nlm_write_reg(lnkbase, PCIE_9XX_BRIDGE_MSIX_ADDR_BASE, + msixaddr >> 8); + nlm_write_reg(lnkbase, PCIE_9XX_BRIDGE_MSIX_ADDR_LIMIT, + (msixaddr + MSI_ADDR_SZ) >> 8); + } else { + /* MSI-X addresses */ + nlm_write_reg(lnkbase, PCIE_BRIDGE_MSIX_ADDR_BASE, + msixaddr >> 8); + nlm_write_reg(lnkbase, PCIE_BRIDGE_MSIX_ADDR_LIMIT, + (msixaddr + MSI_ADDR_SZ) >> 8); + } } /* @@ -369,6 +437,7 @@ static int xlp_setup_msix(uint64_t lnkbase, int node, int link, xirq += t; msixvec = nlm_irq_msixvec(xirq); + msg.address_hi = msixaddr >> 32; msg.address_lo = msixaddr & 0xffffffff; msg.data = 0xc00 | msixvec; @@ -409,7 +478,7 @@ void __init xlp_init_node_msi_irqs(int node, int link) { struct nlm_soc_info *nodep; struct xlp_msi_data *md; - int irq, i, irt, msixvec; + int irq, i, irt, msixvec, val; pr_info("[%d %d] Init node PCI IRT\n", node, link); nodep = nlm_get_node(node); @@ -430,19 +499,28 @@ void __init xlp_init_node_msi_irqs(int node, int link) irq_set_handler_data(i, md); } - for (i = 0; i < XLP_MSIXVEC_PER_LINK; i++) { - /* Initialize MSI-X irts to generate one interrupt per link */ - msixvec = link * XLP_MSIXVEC_PER_LINK + i; - irt = PIC_IRT_PCIE_MSIX_INDEX(msixvec); - nlm_pic_init_irt(nodep->picbase, irt, PIC_PCIE_MSIX_IRQ(link), - node * nlm_threads_per_node(), 1 /* enable */); + for (i = 0; i < XLP_MSIXVEC_PER_LINK ; i++) { + if (cpu_is_xlp9xx()) { + val = ((node * nlm_threads_per_node()) << 7 | + PIC_PCIE_MSIX_IRQ(link) << 1 | 0 << 0); + nlm_write_pcie_reg(md->lnkbase, PCIE_9XX_MSIX_VECX(i + + (link * XLP_MSIXVEC_PER_LINK)), val); + } else { + /* Initialize MSI-X irts to generate one interrupt + * per link + */ + msixvec = link * XLP_MSIXVEC_PER_LINK + i; + irt = PIC_IRT_PCIE_MSIX_INDEX(msixvec); + nlm_pic_init_irt(nodep->picbase, irt, + PIC_PCIE_MSIX_IRQ(link), + node * nlm_threads_per_node(), 1); + } /* Initialize MSI-X extended irq space for the link */ irq = nlm_irq_to_xirq(node, nlm_link_msixirq(link, i)); irq_set_chip_and_handler(irq, &xlp_msix_chip, handle_level_irq); irq_set_handler_data(irq, md); } - } void nlm_dispatch_msi(int node, int lirq) @@ -454,7 +532,11 @@ void nlm_dispatch_msi(int node, int lirq) link = lirq - PIC_PCIE_LINK_MSI_IRQ_BASE; irqbase = nlm_irq_to_xirq(node, nlm_link_msiirq(link, 0)); md = irq_get_handler_data(irqbase); - status = nlm_read_reg(md->lnkbase, PCIE_MSI_STATUS) & + if (cpu_is_xlp9xx()) + status = nlm_read_reg(md->lnkbase, PCIE_9XX_MSI_STATUS) & + md->msi_enabled_mask; + else + status = nlm_read_reg(md->lnkbase, PCIE_MSI_STATUS) & md->msi_enabled_mask; while (status) { i = __ffs(status); @@ -472,10 +554,14 @@ void nlm_dispatch_msix(int node, int lirq) link = lirq - PIC_PCIE_MSIX_IRQ_BASE; irqbase = nlm_irq_to_xirq(node, nlm_link_msixirq(link, 0)); md = irq_get_handler_data(irqbase); - status = nlm_read_reg(md->lnkbase, PCIE_MSIX_STATUS); + if (cpu_is_xlp9xx()) + status = nlm_read_reg(md->lnkbase, PCIE_9XX_MSIX_STATUSX(link)); + else + status = nlm_read_reg(md->lnkbase, PCIE_MSIX_STATUS); /* narrow it down to the MSI-x vectors for our link */ - status = (status >> (link * XLP_MSIXVEC_PER_LINK)) & + if (!cpu_is_xlp9xx()) + status = (status >> (link * XLP_MSIXVEC_PER_LINK)) & ((1 << XLP_MSIXVEC_PER_LINK) - 1); while (status) { diff --git a/arch/mips/pci/ops-pmcmsp.c b/arch/mips/pci/ops-pmcmsp.c index 3d27800edba2..50034f985be1 100644 --- a/arch/mips/pci/ops-pmcmsp.c +++ b/arch/mips/pci/ops-pmcmsp.c @@ -7,7 +7,7 @@ * Author: Jun Sun, jsun@mvista.com or jsun@junsun.net * * Much of the code is derived from the original DDB5074 port by - * Geert Uytterhoeven <geert@sonycom.com> + * Geert Uytterhoeven <geert@linux-m68k.org> * * This program is free software; you can redistribute it and/or modify it * under the terms of the GNU General Public License as published by the diff --git a/arch/mips/pci/ops-tx3927.c b/arch/mips/pci/ops-tx3927.c index 02d64f77e967..d35dc9c9ab9d 100644 --- a/arch/mips/pci/ops-tx3927.c +++ b/arch/mips/pci/ops-tx3927.c @@ -11,7 +11,7 @@ * Define the pci_ops for TX3927. * * Much of the code is derived from the original DDB5074 port by - * Geert Uytterhoeven <geert@sonycom.com> + * Geert Uytterhoeven <geert@linux-m68k.org> * * This program is free software; you can redistribute it and/or modify it * under the terms of the GNU General Public License as published by the diff --git a/arch/mips/pci/ops-tx4927.c b/arch/mips/pci/ops-tx4927.c index 3d5df514d024..0e046d82e4e3 100644 --- a/arch/mips/pci/ops-tx4927.c +++ b/arch/mips/pci/ops-tx4927.c @@ -202,17 +202,20 @@ char *tx4927_pcibios_setup(char *str) unsigned long val; if (!strncmp(str, "trdyto=", 7)) { - if (strict_strtoul(str + 7, 0, &val) == 0) + u8 val = 0; + if (kstrtou8(str + 7, 0, &val) == 0) tx4927_pci_opts.trdyto = val; return NULL; } if (!strncmp(str, "retryto=", 8)) { - if (strict_strtoul(str + 8, 0, &val) == 0) + u8 val = 0; + if (kstrtou8(str + 8, 0, &val) == 0) tx4927_pci_opts.retryto = val; return NULL; } if (!strncmp(str, "gbwc=", 5)) { - if (strict_strtoul(str + 5, 0, &val) == 0) + u16 val; + if (kstrtou16(str + 5, 0, &val) == 0) tx4927_pci_opts.gbwc = val; return NULL; } diff --git a/arch/mips/pci/pci-virtio-guest.c b/arch/mips/pci/pci-virtio-guest.c new file mode 100644 index 000000000000..40a078bc4617 --- /dev/null +++ b/arch/mips/pci/pci-virtio-guest.c @@ -0,0 +1,131 @@ +/* + * This file is subject to the terms and conditions of the GNU General Public + * License. See the file "COPYING" in the main directory of this archive + * for more details. + * + * Copyright (C) 2013 Cavium, Inc. + */ + +#include <linux/kernel.h> +#include <linux/init.h> +#include <linux/interrupt.h> +#include <linux/pci.h> + +#include <uapi/asm/bitfield.h> +#include <asm/byteorder.h> +#include <asm/io.h> + +#define PCI_CONFIG_ADDRESS 0xcf8 +#define PCI_CONFIG_DATA 0xcfc + +union pci_config_address { + struct { + __BITFIELD_FIELD(unsigned enable_bit : 1, /* 31 */ + __BITFIELD_FIELD(unsigned reserved : 7, /* 30 .. 24 */ + __BITFIELD_FIELD(unsigned bus_number : 8, /* 23 .. 16 */ + __BITFIELD_FIELD(unsigned devfn_number : 8, /* 15 .. 8 */ + __BITFIELD_FIELD(unsigned register_number : 8, /* 7 .. 0 */ + ))))); + }; + u32 w; +}; + +int pcibios_plat_dev_init(struct pci_dev *dev) +{ + return 0; +} + +int pcibios_map_irq(const struct pci_dev *dev, u8 slot, u8 pin) +{ + return ((pin + slot) % 4)+ MIPS_IRQ_PCIA; +} + +static void pci_virtio_guest_write_config_addr(struct pci_bus *bus, + unsigned int devfn, int reg) +{ + union pci_config_address pca = { .w = 0 }; + + pca.register_number = reg; + pca.devfn_number = devfn; + pca.bus_number = bus->number; + pca.enable_bit = 1; + + outl(pca.w, PCI_CONFIG_ADDRESS); +} + +static int pci_virtio_guest_write_config(struct pci_bus *bus, + unsigned int devfn, int reg, int size, u32 val) +{ + pci_virtio_guest_write_config_addr(bus, devfn, reg); + + switch (size) { + case 1: + outb(val, PCI_CONFIG_DATA + (reg & 3)); + break; + case 2: + outw(val, PCI_CONFIG_DATA + (reg & 2)); + break; + case 4: + outl(val, PCI_CONFIG_DATA); + break; + } + + return PCIBIOS_SUCCESSFUL; +} + +static int pci_virtio_guest_read_config(struct pci_bus *bus, unsigned int devfn, + int reg, int size, u32 *val) +{ + pci_virtio_guest_write_config_addr(bus, devfn, reg); + + switch (size) { + case 1: + *val = inb(PCI_CONFIG_DATA + (reg & 3)); + break; + case 2: + *val = inw(PCI_CONFIG_DATA + (reg & 2)); + break; + case 4: + *val = inl(PCI_CONFIG_DATA); + break; + } + return PCIBIOS_SUCCESSFUL; +} + +static struct pci_ops pci_virtio_guest_ops = { + .read = pci_virtio_guest_read_config, + .write = pci_virtio_guest_write_config, +}; + +static struct resource pci_virtio_guest_mem_resource = { + .name = "Virtio MEM", + .flags = IORESOURCE_MEM, + .start = 0x10000000, + .end = 0x1dffffff +}; + +static struct resource pci_virtio_guest_io_resource = { + .name = "Virtio IO", + .flags = IORESOURCE_IO, + .start = 0, + .end = 0xffff +}; + +static struct pci_controller pci_virtio_guest_controller = { + .pci_ops = &pci_virtio_guest_ops, + .mem_resource = &pci_virtio_guest_mem_resource, + .io_resource = &pci_virtio_guest_io_resource, +}; + +static int __init pci_virtio_guest_setup(void) +{ + pr_err("pci_virtio_guest_setup\n"); + + /* Virtio comes pre-assigned */ + pci_set_flags(PCI_PROBE_ONLY); + + pci_virtio_guest_controller.io_map_base = mips_io_port_base; + register_pci_controller(&pci_virtio_guest_controller); + return 0; +} +arch_initcall(pci_virtio_guest_setup); diff --git a/arch/mips/pmcs-msp71xx/Makefile b/arch/mips/pmcs-msp71xx/Makefile index 9201c8b3858d..d4f7220f2485 100644 --- a/arch/mips/pmcs-msp71xx/Makefile +++ b/arch/mips/pmcs-msp71xx/Makefile @@ -10,4 +10,3 @@ obj-$(CONFIG_PCI) += msp_pci.o obj-$(CONFIG_MSP_HAS_MAC) += msp_eth.o obj-$(CONFIG_MSP_HAS_USB) += msp_usb.o obj-$(CONFIG_MIPS_MT_SMP) += msp_smp.o -obj-$(CONFIG_MIPS_MT_SMTC) += msp_smtc.o diff --git a/arch/mips/pmcs-msp71xx/msp_eth.c b/arch/mips/pmcs-msp71xx/msp_eth.c index c584df393de2..15679b427f44 100644 --- a/arch/mips/pmcs-msp71xx/msp_eth.c +++ b/arch/mips/pmcs-msp71xx/msp_eth.c @@ -38,73 +38,6 @@ #define MSP_ETHERNET_GPIO1 15 #define MSP_ETHERNET_GPIO2 16 -#ifdef CONFIG_MSP_HAS_TSMAC -#define MSP_TSMAC_SIZE 0x10020 -#define MSP_TSMAC_ID "pmc_tsmac" - -static struct resource msp_tsmac0_resources[] = { - [0] = { - .start = MSP_MAC0_BASE, - .end = MSP_MAC0_BASE + MSP_TSMAC_SIZE - 1, - .flags = IORESOURCE_MEM, - }, - [1] = { - .start = MSP_INT_MAC0, - .end = MSP_INT_MAC0, - .flags = IORESOURCE_IRQ, - }, -}; - -static struct resource msp_tsmac1_resources[] = { - [0] = { - .start = MSP_MAC1_BASE, - .end = MSP_MAC1_BASE + MSP_TSMAC_SIZE - 1, - .flags = IORESOURCE_MEM, - }, - [1] = { - .start = MSP_INT_MAC1, - .end = MSP_INT_MAC1, - .flags = IORESOURCE_IRQ, - }, -}; -static struct resource msp_tsmac2_resources[] = { - [0] = { - .start = MSP_MAC2_BASE, - .end = MSP_MAC2_BASE + MSP_TSMAC_SIZE - 1, - .flags = IORESOURCE_MEM, - }, - [1] = { - .start = MSP_INT_SAR, - .end = MSP_INT_SAR, - .flags = IORESOURCE_IRQ, - }, -}; - - -static struct platform_device tsmac_device[] = { - [0] = { - .name = MSP_TSMAC_ID, - .id = 0, - .num_resources = ARRAY_SIZE(msp_tsmac0_resources), - .resource = msp_tsmac0_resources, - }, - [1] = { - .name = MSP_TSMAC_ID, - .id = 1, - .num_resources = ARRAY_SIZE(msp_tsmac1_resources), - .resource = msp_tsmac1_resources, - }, - [2] = { - .name = MSP_TSMAC_ID, - .id = 2, - .num_resources = ARRAY_SIZE(msp_tsmac2_resources), - .resource = msp_tsmac2_resources, - }, -}; -#define msp_eth_devs tsmac_device - -#else -/* If it is not TSMAC assume MSP_ETH (100Mbps) */ #define MSP_ETH_ID "pmc_mspeth" #define MSP_ETH_SIZE 0xE0 static struct resource msp_eth0_resources[] = { @@ -152,7 +85,6 @@ static struct platform_device mspeth_device[] = { }; #define msp_eth_devs mspeth_device -#endif int __init msp_eth_setup(void) { int i, ret = 0; @@ -161,14 +93,6 @@ int __init msp_eth_setup(void) msp_gpio_pin_mode(MSP_GPIO_OUTPUT, MSP_ETHERNET_GPIO0); msp_gpio_pin_hi(MSP_ETHERNET_GPIO0); -#ifdef CONFIG_MSP_HAS_TSMAC - /* 3 phys on boards with TSMAC */ - msp_gpio_pin_mode(MSP_GPIO_OUTPUT, MSP_ETHERNET_GPIO1); - msp_gpio_pin_hi(MSP_ETHERNET_GPIO1); - - msp_gpio_pin_mode(MSP_GPIO_OUTPUT, MSP_ETHERNET_GPIO2); - msp_gpio_pin_hi(MSP_ETHERNET_GPIO2); -#endif for (i = 0; i < ARRAY_SIZE(msp_eth_devs); i++) { ret = platform_device_register(&msp_eth_devs[i]); printk(KERN_INFO "device: %d, return value = %d\n", i, ret); diff --git a/arch/mips/pmcs-msp71xx/msp_irq.c b/arch/mips/pmcs-msp71xx/msp_irq.c index 9da5619c00a5..941744aabb51 100644 --- a/arch/mips/pmcs-msp71xx/msp_irq.c +++ b/arch/mips/pmcs-msp71xx/msp_irq.c @@ -32,7 +32,7 @@ extern void msp_vsmp_int_init(void); /* vectored interrupt implementation */ -/* SW0/1 interrupts are used for SMP/SMTC */ +/* SW0/1 interrupts are used for SMP */ static inline void mac0_int_dispatch(void) { do_IRQ(MSP_INT_MAC0); } static inline void mac1_int_dispatch(void) { do_IRQ(MSP_INT_MAC1); } static inline void mac2_int_dispatch(void) { do_IRQ(MSP_INT_SAR); } @@ -138,14 +138,6 @@ void __init arch_init_irq(void) set_vi_handler(MSP_INT_SEC, sec_int_dispatch); #ifdef CONFIG_MIPS_MT_SMP msp_vsmp_int_init(); -#elif defined CONFIG_MIPS_MT_SMTC - /*Set hwmask for all platform devices */ - irq_hwmask[MSP_INT_MAC0] = C_IRQ0; - irq_hwmask[MSP_INT_MAC1] = C_IRQ1; - irq_hwmask[MSP_INT_USB] = C_IRQ2; - irq_hwmask[MSP_INT_SAR] = C_IRQ3; - irq_hwmask[MSP_INT_SEC] = C_IRQ5; - #endif /* CONFIG_MIPS_MT_SMP */ #endif /* CONFIG_MIPS_MT */ /* setup the cascaded interrupts */ @@ -153,8 +145,10 @@ void __init arch_init_irq(void) setup_irq(MSP_INT_PER, &per_cascade_msp); #else - /* setup the 2nd-level SLP register based interrupt controller */ - /* VSMP /SMTC support support is not enabled for SLP */ + /* + * Setup the 2nd-level SLP register based interrupt controller. + * VSMP support support is not enabled for SLP. + */ msp_slp_irq_init(); /* setup the cascaded SLP/PER interrupts */ diff --git a/arch/mips/pmcs-msp71xx/msp_irq_cic.c b/arch/mips/pmcs-msp71xx/msp_irq_cic.c index e49b499f66db..b8df2f7b3328 100644 --- a/arch/mips/pmcs-msp71xx/msp_irq_cic.c +++ b/arch/mips/pmcs-msp71xx/msp_irq_cic.c @@ -120,10 +120,9 @@ static void msp_cic_irq_ack(struct irq_data *d) * hurt for the others */ *CIC_STS_REG = (1 << (d->irq - MSP_CIC_INTBASE)); - smtc_im_ack_irq(d->irq); } -/*Note: Limiting to VSMP . Not tested in SMTC */ +/* Note: Limiting to VSMP. */ #ifdef CONFIG_MIPS_MT_SMP static int msp_cic_irq_set_affinity(struct irq_data *d, @@ -183,10 +182,6 @@ void __init msp_cic_irq_init(void) for (i = MSP_CIC_INTBASE ; i < MSP_CIC_INTBASE + 32 ; i++) { irq_set_chip_and_handler(i, &msp_cic_irq_controller, handle_level_irq); -#ifdef CONFIG_MIPS_MT_SMTC - /* Mask of CIC interrupt */ - irq_hwmask[i] = C_IRQ4; -#endif } /* Initialize the PER interrupt sub-system */ diff --git a/arch/mips/pmcs-msp71xx/msp_irq_per.c b/arch/mips/pmcs-msp71xx/msp_irq_per.c index d1fd530479d4..a111836bcec2 100644 --- a/arch/mips/pmcs-msp71xx/msp_irq_per.c +++ b/arch/mips/pmcs-msp71xx/msp_irq_per.c @@ -113,9 +113,6 @@ void __init msp_per_irq_init(void) /* initialize all the IRQ descriptors */ for (i = MSP_PER_INTBASE; i < MSP_PER_INTBASE + 32; i++) { irq_set_chip(i, &msp_per_irq_controller); -#ifdef CONFIG_MIPS_MT_SMTC - irq_hwmask[i] = C_IRQ4; -#endif } } diff --git a/arch/mips/pmcs-msp71xx/msp_setup.c b/arch/mips/pmcs-msp71xx/msp_setup.c index 7e980767679c..4f925e06c414 100644 --- a/arch/mips/pmcs-msp71xx/msp_setup.c +++ b/arch/mips/pmcs-msp71xx/msp_setup.c @@ -27,7 +27,6 @@ #endif extern void msp_serial_setup(void); -extern void pmctwiled_setup(void); #if defined(CONFIG_PMC_MSP7120_EVAL) || \ defined(CONFIG_PMC_MSP7120_GW) || \ @@ -148,8 +147,6 @@ void __init plat_mem_setup(void) pm_power_off = msp_power_off; } -extern struct plat_smp_ops msp_smtc_smp_ops; - void __init prom_init(void) { unsigned long family; @@ -230,17 +227,5 @@ void __init prom_init(void) */ msp_serial_setup(); - if (register_vsmp_smp_ops()) { -#ifdef CONFIG_MIPS_MT_SMTC - register_smp_ops(&msp_smtc_smp_ops); -#endif - } - -#ifdef CONFIG_PMCTWILED - /* - * Setup LED states before the subsys_initcall loads other - * dependent drivers/modules. - */ - pmctwiled_setup(); -#endif + register_vsmp_smp_ops(); } diff --git a/arch/mips/pmcs-msp71xx/msp_smtc.c b/arch/mips/pmcs-msp71xx/msp_smtc.c deleted file mode 100644 index 6b5607fce279..000000000000 --- a/arch/mips/pmcs-msp71xx/msp_smtc.c +++ /dev/null @@ -1,104 +0,0 @@ -/* - * MSP71xx Platform-specific hooks for SMP operation - */ -#include <linux/irq.h> -#include <linux/init.h> - -#include <asm/mipsmtregs.h> -#include <asm/mipsregs.h> -#include <asm/smtc.h> -#include <asm/smtc_ipi.h> - -/* VPE/SMP Prototype implements platform interfaces directly */ - -/* - * Cause the specified action to be performed on a targeted "CPU" - */ - -static void msp_smtc_send_ipi_single(int cpu, unsigned int action) -{ - /* "CPU" may be TC of same VPE, VPE of same CPU, or different CPU */ - smtc_send_ipi(cpu, LINUX_SMP_IPI, action); -} - -static void msp_smtc_send_ipi_mask(const struct cpumask *mask, - unsigned int action) -{ - unsigned int i; - - for_each_cpu(i, mask) - msp_smtc_send_ipi_single(i, action); -} - -/* - * Post-config but pre-boot cleanup entry point - */ -static void msp_smtc_init_secondary(void) -{ - int myvpe; - - /* Don't enable Malta I/O interrupts (IP2) for secondary VPEs */ - myvpe = read_c0_tcbind() & TCBIND_CURVPE; - if (myvpe > 0) - change_c0_status(ST0_IM, STATUSF_IP0 | STATUSF_IP1 | - STATUSF_IP6 | STATUSF_IP7); - smtc_init_secondary(); -} - -/* - * Platform "CPU" startup hook - */ -static void msp_smtc_boot_secondary(int cpu, struct task_struct *idle) -{ - smtc_boot_secondary(cpu, idle); -} - -/* - * SMP initialization finalization entry point - */ -static void msp_smtc_smp_finish(void) -{ - smtc_smp_finish(); -} - -/* - * Hook for after all CPUs are online - */ - -static void msp_smtc_cpus_done(void) -{ -} - -/* - * Platform SMP pre-initialization - * - * As noted above, we can assume a single CPU for now - * but it may be multithreaded. - */ - -static void __init msp_smtc_smp_setup(void) -{ - /* - * we won't get the definitive value until - * we've run smtc_prepare_cpus later, but - */ - - if (read_c0_config3() & (1 << 2)) - smp_num_siblings = smtc_build_cpu_map(0); -} - -static void __init msp_smtc_prepare_cpus(unsigned int max_cpus) -{ - smtc_prepare_cpus(max_cpus); -} - -struct plat_smp_ops msp_smtc_smp_ops = { - .send_ipi_single = msp_smtc_send_ipi_single, - .send_ipi_mask = msp_smtc_send_ipi_mask, - .init_secondary = msp_smtc_init_secondary, - .smp_finish = msp_smtc_smp_finish, - .cpus_done = msp_smtc_cpus_done, - .boot_secondary = msp_smtc_boot_secondary, - .smp_setup = msp_smtc_smp_setup, - .prepare_cpus = msp_smtc_prepare_cpus, -}; diff --git a/arch/mips/pmcs-msp71xx/msp_usb.c b/arch/mips/pmcs-msp71xx/msp_usb.c index 4dab915696e7..c87c5f810cd1 100644 --- a/arch/mips/pmcs-msp71xx/msp_usb.c +++ b/arch/mips/pmcs-msp71xx/msp_usb.c @@ -75,47 +75,6 @@ static struct mspusb_device msp_usbhost0_device = { .resource = msp_usbhost0_resources, }, }; - -/* MSP7140/MSP82XX has two USB2 hosts. */ -#ifdef CONFIG_MSP_HAS_DUAL_USB -static u64 msp_usbhost1_dma_mask = 0xffffffffUL; - -static struct resource msp_usbhost1_resources[] = { - [0] = { /* EHCI-HS operational and capabilities registers */ - .start = MSP_USB1_HS_START, - .end = MSP_USB1_HS_END, - .flags = IORESOURCE_MEM, - }, - [1] = { - .start = MSP_INT_USB, - .end = MSP_INT_USB, - .flags = IORESOURCE_IRQ, - }, - [2] = { /* MSBus-to-AMBA bridge register space */ - .start = MSP_USB1_MAB_START, - .end = MSP_USB1_MAB_END, - .flags = IORESOURCE_MEM, - }, - [3] = { /* Identification and general hardware parameters */ - .start = MSP_USB1_ID_START, - .end = MSP_USB1_ID_END, - .flags = IORESOURCE_MEM, - }, -}; - -static struct mspusb_device msp_usbhost1_device = { - .dev = { - .name = "pmcmsp-ehci", - .id = 1, - .dev = { - .dma_mask = &msp_usbhost1_dma_mask, - .coherent_dma_mask = 0xffffffffUL, - }, - .num_resources = ARRAY_SIZE(msp_usbhost1_resources), - .resource = msp_usbhost1_resources, - }, -}; -#endif /* CONFIG_MSP_HAS_DUAL_USB */ #endif /* CONFIG_USB_EHCI_HCD */ #if defined(CONFIG_USB_GADGET) @@ -157,46 +116,6 @@ static struct mspusb_device msp_usbdev0_device = { .resource = msp_usbdev0_resources, }, }; - -#ifdef CONFIG_MSP_HAS_DUAL_USB -static struct resource msp_usbdev1_resources[] = { - [0] = { /* EHCI-HS operational and capabilities registers */ - .start = MSP_USB1_HS_START, - .end = MSP_USB1_HS_END, - .flags = IORESOURCE_MEM, - }, - [1] = { - .start = MSP_INT_USB, - .end = MSP_INT_USB, - .flags = IORESOURCE_IRQ, - }, - [2] = { /* MSBus-to-AMBA bridge register space */ - .start = MSP_USB1_MAB_START, - .end = MSP_USB1_MAB_END, - .flags = IORESOURCE_MEM, - }, - [3] = { /* Identification and general hardware parameters */ - .start = MSP_USB1_ID_START, - .end = MSP_USB1_ID_END, - .flags = IORESOURCE_MEM, - }, -}; - -/* This may need to be converted to a mspusb_device, too. */ -static struct mspusb_device msp_usbdev1_device = { - .dev = { - .name = "msp71xx_udc", - .id = 0, - .dev = { - .dma_mask = &msp_usbdev_dma_mask, - .coherent_dma_mask = 0xffffffffUL, - }, - .num_resources = ARRAY_SIZE(msp_usbdev1_resources), - .resource = msp_usbdev1_resources, - }, -}; - -#endif /* CONFIG_MSP_HAS_DUAL_USB */ #endif /* CONFIG_USB_GADGET */ static int __init msp_usb_setup(void) @@ -231,10 +150,6 @@ static int __init msp_usb_setup(void) #if defined(CONFIG_USB_EHCI_HCD) msp_devs[0] = &msp_usbhost0_device.dev; ppfinit("platform add USB HOST done %s.\n", msp_devs[0]->name); -#ifdef CONFIG_MSP_HAS_DUAL_USB - msp_devs[1] = &msp_usbhost1_device.dev; - ppfinit("platform add USB HOST done %s.\n", msp_devs[1]->name); -#endif #else ppfinit("%s: echi_hcd not supported\n", __FILE__); #endif /* CONFIG_USB_EHCI_HCD */ @@ -244,11 +159,6 @@ static int __init msp_usb_setup(void) msp_devs[0] = &msp_usbdev0_device.dev; ppfinit("platform add USB DEVICE done %s.\n" , msp_devs[0]->name); -#ifdef CONFIG_MSP_HAS_DUAL_USB - msp_devs[1] = &msp_usbdev1_device.dev; - ppfinit("platform add USB DEVICE done %s.\n" - , msp_devs[1]->name); -#endif #else ppfinit("%s: usb_gadget not supported\n", __FILE__); #endif /* CONFIG_USB_GADGET */ diff --git a/arch/mips/pnx833x/common/platform.c b/arch/mips/pnx833x/common/platform.c index 2b7e837dc2e2..b4b774bc3178 100644 --- a/arch/mips/pnx833x/common/platform.c +++ b/arch/mips/pnx833x/common/platform.c @@ -33,11 +33,6 @@ #include <linux/mtd/nand.h> #include <linux/mtd/partitions.h> -#ifdef CONFIG_I2C_PNX0105 -/* Until i2c driver available in kernel.*/ -#include <linux/i2c-pnx0105.h> -#endif - #include <irq.h> #include <irq-mapping.h> #include <pnx833x.h> @@ -134,70 +129,6 @@ static struct platform_device pnx833x_usb_ehci_device = { .resource = pnx833x_usb_ehci_resources, }; -#ifdef CONFIG_I2C_PNX0105 -static struct resource pnx833x_i2c0_resources[] = { - { - .start = PNX833X_I2C0_PORTS_START, - .end = PNX833X_I2C0_PORTS_END, - .flags = IORESOURCE_MEM, - }, - { - .start = PNX833X_PIC_I2C0_INT, - .end = PNX833X_PIC_I2C0_INT, - .flags = IORESOURCE_IRQ, - }, -}; - -static struct resource pnx833x_i2c1_resources[] = { - { - .start = PNX833X_I2C1_PORTS_START, - .end = PNX833X_I2C1_PORTS_END, - .flags = IORESOURCE_MEM, - }, - { - .start = PNX833X_PIC_I2C1_INT, - .end = PNX833X_PIC_I2C1_INT, - .flags = IORESOURCE_IRQ, - }, -}; - -static struct i2c_pnx0105_dev pnx833x_i2c_dev[] = { - { - .base = PNX833X_I2C0_PORTS_START, - .irq = -1, /* should be PNX833X_PIC_I2C0_INT but polling is faster */ - .clock = 6, /* 0 == 400 kHz, 4 == 100 kHz(Maximum HDMI), 6 = 50kHz(Preferred HDCP) */ - .bus_addr = 0, /* no slave support */ - }, - { - .base = PNX833X_I2C1_PORTS_START, - .irq = -1, /* on high freq, polling is faster */ - /*.irq = PNX833X_PIC_I2C1_INT,*/ - .clock = 4, /* 0 == 400 kHz, 4 == 100 kHz. 100 kHz seems a safe default for now */ - .bus_addr = 0, /* no slave support */ - }, -}; - -static struct platform_device pnx833x_i2c0_device = { - .name = "i2c-pnx0105", - .id = 0, - .dev = { - .platform_data = &pnx833x_i2c_dev[0], - }, - .num_resources = ARRAY_SIZE(pnx833x_i2c0_resources), - .resource = pnx833x_i2c0_resources, -}; - -static struct platform_device pnx833x_i2c1_device = { - .name = "i2c-pnx0105", - .id = 1, - .dev = { - .platform_data = &pnx833x_i2c_dev[1], - }, - .num_resources = ARRAY_SIZE(pnx833x_i2c1_resources), - .resource = pnx833x_i2c1_resources, -}; -#endif - static u64 ethernet_dmamask = DMA_BIT_MASK(32); static struct resource pnx833x_ethernet_resources[] = { @@ -294,10 +225,6 @@ static struct platform_device pnx833x_flash_nand = { static struct platform_device *pnx833x_platform_devices[] __initdata = { &pnx833x_uart_device, &pnx833x_usb_ehci_device, -#ifdef CONFIG_I2C_PNX0105 - &pnx833x_i2c0_device, - &pnx833x_i2c1_device, -#endif &pnx833x_ethernet_device, &pnx833x_sata_device, &pnx833x_flash_nand, diff --git a/arch/mips/sgi-ip22/ip22-gio.c b/arch/mips/sgi-ip22/ip22-gio.c index ab0e379dc7e0..8e52446286ca 100644 --- a/arch/mips/sgi-ip22/ip22-gio.c +++ b/arch/mips/sgi-ip22/ip22-gio.c @@ -19,6 +19,9 @@ static struct { } gio_name_table[] = { { .name = "SGI Impact", .id = 0x10 }, { .name = "Phobos G160", .id = 0x35 }, + { .name = "Phobos G130", .id = 0x36 }, + { .name = "Phobos G100", .id = 0x37 }, + { .name = "Set Engineering GFE", .id = 0x38 }, /* fake IDs */ { .name = "SGI Newport", .id = 0x7e }, { .name = "SGI GR2/GR3", .id = 0x7f }, @@ -293,7 +296,16 @@ static int ip22_gio_id(unsigned long addr, u32 *res) * data matches */ ptr8 = (void *)CKSEG1ADDR(addr + 3); - get_dbe(tmp8, ptr8); + if (get_dbe(tmp8, ptr8)) { + /* + * 32bit access worked, but 8bit doesn't + * so we don't see phantom reads on + * a pipelined bus, but a real card which + * doesn't support 8 bit reads + */ + *res = tmp32; + return 1; + } ptr16 = (void *)CKSEG1ADDR(addr + 2); get_dbe(tmp16, ptr16); if (tmp8 == (tmp16 & 0xff) && @@ -324,7 +336,7 @@ static int ip22_is_gr2(unsigned long addr) } -static void ip22_check_gio(int slotno, unsigned long addr) +static void ip22_check_gio(int slotno, unsigned long addr, int irq) { const char *name = "Unknown"; struct gio_device *gio_dev; @@ -338,9 +350,9 @@ static void ip22_check_gio(int slotno, unsigned long addr) else { if (!ip22_gio_id(addr, &tmp)) { /* - * no GIO signature at start address of slot, but - * Newport doesn't have one, so let's check usea - * status register + * no GIO signature at start address of slot + * since Newport doesn't have one, we check if + * user status register is readable */ if (ip22_gio_id(addr + NEWPORT_USTATUS_OFFS, &tmp)) tmp = 0x7e; @@ -369,6 +381,7 @@ static void ip22_check_gio(int slotno, unsigned long addr) gio_dev->resource.start = addr; gio_dev->resource.end = addr + 0x3fffff; gio_dev->resource.flags = IORESOURCE_MEM; + gio_dev->irq = irq; dev_set_name(&gio_dev->dev, "%d", slotno); gio_device_register(gio_dev); } else @@ -408,16 +421,17 @@ int __init ip22_gio_init(void) request_resource(&iomem_resource, &gio_bus_resource); printk(KERN_INFO "GIO: Probing bus...\n"); - if (ip22_is_fullhouse() || - !get_dbe(pbdma, (unsigned int *)&hpc3c1->pbdma[1])) { - /* Indigo2 and ChallengeS */ - ip22_check_gio(0, GIO_SLOT_GFX_BASE); - ip22_check_gio(1, GIO_SLOT_EXP0_BASE); + if (ip22_is_fullhouse()) { + /* Indigo2 */ + ip22_check_gio(0, GIO_SLOT_GFX_BASE, SGI_GIO_1_IRQ); + ip22_check_gio(1, GIO_SLOT_EXP0_BASE, SGI_GIO_1_IRQ); } else { - /* Indy */ - ip22_check_gio(0, GIO_SLOT_GFX_BASE); - ip22_check_gio(1, GIO_SLOT_EXP0_BASE); - ip22_check_gio(2, GIO_SLOT_EXP1_BASE); + /* Indy/Challenge S */ + if (get_dbe(pbdma, (unsigned int *)&hpc3c1->pbdma[1])) + ip22_check_gio(0, GIO_SLOT_GFX_BASE, + SGI_GIO_0_IRQ); + ip22_check_gio(1, GIO_SLOT_EXP0_BASE, SGI_GIOEXP0_IRQ); + ip22_check_gio(2, GIO_SLOT_EXP1_BASE, SGI_GIOEXP1_IRQ); } } else device_unregister(&gio_bus); diff --git a/arch/mips/sgi-ip22/ip22-int.c b/arch/mips/sgi-ip22/ip22-int.c index 58b40ae59335..c66889fc4913 100644 --- a/arch/mips/sgi-ip22/ip22-int.c +++ b/arch/mips/sgi-ip22/ip22-int.c @@ -119,9 +119,14 @@ static void indy_local0_irqdispatch(void) } else irq = lc0msk_to_irqnr[mask]; - /* if irq == 0, then the interrupt has already been cleared */ + /* + * workaround for INT2 bug; if irq == 0, INT2 has seen a fifo full + * irq, but failed to latch it into status register + */ if (irq) do_IRQ(irq); + else + do_IRQ(SGINT_LOCAL0 + 0); } static void indy_local1_irqdispatch(void) diff --git a/arch/mips/sgi-ip27/ip27-smp.c b/arch/mips/sgi-ip27/ip27-smp.c index f4ea8aa79ba2..f9ae6a8fa7c7 100644 --- a/arch/mips/sgi-ip27/ip27-smp.c +++ b/arch/mips/sgi-ip27/ip27-smp.c @@ -186,10 +186,6 @@ static void ip27_smp_finish(void) local_irq_enable(); } -static void __init ip27_cpus_done(void) -{ -} - /* * Launch a slave into smp_bootstrap(). It doesn't take an argument, and we * set sp to the kernel stack of the newly created idle process, gp to the proc @@ -236,7 +232,6 @@ struct plat_smp_ops ip27_smp_ops = { .send_ipi_mask = ip27_send_ipi_mask, .init_secondary = ip27_init_secondary, .smp_finish = ip27_smp_finish, - .cpus_done = ip27_cpus_done, .boot_secondary = ip27_boot_secondary, .smp_setup = ip27_smp_setup, .prepare_cpus = ip27_prepare_cpus, diff --git a/arch/mips/sibyte/bcm1480/irq.c b/arch/mips/sibyte/bcm1480/irq.c index 59cfe2659771..373fbbc8425c 100644 --- a/arch/mips/sibyte/bcm1480/irq.c +++ b/arch/mips/sibyte/bcm1480/irq.c @@ -347,19 +347,8 @@ asmlinkage void plat_irq_dispatch(void) unsigned int cpu = smp_processor_id(); unsigned int pending; -#ifdef CONFIG_SIBYTE_BCM1480_PROF - /* Set compare to count to silence count/compare timer interrupts */ - write_c0_compare(read_c0_count()); -#endif - pending = read_c0_cause() & read_c0_status(); -#ifdef CONFIG_SIBYTE_BCM1480_PROF - if (pending & CAUSEF_IP7) /* Cpu performance counter interrupt */ - sbprof_cpu_intr(); - else -#endif - if (pending & CAUSEF_IP4) do_IRQ(K_BCM1480_INT_TIMER_0 + cpu); #ifdef CONFIG_SMP diff --git a/arch/mips/sibyte/bcm1480/smp.c b/arch/mips/sibyte/bcm1480/smp.c index 70d9182b26f1..af7d44edd9a8 100644 --- a/arch/mips/sibyte/bcm1480/smp.c +++ b/arch/mips/sibyte/bcm1480/smp.c @@ -115,13 +115,6 @@ static void bcm1480_smp_finish(void) } /* - * Final cleanup after all secondaries booted - */ -static void bcm1480_cpus_done(void) -{ -} - -/* * Setup the PC, SP, and GP of a secondary processor and start it * running! */ @@ -170,7 +163,6 @@ struct plat_smp_ops bcm1480_smp_ops = { .send_ipi_mask = bcm1480_send_ipi_mask, .init_secondary = bcm1480_init_secondary, .smp_finish = bcm1480_smp_finish, - .cpus_done = bcm1480_cpus_done, .boot_secondary = bcm1480_boot_secondary, .smp_setup = bcm1480_smp_setup, .prepare_cpus = bcm1480_prepare_cpus, diff --git a/arch/mips/sibyte/sb1250/smp.c b/arch/mips/sibyte/sb1250/smp.c index db976117dd4d..c0c4b3f88a08 100644 --- a/arch/mips/sibyte/sb1250/smp.c +++ b/arch/mips/sibyte/sb1250/smp.c @@ -103,13 +103,6 @@ static void sb1250_smp_finish(void) } /* - * Final cleanup after all secondaries booted - */ -static void sb1250_cpus_done(void) -{ -} - -/* * Setup the PC, SP, and GP of a secondary processor and start it * running! */ @@ -158,7 +151,6 @@ struct plat_smp_ops sb_smp_ops = { .send_ipi_mask = sb1250_send_ipi_mask, .init_secondary = sb1250_init_secondary, .smp_finish = sb1250_smp_finish, - .cpus_done = sb1250_cpus_done, .boot_secondary = sb1250_boot_secondary, .smp_setup = sb1250_smp_setup, .prepare_cpus = sb1250_prepare_cpus, diff --git a/arch/mips/txx9/generic/setup.c b/arch/mips/txx9/generic/setup.c index 2b0b83c171e0..dd2cf25b5ae5 100644 --- a/arch/mips/txx9/generic/setup.c +++ b/arch/mips/txx9/generic/setup.c @@ -309,8 +309,8 @@ static void __init preprocess_cmdline(void) txx9_board_vec = find_board_byname(str + 6); continue; } else if (strncmp(str, "masterclk=", 10) == 0) { - unsigned long val; - if (strict_strtoul(str + 10, 10, &val) == 0) + unsigned int val; + if (kstrtouint(str + 10, 10, &val) == 0) txx9_master_clock = val; continue; } else if (strcmp(str, "icdisable") == 0) { diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig index e0998997943b..bd6dd6ed3a9f 100644 --- a/arch/powerpc/Kconfig +++ b/arch/powerpc/Kconfig @@ -453,6 +453,14 @@ config NODES_SHIFT default "4" depends on NEED_MULTIPLE_NODES +config USE_PERCPU_NUMA_NODE_ID + def_bool y + depends on NUMA + +config HAVE_MEMORYLESS_NODES + def_bool y + depends on NUMA + config ARCH_SELECT_MEMORY_MODEL def_bool y depends on PPC64 diff --git a/arch/powerpc/Makefile b/arch/powerpc/Makefile index ce4c68a4a823..5687e299d0a5 100644 --- a/arch/powerpc/Makefile +++ b/arch/powerpc/Makefile @@ -113,8 +113,13 @@ else endif endif -CFLAGS-$(CONFIG_PPC64) := -mtraceback=no -mcall-aixdesc -CFLAGS-$(CONFIG_PPC64) += $(call cc-option,-mabi=elfv1) +CFLAGS-$(CONFIG_PPC64) := -mtraceback=no +ifeq ($(CONFIG_CPU_LITTLE_ENDIAN),y) +CFLAGS-$(CONFIG_PPC64) += $(call cc-option,-mabi=elfv2,-mcall-aixdesc) +AFLAGS-$(CONFIG_PPC64) += $(call cc-option,-mabi=elfv2) +else +CFLAGS-$(CONFIG_PPC64) += -mcall-aixdesc +endif CFLAGS-$(CONFIG_PPC64) += $(call cc-option,-mcmodel=medium,-mminimal-toc) CFLAGS-$(CONFIG_PPC64) += $(call cc-option,-mno-pointers-to-nested-functions) CFLAGS-$(CONFIG_PPC32) := -ffixed-r2 $(MULTIPLEWORD) @@ -153,7 +158,7 @@ CFLAGS-$(CONFIG_TUNE_CELL) += $(call cc-option,-mtune=cell) asinstr := $(call as-instr,lis 9$(comma)foo@high,-DHAVE_AS_ATHIGH=1) KBUILD_CPPFLAGS += -Iarch/$(ARCH) $(asinstr) -KBUILD_AFLAGS += -Iarch/$(ARCH) +KBUILD_AFLAGS += -Iarch/$(ARCH) $(AFLAGS-y) KBUILD_CFLAGS += -msoft-float -pipe -Iarch/$(ARCH) $(CFLAGS-y) CPP = $(CC) -E $(KBUILD_CFLAGS) @@ -161,6 +166,11 @@ CHECKFLAGS += -m$(CONFIG_WORD_SIZE) -D__powerpc__ -D__powerpc$(CONFIG_WORD_SIZE) KBUILD_LDFLAGS_MODULE += arch/powerpc/lib/crtsavres.o +ifeq ($(CONFIG_476FPE_ERR46),y) + KBUILD_LDFLAGS_MODULE += --ppc476-workaround \ + -T $(srctree)/arch/powerpc/platforms/44x/ppc476_modules.lds +endif + # No AltiVec or VSX instructions when building kernel KBUILD_CFLAGS += $(call cc-option,-mno-altivec) KBUILD_CFLAGS += $(call cc-option,-mno-vsx) diff --git a/arch/powerpc/boot/Makefile b/arch/powerpc/boot/Makefile index a1f8c7f1ec60..426dce7ae7c4 100644 --- a/arch/powerpc/boot/Makefile +++ b/arch/powerpc/boot/Makefile @@ -22,8 +22,14 @@ all: $(obj)/zImage BOOTCFLAGS := -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs \ -fno-strict-aliasing -Os -msoft-float -pipe \ -fomit-frame-pointer -fno-builtin -fPIC -nostdinc \ - -isystem $(shell $(CROSS32CC) -print-file-name=include) \ - -mbig-endian + -isystem $(shell $(CROSS32CC) -print-file-name=include) +ifdef CONFIG_PPC64_BOOT_WRAPPER +BOOTCFLAGS += -m64 +endif +ifdef CONFIG_CPU_BIG_ENDIAN +BOOTCFLAGS += -mbig-endian +endif + BOOTAFLAGS := -D__ASSEMBLY__ $(BOOTCFLAGS) -traditional -nostdinc ifdef CONFIG_DEBUG_INFO @@ -47,6 +53,7 @@ $(obj)/cuboot-acadia.o: BOOTCFLAGS += -mcpu=405 $(obj)/treeboot-walnut.o: BOOTCFLAGS += -mcpu=405 $(obj)/treeboot-iss4xx.o: BOOTCFLAGS += -mcpu=405 $(obj)/treeboot-currituck.o: BOOTCFLAGS += -mcpu=405 +$(obj)/treeboot-akebono.o: BOOTCFLAGS += -mcpu=405 $(obj)/virtex405-head.o: BOOTAFLAGS += -mcpu=405 @@ -86,6 +93,7 @@ src-plat-$(CONFIG_44x) += treeboot-ebony.c cuboot-ebony.c treeboot-bamboo.c \ cuboot-taishan.c cuboot-katmai.c \ cuboot-warp.c cuboot-yosemite.c \ treeboot-iss4xx.c treeboot-currituck.c \ + treeboot-akebono.c \ simpleboot.c fixed-head.S virtex.c src-plat-$(CONFIG_8xx) += cuboot-8xx.c fixed-head.S ep88xc.c redboot-8xx.c src-plat-$(CONFIG_PPC_MPC52xx) += cuboot-52xx.c @@ -99,6 +107,11 @@ src-plat-$(CONFIG_EMBEDDED6xx) += cuboot-pq2.c cuboot-mpc7448hpc2.c \ src-plat-$(CONFIG_AMIGAONE) += cuboot-amigaone.c src-plat-$(CONFIG_PPC_PS3) += ps3-head.S ps3-hvcall.S ps3.c src-plat-$(CONFIG_EPAPR_BOOT) += epapr.c epapr-wrapper.c +src-plat-$(CONFIG_PPC_PSERIES) += pseries-head.S +src-plat-$(CONFIG_PPC_POWERNV) += pseries-head.S +src-plat-$(CONFIG_PPC_IBM_CELL_BLADE) += pseries-head.S +src-plat-$(CONFIG_PPC_CELLEB) += pseries-head.S +src-plat-$(CONFIG_PPC_CELL_QPACE) += pseries-head.S src-wlib := $(sort $(src-wlib-y)) src-plat := $(sort $(src-plat-y)) @@ -137,7 +150,11 @@ $(addprefix $(obj)/,$(libfdt) $(libfdtheader)): $(obj)/%: $(srctree)/scripts/dtc $(obj)/empty.c: @touch $@ -$(obj)/zImage.lds $(obj)/zImage.coff.lds $(obj)/zImage.ps3.lds: $(obj)/%: $(srctree)/$(src)/%.S +$(obj)/zImage.lds: $(obj)/%: $(srctree)/$(src)/%.S + $(CROSS32CC) $(cpp_flags) -E -Wp,-MD,$(depfile) -P -Upowerpc \ + -D__ASSEMBLY__ -DLINKER_SCRIPT -o $@ $< + +$(obj)/zImage.coff.lds $(obj)/zImage.ps3.lds : $(obj)/%: $(srctree)/$(src)/%.S @cp $< $@ clean-files := $(zlib) $(zlibheader) $(zliblinuxheader) \ @@ -235,6 +252,7 @@ image-$(CONFIG_YOSEMITE) += cuImage.yosemite image-$(CONFIG_ISS4xx) += treeImage.iss4xx \ treeImage.iss4xx-mpic image-$(CONFIG_CURRITUCK) += treeImage.currituck +image-$(CONFIG_AKEBONO) += treeImage.akebono # Board ports in arch/powerpc/platform/8xx/Kconfig image-$(CONFIG_MPC86XADS) += cuImage.mpc866ads diff --git a/arch/powerpc/boot/addnote.c b/arch/powerpc/boot/addnote.c index 349b5530d2c4..9d9f6f334d3c 100644 --- a/arch/powerpc/boot/addnote.c +++ b/arch/powerpc/boot/addnote.c @@ -6,6 +6,8 @@ * * Copyright 2000 Paul Mackerras. * + * Adapted for 64 bit little endian images by Andrew Tauferner. + * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License * as published by the Free Software Foundation; either version @@ -55,36 +57,61 @@ unsigned int rpanote[N_RPA_DESCR] = { #define ROUNDUP(len) (((len) + 3) & ~3) -unsigned char buf[512]; +unsigned char buf[1024]; +#define ELFDATA2LSB 1 +#define ELFDATA2MSB 2 +static int e_data = ELFDATA2MSB; +#define ELFCLASS32 1 +#define ELFCLASS64 2 +static int e_class = ELFCLASS32; #define GET_16BE(off) ((buf[off] << 8) + (buf[(off)+1])) -#define GET_32BE(off) ((GET_16BE(off) << 16) + GET_16BE((off)+2)) - -#define PUT_16BE(off, v) (buf[off] = ((v) >> 8) & 0xff, \ - buf[(off) + 1] = (v) & 0xff) -#define PUT_32BE(off, v) (PUT_16BE((off), (v) >> 16), \ - PUT_16BE((off) + 2, (v))) +#define GET_32BE(off) ((GET_16BE(off) << 16U) + GET_16BE((off)+2U)) +#define GET_64BE(off) ((((unsigned long long)GET_32BE(off)) << 32ULL) + \ + ((unsigned long long)GET_32BE((off)+4ULL))) +#define PUT_16BE(off, v)(buf[off] = ((v) >> 8) & 0xff, \ + buf[(off) + 1] = (v) & 0xff) +#define PUT_32BE(off, v)(PUT_16BE((off), (v) >> 16L), PUT_16BE((off) + 2, (v))) +#define PUT_64BE(off, v)((PUT_32BE((off), (v) >> 32L), \ + PUT_32BE((off) + 4, (v)))) + +#define GET_16LE(off) ((buf[off]) + (buf[(off)+1] << 8)) +#define GET_32LE(off) (GET_16LE(off) + (GET_16LE((off)+2U) << 16U)) +#define GET_64LE(off) ((unsigned long long)GET_32LE(off) + \ + (((unsigned long long)GET_32LE((off)+4ULL)) << 32ULL)) +#define PUT_16LE(off, v) (buf[off] = (v) & 0xff, \ + buf[(off) + 1] = ((v) >> 8) & 0xff) +#define PUT_32LE(off, v) (PUT_16LE((off), (v)), PUT_16LE((off) + 2, (v) >> 16L)) +#define PUT_64LE(off, v) (PUT_32LE((off), (v)), PUT_32LE((off) + 4, (v) >> 32L)) + +#define GET_16(off) (e_data == ELFDATA2MSB ? GET_16BE(off) : GET_16LE(off)) +#define GET_32(off) (e_data == ELFDATA2MSB ? GET_32BE(off) : GET_32LE(off)) +#define GET_64(off) (e_data == ELFDATA2MSB ? GET_64BE(off) : GET_64LE(off)) +#define PUT_16(off, v) (e_data == ELFDATA2MSB ? PUT_16BE(off, v) : \ + PUT_16LE(off, v)) +#define PUT_32(off, v) (e_data == ELFDATA2MSB ? PUT_32BE(off, v) : \ + PUT_32LE(off, v)) +#define PUT_64(off, v) (e_data == ELFDATA2MSB ? PUT_64BE(off, v) : \ + PUT_64LE(off, v)) /* Structure of an ELF file */ #define E_IDENT 0 /* ELF header */ -#define E_PHOFF 28 -#define E_PHENTSIZE 42 -#define E_PHNUM 44 -#define E_HSIZE 52 /* size of ELF header */ +#define E_PHOFF (e_class == ELFCLASS32 ? 28 : 32) +#define E_PHENTSIZE (e_class == ELFCLASS32 ? 42 : 54) +#define E_PHNUM (e_class == ELFCLASS32 ? 44 : 56) +#define E_HSIZE (e_class == ELFCLASS32 ? 52 : 64) #define EI_MAGIC 0 /* offsets in E_IDENT area */ #define EI_CLASS 4 #define EI_DATA 5 #define PH_TYPE 0 /* ELF program header */ -#define PH_OFFSET 4 -#define PH_FILESZ 16 -#define PH_HSIZE 32 /* size of program header */ +#define PH_OFFSET (e_class == ELFCLASS32 ? 4 : 8) +#define PH_FILESZ (e_class == ELFCLASS32 ? 16 : 32) +#define PH_HSIZE (e_class == ELFCLASS32 ? 32 : 56) #define PT_NOTE 4 /* Program header type = note */ -#define ELFCLASS32 1 -#define ELFDATA2MSB 2 unsigned char elf_magic[4] = { 0x7f, 'E', 'L', 'F' }; @@ -92,8 +119,8 @@ int main(int ac, char **av) { int fd, n, i; - int ph, ps, np; - int nnote, nnote2, ns; + unsigned long ph, ps, np; + long nnote, nnote2, ns; if (ac != 2) { fprintf(stderr, "Usage: %s elf-file\n", av[0]); @@ -114,26 +141,27 @@ main(int ac, char **av) exit(1); } - if (n < E_HSIZE || memcmp(&buf[E_IDENT+EI_MAGIC], elf_magic, 4) != 0) + if (memcmp(&buf[E_IDENT+EI_MAGIC], elf_magic, 4) != 0) + goto notelf; + e_class = buf[E_IDENT+EI_CLASS]; + if (e_class != ELFCLASS32 && e_class != ELFCLASS64) + goto notelf; + e_data = buf[E_IDENT+EI_DATA]; + if (e_data != ELFDATA2MSB && e_data != ELFDATA2LSB) + goto notelf; + if (n < E_HSIZE) goto notelf; - if (buf[E_IDENT+EI_CLASS] != ELFCLASS32 - || buf[E_IDENT+EI_DATA] != ELFDATA2MSB) { - fprintf(stderr, "%s is not a big-endian 32-bit ELF image\n", - av[1]); - exit(1); - } - - ph = GET_32BE(E_PHOFF); - ps = GET_16BE(E_PHENTSIZE); - np = GET_16BE(E_PHNUM); + ph = (e_class == ELFCLASS32 ? GET_32(E_PHOFF) : GET_64(E_PHOFF)); + ps = GET_16(E_PHENTSIZE); + np = GET_16(E_PHNUM); if (ph < E_HSIZE || ps < PH_HSIZE || np < 1) goto notelf; if (ph + (np + 2) * ps + nnote + nnote2 > n) goto nospace; for (i = 0; i < np; ++i) { - if (GET_32BE(ph + PH_TYPE) == PT_NOTE) { + if (GET_32(ph + PH_TYPE) == PT_NOTE) { fprintf(stderr, "%s already has a note entry\n", av[1]); exit(0); @@ -148,15 +176,22 @@ main(int ac, char **av) /* fill in the program header entry */ ns = ph + 2 * ps; - PUT_32BE(ph + PH_TYPE, PT_NOTE); - PUT_32BE(ph + PH_OFFSET, ns); - PUT_32BE(ph + PH_FILESZ, nnote); + PUT_32(ph + PH_TYPE, PT_NOTE); + if (e_class == ELFCLASS32) + PUT_32(ph + PH_OFFSET, ns); + else + PUT_64(ph + PH_OFFSET, ns); + + if (e_class == ELFCLASS32) + PUT_32(ph + PH_FILESZ, nnote); + else + PUT_64(ph + PH_FILESZ, nnote); /* fill in the note area we point to */ /* XXX we should probably make this a proper section */ - PUT_32BE(ns, strlen(arch) + 1); - PUT_32BE(ns + 4, N_DESCR * 4); - PUT_32BE(ns + 8, 0x1275); + PUT_32(ns, strlen(arch) + 1); + PUT_32(ns + 4, N_DESCR * 4); + PUT_32(ns + 8, 0x1275); strcpy((char *) &buf[ns + 12], arch); ns += 12 + strlen(arch) + 1; for (i = 0; i < N_DESCR; ++i, ns += 4) @@ -164,21 +199,28 @@ main(int ac, char **av) /* fill in the second program header entry and the RPA note area */ ph += ps; - PUT_32BE(ph + PH_TYPE, PT_NOTE); - PUT_32BE(ph + PH_OFFSET, ns); - PUT_32BE(ph + PH_FILESZ, nnote2); + PUT_32(ph + PH_TYPE, PT_NOTE); + if (e_class == ELFCLASS32) + PUT_32(ph + PH_OFFSET, ns); + else + PUT_64(ph + PH_OFFSET, ns); + + if (e_class == ELFCLASS32) + PUT_32(ph + PH_FILESZ, nnote); + else + PUT_64(ph + PH_FILESZ, nnote2); /* fill in the note area we point to */ - PUT_32BE(ns, strlen(rpaname) + 1); - PUT_32BE(ns + 4, sizeof(rpanote)); - PUT_32BE(ns + 8, 0x12759999); + PUT_32(ns, strlen(rpaname) + 1); + PUT_32(ns + 4, sizeof(rpanote)); + PUT_32(ns + 8, 0x12759999); strcpy((char *) &buf[ns + 12], rpaname); ns += 12 + ROUNDUP(strlen(rpaname) + 1); for (i = 0; i < N_RPA_DESCR; ++i, ns += 4) PUT_32BE(ns, rpanote[i]); /* Update the number of program headers */ - PUT_16BE(E_PHNUM, np + 2); + PUT_16(E_PHNUM, np + 2); /* write back */ lseek(fd, (long) 0, SEEK_SET); diff --git a/arch/powerpc/boot/crt0.S b/arch/powerpc/boot/crt0.S index 0f7428a37efb..14de4f8778a7 100644 --- a/arch/powerpc/boot/crt0.S +++ b/arch/powerpc/boot/crt0.S @@ -1,17 +1,20 @@ /* * Copyright (C) Paul Mackerras 1997. * + * Adapted for 64 bit LE PowerPC by Andrew Tauferner + * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License * as published by the Free Software Foundation; either version * 2 of the License, or (at your option) any later version. * - * NOTE: this code runs in 32 bit mode, is position-independent, - * and is packaged as ELF32. */ #include "ppc_asm.h" +RELA = 7 +RELACOUNT = 0x6ffffff9 + .text /* A procedure descriptor used when booting this as a COFF file. * When making COFF, this comes first in the link and we're @@ -21,6 +24,20 @@ _zimage_start_opd: .long 0x500000, 0, 0, 0 +#ifdef __powerpc64__ +.balign 8 +p_start: .llong _start +p_etext: .llong _etext +p_bss_start: .llong __bss_start +p_end: .llong _end + +p_toc: .llong __toc_start + 0x8000 - p_base +p_dyn: .llong __dynamic_start - p_base +p_rela: .llong __rela_dyn_start - p_base +p_prom: .llong 0 + .weak _platform_stack_top +p_pstack: .llong _platform_stack_top +#else p_start: .long _start p_etext: .long _etext p_bss_start: .long __bss_start @@ -28,6 +45,7 @@ p_end: .long _end .weak _platform_stack_top p_pstack: .long _platform_stack_top +#endif .weak _zimage_start .globl _zimage_start @@ -38,6 +56,7 @@ _zimage_start_lib: and the address where we're running. */ bl .+4 p_base: mflr r10 /* r10 now points to runtime addr of p_base */ +#ifndef __powerpc64__ /* grab the link address of the dynamic section in r11 */ addis r11,r10,(_GLOBAL_OFFSET_TABLE_-p_base)@ha lwz r11,(_GLOBAL_OFFSET_TABLE_-p_base)@l(r11) @@ -51,8 +70,6 @@ p_base: mflr r10 /* r10 now points to runtime addr of p_base */ /* The dynamic section contains a series of tagged entries. * We need the RELA and RELACOUNT entries. */ -RELA = 7 -RELACOUNT = 0x6ffffff9 li r9,0 li r0,0 9: lwz r8,0(r12) /* get tag */ @@ -120,9 +137,164 @@ RELACOUNT = 0x6ffffff9 li r0,0 stwu r0,-16(r1) /* establish a stack frame */ 6: +#else /* __powerpc64__ */ + /* Save the prom pointer at p_prom. */ + std r5,(p_prom-p_base)(r10) + + /* Set r2 to the TOC. */ + ld r2,(p_toc-p_base)(r10) + add r2,r2,r10 + + /* Grab the link address of the dynamic section in r11. */ + ld r11,-32768(r2) + cmpwi r11,0 + beq 3f /* if not linked -pie then no dynamic section */ + + ld r11,(p_dyn-p_base)(r10) + add r11,r11,r10 + ld r9,(p_rela-p_base)(r10) + add r9,r9,r10 + li r7,0 + li r8,0 +9: ld r6,0(r11) /* get tag */ + cmpdi r6,0 + beq 12f /* end of list */ + cmpdi r6,RELA + bne 10f + ld r7,8(r11) /* get RELA pointer in r7 */ + b 11f +10: addis r6,r6,(-RELACOUNT)@ha + cmpdi r6,RELACOUNT@l + bne 11f + ld r8,8(r11) /* get RELACOUNT value in r8 */ +11: addi r11,r11,16 + b 9b +12: + cmpdi r7,0 /* check we have both RELA and RELACOUNT */ + cmpdi cr1,r8,0 + beq 3f + beq cr1,3f + + /* Calcuate the runtime offset. */ + subf r7,r7,r9 + + /* Run through the list of relocations and process the + * R_PPC64_RELATIVE ones. */ + mtctr r8 +13: ld r0,8(r9) /* ELF64_R_TYPE(reloc->r_info) */ + cmpdi r0,22 /* R_PPC64_RELATIVE */ + bne 3f + ld r6,0(r9) /* reloc->r_offset */ + ld r0,16(r9) /* reloc->r_addend */ + add r0,r0,r7 + stdx r0,r7,r6 + addi r9,r9,24 + bdnz 13b + + /* Do a cache flush for our text, in case the loader didn't */ +3: ld r9,p_start-p_base(r10) /* note: these are relocated now */ + ld r8,p_etext-p_base(r10) +4: dcbf r0,r9 + icbi r0,r9 + addi r9,r9,0x20 + cmpld cr0,r9,r8 + blt 4b + sync + isync + + /* Clear the BSS */ + ld r9,p_bss_start-p_base(r10) + ld r8,p_end-p_base(r10) + li r0,0 +5: std r0,0(r9) + addi r9,r9,8 + cmpld cr0,r9,r8 + blt 5b + + /* Possibly set up a custom stack */ + ld r8,p_pstack-p_base(r10) + cmpdi r8,0 + beq 6f + ld r1,0(r8) + li r0,0 + stdu r0,-16(r1) /* establish a stack frame */ +6: +#endif /* __powerpc64__ */ /* Call platform_init() */ bl platform_init /* Call start */ b start + +#ifdef __powerpc64__ + +#define PROM_FRAME_SIZE 512 +#define SAVE_GPR(n, base) std n,8*(n)(base) +#define REST_GPR(n, base) ld n,8*(n)(base) +#define SAVE_2GPRS(n, base) SAVE_GPR(n, base); SAVE_GPR(n+1, base) +#define SAVE_4GPRS(n, base) SAVE_2GPRS(n, base); SAVE_2GPRS(n+2, base) +#define SAVE_8GPRS(n, base) SAVE_4GPRS(n, base); SAVE_4GPRS(n+4, base) +#define SAVE_10GPRS(n, base) SAVE_8GPRS(n, base); SAVE_2GPRS(n+8, base) +#define REST_2GPRS(n, base) REST_GPR(n, base); REST_GPR(n+1, base) +#define REST_4GPRS(n, base) REST_2GPRS(n, base); REST_2GPRS(n+2, base) +#define REST_8GPRS(n, base) REST_4GPRS(n, base); REST_4GPRS(n+4, base) +#define REST_10GPRS(n, base) REST_8GPRS(n, base); REST_2GPRS(n+8, base) + +/* prom handles the jump into and return from firmware. The prom args pointer + is loaded in r3. */ +.globl prom +prom: + mflr r0 + std r0,16(r1) + stdu r1,-PROM_FRAME_SIZE(r1) /* Save SP and create stack space */ + + SAVE_GPR(2, r1) + SAVE_GPR(13, r1) + SAVE_8GPRS(14, r1) + SAVE_10GPRS(22, r1) + mfcr r10 + std r10,8*32(r1) + mfmsr r10 + std r10,8*33(r1) + + /* remove MSR_LE from msr but keep MSR_SF */ + mfmsr r10 + rldicr r10,r10,0,62 + mtsrr1 r10 + + /* Load FW address, set LR to label 1, and jump to FW */ + bl 0f +0: mflr r10 + addi r11,r10,(1f-0b) + mtlr r11 + + ld r10,(p_prom-0b)(r10) + mtsrr0 r10 + + rfid + +1: /* Return from OF */ + FIXUP_ENDIAN + + /* Restore registers and return. */ + rldicl r1,r1,0,32 + + /* Restore the MSR (back to 64 bits) */ + ld r10,8*(33)(r1) + mtmsr r10 + isync + + /* Restore other registers */ + REST_GPR(2, r1) + REST_GPR(13, r1) + REST_8GPRS(14, r1) + REST_10GPRS(22, r1) + ld r10,8*32(r1) + mtcr r10 + + addi r1,r1,PROM_FRAME_SIZE + ld r0,16(r1) + mtlr r0 + blr +#endif diff --git a/arch/powerpc/boot/dcr.h b/arch/powerpc/boot/dcr.h index cc73f7a95e26..bf8f4ede1928 100644 --- a/arch/powerpc/boot/dcr.h +++ b/arch/powerpc/boot/dcr.h @@ -15,6 +15,10 @@ asm volatile("mfdcrx %0,%1" : "=r"(rval) : "r"(rn)); \ rval; \ }) +#define mtdcrx(rn, val) \ + ({ \ + asm volatile("mtdcrx %0,%1" : : "r"(rn), "r" (val)); \ + }) /* 440GP/440GX SDRAM controller DCRs */ #define DCRN_SDRAM0_CFGADDR 0x010 diff --git a/arch/powerpc/boot/dts/akebono.dts b/arch/powerpc/boot/dts/akebono.dts new file mode 100644 index 000000000000..f92ecfed3d2f --- /dev/null +++ b/arch/powerpc/boot/dts/akebono.dts @@ -0,0 +1,415 @@ +/* + * Device Tree Source for IBM Embedded PPC 476 Platform + * + * Copyright © 2013 Tony Breeds IBM Corporation + * Copyright © 2013 Alistair Popple IBM Corporation + * + * This file is licensed under the terms of the GNU General Public + * License version 2. This program is licensed "as is" without + * any warranty of any kind, whether express or implied. + */ + +/dts-v1/; + +/memreserve/ 0x01f00000 0x00100000; // spin table + +/ { + #address-cells = <2>; + #size-cells = <2>; + model = "ibm,akebono"; + compatible = "ibm,akebono", "ibm,476gtr"; + dcr-parent = <&{/cpus/cpu@0}>; + + aliases { + serial0 = &UART0; + }; + + cpus { + #address-cells = <1>; + #size-cells = <0>; + + cpu@0 { + device_type = "cpu"; + model = "PowerPC,476"; + reg = <0>; + clock-frequency = <1600000000>; // 1.6 GHz + timebase-frequency = <100000000>; // 100Mhz + i-cache-line-size = <32>; + d-cache-line-size = <32>; + i-cache-size = <32768>; + d-cache-size = <32768>; + dcr-controller; + dcr-access-method = "native"; + status = "ok"; + }; + cpu@1 { + device_type = "cpu"; + model = "PowerPC,476"; + reg = <1>; + clock-frequency = <1600000000>; // 1.6 GHz + timebase-frequency = <100000000>; // 100Mhz + i-cache-line-size = <32>; + d-cache-line-size = <32>; + i-cache-size = <32768>; + d-cache-size = <32768>; + dcr-controller; + dcr-access-method = "native"; + status = "disabled"; + enable-method = "spin-table"; + cpu-release-addr = <0x0 0x01f00000>; + }; + }; + + memory { + device_type = "memory"; + reg = <0x0 0x0 0x0 0x0>; // filled in by zImage + }; + + MPIC: interrupt-controller { + compatible = "chrp,open-pic"; + interrupt-controller; + dcr-reg = <0xffc00000 0x00040000>; + #address-cells = <0>; + #size-cells = <0>; + #interrupt-cells = <2>; + single-cpu-affinity; + }; + + plb { + compatible = "ibm,plb6"; + #address-cells = <2>; + #size-cells = <2>; + ranges; + clock-frequency = <200000000>; // 200Mhz + + HSTA0: hsta@310000e0000 { + compatible = "ibm,476gtr-hsta-msi", "ibm,hsta-msi"; + reg = <0x310 0x000e0000 0x0 0xf0>; + interrupt-parent = <&MPIC>; + interrupts = <108 0 + 109 0 + 110 0 + 111 0 + 112 0 + 113 0 + 114 0 + 115 0 + 116 0 + 117 0 + 118 0 + 119 0 + 120 0 + 121 0 + 122 0 + 123 0>; + }; + + MAL0: mcmal { + compatible = "ibm,mcmal-476gtr", "ibm,mcmal2"; + dcr-reg = <0xc0000000 0x062>; + num-tx-chans = <1>; + num-rx-chans = <1>; + #address-cells = <0>; + #size-cells = <0>; + interrupt-parent = <&MPIC>; + interrupts = < /*TXEOB*/ 77 0x4 + /*RXEOB*/ 78 0x4 + /*SERR*/ 76 0x4 + /*TXDE*/ 79 0x4 + /*RXDE*/ 80 0x4>; + }; + + SATA0: sata@30000010000 { + compatible = "ibm,476gtr-ahci"; + reg = <0x300 0x00010000 0x0 0x10000>; + interrupt-parent = <&MPIC>; + interrupts = <93 2>; + }; + + EHCI0: ehci@30010000000 { + compatible = "ibm,476gtr-ehci", "generic-ehci"; + reg = <0x300 0x10000000 0x0 0x10000>; + interrupt-parent = <&MPIC>; + interrupts = <85 2>; + }; + + SD0: sd@30000000000 { + compatible = "ibm,476gtr-sdhci", "generic-sdhci"; + reg = <0x300 0x00000000 0x0 0x10000>; + interrupts = <91 2>; + interrupt-parent = <&MPIC>; + }; + + OHCI0: ohci@30010010000 { + compatible = "ibm,476gtr-ohci", "generic-ohci"; + reg = <0x300 0x10010000 0x0 0x10000>; + interrupt-parent = <&MPIC>; + interrupts = <89 1>; + }; + + OHCI1: ohci@30010020000 { + compatible = "ibm,476gtr-ohci", "generic-ohci"; + reg = <0x300 0x10020000 0x0 0x10000>; + interrupt-parent = <&MPIC>; + interrupts = <88 1>; + }; + + POB0: opb { + compatible = "ibm,opb-4xx", "ibm,opb"; + #address-cells = <1>; + #size-cells = <1>; + /* Wish there was a nicer way of specifying a full + * 32-bit range + */ + ranges = <0x00000000 0x0000033f 0x00000000 0x80000000 + 0x80000000 0x0000033f 0x80000000 0x80000000>; + clock-frequency = <100000000>; + + RGMII0: emac-rgmii-wol@50004 { + compatible = "ibm,rgmii-wol-476gtr", "ibm,rgmii-wol"; + reg = <0x50004 0x00000008>; + has-mdio; + }; + + EMAC0: ethernet@30000 { + device_type = "network"; + compatible = "ibm,emac-476gtr", "ibm,emac4sync"; + interrupt-parent = <&EMAC0>; + interrupts = <0x0 0x1>; + #interrupt-cells = <1>; + #address-cells = <0>; + #size-cells = <0>; + interrupt-map = </*Status*/ 0x0 &MPIC 81 0x4 + /*Wake*/ 0x1 &MPIC 82 0x4>; + reg = <0x30000 0x78>; + + /* local-mac-address will normally be added by + * the wrapper. If your device doesn't support + * passing data to the wrapper (in the form + * local-mac-addr=<hwaddr>) then you will need + * to set it manually here. */ + //local-mac-address = [000000000000]; + + mal-device = <&MAL0>; + mal-tx-channel = <0>; + mal-rx-channel = <0>; + cell-index = <0>; + max-frame-size = <9000>; + rx-fifo-size = <4096>; + tx-fifo-size = <2048>; + rx-fifo-size-gige = <16384>; + phy-mode = "rgmii"; + phy-map = <0x00000000>; + rgmii-wol-device = <&RGMII0>; + has-inverted-stacr-oc; + has-new-stacr-staopc; + }; + + UART0: serial@10000 { + device_type = "serial"; + compatible = "ns16750", "ns16550"; + reg = <0x10000 0x00000008>; + virtual-reg = <0xe8010000>; + clock-frequency = <1851851>; + current-speed = <38400>; + interrupt-parent = <&MPIC>; + interrupts = <39 2>; + }; + + IIC0: i2c@00000000 { + compatible = "ibm,iic-476gtr", "ibm,iic"; + reg = <0x0 0x00000020>; + interrupt-parent = <&MPIC>; + interrupts = <37 2>; + #address-cells = <1>; + #size-cells = <0>; + rtc@68 { + compatible = "stm,m41t80", "m41st85"; + reg = <0x68>; + }; + }; + + IIC1: i2c@00000100 { + compatible = "ibm,iic-476gtr", "ibm,iic"; + reg = <0x100 0x00000020>; + interrupt-parent = <&MPIC>; + interrupts = <38 2>; + #address-cells = <1>; + #size-cells = <0>; + avr@58 { + compatible = "ibm,akebono-avr"; + reg = <0x58>; + }; + }; + + FPGA0: fpga@ebc00000 { + compatible = "ibm,akebono-fpga"; + reg = <0xebc00000 0x8>; + }; + }; + + PCIE0: pciex@10100000000 { + device_type = "pci"; + #interrupt-cells = <1>; + #size-cells = <2>; + #address-cells = <3>; + compatible = "ibm,plb-pciex-476fpe", "ibm,plb-pciex"; + primary; + port = <0x0>; /* port number */ + reg = <0x00000101 0x00000000 0x0 0x10000000 /* Config space access */ + 0x00000100 0x00000000 0x0 0x00001000>; /* UTL Registers space access */ + dcr-reg = <0xc0 0x20>; + +// pci_space < pci_addr > < cpu_addr > < size > + ranges = <0x02000000 0x00000000 0x80000000 0x00000110 0x80000000 0x0 0x80000000 + 0x01000000 0x0 0x0 0x00000140 0x0 0x0 0x00010000>; + + /* Inbound starting at 0x0 to 0x40000000000. In order to use MSI + * PCI devices must be able to write to the HSTA module. + */ + dma-ranges = <0x42000000 0x0 0x0 0x0 0x0 0x400 0x0>; + + /* This drives busses 0 to 0xf */ + bus-range = <0x0 0xf>; + + /* Legacy interrupts (note the weird polarity, the bridge seems + * to invert PCIe legacy interrupts). + * We are de-swizzling here because the numbers are actually for + * port of the root complex virtual P2P bridge. But I want + * to avoid putting a node for it in the tree, so the numbers + * below are basically de-swizzled numbers. + * The real slot is on idsel 0, so the swizzling is 1:1 + */ + interrupt-map-mask = <0x0 0x0 0x0 0x7>; + interrupt-map = < + 0x0 0x0 0x0 0x1 &MPIC 45 0x2 /* int A */ + 0x0 0x0 0x0 0x2 &MPIC 46 0x2 /* int B */ + 0x0 0x0 0x0 0x3 &MPIC 47 0x2 /* int C */ + 0x0 0x0 0x0 0x4 &MPIC 48 0x2 /* int D */>; + }; + + PCIE1: pciex@20100000000 { + device_type = "pci"; + #interrupt-cells = <1>; + #size-cells = <2>; + #address-cells = <3>; + compatible = "ibm,plb-pciex-476fpe", "ibm,plb-pciex"; + primary; + port = <0x1>; /* port number */ + reg = <0x00000201 0x00000000 0x0 0x10000000 /* Config space access */ + 0x00000200 0x00000000 0x0 0x00001000>; /* UTL Registers space access */ + dcr-reg = <0x100 0x20>; + +// pci_space < pci_addr > < cpu_addr > < size > + ranges = <0x02000000 0x00000000 0x80000000 0x00000210 0x80000000 0x0 0x80000000 + 0x01000000 0x0 0x0 0x00000240 0x0 0x0 0x00010000>; + + /* Inbound starting at 0x0 to 0x40000000000. In order to use MSI + * PCI devices must be able to write to the HSTA module. + */ + dma-ranges = <0x42000000 0x0 0x0 0x0 0x0 0x400 0x0>; + + /* This drives busses 0 to 0xf */ + bus-range = <0x0 0xf>; + + /* Legacy interrupts (note the weird polarity, the bridge seems + * to invert PCIe legacy interrupts). + * We are de-swizzling here because the numbers are actually for + * port of the root complex virtual P2P bridge. But I want + * to avoid putting a node for it in the tree, so the numbers + * below are basically de-swizzled numbers. + * The real slot is on idsel 0, so the swizzling is 1:1 + */ + interrupt-map-mask = <0x0 0x0 0x0 0x7>; + interrupt-map = < + 0x0 0x0 0x0 0x1 &MPIC 53 0x2 /* int A */ + 0x0 0x0 0x0 0x2 &MPIC 54 0x2 /* int B */ + 0x0 0x0 0x0 0x3 &MPIC 55 0x2 /* int C */ + 0x0 0x0 0x0 0x4 &MPIC 56 0x2 /* int D */>; + }; + + PCIE2: pciex@18100000000 { + device_type = "pci"; + #interrupt-cells = <1>; + #size-cells = <2>; + #address-cells = <3>; + compatible = "ibm,plb-pciex-476fpe", "ibm,plb-pciex"; + primary; + port = <0x2>; /* port number */ + reg = <0x00000181 0x00000000 0x0 0x10000000 /* Config space access */ + 0x00000180 0x00000000 0x0 0x00001000>; /* UTL Registers space access */ + dcr-reg = <0xe0 0x20>; + +// pci_space < pci_addr > < cpu_addr > < size > + ranges = <0x02000000 0x00000000 0x80000000 0x00000190 0x80000000 0x0 0x80000000 + 0x01000000 0x0 0x0 0x000001c0 0x0 0x0 0x00010000>; + + /* Inbound starting at 0x0 to 0x40000000000. In order to use MSI + * PCI devices must be able to write to the HSTA module. + */ + dma-ranges = <0x42000000 0x0 0x0 0x0 0x0 0x400 0x0>; + + /* This drives busses 0 to 0xf */ + bus-range = <0x0 0xf>; + + /* Legacy interrupts (note the weird polarity, the bridge seems + * to invert PCIe legacy interrupts). + * We are de-swizzling here because the numbers are actually for + * port of the root complex virtual P2P bridge. But I want + * to avoid putting a node for it in the tree, so the numbers + * below are basically de-swizzled numbers. + * The real slot is on idsel 0, so the swizzling is 1:1 + */ + interrupt-map-mask = <0x0 0x0 0x0 0x7>; + interrupt-map = < + 0x0 0x0 0x0 0x1 &MPIC 61 0x2 /* int A */ + 0x0 0x0 0x0 0x2 &MPIC 62 0x2 /* int B */ + 0x0 0x0 0x0 0x3 &MPIC 63 0x2 /* int C */ + 0x0 0x0 0x0 0x4 &MPIC 64 0x2 /* int D */>; + }; + + PCIE3: pciex@28100000000 { + device_type = "pci"; + #interrupt-cells = <1>; + #size-cells = <2>; + #address-cells = <3>; + compatible = "ibm,plb-pciex-476fpe", "ibm,plb-pciex"; + primary; + port = <0x3>; /* port number */ + reg = <0x00000281 0x00000000 0x0 0x10000000 /* Config space access */ + 0x00000280 0x00000000 0x0 0x00001000>; /* UTL Registers space access */ + dcr-reg = <0x120 0x20>; + +// pci_space < pci_addr > < cpu_addr > < size > + ranges = <0x02000000 0x00000000 0x80000000 0x00000290 0x80000000 0x0 0x80000000 + 0x01000000 0x0 0x0 0x000002c0 0x0 0x0 0x00010000>; + + /* Inbound starting at 0x0 to 0x40000000000. In order to use MSI + * PCI devices must be able to write to the HSTA module. + */ + dma-ranges = <0x42000000 0x0 0x0 0x0 0x0 0x400 0x0>; + + /* This drives busses 0 to 0xf */ + bus-range = <0x0 0xf>; + + /* Legacy interrupts (note the weird polarity, the bridge seems + * to invert PCIe legacy interrupts). + * We are de-swizzling here because the numbers are actually for + * port of the root complex virtual P2P bridge. But I want + * to avoid putting a node for it in the tree, so the numbers + * below are basically de-swizzled numbers. + * The real slot is on idsel 0, so the swizzling is 1:1 + */ + interrupt-map-mask = <0x0 0x0 0x0 0x7>; + interrupt-map = < + 0x0 0x0 0x0 0x1 &MPIC 69 0x2 /* int A */ + 0x0 0x0 0x0 0x2 &MPIC 70 0x2 /* int B */ + 0x0 0x0 0x0 0x3 &MPIC 71 0x2 /* int C */ + 0x0 0x0 0x0 0x4 &MPIC 72 0x2 /* int D */>; + }; + }; + + chosen { + linux,stdout-path = &UART0; + }; +}; diff --git a/arch/powerpc/boot/dts/b4860emu.dts b/arch/powerpc/boot/dts/b4860emu.dts index 7290021f2dfc..85646b4f96e1 100644 --- a/arch/powerpc/boot/dts/b4860emu.dts +++ b/arch/powerpc/boot/dts/b4860emu.dts @@ -61,21 +61,25 @@ device_type = "cpu"; reg = <0 1>; next-level-cache = <&L2>; + fsl,portid-mapping = <0x80000000>; }; cpu1: PowerPC,e6500@2 { device_type = "cpu"; reg = <2 3>; next-level-cache = <&L2>; + fsl,portid-mapping = <0x80000000>; }; cpu2: PowerPC,e6500@4 { device_type = "cpu"; reg = <4 5>; next-level-cache = <&L2>; + fsl,portid-mapping = <0x80000000>; }; cpu3: PowerPC,e6500@6 { device_type = "cpu"; reg = <6 7>; next-level-cache = <&L2>; + fsl,portid-mapping = <0x80000000>; }; }; }; @@ -157,7 +161,7 @@ }; corenet-cf@18000 { - compatible = "fsl,b4-corenet-cf"; + compatible = "fsl,corenet2-cf", "fsl,corenet-cf"; reg = <0x18000 0x1000>; interrupts = <16 2 1 0>; fsl,ccf-num-csdids = <32>; @@ -167,6 +171,7 @@ iommu@20000 { compatible = "fsl,pamu-v1.0", "fsl,pamu"; reg = <0x20000 0x4000>; + fsl,portid-mapping = <0x8000>; #address-cells = <1>; #size-cells = <1>; interrupts = < diff --git a/arch/powerpc/boot/dts/bsc9132qds.dts b/arch/powerpc/boot/dts/bsc9132qds.dts new file mode 100644 index 000000000000..6cab1062bc74 --- /dev/null +++ b/arch/powerpc/boot/dts/bsc9132qds.dts @@ -0,0 +1,35 @@ +/* + * BSC9132 QDS Device Tree Source + * + * Copyright 2014 Freescale Semiconductor Inc. + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the + * Free Software Foundation; either version 2 of the License, or (at your + * option) any later version. + */ + +/include/ "fsl/bsc9132si-pre.dtsi" + +/ { + model = "fsl,bsc9132qds"; + compatible = "fsl,bsc9132qds"; + + memory { + device_type = "memory"; + }; + + ifc: ifc@ff71e000 { + /* NOR, NAND Flash on board */ + ranges = <0x0 0x0 0x0 0x88000000 0x08000000 + 0x1 0x0 0x0 0xff800000 0x00010000>; + reg = <0x0 0xff71e000 0x0 0x2000>; + }; + + soc: soc@ff700000 { + ranges = <0x0 0x0 0xff700000 0x100000>; + }; +}; + +/include/ "bsc9132qds.dtsi" +/include/ "fsl/bsc9132si-post.dtsi" diff --git a/arch/powerpc/boot/dts/bsc9132qds.dtsi b/arch/powerpc/boot/dts/bsc9132qds.dtsi new file mode 100644 index 000000000000..af8e88830221 --- /dev/null +++ b/arch/powerpc/boot/dts/bsc9132qds.dtsi @@ -0,0 +1,101 @@ +/* + * BSC9132 QDS Device Tree Source stub (no addresses or top-level ranges) + * + * Copyright 2014 Freescale Semiconductor Inc. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are met: + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * * Neither the name of Freescale Semiconductor nor the + * names of its contributors may be used to endorse or promote products + * derived from this software without specific prior written permission. + * + * + * ALTERNATIVELY, this software may be distributed under the terms of the + * GNU General Public License ("GPL") as published by the Free Software + * Foundation, either version 2 of that License or (at your option) any + * later version. + * + * THIS SOFTWARE IS PROVIDED BY Freescale Semiconductor ``AS IS'' AND ANY + * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED + * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE + * DISCLAIMED. IN NO EVENT SHALL Freescale Semiconductor BE LIABLE FOR ANY + * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES + * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; + * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND + * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS + * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +&ifc { + nor@0,0 { + #address-cells = <1>; + #size-cells = <1>; + compatible = "cfi-flash"; + reg = <0x0 0x0 0x8000000>; + bank-width = <2>; + device-width = <1>; + }; + + nand@1,0 { + #address-cells = <1>; + #size-cells = <1>; + compatible = "fsl,ifc-nand"; + reg = <0x1 0x0 0x4000>; + }; +}; + +&soc { + spi@7000 { + flash@0 { + #address-cells = <1>; + #size-cells = <1>; + compatible = "spansion,s25sl12801"; + reg = <0>; + spi-max-frequency = <30000000>; + }; + }; + + i2c@3000 { + fpga: fpga@66 { + compatible = "fsl,bsc9132qds-fpga", "fsl,fpga-qixis-i2c"; + reg = <0x66>; + }; + }; + + usb@22000 { + phy_type = "ulpi"; + }; + + mdio@24000 { + phy0: ethernet-phy@0 { + reg = <0x0>; + }; + + phy1: ethernet-phy@1 { + reg = <0x1>; + }; + + tbi0: tbi-phy@11 { + reg = <0x1f>; + device_type = "tbi-phy"; + }; + }; + + enet0: ethernet@b0000 { + phy-handle = <&phy0>; + tbi-handle = <&tbi0>; + phy-connection-type = "sgmii"; + }; + + enet1: ethernet@b1000 { + phy-handle = <&phy1>; + tbi-handle = <&tbi0>; + phy-connection-type = "sgmii"; + }; +}; diff --git a/arch/powerpc/boot/dts/fsl/b4420si-post.dtsi b/arch/powerpc/boot/dts/fsl/b4420si-post.dtsi index 60566f9927be..d67894459ac8 100644 --- a/arch/powerpc/boot/dts/fsl/b4420si-post.dtsi +++ b/arch/powerpc/boot/dts/fsl/b4420si-post.dtsi @@ -76,10 +76,6 @@ compatible = "fsl,b4420-l3-cache-controller", "cache"; }; - corenet-cf@18000 { - compatible = "fsl,b4420-corenet-cf"; - }; - guts: global-utilities@e0000 { compatible = "fsl,b4420-device-config", "fsl,qoriq-device-config-2.0"; }; diff --git a/arch/powerpc/boot/dts/fsl/b4420si-pre.dtsi b/arch/powerpc/boot/dts/fsl/b4420si-pre.dtsi index 2419731c2c54..338af7e39dd9 100644 --- a/arch/powerpc/boot/dts/fsl/b4420si-pre.dtsi +++ b/arch/powerpc/boot/dts/fsl/b4420si-pre.dtsi @@ -66,12 +66,14 @@ reg = <0 1>; clocks = <&mux0>; next-level-cache = <&L2>; + fsl,portid-mapping = <0x80000000>; }; cpu1: PowerPC,e6500@2 { device_type = "cpu"; reg = <2 3>; clocks = <&mux0>; next-level-cache = <&L2>; + fsl,portid-mapping = <0x80000000>; }; }; }; diff --git a/arch/powerpc/boot/dts/fsl/b4860si-post.dtsi b/arch/powerpc/boot/dts/fsl/b4860si-post.dtsi index cbc354b05117..582381dba1d7 100644 --- a/arch/powerpc/boot/dts/fsl/b4860si-post.dtsi +++ b/arch/powerpc/boot/dts/fsl/b4860si-post.dtsi @@ -120,10 +120,6 @@ compatible = "fsl,b4860-l3-cache-controller", "cache"; }; - corenet-cf@18000 { - compatible = "fsl,b4860-corenet-cf"; - }; - guts: global-utilities@e0000 { compatible = "fsl,b4860-device-config", "fsl,qoriq-device-config-2.0"; }; diff --git a/arch/powerpc/boot/dts/fsl/b4860si-pre.dtsi b/arch/powerpc/boot/dts/fsl/b4860si-pre.dtsi index 142ac862cacf..1948f73fd26b 100644 --- a/arch/powerpc/boot/dts/fsl/b4860si-pre.dtsi +++ b/arch/powerpc/boot/dts/fsl/b4860si-pre.dtsi @@ -66,24 +66,28 @@ reg = <0 1>; clocks = <&mux0>; next-level-cache = <&L2>; + fsl,portid-mapping = <0x80000000>; }; cpu1: PowerPC,e6500@2 { device_type = "cpu"; reg = <2 3>; clocks = <&mux0>; next-level-cache = <&L2>; + fsl,portid-mapping = <0x80000000>; }; cpu2: PowerPC,e6500@4 { device_type = "cpu"; reg = <4 5>; clocks = <&mux0>; next-level-cache = <&L2>; + fsl,portid-mapping = <0x80000000>; }; cpu3: PowerPC,e6500@6 { device_type = "cpu"; reg = <6 7>; clocks = <&mux0>; next-level-cache = <&L2>; + fsl,portid-mapping = <0x80000000>; }; }; }; diff --git a/arch/powerpc/boot/dts/fsl/b4si-post.dtsi b/arch/powerpc/boot/dts/fsl/b4si-post.dtsi index 4f6e48277c46..1a54ba71f685 100644 --- a/arch/powerpc/boot/dts/fsl/b4si-post.dtsi +++ b/arch/powerpc/boot/dts/fsl/b4si-post.dtsi @@ -158,7 +158,7 @@ }; corenet-cf@18000 { - compatible = "fsl,b4-corenet-cf"; + compatible = "fsl,corenet2-cf", "fsl,corenet-cf"; reg = <0x18000 0x1000>; interrupts = <16 2 1 0>; fsl,ccf-num-csdids = <32>; @@ -168,6 +168,7 @@ iommu@20000 { compatible = "fsl,pamu-v1.0", "fsl,pamu"; reg = <0x20000 0x4000>; + fsl,portid-mapping = <0x8000>; #address-cells = <1>; #size-cells = <1>; interrupts = < diff --git a/arch/powerpc/boot/dts/fsl/bsc9132si-post.dtsi b/arch/powerpc/boot/dts/fsl/bsc9132si-post.dtsi new file mode 100644 index 000000000000..c72307198140 --- /dev/null +++ b/arch/powerpc/boot/dts/fsl/bsc9132si-post.dtsi @@ -0,0 +1,185 @@ +/* + * BSC9132 Silicon/SoC Device Tree Source (post include) + * + * Copyright 2014 Freescale Semiconductor Inc. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are met: + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * * Neither the name of Freescale Semiconductor nor the + * names of its contributors may be used to endorse or promote products + * derived from this software without specific prior written permission. + * + * + * ALTERNATIVELY, this software may be distributed under the terms of the + * GNU General Public License ("GPL") as published by the Free Software + * Foundation, either version 2 of that License or (at your option) any + * later version. + * + * THIS SOFTWARE IS PROVIDED BY Freescale Semiconductor ``AS IS'' AND ANY + * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED + * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE + * DISCLAIMED. IN NO EVENT SHALL Freescale Semiconductor BE LIABLE FOR ANY + * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES + * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; + * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND + * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS + * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +&ifc { + #address-cells = <2>; + #size-cells = <1>; + compatible = "fsl,ifc", "simple-bus"; + /* FIXME: Test whether interrupts are split */ + interrupts = <16 2 0 0 20 2 0 0>; +}; + +&soc { + #address-cells = <1>; + #size-cells = <1>; + device_type = "soc"; + compatible = "fsl,bsc9132-immr", "simple-bus"; + bus-frequency = <0>; // Filled out by uboot. + + ecm-law@0 { + compatible = "fsl,ecm-law"; + reg = <0x0 0x1000>; + fsl,num-laws = <12>; + }; + + ecm@1000 { + compatible = "fsl,bsc9132-ecm", "fsl,ecm"; + reg = <0x1000 0x1000>; + interrupts = <16 2 0 0>; + }; + + memory-controller@2000 { + compatible = "fsl,bsc9132-memory-controller"; + reg = <0x2000 0x1000>; + interrupts = <16 2 1 8>; + }; + +/include/ "pq3-i2c-0.dtsi" + i2c@3000 { + interrupts = <17 2 0 0>; + }; + +/include/ "pq3-i2c-1.dtsi" + i2c@3100 { + interrupts = <17 2 0 0>; + }; + +/include/ "pq3-duart-0.dtsi" + serial0: serial@4500 { + interrupts = <18 2 0 0>; + }; + + serial1: serial@4600 { + interrupts = <18 2 0 0 >; + }; +/include/ "pq3-espi-0.dtsi" + spi0: spi@7000 { + fsl,espi-num-chipselects = <1>; + interrupts = <22 0x2 0 0>; + }; + +/include/ "pq3-gpio-0.dtsi" + gpio-controller@f000 { + interrupts = <19 0x2 0 0>; + }; + + L2: l2-cache-controller@20000 { + compatible = "fsl,bsc9132-l2-cache-controller"; + reg = <0x20000 0x1000>; + cache-line-size = <32>; // 32 bytes + cache-size = <0x40000>; // L2,256K + interrupts = <16 2 1 0>; + }; + +/include/ "pq3-dma-0.dtsi" + +dma@21300 { + + dma-channel@0 { + interrupts = <62 2 0 0>; + }; + + dma-channel@80 { + interrupts = <63 2 0 0>; + }; + + dma-channel@100 { + interrupts = <64 2 0 0>; + }; + + dma-channel@180 { + interrupts = <65 2 0 0>; + }; +}; + +/include/ "pq3-usb2-dr-0.dtsi" +usb@22000 { + compatible = "fsl-usb2-dr","fsl-usb2-dr-v2.2"; + interrupts = <40 0x2 0 0>; +}; + +/include/ "pq3-esdhc-0.dtsi" + sdhc@2e000 { + fsl,sdhci-auto-cmd12; + interrupts = <41 0x2 0 0>; + }; + +/include/ "pq3-sec4.4-0.dtsi" +crypto@30000 { + interrupts = <57 2 0 0>; + + sec_jr0: jr@1000 { + interrupts = <58 2 0 0>; + }; + + sec_jr1: jr@2000 { + interrupts = <59 2 0 0>; + }; + + sec_jr2: jr@3000 { + interrupts = <60 2 0 0>; + }; + + sec_jr3: jr@4000 { + interrupts = <61 2 0 0>; + }; +}; + +/include/ "pq3-mpic.dtsi" +/include/ "pq3-mpic-timer-B.dtsi" + +/include/ "pq3-etsec2-0.dtsi" +enet0: ethernet@b0000 { + queue-group@b0000 { + fsl,rx-bit-map = <0xff>; + fsl,tx-bit-map = <0xff>; + interrupts = <26 2 0 0 27 2 0 0 28 2 0 0>; + }; +}; + +/include/ "pq3-etsec2-1.dtsi" +enet1: ethernet@b1000 { + queue-group@b1000 { + fsl,rx-bit-map = <0xff>; + fsl,tx-bit-map = <0xff>; + interrupts = <33 2 0 0 34 2 0 0 35 2 0 0>; + }; +}; + +global-utilities@e0000 { + compatible = "fsl,bsc9132-guts"; + reg = <0xe0000 0x1000>; + fsl,has-rstcr; + }; +}; diff --git a/arch/powerpc/boot/dts/fsl/bsc9132si-pre.dtsi b/arch/powerpc/boot/dts/fsl/bsc9132si-pre.dtsi new file mode 100644 index 000000000000..301a9dba5790 --- /dev/null +++ b/arch/powerpc/boot/dts/fsl/bsc9132si-pre.dtsi @@ -0,0 +1,66 @@ +/* + * BSC9132 Silicon/SoC Device Tree Source (pre include) + * + * Copyright 2014 Freescale Semiconductor Inc. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are met: + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * * Neither the name of Freescale Semiconductor nor the + * names of its contributors may be used to endorse or promote products + * derived from this software without specific prior written permission. + * + * + * ALTERNATIVELY, this software may be distributed under the terms of the + * GNU General Public License ("GPL") as published by the Free Software + * Foundation, either version 2 of that License or (at your option) any + * later version. + * + * THIS SOFTWARE IS PROVIDED BY Freescale Semiconductor ``AS IS'' AND ANY + * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED + * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE + * DISCLAIMED. IN NO EVENT SHALL Freescale Semiconductor BE LIABLE FOR ANY + * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES + * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; + * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND + * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS + * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +/dts-v1/; + +/include/ "e500v2_power_isa.dtsi" + +/ { + #address-cells = <2>; + #size-cells = <2>; + interrupt-parent = <&mpic>; + + aliases { + serial0 = &serial0; + ethernet0 = &enet0; + ethernet1 = &enet1; + }; + + cpus { + #address-cells = <1>; + #size-cells = <0>; + + cpu0: PowerPC,e500v2@0 { + device_type = "cpu"; + reg = <0x0>; + next-level-cache = <&L2>; + }; + + cpu1: PowerPC,e500v2@1 { + device_type = "cpu"; + reg = <0x1>; + next-level-cache = <&L2>; + }; + }; +}; diff --git a/arch/powerpc/boot/dts/fsl/p2041si-post.dtsi b/arch/powerpc/boot/dts/fsl/p2041si-post.dtsi index e2987a33083c..5290df83ff30 100644 --- a/arch/powerpc/boot/dts/fsl/p2041si-post.dtsi +++ b/arch/powerpc/boot/dts/fsl/p2041si-post.dtsi @@ -246,7 +246,7 @@ }; corenet-cf@18000 { - compatible = "fsl,corenet-cf"; + compatible = "fsl,corenet1-cf", "fsl,corenet-cf"; reg = <0x18000 0x1000>; interrupts = <16 2 1 31>; fsl,ccf-num-csdids = <32>; @@ -262,6 +262,7 @@ interrupts = < 24 2 0 0 16 2 1 30>; + fsl,portid-mapping = <0x0f000000>; pamu0: pamu@0 { reg = <0 0x1000>; diff --git a/arch/powerpc/boot/dts/fsl/p2041si-pre.dtsi b/arch/powerpc/boot/dts/fsl/p2041si-pre.dtsi index 22f3b14517de..b1ea147f2995 100644 --- a/arch/powerpc/boot/dts/fsl/p2041si-pre.dtsi +++ b/arch/powerpc/boot/dts/fsl/p2041si-pre.dtsi @@ -83,6 +83,7 @@ reg = <0>; clocks = <&mux0>; next-level-cache = <&L2_0>; + fsl,portid-mapping = <0x80000000>; L2_0: l2-cache { next-level-cache = <&cpc>; }; @@ -92,6 +93,7 @@ reg = <1>; clocks = <&mux1>; next-level-cache = <&L2_1>; + fsl,portid-mapping = <0x40000000>; L2_1: l2-cache { next-level-cache = <&cpc>; }; @@ -101,6 +103,7 @@ reg = <2>; clocks = <&mux2>; next-level-cache = <&L2_2>; + fsl,portid-mapping = <0x20000000>; L2_2: l2-cache { next-level-cache = <&cpc>; }; @@ -110,6 +113,7 @@ reg = <3>; clocks = <&mux3>; next-level-cache = <&L2_3>; + fsl,portid-mapping = <0x10000000>; L2_3: l2-cache { next-level-cache = <&cpc>; }; diff --git a/arch/powerpc/boot/dts/fsl/p3041si-post.dtsi b/arch/powerpc/boot/dts/fsl/p3041si-post.dtsi index 7af6d45fd998..cd63cb1b1042 100644 --- a/arch/powerpc/boot/dts/fsl/p3041si-post.dtsi +++ b/arch/powerpc/boot/dts/fsl/p3041si-post.dtsi @@ -273,7 +273,7 @@ }; corenet-cf@18000 { - compatible = "fsl,corenet-cf"; + compatible = "fsl,corenet1-cf", "fsl,corenet-cf"; reg = <0x18000 0x1000>; interrupts = <16 2 1 31>; fsl,ccf-num-csdids = <32>; @@ -289,6 +289,7 @@ interrupts = < 24 2 0 0 16 2 1 30>; + fsl,portid-mapping = <0x0f000000>; pamu0: pamu@0 { reg = <0 0x1000>; diff --git a/arch/powerpc/boot/dts/fsl/p3041si-pre.dtsi b/arch/powerpc/boot/dts/fsl/p3041si-pre.dtsi index 468e8be8ac6f..dc5f4b362c24 100644 --- a/arch/powerpc/boot/dts/fsl/p3041si-pre.dtsi +++ b/arch/powerpc/boot/dts/fsl/p3041si-pre.dtsi @@ -84,6 +84,7 @@ reg = <0>; clocks = <&mux0>; next-level-cache = <&L2_0>; + fsl,portid-mapping = <0x80000000>; L2_0: l2-cache { next-level-cache = <&cpc>; }; @@ -93,6 +94,7 @@ reg = <1>; clocks = <&mux1>; next-level-cache = <&L2_1>; + fsl,portid-mapping = <0x40000000>; L2_1: l2-cache { next-level-cache = <&cpc>; }; @@ -102,6 +104,7 @@ reg = <2>; clocks = <&mux2>; next-level-cache = <&L2_2>; + fsl,portid-mapping = <0x20000000>; L2_2: l2-cache { next-level-cache = <&cpc>; }; @@ -111,6 +114,7 @@ reg = <3>; clocks = <&mux3>; next-level-cache = <&L2_3>; + fsl,portid-mapping = <0x10000000>; L2_3: l2-cache { next-level-cache = <&cpc>; }; diff --git a/arch/powerpc/boot/dts/fsl/p4080si-post.dtsi b/arch/powerpc/boot/dts/fsl/p4080si-post.dtsi index 2415e1f1d3fa..12947ccddf25 100644 --- a/arch/powerpc/boot/dts/fsl/p4080si-post.dtsi +++ b/arch/powerpc/boot/dts/fsl/p4080si-post.dtsi @@ -281,7 +281,7 @@ }; corenet-cf@18000 { - compatible = "fsl,corenet-cf"; + compatible = "fsl,corenet1-cf", "fsl,corenet-cf"; reg = <0x18000 0x1000>; interrupts = <16 2 1 31>; fsl,ccf-num-csdids = <32>; @@ -297,6 +297,7 @@ interrupts = < 24 2 0 0 16 2 1 30>; + fsl,portid-mapping = <0x00f80000>; pamu0: pamu@0 { reg = <0 0x1000>; diff --git a/arch/powerpc/boot/dts/fsl/p4080si-pre.dtsi b/arch/powerpc/boot/dts/fsl/p4080si-pre.dtsi index 0040b5a5379e..38bde0958672 100644 --- a/arch/powerpc/boot/dts/fsl/p4080si-pre.dtsi +++ b/arch/powerpc/boot/dts/fsl/p4080si-pre.dtsi @@ -83,6 +83,7 @@ reg = <0>; clocks = <&mux0>; next-level-cache = <&L2_0>; + fsl,portid-mapping = <0x80000000>; L2_0: l2-cache { next-level-cache = <&cpc>; }; @@ -92,6 +93,7 @@ reg = <1>; clocks = <&mux1>; next-level-cache = <&L2_1>; + fsl,portid-mapping = <0x40000000>; L2_1: l2-cache { next-level-cache = <&cpc>; }; @@ -101,6 +103,7 @@ reg = <2>; clocks = <&mux2>; next-level-cache = <&L2_2>; + fsl,portid-mapping = <0x20000000>; L2_2: l2-cache { next-level-cache = <&cpc>; }; @@ -110,6 +113,7 @@ reg = <3>; clocks = <&mux3>; next-level-cache = <&L2_3>; + fsl,portid-mapping = <0x10000000>; L2_3: l2-cache { next-level-cache = <&cpc>; }; @@ -119,6 +123,7 @@ reg = <4>; clocks = <&mux4>; next-level-cache = <&L2_4>; + fsl,portid-mapping = <0x08000000>; L2_4: l2-cache { next-level-cache = <&cpc>; }; @@ -128,6 +133,7 @@ reg = <5>; clocks = <&mux5>; next-level-cache = <&L2_5>; + fsl,portid-mapping = <0x04000000>; L2_5: l2-cache { next-level-cache = <&cpc>; }; @@ -137,6 +143,7 @@ reg = <6>; clocks = <&mux6>; next-level-cache = <&L2_6>; + fsl,portid-mapping = <0x02000000>; L2_6: l2-cache { next-level-cache = <&cpc>; }; @@ -146,6 +153,7 @@ reg = <7>; clocks = <&mux7>; next-level-cache = <&L2_7>; + fsl,portid-mapping = <0x01000000>; L2_7: l2-cache { next-level-cache = <&cpc>; }; diff --git a/arch/powerpc/boot/dts/fsl/p5020si-post.dtsi b/arch/powerpc/boot/dts/fsl/p5020si-post.dtsi index 2985de4ad6be..4c4a2b0436b2 100644 --- a/arch/powerpc/boot/dts/fsl/p5020si-post.dtsi +++ b/arch/powerpc/boot/dts/fsl/p5020si-post.dtsi @@ -278,7 +278,7 @@ }; corenet-cf@18000 { - compatible = "fsl,corenet-cf"; + compatible = "fsl,corenet1-cf", "fsl,corenet-cf"; reg = <0x18000 0x1000>; interrupts = <16 2 1 31>; fsl,ccf-num-csdids = <32>; @@ -294,6 +294,7 @@ interrupts = < 24 2 0 0 16 2 1 30>; + fsl,portid-mapping = <0x3c000000>; pamu0: pamu@0 { reg = <0 0x1000>; diff --git a/arch/powerpc/boot/dts/fsl/p5020si-pre.dtsi b/arch/powerpc/boot/dts/fsl/p5020si-pre.dtsi index fe1a2e6613b4..1cc61e126e4c 100644 --- a/arch/powerpc/boot/dts/fsl/p5020si-pre.dtsi +++ b/arch/powerpc/boot/dts/fsl/p5020si-pre.dtsi @@ -90,6 +90,7 @@ reg = <0>; clocks = <&mux0>; next-level-cache = <&L2_0>; + fsl,portid-mapping = <0x80000000>; L2_0: l2-cache { next-level-cache = <&cpc>; }; @@ -99,6 +100,7 @@ reg = <1>; clocks = <&mux1>; next-level-cache = <&L2_1>; + fsl,portid-mapping = <0x40000000>; L2_1: l2-cache { next-level-cache = <&cpc>; }; diff --git a/arch/powerpc/boot/dts/fsl/p5040si-post.dtsi b/arch/powerpc/boot/dts/fsl/p5040si-post.dtsi index 546a899efe20..67296fdd9698 100644 --- a/arch/powerpc/boot/dts/fsl/p5040si-post.dtsi +++ b/arch/powerpc/boot/dts/fsl/p5040si-post.dtsi @@ -233,7 +233,7 @@ }; corenet-cf@18000 { - compatible = "fsl,corenet-cf"; + compatible = "fsl,corenet1-cf", "fsl,corenet-cf"; reg = <0x18000 0x1000>; interrupts = <16 2 1 31>; fsl,ccf-num-csdids = <32>; @@ -248,6 +248,7 @@ #size-cells = <1>; interrupts = <24 2 0 0 16 2 1 30>; + fsl,portid-mapping = <0x0f800000>; pamu0: pamu@0 { reg = <0 0x1000>; diff --git a/arch/powerpc/boot/dts/fsl/p5040si-pre.dtsi b/arch/powerpc/boot/dts/fsl/p5040si-pre.dtsi index 3674686687cb..b048a2be05a8 100644 --- a/arch/powerpc/boot/dts/fsl/p5040si-pre.dtsi +++ b/arch/powerpc/boot/dts/fsl/p5040si-pre.dtsi @@ -83,6 +83,7 @@ reg = <0>; clocks = <&mux0>; next-level-cache = <&L2_0>; + fsl,portid-mapping = <0x80000000>; L2_0: l2-cache { next-level-cache = <&cpc>; }; @@ -92,6 +93,7 @@ reg = <1>; clocks = <&mux1>; next-level-cache = <&L2_1>; + fsl,portid-mapping = <0x40000000>; L2_1: l2-cache { next-level-cache = <&cpc>; }; @@ -101,6 +103,7 @@ reg = <2>; clocks = <&mux2>; next-level-cache = <&L2_2>; + fsl,portid-mapping = <0x20000000>; L2_2: l2-cache { next-level-cache = <&cpc>; }; @@ -110,6 +113,7 @@ reg = <3>; clocks = <&mux3>; next-level-cache = <&L2_3>; + fsl,portid-mapping = <0x10000000>; L2_3: l2-cache { next-level-cache = <&cpc>; }; diff --git a/arch/powerpc/boot/dts/fsl/t1040si-post.dtsi b/arch/powerpc/boot/dts/fsl/t1040si-post.dtsi new file mode 100644 index 000000000000..12e597eea3c8 --- /dev/null +++ b/arch/powerpc/boot/dts/fsl/t1040si-post.dtsi @@ -0,0 +1,430 @@ +/* + * T1040 Silicon/SoC Device Tree Source (post include) + * + * Copyright 2013 Freescale Semiconductor Inc. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are met: + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * * Neither the name of Freescale Semiconductor nor the + * names of its contributors may be used to endorse or promote products + * derived from this software without specific prior written permission. + * + * + * ALTERNATIVELY, this software may be distributed under the terms of the + * GNU General Public License ("GPL") as published by the Free Software + * Foundation, either version 2 of that License or (at your option) any + * later version. + * + * THIS SOFTWARE IS PROVIDED BY Freescale Semiconductor ``AS IS'' AND ANY + * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED + * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE + * DISCLAIMED. IN NO EVENT SHALL Freescale Semiconductor BE LIABLE FOR ANY + * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES + * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; + * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND + * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS + * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +&ifc { + #address-cells = <2>; + #size-cells = <1>; + compatible = "fsl,ifc", "simple-bus"; + interrupts = <25 2 0 0>; +}; + +&pci0 { + compatible = "fsl,t1040-pcie", "fsl,qoriq-pcie-v2.4", "fsl,qoriq-pcie"; + device_type = "pci"; + #size-cells = <2>; + #address-cells = <3>; + bus-range = <0x0 0xff>; + interrupts = <20 2 0 0>; + fsl,iommu-parent = <&pamu0>; + pcie@0 { + reg = <0 0 0 0 0>; + #interrupt-cells = <1>; + #size-cells = <2>; + #address-cells = <3>; + device_type = "pci"; + interrupts = <20 2 0 0>; + interrupt-map-mask = <0xf800 0 0 7>; + interrupt-map = < + /* IDSEL 0x0 */ + 0000 0 0 1 &mpic 40 1 0 0 + 0000 0 0 2 &mpic 1 1 0 0 + 0000 0 0 3 &mpic 2 1 0 0 + 0000 0 0 4 &mpic 3 1 0 0 + >; + }; +}; + +&pci1 { + compatible = "fsl,t1040-pcie", "fsl,qoriq-pcie-v2.4", "fsl,qoriq-pcie"; + device_type = "pci"; + #size-cells = <2>; + #address-cells = <3>; + bus-range = <0 0xff>; + interrupts = <21 2 0 0>; + fsl,iommu-parent = <&pamu0>; + pcie@0 { + reg = <0 0 0 0 0>; + #interrupt-cells = <1>; + #size-cells = <2>; + #address-cells = <3>; + device_type = "pci"; + interrupts = <21 2 0 0>; + interrupt-map-mask = <0xf800 0 0 7>; + interrupt-map = < + /* IDSEL 0x0 */ + 0000 0 0 1 &mpic 41 1 0 0 + 0000 0 0 2 &mpic 5 1 0 0 + 0000 0 0 3 &mpic 6 1 0 0 + 0000 0 0 4 &mpic 7 1 0 0 + >; + }; +}; + +&pci2 { + compatible = "fsl,t1040-pcie", "fsl,qoriq-pcie-v2.4", "fsl,qoriq-pcie"; + device_type = "pci"; + #size-cells = <2>; + #address-cells = <3>; + bus-range = <0x0 0xff>; + interrupts = <22 2 0 0>; + fsl,iommu-parent = <&pamu0>; + pcie@0 { + reg = <0 0 0 0 0>; + #interrupt-cells = <1>; + #size-cells = <2>; + #address-cells = <3>; + device_type = "pci"; + interrupts = <22 2 0 0>; + interrupt-map-mask = <0xf800 0 0 7>; + interrupt-map = < + /* IDSEL 0x0 */ + 0000 0 0 1 &mpic 42 1 0 0 + 0000 0 0 2 &mpic 9 1 0 0 + 0000 0 0 3 &mpic 10 1 0 0 + 0000 0 0 4 &mpic 11 1 0 0 + >; + }; +}; + +&pci3 { + compatible = "fsl,t1040-pcie", "fsl,qoriq-pcie-v2.4", "fsl,qoriq-pcie"; + device_type = "pci"; + #size-cells = <2>; + #address-cells = <3>; + bus-range = <0x0 0xff>; + interrupts = <23 2 0 0>; + fsl,iommu-parent = <&pamu0>; + pcie@0 { + reg = <0 0 0 0 0>; + #interrupt-cells = <1>; + #size-cells = <2>; + #address-cells = <3>; + device_type = "pci"; + interrupts = <23 2 0 0>; + interrupt-map-mask = <0xf800 0 0 7>; + interrupt-map = < + /* IDSEL 0x0 */ + 0000 0 0 1 &mpic 43 1 0 0 + 0000 0 0 2 &mpic 0 1 0 0 + 0000 0 0 3 &mpic 4 1 0 0 + 0000 0 0 4 &mpic 8 1 0 0 + >; + }; +}; + +&dcsr { + #address-cells = <1>; + #size-cells = <1>; + compatible = "fsl,dcsr", "simple-bus"; + + dcsr-epu@0 { + compatible = "fsl,t1040-dcsr-epu", "fsl,dcsr-epu"; + interrupts = <52 2 0 0 + 84 2 0 0 + 85 2 0 0>; + reg = <0x0 0x1000>; + }; + dcsr-npc { + compatible = "fsl,t1040-dcsr-cnpc", "fsl,dcsr-cnpc"; + reg = <0x1000 0x1000 0x1002000 0x10000>; + }; + dcsr-nxc@2000 { + compatible = "fsl,dcsr-nxc"; + reg = <0x2000 0x1000>; + }; + dcsr-corenet { + compatible = "fsl,dcsr-corenet"; + reg = <0x8000 0x1000 0x1A000 0x1000>; + }; + dcsr-dpaa@9000 { + compatible = "fsl,t1040-dcsr-dpaa", "fsl,dcsr-dpaa"; + reg = <0x9000 0x1000>; + }; + dcsr-ocn@11000 { + compatible = "fsl,t1040-dcsr-ocn", "fsl,dcsr-ocn"; + reg = <0x11000 0x1000>; + }; + dcsr-ddr@12000 { + compatible = "fsl,dcsr-ddr"; + dev-handle = <&ddr1>; + reg = <0x12000 0x1000>; + }; + dcsr-nal@18000 { + compatible = "fsl,t1040-dcsr-nal", "fsl,dcsr-nal"; + reg = <0x18000 0x1000>; + }; + dcsr-rcpm@22000 { + compatible = "fsl,t1040-dcsr-rcpm", "fsl,dcsr-rcpm"; + reg = <0x22000 0x1000>; + }; + dcsr-snpc@30000 { + compatible = "fsl,t1040-dcsr-snpc", "fsl,dcsr-snpc"; + reg = <0x30000 0x1000 0x1022000 0x10000>; + }; + dcsr-snpc@31000 { + compatible = "fsl,t1040-dcsr-snpc", "fsl,dcsr-snpc"; + reg = <0x31000 0x1000 0x1042000 0x10000>; + }; + dcsr-cpu-sb-proxy@100000 { + compatible = "fsl,dcsr-e5500-sb-proxy", "fsl,dcsr-cpu-sb-proxy"; + cpu-handle = <&cpu0>; + reg = <0x100000 0x1000 0x101000 0x1000>; + }; + dcsr-cpu-sb-proxy@108000 { + compatible = "fsl,dcsr-e5500-sb-proxy", "fsl,dcsr-cpu-sb-proxy"; + cpu-handle = <&cpu1>; + reg = <0x108000 0x1000 0x109000 0x1000>; + }; + dcsr-cpu-sb-proxy@110000 { + compatible = "fsl,dcsr-e5500-sb-proxy", "fsl,dcsr-cpu-sb-proxy"; + cpu-handle = <&cpu2>; + reg = <0x110000 0x1000 0x111000 0x1000>; + }; + dcsr-cpu-sb-proxy@118000 { + compatible = "fsl,dcsr-e5500-sb-proxy", "fsl,dcsr-cpu-sb-proxy"; + cpu-handle = <&cpu3>; + reg = <0x118000 0x1000 0x119000 0x1000>; + }; +}; + +&soc { + #address-cells = <1>; + #size-cells = <1>; + device_type = "soc"; + compatible = "simple-bus"; + + soc-sram-error { + compatible = "fsl,soc-sram-error"; + interrupts = <16 2 1 29>; + }; + + corenet-law@0 { + compatible = "fsl,corenet-law"; + reg = <0x0 0x1000>; + fsl,num-laws = <16>; + }; + + ddr1: memory-controller@8000 { + compatible = "fsl,qoriq-memory-controller-v5.0", + "fsl,qoriq-memory-controller"; + reg = <0x8000 0x1000>; + interrupts = <16 2 1 23>; + }; + + cpc: l3-cache-controller@10000 { + compatible = "fsl,t1040-l3-cache-controller", "cache"; + reg = <0x10000 0x1000>; + interrupts = <16 2 1 27>; + }; + + corenet-cf@18000 { + compatible = "fsl,corenet2-cf", "fsl,corenet-cf"; + reg = <0x18000 0x1000>; + interrupts = <16 2 1 31>; + fsl,ccf-num-csdids = <32>; + fsl,ccf-num-snoopids = <32>; + }; + + iommu@20000 { + compatible = "fsl,pamu-v1.0", "fsl,pamu"; + reg = <0x20000 0x1000>; + ranges = <0 0x20000 0x1000>; + #address-cells = <1>; + #size-cells = <1>; + interrupts = < + 24 2 0 0 + 16 2 1 30>; + pamu0: pamu@0 { + reg = <0 0x1000>; + fsl,primary-cache-geometry = <128 1>; + fsl,secondary-cache-geometry = <16 2>; + }; + }; + +/include/ "qoriq-mpic.dtsi" + + guts: global-utilities@e0000 { + compatible = "fsl,t1040-device-config", "fsl,qoriq-device-config-2.0"; + reg = <0xe0000 0xe00>; + fsl,has-rstcr; + fsl,liodn-bits = <12>; + }; + + clockgen: global-utilities@e1000 { + compatible = "fsl,t1040-clockgen", "fsl,qoriq-clockgen-2.0"; + ranges = <0x0 0xe1000 0x1000>; + reg = <0xe1000 0x1000>; + #address-cells = <1>; + #size-cells = <1>; + + sysclk: sysclk { + #clock-cells = <0>; + compatible = "fsl,qoriq-sysclk-2.0"; + clock-output-names = "sysclk", "fixed-clock"; + }; + + + pll0: pll0@800 { + #clock-cells = <1>; + reg = <0x800 4>; + compatible = "fsl,qoriq-core-pll-2.0"; + clocks = <&sysclk>; + clock-output-names = "pll0", "pll0-div2", "pll0-div4"; + }; + + pll1: pll1@820 { + #clock-cells = <1>; + reg = <0x820 4>; + compatible = "fsl,qoriq-core-pll-2.0"; + clocks = <&sysclk>; + clock-output-names = "pll1", "pll1-div2", "pll1-div4"; + }; + + mux0: mux0@0 { + #clock-cells = <0>; + reg = <0x0 4>; + compatible = "fsl,qoriq-core-mux-2.0"; + clocks = <&pll0 0>, <&pll0 1>, <&pll0 2>, + <&pll1 0>, <&pll1 1>, <&pll1 2>; + clock-names = "pll0", "pll0-div2", "pll1-div4", + "pll1", "pll1-div2", "pll1-div4"; + clock-output-names = "cmux0"; + }; + + mux1: mux1@20 { + #clock-cells = <0>; + reg = <0x20 4>; + compatible = "fsl,qoriq-core-mux-2.0"; + clocks = <&pll0 0>, <&pll0 1>, <&pll0 2>, + <&pll1 0>, <&pll1 1>, <&pll1 2>; + clock-names = "pll0", "pll0-div2", "pll1-div4", + "pll1", "pll1-div2", "pll1-div4"; + clock-output-names = "cmux1"; + }; + + mux2: mux2@40 { + #clock-cells = <0>; + reg = <0x40 4>; + compatible = "fsl,qoriq-core-mux-2.0"; + clocks = <&pll0 0>, <&pll0 1>, <&pll0 2>, + <&pll1 0>, <&pll1 1>, <&pll1 2>; + clock-names = "pll0", "pll0-div2", "pll1-div4", + "pll1", "pll1-div2", "pll1-div4"; + clock-output-names = "cmux2"; + }; + + mux3: mux3@60 { + #clock-cells = <0>; + reg = <0x60 4>; + compatible = "fsl,qoriq-core-mux-2.0"; + clocks = <&pll0 0>, <&pll0 1>, <&pll0 2>, + <&pll1 0>, <&pll1 1>, <&pll1 2>; + clock-names = "pll0_0", "pll0_1", "pll0_2", + "pll1_0", "pll1_1", "pll1_2"; + clock-output-names = "cmux3"; + }; + }; + + rcpm: global-utilities@e2000 { + compatible = "fsl,t1040-rcpm", "fsl,qoriq-rcpm-2.0"; + reg = <0xe2000 0x1000>; + }; + + sfp: sfp@e8000 { + compatible = "fsl,t1040-sfp"; + reg = <0xe8000 0x1000>; + }; + + serdes: serdes@ea000 { + compatible = "fsl,t1040-serdes"; + reg = <0xea000 0x4000>; + }; + +/include/ "elo3-dma-0.dtsi" +/include/ "elo3-dma-1.dtsi" +/include/ "qoriq-espi-0.dtsi" + spi@110000 { + fsl,espi-num-chipselects = <4>; + }; + +/include/ "qoriq-esdhc-0.dtsi" + sdhc@114000 { + compatible = "fsl,t1040-esdhc", "fsl,esdhc"; + fsl,iommu-parent = <&pamu0>; + fsl,liodn-reg = <&guts 0x530>; /* eSDHCLIODNR */ + sdhci,auto-cmd12; + }; +/include/ "qoriq-i2c-0.dtsi" +/include/ "qoriq-i2c-1.dtsi" +/include/ "qoriq-duart-0.dtsi" +/include/ "qoriq-duart-1.dtsi" +/include/ "qoriq-gpio-0.dtsi" +/include/ "qoriq-gpio-1.dtsi" +/include/ "qoriq-gpio-2.dtsi" +/include/ "qoriq-gpio-3.dtsi" +/include/ "qoriq-usb2-mph-0.dtsi" + usb0: usb@210000 { + compatible = "fsl-usb2-mph-v2.4", "fsl-usb2-mph"; + fsl,iommu-parent = <&pamu0>; + fsl,liodn-reg = <&guts 0x520>; /* USB1LIODNR */ + phy_type = "utmi"; + port0; + }; +/include/ "qoriq-usb2-dr-0.dtsi" + usb1: usb@211000 { + compatible = "fsl-usb2-dr-v2.4", "fsl-usb2-dr"; + fsl,iommu-parent = <&pamu0>; + fsl,liodn-reg = <&guts 0x524>; /* USB2LIODNR */ + dr_mode = "host"; + phy_type = "utmi"; + }; + + display@180000 { + compatible = "fsl,t1040-diu", "fsl,diu"; + reg = <0x180000 1000>; + interrupts = <74 2 0 0>; + }; + +/include/ "qoriq-sata2-0.dtsi" + sata@220000 { + fsl,iommu-parent = <&pamu0>; + fsl,liodn-reg = <&guts 0x550>; /* SATA1LIODNR */ + }; +/include/ "qoriq-sata2-1.dtsi" + sata@221000 { + fsl,iommu-parent = <&pamu0>; + fsl,liodn-reg = <&guts 0x554>; /* SATA2LIODNR */ + }; +/include/ "qoriq-sec5.0-0.dtsi" +}; diff --git a/arch/powerpc/boot/dts/fsl/t1042si-post.dtsi b/arch/powerpc/boot/dts/fsl/t1042si-post.dtsi new file mode 100644 index 000000000000..319b74f29724 --- /dev/null +++ b/arch/powerpc/boot/dts/fsl/t1042si-post.dtsi @@ -0,0 +1,37 @@ +/* + * T1042 Silicon/SoC Device Tree Source (post include) + * + * Copyright 2013 Freescale Semiconductor Inc. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are met: + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * * Neither the name of Freescale Semiconductor nor the + * names of its contributors may be used to endorse or promote products + * derived from this software without specific prior written permission. + * + * + * ALTERNATIVELY, this software may be distributed under the terms of the + * GNU General Public License ("GPL") as published by the Free Software + * Foundation, either version 2 of that License or (at your option) any + * later version. + * + * THIS SOFTWARE IS PROVIDED BY Freescale Semiconductor ``AS IS'' AND ANY + * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED + * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE + * DISCLAIMED. IN NO EVENT SHALL Freescale Semiconductor BE LIABLE FOR ANY + * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES + * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; + * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND + * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS + * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +/include/ "t1040si-post.dtsi" + +/* Place holder for ethernet related device tree nodes */ diff --git a/arch/powerpc/boot/dts/fsl/t104xsi-pre.dtsi b/arch/powerpc/boot/dts/fsl/t104xsi-pre.dtsi new file mode 100644 index 000000000000..bbb7025ca9c2 --- /dev/null +++ b/arch/powerpc/boot/dts/fsl/t104xsi-pre.dtsi @@ -0,0 +1,104 @@ +/* + * T1040/T1042 Silicon/SoC Device Tree Source (pre include) + * + * Copyright 2013 Freescale Semiconductor Inc. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are met: + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * * Neither the name of Freescale Semiconductor nor the + * names of its contributors may be used to endorse or promote products + * derived from this software without specific prior written permission. + * + * + * ALTERNATIVELY, this software may be distributed under the terms of the + * GNU General Public License ("GPL") as published by the Free Software + * Foundation, either version 2 of that License or (at your option) any + * later version. + * + * THIS SOFTWARE IS PROVIDED BY Freescale Semiconductor ``AS IS'' AND ANY + * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED + * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE + * DISCLAIMED. IN NO EVENT SHALL Freescale Semiconductor BE LIABLE FOR ANY + * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES + * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; + * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND + * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS + * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +/dts-v1/; + +/include/ "e5500_power_isa.dtsi" + +/ { + #address-cells = <2>; + #size-cells = <2>; + interrupt-parent = <&mpic>; + + aliases { + ccsr = &soc; + dcsr = &dcsr; + + serial0 = &serial0; + serial1 = &serial1; + serial2 = &serial2; + serial3 = &serial3; + pci0 = &pci0; + pci1 = &pci1; + pci2 = &pci2; + pci3 = &pci3; + usb0 = &usb0; + usb1 = &usb1; + sdhc = &sdhc; + + crypto = &crypto; + }; + + cpus { + #address-cells = <1>; + #size-cells = <0>; + + cpu0: PowerPC,e5500@0 { + device_type = "cpu"; + reg = <0>; + clocks = <&mux0>; + next-level-cache = <&L2_1>; + L2_1: l2-cache { + next-level-cache = <&cpc>; + }; + }; + cpu1: PowerPC,e5500@1 { + device_type = "cpu"; + reg = <1>; + clocks = <&mux1>; + next-level-cache = <&L2_2>; + L2_2: l2-cache { + next-level-cache = <&cpc>; + }; + }; + cpu2: PowerPC,e5500@2 { + device_type = "cpu"; + reg = <2>; + clocks = <&mux2>; + next-level-cache = <&L2_3>; + L2_3: l2-cache { + next-level-cache = <&cpc>; + }; + }; + cpu3: PowerPC,e5500@3 { + device_type = "cpu"; + reg = <3>; + clocks = <&mux3>; + next-level-cache = <&L2_4>; + L2_4: l2-cache { + next-level-cache = <&cpc>; + }; + }; + }; +}; diff --git a/arch/powerpc/boot/dts/fsl/t4240si-post.dtsi b/arch/powerpc/boot/dts/fsl/t4240si-post.dtsi index f99d74ff11b4..793669baa13e 100644 --- a/arch/powerpc/boot/dts/fsl/t4240si-post.dtsi +++ b/arch/powerpc/boot/dts/fsl/t4240si-post.dtsi @@ -343,7 +343,7 @@ }; corenet-cf@18000 { - compatible = "fsl,corenet-cf"; + compatible = "fsl,corenet2-cf", "fsl,corenet-cf"; reg = <0x18000 0x1000>; interrupts = <16 2 1 31>; fsl,ccf-num-csdids = <32>; @@ -353,6 +353,7 @@ iommu@20000 { compatible = "fsl,pamu-v1.0", "fsl,pamu"; reg = <0x20000 0x6000>; + fsl,portid-mapping = <0x8000>; interrupts = < 24 2 0 0 16 2 1 30>; diff --git a/arch/powerpc/boot/dts/fsl/t4240si-pre.dtsi b/arch/powerpc/boot/dts/fsl/t4240si-pre.dtsi index 0b8ccc5b4a46..d2f157edbe81 100644 --- a/arch/powerpc/boot/dts/fsl/t4240si-pre.dtsi +++ b/arch/powerpc/boot/dts/fsl/t4240si-pre.dtsi @@ -69,72 +69,84 @@ reg = <0 1>; clocks = <&mux0>; next-level-cache = <&L2_1>; + fsl,portid-mapping = <0x80000000>; }; cpu1: PowerPC,e6500@2 { device_type = "cpu"; reg = <2 3>; clocks = <&mux0>; next-level-cache = <&L2_1>; + fsl,portid-mapping = <0x80000000>; }; cpu2: PowerPC,e6500@4 { device_type = "cpu"; reg = <4 5>; clocks = <&mux0>; next-level-cache = <&L2_1>; + fsl,portid-mapping = <0x80000000>; }; cpu3: PowerPC,e6500@6 { device_type = "cpu"; reg = <6 7>; clocks = <&mux0>; next-level-cache = <&L2_1>; + fsl,portid-mapping = <0x80000000>; }; cpu4: PowerPC,e6500@8 { device_type = "cpu"; reg = <8 9>; clocks = <&mux1>; next-level-cache = <&L2_2>; + fsl,portid-mapping = <0x40000000>; }; cpu5: PowerPC,e6500@10 { device_type = "cpu"; reg = <10 11>; clocks = <&mux1>; next-level-cache = <&L2_2>; + fsl,portid-mapping = <0x40000000>; }; cpu6: PowerPC,e6500@12 { device_type = "cpu"; reg = <12 13>; clocks = <&mux1>; next-level-cache = <&L2_2>; + fsl,portid-mapping = <0x40000000>; }; cpu7: PowerPC,e6500@14 { device_type = "cpu"; reg = <14 15>; clocks = <&mux1>; next-level-cache = <&L2_2>; + fsl,portid-mapping = <0x40000000>; }; cpu8: PowerPC,e6500@16 { device_type = "cpu"; reg = <16 17>; clocks = <&mux2>; next-level-cache = <&L2_3>; + fsl,portid-mapping = <0x20000000>; }; cpu9: PowerPC,e6500@18 { device_type = "cpu"; reg = <18 19>; clocks = <&mux2>; next-level-cache = <&L2_3>; + fsl,portid-mapping = <0x20000000>; }; cpu10: PowerPC,e6500@20 { device_type = "cpu"; reg = <20 21>; clocks = <&mux2>; next-level-cache = <&L2_3>; + fsl,portid-mapping = <0x20000000>; }; cpu11: PowerPC,e6500@22 { device_type = "cpu"; reg = <22 23>; clocks = <&mux2>; next-level-cache = <&L2_3>; + fsl,portid-mapping = <0x20000000>; }; }; }; diff --git a/arch/powerpc/boot/dts/kmcoge4.dts b/arch/powerpc/boot/dts/kmcoge4.dts new file mode 100644 index 000000000000..89b4119f3b19 --- /dev/null +++ b/arch/powerpc/boot/dts/kmcoge4.dts @@ -0,0 +1,152 @@ +/* + * Keymile kmcoge4 Device Tree Source, based on the P2041RDB DTS + * + * (C) Copyright 2014 + * Valentin Longchamp, Keymile AG, valentin.longchamp@keymile.com + * + * Copyright 2011 Freescale Semiconductor Inc. + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the + * Free Software Foundation; either version 2 of the License, or (at your + * option) any later version. + */ + +/include/ "fsl/p2041si-pre.dtsi" + +/ { + model = "keymile,kmcoge4"; + compatible = "keymile,kmcoge4", "keymile,kmp204x"; + #address-cells = <2>; + #size-cells = <2>; + interrupt-parent = <&mpic>; + + memory { + device_type = "memory"; + }; + + dcsr: dcsr@f00000000 { + ranges = <0x00000000 0xf 0x00000000 0x01008000>; + }; + + soc: soc@ffe000000 { + ranges = <0x00000000 0xf 0xfe000000 0x1000000>; + reg = <0xf 0xfe000000 0 0x00001000>; + spi@110000 { + flash@0 { + #address-cells = <1>; + #size-cells = <1>; + compatible = "spansion,s25fl256s1"; + reg = <0>; + spi-max-frequency = <20000000>; /* input clock */ + }; + + network_clock@1 { + compatible = "zarlink,zl30343"; + reg = <1>; + spi-max-frequency = <8000000>; + }; + + flash@2 { + #address-cells = <1>; + #size-cells = <1>; + compatible = "micron,m25p32"; + reg = <2>; + spi-max-frequency = <15000000>; + }; + }; + + i2c@119000 { + status = "disabled"; + }; + + i2c@119100 { + status = "disabled"; + }; + + usb0: usb@210000 { + status = "disabled"; + }; + + usb1: usb@211000 { + status = "disabled"; + }; + + sata@220000 { + status = "disabled"; + }; + + sata@221000 { + status = "disabled"; + }; + }; + + rio: rapidio@ffe0c0000 { + status = "disabled"; + }; + + lbc: localbus@ffe124000 { + reg = <0xf 0xfe124000 0 0x1000>; + ranges = <0 0 0xf 0xffa00000 0x00040000 /* LB 0 */ + 1 0 0xf 0xfb000000 0x00010000 /* LB 1 */ + 2 0 0xf 0xd0000000 0x10000000 /* LB 2 */ + 3 0 0xf 0xe0000000 0x10000000>; /* LB 3 */ + + nand@0,0 { + #address-cells = <1>; + #size-cells = <1>; + compatible = "fsl,elbc-fcm-nand"; + reg = <0 0 0x40000>; + }; + + board-control@1,0 { + compatible = "keymile,qriox"; + reg = <1 0 0x80>; + }; + + chassis-mgmt@3,0 { + compatible = "keymile,bfticu"; + interrupt-controller; + #interrupt-cells = <2>; + reg = <3 0 0x100>; + interrupt-parent = <&mpic>; + interrupts = <6 1 0 0>; + }; + }; + + pci0: pcie@ffe200000 { + reg = <0xf 0xfe200000 0 0x1000>; + ranges = <0x02000000 0 0xe0000000 0xc 0x00000000 0x0 0x20000000 + 0x01000000 0 0x00000000 0xf 0xf8000000 0x0 0x00010000>; + pcie@0 { + ranges = <0x02000000 0 0xe0000000 + 0x02000000 0 0xe0000000 + 0 0x20000000 + + 0x01000000 0 0x00000000 + 0x01000000 0 0x00000000 + 0 0x00010000>; + }; + }; + + pci1: pcie@ffe201000 { + status = "disabled"; + }; + + pci2: pcie@ffe202000 { + reg = <0xf 0xfe202000 0 0x1000>; + ranges = <0x02000000 0 0xe0000000 0xc 0x20000000 0 0x20000000 + 0x01000000 0 0x00000000 0xf 0xf8010000 0 0x00010000>; + pcie@0 { + ranges = <0x02000000 0 0xe0000000 + 0x02000000 0 0xe0000000 + 0 0x20000000 + + 0x01000000 0 0x00000000 + 0x01000000 0 0x00000000 + 0 0x00010000>; + }; + }; +}; + +/include/ "fsl/p2041si-post.dtsi" diff --git a/arch/powerpc/boot/dts/mpc8308_p1m.dts b/arch/powerpc/boot/dts/mpc8308_p1m.dts index 651e4f55acdb..57f86cdf9f36 100644 --- a/arch/powerpc/boot/dts/mpc8308_p1m.dts +++ b/arch/powerpc/boot/dts/mpc8308_p1m.dts @@ -296,7 +296,7 @@ }; dma@2c000 { - compatible = "fsl,mpc8308-dma", "fsl,mpc5121-dma"; + compatible = "fsl,mpc8308-dma"; reg = <0x2c000 0x1800>; interrupts = <3 0x8 94 0x8>; diff --git a/arch/powerpc/boot/dts/mpc8308rdb.dts b/arch/powerpc/boot/dts/mpc8308rdb.dts index 9ce45f2efd34..d0211f0413c6 100644 --- a/arch/powerpc/boot/dts/mpc8308rdb.dts +++ b/arch/powerpc/boot/dts/mpc8308rdb.dts @@ -265,7 +265,7 @@ }; dma@2c000 { - compatible = "fsl,mpc8308-dma", "fsl,mpc5121-dma"; + compatible = "fsl,mpc8308-dma"; reg = <0x2c000 0x1800>; interrupts = <3 0x8 94 0x8>; diff --git a/arch/powerpc/boot/dts/oca4080.dts b/arch/powerpc/boot/dts/oca4080.dts new file mode 100644 index 000000000000..3d4c751d1608 --- /dev/null +++ b/arch/powerpc/boot/dts/oca4080.dts @@ -0,0 +1,118 @@ +/* + * OCA4080 Device Tree Source + * + * Copyright 2014 Prodrive Technologies B.V. + * + * Based on: + * P4080DS Device Tree Source + * Copyright 2009-2011 Freescale Semiconductor Inc. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are met: + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * * Neither the name of Freescale Semiconductor nor the + * names of its contributors may be used to endorse or promote products + * derived from this software without specific prior written permission. + * + * + * ALTERNATIVELY, this software may be distributed under the terms of the + * GNU General Public License ("GPL") as published by the Free Software + * Foundation, either version 2 of that License or (at your option) any + * later version. + * + * THIS SOFTWARE IS PROVIDED BY Freescale Semiconductor ``AS IS'' AND ANY + * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED + * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE + * DISCLAIMED. IN NO EVENT SHALL Freescale Semiconductor BE LIABLE FOR ANY + * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES + * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; + * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND + * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS + * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +/include/ "fsl/p4080si-pre.dtsi" + +/ { + model = "fsl,OCA4080"; + compatible = "fsl,OCA4080"; + #address-cells = <2>; + #size-cells = <2>; + interrupt-parent = <&mpic>; + + memory { + device_type = "memory"; + }; + + dcsr: dcsr@f00000000 { + ranges = <0x00000000 0xf 0x00000000 0x01008000>; + }; + + soc: soc@ffe000000 { + ranges = <0x00000000 0xf 0xfe000000 0x1000000>; + reg = <0xf 0xfe000000 0 0x00001000>; + + i2c@118000 { + status = "disabled"; + }; + + i2c@118100 { + status = "disabled"; + }; + + i2c@119000 { + status = "disabled"; + }; + + i2c@119100 { + status = "disabled"; + }; + + usb0: usb@210000 { + status = "disabled"; + }; + + usb1: usb@211000 { + status = "disabled"; + }; + }; + + rio: rapidio@ffe0c0000 { + reg = <0xf 0xfe0c0000 0 0x11000>; + + port1 { + ranges = <0 0 0xc 0x20000000 0 0x10000000>; + }; + }; + + lbc: localbus@ffe124000 { + reg = <0xf 0xfe124000 0 0x1000>; + ranges = <0 0 0xf 0xef800000 0x800000>; + + flash@0,0 { + compatible = "cfi-flash"; + reg = <0 0 0x00800000>; + bank-width = <2>; + device-width = <2>; + }; + }; + + pci0: pcie@ffe200000 { + status = "disabled"; + }; + + pci1: pcie@ffe201000 { + status = "disabled"; + }; + + pci2: pcie@ffe202000 { + status = "disabled"; + }; +}; + +/include/ "fsl/p4080si-post.dtsi" diff --git a/arch/powerpc/boot/dts/p1023rds.dts b/arch/powerpc/boot/dts/p1023rds.dts deleted file mode 100644 index beb6cb12e59d..000000000000 --- a/arch/powerpc/boot/dts/p1023rds.dts +++ /dev/null @@ -1,219 +0,0 @@ -/* - * P1023 RDS Device Tree Source - * - * Copyright 2010-2011 Freescale Semiconductor Inc. - * - * Author: Roy Zang <tie-fei.zang@freescale.com> - * - * Redistribution and use in source and binary forms, with or without - * modification, are permitted provided that the following conditions are met: - * * Redistributions of source code must retain the above copyright - * notice, this list of conditions and the following disclaimer. - * * Redistributions in binary form must reproduce the above copyright - * notice, this list of conditions and the following disclaimer in the - * documentation and/or other materials provided with the distribution. - * * Neither the name of Freescale Semiconductor nor the - * names of its contributors may be used to endorse or promote products - * derived from this software without specific prior written permission. - * - * - * ALTERNATIVELY, this software may be distributed under the terms of the - * GNU General Public License ("GPL") as published by the Free Software - * Foundation, either version 2 of that License or (at your option) any - * later version. - * - * THIS SOFTWARE IS PROVIDED BY Freescale Semiconductor ``AS IS'' AND ANY - * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED - * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE - * DISCLAIMED. IN NO EVENT SHALL Freescale Semiconductor BE LIABLE FOR ANY - * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES - * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; - * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND - * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT - * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS - * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - */ - -/include/ "fsl/p1023si-pre.dtsi" - -/ { - model = "fsl,P1023"; - compatible = "fsl,P1023RDS"; - #address-cells = <2>; - #size-cells = <2>; - interrupt-parent = <&mpic>; - - memory { - device_type = "memory"; - }; - - soc: soc@ff600000 { - ranges = <0x0 0x0 0xff600000 0x200000>; - - i2c@3000 { - rtc@68 { - compatible = "dallas,ds1374"; - reg = <0x68>; - }; - }; - - spi@7000 { - fsl_dataflash@0 { - #address-cells = <1>; - #size-cells = <1>; - compatible = "atmel,at45db081d"; - reg = <0>; - spi-max-frequency = <40000000>; /* input clock */ - partition@u-boot { - /* 512KB for u-boot Bootloader Image */ - label = "u-boot-spi"; - reg = <0x00000000 0x00080000>; - read-only; - }; - partition@dtb { - /* 512KB for DTB Image */ - label = "dtb-spi"; - reg = <0x00080000 0x00080000>; - read-only; - }; - }; - }; - - usb@22000 { - dr_mode = "host"; - phy_type = "ulpi"; - }; - }; - - lbc: localbus@ff605000 { - reg = <0 0xff605000 0 0x1000>; - - /* NOR Flash, BCSR */ - ranges = <0x0 0x0 0x0 0xee000000 0x02000000 - 0x1 0x0 0x0 0xe0000000 0x00008000>; - - nor@0,0 { - #address-cells = <1>; - #size-cells = <1>; - compatible = "cfi-flash"; - reg = <0x0 0x0 0x02000000>; - bank-width = <2>; - device-width = <1>; - partition@0 { - label = "ramdisk"; - reg = <0x00000000 0x01c00000>; - }; - partition@1c00000 { - label = "kernel"; - reg = <0x01c00000 0x002e0000>; - }; - partiton@1ee0000 { - label = "dtb"; - reg = <0x01ee0000 0x00020000>; - }; - partition@1f00000 { - label = "firmware"; - reg = <0x01f00000 0x00080000>; - read-only; - }; - partition@1f80000 { - label = "u-boot"; - reg = <0x01f80000 0x00080000>; - read-only; - }; - }; - - fpga@1,0 { - #address-cells = <1>; - #size-cells = <1>; - compatible = "fsl,p1023rds-fpga"; - reg = <1 0 0x8000>; - ranges = <0 1 0 0x8000>; - - bcsr@20 { - compatible = "fsl,p1023rds-bcsr"; - reg = <0x20 0x20>; - }; - }; - }; - - pci0: pcie@ff60a000 { - reg = <0 0xff60a000 0 0x1000>; - ranges = <0x2000000 0x0 0xc0000000 0 0xc0000000 0x0 0x20000000 - 0x1000000 0x0 0x00000000 0 0xffc20000 0x0 0x10000>; - pcie@0 { - /* IRQ[0:3] are pulled up on board, set to active-low */ - interrupt-map-mask = <0xf800 0 0 7>; - interrupt-map = < - /* IDSEL 0x0 */ - 0000 0 0 1 &mpic 0 1 0 0 - 0000 0 0 2 &mpic 1 1 0 0 - 0000 0 0 3 &mpic 2 1 0 0 - 0000 0 0 4 &mpic 3 1 0 0 - >; - ranges = <0x2000000 0x0 0xc0000000 - 0x2000000 0x0 0xc0000000 - 0x0 0x20000000 - - 0x1000000 0x0 0x0 - 0x1000000 0x0 0x0 - 0x0 0x100000>; - }; - }; - - board_pci1: pci1: pcie@ff609000 { - reg = <0 0xff609000 0 0x1000>; - ranges = <0x2000000 0x0 0xa0000000 0 0xa0000000 0x0 0x20000000 - 0x1000000 0x0 0x00000000 0 0xffc10000 0x0 0x10000>; - pcie@0 { - /* - * IRQ[4:6] only for PCIe, set to active-high, - * IRQ[7] is pulled up on board, set to active-low - */ - interrupt-map-mask = <0xf800 0 0 7>; - interrupt-map = < - /* IDSEL 0x0 */ - 0000 0 0 1 &mpic 4 2 0 0 - 0000 0 0 2 &mpic 5 2 0 0 - 0000 0 0 3 &mpic 6 2 0 0 - 0000 0 0 4 &mpic 7 1 0 0 - >; - ranges = <0x2000000 0x0 0xa0000000 - 0x2000000 0x0 0xa0000000 - 0x0 0x20000000 - - 0x1000000 0x0 0x0 - 0x1000000 0x0 0x0 - 0x0 0x100000>; - }; - }; - - pci2: pcie@ff60b000 { - reg = <0 0xff60b000 0 0x1000>; - ranges = <0x2000000 0x0 0x80000000 0 0x80000000 0x0 0x20000000 - 0x1000000 0x0 0x00000000 0 0xffc00000 0x0 0x10000>; - pcie@0 { - /* - * IRQ[8:10] are pulled up on board, set to active-low - * IRQ[11] only for PCIe, set to active-high, - */ - interrupt-map-mask = <0xf800 0 0 7>; - interrupt-map = < - /* IDSEL 0x0 */ - 0000 0 0 1 &mpic 8 1 0 0 - 0000 0 0 2 &mpic 9 1 0 0 - 0000 0 0 3 &mpic 10 1 0 0 - 0000 0 0 4 &mpic 11 2 0 0 - >; - ranges = <0x2000000 0x0 0x80000000 - 0x2000000 0x0 0x80000000 - 0x0 0x20000000 - - 0x1000000 0x0 0x0 - 0x1000000 0x0 0x0 - 0x0 0x100000>; - }; - }; -}; - -/include/ "fsl/p1023si-post.dtsi" diff --git a/arch/powerpc/boot/dts/t1040qds.dts b/arch/powerpc/boot/dts/t1040qds.dts new file mode 100644 index 000000000000..973c29c2f56e --- /dev/null +++ b/arch/powerpc/boot/dts/t1040qds.dts @@ -0,0 +1,46 @@ +/* + * T1040QDS Device Tree Source + * + * Copyright 2013 Freescale Semiconductor Inc. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are met: + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * * Neither the name of Freescale Semiconductor nor the + * names of its contributors may be used to endorse or promote products + * derived from this software without specific prior written permission. + * + * + * ALTERNATIVELY, this software may be distributed under the terms of the + * GNU General Public License ("GPL") as published by the Free Software + * Foundation, either version 2 of that License or (at your option) any + * later version. + * + * THIS SOFTWARE IS PROVIDED BY Freescale Semiconductor "AS IS" AND ANY + * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED + * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE + * DISCLAIMED. IN NO EVENT SHALL Freescale Semiconductor BE LIABLE FOR ANY + * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES + * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; + * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND + * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS + * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +/include/ "fsl/t104xsi-pre.dtsi" +/include/ "t104xqds.dtsi" + +/ { + model = "fsl,T1040QDS"; + compatible = "fsl,T1040QDS"; + #address-cells = <2>; + #size-cells = <2>; + interrupt-parent = <&mpic>; +}; + +/include/ "fsl/t1040si-post.dtsi" diff --git a/arch/powerpc/boot/dts/t1042qds.dts b/arch/powerpc/boot/dts/t1042qds.dts new file mode 100644 index 000000000000..45bd03752154 --- /dev/null +++ b/arch/powerpc/boot/dts/t1042qds.dts @@ -0,0 +1,46 @@ +/* + * T1042QDS Device Tree Source + * + * Copyright 2013 Freescale Semiconductor Inc. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are met: + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * * Neither the name of Freescale Semiconductor nor the + * names of its contributors may be used to endorse or promote products + * derived from this software without specific prior written permission. + * + * + * ALTERNATIVELY, this software may be distributed under the terms of the + * GNU General Public License ("GPL") as published by the Free Software + * Foundation, either version 2 of that License or (at your option) any + * later version. + * + * THIS SOFTWARE IS PROVIDED BY Freescale Semiconductor "AS IS" AND ANY + * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED + * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE + * DISCLAIMED. IN NO EVENT SHALL Freescale Semiconductor BE LIABLE FOR ANY + * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES + * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; + * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND + * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS + * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +/include/ "fsl/t104xsi-pre.dtsi" +/include/ "t104xqds.dtsi" + +/ { + model = "fsl,T1042QDS"; + compatible = "fsl,T1042QDS"; + #address-cells = <2>; + #size-cells = <2>; + interrupt-parent = <&mpic>; +}; + +/include/ "fsl/t1042si-post.dtsi" diff --git a/arch/powerpc/boot/dts/t104xqds.dtsi b/arch/powerpc/boot/dts/t104xqds.dtsi new file mode 100644 index 000000000000..234f4b596c5b --- /dev/null +++ b/arch/powerpc/boot/dts/t104xqds.dtsi @@ -0,0 +1,166 @@ +/* + * T104xQDS Device Tree Source + * + * Copyright 2013 Freescale Semiconductor Inc. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are met: + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * * Neither the name of Freescale Semiconductor nor the + * names of its contributors may be used to endorse or promote products + * derived from this software without specific prior written permission. + * + * + * ALTERNATIVELY, this software may be distributed under the terms of the + * GNU General Public License ("GPL") as published by the Free Software + * Foundation, either version 2 of that License or (at your option) any + * later version. + * + * THIS SOFTWARE IS PROVIDED BY Freescale Semiconductor "AS IS" AND ANY + * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED + * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE + * DISCLAIMED. IN NO EVENT SHALL Freescale Semiconductor BE LIABLE FOR ANY + * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES + * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; + * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND + * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS + * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +/ { + model = "fsl,T1040QDS"; + #address-cells = <2>; + #size-cells = <2>; + interrupt-parent = <&mpic>; + + ifc: localbus@ffe124000 { + reg = <0xf 0xfe124000 0 0x2000>; + ranges = <0 0 0xf 0xe8000000 0x08000000 + 2 0 0xf 0xff800000 0x00010000 + 3 0 0xf 0xffdf0000 0x00008000>; + + nor@0,0 { + #address-cells = <1>; + #size-cells = <1>; + compatible = "cfi-flash"; + reg = <0x0 0x0 0x8000000>; + + bank-width = <2>; + device-width = <1>; + }; + + nand@2,0 { + #address-cells = <1>; + #size-cells = <1>; + compatible = "fsl,ifc-nand"; + reg = <0x2 0x0 0x10000>; + }; + + board-control@3,0 { + #address-cells = <1>; + #size-cells = <1>; + compatible = "fsl,fpga-qixis"; + reg = <3 0 0x300>; + }; + }; + + memory { + device_type = "memory"; + }; + + dcsr: dcsr@f00000000 { + ranges = <0x00000000 0xf 0x00000000 0x01072000>; + }; + + soc: soc@ffe000000 { + ranges = <0x00000000 0xf 0xfe000000 0x1000000>; + reg = <0xf 0xfe000000 0 0x00001000>; + + spi@110000 { + flash@0 { + #address-cells = <1>; + #size-cells = <1>; + compatible = "micron,n25q128a11"; + reg = <0>; + spi-max-frequency = <10000000>; /* input clock */ + }; + }; + + i2c@118000 { + pca9547@77 { + compatible = "philips,pca9547"; + reg = <0x77>; + }; + rtc@68 { + compatible = "dallas,ds3232"; + reg = <0x68>; + interrupts = <0x1 0x1 0 0>; + }; + }; + }; + + pci0: pcie@ffe240000 { + reg = <0xf 0xfe240000 0 0x10000>; + ranges = <0x02000000 0 0xe0000000 0xc 0x00000000 0x0 0x10000000 + 0x01000000 0 0x00000000 0xf 0xf8000000 0x0 0x00010000>; + pcie@0 { + ranges = <0x02000000 0 0xe0000000 + 0x02000000 0 0xe0000000 + 0 0x10000000 + + 0x01000000 0 0x00000000 + 0x01000000 0 0x00000000 + 0 0x00010000>; + }; + }; + + pci1: pcie@ffe250000 { + reg = <0xf 0xfe250000 0 0x10000>; + ranges = <0x02000000 0x0 0xe0000000 0xc 0x10000000 0x0 0x10000000 + 0x01000000 0x0 0x00000000 0xf 0xf8010000 0x0 0x00010000>; + pcie@0 { + ranges = <0x02000000 0 0xe0000000 + 0x02000000 0 0xe0000000 + 0 0x10000000 + + 0x01000000 0 0x00000000 + 0x01000000 0 0x00000000 + 0 0x00010000>; + }; + }; + + pci2: pcie@ffe260000 { + reg = <0xf 0xfe260000 0 0x10000>; + ranges = <0x02000000 0 0xe0000000 0xc 0x20000000 0 0x10000000 + 0x01000000 0 0x00000000 0xf 0xf8020000 0 0x00010000>; + pcie@0 { + ranges = <0x02000000 0 0xe0000000 + 0x02000000 0 0xe0000000 + 0 0x10000000 + + 0x01000000 0 0x00000000 + 0x01000000 0 0x00000000 + 0 0x00010000>; + }; + }; + + pci3: pcie@ffe270000 { + reg = <0xf 0xfe270000 0 0x10000>; + ranges = <0x02000000 0 0xe0000000 0xc 0x30000000 0 0x10000000 + 0x01000000 0 0x00000000 0xf 0xf8030000 0 0x00010000>; + pcie@0 { + ranges = <0x02000000 0 0xe0000000 + 0x02000000 0 0xe0000000 + 0 0x10000000 + + 0x01000000 0 0x00000000 + 0x01000000 0 0x00000000 + 0 0x00010000>; + }; + }; +}; diff --git a/arch/powerpc/boot/dts/t4240emu.dts b/arch/powerpc/boot/dts/t4240emu.dts index ee24ab335598..bc12127a03fb 100644 --- a/arch/powerpc/boot/dts/t4240emu.dts +++ b/arch/powerpc/boot/dts/t4240emu.dts @@ -60,63 +60,75 @@ device_type = "cpu"; reg = <0 1>; next-level-cache = <&L2_1>; + fsl,portid-mapping = <0x80000000>; }; cpu1: PowerPC,e6500@2 { device_type = "cpu"; reg = <2 3>; next-level-cache = <&L2_1>; + fsl,portid-mapping = <0x80000000>; }; cpu2: PowerPC,e6500@4 { device_type = "cpu"; reg = <4 5>; next-level-cache = <&L2_1>; + fsl,portid-mapping = <0x80000000>; }; cpu3: PowerPC,e6500@6 { device_type = "cpu"; reg = <6 7>; next-level-cache = <&L2_1>; + fsl,portid-mapping = <0x80000000>; }; cpu4: PowerPC,e6500@8 { device_type = "cpu"; reg = <8 9>; next-level-cache = <&L2_2>; + fsl,portid-mapping = <0x40000000>; }; cpu5: PowerPC,e6500@10 { device_type = "cpu"; reg = <10 11>; next-level-cache = <&L2_2>; + fsl,portid-mapping = <0x40000000>; }; cpu6: PowerPC,e6500@12 { device_type = "cpu"; reg = <12 13>; next-level-cache = <&L2_2>; + fsl,portid-mapping = <0x40000000>; }; cpu7: PowerPC,e6500@14 { device_type = "cpu"; reg = <14 15>; next-level-cache = <&L2_2>; + fsl,portid-mapping = <0x40000000>; }; cpu8: PowerPC,e6500@16 { device_type = "cpu"; reg = <16 17>; next-level-cache = <&L2_3>; + fsl,portid-mapping = <0x20000000>; }; cpu9: PowerPC,e6500@18 { device_type = "cpu"; reg = <18 19>; next-level-cache = <&L2_3>; + fsl,portid-mapping = <0x20000000>; }; cpu10: PowerPC,e6500@20 { device_type = "cpu"; reg = <20 21>; next-level-cache = <&L2_3>; + fsl,portid-mapping = <0x20000000>; }; cpu11: PowerPC,e6500@22 { device_type = "cpu"; reg = <22 23>; next-level-cache = <&L2_3>; + fsl,portid-mapping = <0x20000000>; }; }; }; @@ -213,7 +225,7 @@ }; corenet-cf@18000 { - compatible = "fsl,corenet-cf"; + compatible = "fsl,corenet2-cf", "fsl,corenet-cf"; reg = <0x18000 0x1000>; interrupts = <16 2 1 31>; fsl,ccf-num-csdids = <32>; @@ -223,6 +235,7 @@ iommu@20000 { compatible = "fsl,pamu-v1.0", "fsl,pamu"; reg = <0x20000 0x6000>; + fsl,portid-mapping = <0x8000>; interrupts = < 24 2 0 0 16 2 1 30>; diff --git a/arch/powerpc/boot/elf_util.c b/arch/powerpc/boot/elf_util.c index 1567a0c0f05c..316552dea4d8 100644 --- a/arch/powerpc/boot/elf_util.c +++ b/arch/powerpc/boot/elf_util.c @@ -26,7 +26,11 @@ int parse_elf64(void *hdr, struct elf_info *info) elf64->e_ident[EI_MAG2] == ELFMAG2 && elf64->e_ident[EI_MAG3] == ELFMAG3 && elf64->e_ident[EI_CLASS] == ELFCLASS64 && +#ifdef __LITTLE_ENDIAN__ + elf64->e_ident[EI_DATA] == ELFDATA2LSB && +#else elf64->e_ident[EI_DATA] == ELFDATA2MSB && +#endif (elf64->e_type == ET_EXEC || elf64->e_type == ET_DYN) && elf64->e_machine == EM_PPC64)) diff --git a/arch/powerpc/boot/of.c b/arch/powerpc/boot/of.c index 62e2f43ec1df..7ca910cb2fc6 100644 --- a/arch/powerpc/boot/of.c +++ b/arch/powerpc/boot/of.c @@ -40,8 +40,8 @@ static void *of_try_claim(unsigned long size) #ifdef DEBUG printf(" trying: 0x%08lx\n\r", claim_base); #endif - addr = (unsigned long)of_claim(claim_base, size, 0); - if ((void *)addr != (void *)-1) + addr = (unsigned long) of_claim(claim_base, size, 0); + if (addr != PROM_ERROR) break; } if (addr == 0) diff --git a/arch/powerpc/boot/of.h b/arch/powerpc/boot/of.h index e4c68f7391c5..c8c1750aba0c 100644 --- a/arch/powerpc/boot/of.h +++ b/arch/powerpc/boot/of.h @@ -1,12 +1,15 @@ #ifndef _PPC_BOOT_OF_H_ #define _PPC_BOOT_OF_H_ +#include "swab.h" + typedef void *phandle; -typedef void *ihandle; +typedef u32 ihandle; void of_init(void *promptr); int of_call_prom(const char *service, int nargs, int nret, ...); -void *of_claim(unsigned long virt, unsigned long size, unsigned long align); +unsigned int of_claim(unsigned long virt, unsigned long size, + unsigned long align); void *of_vmlinux_alloc(unsigned long size); void of_exit(void); void *of_finddevice(const char *name); @@ -18,4 +21,16 @@ int of_setprop(const void *phandle, const char *name, const void *buf, /* Console functions */ void of_console_init(void); +typedef u32 __be32; + +#ifdef __LITTLE_ENDIAN__ +#define cpu_to_be32(x) swab32(x) +#define be32_to_cpu(x) swab32(x) +#else +#define cpu_to_be32(x) (x) +#define be32_to_cpu(x) (x) +#endif + +#define PROM_ERROR (-1u) + #endif /* _PPC_BOOT_OF_H_ */ diff --git a/arch/powerpc/boot/ofconsole.c b/arch/powerpc/boot/ofconsole.c index ce0e02424453..8b754702460a 100644 --- a/arch/powerpc/boot/ofconsole.c +++ b/arch/powerpc/boot/ofconsole.c @@ -18,7 +18,7 @@ #include "of.h" -static void *of_stdout_handle; +static unsigned int of_stdout_handle; static int of_console_open(void) { @@ -27,8 +27,10 @@ static int of_console_open(void) if (((devp = of_finddevice("/chosen")) != NULL) && (of_getprop(devp, "stdout", &of_stdout_handle, sizeof(of_stdout_handle)) - == sizeof(of_stdout_handle))) + == sizeof(of_stdout_handle))) { + of_stdout_handle = be32_to_cpu(of_stdout_handle); return 0; + } return -1; } diff --git a/arch/powerpc/boot/oflib.c b/arch/powerpc/boot/oflib.c index b0ec9cf3eaaf..46c98a47d949 100644 --- a/arch/powerpc/boot/oflib.c +++ b/arch/powerpc/boot/oflib.c @@ -16,74 +16,83 @@ #include "of.h" +typedef u32 prom_arg_t; + +/* The following structure is used to communicate with open firmware. + * All arguments in and out are in big endian format. */ +struct prom_args { + __be32 service; /* Address of service name string. */ + __be32 nargs; /* Number of input arguments. */ + __be32 nret; /* Number of output arguments. */ + __be32 args[10]; /* Input/output arguments. */ +}; + +#ifdef __powerpc64__ +extern int prom(void *); +#else static int (*prom) (void *); +#endif void of_init(void *promptr) { +#ifndef __powerpc64__ prom = (int (*)(void *))promptr; +#endif } +#define ADDR(x) (u32)(unsigned long)(x) + int of_call_prom(const char *service, int nargs, int nret, ...) { int i; - struct prom_args { - const char *service; - int nargs; - int nret; - unsigned int args[12]; - } args; + struct prom_args args; va_list list; - args.service = service; - args.nargs = nargs; - args.nret = nret; + args.service = cpu_to_be32(ADDR(service)); + args.nargs = cpu_to_be32(nargs); + args.nret = cpu_to_be32(nret); va_start(list, nret); for (i = 0; i < nargs; i++) - args.args[i] = va_arg(list, unsigned int); + args.args[i] = cpu_to_be32(va_arg(list, prom_arg_t)); va_end(list); for (i = 0; i < nret; i++) args.args[nargs+i] = 0; if (prom(&args) < 0) - return -1; + return PROM_ERROR; - return (nret > 0)? args.args[nargs]: 0; + return (nret > 0) ? be32_to_cpu(args.args[nargs]) : 0; } static int of_call_prom_ret(const char *service, int nargs, int nret, - unsigned int *rets, ...) + prom_arg_t *rets, ...) { int i; - struct prom_args { - const char *service; - int nargs; - int nret; - unsigned int args[12]; - } args; + struct prom_args args; va_list list; - args.service = service; - args.nargs = nargs; - args.nret = nret; + args.service = cpu_to_be32(ADDR(service)); + args.nargs = cpu_to_be32(nargs); + args.nret = cpu_to_be32(nret); va_start(list, rets); for (i = 0; i < nargs; i++) - args.args[i] = va_arg(list, unsigned int); + args.args[i] = cpu_to_be32(va_arg(list, prom_arg_t)); va_end(list); for (i = 0; i < nret; i++) args.args[nargs+i] = 0; if (prom(&args) < 0) - return -1; + return PROM_ERROR; - if (rets != (void *) 0) + if (rets != NULL) for (i = 1; i < nret; ++i) - rets[i-1] = args.args[nargs+i]; + rets[i-1] = be32_to_cpu(args.args[nargs+i]); - return (nret > 0)? args.args[nargs]: 0; + return (nret > 0) ? be32_to_cpu(args.args[nargs]) : 0; } /* returns true if s2 is a prefix of s1 */ @@ -103,7 +112,7 @@ static int string_match(const char *s1, const char *s2) */ static int need_map = -1; static ihandle chosen_mmu; -static phandle memory; +static ihandle memory; static int check_of_version(void) { @@ -132,10 +141,10 @@ static int check_of_version(void) printf("no mmu\n"); return 0; } - memory = (ihandle) of_call_prom("open", 1, 1, "/memory"); - if (memory == (ihandle) -1) { - memory = (ihandle) of_call_prom("open", 1, 1, "/memory@0"); - if (memory == (ihandle) -1) { + memory = of_call_prom("open", 1, 1, "/memory"); + if (memory == PROM_ERROR) { + memory = of_call_prom("open", 1, 1, "/memory@0"); + if (memory == PROM_ERROR) { printf("no memory node\n"); return 0; } @@ -144,40 +153,41 @@ static int check_of_version(void) return 1; } -void *of_claim(unsigned long virt, unsigned long size, unsigned long align) +unsigned int of_claim(unsigned long virt, unsigned long size, + unsigned long align) { int ret; - unsigned int result; + prom_arg_t result; if (need_map < 0) need_map = check_of_version(); if (align || !need_map) - return (void *) of_call_prom("claim", 3, 1, virt, size, align); + return of_call_prom("claim", 3, 1, virt, size, align); ret = of_call_prom_ret("call-method", 5, 2, &result, "claim", memory, align, size, virt); if (ret != 0 || result == -1) - return (void *) -1; + return -1; ret = of_call_prom_ret("call-method", 5, 2, &result, "claim", chosen_mmu, align, size, virt); /* 0x12 == coherent + read/write */ ret = of_call_prom("call-method", 6, 1, "map", chosen_mmu, 0x12, size, virt, virt); - return (void *) virt; + return virt; } void *of_vmlinux_alloc(unsigned long size) { unsigned long start = (unsigned long)_start, end = (unsigned long)_end; - void *addr; + unsigned long addr; void *p; /* With some older POWER4 firmware we need to claim the area the kernel * will reside in. Newer firmwares don't need this so we just ignore * the return value. */ - addr = of_claim(start, end - start, 0); - printf("Trying to claim from 0x%lx to 0x%lx (0x%lx) got %p\r\n", + addr = (unsigned long) of_claim(start, end - start, 0); + printf("Trying to claim from 0x%lx to 0x%lx (0x%lx) got %lx\r\n", start, end, end - start, addr); p = malloc(size); @@ -197,7 +207,7 @@ void of_exit(void) */ void *of_finddevice(const char *name) { - return (phandle) of_call_prom("finddevice", 1, 1, name); + return (void *) (unsigned long) of_call_prom("finddevice", 1, 1, name); } int of_getprop(const void *phandle, const char *name, void *buf, diff --git a/arch/powerpc/boot/ppc_asm.h b/arch/powerpc/boot/ppc_asm.h index eb0e98be69e0..35ea60c1f070 100644 --- a/arch/powerpc/boot/ppc_asm.h +++ b/arch/powerpc/boot/ppc_asm.h @@ -62,4 +62,16 @@ #define SPRN_TBRL 268 #define SPRN_TBRU 269 +#define FIXUP_ENDIAN \ + tdi 0, 0, 0x48; /* Reverse endian of b . + 8 */ \ + b $+36; /* Skip trampoline if endian is good */ \ + .long 0x05009f42; /* bcl 20,31,$+4 */ \ + .long 0xa602487d; /* mflr r10 */ \ + .long 0x1c004a39; /* addi r10,r10,28 */ \ + .long 0xa600607d; /* mfmsr r11 */ \ + .long 0x01006b69; /* xori r11,r11,1 */ \ + .long 0xa6035a7d; /* mtsrr0 r10 */ \ + .long 0xa6037b7d; /* mtsrr1 r11 */ \ + .long 0x2400004c /* rfid */ + #endif /* _PPC64_PPC_ASM_H */ diff --git a/arch/powerpc/boot/pseries-head.S b/arch/powerpc/boot/pseries-head.S new file mode 100644 index 000000000000..6ef6e02e80f9 --- /dev/null +++ b/arch/powerpc/boot/pseries-head.S @@ -0,0 +1,8 @@ +#include "ppc_asm.h" + + .text + + .globl _zimage_start +_zimage_start: + FIXUP_ENDIAN + b _zimage_start_lib diff --git a/arch/powerpc/boot/stdio.c b/arch/powerpc/boot/stdio.c index 5b57800bbc67..a701261b1781 100644 --- a/arch/powerpc/boot/stdio.c +++ b/arch/powerpc/boot/stdio.c @@ -21,6 +21,18 @@ size_t strnlen(const char * s, size_t count) return sc - s; } +#ifdef __powerpc64__ + +# define do_div(n, base) ({ \ + unsigned int __base = (base); \ + unsigned int __rem; \ + __rem = ((unsigned long long)(n)) % __base; \ + (n) = ((unsigned long long)(n)) / __base; \ + __rem; \ +}) + +#else + extern unsigned int __div64_32(unsigned long long *dividend, unsigned int divisor); @@ -39,6 +51,8 @@ extern unsigned int __div64_32(unsigned long long *dividend, __rem; \ }) +#endif /* __powerpc64__ */ + static int skip_atoi(const char **s) { int i, c; diff --git a/arch/powerpc/boot/swab.h b/arch/powerpc/boot/swab.h new file mode 100644 index 000000000000..d0e1431084ca --- /dev/null +++ b/arch/powerpc/boot/swab.h @@ -0,0 +1,29 @@ +#ifndef _PPC_BOOT_SWAB_H_ +#define _PPC_BOOT_SWAB_H_ + +static inline u16 swab16(u16 x) +{ + return ((x & (u16)0x00ffU) << 8) | + ((x & (u16)0xff00U) >> 8); +} + +static inline u32 swab32(u32 x) +{ + return ((x & (u32)0x000000ffUL) << 24) | + ((x & (u32)0x0000ff00UL) << 8) | + ((x & (u32)0x00ff0000UL) >> 8) | + ((x & (u32)0xff000000UL) >> 24); +} + +static inline u64 swab64(u64 x) +{ + return (u64)((x & (u64)0x00000000000000ffULL) << 56) | + (u64)((x & (u64)0x000000000000ff00ULL) << 40) | + (u64)((x & (u64)0x0000000000ff0000ULL) << 24) | + (u64)((x & (u64)0x00000000ff000000ULL) << 8) | + (u64)((x & (u64)0x000000ff00000000ULL) >> 8) | + (u64)((x & (u64)0x0000ff0000000000ULL) >> 24) | + (u64)((x & (u64)0x00ff000000000000ULL) >> 40) | + (u64)((x & (u64)0xff00000000000000ULL) >> 56); +} +#endif /* _PPC_BOOT_SWAB_H_ */ diff --git a/arch/powerpc/boot/treeboot-akebono.c b/arch/powerpc/boot/treeboot-akebono.c new file mode 100644 index 000000000000..b73174c34fe4 --- /dev/null +++ b/arch/powerpc/boot/treeboot-akebono.c @@ -0,0 +1,163 @@ +/* + * Copyright © 2013 Tony Breeds IBM Corporation + * Copyright © 2013 Alistair Popple IBM Corporation + * + * Based on earlier code: + * Copyright (C) Paul Mackerras 1997. + * + * Matt Porter <mporter@kernel.crashing.org> + * Copyright 2002-2005 MontaVista Software Inc. + * + * Eugene Surovegin <eugene.surovegin@zultys.com> or <ebs@ebshome.net> + * Copyright (c) 2003, 2004 Zultys Technologies + * + * Copyright 2007 David Gibson, IBM Corporation. + * Copyright 2010 Ben. Herrenschmidt, IBM Corporation. + * Copyright © 2011 David Kleikamp IBM Corporation + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License + * as published by the Free Software Foundation; either version + * 2 of the License, or (at your option) any later version. + */ +#include <stdarg.h> +#include <stddef.h> +#include "types.h" +#include "elf.h" +#include "string.h" +#include "stdlib.h" +#include "stdio.h" +#include "page.h" +#include "ops.h" +#include "reg.h" +#include "io.h" +#include "dcr.h" +#include "4xx.h" +#include "44x.h" +#include "libfdt.h" + +BSS_STACK(4096); + +#define SPRN_PIR 0x11E /* Processor Indentification Register */ +#define USERDATA_LEN 256 /* Length of userdata passed in by PIBS */ +#define MAX_RANKS 0x4 +#define DDR3_MR0CF 0x80010011U +#define CCTL0_MCO2 0x8000080FU +#define CCTL0_MCO3 0x80000810U +#define CCTL0_MCO4 0x80000811U +#define CCTL0_MCO5 0x80000812U +#define CCTL0_MCO6 0x80000813U + +static unsigned long long ibm_akebono_memsize; +static long long unsigned mac_addr; + +static unsigned long long ibm_akebono_detect_memsize(void) +{ + u32 reg; + unsigned i; + unsigned long long memsize = 0; + + for (i = 0; i < MAX_RANKS; i++) { + reg = mfdcrx(DDR3_MR0CF + i); + + if (!(reg & 1)) + continue; + + reg &= 0x0000f000; + reg >>= 12; + memsize += (0x800000ULL << reg); + } + + return memsize; +} + +static void ibm_akebono_fixups(void) +{ + void *emac; + u32 reg; + + dt_fixup_memory(0x0ULL, ibm_akebono_memsize); + + /* Fixup the SD timeout frequency */ + mtdcrx(CCTL0_MCO4, 0x1); + + /* Disable SD high-speed mode (which seems to be broken) */ + reg = mfdcrx(CCTL0_MCO2) & ~0x2; + mtdcrx(CCTL0_MCO2, reg); + + /* Set the MAC address */ + emac = finddevice("/plb/opb/ethernet"); + if (emac > 0) { + if (mac_addr) + setprop(emac, "local-mac-address", + ((u8 *) &mac_addr) + 2 , 6); + } +} + +void platform_init(char *userdata) +{ + unsigned long end_of_ram, avail_ram; + u32 pir_reg; + int node, size; + const u32 *timebase; + int len, i, userdata_len; + char *end; + + userdata[USERDATA_LEN - 1] = '\0'; + userdata_len = strlen(userdata); + for (i = 0; i < userdata_len - 15; i++) { + if (strncmp(&userdata[i], "local-mac-addr=", 15) == 0) { + if (i > 0 && userdata[i - 1] != ' ') { + /* We've only found a substring ending + * with local-mac-addr so this isn't + * our mac address. */ + continue; + } + + mac_addr = strtoull(&userdata[i + 15], &end, 16); + + /* Remove the "local-mac-addr=<...>" from the kernel + * command line, including the tailing space if + * present. */ + if (*end == ' ') + end++; + + len = ((int) end) - ((int) &userdata[i]); + memmove(&userdata[i], end, + userdata_len - (len + i) + 1); + break; + } + } + + loader_info.cmdline = userdata; + loader_info.cmdline_len = 256; + + ibm_akebono_memsize = ibm_akebono_detect_memsize(); + if (ibm_akebono_memsize >> 32) + end_of_ram = ~0UL; + else + end_of_ram = ibm_akebono_memsize; + avail_ram = end_of_ram - (unsigned long)_end; + + simple_alloc_init(_end, avail_ram, 128, 64); + platform_ops.fixups = ibm_akebono_fixups; + platform_ops.exit = ibm44x_dbcr_reset; + pir_reg = mfspr(SPRN_PIR); + + /* Make sure FDT blob is sane */ + if (fdt_check_header(_dtb_start) != 0) + fatal("Invalid device tree blob\n"); + + node = fdt_node_offset_by_prop_value(_dtb_start, -1, "device_type", + "cpu", sizeof("cpu")); + if (!node) + fatal("Cannot find cpu node\n"); + timebase = fdt_getprop(_dtb_start, node, "timebase-frequency", &size); + if (timebase && (size == 4)) + timebase_period_ns = 1000000000 / *timebase; + + fdt_set_boot_cpuid_phys(_dtb_start, pir_reg); + fdt_init(_dtb_start); + + serial_console_init(); +} diff --git a/arch/powerpc/boot/util.S b/arch/powerpc/boot/util.S index 6636b1d7821b..243b8497d58b 100644 --- a/arch/powerpc/boot/util.S +++ b/arch/powerpc/boot/util.S @@ -45,7 +45,7 @@ udelay: mfspr r4,SPRN_PVR srwi r4,r4,16 cmpwi 0,r4,1 /* 601 ? */ - bne .udelay_not_601 + bne .Ludelay_not_601 00: li r0,86 /* Instructions / microsecond? */ mtctr r0 10: addi r0,r0,0 /* NOP */ @@ -54,7 +54,7 @@ udelay: bne 00b blr -.udelay_not_601: +.Ludelay_not_601: mulli r4,r3,1000 /* nanoseconds */ /* Change r4 to be the number of ticks using: * (nanoseconds + (timebase_period_ns - 1 )) / timebase_period_ns diff --git a/arch/powerpc/boot/wrapper b/arch/powerpc/boot/wrapper index d27a25518b01..ae0f88ec4a32 100755 --- a/arch/powerpc/boot/wrapper +++ b/arch/powerpc/boot/wrapper @@ -40,6 +40,7 @@ cacheit= binary= gzip=.gz pie= +format= # cross-compilation prefix CROSS= @@ -136,6 +137,14 @@ if [ -z "$kernel" ]; then kernel=vmlinux fi +elfformat="`${CROSS}objdump -p "$kernel" | grep 'file format' | awk '{print $4}'`" +case "$elfformat" in + elf64-powerpcle) format=elf64lppc ;; + elf64-powerpc) format=elf32ppc ;; + elf32-powerpc) format=elf32ppc ;; +esac + + platformo=$object/"$platform".o lds=$object/zImage.lds ext=strip @@ -152,8 +161,12 @@ of) make_space=n ;; pseries) - platformo="$object/of.o $object/epapr.o" + platformo="$object/pseries-head.o $object/of.o $object/epapr.o" link_address='0x4000000' + if [ "$format" != "elf32ppc" ]; then + link_address= + pie=-pie + fi make_space=n ;; maple) @@ -257,6 +270,9 @@ gamecube|wii) treeboot-currituck) link_address='0x1000000' ;; +treeboot-akebono) + link_address='0x1000000' + ;; treeboot-iss4xx-mpic) platformo="$object/treeboot-iss4xx.o" ;; @@ -379,7 +395,7 @@ if [ "$platform" != "miboot" ]; then if [ -n "$link_address" ] ; then text_start="-Ttext $link_address" fi - ${CROSS}ld -m elf32ppc -T $lds $text_start $pie -o "$ofile" \ + ${CROSS}ld -m $format -T $lds $text_start $pie -o "$ofile" \ $platformo $tmp $object/wrapper.a rm $tmp fi diff --git a/arch/powerpc/boot/zImage.lds.S b/arch/powerpc/boot/zImage.lds.S index 2bd8731f1365..861e72109df2 100644 --- a/arch/powerpc/boot/zImage.lds.S +++ b/arch/powerpc/boot/zImage.lds.S @@ -1,4 +1,10 @@ +#include <asm-generic/vmlinux.lds.h> + +#ifdef CONFIG_PPC64_BOOT_WRAPPER +OUTPUT_ARCH(powerpc:common64) +#else OUTPUT_ARCH(powerpc:common) +#endif ENTRY(_zimage_start) EXTERN(_zimage_start) SECTIONS @@ -16,7 +22,9 @@ SECTIONS *(.rodata*) *(.data*) *(.sdata*) +#ifndef CONFIG_PPC64_BOOT_WRAPPER *(.got2) +#endif } .dynsym : { *(.dynsym) } .dynstr : { *(.dynstr) } @@ -27,7 +35,13 @@ SECTIONS } .hash : { *(.hash) } .interp : { *(.interp) } - .rela.dyn : { *(.rela*) } + .rela.dyn : + { +#ifdef CONFIG_PPC64_BOOT_WRAPPER + __rela_dyn_start = .; +#endif + *(.rela*) + } . = ALIGN(8); .kernel:dtb : @@ -53,6 +67,15 @@ SECTIONS _initrd_end = .; } +#ifdef CONFIG_PPC64_BOOT_WRAPPER + .got : + { + __toc_start = .; + *(.got) + *(.toc) + } +#endif + . = ALIGN(4096); .bss : { diff --git a/arch/powerpc/configs/44x/akebono_defconfig b/arch/powerpc/configs/44x/akebono_defconfig new file mode 100644 index 000000000000..7e2530cd9d30 --- /dev/null +++ b/arch/powerpc/configs/44x/akebono_defconfig @@ -0,0 +1,148 @@ +CONFIG_44x=y +CONFIG_SMP=y +CONFIG_SYSVIPC=y +CONFIG_POSIX_MQUEUE=y +CONFIG_LOG_BUF_SHIFT=14 +CONFIG_BLK_DEV_INITRD=y +CONFIG_RD_BZIP2=y +CONFIG_RD_LZMA=y +CONFIG_RD_XZ=y +CONFIG_EXPERT=y +CONFIG_KALLSYMS_ALL=y +# CONFIG_SLUB_CPU_PARTIAL is not set +CONFIG_PROFILING=y +CONFIG_OPROFILE=y +CONFIG_MODULES=y +CONFIG_MODULE_UNLOAD=y +# CONFIG_BLK_DEV_BSG is not set +# CONFIG_POWERNV_MSI is not set +CONFIG_PPC_47x=y +# CONFIG_EBONY is not set +CONFIG_AKEBONO=y +CONFIG_HIGHMEM=y +CONFIG_HZ_100=y +CONFIG_IRQ_ALL_CPUS=y +# CONFIG_COMPACTION is not set +CONFIG_CMDLINE_BOOL=y +CONFIG_CMDLINE="" +# CONFIG_SUSPEND is not set +CONFIG_PCI_MSI=y +CONFIG_NET=y +CONFIG_PACKET=y +CONFIG_UNIX=y +CONFIG_INET=y +CONFIG_IP_PNP=y +CONFIG_IP_PNP_DHCP=y +CONFIG_IP_PNP_BOOTP=y +# CONFIG_INET_XFRM_MODE_TRANSPORT is not set +# CONFIG_INET_XFRM_MODE_TUNNEL is not set +# CONFIG_INET_XFRM_MODE_BEET is not set +# CONFIG_INET_LRO is not set +# CONFIG_IPV6 is not set +CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug" +CONFIG_DEVTMPFS=y +CONFIG_DEVTMPFS_MOUNT=y +CONFIG_CONNECTOR=y +CONFIG_MTD=y +CONFIG_MTD_BLOCK=y +CONFIG_MTD_JEDECPROBE=y +CONFIG_MTD_CFI_AMDSTD=y +CONFIG_MTD_PHYSMAP_OF=y +CONFIG_PROC_DEVICETREE=y +CONFIG_BLK_DEV_RAM=y +CONFIG_BLK_DEV_RAM_SIZE=35000 +# CONFIG_SCSI_PROC_FS is not set +CONFIG_BLK_DEV_SD=y +# CONFIG_SCSI_LOWLEVEL is not set +# CONFIG_SATA_PMP is not set +# CONFIG_ATA_SFF is not set +# CONFIG_NET_VENDOR_3COM is not set +# CONFIG_NET_VENDOR_ADAPTEC is not set +# CONFIG_NET_VENDOR_ALTEON is not set +# CONFIG_NET_VENDOR_AMD is not set +# CONFIG_NET_VENDOR_ARC is not set +# CONFIG_NET_VENDOR_ATHEROS is not set +# CONFIG_NET_CADENCE is not set +# CONFIG_NET_VENDOR_BROADCOM is not set +# CONFIG_NET_VENDOR_BROCADE is not set +# CONFIG_NET_VENDOR_CHELSIO is not set +# CONFIG_NET_VENDOR_CISCO is not set +# CONFIG_NET_VENDOR_DEC is not set +# CONFIG_NET_VENDOR_DLINK is not set +# CONFIG_NET_VENDOR_EMULEX is not set +# CONFIG_NET_VENDOR_EXAR is not set +# CONFIG_NET_VENDOR_HP is not set +CONFIG_IBM_EMAC=y +# CONFIG_NET_VENDOR_MARVELL is not set +# CONFIG_NET_VENDOR_MELLANOX is not set +# CONFIG_NET_VENDOR_MICREL is not set +# CONFIG_NET_VENDOR_MYRI is not set +# CONFIG_NET_VENDOR_NATSEMI is not set +# CONFIG_NET_VENDOR_NVIDIA is not set +# CONFIG_NET_VENDOR_OKI is not set +# CONFIG_NET_VENDOR_QLOGIC is not set +# CONFIG_NET_VENDOR_REALTEK is not set +# CONFIG_NET_VENDOR_RDC is not set +# CONFIG_NET_VENDOR_SEEQ is not set +# CONFIG_NET_VENDOR_SILAN is not set +# CONFIG_NET_VENDOR_SIS is not set +# CONFIG_NET_VENDOR_SMSC is not set +# CONFIG_NET_VENDOR_STMICRO is not set +# CONFIG_NET_VENDOR_SUN is not set +# CONFIG_NET_VENDOR_TEHUTI is not set +# CONFIG_NET_VENDOR_TI is not set +# CONFIG_NET_VENDOR_VIA is not set +# CONFIG_NET_VENDOR_WIZNET is not set +# CONFIG_NET_VENDOR_XILINX is not set +# CONFIG_KEYBOARD_ATKBD is not set +# CONFIG_MOUSE_PS2 is not set +# CONFIG_SERIO is not set +# CONFIG_VT is not set +CONFIG_SERIAL_8250=y +# CONFIG_SERIAL_8250_DEPRECATED_OPTIONS is not set +CONFIG_SERIAL_8250_CONSOLE=y +CONFIG_SERIAL_8250_EXTENDED=y +CONFIG_SERIAL_8250_SHARE_IRQ=y +CONFIG_SERIAL_OF_PLATFORM=y +# CONFIG_HW_RANDOM is not set +CONFIG_I2C_CHARDEV=y +# CONFIG_HWMON is not set +CONFIG_THERMAL=y +# CONFIG_USB_DEFAULT_PERSIST is not set +CONFIG_USB_EHCI_HCD=y +CONFIG_USB_OHCI_HCD=y +# CONFIG_USB_OHCI_HCD_PCI is not set +CONFIG_USB_STORAGE=y +CONFIG_MMC=y +CONFIG_RTC_CLASS=y +CONFIG_RTC_DRV_M41T80=y +CONFIG_EXT2_FS=y +CONFIG_EXT3_FS=y +# CONFIG_EXT3_DEFAULTS_TO_ORDERED is not set +CONFIG_EXT3_FS_POSIX_ACL=y +CONFIG_EXT3_FS_SECURITY=y +# CONFIG_DNOTIFY is not set +# CONFIG_INOTIFY_USER is not set +CONFIG_VFAT_FS=y +CONFIG_PROC_KCORE=y +CONFIG_TMPFS=y +CONFIG_CRAMFS=y +# CONFIG_NETWORK_FILESYSTEMS is not set +CONFIG_NLS_DEFAULT="n" +CONFIG_NLS_CODEPAGE_437=y +CONFIG_NLS_ISO8859_1=y +CONFIG_DEBUG_INFO=y +CONFIG_DEBUG_FS=y +CONFIG_MAGIC_SYSRQ=y +CONFIG_DETECT_HUNG_TASK=y +CONFIG_XMON=y +CONFIG_XMON_DEFAULT=y +CONFIG_PPC_EARLY_DEBUG=y +CONFIG_PPC_EARLY_DEBUG_44x_PHYSLOW=0x00010000 +CONFIG_PPC_EARLY_DEBUG_44x_PHYSHIGH=0x33f +CONFIG_CRYPTO_PCBC=y +CONFIG_CRYPTO_MD5=y +CONFIG_CRYPTO_SHA1_PPC=y +CONFIG_CRYPTO_DES=y +# CONFIG_CRYPTO_ANSI_CPRNG is not set +# CONFIG_CRYPTO_HW is not set diff --git a/arch/powerpc/configs/85xx/kmp204x_defconfig b/arch/powerpc/configs/85xx/kmp204x_defconfig new file mode 100644 index 000000000000..e9a81e5ba273 --- /dev/null +++ b/arch/powerpc/configs/85xx/kmp204x_defconfig @@ -0,0 +1,225 @@ +CONFIG_PPC_85xx=y +CONFIG_SMP=y +CONFIG_NR_CPUS=8 +CONFIG_SYSVIPC=y +CONFIG_POSIX_MQUEUE=y +CONFIG_AUDIT=y +CONFIG_NO_HZ=y +CONFIG_HIGH_RES_TIMERS=y +CONFIG_BSD_PROCESS_ACCT=y +CONFIG_IKCONFIG=y +CONFIG_IKCONFIG_PROC=y +CONFIG_LOG_BUF_SHIFT=14 +CONFIG_CGROUPS=y +CONFIG_CGROUP_SCHED=y +CONFIG_RELAY=y +CONFIG_BLK_DEV_INITRD=y +CONFIG_KALLSYMS_ALL=y +CONFIG_EMBEDDED=y +CONFIG_PERF_EVENTS=y +CONFIG_SLAB=y +CONFIG_MODULES=y +CONFIG_MODULE_UNLOAD=y +CONFIG_MODULE_FORCE_UNLOAD=y +CONFIG_MODVERSIONS=y +# CONFIG_BLK_DEV_BSG is not set +CONFIG_PARTITION_ADVANCED=y +CONFIG_MAC_PARTITION=y +CONFIG_CORENET_GENERIC=y +CONFIG_MPIC_MSGR=y +CONFIG_HIGHMEM=y +# CONFIG_CORE_DUMP_DEFAULT_ELF_HEADERS is not set +CONFIG_BINFMT_MISC=m +CONFIG_KEXEC=y +CONFIG_FORCE_MAX_ZONEORDER=13 +CONFIG_PCI=y +CONFIG_PCIEPORTBUS=y +# CONFIG_PCIEASPM is not set +CONFIG_PCI_MSI=y +CONFIG_ADVANCED_OPTIONS=y +CONFIG_LOWMEM_SIZE_BOOL=y +CONFIG_LOWMEM_SIZE=0x20000000 +CONFIG_NET=y +CONFIG_PACKET=y +CONFIG_UNIX=y +CONFIG_XFRM_USER=y +CONFIG_XFRM_SUB_POLICY=y +CONFIG_XFRM_STATISTICS=y +CONFIG_NET_KEY=y +CONFIG_NET_KEY_MIGRATE=y +CONFIG_INET=y +CONFIG_IP_MULTICAST=y +CONFIG_IP_ADVANCED_ROUTER=y +CONFIG_IP_MULTIPLE_TABLES=y +CONFIG_IP_ROUTE_MULTIPATH=y +CONFIG_IP_ROUTE_VERBOSE=y +CONFIG_IP_PNP=y +CONFIG_IP_PNP_DHCP=y +CONFIG_IP_PNP_BOOTP=y +CONFIG_IP_PNP_RARP=y +CONFIG_NET_IPIP=y +CONFIG_IP_MROUTE=y +CONFIG_IP_PIMSM_V1=y +CONFIG_IP_PIMSM_V2=y +CONFIG_INET_AH=y +CONFIG_INET_ESP=y +CONFIG_INET_IPCOMP=y +# CONFIG_INET_LRO is not set +CONFIG_IPV6=y +CONFIG_IP_SCTP=m +CONFIG_TIPC=y +CONFIG_NET_SCHED=y +CONFIG_NET_SCH_CBQ=y +CONFIG_NET_SCH_HTB=y +CONFIG_NET_SCH_HFSC=y +CONFIG_NET_SCH_PRIO=y +CONFIG_NET_SCH_MULTIQ=y +CONFIG_NET_SCH_RED=y +CONFIG_NET_SCH_SFQ=y +CONFIG_NET_SCH_TEQL=y +CONFIG_NET_SCH_TBF=y +CONFIG_NET_SCH_GRED=y +CONFIG_NET_CLS_BASIC=y +CONFIG_NET_CLS_TCINDEX=y +CONFIG_NET_CLS_U32=y +CONFIG_CLS_U32_PERF=y +CONFIG_CLS_U32_MARK=y +CONFIG_NET_CLS_FLOW=y +CONFIG_NET_CLS_CGROUP=y +CONFIG_UEVENT_HELPER_PATH="/sbin/mdev" +CONFIG_DEVTMPFS=y +CONFIG_MTD=y +CONFIG_MTD_CMDLINE_PARTS=y +CONFIG_MTD_BLOCK=y +CONFIG_MTD_CFI=y +CONFIG_MTD_CFI_AMDSTD=y +CONFIG_MTD_PHYSMAP_OF=y +CONFIG_MTD_M25P80=y +CONFIG_MTD_PHRAM=y +CONFIG_MTD_NAND=y +CONFIG_MTD_NAND_ECC_BCH=y +CONFIG_MTD_NAND_FSL_ELBC=y +CONFIG_MTD_UBI=y +CONFIG_MTD_UBI_GLUEBI=y +CONFIG_BLK_DEV_LOOP=y +CONFIG_BLK_DEV_RAM=y +CONFIG_BLK_DEV_RAM_COUNT=2 +CONFIG_BLK_DEV_RAM_SIZE=2048 +CONFIG_EEPROM_AT24=y +CONFIG_SCSI=y +CONFIG_BLK_DEV_SD=y +CONFIG_CHR_DEV_ST=y +CONFIG_BLK_DEV_SR=y +CONFIG_CHR_DEV_SG=y +CONFIG_SCSI_MULTI_LUN=y +CONFIG_SCSI_LOGGING=y +CONFIG_SCSI_SYM53C8XX_2=y +CONFIG_NETDEVICES=y +# CONFIG_NET_VENDOR_3COM is not set +# CONFIG_NET_VENDOR_ADAPTEC is not set +# CONFIG_NET_VENDOR_ALTEON is not set +# CONFIG_NET_VENDOR_AMD is not set +# CONFIG_NET_VENDOR_ATHEROS is not set +# CONFIG_NET_CADENCE is not set +# CONFIG_NET_VENDOR_BROADCOM is not set +# CONFIG_NET_VENDOR_BROCADE is not set +# CONFIG_NET_VENDOR_CHELSIO is not set +# CONFIG_NET_VENDOR_CISCO is not set +# CONFIG_NET_VENDOR_DEC is not set +# CONFIG_NET_VENDOR_DLINK is not set +# CONFIG_NET_VENDOR_EMULEX is not set +# CONFIG_NET_VENDOR_EXAR is not set +CONFIG_FSL_PQ_MDIO=y +CONFIG_FSL_XGMAC_MDIO=y +# CONFIG_NET_VENDOR_HP is not set +# CONFIG_NET_VENDOR_INTEL is not set +# CONFIG_NET_VENDOR_MARVELL is not set +# CONFIG_NET_VENDOR_MELLANOX is not set +# CONFIG_NET_VENDOR_MICREL is not set +# CONFIG_NET_VENDOR_MICROCHIP is not set +# CONFIG_NET_VENDOR_MYRI is not set +# CONFIG_NET_VENDOR_NATSEMI is not set +# CONFIG_NET_VENDOR_NVIDIA is not set +# CONFIG_NET_VENDOR_OKI is not set +# CONFIG_NET_PACKET_ENGINE is not set +# CONFIG_NET_VENDOR_QLOGIC is not set +# CONFIG_NET_VENDOR_REALTEK is not set +# CONFIG_NET_VENDOR_RDC is not set +# CONFIG_NET_VENDOR_SEEQ is not set +# CONFIG_NET_VENDOR_SILAN is not set +# CONFIG_NET_VENDOR_SIS is not set +# CONFIG_NET_VENDOR_SMSC is not set +# CONFIG_NET_VENDOR_STMICRO is not set +# CONFIG_NET_VENDOR_SUN is not set +# CONFIG_NET_VENDOR_TEHUTI is not set +# CONFIG_NET_VENDOR_TI is not set +# CONFIG_NET_VENDOR_VIA is not set +# CONFIG_NET_VENDOR_WIZNET is not set +# CONFIG_NET_VENDOR_XILINX is not set +CONFIG_MARVELL_PHY=y +CONFIG_VITESSE_PHY=y +CONFIG_FIXED_PHY=y +# CONFIG_WLAN is not set +# CONFIG_INPUT_MOUSEDEV is not set +# CONFIG_INPUT_KEYBOARD is not set +# CONFIG_INPUT_MOUSE is not set +CONFIG_SERIO_LIBPS2=y +# CONFIG_LEGACY_PTYS is not set +CONFIG_PPC_EPAPR_HV_BYTECHAN=y +CONFIG_SERIAL_8250=y +CONFIG_SERIAL_8250_CONSOLE=y +CONFIG_SERIAL_8250_MANY_PORTS=y +CONFIG_SERIAL_8250_DETECT_IRQ=y +CONFIG_SERIAL_8250_RSA=y +CONFIG_NVRAM=y +CONFIG_I2C=y +CONFIG_I2C_CHARDEV=y +CONFIG_I2C_MUX=y +CONFIG_I2C_MUX_PCA954x=y +CONFIG_I2C_MPC=y +CONFIG_SPI=y +CONFIG_SPI_FSL_SPI=y +CONFIG_SPI_FSL_ESPI=y +CONFIG_SPI_SPIDEV=m +CONFIG_PTP_1588_CLOCK=y +# CONFIG_HWMON is not set +# CONFIG_USB_SUPPORT is not set +CONFIG_EDAC=y +CONFIG_EDAC_MM_EDAC=y +CONFIG_EDAC_MPC85XX=y +CONFIG_RTC_CLASS=y +CONFIG_RTC_DRV_DS3232=y +CONFIG_RTC_DRV_CMOS=y +CONFIG_UIO=y +CONFIG_STAGING=y +# CONFIG_NET_VENDOR_SILICOM is not set +CONFIG_CLK_PPC_CORENET=y +CONFIG_EXT2_FS=y +CONFIG_NTFS_FS=y +CONFIG_PROC_KCORE=y +CONFIG_TMPFS=y +CONFIG_JFFS2_FS=y +CONFIG_UBIFS_FS=y +CONFIG_CRAMFS=y +CONFIG_SQUASHFS=y +CONFIG_SQUASHFS_XZ=y +CONFIG_NFS_FS=y +CONFIG_NFS_V4=y +CONFIG_ROOT_NFS=y +CONFIG_NLS_ISO8859_1=y +CONFIG_NLS_UTF8=m +CONFIG_CRC_ITU_T=m +CONFIG_DEBUG_INFO=y +CONFIG_MAGIC_SYSRQ=y +CONFIG_DEBUG_SHIRQ=y +CONFIG_DETECT_HUNG_TASK=y +CONFIG_SCHEDSTATS=y +CONFIG_RCU_TRACE=y +CONFIG_UPROBE_EVENT=y +CONFIG_CRYPTO_NULL=y +CONFIG_CRYPTO_PCBC=m +CONFIG_CRYPTO_MD4=y +CONFIG_CRYPTO_SHA256=y +CONFIG_CRYPTO_SHA512=y +# CONFIG_CRYPTO_ANSI_CPRNG is not set +CONFIG_CRYPTO_DEV_FSL_CAAM=y diff --git a/arch/powerpc/configs/corenet32_smp_defconfig b/arch/powerpc/configs/corenet32_smp_defconfig index bbd794deb6eb..c19ff057d0f9 100644 --- a/arch/powerpc/configs/corenet32_smp_defconfig +++ b/arch/powerpc/configs/corenet32_smp_defconfig @@ -72,6 +72,7 @@ CONFIG_MTD_CMDLINE_PARTS=y CONFIG_MTD_CHAR=y CONFIG_MTD_BLOCK=y CONFIG_MTD_CFI=y +CONFIG_MTD_CFI_INTELEXT=y CONFIG_MTD_CFI_AMDSTD=y CONFIG_MTD_PHYSMAP_OF=y CONFIG_MTD_M25P80=y diff --git a/arch/powerpc/configs/mpc85xx_defconfig b/arch/powerpc/configs/mpc85xx_defconfig index 19f0fbe5ba4b..55765c8cb08f 100644 --- a/arch/powerpc/configs/mpc85xx_defconfig +++ b/arch/powerpc/configs/mpc85xx_defconfig @@ -32,7 +32,6 @@ CONFIG_P1010_RDB=y CONFIG_P1022_DS=y CONFIG_P1022_RDK=y CONFIG_P1023_RDB=y -CONFIG_P1023_RDS=y CONFIG_SOCRATES=y CONFIG_KSI8560=y CONFIG_XES_MPC85xx=y diff --git a/arch/powerpc/configs/mpc85xx_smp_defconfig b/arch/powerpc/configs/mpc85xx_smp_defconfig index 062312e1fe1a..5c6ecdc0f70e 100644 --- a/arch/powerpc/configs/mpc85xx_smp_defconfig +++ b/arch/powerpc/configs/mpc85xx_smp_defconfig @@ -35,7 +35,6 @@ CONFIG_P1010_RDB=y CONFIG_P1022_DS=y CONFIG_P1022_RDK=y CONFIG_P1023_RDB=y -CONFIG_P1023_RDS=y CONFIG_SOCRATES=y CONFIG_KSI8560=y CONFIG_XES_MPC85xx=y diff --git a/arch/powerpc/include/asm/code-patching.h b/arch/powerpc/include/asm/code-patching.h index 97e02f985df8..37991e154ef8 100644 --- a/arch/powerpc/include/asm/code-patching.h +++ b/arch/powerpc/include/asm/code-patching.h @@ -42,15 +42,47 @@ void __patch_exception(int exc, unsigned long addr); } while (0) #endif +#define OP_RT_RA_MASK 0xffff0000UL +#define LIS_R2 0x3c020000UL +#define ADDIS_R2_R12 0x3c4c0000UL +#define ADDI_R2_R2 0x38420000UL + static inline unsigned long ppc_function_entry(void *func) { -#ifdef CONFIG_PPC64 +#if defined(CONFIG_PPC64) +#if defined(_CALL_ELF) && _CALL_ELF == 2 + u32 *insn = func; + + /* + * A PPC64 ABIv2 function may have a local and a global entry + * point. We need to use the local entry point when patching + * functions, so identify and step over the global entry point + * sequence. + * + * The global entry point sequence is always of the form: + * + * addis r2,r12,XXXX + * addi r2,r2,XXXX + * + * A linker optimisation may convert the addis to lis: + * + * lis r2,XXXX + * addi r2,r2,XXXX + */ + if ((((*insn & OP_RT_RA_MASK) == ADDIS_R2_R12) || + ((*insn & OP_RT_RA_MASK) == LIS_R2)) && + ((*(insn+1) & OP_RT_RA_MASK) == ADDI_R2_R2)) + return (unsigned long)(insn + 2); + else + return (unsigned long)func; +#else /* - * On PPC64 the function pointer actually points to the function's - * descriptor. The first entry in the descriptor is the address - * of the function text. + * On PPC64 ABIv1 the function pointer actually points to the + * function's descriptor. The first entry in the descriptor is the + * address of the function text. */ return ((func_descr_t *)func)->entry; +#endif #else return (unsigned long)func; #endif diff --git a/arch/powerpc/include/asm/context_tracking.h b/arch/powerpc/include/asm/context_tracking.h index b6f5a33b8ee2..40014921ffff 100644 --- a/arch/powerpc/include/asm/context_tracking.h +++ b/arch/powerpc/include/asm/context_tracking.h @@ -2,9 +2,9 @@ #define _ASM_POWERPC_CONTEXT_TRACKING_H #ifdef CONFIG_CONTEXT_TRACKING -#define SCHEDULE_USER bl .schedule_user +#define SCHEDULE_USER bl schedule_user #else -#define SCHEDULE_USER bl .schedule +#define SCHEDULE_USER bl schedule #endif #endif diff --git a/arch/powerpc/include/asm/cputhreads.h b/arch/powerpc/include/asm/cputhreads.h index ac3eedb9b74a..2bf8e9307be9 100644 --- a/arch/powerpc/include/asm/cputhreads.h +++ b/arch/powerpc/include/asm/cputhreads.h @@ -18,10 +18,12 @@ #ifdef CONFIG_SMP extern int threads_per_core; +extern int threads_per_subcore; extern int threads_shift; extern cpumask_t threads_core_mask; #else #define threads_per_core 1 +#define threads_per_subcore 1 #define threads_shift 0 #define threads_core_mask (CPU_MASK_CPU0) #endif @@ -74,6 +76,11 @@ static inline int cpu_thread_in_core(int cpu) return cpu & (threads_per_core - 1); } +static inline int cpu_thread_in_subcore(int cpu) +{ + return cpu & (threads_per_subcore - 1); +} + static inline int cpu_first_thread_sibling(int cpu) { return cpu & ~(threads_per_core - 1); diff --git a/arch/powerpc/include/asm/debug.h b/arch/powerpc/include/asm/debug.h index d2516308ed1e..a954e4975049 100644 --- a/arch/powerpc/include/asm/debug.h +++ b/arch/powerpc/include/asm/debug.h @@ -46,7 +46,8 @@ static inline int debugger_break_match(struct pt_regs *regs) { return 0; } static inline int debugger_fault_handler(struct pt_regs *regs) { return 0; } #endif -int set_breakpoint(struct arch_hw_breakpoint *brk); +void set_breakpoint(struct arch_hw_breakpoint *brk); +void __set_breakpoint(struct arch_hw_breakpoint *brk); #ifdef CONFIG_PPC_ADV_DEBUG_REGS extern void do_send_trap(struct pt_regs *regs, unsigned long address, unsigned long error_code, int signal_code, int brkpt); diff --git a/arch/powerpc/include/asm/eeh.h b/arch/powerpc/include/asm/eeh.h index d4dd41fb951b..b76f58c124ca 100644 --- a/arch/powerpc/include/asm/eeh.h +++ b/arch/powerpc/include/asm/eeh.h @@ -32,6 +32,22 @@ struct device_node; #ifdef CONFIG_EEH +/* EEH subsystem flags */ +#define EEH_ENABLED 0x1 /* EEH enabled */ +#define EEH_FORCE_DISABLED 0x2 /* EEH disabled */ +#define EEH_PROBE_MODE_DEV 0x4 /* From PCI device */ +#define EEH_PROBE_MODE_DEVTREE 0x8 /* From device tree */ + +/* + * Delay for PE reset, all in ms + * + * PCI specification has reset hold time of 100 milliseconds. + * We have 250 milliseconds here. The PCI bus settlement time + * is specified as 1.5 seconds and we have 1.8 seconds. + */ +#define EEH_PE_RST_HOLD_TIME 250 +#define EEH_PE_RST_SETTLE_TIME 1800 + /* * The struct is used to trace PE related EEH functionality. * In theory, there will have one instance of the struct to @@ -53,7 +69,7 @@ struct device_node; #define EEH_PE_ISOLATED (1 << 0) /* Isolated PE */ #define EEH_PE_RECOVERING (1 << 1) /* Recovering PE */ -#define EEH_PE_PHB_DEAD (1 << 2) /* Dead PHB */ +#define EEH_PE_RESET (1 << 2) /* PE reset in progress */ #define EEH_PE_KEEP (1 << 8) /* Keep PE on hotplug */ @@ -92,6 +108,7 @@ struct eeh_pe { #define EEH_DEV_NO_HANDLER (1 << 8) /* No error handler */ #define EEH_DEV_SYSFS (1 << 9) /* Sysfs created */ +#define EEH_DEV_REMOVED (1 << 10) /* Removed permanently */ struct eeh_dev { int mode; /* EEH mode */ @@ -99,7 +116,9 @@ struct eeh_dev { int config_addr; /* Config address */ int pe_config_addr; /* PE config address */ u32 config_space[16]; /* Saved PCI config space */ - u8 pcie_cap; /* Saved PCIe capability */ + int pcix_cap; /* Saved PCIx capability */ + int pcie_cap; /* Saved PCIe capability */ + int aer_cap; /* Saved AER capability */ struct eeh_pe *pe; /* Associated PE */ struct list_head list; /* Form link list in the PE */ struct pci_controller *phb; /* Associated PHB */ @@ -171,37 +190,40 @@ struct eeh_ops { int (*restore_config)(struct device_node *dn); }; +extern int eeh_subsystem_flags; extern struct eeh_ops *eeh_ops; -extern bool eeh_subsystem_enabled; extern raw_spinlock_t confirm_error_lock; -extern int eeh_probe_mode; static inline bool eeh_enabled(void) { - return eeh_subsystem_enabled; + if ((eeh_subsystem_flags & EEH_FORCE_DISABLED) || + !(eeh_subsystem_flags & EEH_ENABLED)) + return false; + + return true; } static inline void eeh_set_enable(bool mode) { - eeh_subsystem_enabled = mode; + if (mode) + eeh_subsystem_flags |= EEH_ENABLED; + else + eeh_subsystem_flags &= ~EEH_ENABLED; } -#define EEH_PROBE_MODE_DEV (1<<0) /* From PCI device */ -#define EEH_PROBE_MODE_DEVTREE (1<<1) /* From device tree */ - static inline void eeh_probe_mode_set(int flag) { - eeh_probe_mode = flag; + eeh_subsystem_flags |= flag; } static inline int eeh_probe_mode_devtree(void) { - return (eeh_probe_mode == EEH_PROBE_MODE_DEVTREE); + return (eeh_subsystem_flags & EEH_PROBE_MODE_DEVTREE); } static inline int eeh_probe_mode_dev(void) { - return (eeh_probe_mode == EEH_PROBE_MODE_DEV); + return (eeh_subsystem_flags & EEH_PROBE_MODE_DEV); } static inline void eeh_serialize_lock(unsigned long *flags) diff --git a/arch/powerpc/include/asm/elf.h b/arch/powerpc/include/asm/elf.h index 935b5e7a1436..888d8f3f2524 100644 --- a/arch/powerpc/include/asm/elf.h +++ b/arch/powerpc/include/asm/elf.h @@ -90,6 +90,8 @@ typedef elf_vrregset_t elf_fpxregset_t; do { \ if (((ex).e_flags & 0x3) == 2) \ set_thread_flag(TIF_ELF2ABI); \ + else \ + clear_thread_flag(TIF_ELF2ABI); \ if ((ex).e_ident[EI_CLASS] == ELFCLASS32) \ set_thread_flag(TIF_32BIT); \ else \ diff --git a/arch/powerpc/include/asm/exception-64e.h b/arch/powerpc/include/asm/exception-64e.h index a563d9afd179..a8b52b61043f 100644 --- a/arch/powerpc/include/asm/exception-64e.h +++ b/arch/powerpc/include/asm/exception-64e.h @@ -174,10 +174,10 @@ exc_##label##_book3e: mtlr r16; #define TLB_MISS_STATS_D(name) \ addi r9,r13,MMSTAT_DSTATS+name; \ - bl .tlb_stat_inc; + bl tlb_stat_inc; #define TLB_MISS_STATS_I(name) \ addi r9,r13,MMSTAT_ISTATS+name; \ - bl .tlb_stat_inc; + bl tlb_stat_inc; #define TLB_MISS_STATS_X(name) \ ld r8,PACA_EXTLB+EX_TLB_ESR(r13); \ cmpdi cr2,r8,-1; \ @@ -185,7 +185,7 @@ exc_##label##_book3e: addi r9,r13,MMSTAT_DSTATS+name; \ b 62f; \ 61: addi r9,r13,MMSTAT_ISTATS+name; \ -62: bl .tlb_stat_inc; +62: bl tlb_stat_inc; #define TLB_MISS_STATS_SAVE_INFO \ std r14,EX_TLB_ESR(r12); /* save ESR */ #define TLB_MISS_STATS_SAVE_INFO_BOLTED \ diff --git a/arch/powerpc/include/asm/exception-64s.h b/arch/powerpc/include/asm/exception-64s.h index aeaa56cd9b54..8f35cd7d59cc 100644 --- a/arch/powerpc/include/asm/exception-64s.h +++ b/arch/powerpc/include/asm/exception-64s.h @@ -517,7 +517,7 @@ label##_relon_hv: \ #define DISABLE_INTS RECONCILE_IRQ_STATE(r10,r11) #define ADD_NVGPRS \ - bl .save_nvgprs + bl save_nvgprs #define RUNLATCH_ON \ BEGIN_FTR_SECTION \ diff --git a/arch/powerpc/include/asm/ftrace.h b/arch/powerpc/include/asm/ftrace.h index 169d039ed402..e3661872fbea 100644 --- a/arch/powerpc/include/asm/ftrace.h +++ b/arch/powerpc/include/asm/ftrace.h @@ -61,6 +61,7 @@ struct dyn_arch_ftrace { #endif #if defined(CONFIG_FTRACE_SYSCALLS) && defined(CONFIG_PPC64) && !defined(__ASSEMBLY__) +#if !defined(_CALL_ELF) || _CALL_ELF != 2 #define ARCH_HAS_SYSCALL_MATCH_SYM_NAME static inline bool arch_syscall_match_sym_name(const char *sym, const char *name) { @@ -72,6 +73,7 @@ static inline bool arch_syscall_match_sym_name(const char *sym, const char *name */ return !strcmp(sym + 4, name + 3); } +#endif #endif /* CONFIG_FTRACE_SYSCALLS && CONFIG_PPC64 && !__ASSEMBLY__ */ #endif /* _ASM_POWERPC_FTRACE */ diff --git a/arch/powerpc/include/asm/hw_breakpoint.h b/arch/powerpc/include/asm/hw_breakpoint.h index eb0f4ac75c4c..ac6432d9be46 100644 --- a/arch/powerpc/include/asm/hw_breakpoint.h +++ b/arch/powerpc/include/asm/hw_breakpoint.h @@ -79,7 +79,7 @@ static inline void hw_breakpoint_disable(void) brk.address = 0; brk.type = 0; brk.len = 0; - set_breakpoint(&brk); + __set_breakpoint(&brk); } extern void thread_change_pc(struct task_struct *tsk, struct pt_regs *regs); diff --git a/arch/powerpc/include/asm/irqflags.h b/arch/powerpc/include/asm/irqflags.h index f51a5580bfd0..e20eb95429a8 100644 --- a/arch/powerpc/include/asm/irqflags.h +++ b/arch/powerpc/include/asm/irqflags.h @@ -20,9 +20,9 @@ */ #define TRACE_WITH_FRAME_BUFFER(func) \ mflr r0; \ - stdu r1, -32(r1); \ + stdu r1, -STACK_FRAME_OVERHEAD(r1); \ std r0, 16(r1); \ - stdu r1, -32(r1); \ + stdu r1, -STACK_FRAME_OVERHEAD(r1); \ bl func; \ ld r1, 0(r1); \ ld r1, 0(r1); @@ -36,8 +36,8 @@ * have to call a C function so call a wrapper that saves all the * C-clobbered registers. */ -#define TRACE_ENABLE_INTS TRACE_WITH_FRAME_BUFFER(.trace_hardirqs_on) -#define TRACE_DISABLE_INTS TRACE_WITH_FRAME_BUFFER(.trace_hardirqs_off) +#define TRACE_ENABLE_INTS TRACE_WITH_FRAME_BUFFER(trace_hardirqs_on) +#define TRACE_DISABLE_INTS TRACE_WITH_FRAME_BUFFER(trace_hardirqs_off) /* * This is used by assembly code to soft-disable interrupts first and diff --git a/arch/powerpc/include/asm/kprobes.h b/arch/powerpc/include/asm/kprobes.h index 7b6feab6fd26..af15d4d8d604 100644 --- a/arch/powerpc/include/asm/kprobes.h +++ b/arch/powerpc/include/asm/kprobes.h @@ -30,6 +30,7 @@ #include <linux/ptrace.h> #include <linux/percpu.h> #include <asm/probes.h> +#include <asm/code-patching.h> #define __ARCH_WANT_KPROBES_INSN_SLOT @@ -56,9 +57,9 @@ typedef ppc_opcode_t kprobe_opcode_t; if ((colon = strchr(name, ':')) != NULL) { \ colon++; \ if (*colon != '\0' && *colon != '.') \ - addr = *(kprobe_opcode_t **)addr; \ + addr = (kprobe_opcode_t *)ppc_function_entry(addr); \ } else if (name[0] != '.') \ - addr = *(kprobe_opcode_t **)addr; \ + addr = (kprobe_opcode_t *)ppc_function_entry(addr); \ } else { \ char dot_name[KSYM_NAME_LEN]; \ dot_name[0] = '.'; \ diff --git a/arch/powerpc/include/asm/kvm_ppc.h b/arch/powerpc/include/asm/kvm_ppc.h index 4a7cc453be0b..9c89cdd067a6 100644 --- a/arch/powerpc/include/asm/kvm_ppc.h +++ b/arch/powerpc/include/asm/kvm_ppc.h @@ -337,6 +337,10 @@ static inline void kvmppc_fast_vcpu_kick(struct kvm_vcpu *vcpu) vcpu->kvm->arch.kvm_ops->fast_vcpu_kick(vcpu); } +extern void kvm_hv_vm_activated(void); +extern void kvm_hv_vm_deactivated(void); +extern bool kvm_hv_mode_active(void); + #else static inline void __init kvm_cma_reserve(void) {} @@ -356,6 +360,9 @@ static inline void kvmppc_fast_vcpu_kick(struct kvm_vcpu *vcpu) { kvm_vcpu_kick(vcpu); } + +static inline bool kvm_hv_mode_active(void) { return false; } + #endif #ifdef CONFIG_KVM_XICS diff --git a/arch/powerpc/include/asm/linkage.h b/arch/powerpc/include/asm/linkage.h index b36f650a13ff..e3ad5c72724a 100644 --- a/arch/powerpc/include/asm/linkage.h +++ b/arch/powerpc/include/asm/linkage.h @@ -2,6 +2,7 @@ #define _ASM_POWERPC_LINKAGE_H #ifdef CONFIG_PPC64 +#if !defined(_CALL_ELF) || _CALL_ELF != 2 #define cond_syscall(x) \ asm ("\t.weak " #x "\n\t.set " #x ", sys_ni_syscall\n" \ "\t.weak ." #x "\n\t.set ." #x ", .sys_ni_syscall\n") @@ -9,5 +10,6 @@ asm ("\t.globl " #alias "\n\t.set " #alias ", " #name "\n" \ "\t.globl ." #alias "\n\t.set ." #alias ", ." #name) #endif +#endif #endif /* _ASM_POWERPC_LINKAGE_H */ diff --git a/arch/powerpc/include/asm/machdep.h b/arch/powerpc/include/asm/machdep.h index 5b6c03f1058f..f92b0b54e921 100644 --- a/arch/powerpc/include/asm/machdep.h +++ b/arch/powerpc/include/asm/machdep.h @@ -98,6 +98,9 @@ struct machdep_calls { void (*iommu_save)(void); void (*iommu_restore)(void); #endif +#ifdef CONFIG_MEMORY_HOTPLUG_SPARSE + unsigned long (*memory_block_size)(void); +#endif #endif /* CONFIG_PPC64 */ void (*pci_dma_dev_setup)(struct pci_dev *dev); @@ -113,6 +116,8 @@ struct machdep_calls { /* Optional, may be NULL. */ void (*show_cpuinfo)(struct seq_file *m); void (*show_percpuinfo)(struct seq_file *m, int i); + /* Returns the current operating frequency of "cpu" in Hz */ + unsigned long (*get_proc_freq)(unsigned int cpu); void (*init_IRQ)(void); @@ -241,6 +246,9 @@ struct machdep_calls { /* Called during PCI resource reassignment */ resource_size_t (*pcibios_window_alignment)(struct pci_bus *, unsigned long type); + /* Reset the secondary bus of bridge */ + void (*pcibios_reset_secondary_bus)(struct pci_dev *dev); + /* Called to shutdown machine specific hardware not already controlled * by other drivers. */ diff --git a/arch/powerpc/include/asm/module.h b/arch/powerpc/include/asm/module.h index 49fa55bfbac4..dcfcad139bcc 100644 --- a/arch/powerpc/include/asm/module.h +++ b/arch/powerpc/include/asm/module.h @@ -35,6 +35,7 @@ struct mod_arch_specific { #ifdef __powerpc64__ unsigned int stubs_section; /* Index of stubs section in module */ unsigned int toc_section; /* What section is the TOC? */ + bool toc_fixed; /* Have we fixed up .TOC.? */ #ifdef CONFIG_DYNAMIC_FTRACE unsigned long toc; unsigned long tramp; @@ -77,6 +78,9 @@ struct mod_arch_specific { # endif /* MODULE */ #endif +bool is_module_trampoline(u32 *insns); +int module_trampoline_target(struct module *mod, u32 *trampoline, + unsigned long *target); struct exception_table_entry; void sort_ex_table(struct exception_table_entry *start, diff --git a/arch/powerpc/include/asm/opal.h b/arch/powerpc/include/asm/opal.h index 66ad7a74116f..cb15cbb51600 100644 --- a/arch/powerpc/include/asm/opal.h +++ b/arch/powerpc/include/asm/opal.h @@ -154,6 +154,7 @@ extern int opal_enter_rtas(struct rtas_args *args, #define OPAL_LPC_READ 67 #define OPAL_LPC_WRITE 68 #define OPAL_RETURN_CPU 69 +#define OPAL_REINIT_CPUS 70 #define OPAL_ELOG_READ 71 #define OPAL_ELOG_WRITE 72 #define OPAL_ELOG_ACK 73 @@ -509,7 +510,7 @@ enum OpalMemErr_DynErrType { struct OpalMemoryErrorData { enum OpalMemErr_Version version:8; /* 0x00 */ enum OpalMemErrType type:8; /* 0x01 */ - uint16_t flags; /* 0x02 */ + __be16 flags; /* 0x02 */ uint8_t reserved_1[4]; /* 0x04 */ union { @@ -517,15 +518,15 @@ struct OpalMemoryErrorData { struct { enum OpalMemErr_ResilErrType resil_err_type:8; uint8_t reserved_1[7]; - uint64_t physical_address_start; - uint64_t physical_address_end; + __be64 physical_address_start; + __be64 physical_address_end; } resilience; /* Dynamic memory deallocation error info */ struct { enum OpalMemErr_DynErrType dyn_err_type:8; uint8_t reserved_1[7]; - uint64_t physical_address_start; - uint64_t physical_address_end; + __be64 physical_address_start; + __be64 physical_address_end; } dyn_dealloc; } u; }; @@ -725,6 +726,11 @@ struct OpalIoPhb3ErrorData { uint64_t pestB[OPAL_PHB3_NUM_PEST_REGS]; }; +enum { + OPAL_REINIT_CPUS_HILE_BE = (1 << 0), + OPAL_REINIT_CPUS_HILE_LE = (1 << 1), +}; + typedef struct oppanel_line { const char * line; uint64_t line_len; @@ -849,6 +855,7 @@ int64_t opal_pci_next_error(uint64_t phb_id, uint64_t *first_frozen_pe, uint16_t *pci_error_type, uint16_t *severity); int64_t opal_pci_poll(uint64_t phb_id); int64_t opal_return_cpu(void); +int64_t opal_reinit_cpus(uint64_t flags); int64_t opal_xscom_read(uint32_t gcid, uint64_t pcb_addr, __be64 *val); int64_t opal_xscom_write(uint32_t gcid, uint64_t pcb_addr, uint64_t val); @@ -916,6 +923,7 @@ extern void opal_get_rtc_time(struct rtc_time *tm); extern unsigned long opal_get_boot_time(void); extern void opal_nvram_init(void); extern void opal_flash_init(void); +extern void opal_flash_term_callback(void); extern int opal_elog_init(void); extern void opal_platform_dump_init(void); extern void opal_sys_param_init(void); diff --git a/arch/powerpc/include/asm/paca.h b/arch/powerpc/include/asm/paca.h index 8e956a0b6e85..bb0bd25f20d0 100644 --- a/arch/powerpc/include/asm/paca.h +++ b/arch/powerpc/include/asm/paca.h @@ -92,7 +92,10 @@ struct paca_struct { struct slb_shadow *slb_shadow_ptr; struct dtl_entry *dispatch_log; struct dtl_entry *dispatch_log_end; +#endif /* CONFIG_PPC_STD_MMU_64 */ + u64 dscr_default; /* per-CPU default DSCR */ +#ifdef CONFIG_PPC_STD_MMU_64 /* * Now, starting in cacheline 2, the exception save areas */ diff --git a/arch/powerpc/include/asm/ppc-pci.h b/arch/powerpc/include/asm/ppc-pci.h index ed57fa7920c8..db1e2b8eff3c 100644 --- a/arch/powerpc/include/asm/ppc-pci.h +++ b/arch/powerpc/include/asm/ppc-pci.h @@ -58,6 +58,7 @@ int rtas_write_config(struct pci_dn *, int where, int size, u32 val); int rtas_read_config(struct pci_dn *, int where, int size, u32 *val); void eeh_pe_state_mark(struct eeh_pe *pe, int state); void eeh_pe_state_clear(struct eeh_pe *pe, int state); +void eeh_pe_dev_mode_mark(struct eeh_pe *pe, int mode); void eeh_sysfs_add_device(struct pci_dev *pdev); void eeh_sysfs_remove_device(struct pci_dev *pdev); diff --git a/arch/powerpc/include/asm/ppc_asm.h b/arch/powerpc/include/asm/ppc_asm.h index cded7c1278ef..9ea266eae33e 100644 --- a/arch/powerpc/include/asm/ppc_asm.h +++ b/arch/powerpc/include/asm/ppc_asm.h @@ -57,7 +57,7 @@ BEGIN_FW_FTR_SECTION; \ LDX_BE r10,0,r10; /* get log write index */ \ cmpd cr1,r11,r10; \ beq+ cr1,33f; \ - bl .accumulate_stolen_time; \ + bl accumulate_stolen_time; \ ld r12,_MSR(r1); \ andi. r10,r12,MSR_PR; /* Restore cr0 (coming from user) */ \ 33: \ @@ -189,57 +189,53 @@ END_FW_FTR_SECTION_IFSET(FW_FEATURE_SPLPAR) #define __STK_REG(i) (112 + ((i)-14)*8) #define STK_REG(i) __STK_REG(__REG_##i) +#if defined(_CALL_ELF) && _CALL_ELF == 2 +#define STK_GOT 24 +#define __STK_PARAM(i) (32 + ((i)-3)*8) +#else +#define STK_GOT 40 #define __STK_PARAM(i) (48 + ((i)-3)*8) +#endif #define STK_PARAM(i) __STK_PARAM(__REG_##i) -#define XGLUE(a,b) a##b -#define GLUE(a,b) XGLUE(a,b) +#if defined(_CALL_ELF) && _CALL_ELF == 2 #define _GLOBAL(name) \ .section ".text"; \ .align 2 ; \ + .type name,@function; \ .globl name; \ - .globl GLUE(.,name); \ - .section ".opd","aw"; \ -name: \ - .quad GLUE(.,name); \ - .quad .TOC.@tocbase; \ - .quad 0; \ - .previous; \ - .type GLUE(.,name),@function; \ -GLUE(.,name): +name: -#define _INIT_GLOBAL(name) \ - __REF; \ +#define _GLOBAL_TOC(name) \ + .section ".text"; \ .align 2 ; \ + .type name,@function; \ .globl name; \ - .globl GLUE(.,name); \ - .section ".opd","aw"; \ name: \ - .quad GLUE(.,name); \ - .quad .TOC.@tocbase; \ - .quad 0; \ - .previous; \ - .type GLUE(.,name),@function; \ -GLUE(.,name): +0: addis r2,r12,(.TOC.-0b)@ha; \ + addi r2,r2,(.TOC.-0b)@l; \ + .localentry name,.-name #define _KPROBE(name) \ .section ".kprobes.text","a"; \ .align 2 ; \ + .type name,@function; \ .globl name; \ - .globl GLUE(.,name); \ - .section ".opd","aw"; \ -name: \ - .quad GLUE(.,name); \ - .quad .TOC.@tocbase; \ - .quad 0; \ - .previous; \ - .type GLUE(.,name),@function; \ -GLUE(.,name): +name: + +#define DOTSYM(a) a + +#else + +#define XGLUE(a,b) a##b +#define GLUE(a,b) XGLUE(a,b) -#define _STATIC(name) \ +#define _GLOBAL(name) \ .section ".text"; \ .align 2 ; \ + .globl name; \ + .globl GLUE(.,name); \ .section ".opd","aw"; \ name: \ .quad GLUE(.,name); \ @@ -249,9 +245,13 @@ name: \ .type GLUE(.,name),@function; \ GLUE(.,name): -#define _INIT_STATIC(name) \ - __REF; \ +#define _GLOBAL_TOC(name) _GLOBAL(name) + +#define _KPROBE(name) \ + .section ".kprobes.text","a"; \ .align 2 ; \ + .globl name; \ + .globl GLUE(.,name); \ .section ".opd","aw"; \ name: \ .quad GLUE(.,name); \ @@ -261,6 +261,10 @@ name: \ .type GLUE(.,name),@function; \ GLUE(.,name): +#define DOTSYM(a) GLUE(.,a) + +#endif + #else /* 32-bit */ #define _ENTRY(n) \ diff --git a/arch/powerpc/include/asm/processor.h b/arch/powerpc/include/asm/processor.h index d660dc36831a..6d59072e13a7 100644 --- a/arch/powerpc/include/asm/processor.h +++ b/arch/powerpc/include/asm/processor.h @@ -449,7 +449,7 @@ extern unsigned long cpuidle_disable; enum idle_boot_override {IDLE_NO_OVERRIDE = 0, IDLE_POWERSAVE_OFF}; extern int powersave_nap; /* set if nap mode can be used in idle loop */ -extern void power7_nap(void); +extern void power7_nap(int check_irq); extern void power7_sleep(void); extern void flush_instruction_cache(void); extern void hard_reset_now(void); diff --git a/arch/powerpc/include/asm/reg.h b/arch/powerpc/include/asm/reg.h index 4852bcf270f3..bffd89d27301 100644 --- a/arch/powerpc/include/asm/reg.h +++ b/arch/powerpc/include/asm/reg.h @@ -215,6 +215,7 @@ #define SPRN_TEXASR 0x82 /* Transaction EXception & Summary */ #define TEXASR_FS __MASK(63-36) /* Transaction Failure Summary */ #define SPRN_TEXASRU 0x83 /* '' '' '' Upper 32 */ +#define TEXASR_FS __MASK(63-36) /* TEXASR Failure Summary */ #define SPRN_TFHAR 0x80 /* Transaction Failure Handler Addr */ #define SPRN_CTRLF 0x088 #define SPRN_CTRLT 0x098 @@ -224,6 +225,7 @@ #define CTRL_TE 0x00c00000 /* thread enable */ #define CTRL_RUNLATCH 0x1 #define SPRN_DAWR 0xB4 +#define SPRN_RPR 0xBA /* Relative Priority Register */ #define SPRN_CIABR 0xBB #define CIABR_PRIV 0x3 #define CIABR_PRIV_USER 1 @@ -272,8 +274,10 @@ #define SPRN_HSRR1 0x13B /* Hypervisor Save/Restore 1 */ #define SPRN_IC 0x350 /* Virtual Instruction Count */ #define SPRN_VTB 0x351 /* Virtual Time Base */ +#define SPRN_LDBAR 0x352 /* LD Base Address Register */ #define SPRN_PMICR 0x354 /* Power Management Idle Control Reg */ #define SPRN_PMSR 0x355 /* Power Management Status Reg */ +#define SPRN_PMMAR 0x356 /* Power Management Memory Activity Register */ #define SPRN_PMCR 0x374 /* Power Management Control Register */ /* HFSCR and FSCR bit numbers are the same */ @@ -433,6 +437,12 @@ #define HID0_BTCD (1<<1) /* Branch target cache disable */ #define HID0_NOPDST (1<<1) /* No-op dst, dstt, etc. instr. */ #define HID0_NOPTI (1<<0) /* No-op dcbt and dcbst instr. */ +/* POWER8 HID0 bits */ +#define HID0_POWER8_4LPARMODE __MASK(61) +#define HID0_POWER8_2LPARMODE __MASK(57) +#define HID0_POWER8_1TO2LPAR __MASK(52) +#define HID0_POWER8_1TO4LPAR __MASK(51) +#define HID0_POWER8_DYNLPARDIS __MASK(48) #define SPRN_HID1 0x3F1 /* Hardware Implementation Register 1 */ #ifdef CONFIG_6xx diff --git a/arch/powerpc/include/asm/sections.h b/arch/powerpc/include/asm/sections.h index 521790330672..a5e930aca804 100644 --- a/arch/powerpc/include/asm/sections.h +++ b/arch/powerpc/include/asm/sections.h @@ -50,6 +50,7 @@ static inline int overlaps_kvm_tmp(unsigned long start, unsigned long end) #endif } +#if !defined(_CALL_ELF) || _CALL_ELF != 2 #undef dereference_function_descriptor static inline void *dereference_function_descriptor(void *ptr) { @@ -60,6 +61,7 @@ static inline void *dereference_function_descriptor(void *ptr) ptr = p; return ptr; } +#endif #endif diff --git a/arch/powerpc/include/asm/smp.h b/arch/powerpc/include/asm/smp.h index ff51046b6466..5a6614a7f0b2 100644 --- a/arch/powerpc/include/asm/smp.h +++ b/arch/powerpc/include/asm/smp.h @@ -68,14 +68,6 @@ void generic_mach_cpu_die(void); void generic_set_cpu_dead(unsigned int cpu); void generic_set_cpu_up(unsigned int cpu); int generic_check_cpu_restart(unsigned int cpu); - -extern void inhibit_secondary_onlining(void); -extern void uninhibit_secondary_onlining(void); - -#else /* HOTPLUG_CPU */ -static inline void inhibit_secondary_onlining(void) {} -static inline void uninhibit_secondary_onlining(void) {} - #endif #ifdef CONFIG_PPC64 diff --git a/arch/powerpc/include/asm/string.h b/arch/powerpc/include/asm/string.h index 0dffad6bcc84..e40010abcaf1 100644 --- a/arch/powerpc/include/asm/string.h +++ b/arch/powerpc/include/asm/string.h @@ -10,9 +10,7 @@ #define __HAVE_ARCH_STRNCMP #define __HAVE_ARCH_STRCAT #define __HAVE_ARCH_MEMSET -#ifdef __BIG_ENDIAN__ #define __HAVE_ARCH_MEMCPY -#endif #define __HAVE_ARCH_MEMMOVE #define __HAVE_ARCH_MEMCMP #define __HAVE_ARCH_MEMCHR @@ -24,9 +22,7 @@ extern int strcmp(const char *,const char *); extern int strncmp(const char *, const char *, __kernel_size_t); extern char * strcat(char *, const char *); extern void * memset(void *,int,__kernel_size_t); -#ifdef __BIG_ENDIAN__ extern void * memcpy(void *,const void *,__kernel_size_t); -#endif extern void * memmove(void *,const void *,__kernel_size_t); extern int memcmp(const void *,const void *,__kernel_size_t); extern void * memchr(const void *,int,__kernel_size_t); diff --git a/arch/powerpc/include/asm/systbl.h b/arch/powerpc/include/asm/systbl.h index ea4dc3a89c1f..babbeca6850f 100644 --- a/arch/powerpc/include/asm/systbl.h +++ b/arch/powerpc/include/asm/systbl.h @@ -62,7 +62,7 @@ COMPAT_SYS_SPU(fcntl) SYSCALL(ni_syscall) SYSCALL_SPU(setpgid) SYSCALL(ni_syscall) -SYSX(sys_ni_syscall,sys_olduname, sys_olduname) +SYSX(sys_ni_syscall,sys_olduname,sys_olduname) SYSCALL_SPU(umask) SYSCALL_SPU(chroot) COMPAT_SYS(ustat) @@ -190,7 +190,7 @@ SYSCALL_SPU(getcwd) SYSCALL_SPU(capget) SYSCALL_SPU(capset) COMPAT_SYS(sigaltstack) -COMPAT_SYS_SPU(sendfile) +SYSX_SPU(sys_sendfile64,compat_sys_sendfile,sys_sendfile) SYSCALL(ni_syscall) SYSCALL(ni_syscall) PPC_SYS(vfork) @@ -258,7 +258,7 @@ SYSCALL_SPU(tgkill) COMPAT_SYS_SPU(utimes) COMPAT_SYS_SPU(statfs64) COMPAT_SYS_SPU(fstatfs64) -SYSX(sys_ni_syscall, ppc_fadvise64_64, ppc_fadvise64_64) +SYSX(sys_ni_syscall,ppc_fadvise64_64,ppc_fadvise64_64) PPC_SYS_SPU(rtas) OLDSYS(debug_setcontext) SYSCALL(ni_syscall) @@ -295,7 +295,7 @@ SYSCALL_SPU(mkdirat) SYSCALL_SPU(mknodat) SYSCALL_SPU(fchownat) COMPAT_SYS_SPU(futimesat) -SYSX_SPU(sys_newfstatat, sys_fstatat64, sys_fstatat64) +SYSX_SPU(sys_newfstatat,sys_fstatat64,sys_fstatat64) SYSCALL_SPU(unlinkat) SYSCALL_SPU(renameat) SYSCALL_SPU(linkat) diff --git a/arch/powerpc/include/asm/topology.h b/arch/powerpc/include/asm/topology.h index 6c8a8c5a37a1..5f1048eaa5b6 100644 --- a/arch/powerpc/include/asm/topology.h +++ b/arch/powerpc/include/asm/topology.h @@ -16,19 +16,6 @@ struct device_node; #include <asm/mmzone.h> -static inline int cpu_to_node(int cpu) -{ - int nid; - - nid = numa_cpu_lookup_table[cpu]; - - /* - * During early boot, the numa-cpu lookup table might not have been - * setup for all CPUs yet. In such cases, default to node 0. - */ - return (nid < 0) ? 0 : nid; -} - #define parent_node(node) (node) #define cpumask_of_node(node) ((node) == -1 ? \ diff --git a/arch/powerpc/include/uapi/asm/Kbuild b/arch/powerpc/include/uapi/asm/Kbuild index 48be855ef37b..7a3f795ac218 100644 --- a/arch/powerpc/include/uapi/asm/Kbuild +++ b/arch/powerpc/include/uapi/asm/Kbuild @@ -15,7 +15,6 @@ header-y += ioctls.h header-y += ipcbuf.h header-y += kvm.h header-y += kvm_para.h -header-y += linkage.h header-y += mman.h header-y += msgbuf.h header-y += nvram.h diff --git a/arch/powerpc/include/uapi/asm/elf.h b/arch/powerpc/include/uapi/asm/elf.h index 7e39c9146a71..59dad113897b 100644 --- a/arch/powerpc/include/uapi/asm/elf.h +++ b/arch/powerpc/include/uapi/asm/elf.h @@ -291,9 +291,17 @@ do { \ #define R_PPC64_DTPREL16_HIGHERA 104 /* half16 (sym+add)@dtprel@highera */ #define R_PPC64_DTPREL16_HIGHEST 105 /* half16 (sym+add)@dtprel@highest */ #define R_PPC64_DTPREL16_HIGHESTA 106 /* half16 (sym+add)@dtprel@highesta */ +#define R_PPC64_TLSGD 107 +#define R_PPC64_TLSLD 108 +#define R_PPC64_TOCSAVE 109 + +#define R_PPC64_REL16 249 +#define R_PPC64_REL16_LO 250 +#define R_PPC64_REL16_HI 251 +#define R_PPC64_REL16_HA 252 /* Keep this the last entry. */ -#define R_PPC64_NUM 107 +#define R_PPC64_NUM 253 /* There's actually a third entry here, but it's unused */ struct ppc64_opd_entry diff --git a/arch/powerpc/kernel/asm-offsets.c b/arch/powerpc/kernel/asm-offsets.c index 93e1465c8496..f5995a912213 100644 --- a/arch/powerpc/kernel/asm-offsets.c +++ b/arch/powerpc/kernel/asm-offsets.c @@ -248,6 +248,7 @@ int main(void) #endif DEFINE(PACAHWCPUID, offsetof(struct paca_struct, hw_cpu_id)); DEFINE(PACAKEXECSTATE, offsetof(struct paca_struct, kexec_state)); + DEFINE(PACA_DSCR, offsetof(struct paca_struct, dscr_default)); DEFINE(PACA_STARTTIME, offsetof(struct paca_struct, starttime)); DEFINE(PACA_STARTTIME_USER, offsetof(struct paca_struct, starttime_user)); DEFINE(PACA_USER_TIME, offsetof(struct paca_struct, user_time)); diff --git a/arch/powerpc/kernel/cpu_setup_fsl_booke.S b/arch/powerpc/kernel/cpu_setup_fsl_booke.S index cc2d8962e090..4f1393d20079 100644 --- a/arch/powerpc/kernel/cpu_setup_fsl_booke.S +++ b/arch/powerpc/kernel/cpu_setup_fsl_booke.S @@ -94,12 +94,12 @@ _GLOBAL(setup_altivec_idle) _GLOBAL(__setup_cpu_e6500) mflr r6 #ifdef CONFIG_PPC64 - bl .setup_altivec_ivors + bl setup_altivec_ivors /* Touch IVOR42 only if the CPU supports E.HV category */ mfspr r10,SPRN_MMUCFG rlwinm. r10,r10,0,MMUCFG_LPIDSIZE beq 1f - bl .setup_lrat_ivor + bl setup_lrat_ivor 1: #endif bl setup_pw20_idle @@ -164,15 +164,15 @@ _GLOBAL(__setup_cpu_e5500) #ifdef CONFIG_PPC_BOOK3E_64 _GLOBAL(__restore_cpu_e6500) mflr r5 - bl .setup_altivec_ivors + bl setup_altivec_ivors /* Touch IVOR42 only if the CPU supports E.HV category */ mfspr r10,SPRN_MMUCFG rlwinm. r10,r10,0,MMUCFG_LPIDSIZE beq 1f - bl .setup_lrat_ivor + bl setup_lrat_ivor 1: - bl .setup_pw20_idle - bl .setup_altivec_idle + bl setup_pw20_idle + bl setup_altivec_idle bl __restore_cpu_e5500 mtlr r5 blr @@ -181,9 +181,9 @@ _GLOBAL(__restore_cpu_e5500) mflr r4 bl __e500_icache_setup bl __e500_dcache_setup - bl .__setup_base_ivors - bl .setup_perfmon_ivor - bl .setup_doorbell_ivors + bl __setup_base_ivors + bl setup_perfmon_ivor + bl setup_doorbell_ivors /* * We only want to touch IVOR38-41 if we're running on hardware * that supports category E.HV. The architectural way to determine @@ -192,7 +192,7 @@ _GLOBAL(__restore_cpu_e5500) mfspr r10,SPRN_MMUCFG rlwinm. r10,r10,0,MMUCFG_LPIDSIZE beq 1f - bl .setup_ehv_ivors + bl setup_ehv_ivors 1: mtlr r4 blr @@ -201,9 +201,9 @@ _GLOBAL(__setup_cpu_e5500) mflr r5 bl __e500_icache_setup bl __e500_dcache_setup - bl .__setup_base_ivors - bl .setup_perfmon_ivor - bl .setup_doorbell_ivors + bl __setup_base_ivors + bl setup_perfmon_ivor + bl setup_doorbell_ivors /* * We only want to touch IVOR38-41 if we're running on hardware * that supports category E.HV. The architectural way to determine @@ -212,7 +212,7 @@ _GLOBAL(__setup_cpu_e5500) mfspr r10,SPRN_MMUCFG rlwinm. r10,r10,0,MMUCFG_LPIDSIZE beq 1f - bl .setup_ehv_ivors + bl setup_ehv_ivors b 2f 1: ld r10,CPU_SPEC_FEATURES(r4) diff --git a/arch/powerpc/kernel/eeh.c b/arch/powerpc/kernel/eeh.c index e7b76a6bf150..7051ea3101b9 100644 --- a/arch/powerpc/kernel/eeh.c +++ b/arch/powerpc/kernel/eeh.c @@ -22,6 +22,7 @@ */ #include <linux/delay.h> +#include <linux/debugfs.h> #include <linux/sched.h> #include <linux/init.h> #include <linux/list.h> @@ -35,6 +36,7 @@ #include <linux/of.h> #include <linux/atomic.h> +#include <asm/debug.h> #include <asm/eeh.h> #include <asm/eeh_event.h> #include <asm/io.h> @@ -87,22 +89,21 @@ /* Time to wait for a PCI slot to report status, in milliseconds */ #define PCI_BUS_RESET_WAIT_MSEC (5*60*1000) -/* Platform dependent EEH operations */ -struct eeh_ops *eeh_ops = NULL; - -bool eeh_subsystem_enabled = false; -EXPORT_SYMBOL(eeh_subsystem_enabled); - /* - * EEH probe mode support. The intention is to support multiple - * platforms for EEH. Some platforms like pSeries do PCI emunation - * based on device tree. However, other platforms like powernv probe - * PCI devices from hardware. The flag is used to distinguish that. - * In addition, struct eeh_ops::probe would be invoked for particular - * OF node or PCI device so that the corresponding PE would be created - * there. + * EEH probe mode support, which is part of the flags, + * is to support multiple platforms for EEH. Some platforms + * like pSeries do PCI emunation based on device tree. + * However, other platforms like powernv probe PCI devices + * from hardware. The flag is used to distinguish that. + * In addition, struct eeh_ops::probe would be invoked for + * particular OF node or PCI device so that the corresponding + * PE would be created there. */ -int eeh_probe_mode; +int eeh_subsystem_flags; +EXPORT_SYMBOL(eeh_subsystem_flags); + +/* Platform dependent EEH operations */ +struct eeh_ops *eeh_ops = NULL; /* Lock to avoid races due to multiple reports of an error */ DEFINE_RAW_SPINLOCK(confirm_error_lock); @@ -133,6 +134,15 @@ static struct eeh_stats eeh_stats; #define IS_BRIDGE(class_code) (((class_code)<<16) == PCI_BASE_CLASS_BRIDGE) +static int __init eeh_setup(char *str) +{ + if (!strcmp(str, "off")) + eeh_subsystem_flags |= EEH_FORCE_DISABLED; + + return 1; +} +__setup("eeh=", eeh_setup); + /** * eeh_gather_pci_data - Copy assorted PCI config space registers to buff * @edev: device to report data for @@ -145,73 +155,67 @@ static struct eeh_stats eeh_stats; static size_t eeh_gather_pci_data(struct eeh_dev *edev, char * buf, size_t len) { struct device_node *dn = eeh_dev_to_of_node(edev); - struct pci_dev *dev = eeh_dev_to_pci_dev(edev); u32 cfg; int cap, i; int n = 0; n += scnprintf(buf+n, len-n, "%s\n", dn->full_name); - printk(KERN_WARNING "EEH: of node=%s\n", dn->full_name); + pr_warn("EEH: of node=%s\n", dn->full_name); eeh_ops->read_config(dn, PCI_VENDOR_ID, 4, &cfg); n += scnprintf(buf+n, len-n, "dev/vend:%08x\n", cfg); - printk(KERN_WARNING "EEH: PCI device/vendor: %08x\n", cfg); + pr_warn("EEH: PCI device/vendor: %08x\n", cfg); eeh_ops->read_config(dn, PCI_COMMAND, 4, &cfg); n += scnprintf(buf+n, len-n, "cmd/stat:%x\n", cfg); - printk(KERN_WARNING "EEH: PCI cmd/status register: %08x\n", cfg); - - if (!dev) { - printk(KERN_WARNING "EEH: no PCI device for this of node\n"); - return n; - } + pr_warn("EEH: PCI cmd/status register: %08x\n", cfg); /* Gather bridge-specific registers */ - if (dev->class >> 16 == PCI_BASE_CLASS_BRIDGE) { + if (edev->mode & EEH_DEV_BRIDGE) { eeh_ops->read_config(dn, PCI_SEC_STATUS, 2, &cfg); n += scnprintf(buf+n, len-n, "sec stat:%x\n", cfg); - printk(KERN_WARNING "EEH: Bridge secondary status: %04x\n", cfg); + pr_warn("EEH: Bridge secondary status: %04x\n", cfg); eeh_ops->read_config(dn, PCI_BRIDGE_CONTROL, 2, &cfg); n += scnprintf(buf+n, len-n, "brdg ctl:%x\n", cfg); - printk(KERN_WARNING "EEH: Bridge control: %04x\n", cfg); + pr_warn("EEH: Bridge control: %04x\n", cfg); } /* Dump out the PCI-X command and status regs */ - cap = pci_find_capability(dev, PCI_CAP_ID_PCIX); + cap = edev->pcix_cap; if (cap) { eeh_ops->read_config(dn, cap, 4, &cfg); n += scnprintf(buf+n, len-n, "pcix-cmd:%x\n", cfg); - printk(KERN_WARNING "EEH: PCI-X cmd: %08x\n", cfg); + pr_warn("EEH: PCI-X cmd: %08x\n", cfg); eeh_ops->read_config(dn, cap+4, 4, &cfg); n += scnprintf(buf+n, len-n, "pcix-stat:%x\n", cfg); - printk(KERN_WARNING "EEH: PCI-X status: %08x\n", cfg); + pr_warn("EEH: PCI-X status: %08x\n", cfg); } - /* If PCI-E capable, dump PCI-E cap 10, and the AER */ - if (pci_is_pcie(dev)) { + /* If PCI-E capable, dump PCI-E cap 10 */ + cap = edev->pcie_cap; + if (cap) { n += scnprintf(buf+n, len-n, "pci-e cap10:\n"); - printk(KERN_WARNING - "EEH: PCI-E capabilities and status follow:\n"); + pr_warn("EEH: PCI-E capabilities and status follow:\n"); for (i=0; i<=8; i++) { - eeh_ops->read_config(dn, dev->pcie_cap+4*i, 4, &cfg); + eeh_ops->read_config(dn, cap+4*i, 4, &cfg); n += scnprintf(buf+n, len-n, "%02x:%x\n", 4*i, cfg); - printk(KERN_WARNING "EEH: PCI-E %02x: %08x\n", i, cfg); + pr_warn("EEH: PCI-E %02x: %08x\n", i, cfg); } + } - cap = pci_find_ext_capability(dev, PCI_EXT_CAP_ID_ERR); - if (cap) { - n += scnprintf(buf+n, len-n, "pci-e AER:\n"); - printk(KERN_WARNING - "EEH: PCI-E AER capability register set follows:\n"); - - for (i=0; i<14; i++) { - eeh_ops->read_config(dn, cap+4*i, 4, &cfg); - n += scnprintf(buf+n, len-n, "%02x:%x\n", 4*i, cfg); - printk(KERN_WARNING "EEH: PCI-E AER %02x: %08x\n", i, cfg); - } + /* If AER capable, dump it */ + cap = edev->aer_cap; + if (cap) { + n += scnprintf(buf+n, len-n, "pci-e AER:\n"); + pr_warn("EEH: PCI-E AER capability register set follows:\n"); + + for (i=0; i<14; i++) { + eeh_ops->read_config(dn, cap+4*i, 4, &cfg); + n += scnprintf(buf+n, len-n, "%02x:%x\n", 4*i, cfg); + pr_warn("EEH: PCI-E AER %02x: %08x\n", i, cfg); } } @@ -232,21 +236,19 @@ void eeh_slot_error_detail(struct eeh_pe *pe, int severity) { size_t loglen = 0; struct eeh_dev *edev, *tmp; - bool valid_cfg_log = true; /* * When the PHB is fenced or dead, it's pointless to collect * the data from PCI config space because it should return * 0xFF's. For ER, we still retrieve the data from the PCI * config space. + * + * For pHyp, we have to enable IO for log retrieval. Otherwise, + * 0xFF's is always returned from PCI config space. */ - if (eeh_probe_mode_dev() && - (pe->type & EEH_PE_PHB) && - (pe->state & (EEH_PE_ISOLATED | EEH_PE_PHB_DEAD))) - valid_cfg_log = false; - - if (valid_cfg_log) { - eeh_pci_enable(pe, EEH_OPT_THAW_MMIO); + if (!(pe->type & EEH_PE_PHB)) { + if (eeh_probe_mode_devtree()) + eeh_pci_enable(pe, EEH_OPT_THAW_MMIO); eeh_ops->configure_bridge(pe); eeh_pe_restore_bars(pe); @@ -309,7 +311,7 @@ static int eeh_phb_check_failure(struct eeh_pe *pe) /* If the PHB has been in problematic state */ eeh_serialize_lock(&flags); - if (phb_pe->state & (EEH_PE_ISOLATED | EEH_PE_PHB_DEAD)) { + if (phb_pe->state & EEH_PE_ISOLATED) { ret = 0; goto out; } @@ -515,16 +517,42 @@ EXPORT_SYMBOL(eeh_check_failure); */ int eeh_pci_enable(struct eeh_pe *pe, int function) { - int rc; + int rc, flags = (EEH_STATE_MMIO_ACTIVE | EEH_STATE_DMA_ACTIVE); + + /* + * pHyp doesn't allow to enable IO or DMA on unfrozen PE. + * Also, it's pointless to enable them on unfrozen PE. So + * we have the check here. + */ + if (function == EEH_OPT_THAW_MMIO || + function == EEH_OPT_THAW_DMA) { + rc = eeh_ops->get_state(pe, NULL); + if (rc < 0) + return rc; + + /* Needn't to enable or already enabled */ + if ((rc == EEH_STATE_NOT_SUPPORT) || + ((rc & flags) == flags)) + return 0; + } rc = eeh_ops->set_option(pe, function); if (rc) - pr_warning("%s: Unexpected state change %d on PHB#%d-PE#%x, err=%d\n", - __func__, function, pe->phb->global_number, pe->addr, rc); + pr_warn("%s: Unexpected state change %d on " + "PHB#%d-PE#%x, err=%d\n", + __func__, function, pe->phb->global_number, + pe->addr, rc); rc = eeh_ops->wait_state(pe, PCI_BUS_RESET_WAIT_MSEC); - if (rc > 0 && (rc & EEH_STATE_MMIO_ENABLED) && - (function == EEH_OPT_THAW_MMIO)) + if (rc <= 0) + return rc; + + if ((function == EEH_OPT_THAW_MMIO) && + (rc & EEH_STATE_MMIO_ENABLED)) + return 0; + + if ((function == EEH_OPT_THAW_DMA) && + (rc & EEH_STATE_DMA_ENABLED)) return 0; return rc; @@ -612,26 +640,7 @@ static void eeh_reset_pe_once(struct eeh_pe *pe) else eeh_ops->reset(pe, EEH_RESET_HOT); - /* The PCI bus requires that the reset be held high for at least - * a 100 milliseconds. We wait a bit longer 'just in case'. - */ -#define PCI_BUS_RST_HOLD_TIME_MSEC 250 - msleep(PCI_BUS_RST_HOLD_TIME_MSEC); - - /* We might get hit with another EEH freeze as soon as the - * pci slot reset line is dropped. Make sure we don't miss - * these, and clear the flag now. - */ - eeh_pe_state_clear(pe, EEH_PE_ISOLATED); - eeh_ops->reset(pe, EEH_RESET_DEACTIVATE); - - /* After a PCI slot has been reset, the PCI Express spec requires - * a 1.5 second idle time for the bus to stabilize, before starting - * up traffic. - */ -#define PCI_BUS_SETTLE_TIME_MSEC 1800 - msleep(PCI_BUS_SETTLE_TIME_MSEC); } /** @@ -651,6 +660,10 @@ int eeh_reset_pe(struct eeh_pe *pe) for (i=0; i<3; i++) { eeh_reset_pe_once(pe); + /* + * EEH_PE_ISOLATED is expected to be removed after + * BAR restore. + */ rc = eeh_ops->wait_state(pe, PCI_BUS_RESET_WAIT_MSEC); if ((rc & flags) == flags) return 0; @@ -826,8 +839,8 @@ int eeh_init(void) &hose_list, list_node) pci_walk_bus(hose->bus, eeh_ops->dev_probe, NULL); } else { - pr_warning("%s: Invalid probe mode %d\n", - __func__, eeh_probe_mode); + pr_warn("%s: Invalid probe mode %x", + __func__, eeh_subsystem_flags); return -EINVAL; } @@ -1102,10 +1115,45 @@ static const struct file_operations proc_eeh_operations = { .release = single_release, }; +#ifdef CONFIG_DEBUG_FS +static int eeh_enable_dbgfs_set(void *data, u64 val) +{ + if (val) + eeh_subsystem_flags &= ~EEH_FORCE_DISABLED; + else + eeh_subsystem_flags |= EEH_FORCE_DISABLED; + + /* Notify the backend */ + if (eeh_ops->post_init) + eeh_ops->post_init(); + + return 0; +} + +static int eeh_enable_dbgfs_get(void *data, u64 *val) +{ + if (eeh_enabled()) + *val = 0x1ul; + else + *val = 0x0ul; + return 0; +} + +DEFINE_SIMPLE_ATTRIBUTE(eeh_enable_dbgfs_ops, eeh_enable_dbgfs_get, + eeh_enable_dbgfs_set, "0x%llx\n"); +#endif + static int __init eeh_init_proc(void) { - if (machine_is(pseries) || machine_is(powernv)) + if (machine_is(pseries) || machine_is(powernv)) { proc_create("powerpc/eeh", 0, NULL, &proc_eeh_operations); +#ifdef CONFIG_DEBUG_FS + debugfs_create_file("eeh_enable", 0600, + powerpc_debugfs_root, NULL, + &eeh_enable_dbgfs_ops); +#endif + } + return 0; } __initcall(eeh_init_proc); diff --git a/arch/powerpc/kernel/eeh_driver.c b/arch/powerpc/kernel/eeh_driver.c index bb61ca58ca6d..7100a5b96e70 100644 --- a/arch/powerpc/kernel/eeh_driver.c +++ b/arch/powerpc/kernel/eeh_driver.c @@ -171,6 +171,15 @@ static void eeh_enable_irq(struct pci_dev *dev) } } +static bool eeh_dev_removed(struct eeh_dev *edev) +{ + /* EEH device removed ? */ + if (!edev || (edev->mode & EEH_DEV_REMOVED)) + return true; + + return false; +} + /** * eeh_report_error - Report pci error to each device driver * @data: eeh device @@ -187,10 +196,8 @@ static void *eeh_report_error(void *data, void *userdata) enum pci_ers_result rc, *res = userdata; struct pci_driver *driver; - /* We might not have the associated PCI device, - * then we should continue for next one. - */ - if (!dev) return NULL; + if (!dev || eeh_dev_removed(edev)) + return NULL; dev->error_state = pci_channel_io_frozen; driver = eeh_pcid_get(dev); @@ -230,6 +237,9 @@ static void *eeh_report_mmio_enabled(void *data, void *userdata) enum pci_ers_result rc, *res = userdata; struct pci_driver *driver; + if (!dev || eeh_dev_removed(edev)) + return NULL; + driver = eeh_pcid_get(dev); if (!driver) return NULL; @@ -267,7 +277,8 @@ static void *eeh_report_reset(void *data, void *userdata) enum pci_ers_result rc, *res = userdata; struct pci_driver *driver; - if (!dev) return NULL; + if (!dev || eeh_dev_removed(edev)) + return NULL; dev->error_state = pci_channel_io_normal; driver = eeh_pcid_get(dev); @@ -307,7 +318,8 @@ static void *eeh_report_resume(void *data, void *userdata) struct pci_dev *dev = eeh_dev_to_pci_dev(edev); struct pci_driver *driver; - if (!dev) return NULL; + if (!dev || eeh_dev_removed(edev)) + return NULL; dev->error_state = pci_channel_io_normal; driver = eeh_pcid_get(dev); @@ -343,7 +355,8 @@ static void *eeh_report_failure(void *data, void *userdata) struct pci_dev *dev = eeh_dev_to_pci_dev(edev); struct pci_driver *driver; - if (!dev) return NULL; + if (!dev || eeh_dev_removed(edev)) + return NULL; dev->error_state = pci_channel_io_perm_failure; driver = eeh_pcid_get(dev); @@ -380,6 +393,16 @@ static void *eeh_rmv_device(void *data, void *userdata) if (!dev || (dev->hdr_type & PCI_HEADER_TYPE_BRIDGE)) return NULL; + /* + * We rely on count-based pcibios_release_device() to + * detach permanently offlined PEs. Unfortunately, that's + * not reliable enough. We might have the permanently + * offlined PEs attached, but we needn't take care of + * them and their child devices. + */ + if (eeh_dev_removed(edev)) + return NULL; + driver = eeh_pcid_get(dev); if (driver) { eeh_pcid_put(dev); @@ -417,6 +440,36 @@ static void *eeh_pe_detach_dev(void *data, void *userdata) return NULL; } +/* + * Explicitly clear PE's frozen state for PowerNV where + * we have frozen PE until BAR restore is completed. It's + * harmless to clear it for pSeries. To be consistent with + * PE reset (for 3 times), we try to clear the frozen state + * for 3 times as well. + */ +static int eeh_clear_pe_frozen_state(struct eeh_pe *pe) +{ + int i, rc; + + for (i = 0; i < 3; i++) { + rc = eeh_pci_enable(pe, EEH_OPT_THAW_MMIO); + if (rc) + continue; + rc = eeh_pci_enable(pe, EEH_OPT_THAW_DMA); + if (!rc) + break; + } + + /* The PE has been isolated, clear it */ + if (rc) + pr_warn("%s: Can't clear frozen PHB#%x-PE#%x (%d)\n", + __func__, pe->phb->global_number, pe->addr, rc); + else + eeh_pe_state_clear(pe, EEH_PE_ISOLATED); + + return rc; +} + /** * eeh_reset_device - Perform actual reset of a pci slot * @pe: EEH PE @@ -451,19 +504,33 @@ static int eeh_reset_device(struct eeh_pe *pe, struct pci_bus *bus) eeh_pe_dev_traverse(pe, eeh_rmv_device, &removed); } - /* Reset the pci controller. (Asserts RST#; resets config space). + /* + * Reset the pci controller. (Asserts RST#; resets config space). * Reconfigure bridges and devices. Don't try to bring the system * up if the reset failed for some reason. + * + * During the reset, it's very dangerous to have uncontrolled PCI + * config accesses. So we prefer to block them. However, controlled + * PCI config accesses initiated from EEH itself are allowed. */ + eeh_pe_state_mark(pe, EEH_PE_RESET); rc = eeh_reset_pe(pe); - if (rc) + if (rc) { + eeh_pe_state_clear(pe, EEH_PE_RESET); return rc; + } pci_lock_rescan_remove(); /* Restore PE */ eeh_ops->configure_bridge(pe); eeh_pe_restore_bars(pe); + eeh_pe_state_clear(pe, EEH_PE_RESET); + + /* Clear frozen state */ + rc = eeh_clear_pe_frozen_state(pe); + if (rc) + return rc; /* Give the system 5 seconds to finish running the user-space * hotplug shutdown scripts, e.g. ifdown for ethernet. Yes, @@ -573,7 +640,6 @@ static void eeh_handle_normal_event(struct eeh_pe *pe) result = PCI_ERS_RESULT_NEED_RESET; } else { pr_info("EEH: Notify device drivers to resume I/O\n"); - result = PCI_ERS_RESULT_NONE; eeh_pe_dev_traverse(pe, eeh_report_mmio_enabled, &result); } } @@ -585,10 +651,17 @@ static void eeh_handle_normal_event(struct eeh_pe *pe) if (rc < 0) goto hard_fail; - if (rc) + if (rc) { result = PCI_ERS_RESULT_NEED_RESET; - else + } else { + /* + * We didn't do PE reset for the case. The PE + * is still in frozen state. Clear it before + * resuming the PE. + */ + eeh_pe_state_clear(pe, EEH_PE_ISOLATED); result = PCI_ERS_RESULT_RECOVERED; + } } /* If any device has a hard failure, then shut off everything. */ @@ -650,8 +723,17 @@ perm_error: /* Notify all devices that they're about to go down. */ eeh_pe_dev_traverse(pe, eeh_report_failure, NULL); - /* Shut down the device drivers for good. */ + /* Mark the PE to be removed permanently */ + pe->freeze_count = EEH_MAX_ALLOWED_FREEZES + 1; + + /* + * Shut down the device drivers for good. We mark + * all removed devices correctly to avoid access + * the their PCI config any more. + */ if (frozen_bus) { + eeh_pe_dev_mode_mark(pe, EEH_DEV_REMOVED); + pci_lock_rescan_remove(); pcibios_remove_pci_devices(frozen_bus); pci_unlock_rescan_remove(); @@ -682,8 +764,7 @@ static void eeh_handle_special_event(void) phb_pe = eeh_phb_pe_get(hose); if (!phb_pe) continue; - eeh_pe_state_mark(phb_pe, - EEH_PE_ISOLATED | EEH_PE_PHB_DEAD); + eeh_pe_state_mark(phb_pe, EEH_PE_ISOLATED); } eeh_serialize_unlock(flags); @@ -699,8 +780,7 @@ static void eeh_handle_special_event(void) eeh_remove_event(pe); if (rc == EEH_NEXT_ERR_DEAD_PHB) - eeh_pe_state_mark(pe, - EEH_PE_ISOLATED | EEH_PE_PHB_DEAD); + eeh_pe_state_mark(pe, EEH_PE_ISOLATED); else eeh_pe_state_mark(pe, EEH_PE_ISOLATED | EEH_PE_RECOVERING); @@ -724,12 +804,14 @@ static void eeh_handle_special_event(void) if (rc == EEH_NEXT_ERR_FROZEN_PE || rc == EEH_NEXT_ERR_FENCED_PHB) { eeh_handle_normal_event(pe); + eeh_pe_state_clear(pe, EEH_PE_RECOVERING); } else { pci_lock_rescan_remove(); list_for_each_entry(hose, &hose_list, list_node) { phb_pe = eeh_phb_pe_get(hose); if (!phb_pe || - !(phb_pe->state & EEH_PE_PHB_DEAD)) + !(phb_pe->state & EEH_PE_ISOLATED) || + (phb_pe->state & EEH_PE_RECOVERING)) continue; /* Notify all devices to be down */ diff --git a/arch/powerpc/kernel/eeh_pe.c b/arch/powerpc/kernel/eeh_pe.c index f0c353fa655a..995c2a284630 100644 --- a/arch/powerpc/kernel/eeh_pe.c +++ b/arch/powerpc/kernel/eeh_pe.c @@ -503,13 +503,17 @@ static void *__eeh_pe_state_mark(void *data, void *flag) struct eeh_dev *edev, *tmp; struct pci_dev *pdev; - /* - * Mark the PE with the indicated state. Also, - * the associated PCI device will be put into - * I/O frozen state to avoid I/O accesses from - * the PCI device driver. - */ + /* Keep the state of permanently removed PE intact */ + if ((pe->freeze_count > EEH_MAX_ALLOWED_FREEZES) && + (state & (EEH_PE_ISOLATED | EEH_PE_RECOVERING))) + return NULL; + pe->state |= state; + + /* Offline PCI devices if applicable */ + if (state != EEH_PE_ISOLATED) + return NULL; + eeh_pe_for_each_dev(pe, edev, tmp) { pdev = eeh_dev_to_pci_dev(edev); if (pdev) @@ -532,6 +536,27 @@ void eeh_pe_state_mark(struct eeh_pe *pe, int state) eeh_pe_traverse(pe, __eeh_pe_state_mark, &state); } +static void *__eeh_pe_dev_mode_mark(void *data, void *flag) +{ + struct eeh_dev *edev = data; + int mode = *((int *)flag); + + edev->mode |= mode; + + return NULL; +} + +/** + * eeh_pe_dev_state_mark - Mark state for all device under the PE + * @pe: EEH PE + * + * Mark specific state for all child devices of the PE. + */ +void eeh_pe_dev_mode_mark(struct eeh_pe *pe, int mode) +{ + eeh_pe_dev_traverse(pe, __eeh_pe_dev_mode_mark, &mode); +} + /** * __eeh_pe_state_clear - Clear state for the PE * @data: EEH PE @@ -546,8 +571,16 @@ static void *__eeh_pe_state_clear(void *data, void *flag) struct eeh_pe *pe = (struct eeh_pe *)data; int state = *((int *)flag); + /* Keep the state of permanently removed PE intact */ + if ((pe->freeze_count > EEH_MAX_ALLOWED_FREEZES) && + (state & EEH_PE_ISOLATED)) + return NULL; + pe->state &= ~state; - pe->check_count = 0; + + /* Clear check count since last isolation */ + if (state & EEH_PE_ISOLATED) + pe->check_count = 0; return NULL; } diff --git a/arch/powerpc/kernel/eeh_sysfs.c b/arch/powerpc/kernel/eeh_sysfs.c index 5d753d4f2c75..e2595ba4b720 100644 --- a/arch/powerpc/kernel/eeh_sysfs.c +++ b/arch/powerpc/kernel/eeh_sysfs.c @@ -59,6 +59,9 @@ void eeh_sysfs_add_device(struct pci_dev *pdev) struct eeh_dev *edev = pci_dev_to_eeh_dev(pdev); int rc=0; + if (!eeh_enabled()) + return; + if (edev && (edev->mode & EEH_DEV_SYSFS)) return; diff --git a/arch/powerpc/kernel/entry_64.S b/arch/powerpc/kernel/entry_64.S index 662c6dd98072..911d45366f59 100644 --- a/arch/powerpc/kernel/entry_64.S +++ b/arch/powerpc/kernel/entry_64.S @@ -39,8 +39,8 @@ * System calls. */ .section ".toc","aw" -.SYS_CALL_TABLE: - .tc .sys_call_table[TC],.sys_call_table +SYS_CALL_TABLE: + .tc sys_call_table[TC],sys_call_table /* This value is used to mark exception frames on the stack. */ exception_marker: @@ -106,7 +106,7 @@ BEGIN_FW_FTR_SECTION LDX_BE r10,0,r10 /* get log write index */ cmpd cr1,r11,r10 beq+ cr1,33f - bl .accumulate_stolen_time + bl accumulate_stolen_time REST_GPR(0,r1) REST_4GPRS(3,r1) REST_2GPRS(7,r1) @@ -143,7 +143,7 @@ END_FW_FTR_SECTION_IFSET(FW_FEATURE_SPLPAR) std r10,SOFTE(r1) #ifdef SHOW_SYSCALLS - bl .do_show_syscall + bl do_show_syscall REST_GPR(0,r1) REST_4GPRS(3,r1) REST_2GPRS(7,r1) @@ -162,7 +162,7 @@ system_call: /* label this so stack traces look sane */ * Need to vector to 32 Bit or default sys_call_table here, * based on caller's run-mode / personality. */ - ld r11,.SYS_CALL_TABLE@toc(2) + ld r11,SYS_CALL_TABLE@toc(2) andi. r10,r10,_TIF_32BIT beq 15f addi r11,r11,8 /* use 32-bit syscall entries */ @@ -174,14 +174,14 @@ system_call: /* label this so stack traces look sane */ clrldi r8,r8,32 15: slwi r0,r0,4 - ldx r10,r11,r0 /* Fetch system call handler [ptr] */ - mtctr r10 + ldx r12,r11,r0 /* Fetch system call handler [ptr] */ + mtctr r12 bctrl /* Call handler */ syscall_exit: std r3,RESULT(r1) #ifdef SHOW_SYSCALLS - bl .do_show_syscall_exit + bl do_show_syscall_exit ld r3,RESULT(r1) #endif CURRENT_THREAD_INFO(r12, r1) @@ -248,9 +248,9 @@ syscall_error: /* Traced system call support */ syscall_dotrace: - bl .save_nvgprs + bl save_nvgprs addi r3,r1,STACK_FRAME_OVERHEAD - bl .do_syscall_trace_enter + bl do_syscall_trace_enter /* * Restore argument registers possibly just changed. * We use the return value of do_syscall_trace_enter @@ -308,7 +308,7 @@ syscall_exit_work: 4: /* Anything else left to do? */ SET_DEFAULT_THREAD_PPR(r3, r10) /* Set thread.ppr = 3 */ andi. r0,r9,(_TIF_SYSCALL_T_OR_A|_TIF_SINGLESTEP) - beq .ret_from_except_lite + beq ret_from_except_lite /* Re-enable interrupts */ #ifdef CONFIG_PPC_BOOK3E @@ -319,10 +319,10 @@ syscall_exit_work: mtmsrd r10,1 #endif /* CONFIG_PPC_BOOK3E */ - bl .save_nvgprs + bl save_nvgprs addi r3,r1,STACK_FRAME_OVERHEAD - bl .do_syscall_trace_leave - b .ret_from_except + bl do_syscall_trace_leave + b ret_from_except /* Save non-volatile GPRs, if not already saved. */ _GLOBAL(save_nvgprs) @@ -345,52 +345,48 @@ _GLOBAL(save_nvgprs) */ _GLOBAL(ppc_fork) - bl .save_nvgprs - bl .sys_fork + bl save_nvgprs + bl sys_fork b syscall_exit _GLOBAL(ppc_vfork) - bl .save_nvgprs - bl .sys_vfork + bl save_nvgprs + bl sys_vfork b syscall_exit _GLOBAL(ppc_clone) - bl .save_nvgprs - bl .sys_clone + bl save_nvgprs + bl sys_clone b syscall_exit _GLOBAL(ppc32_swapcontext) - bl .save_nvgprs - bl .compat_sys_swapcontext + bl save_nvgprs + bl compat_sys_swapcontext b syscall_exit _GLOBAL(ppc64_swapcontext) - bl .save_nvgprs - bl .sys_swapcontext + bl save_nvgprs + bl sys_swapcontext b syscall_exit _GLOBAL(ret_from_fork) - bl .schedule_tail + bl schedule_tail REST_NVGPRS(r1) li r3,0 b syscall_exit _GLOBAL(ret_from_kernel_thread) - bl .schedule_tail + bl schedule_tail REST_NVGPRS(r1) - ld r14, 0(r14) mtlr r14 mr r3,r15 +#if defined(_CALL_ELF) && _CALL_ELF == 2 + mr r12,r14 +#endif blrl li r3,0 b syscall_exit - .section ".toc","aw" -DSCR_DEFAULT: - .tc dscr_default[TC],dscr_default - - .section ".text" - /* * This routine switches between two different tasks. The process * state of one is saved on its kernel stack. Then the state @@ -575,11 +571,10 @@ END_FTR_SECTION_IFSET(CPU_FTR_ALTIVEC) #ifdef CONFIG_PPC64 BEGIN_FTR_SECTION lwz r6,THREAD_DSCR_INHERIT(r4) - ld r7,DSCR_DEFAULT@toc(2) ld r0,THREAD_DSCR(r4) cmpwi r6,0 bne 1f - ld r0,0(r7) + ld r0,PACA_DSCR(r13) 1: BEGIN_FTR_SECTION_NESTED(70) mfspr r8, SPRN_FSCR @@ -611,7 +606,7 @@ END_FTR_SECTION_IFSET(CPU_FTR_DSCR) _GLOBAL(ret_from_except) ld r11,_TRAP(r1) andi. r0,r11,1 - bne .ret_from_except_lite + bne ret_from_except_lite REST_NVGPRS(r1) _GLOBAL(ret_from_except_lite) @@ -661,23 +656,23 @@ _GLOBAL(ret_from_except_lite) #endif 1: andi. r0,r4,_TIF_NEED_RESCHED beq 2f - bl .restore_interrupts + bl restore_interrupts SCHEDULE_USER - b .ret_from_except_lite + b ret_from_except_lite 2: #ifdef CONFIG_PPC_TRANSACTIONAL_MEM andi. r0,r4,_TIF_USER_WORK_MASK & ~_TIF_RESTORE_TM bne 3f /* only restore TM if nothing else to do */ addi r3,r1,STACK_FRAME_OVERHEAD - bl .restore_tm_state + bl restore_tm_state b restore 3: #endif - bl .save_nvgprs - bl .restore_interrupts + bl save_nvgprs + bl restore_interrupts addi r3,r1,STACK_FRAME_OVERHEAD - bl .do_notify_resume - b .ret_from_except + bl do_notify_resume + b ret_from_except resume_kernel: /* check current_thread_info, _TIF_EMULATE_STACK_STORE */ @@ -730,7 +725,7 @@ resume_kernel: * sure we are soft-disabled first and reconcile irq state. */ RECONCILE_IRQ_STATE(r3,r4) -1: bl .preempt_schedule_irq +1: bl preempt_schedule_irq /* Re-test flags and eventually loop */ CURRENT_THREAD_INFO(r9, r1) @@ -792,7 +787,7 @@ restore_no_replay: */ do_restore: #ifdef CONFIG_PPC_BOOK3E - b .exception_return_book3e + b exception_return_book3e #else /* * Clear the reservation. If we know the CPU tracks the address of @@ -907,7 +902,7 @@ restore_check_irq_replay: * * Still, this might be useful for things like hash_page */ - bl .__check_irq_replay + bl __check_irq_replay cmpwi cr0,r3,0 beq restore_no_replay @@ -928,13 +923,13 @@ restore_check_irq_replay: cmpwi cr0,r3,0x500 bne 1f addi r3,r1,STACK_FRAME_OVERHEAD; - bl .do_IRQ - b .ret_from_except + bl do_IRQ + b ret_from_except 1: cmpwi cr0,r3,0x900 bne 1f addi r3,r1,STACK_FRAME_OVERHEAD; - bl .timer_interrupt - b .ret_from_except + bl timer_interrupt + b ret_from_except #ifdef CONFIG_PPC_DOORBELL 1: #ifdef CONFIG_PPC_BOOK3E @@ -948,14 +943,14 @@ restore_check_irq_replay: #endif /* CONFIG_PPC_BOOK3E */ bne 1f addi r3,r1,STACK_FRAME_OVERHEAD; - bl .doorbell_exception - b .ret_from_except + bl doorbell_exception + b ret_from_except #endif /* CONFIG_PPC_DOORBELL */ -1: b .ret_from_except /* What else to do here ? */ +1: b ret_from_except /* What else to do here ? */ unrecov_restore: addi r3,r1,STACK_FRAME_OVERHEAD - bl .unrecoverable_exception + bl unrecoverable_exception b unrecov_restore #ifdef CONFIG_PPC_RTAS @@ -1021,7 +1016,7 @@ _GLOBAL(enter_rtas) std r6,PACASAVEDMSR(r13) /* Setup our real return addr */ - LOAD_REG_ADDR(r4,.rtas_return_loc) + LOAD_REG_ADDR(r4,rtas_return_loc) clrldi r4,r4,2 /* convert to realmode address */ mtlr r4 @@ -1045,7 +1040,7 @@ _GLOBAL(enter_rtas) rfid b . /* prevent speculative execution */ -_STATIC(rtas_return_loc) +rtas_return_loc: FIXUP_ENDIAN /* relocation is off at this point */ @@ -1054,7 +1049,7 @@ _STATIC(rtas_return_loc) bcl 20,31,$+4 0: mflr r3 - ld r3,(1f-0b)(r3) /* get &.rtas_restore_regs */ + ld r3,(1f-0b)(r3) /* get &rtas_restore_regs */ mfmsr r6 li r0,MSR_RI @@ -1071,9 +1066,9 @@ _STATIC(rtas_return_loc) b . /* prevent speculative execution */ .align 3 -1: .llong .rtas_restore_regs +1: .llong rtas_restore_regs -_STATIC(rtas_restore_regs) +rtas_restore_regs: /* relocation is on at this point */ REST_GPR(2, r1) /* Restore the TOC */ REST_GPR(13, r1) /* Restore paca */ @@ -1173,7 +1168,7 @@ _GLOBAL(mcount) _GLOBAL(_mcount) blr -_GLOBAL(ftrace_caller) +_GLOBAL_TOC(ftrace_caller) /* Taken from output of objdump from lib64/glibc */ mflr r3 ld r11, 0(r1) @@ -1197,10 +1192,7 @@ _GLOBAL(ftrace_graph_stub) _GLOBAL(ftrace_stub) blr #else -_GLOBAL(mcount) - blr - -_GLOBAL(_mcount) +_GLOBAL_TOC(_mcount) /* Taken from output of objdump from lib64/glibc */ mflr r3 ld r11, 0(r1) @@ -1238,7 +1230,7 @@ _GLOBAL(ftrace_graph_caller) ld r11, 112(r1) addi r3, r11, 16 - bl .prepare_ftrace_return + bl prepare_ftrace_return nop ld r0, 128(r1) @@ -1254,7 +1246,7 @@ _GLOBAL(return_to_handler) mr r31, r1 stdu r1, -112(r1) - bl .ftrace_return_to_handler + bl ftrace_return_to_handler nop /* return value has real return address */ @@ -1284,7 +1276,7 @@ _GLOBAL(mod_return_to_handler) */ ld r2, PACATOC(r13) - bl .ftrace_return_to_handler + bl ftrace_return_to_handler nop /* return value has real return address */ diff --git a/arch/powerpc/kernel/epapr_paravirt.c b/arch/powerpc/kernel/epapr_paravirt.c index 60d1a2259dbe..59e4ba74975d 100644 --- a/arch/powerpc/kernel/epapr_paravirt.c +++ b/arch/powerpc/kernel/epapr_paravirt.c @@ -30,6 +30,7 @@ extern u32 epapr_ev_idle_start[]; #endif bool epapr_paravirt_enabled; +static bool __maybe_unused epapr_has_idle; static int __init early_init_dt_scan_epapr(unsigned long node, const char *uname, @@ -56,7 +57,7 @@ static int __init early_init_dt_scan_epapr(unsigned long node, #if !defined(CONFIG_64BIT) || defined(CONFIG_PPC_BOOK3E_64) if (of_get_flat_dt_prop(node, "has-idle", NULL)) - ppc_md.power_save = epapr_ev_idle; + epapr_has_idle = true; #endif epapr_paravirt_enabled = true; @@ -71,3 +72,14 @@ int __init epapr_paravirt_early_init(void) return 0; } +static int __init epapr_idle_init(void) +{ +#if !defined(CONFIG_64BIT) || defined(CONFIG_PPC_BOOK3E_64) + if (epapr_has_idle) + ppc_md.power_save = epapr_ev_idle; +#endif + + return 0; +} + +postcore_initcall(epapr_idle_init); diff --git a/arch/powerpc/kernel/exceptions-64e.S b/arch/powerpc/kernel/exceptions-64e.S index c1bee3ce9d1f..771b4e92e5d9 100644 --- a/arch/powerpc/kernel/exceptions-64e.S +++ b/arch/powerpc/kernel/exceptions-64e.S @@ -499,7 +499,7 @@ exc_##n##_bad_stack: \ CHECK_NAPPING(); \ addi r3,r1,STACK_FRAME_OVERHEAD; \ bl hdlr; \ - b .ret_from_except_lite; + b ret_from_except_lite; /* This value is used to mark exception frames on the stack. */ .section ".toc","aw" @@ -550,11 +550,11 @@ interrupt_end_book3e: CRIT_EXCEPTION_PROLOG(0x100, BOOKE_INTERRUPT_CRITICAL, PROLOG_ADDITION_NONE) EXCEPTION_COMMON_CRIT(0x100) - bl .save_nvgprs + bl save_nvgprs bl special_reg_save CHECK_NAPPING(); addi r3,r1,STACK_FRAME_OVERHEAD - bl .unknown_exception + bl unknown_exception b ret_from_crit_except /* Machine Check Interrupt */ @@ -562,11 +562,11 @@ interrupt_end_book3e: MC_EXCEPTION_PROLOG(0x000, BOOKE_INTERRUPT_MACHINE_CHECK, PROLOG_ADDITION_NONE) EXCEPTION_COMMON_MC(0x000) - bl .save_nvgprs + bl save_nvgprs bl special_reg_save CHECK_NAPPING(); addi r3,r1,STACK_FRAME_OVERHEAD - bl .machine_check_exception + bl machine_check_exception b ret_from_mc_except /* Data Storage Interrupt */ @@ -591,7 +591,7 @@ interrupt_end_book3e: /* External Input Interrupt */ MASKABLE_EXCEPTION(0x500, BOOKE_INTERRUPT_EXTERNAL, - external_input, .do_IRQ, ACK_NONE) + external_input, do_IRQ, ACK_NONE) /* Alignment */ START_EXCEPTION(alignment); @@ -612,9 +612,9 @@ interrupt_end_book3e: std r14,_DSISR(r1) addi r3,r1,STACK_FRAME_OVERHEAD ld r14,PACA_EXGEN+EX_R14(r13) - bl .save_nvgprs - bl .program_check_exception - b .ret_from_except + bl save_nvgprs + bl program_check_exception + b ret_from_except /* Floating Point Unavailable Interrupt */ START_EXCEPTION(fp_unavailable); @@ -625,13 +625,13 @@ interrupt_end_book3e: ld r12,_MSR(r1) andi. r0,r12,MSR_PR; beq- 1f - bl .load_up_fpu + bl load_up_fpu b fast_exception_return 1: INTS_DISABLE - bl .save_nvgprs + bl save_nvgprs addi r3,r1,STACK_FRAME_OVERHEAD - bl .kernel_fp_unavailable_exception - b .ret_from_except + bl kernel_fp_unavailable_exception + b ret_from_except /* Altivec Unavailable Interrupt */ START_EXCEPTION(altivec_unavailable); @@ -644,16 +644,16 @@ BEGIN_FTR_SECTION ld r12,_MSR(r1) andi. r0,r12,MSR_PR; beq- 1f - bl .load_up_altivec + bl load_up_altivec b fast_exception_return 1: END_FTR_SECTION_IFSET(CPU_FTR_ALTIVEC) #endif INTS_DISABLE - bl .save_nvgprs + bl save_nvgprs addi r3,r1,STACK_FRAME_OVERHEAD - bl .altivec_unavailable_exception - b .ret_from_except + bl altivec_unavailable_exception + b ret_from_except /* AltiVec Assist */ START_EXCEPTION(altivec_assist); @@ -662,39 +662,39 @@ END_FTR_SECTION_IFSET(CPU_FTR_ALTIVEC) PROLOG_ADDITION_NONE) EXCEPTION_COMMON(0x220) INTS_DISABLE - bl .save_nvgprs + bl save_nvgprs addi r3,r1,STACK_FRAME_OVERHEAD #ifdef CONFIG_ALTIVEC BEGIN_FTR_SECTION - bl .altivec_assist_exception + bl altivec_assist_exception END_FTR_SECTION_IFSET(CPU_FTR_ALTIVEC) #else - bl .unknown_exception + bl unknown_exception #endif - b .ret_from_except + b ret_from_except /* Decrementer Interrupt */ MASKABLE_EXCEPTION(0x900, BOOKE_INTERRUPT_DECREMENTER, - decrementer, .timer_interrupt, ACK_DEC) + decrementer, timer_interrupt, ACK_DEC) /* Fixed Interval Timer Interrupt */ MASKABLE_EXCEPTION(0x980, BOOKE_INTERRUPT_FIT, - fixed_interval, .unknown_exception, ACK_FIT) + fixed_interval, unknown_exception, ACK_FIT) /* Watchdog Timer Interrupt */ START_EXCEPTION(watchdog); CRIT_EXCEPTION_PROLOG(0x9f0, BOOKE_INTERRUPT_WATCHDOG, PROLOG_ADDITION_NONE) EXCEPTION_COMMON_CRIT(0x9f0) - bl .save_nvgprs + bl save_nvgprs bl special_reg_save CHECK_NAPPING(); addi r3,r1,STACK_FRAME_OVERHEAD #ifdef CONFIG_BOOKE_WDT - bl .WatchdogException + bl WatchdogException #else - bl .unknown_exception + bl unknown_exception #endif b ret_from_crit_except @@ -712,10 +712,10 @@ END_FTR_SECTION_IFSET(CPU_FTR_ALTIVEC) PROLOG_ADDITION_NONE) EXCEPTION_COMMON(0xf20) INTS_DISABLE - bl .save_nvgprs + bl save_nvgprs addi r3,r1,STACK_FRAME_OVERHEAD - bl .unknown_exception - b .ret_from_except + bl unknown_exception + b ret_from_except /* Debug exception as a critical interrupt*/ START_EXCEPTION(debug_crit); @@ -774,9 +774,9 @@ END_FTR_SECTION_IFSET(CPU_FTR_ALTIVEC) mr r4,r14 ld r14,PACA_EXCRIT+EX_R14(r13) ld r15,PACA_EXCRIT+EX_R15(r13) - bl .save_nvgprs - bl .DebugException - b .ret_from_except + bl save_nvgprs + bl DebugException + b ret_from_except kernel_dbg_exc: b . /* NYI */ @@ -839,9 +839,9 @@ kernel_dbg_exc: mr r4,r14 ld r14,PACA_EXDBG+EX_R14(r13) ld r15,PACA_EXDBG+EX_R15(r13) - bl .save_nvgprs - bl .DebugException - b .ret_from_except + bl save_nvgprs + bl DebugException + b ret_from_except START_EXCEPTION(perfmon); NORMAL_EXCEPTION_PROLOG(0x260, BOOKE_INTERRUPT_PERFORMANCE_MONITOR, @@ -850,23 +850,23 @@ kernel_dbg_exc: INTS_DISABLE CHECK_NAPPING() addi r3,r1,STACK_FRAME_OVERHEAD - bl .performance_monitor_exception - b .ret_from_except_lite + bl performance_monitor_exception + b ret_from_except_lite /* Doorbell interrupt */ MASKABLE_EXCEPTION(0x280, BOOKE_INTERRUPT_DOORBELL, - doorbell, .doorbell_exception, ACK_NONE) + doorbell, doorbell_exception, ACK_NONE) /* Doorbell critical Interrupt */ START_EXCEPTION(doorbell_crit); CRIT_EXCEPTION_PROLOG(0x2a0, BOOKE_INTERRUPT_DOORBELL_CRITICAL, PROLOG_ADDITION_NONE) EXCEPTION_COMMON_CRIT(0x2a0) - bl .save_nvgprs + bl save_nvgprs bl special_reg_save CHECK_NAPPING(); addi r3,r1,STACK_FRAME_OVERHEAD - bl .unknown_exception + bl unknown_exception b ret_from_crit_except /* @@ -878,21 +878,21 @@ kernel_dbg_exc: PROLOG_ADDITION_NONE) EXCEPTION_COMMON(0x2c0) addi r3,r1,STACK_FRAME_OVERHEAD - bl .save_nvgprs + bl save_nvgprs INTS_RESTORE_HARD - bl .unknown_exception - b .ret_from_except + bl unknown_exception + b ret_from_except /* Guest Doorbell critical Interrupt */ START_EXCEPTION(guest_doorbell_crit); CRIT_EXCEPTION_PROLOG(0x2e0, BOOKE_INTERRUPT_GUEST_DBELL_CRIT, PROLOG_ADDITION_NONE) EXCEPTION_COMMON_CRIT(0x2e0) - bl .save_nvgprs + bl save_nvgprs bl special_reg_save CHECK_NAPPING(); addi r3,r1,STACK_FRAME_OVERHEAD - bl .unknown_exception + bl unknown_exception b ret_from_crit_except /* Hypervisor call */ @@ -901,10 +901,10 @@ kernel_dbg_exc: PROLOG_ADDITION_NONE) EXCEPTION_COMMON(0x310) addi r3,r1,STACK_FRAME_OVERHEAD - bl .save_nvgprs + bl save_nvgprs INTS_RESTORE_HARD - bl .unknown_exception - b .ret_from_except + bl unknown_exception + b ret_from_except /* Embedded Hypervisor priviledged */ START_EXCEPTION(ehpriv); @@ -912,10 +912,10 @@ kernel_dbg_exc: PROLOG_ADDITION_NONE) EXCEPTION_COMMON(0x320) addi r3,r1,STACK_FRAME_OVERHEAD - bl .save_nvgprs + bl save_nvgprs INTS_RESTORE_HARD - bl .unknown_exception - b .ret_from_except + bl unknown_exception + b ret_from_except /* LRAT Error interrupt */ START_EXCEPTION(lrat_error); @@ -1014,16 +1014,16 @@ storage_fault_common: mr r5,r15 ld r14,PACA_EXGEN+EX_R14(r13) ld r15,PACA_EXGEN+EX_R15(r13) - bl .do_page_fault + bl do_page_fault cmpdi r3,0 bne- 1f - b .ret_from_except_lite -1: bl .save_nvgprs + b ret_from_except_lite +1: bl save_nvgprs mr r5,r3 addi r3,r1,STACK_FRAME_OVERHEAD ld r4,_DAR(r1) - bl .bad_page_fault - b .ret_from_except + bl bad_page_fault + b ret_from_except /* * Alignment exception doesn't fit entirely in the 0x100 bytes so it @@ -1035,10 +1035,10 @@ alignment_more: addi r3,r1,STACK_FRAME_OVERHEAD ld r14,PACA_EXGEN+EX_R14(r13) ld r15,PACA_EXGEN+EX_R15(r13) - bl .save_nvgprs + bl save_nvgprs INTS_RESTORE_HARD - bl .alignment_exception - b .ret_from_except + bl alignment_exception + b ret_from_except /* * We branch here from entry_64.S for the last stage of the exception @@ -1172,7 +1172,7 @@ bad_stack_book3e: std r12,0(r11) ld r2,PACATOC(r13) 1: addi r3,r1,STACK_FRAME_OVERHEAD - bl .kernel_bad_stack + bl kernel_bad_stack b 1b /* @@ -1521,13 +1521,13 @@ _GLOBAL(start_initialization_book3e) * and always use AS 0, so we just set it up to match our link * address and never use 0 based addresses. */ - bl .initial_tlb_book3e + bl initial_tlb_book3e /* Init global core bits */ - bl .init_core_book3e + bl init_core_book3e /* Init per-thread bits */ - bl .init_thread_book3e + bl init_thread_book3e /* Return to common init code */ tovirt(r28,r28) @@ -1548,7 +1548,7 @@ _GLOBAL(start_initialization_book3e) */ _GLOBAL(book3e_secondary_core_init_tlb_set) li r4,1 - b .generic_secondary_smp_init + b generic_secondary_smp_init _GLOBAL(book3e_secondary_core_init) mflr r28 @@ -1558,18 +1558,18 @@ _GLOBAL(book3e_secondary_core_init) bne 2f /* Setup TLB for this core */ - bl .initial_tlb_book3e + bl initial_tlb_book3e /* We can return from the above running at a different * address, so recalculate r2 (TOC) */ - bl .relative_toc + bl relative_toc /* Init global core bits */ -2: bl .init_core_book3e +2: bl init_core_book3e /* Init per-thread bits */ -3: bl .init_thread_book3e +3: bl init_thread_book3e /* Return to common init code at proper virtual address. * @@ -1596,14 +1596,14 @@ _GLOBAL(book3e_secondary_thread_init) mflr r28 b 3b -_STATIC(init_core_book3e) +init_core_book3e: /* Establish the interrupt vector base */ LOAD_REG_IMMEDIATE(r3, interrupt_base_book3e) mtspr SPRN_IVPR,r3 sync blr -_STATIC(init_thread_book3e) +init_thread_book3e: lis r3,(SPRN_EPCR_ICM | SPRN_EPCR_GICM)@h mtspr SPRN_EPCR,r3 diff --git a/arch/powerpc/kernel/exceptions-64s.S b/arch/powerpc/kernel/exceptions-64s.S index 3afd3915921a..20f11eb4dff7 100644 --- a/arch/powerpc/kernel/exceptions-64s.S +++ b/arch/powerpc/kernel/exceptions-64s.S @@ -132,12 +132,12 @@ BEGIN_FTR_SECTION #endif beq cr1,2f - b .power7_wakeup_noloss -2: b .power7_wakeup_loss + b power7_wakeup_noloss +2: b power7_wakeup_loss /* Fast Sleep wakeup on PowerNV */ 8: GET_PACA(r13) - b .power7_wakeup_tb_loss + b power7_wakeup_tb_loss 9: END_FTR_SECTION_IFSET(CPU_FTR_HVMODE | CPU_FTR_ARCH_206) @@ -211,16 +211,16 @@ data_access_slb_pSeries: #endif /* __DISABLED__ */ mfspr r12,SPRN_SRR1 #ifndef CONFIG_RELOCATABLE - b .slb_miss_realmode + b slb_miss_realmode #else /* - * We can't just use a direct branch to .slb_miss_realmode + * We can't just use a direct branch to slb_miss_realmode * because the distance from here to there depends on where * the kernel ends up being put. */ mfctr r11 ld r10,PACAKBASE(r13) - LOAD_HANDLER(r10, .slb_miss_realmode) + LOAD_HANDLER(r10, slb_miss_realmode) mtctr r10 bctr #endif @@ -243,11 +243,11 @@ instruction_access_slb_pSeries: #endif /* __DISABLED__ */ mfspr r12,SPRN_SRR1 #ifndef CONFIG_RELOCATABLE - b .slb_miss_realmode + b slb_miss_realmode #else mfctr r11 ld r10,PACAKBASE(r13) - LOAD_HANDLER(r10, .slb_miss_realmode) + LOAD_HANDLER(r10, slb_miss_realmode) mtctr r10 bctr #endif @@ -524,7 +524,7 @@ do_stab_bolted_pSeries: std r12,PACA_EXSLB+EX_R12(r13) GET_SCRATCH0(r10) std r10,PACA_EXSLB+EX_R13(r13) - EXCEPTION_PROLOG_PSERIES_1(.do_stab_bolted, EXC_STD) + EXCEPTION_PROLOG_PSERIES_1(do_stab_bolted, EXC_STD) KVM_HANDLER_SKIP(PACA_EXGEN, EXC_STD, 0x300) KVM_HANDLER_SKIP(PACA_EXSLB, EXC_STD, 0x380) @@ -769,38 +769,38 @@ kvmppc_skip_Hinterrupt: /*** Common interrupt handlers ***/ - STD_EXCEPTION_COMMON(0x100, system_reset, .system_reset_exception) + STD_EXCEPTION_COMMON(0x100, system_reset, system_reset_exception) STD_EXCEPTION_COMMON_ASYNC(0x500, hardware_interrupt, do_IRQ) - STD_EXCEPTION_COMMON_ASYNC(0x900, decrementer, .timer_interrupt) - STD_EXCEPTION_COMMON(0x980, hdecrementer, .hdec_interrupt) + STD_EXCEPTION_COMMON_ASYNC(0x900, decrementer, timer_interrupt) + STD_EXCEPTION_COMMON(0x980, hdecrementer, hdec_interrupt) #ifdef CONFIG_PPC_DOORBELL - STD_EXCEPTION_COMMON_ASYNC(0xa00, doorbell_super, .doorbell_exception) + STD_EXCEPTION_COMMON_ASYNC(0xa00, doorbell_super, doorbell_exception) #else - STD_EXCEPTION_COMMON_ASYNC(0xa00, doorbell_super, .unknown_exception) + STD_EXCEPTION_COMMON_ASYNC(0xa00, doorbell_super, unknown_exception) #endif - STD_EXCEPTION_COMMON(0xb00, trap_0b, .unknown_exception) - STD_EXCEPTION_COMMON(0xd00, single_step, .single_step_exception) - STD_EXCEPTION_COMMON(0xe00, trap_0e, .unknown_exception) - STD_EXCEPTION_COMMON(0xe40, emulation_assist, .emulation_assist_interrupt) - STD_EXCEPTION_COMMON(0xe60, hmi_exception, .unknown_exception) + STD_EXCEPTION_COMMON(0xb00, trap_0b, unknown_exception) + STD_EXCEPTION_COMMON(0xd00, single_step, single_step_exception) + STD_EXCEPTION_COMMON(0xe00, trap_0e, unknown_exception) + STD_EXCEPTION_COMMON(0xe40, emulation_assist, emulation_assist_interrupt) + STD_EXCEPTION_COMMON(0xe60, hmi_exception, unknown_exception) #ifdef CONFIG_PPC_DOORBELL - STD_EXCEPTION_COMMON_ASYNC(0xe80, h_doorbell, .doorbell_exception) + STD_EXCEPTION_COMMON_ASYNC(0xe80, h_doorbell, doorbell_exception) #else - STD_EXCEPTION_COMMON_ASYNC(0xe80, h_doorbell, .unknown_exception) + STD_EXCEPTION_COMMON_ASYNC(0xe80, h_doorbell, unknown_exception) #endif - STD_EXCEPTION_COMMON_ASYNC(0xf00, performance_monitor, .performance_monitor_exception) - STD_EXCEPTION_COMMON(0x1300, instruction_breakpoint, .instruction_breakpoint_exception) - STD_EXCEPTION_COMMON(0x1502, denorm, .unknown_exception) + STD_EXCEPTION_COMMON_ASYNC(0xf00, performance_monitor, performance_monitor_exception) + STD_EXCEPTION_COMMON(0x1300, instruction_breakpoint, instruction_breakpoint_exception) + STD_EXCEPTION_COMMON(0x1502, denorm, unknown_exception) #ifdef CONFIG_ALTIVEC - STD_EXCEPTION_COMMON(0x1700, altivec_assist, .altivec_assist_exception) + STD_EXCEPTION_COMMON(0x1700, altivec_assist, altivec_assist_exception) #else - STD_EXCEPTION_COMMON(0x1700, altivec_assist, .unknown_exception) + STD_EXCEPTION_COMMON(0x1700, altivec_assist, unknown_exception) #endif #ifdef CONFIG_CBE_RAS - STD_EXCEPTION_COMMON(0x1200, cbe_system_error, .cbe_system_error_exception) - STD_EXCEPTION_COMMON(0x1600, cbe_maintenance, .cbe_maintenance_exception) - STD_EXCEPTION_COMMON(0x1800, cbe_thermal, .cbe_thermal_exception) + STD_EXCEPTION_COMMON(0x1200, cbe_system_error, cbe_system_error_exception) + STD_EXCEPTION_COMMON(0x1600, cbe_maintenance, cbe_maintenance_exception) + STD_EXCEPTION_COMMON(0x1800, cbe_thermal, cbe_thermal_exception) #endif /* CONFIG_CBE_RAS */ /* @@ -829,16 +829,16 @@ data_access_slb_relon_pSeries: mfspr r3,SPRN_DAR mfspr r12,SPRN_SRR1 #ifndef CONFIG_RELOCATABLE - b .slb_miss_realmode + b slb_miss_realmode #else /* - * We can't just use a direct branch to .slb_miss_realmode + * We can't just use a direct branch to slb_miss_realmode * because the distance from here to there depends on where * the kernel ends up being put. */ mfctr r11 ld r10,PACAKBASE(r13) - LOAD_HANDLER(r10, .slb_miss_realmode) + LOAD_HANDLER(r10, slb_miss_realmode) mtctr r10 bctr #endif @@ -854,11 +854,11 @@ instruction_access_slb_relon_pSeries: mfspr r3,SPRN_SRR0 /* SRR0 is faulting address */ mfspr r12,SPRN_SRR1 #ifndef CONFIG_RELOCATABLE - b .slb_miss_realmode + b slb_miss_realmode #else mfctr r11 ld r10,PACAKBASE(r13) - LOAD_HANDLER(r10, .slb_miss_realmode) + LOAD_HANDLER(r10, slb_miss_realmode) mtctr r10 bctr #endif @@ -966,7 +966,7 @@ system_call_entry: b system_call_common ppc64_runlatch_on_trampoline: - b .__ppc64_runlatch_on + b __ppc64_runlatch_on /* * Here we have detected that the kernel stack pointer is bad. @@ -1025,7 +1025,7 @@ END_FTR_SECTION_IFSET(CPU_FTR_CFAR) std r12,RESULT(r1) std r11,STACK_FRAME_OVERHEAD-16(r1) 1: addi r3,r1,STACK_FRAME_OVERHEAD - bl .kernel_bad_stack + bl kernel_bad_stack b 1b /* @@ -1046,7 +1046,7 @@ data_access_common: ld r3,PACA_EXGEN+EX_DAR(r13) lwz r4,PACA_EXGEN+EX_DSISR(r13) li r5,0x300 - b .do_hash_page /* Try to handle as hpte fault */ + b do_hash_page /* Try to handle as hpte fault */ .align 7 .globl h_data_storage_common @@ -1056,11 +1056,11 @@ h_data_storage_common: mfspr r10,SPRN_HDSISR stw r10,PACA_EXGEN+EX_DSISR(r13) EXCEPTION_PROLOG_COMMON(0xe00, PACA_EXGEN) - bl .save_nvgprs + bl save_nvgprs DISABLE_INTS addi r3,r1,STACK_FRAME_OVERHEAD - bl .unknown_exception - b .ret_from_except + bl unknown_exception + b ret_from_except .align 7 .globl instruction_access_common @@ -1071,9 +1071,9 @@ instruction_access_common: ld r3,_NIP(r1) andis. r4,r12,0x5820 li r5,0x400 - b .do_hash_page /* Try to handle as hpte fault */ + b do_hash_page /* Try to handle as hpte fault */ - STD_EXCEPTION_COMMON(0xe20, h_instr_storage, .unknown_exception) + STD_EXCEPTION_COMMON(0xe20, h_instr_storage, unknown_exception) /* * Here is the common SLB miss user that is used when going to virtual @@ -1088,7 +1088,7 @@ slb_miss_user_common: stw r9,PACA_EXGEN+EX_CCR(r13) std r10,PACA_EXGEN+EX_LR(r13) std r11,PACA_EXGEN+EX_SRR0(r13) - bl .slb_allocate_user + bl slb_allocate_user ld r10,PACA_EXGEN+EX_LR(r13) ld r3,PACA_EXGEN+EX_R3(r13) @@ -1131,9 +1131,9 @@ slb_miss_fault: unrecov_user_slb: EXCEPTION_PROLOG_COMMON(0x4200, PACA_EXGEN) DISABLE_INTS - bl .save_nvgprs + bl save_nvgprs 1: addi r3,r1,STACK_FRAME_OVERHEAD - bl .unrecoverable_exception + bl unrecoverable_exception b 1b #endif /* __DISABLED__ */ @@ -1158,10 +1158,10 @@ machine_check_common: lwz r4,PACA_EXGEN+EX_DSISR(r13) std r3,_DAR(r1) std r4,_DSISR(r1) - bl .save_nvgprs + bl save_nvgprs addi r3,r1,STACK_FRAME_OVERHEAD - bl .machine_check_exception - b .ret_from_except + bl machine_check_exception + b ret_from_except .align 7 .globl alignment_common @@ -1175,31 +1175,31 @@ alignment_common: lwz r4,PACA_EXGEN+EX_DSISR(r13) std r3,_DAR(r1) std r4,_DSISR(r1) - bl .save_nvgprs + bl save_nvgprs DISABLE_INTS addi r3,r1,STACK_FRAME_OVERHEAD - bl .alignment_exception - b .ret_from_except + bl alignment_exception + b ret_from_except .align 7 .globl program_check_common program_check_common: EXCEPTION_PROLOG_COMMON(0x700, PACA_EXGEN) - bl .save_nvgprs + bl save_nvgprs DISABLE_INTS addi r3,r1,STACK_FRAME_OVERHEAD - bl .program_check_exception - b .ret_from_except + bl program_check_exception + b ret_from_except .align 7 .globl fp_unavailable_common fp_unavailable_common: EXCEPTION_PROLOG_COMMON(0x800, PACA_EXGEN) bne 1f /* if from user, just load it up */ - bl .save_nvgprs + bl save_nvgprs DISABLE_INTS addi r3,r1,STACK_FRAME_OVERHEAD - bl .kernel_fp_unavailable_exception + bl kernel_fp_unavailable_exception BUG_OPCODE 1: #ifdef CONFIG_PPC_TRANSACTIONAL_MEM @@ -1211,15 +1211,15 @@ BEGIN_FTR_SECTION bne- 2f END_FTR_SECTION_IFSET(CPU_FTR_TM) #endif - bl .load_up_fpu + bl load_up_fpu b fast_exception_return #ifdef CONFIG_PPC_TRANSACTIONAL_MEM 2: /* User process was in a transaction */ - bl .save_nvgprs + bl save_nvgprs DISABLE_INTS addi r3,r1,STACK_FRAME_OVERHEAD - bl .fp_unavailable_tm - b .ret_from_except + bl fp_unavailable_tm + b ret_from_except #endif .align 7 .globl altivec_unavailable_common @@ -1237,24 +1237,24 @@ BEGIN_FTR_SECTION bne- 2f END_FTR_SECTION_NESTED(CPU_FTR_TM, CPU_FTR_TM, 69) #endif - bl .load_up_altivec + bl load_up_altivec b fast_exception_return #ifdef CONFIG_PPC_TRANSACTIONAL_MEM 2: /* User process was in a transaction */ - bl .save_nvgprs + bl save_nvgprs DISABLE_INTS addi r3,r1,STACK_FRAME_OVERHEAD - bl .altivec_unavailable_tm - b .ret_from_except + bl altivec_unavailable_tm + b ret_from_except #endif 1: END_FTR_SECTION_IFSET(CPU_FTR_ALTIVEC) #endif - bl .save_nvgprs + bl save_nvgprs DISABLE_INTS addi r3,r1,STACK_FRAME_OVERHEAD - bl .altivec_unavailable_exception - b .ret_from_except + bl altivec_unavailable_exception + b ret_from_except .align 7 .globl vsx_unavailable_common @@ -1272,26 +1272,26 @@ BEGIN_FTR_SECTION bne- 2f END_FTR_SECTION_NESTED(CPU_FTR_TM, CPU_FTR_TM, 69) #endif - b .load_up_vsx + b load_up_vsx #ifdef CONFIG_PPC_TRANSACTIONAL_MEM 2: /* User process was in a transaction */ - bl .save_nvgprs + bl save_nvgprs DISABLE_INTS addi r3,r1,STACK_FRAME_OVERHEAD - bl .vsx_unavailable_tm - b .ret_from_except + bl vsx_unavailable_tm + b ret_from_except #endif 1: END_FTR_SECTION_IFSET(CPU_FTR_VSX) #endif - bl .save_nvgprs + bl save_nvgprs DISABLE_INTS addi r3,r1,STACK_FRAME_OVERHEAD - bl .vsx_unavailable_exception - b .ret_from_except + bl vsx_unavailable_exception + b ret_from_except - STD_EXCEPTION_COMMON(0xf60, facility_unavailable, .facility_unavailable_exception) - STD_EXCEPTION_COMMON(0xf80, hv_facility_unavailable, .facility_unavailable_exception) + STD_EXCEPTION_COMMON(0xf60, facility_unavailable, facility_unavailable_exception) + STD_EXCEPTION_COMMON(0xf80, hv_facility_unavailable, facility_unavailable_exception) .align 7 .globl __end_handlers @@ -1386,9 +1386,9 @@ _GLOBAL(opal_mc_secondary_handler) machine_check_handle_early: std r0,GPR0(r1) /* Save r0 */ EXCEPTION_PROLOG_COMMON_3(0x200) - bl .save_nvgprs + bl save_nvgprs addi r3,r1,STACK_FRAME_OVERHEAD - bl .machine_check_early + bl machine_check_early ld r12,_MSR(r1) #ifdef CONFIG_PPC_P7_NAP /* @@ -1408,11 +1408,11 @@ machine_check_handle_early: /* Supervisor state loss */ li r0,1 stb r0,PACA_NAPSTATELOST(r13) -3: bl .machine_check_queue_event +3: bl machine_check_queue_event MACHINE_CHECK_HANDLER_WINDUP GET_PACA(r13) ld r1,PACAR1(r13) - b .power7_enter_nap_mode + b power7_enter_nap_mode 4: #endif /* @@ -1444,7 +1444,7 @@ machine_check_handle_early: andi. r11,r12,MSR_RI bne 2f 1: addi r3,r1,STACK_FRAME_OVERHEAD - bl .unrecoverable_exception + bl unrecoverable_exception b 1b 2: /* @@ -1452,7 +1452,7 @@ machine_check_handle_early: * Queue up the MCE event so that we can log it later, while * returning from kernel or opal call. */ - bl .machine_check_queue_event + bl machine_check_queue_event MACHINE_CHECK_HANDLER_WINDUP rfid 9: @@ -1468,7 +1468,7 @@ machine_check_handle_early: * r3 is saved in paca->slb_r3 * We assume we aren't going to take any exceptions during this procedure. */ -_GLOBAL(slb_miss_realmode) +slb_miss_realmode: mflr r10 #ifdef CONFIG_RELOCATABLE mtctr r11 @@ -1477,7 +1477,7 @@ _GLOBAL(slb_miss_realmode) stw r9,PACA_EXSLB+EX_CCR(r13) /* save CR in exc. frame */ std r10,PACA_EXSLB+EX_LR(r13) /* save LR */ - bl .slb_allocate_realmode + bl slb_allocate_realmode /* All done -- return from exception. */ @@ -1517,9 +1517,9 @@ _GLOBAL(slb_miss_realmode) unrecov_slb: EXCEPTION_PROLOG_COMMON(0x4100, PACA_EXSLB) DISABLE_INTS - bl .save_nvgprs + bl save_nvgprs 1: addi r3,r1,STACK_FRAME_OVERHEAD - bl .unrecoverable_exception + bl unrecoverable_exception b 1b @@ -1536,7 +1536,7 @@ power4_fixup_nap: * Hash table stuff */ .align 7 -_STATIC(do_hash_page) +do_hash_page: std r3,_DAR(r1) std r4,_DSISR(r1) @@ -1573,7 +1573,7 @@ END_MMU_FTR_SECTION_IFCLR(MMU_FTR_SLB) * * at return r3 = 0 for success, 1 for page fault, negative for error */ - bl .hash_page /* build HPTE if possible */ + bl hash_page /* build HPTE if possible */ cmpdi r3,0 /* see if hash_page succeeded */ /* Success */ @@ -1587,35 +1587,35 @@ handle_page_fault: 11: ld r4,_DAR(r1) ld r5,_DSISR(r1) addi r3,r1,STACK_FRAME_OVERHEAD - bl .do_page_fault + bl do_page_fault cmpdi r3,0 beq+ 12f - bl .save_nvgprs + bl save_nvgprs mr r5,r3 addi r3,r1,STACK_FRAME_OVERHEAD lwz r4,_DAR(r1) - bl .bad_page_fault - b .ret_from_except + bl bad_page_fault + b ret_from_except /* We have a data breakpoint exception - handle it */ handle_dabr_fault: - bl .save_nvgprs + bl save_nvgprs ld r4,_DAR(r1) ld r5,_DSISR(r1) addi r3,r1,STACK_FRAME_OVERHEAD - bl .do_break -12: b .ret_from_except_lite + bl do_break +12: b ret_from_except_lite /* We have a page fault that hash_page could handle but HV refused * the PTE insertion */ -13: bl .save_nvgprs +13: bl save_nvgprs mr r5,r3 addi r3,r1,STACK_FRAME_OVERHEAD ld r4,_DAR(r1) - bl .low_hash_fault - b .ret_from_except + bl low_hash_fault + b ret_from_except /* * We come here as a result of a DSI at a point where we don't want @@ -1624,16 +1624,16 @@ handle_dabr_fault: * were soft-disabled. We want to invoke the exception handler for * the access, or panic if there isn't a handler. */ -77: bl .save_nvgprs +77: bl save_nvgprs mr r4,r3 addi r3,r1,STACK_FRAME_OVERHEAD li r5,SIGSEGV - bl .bad_page_fault - b .ret_from_except + bl bad_page_fault + b ret_from_except /* here we have a segment miss */ do_ste_alloc: - bl .ste_allocate /* try to insert stab entry */ + bl ste_allocate /* try to insert stab entry */ cmpdi r3,0 bne- handle_page_fault b fast_exception_return @@ -1646,7 +1646,7 @@ do_ste_alloc: * We assume (DAR >> 60) == 0xc. */ .align 7 -_GLOBAL(do_stab_bolted) +do_stab_bolted: stw r9,PACA_EXSLB+EX_CCR(r13) /* save CR in exc. frame */ std r11,PACA_EXSLB+EX_SRR0(r13) /* save SRR0 in exc. frame */ mfspr r11,SPRN_DAR /* ea */ diff --git a/arch/powerpc/kernel/fadump.c b/arch/powerpc/kernel/fadump.c index 7213d930918d..742694c1d852 100644 --- a/arch/powerpc/kernel/fadump.c +++ b/arch/powerpc/kernel/fadump.c @@ -69,7 +69,7 @@ int __init early_init_dt_scan_fw_dump(unsigned long node, */ token = of_get_flat_dt_prop(node, "ibm,configure-kernel-dump", NULL); if (!token) - return 0; + return 1; fw_dump.fadump_supported = 1; fw_dump.ibm_configure_kernel_dump = *token; @@ -92,7 +92,7 @@ int __init early_init_dt_scan_fw_dump(unsigned long node, &size); if (!sections) - return 0; + return 1; num_sections = size / (3 * sizeof(u32)); @@ -110,6 +110,7 @@ int __init early_init_dt_scan_fw_dump(unsigned long node, break; } } + return 1; } @@ -645,7 +646,7 @@ static int __init fadump_build_cpu_notes(const struct fadump_mem_struct *fdm) } /* Lower 4 bytes of reg_value contains logical cpu id */ cpu = reg_entry->reg_value & FADUMP_CPU_ID_MASK; - if (!cpumask_test_cpu(cpu, &fdh->cpu_online_mask)) { + if (fdh && !cpumask_test_cpu(cpu, &fdh->cpu_online_mask)) { SKIP_TO_NEXT_CPU(reg_entry); continue; } @@ -662,9 +663,11 @@ static int __init fadump_build_cpu_notes(const struct fadump_mem_struct *fdm) } fadump_final_note(note_buf); - pr_debug("Updating elfcore header (%llx) with cpu notes\n", + if (fdh) { + pr_debug("Updating elfcore header (%llx) with cpu notes\n", fdh->elfcorehdr_addr); - fadump_update_elfcore_header((char *)__va(fdh->elfcorehdr_addr)); + fadump_update_elfcore_header((char *)__va(fdh->elfcorehdr_addr)); + } return 0; error_out: diff --git a/arch/powerpc/kernel/ftrace.c b/arch/powerpc/kernel/ftrace.c index 6a014c763cc7..f202d0731b06 100644 --- a/arch/powerpc/kernel/ftrace.c +++ b/arch/powerpc/kernel/ftrace.c @@ -105,11 +105,9 @@ __ftrace_make_nop(struct module *mod, struct dyn_ftrace *rec, unsigned long addr) { unsigned int op; - unsigned int jmp[5]; unsigned long ptr; unsigned long ip = rec->ip; - unsigned long tramp; - int offset; + void *tramp; /* read where this goes */ if (probe_kernel_read(&op, (void *)ip, sizeof(int))) @@ -122,96 +120,41 @@ __ftrace_make_nop(struct module *mod, } /* lets find where the pointer goes */ - tramp = find_bl_target(ip, op); - - /* - * On PPC64 the trampoline looks like: - * 0x3d, 0x82, 0x00, 0x00, addis r12,r2, <high> - * 0x39, 0x8c, 0x00, 0x00, addi r12,r12, <low> - * Where the bytes 2,3,6 and 7 make up the 32bit offset - * to the TOC that holds the pointer. - * to jump to. - * 0xf8, 0x41, 0x00, 0x28, std r2,40(r1) - * 0xe9, 0x6c, 0x00, 0x20, ld r11,32(r12) - * The actually address is 32 bytes from the offset - * into the TOC. - * 0xe8, 0x4c, 0x00, 0x28, ld r2,40(r12) - */ - - pr_devel("ip:%lx jumps to %lx r2: %lx", ip, tramp, mod->arch.toc); - - /* Find where the trampoline jumps to */ - if (probe_kernel_read(jmp, (void *)tramp, sizeof(jmp))) { - printk(KERN_ERR "Failed to read %lx\n", tramp); - return -EFAULT; - } + tramp = (void *)find_bl_target(ip, op); - pr_devel(" %08x %08x", jmp[0], jmp[1]); + pr_devel("ip:%lx jumps to %p", ip, tramp); - /* verify that this is what we expect it to be */ - if (((jmp[0] & 0xffff0000) != 0x3d820000) || - ((jmp[1] & 0xffff0000) != 0x398c0000) || - (jmp[2] != 0xf8410028) || - (jmp[3] != 0xe96c0020) || - (jmp[4] != 0xe84c0028)) { + if (!is_module_trampoline(tramp)) { printk(KERN_ERR "Not a trampoline\n"); return -EINVAL; } - /* The bottom half is signed extended */ - offset = ((unsigned)((unsigned short)jmp[0]) << 16) + - (int)((short)jmp[1]); - - pr_devel(" %x ", offset); - - /* get the address this jumps too */ - tramp = mod->arch.toc + offset + 32; - pr_devel("toc: %lx", tramp); - - if (probe_kernel_read(jmp, (void *)tramp, 8)) { - printk(KERN_ERR "Failed to read %lx\n", tramp); + if (module_trampoline_target(mod, tramp, &ptr)) { + printk(KERN_ERR "Failed to get trampoline target\n"); return -EFAULT; } - pr_devel(" %08x %08x\n", jmp[0], jmp[1]); - -#ifdef __LITTLE_ENDIAN__ - ptr = ((unsigned long)jmp[1] << 32) + jmp[0]; -#else - ptr = ((unsigned long)jmp[0] << 32) + jmp[1]; -#endif + pr_devel("trampoline target %lx", ptr); /* This should match what was called */ if (ptr != ppc_function_entry((void *)addr)) { - printk(KERN_ERR "addr does not match %lx\n", ptr); + printk(KERN_ERR "addr %lx does not match expected %lx\n", + ptr, ppc_function_entry((void *)addr)); return -EINVAL; } /* - * We want to nop the line, but the next line is - * 0xe8, 0x41, 0x00, 0x28 ld r2,40(r1) - * This needs to be turned to a nop too. - */ - if (probe_kernel_read(&op, (void *)(ip+4), MCOUNT_INSN_SIZE)) - return -EFAULT; - - if (op != 0xe8410028) { - printk(KERN_ERR "Next line is not ld! (%08x)\n", op); - return -EINVAL; - } - - /* - * Milton Miller pointed out that we can not blindly do nops. - * If a task was preempted when calling a trace function, - * the nops will remove the way to restore the TOC in r2 - * and the r2 TOC will get corrupted. - */ - - /* - * Replace: - * bl <tramp> <==== will be replaced with "b 1f" - * ld r2,40(r1) - * 1: + * Our original call site looks like: + * + * bl <tramp> + * ld r2,XX(r1) + * + * Milton Miller pointed out that we can not simply nop the branch. + * If a task was preempted when calling a trace function, the nops + * will remove the way to restore the TOC in r2 and the r2 TOC will + * get corrupted. + * + * Use a b +8 to jump over the load. */ op = 0x48000008; /* b +8 */ @@ -349,19 +292,24 @@ static int __ftrace_make_call(struct dyn_ftrace *rec, unsigned long addr) { unsigned int op[2]; - unsigned long ip = rec->ip; + void *ip = (void *)rec->ip; /* read where this goes */ - if (probe_kernel_read(op, (void *)ip, MCOUNT_INSN_SIZE * 2)) + if (probe_kernel_read(op, ip, sizeof(op))) return -EFAULT; /* - * It should be pointing to two nops or - * b +8; ld r2,40(r1) + * We expect to see: + * + * b +8 + * ld r2,XX(r1) + * + * The load offset is different depending on the ABI. For simplicity + * just mask it out when doing the compare. */ - if (((op[0] != 0x48000008) || (op[1] != 0xe8410028)) && - ((op[0] != PPC_INST_NOP) || (op[1] != PPC_INST_NOP))) { - printk(KERN_ERR "Expected NOPs but have %x %x\n", op[0], op[1]); + if ((op[0] != 0x48000008) || ((op[1] & 0xffff00000) != 0xe8410000)) { + printk(KERN_ERR "Unexpected call sequence: %x %x\n", + op[0], op[1]); return -EINVAL; } @@ -371,23 +319,16 @@ __ftrace_make_call(struct dyn_ftrace *rec, unsigned long addr) return -EINVAL; } - /* create the branch to the trampoline */ - op[0] = create_branch((unsigned int *)ip, - rec->arch.mod->arch.tramp, BRANCH_SET_LINK); - if (!op[0]) { - printk(KERN_ERR "REL24 out of range!\n"); + /* Ensure branch is within 24 bits */ + if (create_branch(ip, rec->arch.mod->arch.tramp, BRANCH_SET_LINK)) { + printk(KERN_ERR "Branch out of range"); return -EINVAL; } - /* ld r2,40(r1) */ - op[1] = 0xe8410028; - - pr_devel("write to %lx\n", rec->ip); - - if (probe_kernel_write((void *)ip, op, MCOUNT_INSN_SIZE * 2)) - return -EPERM; - - flush_icache_range(ip, ip + 8); + if (patch_branch(ip, rec->arch.mod->arch.tramp, BRANCH_SET_LINK)) { + printk(KERN_ERR "REL24 out of range!\n"); + return -EINVAL; + } return 0; } diff --git a/arch/powerpc/kernel/head_64.S b/arch/powerpc/kernel/head_64.S index b7363bd42452..a95145d7f61b 100644 --- a/arch/powerpc/kernel/head_64.S +++ b/arch/powerpc/kernel/head_64.S @@ -70,16 +70,15 @@ _GLOBAL(__start) /* NOP this out unconditionally */ BEGIN_FTR_SECTION FIXUP_ENDIAN - b .__start_initialization_multiplatform + b __start_initialization_multiplatform END_FTR_SECTION(0, 1) /* Catch branch to 0 in real mode */ trap - /* Secondary processors spin on this value until it becomes nonzero. - * When it does it contains the real address of the descriptor - * of the function that the cpu should jump to to continue - * initialization. + /* Secondary processors spin on this value until it becomes non-zero. + * When non-zero, it contains the real address of the function the cpu + * should jump to. */ .balign 8 .globl __secondary_hold_spinloop @@ -140,16 +139,15 @@ __secondary_hold: tovirt(r26,r26) #endif /* All secondary cpus wait here until told to start. */ -100: ld r4,__secondary_hold_spinloop-_stext(r26) - cmpdi 0,r4,0 +100: ld r12,__secondary_hold_spinloop-_stext(r26) + cmpdi 0,r12,0 beq 100b #if defined(CONFIG_SMP) || defined(CONFIG_KEXEC) #ifdef CONFIG_PPC_BOOK3E - tovirt(r4,r4) + tovirt(r12,r12) #endif - ld r4,0(r4) /* deref function descriptor */ - mtctr r4 + mtctr r12 mr r3,r24 /* * it may be the case that other platforms have r4 right to @@ -186,16 +184,16 @@ _GLOBAL(generic_secondary_thread_init) mr r24,r3 /* turn on 64-bit mode */ - bl .enable_64b_mode + bl enable_64b_mode /* get a valid TOC pointer, wherever we're mapped at */ - bl .relative_toc + bl relative_toc tovirt(r2,r2) #ifdef CONFIG_PPC_BOOK3E /* Book3E initialization */ mr r3,r24 - bl .book3e_secondary_thread_init + bl book3e_secondary_thread_init #endif b generic_secondary_common_init @@ -214,17 +212,17 @@ _GLOBAL(generic_secondary_smp_init) mr r25,r4 /* turn on 64-bit mode */ - bl .enable_64b_mode + bl enable_64b_mode /* get a valid TOC pointer, wherever we're mapped at */ - bl .relative_toc + bl relative_toc tovirt(r2,r2) #ifdef CONFIG_PPC_BOOK3E /* Book3E initialization */ mr r3,r24 mr r4,r25 - bl .book3e_secondary_core_init + bl book3e_secondary_core_init #endif generic_secondary_common_init: @@ -236,7 +234,7 @@ generic_secondary_common_init: ld r13,0(r13) /* Get base vaddr of paca array */ #ifndef CONFIG_SMP addi r13,r13,PACA_SIZE /* know r13 if used accidentally */ - b .kexec_wait /* wait for next kernel if !SMP */ + b kexec_wait /* wait for next kernel if !SMP */ #else LOAD_REG_ADDR(r7, nr_cpu_ids) /* Load nr_cpu_ids address */ lwz r7,0(r7) /* also the max paca allocated */ @@ -250,7 +248,7 @@ generic_secondary_common_init: blt 1b mr r3,r24 /* not found, copy phys to r3 */ - b .kexec_wait /* next kernel might do better */ + b kexec_wait /* next kernel might do better */ 2: SET_PACA(r13) #ifdef CONFIG_PPC_BOOK3E @@ -264,11 +262,13 @@ generic_secondary_common_init: /* See if we need to call a cpu state restore handler */ LOAD_REG_ADDR(r23, cur_cpu_spec) ld r23,0(r23) - ld r23,CPU_SPEC_RESTORE(r23) - cmpdi 0,r23,0 + ld r12,CPU_SPEC_RESTORE(r23) + cmpdi 0,r12,0 beq 3f - ld r23,0(r23) - mtctr r23 +#if !defined(_CALL_ELF) || _CALL_ELF != 2 + ld r12,0(r12) +#endif + mtctr r12 bctrl 3: LOAD_REG_ADDR(r3, spinning_secondaries) /* Decrement spinning_secondaries */ @@ -299,7 +299,7 @@ generic_secondary_common_init: * Assumes we're mapped EA == RA if the MMU is on. */ #ifdef CONFIG_PPC_BOOK3S -_STATIC(__mmu_off) +__mmu_off: mfmsr r3 andi. r0,r3,MSR_IR|MSR_DR beqlr @@ -324,12 +324,12 @@ _STATIC(__mmu_off) * DT block, r4 is a physical pointer to the kernel itself * */ -_GLOBAL(__start_initialization_multiplatform) +__start_initialization_multiplatform: /* Make sure we are running in 64 bits mode */ - bl .enable_64b_mode + bl enable_64b_mode /* Get TOC pointer (current runtime address) */ - bl .relative_toc + bl relative_toc /* find out where we are now */ bcl 20,31,$+4 @@ -342,7 +342,7 @@ _GLOBAL(__start_initialization_multiplatform) */ cmpldi cr0,r5,0 beq 1f - b .__boot_from_prom /* yes -> prom */ + b __boot_from_prom /* yes -> prom */ 1: /* Save parameters */ mr r31,r3 @@ -354,8 +354,8 @@ _GLOBAL(__start_initialization_multiplatform) #endif #ifdef CONFIG_PPC_BOOK3E - bl .start_initialization_book3e - b .__after_prom_start + bl start_initialization_book3e + b __after_prom_start #else /* Setup some critical 970 SPRs before switching MMU off */ mfspr r0,SPRN_PVR @@ -368,15 +368,15 @@ _GLOBAL(__start_initialization_multiplatform) beq 1f cmpwi r0,0x45 /* 970GX */ bne 2f -1: bl .__cpu_preinit_ppc970 +1: bl __cpu_preinit_ppc970 2: /* Switch off MMU if not already off */ - bl .__mmu_off - b .__after_prom_start + bl __mmu_off + b __after_prom_start #endif /* CONFIG_PPC_BOOK3E */ -_INIT_STATIC(__boot_from_prom) +__boot_from_prom: #ifdef CONFIG_PPC_OF_BOOT_TRAMPOLINE /* Save parameters */ mr r31,r3 @@ -395,7 +395,7 @@ _INIT_STATIC(__boot_from_prom) #ifdef CONFIG_RELOCATABLE /* Relocate code for where we are now */ mr r3,r26 - bl .relocate + bl relocate #endif /* Restore parameters */ @@ -407,14 +407,14 @@ _INIT_STATIC(__boot_from_prom) /* Do all of the interaction with OF client interface */ mr r8,r26 - bl .prom_init + bl prom_init #endif /* #CONFIG_PPC_OF_BOOT_TRAMPOLINE */ /* We never return. We also hit that trap if trying to boot * from OF while CONFIG_PPC_OF_BOOT_TRAMPOLINE isn't selected */ trap -_STATIC(__after_prom_start) +__after_prom_start: #ifdef CONFIG_RELOCATABLE /* process relocations for the final address of the kernel */ lis r25,PAGE_OFFSET@highest /* compute virtual base of kernel */ @@ -424,7 +424,7 @@ _STATIC(__after_prom_start) bne 1f add r25,r25,r26 1: mr r3,r25 - bl .relocate + bl relocate #endif /* @@ -464,12 +464,12 @@ _STATIC(__after_prom_start) lis r5,(copy_to_here - _stext)@ha addi r5,r5,(copy_to_here - _stext)@l /* # bytes of memory to copy */ - bl .copy_and_flush /* copy the first n bytes */ + bl copy_and_flush /* copy the first n bytes */ /* this includes the code being */ /* executed here. */ addis r8,r3,(4f - _stext)@ha /* Jump to the copy of this code */ - addi r8,r8,(4f - _stext)@l /* that we just made */ - mtctr r8 + addi r12,r8,(4f - _stext)@l /* that we just made */ + mtctr r12 bctr .balign 8 @@ -478,9 +478,9 @@ p_end: .llong _end - _stext 4: /* Now copy the rest of the kernel up to _end */ addis r5,r26,(p_end - _stext)@ha ld r5,(p_end - _stext)@l(r5) /* get _end */ -5: bl .copy_and_flush /* copy the rest */ +5: bl copy_and_flush /* copy the rest */ -9: b .start_here_multiplatform +9: b start_here_multiplatform /* * Copy routine used to copy the kernel to start at physical address 0 @@ -544,7 +544,7 @@ __secondary_start_pmac_0: _GLOBAL(pmac_secondary_start) /* turn on 64-bit mode */ - bl .enable_64b_mode + bl enable_64b_mode li r0,0 mfspr r3,SPRN_HID4 @@ -556,11 +556,11 @@ _GLOBAL(pmac_secondary_start) slbia /* get TOC pointer (real address) */ - bl .relative_toc + bl relative_toc tovirt(r2,r2) /* Copy some CPU settings from CPU 0 */ - bl .__restore_cpu_ppc970 + bl __restore_cpu_ppc970 /* pSeries do that early though I don't think we really need it */ mfmsr r3 @@ -619,7 +619,7 @@ __secondary_start: std r14,PACAKSAVE(r13) /* Do early setup for that CPU (stab, slb, hash table pointer) */ - bl .early_setup_secondary + bl early_setup_secondary /* * setup the new stack pointer, but *don't* use this until @@ -639,7 +639,7 @@ __secondary_start: stb r0,PACAIRQHAPPENED(r13) /* enable MMU and jump to start_secondary */ - LOAD_REG_ADDR(r3, .start_secondary_prolog) + LOAD_REG_ADDR(r3, start_secondary_prolog) LOAD_REG_IMMEDIATE(r4, MSR_KERNEL) mtspr SPRN_SRR0,r3 @@ -652,11 +652,11 @@ __secondary_start: * zero the stack back-chain pointer and get the TOC virtual address * before going into C code. */ -_GLOBAL(start_secondary_prolog) +start_secondary_prolog: ld r2,PACATOC(r13) li r3,0 std r3,0(r1) /* Zero the stack frame pointer */ - bl .start_secondary + bl start_secondary b . /* * Reset stack pointer and call start_secondary @@ -667,14 +667,14 @@ _GLOBAL(start_secondary_resume) ld r1,PACAKSAVE(r13) /* Reload kernel stack pointer */ li r3,0 std r3,0(r1) /* Zero the stack frame pointer */ - bl .start_secondary + bl start_secondary b . #endif /* * This subroutine clobbers r11 and r12 */ -_GLOBAL(enable_64b_mode) +enable_64b_mode: mfmsr r11 /* grab the current MSR */ #ifdef CONFIG_PPC_BOOK3E oris r11,r11,0x8000 /* CM bit set, we'll set ICM later */ @@ -715,9 +715,9 @@ p_toc: .llong __toc_start + 0x8000 - 0b /* * This is where the main kernel code starts. */ -_INIT_STATIC(start_here_multiplatform) +start_here_multiplatform: /* set up the TOC */ - bl .relative_toc + bl relative_toc tovirt(r2,r2) /* Clear out the BSS. It may have been done in prom_init, @@ -776,9 +776,9 @@ _INIT_STATIC(start_here_multiplatform) /* Restore parameters passed from prom_init/kexec */ mr r3,r31 - bl .early_setup /* also sets r13 and SPRG_PACA */ + bl early_setup /* also sets r13 and SPRG_PACA */ - LOAD_REG_ADDR(r3, .start_here_common) + LOAD_REG_ADDR(r3, start_here_common) ld r4,PACAKMSR(r13) mtspr SPRN_SRR0,r3 mtspr SPRN_SRR1,r4 @@ -786,7 +786,8 @@ _INIT_STATIC(start_here_multiplatform) b . /* prevent speculative execution */ /* This is where all platforms converge execution */ -_INIT_GLOBAL(start_here_common) + +start_here_common: /* relocation is on at this point */ std r1,PACAKSAVE(r13) @@ -794,7 +795,7 @@ _INIT_GLOBAL(start_here_common) ld r2,PACATOC(r13) /* Do more system initializations in virtual mode */ - bl .setup_system + bl setup_system /* Mark interrupts soft and hard disabled (they might be enabled * in the PACA when doing hotplug) @@ -805,7 +806,7 @@ _INIT_GLOBAL(start_here_common) stb r0,PACAIRQHAPPENED(r13) /* Generic kernel entry */ - bl .start_kernel + bl start_kernel /* Not reached */ BUG_OPCODE diff --git a/arch/powerpc/kernel/hw_breakpoint.c b/arch/powerpc/kernel/hw_breakpoint.c index b0a1792279bb..0bb5918faaaf 100644 --- a/arch/powerpc/kernel/hw_breakpoint.c +++ b/arch/powerpc/kernel/hw_breakpoint.c @@ -72,7 +72,7 @@ int arch_install_hw_breakpoint(struct perf_event *bp) * If so, DABR will be populated in single_step_dabr_instruction(). */ if (current->thread.last_hit_ubp != bp) - set_breakpoint(info); + __set_breakpoint(info); return 0; } @@ -198,7 +198,7 @@ void thread_change_pc(struct task_struct *tsk, struct pt_regs *regs) info = counter_arch_bp(tsk->thread.last_hit_ubp); regs->msr &= ~MSR_SE; - set_breakpoint(info); + __set_breakpoint(info); tsk->thread.last_hit_ubp = NULL; } @@ -284,7 +284,7 @@ int __kprobes hw_breakpoint_handler(struct die_args *args) if (!(info->type & HW_BRK_TYPE_EXTRANEOUS_IRQ)) perf_bp_event(bp, regs); - set_breakpoint(info); + __set_breakpoint(info); out: rcu_read_unlock(); return rc; @@ -316,7 +316,7 @@ int __kprobes single_step_dabr_instruction(struct die_args *args) if (!(info->type & HW_BRK_TYPE_EXTRANEOUS_IRQ)) perf_bp_event(bp, regs); - set_breakpoint(info); + __set_breakpoint(info); current->thread.last_hit_ubp = NULL; /* diff --git a/arch/powerpc/kernel/idle_book3e.S b/arch/powerpc/kernel/idle_book3e.S index bfb73cc209ce..48c21acef915 100644 --- a/arch/powerpc/kernel/idle_book3e.S +++ b/arch/powerpc/kernel/idle_book3e.S @@ -43,7 +43,7 @@ _GLOBAL(\name) */ #ifdef CONFIG_TRACE_IRQFLAGS stdu r1,-128(r1) - bl .trace_hardirqs_on + bl trace_hardirqs_on addi r1,r1,128 #endif li r0,1 diff --git a/arch/powerpc/kernel/idle_power4.S b/arch/powerpc/kernel/idle_power4.S index e3edaa189911..f57a19348bdd 100644 --- a/arch/powerpc/kernel/idle_power4.S +++ b/arch/powerpc/kernel/idle_power4.S @@ -46,7 +46,7 @@ END_FTR_SECTION_IFCLR(CPU_FTR_CAN_NAP) mflr r0 std r0,16(r1) stdu r1,-128(r1) - bl .trace_hardirqs_on + bl trace_hardirqs_on addi r1,r1,128 ld r0,16(r1) mtlr r0 diff --git a/arch/powerpc/kernel/idle_power7.S b/arch/powerpc/kernel/idle_power7.S index c3ab86975614..2480256272d4 100644 --- a/arch/powerpc/kernel/idle_power7.S +++ b/arch/powerpc/kernel/idle_power7.S @@ -39,6 +39,10 @@ * Pass requested state in r3: * 0 - nap * 1 - sleep + * + * To check IRQ_HAPPENED in r4 + * 0 - don't check + * 1 - check */ _GLOBAL(power7_powersave_common) /* Use r3 to pass state nap/sleep/winkle */ @@ -58,7 +62,7 @@ _GLOBAL(power7_powersave_common) /* Make sure FPU, VSX etc... are flushed as we may lose * state when going to nap mode */ - bl .discard_lazy_cpu_state + bl discard_lazy_cpu_state #endif /* CONFIG_SMP */ /* Hard disable interrupts */ @@ -71,6 +75,8 @@ _GLOBAL(power7_powersave_common) lbz r0,PACAIRQHAPPENED(r13) cmpwi cr0,r0,0 beq 1f + cmpwi cr0,r4,0 + beq 1f addi r1,r1,INT_FRAME_SIZE ld r0,16(r1) mtlr r0 @@ -114,15 +120,18 @@ _GLOBAL(power7_idle) lwz r4,ADDROFF(powersave_nap)(r3) cmpwi 0,r4,0 beqlr + li r3, 1 /* fall through */ _GLOBAL(power7_nap) + mr r4,r3 li r3,0 b power7_powersave_common /* No return */ _GLOBAL(power7_sleep) li r3,1 + li r4,0 b power7_powersave_common /* No return */ @@ -168,7 +177,7 @@ _GLOBAL(power7_wakeup_loss) _GLOBAL(power7_wakeup_noloss) lbz r0,PACA_NAPSTATELOST(r13) cmpwi r0,0 - bne .power7_wakeup_loss + bne power7_wakeup_loss ld r1,PACAR1(r13) ld r4,_MSR(r1) ld r5,_NIP(r1) diff --git a/arch/powerpc/kernel/legacy_serial.c b/arch/powerpc/kernel/legacy_serial.c index 40bd7bd4e19a..936258881c98 100644 --- a/arch/powerpc/kernel/legacy_serial.c +++ b/arch/powerpc/kernel/legacy_serial.c @@ -48,6 +48,9 @@ static struct of_device_id legacy_serial_parents[] __initdata = { static unsigned int legacy_serial_count; static int legacy_serial_console = -1; +static const upf_t legacy_port_flags = UPF_BOOT_AUTOCONF | UPF_SKIP_TEST | + UPF_SHARE_IRQ | UPF_FIXED_PORT; + static unsigned int tsi_serial_in(struct uart_port *p, int offset) { unsigned int tmp; @@ -71,8 +74,9 @@ static int __init add_legacy_port(struct device_node *np, int want_index, phys_addr_t taddr, unsigned long irq, upf_t flags, int irq_check_parent) { - const __be32 *clk, *spd; + const __be32 *clk, *spd, *rs; u32 clock = BASE_BAUD * 16; + u32 shift = 0; int index; /* get clock freq. if present */ @@ -83,6 +87,11 @@ static int __init add_legacy_port(struct device_node *np, int want_index, /* get default speed if present */ spd = of_get_property(np, "current-speed", NULL); + /* get register shift if present */ + rs = of_get_property(np, "reg-shift", NULL); + if (rs && *rs) + shift = be32_to_cpup(rs); + /* If we have a location index, then try to use it */ if (want_index >= 0 && want_index < MAX_LEGACY_SERIAL_PORTS) index = want_index; @@ -126,6 +135,7 @@ static int __init add_legacy_port(struct device_node *np, int want_index, legacy_serial_ports[index].uartclk = clock; legacy_serial_ports[index].irq = irq; legacy_serial_ports[index].flags = flags; + legacy_serial_ports[index].regshift = shift; legacy_serial_infos[index].taddr = taddr; legacy_serial_infos[index].np = of_node_get(np); legacy_serial_infos[index].clock = clock; @@ -153,8 +163,6 @@ static int __init add_legacy_soc_port(struct device_node *np, { u64 addr; const __be32 *addrp; - upf_t flags = UPF_BOOT_AUTOCONF | UPF_SKIP_TEST | UPF_SHARE_IRQ - | UPF_FIXED_PORT; struct device_node *tsi = of_get_parent(np); /* We only support ports that have a clock frequency properly @@ -163,9 +171,8 @@ static int __init add_legacy_soc_port(struct device_node *np, if (of_get_property(np, "clock-frequency", NULL) == NULL) return -1; - /* if reg-shift or offset, don't try to use it */ - if ((of_get_property(np, "reg-shift", NULL) != NULL) || - (of_get_property(np, "reg-offset", NULL) != NULL)) + /* if reg-offset don't try to use it */ + if ((of_get_property(np, "reg-offset", NULL) != NULL)) return -1; /* if rtas uses this device, don't try to use it as well */ @@ -185,9 +192,11 @@ static int __init add_legacy_soc_port(struct device_node *np, * IO port value. It will be fixed up later along with the irq */ if (tsi && !strcmp(tsi->type, "tsi-bridge")) - return add_legacy_port(np, -1, UPIO_TSI, addr, addr, NO_IRQ, flags, 0); + return add_legacy_port(np, -1, UPIO_TSI, addr, addr, + NO_IRQ, legacy_port_flags, 0); else - return add_legacy_port(np, -1, UPIO_MEM, addr, addr, NO_IRQ, flags, 0); + return add_legacy_port(np, -1, UPIO_MEM, addr, addr, + NO_IRQ, legacy_port_flags, 0); } static int __init add_legacy_isa_port(struct device_node *np, @@ -233,7 +242,7 @@ static int __init add_legacy_isa_port(struct device_node *np, /* Add port, irq will be dealt with later */ return add_legacy_port(np, index, UPIO_PORT, be32_to_cpu(reg[1]), - taddr, NO_IRQ, UPF_BOOT_AUTOCONF, 0); + taddr, NO_IRQ, legacy_port_flags, 0); } @@ -306,7 +315,7 @@ static int __init add_legacy_pci_port(struct device_node *np, * IO port value. It will be fixed up later along with the irq */ return add_legacy_port(np, index, iotype, base, addr, NO_IRQ, - UPF_BOOT_AUTOCONF, np != pci_dev); + legacy_port_flags, np != pci_dev); } #endif @@ -315,17 +324,20 @@ static void __init setup_legacy_serial_console(int console) struct legacy_serial_info *info = &legacy_serial_infos[console]; struct plat_serial8250_port *port = &legacy_serial_ports[console]; void __iomem *addr; + unsigned int stride; + + stride = 1 << port->regshift; /* Check if a translated MMIO address has been found */ if (info->taddr) { addr = ioremap(info->taddr, 0x1000); if (addr == NULL) return; - udbg_uart_init_mmio(addr, 1); + udbg_uart_init_mmio(addr, stride); } else { /* Check if it's PIO and we support untranslated PIO */ if (port->iotype == UPIO_PORT && isa_io_special) - udbg_uart_init_pio(port->iobase, 1); + udbg_uart_init_pio(port->iobase, stride); else return; } diff --git a/arch/powerpc/kernel/misc_64.S b/arch/powerpc/kernel/misc_64.S index 3d0249599d52..4e314b90c75d 100644 --- a/arch/powerpc/kernel/misc_64.S +++ b/arch/powerpc/kernel/misc_64.S @@ -34,7 +34,7 @@ _GLOBAL(call_do_softirq) std r0,16(r1) stdu r1,THREAD_SIZE-STACK_FRAME_OVERHEAD(r3) mr r1,r3 - bl .__do_softirq + bl __do_softirq ld r1,0(r1) ld r0,16(r1) mtlr r0 @@ -45,7 +45,7 @@ _GLOBAL(call_do_irq) std r0,16(r1) stdu r1,THREAD_SIZE-STACK_FRAME_OVERHEAD(r4) mr r1,r4 - bl .__do_irq + bl __do_irq ld r1,0(r1) ld r0,16(r1) mtlr r0 @@ -506,7 +506,7 @@ _GLOBAL(kexec_smp_wait) stb r4,PACAKEXECSTATE(r13) SYNC - b .kexec_wait + b kexec_wait /* * switch to real mode (turn mmu off) @@ -576,7 +576,7 @@ _GLOBAL(kexec_sequence) /* copy dest pages, flush whole dest image */ mr r3,r29 - bl .kexec_copy_flush /* (image) */ + bl kexec_copy_flush /* (image) */ /* turn off mmu */ bl real_mode @@ -586,7 +586,7 @@ _GLOBAL(kexec_sequence) mr r4,r30 /* start, aka phys mem offset */ li r5,0x100 li r6,0 - bl .copy_and_flush /* (dest, src, copy limit, start offset) */ + bl copy_and_flush /* (dest, src, copy limit, start offset) */ 1: /* assume normal blr return */ /* release other cpus to the new kernel secondary start at 0x60 */ @@ -595,8 +595,12 @@ _GLOBAL(kexec_sequence) stw r6,kexec_flag-1b(5) /* clear out hardware hash page table and tlb */ - ld r5,0(r27) /* deref function descriptor */ - mtctr r5 +#if !defined(_CALL_ELF) || _CALL_ELF != 2 + ld r12,0(r27) /* deref function descriptor */ +#else + mr r12,r27 +#endif + mtctr r12 bctrl /* ppc_md.hpte_clear_all(void); */ /* @@ -630,3 +634,31 @@ _GLOBAL(kexec_sequence) li r5,0 blr /* image->start(physid, image->start, 0); */ #endif /* CONFIG_KEXEC */ + +#ifdef CONFIG_MODULES +#if defined(_CALL_ELF) && _CALL_ELF == 2 + +#ifdef CONFIG_MODVERSIONS +.weak __crc_TOC. +.section "___kcrctab+TOC.","a" +.globl __kcrctab_TOC. +__kcrctab_TOC.: + .llong __crc_TOC. +#endif + +/* + * Export a fake .TOC. since both modpost and depmod will complain otherwise. + * Both modpost and depmod strip the leading . so we do the same here. + */ +.section "__ksymtab_strings","a" +__kstrtab_TOC.: + .asciz "TOC." + +.section "___ksymtab+TOC.","a" +/* This symbol name is important: it's used by modpost to find exported syms */ +.globl __ksymtab_TOC. +__ksymtab_TOC.: + .llong 0 /* .value */ + .llong __kstrtab_TOC. +#endif /* ELFv2 */ +#endif /* MODULES */ diff --git a/arch/powerpc/kernel/module_64.c b/arch/powerpc/kernel/module_64.c index 12664c130d73..077d2ce6c5a7 100644 --- a/arch/powerpc/kernel/module_64.c +++ b/arch/powerpc/kernel/module_64.c @@ -22,6 +22,7 @@ #include <linux/vmalloc.h> #include <linux/ftrace.h> #include <linux/bug.h> +#include <linux/uaccess.h> #include <asm/module.h> #include <asm/firmware.h> #include <asm/code-patching.h> @@ -41,46 +42,170 @@ #define DEBUGP(fmt , ...) #endif +#if defined(_CALL_ELF) && _CALL_ELF == 2 +#define R2_STACK_OFFSET 24 + +/* An address is simply the address of the function. */ +typedef unsigned long func_desc_t; + +static func_desc_t func_desc(unsigned long addr) +{ + return addr; +} +static unsigned long func_addr(unsigned long addr) +{ + return addr; +} +static unsigned long stub_func_addr(func_desc_t func) +{ + return func; +} + +/* PowerPC64 specific values for the Elf64_Sym st_other field. */ +#define STO_PPC64_LOCAL_BIT 5 +#define STO_PPC64_LOCAL_MASK (7 << STO_PPC64_LOCAL_BIT) +#define PPC64_LOCAL_ENTRY_OFFSET(other) \ + (((1 << (((other) & STO_PPC64_LOCAL_MASK) >> STO_PPC64_LOCAL_BIT)) >> 2) << 2) + +static unsigned int local_entry_offset(const Elf64_Sym *sym) +{ + /* sym->st_other indicates offset to local entry point + * (otherwise it will assume r12 is the address of the start + * of function and try to derive r2 from it). */ + return PPC64_LOCAL_ENTRY_OFFSET(sym->st_other); +} +#else +#define R2_STACK_OFFSET 40 + +/* An address is address of the OPD entry, which contains address of fn. */ +typedef struct ppc64_opd_entry func_desc_t; + +static func_desc_t func_desc(unsigned long addr) +{ + return *(struct ppc64_opd_entry *)addr; +} +static unsigned long func_addr(unsigned long addr) +{ + return func_desc(addr).funcaddr; +} +static unsigned long stub_func_addr(func_desc_t func) +{ + return func.funcaddr; +} +static unsigned int local_entry_offset(const Elf64_Sym *sym) +{ + return 0; +} +#endif + /* Like PPC32, we need little trampolines to do > 24-bit jumps (into the kernel itself). But on PPC64, these need to be used for every jump, actually, to reset r2 (TOC+0x8000). */ struct ppc64_stub_entry { - /* 28 byte jump instruction sequence (7 instructions) */ - unsigned char jump[28]; - unsigned char unused[4]; + /* 28 byte jump instruction sequence (7 instructions). We only + * need 6 instructions on ABIv2 but we always allocate 7 so + * so we don't have to modify the trampoline load instruction. */ + u32 jump[7]; + u32 unused; /* Data for the above code */ - struct ppc64_opd_entry opd; + func_desc_t funcdata; }; -/* We use a stub to fix up r2 (TOC ptr) and to jump to the (external) - function which may be more than 24-bits away. We could simply - patch the new r2 value and function pointer into the stub, but it's - significantly shorter to put these values at the end of the stub - code, and patch the stub address (32-bits relative to the TOC ptr, - r2) into the stub. */ -static struct ppc64_stub_entry ppc64_stub = -{ .jump = { -#ifdef __LITTLE_ENDIAN__ - 0x00, 0x00, 0x82, 0x3d, /* addis r12,r2, <high> */ - 0x00, 0x00, 0x8c, 0x39, /* addi r12,r12, <low> */ - /* Save current r2 value in magic place on the stack. */ - 0x28, 0x00, 0x41, 0xf8, /* std r2,40(r1) */ - 0x20, 0x00, 0x6c, 0xe9, /* ld r11,32(r12) */ - 0x28, 0x00, 0x4c, 0xe8, /* ld r2,40(r12) */ - 0xa6, 0x03, 0x69, 0x7d, /* mtctr r11 */ - 0x20, 0x04, 0x80, 0x4e /* bctr */ -#else - 0x3d, 0x82, 0x00, 0x00, /* addis r12,r2, <high> */ - 0x39, 0x8c, 0x00, 0x00, /* addi r12,r12, <low> */ +/* + * PPC64 uses 24 bit jumps, but we need to jump into other modules or + * the kernel which may be further. So we jump to a stub. + * + * For ELFv1 we need to use this to set up the new r2 value (aka TOC + * pointer). For ELFv2 it's the callee's responsibility to set up the + * new r2, but for both we need to save the old r2. + * + * We could simply patch the new r2 value and function pointer into + * the stub, but it's significantly shorter to put these values at the + * end of the stub code, and patch the stub address (32-bits relative + * to the TOC ptr, r2) into the stub. + */ + +static u32 ppc64_stub_insns[] = { + 0x3d620000, /* addis r11,r2, <high> */ + 0x396b0000, /* addi r11,r11, <low> */ /* Save current r2 value in magic place on the stack. */ - 0xf8, 0x41, 0x00, 0x28, /* std r2,40(r1) */ - 0xe9, 0x6c, 0x00, 0x20, /* ld r11,32(r12) */ - 0xe8, 0x4c, 0x00, 0x28, /* ld r2,40(r12) */ - 0x7d, 0x69, 0x03, 0xa6, /* mtctr r11 */ - 0x4e, 0x80, 0x04, 0x20 /* bctr */ + 0xf8410000|R2_STACK_OFFSET, /* std r2,R2_STACK_OFFSET(r1) */ + 0xe98b0020, /* ld r12,32(r11) */ +#if !defined(_CALL_ELF) || _CALL_ELF != 2 + /* Set up new r2 from function descriptor */ + 0xe84b0028, /* ld r2,40(r11) */ +#endif + 0x7d8903a6, /* mtctr r12 */ + 0x4e800420 /* bctr */ +}; + +#ifdef CONFIG_DYNAMIC_FTRACE + +static u32 ppc64_stub_mask[] = { + 0xffff0000, + 0xffff0000, + 0xffffffff, + 0xffffffff, +#if !defined(_CALL_ELF) || _CALL_ELF != 2 + 0xffffffff, +#endif + 0xffffffff, + 0xffffffff +}; + +bool is_module_trampoline(u32 *p) +{ + unsigned int i; + u32 insns[ARRAY_SIZE(ppc64_stub_insns)]; + + BUILD_BUG_ON(sizeof(ppc64_stub_insns) != sizeof(ppc64_stub_mask)); + + if (probe_kernel_read(insns, p, sizeof(insns))) + return -EFAULT; + + for (i = 0; i < ARRAY_SIZE(ppc64_stub_insns); i++) { + u32 insna = insns[i]; + u32 insnb = ppc64_stub_insns[i]; + u32 mask = ppc64_stub_mask[i]; + + if ((insna & mask) != (insnb & mask)) + return false; + } + + return true; +} + +int module_trampoline_target(struct module *mod, u32 *trampoline, + unsigned long *target) +{ + u32 buf[2]; + u16 upper, lower; + long offset; + void *toc_entry; + + if (probe_kernel_read(buf, trampoline, sizeof(buf))) + return -EFAULT; + + upper = buf[0] & 0xffff; + lower = buf[1] & 0xffff; + + /* perform the addis/addi, both signed */ + offset = ((short)upper << 16) + (short)lower; + + /* + * Now get the address this trampoline jumps to. This + * is always 32 bytes into our trampoline stub. + */ + toc_entry = (void *)mod->arch.toc + offset + 32; + + if (probe_kernel_read(target, toc_entry, sizeof(*target))) + return -EFAULT; + + return 0; +} + #endif -} }; /* Count how many different 24-bit relocations (different symbol, different addend) */ @@ -183,6 +308,7 @@ static unsigned long get_stubs_size(const Elf64_Ehdr *hdr, return relocs * sizeof(struct ppc64_stub_entry); } +/* Still needed for ELFv2, for .TOC. */ static void dedotify_versions(struct modversion_info *vers, unsigned long size) { @@ -193,7 +319,7 @@ static void dedotify_versions(struct modversion_info *vers, memmove(vers->name, vers->name+1, strlen(vers->name)); } -/* Undefined symbols which refer to .funcname, hack to funcname */ +/* Undefined symbols which refer to .funcname, hack to funcname (or .TOC.) */ static void dedotify(Elf64_Sym *syms, unsigned int numsyms, char *strtab) { unsigned int i; @@ -207,6 +333,24 @@ static void dedotify(Elf64_Sym *syms, unsigned int numsyms, char *strtab) } } +static Elf64_Sym *find_dot_toc(Elf64_Shdr *sechdrs, + const char *strtab, + unsigned int symindex) +{ + unsigned int i, numsyms; + Elf64_Sym *syms; + + syms = (Elf64_Sym *)sechdrs[symindex].sh_addr; + numsyms = sechdrs[symindex].sh_size / sizeof(Elf64_Sym); + + for (i = 1; i < numsyms; i++) { + if (syms[i].st_shndx == SHN_UNDEF + && strcmp(strtab + syms[i].st_name, "TOC.") == 0) + return &syms[i]; + } + return NULL; +} + int module_frob_arch_sections(Elf64_Ehdr *hdr, Elf64_Shdr *sechdrs, char *secstrings, @@ -271,21 +415,12 @@ static inline unsigned long my_r2(Elf64_Shdr *sechdrs, struct module *me) /* Patch stub to reference function and correct r2 value. */ static inline int create_stub(Elf64_Shdr *sechdrs, struct ppc64_stub_entry *entry, - struct ppc64_opd_entry *opd, + unsigned long addr, struct module *me) { - Elf64_Half *loc1, *loc2; long reladdr; - *entry = ppc64_stub; - -#ifdef __LITTLE_ENDIAN__ - loc1 = (Elf64_Half *)&entry->jump[0]; - loc2 = (Elf64_Half *)&entry->jump[4]; -#else - loc1 = (Elf64_Half *)&entry->jump[2]; - loc2 = (Elf64_Half *)&entry->jump[6]; -#endif + memcpy(entry->jump, ppc64_stub_insns, sizeof(ppc64_stub_insns)); /* Stub uses address relative to r2. */ reladdr = (unsigned long)entry - my_r2(sechdrs, me); @@ -296,35 +431,33 @@ static inline int create_stub(Elf64_Shdr *sechdrs, } DEBUGP("Stub %p get data from reladdr %li\n", entry, reladdr); - *loc1 = PPC_HA(reladdr); - *loc2 = PPC_LO(reladdr); - entry->opd.funcaddr = opd->funcaddr; - entry->opd.r2 = opd->r2; + entry->jump[0] |= PPC_HA(reladdr); + entry->jump[1] |= PPC_LO(reladdr); + entry->funcdata = func_desc(addr); return 1; } -/* Create stub to jump to function described in this OPD: we need the +/* Create stub to jump to function described in this OPD/ptr: we need the stub to set up the TOC ptr (r2) for the function. */ static unsigned long stub_for_addr(Elf64_Shdr *sechdrs, - unsigned long opdaddr, + unsigned long addr, struct module *me) { struct ppc64_stub_entry *stubs; - struct ppc64_opd_entry *opd = (void *)opdaddr; unsigned int i, num_stubs; num_stubs = sechdrs[me->arch.stubs_section].sh_size / sizeof(*stubs); /* Find this stub, or if that fails, the next avail. entry */ stubs = (void *)sechdrs[me->arch.stubs_section].sh_addr; - for (i = 0; stubs[i].opd.funcaddr; i++) { + for (i = 0; stub_func_addr(stubs[i].funcdata); i++) { BUG_ON(i >= num_stubs); - if (stubs[i].opd.funcaddr == opd->funcaddr) + if (stub_func_addr(stubs[i].funcdata) == func_addr(addr)) return (unsigned long)&stubs[i]; } - if (!create_stub(sechdrs, &stubs[i], opd, me)) + if (!create_stub(sechdrs, &stubs[i], addr, me)) return 0; return (unsigned long)&stubs[i]; @@ -339,7 +472,8 @@ static int restore_r2(u32 *instruction, struct module *me) me->name, *instruction); return 0; } - *instruction = 0xe8410028; /* ld r2,40(r1) */ + /* ld r2,R2_STACK_OFFSET(r1) */ + *instruction = 0xe8410000 | R2_STACK_OFFSET; return 1; } @@ -357,6 +491,17 @@ int apply_relocate_add(Elf64_Shdr *sechdrs, DEBUGP("Applying ADD relocate section %u to %u\n", relsec, sechdrs[relsec].sh_info); + + /* First time we're called, we can fix up .TOC. */ + if (!me->arch.toc_fixed) { + sym = find_dot_toc(sechdrs, strtab, symindex); + /* It's theoretically possible that a module doesn't want a + * .TOC. so don't fail it just for that. */ + if (sym) + sym->st_value = my_r2(sechdrs, me); + me->arch.toc_fixed = true; + } + for (i = 0; i < sechdrs[relsec].sh_size / sizeof(*rela); i++) { /* This is where to make the change */ location = (void *)sechdrs[sechdrs[relsec].sh_info].sh_addr @@ -453,7 +598,8 @@ int apply_relocate_add(Elf64_Shdr *sechdrs, return -ENOENT; if (!restore_r2((u32 *)location + 1, me)) return -ENOEXEC; - } + } else + value += local_entry_offset(sym); /* Convert value to relative */ value -= (unsigned long)location; @@ -474,6 +620,31 @@ int apply_relocate_add(Elf64_Shdr *sechdrs, *location = value - (unsigned long)location; break; + case R_PPC64_TOCSAVE: + /* + * Marker reloc indicates we don't have to save r2. + * That would only save us one instruction, so ignore + * it. + */ + break; + + case R_PPC64_REL16_HA: + /* Subtract location pointer */ + value -= (unsigned long)location; + value = ((value + 0x8000) >> 16); + *((uint16_t *) location) + = (*((uint16_t *) location) & ~0xffff) + | (value & 0xffff); + break; + + case R_PPC64_REL16_LO: + /* Subtract location pointer */ + value -= (unsigned long)location; + *((uint16_t *) location) + = (*((uint16_t *) location) & ~0xffff) + | (value & 0xffff); + break; + default: printk("%s: Unknown ADD relocation: %lu\n", me->name, diff --git a/arch/powerpc/kernel/pci-common.c b/arch/powerpc/kernel/pci-common.c index 24d342e91790..b49c72fd7f16 100644 --- a/arch/powerpc/kernel/pci-common.c +++ b/arch/powerpc/kernel/pci-common.c @@ -21,6 +21,7 @@ #include <linux/string.h> #include <linux/init.h> #include <linux/bootmem.h> +#include <linux/delay.h> #include <linux/export.h> #include <linux/of_address.h> #include <linux/of_pci.h> @@ -120,6 +121,25 @@ resource_size_t pcibios_window_alignment(struct pci_bus *bus, return 1; } +void pcibios_reset_secondary_bus(struct pci_dev *dev) +{ + u16 ctrl; + + if (ppc_md.pcibios_reset_secondary_bus) { + ppc_md.pcibios_reset_secondary_bus(dev); + return; + } + + pci_read_config_word(dev, PCI_BRIDGE_CONTROL, &ctrl); + ctrl |= PCI_BRIDGE_CTL_BUS_RESET; + pci_write_config_word(dev, PCI_BRIDGE_CONTROL, ctrl); + msleep(2); + + ctrl &= ~PCI_BRIDGE_CTL_BUS_RESET; + pci_write_config_word(dev, PCI_BRIDGE_CONTROL, ctrl); + ssleep(1); +} + static resource_size_t pcibios_io_size(const struct pci_controller *hose) { #ifdef CONFIG_PPC64 @@ -646,60 +666,36 @@ void pci_resource_to_user(const struct pci_dev *dev, int bar, void pci_process_bridge_OF_ranges(struct pci_controller *hose, struct device_node *dev, int primary) { - const __be32 *ranges; - int rlen; - int pna = of_n_addr_cells(dev); - int np = pna + 5; int memno = 0; - u32 pci_space; - unsigned long long pci_addr, cpu_addr, pci_next, cpu_next, size; struct resource *res; + struct of_pci_range range; + struct of_pci_range_parser parser; printk(KERN_INFO "PCI host bridge %s %s ranges:\n", dev->full_name, primary ? "(primary)" : ""); - /* Get ranges property */ - ranges = of_get_property(dev, "ranges", &rlen); - if (ranges == NULL) + /* Check for ranges property */ + if (of_pci_range_parser_init(&parser, dev)) return; /* Parse it */ - while ((rlen -= np * 4) >= 0) { - /* Read next ranges element */ - pci_space = of_read_number(ranges, 1); - pci_addr = of_read_number(ranges + 1, 2); - cpu_addr = of_translate_address(dev, ranges + 3); - size = of_read_number(ranges + pna + 3, 2); - ranges += np; - + for_each_of_pci_range(&parser, &range) { /* If we failed translation or got a zero-sized region * (some FW try to feed us with non sensical zero sized regions * such as power3 which look like some kind of attempt at exposing * the VGA memory hole) */ - if (cpu_addr == OF_BAD_ADDR || size == 0) + if (range.cpu_addr == OF_BAD_ADDR || range.size == 0) continue; - /* Now consume following elements while they are contiguous */ - for (; rlen >= np * sizeof(u32); - ranges += np, rlen -= np * 4) { - if (of_read_number(ranges, 1) != pci_space) - break; - pci_next = of_read_number(ranges + 1, 2); - cpu_next = of_translate_address(dev, ranges + 3); - if (pci_next != pci_addr + size || - cpu_next != cpu_addr + size) - break; - size += of_read_number(ranges + pna + 3, 2); - } - /* Act based on address space type */ res = NULL; - switch ((pci_space >> 24) & 0x3) { - case 1: /* PCI IO space */ + switch (range.flags & IORESOURCE_TYPE_BITS) { + case IORESOURCE_IO: printk(KERN_INFO " IO 0x%016llx..0x%016llx -> 0x%016llx\n", - cpu_addr, cpu_addr + size - 1, pci_addr); + range.cpu_addr, range.cpu_addr + range.size - 1, + range.pci_addr); /* We support only one IO range */ if (hose->pci_io_size) { @@ -709,11 +705,12 @@ void pci_process_bridge_OF_ranges(struct pci_controller *hose, } #ifdef CONFIG_PPC32 /* On 32 bits, limit I/O space to 16MB */ - if (size > 0x01000000) - size = 0x01000000; + if (range.size > 0x01000000) + range.size = 0x01000000; /* 32 bits needs to map IOs here */ - hose->io_base_virt = ioremap(cpu_addr, size); + hose->io_base_virt = ioremap(range.cpu_addr, + range.size); /* Expect trouble if pci_addr is not 0 */ if (primary) @@ -723,20 +720,20 @@ void pci_process_bridge_OF_ranges(struct pci_controller *hose, /* pci_io_size and io_base_phys always represent IO * space starting at 0 so we factor in pci_addr */ - hose->pci_io_size = pci_addr + size; - hose->io_base_phys = cpu_addr - pci_addr; + hose->pci_io_size = range.pci_addr + range.size; + hose->io_base_phys = range.cpu_addr - range.pci_addr; /* Build resource */ res = &hose->io_resource; - res->flags = IORESOURCE_IO; - res->start = pci_addr; + range.cpu_addr = range.pci_addr; break; - case 2: /* PCI Memory space */ - case 3: /* PCI 64 bits Memory space */ + case IORESOURCE_MEM: printk(KERN_INFO " MEM 0x%016llx..0x%016llx -> 0x%016llx %s\n", - cpu_addr, cpu_addr + size - 1, pci_addr, - (pci_space & 0x40000000) ? "Prefetch" : ""); + range.cpu_addr, range.cpu_addr + range.size - 1, + range.pci_addr, + (range.pci_space & 0x40000000) ? + "Prefetch" : ""); /* We support only 3 memory ranges */ if (memno >= 3) { @@ -745,28 +742,21 @@ void pci_process_bridge_OF_ranges(struct pci_controller *hose, continue; } /* Handles ISA memory hole space here */ - if (pci_addr == 0) { + if (range.pci_addr == 0) { if (primary || isa_mem_base == 0) - isa_mem_base = cpu_addr; - hose->isa_mem_phys = cpu_addr; - hose->isa_mem_size = size; + isa_mem_base = range.cpu_addr; + hose->isa_mem_phys = range.cpu_addr; + hose->isa_mem_size = range.size; } /* Build resource */ - hose->mem_offset[memno] = cpu_addr - pci_addr; + hose->mem_offset[memno] = range.cpu_addr - + range.pci_addr; res = &hose->mem_resources[memno++]; - res->flags = IORESOURCE_MEM; - if (pci_space & 0x40000000) - res->flags |= IORESOURCE_PREFETCH; - res->start = cpu_addr; break; } if (res != NULL) { - res->name = dev->full_name; - res->end = res->start + size - 1; - res->parent = NULL; - res->sibling = NULL; - res->child = NULL; + of_pci_range_to_resource(&range, dev, res); } } } diff --git a/arch/powerpc/kernel/pci_of_scan.c b/arch/powerpc/kernel/pci_of_scan.c index 059e244484fe..44562aa97f16 100644 --- a/arch/powerpc/kernel/pci_of_scan.c +++ b/arch/powerpc/kernel/pci_of_scan.c @@ -304,6 +304,9 @@ static struct pci_dev *of_scan_pci_dev(struct pci_bus *bus, struct pci_dev *dev = NULL; const __be32 *reg; int reglen, devfn; +#ifdef CONFIG_EEH + struct eeh_dev *edev = of_node_to_eeh_dev(dn); +#endif pr_debug(" * %s\n", dn->full_name); if (!of_device_is_available(dn)) @@ -321,6 +324,12 @@ static struct pci_dev *of_scan_pci_dev(struct pci_bus *bus, return dev; } + /* Device removed permanently ? */ +#ifdef CONFIG_EEH + if (edev && (edev->mode & EEH_DEV_REMOVED)) + return NULL; +#endif + /* create a new pci_dev for this device */ dev = of_create_pci_dev(dn, bus, devfn); if (!dev) diff --git a/arch/powerpc/kernel/ppc_ksyms.c b/arch/powerpc/kernel/ppc_ksyms.c index 450850a49dce..48d17d6fca5b 100644 --- a/arch/powerpc/kernel/ppc_ksyms.c +++ b/arch/powerpc/kernel/ppc_ksyms.c @@ -155,9 +155,7 @@ EXPORT_SYMBOL(__cmpdi2); #endif long long __bswapdi2(long long); EXPORT_SYMBOL(__bswapdi2); -#ifdef __BIG_ENDIAN__ EXPORT_SYMBOL(memcpy); -#endif EXPORT_SYMBOL(memset); EXPORT_SYMBOL(memmove); EXPORT_SYMBOL(memcmp); diff --git a/arch/powerpc/kernel/process.c b/arch/powerpc/kernel/process.c index 31d021506d21..8a1edbe26b8f 100644 --- a/arch/powerpc/kernel/process.c +++ b/arch/powerpc/kernel/process.c @@ -54,6 +54,7 @@ #ifdef CONFIG_PPC64 #include <asm/firmware.h> #endif +#include <asm/code-patching.h> #include <linux/kprobes.h> #include <linux/kdebug.h> @@ -495,14 +496,21 @@ static inline int set_dawr(struct arch_hw_breakpoint *brk) return 0; } -int set_breakpoint(struct arch_hw_breakpoint *brk) +void __set_breakpoint(struct arch_hw_breakpoint *brk) { __get_cpu_var(current_brk) = *brk; if (cpu_has_feature(CPU_FTR_DAWR)) - return set_dawr(brk); + set_dawr(brk); + else + set_dabr(brk); +} - return set_dabr(brk); +void set_breakpoint(struct arch_hw_breakpoint *brk) +{ + preempt_disable(); + __set_breakpoint(brk); + preempt_enable(); } #ifdef CONFIG_PPC64 @@ -834,7 +842,7 @@ struct task_struct *__switch_to(struct task_struct *prev, */ #ifndef CONFIG_HAVE_HW_BREAKPOINT if (unlikely(!hw_brk_match(&__get_cpu_var(current_brk), &new->thread.hw_brk))) - set_breakpoint(&new->thread.hw_brk); + __set_breakpoint(&new->thread.hw_brk); #endif /* CONFIG_HAVE_HW_BREAKPOINT */ #endif @@ -1108,7 +1116,9 @@ int copy_thread(unsigned long clone_flags, unsigned long usp, struct thread_info *ti = (void *)task_stack_page(p); memset(childregs, 0, sizeof(struct pt_regs)); childregs->gpr[1] = sp + sizeof(struct pt_regs); - childregs->gpr[14] = usp; /* function */ + /* function */ + if (usp) + childregs->gpr[14] = ppc_function_entry((void *)usp); #ifdef CONFIG_PPC64 clear_tsk_thread_flag(p, TIF_32BIT); childregs->softe = 1; @@ -1187,17 +1197,7 @@ int copy_thread(unsigned long clone_flags, unsigned long usp, if (cpu_has_feature(CPU_FTR_HAS_PPR)) p->thread.ppr = INIT_PPR; #endif - /* - * The PPC64 ABI makes use of a TOC to contain function - * pointers. The function (ret_from_except) is actually a pointer - * to the TOC entry. The first entry is a pointer to the actual - * function. - */ -#ifdef CONFIG_PPC64 - kregs->nip = *((unsigned long *)f); -#else - kregs->nip = (unsigned long)f; -#endif + kregs->nip = ppc_function_entry(f); return 0; } diff --git a/arch/powerpc/kernel/prom_init_check.sh b/arch/powerpc/kernel/prom_init_check.sh index b0c263da219a..77aa1e95e904 100644 --- a/arch/powerpc/kernel/prom_init_check.sh +++ b/arch/powerpc/kernel/prom_init_check.sh @@ -23,7 +23,7 @@ strcmp strcpy strlcpy strlen strncmp strstr logo_linux_clut224 reloc_got2 kernstart_addr memstart_addr linux_banner _stext opal_query_takeover opal_do_takeover opal_enter_rtas opal_secondary_entry boot_command_line __prom_init_toc_start __prom_init_toc_end -btext_setup_display" +btext_setup_display TOC." NM="$1" OBJ="$2" diff --git a/arch/powerpc/kernel/rtas_pci.c b/arch/powerpc/kernel/rtas_pci.c index 7d4c7172f38e..c168337aef9d 100644 --- a/arch/powerpc/kernel/rtas_pci.c +++ b/arch/powerpc/kernel/rtas_pci.c @@ -80,10 +80,6 @@ int rtas_read_config(struct pci_dn *pdn, int where, int size, u32 *val) if (ret) return PCIBIOS_DEVICE_NOT_FOUND; - if (returnval == EEH_IO_ERROR_VALUE(size) && - eeh_dev_check_failure(of_node_to_eeh_dev(pdn->node))) - return PCIBIOS_DEVICE_NOT_FOUND; - return PCIBIOS_SUCCESSFUL; } @@ -92,18 +88,39 @@ static int rtas_pci_read_config(struct pci_bus *bus, int where, int size, u32 *val) { struct device_node *busdn, *dn; - - busdn = pci_bus_to_OF_node(bus); + struct pci_dn *pdn; + bool found = false; +#ifdef CONFIG_EEH + struct eeh_dev *edev; +#endif + int ret; /* Search only direct children of the bus */ + *val = 0xFFFFFFFF; + busdn = pci_bus_to_OF_node(bus); for (dn = busdn->child; dn; dn = dn->sibling) { - struct pci_dn *pdn = PCI_DN(dn); + pdn = PCI_DN(dn); if (pdn && pdn->devfn == devfn - && of_device_is_available(dn)) - return rtas_read_config(pdn, where, size, val); + && of_device_is_available(dn)) { + found = true; + break; + } } - return PCIBIOS_DEVICE_NOT_FOUND; + if (!found) + return PCIBIOS_DEVICE_NOT_FOUND; +#ifdef CONFIG_EEH + edev = of_node_to_eeh_dev(dn); + if (edev && edev->pe && edev->pe->state & EEH_PE_RESET) + return PCIBIOS_DEVICE_NOT_FOUND; +#endif + + ret = rtas_read_config(pdn, where, size, val); + if (*val == EEH_IO_ERROR_VALUE(size) && + eeh_dev_check_failure(of_node_to_eeh_dev(dn))) + return PCIBIOS_DEVICE_NOT_FOUND; + + return ret; } int rtas_write_config(struct pci_dn *pdn, int where, int size, u32 val) @@ -136,17 +153,34 @@ static int rtas_pci_write_config(struct pci_bus *bus, int where, int size, u32 val) { struct device_node *busdn, *dn; - - busdn = pci_bus_to_OF_node(bus); + struct pci_dn *pdn; + bool found = false; +#ifdef CONFIG_EEH + struct eeh_dev *edev; +#endif + int ret; /* Search only direct children of the bus */ + busdn = pci_bus_to_OF_node(bus); for (dn = busdn->child; dn; dn = dn->sibling) { - struct pci_dn *pdn = PCI_DN(dn); + pdn = PCI_DN(dn); if (pdn && pdn->devfn == devfn - && of_device_is_available(dn)) - return rtas_write_config(pdn, where, size, val); + && of_device_is_available(dn)) { + found = true; + break; + } } - return PCIBIOS_DEVICE_NOT_FOUND; + + if (!found) + return PCIBIOS_DEVICE_NOT_FOUND; +#ifdef CONFIG_EEH + edev = of_node_to_eeh_dev(dn); + if (edev && edev->pe && (edev->pe->state & EEH_PE_RESET)) + return PCIBIOS_DEVICE_NOT_FOUND; +#endif + ret = rtas_write_config(pdn, where, size, val); + + return ret; } static struct pci_ops rtas_pci_ops = { diff --git a/arch/powerpc/kernel/setup-common.c b/arch/powerpc/kernel/setup-common.c index fb87e4ac9e86..d4d418376f99 100644 --- a/arch/powerpc/kernel/setup-common.c +++ b/arch/powerpc/kernel/setup-common.c @@ -212,6 +212,7 @@ static int show_cpuinfo(struct seq_file *m, void *v) { unsigned long cpu_id = (unsigned long)v - 1; unsigned int pvr; + unsigned long proc_freq; unsigned short maj; unsigned short min; @@ -263,12 +264,19 @@ static int show_cpuinfo(struct seq_file *m, void *v) #endif /* CONFIG_TAU */ /* - * Assume here that all clock rates are the same in a - * smp system. -- Cort + * Platforms that have variable clock rates, should implement + * the method ppc_md.get_proc_freq() that reports the clock + * rate of a given cpu. The rest can use ppc_proc_freq to + * report the clock rate that is same across all cpus. */ - if (ppc_proc_freq) + if (ppc_md.get_proc_freq) + proc_freq = ppc_md.get_proc_freq(cpu_id); + else + proc_freq = ppc_proc_freq; + + if (proc_freq) seq_printf(m, "clock\t\t: %lu.%06luMHz\n", - ppc_proc_freq / 1000000, ppc_proc_freq % 1000000); + proc_freq / 1000000, proc_freq % 1000000); if (ppc_md.show_percpuinfo != NULL) ppc_md.show_percpuinfo(m, cpu_id); @@ -382,9 +390,10 @@ void __init check_for_initrd(void) #ifdef CONFIG_SMP -int threads_per_core, threads_shift; +int threads_per_core, threads_per_subcore, threads_shift; cpumask_t threads_core_mask; EXPORT_SYMBOL_GPL(threads_per_core); +EXPORT_SYMBOL_GPL(threads_per_subcore); EXPORT_SYMBOL_GPL(threads_shift); EXPORT_SYMBOL_GPL(threads_core_mask); @@ -393,6 +402,7 @@ static void __init cpu_init_thread_core_maps(int tpc) int i; threads_per_core = tpc; + threads_per_subcore = tpc; cpumask_clear(&threads_core_mask); /* This implementation only supports power of 2 number of threads diff --git a/arch/powerpc/kernel/setup_64.c b/arch/powerpc/kernel/setup_64.c index fbe24377eda3..ee082d771178 100644 --- a/arch/powerpc/kernel/setup_64.c +++ b/arch/powerpc/kernel/setup_64.c @@ -36,6 +36,7 @@ #include <linux/lockdep.h> #include <linux/memblock.h> #include <linux/hugetlb.h> +#include <linux/memory.h> #include <asm/io.h> #include <asm/kdump.h> @@ -341,7 +342,7 @@ void smp_release_cpus(void) ptr = (unsigned long *)((unsigned long)&__secondary_hold_spinloop - PHYSICAL_START); - *ptr = __pa(generic_secondary_smp_init); + *ptr = ppc_function_entry(generic_secondary_smp_init); /* And wait a bit for them to catch up */ for (i = 0; i < 100000; i++) { @@ -780,6 +781,15 @@ void __init setup_per_cpu_areas(void) } #endif +#ifdef CONFIG_MEMORY_HOTPLUG_SPARSE +unsigned long memory_block_size_bytes(void) +{ + if (ppc_md.memory_block_size) + return ppc_md.memory_block_size(); + + return MIN_MEMORY_BLOCK_SIZE; +} +#endif #if defined(CONFIG_PPC_INDIRECT_PIO) || defined(CONFIG_PPC_INDIRECT_MMIO) struct ppc_pci_io ppc_pci_io; diff --git a/arch/powerpc/kernel/signal.c b/arch/powerpc/kernel/signal.c index 8fc4177ed65a..1c794cef2883 100644 --- a/arch/powerpc/kernel/signal.c +++ b/arch/powerpc/kernel/signal.c @@ -134,7 +134,7 @@ static int do_signal(struct pt_regs *regs) */ if (current->thread.hw_brk.address && current->thread.hw_brk.type) - set_breakpoint(¤t->thread.hw_brk); + __set_breakpoint(¤t->thread.hw_brk); #endif /* Re-enable the breakpoints for the signal stack */ thread_change_pc(current, regs); diff --git a/arch/powerpc/kernel/smp.c b/arch/powerpc/kernel/smp.c index 10ffffef0414..7753af2d2613 100644 --- a/arch/powerpc/kernel/smp.c +++ b/arch/powerpc/kernel/smp.c @@ -36,6 +36,7 @@ #include <linux/atomic.h> #include <asm/irq.h> #include <asm/hw_irq.h> +#include <asm/kvm_ppc.h> #include <asm/page.h> #include <asm/pgtable.h> #include <asm/prom.h> @@ -390,6 +391,7 @@ void smp_prepare_boot_cpu(void) #ifdef CONFIG_PPC64 paca[boot_cpuid].__current = current; #endif + set_numa_node(numa_cpu_lookup_table[boot_cpuid]); current_set[boot_cpuid] = task_thread_info(current); } @@ -457,38 +459,9 @@ int generic_check_cpu_restart(unsigned int cpu) return per_cpu(cpu_state, cpu) == CPU_UP_PREPARE; } -static atomic_t secondary_inhibit_count; - -/* - * Don't allow secondary CPU threads to come online - */ -void inhibit_secondary_onlining(void) +static bool secondaries_inhibited(void) { - /* - * This makes secondary_inhibit_count stable during cpu - * online/offline operations. - */ - get_online_cpus(); - - atomic_inc(&secondary_inhibit_count); - put_online_cpus(); -} -EXPORT_SYMBOL_GPL(inhibit_secondary_onlining); - -/* - * Allow secondary CPU threads to come online again - */ -void uninhibit_secondary_onlining(void) -{ - get_online_cpus(); - atomic_dec(&secondary_inhibit_count); - put_online_cpus(); -} -EXPORT_SYMBOL_GPL(uninhibit_secondary_onlining); - -static int secondaries_inhibited(void) -{ - return atomic_read(&secondary_inhibit_count); + return kvm_hv_mode_active(); } #else /* HOTPLUG_CPU */ @@ -517,7 +490,7 @@ int __cpu_up(unsigned int cpu, struct task_struct *tidle) * Don't allow secondary threads to come online if inhibited */ if (threads_per_core > 1 && secondaries_inhibited() && - cpu % threads_per_core != 0) + cpu_thread_in_subcore(cpu)) return -EBUSY; if (smp_ops == NULL || @@ -750,6 +723,12 @@ void start_secondary(void *unused) } traverse_core_siblings(cpu, true); + /* + * numa_node_id() works after this. + */ + set_numa_node(numa_cpu_lookup_table[cpu]); + set_numa_mem(local_memory_node(numa_cpu_lookup_table[cpu])); + smp_wmb(); notify_cpu_starting(cpu); set_cpu_online(cpu, true); diff --git a/arch/powerpc/kernel/sysfs.c b/arch/powerpc/kernel/sysfs.c index d90d4b7810d6..67fd2fd2620a 100644 --- a/arch/powerpc/kernel/sysfs.c +++ b/arch/powerpc/kernel/sysfs.c @@ -404,7 +404,7 @@ void ppc_enable_pmcs(void) } EXPORT_SYMBOL(ppc_enable_pmcs); -#define __SYSFS_SPRSETUP(NAME, ADDRESS, EXTRA) \ +#define __SYSFS_SPRSETUP_READ_WRITE(NAME, ADDRESS, EXTRA) \ static void read_##NAME(void *val) \ { \ *(unsigned long *)val = mfspr(ADDRESS); \ @@ -413,7 +413,9 @@ static void write_##NAME(void *val) \ { \ EXTRA; \ mtspr(ADDRESS, *(unsigned long *)val); \ -} \ +} + +#define __SYSFS_SPRSETUP_SHOW_STORE(NAME) \ static ssize_t show_##NAME(struct device *dev, \ struct device_attribute *attr, \ char *buf) \ @@ -436,10 +438,15 @@ static ssize_t __used \ return count; \ } -#define SYSFS_PMCSETUP(NAME, ADDRESS) \ - __SYSFS_SPRSETUP(NAME, ADDRESS, ppc_enable_pmcs()) -#define SYSFS_SPRSETUP(NAME, ADDRESS) \ - __SYSFS_SPRSETUP(NAME, ADDRESS, ) +#define SYSFS_PMCSETUP(NAME, ADDRESS) \ + __SYSFS_SPRSETUP_READ_WRITE(NAME, ADDRESS, ppc_enable_pmcs()) \ + __SYSFS_SPRSETUP_SHOW_STORE(NAME) +#define SYSFS_SPRSETUP(NAME, ADDRESS) \ + __SYSFS_SPRSETUP_READ_WRITE(NAME, ADDRESS, ) \ + __SYSFS_SPRSETUP_SHOW_STORE(NAME) + +#define SYSFS_SPRSETUP_SHOW_STORE(NAME) \ + __SYSFS_SPRSETUP_SHOW_STORE(NAME) /* Let's define all possible registers, we'll only hook up the ones * that are implemented on the current processor @@ -477,7 +484,6 @@ SYSFS_PMCSETUP(pmc8, SPRN_PMC8); SYSFS_PMCSETUP(mmcra, SPRN_MMCRA); SYSFS_SPRSETUP(purr, SPRN_PURR); SYSFS_SPRSETUP(spurr, SPRN_SPURR); -SYSFS_SPRSETUP(dscr, SPRN_DSCR); SYSFS_SPRSETUP(pir, SPRN_PIR); /* @@ -487,12 +493,27 @@ SYSFS_SPRSETUP(pir, SPRN_PIR); */ static DEVICE_ATTR(mmcra, 0600, show_mmcra, store_mmcra); static DEVICE_ATTR(spurr, 0400, show_spurr, NULL); -static DEVICE_ATTR(dscr, 0600, show_dscr, store_dscr); static DEVICE_ATTR(purr, 0400, show_purr, store_purr); static DEVICE_ATTR(pir, 0400, show_pir, NULL); -unsigned long dscr_default = 0; -EXPORT_SYMBOL(dscr_default); +static unsigned long dscr_default; + +static void read_dscr(void *val) +{ + *(unsigned long *)val = get_paca()->dscr_default; +} + +static void write_dscr(void *val) +{ + get_paca()->dscr_default = *(unsigned long *)val; + if (!current->thread.dscr_inherit) { + current->thread.dscr = *(unsigned long *)val; + mtspr(SPRN_DSCR, *(unsigned long *)val); + } +} + +SYSFS_SPRSETUP_SHOW_STORE(dscr); +static DEVICE_ATTR(dscr, 0600, show_dscr, store_dscr); static void add_write_permission_dev_attr(struct device_attribute *attr) { @@ -505,14 +526,6 @@ static ssize_t show_dscr_default(struct device *dev, return sprintf(buf, "%lx\n", dscr_default); } -static void update_dscr(void *dummy) -{ - if (!current->thread.dscr_inherit) { - current->thread.dscr = dscr_default; - mtspr(SPRN_DSCR, dscr_default); - } -} - static ssize_t __used store_dscr_default(struct device *dev, struct device_attribute *attr, const char *buf, size_t count) @@ -525,7 +538,7 @@ static ssize_t __used store_dscr_default(struct device *dev, return -EINVAL; dscr_default = val; - on_each_cpu(update_dscr, NULL, 1); + on_each_cpu(write_dscr, &val, 1); return count; } diff --git a/arch/powerpc/kernel/systbl.S b/arch/powerpc/kernel/systbl.S index 93219c34af32..895c50ca943c 100644 --- a/arch/powerpc/kernel/systbl.S +++ b/arch/powerpc/kernel/systbl.S @@ -17,12 +17,12 @@ #include <asm/ppc_asm.h> #ifdef CONFIG_PPC64 -#define SYSCALL(func) .llong .sys_##func,.sys_##func -#define COMPAT_SYS(func) .llong .sys_##func,.compat_sys_##func -#define PPC_SYS(func) .llong .ppc_##func,.ppc_##func -#define OLDSYS(func) .llong .sys_ni_syscall,.sys_ni_syscall -#define SYS32ONLY(func) .llong .sys_ni_syscall,.compat_sys_##func -#define SYSX(f, f3264, f32) .llong .f,.f3264 +#define SYSCALL(func) .llong DOTSYM(sys_##func),DOTSYM(sys_##func) +#define COMPAT_SYS(func) .llong DOTSYM(sys_##func),DOTSYM(compat_sys_##func) +#define PPC_SYS(func) .llong DOTSYM(ppc_##func),DOTSYM(ppc_##func) +#define OLDSYS(func) .llong DOTSYM(sys_ni_syscall),DOTSYM(sys_ni_syscall) +#define SYS32ONLY(func) .llong DOTSYM(sys_ni_syscall),DOTSYM(compat_sys_##func) +#define SYSX(f, f3264, f32) .llong DOTSYM(f),DOTSYM(f3264) #else #define SYSCALL(func) .long sys_##func #define COMPAT_SYS(func) .long sys_##func @@ -36,6 +36,8 @@ #define PPC_SYS_SPU(func) PPC_SYS(func) #define SYSX_SPU(f, f3264, f32) SYSX(f, f3264, f32) +.section .rodata,"a" + #ifdef CONFIG_PPC64 #define sys_sigpending sys_ni_syscall #define sys_old_getrlimit sys_ni_syscall @@ -43,5 +45,7 @@ .p2align 3 #endif -_GLOBAL(sys_call_table) +.globl sys_call_table +sys_call_table: + #include <asm/systbl.h> diff --git a/arch/powerpc/kernel/tm.S b/arch/powerpc/kernel/tm.S index 03567c05950a..2a324f4cb1b9 100644 --- a/arch/powerpc/kernel/tm.S +++ b/arch/powerpc/kernel/tm.S @@ -10,6 +10,7 @@ #include <asm/ppc-opcode.h> #include <asm/ptrace.h> #include <asm/reg.h> +#include <asm/bug.h> #ifdef CONFIG_VSX /* See fpu.S, this is borrowed from there */ @@ -41,7 +42,6 @@ END_FTR_SECTION_IFSET(CPU_FTR_VSX); \ /* Stack frame offsets for local variables. */ #define TM_FRAME_L0 TM_FRAME_SIZE-16 #define TM_FRAME_L1 TM_FRAME_SIZE-8 -#define STACK_PARAM(x) (48+((x)*8)) /* In order to access the TM SPRs, TM must be enabled. So, do so: */ @@ -78,12 +78,6 @@ _GLOBAL(tm_abort) TABORT(R3) blr - .section ".toc","aw" -DSCR_DEFAULT: - .tc dscr_default[TC],dscr_default - - .section ".text" - /* void tm_reclaim(struct thread_struct *thread, * unsigned long orig_msr, * uint8_t cause) @@ -108,12 +102,12 @@ _GLOBAL(tm_reclaim) mflr r0 stw r6, 8(r1) std r0, 16(r1) - std r2, 40(r1) + std r2, STK_GOT(r1) stdu r1, -TM_FRAME_SIZE(r1) /* We've a struct pt_regs at [r1+STACK_FRAME_OVERHEAD]. */ - std r3, STACK_PARAM(0)(r1) + std r3, STK_PARAM(R3)(r1) SAVE_NVGPRS(r1) /* We need to setup MSR for VSX register save instructions. Here we @@ -175,6 +169,13 @@ dont_backup_vec: stfd fr0,FPSTATE_FPSCR(r7) dont_backup_fp: + /* Do sanity check on MSR to make sure we are suspended */ + li r7, (MSR_TS_S)@higher + srdi r6, r14, 32 + and r6, r6, r7 +1: tdeqi r6, 0 + EMIT_BUG_ENTRY 1b,__FILE__,__LINE__,0 + /* The moment we treclaim, ALL of our GPRs will switch * to user register state. (FPRs, CCR etc. also!) * Use an sprg and a tm_scratch in the PACA to shuffle. @@ -202,7 +203,7 @@ dont_backup_fp: /* Now get some more GPRS free */ std r7, GPR7(r1) /* Temporary stash */ std r12, GPR12(r1) /* '' '' '' */ - ld r12, STACK_PARAM(0)(r1) /* Param 0, thread_struct * */ + ld r12, STK_PARAM(R3)(r1) /* Param 0, thread_struct * */ std r11, THREAD_TM_PPR(r12) /* Store PPR and free r11 */ @@ -289,11 +290,10 @@ dont_backup_fp: ld r0, 16(r1) mtcr r4 mtlr r0 - ld r2, 40(r1) + ld r2, STK_GOT(r1) - /* Load system default DSCR */ - ld r4, DSCR_DEFAULT@toc(r2) - ld r0, 0(r4) + /* Load CPU's default DSCR */ + ld r0, PACA_DSCR(r13) mtspr SPRN_DSCR, r0 blr @@ -312,7 +312,7 @@ _GLOBAL(__tm_recheckpoint) mflr r0 stw r5, 8(r1) std r0, 16(r1) - std r2, 40(r1) + std r2, STK_GOT(r1) stdu r1, -TM_FRAME_SIZE(r1) /* We've a struct pt_regs at [r1+STACK_FRAME_OVERHEAD]. @@ -320,8 +320,6 @@ _GLOBAL(__tm_recheckpoint) */ SAVE_NVGPRS(r1) - std r1, PACAR1(r13) - /* Load complete register state from ts_ckpt* registers */ addi r7, r3, PT_CKPT_REGS /* Thread's ckpt_regs */ @@ -385,12 +383,10 @@ restore_gprs: /* ******************** CR,LR,CCR,MSR ********** */ ld r4, _CTR(r7) ld r5, _LINK(r7) - ld r6, _CCR(r7) ld r8, _XER(r7) mtctr r4 mtlr r5 - mtcr r6 mtxer r8 /* ******************** TAR ******************** */ @@ -406,7 +402,8 @@ restore_gprs: li r4, 0 mtmsrd r4, 1 - REST_4GPRS(0, r7) /* GPR0-3 */ + REST_GPR(0, r7) /* GPR0 */ + REST_2GPRS(2, r7) /* GPR2-3 */ REST_GPR(4, r7) /* GPR4 */ REST_4GPRS(8, r7) /* GPR8-11 */ REST_2GPRS(12, r7) /* GPR12-13 */ @@ -418,6 +415,31 @@ restore_gprs: mtspr SPRN_DSCR, r5 mtspr SPRN_PPR, r6 + /* Do final sanity check on TEXASR to make sure FS is set. Do this + * here before we load up the userspace r1 so any bugs we hit will get + * a call chain */ + mfspr r5, SPRN_TEXASR + srdi r5, r5, 16 + li r6, (TEXASR_FS)@h + and r6, r6, r5 +1: tdeqi r6, 0 + EMIT_BUG_ENTRY 1b,__FILE__,__LINE__,0 + + /* Do final sanity check on MSR to make sure we are not transactional + * or suspended + */ + mfmsr r6 + li r5, (MSR_TS_MASK)@higher + srdi r6, r6, 32 + and r6, r6, r5 +1: tdnei r6, 0 + EMIT_BUG_ENTRY 1b,__FILE__,__LINE__,0 + + /* Restore CR */ + ld r6, _CCR(r7) + mtcr r6 + + REST_GPR(1, r7) /* GPR1 */ REST_GPR(5, r7) /* GPR5-7 */ REST_GPR(6, r7) ld r7, GPR7(r7) @@ -448,11 +470,10 @@ restore_gprs: ld r0, 16(r1) mtcr r4 mtlr r0 - ld r2, 40(r1) + ld r2, STK_GOT(r1) - /* Load system default DSCR */ - ld r4, DSCR_DEFAULT@toc(r2) - ld r0, 0(r4) + /* Load CPU's default DSCR */ + ld r0, PACA_DSCR(r13) mtspr SPRN_DSCR, r0 blr diff --git a/arch/powerpc/kvm/book3s_hv.c b/arch/powerpc/kvm/book3s_hv.c index aba05bbb3e74..7a12edbb61e7 100644 --- a/arch/powerpc/kvm/book3s_hv.c +++ b/arch/powerpc/kvm/book3s_hv.c @@ -1236,7 +1236,7 @@ static struct kvm_vcpu *kvmppc_core_vcpu_create_hv(struct kvm *kvm, int core; struct kvmppc_vcore *vcore; - core = id / threads_per_core; + core = id / threads_per_subcore; if (core >= KVM_MAX_VCORES) goto out; @@ -1286,7 +1286,7 @@ static struct kvm_vcpu *kvmppc_core_vcpu_create_hv(struct kvm *kvm, init_waitqueue_head(&vcore->wq); vcore->preempt_tb = TB_NIL; vcore->lpcr = kvm->arch.lpcr; - vcore->first_vcpuid = core * threads_per_core; + vcore->first_vcpuid = core * threads_per_subcore; vcore->kvm = kvm; } kvm->arch.vcores[core] = vcore; @@ -1476,16 +1476,19 @@ static void kvmppc_wait_for_nap(struct kvmppc_vcore *vc) static int on_primary_thread(void) { int cpu = smp_processor_id(); - int thr = cpu_thread_in_core(cpu); + int thr; - if (thr) + /* Are we on a primary subcore? */ + if (cpu_thread_in_subcore(cpu)) return 0; - while (++thr < threads_per_core) + + thr = 0; + while (++thr < threads_per_subcore) if (cpu_online(cpu + thr)) return 0; /* Grab all hw threads so they can't go into the kernel */ - for (thr = 1; thr < threads_per_core; ++thr) { + for (thr = 1; thr < threads_per_subcore; ++thr) { if (kvmppc_grab_hwthread(cpu + thr)) { /* Couldn't grab one; let the others go */ do { @@ -1544,15 +1547,18 @@ static void kvmppc_run_core(struct kvmppc_vcore *vc) } /* - * Make sure we are running on thread 0, and that - * secondary threads are offline. + * Make sure we are running on primary threads, and that secondary + * threads are offline. Also check if the number of threads in this + * guest are greater than the current system threads per guest. */ - if (threads_per_core > 1 && !on_primary_thread()) { + if ((threads_per_core > 1) && + ((vc->num_threads > threads_per_subcore) || !on_primary_thread())) { list_for_each_entry(vcpu, &vc->runnable_threads, arch.run_list) vcpu->arch.ret = -EBUSY; goto out; } + vc->pcpu = smp_processor_id(); list_for_each_entry(vcpu, &vc->runnable_threads, arch.run_list) { kvmppc_start_thread(vcpu); @@ -1580,7 +1586,7 @@ static void kvmppc_run_core(struct kvmppc_vcore *vc) /* wait for secondary threads to finish writing their state to memory */ if (vc->nap_count < vc->n_woken) kvmppc_wait_for_nap(vc); - for (i = 0; i < threads_per_core; ++i) + for (i = 0; i < threads_per_subcore; ++i) kvmppc_release_hwthread(vc->pcpu + i); /* prevent other vcpu threads from doing kvmppc_start_thread() now */ vc->vcore_state = VCORE_EXITING; @@ -2305,10 +2311,10 @@ static int kvmppc_core_init_vm_hv(struct kvm *kvm) spin_lock_init(&kvm->arch.slot_phys_lock); /* - * Don't allow secondary CPU threads to come online - * while any KVM VMs exist. + * Track that we now have a HV mode VM active. This blocks secondary + * CPU threads from coming online. */ - inhibit_secondary_onlining(); + kvm_hv_vm_activated(); return 0; } @@ -2324,7 +2330,7 @@ static void kvmppc_free_vcores(struct kvm *kvm) static void kvmppc_core_destroy_vm_hv(struct kvm *kvm) { - uninhibit_secondary_onlining(); + kvm_hv_vm_deactivated(); kvmppc_free_vcores(kvm); if (kvm->arch.rma) { diff --git a/arch/powerpc/kvm/book3s_hv_builtin.c b/arch/powerpc/kvm/book3s_hv_builtin.c index 8cd0daebb82d..7cde8a665205 100644 --- a/arch/powerpc/kvm/book3s_hv_builtin.c +++ b/arch/powerpc/kvm/book3s_hv_builtin.c @@ -6,6 +6,7 @@ * published by the Free Software Foundation. */ +#include <linux/cpu.h> #include <linux/kvm_host.h> #include <linux/preempt.h> #include <linux/export.h> @@ -181,3 +182,33 @@ void __init kvm_cma_reserve(void) kvm_cma_declare_contiguous(selected_size, align_size); } } + +/* + * When running HV mode KVM we need to block certain operations while KVM VMs + * exist in the system. We use a counter of VMs to track this. + * + * One of the operations we need to block is onlining of secondaries, so we + * protect hv_vm_count with get/put_online_cpus(). + */ +static atomic_t hv_vm_count; + +void kvm_hv_vm_activated(void) +{ + get_online_cpus(); + atomic_inc(&hv_vm_count); + put_online_cpus(); +} +EXPORT_SYMBOL_GPL(kvm_hv_vm_activated); + +void kvm_hv_vm_deactivated(void) +{ + get_online_cpus(); + atomic_dec(&hv_vm_count); + put_online_cpus(); +} +EXPORT_SYMBOL_GPL(kvm_hv_vm_deactivated); + +bool kvm_hv_mode_active(void) +{ + return atomic_read(&hv_vm_count) != 0; +} diff --git a/arch/powerpc/kvm/book3s_hv_interrupts.S b/arch/powerpc/kvm/book3s_hv_interrupts.S index e18e3cfc32de..8c86422a1e37 100644 --- a/arch/powerpc/kvm/book3s_hv_interrupts.S +++ b/arch/powerpc/kvm/book3s_hv_interrupts.S @@ -171,7 +171,7 @@ END_FTR_SECTION_IFSET(CPU_FTR_ARCH_201) #endif /* CONFIG_SMP */ /* Jump to partition switch code */ - bl .kvmppc_hv_entry_trampoline + bl kvmppc_hv_entry_trampoline nop /* diff --git a/arch/powerpc/kvm/book3s_hv_rmhandlers.S b/arch/powerpc/kvm/book3s_hv_rmhandlers.S index 974793435a2e..77356fd25ccc 100644 --- a/arch/powerpc/kvm/book3s_hv_rmhandlers.S +++ b/arch/powerpc/kvm/book3s_hv_rmhandlers.S @@ -292,8 +292,7 @@ kvm_start_guest: beq kvm_no_guest /* Set HSTATE_DSCR(r13) to something sensible */ - LOAD_REG_ADDR(r6, dscr_default) - ld r6, 0(r6) + ld r6, PACA_DSCR(r13) std r6, HSTATE_DSCR(r13) bl kvmppc_hv_entry @@ -1799,7 +1798,7 @@ kvmppc_hdsi: /* Search the hash table. */ mr r3, r9 /* vcpu pointer */ li r7, 1 /* data fault */ - bl .kvmppc_hpte_hv_fault + bl kvmppc_hpte_hv_fault ld r9, HSTATE_KVM_VCPU(r13) ld r10, VCPU_PC(r9) ld r11, VCPU_MSR(r9) @@ -1873,7 +1872,7 @@ kvmppc_hisi: mr r4, r10 mr r6, r11 li r7, 0 /* instruction fault */ - bl .kvmppc_hpte_hv_fault + bl kvmppc_hpte_hv_fault ld r9, HSTATE_KVM_VCPU(r13) ld r10, VCPU_PC(r9) ld r11, VCPU_MSR(r9) @@ -1947,16 +1946,16 @@ hcall_real_fallback: .globl hcall_real_table hcall_real_table: .long 0 /* 0 - unused */ - .long .kvmppc_h_remove - hcall_real_table - .long .kvmppc_h_enter - hcall_real_table - .long .kvmppc_h_read - hcall_real_table + .long DOTSYM(kvmppc_h_remove) - hcall_real_table + .long DOTSYM(kvmppc_h_enter) - hcall_real_table + .long DOTSYM(kvmppc_h_read) - hcall_real_table .long 0 /* 0x10 - H_CLEAR_MOD */ .long 0 /* 0x14 - H_CLEAR_REF */ - .long .kvmppc_h_protect - hcall_real_table - .long .kvmppc_h_get_tce - hcall_real_table - .long .kvmppc_h_put_tce - hcall_real_table + .long DOTSYM(kvmppc_h_protect) - hcall_real_table + .long DOTSYM(kvmppc_h_get_tce) - hcall_real_table + .long DOTSYM(kvmppc_h_put_tce) - hcall_real_table .long 0 /* 0x24 - H_SET_SPRG0 */ - .long .kvmppc_h_set_dabr - hcall_real_table + .long DOTSYM(kvmppc_h_set_dabr) - hcall_real_table .long 0 /* 0x2c */ .long 0 /* 0x30 */ .long 0 /* 0x34 */ @@ -1972,11 +1971,11 @@ hcall_real_table: .long 0 /* 0x5c */ .long 0 /* 0x60 */ #ifdef CONFIG_KVM_XICS - .long .kvmppc_rm_h_eoi - hcall_real_table - .long .kvmppc_rm_h_cppr - hcall_real_table - .long .kvmppc_rm_h_ipi - hcall_real_table + .long DOTSYM(kvmppc_rm_h_eoi) - hcall_real_table + .long DOTSYM(kvmppc_rm_h_cppr) - hcall_real_table + .long DOTSYM(kvmppc_rm_h_ipi) - hcall_real_table .long 0 /* 0x70 - H_IPOLL */ - .long .kvmppc_rm_h_xirr - hcall_real_table + .long DOTSYM(kvmppc_rm_h_xirr) - hcall_real_table #else .long 0 /* 0x64 - H_EOI */ .long 0 /* 0x68 - H_CPPR */ @@ -2010,7 +2009,7 @@ hcall_real_table: .long 0 /* 0xd4 */ .long 0 /* 0xd8 */ .long 0 /* 0xdc */ - .long .kvmppc_h_cede - hcall_real_table + .long DOTSYM(kvmppc_h_cede) - hcall_real_table .long 0 /* 0xe4 */ .long 0 /* 0xe8 */ .long 0 /* 0xec */ @@ -2027,11 +2026,11 @@ hcall_real_table: .long 0 /* 0x118 */ .long 0 /* 0x11c */ .long 0 /* 0x120 */ - .long .kvmppc_h_bulk_remove - hcall_real_table + .long DOTSYM(kvmppc_h_bulk_remove) - hcall_real_table .long 0 /* 0x128 */ .long 0 /* 0x12c */ .long 0 /* 0x130 */ - .long .kvmppc_h_set_xdabr - hcall_real_table + .long DOTSYM(kvmppc_h_set_xdabr) - hcall_real_table hcall_real_table_end: ignore_hdec: @@ -2256,7 +2255,7 @@ kvm_cede_exit: /* Try to handle a machine check in real mode */ machine_check_realmode: mr r3, r9 /* get vcpu pointer */ - bl .kvmppc_realmode_machine_check + bl kvmppc_realmode_machine_check nop cmpdi r3, 0 /* continue exiting from guest? */ ld r9, HSTATE_KVM_VCPU(r13) diff --git a/arch/powerpc/kvm/powerpc.c b/arch/powerpc/kvm/powerpc.c index bab20f410443..61c738ab1283 100644 --- a/arch/powerpc/kvm/powerpc.c +++ b/arch/powerpc/kvm/powerpc.c @@ -426,7 +426,7 @@ int kvm_dev_ioctl_check_extension(long ext) #ifdef CONFIG_KVM_BOOK3S_HV_POSSIBLE case KVM_CAP_PPC_SMT: if (hv_enabled) - r = threads_per_core; + r = threads_per_subcore; else r = 0; break; diff --git a/arch/powerpc/lib/Makefile b/arch/powerpc/lib/Makefile index 95a20e17dbff..59fa2de9546d 100644 --- a/arch/powerpc/lib/Makefile +++ b/arch/powerpc/lib/Makefile @@ -23,9 +23,7 @@ obj-y += checksum_$(CONFIG_WORD_SIZE).o obj-$(CONFIG_PPC64) += checksum_wrappers_64.o endif -ifeq ($(CONFIG_CPU_LITTLE_ENDIAN),) obj-$(CONFIG_PPC64) += memcpy_power7.o memcpy_64.o -endif obj-$(CONFIG_PPC_EMULATE_SSTEP) += sstep.o ldstfp.o diff --git a/arch/powerpc/lib/copypage_64.S b/arch/powerpc/lib/copypage_64.S index 9f9434a85264..a3c4dc4defdd 100644 --- a/arch/powerpc/lib/copypage_64.S +++ b/arch/powerpc/lib/copypage_64.S @@ -16,11 +16,11 @@ PPC64_CACHES: .tc ppc64_caches[TC],ppc64_caches .section ".text" -_GLOBAL(copy_page) +_GLOBAL_TOC(copy_page) BEGIN_FTR_SECTION lis r5,PAGE_SIZE@h FTR_SECTION_ELSE - b .copypage_power7 + b copypage_power7 ALT_FTR_SECTION_END_IFCLR(CPU_FTR_VMX_COPY) ori r5,r5,PAGE_SIZE@l BEGIN_FTR_SECTION diff --git a/arch/powerpc/lib/copypage_power7.S b/arch/powerpc/lib/copypage_power7.S index 395c594722a2..d7dafb3777ac 100644 --- a/arch/powerpc/lib/copypage_power7.S +++ b/arch/powerpc/lib/copypage_power7.S @@ -56,15 +56,15 @@ _GLOBAL(copypage_power7) #ifdef CONFIG_ALTIVEC mflr r0 - std r3,48(r1) - std r4,56(r1) + std r3,-STACKFRAMESIZE+STK_REG(R31)(r1) + std r4,-STACKFRAMESIZE+STK_REG(R30)(r1) std r0,16(r1) stdu r1,-STACKFRAMESIZE(r1) - bl .enter_vmx_copy + bl enter_vmx_copy cmpwi r3,0 ld r0,STACKFRAMESIZE+16(r1) - ld r3,STACKFRAMESIZE+48(r1) - ld r4,STACKFRAMESIZE+56(r1) + ld r3,STK_REG(R31)(r1) + ld r4,STK_REG(R30)(r1) mtlr r0 li r0,(PAGE_SIZE/128) @@ -103,7 +103,7 @@ _GLOBAL(copypage_power7) addi r3,r3,128 bdnz 1b - b .exit_vmx_copy /* tail call optimise */ + b exit_vmx_copy /* tail call optimise */ #else li r0,(PAGE_SIZE/128) diff --git a/arch/powerpc/lib/copyuser_64.S b/arch/powerpc/lib/copyuser_64.S index 596a285c0755..0860ee46013c 100644 --- a/arch/powerpc/lib/copyuser_64.S +++ b/arch/powerpc/lib/copyuser_64.S @@ -18,7 +18,7 @@ #endif .align 7 -_GLOBAL(__copy_tofrom_user) +_GLOBAL_TOC(__copy_tofrom_user) BEGIN_FTR_SECTION nop FTR_SECTION_ELSE diff --git a/arch/powerpc/lib/copyuser_power7.S b/arch/powerpc/lib/copyuser_power7.S index e8e9c36dc784..c46c876ac96a 100644 --- a/arch/powerpc/lib/copyuser_power7.S +++ b/arch/powerpc/lib/copyuser_power7.S @@ -66,7 +66,7 @@ ld r15,STK_REG(R15)(r1) ld r14,STK_REG(R14)(r1) .Ldo_err3: - bl .exit_vmx_usercopy + bl exit_vmx_usercopy ld r0,STACKFRAMESIZE+16(r1) mtlr r0 b .Lexit @@ -85,9 +85,9 @@ .Lexit: addi r1,r1,STACKFRAMESIZE .Ldo_err1: - ld r3,48(r1) - ld r4,56(r1) - ld r5,64(r1) + ld r3,-STACKFRAMESIZE+STK_REG(R31)(r1) + ld r4,-STACKFRAMESIZE+STK_REG(R30)(r1) + ld r5,-STACKFRAMESIZE+STK_REG(R29)(r1) b __copy_tofrom_user_base @@ -96,18 +96,18 @@ _GLOBAL(__copy_tofrom_user_power7) cmpldi r5,16 cmpldi cr1,r5,4096 - std r3,48(r1) - std r4,56(r1) - std r5,64(r1) + std r3,-STACKFRAMESIZE+STK_REG(R31)(r1) + std r4,-STACKFRAMESIZE+STK_REG(R30)(r1) + std r5,-STACKFRAMESIZE+STK_REG(R29)(r1) blt .Lshort_copy bgt cr1,.Lvmx_copy #else cmpldi r5,16 - std r3,48(r1) - std r4,56(r1) - std r5,64(r1) + std r3,-STACKFRAMESIZE+STK_REG(R31)(r1) + std r4,-STACKFRAMESIZE+STK_REG(R30)(r1) + std r5,-STACKFRAMESIZE+STK_REG(R29)(r1) blt .Lshort_copy #endif @@ -295,12 +295,12 @@ err1; stb r0,0(r3) mflr r0 std r0,16(r1) stdu r1,-STACKFRAMESIZE(r1) - bl .enter_vmx_usercopy + bl enter_vmx_usercopy cmpwi cr1,r3,0 ld r0,STACKFRAMESIZE+16(r1) - ld r3,STACKFRAMESIZE+48(r1) - ld r4,STACKFRAMESIZE+56(r1) - ld r5,STACKFRAMESIZE+64(r1) + ld r3,STK_REG(R31)(r1) + ld r4,STK_REG(R30)(r1) + ld r5,STK_REG(R29)(r1) mtlr r0 /* @@ -514,7 +514,7 @@ err3; lbz r0,0(r4) err3; stb r0,0(r3) 15: addi r1,r1,STACKFRAMESIZE - b .exit_vmx_usercopy /* tail call optimise */ + b exit_vmx_usercopy /* tail call optimise */ .Lvmx_unaligned_copy: /* Get the destination 16B aligned */ @@ -717,5 +717,5 @@ err3; lbz r0,0(r4) err3; stb r0,0(r3) 15: addi r1,r1,STACKFRAMESIZE - b .exit_vmx_usercopy /* tail call optimise */ + b exit_vmx_usercopy /* tail call optimise */ #endif /* CONFiG_ALTIVEC */ diff --git a/arch/powerpc/lib/hweight_64.S b/arch/powerpc/lib/hweight_64.S index 9b96ff2ecd4d..19e66001a4f9 100644 --- a/arch/powerpc/lib/hweight_64.S +++ b/arch/powerpc/lib/hweight_64.S @@ -24,7 +24,7 @@ _GLOBAL(__arch_hweight8) BEGIN_FTR_SECTION - b .__sw_hweight8 + b __sw_hweight8 nop nop FTR_SECTION_ELSE @@ -35,7 +35,7 @@ ALT_FTR_SECTION_END_IFCLR(CPU_FTR_POPCNTB) _GLOBAL(__arch_hweight16) BEGIN_FTR_SECTION - b .__sw_hweight16 + b __sw_hweight16 nop nop nop @@ -57,7 +57,7 @@ ALT_FTR_SECTION_END_IFCLR(CPU_FTR_POPCNTB) _GLOBAL(__arch_hweight32) BEGIN_FTR_SECTION - b .__sw_hweight32 + b __sw_hweight32 nop nop nop @@ -82,7 +82,7 @@ ALT_FTR_SECTION_END_IFCLR(CPU_FTR_POPCNTB) _GLOBAL(__arch_hweight64) BEGIN_FTR_SECTION - b .__sw_hweight64 + b __sw_hweight64 nop nop nop diff --git a/arch/powerpc/lib/mem_64.S b/arch/powerpc/lib/mem_64.S index f4fcb0bc6563..0738f96befbf 100644 --- a/arch/powerpc/lib/mem_64.S +++ b/arch/powerpc/lib/mem_64.S @@ -79,8 +79,8 @@ _GLOBAL(memset) _GLOBAL(memmove) cmplw 0,r3,r4 - bgt .backwards_memcpy - b .memcpy + bgt backwards_memcpy + b memcpy _GLOBAL(backwards_memcpy) rlwinm. r7,r5,32-3,3,31 /* r0 = r5 >> 3 */ diff --git a/arch/powerpc/lib/memcpy_64.S b/arch/powerpc/lib/memcpy_64.S index 72ad055168a3..32a06ec395d2 100644 --- a/arch/powerpc/lib/memcpy_64.S +++ b/arch/powerpc/lib/memcpy_64.S @@ -10,14 +10,29 @@ #include <asm/ppc_asm.h> .align 7 -_GLOBAL(memcpy) +_GLOBAL_TOC(memcpy) BEGIN_FTR_SECTION - std r3,48(r1) /* save destination pointer for return value */ +#ifdef __LITTLE_ENDIAN__ + cmpdi cr7,r5,0 +#else + std r3,-STACKFRAMESIZE+STK_REG(R31)(r1) /* save destination pointer for return value */ +#endif FTR_SECTION_ELSE #ifndef SELFTEST b memcpy_power7 #endif ALT_FTR_SECTION_END_IFCLR(CPU_FTR_VMX_COPY) +#ifdef __LITTLE_ENDIAN__ + /* dumb little-endian memcpy that will get replaced at runtime */ + addi r9,r3,-1 + addi r4,r4,-1 + beqlr cr7 + mtctr r5 +1: lbzu r10,1(r4) + stbu r10,1(r9) + bdnz 1b + blr +#else PPC_MTOCRF(0x01,r5) cmpldi cr1,r5,16 neg r6,r3 # LS 3 bits = # bytes to 8-byte dest bdry @@ -73,7 +88,7 @@ END_FTR_SECTION_IFCLR(CPU_FTR_UNALIGNED_LD_STD) 2: bf cr7*4+3,3f lbz r9,8(r4) stb r9,0(r3) -3: ld r3,48(r1) /* return dest pointer */ +3: ld r3,-STACKFRAMESIZE+STK_REG(R31)(r1) /* return dest pointer */ blr .Lsrc_unaligned: @@ -156,7 +171,7 @@ END_FTR_SECTION_IFCLR(CPU_FTR_UNALIGNED_LD_STD) 2: bf cr7*4+3,3f rotldi r9,r9,8 stb r9,0(r3) -3: ld r3,48(r1) /* return dest pointer */ +3: ld r3,-STACKFRAMESIZE+STK_REG(R31)(r1) /* return dest pointer */ blr .Ldst_unaligned: @@ -201,5 +216,6 @@ END_FTR_SECTION_IFCLR(CPU_FTR_UNALIGNED_LD_STD) 3: bf cr7*4+3,4f lbz r0,0(r4) stb r0,0(r3) -4: ld r3,48(r1) /* return dest pointer */ +4: ld r3,-STACKFRAMESIZE+STK_REG(R31)(r1) /* return dest pointer */ blr +#endif diff --git a/arch/powerpc/lib/memcpy_power7.S b/arch/powerpc/lib/memcpy_power7.S index e4177dbea6bd..2ff5c142f87b 100644 --- a/arch/powerpc/lib/memcpy_power7.S +++ b/arch/powerpc/lib/memcpy_power7.S @@ -33,14 +33,14 @@ _GLOBAL(memcpy_power7) cmpldi r5,16 cmpldi cr1,r5,4096 - std r3,48(r1) + std r3,-STACKFRAMESIZE+STK_REG(R31)(r1) blt .Lshort_copy bgt cr1,.Lvmx_copy #else cmpldi r5,16 - std r3,48(r1) + std r3,-STACKFRAMESIZE+STK_REG(R31)(r1) blt .Lshort_copy #endif @@ -216,7 +216,7 @@ _GLOBAL(memcpy_power7) lbz r0,0(r4) stb r0,0(r3) -15: ld r3,48(r1) +15: ld r3,-STACKFRAMESIZE+STK_REG(R31)(r1) blr .Lunwind_stack_nonvmx_copy: @@ -226,16 +226,16 @@ _GLOBAL(memcpy_power7) #ifdef CONFIG_ALTIVEC .Lvmx_copy: mflr r0 - std r4,56(r1) - std r5,64(r1) + std r4,-STACKFRAMESIZE+STK_REG(R30)(r1) + std r5,-STACKFRAMESIZE+STK_REG(R29)(r1) std r0,16(r1) stdu r1,-STACKFRAMESIZE(r1) - bl .enter_vmx_copy + bl enter_vmx_copy cmpwi cr1,r3,0 ld r0,STACKFRAMESIZE+16(r1) - ld r3,STACKFRAMESIZE+48(r1) - ld r4,STACKFRAMESIZE+56(r1) - ld r5,STACKFRAMESIZE+64(r1) + ld r3,STK_REG(R31)(r1) + ld r4,STK_REG(R30)(r1) + ld r5,STK_REG(R29)(r1) mtlr r0 /* @@ -447,8 +447,8 @@ _GLOBAL(memcpy_power7) stb r0,0(r3) 15: addi r1,r1,STACKFRAMESIZE - ld r3,48(r1) - b .exit_vmx_copy /* tail call optimise */ + ld r3,-STACKFRAMESIZE+STK_REG(R31)(r1) + b exit_vmx_copy /* tail call optimise */ .Lvmx_unaligned_copy: /* Get the destination 16B aligned */ @@ -651,6 +651,6 @@ _GLOBAL(memcpy_power7) stb r0,0(r3) 15: addi r1,r1,STACKFRAMESIZE - ld r3,48(r1) - b .exit_vmx_copy /* tail call optimise */ + ld r3,-STACKFRAMESIZE+STK_REG(R31)(r1) + b exit_vmx_copy /* tail call optimise */ #endif /* CONFiG_ALTIVEC */ diff --git a/arch/powerpc/lib/string_64.S b/arch/powerpc/lib/string_64.S index 3b1e48049faf..7bd9549a90a2 100644 --- a/arch/powerpc/lib/string_64.S +++ b/arch/powerpc/lib/string_64.S @@ -77,7 +77,7 @@ err3; stb r0,0(r3) mr r3,r4 blr -_GLOBAL(__clear_user) +_GLOBAL_TOC(__clear_user) cmpdi r4,32 neg r6,r3 li r0,0 diff --git a/arch/powerpc/mm/hash_low_64.S b/arch/powerpc/mm/hash_low_64.S index 1136d26a95ae..057cbbb4c576 100644 --- a/arch/powerpc/mm/hash_low_64.S +++ b/arch/powerpc/mm/hash_low_64.S @@ -159,7 +159,7 @@ END_MMU_FTR_SECTION_IFSET(MMU_FTR_1T_SEGMENT) BEGIN_FTR_SECTION mr r4,r30 mr r5,r7 - bl .hash_page_do_lazy_icache + bl hash_page_do_lazy_icache END_FTR_SECTION(CPU_FTR_NOEXECUTE|CPU_FTR_COHERENT_ICACHE, CPU_FTR_NOEXECUTE) /* At this point, r3 contains new PP bits, save them in @@ -201,7 +201,8 @@ htab_insert_pte: li r8,MMU_PAGE_4K /* page size */ li r9,MMU_PAGE_4K /* actual page size */ ld r10,STK_PARAM(R9)(r1) /* segment size */ -_GLOBAL(htab_call_hpte_insert1) +.globl htab_call_hpte_insert1 +htab_call_hpte_insert1: bl . /* Patched by htab_finish_init() */ cmpdi 0,r3,0 bge htab_pte_insert_ok /* Insertion successful */ @@ -225,7 +226,8 @@ _GLOBAL(htab_call_hpte_insert1) li r8,MMU_PAGE_4K /* page size */ li r9,MMU_PAGE_4K /* actual page size */ ld r10,STK_PARAM(R9)(r1) /* segment size */ -_GLOBAL(htab_call_hpte_insert2) +.globl htab_call_hpte_insert2 +htab_call_hpte_insert2: bl . /* Patched by htab_finish_init() */ cmpdi 0,r3,0 bge+ htab_pte_insert_ok /* Insertion successful */ @@ -242,7 +244,8 @@ _GLOBAL(htab_call_hpte_insert2) 2: and r0,r5,r27 rldicr r3,r0,3,63-3 /* r0 = (hash & mask) << 3 */ /* Call ppc_md.hpte_remove */ -_GLOBAL(htab_call_hpte_remove) +.globl htab_call_hpte_remove +htab_call_hpte_remove: bl . /* Patched by htab_finish_init() */ /* Try all again */ @@ -296,7 +299,8 @@ htab_modify_pte: li r7,MMU_PAGE_4K /* actual page size */ ld r8,STK_PARAM(R9)(r1) /* segment size */ ld r9,STK_PARAM(R8)(r1) /* get "local" param */ -_GLOBAL(htab_call_hpte_updatepp) +.globl htab_call_hpte_updatepp +htab_call_hpte_updatepp: bl . /* Patched by htab_finish_init() */ /* if we failed because typically the HPTE wasn't really here @@ -471,7 +475,7 @@ END_MMU_FTR_SECTION_IFSET(MMU_FTR_1T_SEGMENT) BEGIN_FTR_SECTION mr r4,r30 mr r5,r7 - bl .hash_page_do_lazy_icache + bl hash_page_do_lazy_icache END_FTR_SECTION(CPU_FTR_NOEXECUTE|CPU_FTR_COHERENT_ICACHE, CPU_FTR_NOEXECUTE) /* At this point, r3 contains new PP bits, save them in @@ -526,7 +530,8 @@ htab_special_pfn: li r8,MMU_PAGE_4K /* page size */ li r9,MMU_PAGE_4K /* actual page size */ ld r10,STK_PARAM(R9)(r1) /* segment size */ -_GLOBAL(htab_call_hpte_insert1) +.globl htab_call_hpte_insert1 +htab_call_hpte_insert1: bl . /* patched by htab_finish_init() */ cmpdi 0,r3,0 bge htab_pte_insert_ok /* Insertion successful */ @@ -554,7 +559,8 @@ _GLOBAL(htab_call_hpte_insert1) li r8,MMU_PAGE_4K /* page size */ li r9,MMU_PAGE_4K /* actual page size */ ld r10,STK_PARAM(R9)(r1) /* segment size */ -_GLOBAL(htab_call_hpte_insert2) +.globl htab_call_hpte_insert2 +htab_call_hpte_insert2: bl . /* patched by htab_finish_init() */ cmpdi 0,r3,0 bge+ htab_pte_insert_ok /* Insertion successful */ @@ -571,7 +577,8 @@ _GLOBAL(htab_call_hpte_insert2) 2: and r0,r5,r27 rldicr r3,r0,3,63-3 /* r0 = (hash & mask) << 3 */ /* Call ppc_md.hpte_remove */ -_GLOBAL(htab_call_hpte_remove) +.globl htab_call_hpte_remove +htab_call_hpte_remove: bl . /* patched by htab_finish_init() */ /* Try all again */ @@ -588,7 +595,7 @@ htab_inval_old_hpte: li r6,MMU_PAGE_64K /* psize */ ld r7,STK_PARAM(R9)(r1) /* ssize */ ld r8,STK_PARAM(R8)(r1) /* local */ - bl .flush_hash_page + bl flush_hash_page /* Clear out _PAGE_HPTE_SUB bits in the new linux PTE */ lis r0,_PAGE_HPTE_SUB@h ori r0,r0,_PAGE_HPTE_SUB@l @@ -660,7 +667,8 @@ htab_modify_pte: li r7,MMU_PAGE_4K /* actual page size */ ld r8,STK_PARAM(R9)(r1) /* segment size */ ld r9,STK_PARAM(R8)(r1) /* get "local" param */ -_GLOBAL(htab_call_hpte_updatepp) +.globl htab_call_hpte_updatepp +htab_call_hpte_updatepp: bl . /* patched by htab_finish_init() */ /* if we failed because typically the HPTE wasn't really here @@ -812,7 +820,7 @@ END_MMU_FTR_SECTION_IFSET(MMU_FTR_1T_SEGMENT) BEGIN_FTR_SECTION mr r4,r30 mr r5,r7 - bl .hash_page_do_lazy_icache + bl hash_page_do_lazy_icache END_FTR_SECTION(CPU_FTR_NOEXECUTE|CPU_FTR_COHERENT_ICACHE, CPU_FTR_NOEXECUTE) /* At this point, r3 contains new PP bits, save them in @@ -857,7 +865,8 @@ ht64_insert_pte: li r8,MMU_PAGE_64K li r9,MMU_PAGE_64K /* actual page size */ ld r10,STK_PARAM(R9)(r1) /* segment size */ -_GLOBAL(ht64_call_hpte_insert1) +.globl ht64_call_hpte_insert1 +ht64_call_hpte_insert1: bl . /* patched by htab_finish_init() */ cmpdi 0,r3,0 bge ht64_pte_insert_ok /* Insertion successful */ @@ -881,7 +890,8 @@ _GLOBAL(ht64_call_hpte_insert1) li r8,MMU_PAGE_64K li r9,MMU_PAGE_64K /* actual page size */ ld r10,STK_PARAM(R9)(r1) /* segment size */ -_GLOBAL(ht64_call_hpte_insert2) +.globl ht64_call_hpte_insert2 +ht64_call_hpte_insert2: bl . /* patched by htab_finish_init() */ cmpdi 0,r3,0 bge+ ht64_pte_insert_ok /* Insertion successful */ @@ -898,7 +908,8 @@ _GLOBAL(ht64_call_hpte_insert2) 2: and r0,r5,r27 rldicr r3,r0,3,63-3 /* r0 = (hash & mask) << 3 */ /* Call ppc_md.hpte_remove */ -_GLOBAL(ht64_call_hpte_remove) +.globl ht64_call_hpte_remove +ht64_call_hpte_remove: bl . /* patched by htab_finish_init() */ /* Try all again */ @@ -952,7 +963,8 @@ ht64_modify_pte: li r7,MMU_PAGE_64K /* actual page size */ ld r8,STK_PARAM(R9)(r1) /* segment size */ ld r9,STK_PARAM(R8)(r1) /* get "local" param */ -_GLOBAL(ht64_call_hpte_updatepp) +.globl ht64_call_hpte_updatepp +ht64_call_hpte_updatepp: bl . /* patched by htab_finish_init() */ /* if we failed because typically the HPTE wasn't really here diff --git a/arch/powerpc/mm/hash_utils_64.c b/arch/powerpc/mm/hash_utils_64.c index 350aa58a6f95..88fdd9d25077 100644 --- a/arch/powerpc/mm/hash_utils_64.c +++ b/arch/powerpc/mm/hash_utils_64.c @@ -449,6 +449,24 @@ static void mmu_psize_set_default_penc(void) mmu_psize_defs[bpsize].penc[apsize] = -1; } +#ifdef CONFIG_PPC_64K_PAGES + +static bool might_have_hea(void) +{ + /* + * The HEA ethernet adapter requires awareness of the + * GX bus. Without that awareness we can easily assume + * we will never see an HEA ethernet device. + */ +#ifdef CONFIG_IBMEBUS + return !cpu_has_feature(CPU_FTR_ARCH_207S); +#else + return false; +#endif +} + +#endif /* #ifdef CONFIG_PPC_64K_PAGES */ + static void __init htab_init_page_sizes(void) { int rc; @@ -503,10 +521,11 @@ static void __init htab_init_page_sizes(void) mmu_linear_psize = MMU_PAGE_64K; if (mmu_has_feature(MMU_FTR_CI_LARGE_PAGE)) { /* - * Don't use 64k pages for ioremap on pSeries, since - * that would stop us accessing the HEA ethernet. + * When running on pSeries using 64k pages for ioremap + * would stop us accessing the HEA ethernet. So if we + * have the chance of ever seeing one, stay at 4k. */ - if (!machine_is(pseries)) + if (!might_have_hea() || !machine_is(pseries)) mmu_io_psize = MMU_PAGE_64K; } else mmu_ci_restrictions = 1; @@ -607,47 +626,43 @@ int remove_section_mapping(unsigned long start, unsigned long end) } #endif /* CONFIG_MEMORY_HOTPLUG */ -#define FUNCTION_TEXT(A) ((*(unsigned long *)(A))) +extern u32 htab_call_hpte_insert1[]; +extern u32 htab_call_hpte_insert2[]; +extern u32 htab_call_hpte_remove[]; +extern u32 htab_call_hpte_updatepp[]; +extern u32 ht64_call_hpte_insert1[]; +extern u32 ht64_call_hpte_insert2[]; +extern u32 ht64_call_hpte_remove[]; +extern u32 ht64_call_hpte_updatepp[]; static void __init htab_finish_init(void) { - extern unsigned int *htab_call_hpte_insert1; - extern unsigned int *htab_call_hpte_insert2; - extern unsigned int *htab_call_hpte_remove; - extern unsigned int *htab_call_hpte_updatepp; - #ifdef CONFIG_PPC_HAS_HASH_64K - extern unsigned int *ht64_call_hpte_insert1; - extern unsigned int *ht64_call_hpte_insert2; - extern unsigned int *ht64_call_hpte_remove; - extern unsigned int *ht64_call_hpte_updatepp; - patch_branch(ht64_call_hpte_insert1, - FUNCTION_TEXT(ppc_md.hpte_insert), + ppc_function_entry(ppc_md.hpte_insert), BRANCH_SET_LINK); patch_branch(ht64_call_hpte_insert2, - FUNCTION_TEXT(ppc_md.hpte_insert), + ppc_function_entry(ppc_md.hpte_insert), BRANCH_SET_LINK); patch_branch(ht64_call_hpte_remove, - FUNCTION_TEXT(ppc_md.hpte_remove), + ppc_function_entry(ppc_md.hpte_remove), BRANCH_SET_LINK); patch_branch(ht64_call_hpte_updatepp, - FUNCTION_TEXT(ppc_md.hpte_updatepp), + ppc_function_entry(ppc_md.hpte_updatepp), BRANCH_SET_LINK); - #endif /* CONFIG_PPC_HAS_HASH_64K */ patch_branch(htab_call_hpte_insert1, - FUNCTION_TEXT(ppc_md.hpte_insert), + ppc_function_entry(ppc_md.hpte_insert), BRANCH_SET_LINK); patch_branch(htab_call_hpte_insert2, - FUNCTION_TEXT(ppc_md.hpte_insert), + ppc_function_entry(ppc_md.hpte_insert), BRANCH_SET_LINK); patch_branch(htab_call_hpte_remove, - FUNCTION_TEXT(ppc_md.hpte_remove), + ppc_function_entry(ppc_md.hpte_remove), BRANCH_SET_LINK); patch_branch(htab_call_hpte_updatepp, - FUNCTION_TEXT(ppc_md.hpte_updatepp), + ppc_function_entry(ppc_md.hpte_updatepp), BRANCH_SET_LINK); } @@ -964,6 +979,22 @@ void hash_failure_debug(unsigned long ea, unsigned long access, trap, vsid, ssize, psize, lpsize, pte); } +static void check_paca_psize(unsigned long ea, struct mm_struct *mm, + int psize, bool user_region) +{ + if (user_region) { + if (psize != get_paca_psize(ea)) { + get_paca()->context = mm->context; + slb_flush_and_rebolt(); + } + } else if (get_paca()->vmalloc_sllp != + mmu_psize_defs[mmu_vmalloc_psize].sllp) { + get_paca()->vmalloc_sllp = + mmu_psize_defs[mmu_vmalloc_psize].sllp; + slb_vmalloc_update(); + } +} + /* Result code is: * 0 - handled * 1 - normal page fault @@ -1085,6 +1116,8 @@ int hash_page(unsigned long ea, unsigned long access, unsigned long trap) WARN_ON(1); } #endif + check_paca_psize(ea, mm, psize, user_region); + goto bail; } @@ -1125,17 +1158,8 @@ int hash_page(unsigned long ea, unsigned long access, unsigned long trap) #endif } } - if (user_region) { - if (psize != get_paca_psize(ea)) { - get_paca()->context = mm->context; - slb_flush_and_rebolt(); - } - } else if (get_paca()->vmalloc_sllp != - mmu_psize_defs[mmu_vmalloc_psize].sllp) { - get_paca()->vmalloc_sllp = - mmu_psize_defs[mmu_vmalloc_psize].sllp; - slb_vmalloc_update(); - } + + check_paca_psize(ea, mm, psize, user_region); #endif /* CONFIG_PPC_64K_PAGES */ #ifdef CONFIG_PPC_HAS_HASH_64K diff --git a/arch/powerpc/mm/slb.c b/arch/powerpc/mm/slb.c index 964a5f61488a..0399a6702958 100644 --- a/arch/powerpc/mm/slb.c +++ b/arch/powerpc/mm/slb.c @@ -256,10 +256,14 @@ static inline void patch_slb_encoding(unsigned int *insn_addr, patch_instruction(insn_addr, insn); } +extern u32 slb_compare_rr_to_size[]; +extern u32 slb_miss_kernel_load_linear[]; +extern u32 slb_miss_kernel_load_io[]; +extern u32 slb_compare_rr_to_size[]; +extern u32 slb_miss_kernel_load_vmemmap[]; + void slb_set_size(u16 size) { - extern unsigned int *slb_compare_rr_to_size; - if (mmu_slb_size == size) return; @@ -272,11 +276,7 @@ void slb_initialize(void) unsigned long linear_llp, vmalloc_llp, io_llp; unsigned long lflags, vflags; static int slb_encoding_inited; - extern unsigned int *slb_miss_kernel_load_linear; - extern unsigned int *slb_miss_kernel_load_io; - extern unsigned int *slb_compare_rr_to_size; #ifdef CONFIG_SPARSEMEM_VMEMMAP - extern unsigned int *slb_miss_kernel_load_vmemmap; unsigned long vmemmap_llp; #endif diff --git a/arch/powerpc/mm/slb_low.S b/arch/powerpc/mm/slb_low.S index 17aa6dfceb34..736d18b3cefd 100644 --- a/arch/powerpc/mm/slb_low.S +++ b/arch/powerpc/mm/slb_low.S @@ -35,7 +35,7 @@ _GLOBAL(slb_allocate_realmode) * check for bad kernel/user address * (ea & ~REGION_MASK) >= PGTABLE_RANGE */ - rldicr. r9,r3,4,(63 - 46 - 4) + rldicr. r9,r3,4,(63 - PGTABLE_EADDR_SIZE - 4) bne- 8f srdi r9,r3,60 /* get region */ @@ -59,7 +59,8 @@ _GLOBAL(slb_allocate_realmode) /* Linear mapping encoding bits, the "li" instruction below will * be patched by the kernel at boot */ -_GLOBAL(slb_miss_kernel_load_linear) +.globl slb_miss_kernel_load_linear +slb_miss_kernel_load_linear: li r11,0 /* * context = (MAX_USER_CONTEXT) + ((ea >> 60) - 0xc) + 1 @@ -79,7 +80,8 @@ END_MMU_FTR_SECTION_IFCLR(MMU_FTR_1T_SEGMENT) /* Check virtual memmap region. To be patches at kernel boot */ cmpldi cr0,r9,0xf bne 1f -_GLOBAL(slb_miss_kernel_load_vmemmap) +.globl slb_miss_kernel_load_vmemmap +slb_miss_kernel_load_vmemmap: li r11,0 b 6f 1: @@ -95,7 +97,8 @@ _GLOBAL(slb_miss_kernel_load_vmemmap) b 6f 5: /* IO mapping */ - _GLOBAL(slb_miss_kernel_load_io) +.globl slb_miss_kernel_load_io +slb_miss_kernel_load_io: li r11,0 6: /* @@ -250,7 +253,8 @@ slb_finish_load: 7: ld r10,PACASTABRR(r13) addi r10,r10,1 /* This gets soft patched on boot. */ -_GLOBAL(slb_compare_rr_to_size) +.globl slb_compare_rr_to_size +slb_compare_rr_to_size: cmpldi r10,0 blt+ 4f diff --git a/arch/powerpc/mm/tlb_nohash.c b/arch/powerpc/mm/tlb_nohash.c index ae3d5b799b90..92cb18d52ea8 100644 --- a/arch/powerpc/mm/tlb_nohash.c +++ b/arch/powerpc/mm/tlb_nohash.c @@ -596,8 +596,13 @@ static void __early_init_mmu(int boot_cpu) /* XXX This should be decided at runtime based on supported * page sizes in the TLB, but for now let's assume 16M is * always there and a good fit (which it probably is) + * + * Freescale booke only supports 4K pages in TLB0, so use that. */ - mmu_vmemmap_psize = MMU_PAGE_16M; + if (mmu_has_feature(MMU_FTR_TYPE_FSL_E)) + mmu_vmemmap_psize = MMU_PAGE_4K; + else + mmu_vmemmap_psize = MMU_PAGE_16M; /* XXX This code only checks for TLB 0 capabilities and doesn't * check what page size combos are supported by the HW. It diff --git a/arch/powerpc/platforms/44x/Kconfig b/arch/powerpc/platforms/44x/Kconfig index dc1a264ec6e6..4d88f6a19058 100644 --- a/arch/powerpc/platforms/44x/Kconfig +++ b/arch/powerpc/platforms/44x/Kconfig @@ -199,6 +199,34 @@ config CURRITUCK help This option enables support for the IBM Currituck (476fpe) evaluation board +config AKEBONO + bool "IBM Akebono (476gtr) Support" + depends on PPC_47x + default n + select SWIOTLB + select 476FPE + select PPC4xx_PCI_EXPRESS + select PCI_MSI + select PPC4xx_HSTA_MSI + select I2C + select I2C_IBM_IIC + select NETDEVICES + select ETHERNET + select NET_VENDOR_IBM + select IBM_EMAC_EMAC4 + select IBM_EMAC_RGMII_WOL + select USB + select USB_OHCI_HCD_PLATFORM + select USB_EHCI_HCD_PLATFORM + select MMC_SDHCI + select MMC_SDHCI_PLTFM + select MMC_SDHCI_OF_476GTR + select ATA + select SATA_AHCI_PLATFORM + help + This option enables support for the IBM Akebono (476gtr) evaluation board + + config ICON bool "Icon" depends on 44x @@ -323,6 +351,20 @@ config APM821xx select IBM_EMAC_EMAC4 select IBM_EMAC_TAH +config 476FPE_ERR46 + depends on 476FPE + bool "Enable linker work around for PPC476FPE errata #46" + help + This option enables a work around for an icache bug on 476 + that can cause execution of stale instructions when falling + through pages (IBM errata #46). It requires a recent version + of binutils which supports the --ppc476-workaround option. + + The work around enables the appropriate linker options and + ensures that all module output sections are aligned to 4K + page boundaries. The work around is only required when + building modules. + # 44x errata/workaround config symbols, selected by the CPU models above config IBM440EP_ERR42 bool diff --git a/arch/powerpc/platforms/44x/Makefile b/arch/powerpc/platforms/44x/Makefile index d03833abec09..26d35b5941f7 100644 --- a/arch/powerpc/platforms/44x/Makefile +++ b/arch/powerpc/platforms/44x/Makefile @@ -10,4 +10,5 @@ obj-$(CONFIG_XILINX_VIRTEX_5_FXT) += virtex.o obj-$(CONFIG_XILINX_ML510) += virtex_ml510.o obj-$(CONFIG_ISS4xx) += iss4xx.o obj-$(CONFIG_CANYONLANDS)+= canyonlands.o -obj-$(CONFIG_CURRITUCK) += currituck.o +obj-$(CONFIG_CURRITUCK) += ppc476.o +obj-$(CONFIG_AKEBONO) += ppc476.o diff --git a/arch/powerpc/platforms/44x/currituck.c b/arch/powerpc/platforms/44x/ppc476.c index 7f1b71a01c6a..33986c1a05da 100644 --- a/arch/powerpc/platforms/44x/currituck.c +++ b/arch/powerpc/platforms/44x/ppc476.c @@ -1,7 +1,8 @@ /* - * Currituck board specific routines + * PowerPC 476FPE board specific routines * - * Copyright © 2011 Tony Breeds IBM Corporation + * Copyright © 2013 Tony Breeds IBM Corporation + * Copyright © 2013 Alistair Popple IBM Corporation * * Based on earlier code: * Matt Porter <mporter@kernel.crashing.org> @@ -35,8 +36,9 @@ #include <asm/mmu.h> #include <linux/pci.h> +#include <linux/i2c.h> -static __initdata struct of_device_id ppc47x_of_bus[] = { +static struct of_device_id ppc47x_of_bus[] __initdata = { { .compatible = "ibm,plb4", }, { .compatible = "ibm,plb6", }, { .compatible = "ibm,opb", }, @@ -55,15 +57,69 @@ static void quirk_ppc_currituck_usb_fixup(struct pci_dev *dev) } DECLARE_PCI_FIXUP_HEADER(0x1033, 0x0035, quirk_ppc_currituck_usb_fixup); +/* Akebono has an AVR microcontroller attached to the I2C bus + * which is used to power off/reset the system. */ + +/* AVR I2C Commands */ +#define AVR_PWRCTL_CMD (0x26) + +/* Flags for the power control I2C commands */ +#define AVR_PWRCTL_PWROFF (0x01) +#define AVR_PWRCTL_RESET (0x02) + +static struct i2c_client *avr_i2c_client; +static void avr_halt_system(int pwrctl_flags) +{ + /* Request the AVR to reset the system */ + i2c_smbus_write_byte_data(avr_i2c_client, + AVR_PWRCTL_CMD, pwrctl_flags); + + /* Wait for system to be reset */ + while (1) + ; +} + +static void avr_power_off_system(void) +{ + avr_halt_system(AVR_PWRCTL_PWROFF); +} + +static void avr_reset_system(char *cmd) +{ + avr_halt_system(AVR_PWRCTL_RESET); +} + +static int avr_probe(struct i2c_client *client, + const struct i2c_device_id *id) +{ + avr_i2c_client = client; + ppc_md.restart = avr_reset_system; + ppc_md.power_off = avr_power_off_system; + return 0; +} + +static const struct i2c_device_id avr_id[] = { + { "akebono-avr", 0 }, + { } +}; + +static struct i2c_driver avr_driver = { + .driver = { + .name = "akebono-avr", + }, + .probe = avr_probe, + .id_table = avr_id, +}; + static int __init ppc47x_device_probe(void) { + i2c_add_driver(&avr_driver); of_platform_bus_probe(NULL, ppc47x_of_bus, NULL); return 0; } machine_device_initcall(ppc47x, ppc47x_device_probe); -/* We can have either UICs or MPICs */ static void __init ppc47x_init_irq(void) { struct device_node *np; @@ -157,43 +213,36 @@ static void __init ppc47x_setup_arch(void) { /* No need to check the DMA config as we /know/ our windows are all of - * RAM. Lets hope that doesn't change */ + * RAM. Lets hope that doesn't change */ swiotlb_detect_4g(); ppc47x_smp_init(); } -/* - * Called very early, MMU is off, device-tree isn't unflattened - */ -static int __init ppc47x_probe(void) -{ - unsigned long root = of_get_flat_dt_root(); - - if (!of_flat_dt_is_compatible(root, "ibm,currituck")) - return 0; - - return 1; -} - static int board_rev = -1; static int __init ppc47x_get_board_rev(void) { - u8 fpga_reg0; - void *fpga; - struct device_node *np; + int reg; + u8 *fpga; + struct device_node *np = NULL; + + if (of_machine_is_compatible("ibm,currituck")) { + np = of_find_compatible_node(NULL, NULL, "ibm,currituck-fpga"); + reg = 0; + } else if (of_machine_is_compatible("ibm,akebono")) { + np = of_find_compatible_node(NULL, NULL, "ibm,akebono-fpga"); + reg = 2; + } - np = of_find_compatible_node(NULL, NULL, "ibm,currituck-fpga"); if (!np) goto fail; - fpga = of_iomap(np, 0); + fpga = (u8 *) of_iomap(np, 0); of_node_put(np); if (!fpga) goto fail; - fpga_reg0 = ioread8(fpga); - board_rev = fpga_reg0 & 0x03; + board_rev = ioread8(fpga + reg) & 0x03; pr_info("%s: Found board revision %d\n", __func__, board_rev); iounmap(fpga); return 0; @@ -208,7 +257,7 @@ machine_arch_initcall(ppc47x, ppc47x_get_board_rev); static void ppc47x_pci_irq_fixup(struct pci_dev *dev) { if (dev->vendor == 0x1033 && (dev->device == 0x0035 || - dev->device == 0x00e0)) { + dev->device == 0x00e0)) { if (board_rev == 0) { dev->irq = irq_create_mapping(NULL, 47); pr_info("%s: Mapping irq %d\n", __func__, dev->irq); @@ -221,13 +270,30 @@ static void ppc47x_pci_irq_fixup(struct pci_dev *dev) } } +/* + * Called very early, MMU is off, device-tree isn't unflattened + */ +static int __init ppc47x_probe(void) +{ + unsigned long root = of_get_flat_dt_root(); + + if (of_flat_dt_is_compatible(root, "ibm,akebono")) + return 1; + + if (of_flat_dt_is_compatible(root, "ibm,currituck")) { + ppc_md.pci_irq_fixup = ppc47x_pci_irq_fixup; + return 1; + } + + return 0; +} + define_machine(ppc47x) { .name = "PowerPC 47x", .probe = ppc47x_probe, .progress = udbg_progress, .init_IRQ = ppc47x_init_irq, .setup_arch = ppc47x_setup_arch, - .pci_irq_fixup = ppc47x_pci_irq_fixup, .restart = ppc4xx_reset_system, .calibrate_decr = generic_calibrate_decr, }; diff --git a/arch/powerpc/platforms/44x/ppc476_modules.lds b/arch/powerpc/platforms/44x/ppc476_modules.lds new file mode 100644 index 000000000000..9fec5d34ba8e --- /dev/null +++ b/arch/powerpc/platforms/44x/ppc476_modules.lds @@ -0,0 +1,15 @@ +SECTIONS +{ + .text : ALIGN(4096) + { + *(.text .text.* .fixup) + } + .init.text : ALIGN(4096) + { + *(.init.text .init.text.*) + } + .exit.text : ALIGN(4096) + { + *(.exit.text .exit.text.*) + } +} diff --git a/arch/powerpc/platforms/85xx/Kconfig b/arch/powerpc/platforms/85xx/Kconfig index c17aae80e7ff..f442120e0033 100644 --- a/arch/powerpc/platforms/85xx/Kconfig +++ b/arch/powerpc/platforms/85xx/Kconfig @@ -38,6 +38,15 @@ config C293_PCIE help This option enables support for the C293PCIE board +config BSC9132_QDS + bool "Freescale BSC9132QDS" + select DEFAULT_UIMAGE + help + This option enables support for the Freescale BSC9132 QDS board. + BSC9132 is a heterogeneous SoC containing dual e500v2 powerpc cores + and dual StarCore SC3850 DSP cores. + Manufacturer : Freescale Semiconductor, Inc + config MPC8540_ADS bool "Freescale MPC8540 ADS" select DEFAULT_UIMAGE @@ -117,11 +126,11 @@ config P1022_RDK This option enables support for the Freescale / iVeia P1022RDK reference board. -config P1023_RDS - bool "Freescale P1023 RDS/RDB" +config P1023_RDB + bool "Freescale P1023 RDB" select DEFAULT_UIMAGE help - This option enables support for the P1023 RDS and RDB boards + This option enables support for the P1023 RDB board. config TWR_P102x bool "Freescale TWR-P102x" @@ -263,11 +272,11 @@ config CORENET_GENERIC help This option enables support for the FSL CoreNet based boards. For 32bit kernel, the following boards are supported: - P2041 RDB, P3041 DS and P4080 DS + P2041 RDB, P3041 DS, P4080 DS, kmcoge4, and OCA4080 For 64bit kernel, the following boards are supported: T4240 QDS and B4 QDS The following boards are supported for both 32bit and 64bit kernel: - P5020 DS and P5040 DS + P5020 DS, P5040 DS and T104xQDS endif # FSL_SOC_BOOKE diff --git a/arch/powerpc/platforms/85xx/Makefile b/arch/powerpc/platforms/85xx/Makefile index 25cebe74ac46..730326046625 100644 --- a/arch/powerpc/platforms/85xx/Makefile +++ b/arch/powerpc/platforms/85xx/Makefile @@ -6,6 +6,7 @@ obj-$(CONFIG_SMP) += smp.o obj-y += common.o obj-$(CONFIG_BSC9131_RDB) += bsc913x_rdb.o +obj-$(CONFIG_BSC9132_QDS) += bsc913x_qds.o obj-$(CONFIG_C293_PCIE) += c293pcie.o obj-$(CONFIG_MPC8540_ADS) += mpc85xx_ads.o obj-$(CONFIG_MPC8560_ADS) += mpc85xx_ads.o @@ -17,7 +18,7 @@ obj-$(CONFIG_MPC85xx_RDB) += mpc85xx_rdb.o obj-$(CONFIG_P1010_RDB) += p1010rdb.o obj-$(CONFIG_P1022_DS) += p1022_ds.o obj-$(CONFIG_P1022_RDK) += p1022_rdk.o -obj-$(CONFIG_P1023_RDS) += p1023_rds.o +obj-$(CONFIG_P1023_RDB) += p1023_rdb.o obj-$(CONFIG_TWR_P102x) += twr_p102x.o obj-$(CONFIG_CORENET_GENERIC) += corenet_generic.o obj-$(CONFIG_STX_GP3) += stx_gp3.o diff --git a/arch/powerpc/platforms/85xx/bsc913x_qds.c b/arch/powerpc/platforms/85xx/bsc913x_qds.c new file mode 100644 index 000000000000..f0927e58af25 --- /dev/null +++ b/arch/powerpc/platforms/85xx/bsc913x_qds.c @@ -0,0 +1,74 @@ +/* + * BSC913xQDS Board Setup + * + * Author: + * Harninder Rai <harninder.rai@freescale.com> + * Priyanka Jain <Priyanka.Jain@freescale.com> + * + * Copyright 2014 Freescale Semiconductor Inc. + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the + * Free Software Foundation; either version 2 of the License, or (at your + * option) any later version. + */ + +#include <linux/of_platform.h> +#include <linux/pci.h> +#include <asm/mpic.h> +#include <sysdev/fsl_soc.h> +#include <asm/udbg.h> + +#include "mpc85xx.h" +#include "smp.h" + +void __init bsc913x_qds_pic_init(void) +{ + struct mpic *mpic = mpic_alloc(NULL, 0, MPIC_BIG_ENDIAN | + MPIC_SINGLE_DEST_CPU, + 0, 256, " OpenPIC "); + + if (!mpic) + pr_err("bsc913x: Failed to allocate MPIC structure\n"); + else + mpic_init(mpic); +} + +/* + * Setup the architecture + */ +static void __init bsc913x_qds_setup_arch(void) +{ + if (ppc_md.progress) + ppc_md.progress("bsc913x_qds_setup_arch()", 0); + +#if defined(CONFIG_SMP) + mpc85xx_smp_init(); +#endif + + pr_info("bsc913x board from Freescale Semiconductor\n"); +} + +machine_device_initcall(bsc9132_qds, mpc85xx_common_publish_devices); + +/* + * Called very early, device-tree isn't unflattened + */ + +static int __init bsc9132_qds_probe(void) +{ + unsigned long root = of_get_flat_dt_root(); + + return of_flat_dt_is_compatible(root, "fsl,bsc9132qds"); +} + +define_machine(bsc9132_qds) { + .name = "BSC9132 QDS", + .probe = bsc9132_qds_probe, + .setup_arch = bsc913x_qds_setup_arch, + .init_IRQ = bsc913x_qds_pic_init, + .get_irq = mpic_get_irq, + .restart = fsl_rstcr_restart, + .calibrate_decr = generic_calibrate_decr, + .progress = udbg_progress, +}; diff --git a/arch/powerpc/platforms/85xx/corenet_generic.c b/arch/powerpc/platforms/85xx/corenet_generic.c index 8e4b1e1a4911..5db1e117fdde 100644 --- a/arch/powerpc/platforms/85xx/corenet_generic.c +++ b/arch/powerpc/platforms/85xx/corenet_generic.c @@ -67,7 +67,7 @@ void __init corenet_gen_setup_arch(void) swiotlb_detect_4g(); - pr_info("%s board from Freescale Semiconductor\n", ppc_md.name); + pr_info("%s board\n", ppc_md.name); mpc85xx_qe_init(); } @@ -115,6 +115,7 @@ int __init corenet_gen_publish_devices(void) static const char * const boards[] __initconst = { "fsl,P2041RDB", "fsl,P3041DS", + "fsl,OCA4080", "fsl,P4080DS", "fsl,P5020DS", "fsl,P5040DS", @@ -122,12 +123,16 @@ static const char * const boards[] __initconst = { "fsl,B4860QDS", "fsl,B4420QDS", "fsl,B4220QDS", + "fsl,T1040QDS", + "fsl,T1042QDS", + "keymile,kmcoge4", NULL }; static const char * const hv_boards[] __initconst = { "fsl,P2041RDB-hv", "fsl,P3041DS-hv", + "fsl,OCA4080-hv", "fsl,P4080DS-hv", "fsl,P5020DS-hv", "fsl,P5040DS-hv", @@ -135,6 +140,8 @@ static const char * const hv_boards[] __initconst = { "fsl,B4860QDS-hv", "fsl,B4420QDS-hv", "fsl,B4220QDS-hv", + "fsl,T1040QDS-hv", + "fsl,T1042QDS-hv", NULL }; diff --git a/arch/powerpc/platforms/85xx/p1023_rds.c b/arch/powerpc/platforms/85xx/p1023_rdb.c index 0e614007acfb..d5b7509825de 100644 --- a/arch/powerpc/platforms/85xx/p1023_rds.c +++ b/arch/powerpc/platforms/85xx/p1023_rdb.c @@ -4,7 +4,7 @@ * Author: Roy Zang <tie-fei.zang@freescale.com> * * Description: - * P1023 RDS Board Setup + * P1023 RDB Board Setup * * This program is free software; you can redistribute it and/or modify it * under the terms of the GNU General Public License as published by the @@ -41,12 +41,12 @@ * Setup the architecture * */ -static void __init mpc85xx_rds_setup_arch(void) +static void __init mpc85xx_rdb_setup_arch(void) { struct device_node *np; if (ppc_md.progress) - ppc_md.progress("p1023_rds_setup_arch()", 0); + ppc_md.progress("p1023_rdb_setup_arch()", 0); /* Map BCSR area */ np = of_find_node_by_name(NULL, "bcsr"); @@ -85,10 +85,9 @@ static void __init mpc85xx_rds_setup_arch(void) fsl_pci_assign_primary(); } -machine_arch_initcall(p1023_rds, mpc85xx_common_publish_devices); machine_arch_initcall(p1023_rdb, mpc85xx_common_publish_devices); -static void __init mpc85xx_rds_pic_init(void) +static void __init mpc85xx_rdb_pic_init(void) { struct mpic *mpic = mpic_alloc(NULL, 0, MPIC_BIG_ENDIAN | MPIC_SINGLE_DEST_CPU, @@ -99,14 +98,6 @@ static void __init mpc85xx_rds_pic_init(void) mpic_init(mpic); } -static int __init p1023_rds_probe(void) -{ - unsigned long root = of_get_flat_dt_root(); - - return of_flat_dt_is_compatible(root, "fsl,P1023RDS"); - -} - static int __init p1023_rdb_probe(void) { unsigned long root = of_get_flat_dt_root(); @@ -115,26 +106,11 @@ static int __init p1023_rdb_probe(void) } -define_machine(p1023_rds) { - .name = "P1023 RDS", - .probe = p1023_rds_probe, - .setup_arch = mpc85xx_rds_setup_arch, - .init_IRQ = mpc85xx_rds_pic_init, - .get_irq = mpic_get_irq, - .restart = fsl_rstcr_restart, - .calibrate_decr = generic_calibrate_decr, - .progress = udbg_progress, -#ifdef CONFIG_PCI - .pcibios_fixup_bus = fsl_pcibios_fixup_bus, - .pcibios_fixup_phb = fsl_pcibios_fixup_phb, -#endif -}; - define_machine(p1023_rdb) { .name = "P1023 RDB", .probe = p1023_rdb_probe, - .setup_arch = mpc85xx_rds_setup_arch, - .init_IRQ = mpc85xx_rds_pic_init, + .setup_arch = mpc85xx_rdb_setup_arch, + .init_IRQ = mpc85xx_rdb_pic_init, .get_irq = mpic_get_irq, .restart = fsl_rstcr_restart, .calibrate_decr = generic_calibrate_decr, diff --git a/arch/powerpc/platforms/85xx/smp.c b/arch/powerpc/platforms/85xx/smp.c index 6382098d6f8d..ba093f553678 100644 --- a/arch/powerpc/platforms/85xx/smp.c +++ b/arch/powerpc/platforms/85xx/smp.c @@ -27,6 +27,7 @@ #include <asm/cacheflush.h> #include <asm/dbell.h> #include <asm/fsl_guts.h> +#include <asm/code-patching.h> #include <sysdev/fsl_soc.h> #include <sysdev/mpic.h> @@ -267,7 +268,7 @@ out: flush_spin_table(spin_table); out_be32(&spin_table->pir, hw_cpu); out_be64((u64 *)(&spin_table->addr_h), - __pa((u64)*((unsigned long long *)generic_secondary_smp_init))); + __pa(ppc_function_entry(generic_secondary_smp_init))); flush_spin_table(spin_table); #endif diff --git a/arch/powerpc/platforms/Kconfig.cputype b/arch/powerpc/platforms/Kconfig.cputype index d9e2b19b7c8d..43b65ad1970a 100644 --- a/arch/powerpc/platforms/Kconfig.cputype +++ b/arch/powerpc/platforms/Kconfig.cputype @@ -422,6 +422,7 @@ config CPU_BIG_ENDIAN config CPU_LITTLE_ENDIAN bool "Build little endian kernel" + select PPC64_BOOT_WRAPPER help Build a little endian kernel. @@ -430,3 +431,7 @@ config CPU_LITTLE_ENDIAN little endian powerpc. endchoice + +config PPC64_BOOT_WRAPPER + def_bool n + depends on CPU_LITTLE_ENDIAN diff --git a/arch/powerpc/platforms/cell/smp.c b/arch/powerpc/platforms/cell/smp.c index 90745eaa45fe..c8017a7bcabd 100644 --- a/arch/powerpc/platforms/cell/smp.c +++ b/arch/powerpc/platforms/cell/smp.c @@ -40,6 +40,7 @@ #include <asm/firmware.h> #include <asm/rtas.h> #include <asm/cputhreads.h> +#include <asm/code-patching.h> #include "interrupt.h" #include <asm/udbg.h> @@ -70,8 +71,8 @@ static cpumask_t of_spin_map; static inline int smp_startup_cpu(unsigned int lcpu) { int status; - unsigned long start_here = __pa((u32)*((unsigned long *) - generic_secondary_smp_init)); + unsigned long start_here = + __pa(ppc_function_entry(generic_secondary_smp_init)); unsigned int pcpu; int start_cpu; diff --git a/arch/powerpc/platforms/embedded6xx/Kconfig b/arch/powerpc/platforms/embedded6xx/Kconfig index 2a7024d8d8b1..a25f496c2ef9 100644 --- a/arch/powerpc/platforms/embedded6xx/Kconfig +++ b/arch/powerpc/platforms/embedded6xx/Kconfig @@ -65,6 +65,7 @@ config MVME5100 select PPC_INDIRECT_PCI select PPC_I8259 select PPC_NATIVE + select PPC_UDBG_16550 help This option enables support for the Motorola (now Emerson) MVME5100 board. diff --git a/arch/powerpc/platforms/pasemi/powersave.S b/arch/powerpc/platforms/pasemi/powersave.S index 56f45adcd089..81ab555aa491 100644 --- a/arch/powerpc/platforms/pasemi/powersave.S +++ b/arch/powerpc/platforms/pasemi/powersave.S @@ -66,7 +66,7 @@ sleep_common: std r3, 48(r1) /* Only do power savings when in astate 0 */ - bl .check_astate + bl check_astate cmpwi r3,0 bne 1f diff --git a/arch/powerpc/platforms/powernv/Makefile b/arch/powerpc/platforms/powernv/Makefile index 63cebb9b4d45..4ad0d345bc96 100644 --- a/arch/powerpc/platforms/powernv/Makefile +++ b/arch/powerpc/platforms/powernv/Makefile @@ -1,7 +1,7 @@ obj-y += setup.o opal-takeover.o opal-wrappers.o opal.o opal-async.o obj-y += opal-rtc.o opal-nvram.o opal-lpc.o opal-flash.o obj-y += rng.o opal-elog.o opal-dump.o opal-sysparam.o opal-sensor.o -obj-y += opal-msglog.o +obj-y += opal-msglog.o subcore.o subcore-asm.o obj-$(CONFIG_SMP) += smp.o obj-$(CONFIG_PCI) += pci.o pci-p5ioc2.o pci-ioda.o diff --git a/arch/powerpc/platforms/powernv/eeh-ioda.c b/arch/powerpc/platforms/powernv/eeh-ioda.c index 5b51079f3e3b..753f08e36dfa 100644 --- a/arch/powerpc/platforms/powernv/eeh-ioda.c +++ b/arch/powerpc/platforms/powernv/eeh-ioda.c @@ -42,11 +42,19 @@ static int ioda_eeh_event(struct notifier_block *nb, { uint64_t changed_evts = (uint64_t)change; - /* We simply send special EEH event */ - if ((changed_evts & OPAL_EVENT_PCI_ERROR) && - (events & OPAL_EVENT_PCI_ERROR) && - eeh_enabled()) + /* + * We simply send special EEH event if EEH has + * been enabled, or clear pending events in + * case that we enable EEH soon + */ + if (!(changed_evts & OPAL_EVENT_PCI_ERROR) || + !(events & OPAL_EVENT_PCI_ERROR)) + return 0; + + if (eeh_enabled()) eeh_send_failure_event(NULL); + else + opal_notifier_update_evt(OPAL_EVENT_PCI_ERROR, 0x0ul); return 0; } @@ -141,7 +149,9 @@ static int ioda_eeh_post_init(struct pci_controller *hose) } #ifdef CONFIG_DEBUG_FS - if (phb->dbgfs) { + if (!phb->has_dbgfs && phb->dbgfs) { + phb->has_dbgfs = 1; + debugfs_create_file("err_injct_outbound", 0600, phb->dbgfs, hose, &ioda_eeh_outb_dbgfs_ops); @@ -154,7 +164,14 @@ static int ioda_eeh_post_init(struct pci_controller *hose) } #endif - phb->eeh_state |= PNV_EEH_STATE_ENABLED; + /* If EEH is enabled, we're going to rely on that. + * Otherwise, we restore to conventional mechanism + * to clear frozen PE during PCI config access. + */ + if (eeh_enabled()) + phb->flags |= PNV_PHB_FLAG_EEH; + else + phb->flags &= ~PNV_PHB_FLAG_EEH; return 0; } @@ -268,6 +285,21 @@ static int ioda_eeh_get_state(struct eeh_pe *pe) return EEH_STATE_NOT_SUPPORT; } + /* + * If we're in middle of PE reset, return normal + * state to keep EEH core going. For PHB reset, we + * still expect to have fenced PHB cleared with + * PHB reset. + */ + if (!(pe->type & EEH_PE_PHB) && + (pe->state & EEH_PE_RESET)) { + result = (EEH_STATE_MMIO_ACTIVE | + EEH_STATE_DMA_ACTIVE | + EEH_STATE_MMIO_ENABLED | + EEH_STATE_DMA_ENABLED); + return result; + } + /* Retrieve PE status through OPAL */ pe_no = pe->addr; ret = opal_pci_eeh_freeze_status(phb->opal_id, pe_no, @@ -347,52 +379,6 @@ static int ioda_eeh_get_state(struct eeh_pe *pe) return result; } -static int ioda_eeh_pe_clear(struct eeh_pe *pe) -{ - struct pci_controller *hose; - struct pnv_phb *phb; - u32 pe_no; - u8 fstate; - u16 pcierr; - s64 ret; - - pe_no = pe->addr; - hose = pe->phb; - phb = pe->phb->private_data; - - /* Clear the EEH error on the PE */ - ret = opal_pci_eeh_freeze_clear(phb->opal_id, - pe_no, OPAL_EEH_ACTION_CLEAR_FREEZE_ALL); - if (ret) { - pr_err("%s: Failed to clear EEH error for " - "PHB#%x-PE#%x, err=%lld\n", - __func__, hose->global_number, pe_no, ret); - return -EIO; - } - - /* - * Read the PE state back and verify that the frozen - * state has been removed. - */ - ret = opal_pci_eeh_freeze_status(phb->opal_id, pe_no, - &fstate, &pcierr, NULL); - if (ret) { - pr_err("%s: Failed to get EEH status on " - "PHB#%x-PE#%x\n, err=%lld\n", - __func__, hose->global_number, pe_no, ret); - return -EIO; - } - - if (fstate != OPAL_EEH_STOPPED_NOT_FROZEN) { - pr_err("%s: Frozen state not cleared on " - "PHB#%x-PE#%x, sts=%x\n", - __func__, hose->global_number, pe_no, fstate); - return -EIO; - } - - return 0; -} - static s64 ioda_eeh_phb_poll(struct pnv_phb *phb) { s64 rc = OPAL_HARDWARE; @@ -402,13 +388,16 @@ static s64 ioda_eeh_phb_poll(struct pnv_phb *phb) if (rc <= 0) break; - msleep(rc); + if (system_state < SYSTEM_RUNNING) + udelay(1000 * rc); + else + msleep(rc); } return rc; } -static int ioda_eeh_phb_reset(struct pci_controller *hose, int option) +int ioda_eeh_phb_reset(struct pci_controller *hose, int option) { struct pnv_phb *phb = hose->private_data; s64 rc = OPAL_HARDWARE; @@ -431,9 +420,17 @@ static int ioda_eeh_phb_reset(struct pci_controller *hose, int option) /* * Poll state of the PHB until the request is done - * successfully. + * successfully. The PHB reset is usually PHB complete + * reset followed by hot reset on root bus. So we also + * need the PCI bus settlement delay. */ rc = ioda_eeh_phb_poll(phb); + if (option == EEH_RESET_DEACTIVATE) { + if (system_state < SYSTEM_RUNNING) + udelay(1000 * EEH_PE_RST_SETTLE_TIME); + else + msleep(EEH_PE_RST_SETTLE_TIME); + } out: if (rc != OPAL_SUCCESS) return -EIO; @@ -471,6 +468,8 @@ static int ioda_eeh_root_reset(struct pci_controller *hose, int option) /* Poll state of the PHB until the request is done */ rc = ioda_eeh_phb_poll(phb); + if (option == EEH_RESET_DEACTIVATE) + msleep(EEH_PE_RST_SETTLE_TIME); out: if (rc != OPAL_SUCCESS) return -EIO; @@ -478,32 +477,71 @@ out: return 0; } -static int ioda_eeh_bridge_reset(struct pci_controller *hose, - struct pci_dev *dev, int option) +static int ioda_eeh_bridge_reset(struct pci_dev *dev, int option) + { - u16 ctrl; + struct device_node *dn = pci_device_to_OF_node(dev); + struct eeh_dev *edev = of_node_to_eeh_dev(dn); + int aer = edev ? edev->aer_cap : 0; + u32 ctrl; - pr_debug("%s: Reset device %04x:%02x:%02x.%01x with option %d\n", - __func__, hose->global_number, dev->bus->number, - PCI_SLOT(dev->devfn), PCI_FUNC(dev->devfn), option); + pr_debug("%s: Reset PCI bus %04x:%02x with option %d\n", + __func__, pci_domain_nr(dev->bus), + dev->bus->number, option); switch (option) { case EEH_RESET_FUNDAMENTAL: case EEH_RESET_HOT: - pci_read_config_word(dev, PCI_BRIDGE_CONTROL, &ctrl); + /* Don't report linkDown event */ + if (aer) { + eeh_ops->read_config(dn, aer + PCI_ERR_UNCOR_MASK, + 4, &ctrl); + ctrl |= PCI_ERR_UNC_SURPDN; + eeh_ops->write_config(dn, aer + PCI_ERR_UNCOR_MASK, + 4, ctrl); + } + + eeh_ops->read_config(dn, PCI_BRIDGE_CONTROL, 2, &ctrl); ctrl |= PCI_BRIDGE_CTL_BUS_RESET; - pci_write_config_word(dev, PCI_BRIDGE_CONTROL, ctrl); + eeh_ops->write_config(dn, PCI_BRIDGE_CONTROL, 2, ctrl); + msleep(EEH_PE_RST_HOLD_TIME); + break; case EEH_RESET_DEACTIVATE: - pci_read_config_word(dev, PCI_BRIDGE_CONTROL, &ctrl); + eeh_ops->read_config(dn, PCI_BRIDGE_CONTROL, 2, &ctrl); ctrl &= ~PCI_BRIDGE_CTL_BUS_RESET; - pci_write_config_word(dev, PCI_BRIDGE_CONTROL, ctrl); + eeh_ops->write_config(dn, PCI_BRIDGE_CONTROL, 2, ctrl); + msleep(EEH_PE_RST_SETTLE_TIME); + + /* Continue reporting linkDown event */ + if (aer) { + eeh_ops->read_config(dn, aer + PCI_ERR_UNCOR_MASK, + 4, &ctrl); + ctrl &= ~PCI_ERR_UNC_SURPDN; + eeh_ops->write_config(dn, aer + PCI_ERR_UNCOR_MASK, + 4, ctrl); + } + break; } return 0; } +void pnv_pci_reset_secondary_bus(struct pci_dev *dev) +{ + struct pci_controller *hose; + + if (pci_is_root_bus(dev->bus)) { + hose = pci_bus_to_host(dev->bus); + ioda_eeh_root_reset(hose, EEH_RESET_HOT); + ioda_eeh_root_reset(hose, EEH_RESET_DEACTIVATE); + } else { + ioda_eeh_bridge_reset(dev, EEH_RESET_HOT); + ioda_eeh_bridge_reset(dev, EEH_RESET_DEACTIVATE); + } +} + /** * ioda_eeh_reset - Reset the indicated PE * @pe: EEH PE @@ -523,27 +561,18 @@ static int ioda_eeh_reset(struct eeh_pe *pe, int option) int ret; /* - * Anyway, we have to clear the problematic state for the - * corresponding PE. However, we needn't do it if the PE - * is PHB associated. That means the PHB is having fatal - * errors and it needs reset. Further more, the AIB interface - * isn't reliable any more. - */ - if (!(pe->type & EEH_PE_PHB) && - (option == EEH_RESET_HOT || - option == EEH_RESET_FUNDAMENTAL)) { - ret = ioda_eeh_pe_clear(pe); - if (ret) - return -EIO; - } - - /* - * The rules applied to reset, either fundamental or hot reset: + * For PHB reset, we always have complete reset. For those PEs whose + * primary bus derived from root complex (root bus) or root port + * (usually bus#1), we apply hot or fundamental reset on the root port. + * For other PEs, we always have hot reset on the PE primary bus. * - * We always reset the direct upstream bridge of the PE. If the - * direct upstream bridge isn't root bridge, we always take hot - * reset no matter what option (fundamental or hot) is. Otherwise, - * we should do the reset according to the required option. + * Here, we have different design to pHyp, which always clear the + * frozen state during PE reset. However, the good idea here from + * benh is to keep frozen state before we get PE reset done completely + * (until BAR restore). With the frozen state, HW drops illegal IO + * or MMIO access, which can incur recrusive frozen PE during PE + * reset. The side effect is that EEH core has to clear the frozen + * state explicitly after BAR restore. */ if (pe->type & EEH_PE_PHB) { ret = ioda_eeh_phb_reset(hose, option); @@ -553,7 +582,7 @@ static int ioda_eeh_reset(struct eeh_pe *pe, int option) pci_is_root_bus(bus->parent)) ret = ioda_eeh_root_reset(hose, option); else - ret = ioda_eeh_bridge_reset(hose, bus->self, option); + ret = ioda_eeh_bridge_reset(bus->self, option); } return ret; @@ -640,22 +669,6 @@ static void ioda_eeh_hub_diag(struct pci_controller *hose) } } -static int ioda_eeh_get_phb_pe(struct pci_controller *hose, - struct eeh_pe **pe) -{ - struct eeh_pe *phb_pe; - - phb_pe = eeh_phb_pe_get(hose); - if (!phb_pe) { - pr_warning("%s Can't find PE for PHB#%d\n", - __func__, hose->global_number); - return -EEXIST; - } - - *pe = phb_pe; - return 0; -} - static int ioda_eeh_get_pe(struct pci_controller *hose, u16 pe_no, struct eeh_pe **pe) { @@ -663,7 +676,8 @@ static int ioda_eeh_get_pe(struct pci_controller *hose, struct eeh_dev dev; /* Find the PHB PE */ - if (ioda_eeh_get_phb_pe(hose, &phb_pe)) + phb_pe = eeh_phb_pe_get(hose); + if (!phb_pe) return -EEXIST; /* Find the PE according to PE# */ @@ -691,6 +705,7 @@ static int ioda_eeh_next_error(struct eeh_pe **pe) { struct pci_controller *hose; struct pnv_phb *phb; + struct eeh_pe *phb_pe; u64 frozen_pe_no; u16 err_type, severity; long rc; @@ -707,10 +722,12 @@ static int ioda_eeh_next_error(struct eeh_pe **pe) list_for_each_entry(hose, &hose_list, list_node) { /* * If the subordinate PCI buses of the PHB has been - * removed, we needn't take care of it any more. + * removed or is exactly under error recovery, we + * needn't take care of it any more. */ phb = hose->private_data; - if (phb->eeh_state & PNV_EEH_STATE_REMOVED) + phb_pe = eeh_phb_pe_get(hose); + if (!phb_pe || (phb_pe->state & EEH_PE_ISOLATED)) continue; rc = opal_pci_next_error(phb->opal_id, @@ -743,12 +760,6 @@ static int ioda_eeh_next_error(struct eeh_pe **pe) switch (err_type) { case OPAL_EEH_IOC_ERROR: if (severity == OPAL_EEH_SEV_IOC_DEAD) { - list_for_each_entry(hose, &hose_list, - list_node) { - phb = hose->private_data; - phb->eeh_state |= PNV_EEH_STATE_REMOVED; - } - pr_err("EEH: dead IOC detected\n"); ret = EEH_NEXT_ERR_DEAD_IOC; } else if (severity == OPAL_EEH_SEV_INF) { @@ -761,17 +772,12 @@ static int ioda_eeh_next_error(struct eeh_pe **pe) break; case OPAL_EEH_PHB_ERROR: if (severity == OPAL_EEH_SEV_PHB_DEAD) { - if (ioda_eeh_get_phb_pe(hose, pe)) - break; - + *pe = phb_pe; pr_err("EEH: dead PHB#%x detected\n", hose->global_number); - phb->eeh_state |= PNV_EEH_STATE_REMOVED; ret = EEH_NEXT_ERR_DEAD_PHB; } else if (severity == OPAL_EEH_SEV_PHB_FENCED) { - if (ioda_eeh_get_phb_pe(hose, pe)) - break; - + *pe = phb_pe; pr_err("EEH: fenced PHB#%x detected\n", hose->global_number); ret = EEH_NEXT_ERR_FENCED_PHB; @@ -789,17 +795,21 @@ static int ioda_eeh_next_error(struct eeh_pe **pe) * If we can't find the corresponding PE, the * PEEV / PEST would be messy. So we force an * fenced PHB so that it can be recovered. + * + * If the PE has been marked as isolated, that + * should have been removed permanently or in + * progress with recovery. We needn't report + * it again. */ if (ioda_eeh_get_pe(hose, frozen_pe_no, pe)) { - if (!ioda_eeh_get_phb_pe(hose, pe)) { - pr_err("EEH: Escalated fenced PHB#%x " - "detected for PE#%llx\n", - hose->global_number, - frozen_pe_no); - ret = EEH_NEXT_ERR_FENCED_PHB; - } else { - ret = EEH_NEXT_ERR_NONE; - } + *pe = phb_pe; + pr_err("EEH: Escalated fenced PHB#%x " + "detected for PE#%llx\n", + hose->global_number, + frozen_pe_no); + ret = EEH_NEXT_ERR_FENCED_PHB; + } else if ((*pe)->state & EEH_PE_ISOLATED) { + ret = EEH_NEXT_ERR_NONE; } else { pr_err("EEH: Frozen PE#%x on PHB#%x detected\n", (*pe)->addr, (*pe)->phb->global_number); diff --git a/arch/powerpc/platforms/powernv/eeh-powernv.c b/arch/powerpc/platforms/powernv/eeh-powernv.c index a59788e83b8b..56a206f32f77 100644 --- a/arch/powerpc/platforms/powernv/eeh-powernv.c +++ b/arch/powerpc/platforms/powernv/eeh-powernv.c @@ -126,6 +126,7 @@ static int powernv_eeh_dev_probe(struct pci_dev *dev, void *flag) edev->mode &= 0xFFFFFF00; if (dev->hdr_type == PCI_HEADER_TYPE_BRIDGE) edev->mode |= EEH_DEV_BRIDGE; + edev->pcix_cap = pci_find_capability(dev, PCI_CAP_ID_PCIX); if (pci_is_pcie(dev)) { edev->pcie_cap = pci_pcie_cap(dev); @@ -133,6 +134,9 @@ static int powernv_eeh_dev_probe(struct pci_dev *dev, void *flag) edev->mode |= EEH_DEV_ROOT_PORT; else if (pci_pcie_type(dev) == PCI_EXP_TYPE_DOWNSTREAM) edev->mode |= EEH_DEV_DS_PORT; + + edev->aer_cap = pci_find_ext_capability(dev, + PCI_EXT_CAP_ID_ERR); } edev->config_addr = ((dev->bus->number << 8) | dev->devfn); diff --git a/arch/powerpc/platforms/powernv/opal-flash.c b/arch/powerpc/platforms/powernv/opal-flash.c index dc487ff04704..5c21d9c07f45 100644 --- a/arch/powerpc/platforms/powernv/opal-flash.c +++ b/arch/powerpc/platforms/powernv/opal-flash.c @@ -20,6 +20,7 @@ #include <linux/mm.h> #include <linux/vmalloc.h> #include <linux/pagemap.h> +#include <linux/delay.h> #include <asm/opal.h> @@ -130,7 +131,8 @@ static inline void opal_flash_validate(void) { long ret; void *buf = validate_flash_data.buf; - __be32 size, result; + __be32 size = cpu_to_be32(validate_flash_data.buf_size); + __be32 result; ret = opal_validate_flash(__pa(buf), &size, &result); @@ -290,11 +292,6 @@ static int opal_flash_update(int op) /* First entry address */ addr = __pa(list); - pr_alert("FLASH: Image is %u bytes\n", image_data.size); - pr_alert("FLASH: Image update requested\n"); - pr_alert("FLASH: Image will be updated during system reboot\n"); - pr_alert("FLASH: This will take several minutes. Do not power off!\n"); - flash: rc = opal_update_flash(addr); @@ -302,6 +299,47 @@ invalid_img: return rc; } +/* Return CPUs to OPAL before starting FW update */ +static void flash_return_cpu(void *info) +{ + int cpu = smp_processor_id(); + + if (!cpu_online(cpu)) + return; + + /* Disable IRQ */ + hard_irq_disable(); + + /* Return the CPU to OPAL */ + opal_return_cpu(); +} + +/* This gets called just before system reboots */ +void opal_flash_term_callback(void) +{ + struct cpumask mask; + + if (update_flash_data.status != FLASH_IMG_READY) + return; + + pr_alert("FLASH: Flashing new firmware\n"); + pr_alert("FLASH: Image is %u bytes\n", image_data.size); + pr_alert("FLASH: Performing flash and reboot/shutdown\n"); + pr_alert("FLASH: This will take several minutes. Do not power off!\n"); + + /* Small delay to help getting the above message out */ + msleep(500); + + /* Return secondary CPUs to firmware */ + cpumask_copy(&mask, cpu_online_mask); + cpumask_clear_cpu(smp_processor_id(), &mask); + if (!cpumask_empty(&mask)) + smp_call_function_many(&mask, + flash_return_cpu, NULL, false); + /* Hard disable interrupts */ + hard_irq_disable(); +} + /* * Show candidate image status */ diff --git a/arch/powerpc/platforms/powernv/opal-lpc.c b/arch/powerpc/platforms/powernv/opal-lpc.c index 79d83cad3d67..f04b4d8aca5a 100644 --- a/arch/powerpc/platforms/powernv/opal-lpc.c +++ b/arch/powerpc/platforms/powernv/opal-lpc.c @@ -12,12 +12,17 @@ #include <linux/kernel.h> #include <linux/of.h> #include <linux/bug.h> +#include <linux/debugfs.h> +#include <linux/io.h> +#include <linux/slab.h> #include <asm/machdep.h> #include <asm/firmware.h> #include <asm/xics.h> #include <asm/opal.h> #include <asm/prom.h> +#include <asm/uaccess.h> +#include <asm/debug.h> static int opal_lpc_chip_id = -1; @@ -176,6 +181,152 @@ static const struct ppc_pci_io opal_lpc_io = { .outsl = opal_lpc_outsl, }; +#ifdef CONFIG_DEBUG_FS +struct lpc_debugfs_entry { + enum OpalLPCAddressType lpc_type; +}; + +static ssize_t lpc_debug_read(struct file *filp, char __user *ubuf, + size_t count, loff_t *ppos) +{ + struct lpc_debugfs_entry *lpc = filp->private_data; + u32 data, pos, len, todo; + int rc; + + if (!access_ok(VERIFY_WRITE, ubuf, count)) + return -EFAULT; + + todo = count; + while (todo) { + pos = *ppos; + + /* + * Select access size based on count and alignment and + * access type. IO and MEM only support byte acceses, + * FW supports all 3. + */ + len = 1; + if (lpc->lpc_type == OPAL_LPC_FW) { + if (todo > 3 && (pos & 3) == 0) + len = 4; + else if (todo > 1 && (pos & 1) == 0) + len = 2; + } + rc = opal_lpc_read(opal_lpc_chip_id, lpc->lpc_type, pos, + &data, len); + if (rc) + return -ENXIO; + switch(len) { + case 4: + rc = __put_user((u32)data, (u32 __user *)ubuf); + break; + case 2: + rc = __put_user((u16)data, (u16 __user *)ubuf); + break; + default: + rc = __put_user((u8)data, (u8 __user *)ubuf); + break; + } + if (rc) + return -EFAULT; + *ppos += len; + ubuf += len; + todo -= len; + } + + return count; +} + +static ssize_t lpc_debug_write(struct file *filp, const char __user *ubuf, + size_t count, loff_t *ppos) +{ + struct lpc_debugfs_entry *lpc = filp->private_data; + u32 data, pos, len, todo; + int rc; + + if (!access_ok(VERIFY_READ, ubuf, count)) + return -EFAULT; + + todo = count; + while (todo) { + pos = *ppos; + + /* + * Select access size based on count and alignment and + * access type. IO and MEM only support byte acceses, + * FW supports all 3. + */ + len = 1; + if (lpc->lpc_type == OPAL_LPC_FW) { + if (todo > 3 && (pos & 3) == 0) + len = 4; + else if (todo > 1 && (pos & 1) == 0) + len = 2; + } + switch(len) { + case 4: + rc = __get_user(data, (u32 __user *)ubuf); + break; + case 2: + rc = __get_user(data, (u16 __user *)ubuf); + break; + default: + rc = __get_user(data, (u8 __user *)ubuf); + break; + } + if (rc) + return -EFAULT; + + rc = opal_lpc_write(opal_lpc_chip_id, lpc->lpc_type, pos, + data, len); + if (rc) + return -ENXIO; + *ppos += len; + ubuf += len; + todo -= len; + } + + return count; +} + +static const struct file_operations lpc_fops = { + .read = lpc_debug_read, + .write = lpc_debug_write, + .open = simple_open, + .llseek = default_llseek, +}; + +static int opal_lpc_debugfs_create_type(struct dentry *folder, + const char *fname, + enum OpalLPCAddressType type) +{ + struct lpc_debugfs_entry *entry; + entry = kzalloc(sizeof(*entry), GFP_KERNEL); + if (!entry) + return -ENOMEM; + entry->lpc_type = type; + debugfs_create_file(fname, 0600, folder, entry, &lpc_fops); + return 0; +} + +static int opal_lpc_init_debugfs(void) +{ + struct dentry *root; + int rc = 0; + + if (opal_lpc_chip_id < 0) + return -ENODEV; + + root = debugfs_create_dir("lpc", powerpc_debugfs_root); + + rc |= opal_lpc_debugfs_create_type(root, "io", OPAL_LPC_IO); + rc |= opal_lpc_debugfs_create_type(root, "mem", OPAL_LPC_MEM); + rc |= opal_lpc_debugfs_create_type(root, "fw", OPAL_LPC_FW); + return rc; +} +device_initcall(opal_lpc_init_debugfs); +#endif /* CONFIG_DEBUG_FS */ + void opal_lpc_init(void) { struct device_node *np; diff --git a/arch/powerpc/platforms/powernv/opal-memory-errors.c b/arch/powerpc/platforms/powernv/opal-memory-errors.c index ec4132239cdf..b17a34b695ef 100644 --- a/arch/powerpc/platforms/powernv/opal-memory-errors.c +++ b/arch/powerpc/platforms/powernv/opal-memory-errors.c @@ -47,12 +47,12 @@ static void handle_memory_error_event(struct OpalMemoryErrorData *merr_evt) __func__, merr_evt->type); switch (merr_evt->type) { case OPAL_MEM_ERR_TYPE_RESILIENCE: - paddr_start = merr_evt->u.resilience.physical_address_start; - paddr_end = merr_evt->u.resilience.physical_address_end; + paddr_start = be64_to_cpu(merr_evt->u.resilience.physical_address_start); + paddr_end = be64_to_cpu(merr_evt->u.resilience.physical_address_end); break; case OPAL_MEM_ERR_TYPE_DYN_DALLOC: - paddr_start = merr_evt->u.dyn_dealloc.physical_address_start; - paddr_end = merr_evt->u.dyn_dealloc.physical_address_end; + paddr_start = be64_to_cpu(merr_evt->u.dyn_dealloc.physical_address_start); + paddr_end = be64_to_cpu(merr_evt->u.dyn_dealloc.physical_address_end); break; default: return; diff --git a/arch/powerpc/platforms/powernv/opal-takeover.S b/arch/powerpc/platforms/powernv/opal-takeover.S index 3cd262897c27..11a3169ee583 100644 --- a/arch/powerpc/platforms/powernv/opal-takeover.S +++ b/arch/powerpc/platforms/powernv/opal-takeover.S @@ -21,11 +21,13 @@ _GLOBAL(opal_query_takeover) mfcr r0 stw r0,8(r1) + stdu r1,-STACKFRAMESIZE(r1) std r3,STK_PARAM(R3)(r1) std r4,STK_PARAM(R4)(r1) li r3,H_HAL_TAKEOVER li r4,H_HAL_TAKEOVER_QUERY_MAGIC HVSC + addi r1,r1,STACKFRAMESIZE ld r10,STK_PARAM(R3)(r1) std r4,0(r10) ld r10,STK_PARAM(R4)(r1) diff --git a/arch/powerpc/platforms/powernv/opal-wrappers.S b/arch/powerpc/platforms/powernv/opal-wrappers.S index f531ffe35b3e..4abbff22a61f 100644 --- a/arch/powerpc/platforms/powernv/opal-wrappers.S +++ b/arch/powerpc/platforms/powernv/opal-wrappers.S @@ -32,7 +32,7 @@ std r12,PACASAVEDMSR(r13); \ andc r12,r12,r0; \ mtmsrd r12,1; \ - LOAD_REG_ADDR(r0,.opal_return); \ + LOAD_REG_ADDR(r0,opal_return); \ mtlr r0; \ li r0,MSR_DR|MSR_IR|MSR_LE;\ andc r12,r12,r0; \ @@ -44,7 +44,7 @@ mtspr SPRN_HSRR0,r12; \ hrfid -_STATIC(opal_return) +opal_return: /* * Fixup endian on OPAL return... we should be able to simplify * this by instead converting the below trampoline to a set of @@ -124,6 +124,7 @@ OPAL_CALL(opal_xscom_write, OPAL_XSCOM_WRITE); OPAL_CALL(opal_lpc_read, OPAL_LPC_READ); OPAL_CALL(opal_lpc_write, OPAL_LPC_WRITE); OPAL_CALL(opal_return_cpu, OPAL_RETURN_CPU); +OPAL_CALL(opal_reinit_cpus, OPAL_REINIT_CPUS); OPAL_CALL(opal_read_elog, OPAL_ELOG_READ); OPAL_CALL(opal_send_ack_elog, OPAL_ELOG_ACK); OPAL_CALL(opal_get_elog_size, OPAL_ELOG_SIZE); diff --git a/arch/powerpc/platforms/powernv/opal.c b/arch/powerpc/platforms/powernv/opal.c index f343183add07..199975613fe9 100644 --- a/arch/powerpc/platforms/powernv/opal.c +++ b/arch/powerpc/platforms/powernv/opal.c @@ -57,6 +57,21 @@ static DEFINE_SPINLOCK(opal_notifier_lock); static uint64_t last_notified_mask = 0x0ul; static atomic_t opal_notifier_hold = ATOMIC_INIT(0); +static void opal_reinit_cores(void) +{ + /* Do the actual re-init, This will clobber all FPRs, VRs, etc... + * + * It will preserve non volatile GPRs and HSPRG0/1. It will + * also restore HIDs and other SPRs to their original value + * but it might clobber a bunch. + */ +#ifdef __BIG_ENDIAN__ + opal_reinit_cpus(OPAL_REINIT_CPUS_HILE_BE); +#else + opal_reinit_cpus(OPAL_REINIT_CPUS_HILE_LE); +#endif +} + int __init early_init_dt_scan_opal(unsigned long node, const char *uname, int depth, void *data) { @@ -96,6 +111,13 @@ int __init early_init_dt_scan_opal(unsigned long node, printk("OPAL V1 detected !\n"); } + /* Reinit all cores with the right endian */ + opal_reinit_cores(); + + /* Restore some bits */ + if (cur_cpu_spec->cpu_restore) + cur_cpu_spec->cpu_restore(); + return 1; } diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c index 98824aa99173..de19edeaa7a7 100644 --- a/arch/powerpc/platforms/powernv/pci-ioda.c +++ b/arch/powerpc/platforms/powernv/pci-ioda.c @@ -13,6 +13,7 @@ #include <linux/kernel.h> #include <linux/pci.h> +#include <linux/crash_dump.h> #include <linux/debugfs.h> #include <linux/delay.h> #include <linux/string.h> @@ -663,15 +664,15 @@ static void pnv_pci_ioda_setup_dma_pe(struct pnv_phb *phb, * errors, and on the first pass the data will be a relative * bus number, print that out instead. */ - tbl->it_busno = 0; pe->tce_inval_reg_phys = be64_to_cpup(swinvp); tbl->it_index = (unsigned long)ioremap(pe->tce_inval_reg_phys, 8); - tbl->it_type = TCE_PCI_SWINV_CREATE | TCE_PCI_SWINV_FREE | - TCE_PCI_SWINV_PAIR; + tbl->it_type |= (TCE_PCI_SWINV_CREATE | + TCE_PCI_SWINV_FREE | + TCE_PCI_SWINV_PAIR); } iommu_init_table(tbl, phb->hose->node); - iommu_register_group(tbl, pci_domain_nr(pe->pbus), pe->pe_number); + iommu_register_group(tbl, phb->hose->global_number, pe->pe_number); if (pe->pdev) set_iommu_table_base_and_group(&pe->pdev->dev, tbl); @@ -793,14 +794,13 @@ static void pnv_pci_ioda2_setup_dma_pe(struct pnv_phb *phb, * errors, and on the first pass the data will be a relative * bus number, print that out instead. */ - tbl->it_busno = 0; pe->tce_inval_reg_phys = be64_to_cpup(swinvp); tbl->it_index = (unsigned long)ioremap(pe->tce_inval_reg_phys, 8); - tbl->it_type = TCE_PCI_SWINV_CREATE | TCE_PCI_SWINV_FREE; + tbl->it_type |= (TCE_PCI_SWINV_CREATE | TCE_PCI_SWINV_FREE); } iommu_init_table(tbl, phb->hose->node); - iommu_register_group(tbl, pci_domain_nr(pe->pbus), pe->pe_number); + iommu_register_group(tbl, phb->hose->global_number, pe->pe_number); if (pe->pdev) set_iommu_table_base_and_group(&pe->pdev->dev, tbl); @@ -1386,12 +1386,24 @@ void __init pnv_pci_init_ioda_phb(struct device_node *np, ppc_md.pcibios_fixup = pnv_pci_ioda_fixup; ppc_md.pcibios_enable_device_hook = pnv_pci_enable_device_hook; ppc_md.pcibios_window_alignment = pnv_pci_window_alignment; + ppc_md.pcibios_reset_secondary_bus = pnv_pci_reset_secondary_bus; pci_add_flags(PCI_REASSIGN_ALL_RSRC); /* Reset IODA tables to a clean state */ rc = opal_pci_reset(phb_id, OPAL_PCI_IODA_TABLE_RESET, OPAL_ASSERT_RESET); if (rc) pr_warning(" OPAL Error %ld performing IODA table reset !\n", rc); + + /* If we're running in kdump kerenl, the previous kerenl never + * shutdown PCI devices correctly. We already got IODA table + * cleaned out. So we have to issue PHB reset to stop all PCI + * transactions from previous kerenl. + */ + if (is_kdump_kernel()) { + pr_info(" Issue PHB reset ...\n"); + ioda_eeh_phb_reset(hose, EEH_RESET_FUNDAMENTAL); + ioda_eeh_phb_reset(hose, OPAL_DEASSERT_RESET); + } } void __init pnv_pci_init_ioda2_phb(struct device_node *np) diff --git a/arch/powerpc/platforms/powernv/pci.c b/arch/powerpc/platforms/powernv/pci.c index 8518817dcdfd..eefbfcc3fd8c 100644 --- a/arch/powerpc/platforms/powernv/pci.c +++ b/arch/powerpc/platforms/powernv/pci.c @@ -131,65 +131,60 @@ static void pnv_pci_dump_p7ioc_diag_data(struct pci_controller *hose, int i; data = (struct OpalIoP7IOCPhbErrorData *)common; - pr_info("P7IOC PHB#%d Diag-data (Version: %d)\n\n", + pr_info("P7IOC PHB#%d Diag-data (Version: %d)\n", hose->global_number, common->version); if (data->brdgCtl) - pr_info(" brdgCtl: %08x\n", + pr_info("brdgCtl: %08x\n", data->brdgCtl); if (data->portStatusReg || data->rootCmplxStatus || data->busAgentStatus) - pr_info(" UtlSts: %08x %08x %08x\n", + pr_info("UtlSts: %08x %08x %08x\n", data->portStatusReg, data->rootCmplxStatus, data->busAgentStatus); if (data->deviceStatus || data->slotStatus || data->linkStatus || data->devCmdStatus || data->devSecStatus) - pr_info(" RootSts: %08x %08x %08x %08x %08x\n", + pr_info("RootSts: %08x %08x %08x %08x %08x\n", data->deviceStatus, data->slotStatus, data->linkStatus, data->devCmdStatus, data->devSecStatus); if (data->rootErrorStatus || data->uncorrErrorStatus || data->corrErrorStatus) - pr_info(" RootErrSts: %08x %08x %08x\n", + pr_info("RootErrSts: %08x %08x %08x\n", data->rootErrorStatus, data->uncorrErrorStatus, data->corrErrorStatus); if (data->tlpHdr1 || data->tlpHdr2 || data->tlpHdr3 || data->tlpHdr4) - pr_info(" RootErrLog: %08x %08x %08x %08x\n", + pr_info("RootErrLog: %08x %08x %08x %08x\n", data->tlpHdr1, data->tlpHdr2, data->tlpHdr3, data->tlpHdr4); if (data->sourceId || data->errorClass || data->correlator) - pr_info(" RootErrLog1: %08x %016llx %016llx\n", + pr_info("RootErrLog1: %08x %016llx %016llx\n", data->sourceId, data->errorClass, data->correlator); if (data->p7iocPlssr || data->p7iocCsr) - pr_info(" PhbSts: %016llx %016llx\n", + pr_info("PhbSts: %016llx %016llx\n", data->p7iocPlssr, data->p7iocCsr); - if (data->lemFir || data->lemErrorMask || - data->lemWOF) - pr_info(" Lem: %016llx %016llx %016llx\n", + if (data->lemFir) + pr_info("Lem: %016llx %016llx %016llx\n", data->lemFir, data->lemErrorMask, data->lemWOF); - if (data->phbErrorStatus || data->phbFirstErrorStatus || - data->phbErrorLog0 || data->phbErrorLog1) - pr_info(" PhbErr: %016llx %016llx %016llx %016llx\n", + if (data->phbErrorStatus) + pr_info("PhbErr: %016llx %016llx %016llx %016llx\n", data->phbErrorStatus, data->phbFirstErrorStatus, data->phbErrorLog0, data->phbErrorLog1); - if (data->mmioErrorStatus || data->mmioFirstErrorStatus || - data->mmioErrorLog0 || data->mmioErrorLog1) - pr_info(" OutErr: %016llx %016llx %016llx %016llx\n", + if (data->mmioErrorStatus) + pr_info("OutErr: %016llx %016llx %016llx %016llx\n", data->mmioErrorStatus, data->mmioFirstErrorStatus, data->mmioErrorLog0, data->mmioErrorLog1); - if (data->dma0ErrorStatus || data->dma0FirstErrorStatus || - data->dma0ErrorLog0 || data->dma0ErrorLog1) - pr_info(" InAErr: %016llx %016llx %016llx %016llx\n", + if (data->dma0ErrorStatus) + pr_info("InAErr: %016llx %016llx %016llx %016llx\n", data->dma0ErrorStatus, data->dma0FirstErrorStatus, data->dma0ErrorLog0, data->dma0ErrorLog1); - if (data->dma1ErrorStatus || data->dma1FirstErrorStatus || - data->dma1ErrorLog0 || data->dma1ErrorLog1) - pr_info(" InBErr: %016llx %016llx %016llx %016llx\n", + if (data->dma1ErrorStatus) + pr_info("InBErr: %016llx %016llx %016llx %016llx\n", data->dma1ErrorStatus, data->dma1FirstErrorStatus, data->dma1ErrorLog0, data->dma1ErrorLog1); @@ -198,7 +193,7 @@ static void pnv_pci_dump_p7ioc_diag_data(struct pci_controller *hose, (data->pestB[i] >> 63) == 0) continue; - pr_info(" PE[%3d] A/B: %016llx %016llx\n", + pr_info("PE[%3d] A/B: %016llx %016llx\n", i, data->pestA[i], data->pestB[i]); } } @@ -210,69 +205,63 @@ static void pnv_pci_dump_phb3_diag_data(struct pci_controller *hose, int i; data = (struct OpalIoPhb3ErrorData*)common; - pr_info("PHB3 PHB#%d Diag-data (Version: %d)\n\n", + pr_info("PHB3 PHB#%d Diag-data (Version: %d)\n", hose->global_number, common->version); if (data->brdgCtl) - pr_info(" brdgCtl: %08x\n", + pr_info("brdgCtl: %08x\n", data->brdgCtl); if (data->portStatusReg || data->rootCmplxStatus || data->busAgentStatus) - pr_info(" UtlSts: %08x %08x %08x\n", + pr_info("UtlSts: %08x %08x %08x\n", data->portStatusReg, data->rootCmplxStatus, data->busAgentStatus); if (data->deviceStatus || data->slotStatus || data->linkStatus || data->devCmdStatus || data->devSecStatus) - pr_info(" RootSts: %08x %08x %08x %08x %08x\n", + pr_info("RootSts: %08x %08x %08x %08x %08x\n", data->deviceStatus, data->slotStatus, data->linkStatus, data->devCmdStatus, data->devSecStatus); if (data->rootErrorStatus || data->uncorrErrorStatus || data->corrErrorStatus) - pr_info(" RootErrSts: %08x %08x %08x\n", + pr_info("RootErrSts: %08x %08x %08x\n", data->rootErrorStatus, data->uncorrErrorStatus, data->corrErrorStatus); if (data->tlpHdr1 || data->tlpHdr2 || data->tlpHdr3 || data->tlpHdr4) - pr_info(" RootErrLog: %08x %08x %08x %08x\n", + pr_info("RootErrLog: %08x %08x %08x %08x\n", data->tlpHdr1, data->tlpHdr2, data->tlpHdr3, data->tlpHdr4); if (data->sourceId || data->errorClass || data->correlator) - pr_info(" RootErrLog1: %08x %016llx %016llx\n", + pr_info("RootErrLog1: %08x %016llx %016llx\n", data->sourceId, data->errorClass, data->correlator); - if (data->nFir || data->nFirMask || - data->nFirWOF) - pr_info(" nFir: %016llx %016llx %016llx\n", + if (data->nFir) + pr_info("nFir: %016llx %016llx %016llx\n", data->nFir, data->nFirMask, data->nFirWOF); if (data->phbPlssr || data->phbCsr) - pr_info(" PhbSts: %016llx %016llx\n", + pr_info("PhbSts: %016llx %016llx\n", data->phbPlssr, data->phbCsr); - if (data->lemFir || data->lemErrorMask || - data->lemWOF) - pr_info(" Lem: %016llx %016llx %016llx\n", + if (data->lemFir) + pr_info("Lem: %016llx %016llx %016llx\n", data->lemFir, data->lemErrorMask, data->lemWOF); - if (data->phbErrorStatus || data->phbFirstErrorStatus || - data->phbErrorLog0 || data->phbErrorLog1) - pr_info(" PhbErr: %016llx %016llx %016llx %016llx\n", + if (data->phbErrorStatus) + pr_info("PhbErr: %016llx %016llx %016llx %016llx\n", data->phbErrorStatus, data->phbFirstErrorStatus, data->phbErrorLog0, data->phbErrorLog1); - if (data->mmioErrorStatus || data->mmioFirstErrorStatus || - data->mmioErrorLog0 || data->mmioErrorLog1) - pr_info(" OutErr: %016llx %016llx %016llx %016llx\n", + if (data->mmioErrorStatus) + pr_info("OutErr: %016llx %016llx %016llx %016llx\n", data->mmioErrorStatus, data->mmioFirstErrorStatus, data->mmioErrorLog0, data->mmioErrorLog1); - if (data->dma0ErrorStatus || data->dma0FirstErrorStatus || - data->dma0ErrorLog0 || data->dma0ErrorLog1) - pr_info(" InAErr: %016llx %016llx %016llx %016llx\n", + if (data->dma0ErrorStatus) + pr_info("InAErr: %016llx %016llx %016llx %016llx\n", data->dma0ErrorStatus, data->dma0FirstErrorStatus, data->dma0ErrorLog0, data->dma0ErrorLog1); - if (data->dma1ErrorStatus || data->dma1FirstErrorStatus || - data->dma1ErrorLog0 || data->dma1ErrorLog1) - pr_info(" InBErr: %016llx %016llx %016llx %016llx\n", + if (data->dma1ErrorStatus) + pr_info("InBErr: %016llx %016llx %016llx %016llx\n", data->dma1ErrorStatus, data->dma1FirstErrorStatus, data->dma1ErrorLog0, data->dma1ErrorLog1); @@ -281,7 +270,7 @@ static void pnv_pci_dump_phb3_diag_data(struct pci_controller *hose, (data->pestB[i] >> 63) == 0) continue; - pr_info(" PE[%3d] A/B: %016llx %016llx\n", + pr_info("PE[%3d] A/B: %016llx %016llx\n", i, data->pestA[i], data->pestB[i]); } } @@ -384,9 +373,6 @@ int pnv_pci_cfg_read(struct device_node *dn, struct pci_dn *pdn = PCI_DN(dn); struct pnv_phb *phb = pdn->phb->private_data; u32 bdfn = (pdn->busno << 8) | pdn->devfn; -#ifdef CONFIG_EEH - struct eeh_pe *phb_pe = NULL; -#endif s64 rc; switch (size) { @@ -412,31 +398,9 @@ int pnv_pci_cfg_read(struct device_node *dn, default: return PCIBIOS_FUNC_NOT_SUPPORTED; } + cfg_dbg("%s: bus: %x devfn: %x +%x/%x -> %08x\n", __func__, pdn->busno, pdn->devfn, where, size, *val); - - /* - * Check if the specified PE has been put into frozen - * state. On the other hand, we needn't do that while - * the PHB has been put into frozen state because of - * PHB-fatal errors. - */ -#ifdef CONFIG_EEH - phb_pe = eeh_phb_pe_get(pdn->phb); - if (phb_pe && (phb_pe->state & EEH_PE_ISOLATED)) - return PCIBIOS_SUCCESSFUL; - - if (phb->eeh_state & PNV_EEH_STATE_ENABLED) { - if (*val == EEH_IO_ERROR_VALUE(size) && - eeh_dev_check_failure(of_node_to_eeh_dev(dn))) - return PCIBIOS_DEVICE_NOT_FOUND; - } else { - pnv_pci_config_check_eeh(phb, dn); - } -#else - pnv_pci_config_check_eeh(phb, dn); -#endif - return PCIBIOS_SUCCESSFUL; } @@ -463,33 +427,74 @@ int pnv_pci_cfg_write(struct device_node *dn, return PCIBIOS_FUNC_NOT_SUPPORTED; } - /* Check if the PHB got frozen due to an error (no response) */ -#ifdef CONFIG_EEH - if (!(phb->eeh_state & PNV_EEH_STATE_ENABLED)) - pnv_pci_config_check_eeh(phb, dn); -#else - pnv_pci_config_check_eeh(phb, dn); -#endif - return PCIBIOS_SUCCESSFUL; } +#if CONFIG_EEH +static bool pnv_pci_cfg_check(struct pci_controller *hose, + struct device_node *dn) +{ + struct eeh_dev *edev = NULL; + struct pnv_phb *phb = hose->private_data; + + /* EEH not enabled ? */ + if (!(phb->flags & PNV_PHB_FLAG_EEH)) + return true; + + /* PE reset or device removed ? */ + edev = of_node_to_eeh_dev(dn); + if (edev) { + if (edev->pe && + (edev->pe->state & EEH_PE_RESET)) + return false; + + if (edev->mode & EEH_DEV_REMOVED) + return false; + } + + return true; +} +#else +static inline pnv_pci_cfg_check(struct pci_controller *hose, + struct device_node *dn) +{ + return true; +} +#endif /* CONFIG_EEH */ + static int pnv_pci_read_config(struct pci_bus *bus, unsigned int devfn, int where, int size, u32 *val) { struct device_node *dn, *busdn = pci_bus_to_OF_node(bus); struct pci_dn *pdn; + struct pnv_phb *phb; + bool found = false; + int ret; + *val = 0xFFFFFFFF; for (dn = busdn->child; dn; dn = dn->sibling) { pdn = PCI_DN(dn); - if (pdn && pdn->devfn == devfn) - return pnv_pci_cfg_read(dn, where, size, val); + if (pdn && pdn->devfn == devfn) { + phb = pdn->phb->private_data; + found = true; + break; + } } - *val = 0xFFFFFFFF; - return PCIBIOS_DEVICE_NOT_FOUND; + if (!found || !pnv_pci_cfg_check(pdn->phb, dn)) + return PCIBIOS_DEVICE_NOT_FOUND; + ret = pnv_pci_cfg_read(dn, where, size, val); + if (phb->flags & PNV_PHB_FLAG_EEH) { + if (*val == EEH_IO_ERROR_VALUE(size) && + eeh_dev_check_failure(of_node_to_eeh_dev(dn))) + return PCIBIOS_DEVICE_NOT_FOUND; + } else { + pnv_pci_config_check_eeh(phb, dn); + } + + return ret; } static int pnv_pci_write_config(struct pci_bus *bus, @@ -498,14 +503,27 @@ static int pnv_pci_write_config(struct pci_bus *bus, { struct device_node *dn, *busdn = pci_bus_to_OF_node(bus); struct pci_dn *pdn; + struct pnv_phb *phb; + bool found = false; + int ret; for (dn = busdn->child; dn; dn = dn->sibling) { pdn = PCI_DN(dn); - if (pdn && pdn->devfn == devfn) - return pnv_pci_cfg_write(dn, where, size, val); + if (pdn && pdn->devfn == devfn) { + phb = pdn->phb->private_data; + found = true; + break; + } } - return PCIBIOS_DEVICE_NOT_FOUND; + if (!found || !pnv_pci_cfg_check(pdn->phb, dn)) + return PCIBIOS_DEVICE_NOT_FOUND; + + ret = pnv_pci_cfg_write(dn, where, size, val); + if (!(phb->flags & PNV_PHB_FLAG_EEH)) + pnv_pci_config_check_eeh(phb, dn); + + return ret; } struct pci_ops pnv_pci_ops = { diff --git a/arch/powerpc/platforms/powernv/pci.h b/arch/powerpc/platforms/powernv/pci.h index cde169442775..676232c34328 100644 --- a/arch/powerpc/platforms/powernv/pci.h +++ b/arch/powerpc/platforms/powernv/pci.h @@ -81,28 +81,27 @@ struct pnv_eeh_ops { int (*configure_bridge)(struct eeh_pe *pe); int (*next_error)(struct eeh_pe **pe); }; - -#define PNV_EEH_STATE_ENABLED (1 << 0) /* EEH enabled */ -#define PNV_EEH_STATE_REMOVED (1 << 1) /* PHB removed */ - #endif /* CONFIG_EEH */ +#define PNV_PHB_FLAG_EEH (1 << 0) + struct pnv_phb { struct pci_controller *hose; enum pnv_phb_type type; enum pnv_phb_model model; u64 hub_id; u64 opal_id; + int flags; void __iomem *regs; int initialized; spinlock_t lock; #ifdef CONFIG_EEH struct pnv_eeh_ops *eeh_ops; - int eeh_state; #endif #ifdef CONFIG_DEBUG_FS + int has_dbgfs; struct dentry *dbgfs; #endif @@ -205,5 +204,7 @@ extern void pnv_pci_init_ioda_hub(struct device_node *np); extern void pnv_pci_init_ioda2_phb(struct device_node *np); extern void pnv_pci_ioda_tce_invalidate(struct iommu_table *tbl, __be64 *startp, __be64 *endp, bool rm); +extern void pnv_pci_reset_secondary_bus(struct pci_dev *dev); +extern int ioda_eeh_phb_reset(struct pci_controller *hose, int option); #endif /* __POWERNV_PCI_H */ diff --git a/arch/powerpc/platforms/powernv/powernv.h b/arch/powerpc/platforms/powernv/powernv.h index 0051e108ef0f..75501bfede7f 100644 --- a/arch/powerpc/platforms/powernv/powernv.h +++ b/arch/powerpc/platforms/powernv/powernv.h @@ -25,4 +25,6 @@ static inline int pnv_pci_dma_set_mask(struct pci_dev *pdev, u64 dma_mask) extern void pnv_lpc_init(void); +bool cpu_core_split_required(void); + #endif /* _POWERNV_H */ diff --git a/arch/powerpc/platforms/powernv/setup.c b/arch/powerpc/platforms/powernv/setup.c index 8723d32632f5..8c16a5f96728 100644 --- a/arch/powerpc/platforms/powernv/setup.c +++ b/arch/powerpc/platforms/powernv/setup.c @@ -27,6 +27,7 @@ #include <linux/interrupt.h> #include <linux/bug.h> #include <linux/pci.h> +#include <linux/cpufreq.h> #include <asm/machdep.h> #include <asm/firmware.h> @@ -98,11 +99,32 @@ static void pnv_show_cpuinfo(struct seq_file *m) of_node_put(root); } +static void pnv_prepare_going_down(void) +{ + /* + * Disable all notifiers from OPAL, we can't + * service interrupts anymore anyway + */ + opal_notifier_disable(); + + /* Soft disable interrupts */ + local_irq_disable(); + + /* + * Return secondary CPUs to firwmare if a flash update + * is pending otherwise we will get all sort of error + * messages about CPU being stuck etc.. This will also + * have the side effect of hard disabling interrupts so + * past this point, the kernel is effectively dead. + */ + opal_flash_term_callback(); +} + static void __noreturn pnv_restart(char *cmd) { long rc = OPAL_BUSY; - opal_notifier_disable(); + pnv_prepare_going_down(); while (rc == OPAL_BUSY || rc == OPAL_BUSY_EVENT) { rc = opal_cec_reboot(); @@ -119,7 +141,7 @@ static void __noreturn pnv_power_off(void) { long rc = OPAL_BUSY; - opal_notifier_disable(); + pnv_prepare_going_down(); while (rc == OPAL_BUSY || rc == OPAL_BUSY_EVENT) { rc = opal_cec_power_down(0); @@ -222,6 +244,13 @@ static void pnv_kexec_cpu_down(int crash_shutdown, int secondary) } #endif /* CONFIG_KEXEC */ +#ifdef CONFIG_MEMORY_HOTPLUG_SPARSE +static unsigned long pnv_memory_block_size(void) +{ + return 256UL * 1024 * 1024; +} +#endif + static void __init pnv_setup_machdep_opal(void) { ppc_md.get_boot_time = opal_get_boot_time; @@ -269,6 +298,25 @@ static int __init pnv_probe(void) return 1; } +/* + * Returns the cpu frequency for 'cpu' in Hz. This is used by + * /proc/cpuinfo + */ +unsigned long pnv_get_proc_freq(unsigned int cpu) +{ + unsigned long ret_freq; + + ret_freq = cpufreq_quick_get(cpu) * 1000ul; + + /* + * If the backend cpufreq driver does not exist, + * then fallback to old way of reporting the clockrate. + */ + if (!ret_freq) + ret_freq = ppc_proc_freq; + return ret_freq; +} + define_machine(powernv) { .name = "PowerNV", .probe = pnv_probe, @@ -276,6 +324,7 @@ define_machine(powernv) { .setup_arch = pnv_setup_arch, .init_IRQ = pnv_init_IRQ, .show_cpuinfo = pnv_show_cpuinfo, + .get_proc_freq = pnv_get_proc_freq, .progress = pnv_progress, .machine_shutdown = pnv_shutdown, .power_save = power7_idle, @@ -284,4 +333,7 @@ define_machine(powernv) { #ifdef CONFIG_KEXEC .kexec_cpu_down = pnv_kexec_cpu_down, #endif +#ifdef CONFIG_MEMORY_HOTPLUG_SPARSE + .memory_block_size = pnv_memory_block_size, +#endif }; diff --git a/arch/powerpc/platforms/powernv/smp.c b/arch/powerpc/platforms/powernv/smp.c index bf5fcd452168..0062a43a2e0d 100644 --- a/arch/powerpc/platforms/powernv/smp.c +++ b/arch/powerpc/platforms/powernv/smp.c @@ -31,6 +31,7 @@ #include <asm/xics.h> #include <asm/opal.h> #include <asm/runlatch.h> +#include <asm/code-patching.h> #include "powernv.h" @@ -50,8 +51,8 @@ static void pnv_smp_setup_cpu(int cpu) int pnv_smp_kick_cpu(int nr) { unsigned int pcpu = get_hard_smp_processor_id(nr); - unsigned long start_here = __pa(*((unsigned long *) - generic_secondary_smp_init)); + unsigned long start_here = + __pa(ppc_function_entry(generic_secondary_smp_init)); long rc; BUG_ON(nr < 0 || nr >= NR_CPUS); @@ -158,17 +159,19 @@ static void pnv_smp_cpu_kill_self(void) mtspr(SPRN_LPCR, mfspr(SPRN_LPCR) & ~(u64)LPCR_PECE1); while (!generic_check_cpu_restart(cpu)) { ppc64_runlatch_off(); - power7_nap(); + power7_nap(1); ppc64_runlatch_on(); - if (!generic_check_cpu_restart(cpu)) { + + /* Reenable IRQs briefly to clear the IPI that woke us */ + local_irq_enable(); + local_irq_disable(); + mb(); + + if (cpu_core_split_required()) + continue; + + if (!generic_check_cpu_restart(cpu)) DBG("CPU%d Unexpected exit while offline !\n", cpu); - /* We may be getting an IPI, so we re-enable - * interrupts to process it, it will be ignored - * since we aren't online (hopefully) - */ - local_irq_enable(); - local_irq_disable(); - } } mtspr(SPRN_LPCR, mfspr(SPRN_LPCR) | LPCR_PECE1); DBG("CPU%d coming online...\n", cpu); diff --git a/arch/powerpc/platforms/powernv/subcore-asm.S b/arch/powerpc/platforms/powernv/subcore-asm.S new file mode 100644 index 000000000000..39bb24aa8f34 --- /dev/null +++ b/arch/powerpc/platforms/powernv/subcore-asm.S @@ -0,0 +1,95 @@ +/* + * Copyright 2013, Michael (Ellerman|Neuling), IBM Corporation. + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License + * as published by the Free Software Foundation; either version + * 2 of the License, or (at your option) any later version. + */ + +#include <asm/asm-offsets.h> +#include <asm/ppc_asm.h> +#include <asm/reg.h> + +#include "subcore.h" + + +_GLOBAL(split_core_secondary_loop) + /* + * r3 = u8 *state, used throughout the routine + * r4 = temp + * r5 = temp + * .. + * r12 = MSR + */ + mfmsr r12 + + /* Disable interrupts so SRR0/1 don't get trashed */ + li r4,0 + ori r4,r4,MSR_EE|MSR_SE|MSR_BE|MSR_RI + andc r4,r12,r4 + sync + mtmsrd r4 + + /* Switch to real mode and leave interrupts off */ + li r5, MSR_IR|MSR_DR + andc r5, r4, r5 + + LOAD_REG_ADDR(r4, real_mode) + + mtspr SPRN_SRR0,r4 + mtspr SPRN_SRR1,r5 + rfid + b . /* prevent speculative execution */ + +real_mode: + /* Grab values from unsplit SPRs */ + mfspr r6, SPRN_LDBAR + mfspr r7, SPRN_PMMAR + mfspr r8, SPRN_PMCR + mfspr r9, SPRN_RPR + mfspr r10, SPRN_SDR1 + + /* Order reading the SPRs vs telling the primary we are ready to split */ + sync + + /* Tell thread 0 we are in real mode */ + li r4, SYNC_STEP_REAL_MODE + stb r4, 0(r3) + + li r5, (HID0_POWER8_4LPARMODE | HID0_POWER8_2LPARMODE)@highest + sldi r5, r5, 48 + + /* Loop until we see the split happen in HID0 */ +1: mfspr r4, SPRN_HID0 + and. r4, r4, r5 + beq 1b + + /* + * We only need to initialise the below regs once for each subcore, + * but it's simpler and harmless to do it on each thread. + */ + + /* Make sure various SPRS have sane values */ + li r4, 0 + mtspr SPRN_LPID, r4 + mtspr SPRN_PCR, r4 + mtspr SPRN_HDEC, r4 + + /* Restore SPR values now we are split */ + mtspr SPRN_LDBAR, r6 + mtspr SPRN_PMMAR, r7 + mtspr SPRN_PMCR, r8 + mtspr SPRN_RPR, r9 + mtspr SPRN_SDR1, r10 + + LOAD_REG_ADDR(r5, virtual_mode) + + /* Get out of real mode */ + mtspr SPRN_SRR0,r5 + mtspr SPRN_SRR1,r12 + rfid + b . /* prevent speculative execution */ + +virtual_mode: + blr diff --git a/arch/powerpc/platforms/powernv/subcore.c b/arch/powerpc/platforms/powernv/subcore.c new file mode 100644 index 000000000000..894ecb3eb596 --- /dev/null +++ b/arch/powerpc/platforms/powernv/subcore.c @@ -0,0 +1,392 @@ +/* + * Copyright 2013, Michael (Ellerman|Neuling), IBM Corporation. + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License + * as published by the Free Software Foundation; either version + * 2 of the License, or (at your option) any later version. + */ + +#define pr_fmt(fmt) "powernv: " fmt + +#include <linux/kernel.h> +#include <linux/cpu.h> +#include <linux/cpumask.h> +#include <linux/device.h> +#include <linux/gfp.h> +#include <linux/smp.h> +#include <linux/stop_machine.h> + +#include <asm/cputhreads.h> +#include <asm/kvm_ppc.h> +#include <asm/machdep.h> +#include <asm/opal.h> +#include <asm/smp.h> + +#include "subcore.h" + + +/* + * Split/unsplit procedure: + * + * A core can be in one of three states, unsplit, 2-way split, and 4-way split. + * + * The mapping to subcores_per_core is simple: + * + * State | subcores_per_core + * ------------|------------------ + * Unsplit | 1 + * 2-way split | 2 + * 4-way split | 4 + * + * The core is split along thread boundaries, the mapping between subcores and + * threads is as follows: + * + * Unsplit: + * ---------------------------- + * Subcore | 0 | + * ---------------------------- + * Thread | 0 1 2 3 4 5 6 7 | + * ---------------------------- + * + * 2-way split: + * ------------------------------------- + * Subcore | 0 | 1 | + * ------------------------------------- + * Thread | 0 1 2 3 | 4 5 6 7 | + * ------------------------------------- + * + * 4-way split: + * ----------------------------------------- + * Subcore | 0 | 1 | 2 | 3 | + * ----------------------------------------- + * Thread | 0 1 | 2 3 | 4 5 | 6 7 | + * ----------------------------------------- + * + * + * Transitions + * ----------- + * + * It is not possible to transition between either of the split states, the + * core must first be unsplit. The legal transitions are: + * + * ----------- --------------- + * | | <----> | 2-way split | + * | | --------------- + * | Unsplit | + * | | --------------- + * | | <----> | 4-way split | + * ----------- --------------- + * + * Unsplitting + * ----------- + * + * Unsplitting is the simpler procedure. It requires thread 0 to request the + * unsplit while all other threads NAP. + * + * Thread 0 clears HID0_POWER8_DYNLPARDIS (Dynamic LPAR Disable). This tells + * the hardware that if all threads except 0 are napping, the hardware should + * unsplit the core. + * + * Non-zero threads are sent to a NAP loop, they don't exit the loop until they + * see the core unsplit. + * + * Core 0 spins waiting for the hardware to see all the other threads napping + * and perform the unsplit. + * + * Once thread 0 sees the unsplit, it IPIs the secondary threads to wake them + * out of NAP. They will then see the core unsplit and exit the NAP loop. + * + * Splitting + * --------- + * + * The basic splitting procedure is fairly straight forward. However it is + * complicated by the fact that after the split occurs, the newly created + * subcores are not in a fully initialised state. + * + * Most notably the subcores do not have the correct value for SDR1, which + * means they must not be running in virtual mode when the split occurs. The + * subcores have separate timebases SPRs but these are pre-synchronised by + * opal. + * + * To begin with secondary threads are sent to an assembly routine. There they + * switch to real mode, so they are immune to the uninitialised SDR1 value. + * Once in real mode they indicate that they are in real mode, and spin waiting + * to see the core split. + * + * Thread 0 waits to see that all secondaries are in real mode, and then begins + * the splitting procedure. It firstly sets HID0_POWER8_DYNLPARDIS, which + * prevents the hardware from unsplitting. Then it sets the appropriate HID bit + * to request the split, and spins waiting to see that the split has happened. + * + * Concurrently the secondaries will notice the split. When they do they set up + * their SPRs, notably SDR1, and then they can return to virtual mode and exit + * the procedure. + */ + +/* Initialised at boot by subcore_init() */ +static int subcores_per_core; + +/* + * Used to communicate to offline cpus that we want them to pop out of the + * offline loop and do a split or unsplit. + * + * 0 - no split happening + * 1 - unsplit in progress + * 2 - split to 2 in progress + * 4 - split to 4 in progress + */ +static int new_split_mode; + +static cpumask_var_t cpu_offline_mask; + +struct split_state { + u8 step; + u8 master; +}; + +static DEFINE_PER_CPU(struct split_state, split_state); + +static void wait_for_sync_step(int step) +{ + int i, cpu = smp_processor_id(); + + for (i = cpu + 1; i < cpu + threads_per_core; i++) + while(per_cpu(split_state, i).step < step) + barrier(); + + /* Order the wait loop vs any subsequent loads/stores. */ + mb(); +} + +static void unsplit_core(void) +{ + u64 hid0, mask; + int i, cpu; + + mask = HID0_POWER8_2LPARMODE | HID0_POWER8_4LPARMODE; + + cpu = smp_processor_id(); + if (cpu_thread_in_core(cpu) != 0) { + while (mfspr(SPRN_HID0) & mask) + power7_nap(0); + + per_cpu(split_state, cpu).step = SYNC_STEP_UNSPLIT; + return; + } + + hid0 = mfspr(SPRN_HID0); + hid0 &= ~HID0_POWER8_DYNLPARDIS; + mtspr(SPRN_HID0, hid0); + + while (mfspr(SPRN_HID0) & mask) + cpu_relax(); + + /* Wake secondaries out of NAP */ + for (i = cpu + 1; i < cpu + threads_per_core; i++) + smp_send_reschedule(i); + + wait_for_sync_step(SYNC_STEP_UNSPLIT); +} + +static void split_core(int new_mode) +{ + struct { u64 value; u64 mask; } split_parms[2] = { + { HID0_POWER8_1TO2LPAR, HID0_POWER8_2LPARMODE }, + { HID0_POWER8_1TO4LPAR, HID0_POWER8_4LPARMODE } + }; + int i, cpu; + u64 hid0; + + /* Convert new_mode (2 or 4) into an index into our parms array */ + i = (new_mode >> 1) - 1; + BUG_ON(i < 0 || i > 1); + + cpu = smp_processor_id(); + if (cpu_thread_in_core(cpu) != 0) { + split_core_secondary_loop(&per_cpu(split_state, cpu).step); + return; + } + + wait_for_sync_step(SYNC_STEP_REAL_MODE); + + /* Write new mode */ + hid0 = mfspr(SPRN_HID0); + hid0 |= HID0_POWER8_DYNLPARDIS | split_parms[i].value; + mtspr(SPRN_HID0, hid0); + + /* Wait for it to happen */ + while (!(mfspr(SPRN_HID0) & split_parms[i].mask)) + cpu_relax(); +} + +static void cpu_do_split(int new_mode) +{ + /* + * At boot subcores_per_core will be 0, so we will always unsplit at + * boot. In the usual case where the core is already unsplit it's a + * nop, and this just ensures the kernel's notion of the mode is + * consistent with the hardware. + */ + if (subcores_per_core != 1) + unsplit_core(); + + if (new_mode != 1) + split_core(new_mode); + + mb(); + per_cpu(split_state, smp_processor_id()).step = SYNC_STEP_FINISHED; +} + +bool cpu_core_split_required(void) +{ + smp_rmb(); + + if (!new_split_mode) + return false; + + cpu_do_split(new_split_mode); + + return true; +} + +static int cpu_update_split_mode(void *data) +{ + int cpu, new_mode = *(int *)data; + + if (this_cpu_ptr(&split_state)->master) { + new_split_mode = new_mode; + smp_wmb(); + + cpumask_andnot(cpu_offline_mask, cpu_present_mask, + cpu_online_mask); + + /* This should work even though the cpu is offline */ + for_each_cpu(cpu, cpu_offline_mask) + smp_send_reschedule(cpu); + } + + cpu_do_split(new_mode); + + if (this_cpu_ptr(&split_state)->master) { + /* Wait for all cpus to finish before we touch subcores_per_core */ + for_each_present_cpu(cpu) { + if (cpu >= setup_max_cpus) + break; + + while(per_cpu(split_state, cpu).step < SYNC_STEP_FINISHED) + barrier(); + } + + new_split_mode = 0; + + /* Make the new mode public */ + subcores_per_core = new_mode; + threads_per_subcore = threads_per_core / subcores_per_core; + + /* Make sure the new mode is written before we exit */ + mb(); + } + + return 0; +} + +static int set_subcores_per_core(int new_mode) +{ + struct split_state *state; + int cpu; + + if (kvm_hv_mode_active()) { + pr_err("Unable to change split core mode while KVM active.\n"); + return -EBUSY; + } + + /* + * We are only called at boot, or from the sysfs write. If that ever + * changes we'll need a lock here. + */ + BUG_ON(new_mode < 1 || new_mode > 4 || new_mode == 3); + + for_each_present_cpu(cpu) { + state = &per_cpu(split_state, cpu); + state->step = SYNC_STEP_INITIAL; + state->master = 0; + } + + get_online_cpus(); + + /* This cpu will update the globals before exiting stop machine */ + this_cpu_ptr(&split_state)->master = 1; + + /* Ensure state is consistent before we call the other cpus */ + mb(); + + stop_machine(cpu_update_split_mode, &new_mode, cpu_online_mask); + + put_online_cpus(); + + return 0; +} + +static ssize_t __used store_subcores_per_core(struct device *dev, + struct device_attribute *attr, const char *buf, + size_t count) +{ + unsigned long val; + int rc; + + /* We are serialised by the attribute lock */ + + rc = sscanf(buf, "%lx", &val); + if (rc != 1) + return -EINVAL; + + switch (val) { + case 1: + case 2: + case 4: + if (subcores_per_core == val) + /* Nothing to do */ + goto out; + break; + default: + return -EINVAL; + } + + rc = set_subcores_per_core(val); + if (rc) + return rc; + +out: + return count; +} + +static ssize_t show_subcores_per_core(struct device *dev, + struct device_attribute *attr, char *buf) +{ + return sprintf(buf, "%x\n", subcores_per_core); +} + +static DEVICE_ATTR(subcores_per_core, 0644, + show_subcores_per_core, store_subcores_per_core); + +static int subcore_init(void) +{ + if (!cpu_has_feature(CPU_FTR_ARCH_207S)) + return 0; + + /* + * We need all threads in a core to be present to split/unsplit so + * continue only if max_cpus are aligned to threads_per_core. + */ + if (setup_max_cpus % threads_per_core) + return 0; + + BUG_ON(!alloc_cpumask_var(&cpu_offline_mask, GFP_KERNEL)); + + set_subcores_per_core(1); + + return device_create_file(cpu_subsys.dev_root, + &dev_attr_subcores_per_core); +} +machine_device_initcall(powernv, subcore_init); diff --git a/arch/powerpc/platforms/powernv/subcore.h b/arch/powerpc/platforms/powernv/subcore.h new file mode 100644 index 000000000000..148abc91debf --- /dev/null +++ b/arch/powerpc/platforms/powernv/subcore.h @@ -0,0 +1,18 @@ +/* + * Copyright 2013, Michael Ellerman, IBM Corporation. + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License + * as published by the Free Software Foundation; either version + * 2 of the License, or (at your option) any later version. + */ + +/* These are ordered and tested with <= */ +#define SYNC_STEP_INITIAL 0 +#define SYNC_STEP_UNSPLIT 1 /* Set by secondary when it sees unsplit */ +#define SYNC_STEP_REAL_MODE 2 /* Set by secondary when in real mode */ +#define SYNC_STEP_FINISHED 3 /* Set by secondary when split/unsplit is done */ + +#ifndef __ASSEMBLY__ +void split_core_secondary_loop(u8 *state); +#endif diff --git a/arch/powerpc/platforms/pseries/eeh_pseries.c b/arch/powerpc/platforms/pseries/eeh_pseries.c index 8a8f0472d98f..0bec0c02c5e7 100644 --- a/arch/powerpc/platforms/pseries/eeh_pseries.c +++ b/arch/powerpc/platforms/pseries/eeh_pseries.c @@ -175,6 +175,36 @@ static int pseries_eeh_find_cap(struct device_node *dn, int cap) return 0; } +static int pseries_eeh_find_ecap(struct device_node *dn, int cap) +{ + struct pci_dn *pdn = PCI_DN(dn); + struct eeh_dev *edev = of_node_to_eeh_dev(dn); + u32 header; + int pos = 256; + int ttl = (4096 - 256) / 8; + + if (!edev || !edev->pcie_cap) + return 0; + if (rtas_read_config(pdn, pos, 4, &header) != PCIBIOS_SUCCESSFUL) + return 0; + else if (!header) + return 0; + + while (ttl-- > 0) { + if (PCI_EXT_CAP_ID(header) == cap && pos) + return pos; + + pos = PCI_EXT_CAP_NEXT(header); + if (pos < 256) + break; + + if (rtas_read_config(pdn, pos, 4, &header) != PCIBIOS_SUCCESSFUL) + break; + } + + return 0; +} + /** * pseries_eeh_of_probe - EEH probe on the given device * @dn: OF node @@ -220,7 +250,9 @@ static void *pseries_eeh_of_probe(struct device_node *dn, void *flag) * or PCIe switch downstream port. */ edev->class_code = class_code; + edev->pcix_cap = pseries_eeh_find_cap(dn, PCI_CAP_ID_PCIX); edev->pcie_cap = pseries_eeh_find_cap(dn, PCI_CAP_ID_EXP); + edev->aer_cap = pseries_eeh_find_ecap(dn, PCI_EXT_CAP_ID_ERR); edev->mode &= 0xFFFFFF00; if ((edev->class_code >> 8) == PCI_CLASS_BRIDGE_PCI) { edev->mode |= EEH_DEV_BRIDGE; @@ -464,6 +496,7 @@ static int pseries_eeh_get_state(struct eeh_pe *pe, int *state) } else { result = EEH_STATE_NOT_SUPPORT; } + break; default: result = EEH_STATE_NOT_SUPPORT; } @@ -499,11 +532,19 @@ static int pseries_eeh_reset(struct eeh_pe *pe, int option) /* If fundamental-reset not supported, try hot-reset */ if (option == EEH_RESET_FUNDAMENTAL && ret == -8) { + option = EEH_RESET_HOT; ret = rtas_call(ibm_set_slot_reset, 4, 1, NULL, config_addr, BUID_HI(pe->phb->buid), - BUID_LO(pe->phb->buid), EEH_RESET_HOT); + BUID_LO(pe->phb->buid), option); } + /* We need reset hold or settlement delay */ + if (option == EEH_RESET_FUNDAMENTAL || + option == EEH_RESET_HOT) + msleep(EEH_PE_RST_HOLD_TIME); + else + msleep(EEH_PE_RST_SETTLE_TIME); + return ret; } diff --git a/arch/powerpc/platforms/pseries/hotplug-memory.c b/arch/powerpc/platforms/pseries/hotplug-memory.c index 7f75c94af822..7995135170a3 100644 --- a/arch/powerpc/platforms/pseries/hotplug-memory.c +++ b/arch/powerpc/platforms/pseries/hotplug-memory.c @@ -21,7 +21,7 @@ #include <asm/prom.h> #include <asm/sparsemem.h> -static unsigned long get_memblock_size(void) +unsigned long pseries_memory_block_size(void) { struct device_node *np; unsigned int memblock_size = MIN_MEMORY_BLOCK_SIZE; @@ -64,17 +64,6 @@ static unsigned long get_memblock_size(void) return memblock_size; } -/* WARNING: This is going to override the generic definition whenever - * pseries is built-in regardless of what platform is active at boot - * time. This is fine for now as this is the only "option" and it - * should work everywhere. If not, we'll have to turn this into a - * ppc_md. callback - */ -unsigned long memory_block_size_bytes(void) -{ - return get_memblock_size(); -} - #ifdef CONFIG_MEMORY_HOTREMOVE static int pseries_remove_memory(u64 start, u64 size) { @@ -105,7 +94,7 @@ static int pseries_remove_memblock(unsigned long base, unsigned int memblock_siz if (!pfn_valid(start_pfn)) goto out; - block_sz = memory_block_size_bytes(); + block_sz = pseries_memory_block_size(); sections_per_block = block_sz / MIN_MEMORY_BLOCK_SIZE; nid = memory_add_physaddr_to_nid(base); @@ -201,7 +190,7 @@ static int pseries_update_drconf_memory(struct of_prop_reconfig *pr) u32 *p; int i, rc = -EINVAL; - memblock_size = get_memblock_size(); + memblock_size = pseries_memory_block_size(); if (!memblock_size) return -EINVAL; diff --git a/arch/powerpc/platforms/pseries/hvCall.S b/arch/powerpc/platforms/pseries/hvCall.S index 444fe7759e55..99ecf0a5a929 100644 --- a/arch/powerpc/platforms/pseries/hvCall.S +++ b/arch/powerpc/platforms/pseries/hvCall.S @@ -49,7 +49,7 @@ END_FTR_SECTION(0, 1); \ std r0,16(r1); \ addi r4,r1,STK_PARAM(FIRST_REG); \ stdu r1,-STACK_FRAME_OVERHEAD(r1); \ - bl .__trace_hcall_entry; \ + bl __trace_hcall_entry; \ addi r1,r1,STACK_FRAME_OVERHEAD; \ ld r0,16(r1); \ ld r3,STK_PARAM(R3)(r1); \ @@ -83,7 +83,7 @@ END_FTR_SECTION(0, 1); \ mr r3,r6; \ std r0,16(r1); \ stdu r1,-STACK_FRAME_OVERHEAD(r1); \ - bl .__trace_hcall_exit; \ + bl __trace_hcall_exit; \ addi r1,r1,STACK_FRAME_OVERHEAD; \ ld r0,16(r1); \ ld r3,STK_PARAM(R3)(r1); \ @@ -106,7 +106,7 @@ END_FTR_SECTION(0, 1); \ .text -_GLOBAL(plpar_hcall_norets) +_GLOBAL_TOC(plpar_hcall_norets) HMT_MEDIUM mfcr r0 @@ -122,7 +122,7 @@ _GLOBAL(plpar_hcall_norets) mtcrf 0xff,r0 blr /* return r3 = status */ -_GLOBAL(plpar_hcall) +_GLOBAL_TOC(plpar_hcall) HMT_MEDIUM mfcr r0 @@ -188,7 +188,7 @@ _GLOBAL(plpar_hcall_raw) blr /* return r3 = status */ -_GLOBAL(plpar_hcall9) +_GLOBAL_TOC(plpar_hcall9) HMT_MEDIUM mfcr r0 diff --git a/arch/powerpc/platforms/pseries/pseries.h b/arch/powerpc/platforms/pseries/pseries.h index 99219530ea4a..361add62abf1 100644 --- a/arch/powerpc/platforms/pseries/pseries.h +++ b/arch/powerpc/platforms/pseries/pseries.h @@ -64,4 +64,6 @@ extern int dlpar_detach_node(struct device_node *); struct pci_host_bridge; int pseries_root_bridge_prepare(struct pci_host_bridge *bridge); +unsigned long pseries_memory_block_size(void); + #endif /* _PSERIES_PSERIES_H */ diff --git a/arch/powerpc/platforms/pseries/setup.c b/arch/powerpc/platforms/pseries/setup.c index 099d2df976a2..f2f40e64658f 100644 --- a/arch/powerpc/platforms/pseries/setup.c +++ b/arch/powerpc/platforms/pseries/setup.c @@ -510,7 +510,11 @@ static void __init pSeries_setup_arch(void) static int __init pSeries_init_panel(void) { /* Manually leave the kernel version on the panel. */ +#ifdef __BIG_ENDIAN__ ppc_md.progress("Linux ppc64\n", 0); +#else + ppc_md.progress("Linux ppc64le\n", 0); +#endif ppc_md.progress(init_utsname()->version, 0); return 0; @@ -806,4 +810,7 @@ define_machine(pseries) { #ifdef CONFIG_KEXEC .machine_kexec = pSeries_machine_kexec, #endif +#ifdef CONFIG_MEMORY_HOTPLUG_SPARSE + .memory_block_size = pseries_memory_block_size, +#endif }; diff --git a/arch/powerpc/platforms/pseries/smp.c b/arch/powerpc/platforms/pseries/smp.c index 24f58cb0a543..a3555b10c1a5 100644 --- a/arch/powerpc/platforms/pseries/smp.c +++ b/arch/powerpc/platforms/pseries/smp.c @@ -44,6 +44,7 @@ #include <asm/xics.h> #include <asm/dbell.h> #include <asm/plpar_wrappers.h> +#include <asm/code-patching.h> #include "pseries.h" #include "offline_states.h" @@ -96,8 +97,8 @@ int smp_query_cpu_stopped(unsigned int pcpu) static inline int smp_startup_cpu(unsigned int lcpu) { int status; - unsigned long start_here = __pa((u32)*((unsigned long *) - generic_secondary_smp_init)); + unsigned long start_here = + __pa(ppc_function_entry(generic_secondary_smp_init)); unsigned int pcpu; int start_cpu; diff --git a/arch/powerpc/platforms/wsp/scom_smp.c b/arch/powerpc/platforms/wsp/scom_smp.c index 268bc899c1f7..8c79ce016cf1 100644 --- a/arch/powerpc/platforms/wsp/scom_smp.c +++ b/arch/powerpc/platforms/wsp/scom_smp.c @@ -20,6 +20,7 @@ #include <asm/reg_a2.h> #include <asm/scom.h> #include <asm/udbg.h> +#include <asm/code-patching.h> #include "wsp.h" @@ -405,7 +406,7 @@ int a2_scom_startup_cpu(unsigned int lcpu, int thr_idx, struct device_node *np) goto fail; } - start_here = *(unsigned long *)(core_setup ? generic_secondary_smp_init + start_here = ppc_function_entry(core_setup ? generic_secondary_smp_init : generic_secondary_thread_init); pr_devel("CPU%d entry point at 0x%lx...\n", lcpu, start_here); diff --git a/arch/powerpc/sysdev/Kconfig b/arch/powerpc/sysdev/Kconfig index 7baa70d6dc01..a19332a38715 100644 --- a/arch/powerpc/sysdev/Kconfig +++ b/arch/powerpc/sysdev/Kconfig @@ -7,6 +7,12 @@ config PPC4xx_PCI_EXPRESS depends on PCI && 4xx default n +config PPC4xx_HSTA_MSI + bool + depends on PCI_MSI + depends on PCI && 4xx + default n + config PPC4xx_MSI bool depends on PCI_MSI diff --git a/arch/powerpc/sysdev/Makefile b/arch/powerpc/sysdev/Makefile index afbcc37aa094..f7cb2a1b01fa 100644 --- a/arch/powerpc/sysdev/Makefile +++ b/arch/powerpc/sysdev/Makefile @@ -45,6 +45,7 @@ obj-$(CONFIG_OF_RTC) += of_rtc.o ifeq ($(CONFIG_PCI),y) obj-$(CONFIG_4xx) += ppc4xx_pci.o endif +obj-$(CONFIG_PPC4xx_HSTA_MSI) += ppc4xx_hsta_msi.o obj-$(CONFIG_PPC4xx_MSI) += ppc4xx_msi.o obj-$(CONFIG_PPC4xx_CPM) += ppc4xx_cpm.o obj-$(CONFIG_PPC4xx_GPIO) += ppc4xx_gpio.o diff --git a/arch/powerpc/sysdev/fsl_pci.c b/arch/powerpc/sysdev/fsl_pci.c index 3f415e252ea5..4bd091a05583 100644 --- a/arch/powerpc/sysdev/fsl_pci.c +++ b/arch/powerpc/sysdev/fsl_pci.c @@ -1150,8 +1150,7 @@ static int fsl_pci_pme_probe(struct pci_controller *hose) pci = hose->private_data; /* Enable PTOD, ENL23D & EXL23D */ - out_be32(&pci->pex_pme_mes_disr, 0); - setbits32(&pci->pex_pme_mes_disr, + clrbits32(&pci->pex_pme_mes_disr, PME_DISR_EN_PTOD | PME_DISR_EN_ENL23D | PME_DISR_EN_EXL23D); out_be32(&pci->pex_pme_mes_ier, 0); diff --git a/arch/powerpc/sysdev/fsl_rio.c b/arch/powerpc/sysdev/fsl_rio.c index cf2b0840a672..c04b718307c8 100644 --- a/arch/powerpc/sysdev/fsl_rio.c +++ b/arch/powerpc/sysdev/fsl_rio.c @@ -391,8 +391,10 @@ int fsl_rio_setup(struct platform_device *dev) ops->get_inb_message = fsl_get_inb_message; rmu_node = of_parse_phandle(dev->dev.of_node, "fsl,srio-rmu-handle", 0); - if (!rmu_node) + if (!rmu_node) { + dev_err(&dev->dev, "No valid fsl,srio-rmu-handle property\n"); goto err_rmu; + } rc = of_address_to_resource(rmu_node, 0, &rmu_regs); if (rc) { dev_err(&dev->dev, "Can't get %s property 'reg'\n", @@ -413,6 +415,7 @@ int fsl_rio_setup(struct platform_device *dev) /*set up doobell node*/ np = of_find_compatible_node(NULL, NULL, "fsl,srio-dbell-unit"); if (!np) { + dev_err(&dev->dev, "No fsl,srio-dbell-unit node\n"); rc = -ENODEV; goto err_dbell; } @@ -441,6 +444,7 @@ int fsl_rio_setup(struct platform_device *dev) /*set up port write node*/ np = of_find_compatible_node(NULL, NULL, "fsl,srio-port-write-unit"); if (!np) { + dev_err(&dev->dev, "No fsl,srio-port-write-unit node\n"); rc = -ENODEV; goto err_pw; } @@ -633,14 +637,18 @@ int fsl_rio_setup(struct platform_device *dev) return 0; err: kfree(pw); + pw = NULL; err_pw: kfree(dbell); + dbell = NULL; err_dbell: iounmap(rmu_regs_win); + rmu_regs_win = NULL; err_rmu: kfree(ops); err_ops: iounmap(rio_regs_win); + rio_regs_win = NULL; err_rio_regs: return rc; } diff --git a/arch/powerpc/sysdev/fsl_rmu.c b/arch/powerpc/sysdev/fsl_rmu.c index 00e224a1048c..b48197ae44d0 100644 --- a/arch/powerpc/sysdev/fsl_rmu.c +++ b/arch/powerpc/sysdev/fsl_rmu.c @@ -881,9 +881,9 @@ fsl_open_inb_mbox(struct rio_mport *mport, void *dev_id, int mbox, int entries) rc = request_irq(IRQ_RIO_RX(mport), fsl_rio_rx_handler, 0, "msg_rx", (void *)mport); if (rc < 0) { - dma_free_coherent(priv->dev, RIO_MSG_BUFFER_SIZE, - rmu->msg_tx_ring.virt_buffer[i], - rmu->msg_tx_ring.phys_buffer[i]); + dma_free_coherent(priv->dev, + rmu->msg_rx_ring.size * RIO_MAX_MSG_SIZE, + rmu->msg_rx_ring.virt, rmu->msg_rx_ring.phys); goto out; } diff --git a/arch/powerpc/sysdev/mpic.c b/arch/powerpc/sysdev/mpic.c index 8209744b2829..be33c9768ea1 100644 --- a/arch/powerpc/sysdev/mpic.c +++ b/arch/powerpc/sysdev/mpic.c @@ -1588,10 +1588,6 @@ void __init mpic_init(struct mpic *mpic) num_timers = 8; } - /* FSL mpic error interrupt intialization */ - if (mpic->flags & MPIC_FSL_HAS_EIMR) - mpic_err_int_init(mpic, MPIC_FSL_ERR_INT); - /* Initialize timers to our reserved vectors and mask them for now */ for (i = 0; i < num_timers; i++) { unsigned int offset = mpic_tm_offset(mpic, i); @@ -1675,6 +1671,10 @@ void __init mpic_init(struct mpic *mpic) irq_set_chained_handler(virq, &mpic_cascade); } } + + /* FSL mpic error interrupt intialization */ + if (mpic->flags & MPIC_FSL_HAS_EIMR) + mpic_err_int_init(mpic, MPIC_FSL_ERR_INT); } void __init mpic_set_clk_ratio(struct mpic *mpic, u32 clock_ratio) diff --git a/arch/powerpc/sysdev/ppc4xx_hsta_msi.c b/arch/powerpc/sysdev/ppc4xx_hsta_msi.c new file mode 100644 index 000000000000..11c888416f0a --- /dev/null +++ b/arch/powerpc/sysdev/ppc4xx_hsta_msi.c @@ -0,0 +1,215 @@ +/* + * MSI support for PPC4xx SoCs using High Speed Transfer Assist (HSTA) for + * generation of the interrupt. + * + * Copyright © 2013 Alistair Popple <alistair@popple.id.au> IBM Corporation + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the + * Free Software Foundation; either version 2 of the License, or (at your + * option) any later version. + */ + +#include <linux/kernel.h> +#include <linux/interrupt.h> +#include <linux/msi.h> +#include <linux/of.h> +#include <linux/of_platform.h> +#include <linux/pci.h> +#include <linux/semaphore.h> +#include <asm/msi_bitmap.h> + +struct ppc4xx_hsta_msi { + struct device *dev; + + /* The ioremapped HSTA MSI IO space */ + u32 __iomem *data; + + /* Physical address of HSTA MSI IO space */ + u64 address; + struct msi_bitmap bmp; + + /* An array mapping offsets to hardware IRQs */ + int *irq_map; + + /* Number of hwirqs supported */ + int irq_count; +}; +static struct ppc4xx_hsta_msi ppc4xx_hsta_msi; + +static int hsta_setup_msi_irqs(struct pci_dev *dev, int nvec, int type) +{ + struct msi_msg msg; + struct msi_desc *entry; + int irq, hwirq; + u64 addr; + + list_for_each_entry(entry, &dev->msi_list, list) { + irq = msi_bitmap_alloc_hwirqs(&ppc4xx_hsta_msi.bmp, 1); + if (irq < 0) { + pr_debug("%s: Failed to allocate msi interrupt\n", + __func__); + return irq; + } + + hwirq = ppc4xx_hsta_msi.irq_map[irq]; + if (hwirq == NO_IRQ) { + pr_err("%s: Failed mapping irq %d\n", __func__, irq); + return -EINVAL; + } + + /* + * HSTA generates interrupts on writes to 128-bit aligned + * addresses. + */ + addr = ppc4xx_hsta_msi.address + irq*0x10; + msg.address_hi = upper_32_bits(addr); + msg.address_lo = lower_32_bits(addr); + + /* Data is not used by the HSTA. */ + msg.data = 0; + + pr_debug("%s: Setup irq %d (0x%0llx)\n", __func__, hwirq, + (((u64) msg.address_hi) << 32) | msg.address_lo); + + if (irq_set_msi_desc(hwirq, entry)) { + pr_err( + "%s: Invalid hwirq %d specified in device tree\n", + __func__, hwirq); + msi_bitmap_free_hwirqs(&ppc4xx_hsta_msi.bmp, irq, 1); + return -EINVAL; + } + write_msi_msg(hwirq, &msg); + } + + return 0; +} + +static int hsta_find_hwirq_offset(int hwirq) +{ + int irq; + + /* Find the offset given the hwirq */ + for (irq = 0; irq < ppc4xx_hsta_msi.irq_count; irq++) + if (ppc4xx_hsta_msi.irq_map[irq] == hwirq) + return irq; + + return -EINVAL; +} + +static void hsta_teardown_msi_irqs(struct pci_dev *dev) +{ + struct msi_desc *entry; + int irq; + + list_for_each_entry(entry, &dev->msi_list, list) { + if (entry->irq == NO_IRQ) + continue; + + irq = hsta_find_hwirq_offset(entry->irq); + + /* entry->irq should always be in irq_map */ + BUG_ON(irq < 0); + irq_set_msi_desc(entry->irq, NULL); + msi_bitmap_free_hwirqs(&ppc4xx_hsta_msi.bmp, irq, 1); + pr_debug("%s: Teardown IRQ %u (index %u)\n", __func__, + entry->irq, irq); + } +} + +static int hsta_msi_check_device(struct pci_dev *pdev, int nvec, int type) +{ + /* We don't support MSI-X */ + if (type == PCI_CAP_ID_MSIX) { + pr_debug("%s: MSI-X not supported.\n", __func__); + return -EINVAL; + } + + return 0; +} + +static int hsta_msi_probe(struct platform_device *pdev) +{ + struct device *dev = &pdev->dev; + struct resource *mem; + int irq, ret, irq_count; + + mem = platform_get_resource(pdev, IORESOURCE_MEM, 0); + if (IS_ERR(mem)) { + dev_err(dev, "Unable to get mmio space\n"); + return -EINVAL; + } + + irq_count = of_irq_count(dev->of_node); + if (!irq_count) { + dev_err(dev, "Unable to find IRQ range\n"); + return -EINVAL; + } + + ppc4xx_hsta_msi.dev = dev; + ppc4xx_hsta_msi.address = mem->start; + ppc4xx_hsta_msi.data = ioremap(mem->start, resource_size(mem)); + ppc4xx_hsta_msi.irq_count = irq_count; + if (IS_ERR(ppc4xx_hsta_msi.data)) { + dev_err(dev, "Unable to map memory\n"); + return -ENOMEM; + } + + ret = msi_bitmap_alloc(&ppc4xx_hsta_msi.bmp, irq_count, dev->of_node); + if (ret) + goto out; + + ppc4xx_hsta_msi.irq_map = kmalloc(sizeof(int) * irq_count, GFP_KERNEL); + if (IS_ERR(ppc4xx_hsta_msi.irq_map)) { + ret = -ENOMEM; + goto out1; + } + + /* Setup a mapping from irq offsets to hardware irq numbers */ + for (irq = 0; irq < irq_count; irq++) { + ppc4xx_hsta_msi.irq_map[irq] = + irq_of_parse_and_map(dev->of_node, irq); + if (ppc4xx_hsta_msi.irq_map[irq] == NO_IRQ) { + dev_err(dev, "Unable to map IRQ\n"); + ret = -EINVAL; + goto out2; + } + } + + ppc_md.setup_msi_irqs = hsta_setup_msi_irqs; + ppc_md.teardown_msi_irqs = hsta_teardown_msi_irqs; + ppc_md.msi_check_device = hsta_msi_check_device; + return 0; + +out2: + kfree(ppc4xx_hsta_msi.irq_map); + +out1: + msi_bitmap_free(&ppc4xx_hsta_msi.bmp); + +out: + iounmap(ppc4xx_hsta_msi.data); + return ret; +} + +static const struct of_device_id hsta_msi_ids[] = { + { + .compatible = "ibm,hsta-msi", + }, + {} +}; + +static struct platform_driver hsta_msi_driver = { + .probe = hsta_msi_probe, + .driver = { + .name = "hsta-msi", + .owner = THIS_MODULE, + .of_match_table = hsta_msi_ids, + }, +}; + +static int hsta_msi_init(void) +{ + return platform_driver_register(&hsta_msi_driver); +} +subsys_initcall(hsta_msi_init); diff --git a/arch/powerpc/sysdev/ppc4xx_pci.c b/arch/powerpc/sysdev/ppc4xx_pci.c index 4914fd3f41ec..df6e2fc4ff92 100644 --- a/arch/powerpc/sysdev/ppc4xx_pci.c +++ b/arch/powerpc/sysdev/ppc4xx_pci.c @@ -176,8 +176,12 @@ static int __init ppc4xx_parse_dma_ranges(struct pci_controller *hose, return -ENXIO; } - /* Check that we are fully contained within 32 bits space */ - if (res->end > 0xffffffff) { + /* Check that we are fully contained within 32 bits space if we are not + * running on a 460sx or 476fpe which have 64 bit bus addresses. + */ + if (res->end > 0xffffffff && + !(of_device_is_compatible(hose->dn, "ibm,plb-pciex-460sx") + || of_device_is_compatible(hose->dn, "ibm,plb-pciex-476fpe"))) { printk(KERN_ERR "%s: dma-ranges outside of 32 bits space\n", hose->dn->full_name); return -ENXIO; @@ -1440,7 +1444,8 @@ static int __init ppc4xx_pciex_check_core_init(struct device_node *np) ppc4xx_pciex_hwops = &ppc405ex_pcie_hwops; #endif #ifdef CONFIG_476FPE - if (of_device_is_compatible(np, "ibm,plb-pciex-476fpe")) + if (of_device_is_compatible(np, "ibm,plb-pciex-476fpe") + || of_device_is_compatible(np, "ibm,plb-pciex-476gtr")) ppc4xx_pciex_hwops = &ppc_476fpe_pcie_hwops; #endif if (ppc4xx_pciex_hwops == NULL) { @@ -1751,7 +1756,10 @@ static int __init ppc4xx_setup_one_pciex_POM(struct ppc4xx_pciex_port *port, dcr_write(port->dcrs, DCRO_PEGPL_OMR1MSKL, sa | DCRO_PEGPL_460SX_OMR1MSKL_UOT | DCRO_PEGPL_OMRxMSKL_VAL); - else if (of_device_is_compatible(port->node, "ibm,plb-pciex-476fpe")) + else if (of_device_is_compatible( + port->node, "ibm,plb-pciex-476fpe") || + of_device_is_compatible( + port->node, "ibm,plb-pciex-476gtr")) dcr_write(port->dcrs, DCRO_PEGPL_OMR1MSKL, sa | DCRO_PEGPL_476FPE_OMR1MSKL_UOT | DCRO_PEGPL_OMRxMSKL_VAL); @@ -1881,7 +1889,10 @@ static void __init ppc4xx_configure_pciex_PIMs(struct ppc4xx_pciex_port *port, sa |= PCI_BASE_ADDRESS_MEM_PREFETCH; if (of_device_is_compatible(port->node, "ibm,plb-pciex-460sx") || - of_device_is_compatible(port->node, "ibm,plb-pciex-476fpe")) + of_device_is_compatible( + port->node, "ibm,plb-pciex-476fpe") || + of_device_is_compatible( + port->node, "ibm,plb-pciex-476gtr")) sa |= PCI_BASE_ADDRESS_MEM_TYPE_64; out_le32(mbase + PECFG_BAR0HMPA, RES_TO_U32_HIGH(sa)); diff --git a/arch/powerpc/xmon/xmon.c b/arch/powerpc/xmon/xmon.c index 08504e75b2c7..d199bfa2f1fa 100644 --- a/arch/powerpc/xmon/xmon.c +++ b/arch/powerpc/xmon/xmon.c @@ -419,7 +419,7 @@ static int xmon_core(struct pt_regs *regs, int fromipi) get_output_lock(); excprint(regs); if (bp) { - printf("cpu 0x%x stopped at breakpoint 0x%x (", + printf("cpu 0x%x stopped at breakpoint 0x%lx (", cpu, BP_NUM(bp)); xmon_print_symbol(regs->nip, " ", ")\n"); } @@ -513,7 +513,7 @@ static int xmon_core(struct pt_regs *regs, int fromipi) excprint(regs); bp = at_breakpoint(regs->nip); if (bp) { - printf("Stopped at breakpoint %x (", BP_NUM(bp)); + printf("Stopped at breakpoint %lx (", BP_NUM(bp)); xmon_print_symbol(regs->nip, " ", ")\n"); } if (unrecoverable_excp(regs)) @@ -759,7 +759,7 @@ static void insert_cpu_bpts(void) brk.address = dabr.address; brk.type = (dabr.enabled & HW_BRK_TYPE_DABR) | HW_BRK_TYPE_PRIV_ALL; brk.len = 8; - set_breakpoint(&brk); + __set_breakpoint(&brk); } if (iabr && cpu_has_feature(CPU_FTR_IABR)) mtspr(SPRN_IABR, iabr->address @@ -997,14 +997,14 @@ static int cpu_cmd(void) last_cpu = cpu; } else { if (last_cpu != first_cpu) - printf("-%lx", last_cpu); + printf("-0x%lx", last_cpu); last_cpu = first_cpu = cpu; - printf(" %lx", cpu); + printf(" 0x%lx", cpu); } } } if (last_cpu != first_cpu) - printf("-%lx", last_cpu); + printf("-0x%lx", last_cpu); printf("\n"); return 0; } @@ -1024,7 +1024,7 @@ static int cpu_cmd(void) /* take control back */ mb(); xmon_owner = smp_processor_id(); - printf("cpu %u didn't take control\n", cpu); + printf("cpu 0x%x didn't take control\n", cpu); return 0; } barrier(); @@ -1086,7 +1086,7 @@ csum(void) fcs = 0xffff; for (i = 0; i < ncsum; ++i) { if (mread(adrs+i, &v, 1) == 0) { - printf("csum stopped at %x\n", adrs+i); + printf("csum stopped at "REG"\n", adrs+i); break; } fcs = FCS(fcs, v); @@ -1202,12 +1202,12 @@ bpt_cmds(void) /* assume a breakpoint address */ bp = at_breakpoint(a); if (bp == NULL) { - printf("No breakpoint at %x\n", a); + printf("No breakpoint at %lx\n", a); break; } } - printf("Cleared breakpoint %x (", BP_NUM(bp)); + printf("Cleared breakpoint %lx (", BP_NUM(bp)); xmon_print_symbol(bp->address, " ", ")\n"); bp->enabled = 0; break; @@ -1746,7 +1746,7 @@ mwrite(unsigned long adrs, void *buf, int size) __delay(200); n = size; } else { - printf("*** Error writing address %x\n", adrs + n); + printf("*** Error writing address "REG"\n", adrs + n); } catch_memory_errors = 0; return n; @@ -2435,7 +2435,7 @@ static void proccall(void) ret = func(args[0], args[1], args[2], args[3], args[4], args[5], args[6], args[7]); sync(); - printf("return value is %x\n", ret); + printf("return value is 0x%lx\n", ret); } else { printf("*** %x exception occurred\n", fault_except); } @@ -2700,7 +2700,7 @@ static void dump_slb(void) unsigned long esid,vsid,valid; unsigned long llp; - printf("SLB contents of cpu %x\n", smp_processor_id()); + printf("SLB contents of cpu 0x%x\n", smp_processor_id()); for (i = 0; i < mmu_slb_size; i++) { asm volatile("slbmfee %0,%1" : "=r" (esid) : "r" (i)); @@ -2732,7 +2732,7 @@ static void dump_stab(void) int i; unsigned long *tmp = (unsigned long *)local_paca->stab_addr; - printf("Segment table contents of cpu %x\n", smp_processor_id()); + printf("Segment table contents of cpu 0x%x\n", smp_processor_id()); for (i = 0; i < PAGE_SIZE/16; i++) { unsigned long a, b; diff --git a/arch/x86/kernel/Makefile b/arch/x86/kernel/Makefile index 491ef3e59850..047f9ff2e36c 100644 --- a/arch/x86/kernel/Makefile +++ b/arch/x86/kernel/Makefile @@ -26,6 +26,7 @@ obj-$(CONFIG_IRQ_WORK) += irq_work.o obj-y += probe_roms.o obj-$(CONFIG_X86_32) += i386_ksyms_32.o obj-$(CONFIG_X86_64) += sys_x86_64.o x8664_ksyms_64.o +obj-$(CONFIG_X86_64) += mcount_64.o obj-y += syscall_$(BITS).o vsyscall_gtod.o obj-$(CONFIG_X86_64) += vsyscall_64.o obj-$(CONFIG_X86_64) += vsyscall_emu_64.o diff --git a/arch/x86/kernel/entry_64.S b/arch/x86/kernel/entry_64.S index 96987987c5de..48a2644a082a 100644 --- a/arch/x86/kernel/entry_64.S +++ b/arch/x86/kernel/entry_64.S @@ -53,7 +53,6 @@ #include <asm/page_types.h> #include <asm/irqflags.h> #include <asm/paravirt.h> -#include <asm/ftrace.h> #include <asm/percpu.h> #include <asm/asm.h> #include <asm/context_tracking.h> @@ -70,209 +69,6 @@ .code64 .section .entry.text, "ax" -#ifdef CONFIG_FUNCTION_TRACER - -#ifdef CC_USING_FENTRY -# define function_hook __fentry__ -#else -# define function_hook mcount -#endif - -#ifdef CONFIG_DYNAMIC_FTRACE - -ENTRY(function_hook) - retq -END(function_hook) - -/* skip is set if stack has been adjusted */ -.macro ftrace_caller_setup skip=0 - MCOUNT_SAVE_FRAME \skip - - /* Load the ftrace_ops into the 3rd parameter */ - movq function_trace_op(%rip), %rdx - - /* Load ip into the first parameter */ - movq RIP(%rsp), %rdi - subq $MCOUNT_INSN_SIZE, %rdi - /* Load the parent_ip into the second parameter */ -#ifdef CC_USING_FENTRY - movq SS+16(%rsp), %rsi -#else - movq 8(%rbp), %rsi -#endif -.endm - -ENTRY(ftrace_caller) - /* Check if tracing was disabled (quick check) */ - cmpl $0, function_trace_stop - jne ftrace_stub - - ftrace_caller_setup - /* regs go into 4th parameter (but make it NULL) */ - movq $0, %rcx - -GLOBAL(ftrace_call) - call ftrace_stub - - MCOUNT_RESTORE_FRAME -ftrace_return: - -#ifdef CONFIG_FUNCTION_GRAPH_TRACER -GLOBAL(ftrace_graph_call) - jmp ftrace_stub -#endif - -GLOBAL(ftrace_stub) - retq -END(ftrace_caller) - -ENTRY(ftrace_regs_caller) - /* Save the current flags before compare (in SS location)*/ - pushfq - - /* Check if tracing was disabled (quick check) */ - cmpl $0, function_trace_stop - jne ftrace_restore_flags - - /* skip=8 to skip flags saved in SS */ - ftrace_caller_setup 8 - - /* Save the rest of pt_regs */ - movq %r15, R15(%rsp) - movq %r14, R14(%rsp) - movq %r13, R13(%rsp) - movq %r12, R12(%rsp) - movq %r11, R11(%rsp) - movq %r10, R10(%rsp) - movq %rbp, RBP(%rsp) - movq %rbx, RBX(%rsp) - /* Copy saved flags */ - movq SS(%rsp), %rcx - movq %rcx, EFLAGS(%rsp) - /* Kernel segments */ - movq $__KERNEL_DS, %rcx - movq %rcx, SS(%rsp) - movq $__KERNEL_CS, %rcx - movq %rcx, CS(%rsp) - /* Stack - skipping return address */ - leaq SS+16(%rsp), %rcx - movq %rcx, RSP(%rsp) - - /* regs go into 4th parameter */ - leaq (%rsp), %rcx - -GLOBAL(ftrace_regs_call) - call ftrace_stub - - /* Copy flags back to SS, to restore them */ - movq EFLAGS(%rsp), %rax - movq %rax, SS(%rsp) - - /* Handlers can change the RIP */ - movq RIP(%rsp), %rax - movq %rax, SS+8(%rsp) - - /* restore the rest of pt_regs */ - movq R15(%rsp), %r15 - movq R14(%rsp), %r14 - movq R13(%rsp), %r13 - movq R12(%rsp), %r12 - movq R10(%rsp), %r10 - movq RBP(%rsp), %rbp - movq RBX(%rsp), %rbx - - /* skip=8 to skip flags saved in SS */ - MCOUNT_RESTORE_FRAME 8 - - /* Restore flags */ - popfq - - jmp ftrace_return -ftrace_restore_flags: - popfq - jmp ftrace_stub - -END(ftrace_regs_caller) - - -#else /* ! CONFIG_DYNAMIC_FTRACE */ - -ENTRY(function_hook) - cmpl $0, function_trace_stop - jne ftrace_stub - - cmpq $ftrace_stub, ftrace_trace_function - jnz trace - -#ifdef CONFIG_FUNCTION_GRAPH_TRACER - cmpq $ftrace_stub, ftrace_graph_return - jnz ftrace_graph_caller - - cmpq $ftrace_graph_entry_stub, ftrace_graph_entry - jnz ftrace_graph_caller -#endif - -GLOBAL(ftrace_stub) - retq - -trace: - MCOUNT_SAVE_FRAME - - movq RIP(%rsp), %rdi -#ifdef CC_USING_FENTRY - movq SS+16(%rsp), %rsi -#else - movq 8(%rbp), %rsi -#endif - subq $MCOUNT_INSN_SIZE, %rdi - - call *ftrace_trace_function - - MCOUNT_RESTORE_FRAME - - jmp ftrace_stub -END(function_hook) -#endif /* CONFIG_DYNAMIC_FTRACE */ -#endif /* CONFIG_FUNCTION_TRACER */ - -#ifdef CONFIG_FUNCTION_GRAPH_TRACER -ENTRY(ftrace_graph_caller) - MCOUNT_SAVE_FRAME - -#ifdef CC_USING_FENTRY - leaq SS+16(%rsp), %rdi - movq $0, %rdx /* No framepointers needed */ -#else - leaq 8(%rbp), %rdi - movq (%rbp), %rdx -#endif - movq RIP(%rsp), %rsi - subq $MCOUNT_INSN_SIZE, %rsi - - call prepare_ftrace_return - - MCOUNT_RESTORE_FRAME - - retq -END(ftrace_graph_caller) - -GLOBAL(return_to_handler) - subq $24, %rsp - - /* Save the return values */ - movq %rax, (%rsp) - movq %rdx, 8(%rsp) - movq %rbp, %rdi - - call ftrace_return_to_handler - - movq %rax, %rdi - movq 8(%rsp), %rdx - movq (%rsp), %rax - addq $24, %rsp - jmp *%rdi -#endif - #ifndef CONFIG_PREEMPT #define retint_kernel retint_restore_args diff --git a/arch/x86/kernel/ftrace.c b/arch/x86/kernel/ftrace.c index 52819e816f87..cbc4a91b131e 100644 --- a/arch/x86/kernel/ftrace.c +++ b/arch/x86/kernel/ftrace.c @@ -297,16 +297,7 @@ int ftrace_int3_handler(struct pt_regs *regs) static int ftrace_write(unsigned long ip, const char *val, int size) { - /* - * On x86_64, kernel text mappings are mapped read-only with - * CONFIG_DEBUG_RODATA. So we use the kernel identity mapping instead - * of the kernel text mapping to modify the kernel text. - * - * For 32bit kernels, these mappings are same and we can use - * kernel identity mapping to modify code. - */ - if (within(ip, (unsigned long)_text, (unsigned long)_etext)) - ip = (unsigned long)__va(__pa_symbol(ip)); + ip = text_ip_addr(ip); if (probe_kernel_write((void *)ip, val, size)) return -EPERM; @@ -349,40 +340,14 @@ static int add_brk_on_nop(struct dyn_ftrace *rec) return add_break(rec->ip, old); } -/* - * If the record has the FTRACE_FL_REGS set, that means that it - * wants to convert to a callback that saves all regs. If FTRACE_FL_REGS - * is not not set, then it wants to convert to the normal callback. - */ -static unsigned long get_ftrace_addr(struct dyn_ftrace *rec) -{ - if (rec->flags & FTRACE_FL_REGS) - return (unsigned long)FTRACE_REGS_ADDR; - else - return (unsigned long)FTRACE_ADDR; -} - -/* - * The FTRACE_FL_REGS_EN is set when the record already points to - * a function that saves all the regs. Basically the '_EN' version - * represents the current state of the function. - */ -static unsigned long get_ftrace_old_addr(struct dyn_ftrace *rec) -{ - if (rec->flags & FTRACE_FL_REGS_EN) - return (unsigned long)FTRACE_REGS_ADDR; - else - return (unsigned long)FTRACE_ADDR; -} - static int add_breakpoints(struct dyn_ftrace *rec, int enable) { unsigned long ftrace_addr; int ret; - ret = ftrace_test_record(rec, enable); + ftrace_addr = ftrace_get_addr_curr(rec); - ftrace_addr = get_ftrace_addr(rec); + ret = ftrace_test_record(rec, enable); switch (ret) { case FTRACE_UPDATE_IGNORE: @@ -392,10 +357,7 @@ static int add_breakpoints(struct dyn_ftrace *rec, int enable) /* converting nop to call */ return add_brk_on_nop(rec); - case FTRACE_UPDATE_MODIFY_CALL_REGS: case FTRACE_UPDATE_MODIFY_CALL: - ftrace_addr = get_ftrace_old_addr(rec); - /* fall through */ case FTRACE_UPDATE_MAKE_NOP: /* converting a call to a nop */ return add_brk_on_call(rec, ftrace_addr); @@ -440,14 +402,14 @@ static int remove_breakpoint(struct dyn_ftrace *rec) * If not, don't touch the breakpoint, we make just create * a disaster. */ - ftrace_addr = get_ftrace_addr(rec); + ftrace_addr = ftrace_get_addr_new(rec); nop = ftrace_call_replace(ip, ftrace_addr); if (memcmp(&ins[1], &nop[1], MCOUNT_INSN_SIZE - 1) == 0) goto update; /* Check both ftrace_addr and ftrace_old_addr */ - ftrace_addr = get_ftrace_old_addr(rec); + ftrace_addr = ftrace_get_addr_curr(rec); nop = ftrace_call_replace(ip, ftrace_addr); if (memcmp(&ins[1], &nop[1], MCOUNT_INSN_SIZE - 1) != 0) @@ -491,13 +453,12 @@ static int add_update(struct dyn_ftrace *rec, int enable) ret = ftrace_test_record(rec, enable); - ftrace_addr = get_ftrace_addr(rec); + ftrace_addr = ftrace_get_addr_new(rec); switch (ret) { case FTRACE_UPDATE_IGNORE: return 0; - case FTRACE_UPDATE_MODIFY_CALL_REGS: case FTRACE_UPDATE_MODIFY_CALL: case FTRACE_UPDATE_MAKE_CALL: /* converting nop to call */ @@ -538,13 +499,12 @@ static int finish_update(struct dyn_ftrace *rec, int enable) ret = ftrace_update_record(rec, enable); - ftrace_addr = get_ftrace_addr(rec); + ftrace_addr = ftrace_get_addr_new(rec); switch (ret) { case FTRACE_UPDATE_IGNORE: return 0; - case FTRACE_UPDATE_MODIFY_CALL_REGS: case FTRACE_UPDATE_MODIFY_CALL: case FTRACE_UPDATE_MAKE_CALL: /* converting nop to call */ @@ -621,8 +581,8 @@ void ftrace_replace_code(int enable) return; remove_breakpoints: + pr_warn("Failed on %s (%d):\n", report, count); ftrace_bug(ret, rec ? rec->ip : 0); - printk(KERN_WARNING "Failed on %s (%d):\n", report, count); for_ftrace_rec_iter(iter) { rec = ftrace_rec_iter_record(iter); /* diff --git a/arch/x86/kernel/mcount_64.S b/arch/x86/kernel/mcount_64.S new file mode 100644 index 000000000000..c050a0153168 --- /dev/null +++ b/arch/x86/kernel/mcount_64.S @@ -0,0 +1,217 @@ +/* + * linux/arch/x86_64/mcount_64.S + * + * Copyright (C) 2014 Steven Rostedt, Red Hat Inc + */ + +#include <linux/linkage.h> +#include <asm/ptrace.h> +#include <asm/ftrace.h> + + + .code64 + .section .entry.text, "ax" + + +#ifdef CONFIG_FUNCTION_TRACER + +#ifdef CC_USING_FENTRY +# define function_hook __fentry__ +#else +# define function_hook mcount +#endif + +#ifdef CONFIG_DYNAMIC_FTRACE + +ENTRY(function_hook) + retq +END(function_hook) + +/* skip is set if stack has been adjusted */ +.macro ftrace_caller_setup skip=0 + MCOUNT_SAVE_FRAME \skip + + /* Load the ftrace_ops into the 3rd parameter */ + movq function_trace_op(%rip), %rdx + + /* Load ip into the first parameter */ + movq RIP(%rsp), %rdi + subq $MCOUNT_INSN_SIZE, %rdi + /* Load the parent_ip into the second parameter */ +#ifdef CC_USING_FENTRY + movq SS+16(%rsp), %rsi +#else + movq 8(%rbp), %rsi +#endif +.endm + +ENTRY(ftrace_caller) + /* Check if tracing was disabled (quick check) */ + cmpl $0, function_trace_stop + jne ftrace_stub + + ftrace_caller_setup + /* regs go into 4th parameter (but make it NULL) */ + movq $0, %rcx + +GLOBAL(ftrace_call) + call ftrace_stub + + MCOUNT_RESTORE_FRAME +ftrace_return: + +#ifdef CONFIG_FUNCTION_GRAPH_TRACER +GLOBAL(ftrace_graph_call) + jmp ftrace_stub +#endif + +GLOBAL(ftrace_stub) + retq +END(ftrace_caller) + +ENTRY(ftrace_regs_caller) + /* Save the current flags before compare (in SS location)*/ + pushfq + + /* Check if tracing was disabled (quick check) */ + cmpl $0, function_trace_stop + jne ftrace_restore_flags + + /* skip=8 to skip flags saved in SS */ + ftrace_caller_setup 8 + + /* Save the rest of pt_regs */ + movq %r15, R15(%rsp) + movq %r14, R14(%rsp) + movq %r13, R13(%rsp) + movq %r12, R12(%rsp) + movq %r11, R11(%rsp) + movq %r10, R10(%rsp) + movq %rbp, RBP(%rsp) + movq %rbx, RBX(%rsp) + /* Copy saved flags */ + movq SS(%rsp), %rcx + movq %rcx, EFLAGS(%rsp) + /* Kernel segments */ + movq $__KERNEL_DS, %rcx + movq %rcx, SS(%rsp) + movq $__KERNEL_CS, %rcx + movq %rcx, CS(%rsp) + /* Stack - skipping return address */ + leaq SS+16(%rsp), %rcx + movq %rcx, RSP(%rsp) + + /* regs go into 4th parameter */ + leaq (%rsp), %rcx + +GLOBAL(ftrace_regs_call) + call ftrace_stub + + /* Copy flags back to SS, to restore them */ + movq EFLAGS(%rsp), %rax + movq %rax, SS(%rsp) + + /* Handlers can change the RIP */ + movq RIP(%rsp), %rax + movq %rax, SS+8(%rsp) + + /* restore the rest of pt_regs */ + movq R15(%rsp), %r15 + movq R14(%rsp), %r14 + movq R13(%rsp), %r13 + movq R12(%rsp), %r12 + movq R10(%rsp), %r10 + movq RBP(%rsp), %rbp + movq RBX(%rsp), %rbx + + /* skip=8 to skip flags saved in SS */ + MCOUNT_RESTORE_FRAME 8 + + /* Restore flags */ + popfq + + jmp ftrace_return +ftrace_restore_flags: + popfq + jmp ftrace_stub + +END(ftrace_regs_caller) + + +#else /* ! CONFIG_DYNAMIC_FTRACE */ + +ENTRY(function_hook) + cmpl $0, function_trace_stop + jne ftrace_stub + + cmpq $ftrace_stub, ftrace_trace_function + jnz trace + +#ifdef CONFIG_FUNCTION_GRAPH_TRACER + cmpq $ftrace_stub, ftrace_graph_return + jnz ftrace_graph_caller + + cmpq $ftrace_graph_entry_stub, ftrace_graph_entry + jnz ftrace_graph_caller +#endif + +GLOBAL(ftrace_stub) + retq + +trace: + MCOUNT_SAVE_FRAME + + movq RIP(%rsp), %rdi +#ifdef CC_USING_FENTRY + movq SS+16(%rsp), %rsi +#else + movq 8(%rbp), %rsi +#endif + subq $MCOUNT_INSN_SIZE, %rdi + + call *ftrace_trace_function + + MCOUNT_RESTORE_FRAME + + jmp ftrace_stub +END(function_hook) +#endif /* CONFIG_DYNAMIC_FTRACE */ +#endif /* CONFIG_FUNCTION_TRACER */ + +#ifdef CONFIG_FUNCTION_GRAPH_TRACER +ENTRY(ftrace_graph_caller) + MCOUNT_SAVE_FRAME + +#ifdef CC_USING_FENTRY + leaq SS+16(%rsp), %rdi + movq $0, %rdx /* No framepointers needed */ +#else + leaq 8(%rbp), %rdi + movq (%rbp), %rdx +#endif + movq RIP(%rsp), %rsi + subq $MCOUNT_INSN_SIZE, %rsi + + call prepare_ftrace_return + + MCOUNT_RESTORE_FRAME + + retq +END(ftrace_graph_caller) + +GLOBAL(return_to_handler) + subq $24, %rsp + + /* Save the return values */ + movq %rax, (%rsp) + movq %rdx, 8(%rsp) + movq %rbp, %rdi + + call ftrace_return_to_handler + + movq %rax, %rdi + movq 8(%rsp), %rdx + movq (%rsp), %rax + addq $24, %rsp + jmp *%rdi +#endif diff --git a/block/bio.c b/block/bio.c index 96d28eee8a1e..1ba33657160f 100644 --- a/block/bio.c +++ b/block/bio.c @@ -1971,7 +1971,7 @@ int bio_associate_current(struct bio *bio) /* associate blkcg if exists */ rcu_read_lock(); css = task_css(current, blkio_cgrp_id); - if (css && css_tryget(css)) + if (css && css_tryget_online(css)) bio->bi_css = css; rcu_read_unlock(); diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c index 1039fb9ff5f5..9f5bce33e6fe 100644 --- a/block/blk-cgroup.c +++ b/block/blk-cgroup.c @@ -185,7 +185,7 @@ static struct blkcg_gq *blkg_create(struct blkcg *blkcg, lockdep_assert_held(q->queue_lock); /* blkg holds a reference to blkcg */ - if (!css_tryget(&blkcg->css)) { + if (!css_tryget_online(&blkcg->css)) { ret = -EINVAL; goto err_free_blkg; } diff --git a/block/blk-cgroup.h b/block/blk-cgroup.h index 371fe8e92ab5..d692b29c083a 100644 --- a/block/blk-cgroup.h +++ b/block/blk-cgroup.h @@ -204,7 +204,7 @@ static inline struct blkcg *bio_blkcg(struct bio *bio) */ static inline struct blkcg *blkcg_parent(struct blkcg *blkcg) { - return css_to_blkcg(css_parent(&blkcg->css)); + return css_to_blkcg(blkcg->css.parent); } /** diff --git a/block/blk-throttle.c b/block/blk-throttle.c index 9353b4683359..3fdb21a390c1 100644 --- a/block/blk-throttle.c +++ b/block/blk-throttle.c @@ -1346,10 +1346,10 @@ static int tg_print_conf_uint(struct seq_file *sf, void *v) return 0; } -static int tg_set_conf(struct cgroup_subsys_state *css, struct cftype *cft, - const char *buf, bool is_u64) +static ssize_t tg_set_conf(struct kernfs_open_file *of, + char *buf, size_t nbytes, loff_t off, bool is_u64) { - struct blkcg *blkcg = css_to_blkcg(css); + struct blkcg *blkcg = css_to_blkcg(of_css(of)); struct blkg_conf_ctx ctx; struct throtl_grp *tg; struct throtl_service_queue *sq; @@ -1368,9 +1368,9 @@ static int tg_set_conf(struct cgroup_subsys_state *css, struct cftype *cft, ctx.v = -1; if (is_u64) - *(u64 *)((void *)tg + cft->private) = ctx.v; + *(u64 *)((void *)tg + of_cft(of)->private) = ctx.v; else - *(unsigned int *)((void *)tg + cft->private) = ctx.v; + *(unsigned int *)((void *)tg + of_cft(of)->private) = ctx.v; throtl_log(&tg->service_queue, "limit change rbps=%llu wbps=%llu riops=%u wiops=%u", @@ -1404,19 +1404,19 @@ static int tg_set_conf(struct cgroup_subsys_state *css, struct cftype *cft, } blkg_conf_finish(&ctx); - return 0; + return nbytes; } -static int tg_set_conf_u64(struct cgroup_subsys_state *css, struct cftype *cft, - char *buf) +static ssize_t tg_set_conf_u64(struct kernfs_open_file *of, + char *buf, size_t nbytes, loff_t off) { - return tg_set_conf(css, cft, buf, true); + return tg_set_conf(of, buf, nbytes, off, true); } -static int tg_set_conf_uint(struct cgroup_subsys_state *css, struct cftype *cft, - char *buf) +static ssize_t tg_set_conf_uint(struct kernfs_open_file *of, + char *buf, size_t nbytes, loff_t off) { - return tg_set_conf(css, cft, buf, false); + return tg_set_conf(of, buf, nbytes, off, false); } static struct cftype throtl_files[] = { @@ -1424,25 +1424,25 @@ static struct cftype throtl_files[] = { .name = "throttle.read_bps_device", .private = offsetof(struct throtl_grp, bps[READ]), .seq_show = tg_print_conf_u64, - .write_string = tg_set_conf_u64, + .write = tg_set_conf_u64, }, { .name = "throttle.write_bps_device", .private = offsetof(struct throtl_grp, bps[WRITE]), .seq_show = tg_print_conf_u64, - .write_string = tg_set_conf_u64, + .write = tg_set_conf_u64, }, { .name = "throttle.read_iops_device", .private = offsetof(struct throtl_grp, iops[READ]), .seq_show = tg_print_conf_uint, - .write_string = tg_set_conf_uint, + .write = tg_set_conf_uint, }, { .name = "throttle.write_iops_device", .private = offsetof(struct throtl_grp, iops[WRITE]), .seq_show = tg_print_conf_uint, - .write_string = tg_set_conf_uint, + .write = tg_set_conf_uint, }, { .name = "throttle.io_service_bytes", diff --git a/block/cfq-iosched.c b/block/cfq-iosched.c index 22dffebc7c73..cadc37841744 100644 --- a/block/cfq-iosched.c +++ b/block/cfq-iosched.c @@ -1670,11 +1670,11 @@ static int cfq_print_leaf_weight(struct seq_file *sf, void *v) return 0; } -static int __cfqg_set_weight_device(struct cgroup_subsys_state *css, - struct cftype *cft, const char *buf, - bool is_leaf_weight) +static ssize_t __cfqg_set_weight_device(struct kernfs_open_file *of, + char *buf, size_t nbytes, loff_t off, + bool is_leaf_weight) { - struct blkcg *blkcg = css_to_blkcg(css); + struct blkcg *blkcg = css_to_blkcg(of_css(of)); struct blkg_conf_ctx ctx; struct cfq_group *cfqg; int ret; @@ -1697,19 +1697,19 @@ static int __cfqg_set_weight_device(struct cgroup_subsys_state *css, } blkg_conf_finish(&ctx); - return ret; + return ret ?: nbytes; } -static int cfqg_set_weight_device(struct cgroup_subsys_state *css, - struct cftype *cft, char *buf) +static ssize_t cfqg_set_weight_device(struct kernfs_open_file *of, + char *buf, size_t nbytes, loff_t off) { - return __cfqg_set_weight_device(css, cft, buf, false); + return __cfqg_set_weight_device(of, buf, nbytes, off, false); } -static int cfqg_set_leaf_weight_device(struct cgroup_subsys_state *css, - struct cftype *cft, char *buf) +static ssize_t cfqg_set_leaf_weight_device(struct kernfs_open_file *of, + char *buf, size_t nbytes, loff_t off) { - return __cfqg_set_weight_device(css, cft, buf, true); + return __cfqg_set_weight_device(of, buf, nbytes, off, true); } static int __cfq_set_weight(struct cgroup_subsys_state *css, struct cftype *cft, @@ -1837,7 +1837,7 @@ static struct cftype cfq_blkcg_files[] = { .name = "weight_device", .flags = CFTYPE_ONLY_ON_ROOT, .seq_show = cfqg_print_leaf_weight_device, - .write_string = cfqg_set_leaf_weight_device, + .write = cfqg_set_leaf_weight_device, }, { .name = "weight", @@ -1851,7 +1851,7 @@ static struct cftype cfq_blkcg_files[] = { .name = "weight_device", .flags = CFTYPE_NOT_ON_ROOT, .seq_show = cfqg_print_weight_device, - .write_string = cfqg_set_weight_device, + .write = cfqg_set_weight_device, }, { .name = "weight", @@ -1863,7 +1863,7 @@ static struct cftype cfq_blkcg_files[] = { { .name = "leaf_weight_device", .seq_show = cfqg_print_leaf_weight_device, - .write_string = cfqg_set_leaf_weight_device, + .write = cfqg_set_leaf_weight_device, }, { .name = "leaf_weight", diff --git a/drivers/ata/Kconfig b/drivers/ata/Kconfig index 0033fafc470b..7671dbac6015 100644 --- a/drivers/ata/Kconfig +++ b/drivers/ata/Kconfig @@ -123,6 +123,15 @@ config AHCI_IMX If unsure, say N. +config AHCI_MVEBU + tristate "Marvell EBU AHCI SATA support" + depends on ARCH_MVEBU + help + This option enables support for the Marvebu EBU SoC's + onboard AHCI SATA. + + If unsure, say N. + config AHCI_SUNXI tristate "Allwinner sunxi AHCI SATA support" depends on ARCH_SUNXI diff --git a/drivers/ata/Makefile b/drivers/ata/Makefile index 44c8016e565c..5a02aeecef5b 100644 --- a/drivers/ata/Makefile +++ b/drivers/ata/Makefile @@ -12,6 +12,7 @@ obj-$(CONFIG_SATA_DWC) += sata_dwc_460ex.o obj-$(CONFIG_SATA_HIGHBANK) += sata_highbank.o libahci.o obj-$(CONFIG_AHCI_DA850) += ahci_da850.o libahci.o libahci_platform.o obj-$(CONFIG_AHCI_IMX) += ahci_imx.o libahci.o libahci_platform.o +obj-$(CONFIG_AHCI_MVEBU) += ahci_mvebu.o libahci.o libahci_platform.o obj-$(CONFIG_AHCI_SUNXI) += ahci_sunxi.o libahci.o libahci_platform.o obj-$(CONFIG_AHCI_ST) += ahci_st.o libahci.o libahci_platform.o obj-$(CONFIG_AHCI_XGENE) += ahci_xgene.o libahci.o libahci_platform.o diff --git a/drivers/ata/acard-ahci.c b/drivers/ata/acard-ahci.c index b51605ac5974..0cd7c7a39e5b 100644 --- a/drivers/ata/acard-ahci.c +++ b/drivers/ata/acard-ahci.c @@ -77,7 +77,7 @@ static bool acard_ahci_qc_fill_rtf(struct ata_queued_cmd *qc); static int acard_ahci_port_start(struct ata_port *ap); static int acard_ahci_init_one(struct pci_dev *pdev, const struct pci_device_id *ent); -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP static int acard_ahci_pci_device_suspend(struct pci_dev *pdev, pm_message_t mesg); static int acard_ahci_pci_device_resume(struct pci_dev *pdev); #endif @@ -118,13 +118,13 @@ static struct pci_driver acard_ahci_pci_driver = { .id_table = acard_ahci_pci_tbl, .probe = acard_ahci_init_one, .remove = ata_pci_remove_one, -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP .suspend = acard_ahci_pci_device_suspend, .resume = acard_ahci_pci_device_resume, #endif }; -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP static int acard_ahci_pci_device_suspend(struct pci_dev *pdev, pm_message_t mesg) { struct ata_host *host = pci_get_drvdata(pdev); diff --git a/drivers/ata/ahci.c b/drivers/ata/ahci.c index 60707814a84b..dae5607e1115 100644 --- a/drivers/ata/ahci.c +++ b/drivers/ata/ahci.c @@ -445,10 +445,14 @@ static const struct pci_device_id ahci_pci_tbl[] = { .driver_data = board_ahci_yes_fbs }, /* 88se9172 */ { PCI_DEVICE(PCI_VENDOR_ID_MARVELL_EXT, 0x9192), .driver_data = board_ahci_yes_fbs }, /* 88se9172 on some Gigabyte */ + { PCI_DEVICE(PCI_VENDOR_ID_MARVELL_EXT, 0x91a0), + .driver_data = board_ahci_yes_fbs }, { PCI_DEVICE(PCI_VENDOR_ID_MARVELL_EXT, 0x91a3), .driver_data = board_ahci_yes_fbs }, { PCI_DEVICE(PCI_VENDOR_ID_MARVELL_EXT, 0x9230), .driver_data = board_ahci_yes_fbs }, + { PCI_DEVICE(PCI_VENDOR_ID_TTI, 0x0642), + .driver_data = board_ahci_yes_fbs }, /* Promise */ { PCI_VDEVICE(PROMISE, 0x3f20), board_ahci }, /* PDC42819 */ diff --git a/drivers/ata/ahci.h b/drivers/ata/ahci.h index af63c75c2001..05882e4445a6 100644 --- a/drivers/ata/ahci.h +++ b/drivers/ata/ahci.h @@ -237,6 +237,7 @@ enum { error-handling stage) */ AHCI_HFLAG_MULTI_MSI = (1 << 16), /* multiple PCI MSIs */ AHCI_HFLAG_NO_DEVSLP = (1 << 17), /* no device sleep */ + AHCI_HFLAG_NO_FBS = (1 << 18), /* no FBS */ /* ap->flags bits */ diff --git a/drivers/ata/ahci_da850.c b/drivers/ata/ahci_da850.c index 2c83613ce2db..2b77d53bccf8 100644 --- a/drivers/ata/ahci_da850.c +++ b/drivers/ata/ahci_da850.c @@ -85,7 +85,8 @@ static int ahci_da850_probe(struct platform_device *pdev) da850_sata_init(dev, pwrdn_reg, hpriv->mmio); - rc = ahci_platform_init_host(pdev, hpriv, &ahci_da850_port_info, 0, 0); + rc = ahci_platform_init_host(pdev, hpriv, &ahci_da850_port_info, + 0, 0, 0); if (rc) goto disable_resources; diff --git a/drivers/ata/ahci_imx.c b/drivers/ata/ahci_imx.c index 8befeb69eeb1..3a901520c62b 100644 --- a/drivers/ata/ahci_imx.c +++ b/drivers/ata/ahci_imx.c @@ -432,7 +432,8 @@ static int imx_ahci_probe(struct platform_device *pdev) reg_val = clk_get_rate(imxpriv->ahb_clk) / 1000; writel(reg_val, hpriv->mmio + IMX_TIMER1MS); - ret = ahci_platform_init_host(pdev, hpriv, &ahci_imx_port_info, 0, 0); + ret = ahci_platform_init_host(pdev, hpriv, &ahci_imx_port_info, + 0, 0, 0); if (ret) imx_sata_disable(hpriv); diff --git a/drivers/ata/ahci_mvebu.c b/drivers/ata/ahci_mvebu.c new file mode 100644 index 000000000000..fd3dfd733b84 --- /dev/null +++ b/drivers/ata/ahci_mvebu.c @@ -0,0 +1,128 @@ +/* + * AHCI glue platform driver for Marvell EBU SOCs + * + * Copyright (C) 2014 Marvell + * + * Thomas Petazzoni <thomas.petazzoni@free-electrons.com> + * Marcin Wojtas <mw@semihalf.com> + * + * This file is licensed under the terms of the GNU General Public + * License version 2. This program is licensed "as is" without any + * warranty of any kind, whether express or implied. + */ + +#include <linux/ahci_platform.h> +#include <linux/kernel.h> +#include <linux/mbus.h> +#include <linux/module.h> +#include <linux/of_device.h> +#include <linux/platform_device.h> +#include "ahci.h" + +#define AHCI_VENDOR_SPECIFIC_0_ADDR 0xa0 +#define AHCI_VENDOR_SPECIFIC_0_DATA 0xa4 + +#define AHCI_WINDOW_CTRL(win) (0x60 + ((win) << 4)) +#define AHCI_WINDOW_BASE(win) (0x64 + ((win) << 4)) +#define AHCI_WINDOW_SIZE(win) (0x68 + ((win) << 4)) + +static void ahci_mvebu_mbus_config(struct ahci_host_priv *hpriv, + const struct mbus_dram_target_info *dram) +{ + int i; + + for (i = 0; i < 4; i++) { + writel(0, hpriv->mmio + AHCI_WINDOW_CTRL(i)); + writel(0, hpriv->mmio + AHCI_WINDOW_BASE(i)); + writel(0, hpriv->mmio + AHCI_WINDOW_SIZE(i)); + } + + for (i = 0; i < dram->num_cs; i++) { + const struct mbus_dram_window *cs = dram->cs + i; + + writel((cs->mbus_attr << 8) | + (dram->mbus_dram_target_id << 4) | 1, + hpriv->mmio + AHCI_WINDOW_CTRL(i)); + writel(cs->base, hpriv->mmio + AHCI_WINDOW_BASE(i)); + writel(((cs->size - 1) & 0xffff0000), + hpriv->mmio + AHCI_WINDOW_SIZE(i)); + } +} + +static void ahci_mvebu_regret_option(struct ahci_host_priv *hpriv) +{ + /* + * Enable the regret bit to allow the SATA unit to regret a + * request that didn't receive an acknowlegde and avoid a + * deadlock + */ + writel(0x4, hpriv->mmio + AHCI_VENDOR_SPECIFIC_0_ADDR); + writel(0x80, hpriv->mmio + AHCI_VENDOR_SPECIFIC_0_DATA); +} + +static const struct ata_port_info ahci_mvebu_port_info = { + .flags = AHCI_FLAG_COMMON, + .pio_mask = ATA_PIO4, + .udma_mask = ATA_UDMA6, + .port_ops = &ahci_platform_ops, +}; + +static int ahci_mvebu_probe(struct platform_device *pdev) +{ + struct ahci_host_priv *hpriv; + const struct mbus_dram_target_info *dram; + int rc; + + hpriv = ahci_platform_get_resources(pdev); + if (IS_ERR(hpriv)) + return PTR_ERR(hpriv); + + rc = ahci_platform_enable_resources(hpriv); + if (rc) + return rc; + + dram = mv_mbus_dram_info(); + if (!dram) + return -ENODEV; + + ahci_mvebu_mbus_config(hpriv, dram); + ahci_mvebu_regret_option(hpriv); + + rc = ahci_platform_init_host(pdev, hpriv, &ahci_mvebu_port_info, + 0, 0, 0); + if (rc) + goto disable_resources; + + return 0; + +disable_resources: + ahci_platform_disable_resources(hpriv); + return rc; +} + +static const struct of_device_id ahci_mvebu_of_match[] = { + { .compatible = "marvell,armada-380-ahci", }, + { }, +}; +MODULE_DEVICE_TABLE(of, ahci_mvebu_of_match); + +/* + * We currently don't provide power management related operations, + * since there is no suspend/resume support at the platform level for + * Armada 38x for the moment. + */ +static struct platform_driver ahci_mvebu_driver = { + .probe = ahci_mvebu_probe, + .remove = ata_platform_remove_one, + .driver = { + .name = "ahci-mvebu", + .owner = THIS_MODULE, + .of_match_table = ahci_mvebu_of_match, + }, +}; +module_platform_driver(ahci_mvebu_driver); + +MODULE_DESCRIPTION("Marvell EBU AHCI SATA driver"); +MODULE_AUTHOR("Thomas Petazzoni <thomas.petazzoni@free-electrons.com>, Marcin Wojtas <mw@semihalf.com>"); +MODULE_LICENSE("GPL"); +MODULE_ALIAS("platform:ahci_mvebu"); diff --git a/drivers/ata/ahci_platform.c b/drivers/ata/ahci_platform.c index ef67e79944f9..ebe505c17763 100644 --- a/drivers/ata/ahci_platform.c +++ b/drivers/ata/ahci_platform.c @@ -16,6 +16,7 @@ #include <linux/module.h> #include <linux/pm.h> #include <linux/device.h> +#include <linux/of_device.h> #include <linux/platform_device.h> #include <linux/libata.h> #include <linux/ahci_platform.h> @@ -33,6 +34,7 @@ static int ahci_probe(struct platform_device *pdev) struct device *dev = &pdev->dev; struct ahci_platform_data *pdata = dev_get_platdata(dev); struct ahci_host_priv *hpriv; + unsigned long hflags = 0; int rc; hpriv = ahci_platform_get_resources(pdev); @@ -55,7 +57,11 @@ static int ahci_probe(struct platform_device *pdev) goto disable_resources; } - rc = ahci_platform_init_host(pdev, hpriv, &ahci_port_info, 0, 0); + if (of_device_is_compatible(dev->of_node, "hisilicon,hisi-ahci")) + hflags |= AHCI_HFLAG_NO_FBS; + + rc = ahci_platform_init_host(pdev, hpriv, &ahci_port_info, + hflags, 0, 0); if (rc) goto pdata_exit; @@ -76,6 +82,7 @@ static const struct of_device_id ahci_of_match[] = { { .compatible = "snps,exynos5440-ahci", }, { .compatible = "ibm,476gtr-ahci", }, { .compatible = "snps,dwc-ahci", }, + { .compatible = "hisilicon,hisi-ahci", }, {}, }; MODULE_DEVICE_TABLE(of, ahci_of_match); diff --git a/drivers/ata/ahci_st.c b/drivers/ata/ahci_st.c index 633222226c19..2595598df9ce 100644 --- a/drivers/ata/ahci_st.c +++ b/drivers/ata/ahci_st.c @@ -166,7 +166,7 @@ static int st_ahci_probe(struct platform_device *pdev) if (err) return err; - err = ahci_platform_init_host(pdev, hpriv, &st_ahci_port_info, 0, 0); + err = ahci_platform_init_host(pdev, hpriv, &st_ahci_port_info, 0, 0, 0); if (err) { ahci_platform_disable_resources(hpriv); return err; diff --git a/drivers/ata/ahci_sunxi.c b/drivers/ata/ahci_sunxi.c index 42d3f64e74b3..02002f125bd4 100644 --- a/drivers/ata/ahci_sunxi.c +++ b/drivers/ata/ahci_sunxi.c @@ -157,8 +157,6 @@ static void ahci_sunxi_start_engine(struct ata_port *ap) } static const struct ata_port_info ahci_sunxi_port_info = { - AHCI_HFLAGS(AHCI_HFLAG_32BIT_ONLY | AHCI_HFLAG_NO_MSI | - AHCI_HFLAG_NO_PMP | AHCI_HFLAG_YES_NCQ), .flags = AHCI_FLAG_COMMON | ATA_FLAG_NCQ, .pio_mask = ATA_PIO4, .udma_mask = ATA_UDMA6, @@ -169,6 +167,7 @@ static int ahci_sunxi_probe(struct platform_device *pdev) { struct device *dev = &pdev->dev; struct ahci_host_priv *hpriv; + unsigned long hflags; int rc; hpriv = ahci_platform_get_resources(pdev); @@ -185,7 +184,11 @@ static int ahci_sunxi_probe(struct platform_device *pdev) if (rc) goto disable_resources; - rc = ahci_platform_init_host(pdev, hpriv, &ahci_sunxi_port_info, 0, 0); + hflags = AHCI_HFLAG_32BIT_ONLY | AHCI_HFLAG_NO_MSI | + AHCI_HFLAG_NO_PMP | AHCI_HFLAG_YES_NCQ; + + rc = ahci_platform_init_host(pdev, hpriv, &ahci_sunxi_port_info, + hflags, 0, 0); if (rc) goto disable_resources; diff --git a/drivers/ata/ahci_xgene.c b/drivers/ata/ahci_xgene.c index 77c89bf171f1..042a9bb45c86 100644 --- a/drivers/ata/ahci_xgene.c +++ b/drivers/ata/ahci_xgene.c @@ -303,7 +303,6 @@ static struct ata_port_operations xgene_ahci_ops = { }; static const struct ata_port_info xgene_ahci_port_info = { - AHCI_HFLAGS(AHCI_HFLAG_NO_PMP | AHCI_HFLAG_YES_NCQ), .flags = AHCI_FLAG_COMMON | ATA_FLAG_NCQ, .pio_mask = ATA_PIO4, .udma_mask = ATA_UDMA6, @@ -382,6 +381,7 @@ static int xgene_ahci_probe(struct platform_device *pdev) struct ahci_host_priv *hpriv; struct xgene_ahci_context *ctx; struct resource *res; + unsigned long hflags; int rc; hpriv = ahci_platform_get_resources(pdev); @@ -450,7 +450,10 @@ static int xgene_ahci_probe(struct platform_device *pdev) goto disable_resources; } - rc = ahci_platform_init_host(pdev, hpriv, &xgene_ahci_port_info, 0, 0); + hflags = AHCI_HFLAG_NO_PMP | AHCI_HFLAG_YES_NCQ; + + rc = ahci_platform_init_host(pdev, hpriv, &xgene_ahci_port_info, + hflags, 0, 0); if (rc) goto disable_resources; diff --git a/drivers/ata/ata_generic.c b/drivers/ata/ata_generic.c index 9498a7d3846f..9ff545ce8da3 100644 --- a/drivers/ata/ata_generic.c +++ b/drivers/ata/ata_generic.c @@ -241,7 +241,7 @@ static struct pci_driver ata_generic_pci_driver = { .id_table = ata_generic, .probe = ata_generic_init_one, .remove = ata_pci_remove_one, -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP .suspend = ata_pci_device_suspend, .resume = ata_pci_device_resume, #endif diff --git a/drivers/ata/ata_piix.c b/drivers/ata/ata_piix.c index 6334c8d7c3f1..893e30e9a9ef 100644 --- a/drivers/ata/ata_piix.c +++ b/drivers/ata/ata_piix.c @@ -830,7 +830,7 @@ static bool piix_irq_check(struct ata_port *ap) return ap->ops->bmdma_status(ap) & ATA_DMA_INTR; } -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP static int piix_broken_suspend(void) { static const struct dmi_system_id sysids[] = { @@ -1767,7 +1767,7 @@ static struct pci_driver piix_pci_driver = { .id_table = piix_pci_tbl, .probe = piix_init_one, .remove = piix_remove_one, -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP .suspend = piix_pci_device_suspend, .resume = piix_pci_device_resume, #endif diff --git a/drivers/ata/libahci.c b/drivers/ata/libahci.c index b9861453fc81..40ea583d3610 100644 --- a/drivers/ata/libahci.c +++ b/drivers/ata/libahci.c @@ -464,6 +464,11 @@ void ahci_save_initial_config(struct device *dev, cap |= HOST_CAP_FBS; } + if ((cap & HOST_CAP_FBS) && (hpriv->flags & AHCI_HFLAG_NO_FBS)) { + dev_info(dev, "controller can't do FBS, turning off CAP_FBS\n"); + cap &= ~HOST_CAP_FBS; + } + if (force_port_map && port_map != force_port_map) { dev_info(dev, "forcing port_map 0x%x -> 0x%x\n", port_map, force_port_map); diff --git a/drivers/ata/libahci_platform.c b/drivers/ata/libahci_platform.c index 7cb3a85719c0..3a5b4ed25a4f 100644 --- a/drivers/ata/libahci_platform.c +++ b/drivers/ata/libahci_platform.c @@ -283,6 +283,7 @@ EXPORT_SYMBOL_GPL(ahci_platform_get_resources); * @pdev: platform device pointer for the host * @hpriv: ahci-host private data for the host * @pi_template: template for the ata_port_info to use + * @host_flags: ahci host flags used in ahci_host_priv * @force_port_map: param passed to ahci_save_initial_config * @mask_port_map: param passed to ahci_save_initial_config * @@ -296,6 +297,7 @@ EXPORT_SYMBOL_GPL(ahci_platform_get_resources); int ahci_platform_init_host(struct platform_device *pdev, struct ahci_host_priv *hpriv, const struct ata_port_info *pi_template, + unsigned long host_flags, unsigned int force_port_map, unsigned int mask_port_map) { @@ -312,7 +314,8 @@ int ahci_platform_init_host(struct platform_device *pdev, } /* prepare host */ - hpriv->flags |= (unsigned long)pi.private_data; + pi.private_data = (void *)host_flags; + hpriv->flags |= host_flags; ahci_save_initial_config(dev, hpriv, force_port_map, mask_port_map); diff --git a/drivers/ata/libata-scsi.c b/drivers/ata/libata-scsi.c index ef8567de6a75..72691fd93948 100644 --- a/drivers/ata/libata-scsi.c +++ b/drivers/ata/libata-scsi.c @@ -1993,7 +1993,11 @@ static unsigned int ata_scsiop_inq_std(struct ata_scsi_args *args, u8 *rbuf) memcpy(rbuf, hdr, sizeof(hdr)); memcpy(&rbuf[8], "ATA ", 8); ata_id_string(args->id, &rbuf[16], ATA_ID_PROD, 16); - ata_id_string(args->id, &rbuf[32], ATA_ID_FW_REV, 4); + + /* From SAT, use last 2 words from fw rev unless they are spaces */ + ata_id_string(args->id, &rbuf[32], ATA_ID_FW_REV + 2, 4); + if (strncmp(&rbuf[32], " ", 4) == 0) + ata_id_string(args->id, &rbuf[32], ATA_ID_FW_REV, 4); if (rbuf[32] == 0 || rbuf[32] == ' ') memcpy(&rbuf[32], "n/a ", 4); diff --git a/drivers/ata/libata-sff.c b/drivers/ata/libata-sff.c index b603720b877d..1121153f1ecd 100644 --- a/drivers/ata/libata-sff.c +++ b/drivers/ata/libata-sff.c @@ -2433,15 +2433,6 @@ int ata_pci_sff_activate_host(struct ata_host *host, mask = (1 << 2) | (1 << 0); if ((tmp8 & mask) != mask) legacy_mode = 1; -#if defined(CONFIG_NO_ATA_LEGACY) - /* Some platforms with PCI limits cannot address compat - port space. In that case we punt if their firmware has - left a device in compatibility mode */ - if (legacy_mode) { - printk(KERN_ERR "ata: Compatibility mode ATA is not supported on this platform, skipping.\n"); - return -EOPNOTSUPP; - } -#endif } if (!devres_open_group(dev, NULL, GFP_KERNEL)) diff --git a/drivers/ata/pata_acpi.c b/drivers/ata/pata_acpi.c index 5108b8744dce..b70fce2a38eb 100644 --- a/drivers/ata/pata_acpi.c +++ b/drivers/ata/pata_acpi.c @@ -265,7 +265,7 @@ static struct pci_driver pacpi_pci_driver = { .id_table = pacpi_pci_tbl, .probe = pacpi_init_one, .remove = ata_pci_remove_one, -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP .suspend = ata_pci_device_suspend, .resume = ata_pci_device_resume, #endif diff --git a/drivers/ata/pata_ali.c b/drivers/ata/pata_ali.c index 1b7b2ccabcff..d19cd88ed2d3 100644 --- a/drivers/ata/pata_ali.c +++ b/drivers/ata/pata_ali.c @@ -589,7 +589,7 @@ static int ali_init_one(struct pci_dev *pdev, const struct pci_device_id *id) return ata_pci_bmdma_init_one(pdev, ppi, &ali_sht, NULL, 0); } -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP static int ali_reinit_one(struct pci_dev *pdev) { struct ata_host *host = pci_get_drvdata(pdev); @@ -616,7 +616,7 @@ static struct pci_driver ali_pci_driver = { .id_table = ali, .probe = ali_init_one, .remove = ata_pci_remove_one, -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP .suspend = ata_pci_device_suspend, .resume = ali_reinit_one, #endif diff --git a/drivers/ata/pata_amd.c b/drivers/ata/pata_amd.c index 1206fa6b62ca..8d4d959a821c 100644 --- a/drivers/ata/pata_amd.c +++ b/drivers/ata/pata_amd.c @@ -574,7 +574,7 @@ static int amd_init_one(struct pci_dev *pdev, const struct pci_device_id *id) return ata_pci_bmdma_init_one(pdev, ppi, &amd_sht, hpriv, 0); } -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP static int amd_reinit_one(struct pci_dev *pdev) { struct ata_host *host = pci_get_drvdata(pdev); @@ -625,7 +625,7 @@ static struct pci_driver amd_pci_driver = { .id_table = amd, .probe = amd_init_one, .remove = ata_pci_remove_one, -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP .suspend = ata_pci_device_suspend, .resume = amd_reinit_one, #endif diff --git a/drivers/ata/pata_artop.c b/drivers/ata/pata_artop.c index 3aa4e655e3c6..96c05c9494fa 100644 --- a/drivers/ata/pata_artop.c +++ b/drivers/ata/pata_artop.c @@ -422,7 +422,7 @@ static const struct pci_device_id artop_pci_tbl[] = { { } /* terminate list */ }; -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP static int atp8xx_reinit_one(struct pci_dev *pdev) { struct ata_host *host = pci_get_drvdata(pdev); @@ -444,7 +444,7 @@ static struct pci_driver artop_pci_driver = { .id_table = artop_pci_tbl, .probe = artop_init_one, .remove = ata_pci_remove_one, -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP .suspend = ata_pci_device_suspend, .resume = atp8xx_reinit_one, #endif diff --git a/drivers/ata/pata_atiixp.c b/drivers/ata/pata_atiixp.c index 30fa4ca4cef6..970f7767e5fd 100644 --- a/drivers/ata/pata_atiixp.c +++ b/drivers/ata/pata_atiixp.c @@ -298,7 +298,7 @@ static struct pci_driver atiixp_pci_driver = { .id_table = atiixp, .probe = atiixp_init_one, .remove = ata_pci_remove_one, -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP .resume = ata_pci_device_resume, .suspend = ata_pci_device_suspend, #endif diff --git a/drivers/ata/pata_atp867x.c b/drivers/ata/pata_atp867x.c index 7e73a0f1e323..a705cfca90f7 100644 --- a/drivers/ata/pata_atp867x.c +++ b/drivers/ata/pata_atp867x.c @@ -530,7 +530,7 @@ err_out: return rc; } -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP static int atp867x_reinit_one(struct pci_dev *pdev) { struct ata_host *host = pci_get_drvdata(pdev); @@ -558,7 +558,7 @@ static struct pci_driver atp867x_driver = { .id_table = atp867x_pci_tbl, .probe = atp867x_init_one, .remove = ata_pci_remove_one, -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP .suspend = ata_pci_device_suspend, .resume = atp867x_reinit_one, #endif diff --git a/drivers/ata/pata_bf54x.c b/drivers/ata/pata_bf54x.c index ba0d8a29dc23..03f2f2bc83bd 100644 --- a/drivers/ata/pata_bf54x.c +++ b/drivers/ata/pata_bf54x.c @@ -1619,7 +1619,7 @@ static int bfin_atapi_remove(struct platform_device *pdev) return 0; } -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP static int bfin_atapi_suspend(struct platform_device *pdev, pm_message_t state) { struct ata_host *host = platform_get_drvdata(pdev); diff --git a/drivers/ata/pata_cmd640.c b/drivers/ata/pata_cmd640.c index 57f1be64dbf2..c47caa807fa9 100644 --- a/drivers/ata/pata_cmd640.c +++ b/drivers/ata/pata_cmd640.c @@ -231,7 +231,7 @@ static int cmd640_init_one(struct pci_dev *pdev, const struct pci_device_id *id) return ata_pci_sff_init_one(pdev, ppi, &cmd640_sht, NULL, 0); } -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP static int cmd640_reinit_one(struct pci_dev *pdev) { struct ata_host *host = pci_get_drvdata(pdev); @@ -256,7 +256,7 @@ static struct pci_driver cmd640_pci_driver = { .id_table = cmd640, .probe = cmd640_init_one, .remove = ata_pci_remove_one, -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP .suspend = ata_pci_device_suspend, .resume = cmd640_reinit_one, #endif diff --git a/drivers/ata/pata_cmd64x.c b/drivers/ata/pata_cmd64x.c index 6bca3505b9e9..13ca5883285b 100644 --- a/drivers/ata/pata_cmd64x.c +++ b/drivers/ata/pata_cmd64x.c @@ -487,7 +487,7 @@ static int cmd64x_init_one(struct pci_dev *pdev, const struct pci_device_id *id) return ata_pci_bmdma_init_one(pdev, ppi, &cmd64x_sht, NULL, 0); } -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP static int cmd64x_reinit_one(struct pci_dev *pdev) { struct ata_host *host = pci_get_drvdata(pdev); @@ -518,7 +518,7 @@ static struct pci_driver cmd64x_pci_driver = { .id_table = cmd64x, .probe = cmd64x_init_one, .remove = ata_pci_remove_one, -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP .suspend = ata_pci_device_suspend, .resume = cmd64x_reinit_one, #endif diff --git a/drivers/ata/pata_cs5520.c b/drivers/ata/pata_cs5520.c index bcde4b786807..d65cb9d2fa8c 100644 --- a/drivers/ata/pata_cs5520.c +++ b/drivers/ata/pata_cs5520.c @@ -229,7 +229,7 @@ static int cs5520_init_one(struct pci_dev *pdev, const struct pci_device_id *id) return ata_host_register(host, &cs5520_sht); } -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP /** * cs5520_reinit_one - device resume * @pdev: PCI device @@ -278,7 +278,7 @@ static int cs5520_pci_device_suspend(struct pci_dev *pdev, pm_message_t mesg) pci_save_state(pdev); return 0; } -#endif /* CONFIG_PM */ +#endif /* CONFIG_PM_SLEEP */ /* For now keep DMA off. We can set it for all but A rev CS5510 once the core ATA code can handle it */ @@ -295,7 +295,7 @@ static struct pci_driver cs5520_pci_driver = { .id_table = pata_cs5520, .probe = cs5520_init_one, .remove = ata_pci_remove_one, -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP .suspend = cs5520_pci_device_suspend, .resume = cs5520_reinit_one, #endif diff --git a/drivers/ata/pata_cs5530.c b/drivers/ata/pata_cs5530.c index 8afe854a5a50..48ae4b434474 100644 --- a/drivers/ata/pata_cs5530.c +++ b/drivers/ata/pata_cs5530.c @@ -326,7 +326,7 @@ static int cs5530_init_one(struct pci_dev *pdev, const struct pci_device_id *id) return ata_pci_bmdma_init_one(pdev, ppi, &cs5530_sht, NULL, 0); } -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP static int cs5530_reinit_one(struct pci_dev *pdev) { struct ata_host *host = pci_get_drvdata(pdev); @@ -343,7 +343,7 @@ static int cs5530_reinit_one(struct pci_dev *pdev) ata_host_resume(host); return 0; } -#endif /* CONFIG_PM */ +#endif /* CONFIG_PM_SLEEP */ static const struct pci_device_id cs5530[] = { { PCI_VDEVICE(CYRIX, PCI_DEVICE_ID_CYRIX_5530_IDE), }, @@ -356,7 +356,7 @@ static struct pci_driver cs5530_pci_driver = { .id_table = cs5530, .probe = cs5530_init_one, .remove = ata_pci_remove_one, -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP .suspend = ata_pci_device_suspend, .resume = cs5530_reinit_one, #endif diff --git a/drivers/ata/pata_cs5535.c b/drivers/ata/pata_cs5535.c index 2c0986fa4bb2..97584e8456d9 100644 --- a/drivers/ata/pata_cs5535.c +++ b/drivers/ata/pata_cs5535.c @@ -200,7 +200,7 @@ static struct pci_driver cs5535_pci_driver = { .id_table = cs5535, .probe = cs5535_init_one, .remove = ata_pci_remove_one, -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP .suspend = ata_pci_device_suspend, .resume = ata_pci_device_resume, #endif diff --git a/drivers/ata/pata_cs5536.c b/drivers/ata/pata_cs5536.c index 32ddcae5a360..6c15a554efbe 100644 --- a/drivers/ata/pata_cs5536.c +++ b/drivers/ata/pata_cs5536.c @@ -297,7 +297,7 @@ static struct pci_driver cs5536_pci_driver = { .id_table = cs5536, .probe = cs5536_init_one, .remove = ata_pci_remove_one, -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP .suspend = ata_pci_device_suspend, .resume = ata_pci_device_resume, #endif diff --git a/drivers/ata/pata_cypress.c b/drivers/ata/pata_cypress.c index 3435bd6a5cc9..793018460d82 100644 --- a/drivers/ata/pata_cypress.c +++ b/drivers/ata/pata_cypress.c @@ -151,7 +151,7 @@ static struct pci_driver cy82c693_pci_driver = { .id_table = cy82c693, .probe = cy82c693_init_one, .remove = ata_pci_remove_one, -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP .suspend = ata_pci_device_suspend, .resume = ata_pci_device_resume, #endif diff --git a/drivers/ata/pata_efar.c b/drivers/ata/pata_efar.c index f440892225f4..4a57a6f032d9 100644 --- a/drivers/ata/pata_efar.c +++ b/drivers/ata/pata_efar.c @@ -288,7 +288,7 @@ static struct pci_driver efar_pci_driver = { .id_table = efar_pci_tbl, .probe = efar_init_one, .remove = ata_pci_remove_one, -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP .suspend = ata_pci_device_suspend, .resume = ata_pci_device_resume, #endif diff --git a/drivers/ata/pata_ep93xx.c b/drivers/ata/pata_ep93xx.c index cad9d45749c4..6ad5c072ce34 100644 --- a/drivers/ata/pata_ep93xx.c +++ b/drivers/ata/pata_ep93xx.c @@ -708,8 +708,8 @@ static void ep93xx_pata_dma_start(struct ata_queued_cmd *qc) struct dma_chan *channel = qc->dma_dir == DMA_TO_DEVICE ? drv_data->dma_tx_channel : drv_data->dma_rx_channel; - txd = channel->device->device_prep_slave_sg(channel, qc->sg, - qc->n_elem, qc->dma_dir, DMA_CTRL_ACK, NULL); + txd = dmaengine_prep_slave_sg(channel, qc->sg, qc->n_elem, qc->dma_dir, + DMA_CTRL_ACK); if (!txd) { dev_err(qc->ap->dev, "failed to prepare slave for sg dma\n"); return; diff --git a/drivers/ata/pata_hpt366.c b/drivers/ata/pata_hpt366.c index 8e76f79689d3..cbc3de793d1d 100644 --- a/drivers/ata/pata_hpt366.c +++ b/drivers/ata/pata_hpt366.c @@ -386,7 +386,7 @@ static int hpt36x_init_one(struct pci_dev *dev, const struct pci_device_id *id) return ata_pci_bmdma_init_one(dev, ppi, &hpt36x_sht, hpriv, 0); } -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP static int hpt36x_reinit_one(struct pci_dev *dev) { struct ata_host *host = pci_get_drvdata(dev); @@ -411,7 +411,7 @@ static struct pci_driver hpt36x_pci_driver = { .id_table = hpt36x, .probe = hpt36x_init_one, .remove = ata_pci_remove_one, -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP .suspend = ata_pci_device_suspend, .resume = hpt36x_reinit_one, #endif diff --git a/drivers/ata/pata_hpt3x3.c b/drivers/ata/pata_hpt3x3.c index 255c5aaff3a8..d019cdd5bc9f 100644 --- a/drivers/ata/pata_hpt3x3.c +++ b/drivers/ata/pata_hpt3x3.c @@ -249,7 +249,7 @@ static int hpt3x3_init_one(struct pci_dev *pdev, const struct pci_device_id *id) IRQF_SHARED, &hpt3x3_sht); } -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP static int hpt3x3_reinit_one(struct pci_dev *dev) { struct ata_host *host = pci_get_drvdata(dev); @@ -277,7 +277,7 @@ static struct pci_driver hpt3x3_pci_driver = { .id_table = hpt3x3, .probe = hpt3x3_init_one, .remove = ata_pci_remove_one, -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP .suspend = ata_pci_device_suspend, .resume = hpt3x3_reinit_one, #endif diff --git a/drivers/ata/pata_imx.c b/drivers/ata/pata_imx.c index e0872db913d6..af424573c2ff 100644 --- a/drivers/ata/pata_imx.c +++ b/drivers/ata/pata_imx.c @@ -185,7 +185,7 @@ static int pata_imx_remove(struct platform_device *pdev) return 0; } -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP static int pata_imx_suspend(struct device *dev) { struct ata_host *host = dev_get_drvdata(dev); @@ -244,7 +244,7 @@ static struct platform_driver pata_imx_driver = { .name = DRV_NAME, .of_match_table = imx_pata_dt_ids, .owner = THIS_MODULE, -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP .pm = &pata_imx_pm_ops, #endif }, diff --git a/drivers/ata/pata_it8213.c b/drivers/ata/pata_it8213.c index 81369d187a5c..4f97d1e52f85 100644 --- a/drivers/ata/pata_it8213.c +++ b/drivers/ata/pata_it8213.c @@ -283,7 +283,7 @@ static struct pci_driver it8213_pci_driver = { .id_table = it8213_pci_tbl, .probe = it8213_init_one, .remove = ata_pci_remove_one, -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP .suspend = ata_pci_device_suspend, .resume = ata_pci_device_resume, #endif diff --git a/drivers/ata/pata_it821x.c b/drivers/ata/pata_it821x.c index dc3d7877f29d..a5088ecb349f 100644 --- a/drivers/ata/pata_it821x.c +++ b/drivers/ata/pata_it821x.c @@ -935,7 +935,7 @@ static int it821x_init_one(struct pci_dev *pdev, const struct pci_device_id *id) return ata_pci_bmdma_init_one(pdev, ppi, &it821x_sht, NULL, 0); } -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP static int it821x_reinit_one(struct pci_dev *pdev) { struct ata_host *host = pci_get_drvdata(pdev); @@ -965,7 +965,7 @@ static struct pci_driver it821x_pci_driver = { .id_table = it821x, .probe = it821x_init_one, .remove = ata_pci_remove_one, -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP .suspend = ata_pci_device_suspend, .resume = it821x_reinit_one, #endif diff --git a/drivers/ata/pata_jmicron.c b/drivers/ata/pata_jmicron.c index b1cfa0258fd3..4d1a5d2c4287 100644 --- a/drivers/ata/pata_jmicron.c +++ b/drivers/ata/pata_jmicron.c @@ -157,7 +157,7 @@ static struct pci_driver jmicron_pci_driver = { .id_table = jmicron_pci_tbl, .probe = jmicron_init_one, .remove = ata_pci_remove_one, -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP .suspend = ata_pci_device_suspend, .resume = ata_pci_device_resume, #endif diff --git a/drivers/ata/pata_macio.c b/drivers/ata/pata_macio.c index c28d0645e851..a02f76fdcfcd 100644 --- a/drivers/ata/pata_macio.c +++ b/drivers/ata/pata_macio.c @@ -845,8 +845,7 @@ static int pata_macio_slave_config(struct scsi_device *sdev) return 0; } -#ifdef CONFIG_PM - +#ifdef CONFIG_PM_SLEEP static int pata_macio_do_suspend(struct pata_macio_priv *priv, pm_message_t mesg) { int rc; @@ -907,8 +906,7 @@ static int pata_macio_do_resume(struct pata_macio_priv *priv) return 0; } - -#endif /* CONFIG_PM */ +#endif /* CONFIG_PM_SLEEP */ static struct scsi_host_template pata_macio_sht = { ATA_BASE_SHT(DRV_NAME), @@ -1208,8 +1206,7 @@ static int pata_macio_detach(struct macio_dev *mdev) return 0; } -#ifdef CONFIG_PM - +#ifdef CONFIG_PM_SLEEP static int pata_macio_suspend(struct macio_dev *mdev, pm_message_t mesg) { struct ata_host *host = macio_get_drvdata(mdev); @@ -1223,8 +1220,7 @@ static int pata_macio_resume(struct macio_dev *mdev) return pata_macio_do_resume(host->private_data); } - -#endif /* CONFIG_PM */ +#endif /* CONFIG_PM_SLEEP */ #ifdef CONFIG_PMAC_MEDIABAY static void pata_macio_mb_event(struct macio_dev* mdev, int mb_state) @@ -1316,8 +1312,7 @@ static void pata_macio_pci_detach(struct pci_dev *pdev) ata_host_detach(host); } -#ifdef CONFIG_PM - +#ifdef CONFIG_PM_SLEEP static int pata_macio_pci_suspend(struct pci_dev *pdev, pm_message_t mesg) { struct ata_host *host = pci_get_drvdata(pdev); @@ -1331,8 +1326,7 @@ static int pata_macio_pci_resume(struct pci_dev *pdev) return pata_macio_do_resume(host->private_data); } - -#endif /* CONFIG_PM */ +#endif /* CONFIG_PM_SLEEP */ static struct of_device_id pata_macio_match[] = { @@ -1360,7 +1354,7 @@ static struct macio_driver pata_macio_driver = }, .probe = pata_macio_attach, .remove = pata_macio_detach, -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP .suspend = pata_macio_suspend, .resume = pata_macio_resume, #endif @@ -1383,7 +1377,7 @@ static struct pci_driver pata_macio_pci_driver = { .id_table = pata_macio_pci_match, .probe = pata_macio_pci_attach, .remove = pata_macio_pci_detach, -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP .suspend = pata_macio_pci_suspend, .resume = pata_macio_pci_resume, #endif diff --git a/drivers/ata/pata_marvell.c b/drivers/ata/pata_marvell.c index 6bad3df3a13c..ae9feb1ba8f7 100644 --- a/drivers/ata/pata_marvell.c +++ b/drivers/ata/pata_marvell.c @@ -171,7 +171,7 @@ static struct pci_driver marvell_pci_driver = { .id_table = marvell_pci_tbl, .probe = marvell_init_one, .remove = ata_pci_remove_one, -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP .suspend = ata_pci_device_suspend, .resume = ata_pci_device_resume, #endif diff --git a/drivers/ata/pata_mpc52xx.c b/drivers/ata/pata_mpc52xx.c index 0024ced3e200..ccd1c83a05cc 100644 --- a/drivers/ata/pata_mpc52xx.c +++ b/drivers/ata/pata_mpc52xx.c @@ -819,9 +819,7 @@ mpc52xx_ata_remove(struct platform_device *op) return 0; } - -#ifdef CONFIG_PM - +#ifdef CONFIG_PM_SLEEP static int mpc52xx_ata_suspend(struct platform_device *op, pm_message_t state) { @@ -847,10 +845,8 @@ mpc52xx_ata_resume(struct platform_device *op) return 0; } - #endif - static struct of_device_id mpc52xx_ata_of_match[] = { { .compatible = "fsl,mpc5200-ata", }, { .compatible = "mpc5200-ata", }, @@ -861,7 +857,7 @@ static struct of_device_id mpc52xx_ata_of_match[] = { static struct platform_driver mpc52xx_ata_of_platform_driver = { .probe = mpc52xx_ata_probe, .remove = mpc52xx_ata_remove, -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP .suspend = mpc52xx_ata_suspend, .resume = mpc52xx_ata_resume, #endif diff --git a/drivers/ata/pata_mpiix.c b/drivers/ata/pata_mpiix.c index f39a5379e816..202b4d601393 100644 --- a/drivers/ata/pata_mpiix.c +++ b/drivers/ata/pata_mpiix.c @@ -223,7 +223,7 @@ static struct pci_driver mpiix_pci_driver = { .id_table = mpiix, .probe = mpiix_init_one, .remove = ata_pci_remove_one, -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP .suspend = ata_pci_device_suspend, .resume = ata_pci_device_resume, #endif diff --git a/drivers/ata/pata_netcell.c b/drivers/ata/pata_netcell.c index e3b97093ef9a..0ea18331d466 100644 --- a/drivers/ata/pata_netcell.c +++ b/drivers/ata/pata_netcell.c @@ -92,7 +92,7 @@ static struct pci_driver netcell_pci_driver = { .id_table = netcell_pci_tbl, .probe = netcell_init_one, .remove = ata_pci_remove_one, -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP .suspend = ata_pci_device_suspend, .resume = ata_pci_device_resume, #endif diff --git a/drivers/ata/pata_ninja32.c b/drivers/ata/pata_ninja32.c index 56201a69af12..efb272da8567 100644 --- a/drivers/ata/pata_ninja32.c +++ b/drivers/ata/pata_ninja32.c @@ -152,8 +152,7 @@ static int ninja32_init_one(struct pci_dev *dev, const struct pci_device_id *id) IRQF_SHARED, &ninja32_sht); } -#ifdef CONFIG_PM - +#ifdef CONFIG_PM_SLEEP static int ninja32_reinit_one(struct pci_dev *pdev) { struct ata_host *host = pci_get_drvdata(pdev); @@ -183,7 +182,7 @@ static struct pci_driver ninja32_pci_driver = { .id_table = ninja32, .probe = ninja32_init_one, .remove = ata_pci_remove_one, -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP .suspend = ata_pci_device_suspend, .resume = ninja32_reinit_one, #endif diff --git a/drivers/ata/pata_ns87410.c b/drivers/ata/pata_ns87410.c index 6154c3ee11a5..200e1eb23a20 100644 --- a/drivers/ata/pata_ns87410.c +++ b/drivers/ata/pata_ns87410.c @@ -161,7 +161,7 @@ static struct pci_driver ns87410_pci_driver = { .id_table = ns87410, .probe = ns87410_init_one, .remove = ata_pci_remove_one, -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP .suspend = ata_pci_device_suspend, .resume = ata_pci_device_resume, #endif diff --git a/drivers/ata/pata_ns87415.c b/drivers/ata/pata_ns87415.c index d44df7ccfe43..84c6b225b56e 100644 --- a/drivers/ata/pata_ns87415.c +++ b/drivers/ata/pata_ns87415.c @@ -385,7 +385,7 @@ static const struct pci_device_id ns87415_pci_tbl[] = { { } /* terminate list */ }; -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP static int ns87415_reinit_one(struct pci_dev *pdev) { struct ata_host *host = pci_get_drvdata(pdev); @@ -407,7 +407,7 @@ static struct pci_driver ns87415_pci_driver = { .id_table = ns87415_pci_tbl, .probe = ns87415_init_one, .remove = ata_pci_remove_one, -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP .suspend = ata_pci_device_suspend, .resume = ns87415_reinit_one, #endif diff --git a/drivers/ata/pata_octeon_cf.c b/drivers/ata/pata_octeon_cf.c index 83c4ddb1bc7f..2a97d3a531ec 100644 --- a/drivers/ata/pata_octeon_cf.c +++ b/drivers/ata/pata_octeon_cf.c @@ -865,7 +865,7 @@ static int octeon_cf_probe(struct platform_device *pdev) if (node == NULL) return -EINVAL; - cf_port = kzalloc(sizeof(*cf_port), GFP_KERNEL); + cf_port = devm_kzalloc(&pdev->dev, sizeof(*cf_port), GFP_KERNEL); if (!cf_port) return -ENOMEM; @@ -881,10 +881,9 @@ static int octeon_cf_probe(struct platform_device *pdev) n_size = of_n_size_cells(node); reg_prop = of_find_property(node, "reg", ®_len); - if (!reg_prop || reg_len < sizeof(__be32)) { - rv = -EINVAL; - goto free_cf_port; - } + if (!reg_prop || reg_len < sizeof(__be32)) + return -EINVAL; + cs_num = reg_prop->value; cf_port->cs0 = be32_to_cpup(cs_num); @@ -901,16 +900,13 @@ static int octeon_cf_probe(struct platform_device *pdev) res_dma = platform_get_resource(dma_dev, IORESOURCE_MEM, 0); if (!res_dma) { of_node_put(dma_node); - rv = -EINVAL; - goto free_cf_port; + return -EINVAL; } cf_port->dma_base = (u64)devm_ioremap_nocache(&pdev->dev, res_dma->start, resource_size(res_dma)); - if (!cf_port->dma_base) { of_node_put(dma_node); - rv = -EINVAL; - goto free_cf_port; + return -EINVAL; } irq_handler = octeon_cf_interrupt; @@ -921,41 +917,34 @@ static int octeon_cf_probe(struct platform_device *pdev) of_node_put(dma_node); } res_cs1 = platform_get_resource(pdev, IORESOURCE_MEM, 1); - if (!res_cs1) { - rv = -EINVAL; - goto free_cf_port; - } + if (!res_cs1) + return -EINVAL; + cs1 = devm_ioremap_nocache(&pdev->dev, res_cs1->start, resource_size(res_cs1)); - if (!cs1) - goto free_cf_port; + return rv; + + if (reg_len < (n_addr + n_size + 1) * sizeof(__be32)) + return -EINVAL; - if (reg_len < (n_addr + n_size + 1) * sizeof(__be32)) { - rv = -EINVAL; - goto free_cf_port; - } cs_num += n_addr + n_size; cf_port->cs1 = be32_to_cpup(cs_num); } res_cs0 = platform_get_resource(pdev, IORESOURCE_MEM, 0); - - if (!res_cs0) { - rv = -EINVAL; - goto free_cf_port; - } + if (!res_cs0) + return -EINVAL; cs0 = devm_ioremap_nocache(&pdev->dev, res_cs0->start, resource_size(res_cs0)); - if (!cs0) - goto free_cf_port; + return rv; /* allocate host */ host = ata_host_alloc(&pdev->dev, 1); if (!host) - goto free_cf_port; + return rv; ap = host->ports[0]; ap->private_data = cf_port; @@ -1020,17 +1009,12 @@ static int octeon_cf_probe(struct platform_device *pdev) ata_port_desc(ap, "cmd %p ctl %p", base, ap->ioaddr.ctl_addr); - dev_info(&pdev->dev, "version " DRV_VERSION" %d bit%s.\n", is_16bit ? 16 : 8, cf_port->is_true_ide ? ", True IDE" : ""); return ata_host_activate(host, irq, irq_handler, IRQF_SHARED, &octeon_cf_sht); - -free_cf_port: - kfree(cf_port); - return rv; } static void octeon_cf_shutdown(struct device *dev) diff --git a/drivers/ata/pata_oldpiix.c b/drivers/ata/pata_oldpiix.c index 319b64491b7b..b9bf78b7d48d 100644 --- a/drivers/ata/pata_oldpiix.c +++ b/drivers/ata/pata_oldpiix.c @@ -258,7 +258,7 @@ static struct pci_driver oldpiix_pci_driver = { .id_table = oldpiix_pci_tbl, .probe = oldpiix_init_one, .remove = ata_pci_remove_one, -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP .suspend = ata_pci_device_suspend, .resume = ata_pci_device_resume, #endif diff --git a/drivers/ata/pata_opti.c b/drivers/ata/pata_opti.c index fb042e0519d0..3a944a029264 100644 --- a/drivers/ata/pata_opti.c +++ b/drivers/ata/pata_opti.c @@ -184,7 +184,7 @@ static struct pci_driver opti_pci_driver = { .id_table = opti, .probe = opti_init_one, .remove = ata_pci_remove_one, -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP .suspend = ata_pci_device_suspend, .resume = ata_pci_device_resume, #endif diff --git a/drivers/ata/pata_optidma.c b/drivers/ata/pata_optidma.c index bb71ea214b99..bdec7efa4643 100644 --- a/drivers/ata/pata_optidma.c +++ b/drivers/ata/pata_optidma.c @@ -440,7 +440,7 @@ static struct pci_driver optidma_pci_driver = { .id_table = optidma, .probe = optidma_init_one, .remove = ata_pci_remove_one, -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP .suspend = ata_pci_device_suspend, .resume = ata_pci_device_resume, #endif diff --git a/drivers/ata/pata_pdc2027x.c b/drivers/ata/pata_pdc2027x.c index 1151f23177bb..4d06a5cda987 100644 --- a/drivers/ata/pata_pdc2027x.c +++ b/drivers/ata/pata_pdc2027x.c @@ -62,7 +62,7 @@ enum { }; static int pdc2027x_init_one(struct pci_dev *pdev, const struct pci_device_id *ent); -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP static int pdc2027x_reinit_one(struct pci_dev *pdev); #endif static int pdc2027x_prereset(struct ata_link *link, unsigned long deadline); @@ -128,7 +128,7 @@ static struct pci_driver pdc2027x_pci_driver = { .id_table = pdc2027x_pci_tbl, .probe = pdc2027x_init_one, .remove = ata_pci_remove_one, -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP .suspend = ata_pci_device_suspend, .resume = pdc2027x_reinit_one, #endif @@ -761,7 +761,7 @@ static int pdc2027x_init_one(struct pci_dev *pdev, IRQF_SHARED, &pdc2027x_sht); } -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP static int pdc2027x_reinit_one(struct pci_dev *pdev) { struct ata_host *host = pci_get_drvdata(pdev); diff --git a/drivers/ata/pata_pdc202xx_old.c b/drivers/ata/pata_pdc202xx_old.c index defa050e1784..9001991d2830 100644 --- a/drivers/ata/pata_pdc202xx_old.c +++ b/drivers/ata/pata_pdc202xx_old.c @@ -377,7 +377,7 @@ static struct pci_driver pdc202xx_pci_driver = { .id_table = pdc202xx, .probe = pdc202xx_init_one, .remove = ata_pci_remove_one, -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP .suspend = ata_pci_device_suspend, .resume = ata_pci_device_resume, #endif diff --git a/drivers/ata/pata_piccolo.c b/drivers/ata/pata_piccolo.c index 0b46be117051..35cb0e263237 100644 --- a/drivers/ata/pata_piccolo.c +++ b/drivers/ata/pata_piccolo.c @@ -110,7 +110,7 @@ static struct pci_driver ata_tosh_pci_driver = { .id_table = ata_tosh, .probe = ata_tosh_init_one, .remove = ata_pci_remove_one, -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP .suspend = ata_pci_device_suspend, .resume = ata_pci_device_resume, #endif diff --git a/drivers/ata/pata_radisys.c b/drivers/ata/pata_radisys.c index be3f10240dca..a3f1123d17aa 100644 --- a/drivers/ata/pata_radisys.c +++ b/drivers/ata/pata_radisys.c @@ -237,7 +237,7 @@ static struct pci_driver radisys_pci_driver = { .id_table = radisys_pci_tbl, .probe = radisys_init_one, .remove = ata_pci_remove_one, -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP .suspend = ata_pci_device_suspend, .resume = ata_pci_device_resume, #endif diff --git a/drivers/ata/pata_rdc.c b/drivers/ata/pata_rdc.c index 521b2137ea3e..9ce5952216bc 100644 --- a/drivers/ata/pata_rdc.c +++ b/drivers/ata/pata_rdc.c @@ -382,7 +382,7 @@ static struct pci_driver rdc_pci_driver = { .id_table = rdc_pci_tbl, .probe = rdc_init_one, .remove = rdc_remove_one, -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP .suspend = ata_pci_device_suspend, .resume = ata_pci_device_resume, #endif diff --git a/drivers/ata/pata_rz1000.c b/drivers/ata/pata_rz1000.c index caedc90855b2..b3ec18c6bcc6 100644 --- a/drivers/ata/pata_rz1000.c +++ b/drivers/ata/pata_rz1000.c @@ -101,7 +101,7 @@ static int rz1000_init_one (struct pci_dev *pdev, const struct pci_device_id *en return -ENODEV; } -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP static int rz1000_reinit_one(struct pci_dev *pdev) { struct ata_host *host = pci_get_drvdata(pdev); @@ -133,7 +133,7 @@ static struct pci_driver rz1000_pci_driver = { .id_table = pata_rz1000, .probe = rz1000_init_one, .remove = ata_pci_remove_one, -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP .suspend = ata_pci_device_suspend, .resume = rz1000_reinit_one, #endif diff --git a/drivers/ata/pata_samsung_cf.c b/drivers/ata/pata_samsung_cf.c index 0610e78c8a2a..fb528831fb92 100644 --- a/drivers/ata/pata_samsung_cf.c +++ b/drivers/ata/pata_samsung_cf.c @@ -619,7 +619,7 @@ static int __exit pata_s3c_remove(struct platform_device *pdev) return 0; } -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP static int pata_s3c_suspend(struct device *dev) { struct platform_device *pdev = to_platform_device(dev); @@ -670,7 +670,7 @@ static struct platform_driver pata_s3c_driver = { .driver = { .name = DRV_NAME, .owner = THIS_MODULE, -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP .pm = &pata_s3c_pm_ops, #endif }, diff --git a/drivers/ata/pata_sc1200.c b/drivers/ata/pata_sc1200.c index 96a232fffae6..c71de5d680d1 100644 --- a/drivers/ata/pata_sc1200.c +++ b/drivers/ata/pata_sc1200.c @@ -254,7 +254,7 @@ static struct pci_driver sc1200_pci_driver = { .id_table = sc1200, .probe = sc1200_init_one, .remove = ata_pci_remove_one, -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP .suspend = ata_pci_device_suspend, .resume = ata_pci_device_resume, #endif diff --git a/drivers/ata/pata_scc.c b/drivers/ata/pata_scc.c index f1f5b5ae3382..4e006d74bef8 100644 --- a/drivers/ata/pata_scc.c +++ b/drivers/ata/pata_scc.c @@ -1096,7 +1096,7 @@ static struct pci_driver scc_pci_driver = { .id_table = scc_pci_tbl, .probe = scc_init_one, .remove = ata_pci_remove_one, -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP .suspend = ata_pci_device_suspend, .resume = ata_pci_device_resume, #endif diff --git a/drivers/ata/pata_sch.c b/drivers/ata/pata_sch.c index 5a1cde0ea360..b920c3407f8b 100644 --- a/drivers/ata/pata_sch.c +++ b/drivers/ata/pata_sch.c @@ -64,7 +64,7 @@ static struct pci_driver sch_pci_driver = { .id_table = sch_pci_tbl, .probe = sch_init_one, .remove = ata_pci_remove_one, -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP .suspend = ata_pci_device_suspend, .resume = ata_pci_device_resume, #endif diff --git a/drivers/ata/pata_serverworks.c b/drivers/ata/pata_serverworks.c index e27f31fe1b67..fc5f31d4828e 100644 --- a/drivers/ata/pata_serverworks.c +++ b/drivers/ata/pata_serverworks.c @@ -436,7 +436,7 @@ static int serverworks_init_one(struct pci_dev *pdev, const struct pci_device_id return ata_pci_bmdma_init_one(pdev, ppi, &serverworks_sht, NULL, 0); } -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP static int serverworks_reinit_one(struct pci_dev *pdev) { struct ata_host *host = pci_get_drvdata(pdev); @@ -468,7 +468,7 @@ static struct pci_driver serverworks_pci_driver = { .id_table = serverworks, .probe = serverworks_init_one, .remove = ata_pci_remove_one, -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP .suspend = ata_pci_device_suspend, .resume = serverworks_reinit_one, #endif diff --git a/drivers/ata/pata_sil680.c b/drivers/ata/pata_sil680.c index 73fe362d9716..f597edccedec 100644 --- a/drivers/ata/pata_sil680.c +++ b/drivers/ata/pata_sil680.c @@ -403,7 +403,7 @@ use_ioports: return ata_pci_bmdma_init_one(pdev, ppi, &sil680_sht, NULL, 0); } -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP static int sil680_reinit_one(struct pci_dev *pdev) { struct ata_host *host = pci_get_drvdata(pdev); @@ -429,7 +429,7 @@ static struct pci_driver sil680_pci_driver = { .id_table = sil680, .probe = sil680_init_one, .remove = ata_pci_remove_one, -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP .suspend = ata_pci_device_suspend, .resume = sil680_reinit_one, #endif diff --git a/drivers/ata/pata_sis.c b/drivers/ata/pata_sis.c index 78d913aa93c8..626f989d5c6a 100644 --- a/drivers/ata/pata_sis.c +++ b/drivers/ata/pata_sis.c @@ -869,7 +869,7 @@ static int sis_init_one (struct pci_dev *pdev, const struct pci_device_id *ent) return ata_pci_bmdma_init_one(pdev, ppi, &sis_sht, chipset, 0); } -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP static int sis_reinit_one(struct pci_dev *pdev) { struct ata_host *host = pci_get_drvdata(pdev); @@ -899,7 +899,7 @@ static struct pci_driver sis_pci_driver = { .id_table = sis_pci_tbl, .probe = sis_init_one, .remove = ata_pci_remove_one, -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP .suspend = ata_pci_device_suspend, .resume = sis_reinit_one, #endif diff --git a/drivers/ata/pata_sl82c105.c b/drivers/ata/pata_sl82c105.c index 900f0e4a1faf..4935f61f629c 100644 --- a/drivers/ata/pata_sl82c105.c +++ b/drivers/ata/pata_sl82c105.c @@ -337,7 +337,7 @@ static int sl82c105_init_one(struct pci_dev *dev, const struct pci_device_id *id return ata_pci_bmdma_init_one(dev, ppi, &sl82c105_sht, NULL, 0); } -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP static int sl82c105_reinit_one(struct pci_dev *pdev) { struct ata_host *host = pci_get_drvdata(pdev); @@ -365,7 +365,7 @@ static struct pci_driver sl82c105_pci_driver = { .id_table = sl82c105, .probe = sl82c105_init_one, .remove = ata_pci_remove_one, -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP .suspend = ata_pci_device_suspend, .resume = sl82c105_reinit_one, #endif diff --git a/drivers/ata/pata_triflex.c b/drivers/ata/pata_triflex.c index 7bc78e264f9e..d9364af8eb63 100644 --- a/drivers/ata/pata_triflex.c +++ b/drivers/ata/pata_triflex.c @@ -207,7 +207,7 @@ static const struct pci_device_id triflex[] = { { }, }; -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP static int triflex_ata_pci_device_suspend(struct pci_dev *pdev, pm_message_t mesg) { struct ata_host *host = pci_get_drvdata(pdev); @@ -233,7 +233,7 @@ static struct pci_driver triflex_pci_driver = { .id_table = triflex, .probe = triflex_init_one, .remove = ata_pci_remove_one, -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP .suspend = triflex_ata_pci_device_suspend, .resume = ata_pci_device_resume, #endif diff --git a/drivers/ata/pata_via.c b/drivers/ata/pata_via.c index f6c9632bdff6..1ca6bcab369f 100644 --- a/drivers/ata/pata_via.c +++ b/drivers/ata/pata_via.c @@ -659,7 +659,7 @@ static int via_init_one(struct pci_dev *pdev, const struct pci_device_id *id) return ata_pci_bmdma_init_one(pdev, ppi, &via_sht, (void *)config, 0); } -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP /** * via_reinit_one - reinit after resume * @pdev; PCI device @@ -704,7 +704,7 @@ static struct pci_driver via_pci_driver = { .id_table = via, .probe = via_init_one, .remove = ata_pci_remove_one, -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP .suspend = ata_pci_device_suspend, .resume = via_reinit_one, #endif diff --git a/drivers/ata/sata_fsl.c b/drivers/ata/sata_fsl.c index fb0b40a191c2..616a6d2ac20c 100644 --- a/drivers/ata/sata_fsl.c +++ b/drivers/ata/sata_fsl.c @@ -774,20 +774,6 @@ static int sata_fsl_port_start(struct ata_port *ap) VPRINTK("HControl = 0x%x\n", ioread32(hcr_base + HCONTROL)); VPRINTK("CHBA = 0x%x\n", ioread32(hcr_base + CHBA)); -#ifdef CONFIG_MPC8315_DS - /* - * Workaround for 8315DS board 3gbps link-up issue, - * currently limit SATA port to GEN1 speed - */ - sata_fsl_scr_read(&ap->link, SCR_CONTROL, &temp); - temp &= ~(0xF << 4); - temp |= (0x1 << 4); - sata_fsl_scr_write(&ap->link, SCR_CONTROL, temp); - - sata_fsl_scr_read(&ap->link, SCR_CONTROL, &temp); - dev_warn(dev, "scr_control, speed limited to %x\n", temp); -#endif - return 0; } @@ -1588,7 +1574,7 @@ static int sata_fsl_remove(struct platform_device *ofdev) return 0; } -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP static int sata_fsl_suspend(struct platform_device *op, pm_message_t state) { struct ata_host *host = platform_get_drvdata(op); @@ -1644,7 +1630,7 @@ static struct platform_driver fsl_sata_driver = { }, .probe = sata_fsl_probe, .remove = sata_fsl_remove, -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP .suspend = sata_fsl_suspend, .resume = sata_fsl_resume, #endif diff --git a/drivers/ata/sata_inic162x.c b/drivers/ata/sata_inic162x.c index 5c54d957370a..069827826b20 100644 --- a/drivers/ata/sata_inic162x.c +++ b/drivers/ata/sata_inic162x.c @@ -785,7 +785,7 @@ static int init_controller(void __iomem *mmio_base, u16 hctl) return 0; } -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP static int inic_pci_device_resume(struct pci_dev *pdev) { struct ata_host *host = pci_get_drvdata(pdev); @@ -898,7 +898,7 @@ static const struct pci_device_id inic_pci_tbl[] = { static struct pci_driver inic_pci_driver = { .name = DRV_NAME, .id_table = inic_pci_tbl, -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP .suspend = ata_pci_device_suspend, .resume = inic_pci_device_resume, #endif diff --git a/drivers/ata/sata_mv.c b/drivers/ata/sata_mv.c index 05c8a44adf8e..391cfda1e83f 100644 --- a/drivers/ata/sata_mv.c +++ b/drivers/ata/sata_mv.c @@ -4222,7 +4222,7 @@ static int mv_platform_remove(struct platform_device *pdev) return 0; } -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP static int mv_platform_suspend(struct platform_device *pdev, pm_message_t state) { struct ata_host *host = platform_get_drvdata(pdev); @@ -4289,7 +4289,7 @@ static struct platform_driver mv_platform_driver = { #ifdef CONFIG_PCI static int mv_pci_init_one(struct pci_dev *pdev, const struct pci_device_id *ent); -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP static int mv_pci_device_resume(struct pci_dev *pdev); #endif @@ -4299,7 +4299,7 @@ static struct pci_driver mv_pci_driver = { .id_table = mv_pci_tbl, .probe = mv_pci_init_one, .remove = ata_pci_remove_one, -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP .suspend = ata_pci_device_suspend, .resume = mv_pci_device_resume, #endif @@ -4457,7 +4457,7 @@ static int mv_pci_init_one(struct pci_dev *pdev, IS_GEN_I(hpriv) ? &mv5_sht : &mv6_sht); } -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP static int mv_pci_device_resume(struct pci_dev *pdev) { struct ata_host *host = pci_get_drvdata(pdev); diff --git a/drivers/ata/sata_nv.c b/drivers/ata/sata_nv.c index ba5f27120332..cdf99fac139a 100644 --- a/drivers/ata/sata_nv.c +++ b/drivers/ata/sata_nv.c @@ -295,7 +295,7 @@ struct nv_swncq_port_priv { #define NV_ADMA_CHECK_INTR(GCTL, PORT) ((GCTL) & (1 << (19 + (12 * (PORT))))) static int nv_init_one(struct pci_dev *pdev, const struct pci_device_id *ent); -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP static int nv_pci_device_resume(struct pci_dev *pdev); #endif static void nv_ck804_host_stop(struct ata_host *host); @@ -379,7 +379,7 @@ static struct pci_driver nv_pci_driver = { .name = DRV_NAME, .id_table = nv_pci_tbl, .probe = nv_init_one, -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP .suspend = ata_pci_device_suspend, .resume = nv_pci_device_resume, #endif @@ -2431,7 +2431,7 @@ static int nv_init_one(struct pci_dev *pdev, const struct pci_device_id *ent) return ata_pci_sff_activate_host(host, ipriv->irq_handler, ipriv->sht); } -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP static int nv_pci_device_resume(struct pci_dev *pdev) { struct ata_host *host = pci_get_drvdata(pdev); diff --git a/drivers/ata/sata_rcar.c b/drivers/ata/sata_rcar.c index 2b25bd83fc9d..61eb6d77dac7 100644 --- a/drivers/ata/sata_rcar.c +++ b/drivers/ata/sata_rcar.c @@ -937,7 +937,7 @@ static int sata_rcar_remove(struct platform_device *pdev) return 0; } -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP static int sata_rcar_suspend(struct device *dev) { struct ata_host *host = dev_get_drvdata(dev); @@ -991,7 +991,7 @@ static struct platform_driver sata_rcar_driver = { .name = DRV_NAME, .owner = THIS_MODULE, .of_match_table = sata_rcar_match, -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP .pm = &sata_rcar_pm_ops, #endif }, diff --git a/drivers/ata/sata_sil.c b/drivers/ata/sata_sil.c index 3062f8605b29..40b76b2d18c6 100644 --- a/drivers/ata/sata_sil.c +++ b/drivers/ata/sata_sil.c @@ -112,7 +112,7 @@ enum { }; static int sil_init_one(struct pci_dev *pdev, const struct pci_device_id *ent); -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP static int sil_pci_device_resume(struct pci_dev *pdev); #endif static void sil_dev_config(struct ata_device *dev); @@ -166,7 +166,7 @@ static struct pci_driver sil_pci_driver = { .id_table = sil_pci_tbl, .probe = sil_init_one, .remove = ata_pci_remove_one, -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP .suspend = ata_pci_device_suspend, .resume = sil_pci_device_resume, #endif @@ -802,7 +802,7 @@ static int sil_init_one(struct pci_dev *pdev, const struct pci_device_id *ent) &sil_sht); } -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP static int sil_pci_device_resume(struct pci_dev *pdev) { struct ata_host *host = pci_get_drvdata(pdev); diff --git a/drivers/ata/sata_sil24.c b/drivers/ata/sata_sil24.c index aa1051ba6d13..0534890f118a 100644 --- a/drivers/ata/sata_sil24.c +++ b/drivers/ata/sata_sil24.c @@ -353,8 +353,10 @@ static void sil24_error_handler(struct ata_port *ap); static void sil24_post_internal_cmd(struct ata_queued_cmd *qc); static int sil24_port_start(struct ata_port *ap); static int sil24_init_one(struct pci_dev *pdev, const struct pci_device_id *ent); -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP static int sil24_pci_device_resume(struct pci_dev *pdev); +#endif +#ifdef CONFIG_PM static int sil24_port_resume(struct ata_port *ap); #endif @@ -375,7 +377,7 @@ static struct pci_driver sil24_pci_driver = { .id_table = sil24_pci_tbl, .probe = sil24_init_one, .remove = ata_pci_remove_one, -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP .suspend = ata_pci_device_suspend, .resume = sil24_pci_device_resume, #endif @@ -1350,7 +1352,7 @@ static int sil24_init_one(struct pci_dev *pdev, const struct pci_device_id *ent) &sil24_sht); } -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP static int sil24_pci_device_resume(struct pci_dev *pdev) { struct ata_host *host = pci_get_drvdata(pdev); @@ -1370,7 +1372,9 @@ static int sil24_pci_device_resume(struct pci_dev *pdev) return 0; } +#endif +#ifdef CONFIG_PM static int sil24_port_resume(struct ata_port *ap) { sil24_config_pmp(ap, ap->nr_pmp_links); diff --git a/drivers/ata/sata_sis.c b/drivers/ata/sata_sis.c index b513428171b3..d1637ac40a73 100644 --- a/drivers/ata/sata_sis.c +++ b/drivers/ata/sata_sis.c @@ -82,7 +82,7 @@ static struct pci_driver sis_pci_driver = { .id_table = sis_pci_tbl, .probe = sis_init_one, .remove = ata_pci_remove_one, -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP .suspend = ata_pci_device_suspend, .resume = ata_pci_device_resume, #endif diff --git a/drivers/ata/sata_via.c b/drivers/ata/sata_via.c index f72e84228c5c..47bf89464cef 100644 --- a/drivers/ata/sata_via.c +++ b/drivers/ata/sata_via.c @@ -103,7 +103,7 @@ static struct pci_driver svia_pci_driver = { .name = DRV_NAME, .id_table = svia_pci_tbl, .probe = svia_init_one, -#ifdef CONFIG_PM +#ifdef CONFIG_PM_SLEEP .suspend = ata_pci_device_suspend, .resume = ata_pci_device_resume, #endif diff --git a/drivers/cpuidle/Kconfig b/drivers/cpuidle/Kconfig index f04e25f6c98d..1b96fb91d32c 100644 --- a/drivers/cpuidle/Kconfig +++ b/drivers/cpuidle/Kconfig @@ -35,6 +35,11 @@ depends on ARM source "drivers/cpuidle/Kconfig.arm" endmenu +menu "MIPS CPU Idle Drivers" +depends on MIPS +source "drivers/cpuidle/Kconfig.mips" +endmenu + menu "POWERPC CPU Idle Drivers" depends on PPC source "drivers/cpuidle/Kconfig.powerpc" diff --git a/drivers/cpuidle/Kconfig.mips b/drivers/cpuidle/Kconfig.mips new file mode 100644 index 000000000000..0e70ee28a5ca --- /dev/null +++ b/drivers/cpuidle/Kconfig.mips @@ -0,0 +1,17 @@ +# +# MIPS CPU Idle Drivers +# +config MIPS_CPS_CPUIDLE + bool "CPU Idle driver for MIPS CPS platforms" + depends on CPU_IDLE + depends on SYS_SUPPORTS_MIPS_CPS + select ARCH_NEEDS_CPU_IDLE_COUPLED if MIPS_MT + select GENERIC_CLOCKEVENTS_BROADCAST if SMP + select MIPS_CPS_PM + default y + help + Select this option to enable processor idle state management + through cpuidle for systems built around the MIPS Coherent + Processing System (CPS) architecture. In order to make use of + the deepest idle states you will need to ensure that you are + also using the CONFIG_MIPS_CPS SMP implementation. diff --git a/drivers/cpuidle/Makefile b/drivers/cpuidle/Makefile index 9b5b2b560d70..d8bb1ff72561 100644 --- a/drivers/cpuidle/Makefile +++ b/drivers/cpuidle/Makefile @@ -18,6 +18,10 @@ obj-$(CONFIG_ARM_AT91_CPUIDLE) += cpuidle-at91.o obj-$(CONFIG_ARM_EXYNOS_CPUIDLE) += cpuidle-exynos.o ############################################################################### +# MIPS drivers +obj-$(CONFIG_MIPS_CPS_CPUIDLE) += cpuidle-cps.o + +############################################################################### # POWERPC drivers obj-$(CONFIG_PSERIES_CPUIDLE) += cpuidle-pseries.o obj-$(CONFIG_POWERNV_CPUIDLE) += cpuidle-powernv.o diff --git a/drivers/cpuidle/cpuidle-cps.c b/drivers/cpuidle/cpuidle-cps.c new file mode 100644 index 000000000000..fc7b62720deb --- /dev/null +++ b/drivers/cpuidle/cpuidle-cps.c @@ -0,0 +1,186 @@ +/* + * Copyright (C) 2014 Imagination Technologies + * Author: Paul Burton <paul.burton@imgtec.com> + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the + * Free Software Foundation; either version 2 of the License, or (at your + * option) any later version. + */ + +#include <linux/cpu_pm.h> +#include <linux/cpuidle.h> +#include <linux/init.h> + +#include <asm/idle.h> +#include <asm/pm-cps.h> + +/* Enumeration of the various idle states this driver may enter */ +enum cps_idle_state { + STATE_WAIT = 0, /* MIPS wait instruction, coherent */ + STATE_NC_WAIT, /* MIPS wait instruction, non-coherent */ + STATE_CLOCK_GATED, /* Core clock gated */ + STATE_POWER_GATED, /* Core power gated */ + STATE_COUNT +}; + +static int cps_nc_enter(struct cpuidle_device *dev, + struct cpuidle_driver *drv, int index) +{ + enum cps_pm_state pm_state; + int err; + + /* + * At least one core must remain powered up & clocked in order for the + * system to have any hope of functioning. + * + * TODO: don't treat core 0 specially, just prevent the final core + * TODO: remap interrupt affinity temporarily + */ + if (!cpu_data[dev->cpu].core && (index > STATE_NC_WAIT)) + index = STATE_NC_WAIT; + + /* Select the appropriate cps_pm_state */ + switch (index) { + case STATE_NC_WAIT: + pm_state = CPS_PM_NC_WAIT; + break; + case STATE_CLOCK_GATED: + pm_state = CPS_PM_CLOCK_GATED; + break; + case STATE_POWER_GATED: + pm_state = CPS_PM_POWER_GATED; + break; + default: + BUG(); + return -EINVAL; + } + + /* Notify listeners the CPU is about to power down */ + if ((pm_state == CPS_PM_POWER_GATED) && cpu_pm_enter()) + return -EINTR; + + /* Enter that state */ + err = cps_pm_enter_state(pm_state); + + /* Notify listeners the CPU is back up */ + if (pm_state == CPS_PM_POWER_GATED) + cpu_pm_exit(); + + return err ?: index; +} + +static struct cpuidle_driver cps_driver = { + .name = "cpc_cpuidle", + .owner = THIS_MODULE, + .states = { + [STATE_WAIT] = MIPS_CPUIDLE_WAIT_STATE, + [STATE_NC_WAIT] = { + .enter = cps_nc_enter, + .exit_latency = 200, + .target_residency = 450, + .flags = CPUIDLE_FLAG_TIME_VALID, + .name = "nc-wait", + .desc = "non-coherent MIPS wait", + }, + [STATE_CLOCK_GATED] = { + .enter = cps_nc_enter, + .exit_latency = 300, + .target_residency = 700, + .flags = CPUIDLE_FLAG_TIME_VALID | + CPUIDLE_FLAG_TIMER_STOP, + .name = "clock-gated", + .desc = "core clock gated", + }, + [STATE_POWER_GATED] = { + .enter = cps_nc_enter, + .exit_latency = 600, + .target_residency = 1000, + .flags = CPUIDLE_FLAG_TIME_VALID | + CPUIDLE_FLAG_TIMER_STOP, + .name = "power-gated", + .desc = "core power gated", + }, + }, + .state_count = STATE_COUNT, + .safe_state_index = 0, +}; + +static void __init cps_cpuidle_unregister(void) +{ + int cpu; + struct cpuidle_device *device; + + for_each_possible_cpu(cpu) { + device = &per_cpu(cpuidle_dev, cpu); + cpuidle_unregister_device(device); + } + + cpuidle_unregister_driver(&cps_driver); +} + +static int __init cps_cpuidle_init(void) +{ + int err, cpu, core, i; + struct cpuidle_device *device; + + /* Detect supported states */ + if (!cps_pm_support_state(CPS_PM_POWER_GATED)) + cps_driver.state_count = STATE_CLOCK_GATED + 1; + if (!cps_pm_support_state(CPS_PM_CLOCK_GATED)) + cps_driver.state_count = STATE_NC_WAIT + 1; + if (!cps_pm_support_state(CPS_PM_NC_WAIT)) + cps_driver.state_count = STATE_WAIT + 1; + + /* Inform the user if some states are unavailable */ + if (cps_driver.state_count < STATE_COUNT) { + pr_info("cpuidle-cps: limited to "); + switch (cps_driver.state_count - 1) { + case STATE_WAIT: + pr_cont("coherent wait\n"); + break; + case STATE_NC_WAIT: + pr_cont("non-coherent wait\n"); + break; + case STATE_CLOCK_GATED: + pr_cont("clock gating\n"); + break; + } + } + + /* + * Set the coupled flag on the appropriate states if this system + * requires it. + */ + if (coupled_coherence) + for (i = STATE_NC_WAIT; i < cps_driver.state_count; i++) + cps_driver.states[i].flags |= CPUIDLE_FLAG_COUPLED; + + err = cpuidle_register_driver(&cps_driver); + if (err) { + pr_err("Failed to register CPS cpuidle driver\n"); + return err; + } + + for_each_possible_cpu(cpu) { + core = cpu_data[cpu].core; + device = &per_cpu(cpuidle_dev, cpu); + device->cpu = cpu; +#ifdef CONFIG_MIPS_MT + cpumask_copy(&device->coupled_cpus, &cpu_sibling_map[cpu]); +#endif + + err = cpuidle_register_device(device); + if (err) { + pr_err("Failed to register CPU%d cpuidle device\n", + cpu); + goto err_out; + } + } + + return 0; +err_out: + cps_cpuidle_unregister(); + return err; +} +device_initcall(cps_cpuidle_init); diff --git a/drivers/dma/Kconfig b/drivers/dma/Kconfig index 5c5863842de9..1eca7b9760e6 100644 --- a/drivers/dma/Kconfig +++ b/drivers/dma/Kconfig @@ -234,7 +234,7 @@ config PL330_DMA config PCH_DMA tristate "Intel EG20T PCH / LAPIS Semicon IOH(ML7213/ML7223/ML7831) DMA" - depends on PCI && X86 + depends on PCI && (X86_32 || COMPILE_TEST) select DMA_ENGINE help Enable support for Intel EG20T PCH DMA engine. @@ -269,7 +269,7 @@ config MXS_DMA select DMA_ENGINE help Support the MXS DMA engine. This engine including APBH-DMA - and APBX-DMA is integrated into Freescale i.MX23/28 chips. + and APBX-DMA is integrated into Freescale i.MX23/28/MX6Q/MX6DL chips. config EP93XX_DMA bool "Cirrus Logic EP93xx DMA support" @@ -361,6 +361,20 @@ config FSL_EDMA multiplexing capability for DMA request sources(slot). This module can be found on Freescale Vybrid and LS-1 SoCs. +config XILINX_VDMA + tristate "Xilinx AXI VDMA Engine" + depends on (ARCH_ZYNQ || MICROBLAZE) + select DMA_ENGINE + help + Enable support for Xilinx AXI VDMA Soft IP. + + This engine provides high-bandwidth direct memory access + between memory and AXI4-Stream video type target + peripherals including peripherals which support AXI4- + Stream Video Protocol. It has two stream interfaces/ + channels, Memory Mapped to Stream (MM2S) and Stream to + Memory Mapped (S2MM) for the data transfers. + config DMA_ENGINE bool diff --git a/drivers/dma/Makefile b/drivers/dma/Makefile index 5150c82c9caf..c779e1eb2db2 100644 --- a/drivers/dma/Makefile +++ b/drivers/dma/Makefile @@ -46,3 +46,4 @@ obj-$(CONFIG_K3_DMA) += k3dma.o obj-$(CONFIG_MOXART_DMA) += moxart-dma.o obj-$(CONFIG_FSL_EDMA) += fsl-edma.o obj-$(CONFIG_QCOM_BAM_DMA) += qcom_bam_dma.o +obj-y += xilinx/ diff --git a/drivers/dma/dw/core.c b/drivers/dma/dw/core.c index 7a740769c2fa..a27ded53ab4f 100644 --- a/drivers/dma/dw/core.c +++ b/drivers/dma/dw/core.c @@ -1493,6 +1493,13 @@ int dw_dma_probe(struct dw_dma_chip *chip, struct dw_dma_platform_data *pdata) dw->regs = chip->regs; chip->dw = dw; + dw->clk = devm_clk_get(chip->dev, "hclk"); + if (IS_ERR(dw->clk)) + return PTR_ERR(dw->clk); + err = clk_prepare_enable(dw->clk); + if (err) + return err; + dw_params = dma_read_byaddr(chip->regs, DW_PARAMS); autocfg = dw_params >> DW_PARAMS_EN & 0x1; @@ -1500,15 +1507,19 @@ int dw_dma_probe(struct dw_dma_chip *chip, struct dw_dma_platform_data *pdata) if (!pdata && autocfg) { pdata = devm_kzalloc(chip->dev, sizeof(*pdata), GFP_KERNEL); - if (!pdata) - return -ENOMEM; + if (!pdata) { + err = -ENOMEM; + goto err_pdata; + } /* Fill platform data with the default values */ pdata->is_private = true; pdata->chan_allocation_order = CHAN_ALLOCATION_ASCENDING; pdata->chan_priority = CHAN_PRIORITY_ASCENDING; - } else if (!pdata || pdata->nr_channels > DW_DMA_MAX_NR_CHANNELS) - return -EINVAL; + } else if (!pdata || pdata->nr_channels > DW_DMA_MAX_NR_CHANNELS) { + err = -EINVAL; + goto err_pdata; + } if (autocfg) nr_channels = (dw_params >> DW_PARAMS_NR_CHAN & 0x7) + 1; @@ -1517,13 +1528,10 @@ int dw_dma_probe(struct dw_dma_chip *chip, struct dw_dma_platform_data *pdata) dw->chan = devm_kcalloc(chip->dev, nr_channels, sizeof(*dw->chan), GFP_KERNEL); - if (!dw->chan) - return -ENOMEM; - - dw->clk = devm_clk_get(chip->dev, "hclk"); - if (IS_ERR(dw->clk)) - return PTR_ERR(dw->clk); - clk_prepare_enable(dw->clk); + if (!dw->chan) { + err = -ENOMEM; + goto err_pdata; + } /* Get hardware configuration parameters */ if (autocfg) { @@ -1553,7 +1561,8 @@ int dw_dma_probe(struct dw_dma_chip *chip, struct dw_dma_platform_data *pdata) sizeof(struct dw_desc), 4, 0); if (!dw->desc_pool) { dev_err(chip->dev, "No memory for descriptors dma pool\n"); - return -ENOMEM; + err = -ENOMEM; + goto err_pdata; } tasklet_init(&dw->tasklet, dw_dma_tasklet, (unsigned long)dw); @@ -1561,7 +1570,7 @@ int dw_dma_probe(struct dw_dma_chip *chip, struct dw_dma_platform_data *pdata) err = request_irq(chip->irq, dw_dma_interrupt, IRQF_SHARED, "dw_dmac", dw); if (err) - return err; + goto err_pdata; INIT_LIST_HEAD(&dw->dma.channels); for (i = 0; i < nr_channels; i++) { @@ -1650,12 +1659,20 @@ int dw_dma_probe(struct dw_dma_chip *chip, struct dw_dma_platform_data *pdata) dma_writel(dw, CFG, DW_CFG_DMA_EN); + err = dma_async_device_register(&dw->dma); + if (err) + goto err_dma_register; + dev_info(chip->dev, "DesignWare DMA Controller, %d channels\n", nr_channels); - dma_async_device_register(&dw->dma); - return 0; + +err_dma_register: + free_irq(chip->irq, dw); +err_pdata: + clk_disable_unprepare(dw->clk); + return err; } EXPORT_SYMBOL_GPL(dw_dma_probe); @@ -1676,6 +1693,8 @@ int dw_dma_remove(struct dw_dma_chip *chip) channel_clear_bit(dw, CH_EN, dwc->mask); } + clk_disable_unprepare(dw->clk); + return 0; } EXPORT_SYMBOL_GPL(dw_dma_remove); diff --git a/drivers/dma/dw/pci.c b/drivers/dma/dw/pci.c index fec59f1a77bb..39e30c3c7a9d 100644 --- a/drivers/dma/dw/pci.c +++ b/drivers/dma/dw/pci.c @@ -93,19 +93,13 @@ static int dw_pci_resume_early(struct device *dev) return dw_dma_resume(chip); }; -#else /* !CONFIG_PM_SLEEP */ - -#define dw_pci_suspend_late NULL -#define dw_pci_resume_early NULL - -#endif /* !CONFIG_PM_SLEEP */ +#endif /* CONFIG_PM_SLEEP */ static const struct dev_pm_ops dw_pci_dev_pm_ops = { - .suspend_late = dw_pci_suspend_late, - .resume_early = dw_pci_resume_early, + SET_LATE_SYSTEM_SLEEP_PM_OPS(dw_pci_suspend_late, dw_pci_resume_early) }; -static DEFINE_PCI_DEVICE_TABLE(dw_pci_id_table) = { +static const struct pci_device_id dw_pci_id_table[] = { /* Medfield */ { PCI_VDEVICE(INTEL, 0x0827), (kernel_ulong_t)&dw_pci_pdata }, { PCI_VDEVICE(INTEL, 0x0830), (kernel_ulong_t)&dw_pci_pdata }, diff --git a/drivers/dma/dw/platform.c b/drivers/dma/dw/platform.c index 453822cc4f9d..c5b339af6be5 100644 --- a/drivers/dma/dw/platform.c +++ b/drivers/dma/dw/platform.c @@ -256,7 +256,7 @@ MODULE_DEVICE_TABLE(acpi, dw_dma_acpi_id_table); #ifdef CONFIG_PM_SLEEP -static int dw_suspend_noirq(struct device *dev) +static int dw_suspend_late(struct device *dev) { struct platform_device *pdev = to_platform_device(dev); struct dw_dma_chip *chip = platform_get_drvdata(pdev); @@ -264,7 +264,7 @@ static int dw_suspend_noirq(struct device *dev) return dw_dma_suspend(chip); } -static int dw_resume_noirq(struct device *dev) +static int dw_resume_early(struct device *dev) { struct platform_device *pdev = to_platform_device(dev); struct dw_dma_chip *chip = platform_get_drvdata(pdev); @@ -272,20 +272,10 @@ static int dw_resume_noirq(struct device *dev) return dw_dma_resume(chip); } -#else /* !CONFIG_PM_SLEEP */ - -#define dw_suspend_noirq NULL -#define dw_resume_noirq NULL - -#endif /* !CONFIG_PM_SLEEP */ +#endif /* CONFIG_PM_SLEEP */ static const struct dev_pm_ops dw_dev_pm_ops = { - .suspend_noirq = dw_suspend_noirq, - .resume_noirq = dw_resume_noirq, - .freeze_noirq = dw_suspend_noirq, - .thaw_noirq = dw_resume_noirq, - .restore_noirq = dw_resume_noirq, - .poweroff_noirq = dw_suspend_noirq, + SET_LATE_SYSTEM_SLEEP_PM_OPS(dw_suspend_late, dw_resume_early) }; static struct platform_driver dw_driver = { diff --git a/drivers/dma/fsldma.c b/drivers/dma/fsldma.c index f157c6f76b32..e0fec68aed25 100644 --- a/drivers/dma/fsldma.c +++ b/drivers/dma/fsldma.c @@ -61,6 +61,16 @@ static u32 get_sr(struct fsldma_chan *chan) return DMA_IN(chan, &chan->regs->sr, 32); } +static void set_mr(struct fsldma_chan *chan, u32 val) +{ + DMA_OUT(chan, &chan->regs->mr, val, 32); +} + +static u32 get_mr(struct fsldma_chan *chan) +{ + return DMA_IN(chan, &chan->regs->mr, 32); +} + static void set_cdar(struct fsldma_chan *chan, dma_addr_t addr) { DMA_OUT(chan, &chan->regs->cdar, addr | FSL_DMA_SNEN, 64); @@ -71,6 +81,11 @@ static dma_addr_t get_cdar(struct fsldma_chan *chan) return DMA_IN(chan, &chan->regs->cdar, 64) & ~FSL_DMA_SNEN; } +static void set_bcr(struct fsldma_chan *chan, u32 val) +{ + DMA_OUT(chan, &chan->regs->bcr, val, 32); +} + static u32 get_bcr(struct fsldma_chan *chan) { return DMA_IN(chan, &chan->regs->bcr, 32); @@ -135,7 +150,7 @@ static void set_ld_eol(struct fsldma_chan *chan, struct fsl_desc_sw *desc) static void dma_init(struct fsldma_chan *chan) { /* Reset the channel */ - DMA_OUT(chan, &chan->regs->mr, 0, 32); + set_mr(chan, 0); switch (chan->feature & FSL_DMA_IP_MASK) { case FSL_DMA_IP_85XX: @@ -144,16 +159,15 @@ static void dma_init(struct fsldma_chan *chan) * EOLNIE - End of links interrupt enable * BWC - Bandwidth sharing among channels */ - DMA_OUT(chan, &chan->regs->mr, FSL_DMA_MR_BWC - | FSL_DMA_MR_EIE | FSL_DMA_MR_EOLNIE, 32); + set_mr(chan, FSL_DMA_MR_BWC | FSL_DMA_MR_EIE + | FSL_DMA_MR_EOLNIE); break; case FSL_DMA_IP_83XX: /* Set the channel to below modes: * EOTIE - End-of-transfer interrupt enable * PRC_RM - PCI read multiple */ - DMA_OUT(chan, &chan->regs->mr, FSL_DMA_MR_EOTIE - | FSL_DMA_MR_PRC_RM, 32); + set_mr(chan, FSL_DMA_MR_EOTIE | FSL_DMA_MR_PRC_RM); break; } } @@ -175,10 +189,10 @@ static void dma_start(struct fsldma_chan *chan) { u32 mode; - mode = DMA_IN(chan, &chan->regs->mr, 32); + mode = get_mr(chan); if (chan->feature & FSL_DMA_CHAN_PAUSE_EXT) { - DMA_OUT(chan, &chan->regs->bcr, 0, 32); + set_bcr(chan, 0); mode |= FSL_DMA_MR_EMP_EN; } else { mode &= ~FSL_DMA_MR_EMP_EN; @@ -191,7 +205,7 @@ static void dma_start(struct fsldma_chan *chan) mode |= FSL_DMA_MR_CS; } - DMA_OUT(chan, &chan->regs->mr, mode, 32); + set_mr(chan, mode); } static void dma_halt(struct fsldma_chan *chan) @@ -200,7 +214,7 @@ static void dma_halt(struct fsldma_chan *chan) int i; /* read the mode register */ - mode = DMA_IN(chan, &chan->regs->mr, 32); + mode = get_mr(chan); /* * The 85xx controller supports channel abort, which will stop @@ -209,14 +223,14 @@ static void dma_halt(struct fsldma_chan *chan) */ if ((chan->feature & FSL_DMA_IP_MASK) == FSL_DMA_IP_85XX) { mode |= FSL_DMA_MR_CA; - DMA_OUT(chan, &chan->regs->mr, mode, 32); + set_mr(chan, mode); mode &= ~FSL_DMA_MR_CA; } /* stop the DMA controller */ mode &= ~(FSL_DMA_MR_CS | FSL_DMA_MR_EMS_EN); - DMA_OUT(chan, &chan->regs->mr, mode, 32); + set_mr(chan, mode); /* wait for the DMA controller to become idle */ for (i = 0; i < 100; i++) { @@ -245,7 +259,7 @@ static void fsl_chan_set_src_loop_size(struct fsldma_chan *chan, int size) { u32 mode; - mode = DMA_IN(chan, &chan->regs->mr, 32); + mode = get_mr(chan); switch (size) { case 0: @@ -259,7 +273,7 @@ static void fsl_chan_set_src_loop_size(struct fsldma_chan *chan, int size) break; } - DMA_OUT(chan, &chan->regs->mr, mode, 32); + set_mr(chan, mode); } /** @@ -277,7 +291,7 @@ static void fsl_chan_set_dst_loop_size(struct fsldma_chan *chan, int size) { u32 mode; - mode = DMA_IN(chan, &chan->regs->mr, 32); + mode = get_mr(chan); switch (size) { case 0: @@ -291,7 +305,7 @@ static void fsl_chan_set_dst_loop_size(struct fsldma_chan *chan, int size) break; } - DMA_OUT(chan, &chan->regs->mr, mode, 32); + set_mr(chan, mode); } /** @@ -312,10 +326,10 @@ static void fsl_chan_set_request_count(struct fsldma_chan *chan, int size) BUG_ON(size > 1024); - mode = DMA_IN(chan, &chan->regs->mr, 32); + mode = get_mr(chan); mode |= (__ilog2(size) << 24) & 0x0f000000; - DMA_OUT(chan, &chan->regs->mr, mode, 32); + set_mr(chan, mode); } /** @@ -404,6 +418,19 @@ static dma_cookie_t fsl_dma_tx_submit(struct dma_async_tx_descriptor *tx) } /** + * fsl_dma_free_descriptor - Free descriptor from channel's DMA pool. + * @chan : Freescale DMA channel + * @desc: descriptor to be freed + */ +static void fsl_dma_free_descriptor(struct fsldma_chan *chan, + struct fsl_desc_sw *desc) +{ + list_del(&desc->node); + chan_dbg(chan, "LD %p free\n", desc); + dma_pool_free(chan->desc_pool, desc, desc->async_tx.phys); +} + +/** * fsl_dma_alloc_descriptor - Allocate descriptor from channel's DMA pool. * @chan : Freescale DMA channel * @@ -426,14 +453,107 @@ static struct fsl_desc_sw *fsl_dma_alloc_descriptor(struct fsldma_chan *chan) desc->async_tx.tx_submit = fsl_dma_tx_submit; desc->async_tx.phys = pdesc; -#ifdef FSL_DMA_LD_DEBUG chan_dbg(chan, "LD %p allocated\n", desc); -#endif return desc; } /** + * fsl_chan_xfer_ld_queue - transfer any pending transactions + * @chan : Freescale DMA channel + * + * HARDWARE STATE: idle + * LOCKING: must hold chan->desc_lock + */ +static void fsl_chan_xfer_ld_queue(struct fsldma_chan *chan) +{ + struct fsl_desc_sw *desc; + + /* + * If the list of pending descriptors is empty, then we + * don't need to do any work at all + */ + if (list_empty(&chan->ld_pending)) { + chan_dbg(chan, "no pending LDs\n"); + return; + } + + /* + * The DMA controller is not idle, which means that the interrupt + * handler will start any queued transactions when it runs after + * this transaction finishes + */ + if (!chan->idle) { + chan_dbg(chan, "DMA controller still busy\n"); + return; + } + + /* + * If there are some link descriptors which have not been + * transferred, we need to start the controller + */ + + /* + * Move all elements from the queue of pending transactions + * onto the list of running transactions + */ + chan_dbg(chan, "idle, starting controller\n"); + desc = list_first_entry(&chan->ld_pending, struct fsl_desc_sw, node); + list_splice_tail_init(&chan->ld_pending, &chan->ld_running); + + /* + * The 85xx DMA controller doesn't clear the channel start bit + * automatically at the end of a transfer. Therefore we must clear + * it in software before starting the transfer. + */ + if ((chan->feature & FSL_DMA_IP_MASK) == FSL_DMA_IP_85XX) { + u32 mode; + + mode = get_mr(chan); + mode &= ~FSL_DMA_MR_CS; + set_mr(chan, mode); + } + + /* + * Program the descriptor's address into the DMA controller, + * then start the DMA transaction + */ + set_cdar(chan, desc->async_tx.phys); + get_cdar(chan); + + dma_start(chan); + chan->idle = false; +} + +/** + * fsldma_cleanup_descriptor - cleanup and free a single link descriptor + * @chan: Freescale DMA channel + * @desc: descriptor to cleanup and free + * + * This function is used on a descriptor which has been executed by the DMA + * controller. It will run any callbacks, submit any dependencies, and then + * free the descriptor. + */ +static void fsldma_cleanup_descriptor(struct fsldma_chan *chan, + struct fsl_desc_sw *desc) +{ + struct dma_async_tx_descriptor *txd = &desc->async_tx; + + /* Run the link descriptor callback function */ + if (txd->callback) { + chan_dbg(chan, "LD %p callback\n", desc); + txd->callback(txd->callback_param); + } + + /* Run any dependencies */ + dma_run_dependencies(txd); + + dma_descriptor_unmap(txd); + chan_dbg(chan, "LD %p free\n", desc); + dma_pool_free(chan->desc_pool, desc, txd->phys); +} + +/** * fsl_dma_alloc_chan_resources - Allocate resources for DMA channel. * @chan : Freescale DMA channel * @@ -477,13 +597,8 @@ static void fsldma_free_desc_list(struct fsldma_chan *chan, { struct fsl_desc_sw *desc, *_desc; - list_for_each_entry_safe(desc, _desc, list, node) { - list_del(&desc->node); -#ifdef FSL_DMA_LD_DEBUG - chan_dbg(chan, "LD %p free\n", desc); -#endif - dma_pool_free(chan->desc_pool, desc, desc->async_tx.phys); - } + list_for_each_entry_safe(desc, _desc, list, node) + fsl_dma_free_descriptor(chan, desc); } static void fsldma_free_desc_list_reverse(struct fsldma_chan *chan, @@ -491,13 +606,8 @@ static void fsldma_free_desc_list_reverse(struct fsldma_chan *chan, { struct fsl_desc_sw *desc, *_desc; - list_for_each_entry_safe_reverse(desc, _desc, list, node) { - list_del(&desc->node); -#ifdef FSL_DMA_LD_DEBUG - chan_dbg(chan, "LD %p free\n", desc); -#endif - dma_pool_free(chan->desc_pool, desc, desc->async_tx.phys); - } + list_for_each_entry_safe_reverse(desc, _desc, list, node) + fsl_dma_free_descriptor(chan, desc); } /** @@ -520,35 +630,6 @@ static void fsl_dma_free_chan_resources(struct dma_chan *dchan) } static struct dma_async_tx_descriptor * -fsl_dma_prep_interrupt(struct dma_chan *dchan, unsigned long flags) -{ - struct fsldma_chan *chan; - struct fsl_desc_sw *new; - - if (!dchan) - return NULL; - - chan = to_fsl_chan(dchan); - - new = fsl_dma_alloc_descriptor(chan); - if (!new) { - chan_err(chan, "%s\n", msg_ld_oom); - return NULL; - } - - new->async_tx.cookie = -EBUSY; - new->async_tx.flags = flags; - - /* Insert the link descriptor to the LD ring */ - list_add_tail(&new->node, &new->tx_list); - - /* Set End-of-link to the last link descriptor of new list */ - set_ld_eol(chan, new); - - return &new->async_tx; -} - -static struct dma_async_tx_descriptor * fsl_dma_prep_memcpy(struct dma_chan *dchan, dma_addr_t dma_dst, dma_addr_t dma_src, size_t len, unsigned long flags) @@ -817,105 +898,6 @@ static int fsl_dma_device_control(struct dma_chan *dchan, } /** - * fsldma_cleanup_descriptor - cleanup and free a single link descriptor - * @chan: Freescale DMA channel - * @desc: descriptor to cleanup and free - * - * This function is used on a descriptor which has been executed by the DMA - * controller. It will run any callbacks, submit any dependencies, and then - * free the descriptor. - */ -static void fsldma_cleanup_descriptor(struct fsldma_chan *chan, - struct fsl_desc_sw *desc) -{ - struct dma_async_tx_descriptor *txd = &desc->async_tx; - - /* Run the link descriptor callback function */ - if (txd->callback) { -#ifdef FSL_DMA_LD_DEBUG - chan_dbg(chan, "LD %p callback\n", desc); -#endif - txd->callback(txd->callback_param); - } - - /* Run any dependencies */ - dma_run_dependencies(txd); - - dma_descriptor_unmap(txd); -#ifdef FSL_DMA_LD_DEBUG - chan_dbg(chan, "LD %p free\n", desc); -#endif - dma_pool_free(chan->desc_pool, desc, txd->phys); -} - -/** - * fsl_chan_xfer_ld_queue - transfer any pending transactions - * @chan : Freescale DMA channel - * - * HARDWARE STATE: idle - * LOCKING: must hold chan->desc_lock - */ -static void fsl_chan_xfer_ld_queue(struct fsldma_chan *chan) -{ - struct fsl_desc_sw *desc; - - /* - * If the list of pending descriptors is empty, then we - * don't need to do any work at all - */ - if (list_empty(&chan->ld_pending)) { - chan_dbg(chan, "no pending LDs\n"); - return; - } - - /* - * The DMA controller is not idle, which means that the interrupt - * handler will start any queued transactions when it runs after - * this transaction finishes - */ - if (!chan->idle) { - chan_dbg(chan, "DMA controller still busy\n"); - return; - } - - /* - * If there are some link descriptors which have not been - * transferred, we need to start the controller - */ - - /* - * Move all elements from the queue of pending transactions - * onto the list of running transactions - */ - chan_dbg(chan, "idle, starting controller\n"); - desc = list_first_entry(&chan->ld_pending, struct fsl_desc_sw, node); - list_splice_tail_init(&chan->ld_pending, &chan->ld_running); - - /* - * The 85xx DMA controller doesn't clear the channel start bit - * automatically at the end of a transfer. Therefore we must clear - * it in software before starting the transfer. - */ - if ((chan->feature & FSL_DMA_IP_MASK) == FSL_DMA_IP_85XX) { - u32 mode; - - mode = DMA_IN(chan, &chan->regs->mr, 32); - mode &= ~FSL_DMA_MR_CS; - DMA_OUT(chan, &chan->regs->mr, mode, 32); - } - - /* - * Program the descriptor's address into the DMA controller, - * then start the DMA transaction - */ - set_cdar(chan, desc->async_tx.phys); - get_cdar(chan); - - dma_start(chan); - chan->idle = false; -} - -/** * fsl_dma_memcpy_issue_pending - Issue the DMA start command * @chan : Freescale DMA channel */ @@ -1304,12 +1286,10 @@ static int fsldma_of_probe(struct platform_device *op) fdev->irq = irq_of_parse_and_map(op->dev.of_node, 0); dma_cap_set(DMA_MEMCPY, fdev->common.cap_mask); - dma_cap_set(DMA_INTERRUPT, fdev->common.cap_mask); dma_cap_set(DMA_SG, fdev->common.cap_mask); dma_cap_set(DMA_SLAVE, fdev->common.cap_mask); fdev->common.device_alloc_chan_resources = fsl_dma_alloc_chan_resources; fdev->common.device_free_chan_resources = fsl_dma_free_chan_resources; - fdev->common.device_prep_dma_interrupt = fsl_dma_prep_interrupt; fdev->common.device_prep_dma_memcpy = fsl_dma_prep_memcpy; fdev->common.device_prep_dma_sg = fsl_dma_prep_sg; fdev->common.device_tx_status = fsl_tx_status; diff --git a/drivers/dma/imx-sdma.c b/drivers/dma/imx-sdma.c index 19041cefabb1..128714622bf5 100644 --- a/drivers/dma/imx-sdma.c +++ b/drivers/dma/imx-sdma.c @@ -607,8 +607,6 @@ static void sdma_handle_channel_loop(struct sdma_channel *sdmac) if (bd->mode.status & BD_RROR) sdmac->status = DMA_ERROR; - else - sdmac->status = DMA_IN_PROGRESS; bd->mode.status |= BD_DONE; sdmac->buf_tail++; diff --git a/drivers/dma/mmp_pdma.c b/drivers/dma/mmp_pdma.c index bf02e7beb51a..a7b186d536b3 100644 --- a/drivers/dma/mmp_pdma.c +++ b/drivers/dma/mmp_pdma.c @@ -29,8 +29,8 @@ #define DALGN 0x00a0 #define DINT 0x00f0 #define DDADR 0x0200 -#define DSADR 0x0204 -#define DTADR 0x0208 +#define DSADR(n) (0x0204 + ((n) << 4)) +#define DTADR(n) (0x0208 + ((n) << 4)) #define DCMD 0x020c #define DCSR_RUN BIT(31) /* Run Bit (read / write) */ @@ -277,7 +277,7 @@ static void mmp_pdma_free_phy(struct mmp_pdma_chan *pchan) return; /* clear the channel mapping in DRCMR */ - reg = DRCMR(pchan->phy->vchan->drcmr); + reg = DRCMR(pchan->drcmr); writel(0, pchan->phy->base + reg); spin_lock_irqsave(&pdev->phy_lock, flags); @@ -748,11 +748,92 @@ static int mmp_pdma_control(struct dma_chan *dchan, enum dma_ctrl_cmd cmd, return 0; } +static unsigned int mmp_pdma_residue(struct mmp_pdma_chan *chan, + dma_cookie_t cookie) +{ + struct mmp_pdma_desc_sw *sw; + u32 curr, residue = 0; + bool passed = false; + bool cyclic = chan->cyclic_first != NULL; + + /* + * If the channel does not have a phy pointer anymore, it has already + * been completed. Therefore, its residue is 0. + */ + if (!chan->phy) + return 0; + + if (chan->dir == DMA_DEV_TO_MEM) + curr = readl(chan->phy->base + DTADR(chan->phy->idx)); + else + curr = readl(chan->phy->base + DSADR(chan->phy->idx)); + + list_for_each_entry(sw, &chan->chain_running, node) { + u32 start, end, len; + + if (chan->dir == DMA_DEV_TO_MEM) + start = sw->desc.dtadr; + else + start = sw->desc.dsadr; + + len = sw->desc.dcmd & DCMD_LENGTH; + end = start + len; + + /* + * 'passed' will be latched once we found the descriptor which + * lies inside the boundaries of the curr pointer. All + * descriptors that occur in the list _after_ we found that + * partially handled descriptor are still to be processed and + * are hence added to the residual bytes counter. + */ + + if (passed) { + residue += len; + } else if (curr >= start && curr <= end) { + residue += end - curr; + passed = true; + } + + /* + * Descriptors that have the ENDIRQEN bit set mark the end of a + * transaction chain, and the cookie assigned with it has been + * returned previously from mmp_pdma_tx_submit(). + * + * In case we have multiple transactions in the running chain, + * and the cookie does not match the one the user asked us + * about, reset the state variables and start over. + * + * This logic does not apply to cyclic transactions, where all + * descriptors have the ENDIRQEN bit set, and for which we + * can't have multiple transactions on one channel anyway. + */ + if (cyclic || !(sw->desc.dcmd & DCMD_ENDIRQEN)) + continue; + + if (sw->async_tx.cookie == cookie) { + return residue; + } else { + residue = 0; + passed = false; + } + } + + /* We should only get here in case of cyclic transactions */ + return residue; +} + static enum dma_status mmp_pdma_tx_status(struct dma_chan *dchan, dma_cookie_t cookie, struct dma_tx_state *txstate) { - return dma_cookie_status(dchan, cookie, txstate); + struct mmp_pdma_chan *chan = to_mmp_pdma_chan(dchan); + enum dma_status ret; + + ret = dma_cookie_status(dchan, cookie, txstate); + if (likely(ret != DMA_ERROR)) + dma_set_residue(txstate, mmp_pdma_residue(chan, cookie)); + + return ret; } /** @@ -858,8 +939,7 @@ static int mmp_pdma_chan_init(struct mmp_pdma_device *pdev, int idx, int irq) struct mmp_pdma_chan *chan; int ret; - chan = devm_kzalloc(pdev->dev, sizeof(struct mmp_pdma_chan), - GFP_KERNEL); + chan = devm_kzalloc(pdev->dev, sizeof(*chan), GFP_KERNEL); if (chan == NULL) return -ENOMEM; @@ -946,8 +1026,7 @@ static int mmp_pdma_probe(struct platform_device *op) irq_num++; } - pdev->phy = devm_kcalloc(pdev->dev, - dma_channels, sizeof(struct mmp_pdma_chan), + pdev->phy = devm_kcalloc(pdev->dev, dma_channels, sizeof(*pdev->phy), GFP_KERNEL); if (pdev->phy == NULL) return -ENOMEM; diff --git a/drivers/dma/mpc512x_dma.c b/drivers/dma/mpc512x_dma.c index 448750da4402..2ad43738ac8b 100644 --- a/drivers/dma/mpc512x_dma.c +++ b/drivers/dma/mpc512x_dma.c @@ -2,6 +2,7 @@ * Copyright (C) Freescale Semicondutor, Inc. 2007, 2008. * Copyright (C) Semihalf 2009 * Copyright (C) Ilya Yanok, Emcraft Systems 2010 + * Copyright (C) Alexander Popov, Promcontroller 2014 * * Written by Piotr Ziecik <kosmo@semihalf.com>. Hardware description * (defines, structures and comments) was taken from MPC5121 DMA driver @@ -29,8 +30,18 @@ */ /* - * This is initial version of MPC5121 DMA driver. Only memory to memory - * transfers are supported (tested using dmatest module). + * MPC512x and MPC8308 DMA driver. It supports + * memory to memory data transfers (tested using dmatest module) and + * data transfers between memory and peripheral I/O memory + * by means of slave scatter/gather with these limitations: + * - chunked transfers (described by s/g lists with more than one item) + * are refused as long as proper support for scatter/gather is missing; + * - transfers on MPC8308 always start from software as this SoC appears + * not to have external request lines for peripheral flow control; + * - only peripheral devices with 4-byte FIFO access register are supported; + * - minimal memory <-> I/O memory transfer chunk is 4 bytes and consequently + * source and destination addresses must be 4-byte aligned + * and transfer size must be aligned on (4 * maxburst) boundary; */ #include <linux/module.h> @@ -52,9 +63,17 @@ #define MPC_DMA_DESCRIPTORS 64 /* Macro definitions */ -#define MPC_DMA_CHANNELS 64 #define MPC_DMA_TCD_OFFSET 0x1000 +/* + * Maximum channel counts for individual hardware variants + * and the maximum channel count over all supported controllers, + * used for data structure size + */ +#define MPC8308_DMACHAN_MAX 16 +#define MPC512x_DMACHAN_MAX 64 +#define MPC_DMA_CHANNELS 64 + /* Arbitration mode of group and channel */ #define MPC_DMA_DMACR_EDCG (1 << 31) #define MPC_DMA_DMACR_ERGA (1 << 3) @@ -181,6 +200,7 @@ struct mpc_dma_desc { dma_addr_t tcd_paddr; int error; struct list_head node; + int will_access_peripheral; }; struct mpc_dma_chan { @@ -193,6 +213,12 @@ struct mpc_dma_chan { struct mpc_dma_tcd *tcd; dma_addr_t tcd_paddr; + /* Settings for access to peripheral FIFO */ + dma_addr_t src_per_paddr; + u32 src_tcd_nunits; + dma_addr_t dst_per_paddr; + u32 dst_tcd_nunits; + /* Lock for this structure */ spinlock_t lock; }; @@ -243,8 +269,23 @@ static void mpc_dma_execute(struct mpc_dma_chan *mchan) struct mpc_dma_desc *mdesc; int cid = mchan->chan.chan_id; - /* Move all queued descriptors to active list */ - list_splice_tail_init(&mchan->queued, &mchan->active); + while (!list_empty(&mchan->queued)) { + mdesc = list_first_entry(&mchan->queued, + struct mpc_dma_desc, node); + /* + * Grab either several mem-to-mem transfer descriptors + * or one peripheral transfer descriptor, + * don't mix mem-to-mem and peripheral transfer descriptors + * within the same 'active' list. + */ + if (mdesc->will_access_peripheral) { + if (list_empty(&mchan->active)) + list_move_tail(&mdesc->node, &mchan->active); + break; + } else { + list_move_tail(&mdesc->node, &mchan->active); + } + } /* Chain descriptors into one transaction */ list_for_each_entry(mdesc, &mchan->active, node) { @@ -270,7 +311,17 @@ static void mpc_dma_execute(struct mpc_dma_chan *mchan) if (first != prev) mdma->tcd[cid].e_sg = 1; - out_8(&mdma->regs->dmassrt, cid); + + if (mdma->is_mpc8308) { + /* MPC8308, no request lines, software initiated start */ + out_8(&mdma->regs->dmassrt, cid); + } else if (first->will_access_peripheral) { + /* Peripherals involved, start by external request signal */ + out_8(&mdma->regs->dmaserq, cid); + } else { + /* Memory to memory transfer, software initiated start */ + out_8(&mdma->regs->dmassrt, cid); + } } /* Handle interrupt on one half of DMA controller (32 channels) */ @@ -588,6 +639,7 @@ mpc_dma_prep_memcpy(struct dma_chan *chan, dma_addr_t dst, dma_addr_t src, } mdesc->error = 0; + mdesc->will_access_peripheral = 0; tcd = mdesc->tcd; /* Prepare Transfer Control Descriptor for this transaction */ @@ -635,6 +687,193 @@ mpc_dma_prep_memcpy(struct dma_chan *chan, dma_addr_t dst, dma_addr_t src, return &mdesc->desc; } +static struct dma_async_tx_descriptor * +mpc_dma_prep_slave_sg(struct dma_chan *chan, struct scatterlist *sgl, + unsigned int sg_len, enum dma_transfer_direction direction, + unsigned long flags, void *context) +{ + struct mpc_dma *mdma = dma_chan_to_mpc_dma(chan); + struct mpc_dma_chan *mchan = dma_chan_to_mpc_dma_chan(chan); + struct mpc_dma_desc *mdesc = NULL; + dma_addr_t per_paddr; + u32 tcd_nunits; + struct mpc_dma_tcd *tcd; + unsigned long iflags; + struct scatterlist *sg; + size_t len; + int iter, i; + + /* Currently there is no proper support for scatter/gather */ + if (sg_len != 1) + return NULL; + + if (!is_slave_direction(direction)) + return NULL; + + for_each_sg(sgl, sg, sg_len, i) { + spin_lock_irqsave(&mchan->lock, iflags); + + mdesc = list_first_entry(&mchan->free, + struct mpc_dma_desc, node); + if (!mdesc) { + spin_unlock_irqrestore(&mchan->lock, iflags); + /* Try to free completed descriptors */ + mpc_dma_process_completed(mdma); + return NULL; + } + + list_del(&mdesc->node); + + if (direction == DMA_DEV_TO_MEM) { + per_paddr = mchan->src_per_paddr; + tcd_nunits = mchan->src_tcd_nunits; + } else { + per_paddr = mchan->dst_per_paddr; + tcd_nunits = mchan->dst_tcd_nunits; + } + + spin_unlock_irqrestore(&mchan->lock, iflags); + + if (per_paddr == 0 || tcd_nunits == 0) + goto err_prep; + + mdesc->error = 0; + mdesc->will_access_peripheral = 1; + + /* Prepare Transfer Control Descriptor for this transaction */ + tcd = mdesc->tcd; + + memset(tcd, 0, sizeof(struct mpc_dma_tcd)); + + if (!IS_ALIGNED(sg_dma_address(sg), 4)) + goto err_prep; + + if (direction == DMA_DEV_TO_MEM) { + tcd->saddr = per_paddr; + tcd->daddr = sg_dma_address(sg); + tcd->soff = 0; + tcd->doff = 4; + } else { + tcd->saddr = sg_dma_address(sg); + tcd->daddr = per_paddr; + tcd->soff = 4; + tcd->doff = 0; + } + + tcd->ssize = MPC_DMA_TSIZE_4; + tcd->dsize = MPC_DMA_TSIZE_4; + + len = sg_dma_len(sg); + tcd->nbytes = tcd_nunits * 4; + if (!IS_ALIGNED(len, tcd->nbytes)) + goto err_prep; + + iter = len / tcd->nbytes; + if (iter >= 1 << 15) { + /* len is too big */ + goto err_prep; + } + /* citer_linkch contains the high bits of iter */ + tcd->biter = iter & 0x1ff; + tcd->biter_linkch = iter >> 9; + tcd->citer = tcd->biter; + tcd->citer_linkch = tcd->biter_linkch; + + tcd->e_sg = 0; + tcd->d_req = 1; + + /* Place descriptor in prepared list */ + spin_lock_irqsave(&mchan->lock, iflags); + list_add_tail(&mdesc->node, &mchan->prepared); + spin_unlock_irqrestore(&mchan->lock, iflags); + } + + return &mdesc->desc; + +err_prep: + /* Put the descriptor back */ + spin_lock_irqsave(&mchan->lock, iflags); + list_add_tail(&mdesc->node, &mchan->free); + spin_unlock_irqrestore(&mchan->lock, iflags); + + return NULL; +} + +static int mpc_dma_device_control(struct dma_chan *chan, enum dma_ctrl_cmd cmd, + unsigned long arg) +{ + struct mpc_dma_chan *mchan; + struct mpc_dma *mdma; + struct dma_slave_config *cfg; + unsigned long flags; + + mchan = dma_chan_to_mpc_dma_chan(chan); + switch (cmd) { + case DMA_TERMINATE_ALL: + /* Disable channel requests */ + mdma = dma_chan_to_mpc_dma(chan); + + spin_lock_irqsave(&mchan->lock, flags); + + out_8(&mdma->regs->dmacerq, chan->chan_id); + list_splice_tail_init(&mchan->prepared, &mchan->free); + list_splice_tail_init(&mchan->queued, &mchan->free); + list_splice_tail_init(&mchan->active, &mchan->free); + + spin_unlock_irqrestore(&mchan->lock, flags); + + return 0; + + case DMA_SLAVE_CONFIG: + /* + * Software constraints: + * - only transfers between a peripheral device and + * memory are supported; + * - only peripheral devices with 4-byte FIFO access register + * are supported; + * - minimal transfer chunk is 4 bytes and consequently + * source and destination addresses must be 4-byte aligned + * and transfer size must be aligned on (4 * maxburst) + * boundary; + * - during the transfer RAM address is being incremented by + * the size of minimal transfer chunk; + * - peripheral port's address is constant during the transfer. + */ + + cfg = (void *)arg; + + if (cfg->src_addr_width != DMA_SLAVE_BUSWIDTH_4_BYTES || + cfg->dst_addr_width != DMA_SLAVE_BUSWIDTH_4_BYTES || + !IS_ALIGNED(cfg->src_addr, 4) || + !IS_ALIGNED(cfg->dst_addr, 4)) { + return -EINVAL; + } + + spin_lock_irqsave(&mchan->lock, flags); + + mchan->src_per_paddr = cfg->src_addr; + mchan->src_tcd_nunits = cfg->src_maxburst; + mchan->dst_per_paddr = cfg->dst_addr; + mchan->dst_tcd_nunits = cfg->dst_maxburst; + + /* Apply defaults */ + if (mchan->src_tcd_nunits == 0) + mchan->src_tcd_nunits = 1; + if (mchan->dst_tcd_nunits == 0) + mchan->dst_tcd_nunits = 1; + + spin_unlock_irqrestore(&mchan->lock, flags); + + return 0; + + default: + /* Unknown command */ + break; + } + + return -ENXIO; +} + static int mpc_dma_probe(struct platform_device *op) { struct device_node *dn = op->dev.of_node; @@ -649,13 +888,15 @@ static int mpc_dma_probe(struct platform_device *op) mdma = devm_kzalloc(dev, sizeof(struct mpc_dma), GFP_KERNEL); if (!mdma) { dev_err(dev, "Memory exhausted!\n"); - return -ENOMEM; + retval = -ENOMEM; + goto err; } mdma->irq = irq_of_parse_and_map(dn, 0); if (mdma->irq == NO_IRQ) { dev_err(dev, "Error mapping IRQ!\n"); - return -EINVAL; + retval = -EINVAL; + goto err; } if (of_device_is_compatible(dn, "fsl,mpc8308-dma")) { @@ -663,14 +904,15 @@ static int mpc_dma_probe(struct platform_device *op) mdma->irq2 = irq_of_parse_and_map(dn, 1); if (mdma->irq2 == NO_IRQ) { dev_err(dev, "Error mapping IRQ!\n"); - return -EINVAL; + retval = -EINVAL; + goto err_dispose1; } } retval = of_address_to_resource(dn, 0, &res); if (retval) { dev_err(dev, "Error parsing memory region!\n"); - return retval; + goto err_dispose2; } regs_start = res.start; @@ -678,31 +920,34 @@ static int mpc_dma_probe(struct platform_device *op) if (!devm_request_mem_region(dev, regs_start, regs_size, DRV_NAME)) { dev_err(dev, "Error requesting memory region!\n"); - return -EBUSY; + retval = -EBUSY; + goto err_dispose2; } mdma->regs = devm_ioremap(dev, regs_start, regs_size); if (!mdma->regs) { dev_err(dev, "Error mapping memory region!\n"); - return -ENOMEM; + retval = -ENOMEM; + goto err_dispose2; } mdma->tcd = (struct mpc_dma_tcd *)((u8 *)(mdma->regs) + MPC_DMA_TCD_OFFSET); - retval = devm_request_irq(dev, mdma->irq, &mpc_dma_irq, 0, DRV_NAME, - mdma); + retval = request_irq(mdma->irq, &mpc_dma_irq, 0, DRV_NAME, mdma); if (retval) { dev_err(dev, "Error requesting IRQ!\n"); - return -EINVAL; + retval = -EINVAL; + goto err_dispose2; } if (mdma->is_mpc8308) { - retval = devm_request_irq(dev, mdma->irq2, &mpc_dma_irq, 0, - DRV_NAME, mdma); + retval = request_irq(mdma->irq2, &mpc_dma_irq, 0, + DRV_NAME, mdma); if (retval) { dev_err(dev, "Error requesting IRQ2!\n"); - return -EINVAL; + retval = -EINVAL; + goto err_free1; } } @@ -710,18 +955,21 @@ static int mpc_dma_probe(struct platform_device *op) dma = &mdma->dma; dma->dev = dev; - if (!mdma->is_mpc8308) - dma->chancnt = MPC_DMA_CHANNELS; + if (mdma->is_mpc8308) + dma->chancnt = MPC8308_DMACHAN_MAX; else - dma->chancnt = 16; /* MPC8308 DMA has only 16 channels */ + dma->chancnt = MPC512x_DMACHAN_MAX; dma->device_alloc_chan_resources = mpc_dma_alloc_chan_resources; dma->device_free_chan_resources = mpc_dma_free_chan_resources; dma->device_issue_pending = mpc_dma_issue_pending; dma->device_tx_status = mpc_dma_tx_status; dma->device_prep_dma_memcpy = mpc_dma_prep_memcpy; + dma->device_prep_slave_sg = mpc_dma_prep_slave_sg; + dma->device_control = mpc_dma_device_control; INIT_LIST_HEAD(&dma->channels); dma_cap_set(DMA_MEMCPY, dma->cap_mask); + dma_cap_set(DMA_SLAVE, dma->cap_mask); for (i = 0; i < dma->chancnt; i++) { mchan = &mdma->channels[i]; @@ -747,7 +995,19 @@ static int mpc_dma_probe(struct platform_device *op) * - Round-robin group arbitration, * - Round-robin channel arbitration. */ - if (!mdma->is_mpc8308) { + if (mdma->is_mpc8308) { + /* MPC8308 has 16 channels and lacks some registers */ + out_be32(&mdma->regs->dmacr, MPC_DMA_DMACR_ERCA); + + /* enable snooping */ + out_be32(&mdma->regs->dmagpor, MPC_DMA_DMAGPOR_SNOOP_ENABLE); + /* Disable error interrupts */ + out_be32(&mdma->regs->dmaeeil, 0); + + /* Clear interrupts status */ + out_be32(&mdma->regs->dmaintl, 0xFFFF); + out_be32(&mdma->regs->dmaerrl, 0xFFFF); + } else { out_be32(&mdma->regs->dmacr, MPC_DMA_DMACR_EDCG | MPC_DMA_DMACR_ERGA | MPC_DMA_DMACR_ERCA); @@ -768,29 +1028,28 @@ static int mpc_dma_probe(struct platform_device *op) /* Route interrupts to IPIC */ out_be32(&mdma->regs->dmaihsa, 0); out_be32(&mdma->regs->dmailsa, 0); - } else { - /* MPC8308 has 16 channels and lacks some registers */ - out_be32(&mdma->regs->dmacr, MPC_DMA_DMACR_ERCA); - - /* enable snooping */ - out_be32(&mdma->regs->dmagpor, MPC_DMA_DMAGPOR_SNOOP_ENABLE); - /* Disable error interrupts */ - out_be32(&mdma->regs->dmaeeil, 0); - - /* Clear interrupts status */ - out_be32(&mdma->regs->dmaintl, 0xFFFF); - out_be32(&mdma->regs->dmaerrl, 0xFFFF); } /* Register DMA engine */ dev_set_drvdata(dev, mdma); retval = dma_async_device_register(dma); - if (retval) { - devm_free_irq(dev, mdma->irq, mdma); - irq_dispose_mapping(mdma->irq); - } + if (retval) + goto err_free2; return retval; + +err_free2: + if (mdma->is_mpc8308) + free_irq(mdma->irq2, mdma); +err_free1: + free_irq(mdma->irq, mdma); +err_dispose2: + if (mdma->is_mpc8308) + irq_dispose_mapping(mdma->irq2); +err_dispose1: + irq_dispose_mapping(mdma->irq); +err: + return retval; } static int mpc_dma_remove(struct platform_device *op) @@ -799,7 +1058,11 @@ static int mpc_dma_remove(struct platform_device *op) struct mpc_dma *mdma = dev_get_drvdata(dev); dma_async_device_unregister(&mdma->dma); - devm_free_irq(dev, mdma->irq, mdma); + if (mdma->is_mpc8308) { + free_irq(mdma->irq2, mdma); + irq_dispose_mapping(mdma->irq2); + } + free_irq(mdma->irq, mdma); irq_dispose_mapping(mdma->irq); return 0; @@ -807,6 +1070,7 @@ static int mpc_dma_remove(struct platform_device *op) static struct of_device_id mpc_dma_match[] = { { .compatible = "fsl,mpc5121-dma", }, + { .compatible = "fsl,mpc8308-dma", }, {}, }; diff --git a/drivers/dma/pch_dma.c b/drivers/dma/pch_dma.c index 05fa548bd659..9f9ca9fe5ce6 100644 --- a/drivers/dma/pch_dma.c +++ b/drivers/dma/pch_dma.c @@ -21,6 +21,7 @@ #include <linux/dma-mapping.h> #include <linux/init.h> #include <linux/pci.h> +#include <linux/slab.h> #include <linux/interrupt.h> #include <linux/module.h> #include <linux/pch_dma.h> @@ -996,7 +997,7 @@ static void pch_dma_remove(struct pci_dev *pdev) #define PCI_DEVICE_ID_ML7831_DMA1_8CH 0x8810 #define PCI_DEVICE_ID_ML7831_DMA2_4CH 0x8815 -DEFINE_PCI_DEVICE_TABLE(pch_dma_id_table) = { +const struct pci_device_id pch_dma_id_table[] = { { PCI_VDEVICE(INTEL, PCI_DEVICE_ID_EG20T_PCH_DMA_8CH), 8 }, { PCI_VDEVICE(INTEL, PCI_DEVICE_ID_EG20T_PCH_DMA_4CH), 4 }, { PCI_VDEVICE(ROHM, PCI_DEVICE_ID_ML7213_DMA1_8CH), 8}, /* UART Video */ diff --git a/drivers/dma/s3c24xx-dma.c b/drivers/dma/s3c24xx-dma.c index b209a0f17344..012520c9fd79 100644 --- a/drivers/dma/s3c24xx-dma.c +++ b/drivers/dma/s3c24xx-dma.c @@ -164,6 +164,7 @@ struct s3c24xx_sg { * @disrcc: value for source control register * @didstc: value for destination control register * @dcon: base value for dcon register + * @cyclic: indicate cyclic transfer */ struct s3c24xx_txd { struct virt_dma_desc vd; @@ -173,6 +174,7 @@ struct s3c24xx_txd { u32 disrcc; u32 didstc; u32 dcon; + bool cyclic; }; struct s3c24xx_dma_chan; @@ -669,8 +671,10 @@ static irqreturn_t s3c24xx_dma_irq(int irq, void *data) /* when more sg's are in this txd, start the next one */ if (!list_is_last(txd->at, &txd->dsg_list)) { txd->at = txd->at->next; + if (txd->cyclic) + vchan_cyclic_callback(&txd->vd); s3c24xx_dma_start_next_sg(s3cchan, txd); - } else { + } else if (!txd->cyclic) { s3cchan->at = NULL; vchan_cookie_complete(&txd->vd); @@ -682,6 +686,12 @@ static irqreturn_t s3c24xx_dma_irq(int irq, void *data) s3c24xx_dma_start_next_txd(s3cchan); else s3c24xx_dma_phy_free(s3cchan); + } else { + vchan_cyclic_callback(&txd->vd); + + /* Cyclic: reset at beginning */ + txd->at = txd->dsg_list.next; + s3c24xx_dma_start_next_sg(s3cchan, txd); } } spin_unlock(&s3cchan->vc.lock); @@ -877,6 +887,104 @@ static struct dma_async_tx_descriptor *s3c24xx_dma_prep_memcpy( return vchan_tx_prep(&s3cchan->vc, &txd->vd, flags); } +static struct dma_async_tx_descriptor *s3c24xx_dma_prep_dma_cyclic( + struct dma_chan *chan, dma_addr_t addr, size_t size, size_t period, + enum dma_transfer_direction direction, unsigned long flags, + void *context) +{ + struct s3c24xx_dma_chan *s3cchan = to_s3c24xx_dma_chan(chan); + struct s3c24xx_dma_engine *s3cdma = s3cchan->host; + const struct s3c24xx_dma_platdata *pdata = s3cdma->pdata; + struct s3c24xx_dma_channel *cdata = &pdata->channels[s3cchan->id]; + struct s3c24xx_txd *txd; + struct s3c24xx_sg *dsg; + unsigned sg_len; + dma_addr_t slave_addr; + u32 hwcfg = 0; + int i; + + dev_dbg(&s3cdma->pdev->dev, + "prepare cyclic transaction of %zu bytes with period %zu from %s\n", + size, period, s3cchan->name); + + if (!is_slave_direction(direction)) { + dev_err(&s3cdma->pdev->dev, + "direction %d unsupported\n", direction); + return NULL; + } + + txd = s3c24xx_dma_get_txd(); + if (!txd) + return NULL; + + txd->cyclic = 1; + + if (cdata->handshake) + txd->dcon |= S3C24XX_DCON_HANDSHAKE; + + switch (cdata->bus) { + case S3C24XX_DMA_APB: + txd->dcon |= S3C24XX_DCON_SYNC_PCLK; + hwcfg |= S3C24XX_DISRCC_LOC_APB; + break; + case S3C24XX_DMA_AHB: + txd->dcon |= S3C24XX_DCON_SYNC_HCLK; + hwcfg |= S3C24XX_DISRCC_LOC_AHB; + break; + } + + /* + * Always assume our peripheral desintation is a fixed + * address in memory. + */ + hwcfg |= S3C24XX_DISRCC_INC_FIXED; + + /* + * Individual dma operations are requested by the slave, + * so serve only single atomic operations (S3C24XX_DCON_SERV_SINGLE). + */ + txd->dcon |= S3C24XX_DCON_SERV_SINGLE; + + if (direction == DMA_MEM_TO_DEV) { + txd->disrcc = S3C24XX_DISRCC_LOC_AHB | + S3C24XX_DISRCC_INC_INCREMENT; + txd->didstc = hwcfg; + slave_addr = s3cchan->cfg.dst_addr; + txd->width = s3cchan->cfg.dst_addr_width; + } else { + txd->disrcc = hwcfg; + txd->didstc = S3C24XX_DIDSTC_LOC_AHB | + S3C24XX_DIDSTC_INC_INCREMENT; + slave_addr = s3cchan->cfg.src_addr; + txd->width = s3cchan->cfg.src_addr_width; + } + + sg_len = size / period; + + for (i = 0; i < sg_len; i++) { + dsg = kzalloc(sizeof(*dsg), GFP_NOWAIT); + if (!dsg) { + s3c24xx_dma_free_txd(txd); + return NULL; + } + list_add_tail(&dsg->node, &txd->dsg_list); + + dsg->len = period; + /* Check last period length */ + if (i == sg_len - 1) + dsg->len = size - period * i; + if (direction == DMA_MEM_TO_DEV) { + dsg->src_addr = addr + period * i; + dsg->dst_addr = slave_addr; + } else { /* DMA_DEV_TO_MEM */ + dsg->src_addr = slave_addr; + dsg->dst_addr = addr + period * i; + } + } + + return vchan_tx_prep(&s3cchan->vc, &txd->vd, flags); +} + static struct dma_async_tx_descriptor *s3c24xx_dma_prep_slave_sg( struct dma_chan *chan, struct scatterlist *sgl, unsigned int sg_len, enum dma_transfer_direction direction, @@ -961,7 +1069,6 @@ static struct dma_async_tx_descriptor *s3c24xx_dma_prep_slave_sg( dsg->src_addr = slave_addr; dsg->dst_addr = sg_dma_address(sg); } - break; } return vchan_tx_prep(&s3cchan->vc, &txd->vd, flags); @@ -1198,6 +1305,7 @@ static int s3c24xx_dma_probe(struct platform_device *pdev) /* Initialize slave engine for SoC internal dedicated peripherals */ dma_cap_set(DMA_SLAVE, s3cdma->slave.cap_mask); + dma_cap_set(DMA_CYCLIC, s3cdma->slave.cap_mask); dma_cap_set(DMA_PRIVATE, s3cdma->slave.cap_mask); s3cdma->slave.dev = &pdev->dev; s3cdma->slave.device_alloc_chan_resources = @@ -1207,6 +1315,7 @@ static int s3c24xx_dma_probe(struct platform_device *pdev) s3cdma->slave.device_tx_status = s3c24xx_dma_tx_status; s3cdma->slave.device_issue_pending = s3c24xx_dma_issue_pending; s3cdma->slave.device_prep_slave_sg = s3c24xx_dma_prep_slave_sg; + s3cdma->slave.device_prep_dma_cyclic = s3c24xx_dma_prep_dma_cyclic; s3cdma->slave.device_control = s3c24xx_dma_control; /* Register as many memcpy channels as there are physical channels */ diff --git a/drivers/dma/sh/Kconfig b/drivers/dma/sh/Kconfig index b4c813831006..0f719816c91b 100644 --- a/drivers/dma/sh/Kconfig +++ b/drivers/dma/sh/Kconfig @@ -4,7 +4,7 @@ config SH_DMAE_BASE bool "Renesas SuperH DMA Engine support" - depends on (SUPERH && SH_DMA) || (ARM && ARCH_SHMOBILE) + depends on (SUPERH && SH_DMA) || ARCH_SHMOBILE || COMPILE_TEST depends on !SH_DMA_API default y select DMA_ENGINE diff --git a/drivers/dma/sh/rcar-hpbdma.c b/drivers/dma/sh/rcar-hpbdma.c index 3083d901a414..b212d9471ab5 100644 --- a/drivers/dma/sh/rcar-hpbdma.c +++ b/drivers/dma/sh/rcar-hpbdma.c @@ -18,6 +18,7 @@ #include <linux/dmaengine.h> #include <linux/delay.h> +#include <linux/err.h> #include <linux/init.h> #include <linux/interrupt.h> #include <linux/module.h> diff --git a/drivers/dma/sh/shdma-base.c b/drivers/dma/sh/shdma-base.c index 52396771acbe..b35007e21e6b 100644 --- a/drivers/dma/sh/shdma-base.c +++ b/drivers/dma/sh/shdma-base.c @@ -73,8 +73,7 @@ static void shdma_chan_xfer_ld_queue(struct shdma_chan *schan) static dma_cookie_t shdma_tx_submit(struct dma_async_tx_descriptor *tx) { struct shdma_desc *chunk, *c, *desc = - container_of(tx, struct shdma_desc, async_tx), - *last = desc; + container_of(tx, struct shdma_desc, async_tx); struct shdma_chan *schan = to_shdma_chan(tx->chan); dma_async_tx_callback callback = tx->callback; dma_cookie_t cookie; @@ -98,19 +97,20 @@ static dma_cookie_t shdma_tx_submit(struct dma_async_tx_descriptor *tx) &chunk->node == &schan->ld_free)) break; chunk->mark = DESC_SUBMITTED; - /* Callback goes to the last chunk */ - chunk->async_tx.callback = NULL; + if (chunk->chunks == 1) { + chunk->async_tx.callback = callback; + chunk->async_tx.callback_param = tx->callback_param; + } else { + /* Callback goes to the last chunk */ + chunk->async_tx.callback = NULL; + } chunk->cookie = cookie; list_move_tail(&chunk->node, &schan->ld_queue); - last = chunk; dev_dbg(schan->dev, "submit #%d@%p on %d\n", - tx->cookie, &last->async_tx, schan->id); + tx->cookie, &chunk->async_tx, schan->id); } - last->async_tx.callback = callback; - last->async_tx.callback_param = tx->callback_param; - if (power_up) { int ret; schan->pm_state = SHDMA_PM_BUSY; @@ -304,6 +304,7 @@ static dma_async_tx_callback __ld_cleanup(struct shdma_chan *schan, bool all) dma_async_tx_callback callback = NULL; void *param = NULL; unsigned long flags; + LIST_HEAD(cyclic_list); spin_lock_irqsave(&schan->chan_lock, flags); list_for_each_entry_safe(desc, _desc, &schan->ld_queue, node) { @@ -369,10 +370,16 @@ static dma_async_tx_callback __ld_cleanup(struct shdma_chan *schan, bool all) if (((desc->mark == DESC_COMPLETED || desc->mark == DESC_WAITING) && async_tx_test_ack(&desc->async_tx)) || all) { - /* Remove from ld_queue list */ - desc->mark = DESC_IDLE; - list_move(&desc->node, &schan->ld_free); + if (all || !desc->cyclic) { + /* Remove from ld_queue list */ + desc->mark = DESC_IDLE; + list_move(&desc->node, &schan->ld_free); + } else { + /* reuse as cyclic */ + desc->mark = DESC_SUBMITTED; + list_move_tail(&desc->node, &cyclic_list); + } if (list_empty(&schan->ld_queue)) { dev_dbg(schan->dev, "Bring down channel %d\n", schan->id); @@ -389,6 +396,8 @@ static dma_async_tx_callback __ld_cleanup(struct shdma_chan *schan, bool all) */ schan->dma_chan.completed_cookie = schan->dma_chan.cookie; + list_splice_tail(&cyclic_list, &schan->ld_queue); + spin_unlock_irqrestore(&schan->chan_lock, flags); if (callback) @@ -521,7 +530,7 @@ static struct shdma_desc *shdma_add_desc(struct shdma_chan *schan, */ static struct dma_async_tx_descriptor *shdma_prep_sg(struct shdma_chan *schan, struct scatterlist *sgl, unsigned int sg_len, dma_addr_t *addr, - enum dma_transfer_direction direction, unsigned long flags) + enum dma_transfer_direction direction, unsigned long flags, bool cyclic) { struct scatterlist *sg; struct shdma_desc *first = NULL, *new = NULL /* compiler... */; @@ -569,7 +578,11 @@ static struct dma_async_tx_descriptor *shdma_prep_sg(struct shdma_chan *schan, if (!new) goto err_get_desc; - new->chunks = chunks--; + new->cyclic = cyclic; + if (cyclic) + new->chunks = 1; + else + new->chunks = chunks--; list_add_tail(&new->node, &tx_list); } while (len); } @@ -612,7 +625,8 @@ static struct dma_async_tx_descriptor *shdma_prep_memcpy( sg_dma_address(&sg) = dma_src; sg_dma_len(&sg) = len; - return shdma_prep_sg(schan, &sg, 1, &dma_dest, DMA_MEM_TO_MEM, flags); + return shdma_prep_sg(schan, &sg, 1, &dma_dest, DMA_MEM_TO_MEM, + flags, false); } static struct dma_async_tx_descriptor *shdma_prep_slave_sg( @@ -640,7 +654,58 @@ static struct dma_async_tx_descriptor *shdma_prep_slave_sg( slave_addr = ops->slave_addr(schan); return shdma_prep_sg(schan, sgl, sg_len, &slave_addr, - direction, flags); + direction, flags, false); +} + +#define SHDMA_MAX_SG_LEN 32 + +static struct dma_async_tx_descriptor *shdma_prep_dma_cyclic( + struct dma_chan *chan, dma_addr_t buf_addr, size_t buf_len, + size_t period_len, enum dma_transfer_direction direction, + unsigned long flags, void *context) +{ + struct shdma_chan *schan = to_shdma_chan(chan); + struct shdma_dev *sdev = to_shdma_dev(schan->dma_chan.device); + const struct shdma_ops *ops = sdev->ops; + unsigned int sg_len = buf_len / period_len; + int slave_id = schan->slave_id; + dma_addr_t slave_addr; + struct scatterlist sgl[SHDMA_MAX_SG_LEN]; + int i; + + if (!chan) + return NULL; + + BUG_ON(!schan->desc_num); + + if (sg_len > SHDMA_MAX_SG_LEN) { + dev_err(schan->dev, "sg length %d exceds limit %d", + sg_len, SHDMA_MAX_SG_LEN); + return NULL; + } + + /* Someone calling slave DMA on a generic channel? */ + if (slave_id < 0 || (buf_len < period_len)) { + dev_warn(schan->dev, + "%s: bad parameter: buf_len=%zu, period_len=%zu, id=%d\n", + __func__, buf_len, period_len, slave_id); + return NULL; + } + + slave_addr = ops->slave_addr(schan); + + sg_init_table(sgl, sg_len); + for (i = 0; i < sg_len; i++) { + dma_addr_t src = buf_addr + (period_len * i); + + sg_set_page(&sgl[i], pfn_to_page(PFN_DOWN(src)), period_len, + offset_in_page(src)); + sg_dma_address(&sgl[i]) = src; + sg_dma_len(&sgl[i]) = period_len; + } + + return shdma_prep_sg(schan, sgl, sg_len, &slave_addr, + direction, flags, true); } static int shdma_control(struct dma_chan *chan, enum dma_ctrl_cmd cmd, @@ -915,6 +980,7 @@ int shdma_init(struct device *dev, struct shdma_dev *sdev, /* Compulsory for DMA_SLAVE fields */ dma_dev->device_prep_slave_sg = shdma_prep_slave_sg; + dma_dev->device_prep_dma_cyclic = shdma_prep_dma_cyclic; dma_dev->device_control = shdma_control; dma_dev->dev = dev; diff --git a/drivers/dma/sh/shdmac.c b/drivers/dma/sh/shdmac.c index dda7e7563f5d..146d5df926db 100644 --- a/drivers/dma/sh/shdmac.c +++ b/drivers/dma/sh/shdmac.c @@ -18,21 +18,22 @@ * */ +#include <linux/delay.h> +#include <linux/dmaengine.h> +#include <linux/err.h> #include <linux/init.h> +#include <linux/interrupt.h> +#include <linux/kdebug.h> #include <linux/module.h> +#include <linux/notifier.h> #include <linux/of.h> #include <linux/of_device.h> -#include <linux/slab.h> -#include <linux/interrupt.h> -#include <linux/dmaengine.h> -#include <linux/delay.h> #include <linux/platform_device.h> #include <linux/pm_runtime.h> +#include <linux/rculist.h> #include <linux/sh_dma.h> -#include <linux/notifier.h> -#include <linux/kdebug.h> +#include <linux/slab.h> #include <linux/spinlock.h> -#include <linux/rculist.h> #include "../dmaengine.h" #include "shdma.h" diff --git a/drivers/dma/sh/sudmac.c b/drivers/dma/sh/sudmac.c index 4e7df43b50d6..3ce103909896 100644 --- a/drivers/dma/sh/sudmac.c +++ b/drivers/dma/sh/sudmac.c @@ -14,12 +14,13 @@ * published by the Free Software Foundation. */ +#include <linux/dmaengine.h> +#include <linux/err.h> #include <linux/init.h> -#include <linux/module.h> -#include <linux/slab.h> #include <linux/interrupt.h> -#include <linux/dmaengine.h> +#include <linux/module.h> #include <linux/platform_device.h> +#include <linux/slab.h> #include <linux/sudmac.h> struct sudmac_chan { diff --git a/drivers/dma/ste_dma40.c b/drivers/dma/ste_dma40.c index bf18c786ed40..c7984459ede7 100644 --- a/drivers/dma/ste_dma40.c +++ b/drivers/dma/ste_dma40.c @@ -556,7 +556,6 @@ struct d40_gen_dmac { * later * @reg_val_backup_chan: Backup data for standard channel parameter registers. * @gcc_pwr_off_mask: Mask to maintain the channels that can be turned off. - * @initialized: true if the dma has been initialized * @gen_dmac: the struct for generic registers values to represent u8500/8540 * DMA controller */ @@ -594,7 +593,6 @@ struct d40_base { u32 reg_val_backup_v4[BACKUP_REGS_SZ_MAX]; u32 *reg_val_backup_chan; u16 gcc_pwr_off_mask; - bool initialized; struct d40_gen_dmac gen_dmac; }; @@ -1056,62 +1054,6 @@ static int d40_sg_2_dmalen(struct scatterlist *sgl, int sg_len, return len; } - -#ifdef CONFIG_PM -static void dma40_backup(void __iomem *baseaddr, u32 *backup, - u32 *regaddr, int num, bool save) -{ - int i; - - for (i = 0; i < num; i++) { - void __iomem *addr = baseaddr + regaddr[i]; - - if (save) - backup[i] = readl_relaxed(addr); - else - writel_relaxed(backup[i], addr); - } -} - -static void d40_save_restore_registers(struct d40_base *base, bool save) -{ - int i; - - /* Save/Restore channel specific registers */ - for (i = 0; i < base->num_phy_chans; i++) { - void __iomem *addr; - int idx; - - if (base->phy_res[i].reserved) - continue; - - addr = base->virtbase + D40_DREG_PCBASE + i * D40_DREG_PCDELTA; - idx = i * ARRAY_SIZE(d40_backup_regs_chan); - - dma40_backup(addr, &base->reg_val_backup_chan[idx], - d40_backup_regs_chan, - ARRAY_SIZE(d40_backup_regs_chan), - save); - } - - /* Save/Restore global registers */ - dma40_backup(base->virtbase, base->reg_val_backup, - d40_backup_regs, ARRAY_SIZE(d40_backup_regs), - save); - - /* Save/Restore registers only existing on dma40 v3 and later */ - if (base->gen_dmac.backup) - dma40_backup(base->virtbase, base->reg_val_backup_v4, - base->gen_dmac.backup, - base->gen_dmac.backup_size, - save); -} -#else -static void d40_save_restore_registers(struct d40_base *base, bool save) -{ -} -#endif - static int __d40_execute_command_phy(struct d40_chan *d40c, enum d40_command command) { @@ -1495,8 +1437,8 @@ static int d40_pause(struct d40_chan *d40c) if (!d40c->busy) return 0; - pm_runtime_get_sync(d40c->base->dev); spin_lock_irqsave(&d40c->lock, flags); + pm_runtime_get_sync(d40c->base->dev); res = d40_channel_execute_command(d40c, D40_DMA_SUSPEND_REQ); @@ -2998,18 +2940,88 @@ failure1: } /* Suspend resume functionality */ -#ifdef CONFIG_PM -static int dma40_pm_suspend(struct device *dev) +#ifdef CONFIG_PM_SLEEP +static int dma40_suspend(struct device *dev) { struct platform_device *pdev = to_platform_device(dev); struct d40_base *base = platform_get_drvdata(pdev); - int ret = 0; + int ret; + + ret = pm_runtime_force_suspend(dev); + if (ret) + return ret; if (base->lcpa_regulator) ret = regulator_disable(base->lcpa_regulator); return ret; } +static int dma40_resume(struct device *dev) +{ + struct platform_device *pdev = to_platform_device(dev); + struct d40_base *base = platform_get_drvdata(pdev); + int ret = 0; + + if (base->lcpa_regulator) { + ret = regulator_enable(base->lcpa_regulator); + if (ret) + return ret; + } + + return pm_runtime_force_resume(dev); +} +#endif + +#ifdef CONFIG_PM +static void dma40_backup(void __iomem *baseaddr, u32 *backup, + u32 *regaddr, int num, bool save) +{ + int i; + + for (i = 0; i < num; i++) { + void __iomem *addr = baseaddr + regaddr[i]; + + if (save) + backup[i] = readl_relaxed(addr); + else + writel_relaxed(backup[i], addr); + } +} + +static void d40_save_restore_registers(struct d40_base *base, bool save) +{ + int i; + + /* Save/Restore channel specific registers */ + for (i = 0; i < base->num_phy_chans; i++) { + void __iomem *addr; + int idx; + + if (base->phy_res[i].reserved) + continue; + + addr = base->virtbase + D40_DREG_PCBASE + i * D40_DREG_PCDELTA; + idx = i * ARRAY_SIZE(d40_backup_regs_chan); + + dma40_backup(addr, &base->reg_val_backup_chan[idx], + d40_backup_regs_chan, + ARRAY_SIZE(d40_backup_regs_chan), + save); + } + + /* Save/Restore global registers */ + dma40_backup(base->virtbase, base->reg_val_backup, + d40_backup_regs, ARRAY_SIZE(d40_backup_regs), + save); + + /* Save/Restore registers only existing on dma40 v3 and later */ + if (base->gen_dmac.backup) + dma40_backup(base->virtbase, base->reg_val_backup_v4, + base->gen_dmac.backup, + base->gen_dmac.backup_size, + save); +} + static int dma40_runtime_suspend(struct device *dev) { struct platform_device *pdev = to_platform_device(dev); @@ -3030,36 +3042,20 @@ static int dma40_runtime_resume(struct device *dev) struct platform_device *pdev = to_platform_device(dev); struct d40_base *base = platform_get_drvdata(pdev); - if (base->initialized) - d40_save_restore_registers(base, false); + d40_save_restore_registers(base, false); writel_relaxed(D40_DREG_GCC_ENABLE_ALL, base->virtbase + D40_DREG_GCC); return 0; } - -static int dma40_resume(struct device *dev) -{ - struct platform_device *pdev = to_platform_device(dev); - struct d40_base *base = platform_get_drvdata(pdev); - int ret = 0; - - if (base->lcpa_regulator) - ret = regulator_enable(base->lcpa_regulator); - - return ret; -} +#endif static const struct dev_pm_ops dma40_pm_ops = { - .suspend = dma40_pm_suspend, - .runtime_suspend = dma40_runtime_suspend, - .runtime_resume = dma40_runtime_resume, - .resume = dma40_resume, + SET_LATE_SYSTEM_SLEEP_PM_OPS(dma40_suspend, dma40_resume) + SET_PM_RUNTIME_PM_OPS(dma40_runtime_suspend, + dma40_runtime_resume, + NULL) }; -#define DMA40_PM_OPS (&dma40_pm_ops) -#else -#define DMA40_PM_OPS NULL -#endif /* Initialization functions. */ @@ -3645,12 +3641,6 @@ static int __init d40_probe(struct platform_device *pdev) goto failure; } - pm_runtime_irq_safe(base->dev); - pm_runtime_set_autosuspend_delay(base->dev, DMA40_AUTOSUSPEND_DELAY); - pm_runtime_use_autosuspend(base->dev); - pm_runtime_enable(base->dev); - pm_runtime_resume(base->dev); - if (base->plat_data->use_esram_lcla) { base->lcpa_regulator = regulator_get(base->dev, "lcla_esram"); @@ -3671,7 +3661,15 @@ static int __init d40_probe(struct platform_device *pdev) } } - base->initialized = true; + writel_relaxed(D40_DREG_GCC_ENABLE_ALL, base->virtbase + D40_DREG_GCC); + + pm_runtime_irq_safe(base->dev); + pm_runtime_set_autosuspend_delay(base->dev, DMA40_AUTOSUSPEND_DELAY); + pm_runtime_use_autosuspend(base->dev); + pm_runtime_mark_last_busy(base->dev); + pm_runtime_set_active(base->dev); + pm_runtime_enable(base->dev); + ret = d40_dmaengine_init(base, num_reserved_chans); if (ret) goto failure; @@ -3754,7 +3752,7 @@ static struct platform_driver d40_driver = { .driver = { .owner = THIS_MODULE, .name = D40_NAME, - .pm = DMA40_PM_OPS, + .pm = &dma40_pm_ops, .of_match_table = d40_match, }, }; diff --git a/drivers/dma/xilinx/Makefile b/drivers/dma/xilinx/Makefile new file mode 100644 index 000000000000..3c4e9f2fea28 --- /dev/null +++ b/drivers/dma/xilinx/Makefile @@ -0,0 +1 @@ +obj-$(CONFIG_XILINX_VDMA) += xilinx_vdma.o diff --git a/drivers/dma/xilinx/xilinx_vdma.c b/drivers/dma/xilinx/xilinx_vdma.c new file mode 100644 index 000000000000..42a13e8d4607 --- /dev/null +++ b/drivers/dma/xilinx/xilinx_vdma.c @@ -0,0 +1,1379 @@ +/* + * DMA driver for Xilinx Video DMA Engine + * + * Copyright (C) 2010-2014 Xilinx, Inc. All rights reserved. + * + * Based on the Freescale DMA driver. + * + * Description: + * The AXI Video Direct Memory Access (AXI VDMA) core is a soft Xilinx IP + * core that provides high-bandwidth direct memory access between memory + * and AXI4-Stream type video target peripherals. The core provides efficient + * two dimensional DMA operations with independent asynchronous read (S2MM) + * and write (MM2S) channel operation. It can be configured to have either + * one channel or two channels. If configured as two channels, one is to + * transmit to the video device (MM2S) and another is to receive from the + * video device (S2MM). Initialization, status, interrupt and management + * registers are accessed through an AXI4-Lite slave interface. + * + * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 2 of the License, or + * (at your option) any later version. + */ + +#include <linux/amba/xilinx_dma.h> +#include <linux/bitops.h> +#include <linux/dmapool.h> +#include <linux/init.h> +#include <linux/interrupt.h> +#include <linux/io.h> +#include <linux/module.h> +#include <linux/of_address.h> +#include <linux/of_dma.h> +#include <linux/of_platform.h> +#include <linux/of_irq.h> +#include <linux/slab.h> + +#include "../dmaengine.h" + +/* Register/Descriptor Offsets */ +#define XILINX_VDMA_MM2S_CTRL_OFFSET 0x0000 +#define XILINX_VDMA_S2MM_CTRL_OFFSET 0x0030 +#define XILINX_VDMA_MM2S_DESC_OFFSET 0x0050 +#define XILINX_VDMA_S2MM_DESC_OFFSET 0x00a0 + +/* Control Registers */ +#define XILINX_VDMA_REG_DMACR 0x0000 +#define XILINX_VDMA_DMACR_DELAY_MAX 0xff +#define XILINX_VDMA_DMACR_DELAY_SHIFT 24 +#define XILINX_VDMA_DMACR_FRAME_COUNT_MAX 0xff +#define XILINX_VDMA_DMACR_FRAME_COUNT_SHIFT 16 +#define XILINX_VDMA_DMACR_ERR_IRQ BIT(14) +#define XILINX_VDMA_DMACR_DLY_CNT_IRQ BIT(13) +#define XILINX_VDMA_DMACR_FRM_CNT_IRQ BIT(12) +#define XILINX_VDMA_DMACR_MASTER_SHIFT 8 +#define XILINX_VDMA_DMACR_FSYNCSRC_SHIFT 5 +#define XILINX_VDMA_DMACR_FRAMECNT_EN BIT(4) +#define XILINX_VDMA_DMACR_GENLOCK_EN BIT(3) +#define XILINX_VDMA_DMACR_RESET BIT(2) +#define XILINX_VDMA_DMACR_CIRC_EN BIT(1) +#define XILINX_VDMA_DMACR_RUNSTOP BIT(0) +#define XILINX_VDMA_DMACR_FSYNCSRC_MASK GENMASK(6, 5) + +#define XILINX_VDMA_REG_DMASR 0x0004 +#define XILINX_VDMA_DMASR_EOL_LATE_ERR BIT(15) +#define XILINX_VDMA_DMASR_ERR_IRQ BIT(14) +#define XILINX_VDMA_DMASR_DLY_CNT_IRQ BIT(13) +#define XILINX_VDMA_DMASR_FRM_CNT_IRQ BIT(12) +#define XILINX_VDMA_DMASR_SOF_LATE_ERR BIT(11) +#define XILINX_VDMA_DMASR_SG_DEC_ERR BIT(10) +#define XILINX_VDMA_DMASR_SG_SLV_ERR BIT(9) +#define XILINX_VDMA_DMASR_EOF_EARLY_ERR BIT(8) +#define XILINX_VDMA_DMASR_SOF_EARLY_ERR BIT(7) +#define XILINX_VDMA_DMASR_DMA_DEC_ERR BIT(6) +#define XILINX_VDMA_DMASR_DMA_SLAVE_ERR BIT(5) +#define XILINX_VDMA_DMASR_DMA_INT_ERR BIT(4) +#define XILINX_VDMA_DMASR_IDLE BIT(1) +#define XILINX_VDMA_DMASR_HALTED BIT(0) +#define XILINX_VDMA_DMASR_DELAY_MASK GENMASK(31, 24) +#define XILINX_VDMA_DMASR_FRAME_COUNT_MASK GENMASK(23, 16) + +#define XILINX_VDMA_REG_CURDESC 0x0008 +#define XILINX_VDMA_REG_TAILDESC 0x0010 +#define XILINX_VDMA_REG_REG_INDEX 0x0014 +#define XILINX_VDMA_REG_FRMSTORE 0x0018 +#define XILINX_VDMA_REG_THRESHOLD 0x001c +#define XILINX_VDMA_REG_FRMPTR_STS 0x0024 +#define XILINX_VDMA_REG_PARK_PTR 0x0028 +#define XILINX_VDMA_PARK_PTR_WR_REF_SHIFT 8 +#define XILINX_VDMA_PARK_PTR_RD_REF_SHIFT 0 +#define XILINX_VDMA_REG_VDMA_VERSION 0x002c + +/* Register Direct Mode Registers */ +#define XILINX_VDMA_REG_VSIZE 0x0000 +#define XILINX_VDMA_REG_HSIZE 0x0004 + +#define XILINX_VDMA_REG_FRMDLY_STRIDE 0x0008 +#define XILINX_VDMA_FRMDLY_STRIDE_FRMDLY_SHIFT 24 +#define XILINX_VDMA_FRMDLY_STRIDE_STRIDE_SHIFT 0 + +#define XILINX_VDMA_REG_START_ADDRESS(n) (0x000c + 4 * (n)) + +/* HW specific definitions */ +#define XILINX_VDMA_MAX_CHANS_PER_DEVICE 0x2 + +#define XILINX_VDMA_DMAXR_ALL_IRQ_MASK \ + (XILINX_VDMA_DMASR_FRM_CNT_IRQ | \ + XILINX_VDMA_DMASR_DLY_CNT_IRQ | \ + XILINX_VDMA_DMASR_ERR_IRQ) + +#define XILINX_VDMA_DMASR_ALL_ERR_MASK \ + (XILINX_VDMA_DMASR_EOL_LATE_ERR | \ + XILINX_VDMA_DMASR_SOF_LATE_ERR | \ + XILINX_VDMA_DMASR_SG_DEC_ERR | \ + XILINX_VDMA_DMASR_SG_SLV_ERR | \ + XILINX_VDMA_DMASR_EOF_EARLY_ERR | \ + XILINX_VDMA_DMASR_SOF_EARLY_ERR | \ + XILINX_VDMA_DMASR_DMA_DEC_ERR | \ + XILINX_VDMA_DMASR_DMA_SLAVE_ERR | \ + XILINX_VDMA_DMASR_DMA_INT_ERR) + +/* + * Recoverable errors are DMA Internal error, SOF Early, EOF Early + * and SOF Late. They are only recoverable when C_FLUSH_ON_FSYNC + * is enabled in the h/w system. + */ +#define XILINX_VDMA_DMASR_ERR_RECOVER_MASK \ + (XILINX_VDMA_DMASR_SOF_LATE_ERR | \ + XILINX_VDMA_DMASR_EOF_EARLY_ERR | \ + XILINX_VDMA_DMASR_SOF_EARLY_ERR | \ + XILINX_VDMA_DMASR_DMA_INT_ERR) + +/* Axi VDMA Flush on Fsync bits */ +#define XILINX_VDMA_FLUSH_S2MM 3 +#define XILINX_VDMA_FLUSH_MM2S 2 +#define XILINX_VDMA_FLUSH_BOTH 1 + +/* Delay loop counter to prevent hardware failure */ +#define XILINX_VDMA_LOOP_COUNT 1000000 + +/** + * struct xilinx_vdma_desc_hw - Hardware Descriptor + * @next_desc: Next Descriptor Pointer @0x00 + * @pad1: Reserved @0x04 + * @buf_addr: Buffer address @0x08 + * @pad2: Reserved @0x0C + * @vsize: Vertical Size @0x10 + * @hsize: Horizontal Size @0x14 + * @stride: Number of bytes between the first + * pixels of each horizontal line @0x18 + */ +struct xilinx_vdma_desc_hw { + u32 next_desc; + u32 pad1; + u32 buf_addr; + u32 pad2; + u32 vsize; + u32 hsize; + u32 stride; +} __aligned(64); + +/** + * struct xilinx_vdma_tx_segment - Descriptor segment + * @hw: Hardware descriptor + * @node: Node in the descriptor segments list + * @phys: Physical address of segment + */ +struct xilinx_vdma_tx_segment { + struct xilinx_vdma_desc_hw hw; + struct list_head node; + dma_addr_t phys; +} __aligned(64); + +/** + * struct xilinx_vdma_tx_descriptor - Per Transaction structure + * @async_tx: Async transaction descriptor + * @segments: TX segments list + * @node: Node in the channel descriptors list + */ +struct xilinx_vdma_tx_descriptor { + struct dma_async_tx_descriptor async_tx; + struct list_head segments; + struct list_head node; +}; + +/** + * struct xilinx_vdma_chan - Driver specific VDMA channel structure + * @xdev: Driver specific device structure + * @ctrl_offset: Control registers offset + * @desc_offset: TX descriptor registers offset + * @lock: Descriptor operation lock + * @pending_list: Descriptors waiting + * @active_desc: Active descriptor + * @allocated_desc: Allocated descriptor + * @done_list: Complete descriptors + * @common: DMA common channel + * @desc_pool: Descriptors pool + * @dev: The dma device + * @irq: Channel IRQ + * @id: Channel ID + * @direction: Transfer direction + * @num_frms: Number of frames + * @has_sg: Support scatter transfers + * @genlock: Support genlock mode + * @err: Channel has errors + * @tasklet: Cleanup work after irq + * @config: Device configuration info + * @flush_on_fsync: Flush on Frame sync + */ +struct xilinx_vdma_chan { + struct xilinx_vdma_device *xdev; + u32 ctrl_offset; + u32 desc_offset; + spinlock_t lock; + struct list_head pending_list; + struct xilinx_vdma_tx_descriptor *active_desc; + struct xilinx_vdma_tx_descriptor *allocated_desc; + struct list_head done_list; + struct dma_chan common; + struct dma_pool *desc_pool; + struct device *dev; + int irq; + int id; + enum dma_transfer_direction direction; + int num_frms; + bool has_sg; + bool genlock; + bool err; + struct tasklet_struct tasklet; + struct xilinx_vdma_config config; + bool flush_on_fsync; +}; + +/** + * struct xilinx_vdma_device - VDMA device structure + * @regs: I/O mapped base address + * @dev: Device Structure + * @common: DMA device structure + * @chan: Driver specific VDMA channel + * @has_sg: Specifies whether Scatter-Gather is present or not + * @flush_on_fsync: Flush on frame sync + */ +struct xilinx_vdma_device { + void __iomem *regs; + struct device *dev; + struct dma_device common; + struct xilinx_vdma_chan *chan[XILINX_VDMA_MAX_CHANS_PER_DEVICE]; + bool has_sg; + u32 flush_on_fsync; +}; + +/* Macros */ +#define to_xilinx_chan(chan) \ + container_of(chan, struct xilinx_vdma_chan, common) +#define to_vdma_tx_descriptor(tx) \ + container_of(tx, struct xilinx_vdma_tx_descriptor, async_tx) + +/* IO accessors */ +static inline u32 vdma_read(struct xilinx_vdma_chan *chan, u32 reg) +{ + return ioread32(chan->xdev->regs + reg); +} + +static inline void vdma_write(struct xilinx_vdma_chan *chan, u32 reg, u32 value) +{ + iowrite32(value, chan->xdev->regs + reg); +} + +static inline void vdma_desc_write(struct xilinx_vdma_chan *chan, u32 reg, + u32 value) +{ + vdma_write(chan, chan->desc_offset + reg, value); +} + +static inline u32 vdma_ctrl_read(struct xilinx_vdma_chan *chan, u32 reg) +{ + return vdma_read(chan, chan->ctrl_offset + reg); +} + +static inline void vdma_ctrl_write(struct xilinx_vdma_chan *chan, u32 reg, + u32 value) +{ + vdma_write(chan, chan->ctrl_offset + reg, value); +} + +static inline void vdma_ctrl_clr(struct xilinx_vdma_chan *chan, u32 reg, + u32 clr) +{ + vdma_ctrl_write(chan, reg, vdma_ctrl_read(chan, reg) & ~clr); +} + +static inline void vdma_ctrl_set(struct xilinx_vdma_chan *chan, u32 reg, + u32 set) +{ + vdma_ctrl_write(chan, reg, vdma_ctrl_read(chan, reg) | set); +} + +/* ----------------------------------------------------------------------------- + * Descriptors and segments alloc and free + */ + +/** + * xilinx_vdma_alloc_tx_segment - Allocate transaction segment + * @chan: Driver specific VDMA channel + * + * Return: The allocated segment on success and NULL on failure. + */ +static struct xilinx_vdma_tx_segment * +xilinx_vdma_alloc_tx_segment(struct xilinx_vdma_chan *chan) +{ + struct xilinx_vdma_tx_segment *segment; + dma_addr_t phys; + + segment = dma_pool_alloc(chan->desc_pool, GFP_ATOMIC, &phys); + if (!segment) + return NULL; + + memset(segment, 0, sizeof(*segment)); + segment->phys = phys; + + return segment; +} + +/** + * xilinx_vdma_free_tx_segment - Free transaction segment + * @chan: Driver specific VDMA channel + * @segment: VDMA transaction segment + */ +static void xilinx_vdma_free_tx_segment(struct xilinx_vdma_chan *chan, + struct xilinx_vdma_tx_segment *segment) +{ + dma_pool_free(chan->desc_pool, segment, segment->phys); +} + +/** + * xilinx_vdma_tx_descriptor - Allocate transaction descriptor + * @chan: Driver specific VDMA channel + * + * Return: The allocated descriptor on success and NULL on failure. + */ +static struct xilinx_vdma_tx_descriptor * +xilinx_vdma_alloc_tx_descriptor(struct xilinx_vdma_chan *chan) +{ + struct xilinx_vdma_tx_descriptor *desc; + unsigned long flags; + + if (chan->allocated_desc) + return chan->allocated_desc; + + desc = kzalloc(sizeof(*desc), GFP_KERNEL); + if (!desc) + return NULL; + + spin_lock_irqsave(&chan->lock, flags); + chan->allocated_desc = desc; + spin_unlock_irqrestore(&chan->lock, flags); + + INIT_LIST_HEAD(&desc->segments); + + return desc; +} + +/** + * xilinx_vdma_free_tx_descriptor - Free transaction descriptor + * @chan: Driver specific VDMA channel + * @desc: VDMA transaction descriptor + */ +static void +xilinx_vdma_free_tx_descriptor(struct xilinx_vdma_chan *chan, + struct xilinx_vdma_tx_descriptor *desc) +{ + struct xilinx_vdma_tx_segment *segment, *next; + + if (!desc) + return; + + list_for_each_entry_safe(segment, next, &desc->segments, node) { + list_del(&segment->node); + xilinx_vdma_free_tx_segment(chan, segment); + } + + kfree(desc); +} + +/* Required functions */ + +/** + * xilinx_vdma_free_desc_list - Free descriptors list + * @chan: Driver specific VDMA channel + * @list: List to parse and delete the descriptor + */ +static void xilinx_vdma_free_desc_list(struct xilinx_vdma_chan *chan, + struct list_head *list) +{ + struct xilinx_vdma_tx_descriptor *desc, *next; + + list_for_each_entry_safe(desc, next, list, node) { + list_del(&desc->node); + xilinx_vdma_free_tx_descriptor(chan, desc); + } +} + +/** + * xilinx_vdma_free_descriptors - Free channel descriptors + * @chan: Driver specific VDMA channel + */ +static void xilinx_vdma_free_descriptors(struct xilinx_vdma_chan *chan) +{ + unsigned long flags; + + spin_lock_irqsave(&chan->lock, flags); + + xilinx_vdma_free_desc_list(chan, &chan->pending_list); + xilinx_vdma_free_desc_list(chan, &chan->done_list); + + xilinx_vdma_free_tx_descriptor(chan, chan->active_desc); + chan->active_desc = NULL; + + spin_unlock_irqrestore(&chan->lock, flags); +} + +/** + * xilinx_vdma_free_chan_resources - Free channel resources + * @dchan: DMA channel + */ +static void xilinx_vdma_free_chan_resources(struct dma_chan *dchan) +{ + struct xilinx_vdma_chan *chan = to_xilinx_chan(dchan); + + dev_dbg(chan->dev, "Free all channel resources.\n"); + + xilinx_vdma_free_descriptors(chan); + dma_pool_destroy(chan->desc_pool); + chan->desc_pool = NULL; +} + +/** + * xilinx_vdma_chan_desc_cleanup - Clean channel descriptors + * @chan: Driver specific VDMA channel + */ +static void xilinx_vdma_chan_desc_cleanup(struct xilinx_vdma_chan *chan) +{ + struct xilinx_vdma_tx_descriptor *desc, *next; + unsigned long flags; + + spin_lock_irqsave(&chan->lock, flags); + + list_for_each_entry_safe(desc, next, &chan->done_list, node) { + dma_async_tx_callback callback; + void *callback_param; + + /* Remove from the list of running transactions */ + list_del(&desc->node); + + /* Run the link descriptor callback function */ + callback = desc->async_tx.callback; + callback_param = desc->async_tx.callback_param; + if (callback) { + spin_unlock_irqrestore(&chan->lock, flags); + callback(callback_param); + spin_lock_irqsave(&chan->lock, flags); + } + + /* Run any dependencies, then free the descriptor */ + dma_run_dependencies(&desc->async_tx); + xilinx_vdma_free_tx_descriptor(chan, desc); + } + + spin_unlock_irqrestore(&chan->lock, flags); +} + +/** + * xilinx_vdma_do_tasklet - Schedule completion tasklet + * @data: Pointer to the Xilinx VDMA channel structure + */ +static void xilinx_vdma_do_tasklet(unsigned long data) +{ + struct xilinx_vdma_chan *chan = (struct xilinx_vdma_chan *)data; + + xilinx_vdma_chan_desc_cleanup(chan); +} + +/** + * xilinx_vdma_alloc_chan_resources - Allocate channel resources + * @dchan: DMA channel + * + * Return: '0' on success and failure value on error + */ +static int xilinx_vdma_alloc_chan_resources(struct dma_chan *dchan) +{ + struct xilinx_vdma_chan *chan = to_xilinx_chan(dchan); + + /* Has this channel already been allocated? */ + if (chan->desc_pool) + return 0; + + /* + * We need the descriptor to be aligned to 64bytes + * for meeting Xilinx VDMA specification requirement. + */ + chan->desc_pool = dma_pool_create("xilinx_vdma_desc_pool", + chan->dev, + sizeof(struct xilinx_vdma_tx_segment), + __alignof__(struct xilinx_vdma_tx_segment), 0); + if (!chan->desc_pool) { + dev_err(chan->dev, + "unable to allocate channel %d descriptor pool\n", + chan->id); + return -ENOMEM; + } + + dma_cookie_init(dchan); + return 0; +} + +/** + * xilinx_vdma_tx_status - Get VDMA transaction status + * @dchan: DMA channel + * @cookie: Transaction identifier + * @txstate: Transaction state + * + * Return: DMA transaction status + */ +static enum dma_status xilinx_vdma_tx_status(struct dma_chan *dchan, + dma_cookie_t cookie, + struct dma_tx_state *txstate) +{ + return dma_cookie_status(dchan, cookie, txstate); +} + +/** + * xilinx_vdma_is_running - Check if VDMA channel is running + * @chan: Driver specific VDMA channel + * + * Return: '1' if running, '0' if not. + */ +static bool xilinx_vdma_is_running(struct xilinx_vdma_chan *chan) +{ + return !(vdma_ctrl_read(chan, XILINX_VDMA_REG_DMASR) & + XILINX_VDMA_DMASR_HALTED) && + (vdma_ctrl_read(chan, XILINX_VDMA_REG_DMACR) & + XILINX_VDMA_DMACR_RUNSTOP); +} + +/** + * xilinx_vdma_is_idle - Check if VDMA channel is idle + * @chan: Driver specific VDMA channel + * + * Return: '1' if idle, '0' if not. + */ +static bool xilinx_vdma_is_idle(struct xilinx_vdma_chan *chan) +{ + return vdma_ctrl_read(chan, XILINX_VDMA_REG_DMASR) & + XILINX_VDMA_DMASR_IDLE; +} + +/** + * xilinx_vdma_halt - Halt VDMA channel + * @chan: Driver specific VDMA channel + */ +static void xilinx_vdma_halt(struct xilinx_vdma_chan *chan) +{ + int loop = XILINX_VDMA_LOOP_COUNT; + + vdma_ctrl_clr(chan, XILINX_VDMA_REG_DMACR, XILINX_VDMA_DMACR_RUNSTOP); + + /* Wait for the hardware to halt */ + do { + if (vdma_ctrl_read(chan, XILINX_VDMA_REG_DMASR) & + XILINX_VDMA_DMASR_HALTED) + break; + } while (loop--); + + if (!loop) { + dev_err(chan->dev, "Cannot stop channel %p: %x\n", + chan, vdma_ctrl_read(chan, XILINX_VDMA_REG_DMASR)); + chan->err = true; + } + + return; +} + +/** + * xilinx_vdma_start - Start VDMA channel + * @chan: Driver specific VDMA channel + */ +static void xilinx_vdma_start(struct xilinx_vdma_chan *chan) +{ + int loop = XILINX_VDMA_LOOP_COUNT; + + vdma_ctrl_set(chan, XILINX_VDMA_REG_DMACR, XILINX_VDMA_DMACR_RUNSTOP); + + /* Wait for the hardware to start */ + do { + if (!(vdma_ctrl_read(chan, XILINX_VDMA_REG_DMASR) & + XILINX_VDMA_DMASR_HALTED)) + break; + } while (loop--); + + if (!loop) { + dev_err(chan->dev, "Cannot start channel %p: %x\n", + chan, vdma_ctrl_read(chan, XILINX_VDMA_REG_DMASR)); + + chan->err = true; + } + + return; +} + +/** + * xilinx_vdma_start_transfer - Starts VDMA transfer + * @chan: Driver specific channel struct pointer + */ +static void xilinx_vdma_start_transfer(struct xilinx_vdma_chan *chan) +{ + struct xilinx_vdma_config *config = &chan->config; + struct xilinx_vdma_tx_descriptor *desc; + unsigned long flags; + u32 reg; + struct xilinx_vdma_tx_segment *head, *tail = NULL; + + if (chan->err) + return; + + spin_lock_irqsave(&chan->lock, flags); + + /* There's already an active descriptor, bail out. */ + if (chan->active_desc) + goto out_unlock; + + if (list_empty(&chan->pending_list)) + goto out_unlock; + + desc = list_first_entry(&chan->pending_list, + struct xilinx_vdma_tx_descriptor, node); + + /* If it is SG mode and hardware is busy, cannot submit */ + if (chan->has_sg && xilinx_vdma_is_running(chan) && + !xilinx_vdma_is_idle(chan)) { + dev_dbg(chan->dev, "DMA controller still busy\n"); + goto out_unlock; + } + + /* + * If hardware is idle, then all descriptors on the running lists are + * done, start new transfers + */ + if (chan->has_sg) { + head = list_first_entry(&desc->segments, + struct xilinx_vdma_tx_segment, node); + tail = list_entry(desc->segments.prev, + struct xilinx_vdma_tx_segment, node); + + vdma_ctrl_write(chan, XILINX_VDMA_REG_CURDESC, head->phys); + } + + /* Configure the hardware using info in the config structure */ + reg = vdma_ctrl_read(chan, XILINX_VDMA_REG_DMACR); + + if (config->frm_cnt_en) + reg |= XILINX_VDMA_DMACR_FRAMECNT_EN; + else + reg &= ~XILINX_VDMA_DMACR_FRAMECNT_EN; + + /* + * With SG, start with circular mode, so that BDs can be fetched. + * In direct register mode, if not parking, enable circular mode + */ + if (chan->has_sg || !config->park) + reg |= XILINX_VDMA_DMACR_CIRC_EN; + + if (config->park) + reg &= ~XILINX_VDMA_DMACR_CIRC_EN; + + vdma_ctrl_write(chan, XILINX_VDMA_REG_DMACR, reg); + + if (config->park && (config->park_frm >= 0) && + (config->park_frm < chan->num_frms)) { + if (chan->direction == DMA_MEM_TO_DEV) + vdma_write(chan, XILINX_VDMA_REG_PARK_PTR, + config->park_frm << + XILINX_VDMA_PARK_PTR_RD_REF_SHIFT); + else + vdma_write(chan, XILINX_VDMA_REG_PARK_PTR, + config->park_frm << + XILINX_VDMA_PARK_PTR_WR_REF_SHIFT); + } + + /* Start the hardware */ + xilinx_vdma_start(chan); + + if (chan->err) + goto out_unlock; + + /* Start the transfer */ + if (chan->has_sg) { + vdma_ctrl_write(chan, XILINX_VDMA_REG_TAILDESC, tail->phys); + } else { + struct xilinx_vdma_tx_segment *segment, *last = NULL; + int i = 0; + + list_for_each_entry(segment, &desc->segments, node) { + vdma_desc_write(chan, + XILINX_VDMA_REG_START_ADDRESS(i++), + segment->hw.buf_addr); + last = segment; + } + + if (!last) + goto out_unlock; + + /* HW expects these parameters to be same for one transaction */ + vdma_desc_write(chan, XILINX_VDMA_REG_HSIZE, last->hw.hsize); + vdma_desc_write(chan, XILINX_VDMA_REG_FRMDLY_STRIDE, + last->hw.stride); + vdma_desc_write(chan, XILINX_VDMA_REG_VSIZE, last->hw.vsize); + } + + list_del(&desc->node); + chan->active_desc = desc; + +out_unlock: + spin_unlock_irqrestore(&chan->lock, flags); +} + +/** + * xilinx_vdma_issue_pending - Issue pending transactions + * @dchan: DMA channel + */ +static void xilinx_vdma_issue_pending(struct dma_chan *dchan) +{ + struct xilinx_vdma_chan *chan = to_xilinx_chan(dchan); + + xilinx_vdma_start_transfer(chan); +} + +/** + * xilinx_vdma_complete_descriptor - Mark the active descriptor as complete + * @chan : xilinx DMA channel + * + * CONTEXT: hardirq + */ +static void xilinx_vdma_complete_descriptor(struct xilinx_vdma_chan *chan) +{ + struct xilinx_vdma_tx_descriptor *desc; + unsigned long flags; + + spin_lock_irqsave(&chan->lock, flags); + + desc = chan->active_desc; + if (!desc) { + dev_dbg(chan->dev, "no running descriptors\n"); + goto out_unlock; + } + + dma_cookie_complete(&desc->async_tx); + list_add_tail(&desc->node, &chan->done_list); + + chan->active_desc = NULL; + +out_unlock: + spin_unlock_irqrestore(&chan->lock, flags); +} + +/** + * xilinx_vdma_reset - Reset VDMA channel + * @chan: Driver specific VDMA channel + * + * Return: '0' on success and failure value on error + */ +static int xilinx_vdma_reset(struct xilinx_vdma_chan *chan) +{ + int loop = XILINX_VDMA_LOOP_COUNT; + u32 tmp; + + vdma_ctrl_set(chan, XILINX_VDMA_REG_DMACR, XILINX_VDMA_DMACR_RESET); + + tmp = vdma_ctrl_read(chan, XILINX_VDMA_REG_DMACR) & + XILINX_VDMA_DMACR_RESET; + + /* Wait for the hardware to finish reset */ + do { + tmp = vdma_ctrl_read(chan, XILINX_VDMA_REG_DMACR) & + XILINX_VDMA_DMACR_RESET; + } while (loop-- && tmp); + + if (!loop) { + dev_err(chan->dev, "reset timeout, cr %x, sr %x\n", + vdma_ctrl_read(chan, XILINX_VDMA_REG_DMACR), + vdma_ctrl_read(chan, XILINX_VDMA_REG_DMASR)); + return -ETIMEDOUT; + } + + chan->err = false; + + return 0; +} + +/** + * xilinx_vdma_chan_reset - Reset VDMA channel and enable interrupts + * @chan: Driver specific VDMA channel + * + * Return: '0' on success and failure value on error + */ +static int xilinx_vdma_chan_reset(struct xilinx_vdma_chan *chan) +{ + int err; + + /* Reset VDMA */ + err = xilinx_vdma_reset(chan); + if (err) + return err; + + /* Enable interrupts */ + vdma_ctrl_set(chan, XILINX_VDMA_REG_DMACR, + XILINX_VDMA_DMAXR_ALL_IRQ_MASK); + + return 0; +} + +/** + * xilinx_vdma_irq_handler - VDMA Interrupt handler + * @irq: IRQ number + * @data: Pointer to the Xilinx VDMA channel structure + * + * Return: IRQ_HANDLED/IRQ_NONE + */ +static irqreturn_t xilinx_vdma_irq_handler(int irq, void *data) +{ + struct xilinx_vdma_chan *chan = data; + u32 status; + + /* Read the status and ack the interrupts. */ + status = vdma_ctrl_read(chan, XILINX_VDMA_REG_DMASR); + if (!(status & XILINX_VDMA_DMAXR_ALL_IRQ_MASK)) + return IRQ_NONE; + + vdma_ctrl_write(chan, XILINX_VDMA_REG_DMASR, + status & XILINX_VDMA_DMAXR_ALL_IRQ_MASK); + + if (status & XILINX_VDMA_DMASR_ERR_IRQ) { + /* + * An error occurred. If C_FLUSH_ON_FSYNC is enabled and the + * error is recoverable, ignore it. Otherwise flag the error. + * + * Only recoverable errors can be cleared in the DMASR register, + * make sure not to write to other error bits to 1. + */ + u32 errors = status & XILINX_VDMA_DMASR_ALL_ERR_MASK; + vdma_ctrl_write(chan, XILINX_VDMA_REG_DMASR, + errors & XILINX_VDMA_DMASR_ERR_RECOVER_MASK); + + if (!chan->flush_on_fsync || + (errors & ~XILINX_VDMA_DMASR_ERR_RECOVER_MASK)) { + dev_err(chan->dev, + "Channel %p has errors %x, cdr %x tdr %x\n", + chan, errors, + vdma_ctrl_read(chan, XILINX_VDMA_REG_CURDESC), + vdma_ctrl_read(chan, XILINX_VDMA_REG_TAILDESC)); + chan->err = true; + } + } + + if (status & XILINX_VDMA_DMASR_DLY_CNT_IRQ) { + /* + * Device takes too long to do the transfer when user requires + * responsiveness. + */ + dev_dbg(chan->dev, "Inter-packet latency too long\n"); + } + + if (status & XILINX_VDMA_DMASR_FRM_CNT_IRQ) { + xilinx_vdma_complete_descriptor(chan); + xilinx_vdma_start_transfer(chan); + } + + tasklet_schedule(&chan->tasklet); + return IRQ_HANDLED; +} + +/** + * xilinx_vdma_tx_submit - Submit DMA transaction + * @tx: Async transaction descriptor + * + * Return: cookie value on success and failure value on error + */ +static dma_cookie_t xilinx_vdma_tx_submit(struct dma_async_tx_descriptor *tx) +{ + struct xilinx_vdma_tx_descriptor *desc = to_vdma_tx_descriptor(tx); + struct xilinx_vdma_chan *chan = to_xilinx_chan(tx->chan); + dma_cookie_t cookie; + unsigned long flags; + int err; + + if (chan->err) { + /* + * If reset fails, need to hard reset the system. + * Channel is no longer functional + */ + err = xilinx_vdma_chan_reset(chan); + if (err < 0) + return err; + } + + spin_lock_irqsave(&chan->lock, flags); + + cookie = dma_cookie_assign(tx); + + /* Append the transaction to the pending transactions queue. */ + list_add_tail(&desc->node, &chan->pending_list); + + /* Free the allocated desc */ + chan->allocated_desc = NULL; + + spin_unlock_irqrestore(&chan->lock, flags); + + return cookie; +} + +/** + * xilinx_vdma_dma_prep_interleaved - prepare a descriptor for a + * DMA_SLAVE transaction + * @dchan: DMA channel + * @xt: Interleaved template pointer + * @flags: transfer ack flags + * + * Return: Async transaction descriptor on success and NULL on failure + */ +static struct dma_async_tx_descriptor * +xilinx_vdma_dma_prep_interleaved(struct dma_chan *dchan, + struct dma_interleaved_template *xt, + unsigned long flags) +{ + struct xilinx_vdma_chan *chan = to_xilinx_chan(dchan); + struct xilinx_vdma_tx_descriptor *desc; + struct xilinx_vdma_tx_segment *segment, *prev = NULL; + struct xilinx_vdma_desc_hw *hw; + + if (!is_slave_direction(xt->dir)) + return NULL; + + if (!xt->numf || !xt->sgl[0].size) + return NULL; + + /* Allocate a transaction descriptor. */ + desc = xilinx_vdma_alloc_tx_descriptor(chan); + if (!desc) + return NULL; + + dma_async_tx_descriptor_init(&desc->async_tx, &chan->common); + desc->async_tx.tx_submit = xilinx_vdma_tx_submit; + async_tx_ack(&desc->async_tx); + + /* Allocate the link descriptor from DMA pool */ + segment = xilinx_vdma_alloc_tx_segment(chan); + if (!segment) + goto error; + + /* Fill in the hardware descriptor */ + hw = &segment->hw; + hw->vsize = xt->numf; + hw->hsize = xt->sgl[0].size; + hw->stride = xt->sgl[0].icg << + XILINX_VDMA_FRMDLY_STRIDE_STRIDE_SHIFT; + hw->stride |= chan->config.frm_dly << + XILINX_VDMA_FRMDLY_STRIDE_FRMDLY_SHIFT; + + if (xt->dir != DMA_MEM_TO_DEV) + hw->buf_addr = xt->dst_start; + else + hw->buf_addr = xt->src_start; + + /* Link the previous next descriptor to current */ + prev = list_last_entry(&desc->segments, + struct xilinx_vdma_tx_segment, node); + prev->hw.next_desc = segment->phys; + + /* Insert the segment into the descriptor segments list. */ + list_add_tail(&segment->node, &desc->segments); + + prev = segment; + + /* Link the last hardware descriptor with the first. */ + segment = list_first_entry(&desc->segments, + struct xilinx_vdma_tx_segment, node); + prev->hw.next_desc = segment->phys; + + return &desc->async_tx; + +error: + xilinx_vdma_free_tx_descriptor(chan, desc); + return NULL; +} + +/** + * xilinx_vdma_terminate_all - Halt the channel and free descriptors + * @chan: Driver specific VDMA Channel pointer + */ +static void xilinx_vdma_terminate_all(struct xilinx_vdma_chan *chan) +{ + /* Halt the DMA engine */ + xilinx_vdma_halt(chan); + + /* Remove and free all of the descriptors in the lists */ + xilinx_vdma_free_descriptors(chan); +} + +/** + * xilinx_vdma_channel_set_config - Configure VDMA channel + * Run-time configuration for Axi VDMA, supports: + * . halt the channel + * . configure interrupt coalescing and inter-packet delay threshold + * . start/stop parking + * . enable genlock + * + * @dchan: DMA channel + * @cfg: VDMA device configuration pointer + * + * Return: '0' on success and failure value on error + */ +int xilinx_vdma_channel_set_config(struct dma_chan *dchan, + struct xilinx_vdma_config *cfg) +{ + struct xilinx_vdma_chan *chan = to_xilinx_chan(dchan); + u32 dmacr; + + if (cfg->reset) + return xilinx_vdma_chan_reset(chan); + + dmacr = vdma_ctrl_read(chan, XILINX_VDMA_REG_DMACR); + + chan->config.frm_dly = cfg->frm_dly; + chan->config.park = cfg->park; + + /* genlock settings */ + chan->config.gen_lock = cfg->gen_lock; + chan->config.master = cfg->master; + + if (cfg->gen_lock && chan->genlock) { + dmacr |= XILINX_VDMA_DMACR_GENLOCK_EN; + dmacr |= cfg->master << XILINX_VDMA_DMACR_MASTER_SHIFT; + } + + chan->config.frm_cnt_en = cfg->frm_cnt_en; + if (cfg->park) + chan->config.park_frm = cfg->park_frm; + else + chan->config.park_frm = -1; + + chan->config.coalesc = cfg->coalesc; + chan->config.delay = cfg->delay; + + if (cfg->coalesc <= XILINX_VDMA_DMACR_FRAME_COUNT_MAX) { + dmacr |= cfg->coalesc << XILINX_VDMA_DMACR_FRAME_COUNT_SHIFT; + chan->config.coalesc = cfg->coalesc; + } + + if (cfg->delay <= XILINX_VDMA_DMACR_DELAY_MAX) { + dmacr |= cfg->delay << XILINX_VDMA_DMACR_DELAY_SHIFT; + chan->config.delay = cfg->delay; + } + + /* FSync Source selection */ + dmacr &= ~XILINX_VDMA_DMACR_FSYNCSRC_MASK; + dmacr |= cfg->ext_fsync << XILINX_VDMA_DMACR_FSYNCSRC_SHIFT; + + vdma_ctrl_write(chan, XILINX_VDMA_REG_DMACR, dmacr); + + return 0; +} +EXPORT_SYMBOL(xilinx_vdma_channel_set_config); + +/** + * xilinx_vdma_device_control - Configure DMA channel of the device + * @dchan: DMA Channel pointer + * @cmd: DMA control command + * @arg: Channel configuration + * + * Return: '0' on success and failure value on error + */ +static int xilinx_vdma_device_control(struct dma_chan *dchan, + enum dma_ctrl_cmd cmd, unsigned long arg) +{ + struct xilinx_vdma_chan *chan = to_xilinx_chan(dchan); + + if (cmd != DMA_TERMINATE_ALL) + return -ENXIO; + + xilinx_vdma_terminate_all(chan); + + return 0; +} + +/* ----------------------------------------------------------------------------- + * Probe and remove + */ + +/** + * xilinx_vdma_chan_remove - Per Channel remove function + * @chan: Driver specific VDMA channel + */ +static void xilinx_vdma_chan_remove(struct xilinx_vdma_chan *chan) +{ + /* Disable all interrupts */ + vdma_ctrl_clr(chan, XILINX_VDMA_REG_DMACR, + XILINX_VDMA_DMAXR_ALL_IRQ_MASK); + + if (chan->irq > 0) + free_irq(chan->irq, chan); + + tasklet_kill(&chan->tasklet); + + list_del(&chan->common.device_node); +} + +/** + * xilinx_vdma_chan_probe - Per Channel Probing + * It get channel features from the device tree entry and + * initialize special channel handling routines + * + * @xdev: Driver specific device structure + * @node: Device node + * + * Return: '0' on success and failure value on error + */ +static int xilinx_vdma_chan_probe(struct xilinx_vdma_device *xdev, + struct device_node *node) +{ + struct xilinx_vdma_chan *chan; + bool has_dre = false; + u32 value, width; + int err; + + /* Allocate and initialize the channel structure */ + chan = devm_kzalloc(xdev->dev, sizeof(*chan), GFP_KERNEL); + if (!chan) + return -ENOMEM; + + chan->dev = xdev->dev; + chan->xdev = xdev; + chan->has_sg = xdev->has_sg; + + spin_lock_init(&chan->lock); + INIT_LIST_HEAD(&chan->pending_list); + INIT_LIST_HEAD(&chan->done_list); + + /* Retrieve the channel properties from the device tree */ + has_dre = of_property_read_bool(node, "xlnx,include-dre"); + + chan->genlock = of_property_read_bool(node, "xlnx,genlock-mode"); + + err = of_property_read_u32(node, "xlnx,datawidth", &value); + if (err) { + dev_err(xdev->dev, "missing xlnx,datawidth property\n"); + return err; + } + width = value >> 3; /* Convert bits to bytes */ + + /* If data width is greater than 8 bytes, DRE is not in hw */ + if (width > 8) + has_dre = false; + + if (!has_dre) + xdev->common.copy_align = fls(width - 1); + + if (of_device_is_compatible(node, "xlnx,axi-vdma-mm2s-channel")) { + chan->direction = DMA_MEM_TO_DEV; + chan->id = 0; + + chan->ctrl_offset = XILINX_VDMA_MM2S_CTRL_OFFSET; + chan->desc_offset = XILINX_VDMA_MM2S_DESC_OFFSET; + + if (xdev->flush_on_fsync == XILINX_VDMA_FLUSH_BOTH || + xdev->flush_on_fsync == XILINX_VDMA_FLUSH_MM2S) + chan->flush_on_fsync = true; + } else if (of_device_is_compatible(node, + "xlnx,axi-vdma-s2mm-channel")) { + chan->direction = DMA_DEV_TO_MEM; + chan->id = 1; + + chan->ctrl_offset = XILINX_VDMA_S2MM_CTRL_OFFSET; + chan->desc_offset = XILINX_VDMA_S2MM_DESC_OFFSET; + + if (xdev->flush_on_fsync == XILINX_VDMA_FLUSH_BOTH || + xdev->flush_on_fsync == XILINX_VDMA_FLUSH_S2MM) + chan->flush_on_fsync = true; + } else { + dev_err(xdev->dev, "Invalid channel compatible node\n"); + return -EINVAL; + } + + /* Request the interrupt */ + chan->irq = irq_of_parse_and_map(node, 0); + err = request_irq(chan->irq, xilinx_vdma_irq_handler, IRQF_SHARED, + "xilinx-vdma-controller", chan); + if (err) { + dev_err(xdev->dev, "unable to request IRQ %d\n", chan->irq); + return err; + } + + /* Initialize the tasklet */ + tasklet_init(&chan->tasklet, xilinx_vdma_do_tasklet, + (unsigned long)chan); + + /* + * Initialize the DMA channel and add it to the DMA engine channels + * list. + */ + chan->common.device = &xdev->common; + + list_add_tail(&chan->common.device_node, &xdev->common.channels); + xdev->chan[chan->id] = chan; + + /* Reset the channel */ + err = xilinx_vdma_chan_reset(chan); + if (err < 0) { + dev_err(xdev->dev, "Reset channel failed\n"); + return err; + } + + return 0; +} + +/** + * of_dma_xilinx_xlate - Translation function + * @dma_spec: Pointer to DMA specifier as found in the device tree + * @ofdma: Pointer to DMA controller data + * + * Return: DMA channel pointer on success and NULL on error + */ +static struct dma_chan *of_dma_xilinx_xlate(struct of_phandle_args *dma_spec, + struct of_dma *ofdma) +{ + struct xilinx_vdma_device *xdev = ofdma->of_dma_data; + int chan_id = dma_spec->args[0]; + + if (chan_id >= XILINX_VDMA_MAX_CHANS_PER_DEVICE) + return NULL; + + return dma_get_slave_channel(&xdev->chan[chan_id]->common); +} + +/** + * xilinx_vdma_probe - Driver probe function + * @pdev: Pointer to the platform_device structure + * + * Return: '0' on success and failure value on error + */ +static int xilinx_vdma_probe(struct platform_device *pdev) +{ + struct device_node *node = pdev->dev.of_node; + struct xilinx_vdma_device *xdev; + struct device_node *child; + struct resource *io; + u32 num_frames; + int i, err; + + /* Allocate and initialize the DMA engine structure */ + xdev = devm_kzalloc(&pdev->dev, sizeof(*xdev), GFP_KERNEL); + if (!xdev) + return -ENOMEM; + + xdev->dev = &pdev->dev; + + /* Request and map I/O memory */ + io = platform_get_resource(pdev, IORESOURCE_MEM, 0); + xdev->regs = devm_ioremap_resource(&pdev->dev, io); + if (IS_ERR(xdev->regs)) + return PTR_ERR(xdev->regs); + + /* Retrieve the DMA engine properties from the device tree */ + xdev->has_sg = of_property_read_bool(node, "xlnx,include-sg"); + + err = of_property_read_u32(node, "xlnx,num-fstores", &num_frames); + if (err < 0) { + dev_err(xdev->dev, "missing xlnx,num-fstores property\n"); + return err; + } + + err = of_property_read_u32(node, "xlnx,flush-fsync", + &xdev->flush_on_fsync); + if (err < 0) + dev_warn(xdev->dev, "missing xlnx,flush-fsync property\n"); + + /* Initialize the DMA engine */ + xdev->common.dev = &pdev->dev; + + INIT_LIST_HEAD(&xdev->common.channels); + dma_cap_set(DMA_SLAVE, xdev->common.cap_mask); + dma_cap_set(DMA_PRIVATE, xdev->common.cap_mask); + + xdev->common.device_alloc_chan_resources = + xilinx_vdma_alloc_chan_resources; + xdev->common.device_free_chan_resources = + xilinx_vdma_free_chan_resources; + xdev->common.device_prep_interleaved_dma = + xilinx_vdma_dma_prep_interleaved; + xdev->common.device_control = xilinx_vdma_device_control; + xdev->common.device_tx_status = xilinx_vdma_tx_status; + xdev->common.device_issue_pending = xilinx_vdma_issue_pending; + + platform_set_drvdata(pdev, xdev); + + /* Initialize the channels */ + for_each_child_of_node(node, child) { + err = xilinx_vdma_chan_probe(xdev, child); + if (err < 0) + goto error; + } + + for (i = 0; i < XILINX_VDMA_MAX_CHANS_PER_DEVICE; i++) + if (xdev->chan[i]) + xdev->chan[i]->num_frms = num_frames; + + /* Register the DMA engine with the core */ + dma_async_device_register(&xdev->common); + + err = of_dma_controller_register(node, of_dma_xilinx_xlate, + xdev); + if (err < 0) { + dev_err(&pdev->dev, "Unable to register DMA to DT\n"); + dma_async_device_unregister(&xdev->common); + goto error; + } + + dev_info(&pdev->dev, "Xilinx AXI VDMA Engine Driver Probed!!\n"); + + return 0; + +error: + for (i = 0; i < XILINX_VDMA_MAX_CHANS_PER_DEVICE; i++) + if (xdev->chan[i]) + xilinx_vdma_chan_remove(xdev->chan[i]); + + return err; +} + +/** + * xilinx_vdma_remove - Driver remove function + * @pdev: Pointer to the platform_device structure + * + * Return: Always '0' + */ +static int xilinx_vdma_remove(struct platform_device *pdev) +{ + struct xilinx_vdma_device *xdev = platform_get_drvdata(pdev); + int i; + + of_dma_controller_free(pdev->dev.of_node); + + dma_async_device_unregister(&xdev->common); + + for (i = 0; i < XILINX_VDMA_MAX_CHANS_PER_DEVICE; i++) + if (xdev->chan[i]) + xilinx_vdma_chan_remove(xdev->chan[i]); + + return 0; +} + +static const struct of_device_id xilinx_vdma_of_ids[] = { + { .compatible = "xlnx,axi-vdma-1.00.a",}, + {} +}; + +static struct platform_driver xilinx_vdma_driver = { + .driver = { + .name = "xilinx-vdma", + .owner = THIS_MODULE, + .of_match_table = xilinx_vdma_of_ids, + }, + .probe = xilinx_vdma_probe, + .remove = xilinx_vdma_remove, +}; + +module_platform_driver(xilinx_vdma_driver); + +MODULE_AUTHOR("Xilinx, Inc."); +MODULE_DESCRIPTION("Xilinx VDMA driver"); +MODULE_LICENSE("GPL v2"); diff --git a/drivers/infiniband/Makefile b/drivers/infiniband/Makefile index bf508b5550c4..dc21836b5a8d 100644 --- a/drivers/infiniband/Makefile +++ b/drivers/infiniband/Makefile @@ -1,18 +1,3 @@ obj-$(CONFIG_INFINIBAND) += core/ -obj-$(CONFIG_INFINIBAND_MTHCA) += hw/mthca/ -obj-$(CONFIG_INFINIBAND_IPATH) += hw/ipath/ -obj-$(CONFIG_INFINIBAND_QIB) += hw/qib/ -obj-$(CONFIG_INFINIBAND_EHCA) += hw/ehca/ -obj-$(CONFIG_INFINIBAND_AMSO1100) += hw/amso1100/ -obj-$(CONFIG_INFINIBAND_CXGB3) += hw/cxgb3/ -obj-$(CONFIG_INFINIBAND_CXGB4) += hw/cxgb4/ -obj-$(CONFIG_MLX4_INFINIBAND) += hw/mlx4/ -obj-$(CONFIG_MLX5_INFINIBAND) += hw/mlx5/ -obj-$(CONFIG_INFINIBAND_NES) += hw/nes/ -obj-$(CONFIG_INFINIBAND_OCRDMA) += hw/ocrdma/ -obj-$(CONFIG_INFINIBAND_USNIC) += hw/usnic/ -obj-$(CONFIG_INFINIBAND_IPOIB) += ulp/ipoib/ -obj-$(CONFIG_INFINIBAND_SRP) += ulp/srp/ -obj-$(CONFIG_INFINIBAND_SRPT) += ulp/srpt/ -obj-$(CONFIG_INFINIBAND_ISER) += ulp/iser/ -obj-$(CONFIG_INFINIBAND_ISERT) += ulp/isert/ +obj-$(CONFIG_INFINIBAND) += hw/ +obj-$(CONFIG_INFINIBAND) += ulp/ diff --git a/drivers/infiniband/core/Makefile b/drivers/infiniband/core/Makefile index 3ab3865544bb..ffd0af6734af 100644 --- a/drivers/infiniband/core/Makefile +++ b/drivers/infiniband/core/Makefile @@ -18,7 +18,7 @@ ib_sa-y := sa_query.o multicast.o ib_cm-y := cm.o -iw_cm-y := iwcm.o +iw_cm-y := iwcm.o iwpm_util.o iwpm_msg.o rdma_cm-y := cma.o diff --git a/drivers/infiniband/core/cma.c b/drivers/infiniband/core/cma.c index 42c3058e6e9c..d570030d899c 100644 --- a/drivers/infiniband/core/cma.c +++ b/drivers/infiniband/core/cma.c @@ -3607,7 +3607,8 @@ static int cma_get_id_stats(struct sk_buff *skb, struct netlink_callback *cb) id_stats = ibnl_put_msg(skb, &nlh, cb->nlh->nlmsg_seq, sizeof *id_stats, RDMA_NL_RDMA_CM, - RDMA_NL_RDMA_CM_ID_STATS); + RDMA_NL_RDMA_CM_ID_STATS, + NLM_F_MULTI); if (!id_stats) goto out; diff --git a/drivers/infiniband/core/iwpm_msg.c b/drivers/infiniband/core/iwpm_msg.c new file mode 100644 index 000000000000..b85ddbc979e0 --- /dev/null +++ b/drivers/infiniband/core/iwpm_msg.c @@ -0,0 +1,685 @@ +/* + * Copyright (c) 2014 Intel Corporation. All rights reserved. + * Copyright (c) 2014 Chelsio, Inc. All rights reserved. + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the + * OpenIB.org BSD license below: + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ + +#include "iwpm_util.h" + +static const char iwpm_ulib_name[] = "iWarpPortMapperUser"; +static int iwpm_ulib_version = 3; +static int iwpm_user_pid = IWPM_PID_UNDEFINED; +static atomic_t echo_nlmsg_seq; + +int iwpm_valid_pid(void) +{ + return iwpm_user_pid > 0; +} +EXPORT_SYMBOL(iwpm_valid_pid); + +/* + * iwpm_register_pid - Send a netlink query to user space + * for the iwarp port mapper pid + * + * nlmsg attributes: + * [IWPM_NLA_REG_PID_SEQ] + * [IWPM_NLA_REG_IF_NAME] + * [IWPM_NLA_REG_IBDEV_NAME] + * [IWPM_NLA_REG_ULIB_NAME] + */ +int iwpm_register_pid(struct iwpm_dev_data *pm_msg, u8 nl_client) +{ + struct sk_buff *skb = NULL; + struct iwpm_nlmsg_request *nlmsg_request = NULL; + struct nlmsghdr *nlh; + u32 msg_seq; + const char *err_str = ""; + int ret = -EINVAL; + + if (!iwpm_valid_client(nl_client)) { + err_str = "Invalid port mapper client"; + goto pid_query_error; + } + if (iwpm_registered_client(nl_client)) + return 0; + skb = iwpm_create_nlmsg(RDMA_NL_IWPM_REG_PID, &nlh, nl_client); + if (!skb) { + err_str = "Unable to create a nlmsg"; + goto pid_query_error; + } + nlh->nlmsg_seq = iwpm_get_nlmsg_seq(); + nlmsg_request = iwpm_get_nlmsg_request(nlh->nlmsg_seq, nl_client, GFP_KERNEL); + if (!nlmsg_request) { + err_str = "Unable to allocate netlink request"; + goto pid_query_error; + } + msg_seq = atomic_read(&echo_nlmsg_seq); + + /* fill in the pid request message */ + err_str = "Unable to put attribute of the nlmsg"; + ret = ibnl_put_attr(skb, nlh, sizeof(u32), &msg_seq, IWPM_NLA_REG_PID_SEQ); + if (ret) + goto pid_query_error; + ret = ibnl_put_attr(skb, nlh, IWPM_IFNAME_SIZE, + pm_msg->if_name, IWPM_NLA_REG_IF_NAME); + if (ret) + goto pid_query_error; + ret = ibnl_put_attr(skb, nlh, IWPM_DEVNAME_SIZE, + pm_msg->dev_name, IWPM_NLA_REG_IBDEV_NAME); + if (ret) + goto pid_query_error; + ret = ibnl_put_attr(skb, nlh, IWPM_ULIBNAME_SIZE, + (char *)iwpm_ulib_name, IWPM_NLA_REG_ULIB_NAME); + if (ret) + goto pid_query_error; + + pr_debug("%s: Multicasting a nlmsg (dev = %s ifname = %s iwpm = %s)\n", + __func__, pm_msg->dev_name, pm_msg->if_name, iwpm_ulib_name); + + ret = ibnl_multicast(skb, nlh, RDMA_NL_GROUP_IWPM, GFP_KERNEL); + if (ret) { + skb = NULL; /* skb is freed in the netlink send-op handling */ + iwpm_set_registered(nl_client, 1); + iwpm_user_pid = IWPM_PID_UNAVAILABLE; + err_str = "Unable to send a nlmsg"; + goto pid_query_error; + } + nlmsg_request->req_buffer = pm_msg; + ret = iwpm_wait_complete_req(nlmsg_request); + return ret; +pid_query_error: + pr_info("%s: %s (client = %d)\n", __func__, err_str, nl_client); + if (skb) + dev_kfree_skb(skb); + if (nlmsg_request) + iwpm_free_nlmsg_request(&nlmsg_request->kref); + return ret; +} +EXPORT_SYMBOL(iwpm_register_pid); + +/* + * iwpm_add_mapping - Send a netlink add mapping message + * to the port mapper + * nlmsg attributes: + * [IWPM_NLA_MANAGE_MAPPING_SEQ] + * [IWPM_NLA_MANAGE_ADDR] + */ +int iwpm_add_mapping(struct iwpm_sa_data *pm_msg, u8 nl_client) +{ + struct sk_buff *skb = NULL; + struct iwpm_nlmsg_request *nlmsg_request = NULL; + struct nlmsghdr *nlh; + u32 msg_seq; + const char *err_str = ""; + int ret = -EINVAL; + + if (!iwpm_valid_client(nl_client)) { + err_str = "Invalid port mapper client"; + goto add_mapping_error; + } + if (!iwpm_registered_client(nl_client)) { + err_str = "Unregistered port mapper client"; + goto add_mapping_error; + } + if (!iwpm_valid_pid()) + return 0; + skb = iwpm_create_nlmsg(RDMA_NL_IWPM_ADD_MAPPING, &nlh, nl_client); + if (!skb) { + err_str = "Unable to create a nlmsg"; + goto add_mapping_error; + } + nlh->nlmsg_seq = iwpm_get_nlmsg_seq(); + nlmsg_request = iwpm_get_nlmsg_request(nlh->nlmsg_seq, nl_client, GFP_KERNEL); + if (!nlmsg_request) { + err_str = "Unable to allocate netlink request"; + goto add_mapping_error; + } + msg_seq = atomic_read(&echo_nlmsg_seq); + /* fill in the add mapping message */ + err_str = "Unable to put attribute of the nlmsg"; + ret = ibnl_put_attr(skb, nlh, sizeof(u32), &msg_seq, + IWPM_NLA_MANAGE_MAPPING_SEQ); + if (ret) + goto add_mapping_error; + ret = ibnl_put_attr(skb, nlh, sizeof(struct sockaddr_storage), + &pm_msg->loc_addr, IWPM_NLA_MANAGE_ADDR); + if (ret) + goto add_mapping_error; + nlmsg_request->req_buffer = pm_msg; + + ret = ibnl_unicast(skb, nlh, iwpm_user_pid); + if (ret) { + skb = NULL; /* skb is freed in the netlink send-op handling */ + iwpm_user_pid = IWPM_PID_UNDEFINED; + err_str = "Unable to send a nlmsg"; + goto add_mapping_error; + } + ret = iwpm_wait_complete_req(nlmsg_request); + return ret; +add_mapping_error: + pr_info("%s: %s (client = %d)\n", __func__, err_str, nl_client); + if (skb) + dev_kfree_skb(skb); + if (nlmsg_request) + iwpm_free_nlmsg_request(&nlmsg_request->kref); + return ret; +} +EXPORT_SYMBOL(iwpm_add_mapping); + +/* + * iwpm_add_and_query_mapping - Send a netlink add and query + * mapping message to the port mapper + * nlmsg attributes: + * [IWPM_NLA_QUERY_MAPPING_SEQ] + * [IWPM_NLA_QUERY_LOCAL_ADDR] + * [IWPM_NLA_QUERY_REMOTE_ADDR] + */ +int iwpm_add_and_query_mapping(struct iwpm_sa_data *pm_msg, u8 nl_client) +{ + struct sk_buff *skb = NULL; + struct iwpm_nlmsg_request *nlmsg_request = NULL; + struct nlmsghdr *nlh; + u32 msg_seq; + const char *err_str = ""; + int ret = -EINVAL; + + if (!iwpm_valid_client(nl_client)) { + err_str = "Invalid port mapper client"; + goto query_mapping_error; + } + if (!iwpm_registered_client(nl_client)) { + err_str = "Unregistered port mapper client"; + goto query_mapping_error; + } + if (!iwpm_valid_pid()) + return 0; + ret = -ENOMEM; + skb = iwpm_create_nlmsg(RDMA_NL_IWPM_QUERY_MAPPING, &nlh, nl_client); + if (!skb) { + err_str = "Unable to create a nlmsg"; + goto query_mapping_error; + } + nlh->nlmsg_seq = iwpm_get_nlmsg_seq(); + nlmsg_request = iwpm_get_nlmsg_request(nlh->nlmsg_seq, + nl_client, GFP_KERNEL); + if (!nlmsg_request) { + err_str = "Unable to allocate netlink request"; + goto query_mapping_error; + } + msg_seq = atomic_read(&echo_nlmsg_seq); + + /* fill in the query message */ + err_str = "Unable to put attribute of the nlmsg"; + ret = ibnl_put_attr(skb, nlh, sizeof(u32), &msg_seq, + IWPM_NLA_QUERY_MAPPING_SEQ); + if (ret) + goto query_mapping_error; + ret = ibnl_put_attr(skb, nlh, sizeof(struct sockaddr_storage), + &pm_msg->loc_addr, IWPM_NLA_QUERY_LOCAL_ADDR); + if (ret) + goto query_mapping_error; + ret = ibnl_put_attr(skb, nlh, sizeof(struct sockaddr_storage), + &pm_msg->rem_addr, IWPM_NLA_QUERY_REMOTE_ADDR); + if (ret) + goto query_mapping_error; + nlmsg_request->req_buffer = pm_msg; + + ret = ibnl_unicast(skb, nlh, iwpm_user_pid); + if (ret) { + skb = NULL; /* skb is freed in the netlink send-op handling */ + err_str = "Unable to send a nlmsg"; + goto query_mapping_error; + } + ret = iwpm_wait_complete_req(nlmsg_request); + return ret; +query_mapping_error: + pr_info("%s: %s (client = %d)\n", __func__, err_str, nl_client); + if (skb) + dev_kfree_skb(skb); + if (nlmsg_request) + iwpm_free_nlmsg_request(&nlmsg_request->kref); + return ret; +} +EXPORT_SYMBOL(iwpm_add_and_query_mapping); + +/* + * iwpm_remove_mapping - Send a netlink remove mapping message + * to the port mapper + * nlmsg attributes: + * [IWPM_NLA_MANAGE_MAPPING_SEQ] + * [IWPM_NLA_MANAGE_ADDR] + */ +int iwpm_remove_mapping(struct sockaddr_storage *local_addr, u8 nl_client) +{ + struct sk_buff *skb = NULL; + struct nlmsghdr *nlh; + u32 msg_seq; + const char *err_str = ""; + int ret = -EINVAL; + + if (!iwpm_valid_client(nl_client)) { + err_str = "Invalid port mapper client"; + goto remove_mapping_error; + } + if (!iwpm_registered_client(nl_client)) { + err_str = "Unregistered port mapper client"; + goto remove_mapping_error; + } + if (!iwpm_valid_pid()) + return 0; + skb = iwpm_create_nlmsg(RDMA_NL_IWPM_REMOVE_MAPPING, &nlh, nl_client); + if (!skb) { + ret = -ENOMEM; + err_str = "Unable to create a nlmsg"; + goto remove_mapping_error; + } + msg_seq = atomic_read(&echo_nlmsg_seq); + nlh->nlmsg_seq = iwpm_get_nlmsg_seq(); + err_str = "Unable to put attribute of the nlmsg"; + ret = ibnl_put_attr(skb, nlh, sizeof(u32), &msg_seq, + IWPM_NLA_MANAGE_MAPPING_SEQ); + if (ret) + goto remove_mapping_error; + ret = ibnl_put_attr(skb, nlh, sizeof(struct sockaddr_storage), + local_addr, IWPM_NLA_MANAGE_ADDR); + if (ret) + goto remove_mapping_error; + + ret = ibnl_unicast(skb, nlh, iwpm_user_pid); + if (ret) { + skb = NULL; /* skb is freed in the netlink send-op handling */ + iwpm_user_pid = IWPM_PID_UNDEFINED; + err_str = "Unable to send a nlmsg"; + goto remove_mapping_error; + } + iwpm_print_sockaddr(local_addr, + "remove_mapping: Local sockaddr:"); + return 0; +remove_mapping_error: + pr_info("%s: %s (client = %d)\n", __func__, err_str, nl_client); + if (skb) + dev_kfree_skb_any(skb); + return ret; +} +EXPORT_SYMBOL(iwpm_remove_mapping); + +/* netlink attribute policy for the received response to register pid request */ +static const struct nla_policy resp_reg_policy[IWPM_NLA_RREG_PID_MAX] = { + [IWPM_NLA_RREG_PID_SEQ] = { .type = NLA_U32 }, + [IWPM_NLA_RREG_IBDEV_NAME] = { .type = NLA_STRING, + .len = IWPM_DEVNAME_SIZE - 1 }, + [IWPM_NLA_RREG_ULIB_NAME] = { .type = NLA_STRING, + .len = IWPM_ULIBNAME_SIZE - 1 }, + [IWPM_NLA_RREG_ULIB_VER] = { .type = NLA_U16 }, + [IWPM_NLA_RREG_PID_ERR] = { .type = NLA_U16 } +}; + +/* + * iwpm_register_pid_cb - Process a port mapper response to + * iwpm_register_pid() + */ +int iwpm_register_pid_cb(struct sk_buff *skb, struct netlink_callback *cb) +{ + struct iwpm_nlmsg_request *nlmsg_request = NULL; + struct nlattr *nltb[IWPM_NLA_RREG_PID_MAX]; + struct iwpm_dev_data *pm_msg; + char *dev_name, *iwpm_name; + u32 msg_seq; + u8 nl_client; + u16 iwpm_version; + const char *msg_type = "Register Pid response"; + + if (iwpm_parse_nlmsg(cb, IWPM_NLA_RREG_PID_MAX, + resp_reg_policy, nltb, msg_type)) + return -EINVAL; + + msg_seq = nla_get_u32(nltb[IWPM_NLA_RREG_PID_SEQ]); + nlmsg_request = iwpm_find_nlmsg_request(msg_seq); + if (!nlmsg_request) { + pr_info("%s: Could not find a matching request (seq = %u)\n", + __func__, msg_seq); + return -EINVAL; + } + pm_msg = nlmsg_request->req_buffer; + nl_client = nlmsg_request->nl_client; + dev_name = (char *)nla_data(nltb[IWPM_NLA_RREG_IBDEV_NAME]); + iwpm_name = (char *)nla_data(nltb[IWPM_NLA_RREG_ULIB_NAME]); + iwpm_version = nla_get_u16(nltb[IWPM_NLA_RREG_ULIB_VER]); + + /* check device name, ulib name and version */ + if (strcmp(pm_msg->dev_name, dev_name) || + strcmp(iwpm_ulib_name, iwpm_name) || + iwpm_version != iwpm_ulib_version) { + + pr_info("%s: Incorrect info (dev = %s name = %s version = %d)\n", + __func__, dev_name, iwpm_name, iwpm_version); + nlmsg_request->err_code = IWPM_USER_LIB_INFO_ERR; + goto register_pid_response_exit; + } + iwpm_user_pid = cb->nlh->nlmsg_pid; + atomic_set(&echo_nlmsg_seq, cb->nlh->nlmsg_seq); + pr_debug("%s: iWarp Port Mapper (pid = %d) is available!\n", + __func__, iwpm_user_pid); + if (iwpm_valid_client(nl_client)) + iwpm_set_registered(nl_client, 1); +register_pid_response_exit: + nlmsg_request->request_done = 1; + /* always for found nlmsg_request */ + kref_put(&nlmsg_request->kref, iwpm_free_nlmsg_request); + barrier(); + wake_up(&nlmsg_request->waitq); + return 0; +} +EXPORT_SYMBOL(iwpm_register_pid_cb); + +/* netlink attribute policy for the received response to add mapping request */ +static const struct nla_policy resp_add_policy[IWPM_NLA_RMANAGE_MAPPING_MAX] = { + [IWPM_NLA_MANAGE_MAPPING_SEQ] = { .type = NLA_U32 }, + [IWPM_NLA_MANAGE_ADDR] = { .len = sizeof(struct sockaddr_storage) }, + [IWPM_NLA_MANAGE_MAPPED_LOC_ADDR] = { .len = sizeof(struct sockaddr_storage) }, + [IWPM_NLA_RMANAGE_MAPPING_ERR] = { .type = NLA_U16 } +}; + +/* + * iwpm_add_mapping_cb - Process a port mapper response to + * iwpm_add_mapping() + */ +int iwpm_add_mapping_cb(struct sk_buff *skb, struct netlink_callback *cb) +{ + struct iwpm_sa_data *pm_msg; + struct iwpm_nlmsg_request *nlmsg_request = NULL; + struct nlattr *nltb[IWPM_NLA_RMANAGE_MAPPING_MAX]; + struct sockaddr_storage *local_sockaddr; + struct sockaddr_storage *mapped_sockaddr; + const char *msg_type; + u32 msg_seq; + + msg_type = "Add Mapping response"; + if (iwpm_parse_nlmsg(cb, IWPM_NLA_RMANAGE_MAPPING_MAX, + resp_add_policy, nltb, msg_type)) + return -EINVAL; + + atomic_set(&echo_nlmsg_seq, cb->nlh->nlmsg_seq); + + msg_seq = nla_get_u32(nltb[IWPM_NLA_MANAGE_MAPPING_SEQ]); + nlmsg_request = iwpm_find_nlmsg_request(msg_seq); + if (!nlmsg_request) { + pr_info("%s: Could not find a matching request (seq = %u)\n", + __func__, msg_seq); + return -EINVAL; + } + pm_msg = nlmsg_request->req_buffer; + local_sockaddr = (struct sockaddr_storage *) + nla_data(nltb[IWPM_NLA_MANAGE_ADDR]); + mapped_sockaddr = (struct sockaddr_storage *) + nla_data(nltb[IWPM_NLA_MANAGE_MAPPED_LOC_ADDR]); + + if (iwpm_compare_sockaddr(local_sockaddr, &pm_msg->loc_addr)) { + nlmsg_request->err_code = IWPM_USER_LIB_INFO_ERR; + goto add_mapping_response_exit; + } + if (mapped_sockaddr->ss_family != local_sockaddr->ss_family) { + pr_info("%s: Sockaddr family doesn't match the requested one\n", + __func__); + nlmsg_request->err_code = IWPM_USER_LIB_INFO_ERR; + goto add_mapping_response_exit; + } + memcpy(&pm_msg->mapped_loc_addr, mapped_sockaddr, + sizeof(*mapped_sockaddr)); + iwpm_print_sockaddr(&pm_msg->loc_addr, + "add_mapping: Local sockaddr:"); + iwpm_print_sockaddr(&pm_msg->mapped_loc_addr, + "add_mapping: Mapped local sockaddr:"); + +add_mapping_response_exit: + nlmsg_request->request_done = 1; + /* always for found request */ + kref_put(&nlmsg_request->kref, iwpm_free_nlmsg_request); + barrier(); + wake_up(&nlmsg_request->waitq); + return 0; +} +EXPORT_SYMBOL(iwpm_add_mapping_cb); + +/* netlink attribute policy for the response to add and query mapping request */ +static const struct nla_policy resp_query_policy[IWPM_NLA_RQUERY_MAPPING_MAX] = { + [IWPM_NLA_QUERY_MAPPING_SEQ] = { .type = NLA_U32 }, + [IWPM_NLA_QUERY_LOCAL_ADDR] = { .len = sizeof(struct sockaddr_storage) }, + [IWPM_NLA_QUERY_REMOTE_ADDR] = { .len = sizeof(struct sockaddr_storage) }, + [IWPM_NLA_RQUERY_MAPPED_LOC_ADDR] = { .len = sizeof(struct sockaddr_storage) }, + [IWPM_NLA_RQUERY_MAPPED_REM_ADDR] = { .len = sizeof(struct sockaddr_storage) }, + [IWPM_NLA_RQUERY_MAPPING_ERR] = { .type = NLA_U16 } +}; + +/* + * iwpm_add_and_query_mapping_cb - Process a port mapper response to + * iwpm_add_and_query_mapping() + */ +int iwpm_add_and_query_mapping_cb(struct sk_buff *skb, + struct netlink_callback *cb) +{ + struct iwpm_sa_data *pm_msg; + struct iwpm_nlmsg_request *nlmsg_request = NULL; + struct nlattr *nltb[IWPM_NLA_RQUERY_MAPPING_MAX]; + struct sockaddr_storage *local_sockaddr, *remote_sockaddr; + struct sockaddr_storage *mapped_loc_sockaddr, *mapped_rem_sockaddr; + const char *msg_type; + u32 msg_seq; + u16 err_code; + + msg_type = "Query Mapping response"; + if (iwpm_parse_nlmsg(cb, IWPM_NLA_RQUERY_MAPPING_MAX, + resp_query_policy, nltb, msg_type)) + return -EINVAL; + atomic_set(&echo_nlmsg_seq, cb->nlh->nlmsg_seq); + + msg_seq = nla_get_u32(nltb[IWPM_NLA_QUERY_MAPPING_SEQ]); + nlmsg_request = iwpm_find_nlmsg_request(msg_seq); + if (!nlmsg_request) { + pr_info("%s: Could not find a matching request (seq = %u)\n", + __func__, msg_seq); + return -EINVAL; + } + pm_msg = nlmsg_request->req_buffer; + local_sockaddr = (struct sockaddr_storage *) + nla_data(nltb[IWPM_NLA_QUERY_LOCAL_ADDR]); + remote_sockaddr = (struct sockaddr_storage *) + nla_data(nltb[IWPM_NLA_QUERY_REMOTE_ADDR]); + mapped_loc_sockaddr = (struct sockaddr_storage *) + nla_data(nltb[IWPM_NLA_RQUERY_MAPPED_LOC_ADDR]); + mapped_rem_sockaddr = (struct sockaddr_storage *) + nla_data(nltb[IWPM_NLA_RQUERY_MAPPED_REM_ADDR]); + + err_code = nla_get_u16(nltb[IWPM_NLA_RQUERY_MAPPING_ERR]); + if (err_code == IWPM_REMOTE_QUERY_REJECT) { + pr_info("%s: Received a Reject (pid = %u, echo seq = %u)\n", + __func__, cb->nlh->nlmsg_pid, msg_seq); + nlmsg_request->err_code = IWPM_REMOTE_QUERY_REJECT; + } + if (iwpm_compare_sockaddr(local_sockaddr, &pm_msg->loc_addr) || + iwpm_compare_sockaddr(remote_sockaddr, &pm_msg->rem_addr)) { + pr_info("%s: Incorrect local sockaddr\n", __func__); + nlmsg_request->err_code = IWPM_USER_LIB_INFO_ERR; + goto query_mapping_response_exit; + } + if (mapped_loc_sockaddr->ss_family != local_sockaddr->ss_family || + mapped_rem_sockaddr->ss_family != remote_sockaddr->ss_family) { + pr_info("%s: Sockaddr family doesn't match the requested one\n", + __func__); + nlmsg_request->err_code = IWPM_USER_LIB_INFO_ERR; + goto query_mapping_response_exit; + } + memcpy(&pm_msg->mapped_loc_addr, mapped_loc_sockaddr, + sizeof(*mapped_loc_sockaddr)); + memcpy(&pm_msg->mapped_rem_addr, mapped_rem_sockaddr, + sizeof(*mapped_rem_sockaddr)); + + iwpm_print_sockaddr(&pm_msg->loc_addr, + "query_mapping: Local sockaddr:"); + iwpm_print_sockaddr(&pm_msg->mapped_loc_addr, + "query_mapping: Mapped local sockaddr:"); + iwpm_print_sockaddr(&pm_msg->rem_addr, + "query_mapping: Remote sockaddr:"); + iwpm_print_sockaddr(&pm_msg->mapped_rem_addr, + "query_mapping: Mapped remote sockaddr:"); +query_mapping_response_exit: + nlmsg_request->request_done = 1; + /* always for found request */ + kref_put(&nlmsg_request->kref, iwpm_free_nlmsg_request); + barrier(); + wake_up(&nlmsg_request->waitq); + return 0; +} +EXPORT_SYMBOL(iwpm_add_and_query_mapping_cb); + +/* netlink attribute policy for the received request for mapping info */ +static const struct nla_policy resp_mapinfo_policy[IWPM_NLA_MAPINFO_REQ_MAX] = { + [IWPM_NLA_MAPINFO_ULIB_NAME] = { .type = NLA_STRING, + .len = IWPM_ULIBNAME_SIZE - 1 }, + [IWPM_NLA_MAPINFO_ULIB_VER] = { .type = NLA_U16 } +}; + +/* + * iwpm_mapping_info_cb - Process a port mapper request for mapping info + */ +int iwpm_mapping_info_cb(struct sk_buff *skb, struct netlink_callback *cb) +{ + struct nlattr *nltb[IWPM_NLA_MAPINFO_REQ_MAX]; + const char *msg_type = "Mapping Info response"; + int iwpm_pid; + u8 nl_client; + char *iwpm_name; + u16 iwpm_version; + int ret = -EINVAL; + + if (iwpm_parse_nlmsg(cb, IWPM_NLA_MAPINFO_REQ_MAX, + resp_mapinfo_policy, nltb, msg_type)) { + pr_info("%s: Unable to parse nlmsg\n", __func__); + return ret; + } + iwpm_name = (char *)nla_data(nltb[IWPM_NLA_MAPINFO_ULIB_NAME]); + iwpm_version = nla_get_u16(nltb[IWPM_NLA_MAPINFO_ULIB_VER]); + if (strcmp(iwpm_ulib_name, iwpm_name) || + iwpm_version != iwpm_ulib_version) { + pr_info("%s: Invalid port mapper name = %s version = %d\n", + __func__, iwpm_name, iwpm_version); + return ret; + } + nl_client = RDMA_NL_GET_CLIENT(cb->nlh->nlmsg_type); + if (!iwpm_valid_client(nl_client)) { + pr_info("%s: Invalid port mapper client = %d\n", + __func__, nl_client); + return ret; + } + iwpm_set_registered(nl_client, 0); + atomic_set(&echo_nlmsg_seq, cb->nlh->nlmsg_seq); + if (!iwpm_mapinfo_available()) + return 0; + iwpm_pid = cb->nlh->nlmsg_pid; + pr_debug("%s: iWarp Port Mapper (pid = %d) is available!\n", + __func__, iwpm_pid); + ret = iwpm_send_mapinfo(nl_client, iwpm_pid); + return ret; +} +EXPORT_SYMBOL(iwpm_mapping_info_cb); + +/* netlink attribute policy for the received mapping info ack */ +static const struct nla_policy ack_mapinfo_policy[IWPM_NLA_MAPINFO_NUM_MAX] = { + [IWPM_NLA_MAPINFO_SEQ] = { .type = NLA_U32 }, + [IWPM_NLA_MAPINFO_SEND_NUM] = { .type = NLA_U32 }, + [IWPM_NLA_MAPINFO_ACK_NUM] = { .type = NLA_U32 } +}; + +/* + * iwpm_ack_mapping_info_cb - Process a port mapper ack for + * the provided mapping info records + */ +int iwpm_ack_mapping_info_cb(struct sk_buff *skb, struct netlink_callback *cb) +{ + struct nlattr *nltb[IWPM_NLA_MAPINFO_NUM_MAX]; + u32 mapinfo_send, mapinfo_ack; + const char *msg_type = "Mapping Info Ack"; + + if (iwpm_parse_nlmsg(cb, IWPM_NLA_MAPINFO_NUM_MAX, + ack_mapinfo_policy, nltb, msg_type)) + return -EINVAL; + mapinfo_send = nla_get_u32(nltb[IWPM_NLA_MAPINFO_SEND_NUM]); + mapinfo_ack = nla_get_u32(nltb[IWPM_NLA_MAPINFO_ACK_NUM]); + if (mapinfo_ack != mapinfo_send) + pr_info("%s: Invalid mapinfo number (sent = %u ack-ed = %u)\n", + __func__, mapinfo_send, mapinfo_ack); + atomic_set(&echo_nlmsg_seq, cb->nlh->nlmsg_seq); + return 0; +} +EXPORT_SYMBOL(iwpm_ack_mapping_info_cb); + +/* netlink attribute policy for the received port mapper error message */ +static const struct nla_policy map_error_policy[IWPM_NLA_ERR_MAX] = { + [IWPM_NLA_ERR_SEQ] = { .type = NLA_U32 }, + [IWPM_NLA_ERR_CODE] = { .type = NLA_U16 }, +}; + +/* + * iwpm_mapping_error_cb - Process a port mapper error message + */ +int iwpm_mapping_error_cb(struct sk_buff *skb, struct netlink_callback *cb) +{ + struct iwpm_nlmsg_request *nlmsg_request = NULL; + int nl_client = RDMA_NL_GET_CLIENT(cb->nlh->nlmsg_type); + struct nlattr *nltb[IWPM_NLA_ERR_MAX]; + u32 msg_seq; + u16 err_code; + const char *msg_type = "Mapping Error Msg"; + + if (iwpm_parse_nlmsg(cb, IWPM_NLA_ERR_MAX, + map_error_policy, nltb, msg_type)) + return -EINVAL; + + msg_seq = nla_get_u32(nltb[IWPM_NLA_ERR_SEQ]); + err_code = nla_get_u16(nltb[IWPM_NLA_ERR_CODE]); + pr_info("%s: Received msg seq = %u err code = %u client = %d\n", + __func__, msg_seq, err_code, nl_client); + /* look for nlmsg_request */ + nlmsg_request = iwpm_find_nlmsg_request(msg_seq); + if (!nlmsg_request) { + /* not all errors have associated requests */ + pr_debug("Could not find matching req (seq = %u)\n", msg_seq); + return 0; + } + atomic_set(&echo_nlmsg_seq, cb->nlh->nlmsg_seq); + nlmsg_request->err_code = err_code; + nlmsg_request->request_done = 1; + /* always for found request */ + kref_put(&nlmsg_request->kref, iwpm_free_nlmsg_request); + barrier(); + wake_up(&nlmsg_request->waitq); + return 0; +} +EXPORT_SYMBOL(iwpm_mapping_error_cb); diff --git a/drivers/infiniband/core/iwpm_util.c b/drivers/infiniband/core/iwpm_util.c new file mode 100644 index 000000000000..69e9f84c1605 --- /dev/null +++ b/drivers/infiniband/core/iwpm_util.c @@ -0,0 +1,607 @@ +/* + * Copyright (c) 2014 Chelsio, Inc. All rights reserved. + * Copyright (c) 2014 Intel Corporation. All rights reserved. + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the + * OpenIB.org BSD license below: + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ + +#include "iwpm_util.h" + +#define IWPM_HASH_BUCKET_SIZE 512 +#define IWPM_HASH_BUCKET_MASK (IWPM_HASH_BUCKET_SIZE - 1) + +static LIST_HEAD(iwpm_nlmsg_req_list); +static DEFINE_SPINLOCK(iwpm_nlmsg_req_lock); + +static struct hlist_head *iwpm_hash_bucket; +static DEFINE_SPINLOCK(iwpm_mapinfo_lock); + +static DEFINE_MUTEX(iwpm_admin_lock); +static struct iwpm_admin_data iwpm_admin; + +int iwpm_init(u8 nl_client) +{ + if (iwpm_valid_client(nl_client)) + return -EINVAL; + mutex_lock(&iwpm_admin_lock); + if (atomic_read(&iwpm_admin.refcount) == 0) { + iwpm_hash_bucket = kzalloc(IWPM_HASH_BUCKET_SIZE * + sizeof(struct hlist_head), GFP_KERNEL); + if (!iwpm_hash_bucket) { + mutex_unlock(&iwpm_admin_lock); + pr_err("%s Unable to create mapinfo hash table\n", __func__); + return -ENOMEM; + } + } + atomic_inc(&iwpm_admin.refcount); + mutex_unlock(&iwpm_admin_lock); + iwpm_set_valid(nl_client, 1); + return 0; +} +EXPORT_SYMBOL(iwpm_init); + +static void free_hash_bucket(void); + +int iwpm_exit(u8 nl_client) +{ + + if (!iwpm_valid_client(nl_client)) + return -EINVAL; + mutex_lock(&iwpm_admin_lock); + if (atomic_read(&iwpm_admin.refcount) == 0) { + mutex_unlock(&iwpm_admin_lock); + pr_err("%s Incorrect usage - negative refcount\n", __func__); + return -EINVAL; + } + if (atomic_dec_and_test(&iwpm_admin.refcount)) { + free_hash_bucket(); + pr_debug("%s: Mapinfo hash table is destroyed\n", __func__); + } + mutex_unlock(&iwpm_admin_lock); + iwpm_set_valid(nl_client, 0); + return 0; +} +EXPORT_SYMBOL(iwpm_exit); + +static struct hlist_head *get_hash_bucket_head(struct sockaddr_storage *, + struct sockaddr_storage *); + +int iwpm_create_mapinfo(struct sockaddr_storage *local_sockaddr, + struct sockaddr_storage *mapped_sockaddr, + u8 nl_client) +{ + struct hlist_head *hash_bucket_head; + struct iwpm_mapping_info *map_info; + unsigned long flags; + + if (!iwpm_valid_client(nl_client)) + return -EINVAL; + map_info = kzalloc(sizeof(struct iwpm_mapping_info), GFP_KERNEL); + if (!map_info) { + pr_err("%s: Unable to allocate a mapping info\n", __func__); + return -ENOMEM; + } + memcpy(&map_info->local_sockaddr, local_sockaddr, + sizeof(struct sockaddr_storage)); + memcpy(&map_info->mapped_sockaddr, mapped_sockaddr, + sizeof(struct sockaddr_storage)); + map_info->nl_client = nl_client; + + spin_lock_irqsave(&iwpm_mapinfo_lock, flags); + if (iwpm_hash_bucket) { + hash_bucket_head = get_hash_bucket_head( + &map_info->local_sockaddr, + &map_info->mapped_sockaddr); + hlist_add_head(&map_info->hlist_node, hash_bucket_head); + } + spin_unlock_irqrestore(&iwpm_mapinfo_lock, flags); + return 0; +} +EXPORT_SYMBOL(iwpm_create_mapinfo); + +int iwpm_remove_mapinfo(struct sockaddr_storage *local_sockaddr, + struct sockaddr_storage *mapped_local_addr) +{ + struct hlist_node *tmp_hlist_node; + struct hlist_head *hash_bucket_head; + struct iwpm_mapping_info *map_info = NULL; + unsigned long flags; + int ret = -EINVAL; + + spin_lock_irqsave(&iwpm_mapinfo_lock, flags); + if (iwpm_hash_bucket) { + hash_bucket_head = get_hash_bucket_head( + local_sockaddr, + mapped_local_addr); + hlist_for_each_entry_safe(map_info, tmp_hlist_node, + hash_bucket_head, hlist_node) { + + if (!iwpm_compare_sockaddr(&map_info->mapped_sockaddr, + mapped_local_addr)) { + + hlist_del_init(&map_info->hlist_node); + kfree(map_info); + ret = 0; + break; + } + } + } + spin_unlock_irqrestore(&iwpm_mapinfo_lock, flags); + return ret; +} +EXPORT_SYMBOL(iwpm_remove_mapinfo); + +static void free_hash_bucket(void) +{ + struct hlist_node *tmp_hlist_node; + struct iwpm_mapping_info *map_info; + unsigned long flags; + int i; + + /* remove all the mapinfo data from the list */ + spin_lock_irqsave(&iwpm_mapinfo_lock, flags); + for (i = 0; i < IWPM_HASH_BUCKET_SIZE; i++) { + hlist_for_each_entry_safe(map_info, tmp_hlist_node, + &iwpm_hash_bucket[i], hlist_node) { + + hlist_del_init(&map_info->hlist_node); + kfree(map_info); + } + } + /* free the hash list */ + kfree(iwpm_hash_bucket); + iwpm_hash_bucket = NULL; + spin_unlock_irqrestore(&iwpm_mapinfo_lock, flags); +} + +struct iwpm_nlmsg_request *iwpm_get_nlmsg_request(__u32 nlmsg_seq, + u8 nl_client, gfp_t gfp) +{ + struct iwpm_nlmsg_request *nlmsg_request = NULL; + unsigned long flags; + + nlmsg_request = kzalloc(sizeof(struct iwpm_nlmsg_request), gfp); + if (!nlmsg_request) { + pr_err("%s Unable to allocate a nlmsg_request\n", __func__); + return NULL; + } + spin_lock_irqsave(&iwpm_nlmsg_req_lock, flags); + list_add_tail(&nlmsg_request->inprocess_list, &iwpm_nlmsg_req_list); + spin_unlock_irqrestore(&iwpm_nlmsg_req_lock, flags); + + kref_init(&nlmsg_request->kref); + kref_get(&nlmsg_request->kref); + nlmsg_request->nlmsg_seq = nlmsg_seq; + nlmsg_request->nl_client = nl_client; + nlmsg_request->request_done = 0; + nlmsg_request->err_code = 0; + return nlmsg_request; +} + +void iwpm_free_nlmsg_request(struct kref *kref) +{ + struct iwpm_nlmsg_request *nlmsg_request; + unsigned long flags; + + nlmsg_request = container_of(kref, struct iwpm_nlmsg_request, kref); + + spin_lock_irqsave(&iwpm_nlmsg_req_lock, flags); + list_del_init(&nlmsg_request->inprocess_list); + spin_unlock_irqrestore(&iwpm_nlmsg_req_lock, flags); + + if (!nlmsg_request->request_done) + pr_debug("%s Freeing incomplete nlmsg request (seq = %u).\n", + __func__, nlmsg_request->nlmsg_seq); + kfree(nlmsg_request); +} + +struct iwpm_nlmsg_request *iwpm_find_nlmsg_request(__u32 echo_seq) +{ + struct iwpm_nlmsg_request *nlmsg_request; + struct iwpm_nlmsg_request *found_request = NULL; + unsigned long flags; + + spin_lock_irqsave(&iwpm_nlmsg_req_lock, flags); + list_for_each_entry(nlmsg_request, &iwpm_nlmsg_req_list, + inprocess_list) { + if (nlmsg_request->nlmsg_seq == echo_seq) { + found_request = nlmsg_request; + kref_get(&nlmsg_request->kref); + break; + } + } + spin_unlock_irqrestore(&iwpm_nlmsg_req_lock, flags); + return found_request; +} + +int iwpm_wait_complete_req(struct iwpm_nlmsg_request *nlmsg_request) +{ + int ret; + init_waitqueue_head(&nlmsg_request->waitq); + + ret = wait_event_timeout(nlmsg_request->waitq, + (nlmsg_request->request_done != 0), IWPM_NL_TIMEOUT); + if (!ret) { + ret = -EINVAL; + pr_info("%s: Timeout %d sec for netlink request (seq = %u)\n", + __func__, (IWPM_NL_TIMEOUT/HZ), nlmsg_request->nlmsg_seq); + } else { + ret = nlmsg_request->err_code; + } + kref_put(&nlmsg_request->kref, iwpm_free_nlmsg_request); + return ret; +} + +int iwpm_get_nlmsg_seq(void) +{ + return atomic_inc_return(&iwpm_admin.nlmsg_seq); +} + +int iwpm_valid_client(u8 nl_client) +{ + if (nl_client >= RDMA_NL_NUM_CLIENTS) + return 0; + return iwpm_admin.client_list[nl_client]; +} + +void iwpm_set_valid(u8 nl_client, int valid) +{ + if (nl_client >= RDMA_NL_NUM_CLIENTS) + return; + iwpm_admin.client_list[nl_client] = valid; +} + +/* valid client */ +int iwpm_registered_client(u8 nl_client) +{ + return iwpm_admin.reg_list[nl_client]; +} + +/* valid client */ +void iwpm_set_registered(u8 nl_client, int reg) +{ + iwpm_admin.reg_list[nl_client] = reg; +} + +int iwpm_compare_sockaddr(struct sockaddr_storage *a_sockaddr, + struct sockaddr_storage *b_sockaddr) +{ + if (a_sockaddr->ss_family != b_sockaddr->ss_family) + return 1; + if (a_sockaddr->ss_family == AF_INET) { + struct sockaddr_in *a4_sockaddr = + (struct sockaddr_in *)a_sockaddr; + struct sockaddr_in *b4_sockaddr = + (struct sockaddr_in *)b_sockaddr; + if (!memcmp(&a4_sockaddr->sin_addr, + &b4_sockaddr->sin_addr, sizeof(struct in_addr)) + && a4_sockaddr->sin_port == b4_sockaddr->sin_port) + return 0; + + } else if (a_sockaddr->ss_family == AF_INET6) { + struct sockaddr_in6 *a6_sockaddr = + (struct sockaddr_in6 *)a_sockaddr; + struct sockaddr_in6 *b6_sockaddr = + (struct sockaddr_in6 *)b_sockaddr; + if (!memcmp(&a6_sockaddr->sin6_addr, + &b6_sockaddr->sin6_addr, sizeof(struct in6_addr)) + && a6_sockaddr->sin6_port == b6_sockaddr->sin6_port) + return 0; + + } else { + pr_err("%s: Invalid sockaddr family\n", __func__); + } + return 1; +} + +struct sk_buff *iwpm_create_nlmsg(u32 nl_op, struct nlmsghdr **nlh, + int nl_client) +{ + struct sk_buff *skb = NULL; + + skb = dev_alloc_skb(NLMSG_GOODSIZE); + if (!skb) { + pr_err("%s Unable to allocate skb\n", __func__); + goto create_nlmsg_exit; + } + if (!(ibnl_put_msg(skb, nlh, 0, 0, nl_client, nl_op, + NLM_F_REQUEST))) { + pr_warn("%s: Unable to put the nlmsg header\n", __func__); + dev_kfree_skb(skb); + skb = NULL; + } +create_nlmsg_exit: + return skb; +} + +int iwpm_parse_nlmsg(struct netlink_callback *cb, int policy_max, + const struct nla_policy *nlmsg_policy, + struct nlattr *nltb[], const char *msg_type) +{ + int nlh_len = 0; + int ret; + const char *err_str = ""; + + ret = nlmsg_validate(cb->nlh, nlh_len, policy_max-1, nlmsg_policy); + if (ret) { + err_str = "Invalid attribute"; + goto parse_nlmsg_error; + } + ret = nlmsg_parse(cb->nlh, nlh_len, nltb, policy_max-1, nlmsg_policy); + if (ret) { + err_str = "Unable to parse the nlmsg"; + goto parse_nlmsg_error; + } + ret = iwpm_validate_nlmsg_attr(nltb, policy_max); + if (ret) { + err_str = "Invalid NULL attribute"; + goto parse_nlmsg_error; + } + return 0; +parse_nlmsg_error: + pr_warn("%s: %s (msg type %s ret = %d)\n", + __func__, err_str, msg_type, ret); + return ret; +} + +void iwpm_print_sockaddr(struct sockaddr_storage *sockaddr, char *msg) +{ + struct sockaddr_in6 *sockaddr_v6; + struct sockaddr_in *sockaddr_v4; + + switch (sockaddr->ss_family) { + case AF_INET: + sockaddr_v4 = (struct sockaddr_in *)sockaddr; + pr_debug("%s IPV4 %pI4: %u(0x%04X)\n", + msg, &sockaddr_v4->sin_addr, + ntohs(sockaddr_v4->sin_port), + ntohs(sockaddr_v4->sin_port)); + break; + case AF_INET6: + sockaddr_v6 = (struct sockaddr_in6 *)sockaddr; + pr_debug("%s IPV6 %pI6: %u(0x%04X)\n", + msg, &sockaddr_v6->sin6_addr, + ntohs(sockaddr_v6->sin6_port), + ntohs(sockaddr_v6->sin6_port)); + break; + default: + break; + } +} + +static u32 iwpm_ipv6_jhash(struct sockaddr_in6 *ipv6_sockaddr) +{ + u32 ipv6_hash = jhash(&ipv6_sockaddr->sin6_addr, sizeof(struct in6_addr), 0); + u32 hash = jhash_2words(ipv6_hash, (__force u32) ipv6_sockaddr->sin6_port, 0); + return hash; +} + +static u32 iwpm_ipv4_jhash(struct sockaddr_in *ipv4_sockaddr) +{ + u32 ipv4_hash = jhash(&ipv4_sockaddr->sin_addr, sizeof(struct in_addr), 0); + u32 hash = jhash_2words(ipv4_hash, (__force u32) ipv4_sockaddr->sin_port, 0); + return hash; +} + +static struct hlist_head *get_hash_bucket_head(struct sockaddr_storage + *local_sockaddr, + struct sockaddr_storage + *mapped_sockaddr) +{ + u32 local_hash, mapped_hash, hash; + + if (local_sockaddr->ss_family == AF_INET) { + local_hash = iwpm_ipv4_jhash((struct sockaddr_in *) local_sockaddr); + mapped_hash = iwpm_ipv4_jhash((struct sockaddr_in *) mapped_sockaddr); + + } else if (local_sockaddr->ss_family == AF_INET6) { + local_hash = iwpm_ipv6_jhash((struct sockaddr_in6 *) local_sockaddr); + mapped_hash = iwpm_ipv6_jhash((struct sockaddr_in6 *) mapped_sockaddr); + } else { + pr_err("%s: Invalid sockaddr family\n", __func__); + return NULL; + } + + if (local_hash == mapped_hash) /* if port mapper isn't available */ + hash = local_hash; + else + hash = jhash_2words(local_hash, mapped_hash, 0); + + return &iwpm_hash_bucket[hash & IWPM_HASH_BUCKET_MASK]; +} + +static int send_mapinfo_num(u32 mapping_num, u8 nl_client, int iwpm_pid) +{ + struct sk_buff *skb = NULL; + struct nlmsghdr *nlh; + u32 msg_seq; + const char *err_str = ""; + int ret = -EINVAL; + + skb = iwpm_create_nlmsg(RDMA_NL_IWPM_MAPINFO_NUM, &nlh, nl_client); + if (!skb) { + err_str = "Unable to create a nlmsg"; + goto mapinfo_num_error; + } + nlh->nlmsg_seq = iwpm_get_nlmsg_seq(); + msg_seq = 0; + err_str = "Unable to put attribute of mapinfo number nlmsg"; + ret = ibnl_put_attr(skb, nlh, sizeof(u32), &msg_seq, IWPM_NLA_MAPINFO_SEQ); + if (ret) + goto mapinfo_num_error; + ret = ibnl_put_attr(skb, nlh, sizeof(u32), + &mapping_num, IWPM_NLA_MAPINFO_SEND_NUM); + if (ret) + goto mapinfo_num_error; + ret = ibnl_unicast(skb, nlh, iwpm_pid); + if (ret) { + skb = NULL; + err_str = "Unable to send a nlmsg"; + goto mapinfo_num_error; + } + pr_debug("%s: Sent mapping number = %d\n", __func__, mapping_num); + return 0; +mapinfo_num_error: + pr_info("%s: %s\n", __func__, err_str); + if (skb) + dev_kfree_skb(skb); + return ret; +} + +static int send_nlmsg_done(struct sk_buff *skb, u8 nl_client, int iwpm_pid) +{ + struct nlmsghdr *nlh = NULL; + int ret = 0; + + if (!skb) + return ret; + if (!(ibnl_put_msg(skb, &nlh, 0, 0, nl_client, + RDMA_NL_IWPM_MAPINFO, NLM_F_MULTI))) { + pr_warn("%s Unable to put NLMSG_DONE\n", __func__); + return -ENOMEM; + } + nlh->nlmsg_type = NLMSG_DONE; + ret = ibnl_unicast(skb, (struct nlmsghdr *)skb->data, iwpm_pid); + if (ret) + pr_warn("%s Unable to send a nlmsg\n", __func__); + return ret; +} + +int iwpm_send_mapinfo(u8 nl_client, int iwpm_pid) +{ + struct iwpm_mapping_info *map_info; + struct sk_buff *skb = NULL; + struct nlmsghdr *nlh; + int skb_num = 0, mapping_num = 0; + int i = 0, nlmsg_bytes = 0; + unsigned long flags; + const char *err_str = ""; + int ret; + + skb = dev_alloc_skb(NLMSG_GOODSIZE); + if (!skb) { + ret = -ENOMEM; + err_str = "Unable to allocate skb"; + goto send_mapping_info_exit; + } + skb_num++; + spin_lock_irqsave(&iwpm_mapinfo_lock, flags); + for (i = 0; i < IWPM_HASH_BUCKET_SIZE; i++) { + hlist_for_each_entry(map_info, &iwpm_hash_bucket[i], + hlist_node) { + if (map_info->nl_client != nl_client) + continue; + nlh = NULL; + if (!(ibnl_put_msg(skb, &nlh, 0, 0, nl_client, + RDMA_NL_IWPM_MAPINFO, NLM_F_MULTI))) { + ret = -ENOMEM; + err_str = "Unable to put the nlmsg header"; + goto send_mapping_info_unlock; + } + err_str = "Unable to put attribute of the nlmsg"; + ret = ibnl_put_attr(skb, nlh, + sizeof(struct sockaddr_storage), + &map_info->local_sockaddr, + IWPM_NLA_MAPINFO_LOCAL_ADDR); + if (ret) + goto send_mapping_info_unlock; + + ret = ibnl_put_attr(skb, nlh, + sizeof(struct sockaddr_storage), + &map_info->mapped_sockaddr, + IWPM_NLA_MAPINFO_MAPPED_ADDR); + if (ret) + goto send_mapping_info_unlock; + + iwpm_print_sockaddr(&map_info->local_sockaddr, + "send_mapping_info: Local sockaddr:"); + iwpm_print_sockaddr(&map_info->mapped_sockaddr, + "send_mapping_info: Mapped local sockaddr:"); + mapping_num++; + nlmsg_bytes += nlh->nlmsg_len; + + /* check if all mappings can fit in one skb */ + if (NLMSG_GOODSIZE - nlmsg_bytes < nlh->nlmsg_len * 2) { + /* and leave room for NLMSG_DONE */ + nlmsg_bytes = 0; + skb_num++; + spin_unlock_irqrestore(&iwpm_mapinfo_lock, + flags); + /* send the skb */ + ret = send_nlmsg_done(skb, nl_client, iwpm_pid); + skb = NULL; + if (ret) { + err_str = "Unable to send map info"; + goto send_mapping_info_exit; + } + if (skb_num == IWPM_MAPINFO_SKB_COUNT) { + ret = -ENOMEM; + err_str = "Insufficient skbs for map info"; + goto send_mapping_info_exit; + } + skb = dev_alloc_skb(NLMSG_GOODSIZE); + if (!skb) { + ret = -ENOMEM; + err_str = "Unable to allocate skb"; + goto send_mapping_info_exit; + } + spin_lock_irqsave(&iwpm_mapinfo_lock, flags); + } + } + } +send_mapping_info_unlock: + spin_unlock_irqrestore(&iwpm_mapinfo_lock, flags); +send_mapping_info_exit: + if (ret) { + pr_warn("%s: %s (ret = %d)\n", __func__, err_str, ret); + if (skb) + dev_kfree_skb(skb); + return ret; + } + send_nlmsg_done(skb, nl_client, iwpm_pid); + return send_mapinfo_num(mapping_num, nl_client, iwpm_pid); +} + +int iwpm_mapinfo_available(void) +{ + unsigned long flags; + int full_bucket = 0, i = 0; + + spin_lock_irqsave(&iwpm_mapinfo_lock, flags); + if (iwpm_hash_bucket) { + for (i = 0; i < IWPM_HASH_BUCKET_SIZE; i++) { + if (!hlist_empty(&iwpm_hash_bucket[i])) { + full_bucket = 1; + break; + } + } + } + spin_unlock_irqrestore(&iwpm_mapinfo_lock, flags); + return full_bucket; +} diff --git a/drivers/infiniband/core/iwpm_util.h b/drivers/infiniband/core/iwpm_util.h new file mode 100644 index 000000000000..9777c869a140 --- /dev/null +++ b/drivers/infiniband/core/iwpm_util.h @@ -0,0 +1,238 @@ +/* + * Copyright (c) 2014 Intel Corporation. All rights reserved. + * Copyright (c) 2014 Chelsio, Inc. All rights reserved. + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the + * OpenIB.org BSD license below: + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ +#ifndef _IWPM_UTIL_H +#define _IWPM_UTIL_H + +#include <linux/module.h> +#include <linux/io.h> +#include <linux/in.h> +#include <linux/in6.h> +#include <linux/spinlock.h> +#include <linux/kernel.h> +#include <linux/netdevice.h> +#include <linux/delay.h> +#include <linux/workqueue.h> +#include <linux/mutex.h> +#include <linux/jhash.h> +#include <linux/kref.h> +#include <net/netlink.h> +#include <linux/errno.h> +#include <rdma/iw_portmap.h> +#include <rdma/rdma_netlink.h> + + +#define IWPM_NL_RETRANS 3 +#define IWPM_NL_TIMEOUT (10*HZ) +#define IWPM_MAPINFO_SKB_COUNT 20 + +#define IWPM_PID_UNDEFINED -1 +#define IWPM_PID_UNAVAILABLE -2 + +struct iwpm_nlmsg_request { + struct list_head inprocess_list; + __u32 nlmsg_seq; + void *req_buffer; + u8 nl_client; + u8 request_done; + u16 err_code; + wait_queue_head_t waitq; + struct kref kref; +}; + +struct iwpm_mapping_info { + struct hlist_node hlist_node; + struct sockaddr_storage local_sockaddr; + struct sockaddr_storage mapped_sockaddr; + u8 nl_client; +}; + +struct iwpm_admin_data { + atomic_t refcount; + atomic_t nlmsg_seq; + int client_list[RDMA_NL_NUM_CLIENTS]; + int reg_list[RDMA_NL_NUM_CLIENTS]; +}; + +/** + * iwpm_get_nlmsg_request - Allocate and initialize netlink message request + * @nlmsg_seq: Sequence number of the netlink message + * @nl_client: The index of the netlink client + * @gfp: Indicates how the memory for the request should be allocated + * + * Returns the newly allocated netlink request object if successful, + * otherwise returns NULL + */ +struct iwpm_nlmsg_request *iwpm_get_nlmsg_request(__u32 nlmsg_seq, + u8 nl_client, gfp_t gfp); + +/** + * iwpm_free_nlmsg_request - Deallocate netlink message request + * @kref: Holds reference of netlink message request + */ +void iwpm_free_nlmsg_request(struct kref *kref); + +/** + * iwpm_find_nlmsg_request - Find netlink message request in the request list + * @echo_seq: Sequence number of the netlink request to find + * + * Returns the found netlink message request, + * if not found, returns NULL + */ +struct iwpm_nlmsg_request *iwpm_find_nlmsg_request(__u32 echo_seq); + +/** + * iwpm_wait_complete_req - Block while servicing the netlink request + * @nlmsg_request: Netlink message request to service + * + * Wakes up, after the request is completed or expired + * Returns 0 if the request is complete without error + */ +int iwpm_wait_complete_req(struct iwpm_nlmsg_request *nlmsg_request); + +/** + * iwpm_get_nlmsg_seq - Get the sequence number for a netlink + * message to send to the port mapper + * + * Returns the sequence number for the netlink message. + */ +int iwpm_get_nlmsg_seq(void); + +/** + * iwpm_valid_client - Check if the port mapper client is valid + * @nl_client: The index of the netlink client + * + * Valid clients need to call iwpm_init() before using + * the port mapper + */ +int iwpm_valid_client(u8 nl_client); + +/** + * iwpm_set_valid - Set the port mapper client to valid or not + * @nl_client: The index of the netlink client + * @valid: 1 if valid or 0 if invalid + */ +void iwpm_set_valid(u8 nl_client, int valid); + +/** + * iwpm_registered_client - Check if the port mapper client is registered + * @nl_client: The index of the netlink client + * + * Call iwpm_register_pid() to register a client + */ +int iwpm_registered_client(u8 nl_client); + +/** + * iwpm_set_registered - Set the port mapper client to registered or not + * @nl_client: The index of the netlink client + * @reg: 1 if registered or 0 if not + */ +void iwpm_set_registered(u8 nl_client, int reg); + +/** + * iwpm_send_mapinfo - Send local and mapped IPv4/IPv6 address info of + * a client to the user space port mapper + * @nl_client: The index of the netlink client + * @iwpm_pid: The pid of the user space port mapper + * + * If successful, returns the number of sent mapping info records + */ +int iwpm_send_mapinfo(u8 nl_client, int iwpm_pid); + +/** + * iwpm_mapinfo_available - Check if any mapping info records is available + * in the hash table + * + * Returns 1 if mapping information is available, otherwise returns 0 + */ +int iwpm_mapinfo_available(void); + +/** + * iwpm_compare_sockaddr - Compare two sockaddr storage structs + * + * Returns 0 if they are holding the same ip/tcp address info, + * otherwise returns 1 + */ +int iwpm_compare_sockaddr(struct sockaddr_storage *a_sockaddr, + struct sockaddr_storage *b_sockaddr); + +/** + * iwpm_validate_nlmsg_attr - Check for NULL netlink attributes + * @nltb: Holds address of each netlink message attributes + * @nla_count: Number of netlink message attributes + * + * Returns error if any of the nla_count attributes is NULL + */ +static inline int iwpm_validate_nlmsg_attr(struct nlattr *nltb[], + int nla_count) +{ + int i; + for (i = 1; i < nla_count; i++) { + if (!nltb[i]) + return -EINVAL; + } + return 0; +} + +/** + * iwpm_create_nlmsg - Allocate skb and form a netlink message + * @nl_op: Netlink message opcode + * @nlh: Holds address of the netlink message header in skb + * @nl_client: The index of the netlink client + * + * Returns the newly allcated skb, or NULL if the tailroom of the skb + * is insufficient to store the message header and payload + */ +struct sk_buff *iwpm_create_nlmsg(u32 nl_op, struct nlmsghdr **nlh, + int nl_client); + +/** + * iwpm_parse_nlmsg - Validate and parse the received netlink message + * @cb: Netlink callback structure + * @policy_max: Maximum attribute type to be expected + * @nlmsg_policy: Validation policy + * @nltb: Array to store policy_max parsed elements + * @msg_type: Type of netlink message + * + * Returns 0 on success or a negative error code + */ +int iwpm_parse_nlmsg(struct netlink_callback *cb, int policy_max, + const struct nla_policy *nlmsg_policy, + struct nlattr *nltb[], const char *msg_type); + +/** + * iwpm_print_sockaddr - Print IPv4/IPv6 address and TCP port + * @sockaddr: Socket address to print + * @msg: Message to print + */ +void iwpm_print_sockaddr(struct sockaddr_storage *sockaddr, char *msg); +#endif diff --git a/drivers/infiniband/core/netlink.c b/drivers/infiniband/core/netlink.c index a1e9cba84944..23dd5a5c7597 100644 --- a/drivers/infiniband/core/netlink.c +++ b/drivers/infiniband/core/netlink.c @@ -103,13 +103,13 @@ int ibnl_remove_client(int index) EXPORT_SYMBOL(ibnl_remove_client); void *ibnl_put_msg(struct sk_buff *skb, struct nlmsghdr **nlh, int seq, - int len, int client, int op) + int len, int client, int op, int flags) { unsigned char *prev_tail; prev_tail = skb_tail_pointer(skb); *nlh = nlmsg_put(skb, 0, seq, RDMA_NL_GET_TYPE(client, op), - len, NLM_F_MULTI); + len, flags); if (!*nlh) goto out_nlmsg_trim; (*nlh)->nlmsg_len = skb_tail_pointer(skb) - prev_tail; @@ -172,6 +172,20 @@ static void ibnl_rcv(struct sk_buff *skb) mutex_unlock(&ibnl_mutex); } +int ibnl_unicast(struct sk_buff *skb, struct nlmsghdr *nlh, + __u32 pid) +{ + return nlmsg_unicast(nls, skb, pid); +} +EXPORT_SYMBOL(ibnl_unicast); + +int ibnl_multicast(struct sk_buff *skb, struct nlmsghdr *nlh, + unsigned int group, gfp_t flags) +{ + return nlmsg_multicast(nls, skb, 0, group, flags); +} +EXPORT_SYMBOL(ibnl_multicast); + int __init ibnl_init(void) { struct netlink_kernel_cfg cfg = { diff --git a/drivers/infiniband/core/sa_query.c b/drivers/infiniband/core/sa_query.c index f820958e4047..233eaf541f55 100644 --- a/drivers/infiniband/core/sa_query.c +++ b/drivers/infiniband/core/sa_query.c @@ -618,7 +618,7 @@ static void init_mad(struct ib_sa_mad *mad, struct ib_mad_agent *agent) static int send_mad(struct ib_sa_query *query, int timeout_ms, gfp_t gfp_mask) { - bool preload = gfp_mask & __GFP_WAIT; + bool preload = !!(gfp_mask & __GFP_WAIT); unsigned long flags; int ret, id; diff --git a/drivers/infiniband/core/sysfs.c b/drivers/infiniband/core/sysfs.c index 7d3292c7b4b4..cbd0383f622e 100644 --- a/drivers/infiniband/core/sysfs.c +++ b/drivers/infiniband/core/sysfs.c @@ -429,15 +429,19 @@ static void ib_port_release(struct kobject *kobj) struct attribute *a; int i; - for (i = 0; (a = p->gid_group.attrs[i]); ++i) - kfree(a); + if (p->gid_group.attrs) { + for (i = 0; (a = p->gid_group.attrs[i]); ++i) + kfree(a); - kfree(p->gid_group.attrs); + kfree(p->gid_group.attrs); + } - for (i = 0; (a = p->pkey_group.attrs[i]); ++i) - kfree(a); + if (p->pkey_group.attrs) { + for (i = 0; (a = p->pkey_group.attrs[i]); ++i) + kfree(a); - kfree(p->pkey_group.attrs); + kfree(p->pkey_group.attrs); + } kfree(p); } @@ -534,10 +538,12 @@ static int add_port(struct ib_device *device, int port_num, p->port_num = port_num; ret = kobject_init_and_add(&p->kobj, &port_type, - kobject_get(device->ports_parent), + device->ports_parent, "%d", port_num); - if (ret) - goto err_put; + if (ret) { + kfree(p); + return ret; + } ret = sysfs_create_group(&p->kobj, &pma_group); if (ret) @@ -585,6 +591,7 @@ err_free_pkey: kfree(p->pkey_group.attrs[i]); kfree(p->pkey_group.attrs); + p->pkey_group.attrs = NULL; err_remove_gid: sysfs_remove_group(&p->kobj, &p->gid_group); @@ -594,13 +601,13 @@ err_free_gid: kfree(p->gid_group.attrs[i]); kfree(p->gid_group.attrs); + p->gid_group.attrs = NULL; err_remove_pma: sysfs_remove_group(&p->kobj, &pma_group); err_put: - kobject_put(device->ports_parent); - kfree(p); + kobject_put(&p->kobj); return ret; } @@ -809,6 +816,22 @@ static struct attribute_group iw_stats_group = { .attrs = iw_proto_stats_attrs, }; +static void free_port_list_attributes(struct ib_device *device) +{ + struct kobject *p, *t; + + list_for_each_entry_safe(p, t, &device->port_list, entry) { + struct ib_port *port = container_of(p, struct ib_port, kobj); + list_del(&p->entry); + sysfs_remove_group(p, &pma_group); + sysfs_remove_group(p, &port->pkey_group); + sysfs_remove_group(p, &port->gid_group); + kobject_put(p); + } + + kobject_put(device->ports_parent); +} + int ib_device_register_sysfs(struct ib_device *device, int (*port_callback)(struct ib_device *, u8, struct kobject *)) @@ -835,7 +858,7 @@ int ib_device_register_sysfs(struct ib_device *device, } device->ports_parent = kobject_create_and_add("ports", - kobject_get(&class_dev->kobj)); + &class_dev->kobj); if (!device->ports_parent) { ret = -ENOMEM; goto err_put; @@ -862,21 +885,7 @@ int ib_device_register_sysfs(struct ib_device *device, return 0; err_put: - { - struct kobject *p, *t; - struct ib_port *port; - - list_for_each_entry_safe(p, t, &device->port_list, entry) { - list_del(&p->entry); - port = container_of(p, struct ib_port, kobj); - sysfs_remove_group(p, &pma_group); - sysfs_remove_group(p, &port->pkey_group); - sysfs_remove_group(p, &port->gid_group); - kobject_put(p); - } - } - - kobject_put(&class_dev->kobj); + free_port_list_attributes(device); err_unregister: device_unregister(class_dev); @@ -887,22 +896,18 @@ err: void ib_device_unregister_sysfs(struct ib_device *device) { - struct kobject *p, *t; - struct ib_port *port; - /* Hold kobject until ib_dealloc_device() */ - kobject_get(&device->dev.kobj); + struct kobject *kobj_dev = kobject_get(&device->dev.kobj); + int i; - list_for_each_entry_safe(p, t, &device->port_list, entry) { - list_del(&p->entry); - port = container_of(p, struct ib_port, kobj); - sysfs_remove_group(p, &pma_group); - sysfs_remove_group(p, &port->pkey_group); - sysfs_remove_group(p, &port->gid_group); - kobject_put(p); - } + if (device->node_type == RDMA_NODE_RNIC && device->get_protocol_stats) + sysfs_remove_group(kobj_dev, &iw_stats_group); + + free_port_list_attributes(device); + + for (i = 0; i < ARRAY_SIZE(ib_class_attributes); ++i) + device_remove_file(&device->dev, ib_class_attributes[i]); - kobject_put(device->ports_parent); device_unregister(&device->dev); } diff --git a/drivers/infiniband/core/user_mad.c b/drivers/infiniband/core/user_mad.c index f0d588f8859e..1acb99100556 100644 --- a/drivers/infiniband/core/user_mad.c +++ b/drivers/infiniband/core/user_mad.c @@ -98,7 +98,7 @@ struct ib_umad_port { struct ib_umad_device { int start_port, end_port; - struct kref ref; + struct kobject kobj; struct ib_umad_port port[0]; }; @@ -134,14 +134,18 @@ static DECLARE_BITMAP(dev_map, IB_UMAD_MAX_PORTS); static void ib_umad_add_one(struct ib_device *device); static void ib_umad_remove_one(struct ib_device *device); -static void ib_umad_release_dev(struct kref *ref) +static void ib_umad_release_dev(struct kobject *kobj) { struct ib_umad_device *dev = - container_of(ref, struct ib_umad_device, ref); + container_of(kobj, struct ib_umad_device, kobj); kfree(dev); } +static struct kobj_type ib_umad_dev_ktype = { + .release = ib_umad_release_dev, +}; + static int hdr_size(struct ib_umad_file *file) { return file->use_pkey_index ? sizeof (struct ib_user_mad_hdr) : @@ -780,27 +784,19 @@ static int ib_umad_open(struct inode *inode, struct file *filp) { struct ib_umad_port *port; struct ib_umad_file *file; - int ret; + int ret = -ENXIO; port = container_of(inode->i_cdev, struct ib_umad_port, cdev); - if (port) - kref_get(&port->umad_dev->ref); - else - return -ENXIO; mutex_lock(&port->file_mutex); - if (!port->ib_dev) { - ret = -ENXIO; + if (!port->ib_dev) goto out; - } + ret = -ENOMEM; file = kzalloc(sizeof *file, GFP_KERNEL); - if (!file) { - kref_put(&port->umad_dev->ref, ib_umad_release_dev); - ret = -ENOMEM; + if (!file) goto out; - } mutex_init(&file->mutex); spin_lock_init(&file->send_lock); @@ -814,6 +810,13 @@ static int ib_umad_open(struct inode *inode, struct file *filp) list_add_tail(&file->port_list, &port->file_list); ret = nonseekable_open(inode, filp); + if (ret) { + list_del(&file->port_list); + kfree(file); + goto out; + } + + kobject_get(&port->umad_dev->kobj); out: mutex_unlock(&port->file_mutex); @@ -852,7 +855,7 @@ static int ib_umad_close(struct inode *inode, struct file *filp) mutex_unlock(&file->port->file_mutex); kfree(file); - kref_put(&dev->ref, ib_umad_release_dev); + kobject_put(&dev->kobj); return 0; } @@ -880,10 +883,6 @@ static int ib_umad_sm_open(struct inode *inode, struct file *filp) int ret; port = container_of(inode->i_cdev, struct ib_umad_port, sm_cdev); - if (port) - kref_get(&port->umad_dev->ref); - else - return -ENXIO; if (filp->f_flags & O_NONBLOCK) { if (down_trylock(&port->sm_sem)) { @@ -898,17 +897,27 @@ static int ib_umad_sm_open(struct inode *inode, struct file *filp) } ret = ib_modify_port(port->ib_dev, port->port_num, 0, &props); - if (ret) { - up(&port->sm_sem); - goto fail; - } + if (ret) + goto err_up_sem; filp->private_data = port; - return nonseekable_open(inode, filp); + ret = nonseekable_open(inode, filp); + if (ret) + goto err_clr_sm_cap; + + kobject_get(&port->umad_dev->kobj); + + return 0; + +err_clr_sm_cap: + swap(props.set_port_cap_mask, props.clr_port_cap_mask); + ib_modify_port(port->ib_dev, port->port_num, 0, &props); + +err_up_sem: + up(&port->sm_sem); fail: - kref_put(&port->umad_dev->ref, ib_umad_release_dev); return ret; } @@ -927,7 +936,7 @@ static int ib_umad_sm_close(struct inode *inode, struct file *filp) up(&port->sm_sem); - kref_put(&port->umad_dev->ref, ib_umad_release_dev); + kobject_put(&port->umad_dev->kobj); return ret; } @@ -995,6 +1004,7 @@ static int find_overflow_devnum(void) } static int ib_umad_init_port(struct ib_device *device, int port_num, + struct ib_umad_device *umad_dev, struct ib_umad_port *port) { int devnum; @@ -1027,6 +1037,7 @@ static int ib_umad_init_port(struct ib_device *device, int port_num, cdev_init(&port->cdev, &umad_fops); port->cdev.owner = THIS_MODULE; + port->cdev.kobj.parent = &umad_dev->kobj; kobject_set_name(&port->cdev.kobj, "umad%d", port->dev_num); if (cdev_add(&port->cdev, base, 1)) goto err_cdev; @@ -1045,6 +1056,7 @@ static int ib_umad_init_port(struct ib_device *device, int port_num, base += IB_UMAD_MAX_PORTS; cdev_init(&port->sm_cdev, &umad_sm_fops); port->sm_cdev.owner = THIS_MODULE; + port->sm_cdev.kobj.parent = &umad_dev->kobj; kobject_set_name(&port->sm_cdev.kobj, "issm%d", port->dev_num); if (cdev_add(&port->sm_cdev, base, 1)) goto err_sm_cdev; @@ -1138,7 +1150,7 @@ static void ib_umad_add_one(struct ib_device *device) if (!umad_dev) return; - kref_init(&umad_dev->ref); + kobject_init(&umad_dev->kobj, &ib_umad_dev_ktype); umad_dev->start_port = s; umad_dev->end_port = e; @@ -1146,7 +1158,8 @@ static void ib_umad_add_one(struct ib_device *device) for (i = s; i <= e; ++i) { umad_dev->port[i - s].umad_dev = umad_dev; - if (ib_umad_init_port(device, i, &umad_dev->port[i - s])) + if (ib_umad_init_port(device, i, umad_dev, + &umad_dev->port[i - s])) goto err; } @@ -1158,7 +1171,7 @@ err: while (--i >= s) ib_umad_kill_port(&umad_dev->port[i - s]); - kref_put(&umad_dev->ref, ib_umad_release_dev); + kobject_put(&umad_dev->kobj); } static void ib_umad_remove_one(struct ib_device *device) @@ -1172,7 +1185,7 @@ static void ib_umad_remove_one(struct ib_device *device) for (i = 0; i <= umad_dev->end_port - umad_dev->start_port; ++i) ib_umad_kill_port(&umad_dev->port[i]); - kref_put(&umad_dev->ref, ib_umad_release_dev); + kobject_put(&umad_dev->kobj); } static char *umad_devnode(struct device *dev, umode_t *mode) diff --git a/drivers/infiniband/core/verbs.c b/drivers/infiniband/core/verbs.c index 92525f855d82..c2b89cc5dbca 100644 --- a/drivers/infiniband/core/verbs.c +++ b/drivers/infiniband/core/verbs.c @@ -48,7 +48,7 @@ #include "core_priv.h" -int ib_rate_to_mult(enum ib_rate rate) +__attribute_const__ int ib_rate_to_mult(enum ib_rate rate) { switch (rate) { case IB_RATE_2_5_GBPS: return 1; @@ -65,7 +65,7 @@ int ib_rate_to_mult(enum ib_rate rate) } EXPORT_SYMBOL(ib_rate_to_mult); -enum ib_rate mult_to_ib_rate(int mult) +__attribute_const__ enum ib_rate mult_to_ib_rate(int mult) { switch (mult) { case 1: return IB_RATE_2_5_GBPS; @@ -82,7 +82,7 @@ enum ib_rate mult_to_ib_rate(int mult) } EXPORT_SYMBOL(mult_to_ib_rate); -int ib_rate_to_mbps(enum ib_rate rate) +__attribute_const__ int ib_rate_to_mbps(enum ib_rate rate) { switch (rate) { case IB_RATE_2_5_GBPS: return 2500; @@ -107,7 +107,7 @@ int ib_rate_to_mbps(enum ib_rate rate) } EXPORT_SYMBOL(ib_rate_to_mbps); -enum rdma_transport_type +__attribute_const__ enum rdma_transport_type rdma_node_get_transport(enum rdma_node_type node_type) { switch (node_type) { diff --git a/drivers/infiniband/hw/Makefile b/drivers/infiniband/hw/Makefile new file mode 100644 index 000000000000..e900b03531a9 --- /dev/null +++ b/drivers/infiniband/hw/Makefile @@ -0,0 +1,12 @@ +obj-$(CONFIG_INFINIBAND_MTHCA) += mthca/ +obj-$(CONFIG_INFINIBAND_IPATH) += ipath/ +obj-$(CONFIG_INFINIBAND_QIB) += qib/ +obj-$(CONFIG_INFINIBAND_EHCA) += ehca/ +obj-$(CONFIG_INFINIBAND_AMSO1100) += amso1100/ +obj-$(CONFIG_INFINIBAND_CXGB3) += cxgb3/ +obj-$(CONFIG_INFINIBAND_CXGB4) += cxgb4/ +obj-$(CONFIG_MLX4_INFINIBAND) += mlx4/ +obj-$(CONFIG_MLX5_INFINIBAND) += mlx5/ +obj-$(CONFIG_INFINIBAND_NES) += nes/ +obj-$(CONFIG_INFINIBAND_OCRDMA) += ocrdma/ +obj-$(CONFIG_INFINIBAND_USNIC) += usnic/ diff --git a/drivers/infiniband/hw/cxgb3/cxio_hal.c b/drivers/infiniband/hw/cxgb3/cxio_hal.c index c3f5aca4ef00..de1c61b417d6 100644 --- a/drivers/infiniband/hw/cxgb3/cxio_hal.c +++ b/drivers/infiniband/hw/cxgb3/cxio_hal.c @@ -735,14 +735,12 @@ static int __cxio_tpt_op(struct cxio_rdev *rdev_p, u32 reset_tpt_entry, ((perm & TPT_MW_BIND) ? F_TPT_MW_BIND_ENABLE : 0) | V_TPT_ADDR_TYPE((zbva ? TPT_ZBTO : TPT_VATO)) | V_TPT_PAGE_SIZE(page_size)); - tpt.rsvd_pbl_addr = reset_tpt_entry ? 0 : - cpu_to_be32(V_TPT_PBL_ADDR(PBL_OFF(rdev_p, pbl_addr)>>3)); + tpt.rsvd_pbl_addr = cpu_to_be32(V_TPT_PBL_ADDR(PBL_OFF(rdev_p, pbl_addr)>>3)); tpt.len = cpu_to_be32(len); tpt.va_hi = cpu_to_be32((u32) (to >> 32)); tpt.va_low_or_fbo = cpu_to_be32((u32) (to & 0xFFFFFFFFULL)); tpt.rsvd_bind_cnt_or_pstag = 0; - tpt.rsvd_pbl_size = reset_tpt_entry ? 0 : - cpu_to_be32(V_TPT_PBL_SIZE(pbl_size >> 2)); + tpt.rsvd_pbl_size = cpu_to_be32(V_TPT_PBL_SIZE(pbl_size >> 2)); } err = cxio_hal_ctrl_qp_write_mem(rdev_p, stag_idx + diff --git a/drivers/infiniband/hw/cxgb3/iwch_cm.c b/drivers/infiniband/hw/cxgb3/iwch_cm.c index 095bb046e2c8..cb78b1e9bcd9 100644 --- a/drivers/infiniband/hw/cxgb3/iwch_cm.c +++ b/drivers/infiniband/hw/cxgb3/iwch_cm.c @@ -418,6 +418,7 @@ static int send_abort(struct iwch_ep *ep, struct sk_buff *skb, gfp_t gfp) skb->priority = CPL_PRIORITY_DATA; set_arp_failure_handler(skb, abort_arp_failure); req = (struct cpl_abort_req *) skb_put(skb, sizeof(*req)); + memset(req, 0, sizeof(*req)); req->wr.wr_hi = htonl(V_WR_OP(FW_WROPCODE_OFLD_HOST_ABORT_CON_REQ)); req->wr.wr_lo = htonl(V_WR_TID(ep->hwtid)); OPCODE_TID(req) = htonl(MK_OPCODE_TID(CPL_ABORT_REQ, ep->hwtid)); diff --git a/drivers/infiniband/hw/cxgb4/cm.c b/drivers/infiniband/hw/cxgb4/cm.c index 1f863a96a480..96d7131ab974 100644 --- a/drivers/infiniband/hw/cxgb4/cm.c +++ b/drivers/infiniband/hw/cxgb4/cm.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2009-2010 Chelsio, Inc. All rights reserved. + * Copyright (c) 2009-2014 Chelsio, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -47,6 +47,8 @@ #include <net/ip6_route.h> #include <net/addrconf.h> +#include <rdma/ib_addr.h> + #include "iw_cxgb4.h" static char *states[] = { @@ -294,6 +296,12 @@ void _c4iw_free_ep(struct kref *kref) dst_release(ep->dst); cxgb4_l2t_release(ep->l2t); } + if (test_bit(RELEASE_MAPINFO, &ep->com.flags)) { + print_addr(&ep->com, __func__, "remove_mapinfo/mapping"); + iwpm_remove_mapinfo(&ep->com.local_addr, + &ep->com.mapped_local_addr); + iwpm_remove_mapping(&ep->com.local_addr, RDMA_NL_C4IW); + } kfree(ep); } @@ -341,10 +349,7 @@ static struct sk_buff *get_skb(struct sk_buff *skb, int len, gfp_t gfp) static struct net_device *get_real_dev(struct net_device *egress_dev) { - struct net_device *phys_dev = egress_dev; - if (egress_dev->priv_flags & IFF_802_1Q_VLAN) - phys_dev = vlan_dev_real_dev(egress_dev); - return phys_dev; + return rdma_vlan_dev_real_dev(egress_dev) ? : egress_dev; } static int our_interface(struct c4iw_dev *dev, struct net_device *egress_dev) @@ -528,6 +533,38 @@ static int send_abort(struct c4iw_ep *ep, struct sk_buff *skb, gfp_t gfp) return c4iw_l2t_send(&ep->com.dev->rdev, skb, ep->l2t); } +/* + * c4iw_form_pm_msg - Form a port mapper message with mapping info + */ +static void c4iw_form_pm_msg(struct c4iw_ep *ep, + struct iwpm_sa_data *pm_msg) +{ + memcpy(&pm_msg->loc_addr, &ep->com.local_addr, + sizeof(ep->com.local_addr)); + memcpy(&pm_msg->rem_addr, &ep->com.remote_addr, + sizeof(ep->com.remote_addr)); +} + +/* + * c4iw_form_reg_msg - Form a port mapper message with dev info + */ +static void c4iw_form_reg_msg(struct c4iw_dev *dev, + struct iwpm_dev_data *pm_msg) +{ + memcpy(pm_msg->dev_name, dev->ibdev.name, IWPM_DEVNAME_SIZE); + memcpy(pm_msg->if_name, dev->rdev.lldi.ports[0]->name, + IWPM_IFNAME_SIZE); +} + +static void c4iw_record_pm_msg(struct c4iw_ep *ep, + struct iwpm_sa_data *pm_msg) +{ + memcpy(&ep->com.mapped_local_addr, &pm_msg->mapped_loc_addr, + sizeof(ep->com.mapped_local_addr)); + memcpy(&ep->com.mapped_remote_addr, &pm_msg->mapped_rem_addr, + sizeof(ep->com.mapped_remote_addr)); +} + static int send_connect(struct c4iw_ep *ep) { struct cpl_act_open_req *req; @@ -546,10 +583,14 @@ static int send_connect(struct c4iw_ep *ep) int sizev6 = is_t4(ep->com.dev->rdev.lldi.adapter_type) ? sizeof(struct cpl_act_open_req6) : sizeof(struct cpl_t5_act_open_req6); - struct sockaddr_in *la = (struct sockaddr_in *)&ep->com.local_addr; - struct sockaddr_in *ra = (struct sockaddr_in *)&ep->com.remote_addr; - struct sockaddr_in6 *la6 = (struct sockaddr_in6 *)&ep->com.local_addr; - struct sockaddr_in6 *ra6 = (struct sockaddr_in6 *)&ep->com.remote_addr; + struct sockaddr_in *la = (struct sockaddr_in *) + &ep->com.mapped_local_addr; + struct sockaddr_in *ra = (struct sockaddr_in *) + &ep->com.mapped_remote_addr; + struct sockaddr_in6 *la6 = (struct sockaddr_in6 *) + &ep->com.mapped_local_addr; + struct sockaddr_in6 *ra6 = (struct sockaddr_in6 *) + &ep->com.mapped_remote_addr; wrlen = (ep->com.remote_addr.ss_family == AF_INET) ? roundup(sizev4, 16) : @@ -1627,10 +1668,10 @@ static void send_fw_act_open_req(struct c4iw_ep *ep, unsigned int atid) req->le.filter = cpu_to_be32(cxgb4_select_ntuple( ep->com.dev->rdev.lldi.ports[0], ep->l2t)); - sin = (struct sockaddr_in *)&ep->com.local_addr; + sin = (struct sockaddr_in *)&ep->com.mapped_local_addr; req->le.lport = sin->sin_port; req->le.u.ipv4.lip = sin->sin_addr.s_addr; - sin = (struct sockaddr_in *)&ep->com.remote_addr; + sin = (struct sockaddr_in *)&ep->com.mapped_remote_addr; req->le.pport = sin->sin_port; req->le.u.ipv4.pip = sin->sin_addr.s_addr; req->tcb.t_state_to_astid = @@ -1746,16 +1787,16 @@ static int import_ep(struct c4iw_ep *ep, int iptype, __u8 *peer_ip, if (!ep->l2t) goto out; ep->mtu = dst_mtu(dst); - ep->tx_chan = cxgb4_port_chan(n->dev); - ep->smac_idx = (cxgb4_port_viid(n->dev) & 0x7F) << 1; + ep->tx_chan = cxgb4_port_chan(pdev); + ep->smac_idx = (cxgb4_port_viid(pdev) & 0x7F) << 1; step = cdev->rdev.lldi.ntxq / cdev->rdev.lldi.nchan; - ep->txq_idx = cxgb4_port_idx(n->dev) * step; - ep->ctrlq_idx = cxgb4_port_idx(n->dev); + ep->txq_idx = cxgb4_port_idx(pdev) * step; + ep->ctrlq_idx = cxgb4_port_idx(pdev); step = cdev->rdev.lldi.nrxq / cdev->rdev.lldi.nchan; ep->rss_qid = cdev->rdev.lldi.rxq_ids[ - cxgb4_port_idx(n->dev) * step]; + cxgb4_port_idx(pdev) * step]; if (clear_mpa_v1) { ep->retry_with_mpa_v1 = 0; @@ -1870,10 +1911,10 @@ static int act_open_rpl(struct c4iw_dev *dev, struct sk_buff *skb) struct sockaddr_in6 *ra6; ep = lookup_atid(t, atid); - la = (struct sockaddr_in *)&ep->com.local_addr; - ra = (struct sockaddr_in *)&ep->com.remote_addr; - la6 = (struct sockaddr_in6 *)&ep->com.local_addr; - ra6 = (struct sockaddr_in6 *)&ep->com.remote_addr; + la = (struct sockaddr_in *)&ep->com.mapped_local_addr; + ra = (struct sockaddr_in *)&ep->com.mapped_remote_addr; + la6 = (struct sockaddr_in6 *)&ep->com.mapped_local_addr; + ra6 = (struct sockaddr_in6 *)&ep->com.mapped_remote_addr; PDBG("%s ep %p atid %u status %u errno %d\n", __func__, ep, atid, status, status2errno(status)); @@ -2730,13 +2771,15 @@ int c4iw_connect(struct iw_cm_id *cm_id, struct iw_cm_conn_param *conn_param) struct c4iw_dev *dev = to_c4iw_dev(cm_id->device); struct c4iw_ep *ep; int err = 0; - struct sockaddr_in *laddr = (struct sockaddr_in *)&cm_id->local_addr; - struct sockaddr_in *raddr = (struct sockaddr_in *)&cm_id->remote_addr; - struct sockaddr_in6 *laddr6 = (struct sockaddr_in6 *)&cm_id->local_addr; - struct sockaddr_in6 *raddr6 = (struct sockaddr_in6 *) - &cm_id->remote_addr; + struct sockaddr_in *laddr; + struct sockaddr_in *raddr; + struct sockaddr_in6 *laddr6; + struct sockaddr_in6 *raddr6; + struct iwpm_dev_data pm_reg_msg; + struct iwpm_sa_data pm_msg; __u8 *ra; int iptype; + int iwpm_err = 0; if ((conn_param->ord > c4iw_max_read_depth) || (conn_param->ird > c4iw_max_read_depth)) { @@ -2767,7 +2810,7 @@ int c4iw_connect(struct iw_cm_id *cm_id, struct iw_cm_conn_param *conn_param) if (!ep->com.qp) { PDBG("%s qpn 0x%x not found!\n", __func__, conn_param->qpn); err = -EINVAL; - goto fail2; + goto fail1; } ref_qp(ep); PDBG("%s qpn 0x%x qp %p cm_id %p\n", __func__, conn_param->qpn, @@ -2780,10 +2823,50 @@ int c4iw_connect(struct iw_cm_id *cm_id, struct iw_cm_conn_param *conn_param) if (ep->atid == -1) { printk(KERN_ERR MOD "%s - cannot alloc atid.\n", __func__); err = -ENOMEM; - goto fail2; + goto fail1; } insert_handle(dev, &dev->atid_idr, ep, ep->atid); + memcpy(&ep->com.local_addr, &cm_id->local_addr, + sizeof(ep->com.local_addr)); + memcpy(&ep->com.remote_addr, &cm_id->remote_addr, + sizeof(ep->com.remote_addr)); + + /* No port mapper available, go with the specified peer information */ + memcpy(&ep->com.mapped_local_addr, &cm_id->local_addr, + sizeof(ep->com.mapped_local_addr)); + memcpy(&ep->com.mapped_remote_addr, &cm_id->remote_addr, + sizeof(ep->com.mapped_remote_addr)); + + c4iw_form_reg_msg(dev, &pm_reg_msg); + iwpm_err = iwpm_register_pid(&pm_reg_msg, RDMA_NL_C4IW); + if (iwpm_err) { + PDBG("%s: Port Mapper reg pid fail (err = %d).\n", + __func__, iwpm_err); + } + if (iwpm_valid_pid() && !iwpm_err) { + c4iw_form_pm_msg(ep, &pm_msg); + iwpm_err = iwpm_add_and_query_mapping(&pm_msg, RDMA_NL_C4IW); + if (iwpm_err) + PDBG("%s: Port Mapper query fail (err = %d).\n", + __func__, iwpm_err); + else + c4iw_record_pm_msg(ep, &pm_msg); + } + if (iwpm_create_mapinfo(&ep->com.local_addr, + &ep->com.mapped_local_addr, RDMA_NL_C4IW)) { + iwpm_remove_mapping(&ep->com.local_addr, RDMA_NL_C4IW); + err = -ENOMEM; + goto fail1; + } + print_addr(&ep->com, __func__, "add_query/create_mapinfo"); + set_bit(RELEASE_MAPINFO, &ep->com.flags); + + laddr = (struct sockaddr_in *)&ep->com.mapped_local_addr; + raddr = (struct sockaddr_in *)&ep->com.mapped_remote_addr; + laddr6 = (struct sockaddr_in6 *)&ep->com.mapped_local_addr; + raddr6 = (struct sockaddr_in6 *) &ep->com.mapped_remote_addr; + if (cm_id->remote_addr.ss_family == AF_INET) { iptype = 4; ra = (__u8 *)&raddr->sin_addr; @@ -2794,7 +2877,7 @@ int c4iw_connect(struct iw_cm_id *cm_id, struct iw_cm_conn_param *conn_param) if ((__force int)raddr->sin_addr.s_addr == INADDR_ANY) { err = pick_local_ipaddrs(dev, cm_id); if (err) - goto fail2; + goto fail1; } /* find a route */ @@ -2814,7 +2897,7 @@ int c4iw_connect(struct iw_cm_id *cm_id, struct iw_cm_conn_param *conn_param) if (ipv6_addr_type(&raddr6->sin6_addr) == IPV6_ADDR_ANY) { err = pick_local_ip6addrs(dev, cm_id); if (err) - goto fail2; + goto fail1; } /* find a route */ @@ -2830,13 +2913,13 @@ int c4iw_connect(struct iw_cm_id *cm_id, struct iw_cm_conn_param *conn_param) if (!ep->dst) { printk(KERN_ERR MOD "%s - cannot find route.\n", __func__); err = -EHOSTUNREACH; - goto fail3; + goto fail2; } err = import_ep(ep, iptype, ra, ep->dst, ep->com.dev, true); if (err) { printk(KERN_ERR MOD "%s - cannot alloc l2e.\n", __func__); - goto fail4; + goto fail3; } PDBG("%s txq_idx %u tx_chan %u smac_idx %u rss_qid %u l2t_idx %u\n", @@ -2845,10 +2928,6 @@ int c4iw_connect(struct iw_cm_id *cm_id, struct iw_cm_conn_param *conn_param) state_set(&ep->com, CONNECTING); ep->tos = 0; - memcpy(&ep->com.local_addr, &cm_id->local_addr, - sizeof(ep->com.local_addr)); - memcpy(&ep->com.remote_addr, &cm_id->remote_addr, - sizeof(ep->com.remote_addr)); /* send connect request to rnic */ err = send_connect(ep); @@ -2856,12 +2935,12 @@ int c4iw_connect(struct iw_cm_id *cm_id, struct iw_cm_conn_param *conn_param) goto out; cxgb4_l2t_release(ep->l2t); -fail4: - dst_release(ep->dst); fail3: + dst_release(ep->dst); +fail2: remove_handle(ep->com.dev, &ep->com.dev->atid_idr, ep->atid); cxgb4_free_atid(ep->com.dev->rdev.lldi.tids, ep->atid); -fail2: +fail1: cm_id->rem_ref(cm_id); c4iw_put_ep(&ep->com); out: @@ -2871,7 +2950,8 @@ out: static int create_server6(struct c4iw_dev *dev, struct c4iw_listen_ep *ep) { int err; - struct sockaddr_in6 *sin6 = (struct sockaddr_in6 *)&ep->com.local_addr; + struct sockaddr_in6 *sin6 = (struct sockaddr_in6 *) + &ep->com.mapped_local_addr; c4iw_init_wr_wait(&ep->com.wr_wait); err = cxgb4_create_server6(ep->com.dev->rdev.lldi.ports[0], @@ -2892,7 +2972,8 @@ static int create_server6(struct c4iw_dev *dev, struct c4iw_listen_ep *ep) static int create_server4(struct c4iw_dev *dev, struct c4iw_listen_ep *ep) { int err; - struct sockaddr_in *sin = (struct sockaddr_in *)&ep->com.local_addr; + struct sockaddr_in *sin = (struct sockaddr_in *) + &ep->com.mapped_local_addr; if (dev->rdev.lldi.enable_fw_ofld_conn) { do { @@ -2927,6 +3008,9 @@ int c4iw_create_listen(struct iw_cm_id *cm_id, int backlog) int err = 0; struct c4iw_dev *dev = to_c4iw_dev(cm_id->device); struct c4iw_listen_ep *ep; + struct iwpm_dev_data pm_reg_msg; + struct iwpm_sa_data pm_msg; + int iwpm_err = 0; might_sleep(); @@ -2961,6 +3045,37 @@ int c4iw_create_listen(struct iw_cm_id *cm_id, int backlog) goto fail2; } insert_handle(dev, &dev->stid_idr, ep, ep->stid); + + /* No port mapper available, go with the specified info */ + memcpy(&ep->com.mapped_local_addr, &cm_id->local_addr, + sizeof(ep->com.mapped_local_addr)); + + c4iw_form_reg_msg(dev, &pm_reg_msg); + iwpm_err = iwpm_register_pid(&pm_reg_msg, RDMA_NL_C4IW); + if (iwpm_err) { + PDBG("%s: Port Mapper reg pid fail (err = %d).\n", + __func__, iwpm_err); + } + if (iwpm_valid_pid() && !iwpm_err) { + memcpy(&pm_msg.loc_addr, &ep->com.local_addr, + sizeof(ep->com.local_addr)); + iwpm_err = iwpm_add_mapping(&pm_msg, RDMA_NL_C4IW); + if (iwpm_err) + PDBG("%s: Port Mapper query fail (err = %d).\n", + __func__, iwpm_err); + else + memcpy(&ep->com.mapped_local_addr, + &pm_msg.mapped_loc_addr, + sizeof(ep->com.mapped_local_addr)); + } + if (iwpm_create_mapinfo(&ep->com.local_addr, + &ep->com.mapped_local_addr, RDMA_NL_C4IW)) { + err = -ENOMEM; + goto fail3; + } + print_addr(&ep->com, __func__, "add_mapping/create_mapinfo"); + + set_bit(RELEASE_MAPINFO, &ep->com.flags); state_set(&ep->com, LISTEN); if (ep->com.local_addr.ss_family == AF_INET) err = create_server4(dev, ep); @@ -2970,6 +3085,8 @@ int c4iw_create_listen(struct iw_cm_id *cm_id, int backlog) cm_id->provider_data = ep; goto out; } + +fail3: cxgb4_free_stid(ep->com.dev->rdev.lldi.tids, ep->stid, ep->com.local_addr.ss_family); fail2: diff --git a/drivers/infiniband/hw/cxgb4/cq.c b/drivers/infiniband/hw/cxgb4/cq.c index cfaa56ada189..7151a02b4ebb 100644 --- a/drivers/infiniband/hw/cxgb4/cq.c +++ b/drivers/infiniband/hw/cxgb4/cq.c @@ -940,7 +940,6 @@ struct ib_cq *c4iw_create_cq(struct ib_device *ibdev, int entries, if (!mm2) goto err4; - memset(&uresp, 0, sizeof(uresp)); uresp.qid_mask = rhp->rdev.cqmask; uresp.cqid = chp->cq.cqid; uresp.size = chp->cq.size; @@ -951,7 +950,8 @@ struct ib_cq *c4iw_create_cq(struct ib_device *ibdev, int entries, uresp.gts_key = ucontext->key; ucontext->key += PAGE_SIZE; spin_unlock(&ucontext->mmap_lock); - ret = ib_copy_to_udata(udata, &uresp, sizeof uresp); + ret = ib_copy_to_udata(udata, &uresp, + sizeof(uresp) - sizeof(uresp.reserved)); if (ret) goto err5; diff --git a/drivers/infiniband/hw/cxgb4/device.c b/drivers/infiniband/hw/cxgb4/device.c index f4fa50a609e2..dd93aadc996e 100644 --- a/drivers/infiniband/hw/cxgb4/device.c +++ b/drivers/infiniband/hw/cxgb4/device.c @@ -77,6 +77,16 @@ struct c4iw_debugfs_data { int pos; }; +/* registered cxgb4 netlink callbacks */ +static struct ibnl_client_cbs c4iw_nl_cb_table[] = { + [RDMA_NL_IWPM_REG_PID] = {.dump = iwpm_register_pid_cb}, + [RDMA_NL_IWPM_ADD_MAPPING] = {.dump = iwpm_add_mapping_cb}, + [RDMA_NL_IWPM_QUERY_MAPPING] = {.dump = iwpm_add_and_query_mapping_cb}, + [RDMA_NL_IWPM_HANDLE_ERR] = {.dump = iwpm_mapping_error_cb}, + [RDMA_NL_IWPM_MAPINFO] = {.dump = iwpm_mapping_info_cb}, + [RDMA_NL_IWPM_MAPINFO_NUM] = {.dump = iwpm_ack_mapping_info_cb} +}; + static int count_idrs(int id, void *p, void *data) { int *countp = data; @@ -113,35 +123,49 @@ static int dump_qp(int id, void *p, void *data) &qp->ep->com.local_addr; struct sockaddr_in *rsin = (struct sockaddr_in *) &qp->ep->com.remote_addr; + struct sockaddr_in *mapped_lsin = (struct sockaddr_in *) + &qp->ep->com.mapped_local_addr; + struct sockaddr_in *mapped_rsin = (struct sockaddr_in *) + &qp->ep->com.mapped_remote_addr; cc = snprintf(qpd->buf + qpd->pos, space, "rc qp sq id %u rq id %u state %u " "onchip %u ep tid %u state %u " - "%pI4:%u->%pI4:%u\n", + "%pI4:%u/%u->%pI4:%u/%u\n", qp->wq.sq.qid, qp->wq.rq.qid, (int)qp->attr.state, qp->wq.sq.flags & T4_SQ_ONCHIP, qp->ep->hwtid, (int)qp->ep->com.state, &lsin->sin_addr, ntohs(lsin->sin_port), - &rsin->sin_addr, ntohs(rsin->sin_port)); + ntohs(mapped_lsin->sin_port), + &rsin->sin_addr, ntohs(rsin->sin_port), + ntohs(mapped_rsin->sin_port)); } else { struct sockaddr_in6 *lsin6 = (struct sockaddr_in6 *) &qp->ep->com.local_addr; struct sockaddr_in6 *rsin6 = (struct sockaddr_in6 *) &qp->ep->com.remote_addr; + struct sockaddr_in6 *mapped_lsin6 = + (struct sockaddr_in6 *) + &qp->ep->com.mapped_local_addr; + struct sockaddr_in6 *mapped_rsin6 = + (struct sockaddr_in6 *) + &qp->ep->com.mapped_remote_addr; cc = snprintf(qpd->buf + qpd->pos, space, "rc qp sq id %u rq id %u state %u " "onchip %u ep tid %u state %u " - "%pI6:%u->%pI6:%u\n", + "%pI6:%u/%u->%pI6:%u/%u\n", qp->wq.sq.qid, qp->wq.rq.qid, (int)qp->attr.state, qp->wq.sq.flags & T4_SQ_ONCHIP, qp->ep->hwtid, (int)qp->ep->com.state, &lsin6->sin6_addr, ntohs(lsin6->sin6_port), + ntohs(mapped_lsin6->sin6_port), &rsin6->sin6_addr, - ntohs(rsin6->sin6_port)); + ntohs(rsin6->sin6_port), + ntohs(mapped_rsin6->sin6_port)); } } else cc = snprintf(qpd->buf + qpd->pos, space, @@ -386,31 +410,43 @@ static int dump_ep(int id, void *p, void *data) &ep->com.local_addr; struct sockaddr_in *rsin = (struct sockaddr_in *) &ep->com.remote_addr; + struct sockaddr_in *mapped_lsin = (struct sockaddr_in *) + &ep->com.mapped_local_addr; + struct sockaddr_in *mapped_rsin = (struct sockaddr_in *) + &ep->com.mapped_remote_addr; cc = snprintf(epd->buf + epd->pos, space, "ep %p cm_id %p qp %p state %d flags 0x%lx " "history 0x%lx hwtid %d atid %d " - "%pI4:%d <-> %pI4:%d\n", + "%pI4:%d/%d <-> %pI4:%d/%d\n", ep, ep->com.cm_id, ep->com.qp, (int)ep->com.state, ep->com.flags, ep->com.history, ep->hwtid, ep->atid, &lsin->sin_addr, ntohs(lsin->sin_port), - &rsin->sin_addr, ntohs(rsin->sin_port)); + ntohs(mapped_lsin->sin_port), + &rsin->sin_addr, ntohs(rsin->sin_port), + ntohs(mapped_rsin->sin_port)); } else { struct sockaddr_in6 *lsin6 = (struct sockaddr_in6 *) &ep->com.local_addr; struct sockaddr_in6 *rsin6 = (struct sockaddr_in6 *) &ep->com.remote_addr; + struct sockaddr_in6 *mapped_lsin6 = (struct sockaddr_in6 *) + &ep->com.mapped_local_addr; + struct sockaddr_in6 *mapped_rsin6 = (struct sockaddr_in6 *) + &ep->com.mapped_remote_addr; cc = snprintf(epd->buf + epd->pos, space, "ep %p cm_id %p qp %p state %d flags 0x%lx " "history 0x%lx hwtid %d atid %d " - "%pI6:%d <-> %pI6:%d\n", + "%pI6:%d/%d <-> %pI6:%d/%d\n", ep, ep->com.cm_id, ep->com.qp, (int)ep->com.state, ep->com.flags, ep->com.history, ep->hwtid, ep->atid, &lsin6->sin6_addr, ntohs(lsin6->sin6_port), - &rsin6->sin6_addr, ntohs(rsin6->sin6_port)); + ntohs(mapped_lsin6->sin6_port), + &rsin6->sin6_addr, ntohs(rsin6->sin6_port), + ntohs(mapped_rsin6->sin6_port)); } if (cc < space) epd->pos += cc; @@ -431,23 +467,29 @@ static int dump_listen_ep(int id, void *p, void *data) if (ep->com.local_addr.ss_family == AF_INET) { struct sockaddr_in *lsin = (struct sockaddr_in *) &ep->com.local_addr; + struct sockaddr_in *mapped_lsin = (struct sockaddr_in *) + &ep->com.mapped_local_addr; cc = snprintf(epd->buf + epd->pos, space, "ep %p cm_id %p state %d flags 0x%lx stid %d " - "backlog %d %pI4:%d\n", + "backlog %d %pI4:%d/%d\n", ep, ep->com.cm_id, (int)ep->com.state, ep->com.flags, ep->stid, ep->backlog, - &lsin->sin_addr, ntohs(lsin->sin_port)); + &lsin->sin_addr, ntohs(lsin->sin_port), + ntohs(mapped_lsin->sin_port)); } else { struct sockaddr_in6 *lsin6 = (struct sockaddr_in6 *) &ep->com.local_addr; + struct sockaddr_in6 *mapped_lsin6 = (struct sockaddr_in6 *) + &ep->com.mapped_local_addr; cc = snprintf(epd->buf + epd->pos, space, "ep %p cm_id %p state %d flags 0x%lx stid %d " - "backlog %d %pI6:%d\n", + "backlog %d %pI6:%d/%d\n", ep, ep->com.cm_id, (int)ep->com.state, ep->com.flags, ep->stid, ep->backlog, - &lsin6->sin6_addr, ntohs(lsin6->sin6_port)); + &lsin6->sin6_addr, ntohs(lsin6->sin6_port), + ntohs(mapped_lsin6->sin6_port)); } if (cc < space) epd->pos += cc; @@ -687,6 +729,7 @@ static void c4iw_dealloc(struct uld_ctx *ctx) if (ctx->dev->rdev.oc_mw_kva) iounmap(ctx->dev->rdev.oc_mw_kva); ib_dealloc_device(&ctx->dev->ibdev); + iwpm_exit(RDMA_NL_C4IW); ctx->dev = NULL; } @@ -736,6 +779,7 @@ static struct c4iw_dev *c4iw_alloc(const struct cxgb4_lld_info *infop) pci_resource_len(devp->rdev.lldi.pdev, 2)); if (!devp->rdev.bar2_kva) { pr_err(MOD "Unable to ioremap BAR2\n"); + ib_dealloc_device(&devp->ibdev); return ERR_PTR(-EINVAL); } } else if (ocqp_supported(infop)) { @@ -747,6 +791,7 @@ static struct c4iw_dev *c4iw_alloc(const struct cxgb4_lld_info *infop) devp->rdev.lldi.vr->ocq.size); if (!devp->rdev.oc_mw_kva) { pr_err(MOD "Unable to ioremap onchip mem\n"); + ib_dealloc_device(&devp->ibdev); return ERR_PTR(-EINVAL); } } @@ -780,6 +825,14 @@ static struct c4iw_dev *c4iw_alloc(const struct cxgb4_lld_info *infop) c4iw_debugfs_root); setup_debugfs(devp); } + + ret = iwpm_init(RDMA_NL_C4IW); + if (ret) { + pr_err("port mapper initialization failed with %d\n", ret); + ib_dealloc_device(&devp->ibdev); + return ERR_PTR(ret); + } + return devp; } @@ -1274,6 +1327,11 @@ static int __init c4iw_init_module(void) printk(KERN_WARNING MOD "could not create debugfs entry, continuing\n"); + if (ibnl_add_client(RDMA_NL_C4IW, RDMA_NL_IWPM_NUM_OPS, + c4iw_nl_cb_table)) + pr_err("%s[%u]: Failed to add netlink callback\n" + , __func__, __LINE__); + cxgb4_register_uld(CXGB4_ULD_RDMA, &c4iw_uld_info); return 0; @@ -1291,6 +1349,7 @@ static void __exit c4iw_exit_module(void) } mutex_unlock(&dev_mutex); cxgb4_unregister_uld(CXGB4_ULD_RDMA); + ibnl_remove_client(RDMA_NL_C4IW); c4iw_cm_term(); debugfs_remove_recursive(c4iw_debugfs_root); } diff --git a/drivers/infiniband/hw/cxgb4/iw_cxgb4.h b/drivers/infiniband/hw/cxgb4/iw_cxgb4.h index 7474b490760a..6f533fbcc4b3 100644 --- a/drivers/infiniband/hw/cxgb4/iw_cxgb4.h +++ b/drivers/infiniband/hw/cxgb4/iw_cxgb4.h @@ -52,6 +52,8 @@ #include <rdma/ib_verbs.h> #include <rdma/iw_cm.h> +#include <rdma/rdma_netlink.h> +#include <rdma/iw_portmap.h> #include "cxgb4.h" #include "cxgb4_uld.h" @@ -728,6 +730,7 @@ enum c4iw_ep_flags { CLOSE_SENT = 3, TIMEOUT = 4, QP_REFERENCED = 5, + RELEASE_MAPINFO = 6, }; enum c4iw_ep_history { @@ -764,6 +767,8 @@ struct c4iw_ep_common { struct mutex mutex; struct sockaddr_storage local_addr; struct sockaddr_storage remote_addr; + struct sockaddr_storage mapped_local_addr; + struct sockaddr_storage mapped_remote_addr; struct c4iw_wr_wait wr_wait; unsigned long flags; unsigned long history; @@ -807,6 +812,45 @@ struct c4iw_ep { unsigned int retry_count; }; +static inline void print_addr(struct c4iw_ep_common *epc, const char *func, + const char *msg) +{ + +#define SINA(a) (&(((struct sockaddr_in *)(a))->sin_addr.s_addr)) +#define SINP(a) ntohs(((struct sockaddr_in *)(a))->sin_port) +#define SIN6A(a) (&(((struct sockaddr_in6 *)(a))->sin6_addr)) +#define SIN6P(a) ntohs(((struct sockaddr_in6 *)(a))->sin6_port) + + if (c4iw_debug) { + switch (epc->local_addr.ss_family) { + case AF_INET: + PDBG("%s %s %pI4:%u/%u <-> %pI4:%u/%u\n", + func, msg, SINA(&epc->local_addr), + SINP(&epc->local_addr), + SINP(&epc->mapped_local_addr), + SINA(&epc->remote_addr), + SINP(&epc->remote_addr), + SINP(&epc->mapped_remote_addr)); + break; + case AF_INET6: + PDBG("%s %s %pI6:%u/%u <-> %pI6:%u/%u\n", + func, msg, SIN6A(&epc->local_addr), + SIN6P(&epc->local_addr), + SIN6P(&epc->mapped_local_addr), + SIN6A(&epc->remote_addr), + SIN6P(&epc->remote_addr), + SIN6P(&epc->mapped_remote_addr)); + break; + default: + break; + } + } +#undef SINA +#undef SINP +#undef SIN6A +#undef SIN6P +} + static inline struct c4iw_ep *to_ep(struct iw_cm_id *cm_id) { return cm_id->provider_data; diff --git a/drivers/infiniband/hw/cxgb4/provider.c b/drivers/infiniband/hw/cxgb4/provider.c index a94a3e12c349..c777e22bd8d5 100644 --- a/drivers/infiniband/hw/cxgb4/provider.c +++ b/drivers/infiniband/hw/cxgb4/provider.c @@ -122,7 +122,7 @@ static struct ib_ucontext *c4iw_alloc_ucontext(struct ib_device *ibdev, INIT_LIST_HEAD(&context->mmaps); spin_lock_init(&context->mmap_lock); - if (udata->outlen < sizeof(uresp)) { + if (udata->outlen < sizeof(uresp) - sizeof(uresp.reserved)) { if (!warned++) pr_err(MOD "Warning - downlevel libcxgb4 (non-fatal), device status page disabled."); rhp->rdev.flags |= T4_STATUS_PAGE_DISABLED; @@ -140,7 +140,8 @@ static struct ib_ucontext *c4iw_alloc_ucontext(struct ib_device *ibdev, context->key += PAGE_SIZE; spin_unlock(&context->mmap_lock); - ret = ib_copy_to_udata(udata, &uresp, sizeof(uresp)); + ret = ib_copy_to_udata(udata, &uresp, + sizeof(uresp) - sizeof(uresp.reserved)); if (ret) goto err_mm; diff --git a/drivers/infiniband/hw/cxgb4/user.h b/drivers/infiniband/hw/cxgb4/user.h index 11ccd276e5d9..cbd0ce170728 100644 --- a/drivers/infiniband/hw/cxgb4/user.h +++ b/drivers/infiniband/hw/cxgb4/user.h @@ -48,6 +48,7 @@ struct c4iw_create_cq_resp { __u32 cqid; __u32 size; __u32 qid_mask; + __u32 reserved; /* explicit padding (optional for i386) */ }; @@ -74,5 +75,6 @@ struct c4iw_create_qp_resp { struct c4iw_alloc_ucontext_resp { __u64 status_page_key; __u32 status_page_size; + __u32 reserved; /* explicit padding (optional for i386) */ }; #endif diff --git a/drivers/infiniband/hw/ipath/ipath_diag.c b/drivers/infiniband/hw/ipath/ipath_diag.c index e2f9a51f4a38..45802e97332e 100644 --- a/drivers/infiniband/hw/ipath/ipath_diag.c +++ b/drivers/infiniband/hw/ipath/ipath_diag.c @@ -346,6 +346,10 @@ static ssize_t ipath_diagpkt_write(struct file *fp, ret = -EFAULT; goto bail; } + dp.len = odp.len; + dp.unit = odp.unit; + dp.data = odp.data; + dp.pbc_wd = 0; } else { ret = -EINVAL; goto bail; diff --git a/drivers/infiniband/hw/ipath/ipath_intr.c b/drivers/infiniband/hw/ipath/ipath_intr.c index 26dfbc8ee0f1..01ba792791a0 100644 --- a/drivers/infiniband/hw/ipath/ipath_intr.c +++ b/drivers/infiniband/hw/ipath/ipath_intr.c @@ -70,7 +70,7 @@ void ipath_disarm_senderrbufs(struct ipath_devdata *dd) if (sbuf[0] || sbuf[1] || (piobcnt > 128 && (sbuf[2] || sbuf[3]))) { int i; if (ipath_debug & (__IPATH_PKTDBG|__IPATH_DBG) && - dd->ipath_lastcancel > jiffies) { + time_after(dd->ipath_lastcancel, jiffies)) { __IPATH_DBG_WHICH(__IPATH_PKTDBG|__IPATH_DBG, "SendbufErrs %lx %lx", sbuf[0], sbuf[1]); @@ -755,7 +755,7 @@ static int handle_errors(struct ipath_devdata *dd, ipath_err_t errs) /* likely due to cancel; so suppress message unless verbose */ if ((errs & (INFINIPATH_E_SPKTLEN | INFINIPATH_E_SPIOARMLAUNCH)) && - dd->ipath_lastcancel > jiffies) { + time_after(dd->ipath_lastcancel, jiffies)) { /* armlaunch takes precedence; it often causes both. */ ipath_cdbg(VERBOSE, "Suppressed %s error (%llx) after sendbuf cancel\n", diff --git a/drivers/infiniband/hw/ipath/ipath_sdma.c b/drivers/infiniband/hw/ipath/ipath_sdma.c index 98ac18ec977e..17a517766ad2 100644 --- a/drivers/infiniband/hw/ipath/ipath_sdma.c +++ b/drivers/infiniband/hw/ipath/ipath_sdma.c @@ -247,7 +247,7 @@ static void sdma_abort_task(unsigned long opaque) /* ipath_sdma_abort() is done, waiting for interrupt */ if (status == IPATH_SDMA_ABORT_DISARMED) { - if (jiffies < dd->ipath_sdma_abort_intr_timeout) + if (time_before(jiffies, dd->ipath_sdma_abort_intr_timeout)) goto resched_noprint; /* give up, intr got lost somewhere */ ipath_dbg("give up waiting for SDMADISABLED intr\n"); @@ -341,7 +341,7 @@ resched: * JAG - this is bad to just have default be a loop without * state change */ - if (jiffies > dd->ipath_sdma_abort_jiffies) { + if (time_after(jiffies, dd->ipath_sdma_abort_jiffies)) { ipath_dbg("looping with status 0x%08lx\n", dd->ipath_sdma_status); dd->ipath_sdma_abort_jiffies = jiffies + 5 * HZ; diff --git a/drivers/infiniband/hw/mlx4/ah.c b/drivers/infiniband/hw/mlx4/ah.c index 170dca608042..2d8c3397774f 100644 --- a/drivers/infiniband/hw/mlx4/ah.c +++ b/drivers/infiniband/hw/mlx4/ah.c @@ -73,7 +73,7 @@ static struct ib_ah *create_iboe_ah(struct ib_pd *pd, struct ib_ah_attr *ah_attr { struct mlx4_ib_dev *ibdev = to_mdev(pd->device); struct mlx4_dev *dev = ibdev->dev; - int is_mcast; + int is_mcast = 0; struct in6_addr in6; u16 vlan_tag; diff --git a/drivers/infiniband/hw/mlx4/cq.c b/drivers/infiniband/hw/mlx4/cq.c index 5f640814cc81..1066eec854a9 100644 --- a/drivers/infiniband/hw/mlx4/cq.c +++ b/drivers/infiniband/hw/mlx4/cq.c @@ -102,7 +102,7 @@ static int mlx4_ib_alloc_cq_buf(struct mlx4_ib_dev *dev, struct mlx4_ib_cq_buf * int err; err = mlx4_buf_alloc(dev->dev, nent * dev->dev->caps.cqe_size, - PAGE_SIZE * 2, &buf->buf); + PAGE_SIZE * 2, &buf->buf, GFP_KERNEL); if (err) goto out; @@ -113,7 +113,7 @@ static int mlx4_ib_alloc_cq_buf(struct mlx4_ib_dev *dev, struct mlx4_ib_cq_buf * if (err) goto err_buf; - err = mlx4_buf_write_mtt(dev->dev, &buf->mtt, &buf->buf); + err = mlx4_buf_write_mtt(dev->dev, &buf->mtt, &buf->buf, GFP_KERNEL); if (err) goto err_mtt; @@ -209,7 +209,7 @@ struct ib_cq *mlx4_ib_create_cq(struct ib_device *ibdev, int entries, int vector uar = &to_mucontext(context)->uar; } else { - err = mlx4_db_alloc(dev->dev, &cq->db, 1); + err = mlx4_db_alloc(dev->dev, &cq->db, 1, GFP_KERNEL); if (err) goto err_cq; diff --git a/drivers/infiniband/hw/mlx4/mad.c b/drivers/infiniband/hw/mlx4/mad.c index fd36ec672632..287ad0564acd 100644 --- a/drivers/infiniband/hw/mlx4/mad.c +++ b/drivers/infiniband/hw/mlx4/mad.c @@ -478,10 +478,6 @@ int mlx4_ib_send_to_slave(struct mlx4_ib_dev *dev, int slave, u8 port, if (!tun_ctx || tun_ctx->state != DEMUX_PV_STATE_ACTIVE) return -EAGAIN; - /* QP0 forwarding only for Dom0 */ - if (!dest_qpt && (mlx4_master_func_num(dev->dev) != slave)) - return -EINVAL; - if (!dest_qpt) tun_qp = &tun_ctx->qp[0]; else @@ -667,6 +663,21 @@ static int mlx4_ib_demux_mad(struct ib_device *ibdev, u8 port, } /* Class-specific handling */ switch (mad->mad_hdr.mgmt_class) { + case IB_MGMT_CLASS_SUBN_LID_ROUTED: + case IB_MGMT_CLASS_SUBN_DIRECTED_ROUTE: + /* 255 indicates the dom0 */ + if (slave != 255 && slave != mlx4_master_func_num(dev->dev)) { + if (!mlx4_vf_smi_enabled(dev->dev, slave, port)) + return -EPERM; + /* for a VF. drop unsolicited MADs */ + if (!(mad->mad_hdr.method & IB_MGMT_METHOD_RESP)) { + mlx4_ib_warn(ibdev, "demux QP0. rejecting unsolicited mad for slave %d class 0x%x, method 0x%x\n", + slave, mad->mad_hdr.mgmt_class, + mad->mad_hdr.method); + return -EINVAL; + } + } + break; case IB_MGMT_CLASS_SUBN_ADM: if (mlx4_ib_demux_sa_handler(ibdev, port, slave, (struct ib_sa_mad *) mad)) @@ -1165,10 +1176,6 @@ int mlx4_ib_send_to_wire(struct mlx4_ib_dev *dev, int slave, u8 port, if (!sqp_ctx || sqp_ctx->state != DEMUX_PV_STATE_ACTIVE) return -EAGAIN; - /* QP0 forwarding only for Dom0 */ - if (dest_qpt == IB_QPT_SMI && (mlx4_master_func_num(dev->dev) != slave)) - return -EINVAL; - if (dest_qpt == IB_QPT_SMI) { src_qpnum = 0; sqp = &sqp_ctx->qp[0]; @@ -1285,11 +1292,6 @@ static void mlx4_ib_multiplex_mad(struct mlx4_ib_demux_pv_ctx *ctx, struct ib_wc "belongs to another slave\n", wc->src_qp); return; } - if (slave != mlx4_master_func_num(dev->dev) && !(wc->src_qp & 0x2)) { - mlx4_ib_warn(ctx->ib_dev, "can't multiplex bad sqp:%d: " - "non-master trying to send QP0 packets\n", wc->src_qp); - return; - } /* Map transaction ID */ ib_dma_sync_single_for_cpu(ctx->ib_dev, tun_qp->ring[wr_ix].map, @@ -1317,6 +1319,12 @@ static void mlx4_ib_multiplex_mad(struct mlx4_ib_demux_pv_ctx *ctx, struct ib_wc /* Class-specific handling */ switch (tunnel->mad.mad_hdr.mgmt_class) { + case IB_MGMT_CLASS_SUBN_LID_ROUTED: + case IB_MGMT_CLASS_SUBN_DIRECTED_ROUTE: + if (slave != mlx4_master_func_num(dev->dev) && + !mlx4_vf_smi_enabled(dev->dev, slave, ctx->port)) + return; + break; case IB_MGMT_CLASS_SUBN_ADM: if (mlx4_ib_multiplex_sa_handler(ctx->ib_dev, ctx->port, slave, (struct ib_sa_mad *) &tunnel->mad)) @@ -1749,9 +1757,9 @@ static int create_pv_resources(struct ib_device *ibdev, int slave, int port, return -EEXIST; ctx->state = DEMUX_PV_STATE_STARTING; - /* have QP0 only on port owner, and only if link layer is IB */ - if (ctx->slave == mlx4_master_func_num(to_mdev(ctx->ib_dev)->dev) && - rdma_port_get_link_layer(ibdev, ctx->port) == IB_LINK_LAYER_INFINIBAND) + /* have QP0 only if link layer is IB */ + if (rdma_port_get_link_layer(ibdev, ctx->port) == + IB_LINK_LAYER_INFINIBAND) ctx->has_smi = 1; if (ctx->has_smi) { diff --git a/drivers/infiniband/hw/mlx4/main.c b/drivers/infiniband/hw/mlx4/main.c index 199c7896f081..0f7027e7db13 100644 --- a/drivers/infiniband/hw/mlx4/main.c +++ b/drivers/infiniband/hw/mlx4/main.c @@ -545,12 +545,11 @@ static int mlx4_ib_modify_device(struct ib_device *ibdev, int mask, return 0; } -static int mlx4_SET_PORT(struct mlx4_ib_dev *dev, u8 port, int reset_qkey_viols, - u32 cap_mask) +static int mlx4_ib_SET_PORT(struct mlx4_ib_dev *dev, u8 port, int reset_qkey_viols, + u32 cap_mask) { struct mlx4_cmd_mailbox *mailbox; int err; - u8 is_eth = dev->dev->caps.port_type[port] == MLX4_PORT_TYPE_ETH; mailbox = mlx4_alloc_cmd_mailbox(dev->dev); if (IS_ERR(mailbox)) @@ -564,8 +563,8 @@ static int mlx4_SET_PORT(struct mlx4_ib_dev *dev, u8 port, int reset_qkey_viols, ((__be32 *) mailbox->buf)[1] = cpu_to_be32(cap_mask); } - err = mlx4_cmd(dev->dev, mailbox->dma, port, is_eth, MLX4_CMD_SET_PORT, - MLX4_CMD_TIME_CLASS_B, MLX4_CMD_NATIVE); + err = mlx4_cmd(dev->dev, mailbox->dma, port, 0, MLX4_CMD_SET_PORT, + MLX4_CMD_TIME_CLASS_B, MLX4_CMD_WRAPPED); mlx4_free_cmd_mailbox(dev->dev, mailbox); return err; @@ -574,11 +573,20 @@ static int mlx4_SET_PORT(struct mlx4_ib_dev *dev, u8 port, int reset_qkey_viols, static int mlx4_ib_modify_port(struct ib_device *ibdev, u8 port, int mask, struct ib_port_modify *props) { + struct mlx4_ib_dev *mdev = to_mdev(ibdev); + u8 is_eth = mdev->dev->caps.port_type[port] == MLX4_PORT_TYPE_ETH; struct ib_port_attr attr; u32 cap_mask; int err; - mutex_lock(&to_mdev(ibdev)->cap_mask_mutex); + /* return OK if this is RoCE. CM calls ib_modify_port() regardless + * of whether port link layer is ETH or IB. For ETH ports, qkey + * violations and port capabilities are not meaningful. + */ + if (is_eth) + return 0; + + mutex_lock(&mdev->cap_mask_mutex); err = mlx4_ib_query_port(ibdev, port, &attr); if (err) @@ -587,9 +595,9 @@ static int mlx4_ib_modify_port(struct ib_device *ibdev, u8 port, int mask, cap_mask = (attr.port_cap_flags | props->set_port_cap_mask) & ~props->clr_port_cap_mask; - err = mlx4_SET_PORT(to_mdev(ibdev), port, - !!(mask & IB_PORT_RESET_QKEY_CNTR), - cap_mask); + err = mlx4_ib_SET_PORT(mdev, port, + !!(mask & IB_PORT_RESET_QKEY_CNTR), + cap_mask); out: mutex_unlock(&to_mdev(ibdev)->cap_mask_mutex); diff --git a/drivers/infiniband/hw/mlx4/mlx4_ib.h b/drivers/infiniband/hw/mlx4/mlx4_ib.h index 66b0b7dbd9f4..369da3ca5d64 100644 --- a/drivers/infiniband/hw/mlx4/mlx4_ib.h +++ b/drivers/infiniband/hw/mlx4/mlx4_ib.h @@ -156,6 +156,7 @@ enum mlx4_ib_qp_flags { MLX4_IB_QP_LSO = IB_QP_CREATE_IPOIB_UD_LSO, MLX4_IB_QP_BLOCK_MULTICAST_LOOPBACK = IB_QP_CREATE_BLOCK_MULTICAST_LOOPBACK, MLX4_IB_QP_NETIF = IB_QP_CREATE_NETIF_QP, + MLX4_IB_QP_CREATE_USE_GFP_NOIO = IB_QP_CREATE_USE_GFP_NOIO, MLX4_IB_SRIOV_TUNNEL_QP = 1 << 30, MLX4_IB_SRIOV_SQP = 1 << 31, }; diff --git a/drivers/infiniband/hw/mlx4/qp.c b/drivers/infiniband/hw/mlx4/qp.c index dc57482ae7af..67780452f0cf 100644 --- a/drivers/infiniband/hw/mlx4/qp.c +++ b/drivers/infiniband/hw/mlx4/qp.c @@ -608,9 +608,20 @@ static int qp_has_rq(struct ib_qp_init_attr *attr) return !attr->srq; } +static int qp0_enabled_vf(struct mlx4_dev *dev, int qpn) +{ + int i; + for (i = 0; i < dev->caps.num_ports; i++) { + if (qpn == dev->caps.qp0_proxy[i]) + return !!dev->caps.qp0_qkey[i]; + } + return 0; +} + static int create_qp_common(struct mlx4_ib_dev *dev, struct ib_pd *pd, struct ib_qp_init_attr *init_attr, - struct ib_udata *udata, int sqpn, struct mlx4_ib_qp **caller_qp) + struct ib_udata *udata, int sqpn, struct mlx4_ib_qp **caller_qp, + gfp_t gfp) { int qpn; int err; @@ -625,10 +636,13 @@ static int create_qp_common(struct mlx4_ib_dev *dev, struct ib_pd *pd, !(init_attr->create_flags & MLX4_IB_SRIOV_SQP))) { if (init_attr->qp_type == IB_QPT_GSI) qp_type = MLX4_IB_QPT_PROXY_GSI; - else if (mlx4_is_master(dev->dev)) - qp_type = MLX4_IB_QPT_PROXY_SMI_OWNER; - else - qp_type = MLX4_IB_QPT_PROXY_SMI; + else { + if (mlx4_is_master(dev->dev) || + qp0_enabled_vf(dev->dev, sqpn)) + qp_type = MLX4_IB_QPT_PROXY_SMI_OWNER; + else + qp_type = MLX4_IB_QPT_PROXY_SMI; + } } qpn = sqpn; /* add extra sg entry for tunneling */ @@ -643,7 +657,9 @@ static int create_qp_common(struct mlx4_ib_dev *dev, struct ib_pd *pd, return -EINVAL; if (tnl_init->proxy_qp_type == IB_QPT_GSI) qp_type = MLX4_IB_QPT_TUN_GSI; - else if (tnl_init->slave == mlx4_master_func_num(dev->dev)) + else if (tnl_init->slave == mlx4_master_func_num(dev->dev) || + mlx4_vf_smi_enabled(dev->dev, tnl_init->slave, + tnl_init->port)) qp_type = MLX4_IB_QPT_TUN_SMI_OWNER; else qp_type = MLX4_IB_QPT_TUN_SMI; @@ -658,14 +674,14 @@ static int create_qp_common(struct mlx4_ib_dev *dev, struct ib_pd *pd, if (qp_type == MLX4_IB_QPT_SMI || qp_type == MLX4_IB_QPT_GSI || (qp_type & (MLX4_IB_QPT_PROXY_SMI | MLX4_IB_QPT_PROXY_SMI_OWNER | MLX4_IB_QPT_PROXY_GSI | MLX4_IB_QPT_TUN_SMI_OWNER))) { - sqp = kzalloc(sizeof (struct mlx4_ib_sqp), GFP_KERNEL); + sqp = kzalloc(sizeof (struct mlx4_ib_sqp), gfp); if (!sqp) return -ENOMEM; qp = &sqp->qp; qp->pri.vid = 0xFFFF; qp->alt.vid = 0xFFFF; } else { - qp = kzalloc(sizeof (struct mlx4_ib_qp), GFP_KERNEL); + qp = kzalloc(sizeof (struct mlx4_ib_qp), gfp); if (!qp) return -ENOMEM; qp->pri.vid = 0xFFFF; @@ -748,14 +764,14 @@ static int create_qp_common(struct mlx4_ib_dev *dev, struct ib_pd *pd, goto err; if (qp_has_rq(init_attr)) { - err = mlx4_db_alloc(dev->dev, &qp->db, 0); + err = mlx4_db_alloc(dev->dev, &qp->db, 0, gfp); if (err) goto err; *qp->db.db = 0; } - if (mlx4_buf_alloc(dev->dev, qp->buf_size, PAGE_SIZE * 2, &qp->buf)) { + if (mlx4_buf_alloc(dev->dev, qp->buf_size, PAGE_SIZE * 2, &qp->buf, gfp)) { err = -ENOMEM; goto err_db; } @@ -765,13 +781,12 @@ static int create_qp_common(struct mlx4_ib_dev *dev, struct ib_pd *pd, if (err) goto err_buf; - err = mlx4_buf_write_mtt(dev->dev, &qp->mtt, &qp->buf); + err = mlx4_buf_write_mtt(dev->dev, &qp->mtt, &qp->buf, gfp); if (err) goto err_mtt; - qp->sq.wrid = kmalloc(qp->sq.wqe_cnt * sizeof (u64), GFP_KERNEL); - qp->rq.wrid = kmalloc(qp->rq.wqe_cnt * sizeof (u64), GFP_KERNEL); - + qp->sq.wrid = kmalloc(qp->sq.wqe_cnt * sizeof (u64), gfp); + qp->rq.wrid = kmalloc(qp->rq.wqe_cnt * sizeof (u64), gfp); if (!qp->sq.wrid || !qp->rq.wrid) { err = -ENOMEM; goto err_wrid; @@ -801,7 +816,7 @@ static int create_qp_common(struct mlx4_ib_dev *dev, struct ib_pd *pd, goto err_proxy; } - err = mlx4_qp_alloc(dev->dev, qpn, &qp->mqp); + err = mlx4_qp_alloc(dev->dev, qpn, &qp->mqp, gfp); if (err) goto err_qpn; @@ -1040,7 +1055,10 @@ struct ib_qp *mlx4_ib_create_qp(struct ib_pd *pd, struct mlx4_ib_qp *qp = NULL; int err; u16 xrcdn = 0; + gfp_t gfp; + gfp = (init_attr->create_flags & MLX4_IB_QP_CREATE_USE_GFP_NOIO) ? + GFP_NOIO : GFP_KERNEL; /* * We only support LSO, vendor flag1, and multicast loopback blocking, * and only for kernel UD QPs. @@ -1049,7 +1067,8 @@ struct ib_qp *mlx4_ib_create_qp(struct ib_pd *pd, MLX4_IB_QP_BLOCK_MULTICAST_LOOPBACK | MLX4_IB_SRIOV_TUNNEL_QP | MLX4_IB_SRIOV_SQP | - MLX4_IB_QP_NETIF)) + MLX4_IB_QP_NETIF | + MLX4_IB_QP_CREATE_USE_GFP_NOIO)) return ERR_PTR(-EINVAL); if (init_attr->create_flags & IB_QP_CREATE_NETIF_QP) { @@ -1059,7 +1078,7 @@ struct ib_qp *mlx4_ib_create_qp(struct ib_pd *pd, if (init_attr->create_flags && (udata || - ((init_attr->create_flags & ~MLX4_IB_SRIOV_SQP) && + ((init_attr->create_flags & ~(MLX4_IB_SRIOV_SQP | MLX4_IB_QP_CREATE_USE_GFP_NOIO)) && init_attr->qp_type != IB_QPT_UD) || ((init_attr->create_flags & MLX4_IB_SRIOV_SQP) && init_attr->qp_type > IB_QPT_GSI))) @@ -1079,7 +1098,7 @@ struct ib_qp *mlx4_ib_create_qp(struct ib_pd *pd, case IB_QPT_RC: case IB_QPT_UC: case IB_QPT_RAW_PACKET: - qp = kzalloc(sizeof *qp, GFP_KERNEL); + qp = kzalloc(sizeof *qp, gfp); if (!qp) return ERR_PTR(-ENOMEM); qp->pri.vid = 0xFFFF; @@ -1088,7 +1107,7 @@ struct ib_qp *mlx4_ib_create_qp(struct ib_pd *pd, case IB_QPT_UD: { err = create_qp_common(to_mdev(pd->device), pd, init_attr, - udata, 0, &qp); + udata, 0, &qp, gfp); if (err) return ERR_PTR(err); @@ -1106,7 +1125,7 @@ struct ib_qp *mlx4_ib_create_qp(struct ib_pd *pd, err = create_qp_common(to_mdev(pd->device), pd, init_attr, udata, get_sqp_num(to_mdev(pd->device), init_attr), - &qp); + &qp, gfp); if (err) return ERR_PTR(err); @@ -1938,6 +1957,19 @@ out: return err; } +static int vf_get_qp0_qkey(struct mlx4_dev *dev, int qpn, u32 *qkey) +{ + int i; + for (i = 0; i < dev->caps.num_ports; i++) { + if (qpn == dev->caps.qp0_proxy[i] || + qpn == dev->caps.qp0_tunnel[i]) { + *qkey = dev->caps.qp0_qkey[i]; + return 0; + } + } + return -EINVAL; +} + static int build_sriov_qp0_header(struct mlx4_ib_sqp *sqp, struct ib_send_wr *wr, void *wqe, unsigned *mlx_seg_len) @@ -1995,8 +2027,13 @@ static int build_sriov_qp0_header(struct mlx4_ib_sqp *sqp, cpu_to_be32(mdev->dev->caps.qp0_tunnel[sqp->qp.port - 1]); sqp->ud_header.bth.psn = cpu_to_be32((sqp->send_psn++) & ((1 << 24) - 1)); - if (mlx4_get_parav_qkey(mdev->dev, sqp->qp.mqp.qpn, &qkey)) - return -EINVAL; + if (mlx4_is_master(mdev->dev)) { + if (mlx4_get_parav_qkey(mdev->dev, sqp->qp.mqp.qpn, &qkey)) + return -EINVAL; + } else { + if (vf_get_qp0_qkey(mdev->dev, sqp->qp.mqp.qpn, &qkey)) + return -EINVAL; + } sqp->ud_header.deth.qkey = cpu_to_be32(qkey); sqp->ud_header.deth.source_qpn = cpu_to_be32(sqp->qp.mqp.qpn); @@ -2378,7 +2415,8 @@ static void set_datagram_seg(struct mlx4_wqe_datagram_seg *dseg, static void set_tunnel_datagram_seg(struct mlx4_ib_dev *dev, struct mlx4_wqe_datagram_seg *dseg, - struct ib_send_wr *wr, enum ib_qp_type qpt) + struct ib_send_wr *wr, + enum mlx4_ib_qp_type qpt) { union mlx4_ext_av *av = &to_mah(wr->wr.ud.ah)->av; struct mlx4_av sqp_av = {0}; @@ -2391,8 +2429,10 @@ static void set_tunnel_datagram_seg(struct mlx4_ib_dev *dev, cpu_to_be32(0xf0000000); memcpy(dseg->av, &sqp_av, sizeof (struct mlx4_av)); - /* This function used only for sending on QP1 proxies */ - dseg->dqpn = cpu_to_be32(dev->dev->caps.qp1_tunnel[port - 1]); + if (qpt == MLX4_IB_QPT_PROXY_GSI) + dseg->dqpn = cpu_to_be32(dev->dev->caps.qp1_tunnel[port - 1]); + else + dseg->dqpn = cpu_to_be32(dev->dev->caps.qp0_tunnel[port - 1]); /* Use QKEY from the QP context, which is set by master */ dseg->qkey = cpu_to_be32(IB_QP_SET_QKEY); } @@ -2687,11 +2727,6 @@ int mlx4_ib_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr, break; case MLX4_IB_QPT_PROXY_SMI_OWNER: - if (unlikely(!mlx4_is_master(to_mdev(ibqp->device)->dev))) { - err = -ENOSYS; - *bad_wr = wr; - goto out; - } err = build_sriov_qp0_header(to_msqp(qp), wr, ctrl, &seglen); if (unlikely(err)) { *bad_wr = wr; @@ -2708,16 +2743,13 @@ int mlx4_ib_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr, size += seglen / 16; break; case MLX4_IB_QPT_PROXY_SMI: - /* don't allow QP0 sends on guests */ - err = -ENOSYS; - *bad_wr = wr; - goto out; case MLX4_IB_QPT_PROXY_GSI: /* If we are tunneling special qps, this is a UD qp. * In this case we first add a UD segment targeting * the tunnel qp, and then add a header with address * information */ - set_tunnel_datagram_seg(to_mdev(ibqp->device), wqe, wr, ibqp->qp_type); + set_tunnel_datagram_seg(to_mdev(ibqp->device), wqe, wr, + qp->mlx4_ib_qp_type); wqe += sizeof (struct mlx4_wqe_datagram_seg); size += sizeof (struct mlx4_wqe_datagram_seg) / 16; build_tunnel_header(wr, wqe, &seglen); diff --git a/drivers/infiniband/hw/mlx4/srq.c b/drivers/infiniband/hw/mlx4/srq.c index 60c5fb025fc7..62d9285300af 100644 --- a/drivers/infiniband/hw/mlx4/srq.c +++ b/drivers/infiniband/hw/mlx4/srq.c @@ -134,13 +134,14 @@ struct ib_srq *mlx4_ib_create_srq(struct ib_pd *pd, if (err) goto err_mtt; } else { - err = mlx4_db_alloc(dev->dev, &srq->db, 0); + err = mlx4_db_alloc(dev->dev, &srq->db, 0, GFP_KERNEL); if (err) goto err_srq; *srq->db.db = 0; - if (mlx4_buf_alloc(dev->dev, buf_size, PAGE_SIZE * 2, &srq->buf)) { + if (mlx4_buf_alloc(dev->dev, buf_size, PAGE_SIZE * 2, &srq->buf, + GFP_KERNEL)) { err = -ENOMEM; goto err_db; } @@ -165,7 +166,7 @@ struct ib_srq *mlx4_ib_create_srq(struct ib_pd *pd, if (err) goto err_buf; - err = mlx4_buf_write_mtt(dev->dev, &srq->mtt, &srq->buf); + err = mlx4_buf_write_mtt(dev->dev, &srq->mtt, &srq->buf, GFP_KERNEL); if (err) goto err_mtt; diff --git a/drivers/infiniband/hw/mlx4/sysfs.c b/drivers/infiniband/hw/mlx4/sysfs.c index 5a38e43eca65..cb4c66e723b5 100644 --- a/drivers/infiniband/hw/mlx4/sysfs.c +++ b/drivers/infiniband/hw/mlx4/sysfs.c @@ -389,8 +389,10 @@ struct mlx4_port { struct mlx4_ib_dev *dev; struct attribute_group pkey_group; struct attribute_group gid_group; - u8 port_num; + struct device_attribute enable_smi_admin; + struct device_attribute smi_enabled; int slave; + u8 port_num; }; @@ -558,6 +560,101 @@ err: return NULL; } +static ssize_t sysfs_show_smi_enabled(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct mlx4_port *p = + container_of(attr, struct mlx4_port, smi_enabled); + ssize_t len = 0; + + if (mlx4_vf_smi_enabled(p->dev->dev, p->slave, p->port_num)) + len = sprintf(buf, "%d\n", 1); + else + len = sprintf(buf, "%d\n", 0); + + return len; +} + +static ssize_t sysfs_show_enable_smi_admin(struct device *dev, + struct device_attribute *attr, + char *buf) +{ + struct mlx4_port *p = + container_of(attr, struct mlx4_port, enable_smi_admin); + ssize_t len = 0; + + if (mlx4_vf_get_enable_smi_admin(p->dev->dev, p->slave, p->port_num)) + len = sprintf(buf, "%d\n", 1); + else + len = sprintf(buf, "%d\n", 0); + + return len; +} + +static ssize_t sysfs_store_enable_smi_admin(struct device *dev, + struct device_attribute *attr, + const char *buf, size_t count) +{ + struct mlx4_port *p = + container_of(attr, struct mlx4_port, enable_smi_admin); + int enable; + + if (sscanf(buf, "%i", &enable) != 1 || + enable < 0 || enable > 1) + return -EINVAL; + + if (mlx4_vf_set_enable_smi_admin(p->dev->dev, p->slave, p->port_num, enable)) + return -EINVAL; + return count; +} + +static int add_vf_smi_entries(struct mlx4_port *p) +{ + int is_eth = rdma_port_get_link_layer(&p->dev->ib_dev, p->port_num) == + IB_LINK_LAYER_ETHERNET; + int ret; + + /* do not display entries if eth transport, or if master */ + if (is_eth || p->slave == mlx4_master_func_num(p->dev->dev)) + return 0; + + sysfs_attr_init(&p->smi_enabled.attr); + p->smi_enabled.show = sysfs_show_smi_enabled; + p->smi_enabled.store = NULL; + p->smi_enabled.attr.name = "smi_enabled"; + p->smi_enabled.attr.mode = 0444; + ret = sysfs_create_file(&p->kobj, &p->smi_enabled.attr); + if (ret) { + pr_err("failed to create smi_enabled\n"); + return ret; + } + + sysfs_attr_init(&p->enable_smi_admin.attr); + p->enable_smi_admin.show = sysfs_show_enable_smi_admin; + p->enable_smi_admin.store = sysfs_store_enable_smi_admin; + p->enable_smi_admin.attr.name = "enable_smi_admin"; + p->enable_smi_admin.attr.mode = 0644; + ret = sysfs_create_file(&p->kobj, &p->enable_smi_admin.attr); + if (ret) { + pr_err("failed to create enable_smi_admin\n"); + sysfs_remove_file(&p->kobj, &p->smi_enabled.attr); + return ret; + } + return 0; +} + +static void remove_vf_smi_entries(struct mlx4_port *p) +{ + int is_eth = rdma_port_get_link_layer(&p->dev->ib_dev, p->port_num) == + IB_LINK_LAYER_ETHERNET; + + if (is_eth || p->slave == mlx4_master_func_num(p->dev->dev)) + return; + + sysfs_remove_file(&p->kobj, &p->smi_enabled.attr); + sysfs_remove_file(&p->kobj, &p->enable_smi_admin.attr); +} + static int add_port(struct mlx4_ib_dev *dev, int port_num, int slave) { struct mlx4_port *p; @@ -602,6 +699,10 @@ static int add_port(struct mlx4_ib_dev *dev, int port_num, int slave) if (ret) goto err_free_gid; + ret = add_vf_smi_entries(p); + if (ret) + goto err_free_gid; + list_add_tail(&p->kobj.entry, &dev->pkeys.pkey_port_list[slave]); return 0; @@ -669,6 +770,7 @@ err_add: mport = container_of(p, struct mlx4_port, kobj); sysfs_remove_group(p, &mport->pkey_group); sysfs_remove_group(p, &mport->gid_group); + remove_vf_smi_entries(mport); kobject_put(p); } kobject_put(dev->dev_ports_parent[slave]); @@ -713,6 +815,7 @@ static void unregister_pkey_tree(struct mlx4_ib_dev *device) port = container_of(p, struct mlx4_port, kobj); sysfs_remove_group(p, &port->pkey_group); sysfs_remove_group(p, &port->gid_group); + remove_vf_smi_entries(port); kobject_put(p); kobject_put(device->dev_ports_parent[slave]); } diff --git a/drivers/infiniband/hw/mlx5/cq.c b/drivers/infiniband/hw/mlx5/cq.c index 62bb6b49dc1d..8ae4f896cb41 100644 --- a/drivers/infiniband/hw/mlx5/cq.c +++ b/drivers/infiniband/hw/mlx5/cq.c @@ -32,6 +32,7 @@ #include <linux/kref.h> #include <rdma/ib_umem.h> +#include <rdma/ib_user_verbs.h> #include "mlx5_ib.h" #include "user.h" @@ -602,14 +603,24 @@ static int create_cq_user(struct mlx5_ib_dev *dev, struct ib_udata *udata, int *cqe_size, int *index, int *inlen) { struct mlx5_ib_create_cq ucmd; + size_t ucmdlen; int page_shift; int npages; int ncont; int err; - if (ib_copy_from_udata(&ucmd, udata, sizeof(ucmd))) + ucmdlen = + (udata->inlen - sizeof(struct ib_uverbs_cmd_hdr) < + sizeof(ucmd)) ? (sizeof(ucmd) - + sizeof(ucmd.reserved)) : sizeof(ucmd); + + if (ib_copy_from_udata(&ucmd, udata, ucmdlen)) return -EFAULT; + if (ucmdlen == sizeof(ucmd) && + ucmd.reserved != 0) + return -EINVAL; + if (ucmd.cqe_size != 64 && ucmd.cqe_size != 128) return -EINVAL; diff --git a/drivers/infiniband/hw/mlx5/mlx5_ib.h b/drivers/infiniband/hw/mlx5/mlx5_ib.h index 50541586e0a6..f2ccf1a5a291 100644 --- a/drivers/infiniband/hw/mlx5/mlx5_ib.h +++ b/drivers/infiniband/hw/mlx5/mlx5_ib.h @@ -264,8 +264,6 @@ struct mlx5_ib_mr { __be64 *pas; dma_addr_t dma; int npages; - struct completion done; - enum ib_wc_status status; struct mlx5_ib_dev *dev; struct mlx5_create_mkey_mbox_out out; struct mlx5_core_sig_ctx *sig; @@ -277,6 +275,17 @@ struct mlx5_ib_fast_reg_page_list { dma_addr_t map; }; +struct mlx5_ib_umr_context { + enum ib_wc_status status; + struct completion done; +}; + +static inline void mlx5_ib_init_umr_context(struct mlx5_ib_umr_context *context) +{ + context->status = -1; + init_completion(&context->done); +} + struct umr_common { struct ib_pd *pd; struct ib_cq *cq; diff --git a/drivers/infiniband/hw/mlx5/mr.c b/drivers/infiniband/hw/mlx5/mr.c index 81392b26d078..afa873bd028e 100644 --- a/drivers/infiniband/hw/mlx5/mr.c +++ b/drivers/infiniband/hw/mlx5/mr.c @@ -73,6 +73,8 @@ static void reg_mr_callback(int status, void *context) struct mlx5_cache_ent *ent = &cache->ent[c]; u8 key; unsigned long flags; + struct mlx5_mr_table *table = &dev->mdev.priv.mr_table; + int err; spin_lock_irqsave(&ent->lock, flags); ent->pending--; @@ -107,6 +109,13 @@ static void reg_mr_callback(int status, void *context) ent->cur++; ent->size++; spin_unlock_irqrestore(&ent->lock, flags); + + write_lock_irqsave(&table->lock, flags); + err = radix_tree_insert(&table->tree, mlx5_base_mkey(mr->mmr.key), + &mr->mmr); + if (err) + pr_err("Error inserting to mr tree. 0x%x\n", -err); + write_unlock_irqrestore(&table->lock, flags); } static int add_keys(struct mlx5_ib_dev *dev, int c, int num) @@ -699,7 +708,7 @@ static void prep_umr_unreg_wqe(struct mlx5_ib_dev *dev, void mlx5_umr_cq_handler(struct ib_cq *cq, void *cq_context) { - struct mlx5_ib_mr *mr; + struct mlx5_ib_umr_context *context; struct ib_wc wc; int err; @@ -712,9 +721,9 @@ void mlx5_umr_cq_handler(struct ib_cq *cq, void *cq_context) if (err == 0) break; - mr = (struct mlx5_ib_mr *)(unsigned long)wc.wr_id; - mr->status = wc.status; - complete(&mr->done); + context = (struct mlx5_ib_umr_context *) (unsigned long) wc.wr_id; + context->status = wc.status; + complete(&context->done); } ib_req_notify_cq(cq, IB_CQ_NEXT_COMP); } @@ -726,11 +735,12 @@ static struct mlx5_ib_mr *reg_umr(struct ib_pd *pd, struct ib_umem *umem, struct mlx5_ib_dev *dev = to_mdev(pd->device); struct device *ddev = dev->ib_dev.dma_device; struct umr_common *umrc = &dev->umrc; + struct mlx5_ib_umr_context umr_context; struct ib_send_wr wr, *bad; struct mlx5_ib_mr *mr; struct ib_sge sg; int size = sizeof(u64) * npages; - int err; + int err = 0; int i; for (i = 0; i < 1; i++) { @@ -751,7 +761,7 @@ static struct mlx5_ib_mr *reg_umr(struct ib_pd *pd, struct ib_umem *umem, mr->pas = kmalloc(size + MLX5_UMR_ALIGN - 1, GFP_KERNEL); if (!mr->pas) { err = -ENOMEM; - goto error; + goto free_mr; } mlx5_ib_populate_pas(dev, umem, page_shift, @@ -760,44 +770,46 @@ static struct mlx5_ib_mr *reg_umr(struct ib_pd *pd, struct ib_umem *umem, mr->dma = dma_map_single(ddev, mr_align(mr->pas, MLX5_UMR_ALIGN), size, DMA_TO_DEVICE); if (dma_mapping_error(ddev, mr->dma)) { - kfree(mr->pas); err = -ENOMEM; - goto error; + goto free_pas; } memset(&wr, 0, sizeof(wr)); - wr.wr_id = (u64)(unsigned long)mr; + wr.wr_id = (u64)(unsigned long)&umr_context; prep_umr_reg_wqe(pd, &wr, &sg, mr->dma, npages, mr->mmr.key, page_shift, virt_addr, len, access_flags); - /* We serialize polls so one process does not kidnap another's - * completion. This is not a problem since wr is completed in - * around 1 usec - */ + mlx5_ib_init_umr_context(&umr_context); down(&umrc->sem); - init_completion(&mr->done); err = ib_post_send(umrc->qp, &wr, &bad); if (err) { mlx5_ib_warn(dev, "post send failed, err %d\n", err); - up(&umrc->sem); - goto error; + goto unmap_dma; + } else { + wait_for_completion(&umr_context.done); + if (umr_context.status != IB_WC_SUCCESS) { + mlx5_ib_warn(dev, "reg umr failed\n"); + err = -EFAULT; + } } - wait_for_completion(&mr->done); - up(&umrc->sem); + mr->mmr.iova = virt_addr; + mr->mmr.size = len; + mr->mmr.pd = to_mpd(pd)->pdn; + +unmap_dma: + up(&umrc->sem); dma_unmap_single(ddev, mr->dma, size, DMA_TO_DEVICE); + +free_pas: kfree(mr->pas); - if (mr->status != IB_WC_SUCCESS) { - mlx5_ib_warn(dev, "reg umr failed\n"); - err = -EFAULT; - goto error; +free_mr: + if (err) { + free_cached_mr(dev, mr); + return ERR_PTR(err); } return mr; - -error: - free_cached_mr(dev, mr); - return ERR_PTR(err); } static struct mlx5_ib_mr *reg_create(struct ib_pd *pd, u64 virt_addr, @@ -926,24 +938,26 @@ error: static int unreg_umr(struct mlx5_ib_dev *dev, struct mlx5_ib_mr *mr) { struct umr_common *umrc = &dev->umrc; + struct mlx5_ib_umr_context umr_context; struct ib_send_wr wr, *bad; int err; memset(&wr, 0, sizeof(wr)); - wr.wr_id = (u64)(unsigned long)mr; + wr.wr_id = (u64)(unsigned long)&umr_context; prep_umr_unreg_wqe(dev, &wr, mr->mmr.key); + mlx5_ib_init_umr_context(&umr_context); down(&umrc->sem); - init_completion(&mr->done); err = ib_post_send(umrc->qp, &wr, &bad); if (err) { up(&umrc->sem); mlx5_ib_dbg(dev, "err %d\n", err); goto error; + } else { + wait_for_completion(&umr_context.done); + up(&umrc->sem); } - wait_for_completion(&mr->done); - up(&umrc->sem); - if (mr->status != IB_WC_SUCCESS) { + if (umr_context.status != IB_WC_SUCCESS) { mlx5_ib_warn(dev, "unreg umr failed\n"); err = -EFAULT; goto error; diff --git a/drivers/infiniband/hw/mlx5/qp.c b/drivers/infiniband/hw/mlx5/qp.c index dc930ed21eca..d13ddf1c0033 100644 --- a/drivers/infiniband/hw/mlx5/qp.c +++ b/drivers/infiniband/hw/mlx5/qp.c @@ -574,6 +574,10 @@ static int create_user_qp(struct mlx5_ib_dev *dev, struct ib_pd *pd, uar_index = uuarn_to_uar_index(&context->uuari, uuarn); mlx5_ib_dbg(dev, "uuarn 0x%x, uar_index 0x%x\n", uuarn, uar_index); + qp->rq.offset = 0; + qp->sq.wqe_shift = ilog2(MLX5_SEND_WQE_BB); + qp->sq.offset = qp->rq.wqe_cnt << qp->rq.wqe_shift; + err = set_user_buf_size(dev, qp, &ucmd); if (err) goto err_uuar; @@ -2078,6 +2082,7 @@ static int mlx5_set_bsf(struct ib_mr *sig_mr, struct ib_sig_domain *wire = &sig_attrs->wire; int ret, selector; + memset(bsf, 0, sizeof(*bsf)); switch (sig_attrs->mem.sig_type) { case IB_SIG_TYPE_T10_DIF: if (sig_attrs->wire.sig_type != IB_SIG_TYPE_T10_DIF) @@ -2090,9 +2095,11 @@ static int mlx5_set_bsf(struct ib_mr *sig_mr, /* Same block structure */ basic->bsf_size_sbs = 1 << 4; if (mem->sig.dif.bg_type == wire->sig.dif.bg_type) - basic->wire.copy_byte_mask = 0xff; - else - basic->wire.copy_byte_mask = 0x3f; + basic->wire.copy_byte_mask |= 0xc0; + if (mem->sig.dif.app_tag == wire->sig.dif.app_tag) + basic->wire.copy_byte_mask |= 0x30; + if (mem->sig.dif.ref_tag == wire->sig.dif.ref_tag) + basic->wire.copy_byte_mask |= 0x0f; } else basic->wire.bs_selector = bs_selector(wire->sig.dif.pi_interval); @@ -2131,9 +2138,13 @@ static int set_sig_data_segment(struct ib_send_wr *wr, struct mlx5_ib_qp *qp, int ret; int wqe_size; - if (!wr->wr.sig_handover.prot) { + if (!wr->wr.sig_handover.prot || + (data_key == wr->wr.sig_handover.prot->lkey && + data_va == wr->wr.sig_handover.prot->addr && + data_len == wr->wr.sig_handover.prot->length)) { /** * Source domain doesn't contain signature information + * or data and protection are interleaved in memory. * So need construct: * ------------------ * | data_klm | @@ -2187,23 +2198,13 @@ static int set_sig_data_segment(struct ib_send_wr *wr, struct mlx5_ib_qp *qp, data_sentry->bcount = cpu_to_be16(block_size); data_sentry->key = cpu_to_be32(data_key); data_sentry->va = cpu_to_be64(data_va); + data_sentry->stride = cpu_to_be16(block_size); + prot_sentry->bcount = cpu_to_be16(prot_size); prot_sentry->key = cpu_to_be32(prot_key); + prot_sentry->va = cpu_to_be64(prot_va); + prot_sentry->stride = cpu_to_be16(prot_size); - if (prot_key == data_key && prot_va == data_va) { - /** - * The data and protection are interleaved - * in a single memory region - **/ - prot_sentry->va = cpu_to_be64(data_va + block_size); - prot_sentry->stride = cpu_to_be16(block_size + prot_size); - data_sentry->stride = prot_sentry->stride; - } else { - /* The data and protection are two different buffers */ - prot_sentry->va = cpu_to_be64(prot_va); - data_sentry->stride = cpu_to_be16(block_size); - prot_sentry->stride = cpu_to_be16(prot_size); - } wqe_size = ALIGN(sizeof(*sblock_ctrl) + sizeof(*data_sentry) + sizeof(*prot_sentry), 64); } @@ -2275,7 +2276,10 @@ static int set_sig_umr_wr(struct ib_send_wr *wr, struct mlx5_ib_qp *qp, /* length of the protected region, data + protection */ region_len = wr->sg_list->length; - if (wr->wr.sig_handover.prot) + if (wr->wr.sig_handover.prot && + (wr->wr.sig_handover.prot->lkey != wr->sg_list->lkey || + wr->wr.sig_handover.prot->addr != wr->sg_list->addr || + wr->wr.sig_handover.prot->length != wr->sg_list->length)) region_len += wr->wr.sig_handover.prot->length; /** diff --git a/drivers/infiniband/hw/mlx5/srq.c b/drivers/infiniband/hw/mlx5/srq.c index 210b3eaf188a..384af6dec5eb 100644 --- a/drivers/infiniband/hw/mlx5/srq.c +++ b/drivers/infiniband/hw/mlx5/srq.c @@ -35,6 +35,7 @@ #include <linux/mlx5/srq.h> #include <linux/slab.h> #include <rdma/ib_umem.h> +#include <rdma/ib_user_verbs.h> #include "mlx5_ib.h" #include "user.h" @@ -78,16 +79,27 @@ static int create_srq_user(struct ib_pd *pd, struct mlx5_ib_srq *srq, { struct mlx5_ib_dev *dev = to_mdev(pd->device); struct mlx5_ib_create_srq ucmd; + size_t ucmdlen; int err; int npages; int page_shift; int ncont; u32 offset; - if (ib_copy_from_udata(&ucmd, udata, sizeof(ucmd))) { + ucmdlen = + (udata->inlen - sizeof(struct ib_uverbs_cmd_hdr) < + sizeof(ucmd)) ? (sizeof(ucmd) - + sizeof(ucmd.reserved)) : sizeof(ucmd); + + if (ib_copy_from_udata(&ucmd, udata, ucmdlen)) { mlx5_ib_dbg(dev, "failed copy udata\n"); return -EFAULT; } + + if (ucmdlen == sizeof(ucmd) && + ucmd.reserved != 0) + return -EINVAL; + srq->wq_sig = !!(ucmd.flags & MLX5_SRQ_FLAG_SIGNATURE); srq->umem = ib_umem_get(pd->uobject->context, ucmd.buf_addr, buf_size, diff --git a/drivers/infiniband/hw/mlx5/user.h b/drivers/infiniband/hw/mlx5/user.h index 0f4f8e42a17f..d0ba264ac1ed 100644 --- a/drivers/infiniband/hw/mlx5/user.h +++ b/drivers/infiniband/hw/mlx5/user.h @@ -91,6 +91,7 @@ struct mlx5_ib_create_cq { __u64 buf_addr; __u64 db_addr; __u32 cqe_size; + __u32 reserved; /* explicit padding (optional on i386) */ }; struct mlx5_ib_create_cq_resp { @@ -109,6 +110,7 @@ struct mlx5_ib_create_srq { __u64 buf_addr; __u64 db_addr; __u32 flags; + __u32 reserved; /* explicit padding (optional on i386) */ }; struct mlx5_ib_create_srq_resp { diff --git a/drivers/infiniband/hw/nes/nes.c b/drivers/infiniband/hw/nes/nes.c index 353c7b05a90a..3b2a6dc8ea99 100644 --- a/drivers/infiniband/hw/nes/nes.c +++ b/drivers/infiniband/hw/nes/nes.c @@ -68,7 +68,6 @@ MODULE_VERSION(DRV_VERSION); int max_mtu = 9000; int interrupt_mod_interval = 0; - /* Interoperability */ int mpa_version = 1; module_param(mpa_version, int, 0644); @@ -112,6 +111,16 @@ static struct pci_device_id nes_pci_table[] = { MODULE_DEVICE_TABLE(pci, nes_pci_table); +/* registered nes netlink callbacks */ +static struct ibnl_client_cbs nes_nl_cb_table[] = { + [RDMA_NL_IWPM_REG_PID] = {.dump = iwpm_register_pid_cb}, + [RDMA_NL_IWPM_ADD_MAPPING] = {.dump = iwpm_add_mapping_cb}, + [RDMA_NL_IWPM_QUERY_MAPPING] = {.dump = iwpm_add_and_query_mapping_cb}, + [RDMA_NL_IWPM_HANDLE_ERR] = {.dump = iwpm_mapping_error_cb}, + [RDMA_NL_IWPM_MAPINFO] = {.dump = iwpm_mapping_info_cb}, + [RDMA_NL_IWPM_MAPINFO_NUM] = {.dump = iwpm_ack_mapping_info_cb} +}; + static int nes_inetaddr_event(struct notifier_block *, unsigned long, void *); static int nes_net_event(struct notifier_block *, unsigned long, void *); static int nes_notifiers_registered; @@ -672,6 +681,17 @@ static int nes_probe(struct pci_dev *pcidev, const struct pci_device_id *ent) } nes_notifiers_registered++; + if (ibnl_add_client(RDMA_NL_NES, RDMA_NL_IWPM_NUM_OPS, nes_nl_cb_table)) + printk(KERN_ERR PFX "%s[%u]: Failed to add netlink callback\n", + __func__, __LINE__); + + ret = iwpm_init(RDMA_NL_NES); + if (ret) { + printk(KERN_ERR PFX "%s: port mapper initialization failed\n", + pci_name(pcidev)); + goto bail7; + } + INIT_DELAYED_WORK(&nesdev->work, nes_recheck_link_status); /* Initialize network devices */ @@ -710,6 +730,7 @@ static int nes_probe(struct pci_dev *pcidev, const struct pci_device_id *ent) nes_debug(NES_DBG_INIT, "netdev_count=%d, nesadapter->netdev_count=%d\n", nesdev->netdev_count, nesdev->nesadapter->netdev_count); + ibnl_remove_client(RDMA_NL_NES); nes_notifiers_registered--; if (nes_notifiers_registered == 0) { @@ -773,6 +794,8 @@ static void nes_remove(struct pci_dev *pcidev) nesdev->nesadapter->netdev_count--; } } + ibnl_remove_client(RDMA_NL_NES); + iwpm_exit(RDMA_NL_NES); nes_notifiers_registered--; if (nes_notifiers_registered == 0) { diff --git a/drivers/infiniband/hw/nes/nes.h b/drivers/infiniband/hw/nes/nes.h index 33cc58941a3e..bd9d132f11c7 100644 --- a/drivers/infiniband/hw/nes/nes.h +++ b/drivers/infiniband/hw/nes/nes.h @@ -51,6 +51,8 @@ #include <rdma/ib_pack.h> #include <rdma/rdma_cm.h> #include <rdma/iw_cm.h> +#include <rdma/rdma_netlink.h> +#include <rdma/iw_portmap.h> #define NES_SEND_FIRST_WRITE @@ -130,6 +132,7 @@ #define NES_DBG_IW_TX 0x00040000 #define NES_DBG_SHUTDOWN 0x00080000 #define NES_DBG_PAU 0x00100000 +#define NES_DBG_NLMSG 0x00200000 #define NES_DBG_RSVD1 0x10000000 #define NES_DBG_RSVD2 0x20000000 #define NES_DBG_RSVD3 0x40000000 diff --git a/drivers/infiniband/hw/nes/nes_cm.c b/drivers/infiniband/hw/nes/nes_cm.c index dfa9df484505..6f09a72e78d7 100644 --- a/drivers/infiniband/hw/nes/nes_cm.c +++ b/drivers/infiniband/hw/nes/nes_cm.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2006 - 2011 Intel Corporation. All rights reserved. + * Copyright (c) 2006 - 2014 Intel Corporation. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -59,6 +59,7 @@ #include <net/route.h> #include <net/ip_fib.h> #include <net/tcp.h> +#include <linux/fcntl.h> #include "nes.h" @@ -166,7 +167,6 @@ int nes_rem_ref_cm_node(struct nes_cm_node *cm_node) { return rem_ref_cm_node(cm_node->cm_core, cm_node); } - /** * create_event */ @@ -482,11 +482,11 @@ static void form_cm_frame(struct sk_buff *skb, iph->ttl = 0x40; iph->protocol = 0x06; /* IPPROTO_TCP */ - iph->saddr = htonl(cm_node->loc_addr); - iph->daddr = htonl(cm_node->rem_addr); + iph->saddr = htonl(cm_node->mapped_loc_addr); + iph->daddr = htonl(cm_node->mapped_rem_addr); - tcph->source = htons(cm_node->loc_port); - tcph->dest = htons(cm_node->rem_port); + tcph->source = htons(cm_node->mapped_loc_port); + tcph->dest = htons(cm_node->mapped_rem_port); tcph->seq = htonl(cm_node->tcp_cntxt.loc_seq_num); if (flags & SET_ACK) { @@ -525,6 +525,100 @@ static void form_cm_frame(struct sk_buff *skb, cm_packets_created++; } +/* + * nes_create_sockaddr - Record ip addr and tcp port in a sockaddr struct + */ +static void nes_create_sockaddr(__be32 ip_addr, __be16 port, + struct sockaddr_storage *addr) +{ + struct sockaddr_in *nes_sockaddr = (struct sockaddr_in *)addr; + nes_sockaddr->sin_family = AF_INET; + memcpy(&nes_sockaddr->sin_addr.s_addr, &ip_addr, sizeof(__be32)); + nes_sockaddr->sin_port = port; +} + +/* + * nes_create_mapinfo - Create a mapinfo object in the port mapper data base + */ +static int nes_create_mapinfo(struct nes_cm_info *cm_info) +{ + struct sockaddr_storage local_sockaddr; + struct sockaddr_storage mapped_sockaddr; + + nes_create_sockaddr(htonl(cm_info->loc_addr), htons(cm_info->loc_port), + &local_sockaddr); + nes_create_sockaddr(htonl(cm_info->mapped_loc_addr), + htons(cm_info->mapped_loc_port), &mapped_sockaddr); + + return iwpm_create_mapinfo(&local_sockaddr, + &mapped_sockaddr, RDMA_NL_NES); +} + +/* + * nes_remove_mapinfo - Remove a mapinfo object from the port mapper data base + * and send a remove mapping op message to + * the userspace port mapper + */ +static int nes_remove_mapinfo(u32 loc_addr, u16 loc_port, + u32 mapped_loc_addr, u16 mapped_loc_port) +{ + struct sockaddr_storage local_sockaddr; + struct sockaddr_storage mapped_sockaddr; + + nes_create_sockaddr(htonl(loc_addr), htons(loc_port), &local_sockaddr); + nes_create_sockaddr(htonl(mapped_loc_addr), htons(mapped_loc_port), + &mapped_sockaddr); + + iwpm_remove_mapinfo(&local_sockaddr, &mapped_sockaddr); + return iwpm_remove_mapping(&local_sockaddr, RDMA_NL_NES); +} + +/* + * nes_form_pm_msg - Form a port mapper message with mapping info + */ +static void nes_form_pm_msg(struct nes_cm_info *cm_info, + struct iwpm_sa_data *pm_msg) +{ + nes_create_sockaddr(htonl(cm_info->loc_addr), htons(cm_info->loc_port), + &pm_msg->loc_addr); + nes_create_sockaddr(htonl(cm_info->rem_addr), htons(cm_info->rem_port), + &pm_msg->rem_addr); +} + +/* + * nes_form_reg_msg - Form a port mapper message with dev info + */ +static void nes_form_reg_msg(struct nes_vnic *nesvnic, + struct iwpm_dev_data *pm_msg) +{ + memcpy(pm_msg->dev_name, nesvnic->nesibdev->ibdev.name, + IWPM_DEVNAME_SIZE); + memcpy(pm_msg->if_name, nesvnic->netdev->name, IWPM_IFNAME_SIZE); +} + +/* + * nes_record_pm_msg - Save the received mapping info + */ +static void nes_record_pm_msg(struct nes_cm_info *cm_info, + struct iwpm_sa_data *pm_msg) +{ + struct sockaddr_in *mapped_loc_addr = + (struct sockaddr_in *)&pm_msg->mapped_loc_addr; + struct sockaddr_in *mapped_rem_addr = + (struct sockaddr_in *)&pm_msg->mapped_rem_addr; + + if (mapped_loc_addr->sin_family == AF_INET) { + cm_info->mapped_loc_addr = + ntohl(mapped_loc_addr->sin_addr.s_addr); + cm_info->mapped_loc_port = ntohs(mapped_loc_addr->sin_port); + } + if (mapped_rem_addr->sin_family == AF_INET) { + cm_info->mapped_rem_addr = + ntohl(mapped_rem_addr->sin_addr.s_addr); + cm_info->mapped_rem_port = ntohs(mapped_rem_addr->sin_port); + } +} + /** * print_core - dump a cm core */ @@ -1147,8 +1241,11 @@ static struct nes_cm_node *find_node(struct nes_cm_core *cm_core, loc_addr, loc_port, cm_node->rem_addr, cm_node->rem_port, rem_addr, rem_port); - if ((cm_node->loc_addr == loc_addr) && (cm_node->loc_port == loc_port) && - (cm_node->rem_addr == rem_addr) && (cm_node->rem_port == rem_port)) { + if ((cm_node->mapped_loc_addr == loc_addr) && + (cm_node->mapped_loc_port == loc_port) && + (cm_node->mapped_rem_addr == rem_addr) && + (cm_node->mapped_rem_port == rem_port)) { + add_ref_cm_node(cm_node); spin_unlock_irqrestore(&cm_core->ht_lock, flags); return cm_node; @@ -1165,18 +1262,28 @@ static struct nes_cm_node *find_node(struct nes_cm_core *cm_core, * find_listener - find a cm node listening on this addr-port pair */ static struct nes_cm_listener *find_listener(struct nes_cm_core *cm_core, - nes_addr_t dst_addr, u16 dst_port, enum nes_cm_listener_state listener_state) + nes_addr_t dst_addr, u16 dst_port, + enum nes_cm_listener_state listener_state, int local) { unsigned long flags; struct nes_cm_listener *listen_node; + nes_addr_t listen_addr; + u16 listen_port; /* walk list and find cm_node associated with this session ID */ spin_lock_irqsave(&cm_core->listen_list_lock, flags); list_for_each_entry(listen_node, &cm_core->listen_list.list, list) { + if (local) { + listen_addr = listen_node->loc_addr; + listen_port = listen_node->loc_port; + } else { + listen_addr = listen_node->mapped_loc_addr; + listen_port = listen_node->mapped_loc_port; + } /* compare node pair, return node handle if a match */ - if (((listen_node->loc_addr == dst_addr) || - listen_node->loc_addr == 0x00000000) && - (listen_node->loc_port == dst_port) && + if (((listen_addr == dst_addr) || + listen_addr == 0x00000000) && + (listen_port == dst_port) && (listener_state & listen_node->listener_state)) { atomic_inc(&listen_node->ref_count); spin_unlock_irqrestore(&cm_core->listen_list_lock, flags); @@ -1189,7 +1296,6 @@ static struct nes_cm_listener *find_listener(struct nes_cm_core *cm_core, return NULL; } - /** * add_hte_node - add a cm node to the hash table */ @@ -1310,9 +1416,20 @@ static int mini_cm_dec_refcnt_listen(struct nes_cm_core *cm_core, spin_unlock_irqrestore(&cm_core->listen_list_lock, flags); - if (listener->nesvnic) - nes_manage_apbvt(listener->nesvnic, listener->loc_port, - PCI_FUNC(listener->nesvnic->nesdev->pcidev->devfn), NES_MANAGE_APBVT_DEL); + if (listener->nesvnic) { + nes_manage_apbvt(listener->nesvnic, + listener->mapped_loc_port, + PCI_FUNC(listener->nesvnic->nesdev->pcidev->devfn), + NES_MANAGE_APBVT_DEL); + + nes_remove_mapinfo(listener->loc_addr, + listener->loc_port, + listener->mapped_loc_addr, + listener->mapped_loc_port); + nes_debug(NES_DBG_NLMSG, + "Delete APBVT mapped_loc_port = %04X\n", + listener->mapped_loc_port); + } nes_debug(NES_DBG_CM, "destroying listener (%p)\n", listener); @@ -1454,6 +1571,11 @@ static struct nes_cm_node *make_cm_node(struct nes_cm_core *cm_core, cm_node->loc_port = cm_info->loc_port; cm_node->rem_port = cm_info->rem_port; + cm_node->mapped_loc_addr = cm_info->mapped_loc_addr; + cm_node->mapped_rem_addr = cm_info->mapped_rem_addr; + cm_node->mapped_loc_port = cm_info->mapped_loc_port; + cm_node->mapped_rem_port = cm_info->mapped_rem_port; + cm_node->mpa_frame_rev = mpa_version; cm_node->send_rdma0_op = SEND_RDMA_READ_ZERO; cm_node->mpav2_ird_ord = 0; @@ -1500,8 +1622,10 @@ static struct nes_cm_node *make_cm_node(struct nes_cm_core *cm_core, cm_node->loopbackpartner = NULL; /* get the mac addr for the remote node */ - oldarpindex = nes_arp_table(nesdev, cm_node->rem_addr, NULL, NES_ARP_RESOLVE); - arpindex = nes_addr_resolve_neigh(nesvnic, cm_info->rem_addr, oldarpindex); + oldarpindex = nes_arp_table(nesdev, cm_node->mapped_rem_addr, + NULL, NES_ARP_RESOLVE); + arpindex = nes_addr_resolve_neigh(nesvnic, + cm_node->mapped_rem_addr, oldarpindex); if (arpindex < 0) { kfree(cm_node); return NULL; @@ -1563,11 +1687,14 @@ static int rem_ref_cm_node(struct nes_cm_core *cm_core, mini_cm_dec_refcnt_listen(cm_core, cm_node->listener, 0); } else { if (cm_node->apbvt_set && cm_node->nesvnic) { - nes_manage_apbvt(cm_node->nesvnic, cm_node->loc_port, - PCI_FUNC( - cm_node->nesvnic->nesdev->pcidev->devfn), + nes_manage_apbvt(cm_node->nesvnic, cm_node->mapped_loc_port, + PCI_FUNC(cm_node->nesvnic->nesdev->pcidev->devfn), NES_MANAGE_APBVT_DEL); } + nes_debug(NES_DBG_NLMSG, "Delete APBVT mapped_loc_port = %04X\n", + cm_node->mapped_loc_port); + nes_remove_mapinfo(cm_node->loc_addr, cm_node->loc_port, + cm_node->mapped_loc_addr, cm_node->mapped_loc_port); } atomic_dec(&cm_core->node_cnt); @@ -2235,17 +2362,21 @@ static void process_packet(struct nes_cm_node *cm_node, struct sk_buff *skb, * mini_cm_listen - create a listen node with params */ static struct nes_cm_listener *mini_cm_listen(struct nes_cm_core *cm_core, - struct nes_vnic *nesvnic, struct nes_cm_info *cm_info) + struct nes_vnic *nesvnic, struct nes_cm_info *cm_info) { struct nes_cm_listener *listener; + struct iwpm_dev_data pm_reg_msg; + struct iwpm_sa_data pm_msg; unsigned long flags; + int iwpm_err = 0; nes_debug(NES_DBG_CM, "Search for 0x%08x : 0x%04x\n", cm_info->loc_addr, cm_info->loc_port); /* cannot have multiple matching listeners */ - listener = find_listener(cm_core, htonl(cm_info->loc_addr), - htons(cm_info->loc_port), NES_CM_LISTENER_EITHER_STATE); + listener = find_listener(cm_core, cm_info->loc_addr, cm_info->loc_port, + NES_CM_LISTENER_EITHER_STATE, 1); + if (listener && listener->listener_state == NES_CM_LISTENER_ACTIVE_STATE) { /* find automatically incs ref count ??? */ atomic_dec(&listener->ref_count); @@ -2254,6 +2385,22 @@ static struct nes_cm_listener *mini_cm_listen(struct nes_cm_core *cm_core, } if (!listener) { + nes_form_reg_msg(nesvnic, &pm_reg_msg); + iwpm_err = iwpm_register_pid(&pm_reg_msg, RDMA_NL_NES); + if (iwpm_err) { + nes_debug(NES_DBG_NLMSG, + "Port Mapper reg pid fail (err = %d).\n", iwpm_err); + } + if (iwpm_valid_pid() && !iwpm_err) { + nes_form_pm_msg(cm_info, &pm_msg); + iwpm_err = iwpm_add_mapping(&pm_msg, RDMA_NL_NES); + if (iwpm_err) + nes_debug(NES_DBG_NLMSG, + "Port Mapper query fail (err = %d).\n", iwpm_err); + else + nes_record_pm_msg(cm_info, &pm_msg); + } + /* create a CM listen node (1/2 node to compare incoming traffic to) */ listener = kzalloc(sizeof(*listener), GFP_ATOMIC); if (!listener) { @@ -2261,8 +2408,10 @@ static struct nes_cm_listener *mini_cm_listen(struct nes_cm_core *cm_core, return NULL; } - listener->loc_addr = htonl(cm_info->loc_addr); - listener->loc_port = htons(cm_info->loc_port); + listener->loc_addr = cm_info->loc_addr; + listener->loc_port = cm_info->loc_port; + listener->mapped_loc_addr = cm_info->mapped_loc_addr; + listener->mapped_loc_port = cm_info->mapped_loc_port; listener->reused_node = 0; atomic_set(&listener->ref_count, 1); @@ -2324,14 +2473,18 @@ static struct nes_cm_node *mini_cm_connect(struct nes_cm_core *cm_core, if (cm_info->loc_addr == cm_info->rem_addr) { loopbackremotelistener = find_listener(cm_core, - ntohl(nesvnic->local_ipaddr), cm_node->rem_port, - NES_CM_LISTENER_ACTIVE_STATE); + cm_node->mapped_loc_addr, cm_node->mapped_rem_port, + NES_CM_LISTENER_ACTIVE_STATE, 0); if (loopbackremotelistener == NULL) { create_event(cm_node, NES_CM_EVENT_ABORTED); } else { loopback_cm_info = *cm_info; loopback_cm_info.loc_port = cm_info->rem_port; loopback_cm_info.rem_port = cm_info->loc_port; + loopback_cm_info.mapped_loc_port = + cm_info->mapped_rem_port; + loopback_cm_info.mapped_rem_port = + cm_info->mapped_loc_port; loopback_cm_info.cm_id = loopbackremotelistener->cm_id; loopbackremotenode = make_cm_node(cm_core, nesvnic, &loopback_cm_info, loopbackremotelistener); @@ -2560,6 +2713,12 @@ static int mini_cm_recv_pkt(struct nes_cm_core *cm_core, nfo.rem_addr = ntohl(iph->saddr); nfo.rem_port = ntohs(tcph->source); + /* If port mapper is available these should be mapped address info */ + nfo.mapped_loc_addr = ntohl(iph->daddr); + nfo.mapped_loc_port = ntohs(tcph->dest); + nfo.mapped_rem_addr = ntohl(iph->saddr); + nfo.mapped_rem_port = ntohs(tcph->source); + tmp_daddr = cpu_to_be32(iph->daddr); tmp_saddr = cpu_to_be32(iph->saddr); @@ -2568,8 +2727,8 @@ static int mini_cm_recv_pkt(struct nes_cm_core *cm_core, do { cm_node = find_node(cm_core, - nfo.rem_port, nfo.rem_addr, - nfo.loc_port, nfo.loc_addr); + nfo.mapped_rem_port, nfo.mapped_rem_addr, + nfo.mapped_loc_port, nfo.mapped_loc_addr); if (!cm_node) { /* Only type of packet accepted are for */ @@ -2578,9 +2737,9 @@ static int mini_cm_recv_pkt(struct nes_cm_core *cm_core, skb_handled = 0; break; } - listener = find_listener(cm_core, nfo.loc_addr, - nfo.loc_port, - NES_CM_LISTENER_ACTIVE_STATE); + listener = find_listener(cm_core, nfo.mapped_loc_addr, + nfo.mapped_loc_port, + NES_CM_LISTENER_ACTIVE_STATE, 0); if (!listener) { nfo.cm_id = NULL; nfo.conn_type = 0; @@ -3184,10 +3343,12 @@ int nes_accept(struct iw_cm_id *cm_id, struct iw_cm_conn_param *conn_param) nes_cm_init_tsa_conn(nesqp, cm_node); - nesqp->nesqp_context->tcpPorts[0] = cpu_to_le16(ntohs(laddr->sin_port)); - nesqp->nesqp_context->tcpPorts[1] = cpu_to_le16(ntohs(raddr->sin_port)); + nesqp->nesqp_context->tcpPorts[0] = + cpu_to_le16(cm_node->mapped_loc_port); + nesqp->nesqp_context->tcpPorts[1] = + cpu_to_le16(cm_node->mapped_rem_port); - nesqp->nesqp_context->ip0 = cpu_to_le32(ntohl(raddr->sin_addr.s_addr)); + nesqp->nesqp_context->ip0 = cpu_to_le32(cm_node->mapped_rem_addr); nesqp->nesqp_context->misc2 |= cpu_to_le32( (u32)PCI_FUNC(nesdev->pcidev->devfn) << @@ -3211,9 +3372,9 @@ int nes_accept(struct iw_cm_id *cm_id, struct iw_cm_conn_param *conn_param) memset(&nes_quad, 0, sizeof(nes_quad)); nes_quad.DstIpAdrIndex = cpu_to_le32((u32)PCI_FUNC(nesdev->pcidev->devfn) << 24); - nes_quad.SrcIpadr = raddr->sin_addr.s_addr; - nes_quad.TcpPorts[0] = raddr->sin_port; - nes_quad.TcpPorts[1] = laddr->sin_port; + nes_quad.SrcIpadr = htonl(cm_node->mapped_rem_addr); + nes_quad.TcpPorts[0] = htons(cm_node->mapped_rem_port); + nes_quad.TcpPorts[1] = htons(cm_node->mapped_loc_port); /* Produce hash key */ crc_value = get_crc_value(&nes_quad); @@ -3315,6 +3476,9 @@ int nes_connect(struct iw_cm_id *cm_id, struct iw_cm_conn_param *conn_param) int apbvt_set = 0; struct sockaddr_in *laddr = (struct sockaddr_in *)&cm_id->local_addr; struct sockaddr_in *raddr = (struct sockaddr_in *)&cm_id->remote_addr; + struct iwpm_dev_data pm_reg_msg; + struct iwpm_sa_data pm_msg; + int iwpm_err = 0; if (cm_id->remote_addr.ss_family != AF_INET) return -ENOSYS; @@ -3352,20 +3516,44 @@ int nes_connect(struct iw_cm_id *cm_id, struct iw_cm_conn_param *conn_param) nes_debug(NES_DBG_CM, "mpa private data len =%u\n", conn_param->private_data_len); + /* set up the connection params for the node */ + cm_info.loc_addr = ntohl(laddr->sin_addr.s_addr); + cm_info.loc_port = ntohs(laddr->sin_port); + cm_info.rem_addr = ntohl(raddr->sin_addr.s_addr); + cm_info.rem_port = ntohs(raddr->sin_port); + cm_info.cm_id = cm_id; + cm_info.conn_type = NES_CM_IWARP_CONN_TYPE; + + /* No port mapper available, go with the specified peer information */ + cm_info.mapped_loc_addr = cm_info.loc_addr; + cm_info.mapped_loc_port = cm_info.loc_port; + cm_info.mapped_rem_addr = cm_info.rem_addr; + cm_info.mapped_rem_port = cm_info.rem_port; + + nes_form_reg_msg(nesvnic, &pm_reg_msg); + iwpm_err = iwpm_register_pid(&pm_reg_msg, RDMA_NL_NES); + if (iwpm_err) { + nes_debug(NES_DBG_NLMSG, + "Port Mapper reg pid fail (err = %d).\n", iwpm_err); + } + if (iwpm_valid_pid() && !iwpm_err) { + nes_form_pm_msg(&cm_info, &pm_msg); + iwpm_err = iwpm_add_and_query_mapping(&pm_msg, RDMA_NL_NES); + if (iwpm_err) + nes_debug(NES_DBG_NLMSG, + "Port Mapper query fail (err = %d).\n", iwpm_err); + else + nes_record_pm_msg(&cm_info, &pm_msg); + } + if (laddr->sin_addr.s_addr != raddr->sin_addr.s_addr) { - nes_manage_apbvt(nesvnic, ntohs(laddr->sin_port), - PCI_FUNC(nesdev->pcidev->devfn), - NES_MANAGE_APBVT_ADD); + nes_manage_apbvt(nesvnic, cm_info.mapped_loc_port, + PCI_FUNC(nesdev->pcidev->devfn), NES_MANAGE_APBVT_ADD); apbvt_set = 1; } - /* set up the connection params for the node */ - cm_info.loc_addr = htonl(laddr->sin_addr.s_addr); - cm_info.loc_port = htons(laddr->sin_port); - cm_info.rem_addr = htonl(raddr->sin_addr.s_addr); - cm_info.rem_port = htons(raddr->sin_port); - cm_info.cm_id = cm_id; - cm_info.conn_type = NES_CM_IWARP_CONN_TYPE; + if (nes_create_mapinfo(&cm_info)) + return -ENOMEM; cm_id->add_ref(cm_id); @@ -3375,10 +3563,14 @@ int nes_connect(struct iw_cm_id *cm_id, struct iw_cm_conn_param *conn_param) &cm_info); if (!cm_node) { if (apbvt_set) - nes_manage_apbvt(nesvnic, ntohs(laddr->sin_port), + nes_manage_apbvt(nesvnic, cm_info.mapped_loc_port, PCI_FUNC(nesdev->pcidev->devfn), NES_MANAGE_APBVT_DEL); + nes_debug(NES_DBG_NLMSG, "Delete mapped_loc_port = %04X\n", + cm_info.mapped_loc_port); + nes_remove_mapinfo(cm_info.loc_addr, cm_info.loc_port, + cm_info.mapped_loc_addr, cm_info.mapped_loc_port); cm_id->rem_ref(cm_id); return -ENOMEM; } @@ -3424,13 +3616,16 @@ int nes_create_listen(struct iw_cm_id *cm_id, int backlog) nesvnic->local_ipaddr, laddr->sin_addr.s_addr); /* setup listen params in our api call struct */ - cm_info.loc_addr = nesvnic->local_ipaddr; - cm_info.loc_port = laddr->sin_port; + cm_info.loc_addr = ntohl(nesvnic->local_ipaddr); + cm_info.loc_port = ntohs(laddr->sin_port); cm_info.backlog = backlog; cm_info.cm_id = cm_id; cm_info.conn_type = NES_CM_IWARP_CONN_TYPE; + /* No port mapper available, go with the specified info */ + cm_info.mapped_loc_addr = cm_info.loc_addr; + cm_info.mapped_loc_port = cm_info.loc_port; cm_node = g_cm_core->api->listen(g_cm_core, nesvnic, &cm_info); if (!cm_node) { @@ -3442,7 +3637,10 @@ int nes_create_listen(struct iw_cm_id *cm_id, int backlog) cm_id->provider_data = cm_node; if (!cm_node->reused_node) { - err = nes_manage_apbvt(nesvnic, ntohs(laddr->sin_port), + if (nes_create_mapinfo(&cm_info)) + return -ENOMEM; + + err = nes_manage_apbvt(nesvnic, cm_node->mapped_loc_port, PCI_FUNC(nesvnic->nesdev->pcidev->devfn), NES_MANAGE_APBVT_ADD); if (err) { @@ -3567,9 +3765,11 @@ static void cm_event_connected(struct nes_cm_event *event) nes_cm_init_tsa_conn(nesqp, cm_node); /* set the QP tsa context */ - nesqp->nesqp_context->tcpPorts[0] = cpu_to_le16(ntohs(laddr->sin_port)); - nesqp->nesqp_context->tcpPorts[1] = cpu_to_le16(ntohs(raddr->sin_port)); - nesqp->nesqp_context->ip0 = cpu_to_le32(ntohl(raddr->sin_addr.s_addr)); + nesqp->nesqp_context->tcpPorts[0] = + cpu_to_le16(cm_node->mapped_loc_port); + nesqp->nesqp_context->tcpPorts[1] = + cpu_to_le16(cm_node->mapped_rem_port); + nesqp->nesqp_context->ip0 = cpu_to_le32(cm_node->mapped_rem_addr); nesqp->nesqp_context->misc2 |= cpu_to_le32( (u32)PCI_FUNC(nesdev->pcidev->devfn) << @@ -3599,9 +3799,9 @@ static void cm_event_connected(struct nes_cm_event *event) nes_quad.DstIpAdrIndex = cpu_to_le32((u32)PCI_FUNC(nesdev->pcidev->devfn) << 24); - nes_quad.SrcIpadr = raddr->sin_addr.s_addr; - nes_quad.TcpPorts[0] = raddr->sin_port; - nes_quad.TcpPorts[1] = laddr->sin_port; + nes_quad.SrcIpadr = htonl(cm_node->mapped_rem_addr); + nes_quad.TcpPorts[0] = htons(cm_node->mapped_rem_port); + nes_quad.TcpPorts[1] = htons(cm_node->mapped_loc_port); /* Produce hash key */ crc_value = get_crc_value(&nes_quad); @@ -3629,7 +3829,7 @@ static void cm_event_connected(struct nes_cm_event *event) cm_event.ird = cm_node->ird_size; cm_event.ord = cm_node->ord_size; - cm_event_laddr->sin_addr.s_addr = event->cm_info.rem_addr; + cm_event_laddr->sin_addr.s_addr = htonl(event->cm_info.rem_addr); ret = cm_id->event_handler(cm_id, &cm_event); nes_debug(NES_DBG_CM, "OFA CM event_handler returned, ret=%d\n", ret); diff --git a/drivers/infiniband/hw/nes/nes_cm.h b/drivers/infiniband/hw/nes/nes_cm.h index 522c99cd07c4..f522cf639789 100644 --- a/drivers/infiniband/hw/nes/nes_cm.h +++ b/drivers/infiniband/hw/nes/nes_cm.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2006 - 2011 Intel Corporation. All rights reserved. + * Copyright (c) 2006 - 2014 Intel Corporation. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -293,8 +293,8 @@ struct nes_cm_listener { struct list_head list; struct nes_cm_core *cm_core; u8 loc_mac[ETH_ALEN]; - nes_addr_t loc_addr; - u16 loc_port; + nes_addr_t loc_addr, mapped_loc_addr; + u16 loc_port, mapped_loc_port; struct iw_cm_id *cm_id; enum nes_cm_conn_type conn_type; atomic_t ref_count; @@ -308,7 +308,9 @@ struct nes_cm_listener { /* per connection node and node state information */ struct nes_cm_node { nes_addr_t loc_addr, rem_addr; + nes_addr_t mapped_loc_addr, mapped_rem_addr; u16 loc_port, rem_port; + u16 mapped_loc_port, mapped_rem_port; u8 loc_mac[ETH_ALEN]; u8 rem_mac[ETH_ALEN]; @@ -364,6 +366,10 @@ struct nes_cm_info { u16 rem_port; nes_addr_t loc_addr; nes_addr_t rem_addr; + u16 mapped_loc_port; + u16 mapped_rem_port; + nes_addr_t mapped_loc_addr; + nes_addr_t mapped_rem_addr; enum nes_cm_conn_type conn_type; int backlog; diff --git a/drivers/infiniband/hw/ocrdma/ocrdma_stats.c b/drivers/infiniband/hw/ocrdma/ocrdma_stats.c index 6c54106f5e64..41a9aec9998d 100644 --- a/drivers/infiniband/hw/ocrdma/ocrdma_stats.c +++ b/drivers/infiniband/hw/ocrdma/ocrdma_stats.c @@ -510,16 +510,9 @@ exit: return status; } -static int ocrdma_debugfs_open(struct inode *inode, struct file *file) -{ - if (inode->i_private) - file->private_data = inode->i_private; - return 0; -} - static const struct file_operations ocrdma_dbg_ops = { .owner = THIS_MODULE, - .open = ocrdma_debugfs_open, + .open = simple_open, .read = ocrdma_dbgfs_ops_read, }; diff --git a/drivers/infiniband/hw/qib/qib_init.c b/drivers/infiniband/hw/qib/qib_init.c index 5b7aeb224a30..8d3c78ddc906 100644 --- a/drivers/infiniband/hw/qib/qib_init.c +++ b/drivers/infiniband/hw/qib/qib_init.c @@ -1272,7 +1272,7 @@ static int qib_notify_dca(struct notifier_block *nb, unsigned long event, * Do all the generic driver unit- and chip-independent memory * allocation and initialization. */ -static int __init qlogic_ib_init(void) +static int __init qib_ib_init(void) { int ret; @@ -1316,12 +1316,12 @@ bail: return ret; } -module_init(qlogic_ib_init); +module_init(qib_ib_init); /* * Do the non-unit driver cleanup, memory free, etc. at unload. */ -static void __exit qlogic_ib_cleanup(void) +static void __exit qib_ib_cleanup(void) { int ret; @@ -1346,7 +1346,7 @@ static void __exit qlogic_ib_cleanup(void) qib_dev_cleanup(); } -module_exit(qlogic_ib_cleanup); +module_exit(qib_ib_cleanup); /* this can only be called after a successful initialization */ static void cleanup_device_data(struct qib_devdata *dd) diff --git a/drivers/infiniband/hw/qib/qib_mad.c b/drivers/infiniband/hw/qib/qib_mad.c index edad991d60ed..22c720e5740d 100644 --- a/drivers/infiniband/hw/qib/qib_mad.c +++ b/drivers/infiniband/hw/qib/qib_mad.c @@ -1028,7 +1028,7 @@ static int set_pkeys(struct qib_devdata *dd, u8 port, u16 *pkeys) event.event = IB_EVENT_PKEY_CHANGE; event.device = &dd->verbs_dev.ibdev; - event.element.port_num = 1; + event.element.port_num = port; ib_dispatch_event(&event); } return 0; diff --git a/drivers/infiniband/hw/qib/qib_qp.c b/drivers/infiniband/hw/qib/qib_qp.c index 0cad0c40d742..7fcc150d603c 100644 --- a/drivers/infiniband/hw/qib/qib_qp.c +++ b/drivers/infiniband/hw/qib/qib_qp.c @@ -985,7 +985,8 @@ struct ib_qp *qib_create_qp(struct ib_pd *ibpd, struct ib_qp *ret; if (init_attr->cap.max_send_sge > ib_qib_max_sges || - init_attr->cap.max_send_wr > ib_qib_max_qp_wrs) { + init_attr->cap.max_send_wr > ib_qib_max_qp_wrs || + init_attr->create_flags) { ret = ERR_PTR(-EINVAL); goto bail; } diff --git a/drivers/infiniband/hw/usnic/usnic_ib_verbs.c b/drivers/infiniband/hw/usnic/usnic_ib_verbs.c index d48d2c0a2e3c..53bd6a2d9cdb 100644 --- a/drivers/infiniband/hw/usnic/usnic_ib_verbs.c +++ b/drivers/infiniband/hw/usnic/usnic_ib_verbs.c @@ -466,6 +466,9 @@ struct ib_qp *usnic_ib_create_qp(struct ib_pd *pd, ucontext = to_uucontext(pd->uobject->context); us_ibdev = to_usdev(pd->device); + if (init_attr->create_flags) + return ERR_PTR(-EINVAL); + err = ib_copy_from_udata(&cmd, udata, sizeof(cmd)); if (err) { usnic_err("%s: cannot copy udata for create_qp\n", diff --git a/drivers/infiniband/hw/usnic/usnic_uiom_interval_tree.c b/drivers/infiniband/hw/usnic/usnic_uiom_interval_tree.c index d135ad90d914..3a4288e0fbac 100644 --- a/drivers/infiniband/hw/usnic/usnic_uiom_interval_tree.c +++ b/drivers/infiniband/hw/usnic/usnic_uiom_interval_tree.c @@ -1,3 +1,21 @@ +/* + * Copyright (c) 2014, Cisco Systems, Inc. All rights reserved. + * + * This program is free software; you may redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; version 2 of the License. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + * + */ + #include <linux/init.h> #include <linux/list.h> #include <linux/slab.h> diff --git a/drivers/infiniband/ulp/Makefile b/drivers/infiniband/ulp/Makefile new file mode 100644 index 000000000000..f3c7dcf03098 --- /dev/null +++ b/drivers/infiniband/ulp/Makefile @@ -0,0 +1,5 @@ +obj-$(CONFIG_INFINIBAND_IPOIB) += ipoib/ +obj-$(CONFIG_INFINIBAND_SRP) += srp/ +obj-$(CONFIG_INFINIBAND_SRPT) += srpt/ +obj-$(CONFIG_INFINIBAND_ISER) += iser/ +obj-$(CONFIG_INFINIBAND_ISERT) += isert/ diff --git a/drivers/infiniband/ulp/ipoib/ipoib_cm.c b/drivers/infiniband/ulp/ipoib/ipoib_cm.c index 1377f85911c2..933efcea0d03 100644 --- a/drivers/infiniband/ulp/ipoib/ipoib_cm.c +++ b/drivers/infiniband/ulp/ipoib/ipoib_cm.c @@ -1030,10 +1030,20 @@ static struct ib_qp *ipoib_cm_create_tx_qp(struct net_device *dev, struct ipoib_ .cap.max_send_sge = 1, .sq_sig_type = IB_SIGNAL_ALL_WR, .qp_type = IB_QPT_RC, - .qp_context = tx + .qp_context = tx, + .create_flags = IB_QP_CREATE_USE_GFP_NOIO }; - return ib_create_qp(priv->pd, &attr); + struct ib_qp *tx_qp; + + tx_qp = ib_create_qp(priv->pd, &attr); + if (PTR_ERR(tx_qp) == -EINVAL) { + ipoib_warn(priv, "can't use GFP_NOIO for QPs on device %s, using GFP_KERNEL\n", + priv->ca->name); + attr.create_flags &= ~IB_QP_CREATE_USE_GFP_NOIO; + tx_qp = ib_create_qp(priv->pd, &attr); + } + return tx_qp; } static int ipoib_cm_send_req(struct net_device *dev, @@ -1104,12 +1114,14 @@ static int ipoib_cm_tx_init(struct ipoib_cm_tx *p, u32 qpn, struct ipoib_dev_priv *priv = netdev_priv(p->dev); int ret; - p->tx_ring = vzalloc(ipoib_sendq_size * sizeof *p->tx_ring); + p->tx_ring = __vmalloc(ipoib_sendq_size * sizeof *p->tx_ring, + GFP_NOIO, PAGE_KERNEL); if (!p->tx_ring) { ipoib_warn(priv, "failed to allocate tx ring\n"); ret = -ENOMEM; goto err_tx; } + memset(p->tx_ring, 0, ipoib_sendq_size * sizeof *p->tx_ring); p->qp = ipoib_cm_create_tx_qp(p->dev, p); if (IS_ERR(p->qp)) { diff --git a/drivers/infiniband/ulp/iser/iscsi_iser.c b/drivers/infiniband/ulp/iser/iscsi_iser.c index 25f195ef44b0..eb7973957a6e 100644 --- a/drivers/infiniband/ulp/iser/iscsi_iser.c +++ b/drivers/infiniband/ulp/iser/iscsi_iser.c @@ -99,6 +99,7 @@ MODULE_PARM_DESC(pi_enable, "Enable T10-PI offload support (default:disabled)"); module_param_named(pi_guard, iser_pi_guard, int, 0644); MODULE_PARM_DESC(pi_guard, "T10-PI guard_type, 0:CRC|1:IP_CSUM (default:CRC)"); +static struct workqueue_struct *release_wq; struct iser_global ig; void @@ -337,24 +338,6 @@ iscsi_iser_conn_create(struct iscsi_cls_session *cls_session, uint32_t conn_idx) return cls_conn; } -static void -iscsi_iser_conn_destroy(struct iscsi_cls_conn *cls_conn) -{ - struct iscsi_conn *conn = cls_conn->dd_data; - struct iser_conn *ib_conn = conn->dd_data; - - iscsi_conn_teardown(cls_conn); - /* - * Userspace will normally call the stop callback and - * already have freed the ib_conn, but if it goofed up then - * we free it here. - */ - if (ib_conn) { - ib_conn->iscsi_conn = NULL; - iser_conn_put(ib_conn, 1); /* deref iscsi/ib conn unbinding */ - } -} - static int iscsi_iser_conn_bind(struct iscsi_cls_session *cls_session, struct iscsi_cls_conn *cls_conn, uint64_t transport_eph, @@ -392,29 +375,39 @@ iscsi_iser_conn_bind(struct iscsi_cls_session *cls_session, conn->dd_data = ib_conn; ib_conn->iscsi_conn = conn; - iser_conn_get(ib_conn); /* ref iscsi/ib conn binding */ return 0; } +static int +iscsi_iser_conn_start(struct iscsi_cls_conn *cls_conn) +{ + struct iscsi_conn *iscsi_conn; + struct iser_conn *ib_conn; + + iscsi_conn = cls_conn->dd_data; + ib_conn = iscsi_conn->dd_data; + reinit_completion(&ib_conn->stop_completion); + + return iscsi_conn_start(cls_conn); +} + static void iscsi_iser_conn_stop(struct iscsi_cls_conn *cls_conn, int flag) { struct iscsi_conn *conn = cls_conn->dd_data; struct iser_conn *ib_conn = conn->dd_data; + iser_dbg("stopping iscsi_conn: %p, ib_conn: %p\n", conn, ib_conn); + iscsi_conn_stop(cls_conn, flag); + /* * Userspace may have goofed up and not bound the connection or * might have only partially setup the connection. */ if (ib_conn) { - iscsi_conn_stop(cls_conn, flag); - /* - * There is no unbind event so the stop callback - * must release the ref from the bind. - */ - iser_conn_put(ib_conn, 1); /* deref iscsi/ib conn unbinding */ + conn->dd_data = NULL; + complete(&ib_conn->stop_completion); } - conn->dd_data = NULL; } static void iscsi_iser_session_destroy(struct iscsi_cls_session *cls_session) @@ -515,28 +508,28 @@ iscsi_iser_set_param(struct iscsi_cls_conn *cls_conn, case ISCSI_PARAM_HDRDGST_EN: sscanf(buf, "%d", &value); if (value) { - iser_err("DataDigest wasn't negotiated to None"); + iser_err("DataDigest wasn't negotiated to None\n"); return -EPROTO; } break; case ISCSI_PARAM_DATADGST_EN: sscanf(buf, "%d", &value); if (value) { - iser_err("DataDigest wasn't negotiated to None"); + iser_err("DataDigest wasn't negotiated to None\n"); return -EPROTO; } break; case ISCSI_PARAM_IFMARKER_EN: sscanf(buf, "%d", &value); if (value) { - iser_err("IFMarker wasn't negotiated to No"); + iser_err("IFMarker wasn't negotiated to No\n"); return -EPROTO; } break; case ISCSI_PARAM_OFMARKER_EN: sscanf(buf, "%d", &value); if (value) { - iser_err("OFMarker wasn't negotiated to No"); + iser_err("OFMarker wasn't negotiated to No\n"); return -EPROTO; } break; @@ -652,19 +645,20 @@ iscsi_iser_ep_disconnect(struct iscsi_endpoint *ep) struct iser_conn *ib_conn; ib_conn = ep->dd_data; - if (ib_conn->iscsi_conn) - /* - * Must suspend xmit path if the ep is bound to the - * iscsi_conn, so we know we are not accessing the ib_conn - * when we free it. - * - * This may not be bound if the ep poll failed. - */ - iscsi_suspend_tx(ib_conn->iscsi_conn); - - - iser_info("ib conn %p state %d\n", ib_conn, ib_conn->state); + iser_info("ep %p ib conn %p state %d\n", ep, ib_conn, ib_conn->state); iser_conn_terminate(ib_conn); + + /* + * if iser_conn and iscsi_conn are bound, we must wait iscsi_conn_stop + * call and ISER_CONN_DOWN state before freeing the iser resources. + * otherwise we are safe to free resources immediately. + */ + if (ib_conn->iscsi_conn) { + INIT_WORK(&ib_conn->release_work, iser_release_work); + queue_work(release_wq, &ib_conn->release_work); + } else { + iser_conn_release(ib_conn); + } } static umode_t iser_attr_is_visible(int param_type, int param) @@ -748,13 +742,13 @@ static struct iscsi_transport iscsi_iser_transport = { /* connection management */ .create_conn = iscsi_iser_conn_create, .bind_conn = iscsi_iser_conn_bind, - .destroy_conn = iscsi_iser_conn_destroy, + .destroy_conn = iscsi_conn_teardown, .attr_is_visible = iser_attr_is_visible, .set_param = iscsi_iser_set_param, .get_conn_param = iscsi_conn_get_param, .get_ep_param = iscsi_iser_get_ep_param, .get_session_param = iscsi_session_get_param, - .start_conn = iscsi_conn_start, + .start_conn = iscsi_iser_conn_start, .stop_conn = iscsi_iser_conn_stop, /* iscsi host params */ .get_host_param = iscsi_host_get_param, @@ -801,6 +795,12 @@ static int __init iser_init(void) mutex_init(&ig.connlist_mutex); INIT_LIST_HEAD(&ig.connlist); + release_wq = alloc_workqueue("release workqueue", 0, 0); + if (!release_wq) { + iser_err("failed to allocate release workqueue\n"); + return -ENOMEM; + } + iscsi_iser_scsi_transport = iscsi_register_transport( &iscsi_iser_transport); if (!iscsi_iser_scsi_transport) { @@ -819,7 +819,24 @@ register_transport_failure: static void __exit iser_exit(void) { + struct iser_conn *ib_conn, *n; + int connlist_empty; + iser_dbg("Removing iSER datamover...\n"); + destroy_workqueue(release_wq); + + mutex_lock(&ig.connlist_mutex); + connlist_empty = list_empty(&ig.connlist); + mutex_unlock(&ig.connlist_mutex); + + if (!connlist_empty) { + iser_err("Error cleanup stage completed but we still have iser " + "connections, destroying them anyway.\n"); + list_for_each_entry_safe(ib_conn, n, &ig.connlist, conn_list) { + iser_conn_release(ib_conn); + } + } + iscsi_unregister_transport(&iscsi_iser_transport); kmem_cache_destroy(ig.desc_cache); } diff --git a/drivers/infiniband/ulp/iser/iscsi_iser.h b/drivers/infiniband/ulp/iser/iscsi_iser.h index 324129f80d40..97cd385bf7f7 100644 --- a/drivers/infiniband/ulp/iser/iscsi_iser.h +++ b/drivers/infiniband/ulp/iser/iscsi_iser.h @@ -69,7 +69,7 @@ #define DRV_NAME "iser" #define PFX DRV_NAME ": " -#define DRV_VER "1.3" +#define DRV_VER "1.4" #define iser_dbg(fmt, arg...) \ do { \ @@ -333,6 +333,8 @@ struct iser_conn { int post_recv_buf_count; /* posted rx count */ atomic_t post_send_buf_count; /* posted tx count */ char name[ISER_OBJECT_NAME_SIZE]; + struct work_struct release_work; + struct completion stop_completion; struct list_head conn_list; /* entry in ig conn list */ char *login_buf; @@ -417,12 +419,12 @@ void iscsi_iser_recv(struct iscsi_conn *conn, void iser_conn_init(struct iser_conn *ib_conn); -void iser_conn_get(struct iser_conn *ib_conn); - -int iser_conn_put(struct iser_conn *ib_conn, int destroy_cma_id_allowed); +void iser_conn_release(struct iser_conn *ib_conn); void iser_conn_terminate(struct iser_conn *ib_conn); +void iser_release_work(struct work_struct *work); + void iser_rcv_completion(struct iser_rx_desc *desc, unsigned long dto_xfer_len, struct iser_conn *ib_conn); diff --git a/drivers/infiniband/ulp/iser/iser_verbs.c b/drivers/infiniband/ulp/iser/iser_verbs.c index 32849f2becde..ea01075f9f9b 100644 --- a/drivers/infiniband/ulp/iser/iser_verbs.c +++ b/drivers/infiniband/ulp/iser/iser_verbs.c @@ -581,14 +581,30 @@ static int iser_conn_state_comp_exch(struct iser_conn *ib_conn, return ret; } +void iser_release_work(struct work_struct *work) +{ + struct iser_conn *ib_conn; + + ib_conn = container_of(work, struct iser_conn, release_work); + + /* wait for .conn_stop callback */ + wait_for_completion(&ib_conn->stop_completion); + + /* wait for the qp`s post send and post receive buffers to empty */ + wait_event_interruptible(ib_conn->wait, + ib_conn->state == ISER_CONN_DOWN); + + iser_conn_release(ib_conn); +} + /** * Frees all conn objects and deallocs conn descriptor */ -static void iser_conn_release(struct iser_conn *ib_conn, int can_destroy_id) +void iser_conn_release(struct iser_conn *ib_conn) { struct iser_device *device = ib_conn->device; - BUG_ON(ib_conn->state != ISER_CONN_DOWN); + BUG_ON(ib_conn->state == ISER_CONN_UP); mutex_lock(&ig.connlist_mutex); list_del(&ib_conn->conn_list); @@ -600,27 +616,13 @@ static void iser_conn_release(struct iser_conn *ib_conn, int can_destroy_id) if (device != NULL) iser_device_try_release(device); /* if cma handler context, the caller actually destroy the id */ - if (ib_conn->cma_id != NULL && can_destroy_id) { + if (ib_conn->cma_id != NULL) { rdma_destroy_id(ib_conn->cma_id); ib_conn->cma_id = NULL; } iscsi_destroy_endpoint(ib_conn->ep); } -void iser_conn_get(struct iser_conn *ib_conn) -{ - atomic_inc(&ib_conn->refcount); -} - -int iser_conn_put(struct iser_conn *ib_conn, int can_destroy_id) -{ - if (atomic_dec_and_test(&ib_conn->refcount)) { - iser_conn_release(ib_conn, can_destroy_id); - return 1; - } - return 0; -} - /** * triggers start of the disconnect procedures and wait for them to be done */ @@ -638,24 +640,19 @@ void iser_conn_terminate(struct iser_conn *ib_conn) if (err) iser_err("Failed to disconnect, conn: 0x%p err %d\n", ib_conn,err); - - wait_event_interruptible(ib_conn->wait, - ib_conn->state == ISER_CONN_DOWN); - - iser_conn_put(ib_conn, 1); /* deref ib conn deallocate */ } -static int iser_connect_error(struct rdma_cm_id *cma_id) +static void iser_connect_error(struct rdma_cm_id *cma_id) { struct iser_conn *ib_conn; + ib_conn = (struct iser_conn *)cma_id->context; ib_conn->state = ISER_CONN_DOWN; wake_up_interruptible(&ib_conn->wait); - return iser_conn_put(ib_conn, 0); /* deref ib conn's cma id */ } -static int iser_addr_handler(struct rdma_cm_id *cma_id) +static void iser_addr_handler(struct rdma_cm_id *cma_id) { struct iser_device *device; struct iser_conn *ib_conn; @@ -664,7 +661,8 @@ static int iser_addr_handler(struct rdma_cm_id *cma_id) device = iser_device_find_by_ib_device(cma_id); if (!device) { iser_err("device lookup/creation failed\n"); - return iser_connect_error(cma_id); + iser_connect_error(cma_id); + return; } ib_conn = (struct iser_conn *)cma_id->context; @@ -686,13 +684,12 @@ static int iser_addr_handler(struct rdma_cm_id *cma_id) ret = rdma_resolve_route(cma_id, 1000); if (ret) { iser_err("resolve route failed: %d\n", ret); - return iser_connect_error(cma_id); + iser_connect_error(cma_id); + return; } - - return 0; } -static int iser_route_handler(struct rdma_cm_id *cma_id) +static void iser_route_handler(struct rdma_cm_id *cma_id) { struct rdma_conn_param conn_param; int ret; @@ -720,9 +717,9 @@ static int iser_route_handler(struct rdma_cm_id *cma_id) goto failure; } - return 0; + return; failure: - return iser_connect_error(cma_id); + iser_connect_error(cma_id); } static void iser_connected_handler(struct rdma_cm_id *cma_id) @@ -735,14 +732,13 @@ static void iser_connected_handler(struct rdma_cm_id *cma_id) iser_info("remote qpn:%x my qpn:%x\n", attr.dest_qp_num, cma_id->qp->qp_num); ib_conn = (struct iser_conn *)cma_id->context; - ib_conn->state = ISER_CONN_UP; - wake_up_interruptible(&ib_conn->wait); + if (iser_conn_state_comp_exch(ib_conn, ISER_CONN_PENDING, ISER_CONN_UP)) + wake_up_interruptible(&ib_conn->wait); } -static int iser_disconnected_handler(struct rdma_cm_id *cma_id) +static void iser_disconnected_handler(struct rdma_cm_id *cma_id) { struct iser_conn *ib_conn; - int ret; ib_conn = (struct iser_conn *)cma_id->context; @@ -762,24 +758,19 @@ static int iser_disconnected_handler(struct rdma_cm_id *cma_id) ib_conn->state = ISER_CONN_DOWN; wake_up_interruptible(&ib_conn->wait); } - - ret = iser_conn_put(ib_conn, 0); /* deref ib conn's cma id */ - return ret; } static int iser_cma_handler(struct rdma_cm_id *cma_id, struct rdma_cm_event *event) { - int ret = 0; - iser_info("event %d status %d conn %p id %p\n", event->event, event->status, cma_id->context, cma_id); switch (event->event) { case RDMA_CM_EVENT_ADDR_RESOLVED: - ret = iser_addr_handler(cma_id); + iser_addr_handler(cma_id); break; case RDMA_CM_EVENT_ROUTE_RESOLVED: - ret = iser_route_handler(cma_id); + iser_route_handler(cma_id); break; case RDMA_CM_EVENT_ESTABLISHED: iser_connected_handler(cma_id); @@ -789,18 +780,18 @@ static int iser_cma_handler(struct rdma_cm_id *cma_id, struct rdma_cm_event *eve case RDMA_CM_EVENT_CONNECT_ERROR: case RDMA_CM_EVENT_UNREACHABLE: case RDMA_CM_EVENT_REJECTED: - ret = iser_connect_error(cma_id); + iser_connect_error(cma_id); break; case RDMA_CM_EVENT_DISCONNECTED: case RDMA_CM_EVENT_DEVICE_REMOVAL: case RDMA_CM_EVENT_ADDR_CHANGE: - ret = iser_disconnected_handler(cma_id); + iser_disconnected_handler(cma_id); break; default: iser_err("Unexpected RDMA CM event (%d)\n", event->event); break; } - return ret; + return 0; } void iser_conn_init(struct iser_conn *ib_conn) @@ -809,7 +800,7 @@ void iser_conn_init(struct iser_conn *ib_conn) init_waitqueue_head(&ib_conn->wait); ib_conn->post_recv_buf_count = 0; atomic_set(&ib_conn->post_send_buf_count, 0); - atomic_set(&ib_conn->refcount, 1); /* ref ib conn allocation */ + init_completion(&ib_conn->stop_completion); INIT_LIST_HEAD(&ib_conn->conn_list); spin_lock_init(&ib_conn->lock); } @@ -837,7 +828,6 @@ int iser_connect(struct iser_conn *ib_conn, ib_conn->state = ISER_CONN_PENDING; - iser_conn_get(ib_conn); /* ref ib conn's cma id */ ib_conn->cma_id = rdma_create_id(iser_cma_handler, (void *)ib_conn, RDMA_PS_TCP, IB_QPT_RC); @@ -874,9 +864,8 @@ id_failure: ib_conn->cma_id = NULL; addr_failure: ib_conn->state = ISER_CONN_DOWN; - iser_conn_put(ib_conn, 1); /* deref ib conn's cma id */ connect_failure: - iser_conn_put(ib_conn, 1); /* deref ib conn deallocate */ + iser_conn_release(ib_conn); return err; } diff --git a/drivers/infiniband/ulp/srp/ib_srp.c b/drivers/infiniband/ulp/srp/ib_srp.c index 66a908bf3fb9..e3c2c5b4297f 100644 --- a/drivers/infiniband/ulp/srp/ib_srp.c +++ b/drivers/infiniband/ulp/srp/ib_srp.c @@ -30,7 +30,7 @@ * SOFTWARE. */ -#define pr_fmt(fmt) PFX fmt +#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt #include <linux/module.h> #include <linux/init.h> @@ -66,6 +66,8 @@ static unsigned int srp_sg_tablesize; static unsigned int cmd_sg_entries; static unsigned int indirect_sg_entries; static bool allow_ext_sg; +static bool prefer_fr; +static bool register_always; static int topspin_workarounds = 1; module_param(srp_sg_tablesize, uint, 0444); @@ -87,6 +89,14 @@ module_param(topspin_workarounds, int, 0444); MODULE_PARM_DESC(topspin_workarounds, "Enable workarounds for Topspin/Cisco SRP target bugs if != 0"); +module_param(prefer_fr, bool, 0444); +MODULE_PARM_DESC(prefer_fr, +"Whether to use fast registration if both FMR and fast registration are supported"); + +module_param(register_always, bool, 0444); +MODULE_PARM_DESC(register_always, + "Use memory registration even for contiguous memory regions"); + static struct kernel_param_ops srp_tmo_ops; static int srp_reconnect_delay = 10; @@ -288,28 +298,174 @@ static int srp_new_cm_id(struct srp_target_port *target) return 0; } +static struct ib_fmr_pool *srp_alloc_fmr_pool(struct srp_target_port *target) +{ + struct srp_device *dev = target->srp_host->srp_dev; + struct ib_fmr_pool_param fmr_param; + + memset(&fmr_param, 0, sizeof(fmr_param)); + fmr_param.pool_size = target->scsi_host->can_queue; + fmr_param.dirty_watermark = fmr_param.pool_size / 4; + fmr_param.cache = 1; + fmr_param.max_pages_per_fmr = dev->max_pages_per_mr; + fmr_param.page_shift = ilog2(dev->mr_page_size); + fmr_param.access = (IB_ACCESS_LOCAL_WRITE | + IB_ACCESS_REMOTE_WRITE | + IB_ACCESS_REMOTE_READ); + + return ib_create_fmr_pool(dev->pd, &fmr_param); +} + +/** + * srp_destroy_fr_pool() - free the resources owned by a pool + * @pool: Fast registration pool to be destroyed. + */ +static void srp_destroy_fr_pool(struct srp_fr_pool *pool) +{ + int i; + struct srp_fr_desc *d; + + if (!pool) + return; + + for (i = 0, d = &pool->desc[0]; i < pool->size; i++, d++) { + if (d->frpl) + ib_free_fast_reg_page_list(d->frpl); + if (d->mr) + ib_dereg_mr(d->mr); + } + kfree(pool); +} + +/** + * srp_create_fr_pool() - allocate and initialize a pool for fast registration + * @device: IB device to allocate fast registration descriptors for. + * @pd: Protection domain associated with the FR descriptors. + * @pool_size: Number of descriptors to allocate. + * @max_page_list_len: Maximum fast registration work request page list length. + */ +static struct srp_fr_pool *srp_create_fr_pool(struct ib_device *device, + struct ib_pd *pd, int pool_size, + int max_page_list_len) +{ + struct srp_fr_pool *pool; + struct srp_fr_desc *d; + struct ib_mr *mr; + struct ib_fast_reg_page_list *frpl; + int i, ret = -EINVAL; + + if (pool_size <= 0) + goto err; + ret = -ENOMEM; + pool = kzalloc(sizeof(struct srp_fr_pool) + + pool_size * sizeof(struct srp_fr_desc), GFP_KERNEL); + if (!pool) + goto err; + pool->size = pool_size; + pool->max_page_list_len = max_page_list_len; + spin_lock_init(&pool->lock); + INIT_LIST_HEAD(&pool->free_list); + + for (i = 0, d = &pool->desc[0]; i < pool->size; i++, d++) { + mr = ib_alloc_fast_reg_mr(pd, max_page_list_len); + if (IS_ERR(mr)) { + ret = PTR_ERR(mr); + goto destroy_pool; + } + d->mr = mr; + frpl = ib_alloc_fast_reg_page_list(device, max_page_list_len); + if (IS_ERR(frpl)) { + ret = PTR_ERR(frpl); + goto destroy_pool; + } + d->frpl = frpl; + list_add_tail(&d->entry, &pool->free_list); + } + +out: + return pool; + +destroy_pool: + srp_destroy_fr_pool(pool); + +err: + pool = ERR_PTR(ret); + goto out; +} + +/** + * srp_fr_pool_get() - obtain a descriptor suitable for fast registration + * @pool: Pool to obtain descriptor from. + */ +static struct srp_fr_desc *srp_fr_pool_get(struct srp_fr_pool *pool) +{ + struct srp_fr_desc *d = NULL; + unsigned long flags; + + spin_lock_irqsave(&pool->lock, flags); + if (!list_empty(&pool->free_list)) { + d = list_first_entry(&pool->free_list, typeof(*d), entry); + list_del(&d->entry); + } + spin_unlock_irqrestore(&pool->lock, flags); + + return d; +} + +/** + * srp_fr_pool_put() - put an FR descriptor back in the free list + * @pool: Pool the descriptor was allocated from. + * @desc: Pointer to an array of fast registration descriptor pointers. + * @n: Number of descriptors to put back. + * + * Note: The caller must already have queued an invalidation request for + * desc->mr->rkey before calling this function. + */ +static void srp_fr_pool_put(struct srp_fr_pool *pool, struct srp_fr_desc **desc, + int n) +{ + unsigned long flags; + int i; + + spin_lock_irqsave(&pool->lock, flags); + for (i = 0; i < n; i++) + list_add(&desc[i]->entry, &pool->free_list); + spin_unlock_irqrestore(&pool->lock, flags); +} + +static struct srp_fr_pool *srp_alloc_fr_pool(struct srp_target_port *target) +{ + struct srp_device *dev = target->srp_host->srp_dev; + + return srp_create_fr_pool(dev->dev, dev->pd, + target->scsi_host->can_queue, + dev->max_pages_per_mr); +} + static int srp_create_target_ib(struct srp_target_port *target) { + struct srp_device *dev = target->srp_host->srp_dev; struct ib_qp_init_attr *init_attr; struct ib_cq *recv_cq, *send_cq; struct ib_qp *qp; + struct ib_fmr_pool *fmr_pool = NULL; + struct srp_fr_pool *fr_pool = NULL; + const int m = 1 + dev->use_fast_reg; int ret; init_attr = kzalloc(sizeof *init_attr, GFP_KERNEL); if (!init_attr) return -ENOMEM; - recv_cq = ib_create_cq(target->srp_host->srp_dev->dev, - srp_recv_completion, NULL, target, + recv_cq = ib_create_cq(dev->dev, srp_recv_completion, NULL, target, target->queue_size, target->comp_vector); if (IS_ERR(recv_cq)) { ret = PTR_ERR(recv_cq); goto err; } - send_cq = ib_create_cq(target->srp_host->srp_dev->dev, - srp_send_completion, NULL, target, - target->queue_size, target->comp_vector); + send_cq = ib_create_cq(dev->dev, srp_send_completion, NULL, target, + m * target->queue_size, target->comp_vector); if (IS_ERR(send_cq)) { ret = PTR_ERR(send_cq); goto err_recv_cq; @@ -318,16 +474,16 @@ static int srp_create_target_ib(struct srp_target_port *target) ib_req_notify_cq(recv_cq, IB_CQ_NEXT_COMP); init_attr->event_handler = srp_qp_event; - init_attr->cap.max_send_wr = target->queue_size; + init_attr->cap.max_send_wr = m * target->queue_size; init_attr->cap.max_recv_wr = target->queue_size; init_attr->cap.max_recv_sge = 1; init_attr->cap.max_send_sge = 1; - init_attr->sq_sig_type = IB_SIGNAL_ALL_WR; + init_attr->sq_sig_type = IB_SIGNAL_REQ_WR; init_attr->qp_type = IB_QPT_RC; init_attr->send_cq = send_cq; init_attr->recv_cq = recv_cq; - qp = ib_create_qp(target->srp_host->srp_dev->pd, init_attr); + qp = ib_create_qp(dev->pd, init_attr); if (IS_ERR(qp)) { ret = PTR_ERR(qp); goto err_send_cq; @@ -337,6 +493,30 @@ static int srp_create_target_ib(struct srp_target_port *target) if (ret) goto err_qp; + if (dev->use_fast_reg && dev->has_fr) { + fr_pool = srp_alloc_fr_pool(target); + if (IS_ERR(fr_pool)) { + ret = PTR_ERR(fr_pool); + shost_printk(KERN_WARNING, target->scsi_host, PFX + "FR pool allocation failed (%d)\n", ret); + goto err_qp; + } + if (target->fr_pool) + srp_destroy_fr_pool(target->fr_pool); + target->fr_pool = fr_pool; + } else if (!dev->use_fast_reg && dev->has_fmr) { + fmr_pool = srp_alloc_fmr_pool(target); + if (IS_ERR(fmr_pool)) { + ret = PTR_ERR(fmr_pool); + shost_printk(KERN_WARNING, target->scsi_host, PFX + "FMR pool allocation failed (%d)\n", ret); + goto err_qp; + } + if (target->fmr_pool) + ib_destroy_fmr_pool(target->fmr_pool); + target->fmr_pool = fmr_pool; + } + if (target->qp) ib_destroy_qp(target->qp); if (target->recv_cq) @@ -371,8 +551,16 @@ err: */ static void srp_free_target_ib(struct srp_target_port *target) { + struct srp_device *dev = target->srp_host->srp_dev; int i; + if (dev->use_fast_reg) { + if (target->fr_pool) + srp_destroy_fr_pool(target->fr_pool); + } else { + if (target->fmr_pool) + ib_destroy_fmr_pool(target->fmr_pool); + } ib_destroy_qp(target->qp); ib_destroy_cq(target->send_cq); ib_destroy_cq(target->recv_cq); @@ -577,7 +765,8 @@ static void srp_disconnect_target(struct srp_target_port *target) static void srp_free_req_data(struct srp_target_port *target) { - struct ib_device *ibdev = target->srp_host->srp_dev->dev; + struct srp_device *dev = target->srp_host->srp_dev; + struct ib_device *ibdev = dev->dev; struct srp_request *req; int i; @@ -586,7 +775,10 @@ static void srp_free_req_data(struct srp_target_port *target) for (i = 0; i < target->req_ring_size; ++i) { req = &target->req_ring[i]; - kfree(req->fmr_list); + if (dev->use_fast_reg) + kfree(req->fr_list); + else + kfree(req->fmr_list); kfree(req->map_page); if (req->indirect_dma_addr) { ib_dma_unmap_single(ibdev, req->indirect_dma_addr, @@ -605,6 +797,7 @@ static int srp_alloc_req_data(struct srp_target_port *target) struct srp_device *srp_dev = target->srp_host->srp_dev; struct ib_device *ibdev = srp_dev->dev; struct srp_request *req; + void *mr_list; dma_addr_t dma_addr; int i, ret = -ENOMEM; @@ -617,12 +810,20 @@ static int srp_alloc_req_data(struct srp_target_port *target) for (i = 0; i < target->req_ring_size; ++i) { req = &target->req_ring[i]; - req->fmr_list = kmalloc(target->cmd_sg_cnt * sizeof(void *), - GFP_KERNEL); - req->map_page = kmalloc(SRP_FMR_SIZE * sizeof(void *), - GFP_KERNEL); + mr_list = kmalloc(target->cmd_sg_cnt * sizeof(void *), + GFP_KERNEL); + if (!mr_list) + goto out; + if (srp_dev->use_fast_reg) + req->fr_list = mr_list; + else + req->fmr_list = mr_list; + req->map_page = kmalloc(srp_dev->max_pages_per_mr * + sizeof(void *), GFP_KERNEL); + if (!req->map_page) + goto out; req->indirect_desc = kmalloc(target->indirect_size, GFP_KERNEL); - if (!req->fmr_list || !req->map_page || !req->indirect_desc) + if (!req->indirect_desc) goto out; dma_addr = ib_dma_map_single(ibdev, req->indirect_desc, @@ -759,21 +960,56 @@ static int srp_connect_target(struct srp_target_port *target) } } +static int srp_inv_rkey(struct srp_target_port *target, u32 rkey) +{ + struct ib_send_wr *bad_wr; + struct ib_send_wr wr = { + .opcode = IB_WR_LOCAL_INV, + .wr_id = LOCAL_INV_WR_ID_MASK, + .next = NULL, + .num_sge = 0, + .send_flags = 0, + .ex.invalidate_rkey = rkey, + }; + + return ib_post_send(target->qp, &wr, &bad_wr); +} + static void srp_unmap_data(struct scsi_cmnd *scmnd, struct srp_target_port *target, struct srp_request *req) { - struct ib_device *ibdev = target->srp_host->srp_dev->dev; - struct ib_pool_fmr **pfmr; + struct srp_device *dev = target->srp_host->srp_dev; + struct ib_device *ibdev = dev->dev; + int i, res; if (!scsi_sglist(scmnd) || (scmnd->sc_data_direction != DMA_TO_DEVICE && scmnd->sc_data_direction != DMA_FROM_DEVICE)) return; - pfmr = req->fmr_list; - while (req->nfmr--) - ib_fmr_pool_unmap(*pfmr++); + if (dev->use_fast_reg) { + struct srp_fr_desc **pfr; + + for (i = req->nmdesc, pfr = req->fr_list; i > 0; i--, pfr++) { + res = srp_inv_rkey(target, (*pfr)->mr->rkey); + if (res < 0) { + shost_printk(KERN_ERR, target->scsi_host, PFX + "Queueing INV WR for rkey %#x failed (%d)\n", + (*pfr)->mr->rkey, res); + queue_work(system_long_wq, + &target->tl_err_work); + } + } + if (req->nmdesc) + srp_fr_pool_put(target->fr_pool, req->fr_list, + req->nmdesc); + } else { + struct ib_pool_fmr **pfmr; + + for (i = req->nmdesc, pfmr = req->fmr_list; i > 0; i--, pfmr++) + ib_fmr_pool_unmap(*pfmr); + } ib_dma_unmap_sg(ibdev, scsi_sglist(scmnd), scsi_sg_count(scmnd), scmnd->sc_data_direction); @@ -813,6 +1049,10 @@ static struct scsi_cmnd *srp_claim_req(struct srp_target_port *target, /** * srp_free_req() - Unmap data and add request to the free request list. + * @target: SRP target port. + * @req: Request to be freed. + * @scmnd: SCSI command associated with @req. + * @req_lim_delta: Amount to be added to @target->req_lim. */ static void srp_free_req(struct srp_target_port *target, struct srp_request *req, struct scsi_cmnd *scmnd, @@ -882,21 +1122,19 @@ static int srp_rport_reconnect(struct srp_rport *rport) * callbacks will have finished before a new QP is allocated. */ ret = srp_new_cm_id(target); - /* - * Whether or not creating a new CM ID succeeded, create a new - * QP. This guarantees that all completion callback function - * invocations have finished before request resetting starts. - */ - if (ret == 0) - ret = srp_create_target_ib(target); - else - srp_create_target_ib(target); for (i = 0; i < target->req_ring_size; ++i) { struct srp_request *req = &target->req_ring[i]; srp_finish_req(target, req, NULL, DID_RESET << 16); } + /* + * Whether or not creating a new CM ID succeeded, create a new + * QP. This guarantees that all callback functions for the old QP have + * finished before any send requests are posted on the new QP. + */ + ret += srp_create_target_ib(target); + INIT_LIST_HEAD(&target->free_tx); for (i = 0; i < target->queue_size; ++i) list_add(&target->tx_ring[i]->list, &target->free_tx); @@ -928,33 +1166,87 @@ static void srp_map_desc(struct srp_map_state *state, dma_addr_t dma_addr, static int srp_map_finish_fmr(struct srp_map_state *state, struct srp_target_port *target) { - struct srp_device *dev = target->srp_host->srp_dev; struct ib_pool_fmr *fmr; u64 io_addr = 0; - if (!state->npages) - return 0; - - if (state->npages == 1) { - srp_map_desc(state, state->base_dma_addr, state->fmr_len, - target->rkey); - state->npages = state->fmr_len = 0; - return 0; - } - - fmr = ib_fmr_pool_map_phys(dev->fmr_pool, state->pages, + fmr = ib_fmr_pool_map_phys(target->fmr_pool, state->pages, state->npages, io_addr); if (IS_ERR(fmr)) return PTR_ERR(fmr); *state->next_fmr++ = fmr; - state->nfmr++; + state->nmdesc++; + + srp_map_desc(state, 0, state->dma_len, fmr->fmr->rkey); - srp_map_desc(state, 0, state->fmr_len, fmr->fmr->rkey); - state->npages = state->fmr_len = 0; return 0; } +static int srp_map_finish_fr(struct srp_map_state *state, + struct srp_target_port *target) +{ + struct srp_device *dev = target->srp_host->srp_dev; + struct ib_send_wr *bad_wr; + struct ib_send_wr wr; + struct srp_fr_desc *desc; + u32 rkey; + + desc = srp_fr_pool_get(target->fr_pool); + if (!desc) + return -ENOMEM; + + rkey = ib_inc_rkey(desc->mr->rkey); + ib_update_fast_reg_key(desc->mr, rkey); + + memcpy(desc->frpl->page_list, state->pages, + sizeof(state->pages[0]) * state->npages); + + memset(&wr, 0, sizeof(wr)); + wr.opcode = IB_WR_FAST_REG_MR; + wr.wr_id = FAST_REG_WR_ID_MASK; + wr.wr.fast_reg.iova_start = state->base_dma_addr; + wr.wr.fast_reg.page_list = desc->frpl; + wr.wr.fast_reg.page_list_len = state->npages; + wr.wr.fast_reg.page_shift = ilog2(dev->mr_page_size); + wr.wr.fast_reg.length = state->dma_len; + wr.wr.fast_reg.access_flags = (IB_ACCESS_LOCAL_WRITE | + IB_ACCESS_REMOTE_READ | + IB_ACCESS_REMOTE_WRITE); + wr.wr.fast_reg.rkey = desc->mr->lkey; + + *state->next_fr++ = desc; + state->nmdesc++; + + srp_map_desc(state, state->base_dma_addr, state->dma_len, + desc->mr->rkey); + + return ib_post_send(target->qp, &wr, &bad_wr); +} + +static int srp_finish_mapping(struct srp_map_state *state, + struct srp_target_port *target) +{ + int ret = 0; + + if (state->npages == 0) + return 0; + + if (state->npages == 1 && !register_always) + srp_map_desc(state, state->base_dma_addr, state->dma_len, + target->rkey); + else + ret = target->srp_host->srp_dev->use_fast_reg ? + srp_map_finish_fr(state, target) : + srp_map_finish_fmr(state, target); + + if (ret == 0) { + state->npages = 0; + state->dma_len = 0; + } + + return ret; +} + static void srp_map_update_start(struct srp_map_state *state, struct scatterlist *sg, int sg_index, dma_addr_t dma_addr) @@ -967,7 +1259,7 @@ static void srp_map_update_start(struct srp_map_state *state, static int srp_map_sg_entry(struct srp_map_state *state, struct srp_target_port *target, struct scatterlist *sg, int sg_index, - int use_fmr) + bool use_mr) { struct srp_device *dev = target->srp_host->srp_dev; struct ib_device *ibdev = dev->dev; @@ -979,23 +1271,25 @@ static int srp_map_sg_entry(struct srp_map_state *state, if (!dma_len) return 0; - if (use_fmr == SRP_MAP_NO_FMR) { - /* Once we're in direct map mode for a request, we don't - * go back to FMR mode, so no need to update anything + if (!use_mr) { + /* + * Once we're in direct map mode for a request, we don't + * go back to FMR or FR mode, so no need to update anything * other than the descriptor. */ srp_map_desc(state, dma_addr, dma_len, target->rkey); return 0; } - /* If we start at an offset into the FMR page, don't merge into - * the current FMR. Finish it out, and use the kernel's MR for this - * sg entry. This is to avoid potential bugs on some SRP targets - * that were never quite defined, but went away when the initiator - * avoided using FMR on such page fragments. + /* + * Since not all RDMA HW drivers support non-zero page offsets for + * FMR, if we start at an offset into a page, don't merge into the + * current FMR mapping. Finish it out, and use the kernel's MR for + * this sg entry. */ - if (dma_addr & ~dev->fmr_page_mask || dma_len > dev->fmr_max_size) { - ret = srp_map_finish_fmr(state, target); + if ((!dev->use_fast_reg && dma_addr & ~dev->mr_page_mask) || + dma_len > dev->mr_max_size) { + ret = srp_finish_mapping(state, target); if (ret) return ret; @@ -1004,52 +1298,106 @@ static int srp_map_sg_entry(struct srp_map_state *state, return 0; } - /* If this is the first sg to go into the FMR, save our position. - * We need to know the first unmapped entry, its index, and the - * first unmapped address within that entry to be able to restart - * mapping after an error. + /* + * If this is the first sg that will be mapped via FMR or via FR, save + * our position. We need to know the first unmapped entry, its index, + * and the first unmapped address within that entry to be able to + * restart mapping after an error. */ if (!state->unmapped_sg) srp_map_update_start(state, sg, sg_index, dma_addr); while (dma_len) { - if (state->npages == SRP_FMR_SIZE) { - ret = srp_map_finish_fmr(state, target); + unsigned offset = dma_addr & ~dev->mr_page_mask; + if (state->npages == dev->max_pages_per_mr || offset != 0) { + ret = srp_finish_mapping(state, target); if (ret) return ret; srp_map_update_start(state, sg, sg_index, dma_addr); } - len = min_t(unsigned int, dma_len, dev->fmr_page_size); + len = min_t(unsigned int, dma_len, dev->mr_page_size - offset); if (!state->npages) state->base_dma_addr = dma_addr; - state->pages[state->npages++] = dma_addr; - state->fmr_len += len; + state->pages[state->npages++] = dma_addr & dev->mr_page_mask; + state->dma_len += len; dma_addr += len; dma_len -= len; } - /* If the last entry of the FMR wasn't a full page, then we need to + /* + * If the last entry of the MR wasn't a full page, then we need to * close it out and start a new one -- we can only merge at page * boundries. */ ret = 0; - if (len != dev->fmr_page_size) { - ret = srp_map_finish_fmr(state, target); + if (len != dev->mr_page_size) { + ret = srp_finish_mapping(state, target); if (!ret) srp_map_update_start(state, NULL, 0, 0); } return ret; } +static int srp_map_sg(struct srp_map_state *state, + struct srp_target_port *target, struct srp_request *req, + struct scatterlist *scat, int count) +{ + struct srp_device *dev = target->srp_host->srp_dev; + struct ib_device *ibdev = dev->dev; + struct scatterlist *sg; + int i; + bool use_mr; + + state->desc = req->indirect_desc; + state->pages = req->map_page; + if (dev->use_fast_reg) { + state->next_fr = req->fr_list; + use_mr = !!target->fr_pool; + } else { + state->next_fmr = req->fmr_list; + use_mr = !!target->fmr_pool; + } + + for_each_sg(scat, sg, count, i) { + if (srp_map_sg_entry(state, target, sg, i, use_mr)) { + /* + * Memory registration failed, so backtrack to the + * first unmapped entry and continue on without using + * memory registration. + */ + dma_addr_t dma_addr; + unsigned int dma_len; + +backtrack: + sg = state->unmapped_sg; + i = state->unmapped_index; + + dma_addr = ib_sg_dma_address(ibdev, sg); + dma_len = ib_sg_dma_len(ibdev, sg); + dma_len -= (state->unmapped_addr - dma_addr); + dma_addr = state->unmapped_addr; + use_mr = false; + srp_map_desc(state, dma_addr, dma_len, target->rkey); + } + } + + if (use_mr && srp_finish_mapping(state, target)) + goto backtrack; + + req->nmdesc = state->nmdesc; + + return 0; +} + static int srp_map_data(struct scsi_cmnd *scmnd, struct srp_target_port *target, struct srp_request *req) { - struct scatterlist *scat, *sg; + struct scatterlist *scat; struct srp_cmd *cmd = req->cmd->buf; - int i, len, nents, count, use_fmr; + int len, nents, count; struct srp_device *dev; struct ib_device *ibdev; struct srp_map_state state; @@ -1081,7 +1429,7 @@ static int srp_map_data(struct scsi_cmnd *scmnd, struct srp_target_port *target, fmt = SRP_DATA_DESC_DIRECT; len = sizeof (struct srp_cmd) + sizeof (struct srp_direct_buf); - if (count == 1) { + if (count == 1 && !register_always) { /* * The midlayer only generated a single gather/scatter * entry, or DMA mapping coalesced everything to a @@ -1094,13 +1442,13 @@ static int srp_map_data(struct scsi_cmnd *scmnd, struct srp_target_port *target, buf->key = cpu_to_be32(target->rkey); buf->len = cpu_to_be32(ib_sg_dma_len(ibdev, scat)); - req->nfmr = 0; + req->nmdesc = 0; goto map_complete; } - /* We have more than one scatter/gather entry, so build our indirect - * descriptor table, trying to merge as many entries with FMR as we - * can. + /* + * We have more than one scatter/gather entry, so build our indirect + * descriptor table, trying to merge as many entries as we can. */ indirect_hdr = (void *) cmd->add_data; @@ -1108,35 +1456,7 @@ static int srp_map_data(struct scsi_cmnd *scmnd, struct srp_target_port *target, target->indirect_size, DMA_TO_DEVICE); memset(&state, 0, sizeof(state)); - state.desc = req->indirect_desc; - state.pages = req->map_page; - state.next_fmr = req->fmr_list; - - use_fmr = dev->fmr_pool ? SRP_MAP_ALLOW_FMR : SRP_MAP_NO_FMR; - - for_each_sg(scat, sg, count, i) { - if (srp_map_sg_entry(&state, target, sg, i, use_fmr)) { - /* FMR mapping failed, so backtrack to the first - * unmapped entry and continue on without using FMR. - */ - dma_addr_t dma_addr; - unsigned int dma_len; - -backtrack: - sg = state.unmapped_sg; - i = state.unmapped_index; - - dma_addr = ib_sg_dma_address(ibdev, sg); - dma_len = ib_sg_dma_len(ibdev, sg); - dma_len -= (state.unmapped_addr - dma_addr); - dma_addr = state.unmapped_addr; - use_fmr = SRP_MAP_NO_FMR; - srp_map_desc(&state, dma_addr, dma_len, target->rkey); - } - } - - if (use_fmr == SRP_MAP_ALLOW_FMR && srp_map_finish_fmr(&state, target)) - goto backtrack; + srp_map_sg(&state, target, req, scat, count); /* We've mapped the request, now pull as much of the indirect * descriptor table as we can into the command buffer. If this @@ -1144,9 +1464,9 @@ backtrack: * guaranteed to fit into the command, as the SCSI layer won't * give us more S/G entries than we allow. */ - req->nfmr = state.nfmr; if (state.ndesc == 1) { - /* FMR mapping was able to collapse this to one entry, + /* + * Memory registration collapsed the sg-list into one entry, * so use a direct descriptor. */ struct srp_direct_buf *buf = (void *) cmd->add_data; @@ -1455,6 +1775,7 @@ static void srp_handle_recv(struct srp_target_port *target, struct ib_wc *wc) /** * srp_tl_err_work() - handle a transport layer error + * @work: Work structure embedded in an SRP target port. * * Note: This function may get invoked before the rport has been created, * hence the target->rport test. @@ -1468,14 +1789,24 @@ static void srp_tl_err_work(struct work_struct *work) srp_start_tl_fail_timers(target->rport); } -static void srp_handle_qp_err(enum ib_wc_status wc_status, bool send_err, - struct srp_target_port *target) +static void srp_handle_qp_err(u64 wr_id, enum ib_wc_status wc_status, + bool send_err, struct srp_target_port *target) { if (target->connected && !target->qp_in_error) { - shost_printk(KERN_ERR, target->scsi_host, - PFX "failed %s status %d\n", - send_err ? "send" : "receive", - wc_status); + if (wr_id & LOCAL_INV_WR_ID_MASK) { + shost_printk(KERN_ERR, target->scsi_host, PFX + "LOCAL_INV failed with status %d\n", + wc_status); + } else if (wr_id & FAST_REG_WR_ID_MASK) { + shost_printk(KERN_ERR, target->scsi_host, PFX + "FAST_REG_MR failed status %d\n", + wc_status); + } else { + shost_printk(KERN_ERR, target->scsi_host, + PFX "failed %s status %d for iu %p\n", + send_err ? "send" : "receive", + wc_status, (void *)(uintptr_t)wr_id); + } queue_work(system_long_wq, &target->tl_err_work); } target->qp_in_error = true; @@ -1491,7 +1822,7 @@ static void srp_recv_completion(struct ib_cq *cq, void *target_ptr) if (likely(wc.status == IB_WC_SUCCESS)) { srp_handle_recv(target, &wc); } else { - srp_handle_qp_err(wc.status, false, target); + srp_handle_qp_err(wc.wr_id, wc.status, false, target); } } } @@ -1507,7 +1838,7 @@ static void srp_send_completion(struct ib_cq *cq, void *target_ptr) iu = (struct srp_iu *) (uintptr_t) wc.wr_id; list_add(&iu->list, &target->free_tx); } else { - srp_handle_qp_err(wc.status, true, target); + srp_handle_qp_err(wc.wr_id, wc.status, true, target); } } } @@ -1521,7 +1852,7 @@ static int srp_queuecommand(struct Scsi_Host *shost, struct scsi_cmnd *scmnd) struct srp_cmd *cmd; struct ib_device *dev; unsigned long flags; - int len, result; + int len, ret; const bool in_scsi_eh = !in_interrupt() && current == shost->ehandler; /* @@ -1533,12 +1864,9 @@ static int srp_queuecommand(struct Scsi_Host *shost, struct scsi_cmnd *scmnd) if (in_scsi_eh) mutex_lock(&rport->mutex); - result = srp_chkready(target->rport); - if (unlikely(result)) { - scmnd->result = result; - scmnd->scsi_done(scmnd); - goto unlock_rport; - } + scmnd->result = srp_chkready(target->rport); + if (unlikely(scmnd->result)) + goto err; spin_lock_irqsave(&target->lock, flags); iu = __srp_get_tx_iu(target, SRP_IU_CMD); @@ -1553,7 +1881,6 @@ static int srp_queuecommand(struct Scsi_Host *shost, struct scsi_cmnd *scmnd) ib_dma_sync_single_for_cpu(dev, iu->dma, target->max_iu_len, DMA_TO_DEVICE); - scmnd->result = 0; scmnd->host_scribble = (void *) req; cmd = iu->buf; @@ -1570,7 +1897,15 @@ static int srp_queuecommand(struct Scsi_Host *shost, struct scsi_cmnd *scmnd) len = srp_map_data(scmnd, target, req); if (len < 0) { shost_printk(KERN_ERR, target->scsi_host, - PFX "Failed to map data\n"); + PFX "Failed to map data (%d)\n", len); + /* + * If we ran out of memory descriptors (-ENOMEM) because an + * application is queuing many requests with more than + * max_pages_per_mr sg-list elements, tell the SCSI mid-layer + * to reduce queue depth temporarily. + */ + scmnd->result = len == -ENOMEM ? + DID_OK << 16 | QUEUE_FULL << 1 : DID_ERROR << 16; goto err_iu; } @@ -1582,11 +1917,13 @@ static int srp_queuecommand(struct Scsi_Host *shost, struct scsi_cmnd *scmnd) goto err_unmap; } + ret = 0; + unlock_rport: if (in_scsi_eh) mutex_unlock(&rport->mutex); - return 0; + return ret; err_unmap: srp_unmap_data(scmnd, target, req); @@ -1594,16 +1931,27 @@ err_unmap: err_iu: srp_put_tx_iu(target, iu, SRP_IU_CMD); + /* + * Avoid that the loops that iterate over the request ring can + * encounter a dangling SCSI command pointer. + */ + req->scmnd = NULL; + spin_lock_irqsave(&target->lock, flags); list_add(&req->list, &target->free_reqs); err_unlock: spin_unlock_irqrestore(&target->lock, flags); - if (in_scsi_eh) - mutex_unlock(&rport->mutex); +err: + if (scmnd->result) { + scmnd->scsi_done(scmnd); + ret = 0; + } else { + ret = SCSI_MLQUEUE_HOST_BUSY; + } - return SCSI_MLQUEUE_HOST_BUSY; + goto unlock_rport; } /* @@ -2310,6 +2658,8 @@ static struct class srp_class = { /** * srp_conn_unique() - check whether the connection to a target is unique + * @host: SRP host. + * @target: SRP target port. */ static bool srp_conn_unique(struct srp_host *host, struct srp_target_port *target) @@ -2605,7 +2955,8 @@ static ssize_t srp_create_target(struct device *dev, container_of(dev, struct srp_host, dev); struct Scsi_Host *target_host; struct srp_target_port *target; - struct ib_device *ibdev = host->srp_dev->dev; + struct srp_device *srp_dev = host->srp_dev; + struct ib_device *ibdev = srp_dev->dev; int ret; target_host = scsi_host_alloc(&srp_template, @@ -2650,9 +3001,9 @@ static ssize_t srp_create_target(struct device *dev, goto err; } - if (!host->srp_dev->fmr_pool && !target->allow_ext_sg && - target->cmd_sg_cnt < target->sg_tablesize) { - pr_warn("No FMR pool and no external indirect descriptors, limiting sg_tablesize to cmd_sg_cnt\n"); + if (!srp_dev->has_fmr && !srp_dev->has_fr && !target->allow_ext_sg && + target->cmd_sg_cnt < target->sg_tablesize) { + pr_warn("No MR pool and no external indirect descriptors, limiting sg_tablesize to cmd_sg_cnt\n"); target->sg_tablesize = target->cmd_sg_cnt; } @@ -2790,9 +3141,9 @@ static void srp_add_one(struct ib_device *device) { struct srp_device *srp_dev; struct ib_device_attr *dev_attr; - struct ib_fmr_pool_param fmr_param; struct srp_host *host; - int max_pages_per_fmr, fmr_page_shift, s, e, p; + int mr_page_shift, s, e, p; + u64 max_pages_per_mr; dev_attr = kmalloc(sizeof *dev_attr, GFP_KERNEL); if (!dev_attr) @@ -2807,15 +3158,39 @@ static void srp_add_one(struct ib_device *device) if (!srp_dev) goto free_attr; + srp_dev->has_fmr = (device->alloc_fmr && device->dealloc_fmr && + device->map_phys_fmr && device->unmap_fmr); + srp_dev->has_fr = (dev_attr->device_cap_flags & + IB_DEVICE_MEM_MGT_EXTENSIONS); + if (!srp_dev->has_fmr && !srp_dev->has_fr) + dev_warn(&device->dev, "neither FMR nor FR is supported\n"); + + srp_dev->use_fast_reg = (srp_dev->has_fr && + (!srp_dev->has_fmr || prefer_fr)); + /* * Use the smallest page size supported by the HCA, down to a * minimum of 4096 bytes. We're unlikely to build large sglists * out of smaller entries. */ - fmr_page_shift = max(12, ffs(dev_attr->page_size_cap) - 1); - srp_dev->fmr_page_size = 1 << fmr_page_shift; - srp_dev->fmr_page_mask = ~((u64) srp_dev->fmr_page_size - 1); - srp_dev->fmr_max_size = srp_dev->fmr_page_size * SRP_FMR_SIZE; + mr_page_shift = max(12, ffs(dev_attr->page_size_cap) - 1); + srp_dev->mr_page_size = 1 << mr_page_shift; + srp_dev->mr_page_mask = ~((u64) srp_dev->mr_page_size - 1); + max_pages_per_mr = dev_attr->max_mr_size; + do_div(max_pages_per_mr, srp_dev->mr_page_size); + srp_dev->max_pages_per_mr = min_t(u64, SRP_MAX_PAGES_PER_MR, + max_pages_per_mr); + if (srp_dev->use_fast_reg) { + srp_dev->max_pages_per_mr = + min_t(u32, srp_dev->max_pages_per_mr, + dev_attr->max_fast_reg_page_list_len); + } + srp_dev->mr_max_size = srp_dev->mr_page_size * + srp_dev->max_pages_per_mr; + pr_debug("%s: mr_page_shift = %d, dev_attr->max_mr_size = %#llx, dev_attr->max_fast_reg_page_list_len = %u, max_pages_per_mr = %d, mr_max_size = %#x\n", + device->name, mr_page_shift, dev_attr->max_mr_size, + dev_attr->max_fast_reg_page_list_len, + srp_dev->max_pages_per_mr, srp_dev->mr_max_size); INIT_LIST_HEAD(&srp_dev->dev_list); @@ -2831,27 +3206,6 @@ static void srp_add_one(struct ib_device *device) if (IS_ERR(srp_dev->mr)) goto err_pd; - for (max_pages_per_fmr = SRP_FMR_SIZE; - max_pages_per_fmr >= SRP_FMR_MIN_SIZE; - max_pages_per_fmr /= 2, srp_dev->fmr_max_size /= 2) { - memset(&fmr_param, 0, sizeof fmr_param); - fmr_param.pool_size = SRP_FMR_POOL_SIZE; - fmr_param.dirty_watermark = SRP_FMR_DIRTY_SIZE; - fmr_param.cache = 1; - fmr_param.max_pages_per_fmr = max_pages_per_fmr; - fmr_param.page_shift = fmr_page_shift; - fmr_param.access = (IB_ACCESS_LOCAL_WRITE | - IB_ACCESS_REMOTE_WRITE | - IB_ACCESS_REMOTE_READ); - - srp_dev->fmr_pool = ib_create_fmr_pool(srp_dev->pd, &fmr_param); - if (!IS_ERR(srp_dev->fmr_pool)) - break; - } - - if (IS_ERR(srp_dev->fmr_pool)) - srp_dev->fmr_pool = NULL; - if (device->node_type == RDMA_NODE_IB_SWITCH) { s = 0; e = 0; @@ -2914,8 +3268,6 @@ static void srp_remove_one(struct ib_device *device) kfree(host); } - if (srp_dev->fmr_pool) - ib_destroy_fmr_pool(srp_dev->fmr_pool); ib_dereg_mr(srp_dev->mr); ib_dealloc_pd(srp_dev->pd); diff --git a/drivers/infiniband/ulp/srp/ib_srp.h b/drivers/infiniband/ulp/srp/ib_srp.h index aad27b7b4a46..e46ecb15aa0d 100644 --- a/drivers/infiniband/ulp/srp/ib_srp.h +++ b/drivers/infiniband/ulp/srp/ib_srp.h @@ -66,13 +66,10 @@ enum { SRP_TAG_NO_REQ = ~0U, SRP_TAG_TSK_MGMT = 1U << 31, - SRP_FMR_SIZE = 512, - SRP_FMR_MIN_SIZE = 128, - SRP_FMR_POOL_SIZE = 1024, - SRP_FMR_DIRTY_SIZE = SRP_FMR_POOL_SIZE / 4, + SRP_MAX_PAGES_PER_MR = 512, - SRP_MAP_ALLOW_FMR = 0, - SRP_MAP_NO_FMR = 1, + LOCAL_INV_WR_ID_MASK = 1, + FAST_REG_WR_ID_MASK = 2, }; enum srp_target_state { @@ -86,15 +83,24 @@ enum srp_iu_type { SRP_IU_RSP, }; +/* + * @mr_page_mask: HCA memory registration page mask. + * @mr_page_size: HCA memory registration page size. + * @mr_max_size: Maximum size in bytes of a single FMR / FR registration + * request. + */ struct srp_device { struct list_head dev_list; struct ib_device *dev; struct ib_pd *pd; struct ib_mr *mr; - struct ib_fmr_pool *fmr_pool; - u64 fmr_page_mask; - int fmr_page_size; - int fmr_max_size; + u64 mr_page_mask; + int mr_page_size; + int mr_max_size; + int max_pages_per_mr; + bool has_fmr; + bool has_fr; + bool use_fast_reg; }; struct srp_host { @@ -112,11 +118,14 @@ struct srp_request { struct list_head list; struct scsi_cmnd *scmnd; struct srp_iu *cmd; - struct ib_pool_fmr **fmr_list; + union { + struct ib_pool_fmr **fmr_list; + struct srp_fr_desc **fr_list; + }; u64 *map_page; struct srp_direct_buf *indirect_desc; dma_addr_t indirect_dma_addr; - short nfmr; + short nmdesc; short index; }; @@ -131,6 +140,10 @@ struct srp_target_port { struct ib_cq *send_cq ____cacheline_aligned_in_smp; struct ib_cq *recv_cq; struct ib_qp *qp; + union { + struct ib_fmr_pool *fmr_pool; + struct srp_fr_pool *fr_pool; + }; u32 lkey; u32 rkey; enum srp_target_state state; @@ -197,15 +210,66 @@ struct srp_iu { enum dma_data_direction direction; }; +/** + * struct srp_fr_desc - fast registration work request arguments + * @entry: Entry in srp_fr_pool.free_list. + * @mr: Memory region. + * @frpl: Fast registration page list. + */ +struct srp_fr_desc { + struct list_head entry; + struct ib_mr *mr; + struct ib_fast_reg_page_list *frpl; +}; + +/** + * struct srp_fr_pool - pool of fast registration descriptors + * + * An entry is available for allocation if and only if it occurs in @free_list. + * + * @size: Number of descriptors in this pool. + * @max_page_list_len: Maximum fast registration work request page list length. + * @lock: Protects free_list. + * @free_list: List of free descriptors. + * @desc: Fast registration descriptor pool. + */ +struct srp_fr_pool { + int size; + int max_page_list_len; + spinlock_t lock; + struct list_head free_list; + struct srp_fr_desc desc[0]; +}; + +/** + * struct srp_map_state - per-request DMA memory mapping state + * @desc: Pointer to the element of the SRP buffer descriptor array + * that is being filled in. + * @pages: Array with DMA addresses of pages being considered for + * memory registration. + * @base_dma_addr: DMA address of the first page that has not yet been mapped. + * @dma_len: Number of bytes that will be registered with the next + * FMR or FR memory registration call. + * @total_len: Total number of bytes in the sg-list being mapped. + * @npages: Number of page addresses in the pages[] array. + * @nmdesc: Number of FMR or FR memory descriptors used for mapping. + * @ndesc: Number of SRP buffer descriptors that have been filled in. + * @unmapped_sg: First element of the sg-list that is mapped via FMR or FR. + * @unmapped_index: Index of the first element mapped via FMR or FR. + * @unmapped_addr: DMA address of the first element mapped via FMR or FR. + */ struct srp_map_state { - struct ib_pool_fmr **next_fmr; + union { + struct ib_pool_fmr **next_fmr; + struct srp_fr_desc **next_fr; + }; struct srp_direct_buf *desc; u64 *pages; dma_addr_t base_dma_addr; - u32 fmr_len; + u32 dma_len; u32 total_len; unsigned int npages; - unsigned int nfmr; + unsigned int nmdesc; unsigned int ndesc; struct scatterlist *unmapped_sg; int unmapped_index; diff --git a/drivers/input/evdev.c b/drivers/input/evdev.c index ce953d895f5b..fd325ec9f064 100644 --- a/drivers/input/evdev.c +++ b/drivers/input/evdev.c @@ -629,12 +629,10 @@ static int str_to_user(const char *str, unsigned int maxlen, void __user *p) return copy_to_user(p, str, len) ? -EFAULT : len; } -#define OLD_KEY_MAX 0x1ff static int handle_eviocgbit(struct input_dev *dev, unsigned int type, unsigned int size, void __user *p, int compat_mode) { - static unsigned long keymax_warn_time; unsigned long *bits; int len; @@ -652,24 +650,8 @@ static int handle_eviocgbit(struct input_dev *dev, default: return -EINVAL; } - /* - * Work around bugs in userspace programs that like to do - * EVIOCGBIT(EV_KEY, KEY_MAX) and not realize that 'len' - * should be in bytes, not in bits. - */ - if (type == EV_KEY && size == OLD_KEY_MAX) { - len = OLD_KEY_MAX; - if (printk_timed_ratelimit(&keymax_warn_time, 10 * 1000)) - pr_warning("(EVIOCGBIT): Suspicious buffer size %u, " - "limiting output to %zu bytes. See " - "http://userweb.kernel.org/~dtor/eviocgbit-bug.html\n", - OLD_KEY_MAX, - BITS_TO_LONGS(OLD_KEY_MAX) * sizeof(long)); - } - return bits_to_user(bits, len, size, p, compat_mode); } -#undef OLD_KEY_MAX static int evdev_handle_get_keycode(struct input_dev *dev, void __user *p) { diff --git a/drivers/input/input-polldev.c b/drivers/input/input-polldev.c index 7f161d93203c..3664f81655ca 100644 --- a/drivers/input/input-polldev.c +++ b/drivers/input/input-polldev.c @@ -147,6 +147,11 @@ static struct attribute_group input_polldev_attribute_group = { .attrs = sysfs_attrs }; +static const struct attribute_group *input_polldev_attribute_groups[] = { + &input_polldev_attribute_group, + NULL +}; + /** * input_allocate_polled_device - allocate memory for polled device * @@ -171,6 +176,91 @@ struct input_polled_dev *input_allocate_polled_device(void) } EXPORT_SYMBOL(input_allocate_polled_device); +struct input_polled_devres { + struct input_polled_dev *polldev; +}; + +static int devm_input_polldev_match(struct device *dev, void *res, void *data) +{ + struct input_polled_devres *devres = res; + + return devres->polldev == data; +} + +static void devm_input_polldev_release(struct device *dev, void *res) +{ + struct input_polled_devres *devres = res; + struct input_polled_dev *polldev = devres->polldev; + + dev_dbg(dev, "%s: dropping reference/freeing %s\n", + __func__, dev_name(&polldev->input->dev)); + + input_put_device(polldev->input); + kfree(polldev); +} + +static void devm_input_polldev_unregister(struct device *dev, void *res) +{ + struct input_polled_devres *devres = res; + struct input_polled_dev *polldev = devres->polldev; + + dev_dbg(dev, "%s: unregistering device %s\n", + __func__, dev_name(&polldev->input->dev)); + input_unregister_device(polldev->input); + + /* + * Note that we are still holding extra reference to the input + * device so it will stick around until devm_input_polldev_release() + * is called. + */ +} + +/** + * devm_input_allocate_polled_device - allocate managed polled device + * @dev: device owning the polled device being created + * + * Returns prepared &struct input_polled_dev or %NULL. + * + * Managed polled input devices do not need to be explicitly unregistered + * or freed as it will be done automatically when owner device unbinds + * from * its driver (or binding fails). Once such managed polled device + * is allocated, it is ready to be set up and registered in the same + * fashion as regular polled input devices (using + * input_register_polled_device() function). + * + * If you want to manually unregister and free such managed polled devices, + * it can be still done by calling input_unregister_polled_device() and + * input_free_polled_device(), although it is rarely needed. + * + * NOTE: the owner device is set up as parent of input device and users + * should not override it. + */ +struct input_polled_dev *devm_input_allocate_polled_device(struct device *dev) +{ + struct input_polled_dev *polldev; + struct input_polled_devres *devres; + + devres = devres_alloc(devm_input_polldev_release, sizeof(*devres), + GFP_KERNEL); + if (!devres) + return NULL; + + polldev = input_allocate_polled_device(); + if (!polldev) { + devres_free(devres); + return NULL; + } + + polldev->input->dev.parent = dev; + polldev->devres_managed = true; + + devres->polldev = polldev; + devres_add(dev, devres); + + return polldev; +} +EXPORT_SYMBOL(devm_input_allocate_polled_device); + /** * input_free_polled_device - free memory allocated for polled device * @dev: device to free @@ -181,7 +271,12 @@ EXPORT_SYMBOL(input_allocate_polled_device); void input_free_polled_device(struct input_polled_dev *dev) { if (dev) { - input_free_device(dev->input); + if (dev->devres_managed) + WARN_ON(devres_destroy(dev->input->dev.parent, + devm_input_polldev_release, + devm_input_polldev_match, + dev)); + input_put_device(dev->input); kfree(dev); } } @@ -199,26 +294,35 @@ EXPORT_SYMBOL(input_free_polled_device); */ int input_register_polled_device(struct input_polled_dev *dev) { + struct input_polled_devres *devres = NULL; struct input_dev *input = dev->input; int error; + if (dev->devres_managed) { + devres = devres_alloc(devm_input_polldev_unregister, + sizeof(*devres), GFP_KERNEL); + if (!devres) + return -ENOMEM; + + devres->polldev = dev; + } + input_set_drvdata(input, dev); INIT_DELAYED_WORK(&dev->work, input_polled_device_work); + if (!dev->poll_interval) dev->poll_interval = 500; if (!dev->poll_interval_max) dev->poll_interval_max = dev->poll_interval; + input->open = input_open_polled_device; input->close = input_close_polled_device; - error = input_register_device(input); - if (error) - return error; + input->dev.groups = input_polldev_attribute_groups; - error = sysfs_create_group(&input->dev.kobj, - &input_polldev_attribute_group); + error = input_register_device(input); if (error) { - input_unregister_device(input); + devres_free(devres); return error; } @@ -231,6 +335,12 @@ int input_register_polled_device(struct input_polled_dev *dev) */ input_get_device(input); + if (dev->devres_managed) { + dev_dbg(input->dev.parent, "%s: registering %s with devres.\n", + __func__, dev_name(&input->dev)); + devres_add(input->dev.parent, devres); + } + return 0; } EXPORT_SYMBOL(input_register_polled_device); @@ -245,8 +355,11 @@ EXPORT_SYMBOL(input_register_polled_device); */ void input_unregister_polled_device(struct input_polled_dev *dev) { - sysfs_remove_group(&dev->input->dev.kobj, - &input_polldev_attribute_group); + if (dev->devres_managed) + WARN_ON(devres_destroy(dev->input->dev.parent, + devm_input_polldev_unregister, + devm_input_polldev_match, + dev)); input_unregister_device(dev->input); } diff --git a/drivers/input/keyboard/Kconfig b/drivers/input/keyboard/Kconfig index ffc7ad3a2c88..f7e79b481349 100644 --- a/drivers/input/keyboard/Kconfig +++ b/drivers/input/keyboard/Kconfig @@ -524,6 +524,17 @@ config KEYBOARD_STOWAWAY To compile this driver as a module, choose M here: the module will be called stowaway. +config KEYBOARD_ST_KEYSCAN + tristate "STMicroelectronics keyscan support" + depends on ARCH_STI || COMPILE_TEST + select INPUT_MATRIXKMAP + help + Say Y here if you want to use a keypad attached to the keyscan block + on some STMicroelectronics SoC devices. + + To compile this driver as a module, choose M here: the + module will be called st-keyscan. + config KEYBOARD_SUNKBD tristate "Sun Type 4 and Type 5 keyboard" select SERIO @@ -578,7 +589,7 @@ config KEYBOARD_OMAP config KEYBOARD_OMAP4 tristate "TI OMAP4+ keypad support" - depends on ARCH_OMAP2PLUS + depends on OF || ARCH_OMAP2PLUS select INPUT_MATRIXKMAP help Say Y here if you want to use the OMAP4+ keypad. diff --git a/drivers/input/keyboard/Makefile b/drivers/input/keyboard/Makefile index 11cff7b84b47..7504ae19049d 100644 --- a/drivers/input/keyboard/Makefile +++ b/drivers/input/keyboard/Makefile @@ -51,6 +51,7 @@ obj-$(CONFIG_KEYBOARD_SH_KEYSC) += sh_keysc.o obj-$(CONFIG_KEYBOARD_SPEAR) += spear-keyboard.o obj-$(CONFIG_KEYBOARD_STMPE) += stmpe-keypad.o obj-$(CONFIG_KEYBOARD_STOWAWAY) += stowaway.o +obj-$(CONFIG_KEYBOARD_ST_KEYSCAN) += st-keyscan.o obj-$(CONFIG_KEYBOARD_SUNKBD) += sunkbd.o obj-$(CONFIG_KEYBOARD_TC3589X) += tc3589x-keypad.o obj-$(CONFIG_KEYBOARD_TEGRA) += tegra-kbc.o diff --git a/drivers/input/keyboard/adp5520-keys.c b/drivers/input/keyboard/adp5520-keys.c index 4cc14c2fa7d5..7f4a8b58efc1 100644 --- a/drivers/input/keyboard/adp5520-keys.c +++ b/drivers/input/keyboard/adp5520-keys.c @@ -12,6 +12,7 @@ #include <linux/input.h> #include <linux/mfd/adp5520.h> #include <linux/slab.h> +#include <linux/device.h> struct adp5520_keys { struct input_dev *input; @@ -81,7 +82,7 @@ static int adp5520_keys_probe(struct platform_device *pdev) return -EINVAL; } - if (pdata == NULL) { + if (!pdata) { dev_err(&pdev->dev, "missing platform data\n"); return -EINVAL; } @@ -89,17 +90,15 @@ static int adp5520_keys_probe(struct platform_device *pdev) if (!(pdata->rows_en_mask && pdata->cols_en_mask)) return -EINVAL; - dev = kzalloc(sizeof(*dev), GFP_KERNEL); - if (dev == NULL) { + dev = devm_kzalloc(&pdev->dev, sizeof(*dev), GFP_KERNEL); + if (!dev) { dev_err(&pdev->dev, "failed to alloc memory\n"); return -ENOMEM; } - input = input_allocate_device(); - if (!input) { - ret = -ENOMEM; - goto err; - } + input = devm_input_allocate_device(&pdev->dev); + if (!input) + return -ENOMEM; dev->master = pdev->dev.parent; dev->input = input; @@ -135,7 +134,7 @@ static int adp5520_keys_probe(struct platform_device *pdev) ret = input_register_device(input); if (ret) { dev_err(&pdev->dev, "unable to register input device\n"); - goto err; + return ret; } en_mask = pdata->rows_en_mask | pdata->cols_en_mask; @@ -157,8 +156,7 @@ static int adp5520_keys_probe(struct platform_device *pdev) if (ret) { dev_err(&pdev->dev, "failed to write\n"); - ret = -EIO; - goto err1; + return -EIO; } dev->notifier.notifier_call = adp5520_keys_notifier; @@ -166,19 +164,11 @@ static int adp5520_keys_probe(struct platform_device *pdev) ADP5520_KP_IEN | ADP5520_KR_IEN); if (ret) { dev_err(&pdev->dev, "failed to register notifier\n"); - goto err1; + return ret; } platform_set_drvdata(pdev, dev); return 0; - -err1: - input_unregister_device(input); - input = NULL; -err: - input_free_device(input); - kfree(dev); - return ret; } static int adp5520_keys_remove(struct platform_device *pdev) @@ -188,8 +178,6 @@ static int adp5520_keys_remove(struct platform_device *pdev) adp5520_unregister_notifier(dev->master, &dev->notifier, ADP5520_KP_IEN | ADP5520_KR_IEN); - input_unregister_device(dev->input); - kfree(dev); return 0; } diff --git a/drivers/input/keyboard/clps711x-keypad.c b/drivers/input/keyboard/clps711x-keypad.c index 3955aecee44b..552b65c6e6b0 100644 --- a/drivers/input/keyboard/clps711x-keypad.c +++ b/drivers/input/keyboard/clps711x-keypad.c @@ -185,7 +185,7 @@ static int clps711x_keypad_remove(struct platform_device *pdev) return 0; } -static struct of_device_id clps711x_keypad_of_match[] = { +static const struct of_device_id clps711x_keypad_of_match[] = { { .compatible = "cirrus,clps711x-keypad", }, { } }; diff --git a/drivers/input/keyboard/gpio_keys.c b/drivers/input/keyboard/gpio_keys.c index 2db13246eb8e..8c98e97f8e41 100644 --- a/drivers/input/keyboard/gpio_keys.c +++ b/drivers/input/keyboard/gpio_keys.c @@ -424,6 +424,16 @@ out: return IRQ_HANDLED; } +static void gpio_keys_quiesce_key(void *data) +{ + struct gpio_button_data *bdata = data; + + if (bdata->timer_debounce) + del_timer_sync(&bdata->timer); + + cancel_work_sync(&bdata->work); +} + static int gpio_keys_setup_key(struct platform_device *pdev, struct input_dev *input, struct gpio_button_data *bdata, @@ -433,7 +443,8 @@ static int gpio_keys_setup_key(struct platform_device *pdev, struct device *dev = &pdev->dev; irq_handler_t isr; unsigned long irqflags; - int irq, error; + int irq; + int error; bdata->input = input; bdata->button = button; @@ -441,7 +452,8 @@ static int gpio_keys_setup_key(struct platform_device *pdev, if (gpio_is_valid(button->gpio)) { - error = gpio_request_one(button->gpio, GPIOF_IN, desc); + error = devm_gpio_request_one(&pdev->dev, button->gpio, + GPIOF_IN, desc); if (error < 0) { dev_err(dev, "Failed to request GPIO %d, error %d\n", button->gpio, error); @@ -463,7 +475,7 @@ static int gpio_keys_setup_key(struct platform_device *pdev, dev_err(dev, "Unable to get irq number for GPIO %d, error %d\n", button->gpio, error); - goto fail; + return error; } bdata->irq = irq; @@ -497,26 +509,33 @@ static int gpio_keys_setup_key(struct platform_device *pdev, input_set_capability(input, button->type ?: EV_KEY, button->code); /* + * Install custom action to cancel debounce timer and + * workqueue item. + */ + error = devm_add_action(&pdev->dev, gpio_keys_quiesce_key, bdata); + if (error) { + dev_err(&pdev->dev, + "failed to register quiesce action, error: %d\n", + error); + return error; + } + + /* * If platform has specified that the button can be disabled, * we don't want it to share the interrupt line. */ if (!button->can_disable) irqflags |= IRQF_SHARED; - error = request_any_context_irq(bdata->irq, isr, irqflags, desc, bdata); + error = devm_request_any_context_irq(&pdev->dev, bdata->irq, + isr, irqflags, desc, bdata); if (error < 0) { dev_err(dev, "Unable to claim irq %d; error %d\n", bdata->irq, error); - goto fail; + return error; } return 0; - -fail: - if (gpio_is_valid(button->gpio)) - gpio_free(button->gpio); - - return error; } static void gpio_keys_report_state(struct gpio_keys_drvdata *ddata) @@ -578,23 +597,18 @@ gpio_keys_get_devtree_pdata(struct device *dev) int i; node = dev->of_node; - if (!node) { - error = -ENODEV; - goto err_out; - } + if (!node) + return ERR_PTR(-ENODEV); nbuttons = of_get_child_count(node); - if (nbuttons == 0) { - error = -ENODEV; - goto err_out; - } + if (nbuttons == 0) + return ERR_PTR(-ENODEV); - pdata = kzalloc(sizeof(*pdata) + nbuttons * (sizeof *button), - GFP_KERNEL); - if (!pdata) { - error = -ENOMEM; - goto err_out; - } + pdata = devm_kzalloc(dev, + sizeof(*pdata) + nbuttons * sizeof(*button), + GFP_KERNEL); + if (!pdata) + return ERR_PTR(-ENOMEM); pdata->buttons = (struct gpio_keys_button *)(pdata + 1); pdata->nbuttons = nbuttons; @@ -619,7 +633,7 @@ gpio_keys_get_devtree_pdata(struct device *dev) dev_err(dev, "Failed to get gpio flags, error: %d\n", error); - goto err_free_pdata; + return ERR_PTR(error); } button = &pdata->buttons[i++]; @@ -630,8 +644,7 @@ gpio_keys_get_devtree_pdata(struct device *dev) if (of_property_read_u32(pp, "linux,code", &button->code)) { dev_err(dev, "Button without keycode: 0x%x\n", button->gpio); - error = -EINVAL; - goto err_free_pdata; + return ERR_PTR(-EINVAL); } button->desc = of_get_property(pp, "label", NULL); @@ -646,20 +659,13 @@ gpio_keys_get_devtree_pdata(struct device *dev) button->debounce_interval = 5; } - if (pdata->nbuttons == 0) { - error = -EINVAL; - goto err_free_pdata; - } + if (pdata->nbuttons == 0) + return ERR_PTR(-EINVAL); return pdata; - -err_free_pdata: - kfree(pdata); -err_out: - return ERR_PTR(error); } -static struct of_device_id gpio_keys_of_match[] = { +static const struct of_device_id gpio_keys_of_match[] = { { .compatible = "gpio-keys", }, { }, }; @@ -675,22 +681,13 @@ gpio_keys_get_devtree_pdata(struct device *dev) #endif -static void gpio_remove_key(struct gpio_button_data *bdata) -{ - free_irq(bdata->irq, bdata); - if (bdata->timer_debounce) - del_timer_sync(&bdata->timer); - cancel_work_sync(&bdata->work); - if (gpio_is_valid(bdata->button->gpio)) - gpio_free(bdata->button->gpio); -} - static int gpio_keys_probe(struct platform_device *pdev) { struct device *dev = &pdev->dev; const struct gpio_keys_platform_data *pdata = dev_get_platdata(dev); struct gpio_keys_drvdata *ddata; struct input_dev *input; + size_t size; int i, error; int wakeup = 0; @@ -700,14 +697,18 @@ static int gpio_keys_probe(struct platform_device *pdev) return PTR_ERR(pdata); } - ddata = kzalloc(sizeof(struct gpio_keys_drvdata) + - pdata->nbuttons * sizeof(struct gpio_button_data), - GFP_KERNEL); - input = input_allocate_device(); - if (!ddata || !input) { + size = sizeof(struct gpio_keys_drvdata) + + pdata->nbuttons * sizeof(struct gpio_button_data); + ddata = devm_kzalloc(dev, size, GFP_KERNEL); + if (!ddata) { dev_err(dev, "failed to allocate state\n"); - error = -ENOMEM; - goto fail1; + return -ENOMEM; + } + + input = devm_input_allocate_device(dev); + if (!input) { + dev_err(dev, "failed to allocate input device\n"); + return -ENOMEM; } ddata->pdata = pdata; @@ -738,7 +739,7 @@ static int gpio_keys_probe(struct platform_device *pdev) error = gpio_keys_setup_key(pdev, input, bdata, button); if (error) - goto fail2; + return error; if (button->wakeup) wakeup = 1; @@ -748,57 +749,31 @@ static int gpio_keys_probe(struct platform_device *pdev) if (error) { dev_err(dev, "Unable to export keys/switches, error: %d\n", error); - goto fail2; + return error; } error = input_register_device(input); if (error) { dev_err(dev, "Unable to register input device, error: %d\n", error); - goto fail3; + goto err_remove_group; } device_init_wakeup(&pdev->dev, wakeup); return 0; - fail3: +err_remove_group: sysfs_remove_group(&pdev->dev.kobj, &gpio_keys_attr_group); - fail2: - while (--i >= 0) - gpio_remove_key(&ddata->data[i]); - - fail1: - input_free_device(input); - kfree(ddata); - /* If we have no platform data, we allocated pdata dynamically. */ - if (!dev_get_platdata(&pdev->dev)) - kfree(pdata); - return error; } static int gpio_keys_remove(struct platform_device *pdev) { - struct gpio_keys_drvdata *ddata = platform_get_drvdata(pdev); - struct input_dev *input = ddata->input; - int i; - sysfs_remove_group(&pdev->dev.kobj, &gpio_keys_attr_group); device_init_wakeup(&pdev->dev, 0); - for (i = 0; i < ddata->pdata->nbuttons; i++) - gpio_remove_key(&ddata->data[i]); - - input_unregister_device(input); - - /* If we have no platform data, we allocated pdata dynamically. */ - if (!dev_get_platdata(&pdev->dev)) - kfree(ddata->pdata); - - kfree(ddata); - return 0; } diff --git a/drivers/input/keyboard/gpio_keys_polled.c b/drivers/input/keyboard/gpio_keys_polled.c index e571e194ff84..432d36395f35 100644 --- a/drivers/input/keyboard/gpio_keys_polled.c +++ b/drivers/input/keyboard/gpio_keys_polled.c @@ -120,12 +120,10 @@ static struct gpio_keys_platform_data *gpio_keys_polled_get_devtree_pdata(struct if (nbuttons == 0) return NULL; - pdata = kzalloc(sizeof(*pdata) + nbuttons * (sizeof *button), - GFP_KERNEL); - if (!pdata) { - error = -ENOMEM; - goto err_out; - } + pdata = devm_kzalloc(dev, sizeof(*pdata) + nbuttons * sizeof(*button), + GFP_KERNEL); + if (!pdata) + return ERR_PTR(-ENOMEM); pdata->buttons = (struct gpio_keys_button *)(pdata + 1); pdata->nbuttons = nbuttons; @@ -151,7 +149,7 @@ static struct gpio_keys_platform_data *gpio_keys_polled_get_devtree_pdata(struct dev_err(dev, "Failed to get gpio flags, error: %d\n", error); - goto err_free_pdata; + return ERR_PTR(error); } button = &pdata->buttons[i++]; @@ -162,8 +160,7 @@ static struct gpio_keys_platform_data *gpio_keys_polled_get_devtree_pdata(struct if (of_property_read_u32(pp, "linux,code", &button->code)) { dev_err(dev, "Button without keycode: 0x%x\n", button->gpio); - error = -EINVAL; - goto err_free_pdata; + return ERR_PTR(-EINVAL); } button->desc = of_get_property(pp, "label", NULL); @@ -178,20 +175,13 @@ static struct gpio_keys_platform_data *gpio_keys_polled_get_devtree_pdata(struct button->debounce_interval = 5; } - if (pdata->nbuttons == 0) { - error = -EINVAL; - goto err_free_pdata; - } + if (pdata->nbuttons == 0) + return ERR_PTR(-EINVAL); return pdata; - -err_free_pdata: - kfree(pdata); -err_out: - return ERR_PTR(error); } -static struct of_device_id gpio_keys_polled_of_match[] = { +static const struct of_device_id gpio_keys_polled_of_match[] = { { .compatible = "gpio-keys-polled", }, { }, }; @@ -213,6 +203,7 @@ static int gpio_keys_polled_probe(struct platform_device *pdev) struct gpio_keys_polled_dev *bdev; struct input_polled_dev *poll_dev; struct input_dev *input; + size_t size; int error; int i; @@ -228,24 +219,21 @@ static int gpio_keys_polled_probe(struct platform_device *pdev) if (!pdata->poll_interval) { dev_err(dev, "missing poll_interval value\n"); - error = -EINVAL; - goto err_free_pdata; + return -EINVAL; } - bdev = kzalloc(sizeof(struct gpio_keys_polled_dev) + - pdata->nbuttons * sizeof(struct gpio_keys_button_data), - GFP_KERNEL); + size = sizeof(struct gpio_keys_polled_dev) + + pdata->nbuttons * sizeof(struct gpio_keys_button_data); + bdev = devm_kzalloc(&pdev->dev, size, GFP_KERNEL); if (!bdev) { dev_err(dev, "no memory for private data\n"); - error = -ENOMEM; - goto err_free_pdata; + return -ENOMEM; } - poll_dev = input_allocate_polled_device(); + poll_dev = devm_input_allocate_polled_device(&pdev->dev); if (!poll_dev) { dev_err(dev, "no memory for polled device\n"); - error = -ENOMEM; - goto err_free_bdev; + return -ENOMEM; } poll_dev->private = bdev; @@ -258,7 +246,6 @@ static int gpio_keys_polled_probe(struct platform_device *pdev) input->name = pdev->name; input->phys = DRV_NAME"/input0"; - input->dev.parent = &pdev->dev; input->id.bustype = BUS_HOST; input->id.vendor = 0x0001; @@ -277,16 +264,15 @@ static int gpio_keys_polled_probe(struct platform_device *pdev) if (button->wakeup) { dev_err(dev, DRV_NAME " does not support wakeup\n"); - error = -EINVAL; - goto err_free_gpio; + return -EINVAL; } - error = gpio_request_one(gpio, GPIOF_IN, - button->desc ?: DRV_NAME); + error = devm_gpio_request_one(&pdev->dev, gpio, GPIOF_IN, + button->desc ? : DRV_NAME); if (error) { dev_err(dev, "unable to claim gpio %u, err=%d\n", gpio, error); - goto err_free_gpio; + return error; } bdata->can_sleep = gpio_cansleep(gpio); @@ -306,7 +292,7 @@ static int gpio_keys_polled_probe(struct platform_device *pdev) if (error) { dev_err(dev, "unable to register polled device, err=%d\n", error); - goto err_free_gpio; + return error; } /* report initial state of the buttons */ @@ -315,52 +301,10 @@ static int gpio_keys_polled_probe(struct platform_device *pdev) &bdev->data[i]); return 0; - -err_free_gpio: - while (--i >= 0) - gpio_free(pdata->buttons[i].gpio); - - input_free_polled_device(poll_dev); - -err_free_bdev: - kfree(bdev); - -err_free_pdata: - /* If we have no platform_data, we allocated pdata dynamically. */ - if (!dev_get_platdata(&pdev->dev)) - kfree(pdata); - - return error; -} - -static int gpio_keys_polled_remove(struct platform_device *pdev) -{ - struct gpio_keys_polled_dev *bdev = platform_get_drvdata(pdev); - const struct gpio_keys_platform_data *pdata = bdev->pdata; - int i; - - input_unregister_polled_device(bdev->poll_dev); - - for (i = 0; i < pdata->nbuttons; i++) - gpio_free(pdata->buttons[i].gpio); - - input_free_polled_device(bdev->poll_dev); - - /* - * If we had no platform_data, we allocated pdata dynamically and - * must free it here. - */ - if (!dev_get_platdata(&pdev->dev)) - kfree(pdata); - - kfree(bdev); - - return 0; } static struct platform_driver gpio_keys_polled_driver = { .probe = gpio_keys_polled_probe, - .remove = gpio_keys_polled_remove, .driver = { .name = DRV_NAME, .owner = THIS_MODULE, diff --git a/drivers/input/keyboard/imx_keypad.c b/drivers/input/keyboard/imx_keypad.c index 97ec33572e56..8280cb16260b 100644 --- a/drivers/input/keyboard/imx_keypad.c +++ b/drivers/input/keyboard/imx_keypad.c @@ -415,7 +415,7 @@ open_err: } #ifdef CONFIG_OF -static struct of_device_id imx_keypad_of_match[] = { +static const struct of_device_id imx_keypad_of_match[] = { { .compatible = "fsl,imx21-kpp", }, { /* sentinel */ } }; diff --git a/drivers/input/keyboard/jornada680_kbd.c b/drivers/input/keyboard/jornada680_kbd.c index 69b1f002ff52..0ba4428da24a 100644 --- a/drivers/input/keyboard/jornada680_kbd.c +++ b/drivers/input/keyboard/jornada680_kbd.c @@ -16,6 +16,7 @@ * published by the Free Software Foundation. */ +#include <linux/device.h> #include <linux/input.h> #include <linux/input-polldev.h> #include <linux/interrupt.h> @@ -185,14 +186,15 @@ static int jornada680kbd_probe(struct platform_device *pdev) struct input_dev *input_dev; int i, error; - jornadakbd = kzalloc(sizeof(struct jornadakbd), GFP_KERNEL); + jornadakbd = devm_kzalloc(&pdev->dev, sizeof(struct jornadakbd), + GFP_KERNEL); if (!jornadakbd) return -ENOMEM; - poll_dev = input_allocate_polled_device(); + poll_dev = devm_input_allocate_polled_device(&pdev->dev); if (!poll_dev) { - error = -ENOMEM; - goto failed; + dev_err(&pdev->dev, "failed to allocate polled input device\n"); + return -ENOMEM; } platform_set_drvdata(pdev, jornadakbd); @@ -224,27 +226,10 @@ static int jornada680kbd_probe(struct platform_device *pdev) input_set_capability(input_dev, EV_MSC, MSC_SCAN); error = input_register_polled_device(jornadakbd->poll_dev); - if (error) - goto failed; - - return 0; - - failed: - printk(KERN_ERR "Jornadakbd: failed to register driver, error: %d\n", - error); - input_free_polled_device(poll_dev); - kfree(jornadakbd); - return error; - -} - -static int jornada680kbd_remove(struct platform_device *pdev) -{ - struct jornadakbd *jornadakbd = platform_get_drvdata(pdev); - - input_unregister_polled_device(jornadakbd->poll_dev); - input_free_polled_device(jornadakbd->poll_dev); - kfree(jornadakbd); + if (error) { + dev_err(&pdev->dev, "failed to register polled input device\n"); + return error; + } return 0; } @@ -255,7 +240,6 @@ static struct platform_driver jornada680kbd_driver = { .owner = THIS_MODULE, }, .probe = jornada680kbd_probe, - .remove = jornada680kbd_remove, }; module_platform_driver(jornada680kbd_driver); diff --git a/drivers/input/keyboard/mcs_touchkey.c b/drivers/input/keyboard/mcs_touchkey.c index 1da8e0b44b56..375b05ca8e2a 100644 --- a/drivers/input/keyboard/mcs_touchkey.c +++ b/drivers/input/keyboard/mcs_touchkey.c @@ -147,7 +147,7 @@ static int mcs_touchkey_probe(struct i2c_client *client, } dev_info(&client->dev, "Firmware version: %d\n", fw_ver); - input_dev->name = "MELPAS MCS Touchkey"; + input_dev->name = "MELFAS MCS Touchkey"; input_dev->id.bustype = BUS_I2C; input_dev->dev.parent = &client->dev; input_dev->evbit[0] = BIT_MASK(EV_KEY); diff --git a/drivers/input/keyboard/omap4-keypad.c b/drivers/input/keyboard/omap4-keypad.c index 0400b3f2b4b9..024b7bdffe5b 100644 --- a/drivers/input/keyboard/omap4-keypad.c +++ b/drivers/input/keyboard/omap4-keypad.c @@ -28,11 +28,10 @@ #include <linux/io.h> #include <linux/of.h> #include <linux/input.h> +#include <linux/input/matrix_keypad.h> #include <linux/slab.h> #include <linux/pm_runtime.h> -#include <linux/platform_data/omap4-keypad.h> - /* OMAP4 registers */ #define OMAP4_KBD_REVISION 0x00 #define OMAP4_KBD_SYSCONFIG 0x10 @@ -218,7 +217,6 @@ static void omap4_keypad_close(struct input_dev *input) pm_runtime_put_sync(input->dev.parent); } -#ifdef CONFIG_OF static int omap4_keypad_parse_dt(struct device *dev, struct omap4_keypad *keypad_data) { @@ -235,20 +233,9 @@ static int omap4_keypad_parse_dt(struct device *dev, return 0; } -#else -static inline int omap4_keypad_parse_dt(struct device *dev, - struct omap4_keypad *keypad_data) -{ - return -ENOSYS; -} -#endif static int omap4_keypad_probe(struct platform_device *pdev) { - const struct omap4_keypad_platform_data *pdata = - dev_get_platdata(&pdev->dev); - const struct matrix_keymap_data *keymap_data = - pdata ? pdata->keymap_data : NULL; struct omap4_keypad *keypad_data; struct input_dev *input_dev; struct resource *res; @@ -277,14 +264,9 @@ static int omap4_keypad_probe(struct platform_device *pdev) keypad_data->irq = irq; - if (pdata) { - keypad_data->rows = pdata->rows; - keypad_data->cols = pdata->cols; - } else { - error = omap4_keypad_parse_dt(&pdev->dev, keypad_data); - if (error) - return error; - } + error = omap4_keypad_parse_dt(&pdev->dev, keypad_data); + if (error) + return error; res = request_mem_region(res->start, resource_size(res), pdev->name); if (!res) { @@ -363,7 +345,7 @@ static int omap4_keypad_probe(struct platform_device *pdev) goto err_free_input; } - error = matrix_keypad_build_keymap(keymap_data, NULL, + error = matrix_keypad_build_keymap(NULL, NULL, keypad_data->rows, keypad_data->cols, keypad_data->keymap, input_dev); if (error) { @@ -434,13 +416,11 @@ static int omap4_keypad_remove(struct platform_device *pdev) return 0; } -#ifdef CONFIG_OF static const struct of_device_id omap_keypad_dt_match[] = { { .compatible = "ti,omap4-keypad" }, {}, }; MODULE_DEVICE_TABLE(of, omap_keypad_dt_match); -#endif #ifdef CONFIG_PM_SLEEP static int omap4_keypad_suspend(struct device *dev) @@ -482,7 +462,7 @@ static struct platform_driver omap4_keypad_driver = { .name = "omap4-keypad", .owner = THIS_MODULE, .pm = &omap4_keypad_pm_ops, - .of_match_table = of_match_ptr(omap_keypad_dt_match), + .of_match_table = omap_keypad_dt_match, }, }; module_platform_driver(omap4_keypad_driver); diff --git a/drivers/input/keyboard/st-keyscan.c b/drivers/input/keyboard/st-keyscan.c new file mode 100644 index 000000000000..758b48731415 --- /dev/null +++ b/drivers/input/keyboard/st-keyscan.c @@ -0,0 +1,274 @@ +/* + * STMicroelectronics Key Scanning driver + * + * Copyright (c) 2014 STMicroelectonics Ltd. + * Author: Stuart Menefy <stuart.menefy@st.com> + * + * Based on sh_keysc.c, copyright 2008 Magnus Damm + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +#include <linux/module.h> +#include <linux/interrupt.h> +#include <linux/platform_device.h> +#include <linux/clk.h> +#include <linux/io.h> +#include <linux/input/matrix_keypad.h> + +#define ST_KEYSCAN_MAXKEYS 16 + +#define KEYSCAN_CONFIG_OFF 0x0 +#define KEYSCAN_CONFIG_ENABLE 0x1 +#define KEYSCAN_DEBOUNCE_TIME_OFF 0x4 +#define KEYSCAN_MATRIX_STATE_OFF 0x8 +#define KEYSCAN_MATRIX_DIM_OFF 0xc +#define KEYSCAN_MATRIX_DIM_X_SHIFT 0x0 +#define KEYSCAN_MATRIX_DIM_Y_SHIFT 0x2 + +struct st_keyscan { + void __iomem *base; + int irq; + struct clk *clk; + struct input_dev *input_dev; + unsigned long last_state; + unsigned int n_rows; + unsigned int n_cols; + unsigned int debounce_us; +}; + +static irqreturn_t keyscan_isr(int irq, void *dev_id) +{ + struct st_keyscan *keypad = dev_id; + unsigned short *keycode = keypad->input_dev->keycode; + unsigned long state, change; + int bit_nr; + + state = readl(keypad->base + KEYSCAN_MATRIX_STATE_OFF) & 0xffff; + change = keypad->last_state ^ state; + keypad->last_state = state; + + for_each_set_bit(bit_nr, &change, BITS_PER_LONG) + input_report_key(keypad->input_dev, + keycode[bit_nr], state & BIT(bit_nr)); + + input_sync(keypad->input_dev); + + return IRQ_HANDLED; +} + +static int keyscan_start(struct st_keyscan *keypad) +{ + int error; + + error = clk_enable(keypad->clk); + if (error) + return error; + + writel(keypad->debounce_us * (clk_get_rate(keypad->clk) / 1000000), + keypad->base + KEYSCAN_DEBOUNCE_TIME_OFF); + + writel(((keypad->n_cols - 1) << KEYSCAN_MATRIX_DIM_X_SHIFT) | + ((keypad->n_rows - 1) << KEYSCAN_MATRIX_DIM_Y_SHIFT), + keypad->base + KEYSCAN_MATRIX_DIM_OFF); + + writel(KEYSCAN_CONFIG_ENABLE, keypad->base + KEYSCAN_CONFIG_OFF); + + return 0; +} + +static void keyscan_stop(struct st_keyscan *keypad) +{ + writel(0, keypad->base + KEYSCAN_CONFIG_OFF); + + clk_disable(keypad->clk); +} + +static int keyscan_open(struct input_dev *dev) +{ + struct st_keyscan *keypad = input_get_drvdata(dev); + + return keyscan_start(keypad); +} + +static void keyscan_close(struct input_dev *dev) +{ + struct st_keyscan *keypad = input_get_drvdata(dev); + + keyscan_stop(keypad); +} + +static int keypad_matrix_key_parse_dt(struct st_keyscan *keypad_data) +{ + struct device *dev = keypad_data->input_dev->dev.parent; + struct device_node *np = dev->of_node; + int error; + + error = matrix_keypad_parse_of_params(dev, &keypad_data->n_rows, + &keypad_data->n_cols); + if (error) { + dev_err(dev, "failed to parse keypad params\n"); + return error; + } + + of_property_read_u32(np, "st,debounce-us", &keypad_data->debounce_us); + + dev_dbg(dev, "n_rows=%d n_col=%d debounce=%d\n", + keypad_data->n_rows, keypad_data->n_cols, + keypad_data->debounce_us); + + return 0; +} + +static int keyscan_probe(struct platform_device *pdev) +{ + struct st_keyscan *keypad_data; + struct input_dev *input_dev; + struct resource *res; + int error; + + if (!pdev->dev.of_node) { + dev_err(&pdev->dev, "no DT data present\n"); + return -EINVAL; + } + + keypad_data = devm_kzalloc(&pdev->dev, sizeof(*keypad_data), + GFP_KERNEL); + if (!keypad_data) + return -ENOMEM; + + input_dev = devm_input_allocate_device(&pdev->dev); + if (!input_dev) { + dev_err(&pdev->dev, "failed to allocate the input device\n"); + return -ENOMEM; + } + + input_dev->name = pdev->name; + input_dev->phys = "keyscan-keys/input0"; + input_dev->dev.parent = &pdev->dev; + input_dev->open = keyscan_open; + input_dev->close = keyscan_close; + + input_dev->id.bustype = BUS_HOST; + + error = keypad_matrix_key_parse_dt(keypad_data); + if (error) + return error; + + error = matrix_keypad_build_keymap(NULL, NULL, + keypad_data->n_rows, + keypad_data->n_cols, + NULL, input_dev); + if (error) { + dev_err(&pdev->dev, "failed to build keymap\n"); + return error; + } + + input_set_drvdata(input_dev, keypad_data); + + keypad_data->input_dev = input_dev; + + res = platform_get_resource(pdev, IORESOURCE_MEM, 0); + keypad_data->base = devm_ioremap_resource(&pdev->dev, res); + if (IS_ERR(keypad_data->base)) + return PTR_ERR(keypad_data->base); + + keypad_data->clk = devm_clk_get(&pdev->dev, NULL); + if (IS_ERR(keypad_data->clk)) { + dev_err(&pdev->dev, "cannot get clock\n"); + return PTR_ERR(keypad_data->clk); + } + + error = clk_enable(keypad_data->clk); + if (error) { + dev_err(&pdev->dev, "failed to enable clock\n"); + return error; + } + + keyscan_stop(keypad_data); + + keypad_data->irq = platform_get_irq(pdev, 0); + if (keypad_data->irq < 0) { + dev_err(&pdev->dev, "no IRQ specified\n"); + return -EINVAL; + } + + error = devm_request_irq(&pdev->dev, keypad_data->irq, keyscan_isr, 0, + pdev->name, keypad_data); + if (error) { + dev_err(&pdev->dev, "failed to request IRQ\n"); + return error; + } + + error = input_register_device(input_dev); + if (error) { + dev_err(&pdev->dev, "failed to register input device\n"); + return error; + } + + platform_set_drvdata(pdev, keypad_data); + + device_set_wakeup_capable(&pdev->dev, 1); + + return 0; +} + +static int keyscan_suspend(struct device *dev) +{ + struct platform_device *pdev = to_platform_device(dev); + struct st_keyscan *keypad = platform_get_drvdata(pdev); + struct input_dev *input = keypad->input_dev; + + mutex_lock(&input->mutex); + + if (device_may_wakeup(dev)) + enable_irq_wake(keypad->irq); + else if (input->users) + keyscan_stop(keypad); + + mutex_unlock(&input->mutex); + return 0; +} + +static int keyscan_resume(struct device *dev) +{ + struct platform_device *pdev = to_platform_device(dev); + struct st_keyscan *keypad = platform_get_drvdata(pdev); + struct input_dev *input = keypad->input_dev; + int retval = 0; + + mutex_lock(&input->mutex); + + if (device_may_wakeup(dev)) + disable_irq_wake(keypad->irq); + else if (input->users) + retval = keyscan_start(keypad); + + mutex_unlock(&input->mutex); + return retval; +} + +static SIMPLE_DEV_PM_OPS(keyscan_dev_pm_ops, keyscan_suspend, keyscan_resume); + +static const struct of_device_id keyscan_of_match[] = { + { .compatible = "st,sti-keyscan" }, + { }, +}; +MODULE_DEVICE_TABLE(of, keyscan_of_match); + +static struct platform_driver keyscan_device_driver = { + .probe = keyscan_probe, + .driver = { + .name = "st-keyscan", + .pm = &keyscan_dev_pm_ops, + .of_match_table = of_match_ptr(keyscan_of_match), + } +}; + +module_platform_driver(keyscan_device_driver); + +MODULE_AUTHOR("Stuart Menefy <stuart.menefy@st.com>"); +MODULE_DESCRIPTION("STMicroelectronics keyscan device driver"); +MODULE_LICENSE("GPL"); diff --git a/drivers/input/keyboard/tc3589x-keypad.c b/drivers/input/keyboard/tc3589x-keypad.c index 74494a357522..ad7abae69078 100644 --- a/drivers/input/keyboard/tc3589x-keypad.c +++ b/drivers/input/keyboard/tc3589x-keypad.c @@ -296,6 +296,65 @@ static void tc3589x_keypad_close(struct input_dev *input) tc3589x_keypad_disable(keypad); } +#ifdef CONFIG_OF +static const struct tc3589x_keypad_platform_data * +tc3589x_keypad_of_probe(struct device *dev) +{ + struct device_node *np = dev->of_node; + struct tc3589x_keypad_platform_data *plat; + u32 cols, rows; + u32 debounce_ms; + int proplen; + + if (!np) + return ERR_PTR(-ENODEV); + + plat = devm_kzalloc(dev, sizeof(*plat), GFP_KERNEL); + if (!plat) + return ERR_PTR(-ENOMEM); + + of_property_read_u32(np, "keypad,num-columns", &cols); + of_property_read_u32(np, "keypad,num-rows", &rows); + plat->kcol = (u8) cols; + plat->krow = (u8) rows; + if (!plat->krow || !plat->kcol || + plat->krow > TC_KPD_ROWS || plat->kcol > TC_KPD_COLUMNS) { + dev_err(dev, + "keypad columns/rows not properly specified (%ux%u)\n", + plat->kcol, plat->krow); + return ERR_PTR(-EINVAL); + } + + if (!of_get_property(np, "linux,keymap", &proplen)) { + dev_err(dev, "property linux,keymap not found\n"); + return ERR_PTR(-ENOENT); + } + + plat->no_autorepeat = of_property_read_bool(np, "linux,no-autorepeat"); + plat->enable_wakeup = of_property_read_bool(np, "linux,wakeup"); + + /* The custom delay format is ms/16 */ + of_property_read_u32(np, "debounce-delay-ms", &debounce_ms); + if (debounce_ms) + plat->debounce_period = debounce_ms * 16; + else + plat->debounce_period = TC_KPD_DEBOUNCE_PERIOD; + + plat->settle_time = TC_KPD_SETTLE_TIME; + /* FIXME: should be property of the IRQ resource? */ + plat->irqtype = IRQF_TRIGGER_FALLING; + + return plat; +} +#else +static inline const struct tc3589x_keypad_platform_data * +tc3589x_keypad_of_probe(struct device *dev) +{ + return ERR_PTR(-ENODEV); +} +#endif + + static int tc3589x_keypad_probe(struct platform_device *pdev) { struct tc3589x *tc3589x = dev_get_drvdata(pdev->dev.parent); @@ -306,8 +365,11 @@ static int tc3589x_keypad_probe(struct platform_device *pdev) plat = tc3589x->pdata->keypad; if (!plat) { - dev_err(&pdev->dev, "invalid keypad platform data\n"); - return -EINVAL; + plat = tc3589x_keypad_of_probe(&pdev->dev); + if (IS_ERR(plat)) { + dev_err(&pdev->dev, "invalid keypad platform data\n"); + return PTR_ERR(plat); + } } irq = platform_get_irq(pdev, 0); diff --git a/drivers/input/misc/88pm860x_onkey.c b/drivers/input/misc/88pm860x_onkey.c index abd8453e5212..220ce0fa15d9 100644 --- a/drivers/input/misc/88pm860x_onkey.c +++ b/drivers/input/misc/88pm860x_onkey.c @@ -26,6 +26,7 @@ #include <linux/interrupt.h> #include <linux/mfd/88pm860x.h> #include <linux/slab.h> +#include <linux/device.h> #define PM8607_WAKEUP 0x0b @@ -68,7 +69,8 @@ static int pm860x_onkey_probe(struct platform_device *pdev) return -EINVAL; } - info = kzalloc(sizeof(struct pm860x_onkey_info), GFP_KERNEL); + info = devm_kzalloc(&pdev->dev, sizeof(struct pm860x_onkey_info), + GFP_KERNEL); if (!info) return -ENOMEM; info->chip = chip; @@ -76,11 +78,10 @@ static int pm860x_onkey_probe(struct platform_device *pdev) info->dev = &pdev->dev; info->irq = irq; - info->idev = input_allocate_device(); + info->idev = devm_input_allocate_device(&pdev->dev); if (!info->idev) { dev_err(chip->dev, "Failed to allocate input dev\n"); - ret = -ENOMEM; - goto out; + return -ENOMEM; } info->idev->name = "88pm860x_on"; @@ -93,42 +94,22 @@ static int pm860x_onkey_probe(struct platform_device *pdev) ret = input_register_device(info->idev); if (ret) { dev_err(chip->dev, "Can't register input device: %d\n", ret); - goto out_reg; + return ret; } - ret = request_threaded_irq(info->irq, NULL, pm860x_onkey_handler, - IRQF_ONESHOT, "onkey", info); + ret = devm_request_threaded_irq(&pdev->dev, info->irq, NULL, + pm860x_onkey_handler, IRQF_ONESHOT, + "onkey", info); if (ret < 0) { dev_err(chip->dev, "Failed to request IRQ: #%d: %d\n", info->irq, ret); - goto out_irq; + return ret; } platform_set_drvdata(pdev, info); device_init_wakeup(&pdev->dev, 1); return 0; - -out_irq: - input_unregister_device(info->idev); - kfree(info); - return ret; - -out_reg: - input_free_device(info->idev); -out: - kfree(info); - return ret; -} - -static int pm860x_onkey_remove(struct platform_device *pdev) -{ - struct pm860x_onkey_info *info = platform_get_drvdata(pdev); - - free_irq(info->irq, info); - input_unregister_device(info->idev); - kfree(info); - return 0; } #ifdef CONFIG_PM_SLEEP @@ -161,7 +142,6 @@ static struct platform_driver pm860x_onkey_driver = { .pm = &pm860x_onkey_pm_ops, }, .probe = pm860x_onkey_probe, - .remove = pm860x_onkey_remove, }; module_platform_driver(pm860x_onkey_driver); diff --git a/drivers/input/misc/Kconfig b/drivers/input/misc/Kconfig index 5928ea71dd69..2ff4425a893b 100644 --- a/drivers/input/misc/Kconfig +++ b/drivers/input/misc/Kconfig @@ -224,7 +224,7 @@ config INPUT_GP2A config INPUT_GPIO_BEEPER tristate "Generic GPIO Beeper support" - depends on OF_GPIO + depends on GPIOLIB help Say Y here if you have a beeper connected to a GPIO pin. diff --git a/drivers/input/misc/ab8500-ponkey.c b/drivers/input/misc/ab8500-ponkey.c index f2fbdd88ed20..95ef7dd6442d 100644 --- a/drivers/input/misc/ab8500-ponkey.c +++ b/drivers/input/misc/ab8500-ponkey.c @@ -7,6 +7,7 @@ * AB8500 Power-On Key handler */ +#include <linux/device.h> #include <linux/kernel.h> #include <linux/module.h> #include <linux/platform_device.h> @@ -65,12 +66,14 @@ static int ab8500_ponkey_probe(struct platform_device *pdev) return irq_dbr; } - ponkey = kzalloc(sizeof(struct ab8500_ponkey), GFP_KERNEL); - input = input_allocate_device(); - if (!ponkey || !input) { - error = -ENOMEM; - goto err_free_mem; - } + ponkey = devm_kzalloc(&pdev->dev, sizeof(struct ab8500_ponkey), + GFP_KERNEL); + if (!ponkey) + return -ENOMEM; + + input = devm_input_allocate_device(&pdev->dev); + if (!input) + return -ENOMEM; ponkey->idev = input; ponkey->ab8500 = ab8500; @@ -82,52 +85,32 @@ static int ab8500_ponkey_probe(struct platform_device *pdev) input_set_capability(input, EV_KEY, KEY_POWER); - error = request_any_context_irq(ponkey->irq_dbf, ab8500_ponkey_handler, - 0, "ab8500-ponkey-dbf", ponkey); + error = devm_request_any_context_irq(&pdev->dev, ponkey->irq_dbf, + ab8500_ponkey_handler, 0, + "ab8500-ponkey-dbf", ponkey); if (error < 0) { dev_err(ab8500->dev, "Failed to request dbf IRQ#%d: %d\n", ponkey->irq_dbf, error); - goto err_free_mem; + return error; } - error = request_any_context_irq(ponkey->irq_dbr, ab8500_ponkey_handler, - 0, "ab8500-ponkey-dbr", ponkey); + error = devm_request_any_context_irq(&pdev->dev, ponkey->irq_dbr, + ab8500_ponkey_handler, 0, + "ab8500-ponkey-dbr", ponkey); if (error < 0) { dev_err(ab8500->dev, "Failed to request dbr IRQ#%d: %d\n", ponkey->irq_dbr, error); - goto err_free_dbf_irq; + return error; } error = input_register_device(ponkey->idev); if (error) { dev_err(ab8500->dev, "Can't register input device: %d\n", error); - goto err_free_dbr_irq; + return error; } platform_set_drvdata(pdev, ponkey); return 0; - -err_free_dbr_irq: - free_irq(ponkey->irq_dbr, ponkey); -err_free_dbf_irq: - free_irq(ponkey->irq_dbf, ponkey); -err_free_mem: - input_free_device(input); - kfree(ponkey); - - return error; -} - -static int ab8500_ponkey_remove(struct platform_device *pdev) -{ - struct ab8500_ponkey *ponkey = platform_get_drvdata(pdev); - - free_irq(ponkey->irq_dbf, ponkey); - free_irq(ponkey->irq_dbr, ponkey); - input_unregister_device(ponkey->idev); - kfree(ponkey); - - return 0; } #ifdef CONFIG_OF @@ -144,7 +127,6 @@ static struct platform_driver ab8500_ponkey_driver = { .of_match_table = of_match_ptr(ab8500_ponkey_match), }, .probe = ab8500_ponkey_probe, - .remove = ab8500_ponkey_remove, }; module_platform_driver(ab8500_ponkey_driver); diff --git a/drivers/input/misc/gpio-beeper.c b/drivers/input/misc/gpio-beeper.c index b757435e2b3d..8886af63eae3 100644 --- a/drivers/input/misc/gpio-beeper.c +++ b/drivers/input/misc/gpio-beeper.c @@ -1,7 +1,7 @@ /* * Generic GPIO beeper driver * - * Copyright (C) 2013 Alexander Shiyan <shc_work@mail.ru> + * Copyright (C) 2013-2014 Alexander Shiyan <shc_work@mail.ru> * * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License as published by @@ -11,7 +11,8 @@ #include <linux/input.h> #include <linux/module.h> -#include <linux/of_gpio.h> +#include <linux/gpio/consumer.h> +#include <linux/of.h> #include <linux/workqueue.h> #include <linux/platform_device.h> @@ -19,14 +20,13 @@ struct gpio_beeper { struct work_struct work; - int gpio; - bool active_low; + struct gpio_desc *desc; bool beeping; }; static void gpio_beeper_toggle(struct gpio_beeper *beep, bool on) { - gpio_set_value_cansleep(beep->gpio, on ^ beep->active_low); + gpiod_set_value_cansleep(beep->desc, on); } static void gpio_beeper_work(struct work_struct *work) @@ -65,18 +65,16 @@ static void gpio_beeper_close(struct input_dev *input) static int gpio_beeper_probe(struct platform_device *pdev) { struct gpio_beeper *beep; - enum of_gpio_flags flags; struct input_dev *input; - unsigned long gflags; int err; beep = devm_kzalloc(&pdev->dev, sizeof(*beep), GFP_KERNEL); if (!beep) return -ENOMEM; - beep->gpio = of_get_gpio_flags(pdev->dev.of_node, 0, &flags); - if (!gpio_is_valid(beep->gpio)) - return beep->gpio; + beep->desc = devm_gpiod_get(&pdev->dev, NULL); + if (IS_ERR(beep->desc)) + return PTR_ERR(beep->desc); input = devm_input_allocate_device(&pdev->dev); if (!input) @@ -94,10 +92,7 @@ static int gpio_beeper_probe(struct platform_device *pdev) input_set_capability(input, EV_SND, SND_BELL); - beep->active_low = flags & OF_GPIO_ACTIVE_LOW; - gflags = beep->active_low ? GPIOF_OUT_INIT_HIGH : GPIOF_OUT_INIT_LOW; - - err = devm_gpio_request_one(&pdev->dev, beep->gpio, gflags, pdev->name); + err = gpiod_direction_output(beep->desc, 0); if (err) return err; @@ -106,17 +101,19 @@ static int gpio_beeper_probe(struct platform_device *pdev) return input_register_device(input); } -static struct of_device_id gpio_beeper_of_match[] = { +#ifdef CONFIG_OF +static const struct of_device_id gpio_beeper_of_match[] = { { .compatible = BEEPER_MODNAME, }, { } }; MODULE_DEVICE_TABLE(of, gpio_beeper_of_match); +#endif static struct platform_driver gpio_beeper_platform_driver = { .driver = { .name = BEEPER_MODNAME, .owner = THIS_MODULE, - .of_match_table = gpio_beeper_of_match, + .of_match_table = of_match_ptr(gpio_beeper_of_match), }, .probe = gpio_beeper_probe, }; diff --git a/drivers/input/misc/ims-pcu.c b/drivers/input/misc/ims-pcu.c index 5a736397d9c8..719410feb84b 100644 --- a/drivers/input/misc/ims-pcu.c +++ b/drivers/input/misc/ims-pcu.c @@ -1566,6 +1566,7 @@ static int ims_pcu_buffers_alloc(struct ims_pcu *pcu) if (!pcu->urb_ctrl_buf) { dev_err(pcu->dev, "Failed to allocate memory for read buffer\n"); + error = -ENOMEM; goto err_free_urb_out_buf; } diff --git a/drivers/input/misc/max8925_onkey.c b/drivers/input/misc/max8925_onkey.c index eef41cfc054d..3809618e6a5d 100644 --- a/drivers/input/misc/max8925_onkey.c +++ b/drivers/input/misc/max8925_onkey.c @@ -26,6 +26,7 @@ #include <linux/interrupt.h> #include <linux/mfd/max8925.h> #include <linux/slab.h> +#include <linux/device.h> #define SW_INPUT (1 << 7) /* 0/1 -- up/down */ #define HARDRESET_EN (1 << 7) @@ -81,12 +82,14 @@ static int max8925_onkey_probe(struct platform_device *pdev) return -EINVAL; } - info = kzalloc(sizeof(struct max8925_onkey_info), GFP_KERNEL); - input = input_allocate_device(); - if (!info || !input) { - error = -ENOMEM; - goto err_free_mem; - } + info = devm_kzalloc(&pdev->dev, sizeof(struct max8925_onkey_info), + GFP_KERNEL); + if (!info) + return -ENOMEM; + + input = devm_input_allocate_device(&pdev->dev); + if (!input) + return -ENOMEM; info->idev = input; info->i2c = chip->i2c; @@ -100,55 +103,34 @@ static int max8925_onkey_probe(struct platform_device *pdev) input->dev.parent = &pdev->dev; input_set_capability(input, EV_KEY, KEY_POWER); - error = request_threaded_irq(irq[0], NULL, max8925_onkey_handler, - IRQF_ONESHOT, "onkey-down", info); + error = devm_request_threaded_irq(&pdev->dev, irq[0], NULL, + max8925_onkey_handler, IRQF_ONESHOT, + "onkey-down", info); if (error < 0) { dev_err(chip->dev, "Failed to request IRQ: #%d: %d\n", irq[0], error); - goto err_free_mem; + return error; } - error = request_threaded_irq(irq[1], NULL, max8925_onkey_handler, - IRQF_ONESHOT, "onkey-up", info); + error = devm_request_threaded_irq(&pdev->dev, irq[1], NULL, + max8925_onkey_handler, IRQF_ONESHOT, + "onkey-up", info); if (error < 0) { dev_err(chip->dev, "Failed to request IRQ: #%d: %d\n", irq[1], error); - goto err_free_irq0; + return error; } error = input_register_device(info->idev); if (error) { dev_err(chip->dev, "Can't register input device: %d\n", error); - goto err_free_irq1; + return error; } platform_set_drvdata(pdev, info); device_init_wakeup(&pdev->dev, 1); return 0; - -err_free_irq1: - free_irq(irq[1], info); -err_free_irq0: - free_irq(irq[0], info); -err_free_mem: - input_free_device(input); - kfree(info); - - return error; -} - -static int max8925_onkey_remove(struct platform_device *pdev) -{ - struct max8925_onkey_info *info = platform_get_drvdata(pdev); - struct max8925_chip *chip = dev_get_drvdata(pdev->dev.parent); - - free_irq(info->irq[0] + chip->irq_base, info); - free_irq(info->irq[1] + chip->irq_base, info); - input_unregister_device(info->idev); - kfree(info); - - return 0; } #ifdef CONFIG_PM_SLEEP @@ -190,7 +172,6 @@ static struct platform_driver max8925_onkey_driver = { .pm = &max8925_onkey_pm_ops, }, .probe = max8925_onkey_probe, - .remove = max8925_onkey_remove, }; module_platform_driver(max8925_onkey_driver); diff --git a/drivers/input/misc/max8997_haptic.c b/drivers/input/misc/max8997_haptic.c index 1fea5484941f..a363ebbd9cc0 100644 --- a/drivers/input/misc/max8997_haptic.c +++ b/drivers/input/misc/max8997_haptic.c @@ -181,11 +181,21 @@ static void max8997_haptic_enable(struct max8997_haptic *chip) } if (!chip->enabled) { - chip->enabled = true; - regulator_enable(chip->regulator); + error = regulator_enable(chip->regulator); + if (error) { + dev_err(chip->dev, "Failed to enable regulator\n"); + goto out; + } max8997_haptic_configure(chip); - if (chip->mode == MAX8997_EXTERNAL_MODE) - pwm_enable(chip->pwm); + if (chip->mode == MAX8997_EXTERNAL_MODE) { + error = pwm_enable(chip->pwm); + if (error) { + dev_err(chip->dev, "Failed to enable PWM\n"); + regulator_disable(chip->regulator); + goto out; + } + } + chip->enabled = true; } out: diff --git a/drivers/input/misc/pmic8xxx-pwrkey.c b/drivers/input/misc/pmic8xxx-pwrkey.c index 1cb8fda7a166..c91e3d33aea9 100644 --- a/drivers/input/misc/pmic8xxx-pwrkey.c +++ b/drivers/input/misc/pmic8xxx-pwrkey.c @@ -92,15 +92,15 @@ static int pmic8xxx_pwrkey_probe(struct platform_device *pdev) bool pull_up; if (of_property_read_u32(pdev->dev.of_node, "debounce", &kpd_delay)) - kpd_delay = 0; + kpd_delay = 15625; - pull_up = of_property_read_bool(pdev->dev.of_node, "pull-up"); - - if (kpd_delay > 62500) { + if (kpd_delay > 62500 || kpd_delay == 0) { dev_err(&pdev->dev, "invalid power key trigger delay\n"); return -EINVAL; } + pull_up = of_property_read_bool(pdev->dev.of_node, "pull-up"); + regmap = dev_get_regmap(pdev->dev.parent, NULL); if (!regmap) { dev_err(&pdev->dev, "failed to locate regmap for the device\n"); diff --git a/drivers/input/misc/rotary_encoder.c b/drivers/input/misc/rotary_encoder.c index 99b9e42aa748..93558a1c7f70 100644 --- a/drivers/input/misc/rotary_encoder.c +++ b/drivers/input/misc/rotary_encoder.c @@ -143,7 +143,7 @@ static irqreturn_t rotary_encoder_half_period_irq(int irq, void *dev_id) } #ifdef CONFIG_OF -static struct of_device_id rotary_encoder_of_match[] = { +static const struct of_device_id rotary_encoder_of_match[] = { { .compatible = "rotary-encoder", }, { }, }; diff --git a/drivers/input/misc/soc_button_array.c b/drivers/input/misc/soc_button_array.c index 20c80f543d5e..5a6334be30b8 100644 --- a/drivers/input/misc/soc_button_array.c +++ b/drivers/input/misc/soc_button_array.c @@ -17,7 +17,6 @@ #include <linux/acpi.h> #include <linux/gpio/consumer.h> #include <linux/gpio_keys.h> -#include <linux/input.h> #include <linux/platform_device.h> #include <linux/pnp.h> diff --git a/drivers/input/misc/twl6040-vibra.c b/drivers/input/misc/twl6040-vibra.c index 77dc23b94eb1..6d26eecc278c 100644 --- a/drivers/input/misc/twl6040-vibra.c +++ b/drivers/input/misc/twl6040-vibra.c @@ -262,7 +262,7 @@ static int twl6040_vibra_probe(struct platform_device *pdev) struct vibra_info *info; int vddvibl_uV = 0; int vddvibr_uV = 0; - int ret; + int error; twl6040_core_node = of_find_node_by_name(twl6040_core_dev->of_node, "vibra"); @@ -309,12 +309,12 @@ static int twl6040_vibra_probe(struct platform_device *pdev) mutex_init(&info->mutex); - ret = devm_request_threaded_irq(&pdev->dev, info->irq, NULL, - twl6040_vib_irq_handler, 0, - "twl6040_irq_vib", info); - if (ret) { - dev_err(info->dev, "VIB IRQ request failed: %d\n", ret); - return ret; + error = devm_request_threaded_irq(&pdev->dev, info->irq, NULL, + twl6040_vib_irq_handler, 0, + "twl6040_irq_vib", info); + if (error) { + dev_err(info->dev, "VIB IRQ request failed: %d\n", error); + return error; } info->supplies[0].supply = "vddvibl"; @@ -323,40 +323,40 @@ static int twl6040_vibra_probe(struct platform_device *pdev) * When booted with Device tree the regulators are attached to the * parent device (twl6040 MFD core) */ - ret = regulator_bulk_get(twl6040_core_dev, ARRAY_SIZE(info->supplies), - info->supplies); - if (ret) { - dev_err(info->dev, "couldn't get regulators %d\n", ret); - return ret; + error = devm_regulator_bulk_get(twl6040_core_dev, + ARRAY_SIZE(info->supplies), + info->supplies); + if (error) { + dev_err(info->dev, "couldn't get regulators %d\n", error); + return error; } if (vddvibl_uV) { - ret = regulator_set_voltage(info->supplies[0].consumer, - vddvibl_uV, vddvibl_uV); - if (ret) { + error = regulator_set_voltage(info->supplies[0].consumer, + vddvibl_uV, vddvibl_uV); + if (error) { dev_err(info->dev, "failed to set VDDVIBL volt %d\n", - ret); - goto err_regulator; + error); + return error; } } if (vddvibr_uV) { - ret = regulator_set_voltage(info->supplies[1].consumer, - vddvibr_uV, vddvibr_uV); - if (ret) { + error = regulator_set_voltage(info->supplies[1].consumer, + vddvibr_uV, vddvibr_uV); + if (error) { dev_err(info->dev, "failed to set VDDVIBR volt %d\n", - ret); - goto err_regulator; + error); + return error; } } INIT_WORK(&info->play_work, vibra_play_work); - info->input_dev = input_allocate_device(); - if (info->input_dev == NULL) { + info->input_dev = devm_input_allocate_device(&pdev->dev); + if (!info->input_dev) { dev_err(info->dev, "couldn't allocate input device\n"); - ret = -ENOMEM; - goto err_regulator; + return -ENOMEM; } input_set_drvdata(info->input_dev, info); @@ -367,44 +367,25 @@ static int twl6040_vibra_probe(struct platform_device *pdev) info->input_dev->close = twl6040_vibra_close; __set_bit(FF_RUMBLE, info->input_dev->ffbit); - ret = input_ff_create_memless(info->input_dev, NULL, vibra_play); - if (ret < 0) { + error = input_ff_create_memless(info->input_dev, NULL, vibra_play); + if (error) { dev_err(info->dev, "couldn't register vibrator to FF\n"); - goto err_ialloc; + return error; } - ret = input_register_device(info->input_dev); - if (ret < 0) { + error = input_register_device(info->input_dev); + if (error) { dev_err(info->dev, "couldn't register input device\n"); - goto err_iff; + return error; } platform_set_drvdata(pdev, info); return 0; - -err_iff: - input_ff_destroy(info->input_dev); -err_ialloc: - input_free_device(info->input_dev); -err_regulator: - regulator_bulk_free(ARRAY_SIZE(info->supplies), info->supplies); - return ret; -} - -static int twl6040_vibra_remove(struct platform_device *pdev) -{ - struct vibra_info *info = platform_get_drvdata(pdev); - - input_unregister_device(info->input_dev); - regulator_bulk_free(ARRAY_SIZE(info->supplies), info->supplies); - - return 0; } static struct platform_driver twl6040_vibra_driver = { .probe = twl6040_vibra_probe, - .remove = twl6040_vibra_remove, .driver = { .name = "twl6040-vibra", .owner = THIS_MODULE, diff --git a/drivers/input/mouse/Kconfig b/drivers/input/mouse/Kconfig index 6b8441f7bc32..366fc7ad5eb6 100644 --- a/drivers/input/mouse/Kconfig +++ b/drivers/input/mouse/Kconfig @@ -53,7 +53,7 @@ config MOUSE_PS2_LOGIPS2PP default y depends on MOUSE_PS2 help - Say Y here if you have a Logictech PS/2++ mouse connected to + Say Y here if you have a Logitech PS/2++ mouse connected to your system. If unsure, say Y. diff --git a/drivers/input/mouse/elantech.c b/drivers/input/mouse/elantech.c index b96e978a37b7..ee2a04d90d20 100644 --- a/drivers/input/mouse/elantech.c +++ b/drivers/input/mouse/elantech.c @@ -473,8 +473,15 @@ static void elantech_report_absolute_v3(struct psmouse *psmouse, input_report_key(dev, BTN_TOOL_FINGER, fingers == 1); input_report_key(dev, BTN_TOOL_DOUBLETAP, fingers == 2); input_report_key(dev, BTN_TOOL_TRIPLETAP, fingers == 3); - input_report_key(dev, BTN_LEFT, packet[0] & 0x01); - input_report_key(dev, BTN_RIGHT, packet[0] & 0x02); + + /* For clickpads map both buttons to BTN_LEFT */ + if (etd->fw_version & 0x001000) { + input_report_key(dev, BTN_LEFT, packet[0] & 0x03); + } else { + input_report_key(dev, BTN_LEFT, packet[0] & 0x01); + input_report_key(dev, BTN_RIGHT, packet[0] & 0x02); + } + input_report_abs(dev, ABS_PRESSURE, pres); input_report_abs(dev, ABS_TOOL_WIDTH, width); @@ -484,10 +491,17 @@ static void elantech_report_absolute_v3(struct psmouse *psmouse, static void elantech_input_sync_v4(struct psmouse *psmouse) { struct input_dev *dev = psmouse->dev; + struct elantech_data *etd = psmouse->private; unsigned char *packet = psmouse->packet; - input_report_key(dev, BTN_LEFT, packet[0] & 0x01); - input_report_key(dev, BTN_RIGHT, packet[0] & 0x02); + /* For clickpads map both buttons to BTN_LEFT */ + if (etd->fw_version & 0x001000) { + input_report_key(dev, BTN_LEFT, packet[0] & 0x03); + } else { + input_report_key(dev, BTN_LEFT, packet[0] & 0x01); + input_report_key(dev, BTN_RIGHT, packet[0] & 0x02); + } + input_mt_report_pointer_emulation(dev, true); input_sync(dev); } @@ -835,7 +849,7 @@ static int elantech_set_absolute_mode(struct psmouse *psmouse) if (etd->set_hw_resolution) etd->reg_10 = 0x0b; else - etd->reg_10 = 0x03; + etd->reg_10 = 0x01; if (elantech_write_reg(psmouse, 0x10, etd->reg_10)) rc = -1; @@ -1336,7 +1350,8 @@ static int elantech_reconnect(struct psmouse *psmouse) } /* - * Some hw_version 3 models go into error state when we try to set bit 3 of r10 + * Some hw_version 3 models go into error state when we try to set + * bit 3 and/or bit 1 of r10. */ static const struct dmi_system_id no_hw_res_dmi_table[] = { #if defined(CONFIG_DMI) && defined(CONFIG_X86) diff --git a/drivers/input/mouse/synaptics.c b/drivers/input/mouse/synaptics.c index c5ec703c727e..ec772d962f06 100644 --- a/drivers/input/mouse/synaptics.c +++ b/drivers/input/mouse/synaptics.c @@ -347,15 +347,6 @@ static int synaptics_resolution(struct psmouse *psmouse) unsigned char resp[3]; int i; - for (i = 0; min_max_pnpid_table[i].pnp_ids; i++) - if (matches_pnp_id(psmouse, min_max_pnpid_table[i].pnp_ids)) { - priv->x_min = min_max_pnpid_table[i].x_min; - priv->x_max = min_max_pnpid_table[i].x_max; - priv->y_min = min_max_pnpid_table[i].y_min; - priv->y_max = min_max_pnpid_table[i].y_max; - return 0; - } - if (SYN_ID_MAJOR(priv->identity) < 4) return 0; @@ -366,6 +357,16 @@ static int synaptics_resolution(struct psmouse *psmouse) } } + for (i = 0; min_max_pnpid_table[i].pnp_ids; i++) { + if (matches_pnp_id(psmouse, min_max_pnpid_table[i].pnp_ids)) { + priv->x_min = min_max_pnpid_table[i].x_min; + priv->x_max = min_max_pnpid_table[i].x_max; + priv->y_min = min_max_pnpid_table[i].y_min; + priv->y_max = min_max_pnpid_table[i].y_max; + return 0; + } + } + if (SYN_EXT_CAP_REQUESTS(priv->capabilities) >= 5 && SYN_CAP_MAX_DIMENSIONS(priv->ext_cap_0c)) { if (synaptics_send_cmd(psmouse, SYN_QUE_EXT_MAX_COORDS, resp)) { diff --git a/drivers/input/serio/apbps2.c b/drivers/input/serio/apbps2.c index 17e01a807ddc..98be824544a5 100644 --- a/drivers/input/serio/apbps2.c +++ b/drivers/input/serio/apbps2.c @@ -203,7 +203,7 @@ static int apbps2_of_remove(struct platform_device *of_dev) return 0; } -static struct of_device_id apbps2_of_match[] = { +static const struct of_device_id apbps2_of_match[] = { { .name = "GAISLER_APBPS2", }, { .name = "01_060", }, {} diff --git a/drivers/input/serio/olpc_apsp.c b/drivers/input/serio/olpc_apsp.c index 5d2fe7ece7ca..d906f3ebc8c8 100644 --- a/drivers/input/serio/olpc_apsp.c +++ b/drivers/input/serio/olpc_apsp.c @@ -262,7 +262,7 @@ static int olpc_apsp_remove(struct platform_device *pdev) return 0; } -static struct of_device_id olpc_apsp_dt_ids[] = { +static const struct of_device_id olpc_apsp_dt_ids[] = { { .compatible = "olpc,ap-sp", }, {} }; diff --git a/drivers/input/tablet/wacom_sys.c b/drivers/input/tablet/wacom_sys.c index 611fc3905d00..2c613cd41dd6 100644 --- a/drivers/input/tablet/wacom_sys.c +++ b/drivers/input/tablet/wacom_sys.c @@ -349,6 +349,7 @@ static int wacom_parse_hid(struct usb_interface *intf, break; case MTTPC: + case MTTPC_B: features->pktlen = WACOM_PKGLEN_MTTPC; break; @@ -380,6 +381,16 @@ static int wacom_parse_hid(struct usb_interface *intf, i += 12; break; + case MTTPC_B: + features->x_max = + get_unaligned_le16(&report[i + 3]); + features->x_phy = + get_unaligned_le16(&report[i + 6]); + features->unit = report[i - 5]; + features->unitExpo = report[i - 3]; + i += 9; + break; + default: features->x_max = get_unaligned_le16(&report[i + 3]); @@ -430,6 +441,14 @@ static int wacom_parse_hid(struct usb_interface *intf, i += 12; break; + case MTTPC_B: + features->y_max = + get_unaligned_le16(&report[i + 3]); + features->y_phy = + get_unaligned_le16(&report[i + 6]); + i += 9; + break; + default: features->y_max = features->x_max; diff --git a/drivers/input/tablet/wacom_wac.c b/drivers/input/tablet/wacom_wac.c index 4822c57a3756..977d05cd9e2e 100644 --- a/drivers/input/tablet/wacom_wac.c +++ b/drivers/input/tablet/wacom_wac.c @@ -484,6 +484,8 @@ static int wacom_intuos_inout(struct wacom_wac *wacom) input_report_key(input, BTN_TOUCH, 0); input_report_abs(input, ABS_PRESSURE, 0); input_report_abs(input, ABS_DISTANCE, wacom->features.distance_max); + if (features->quirks & WACOM_QUIRK_MULTI_INPUT) + wacom->shared->stylus_in_proximity = true; } /* Exit report */ @@ -928,12 +930,12 @@ static int wacom_24hdt_irq(struct wacom_wac *wacom) input_mt_report_slot_state(input, MT_TOOL_FINGER, touch); if (touch) { - int t_x = le16_to_cpup((__le16 *)&data[offset + 2]); - int c_x = le16_to_cpup((__le16 *)&data[offset + 4]); - int t_y = le16_to_cpup((__le16 *)&data[offset + 6]); - int c_y = le16_to_cpup((__le16 *)&data[offset + 8]); - int w = le16_to_cpup((__le16 *)&data[offset + 10]); - int h = le16_to_cpup((__le16 *)&data[offset + 12]); + int t_x = get_unaligned_le16(&data[offset + 2]); + int c_x = get_unaligned_le16(&data[offset + 4]); + int t_y = get_unaligned_le16(&data[offset + 6]); + int c_y = get_unaligned_le16(&data[offset + 8]); + int w = get_unaligned_le16(&data[offset + 10]); + int h = get_unaligned_le16(&data[offset + 12]); input_report_abs(input, ABS_MT_POSITION_X, t_x); input_report_abs(input, ABS_MT_POSITION_Y, t_y); @@ -962,7 +964,7 @@ static int wacom_mt_touch(struct wacom_wac *wacom) int x_offset = 0; /* MTTPC does not support Height and Width */ - if (wacom->features.type == MTTPC) + if (wacom->features.type == MTTPC || wacom->features.type == MTTPC_B) x_offset = -4; /* @@ -978,7 +980,7 @@ static int wacom_mt_touch(struct wacom_wac *wacom) for (i = 0; i < contacts_to_send; i++) { int offset = (WACOM_BYTES_PER_MT_PACKET + x_offset) * i + 3; bool touch = data[offset] & 0x1; - int id = le16_to_cpup((__le16 *)&data[offset + 1]); + int id = get_unaligned_le16(&data[offset + 1]); int slot = input_mt_get_slot_by_key(input, id); if (slot < 0) @@ -987,8 +989,8 @@ static int wacom_mt_touch(struct wacom_wac *wacom) input_mt_slot(input, slot); input_mt_report_slot_state(input, MT_TOOL_FINGER, touch); if (touch) { - int x = le16_to_cpup((__le16 *)&data[offset + x_offset + 7]); - int y = le16_to_cpup((__le16 *)&data[offset + x_offset + 9]); + int x = get_unaligned_le16(&data[offset + x_offset + 7]); + int y = get_unaligned_le16(&data[offset + x_offset + 9]); input_report_abs(input, ABS_MT_POSITION_X, x); input_report_abs(input, ABS_MT_POSITION_Y, y); } @@ -1047,6 +1049,10 @@ static int wacom_tpc_single_touch(struct wacom_wac *wacom, size_t len) prox = data[0] & 0x01; x = get_unaligned_le16(&data[1]); y = get_unaligned_le16(&data[3]); + } else if (len == WACOM_PKGLEN_TPC1FG_B) { + prox = data[2] & 0x01; + x = get_unaligned_le16(&data[3]); + y = get_unaligned_le16(&data[5]); } else { prox = data[1] & 0x01; x = le16_to_cpup((__le16 *)&data[2]); @@ -1110,6 +1116,9 @@ static int wacom_tpc_irq(struct wacom_wac *wacom, size_t len) case WACOM_PKGLEN_TPC2FG: return wacom_tpc_mt_touch(wacom); + case WACOM_PKGLEN_PENABLED: + return wacom_tpc_pen(wacom); + default: switch (data[0]) { case WACOM_REPORT_TPC1FG: @@ -1119,6 +1128,7 @@ static int wacom_tpc_irq(struct wacom_wac *wacom, size_t len) return wacom_tpc_single_touch(wacom, len); case WACOM_REPORT_TPCMT: + case WACOM_REPORT_TPCMT2: return wacom_mt_touch(wacom); case WACOM_REPORT_PENABLED: @@ -1461,6 +1471,7 @@ void wacom_wac_irq(struct wacom_wac *wacom_wac, size_t len) case TABLETPC2FG: case MTSCREEN: case MTTPC: + case MTTPC_B: sync = wacom_tpc_irq(wacom_wac, len); break; @@ -1565,10 +1576,10 @@ static void wacom_abs_set_axis(struct input_dev *input_dev, struct wacom_features *features = &wacom_wac->features; if (features->device_type == BTN_TOOL_PEN) { - input_set_abs_params(input_dev, ABS_X, 0, features->x_max, - features->x_fuzz, 0); - input_set_abs_params(input_dev, ABS_Y, 0, features->y_max, - features->y_fuzz, 0); + input_set_abs_params(input_dev, ABS_X, features->x_min, + features->x_max, features->x_fuzz, 0); + input_set_abs_params(input_dev, ABS_Y, features->y_min, + features->y_max, features->y_fuzz, 0); input_set_abs_params(input_dev, ABS_PRESSURE, 0, features->pressure_max, features->pressure_fuzz, 0); @@ -1802,6 +1813,7 @@ int wacom_setup_input_capabilities(struct input_dev *input_dev, case MTSCREEN: case MTTPC: + case MTTPC_B: case TABLETPC2FG: if (features->device_type == BTN_TOOL_FINGER) { unsigned int flags = INPUT_MT_DIRECT; @@ -2123,11 +2135,11 @@ static const struct wacom_features wacom_features_0x317 = 63, INTUOSPL, WACOM_INTUOS3_RES, WACOM_INTUOS3_RES, .touch_max = 16 }; static const struct wacom_features wacom_features_0xF4 = - { "Wacom Cintiq 24HD", WACOM_PKGLEN_INTUOS, 104480, 65600, 2047, - 63, WACOM_24HD, WACOM_INTUOS3_RES, WACOM_INTUOS3_RES }; + { "Wacom Cintiq 24HD", WACOM_PKGLEN_INTUOS, 104280, 65400, 2047, + 63, WACOM_24HD, WACOM_INTUOS3_RES, WACOM_INTUOS3_RES, 200, 200 }; static const struct wacom_features wacom_features_0xF8 = - { "Wacom Cintiq 24HD touch", WACOM_PKGLEN_INTUOS, 104480, 65600, 2047, /* Pen */ - 63, WACOM_24HD, WACOM_INTUOS3_RES, WACOM_INTUOS3_RES, + { "Wacom Cintiq 24HD touch", WACOM_PKGLEN_INTUOS, 104280, 65400, 2047, /* Pen */ + 63, WACOM_24HD, WACOM_INTUOS3_RES, WACOM_INTUOS3_RES, 200, 200, .oVid = USB_VENDOR_ID_WACOM, .oPid = 0xf6 }; static const struct wacom_features wacom_features_0xF6 = { "Wacom Cintiq 24HD touch", .type = WACOM_24HDT, /* Touch */ @@ -2142,8 +2154,8 @@ static const struct wacom_features wacom_features_0xC6 = { "Wacom Cintiq 12WX", WACOM_PKGLEN_INTUOS, 53020, 33440, 1023, 63, WACOM_BEE, WACOM_INTUOS3_RES, WACOM_INTUOS3_RES }; static const struct wacom_features wacom_features_0x304 = - { "Wacom Cintiq 13HD", WACOM_PKGLEN_INTUOS, 59552, 33848, 1023, - 63, WACOM_13HD, WACOM_INTUOS3_RES, WACOM_INTUOS3_RES }; + { "Wacom Cintiq 13HD", WACOM_PKGLEN_INTUOS, 59352, 33648, 1023, + 63, WACOM_13HD, WACOM_INTUOS3_RES, WACOM_INTUOS3_RES, 200, 200 }; static const struct wacom_features wacom_features_0xC7 = { "Wacom DTU1931", WACOM_PKGLEN_GRAPHIRE, 37832, 30305, 511, 0, PL, WACOM_INTUOS_RES, WACOM_INTUOS_RES }; @@ -2157,24 +2169,24 @@ static const struct wacom_features wacom_features_0xFB = { "Wacom DTU1031", WACOM_PKGLEN_DTUS, 22096, 13960, 511, 0, DTUS, WACOM_INTUOS_RES, WACOM_INTUOS_RES }; static const struct wacom_features wacom_features_0x57 = - { "Wacom DTK2241", WACOM_PKGLEN_INTUOS, 95840, 54260, 2047, - 63, DTK, WACOM_INTUOS3_RES, WACOM_INTUOS3_RES}; + { "Wacom DTK2241", WACOM_PKGLEN_INTUOS, 95640, 54060, 2047, + 63, DTK, WACOM_INTUOS3_RES, WACOM_INTUOS3_RES, 200, 200 }; static const struct wacom_features wacom_features_0x59 = /* Pen */ - { "Wacom DTH2242", WACOM_PKGLEN_INTUOS, 95840, 54260, 2047, - 63, DTK, WACOM_INTUOS3_RES, WACOM_INTUOS3_RES, + { "Wacom DTH2242", WACOM_PKGLEN_INTUOS, 95640, 54060, 2047, + 63, DTK, WACOM_INTUOS3_RES, WACOM_INTUOS3_RES, 200, 200, .oVid = USB_VENDOR_ID_WACOM, .oPid = 0x5D }; static const struct wacom_features wacom_features_0x5D = /* Touch */ { "Wacom DTH2242", .type = WACOM_24HDT, .oVid = USB_VENDOR_ID_WACOM, .oPid = 0x59, .touch_max = 10 }; static const struct wacom_features wacom_features_0xCC = - { "Wacom Cintiq 21UX2", WACOM_PKGLEN_INTUOS, 87200, 65600, 2047, - 63, WACOM_21UX2, WACOM_INTUOS3_RES, WACOM_INTUOS3_RES }; + { "Wacom Cintiq 21UX2", WACOM_PKGLEN_INTUOS, 87000, 65400, 2047, + 63, WACOM_21UX2, WACOM_INTUOS3_RES, WACOM_INTUOS3_RES, 200, 200 }; static const struct wacom_features wacom_features_0xFA = - { "Wacom Cintiq 22HD", WACOM_PKGLEN_INTUOS, 95840, 54260, 2047, - 63, WACOM_22HD, WACOM_INTUOS3_RES, WACOM_INTUOS3_RES }; + { "Wacom Cintiq 22HD", WACOM_PKGLEN_INTUOS, 95640, 54060, 2047, + 63, WACOM_22HD, WACOM_INTUOS3_RES, WACOM_INTUOS3_RES, 200, 200 }; static const struct wacom_features wacom_features_0x5B = - { "Wacom Cintiq 22HDT", WACOM_PKGLEN_INTUOS, 95840, 54260, 2047, - 63, WACOM_22HD, WACOM_INTUOS3_RES, WACOM_INTUOS3_RES, + { "Wacom Cintiq 22HDT", WACOM_PKGLEN_INTUOS, 95640, 54060, 2047, + 63, WACOM_22HD, WACOM_INTUOS3_RES, WACOM_INTUOS3_RES, 200, 200, .oVid = USB_VENDOR_ID_WACOM, .oPid = 0x5e }; static const struct wacom_features wacom_features_0x5E = { "Wacom Cintiq 22HDT", .type = WACOM_24HDT, @@ -2233,9 +2245,21 @@ static const struct wacom_features wacom_features_0x10E = static const struct wacom_features wacom_features_0x10F = { "Wacom ISDv4 10F", WACOM_PKGLEN_MTTPC, 27760, 15694, 255, 0, MTTPC, WACOM_INTUOS_RES, WACOM_INTUOS_RES }; +static const struct wacom_features wacom_features_0x116 = + { "Wacom ISDv4 116", WACOM_PKGLEN_GRAPHIRE, 26202, 16325, 255, + 0, TABLETPCE, WACOM_INTUOS_RES, WACOM_INTUOS_RES }; static const struct wacom_features wacom_features_0x4001 = { "Wacom ISDv4 4001", WACOM_PKGLEN_MTTPC, 26202, 16325, 255, 0, MTTPC, WACOM_INTUOS_RES, WACOM_INTUOS_RES }; +static const struct wacom_features wacom_features_0x4004 = + { "Wacom ISDv4 4004", WACOM_PKGLEN_MTTPC, 11060, 6220, 255, + 0, MTTPC_B, WACOM_INTUOS_RES, WACOM_INTUOS_RES }; +static const struct wacom_features wacom_features_0x5000 = + { "Wacom ISDv4 5000", WACOM_PKGLEN_MTTPC, 27848, 15752, 1023, + 0, MTTPC_B, WACOM_INTUOS_RES, WACOM_INTUOS_RES }; +static const struct wacom_features wacom_features_0x5002 = + { "Wacom ISDv4 5002", WACOM_PKGLEN_MTTPC, 29576, 16724, 1023, + 0, MTTPC_B, WACOM_INTUOS_RES, WACOM_INTUOS_RES }; static const struct wacom_features wacom_features_0x47 = { "Wacom Intuos2 6x8", WACOM_PKGLEN_INTUOS, 20320, 16240, 1023, 31, INTUOS, WACOM_INTUOS_RES, WACOM_INTUOS_RES }; @@ -2316,8 +2340,8 @@ static const struct wacom_features wacom_features_0x6004 = { "ISD-V4", WACOM_PKGLEN_GRAPHIRE, 12800, 8000, 255, 0, TABLETPC, WACOM_INTUOS_RES, WACOM_INTUOS_RES }; static const struct wacom_features wacom_features_0x0307 = - { "Wacom ISDv5 307", WACOM_PKGLEN_INTUOS, 59552, 33848, 2047, - 63, CINTIQ_HYBRID, WACOM_INTUOS3_RES, WACOM_INTUOS3_RES, + { "Wacom ISDv5 307", WACOM_PKGLEN_INTUOS, 59352, 33648, 2047, + 63, CINTIQ_HYBRID, WACOM_INTUOS3_RES, WACOM_INTUOS3_RES, 200, 200, .oVid = USB_VENDOR_ID_WACOM, .oPid = 0x309 }; static const struct wacom_features wacom_features_0x0309 = { "Wacom ISDv5 309", .type = WACOM_24HDT, /* Touch */ @@ -2447,6 +2471,7 @@ const struct usb_device_id wacom_ids[] = { { USB_DEVICE_WACOM(0x10D) }, { USB_DEVICE_WACOM(0x10E) }, { USB_DEVICE_WACOM(0x10F) }, + { USB_DEVICE_WACOM(0x116) }, { USB_DEVICE_WACOM(0x300) }, { USB_DEVICE_WACOM(0x301) }, { USB_DEVICE_DETAILED(0x302, USB_CLASS_HID, 0, 0) }, @@ -2457,6 +2482,9 @@ const struct usb_device_id wacom_ids[] = { { USB_DEVICE_DETAILED(0x315, USB_CLASS_HID, 0, 0) }, { USB_DEVICE_DETAILED(0x317, USB_CLASS_HID, 0, 0) }, { USB_DEVICE_WACOM(0x4001) }, + { USB_DEVICE_WACOM(0x4004) }, + { USB_DEVICE_WACOM(0x5000) }, + { USB_DEVICE_WACOM(0x5002) }, { USB_DEVICE_WACOM(0x47) }, { USB_DEVICE_WACOM(0xF4) }, { USB_DEVICE_WACOM(0xF8) }, diff --git a/drivers/input/tablet/wacom_wac.h b/drivers/input/tablet/wacom_wac.h index f69c0ebe7fa9..b2c9a9c1b551 100644 --- a/drivers/input/tablet/wacom_wac.h +++ b/drivers/input/tablet/wacom_wac.h @@ -22,6 +22,7 @@ #define WACOM_PKGLEN_BBFUN 9 #define WACOM_PKGLEN_INTUOS 10 #define WACOM_PKGLEN_TPC1FG 5 +#define WACOM_PKGLEN_TPC1FG_B 10 #define WACOM_PKGLEN_TPC2FG 14 #define WACOM_PKGLEN_BBTOUCH 20 #define WACOM_PKGLEN_BBTOUCH3 64 @@ -30,6 +31,7 @@ #define WACOM_PKGLEN_MTOUCH 62 #define WACOM_PKGLEN_MTTPC 40 #define WACOM_PKGLEN_DTUS 68 +#define WACOM_PKGLEN_PENABLED 8 /* wacom data size per MT contact */ #define WACOM_BYTES_PER_MT_PACKET 11 @@ -52,6 +54,7 @@ #define WACOM_REPORT_TPC1FG 6 #define WACOM_REPORT_TPC2FG 13 #define WACOM_REPORT_TPCMT 13 +#define WACOM_REPORT_TPCMT2 3 #define WACOM_REPORT_TPCHID 15 #define WACOM_REPORT_TPCST 16 #define WACOM_REPORT_DTUS 17 @@ -105,6 +108,7 @@ enum { TABLETPC2FG, MTSCREEN, MTTPC, + MTTPC_B, MAX_TYPE }; @@ -118,6 +122,8 @@ struct wacom_features { int type; int x_resolution; int y_resolution; + int x_min; + int y_min; int device_type; int x_phy; int y_phy; diff --git a/drivers/input/touchscreen/88pm860x-ts.c b/drivers/input/touchscreen/88pm860x-ts.c index 544e20c551f8..0d4a9fad4a78 100644 --- a/drivers/input/touchscreen/88pm860x-ts.c +++ b/drivers/input/touchscreen/88pm860x-ts.c @@ -16,6 +16,7 @@ #include <linux/input.h> #include <linux/mfd/88pm860x.h> #include <linux/slab.h> +#include <linux/device.h> #define MEAS_LEN (8) #define ACCURATE_BIT (12) @@ -234,16 +235,17 @@ static int pm860x_touch_probe(struct platform_device *pdev) if (ret) return ret; - touch = kzalloc(sizeof(struct pm860x_touch), GFP_KERNEL); - if (touch == NULL) + touch = devm_kzalloc(&pdev->dev, sizeof(struct pm860x_touch), + GFP_KERNEL); + if (!touch) return -ENOMEM; + platform_set_drvdata(pdev, touch); - touch->idev = input_allocate_device(); - if (touch->idev == NULL) { + touch->idev = devm_input_allocate_device(&pdev->dev); + if (!touch->idev) { dev_err(&pdev->dev, "Failed to allocate input device!\n"); - ret = -ENOMEM; - goto out; + return -ENOMEM; } touch->idev->name = "88pm860x-touch"; @@ -258,10 +260,11 @@ static int pm860x_touch_probe(struct platform_device *pdev) touch->res_x = res_x; input_set_drvdata(touch->idev, touch); - ret = request_threaded_irq(touch->irq, NULL, pm860x_touch_handler, - IRQF_ONESHOT, "touch", touch); + ret = devm_request_threaded_irq(&pdev->dev, touch->irq, NULL, + pm860x_touch_handler, IRQF_ONESHOT, + "touch", touch); if (ret < 0) - goto out_irq; + return ret; __set_bit(EV_ABS, touch->idev->evbit); __set_bit(ABS_X, touch->idev->absbit); @@ -279,28 +282,11 @@ static int pm860x_touch_probe(struct platform_device *pdev) ret = input_register_device(touch->idev); if (ret < 0) { dev_err(chip->dev, "Failed to register touch!\n"); - goto out_rg; + return ret; } platform_set_drvdata(pdev, touch); return 0; -out_rg: - free_irq(touch->irq, touch); -out_irq: - input_free_device(touch->idev); -out: - kfree(touch); - return ret; -} - -static int pm860x_touch_remove(struct platform_device *pdev) -{ - struct pm860x_touch *touch = platform_get_drvdata(pdev); - - input_unregister_device(touch->idev); - free_irq(touch->irq, touch); - kfree(touch); - return 0; } static struct platform_driver pm860x_touch_driver = { @@ -309,7 +295,6 @@ static struct platform_driver pm860x_touch_driver = { .owner = THIS_MODULE, }, .probe = pm860x_touch_probe, - .remove = pm860x_touch_remove, }; module_platform_driver(pm860x_touch_driver); diff --git a/drivers/input/touchscreen/Kconfig b/drivers/input/touchscreen/Kconfig index d4e5ab57909f..a23a94bb4bcb 100644 --- a/drivers/input/touchscreen/Kconfig +++ b/drivers/input/touchscreen/Kconfig @@ -11,6 +11,10 @@ menuconfig INPUT_TOUCHSCREEN if INPUT_TOUCHSCREEN +config OF_TOUCHSCREEN + def_tristate INPUT + depends on INPUT && OF + config TOUCHSCREEN_88PM860X tristate "Marvell 88PM860x touchscreen" depends on MFD_88PM860X @@ -89,6 +93,7 @@ config TOUCHSCREEN_AD7879_SPI config TOUCHSCREEN_ATMEL_MXT tristate "Atmel mXT I2C Touchscreen" depends on I2C + select FW_LOADER help Say Y here if you have Atmel mXT series I2C touchscreen, such as AT42QT602240/ATMXT224, connected to your system. @@ -846,7 +851,7 @@ config TOUCHSCREEN_TSC2007 config TOUCHSCREEN_W90X900 tristate "W90P910 touchscreen driver" - depends on HAVE_CLK + depends on ARCH_W90X900 help Say Y here if you have a W90P910 based touchscreen. @@ -885,6 +890,17 @@ config TOUCHSCREEN_STMPE To compile this driver as a module, choose M here: the module will be called stmpe-ts. +config TOUCHSCREEN_SUN4I + tristate "Allwinner sun4i resistive touchscreen controller support" + depends on ARCH_SUNXI || COMPILE_TEST + depends on HWMON + help + This selects support for the resistive touchscreen controller + found on Allwinner sunxi SoCs. + + To compile this driver as a module, choose M here: the + module will be called sun4i-ts. + config TOUCHSCREEN_SUR40 tristate "Samsung SUR40 (Surface 2.0/PixelSense) touchscreen" depends on USB diff --git a/drivers/input/touchscreen/Makefile b/drivers/input/touchscreen/Makefile index 03f12a1f2218..126479d8c29a 100644 --- a/drivers/input/touchscreen/Makefile +++ b/drivers/input/touchscreen/Makefile @@ -6,6 +6,7 @@ wm97xx-ts-y := wm97xx-core.o +obj-$(CONFIG_OF_TOUCHSCREEN) += of_touchscreen.o obj-$(CONFIG_TOUCHSCREEN_88PM860X) += 88pm860x-ts.o obj-$(CONFIG_TOUCHSCREEN_AD7877) += ad7877.o obj-$(CONFIG_TOUCHSCREEN_AD7879) += ad7879.o @@ -53,6 +54,7 @@ obj-$(CONFIG_TOUCHSCREEN_PIXCIR) += pixcir_i2c_ts.o obj-$(CONFIG_TOUCHSCREEN_S3C2410) += s3c2410_ts.o obj-$(CONFIG_TOUCHSCREEN_ST1232) += st1232.o obj-$(CONFIG_TOUCHSCREEN_STMPE) += stmpe-ts.o +obj-$(CONFIG_TOUCHSCREEN_SUN4I) += sun4i-ts.o obj-$(CONFIG_TOUCHSCREEN_SUR40) += sur40.o obj-$(CONFIG_TOUCHSCREEN_TI_AM335X_TSC) += ti_am335x_tsc.o obj-$(CONFIG_TOUCHSCREEN_TOUCHIT213) += touchit213.o diff --git a/drivers/input/touchscreen/ad7877.c b/drivers/input/touchscreen/ad7877.c index 6793c85903ae..523865daa1d3 100644 --- a/drivers/input/touchscreen/ad7877.c +++ b/drivers/input/touchscreen/ad7877.c @@ -210,11 +210,6 @@ static bool gpio3; module_param(gpio3, bool, 0); MODULE_PARM_DESC(gpio3, "If gpio3 is set to 1 AUX3 acts as GPIO3"); -/* - * ad7877_read/write are only used for initial setup and for sysfs controls. - * The main traffic is done using spi_async() in the interrupt handler. - */ - static int ad7877_read(struct spi_device *spi, u16 reg) { struct ser_req *req; diff --git a/drivers/input/touchscreen/ads7846.c b/drivers/input/touchscreen/ads7846.c index 7f8aa981500d..da201b8e37dc 100644 --- a/drivers/input/touchscreen/ads7846.c +++ b/drivers/input/touchscreen/ads7846.c @@ -706,7 +706,7 @@ static void ads7846_read_state(struct ads7846 *ts) m = &ts->msg[msg_idx]; error = spi_sync(ts->spi, m); if (error) { - dev_err(&ts->spi->dev, "spi_async --> %d\n", error); + dev_err(&ts->spi->dev, "spi_sync --> %d\n", error); packet->tc.ignore = true; return; } diff --git a/drivers/input/touchscreen/atmel_mxt_ts.c b/drivers/input/touchscreen/atmel_mxt_ts.c index a70400754e92..6e0b4a2120d3 100644 --- a/drivers/input/touchscreen/atmel_mxt_ts.c +++ b/drivers/input/touchscreen/atmel_mxt_ts.c @@ -2,6 +2,8 @@ * Atmel maXTouch Touchscreen driver * * Copyright (C) 2010 Samsung Electronics Co.Ltd + * Copyright (C) 2012 Google, Inc. + * * Author: Joonyoung Shim <jy0922.shim@samsung.com> * * This program is free software; you can redistribute it and/or modify it @@ -12,6 +14,8 @@ */ #include <linux/module.h> +#include <linux/init.h> +#include <linux/completion.h> #include <linux/delay.h> #include <linux/firmware.h> #include <linux/i2c.h> @@ -25,12 +29,6 @@ #define MXT_VER_21 21 #define MXT_VER_22 22 -/* Slave addresses */ -#define MXT_APP_LOW 0x4a -#define MXT_APP_HIGH 0x4b -#define MXT_BOOT_LOW 0x24 -#define MXT_BOOT_HIGH 0x25 - /* Firmware */ #define MXT_FW_NAME "maxtouch.fw" @@ -83,6 +81,9 @@ #define MXT_COMMAND_REPORTALL 3 #define MXT_COMMAND_DIAGNOSTIC 5 +/* Define for T6 status byte */ +#define MXT_T6_STATUS_RESET (1 << 7) + /* MXT_GEN_POWER_T7 field */ #define MXT_POWER_IDLEACQINT 0 #define MXT_POWER_ACTVACQINT 1 @@ -99,33 +100,26 @@ /* MXT_TOUCH_MULTI_T9 field */ #define MXT_TOUCH_CTRL 0 -#define MXT_TOUCH_XORIGIN 1 -#define MXT_TOUCH_YORIGIN 2 -#define MXT_TOUCH_XSIZE 3 -#define MXT_TOUCH_YSIZE 4 -#define MXT_TOUCH_BLEN 6 -#define MXT_TOUCH_TCHTHR 7 -#define MXT_TOUCH_TCHDI 8 -#define MXT_TOUCH_ORIENT 9 -#define MXT_TOUCH_MOVHYSTI 11 -#define MXT_TOUCH_MOVHYSTN 12 -#define MXT_TOUCH_NUMTOUCH 14 -#define MXT_TOUCH_MRGHYST 15 -#define MXT_TOUCH_MRGTHR 16 -#define MXT_TOUCH_AMPHYST 17 -#define MXT_TOUCH_XRANGE_LSB 18 -#define MXT_TOUCH_XRANGE_MSB 19 -#define MXT_TOUCH_YRANGE_LSB 20 -#define MXT_TOUCH_YRANGE_MSB 21 -#define MXT_TOUCH_XLOCLIP 22 -#define MXT_TOUCH_XHICLIP 23 -#define MXT_TOUCH_YLOCLIP 24 -#define MXT_TOUCH_YHICLIP 25 -#define MXT_TOUCH_XEDGECTRL 26 -#define MXT_TOUCH_XEDGEDIST 27 -#define MXT_TOUCH_YEDGECTRL 28 -#define MXT_TOUCH_YEDGEDIST 29 -#define MXT_TOUCH_JUMPLIMIT 30 +#define MXT_T9_ORIENT 9 +#define MXT_T9_RANGE 18 + +/* MXT_TOUCH_MULTI_T9 status */ +#define MXT_T9_UNGRIP (1 << 0) +#define MXT_T9_SUPPRESS (1 << 1) +#define MXT_T9_AMP (1 << 2) +#define MXT_T9_VECTOR (1 << 3) +#define MXT_T9_MOVE (1 << 4) +#define MXT_T9_RELEASE (1 << 5) +#define MXT_T9_PRESS (1 << 6) +#define MXT_T9_DETECT (1 << 7) + +struct t9_range { + u16 x; + u16 y; +} __packed; + +/* MXT_TOUCH_MULTI_T9 orient */ +#define MXT_T9_ORIENT_SWITCH (1 << 0) /* MXT_PROCI_GRIPFACE_T20 field */ #define MXT_GRIPFACE_CTRL 0 @@ -174,17 +168,16 @@ /* Define for MXT_GEN_COMMAND_T6 */ #define MXT_BOOT_VALUE 0xa5 +#define MXT_RESET_VALUE 0x01 #define MXT_BACKUP_VALUE 0x55 + +/* Delay times */ #define MXT_BACKUP_TIME 50 /* msec */ #define MXT_RESET_TIME 200 /* msec */ - -#define MXT_FWRESET_TIME 175 /* msec */ - -/* MXT_SPT_GPIOPWM_T19 field */ -#define MXT_GPIO0_MASK 0x04 -#define MXT_GPIO1_MASK 0x08 -#define MXT_GPIO2_MASK 0x10 -#define MXT_GPIO3_MASK 0x20 +#define MXT_RESET_TIMEOUT 3000 /* msec */ +#define MXT_CRC_TIMEOUT 1000 /* msec */ +#define MXT_FW_RESET_TIME 3000 /* msec */ +#define MXT_FW_CHG_TIMEOUT 300 /* msec */ /* Command to unlock bootloader */ #define MXT_UNLOCK_CMD_MSB 0xaa @@ -198,21 +191,8 @@ #define MXT_FRAME_CRC_PASS 0x04 #define MXT_APP_CRC_FAIL 0x40 /* valid 7 8 bit only */ #define MXT_BOOT_STATUS_MASK 0x3f - -/* Touch status */ -#define MXT_UNGRIP (1 << 0) -#define MXT_SUPPRESS (1 << 1) -#define MXT_AMP (1 << 2) -#define MXT_VECTOR (1 << 3) -#define MXT_MOVE (1 << 4) -#define MXT_RELEASE (1 << 5) -#define MXT_PRESS (1 << 6) -#define MXT_DETECT (1 << 7) - -/* Touch orient bits */ -#define MXT_XY_SWITCH (1 << 0) -#define MXT_X_INVERT (1 << 1) -#define MXT_Y_INVERT (1 << 2) +#define MXT_BOOT_EXTENDED_ID (1 << 5) +#define MXT_BOOT_ID_MASK 0x1f /* Touchscreen absolute values */ #define MXT_MAX_AREA 0xff @@ -232,8 +212,8 @@ struct mxt_info { struct mxt_object { u8 type; u16 start_address; - u8 size; /* Size of each instance - 1 */ - u8 instances; /* Number of instances - 1 */ + u8 size_minus_one; + u8 instances_minus_one; u8 num_report_ids; } __packed; @@ -250,19 +230,40 @@ struct mxt_data { const struct mxt_platform_data *pdata; struct mxt_object *object_table; struct mxt_info info; - bool is_tp; - unsigned int irq; unsigned int max_x; unsigned int max_y; + bool in_bootloader; + u32 config_crc; + u8 bootloader_addr; /* Cached parameters from object table */ u8 T6_reportid; + u16 T6_address; u8 T9_reportid_min; u8 T9_reportid_max; u8 T19_reportid; + + /* for fw update in bootloader */ + struct completion bl_completion; + + /* for reset handling */ + struct completion reset_completion; + + /* for config update handling */ + struct completion crc_completion; }; +static size_t mxt_obj_size(const struct mxt_object *obj) +{ + return obj->size_minus_one + 1; +} + +static size_t mxt_obj_instances(const struct mxt_object *obj) +{ + return obj->instances_minus_one + 1; +} + static bool mxt_object_readable(unsigned int type) { switch (type) { @@ -334,60 +335,190 @@ static void mxt_dump_message(struct device *dev, message->reportid, 7, message->message); } -static int mxt_check_bootloader(struct i2c_client *client, - unsigned int state) +static int mxt_wait_for_completion(struct mxt_data *data, + struct completion *comp, + unsigned int timeout_ms) +{ + struct device *dev = &data->client->dev; + unsigned long timeout = msecs_to_jiffies(timeout_ms); + long ret; + + ret = wait_for_completion_interruptible_timeout(comp, timeout); + if (ret < 0) { + return ret; + } else if (ret == 0) { + dev_err(dev, "Wait for completion timed out.\n"); + return -ETIMEDOUT; + } + return 0; +} + +static int mxt_bootloader_read(struct mxt_data *data, + u8 *val, unsigned int count) +{ + int ret; + struct i2c_msg msg; + + msg.addr = data->bootloader_addr; + msg.flags = data->client->flags & I2C_M_TEN; + msg.flags |= I2C_M_RD; + msg.len = count; + msg.buf = val; + + ret = i2c_transfer(data->client->adapter, &msg, 1); + + if (ret == 1) { + ret = 0; + } else { + ret = ret < 0 ? ret : -EIO; + dev_err(&data->client->dev, "%s: i2c recv failed (%d)\n", + __func__, ret); + } + + return ret; +} + +static int mxt_bootloader_write(struct mxt_data *data, + const u8 * const val, unsigned int count) +{ + int ret; + struct i2c_msg msg; + + msg.addr = data->bootloader_addr; + msg.flags = data->client->flags & I2C_M_TEN; + msg.len = count; + msg.buf = (u8 *)val; + + ret = i2c_transfer(data->client->adapter, &msg, 1); + if (ret == 1) { + ret = 0; + } else { + ret = ret < 0 ? ret : -EIO; + dev_err(&data->client->dev, "%s: i2c send failed (%d)\n", + __func__, ret); + } + + return ret; +} + +static int mxt_lookup_bootloader_address(struct mxt_data *data) +{ + u8 appmode = data->client->addr; + u8 bootloader; + + switch (appmode) { + case 0x4a: + case 0x4b: + case 0x4c: + case 0x4d: + case 0x5a: + case 0x5b: + bootloader = appmode - 0x26; + break; + default: + dev_err(&data->client->dev, + "Appmode i2c address 0x%02x not found\n", + appmode); + return -EINVAL; + } + + data->bootloader_addr = bootloader; + return 0; +} + +static u8 mxt_get_bootloader_version(struct mxt_data *data, u8 val) +{ + struct device *dev = &data->client->dev; + u8 buf[3]; + + if (val & MXT_BOOT_EXTENDED_ID) { + if (mxt_bootloader_read(data, &buf[0], 3) != 0) { + dev_err(dev, "%s: i2c failure\n", __func__); + return val; + } + + dev_dbg(dev, "Bootloader ID:%d Version:%d\n", buf[1], buf[2]); + + return buf[0]; + } else { + dev_dbg(dev, "Bootloader ID:%d\n", val & MXT_BOOT_ID_MASK); + + return val; + } +} + +static int mxt_check_bootloader(struct mxt_data *data, unsigned int state) { + struct device *dev = &data->client->dev; u8 val; + int ret; recheck: - if (i2c_master_recv(client, &val, 1) != 1) { - dev_err(&client->dev, "%s: i2c recv failed\n", __func__); - return -EIO; + if (state != MXT_WAITING_BOOTLOAD_CMD) { + /* + * In application update mode, the interrupt + * line signals state transitions. We must wait for the + * CHG assertion before reading the status byte. + * Once the status byte has been read, the line is deasserted. + */ + ret = mxt_wait_for_completion(data, &data->bl_completion, + MXT_FW_CHG_TIMEOUT); + if (ret) { + /* + * TODO: handle -ERESTARTSYS better by terminating + * fw update process before returning to userspace + * by writing length 0x000 to device (iff we are in + * WAITING_FRAME_DATA state). + */ + dev_err(dev, "Update wait error %d\n", ret); + return ret; + } } + ret = mxt_bootloader_read(data, &val, 1); + if (ret) + return ret; + + if (state == MXT_WAITING_BOOTLOAD_CMD) + val = mxt_get_bootloader_version(data, val); + switch (state) { case MXT_WAITING_BOOTLOAD_CMD: case MXT_WAITING_FRAME_DATA: val &= ~MXT_BOOT_STATUS_MASK; break; case MXT_FRAME_CRC_PASS: - if (val == MXT_FRAME_CRC_CHECK) + if (val == MXT_FRAME_CRC_CHECK) { goto recheck; + } else if (val == MXT_FRAME_CRC_FAIL) { + dev_err(dev, "Bootloader CRC fail\n"); + return -EINVAL; + } break; default: return -EINVAL; } if (val != state) { - dev_err(&client->dev, "Unvalid bootloader mode state\n"); + dev_err(dev, "Invalid bootloader state %02X != %02X\n", + val, state); return -EINVAL; } return 0; } -static int mxt_unlock_bootloader(struct i2c_client *client) +static int mxt_unlock_bootloader(struct mxt_data *data) { + int ret; u8 buf[2]; buf[0] = MXT_UNLOCK_CMD_LSB; buf[1] = MXT_UNLOCK_CMD_MSB; - if (i2c_master_send(client, buf, 2) != 2) { - dev_err(&client->dev, "%s: i2c send failed\n", __func__); - return -EIO; - } - - return 0; -} - -static int mxt_fw_write(struct i2c_client *client, - const u8 *data, unsigned int frame_size) -{ - if (i2c_master_send(client, data, frame_size) != frame_size) { - dev_err(&client->dev, "%s: i2c send failed\n", __func__); - return -EIO; - } + ret = mxt_bootloader_write(data, buf, 2); + if (ret) + return ret; return 0; } @@ -427,11 +558,6 @@ static int __mxt_read_reg(struct i2c_client *client, return ret; } -static int mxt_read_reg(struct i2c_client *client, u16 reg, u8 *val) -{ - return __mxt_read_reg(client, reg, 1, val); -} - static int __mxt_write_reg(struct i2c_client *client, u16 reg, u16 len, const void *val) { @@ -479,7 +605,7 @@ mxt_get_object(struct mxt_data *data, u8 type) return object; } - dev_err(&data->client->dev, "Invalid object type\n"); + dev_err(&data->client->dev, "Invalid object type T%u\n", type); return NULL; } @@ -505,7 +631,7 @@ static int mxt_write_object(struct mxt_data *data, u16 reg; object = mxt_get_object(data, type); - if (!object || offset >= object->size + 1) + if (!object || offset >= mxt_obj_size(object)) return -EINVAL; reg = object->start_address; @@ -515,18 +641,25 @@ static int mxt_write_object(struct mxt_data *data, static void mxt_input_button(struct mxt_data *data, struct mxt_message *message) { struct input_dev *input = data->input_dev; + const struct mxt_platform_data *pdata = data->pdata; bool button; int i; /* Active-low switch */ - for (i = 0; i < MXT_NUM_GPIO; i++) { - if (data->pdata->key_map[i] == KEY_RESERVED) + for (i = 0; i < pdata->t19_num_keys; i++) { + if (pdata->t19_keymap[i] == KEY_RESERVED) continue; - button = !(message->message[0] & MXT_GPIO0_MASK << i); - input_report_key(input, data->pdata->key_map[i], button); + button = !(message->message[0] & (1 << i)); + input_report_key(input, pdata->t19_keymap[i], button); } } +static void mxt_input_sync(struct input_dev *input_dev) +{ + input_mt_report_pointer_emulation(input_dev, false); + input_sync(input_dev); +} + static void mxt_input_touchevent(struct mxt_data *data, struct mxt_message *message, int id) { @@ -536,44 +669,60 @@ static void mxt_input_touchevent(struct mxt_data *data, int x; int y; int area; - int pressure; + int amplitude; x = (message->message[1] << 4) | ((message->message[3] >> 4) & 0xf); y = (message->message[2] << 4) | ((message->message[3] & 0xf)); + + /* Handle 10/12 bit switching */ if (data->max_x < 1024) - x = x >> 2; + x >>= 2; if (data->max_y < 1024) - y = y >> 2; + y >>= 2; area = message->message[4]; - pressure = message->message[5]; + amplitude = message->message[5]; dev_dbg(dev, "[%u] %c%c%c%c%c%c%c%c x: %5u y: %5u area: %3u amp: %3u\n", id, - (status & MXT_DETECT) ? 'D' : '.', - (status & MXT_PRESS) ? 'P' : '.', - (status & MXT_RELEASE) ? 'R' : '.', - (status & MXT_MOVE) ? 'M' : '.', - (status & MXT_VECTOR) ? 'V' : '.', - (status & MXT_AMP) ? 'A' : '.', - (status & MXT_SUPPRESS) ? 'S' : '.', - (status & MXT_UNGRIP) ? 'U' : '.', - x, y, area, pressure); + (status & MXT_T9_DETECT) ? 'D' : '.', + (status & MXT_T9_PRESS) ? 'P' : '.', + (status & MXT_T9_RELEASE) ? 'R' : '.', + (status & MXT_T9_MOVE) ? 'M' : '.', + (status & MXT_T9_VECTOR) ? 'V' : '.', + (status & MXT_T9_AMP) ? 'A' : '.', + (status & MXT_T9_SUPPRESS) ? 'S' : '.', + (status & MXT_T9_UNGRIP) ? 'U' : '.', + x, y, area, amplitude); input_mt_slot(input_dev, id); - input_mt_report_slot_state(input_dev, MT_TOOL_FINGER, - status & MXT_DETECT); - if (status & MXT_DETECT) { + if (status & MXT_T9_DETECT) { + /* + * Multiple bits may be set if the host is slow to read + * the status messages, indicating all the events that + * have happened. + */ + if (status & MXT_T9_RELEASE) { + input_mt_report_slot_state(input_dev, + MT_TOOL_FINGER, 0); + mxt_input_sync(input_dev); + } + + /* Touch active */ + input_mt_report_slot_state(input_dev, MT_TOOL_FINGER, 1); input_report_abs(input_dev, ABS_MT_POSITION_X, x); input_report_abs(input_dev, ABS_MT_POSITION_Y, y); - input_report_abs(input_dev, ABS_MT_PRESSURE, pressure); + input_report_abs(input_dev, ABS_MT_PRESSURE, amplitude); input_report_abs(input_dev, ABS_MT_TOUCH_MAJOR, area); + } else { + /* Touch no longer active, close out slot */ + input_mt_report_slot_state(input_dev, MT_TOOL_FINGER, 0); } } -static unsigned mxt_extract_T6_csum(const u8 *csum) +static u16 mxt_extract_T6_csum(const u8 *csum) { return csum[0] | (csum[1] << 8) | (csum[2] << 16); } @@ -584,28 +733,37 @@ static bool mxt_is_T9_message(struct mxt_data *data, struct mxt_message *msg) return (id >= data->T9_reportid_min && id <= data->T9_reportid_max); } -static irqreturn_t mxt_interrupt(int irq, void *dev_id) +static irqreturn_t mxt_process_messages_until_invalid(struct mxt_data *data) { - struct mxt_data *data = dev_id; struct mxt_message message; const u8 *payload = &message.message[0]; struct device *dev = &data->client->dev; u8 reportid; bool update_input = false; + u32 crc; do { if (mxt_read_message(data, &message)) { dev_err(dev, "Failed to read message\n"); - goto end; + return IRQ_NONE; } reportid = message.reportid; if (reportid == data->T6_reportid) { u8 status = payload[0]; - unsigned csum = mxt_extract_T6_csum(&payload[1]); + + crc = mxt_extract_T6_csum(&payload[1]); + if (crc != data->config_crc) { + data->config_crc = crc; + complete(&data->crc_completion); + } + dev_dbg(dev, "Status: %02x Config Checksum: %06x\n", - status, csum); + status, data->config_crc); + + if (status & MXT_T6_STATUS_RESET) + complete(&data->reset_completion); } else if (mxt_is_T9_message(data, &message)) { int id = reportid - data->T9_reportid_min; mxt_input_touchevent(data, &message, id); @@ -618,15 +776,96 @@ static irqreturn_t mxt_interrupt(int irq, void *dev_id) } } while (reportid != 0xff); - if (update_input) { - input_mt_report_pointer_emulation(data->input_dev, false); - input_sync(data->input_dev); - } + if (update_input) + mxt_input_sync(data->input_dev); -end: return IRQ_HANDLED; } +static irqreturn_t mxt_interrupt(int irq, void *dev_id) +{ + struct mxt_data *data = dev_id; + + if (data->in_bootloader) { + /* bootloader state transition completion */ + complete(&data->bl_completion); + return IRQ_HANDLED; + } + + return mxt_process_messages_until_invalid(data); +} + +static int mxt_t6_command(struct mxt_data *data, u16 cmd_offset, + u8 value, bool wait) +{ + u16 reg; + u8 command_register; + int timeout_counter = 0; + int ret; + + reg = data->T6_address + cmd_offset; + + ret = mxt_write_reg(data->client, reg, value); + if (ret) + return ret; + + if (!wait) + return 0; + + do { + msleep(20); + ret = __mxt_read_reg(data->client, reg, 1, &command_register); + if (ret) + return ret; + } while (command_register != 0 && timeout_counter++ <= 100); + + if (timeout_counter > 100) { + dev_err(&data->client->dev, "Command failed!\n"); + return -EIO; + } + + return 0; +} + +static int mxt_soft_reset(struct mxt_data *data) +{ + struct device *dev = &data->client->dev; + int ret = 0; + + dev_info(dev, "Resetting chip\n"); + + reinit_completion(&data->reset_completion); + + ret = mxt_t6_command(data, MXT_COMMAND_RESET, MXT_RESET_VALUE, false); + if (ret) + return ret; + + ret = mxt_wait_for_completion(data, &data->reset_completion, + MXT_RESET_TIMEOUT); + if (ret) + return ret; + + return 0; +} + +static void mxt_update_crc(struct mxt_data *data, u8 cmd, u8 value) +{ + /* + * On failure, CRC is set to 0 and config will always be + * downloaded. + */ + data->config_crc = 0; + reinit_completion(&data->crc_completion); + + mxt_t6_command(data, cmd, value, true); + + /* + * Wait for crc message. On failure, CRC is set to 0 and config will + * always be downloaded. + */ + mxt_wait_for_completion(data, &data->crc_completion, MXT_CRC_TIMEOUT); +} + static int mxt_check_reg_init(struct mxt_data *data) { const struct mxt_platform_data *pdata = data->pdata; @@ -641,13 +880,23 @@ static int mxt_check_reg_init(struct mxt_data *data) return 0; } + mxt_update_crc(data, MXT_COMMAND_REPORTALL, 1); + + if (data->config_crc == pdata->config_crc) { + dev_info(dev, "Config CRC 0x%06X: OK\n", data->config_crc); + return 0; + } + + dev_info(dev, "Config CRC 0x%06X: does not match 0x%06X\n", + data->config_crc, pdata->config_crc); + for (i = 0; i < data->info.object_num; i++) { object = data->object_table + i; if (!mxt_object_writable(object->type)) continue; - size = (object->size + 1) * (object->instances + 1); + size = mxt_obj_size(object) * mxt_obj_instances(object); if (index + size > pdata->config_length) { dev_err(dev, "Not enough config data!\n"); return -EINVAL; @@ -660,6 +909,14 @@ static int mxt_check_reg_init(struct mxt_data *data) index += size; } + mxt_update_crc(data, MXT_COMMAND_BACKUPNV, MXT_BACKUP_VALUE); + + ret = mxt_soft_reset(data); + if (ret) + return ret; + + dev_info(dev, "Config successfully updated\n"); + return 0; } @@ -685,54 +942,6 @@ static int mxt_make_highchg(struct mxt_data *data) return 0; } -static void mxt_handle_pdata(struct mxt_data *data) -{ - const struct mxt_platform_data *pdata = data->pdata; - u8 voltage; - - /* Set touchscreen lines */ - mxt_write_object(data, MXT_TOUCH_MULTI_T9, MXT_TOUCH_XSIZE, - pdata->x_line); - mxt_write_object(data, MXT_TOUCH_MULTI_T9, MXT_TOUCH_YSIZE, - pdata->y_line); - - /* Set touchscreen orient */ - mxt_write_object(data, MXT_TOUCH_MULTI_T9, MXT_TOUCH_ORIENT, - pdata->orient); - - /* Set touchscreen burst length */ - mxt_write_object(data, MXT_TOUCH_MULTI_T9, - MXT_TOUCH_BLEN, pdata->blen); - - /* Set touchscreen threshold */ - mxt_write_object(data, MXT_TOUCH_MULTI_T9, - MXT_TOUCH_TCHTHR, pdata->threshold); - - /* Set touchscreen resolution */ - mxt_write_object(data, MXT_TOUCH_MULTI_T9, - MXT_TOUCH_XRANGE_LSB, (pdata->x_size - 1) & 0xff); - mxt_write_object(data, MXT_TOUCH_MULTI_T9, - MXT_TOUCH_XRANGE_MSB, (pdata->x_size - 1) >> 8); - mxt_write_object(data, MXT_TOUCH_MULTI_T9, - MXT_TOUCH_YRANGE_LSB, (pdata->y_size - 1) & 0xff); - mxt_write_object(data, MXT_TOUCH_MULTI_T9, - MXT_TOUCH_YRANGE_MSB, (pdata->y_size - 1) >> 8); - - /* Set touchscreen voltage */ - if (pdata->voltage) { - if (pdata->voltage < MXT_VOLTAGE_DEFAULT) { - voltage = (MXT_VOLTAGE_DEFAULT - pdata->voltage) / - MXT_VOLTAGE_STEP; - voltage = 0xff - voltage + 1; - } else - voltage = (pdata->voltage - MXT_VOLTAGE_DEFAULT) / - MXT_VOLTAGE_STEP; - - mxt_write_object(data, MXT_SPT_CTECONFIG_T28, - MXT_CTE_VOLTAGE, voltage); - } -} - static int mxt_get_info(struct mxt_data *data) { struct i2c_client *client = data->client; @@ -772,7 +981,7 @@ static int mxt_get_object_table(struct mxt_data *data) if (object->num_report_ids) { min_id = reportid; reportid += object->num_report_ids * - (object->instances + 1); + mxt_obj_instances(object); max_id = reportid - 1; } else { min_id = 0; @@ -780,13 +989,15 @@ static int mxt_get_object_table(struct mxt_data *data) } dev_dbg(&data->client->dev, - "Type %2d Start %3d Size %3d Instances %2d ReportIDs %3u : %3u\n", - object->type, object->start_address, object->size + 1, - object->instances + 1, min_id, max_id); + "T%u Start:%u Size:%zu Instances:%zu Report IDs:%u-%u\n", + object->type, object->start_address, + mxt_obj_size(object), mxt_obj_instances(object), + min_id, max_id); switch (object->type) { case MXT_GEN_COMMAND_T6: data->T6_reportid = min_id; + data->T6_address = object->start_address; break; case MXT_TOUCH_MULTI_T9: data->T9_reportid_min = min_id; @@ -811,12 +1022,59 @@ static void mxt_free_object_table(struct mxt_data *data) data->T19_reportid = 0; } +static int mxt_read_t9_resolution(struct mxt_data *data) +{ + struct i2c_client *client = data->client; + int error; + struct t9_range range; + unsigned char orient; + struct mxt_object *object; + + object = mxt_get_object(data, MXT_TOUCH_MULTI_T9); + if (!object) + return -EINVAL; + + error = __mxt_read_reg(client, + object->start_address + MXT_T9_RANGE, + sizeof(range), &range); + if (error) + return error; + + le16_to_cpus(&range.x); + le16_to_cpus(&range.y); + + error = __mxt_read_reg(client, + object->start_address + MXT_T9_ORIENT, + 1, &orient); + if (error) + return error; + + /* Handle default values */ + if (range.x == 0) + range.x = 1023; + + if (range.y == 0) + range.y = 1023; + + if (orient & MXT_T9_ORIENT_SWITCH) { + data->max_x = range.y; + data->max_y = range.x; + } else { + data->max_x = range.x; + data->max_y = range.y; + } + + dev_dbg(&client->dev, + "Touchscreen size X%uY%u\n", data->max_x, data->max_y); + + return 0; +} + static int mxt_initialize(struct mxt_data *data) { struct i2c_client *client = data->client; struct mxt_info *info = &data->info; int error; - u8 val; error = mxt_get_info(data); if (error) @@ -832,47 +1090,29 @@ static int mxt_initialize(struct mxt_data *data) /* Get object table information */ error = mxt_get_object_table(data); - if (error) + if (error) { + dev_err(&client->dev, "Error %d reading object table\n", error); goto err_free_object_table; + } /* Check register init values */ error = mxt_check_reg_init(data); - if (error) - goto err_free_object_table; - - mxt_handle_pdata(data); - - /* Backup to memory */ - mxt_write_object(data, MXT_GEN_COMMAND_T6, - MXT_COMMAND_BACKUPNV, - MXT_BACKUP_VALUE); - msleep(MXT_BACKUP_TIME); - - /* Soft reset */ - mxt_write_object(data, MXT_GEN_COMMAND_T6, - MXT_COMMAND_RESET, 1); - msleep(MXT_RESET_TIME); - - /* Update matrix size at info struct */ - error = mxt_read_reg(client, MXT_MATRIX_X_SIZE, &val); - if (error) + if (error) { + dev_err(&client->dev, "Error %d initializing configuration\n", + error); goto err_free_object_table; - info->matrix_xsize = val; + } - error = mxt_read_reg(client, MXT_MATRIX_Y_SIZE, &val); - if (error) + error = mxt_read_t9_resolution(data); + if (error) { + dev_err(&client->dev, "Failed to initialize T9 resolution\n"); goto err_free_object_table; - info->matrix_ysize = val; - - dev_info(&client->dev, - "Family ID: %u Variant ID: %u Major.Minor.Build: %u.%u.%02X\n", - info->family_id, info->variant_id, info->version >> 4, - info->version & 0xf, info->build); + } dev_info(&client->dev, - "Matrix X Size: %u Matrix Y Size: %u Object Num: %u\n", - info->matrix_xsize, info->matrix_ysize, - info->object_num); + "Family: %u Variant: %u Firmware V%u.%u.%02X Objects: %u\n", + info->family_id, info->variant_id, info->version >> 4, + info->version & 0xf, info->build, info->object_num); return 0; @@ -881,20 +1121,6 @@ err_free_object_table: return error; } -static void mxt_calc_resolution(struct mxt_data *data) -{ - unsigned int max_x = data->pdata->x_size - 1; - unsigned int max_y = data->pdata->y_size - 1; - - if (data->pdata->orient & MXT_XY_SWITCH) { - data->max_x = max_y; - data->max_y = max_x; - } else { - data->max_x = max_x; - data->max_y = max_y; - } -} - /* Firmware Version is returned as Major.Minor.Build */ static ssize_t mxt_fw_version_show(struct device *dev, struct device_attribute *attr, char *buf) @@ -921,11 +1147,11 @@ static ssize_t mxt_show_instance(char *buf, int count, { int i; - if (object->instances > 0) + if (mxt_obj_instances(object) > 1) count += scnprintf(buf + count, PAGE_SIZE - count, "Instance %u\n", instance); - for (i = 0; i < object->size + 1; i++) + for (i = 0; i < mxt_obj_size(object); i++) count += scnprintf(buf + count, PAGE_SIZE - count, "\t[%2u]: %02x (%d)\n", i, val[i], val[i]); count += scnprintf(buf + count, PAGE_SIZE - count, "\n"); @@ -958,8 +1184,8 @@ static ssize_t mxt_object_show(struct device *dev, count += scnprintf(buf + count, PAGE_SIZE - count, "T%u:\n", object->type); - for (j = 0; j < object->instances + 1; j++) { - u16 size = object->size + 1; + for (j = 0; j < mxt_obj_instances(object); j++) { + u16 size = mxt_obj_size(object); u16 addr = object->start_address + j * size; error = __mxt_read_reg(data->client, addr, size, obuf); @@ -975,13 +1201,38 @@ done: return error ?: count; } +static int mxt_check_firmware_format(struct device *dev, + const struct firmware *fw) +{ + unsigned int pos = 0; + char c; + + while (pos < fw->size) { + c = *(fw->data + pos); + + if (c < '0' || (c > '9' && c < 'A') || c > 'F') + return 0; + + pos++; + } + + /* + * To convert file try: + * xxd -r -p mXTXXX__APP_VX-X-XX.enc > maxtouch.fw + */ + dev_err(dev, "Aborting: firmware file must be in binary format\n"); + + return -EINVAL; +} + static int mxt_load_fw(struct device *dev, const char *fn) { struct mxt_data *data = dev_get_drvdata(dev); - struct i2c_client *client = data->client; const struct firmware *fw = NULL; unsigned int frame_size; unsigned int pos = 0; + unsigned int retry = 0; + unsigned int frame = 0; int ret; ret = request_firmware(&fw, fn, dev); @@ -990,59 +1241,91 @@ static int mxt_load_fw(struct device *dev, const char *fn) return ret; } + /* Check for incorrect enc file */ + ret = mxt_check_firmware_format(dev, fw); + if (ret) + goto release_firmware; + + ret = mxt_lookup_bootloader_address(data); + if (ret) + goto release_firmware; + /* Change to the bootloader mode */ - mxt_write_object(data, MXT_GEN_COMMAND_T6, - MXT_COMMAND_RESET, MXT_BOOT_VALUE); + data->in_bootloader = true; + + ret = mxt_t6_command(data, MXT_COMMAND_RESET, MXT_BOOT_VALUE, false); + if (ret) + goto release_firmware; + msleep(MXT_RESET_TIME); - /* Change to slave address of bootloader */ - if (client->addr == MXT_APP_LOW) - client->addr = MXT_BOOT_LOW; - else - client->addr = MXT_BOOT_HIGH; + reinit_completion(&data->bl_completion); - ret = mxt_check_bootloader(client, MXT_WAITING_BOOTLOAD_CMD); + ret = mxt_check_bootloader(data, MXT_WAITING_BOOTLOAD_CMD); if (ret) - goto out; + goto disable_irq; /* Unlock bootloader */ - mxt_unlock_bootloader(client); + mxt_unlock_bootloader(data); while (pos < fw->size) { - ret = mxt_check_bootloader(client, - MXT_WAITING_FRAME_DATA); + ret = mxt_check_bootloader(data, MXT_WAITING_FRAME_DATA); if (ret) - goto out; + goto disable_irq; frame_size = ((*(fw->data + pos) << 8) | *(fw->data + pos + 1)); - /* We should add 2 at frame size as the the firmware data is not - * included the CRC bytes. - */ + /* Take account of CRC bytes */ frame_size += 2; /* Write one frame to device */ - mxt_fw_write(client, fw->data + pos, frame_size); - - ret = mxt_check_bootloader(client, - MXT_FRAME_CRC_PASS); + ret = mxt_bootloader_write(data, fw->data + pos, frame_size); if (ret) - goto out; + goto disable_irq; - pos += frame_size; + ret = mxt_check_bootloader(data, MXT_FRAME_CRC_PASS); + if (ret) { + retry++; - dev_dbg(dev, "Updated %d bytes / %zd bytes\n", pos, fw->size); + /* Back off by 20ms per retry */ + msleep(retry * 20); + + if (retry > 20) { + dev_err(dev, "Retry count exceeded\n"); + goto disable_irq; + } + } else { + retry = 0; + pos += frame_size; + frame++; + } + + if (frame % 50 == 0) + dev_dbg(dev, "Sent %d frames, %d/%zd bytes\n", + frame, pos, fw->size); } -out: - release_firmware(fw); + /* Wait for flash. */ + ret = mxt_wait_for_completion(data, &data->bl_completion, + MXT_FW_RESET_TIME); + if (ret) + goto disable_irq; - /* Change to slave address of application */ - if (client->addr == MXT_BOOT_LOW) - client->addr = MXT_APP_LOW; - else - client->addr = MXT_APP_HIGH; + dev_dbg(dev, "Sent %d frames, %d bytes\n", frame, pos); + /* + * Wait for device to reset. Some bootloader versions do not assert + * the CHG line after bootloading has finished, so ignore potential + * errors. + */ + mxt_wait_for_completion(data, &data->bl_completion, MXT_FW_RESET_TIME); + + data->in_bootloader = false; + +disable_irq: + disable_irq(data->irq); +release_firmware: + release_firmware(fw); return ret; } @@ -1053,28 +1336,23 @@ static ssize_t mxt_update_fw_store(struct device *dev, struct mxt_data *data = dev_get_drvdata(dev); int error; - disable_irq(data->irq); - error = mxt_load_fw(dev, MXT_FW_NAME); if (error) { dev_err(dev, "The firmware update failed(%d)\n", error); count = error; } else { - dev_dbg(dev, "The firmware update succeeded\n"); - - /* Wait for reset */ - msleep(MXT_FWRESET_TIME); + dev_info(dev, "The firmware update succeeded\n"); mxt_free_object_table(data); mxt_initialize(data); - } - enable_irq(data->irq); + enable_irq(data->irq); - error = mxt_make_highchg(data); - if (error) - return error; + error = mxt_make_highchg(data); + if (error) + return error; + } return count; } @@ -1134,6 +1412,8 @@ static int mxt_probe(struct i2c_client *client, struct input_dev *input_dev; int error; unsigned int num_mt_slots; + unsigned int mt_flags = 0; + int i; if (!pdata) return -EINVAL; @@ -1146,10 +1426,7 @@ static int mxt_probe(struct i2c_client *client, goto err_free_mem; } - data->is_tp = pdata && pdata->is_tp; - - input_dev->name = (data->is_tp) ? "Atmel maXTouch Touchpad" : - "Atmel maXTouch Touchscreen"; + input_dev->name = "Atmel maXTouch Touchscreen"; snprintf(data->phys, sizeof(data->phys), "i2c-%u-%04x/input0", client->adapter->nr, client->addr); @@ -1165,7 +1442,9 @@ static int mxt_probe(struct i2c_client *client, data->pdata = pdata; data->irq = client->irq; - mxt_calc_resolution(data); + init_completion(&data->bl_completion); + init_completion(&data->reset_completion); + init_completion(&data->crc_completion); error = mxt_initialize(data); if (error) @@ -1175,20 +1454,15 @@ static int mxt_probe(struct i2c_client *client, __set_bit(EV_KEY, input_dev->evbit); __set_bit(BTN_TOUCH, input_dev->keybit); - if (data->is_tp) { - int i; - __set_bit(INPUT_PROP_POINTER, input_dev->propbit); + if (pdata->t19_num_keys) { __set_bit(INPUT_PROP_BUTTONPAD, input_dev->propbit); - for (i = 0; i < MXT_NUM_GPIO; i++) - if (pdata->key_map[i] != KEY_RESERVED) - __set_bit(pdata->key_map[i], input_dev->keybit); + for (i = 0; i < pdata->t19_num_keys; i++) + if (pdata->t19_keymap[i] != KEY_RESERVED) + input_set_capability(input_dev, EV_KEY, + pdata->t19_keymap[i]); - __set_bit(BTN_TOOL_FINGER, input_dev->keybit); - __set_bit(BTN_TOOL_DOUBLETAP, input_dev->keybit); - __set_bit(BTN_TOOL_TRIPLETAP, input_dev->keybit); - __set_bit(BTN_TOOL_QUADTAP, input_dev->keybit); - __set_bit(BTN_TOOL_QUINTTAP, input_dev->keybit); + mt_flags |= INPUT_MT_POINTER; input_abs_set_res(input_dev, ABS_X, MXT_PIXELS_PER_MM); input_abs_set_res(input_dev, ABS_Y, MXT_PIXELS_PER_MM); @@ -1196,6 +1470,8 @@ static int mxt_probe(struct i2c_client *client, MXT_PIXELS_PER_MM); input_abs_set_res(input_dev, ABS_MT_POSITION_Y, MXT_PIXELS_PER_MM); + + input_dev->name = "Atmel maXTouch Touchpad"; } /* For single touch */ @@ -1208,7 +1484,7 @@ static int mxt_probe(struct i2c_client *client, /* For multi touch */ num_mt_slots = data->T9_reportid_max - data->T9_reportid_min + 1; - error = input_mt_init_slots(input_dev, num_mt_slots, 0); + error = input_mt_init_slots(input_dev, num_mt_slots, mt_flags); if (error) goto err_free_object; input_set_abs_params(input_dev, ABS_MT_TOUCH_MAJOR, @@ -1236,12 +1512,18 @@ static int mxt_probe(struct i2c_client *client, goto err_free_irq; error = input_register_device(input_dev); - if (error) + if (error) { + dev_err(&client->dev, "Error %d registering input device\n", + error); goto err_free_irq; + } error = sysfs_create_group(&client->dev.kobj, &mxt_attr_group); - if (error) + if (error) { + dev_err(&client->dev, "Failure %d creating sysfs group\n", + error); goto err_unregister_device; + } return 0; @@ -1294,11 +1576,7 @@ static int mxt_resume(struct device *dev) struct mxt_data *data = i2c_get_clientdata(client); struct input_dev *input_dev = data->input_dev; - /* Soft reset */ - mxt_write_object(data, MXT_GEN_COMMAND_T6, - MXT_COMMAND_RESET, 1); - - msleep(MXT_RESET_TIME); + mxt_soft_reset(data); mutex_lock(&input_dev->mutex); diff --git a/drivers/input/touchscreen/auo-pixcir-ts.c b/drivers/input/touchscreen/auo-pixcir-ts.c index d3f9f6b0f9b7..7f3c94787787 100644 --- a/drivers/input/touchscreen/auo-pixcir-ts.c +++ b/drivers/input/touchscreen/auo-pixcir-ts.c @@ -679,7 +679,7 @@ static const struct i2c_device_id auo_pixcir_idtable[] = { MODULE_DEVICE_TABLE(i2c, auo_pixcir_idtable); #ifdef CONFIG_OF -static struct of_device_id auo_pixcir_ts_dt_idtable[] = { +static const struct of_device_id auo_pixcir_ts_dt_idtable[] = { { .compatible = "auo,auo_pixcir_ts" }, {}, }; diff --git a/drivers/input/touchscreen/da9034-ts.c b/drivers/input/touchscreen/da9034-ts.c index 8ccf7bb4028a..cf6f4b31db4d 100644 --- a/drivers/input/touchscreen/da9034-ts.c +++ b/drivers/input/touchscreen/da9034-ts.c @@ -301,10 +301,11 @@ static int da9034_touch_probe(struct platform_device *pdev) struct da9034_touch_pdata *pdata = dev_get_platdata(&pdev->dev); struct da9034_touch *touch; struct input_dev *input_dev; - int ret; + int error; - touch = kzalloc(sizeof(struct da9034_touch), GFP_KERNEL); - if (touch == NULL) { + touch = devm_kzalloc(&pdev->dev, sizeof(struct da9034_touch), + GFP_KERNEL); + if (!touch) { dev_err(&pdev->dev, "failed to allocate driver data\n"); return -ENOMEM; } @@ -315,18 +316,18 @@ static int da9034_touch_probe(struct platform_device *pdev) touch->interval_ms = pdata->interval_ms; touch->x_inverted = pdata->x_inverted; touch->y_inverted = pdata->y_inverted; - } else + } else { /* fallback into default */ touch->interval_ms = 10; + } INIT_DELAYED_WORK(&touch->tsi_work, da9034_tsi_work); touch->notifier.notifier_call = da9034_touch_notifier; - input_dev = input_allocate_device(); + input_dev = devm_input_allocate_device(&pdev->dev); if (!input_dev) { dev_err(&pdev->dev, "failed to allocate input device\n"); - ret = -ENOMEM; - goto err_free_touch; + return -ENOMEM; } input_dev->name = pdev->name; @@ -346,26 +347,9 @@ static int da9034_touch_probe(struct platform_device *pdev) touch->input_dev = input_dev; input_set_drvdata(input_dev, touch); - ret = input_register_device(input_dev); - if (ret) - goto err_free_input; - - platform_set_drvdata(pdev, touch); - return 0; - -err_free_input: - input_free_device(input_dev); -err_free_touch: - kfree(touch); - return ret; -} - -static int da9034_touch_remove(struct platform_device *pdev) -{ - struct da9034_touch *touch = platform_get_drvdata(pdev); - - input_unregister_device(touch->input_dev); - kfree(touch); + error = input_register_device(input_dev); + if (error) + return error; return 0; } @@ -376,7 +360,6 @@ static struct platform_driver da9034_touch_driver = { .owner = THIS_MODULE, }, .probe = da9034_touch_probe, - .remove = da9034_touch_remove, }; module_platform_driver(da9034_touch_driver); diff --git a/drivers/input/touchscreen/edt-ft5x06.c b/drivers/input/touchscreen/edt-ft5x06.c index f8815bebc9ef..d4f33992ad8c 100644 --- a/drivers/input/touchscreen/edt-ft5x06.c +++ b/drivers/input/touchscreen/edt-ft5x06.c @@ -271,7 +271,7 @@ static int edt_ft5x06_register_write(struct edt_ft5x06_ts_data *tsdata, wrbuf[0] = addr; wrbuf[1] = value; - return edt_ft5x06_ts_readwrite(tsdata->client, 3, + return edt_ft5x06_ts_readwrite(tsdata->client, 2, wrbuf, 0, NULL); default: diff --git a/drivers/input/touchscreen/egalax_ts.c b/drivers/input/touchscreen/egalax_ts.c index e6bcb13680b2..c8057847d71d 100644 --- a/drivers/input/touchscreen/egalax_ts.c +++ b/drivers/input/touchscreen/egalax_ts.c @@ -262,7 +262,7 @@ static int egalax_ts_resume(struct device *dev) static SIMPLE_DEV_PM_OPS(egalax_ts_pm_ops, egalax_ts_suspend, egalax_ts_resume); -static struct of_device_id egalax_ts_dt_ids[] = { +static const struct of_device_id egalax_ts_dt_ids[] = { { .compatible = "eeti,egalax_ts" }, { /* sentinel */ } }; diff --git a/drivers/input/touchscreen/intel-mid-touch.c b/drivers/input/touchscreen/intel-mid-touch.c index 4f6b156144e9..c38ca4a7e386 100644 --- a/drivers/input/touchscreen/intel-mid-touch.c +++ b/drivers/input/touchscreen/intel-mid-touch.c @@ -36,6 +36,7 @@ #include <linux/irq.h> #include <linux/delay.h> #include <asm/intel_scu_ipc.h> +#include <linux/device.h> /* PMIC Interrupt registers */ #define PMIC_REG_ID1 0x00 /* PMIC ID1 register */ @@ -580,12 +581,17 @@ static int mrstouch_probe(struct platform_device *pdev) return -EINVAL; } - tsdev = kzalloc(sizeof(struct mrstouch_dev), GFP_KERNEL); - input = input_allocate_device(); - if (!tsdev || !input) { + tsdev = devm_kzalloc(&pdev->dev, sizeof(struct mrstouch_dev), + GFP_KERNEL); + if (!tsdev) { dev_err(&pdev->dev, "unable to allocate memory\n"); - err = -ENOMEM; - goto err_free_mem; + return -ENOMEM; + } + + input = devm_input_allocate_device(&pdev->dev); + if (!input) { + dev_err(&pdev->dev, "unable to allocate input device\n"); + return -ENOMEM; } tsdev->dev = &pdev->dev; @@ -598,7 +604,7 @@ static int mrstouch_probe(struct platform_device *pdev) err = mrstouch_adc_init(tsdev); if (err) { dev_err(&pdev->dev, "ADC initialization failed\n"); - goto err_free_mem; + return err; } input->name = "mrst_touchscreen"; @@ -618,38 +624,20 @@ static int mrstouch_probe(struct platform_device *pdev) input_set_abs_params(tsdev->input, ABS_PRESSURE, MRST_PRESSURE_MIN, MRST_PRESSURE_MAX, 0, 0); - err = request_threaded_irq(tsdev->irq, NULL, mrstouch_pendet_irq, - IRQF_ONESHOT, "mrstouch", tsdev); + err = devm_request_threaded_irq(&pdev->dev, tsdev->irq, NULL, + mrstouch_pendet_irq, IRQF_ONESHOT, + "mrstouch", tsdev); if (err) { dev_err(tsdev->dev, "unable to allocate irq\n"); - goto err_free_mem; + return err; } err = input_register_device(tsdev->input); if (err) { dev_err(tsdev->dev, "unable to register input device\n"); - goto err_free_irq; + return err; } - platform_set_drvdata(pdev, tsdev); - return 0; - -err_free_irq: - free_irq(tsdev->irq, tsdev); -err_free_mem: - input_free_device(input); - kfree(tsdev); - return err; -} - -static int mrstouch_remove(struct platform_device *pdev) -{ - struct mrstouch_dev *tsdev = platform_get_drvdata(pdev); - - free_irq(tsdev->irq, tsdev); - input_unregister_device(tsdev->input); - kfree(tsdev); - return 0; } @@ -659,7 +647,6 @@ static struct platform_driver mrstouch_driver = { .owner = THIS_MODULE, }, .probe = mrstouch_probe, - .remove = mrstouch_remove, }; module_platform_driver(mrstouch_driver); diff --git a/drivers/input/touchscreen/lpc32xx_ts.c b/drivers/input/touchscreen/lpc32xx_ts.c index 2058253b55d9..bb47d3442a35 100644 --- a/drivers/input/touchscreen/lpc32xx_ts.c +++ b/drivers/input/touchscreen/lpc32xx_ts.c @@ -384,7 +384,7 @@ static const struct dev_pm_ops lpc32xx_ts_pm_ops = { #endif #ifdef CONFIG_OF -static struct of_device_id lpc32xx_tsc_of_match[] = { +static const struct of_device_id lpc32xx_tsc_of_match[] = { { .compatible = "nxp,lpc3220-tsc", }, { }, }; diff --git a/drivers/input/touchscreen/mcs5000_ts.c b/drivers/input/touchscreen/mcs5000_ts.c index 647e36f5930e..00510a9836b3 100644 --- a/drivers/input/touchscreen/mcs5000_ts.c +++ b/drivers/input/touchscreen/mcs5000_ts.c @@ -161,10 +161,9 @@ static irqreturn_t mcs5000_ts_interrupt(int irq, void *dev_id) return IRQ_HANDLED; } -static void mcs5000_ts_phys_init(struct mcs5000_ts_data *data) +static void mcs5000_ts_phys_init(struct mcs5000_ts_data *data, + const struct mcs_platform_data *platform_data) { - const struct mcs_platform_data *platform_data = - data->platform_data; struct i2c_client *client = data->client; /* Touch reset & sleep mode */ @@ -187,28 +186,32 @@ static void mcs5000_ts_phys_init(struct mcs5000_ts_data *data) } static int mcs5000_ts_probe(struct i2c_client *client, - const struct i2c_device_id *id) + const struct i2c_device_id *id) { + const struct mcs_platform_data *pdata; struct mcs5000_ts_data *data; struct input_dev *input_dev; - int ret; + int error; - if (!dev_get_platdata(&client->dev)) + pdata = dev_get_platdata(&client->dev); + if (!pdata) return -EINVAL; - data = kzalloc(sizeof(struct mcs5000_ts_data), GFP_KERNEL); - input_dev = input_allocate_device(); - if (!data || !input_dev) { + data = devm_kzalloc(&client->dev, sizeof(*data), GFP_KERNEL); + if (!data) { dev_err(&client->dev, "Failed to allocate memory\n"); - ret = -ENOMEM; - goto err_free_mem; + return -ENOMEM; } data->client = client; - data->input_dev = input_dev; - data->platform_data = dev_get_platdata(&client->dev); - input_dev->name = "MELPAS MCS-5000 Touchscreen"; + input_dev = devm_input_allocate_device(&client->dev); + if (!input_dev) { + dev_err(&client->dev, "Failed to allocate input device\n"); + return -ENOMEM; + } + + input_dev->name = "MELFAS MCS-5000 Touchscreen"; input_dev->id.bustype = BUS_I2C; input_dev->dev.parent = &client->dev; @@ -219,44 +222,30 @@ static int mcs5000_ts_probe(struct i2c_client *client, input_set_abs_params(input_dev, ABS_Y, 0, MCS5000_MAX_YC, 0, 0); input_set_drvdata(input_dev, data); + data->input_dev = input_dev; - if (data->platform_data->cfg_pin) - data->platform_data->cfg_pin(); - - ret = request_threaded_irq(client->irq, NULL, mcs5000_ts_interrupt, - IRQF_TRIGGER_LOW | IRQF_ONESHOT, "mcs5000_ts", data); + if (pdata->cfg_pin) + pdata->cfg_pin(); - if (ret < 0) { + error = devm_request_threaded_irq(&client->dev, client->irq, + NULL, mcs5000_ts_interrupt, + IRQF_TRIGGER_LOW | IRQF_ONESHOT, + "mcs5000_ts", data); + if (error) { dev_err(&client->dev, "Failed to register interrupt\n"); - goto err_free_mem; + return error; } - ret = input_register_device(data->input_dev); - if (ret < 0) - goto err_free_irq; + error = input_register_device(data->input_dev); + if (error) { + dev_err(&client->dev, "Failed to register input device\n"); + return error; + } - mcs5000_ts_phys_init(data); + mcs5000_ts_phys_init(data, pdata); i2c_set_clientdata(client, data); return 0; - -err_free_irq: - free_irq(client->irq, data); -err_free_mem: - input_free_device(input_dev); - kfree(data); - return ret; -} - -static int mcs5000_ts_remove(struct i2c_client *client) -{ - struct mcs5000_ts_data *data = i2c_get_clientdata(client); - - free_irq(client->irq, data); - input_unregister_device(data->input_dev); - kfree(data); - - return 0; } #ifdef CONFIG_PM @@ -274,14 +263,15 @@ static int mcs5000_ts_resume(struct device *dev) { struct i2c_client *client = to_i2c_client(dev); struct mcs5000_ts_data *data = i2c_get_clientdata(client); + const struct mcs_platform_data *pdata = dev_get_platdata(dev); - mcs5000_ts_phys_init(data); + mcs5000_ts_phys_init(data, pdata); return 0; } +#endif static SIMPLE_DEV_PM_OPS(mcs5000_ts_pm, mcs5000_ts_suspend, mcs5000_ts_resume); -#endif static const struct i2c_device_id mcs5000_ts_id[] = { { "mcs5000_ts", 0 }, @@ -291,12 +281,9 @@ MODULE_DEVICE_TABLE(i2c, mcs5000_ts_id); static struct i2c_driver mcs5000_ts_driver = { .probe = mcs5000_ts_probe, - .remove = mcs5000_ts_remove, .driver = { .name = "mcs5000_ts", -#ifdef CONFIG_PM .pm = &mcs5000_ts_pm, -#endif }, .id_table = mcs5000_ts_id, }; diff --git a/drivers/input/touchscreen/mms114.c b/drivers/input/touchscreen/mms114.c index 8a598c065391..372bbf7658fe 100644 --- a/drivers/input/touchscreen/mms114.c +++ b/drivers/input/touchscreen/mms114.c @@ -456,7 +456,7 @@ static int mms114_probe(struct i2c_client *client, data->input_dev = input_dev; data->pdata = pdata; - input_dev->name = "MELPAS MMS114 Touchscreen"; + input_dev->name = "MELFAS MMS114 Touchscreen"; input_dev->id.bustype = BUS_I2C; input_dev->dev.parent = &client->dev; input_dev->open = mms114_input_open; @@ -570,7 +570,7 @@ static const struct i2c_device_id mms114_id[] = { MODULE_DEVICE_TABLE(i2c, mms114_id); #ifdef CONFIG_OF -static struct of_device_id mms114_dt_match[] = { +static const struct of_device_id mms114_dt_match[] = { { .compatible = "melfas,mms114" }, { } }; diff --git a/drivers/input/touchscreen/of_touchscreen.c b/drivers/input/touchscreen/of_touchscreen.c new file mode 100644 index 000000000000..f8f9b84230b1 --- /dev/null +++ b/drivers/input/touchscreen/of_touchscreen.c @@ -0,0 +1,45 @@ +/* + * Generic DT helper functions for touchscreen devices + * + * Copyright (c) 2014 Sebastian Reichel <sre@kernel.org> + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + * + */ + +#include <linux/of.h> +#include <linux/input.h> +#include <linux/input/touchscreen.h> + +/** + * touchscreen_parse_of_params - parse common touchscreen DT properties + * @dev: device that should be parsed + * + * This function parses common DT properties for touchscreens and setups the + * input device accordingly. The function keeps previously setuped default + * values if no value is specified via DT. + */ +void touchscreen_parse_of_params(struct input_dev *dev) +{ + struct device_node *np = dev->dev.parent->of_node; + struct input_absinfo *absinfo; + + input_alloc_absinfo(dev); + if (!dev->absinfo) + return; + + absinfo = &dev->absinfo[ABS_X]; + of_property_read_u32(np, "touchscreen-size-x", &absinfo->maximum); + of_property_read_u32(np, "touchscreen-fuzz-x", &absinfo->fuzz); + + absinfo = &dev->absinfo[ABS_Y]; + of_property_read_u32(np, "touchscreen-size-y", &absinfo->maximum); + of_property_read_u32(np, "touchscreen-fuzz-y", &absinfo->fuzz); + + absinfo = &dev->absinfo[ABS_PRESSURE]; + of_property_read_u32(np, "touchscreen-max-pressure", &absinfo->maximum); + of_property_read_u32(np, "touchscreen-fuzz-pressure", &absinfo->fuzz); +} +EXPORT_SYMBOL(touchscreen_parse_of_params); diff --git a/drivers/input/touchscreen/pixcir_i2c_ts.c b/drivers/input/touchscreen/pixcir_i2c_ts.c index 02392d2061d6..19c6c0fdc94b 100644 --- a/drivers/input/touchscreen/pixcir_i2c_ts.c +++ b/drivers/input/touchscreen/pixcir_i2c_ts.c @@ -24,12 +24,13 @@ #include <linux/i2c.h> #include <linux/input.h> #include <linux/input/pixcir_ts.h> +#include <linux/gpio.h> struct pixcir_i2c_ts_data { struct i2c_client *client; struct input_dev *input; const struct pixcir_ts_platform_data *chip; - bool exiting; + bool running; }; static void pixcir_ts_poscheck(struct pixcir_i2c_ts_data *data) @@ -87,11 +88,12 @@ static void pixcir_ts_poscheck(struct pixcir_i2c_ts_data *data) static irqreturn_t pixcir_ts_isr(int irq, void *dev_id) { struct pixcir_i2c_ts_data *tsdata = dev_id; + const struct pixcir_ts_platform_data *pdata = tsdata->chip; - while (!tsdata->exiting) { + while (tsdata->running) { pixcir_ts_poscheck(tsdata); - if (tsdata->chip->attb_read_val()) + if (gpio_get_value(pdata->gpio_attb)) break; msleep(20); @@ -100,25 +102,221 @@ static irqreturn_t pixcir_ts_isr(int irq, void *dev_id) return IRQ_HANDLED; } +static int pixcir_set_power_mode(struct pixcir_i2c_ts_data *ts, + enum pixcir_power_mode mode) +{ + struct device *dev = &ts->client->dev; + int ret; + + ret = i2c_smbus_read_byte_data(ts->client, PIXCIR_REG_POWER_MODE); + if (ret < 0) { + dev_err(dev, "%s: can't read reg 0x%x : %d\n", + __func__, PIXCIR_REG_POWER_MODE, ret); + return ret; + } + + ret &= ~PIXCIR_POWER_MODE_MASK; + ret |= mode; + + /* Always AUTO_IDLE */ + ret |= PIXCIR_POWER_ALLOW_IDLE; + + ret = i2c_smbus_write_byte_data(ts->client, PIXCIR_REG_POWER_MODE, ret); + if (ret < 0) { + dev_err(dev, "%s: can't write reg 0x%x : %d\n", + __func__, PIXCIR_REG_POWER_MODE, ret); + return ret; + } + + return 0; +} + +/* + * Set the interrupt mode for the device i.e. ATTB line behaviour + * + * @polarity : 1 for active high, 0 for active low. + */ +static int pixcir_set_int_mode(struct pixcir_i2c_ts_data *ts, + enum pixcir_int_mode mode, bool polarity) +{ + struct device *dev = &ts->client->dev; + int ret; + + ret = i2c_smbus_read_byte_data(ts->client, PIXCIR_REG_INT_MODE); + if (ret < 0) { + dev_err(dev, "%s: can't read reg 0x%x : %d\n", + __func__, PIXCIR_REG_INT_MODE, ret); + return ret; + } + + ret &= ~PIXCIR_INT_MODE_MASK; + ret |= mode; + + if (polarity) + ret |= PIXCIR_INT_POL_HIGH; + else + ret &= ~PIXCIR_INT_POL_HIGH; + + ret = i2c_smbus_write_byte_data(ts->client, PIXCIR_REG_INT_MODE, ret); + if (ret < 0) { + dev_err(dev, "%s: can't write reg 0x%x : %d\n", + __func__, PIXCIR_REG_INT_MODE, ret); + return ret; + } + + return 0; +} + +/* + * Enable/disable interrupt generation + */ +static int pixcir_int_enable(struct pixcir_i2c_ts_data *ts, bool enable) +{ + struct device *dev = &ts->client->dev; + int ret; + + ret = i2c_smbus_read_byte_data(ts->client, PIXCIR_REG_INT_MODE); + if (ret < 0) { + dev_err(dev, "%s: can't read reg 0x%x : %d\n", + __func__, PIXCIR_REG_INT_MODE, ret); + return ret; + } + + if (enable) + ret |= PIXCIR_INT_ENABLE; + else + ret &= ~PIXCIR_INT_ENABLE; + + ret = i2c_smbus_write_byte_data(ts->client, PIXCIR_REG_INT_MODE, ret); + if (ret < 0) { + dev_err(dev, "%s: can't write reg 0x%x : %d\n", + __func__, PIXCIR_REG_INT_MODE, ret); + return ret; + } + + return 0; +} + +static int pixcir_start(struct pixcir_i2c_ts_data *ts) +{ + struct device *dev = &ts->client->dev; + int error; + + /* LEVEL_TOUCH interrupt with active low polarity */ + error = pixcir_set_int_mode(ts, PIXCIR_INT_LEVEL_TOUCH, 0); + if (error) { + dev_err(dev, "Failed to set interrupt mode: %d\n", error); + return error; + } + + ts->running = true; + mb(); /* Update status before IRQ can fire */ + + /* enable interrupt generation */ + error = pixcir_int_enable(ts, true); + if (error) { + dev_err(dev, "Failed to enable interrupt generation: %d\n", + error); + return error; + } + + return 0; +} + +static int pixcir_stop(struct pixcir_i2c_ts_data *ts) +{ + int error; + + /* Disable interrupt generation */ + error = pixcir_int_enable(ts, false); + if (error) { + dev_err(&ts->client->dev, + "Failed to disable interrupt generation: %d\n", + error); + return error; + } + + /* Exit ISR if running, no more report parsing */ + ts->running = false; + mb(); /* update status before we synchronize irq */ + + /* Wait till running ISR is complete */ + synchronize_irq(ts->client->irq); + + return 0; +} + +static int pixcir_input_open(struct input_dev *dev) +{ + struct pixcir_i2c_ts_data *ts = input_get_drvdata(dev); + + return pixcir_start(ts); +} + +static void pixcir_input_close(struct input_dev *dev) +{ + struct pixcir_i2c_ts_data *ts = input_get_drvdata(dev); + + pixcir_stop(ts); +} + #ifdef CONFIG_PM_SLEEP static int pixcir_i2c_ts_suspend(struct device *dev) { struct i2c_client *client = to_i2c_client(dev); + struct pixcir_i2c_ts_data *ts = i2c_get_clientdata(client); + struct input_dev *input = ts->input; + int ret = 0; + + mutex_lock(&input->mutex); + + if (device_may_wakeup(&client->dev)) { + if (!input->users) { + ret = pixcir_start(ts); + if (ret) { + dev_err(dev, "Failed to start\n"); + goto unlock; + } + } - if (device_may_wakeup(&client->dev)) enable_irq_wake(client->irq); + } else if (input->users) { + ret = pixcir_stop(ts); + } - return 0; +unlock: + mutex_unlock(&input->mutex); + + return ret; } static int pixcir_i2c_ts_resume(struct device *dev) { struct i2c_client *client = to_i2c_client(dev); + struct pixcir_i2c_ts_data *ts = i2c_get_clientdata(client); + struct input_dev *input = ts->input; + int ret = 0; + + mutex_lock(&input->mutex); - if (device_may_wakeup(&client->dev)) + if (device_may_wakeup(&client->dev)) { disable_irq_wake(client->irq); - return 0; + if (!input->users) { + ret = pixcir_stop(ts); + if (ret) { + dev_err(dev, "Failed to stop\n"); + goto unlock; + } + } + } else if (input->users) { + ret = pixcir_start(ts); + } + +unlock: + mutex_unlock(&input->mutex); + + return ret; } #endif @@ -130,6 +328,7 @@ static int pixcir_i2c_ts_probe(struct i2c_client *client, { const struct pixcir_ts_platform_data *pdata = dev_get_platdata(&client->dev); + struct device *dev = &client->dev; struct pixcir_i2c_ts_data *tsdata; struct input_dev *input; int error; @@ -139,12 +338,19 @@ static int pixcir_i2c_ts_probe(struct i2c_client *client, return -EINVAL; } - tsdata = kzalloc(sizeof(*tsdata), GFP_KERNEL); - input = input_allocate_device(); - if (!tsdata || !input) { - dev_err(&client->dev, "Failed to allocate driver data!\n"); - error = -ENOMEM; - goto err_free_mem; + if (!gpio_is_valid(pdata->gpio_attb)) { + dev_err(dev, "Invalid gpio_attb in pdata\n"); + return -EINVAL; + } + + tsdata = devm_kzalloc(dev, sizeof(*tsdata), GFP_KERNEL); + if (!tsdata) + return -ENOMEM; + + input = devm_input_allocate_device(dev); + if (!input) { + dev_err(dev, "Failed to allocate input device\n"); + return -ENOMEM; } tsdata->client = client; @@ -153,6 +359,8 @@ static int pixcir_i2c_ts_probe(struct i2c_client *client, input->name = client->name; input->id.bustype = BUS_I2C; + input->open = pixcir_input_open; + input->close = pixcir_input_close; input->dev.parent = &client->dev; __set_bit(EV_KEY, input->evbit); @@ -165,44 +373,47 @@ static int pixcir_i2c_ts_probe(struct i2c_client *client, input_set_drvdata(input, tsdata); - error = request_threaded_irq(client->irq, NULL, pixcir_ts_isr, - IRQF_TRIGGER_FALLING | IRQF_ONESHOT, - client->name, tsdata); + error = devm_gpio_request_one(dev, pdata->gpio_attb, + GPIOF_DIR_IN, "pixcir_i2c_attb"); if (error) { - dev_err(&client->dev, "Unable to request touchscreen IRQ.\n"); - goto err_free_mem; + dev_err(dev, "Failed to request ATTB gpio\n"); + return error; } + error = devm_request_threaded_irq(dev, client->irq, NULL, pixcir_ts_isr, + IRQF_TRIGGER_FALLING | IRQF_ONESHOT, + client->name, tsdata); + if (error) { + dev_err(dev, "failed to request irq %d\n", client->irq); + return error; + } + + /* Always be in IDLE mode to save power, device supports auto wake */ + error = pixcir_set_power_mode(tsdata, PIXCIR_POWER_IDLE); + if (error) { + dev_err(dev, "Failed to set IDLE mode\n"); + return error; + } + + /* Stop device till opened */ + error = pixcir_stop(tsdata); + if (error) + return error; + error = input_register_device(input); if (error) - goto err_free_irq; + return error; i2c_set_clientdata(client, tsdata); device_init_wakeup(&client->dev, 1); return 0; - -err_free_irq: - free_irq(client->irq, tsdata); -err_free_mem: - input_free_device(input); - kfree(tsdata); - return error; } static int pixcir_i2c_ts_remove(struct i2c_client *client) { - struct pixcir_i2c_ts_data *tsdata = i2c_get_clientdata(client); - device_init_wakeup(&client->dev, 0); - tsdata->exiting = true; - mb(); - free_irq(client->irq, tsdata); - - input_unregister_device(tsdata->input); - kfree(tsdata); - return 0; } diff --git a/drivers/input/touchscreen/sun4i-ts.c b/drivers/input/touchscreen/sun4i-ts.c new file mode 100644 index 000000000000..2ba826024954 --- /dev/null +++ b/drivers/input/touchscreen/sun4i-ts.c @@ -0,0 +1,339 @@ +/* + * Allwinner sunxi resistive touchscreen controller driver + * + * Copyright (C) 2013 - 2014 Hans de Goede <hdegoede@redhat.com> + * + * The hwmon parts are based on work by Corentin LABBE which is: + * Copyright (C) 2013 Corentin LABBE <clabbe.montjoie@gmail.com> + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + */ + +/* + * The sun4i-ts controller is capable of detecting a second touch, but when a + * second touch is present then the accuracy becomes so bad the reported touch + * location is not useable. + * + * The original android driver contains some complicated heuristics using the + * aprox. distance between the 2 touches to see if the user is making a pinch + * open / close movement, and then reports emulated multi-touch events around + * the last touch coordinate (as the dual-touch coordinates are worthless). + * + * These kinds of heuristics are just asking for trouble (and don't belong + * in the kernel). So this driver offers straight forward, reliable single + * touch functionality only. + */ + +#include <linux/err.h> +#include <linux/hwmon.h> +#include <linux/init.h> +#include <linux/input.h> +#include <linux/interrupt.h> +#include <linux/io.h> +#include <linux/module.h> +#include <linux/of_platform.h> +#include <linux/platform_device.h> +#include <linux/slab.h> + +#define TP_CTRL0 0x00 +#define TP_CTRL1 0x04 +#define TP_CTRL2 0x08 +#define TP_CTRL3 0x0c +#define TP_INT_FIFOC 0x10 +#define TP_INT_FIFOS 0x14 +#define TP_TPR 0x18 +#define TP_CDAT 0x1c +#define TEMP_DATA 0x20 +#define TP_DATA 0x24 + +/* TP_CTRL0 bits */ +#define ADC_FIRST_DLY(x) ((x) << 24) /* 8 bits */ +#define ADC_FIRST_DLY_MODE(x) ((x) << 23) +#define ADC_CLK_SEL(x) ((x) << 22) +#define ADC_CLK_DIV(x) ((x) << 20) /* 3 bits */ +#define FS_DIV(x) ((x) << 16) /* 4 bits */ +#define T_ACQ(x) ((x) << 0) /* 16 bits */ + +/* TP_CTRL1 bits */ +#define STYLUS_UP_DEBOUN(x) ((x) << 12) /* 8 bits */ +#define STYLUS_UP_DEBOUN_EN(x) ((x) << 9) +#define TOUCH_PAN_CALI_EN(x) ((x) << 6) +#define TP_DUAL_EN(x) ((x) << 5) +#define TP_MODE_EN(x) ((x) << 4) +#define TP_ADC_SELECT(x) ((x) << 3) +#define ADC_CHAN_SELECT(x) ((x) << 0) /* 3 bits */ + +/* TP_CTRL2 bits */ +#define TP_SENSITIVE_ADJUST(x) ((x) << 28) /* 4 bits */ +#define TP_MODE_SELECT(x) ((x) << 26) /* 2 bits */ +#define PRE_MEA_EN(x) ((x) << 24) +#define PRE_MEA_THRE_CNT(x) ((x) << 0) /* 24 bits */ + +/* TP_CTRL3 bits */ +#define FILTER_EN(x) ((x) << 2) +#define FILTER_TYPE(x) ((x) << 0) /* 2 bits */ + +/* TP_INT_FIFOC irq and fifo mask / control bits */ +#define TEMP_IRQ_EN(x) ((x) << 18) +#define OVERRUN_IRQ_EN(x) ((x) << 17) +#define DATA_IRQ_EN(x) ((x) << 16) +#define TP_DATA_XY_CHANGE(x) ((x) << 13) +#define FIFO_TRIG(x) ((x) << 8) /* 5 bits */ +#define DATA_DRQ_EN(x) ((x) << 7) +#define FIFO_FLUSH(x) ((x) << 4) +#define TP_UP_IRQ_EN(x) ((x) << 1) +#define TP_DOWN_IRQ_EN(x) ((x) << 0) + +/* TP_INT_FIFOS irq and fifo status bits */ +#define TEMP_DATA_PENDING BIT(18) +#define FIFO_OVERRUN_PENDING BIT(17) +#define FIFO_DATA_PENDING BIT(16) +#define TP_IDLE_FLG BIT(2) +#define TP_UP_PENDING BIT(1) +#define TP_DOWN_PENDING BIT(0) + +/* TP_TPR bits */ +#define TEMP_ENABLE(x) ((x) << 16) +#define TEMP_PERIOD(x) ((x) << 0) /* t = x * 256 * 16 / clkin */ + +struct sun4i_ts_data { + struct device *dev; + struct input_dev *input; + void __iomem *base; + unsigned int irq; + bool ignore_fifo_data; + int temp_data; +}; + +static void sun4i_ts_irq_handle_input(struct sun4i_ts_data *ts, u32 reg_val) +{ + u32 x, y; + + if (reg_val & FIFO_DATA_PENDING) { + x = readl(ts->base + TP_DATA); + y = readl(ts->base + TP_DATA); + /* The 1st location reported after an up event is unreliable */ + if (!ts->ignore_fifo_data) { + input_report_abs(ts->input, ABS_X, x); + input_report_abs(ts->input, ABS_Y, y); + /* + * The hardware has a separate down status bit, but + * that gets set before we get the first location, + * resulting in reporting a click on the old location. + */ + input_report_key(ts->input, BTN_TOUCH, 1); + input_sync(ts->input); + } else { + ts->ignore_fifo_data = false; + } + } + + if (reg_val & TP_UP_PENDING) { + ts->ignore_fifo_data = true; + input_report_key(ts->input, BTN_TOUCH, 0); + input_sync(ts->input); + } +} + +static irqreturn_t sun4i_ts_irq(int irq, void *dev_id) +{ + struct sun4i_ts_data *ts = dev_id; + u32 reg_val; + + reg_val = readl(ts->base + TP_INT_FIFOS); + + if (reg_val & TEMP_DATA_PENDING) + ts->temp_data = readl(ts->base + TEMP_DATA); + + if (ts->input) + sun4i_ts_irq_handle_input(ts, reg_val); + + writel(reg_val, ts->base + TP_INT_FIFOS); + + return IRQ_HANDLED; +} + +static int sun4i_ts_open(struct input_dev *dev) +{ + struct sun4i_ts_data *ts = input_get_drvdata(dev); + + /* Flush, set trig level to 1, enable temp, data and up irqs */ + writel(TEMP_IRQ_EN(1) | DATA_IRQ_EN(1) | FIFO_TRIG(1) | FIFO_FLUSH(1) | + TP_UP_IRQ_EN(1), ts->base + TP_INT_FIFOC); + + return 0; +} + +static void sun4i_ts_close(struct input_dev *dev) +{ + struct sun4i_ts_data *ts = input_get_drvdata(dev); + + /* Deactivate all input IRQs */ + writel(TEMP_IRQ_EN(1), ts->base + TP_INT_FIFOC); +} + +static ssize_t show_temp(struct device *dev, struct device_attribute *devattr, + char *buf) +{ + struct sun4i_ts_data *ts = dev_get_drvdata(dev); + + /* No temp_data until the first irq */ + if (ts->temp_data == -1) + return -EAGAIN; + + return sprintf(buf, "%d\n", (ts->temp_data - 1447) * 100); +} + +static ssize_t show_temp_label(struct device *dev, + struct device_attribute *devattr, char *buf) +{ + return sprintf(buf, "SoC temperature\n"); +} + +static DEVICE_ATTR(temp1_input, S_IRUGO, show_temp, NULL); +static DEVICE_ATTR(temp1_label, S_IRUGO, show_temp_label, NULL); + +static struct attribute *sun4i_ts_attrs[] = { + &dev_attr_temp1_input.attr, + &dev_attr_temp1_label.attr, + NULL +}; +ATTRIBUTE_GROUPS(sun4i_ts); + +static int sun4i_ts_probe(struct platform_device *pdev) +{ + struct sun4i_ts_data *ts; + struct device *dev = &pdev->dev; + struct device_node *np = dev->of_node; + struct device *hwmon; + int error; + bool ts_attached; + + ts = devm_kzalloc(dev, sizeof(struct sun4i_ts_data), GFP_KERNEL); + if (!ts) + return -ENOMEM; + + ts->dev = dev; + ts->ignore_fifo_data = true; + ts->temp_data = -1; + + ts_attached = of_property_read_bool(np, "allwinner,ts-attached"); + if (ts_attached) { + ts->input = devm_input_allocate_device(dev); + if (!ts->input) + return -ENOMEM; + + ts->input->name = pdev->name; + ts->input->phys = "sun4i_ts/input0"; + ts->input->open = sun4i_ts_open; + ts->input->close = sun4i_ts_close; + ts->input->id.bustype = BUS_HOST; + ts->input->id.vendor = 0x0001; + ts->input->id.product = 0x0001; + ts->input->id.version = 0x0100; + ts->input->evbit[0] = BIT(EV_SYN) | BIT(EV_KEY) | BIT(EV_ABS); + __set_bit(BTN_TOUCH, ts->input->keybit); + input_set_abs_params(ts->input, ABS_X, 0, 4095, 0, 0); + input_set_abs_params(ts->input, ABS_Y, 0, 4095, 0, 0); + input_set_drvdata(ts->input, ts); + } + + ts->base = devm_ioremap_resource(dev, + platform_get_resource(pdev, IORESOURCE_MEM, 0)); + if (IS_ERR(ts->base)) + return PTR_ERR(ts->base); + + ts->irq = platform_get_irq(pdev, 0); + error = devm_request_irq(dev, ts->irq, sun4i_ts_irq, 0, "sun4i-ts", ts); + if (error) + return error; + + /* + * Select HOSC clk, clkin = clk / 6, adc samplefreq = clkin / 8192, + * t_acq = clkin / (16 * 64) + */ + writel(ADC_CLK_SEL(0) | ADC_CLK_DIV(2) | FS_DIV(7) | T_ACQ(63), + ts->base + TP_CTRL0); + + /* + * sensitive_adjust = 15 : max, which is not all that sensitive, + * tp_mode = 0 : only x and y coordinates, as we don't use dual touch + */ + writel(TP_SENSITIVE_ADJUST(15) | TP_MODE_SELECT(0), + ts->base + TP_CTRL2); + + /* Enable median filter, type 1 : 5/3 */ + writel(FILTER_EN(1) | FILTER_TYPE(1), ts->base + TP_CTRL3); + + /* Enable temperature measurement, period 1953 (2 seconds) */ + writel(TEMP_ENABLE(1) | TEMP_PERIOD(1953), ts->base + TP_TPR); + + /* + * Set stylus up debounce to aprox 10 ms, enable debounce, and + * finally enable tp mode. + */ + writel(STYLUS_UP_DEBOUN(5) | STYLUS_UP_DEBOUN_EN(1) | TP_MODE_EN(1), + ts->base + TP_CTRL1); + + hwmon = devm_hwmon_device_register_with_groups(ts->dev, "sun4i_ts", + ts, sun4i_ts_groups); + if (IS_ERR(hwmon)) + return PTR_ERR(hwmon); + + writel(TEMP_IRQ_EN(1), ts->base + TP_INT_FIFOC); + + if (ts_attached) { + error = input_register_device(ts->input); + if (error) { + writel(0, ts->base + TP_INT_FIFOC); + return error; + } + } + + platform_set_drvdata(pdev, ts); + return 0; +} + +static int sun4i_ts_remove(struct platform_device *pdev) +{ + struct sun4i_ts_data *ts = platform_get_drvdata(pdev); + + /* Explicit unregister to avoid open/close changing the imask later */ + if (ts->input) + input_unregister_device(ts->input); + + /* Deactivate all IRQs */ + writel(0, ts->base + TP_INT_FIFOC); + + return 0; +} + +static const struct of_device_id sun4i_ts_of_match[] = { + { .compatible = "allwinner,sun4i-a10-ts", }, + { /* sentinel */ } +}; +MODULE_DEVICE_TABLE(of, sun4i_ts_of_match); + +static struct platform_driver sun4i_ts_driver = { + .driver = { + .owner = THIS_MODULE, + .name = "sun4i-ts", + .of_match_table = of_match_ptr(sun4i_ts_of_match), + }, + .probe = sun4i_ts_probe, + .remove = sun4i_ts_remove, +}; + +module_platform_driver(sun4i_ts_driver); + +MODULE_DESCRIPTION("Allwinner sun4i resistive touchscreen controller driver"); +MODULE_AUTHOR("Hans de Goede <hdegoede@redhat.com>"); +MODULE_LICENSE("GPL"); diff --git a/drivers/input/touchscreen/tsc2005.c b/drivers/input/touchscreen/tsc2005.c index 550adcbbfc23..52380b68ebdf 100644 --- a/drivers/input/touchscreen/tsc2005.c +++ b/drivers/input/touchscreen/tsc2005.c @@ -25,11 +25,15 @@ #include <linux/kernel.h> #include <linux/module.h> #include <linux/input.h> +#include <linux/input/touchscreen.h> #include <linux/interrupt.h> #include <linux/delay.h> #include <linux/pm.h> +#include <linux/of.h> +#include <linux/of_gpio.h> #include <linux/spi/spi.h> #include <linux/spi/tsc2005.h> +#include <linux/regulator/consumer.h> /* * The touchscreen interface operates as follows: @@ -100,6 +104,11 @@ TSC2005_CFR2_AVG_7) #define MAX_12BIT 0xfff +#define TSC2005_DEF_X_FUZZ 4 +#define TSC2005_DEF_Y_FUZZ 8 +#define TSC2005_DEF_P_FUZZ 2 +#define TSC2005_DEF_RESISTOR 280 + #define TSC2005_SPI_MAX_SPEED_HZ 10000000 #define TSC2005_PENUP_TIME_MS 40 @@ -143,6 +152,9 @@ struct tsc2005 { bool pen_down; + struct regulator *vio; + + int reset_gpio; void (*set_reset)(bool enable); }; @@ -337,6 +349,14 @@ static void tsc2005_stop_scan(struct tsc2005 *ts) tsc2005_cmd(ts, TSC2005_CMD_STOP); } +static void tsc2005_set_reset(struct tsc2005 *ts, bool enable) +{ + if (ts->reset_gpio >= 0) + gpio_set_value(ts->reset_gpio, enable); + else if (ts->set_reset) + ts->set_reset(enable); +} + /* must be called with ts->mutex held */ static void __tsc2005_disable(struct tsc2005 *ts) { @@ -355,7 +375,7 @@ static void __tsc2005_enable(struct tsc2005 *ts) { tsc2005_start_scan(ts); - if (ts->esd_timeout && ts->set_reset) { + if (ts->esd_timeout && (ts->set_reset || ts->reset_gpio)) { ts->last_valid_interrupt = jiffies; schedule_delayed_work(&ts->esd_work, round_jiffies_relative( @@ -414,9 +434,9 @@ static ssize_t tsc2005_selftest_show(struct device *dev, } /* hardware reset */ - ts->set_reset(false); + tsc2005_set_reset(ts, false); usleep_range(100, 500); /* only 10us required */ - ts->set_reset(true); + tsc2005_set_reset(ts, true); if (!success) goto out; @@ -459,7 +479,7 @@ static umode_t tsc2005_attr_is_visible(struct kobject *kobj, umode_t mode = attr->mode; if (attr == &dev_attr_selftest.attr) { - if (!ts->set_reset) + if (!ts->set_reset && !ts->reset_gpio) mode = 0; } @@ -509,9 +529,9 @@ static void tsc2005_esd_work(struct work_struct *work) tsc2005_update_pen_state(ts, 0, 0, 0); - ts->set_reset(false); + tsc2005_set_reset(ts, false); usleep_range(100, 500); /* only 10us required */ - ts->set_reset(true); + tsc2005_set_reset(ts, true); enable_irq(ts->spi->irq); tsc2005_start_scan(ts); @@ -572,29 +592,47 @@ static void tsc2005_setup_spi_xfer(struct tsc2005 *ts) static int tsc2005_probe(struct spi_device *spi) { const struct tsc2005_platform_data *pdata = dev_get_platdata(&spi->dev); + struct device_node *np = spi->dev.of_node; + struct tsc2005 *ts; struct input_dev *input_dev; - unsigned int max_x, max_y, max_p; - unsigned int fudge_x, fudge_y, fudge_p; + unsigned int max_x = MAX_12BIT; + unsigned int max_y = MAX_12BIT; + unsigned int max_p = MAX_12BIT; + unsigned int fudge_x = TSC2005_DEF_X_FUZZ; + unsigned int fudge_y = TSC2005_DEF_Y_FUZZ; + unsigned int fudge_p = TSC2005_DEF_P_FUZZ; + unsigned int x_plate_ohm = TSC2005_DEF_RESISTOR; + unsigned int esd_timeout; int error; - if (!pdata) { - dev_dbg(&spi->dev, "no platform data\n"); + if (!np && !pdata) { + dev_err(&spi->dev, "no platform data\n"); return -ENODEV; } - fudge_x = pdata->ts_x_fudge ? : 4; - fudge_y = pdata->ts_y_fudge ? : 8; - fudge_p = pdata->ts_pressure_fudge ? : 2; - max_x = pdata->ts_x_max ? : MAX_12BIT; - max_y = pdata->ts_y_max ? : MAX_12BIT; - max_p = pdata->ts_pressure_max ? : MAX_12BIT; - if (spi->irq <= 0) { - dev_dbg(&spi->dev, "no irq\n"); + dev_err(&spi->dev, "no irq\n"); return -ENODEV; } + if (pdata) { + fudge_x = pdata->ts_x_fudge; + fudge_y = pdata->ts_y_fudge; + fudge_p = pdata->ts_pressure_fudge; + max_x = pdata->ts_x_max; + max_y = pdata->ts_y_max; + max_p = pdata->ts_pressure_max; + x_plate_ohm = pdata->ts_x_plate_ohm; + esd_timeout = pdata->esd_timeout_ms; + } else { + x_plate_ohm = TSC2005_DEF_RESISTOR; + of_property_read_u32(np, "ti,x-plate-ohms", &x_plate_ohm); + esd_timeout = 0; + of_property_read_u32(np, "ti,esd-recovery-timeout-ms", + &esd_timeout); + } + spi->mode = SPI_MODE_0; spi->bits_per_word = 8; if (!spi->max_speed_hz) @@ -604,19 +642,48 @@ static int tsc2005_probe(struct spi_device *spi) if (error) return error; - ts = kzalloc(sizeof(*ts), GFP_KERNEL); - input_dev = input_allocate_device(); - if (!ts || !input_dev) { - error = -ENOMEM; - goto err_free_mem; - } + ts = devm_kzalloc(&spi->dev, sizeof(*ts), GFP_KERNEL); + if (!ts) + return -ENOMEM; + + input_dev = devm_input_allocate_device(&spi->dev); + if (!input_dev) + return -ENOMEM; ts->spi = spi; ts->idev = input_dev; - ts->x_plate_ohm = pdata->ts_x_plate_ohm ? : 280; - ts->esd_timeout = pdata->esd_timeout_ms; - ts->set_reset = pdata->set_reset; + ts->x_plate_ohm = x_plate_ohm; + ts->esd_timeout = esd_timeout; + + if (np) { + ts->reset_gpio = of_get_named_gpio(np, "reset-gpios", 0); + if (ts->reset_gpio == -EPROBE_DEFER) + return ts->reset_gpio; + if (ts->reset_gpio < 0) { + dev_err(&spi->dev, "error acquiring reset gpio: %d\n", + ts->reset_gpio); + return ts->reset_gpio; + } + + error = devm_gpio_request_one(&spi->dev, ts->reset_gpio, 0, + "reset-gpios"); + if (error) { + dev_err(&spi->dev, "error requesting reset gpio: %d\n", + error); + return error; + } + + ts->vio = devm_regulator_get(&spi->dev, "vio"); + if (IS_ERR(ts->vio)) { + error = PTR_ERR(ts->vio); + dev_err(&spi->dev, "vio regulator missing (%d)", error); + return error; + } + } else { + ts->reset_gpio = -1; + ts->set_reset = pdata->set_reset; + } mutex_init(&ts->mutex); @@ -641,6 +708,9 @@ static int tsc2005_probe(struct spi_device *spi) input_set_abs_params(input_dev, ABS_Y, 0, max_y, fudge_y, 0); input_set_abs_params(input_dev, ABS_PRESSURE, 0, max_p, fudge_p, 0); + if (np) + touchscreen_parse_of_params(input_dev); + input_dev->open = tsc2005_open; input_dev->close = tsc2005_close; @@ -649,12 +719,20 @@ static int tsc2005_probe(struct spi_device *spi) /* Ensure the touchscreen is off */ tsc2005_stop_scan(ts); - error = request_threaded_irq(spi->irq, NULL, tsc2005_irq_thread, - IRQF_TRIGGER_RISING | IRQF_ONESHOT, - "tsc2005", ts); + error = devm_request_threaded_irq(&spi->dev, spi->irq, NULL, + tsc2005_irq_thread, + IRQF_TRIGGER_RISING | IRQF_ONESHOT, + "tsc2005", ts); if (error) { dev_err(&spi->dev, "Failed to request irq, err: %d\n", error); - goto err_free_mem; + return error; + } + + /* enable regulator for DT */ + if (ts->vio) { + error = regulator_enable(ts->vio); + if (error) + return error; } spi_set_drvdata(spi, ts); @@ -662,7 +740,7 @@ static int tsc2005_probe(struct spi_device *spi) if (error) { dev_err(&spi->dev, "Failed to create sysfs attributes, err: %d\n", error); - goto err_clear_drvdata; + goto disable_regulator; } error = input_register_device(ts->idev); @@ -677,11 +755,9 @@ static int tsc2005_probe(struct spi_device *spi) err_remove_sysfs: sysfs_remove_group(&spi->dev.kobj, &tsc2005_attr_group); -err_clear_drvdata: - free_irq(spi->irq, ts); -err_free_mem: - input_free_device(input_dev); - kfree(ts); +disable_regulator: + if (ts->vio) + regulator_disable(ts->vio); return error; } @@ -689,11 +765,10 @@ static int tsc2005_remove(struct spi_device *spi) { struct tsc2005 *ts = spi_get_drvdata(spi); - sysfs_remove_group(&ts->spi->dev.kobj, &tsc2005_attr_group); + sysfs_remove_group(&spi->dev.kobj, &tsc2005_attr_group); - free_irq(ts->spi->irq, ts); - input_unregister_device(ts->idev); - kfree(ts); + if (ts->vio) + regulator_disable(ts->vio); return 0; } diff --git a/drivers/input/touchscreen/zforce_ts.c b/drivers/input/touchscreen/zforce_ts.c index 01d30cedde46..feea85b52fa8 100644 --- a/drivers/input/touchscreen/zforce_ts.c +++ b/drivers/input/touchscreen/zforce_ts.c @@ -880,7 +880,7 @@ static struct i2c_device_id zforce_idtable[] = { MODULE_DEVICE_TABLE(i2c, zforce_idtable); #ifdef CONFIG_OF -static struct of_device_id zforce_dt_idtable[] = { +static const struct of_device_id zforce_dt_idtable[] = { { .compatible = "neonode,zforce" }, {}, }; diff --git a/drivers/macintosh/windfarm_pm121.c b/drivers/macintosh/windfarm_pm121.c index 7fe58b0ae8b4..b350fb86ff08 100644 --- a/drivers/macintosh/windfarm_pm121.c +++ b/drivers/macintosh/windfarm_pm121.c @@ -555,8 +555,18 @@ static void pm121_create_sys_fans(int loop_id) pid_param.interval = PM121_SYS_INTERVAL; pid_param.history_len = PM121_SYS_HISTORY_SIZE; pid_param.itarget = param->itarget; - pid_param.min = control->ops->get_min(control); - pid_param.max = control->ops->get_max(control); + if(control) + { + pid_param.min = control->ops->get_min(control); + pid_param.max = control->ops->get_max(control); + } else { + /* + * This is probably not the right!? + * Perhaps goto fail if control == NULL above? + */ + pid_param.min = 0; + pid_param.max = 0; + } wf_pid_init(&pm121_sys_state[loop_id]->pid, &pid_param); @@ -571,7 +581,7 @@ static void pm121_create_sys_fans(int loop_id) control the same control */ printk(KERN_WARNING "pm121: failed to set up %s loop " "setting \"%s\" to max speed.\n", - loop_names[loop_id], control->name); + loop_names[loop_id], control ? control->name : "uninitialized value"); if (control) wf_control_set_max(control); diff --git a/drivers/message/fusion/mptbase.c b/drivers/message/fusion/mptbase.c index 570b18a113ff..ebc0af7d769c 100644 --- a/drivers/message/fusion/mptbase.c +++ b/drivers/message/fusion/mptbase.c @@ -1037,7 +1037,7 @@ mpt_free_msg_frame(MPT_ADAPTER *ioc, MPT_FRAME_HDR *mf) goto out; /* signature to know if this mf is freed */ mf->u.frame.linkage.arg1 = cpu_to_le32(0xdeadbeaf); - list_add_tail(&mf->u.frame.linkage.list, &ioc->FreeQ); + list_add(&mf->u.frame.linkage.list, &ioc->FreeQ); #ifdef MFCNT ioc->mfcnt--; #endif diff --git a/drivers/message/fusion/mptctl.c b/drivers/message/fusion/mptctl.c index dcc8385adeb3..8a050e885688 100644 --- a/drivers/message/fusion/mptctl.c +++ b/drivers/message/fusion/mptctl.c @@ -2432,9 +2432,9 @@ mptctl_hp_hostinfo(unsigned long arg, unsigned int data_size) int rc, cim_rev; ToolboxIstwiReadWriteRequest_t *IstwiRWRequest; MPT_FRAME_HDR *mf = NULL; - MPIHeader_t *mpi_hdr; unsigned long timeleft; int retval; + u32 msgcontext; /* Reset long to int. Should affect IA64 and SPARC only */ @@ -2581,11 +2581,11 @@ mptctl_hp_hostinfo(unsigned long arg, unsigned int data_size) } IstwiRWRequest = (ToolboxIstwiReadWriteRequest_t *)mf; - mpi_hdr = (MPIHeader_t *) mf; + msgcontext = IstwiRWRequest->MsgContext; memset(IstwiRWRequest,0,sizeof(ToolboxIstwiReadWriteRequest_t)); + IstwiRWRequest->MsgContext = msgcontext; IstwiRWRequest->Function = MPI_FUNCTION_TOOLBOX; IstwiRWRequest->Tool = MPI_TOOLBOX_ISTWI_READ_WRITE_TOOL; - IstwiRWRequest->MsgContext = mpi_hdr->MsgContext; IstwiRWRequest->Flags = MPI_TB_ISTWI_FLAGS_READ; IstwiRWRequest->NumAddressBytes = 0x01; IstwiRWRequest->DataLength = cpu_to_le16(0x04); diff --git a/drivers/message/fusion/mptfc.c b/drivers/message/fusion/mptfc.c index fd75108c355e..02a3eefd6931 100644 --- a/drivers/message/fusion/mptfc.c +++ b/drivers/message/fusion/mptfc.c @@ -649,7 +649,7 @@ mptfc_slave_alloc(struct scsi_device *sdev) } static int -mptfc_qcmd_lck(struct scsi_cmnd *SCpnt, void (*done)(struct scsi_cmnd *)) +mptfc_qcmd(struct Scsi_Host *shost, struct scsi_cmnd *SCpnt) { struct mptfc_rport_info *ri; struct fc_rport *rport = starget_to_rport(scsi_target(SCpnt->device)); @@ -658,14 +658,14 @@ mptfc_qcmd_lck(struct scsi_cmnd *SCpnt, void (*done)(struct scsi_cmnd *)) if (!vdevice || !vdevice->vtarget) { SCpnt->result = DID_NO_CONNECT << 16; - done(SCpnt); + SCpnt->scsi_done(SCpnt); return 0; } err = fc_remote_port_chkready(rport); if (unlikely(err)) { SCpnt->result = err; - done(SCpnt); + SCpnt->scsi_done(SCpnt); return 0; } @@ -673,15 +673,13 @@ mptfc_qcmd_lck(struct scsi_cmnd *SCpnt, void (*done)(struct scsi_cmnd *)) ri = *((struct mptfc_rport_info **)rport->dd_data); if (unlikely(!ri)) { SCpnt->result = DID_IMM_RETRY << 16; - done(SCpnt); + SCpnt->scsi_done(SCpnt); return 0; } - return mptscsih_qcmd(SCpnt,done); + return mptscsih_qcmd(SCpnt); } -static DEF_SCSI_QCMD(mptfc_qcmd) - /* * mptfc_display_port_link_speed - displaying link speed * @ioc: Pointer to MPT_ADAPTER structure diff --git a/drivers/message/fusion/mptsas.c b/drivers/message/fusion/mptsas.c index 00d339c361fc..711fcb5cec87 100644 --- a/drivers/message/fusion/mptsas.c +++ b/drivers/message/fusion/mptsas.c @@ -1896,7 +1896,7 @@ mptsas_slave_alloc(struct scsi_device *sdev) } static int -mptsas_qcmd_lck(struct scsi_cmnd *SCpnt, void (*done)(struct scsi_cmnd *)) +mptsas_qcmd(struct Scsi_Host *shost, struct scsi_cmnd *SCpnt) { MPT_SCSI_HOST *hd; MPT_ADAPTER *ioc; @@ -1904,11 +1904,11 @@ mptsas_qcmd_lck(struct scsi_cmnd *SCpnt, void (*done)(struct scsi_cmnd *)) if (!vdevice || !vdevice->vtarget || vdevice->vtarget->deleted) { SCpnt->result = DID_NO_CONNECT << 16; - done(SCpnt); + SCpnt->scsi_done(SCpnt); return 0; } - hd = shost_priv(SCpnt->device->host); + hd = shost_priv(shost); ioc = hd->ioc; if (ioc->sas_discovery_quiesce_io) @@ -1917,11 +1917,9 @@ mptsas_qcmd_lck(struct scsi_cmnd *SCpnt, void (*done)(struct scsi_cmnd *)) if (ioc->debug_level & MPT_DEBUG_SCSI) scsi_print_command(SCpnt); - return mptscsih_qcmd(SCpnt,done); + return mptscsih_qcmd(SCpnt); } -static DEF_SCSI_QCMD(mptsas_qcmd) - /** * mptsas_mptsas_eh_timed_out - resets the scsi_cmnd timeout * if the device under question is currently in the diff --git a/drivers/message/fusion/mptscsih.c b/drivers/message/fusion/mptscsih.c index 727819cc7034..2a1c6f21af27 100644 --- a/drivers/message/fusion/mptscsih.c +++ b/drivers/message/fusion/mptscsih.c @@ -1304,7 +1304,6 @@ int mptscsih_show_info(struct seq_file *m, struct Scsi_Host *host) /** * mptscsih_qcmd - Primary Fusion MPT SCSI initiator IO start routine. * @SCpnt: Pointer to scsi_cmnd structure - * @done: Pointer SCSI mid-layer IO completion function * * (linux scsi_host_template.queuecommand routine) * This is the primary SCSI IO start routine. Create a MPI SCSIIORequest @@ -1313,7 +1312,7 @@ int mptscsih_show_info(struct seq_file *m, struct Scsi_Host *host) * Returns 0. (rtn value discarded by linux scsi mid-layer) */ int -mptscsih_qcmd(struct scsi_cmnd *SCpnt, void (*done)(struct scsi_cmnd *)) +mptscsih_qcmd(struct scsi_cmnd *SCpnt) { MPT_SCSI_HOST *hd; MPT_FRAME_HDR *mf; @@ -1329,10 +1328,9 @@ mptscsih_qcmd(struct scsi_cmnd *SCpnt, void (*done)(struct scsi_cmnd *)) hd = shost_priv(SCpnt->device->host); ioc = hd->ioc; - SCpnt->scsi_done = done; - dmfprintk(ioc, printk(MYIOC_s_DEBUG_FMT "qcmd: SCpnt=%p, done()=%p\n", - ioc->name, SCpnt, done)); + dmfprintk(ioc, printk(MYIOC_s_DEBUG_FMT "qcmd: SCpnt=%p\n", + ioc->name, SCpnt)); if (ioc->taskmgmt_quiesce_io) return SCSI_MLQUEUE_HOST_BUSY; diff --git a/drivers/message/fusion/mptscsih.h b/drivers/message/fusion/mptscsih.h index 83f503162f7a..99e3390807f3 100644 --- a/drivers/message/fusion/mptscsih.h +++ b/drivers/message/fusion/mptscsih.h @@ -113,7 +113,7 @@ extern int mptscsih_resume(struct pci_dev *pdev); #endif extern int mptscsih_show_info(struct seq_file *, struct Scsi_Host *); extern const char * mptscsih_info(struct Scsi_Host *SChost); -extern int mptscsih_qcmd(struct scsi_cmnd *SCpnt, void (*done)(struct scsi_cmnd *)); +extern int mptscsih_qcmd(struct scsi_cmnd *SCpnt); extern int mptscsih_IssueTaskMgmt(MPT_SCSI_HOST *hd, u8 type, u8 channel, u8 id, int lun, int ctx2abort, ulong timeout); extern void mptscsih_slave_destroy(struct scsi_device *device); diff --git a/drivers/message/fusion/mptspi.c b/drivers/message/fusion/mptspi.c index 5653e505f91f..49d11338294b 100644 --- a/drivers/message/fusion/mptspi.c +++ b/drivers/message/fusion/mptspi.c @@ -780,33 +780,31 @@ static int mptspi_slave_configure(struct scsi_device *sdev) } static int -mptspi_qcmd_lck(struct scsi_cmnd *SCpnt, void (*done)(struct scsi_cmnd *)) +mptspi_qcmd(struct Scsi_Host *shost, struct scsi_cmnd *SCpnt) { - struct _MPT_SCSI_HOST *hd = shost_priv(SCpnt->device->host); + struct _MPT_SCSI_HOST *hd = shost_priv(shost); VirtDevice *vdevice = SCpnt->device->hostdata; MPT_ADAPTER *ioc = hd->ioc; if (!vdevice || !vdevice->vtarget) { SCpnt->result = DID_NO_CONNECT << 16; - done(SCpnt); + SCpnt->scsi_done(SCpnt); return 0; } if (SCpnt->device->channel == 1 && mptscsih_is_phys_disk(ioc, 0, SCpnt->device->id) == 0) { SCpnt->result = DID_NO_CONNECT << 16; - done(SCpnt); + SCpnt->scsi_done(SCpnt); return 0; } if (spi_dv_pending(scsi_target(SCpnt->device))) ddvprintk(ioc, scsi_print_command(SCpnt)); - return mptscsih_qcmd(SCpnt,done); + return mptscsih_qcmd(SCpnt); } -static DEF_SCSI_QCMD(mptspi_qcmd) - static void mptspi_slave_destroy(struct scsi_device *sdev) { struct scsi_target *starget = scsi_target(sdev); diff --git a/drivers/mmc/core/bus.c b/drivers/mmc/core/bus.c index 824644875d41..d2dbf02022bd 100644 --- a/drivers/mmc/core/bus.c +++ b/drivers/mmc/core/bus.c @@ -341,16 +341,17 @@ int mmc_add_card(struct mmc_card *card) if (mmc_host_is_spi(card->host)) { pr_info("%s: new %s%s%s card on SPI\n", mmc_hostname(card->host), - mmc_card_highspeed(card) ? "high speed " : "", - mmc_card_ddr_mode(card) ? "DDR " : "", + mmc_card_hs(card) ? "high speed " : "", + mmc_card_ddr52(card) ? "DDR " : "", type); } else { pr_info("%s: new %s%s%s%s%s card at address %04x\n", mmc_hostname(card->host), mmc_card_uhs(card) ? "ultra high speed " : - (mmc_card_highspeed(card) ? "high speed " : ""), + (mmc_card_hs(card) ? "high speed " : ""), + mmc_card_hs400(card) ? "HS400 " : (mmc_card_hs200(card) ? "HS200 " : ""), - mmc_card_ddr_mode(card) ? "DDR " : "", + mmc_card_ddr52(card) ? "DDR " : "", uhs_bus_speed_mode, type, card->rca); } diff --git a/drivers/mmc/core/core.c b/drivers/mmc/core/core.c index acbc3f2aaaf9..7dc0c85fdb60 100644 --- a/drivers/mmc/core/core.c +++ b/drivers/mmc/core/core.c @@ -800,6 +800,10 @@ void mmc_set_data_timeout(struct mmc_data *data, const struct mmc_card *card) data->timeout_ns = limit_us * 1000; data->timeout_clks = 0; } + + /* assign limit value if invalid */ + if (timeout_us == 0) + data->timeout_ns = limit_us * 1000; } /* @@ -1310,31 +1314,38 @@ int mmc_regulator_set_ocr(struct mmc_host *mmc, } EXPORT_SYMBOL_GPL(mmc_regulator_set_ocr); +#endif /* CONFIG_REGULATOR */ + int mmc_regulator_get_supply(struct mmc_host *mmc) { struct device *dev = mmc_dev(mmc); - struct regulator *supply; int ret; - supply = devm_regulator_get(dev, "vmmc"); - mmc->supply.vmmc = supply; + mmc->supply.vmmc = devm_regulator_get_optional(dev, "vmmc"); mmc->supply.vqmmc = devm_regulator_get_optional(dev, "vqmmc"); - if (IS_ERR(supply)) - return PTR_ERR(supply); + if (IS_ERR(mmc->supply.vmmc)) { + if (PTR_ERR(mmc->supply.vmmc) == -EPROBE_DEFER) + return -EPROBE_DEFER; + dev_info(dev, "No vmmc regulator found\n"); + } else { + ret = mmc_regulator_get_ocrmask(mmc->supply.vmmc); + if (ret > 0) + mmc->ocr_avail = ret; + else + dev_warn(dev, "Failed getting OCR mask: %d\n", ret); + } - ret = mmc_regulator_get_ocrmask(supply); - if (ret > 0) - mmc->ocr_avail = ret; - else - dev_warn(mmc_dev(mmc), "Failed getting OCR mask: %d\n", ret); + if (IS_ERR(mmc->supply.vqmmc)) { + if (PTR_ERR(mmc->supply.vqmmc) == -EPROBE_DEFER) + return -EPROBE_DEFER; + dev_info(dev, "No vqmmc regulator found\n"); + } return 0; } EXPORT_SYMBOL_GPL(mmc_regulator_get_supply); -#endif /* CONFIG_REGULATOR */ - /* * Mask off any voltages we don't support and select * the lowest voltage @@ -1533,8 +1544,13 @@ void mmc_power_up(struct mmc_host *host, u32 ocr) host->ios.timing = MMC_TIMING_LEGACY; mmc_set_ios(host); - /* Set signal voltage to 3.3V */ - __mmc_set_signal_voltage(host, MMC_SIGNAL_VOLTAGE_330); + /* Try to set signal voltage to 3.3V but fall back to 1.8v or 1.2v */ + if (__mmc_set_signal_voltage(host, MMC_SIGNAL_VOLTAGE_330) == 0) + dev_dbg(mmc_dev(host), "Initial signal voltage of 3.3v\n"); + else if (__mmc_set_signal_voltage(host, MMC_SIGNAL_VOLTAGE_180) == 0) + dev_dbg(mmc_dev(host), "Initial signal voltage of 1.8v\n"); + else if (__mmc_set_signal_voltage(host, MMC_SIGNAL_VOLTAGE_120) == 0) + dev_dbg(mmc_dev(host), "Initial signal voltage of 1.2v\n"); /* * This delay should be sufficient to allow the power supply @@ -2183,7 +2199,7 @@ int mmc_set_blocklen(struct mmc_card *card, unsigned int blocklen) { struct mmc_command cmd = {0}; - if (mmc_card_blockaddr(card) || mmc_card_ddr_mode(card)) + if (mmc_card_blockaddr(card) || mmc_card_ddr52(card)) return 0; cmd.opcode = MMC_SET_BLOCKLEN; @@ -2263,7 +2279,6 @@ static int mmc_do_hw_reset(struct mmc_host *host, int check) } } - host->card->state &= ~(MMC_STATE_HIGHSPEED | MMC_STATE_HIGHSPEED_DDR); if (mmc_host_is_spi(host)) { host->ios.chip_select = MMC_CS_HIGH; host->ios.bus_mode = MMC_BUSMODE_PUSHPULL; @@ -2403,6 +2418,11 @@ void mmc_rescan(struct work_struct *work) container_of(work, struct mmc_host, detect.work); int i; + if (host->trigger_card_event && host->ops->card_event) { + host->ops->card_event(host); + host->trigger_card_event = false; + } + if (host->rescan_disable) return; diff --git a/drivers/mmc/core/debugfs.c b/drivers/mmc/core/debugfs.c index 54829c0ed000..91eb16223246 100644 --- a/drivers/mmc/core/debugfs.c +++ b/drivers/mmc/core/debugfs.c @@ -135,8 +135,14 @@ static int mmc_ios_show(struct seq_file *s, void *data) case MMC_TIMING_UHS_DDR50: str = "sd uhs DDR50"; break; + case MMC_TIMING_MMC_DDR52: + str = "mmc DDR52"; + break; case MMC_TIMING_MMC_HS200: - str = "mmc high-speed SDR200"; + str = "mmc HS200"; + break; + case MMC_TIMING_MMC_HS400: + str = "mmc HS400"; break; default: str = "invalid"; diff --git a/drivers/mmc/core/host.c b/drivers/mmc/core/host.c index fdea825dbb24..95cceae96944 100644 --- a/drivers/mmc/core/host.c +++ b/drivers/mmc/core/host.c @@ -447,6 +447,10 @@ int mmc_of_parse(struct mmc_host *host) host->caps2 |= MMC_CAP2_HS200_1_8V_SDR; if (of_find_property(np, "mmc-hs200-1_2v", &len)) host->caps2 |= MMC_CAP2_HS200_1_2V_SDR; + if (of_find_property(np, "mmc-hs400-1_8v", &len)) + host->caps2 |= MMC_CAP2_HS400_1_8V | MMC_CAP2_HS200_1_8V_SDR; + if (of_find_property(np, "mmc-hs400-1_2v", &len)) + host->caps2 |= MMC_CAP2_HS400_1_2V | MMC_CAP2_HS200_1_2V_SDR; return 0; diff --git a/drivers/mmc/core/mmc.c b/drivers/mmc/core/mmc.c index 1ab5f3a0af5b..793c6f7ddb04 100644 --- a/drivers/mmc/core/mmc.c +++ b/drivers/mmc/core/mmc.c @@ -240,31 +240,62 @@ static int mmc_get_ext_csd(struct mmc_card *card, u8 **new_ext_csd) static void mmc_select_card_type(struct mmc_card *card) { struct mmc_host *host = card->host; - u8 card_type = card->ext_csd.raw_card_type & EXT_CSD_CARD_TYPE_MASK; + u8 card_type = card->ext_csd.raw_card_type; u32 caps = host->caps, caps2 = host->caps2; - unsigned int hs_max_dtr = 0; + unsigned int hs_max_dtr = 0, hs200_max_dtr = 0; + unsigned int avail_type = 0; - if (card_type & EXT_CSD_CARD_TYPE_26) + if (caps & MMC_CAP_MMC_HIGHSPEED && + card_type & EXT_CSD_CARD_TYPE_HS_26) { hs_max_dtr = MMC_HIGH_26_MAX_DTR; + avail_type |= EXT_CSD_CARD_TYPE_HS_26; + } if (caps & MMC_CAP_MMC_HIGHSPEED && - card_type & EXT_CSD_CARD_TYPE_52) + card_type & EXT_CSD_CARD_TYPE_HS_52) { hs_max_dtr = MMC_HIGH_52_MAX_DTR; + avail_type |= EXT_CSD_CARD_TYPE_HS_52; + } + + if (caps & MMC_CAP_1_8V_DDR && + card_type & EXT_CSD_CARD_TYPE_DDR_1_8V) { + hs_max_dtr = MMC_HIGH_DDR_MAX_DTR; + avail_type |= EXT_CSD_CARD_TYPE_DDR_1_8V; + } - if ((caps & MMC_CAP_1_8V_DDR && - card_type & EXT_CSD_CARD_TYPE_DDR_1_8V) || - (caps & MMC_CAP_1_2V_DDR && - card_type & EXT_CSD_CARD_TYPE_DDR_1_2V)) + if (caps & MMC_CAP_1_2V_DDR && + card_type & EXT_CSD_CARD_TYPE_DDR_1_2V) { hs_max_dtr = MMC_HIGH_DDR_MAX_DTR; + avail_type |= EXT_CSD_CARD_TYPE_DDR_1_2V; + } + + if (caps2 & MMC_CAP2_HS200_1_8V_SDR && + card_type & EXT_CSD_CARD_TYPE_HS200_1_8V) { + hs200_max_dtr = MMC_HS200_MAX_DTR; + avail_type |= EXT_CSD_CARD_TYPE_HS200_1_8V; + } + + if (caps2 & MMC_CAP2_HS200_1_2V_SDR && + card_type & EXT_CSD_CARD_TYPE_HS200_1_2V) { + hs200_max_dtr = MMC_HS200_MAX_DTR; + avail_type |= EXT_CSD_CARD_TYPE_HS200_1_2V; + } + + if (caps2 & MMC_CAP2_HS400_1_8V && + card_type & EXT_CSD_CARD_TYPE_HS400_1_8V) { + hs200_max_dtr = MMC_HS200_MAX_DTR; + avail_type |= EXT_CSD_CARD_TYPE_HS400_1_8V; + } - if ((caps2 & MMC_CAP2_HS200_1_8V_SDR && - card_type & EXT_CSD_CARD_TYPE_SDR_1_8V) || - (caps2 & MMC_CAP2_HS200_1_2V_SDR && - card_type & EXT_CSD_CARD_TYPE_SDR_1_2V)) - hs_max_dtr = MMC_HS200_MAX_DTR; + if (caps2 & MMC_CAP2_HS400_1_2V && + card_type & EXT_CSD_CARD_TYPE_HS400_1_2V) { + hs200_max_dtr = MMC_HS200_MAX_DTR; + avail_type |= EXT_CSD_CARD_TYPE_HS400_1_2V; + } card->ext_csd.hs_max_dtr = hs_max_dtr; - card->ext_csd.card_type = card_type; + card->ext_csd.hs200_max_dtr = hs200_max_dtr; + card->mmc_avail_type = avail_type; } /* @@ -480,6 +511,8 @@ static int mmc_read_ext_csd(struct mmc_card *card, u8 *ext_csd) ext_csd[EXT_CSD_PWR_CL_DDR_52_195]; card->ext_csd.raw_pwr_cl_ddr_52_360 = ext_csd[EXT_CSD_PWR_CL_DDR_52_360]; + card->ext_csd.raw_pwr_cl_ddr_200_360 = + ext_csd[EXT_CSD_PWR_CL_DDR_200_360]; } if (card->ext_csd.rev >= 5) { @@ -646,7 +679,10 @@ static int mmc_compare_ext_csds(struct mmc_card *card, unsigned bus_width) (card->ext_csd.raw_pwr_cl_ddr_52_195 == bw_ext_csd[EXT_CSD_PWR_CL_DDR_52_195]) && (card->ext_csd.raw_pwr_cl_ddr_52_360 == - bw_ext_csd[EXT_CSD_PWR_CL_DDR_52_360])); + bw_ext_csd[EXT_CSD_PWR_CL_DDR_52_360]) && + (card->ext_csd.raw_pwr_cl_ddr_200_360 == + bw_ext_csd[EXT_CSD_PWR_CL_DDR_200_360])); + if (err) err = -EINVAL; @@ -694,18 +730,10 @@ static struct attribute *mmc_std_attrs[] = { &dev_attr_rel_sectors.attr, NULL, }; - -static struct attribute_group mmc_std_attr_group = { - .attrs = mmc_std_attrs, -}; - -static const struct attribute_group *mmc_attr_groups[] = { - &mmc_std_attr_group, - NULL, -}; +ATTRIBUTE_GROUPS(mmc_std); static struct device_type mmc_type = { - .groups = mmc_attr_groups, + .groups = mmc_std_groups, }; /* @@ -714,17 +742,13 @@ static struct device_type mmc_type = { * extended CSD register, select it by executing the * mmc_switch command. */ -static int mmc_select_powerclass(struct mmc_card *card, - unsigned int bus_width) +static int __mmc_select_powerclass(struct mmc_card *card, + unsigned int bus_width) { - int err = 0; + struct mmc_host *host = card->host; + struct mmc_ext_csd *ext_csd = &card->ext_csd; unsigned int pwrclass_val = 0; - struct mmc_host *host; - - BUG_ON(!card); - - host = card->host; - BUG_ON(!host); + int err = 0; /* Power class selection is supported for versions >= 4.0 */ if (card->csd.mmca_vsn < CSD_SPEC_VER_4) @@ -736,14 +760,14 @@ static int mmc_select_powerclass(struct mmc_card *card, switch (1 << host->ios.vdd) { case MMC_VDD_165_195: - if (host->ios.clock <= 26000000) - pwrclass_val = card->ext_csd.raw_pwr_cl_26_195; - else if (host->ios.clock <= 52000000) + if (host->ios.clock <= MMC_HIGH_26_MAX_DTR) + pwrclass_val = ext_csd->raw_pwr_cl_26_195; + else if (host->ios.clock <= MMC_HIGH_52_MAX_DTR) pwrclass_val = (bus_width <= EXT_CSD_BUS_WIDTH_8) ? - card->ext_csd.raw_pwr_cl_52_195 : - card->ext_csd.raw_pwr_cl_ddr_52_195; - else if (host->ios.clock <= 200000000) - pwrclass_val = card->ext_csd.raw_pwr_cl_200_195; + ext_csd->raw_pwr_cl_52_195 : + ext_csd->raw_pwr_cl_ddr_52_195; + else if (host->ios.clock <= MMC_HS200_MAX_DTR) + pwrclass_val = ext_csd->raw_pwr_cl_200_195; break; case MMC_VDD_27_28: case MMC_VDD_28_29: @@ -754,14 +778,16 @@ static int mmc_select_powerclass(struct mmc_card *card, case MMC_VDD_33_34: case MMC_VDD_34_35: case MMC_VDD_35_36: - if (host->ios.clock <= 26000000) - pwrclass_val = card->ext_csd.raw_pwr_cl_26_360; - else if (host->ios.clock <= 52000000) + if (host->ios.clock <= MMC_HIGH_26_MAX_DTR) + pwrclass_val = ext_csd->raw_pwr_cl_26_360; + else if (host->ios.clock <= MMC_HIGH_52_MAX_DTR) pwrclass_val = (bus_width <= EXT_CSD_BUS_WIDTH_8) ? - card->ext_csd.raw_pwr_cl_52_360 : - card->ext_csd.raw_pwr_cl_ddr_52_360; - else if (host->ios.clock <= 200000000) - pwrclass_val = card->ext_csd.raw_pwr_cl_200_360; + ext_csd->raw_pwr_cl_52_360 : + ext_csd->raw_pwr_cl_ddr_52_360; + else if (host->ios.clock <= MMC_HS200_MAX_DTR) + pwrclass_val = (bus_width == EXT_CSD_DDR_BUS_WIDTH_8) ? + ext_csd->raw_pwr_cl_ddr_200_360 : + ext_csd->raw_pwr_cl_200_360; break; default: pr_warning("%s: Voltage range not supported " @@ -787,40 +813,79 @@ static int mmc_select_powerclass(struct mmc_card *card, return err; } +static int mmc_select_powerclass(struct mmc_card *card) +{ + struct mmc_host *host = card->host; + u32 bus_width, ext_csd_bits; + int err, ddr; + + /* Power class selection is supported for versions >= 4.0 */ + if (card->csd.mmca_vsn < CSD_SPEC_VER_4) + return 0; + + bus_width = host->ios.bus_width; + /* Power class values are defined only for 4/8 bit bus */ + if (bus_width == MMC_BUS_WIDTH_1) + return 0; + + ddr = card->mmc_avail_type & EXT_CSD_CARD_TYPE_DDR_52; + if (ddr) + ext_csd_bits = (bus_width == MMC_BUS_WIDTH_8) ? + EXT_CSD_DDR_BUS_WIDTH_8 : EXT_CSD_DDR_BUS_WIDTH_4; + else + ext_csd_bits = (bus_width == MMC_BUS_WIDTH_8) ? + EXT_CSD_BUS_WIDTH_8 : EXT_CSD_BUS_WIDTH_4; + + err = __mmc_select_powerclass(card, ext_csd_bits); + if (err) + pr_warn("%s: power class selection to bus width %d ddr %d failed\n", + mmc_hostname(host), 1 << bus_width, ddr); + + return err; +} + /* - * Selects the desired buswidth and switch to the HS200 mode - * if bus width set without error + * Set the bus speed for the selected speed mode. */ -static int mmc_select_hs200(struct mmc_card *card) +static void mmc_set_bus_speed(struct mmc_card *card) +{ + unsigned int max_dtr = (unsigned int)-1; + + if ((mmc_card_hs200(card) || mmc_card_hs400(card)) && + max_dtr > card->ext_csd.hs200_max_dtr) + max_dtr = card->ext_csd.hs200_max_dtr; + else if (mmc_card_hs(card) && max_dtr > card->ext_csd.hs_max_dtr) + max_dtr = card->ext_csd.hs_max_dtr; + else if (max_dtr > card->csd.max_dtr) + max_dtr = card->csd.max_dtr; + + mmc_set_clock(card->host, max_dtr); +} + +/* + * Select the bus width amoung 4-bit and 8-bit(SDR). + * If the bus width is changed successfully, return the selected width value. + * Zero is returned instead of error value if the wide width is not supported. + */ +static int mmc_select_bus_width(struct mmc_card *card) { - int idx, err = -EINVAL; - struct mmc_host *host; static unsigned ext_csd_bits[] = { - EXT_CSD_BUS_WIDTH_4, EXT_CSD_BUS_WIDTH_8, + EXT_CSD_BUS_WIDTH_4, }; static unsigned bus_widths[] = { - MMC_BUS_WIDTH_4, MMC_BUS_WIDTH_8, + MMC_BUS_WIDTH_4, }; + struct mmc_host *host = card->host; + unsigned idx, bus_width = 0; + int err = 0; - BUG_ON(!card); - - host = card->host; - - if (card->ext_csd.card_type & EXT_CSD_CARD_TYPE_SDR_1_2V && - host->caps2 & MMC_CAP2_HS200_1_2V_SDR) - err = __mmc_set_signal_voltage(host, MMC_SIGNAL_VOLTAGE_120); - - if (err && card->ext_csd.card_type & EXT_CSD_CARD_TYPE_SDR_1_8V && - host->caps2 & MMC_CAP2_HS200_1_8V_SDR) - err = __mmc_set_signal_voltage(host, MMC_SIGNAL_VOLTAGE_180); - - /* If fails try again during next card power cycle */ - if (err) - goto err; + if ((card->csd.mmca_vsn < CSD_SPEC_VER_4) && + !(host->caps & (MMC_CAP_4_BIT_DATA | MMC_CAP_8_BIT_DATA))) + return 0; - idx = (host->caps & MMC_CAP_8_BIT_DATA) ? 1 : 0; + idx = (host->caps & MMC_CAP_8_BIT_DATA) ? 0 : 1; /* * Unlike SD, MMC cards dont have a configuration register to notify @@ -828,8 +893,7 @@ static int mmc_select_hs200(struct mmc_card *card) * the supported bus width or compare the ext csd values of current * bus width and ext csd values of 1 bit mode read earlier. */ - for (; idx >= 0; idx--) { - + for (; idx < ARRAY_SIZE(bus_widths); idx++) { /* * Host is capable of 8bit transfer, then switch * the device to work in 8bit transfer mode. If the @@ -844,27 +908,266 @@ static int mmc_select_hs200(struct mmc_card *card) if (err) continue; - mmc_set_bus_width(card->host, bus_widths[idx]); + bus_width = bus_widths[idx]; + mmc_set_bus_width(host, bus_width); + /* + * If controller can't handle bus width test, + * compare ext_csd previously read in 1 bit mode + * against ext_csd at new bus width + */ if (!(host->caps & MMC_CAP_BUS_WIDTH_TEST)) - err = mmc_compare_ext_csds(card, bus_widths[idx]); + err = mmc_compare_ext_csds(card, bus_width); else - err = mmc_bus_test(card, bus_widths[idx]); - if (!err) + err = mmc_bus_test(card, bus_width); + + if (!err) { + err = bus_width; break; + } else { + pr_warn("%s: switch to bus width %d failed\n", + mmc_hostname(host), ext_csd_bits[idx]); + } } - /* switch to HS200 mode if bus width set successfully */ + return err; +} + +/* + * Switch to the high-speed mode + */ +static int mmc_select_hs(struct mmc_card *card) +{ + int err; + + err = __mmc_switch(card, EXT_CSD_CMD_SET_NORMAL, + EXT_CSD_HS_TIMING, EXT_CSD_TIMING_HS, + card->ext_csd.generic_cmd6_time, + true, true, true); if (!err) + mmc_set_timing(card->host, MMC_TIMING_MMC_HS); + + return err; +} + +/* + * Activate wide bus and DDR if supported. + */ +static int mmc_select_hs_ddr(struct mmc_card *card) +{ + struct mmc_host *host = card->host; + u32 bus_width, ext_csd_bits; + int err = 0; + + if (!(card->mmc_avail_type & EXT_CSD_CARD_TYPE_DDR_52)) + return 0; + + bus_width = host->ios.bus_width; + if (bus_width == MMC_BUS_WIDTH_1) + return 0; + + ext_csd_bits = (bus_width == MMC_BUS_WIDTH_8) ? + EXT_CSD_DDR_BUS_WIDTH_8 : EXT_CSD_DDR_BUS_WIDTH_4; + + err = mmc_switch(card, EXT_CSD_CMD_SET_NORMAL, + EXT_CSD_BUS_WIDTH, + ext_csd_bits, + card->ext_csd.generic_cmd6_time); + if (err) { + pr_warn("%s: switch to bus width %d ddr failed\n", + mmc_hostname(host), 1 << bus_width); + return err; + } + + /* + * eMMC cards can support 3.3V to 1.2V i/o (vccq) + * signaling. + * + * EXT_CSD_CARD_TYPE_DDR_1_8V means 3.3V or 1.8V vccq. + * + * 1.8V vccq at 3.3V core voltage (vcc) is not required + * in the JEDEC spec for DDR. + * + * Do not force change in vccq since we are obviously + * working and no change to vccq is needed. + * + * WARNING: eMMC rules are NOT the same as SD DDR + */ + if (card->mmc_avail_type & EXT_CSD_CARD_TYPE_DDR_1_2V) { + err = __mmc_set_signal_voltage(host, + MMC_SIGNAL_VOLTAGE_120); + if (err) + return err; + } + + mmc_set_timing(host, MMC_TIMING_MMC_DDR52); + + return err; +} + +static int mmc_select_hs400(struct mmc_card *card) +{ + struct mmc_host *host = card->host; + int err = 0; + + /* + * HS400 mode requires 8-bit bus width + */ + if (!(card->mmc_avail_type & EXT_CSD_CARD_TYPE_HS400 && + host->ios.bus_width == MMC_BUS_WIDTH_8)) + return 0; + + /* + * Before switching to dual data rate operation for HS400, + * it is required to convert from HS200 mode to HS mode. + */ + mmc_set_timing(card->host, MMC_TIMING_MMC_HS); + mmc_set_bus_speed(card); + + err = __mmc_switch(card, EXT_CSD_CMD_SET_NORMAL, + EXT_CSD_HS_TIMING, EXT_CSD_TIMING_HS, + card->ext_csd.generic_cmd6_time, + true, true, true); + if (err) { + pr_warn("%s: switch to high-speed from hs200 failed, err:%d\n", + mmc_hostname(host), err); + return err; + } + + err = mmc_switch(card, EXT_CSD_CMD_SET_NORMAL, + EXT_CSD_BUS_WIDTH, + EXT_CSD_DDR_BUS_WIDTH_8, + card->ext_csd.generic_cmd6_time); + if (err) { + pr_warn("%s: switch to bus width for hs400 failed, err:%d\n", + mmc_hostname(host), err); + return err; + } + + err = __mmc_switch(card, EXT_CSD_CMD_SET_NORMAL, + EXT_CSD_HS_TIMING, EXT_CSD_TIMING_HS400, + card->ext_csd.generic_cmd6_time, + true, true, true); + if (err) { + pr_warn("%s: switch to hs400 failed, err:%d\n", + mmc_hostname(host), err); + return err; + } + + mmc_set_timing(host, MMC_TIMING_MMC_HS400); + mmc_set_bus_speed(card); + + return 0; +} + +/* + * For device supporting HS200 mode, the following sequence + * should be done before executing the tuning process. + * 1. set the desired bus width(4-bit or 8-bit, 1-bit is not supported) + * 2. switch to HS200 mode + * 3. set the clock to > 52Mhz and <=200MHz + */ +static int mmc_select_hs200(struct mmc_card *card) +{ + struct mmc_host *host = card->host; + int err = -EINVAL; + + if (card->mmc_avail_type & EXT_CSD_CARD_TYPE_HS200_1_2V) + err = __mmc_set_signal_voltage(host, MMC_SIGNAL_VOLTAGE_120); + + if (err && card->mmc_avail_type & EXT_CSD_CARD_TYPE_HS200_1_8V) + err = __mmc_set_signal_voltage(host, MMC_SIGNAL_VOLTAGE_180); + + /* If fails try again during next card power cycle */ + if (err) + goto err; + + /* + * Set the bus width(4 or 8) with host's support and + * switch to HS200 mode if bus width is set successfully. + */ + err = mmc_select_bus_width(card); + if (!IS_ERR_VALUE(err)) { err = __mmc_switch(card, EXT_CSD_CMD_SET_NORMAL, - EXT_CSD_HS_TIMING, 2, - card->ext_csd.generic_cmd6_time, - true, true, true); + EXT_CSD_HS_TIMING, EXT_CSD_TIMING_HS200, + card->ext_csd.generic_cmd6_time, + true, true, true); + if (!err) + mmc_set_timing(host, MMC_TIMING_MMC_HS200); + } err: return err; } /* + * Activate High Speed or HS200 mode if supported. + */ +static int mmc_select_timing(struct mmc_card *card) +{ + int err = 0; + + if ((card->csd.mmca_vsn < CSD_SPEC_VER_4 && + card->ext_csd.hs_max_dtr == 0)) + goto bus_speed; + + if (card->mmc_avail_type & EXT_CSD_CARD_TYPE_HS200) + err = mmc_select_hs200(card); + else if (card->mmc_avail_type & EXT_CSD_CARD_TYPE_HS) + err = mmc_select_hs(card); + + if (err && err != -EBADMSG) + return err; + + if (err) { + pr_warn("%s: switch to %s failed\n", + mmc_card_hs(card) ? "high-speed" : + (mmc_card_hs200(card) ? "hs200" : ""), + mmc_hostname(card->host)); + err = 0; + } + +bus_speed: + /* + * Set the bus speed to the selected bus timing. + * If timing is not selected, backward compatible is the default. + */ + mmc_set_bus_speed(card); + return err; +} + +/* + * Execute tuning sequence to seek the proper bus operating + * conditions for HS200 and HS400, which sends CMD21 to the device. + */ +static int mmc_hs200_tuning(struct mmc_card *card) +{ + struct mmc_host *host = card->host; + int err = 0; + + /* + * Timing should be adjusted to the HS400 target + * operation frequency for tuning process + */ + if (card->mmc_avail_type & EXT_CSD_CARD_TYPE_HS400 && + host->ios.bus_width == MMC_BUS_WIDTH_8) + if (host->ops->prepare_hs400_tuning) + host->ops->prepare_hs400_tuning(host, &host->ios); + + if (host->ops->execute_tuning) { + mmc_host_clk_hold(host); + err = host->ops->execute_tuning(host, + MMC_SEND_TUNING_BLOCK_HS200); + mmc_host_clk_release(host); + + if (err) + pr_warn("%s: tuning execution failed\n", + mmc_hostname(host)); + } + + return err; +} + +/* * Handle the detection and initialisation of a card. * * In the case of a resume, "oldcard" will contain the card @@ -874,9 +1177,8 @@ static int mmc_init_card(struct mmc_host *host, u32 ocr, struct mmc_card *oldcard) { struct mmc_card *card; - int err, ddr = 0; + int err; u32 cid[4]; - unsigned int max_dtr; u32 rocr; u8 *ext_csd = NULL; @@ -1068,206 +1370,34 @@ static int mmc_init_card(struct mmc_host *host, u32 ocr, } /* - * Activate high speed (if supported) - */ - if (card->ext_csd.hs_max_dtr != 0) { - err = 0; - if (card->ext_csd.hs_max_dtr > 52000000 && - host->caps2 & MMC_CAP2_HS200) - err = mmc_select_hs200(card); - else if (host->caps & MMC_CAP_MMC_HIGHSPEED) - err = __mmc_switch(card, EXT_CSD_CMD_SET_NORMAL, - EXT_CSD_HS_TIMING, 1, - card->ext_csd.generic_cmd6_time, - true, true, true); - - if (err && err != -EBADMSG) - goto free_card; - - if (err) { - pr_warning("%s: switch to highspeed failed\n", - mmc_hostname(card->host)); - err = 0; - } else { - if (card->ext_csd.hs_max_dtr > 52000000 && - host->caps2 & MMC_CAP2_HS200) { - mmc_card_set_hs200(card); - mmc_set_timing(card->host, - MMC_TIMING_MMC_HS200); - } else { - mmc_card_set_highspeed(card); - mmc_set_timing(card->host, MMC_TIMING_MMC_HS); - } - } - } - - /* - * Compute bus speed. - */ - max_dtr = (unsigned int)-1; - - if (mmc_card_highspeed(card) || mmc_card_hs200(card)) { - if (max_dtr > card->ext_csd.hs_max_dtr) - max_dtr = card->ext_csd.hs_max_dtr; - if (mmc_card_highspeed(card) && (max_dtr > 52000000)) - max_dtr = 52000000; - } else if (max_dtr > card->csd.max_dtr) { - max_dtr = card->csd.max_dtr; - } - - mmc_set_clock(host, max_dtr); - - /* - * Indicate DDR mode (if supported). + * Select timing interface */ - if (mmc_card_highspeed(card)) { - if ((card->ext_csd.card_type & EXT_CSD_CARD_TYPE_DDR_1_8V) - && (host->caps & MMC_CAP_1_8V_DDR)) - ddr = MMC_1_8V_DDR_MODE; - else if ((card->ext_csd.card_type & EXT_CSD_CARD_TYPE_DDR_1_2V) - && (host->caps & MMC_CAP_1_2V_DDR)) - ddr = MMC_1_2V_DDR_MODE; - } + err = mmc_select_timing(card); + if (err) + goto free_card; - /* - * Indicate HS200 SDR mode (if supported). - */ if (mmc_card_hs200(card)) { - u32 ext_csd_bits; - u32 bus_width = card->host->ios.bus_width; - - /* - * For devices supporting HS200 mode, the bus width has - * to be set before executing the tuning function. If - * set before tuning, then device will respond with CRC - * errors for responses on CMD line. So for HS200 the - * sequence will be - * 1. set bus width 4bit / 8 bit (1 bit not supported) - * 2. switch to HS200 mode - * 3. set the clock to > 52Mhz <=200MHz and - * 4. execute tuning for HS200 - */ - if ((host->caps2 & MMC_CAP2_HS200) && - card->host->ops->execute_tuning) { - mmc_host_clk_hold(card->host); - err = card->host->ops->execute_tuning(card->host, - MMC_SEND_TUNING_BLOCK_HS200); - mmc_host_clk_release(card->host); - } - if (err) { - pr_warning("%s: tuning execution failed\n", - mmc_hostname(card->host)); + err = mmc_hs200_tuning(card); + if (err) goto err; - } - ext_csd_bits = (bus_width == MMC_BUS_WIDTH_8) ? - EXT_CSD_BUS_WIDTH_8 : EXT_CSD_BUS_WIDTH_4; - err = mmc_select_powerclass(card, ext_csd_bits); + err = mmc_select_hs400(card); if (err) - pr_warning("%s: power class selection to bus width %d" - " failed\n", mmc_hostname(card->host), - 1 << bus_width); + goto err; + } else if (mmc_card_hs(card)) { + /* Select the desired bus width optionally */ + err = mmc_select_bus_width(card); + if (!IS_ERR_VALUE(err)) { + err = mmc_select_hs_ddr(card); + if (err) + goto err; + } } /* - * Activate wide bus and DDR (if supported). + * Choose the power class with selected bus interface */ - if (!mmc_card_hs200(card) && - (card->csd.mmca_vsn >= CSD_SPEC_VER_4) && - (host->caps & (MMC_CAP_4_BIT_DATA | MMC_CAP_8_BIT_DATA))) { - static unsigned ext_csd_bits[][2] = { - { EXT_CSD_BUS_WIDTH_8, EXT_CSD_DDR_BUS_WIDTH_8 }, - { EXT_CSD_BUS_WIDTH_4, EXT_CSD_DDR_BUS_WIDTH_4 }, - { EXT_CSD_BUS_WIDTH_1, EXT_CSD_BUS_WIDTH_1 }, - }; - static unsigned bus_widths[] = { - MMC_BUS_WIDTH_8, - MMC_BUS_WIDTH_4, - MMC_BUS_WIDTH_1 - }; - unsigned idx, bus_width = 0; - - if (host->caps & MMC_CAP_8_BIT_DATA) - idx = 0; - else - idx = 1; - for (; idx < ARRAY_SIZE(bus_widths); idx++) { - bus_width = bus_widths[idx]; - if (bus_width == MMC_BUS_WIDTH_1) - ddr = 0; /* no DDR for 1-bit width */ - err = mmc_select_powerclass(card, ext_csd_bits[idx][0]); - if (err) - pr_warning("%s: power class selection to " - "bus width %d failed\n", - mmc_hostname(card->host), - 1 << bus_width); - - err = mmc_switch(card, EXT_CSD_CMD_SET_NORMAL, - EXT_CSD_BUS_WIDTH, - ext_csd_bits[idx][0], - card->ext_csd.generic_cmd6_time); - if (!err) { - mmc_set_bus_width(card->host, bus_width); - - /* - * If controller can't handle bus width test, - * compare ext_csd previously read in 1 bit mode - * against ext_csd at new bus width - */ - if (!(host->caps & MMC_CAP_BUS_WIDTH_TEST)) - err = mmc_compare_ext_csds(card, - bus_width); - else - err = mmc_bus_test(card, bus_width); - if (!err) - break; - } - } - - if (!err && ddr) { - err = mmc_select_powerclass(card, ext_csd_bits[idx][1]); - if (err) - pr_warning("%s: power class selection to " - "bus width %d ddr %d failed\n", - mmc_hostname(card->host), - 1 << bus_width, ddr); - - err = mmc_switch(card, EXT_CSD_CMD_SET_NORMAL, - EXT_CSD_BUS_WIDTH, - ext_csd_bits[idx][1], - card->ext_csd.generic_cmd6_time); - } - if (err) { - pr_warning("%s: switch to bus width %d ddr %d " - "failed\n", mmc_hostname(card->host), - 1 << bus_width, ddr); - goto free_card; - } else if (ddr) { - /* - * eMMC cards can support 3.3V to 1.2V i/o (vccq) - * signaling. - * - * EXT_CSD_CARD_TYPE_DDR_1_8V means 3.3V or 1.8V vccq. - * - * 1.8V vccq at 3.3V core voltage (vcc) is not required - * in the JEDEC spec for DDR. - * - * Do not force change in vccq since we are obviously - * working and no change to vccq is needed. - * - * WARNING: eMMC rules are NOT the same as SD DDR - */ - if (ddr == MMC_1_2V_DDR_MODE) { - err = __mmc_set_signal_voltage(host, - MMC_SIGNAL_VOLTAGE_120); - if (err) - goto err; - } - mmc_card_set_ddr_mode(card); - mmc_set_timing(card->host, MMC_TIMING_UHS_DDR50); - mmc_set_bus_width(card->host, bus_width); - } - } + mmc_select_powerclass(card); /* * Enable HPI feature (if supported) @@ -1507,7 +1637,6 @@ static int _mmc_suspend(struct mmc_host *host, bool is_suspend) err = mmc_sleep(host); else if (!mmc_host_is_spi(host)) err = mmc_deselect_cards(host); - host->card->state &= ~(MMC_STATE_HIGHSPEED | MMC_STATE_HIGHSPEED_200); if (!err) { mmc_power_off(host); @@ -1637,7 +1766,6 @@ static int mmc_power_restore(struct mmc_host *host) { int ret; - host->card->state &= ~(MMC_STATE_HIGHSPEED | MMC_STATE_HIGHSPEED_200); mmc_claim_host(host); ret = mmc_init_card(host, host->card->ocr, host->card); mmc_release_host(host); diff --git a/drivers/mmc/core/sd.c b/drivers/mmc/core/sd.c index 2dd359d2242f..0c44510bf717 100644 --- a/drivers/mmc/core/sd.c +++ b/drivers/mmc/core/sd.c @@ -707,18 +707,10 @@ static struct attribute *sd_std_attrs[] = { &dev_attr_serial.attr, NULL, }; - -static struct attribute_group sd_std_attr_group = { - .attrs = sd_std_attrs, -}; - -static const struct attribute_group *sd_attr_groups[] = { - &sd_std_attr_group, - NULL, -}; +ATTRIBUTE_GROUPS(sd_std); struct device_type sd_type = { - .groups = sd_attr_groups, + .groups = sd_std_groups, }; /* @@ -895,7 +887,7 @@ unsigned mmc_sd_get_max_clock(struct mmc_card *card) { unsigned max_dtr = (unsigned int)-1; - if (mmc_card_highspeed(card)) { + if (mmc_card_hs(card)) { if (max_dtr > card->sw_caps.hs_max_dtr) max_dtr = card->sw_caps.hs_max_dtr; } else if (max_dtr > card->csd.max_dtr) { @@ -905,12 +897,6 @@ unsigned mmc_sd_get_max_clock(struct mmc_card *card) return max_dtr; } -void mmc_sd_go_highspeed(struct mmc_card *card) -{ - mmc_card_set_highspeed(card); - mmc_set_timing(card->host, MMC_TIMING_SD_HS); -} - /* * Handle the detection and initialisation of a card. * @@ -985,16 +971,13 @@ static int mmc_sd_init_card(struct mmc_host *host, u32 ocr, err = mmc_sd_init_uhs_card(card); if (err) goto free_card; - - /* Card is an ultra-high-speed card */ - mmc_card_set_uhs(card); } else { /* * Attempt to change to high-speed (if supported) */ err = mmc_sd_switch_hs(card); if (err > 0) - mmc_sd_go_highspeed(card); + mmc_set_timing(card->host, MMC_TIMING_SD_HS); else if (err) goto free_card; @@ -1089,7 +1072,7 @@ static int _mmc_sd_suspend(struct mmc_host *host) if (!mmc_host_is_spi(host)) err = mmc_deselect_cards(host); - host->card->state &= ~MMC_STATE_HIGHSPEED; + if (!err) { mmc_power_off(host); mmc_card_set_suspended(host->card); @@ -1198,7 +1181,6 @@ static int mmc_sd_power_restore(struct mmc_host *host) { int ret; - host->card->state &= ~MMC_STATE_HIGHSPEED; mmc_claim_host(host); ret = mmc_sd_init_card(host, host->card->ocr, host->card); mmc_release_host(host); diff --git a/drivers/mmc/core/sd.h b/drivers/mmc/core/sd.h index 4b34b24f3f76..aab824a9a7f3 100644 --- a/drivers/mmc/core/sd.h +++ b/drivers/mmc/core/sd.h @@ -12,6 +12,5 @@ int mmc_sd_setup_card(struct mmc_host *host, struct mmc_card *card, bool reinit); unsigned mmc_sd_get_max_clock(struct mmc_card *card); int mmc_sd_switch_hs(struct mmc_card *card); -void mmc_sd_go_highspeed(struct mmc_card *card); #endif diff --git a/drivers/mmc/core/sdio.c b/drivers/mmc/core/sdio.c index 4d721c6e2af0..e636d9e99e4a 100644 --- a/drivers/mmc/core/sdio.c +++ b/drivers/mmc/core/sdio.c @@ -363,7 +363,7 @@ static unsigned mmc_sdio_get_max_clock(struct mmc_card *card) { unsigned max_dtr; - if (mmc_card_highspeed(card)) { + if (mmc_card_hs(card)) { /* * The SDIO specification doesn't mention how * the CIS transfer speed register relates to @@ -733,7 +733,6 @@ try_again: mmc_set_clock(host, card->cis.max_dtr); if (card->cccr.high_speed) { - mmc_card_set_highspeed(card); mmc_set_timing(card->host, MMC_TIMING_SD_HS); } @@ -792,16 +791,13 @@ try_again: err = mmc_sdio_init_uhs_card(card); if (err) goto remove; - - /* Card is an ultra-high-speed card */ - mmc_card_set_uhs(card); } else { /* * Switch to high-speed (if supported). */ err = sdio_enable_hs(card); if (err > 0) - mmc_sd_go_highspeed(card); + mmc_set_timing(card->host, MMC_TIMING_SD_HS); else if (err) goto remove; @@ -943,40 +939,21 @@ static int mmc_sdio_pre_suspend(struct mmc_host *host) */ static int mmc_sdio_suspend(struct mmc_host *host) { - int i, err = 0; - - for (i = 0; i < host->card->sdio_funcs; i++) { - struct sdio_func *func = host->card->sdio_func[i]; - if (func && sdio_func_present(func) && func->dev.driver) { - const struct dev_pm_ops *pmops = func->dev.driver->pm; - err = pmops->suspend(&func->dev); - if (err) - break; - } - } - while (err && --i >= 0) { - struct sdio_func *func = host->card->sdio_func[i]; - if (func && sdio_func_present(func) && func->dev.driver) { - const struct dev_pm_ops *pmops = func->dev.driver->pm; - pmops->resume(&func->dev); - } - } - - if (!err && mmc_card_keep_power(host) && mmc_card_wake_sdio_irq(host)) { + if (mmc_card_keep_power(host) && mmc_card_wake_sdio_irq(host)) { mmc_claim_host(host); sdio_disable_wide(host->card); mmc_release_host(host); } - if (!err && !mmc_card_keep_power(host)) + if (!mmc_card_keep_power(host)) mmc_power_off(host); - return err; + return 0; } static int mmc_sdio_resume(struct mmc_host *host) { - int i, err = 0; + int err = 0; BUG_ON(!host); BUG_ON(!host->card); @@ -1019,24 +996,6 @@ static int mmc_sdio_resume(struct mmc_host *host) wake_up_process(host->sdio_irq_thread); mmc_release_host(host); - /* - * If the card looked to be the same as before suspending, then - * we proceed to resume all card functions. If one of them returns - * an error then we simply return that error to the core and the - * card will be redetected as new. It is the responsibility of - * the function driver to perform further tests with the extra - * knowledge it has of the card to confirm the card is indeed the - * same as before suspending (same MAC address for network cards, - * etc.) and return an error otherwise. - */ - for (i = 0; !err && i < host->card->sdio_funcs; i++) { - struct sdio_func *func = host->card->sdio_func[i]; - if (func && sdio_func_present(func) && func->dev.driver) { - const struct dev_pm_ops *pmops = func->dev.driver->pm; - err = pmops->resume(&func->dev); - } - } - host->pm_flags &= ~MMC_PM_KEEP_POWER; return err; } diff --git a/drivers/mmc/core/sdio_bus.c b/drivers/mmc/core/sdio_bus.c index 92d1ba8e8153..4fa8fef9147f 100644 --- a/drivers/mmc/core/sdio_bus.c +++ b/drivers/mmc/core/sdio_bus.c @@ -197,20 +197,8 @@ static int sdio_bus_remove(struct device *dev) #ifdef CONFIG_PM -#ifdef CONFIG_PM_SLEEP -static int pm_no_operation(struct device *dev) -{ - /* - * Prevent the PM core from calling SDIO device drivers' suspend - * callback routines, which it is not supposed to do, by using this - * empty function as the bus type suspend callaback for SDIO. - */ - return 0; -} -#endif - static const struct dev_pm_ops sdio_bus_pm_ops = { - SET_SYSTEM_SLEEP_PM_OPS(pm_no_operation, pm_no_operation) + SET_SYSTEM_SLEEP_PM_OPS(pm_generic_suspend, pm_generic_resume) SET_RUNTIME_PM_OPS( pm_generic_runtime_suspend, pm_generic_runtime_resume, diff --git a/drivers/mmc/core/sdio_irq.c b/drivers/mmc/core/sdio_irq.c index aaa90460ed23..5cc13c8d35bb 100644 --- a/drivers/mmc/core/sdio_irq.c +++ b/drivers/mmc/core/sdio_irq.c @@ -90,6 +90,15 @@ static int process_sdio_pending_irqs(struct mmc_host *host) return ret; } +void sdio_run_irqs(struct mmc_host *host) +{ + mmc_claim_host(host); + host->sdio_irq_pending = true; + process_sdio_pending_irqs(host); + mmc_release_host(host); +} +EXPORT_SYMBOL_GPL(sdio_run_irqs); + static int sdio_irq_thread(void *_host) { struct mmc_host *host = _host; @@ -189,14 +198,20 @@ static int sdio_card_irq_get(struct mmc_card *card) WARN_ON(!host->claimed); if (!host->sdio_irqs++) { - atomic_set(&host->sdio_irq_thread_abort, 0); - host->sdio_irq_thread = - kthread_run(sdio_irq_thread, host, "ksdioirqd/%s", - mmc_hostname(host)); - if (IS_ERR(host->sdio_irq_thread)) { - int err = PTR_ERR(host->sdio_irq_thread); - host->sdio_irqs--; - return err; + if (!(host->caps2 & MMC_CAP2_SDIO_IRQ_NOTHREAD)) { + atomic_set(&host->sdio_irq_thread_abort, 0); + host->sdio_irq_thread = + kthread_run(sdio_irq_thread, host, + "ksdioirqd/%s", mmc_hostname(host)); + if (IS_ERR(host->sdio_irq_thread)) { + int err = PTR_ERR(host->sdio_irq_thread); + host->sdio_irqs--; + return err; + } + } else { + mmc_host_clk_hold(host); + host->ops->enable_sdio_irq(host, 1); + mmc_host_clk_release(host); } } @@ -211,8 +226,14 @@ static int sdio_card_irq_put(struct mmc_card *card) BUG_ON(host->sdio_irqs < 1); if (!--host->sdio_irqs) { - atomic_set(&host->sdio_irq_thread_abort, 1); - kthread_stop(host->sdio_irq_thread); + if (!(host->caps2 & MMC_CAP2_SDIO_IRQ_NOTHREAD)) { + atomic_set(&host->sdio_irq_thread_abort, 1); + kthread_stop(host->sdio_irq_thread); + } else { + mmc_host_clk_hold(host); + host->ops->enable_sdio_irq(host, 0); + mmc_host_clk_release(host); + } } return 0; diff --git a/drivers/mmc/core/slot-gpio.c b/drivers/mmc/core/slot-gpio.c index f7650b899e3d..5f89cb83d5f0 100644 --- a/drivers/mmc/core/slot-gpio.c +++ b/drivers/mmc/core/slot-gpio.c @@ -32,9 +32,7 @@ static irqreturn_t mmc_gpio_cd_irqt(int irq, void *dev_id) /* Schedule a card detection after a debounce timeout */ struct mmc_host *host = dev_id; - if (host->ops->card_event) - host->ops->card_event(host); - + host->trigger_card_event = true; mmc_detect_change(host, msecs_to_jiffies(200)); return IRQ_HANDLED; diff --git a/drivers/mmc/host/Kconfig b/drivers/mmc/host/Kconfig index 779368b683d0..7fee22432e94 100644 --- a/drivers/mmc/host/Kconfig +++ b/drivers/mmc/host/Kconfig @@ -168,7 +168,7 @@ config MMC_SDHCI_ESDHC_IMX config MMC_SDHCI_DOVE tristate "SDHCI support on Marvell's Dove SoC" - depends on ARCH_DOVE + depends on ARCH_DOVE || MACH_DOVE depends on MMC_SDHCI_PLTFM select MMC_SDHCI_IO_ACCESSORS help @@ -283,6 +283,15 @@ config MMC_SDHCI_BCM2835 If unsure, say N. +config MMC_MOXART + tristate "MOXART SD/MMC Host Controller support" + depends on ARCH_MOXART && MMC + help + This selects support for the MOXART SD/MMC Host Controller. + MOXA provides one multi-functional card reader which can + be found on some embedded hardware such as UC-7112-LX. + If you have a controller with this interface, say Y here. + config MMC_OMAP tristate "TI OMAP Multimedia Card Interface support" depends on ARCH_OMAP @@ -688,6 +697,12 @@ config MMC_WMT To compile this driver as a module, choose M here: the module will be called wmt-sdmmc. +config MMC_USDHI6ROL0 + tristate "Renesas USDHI6ROL0 SD/SDIO Host Controller support" + help + This selects support for the Renesas USDHI6ROL0 SD/SDIO + Host Controller + config MMC_REALTEK_PCI tristate "Realtek PCI-E SD/MMC Card Interface Driver" depends on MFD_RTSX_PCI diff --git a/drivers/mmc/host/Makefile b/drivers/mmc/host/Makefile index 61cbc241935b..7f81ddf1dd2c 100644 --- a/drivers/mmc/host/Makefile +++ b/drivers/mmc/host/Makefile @@ -50,7 +50,9 @@ obj-$(CONFIG_MMC_JZ4740) += jz4740_mmc.o obj-$(CONFIG_MMC_VUB300) += vub300.o obj-$(CONFIG_MMC_USHC) += ushc.o obj-$(CONFIG_MMC_WMT) += wmt-sdmmc.o +obj-$(CONFIG_MMC_MOXART) += moxart-mmc.o obj-$(CONFIG_MMC_SUNXI) += sunxi-mmc.o +obj-$(CONFIG_MMC_USDHI6ROL0) += usdhi6rol0.o obj-$(CONFIG_MMC_REALTEK_PCI) += rtsx_pci_sdmmc.o obj-$(CONFIG_MMC_REALTEK_USB) += rtsx_usb_sdmmc.o diff --git a/drivers/mmc/host/atmel-mci.c b/drivers/mmc/host/atmel-mci.c index 42706ea0ba85..aece7cafbb97 100644 --- a/drivers/mmc/host/atmel-mci.c +++ b/drivers/mmc/host/atmel-mci.c @@ -820,16 +820,9 @@ static void atmci_pdc_complete(struct atmel_mci *host) atmci_pdc_cleanup(host); - /* - * If the card was removed, data will be NULL. No point trying - * to send the stop command or waiting for NBUSY in this case. - */ - if (host->data) { - dev_dbg(&host->pdev->dev, - "(%s) set pending xfer complete\n", __func__); - atmci_set_pending(host, EVENT_XFER_COMPLETE); - tasklet_schedule(&host->tasklet); - } + dev_dbg(&host->pdev->dev, "(%s) set pending xfer complete\n", __func__); + atmci_set_pending(host, EVENT_XFER_COMPLETE); + tasklet_schedule(&host->tasklet); } static void atmci_dma_cleanup(struct atmel_mci *host) diff --git a/drivers/mmc/host/dw_mmc-exynos.c b/drivers/mmc/host/dw_mmc-exynos.c index 3423c5ed50c7..0fbc53ac7eae 100644 --- a/drivers/mmc/host/dw_mmc-exynos.c +++ b/drivers/mmc/host/dw_mmc-exynos.c @@ -187,7 +187,7 @@ static void dw_mci_exynos_set_ios(struct dw_mci *host, struct mmc_ios *ios) unsigned long actual; u8 div = priv->ciu_div + 1; - if (ios->timing == MMC_TIMING_UHS_DDR50) { + if (ios->timing == MMC_TIMING_MMC_DDR52) { mci_writel(host, CLKSEL, priv->ddr_timing); /* Should be double rate for DDR mode */ if (ios->bus_width == MMC_BUS_WIDTH_8) @@ -386,8 +386,7 @@ static int dw_mci_exynos_execute_tuning(struct dw_mci_slot *slot, u32 opcode, /* Common capabilities of Exynos4/Exynos5 SoC */ static unsigned long exynos_dwmmc_caps[4] = { - MMC_CAP_UHS_DDR50 | MMC_CAP_1_8V_DDR | - MMC_CAP_8_BIT_DATA | MMC_CAP_CMD23, + MMC_CAP_1_8V_DDR | MMC_CAP_8_BIT_DATA | MMC_CAP_CMD23, MMC_CAP_CMD23, MMC_CAP_CMD23, MMC_CAP_CMD23, @@ -426,7 +425,7 @@ static int dw_mci_exynos_probe(struct platform_device *pdev) return dw_mci_pltfm_register(pdev, drv_data); } -const struct dev_pm_ops dw_mci_exynos_pmops = { +static const struct dev_pm_ops dw_mci_exynos_pmops = { SET_SYSTEM_SLEEP_PM_OPS(dw_mci_exynos_suspend, dw_mci_exynos_resume) .resume_noirq = dw_mci_exynos_resume_noirq, .thaw_noirq = dw_mci_exynos_resume_noirq, diff --git a/drivers/mmc/host/dw_mmc.c b/drivers/mmc/host/dw_mmc.c index cced599d5aeb..1ac227c603b7 100644 --- a/drivers/mmc/host/dw_mmc.c +++ b/drivers/mmc/host/dw_mmc.c @@ -235,12 +235,6 @@ err: } #endif /* defined(CONFIG_DEBUG_FS) */ -static void dw_mci_set_timeout(struct dw_mci *host) -{ - /* timeout (maximum) */ - mci_writel(host, TMOUT, 0xffffffff); -} - static u32 dw_mci_prepare_command(struct mmc_host *mmc, struct mmc_command *cmd) { struct mmc_data *data; @@ -257,9 +251,8 @@ static u32 dw_mci_prepare_command(struct mmc_host *mmc, struct mmc_command *cmd) (cmd->opcode == SD_IO_RW_DIRECT && ((cmd->arg >> 9) & 0x1FFFF) == SDIO_CCCR_ABORT)) cmdr |= SDMMC_CMD_STOP; - else - if (cmd->opcode != MMC_SEND_STATUS && cmd->data) - cmdr |= SDMMC_CMD_PRV_DAT_WAIT; + else if (cmd->opcode != MMC_SEND_STATUS && cmd->data) + cmdr |= SDMMC_CMD_PRV_DAT_WAIT; if (cmd->flags & MMC_RSP_PRESENT) { /* We expect a response, so set this bit */ @@ -850,8 +843,6 @@ static void __dw_mci_start_request(struct dw_mci *host, u32 cmdflags; mrq = slot->mrq; - if (host->pdata->select_slot) - host->pdata->select_slot(slot->id); host->cur_slot = slot; host->mrq = mrq; @@ -864,7 +855,7 @@ static void __dw_mci_start_request(struct dw_mci *host, data = cmd->data; if (data) { - dw_mci_set_timeout(host); + mci_writel(host, TMOUT, 0xFFFFFFFF); mci_writel(host, BYTCNT, data->blksz*data->blocks); mci_writel(host, BLKSIZ, data->blksz); } @@ -962,7 +953,7 @@ static void dw_mci_set_ios(struct mmc_host *mmc, struct mmc_ios *ios) regs = mci_readl(slot->host, UHS_REG); /* DDR mode set */ - if (ios->timing == MMC_TIMING_UHS_DDR50) + if (ios->timing == MMC_TIMING_MMC_DDR52) regs |= ((0x1 << slot->id) << 16); else regs &= ~((0x1 << slot->id) << 16); @@ -985,17 +976,11 @@ static void dw_mci_set_ios(struct mmc_host *mmc, struct mmc_ios *ios) switch (ios->power_mode) { case MMC_POWER_UP: set_bit(DW_MMC_CARD_NEED_INIT, &slot->flags); - /* Power up slot */ - if (slot->host->pdata->setpower) - slot->host->pdata->setpower(slot->id, mmc->ocr_avail); regs = mci_readl(slot->host, PWREN); regs |= (1 << slot->id); mci_writel(slot->host, PWREN, regs); break; case MMC_POWER_OFF: - /* Power down slot */ - if (slot->host->pdata->setpower) - slot->host->pdata->setpower(slot->id, 0); regs = mci_readl(slot->host, PWREN); regs &= ~(1 << slot->id); mci_writel(slot->host, PWREN, regs); @@ -1009,15 +994,13 @@ static int dw_mci_get_ro(struct mmc_host *mmc) { int read_only; struct dw_mci_slot *slot = mmc_priv(mmc); - struct dw_mci_board *brd = slot->host->pdata; + int gpio_ro = mmc_gpio_get_ro(mmc); /* Use platform get_ro function, else try on board write protect */ if (slot->quirks & DW_MCI_SLOT_QUIRK_NO_WRITE_PROTECT) read_only = 0; - else if (brd->get_ro) - read_only = brd->get_ro(slot->id); - else if (gpio_is_valid(slot->wp_gpio)) - read_only = gpio_get_value(slot->wp_gpio); + else if (!IS_ERR_VALUE(gpio_ro)) + read_only = gpio_ro; else read_only = mci_readl(slot->host, WRTPRT) & (1 << slot->id) ? 1 : 0; @@ -1039,8 +1022,6 @@ static int dw_mci_get_cd(struct mmc_host *mmc) /* Use platform get_cd function, else try onboard card detect */ if (brd->quirks & DW_MCI_QUIRK_BROKEN_CARD_DETECTION) present = 1; - else if (brd->get_cd) - present = !brd->get_cd(slot->id); else if (!IS_ERR_VALUE(gpio_cd)) present = gpio_cd; else @@ -1248,7 +1229,7 @@ static int dw_mci_data_complete(struct dw_mci *host, struct mmc_data *data) data->error = -EIO; } - dev_err(host->dev, "data error, status 0x%08x\n", status); + dev_dbg(host->dev, "data error, status 0x%08x\n", status); /* * After an error, there may be data lingering @@ -2045,86 +2026,15 @@ static int dw_mci_of_get_slot_quirks(struct device *dev, u8 slot) return quirks; } - -/* find out bus-width for a given slot */ -static u32 dw_mci_of_get_bus_wd(struct device *dev, u8 slot) -{ - struct device_node *np = dw_mci_of_find_slot_node(dev, slot); - u32 bus_wd = 1; - - if (!np) - return 1; - - if (of_property_read_u32(np, "bus-width", &bus_wd)) - dev_err(dev, "bus-width property not found, assuming width" - " as 1\n"); - return bus_wd; -} - -/* find the write protect gpio for a given slot; or -1 if none specified */ -static int dw_mci_of_get_wp_gpio(struct device *dev, u8 slot) -{ - struct device_node *np = dw_mci_of_find_slot_node(dev, slot); - int gpio; - - if (!np) - return -EINVAL; - - gpio = of_get_named_gpio(np, "wp-gpios", 0); - - /* Having a missing entry is valid; return silently */ - if (!gpio_is_valid(gpio)) - return -EINVAL; - - if (devm_gpio_request(dev, gpio, "dw-mci-wp")) { - dev_warn(dev, "gpio [%d] request failed\n", gpio); - return -EINVAL; - } - - return gpio; -} - -/* find the cd gpio for a given slot */ -static void dw_mci_of_get_cd_gpio(struct device *dev, u8 slot, - struct mmc_host *mmc) -{ - struct device_node *np = dw_mci_of_find_slot_node(dev, slot); - int gpio; - - if (!np) - return; - - gpio = of_get_named_gpio(np, "cd-gpios", 0); - - /* Having a missing entry is valid; return silently */ - if (!gpio_is_valid(gpio)) - return; - - if (mmc_gpio_request_cd(mmc, gpio, 0)) - dev_warn(dev, "gpio [%d] request failed\n", gpio); -} #else /* CONFIG_OF */ static int dw_mci_of_get_slot_quirks(struct device *dev, u8 slot) { return 0; } -static u32 dw_mci_of_get_bus_wd(struct device *dev, u8 slot) -{ - return 1; -} static struct device_node *dw_mci_of_find_slot_node(struct device *dev, u8 slot) { return NULL; } -static int dw_mci_of_get_wp_gpio(struct device *dev, u8 slot) -{ - return -EINVAL; -} -static void dw_mci_of_get_cd_gpio(struct device *dev, u8 slot, - struct mmc_host *mmc) -{ - return; -} #endif /* CONFIG_OF */ static int dw_mci_init_slot(struct dw_mci *host, unsigned int id) @@ -2134,7 +2044,6 @@ static int dw_mci_init_slot(struct dw_mci *host, unsigned int id) const struct dw_mci_drv_data *drv_data = host->drv_data; int ctrl_id, ret; u32 freq[2]; - u8 bus_width; mmc = mmc_alloc_host(sizeof(struct dw_mci_slot), host->dev); if (!mmc) @@ -2158,17 +2067,7 @@ static int dw_mci_init_slot(struct dw_mci *host, unsigned int id) mmc->f_max = freq[1]; } - if (host->pdata->get_ocr) - mmc->ocr_avail = host->pdata->get_ocr(id); - else - mmc->ocr_avail = MMC_VDD_32_33 | MMC_VDD_33_34; - - /* - * Start with slot power disabled, it will be enabled when a card - * is detected. - */ - if (host->pdata->setpower) - host->pdata->setpower(id, 0); + mmc->ocr_avail = MMC_VDD_32_33 | MMC_VDD_33_34; if (host->pdata->caps) mmc->caps = host->pdata->caps; @@ -2189,19 +2088,7 @@ static int dw_mci_init_slot(struct dw_mci *host, unsigned int id) if (host->pdata->caps2) mmc->caps2 = host->pdata->caps2; - if (host->pdata->get_bus_wd) - bus_width = host->pdata->get_bus_wd(slot->id); - else if (host->dev->of_node) - bus_width = dw_mci_of_get_bus_wd(host->dev, slot->id); - else - bus_width = 1; - - switch (bus_width) { - case 8: - mmc->caps |= MMC_CAP_8_BIT_DATA; - case 4: - mmc->caps |= MMC_CAP_4_BIT_DATA; - } + mmc_of_parse(mmc); if (host->pdata->blk_settings) { mmc->max_segs = host->pdata->blk_settings->max_segs; @@ -2226,8 +2113,10 @@ static int dw_mci_init_slot(struct dw_mci *host, unsigned int id) #endif /* CONFIG_MMC_DW_IDMAC */ } - slot->wp_gpio = dw_mci_of_get_wp_gpio(host->dev, slot->id); - dw_mci_of_get_cd_gpio(host->dev, slot->id, mmc); + if (dw_mci_get_cd(mmc)) + set_bit(DW_MMC_CARD_PRESENT, &slot->flags); + else + clear_bit(DW_MMC_CARD_PRESENT, &slot->flags); ret = mmc_add_host(mmc); if (ret) @@ -2249,10 +2138,6 @@ err_setup_bus: static void dw_mci_cleanup_slot(struct dw_mci_slot *slot, unsigned int id) { - /* Shutdown detect IRQ */ - if (slot->host->pdata->exit) - slot->host->pdata->exit(id); - /* Debugfs stuff is cleaned up by mmc core */ mmc_remove_host(slot->mmc); slot->host->slot[id] = NULL; @@ -2399,24 +2284,9 @@ static struct dw_mci_board *dw_mci_parse_dt(struct dw_mci *host) return ERR_PTR(ret); } - if (of_find_property(np, "keep-power-in-suspend", NULL)) - pdata->pm_caps |= MMC_PM_KEEP_POWER; - - if (of_find_property(np, "enable-sdio-wakeup", NULL)) - pdata->pm_caps |= MMC_PM_WAKE_SDIO_IRQ; - if (of_find_property(np, "supports-highspeed", NULL)) pdata->caps |= MMC_CAP_SD_HIGHSPEED | MMC_CAP_MMC_HIGHSPEED; - if (of_find_property(np, "caps2-mmc-hs200-1_8v", NULL)) - pdata->caps2 |= MMC_CAP2_HS200_1_8V_SDR; - - if (of_find_property(np, "caps2-mmc-hs200-1_2v", NULL)) - pdata->caps2 |= MMC_CAP2_HS200_1_2V_SDR; - - if (of_get_property(np, "cd-inverted", NULL)) - pdata->caps2 |= MMC_CAP2_CD_ACTIVE_HIGH; - return pdata; } @@ -2442,9 +2312,9 @@ int dw_mci_probe(struct dw_mci *host) } } - if (!host->pdata->select_slot && host->pdata->num_slots > 1) { + if (host->pdata->num_slots > 1) { dev_err(host->dev, - "Platform data must supply select_slot function\n"); + "Platform data must supply num_slots.\n"); return -ENODEV; } @@ -2474,12 +2344,19 @@ int dw_mci_probe(struct dw_mci *host) ret = clk_set_rate(host->ciu_clk, host->pdata->bus_hz); if (ret) dev_warn(host->dev, - "Unable to set bus rate to %ul\n", + "Unable to set bus rate to %uHz\n", host->pdata->bus_hz); } host->bus_hz = clk_get_rate(host->ciu_clk); } + if (!host->bus_hz) { + dev_err(host->dev, + "Platform data must supply bus speed\n"); + ret = -ENODEV; + goto err_clk_ciu; + } + if (drv_data && drv_data->init) { ret = drv_data->init(host); if (ret) { @@ -2516,13 +2393,6 @@ int dw_mci_probe(struct dw_mci *host) } } - if (!host->bus_hz) { - dev_err(host->dev, - "Platform data must supply bus speed\n"); - ret = -ENODEV; - goto err_regulator; - } - host->quirks = host->pdata->quirks; spin_lock_init(&host->lock); @@ -2666,8 +2536,6 @@ err_workqueue: err_dmaunmap: if (host->use_dma && host->dma_ops->exit) host->dma_ops->exit(host); - -err_regulator: if (host->vmmc) regulator_disable(host->vmmc); diff --git a/drivers/mmc/host/dw_mmc.h b/drivers/mmc/host/dw_mmc.h index 68349779c396..738fa241d058 100644 --- a/drivers/mmc/host/dw_mmc.h +++ b/drivers/mmc/host/dw_mmc.h @@ -195,7 +195,6 @@ extern int dw_mci_resume(struct dw_mci *host); * @mmc: The mmc_host representing this slot. * @host: The MMC controller this slot is using. * @quirks: Slot-level quirks (DW_MCI_SLOT_QUIRK_XXX) - * @wp_gpio: If gpio_is_valid() we'll use this to read write protect. * @ctype: Card type for this slot. * @mrq: mmc_request currently being processed or waiting to be * processed, or NULL when the slot is idle. @@ -214,7 +213,6 @@ struct dw_mci_slot { struct dw_mci *host; int quirks; - int wp_gpio; u32 ctype; diff --git a/drivers/mmc/host/jz4740_mmc.c b/drivers/mmc/host/jz4740_mmc.c index de2139cf3444..537d6c7a5ae4 100644 --- a/drivers/mmc/host/jz4740_mmc.c +++ b/drivers/mmc/host/jz4740_mmc.c @@ -515,10 +515,13 @@ static irqreturn_t jz_mmc_irq_worker(int irq, void *devid) jz4740_mmc_send_command(host, req->stop); - timeout = jz4740_mmc_poll_irq(host, JZ_MMC_IRQ_PRG_DONE); - if (timeout) { - host->state = JZ4740_MMC_STATE_DONE; - break; + if (mmc_resp_type(req->stop) & MMC_RSP_BUSY) { + timeout = jz4740_mmc_poll_irq(host, + JZ_MMC_IRQ_PRG_DONE); + if (timeout) { + host->state = JZ4740_MMC_STATE_DONE; + break; + } } case JZ4740_MMC_STATE_DONE: break; diff --git a/drivers/mmc/host/mmci.c b/drivers/mmc/host/mmci.c index a084edd37af5..7ad463e9741c 100644 --- a/drivers/mmc/host/mmci.c +++ b/drivers/mmc/host/mmci.c @@ -301,7 +301,8 @@ static void mmci_set_clkreg(struct mmci_host *host, unsigned int desired) if (host->mmc->ios.bus_width == MMC_BUS_WIDTH_8) clk |= MCI_ST_8BIT_BUS; - if (host->mmc->ios.timing == MMC_TIMING_UHS_DDR50) + if (host->mmc->ios.timing == MMC_TIMING_UHS_DDR50 || + host->mmc->ios.timing == MMC_TIMING_MMC_DDR52) clk |= MCI_ST_UX500_NEG_EDGE; mmci_write_clkreg(host, clk); @@ -764,7 +765,8 @@ static void mmci_start_data(struct mmci_host *host, struct mmc_data *data) mmci_write_clkreg(host, clk); } - if (host->mmc->ios.timing == MMC_TIMING_UHS_DDR50) + if (host->mmc->ios.timing == MMC_TIMING_UHS_DDR50 || + host->mmc->ios.timing == MMC_TIMING_MMC_DDR52) datactrl |= MCI_ST_DPSM_DDRMODE; /* diff --git a/drivers/mmc/host/moxart-mmc.c b/drivers/mmc/host/moxart-mmc.c new file mode 100644 index 000000000000..74924a04026e --- /dev/null +++ b/drivers/mmc/host/moxart-mmc.c @@ -0,0 +1,730 @@ +/* + * MOXA ART MMC host driver. + * + * Copyright (C) 2014 Jonas Jensen + * + * Jonas Jensen <jonas.jensen@gmail.com> + * + * Based on code from + * Moxa Technologies Co., Ltd. <www.moxa.com> + * + * This file is licensed under the terms of the GNU General Public + * License version 2. This program is licensed "as is" without any + * warranty of any kind, whether express or implied. + */ + +#include <linux/version.h> +#include <linux/module.h> +#include <linux/init.h> +#include <linux/platform_device.h> +#include <linux/delay.h> +#include <linux/interrupt.h> +#include <linux/blkdev.h> +#include <linux/dma-mapping.h> +#include <linux/dmaengine.h> +#include <linux/mmc/host.h> +#include <linux/mmc/sd.h> +#include <linux/sched.h> +#include <linux/io.h> +#include <linux/of_address.h> +#include <linux/of_irq.h> +#include <linux/clk.h> +#include <linux/bitops.h> +#include <linux/of_dma.h> +#include <linux/spinlock.h> + +#define REG_COMMAND 0 +#define REG_ARGUMENT 4 +#define REG_RESPONSE0 8 +#define REG_RESPONSE1 12 +#define REG_RESPONSE2 16 +#define REG_RESPONSE3 20 +#define REG_RESPONSE_COMMAND 24 +#define REG_DATA_CONTROL 28 +#define REG_DATA_TIMER 32 +#define REG_DATA_LENGTH 36 +#define REG_STATUS 40 +#define REG_CLEAR 44 +#define REG_INTERRUPT_MASK 48 +#define REG_POWER_CONTROL 52 +#define REG_CLOCK_CONTROL 56 +#define REG_BUS_WIDTH 60 +#define REG_DATA_WINDOW 64 +#define REG_FEATURE 68 +#define REG_REVISION 72 + +/* REG_COMMAND */ +#define CMD_SDC_RESET BIT(10) +#define CMD_EN BIT(9) +#define CMD_APP_CMD BIT(8) +#define CMD_LONG_RSP BIT(7) +#define CMD_NEED_RSP BIT(6) +#define CMD_IDX_MASK 0x3f + +/* REG_RESPONSE_COMMAND */ +#define RSP_CMD_APP BIT(6) +#define RSP_CMD_IDX_MASK 0x3f + +/* REG_DATA_CONTROL */ +#define DCR_DATA_FIFO_RESET BIT(8) +#define DCR_DATA_THRES BIT(7) +#define DCR_DATA_EN BIT(6) +#define DCR_DMA_EN BIT(5) +#define DCR_DATA_WRITE BIT(4) +#define DCR_BLK_SIZE 0x0f + +/* REG_DATA_LENGTH */ +#define DATA_LEN_MASK 0xffffff + +/* REG_STATUS */ +#define WRITE_PROT BIT(12) +#define CARD_DETECT BIT(11) +/* 1-10 below can be sent to either registers, interrupt or clear. */ +#define CARD_CHANGE BIT(10) +#define FIFO_ORUN BIT(9) +#define FIFO_URUN BIT(8) +#define DATA_END BIT(7) +#define CMD_SENT BIT(6) +#define DATA_CRC_OK BIT(5) +#define RSP_CRC_OK BIT(4) +#define DATA_TIMEOUT BIT(3) +#define RSP_TIMEOUT BIT(2) +#define DATA_CRC_FAIL BIT(1) +#define RSP_CRC_FAIL BIT(0) + +#define MASK_RSP (RSP_TIMEOUT | RSP_CRC_FAIL | \ + RSP_CRC_OK | CARD_DETECT | CMD_SENT) + +#define MASK_DATA (DATA_CRC_OK | DATA_END | \ + DATA_CRC_FAIL | DATA_TIMEOUT) + +#define MASK_INTR_PIO (FIFO_URUN | FIFO_ORUN | CARD_CHANGE) + +/* REG_POWER_CONTROL */ +#define SD_POWER_ON BIT(4) +#define SD_POWER_MASK 0x0f + +/* REG_CLOCK_CONTROL */ +#define CLK_HISPD BIT(9) +#define CLK_OFF BIT(8) +#define CLK_SD BIT(7) +#define CLK_DIV_MASK 0x7f + +/* REG_BUS_WIDTH */ +#define BUS_WIDTH_8 BIT(2) +#define BUS_WIDTH_4 BIT(1) +#define BUS_WIDTH_1 BIT(0) + +#define MMC_VDD_360 23 +#define MIN_POWER (MMC_VDD_360 - SD_POWER_MASK) +#define MAX_RETRIES 500000 + +struct moxart_host { + spinlock_t lock; + + void __iomem *base; + + phys_addr_t reg_phys; + + struct dma_chan *dma_chan_tx; + struct dma_chan *dma_chan_rx; + struct dma_async_tx_descriptor *tx_desc; + struct mmc_host *mmc; + struct mmc_request *mrq; + struct scatterlist *cur_sg; + struct completion dma_complete; + struct completion pio_complete; + + u32 num_sg; + u32 data_remain; + u32 data_len; + u32 fifo_width; + u32 timeout; + u32 rate; + + long sysclk; + + bool have_dma; + bool is_removed; +}; + +static inline void moxart_init_sg(struct moxart_host *host, + struct mmc_data *data) +{ + host->cur_sg = data->sg; + host->num_sg = data->sg_len; + host->data_remain = host->cur_sg->length; + + if (host->data_remain > host->data_len) + host->data_remain = host->data_len; +} + +static inline int moxart_next_sg(struct moxart_host *host) +{ + int remain; + struct mmc_data *data = host->mrq->cmd->data; + + host->cur_sg++; + host->num_sg--; + + if (host->num_sg > 0) { + host->data_remain = host->cur_sg->length; + remain = host->data_len - data->bytes_xfered; + if (remain > 0 && remain < host->data_remain) + host->data_remain = remain; + } + + return host->num_sg; +} + +static int moxart_wait_for_status(struct moxart_host *host, + u32 mask, u32 *status) +{ + int ret = -ETIMEDOUT; + u32 i; + + for (i = 0; i < MAX_RETRIES; i++) { + *status = readl(host->base + REG_STATUS); + if (!(*status & mask)) { + udelay(5); + continue; + } + writel(*status & mask, host->base + REG_CLEAR); + ret = 0; + break; + } + + if (ret) + dev_err(mmc_dev(host->mmc), "timed out waiting for status\n"); + + return ret; +} + + +static void moxart_send_command(struct moxart_host *host, + struct mmc_command *cmd) +{ + u32 status, cmdctrl; + + writel(RSP_TIMEOUT | RSP_CRC_OK | + RSP_CRC_FAIL | CMD_SENT, host->base + REG_CLEAR); + writel(cmd->arg, host->base + REG_ARGUMENT); + + cmdctrl = cmd->opcode & CMD_IDX_MASK; + if (cmdctrl == SD_APP_SET_BUS_WIDTH || cmdctrl == SD_APP_OP_COND || + cmdctrl == SD_APP_SEND_SCR || cmdctrl == SD_APP_SD_STATUS || + cmdctrl == SD_APP_SEND_NUM_WR_BLKS) + cmdctrl |= CMD_APP_CMD; + + if (cmd->flags & MMC_RSP_PRESENT) + cmdctrl |= CMD_NEED_RSP; + + if (cmd->flags & MMC_RSP_136) + cmdctrl |= CMD_LONG_RSP; + + writel(cmdctrl | CMD_EN, host->base + REG_COMMAND); + + if (moxart_wait_for_status(host, MASK_RSP, &status) == -ETIMEDOUT) + cmd->error = -ETIMEDOUT; + + if (status & RSP_TIMEOUT) { + cmd->error = -ETIMEDOUT; + return; + } + if (status & RSP_CRC_FAIL) { + cmd->error = -EIO; + return; + } + if (status & RSP_CRC_OK) { + if (cmd->flags & MMC_RSP_136) { + cmd->resp[3] = readl(host->base + REG_RESPONSE0); + cmd->resp[2] = readl(host->base + REG_RESPONSE1); + cmd->resp[1] = readl(host->base + REG_RESPONSE2); + cmd->resp[0] = readl(host->base + REG_RESPONSE3); + } else { + cmd->resp[0] = readl(host->base + REG_RESPONSE0); + } + } +} + +static void moxart_dma_complete(void *param) +{ + struct moxart_host *host = param; + + complete(&host->dma_complete); +} + +static void moxart_transfer_dma(struct mmc_data *data, struct moxart_host *host) +{ + u32 len, dir_data, dir_slave; + unsigned long dma_time; + struct dma_async_tx_descriptor *desc = NULL; + struct dma_chan *dma_chan; + + if (host->data_len == data->bytes_xfered) + return; + + if (data->flags & MMC_DATA_WRITE) { + dma_chan = host->dma_chan_tx; + dir_data = DMA_TO_DEVICE; + dir_slave = DMA_MEM_TO_DEV; + } else { + dma_chan = host->dma_chan_rx; + dir_data = DMA_FROM_DEVICE; + dir_slave = DMA_DEV_TO_MEM; + } + + len = dma_map_sg(dma_chan->device->dev, data->sg, + data->sg_len, dir_data); + + if (len > 0) { + desc = dmaengine_prep_slave_sg(dma_chan, data->sg, + len, dir_slave, + DMA_PREP_INTERRUPT | + DMA_CTRL_ACK); + } else { + dev_err(mmc_dev(host->mmc), "dma_map_sg returned zero length\n"); + } + + if (desc) { + host->tx_desc = desc; + desc->callback = moxart_dma_complete; + desc->callback_param = host; + dmaengine_submit(desc); + dma_async_issue_pending(dma_chan); + } + + data->bytes_xfered += host->data_remain; + + dma_time = wait_for_completion_interruptible_timeout( + &host->dma_complete, host->timeout); + + dma_unmap_sg(dma_chan->device->dev, + data->sg, data->sg_len, + dir_data); +} + + +static void moxart_transfer_pio(struct moxart_host *host) +{ + struct mmc_data *data = host->mrq->cmd->data; + u32 *sgp, len = 0, remain, status; + + if (host->data_len == data->bytes_xfered) + return; + + sgp = sg_virt(host->cur_sg); + remain = host->data_remain; + + if (data->flags & MMC_DATA_WRITE) { + while (remain > 0) { + if (moxart_wait_for_status(host, FIFO_URUN, &status) + == -ETIMEDOUT) { + data->error = -ETIMEDOUT; + complete(&host->pio_complete); + return; + } + for (len = 0; len < remain && len < host->fifo_width;) { + iowrite32(*sgp, host->base + REG_DATA_WINDOW); + sgp++; + len += 4; + } + remain -= len; + } + + } else { + while (remain > 0) { + if (moxart_wait_for_status(host, FIFO_ORUN, &status) + == -ETIMEDOUT) { + data->error = -ETIMEDOUT; + complete(&host->pio_complete); + return; + } + for (len = 0; len < remain && len < host->fifo_width;) { + /* SCR data must be read in big endian. */ + if (data->mrq->cmd->opcode == SD_APP_SEND_SCR) + *sgp = ioread32be(host->base + + REG_DATA_WINDOW); + else + *sgp = ioread32(host->base + + REG_DATA_WINDOW); + sgp++; + len += 4; + } + remain -= len; + } + } + + data->bytes_xfered += host->data_remain - remain; + host->data_remain = remain; + + if (host->data_len != data->bytes_xfered) + moxart_next_sg(host); + else + complete(&host->pio_complete); +} + +static void moxart_prepare_data(struct moxart_host *host) +{ + struct mmc_data *data = host->mrq->cmd->data; + u32 datactrl; + int blksz_bits; + + if (!data) + return; + + host->data_len = data->blocks * data->blksz; + blksz_bits = ffs(data->blksz) - 1; + BUG_ON(1 << blksz_bits != data->blksz); + + moxart_init_sg(host, data); + + datactrl = DCR_DATA_EN | (blksz_bits & DCR_BLK_SIZE); + + if (data->flags & MMC_DATA_WRITE) + datactrl |= DCR_DATA_WRITE; + + if ((host->data_len > host->fifo_width) && host->have_dma) + datactrl |= DCR_DMA_EN; + + writel(DCR_DATA_FIFO_RESET, host->base + REG_DATA_CONTROL); + writel(MASK_DATA | FIFO_URUN | FIFO_ORUN, host->base + REG_CLEAR); + writel(host->rate, host->base + REG_DATA_TIMER); + writel(host->data_len, host->base + REG_DATA_LENGTH); + writel(datactrl, host->base + REG_DATA_CONTROL); +} + +static void moxart_request(struct mmc_host *mmc, struct mmc_request *mrq) +{ + struct moxart_host *host = mmc_priv(mmc); + unsigned long pio_time, flags; + u32 status; + + spin_lock_irqsave(&host->lock, flags); + + init_completion(&host->dma_complete); + init_completion(&host->pio_complete); + + host->mrq = mrq; + + if (readl(host->base + REG_STATUS) & CARD_DETECT) { + mrq->cmd->error = -ETIMEDOUT; + goto request_done; + } + + moxart_prepare_data(host); + moxart_send_command(host, host->mrq->cmd); + + if (mrq->cmd->data) { + if ((host->data_len > host->fifo_width) && host->have_dma) { + + writel(CARD_CHANGE, host->base + REG_INTERRUPT_MASK); + + spin_unlock_irqrestore(&host->lock, flags); + + moxart_transfer_dma(mrq->cmd->data, host); + + spin_lock_irqsave(&host->lock, flags); + } else { + + writel(MASK_INTR_PIO, host->base + REG_INTERRUPT_MASK); + + spin_unlock_irqrestore(&host->lock, flags); + + /* PIO transfers start from interrupt. */ + pio_time = wait_for_completion_interruptible_timeout( + &host->pio_complete, host->timeout); + + spin_lock_irqsave(&host->lock, flags); + } + + if (host->is_removed) { + dev_err(mmc_dev(host->mmc), "card removed\n"); + mrq->cmd->error = -ETIMEDOUT; + goto request_done; + } + + if (moxart_wait_for_status(host, MASK_DATA, &status) + == -ETIMEDOUT) { + mrq->cmd->data->error = -ETIMEDOUT; + goto request_done; + } + + if (status & DATA_CRC_FAIL) + mrq->cmd->data->error = -ETIMEDOUT; + + if (mrq->cmd->data->stop) + moxart_send_command(host, mrq->cmd->data->stop); + } + +request_done: + spin_unlock_irqrestore(&host->lock, flags); + mmc_request_done(host->mmc, mrq); +} + +static irqreturn_t moxart_irq(int irq, void *devid) +{ + struct moxart_host *host = (struct moxart_host *)devid; + u32 status; + unsigned long flags; + + spin_lock_irqsave(&host->lock, flags); + + status = readl(host->base + REG_STATUS); + if (status & CARD_CHANGE) { + host->is_removed = status & CARD_DETECT; + if (host->is_removed && host->have_dma) { + dmaengine_terminate_all(host->dma_chan_tx); + dmaengine_terminate_all(host->dma_chan_rx); + } + host->mrq = NULL; + writel(MASK_INTR_PIO, host->base + REG_CLEAR); + writel(CARD_CHANGE, host->base + REG_INTERRUPT_MASK); + mmc_detect_change(host->mmc, 0); + } + if (status & (FIFO_ORUN | FIFO_URUN) && host->mrq) + moxart_transfer_pio(host); + + spin_unlock_irqrestore(&host->lock, flags); + + return IRQ_HANDLED; +} + +static void moxart_set_ios(struct mmc_host *mmc, struct mmc_ios *ios) +{ + struct moxart_host *host = mmc_priv(mmc); + unsigned long flags; + u8 power, div; + u32 ctrl; + + spin_lock_irqsave(&host->lock, flags); + + if (ios->clock) { + for (div = 0; div < CLK_DIV_MASK; ++div) { + if (ios->clock >= host->sysclk / (2 * (div + 1))) + break; + } + ctrl = CLK_SD | div; + host->rate = host->sysclk / (2 * (div + 1)); + if (host->rate > host->sysclk) + ctrl |= CLK_HISPD; + writel(ctrl, host->base + REG_CLOCK_CONTROL); + } + + if (ios->power_mode == MMC_POWER_OFF) { + writel(readl(host->base + REG_POWER_CONTROL) & ~SD_POWER_ON, + host->base + REG_POWER_CONTROL); + } else { + if (ios->vdd < MIN_POWER) + power = 0; + else + power = ios->vdd - MIN_POWER; + + writel(SD_POWER_ON | (u32) power, + host->base + REG_POWER_CONTROL); + } + + switch (ios->bus_width) { + case MMC_BUS_WIDTH_4: + writel(BUS_WIDTH_4, host->base + REG_BUS_WIDTH); + break; + case MMC_BUS_WIDTH_8: + writel(BUS_WIDTH_8, host->base + REG_BUS_WIDTH); + break; + default: + writel(BUS_WIDTH_1, host->base + REG_BUS_WIDTH); + break; + } + + spin_unlock_irqrestore(&host->lock, flags); +} + + +static int moxart_get_ro(struct mmc_host *mmc) +{ + struct moxart_host *host = mmc_priv(mmc); + + return !!(readl(host->base + REG_STATUS) & WRITE_PROT); +} + +static struct mmc_host_ops moxart_ops = { + .request = moxart_request, + .set_ios = moxart_set_ios, + .get_ro = moxart_get_ro, +}; + +static int moxart_probe(struct platform_device *pdev) +{ + struct device *dev = &pdev->dev; + struct device_node *node = dev->of_node; + struct resource res_mmc; + struct mmc_host *mmc; + struct moxart_host *host = NULL; + struct dma_slave_config cfg; + struct clk *clk; + void __iomem *reg_mmc; + dma_cap_mask_t mask; + int irq, ret; + u32 i; + + mmc = mmc_alloc_host(sizeof(struct moxart_host), dev); + if (!mmc) { + dev_err(dev, "mmc_alloc_host failed\n"); + ret = -ENOMEM; + goto out; + } + + ret = of_address_to_resource(node, 0, &res_mmc); + if (ret) { + dev_err(dev, "of_address_to_resource failed\n"); + goto out; + } + + irq = irq_of_parse_and_map(node, 0); + if (irq <= 0) { + dev_err(dev, "irq_of_parse_and_map failed\n"); + ret = -EINVAL; + goto out; + } + + clk = of_clk_get(node, 0); + if (IS_ERR(clk)) { + dev_err(dev, "of_clk_get failed\n"); + ret = PTR_ERR(clk); + goto out; + } + + reg_mmc = devm_ioremap_resource(dev, &res_mmc); + if (IS_ERR(reg_mmc)) { + ret = PTR_ERR(reg_mmc); + goto out; + } + + mmc_of_parse(mmc); + + dma_cap_zero(mask); + dma_cap_set(DMA_SLAVE, mask); + + host = mmc_priv(mmc); + host->mmc = mmc; + host->base = reg_mmc; + host->reg_phys = res_mmc.start; + host->timeout = msecs_to_jiffies(1000); + host->sysclk = clk_get_rate(clk); + host->fifo_width = readl(host->base + REG_FEATURE) << 2; + host->dma_chan_tx = of_dma_request_slave_channel(node, "tx"); + host->dma_chan_rx = of_dma_request_slave_channel(node, "rx"); + + spin_lock_init(&host->lock); + + mmc->ops = &moxart_ops; + mmc->f_max = DIV_ROUND_CLOSEST(host->sysclk, 2); + mmc->f_min = DIV_ROUND_CLOSEST(host->sysclk, CLK_DIV_MASK * 2); + mmc->ocr_avail = 0xffff00; /* Support 2.0v - 3.6v power. */ + + if (IS_ERR(host->dma_chan_tx) || IS_ERR(host->dma_chan_rx)) { + dev_dbg(dev, "PIO mode transfer enabled\n"); + host->have_dma = false; + } else { + dev_dbg(dev, "DMA channels found (%p,%p)\n", + host->dma_chan_tx, host->dma_chan_rx); + host->have_dma = true; + + cfg.src_addr_width = DMA_SLAVE_BUSWIDTH_4_BYTES; + cfg.dst_addr_width = DMA_SLAVE_BUSWIDTH_4_BYTES; + + cfg.direction = DMA_MEM_TO_DEV; + cfg.src_addr = 0; + cfg.dst_addr = host->reg_phys + REG_DATA_WINDOW; + dmaengine_slave_config(host->dma_chan_tx, &cfg); + + cfg.direction = DMA_DEV_TO_MEM; + cfg.src_addr = host->reg_phys + REG_DATA_WINDOW; + cfg.dst_addr = 0; + dmaengine_slave_config(host->dma_chan_rx, &cfg); + } + + switch ((readl(host->base + REG_BUS_WIDTH) >> 3) & 3) { + case 1: + mmc->caps |= MMC_CAP_4_BIT_DATA; + break; + case 2: + mmc->caps |= MMC_CAP_4_BIT_DATA | MMC_CAP_8_BIT_DATA; + break; + default: + break; + } + + writel(0, host->base + REG_INTERRUPT_MASK); + + writel(CMD_SDC_RESET, host->base + REG_COMMAND); + for (i = 0; i < MAX_RETRIES; i++) { + if (!(readl(host->base + REG_COMMAND) & CMD_SDC_RESET)) + break; + udelay(5); + } + + ret = devm_request_irq(dev, irq, moxart_irq, 0, "moxart-mmc", host); + if (ret) + goto out; + + dev_set_drvdata(dev, mmc); + mmc_add_host(mmc); + + dev_dbg(dev, "IRQ=%d, FIFO is %d bytes\n", irq, host->fifo_width); + + return 0; + +out: + if (mmc) + mmc_free_host(mmc); + return ret; +} + +static int moxart_remove(struct platform_device *pdev) +{ + struct mmc_host *mmc = dev_get_drvdata(&pdev->dev); + struct moxart_host *host = mmc_priv(mmc); + + dev_set_drvdata(&pdev->dev, NULL); + + if (mmc) { + if (!IS_ERR(host->dma_chan_tx)) + dma_release_channel(host->dma_chan_tx); + if (!IS_ERR(host->dma_chan_rx)) + dma_release_channel(host->dma_chan_rx); + mmc_remove_host(mmc); + mmc_free_host(mmc); + + writel(0, host->base + REG_INTERRUPT_MASK); + writel(0, host->base + REG_POWER_CONTROL); + writel(readl(host->base + REG_CLOCK_CONTROL) | CLK_OFF, + host->base + REG_CLOCK_CONTROL); + } + + kfree(host); + + return 0; +} + +static const struct of_device_id moxart_mmc_match[] = { + { .compatible = "moxa,moxart-mmc" }, + { .compatible = "faraday,ftsdc010" }, + { } +}; + +static struct platform_driver moxart_mmc_driver = { + .probe = moxart_probe, + .remove = moxart_remove, + .driver = { + .name = "mmc-moxart", + .owner = THIS_MODULE, + .of_match_table = moxart_mmc_match, + }, +}; +module_platform_driver(moxart_mmc_driver); + +MODULE_ALIAS("platform:mmc-moxart"); +MODULE_DESCRIPTION("MOXA ART MMC driver"); +MODULE_LICENSE("GPL v2"); +MODULE_AUTHOR("Jonas Jensen <jonas.jensen@gmail.com>"); diff --git a/drivers/mmc/host/mvsdio.c b/drivers/mmc/host/mvsdio.c index 45aa2206741d..9377284f8544 100644 --- a/drivers/mmc/host/mvsdio.c +++ b/drivers/mmc/host/mvsdio.c @@ -354,6 +354,20 @@ static irqreturn_t mvsd_irq(int irq, void *dev) intr_status, mvsd_read(MVSD_NOR_INTR_EN), mvsd_read(MVSD_HW_STATE)); + /* + * It looks like, SDIO IP can issue one late, spurious irq + * although all irqs should be disabled. To work around this, + * bail out early, if we didn't expect any irqs to occur. + */ + if (!mvsd_read(MVSD_NOR_INTR_EN) && !mvsd_read(MVSD_ERR_INTR_EN)) { + dev_dbg(host->dev, "spurious irq detected intr 0x%04x intr_en 0x%04x erri 0x%04x erri_en 0x%04x\n", + mvsd_read(MVSD_NOR_INTR_STATUS), + mvsd_read(MVSD_NOR_INTR_EN), + mvsd_read(MVSD_ERR_INTR_STATUS), + mvsd_read(MVSD_ERR_INTR_EN)); + return IRQ_HANDLED; + } + spin_lock(&host->lock); /* PIO handling, if needed. Messy business... */ @@ -801,10 +815,10 @@ static int mvsd_probe(struct platform_device *pdev) goto out; if (!(mmc->caps & MMC_CAP_NEEDS_POLL)) - dev_notice(&pdev->dev, "using GPIO for card detection\n"); + dev_dbg(&pdev->dev, "using GPIO for card detection\n"); else - dev_notice(&pdev->dev, - "lacking card detect (fall back to polling)\n"); + dev_dbg(&pdev->dev, "lacking card detect (fall back to polling)\n"); + return 0; out: diff --git a/drivers/mmc/host/mxcmmc.c b/drivers/mmc/host/mxcmmc.c index f7199c83f5cf..ed1cb93c3784 100644 --- a/drivers/mmc/host/mxcmmc.c +++ b/drivers/mmc/host/mxcmmc.c @@ -124,9 +124,8 @@ enum mxcmci_type { struct mxcmci_host { struct mmc_host *mmc; - struct resource *res; void __iomem *base; - int irq; + dma_addr_t phys_base; int detect_irq; struct dma_chan *dma; struct dma_async_tx_descriptor *desc; @@ -154,8 +153,6 @@ struct mxcmci_host { struct work_struct datawork; spinlock_t lock; - struct regulator *vcc; - int burstlen; int dmareq; struct dma_slave_config dma_slave_config; @@ -241,37 +238,15 @@ static inline void mxcmci_writew(struct mxcmci_host *host, u16 val, int reg) static void mxcmci_set_clk_rate(struct mxcmci_host *host, unsigned int clk_ios); -static inline void mxcmci_init_ocr(struct mxcmci_host *host) -{ - host->vcc = regulator_get(mmc_dev(host->mmc), "vmmc"); - - if (IS_ERR(host->vcc)) { - host->vcc = NULL; - } else { - host->mmc->ocr_avail = mmc_regulator_get_ocrmask(host->vcc); - if (host->pdata && host->pdata->ocr_avail) - dev_warn(mmc_dev(host->mmc), - "pdata->ocr_avail will not be used\n"); - } - - if (host->vcc == NULL) { - /* fall-back to platform data */ - if (host->pdata && host->pdata->ocr_avail) - host->mmc->ocr_avail = host->pdata->ocr_avail; - else - host->mmc->ocr_avail = MMC_VDD_32_33 | MMC_VDD_33_34; - } -} - -static inline void mxcmci_set_power(struct mxcmci_host *host, - unsigned char power_mode, - unsigned int vdd) +static void mxcmci_set_power(struct mxcmci_host *host, unsigned int vdd) { - if (host->vcc) { - if (power_mode == MMC_POWER_UP) - mmc_regulator_set_ocr(host->mmc, host->vcc, vdd); - else if (power_mode == MMC_POWER_OFF) - mmc_regulator_set_ocr(host->mmc, host->vcc, 0); + if (!IS_ERR(host->mmc->supply.vmmc)) { + if (host->power_mode == MMC_POWER_UP) + mmc_regulator_set_ocr(host->mmc, + host->mmc->supply.vmmc, vdd); + else if (host->power_mode == MMC_POWER_OFF) + mmc_regulator_set_ocr(host->mmc, + host->mmc->supply.vmmc, 0); } if (host->pdata && host->pdata->setpower) @@ -299,7 +274,6 @@ static void mxcmci_softreset(struct mxcmci_host *host) mxcmci_writew(host, 0xff, MMC_REG_RES_TO); } -static int mxcmci_setup_dma(struct mmc_host *mmc); #if IS_ENABLED(CONFIG_PPC_MPC512x) static inline void buffer_swap32(u32 *buf, int len) @@ -868,8 +842,8 @@ static int mxcmci_setup_dma(struct mmc_host *mmc) struct mxcmci_host *host = mmc_priv(mmc); struct dma_slave_config *config = &host->dma_slave_config; - config->dst_addr = host->res->start + MMC_REG_BUFFER_ACCESS; - config->src_addr = host->res->start + MMC_REG_BUFFER_ACCESS; + config->dst_addr = host->phys_base + MMC_REG_BUFFER_ACCESS; + config->src_addr = host->phys_base + MMC_REG_BUFFER_ACCESS; config->dst_addr_width = 4; config->src_addr_width = 4; config->dst_maxburst = host->burstlen; @@ -911,8 +885,8 @@ static void mxcmci_set_ios(struct mmc_host *mmc, struct mmc_ios *ios) host->cmdat &= ~CMD_DAT_CONT_BUS_WIDTH_4; if (host->power_mode != ios->power_mode) { - mxcmci_set_power(host, ios->power_mode, ios->vdd); host->power_mode = ios->power_mode; + mxcmci_set_power(host, ios->vdd); if (ios->power_mode == MMC_POWER_ON) host->cmdat |= CMD_DAT_CONT_INIT; @@ -1040,8 +1014,8 @@ static const struct mmc_host_ops mxcmci_ops = { static int mxcmci_probe(struct platform_device *pdev) { struct mmc_host *mmc; - struct mxcmci_host *host = NULL; - struct resource *iores, *r; + struct mxcmci_host *host; + struct resource *res; int ret = 0, irq; bool dat3_card_detect = false; dma_cap_mask_t mask; @@ -1052,21 +1026,25 @@ static int mxcmci_probe(struct platform_device *pdev) of_id = of_match_device(mxcmci_of_match, &pdev->dev); - iores = platform_get_resource(pdev, IORESOURCE_MEM, 0); + res = platform_get_resource(pdev, IORESOURCE_MEM, 0); irq = platform_get_irq(pdev, 0); - if (!iores || irq < 0) + if (irq < 0) return -EINVAL; - r = request_mem_region(iores->start, resource_size(iores), pdev->name); - if (!r) - return -EBUSY; + mmc = mmc_alloc_host(sizeof(*host), &pdev->dev); + if (!mmc) + return -ENOMEM; + + host = mmc_priv(mmc); - mmc = mmc_alloc_host(sizeof(struct mxcmci_host), &pdev->dev); - if (!mmc) { - ret = -ENOMEM; - goto out_release_mem; + host->base = devm_ioremap_resource(&pdev->dev, res); + if (IS_ERR(host->base)) { + ret = PTR_ERR(host->base); + goto out_free; } + host->phys_base = res->start; + ret = mmc_of_parse(mmc); if (ret) goto out_free; @@ -1084,13 +1062,6 @@ static int mxcmci_probe(struct platform_device *pdev) mmc->max_req_size = mmc->max_blk_size * mmc->max_blk_count; mmc->max_seg_size = mmc->max_req_size; - host = mmc_priv(mmc); - host->base = ioremap(r->start, resource_size(r)); - if (!host->base) { - ret = -ENOMEM; - goto out_free; - } - if (of_id) { const struct platform_device_id *id_entry = of_id->data; host->devtype = id_entry->driver_data; @@ -1112,7 +1083,14 @@ static int mxcmci_probe(struct platform_device *pdev) && !of_property_read_bool(pdev->dev.of_node, "cd-gpios")) dat3_card_detect = true; - mxcmci_init_ocr(host); + ret = mmc_regulator_get_supply(mmc); + if (ret) { + if (pdata && ret != -EPROBE_DEFER) + mmc->ocr_avail = pdata->ocr_avail ? : + MMC_VDD_32_33 | MMC_VDD_33_34; + else + goto out_free; + } if (dat3_card_detect) host->default_irq_mask = @@ -1120,19 +1098,16 @@ static int mxcmci_probe(struct platform_device *pdev) else host->default_irq_mask = 0; - host->res = r; - host->irq = irq; - host->clk_ipg = devm_clk_get(&pdev->dev, "ipg"); if (IS_ERR(host->clk_ipg)) { ret = PTR_ERR(host->clk_ipg); - goto out_iounmap; + goto out_free; } host->clk_per = devm_clk_get(&pdev->dev, "per"); if (IS_ERR(host->clk_per)) { ret = PTR_ERR(host->clk_per); - goto out_iounmap; + goto out_free; } clk_prepare_enable(host->clk_per); @@ -1159,9 +1134,9 @@ static int mxcmci_probe(struct platform_device *pdev) if (!host->pdata) { host->dma = dma_request_slave_channel(&pdev->dev, "rx-tx"); } else { - r = platform_get_resource(pdev, IORESOURCE_DMA, 0); - if (r) { - host->dmareq = r->start; + res = platform_get_resource(pdev, IORESOURCE_DMA, 0); + if (res) { + host->dmareq = res->start; host->dma_data.peripheral_type = IMX_DMATYPE_SDHC; host->dma_data.priority = DMA_PRIO_LOW; host->dma_data.dma_request = host->dmareq; @@ -1178,7 +1153,8 @@ static int mxcmci_probe(struct platform_device *pdev) INIT_WORK(&host->datawork, mxcmci_datawork); - ret = request_irq(host->irq, mxcmci_irq, 0, DRIVER_NAME, host); + ret = devm_request_irq(&pdev->dev, irq, mxcmci_irq, 0, + dev_name(&pdev->dev), host); if (ret) goto out_free_dma; @@ -1188,7 +1164,7 @@ static int mxcmci_probe(struct platform_device *pdev) ret = host->pdata->init(&pdev->dev, mxcmci_detect_irq, host->mmc); if (ret) - goto out_free_irq; + goto out_free_dma; } init_timer(&host->watchdog); @@ -1199,20 +1175,17 @@ static int mxcmci_probe(struct platform_device *pdev) return 0; -out_free_irq: - free_irq(host->irq, host); out_free_dma: if (host->dma) dma_release_channel(host->dma); + out_clk_put: clk_disable_unprepare(host->clk_per); clk_disable_unprepare(host->clk_ipg); -out_iounmap: - iounmap(host->base); + out_free: mmc_free_host(mmc); -out_release_mem: - release_mem_region(iores->start, resource_size(iores)); + return ret; } @@ -1223,30 +1196,21 @@ static int mxcmci_remove(struct platform_device *pdev) mmc_remove_host(mmc); - if (host->vcc) - regulator_put(host->vcc); - if (host->pdata && host->pdata->exit) host->pdata->exit(&pdev->dev, mmc); - free_irq(host->irq, host); - iounmap(host->base); - if (host->dma) dma_release_channel(host->dma); clk_disable_unprepare(host->clk_per); clk_disable_unprepare(host->clk_ipg); - release_mem_region(host->res->start, resource_size(host->res)); - mmc_free_host(mmc); return 0; } -#ifdef CONFIG_PM -static int mxcmci_suspend(struct device *dev) +static int __maybe_unused mxcmci_suspend(struct device *dev) { struct mmc_host *mmc = dev_get_drvdata(dev); struct mxcmci_host *host = mmc_priv(mmc); @@ -1256,7 +1220,7 @@ static int mxcmci_suspend(struct device *dev) return 0; } -static int mxcmci_resume(struct device *dev) +static int __maybe_unused mxcmci_resume(struct device *dev) { struct mmc_host *mmc = dev_get_drvdata(dev); struct mxcmci_host *host = mmc_priv(mmc); @@ -1266,11 +1230,7 @@ static int mxcmci_resume(struct device *dev) return 0; } -static const struct dev_pm_ops mxcmci_pm_ops = { - .suspend = mxcmci_suspend, - .resume = mxcmci_resume, -}; -#endif +static SIMPLE_DEV_PM_OPS(mxcmci_pm_ops, mxcmci_suspend, mxcmci_resume); static struct platform_driver mxcmci_driver = { .probe = mxcmci_probe, @@ -1279,9 +1239,7 @@ static struct platform_driver mxcmci_driver = { .driver = { .name = DRIVER_NAME, .owner = THIS_MODULE, -#ifdef CONFIG_PM .pm = &mxcmci_pm_ops, -#endif .of_match_table = mxcmci_of_match, } }; diff --git a/drivers/mmc/host/mxs-mmc.c b/drivers/mmc/host/mxs-mmc.c index 073e871a0fc8..babfea03ba8a 100644 --- a/drivers/mmc/host/mxs-mmc.c +++ b/drivers/mmc/host/mxs-mmc.c @@ -70,6 +70,7 @@ struct mxs_mmc_host { unsigned char bus_width; spinlock_t lock; int sdio_irq_en; + bool broken_cd; }; static int mxs_mmc_get_cd(struct mmc_host *mmc) @@ -78,6 +79,9 @@ static int mxs_mmc_get_cd(struct mmc_host *mmc) struct mxs_ssp *ssp = &host->ssp; int present, ret; + if (host->broken_cd) + return -ENOSYS; + ret = mmc_gpio_get_cd(mmc); if (ret >= 0) return ret; @@ -568,6 +572,7 @@ static int mxs_mmc_probe(struct platform_device *pdev) { const struct of_device_id *of_id = of_match_device(mxs_mmc_dt_ids, &pdev->dev); + struct device_node *np = pdev->dev.of_node; struct mxs_mmc_host *host; struct mmc_host *mmc; struct resource *iores; @@ -634,6 +639,8 @@ static int mxs_mmc_probe(struct platform_device *pdev) mmc->caps = MMC_CAP_SD_HIGHSPEED | MMC_CAP_MMC_HIGHSPEED | MMC_CAP_SDIO_IRQ | MMC_CAP_NEEDS_POLL; + host->broken_cd = of_property_read_bool(np, "broken-cd"); + mmc->f_min = 400000; mmc->f_max = 288000000; diff --git a/drivers/mmc/host/omap.c b/drivers/mmc/host/omap.c index 5c2e58b29305..81974ecdfcbc 100644 --- a/drivers/mmc/host/omap.c +++ b/drivers/mmc/host/omap.c @@ -177,7 +177,7 @@ static void mmc_omap_fclk_offdelay(struct mmc_omap_slot *slot) unsigned long tick_ns; if (slot != NULL && slot->host->fclk_enabled && slot->fclk_freq > 0) { - tick_ns = (1000000000 + slot->fclk_freq - 1) / slot->fclk_freq; + tick_ns = DIV_ROUND_UP(NSEC_PER_SEC, slot->fclk_freq); ndelay(8 * tick_ns); } } @@ -435,7 +435,7 @@ static void mmc_omap_send_stop_work(struct work_struct *work) struct mmc_data *data = host->stop_data; unsigned long tick_ns; - tick_ns = (1000000000 + slot->fclk_freq - 1)/slot->fclk_freq; + tick_ns = DIV_ROUND_UP(NSEC_PER_SEC, slot->fclk_freq); ndelay(8*tick_ns); mmc_omap_start_command(host, data->stop); @@ -477,7 +477,7 @@ mmc_omap_send_abort(struct mmc_omap_host *host, int maxloops) u16 stat = 0; /* Sending abort takes 80 clocks. Have some extra and round up */ - timeout = (120*1000000 + slot->fclk_freq - 1)/slot->fclk_freq; + timeout = DIV_ROUND_UP(120 * USEC_PER_SEC, slot->fclk_freq); restarts = 0; while (restarts < maxloops) { OMAP_MMC_WRITE(host, STAT, 0xFFFF); @@ -677,8 +677,8 @@ mmc_omap_xfer_data(struct mmc_omap_host *host, int write) if (n > host->buffer_bytes_left) n = host->buffer_bytes_left; - nwords = n / 2; - nwords += n & 1; /* handle odd number of bytes to transfer */ + /* Round up to handle odd number of bytes to transfer */ + nwords = DIV_ROUND_UP(n, 2); host->buffer_bytes_left -= n; host->total_bytes_left -= n; diff --git a/drivers/mmc/host/omap_hsmmc.c b/drivers/mmc/host/omap_hsmmc.c index e91ee21549d0..6b7b75585926 100644 --- a/drivers/mmc/host/omap_hsmmc.c +++ b/drivers/mmc/host/omap_hsmmc.c @@ -31,7 +31,7 @@ #include <linux/of.h> #include <linux/of_gpio.h> #include <linux/of_device.h> -#include <linux/omap-dma.h> +#include <linux/omap-dmaengine.h> #include <linux/mmc/host.h> #include <linux/mmc/core.h> #include <linux/mmc/mmc.h> @@ -582,7 +582,7 @@ static void omap_hsmmc_set_clock(struct omap_hsmmc_host *host) * - MMC/SD clock coming out of controller > 25MHz */ if ((mmc_slot(host).features & HSMMC_HAS_HSPE_SUPPORT) && - (ios->timing != MMC_TIMING_UHS_DDR50) && + (ios->timing != MMC_TIMING_MMC_DDR52) && ((OMAP_HSMMC_READ(host->base, CAPA) & HSS) == HSS)) { regval = OMAP_HSMMC_READ(host->base, HCTL); if (clkdiv && (clk_get_rate(host->fclk)/clkdiv) > 25000000) @@ -602,7 +602,7 @@ static void omap_hsmmc_set_bus_width(struct omap_hsmmc_host *host) u32 con; con = OMAP_HSMMC_READ(host->base, CON); - if (ios->timing == MMC_TIMING_UHS_DDR50) + if (ios->timing == MMC_TIMING_MMC_DDR52) con |= DDR; /* configure in DDR mode */ else con &= ~DDR; @@ -920,16 +920,17 @@ omap_hsmmc_xfer_done(struct omap_hsmmc_host *host, struct mmc_data *data) static void omap_hsmmc_cmd_done(struct omap_hsmmc_host *host, struct mmc_command *cmd) { - host->cmd = NULL; - if (host->mrq->sbc && (host->cmd == host->mrq->sbc) && !host->mrq->sbc->error && !(host->flags & AUTO_CMD23)) { + host->cmd = NULL; omap_hsmmc_start_dma_transfer(host); omap_hsmmc_start_command(host, host->mrq->cmd, host->mrq->data); return; } + host->cmd = NULL; + if (cmd->flags & MMC_RSP_PRESENT) { if (cmd->flags & MMC_RSP_136) { /* response type 2 */ @@ -1851,6 +1852,7 @@ static int omap_hsmmc_probe(struct platform_device *pdev) unsigned tx_req, rx_req; struct pinctrl *pinctrl; const struct omap_mmc_of_data *data; + void __iomem *base; match = of_match_device(of_match_ptr(omap_mmc_of_match), &pdev->dev); if (match) { @@ -1881,9 +1883,9 @@ static int omap_hsmmc_probe(struct platform_device *pdev) if (res == NULL || irq < 0) return -ENXIO; - res = request_mem_region(res->start, resource_size(res), pdev->name); - if (res == NULL) - return -EBUSY; + base = devm_ioremap_resource(&pdev->dev, res); + if (IS_ERR(base)) + return PTR_ERR(base); ret = omap_hsmmc_gpio_init(pdata); if (ret) @@ -1904,7 +1906,7 @@ static int omap_hsmmc_probe(struct platform_device *pdev) host->irq = irq; host->slot_id = 0; host->mapbase = res->start + pdata->reg_offset; - host->base = ioremap(host->mapbase, SZ_4K); + host->base = base + pdata->reg_offset; host->power_mode = MMC_POWER_OFF; host->next_data.cookie = 1; host->pbias_enabled = 0; @@ -1922,7 +1924,7 @@ static int omap_hsmmc_probe(struct platform_device *pdev) spin_lock_init(&host->irq_lock); - host->fclk = clk_get(&pdev->dev, "fck"); + host->fclk = devm_clk_get(&pdev->dev, "fck"); if (IS_ERR(host->fclk)) { ret = PTR_ERR(host->fclk); host->fclk = NULL; @@ -1941,7 +1943,7 @@ static int omap_hsmmc_probe(struct platform_device *pdev) omap_hsmmc_context_save(host); - host->dbclk = clk_get(&pdev->dev, "mmchsdb_fck"); + host->dbclk = devm_clk_get(&pdev->dev, "mmchsdb_fck"); /* * MMC can still work without debounce clock. */ @@ -1949,7 +1951,6 @@ static int omap_hsmmc_probe(struct platform_device *pdev) host->dbclk = NULL; } else if (clk_prepare_enable(host->dbclk) != 0) { dev_warn(mmc_dev(host->mmc), "Failed to enable debounce clk\n"); - clk_put(host->dbclk); host->dbclk = NULL; } @@ -2018,7 +2019,7 @@ static int omap_hsmmc_probe(struct platform_device *pdev) } /* Request IRQ for MMC operations */ - ret = request_irq(host->irq, omap_hsmmc_irq, 0, + ret = devm_request_irq(&pdev->dev, host->irq, omap_hsmmc_irq, 0, mmc_hostname(mmc), host); if (ret) { dev_err(mmc_dev(host->mmc), "Unable to grab HSMMC IRQ\n"); @@ -2029,7 +2030,7 @@ static int omap_hsmmc_probe(struct platform_device *pdev) if (pdata->init(&pdev->dev) != 0) { dev_err(mmc_dev(host->mmc), "Unable to configure MMC IRQs\n"); - goto err_irq_cd_init; + goto err_irq; } } @@ -2044,9 +2045,9 @@ static int omap_hsmmc_probe(struct platform_device *pdev) /* Request IRQ for card detect */ if ((mmc_slot(host).card_detect_irq)) { - ret = request_threaded_irq(mmc_slot(host).card_detect_irq, - NULL, - omap_hsmmc_detect, + ret = devm_request_threaded_irq(&pdev->dev, + mmc_slot(host).card_detect_irq, + NULL, omap_hsmmc_detect, IRQF_TRIGGER_RISING | IRQF_TRIGGER_FALLING | IRQF_ONESHOT, mmc_hostname(mmc), host); if (ret) { @@ -2089,15 +2090,12 @@ static int omap_hsmmc_probe(struct platform_device *pdev) err_slot_name: mmc_remove_host(mmc); - free_irq(mmc_slot(host).card_detect_irq, host); err_irq_cd: if (host->use_reg) omap_hsmmc_reg_put(host); err_reg: if (host->pdata->cleanup) host->pdata->cleanup(&pdev->dev); -err_irq_cd_init: - free_irq(host->irq, host); err_irq: if (host->tx_chan) dma_release_channel(host->tx_chan); @@ -2105,27 +2103,19 @@ err_irq: dma_release_channel(host->rx_chan); pm_runtime_put_sync(host->dev); pm_runtime_disable(host->dev); - clk_put(host->fclk); - if (host->dbclk) { + if (host->dbclk) clk_disable_unprepare(host->dbclk); - clk_put(host->dbclk); - } err1: - iounmap(host->base); mmc_free_host(mmc); err_alloc: omap_hsmmc_gpio_free(pdata); err: - res = platform_get_resource(pdev, IORESOURCE_MEM, 0); - if (res) - release_mem_region(res->start, resource_size(res)); return ret; } static int omap_hsmmc_remove(struct platform_device *pdev) { struct omap_hsmmc_host *host = platform_get_drvdata(pdev); - struct resource *res; pm_runtime_get_sync(host->dev); mmc_remove_host(host->mmc); @@ -2133,9 +2123,6 @@ static int omap_hsmmc_remove(struct platform_device *pdev) omap_hsmmc_reg_put(host); if (host->pdata->cleanup) host->pdata->cleanup(&pdev->dev); - free_irq(host->irq, host); - if (mmc_slot(host).card_detect_irq) - free_irq(mmc_slot(host).card_detect_irq, host); if (host->tx_chan) dma_release_channel(host->tx_chan); @@ -2144,20 +2131,12 @@ static int omap_hsmmc_remove(struct platform_device *pdev) pm_runtime_put_sync(host->dev); pm_runtime_disable(host->dev); - clk_put(host->fclk); - if (host->dbclk) { + if (host->dbclk) clk_disable_unprepare(host->dbclk); - clk_put(host->dbclk); - } omap_hsmmc_gpio_free(host->pdata); - iounmap(host->base); mmc_free_host(host->mmc); - res = platform_get_resource(pdev, IORESOURCE_MEM, 0); - if (res) - release_mem_region(res->start, resource_size(res)); - return 0; } diff --git a/drivers/mmc/host/rtsx_pci_sdmmc.c b/drivers/mmc/host/rtsx_pci_sdmmc.c index 0b9ded13a3ae..0d519649b575 100644 --- a/drivers/mmc/host/rtsx_pci_sdmmc.c +++ b/drivers/mmc/host/rtsx_pci_sdmmc.c @@ -236,6 +236,9 @@ static void sd_send_cmd_get_rsp(struct realtek_pci_sdmmc *host, case MMC_RSP_R1: rsp_type = SD_RSP_TYPE_R1; break; + case MMC_RSP_R1 & ~MMC_RSP_CRC: + rsp_type = SD_RSP_TYPE_R1 | SD_NO_CHECK_CRC7; + break; case MMC_RSP_R1B: rsp_type = SD_RSP_TYPE_R1b; break; @@ -816,6 +819,7 @@ static int sd_set_timing(struct realtek_pci_sdmmc *host, unsigned char timing) rtsx_pci_add_cmd(pcr, WRITE_REG_CMD, CLK_CTL, CLK_LOW_FREQ, 0); break; + case MMC_TIMING_MMC_DDR52: case MMC_TIMING_UHS_DDR50: rtsx_pci_add_cmd(pcr, WRITE_REG_CMD, SD_CFG1, 0x0C | SD_ASYNC_FIFO_NOT_RST, @@ -896,6 +900,7 @@ static void sdmmc_set_ios(struct mmc_host *mmc, struct mmc_ios *ios) host->vpclk = true; host->double_clk = false; break; + case MMC_TIMING_MMC_DDR52: case MMC_TIMING_UHS_DDR50: case MMC_TIMING_UHS_SDR25: host->ssc_depth = RTSX_SSC_DEPTH_1M; diff --git a/drivers/mmc/host/rtsx_usb_sdmmc.c b/drivers/mmc/host/rtsx_usb_sdmmc.c index e11fafa6fc6b..5d3766e792f0 100644 --- a/drivers/mmc/host/rtsx_usb_sdmmc.c +++ b/drivers/mmc/host/rtsx_usb_sdmmc.c @@ -34,7 +34,8 @@ #include <linux/mfd/rtsx_usb.h> #include <asm/unaligned.h> -#if defined(CONFIG_LEDS_CLASS) || defined(CONFIG_LEDS_CLASS_MODULE) +#if defined(CONFIG_LEDS_CLASS) || (defined(CONFIG_LEDS_CLASS_MODULE) && \ + defined(CONFIG_MMC_REALTEK_USB_MODULE)) #include <linux/leds.h> #include <linux/workqueue.h> #define RTSX_USB_USE_LEDS_CLASS @@ -59,7 +60,7 @@ struct rtsx_usb_sdmmc { unsigned char power_mode; -#if defined(CONFIG_LEDS_CLASS) || defined(CONFIG_LEDS_CLASS_MODULE) +#ifdef RTSX_USB_USE_LEDS_CLASS struct led_classdev led; char led_name[32]; struct work_struct led_work; diff --git a/drivers/mmc/host/sdhci-acpi.c b/drivers/mmc/host/sdhci-acpi.c index ebb3f392b589..8ce3c28cb76e 100644 --- a/drivers/mmc/host/sdhci-acpi.c +++ b/drivers/mmc/host/sdhci-acpi.c @@ -102,11 +102,19 @@ static void sdhci_acpi_int_hw_reset(struct sdhci_host *host) } static const struct sdhci_ops sdhci_acpi_ops_dflt = { + .set_clock = sdhci_set_clock, .enable_dma = sdhci_acpi_enable_dma, + .set_bus_width = sdhci_set_bus_width, + .reset = sdhci_reset, + .set_uhs_signaling = sdhci_set_uhs_signaling, }; static const struct sdhci_ops sdhci_acpi_ops_int = { + .set_clock = sdhci_set_clock, .enable_dma = sdhci_acpi_enable_dma, + .set_bus_width = sdhci_set_bus_width, + .reset = sdhci_reset, + .set_uhs_signaling = sdhci_set_uhs_signaling, .hw_reset = sdhci_acpi_int_hw_reset, }; diff --git a/drivers/mmc/host/sdhci-bcm-kona.c b/drivers/mmc/host/sdhci-bcm-kona.c index 6f166e63b817..dd780c315a63 100644 --- a/drivers/mmc/host/sdhci-bcm-kona.c +++ b/drivers/mmc/host/sdhci-bcm-kona.c @@ -206,9 +206,13 @@ static void sdhci_bcm_kona_init_74_clocks(struct sdhci_host *host, } static struct sdhci_ops sdhci_bcm_kona_ops = { + .set_clock = sdhci_set_clock, .get_max_clock = sdhci_bcm_kona_get_max_clk, .get_timeout_clock = sdhci_bcm_kona_get_timeout_clock, .platform_send_init_74_clocks = sdhci_bcm_kona_init_74_clocks, + .set_bus_width = sdhci_set_bus_width, + .reset = sdhci_reset, + .set_uhs_signaling = sdhci_set_uhs_signaling, .card_event = sdhci_bcm_kona_card_event, }; diff --git a/drivers/mmc/host/sdhci-bcm2835.c b/drivers/mmc/host/sdhci-bcm2835.c index f6d8d67c545f..46af9a439d7b 100644 --- a/drivers/mmc/host/sdhci-bcm2835.c +++ b/drivers/mmc/host/sdhci-bcm2835.c @@ -131,8 +131,12 @@ static const struct sdhci_ops bcm2835_sdhci_ops = { .read_l = bcm2835_sdhci_readl, .read_w = bcm2835_sdhci_readw, .read_b = bcm2835_sdhci_readb, + .set_clock = sdhci_set_clock, .get_max_clock = sdhci_pltfm_clk_get_max_clock, .get_min_clock = bcm2835_sdhci_get_min_clock, + .set_bus_width = sdhci_set_bus_width, + .reset = sdhci_reset, + .set_uhs_signaling = sdhci_set_uhs_signaling, }; static const struct sdhci_pltfm_data bcm2835_sdhci_pdata = { diff --git a/drivers/mmc/host/sdhci-cns3xxx.c b/drivers/mmc/host/sdhci-cns3xxx.c index f2cc26633cb2..14b74075589a 100644 --- a/drivers/mmc/host/sdhci-cns3xxx.c +++ b/drivers/mmc/host/sdhci-cns3xxx.c @@ -30,13 +30,12 @@ static void sdhci_cns3xxx_set_clock(struct sdhci_host *host, unsigned int clock) u16 clk; unsigned long timeout; - if (clock == host->clock) - return; + host->mmc->actual_clock = 0; sdhci_writew(host, 0, SDHCI_CLOCK_CONTROL); if (clock == 0) - goto out; + return; while (host->max_clk / div > clock) { /* @@ -75,13 +74,14 @@ static void sdhci_cns3xxx_set_clock(struct sdhci_host *host, unsigned int clock) clk |= SDHCI_CLOCK_CARD_EN; sdhci_writew(host, clk, SDHCI_CLOCK_CONTROL); -out: - host->clock = clock; } static const struct sdhci_ops sdhci_cns3xxx_ops = { .get_max_clock = sdhci_cns3xxx_get_max_clk, .set_clock = sdhci_cns3xxx_set_clock, + .set_bus_width = sdhci_set_bus_width, + .reset = sdhci_reset, + .set_uhs_signaling = sdhci_set_uhs_signaling, }; static const struct sdhci_pltfm_data sdhci_cns3xxx_pdata = { @@ -90,8 +90,7 @@ static const struct sdhci_pltfm_data sdhci_cns3xxx_pdata = { SDHCI_QUIRK_DATA_TIMEOUT_USES_SDCLK | SDHCI_QUIRK_INVERTED_WRITE_PROTECT | SDHCI_QUIRK_CAP_CLOCK_BASE_BROKEN | - SDHCI_QUIRK_BROKEN_TIMEOUT_VAL | - SDHCI_QUIRK_NONSTANDARD_CLOCK, + SDHCI_QUIRK_BROKEN_TIMEOUT_VAL, }; static int sdhci_cns3xxx_probe(struct platform_device *pdev) diff --git a/drivers/mmc/host/sdhci-dove.c b/drivers/mmc/host/sdhci-dove.c index 736d7a2eb7ec..e6278ec007d7 100644 --- a/drivers/mmc/host/sdhci-dove.c +++ b/drivers/mmc/host/sdhci-dove.c @@ -21,28 +21,17 @@ #include <linux/clk.h> #include <linux/err.h> -#include <linux/gpio.h> #include <linux/io.h> #include <linux/mmc/host.h> #include <linux/module.h> #include <linux/of.h> -#include <linux/of_gpio.h> #include "sdhci-pltfm.h" struct sdhci_dove_priv { struct clk *clk; - int gpio_cd; }; -static irqreturn_t sdhci_dove_carddetect_irq(int irq, void *data) -{ - struct sdhci_host *host = data; - - tasklet_schedule(&host->card_tasklet); - return IRQ_HANDLED; -} - static u16 sdhci_dove_readw(struct sdhci_host *host, int reg) { u16 ret; @@ -60,8 +49,6 @@ static u16 sdhci_dove_readw(struct sdhci_host *host, int reg) static u32 sdhci_dove_readl(struct sdhci_host *host, int reg) { - struct sdhci_pltfm_host *pltfm_host = sdhci_priv(host); - struct sdhci_dove_priv *priv = pltfm_host->priv; u32 ret; ret = readl(host->ioaddr + reg); @@ -71,14 +58,6 @@ static u32 sdhci_dove_readl(struct sdhci_host *host, int reg) /* Mask the support for 3.0V */ ret &= ~SDHCI_CAN_VDD_300; break; - case SDHCI_PRESENT_STATE: - if (gpio_is_valid(priv->gpio_cd)) { - if (gpio_get_value(priv->gpio_cd) == 0) - ret |= SDHCI_CARD_PRESENT; - else - ret &= ~SDHCI_CARD_PRESENT; - } - break; } return ret; } @@ -86,6 +65,10 @@ static u32 sdhci_dove_readl(struct sdhci_host *host, int reg) static const struct sdhci_ops sdhci_dove_ops = { .read_w = sdhci_dove_readw, .read_l = sdhci_dove_readl, + .set_clock = sdhci_set_clock, + .set_bus_width = sdhci_set_bus_width, + .reset = sdhci_reset, + .set_uhs_signaling = sdhci_set_uhs_signaling, }; static const struct sdhci_pltfm_data sdhci_dove_pdata = { @@ -113,28 +96,9 @@ static int sdhci_dove_probe(struct platform_device *pdev) priv->clk = devm_clk_get(&pdev->dev, NULL); - if (pdev->dev.of_node) { - priv->gpio_cd = of_get_named_gpio(pdev->dev.of_node, - "cd-gpios", 0); - } else { - priv->gpio_cd = -EINVAL; - } - - if (gpio_is_valid(priv->gpio_cd)) { - ret = gpio_request(priv->gpio_cd, "sdhci-cd"); - if (ret) { - dev_err(&pdev->dev, "card detect gpio request failed: %d\n", - ret); - return ret; - } - gpio_direction_input(priv->gpio_cd); - } - host = sdhci_pltfm_init(pdev, &sdhci_dove_pdata, 0); - if (IS_ERR(host)) { - ret = PTR_ERR(host); - goto err_sdhci_pltfm_init; - } + if (IS_ERR(host)) + return PTR_ERR(host); pltfm_host = sdhci_priv(host); pltfm_host->priv = priv; @@ -142,39 +106,20 @@ static int sdhci_dove_probe(struct platform_device *pdev) if (!IS_ERR(priv->clk)) clk_prepare_enable(priv->clk); - sdhci_get_of_property(pdev); + ret = mmc_of_parse(host->mmc); + if (ret) + goto err_sdhci_add; ret = sdhci_add_host(host); if (ret) goto err_sdhci_add; - /* - * We must request the IRQ after sdhci_add_host(), as the tasklet only - * gets setup in sdhci_add_host() and we oops. - */ - if (gpio_is_valid(priv->gpio_cd)) { - ret = request_irq(gpio_to_irq(priv->gpio_cd), - sdhci_dove_carddetect_irq, - IRQF_TRIGGER_FALLING | IRQF_TRIGGER_RISING, - mmc_hostname(host->mmc), host); - if (ret) { - dev_err(&pdev->dev, "card detect irq request failed: %d\n", - ret); - goto err_request_irq; - } - } - return 0; -err_request_irq: - sdhci_remove_host(host, 0); err_sdhci_add: if (!IS_ERR(priv->clk)) clk_disable_unprepare(priv->clk); sdhci_pltfm_free(pdev); -err_sdhci_pltfm_init: - if (gpio_is_valid(priv->gpio_cd)) - gpio_free(priv->gpio_cd); return ret; } @@ -186,11 +131,6 @@ static int sdhci_dove_remove(struct platform_device *pdev) sdhci_pltfm_unregister(pdev); - if (gpio_is_valid(priv->gpio_cd)) { - free_irq(gpio_to_irq(priv->gpio_cd), host); - gpio_free(priv->gpio_cd); - } - if (!IS_ERR(priv->clk)) clk_disable_unprepare(priv->clk); diff --git a/drivers/mmc/host/sdhci-esdhc-imx.c b/drivers/mmc/host/sdhci-esdhc-imx.c index b841bb7cd371..ccec0e32590f 100644 --- a/drivers/mmc/host/sdhci-esdhc-imx.c +++ b/drivers/mmc/host/sdhci-esdhc-imx.c @@ -160,7 +160,6 @@ struct pltfm_imx_data { MULTIBLK_IN_PROCESS, /* exact multiblock cmd in process */ WAIT_FOR_INT, /* sent CMD12, waiting for response INT */ } multiblock_status; - u32 uhs_mode; u32 is_ddr; }; @@ -382,7 +381,6 @@ static u16 esdhc_readw_le(struct sdhci_host *host, int reg) if (val & ESDHC_MIX_CTRL_SMPCLK_SEL) ret |= SDHCI_CTRL_TUNED_CLK; - ret |= (imx_data->uhs_mode & SDHCI_CTRL_UHS_MASK); ret &= ~SDHCI_CTRL_PRESET_VAL_ENABLE; return ret; @@ -429,7 +427,6 @@ static void esdhc_writew_le(struct sdhci_host *host, u16 val, int reg) else new_val &= ~ESDHC_VENDOR_SPEC_VSELECT; writel(new_val, host->ioaddr + ESDHC_VENDOR_SPEC); - imx_data->uhs_mode = val & SDHCI_CTRL_UHS_MASK; if (imx_data->socdata->flags & ESDHC_FLAG_MAN_TUNING) { new_val = readl(host->ioaddr + ESDHC_MIX_CTRL); if (val & SDHCI_CTRL_TUNED_CLK) @@ -600,12 +597,14 @@ static inline void esdhc_pltfm_set_clock(struct sdhci_host *host, u32 temp, val; if (clock == 0) { + host->mmc->actual_clock = 0; + if (esdhc_is_usdhc(imx_data)) { val = readl(host->ioaddr + ESDHC_VENDOR_SPEC); writel(val & ~ESDHC_VENDOR_SPEC_FRC_SDCLK_ON, host->ioaddr + ESDHC_VENDOR_SPEC); } - goto out; + return; } if (esdhc_is_usdhc(imx_data) && !imx_data->is_ddr) @@ -645,8 +644,6 @@ static inline void esdhc_pltfm_set_clock(struct sdhci_host *host, } mdelay(1); -out: - host->clock = clock; } static unsigned int esdhc_pltfm_get_ro(struct sdhci_host *host) @@ -668,7 +665,7 @@ static unsigned int esdhc_pltfm_get_ro(struct sdhci_host *host) return -ENOSYS; } -static int esdhc_pltfm_bus_width(struct sdhci_host *host, int width) +static void esdhc_pltfm_set_bus_width(struct sdhci_host *host, int width) { u32 ctrl; @@ -686,8 +683,6 @@ static int esdhc_pltfm_bus_width(struct sdhci_host *host, int width) esdhc_clrset_le(host, ESDHC_CTRL_BUSWIDTH_MASK, ctrl, SDHCI_HOST_CONTROL); - - return 0; } static void esdhc_prepare_tuning(struct sdhci_host *host, u32 val) @@ -697,6 +692,7 @@ static void esdhc_prepare_tuning(struct sdhci_host *host, u32 val) /* FIXME: delay a bit for card to be ready for next tuning due to errors */ mdelay(1); + /* This is balanced by the runtime put in sdhci_tasklet_finish */ pm_runtime_get_sync(host->mmc->parent); reg = readl(host->ioaddr + ESDHC_MIX_CTRL); reg |= ESDHC_MIX_CTRL_EXE_TUNE | ESDHC_MIX_CTRL_SMPCLK_SEL | @@ -713,13 +709,12 @@ static void esdhc_request_done(struct mmc_request *mrq) complete(&mrq->completion); } -static int esdhc_send_tuning_cmd(struct sdhci_host *host, u32 opcode) +static int esdhc_send_tuning_cmd(struct sdhci_host *host, u32 opcode, + struct scatterlist *sg) { struct mmc_command cmd = {0}; struct mmc_request mrq = {NULL}; struct mmc_data data = {0}; - struct scatterlist sg; - char tuning_pattern[ESDHC_TUNING_BLOCK_PATTERN_LEN]; cmd.opcode = opcode; cmd.arg = 0; @@ -728,11 +723,9 @@ static int esdhc_send_tuning_cmd(struct sdhci_host *host, u32 opcode) data.blksz = ESDHC_TUNING_BLOCK_PATTERN_LEN; data.blocks = 1; data.flags = MMC_DATA_READ; - data.sg = &sg; + data.sg = sg; data.sg_len = 1; - sg_init_one(&sg, tuning_pattern, sizeof(tuning_pattern)); - mrq.cmd = &cmd; mrq.cmd->mrq = &mrq; mrq.data = &data; @@ -742,14 +735,12 @@ static int esdhc_send_tuning_cmd(struct sdhci_host *host, u32 opcode) mrq.done = esdhc_request_done; init_completion(&(mrq.completion)); - disable_irq(host->irq); - spin_lock(&host->lock); + spin_lock_irq(&host->lock); host->mrq = &mrq; sdhci_send_command(host, mrq.cmd); - spin_unlock(&host->lock); - enable_irq(host->irq); + spin_unlock_irq(&host->lock); wait_for_completion(&mrq.completion); @@ -772,13 +763,21 @@ static void esdhc_post_tuning(struct sdhci_host *host) static int esdhc_executing_tuning(struct sdhci_host *host, u32 opcode) { + struct scatterlist sg; + char *tuning_pattern; int min, max, avg, ret; + tuning_pattern = kmalloc(ESDHC_TUNING_BLOCK_PATTERN_LEN, GFP_KERNEL); + if (!tuning_pattern) + return -ENOMEM; + + sg_init_one(&sg, tuning_pattern, ESDHC_TUNING_BLOCK_PATTERN_LEN); + /* find the mininum delay first which can pass tuning */ min = ESDHC_TUNE_CTRL_MIN; while (min < ESDHC_TUNE_CTRL_MAX) { esdhc_prepare_tuning(host, min); - if (!esdhc_send_tuning_cmd(host, opcode)) + if (!esdhc_send_tuning_cmd(host, opcode, &sg)) break; min += ESDHC_TUNE_CTRL_STEP; } @@ -787,7 +786,7 @@ static int esdhc_executing_tuning(struct sdhci_host *host, u32 opcode) max = min + ESDHC_TUNE_CTRL_STEP; while (max < ESDHC_TUNE_CTRL_MAX) { esdhc_prepare_tuning(host, max); - if (esdhc_send_tuning_cmd(host, opcode)) { + if (esdhc_send_tuning_cmd(host, opcode, &sg)) { max -= ESDHC_TUNE_CTRL_STEP; break; } @@ -797,9 +796,11 @@ static int esdhc_executing_tuning(struct sdhci_host *host, u32 opcode) /* use average delay to get the best timing */ avg = (min + max) / 2; esdhc_prepare_tuning(host, avg); - ret = esdhc_send_tuning_cmd(host, opcode); + ret = esdhc_send_tuning_cmd(host, opcode, &sg); esdhc_post_tuning(host); + kfree(tuning_pattern); + dev_dbg(mmc_dev(host->mmc), "tunning %s at 0x%x ret %d\n", ret ? "failed" : "passed", avg, ret); @@ -837,28 +838,21 @@ static int esdhc_change_pinstate(struct sdhci_host *host, return pinctrl_select_state(imx_data->pinctrl, pinctrl); } -static int esdhc_set_uhs_signaling(struct sdhci_host *host, unsigned int uhs) +static void esdhc_set_uhs_signaling(struct sdhci_host *host, unsigned timing) { struct sdhci_pltfm_host *pltfm_host = sdhci_priv(host); struct pltfm_imx_data *imx_data = pltfm_host->priv; struct esdhc_platform_data *boarddata = &imx_data->boarddata; - switch (uhs) { + switch (timing) { case MMC_TIMING_UHS_SDR12: - imx_data->uhs_mode = SDHCI_CTRL_UHS_SDR12; - break; case MMC_TIMING_UHS_SDR25: - imx_data->uhs_mode = SDHCI_CTRL_UHS_SDR25; - break; case MMC_TIMING_UHS_SDR50: - imx_data->uhs_mode = SDHCI_CTRL_UHS_SDR50; - break; case MMC_TIMING_UHS_SDR104: case MMC_TIMING_MMC_HS200: - imx_data->uhs_mode = SDHCI_CTRL_UHS_SDR104; break; case MMC_TIMING_UHS_DDR50: - imx_data->uhs_mode = SDHCI_CTRL_UHS_DDR50; + case MMC_TIMING_MMC_DDR52: writel(readl(host->ioaddr + ESDHC_MIX_CTRL) | ESDHC_MIX_CTRL_DDREN, host->ioaddr + ESDHC_MIX_CTRL); @@ -875,7 +869,15 @@ static int esdhc_set_uhs_signaling(struct sdhci_host *host, unsigned int uhs) break; } - return esdhc_change_pinstate(host, uhs); + esdhc_change_pinstate(host, timing); +} + +static void esdhc_reset(struct sdhci_host *host, u8 mask) +{ + sdhci_reset(host, mask); + + sdhci_writel(host, host->ier, SDHCI_INT_ENABLE); + sdhci_writel(host, host->ier, SDHCI_SIGNAL_ENABLE); } static struct sdhci_ops sdhci_esdhc_ops = { @@ -888,8 +890,9 @@ static struct sdhci_ops sdhci_esdhc_ops = { .get_max_clock = esdhc_pltfm_get_max_clock, .get_min_clock = esdhc_pltfm_get_min_clock, .get_ro = esdhc_pltfm_get_ro, - .platform_bus_width = esdhc_pltfm_bus_width, + .set_bus_width = esdhc_pltfm_set_bus_width, .set_uhs_signaling = esdhc_set_uhs_signaling, + .reset = esdhc_reset, }; static const struct sdhci_pltfm_data sdhci_esdhc_imx_pdata = { @@ -1170,8 +1173,10 @@ static int sdhci_esdhc_runtime_suspend(struct device *dev) ret = sdhci_runtime_suspend_host(host); - clk_disable_unprepare(imx_data->clk_per); - clk_disable_unprepare(imx_data->clk_ipg); + if (!sdhci_sdio_irq_enabled(host)) { + clk_disable_unprepare(imx_data->clk_per); + clk_disable_unprepare(imx_data->clk_ipg); + } clk_disable_unprepare(imx_data->clk_ahb); return ret; @@ -1183,8 +1188,10 @@ static int sdhci_esdhc_runtime_resume(struct device *dev) struct sdhci_pltfm_host *pltfm_host = sdhci_priv(host); struct pltfm_imx_data *imx_data = pltfm_host->priv; - clk_prepare_enable(imx_data->clk_per); - clk_prepare_enable(imx_data->clk_ipg); + if (!sdhci_sdio_irq_enabled(host)) { + clk_prepare_enable(imx_data->clk_per); + clk_prepare_enable(imx_data->clk_ipg); + } clk_prepare_enable(imx_data->clk_ahb); return sdhci_runtime_resume_host(host); diff --git a/drivers/mmc/host/sdhci-esdhc.h b/drivers/mmc/host/sdhci-esdhc.h index a7d9f95a7b03..3497cfaf683c 100644 --- a/drivers/mmc/host/sdhci-esdhc.h +++ b/drivers/mmc/host/sdhci-esdhc.h @@ -20,10 +20,8 @@ #define ESDHC_DEFAULT_QUIRKS (SDHCI_QUIRK_FORCE_BLK_SZ_2048 | \ SDHCI_QUIRK_NO_BUSY_IRQ | \ - SDHCI_QUIRK_NONSTANDARD_CLOCK | \ SDHCI_QUIRK_DATA_TIMEOUT_USES_SDCLK | \ - SDHCI_QUIRK_PIO_NEEDS_DELAY | \ - SDHCI_QUIRK_RESTORE_IRQS_AFTER_RESET) + SDHCI_QUIRK_PIO_NEEDS_DELAY) #define ESDHC_SYSTEM_CONTROL 0x2c #define ESDHC_CLOCK_MASK 0x0000fff0 diff --git a/drivers/mmc/host/sdhci-of-arasan.c b/drivers/mmc/host/sdhci-of-arasan.c index f7c7cf62437d..5bd1092310f2 100644 --- a/drivers/mmc/host/sdhci-of-arasan.c +++ b/drivers/mmc/host/sdhci-of-arasan.c @@ -52,8 +52,12 @@ static unsigned int sdhci_arasan_get_timeout_clock(struct sdhci_host *host) } static struct sdhci_ops sdhci_arasan_ops = { + .set_clock = sdhci_set_clock, .get_max_clock = sdhci_pltfm_clk_get_max_clock, .get_timeout_clock = sdhci_arasan_get_timeout_clock, + .set_bus_width = sdhci_set_bus_width, + .reset = sdhci_reset, + .set_uhs_signaling = sdhci_set_uhs_signaling, }; static struct sdhci_pltfm_data sdhci_arasan_pdata = { diff --git a/drivers/mmc/host/sdhci-of-esdhc.c b/drivers/mmc/host/sdhci-of-esdhc.c index 0b249970b119..8be4dcfb49a0 100644 --- a/drivers/mmc/host/sdhci-of-esdhc.c +++ b/drivers/mmc/host/sdhci-of-esdhc.c @@ -199,13 +199,14 @@ static unsigned int esdhc_of_get_min_clock(struct sdhci_host *host) static void esdhc_of_set_clock(struct sdhci_host *host, unsigned int clock) { - int pre_div = 2; int div = 1; u32 temp; + host->mmc->actual_clock = 0; + if (clock == 0) - goto out; + return; /* Workaround to reduce the clock frequency for p1010 esdhc */ if (of_find_compatible_node(NULL, NULL, "fsl,p1010-esdhc")) { @@ -238,24 +239,8 @@ static void esdhc_of_set_clock(struct sdhci_host *host, unsigned int clock) | (pre_div << ESDHC_PREDIV_SHIFT)); sdhci_writel(host, temp, ESDHC_SYSTEM_CONTROL); mdelay(1); -out: - host->clock = clock; -} - -#ifdef CONFIG_PM -static u32 esdhc_proctl; -static void esdhc_of_suspend(struct sdhci_host *host) -{ - esdhc_proctl = sdhci_be32bs_readl(host, SDHCI_HOST_CONTROL); } -static void esdhc_of_resume(struct sdhci_host *host) -{ - esdhc_of_enable_dma(host); - sdhci_be32bs_writel(host, esdhc_proctl, SDHCI_HOST_CONTROL); -} -#endif - static void esdhc_of_platform_init(struct sdhci_host *host) { u32 vvn; @@ -269,7 +254,7 @@ static void esdhc_of_platform_init(struct sdhci_host *host) host->quirks &= ~SDHCI_QUIRK_NO_BUSY_IRQ; } -static int esdhc_pltfm_bus_width(struct sdhci_host *host, int width) +static void esdhc_pltfm_set_bus_width(struct sdhci_host *host, int width) { u32 ctrl; @@ -289,8 +274,6 @@ static int esdhc_pltfm_bus_width(struct sdhci_host *host, int width) clrsetbits_be32(host->ioaddr + SDHCI_HOST_CONTROL, ESDHC_CTRL_BUSWIDTH_MASK, ctrl); - - return 0; } static const struct sdhci_ops sdhci_esdhc_ops = { @@ -305,13 +288,46 @@ static const struct sdhci_ops sdhci_esdhc_ops = { .get_max_clock = esdhc_of_get_max_clock, .get_min_clock = esdhc_of_get_min_clock, .platform_init = esdhc_of_platform_init, -#ifdef CONFIG_PM - .platform_suspend = esdhc_of_suspend, - .platform_resume = esdhc_of_resume, -#endif .adma_workaround = esdhci_of_adma_workaround, - .platform_bus_width = esdhc_pltfm_bus_width, + .set_bus_width = esdhc_pltfm_set_bus_width, + .reset = sdhci_reset, + .set_uhs_signaling = sdhci_set_uhs_signaling, +}; + +#ifdef CONFIG_PM + +static u32 esdhc_proctl; +static int esdhc_of_suspend(struct device *dev) +{ + struct sdhci_host *host = dev_get_drvdata(dev); + + esdhc_proctl = sdhci_be32bs_readl(host, SDHCI_HOST_CONTROL); + + return sdhci_suspend_host(host); +} + +static int esdhc_of_resume(struct device *dev) +{ + struct sdhci_host *host = dev_get_drvdata(dev); + int ret = sdhci_resume_host(host); + + if (ret == 0) { + /* Isn't this already done by sdhci_resume_host() ? --rmk */ + esdhc_of_enable_dma(host); + sdhci_be32bs_writel(host, esdhc_proctl, SDHCI_HOST_CONTROL); + } + + return ret; +} + +static const struct dev_pm_ops esdhc_pmops = { + .suspend = esdhc_of_suspend, + .resume = esdhc_of_resume, }; +#define ESDHC_PMOPS (&esdhc_pmops) +#else +#define ESDHC_PMOPS NULL +#endif static const struct sdhci_pltfm_data sdhci_esdhc_pdata = { /* @@ -374,7 +390,7 @@ static struct platform_driver sdhci_esdhc_driver = { .name = "sdhci-esdhc", .owner = THIS_MODULE, .of_match_table = sdhci_esdhc_of_match, - .pm = SDHCI_PLTFM_PMOPS, + .pm = ESDHC_PMOPS, }, .probe = sdhci_esdhc_probe, .remove = sdhci_esdhc_remove, diff --git a/drivers/mmc/host/sdhci-of-hlwd.c b/drivers/mmc/host/sdhci-of-hlwd.c index 57c514a81ca5..b341661369a2 100644 --- a/drivers/mmc/host/sdhci-of-hlwd.c +++ b/drivers/mmc/host/sdhci-of-hlwd.c @@ -58,6 +58,10 @@ static const struct sdhci_ops sdhci_hlwd_ops = { .write_l = sdhci_hlwd_writel, .write_w = sdhci_hlwd_writew, .write_b = sdhci_hlwd_writeb, + .set_clock = sdhci_set_clock, + .set_bus_width = sdhci_set_bus_width, + .reset = sdhci_reset, + .set_uhs_signaling = sdhci_set_uhs_signaling, }; static const struct sdhci_pltfm_data sdhci_hlwd_pdata = { diff --git a/drivers/mmc/host/sdhci-pci-o2micro.c b/drivers/mmc/host/sdhci-pci-o2micro.c index f49666bcc52a..5670e381b0cf 100644 --- a/drivers/mmc/host/sdhci-pci-o2micro.c +++ b/drivers/mmc/host/sdhci-pci-o2micro.c @@ -21,6 +21,45 @@ #include "sdhci-pci.h" #include "sdhci-pci-o2micro.h" +static void o2_pci_set_baseclk(struct sdhci_pci_chip *chip, u32 value) +{ + u32 scratch_32; + pci_read_config_dword(chip->pdev, + O2_SD_PLL_SETTING, &scratch_32); + + scratch_32 &= 0x0000FFFF; + scratch_32 |= value; + + pci_write_config_dword(chip->pdev, + O2_SD_PLL_SETTING, scratch_32); +} + +static void o2_pci_led_enable(struct sdhci_pci_chip *chip) +{ + int ret; + u32 scratch_32; + + /* Set led of SD host function enable */ + ret = pci_read_config_dword(chip->pdev, + O2_SD_FUNC_REG0, &scratch_32); + if (ret) + return; + + scratch_32 &= ~O2_SD_FREG0_LEDOFF; + pci_write_config_dword(chip->pdev, + O2_SD_FUNC_REG0, scratch_32); + + ret = pci_read_config_dword(chip->pdev, + O2_SD_TEST_REG, &scratch_32); + if (ret) + return; + + scratch_32 |= O2_SD_LED_ENABLE; + pci_write_config_dword(chip->pdev, + O2_SD_TEST_REG, scratch_32); + +} + void sdhci_pci_o2_fujin2_pci_init(struct sdhci_pci_chip *chip) { u32 scratch_32; @@ -216,6 +255,40 @@ int sdhci_pci_o2_probe(struct sdhci_pci_chip *chip) scratch &= 0x7f; pci_write_config_byte(chip->pdev, O2_SD_LOCK_WP, scratch); + /* DevId=8520 subId= 0x11 or 0x12 Type Chip support */ + if (chip->pdev->device == PCI_DEVICE_ID_O2_FUJIN2) { + ret = pci_read_config_dword(chip->pdev, + O2_SD_FUNC_REG0, + &scratch_32); + scratch_32 = ((scratch_32 & 0xFF000000) >> 24); + + /* Check Whether subId is 0x11 or 0x12 */ + if ((scratch_32 == 0x11) || (scratch_32 == 0x12)) { + scratch_32 = 0x2c280000; + + /* Set Base Clock to 208MZ */ + o2_pci_set_baseclk(chip, scratch_32); + ret = pci_read_config_dword(chip->pdev, + O2_SD_FUNC_REG4, + &scratch_32); + + /* Enable Base Clk setting change */ + scratch_32 |= O2_SD_FREG4_ENABLE_CLK_SET; + pci_write_config_dword(chip->pdev, + O2_SD_FUNC_REG4, + scratch_32); + + /* Set Tuning Window to 4 */ + pci_write_config_byte(chip->pdev, + O2_SD_TUNING_CTRL, 0x44); + + break; + } + } + + /* Enable 8520 led function */ + o2_pci_led_enable(chip); + /* Set timeout CLK */ ret = pci_read_config_dword(chip->pdev, O2_SD_CLK_SETTING, &scratch_32); @@ -276,7 +349,7 @@ int sdhci_pci_o2_probe(struct sdhci_pci_chip *chip) pci_write_config_byte(chip->pdev, O2_SD_LOCK_WP, scratch); ret = pci_read_config_dword(chip->pdev, - O2_SD_FUNC_REG0, &scratch_32); + O2_SD_PLL_SETTING, &scratch_32); if ((scratch_32 & 0xff000000) == 0x01000000) { scratch_32 &= 0x0000FFFF; @@ -299,6 +372,9 @@ int sdhci_pci_o2_probe(struct sdhci_pci_chip *chip) O2_SD_FUNC_REG4, scratch_32); } + /* Set Tuning Windows to 5 */ + pci_write_config_byte(chip->pdev, + O2_SD_TUNING_CTRL, 0x55); /* Lock WP */ ret = pci_read_config_byte(chip->pdev, O2_SD_LOCK_WP, &scratch); diff --git a/drivers/mmc/host/sdhci-pci-o2micro.h b/drivers/mmc/host/sdhci-pci-o2micro.h index dbec4c933488..f7ffc908d9a0 100644 --- a/drivers/mmc/host/sdhci-pci-o2micro.h +++ b/drivers/mmc/host/sdhci-pci-o2micro.h @@ -57,6 +57,9 @@ #define O2_SD_UHS2_L1_CTRL 0x35C #define O2_SD_FUNC_REG3 0x3E0 #define O2_SD_FUNC_REG4 0x3E4 +#define O2_SD_LED_ENABLE BIT(6) +#define O2_SD_FREG0_LEDOFF BIT(13) +#define O2_SD_FREG4_ENABLE_CLK_SET BIT(22) #define O2_SD_VENDOR_SETTING 0x110 #define O2_SD_VENDOR_SETTING2 0x1C8 diff --git a/drivers/mmc/host/sdhci-pci.c b/drivers/mmc/host/sdhci-pci.c index fdc612120362..52c42fcc284c 100644 --- a/drivers/mmc/host/sdhci-pci.c +++ b/drivers/mmc/host/sdhci-pci.c @@ -1031,7 +1031,7 @@ static int sdhci_pci_enable_dma(struct sdhci_host *host) return 0; } -static int sdhci_pci_bus_width(struct sdhci_host *host, int width) +static void sdhci_pci_set_bus_width(struct sdhci_host *host, int width) { u8 ctrl; @@ -1052,8 +1052,6 @@ static int sdhci_pci_bus_width(struct sdhci_host *host, int width) } sdhci_writeb(host, ctrl, SDHCI_HOST_CONTROL); - - return 0; } static void sdhci_pci_gpio_hw_reset(struct sdhci_host *host) @@ -1080,8 +1078,11 @@ static void sdhci_pci_hw_reset(struct sdhci_host *host) } static const struct sdhci_ops sdhci_pci_ops = { + .set_clock = sdhci_set_clock, .enable_dma = sdhci_pci_enable_dma, - .platform_bus_width = sdhci_pci_bus_width, + .set_bus_width = sdhci_pci_set_bus_width, + .reset = sdhci_reset, + .set_uhs_signaling = sdhci_set_uhs_signaling, .hw_reset = sdhci_pci_hw_reset, }; diff --git a/drivers/mmc/host/sdhci-pltfm.c b/drivers/mmc/host/sdhci-pltfm.c index bef250e95418..7e834fb78f42 100644 --- a/drivers/mmc/host/sdhci-pltfm.c +++ b/drivers/mmc/host/sdhci-pltfm.c @@ -45,6 +45,10 @@ unsigned int sdhci_pltfm_clk_get_max_clock(struct sdhci_host *host) EXPORT_SYMBOL_GPL(sdhci_pltfm_clk_get_max_clock); static const struct sdhci_ops sdhci_pltfm_ops = { + .set_clock = sdhci_set_clock, + .set_bus_width = sdhci_set_bus_width, + .reset = sdhci_reset, + .set_uhs_signaling = sdhci_set_uhs_signaling, }; #ifdef CONFIG_OF diff --git a/drivers/mmc/host/sdhci-pxav2.c b/drivers/mmc/host/sdhci-pxav2.c index d51e061ec576..3c0f3c0a1cc8 100644 --- a/drivers/mmc/host/sdhci-pxav2.c +++ b/drivers/mmc/host/sdhci-pxav2.c @@ -51,11 +51,13 @@ #define MMC_CARD 0x1000 #define MMC_WIDTH 0x0100 -static void pxav2_set_private_registers(struct sdhci_host *host, u8 mask) +static void pxav2_reset(struct sdhci_host *host, u8 mask) { struct platform_device *pdev = to_platform_device(mmc_dev(host->mmc)); struct sdhci_pxa_platdata *pdata = pdev->dev.platform_data; + sdhci_reset(host, mask); + if (mask == SDHCI_RESET_ALL) { u16 tmp = 0; @@ -88,7 +90,7 @@ static void pxav2_set_private_registers(struct sdhci_host *host, u8 mask) } } -static int pxav2_mmc_set_width(struct sdhci_host *host, int width) +static void pxav2_mmc_set_bus_width(struct sdhci_host *host, int width) { u8 ctrl; u16 tmp; @@ -107,14 +109,14 @@ static int pxav2_mmc_set_width(struct sdhci_host *host, int width) } writew(tmp, host->ioaddr + SD_CE_ATA_2); writeb(ctrl, host->ioaddr + SDHCI_HOST_CONTROL); - - return 0; } static const struct sdhci_ops pxav2_sdhci_ops = { + .set_clock = sdhci_set_clock, .get_max_clock = sdhci_pltfm_clk_get_max_clock, - .platform_reset_exit = pxav2_set_private_registers, - .platform_bus_width = pxav2_mmc_set_width, + .set_bus_width = pxav2_mmc_set_bus_width, + .reset = pxav2_reset, + .set_uhs_signaling = sdhci_set_uhs_signaling, }; #ifdef CONFIG_OF diff --git a/drivers/mmc/host/sdhci-pxav3.c b/drivers/mmc/host/sdhci-pxav3.c index 2fd73b38c303..f4f128947561 100644 --- a/drivers/mmc/host/sdhci-pxav3.c +++ b/drivers/mmc/host/sdhci-pxav3.c @@ -112,11 +112,13 @@ static int mv_conf_mbus_windows(struct platform_device *pdev, return 0; } -static void pxav3_set_private_registers(struct sdhci_host *host, u8 mask) +static void pxav3_reset(struct sdhci_host *host, u8 mask) { struct platform_device *pdev = to_platform_device(mmc_dev(host->mmc)); struct sdhci_pxa_platdata *pdata = pdev->dev.platform_data; + sdhci_reset(host, mask); + if (mask == SDHCI_RESET_ALL) { /* * tune timing of read data/command when crc error happen @@ -184,7 +186,7 @@ static void pxav3_gen_init_74_clocks(struct sdhci_host *host, u8 power_mode) pxa->power_mode = power_mode; } -static int pxav3_set_uhs_signaling(struct sdhci_host *host, unsigned int uhs) +static void pxav3_set_uhs_signaling(struct sdhci_host *host, unsigned int uhs) { u16 ctrl_2; @@ -218,15 +220,16 @@ static int pxav3_set_uhs_signaling(struct sdhci_host *host, unsigned int uhs) dev_dbg(mmc_dev(host->mmc), "%s uhs = %d, ctrl_2 = %04X\n", __func__, uhs, ctrl_2); - - return 0; } static const struct sdhci_ops pxav3_sdhci_ops = { - .platform_reset_exit = pxav3_set_private_registers, + .set_clock = sdhci_set_clock, .set_uhs_signaling = pxav3_set_uhs_signaling, .platform_send_init_74_clocks = pxav3_gen_init_74_clocks, .get_max_clock = sdhci_pltfm_clk_get_max_clock, + .set_bus_width = sdhci_set_bus_width, + .reset = pxav3_reset, + .set_uhs_signaling = sdhci_set_uhs_signaling, }; static struct sdhci_pltfm_data sdhci_pxav3_pdata = { diff --git a/drivers/mmc/host/sdhci-s3c.c b/drivers/mmc/host/sdhci-s3c.c index d61eb5a70833..fa5954a05449 100644 --- a/drivers/mmc/host/sdhci-s3c.c +++ b/drivers/mmc/host/sdhci-s3c.c @@ -33,9 +33,6 @@ #define MAX_BUS_CLK (4) -/* Number of gpio's used is max data bus width + command and clock lines */ -#define NUM_GPIOS(x) (x + 2) - /** * struct sdhci_s3c - S3C SDHCI instance * @host: The SDHCI host created @@ -58,6 +55,8 @@ struct sdhci_s3c { struct clk *clk_io; struct clk *clk_bus[MAX_BUS_CLK]; unsigned long clk_rates[MAX_BUS_CLK]; + + bool no_divider; }; /** @@ -70,6 +69,7 @@ struct sdhci_s3c { */ struct sdhci_s3c_drv_data { unsigned int sdhci_quirks; + bool no_divider; }; static inline struct sdhci_s3c *to_s3c(struct sdhci_host *host) @@ -119,7 +119,7 @@ static unsigned int sdhci_s3c_consider_clock(struct sdhci_s3c *ourhost, * If controller uses a non-standard clock division, find the best clock * speed possible with selected clock source and skip the division. */ - if (ourhost->host->quirks & SDHCI_QUIRK_NONSTANDARD_CLOCK) { + if (ourhost->no_divider) { rate = clk_round_rate(clksrc, wanted); return wanted - rate; } @@ -161,9 +161,13 @@ static void sdhci_s3c_set_clock(struct sdhci_host *host, unsigned int clock) int src; u32 ctrl; + host->mmc->actual_clock = 0; + /* don't bother if the clock is going off. */ - if (clock == 0) + if (clock == 0) { + sdhci_set_clock(host, clock); return; + } for (src = 0; src < MAX_BUS_CLK; src++) { delta = sdhci_s3c_consider_clock(ourhost, src, clock); @@ -215,6 +219,8 @@ static void sdhci_s3c_set_clock(struct sdhci_host *host, unsigned int clock) if (clock < 25 * 1000000) ctrl |= (S3C_SDHCI_CTRL3_FCSEL3 | S3C_SDHCI_CTRL3_FCSEL2); writel(ctrl, host->ioaddr + S3C_SDHCI_CONTROL3); + + sdhci_set_clock(host, clock); } /** @@ -295,10 +301,11 @@ static void sdhci_cmu_set_clock(struct sdhci_host *host, unsigned int clock) unsigned long timeout; u16 clk = 0; + host->mmc->actual_clock = 0; + /* If the clock is going off, set to 0 at clock control register */ if (clock == 0) { sdhci_writew(host, 0, SDHCI_CLOCK_CONTROL); - host->clock = clock; return; } @@ -306,8 +313,6 @@ static void sdhci_cmu_set_clock(struct sdhci_host *host, unsigned int clock) clk_set_rate(ourhost->clk_bus[ourhost->cur_clk], clock); - host->clock = clock; - clk = SDHCI_CLOCK_INT_EN; sdhci_writew(host, clk, SDHCI_CLOCK_CONTROL); @@ -329,14 +334,14 @@ static void sdhci_cmu_set_clock(struct sdhci_host *host, unsigned int clock) } /** - * sdhci_s3c_platform_bus_width - support 8bit buswidth + * sdhci_s3c_set_bus_width - support 8bit buswidth * @host: The SDHCI host being queried * @width: MMC_BUS_WIDTH_ macro for the bus width being requested * * We have 8-bit width support but is not a v3 controller. * So we add platform_bus_width() and support 8bit width. */ -static int sdhci_s3c_platform_bus_width(struct sdhci_host *host, int width) +static void sdhci_s3c_set_bus_width(struct sdhci_host *host, int width) { u8 ctrl; @@ -358,93 +363,23 @@ static int sdhci_s3c_platform_bus_width(struct sdhci_host *host, int width) } sdhci_writeb(host, ctrl, SDHCI_HOST_CONTROL); - - return 0; } static struct sdhci_ops sdhci_s3c_ops = { .get_max_clock = sdhci_s3c_get_max_clk, .set_clock = sdhci_s3c_set_clock, .get_min_clock = sdhci_s3c_get_min_clock, - .platform_bus_width = sdhci_s3c_platform_bus_width, + .set_bus_width = sdhci_s3c_set_bus_width, + .reset = sdhci_reset, + .set_uhs_signaling = sdhci_set_uhs_signaling, }; -static void sdhci_s3c_notify_change(struct platform_device *dev, int state) -{ - struct sdhci_host *host = platform_get_drvdata(dev); -#ifdef CONFIG_PM_RUNTIME - struct sdhci_s3c *sc = sdhci_priv(host); -#endif - unsigned long flags; - - if (host) { - spin_lock_irqsave(&host->lock, flags); - if (state) { - dev_dbg(&dev->dev, "card inserted.\n"); -#ifdef CONFIG_PM_RUNTIME - clk_prepare_enable(sc->clk_io); -#endif - host->flags &= ~SDHCI_DEVICE_DEAD; - host->quirks |= SDHCI_QUIRK_BROKEN_CARD_DETECTION; - } else { - dev_dbg(&dev->dev, "card removed.\n"); - host->flags |= SDHCI_DEVICE_DEAD; - host->quirks &= ~SDHCI_QUIRK_BROKEN_CARD_DETECTION; -#ifdef CONFIG_PM_RUNTIME - clk_disable_unprepare(sc->clk_io); -#endif - } - tasklet_schedule(&host->card_tasklet); - spin_unlock_irqrestore(&host->lock, flags); - } -} - -static irqreturn_t sdhci_s3c_gpio_card_detect_thread(int irq, void *dev_id) -{ - struct sdhci_s3c *sc = dev_id; - int status = gpio_get_value(sc->ext_cd_gpio); - if (sc->pdata->ext_cd_gpio_invert) - status = !status; - sdhci_s3c_notify_change(sc->pdev, status); - return IRQ_HANDLED; -} - -static void sdhci_s3c_setup_card_detect_gpio(struct sdhci_s3c *sc) -{ - struct s3c_sdhci_platdata *pdata = sc->pdata; - struct device *dev = &sc->pdev->dev; - - if (devm_gpio_request(dev, pdata->ext_cd_gpio, "SDHCI EXT CD") == 0) { - sc->ext_cd_gpio = pdata->ext_cd_gpio; - sc->ext_cd_irq = gpio_to_irq(pdata->ext_cd_gpio); - if (sc->ext_cd_irq && - request_threaded_irq(sc->ext_cd_irq, NULL, - sdhci_s3c_gpio_card_detect_thread, - IRQF_TRIGGER_RISING | - IRQF_TRIGGER_FALLING | - IRQF_ONESHOT, - dev_name(dev), sc) == 0) { - int status = gpio_get_value(sc->ext_cd_gpio); - if (pdata->ext_cd_gpio_invert) - status = !status; - sdhci_s3c_notify_change(sc->pdev, status); - } else { - dev_warn(dev, "cannot request irq for card detect\n"); - sc->ext_cd_irq = 0; - } - } else { - dev_err(dev, "cannot request gpio for card detect\n"); - } -} - #ifdef CONFIG_OF static int sdhci_s3c_parse_dt(struct device *dev, struct sdhci_host *host, struct s3c_sdhci_platdata *pdata) { struct device_node *node = dev->of_node; - struct sdhci_s3c *ourhost = to_s3c(host); u32 max_width; - int gpio; /* if the bus-width property is not specified, assume width as 1 */ if (of_property_read_u32(node, "bus-width", &max_width)) @@ -462,18 +397,8 @@ static int sdhci_s3c_parse_dt(struct device *dev, return 0; } - gpio = of_get_named_gpio(node, "cd-gpios", 0); - if (gpio_is_valid(gpio)) { - pdata->cd_type = S3C_SDHCI_CD_GPIO; - pdata->ext_cd_gpio = gpio; - ourhost->ext_cd_gpio = -1; - if (of_get_property(node, "cd-inverted", NULL)) - pdata->ext_cd_gpio_invert = 1; + if (of_get_named_gpio(node, "cd-gpios", 0)) return 0; - } else if (gpio != -ENOENT) { - dev_err(dev, "invalid card detect gpio specified\n"); - return -EINVAL; - } /* assuming internal card detect that will be configured by pinctrl */ pdata->cd_type = S3C_SDHCI_CD_INTERNAL; @@ -606,8 +531,10 @@ static int sdhci_s3c_probe(struct platform_device *pdev) /* Setup quirks for the controller */ host->quirks |= SDHCI_QUIRK_NO_ENDATTR_IN_NOPDESC; host->quirks |= SDHCI_QUIRK_NO_HISPD_BIT; - if (drv_data) + if (drv_data) { host->quirks |= drv_data->sdhci_quirks; + sc->no_divider = drv_data->no_divider; + } #ifndef CONFIG_MMC_SDHCI_S3C_DMA @@ -656,7 +583,7 @@ static int sdhci_s3c_probe(struct platform_device *pdev) * If controller does not have internal clock divider, * we can use overriding functions instead of default. */ - if (host->quirks & SDHCI_QUIRK_NONSTANDARD_CLOCK) { + if (sc->no_divider) { sdhci_s3c_ops.set_clock = sdhci_cmu_set_clock; sdhci_s3c_ops.get_min_clock = sdhci_cmu_get_min_clock; sdhci_s3c_ops.get_max_clock = sdhci_cmu_get_max_clock; @@ -674,6 +601,8 @@ static int sdhci_s3c_probe(struct platform_device *pdev) pm_runtime_use_autosuspend(&pdev->dev); pm_suspend_ignore_children(&pdev->dev, 1); + mmc_of_parse(host->mmc); + ret = sdhci_add_host(host); if (ret) { dev_err(dev, "sdhci_add_host() failed\n"); @@ -682,15 +611,6 @@ static int sdhci_s3c_probe(struct platform_device *pdev) goto err_req_regs; } - /* The following two methods of card detection might call - sdhci_s3c_notify_change() immediately, so they can be called - only after sdhci_add_host(). Setup errors are ignored. */ - if (pdata->cd_type == S3C_SDHCI_CD_EXTERNAL && pdata->ext_cd_init) - pdata->ext_cd_init(&sdhci_s3c_notify_change); - if (pdata->cd_type == S3C_SDHCI_CD_GPIO && - gpio_is_valid(pdata->ext_cd_gpio)) - sdhci_s3c_setup_card_detect_gpio(sc); - #ifdef CONFIG_PM_RUNTIME if (pdata->cd_type != S3C_SDHCI_CD_INTERNAL) clk_disable_unprepare(sc->clk_io); @@ -711,16 +631,12 @@ static int sdhci_s3c_remove(struct platform_device *pdev) { struct sdhci_host *host = platform_get_drvdata(pdev); struct sdhci_s3c *sc = sdhci_priv(host); - struct s3c_sdhci_platdata *pdata = sc->pdata; - - if (pdata->cd_type == S3C_SDHCI_CD_EXTERNAL && pdata->ext_cd_cleanup) - pdata->ext_cd_cleanup(&sdhci_s3c_notify_change); if (sc->ext_cd_irq) free_irq(sc->ext_cd_irq, sc); #ifdef CONFIG_PM_RUNTIME - if (pdata->cd_type != S3C_SDHCI_CD_INTERNAL) + if (sc->pdata->cd_type != S3C_SDHCI_CD_INTERNAL) clk_prepare_enable(sc->clk_io); #endif sdhci_remove_host(host, 1); @@ -797,7 +713,7 @@ static const struct dev_pm_ops sdhci_s3c_pmops = { #if defined(CONFIG_CPU_EXYNOS4210) || defined(CONFIG_SOC_EXYNOS4212) static struct sdhci_s3c_drv_data exynos4_sdhci_drv_data = { - .sdhci_quirks = SDHCI_QUIRK_NONSTANDARD_CLOCK, + .no_divider = true, }; #define EXYNOS4_SDHCI_DRV_DATA ((kernel_ulong_t)&exynos4_sdhci_drv_data) #else diff --git a/drivers/mmc/host/sdhci-sirf.c b/drivers/mmc/host/sdhci-sirf.c index 696122c1b468..17004531d089 100644 --- a/drivers/mmc/host/sdhci-sirf.c +++ b/drivers/mmc/host/sdhci-sirf.c @@ -28,7 +28,11 @@ static unsigned int sdhci_sirf_get_max_clk(struct sdhci_host *host) } static struct sdhci_ops sdhci_sirf_ops = { + .set_clock = sdhci_set_clock, .get_max_clock = sdhci_sirf_get_max_clk, + .set_bus_width = sdhci_set_bus_width, + .reset = sdhci_reset, + .set_uhs_signaling = sdhci_set_uhs_signaling, }; static struct sdhci_pltfm_data sdhci_sirf_pdata = { diff --git a/drivers/mmc/host/sdhci-spear.c b/drivers/mmc/host/sdhci-spear.c index 0316dec3f006..9d535c7336ef 100644 --- a/drivers/mmc/host/sdhci-spear.c +++ b/drivers/mmc/host/sdhci-spear.c @@ -38,7 +38,10 @@ struct spear_sdhci { /* sdhci ops */ static const struct sdhci_ops sdhci_pltfm_ops = { - /* Nothing to do for now. */ + .set_clock = sdhci_set_clock, + .set_bus_width = sdhci_set_bus_width, + .reset = sdhci_reset, + .set_uhs_signaling = sdhci_set_uhs_signaling, }; #ifdef CONFIG_OF diff --git a/drivers/mmc/host/sdhci-tegra.c b/drivers/mmc/host/sdhci-tegra.c index a835898a68dd..d93a063a36f3 100644 --- a/drivers/mmc/host/sdhci-tegra.c +++ b/drivers/mmc/host/sdhci-tegra.c @@ -32,11 +32,17 @@ /* Tegra SDHOST controller vendor register definitions */ #define SDHCI_TEGRA_VENDOR_MISC_CTRL 0x120 +#define SDHCI_MISC_CTRL_ENABLE_SDR104 0x8 +#define SDHCI_MISC_CTRL_ENABLE_SDR50 0x10 #define SDHCI_MISC_CTRL_ENABLE_SDHCI_SPEC_300 0x20 +#define SDHCI_MISC_CTRL_ENABLE_DDR50 0x200 #define NVQUIRK_FORCE_SDHCI_SPEC_200 BIT(0) #define NVQUIRK_ENABLE_BLOCK_GAP_DET BIT(1) #define NVQUIRK_ENABLE_SDHCI_SPEC_300 BIT(2) +#define NVQUIRK_DISABLE_SDR50 BIT(3) +#define NVQUIRK_DISABLE_SDR104 BIT(4) +#define NVQUIRK_DISABLE_DDR50 BIT(5) struct sdhci_tegra_soc_data { const struct sdhci_pltfm_data *pdata; @@ -48,19 +54,6 @@ struct sdhci_tegra { int power_gpio; }; -static u32 tegra_sdhci_readl(struct sdhci_host *host, int reg) -{ - u32 val; - - if (unlikely(reg == SDHCI_PRESENT_STATE)) { - /* Use wp_gpio here instead? */ - val = readl(host->ioaddr + reg); - return val | SDHCI_WRITE_PROTECT; - } - - return readl(host->ioaddr + reg); -} - static u16 tegra_sdhci_readw(struct sdhci_host *host, int reg) { struct sdhci_pltfm_host *pltfm_host = sdhci_priv(host); @@ -108,26 +101,33 @@ static unsigned int tegra_sdhci_get_ro(struct sdhci_host *host) return mmc_gpio_get_ro(host->mmc); } -static void tegra_sdhci_reset_exit(struct sdhci_host *host, u8 mask) +static void tegra_sdhci_reset(struct sdhci_host *host, u8 mask) { struct sdhci_pltfm_host *pltfm_host = sdhci_priv(host); struct sdhci_tegra *tegra_host = pltfm_host->priv; const struct sdhci_tegra_soc_data *soc_data = tegra_host->soc_data; + u32 misc_ctrl; + + sdhci_reset(host, mask); if (!(mask & SDHCI_RESET_ALL)) return; + misc_ctrl = sdhci_readw(host, SDHCI_TEGRA_VENDOR_MISC_CTRL); /* Erratum: Enable SDHCI spec v3.00 support */ - if (soc_data->nvquirks & NVQUIRK_ENABLE_SDHCI_SPEC_300) { - u32 misc_ctrl; - - misc_ctrl = sdhci_readb(host, SDHCI_TEGRA_VENDOR_MISC_CTRL); + if (soc_data->nvquirks & NVQUIRK_ENABLE_SDHCI_SPEC_300) misc_ctrl |= SDHCI_MISC_CTRL_ENABLE_SDHCI_SPEC_300; - sdhci_writeb(host, misc_ctrl, SDHCI_TEGRA_VENDOR_MISC_CTRL); - } + /* Don't advertise UHS modes which aren't supported yet */ + if (soc_data->nvquirks & NVQUIRK_DISABLE_SDR50) + misc_ctrl &= ~SDHCI_MISC_CTRL_ENABLE_SDR50; + if (soc_data->nvquirks & NVQUIRK_DISABLE_DDR50) + misc_ctrl &= ~SDHCI_MISC_CTRL_ENABLE_DDR50; + if (soc_data->nvquirks & NVQUIRK_DISABLE_SDR104) + misc_ctrl &= ~SDHCI_MISC_CTRL_ENABLE_SDR104; + sdhci_writew(host, misc_ctrl, SDHCI_TEGRA_VENDOR_MISC_CTRL); } -static int tegra_sdhci_buswidth(struct sdhci_host *host, int bus_width) +static void tegra_sdhci_set_bus_width(struct sdhci_host *host, int bus_width) { u32 ctrl; @@ -144,23 +144,25 @@ static int tegra_sdhci_buswidth(struct sdhci_host *host, int bus_width) ctrl &= ~SDHCI_CTRL_4BITBUS; } sdhci_writeb(host, ctrl, SDHCI_HOST_CONTROL); - return 0; } static const struct sdhci_ops tegra_sdhci_ops = { .get_ro = tegra_sdhci_get_ro, - .read_l = tegra_sdhci_readl, .read_w = tegra_sdhci_readw, .write_l = tegra_sdhci_writel, - .platform_bus_width = tegra_sdhci_buswidth, - .platform_reset_exit = tegra_sdhci_reset_exit, + .set_clock = sdhci_set_clock, + .set_bus_width = tegra_sdhci_set_bus_width, + .reset = tegra_sdhci_reset, + .set_uhs_signaling = sdhci_set_uhs_signaling, + .get_max_clock = sdhci_pltfm_clk_get_max_clock, }; static const struct sdhci_pltfm_data sdhci_tegra20_pdata = { .quirks = SDHCI_QUIRK_BROKEN_TIMEOUT_VAL | SDHCI_QUIRK_SINGLE_POWER_WRITE | SDHCI_QUIRK_NO_HISPD_BIT | - SDHCI_QUIRK_BROKEN_ADMA_ZEROLEN_DESC, + SDHCI_QUIRK_BROKEN_ADMA_ZEROLEN_DESC | + SDHCI_QUIRK_CAP_CLOCK_BASE_BROKEN, .ops = &tegra_sdhci_ops, }; @@ -175,13 +177,16 @@ static const struct sdhci_pltfm_data sdhci_tegra30_pdata = { SDHCI_QUIRK_DATA_TIMEOUT_USES_SDCLK | SDHCI_QUIRK_SINGLE_POWER_WRITE | SDHCI_QUIRK_NO_HISPD_BIT | - SDHCI_QUIRK_BROKEN_ADMA_ZEROLEN_DESC, + SDHCI_QUIRK_BROKEN_ADMA_ZEROLEN_DESC | + SDHCI_QUIRK_CAP_CLOCK_BASE_BROKEN, .ops = &tegra_sdhci_ops, }; static struct sdhci_tegra_soc_data soc_data_tegra30 = { .pdata = &sdhci_tegra30_pdata, - .nvquirks = NVQUIRK_ENABLE_SDHCI_SPEC_300, + .nvquirks = NVQUIRK_ENABLE_SDHCI_SPEC_300 | + NVQUIRK_DISABLE_SDR50 | + NVQUIRK_DISABLE_SDR104, }; static const struct sdhci_pltfm_data sdhci_tegra114_pdata = { @@ -189,12 +194,16 @@ static const struct sdhci_pltfm_data sdhci_tegra114_pdata = { SDHCI_QUIRK_DATA_TIMEOUT_USES_SDCLK | SDHCI_QUIRK_SINGLE_POWER_WRITE | SDHCI_QUIRK_NO_HISPD_BIT | - SDHCI_QUIRK_BROKEN_ADMA_ZEROLEN_DESC, + SDHCI_QUIRK_BROKEN_ADMA_ZEROLEN_DESC | + SDHCI_QUIRK_CAP_CLOCK_BASE_BROKEN, .ops = &tegra_sdhci_ops, }; static struct sdhci_tegra_soc_data soc_data_tegra114 = { .pdata = &sdhci_tegra114_pdata, + .nvquirks = NVQUIRK_DISABLE_SDR50 | + NVQUIRK_DISABLE_DDR50 | + NVQUIRK_DISABLE_SDR104, }; static const struct of_device_id sdhci_tegra_dt_match[] = { diff --git a/drivers/mmc/host/sdhci.c b/drivers/mmc/host/sdhci.c index 9a79fc4b60ca..47055f3f01b8 100644 --- a/drivers/mmc/host/sdhci.c +++ b/drivers/mmc/host/sdhci.c @@ -44,6 +44,8 @@ #define MAX_TUNING_LOOP 40 +#define ADMA_SIZE ((128 * 2 + 1) * 4) + static unsigned int debug_quirks = 0; static unsigned int debug_quirks2; @@ -131,43 +133,26 @@ static void sdhci_dumpregs(struct sdhci_host *host) * * \*****************************************************************************/ -static void sdhci_clear_set_irqs(struct sdhci_host *host, u32 clear, u32 set) -{ - u32 ier; - - ier = sdhci_readl(host, SDHCI_INT_ENABLE); - ier &= ~clear; - ier |= set; - sdhci_writel(host, ier, SDHCI_INT_ENABLE); - sdhci_writel(host, ier, SDHCI_SIGNAL_ENABLE); -} - -static void sdhci_unmask_irqs(struct sdhci_host *host, u32 irqs) -{ - sdhci_clear_set_irqs(host, 0, irqs); -} - -static void sdhci_mask_irqs(struct sdhci_host *host, u32 irqs) -{ - sdhci_clear_set_irqs(host, irqs, 0); -} - static void sdhci_set_card_detection(struct sdhci_host *host, bool enable) { - u32 present, irqs; + u32 present; if ((host->quirks & SDHCI_QUIRK_BROKEN_CARD_DETECTION) || (host->mmc->caps & MMC_CAP_NONREMOVABLE)) return; - present = sdhci_readl(host, SDHCI_PRESENT_STATE) & - SDHCI_CARD_PRESENT; - irqs = present ? SDHCI_INT_CARD_REMOVE : SDHCI_INT_CARD_INSERT; + if (enable) { + present = sdhci_readl(host, SDHCI_PRESENT_STATE) & + SDHCI_CARD_PRESENT; - if (enable) - sdhci_unmask_irqs(host, irqs); - else - sdhci_mask_irqs(host, irqs); + host->ier |= present ? SDHCI_INT_CARD_REMOVE : + SDHCI_INT_CARD_INSERT; + } else { + host->ier &= ~(SDHCI_INT_CARD_REMOVE | SDHCI_INT_CARD_INSERT); + } + + sdhci_writel(host, host->ier, SDHCI_INT_ENABLE); + sdhci_writel(host, host->ier, SDHCI_SIGNAL_ENABLE); } static void sdhci_enable_card_detection(struct sdhci_host *host) @@ -180,22 +165,9 @@ static void sdhci_disable_card_detection(struct sdhci_host *host) sdhci_set_card_detection(host, false); } -static void sdhci_reset(struct sdhci_host *host, u8 mask) +void sdhci_reset(struct sdhci_host *host, u8 mask) { unsigned long timeout; - u32 uninitialized_var(ier); - - if (host->quirks & SDHCI_QUIRK_NO_CARD_NO_RESET) { - if (!(sdhci_readl(host, SDHCI_PRESENT_STATE) & - SDHCI_CARD_PRESENT)) - return; - } - - if (host->quirks & SDHCI_QUIRK_RESTORE_IRQS_AFTER_RESET) - ier = sdhci_readl(host, SDHCI_INT_ENABLE); - - if (host->ops->platform_reset_enter) - host->ops->platform_reset_enter(host, mask); sdhci_writeb(host, mask, SDHCI_SOFTWARE_RESET); @@ -220,16 +192,27 @@ static void sdhci_reset(struct sdhci_host *host, u8 mask) timeout--; mdelay(1); } +} +EXPORT_SYMBOL_GPL(sdhci_reset); + +static void sdhci_do_reset(struct sdhci_host *host, u8 mask) +{ + if (host->quirks & SDHCI_QUIRK_NO_CARD_NO_RESET) { + if (!(sdhci_readl(host, SDHCI_PRESENT_STATE) & + SDHCI_CARD_PRESENT)) + return; + } - if (host->ops->platform_reset_exit) - host->ops->platform_reset_exit(host, mask); + host->ops->reset(host, mask); - if (host->quirks & SDHCI_QUIRK_RESTORE_IRQS_AFTER_RESET) - sdhci_clear_set_irqs(host, SDHCI_INT_ALL_MASK, ier); + if (mask & SDHCI_RESET_ALL) { + if (host->flags & (SDHCI_USE_SDMA | SDHCI_USE_ADMA)) { + if (host->ops->enable_dma) + host->ops->enable_dma(host); + } - if (host->flags & (SDHCI_USE_SDMA | SDHCI_USE_ADMA)) { - if ((host->ops->enable_dma) && (mask & SDHCI_RESET_ALL)) - host->ops->enable_dma(host); + /* Resetting the controller clears many */ + host->preset_enabled = false; } } @@ -238,15 +221,18 @@ static void sdhci_set_ios(struct mmc_host *mmc, struct mmc_ios *ios); static void sdhci_init(struct sdhci_host *host, int soft) { if (soft) - sdhci_reset(host, SDHCI_RESET_CMD|SDHCI_RESET_DATA); + sdhci_do_reset(host, SDHCI_RESET_CMD|SDHCI_RESET_DATA); else - sdhci_reset(host, SDHCI_RESET_ALL); + sdhci_do_reset(host, SDHCI_RESET_ALL); - sdhci_clear_set_irqs(host, SDHCI_INT_ALL_MASK, - SDHCI_INT_BUS_POWER | SDHCI_INT_DATA_END_BIT | - SDHCI_INT_DATA_CRC | SDHCI_INT_DATA_TIMEOUT | SDHCI_INT_INDEX | - SDHCI_INT_END_BIT | SDHCI_INT_CRC | SDHCI_INT_TIMEOUT | - SDHCI_INT_DATA_END | SDHCI_INT_RESPONSE); + host->ier = SDHCI_INT_BUS_POWER | SDHCI_INT_DATA_END_BIT | + SDHCI_INT_DATA_CRC | SDHCI_INT_DATA_TIMEOUT | + SDHCI_INT_INDEX | SDHCI_INT_END_BIT | SDHCI_INT_CRC | + SDHCI_INT_TIMEOUT | SDHCI_INT_DATA_END | + SDHCI_INT_RESPONSE; + + sdhci_writel(host, host->ier, SDHCI_INT_ENABLE); + sdhci_writel(host, host->ier, SDHCI_SIGNAL_ENABLE); if (soft) { /* force clock reconfiguration */ @@ -502,11 +488,6 @@ static int sdhci_adma_table_pre(struct sdhci_host *host, else direction = DMA_TO_DEVICE; - /* - * The ADMA descriptor table is mapped further down as we - * need to fill it with data first. - */ - host->align_addr = dma_map_single(mmc_dev(host->mmc), host->align_buffer, 128 * 4, direction); if (dma_mapping_error(mmc_dev(host->mmc), host->align_addr)) @@ -567,7 +548,7 @@ static int sdhci_adma_table_pre(struct sdhci_host *host, * If this triggers then we have a calculation bug * somewhere. :/ */ - WARN_ON((desc - host->adma_desc) > (128 * 2 + 1) * 4); + WARN_ON((desc - host->adma_desc) > ADMA_SIZE); } if (host->quirks & SDHCI_QUIRK_NO_ENDATTR_IN_NOPDESC) { @@ -595,17 +576,8 @@ static int sdhci_adma_table_pre(struct sdhci_host *host, host->align_addr, 128 * 4, direction); } - host->adma_addr = dma_map_single(mmc_dev(host->mmc), - host->adma_desc, (128 * 2 + 1) * 4, DMA_TO_DEVICE); - if (dma_mapping_error(mmc_dev(host->mmc), host->adma_addr)) - goto unmap_entries; - BUG_ON(host->adma_addr & 0x3); - return 0; -unmap_entries: - dma_unmap_sg(mmc_dev(host->mmc), data->sg, - data->sg_len, direction); unmap_align: dma_unmap_single(mmc_dev(host->mmc), host->align_addr, 128 * 4, direction); @@ -623,19 +595,25 @@ static void sdhci_adma_table_post(struct sdhci_host *host, u8 *align; char *buffer; unsigned long flags; + bool has_unaligned; if (data->flags & MMC_DATA_READ) direction = DMA_FROM_DEVICE; else direction = DMA_TO_DEVICE; - dma_unmap_single(mmc_dev(host->mmc), host->adma_addr, - (128 * 2 + 1) * 4, DMA_TO_DEVICE); - dma_unmap_single(mmc_dev(host->mmc), host->align_addr, 128 * 4, direction); - if (data->flags & MMC_DATA_READ) { + /* Do a quick scan of the SG list for any unaligned mappings */ + has_unaligned = false; + for_each_sg(data->sg, sg, host->sg_count, i) + if (sg_dma_address(sg) & 3) { + has_unaligned = true; + break; + } + + if (has_unaligned && data->flags & MMC_DATA_READ) { dma_sync_sg_for_cpu(mmc_dev(host->mmc), data->sg, data->sg_len, direction); @@ -721,9 +699,12 @@ static void sdhci_set_transfer_irqs(struct sdhci_host *host) u32 dma_irqs = SDHCI_INT_DMA_END | SDHCI_INT_ADMA_ERROR; if (host->flags & SDHCI_REQ_USE_DMA) - sdhci_clear_set_irqs(host, pio_irqs, dma_irqs); + host->ier = (host->ier & ~pio_irqs) | dma_irqs; else - sdhci_clear_set_irqs(host, dma_irqs, pio_irqs); + host->ier = (host->ier & ~dma_irqs) | pio_irqs; + + sdhci_writel(host, host->ier, SDHCI_INT_ENABLE); + sdhci_writel(host, host->ier, SDHCI_SIGNAL_ENABLE); } static void sdhci_prepare_data(struct sdhci_host *host, struct mmc_command *cmd) @@ -976,8 +957,8 @@ static void sdhci_finish_data(struct sdhci_host *host) * upon error conditions. */ if (data->error) { - sdhci_reset(host, SDHCI_RESET_CMD); - sdhci_reset(host, SDHCI_RESET_DATA); + sdhci_do_reset(host, SDHCI_RESET_CMD); + sdhci_do_reset(host, SDHCI_RESET_DATA); } sdhci_send_command(host, data->stop); @@ -1107,24 +1088,23 @@ static void sdhci_finish_command(struct sdhci_host *host) static u16 sdhci_get_preset_value(struct sdhci_host *host) { - u16 ctrl, preset = 0; - - ctrl = sdhci_readw(host, SDHCI_HOST_CONTROL2); + u16 preset = 0; - switch (ctrl & SDHCI_CTRL_UHS_MASK) { - case SDHCI_CTRL_UHS_SDR12: + switch (host->timing) { + case MMC_TIMING_UHS_SDR12: preset = sdhci_readw(host, SDHCI_PRESET_FOR_SDR12); break; - case SDHCI_CTRL_UHS_SDR25: + case MMC_TIMING_UHS_SDR25: preset = sdhci_readw(host, SDHCI_PRESET_FOR_SDR25); break; - case SDHCI_CTRL_UHS_SDR50: + case MMC_TIMING_UHS_SDR50: preset = sdhci_readw(host, SDHCI_PRESET_FOR_SDR50); break; - case SDHCI_CTRL_UHS_SDR104: + case MMC_TIMING_UHS_SDR104: + case MMC_TIMING_MMC_HS200: preset = sdhci_readw(host, SDHCI_PRESET_FOR_SDR104); break; - case SDHCI_CTRL_UHS_DDR50: + case MMC_TIMING_UHS_DDR50: preset = sdhci_readw(host, SDHCI_PRESET_FOR_DDR50); break; default: @@ -1136,32 +1116,22 @@ static u16 sdhci_get_preset_value(struct sdhci_host *host) return preset; } -static void sdhci_set_clock(struct sdhci_host *host, unsigned int clock) +void sdhci_set_clock(struct sdhci_host *host, unsigned int clock) { int div = 0; /* Initialized for compiler warning */ int real_div = div, clk_mul = 1; u16 clk = 0; unsigned long timeout; - if (clock && clock == host->clock) - return; - host->mmc->actual_clock = 0; - if (host->ops->set_clock) { - host->ops->set_clock(host, clock); - if (host->quirks & SDHCI_QUIRK_NONSTANDARD_CLOCK) - return; - } - sdhci_writew(host, 0, SDHCI_CLOCK_CONTROL); if (clock == 0) - goto out; + return; if (host->version >= SDHCI_SPEC_300) { - if (sdhci_readw(host, SDHCI_HOST_CONTROL2) & - SDHCI_CTRL_PRESET_VAL_ENABLE) { + if (host->preset_enabled) { u16 pre_val; clk = sdhci_readw(host, SDHCI_CLOCK_CONTROL); @@ -1247,26 +1217,16 @@ clock_set: clk |= SDHCI_CLOCK_CARD_EN; sdhci_writew(host, clk, SDHCI_CLOCK_CONTROL); - -out: - host->clock = clock; } +EXPORT_SYMBOL_GPL(sdhci_set_clock); -static inline void sdhci_update_clock(struct sdhci_host *host) -{ - unsigned int clock; - - clock = host->clock; - host->clock = 0; - sdhci_set_clock(host, clock); -} - -static int sdhci_set_power(struct sdhci_host *host, unsigned short power) +static void sdhci_set_power(struct sdhci_host *host, unsigned char mode, + unsigned short vdd) { u8 pwr = 0; - if (power != (unsigned short)-1) { - switch (1 << power) { + if (mode != MMC_POWER_OFF) { + switch (1 << vdd) { case MMC_VDD_165_195: pwr = SDHCI_POWER_180; break; @@ -1284,7 +1244,7 @@ static int sdhci_set_power(struct sdhci_host *host, unsigned short power) } if (host->pwr == pwr) - return -1; + return; host->pwr = pwr; @@ -1292,38 +1252,43 @@ static int sdhci_set_power(struct sdhci_host *host, unsigned short power) sdhci_writeb(host, 0, SDHCI_POWER_CONTROL); if (host->quirks2 & SDHCI_QUIRK2_CARD_ON_NEEDS_BUS_ON) sdhci_runtime_pm_bus_off(host); - return 0; - } - - /* - * Spec says that we should clear the power reg before setting - * a new value. Some controllers don't seem to like this though. - */ - if (!(host->quirks & SDHCI_QUIRK_SINGLE_POWER_WRITE)) - sdhci_writeb(host, 0, SDHCI_POWER_CONTROL); + vdd = 0; + } else { + /* + * Spec says that we should clear the power reg before setting + * a new value. Some controllers don't seem to like this though. + */ + if (!(host->quirks & SDHCI_QUIRK_SINGLE_POWER_WRITE)) + sdhci_writeb(host, 0, SDHCI_POWER_CONTROL); - /* - * At least the Marvell CaFe chip gets confused if we set the voltage - * and set turn on power at the same time, so set the voltage first. - */ - if (host->quirks & SDHCI_QUIRK_NO_SIMULT_VDD_AND_POWER) - sdhci_writeb(host, pwr, SDHCI_POWER_CONTROL); + /* + * At least the Marvell CaFe chip gets confused if we set the + * voltage and set turn on power at the same time, so set the + * voltage first. + */ + if (host->quirks & SDHCI_QUIRK_NO_SIMULT_VDD_AND_POWER) + sdhci_writeb(host, pwr, SDHCI_POWER_CONTROL); - pwr |= SDHCI_POWER_ON; + pwr |= SDHCI_POWER_ON; - sdhci_writeb(host, pwr, SDHCI_POWER_CONTROL); + sdhci_writeb(host, pwr, SDHCI_POWER_CONTROL); - if (host->quirks2 & SDHCI_QUIRK2_CARD_ON_NEEDS_BUS_ON) - sdhci_runtime_pm_bus_on(host); + if (host->quirks2 & SDHCI_QUIRK2_CARD_ON_NEEDS_BUS_ON) + sdhci_runtime_pm_bus_on(host); - /* - * Some controllers need an extra 10ms delay of 10ms before they - * can apply clock after applying power - */ - if (host->quirks & SDHCI_QUIRK_DELAY_AFTER_POWER) - mdelay(10); + /* + * Some controllers need an extra 10ms delay of 10ms before + * they can apply clock after applying power + */ + if (host->quirks & SDHCI_QUIRK_DELAY_AFTER_POWER) + mdelay(10); + } - return power; + if (host->vmmc) { + spin_unlock_irq(&host->lock); + mmc_regulator_set_ocr(host->mmc, host->vmmc, vdd); + spin_lock_irq(&host->lock); + } } /*****************************************************************************\ @@ -1427,10 +1392,53 @@ static void sdhci_request(struct mmc_host *mmc, struct mmc_request *mrq) spin_unlock_irqrestore(&host->lock, flags); } +void sdhci_set_bus_width(struct sdhci_host *host, int width) +{ + u8 ctrl; + + ctrl = sdhci_readb(host, SDHCI_HOST_CONTROL); + if (width == MMC_BUS_WIDTH_8) { + ctrl &= ~SDHCI_CTRL_4BITBUS; + if (host->version >= SDHCI_SPEC_300) + ctrl |= SDHCI_CTRL_8BITBUS; + } else { + if (host->version >= SDHCI_SPEC_300) + ctrl &= ~SDHCI_CTRL_8BITBUS; + if (width == MMC_BUS_WIDTH_4) + ctrl |= SDHCI_CTRL_4BITBUS; + else + ctrl &= ~SDHCI_CTRL_4BITBUS; + } + sdhci_writeb(host, ctrl, SDHCI_HOST_CONTROL); +} +EXPORT_SYMBOL_GPL(sdhci_set_bus_width); + +void sdhci_set_uhs_signaling(struct sdhci_host *host, unsigned timing) +{ + u16 ctrl_2; + + ctrl_2 = sdhci_readw(host, SDHCI_HOST_CONTROL2); + /* Select Bus Speed Mode for host */ + ctrl_2 &= ~SDHCI_CTRL_UHS_MASK; + if ((timing == MMC_TIMING_MMC_HS200) || + (timing == MMC_TIMING_UHS_SDR104)) + ctrl_2 |= SDHCI_CTRL_UHS_SDR104; + else if (timing == MMC_TIMING_UHS_SDR12) + ctrl_2 |= SDHCI_CTRL_UHS_SDR12; + else if (timing == MMC_TIMING_UHS_SDR25) + ctrl_2 |= SDHCI_CTRL_UHS_SDR25; + else if (timing == MMC_TIMING_UHS_SDR50) + ctrl_2 |= SDHCI_CTRL_UHS_SDR50; + else if ((timing == MMC_TIMING_UHS_DDR50) || + (timing == MMC_TIMING_MMC_DDR52)) + ctrl_2 |= SDHCI_CTRL_UHS_DDR50; + sdhci_writew(host, ctrl_2, SDHCI_HOST_CONTROL2); +} +EXPORT_SYMBOL_GPL(sdhci_set_uhs_signaling); + static void sdhci_do_set_ios(struct sdhci_host *host, struct mmc_ios *ios) { unsigned long flags; - int vdd_bit = -1; u8 ctrl; spin_lock_irqsave(&host->lock, flags); @@ -1456,45 +1464,17 @@ static void sdhci_do_set_ios(struct sdhci_host *host, struct mmc_ios *ios) !(host->quirks2 & SDHCI_QUIRK2_PRESET_VALUE_BROKEN)) sdhci_enable_preset_value(host, false); - sdhci_set_clock(host, ios->clock); - - if (ios->power_mode == MMC_POWER_OFF) - vdd_bit = sdhci_set_power(host, -1); - else - vdd_bit = sdhci_set_power(host, ios->vdd); - - if (host->vmmc && vdd_bit != -1) { - spin_unlock_irqrestore(&host->lock, flags); - mmc_regulator_set_ocr(host->mmc, host->vmmc, vdd_bit); - spin_lock_irqsave(&host->lock, flags); + if (!ios->clock || ios->clock != host->clock) { + host->ops->set_clock(host, ios->clock); + host->clock = ios->clock; } + sdhci_set_power(host, ios->power_mode, ios->vdd); + if (host->ops->platform_send_init_74_clocks) host->ops->platform_send_init_74_clocks(host, ios->power_mode); - /* - * If your platform has 8-bit width support but is not a v3 controller, - * or if it requires special setup code, you should implement that in - * platform_bus_width(). - */ - if (host->ops->platform_bus_width) { - host->ops->platform_bus_width(host, ios->bus_width); - } else { - ctrl = sdhci_readb(host, SDHCI_HOST_CONTROL); - if (ios->bus_width == MMC_BUS_WIDTH_8) { - ctrl &= ~SDHCI_CTRL_4BITBUS; - if (host->version >= SDHCI_SPEC_300) - ctrl |= SDHCI_CTRL_8BITBUS; - } else { - if (host->version >= SDHCI_SPEC_300) - ctrl &= ~SDHCI_CTRL_8BITBUS; - if (ios->bus_width == MMC_BUS_WIDTH_4) - ctrl |= SDHCI_CTRL_4BITBUS; - else - ctrl &= ~SDHCI_CTRL_4BITBUS; - } - sdhci_writeb(host, ctrl, SDHCI_HOST_CONTROL); - } + host->ops->set_bus_width(host, ios->bus_width); ctrl = sdhci_readb(host, SDHCI_HOST_CONTROL); @@ -1510,19 +1490,20 @@ static void sdhci_do_set_ios(struct sdhci_host *host, struct mmc_ios *ios) /* In case of UHS-I modes, set High Speed Enable */ if ((ios->timing == MMC_TIMING_MMC_HS200) || + (ios->timing == MMC_TIMING_MMC_DDR52) || (ios->timing == MMC_TIMING_UHS_SDR50) || (ios->timing == MMC_TIMING_UHS_SDR104) || (ios->timing == MMC_TIMING_UHS_DDR50) || (ios->timing == MMC_TIMING_UHS_SDR25)) ctrl |= SDHCI_CTRL_HISPD; - ctrl_2 = sdhci_readw(host, SDHCI_HOST_CONTROL2); - if (!(ctrl_2 & SDHCI_CTRL_PRESET_VAL_ENABLE)) { + if (!host->preset_enabled) { sdhci_writeb(host, ctrl, SDHCI_HOST_CONTROL); /* * We only need to set Driver Strength if the * preset value enable is not set. */ + ctrl_2 = sdhci_readw(host, SDHCI_HOST_CONTROL2); ctrl_2 &= ~SDHCI_CTRL_DRV_TYPE_MASK; if (ios->drv_type == MMC_SET_DRIVER_TYPE_A) ctrl_2 |= SDHCI_CTRL_DRV_TYPE_A; @@ -1546,7 +1527,7 @@ static void sdhci_do_set_ios(struct sdhci_host *host, struct mmc_ios *ios) sdhci_writeb(host, ctrl, SDHCI_HOST_CONTROL); /* Re-enable SD Clock */ - sdhci_update_clock(host); + host->ops->set_clock(host, host->clock); } @@ -1555,25 +1536,8 @@ static void sdhci_do_set_ios(struct sdhci_host *host, struct mmc_ios *ios) clk &= ~SDHCI_CLOCK_CARD_EN; sdhci_writew(host, clk, SDHCI_CLOCK_CONTROL); - if (host->ops->set_uhs_signaling) - host->ops->set_uhs_signaling(host, ios->timing); - else { - ctrl_2 = sdhci_readw(host, SDHCI_HOST_CONTROL2); - /* Select Bus Speed Mode for host */ - ctrl_2 &= ~SDHCI_CTRL_UHS_MASK; - if ((ios->timing == MMC_TIMING_MMC_HS200) || - (ios->timing == MMC_TIMING_UHS_SDR104)) - ctrl_2 |= SDHCI_CTRL_UHS_SDR104; - else if (ios->timing == MMC_TIMING_UHS_SDR12) - ctrl_2 |= SDHCI_CTRL_UHS_SDR12; - else if (ios->timing == MMC_TIMING_UHS_SDR25) - ctrl_2 |= SDHCI_CTRL_UHS_SDR25; - else if (ios->timing == MMC_TIMING_UHS_SDR50) - ctrl_2 |= SDHCI_CTRL_UHS_SDR50; - else if (ios->timing == MMC_TIMING_UHS_DDR50) - ctrl_2 |= SDHCI_CTRL_UHS_DDR50; - sdhci_writew(host, ctrl_2, SDHCI_HOST_CONTROL2); - } + host->ops->set_uhs_signaling(host, ios->timing); + host->timing = ios->timing; if (!(host->quirks2 & SDHCI_QUIRK2_PRESET_VALUE_BROKEN) && ((ios->timing == MMC_TIMING_UHS_SDR12) || @@ -1590,7 +1554,7 @@ static void sdhci_do_set_ios(struct sdhci_host *host, struct mmc_ios *ios) } /* Re-enable SD Clock */ - sdhci_update_clock(host); + host->ops->set_clock(host, host->clock); } else sdhci_writeb(host, ctrl, SDHCI_HOST_CONTROL); @@ -1600,7 +1564,7 @@ static void sdhci_do_set_ios(struct sdhci_host *host, struct mmc_ios *ios) * it on each ios seems to solve the problem. */ if(host->quirks & SDHCI_QUIRK_RESET_CMD_DATA_ON_IOS) - sdhci_reset(host, SDHCI_RESET_CMD | SDHCI_RESET_DATA); + sdhci_do_reset(host, SDHCI_RESET_CMD | SDHCI_RESET_DATA); mmiowb(); spin_unlock_irqrestore(&host->lock, flags); @@ -1709,24 +1673,16 @@ static int sdhci_get_ro(struct mmc_host *mmc) static void sdhci_enable_sdio_irq_nolock(struct sdhci_host *host, int enable) { - if (host->flags & SDHCI_DEVICE_DEAD) - goto out; - - if (enable) - host->flags |= SDHCI_SDIO_IRQ_ENABLED; - else - host->flags &= ~SDHCI_SDIO_IRQ_ENABLED; - - /* SDIO IRQ will be enabled as appropriate in runtime resume */ - if (host->runtime_suspended) - goto out; + if (!(host->flags & SDHCI_DEVICE_DEAD)) { + if (enable) + host->ier |= SDHCI_INT_CARD_INT; + else + host->ier &= ~SDHCI_INT_CARD_INT; - if (enable) - sdhci_unmask_irqs(host, SDHCI_INT_CARD_INT); - else - sdhci_mask_irqs(host, SDHCI_INT_CARD_INT); -out: - mmiowb(); + sdhci_writel(host, host->ier, SDHCI_INT_ENABLE); + sdhci_writel(host, host->ier, SDHCI_SIGNAL_ENABLE); + mmiowb(); + } } static void sdhci_enable_sdio_irq(struct mmc_host *mmc, int enable) @@ -1734,9 +1690,18 @@ static void sdhci_enable_sdio_irq(struct mmc_host *mmc, int enable) struct sdhci_host *host = mmc_priv(mmc); unsigned long flags; + sdhci_runtime_pm_get(host); + spin_lock_irqsave(&host->lock, flags); + if (enable) + host->flags |= SDHCI_SDIO_IRQ_ENABLED; + else + host->flags &= ~SDHCI_SDIO_IRQ_ENABLED; + sdhci_enable_sdio_irq_nolock(host, enable); spin_unlock_irqrestore(&host->lock, flags); + + sdhci_runtime_pm_put(host); } static int sdhci_do_start_signal_voltage_switch(struct sdhci_host *host, @@ -1855,22 +1820,15 @@ static int sdhci_card_busy(struct mmc_host *mmc) static int sdhci_execute_tuning(struct mmc_host *mmc, u32 opcode) { - struct sdhci_host *host; + struct sdhci_host *host = mmc_priv(mmc); u16 ctrl; - u32 ier; int tuning_loop_counter = MAX_TUNING_LOOP; - unsigned long timeout; int err = 0; - bool requires_tuning_nonuhs = false; unsigned long flags; - host = mmc_priv(mmc); - sdhci_runtime_pm_get(host); spin_lock_irqsave(&host->lock, flags); - ctrl = sdhci_readw(host, SDHCI_HOST_CONTROL2); - /* * The Host Controller needs tuning only in case of SDR104 mode * and for SDR50 mode when Use Tuning for SDR50 is set in the @@ -1878,15 +1836,18 @@ static int sdhci_execute_tuning(struct mmc_host *mmc, u32 opcode) * If the Host Controller supports the HS200 mode then the * tuning function has to be executed. */ - if (((ctrl & SDHCI_CTRL_UHS_MASK) == SDHCI_CTRL_UHS_SDR50) && - (host->flags & SDHCI_SDR50_NEEDS_TUNING || - host->flags & SDHCI_SDR104_NEEDS_TUNING)) - requires_tuning_nonuhs = true; - - if (((ctrl & SDHCI_CTRL_UHS_MASK) == SDHCI_CTRL_UHS_SDR104) || - requires_tuning_nonuhs) - ctrl |= SDHCI_CTRL_EXEC_TUNING; - else { + switch (host->timing) { + case MMC_TIMING_MMC_HS200: + case MMC_TIMING_UHS_SDR104: + break; + + case MMC_TIMING_UHS_SDR50: + if (host->flags & SDHCI_SDR50_NEEDS_TUNING || + host->flags & SDHCI_SDR104_NEEDS_TUNING) + break; + /* FALLTHROUGH */ + + default: spin_unlock_irqrestore(&host->lock, flags); sdhci_runtime_pm_put(host); return 0; @@ -1899,6 +1860,8 @@ static int sdhci_execute_tuning(struct mmc_host *mmc, u32 opcode) return err; } + ctrl = sdhci_readw(host, SDHCI_HOST_CONTROL2); + ctrl |= SDHCI_CTRL_EXEC_TUNING; sdhci_writew(host, ctrl, SDHCI_HOST_CONTROL2); /* @@ -1911,21 +1874,17 @@ static int sdhci_execute_tuning(struct mmc_host *mmc, u32 opcode) * to make sure we don't hit a controller bug, we _only_ * enable Buffer Read Ready interrupt here. */ - ier = sdhci_readl(host, SDHCI_INT_ENABLE); - sdhci_clear_set_irqs(host, ier, SDHCI_INT_DATA_AVAIL); + sdhci_writel(host, SDHCI_INT_DATA_AVAIL, SDHCI_INT_ENABLE); + sdhci_writel(host, SDHCI_INT_DATA_AVAIL, SDHCI_SIGNAL_ENABLE); /* * Issue CMD19 repeatedly till Execute Tuning is set to 0 or the number * of loops reaches 40 times or a timeout of 150ms occurs. */ - timeout = 150; do { struct mmc_command cmd = {0}; struct mmc_request mrq = {NULL}; - if (!tuning_loop_counter && !timeout) - break; - cmd.opcode = opcode; cmd.arg = 0; cmd.flags = MMC_RSP_R1 | MMC_CMD_ADTC; @@ -1933,6 +1892,9 @@ static int sdhci_execute_tuning(struct mmc_host *mmc, u32 opcode) cmd.data = NULL; cmd.error = 0; + if (tuning_loop_counter-- == 0) + break; + mrq.cmd = &cmd; host->mrq = &mrq; @@ -1990,26 +1952,25 @@ static int sdhci_execute_tuning(struct mmc_host *mmc, u32 opcode) host->tuning_done = 0; ctrl = sdhci_readw(host, SDHCI_HOST_CONTROL2); - tuning_loop_counter--; - timeout--; - mdelay(1); + + /* eMMC spec does not require a delay between tuning cycles */ + if (opcode == MMC_SEND_TUNING_BLOCK) + mdelay(1); } while (ctrl & SDHCI_CTRL_EXEC_TUNING); /* * The Host Driver has exhausted the maximum number of loops allowed, * so use fixed sampling frequency. */ - if (!tuning_loop_counter || !timeout) { + if (tuning_loop_counter < 0) { ctrl &= ~SDHCI_CTRL_TUNED_CLK; sdhci_writew(host, ctrl, SDHCI_HOST_CONTROL2); + } + if (!(ctrl & SDHCI_CTRL_TUNED_CLK)) { + pr_info(DRIVER_NAME ": Tuning procedure" + " failed, falling back to fixed sampling" + " clock\n"); err = -EIO; - } else { - if (!(ctrl & SDHCI_CTRL_TUNED_CLK)) { - pr_info(DRIVER_NAME ": Tuning procedure" - " failed, falling back to fixed sampling" - " clock\n"); - err = -EIO; - } } out: @@ -2044,7 +2005,8 @@ out: if (err && (host->flags & SDHCI_USING_RETUNING_TIMER)) err = 0; - sdhci_clear_set_irqs(host, SDHCI_INT_DATA_AVAIL, ier); + sdhci_writel(host, host->ier, SDHCI_INT_ENABLE); + sdhci_writel(host, host->ier, SDHCI_SIGNAL_ENABLE); spin_unlock_irqrestore(&host->lock, flags); sdhci_runtime_pm_put(host); @@ -2054,26 +2016,30 @@ out: static void sdhci_enable_preset_value(struct sdhci_host *host, bool enable) { - u16 ctrl; - /* Host Controller v3.00 defines preset value registers */ if (host->version < SDHCI_SPEC_300) return; - ctrl = sdhci_readw(host, SDHCI_HOST_CONTROL2); - /* * We only enable or disable Preset Value if they are not already * enabled or disabled respectively. Otherwise, we bail out. */ - if (enable && !(ctrl & SDHCI_CTRL_PRESET_VAL_ENABLE)) { - ctrl |= SDHCI_CTRL_PRESET_VAL_ENABLE; - sdhci_writew(host, ctrl, SDHCI_HOST_CONTROL2); - host->flags |= SDHCI_PV_ENABLED; - } else if (!enable && (ctrl & SDHCI_CTRL_PRESET_VAL_ENABLE)) { - ctrl &= ~SDHCI_CTRL_PRESET_VAL_ENABLE; + if (host->preset_enabled != enable) { + u16 ctrl = sdhci_readw(host, SDHCI_HOST_CONTROL2); + + if (enable) + ctrl |= SDHCI_CTRL_PRESET_VAL_ENABLE; + else + ctrl &= ~SDHCI_CTRL_PRESET_VAL_ENABLE; + sdhci_writew(host, ctrl, SDHCI_HOST_CONTROL2); - host->flags &= ~SDHCI_PV_ENABLED; + + if (enable) + host->flags |= SDHCI_PV_ENABLED; + else + host->flags &= ~SDHCI_PV_ENABLED; + + host->preset_enabled = enable; } } @@ -2095,8 +2061,8 @@ static void sdhci_card_event(struct mmc_host *mmc) pr_err("%s: Resetting controller.\n", mmc_hostname(host->mmc)); - sdhci_reset(host, SDHCI_RESET_CMD); - sdhci_reset(host, SDHCI_RESET_DATA); + sdhci_do_reset(host, SDHCI_RESET_CMD); + sdhci_do_reset(host, SDHCI_RESET_DATA); host->mrq->cmd->error = -ENOMEDIUM; tasklet_schedule(&host->finish_tasklet); @@ -2124,15 +2090,6 @@ static const struct mmc_host_ops sdhci_ops = { * * \*****************************************************************************/ -static void sdhci_tasklet_card(unsigned long param) -{ - struct sdhci_host *host = (struct sdhci_host*)param; - - sdhci_card_event(host->mmc); - - mmc_detect_change(host->mmc, msecs_to_jiffies(200)); -} - static void sdhci_tasklet_finish(unsigned long param) { struct sdhci_host *host; @@ -2169,12 +2126,12 @@ static void sdhci_tasklet_finish(unsigned long param) /* Some controllers need this kick or reset won't work here */ if (host->quirks & SDHCI_QUIRK_CLOCK_BEFORE_RESET) /* This is to force an update */ - sdhci_update_clock(host); + host->ops->set_clock(host, host->clock); /* Spec says we should do both at the same time, but Ricoh controllers do not like that. */ - sdhci_reset(host, SDHCI_RESET_CMD); - sdhci_reset(host, SDHCI_RESET_DATA); + sdhci_do_reset(host, SDHCI_RESET_CMD); + sdhci_do_reset(host, SDHCI_RESET_DATA); } host->mrq = NULL; @@ -2424,101 +2381,94 @@ static void sdhci_data_irq(struct sdhci_host *host, u32 intmask) static irqreturn_t sdhci_irq(int irq, void *dev_id) { - irqreturn_t result; + irqreturn_t result = IRQ_NONE; struct sdhci_host *host = dev_id; - u32 intmask, unexpected = 0; - int cardint = 0, max_loops = 16; + u32 intmask, mask, unexpected = 0; + int max_loops = 16; spin_lock(&host->lock); - if (host->runtime_suspended) { + if (host->runtime_suspended && !sdhci_sdio_irq_enabled(host)) { spin_unlock(&host->lock); return IRQ_NONE; } intmask = sdhci_readl(host, SDHCI_INT_STATUS); - if (!intmask || intmask == 0xffffffff) { result = IRQ_NONE; goto out; } -again: - DBG("*** %s got interrupt: 0x%08x\n", - mmc_hostname(host->mmc), intmask); - - if (intmask & (SDHCI_INT_CARD_INSERT | SDHCI_INT_CARD_REMOVE)) { - u32 present = sdhci_readl(host, SDHCI_PRESENT_STATE) & - SDHCI_CARD_PRESENT; - - /* - * There is a observation on i.mx esdhc. INSERT bit will be - * immediately set again when it gets cleared, if a card is - * inserted. We have to mask the irq to prevent interrupt - * storm which will freeze the system. And the REMOVE gets - * the same situation. - * - * More testing are needed here to ensure it works for other - * platforms though. - */ - sdhci_mask_irqs(host, present ? SDHCI_INT_CARD_INSERT : - SDHCI_INT_CARD_REMOVE); - sdhci_unmask_irqs(host, present ? SDHCI_INT_CARD_REMOVE : - SDHCI_INT_CARD_INSERT); - - sdhci_writel(host, intmask & (SDHCI_INT_CARD_INSERT | - SDHCI_INT_CARD_REMOVE), SDHCI_INT_STATUS); - intmask &= ~(SDHCI_INT_CARD_INSERT | SDHCI_INT_CARD_REMOVE); - tasklet_schedule(&host->card_tasklet); - } - - if (intmask & SDHCI_INT_CMD_MASK) { - sdhci_writel(host, intmask & SDHCI_INT_CMD_MASK, - SDHCI_INT_STATUS); - sdhci_cmd_irq(host, intmask & SDHCI_INT_CMD_MASK); - } + do { + /* Clear selected interrupts. */ + mask = intmask & (SDHCI_INT_CMD_MASK | SDHCI_INT_DATA_MASK | + SDHCI_INT_BUS_POWER); + sdhci_writel(host, mask, SDHCI_INT_STATUS); - if (intmask & SDHCI_INT_DATA_MASK) { - sdhci_writel(host, intmask & SDHCI_INT_DATA_MASK, - SDHCI_INT_STATUS); - sdhci_data_irq(host, intmask & SDHCI_INT_DATA_MASK); - } + DBG("*** %s got interrupt: 0x%08x\n", + mmc_hostname(host->mmc), intmask); - intmask &= ~(SDHCI_INT_CMD_MASK | SDHCI_INT_DATA_MASK); + if (intmask & (SDHCI_INT_CARD_INSERT | SDHCI_INT_CARD_REMOVE)) { + u32 present = sdhci_readl(host, SDHCI_PRESENT_STATE) & + SDHCI_CARD_PRESENT; - intmask &= ~SDHCI_INT_ERROR; + /* + * There is a observation on i.mx esdhc. INSERT + * bit will be immediately set again when it gets + * cleared, if a card is inserted. We have to mask + * the irq to prevent interrupt storm which will + * freeze the system. And the REMOVE gets the + * same situation. + * + * More testing are needed here to ensure it works + * for other platforms though. + */ + host->ier &= ~(SDHCI_INT_CARD_INSERT | + SDHCI_INT_CARD_REMOVE); + host->ier |= present ? SDHCI_INT_CARD_REMOVE : + SDHCI_INT_CARD_INSERT; + sdhci_writel(host, host->ier, SDHCI_INT_ENABLE); + sdhci_writel(host, host->ier, SDHCI_SIGNAL_ENABLE); + + sdhci_writel(host, intmask & (SDHCI_INT_CARD_INSERT | + SDHCI_INT_CARD_REMOVE), SDHCI_INT_STATUS); + + host->thread_isr |= intmask & (SDHCI_INT_CARD_INSERT | + SDHCI_INT_CARD_REMOVE); + result = IRQ_WAKE_THREAD; + } - if (intmask & SDHCI_INT_BUS_POWER) { - pr_err("%s: Card is consuming too much power!\n", - mmc_hostname(host->mmc)); - sdhci_writel(host, SDHCI_INT_BUS_POWER, SDHCI_INT_STATUS); - } + if (intmask & SDHCI_INT_CMD_MASK) + sdhci_cmd_irq(host, intmask & SDHCI_INT_CMD_MASK); - intmask &= ~SDHCI_INT_BUS_POWER; + if (intmask & SDHCI_INT_DATA_MASK) + sdhci_data_irq(host, intmask & SDHCI_INT_DATA_MASK); - if (intmask & SDHCI_INT_CARD_INT) - cardint = 1; + if (intmask & SDHCI_INT_BUS_POWER) + pr_err("%s: Card is consuming too much power!\n", + mmc_hostname(host->mmc)); - intmask &= ~SDHCI_INT_CARD_INT; + if (intmask & SDHCI_INT_CARD_INT) { + sdhci_enable_sdio_irq_nolock(host, false); + host->thread_isr |= SDHCI_INT_CARD_INT; + result = IRQ_WAKE_THREAD; + } - if (intmask) { - unexpected |= intmask; - sdhci_writel(host, intmask, SDHCI_INT_STATUS); - } + intmask &= ~(SDHCI_INT_CARD_INSERT | SDHCI_INT_CARD_REMOVE | + SDHCI_INT_CMD_MASK | SDHCI_INT_DATA_MASK | + SDHCI_INT_ERROR | SDHCI_INT_BUS_POWER | + SDHCI_INT_CARD_INT); - result = IRQ_HANDLED; + if (intmask) { + unexpected |= intmask; + sdhci_writel(host, intmask, SDHCI_INT_STATUS); + } - intmask = sdhci_readl(host, SDHCI_INT_STATUS); + if (result == IRQ_NONE) + result = IRQ_HANDLED; - /* - * If we know we'll call the driver to signal SDIO IRQ, disregard - * further indications of Card Interrupt in the status to avoid a - * needless loop. - */ - if (cardint) - intmask &= ~SDHCI_INT_CARD_INT; - if (intmask && --max_loops) - goto again; + intmask = sdhci_readl(host, SDHCI_INT_STATUS); + } while (intmask && --max_loops); out: spin_unlock(&host->lock); @@ -2527,15 +2477,38 @@ out: mmc_hostname(host->mmc), unexpected); sdhci_dumpregs(host); } - /* - * We have to delay this as it calls back into the driver. - */ - if (cardint) - mmc_signal_sdio_irq(host->mmc); return result; } +static irqreturn_t sdhci_thread_irq(int irq, void *dev_id) +{ + struct sdhci_host *host = dev_id; + unsigned long flags; + u32 isr; + + spin_lock_irqsave(&host->lock, flags); + isr = host->thread_isr; + host->thread_isr = 0; + spin_unlock_irqrestore(&host->lock, flags); + + if (isr & (SDHCI_INT_CARD_INSERT | SDHCI_INT_CARD_REMOVE)) { + sdhci_card_event(host->mmc); + mmc_detect_change(host->mmc, msecs_to_jiffies(200)); + } + + if (isr & SDHCI_INT_CARD_INT) { + sdio_run_irqs(host->mmc); + + spin_lock_irqsave(&host->lock, flags); + if (host->flags & SDHCI_SDIO_IRQ_ENABLED) + sdhci_enable_sdio_irq_nolock(host, true); + spin_unlock_irqrestore(&host->lock, flags); + } + + return isr ? IRQ_HANDLED : IRQ_NONE; +} + /*****************************************************************************\ * * * Suspend/resume * @@ -2572,9 +2545,6 @@ EXPORT_SYMBOL_GPL(sdhci_disable_irq_wakeups); int sdhci_suspend_host(struct sdhci_host *host) { - if (host->ops->platform_suspend) - host->ops->platform_suspend(host); - sdhci_disable_card_detection(host); /* Disable tuning since we are suspending */ @@ -2584,7 +2554,9 @@ int sdhci_suspend_host(struct sdhci_host *host) } if (!device_may_wakeup(mmc_dev(host->mmc))) { - sdhci_mask_irqs(host, SDHCI_INT_ALL_MASK); + host->ier = 0; + sdhci_writel(host, 0, SDHCI_INT_ENABLE); + sdhci_writel(host, 0, SDHCI_SIGNAL_ENABLE); free_irq(host->irq, host); } else { sdhci_enable_irq_wakeups(host); @@ -2605,8 +2577,9 @@ int sdhci_resume_host(struct sdhci_host *host) } if (!device_may_wakeup(mmc_dev(host->mmc))) { - ret = request_irq(host->irq, sdhci_irq, IRQF_SHARED, - mmc_hostname(host->mmc), host); + ret = request_threaded_irq(host->irq, sdhci_irq, + sdhci_thread_irq, IRQF_SHARED, + mmc_hostname(host->mmc), host); if (ret) return ret; } else { @@ -2628,9 +2601,6 @@ int sdhci_resume_host(struct sdhci_host *host) sdhci_enable_card_detection(host); - if (host->ops->platform_resume) - host->ops->platform_resume(host); - /* Set the re-tuning expiration flag */ if (host->flags & SDHCI_USING_RETUNING_TIMER) host->flags |= SDHCI_NEEDS_RETUNING; @@ -2682,10 +2652,12 @@ int sdhci_runtime_suspend_host(struct sdhci_host *host) } spin_lock_irqsave(&host->lock, flags); - sdhci_mask_irqs(host, SDHCI_INT_ALL_MASK); + host->ier &= SDHCI_INT_CARD_INT; + sdhci_writel(host, host->ier, SDHCI_INT_ENABLE); + sdhci_writel(host, host->ier, SDHCI_SIGNAL_ENABLE); spin_unlock_irqrestore(&host->lock, flags); - synchronize_irq(host->irq); + synchronize_hardirq(host->irq); spin_lock_irqsave(&host->lock, flags); host->runtime_suspended = true; @@ -2729,7 +2701,7 @@ int sdhci_runtime_resume_host(struct sdhci_host *host) host->runtime_suspended = false; /* Enable SDIO IRQ */ - if ((host->flags & SDHCI_SDIO_IRQ_ENABLED)) + if (host->flags & SDHCI_SDIO_IRQ_ENABLED) sdhci_enable_sdio_irq_nolock(host, true); /* Enable Card Detection */ @@ -2788,7 +2760,7 @@ int sdhci_add_host(struct sdhci_host *host) if (debug_quirks2) host->quirks2 = debug_quirks2; - sdhci_reset(host, SDHCI_RESET_ALL); + sdhci_do_reset(host, SDHCI_RESET_ALL); host->version = sdhci_readw(host, SDHCI_HOST_VERSION); host->version = (host->version & SDHCI_SPEC_VER_MASK) @@ -2848,15 +2820,29 @@ int sdhci_add_host(struct sdhci_host *host) * (128) and potentially one alignment transfer for * each of those entries. */ - host->adma_desc = kmalloc((128 * 2 + 1) * 4, GFP_KERNEL); + host->adma_desc = dma_alloc_coherent(mmc_dev(host->mmc), + ADMA_SIZE, &host->adma_addr, + GFP_KERNEL); host->align_buffer = kmalloc(128 * 4, GFP_KERNEL); if (!host->adma_desc || !host->align_buffer) { - kfree(host->adma_desc); + dma_free_coherent(mmc_dev(host->mmc), ADMA_SIZE, + host->adma_desc, host->adma_addr); kfree(host->align_buffer); pr_warning("%s: Unable to allocate ADMA " "buffers. Falling back to standard DMA.\n", mmc_hostname(mmc)); host->flags &= ~SDHCI_USE_ADMA; + host->adma_desc = NULL; + host->align_buffer = NULL; + } else if (host->adma_addr & 3) { + pr_warning("%s: unable to allocate aligned ADMA descriptor\n", + mmc_hostname(mmc)); + host->flags &= ~SDHCI_USE_ADMA; + dma_free_coherent(mmc_dev(host->mmc), ADMA_SIZE, + host->adma_desc, host->adma_addr); + kfree(host->align_buffer); + host->adma_desc = NULL; + host->align_buffer = NULL; } } @@ -2941,6 +2927,7 @@ int sdhci_add_host(struct sdhci_host *host) mmc->max_busy_timeout = (1 << 27) / host->timeout_clk; mmc->caps |= MMC_CAP_SDIO_IRQ | MMC_CAP_ERASE | MMC_CAP_CMD23; + mmc->caps2 |= MMC_CAP2_SDIO_IRQ_NOTHREAD; if (host->quirks & SDHCI_QUIRK_MULTIBLOCK_READ_ACMD12) host->flags |= SDHCI_AUTO_CMD12; @@ -3212,8 +3199,6 @@ int sdhci_add_host(struct sdhci_host *host) /* * Init tasklets. */ - tasklet_init(&host->card_tasklet, - sdhci_tasklet_card, (unsigned long)host); tasklet_init(&host->finish_tasklet, sdhci_tasklet_finish, (unsigned long)host); @@ -3230,8 +3215,8 @@ int sdhci_add_host(struct sdhci_host *host) sdhci_init(host, 0); - ret = request_irq(host->irq, sdhci_irq, IRQF_SHARED, - mmc_hostname(mmc), host); + ret = request_threaded_irq(host->irq, sdhci_irq, sdhci_thread_irq, + IRQF_SHARED, mmc_hostname(mmc), host); if (ret) { pr_err("%s: Failed to request IRQ %d: %d\n", mmc_hostname(mmc), host->irq, ret); @@ -3273,12 +3258,12 @@ int sdhci_add_host(struct sdhci_host *host) #ifdef SDHCI_USE_LEDS_CLASS reset: - sdhci_reset(host, SDHCI_RESET_ALL); - sdhci_mask_irqs(host, SDHCI_INT_ALL_MASK); + sdhci_do_reset(host, SDHCI_RESET_ALL); + sdhci_writel(host, 0, SDHCI_INT_ENABLE); + sdhci_writel(host, 0, SDHCI_SIGNAL_ENABLE); free_irq(host->irq, host); #endif untasklet: - tasklet_kill(&host->card_tasklet); tasklet_kill(&host->finish_tasklet); return ret; @@ -3315,14 +3300,14 @@ void sdhci_remove_host(struct sdhci_host *host, int dead) #endif if (!dead) - sdhci_reset(host, SDHCI_RESET_ALL); + sdhci_do_reset(host, SDHCI_RESET_ALL); - sdhci_mask_irqs(host, SDHCI_INT_ALL_MASK); + sdhci_writel(host, 0, SDHCI_INT_ENABLE); + sdhci_writel(host, 0, SDHCI_SIGNAL_ENABLE); free_irq(host->irq, host); del_timer_sync(&host->timer); - tasklet_kill(&host->card_tasklet); tasklet_kill(&host->finish_tasklet); if (host->vmmc) { @@ -3335,7 +3320,9 @@ void sdhci_remove_host(struct sdhci_host *host, int dead) regulator_put(host->vqmmc); } - kfree(host->adma_desc); + if (host->adma_desc) + dma_free_coherent(mmc_dev(host->mmc), ADMA_SIZE, + host->adma_desc, host->adma_addr); kfree(host->align_buffer); host->adma_desc = NULL; diff --git a/drivers/mmc/host/sdhci.h b/drivers/mmc/host/sdhci.h index 0a3ed01887db..4a5cd5e3fa3e 100644 --- a/drivers/mmc/host/sdhci.h +++ b/drivers/mmc/host/sdhci.h @@ -281,18 +281,14 @@ struct sdhci_ops { unsigned int (*get_max_clock)(struct sdhci_host *host); unsigned int (*get_min_clock)(struct sdhci_host *host); unsigned int (*get_timeout_clock)(struct sdhci_host *host); - int (*platform_bus_width)(struct sdhci_host *host, - int width); + void (*set_bus_width)(struct sdhci_host *host, int width); void (*platform_send_init_74_clocks)(struct sdhci_host *host, u8 power_mode); unsigned int (*get_ro)(struct sdhci_host *host); - void (*platform_reset_enter)(struct sdhci_host *host, u8 mask); - void (*platform_reset_exit)(struct sdhci_host *host, u8 mask); + void (*reset)(struct sdhci_host *host, u8 mask); int (*platform_execute_tuning)(struct sdhci_host *host, u32 opcode); - int (*set_uhs_signaling)(struct sdhci_host *host, unsigned int uhs); + void (*set_uhs_signaling)(struct sdhci_host *host, unsigned int uhs); void (*hw_reset)(struct sdhci_host *host); - void (*platform_suspend)(struct sdhci_host *host); - void (*platform_resume)(struct sdhci_host *host); void (*adma_workaround)(struct sdhci_host *host, u32 intmask); void (*platform_init)(struct sdhci_host *host); void (*card_event)(struct sdhci_host *host); @@ -397,6 +393,16 @@ extern void sdhci_remove_host(struct sdhci_host *host, int dead); extern void sdhci_send_command(struct sdhci_host *host, struct mmc_command *cmd); +static inline bool sdhci_sdio_irq_enabled(struct sdhci_host *host) +{ + return !!(host->flags & SDHCI_SDIO_IRQ_ENABLED); +} + +void sdhci_set_clock(struct sdhci_host *host, unsigned int clock); +void sdhci_set_bus_width(struct sdhci_host *host, int width); +void sdhci_reset(struct sdhci_host *host, u8 mask); +void sdhci_set_uhs_signaling(struct sdhci_host *host, unsigned timing); + #ifdef CONFIG_PM extern int sdhci_suspend_host(struct sdhci_host *host); extern int sdhci_resume_host(struct sdhci_host *host); diff --git a/drivers/mmc/host/sh_mmcif.c b/drivers/mmc/host/sh_mmcif.c index 54730f4aac87..656fbba4c422 100644 --- a/drivers/mmc/host/sh_mmcif.c +++ b/drivers/mmc/host/sh_mmcif.c @@ -803,12 +803,13 @@ static u32 sh_mmcif_set_cmd(struct sh_mmcif_host *host, break; } switch (host->timing) { - case MMC_TIMING_UHS_DDR50: + case MMC_TIMING_MMC_DDR52: /* * MMC core will only set this timing, if the host - * advertises the MMC_CAP_UHS_DDR50 capability. MMCIF - * implementations with this capability, e.g. sh73a0, - * will have to set it in their platform data. + * advertises the MMC_CAP_1_8V_DDR/MMC_CAP_1_2V_DDR + * capability. MMCIF implementations with this + * capability, e.g. sh73a0, will have to set it + * in their platform data. */ tmp |= CMD_SET_DARS; break; diff --git a/drivers/mmc/host/usdhi6rol0.c b/drivers/mmc/host/usdhi6rol0.c new file mode 100644 index 000000000000..eb2bbbef19c6 --- /dev/null +++ b/drivers/mmc/host/usdhi6rol0.c @@ -0,0 +1,1847 @@ +/* + * Copyright (C) 2013-2014 Renesas Electronics Europe Ltd. + * Author: Guennadi Liakhovetski <g.liakhovetski@gmx.de> + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of version 2 of the GNU General Public License as + * published by the Free Software Foundation. + */ + +#include <linux/clk.h> +#include <linux/delay.h> +#include <linux/device.h> +#include <linux/dma-mapping.h> +#include <linux/dmaengine.h> +#include <linux/highmem.h> +#include <linux/interrupt.h> +#include <linux/io.h> +#include <linux/log2.h> +#include <linux/mmc/host.h> +#include <linux/mmc/mmc.h> +#include <linux/mmc/sd.h> +#include <linux/mmc/sdio.h> +#include <linux/module.h> +#include <linux/pagemap.h> +#include <linux/platform_device.h> +#include <linux/scatterlist.h> +#include <linux/string.h> +#include <linux/time.h> +#include <linux/virtio.h> +#include <linux/workqueue.h> + +#define USDHI6_SD_CMD 0x0000 +#define USDHI6_SD_PORT_SEL 0x0004 +#define USDHI6_SD_ARG 0x0008 +#define USDHI6_SD_STOP 0x0010 +#define USDHI6_SD_SECCNT 0x0014 +#define USDHI6_SD_RSP10 0x0018 +#define USDHI6_SD_RSP32 0x0020 +#define USDHI6_SD_RSP54 0x0028 +#define USDHI6_SD_RSP76 0x0030 +#define USDHI6_SD_INFO1 0x0038 +#define USDHI6_SD_INFO2 0x003c +#define USDHI6_SD_INFO1_MASK 0x0040 +#define USDHI6_SD_INFO2_MASK 0x0044 +#define USDHI6_SD_CLK_CTRL 0x0048 +#define USDHI6_SD_SIZE 0x004c +#define USDHI6_SD_OPTION 0x0050 +#define USDHI6_SD_ERR_STS1 0x0058 +#define USDHI6_SD_ERR_STS2 0x005c +#define USDHI6_SD_BUF0 0x0060 +#define USDHI6_SDIO_MODE 0x0068 +#define USDHI6_SDIO_INFO1 0x006c +#define USDHI6_SDIO_INFO1_MASK 0x0070 +#define USDHI6_CC_EXT_MODE 0x01b0 +#define USDHI6_SOFT_RST 0x01c0 +#define USDHI6_VERSION 0x01c4 +#define USDHI6_HOST_MODE 0x01c8 +#define USDHI6_SDIF_MODE 0x01cc + +#define USDHI6_SD_CMD_APP 0x0040 +#define USDHI6_SD_CMD_MODE_RSP_AUTO 0x0000 +#define USDHI6_SD_CMD_MODE_RSP_NONE 0x0300 +#define USDHI6_SD_CMD_MODE_RSP_R1 0x0400 /* Also R5, R6, R7 */ +#define USDHI6_SD_CMD_MODE_RSP_R1B 0x0500 /* R1b */ +#define USDHI6_SD_CMD_MODE_RSP_R2 0x0600 +#define USDHI6_SD_CMD_MODE_RSP_R3 0x0700 /* Also R4 */ +#define USDHI6_SD_CMD_DATA 0x0800 +#define USDHI6_SD_CMD_READ 0x1000 +#define USDHI6_SD_CMD_MULTI 0x2000 +#define USDHI6_SD_CMD_CMD12_AUTO_OFF 0x4000 + +#define USDHI6_CC_EXT_MODE_SDRW BIT(1) + +#define USDHI6_SD_INFO1_RSP_END BIT(0) +#define USDHI6_SD_INFO1_ACCESS_END BIT(2) +#define USDHI6_SD_INFO1_CARD_OUT BIT(3) +#define USDHI6_SD_INFO1_CARD_IN BIT(4) +#define USDHI6_SD_INFO1_CD BIT(5) +#define USDHI6_SD_INFO1_WP BIT(7) +#define USDHI6_SD_INFO1_D3_CARD_OUT BIT(8) +#define USDHI6_SD_INFO1_D3_CARD_IN BIT(9) + +#define USDHI6_SD_INFO2_CMD_ERR BIT(0) +#define USDHI6_SD_INFO2_CRC_ERR BIT(1) +#define USDHI6_SD_INFO2_END_ERR BIT(2) +#define USDHI6_SD_INFO2_TOUT BIT(3) +#define USDHI6_SD_INFO2_IWA_ERR BIT(4) +#define USDHI6_SD_INFO2_IRA_ERR BIT(5) +#define USDHI6_SD_INFO2_RSP_TOUT BIT(6) +#define USDHI6_SD_INFO2_SDDAT0 BIT(7) +#define USDHI6_SD_INFO2_BRE BIT(8) +#define USDHI6_SD_INFO2_BWE BIT(9) +#define USDHI6_SD_INFO2_SCLKDIVEN BIT(13) +#define USDHI6_SD_INFO2_CBSY BIT(14) +#define USDHI6_SD_INFO2_ILA BIT(15) + +#define USDHI6_SD_INFO1_CARD_INSERT (USDHI6_SD_INFO1_CARD_IN | USDHI6_SD_INFO1_D3_CARD_IN) +#define USDHI6_SD_INFO1_CARD_EJECT (USDHI6_SD_INFO1_CARD_OUT | USDHI6_SD_INFO1_D3_CARD_OUT) +#define USDHI6_SD_INFO1_CARD (USDHI6_SD_INFO1_CARD_INSERT | USDHI6_SD_INFO1_CARD_EJECT) +#define USDHI6_SD_INFO1_CARD_CD (USDHI6_SD_INFO1_CARD_IN | USDHI6_SD_INFO1_CARD_OUT) + +#define USDHI6_SD_INFO2_ERR (USDHI6_SD_INFO2_CMD_ERR | \ + USDHI6_SD_INFO2_CRC_ERR | USDHI6_SD_INFO2_END_ERR | \ + USDHI6_SD_INFO2_TOUT | USDHI6_SD_INFO2_IWA_ERR | \ + USDHI6_SD_INFO2_IRA_ERR | USDHI6_SD_INFO2_RSP_TOUT | \ + USDHI6_SD_INFO2_ILA) + +#define USDHI6_SD_INFO1_IRQ (USDHI6_SD_INFO1_RSP_END | USDHI6_SD_INFO1_ACCESS_END | \ + USDHI6_SD_INFO1_CARD) + +#define USDHI6_SD_INFO2_IRQ (USDHI6_SD_INFO2_ERR | USDHI6_SD_INFO2_BRE | \ + USDHI6_SD_INFO2_BWE | 0x0800 | USDHI6_SD_INFO2_ILA) + +#define USDHI6_SD_CLK_CTRL_SCLKEN BIT(8) + +#define USDHI6_SD_STOP_STP BIT(0) +#define USDHI6_SD_STOP_SEC BIT(8) + +#define USDHI6_SDIO_INFO1_IOIRQ BIT(0) +#define USDHI6_SDIO_INFO1_EXPUB52 BIT(14) +#define USDHI6_SDIO_INFO1_EXWT BIT(15) + +#define USDHI6_SD_ERR_STS1_CRC_NO_ERROR BIT(13) + +#define USDHI6_SOFT_RST_RESERVED (BIT(1) | BIT(2)) +#define USDHI6_SOFT_RST_RESET BIT(0) + +#define USDHI6_SD_OPTION_TIMEOUT_SHIFT 4 +#define USDHI6_SD_OPTION_TIMEOUT_MASK (0xf << USDHI6_SD_OPTION_TIMEOUT_SHIFT) +#define USDHI6_SD_OPTION_WIDTH_1 BIT(15) + +#define USDHI6_SD_PORT_SEL_PORTS_SHIFT 8 + +#define USDHI6_SD_CLK_CTRL_DIV_MASK 0xff + +#define USDHI6_SDIO_INFO1_IRQ (USDHI6_SDIO_INFO1_IOIRQ | 3 | \ + USDHI6_SDIO_INFO1_EXPUB52 | USDHI6_SDIO_INFO1_EXWT) + +#define USDHI6_MIN_DMA 64 + +enum usdhi6_wait_for { + USDHI6_WAIT_FOR_REQUEST, + USDHI6_WAIT_FOR_CMD, + USDHI6_WAIT_FOR_MREAD, + USDHI6_WAIT_FOR_MWRITE, + USDHI6_WAIT_FOR_READ, + USDHI6_WAIT_FOR_WRITE, + USDHI6_WAIT_FOR_DATA_END, + USDHI6_WAIT_FOR_STOP, + USDHI6_WAIT_FOR_DMA, +}; + +struct usdhi6_page { + struct page *page; + void *mapped; /* mapped page */ +}; + +struct usdhi6_host { + struct mmc_host *mmc; + struct mmc_request *mrq; + void __iomem *base; + struct clk *clk; + + /* SG memory handling */ + + /* Common for multiple and single block requests */ + struct usdhi6_page pg; /* current page from an SG */ + void *blk_page; /* either a mapped page, or the bounce buffer */ + size_t offset; /* offset within a page, including sg->offset */ + + /* Blocks, crossing a page boundary */ + size_t head_len; + struct usdhi6_page head_pg; + + /* A bounce buffer for unaligned blocks or blocks, crossing a page boundary */ + struct scatterlist bounce_sg; + u8 bounce_buf[512]; + + /* Multiple block requests only */ + struct scatterlist *sg; /* current SG segment */ + int page_idx; /* page index within an SG segment */ + + enum usdhi6_wait_for wait; + u32 status_mask; + u32 status2_mask; + u32 sdio_mask; + u32 io_error; + u32 irq_status; + unsigned long imclk; + unsigned long rate; + bool app_cmd; + + /* Timeout handling */ + struct delayed_work timeout_work; + unsigned long timeout; + + /* DMA support */ + struct dma_chan *chan_rx; + struct dma_chan *chan_tx; + bool dma_active; +}; + +/* I/O primitives */ + +static void usdhi6_write(struct usdhi6_host *host, u32 reg, u32 data) +{ + iowrite32(data, host->base + reg); + dev_vdbg(mmc_dev(host->mmc), "%s(0x%p + 0x%x) = 0x%x\n", __func__, + host->base, reg, data); +} + +static void usdhi6_write16(struct usdhi6_host *host, u32 reg, u16 data) +{ + iowrite16(data, host->base + reg); + dev_vdbg(mmc_dev(host->mmc), "%s(0x%p + 0x%x) = 0x%x\n", __func__, + host->base, reg, data); +} + +static u32 usdhi6_read(struct usdhi6_host *host, u32 reg) +{ + u32 data = ioread32(host->base + reg); + dev_vdbg(mmc_dev(host->mmc), "%s(0x%p + 0x%x) = 0x%x\n", __func__, + host->base, reg, data); + return data; +} + +static u16 usdhi6_read16(struct usdhi6_host *host, u32 reg) +{ + u16 data = ioread16(host->base + reg); + dev_vdbg(mmc_dev(host->mmc), "%s(0x%p + 0x%x) = 0x%x\n", __func__, + host->base, reg, data); + return data; +} + +static void usdhi6_irq_enable(struct usdhi6_host *host, u32 info1, u32 info2) +{ + host->status_mask = USDHI6_SD_INFO1_IRQ & ~info1; + host->status2_mask = USDHI6_SD_INFO2_IRQ & ~info2; + usdhi6_write(host, USDHI6_SD_INFO1_MASK, host->status_mask); + usdhi6_write(host, USDHI6_SD_INFO2_MASK, host->status2_mask); +} + +static void usdhi6_wait_for_resp(struct usdhi6_host *host) +{ + usdhi6_irq_enable(host, USDHI6_SD_INFO1_RSP_END | + USDHI6_SD_INFO1_ACCESS_END | USDHI6_SD_INFO1_CARD_CD, + USDHI6_SD_INFO2_ERR); +} + +static void usdhi6_wait_for_brwe(struct usdhi6_host *host, bool read) +{ + usdhi6_irq_enable(host, USDHI6_SD_INFO1_ACCESS_END | + USDHI6_SD_INFO1_CARD_CD, USDHI6_SD_INFO2_ERR | + (read ? USDHI6_SD_INFO2_BRE : USDHI6_SD_INFO2_BWE)); +} + +static void usdhi6_only_cd(struct usdhi6_host *host) +{ + /* Mask all except card hotplug */ + usdhi6_irq_enable(host, USDHI6_SD_INFO1_CARD_CD, 0); +} + +static void usdhi6_mask_all(struct usdhi6_host *host) +{ + usdhi6_irq_enable(host, 0, 0); +} + +static int usdhi6_error_code(struct usdhi6_host *host) +{ + u32 err; + + usdhi6_write(host, USDHI6_SD_STOP, USDHI6_SD_STOP_STP); + + if (host->io_error & + (USDHI6_SD_INFO2_RSP_TOUT | USDHI6_SD_INFO2_TOUT)) { + u32 rsp54 = usdhi6_read(host, USDHI6_SD_RSP54); + int opc = host->mrq ? host->mrq->cmd->opcode : -1; + + err = usdhi6_read(host, USDHI6_SD_ERR_STS2); + /* Response timeout is often normal, don't spam the log */ + if (host->wait == USDHI6_WAIT_FOR_CMD) + dev_dbg(mmc_dev(host->mmc), + "T-out sts 0x%x, resp 0x%x, state %u, CMD%d\n", + err, rsp54, host->wait, opc); + else + dev_warn(mmc_dev(host->mmc), + "T-out sts 0x%x, resp 0x%x, state %u, CMD%d\n", + err, rsp54, host->wait, opc); + return -ETIMEDOUT; + } + + err = usdhi6_read(host, USDHI6_SD_ERR_STS1); + if (err != USDHI6_SD_ERR_STS1_CRC_NO_ERROR) + dev_warn(mmc_dev(host->mmc), "Err sts 0x%x, state %u, CMD%d\n", + err, host->wait, host->mrq ? host->mrq->cmd->opcode : -1); + if (host->io_error & USDHI6_SD_INFO2_ILA) + return -EILSEQ; + + return -EIO; +} + +/* Scatter-Gather management */ + +/* + * In PIO mode we have to map each page separately, using kmap(). That way + * adjacent pages are mapped to non-adjacent virtual addresses. That's why we + * have to use a bounce buffer for blocks, crossing page boundaries. Such blocks + * have been observed with an SDIO WiFi card (b43 driver). + */ +static void usdhi6_blk_bounce(struct usdhi6_host *host, + struct scatterlist *sg) +{ + struct mmc_data *data = host->mrq->data; + size_t blk_head = host->head_len; + + dev_dbg(mmc_dev(host->mmc), "%s(): CMD%u of %u SG: %ux%u @ 0x%x\n", + __func__, host->mrq->cmd->opcode, data->sg_len, + data->blksz, data->blocks, sg->offset); + + host->head_pg.page = host->pg.page; + host->head_pg.mapped = host->pg.mapped; + host->pg.page = nth_page(host->pg.page, 1); + host->pg.mapped = kmap(host->pg.page); + + host->blk_page = host->bounce_buf; + host->offset = 0; + + if (data->flags & MMC_DATA_READ) + return; + + memcpy(host->bounce_buf, host->head_pg.mapped + PAGE_SIZE - blk_head, + blk_head); + memcpy(host->bounce_buf + blk_head, host->pg.mapped, + data->blksz - blk_head); +} + +/* Only called for multiple block IO */ +static void usdhi6_sg_prep(struct usdhi6_host *host) +{ + struct mmc_request *mrq = host->mrq; + struct mmc_data *data = mrq->data; + + usdhi6_write(host, USDHI6_SD_SECCNT, data->blocks); + + host->sg = data->sg; + /* TODO: if we always map, this is redundant */ + host->offset = host->sg->offset; +} + +/* Map the first page in an SG segment: common for multiple and single block IO */ +static void *usdhi6_sg_map(struct usdhi6_host *host) +{ + struct mmc_data *data = host->mrq->data; + struct scatterlist *sg = data->sg_len > 1 ? host->sg : data->sg; + size_t head = PAGE_SIZE - sg->offset; + size_t blk_head = head % data->blksz; + + WARN(host->pg.page, "%p not properly unmapped!\n", host->pg.page); + if (WARN(sg_dma_len(sg) % data->blksz, + "SG size %zd isn't a multiple of block size %zd\n", + sg_dma_len(sg), data->blksz)) + return NULL; + + host->pg.page = sg_page(sg); + host->pg.mapped = kmap(host->pg.page); + host->offset = sg->offset; + + /* + * Block size must be a power of 2 for multi-block transfers, + * therefore blk_head is equal for all pages in this SG + */ + host->head_len = blk_head; + + if (head < data->blksz) + /* + * The first block in the SG crosses a page boundary. + * Max blksz = 512, so blocks can only span 2 pages + */ + usdhi6_blk_bounce(host, sg); + else + host->blk_page = host->pg.mapped; + + dev_dbg(mmc_dev(host->mmc), "Mapped %p (%lx) at %p + %u for CMD%u @ 0x%p\n", + host->pg.page, page_to_pfn(host->pg.page), host->pg.mapped, + sg->offset, host->mrq->cmd->opcode, host->mrq); + + return host->blk_page + host->offset; +} + +/* Unmap the current page: common for multiple and single block IO */ +static void usdhi6_sg_unmap(struct usdhi6_host *host, bool force) +{ + struct mmc_data *data = host->mrq->data; + struct page *page = host->head_pg.page; + + if (page) { + /* Previous block was cross-page boundary */ + struct scatterlist *sg = data->sg_len > 1 ? + host->sg : data->sg; + size_t blk_head = host->head_len; + + if (!data->error && data->flags & MMC_DATA_READ) { + memcpy(host->head_pg.mapped + PAGE_SIZE - blk_head, + host->bounce_buf, blk_head); + memcpy(host->pg.mapped, host->bounce_buf + blk_head, + data->blksz - blk_head); + } + + flush_dcache_page(page); + kunmap(page); + + host->head_pg.page = NULL; + + if (!force && sg_dma_len(sg) + sg->offset > + (host->page_idx << PAGE_SHIFT) + data->blksz - blk_head) + /* More blocks in this SG, don't unmap the next page */ + return; + } + + page = host->pg.page; + if (!page) + return; + + flush_dcache_page(page); + kunmap(page); + + host->pg.page = NULL; +} + +/* Called from MMC_WRITE_MULTIPLE_BLOCK or MMC_READ_MULTIPLE_BLOCK */ +static void usdhi6_sg_advance(struct usdhi6_host *host) +{ + struct mmc_data *data = host->mrq->data; + size_t done, total; + + /* New offset: set at the end of the previous block */ + if (host->head_pg.page) { + /* Finished a cross-page block, jump to the new page */ + host->page_idx++; + host->offset = data->blksz - host->head_len; + host->blk_page = host->pg.mapped; + usdhi6_sg_unmap(host, false); + } else { + host->offset += data->blksz; + /* The completed block didn't cross a page boundary */ + if (host->offset == PAGE_SIZE) { + /* If required, we'll map the page below */ + host->offset = 0; + host->page_idx++; + } + } + + /* + * Now host->blk_page + host->offset point at the end of our last block + * and host->page_idx is the index of the page, in which our new block + * is located, if any + */ + + done = (host->page_idx << PAGE_SHIFT) + host->offset; + total = host->sg->offset + sg_dma_len(host->sg); + + dev_dbg(mmc_dev(host->mmc), "%s(): %zu of %zu @ %u\n", __func__, + done, total, host->offset); + + if (done < total && host->offset) { + /* More blocks in this page */ + if (host->offset + data->blksz > PAGE_SIZE) + /* We approached at a block, that spans 2 pages */ + usdhi6_blk_bounce(host, host->sg); + + return; + } + + /* Finished current page or an SG segment */ + usdhi6_sg_unmap(host, false); + + if (done == total) { + /* + * End of an SG segment or the complete SG: jump to the next + * segment, we'll map it later in usdhi6_blk_read() or + * usdhi6_blk_write() + */ + struct scatterlist *next = sg_next(host->sg); + + host->page_idx = 0; + + if (!next) + host->wait = USDHI6_WAIT_FOR_DATA_END; + host->sg = next; + + if (WARN(next && sg_dma_len(next) % data->blksz, + "SG size %zd isn't a multiple of block size %zd\n", + sg_dma_len(next), data->blksz)) + data->error = -EINVAL; + + return; + } + + /* We cannot get here after crossing a page border */ + + /* Next page in the same SG */ + host->pg.page = nth_page(sg_page(host->sg), host->page_idx); + host->pg.mapped = kmap(host->pg.page); + host->blk_page = host->pg.mapped; + + dev_dbg(mmc_dev(host->mmc), "Mapped %p (%lx) at %p for CMD%u @ 0x%p\n", + host->pg.page, page_to_pfn(host->pg.page), host->pg.mapped, + host->mrq->cmd->opcode, host->mrq); +} + +/* DMA handling */ + +static void usdhi6_dma_release(struct usdhi6_host *host) +{ + host->dma_active = false; + if (host->chan_tx) { + struct dma_chan *chan = host->chan_tx; + host->chan_tx = NULL; + dma_release_channel(chan); + } + if (host->chan_rx) { + struct dma_chan *chan = host->chan_rx; + host->chan_rx = NULL; + dma_release_channel(chan); + } +} + +static void usdhi6_dma_stop_unmap(struct usdhi6_host *host) +{ + struct mmc_data *data = host->mrq->data; + + if (!host->dma_active) + return; + + usdhi6_write(host, USDHI6_CC_EXT_MODE, 0); + host->dma_active = false; + + if (data->flags & MMC_DATA_READ) + dma_unmap_sg(host->chan_rx->device->dev, data->sg, + data->sg_len, DMA_FROM_DEVICE); + else + dma_unmap_sg(host->chan_tx->device->dev, data->sg, + data->sg_len, DMA_TO_DEVICE); +} + +static void usdhi6_dma_complete(void *arg) +{ + struct usdhi6_host *host = arg; + struct mmc_request *mrq = host->mrq; + + if (WARN(!mrq || !mrq->data, "%s: NULL data in DMA completion for %p!\n", + dev_name(mmc_dev(host->mmc)), mrq)) + return; + + dev_dbg(mmc_dev(host->mmc), "%s(): CMD%u DMA completed\n", __func__, + mrq->cmd->opcode); + + usdhi6_dma_stop_unmap(host); + usdhi6_wait_for_brwe(host, mrq->data->flags & MMC_DATA_READ); +} + +static int usdhi6_dma_setup(struct usdhi6_host *host, struct dma_chan *chan, + enum dma_transfer_direction dir) +{ + struct mmc_data *data = host->mrq->data; + struct scatterlist *sg = data->sg; + struct dma_async_tx_descriptor *desc = NULL; + dma_cookie_t cookie = -EINVAL; + enum dma_data_direction data_dir; + int ret; + + switch (dir) { + case DMA_MEM_TO_DEV: + data_dir = DMA_TO_DEVICE; + break; + case DMA_DEV_TO_MEM: + data_dir = DMA_FROM_DEVICE; + break; + default: + return -EINVAL; + } + + ret = dma_map_sg(chan->device->dev, sg, data->sg_len, data_dir); + if (ret > 0) { + host->dma_active = true; + desc = dmaengine_prep_slave_sg(chan, sg, ret, dir, + DMA_PREP_INTERRUPT | DMA_CTRL_ACK); + } + + if (desc) { + desc->callback = usdhi6_dma_complete; + desc->callback_param = host; + cookie = dmaengine_submit(desc); + } + + dev_dbg(mmc_dev(host->mmc), "%s(): mapped %d -> %d, cookie %d @ %p\n", + __func__, data->sg_len, ret, cookie, desc); + + if (cookie < 0) { + /* DMA failed, fall back to PIO */ + if (ret >= 0) + ret = cookie; + usdhi6_dma_release(host); + dev_warn(mmc_dev(host->mmc), + "DMA failed: %d, falling back to PIO\n", ret); + } + + return cookie; +} + +static int usdhi6_dma_start(struct usdhi6_host *host) +{ + if (!host->chan_rx || !host->chan_tx) + return -ENODEV; + + if (host->mrq->data->flags & MMC_DATA_READ) + return usdhi6_dma_setup(host, host->chan_rx, DMA_DEV_TO_MEM); + + return usdhi6_dma_setup(host, host->chan_tx, DMA_MEM_TO_DEV); +} + +static void usdhi6_dma_kill(struct usdhi6_host *host) +{ + struct mmc_data *data = host->mrq->data; + + dev_dbg(mmc_dev(host->mmc), "%s(): SG of %u: %ux%u\n", + __func__, data->sg_len, data->blocks, data->blksz); + /* Abort DMA */ + if (data->flags & MMC_DATA_READ) + dmaengine_terminate_all(host->chan_rx); + else + dmaengine_terminate_all(host->chan_tx); +} + +static void usdhi6_dma_check_error(struct usdhi6_host *host) +{ + struct mmc_data *data = host->mrq->data; + + dev_dbg(mmc_dev(host->mmc), "%s(): IO error %d, status 0x%x\n", + __func__, host->io_error, usdhi6_read(host, USDHI6_SD_INFO1)); + + if (host->io_error) { + data->error = usdhi6_error_code(host); + data->bytes_xfered = 0; + usdhi6_dma_kill(host); + usdhi6_dma_release(host); + dev_warn(mmc_dev(host->mmc), + "DMA failed: %d, falling back to PIO\n", data->error); + return; + } + + /* + * The datasheet tells us to check a response from the card, whereas + * responses only come after the command phase, not after the data + * phase. Let's check anyway. + */ + if (host->irq_status & USDHI6_SD_INFO1_RSP_END) + dev_warn(mmc_dev(host->mmc), "Unexpected response received!\n"); +} + +static void usdhi6_dma_kick(struct usdhi6_host *host) +{ + if (host->mrq->data->flags & MMC_DATA_READ) + dma_async_issue_pending(host->chan_rx); + else + dma_async_issue_pending(host->chan_tx); +} + +static void usdhi6_dma_request(struct usdhi6_host *host, phys_addr_t start) +{ + struct dma_slave_config cfg = { + .src_addr_width = DMA_SLAVE_BUSWIDTH_4_BYTES, + .dst_addr_width = DMA_SLAVE_BUSWIDTH_4_BYTES, + }; + int ret; + + host->chan_tx = dma_request_slave_channel(mmc_dev(host->mmc), "tx"); + dev_dbg(mmc_dev(host->mmc), "%s: TX: got channel %p\n", __func__, + host->chan_tx); + + if (!host->chan_tx) + return; + + cfg.direction = DMA_MEM_TO_DEV; + cfg.dst_addr = start + USDHI6_SD_BUF0; + cfg.dst_maxburst = 128; /* 128 words * 4 bytes = 512 bytes */ + cfg.src_addr = 0; + ret = dmaengine_slave_config(host->chan_tx, &cfg); + if (ret < 0) + goto e_release_tx; + + host->chan_rx = dma_request_slave_channel(mmc_dev(host->mmc), "rx"); + dev_dbg(mmc_dev(host->mmc), "%s: RX: got channel %p\n", __func__, + host->chan_rx); + + if (!host->chan_rx) + goto e_release_tx; + + cfg.direction = DMA_DEV_TO_MEM; + cfg.src_addr = cfg.dst_addr; + cfg.src_maxburst = 128; /* 128 words * 4 bytes = 512 bytes */ + cfg.dst_addr = 0; + ret = dmaengine_slave_config(host->chan_rx, &cfg); + if (ret < 0) + goto e_release_rx; + + return; + +e_release_rx: + dma_release_channel(host->chan_rx); + host->chan_rx = NULL; +e_release_tx: + dma_release_channel(host->chan_tx); + host->chan_tx = NULL; +} + +/* API helpers */ + +static void usdhi6_clk_set(struct usdhi6_host *host, struct mmc_ios *ios) +{ + unsigned long rate = ios->clock; + u32 val; + unsigned int i; + + for (i = 1000; i; i--) { + if (usdhi6_read(host, USDHI6_SD_INFO2) & USDHI6_SD_INFO2_SCLKDIVEN) + break; + usleep_range(10, 100); + } + + if (!i) { + dev_err(mmc_dev(host->mmc), "SD bus busy, clock set aborted\n"); + return; + } + + val = usdhi6_read(host, USDHI6_SD_CLK_CTRL) & ~USDHI6_SD_CLK_CTRL_DIV_MASK; + + if (rate) { + unsigned long new_rate; + + if (host->imclk <= rate) { + if (ios->timing != MMC_TIMING_UHS_DDR50) { + /* Cannot have 1-to-1 clock in DDR mode */ + new_rate = host->imclk; + val |= 0xff; + } else { + new_rate = host->imclk / 2; + } + } else { + unsigned long div = + roundup_pow_of_two(DIV_ROUND_UP(host->imclk, rate)); + val |= div >> 2; + new_rate = host->imclk / div; + } + + if (host->rate == new_rate) + return; + + host->rate = new_rate; + + dev_dbg(mmc_dev(host->mmc), "target %lu, div %u, set %lu\n", + rate, (val & 0xff) << 2, new_rate); + } + + /* + * if old or new rate is equal to input rate, have to switch the clock + * off before changing and on after + */ + if (host->imclk == rate || host->imclk == host->rate || !rate) + usdhi6_write(host, USDHI6_SD_CLK_CTRL, + val & ~USDHI6_SD_CLK_CTRL_SCLKEN); + + if (!rate) { + host->rate = 0; + return; + } + + usdhi6_write(host, USDHI6_SD_CLK_CTRL, val); + + if (host->imclk == rate || host->imclk == host->rate || + !(val & USDHI6_SD_CLK_CTRL_SCLKEN)) + usdhi6_write(host, USDHI6_SD_CLK_CTRL, + val | USDHI6_SD_CLK_CTRL_SCLKEN); +} + +static void usdhi6_set_power(struct usdhi6_host *host, struct mmc_ios *ios) +{ + struct mmc_host *mmc = host->mmc; + + if (!IS_ERR(mmc->supply.vmmc)) + /* Errors ignored... */ + mmc_regulator_set_ocr(mmc, mmc->supply.vmmc, + ios->power_mode ? ios->vdd : 0); +} + +static int usdhi6_reset(struct usdhi6_host *host) +{ + int i; + + usdhi6_write(host, USDHI6_SOFT_RST, USDHI6_SOFT_RST_RESERVED); + cpu_relax(); + usdhi6_write(host, USDHI6_SOFT_RST, USDHI6_SOFT_RST_RESERVED | USDHI6_SOFT_RST_RESET); + for (i = 1000; i; i--) + if (usdhi6_read(host, USDHI6_SOFT_RST) & USDHI6_SOFT_RST_RESET) + break; + + return i ? 0 : -ETIMEDOUT; +} + +static void usdhi6_set_ios(struct mmc_host *mmc, struct mmc_ios *ios) +{ + struct usdhi6_host *host = mmc_priv(mmc); + u32 option, mode; + int ret; + + dev_dbg(mmc_dev(mmc), "%uHz, OCR: %u, power %u, bus-width %u, timing %u\n", + ios->clock, ios->vdd, ios->power_mode, ios->bus_width, ios->timing); + + switch (ios->power_mode) { + case MMC_POWER_OFF: + usdhi6_set_power(host, ios); + usdhi6_only_cd(host); + break; + case MMC_POWER_UP: + /* + * We only also touch USDHI6_SD_OPTION from .request(), which + * cannot race with MMC_POWER_UP + */ + ret = usdhi6_reset(host); + if (ret < 0) { + dev_err(mmc_dev(mmc), "Cannot reset the interface!\n"); + } else { + usdhi6_set_power(host, ios); + usdhi6_only_cd(host); + } + break; + case MMC_POWER_ON: + option = usdhi6_read(host, USDHI6_SD_OPTION); + /* + * The eMMC standard only allows 4 or 8 bits in the DDR mode, + * the same probably holds for SD cards. We check here anyway, + * since the datasheet explicitly requires 4 bits for DDR. + */ + if (ios->bus_width == MMC_BUS_WIDTH_1) { + if (ios->timing == MMC_TIMING_UHS_DDR50) + dev_err(mmc_dev(mmc), + "4 bits are required for DDR\n"); + option |= USDHI6_SD_OPTION_WIDTH_1; + mode = 0; + } else { + option &= ~USDHI6_SD_OPTION_WIDTH_1; + mode = ios->timing == MMC_TIMING_UHS_DDR50; + } + usdhi6_write(host, USDHI6_SD_OPTION, option); + usdhi6_write(host, USDHI6_SDIF_MODE, mode); + break; + } + + if (host->rate != ios->clock) + usdhi6_clk_set(host, ios); +} + +/* This is data timeout. Response timeout is fixed to 640 clock cycles */ +static void usdhi6_timeout_set(struct usdhi6_host *host) +{ + struct mmc_request *mrq = host->mrq; + u32 val; + unsigned long ticks; + + if (!mrq->data) + ticks = host->rate / 1000 * mrq->cmd->busy_timeout; + else + ticks = host->rate / 1000000 * (mrq->data->timeout_ns / 1000) + + mrq->data->timeout_clks; + + if (!ticks || ticks > 1 << 27) + /* Max timeout */ + val = 14; + else if (ticks < 1 << 13) + /* Min timeout */ + val = 0; + else + val = order_base_2(ticks) - 13; + + dev_dbg(mmc_dev(host->mmc), "Set %s timeout %lu ticks @ %lu Hz\n", + mrq->data ? "data" : "cmd", ticks, host->rate); + + /* Timeout Counter mask: 0xf0 */ + usdhi6_write(host, USDHI6_SD_OPTION, (val << USDHI6_SD_OPTION_TIMEOUT_SHIFT) | + (usdhi6_read(host, USDHI6_SD_OPTION) & ~USDHI6_SD_OPTION_TIMEOUT_MASK)); +} + +static void usdhi6_request_done(struct usdhi6_host *host) +{ + struct mmc_request *mrq = host->mrq; + struct mmc_data *data = mrq->data; + + if (WARN(host->pg.page || host->head_pg.page, + "Page %p or %p not unmapped: wait %u, CMD%d(%c) @ +0x%x %ux%u in SG%u!\n", + host->pg.page, host->head_pg.page, host->wait, mrq->cmd->opcode, + data ? (data->flags & MMC_DATA_READ ? 'R' : 'W') : '-', + data ? host->offset : 0, data ? data->blocks : 0, + data ? data->blksz : 0, data ? data->sg_len : 0)) + usdhi6_sg_unmap(host, true); + + if (mrq->cmd->error || + (data && data->error) || + (mrq->stop && mrq->stop->error)) + dev_dbg(mmc_dev(host->mmc), "%s(CMD%d: %ux%u): err %d %d %d\n", + __func__, mrq->cmd->opcode, data ? data->blocks : 0, + data ? data->blksz : 0, + mrq->cmd->error, + data ? data->error : 1, + mrq->stop ? mrq->stop->error : 1); + + /* Disable DMA */ + usdhi6_write(host, USDHI6_CC_EXT_MODE, 0); + host->wait = USDHI6_WAIT_FOR_REQUEST; + host->mrq = NULL; + + mmc_request_done(host->mmc, mrq); +} + +static int usdhi6_cmd_flags(struct usdhi6_host *host) +{ + struct mmc_request *mrq = host->mrq; + struct mmc_command *cmd = mrq->cmd; + u16 opc = cmd->opcode; + + if (host->app_cmd) { + host->app_cmd = false; + opc |= USDHI6_SD_CMD_APP; + } + + if (mrq->data) { + opc |= USDHI6_SD_CMD_DATA; + + if (mrq->data->flags & MMC_DATA_READ) + opc |= USDHI6_SD_CMD_READ; + + if (cmd->opcode == MMC_READ_MULTIPLE_BLOCK || + cmd->opcode == MMC_WRITE_MULTIPLE_BLOCK || + (cmd->opcode == SD_IO_RW_EXTENDED && + mrq->data->blocks > 1)) { + opc |= USDHI6_SD_CMD_MULTI; + if (!mrq->stop) + opc |= USDHI6_SD_CMD_CMD12_AUTO_OFF; + } + + switch (mmc_resp_type(cmd)) { + case MMC_RSP_NONE: + opc |= USDHI6_SD_CMD_MODE_RSP_NONE; + break; + case MMC_RSP_R1: + opc |= USDHI6_SD_CMD_MODE_RSP_R1; + break; + case MMC_RSP_R1B: + opc |= USDHI6_SD_CMD_MODE_RSP_R1B; + break; + case MMC_RSP_R2: + opc |= USDHI6_SD_CMD_MODE_RSP_R2; + break; + case MMC_RSP_R3: + opc |= USDHI6_SD_CMD_MODE_RSP_R3; + break; + default: + dev_warn(mmc_dev(host->mmc), + "Unknown response type %d\n", + mmc_resp_type(cmd)); + return -EINVAL; + } + } + + return opc; +} + +static int usdhi6_rq_start(struct usdhi6_host *host) +{ + struct mmc_request *mrq = host->mrq; + struct mmc_command *cmd = mrq->cmd; + struct mmc_data *data = mrq->data; + int opc = usdhi6_cmd_flags(host); + int i; + + if (opc < 0) + return opc; + + for (i = 1000; i; i--) { + if (!(usdhi6_read(host, USDHI6_SD_INFO2) & USDHI6_SD_INFO2_CBSY)) + break; + usleep_range(10, 100); + } + + if (!i) { + dev_dbg(mmc_dev(host->mmc), "Command active, request aborted\n"); + return -EAGAIN; + } + + if (data) { + bool use_dma; + int ret = 0; + + host->page_idx = 0; + + if (cmd->opcode == SD_IO_RW_EXTENDED && data->blocks > 1) { + switch (data->blksz) { + case 512: + break; + case 32: + case 64: + case 128: + case 256: + if (mrq->stop) + ret = -EINVAL; + break; + default: + ret = -EINVAL; + } + } else if ((cmd->opcode == MMC_READ_MULTIPLE_BLOCK || + cmd->opcode == MMC_WRITE_MULTIPLE_BLOCK) && + data->blksz != 512) { + ret = -EINVAL; + } + + if (ret < 0) { + dev_warn(mmc_dev(host->mmc), "%s(): %u blocks of %u bytes\n", + __func__, data->blocks, data->blksz); + return -EINVAL; + } + + if (cmd->opcode == MMC_READ_MULTIPLE_BLOCK || + cmd->opcode == MMC_WRITE_MULTIPLE_BLOCK || + (cmd->opcode == SD_IO_RW_EXTENDED && + data->blocks > 1)) + usdhi6_sg_prep(host); + + usdhi6_write(host, USDHI6_SD_SIZE, data->blksz); + + if ((data->blksz >= USDHI6_MIN_DMA || + data->blocks > 1) && + (data->blksz % 4 || + data->sg->offset % 4)) + dev_dbg(mmc_dev(host->mmc), + "Bad SG of %u: %ux%u @ %u\n", data->sg_len, + data->blksz, data->blocks, data->sg->offset); + + /* Enable DMA for USDHI6_MIN_DMA bytes or more */ + use_dma = data->blksz >= USDHI6_MIN_DMA && + !(data->blksz % 4) && + usdhi6_dma_start(host) >= DMA_MIN_COOKIE; + + if (use_dma) + usdhi6_write(host, USDHI6_CC_EXT_MODE, USDHI6_CC_EXT_MODE_SDRW); + + dev_dbg(mmc_dev(host->mmc), + "%s(): request opcode %u, %u blocks of %u bytes in %u segments, %s %s @+0x%x%s\n", + __func__, cmd->opcode, data->blocks, data->blksz, + data->sg_len, use_dma ? "DMA" : "PIO", + data->flags & MMC_DATA_READ ? "read" : "write", + data->sg->offset, mrq->stop ? " + stop" : ""); + } else { + dev_dbg(mmc_dev(host->mmc), "%s(): request opcode %u\n", + __func__, cmd->opcode); + } + + /* We have to get a command completion interrupt with DMA too */ + usdhi6_wait_for_resp(host); + + host->wait = USDHI6_WAIT_FOR_CMD; + schedule_delayed_work(&host->timeout_work, host->timeout); + + /* SEC bit is required to enable block counting by the core */ + usdhi6_write(host, USDHI6_SD_STOP, + data && data->blocks > 1 ? USDHI6_SD_STOP_SEC : 0); + usdhi6_write(host, USDHI6_SD_ARG, cmd->arg); + + /* Kick command execution */ + usdhi6_write(host, USDHI6_SD_CMD, opc); + + return 0; +} + +static void usdhi6_request(struct mmc_host *mmc, struct mmc_request *mrq) +{ + struct usdhi6_host *host = mmc_priv(mmc); + int ret; + + cancel_delayed_work_sync(&host->timeout_work); + + host->mrq = mrq; + host->sg = NULL; + + usdhi6_timeout_set(host); + ret = usdhi6_rq_start(host); + if (ret < 0) { + mrq->cmd->error = ret; + usdhi6_request_done(host); + } +} + +static int usdhi6_get_cd(struct mmc_host *mmc) +{ + struct usdhi6_host *host = mmc_priv(mmc); + /* Read is atomic, no need to lock */ + u32 status = usdhi6_read(host, USDHI6_SD_INFO1) & USDHI6_SD_INFO1_CD; + +/* + * level status.CD CD_ACTIVE_HIGH card present + * 1 0 0 0 + * 1 0 1 1 + * 0 1 0 1 + * 0 1 1 0 + */ + return !status ^ !(mmc->caps2 & MMC_CAP2_CD_ACTIVE_HIGH); +} + +static int usdhi6_get_ro(struct mmc_host *mmc) +{ + struct usdhi6_host *host = mmc_priv(mmc); + /* No locking as above */ + u32 status = usdhi6_read(host, USDHI6_SD_INFO1) & USDHI6_SD_INFO1_WP; + +/* + * level status.WP RO_ACTIVE_HIGH card read-only + * 1 0 0 0 + * 1 0 1 1 + * 0 1 0 1 + * 0 1 1 0 + */ + return !status ^ !(mmc->caps2 & MMC_CAP2_RO_ACTIVE_HIGH); +} + +static void usdhi6_enable_sdio_irq(struct mmc_host *mmc, int enable) +{ + struct usdhi6_host *host = mmc_priv(mmc); + + dev_dbg(mmc_dev(mmc), "%s(): %sable\n", __func__, enable ? "en" : "dis"); + + if (enable) { + host->sdio_mask = USDHI6_SDIO_INFO1_IRQ & ~USDHI6_SDIO_INFO1_IOIRQ; + usdhi6_write(host, USDHI6_SDIO_INFO1_MASK, host->sdio_mask); + usdhi6_write(host, USDHI6_SDIO_MODE, 1); + } else { + usdhi6_write(host, USDHI6_SDIO_MODE, 0); + usdhi6_write(host, USDHI6_SDIO_INFO1_MASK, USDHI6_SDIO_INFO1_IRQ); + host->sdio_mask = USDHI6_SDIO_INFO1_IRQ; + } +} + +static struct mmc_host_ops usdhi6_ops = { + .request = usdhi6_request, + .set_ios = usdhi6_set_ios, + .get_cd = usdhi6_get_cd, + .get_ro = usdhi6_get_ro, + .enable_sdio_irq = usdhi6_enable_sdio_irq, +}; + +/* State machine handlers */ + +static void usdhi6_resp_cmd12(struct usdhi6_host *host) +{ + struct mmc_command *cmd = host->mrq->stop; + cmd->resp[0] = usdhi6_read(host, USDHI6_SD_RSP10); +} + +static void usdhi6_resp_read(struct usdhi6_host *host) +{ + struct mmc_command *cmd = host->mrq->cmd; + u32 *rsp = cmd->resp, tmp = 0; + int i; + +/* + * RSP10 39-8 + * RSP32 71-40 + * RSP54 103-72 + * RSP76 127-104 + * R2-type response: + * resp[0] = r[127..96] + * resp[1] = r[95..64] + * resp[2] = r[63..32] + * resp[3] = r[31..0] + * Other responses: + * resp[0] = r[39..8] + */ + + if (mmc_resp_type(cmd) == MMC_RSP_NONE) + return; + + if (!(host->irq_status & USDHI6_SD_INFO1_RSP_END)) { + dev_err(mmc_dev(host->mmc), + "CMD%d: response expected but is missing!\n", cmd->opcode); + return; + } + + if (mmc_resp_type(cmd) & MMC_RSP_136) + for (i = 0; i < 4; i++) { + if (i) + rsp[3 - i] = tmp >> 24; + tmp = usdhi6_read(host, USDHI6_SD_RSP10 + i * 8); + rsp[3 - i] |= tmp << 8; + } + else if (cmd->opcode == MMC_READ_MULTIPLE_BLOCK || + cmd->opcode == MMC_WRITE_MULTIPLE_BLOCK) + /* Read RSP54 to avoid conflict with auto CMD12 */ + rsp[0] = usdhi6_read(host, USDHI6_SD_RSP54); + else + rsp[0] = usdhi6_read(host, USDHI6_SD_RSP10); + + dev_dbg(mmc_dev(host->mmc), "Response 0x%x\n", rsp[0]); +} + +static int usdhi6_blk_read(struct usdhi6_host *host) +{ + struct mmc_data *data = host->mrq->data; + u32 *p; + int i, rest; + + if (host->io_error) { + data->error = usdhi6_error_code(host); + goto error; + } + + if (host->pg.page) { + p = host->blk_page + host->offset; + } else { + p = usdhi6_sg_map(host); + if (!p) { + data->error = -ENOMEM; + goto error; + } + } + + for (i = 0; i < data->blksz / 4; i++, p++) + *p = usdhi6_read(host, USDHI6_SD_BUF0); + + rest = data->blksz % 4; + for (i = 0; i < (rest + 1) / 2; i++) { + u16 d = usdhi6_read16(host, USDHI6_SD_BUF0); + ((u8 *)p)[2 * i] = ((u8 *)&d)[0]; + if (rest > 1 && !i) + ((u8 *)p)[2 * i + 1] = ((u8 *)&d)[1]; + } + + return 0; + +error: + dev_dbg(mmc_dev(host->mmc), "%s(): %d\n", __func__, data->error); + host->wait = USDHI6_WAIT_FOR_REQUEST; + return data->error; +} + +static int usdhi6_blk_write(struct usdhi6_host *host) +{ + struct mmc_data *data = host->mrq->data; + u32 *p; + int i, rest; + + if (host->io_error) { + data->error = usdhi6_error_code(host); + goto error; + } + + if (host->pg.page) { + p = host->blk_page + host->offset; + } else { + p = usdhi6_sg_map(host); + if (!p) { + data->error = -ENOMEM; + goto error; + } + } + + for (i = 0; i < data->blksz / 4; i++, p++) + usdhi6_write(host, USDHI6_SD_BUF0, *p); + + rest = data->blksz % 4; + for (i = 0; i < (rest + 1) / 2; i++) { + u16 d; + ((u8 *)&d)[0] = ((u8 *)p)[2 * i]; + if (rest > 1 && !i) + ((u8 *)&d)[1] = ((u8 *)p)[2 * i + 1]; + else + ((u8 *)&d)[1] = 0; + usdhi6_write16(host, USDHI6_SD_BUF0, d); + } + + return 0; + +error: + dev_dbg(mmc_dev(host->mmc), "%s(): %d\n", __func__, data->error); + host->wait = USDHI6_WAIT_FOR_REQUEST; + return data->error; +} + +static int usdhi6_stop_cmd(struct usdhi6_host *host) +{ + struct mmc_request *mrq = host->mrq; + + switch (mrq->cmd->opcode) { + case MMC_READ_MULTIPLE_BLOCK: + case MMC_WRITE_MULTIPLE_BLOCK: + if (mrq->stop->opcode == MMC_STOP_TRANSMISSION) { + host->wait = USDHI6_WAIT_FOR_STOP; + return 0; + } + /* Unsupported STOP command */ + default: + dev_err(mmc_dev(host->mmc), + "unsupported stop CMD%d for CMD%d\n", + mrq->stop->opcode, mrq->cmd->opcode); + mrq->stop->error = -EOPNOTSUPP; + } + + return -EOPNOTSUPP; +} + +static bool usdhi6_end_cmd(struct usdhi6_host *host) +{ + struct mmc_request *mrq = host->mrq; + struct mmc_command *cmd = mrq->cmd; + + if (host->io_error) { + cmd->error = usdhi6_error_code(host); + return false; + } + + usdhi6_resp_read(host); + + if (!mrq->data) + return false; + + if (host->dma_active) { + usdhi6_dma_kick(host); + if (!mrq->stop) + host->wait = USDHI6_WAIT_FOR_DMA; + else if (usdhi6_stop_cmd(host) < 0) + return false; + } else if (mrq->data->flags & MMC_DATA_READ) { + if (cmd->opcode == MMC_READ_MULTIPLE_BLOCK || + (cmd->opcode == SD_IO_RW_EXTENDED && + mrq->data->blocks > 1)) + host->wait = USDHI6_WAIT_FOR_MREAD; + else + host->wait = USDHI6_WAIT_FOR_READ; + } else { + if (cmd->opcode == MMC_WRITE_MULTIPLE_BLOCK || + (cmd->opcode == SD_IO_RW_EXTENDED && + mrq->data->blocks > 1)) + host->wait = USDHI6_WAIT_FOR_MWRITE; + else + host->wait = USDHI6_WAIT_FOR_WRITE; + } + + return true; +} + +static bool usdhi6_read_block(struct usdhi6_host *host) +{ + /* ACCESS_END IRQ is already unmasked */ + int ret = usdhi6_blk_read(host); + + /* + * Have to force unmapping both pages: the single block could have been + * cross-page, in which case for single-block IO host->page_idx == 0. + * So, if we don't force, the second page won't be unmapped. + */ + usdhi6_sg_unmap(host, true); + + if (ret < 0) + return false; + + host->wait = USDHI6_WAIT_FOR_DATA_END; + return true; +} + +static bool usdhi6_mread_block(struct usdhi6_host *host) +{ + int ret = usdhi6_blk_read(host); + + if (ret < 0) + return false; + + usdhi6_sg_advance(host); + + return !host->mrq->data->error && + (host->wait != USDHI6_WAIT_FOR_DATA_END || !host->mrq->stop); +} + +static bool usdhi6_write_block(struct usdhi6_host *host) +{ + int ret = usdhi6_blk_write(host); + + /* See comment in usdhi6_read_block() */ + usdhi6_sg_unmap(host, true); + + if (ret < 0) + return false; + + host->wait = USDHI6_WAIT_FOR_DATA_END; + return true; +} + +static bool usdhi6_mwrite_block(struct usdhi6_host *host) +{ + int ret = usdhi6_blk_write(host); + + if (ret < 0) + return false; + + usdhi6_sg_advance(host); + + return !host->mrq->data->error && + (host->wait != USDHI6_WAIT_FOR_DATA_END || !host->mrq->stop); +} + +/* Interrupt & timeout handlers */ + +static irqreturn_t usdhi6_sd_bh(int irq, void *dev_id) +{ + struct usdhi6_host *host = dev_id; + struct mmc_request *mrq; + struct mmc_command *cmd; + struct mmc_data *data; + bool io_wait = false; + + cancel_delayed_work_sync(&host->timeout_work); + + mrq = host->mrq; + if (!mrq) + return IRQ_HANDLED; + + cmd = mrq->cmd; + data = mrq->data; + + switch (host->wait) { + case USDHI6_WAIT_FOR_REQUEST: + /* We're too late, the timeout has already kicked in */ + return IRQ_HANDLED; + case USDHI6_WAIT_FOR_CMD: + /* Wait for data? */ + io_wait = usdhi6_end_cmd(host); + break; + case USDHI6_WAIT_FOR_MREAD: + /* Wait for more data? */ + io_wait = usdhi6_mread_block(host); + break; + case USDHI6_WAIT_FOR_READ: + /* Wait for data end? */ + io_wait = usdhi6_read_block(host); + break; + case USDHI6_WAIT_FOR_MWRITE: + /* Wait data to write? */ + io_wait = usdhi6_mwrite_block(host); + break; + case USDHI6_WAIT_FOR_WRITE: + /* Wait for data end? */ + io_wait = usdhi6_write_block(host); + break; + case USDHI6_WAIT_FOR_DMA: + usdhi6_dma_check_error(host); + break; + case USDHI6_WAIT_FOR_STOP: + usdhi6_write(host, USDHI6_SD_STOP, 0); + if (host->io_error) { + int ret = usdhi6_error_code(host); + if (mrq->stop) + mrq->stop->error = ret; + else + mrq->data->error = ret; + dev_warn(mmc_dev(host->mmc), "%s(): %d\n", __func__, ret); + break; + } + usdhi6_resp_cmd12(host); + mrq->stop->error = 0; + break; + case USDHI6_WAIT_FOR_DATA_END: + if (host->io_error) { + mrq->data->error = usdhi6_error_code(host); + dev_warn(mmc_dev(host->mmc), "%s(): %d\n", __func__, + mrq->data->error); + } + break; + default: + cmd->error = -EFAULT; + dev_err(mmc_dev(host->mmc), "Invalid state %u\n", host->wait); + usdhi6_request_done(host); + return IRQ_HANDLED; + } + + if (io_wait) { + schedule_delayed_work(&host->timeout_work, host->timeout); + /* Wait for more data or ACCESS_END */ + if (!host->dma_active) + usdhi6_wait_for_brwe(host, mrq->data->flags & MMC_DATA_READ); + return IRQ_HANDLED; + } + + if (!cmd->error) { + if (data) { + if (!data->error) { + if (host->wait != USDHI6_WAIT_FOR_STOP && + host->mrq->stop && + !host->mrq->stop->error && + !usdhi6_stop_cmd(host)) { + /* Sending STOP */ + usdhi6_wait_for_resp(host); + + schedule_delayed_work(&host->timeout_work, + host->timeout); + + return IRQ_HANDLED; + } + + data->bytes_xfered = data->blocks * data->blksz; + } else { + /* Data error: might need to unmap the last page */ + dev_warn(mmc_dev(host->mmc), "%s(): data error %d\n", + __func__, data->error); + usdhi6_sg_unmap(host, true); + } + } else if (cmd->opcode == MMC_APP_CMD) { + host->app_cmd = true; + } + } + + usdhi6_request_done(host); + + return IRQ_HANDLED; +} + +static irqreturn_t usdhi6_sd(int irq, void *dev_id) +{ + struct usdhi6_host *host = dev_id; + u16 status, status2, error; + + status = usdhi6_read(host, USDHI6_SD_INFO1) & ~host->status_mask & + ~USDHI6_SD_INFO1_CARD; + status2 = usdhi6_read(host, USDHI6_SD_INFO2) & ~host->status2_mask; + + usdhi6_only_cd(host); + + dev_dbg(mmc_dev(host->mmc), + "IRQ status = 0x%08x, status2 = 0x%08x\n", status, status2); + + if (!status && !status2) + return IRQ_NONE; + + error = status2 & USDHI6_SD_INFO2_ERR; + + /* Ack / clear interrupts */ + if (USDHI6_SD_INFO1_IRQ & status) + usdhi6_write(host, USDHI6_SD_INFO1, + 0xffff & ~(USDHI6_SD_INFO1_IRQ & status)); + + if (USDHI6_SD_INFO2_IRQ & status2) { + if (error) + /* In error cases BWE and BRE aren't cleared automatically */ + status2 |= USDHI6_SD_INFO2_BWE | USDHI6_SD_INFO2_BRE; + + usdhi6_write(host, USDHI6_SD_INFO2, + 0xffff & ~(USDHI6_SD_INFO2_IRQ & status2)); + } + + host->io_error = error; + host->irq_status = status; + + if (error) { + /* Don't pollute the log with unsupported command timeouts */ + if (host->wait != USDHI6_WAIT_FOR_CMD || + error != USDHI6_SD_INFO2_RSP_TOUT) + dev_warn(mmc_dev(host->mmc), + "%s(): INFO2 error bits 0x%08x\n", + __func__, error); + else + dev_dbg(mmc_dev(host->mmc), + "%s(): INFO2 error bits 0x%08x\n", + __func__, error); + } + + return IRQ_WAKE_THREAD; +} + +static irqreturn_t usdhi6_sdio(int irq, void *dev_id) +{ + struct usdhi6_host *host = dev_id; + u32 status = usdhi6_read(host, USDHI6_SDIO_INFO1) & ~host->sdio_mask; + + dev_dbg(mmc_dev(host->mmc), "%s(): status 0x%x\n", __func__, status); + + if (!status) + return IRQ_NONE; + + usdhi6_write(host, USDHI6_SDIO_INFO1, ~status); + + mmc_signal_sdio_irq(host->mmc); + + return IRQ_HANDLED; +} + +static irqreturn_t usdhi6_cd(int irq, void *dev_id) +{ + struct usdhi6_host *host = dev_id; + struct mmc_host *mmc = host->mmc; + u16 status; + + /* We're only interested in hotplug events here */ + status = usdhi6_read(host, USDHI6_SD_INFO1) & ~host->status_mask & + USDHI6_SD_INFO1_CARD; + + if (!status) + return IRQ_NONE; + + /* Ack */ + usdhi6_write(host, USDHI6_SD_INFO1, !status); + + if (!work_pending(&mmc->detect.work) && + (((status & USDHI6_SD_INFO1_CARD_INSERT) && + !mmc->card) || + ((status & USDHI6_SD_INFO1_CARD_EJECT) && + mmc->card))) + mmc_detect_change(mmc, msecs_to_jiffies(100)); + + return IRQ_HANDLED; +} + +/* + * Actually this should not be needed, if the built-in timeout works reliably in + * the both PIO cases and DMA never fails. But if DMA does fail, a timeout + * handler might be the only way to catch the error. + */ +static void usdhi6_timeout_work(struct work_struct *work) +{ + struct delayed_work *d = container_of(work, struct delayed_work, work); + struct usdhi6_host *host = container_of(d, struct usdhi6_host, timeout_work); + struct mmc_request *mrq = host->mrq; + struct mmc_data *data = mrq ? mrq->data : NULL; + + dev_warn(mmc_dev(host->mmc), + "%s timeout wait %u CMD%d: IRQ 0x%08x:0x%08x, last IRQ 0x%08x\n", + host->dma_active ? "DMA" : "PIO", + host->wait, mrq ? mrq->cmd->opcode : -1, + usdhi6_read(host, USDHI6_SD_INFO1), + usdhi6_read(host, USDHI6_SD_INFO2), host->irq_status); + + if (host->dma_active) { + usdhi6_dma_kill(host); + usdhi6_dma_stop_unmap(host); + } + + switch (host->wait) { + default: + dev_err(mmc_dev(host->mmc), "Invalid state %u\n", host->wait); + /* mrq can be NULL in this actually impossible case */ + case USDHI6_WAIT_FOR_CMD: + usdhi6_error_code(host); + if (mrq) + mrq->cmd->error = -ETIMEDOUT; + break; + case USDHI6_WAIT_FOR_STOP: + usdhi6_error_code(host); + mrq->stop->error = -ETIMEDOUT; + break; + case USDHI6_WAIT_FOR_DMA: + case USDHI6_WAIT_FOR_MREAD: + case USDHI6_WAIT_FOR_MWRITE: + case USDHI6_WAIT_FOR_READ: + case USDHI6_WAIT_FOR_WRITE: + dev_dbg(mmc_dev(host->mmc), + "%c: page #%u @ +0x%x %ux%u in SG%u. Current SG %u bytes @ %u\n", + data->flags & MMC_DATA_READ ? 'R' : 'W', host->page_idx, + host->offset, data->blocks, data->blksz, data->sg_len, + sg_dma_len(host->sg), host->sg->offset); + usdhi6_sg_unmap(host, true); + /* + * If USDHI6_WAIT_FOR_DATA_END times out, we have already unmapped + * the page + */ + case USDHI6_WAIT_FOR_DATA_END: + usdhi6_error_code(host); + data->error = -ETIMEDOUT; + } + + if (mrq) + usdhi6_request_done(host); +} + +/* Probe / release */ + +static const struct of_device_id usdhi6_of_match[] = { + {.compatible = "renesas,usdhi6rol0"}, + {} +}; +MODULE_DEVICE_TABLE(of, usdhi6_of_match); + +static int usdhi6_probe(struct platform_device *pdev) +{ + struct device *dev = &pdev->dev; + struct mmc_host *mmc; + struct usdhi6_host *host; + struct resource *res; + int irq_cd, irq_sd, irq_sdio; + u32 version; + int ret; + + if (!dev->of_node) + return -ENODEV; + + irq_cd = platform_get_irq_byname(pdev, "card detect"); + irq_sd = platform_get_irq_byname(pdev, "data"); + irq_sdio = platform_get_irq_byname(pdev, "SDIO"); + if (irq_sd < 0 || irq_sdio < 0) + return -ENODEV; + + mmc = mmc_alloc_host(sizeof(struct usdhi6_host), dev); + if (!mmc) + return -ENOMEM; + + ret = mmc_of_parse(mmc); + if (ret < 0) + goto e_free_mmc; + + mmc_regulator_get_supply(mmc); + + host = mmc_priv(mmc); + host->mmc = mmc; + host->wait = USDHI6_WAIT_FOR_REQUEST; + host->timeout = msecs_to_jiffies(4000); + + res = platform_get_resource(pdev, IORESOURCE_MEM, 0); + host->base = devm_ioremap_resource(dev, res); + if (IS_ERR(host->base)) { + ret = PTR_ERR(host->base); + goto e_free_mmc; + } + + host->clk = devm_clk_get(dev, NULL); + if (IS_ERR(host->clk)) + goto e_free_mmc; + + host->imclk = clk_get_rate(host->clk); + + ret = clk_prepare_enable(host->clk); + if (ret < 0) + goto e_free_mmc; + + version = usdhi6_read(host, USDHI6_VERSION); + if ((version & 0xfff) != 0xa0d) { + dev_err(dev, "Version not recognized %x\n", version); + goto e_clk_off; + } + + dev_info(dev, "A USDHI6ROL0 SD host detected with %d ports\n", + usdhi6_read(host, USDHI6_SD_PORT_SEL) >> USDHI6_SD_PORT_SEL_PORTS_SHIFT); + + usdhi6_mask_all(host); + + if (irq_cd >= 0) { + ret = devm_request_irq(dev, irq_cd, usdhi6_cd, 0, + dev_name(dev), host); + if (ret < 0) + goto e_clk_off; + } else { + mmc->caps |= MMC_CAP_NEEDS_POLL; + } + + ret = devm_request_threaded_irq(dev, irq_sd, usdhi6_sd, usdhi6_sd_bh, 0, + dev_name(dev), host); + if (ret < 0) + goto e_clk_off; + + ret = devm_request_irq(dev, irq_sdio, usdhi6_sdio, 0, + dev_name(dev), host); + if (ret < 0) + goto e_clk_off; + + INIT_DELAYED_WORK(&host->timeout_work, usdhi6_timeout_work); + + usdhi6_dma_request(host, res->start); + + mmc->ops = &usdhi6_ops; + mmc->caps |= MMC_CAP_SD_HIGHSPEED | MMC_CAP_MMC_HIGHSPEED | + MMC_CAP_UHS_SDR50 | MMC_CAP_UHS_DDR50 | MMC_CAP_SDIO_IRQ; + /* Set .max_segs to some random number. Feel free to adjust. */ + mmc->max_segs = 32; + mmc->max_blk_size = 512; + mmc->max_req_size = PAGE_CACHE_SIZE * mmc->max_segs; + mmc->max_blk_count = mmc->max_req_size / mmc->max_blk_size; + /* + * Setting .max_seg_size to 1 page would simplify our page-mapping code, + * But OTOH, having large segments makes DMA more efficient. We could + * check, whether we managed to get DMA and fall back to 1 page + * segments, but if we do manage to obtain DMA and then it fails at + * run-time and we fall back to PIO, we will continue getting large + * segments. So, we wouldn't be able to get rid of the code anyway. + */ + mmc->max_seg_size = mmc->max_req_size; + if (!mmc->f_max) + mmc->f_max = host->imclk; + mmc->f_min = host->imclk / 512; + + platform_set_drvdata(pdev, host); + + ret = mmc_add_host(mmc); + if (ret < 0) + goto e_clk_off; + + return 0; + +e_clk_off: + clk_disable_unprepare(host->clk); +e_free_mmc: + mmc_free_host(mmc); + + return ret; +} + +static int usdhi6_remove(struct platform_device *pdev) +{ + struct usdhi6_host *host = platform_get_drvdata(pdev); + + mmc_remove_host(host->mmc); + + usdhi6_mask_all(host); + cancel_delayed_work_sync(&host->timeout_work); + usdhi6_dma_release(host); + clk_disable_unprepare(host->clk); + mmc_free_host(host->mmc); + + return 0; +} + +static struct platform_driver usdhi6_driver = { + .probe = usdhi6_probe, + .remove = usdhi6_remove, + .driver = { + .name = "usdhi6rol0", + .owner = THIS_MODULE, + .of_match_table = usdhi6_of_match, + }, +}; + +module_platform_driver(usdhi6_driver); + +MODULE_DESCRIPTION("Renesas usdhi6rol0 SD/SDIO host driver"); +MODULE_LICENSE("GPL v2"); +MODULE_ALIAS("platform:usdhi6rol0"); +MODULE_AUTHOR("Guennadi Liakhovetski <g.liakhovetski@gmx.de>"); diff --git a/drivers/mmc/host/wmt-sdmmc.c b/drivers/mmc/host/wmt-sdmmc.c index 498d1f799085..282891a8e451 100644 --- a/drivers/mmc/host/wmt-sdmmc.c +++ b/drivers/mmc/host/wmt-sdmmc.c @@ -840,7 +840,7 @@ static int wmt_mci_probe(struct platform_device *pdev) priv->dma_desc_buffer = dma_alloc_coherent(&pdev->dev, mmc->max_blk_count * 16, &priv->dma_desc_device_addr, - 208); + GFP_KERNEL); if (!priv->dma_desc_buffer) { dev_err(&pdev->dev, "DMA alloc fail\n"); ret = -EPERM; diff --git a/drivers/mtd/ubi/block.c b/drivers/mtd/ubi/block.c index 20a667c95da4..8457df7ec5af 100644 --- a/drivers/mtd/ubi/block.c +++ b/drivers/mtd/ubi/block.c @@ -432,8 +432,10 @@ int ubiblock_create(struct ubi_volume_info *vi) * Rembember workqueues are cheap, they're not threads. */ dev->wq = alloc_workqueue("%s", 0, 0, gd->disk_name); - if (!dev->wq) + if (!dev->wq) { + ret = -ENOMEM; goto out_free_queue; + } INIT_WORK(&dev->work, ubiblock_do_work); mutex_lock(&devices_mutex); diff --git a/drivers/mtd/ubi/cdev.c b/drivers/mtd/ubi/cdev.c index f54562a5998e..7646220ca6e2 100644 --- a/drivers/mtd/ubi/cdev.c +++ b/drivers/mtd/ubi/cdev.c @@ -731,7 +731,7 @@ static int rename_volumes(struct ubi_device *ubi, goto out_free; } - re->desc = ubi_open_volume(ubi->ubi_num, vol_id, UBI_EXCLUSIVE); + re->desc = ubi_open_volume(ubi->ubi_num, vol_id, UBI_READWRITE); if (IS_ERR(re->desc)) { err = PTR_ERR(re->desc); ubi_err("cannot open volume %d, error %d", vol_id, err); diff --git a/drivers/mtd/ubi/fastmap.c b/drivers/mtd/ubi/fastmap.c index c5dad652614d..b04e7d059888 100644 --- a/drivers/mtd/ubi/fastmap.c +++ b/drivers/mtd/ubi/fastmap.c @@ -125,9 +125,9 @@ static struct ubi_ainf_volume *add_vol(struct ubi_attach_info *ai, int vol_id, parent = *p; av = rb_entry(parent, struct ubi_ainf_volume, rb); - if (vol_id > av->vol_id) + if (vol_id < av->vol_id) p = &(*p)->rb_left; - else if (vol_id > av->vol_id) + else p = &(*p)->rb_right; } diff --git a/drivers/net/ethernet/mellanox/mlx4/alloc.c b/drivers/net/ethernet/mellanox/mlx4/alloc.c index c3ad464d0627..b0297da50304 100644 --- a/drivers/net/ethernet/mellanox/mlx4/alloc.c +++ b/drivers/net/ethernet/mellanox/mlx4/alloc.c @@ -171,7 +171,7 @@ void mlx4_bitmap_cleanup(struct mlx4_bitmap *bitmap) */ int mlx4_buf_alloc(struct mlx4_dev *dev, int size, int max_direct, - struct mlx4_buf *buf) + struct mlx4_buf *buf, gfp_t gfp) { dma_addr_t t; @@ -180,7 +180,7 @@ int mlx4_buf_alloc(struct mlx4_dev *dev, int size, int max_direct, buf->npages = 1; buf->page_shift = get_order(size) + PAGE_SHIFT; buf->direct.buf = dma_alloc_coherent(&dev->pdev->dev, - size, &t, GFP_KERNEL); + size, &t, gfp); if (!buf->direct.buf) return -ENOMEM; @@ -200,14 +200,14 @@ int mlx4_buf_alloc(struct mlx4_dev *dev, int size, int max_direct, buf->npages = buf->nbufs; buf->page_shift = PAGE_SHIFT; buf->page_list = kcalloc(buf->nbufs, sizeof(*buf->page_list), - GFP_KERNEL); + gfp); if (!buf->page_list) return -ENOMEM; for (i = 0; i < buf->nbufs; ++i) { buf->page_list[i].buf = dma_alloc_coherent(&dev->pdev->dev, PAGE_SIZE, - &t, GFP_KERNEL); + &t, gfp); if (!buf->page_list[i].buf) goto err_free; @@ -218,7 +218,7 @@ int mlx4_buf_alloc(struct mlx4_dev *dev, int size, int max_direct, if (BITS_PER_LONG == 64) { struct page **pages; - pages = kmalloc(sizeof *pages * buf->nbufs, GFP_KERNEL); + pages = kmalloc(sizeof *pages * buf->nbufs, gfp); if (!pages) goto err_free; for (i = 0; i < buf->nbufs; ++i) @@ -260,11 +260,12 @@ void mlx4_buf_free(struct mlx4_dev *dev, int size, struct mlx4_buf *buf) } EXPORT_SYMBOL_GPL(mlx4_buf_free); -static struct mlx4_db_pgdir *mlx4_alloc_db_pgdir(struct device *dma_device) +static struct mlx4_db_pgdir *mlx4_alloc_db_pgdir(struct device *dma_device, + gfp_t gfp) { struct mlx4_db_pgdir *pgdir; - pgdir = kzalloc(sizeof *pgdir, GFP_KERNEL); + pgdir = kzalloc(sizeof *pgdir, gfp); if (!pgdir) return NULL; @@ -272,7 +273,7 @@ static struct mlx4_db_pgdir *mlx4_alloc_db_pgdir(struct device *dma_device) pgdir->bits[0] = pgdir->order0; pgdir->bits[1] = pgdir->order1; pgdir->db_page = dma_alloc_coherent(dma_device, PAGE_SIZE, - &pgdir->db_dma, GFP_KERNEL); + &pgdir->db_dma, gfp); if (!pgdir->db_page) { kfree(pgdir); return NULL; @@ -312,7 +313,7 @@ found: return 0; } -int mlx4_db_alloc(struct mlx4_dev *dev, struct mlx4_db *db, int order) +int mlx4_db_alloc(struct mlx4_dev *dev, struct mlx4_db *db, int order, gfp_t gfp) { struct mlx4_priv *priv = mlx4_priv(dev); struct mlx4_db_pgdir *pgdir; @@ -324,7 +325,7 @@ int mlx4_db_alloc(struct mlx4_dev *dev, struct mlx4_db *db, int order) if (!mlx4_alloc_db_from_pgdir(pgdir, db, order)) goto out; - pgdir = mlx4_alloc_db_pgdir(&(dev->pdev->dev)); + pgdir = mlx4_alloc_db_pgdir(&(dev->pdev->dev), gfp); if (!pgdir) { ret = -ENOMEM; goto out; @@ -376,13 +377,13 @@ int mlx4_alloc_hwq_res(struct mlx4_dev *dev, struct mlx4_hwq_resources *wqres, { int err; - err = mlx4_db_alloc(dev, &wqres->db, 1); + err = mlx4_db_alloc(dev, &wqres->db, 1, GFP_KERNEL); if (err) return err; *wqres->db.db = 0; - err = mlx4_buf_alloc(dev, size, max_direct, &wqres->buf); + err = mlx4_buf_alloc(dev, size, max_direct, &wqres->buf, GFP_KERNEL); if (err) goto err_db; @@ -391,7 +392,7 @@ int mlx4_alloc_hwq_res(struct mlx4_dev *dev, struct mlx4_hwq_resources *wqres, if (err) goto err_buf; - err = mlx4_buf_write_mtt(dev, &wqres->mtt, &wqres->buf); + err = mlx4_buf_write_mtt(dev, &wqres->mtt, &wqres->buf, GFP_KERNEL); if (err) goto err_mtt; diff --git a/drivers/net/ethernet/mellanox/mlx4/cmd.c b/drivers/net/ethernet/mellanox/mlx4/cmd.c index 92d3249f63f1..29b616990e52 100644 --- a/drivers/net/ethernet/mellanox/mlx4/cmd.c +++ b/drivers/net/ethernet/mellanox/mlx4/cmd.c @@ -705,20 +705,28 @@ static int mlx4_MAD_IFC_wrapper(struct mlx4_dev *dev, int slave, struct ib_smp *smp = inbox->buf; u32 index; u8 port; + u8 opcode_modifier; u16 *table; int err; int vidx, pidx; + int network_view; struct mlx4_priv *priv = mlx4_priv(dev); struct ib_smp *outsmp = outbox->buf; __be16 *outtab = (__be16 *)(outsmp->data); __be32 slave_cap_mask; __be64 slave_node_guid; + port = vhcr->in_modifier; + /* network-view bit is for driver use only, and should not be passed to FW */ + opcode_modifier = vhcr->op_modifier & ~0x8; /* clear netw view bit */ + network_view = !!(vhcr->op_modifier & 0x8); + if (smp->base_version == 1 && smp->mgmt_class == IB_MGMT_CLASS_SUBN_LID_ROUTED && smp->class_version == 1) { - if (smp->method == IB_MGMT_METHOD_GET) { + /* host view is paravirtualized */ + if (!network_view && smp->method == IB_MGMT_METHOD_GET) { if (smp->attr_id == IB_SMP_ATTR_PKEY_TABLE) { index = be32_to_cpu(smp->attr_mod); if (port < 1 || port > dev->caps.num_ports) @@ -743,7 +751,7 @@ static int mlx4_MAD_IFC_wrapper(struct mlx4_dev *dev, int slave, /*get the slave specific caps:*/ /*do the command */ err = mlx4_cmd_box(dev, inbox->dma, outbox->dma, - vhcr->in_modifier, vhcr->op_modifier, + vhcr->in_modifier, opcode_modifier, vhcr->op, MLX4_CMD_TIME_CLASS_C, MLX4_CMD_NATIVE); /* modify the response for slaves */ if (!err && slave != mlx4_master_func_num(dev)) { @@ -760,7 +768,7 @@ static int mlx4_MAD_IFC_wrapper(struct mlx4_dev *dev, int slave, smp->attr_mod = cpu_to_be32(slave / 8); /* execute cmd */ err = mlx4_cmd_box(dev, inbox->dma, outbox->dma, - vhcr->in_modifier, vhcr->op_modifier, + vhcr->in_modifier, opcode_modifier, vhcr->op, MLX4_CMD_TIME_CLASS_C, MLX4_CMD_NATIVE); if (!err) { /* if needed, move slave gid to index 0 */ @@ -774,7 +782,7 @@ static int mlx4_MAD_IFC_wrapper(struct mlx4_dev *dev, int slave, } if (smp->attr_id == IB_SMP_ATTR_NODE_INFO) { err = mlx4_cmd_box(dev, inbox->dma, outbox->dma, - vhcr->in_modifier, vhcr->op_modifier, + vhcr->in_modifier, opcode_modifier, vhcr->op, MLX4_CMD_TIME_CLASS_C, MLX4_CMD_NATIVE); if (!err) { slave_node_guid = mlx4_get_slave_node_guid(dev, slave); @@ -784,19 +792,24 @@ static int mlx4_MAD_IFC_wrapper(struct mlx4_dev *dev, int slave, } } } + + /* Non-privileged VFs are only allowed "host" view LID-routed 'Get' MADs. + * These are the MADs used by ib verbs (such as ib_query_gids). + */ if (slave != mlx4_master_func_num(dev) && - ((smp->mgmt_class == IB_MGMT_CLASS_SUBN_DIRECTED_ROUTE) || - (smp->mgmt_class == IB_MGMT_CLASS_SUBN_LID_ROUTED && - smp->method == IB_MGMT_METHOD_SET))) { - mlx4_err(dev, "slave %d is trying to execute a Subnet MGMT MAD, " - "class 0x%x, method 0x%x for attr 0x%x. Rejecting\n", - slave, smp->method, smp->mgmt_class, - be16_to_cpu(smp->attr_id)); - return -EPERM; + !mlx4_vf_smi_enabled(dev, slave, port)) { + if (!(smp->mgmt_class == IB_MGMT_CLASS_SUBN_LID_ROUTED && + smp->method == IB_MGMT_METHOD_GET) || network_view) { + mlx4_err(dev, "Unprivileged slave %d is trying to execute a Subnet MGMT MAD, class 0x%x, method 0x%x, view=%s for attr 0x%x. Rejecting\n", + slave, smp->method, smp->mgmt_class, + network_view ? "Network" : "Host", + be16_to_cpu(smp->attr_id)); + return -EPERM; + } } - /*default:*/ + return mlx4_cmd_box(dev, inbox->dma, outbox->dma, - vhcr->in_modifier, vhcr->op_modifier, + vhcr->in_modifier, opcode_modifier, vhcr->op, MLX4_CMD_TIME_CLASS_C, MLX4_CMD_NATIVE); } @@ -1653,6 +1666,8 @@ static int mlx4_master_activate_admin_state(struct mlx4_priv *priv, int slave) for (port = min_port; port <= max_port; port++) { if (!test_bit(port - 1, actv_ports.ports)) continue; + priv->mfunc.master.vf_oper[slave].smi_enabled[port] = + priv->mfunc.master.vf_admin[slave].enable_smi[port]; vp_oper = &priv->mfunc.master.vf_oper[slave].vport[port]; vp_admin = &priv->mfunc.master.vf_admin[slave].vport[port]; vp_oper->state = *vp_admin; @@ -1704,6 +1719,8 @@ static void mlx4_master_deactivate_admin_state(struct mlx4_priv *priv, int slave for (port = min_port; port <= max_port; port++) { if (!test_bit(port - 1, actv_ports.ports)) continue; + priv->mfunc.master.vf_oper[slave].smi_enabled[port] = + MLX4_VF_SMI_DISABLED; vp_oper = &priv->mfunc.master.vf_oper[slave].vport[port]; if (NO_INDX != vp_oper->vlan_idx) { __mlx4_unregister_vlan(&priv->dev, @@ -2537,3 +2554,50 @@ int mlx4_set_vf_link_state(struct mlx4_dev *dev, int port, int vf, int link_stat return 0; } EXPORT_SYMBOL_GPL(mlx4_set_vf_link_state); + +int mlx4_vf_smi_enabled(struct mlx4_dev *dev, int slave, int port) +{ + struct mlx4_priv *priv = mlx4_priv(dev); + + if (slave < 1 || slave >= dev->num_slaves || + port < 1 || port > MLX4_MAX_PORTS) + return 0; + + return priv->mfunc.master.vf_oper[slave].smi_enabled[port] == + MLX4_VF_SMI_ENABLED; +} +EXPORT_SYMBOL_GPL(mlx4_vf_smi_enabled); + +int mlx4_vf_get_enable_smi_admin(struct mlx4_dev *dev, int slave, int port) +{ + struct mlx4_priv *priv = mlx4_priv(dev); + + if (slave == mlx4_master_func_num(dev)) + return 1; + + if (slave < 1 || slave >= dev->num_slaves || + port < 1 || port > MLX4_MAX_PORTS) + return 0; + + return priv->mfunc.master.vf_admin[slave].enable_smi[port] == + MLX4_VF_SMI_ENABLED; +} +EXPORT_SYMBOL_GPL(mlx4_vf_get_enable_smi_admin); + +int mlx4_vf_set_enable_smi_admin(struct mlx4_dev *dev, int slave, int port, + int enabled) +{ + struct mlx4_priv *priv = mlx4_priv(dev); + + if (slave == mlx4_master_func_num(dev)) + return 0; + + if (slave < 1 || slave >= dev->num_slaves || + port < 1 || port > MLX4_MAX_PORTS || + enabled < 0 || enabled > 1) + return -EINVAL; + + priv->mfunc.master.vf_admin[slave].enable_smi[port] = enabled; + return 0; +} +EXPORT_SYMBOL_GPL(mlx4_vf_set_enable_smi_admin); diff --git a/drivers/net/ethernet/mellanox/mlx4/cq.c b/drivers/net/ethernet/mellanox/mlx4/cq.c index 0487121e4a0f..c90cde5b4aee 100644 --- a/drivers/net/ethernet/mellanox/mlx4/cq.c +++ b/drivers/net/ethernet/mellanox/mlx4/cq.c @@ -173,11 +173,11 @@ int __mlx4_cq_alloc_icm(struct mlx4_dev *dev, int *cqn) if (*cqn == -1) return -ENOMEM; - err = mlx4_table_get(dev, &cq_table->table, *cqn); + err = mlx4_table_get(dev, &cq_table->table, *cqn, GFP_KERNEL); if (err) goto err_out; - err = mlx4_table_get(dev, &cq_table->cmpt_table, *cqn); + err = mlx4_table_get(dev, &cq_table->cmpt_table, *cqn, GFP_KERNEL); if (err) goto err_put; return 0; diff --git a/drivers/net/ethernet/mellanox/mlx4/en_rx.c b/drivers/net/ethernet/mellanox/mlx4/en_rx.c index ba049ae88749..87857a6463eb 100644 --- a/drivers/net/ethernet/mellanox/mlx4/en_rx.c +++ b/drivers/net/ethernet/mellanox/mlx4/en_rx.c @@ -972,7 +972,7 @@ static int mlx4_en_config_rss_qp(struct mlx4_en_priv *priv, int qpn, if (!context) return -ENOMEM; - err = mlx4_qp_alloc(mdev->dev, qpn, qp); + err = mlx4_qp_alloc(mdev->dev, qpn, qp, GFP_KERNEL); if (err) { en_err(priv, "Failed to allocate qp #%x\n", qpn); goto out; @@ -1012,7 +1012,7 @@ int mlx4_en_create_drop_qp(struct mlx4_en_priv *priv) en_err(priv, "Failed reserving drop qpn\n"); return err; } - err = mlx4_qp_alloc(priv->mdev->dev, qpn, &priv->drop_qp); + err = mlx4_qp_alloc(priv->mdev->dev, qpn, &priv->drop_qp, GFP_KERNEL); if (err) { en_err(priv, "Failed allocating drop qp\n"); mlx4_qp_release_range(priv->mdev->dev, qpn, 1); @@ -1071,7 +1071,7 @@ int mlx4_en_config_rss_steer(struct mlx4_en_priv *priv) } /* Configure RSS indirection qp */ - err = mlx4_qp_alloc(mdev->dev, priv->base_qpn, &rss_map->indir_qp); + err = mlx4_qp_alloc(mdev->dev, priv->base_qpn, &rss_map->indir_qp, GFP_KERNEL); if (err) { en_err(priv, "Failed to allocate RSS indirection QP\n"); goto rss_err; diff --git a/drivers/net/ethernet/mellanox/mlx4/en_tx.c b/drivers/net/ethernet/mellanox/mlx4/en_tx.c index dd1f6d346459..bc0cc1eb214d 100644 --- a/drivers/net/ethernet/mellanox/mlx4/en_tx.c +++ b/drivers/net/ethernet/mellanox/mlx4/en_tx.c @@ -113,7 +113,7 @@ int mlx4_en_create_tx_ring(struct mlx4_en_priv *priv, ring->buf_size, (unsigned long long) ring->wqres.buf.direct.map); ring->qpn = qpn; - err = mlx4_qp_alloc(mdev->dev, ring->qpn, &ring->qp); + err = mlx4_qp_alloc(mdev->dev, ring->qpn, &ring->qp, GFP_KERNEL); if (err) { en_err(priv, "Failed allocating qp %d\n", ring->qpn); goto err_map; diff --git a/drivers/net/ethernet/mellanox/mlx4/fw.c b/drivers/net/ethernet/mellanox/mlx4/fw.c index d16a4d118903..01e6dd61ee3c 100644 --- a/drivers/net/ethernet/mellanox/mlx4/fw.c +++ b/drivers/net/ethernet/mellanox/mlx4/fw.c @@ -178,8 +178,8 @@ int mlx4_QUERY_FUNC_CAP_wrapper(struct mlx4_dev *dev, int slave, struct mlx4_cmd_info *cmd) { struct mlx4_priv *priv = mlx4_priv(dev); - u8 field; - u32 size; + u8 field, port; + u32 size, proxy_qp, qkey; int err = 0; #define QUERY_FUNC_CAP_FLAGS_OFFSET 0x0 @@ -209,6 +209,7 @@ int mlx4_QUERY_FUNC_CAP_wrapper(struct mlx4_dev *dev, int slave, /* when opcode modifier = 1 */ #define QUERY_FUNC_CAP_PHYS_PORT_OFFSET 0x3 +#define QUERY_FUNC_CAP_PRIV_VF_QKEY_OFFSET 0x4 #define QUERY_FUNC_CAP_FLAGS0_OFFSET 0x8 #define QUERY_FUNC_CAP_FLAGS1_OFFSET 0xc @@ -221,6 +222,7 @@ int mlx4_QUERY_FUNC_CAP_wrapper(struct mlx4_dev *dev, int slave, #define QUERY_FUNC_CAP_FLAGS1_FORCE_MAC 0x40 #define QUERY_FUNC_CAP_FLAGS1_FORCE_VLAN 0x80 #define QUERY_FUNC_CAP_FLAGS1_NIC_INFO 0x10 +#define QUERY_FUNC_CAP_VF_ENABLE_QP0 0x08 #define QUERY_FUNC_CAP_FLAGS0_FORCE_PHY_WQE_GID 0x80 @@ -234,28 +236,35 @@ int mlx4_QUERY_FUNC_CAP_wrapper(struct mlx4_dev *dev, int slave, return -EINVAL; vhcr->in_modifier = converted_port; - /* Set nic_info bit to mark new fields support */ - field = QUERY_FUNC_CAP_FLAGS1_NIC_INFO; - MLX4_PUT(outbox->buf, field, QUERY_FUNC_CAP_FLAGS1_OFFSET); - /* phys-port = logical-port */ field = vhcr->in_modifier - find_first_bit(actv_ports.ports, dev->caps.num_ports); MLX4_PUT(outbox->buf, field, QUERY_FUNC_CAP_PHYS_PORT_OFFSET); - field = vhcr->in_modifier; + port = vhcr->in_modifier; + proxy_qp = dev->phys_caps.base_proxy_sqpn + 8 * slave + port - 1; + + /* Set nic_info bit to mark new fields support */ + field = QUERY_FUNC_CAP_FLAGS1_NIC_INFO; + + if (mlx4_vf_smi_enabled(dev, slave, port) && + !mlx4_get_parav_qkey(dev, proxy_qp, &qkey)) { + field |= QUERY_FUNC_CAP_VF_ENABLE_QP0; + MLX4_PUT(outbox->buf, qkey, + QUERY_FUNC_CAP_PRIV_VF_QKEY_OFFSET); + } + MLX4_PUT(outbox->buf, field, QUERY_FUNC_CAP_FLAGS1_OFFSET); + /* size is now the QP number */ - size = dev->phys_caps.base_tunnel_sqpn + 8 * slave + field - 1; + size = dev->phys_caps.base_tunnel_sqpn + 8 * slave + port - 1; MLX4_PUT(outbox->buf, size, QUERY_FUNC_CAP_QP0_TUNNEL); size += 2; MLX4_PUT(outbox->buf, size, QUERY_FUNC_CAP_QP1_TUNNEL); - size = dev->phys_caps.base_proxy_sqpn + 8 * slave + field - 1; - MLX4_PUT(outbox->buf, size, QUERY_FUNC_CAP_QP0_PROXY); - - size += 2; - MLX4_PUT(outbox->buf, size, QUERY_FUNC_CAP_QP1_PROXY); + MLX4_PUT(outbox->buf, proxy_qp, QUERY_FUNC_CAP_QP0_PROXY); + proxy_qp += 2; + MLX4_PUT(outbox->buf, proxy_qp, QUERY_FUNC_CAP_QP1_PROXY); MLX4_PUT(outbox->buf, dev->caps.phys_port_id[vhcr->in_modifier], QUERY_FUNC_CAP_PHYS_PORT_ID); @@ -326,7 +335,7 @@ int mlx4_QUERY_FUNC_CAP(struct mlx4_dev *dev, u32 gen_or_port, struct mlx4_cmd_mailbox *mailbox; u32 *outbox; u8 field, op_modifier; - u32 size; + u32 size, qkey; int err = 0, quotas = 0; op_modifier = !!gen_or_port; /* 0 = general, 1 = logical port */ @@ -414,7 +423,7 @@ int mlx4_QUERY_FUNC_CAP(struct mlx4_dev *dev, u32 gen_or_port, MLX4_GET(func_cap->flags1, outbox, QUERY_FUNC_CAP_FLAGS1_OFFSET); if (dev->caps.port_type[gen_or_port] == MLX4_PORT_TYPE_ETH) { - if (func_cap->flags1 & QUERY_FUNC_CAP_FLAGS1_OFFSET) { + if (func_cap->flags1 & QUERY_FUNC_CAP_FLAGS1_FORCE_VLAN) { mlx4_err(dev, "VLAN is enforced on this port\n"); err = -EPROTONOSUPPORT; goto out; @@ -442,6 +451,13 @@ int mlx4_QUERY_FUNC_CAP(struct mlx4_dev *dev, u32 gen_or_port, goto out; } + if (func_cap->flags1 & QUERY_FUNC_CAP_VF_ENABLE_QP0) { + MLX4_GET(qkey, outbox, QUERY_FUNC_CAP_PRIV_VF_QKEY_OFFSET); + func_cap->qp0_qkey = qkey; + } else { + func_cap->qp0_qkey = 0; + } + MLX4_GET(size, outbox, QUERY_FUNC_CAP_QP0_TUNNEL); func_cap->qp0_tunnel_qpn = size & 0xFFFFFF; diff --git a/drivers/net/ethernet/mellanox/mlx4/fw.h b/drivers/net/ethernet/mellanox/mlx4/fw.h index 6811ee00ba7c..1fce03ebe5c4 100644 --- a/drivers/net/ethernet/mellanox/mlx4/fw.h +++ b/drivers/net/ethernet/mellanox/mlx4/fw.h @@ -134,6 +134,7 @@ struct mlx4_func_cap { int max_eq; int reserved_eq; int mcg_quota; + u32 qp0_qkey; u32 qp0_tunnel_qpn; u32 qp0_proxy_qpn; u32 qp1_tunnel_qpn; diff --git a/drivers/net/ethernet/mellanox/mlx4/icm.c b/drivers/net/ethernet/mellanox/mlx4/icm.c index 5fbf4924c272..97c9b1db1d27 100644 --- a/drivers/net/ethernet/mellanox/mlx4/icm.c +++ b/drivers/net/ethernet/mellanox/mlx4/icm.c @@ -245,7 +245,8 @@ int mlx4_UNMAP_ICM_AUX(struct mlx4_dev *dev) MLX4_CMD_TIME_CLASS_B, MLX4_CMD_NATIVE); } -int mlx4_table_get(struct mlx4_dev *dev, struct mlx4_icm_table *table, u32 obj) +int mlx4_table_get(struct mlx4_dev *dev, struct mlx4_icm_table *table, u32 obj, + gfp_t gfp) { u32 i = (obj & (table->num_obj - 1)) / (MLX4_TABLE_CHUNK_SIZE / table->obj_size); @@ -259,7 +260,7 @@ int mlx4_table_get(struct mlx4_dev *dev, struct mlx4_icm_table *table, u32 obj) } table->icm[i] = mlx4_alloc_icm(dev, MLX4_TABLE_CHUNK_SIZE >> PAGE_SHIFT, - (table->lowmem ? GFP_KERNEL : GFP_HIGHUSER) | + (table->lowmem ? gfp : GFP_HIGHUSER) | __GFP_NOWARN, table->coherent); if (!table->icm[i]) { ret = -ENOMEM; @@ -356,7 +357,7 @@ int mlx4_table_get_range(struct mlx4_dev *dev, struct mlx4_icm_table *table, u32 i; for (i = start; i <= end; i += inc) { - err = mlx4_table_get(dev, table, i); + err = mlx4_table_get(dev, table, i, GFP_KERNEL); if (err) goto fail; } diff --git a/drivers/net/ethernet/mellanox/mlx4/icm.h b/drivers/net/ethernet/mellanox/mlx4/icm.h index dee67fa39107..0c7364550150 100644 --- a/drivers/net/ethernet/mellanox/mlx4/icm.h +++ b/drivers/net/ethernet/mellanox/mlx4/icm.h @@ -71,7 +71,8 @@ struct mlx4_icm *mlx4_alloc_icm(struct mlx4_dev *dev, int npages, gfp_t gfp_mask, int coherent); void mlx4_free_icm(struct mlx4_dev *dev, struct mlx4_icm *icm, int coherent); -int mlx4_table_get(struct mlx4_dev *dev, struct mlx4_icm_table *table, u32 obj); +int mlx4_table_get(struct mlx4_dev *dev, struct mlx4_icm_table *table, u32 obj, + gfp_t gfp); void mlx4_table_put(struct mlx4_dev *dev, struct mlx4_icm_table *table, u32 obj); int mlx4_table_get_range(struct mlx4_dev *dev, struct mlx4_icm_table *table, u32 start, u32 end); diff --git a/drivers/net/ethernet/mellanox/mlx4/main.c b/drivers/net/ethernet/mellanox/mlx4/main.c index c187d748115f..26169b3eaed8 100644 --- a/drivers/net/ethernet/mellanox/mlx4/main.c +++ b/drivers/net/ethernet/mellanox/mlx4/main.c @@ -666,13 +666,15 @@ static int mlx4_slave_cap(struct mlx4_dev *dev) return -ENODEV; } + dev->caps.qp0_qkey = kcalloc(dev->caps.num_ports, sizeof(u32), GFP_KERNEL); dev->caps.qp0_tunnel = kcalloc(dev->caps.num_ports, sizeof (u32), GFP_KERNEL); dev->caps.qp0_proxy = kcalloc(dev->caps.num_ports, sizeof (u32), GFP_KERNEL); dev->caps.qp1_tunnel = kcalloc(dev->caps.num_ports, sizeof (u32), GFP_KERNEL); dev->caps.qp1_proxy = kcalloc(dev->caps.num_ports, sizeof (u32), GFP_KERNEL); if (!dev->caps.qp0_tunnel || !dev->caps.qp0_proxy || - !dev->caps.qp1_tunnel || !dev->caps.qp1_proxy) { + !dev->caps.qp1_tunnel || !dev->caps.qp1_proxy || + !dev->caps.qp0_qkey) { err = -ENOMEM; goto err_mem; } @@ -684,6 +686,7 @@ static int mlx4_slave_cap(struct mlx4_dev *dev) " port %d, aborting (%d).\n", i, err); goto err_mem; } + dev->caps.qp0_qkey[i - 1] = func_cap.qp0_qkey; dev->caps.qp0_tunnel[i - 1] = func_cap.qp0_tunnel_qpn; dev->caps.qp0_proxy[i - 1] = func_cap.qp0_proxy_qpn; dev->caps.qp1_tunnel[i - 1] = func_cap.qp1_tunnel_qpn; @@ -729,12 +732,16 @@ static int mlx4_slave_cap(struct mlx4_dev *dev) return 0; err_mem: + kfree(dev->caps.qp0_qkey); kfree(dev->caps.qp0_tunnel); kfree(dev->caps.qp0_proxy); kfree(dev->caps.qp1_tunnel); kfree(dev->caps.qp1_proxy); - dev->caps.qp0_tunnel = dev->caps.qp0_proxy = - dev->caps.qp1_tunnel = dev->caps.qp1_proxy = NULL; + dev->caps.qp0_qkey = NULL; + dev->caps.qp0_tunnel = NULL; + dev->caps.qp0_proxy = NULL; + dev->caps.qp1_tunnel = NULL; + dev->caps.qp1_proxy = NULL; return err; } @@ -1696,6 +1703,14 @@ unmap_bf: unmap_internal_clock(dev); unmap_bf_area(dev); + if (mlx4_is_slave(dev)) { + kfree(dev->caps.qp0_qkey); + kfree(dev->caps.qp0_tunnel); + kfree(dev->caps.qp0_proxy); + kfree(dev->caps.qp1_tunnel); + kfree(dev->caps.qp1_proxy); + } + err_close: if (mlx4_is_slave(dev)) mlx4_slave_exit(dev); @@ -2566,6 +2581,14 @@ err_master_mfunc: if (mlx4_is_master(dev)) mlx4_multi_func_cleanup(dev); + if (mlx4_is_slave(dev)) { + kfree(dev->caps.qp0_qkey); + kfree(dev->caps.qp0_tunnel); + kfree(dev->caps.qp0_proxy); + kfree(dev->caps.qp1_tunnel); + kfree(dev->caps.qp1_proxy); + } + err_close: if (dev->flags & MLX4_FLAG_MSI_X) pci_disable_msix(pdev); @@ -2689,6 +2712,7 @@ static void __mlx4_remove_one(struct pci_dev *pdev) if (!mlx4_is_slave(dev)) mlx4_free_ownership(dev); + kfree(dev->caps.qp0_qkey); kfree(dev->caps.qp0_tunnel); kfree(dev->caps.qp0_proxy); kfree(dev->caps.qp1_tunnel); diff --git a/drivers/net/ethernet/mellanox/mlx4/mlx4.h b/drivers/net/ethernet/mellanox/mlx4/mlx4.h index 8e9eb02e09cb..7a0665beebb1 100644 --- a/drivers/net/ethernet/mellanox/mlx4/mlx4.h +++ b/drivers/net/ethernet/mellanox/mlx4/mlx4.h @@ -133,6 +133,11 @@ enum { MLX4_COMM_CMD_FLR = 254 }; +enum { + MLX4_VF_SMI_DISABLED, + MLX4_VF_SMI_ENABLED +}; + /*The flag indicates that the slave should delay the RESET cmd*/ #define MLX4_DELAY_RESET_SLAVE 0xbbbbbbb /*indicates how many retries will be done if we are in the middle of FLR*/ @@ -488,6 +493,7 @@ struct mlx4_vport_state { struct mlx4_vf_admin_state { struct mlx4_vport_state vport[MLX4_MAX_PORTS + 1]; + u8 enable_smi[MLX4_MAX_PORTS + 1]; }; struct mlx4_vport_oper_state { @@ -495,8 +501,10 @@ struct mlx4_vport_oper_state { int mac_idx; int vlan_idx; }; + struct mlx4_vf_oper_state { struct mlx4_vport_oper_state vport[MLX4_MAX_PORTS + 1]; + u8 smi_enabled[MLX4_MAX_PORTS + 1]; }; struct slave_list { @@ -895,7 +903,7 @@ void mlx4_cleanup_cq_table(struct mlx4_dev *dev); void mlx4_cleanup_qp_table(struct mlx4_dev *dev); void mlx4_cleanup_srq_table(struct mlx4_dev *dev); void mlx4_cleanup_mcg_table(struct mlx4_dev *dev); -int __mlx4_qp_alloc_icm(struct mlx4_dev *dev, int qpn); +int __mlx4_qp_alloc_icm(struct mlx4_dev *dev, int qpn, gfp_t gfp); void __mlx4_qp_free_icm(struct mlx4_dev *dev, int qpn); int __mlx4_cq_alloc_icm(struct mlx4_dev *dev, int *cqn); void __mlx4_cq_free_icm(struct mlx4_dev *dev, int cqn); @@ -903,7 +911,7 @@ int __mlx4_srq_alloc_icm(struct mlx4_dev *dev, int *srqn); void __mlx4_srq_free_icm(struct mlx4_dev *dev, int srqn); int __mlx4_mpt_reserve(struct mlx4_dev *dev); void __mlx4_mpt_release(struct mlx4_dev *dev, u32 index); -int __mlx4_mpt_alloc_icm(struct mlx4_dev *dev, u32 index); +int __mlx4_mpt_alloc_icm(struct mlx4_dev *dev, u32 index, gfp_t gfp); void __mlx4_mpt_free_icm(struct mlx4_dev *dev, u32 index); u32 __mlx4_alloc_mtt_range(struct mlx4_dev *dev, int order); void __mlx4_free_mtt_range(struct mlx4_dev *dev, u32 first_seg, int order); diff --git a/drivers/net/ethernet/mellanox/mlx4/mr.c b/drivers/net/ethernet/mellanox/mlx4/mr.c index 24835853b753..4c71dafad217 100644 --- a/drivers/net/ethernet/mellanox/mlx4/mr.c +++ b/drivers/net/ethernet/mellanox/mlx4/mr.c @@ -364,14 +364,14 @@ static void mlx4_mpt_release(struct mlx4_dev *dev, u32 index) __mlx4_mpt_release(dev, index); } -int __mlx4_mpt_alloc_icm(struct mlx4_dev *dev, u32 index) +int __mlx4_mpt_alloc_icm(struct mlx4_dev *dev, u32 index, gfp_t gfp) { struct mlx4_mr_table *mr_table = &mlx4_priv(dev)->mr_table; - return mlx4_table_get(dev, &mr_table->dmpt_table, index); + return mlx4_table_get(dev, &mr_table->dmpt_table, index, gfp); } -static int mlx4_mpt_alloc_icm(struct mlx4_dev *dev, u32 index) +static int mlx4_mpt_alloc_icm(struct mlx4_dev *dev, u32 index, gfp_t gfp) { u64 param = 0; @@ -382,7 +382,7 @@ static int mlx4_mpt_alloc_icm(struct mlx4_dev *dev, u32 index) MLX4_CMD_TIME_CLASS_A, MLX4_CMD_WRAPPED); } - return __mlx4_mpt_alloc_icm(dev, index); + return __mlx4_mpt_alloc_icm(dev, index, gfp); } void __mlx4_mpt_free_icm(struct mlx4_dev *dev, u32 index) @@ -469,7 +469,7 @@ int mlx4_mr_enable(struct mlx4_dev *dev, struct mlx4_mr *mr) struct mlx4_mpt_entry *mpt_entry; int err; - err = mlx4_mpt_alloc_icm(dev, key_to_hw_index(mr->key)); + err = mlx4_mpt_alloc_icm(dev, key_to_hw_index(mr->key), GFP_KERNEL); if (err) return err; @@ -627,13 +627,14 @@ int mlx4_write_mtt(struct mlx4_dev *dev, struct mlx4_mtt *mtt, EXPORT_SYMBOL_GPL(mlx4_write_mtt); int mlx4_buf_write_mtt(struct mlx4_dev *dev, struct mlx4_mtt *mtt, - struct mlx4_buf *buf) + struct mlx4_buf *buf, gfp_t gfp) { u64 *page_list; int err; int i; - page_list = kmalloc(buf->npages * sizeof *page_list, GFP_KERNEL); + page_list = kmalloc(buf->npages * sizeof *page_list, + gfp); if (!page_list) return -ENOMEM; @@ -680,7 +681,7 @@ int mlx4_mw_enable(struct mlx4_dev *dev, struct mlx4_mw *mw) struct mlx4_mpt_entry *mpt_entry; int err; - err = mlx4_mpt_alloc_icm(dev, key_to_hw_index(mw->key)); + err = mlx4_mpt_alloc_icm(dev, key_to_hw_index(mw->key), GFP_KERNEL); if (err) return err; diff --git a/drivers/net/ethernet/mellanox/mlx4/qp.c b/drivers/net/ethernet/mellanox/mlx4/qp.c index fbd32af89c7c..40af61947925 100644 --- a/drivers/net/ethernet/mellanox/mlx4/qp.c +++ b/drivers/net/ethernet/mellanox/mlx4/qp.c @@ -272,29 +272,29 @@ void mlx4_qp_release_range(struct mlx4_dev *dev, int base_qpn, int cnt) } EXPORT_SYMBOL_GPL(mlx4_qp_release_range); -int __mlx4_qp_alloc_icm(struct mlx4_dev *dev, int qpn) +int __mlx4_qp_alloc_icm(struct mlx4_dev *dev, int qpn, gfp_t gfp) { struct mlx4_priv *priv = mlx4_priv(dev); struct mlx4_qp_table *qp_table = &priv->qp_table; int err; - err = mlx4_table_get(dev, &qp_table->qp_table, qpn); + err = mlx4_table_get(dev, &qp_table->qp_table, qpn, gfp); if (err) goto err_out; - err = mlx4_table_get(dev, &qp_table->auxc_table, qpn); + err = mlx4_table_get(dev, &qp_table->auxc_table, qpn, gfp); if (err) goto err_put_qp; - err = mlx4_table_get(dev, &qp_table->altc_table, qpn); + err = mlx4_table_get(dev, &qp_table->altc_table, qpn, gfp); if (err) goto err_put_auxc; - err = mlx4_table_get(dev, &qp_table->rdmarc_table, qpn); + err = mlx4_table_get(dev, &qp_table->rdmarc_table, qpn, gfp); if (err) goto err_put_altc; - err = mlx4_table_get(dev, &qp_table->cmpt_table, qpn); + err = mlx4_table_get(dev, &qp_table->cmpt_table, qpn, gfp); if (err) goto err_put_rdmarc; @@ -316,7 +316,7 @@ err_out: return err; } -static int mlx4_qp_alloc_icm(struct mlx4_dev *dev, int qpn) +static int mlx4_qp_alloc_icm(struct mlx4_dev *dev, int qpn, gfp_t gfp) { u64 param = 0; @@ -326,7 +326,7 @@ static int mlx4_qp_alloc_icm(struct mlx4_dev *dev, int qpn) MLX4_CMD_ALLOC_RES, MLX4_CMD_TIME_CLASS_A, MLX4_CMD_WRAPPED); } - return __mlx4_qp_alloc_icm(dev, qpn); + return __mlx4_qp_alloc_icm(dev, qpn, gfp); } void __mlx4_qp_free_icm(struct mlx4_dev *dev, int qpn) @@ -355,7 +355,7 @@ static void mlx4_qp_free_icm(struct mlx4_dev *dev, int qpn) __mlx4_qp_free_icm(dev, qpn); } -int mlx4_qp_alloc(struct mlx4_dev *dev, int qpn, struct mlx4_qp *qp) +int mlx4_qp_alloc(struct mlx4_dev *dev, int qpn, struct mlx4_qp *qp, gfp_t gfp) { struct mlx4_priv *priv = mlx4_priv(dev); struct mlx4_qp_table *qp_table = &priv->qp_table; @@ -366,7 +366,7 @@ int mlx4_qp_alloc(struct mlx4_dev *dev, int qpn, struct mlx4_qp *qp) qp->qpn = qpn; - err = mlx4_qp_alloc_icm(dev, qpn); + err = mlx4_qp_alloc_icm(dev, qpn, gfp); if (err) return err; diff --git a/drivers/net/ethernet/mellanox/mlx4/resource_tracker.c b/drivers/net/ethernet/mellanox/mlx4/resource_tracker.c index f16e539749c4..2ba3b7623960 100644 --- a/drivers/net/ethernet/mellanox/mlx4/resource_tracker.c +++ b/drivers/net/ethernet/mellanox/mlx4/resource_tracker.c @@ -1533,7 +1533,7 @@ static int qp_alloc_res(struct mlx4_dev *dev, int slave, int op, int cmd, return err; if (!fw_reserved(dev, qpn)) { - err = __mlx4_qp_alloc_icm(dev, qpn); + err = __mlx4_qp_alloc_icm(dev, qpn, GFP_KERNEL); if (err) { res_abort_move(dev, slave, RES_QP, qpn); return err; @@ -1620,7 +1620,7 @@ static int mpt_alloc_res(struct mlx4_dev *dev, int slave, int op, int cmd, if (err) return err; - err = __mlx4_mpt_alloc_icm(dev, mpt->key); + err = __mlx4_mpt_alloc_icm(dev, mpt->key, GFP_KERNEL); if (err) { res_abort_move(dev, slave, RES_MPT, id); return err; @@ -2828,10 +2828,12 @@ static int get_containing_mtt(struct mlx4_dev *dev, int slave, int start, } static int verify_qp_parameters(struct mlx4_dev *dev, + struct mlx4_vhcr *vhcr, struct mlx4_cmd_mailbox *inbox, enum qp_transition transition, u8 slave) { u32 qp_type; + u32 qpn; struct mlx4_qp_context *qp_ctx; enum mlx4_qp_optpar optpar; int port; @@ -2874,8 +2876,22 @@ static int verify_qp_parameters(struct mlx4_dev *dev, default: break; } + break; + case MLX4_QP_ST_MLX: + qpn = vhcr->in_modifier & 0x7fffff; + port = (qp_ctx->pri_path.sched_queue >> 6 & 1) + 1; + if (transition == QP_TRANS_INIT2RTR && + slave != mlx4_master_func_num(dev) && + mlx4_is_qp_reserved(dev, qpn) && + !mlx4_vf_smi_enabled(dev, slave, port)) { + /* only enabled VFs may create MLX proxy QPs */ + mlx4_err(dev, "%s: unprivileged slave %d attempting to create an MLX proxy special QP on port %d\n", + __func__, slave, port); + return -EPERM; + } break; + default: break; } @@ -3455,7 +3471,7 @@ int mlx4_INIT2RTR_QP_wrapper(struct mlx4_dev *dev, int slave, err = adjust_qp_sched_queue(dev, slave, qpc, inbox); if (err) return err; - err = verify_qp_parameters(dev, inbox, QP_TRANS_INIT2RTR, slave); + err = verify_qp_parameters(dev, vhcr, inbox, QP_TRANS_INIT2RTR, slave); if (err) return err; @@ -3509,7 +3525,7 @@ int mlx4_RTR2RTS_QP_wrapper(struct mlx4_dev *dev, int slave, err = adjust_qp_sched_queue(dev, slave, context, inbox); if (err) return err; - err = verify_qp_parameters(dev, inbox, QP_TRANS_RTR2RTS, slave); + err = verify_qp_parameters(dev, vhcr, inbox, QP_TRANS_RTR2RTS, slave); if (err) return err; @@ -3531,7 +3547,7 @@ int mlx4_RTS2RTS_QP_wrapper(struct mlx4_dev *dev, int slave, err = adjust_qp_sched_queue(dev, slave, context, inbox); if (err) return err; - err = verify_qp_parameters(dev, inbox, QP_TRANS_RTS2RTS, slave); + err = verify_qp_parameters(dev, vhcr, inbox, QP_TRANS_RTS2RTS, slave); if (err) return err; @@ -3568,7 +3584,7 @@ int mlx4_SQD2SQD_QP_wrapper(struct mlx4_dev *dev, int slave, err = adjust_qp_sched_queue(dev, slave, context, inbox); if (err) return err; - err = verify_qp_parameters(dev, inbox, QP_TRANS_SQD2SQD, slave); + err = verify_qp_parameters(dev, vhcr, inbox, QP_TRANS_SQD2SQD, slave); if (err) return err; @@ -3590,7 +3606,7 @@ int mlx4_SQD2RTS_QP_wrapper(struct mlx4_dev *dev, int slave, err = adjust_qp_sched_queue(dev, slave, context, inbox); if (err) return err; - err = verify_qp_parameters(dev, inbox, QP_TRANS_SQD2RTS, slave); + err = verify_qp_parameters(dev, vhcr, inbox, QP_TRANS_SQD2RTS, slave); if (err) return err; diff --git a/drivers/net/ethernet/mellanox/mlx4/srq.c b/drivers/net/ethernet/mellanox/mlx4/srq.c index 98faf870b0b0..67146624eb58 100644 --- a/drivers/net/ethernet/mellanox/mlx4/srq.c +++ b/drivers/net/ethernet/mellanox/mlx4/srq.c @@ -103,11 +103,11 @@ int __mlx4_srq_alloc_icm(struct mlx4_dev *dev, int *srqn) if (*srqn == -1) return -ENOMEM; - err = mlx4_table_get(dev, &srq_table->table, *srqn); + err = mlx4_table_get(dev, &srq_table->table, *srqn, GFP_KERNEL); if (err) goto err_out; - err = mlx4_table_get(dev, &srq_table->cmpt_table, *srqn); + err = mlx4_table_get(dev, &srq_table->cmpt_table, *srqn, GFP_KERNEL); if (err) goto err_put; return 0; diff --git a/drivers/net/ethernet/mellanox/mlx5/core/mr.c b/drivers/net/ethernet/mellanox/mlx5/core/mr.c index 4cc927649404..ac52a0fe2d3a 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/mr.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/mr.c @@ -82,7 +82,11 @@ int mlx5_core_create_mkey(struct mlx5_core_dev *dev, struct mlx5_core_mr *mr, return mlx5_cmd_status_to_err(&lout.hdr); } + mr->iova = be64_to_cpu(in->seg.start_addr); + mr->size = be64_to_cpu(in->seg.len); mr->key = mlx5_idx_to_mkey(be32_to_cpu(lout.mkey) & 0xffffff) | key; + mr->pd = be32_to_cpu(in->seg.flags_pd) & 0xffffff; + mlx5_core_dbg(dev, "out 0x%x, key 0x%x, mkey 0x%x\n", be32_to_cpu(lout.mkey), key, mr->key); diff --git a/drivers/net/wireless/rsi/rsi_91x_sdio.c b/drivers/net/wireless/rsi/rsi_91x_sdio.c index 2e39d38d6a9e..46e7af446f01 100644 --- a/drivers/net/wireless/rsi/rsi_91x_sdio.c +++ b/drivers/net/wireless/rsi/rsi_91x_sdio.c @@ -285,7 +285,6 @@ static void rsi_reset_card(struct sdio_func *pfunction) if (err) { rsi_dbg(ERR_ZONE, "%s: CCCR speed reg read failed: %d\n", __func__, err); - card->state &= ~MMC_STATE_HIGHSPEED; } else { err = rsi_cmd52writebyte(card, SDIO_CCCR_SPEED, @@ -296,14 +295,13 @@ static void rsi_reset_card(struct sdio_func *pfunction) __func__, err); return; } - mmc_card_set_highspeed(card); host->ios.timing = MMC_TIMING_SD_HS; host->ops->set_ios(host, &host->ios); } } /* Set clock */ - if (mmc_card_highspeed(card)) + if (mmc_card_hs(card)) clock = 50000000; else clock = card->cis.max_dtr; diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index 436a76ab4bb1..212c63d780e7 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -3179,14 +3179,7 @@ static int pci_pm_reset(struct pci_dev *dev, int probe) return 0; } -/** - * pci_reset_bridge_secondary_bus - Reset the secondary bus on a PCI bridge. - * @dev: Bridge device - * - * Use the bridge control register to assert reset on the secondary bus. - * Devices on the secondary bus are left in power-on state. - */ -void pci_reset_bridge_secondary_bus(struct pci_dev *dev) +void __weak pcibios_reset_secondary_bus(struct pci_dev *dev) { u16 ctrl; @@ -3211,6 +3204,18 @@ void pci_reset_bridge_secondary_bus(struct pci_dev *dev) */ ssleep(1); } + +/** + * pci_reset_bridge_secondary_bus - Reset the secondary bus on a PCI bridge. + * @dev: Bridge device + * + * Use the bridge control register to assert reset on the secondary bus. + * Devices on the secondary bus are left in power-on state. + */ +void pci_reset_bridge_secondary_bus(struct pci_dev *dev) +{ + pcibios_reset_secondary_bus(dev); +} EXPORT_SYMBOL_GPL(pci_reset_bridge_secondary_bus); static int pci_parent_bus_reset(struct pci_dev *dev, int probe) diff --git a/drivers/platform/chrome/chromeos_laptop.c b/drivers/platform/chrome/chromeos_laptop.c index 7f3aad0e115c..7f1a2e2711bd 100644 --- a/drivers/platform/chrome/chromeos_laptop.c +++ b/drivers/platform/chrome/chromeos_laptop.c @@ -84,21 +84,19 @@ static struct i2c_board_info tsl2563_als_device = { I2C_BOARD_INFO("tsl2563", TAOS_ALS_I2C_ADDR), }; +static int mxt_t19_keys[] = { + KEY_RESERVED, + KEY_RESERVED, + KEY_RESERVED, + KEY_RESERVED, + KEY_RESERVED, + BTN_LEFT +}; + static struct mxt_platform_data atmel_224s_tp_platform_data = { - .x_line = 18, - .y_line = 12, - .x_size = 102*20, - .y_size = 68*20, - .blen = 0x80, /* Gain setting is in upper 4 bits */ - .threshold = 0x32, - .voltage = 0, /* 3.3V */ - .orient = MXT_VERTICAL_FLIP, .irqflags = IRQF_TRIGGER_FALLING, - .is_tp = true, - .key_map = { KEY_RESERVED, - KEY_RESERVED, - KEY_RESERVED, - BTN_LEFT }, + .t19_num_keys = ARRAY_SIZE(mxt_t19_keys), + .t19_keymap = mxt_t19_keys, .config = NULL, .config_length = 0, }; @@ -110,16 +108,7 @@ static struct i2c_board_info atmel_224s_tp_device = { }; static struct mxt_platform_data atmel_1664s_platform_data = { - .x_line = 32, - .y_line = 50, - .x_size = 1700, - .y_size = 2560, - .blen = 0x89, /* Gain setting is in upper 4 bits */ - .threshold = 0x28, - .voltage = 0, /* 3.3V */ - .orient = MXT_ROTATED_90_COUNTER, .irqflags = IRQF_TRIGGER_FALLING, - .is_tp = false, .config = NULL, .config_length = 0, }; diff --git a/drivers/platform/x86/Kconfig b/drivers/platform/x86/Kconfig index 27df2c533b09..172f26ce59ac 100644 --- a/drivers/platform/x86/Kconfig +++ b/drivers/platform/x86/Kconfig @@ -102,7 +102,7 @@ config DELL_LAPTOP default n ---help--- This driver adds support for rfkill and backlight control to Dell - laptops. + laptops (except for some models covered by the Compal driver). config DELL_WMI tristate "Dell WMI extras" @@ -127,6 +127,16 @@ config DELL_WMI_AIO To compile this driver as a module, choose M here: the module will be called dell-wmi-aio. +config DELL_SMO8800 + tristate "Dell Latitude freefall driver (ACPI SMO8800/SMO8810)" + depends on ACPI + ---help--- + Say Y here if you want to support SMO8800/SMO8810 freefall device + on Dell Latitude laptops. + + To compile this driver as a module, choose M here: the module will + be called dell-smo8800. + config FUJITSU_LAPTOP tristate "Fujitsu Laptop Extras" @@ -265,23 +275,21 @@ config PANASONIC_LAPTOP R2, R3, R5, T2, W2 and Y2 series), say Y. config COMPAL_LAPTOP - tristate "Compal Laptop Extras" + tristate "Compal (and others) Laptop Extras" depends on ACPI depends on BACKLIGHT_CLASS_DEVICE depends on RFKILL depends on HWMON depends on POWER_SUPPLY ---help--- - This is a driver for laptops built by Compal: - - Compal FL90/IFL90 - Compal FL91/IFL91 - Compal FL92/JFL92 - Compal FT00/IFT00 + This is a driver for laptops built by Compal, and some models by + other brands (e.g. Dell, Toshiba). - It adds support for Bluetooth, WLAN and LCD brightness control. + It adds support for rfkill, Bluetooth, WLAN and LCD brightness + control. - If you have an Compal FL9x/IFL9x/FT00 laptop, say Y or M here. + For a (possibly incomplete) list of supported laptops, please refer + to: Documentation/platform/x86-laptop-drivers.txt config SONY_LAPTOP tristate "Sony Laptop Extras" @@ -724,7 +732,7 @@ config IBM_RTL config XO1_RFKILL tristate "OLPC XO-1 software RF kill switch" - depends on OLPC + depends on OLPC || COMPILE_TEST depends on RFKILL ---help--- Support for enabling/disabling the WLAN interface on the OLPC XO-1 @@ -732,6 +740,7 @@ config XO1_RFKILL config XO15_EBOOK tristate "OLPC XO-1.5 ebook switch" + depends on OLPC || COMPILE_TEST depends on ACPI && INPUT ---help--- Support for the ebook switch on the OLPC XO-1.5 laptop. diff --git a/drivers/platform/x86/Makefile b/drivers/platform/x86/Makefile index 1a2eafc9d48e..c4ca428fd3db 100644 --- a/drivers/platform/x86/Makefile +++ b/drivers/platform/x86/Makefile @@ -13,6 +13,7 @@ obj-$(CONFIG_COMPAL_LAPTOP) += compal-laptop.o obj-$(CONFIG_DELL_LAPTOP) += dell-laptop.o obj-$(CONFIG_DELL_WMI) += dell-wmi.o obj-$(CONFIG_DELL_WMI_AIO) += dell-wmi-aio.o +obj-$(CONFIG_DELL_SMO8800) += dell-smo8800.o obj-$(CONFIG_ACER_WMI) += acer-wmi.o obj-$(CONFIG_ACERHDF) += acerhdf.o obj-$(CONFIG_HP_ACCEL) += hp_accel.o diff --git a/drivers/platform/x86/alienware-wmi.c b/drivers/platform/x86/alienware-wmi.c index 541f9514f76f..297b6640213f 100644 --- a/drivers/platform/x86/alienware-wmi.c +++ b/drivers/platform/x86/alienware-wmi.c @@ -32,6 +32,7 @@ #define WMAX_METHOD_HDMI_STATUS 0x2 #define WMAX_METHOD_BRIGHTNESS 0x3 #define WMAX_METHOD_ZONE_CONTROL 0x4 +#define WMAX_METHOD_HDMI_CABLE 0x5 MODULE_AUTHOR("Mario Limonciello <mario_limonciello@dell.com>"); MODULE_DESCRIPTION("Alienware special feature control"); @@ -350,12 +351,11 @@ static int alienware_zone_init(struct platform_device *dev) char *name; if (interface == WMAX) { - global_led.max_brightness = 100; lighting_control_state = WMAX_RUNNING; } else if (interface == LEGACY) { - global_led.max_brightness = 0x0F; lighting_control_state = LEGACY_RUNNING; } + global_led.max_brightness = 0x0F; global_brightness = global_led.max_brightness; /* @@ -423,41 +423,85 @@ static void alienware_zone_exit(struct platform_device *dev) The HDMI mux sysfs node indicates the status of the HDMI input mux. It can toggle between standard system GPU output and HDMI input. */ -static ssize_t show_hdmi(struct device *dev, struct device_attribute *attr, - char *buf) +static acpi_status alienware_hdmi_command(struct hdmi_args *in_args, + u32 command, int *out_data) { acpi_status status; - struct acpi_buffer input; union acpi_object *obj; - u32 tmp = 0; - struct acpi_buffer output = { ACPI_ALLOCATE_BUFFER, NULL }; + struct acpi_buffer input; + struct acpi_buffer output; + + input.length = (acpi_size) sizeof(*in_args); + input.pointer = in_args; + if (out_data != NULL) { + output.length = ACPI_ALLOCATE_BUFFER; + output.pointer = NULL; + status = wmi_evaluate_method(WMAX_CONTROL_GUID, 1, + command, &input, &output); + } else + status = wmi_evaluate_method(WMAX_CONTROL_GUID, 1, + command, &input, NULL); + + if (ACPI_SUCCESS(status) && out_data != NULL) { + obj = (union acpi_object *)output.pointer; + if (obj && obj->type == ACPI_TYPE_INTEGER) + *out_data = (u32) obj->integer.value; + } + return status; + +} + +static ssize_t show_hdmi_cable(struct device *dev, + struct device_attribute *attr, char *buf) +{ + acpi_status status; + u32 out_data; struct hdmi_args in_args = { .arg = 0, }; - input.length = (acpi_size) sizeof(in_args); - input.pointer = &in_args; - status = wmi_evaluate_method(WMAX_CONTROL_GUID, 1, - WMAX_METHOD_HDMI_STATUS, &input, &output); + status = + alienware_hdmi_command(&in_args, WMAX_METHOD_HDMI_CABLE, + (u32 *) &out_data); + if (ACPI_SUCCESS(status)) { + if (out_data == 0) + return scnprintf(buf, PAGE_SIZE, + "[unconnected] connected unknown\n"); + else if (out_data == 1) + return scnprintf(buf, PAGE_SIZE, + "unconnected [connected] unknown\n"); + } + pr_err("alienware-wmi: unknown HDMI cable status: %d\n", status); + return scnprintf(buf, PAGE_SIZE, "unconnected connected [unknown]\n"); +} + +static ssize_t show_hdmi_source(struct device *dev, + struct device_attribute *attr, char *buf) +{ + acpi_status status; + u32 out_data; + struct hdmi_args in_args = { + .arg = 0, + }; + status = + alienware_hdmi_command(&in_args, WMAX_METHOD_HDMI_STATUS, + (u32 *) &out_data); if (ACPI_SUCCESS(status)) { - obj = (union acpi_object *)output.pointer; - if (obj && obj->type == ACPI_TYPE_INTEGER) - tmp = (u32) obj->integer.value; - if (tmp == 1) + if (out_data == 1) return scnprintf(buf, PAGE_SIZE, "[input] gpu unknown\n"); - else if (tmp == 2) + else if (out_data == 2) return scnprintf(buf, PAGE_SIZE, "input [gpu] unknown\n"); } - pr_err("alienware-wmi: unknown HDMI status: %d\n", status); + pr_err("alienware-wmi: unknown HDMI source status: %d\n", out_data); return scnprintf(buf, PAGE_SIZE, "input gpu [unknown]\n"); } -static ssize_t toggle_hdmi(struct device *dev, struct device_attribute *attr, - const char *buf, size_t count) +static ssize_t toggle_hdmi_source(struct device *dev, + struct device_attribute *attr, + const char *buf, size_t count) { - struct acpi_buffer input; acpi_status status; struct hdmi_args args; if (strcmp(buf, "gpu\n") == 0) @@ -467,33 +511,46 @@ static ssize_t toggle_hdmi(struct device *dev, struct device_attribute *attr, else args.arg = 3; pr_debug("alienware-wmi: setting hdmi to %d : %s", args.arg, buf); - input.length = (acpi_size) sizeof(args); - input.pointer = &args; - status = wmi_evaluate_method(WMAX_CONTROL_GUID, 1, - WMAX_METHOD_HDMI_SOURCE, &input, NULL); + + status = alienware_hdmi_command(&args, WMAX_METHOD_HDMI_SOURCE, NULL); + if (ACPI_FAILURE(status)) pr_err("alienware-wmi: HDMI toggle failed: results: %u\n", status); return count; } -static DEVICE_ATTR(hdmi, S_IRUGO | S_IWUSR, show_hdmi, toggle_hdmi); +static DEVICE_ATTR(cable, S_IRUGO, show_hdmi_cable, NULL); +static DEVICE_ATTR(source, S_IRUGO | S_IWUSR, show_hdmi_source, + toggle_hdmi_source); + +static struct attribute *hdmi_attrs[] = { + &dev_attr_cable.attr, + &dev_attr_source.attr, + NULL, +}; -static void remove_hdmi(struct platform_device *device) +static struct attribute_group hdmi_attribute_group = { + .name = "hdmi", + .attrs = hdmi_attrs, +}; + +static void remove_hdmi(struct platform_device *dev) { - device_remove_file(&device->dev, &dev_attr_hdmi); + sysfs_remove_group(&dev->dev.kobj, &hdmi_attribute_group); } -static int create_hdmi(void) +static int create_hdmi(struct platform_device *dev) { - int ret = -ENOMEM; - ret = device_create_file(&platform_device->dev, &dev_attr_hdmi); + int ret; + + ret = sysfs_create_group(&dev->dev.kobj, &hdmi_attribute_group); if (ret) goto error_create_hdmi; return 0; error_create_hdmi: - remove_hdmi(platform_device); + remove_hdmi(dev); return ret; } @@ -527,7 +584,7 @@ static int __init alienware_wmi_init(void) goto fail_platform_device2; if (interface == WMAX) { - ret = create_hdmi(); + ret = create_hdmi(platform_device); if (ret) goto fail_prep_hdmi; } diff --git a/drivers/platform/x86/asus-nb-wmi.c b/drivers/platform/x86/asus-nb-wmi.c index 563f59efa669..ddf0eefd862c 100644 --- a/drivers/platform/x86/asus-nb-wmi.c +++ b/drivers/platform/x86/asus-nb-wmi.c @@ -137,6 +137,15 @@ static struct dmi_system_id asus_quirks[] = { }, { .callback = dmi_matched, + .ident = "ASUSTeK COMPUTER INC. X550CA", + .matches = { + DMI_MATCH(DMI_SYS_VENDOR, "ASUSTeK COMPUTER INC."), + DMI_MATCH(DMI_PRODUCT_NAME, "X550CA"), + }, + .driver_data = &quirk_asus_x401u, + }, + { + .callback = dmi_matched, .ident = "ASUSTeK COMPUTER INC. X55A", .matches = { DMI_MATCH(DMI_SYS_VENDOR, "ASUSTeK COMPUTER INC."), diff --git a/drivers/platform/x86/asus-wmi.c b/drivers/platform/x86/asus-wmi.c index 91ef69a52263..3c6ccedc82b6 100644 --- a/drivers/platform/x86/asus-wmi.c +++ b/drivers/platform/x86/asus-wmi.c @@ -266,7 +266,7 @@ static int asus_wmi_evaluate_method(u32 method_id, u32 arg0, u32 arg1, struct acpi_buffer output = { ACPI_ALLOCATE_BUFFER, NULL }; acpi_status status; union acpi_object *obj; - u32 tmp; + u32 tmp = 0; status = wmi_evaluate_method(ASUS_WMI_MGMT_GUID, 1, method_id, &input, &output); @@ -277,8 +277,6 @@ static int asus_wmi_evaluate_method(u32 method_id, u32 arg0, u32 arg1, obj = (union acpi_object *)output.pointer; if (obj && obj->type == ACPI_TYPE_INTEGER) tmp = (u32) obj->integer.value; - else - tmp = 0; if (retval) *retval = tmp; diff --git a/drivers/platform/x86/dell-smo8800.c b/drivers/platform/x86/dell-smo8800.c new file mode 100644 index 000000000000..a653716055d1 --- /dev/null +++ b/drivers/platform/x86/dell-smo8800.c @@ -0,0 +1,233 @@ +/* + * dell-smo8800.c - Dell Latitude ACPI SMO8800/SMO8810 freefall sensor driver + * + * Copyright (C) 2012 Sonal Santan <sonal.santan@gmail.com> + * Copyright (C) 2014 Pali Rohár <pali.rohar@gmail.com> + * + * This is loosely based on lis3lv02d driver. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + */ + +#define DRIVER_NAME "smo8800" + +#include <linux/kernel.h> +#include <linux/module.h> +#include <linux/acpi.h> +#include <linux/interrupt.h> +#include <linux/miscdevice.h> + +struct smo8800_device { + u32 irq; /* acpi device irq */ + atomic_t counter; /* count after last read */ + struct miscdevice miscdev; /* for /dev/freefall */ + unsigned long misc_opened; /* whether the device is open */ + wait_queue_head_t misc_wait; /* Wait queue for the misc dev */ + struct device *dev; /* acpi device */ +}; + +static irqreturn_t smo8800_interrupt_quick(int irq, void *data) +{ + struct smo8800_device *smo8800 = data; + + atomic_inc(&smo8800->counter); + wake_up_interruptible(&smo8800->misc_wait); + return IRQ_WAKE_THREAD; +} + +static irqreturn_t smo8800_interrupt_thread(int irq, void *data) +{ + struct smo8800_device *smo8800 = data; + + dev_info(smo8800->dev, "detected free fall\n"); + return IRQ_HANDLED; +} + +static acpi_status smo8800_get_resource(struct acpi_resource *resource, + void *context) +{ + struct acpi_resource_extended_irq *irq; + + if (resource->type != ACPI_RESOURCE_TYPE_EXTENDED_IRQ) + return AE_OK; + + irq = &resource->data.extended_irq; + if (!irq || !irq->interrupt_count) + return AE_OK; + + *((u32 *)context) = irq->interrupts[0]; + return AE_CTRL_TERMINATE; +} + +static u32 smo8800_get_irq(struct acpi_device *device) +{ + u32 irq = 0; + acpi_status status; + + status = acpi_walk_resources(device->handle, METHOD_NAME__CRS, + smo8800_get_resource, &irq); + if (ACPI_FAILURE(status)) { + dev_err(&device->dev, "acpi_walk_resources failed\n"); + return 0; + } + + return irq; +} + +static ssize_t smo8800_misc_read(struct file *file, char __user *buf, + size_t count, loff_t *pos) +{ + struct smo8800_device *smo8800 = container_of(file->private_data, + struct smo8800_device, miscdev); + + u32 data = 0; + unsigned char byte_data = 0; + ssize_t retval = 1; + + if (count < 1) + return -EINVAL; + + atomic_set(&smo8800->counter, 0); + retval = wait_event_interruptible(smo8800->misc_wait, + (data = atomic_xchg(&smo8800->counter, 0))); + + if (retval) + return retval; + + byte_data = 1; + retval = 1; + + if (data < 255) + byte_data = data; + else + byte_data = 255; + + if (put_user(byte_data, buf)) + retval = -EFAULT; + + return retval; +} + +static int smo8800_misc_open(struct inode *inode, struct file *file) +{ + struct smo8800_device *smo8800 = container_of(file->private_data, + struct smo8800_device, miscdev); + + if (test_and_set_bit(0, &smo8800->misc_opened)) + return -EBUSY; /* already open */ + + atomic_set(&smo8800->counter, 0); + return 0; +} + +static int smo8800_misc_release(struct inode *inode, struct file *file) +{ + struct smo8800_device *smo8800 = container_of(file->private_data, + struct smo8800_device, miscdev); + + clear_bit(0, &smo8800->misc_opened); /* release the device */ + return 0; +} + +static const struct file_operations smo8800_misc_fops = { + .owner = THIS_MODULE, + .read = smo8800_misc_read, + .open = smo8800_misc_open, + .release = smo8800_misc_release, +}; + +static int smo8800_add(struct acpi_device *device) +{ + int err; + struct smo8800_device *smo8800; + + smo8800 = devm_kzalloc(&device->dev, sizeof(*smo8800), GFP_KERNEL); + if (!smo8800) { + dev_err(&device->dev, "failed to allocate device data\n"); + return -ENOMEM; + } + + smo8800->dev = &device->dev; + smo8800->miscdev.minor = MISC_DYNAMIC_MINOR; + smo8800->miscdev.name = "freefall"; + smo8800->miscdev.fops = &smo8800_misc_fops; + + init_waitqueue_head(&smo8800->misc_wait); + + err = misc_register(&smo8800->miscdev); + if (err) { + dev_err(&device->dev, "failed to register misc dev: %d\n", err); + return err; + } + + device->driver_data = smo8800; + + smo8800->irq = smo8800_get_irq(device); + if (!smo8800->irq) { + dev_err(&device->dev, "failed to obtain IRQ\n"); + err = -EINVAL; + goto error; + } + + err = request_threaded_irq(smo8800->irq, smo8800_interrupt_quick, + smo8800_interrupt_thread, + IRQF_TRIGGER_RISING | IRQF_ONESHOT, + DRIVER_NAME, smo8800); + if (err) { + dev_err(&device->dev, + "failed to request thread for IRQ %d: %d\n", + smo8800->irq, err); + goto error; + } + + dev_dbg(&device->dev, "device /dev/freefall registered with IRQ %d\n", + smo8800->irq); + return 0; + +error: + misc_deregister(&smo8800->miscdev); + return err; +} + +static int smo8800_remove(struct acpi_device *device) +{ + struct smo8800_device *smo8800 = device->driver_data; + + free_irq(smo8800->irq, smo8800); + misc_deregister(&smo8800->miscdev); + dev_dbg(&device->dev, "device /dev/freefall unregistered\n"); + return 0; +} + +static const struct acpi_device_id smo8800_ids[] = { + { "SMO8800", 0 }, + { "SMO8810", 0 }, + { "", 0 }, +}; + +MODULE_DEVICE_TABLE(acpi, smo8800_ids); + +static struct acpi_driver smo8800_driver = { + .name = DRIVER_NAME, + .class = "Latitude", + .ids = smo8800_ids, + .ops = { + .add = smo8800_add, + .remove = smo8800_remove, + }, + .owner = THIS_MODULE, +}; + +module_acpi_driver(smo8800_driver); + +MODULE_DESCRIPTION("Dell Latitude freefall driver (ACPI SMO8800/SMO8810)"); +MODULE_LICENSE("GPL"); +MODULE_AUTHOR("Sonal Santan, Pali Rohár"); diff --git a/drivers/platform/x86/hp-wmi.c b/drivers/platform/x86/hp-wmi.c index 8ba8956b5a48..484a8673b835 100644 --- a/drivers/platform/x86/hp-wmi.c +++ b/drivers/platform/x86/hp-wmi.c @@ -53,6 +53,7 @@ MODULE_ALIAS("wmi:5FB7F034-2C63-45e9-BE91-3D44E2C707E4"); #define HPWMI_ALS_QUERY 0x3 #define HPWMI_HARDWARE_QUERY 0x4 #define HPWMI_WIRELESS_QUERY 0x5 +#define HPWMI_BIOS_QUERY 0x9 #define HPWMI_HOTKEY_QUERY 0xc #define HPWMI_FEATURE_QUERY 0xd #define HPWMI_WIRELESS2_QUERY 0x1b @@ -144,6 +145,7 @@ static const struct key_entry hp_wmi_keymap[] = { { KE_KEY, 0x2142, { KEY_MEDIA } }, { KE_KEY, 0x213b, { KEY_INFO } }, { KE_KEY, 0x2169, { KEY_DIRECTION } }, + { KE_KEY, 0x216a, { KEY_SETUP } }, { KE_KEY, 0x231b, { KEY_HELP } }, { KE_END, 0 } }; @@ -304,6 +306,19 @@ static int hp_wmi_bios_2009_later(void) return (state & 0x10) ? 1 : 0; } +static int hp_wmi_enable_hotkeys(void) +{ + int ret; + int query = 0x6e; + + ret = hp_wmi_perform_query(HPWMI_BIOS_QUERY, 1, &query, sizeof(query), + 0); + + if (ret) + return -EINVAL; + return 0; +} + static int hp_wmi_set_block(void *data, bool blocked) { enum hp_wmi_radio r = (enum hp_wmi_radio) data; @@ -648,6 +663,9 @@ static int __init hp_wmi_input_setup(void) hp_wmi_tablet_state()); input_sync(hp_wmi_input_dev); + if (hp_wmi_bios_2009_later() == 4) + hp_wmi_enable_hotkeys(); + status = wmi_install_notify_handler(HPWMI_EVENT_GUID, hp_wmi_notify, NULL); if (ACPI_FAILURE(status)) { err = -EIO; diff --git a/drivers/platform/x86/ideapad-laptop.c b/drivers/platform/x86/ideapad-laptop.c index 6dd060a0bb65..b4c495a62eec 100644 --- a/drivers/platform/x86/ideapad-laptop.c +++ b/drivers/platform/x86/ideapad-laptop.c @@ -36,6 +36,8 @@ #include <linux/debugfs.h> #include <linux/seq_file.h> #include <linux/i8042.h> +#include <linux/dmi.h> +#include <linux/device.h> #define IDEAPAD_RFKILL_DEV_NUM (3) @@ -819,6 +821,19 @@ static void ideapad_acpi_notify(acpi_handle handle, u32 event, void *data) } } +/* Blacklist for devices where the ideapad rfkill interface does not work */ +static struct dmi_system_id rfkill_blacklist[] = { + /* The Lenovo Yoga 2 11 always reports everything as blocked */ + { + .ident = "Lenovo Yoga 2 11", + .matches = { + DMI_MATCH(DMI_SYS_VENDOR, "LENOVO"), + DMI_MATCH(DMI_PRODUCT_VERSION, "Lenovo Yoga 2 11"), + }, + }, + {} +}; + static int ideapad_acpi_add(struct platform_device *pdev) { int ret, i; @@ -833,7 +848,7 @@ static int ideapad_acpi_add(struct platform_device *pdev) if (read_method_int(adev->handle, "_CFG", &cfg)) return -ENODEV; - priv = kzalloc(sizeof(*priv), GFP_KERNEL); + priv = devm_kzalloc(&pdev->dev, sizeof(*priv), GFP_KERNEL); if (!priv) return -ENOMEM; @@ -844,7 +859,7 @@ static int ideapad_acpi_add(struct platform_device *pdev) ret = ideapad_sysfs_init(priv); if (ret) - goto sysfs_failed; + return ret; ret = ideapad_debugfs_init(priv); if (ret) @@ -854,11 +869,10 @@ static int ideapad_acpi_add(struct platform_device *pdev) if (ret) goto input_failed; - for (i = 0; i < IDEAPAD_RFKILL_DEV_NUM; i++) { - if (test_bit(ideapad_rfk_data[i].cfgbit, &priv->cfg)) - ideapad_register_rfkill(priv, i); - else - priv->rfk[i] = NULL; + if (!dmi_check_system(rfkill_blacklist)) { + for (i = 0; i < IDEAPAD_RFKILL_DEV_NUM; i++) + if (test_bit(ideapad_rfk_data[i].cfgbit, &priv->cfg)) + ideapad_register_rfkill(priv, i); } ideapad_sync_rfk_state(priv); ideapad_sync_touchpad_state(priv); @@ -884,8 +898,6 @@ input_failed: ideapad_debugfs_exit(priv); debugfs_failed: ideapad_sysfs_exit(priv); -sysfs_failed: - kfree(priv); return ret; } @@ -903,7 +915,6 @@ static int ideapad_acpi_remove(struct platform_device *pdev) ideapad_debugfs_exit(priv); ideapad_sysfs_exit(priv); dev_set_drvdata(&pdev->dev, NULL); - kfree(priv); return 0; } diff --git a/drivers/platform/x86/intel_mid_thermal.c b/drivers/platform/x86/intel_mid_thermal.c index 93fab8b70ce1..ab7860a21a22 100644 --- a/drivers/platform/x86/intel_mid_thermal.c +++ b/drivers/platform/x86/intel_mid_thermal.c @@ -481,7 +481,8 @@ static int mid_thermal_probe(struct platform_device *pdev) int i; struct platform_info *pinfo; - pinfo = kzalloc(sizeof(struct platform_info), GFP_KERNEL); + pinfo = devm_kzalloc(&pdev->dev, sizeof(struct platform_info), + GFP_KERNEL); if (!pinfo) return -ENOMEM; @@ -489,7 +490,6 @@ static int mid_thermal_probe(struct platform_device *pdev) ret = mid_initialize_adc(&pdev->dev); if (ret) { dev_err(&pdev->dev, "ADC init failed"); - kfree(pinfo); return ret; } @@ -520,7 +520,6 @@ err: thermal_zone_device_unregister(pinfo->tzd[i]); } configure_adc(0); - kfree(pinfo); return ret; } @@ -541,8 +540,6 @@ static int mid_thermal_remove(struct platform_device *pdev) thermal_zone_device_unregister(pinfo->tzd[i]); } - kfree(pinfo); - /* Stop the ADC */ return configure_adc(0); } diff --git a/drivers/platform/x86/intel_pmic_gpio.c b/drivers/platform/x86/intel_pmic_gpio.c index 2805988485f6..40929e4f7ad7 100644 --- a/drivers/platform/x86/intel_pmic_gpio.c +++ b/drivers/platform/x86/intel_pmic_gpio.c @@ -91,7 +91,7 @@ static void pmic_program_irqtype(int gpio, int type) static int pmic_gpio_direction_input(struct gpio_chip *chip, unsigned offset) { - if (offset > 8) { + if (offset >= 8) { pr_err("only pin 0-7 support input\n"); return -1;/* we only have 8 GPIO can use as input */ } @@ -130,7 +130,7 @@ static int pmic_gpio_get(struct gpio_chip *chip, unsigned offset) int ret; /* we only have 8 GPIO pins we can use as input */ - if (offset > 8) + if (offset >= 8) return -EOPNOTSUPP; ret = intel_scu_ipc_ioread8(GPIO0 + offset, &r); if (ret < 0) diff --git a/drivers/platform/x86/pvpanic.c b/drivers/platform/x86/pvpanic.c index c9f6e511daa6..073a90a63dbc 100644 --- a/drivers/platform/x86/pvpanic.c +++ b/drivers/platform/x86/pvpanic.c @@ -70,6 +70,7 @@ pvpanic_panic_notify(struct notifier_block *nb, unsigned long code, static struct notifier_block pvpanic_panic_nb = { .notifier_call = pvpanic_panic_notify, + .priority = 1, /* let this called before broken drm_fb_helper */ }; diff --git a/drivers/platform/x86/samsung-laptop.c b/drivers/platform/x86/samsung-laptop.c index d1f030053176..5a5966512277 100644 --- a/drivers/platform/x86/samsung-laptop.c +++ b/drivers/platform/x86/samsung-laptop.c @@ -27,6 +27,7 @@ #include <linux/debugfs.h> #include <linux/ctype.h> #include <linux/efi.h> +#include <linux/suspend.h> #include <acpi/video.h> /* @@ -340,6 +341,8 @@ struct samsung_laptop { struct samsung_laptop_debug debug; struct samsung_quirks *quirks; + struct notifier_block pm_nb; + bool handle_backlight; bool has_stepping_quirk; @@ -348,6 +351,8 @@ struct samsung_laptop { struct samsung_quirks { bool broken_acpi_video; + bool four_kbd_backlight_levels; + bool enable_kbd_backlight; }; static struct samsung_quirks samsung_unknown = {}; @@ -356,6 +361,11 @@ static struct samsung_quirks samsung_broken_acpi_video = { .broken_acpi_video = true, }; +static struct samsung_quirks samsung_np740u3e = { + .four_kbd_backlight_levels = true, + .enable_kbd_backlight = true, +}; + static bool force; module_param(force, bool, 0); MODULE_PARM_DESC(force, @@ -1051,6 +1061,8 @@ static int __init samsung_leds_init(struct samsung_laptop *samsung) samsung->kbd_led.brightness_set = kbd_led_set; samsung->kbd_led.brightness_get = kbd_led_get; samsung->kbd_led.max_brightness = 8; + if (samsung->quirks->four_kbd_backlight_levels) + samsung->kbd_led.max_brightness = 4; ret = led_classdev_register(&samsung->platform_device->dev, &samsung->kbd_led); @@ -1414,6 +1426,19 @@ static void samsung_platform_exit(struct samsung_laptop *samsung) } } +static int samsung_pm_notification(struct notifier_block *nb, + unsigned long val, void *ptr) +{ + struct samsung_laptop *samsung; + + samsung = container_of(nb, struct samsung_laptop, pm_nb); + if (val == PM_POST_HIBERNATION && + samsung->quirks->enable_kbd_backlight) + kbd_backlight_enable(samsung); + + return 0; +} + static int __init samsung_platform_init(struct samsung_laptop *samsung) { struct platform_device *pdev; @@ -1534,6 +1559,15 @@ static struct dmi_system_id __initdata samsung_dmi_table[] = { }, .driver_data = &samsung_broken_acpi_video, }, + { + .callback = samsung_dmi_matched, + .ident = "730U3E/740U3E", + .matches = { + DMI_MATCH(DMI_SYS_VENDOR, "SAMSUNG ELECTRONICS CO., LTD."), + DMI_MATCH(DMI_PRODUCT_NAME, "730U3E/740U3E"), + }, + .driver_data = &samsung_np740u3e, + }, { }, }; MODULE_DEVICE_TABLE(dmi, samsung_dmi_table); @@ -1608,6 +1642,9 @@ static int __init samsung_init(void) if (ret) goto error_debugfs; + samsung->pm_nb.notifier_call = samsung_pm_notification; + register_pm_notifier(&samsung->pm_nb); + samsung_platform_device = samsung->platform_device; return ret; @@ -1633,6 +1670,7 @@ static void __exit samsung_exit(void) struct samsung_laptop *samsung; samsung = platform_get_drvdata(samsung_platform_device); + unregister_pm_notifier(&samsung->pm_nb); samsung_debugfs_exit(samsung); samsung_leds_exit(samsung); diff --git a/drivers/platform/x86/thinkpad_acpi.c b/drivers/platform/x86/thinkpad_acpi.c index 15e61c16736e..d82f196e3cfe 100644 --- a/drivers/platform/x86/thinkpad_acpi.c +++ b/drivers/platform/x86/thinkpad_acpi.c @@ -3171,8 +3171,10 @@ static int __init hotkey_init(struct ibm_init_struct *iibm) KEY_MICMUTE, /* 0x1a: Mic mute (since ?400 or so) */ /* (assignments unknown, please report if found) */ - KEY_UNKNOWN, KEY_UNKNOWN, KEY_UNKNOWN, KEY_UNKNOWN, KEY_UNKNOWN, + + /* Extra keys in use since the X240 / T440 / T540 */ + KEY_CONFIG, KEY_SEARCH, KEY_SCALE, KEY_COMPUTER, }, }; diff --git a/drivers/platform/x86/toshiba_acpi.c b/drivers/platform/x86/toshiba_acpi.c index 46473ca7566b..76441dcbe5ff 100644 --- a/drivers/platform/x86/toshiba_acpi.c +++ b/drivers/platform/x86/toshiba_acpi.c @@ -56,6 +56,7 @@ #include <linux/workqueue.h> #include <linux/i8042.h> #include <linux/acpi.h> +#include <linux/dmi.h> #include <asm/uaccess.h> MODULE_AUTHOR("John Belmonte"); @@ -213,6 +214,30 @@ static const struct key_entry toshiba_acpi_keymap[] = { { KE_END, 0 }, }; +/* alternative keymap */ +static const struct dmi_system_id toshiba_alt_keymap_dmi[] = { + { + .matches = { + DMI_MATCH(DMI_SYS_VENDOR, "TOSHIBA"), + DMI_MATCH(DMI_PRODUCT_NAME, "Satellite M840"), + }, + }, + {} +}; + +static const struct key_entry toshiba_acpi_alt_keymap[] = { + { KE_KEY, 0x157, { KEY_MUTE } }, + { KE_KEY, 0x102, { KEY_ZOOMOUT } }, + { KE_KEY, 0x103, { KEY_ZOOMIN } }, + { KE_KEY, 0x139, { KEY_ZOOMRESET } }, + { KE_KEY, 0x13e, { KEY_SWITCHVIDEOMODE } }, + { KE_KEY, 0x13c, { KEY_BRIGHTNESSDOWN } }, + { KE_KEY, 0x13d, { KEY_BRIGHTNESSUP } }, + { KE_KEY, 0x158, { KEY_WLAN } }, + { KE_KEY, 0x13f, { KEY_TOUCHPAD_TOGGLE } }, + { KE_END, 0 }, +}; + /* utility */ @@ -1440,6 +1465,7 @@ static int toshiba_acpi_setup_keyboard(struct toshiba_acpi_dev *dev) acpi_handle ec_handle; int error; u32 hci_result; + const struct key_entry *keymap = toshiba_acpi_keymap; dev->hotkey_dev = input_allocate_device(); if (!dev->hotkey_dev) @@ -1449,7 +1475,9 @@ static int toshiba_acpi_setup_keyboard(struct toshiba_acpi_dev *dev) dev->hotkey_dev->phys = "toshiba_acpi/input0"; dev->hotkey_dev->id.bustype = BUS_HOST; - error = sparse_keymap_setup(dev->hotkey_dev, toshiba_acpi_keymap, NULL); + if (dmi_check_system(toshiba_alt_keymap_dmi)) + keymap = toshiba_acpi_alt_keymap; + error = sparse_keymap_setup(dev->hotkey_dev, keymap, NULL); if (error) goto err_free_dev; diff --git a/drivers/rtc/Kconfig b/drivers/rtc/Kconfig index 71988b69eca6..0754f5c7cb3b 100644 --- a/drivers/rtc/Kconfig +++ b/drivers/rtc/Kconfig @@ -530,11 +530,11 @@ config RTC_DRV_RV3029C2 will be called rtc-rv3029c2. config RTC_DRV_S5M - tristate "Samsung S5M series" + tristate "Samsung S2M/S5M series" depends on MFD_SEC_CORE help If you say yes here you will get support for the - RTC of Samsung S5M PMIC series. + RTC of Samsung S2MPS14 and S5M PMIC series. This driver can also be built as a module. If so, the module will be called rtc-s5m. diff --git a/drivers/rtc/rtc-s5m.c b/drivers/rtc/rtc-s5m.c index 8ec2d6a1dbe1..8f06250a0389 100644 --- a/drivers/rtc/rtc-s5m.c +++ b/drivers/rtc/rtc-s5m.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2013 Samsung Electronics Co., Ltd + * Copyright (c) 2013-2014 Samsung Electronics Co., Ltd * http://www.samsung.com * * Copyright (C) 2013 Google, Inc @@ -17,27 +17,76 @@ #include <linux/module.h> #include <linux/i2c.h> -#include <linux/slab.h> #include <linux/bcd.h> -#include <linux/bitops.h> #include <linux/regmap.h> #include <linux/rtc.h> -#include <linux/delay.h> #include <linux/platform_device.h> #include <linux/mfd/samsung/core.h> #include <linux/mfd/samsung/irq.h> #include <linux/mfd/samsung/rtc.h> +#include <linux/mfd/samsung/s2mps14.h> /* * Maximum number of retries for checking changes in UDR field - * of SEC_RTC_UDR_CON register (to limit possible endless loop). + * of S5M_RTC_UDR_CON register (to limit possible endless loop). * * After writing to RTC registers (setting time or alarm) read the UDR field - * in SEC_RTC_UDR_CON register. UDR is auto-cleared when data have + * in S5M_RTC_UDR_CON register. UDR is auto-cleared when data have * been transferred. */ #define UDR_READ_RETRY_CNT 5 +/* Registers used by the driver which are different between chipsets. */ +struct s5m_rtc_reg_config { + /* Number of registers used for setting time/alarm0/alarm1 */ + unsigned int regs_count; + /* First register for time, seconds */ + unsigned int time; + /* RTC control register */ + unsigned int ctrl; + /* First register for alarm 0, seconds */ + unsigned int alarm0; + /* First register for alarm 1, seconds */ + unsigned int alarm1; + /* SMPL/WTSR register */ + unsigned int smpl_wtsr; + /* + * Register for update flag (UDR). Typically setting UDR field to 1 + * will enable update of time or alarm register. Then it will be + * auto-cleared after successful update. + */ + unsigned int rtc_udr_update; + /* Mask for UDR field in 'rtc_udr_update' register */ + unsigned int rtc_udr_mask; +}; + +/* Register map for S5M8763 and S5M8767 */ +static const struct s5m_rtc_reg_config s5m_rtc_regs = { + .regs_count = 8, + .time = S5M_RTC_SEC, + .ctrl = S5M_ALARM1_CONF, + .alarm0 = S5M_ALARM0_SEC, + .alarm1 = S5M_ALARM1_SEC, + .smpl_wtsr = S5M_WTSR_SMPL_CNTL, + .rtc_udr_update = S5M_RTC_UDR_CON, + .rtc_udr_mask = S5M_RTC_UDR_MASK, +}; + +/* + * Register map for S2MPS14. + * It may be also suitable for S2MPS11 but this was not tested. + */ +static const struct s5m_rtc_reg_config s2mps_rtc_regs = { + .regs_count = 7, + .time = S2MPS_RTC_SEC, + .ctrl = S2MPS_RTC_CTRL, + .alarm0 = S2MPS_ALARM0_SEC, + .alarm1 = S2MPS_ALARM1_SEC, + .smpl_wtsr = S2MPS_WTSR_SMPL_CNTL, + .rtc_udr_update = S2MPS_RTC_UDR_CON, + .rtc_udr_mask = S2MPS_RTC_WUDR_MASK, +}; + struct s5m_rtc_info { struct device *dev; struct i2c_client *i2c; @@ -48,13 +97,14 @@ struct s5m_rtc_info { int device_type; int rtc_24hr_mode; bool wtsr_smpl; + const struct s5m_rtc_reg_config *regs; }; static const struct regmap_config s5m_rtc_regmap_config = { .reg_bits = 8, .val_bits = 8, - .max_register = SEC_RTC_REG_MAX, + .max_register = S5M_RTC_REG_MAX, }; static const struct regmap_config s2mps14_rtc_regmap_config = { @@ -119,8 +169,9 @@ static inline int s5m8767_wait_for_udr_update(struct s5m_rtc_info *info) unsigned int data; do { - ret = regmap_read(info->regmap, SEC_RTC_UDR_CON, &data); - } while (--retry && (data & RTC_UDR_MASK) && !ret); + ret = regmap_read(info->regmap, info->regs->rtc_udr_update, + &data); + } while (--retry && (data & info->regs->rtc_udr_mask) && !ret); if (!retry) dev_err(info->dev, "waiting for UDR update, reached max number of retries\n"); @@ -128,21 +179,53 @@ static inline int s5m8767_wait_for_udr_update(struct s5m_rtc_info *info) return ret; } +static inline int s5m_check_peding_alarm_interrupt(struct s5m_rtc_info *info, + struct rtc_wkalrm *alarm) +{ + int ret; + unsigned int val; + + switch (info->device_type) { + case S5M8767X: + case S5M8763X: + ret = regmap_read(info->regmap, S5M_RTC_STATUS, &val); + val &= S5M_ALARM0_STATUS; + break; + case S2MPS14X: + ret = regmap_read(info->s5m87xx->regmap_pmic, S2MPS14_REG_ST2, + &val); + val &= S2MPS_ALARM0_STATUS; + break; + default: + return -EINVAL; + } + if (ret < 0) + return ret; + + if (val) + alarm->pending = 1; + else + alarm->pending = 0; + + return 0; +} + static inline int s5m8767_rtc_set_time_reg(struct s5m_rtc_info *info) { int ret; unsigned int data; - ret = regmap_read(info->regmap, SEC_RTC_UDR_CON, &data); + ret = regmap_read(info->regmap, info->regs->rtc_udr_update, &data); if (ret < 0) { dev_err(info->dev, "failed to read update reg(%d)\n", ret); return ret; } - data |= RTC_TIME_EN_MASK; - data |= RTC_UDR_MASK; + data |= info->regs->rtc_udr_mask; + if (info->device_type == S5M8763X || info->device_type == S5M8767X) + data |= S5M_RTC_TIME_EN_MASK; - ret = regmap_write(info->regmap, SEC_RTC_UDR_CON, data); + ret = regmap_write(info->regmap, info->regs->rtc_udr_update, data); if (ret < 0) { dev_err(info->dev, "failed to write update reg(%d)\n", ret); return ret; @@ -158,17 +241,27 @@ static inline int s5m8767_rtc_set_alarm_reg(struct s5m_rtc_info *info) int ret; unsigned int data; - ret = regmap_read(info->regmap, SEC_RTC_UDR_CON, &data); + ret = regmap_read(info->regmap, info->regs->rtc_udr_update, &data); if (ret < 0) { dev_err(info->dev, "%s: fail to read update reg(%d)\n", __func__, ret); return ret; } - data &= ~RTC_TIME_EN_MASK; - data |= RTC_UDR_MASK; + data |= info->regs->rtc_udr_mask; + switch (info->device_type) { + case S5M8763X: + case S5M8767X: + data &= ~S5M_RTC_TIME_EN_MASK; + break; + case S2MPS14X: + data |= S2MPS_RTC_RUDR_MASK; + break; + default: + return -EINVAL; + } - ret = regmap_write(info->regmap, SEC_RTC_UDR_CON, data); + ret = regmap_write(info->regmap, info->regs->rtc_udr_update, data); if (ret < 0) { dev_err(info->dev, "%s: fail to write update reg(%d)\n", __func__, ret); @@ -215,10 +308,22 @@ static void s5m8763_tm_to_data(struct rtc_time *tm, u8 *data) static int s5m_rtc_read_time(struct device *dev, struct rtc_time *tm) { struct s5m_rtc_info *info = dev_get_drvdata(dev); - u8 data[8]; + u8 data[info->regs->regs_count]; int ret; - ret = regmap_bulk_read(info->regmap, SEC_RTC_SEC, data, 8); + if (info->device_type == S2MPS14X) { + ret = regmap_update_bits(info->regmap, + info->regs->rtc_udr_update, + S2MPS_RTC_RUDR_MASK, S2MPS_RTC_RUDR_MASK); + if (ret) { + dev_err(dev, + "Failed to prepare registers for time reading: %d\n", + ret); + return ret; + } + } + ret = regmap_bulk_read(info->regmap, info->regs->time, data, + info->regs->regs_count); if (ret < 0) return ret; @@ -228,6 +333,7 @@ static int s5m_rtc_read_time(struct device *dev, struct rtc_time *tm) break; case S5M8767X: + case S2MPS14X: s5m8767_data_to_tm(data, tm, info->rtc_24hr_mode); break; @@ -245,7 +351,7 @@ static int s5m_rtc_read_time(struct device *dev, struct rtc_time *tm) static int s5m_rtc_set_time(struct device *dev, struct rtc_time *tm) { struct s5m_rtc_info *info = dev_get_drvdata(dev); - u8 data[8]; + u8 data[info->regs->regs_count]; int ret = 0; switch (info->device_type) { @@ -253,6 +359,7 @@ static int s5m_rtc_set_time(struct device *dev, struct rtc_time *tm) s5m8763_tm_to_data(tm, data); break; case S5M8767X: + case S2MPS14X: ret = s5m8767_tm_to_data(tm, data); break; default: @@ -266,7 +373,8 @@ static int s5m_rtc_set_time(struct device *dev, struct rtc_time *tm) 1900 + tm->tm_year, 1 + tm->tm_mon, tm->tm_mday, tm->tm_hour, tm->tm_min, tm->tm_sec, tm->tm_wday); - ret = regmap_raw_write(info->regmap, SEC_RTC_SEC, data, 8); + ret = regmap_raw_write(info->regmap, info->regs->time, data, + info->regs->regs_count); if (ret < 0) return ret; @@ -278,70 +386,60 @@ static int s5m_rtc_set_time(struct device *dev, struct rtc_time *tm) static int s5m_rtc_read_alarm(struct device *dev, struct rtc_wkalrm *alrm) { struct s5m_rtc_info *info = dev_get_drvdata(dev); - u8 data[8]; + u8 data[info->regs->regs_count]; unsigned int val; int ret, i; - ret = regmap_bulk_read(info->regmap, SEC_ALARM0_SEC, data, 8); + ret = regmap_bulk_read(info->regmap, info->regs->alarm0, data, + info->regs->regs_count); if (ret < 0) return ret; switch (info->device_type) { case S5M8763X: s5m8763_data_to_tm(data, &alrm->time); - ret = regmap_read(info->regmap, SEC_ALARM0_CONF, &val); + ret = regmap_read(info->regmap, S5M_ALARM0_CONF, &val); if (ret < 0) return ret; alrm->enabled = !!val; - - ret = regmap_read(info->regmap, SEC_RTC_STATUS, &val); - if (ret < 0) - return ret; - break; case S5M8767X: + case S2MPS14X: s5m8767_data_to_tm(data, &alrm->time, info->rtc_24hr_mode); - dev_dbg(dev, "%s: %d/%d/%d %d:%d:%d(%d)\n", __func__, - 1900 + alrm->time.tm_year, 1 + alrm->time.tm_mon, - alrm->time.tm_mday, alrm->time.tm_hour, - alrm->time.tm_min, alrm->time.tm_sec, - alrm->time.tm_wday); - alrm->enabled = 0; - for (i = 0; i < 7; i++) { + for (i = 0; i < info->regs->regs_count; i++) { if (data[i] & ALARM_ENABLE_MASK) { alrm->enabled = 1; break; } } - - alrm->pending = 0; - ret = regmap_read(info->regmap, SEC_RTC_STATUS, &val); - if (ret < 0) - return ret; break; default: return -EINVAL; } - if (val & ALARM0_STATUS) - alrm->pending = 1; - else - alrm->pending = 0; + dev_dbg(dev, "%s: %d/%d/%d %d:%d:%d(%d)\n", __func__, + 1900 + alrm->time.tm_year, 1 + alrm->time.tm_mon, + alrm->time.tm_mday, alrm->time.tm_hour, + alrm->time.tm_min, alrm->time.tm_sec, + alrm->time.tm_wday); + + ret = s5m_check_peding_alarm_interrupt(info, alrm); return 0; } static int s5m_rtc_stop_alarm(struct s5m_rtc_info *info) { - u8 data[8]; + u8 data[info->regs->regs_count]; int ret, i; struct rtc_time tm; - ret = regmap_bulk_read(info->regmap, SEC_ALARM0_SEC, data, 8); + ret = regmap_bulk_read(info->regmap, info->regs->alarm0, data, + info->regs->regs_count); if (ret < 0) return ret; @@ -352,14 +450,16 @@ static int s5m_rtc_stop_alarm(struct s5m_rtc_info *info) switch (info->device_type) { case S5M8763X: - ret = regmap_write(info->regmap, SEC_ALARM0_CONF, 0); + ret = regmap_write(info->regmap, S5M_ALARM0_CONF, 0); break; case S5M8767X: - for (i = 0; i < 7; i++) + case S2MPS14X: + for (i = 0; i < info->regs->regs_count; i++) data[i] &= ~ALARM_ENABLE_MASK; - ret = regmap_raw_write(info->regmap, SEC_ALARM0_SEC, data, 8); + ret = regmap_raw_write(info->regmap, info->regs->alarm0, data, + info->regs->regs_count); if (ret < 0) return ret; @@ -377,11 +477,12 @@ static int s5m_rtc_stop_alarm(struct s5m_rtc_info *info) static int s5m_rtc_start_alarm(struct s5m_rtc_info *info) { int ret; - u8 data[8]; + u8 data[info->regs->regs_count]; u8 alarm0_conf; struct rtc_time tm; - ret = regmap_bulk_read(info->regmap, SEC_ALARM0_SEC, data, 8); + ret = regmap_bulk_read(info->regmap, info->regs->alarm0, data, + info->regs->regs_count); if (ret < 0) return ret; @@ -393,10 +494,11 @@ static int s5m_rtc_start_alarm(struct s5m_rtc_info *info) switch (info->device_type) { case S5M8763X: alarm0_conf = 0x77; - ret = regmap_write(info->regmap, SEC_ALARM0_CONF, alarm0_conf); + ret = regmap_write(info->regmap, S5M_ALARM0_CONF, alarm0_conf); break; case S5M8767X: + case S2MPS14X: data[RTC_SEC] |= ALARM_ENABLE_MASK; data[RTC_MIN] |= ALARM_ENABLE_MASK; data[RTC_HOUR] |= ALARM_ENABLE_MASK; @@ -408,7 +510,8 @@ static int s5m_rtc_start_alarm(struct s5m_rtc_info *info) if (data[RTC_YEAR1] & 0x7f) data[RTC_YEAR1] |= ALARM_ENABLE_MASK; - ret = regmap_raw_write(info->regmap, SEC_ALARM0_SEC, data, 8); + ret = regmap_raw_write(info->regmap, info->regs->alarm0, data, + info->regs->regs_count); if (ret < 0) return ret; ret = s5m8767_rtc_set_alarm_reg(info); @@ -425,7 +528,7 @@ static int s5m_rtc_start_alarm(struct s5m_rtc_info *info) static int s5m_rtc_set_alarm(struct device *dev, struct rtc_wkalrm *alrm) { struct s5m_rtc_info *info = dev_get_drvdata(dev); - u8 data[8]; + u8 data[info->regs->regs_count]; int ret; switch (info->device_type) { @@ -434,6 +537,7 @@ static int s5m_rtc_set_alarm(struct device *dev, struct rtc_wkalrm *alrm) break; case S5M8767X: + case S2MPS14X: s5m8767_tm_to_data(&alrm->time, data); break; @@ -450,7 +554,8 @@ static int s5m_rtc_set_alarm(struct device *dev, struct rtc_wkalrm *alrm) if (ret < 0) return ret; - ret = regmap_raw_write(info->regmap, SEC_ALARM0_SEC, data, 8); + ret = regmap_raw_write(info->regmap, info->regs->alarm0, data, + info->regs->regs_count); if (ret < 0) return ret; @@ -495,7 +600,7 @@ static const struct rtc_class_ops s5m_rtc_ops = { static void s5m_rtc_enable_wtsr(struct s5m_rtc_info *info, bool enable) { int ret; - ret = regmap_update_bits(info->regmap, SEC_WTSR_SMPL_CNTL, + ret = regmap_update_bits(info->regmap, info->regs->smpl_wtsr, WTSR_ENABLE_MASK, enable ? WTSR_ENABLE_MASK : 0); if (ret < 0) @@ -506,7 +611,7 @@ static void s5m_rtc_enable_wtsr(struct s5m_rtc_info *info, bool enable) static void s5m_rtc_enable_smpl(struct s5m_rtc_info *info, bool enable) { int ret; - ret = regmap_update_bits(info->regmap, SEC_WTSR_SMPL_CNTL, + ret = regmap_update_bits(info->regmap, info->regs->smpl_wtsr, SMPL_ENABLE_MASK, enable ? SMPL_ENABLE_MASK : 0); if (ret < 0) @@ -517,50 +622,41 @@ static void s5m_rtc_enable_smpl(struct s5m_rtc_info *info, bool enable) static int s5m8767_rtc_init_reg(struct s5m_rtc_info *info) { u8 data[2]; - unsigned int tp_read; int ret; - struct rtc_time tm; - ret = regmap_read(info->regmap, SEC_RTC_UDR_CON, &tp_read); - if (ret < 0) { - dev_err(info->dev, "%s: fail to read control reg(%d)\n", - __func__, ret); - return ret; - } + switch (info->device_type) { + case S5M8763X: + case S5M8767X: + /* UDR update time. Default of 7.32 ms is too long. */ + ret = regmap_update_bits(info->regmap, S5M_RTC_UDR_CON, + S5M_RTC_UDR_T_MASK, S5M_RTC_UDR_T_450_US); + if (ret < 0) + dev_err(info->dev, "%s: fail to change UDR time: %d\n", + __func__, ret); - /* Set RTC control register : Binary mode, 24hour mode */ - data[0] = (1 << BCD_EN_SHIFT) | (1 << MODEL24_SHIFT); - data[1] = (0 << BCD_EN_SHIFT) | (1 << MODEL24_SHIFT); + /* Set RTC control register : Binary mode, 24hour mode */ + data[0] = (1 << BCD_EN_SHIFT) | (1 << MODEL24_SHIFT); + data[1] = (0 << BCD_EN_SHIFT) | (1 << MODEL24_SHIFT); + + ret = regmap_raw_write(info->regmap, S5M_ALARM0_CONF, data, 2); + break; + + case S2MPS14X: + data[0] = (0 << BCD_EN_SHIFT) | (1 << MODEL24_SHIFT); + ret = regmap_write(info->regmap, info->regs->ctrl, data[0]); + break; + + default: + return -EINVAL; + } info->rtc_24hr_mode = 1; - ret = regmap_raw_write(info->regmap, SEC_ALARM0_CONF, data, 2); if (ret < 0) { dev_err(info->dev, "%s: fail to write controlm reg(%d)\n", __func__, ret); return ret; } - /* In first boot time, Set rtc time to 1/1/2012 00:00:00(SUN) */ - if ((tp_read & RTC_TCON_MASK) == 0) { - dev_dbg(info->dev, "rtc init\n"); - tm.tm_sec = 0; - tm.tm_min = 0; - tm.tm_hour = 0; - tm.tm_wday = 0; - tm.tm_mday = 1; - tm.tm_mon = 0; - tm.tm_year = 112; - tm.tm_yday = 0; - tm.tm_isdst = 0; - ret = s5m_rtc_set_time(info->dev, &tm); - } - - ret = regmap_update_bits(info->regmap, SEC_RTC_UDR_CON, - RTC_TCON_MASK, tp_read | RTC_TCON_MASK); - if (ret < 0) - dev_err(info->dev, "%s: fail to update TCON reg(%d)\n", - __func__, ret); - return ret; } @@ -570,7 +666,7 @@ static int s5m_rtc_probe(struct platform_device *pdev) struct sec_platform_data *pdata = s5m87xx->pdata; struct s5m_rtc_info *info; const struct regmap_config *regmap_cfg; - int ret; + int ret, alarm_irq; if (!pdata) { dev_err(pdev->dev.parent, "Platform data not supplied\n"); @@ -584,12 +680,18 @@ static int s5m_rtc_probe(struct platform_device *pdev) switch (pdata->device_type) { case S2MPS14X: regmap_cfg = &s2mps14_rtc_regmap_config; + info->regs = &s2mps_rtc_regs; + alarm_irq = S2MPS14_IRQ_RTCA0; break; case S5M8763X: regmap_cfg = &s5m_rtc_regmap_config; + info->regs = &s5m_rtc_regs; + alarm_irq = S5M8763_IRQ_ALARM0; break; case S5M8767X: regmap_cfg = &s5m_rtc_regmap_config; + info->regs = &s5m_rtc_regs; + alarm_irq = S5M8767_IRQ_RTCA1; break; default: dev_err(&pdev->dev, "Device type is not supported by RTC driver\n"); @@ -615,20 +717,11 @@ static int s5m_rtc_probe(struct platform_device *pdev) info->device_type = s5m87xx->device_type; info->wtsr_smpl = s5m87xx->wtsr_smpl; - switch (pdata->device_type) { - case S5M8763X: - info->irq = regmap_irq_get_virq(s5m87xx->irq_data, - S5M8763_IRQ_ALARM0); - break; - - case S5M8767X: - info->irq = regmap_irq_get_virq(s5m87xx->irq_data, - S5M8767_IRQ_RTCA1); - break; - - default: + info->irq = regmap_irq_get_virq(s5m87xx->irq_data, alarm_irq); + if (info->irq <= 0) { ret = -EINVAL; - dev_err(&pdev->dev, "Unsupported device type: %d\n", ret); + dev_err(&pdev->dev, "Failed to get virtual IRQ %d\n", + alarm_irq); goto err; } @@ -676,7 +769,7 @@ static void s5m_rtc_shutdown(struct platform_device *pdev) if (info->wtsr_smpl) { for (i = 0; i < 3; i++) { s5m_rtc_enable_wtsr(info, false); - regmap_read(info->regmap, SEC_WTSR_SMPL_CNTL, &val); + regmap_read(info->regmap, info->regs->smpl_wtsr, &val); pr_debug("%s: WTSR_SMPL reg(0x%02x)\n", __func__, val); if (val & WTSR_ENABLE_MASK) pr_emerg("%s: fail to disable WTSR\n", @@ -730,7 +823,8 @@ static int s5m_rtc_suspend(struct device *dev) static SIMPLE_DEV_PM_OPS(s5m_rtc_pm_ops, s5m_rtc_suspend, s5m_rtc_resume); static const struct platform_device_id s5m_rtc_id[] = { - { "s5m-rtc", 0 }, + { "s5m-rtc", S5M8767X }, + { "s2mps14-rtc", S2MPS14X }, }; static struct platform_driver s5m_rtc_driver = { @@ -749,6 +843,6 @@ module_platform_driver(s5m_rtc_driver); /* Module information */ MODULE_AUTHOR("Sangbeom Kim <sbkim73@samsung.com>"); -MODULE_DESCRIPTION("Samsung S5M RTC driver"); +MODULE_DESCRIPTION("Samsung S5M/S2MPS14 RTC driver"); MODULE_LICENSE("GPL"); MODULE_ALIAS("platform:s5m-rtc"); diff --git a/drivers/scsi/NCR5380.c b/drivers/scsi/NCR5380.c index bcd223868227..93d13fc9a293 100644 --- a/drivers/scsi/NCR5380.c +++ b/drivers/scsi/NCR5380.c @@ -27,8 +27,6 @@ */ /* - * $Log: NCR5380.c,v $ - * Revision 1.10 1998/9/2 Alan Cox * (alan@lxorguk.ukuu.org.uk) * Fixed up the timer lockups reported so far. Things still suck. Looking @@ -89,13 +87,6 @@ #include <scsi/scsi_dbg.h> #include <scsi/scsi_transport_spi.h> -#ifndef NDEBUG -#define NDEBUG 0 -#endif -#ifndef NDEBUG_ABORT -#define NDEBUG_ABORT 0 -#endif - #if (NDEBUG & NDEBUG_LISTS) #define LIST(x,y) {printk("LINE:%d Adding %p to %p\n", __LINE__, (void*)(x), (void*)(y)); if ((x)==(y)) udelay(5); } #define REMOVE(w,x,y,z) {printk("LINE:%d Removing: %p->%p %p->%p \n", __LINE__, (void*)(w), (void*)(x), (void*)(y), (void*)(z)); if ((x)==(y)) udelay(5); } @@ -1005,7 +996,7 @@ static int NCR5380_queue_command_lck(Scsi_Cmnd * cmd, void (*done) (Scsi_Cmnd *) LIST(cmd, tmp); tmp->host_scribble = (unsigned char *) cmd; } - dprintk(NDEBUG_QUEUES, ("scsi%d : command added to %s of queue\n", instance->host_no, (cmd->cmnd[0] == REQUEST_SENSE) ? "head" : "tail")); + dprintk(NDEBUG_QUEUES, "scsi%d : command added to %s of queue\n", instance->host_no, (cmd->cmnd[0] == REQUEST_SENSE) ? "head" : "tail"); /* Run the coroutine if it isn't already running. */ /* Kick off command processing */ @@ -1040,7 +1031,7 @@ static void NCR5380_main(struct work_struct *work) /* Lock held here */ done = 1; if (!hostdata->connected && !hostdata->selecting) { - dprintk(NDEBUG_MAIN, ("scsi%d : not connected\n", instance->host_no)); + dprintk(NDEBUG_MAIN, "scsi%d : not connected\n", instance->host_no); /* * Search through the issue_queue for a command destined * for a target that's not busy. @@ -1048,7 +1039,7 @@ static void NCR5380_main(struct work_struct *work) for (tmp = (Scsi_Cmnd *) hostdata->issue_queue, prev = NULL; tmp; prev = tmp, tmp = (Scsi_Cmnd *) tmp->host_scribble) { if (prev != tmp) - dprintk(NDEBUG_LISTS, ("MAIN tmp=%p target=%d busy=%d lun=%d\n", tmp, tmp->target, hostdata->busy[tmp->target], tmp->lun)); + dprintk(NDEBUG_LISTS, "MAIN tmp=%p target=%d busy=%d lun=%d\n", tmp, tmp->device->id, hostdata->busy[tmp->device->id], tmp->device->lun); /* When we find one, remove it from the issue queue. */ if (!(hostdata->busy[tmp->device->id] & (1 << tmp->device->lun))) { if (prev) { @@ -1066,7 +1057,7 @@ static void NCR5380_main(struct work_struct *work) * On failure, we must add the command back to the * issue queue so we can keep trying. */ - dprintk(NDEBUG_MAIN|NDEBUG_QUEUES, ("scsi%d : main() : command for target %d lun %d removed from issue_queue\n", instance->host_no, tmp->target, tmp->lun)); + dprintk(NDEBUG_MAIN|NDEBUG_QUEUES, "scsi%d : main() : command for target %d lun %d removed from issue_queue\n", instance->host_no, tmp->device->id, tmp->device->lun); /* * A successful selection is defined as one that @@ -1095,7 +1086,7 @@ static void NCR5380_main(struct work_struct *work) tmp->host_scribble = (unsigned char *) hostdata->issue_queue; hostdata->issue_queue = tmp; done = 0; - dprintk(NDEBUG_MAIN|NDEBUG_QUEUES, ("scsi%d : main(): select() failed, returned to issue_queue\n", instance->host_no)); + dprintk(NDEBUG_MAIN|NDEBUG_QUEUES, "scsi%d : main(): select() failed, returned to issue_queue\n", instance->host_no); } /* lock held here still */ } /* if target/lun is not busy */ @@ -1125,9 +1116,9 @@ static void NCR5380_main(struct work_struct *work) #endif && (!hostdata->time_expires || time_before_eq(hostdata->time_expires, jiffies)) ) { - dprintk(NDEBUG_MAIN, ("scsi%d : main() : performing information transfer\n", instance->host_no)); + dprintk(NDEBUG_MAIN, "scsi%d : main() : performing information transfer\n", instance->host_no); NCR5380_information_transfer(instance); - dprintk(NDEBUG_MAIN, ("scsi%d : main() : done set false\n", instance->host_no)); + dprintk(NDEBUG_MAIN, "scsi%d : main() : done set false\n", instance->host_no); done = 0; } else break; @@ -1159,8 +1150,8 @@ static irqreturn_t NCR5380_intr(int dummy, void *dev_id) unsigned char basr; unsigned long flags; - dprintk(NDEBUG_INTR, ("scsi : NCR5380 irq %d triggered\n", - instance->irq)); + dprintk(NDEBUG_INTR, "scsi : NCR5380 irq %d triggered\n", + instance->irq); do { done = 1; @@ -1173,14 +1164,14 @@ static irqreturn_t NCR5380_intr(int dummy, void *dev_id) NCR5380_dprint(NDEBUG_INTR, instance); if ((NCR5380_read(STATUS_REG) & (SR_SEL | SR_IO)) == (SR_SEL | SR_IO)) { done = 0; - dprintk(NDEBUG_INTR, ("scsi%d : SEL interrupt\n", instance->host_no)); + dprintk(NDEBUG_INTR, "scsi%d : SEL interrupt\n", instance->host_no); NCR5380_reselect(instance); (void) NCR5380_read(RESET_PARITY_INTERRUPT_REG); } else if (basr & BASR_PARITY_ERROR) { - dprintk(NDEBUG_INTR, ("scsi%d : PARITY interrupt\n", instance->host_no)); + dprintk(NDEBUG_INTR, "scsi%d : PARITY interrupt\n", instance->host_no); (void) NCR5380_read(RESET_PARITY_INTERRUPT_REG); } else if ((NCR5380_read(STATUS_REG) & SR_RST) == SR_RST) { - dprintk(NDEBUG_INTR, ("scsi%d : RESET interrupt\n", instance->host_no)); + dprintk(NDEBUG_INTR, "scsi%d : RESET interrupt\n", instance->host_no); (void) NCR5380_read(RESET_PARITY_INTERRUPT_REG); } else { #if defined(REAL_DMA) @@ -1210,7 +1201,7 @@ static irqreturn_t NCR5380_intr(int dummy, void *dev_id) NCR5380_write(INITIATOR_COMMAND_REG, ICR_BASE); } #else - dprintk(NDEBUG_INTR, ("scsi : unknown interrupt, BASR 0x%X, MR 0x%X, SR 0x%x\n", basr, NCR5380_read(MODE_REG), NCR5380_read(STATUS_REG))); + dprintk(NDEBUG_INTR, "scsi : unknown interrupt, BASR 0x%X, MR 0x%X, SR 0x%x\n", basr, NCR5380_read(MODE_REG), NCR5380_read(STATUS_REG)); (void) NCR5380_read(RESET_PARITY_INTERRUPT_REG); #endif } @@ -1304,7 +1295,7 @@ static int NCR5380_select(struct Scsi_Host *instance, Scsi_Cmnd * cmd, int tag) hostdata->restart_select = 0; NCR5380_dprint(NDEBUG_ARBITRATION, instance); - dprintk(NDEBUG_ARBITRATION, ("scsi%d : starting arbitration, id = %d\n", instance->host_no, instance->this_id)); + dprintk(NDEBUG_ARBITRATION, "scsi%d : starting arbitration, id = %d\n", instance->host_no, instance->this_id); /* * Set the phase bits to 0, otherwise the NCR5380 won't drive the @@ -1333,7 +1324,7 @@ static int NCR5380_select(struct Scsi_Host *instance, Scsi_Cmnd * cmd, int tag) goto failed; } - dprintk(NDEBUG_ARBITRATION, ("scsi%d : arbitration complete\n", instance->host_no)); + dprintk(NDEBUG_ARBITRATION, "scsi%d : arbitration complete\n", instance->host_no); /* * The arbitration delay is 2.2us, but this is a minimum and there is @@ -1347,7 +1338,7 @@ static int NCR5380_select(struct Scsi_Host *instance, Scsi_Cmnd * cmd, int tag) /* Check for lost arbitration */ if ((NCR5380_read(INITIATOR_COMMAND_REG) & ICR_ARBITRATION_LOST) || (NCR5380_read(CURRENT_SCSI_DATA_REG) & hostdata->id_higher_mask) || (NCR5380_read(INITIATOR_COMMAND_REG) & ICR_ARBITRATION_LOST)) { NCR5380_write(MODE_REG, MR_BASE); - dprintk(NDEBUG_ARBITRATION, ("scsi%d : lost arbitration, deasserting MR_ARBITRATE\n", instance->host_no)); + dprintk(NDEBUG_ARBITRATION, "scsi%d : lost arbitration, deasserting MR_ARBITRATE\n", instance->host_no); goto failed; } NCR5380_write(INITIATOR_COMMAND_REG, ICR_BASE | ICR_ASSERT_SEL); @@ -1360,7 +1351,7 @@ static int NCR5380_select(struct Scsi_Host *instance, Scsi_Cmnd * cmd, int tag) (NCR5380_read(INITIATOR_COMMAND_REG) & ICR_ARBITRATION_LOST)) { NCR5380_write(MODE_REG, MR_BASE); NCR5380_write(INITIATOR_COMMAND_REG, ICR_BASE); - dprintk(NDEBUG_ARBITRATION, ("scsi%d : lost arbitration, deasserting ICR_ASSERT_SEL\n", instance->host_no)); + dprintk(NDEBUG_ARBITRATION, "scsi%d : lost arbitration, deasserting ICR_ASSERT_SEL\n", instance->host_no); goto failed; } /* @@ -1370,7 +1361,7 @@ static int NCR5380_select(struct Scsi_Host *instance, Scsi_Cmnd * cmd, int tag) udelay(2); - dprintk(NDEBUG_ARBITRATION, ("scsi%d : won arbitration\n", instance->host_no)); + dprintk(NDEBUG_ARBITRATION, "scsi%d : won arbitration\n", instance->host_no); /* * Now that we have won arbitration, start Selection process, asserting @@ -1422,7 +1413,7 @@ static int NCR5380_select(struct Scsi_Host *instance, Scsi_Cmnd * cmd, int tag) udelay(1); - dprintk(NDEBUG_SELECTION, ("scsi%d : selecting target %d\n", instance->host_no, scmd_id(cmd))); + dprintk(NDEBUG_SELECTION, "scsi%d : selecting target %d\n", instance->host_no, scmd_id(cmd)); /* * The SCSI specification calls for a 250 ms timeout for the actual @@ -1487,7 +1478,7 @@ part2: collect_stats(hostdata, cmd); cmd->scsi_done(cmd); NCR5380_write(SELECT_ENABLE_REG, hostdata->id_mask); - dprintk(NDEBUG_SELECTION, ("scsi%d : target did not respond within 250ms\n", instance->host_no)); + dprintk(NDEBUG_SELECTION, "scsi%d : target did not respond within 250ms\n", instance->host_no); NCR5380_write(SELECT_ENABLE_REG, hostdata->id_mask); return 0; } @@ -1520,7 +1511,7 @@ part2: goto failed; } - dprintk(NDEBUG_SELECTION, ("scsi%d : target %d selected, going into MESSAGE OUT phase.\n", instance->host_no, cmd->device->id)); + dprintk(NDEBUG_SELECTION, "scsi%d : target %d selected, going into MESSAGE OUT phase.\n", instance->host_no, cmd->device->id); tmp[0] = IDENTIFY(((instance->irq == SCSI_IRQ_NONE) ? 0 : 1), cmd->device->lun); len = 1; @@ -1530,7 +1521,7 @@ part2: data = tmp; phase = PHASE_MSGOUT; NCR5380_transfer_pio(instance, &phase, &len, &data); - dprintk(NDEBUG_SELECTION, ("scsi%d : nexus established.\n", instance->host_no)); + dprintk(NDEBUG_SELECTION, "scsi%d : nexus established.\n", instance->host_no); /* XXX need to handle errors here */ hostdata->connected = cmd; hostdata->busy[cmd->device->id] |= (1 << cmd->device->lun); @@ -1583,9 +1574,9 @@ static int NCR5380_transfer_pio(struct Scsi_Host *instance, unsigned char *phase NCR5380_setup(instance); if (!(p & SR_IO)) - dprintk(NDEBUG_PIO, ("scsi%d : pio write %d bytes\n", instance->host_no, c)); + dprintk(NDEBUG_PIO, "scsi%d : pio write %d bytes\n", instance->host_no, c); else - dprintk(NDEBUG_PIO, ("scsi%d : pio read %d bytes\n", instance->host_no, c)); + dprintk(NDEBUG_PIO, "scsi%d : pio read %d bytes\n", instance->host_no, c); /* * The NCR5380 chip will only drive the SCSI bus when the @@ -1620,11 +1611,11 @@ static int NCR5380_transfer_pio(struct Scsi_Host *instance, unsigned char *phase break; } - dprintk(NDEBUG_HANDSHAKE, ("scsi%d : REQ detected\n", instance->host_no)); + dprintk(NDEBUG_HANDSHAKE, "scsi%d : REQ detected\n", instance->host_no); /* Check for phase mismatch */ if ((tmp & PHASE_MASK) != p) { - dprintk(NDEBUG_HANDSHAKE, ("scsi%d : phase mismatch\n", instance->host_no)); + dprintk(NDEBUG_HANDSHAKE, "scsi%d : phase mismatch\n", instance->host_no); NCR5380_dprint_phase(NDEBUG_HANDSHAKE, instance); break; } @@ -1660,7 +1651,7 @@ static int NCR5380_transfer_pio(struct Scsi_Host *instance, unsigned char *phase /* FIXME - if this fails bus reset ?? */ NCR5380_poll_politely(instance, STATUS_REG, SR_REQ, 0, 5*HZ); - dprintk(NDEBUG_HANDSHAKE, ("scsi%d : req false, handshake complete\n", instance->host_no)); + dprintk(NDEBUG_HANDSHAKE, "scsi%d : req false, handshake complete\n", instance->host_no); /* * We have several special cases to consider during REQ/ACK handshaking : @@ -1681,7 +1672,7 @@ static int NCR5380_transfer_pio(struct Scsi_Host *instance, unsigned char *phase } } while (--c); - dprintk(NDEBUG_PIO, ("scsi%d : residual %d\n", instance->host_no, c)); + dprintk(NDEBUG_PIO, "scsi%d : residual %d\n", instance->host_no, c); *count = c; *data = d; @@ -1828,7 +1819,7 @@ static int NCR5380_transfer_dma(struct Scsi_Host *instance, unsigned char *phase c -= 2; } #endif - dprintk(NDEBUG_DMA, ("scsi%d : initializing DMA channel %d for %s, %d bytes %s %0x\n", instance->host_no, instance->dma_channel, (p & SR_IO) ? "reading" : "writing", c, (p & SR_IO) ? "to" : "from", (unsigned) d)); + dprintk(NDEBUG_DMA, "scsi%d : initializing DMA channel %d for %s, %d bytes %s %0x\n", instance->host_no, instance->dma_channel, (p & SR_IO) ? "reading" : "writing", c, (p & SR_IO) ? "to" : "from", (unsigned) d); hostdata->dma_len = (p & SR_IO) ? NCR5380_dma_read_setup(instance, d, c) : NCR5380_dma_write_setup(instance, d, c); #endif @@ -1857,7 +1848,7 @@ static int NCR5380_transfer_dma(struct Scsi_Host *instance, unsigned char *phase NCR5380_write(MODE_REG, MR_BASE | MR_DMA_MODE); #endif /* def REAL_DMA */ - dprintk(NDEBUG_DMA, ("scsi%d : mode reg = 0x%X\n", instance->host_no, NCR5380_read(MODE_REG))); + dprintk(NDEBUG_DMA, "scsi%d : mode reg = 0x%X\n", instance->host_no, NCR5380_read(MODE_REG)); /* * On the PAS16 at least I/O recovery delays are not needed here. @@ -1934,7 +1925,7 @@ static int NCR5380_transfer_dma(struct Scsi_Host *instance, unsigned char *phase } } - dprintk(NDEBUG_DMA, ("scsi%d : polled DMA transfer complete, basr 0x%X, sr 0x%X\n", instance->host_no, tmp, NCR5380_read(STATUS_REG))); + dprintk(NDEBUG_DMA, "scsi%d : polled DMA transfer complete, basr 0x%X, sr 0x%X\n", instance->host_no, tmp, NCR5380_read(STATUS_REG)); NCR5380_write(MODE_REG, MR_BASE); NCR5380_write(INITIATOR_COMMAND_REG, ICR_BASE); @@ -1948,7 +1939,7 @@ static int NCR5380_transfer_dma(struct Scsi_Host *instance, unsigned char *phase #ifdef READ_OVERRUNS if (*phase == p && (p & SR_IO) && residue == 0) { if (overrun) { - dprintk(NDEBUG_DMA, ("Got an input overrun, using saved byte\n")); + dprintk(NDEBUG_DMA, "Got an input overrun, using saved byte\n"); **data = saved_data; *data += 1; *count -= 1; @@ -1957,13 +1948,13 @@ static int NCR5380_transfer_dma(struct Scsi_Host *instance, unsigned char *phase printk("No overrun??\n"); cnt = toPIO = 2; } - dprintk(NDEBUG_DMA, ("Doing %d-byte PIO to 0x%X\n", cnt, *data)); + dprintk(NDEBUG_DMA, "Doing %d-byte PIO to 0x%X\n", cnt, *data); NCR5380_transfer_pio(instance, phase, &cnt, data); *count -= toPIO - cnt; } #endif - dprintk(NDEBUG_DMA, ("Return with data ptr = 0x%X, count %d, last 0x%X, next 0x%X\n", *data, *count, *(*data + *count - 1), *(*data + *count))); + dprintk(NDEBUG_DMA, "Return with data ptr = 0x%X, count %d, last 0x%X, next 0x%X\n", *data, *count, *(*data + *count - 1), *(*data + *count)); return 0; #elif defined(REAL_DMA) @@ -2013,7 +2004,7 @@ static int NCR5380_transfer_dma(struct Scsi_Host *instance, unsigned char *phase foo = NCR5380_pwrite(instance, d, c); #else int timeout; - dprintk(NDEBUG_C400_PWRITE, ("About to pwrite %d bytes\n", c)); + dprintk(NDEBUG_C400_PWRITE, "About to pwrite %d bytes\n", c); if (!(foo = NCR5380_pwrite(instance, d, c))) { /* * Wait for the last byte to be sent. If REQ is being asserted for @@ -2024,19 +2015,19 @@ static int NCR5380_transfer_dma(struct Scsi_Host *instance, unsigned char *phase while (!(NCR5380_read(BUS_AND_STATUS_REG) & BASR_DRQ) && (NCR5380_read(BUS_AND_STATUS_REG) & BASR_PHASE_MATCH)); if (!timeout) - dprintk(NDEBUG_LAST_BYTE_SENT, ("scsi%d : timed out on last byte\n", instance->host_no)); + dprintk(NDEBUG_LAST_BYTE_SENT, "scsi%d : timed out on last byte\n", instance->host_no); if (hostdata->flags & FLAG_CHECK_LAST_BYTE_SENT) { hostdata->flags &= ~FLAG_CHECK_LAST_BYTE_SENT; if (NCR5380_read(TARGET_COMMAND_REG) & TCR_LAST_BYTE_SENT) { hostdata->flags |= FLAG_HAS_LAST_BYTE_SENT; - dprintk(NDEBUG_LAST_WRITE_SENT, ("scsi%d : last bit sent works\n", instance->host_no)); + dprintk(NDEBUG_LAST_BYTE_SENT, "scsi%d : last byte sent works\n", instance->host_no); } } } else { - dprintk(NDEBUG_C400_PWRITE, ("Waiting for LASTBYTE\n")); + dprintk(NDEBUG_C400_PWRITE, "Waiting for LASTBYTE\n"); while (!(NCR5380_read(TARGET_COMMAND_REG) & TCR_LAST_BYTE_SENT)); - dprintk(NDEBUG_C400_PWRITE, ("Got LASTBYTE\n")); + dprintk(NDEBUG_C400_PWRITE, "Got LASTBYTE\n"); } } #endif @@ -2045,9 +2036,9 @@ static int NCR5380_transfer_dma(struct Scsi_Host *instance, unsigned char *phase NCR5380_write(INITIATOR_COMMAND_REG, ICR_BASE); if ((!(p & SR_IO)) && (hostdata->flags & FLAG_NCR53C400)) { - dprintk(NDEBUG_C400_PWRITE, ("53C400w: Checking for IRQ\n")); + dprintk(NDEBUG_C400_PWRITE, "53C400w: Checking for IRQ\n"); if (NCR5380_read(BUS_AND_STATUS_REG) & BASR_IRQ) { - dprintk(NDEBUG_C400_PWRITE, ("53C400w: got it, reading reset interrupt reg\n")); + dprintk(NDEBUG_C400_PWRITE, "53C400w: got it, reading reset interrupt reg\n"); NCR5380_read(RESET_PARITY_INTERRUPT_REG); } else { printk("53C400w: IRQ NOT THERE!\n"); @@ -2139,7 +2130,7 @@ static void NCR5380_information_transfer(struct Scsi_Host *instance) { --cmd->SCp.buffers_residual; cmd->SCp.this_residual = cmd->SCp.buffer->length; cmd->SCp.ptr = sg_virt(cmd->SCp.buffer); - dprintk(NDEBUG_INFORMATION, ("scsi%d : %d bytes and %d buffers left\n", instance->host_no, cmd->SCp.this_residual, cmd->SCp.buffers_residual)); + dprintk(NDEBUG_INFORMATION, "scsi%d : %d bytes and %d buffers left\n", instance->host_no, cmd->SCp.this_residual, cmd->SCp.buffers_residual); } /* * The preferred transfer method is going to be @@ -2219,7 +2210,7 @@ static void NCR5380_information_transfer(struct Scsi_Host *instance) { case LINKED_FLG_CMD_COMPLETE: /* Accept message by clearing ACK */ NCR5380_write(INITIATOR_COMMAND_REG, ICR_BASE); - dprintk(NDEBUG_LINKED, ("scsi%d : target %d lun %d linked command complete.\n", instance->host_no, cmd->device->id, cmd->device->lun)); + dprintk(NDEBUG_LINKED, "scsi%d : target %d lun %d linked command complete.\n", instance->host_no, cmd->device->id, cmd->device->lun); /* * Sanity check : A linked command should only terminate with * one of these messages if there are more linked commands @@ -2235,7 +2226,7 @@ static void NCR5380_information_transfer(struct Scsi_Host *instance) { /* The next command is still part of this process */ cmd->next_link->tag = cmd->tag; cmd->result = cmd->SCp.Status | (cmd->SCp.Message << 8); - dprintk(NDEBUG_LINKED, ("scsi%d : target %d lun %d linked request done, calling scsi_done().\n", instance->host_no, cmd->device->id, cmd->device->lun)); + dprintk(NDEBUG_LINKED, "scsi%d : target %d lun %d linked request done, calling scsi_done().\n", instance->host_no, cmd->device->id, cmd->device->lun); collect_stats(hostdata, cmd); cmd->scsi_done(cmd); cmd = hostdata->connected; @@ -2247,7 +2238,7 @@ static void NCR5380_information_transfer(struct Scsi_Host *instance) { sink = 1; NCR5380_write(INITIATOR_COMMAND_REG, ICR_BASE); hostdata->connected = NULL; - dprintk(NDEBUG_QUEUES, ("scsi%d : command for target %d, lun %d completed\n", instance->host_no, cmd->device->id, cmd->device->lun)); + dprintk(NDEBUG_QUEUES, "scsi%d : command for target %d, lun %d completed\n", instance->host_no, cmd->device->id, cmd->device->lun); hostdata->busy[cmd->device->id] &= ~(1 << cmd->device->lun); /* @@ -2281,13 +2272,13 @@ static void NCR5380_information_transfer(struct Scsi_Host *instance) { if ((cmd->cmnd[0] != REQUEST_SENSE) && (status_byte(cmd->SCp.Status) == CHECK_CONDITION)) { scsi_eh_prep_cmnd(cmd, &hostdata->ses, NULL, 0, ~0); - dprintk(NDEBUG_AUTOSENSE, ("scsi%d : performing request sense\n", instance->host_no)); + dprintk(NDEBUG_AUTOSENSE, "scsi%d : performing request sense\n", instance->host_no); LIST(cmd, hostdata->issue_queue); cmd->host_scribble = (unsigned char *) hostdata->issue_queue; hostdata->issue_queue = (Scsi_Cmnd *) cmd; - dprintk(NDEBUG_QUEUES, ("scsi%d : REQUEST SENSE added to head of issue queue\n", instance->host_no)); + dprintk(NDEBUG_QUEUES, "scsi%d : REQUEST SENSE added to head of issue queue\n", instance->host_no); } else #endif /* def AUTOSENSE */ { @@ -2327,7 +2318,7 @@ static void NCR5380_information_transfer(struct Scsi_Host *instance) { hostdata->disconnected_queue; hostdata->connected = NULL; hostdata->disconnected_queue = cmd; - dprintk(NDEBUG_QUEUES, ("scsi%d : command for target %d lun %d was moved from connected to" " the disconnected_queue\n", instance->host_no, cmd->device->id, cmd->device->lun)); + dprintk(NDEBUG_QUEUES, "scsi%d : command for target %d lun %d was moved from connected to" " the disconnected_queue\n", instance->host_no, cmd->device->id, cmd->device->lun); /* * Restore phase bits to 0 so an interrupted selection, * arbitration can resume. @@ -2373,14 +2364,14 @@ static void NCR5380_information_transfer(struct Scsi_Host *instance) { extended_msg[0] = EXTENDED_MESSAGE; /* Accept first byte by clearing ACK */ NCR5380_write(INITIATOR_COMMAND_REG, ICR_BASE); - dprintk(NDEBUG_EXTENDED, ("scsi%d : receiving extended message\n", instance->host_no)); + dprintk(NDEBUG_EXTENDED, "scsi%d : receiving extended message\n", instance->host_no); len = 2; data = extended_msg + 1; phase = PHASE_MSGIN; NCR5380_transfer_pio(instance, &phase, &len, &data); - dprintk(NDEBUG_EXTENDED, ("scsi%d : length=%d, code=0x%02x\n", instance->host_no, (int) extended_msg[1], (int) extended_msg[2])); + dprintk(NDEBUG_EXTENDED, "scsi%d : length=%d, code=0x%02x\n", instance->host_no, (int) extended_msg[1], (int) extended_msg[2]); if (!len && extended_msg[1] <= (sizeof(extended_msg) - 1)) { /* Accept third byte by clearing ACK */ @@ -2390,7 +2381,7 @@ static void NCR5380_information_transfer(struct Scsi_Host *instance) { phase = PHASE_MSGIN; NCR5380_transfer_pio(instance, &phase, &len, &data); - dprintk(NDEBUG_EXTENDED, ("scsi%d : message received, residual %d\n", instance->host_no, len)); + dprintk(NDEBUG_EXTENDED, "scsi%d : message received, residual %d\n", instance->host_no, len); switch (extended_msg[2]) { case EXTENDED_SDTR: @@ -2456,7 +2447,7 @@ static void NCR5380_information_transfer(struct Scsi_Host *instance) { NCR5380_transfer_pio(instance, &phase, &len, &data); if (!cmd->device->disconnect && should_disconnect(cmd->cmnd[0])) { NCR5380_set_timer(hostdata, USLEEP_SLEEP); - dprintk(NDEBUG_USLEEP, ("scsi%d : issued command, sleeping until %ul\n", instance->host_no, hostdata->time_expires)); + dprintk(NDEBUG_USLEEP, "scsi%d : issued command, sleeping until %lu\n", instance->host_no, hostdata->time_expires); return; } break; @@ -2468,7 +2459,7 @@ static void NCR5380_information_transfer(struct Scsi_Host *instance) { break; default: printk("scsi%d : unknown phase\n", instance->host_no); - NCR5380_dprint(NDEBUG_ALL, instance); + NCR5380_dprint(NDEBUG_ANY, instance); } /* switch(phase) */ } /* if (tmp * SR_REQ) */ else { @@ -2476,7 +2467,7 @@ static void NCR5380_information_transfer(struct Scsi_Host *instance) { */ if (!cmd->device->disconnect && time_after_eq(jiffies, poll_time)) { NCR5380_set_timer(hostdata, USLEEP_SLEEP); - dprintk(NDEBUG_USLEEP, ("scsi%d : poll timed out, sleeping until %ul\n", instance->host_no, hostdata->time_expires)); + dprintk(NDEBUG_USLEEP, "scsi%d : poll timed out, sleeping until %lu\n", instance->host_no, hostdata->time_expires); return; } } @@ -2517,7 +2508,7 @@ static void NCR5380_reselect(struct Scsi_Host *instance) { hostdata->restart_select = 1; target_mask = NCR5380_read(CURRENT_SCSI_DATA_REG) & ~(hostdata->id_mask); - dprintk(NDEBUG_SELECTION, ("scsi%d : reselect\n", instance->host_no)); + dprintk(NDEBUG_SELECTION, "scsi%d : reselect\n", instance->host_no); /* * At this point, we have detected that our SCSI ID is on the bus, @@ -2597,7 +2588,7 @@ static void NCR5380_reselect(struct Scsi_Host *instance) { do_abort(instance); } else { hostdata->connected = tmp; - dprintk(NDEBUG_RESELECTION, ("scsi%d : nexus established, target = %d, lun = %d, tag = %d\n", instance->host_no, tmp->target, tmp->lun, tmp->tag)); + dprintk(NDEBUG_RESELECTION, "scsi%d : nexus established, target = %d, lun = %d, tag = %d\n", instance->host_no, tmp->device->id, tmp->device->lun, tmp->tag); } } @@ -2682,8 +2673,8 @@ static int NCR5380_abort(Scsi_Cmnd * cmd) { NCR5380_setup(instance); - dprintk(NDEBUG_ABORT, ("scsi%d : abort called\n", instance->host_no)); - dprintk(NDEBUG_ABORT, (" basr 0x%X, sr 0x%X\n", NCR5380_read(BUS_AND_STATUS_REG), NCR5380_read(STATUS_REG))); + dprintk(NDEBUG_ABORT, "scsi%d : abort called\n", instance->host_no); + dprintk(NDEBUG_ABORT, " basr 0x%X, sr 0x%X\n", NCR5380_read(BUS_AND_STATUS_REG), NCR5380_read(STATUS_REG)); #if 0 /* @@ -2693,7 +2684,7 @@ static int NCR5380_abort(Scsi_Cmnd * cmd) { */ if (hostdata->connected == cmd) { - dprintk(NDEBUG_ABORT, ("scsi%d : aborting connected command\n", instance->host_no)); + dprintk(NDEBUG_ABORT, "scsi%d : aborting connected command\n", instance->host_no); hostdata->aborted = 1; /* * We should perform BSY checking, and make sure we haven't slipped @@ -2721,14 +2712,14 @@ static int NCR5380_abort(Scsi_Cmnd * cmd) { * from the issue queue. */ - dprintk(NDEBUG_ABORT, ("scsi%d : abort going into loop.\n", instance->host_no)); + dprintk(NDEBUG_ABORT, "scsi%d : abort going into loop.\n", instance->host_no); for (prev = (Scsi_Cmnd **) & (hostdata->issue_queue), tmp = (Scsi_Cmnd *) hostdata->issue_queue; tmp; prev = (Scsi_Cmnd **) & (tmp->host_scribble), tmp = (Scsi_Cmnd *) tmp->host_scribble) if (cmd == tmp) { REMOVE(5, *prev, tmp, tmp->host_scribble); (*prev) = (Scsi_Cmnd *) tmp->host_scribble; tmp->host_scribble = NULL; tmp->result = DID_ABORT << 16; - dprintk(NDEBUG_ABORT, ("scsi%d : abort removed command from issue queue.\n", instance->host_no)); + dprintk(NDEBUG_ABORT, "scsi%d : abort removed command from issue queue.\n", instance->host_no); tmp->scsi_done(tmp); return SUCCESS; } @@ -2750,7 +2741,7 @@ static int NCR5380_abort(Scsi_Cmnd * cmd) { */ if (hostdata->connected) { - dprintk(NDEBUG_ABORT, ("scsi%d : abort failed, command connected.\n", instance->host_no)); + dprintk(NDEBUG_ABORT, "scsi%d : abort failed, command connected.\n", instance->host_no); return FAILED; } /* @@ -2780,11 +2771,11 @@ static int NCR5380_abort(Scsi_Cmnd * cmd) { for (tmp = (Scsi_Cmnd *) hostdata->disconnected_queue; tmp; tmp = (Scsi_Cmnd *) tmp->host_scribble) if (cmd == tmp) { - dprintk(NDEBUG_ABORT, ("scsi%d : aborting disconnected command.\n", instance->host_no)); + dprintk(NDEBUG_ABORT, "scsi%d : aborting disconnected command.\n", instance->host_no); if (NCR5380_select(instance, cmd, (int) cmd->tag)) return FAILED; - dprintk(NDEBUG_ABORT, ("scsi%d : nexus reestablished.\n", instance->host_no)); + dprintk(NDEBUG_ABORT, "scsi%d : nexus reestablished.\n", instance->host_no); do_abort(instance); diff --git a/drivers/scsi/NCR5380.h b/drivers/scsi/NCR5380.h index 14964d0a0e9d..c79ddfa6f53c 100644 --- a/drivers/scsi/NCR5380.h +++ b/drivers/scsi/NCR5380.h @@ -21,10 +21,6 @@ * 1+ (800) 334-5454 */ -/* - * $Log: NCR5380.h,v $ - */ - #ifndef NCR5380_H #define NCR5380_H @@ -60,6 +56,9 @@ #define NDEBUG_C400_PREAD 0x100000 #define NDEBUG_C400_PWRITE 0x200000 #define NDEBUG_LISTS 0x400000 +#define NDEBUG_ABORT 0x800000 +#define NDEBUG_TAGS 0x1000000 +#define NDEBUG_MERGING 0x2000000 #define NDEBUG_ANY 0xFFFFFFFFUL @@ -292,9 +291,24 @@ struct NCR5380_hostdata { #ifdef __KERNEL__ -#define dprintk(a,b) do {} while(0) -#define NCR5380_dprint(a,b) do {} while(0) -#define NCR5380_dprint_phase(a,b) do {} while(0) +#ifndef NDEBUG +#define NDEBUG (0) +#endif + +#define dprintk(flg, fmt, ...) \ + do { if ((NDEBUG) & (flg)) pr_debug(fmt, ## __VA_ARGS__); } while (0) + +#if NDEBUG +#define NCR5380_dprint(flg, arg) \ + do { if ((NDEBUG) & (flg)) NCR5380_print(arg); } while (0) +#define NCR5380_dprint_phase(flg, arg) \ + do { if ((NDEBUG) & (flg)) NCR5380_print_phase(arg); } while (0) +static void NCR5380_print_phase(struct Scsi_Host *instance); +static void NCR5380_print(struct Scsi_Host *instance); +#else +#define NCR5380_dprint(flg, arg) do {} while (0) +#define NCR5380_dprint_phase(flg, arg) do {} while (0) +#endif #if defined(AUTOPROBE_IRQ) static int NCR5380_probe_irq(struct Scsi_Host *instance, int possible); @@ -307,10 +321,6 @@ static irqreturn_t NCR5380_intr(int irq, void *dev_id); #endif static void NCR5380_main(struct work_struct *work); static void __maybe_unused NCR5380_print_options(struct Scsi_Host *instance); -#ifdef NDEBUG -static void NCR5380_print_phase(struct Scsi_Host *instance); -static void NCR5380_print(struct Scsi_Host *instance); -#endif static int NCR5380_abort(Scsi_Cmnd * cmd); static int NCR5380_bus_reset(Scsi_Cmnd * cmd); static int NCR5380_queue_command(struct Scsi_Host *, struct scsi_cmnd *); diff --git a/drivers/scsi/aic7xxx/aic79xx_pci.c b/drivers/scsi/aic7xxx/aic79xx_pci.c index 14b5f8d0e7f4..cc9bd26f5d1a 100644 --- a/drivers/scsi/aic7xxx/aic79xx_pci.c +++ b/drivers/scsi/aic7xxx/aic79xx_pci.c @@ -827,7 +827,7 @@ ahd_pci_intr(struct ahd_softc *ahd) for (bit = 0; bit < 8; bit++) { if ((pci_status[i] & (0x1 << bit)) != 0) { - static const char *s; + const char *s; s = pci_status_strings[bit]; if (i == 7/*TARG*/ && bit == 3) @@ -887,23 +887,15 @@ ahd_pci_split_intr(struct ahd_softc *ahd, u_int intstat) for (bit = 0; bit < 8; bit++) { - if ((split_status[i] & (0x1 << bit)) != 0) { - static const char *s; - - s = split_status_strings[bit]; - printk(s, ahd_name(ahd), + if ((split_status[i] & (0x1 << bit)) != 0) + printk(split_status_strings[bit], ahd_name(ahd), split_status_source[i]); - } if (i > 1) continue; - if ((sg_split_status[i] & (0x1 << bit)) != 0) { - static const char *s; - - s = split_status_strings[bit]; - printk(s, ahd_name(ahd), "SG"); - } + if ((sg_split_status[i] & (0x1 << bit)) != 0) + printk(split_status_strings[bit], ahd_name(ahd), "SG"); } } /* diff --git a/drivers/scsi/arm/acornscsi.c b/drivers/scsi/arm/acornscsi.c index 059ff477a398..2e797a367608 100644 --- a/drivers/scsi/arm/acornscsi.c +++ b/drivers/scsi/arm/acornscsi.c @@ -62,13 +62,6 @@ */ #undef CONFIG_SCSI_ACORNSCSI_TAGGED_QUEUE /* - * SCSI-II Linked command support. - * - * The higher level code doesn't support linked commands yet, and so the option - * is undef'd here. - */ -#undef CONFIG_SCSI_ACORNSCSI_LINK -/* * SCSI-II Synchronous transfer support. * * Tried and tested... @@ -160,10 +153,6 @@ #error "Yippee! ABORT TAG is now defined! Remove this error!" #endif -#ifdef CONFIG_SCSI_ACORNSCSI_LINK -#error SCSI2 LINKed commands not supported (yet)! -#endif - #ifdef USE_DMAC /* * DMAC setup parameters @@ -1668,42 +1657,6 @@ void acornscsi_message(AS_Host *host) } break; -#ifdef CONFIG_SCSI_ACORNSCSI_LINK - case LINKED_CMD_COMPLETE: - case LINKED_FLG_CMD_COMPLETE: - /* - * We don't support linked commands yet - */ - if (0) { -#if (DEBUG & DEBUG_LINK) - printk("scsi%d.%c: lun %d tag %d linked command complete\n", - host->host->host_no, acornscsi_target(host), host->SCpnt->tag); -#endif - /* - * A linked command should only terminate with one of these messages - * if there are more linked commands available. - */ - if (!host->SCpnt->next_link) { - printk(KERN_WARNING "scsi%d.%c: lun %d tag %d linked command complete, but no next_link\n", - instance->host_no, acornscsi_target(host), host->SCpnt->tag); - acornscsi_sbic_issuecmd(host, CMND_ASSERTATN); - msgqueue_addmsg(&host->scsi.msgs, 1, ABORT); - } else { - struct scsi_cmnd *SCpnt = host->SCpnt; - - acornscsi_dma_cleanup(host); - - host->SCpnt = host->SCpnt->next_link; - host->SCpnt->tag = SCpnt->tag; - SCpnt->result = DID_OK | host->scsi.SCp.Message << 8 | host->Scsi.SCp.Status; - SCpnt->done(SCpnt); - - /* initialise host->SCpnt->SCp */ - } - break; - } -#endif - default: /* reject message */ printk(KERN_ERR "scsi%d.%c: unrecognised message %02X, rejecting\n", host->host->host_no, acornscsi_target(host), @@ -2825,9 +2778,6 @@ char *acornscsi_info(struct Scsi_Host *host) #ifdef CONFIG_SCSI_ACORNSCSI_TAGGED_QUEUE " TAG" #endif -#ifdef CONFIG_SCSI_ACORNSCSI_LINK - " LINK" -#endif #if (DEBUG & DEBUG_NO_WRITE) " NOWRITE (" __stringify(NO_WRITE) ")" #endif @@ -2851,9 +2801,6 @@ static int acornscsi_show_info(struct seq_file *m, struct Scsi_Host *instance) #ifdef CONFIG_SCSI_ACORNSCSI_TAGGED_QUEUE " TAG" #endif -#ifdef CONFIG_SCSI_ACORNSCSI_LINK - " LINK" -#endif #if (DEBUG & DEBUG_NO_WRITE) " NOWRITE (" __stringify(NO_WRITE) ")" #endif diff --git a/drivers/scsi/arm/cumana_1.c b/drivers/scsi/arm/cumana_1.c index f8e060900052..8ef810a4476e 100644 --- a/drivers/scsi/arm/cumana_1.c +++ b/drivers/scsi/arm/cumana_1.c @@ -36,9 +36,6 @@ void __iomem *base; \ void __iomem *dma -#define BOARD_NORMAL 0 -#define BOARD_NCR53C400 1 - #include "../NCR5380.h" void cumanascsi_setup(char *str, int *ints) diff --git a/drivers/scsi/arm/oak.c b/drivers/scsi/arm/oak.c index 4266eef8aca1..188e734c7ff0 100644 --- a/drivers/scsi/arm/oak.c +++ b/drivers/scsi/arm/oak.c @@ -37,9 +37,6 @@ #define NCR5380_implementation_fields \ void __iomem *base -#define BOARD_NORMAL 0 -#define BOARD_NCR53C400 1 - #include "../NCR5380.h" #undef START_DMA_INITIATOR_RECEIVE_REG diff --git a/drivers/scsi/atari_NCR5380.c b/drivers/scsi/atari_NCR5380.c index 0f3cdbc80ba6..1814aa20b724 100644 --- a/drivers/scsi/atari_NCR5380.c +++ b/drivers/scsi/atari_NCR5380.c @@ -370,7 +370,7 @@ static int is_lun_busy(Scsi_Cmnd *cmd, int should_be_tagged) return 0; if (TagAlloc[cmd->device->id][cmd->device->lun].nr_allocated >= TagAlloc[cmd->device->id][cmd->device->lun].queue_size) { - TAG_PRINTK("scsi%d: target %d lun %d: no free tags\n", + dprintk(NDEBUG_TAGS, "scsi%d: target %d lun %d: no free tags\n", H_NO(cmd), cmd->device->id, cmd->device->lun); return 1; } @@ -394,7 +394,7 @@ static void cmd_get_tag(Scsi_Cmnd *cmd, int should_be_tagged) !setup_use_tagged_queuing || !cmd->device->tagged_supported) { cmd->tag = TAG_NONE; hostdata->busy[cmd->device->id] |= (1 << cmd->device->lun); - TAG_PRINTK("scsi%d: target %d lun %d now allocated by untagged " + dprintk(NDEBUG_TAGS, "scsi%d: target %d lun %d now allocated by untagged " "command\n", H_NO(cmd), cmd->device->id, cmd->device->lun); } else { TAG_ALLOC *ta = &TagAlloc[cmd->device->id][cmd->device->lun]; @@ -402,7 +402,7 @@ static void cmd_get_tag(Scsi_Cmnd *cmd, int should_be_tagged) cmd->tag = find_first_zero_bit(ta->allocated, MAX_TAGS); set_bit(cmd->tag, ta->allocated); ta->nr_allocated++; - TAG_PRINTK("scsi%d: using tag %d for target %d lun %d " + dprintk(NDEBUG_TAGS, "scsi%d: using tag %d for target %d lun %d " "(now %d tags in use)\n", H_NO(cmd), cmd->tag, cmd->device->id, cmd->device->lun, ta->nr_allocated); @@ -420,7 +420,7 @@ static void cmd_free_tag(Scsi_Cmnd *cmd) if (cmd->tag == TAG_NONE) { hostdata->busy[cmd->device->id] &= ~(1 << cmd->device->lun); - TAG_PRINTK("scsi%d: target %d lun %d untagged cmd finished\n", + dprintk(NDEBUG_TAGS, "scsi%d: target %d lun %d untagged cmd finished\n", H_NO(cmd), cmd->device->id, cmd->device->lun); } else if (cmd->tag >= MAX_TAGS) { printk(KERN_NOTICE "scsi%d: trying to free bad tag %d!\n", @@ -429,7 +429,7 @@ static void cmd_free_tag(Scsi_Cmnd *cmd) TAG_ALLOC *ta = &TagAlloc[cmd->device->id][cmd->device->lun]; clear_bit(cmd->tag, ta->allocated); ta->nr_allocated--; - TAG_PRINTK("scsi%d: freed tag %d for target %d lun %d\n", + dprintk(NDEBUG_TAGS, "scsi%d: freed tag %d for target %d lun %d\n", H_NO(cmd), cmd->tag, cmd->device->id, cmd->device->lun); } } @@ -478,7 +478,7 @@ static void merge_contiguous_buffers(Scsi_Cmnd *cmd) for (endaddr = virt_to_phys(cmd->SCp.ptr + cmd->SCp.this_residual - 1) + 1; cmd->SCp.buffers_residual && virt_to_phys(sg_virt(&cmd->SCp.buffer[1])) == endaddr;) { - MER_PRINTK("VTOP(%p) == %08lx -> merging\n", + dprintk(NDEBUG_MERGING, "VTOP(%p) == %08lx -> merging\n", page_address(sg_page(&cmd->SCp.buffer[1])), endaddr); #if (NDEBUG & NDEBUG_MERGING) ++cnt; @@ -490,7 +490,7 @@ static void merge_contiguous_buffers(Scsi_Cmnd *cmd) } #if (NDEBUG & NDEBUG_MERGING) if (oldlen != cmd->SCp.this_residual) - MER_PRINTK("merged %d buffers from %p, new length %08x\n", + dprintk(NDEBUG_MERGING, "merged %d buffers from %p, new length %08x\n", cnt, cmd->SCp.ptr, cmd->SCp.this_residual); #endif } @@ -626,16 +626,6 @@ static void NCR5380_print_phase(struct Scsi_Host *instance) } } -#else /* !NDEBUG */ - -/* dummies... */ -static inline void NCR5380_print(struct Scsi_Host *instance) -{ -}; -static inline void NCR5380_print_phase(struct Scsi_Host *instance) -{ -}; - #endif /* @@ -676,7 +666,7 @@ static inline void NCR5380_all_init(void) { static int done = 0; if (!done) { - INI_PRINTK("scsi : NCR5380_all_init()\n"); + dprintk(NDEBUG_INIT, "scsi : NCR5380_all_init()\n"); done = 1; } } @@ -739,8 +729,8 @@ static void NCR5380_print_status(struct Scsi_Host *instance) Scsi_Cmnd *ptr; unsigned long flags; - NCR_PRINT(NDEBUG_ANY); - NCR_PRINT_PHASE(NDEBUG_ANY); + NCR5380_dprint(NDEBUG_ANY, instance); + NCR5380_dprint_phase(NDEBUG_ANY, instance); hostdata = (struct NCR5380_hostdata *)instance->hostdata; @@ -984,7 +974,7 @@ static int NCR5380_queue_command_lck(Scsi_Cmnd *cmd, void (*done)(Scsi_Cmnd *)) } local_irq_restore(flags); - QU_PRINTK("scsi%d: command added to %s of queue\n", H_NO(cmd), + dprintk(NDEBUG_QUEUES, "scsi%d: command added to %s of queue\n", H_NO(cmd), (cmd->cmnd[0] == REQUEST_SENSE) ? "head" : "tail"); /* If queue_command() is called from an interrupt (real one or bottom @@ -1054,7 +1044,7 @@ static void NCR5380_main(struct work_struct *work) done = 1; if (!hostdata->connected) { - MAIN_PRINTK("scsi%d: not connected\n", HOSTNO); + dprintk(NDEBUG_MAIN, "scsi%d: not connected\n", HOSTNO); /* * Search through the issue_queue for a command destined * for a target that's not busy. @@ -1107,7 +1097,7 @@ static void NCR5380_main(struct work_struct *work) * On failure, we must add the command back to the * issue queue so we can keep trying. */ - MAIN_PRINTK("scsi%d: main(): command for target %d " + dprintk(NDEBUG_MAIN, "scsi%d: main(): command for target %d " "lun %d removed from issue_queue\n", HOSTNO, tmp->device->id, tmp->device->lun); /* @@ -1140,7 +1130,7 @@ static void NCR5380_main(struct work_struct *work) #endif falcon_dont_release--; local_irq_restore(flags); - MAIN_PRINTK("scsi%d: main(): select() failed, " + dprintk(NDEBUG_MAIN, "scsi%d: main(): select() failed, " "returned to issue_queue\n", HOSTNO); if (hostdata->connected) break; @@ -1155,10 +1145,10 @@ static void NCR5380_main(struct work_struct *work) #endif ) { local_irq_restore(flags); - MAIN_PRINTK("scsi%d: main: performing information transfer\n", + dprintk(NDEBUG_MAIN, "scsi%d: main: performing information transfer\n", HOSTNO); NCR5380_information_transfer(instance); - MAIN_PRINTK("scsi%d: main: done set false\n", HOSTNO); + dprintk(NDEBUG_MAIN, "scsi%d: main: done set false\n", HOSTNO); done = 0; } } while (!done); @@ -1204,12 +1194,12 @@ static void NCR5380_dma_complete(struct Scsi_Host *instance) (BASR_PHASE_MATCH|BASR_ACK)) { saved_data = NCR5380_read(INPUT_DATA_REG); overrun = 1; - DMA_PRINTK("scsi%d: read overrun handled\n", HOSTNO); + dprintk(NDEBUG_DMA, "scsi%d: read overrun handled\n", HOSTNO); } } } - DMA_PRINTK("scsi%d: real DMA transfer complete, basr 0x%X, sr 0x%X\n", + dprintk(NDEBUG_DMA, "scsi%d: real DMA transfer complete, basr 0x%X, sr 0x%X\n", HOSTNO, NCR5380_read(BUS_AND_STATUS_REG), NCR5380_read(STATUS_REG)); @@ -1229,13 +1219,13 @@ static void NCR5380_dma_complete(struct Scsi_Host *instance) if ((NCR5380_read(STATUS_REG) & PHASE_MASK) == p && (p & SR_IO)) { cnt = toPIO = atari_read_overruns; if (overrun) { - DMA_PRINTK("Got an input overrun, using saved byte\n"); + dprintk(NDEBUG_DMA, "Got an input overrun, using saved byte\n"); *(*data)++ = saved_data; (*count)--; cnt--; toPIO--; } - DMA_PRINTK("Doing %d-byte PIO to 0x%08lx\n", cnt, (long)*data); + dprintk(NDEBUG_DMA, "Doing %d-byte PIO to 0x%08lx\n", cnt, (long)*data); NCR5380_transfer_pio(instance, &p, &cnt, data); *count -= toPIO - cnt; } @@ -1261,25 +1251,25 @@ static irqreturn_t NCR5380_intr(int irq, void *dev_id) int done = 1, handled = 0; unsigned char basr; - INT_PRINTK("scsi%d: NCR5380 irq triggered\n", HOSTNO); + dprintk(NDEBUG_INTR, "scsi%d: NCR5380 irq triggered\n", HOSTNO); /* Look for pending interrupts */ basr = NCR5380_read(BUS_AND_STATUS_REG); - INT_PRINTK("scsi%d: BASR=%02x\n", HOSTNO, basr); + dprintk(NDEBUG_INTR, "scsi%d: BASR=%02x\n", HOSTNO, basr); /* dispatch to appropriate routine if found and done=0 */ if (basr & BASR_IRQ) { - NCR_PRINT(NDEBUG_INTR); + NCR5380_dprint(NDEBUG_INTR, instance); if ((NCR5380_read(STATUS_REG) & (SR_SEL|SR_IO)) == (SR_SEL|SR_IO)) { done = 0; ENABLE_IRQ(); - INT_PRINTK("scsi%d: SEL interrupt\n", HOSTNO); + dprintk(NDEBUG_INTR, "scsi%d: SEL interrupt\n", HOSTNO); NCR5380_reselect(instance); (void)NCR5380_read(RESET_PARITY_INTERRUPT_REG); } else if (basr & BASR_PARITY_ERROR) { - INT_PRINTK("scsi%d: PARITY interrupt\n", HOSTNO); + dprintk(NDEBUG_INTR, "scsi%d: PARITY interrupt\n", HOSTNO); (void)NCR5380_read(RESET_PARITY_INTERRUPT_REG); } else if ((NCR5380_read(STATUS_REG) & SR_RST) == SR_RST) { - INT_PRINTK("scsi%d: RESET interrupt\n", HOSTNO); + dprintk(NDEBUG_INTR, "scsi%d: RESET interrupt\n", HOSTNO); (void)NCR5380_read(RESET_PARITY_INTERRUPT_REG); } else { /* @@ -1298,7 +1288,7 @@ static irqreturn_t NCR5380_intr(int irq, void *dev_id) ((basr & BASR_END_DMA_TRANSFER) || !(basr & BASR_PHASE_MATCH))) { - INT_PRINTK("scsi%d: PHASE MISM or EOP interrupt\n", HOSTNO); + dprintk(NDEBUG_INTR, "scsi%d: PHASE MISM or EOP interrupt\n", HOSTNO); NCR5380_dma_complete( instance ); done = 0; ENABLE_IRQ(); @@ -1323,7 +1313,7 @@ static irqreturn_t NCR5380_intr(int irq, void *dev_id) } if (!done) { - INT_PRINTK("scsi%d: in int routine, calling main\n", HOSTNO); + dprintk(NDEBUG_INTR, "scsi%d: in int routine, calling main\n", HOSTNO); /* Put a call to NCR5380_main() on the queue... */ queue_main(); } @@ -1396,8 +1386,8 @@ static int NCR5380_select(struct Scsi_Host *instance, Scsi_Cmnd *cmd, int tag) unsigned long flags; hostdata->restart_select = 0; - NCR_PRINT(NDEBUG_ARBITRATION); - ARB_PRINTK("scsi%d: starting arbitration, id = %d\n", HOSTNO, + NCR5380_dprint(NDEBUG_ARBITRATION, instance); + dprintk(NDEBUG_ARBITRATION, "scsi%d: starting arbitration, id = %d\n", HOSTNO, instance->this_id); /* @@ -1442,7 +1432,7 @@ static int NCR5380_select(struct Scsi_Host *instance, Scsi_Cmnd *cmd, int tag) ; #endif - ARB_PRINTK("scsi%d: arbitration complete\n", HOSTNO); + dprintk(NDEBUG_ARBITRATION, "scsi%d: arbitration complete\n", HOSTNO); if (hostdata->connected) { NCR5380_write(MODE_REG, MR_BASE); @@ -1463,7 +1453,7 @@ static int NCR5380_select(struct Scsi_Host *instance, Scsi_Cmnd *cmd, int tag) (NCR5380_read(INITIATOR_COMMAND_REG) & ICR_ARBITRATION_LOST) || hostdata->connected) { NCR5380_write(MODE_REG, MR_BASE); - ARB_PRINTK("scsi%d: lost arbitration, deasserting MR_ARBITRATE\n", + dprintk(NDEBUG_ARBITRATION, "scsi%d: lost arbitration, deasserting MR_ARBITRATE\n", HOSTNO); return -1; } @@ -1478,7 +1468,7 @@ static int NCR5380_select(struct Scsi_Host *instance, Scsi_Cmnd *cmd, int tag) hostdata->connected) { NCR5380_write(MODE_REG, MR_BASE); NCR5380_write(INITIATOR_COMMAND_REG, ICR_BASE); - ARB_PRINTK("scsi%d: lost arbitration, deasserting ICR_ASSERT_SEL\n", + dprintk(NDEBUG_ARBITRATION, "scsi%d: lost arbitration, deasserting ICR_ASSERT_SEL\n", HOSTNO); return -1; } @@ -1501,7 +1491,7 @@ static int NCR5380_select(struct Scsi_Host *instance, Scsi_Cmnd *cmd, int tag) return -1; } - ARB_PRINTK("scsi%d: won arbitration\n", HOSTNO); + dprintk(NDEBUG_ARBITRATION, "scsi%d: won arbitration\n", HOSTNO); /* * Now that we have won arbitration, start Selection process, asserting @@ -1561,7 +1551,7 @@ static int NCR5380_select(struct Scsi_Host *instance, Scsi_Cmnd *cmd, int tag) udelay(1); - SEL_PRINTK("scsi%d: selecting target %d\n", HOSTNO, cmd->device->id); + dprintk(NDEBUG_SELECTION, "scsi%d: selecting target %d\n", HOSTNO, cmd->device->id); /* * The SCSI specification calls for a 250 ms timeout for the actual @@ -1617,7 +1607,7 @@ static int NCR5380_select(struct Scsi_Host *instance, Scsi_Cmnd *cmd, int tag) printk(KERN_ERR "scsi%d: weirdness\n", HOSTNO); if (hostdata->restart_select) printk(KERN_NOTICE "\trestart select\n"); - NCR_PRINT(NDEBUG_ANY); + NCR5380_dprint(NDEBUG_ANY, instance); NCR5380_write(SELECT_ENABLE_REG, hostdata->id_mask); return -1; } @@ -1630,7 +1620,7 @@ static int NCR5380_select(struct Scsi_Host *instance, Scsi_Cmnd *cmd, int tag) #endif cmd->scsi_done(cmd); NCR5380_write(SELECT_ENABLE_REG, hostdata->id_mask); - SEL_PRINTK("scsi%d: target did not respond within 250ms\n", HOSTNO); + dprintk(NDEBUG_SELECTION, "scsi%d: target did not respond within 250ms\n", HOSTNO); NCR5380_write(SELECT_ENABLE_REG, hostdata->id_mask); return 0; } @@ -1656,7 +1646,7 @@ static int NCR5380_select(struct Scsi_Host *instance, Scsi_Cmnd *cmd, int tag) while (!(NCR5380_read(STATUS_REG) & SR_REQ)) ; - SEL_PRINTK("scsi%d: target %d selected, going into MESSAGE OUT phase.\n", + dprintk(NDEBUG_SELECTION, "scsi%d: target %d selected, going into MESSAGE OUT phase.\n", HOSTNO, cmd->device->id); tmp[0] = IDENTIFY(1, cmd->device->lun); @@ -1676,7 +1666,7 @@ static int NCR5380_select(struct Scsi_Host *instance, Scsi_Cmnd *cmd, int tag) data = tmp; phase = PHASE_MSGOUT; NCR5380_transfer_pio(instance, &phase, &len, &data); - SEL_PRINTK("scsi%d: nexus established.\n", HOSTNO); + dprintk(NDEBUG_SELECTION, "scsi%d: nexus established.\n", HOSTNO); /* XXX need to handle errors here */ hostdata->connected = cmd; #ifndef SUPPORT_TAGS @@ -1737,12 +1727,12 @@ static int NCR5380_transfer_pio(struct Scsi_Host *instance, while (!((tmp = NCR5380_read(STATUS_REG)) & SR_REQ)) ; - HSH_PRINTK("scsi%d: REQ detected\n", HOSTNO); + dprintk(NDEBUG_HANDSHAKE, "scsi%d: REQ detected\n", HOSTNO); /* Check for phase mismatch */ if ((tmp & PHASE_MASK) != p) { - PIO_PRINTK("scsi%d: phase mismatch\n", HOSTNO); - NCR_PRINT_PHASE(NDEBUG_PIO); + dprintk(NDEBUG_PIO, "scsi%d: phase mismatch\n", HOSTNO); + NCR5380_dprint_phase(NDEBUG_PIO, instance); break; } @@ -1764,25 +1754,25 @@ static int NCR5380_transfer_pio(struct Scsi_Host *instance, if (!(p & SR_IO)) { if (!((p & SR_MSG) && c > 1)) { NCR5380_write(INITIATOR_COMMAND_REG, ICR_BASE | ICR_ASSERT_DATA); - NCR_PRINT(NDEBUG_PIO); + NCR5380_dprint(NDEBUG_PIO, instance); NCR5380_write(INITIATOR_COMMAND_REG, ICR_BASE | ICR_ASSERT_DATA | ICR_ASSERT_ACK); } else { NCR5380_write(INITIATOR_COMMAND_REG, ICR_BASE | ICR_ASSERT_DATA | ICR_ASSERT_ATN); - NCR_PRINT(NDEBUG_PIO); + NCR5380_dprint(NDEBUG_PIO, instance); NCR5380_write(INITIATOR_COMMAND_REG, ICR_BASE | ICR_ASSERT_DATA | ICR_ASSERT_ATN | ICR_ASSERT_ACK); } } else { - NCR_PRINT(NDEBUG_PIO); + NCR5380_dprint(NDEBUG_PIO, instance); NCR5380_write(INITIATOR_COMMAND_REG, ICR_BASE | ICR_ASSERT_ACK); } while (NCR5380_read(STATUS_REG) & SR_REQ) ; - HSH_PRINTK("scsi%d: req false, handshake complete\n", HOSTNO); + dprintk(NDEBUG_HANDSHAKE, "scsi%d: req false, handshake complete\n", HOSTNO); /* * We have several special cases to consider during REQ/ACK handshaking : @@ -1803,7 +1793,7 @@ static int NCR5380_transfer_pio(struct Scsi_Host *instance, } } while (--c); - PIO_PRINTK("scsi%d: residual %d\n", HOSTNO, c); + dprintk(NDEBUG_PIO, "scsi%d: residual %d\n", HOSTNO, c); *count = c; *data = d; @@ -1917,7 +1907,7 @@ static int NCR5380_transfer_dma(struct Scsi_Host *instance, if (atari_read_overruns && (p & SR_IO)) c -= atari_read_overruns; - DMA_PRINTK("scsi%d: initializing DMA for %s, %d bytes %s %p\n", + dprintk(NDEBUG_DMA, "scsi%d: initializing DMA for %s, %d bytes %s %p\n", HOSTNO, (p & SR_IO) ? "reading" : "writing", c, (p & SR_IO) ? "to" : "from", d); @@ -1997,7 +1987,7 @@ static void NCR5380_information_transfer(struct Scsi_Host *instance) phase = (tmp & PHASE_MASK); if (phase != old_phase) { old_phase = phase; - NCR_PRINT_PHASE(NDEBUG_INFORMATION); + NCR5380_dprint_phase(NDEBUG_INFORMATION, instance); } if (sink && (phase != PHASE_MSGOUT)) { @@ -2039,7 +2029,7 @@ static void NCR5380_information_transfer(struct Scsi_Host *instance) * they are at contiguous physical addresses. */ merge_contiguous_buffers(cmd); - INF_PRINTK("scsi%d: %d bytes and %d buffers left\n", + dprintk(NDEBUG_INFORMATION, "scsi%d: %d bytes and %d buffers left\n", HOSTNO, cmd->SCp.this_residual, cmd->SCp.buffers_residual); } @@ -2123,7 +2113,7 @@ static void NCR5380_information_transfer(struct Scsi_Host *instance) /* Accept message by clearing ACK */ NCR5380_write(INITIATOR_COMMAND_REG, ICR_BASE); - LNK_PRINTK("scsi%d: target %d lun %d linked command " + dprintk(NDEBUG_LINKED, "scsi%d: target %d lun %d linked command " "complete.\n", HOSTNO, cmd->device->id, cmd->device->lun); /* Enable reselect interrupts */ @@ -2148,7 +2138,7 @@ static void NCR5380_information_transfer(struct Scsi_Host *instance) * and don't free it! */ cmd->next_link->tag = cmd->tag; cmd->result = cmd->SCp.Status | (cmd->SCp.Message << 8); - LNK_PRINTK("scsi%d: target %d lun %d linked request " + dprintk(NDEBUG_LINKED, "scsi%d: target %d lun %d linked request " "done, calling scsi_done().\n", HOSTNO, cmd->device->id, cmd->device->lun); #ifdef NCR5380_STATS @@ -2165,7 +2155,7 @@ static void NCR5380_information_transfer(struct Scsi_Host *instance) /* ++guenther: possible race with Falcon locking */ falcon_dont_release++; hostdata->connected = NULL; - QU_PRINTK("scsi%d: command for target %d, lun %d " + dprintk(NDEBUG_QUEUES, "scsi%d: command for target %d, lun %d " "completed\n", HOSTNO, cmd->device->id, cmd->device->lun); #ifdef SUPPORT_TAGS cmd_free_tag(cmd); @@ -2179,7 +2169,7 @@ static void NCR5380_information_transfer(struct Scsi_Host *instance) /* ++Andreas: the mid level code knows about QUEUE_FULL now. */ TAG_ALLOC *ta = &TagAlloc[cmd->device->id][cmd->device->lun]; - TAG_PRINTK("scsi%d: target %d lun %d returned " + dprintk(NDEBUG_TAGS, "scsi%d: target %d lun %d returned " "QUEUE_FULL after %d commands\n", HOSTNO, cmd->device->id, cmd->device->lun, ta->nr_allocated); @@ -2224,14 +2214,14 @@ static void NCR5380_information_transfer(struct Scsi_Host *instance) (status_byte(cmd->SCp.Status) == CHECK_CONDITION)) { scsi_eh_prep_cmnd(cmd, &hostdata->ses, NULL, 0, ~0); - ASEN_PRINTK("scsi%d: performing request sense\n", HOSTNO); + dprintk(NDEBUG_AUTOSENSE, "scsi%d: performing request sense\n", HOSTNO); local_irq_save(flags); LIST(cmd,hostdata->issue_queue); SET_NEXT(cmd, hostdata->issue_queue); hostdata->issue_queue = (Scsi_Cmnd *) cmd; local_irq_restore(flags); - QU_PRINTK("scsi%d: REQUEST SENSE added to head of " + dprintk(NDEBUG_QUEUES, "scsi%d: REQUEST SENSE added to head of " "issue queue\n", H_NO(cmd)); } else #endif /* def AUTOSENSE */ @@ -2277,7 +2267,7 @@ static void NCR5380_information_transfer(struct Scsi_Host *instance) cmd->device->tagged_supported = 0; hostdata->busy[cmd->device->id] |= (1 << cmd->device->lun); cmd->tag = TAG_NONE; - TAG_PRINTK("scsi%d: target %d lun %d rejected " + dprintk(NDEBUG_TAGS, "scsi%d: target %d lun %d rejected " "QUEUE_TAG message; tagged queuing " "disabled\n", HOSTNO, cmd->device->id, cmd->device->lun); @@ -2294,7 +2284,7 @@ static void NCR5380_information_transfer(struct Scsi_Host *instance) hostdata->connected = NULL; hostdata->disconnected_queue = cmd; local_irq_restore(flags); - QU_PRINTK("scsi%d: command for target %d lun %d was " + dprintk(NDEBUG_QUEUES, "scsi%d: command for target %d lun %d was " "moved from connected to the " "disconnected_queue\n", HOSTNO, cmd->device->id, cmd->device->lun); @@ -2344,13 +2334,13 @@ static void NCR5380_information_transfer(struct Scsi_Host *instance) /* Accept first byte by clearing ACK */ NCR5380_write(INITIATOR_COMMAND_REG, ICR_BASE); - EXT_PRINTK("scsi%d: receiving extended message\n", HOSTNO); + dprintk(NDEBUG_EXTENDED, "scsi%d: receiving extended message\n", HOSTNO); len = 2; data = extended_msg + 1; phase = PHASE_MSGIN; NCR5380_transfer_pio(instance, &phase, &len, &data); - EXT_PRINTK("scsi%d: length=%d, code=0x%02x\n", HOSTNO, + dprintk(NDEBUG_EXTENDED, "scsi%d: length=%d, code=0x%02x\n", HOSTNO, (int)extended_msg[1], (int)extended_msg[2]); if (!len && extended_msg[1] <= @@ -2362,7 +2352,7 @@ static void NCR5380_information_transfer(struct Scsi_Host *instance) phase = PHASE_MSGIN; NCR5380_transfer_pio(instance, &phase, &len, &data); - EXT_PRINTK("scsi%d: message received, residual %d\n", + dprintk(NDEBUG_EXTENDED, "scsi%d: message received, residual %d\n", HOSTNO, len); switch (extended_msg[2]) { @@ -2451,7 +2441,7 @@ static void NCR5380_information_transfer(struct Scsi_Host *instance) break; default: printk("scsi%d: unknown phase\n", HOSTNO); - NCR_PRINT(NDEBUG_ANY); + NCR5380_dprint(NDEBUG_ANY, instance); } /* switch(phase) */ } /* if (tmp * SR_REQ) */ } /* while (1) */ @@ -2493,7 +2483,7 @@ static void NCR5380_reselect(struct Scsi_Host *instance) target_mask = NCR5380_read(CURRENT_SCSI_DATA_REG) & ~(hostdata->id_mask); - RSL_PRINTK("scsi%d: reselect\n", HOSTNO); + dprintk(NDEBUG_RESELECTION, "scsi%d: reselect\n", HOSTNO); /* * At this point, we have detected that our SCSI ID is on the bus, @@ -2544,7 +2534,7 @@ static void NCR5380_reselect(struct Scsi_Host *instance) if (!NCR5380_transfer_pio(instance, &phase, &len, &data) && msg[1] == SIMPLE_QUEUE_TAG) tag = msg[2]; - TAG_PRINTK("scsi%d: target mask %02x, lun %d sent tag %d at " + dprintk(NDEBUG_TAGS, "scsi%d: target mask %02x, lun %d sent tag %d at " "reselection\n", HOSTNO, target_mask, lun, tag); } #endif @@ -2598,7 +2588,7 @@ static void NCR5380_reselect(struct Scsi_Host *instance) NCR5380_write(INITIATOR_COMMAND_REG, ICR_BASE); hostdata->connected = tmp; - RSL_PRINTK("scsi%d: nexus established, target = %d, lun = %d, tag = %d\n", + dprintk(NDEBUG_RESELECTION, "scsi%d: nexus established, target = %d, lun = %d, tag = %d\n", HOSTNO, tmp->device->id, tmp->device->lun, tmp->tag); falcon_dont_release--; } @@ -2640,7 +2630,7 @@ int NCR5380_abort(Scsi_Cmnd *cmd) printk(KERN_ERR "scsi%d: !!BINGO!! Falcon has no lock in NCR5380_abort\n", HOSTNO); - ABRT_PRINTK("scsi%d: abort called basr 0x%02x, sr 0x%02x\n", HOSTNO, + dprintk(NDEBUG_ABORT, "scsi%d: abort called basr 0x%02x, sr 0x%02x\n", HOSTNO, NCR5380_read(BUS_AND_STATUS_REG), NCR5380_read(STATUS_REG)); @@ -2653,7 +2643,7 @@ int NCR5380_abort(Scsi_Cmnd *cmd) if (hostdata->connected == cmd) { - ABRT_PRINTK("scsi%d: aborting connected command\n", HOSTNO); + dprintk(NDEBUG_ABORT, "scsi%d: aborting connected command\n", HOSTNO); /* * We should perform BSY checking, and make sure we haven't slipped * into BUS FREE. @@ -2683,11 +2673,11 @@ int NCR5380_abort(Scsi_Cmnd *cmd) local_irq_restore(flags); cmd->scsi_done(cmd); falcon_release_lock_if_possible(hostdata); - return SCSI_ABORT_SUCCESS; + return SUCCESS; } else { /* local_irq_restore(flags); */ printk("scsi%d: abort of connected command failed!\n", HOSTNO); - return SCSI_ABORT_ERROR; + return FAILED; } } #endif @@ -2705,13 +2695,13 @@ int NCR5380_abort(Scsi_Cmnd *cmd) SET_NEXT(tmp, NULL); tmp->result = DID_ABORT << 16; local_irq_restore(flags); - ABRT_PRINTK("scsi%d: abort removed command from issue queue.\n", + dprintk(NDEBUG_ABORT, "scsi%d: abort removed command from issue queue.\n", HOSTNO); /* Tagged queuing note: no tag to free here, hasn't been assigned * yet... */ tmp->scsi_done(tmp); falcon_release_lock_if_possible(hostdata); - return SCSI_ABORT_SUCCESS; + return SUCCESS; } } @@ -2728,8 +2718,8 @@ int NCR5380_abort(Scsi_Cmnd *cmd) if (hostdata->connected) { local_irq_restore(flags); - ABRT_PRINTK("scsi%d: abort failed, command connected.\n", HOSTNO); - return SCSI_ABORT_SNOOZE; + dprintk(NDEBUG_ABORT, "scsi%d: abort failed, command connected.\n", HOSTNO); + return FAILED; } /* @@ -2761,12 +2751,12 @@ int NCR5380_abort(Scsi_Cmnd *cmd) tmp = NEXT(tmp)) { if (cmd == tmp) { local_irq_restore(flags); - ABRT_PRINTK("scsi%d: aborting disconnected command.\n", HOSTNO); + dprintk(NDEBUG_ABORT, "scsi%d: aborting disconnected command.\n", HOSTNO); if (NCR5380_select(instance, cmd, (int)cmd->tag)) - return SCSI_ABORT_BUSY; + return FAILED; - ABRT_PRINTK("scsi%d: nexus reestablished.\n", HOSTNO); + dprintk(NDEBUG_ABORT, "scsi%d: nexus reestablished.\n", HOSTNO); do_abort(instance); @@ -2791,7 +2781,7 @@ int NCR5380_abort(Scsi_Cmnd *cmd) local_irq_restore(flags); tmp->scsi_done(tmp); falcon_release_lock_if_possible(hostdata); - return SCSI_ABORT_SUCCESS; + return SUCCESS; } } } @@ -2816,7 +2806,7 @@ int NCR5380_abort(Scsi_Cmnd *cmd) */ falcon_release_lock_if_possible(hostdata); - return SCSI_ABORT_NOT_RUNNING; + return FAILED; } @@ -2825,7 +2815,7 @@ int NCR5380_abort(Scsi_Cmnd *cmd) * * Purpose : reset the SCSI bus. * - * Returns : SCSI_RESET_WAKEUP + * Returns : SUCCESS or FAILURE * */ @@ -2834,7 +2824,7 @@ static int NCR5380_bus_reset(Scsi_Cmnd *cmd) SETUP_HOSTDATA(cmd->device->host); int i; unsigned long flags; -#if 1 +#if defined(RESET_RUN_DONE) Scsi_Cmnd *connected, *disconnected_queue; #endif @@ -2859,7 +2849,14 @@ static int NCR5380_bus_reset(Scsi_Cmnd *cmd) * through anymore ... */ (void)NCR5380_read(RESET_PARITY_INTERRUPT_REG); -#if 1 /* XXX Should now be done by midlevel code, but it's broken XXX */ + /* MSch 20140115 - looking at the generic NCR5380 driver, all of this + * should go. + * Catch-22: if we don't clear all queues, the SCSI driver lock will + * not be reset by atari_scsi_reset()! + */ + +#if defined(RESET_RUN_DONE) + /* XXX Should now be done by midlevel code, but it's broken XXX */ /* XXX see below XXX */ /* MSch: old-style reset: actually abort all command processing here */ @@ -2890,7 +2887,7 @@ static int NCR5380_bus_reset(Scsi_Cmnd *cmd) */ if ((cmd = connected)) { - ABRT_PRINTK("scsi%d: reset aborted a connected command\n", H_NO(cmd)); + dprintk(NDEBUG_ABORT, "scsi%d: reset aborted a connected command\n", H_NO(cmd)); cmd->result = (cmd->result & 0xffff) | (DID_RESET << 16); cmd->scsi_done(cmd); } @@ -2902,7 +2899,7 @@ static int NCR5380_bus_reset(Scsi_Cmnd *cmd) cmd->scsi_done(cmd); } if (i > 0) - ABRT_PRINTK("scsi: reset aborted %d disconnected command(s)\n", i); + dprintk(NDEBUG_ABORT, "scsi: reset aborted %d disconnected command(s)\n", i); /* The Falcon lock should be released after a reset... */ @@ -2915,7 +2912,7 @@ static int NCR5380_bus_reset(Scsi_Cmnd *cmd) * the midlevel code that the reset was SUCCESSFUL, and there is no * need to 'wake up' the commands by a request_sense */ - return SCSI_RESET_SUCCESS | SCSI_RESET_BUS_RESET; + return SUCCESS; #else /* 1 */ /* MSch: new-style reset handling: let the mid-level do what it can */ @@ -2942,11 +2939,11 @@ static int NCR5380_bus_reset(Scsi_Cmnd *cmd) */ if (hostdata->issue_queue) - ABRT_PRINTK("scsi%d: reset aborted issued command(s)\n", H_NO(cmd)); + dprintk(NDEBUG_ABORT, "scsi%d: reset aborted issued command(s)\n", H_NO(cmd)); if (hostdata->connected) - ABRT_PRINTK("scsi%d: reset aborted a connected command\n", H_NO(cmd)); + dprintk(NDEBUG_ABORT, "scsi%d: reset aborted a connected command\n", H_NO(cmd)); if (hostdata->disconnected_queue) - ABRT_PRINTK("scsi%d: reset aborted disconnected command(s)\n", H_NO(cmd)); + dprintk(NDEBUG_ABORT, "scsi%d: reset aborted disconnected command(s)\n", H_NO(cmd)); local_irq_save(flags); hostdata->issue_queue = NULL; @@ -2963,6 +2960,6 @@ static int NCR5380_bus_reset(Scsi_Cmnd *cmd) local_irq_restore(flags); /* we did no complete reset of all commands, so a wakeup is required */ - return SCSI_RESET_WAKEUP | SCSI_RESET_BUS_RESET; + return SUCCESS; #endif /* 1 */ } diff --git a/drivers/scsi/atari_scsi.c b/drivers/scsi/atari_scsi.c index a8d721ff19eb..b522134528d6 100644 --- a/drivers/scsi/atari_scsi.c +++ b/drivers/scsi/atari_scsi.c @@ -67,12 +67,6 @@ #include <linux/module.h> -#define NDEBUG (0) - -#define NDEBUG_ABORT 0x00100000 -#define NDEBUG_TAGS 0x00200000 -#define NDEBUG_MERGING 0x00400000 - #define AUTOSENSE /* For the Atari version, use only polled IO or REAL_DMA */ #define REAL_DMA @@ -314,7 +308,7 @@ static irqreturn_t scsi_tt_intr(int irq, void *dummy) dma_stat = tt_scsi_dma.dma_ctrl; - INT_PRINTK("scsi%d: NCR5380 interrupt, DMA status = %02x\n", + dprintk(NDEBUG_INTR, "scsi%d: NCR5380 interrupt, DMA status = %02x\n", atari_scsi_host->host_no, dma_stat & 0xff); /* Look if it was the DMA that has interrupted: First possibility @@ -340,7 +334,7 @@ static irqreturn_t scsi_tt_intr(int irq, void *dummy) if ((dma_stat & 0x02) && !(dma_stat & 0x40)) { atari_dma_residual = HOSTDATA_DMALEN - (SCSI_DMA_READ_P(dma_addr) - atari_dma_startaddr); - DMA_PRINTK("SCSI DMA: There are %ld residual bytes.\n", + dprintk(NDEBUG_DMA, "SCSI DMA: There are %ld residual bytes.\n", atari_dma_residual); if ((signed int)atari_dma_residual < 0) @@ -371,7 +365,7 @@ static irqreturn_t scsi_tt_intr(int irq, void *dummy) * other command. These shouldn't disconnect anyway. */ if (atari_dma_residual & 0x1ff) { - DMA_PRINTK("SCSI DMA: DMA bug corrected, " + dprintk(NDEBUG_DMA, "SCSI DMA: DMA bug corrected, " "difference %ld bytes\n", 512 - (atari_dma_residual & 0x1ff)); atari_dma_residual = (atari_dma_residual + 511) & ~0x1ff; @@ -438,7 +432,7 @@ static irqreturn_t scsi_falcon_intr(int irq, void *dummy) "ST-DMA fifo\n", transferred & 15); atari_dma_residual = HOSTDATA_DMALEN - transferred; - DMA_PRINTK("SCSI DMA: There are %ld residual bytes.\n", + dprintk(NDEBUG_DMA, "SCSI DMA: There are %ld residual bytes.\n", atari_dma_residual); } else atari_dma_residual = 0; @@ -474,11 +468,11 @@ static void atari_scsi_fetch_restbytes(void) /* there are 'nr' bytes left for the last long address before the DMA pointer */ phys_dst ^= nr; - DMA_PRINTK("SCSI DMA: there are %d rest bytes for phys addr 0x%08lx", + dprintk(NDEBUG_DMA, "SCSI DMA: there are %d rest bytes for phys addr 0x%08lx", nr, phys_dst); /* The content of the DMA pointer is a physical address! */ dst = phys_to_virt(phys_dst); - DMA_PRINTK(" = virt addr %p\n", dst); + dprintk(NDEBUG_DMA, " = virt addr %p\n", dst); for (src = (char *)&tt_scsi_dma.dma_restdata; nr != 0; --nr) *dst++ = *src++; } @@ -827,7 +821,7 @@ static int atari_scsi_bus_reset(Scsi_Cmnd *cmd) } else { atari_turnon_irq(IRQ_MFP_FSCSI); } - if ((rv & SCSI_RESET_ACTION) == SCSI_RESET_SUCCESS) + if (rv == SUCCESS) falcon_release_lock_if_possible(hostdata); return rv; @@ -883,7 +877,7 @@ static unsigned long atari_scsi_dma_setup(struct Scsi_Host *instance, { unsigned long addr = virt_to_phys(data); - DMA_PRINTK("scsi%d: setting up dma, data = %p, phys = %lx, count = %ld, " + dprintk(NDEBUG_DMA, "scsi%d: setting up dma, data = %p, phys = %lx, count = %ld, " "dir = %d\n", instance->host_no, data, addr, count, dir); if (!IS_A_TT() && !STRAM_ADDR(addr)) { @@ -1063,7 +1057,7 @@ static unsigned long atari_dma_xfer_len(unsigned long wanted_len, possible_len = limit; if (possible_len != wanted_len) - DMA_PRINTK("Sorry, must cut DMA transfer size to %ld bytes " + dprintk(NDEBUG_DMA, "Sorry, must cut DMA transfer size to %ld bytes " "instead of %ld\n", possible_len, wanted_len); return possible_len; diff --git a/drivers/scsi/atari_scsi.h b/drivers/scsi/atari_scsi.h index 11c624bb122d..3299d91d7336 100644 --- a/drivers/scsi/atari_scsi.h +++ b/drivers/scsi/atari_scsi.h @@ -54,125 +54,6 @@ #define NCR5380_dma_xfer_len(i,cmd,phase) \ atari_dma_xfer_len(cmd->SCp.this_residual,cmd,((phase) & SR_IO) ? 0 : 1) -/* former generic SCSI error handling stuff */ - -#define SCSI_ABORT_SNOOZE 0 -#define SCSI_ABORT_SUCCESS 1 -#define SCSI_ABORT_PENDING 2 -#define SCSI_ABORT_BUSY 3 -#define SCSI_ABORT_NOT_RUNNING 4 -#define SCSI_ABORT_ERROR 5 - -#define SCSI_RESET_SNOOZE 0 -#define SCSI_RESET_PUNT 1 -#define SCSI_RESET_SUCCESS 2 -#define SCSI_RESET_PENDING 3 -#define SCSI_RESET_WAKEUP 4 -#define SCSI_RESET_NOT_RUNNING 5 -#define SCSI_RESET_ERROR 6 - -#define SCSI_RESET_SYNCHRONOUS 0x01 -#define SCSI_RESET_ASYNCHRONOUS 0x02 -#define SCSI_RESET_SUGGEST_BUS_RESET 0x04 -#define SCSI_RESET_SUGGEST_HOST_RESET 0x08 - -#define SCSI_RESET_BUS_RESET 0x100 -#define SCSI_RESET_HOST_RESET 0x200 -#define SCSI_RESET_ACTION 0xff - -/* Debugging printk definitions: - * - * ARB -> arbitration - * ASEN -> auto-sense - * DMA -> DMA - * HSH -> PIO handshake - * INF -> information transfer - * INI -> initialization - * INT -> interrupt - * LNK -> linked commands - * MAIN -> NCR5380_main() control flow - * NDAT -> no data-out phase - * NWR -> no write commands - * PIO -> PIO transfers - * PDMA -> pseudo DMA (unused on Atari) - * QU -> queues - * RSL -> reselections - * SEL -> selections - * USL -> usleep cpde (unused on Atari) - * LBS -> last byte sent (unused on Atari) - * RSS -> restarting of selections - * EXT -> extended messages - * ABRT -> aborting and resetting - * TAG -> queue tag handling - * MER -> merging of consec. buffers - * - */ - -#define dprint(flg, format...) \ -({ \ - if (NDEBUG & (flg)) \ - printk(KERN_DEBUG format); \ -}) - -#define ARB_PRINTK(format, args...) \ - dprint(NDEBUG_ARBITRATION, format , ## args) -#define ASEN_PRINTK(format, args...) \ - dprint(NDEBUG_AUTOSENSE, format , ## args) -#define DMA_PRINTK(format, args...) \ - dprint(NDEBUG_DMA, format , ## args) -#define HSH_PRINTK(format, args...) \ - dprint(NDEBUG_HANDSHAKE, format , ## args) -#define INF_PRINTK(format, args...) \ - dprint(NDEBUG_INFORMATION, format , ## args) -#define INI_PRINTK(format, args...) \ - dprint(NDEBUG_INIT, format , ## args) -#define INT_PRINTK(format, args...) \ - dprint(NDEBUG_INTR, format , ## args) -#define LNK_PRINTK(format, args...) \ - dprint(NDEBUG_LINKED, format , ## args) -#define MAIN_PRINTK(format, args...) \ - dprint(NDEBUG_MAIN, format , ## args) -#define NDAT_PRINTK(format, args...) \ - dprint(NDEBUG_NO_DATAOUT, format , ## args) -#define NWR_PRINTK(format, args...) \ - dprint(NDEBUG_NO_WRITE, format , ## args) -#define PIO_PRINTK(format, args...) \ - dprint(NDEBUG_PIO, format , ## args) -#define PDMA_PRINTK(format, args...) \ - dprint(NDEBUG_PSEUDO_DMA, format , ## args) -#define QU_PRINTK(format, args...) \ - dprint(NDEBUG_QUEUES, format , ## args) -#define RSL_PRINTK(format, args...) \ - dprint(NDEBUG_RESELECTION, format , ## args) -#define SEL_PRINTK(format, args...) \ - dprint(NDEBUG_SELECTION, format , ## args) -#define USL_PRINTK(format, args...) \ - dprint(NDEBUG_USLEEP, format , ## args) -#define LBS_PRINTK(format, args...) \ - dprint(NDEBUG_LAST_BYTE_SENT, format , ## args) -#define RSS_PRINTK(format, args...) \ - dprint(NDEBUG_RESTART_SELECT, format , ## args) -#define EXT_PRINTK(format, args...) \ - dprint(NDEBUG_EXTENDED, format , ## args) -#define ABRT_PRINTK(format, args...) \ - dprint(NDEBUG_ABORT, format , ## args) -#define TAG_PRINTK(format, args...) \ - dprint(NDEBUG_TAGS, format , ## args) -#define MER_PRINTK(format, args...) \ - dprint(NDEBUG_MERGING, format , ## args) - -/* conditional macros for NCR5380_print_{,phase,status} */ - -#define NCR_PRINT(mask) \ - ((NDEBUG & (mask)) ? NCR5380_print(instance) : (void)0) - -#define NCR_PRINT_PHASE(mask) \ - ((NDEBUG & (mask)) ? NCR5380_print_phase(instance) : (void)0) - -#define NCR_PRINT_STATUS(mask) \ - ((NDEBUG & (mask)) ? NCR5380_print_status(instance) : (void)0) - - #endif /* ndef ASM */ #endif /* ATARI_SCSI_H */ diff --git a/drivers/scsi/be2iscsi/be.h b/drivers/scsi/be2iscsi/be.h index 1bfb0bd01198..860f527d8f26 100644 --- a/drivers/scsi/be2iscsi/be.h +++ b/drivers/scsi/be2iscsi/be.h @@ -83,9 +83,20 @@ static inline void queue_tail_inc(struct be_queue_info *q) /*ISCSI */ +struct be_aic_obj { /* Adaptive interrupt coalescing (AIC) info */ + bool enable; + u32 min_eqd; /* in usecs */ + u32 max_eqd; /* in usecs */ + u32 prev_eqd; /* in usecs */ + u32 et_eqd; /* configured val when aic is off */ + ulong jiffs; + u64 eq_prev; /* Used to calculate eqe */ +}; + struct be_eq_obj { bool todo_mcc_cq; bool todo_cq; + u32 cq_count; struct be_queue_info q; struct beiscsi_hba *phba; struct be_queue_info *cq; diff --git a/drivers/scsi/be2iscsi/be_cmds.h b/drivers/scsi/be2iscsi/be_cmds.h index 7cf7f99ee442..cc7405c0eca0 100644 --- a/drivers/scsi/be2iscsi/be_cmds.h +++ b/drivers/scsi/be2iscsi/be_cmds.h @@ -71,6 +71,7 @@ struct be_mcc_wrb { #define BEISCSI_FW_MBX_TIMEOUT 100 /* MBOX Command VER */ +#define MBX_CMD_VER1 0x01 #define MBX_CMD_VER2 0x02 struct be_mcc_compl { @@ -271,6 +272,12 @@ struct be_cmd_resp_eq_create { u16 rsvd0; /* sword */ } __packed; +struct be_set_eqd { + u32 eq_id; + u32 phase; + u32 delay_multiplier; +} __packed; + struct mgmt_chap_format { u32 flags; u8 intr_chap_name[256]; @@ -622,7 +629,7 @@ struct be_cmd_req_modify_eq_delay { u32 eq_id; u32 phase; u32 delay_multiplier; - } delay[8]; + } delay[MAX_CPUS]; } __packed; /******************** Get MAC ADDR *******************/ @@ -708,6 +715,8 @@ unsigned int be_cmd_get_port_speed(struct beiscsi_hba *phba); void free_mcc_tag(struct be_ctrl_info *ctrl, unsigned int tag); +int be_cmd_modify_eq_delay(struct beiscsi_hba *phba, struct be_set_eqd *, + int num); int beiscsi_mccq_compl(struct beiscsi_hba *phba, uint32_t tag, struct be_mcc_wrb **wrb, struct be_dma_mem *mbx_cmd_mem); @@ -1005,6 +1014,26 @@ struct tcp_connect_and_offload_in { u8 rsvd0[3]; } __packed; +struct tcp_connect_and_offload_in_v1 { + struct be_cmd_req_hdr hdr; + struct ip_addr_format ip_address; + u16 tcp_port; + u16 cid; + u16 cq_id; + u16 defq_id; + struct phys_addr dataout_template_pa; + u16 hdr_ring_id; + u16 data_ring_id; + u8 do_offload; + u8 ifd_state; + u8 rsvd0[2]; + u16 tcp_window_size; + u8 tcp_window_scale_count; + u8 rsvd1; + u32 tcp_mss:24; + u8 rsvd2; +} __packed; + struct tcp_connect_and_offload_out { struct be_cmd_resp_hdr hdr; u32 connection_handle; diff --git a/drivers/scsi/be2iscsi/be_iscsi.c b/drivers/scsi/be2iscsi/be_iscsi.c index a3df43324c98..fd284ff36ecf 100644 --- a/drivers/scsi/be2iscsi/be_iscsi.c +++ b/drivers/scsi/be2iscsi/be_iscsi.c @@ -1106,7 +1106,7 @@ static int beiscsi_open_conn(struct iscsi_endpoint *ep, struct beiscsi_hba *phba = beiscsi_ep->phba; struct tcp_connect_and_offload_out *ptcpcnct_out; struct be_dma_mem nonemb_cmd; - unsigned int tag; + unsigned int tag, req_memsize; int ret = -ENOMEM; beiscsi_log(phba, KERN_INFO, BEISCSI_LOG_CONFIG, @@ -1127,8 +1127,14 @@ static int beiscsi_open_conn(struct iscsi_endpoint *ep, (beiscsi_ep->ep_cid)] = ep; beiscsi_ep->cid_vld = 0; + + if (is_chip_be2_be3r(phba)) + req_memsize = sizeof(struct tcp_connect_and_offload_in); + else + req_memsize = sizeof(struct tcp_connect_and_offload_in_v1); + nonemb_cmd.va = pci_alloc_consistent(phba->ctrl.pdev, - sizeof(struct tcp_connect_and_offload_in), + req_memsize, &nonemb_cmd.dma); if (nonemb_cmd.va == NULL) { @@ -1139,7 +1145,7 @@ static int beiscsi_open_conn(struct iscsi_endpoint *ep, beiscsi_free_ep(beiscsi_ep); return -ENOMEM; } - nonemb_cmd.size = sizeof(struct tcp_connect_and_offload_in); + nonemb_cmd.size = req_memsize; memset(nonemb_cmd.va, 0, nonemb_cmd.size); tag = mgmt_open_connection(phba, dst_addr, beiscsi_ep, &nonemb_cmd); if (tag <= 0) { diff --git a/drivers/scsi/be2iscsi/be_main.c b/drivers/scsi/be2iscsi/be_main.c index 0d822297aa80..554349029628 100644 --- a/drivers/scsi/be2iscsi/be_main.c +++ b/drivers/scsi/be2iscsi/be_main.c @@ -599,15 +599,7 @@ static struct beiscsi_hba *beiscsi_hba_alloc(struct pci_dev *pcidev) pci_set_drvdata(pcidev, phba); phba->interface_handle = 0xFFFFFFFF; - if (iscsi_host_add(shost, &phba->pcidev->dev)) - goto free_devices; - return phba; - -free_devices: - pci_dev_put(phba->pcidev); - iscsi_host_free(phba->shost); - return NULL; } static void beiscsi_unmap_pci_function(struct beiscsi_hba *phba) @@ -2279,6 +2271,7 @@ static int be_iopoll(struct blk_iopoll *iop, int budget) pbe_eq = container_of(iop, struct be_eq_obj, iopoll); ret = beiscsi_process_cq(pbe_eq); + pbe_eq->cq_count += ret; if (ret < budget) { phba = pbe_eq->phba; blk_iopoll_complete(iop); @@ -3692,7 +3685,7 @@ static void hwi_cleanup(struct beiscsi_hba *phba) struct hwi_controller *phwi_ctrlr; struct hwi_context_memory *phwi_context; struct hwi_async_pdu_context *pasync_ctx; - int i, eq_num, ulp_num; + int i, eq_for_mcc, ulp_num; phwi_ctrlr = phba->phwi_ctrlr; phwi_context = phwi_ctrlr->phwi_ctxt; @@ -3729,16 +3722,17 @@ static void hwi_cleanup(struct beiscsi_hba *phba) if (q->created) beiscsi_cmd_q_destroy(ctrl, q, QTYPE_CQ); } + + be_mcc_queues_destroy(phba); if (phba->msix_enabled) - eq_num = 1; + eq_for_mcc = 1; else - eq_num = 0; - for (i = 0; i < (phba->num_cpus + eq_num); i++) { + eq_for_mcc = 0; + for (i = 0; i < (phba->num_cpus + eq_for_mcc); i++) { q = &phwi_context->be_eq[i].q; if (q->created) beiscsi_cmd_q_destroy(ctrl, q, QTYPE_EQ); } - be_mcc_queues_destroy(phba); be_cmd_fw_uninit(ctrl); } @@ -3833,9 +3827,9 @@ static int hwi_init_port(struct beiscsi_hba *phba) phwi_ctrlr = phba->phwi_ctrlr; phwi_context = phwi_ctrlr->phwi_ctxt; - phwi_context->max_eqd = 0; + phwi_context->max_eqd = 128; phwi_context->min_eqd = 0; - phwi_context->cur_eqd = 64; + phwi_context->cur_eqd = 0; be_cmd_fw_initialize(&phba->ctrl); status = beiscsi_create_eqs(phba, phwi_context); @@ -5290,6 +5284,57 @@ static void beiscsi_msix_enable(struct beiscsi_hba *phba) return; } +static void be_eqd_update(struct beiscsi_hba *phba) +{ + struct be_set_eqd set_eqd[MAX_CPUS]; + struct be_aic_obj *aic; + struct be_eq_obj *pbe_eq; + struct hwi_controller *phwi_ctrlr; + struct hwi_context_memory *phwi_context; + int eqd, i, num = 0; + ulong now; + u32 pps, delta; + unsigned int tag; + + phwi_ctrlr = phba->phwi_ctrlr; + phwi_context = phwi_ctrlr->phwi_ctxt; + + for (i = 0; i <= phba->num_cpus; i++) { + aic = &phba->aic_obj[i]; + pbe_eq = &phwi_context->be_eq[i]; + now = jiffies; + if (!aic->jiffs || time_before(now, aic->jiffs) || + pbe_eq->cq_count < aic->eq_prev) { + aic->jiffs = now; + aic->eq_prev = pbe_eq->cq_count; + continue; + } + delta = jiffies_to_msecs(now - aic->jiffs); + pps = (((u32)(pbe_eq->cq_count - aic->eq_prev) * 1000) / delta); + eqd = (pps / 1500) << 2; + + if (eqd < 8) + eqd = 0; + eqd = min_t(u32, eqd, phwi_context->max_eqd); + eqd = max_t(u32, eqd, phwi_context->min_eqd); + + aic->jiffs = now; + aic->eq_prev = pbe_eq->cq_count; + + if (eqd != aic->prev_eqd) { + set_eqd[num].delay_multiplier = (eqd * 65)/100; + set_eqd[num].eq_id = pbe_eq->q.id; + aic->prev_eqd = eqd; + num++; + } + } + if (num) { + tag = be_cmd_modify_eq_delay(phba, set_eqd, num); + if (tag) + beiscsi_mccq_compl(phba, tag, NULL, NULL); + } +} + /* * beiscsi_hw_health_check()- Check adapter health * @work: work item to check HW health @@ -5303,6 +5348,8 @@ beiscsi_hw_health_check(struct work_struct *work) container_of(work, struct beiscsi_hba, beiscsi_hw_check_task.work); + be_eqd_update(phba); + beiscsi_ue_detect(phba); schedule_delayed_work(&phba->beiscsi_hw_check_task, @@ -5579,7 +5626,7 @@ static int beiscsi_dev_probe(struct pci_dev *pcidev, phba->ctrl.mcc_numtag[i + 1] = 0; phba->ctrl.mcc_tag_available++; memset(&phba->ctrl.ptag_state[i].tag_mem_state, 0, - sizeof(struct beiscsi_mcc_tag_state)); + sizeof(struct be_dma_mem)); } phba->ctrl.mcc_alloc_index = phba->ctrl.mcc_free_index = 0; @@ -5621,6 +5668,9 @@ static int beiscsi_dev_probe(struct pci_dev *pcidev, } hwi_enable_intr(phba); + if (iscsi_host_add(phba->shost, &phba->pcidev->dev)) + goto free_blkenbld; + if (beiscsi_setup_boot_info(phba)) /* * log error but continue, because we may not be using diff --git a/drivers/scsi/be2iscsi/be_main.h b/drivers/scsi/be2iscsi/be_main.h index 9380b55bdeaf..9ceab426eec9 100644 --- a/drivers/scsi/be2iscsi/be_main.h +++ b/drivers/scsi/be2iscsi/be_main.h @@ -36,7 +36,7 @@ #include <scsi/scsi_transport_iscsi.h> #define DRV_NAME "be2iscsi" -#define BUILD_STR "10.2.125.0" +#define BUILD_STR "10.2.273.0" #define BE_NAME "Emulex OneConnect" \ "Open-iSCSI Driver version" BUILD_STR #define DRV_DESC BE_NAME " " "Driver" @@ -71,8 +71,8 @@ #define BEISCSI_SGLIST_ELEMENTS 30 -#define BEISCSI_CMD_PER_LUN 128 /* scsi_host->cmd_per_lun */ -#define BEISCSI_MAX_SECTORS 2048 /* scsi_host->max_sectors */ +#define BEISCSI_CMD_PER_LUN 128 /* scsi_host->cmd_per_lun */ +#define BEISCSI_MAX_SECTORS 1024 /* scsi_host->max_sectors */ #define BEISCSI_TEMPLATE_HDR_PER_CXN_SIZE 128 /* Template size per cxn */ #define BEISCSI_MAX_CMD_LEN 16 /* scsi_host->max_cmd_len */ @@ -427,6 +427,7 @@ struct beiscsi_hba { struct mgmt_session_info boot_sess; struct invalidate_command_table inv_tbl[128]; + struct be_aic_obj aic_obj[MAX_CPUS]; unsigned int attr_log_enable; int (*iotask_fn)(struct iscsi_task *, struct scatterlist *sg, diff --git a/drivers/scsi/be2iscsi/be_mgmt.c b/drivers/scsi/be2iscsi/be_mgmt.c index 088bdf752cfa..6045aa78986a 100644 --- a/drivers/scsi/be2iscsi/be_mgmt.c +++ b/drivers/scsi/be2iscsi/be_mgmt.c @@ -155,6 +155,43 @@ void beiscsi_ue_detect(struct beiscsi_hba *phba) } } +int be_cmd_modify_eq_delay(struct beiscsi_hba *phba, + struct be_set_eqd *set_eqd, int num) +{ + struct be_ctrl_info *ctrl = &phba->ctrl; + struct be_mcc_wrb *wrb; + struct be_cmd_req_modify_eq_delay *req; + unsigned int tag = 0; + int i; + + spin_lock(&ctrl->mbox_lock); + tag = alloc_mcc_tag(phba); + if (!tag) { + spin_unlock(&ctrl->mbox_lock); + return tag; + } + + wrb = wrb_from_mccq(phba); + req = embedded_payload(wrb); + + wrb->tag0 |= tag; + be_wrb_hdr_prepare(wrb, sizeof(*req), true, 0); + be_cmd_hdr_prepare(&req->hdr, CMD_SUBSYSTEM_COMMON, + OPCODE_COMMON_MODIFY_EQ_DELAY, sizeof(*req)); + + req->num_eq = cpu_to_le32(num); + for (i = 0; i < num; i++) { + req->delay[i].eq_id = cpu_to_le32(set_eqd[i].eq_id); + req->delay[i].phase = 0; + req->delay[i].delay_multiplier = + cpu_to_le32(set_eqd[i].delay_multiplier); + } + + be_mcc_notify(phba); + spin_unlock(&ctrl->mbox_lock); + return tag; +} + /** * mgmt_reopen_session()- Reopen a session based on reopen_type * @phba: Device priv structure instance @@ -447,8 +484,8 @@ unsigned int mgmt_vendor_specific_fw_cmd(struct be_ctrl_info *ctrl, struct be_dma_mem *nonemb_cmd) { struct be_cmd_resp_hdr *resp; - struct be_mcc_wrb *wrb = wrb_from_mccq(phba); - struct be_sge *mcc_sge = nonembedded_sgl(wrb); + struct be_mcc_wrb *wrb; + struct be_sge *mcc_sge; unsigned int tag = 0; struct iscsi_bsg_request *bsg_req = job->request; struct be_bsg_vendor_cmd *req = nonemb_cmd->va; @@ -465,7 +502,6 @@ unsigned int mgmt_vendor_specific_fw_cmd(struct be_ctrl_info *ctrl, req->sector = sector; req->offset = offset; spin_lock(&ctrl->mbox_lock); - memset(wrb, 0, sizeof(*wrb)); switch (bsg_req->rqst_data.h_vendor.vendor_cmd[0]) { case BEISCSI_WRITE_FLASH: @@ -495,6 +531,8 @@ unsigned int mgmt_vendor_specific_fw_cmd(struct be_ctrl_info *ctrl, return tag; } + wrb = wrb_from_mccq(phba); + mcc_sge = nonembedded_sgl(wrb); be_wrb_hdr_prepare(wrb, nonemb_cmd->size, false, job->request_payload.sg_cnt); mcc_sge->pa_hi = cpu_to_le32(upper_32_bits(nonemb_cmd->dma)); @@ -525,7 +563,6 @@ int mgmt_epfw_cleanup(struct beiscsi_hba *phba, unsigned short ulp_num) int status = 0; spin_lock(&ctrl->mbox_lock); - memset(wrb, 0, sizeof(*wrb)); be_wrb_hdr_prepare(wrb, sizeof(*req), true, 0); be_cmd_hdr_prepare(&req->hdr, CMD_SUBSYSTEM_ISCSI, @@ -675,7 +712,7 @@ int mgmt_open_connection(struct beiscsi_hba *phba, struct sockaddr_in6 *daddr_in6 = (struct sockaddr_in6 *)dst_addr; struct be_ctrl_info *ctrl = &phba->ctrl; struct be_mcc_wrb *wrb; - struct tcp_connect_and_offload_in *req; + struct tcp_connect_and_offload_in_v1 *req; unsigned short def_hdr_id; unsigned short def_data_id; struct phys_addr template_address = { 0, 0 }; @@ -702,17 +739,16 @@ int mgmt_open_connection(struct beiscsi_hba *phba, return tag; } wrb = wrb_from_mccq(phba); - memset(wrb, 0, sizeof(*wrb)); sge = nonembedded_sgl(wrb); req = nonemb_cmd->va; memset(req, 0, sizeof(*req)); wrb->tag0 |= tag; - be_wrb_hdr_prepare(wrb, sizeof(*req), false, 1); + be_wrb_hdr_prepare(wrb, nonemb_cmd->size, false, 1); be_cmd_hdr_prepare(&req->hdr, CMD_SUBSYSTEM_ISCSI, OPCODE_COMMON_ISCSI_TCP_CONNECT_AND_OFFLOAD, - sizeof(*req)); + nonemb_cmd->size); if (dst_addr->sa_family == PF_INET) { __be32 s_addr = daddr_in->sin_addr.s_addr; req->ip_address.ip_type = BE2_IPV4; @@ -758,6 +794,13 @@ int mgmt_open_connection(struct beiscsi_hba *phba, sge->pa_hi = cpu_to_le32(upper_32_bits(nonemb_cmd->dma)); sge->pa_lo = cpu_to_le32(nonemb_cmd->dma & 0xFFFFFFFF); sge->len = cpu_to_le32(nonemb_cmd->size); + + if (!is_chip_be2_be3r(phba)) { + req->hdr.version = MBX_CMD_VER1; + req->tcp_window_size = 0; + req->tcp_window_scale_count = 2; + } + be_mcc_notify(phba); spin_unlock(&ctrl->mbox_lock); return tag; @@ -804,7 +847,7 @@ static int mgmt_exec_nonemb_cmd(struct beiscsi_hba *phba, int resp_buf_len) { struct be_ctrl_info *ctrl = &phba->ctrl; - struct be_mcc_wrb *wrb = wrb_from_mccq(phba); + struct be_mcc_wrb *wrb; struct be_sge *sge; unsigned int tag; int rc = 0; @@ -816,7 +859,8 @@ static int mgmt_exec_nonemb_cmd(struct beiscsi_hba *phba, rc = -ENOMEM; goto free_cmd; } - memset(wrb, 0, sizeof(*wrb)); + + wrb = wrb_from_mccq(phba); wrb->tag0 |= tag; sge = nonembedded_sgl(wrb); diff --git a/drivers/scsi/be2iscsi/be_mgmt.h b/drivers/scsi/be2iscsi/be_mgmt.h index 01b8c97284c0..24a8fc577477 100644 --- a/drivers/scsi/be2iscsi/be_mgmt.h +++ b/drivers/scsi/be2iscsi/be_mgmt.h @@ -335,5 +335,7 @@ void beiscsi_offload_cxn_v0(struct beiscsi_offload_params *params, void beiscsi_offload_cxn_v2(struct beiscsi_offload_params *params, struct wrb_handle *pwrb_handle); void beiscsi_ue_detect(struct beiscsi_hba *phba); +int be_cmd_modify_eq_delay(struct beiscsi_hba *phba, + struct be_set_eqd *, int num); #endif diff --git a/drivers/scsi/bfa/bfad.c b/drivers/scsi/bfa/bfad.c index cc0fbcdc5192..7593b7c1d336 100644 --- a/drivers/scsi/bfa/bfad.c +++ b/drivers/scsi/bfa/bfad.c @@ -507,7 +507,7 @@ bfa_fcb_pbc_vport_create(struct bfad_s *bfad, struct bfi_pbc_vport_s pbc_vport) struct bfad_vport_s *vport; int rc; - vport = kzalloc(sizeof(struct bfad_vport_s), GFP_KERNEL); + vport = kzalloc(sizeof(struct bfad_vport_s), GFP_ATOMIC); if (!vport) { bfa_trc(bfad, 0); return; diff --git a/drivers/scsi/bnx2fc/bnx2fc_hwi.c b/drivers/scsi/bnx2fc/bnx2fc_hwi.c index 46a37657307f..512aed3ae4f1 100644 --- a/drivers/scsi/bnx2fc/bnx2fc_hwi.c +++ b/drivers/scsi/bnx2fc/bnx2fc_hwi.c @@ -1966,26 +1966,29 @@ static void bnx2fc_free_hash_table(struct bnx2fc_hba *hba) { int i; int segment_count; - int hash_table_size; u32 *pbl; - segment_count = hba->hash_tbl_segment_count; - hash_table_size = BNX2FC_NUM_MAX_SESS * BNX2FC_MAX_ROWS_IN_HASH_TBL * - sizeof(struct fcoe_hash_table_entry); + if (hba->hash_tbl_segments) { - pbl = hba->hash_tbl_pbl; - for (i = 0; i < segment_count; ++i) { - dma_addr_t dma_address; + pbl = hba->hash_tbl_pbl; + if (pbl) { + segment_count = hba->hash_tbl_segment_count; + for (i = 0; i < segment_count; ++i) { + dma_addr_t dma_address; - dma_address = le32_to_cpu(*pbl); - ++pbl; - dma_address += ((u64)le32_to_cpu(*pbl)) << 32; - ++pbl; - dma_free_coherent(&hba->pcidev->dev, - BNX2FC_HASH_TBL_CHUNK_SIZE, - hba->hash_tbl_segments[i], - dma_address); + dma_address = le32_to_cpu(*pbl); + ++pbl; + dma_address += ((u64)le32_to_cpu(*pbl)) << 32; + ++pbl; + dma_free_coherent(&hba->pcidev->dev, + BNX2FC_HASH_TBL_CHUNK_SIZE, + hba->hash_tbl_segments[i], + dma_address); + } + } + kfree(hba->hash_tbl_segments); + hba->hash_tbl_segments = NULL; } if (hba->hash_tbl_pbl) { @@ -2023,7 +2026,7 @@ static int bnx2fc_allocate_hash_table(struct bnx2fc_hba *hba) dma_segment_array = kzalloc(dma_segment_array_size, GFP_KERNEL); if (!dma_segment_array) { printk(KERN_ERR PFX "hash table pointers (dma) alloc failed\n"); - return -ENOMEM; + goto cleanup_ht; } for (i = 0; i < segment_count; ++i) { @@ -2034,15 +2037,7 @@ static int bnx2fc_allocate_hash_table(struct bnx2fc_hba *hba) GFP_KERNEL); if (!hba->hash_tbl_segments[i]) { printk(KERN_ERR PFX "hash segment alloc failed\n"); - while (--i >= 0) { - dma_free_coherent(&hba->pcidev->dev, - BNX2FC_HASH_TBL_CHUNK_SIZE, - hba->hash_tbl_segments[i], - dma_segment_array[i]); - hba->hash_tbl_segments[i] = NULL; - } - kfree(dma_segment_array); - return -ENOMEM; + goto cleanup_dma; } memset(hba->hash_tbl_segments[i], 0, BNX2FC_HASH_TBL_CHUNK_SIZE); @@ -2054,8 +2049,7 @@ static int bnx2fc_allocate_hash_table(struct bnx2fc_hba *hba) GFP_KERNEL); if (!hba->hash_tbl_pbl) { printk(KERN_ERR PFX "hash table pbl alloc failed\n"); - kfree(dma_segment_array); - return -ENOMEM; + goto cleanup_dma; } memset(hba->hash_tbl_pbl, 0, PAGE_SIZE); @@ -2080,6 +2074,22 @@ static int bnx2fc_allocate_hash_table(struct bnx2fc_hba *hba) } kfree(dma_segment_array); return 0; + +cleanup_dma: + for (i = 0; i < segment_count; ++i) { + if (hba->hash_tbl_segments[i]) + dma_free_coherent(&hba->pcidev->dev, + BNX2FC_HASH_TBL_CHUNK_SIZE, + hba->hash_tbl_segments[i], + dma_segment_array[i]); + } + + kfree(dma_segment_array); + +cleanup_ht: + kfree(hba->hash_tbl_segments); + hba->hash_tbl_segments = NULL; + return -ENOMEM; } /** diff --git a/drivers/scsi/dtc.c b/drivers/scsi/dtc.c index eb29fe7eaf49..0a667fe05006 100644 --- a/drivers/scsi/dtc.c +++ b/drivers/scsi/dtc.c @@ -3,8 +3,6 @@ #define PSEUDO_DMA #define DONT_USE_INTR #define UNSAFE /* Leave interrupts enabled during pseudo-dma I/O */ -#define xNDEBUG (NDEBUG_INTR+NDEBUG_RESELECTION+\ - NDEBUG_SELECTION+NDEBUG_ARBITRATION) #define DMA_WORKS_RIGHT diff --git a/drivers/scsi/esas2r/esas2r_main.c b/drivers/scsi/esas2r/esas2r_main.c index f37f3e3dd5d5..6504a195c874 100644 --- a/drivers/scsi/esas2r/esas2r_main.c +++ b/drivers/scsi/esas2r/esas2r_main.c @@ -390,7 +390,7 @@ static int esas2r_probe(struct pci_dev *pcid, esas2r_log_dev(ESAS2R_LOG_INFO, &(pcid->dev), "pci_enable_device() OK"); esas2r_log_dev(ESAS2R_LOG_INFO, &(pcid->dev), - "after pci_device_enable() enable_cnt: %d", + "after pci_enable_device() enable_cnt: %d", pcid->enable_cnt.counter); host = scsi_host_alloc(&driver_template, host_alloc_size); diff --git a/drivers/scsi/fnic/fnic.h b/drivers/scsi/fnic/fnic.h index 528d43b7b569..1d3521e13d77 100644 --- a/drivers/scsi/fnic/fnic.h +++ b/drivers/scsi/fnic/fnic.h @@ -39,14 +39,15 @@ #define DRV_NAME "fnic" #define DRV_DESCRIPTION "Cisco FCoE HBA Driver" -#define DRV_VERSION "1.5.0.45" +#define DRV_VERSION "1.6.0.10" #define PFX DRV_NAME ": " #define DFX DRV_NAME "%d: " #define DESC_CLEAN_LOW_WATERMARK 8 #define FNIC_UCSM_DFLT_THROTTLE_CNT_BLD 16 /* UCSM default throttle count */ #define FNIC_MIN_IO_REQ 256 /* Min IO throttle count */ -#define FNIC_MAX_IO_REQ 2048 /* scsi_cmnd tag map entries */ +#define FNIC_MAX_IO_REQ 1024 /* scsi_cmnd tag map entries */ +#define FNIC_DFLT_IO_REQ 256 /* Default scsi_cmnd tag map entries */ #define FNIC_IO_LOCKS 64 /* IO locks: power of 2 */ #define FNIC_DFLT_QUEUE_DEPTH 32 #define FNIC_STATS_RATE_LIMIT 4 /* limit rate at which stats are pulled up */ diff --git a/drivers/scsi/fnic/fnic_debugfs.c b/drivers/scsi/fnic/fnic_debugfs.c index b6073f875761..2c613bdea78f 100644 --- a/drivers/scsi/fnic/fnic_debugfs.c +++ b/drivers/scsi/fnic/fnic_debugfs.c @@ -25,6 +25,21 @@ static struct dentry *fnic_trace_debugfs_file; static struct dentry *fnic_trace_enable; static struct dentry *fnic_stats_debugfs_root; +static struct dentry *fnic_fc_trace_debugfs_file; +static struct dentry *fnic_fc_rdata_trace_debugfs_file; +static struct dentry *fnic_fc_trace_enable; +static struct dentry *fnic_fc_trace_clear; + +struct fc_trace_flag_type { + u8 fc_row_file; + u8 fc_normal_file; + u8 fnic_trace; + u8 fc_trace; + u8 fc_clear; +}; + +static struct fc_trace_flag_type *fc_trc_flag; + /* * fnic_debugfs_init - Initialize debugfs for fnic debug logging * @@ -56,6 +71,18 @@ int fnic_debugfs_init(void) return rc; } + /* Allocate memory to structure */ + fc_trc_flag = (struct fc_trace_flag_type *) + vmalloc(sizeof(struct fc_trace_flag_type)); + + if (fc_trc_flag) { + fc_trc_flag->fc_row_file = 0; + fc_trc_flag->fc_normal_file = 1; + fc_trc_flag->fnic_trace = 2; + fc_trc_flag->fc_trace = 3; + fc_trc_flag->fc_clear = 4; + } + rc = 0; return rc; } @@ -74,15 +101,19 @@ void fnic_debugfs_terminate(void) debugfs_remove(fnic_trace_debugfs_root); fnic_trace_debugfs_root = NULL; + + if (fc_trc_flag) + vfree(fc_trc_flag); } /* - * fnic_trace_ctrl_open - Open the trace_enable file + * fnic_trace_ctrl_open - Open the trace_enable file for fnic_trace + * Or Open fc_trace_enable file for fc_trace * @inode: The inode pointer. * @file: The file pointer to attach the trace enable/disable flag. * * Description: - * This routine opens a debugsfs file trace_enable. + * This routine opens a debugsfs file trace_enable or fc_trace_enable. * * Returns: * This function returns zero if successful. @@ -94,15 +125,19 @@ static int fnic_trace_ctrl_open(struct inode *inode, struct file *filp) } /* - * fnic_trace_ctrl_read - Read a trace_enable debugfs file + * fnic_trace_ctrl_read - + * Read trace_enable ,fc_trace_enable + * or fc_trace_clear debugfs file * @filp: The file pointer to read from. * @ubuf: The buffer to copy the data to. * @cnt: The number of bytes to read. * @ppos: The position in the file to start reading from. * * Description: - * This routine reads value of variable fnic_tracing_enabled - * and stores into local @buf. It will start reading file at @ppos and + * This routine reads value of variable fnic_tracing_enabled or + * fnic_fc_tracing_enabled or fnic_fc_trace_cleared + * and stores into local @buf. + * It will start reading file at @ppos and * copy up to @cnt of data to @ubuf from @buf. * * Returns: @@ -114,13 +149,25 @@ static ssize_t fnic_trace_ctrl_read(struct file *filp, { char buf[64]; int len; - len = sprintf(buf, "%u\n", fnic_tracing_enabled); + u8 *trace_type; + len = 0; + trace_type = (u8 *)filp->private_data; + if (*trace_type == fc_trc_flag->fnic_trace) + len = sprintf(buf, "%u\n", fnic_tracing_enabled); + else if (*trace_type == fc_trc_flag->fc_trace) + len = sprintf(buf, "%u\n", fnic_fc_tracing_enabled); + else if (*trace_type == fc_trc_flag->fc_clear) + len = sprintf(buf, "%u\n", fnic_fc_trace_cleared); + else + pr_err("fnic: Cannot read to any debugfs file\n"); return simple_read_from_buffer(ubuf, cnt, ppos, buf, len); } /* - * fnic_trace_ctrl_write - Write to trace_enable debugfs file + * fnic_trace_ctrl_write - + * Write to trace_enable, fc_trace_enable or + * fc_trace_clear debugfs file * @filp: The file pointer to write from. * @ubuf: The buffer to copy the data from. * @cnt: The number of bytes to write. @@ -128,7 +175,8 @@ static ssize_t fnic_trace_ctrl_read(struct file *filp, * * Description: * This routine writes data from user buffer @ubuf to buffer @buf and - * sets fnic_tracing_enabled value as per user input. + * sets fc_trace_enable ,tracing_enable or fnic_fc_trace_cleared + * value as per user input. * * Returns: * This function returns the amount of data that was written. @@ -140,6 +188,8 @@ static ssize_t fnic_trace_ctrl_write(struct file *filp, char buf[64]; unsigned long val; int ret; + u8 *trace_type; + trace_type = (u8 *)filp->private_data; if (cnt >= sizeof(buf)) return -EINVAL; @@ -153,12 +203,27 @@ static ssize_t fnic_trace_ctrl_write(struct file *filp, if (ret < 0) return ret; - fnic_tracing_enabled = val; + if (*trace_type == fc_trc_flag->fnic_trace) + fnic_tracing_enabled = val; + else if (*trace_type == fc_trc_flag->fc_trace) + fnic_fc_tracing_enabled = val; + else if (*trace_type == fc_trc_flag->fc_clear) + fnic_fc_trace_cleared = val; + else + pr_err("fnic: cannot write to any debufs file\n"); + (*ppos)++; return cnt; } +static const struct file_operations fnic_trace_ctrl_fops = { + .owner = THIS_MODULE, + .open = fnic_trace_ctrl_open, + .read = fnic_trace_ctrl_read, + .write = fnic_trace_ctrl_write, +}; + /* * fnic_trace_debugfs_open - Open the fnic trace log * @inode: The inode pointer @@ -178,19 +243,36 @@ static int fnic_trace_debugfs_open(struct inode *inode, struct file *file) { fnic_dbgfs_t *fnic_dbg_prt; + u8 *rdata_ptr; + rdata_ptr = (u8 *)inode->i_private; fnic_dbg_prt = kzalloc(sizeof(fnic_dbgfs_t), GFP_KERNEL); if (!fnic_dbg_prt) return -ENOMEM; - fnic_dbg_prt->buffer = vmalloc((3*(trace_max_pages * PAGE_SIZE))); - if (!fnic_dbg_prt->buffer) { - kfree(fnic_dbg_prt); - return -ENOMEM; + if (*rdata_ptr == fc_trc_flag->fnic_trace) { + fnic_dbg_prt->buffer = vmalloc(3 * + (trace_max_pages * PAGE_SIZE)); + if (!fnic_dbg_prt->buffer) { + kfree(fnic_dbg_prt); + return -ENOMEM; + } + memset((void *)fnic_dbg_prt->buffer, 0, + 3 * (trace_max_pages * PAGE_SIZE)); + fnic_dbg_prt->buffer_len = fnic_get_trace_data(fnic_dbg_prt); + } else { + fnic_dbg_prt->buffer = + vmalloc(3 * (fnic_fc_trace_max_pages * PAGE_SIZE)); + if (!fnic_dbg_prt->buffer) { + kfree(fnic_dbg_prt); + return -ENOMEM; + } + memset((void *)fnic_dbg_prt->buffer, 0, + 3 * (fnic_fc_trace_max_pages * PAGE_SIZE)); + fnic_dbg_prt->buffer_len = + fnic_fc_trace_get_data(fnic_dbg_prt, *rdata_ptr); } - memset((void *)fnic_dbg_prt->buffer, 0, - (3*(trace_max_pages * PAGE_SIZE))); - fnic_dbg_prt->buffer_len = fnic_get_trace_data(fnic_dbg_prt); file->private_data = fnic_dbg_prt; + return 0; } @@ -272,13 +354,6 @@ static int fnic_trace_debugfs_release(struct inode *inode, return 0; } -static const struct file_operations fnic_trace_ctrl_fops = { - .owner = THIS_MODULE, - .open = fnic_trace_ctrl_open, - .read = fnic_trace_ctrl_read, - .write = fnic_trace_ctrl_write, -}; - static const struct file_operations fnic_trace_debugfs_fops = { .owner = THIS_MODULE, .open = fnic_trace_debugfs_open, @@ -306,9 +381,10 @@ int fnic_trace_debugfs_init(void) return rc; } fnic_trace_enable = debugfs_create_file("tracing_enable", - S_IFREG|S_IRUGO|S_IWUSR, - fnic_trace_debugfs_root, - NULL, &fnic_trace_ctrl_fops); + S_IFREG|S_IRUGO|S_IWUSR, + fnic_trace_debugfs_root, + &(fc_trc_flag->fnic_trace), + &fnic_trace_ctrl_fops); if (!fnic_trace_enable) { printk(KERN_DEBUG @@ -317,10 +393,10 @@ int fnic_trace_debugfs_init(void) } fnic_trace_debugfs_file = debugfs_create_file("trace", - S_IFREG|S_IRUGO|S_IWUSR, - fnic_trace_debugfs_root, - NULL, - &fnic_trace_debugfs_fops); + S_IFREG|S_IRUGO|S_IWUSR, + fnic_trace_debugfs_root, + &(fc_trc_flag->fnic_trace), + &fnic_trace_debugfs_fops); if (!fnic_trace_debugfs_file) { printk(KERN_DEBUG @@ -340,14 +416,104 @@ int fnic_trace_debugfs_init(void) */ void fnic_trace_debugfs_terminate(void) { - if (fnic_trace_debugfs_file) { - debugfs_remove(fnic_trace_debugfs_file); - fnic_trace_debugfs_file = NULL; + debugfs_remove(fnic_trace_debugfs_file); + fnic_trace_debugfs_file = NULL; + + debugfs_remove(fnic_trace_enable); + fnic_trace_enable = NULL; +} + +/* + * fnic_fc_trace_debugfs_init - + * Initialize debugfs for fnic control frame trace logging + * + * Description: + * When Debugfs is configured this routine sets up the fnic_fc debugfs + * file system. If not already created, this routine will create the + * create file trace to log fnic fc trace buffer output into debugfs and + * it will also create file fc_trace_enable to control enable/disable of + * trace logging into trace buffer. + */ + +int fnic_fc_trace_debugfs_init(void) +{ + int rc = -1; + + if (!fnic_trace_debugfs_root) { + pr_err("fnic:Debugfs root directory doesn't exist\n"); + return rc; + } + + fnic_fc_trace_enable = debugfs_create_file("fc_trace_enable", + S_IFREG|S_IRUGO|S_IWUSR, + fnic_trace_debugfs_root, + &(fc_trc_flag->fc_trace), + &fnic_trace_ctrl_fops); + + if (!fnic_fc_trace_enable) { + pr_err("fnic: Failed create fc_trace_enable file\n"); + return rc; + } + + fnic_fc_trace_clear = debugfs_create_file("fc_trace_clear", + S_IFREG|S_IRUGO|S_IWUSR, + fnic_trace_debugfs_root, + &(fc_trc_flag->fc_clear), + &fnic_trace_ctrl_fops); + + if (!fnic_fc_trace_clear) { + pr_err("fnic: Failed to create fc_trace_enable file\n"); + return rc; + } + + fnic_fc_rdata_trace_debugfs_file = + debugfs_create_file("fc_trace_rdata", + S_IFREG|S_IRUGO|S_IWUSR, + fnic_trace_debugfs_root, + &(fc_trc_flag->fc_normal_file), + &fnic_trace_debugfs_fops); + + if (!fnic_fc_rdata_trace_debugfs_file) { + pr_err("fnic: Failed create fc_rdata_trace file\n"); + return rc; } - if (fnic_trace_enable) { - debugfs_remove(fnic_trace_enable); - fnic_trace_enable = NULL; + + fnic_fc_trace_debugfs_file = + debugfs_create_file("fc_trace", + S_IFREG|S_IRUGO|S_IWUSR, + fnic_trace_debugfs_root, + &(fc_trc_flag->fc_row_file), + &fnic_trace_debugfs_fops); + + if (!fnic_fc_trace_debugfs_file) { + pr_err("fnic: Failed to create fc_trace file\n"); + return rc; } + rc = 0; + return rc; +} + +/* + * fnic_fc_trace_debugfs_terminate - Tear down debugfs infrastructure + * + * Description: + * When Debugfs is configured this routine removes debugfs file system + * elements that are specific to fnic_fc trace logging. + */ + +void fnic_fc_trace_debugfs_terminate(void) +{ + debugfs_remove(fnic_fc_trace_debugfs_file); + fnic_fc_trace_debugfs_file = NULL; + + debugfs_remove(fnic_fc_rdata_trace_debugfs_file); + fnic_fc_rdata_trace_debugfs_file = NULL; + + debugfs_remove(fnic_fc_trace_enable); + fnic_fc_trace_enable = NULL; + + debugfs_remove(fnic_fc_trace_clear); + fnic_fc_trace_clear = NULL; } /* diff --git a/drivers/scsi/fnic/fnic_fcs.c b/drivers/scsi/fnic/fnic_fcs.c index 1671325aec7f..1b948f633fc5 100644 --- a/drivers/scsi/fnic/fnic_fcs.c +++ b/drivers/scsi/fnic/fnic_fcs.c @@ -66,19 +66,35 @@ void fnic_handle_link(struct work_struct *work) fnic->link_down_cnt = vnic_dev_link_down_cnt(fnic->vdev); if (old_link_status == fnic->link_status) { - if (!fnic->link_status) + if (!fnic->link_status) { /* DOWN -> DOWN */ spin_unlock_irqrestore(&fnic->fnic_lock, flags); - else { + fnic_fc_trace_set_data(fnic->lport->host->host_no, + FNIC_FC_LE, "Link Status: DOWN->DOWN", + strlen("Link Status: DOWN->DOWN")); + } else { if (old_link_down_cnt != fnic->link_down_cnt) { /* UP -> DOWN -> UP */ fnic->lport->host_stats.link_failure_count++; spin_unlock_irqrestore(&fnic->fnic_lock, flags); + fnic_fc_trace_set_data( + fnic->lport->host->host_no, + FNIC_FC_LE, + "Link Status:UP_DOWN_UP", + strlen("Link_Status:UP_DOWN_UP") + ); FNIC_FCS_DBG(KERN_DEBUG, fnic->lport->host, "link down\n"); fcoe_ctlr_link_down(&fnic->ctlr); if (fnic->config.flags & VFCF_FIP_CAPABLE) { /* start FCoE VLAN discovery */ + fnic_fc_trace_set_data( + fnic->lport->host->host_no, + FNIC_FC_LE, + "Link Status: UP_DOWN_UP_VLAN", + strlen( + "Link Status: UP_DOWN_UP_VLAN") + ); fnic_fcoe_send_vlan_req(fnic); return; } @@ -88,22 +104,36 @@ void fnic_handle_link(struct work_struct *work) } else /* UP -> UP */ spin_unlock_irqrestore(&fnic->fnic_lock, flags); + fnic_fc_trace_set_data( + fnic->lport->host->host_no, FNIC_FC_LE, + "Link Status: UP_UP", + strlen("Link Status: UP_UP")); } } else if (fnic->link_status) { /* DOWN -> UP */ spin_unlock_irqrestore(&fnic->fnic_lock, flags); if (fnic->config.flags & VFCF_FIP_CAPABLE) { /* start FCoE VLAN discovery */ + fnic_fc_trace_set_data( + fnic->lport->host->host_no, + FNIC_FC_LE, "Link Status: DOWN_UP_VLAN", + strlen("Link Status: DOWN_UP_VLAN")); fnic_fcoe_send_vlan_req(fnic); return; } FNIC_FCS_DBG(KERN_DEBUG, fnic->lport->host, "link up\n"); + fnic_fc_trace_set_data(fnic->lport->host->host_no, FNIC_FC_LE, + "Link Status: DOWN_UP", strlen("Link Status: DOWN_UP")); fcoe_ctlr_link_up(&fnic->ctlr); } else { /* UP -> DOWN */ fnic->lport->host_stats.link_failure_count++; spin_unlock_irqrestore(&fnic->fnic_lock, flags); FNIC_FCS_DBG(KERN_DEBUG, fnic->lport->host, "link down\n"); + fnic_fc_trace_set_data( + fnic->lport->host->host_no, FNIC_FC_LE, + "Link Status: UP_DOWN", + strlen("Link Status: UP_DOWN")); fcoe_ctlr_link_down(&fnic->ctlr); } @@ -267,11 +297,6 @@ static inline int is_fnic_fip_flogi_reject(struct fcoe_ctlr *fip, if (desc->fip_dtype == FIP_DT_FLOGI) { - shost_printk(KERN_DEBUG, lport->host, - " FIP TYPE FLOGI: fab name:%llx " - "vfid:%d map:%x\n", - fip->sel_fcf->fabric_name, fip->sel_fcf->vfid, - fip->sel_fcf->fc_map); if (dlen < sizeof(*els) + sizeof(*fh) + 1) return 0; @@ -616,6 +641,10 @@ static inline int fnic_import_rq_eth_pkt(struct fnic *fnic, struct sk_buff *skb) "using UCSM\n"); goto drop; } + if ((fnic_fc_trace_set_data(fnic->lport->host->host_no, + FNIC_FC_RECV|0x80, (char *)skb->data, skb->len)) != 0) { + printk(KERN_ERR "fnic ctlr frame trace error!!!"); + } skb_queue_tail(&fnic->fip_frame_queue, skb); queue_work(fnic_fip_queue, &fnic->fip_frame_work); return 1; /* let caller know packet was used */ @@ -844,6 +873,10 @@ static void fnic_rq_cmpl_frame_recv(struct vnic_rq *rq, struct cq_desc } fr_dev(fp) = fnic->lport; spin_unlock_irqrestore(&fnic->fnic_lock, flags); + if ((fnic_fc_trace_set_data(fnic->lport->host->host_no, FNIC_FC_RECV, + (char *)skb->data, skb->len)) != 0) { + printk(KERN_ERR "fnic ctlr frame trace error!!!"); + } skb_queue_tail(&fnic->frame_queue, skb); queue_work(fnic_event_queue, &fnic->frame_work); @@ -951,6 +984,15 @@ void fnic_eth_send(struct fcoe_ctlr *fip, struct sk_buff *skb) vlan_hdr->h_vlan_proto = htons(ETH_P_8021Q); vlan_hdr->h_vlan_encapsulated_proto = eth_hdr->h_proto; vlan_hdr->h_vlan_TCI = htons(fnic->vlan_id); + if ((fnic_fc_trace_set_data(fnic->lport->host->host_no, + FNIC_FC_SEND|0x80, (char *)eth_hdr, skb->len)) != 0) { + printk(KERN_ERR "fnic ctlr frame trace error!!!"); + } + } else { + if ((fnic_fc_trace_set_data(fnic->lport->host->host_no, + FNIC_FC_SEND|0x80, (char *)skb->data, skb->len)) != 0) { + printk(KERN_ERR "fnic ctlr frame trace error!!!"); + } } pa = pci_map_single(fnic->pdev, skb->data, skb->len, PCI_DMA_TODEVICE); @@ -1023,6 +1065,11 @@ static int fnic_send_frame(struct fnic *fnic, struct fc_frame *fp) pa = pci_map_single(fnic->pdev, eth_hdr, tot_len, PCI_DMA_TODEVICE); + if ((fnic_fc_trace_set_data(fnic->lport->host->host_no, FNIC_FC_SEND, + (char *)eth_hdr, tot_len)) != 0) { + printk(KERN_ERR "fnic ctlr frame trace error!!!"); + } + spin_lock_irqsave(&fnic->wq_lock[0], flags); if (!vnic_wq_desc_avail(wq)) { diff --git a/drivers/scsi/fnic/fnic_main.c b/drivers/scsi/fnic/fnic_main.c index 33e4ec2bfe73..8c56fdc3a456 100644 --- a/drivers/scsi/fnic/fnic_main.c +++ b/drivers/scsi/fnic/fnic_main.c @@ -74,6 +74,11 @@ module_param(fnic_trace_max_pages, uint, S_IRUGO|S_IWUSR); MODULE_PARM_DESC(fnic_trace_max_pages, "Total allocated memory pages " "for fnic trace buffer"); +unsigned int fnic_fc_trace_max_pages = 64; +module_param(fnic_fc_trace_max_pages, uint, S_IRUGO|S_IWUSR); +MODULE_PARM_DESC(fnic_fc_trace_max_pages, + "Total allocated memory pages for fc trace buffer"); + static unsigned int fnic_max_qdepth = FNIC_DFLT_QUEUE_DEPTH; module_param(fnic_max_qdepth, uint, S_IRUGO|S_IWUSR); MODULE_PARM_DESC(fnic_max_qdepth, "Queue depth to report for each LUN"); @@ -111,7 +116,7 @@ static struct scsi_host_template fnic_host_template = { .change_queue_type = fc_change_queue_type, .this_id = -1, .cmd_per_lun = 3, - .can_queue = FNIC_MAX_IO_REQ, + .can_queue = FNIC_DFLT_IO_REQ, .use_clustering = ENABLE_CLUSTERING, .sg_tablesize = FNIC_MAX_SG_DESC_CNT, .max_sectors = 0xffff, @@ -773,6 +778,7 @@ static int fnic_probe(struct pci_dev *pdev, const struct pci_device_id *ent) shost_printk(KERN_INFO, fnic->lport->host, "firmware uses non-FIP mode\n"); fcoe_ctlr_init(&fnic->ctlr, FIP_MODE_NON_FIP); + fnic->ctlr.state = FIP_ST_NON_FIP; } fnic->state = FNIC_IN_FC_MODE; @@ -1033,11 +1039,20 @@ static int __init fnic_init_module(void) /* Allocate memory for trace buffer */ err = fnic_trace_buf_init(); if (err < 0) { - printk(KERN_ERR PFX "Trace buffer initialization Failed " - "Fnic Tracing utility is disabled\n"); + printk(KERN_ERR PFX + "Trace buffer initialization Failed. " + "Fnic Tracing utility is disabled\n"); fnic_trace_free(); } + /* Allocate memory for fc trace buffer */ + err = fnic_fc_trace_init(); + if (err < 0) { + printk(KERN_ERR PFX "FC trace buffer initialization Failed " + "FC frame tracing utility is disabled\n"); + fnic_fc_trace_free(); + } + /* Create a cache for allocation of default size sgls */ len = sizeof(struct fnic_dflt_sgl_list); fnic_sgl_cache[FNIC_SGL_CACHE_DFLT] = kmem_cache_create @@ -1118,6 +1133,7 @@ err_create_fnic_sgl_slab_max: kmem_cache_destroy(fnic_sgl_cache[FNIC_SGL_CACHE_DFLT]); err_create_fnic_sgl_slab_dflt: fnic_trace_free(); + fnic_fc_trace_free(); fnic_debugfs_terminate(); return err; } @@ -1135,6 +1151,7 @@ static void __exit fnic_cleanup_module(void) kmem_cache_destroy(fnic_io_req_cache); fc_release_transport(fnic_fc_transport); fnic_trace_free(); + fnic_fc_trace_free(); fnic_debugfs_terminate(); } diff --git a/drivers/scsi/fnic/fnic_scsi.c b/drivers/scsi/fnic/fnic_scsi.c index 0521436d05d6..ea28b5ca4c73 100644 --- a/drivers/scsi/fnic/fnic_scsi.c +++ b/drivers/scsi/fnic/fnic_scsi.c @@ -1312,8 +1312,9 @@ static void fnic_cleanup_io(struct fnic *fnic, int exclude_id) cleanup_scsi_cmd: sc->result = DID_TRANSPORT_DISRUPTED << 16; - FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, "fnic_cleanup_io:" - " DID_TRANSPORT_DISRUPTED\n"); + FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, + "%s: sc duration = %lu DID_TRANSPORT_DISRUPTED\n", + __func__, (jiffies - start_time)); if (atomic64_read(&fnic->io_cmpl_skip)) atomic64_dec(&fnic->io_cmpl_skip); @@ -1733,6 +1734,7 @@ int fnic_abort_cmd(struct scsi_cmnd *sc) struct fnic_stats *fnic_stats; struct abort_stats *abts_stats; struct terminate_stats *term_stats; + enum fnic_ioreq_state old_ioreq_state; int tag; DECLARE_COMPLETION_ONSTACK(tm_done); @@ -1793,6 +1795,7 @@ int fnic_abort_cmd(struct scsi_cmnd *sc) * the completion wont be done till mid-layer, since abort * has already started. */ + old_ioreq_state = CMD_STATE(sc); CMD_STATE(sc) = FNIC_IOREQ_ABTS_PENDING; CMD_ABTS_STATUS(sc) = FCPIO_INVALID_CODE; @@ -1816,6 +1819,8 @@ int fnic_abort_cmd(struct scsi_cmnd *sc) if (fnic_queue_abort_io_req(fnic, sc->request->tag, task_req, fc_lun.scsi_lun, io_req)) { spin_lock_irqsave(io_lock, flags); + if (CMD_STATE(sc) == FNIC_IOREQ_ABTS_PENDING) + CMD_STATE(sc) = old_ioreq_state; io_req = (struct fnic_io_req *)CMD_SP(sc); if (io_req) io_req->abts_done = NULL; @@ -1859,12 +1864,8 @@ int fnic_abort_cmd(struct scsi_cmnd *sc) if (CMD_ABTS_STATUS(sc) == FCPIO_INVALID_CODE) { spin_unlock_irqrestore(io_lock, flags); if (task_req == FCPIO_ITMF_ABT_TASK) { - FNIC_SCSI_DBG(KERN_INFO, - fnic->lport->host, "Abort Driver Timeout\n"); atomic64_inc(&abts_stats->abort_drv_timeouts); } else { - FNIC_SCSI_DBG(KERN_INFO, fnic->lport->host, - "Terminate Driver Timeout\n"); atomic64_inc(&term_stats->terminate_drv_timeouts); } CMD_FLAGS(sc) |= FNIC_IO_ABT_TERM_TIMED_OUT; diff --git a/drivers/scsi/fnic/fnic_trace.c b/drivers/scsi/fnic/fnic_trace.c index e002e7187dc0..c77285926827 100644 --- a/drivers/scsi/fnic/fnic_trace.c +++ b/drivers/scsi/fnic/fnic_trace.c @@ -20,6 +20,7 @@ #include <linux/errno.h> #include <linux/spinlock.h> #include <linux/kallsyms.h> +#include <linux/time.h> #include "fnic_io.h" #include "fnic.h" @@ -32,6 +33,16 @@ static DEFINE_SPINLOCK(fnic_trace_lock); static fnic_trace_dbg_t fnic_trace_entries; int fnic_tracing_enabled = 1; +/* static char *fnic_fc_ctlr_trace_buf_p; */ + +static int fc_trace_max_entries; +static unsigned long fnic_fc_ctlr_trace_buf_p; +static fnic_trace_dbg_t fc_trace_entries; +int fnic_fc_tracing_enabled = 1; +int fnic_fc_trace_cleared = 1; +static DEFINE_SPINLOCK(fnic_fc_trace_lock); + + /* * fnic_trace_get_buf - Give buffer pointer to user to fill up trace information * @@ -428,10 +439,10 @@ int fnic_trace_buf_init(void) } err = fnic_trace_debugfs_init(); if (err < 0) { - printk(KERN_ERR PFX "Failed to initialize debugfs for tracing\n"); + pr_err("fnic: Failed to initialize debugfs for tracing\n"); goto err_fnic_trace_debugfs_init; } - printk(KERN_INFO PFX "Successfully Initialized Trace Buffer\n"); + pr_info("fnic: Successfully Initialized Trace Buffer\n"); return err; err_fnic_trace_debugfs_init: fnic_trace_free(); @@ -456,3 +467,314 @@ void fnic_trace_free(void) } printk(KERN_INFO PFX "Successfully Freed Trace Buffer\n"); } + +/* + * fnic_fc_ctlr_trace_buf_init - + * Initialize trace buffer to log fnic control frames + * Description: + * Initialize trace buffer data structure by allocating + * required memory for trace data as well as for Indexes. + * Frame size is 256 bytes and + * memory is allocated for 1024 entries of 256 bytes. + * Page_offset(Index) is set to the address of trace entry + * and page_offset is initialized by adding frame size + * to the previous page_offset entry. + */ + +int fnic_fc_trace_init(void) +{ + unsigned long fc_trace_buf_head; + int err = 0; + int i; + + fc_trace_max_entries = (fnic_fc_trace_max_pages * PAGE_SIZE)/ + FC_TRC_SIZE_BYTES; + fnic_fc_ctlr_trace_buf_p = (unsigned long)vmalloc( + fnic_fc_trace_max_pages * PAGE_SIZE); + if (!fnic_fc_ctlr_trace_buf_p) { + pr_err("fnic: Failed to allocate memory for " + "FC Control Trace Buf\n"); + err = -ENOMEM; + goto err_fnic_fc_ctlr_trace_buf_init; + } + + memset((void *)fnic_fc_ctlr_trace_buf_p, 0, + fnic_fc_trace_max_pages * PAGE_SIZE); + + /* Allocate memory for page offset */ + fc_trace_entries.page_offset = vmalloc(fc_trace_max_entries * + sizeof(unsigned long)); + if (!fc_trace_entries.page_offset) { + pr_err("fnic:Failed to allocate memory for page_offset\n"); + if (fnic_fc_ctlr_trace_buf_p) { + pr_err("fnic: Freeing FC Control Trace Buf\n"); + vfree((void *)fnic_fc_ctlr_trace_buf_p); + fnic_fc_ctlr_trace_buf_p = 0; + } + err = -ENOMEM; + goto err_fnic_fc_ctlr_trace_buf_init; + } + memset((void *)fc_trace_entries.page_offset, 0, + (fc_trace_max_entries * sizeof(unsigned long))); + + fc_trace_entries.rd_idx = fc_trace_entries.wr_idx = 0; + fc_trace_buf_head = fnic_fc_ctlr_trace_buf_p; + + /* + * Set up fc_trace_entries.page_offset field with memory location + * for every trace entry + */ + for (i = 0; i < fc_trace_max_entries; i++) { + fc_trace_entries.page_offset[i] = fc_trace_buf_head; + fc_trace_buf_head += FC_TRC_SIZE_BYTES; + } + err = fnic_fc_trace_debugfs_init(); + if (err < 0) { + pr_err("fnic: Failed to initialize FC_CTLR tracing.\n"); + goto err_fnic_fc_ctlr_trace_debugfs_init; + } + pr_info("fnic: Successfully Initialized FC_CTLR Trace Buffer\n"); + return err; + +err_fnic_fc_ctlr_trace_debugfs_init: + fnic_fc_trace_free(); +err_fnic_fc_ctlr_trace_buf_init: + return err; +} + +/* + * Fnic_fc_ctlr_trace_free - Free memory of fnic_fc_ctlr trace data structures. + */ +void fnic_fc_trace_free(void) +{ + fnic_fc_tracing_enabled = 0; + fnic_fc_trace_debugfs_terminate(); + if (fc_trace_entries.page_offset) { + vfree((void *)fc_trace_entries.page_offset); + fc_trace_entries.page_offset = NULL; + } + if (fnic_fc_ctlr_trace_buf_p) { + vfree((void *)fnic_fc_ctlr_trace_buf_p); + fnic_fc_ctlr_trace_buf_p = 0; + } + pr_info("fnic:Successfully FC_CTLR Freed Trace Buffer\n"); +} + +/* + * fnic_fc_ctlr_set_trace_data: + * Maintain rd & wr idx accordingly and set data + * Passed parameters: + * host_no: host number accociated with fnic + * frame_type: send_frame, rece_frame or link event + * fc_frame: pointer to fc_frame + * frame_len: Length of the fc_frame + * Description: + * This routine will get next available wr_idx and + * copy all passed trace data to the buffer pointed by wr_idx + * and increment wr_idx. It will also make sure that we dont + * overwrite the entry which we are reading and also + * wrap around if we reach the maximum entries. + * Returned Value: + * It will return 0 for success or -1 for failure + */ +int fnic_fc_trace_set_data(u32 host_no, u8 frame_type, + char *frame, u32 fc_trc_frame_len) +{ + unsigned long flags; + struct fc_trace_hdr *fc_buf; + unsigned long eth_fcoe_hdr_len; + char *fc_trace; + + if (fnic_fc_tracing_enabled == 0) + return 0; + + spin_lock_irqsave(&fnic_fc_trace_lock, flags); + + if (fnic_fc_trace_cleared == 1) { + fc_trace_entries.rd_idx = fc_trace_entries.wr_idx = 0; + pr_info("fnic: Reseting the read idx\n"); + memset((void *)fnic_fc_ctlr_trace_buf_p, 0, + fnic_fc_trace_max_pages * PAGE_SIZE); + fnic_fc_trace_cleared = 0; + } + + fc_buf = (struct fc_trace_hdr *) + fc_trace_entries.page_offset[fc_trace_entries.wr_idx]; + + fc_trace_entries.wr_idx++; + + if (fc_trace_entries.wr_idx >= fc_trace_max_entries) + fc_trace_entries.wr_idx = 0; + + if (fc_trace_entries.wr_idx == fc_trace_entries.rd_idx) { + fc_trace_entries.rd_idx++; + if (fc_trace_entries.rd_idx >= fc_trace_max_entries) + fc_trace_entries.rd_idx = 0; + } + + fc_buf->time_stamp = CURRENT_TIME; + fc_buf->host_no = host_no; + fc_buf->frame_type = frame_type; + + fc_trace = (char *)FC_TRACE_ADDRESS(fc_buf); + + /* During the receive path, we do not have eth hdr as well as fcoe hdr + * at trace entry point so we will stuff 0xff just to make it generic. + */ + if (frame_type == FNIC_FC_RECV) { + eth_fcoe_hdr_len = sizeof(struct ethhdr) + + sizeof(struct fcoe_hdr); + fc_trc_frame_len = fc_trc_frame_len + eth_fcoe_hdr_len; + memset((char *)fc_trace, 0xff, eth_fcoe_hdr_len); + /* Copy the rest of data frame */ + memcpy((char *)(fc_trace + eth_fcoe_hdr_len), (void *)frame, + min_t(u8, fc_trc_frame_len, + (u8)(FC_TRC_SIZE_BYTES - FC_TRC_HEADER_SIZE))); + } else { + memcpy((char *)fc_trace, (void *)frame, + min_t(u8, fc_trc_frame_len, + (u8)(FC_TRC_SIZE_BYTES - FC_TRC_HEADER_SIZE))); + } + + /* Store the actual received length */ + fc_buf->frame_len = fc_trc_frame_len; + + spin_unlock_irqrestore(&fnic_fc_trace_lock, flags); + return 0; +} + +/* + * fnic_fc_ctlr_get_trace_data: Copy trace buffer to a memory file + * Passed parameter: + * @fnic_dbgfs_t: pointer to debugfs trace buffer + * rdata_flag: 1 => Unformated file + * 0 => formated file + * Description: + * This routine will copy the trace data to memory file with + * proper formatting and also copy to another memory + * file without formatting for further procesing. + * Retrun Value: + * Number of bytes that were dumped into fnic_dbgfs_t + */ + +int fnic_fc_trace_get_data(fnic_dbgfs_t *fnic_dbgfs_prt, u8 rdata_flag) +{ + int rd_idx, wr_idx; + unsigned long flags; + int len = 0, j; + struct fc_trace_hdr *tdata; + char *fc_trace; + + spin_lock_irqsave(&fnic_fc_trace_lock, flags); + if (fc_trace_entries.wr_idx == fc_trace_entries.rd_idx) { + spin_unlock_irqrestore(&fnic_fc_trace_lock, flags); + pr_info("fnic: Buffer is empty\n"); + return 0; + } + rd_idx = fc_trace_entries.rd_idx; + wr_idx = fc_trace_entries.wr_idx; + if (rdata_flag == 0) { + len += snprintf(fnic_dbgfs_prt->buffer + len, + (fnic_fc_trace_max_pages * PAGE_SIZE * 3) - len, + "Time Stamp (UTC)\t\t" + "Host No: F Type: len: FCoE_FRAME:\n"); + } + + while (rd_idx != wr_idx) { + tdata = (struct fc_trace_hdr *) + fc_trace_entries.page_offset[rd_idx]; + if (!tdata) { + pr_info("fnic: Rd data is NULL\n"); + spin_unlock_irqrestore(&fnic_fc_trace_lock, flags); + return 0; + } + if (rdata_flag == 0) { + copy_and_format_trace_data(tdata, + fnic_dbgfs_prt, &len, rdata_flag); + } else { + fc_trace = (char *)tdata; + for (j = 0; j < FC_TRC_SIZE_BYTES; j++) { + len += snprintf(fnic_dbgfs_prt->buffer + len, + (fnic_fc_trace_max_pages * PAGE_SIZE * 3) + - len, "%02x", fc_trace[j] & 0xff); + } /* for loop */ + len += snprintf(fnic_dbgfs_prt->buffer + len, + (fnic_fc_trace_max_pages * PAGE_SIZE * 3) - len, + "\n"); + } + rd_idx++; + if (rd_idx > (fc_trace_max_entries - 1)) + rd_idx = 0; + } + + spin_unlock_irqrestore(&fnic_fc_trace_lock, flags); + return len; +} + +/* + * copy_and_format_trace_data: Copy formatted data to char * buffer + * Passed Parameter: + * @fc_trace_hdr_t: pointer to trace data + * @fnic_dbgfs_t: pointer to debugfs trace buffer + * @orig_len: pointer to len + * rdata_flag: 0 => Formated file, 1 => Unformated file + * Description: + * This routine will format and copy the passed trace data + * for formated file or unformated file accordingly. + */ + +void copy_and_format_trace_data(struct fc_trace_hdr *tdata, + fnic_dbgfs_t *fnic_dbgfs_prt, int *orig_len, + u8 rdata_flag) +{ + struct tm tm; + int j, i = 1, len; + char *fc_trace, *fmt; + int ethhdr_len = sizeof(struct ethhdr) - 1; + int fcoehdr_len = sizeof(struct fcoe_hdr); + int fchdr_len = sizeof(struct fc_frame_header); + int max_size = fnic_fc_trace_max_pages * PAGE_SIZE * 3; + + tdata->frame_type = tdata->frame_type & 0x7F; + + len = *orig_len; + + time_to_tm(tdata->time_stamp.tv_sec, 0, &tm); + + fmt = "%02d:%02d:%04ld %02d:%02d:%02d.%09lu ns%8x %c%8x\t"; + len += snprintf(fnic_dbgfs_prt->buffer + len, + (fnic_fc_trace_max_pages * PAGE_SIZE * 3) - len, + fmt, + tm.tm_mon + 1, tm.tm_mday, tm.tm_year + 1900, + tm.tm_hour, tm.tm_min, tm.tm_sec, + tdata->time_stamp.tv_nsec, tdata->host_no, + tdata->frame_type, tdata->frame_len); + + fc_trace = (char *)FC_TRACE_ADDRESS(tdata); + + for (j = 0; j < min_t(u8, tdata->frame_len, + (u8)(FC_TRC_SIZE_BYTES - FC_TRC_HEADER_SIZE)); j++) { + if (tdata->frame_type == FNIC_FC_LE) { + len += snprintf(fnic_dbgfs_prt->buffer + len, + max_size - len, "%c", fc_trace[j]); + } else { + len += snprintf(fnic_dbgfs_prt->buffer + len, + max_size - len, "%02x", fc_trace[j] & 0xff); + len += snprintf(fnic_dbgfs_prt->buffer + len, + max_size - len, " "); + if (j == ethhdr_len || + j == ethhdr_len + fcoehdr_len || + j == ethhdr_len + fcoehdr_len + fchdr_len || + (i > 3 && j%fchdr_len == 0)) { + len += snprintf(fnic_dbgfs_prt->buffer + + len, (fnic_fc_trace_max_pages + * PAGE_SIZE * 3) - len, + "\n\t\t\t\t\t\t\t\t"); + i++; + } + } /* end of else*/ + } /* End of for loop*/ + len += snprintf(fnic_dbgfs_prt->buffer + len, + max_size - len, "\n"); + *orig_len = len; +} diff --git a/drivers/scsi/fnic/fnic_trace.h b/drivers/scsi/fnic/fnic_trace.h index d412f2ee3c4f..a8aa0578fcb0 100644 --- a/drivers/scsi/fnic/fnic_trace.h +++ b/drivers/scsi/fnic/fnic_trace.h @@ -19,6 +19,17 @@ #define __FNIC_TRACE_H__ #define FNIC_ENTRY_SIZE_BYTES 64 +#define FC_TRC_SIZE_BYTES 256 +#define FC_TRC_HEADER_SIZE sizeof(struct fc_trace_hdr) + +/* + * Fisrt bit of FNIC_FC_RECV and FNIC_FC_SEND is used to represent the type + * of frame 1 => Eth frame, 0=> FC frame + */ + +#define FNIC_FC_RECV 0x52 /* Character R */ +#define FNIC_FC_SEND 0x54 /* Character T */ +#define FNIC_FC_LE 0x4C /* Character L */ extern ssize_t simple_read_from_buffer(void __user *to, size_t count, @@ -30,6 +41,10 @@ extern unsigned int fnic_trace_max_pages; extern int fnic_tracing_enabled; extern unsigned int trace_max_pages; +extern unsigned int fnic_fc_trace_max_pages; +extern int fnic_fc_tracing_enabled; +extern int fnic_fc_trace_cleared; + typedef struct fnic_trace_dbg { int wr_idx; int rd_idx; @@ -56,6 +71,16 @@ struct fnic_trace_data { typedef struct fnic_trace_data fnic_trace_data_t; +struct fc_trace_hdr { + struct timespec time_stamp; + u32 host_no; + u8 frame_type; + u8 frame_len; +} __attribute__((__packed__)); + +#define FC_TRACE_ADDRESS(a) \ + ((unsigned long)(a) + sizeof(struct fc_trace_hdr)) + #define FNIC_TRACE_ENTRY_SIZE \ (FNIC_ENTRY_SIZE_BYTES - sizeof(fnic_trace_data_t)) @@ -88,4 +113,17 @@ int fnic_debugfs_init(void); void fnic_debugfs_terminate(void); int fnic_trace_debugfs_init(void); void fnic_trace_debugfs_terminate(void); + +/* Fnic FC CTLR Trace releated function */ +int fnic_fc_trace_init(void); +void fnic_fc_trace_free(void); +int fnic_fc_trace_set_data(u32 host_no, u8 frame_type, + char *frame, u32 fc_frame_len); +int fnic_fc_trace_get_data(fnic_dbgfs_t *fnic_dbgfs_prt, u8 rdata_flag); +void copy_and_format_trace_data(struct fc_trace_hdr *tdata, + fnic_dbgfs_t *fnic_dbgfs_prt, + int *len, u8 rdata_flag); +int fnic_fc_trace_debugfs_init(void); +void fnic_fc_trace_debugfs_terminate(void); + #endif diff --git a/drivers/scsi/g_NCR5380.c b/drivers/scsi/g_NCR5380.c index 7176365e916b..a1bc8ca958e1 100644 --- a/drivers/scsi/g_NCR5380.c +++ b/drivers/scsi/g_NCR5380.c @@ -78,10 +78,6 @@ * */ -/* - * $Log: generic_NCR5380.c,v $ - */ - /* settings for DTC3181E card with only Mustek scanner attached */ #define USLEEP #define USLEEP_POLL 1 diff --git a/drivers/scsi/g_NCR5380.h b/drivers/scsi/g_NCR5380.h index 1bcdb7beb77b..703adf78e0b2 100644 --- a/drivers/scsi/g_NCR5380.h +++ b/drivers/scsi/g_NCR5380.h @@ -25,10 +25,6 @@ * 1+ (800) 334-5454 */ -/* - * $Log: generic_NCR5380.h,v $ - */ - #ifndef GENERIC_NCR5380_H #define GENERIC_NCR5380_H @@ -58,8 +54,6 @@ static const char* generic_NCR5380_info(struct Scsi_Host *); #define CAN_QUEUE 16 #endif -#ifndef HOSTS_C - #define __STRVAL(x) #x #define STRVAL(x) __STRVAL(x) @@ -131,7 +125,6 @@ static const char* generic_NCR5380_info(struct Scsi_Host *); #define BOARD_NCR53C400A 2 #define BOARD_DTC3181E 3 -#endif /* else def HOSTS_C */ #endif /* ndef ASM */ #endif /* GENERIC_NCR5380_H */ diff --git a/drivers/scsi/hpsa.c b/drivers/scsi/hpsa.c index 9a6e4a2cd072..5858600bfe59 100644 --- a/drivers/scsi/hpsa.c +++ b/drivers/scsi/hpsa.c @@ -115,9 +115,15 @@ static const struct pci_device_id hpsa_pci_device_id[] = { {PCI_VENDOR_ID_HP, PCI_DEVICE_ID_HP_CISSI, 0x103C, 0x21C3}, {PCI_VENDOR_ID_HP, PCI_DEVICE_ID_HP_CISSI, 0x103C, 0x21C4}, {PCI_VENDOR_ID_HP, PCI_DEVICE_ID_HP_CISSI, 0x103C, 0x21C5}, + {PCI_VENDOR_ID_HP, PCI_DEVICE_ID_HP_CISSI, 0x103C, 0x21C6}, {PCI_VENDOR_ID_HP, PCI_DEVICE_ID_HP_CISSI, 0x103C, 0x21C7}, {PCI_VENDOR_ID_HP, PCI_DEVICE_ID_HP_CISSI, 0x103C, 0x21C8}, {PCI_VENDOR_ID_HP, PCI_DEVICE_ID_HP_CISSI, 0x103C, 0x21C9}, + {PCI_VENDOR_ID_HP, PCI_DEVICE_ID_HP_CISSI, 0x103C, 0x21CA}, + {PCI_VENDOR_ID_HP, PCI_DEVICE_ID_HP_CISSI, 0x103C, 0x21CB}, + {PCI_VENDOR_ID_HP, PCI_DEVICE_ID_HP_CISSI, 0x103C, 0x21CC}, + {PCI_VENDOR_ID_HP, PCI_DEVICE_ID_HP_CISSI, 0x103C, 0x21CD}, + {PCI_VENDOR_ID_HP, PCI_DEVICE_ID_HP_CISSI, 0x103C, 0x21CE}, {PCI_VENDOR_ID_HP_3PAR, 0x0075, 0x1590, 0x0076}, {PCI_VENDOR_ID_HP_3PAR, 0x0075, 0x1590, 0x0087}, {PCI_VENDOR_ID_HP_3PAR, 0x0075, 0x1590, 0x007D}, @@ -165,9 +171,15 @@ static struct board_type products[] = { {0x21C3103C, "Smart Array", &SA5_access}, {0x21C4103C, "Smart Array", &SA5_access}, {0x21C5103C, "Smart Array", &SA5_access}, + {0x21C6103C, "Smart Array", &SA5_access}, {0x21C7103C, "Smart Array", &SA5_access}, {0x21C8103C, "Smart Array", &SA5_access}, {0x21C9103C, "Smart Array", &SA5_access}, + {0x21CA103C, "Smart Array", &SA5_access}, + {0x21CB103C, "Smart Array", &SA5_access}, + {0x21CC103C, "Smart Array", &SA5_access}, + {0x21CD103C, "Smart Array", &SA5_access}, + {0x21CE103C, "Smart Array", &SA5_access}, {0x00761590, "HP Storage P1224 Array Controller", &SA5_access}, {0x00871590, "HP Storage P1224e Array Controller", &SA5_access}, {0x007D1590, "HP Storage P1228 Array Controller", &SA5_access}, @@ -2836,6 +2848,8 @@ static int hpsa_get_pdisk_of_ioaccel2(struct ctlr_info *h, /* Get the list of physical devices */ physicals = kzalloc(reportsize, GFP_KERNEL); + if (physicals == NULL) + return 0; if (hpsa_scsi_do_report_phys_luns(h, (struct ReportLUNdata *) physicals, reportsize, extended)) { dev_err(&h->pdev->dev, @@ -2963,19 +2977,24 @@ u8 *figure_lunaddrbytes(struct ctlr_info *h, int raid_ctlr_position, int i, static int hpsa_hba_mode_enabled(struct ctlr_info *h) { int rc; + int hba_mode_enabled; struct bmic_controller_parameters *ctlr_params; ctlr_params = kzalloc(sizeof(struct bmic_controller_parameters), GFP_KERNEL); if (!ctlr_params) - return 0; + return -ENOMEM; rc = hpsa_bmic_ctrl_mode_sense(h, RAID_CTLR_LUNID, 0, ctlr_params, sizeof(struct bmic_controller_parameters)); - if (rc != 0) { + if (rc) { kfree(ctlr_params); - return 0; + return rc; } - return ctlr_params->nvram_flags & (1 << 3) ? 1 : 0; + + hba_mode_enabled = + ((ctlr_params->nvram_flags & HBA_MODE_ENABLED_FLAG) != 0); + kfree(ctlr_params); + return hba_mode_enabled; } static void hpsa_update_scsi_devices(struct ctlr_info *h, int hostno) @@ -3001,7 +3020,7 @@ static void hpsa_update_scsi_devices(struct ctlr_info *h, int hostno) int reportlunsize = sizeof(*physdev_list) + HPSA_MAX_PHYS_LUN * 24; int i, n_ext_target_devs, ndevs_to_allocate; int raid_ctlr_position; - u8 rescan_hba_mode; + int rescan_hba_mode; DECLARE_BITMAP(lunzerobits, MAX_EXT_TARGETS); currentsd = kzalloc(sizeof(*currentsd) * HPSA_MAX_DEVICES, GFP_KERNEL); @@ -3016,6 +3035,8 @@ static void hpsa_update_scsi_devices(struct ctlr_info *h, int hostno) memset(lunzerobits, 0, sizeof(lunzerobits)); rescan_hba_mode = hpsa_hba_mode_enabled(h); + if (rescan_hba_mode < 0) + goto out; if (!h->hba_mode_enabled && rescan_hba_mode) dev_warn(&h->pdev->dev, "HBA mode enabled\n"); diff --git a/drivers/scsi/hpsa.h b/drivers/scsi/hpsa.h index 44235a27e1b6..1e3cf33a82cf 100644 --- a/drivers/scsi/hpsa.h +++ b/drivers/scsi/hpsa.h @@ -90,6 +90,7 @@ struct bmic_controller_parameters { u8 automatic_drive_slamming; u8 reserved1; u8 nvram_flags; +#define HBA_MODE_ENABLED_FLAG (1 << 3) u8 cache_nvram_flags; u8 drive_config_flags; u16 reserved2; diff --git a/drivers/scsi/libiscsi.c b/drivers/scsi/libiscsi.c index 26dc005bb0f0..ecd7bd304efe 100644 --- a/drivers/scsi/libiscsi.c +++ b/drivers/scsi/libiscsi.c @@ -1442,9 +1442,9 @@ static int iscsi_xmit_task(struct iscsi_conn *conn) conn->task = NULL; } /* regular RX path uses back_lock */ - spin_lock_bh(&conn->session->back_lock); + spin_lock(&conn->session->back_lock); __iscsi_put_task(task); - spin_unlock_bh(&conn->session->back_lock); + spin_unlock(&conn->session->back_lock); return rc; } diff --git a/drivers/scsi/lpfc/lpfc_sli.c b/drivers/scsi/lpfc/lpfc_sli.c index 6bb51f8e3c1b..393662c24df5 100644 --- a/drivers/scsi/lpfc/lpfc_sli.c +++ b/drivers/scsi/lpfc/lpfc_sli.c @@ -265,6 +265,16 @@ lpfc_sli4_eq_get(struct lpfc_queue *q) return NULL; q->hba_index = idx; + + /* + * insert barrier for instruction interlock : data from the hardware + * must have the valid bit checked before it can be copied and acted + * upon. Given what was seen in lpfc_sli4_cq_get() of speculative + * instructions allowing action on content before valid bit checked, + * add barrier here as well. May not be needed as "content" is a + * single 32-bit entity here (vs multi word structure for cq's). + */ + mb(); return eqe; } @@ -370,6 +380,17 @@ lpfc_sli4_cq_get(struct lpfc_queue *q) cqe = q->qe[q->hba_index].cqe; q->hba_index = idx; + + /* + * insert barrier for instruction interlock : data from the hardware + * must have the valid bit checked before it can be copied and acted + * upon. Speculative instructions were allowing a bcopy at the start + * of lpfc_sli4_fp_handle_wcqe(), which is called immediately + * after our return, to copy data before the valid bit check above + * was done. As such, some of the copied data was stale. The barrier + * ensures the check is before any data is copied. + */ + mb(); return cqe; } diff --git a/drivers/scsi/mac_scsi.c b/drivers/scsi/mac_scsi.c index f5cdc68cd5b6..6a039eb1cbce 100644 --- a/drivers/scsi/mac_scsi.c +++ b/drivers/scsi/mac_scsi.c @@ -25,10 +25,6 @@ * 1+ (800) 334-5454 */ -/* - * $Log: mac_NCR5380.c,v $ - */ - #include <linux/types.h> #include <linux/stddef.h> #include <linux/ctype.h> @@ -58,12 +54,6 @@ #include "NCR5380.h" -#if 0 -#define NDEBUG (NDEBUG_INTR | NDEBUG_PSEUDO_DMA | NDEBUG_ARBITRATION | NDEBUG_SELECTION | NDEBUG_RESELECTION) -#else -#define NDEBUG (NDEBUG_ABORT) -#endif - #define RESET_BOOT #define DRIVER_SETUP diff --git a/drivers/scsi/mac_scsi.h b/drivers/scsi/mac_scsi.h index 7dc62fce1c4c..06969b06e54b 100644 --- a/drivers/scsi/mac_scsi.h +++ b/drivers/scsi/mac_scsi.h @@ -22,10 +22,6 @@ * 1+ (800) 334-5454 */ -/* - * $Log: cumana_NCR5380.h,v $ - */ - #ifndef MAC_NCR5380_H #define MAC_NCR5380_H @@ -51,8 +47,6 @@ #include <scsi/scsicam.h> -#ifndef HOSTS_C - #define NCR5380_implementation_fields \ int port, ctrl @@ -75,10 +69,6 @@ #define NCR5380_show_info macscsi_show_info #define NCR5380_write_info macscsi_write_info -#define BOARD_NORMAL 0 -#define BOARD_NCR53C400 1 - -#endif /* ndef HOSTS_C */ #endif /* ndef ASM */ #endif /* MAC_NCR5380_H */ diff --git a/drivers/scsi/megaraid/megaraid_sas_base.c b/drivers/scsi/megaraid/megaraid_sas_base.c index d84d02c2aad9..112799b131a9 100644 --- a/drivers/scsi/megaraid/megaraid_sas_base.c +++ b/drivers/scsi/megaraid/megaraid_sas_base.c @@ -3061,7 +3061,8 @@ megasas_transition_to_ready(struct megasas_instance *instance, int ocr) u32 cur_state; u32 abs_state, curr_abs_state; - fw_state = instance->instancet->read_fw_status_reg(instance->reg_set) & MFI_STATE_MASK; + abs_state = instance->instancet->read_fw_status_reg(instance->reg_set); + fw_state = abs_state & MFI_STATE_MASK; if (fw_state != MFI_STATE_READY) printk(KERN_INFO "megasas: Waiting for FW to come to ready" @@ -3069,9 +3070,6 @@ megasas_transition_to_ready(struct megasas_instance *instance, int ocr) while (fw_state != MFI_STATE_READY) { - abs_state = - instance->instancet->read_fw_status_reg(instance->reg_set); - switch (fw_state) { case MFI_STATE_FAULT: @@ -3223,10 +3221,8 @@ megasas_transition_to_ready(struct megasas_instance *instance, int ocr) * The cur_state should not last for more than max_wait secs */ for (i = 0; i < (max_wait * 1000); i++) { - fw_state = instance->instancet->read_fw_status_reg(instance->reg_set) & - MFI_STATE_MASK ; - curr_abs_state = - instance->instancet->read_fw_status_reg(instance->reg_set); + curr_abs_state = instance->instancet-> + read_fw_status_reg(instance->reg_set); if (abs_state == curr_abs_state) { msleep(1); @@ -3242,6 +3238,9 @@ megasas_transition_to_ready(struct megasas_instance *instance, int ocr) "in %d secs\n", fw_state, max_wait); return -ENODEV; } + + abs_state = curr_abs_state; + fw_state = curr_abs_state & MFI_STATE_MASK; } printk(KERN_INFO "megasas: FW now in Ready state\n"); diff --git a/drivers/scsi/mpt2sas/mpt2sas_base.c b/drivers/scsi/mpt2sas/mpt2sas_base.c index bde63f7452bd..8b88118e20e6 100644 --- a/drivers/scsi/mpt2sas/mpt2sas_base.c +++ b/drivers/scsi/mpt2sas/mpt2sas_base.c @@ -1739,14 +1739,14 @@ mpt2sas_base_free_smid(struct MPT2SAS_ADAPTER *ioc, u16 smid) list_for_each_entry_safe(chain_req, next, &ioc->scsi_lookup[i].chain_list, tracker_list) { list_del_init(&chain_req->tracker_list); - list_add_tail(&chain_req->tracker_list, + list_add(&chain_req->tracker_list, &ioc->free_chain_list); } } ioc->scsi_lookup[i].cb_idx = 0xFF; ioc->scsi_lookup[i].scmd = NULL; ioc->scsi_lookup[i].direct_io = 0; - list_add_tail(&ioc->scsi_lookup[i].tracker_list, + list_add(&ioc->scsi_lookup[i].tracker_list, &ioc->free_list); spin_unlock_irqrestore(&ioc->scsi_lookup_lock, flags); @@ -1764,13 +1764,13 @@ mpt2sas_base_free_smid(struct MPT2SAS_ADAPTER *ioc, u16 smid) /* hi-priority */ i = smid - ioc->hi_priority_smid; ioc->hpr_lookup[i].cb_idx = 0xFF; - list_add_tail(&ioc->hpr_lookup[i].tracker_list, + list_add(&ioc->hpr_lookup[i].tracker_list, &ioc->hpr_free_list); } else if (smid <= ioc->hba_queue_depth) { /* internal queue */ i = smid - ioc->internal_smid; ioc->internal_lookup[i].cb_idx = 0xFF; - list_add_tail(&ioc->internal_lookup[i].tracker_list, + list_add(&ioc->internal_lookup[i].tracker_list, &ioc->internal_free_list); } spin_unlock_irqrestore(&ioc->scsi_lookup_lock, flags); diff --git a/drivers/scsi/mpt2sas/mpt2sas_base.h b/drivers/scsi/mpt2sas/mpt2sas_base.h index 1f2ac3a28621..fd3b998c75b1 100644 --- a/drivers/scsi/mpt2sas/mpt2sas_base.h +++ b/drivers/scsi/mpt2sas/mpt2sas_base.h @@ -1065,7 +1065,7 @@ void mpt2sas_scsih_event_callback(struct MPT2SAS_ADAPTER *ioc, u8 msix_index, u32 reply); int mpt2sas_scsih_issue_tm(struct MPT2SAS_ADAPTER *ioc, u16 handle, uint channel, uint id, uint lun, u8 type, u16 smid_task, - ulong timeout, unsigned long serial_number, enum mutex_type m_type); + ulong timeout, enum mutex_type m_type); void mpt2sas_scsih_set_tm_flag(struct MPT2SAS_ADAPTER *ioc, u16 handle); void mpt2sas_scsih_clear_tm_flag(struct MPT2SAS_ADAPTER *ioc, u16 handle); void mpt2sas_expander_remove(struct MPT2SAS_ADAPTER *ioc, u64 sas_address); diff --git a/drivers/scsi/mpt2sas/mpt2sas_ctl.c b/drivers/scsi/mpt2sas/mpt2sas_ctl.c index b7f887c9b0bf..62df8f9d4271 100644 --- a/drivers/scsi/mpt2sas/mpt2sas_ctl.c +++ b/drivers/scsi/mpt2sas/mpt2sas_ctl.c @@ -987,7 +987,7 @@ _ctl_do_mpt_command(struct MPT2SAS_ADAPTER *ioc, struct mpt2_ioctl_command karg, mpt2sas_scsih_issue_tm(ioc, le16_to_cpu(mpi_request->FunctionDependent1), 0, 0, 0, MPI2_SCSITASKMGMT_TASKTYPE_TARGET_RESET, 0, 10, - 0, TM_MUTEX_ON); + TM_MUTEX_ON); ioc->tm_cmds.status = MPT2_CMD_NOT_USED; } else mpt2sas_base_hard_reset_handler(ioc, CAN_SLEEP, diff --git a/drivers/scsi/mpt2sas/mpt2sas_scsih.c b/drivers/scsi/mpt2sas/mpt2sas_scsih.c index 6fd7d40b2c4d..5055f925d2cd 100644 --- a/drivers/scsi/mpt2sas/mpt2sas_scsih.c +++ b/drivers/scsi/mpt2sas/mpt2sas_scsih.c @@ -2368,7 +2368,6 @@ mpt2sas_scsih_clear_tm_flag(struct MPT2SAS_ADAPTER *ioc, u16 handle) * @type: MPI2_SCSITASKMGMT_TASKTYPE__XXX (defined in mpi2_init.h) * @smid_task: smid assigned to the task * @timeout: timeout in seconds - * @serial_number: the serial_number from scmd * @m_type: TM_MUTEX_ON or TM_MUTEX_OFF * Context: user * @@ -2381,7 +2380,7 @@ mpt2sas_scsih_clear_tm_flag(struct MPT2SAS_ADAPTER *ioc, u16 handle) int mpt2sas_scsih_issue_tm(struct MPT2SAS_ADAPTER *ioc, u16 handle, uint channel, uint id, uint lun, u8 type, u16 smid_task, ulong timeout, - unsigned long serial_number, enum mutex_type m_type) + enum mutex_type m_type) { Mpi2SCSITaskManagementRequest_t *mpi_request; Mpi2SCSITaskManagementReply_t *mpi_reply; @@ -2634,8 +2633,7 @@ _scsih_abort(struct scsi_cmnd *scmd) handle = sas_device_priv_data->sas_target->handle; r = mpt2sas_scsih_issue_tm(ioc, handle, scmd->device->channel, scmd->device->id, scmd->device->lun, - MPI2_SCSITASKMGMT_TASKTYPE_ABORT_TASK, smid, 30, - scmd->serial_number, TM_MUTEX_ON); + MPI2_SCSITASKMGMT_TASKTYPE_ABORT_TASK, smid, 30, TM_MUTEX_ON); out: sdev_printk(KERN_INFO, scmd->device, "task abort: %s scmd(%p)\n", @@ -2696,8 +2694,7 @@ _scsih_dev_reset(struct scsi_cmnd *scmd) r = mpt2sas_scsih_issue_tm(ioc, handle, scmd->device->channel, scmd->device->id, scmd->device->lun, - MPI2_SCSITASKMGMT_TASKTYPE_LOGICAL_UNIT_RESET, 0, 30, 0, - TM_MUTEX_ON); + MPI2_SCSITASKMGMT_TASKTYPE_LOGICAL_UNIT_RESET, 0, 30, TM_MUTEX_ON); out: sdev_printk(KERN_INFO, scmd->device, "device reset: %s scmd(%p)\n", @@ -2757,7 +2754,7 @@ _scsih_target_reset(struct scsi_cmnd *scmd) r = mpt2sas_scsih_issue_tm(ioc, handle, scmd->device->channel, scmd->device->id, 0, MPI2_SCSITASKMGMT_TASKTYPE_TARGET_RESET, 0, - 30, 0, TM_MUTEX_ON); + 30, TM_MUTEX_ON); out: starget_printk(KERN_INFO, starget, "target reset: %s scmd(%p)\n", @@ -3953,9 +3950,9 @@ _scsih_setup_direct_io(struct MPT2SAS_ADAPTER *ioc, struct scsi_cmnd *scmd, * SCSI_MLQUEUE_HOST_BUSY if the entire host queue is full */ static int -_scsih_qcmd_lck(struct scsi_cmnd *scmd, void (*done)(struct scsi_cmnd *)) +_scsih_qcmd(struct Scsi_Host *shost, struct scsi_cmnd *scmd) { - struct MPT2SAS_ADAPTER *ioc = shost_priv(scmd->device->host); + struct MPT2SAS_ADAPTER *ioc = shost_priv(shost); struct MPT2SAS_DEVICE *sas_device_priv_data; struct MPT2SAS_TARGET *sas_target_priv_data; struct _raid_device *raid_device; @@ -3963,7 +3960,6 @@ _scsih_qcmd_lck(struct scsi_cmnd *scmd, void (*done)(struct scsi_cmnd *)) u32 mpi_control; u16 smid; - scmd->scsi_done = done; sas_device_priv_data = scmd->device->hostdata; if (!sas_device_priv_data || !sas_device_priv_data->sas_target) { scmd->result = DID_NO_CONNECT << 16; @@ -4039,7 +4035,7 @@ _scsih_qcmd_lck(struct scsi_cmnd *scmd, void (*done)(struct scsi_cmnd *)) MPT_TARGET_FLAGS_RAID_COMPONENT) mpi_request->Function = MPI2_FUNCTION_RAID_SCSI_IO_PASSTHROUGH; else - mpi_request->Function = MPI2_FUNCTION_SCSI_IO_REQUEST; + mpi_request->Function = MPI2_FUNCTION_SCSI_IO_REQUEST; mpi_request->DevHandle = cpu_to_le16(sas_device_priv_data->sas_target->handle); mpi_request->DataLength = cpu_to_le32(scsi_bufflen(scmd)); @@ -4083,8 +4079,6 @@ _scsih_qcmd_lck(struct scsi_cmnd *scmd, void (*done)(struct scsi_cmnd *)) return SCSI_MLQUEUE_HOST_BUSY; } -static DEF_SCSI_QCMD(_scsih_qcmd) - /** * _scsih_normalize_sense - normalize descriptor and fixed format sense data * @sense_buffer: sense data returned by target @@ -5880,7 +5874,7 @@ broadcast_aen_retry: spin_unlock_irqrestore(&ioc->scsi_lookup_lock, flags); r = mpt2sas_scsih_issue_tm(ioc, handle, 0, 0, lun, - MPI2_SCSITASKMGMT_TASKTYPE_QUERY_TASK, smid, 30, 0, + MPI2_SCSITASKMGMT_TASKTYPE_QUERY_TASK, smid, 30, TM_MUTEX_OFF); if (r == FAILED) { sdev_printk(KERN_WARNING, sdev, @@ -5922,7 +5916,7 @@ broadcast_aen_retry: r = mpt2sas_scsih_issue_tm(ioc, handle, sdev->channel, sdev->id, sdev->lun, MPI2_SCSITASKMGMT_TASKTYPE_ABORT_TASK, smid, 30, - scmd->serial_number, TM_MUTEX_OFF); + TM_MUTEX_OFF); if (r == FAILED) { sdev_printk(KERN_WARNING, sdev, "mpt2sas_scsih_issue_tm: ABORT_TASK: FAILED : " diff --git a/drivers/scsi/mpt3sas/mpt3sas_base.h b/drivers/scsi/mpt3sas/mpt3sas_base.h index 0ebf5d913c80..9b90a6fef706 100644 --- a/drivers/scsi/mpt3sas/mpt3sas_base.h +++ b/drivers/scsi/mpt3sas/mpt3sas_base.h @@ -993,7 +993,7 @@ void mpt3sas_scsih_reset_handler(struct MPT3SAS_ADAPTER *ioc, int reset_phase); int mpt3sas_scsih_issue_tm(struct MPT3SAS_ADAPTER *ioc, u16 handle, uint channel, uint id, uint lun, u8 type, u16 smid_task, - ulong timeout, unsigned long serial_number, enum mutex_type m_type); + ulong timeout, enum mutex_type m_type); void mpt3sas_scsih_set_tm_flag(struct MPT3SAS_ADAPTER *ioc, u16 handle); void mpt3sas_scsih_clear_tm_flag(struct MPT3SAS_ADAPTER *ioc, u16 handle); void mpt3sas_expander_remove(struct MPT3SAS_ADAPTER *ioc, u64 sas_address); diff --git a/drivers/scsi/mpt3sas/mpt3sas_ctl.c b/drivers/scsi/mpt3sas/mpt3sas_ctl.c index 9b89de14a0a3..ba9cbe598a91 100644 --- a/drivers/scsi/mpt3sas/mpt3sas_ctl.c +++ b/drivers/scsi/mpt3sas/mpt3sas_ctl.c @@ -980,7 +980,7 @@ _ctl_do_mpt_command(struct MPT3SAS_ADAPTER *ioc, struct mpt3_ioctl_command karg, mpt3sas_scsih_issue_tm(ioc, le16_to_cpu(mpi_request->FunctionDependent1), 0, 0, 0, MPI2_SCSITASKMGMT_TASKTYPE_TARGET_RESET, 0, 30, - 0, TM_MUTEX_ON); + TM_MUTEX_ON); } else mpt3sas_base_hard_reset_handler(ioc, CAN_SLEEP, FORCE_BIG_HAMMER); diff --git a/drivers/scsi/mpt3sas/mpt3sas_scsih.c b/drivers/scsi/mpt3sas/mpt3sas_scsih.c index a961fe11b527..18e713db1d32 100644 --- a/drivers/scsi/mpt3sas/mpt3sas_scsih.c +++ b/drivers/scsi/mpt3sas/mpt3sas_scsih.c @@ -2029,7 +2029,6 @@ mpt3sas_scsih_clear_tm_flag(struct MPT3SAS_ADAPTER *ioc, u16 handle) * @type: MPI2_SCSITASKMGMT_TASKTYPE__XXX (defined in mpi2_init.h) * @smid_task: smid assigned to the task * @timeout: timeout in seconds - * @serial_number: the serial_number from scmd * @m_type: TM_MUTEX_ON or TM_MUTEX_OFF * Context: user * @@ -2042,7 +2041,7 @@ mpt3sas_scsih_clear_tm_flag(struct MPT3SAS_ADAPTER *ioc, u16 handle) int mpt3sas_scsih_issue_tm(struct MPT3SAS_ADAPTER *ioc, u16 handle, uint channel, uint id, uint lun, u8 type, u16 smid_task, ulong timeout, - unsigned long serial_number, enum mutex_type m_type) + enum mutex_type m_type) { Mpi2SCSITaskManagementRequest_t *mpi_request; Mpi2SCSITaskManagementReply_t *mpi_reply; @@ -2293,8 +2292,7 @@ _scsih_abort(struct scsi_cmnd *scmd) handle = sas_device_priv_data->sas_target->handle; r = mpt3sas_scsih_issue_tm(ioc, handle, scmd->device->channel, scmd->device->id, scmd->device->lun, - MPI2_SCSITASKMGMT_TASKTYPE_ABORT_TASK, smid, 30, - scmd->serial_number, TM_MUTEX_ON); + MPI2_SCSITASKMGMT_TASKTYPE_ABORT_TASK, smid, 30, TM_MUTEX_ON); out: sdev_printk(KERN_INFO, scmd->device, "task abort: %s scmd(%p)\n", @@ -2353,8 +2351,7 @@ _scsih_dev_reset(struct scsi_cmnd *scmd) r = mpt3sas_scsih_issue_tm(ioc, handle, scmd->device->channel, scmd->device->id, scmd->device->lun, - MPI2_SCSITASKMGMT_TASKTYPE_LOGICAL_UNIT_RESET, 0, 30, 0, - TM_MUTEX_ON); + MPI2_SCSITASKMGMT_TASKTYPE_LOGICAL_UNIT_RESET, 0, 30, TM_MUTEX_ON); out: sdev_printk(KERN_INFO, scmd->device, "device reset: %s scmd(%p)\n", @@ -2414,7 +2411,7 @@ _scsih_target_reset(struct scsi_cmnd *scmd) r = mpt3sas_scsih_issue_tm(ioc, handle, scmd->device->channel, scmd->device->id, 0, MPI2_SCSITASKMGMT_TASKTYPE_TARGET_RESET, 0, - 30, 0, TM_MUTEX_ON); + 30, TM_MUTEX_ON); out: starget_printk(KERN_INFO, starget, "target reset: %s scmd(%p)\n", @@ -3518,7 +3515,7 @@ _scsih_eedp_error_handling(struct scsi_cmnd *scmd, u16 ioc_status) /** - * _scsih_qcmd_lck - main scsi request entry point + * _scsih_qcmd - main scsi request entry point * @scmd: pointer to scsi command object * @done: function pointer to be invoked on completion * @@ -3529,9 +3526,9 @@ _scsih_eedp_error_handling(struct scsi_cmnd *scmd, u16 ioc_status) * SCSI_MLQUEUE_HOST_BUSY if the entire host queue is full */ static int -_scsih_qcmd_lck(struct scsi_cmnd *scmd, void (*done)(struct scsi_cmnd *)) +_scsih_qcmd(struct Scsi_Host *shost, struct scsi_cmnd *scmd) { - struct MPT3SAS_ADAPTER *ioc = shost_priv(scmd->device->host); + struct MPT3SAS_ADAPTER *ioc = shost_priv(shost); struct MPT3SAS_DEVICE *sas_device_priv_data; struct MPT3SAS_TARGET *sas_target_priv_data; Mpi2SCSIIORequest_t *mpi_request; @@ -3544,7 +3541,6 @@ _scsih_qcmd_lck(struct scsi_cmnd *scmd, void (*done)(struct scsi_cmnd *)) scsi_print_command(scmd); #endif - scmd->scsi_done = done; sas_device_priv_data = scmd->device->hostdata; if (!sas_device_priv_data || !sas_device_priv_data->sas_target) { scmd->result = DID_NO_CONNECT << 16; @@ -3659,8 +3655,6 @@ _scsih_qcmd_lck(struct scsi_cmnd *scmd, void (*done)(struct scsi_cmnd *)) out: return SCSI_MLQUEUE_HOST_BUSY; } -static DEF_SCSI_QCMD(_scsih_qcmd) - /** * _scsih_normalize_sense - normalize descriptor and fixed format sense data @@ -5425,7 +5419,7 @@ _scsih_sas_broadcast_primitive_event(struct MPT3SAS_ADAPTER *ioc, spin_unlock_irqrestore(&ioc->scsi_lookup_lock, flags); r = mpt3sas_scsih_issue_tm(ioc, handle, 0, 0, lun, - MPI2_SCSITASKMGMT_TASKTYPE_QUERY_TASK, smid, 30, 0, + MPI2_SCSITASKMGMT_TASKTYPE_QUERY_TASK, smid, 30, TM_MUTEX_OFF); if (r == FAILED) { sdev_printk(KERN_WARNING, sdev, @@ -5467,7 +5461,7 @@ _scsih_sas_broadcast_primitive_event(struct MPT3SAS_ADAPTER *ioc, r = mpt3sas_scsih_issue_tm(ioc, handle, sdev->channel, sdev->id, sdev->lun, MPI2_SCSITASKMGMT_TASKTYPE_ABORT_TASK, smid, 30, - scmd->serial_number, TM_MUTEX_OFF); + TM_MUTEX_OFF); if (r == FAILED) { sdev_printk(KERN_WARNING, sdev, "mpt3sas_scsih_issue_tm: ABORT_TASK: FAILED : " diff --git a/drivers/scsi/mvsas/mv_init.c b/drivers/scsi/mvsas/mv_init.c index 5ff978be249d..eacee48a955c 100644 --- a/drivers/scsi/mvsas/mv_init.c +++ b/drivers/scsi/mvsas/mv_init.c @@ -728,6 +728,15 @@ static struct pci_device_id mvs_pci_table[] = { .class_mask = 0, .driver_data = chip_9485, }, + { + .vendor = PCI_VENDOR_ID_MARVELL_EXT, + .device = 0x9485, + .subvendor = PCI_ANY_ID, + .subdevice = 0x9485, + .class = 0, + .class_mask = 0, + .driver_data = chip_9485, + }, { PCI_VDEVICE(OCZ, 0x1021), chip_9485}, /* OCZ RevoDrive3 */ { PCI_VDEVICE(OCZ, 0x1022), chip_9485}, /* OCZ RevoDrive3/zDriveR4 (exact model unknown) */ { PCI_VDEVICE(OCZ, 0x1040), chip_9485}, /* OCZ RevoDrive3/zDriveR4 (exact model unknown) */ diff --git a/drivers/scsi/pas16.h b/drivers/scsi/pas16.h index 3721342835e9..aa528f53c533 100644 --- a/drivers/scsi/pas16.h +++ b/drivers/scsi/pas16.h @@ -129,8 +129,6 @@ static int pas16_bus_reset(Scsi_Cmnd *); #define CAN_QUEUE 32 #endif -#ifndef HOSTS_C - #define NCR5380_implementation_fields \ volatile unsigned short io_port @@ -171,6 +169,5 @@ static int pas16_bus_reset(Scsi_Cmnd *); #define PAS16_IRQS 0xd4a8 -#endif /* else def HOSTS_C */ #endif /* ndef ASM */ #endif /* PAS16_H */ diff --git a/drivers/scsi/pm8001/pm8001_ctl.c b/drivers/scsi/pm8001/pm8001_ctl.c index 28b4e8139153..fe5eee4d0a11 100644 --- a/drivers/scsi/pm8001/pm8001_ctl.c +++ b/drivers/scsi/pm8001/pm8001_ctl.c @@ -395,6 +395,8 @@ static ssize_t pm8001_ctl_bios_version_show(struct device *cdev, payload.offset = 0; payload.length = 4096; payload.func_specific = kzalloc(4096, GFP_KERNEL); + if (!payload.func_specific) + return -ENOMEM; PM8001_CHIP_DISP->get_nvmd_req(pm8001_ha, &payload); wait_for_completion(&completion); virt_addr = pm8001_ha->memoryMap.region[NVMD].virt_ptr; @@ -402,6 +404,7 @@ static ssize_t pm8001_ctl_bios_version_show(struct device *cdev, bios_index++) str += sprintf(str, "%c", *((u8 *)((u8 *)virt_addr+bios_index))); + kfree(payload.func_specific); return str - buf; } static DEVICE_ATTR(bios_version, S_IRUGO, pm8001_ctl_bios_version_show, NULL); diff --git a/drivers/scsi/qla2xxx/qla_attr.c b/drivers/scsi/qla2xxx/qla_attr.c index 07befcf365b8..16fe5196e6d9 100644 --- a/drivers/scsi/qla2xxx/qla_attr.c +++ b/drivers/scsi/qla2xxx/qla_attr.c @@ -1,6 +1,6 @@ /* * QLogic Fibre Channel HBA Driver - * Copyright (c) 2003-2013 QLogic Corporation + * Copyright (c) 2003-2014 QLogic Corporation * * See LICENSE.qla2xxx for copyright and licensing details. */ @@ -664,7 +664,7 @@ do_read: } rval = qla2x00_read_sfp(vha, ha->sfp_data_dma, ha->sfp_data, - addr, offset, SFP_BLOCK_SIZE, 0); + addr, offset, SFP_BLOCK_SIZE, BIT_1); if (rval != QLA_SUCCESS) { ql_log(ql_log_warn, vha, 0x706d, "Unable to read SFP data (%x/%x/%x).\n", rval, @@ -1495,7 +1495,7 @@ qla2x00_fw_dump_size_show(struct device *dev, struct device_attribute *attr, if (!ha->fw_dumped) size = 0; - else if (IS_QLA82XX(ha)) + else if (IS_P3P_TYPE(ha)) size = ha->md_template_size + ha->md_dump_size; else size = ha->fw_dump_len; diff --git a/drivers/scsi/qla2xxx/qla_bsg.c b/drivers/scsi/qla2xxx/qla_bsg.c index 71ff340f6de4..524f9eb7fcd1 100644 --- a/drivers/scsi/qla2xxx/qla_bsg.c +++ b/drivers/scsi/qla2xxx/qla_bsg.c @@ -1,6 +1,6 @@ /* * QLogic Fibre Channel HBA Driver - * Copyright (c) 2003-2012 QLogic Corporation + * Copyright (c) 2003-2014 QLogic Corporation * * See LICENSE.qla2xxx for copyright and licensing details. */ @@ -2054,9 +2054,49 @@ qla26xx_serdes_op(struct fc_bsg_job *bsg_job) bsg_job->reply->reply_payload_rcv_len = sizeof(sr); break; default: - ql_log(ql_log_warn, vha, 0x708c, + ql_dbg(ql_dbg_user, vha, 0x708c, "Unknown serdes cmd %x.\n", sr.cmd); - rval = -EDOM; + rval = -EINVAL; + break; + } + + bsg_job->reply->reply_data.vendor_reply.vendor_rsp[0] = + rval ? EXT_STATUS_MAILBOX : 0; + + bsg_job->reply_len = sizeof(struct fc_bsg_reply); + bsg_job->reply->result = DID_OK << 16; + bsg_job->job_done(bsg_job); + return 0; +} + +static int +qla8044_serdes_op(struct fc_bsg_job *bsg_job) +{ + struct Scsi_Host *host = bsg_job->shost; + scsi_qla_host_t *vha = shost_priv(host); + int rval = 0; + struct qla_serdes_reg_ex sr; + + memset(&sr, 0, sizeof(sr)); + + sg_copy_to_buffer(bsg_job->request_payload.sg_list, + bsg_job->request_payload.sg_cnt, &sr, sizeof(sr)); + + switch (sr.cmd) { + case INT_SC_SERDES_WRITE_REG: + rval = qla8044_write_serdes_word(vha, sr.addr, sr.val); + bsg_job->reply->reply_payload_rcv_len = 0; + break; + case INT_SC_SERDES_READ_REG: + rval = qla8044_read_serdes_word(vha, sr.addr, &sr.val); + sg_copy_from_buffer(bsg_job->reply_payload.sg_list, + bsg_job->reply_payload.sg_cnt, &sr, sizeof(sr)); + bsg_job->reply->reply_payload_rcv_len = sizeof(sr); + break; + default: + ql_dbg(ql_dbg_user, vha, 0x70cf, + "Unknown serdes cmd %x.\n", sr.cmd); + rval = -EINVAL; break; } @@ -2121,6 +2161,9 @@ qla2x00_process_vendor_specific(struct fc_bsg_job *bsg_job) case QL_VND_SERDES_OP: return qla26xx_serdes_op(bsg_job); + case QL_VND_SERDES_OP_EX: + return qla8044_serdes_op(bsg_job); + default: return -ENOSYS; } diff --git a/drivers/scsi/qla2xxx/qla_bsg.h b/drivers/scsi/qla2xxx/qla_bsg.h index e5c2126221e9..d38f9efa56fa 100644 --- a/drivers/scsi/qla2xxx/qla_bsg.h +++ b/drivers/scsi/qla2xxx/qla_bsg.h @@ -1,6 +1,6 @@ /* * QLogic Fibre Channel HBA Driver - * Copyright (c) 2003-2013 QLogic Corporation + * Copyright (c) 2003-2014 QLogic Corporation * * See LICENSE.qla2xxx for copyright and licensing details. */ @@ -24,6 +24,7 @@ #define QL_VND_READ_I2C 0x11 #define QL_VND_FX00_MGMT_CMD 0x12 #define QL_VND_SERDES_OP 0x13 +#define QL_VND_SERDES_OP_EX 0x14 /* BSG Vendor specific subcode returns */ #define EXT_STATUS_OK 0 @@ -225,4 +226,10 @@ struct qla_serdes_reg { uint16_t val; } __packed; +struct qla_serdes_reg_ex { + uint16_t cmd; + uint32_t addr; + uint32_t val; +} __packed; + #endif diff --git a/drivers/scsi/qla2xxx/qla_dbg.c b/drivers/scsi/qla2xxx/qla_dbg.c index 97255f7c3975..c72ee97bf3f7 100644 --- a/drivers/scsi/qla2xxx/qla_dbg.c +++ b/drivers/scsi/qla2xxx/qla_dbg.c @@ -1,6 +1,6 @@ /* * QLogic Fibre Channel HBA Driver - * Copyright (c) 2003-2013 QLogic Corporation + * Copyright (c) 2003-2014 QLogic Corporation * * See LICENSE.qla2xxx for copyright and licensing details. */ @@ -15,7 +15,7 @@ * | | | 0x0144,0x0146 | * | | | 0x015b-0x0160 | * | | | 0x016e-0x0170 | - * | Mailbox commands | 0x1187 | 0x1018-0x1019 | + * | Mailbox commands | 0x118d | 0x1018-0x1019 | * | | | 0x10ca | * | | | 0x1115-0x1116 | * | | | 0x111a-0x111b | @@ -45,12 +45,16 @@ * | | | 0x70ad-0x70ae | * | | | 0x70d7-0x70db | * | | | 0x70de-0x70df | - * | Task Management | 0x803d | 0x8025-0x8026 | - * | | | 0x800b,0x8039 | + * | Task Management | 0x803d | 0x8000,0x800b | + * | | | 0x8019 | + * | | | 0x8025,0x8026 | + * | | | 0x8031,0x8032 | + * | | | 0x8039,0x803c | * | AER/EEH | 0x9011 | | * | Virtual Port | 0xa007 | | - * | ISP82XX Specific | 0xb14c | 0xb002,0xb024 | + * | ISP82XX Specific | 0xb157 | 0xb002,0xb024 | * | | | 0xb09e,0xb0ae | + * | | | 0xb0c3,0xb0c6 | * | | | 0xb0e0-0xb0ef | * | | | 0xb085,0xb0dc | * | | | 0xb107,0xb108 | @@ -60,12 +64,12 @@ * | | | 0xb13c-0xb140 | * | | | 0xb149 | * | MultiQ | 0xc00c | | - * | Misc | 0xd2ff | 0xd017-0xd019 | + * | Misc | 0xd212 | 0xd017-0xd019 | * | | | 0xd020 | - * | | | 0xd02e-0xd0ff | + * | | | 0xd030-0xd0ff | * | | | 0xd101-0xd1fe | - * | | | 0xd212-0xd2fe | - * | Target Mode | 0xe070 | 0xe021 | + * | | | 0xd213-0xd2fe | + * | Target Mode | 0xe078 | | * | Target Mode Management | 0xf072 | 0xf002-0xf003 | * | | | 0xf046-0xf049 | * | Target Mode Task Management | 0x1000b | | @@ -277,9 +281,15 @@ qla24xx_dump_memory(struct qla_hw_data *ha, uint32_t *code_ram, if (rval != QLA_SUCCESS) return rval; + set_bit(RISC_SRAM_DUMP_CMPL, &ha->fw_dump_cap_flags); + /* External Memory. */ - return qla24xx_dump_ram(ha, 0x100000, *nxt, + rval = qla24xx_dump_ram(ha, 0x100000, *nxt, ha->fw_memory_size - 0x100000 + 1, nxt); + if (rval == QLA_SUCCESS) + set_bit(RISC_EXT_MEM_DUMP_CMPL, &ha->fw_dump_cap_flags); + + return rval; } static uint32_t * @@ -296,23 +306,15 @@ qla24xx_read_window(struct device_reg_24xx __iomem *reg, uint32_t iobase, return buf; } -int -qla24xx_pause_risc(struct device_reg_24xx __iomem *reg) +void +qla24xx_pause_risc(struct device_reg_24xx __iomem *reg, struct qla_hw_data *ha) { - int rval = QLA_SUCCESS; - uint32_t cnt; - WRT_REG_DWORD(®->hccr, HCCRX_SET_RISC_PAUSE); - for (cnt = 30000; - ((RD_REG_DWORD(®->host_status) & HSRX_RISC_PAUSED) == 0) && - rval == QLA_SUCCESS; cnt--) { - if (cnt) - udelay(100); - else - rval = QLA_FUNCTION_TIMEOUT; - } - return rval; + /* 100 usec delay is sufficient enough for hardware to pause RISC */ + udelay(100); + if (RD_REG_DWORD(®->host_status) & HSRX_RISC_PAUSED) + set_bit(RISC_PAUSE_CMPL, &ha->fw_dump_cap_flags); } int @@ -320,10 +322,14 @@ qla24xx_soft_reset(struct qla_hw_data *ha) { int rval = QLA_SUCCESS; uint32_t cnt; - uint16_t mb0, wd; + uint16_t wd; struct device_reg_24xx __iomem *reg = &ha->iobase->isp24; - /* Reset RISC. */ + /* + * Reset RISC. The delay is dependent on system architecture. + * Driver can proceed with the reset sequence after waiting + * for a timeout period. + */ WRT_REG_DWORD(®->ctrl_status, CSRX_DMA_SHUTDOWN|MWB_4096_BYTES); for (cnt = 0; cnt < 30000; cnt++) { if ((RD_REG_DWORD(®->ctrl_status) & CSRX_DMA_ACTIVE) == 0) @@ -331,19 +337,14 @@ qla24xx_soft_reset(struct qla_hw_data *ha) udelay(10); } + if (!(RD_REG_DWORD(®->ctrl_status) & CSRX_DMA_ACTIVE)) + set_bit(DMA_SHUTDOWN_CMPL, &ha->fw_dump_cap_flags); WRT_REG_DWORD(®->ctrl_status, CSRX_ISP_SOFT_RESET|CSRX_DMA_SHUTDOWN|MWB_4096_BYTES); pci_read_config_word(ha->pdev, PCI_COMMAND, &wd); udelay(100); - /* Wait for firmware to complete NVRAM accesses. */ - mb0 = (uint32_t) RD_REG_WORD(®->mailbox0); - for (cnt = 10000 ; cnt && mb0; cnt--) { - udelay(5); - mb0 = (uint32_t) RD_REG_WORD(®->mailbox0); - barrier(); - } /* Wait for soft-reset to complete. */ for (cnt = 0; cnt < 30000; cnt++) { @@ -353,16 +354,21 @@ qla24xx_soft_reset(struct qla_hw_data *ha) udelay(10); } + if (!(RD_REG_DWORD(®->ctrl_status) & CSRX_ISP_SOFT_RESET)) + set_bit(ISP_RESET_CMPL, &ha->fw_dump_cap_flags); + WRT_REG_DWORD(®->hccr, HCCRX_CLR_RISC_RESET); RD_REG_DWORD(®->hccr); /* PCI Posting. */ - for (cnt = 30000; RD_REG_WORD(®->mailbox0) != 0 && + for (cnt = 10000; RD_REG_WORD(®->mailbox0) != 0 && rval == QLA_SUCCESS; cnt--) { if (cnt) - udelay(100); + udelay(10); else rval = QLA_FUNCTION_TIMEOUT; } + if (rval == QLA_SUCCESS) + set_bit(RISC_RDY_AFT_RESET, &ha->fw_dump_cap_flags); return rval; } @@ -659,12 +665,13 @@ qla2xxx_dump_post_process(scsi_qla_host_t *vha, int rval) if (rval != QLA_SUCCESS) { ql_log(ql_log_warn, vha, 0xd000, - "Failed to dump firmware (%x).\n", rval); + "Failed to dump firmware (%x), dump status flags (0x%lx).\n", + rval, ha->fw_dump_cap_flags); ha->fw_dumped = 0; } else { ql_log(ql_log_info, vha, 0xd001, - "Firmware dump saved to temp buffer (%ld/%p).\n", - vha->host_no, ha->fw_dump); + "Firmware dump saved to temp buffer (%ld/%p), dump status flags (0x%lx).\n", + vha->host_no, ha->fw_dump, ha->fw_dump_cap_flags); ha->fw_dumped = 1; qla2x00_post_uevent_work(vha, QLA_UEVENT_CODE_FW_DUMP); } @@ -1053,6 +1060,7 @@ qla24xx_fw_dump(scsi_qla_host_t *vha, int hardware_locked) risc_address = ext_mem_cnt = 0; flags = 0; + ha->fw_dump_cap_flags = 0; if (!hardware_locked) spin_lock_irqsave(&ha->hardware_lock, flags); @@ -1075,10 +1083,11 @@ qla24xx_fw_dump(scsi_qla_host_t *vha, int hardware_locked) fw->host_status = htonl(RD_REG_DWORD(®->host_status)); - /* Pause RISC. */ - rval = qla24xx_pause_risc(reg); - if (rval != QLA_SUCCESS) - goto qla24xx_fw_dump_failed_0; + /* + * Pause RISC. No need to track timeout, as resetting the chip + * is the right approach incase of pause timeout + */ + qla24xx_pause_risc(reg, ha); /* Host interface registers. */ dmp_reg = ®->flash_addr; @@ -1302,6 +1311,7 @@ qla25xx_fw_dump(scsi_qla_host_t *vha, int hardware_locked) risc_address = ext_mem_cnt = 0; flags = 0; + ha->fw_dump_cap_flags = 0; if (!hardware_locked) spin_lock_irqsave(&ha->hardware_lock, flags); @@ -1325,10 +1335,11 @@ qla25xx_fw_dump(scsi_qla_host_t *vha, int hardware_locked) fw->host_status = htonl(RD_REG_DWORD(®->host_status)); - /* Pause RISC. */ - rval = qla24xx_pause_risc(reg); - if (rval != QLA_SUCCESS) - goto qla25xx_fw_dump_failed_0; + /* + * Pause RISC. No need to track timeout, as resetting the chip + * is the right approach incase of pause timeout + */ + qla24xx_pause_risc(reg, ha); /* Host/Risc registers. */ iter_reg = fw->host_risc_reg; @@ -1619,6 +1630,7 @@ qla81xx_fw_dump(scsi_qla_host_t *vha, int hardware_locked) risc_address = ext_mem_cnt = 0; flags = 0; + ha->fw_dump_cap_flags = 0; if (!hardware_locked) spin_lock_irqsave(&ha->hardware_lock, flags); @@ -1641,10 +1653,11 @@ qla81xx_fw_dump(scsi_qla_host_t *vha, int hardware_locked) fw->host_status = htonl(RD_REG_DWORD(®->host_status)); - /* Pause RISC. */ - rval = qla24xx_pause_risc(reg); - if (rval != QLA_SUCCESS) - goto qla81xx_fw_dump_failed_0; + /* + * Pause RISC. No need to track timeout, as resetting the chip + * is the right approach incase of pause timeout + */ + qla24xx_pause_risc(reg, ha); /* Host/Risc registers. */ iter_reg = fw->host_risc_reg; @@ -1938,6 +1951,7 @@ qla83xx_fw_dump(scsi_qla_host_t *vha, int hardware_locked) risc_address = ext_mem_cnt = 0; flags = 0; + ha->fw_dump_cap_flags = 0; if (!hardware_locked) spin_lock_irqsave(&ha->hardware_lock, flags); @@ -1959,10 +1973,11 @@ qla83xx_fw_dump(scsi_qla_host_t *vha, int hardware_locked) fw->host_status = htonl(RD_REG_DWORD(®->host_status)); - /* Pause RISC. */ - rval = qla24xx_pause_risc(reg); - if (rval != QLA_SUCCESS) - goto qla83xx_fw_dump_failed_0; + /* + * Pause RISC. No need to track timeout, as resetting the chip + * is the right approach incase of pause timeout + */ + qla24xx_pause_risc(reg, ha); WRT_REG_DWORD(®->iobase_addr, 0x6000); dmp_reg = ®->iobase_window; @@ -2385,9 +2400,11 @@ qla83xx_fw_dump(scsi_qla_host_t *vha, int hardware_locked) nxt += sizeof(fw->code_ram); nxt += (ha->fw_memory_size - 0x100000 + 1); goto copy_queue; - } else + } else { + set_bit(RISC_RDY_AFT_RESET, &ha->fw_dump_cap_flags); ql_log(ql_log_warn, vha, 0xd010, "bigger hammer success?\n"); + } } rval = qla24xx_dump_memory(ha, fw->code_ram, sizeof(fw->code_ram), diff --git a/drivers/scsi/qla2xxx/qla_dbg.h b/drivers/scsi/qla2xxx/qla_dbg.h index cc961040f8b1..e1fc4e66966a 100644 --- a/drivers/scsi/qla2xxx/qla_dbg.h +++ b/drivers/scsi/qla2xxx/qla_dbg.h @@ -1,6 +1,6 @@ /* * QLogic Fibre Channel HBA Driver - * Copyright (c) 2003-2013 QLogic Corporation + * Copyright (c) 2003-2014 QLogic Corporation * * See LICENSE.qla2xxx for copyright and licensing details. */ @@ -353,5 +353,6 @@ extern int qla27xx_dump_mpi_ram(struct qla_hw_data *, uint32_t, uint32_t *, uint32_t, void **); extern int qla24xx_dump_ram(struct qla_hw_data *, uint32_t, uint32_t *, uint32_t, void **); -extern int qla24xx_pause_risc(struct device_reg_24xx __iomem *); +extern void qla24xx_pause_risc(struct device_reg_24xx __iomem *, + struct qla_hw_data *); extern int qla24xx_soft_reset(struct qla_hw_data *); diff --git a/drivers/scsi/qla2xxx/qla_def.h b/drivers/scsi/qla2xxx/qla_def.h index 6a106136716c..1fa010448666 100644 --- a/drivers/scsi/qla2xxx/qla_def.h +++ b/drivers/scsi/qla2xxx/qla_def.h @@ -1,6 +1,6 @@ /* * QLogic Fibre Channel HBA Driver - * Copyright (c) 2003-2013 QLogic Corporation + * Copyright (c) 2003-2014 QLogic Corporation * * See LICENSE.qla2xxx for copyright and licensing details. */ @@ -965,6 +965,13 @@ struct mbx_cmd_32 { */ #define MBC_WRITE_MPI_REGISTER 0x01 /* Write MPI Register. */ +/* + * ISP8044 mailbox commands + */ +#define MBC_SET_GET_ETH_SERDES_REG 0x150 +#define HCS_WRITE_SERDES 0x3 +#define HCS_READ_SERDES 0x4 + /* Firmware return data sizes */ #define FCAL_MAP_SIZE 128 @@ -1622,10 +1629,20 @@ typedef struct { #define PO_MODE_DIF_PASS 2 #define PO_MODE_DIF_REPLACE 3 #define PO_MODE_DIF_TCP_CKSUM 6 -#define PO_ENABLE_DIF_BUNDLING BIT_8 #define PO_ENABLE_INCR_GUARD_SEED BIT_3 -#define PO_DISABLE_INCR_REF_TAG BIT_5 #define PO_DISABLE_GUARD_CHECK BIT_4 +#define PO_DISABLE_INCR_REF_TAG BIT_5 +#define PO_DIS_HEADER_MODE BIT_7 +#define PO_ENABLE_DIF_BUNDLING BIT_8 +#define PO_DIS_FRAME_MODE BIT_9 +#define PO_DIS_VALD_APP_ESC BIT_10 /* Dis validation for escape tag/ffffh */ +#define PO_DIS_VALD_APP_REF_ESC BIT_11 + +#define PO_DIS_APP_TAG_REPL BIT_12 /* disable REG Tag replacement */ +#define PO_DIS_REF_TAG_REPL BIT_13 +#define PO_DIS_APP_TAG_VALD BIT_14 /* disable REF Tag validation */ +#define PO_DIS_REF_TAG_VALD BIT_15 + /* * ISP queue - 64-Bit addressing, continuation crc entry structure definition. */ @@ -1748,6 +1765,8 @@ typedef struct { #define CS_PORT_CONFIG_CHG 0x2A /* Port Configuration Changed */ #define CS_PORT_BUSY 0x2B /* Port Busy */ #define CS_COMPLETE_CHKCOND 0x30 /* Error? */ +#define CS_IOCB_ERROR 0x31 /* Generic error for IOCB request + failure */ #define CS_BAD_PAYLOAD 0x80 /* Driver defined */ #define CS_UNKNOWN 0x81 /* Driver defined */ #define CS_RETRY 0x82 /* Driver defined */ @@ -2676,6 +2695,7 @@ struct rsp_que { uint32_t __iomem *rsp_q_out; uint16_t ring_index; uint16_t out_ptr; + uint16_t *in_ptr; /* queue shadow in index */ uint16_t length; uint16_t options; uint16_t rid; @@ -2702,6 +2722,7 @@ struct req_que { uint32_t __iomem *req_q_out; uint16_t ring_index; uint16_t in_ptr; + uint16_t *out_ptr; /* queue shadow out index */ uint16_t cnt; uint16_t length; uint16_t options; @@ -2907,6 +2928,8 @@ struct qla_hw_data { #define PCI_DEVICE_ID_QLOGIC_ISP8031 0x8031 #define PCI_DEVICE_ID_QLOGIC_ISP2031 0x2031 #define PCI_DEVICE_ID_QLOGIC_ISP2071 0x2071 +#define PCI_DEVICE_ID_QLOGIC_ISP2271 0x2271 + uint32_t device_type; #define DT_ISP2100 BIT_0 #define DT_ISP2200 BIT_1 @@ -2928,7 +2951,8 @@ struct qla_hw_data { #define DT_ISPFX00 BIT_17 #define DT_ISP8044 BIT_18 #define DT_ISP2071 BIT_19 -#define DT_ISP_LAST (DT_ISP2071 << 1) +#define DT_ISP2271 BIT_20 +#define DT_ISP_LAST (DT_ISP2271 << 1) #define DT_T10_PI BIT_25 #define DT_IIDMA BIT_26 @@ -2959,6 +2983,7 @@ struct qla_hw_data { #define IS_QLA8031(ha) (DT_MASK(ha) & DT_ISP8031) #define IS_QLAFX00(ha) (DT_MASK(ha) & DT_ISPFX00) #define IS_QLA2071(ha) (DT_MASK(ha) & DT_ISP2071) +#define IS_QLA2271(ha) (DT_MASK(ha) & DT_ISP2271) #define IS_QLA23XX(ha) (IS_QLA2300(ha) || IS_QLA2312(ha) || IS_QLA2322(ha) || \ IS_QLA6312(ha) || IS_QLA6322(ha)) @@ -2967,7 +2992,7 @@ struct qla_hw_data { #define IS_QLA25XX(ha) (IS_QLA2532(ha)) #define IS_QLA83XX(ha) (IS_QLA2031(ha) || IS_QLA8031(ha)) #define IS_QLA84XX(ha) (IS_QLA8432(ha)) -#define IS_QLA27XX(ha) (IS_QLA2071(ha)) +#define IS_QLA27XX(ha) (IS_QLA2071(ha) || IS_QLA2271(ha)) #define IS_QLA24XX_TYPE(ha) (IS_QLA24XX(ha) || IS_QLA54XX(ha) || \ IS_QLA84XX(ha)) #define IS_CNA_CAPABLE(ha) (IS_QLA81XX(ha) || IS_QLA82XX(ha) || \ @@ -3006,6 +3031,7 @@ struct qla_hw_data { (((ha)->fw_attributes_h << 16 | (ha)->fw_attributes) & BIT_22)) #define IS_ATIO_MSIX_CAPABLE(ha) (IS_QLA83XX(ha)) #define IS_TGT_MODE_CAPABLE(ha) (ha->tgt.atio_q_length) +#define IS_SHADOW_REG_CAPABLE(ha) (IS_QLA27XX(ha)) /* HBA serial number */ uint8_t serial0; @@ -3136,7 +3162,15 @@ struct qla_hw_data { struct qla2xxx_fw_dump *fw_dump; uint32_t fw_dump_len; int fw_dumped; + unsigned long fw_dump_cap_flags; +#define RISC_PAUSE_CMPL 0 +#define DMA_SHUTDOWN_CMPL 1 +#define ISP_RESET_CMPL 2 +#define RISC_RDY_AFT_RESET 3 +#define RISC_SRAM_DUMP_CMPL 4 +#define RISC_EXT_MEM_DUMP_CMPL 5 int fw_dump_reading; + int prev_minidump_failed; dma_addr_t eft_dma; void *eft; /* Current size of mctp dump is 0x086064 bytes */ diff --git a/drivers/scsi/qla2xxx/qla_dfs.c b/drivers/scsi/qla2xxx/qla_dfs.c index 32ab80957688..2ca39b8e7166 100644 --- a/drivers/scsi/qla2xxx/qla_dfs.c +++ b/drivers/scsi/qla2xxx/qla_dfs.c @@ -1,6 +1,6 @@ /* * QLogic Fibre Channel HBA Driver - * Copyright (c) 2003-2013 QLogic Corporation + * Copyright (c) 2003-2014 QLogic Corporation * * See LICENSE.qla2xxx for copyright and licensing details. */ diff --git a/drivers/scsi/qla2xxx/qla_fw.h b/drivers/scsi/qla2xxx/qla_fw.h index 3a7353eaccbd..eb8f57249f1d 100644 --- a/drivers/scsi/qla2xxx/qla_fw.h +++ b/drivers/scsi/qla2xxx/qla_fw.h @@ -1,6 +1,6 @@ /* * QLogic Fibre Channel HBA Driver - * Copyright (c) 2003-2013 QLogic Corporation + * Copyright (c) 2003-2014 QLogic Corporation * * See LICENSE.qla2xxx for copyright and licensing details. */ @@ -371,7 +371,10 @@ struct init_cb_24xx { * BIT 14 = Data Rate bit 1 * BIT 15 = Data Rate bit 2 * BIT 16 = Enable 75 ohm Termination Select - * BIT 17-31 = Reserved + * BIT 17-28 = Reserved + * BIT 29 = Enable response queue 0 in index shadowing + * BIT 30 = Enable request queue 0 out index shadowing + * BIT 31 = Reserved */ uint32_t firmware_options_3; uint16_t qos; @@ -1134,13 +1137,6 @@ struct device_reg_24xx { #define MIN_MULTI_ID_FABRIC 64 /* Must be power-of-2. */ #define MAX_MULTI_ID_FABRIC 256 /* ... */ -#define for_each_mapped_vp_idx(_ha, _idx) \ - for (_idx = find_next_bit((_ha)->vp_idx_map, \ - (_ha)->max_npiv_vports + 1, 1); \ - _idx <= (_ha)->max_npiv_vports; \ - _idx = find_next_bit((_ha)->vp_idx_map, \ - (_ha)->max_npiv_vports + 1, _idx + 1)) \ - struct mid_conf_entry_24xx { uint16_t reserved_1; diff --git a/drivers/scsi/qla2xxx/qla_gbl.h b/drivers/scsi/qla2xxx/qla_gbl.h index e665e8109933..d48dea8fab1b 100644 --- a/drivers/scsi/qla2xxx/qla_gbl.h +++ b/drivers/scsi/qla2xxx/qla_gbl.h @@ -1,6 +1,6 @@ /* * QLogic Fibre Channel HBA Driver - * Copyright (c) 2003-2013 QLogic Corporation + * Copyright (c) 2003-2014 QLogic Corporation * * See LICENSE.qla2xxx for copyright and licensing details. */ @@ -220,6 +220,13 @@ extern unsigned long qla2x00_get_async_timeout(struct scsi_qla_host *); extern void *qla2x00_alloc_iocbs(scsi_qla_host_t *, srb_t *); extern int qla2x00_issue_marker(scsi_qla_host_t *, int); +extern int qla24xx_walk_and_build_sglist_no_difb(struct qla_hw_data *, srb_t *, + uint32_t *, uint16_t, struct qla_tgt_cmd *); +extern int qla24xx_walk_and_build_sglist(struct qla_hw_data *, srb_t *, + uint32_t *, uint16_t, struct qla_tgt_cmd *); +extern int qla24xx_walk_and_build_prot_sglist(struct qla_hw_data *, srb_t *, + uint32_t *, uint16_t, struct qla_tgt_cmd *); + /* * Global Function Prototypes in qla_mbx.c source file. @@ -347,6 +354,11 @@ extern int qla2x00_read_serdes_word(scsi_qla_host_t *, uint16_t, uint16_t *); extern int +qla8044_write_serdes_word(scsi_qla_host_t *, uint32_t, uint32_t); +extern int +qla8044_read_serdes_word(scsi_qla_host_t *, uint32_t, uint32_t *); + +extern int qla2x00_set_serdes_params(scsi_qla_host_t *, uint16_t, uint16_t, uint16_t); extern int diff --git a/drivers/scsi/qla2xxx/qla_gs.c b/drivers/scsi/qla2xxx/qla_gs.c index e377f9d2f92a..a0df3b1b3823 100644 --- a/drivers/scsi/qla2xxx/qla_gs.c +++ b/drivers/scsi/qla2xxx/qla_gs.c @@ -1,6 +1,6 @@ /* * QLogic Fibre Channel HBA Driver - * Copyright (c) 2003-2013 QLogic Corporation + * Copyright (c) 2003-2014 QLogic Corporation * * See LICENSE.qla2xxx for copyright and licensing details. */ diff --git a/drivers/scsi/qla2xxx/qla_init.c b/drivers/scsi/qla2xxx/qla_init.c index 38aeb54cd9d8..e2184412617d 100644 --- a/drivers/scsi/qla2xxx/qla_init.c +++ b/drivers/scsi/qla2xxx/qla_init.c @@ -1,6 +1,6 @@ /* * QLogic Fibre Channel HBA Driver - * Copyright (c) 2003-2013 QLogic Corporation + * Copyright (c) 2003-2014 QLogic Corporation * * See LICENSE.qla2xxx for copyright and licensing details. */ @@ -1476,6 +1476,7 @@ qla2x00_alloc_fw_dump(scsi_qla_host_t *vha) } ha->fw_dumped = 0; + ha->fw_dump_cap_flags = 0; dump_size = fixed_size = mem_size = eft_size = fce_size = mq_size = 0; req_q_size = rsp_q_size = 0; @@ -2061,6 +2062,10 @@ qla24xx_config_rings(struct scsi_qla_host *vha) icb->atio_q_address[0] = cpu_to_le32(LSD(ha->tgt.atio_dma)); icb->atio_q_address[1] = cpu_to_le32(MSD(ha->tgt.atio_dma)); + if (IS_SHADOW_REG_CAPABLE(ha)) + icb->firmware_options_2 |= + __constant_cpu_to_le32(BIT_30|BIT_29); + if (ha->mqenable || IS_QLA83XX(ha) || IS_QLA27XX(ha)) { icb->qos = __constant_cpu_to_le16(QLA_DEFAULT_QUE_QOS); icb->rid = __constant_cpu_to_le16(rid); @@ -2138,6 +2143,8 @@ qla2x00_init_rings(scsi_qla_host_t *vha) req = ha->req_q_map[que]; if (!req) continue; + req->out_ptr = (void *)(req->ring + req->length); + *req->out_ptr = 0; for (cnt = 1; cnt < req->num_outstanding_cmds; cnt++) req->outstanding_cmds[cnt] = NULL; @@ -2153,6 +2160,8 @@ qla2x00_init_rings(scsi_qla_host_t *vha) rsp = ha->rsp_q_map[que]; if (!rsp) continue; + rsp->in_ptr = (void *)(rsp->ring + rsp->length); + *rsp->in_ptr = 0; /* Initialize response queue entries */ if (IS_QLAFX00(ha)) qlafx00_init_response_q_entries(rsp); @@ -3406,7 +3415,7 @@ qla2x00_configure_fabric(scsi_qla_host_t *vha) fcport->d_id.b.domain, fcport->d_id.b.area, fcport->d_id.b.al_pa); - fcport->loop_id = FC_NO_LOOP_ID; + qla2x00_clear_loop_id(fcport); } } } @@ -4727,7 +4736,6 @@ static int qla2x00_restart_isp(scsi_qla_host_t *vha) { int status = 0; - uint32_t wait_time; struct qla_hw_data *ha = vha->hw; struct req_que *req = ha->req_q_map[0]; struct rsp_que *rsp = ha->rsp_q_map[0]; @@ -4744,14 +4752,12 @@ qla2x00_restart_isp(scsi_qla_host_t *vha) if (!status && !(status = qla2x00_init_rings(vha))) { clear_bit(RESET_MARKER_NEEDED, &vha->dpc_flags); ha->flags.chip_reset_done = 1; + /* Initialize the queues in use */ qla25xx_init_queues(ha); status = qla2x00_fw_ready(vha); if (!status) { - ql_dbg(ql_dbg_taskm, vha, 0x8031, - "Start configure loop status = %d.\n", status); - /* Issue a marker after FW becomes ready. */ qla2x00_marker(vha, req, rsp, 0, 0, MK_SYNC_ALL); @@ -4766,24 +4772,12 @@ qla2x00_restart_isp(scsi_qla_host_t *vha) qlt_24xx_process_atio_queue(vha); spin_unlock_irqrestore(&ha->hardware_lock, flags); - /* Wait at most MAX_TARGET RSCNs for a stable link. */ - wait_time = 256; - do { - clear_bit(LOOP_RESYNC_NEEDED, &vha->dpc_flags); - qla2x00_configure_loop(vha); - wait_time--; - } while (!atomic_read(&vha->loop_down_timer) && - !(test_bit(ISP_ABORT_NEEDED, &vha->dpc_flags)) - && wait_time && (test_bit(LOOP_RESYNC_NEEDED, - &vha->dpc_flags))); + set_bit(LOOP_RESYNC_NEEDED, &vha->dpc_flags); } /* if no cable then assume it's good */ if ((vha->device_flags & DFLG_NO_CABLE)) status = 0; - - ql_dbg(ql_dbg_taskm, vha, 0x8032, - "Configure loop done, status = 0x%x.\n", status); } return (status); } @@ -6130,7 +6124,6 @@ int qla82xx_restart_isp(scsi_qla_host_t *vha) { int status, rval; - uint32_t wait_time; struct qla_hw_data *ha = vha->hw; struct req_que *req = ha->req_q_map[0]; struct rsp_que *rsp = ha->rsp_q_map[0]; @@ -6144,31 +6137,15 @@ qla82xx_restart_isp(scsi_qla_host_t *vha) status = qla2x00_fw_ready(vha); if (!status) { - ql_log(ql_log_info, vha, 0x803c, - "Start configure loop, status =%d.\n", status); - /* Issue a marker after FW becomes ready. */ qla2x00_marker(vha, req, rsp, 0, 0, MK_SYNC_ALL); - vha->flags.online = 1; - /* Wait at most MAX_TARGET RSCNs for a stable link. */ - wait_time = 256; - do { - clear_bit(LOOP_RESYNC_NEEDED, &vha->dpc_flags); - qla2x00_configure_loop(vha); - wait_time--; - } while (!atomic_read(&vha->loop_down_timer) && - !(test_bit(ISP_ABORT_NEEDED, &vha->dpc_flags)) && - wait_time && - (test_bit(LOOP_RESYNC_NEEDED, &vha->dpc_flags))); + set_bit(LOOP_RESYNC_NEEDED, &vha->dpc_flags); } /* if no cable then assume it's good */ if ((vha->device_flags & DFLG_NO_CABLE)) status = 0; - - ql_log(ql_log_info, vha, 0x8000, - "Configure loop done, status = 0x%x.\n", status); } if (!status) { @@ -6182,8 +6159,6 @@ qla82xx_restart_isp(scsi_qla_host_t *vha) vha->marker_needed = 1; } - vha->flags.online = 1; - ha->isp_ops->enable_intrs(ha); ha->isp_abort_cnt = 0; diff --git a/drivers/scsi/qla2xxx/qla_inline.h b/drivers/scsi/qla2xxx/qla_inline.h index ce8b5fb0f347..b3b1d6fc2d6c 100644 --- a/drivers/scsi/qla2xxx/qla_inline.h +++ b/drivers/scsi/qla2xxx/qla_inline.h @@ -1,10 +1,11 @@ /* * QLogic Fibre Channel HBA Driver - * Copyright (c) 2003-2013 QLogic Corporation + * Copyright (c) 2003-2014 QLogic Corporation * * See LICENSE.qla2xxx for copyright and licensing details. */ +#include "qla_target.h" /** * qla24xx_calc_iocbs() - Determine number of Command Type 3 and * Continuation Type 1 IOCBs to allocate. @@ -128,12 +129,20 @@ qla2x00_clear_loop_id(fc_port_t *fcport) { } static inline void -qla2x00_clean_dsd_pool(struct qla_hw_data *ha, srb_t *sp) +qla2x00_clean_dsd_pool(struct qla_hw_data *ha, srb_t *sp, + struct qla_tgt_cmd *tc) { struct dsd_dma *dsd_ptr, *tdsd_ptr; struct crc_context *ctx; - ctx = (struct crc_context *)GET_CMD_CTX_SP(sp); + if (sp) + ctx = (struct crc_context *)GET_CMD_CTX_SP(sp); + else if (tc) + ctx = (struct crc_context *)tc->ctx; + else { + BUG(); + return; + } /* clean up allocated prev pool */ list_for_each_entry_safe(dsd_ptr, tdsd_ptr, diff --git a/drivers/scsi/qla2xxx/qla_iocb.c b/drivers/scsi/qla2xxx/qla_iocb.c index e607568bce49..760931529592 100644 --- a/drivers/scsi/qla2xxx/qla_iocb.c +++ b/drivers/scsi/qla2xxx/qla_iocb.c @@ -1,6 +1,6 @@ /* * QLogic Fibre Channel HBA Driver - * Copyright (c) 2003-2013 QLogic Corporation + * Copyright (c) 2003-2014 QLogic Corporation * * See LICENSE.qla2xxx for copyright and licensing details. */ @@ -936,9 +936,9 @@ qla24xx_get_one_block_sg(uint32_t blk_sz, struct qla2_sgx *sgx, return 1; } -static int +int qla24xx_walk_and_build_sglist_no_difb(struct qla_hw_data *ha, srb_t *sp, - uint32_t *dsd, uint16_t tot_dsds) + uint32_t *dsd, uint16_t tot_dsds, struct qla_tgt_cmd *tc) { void *next_dsd; uint8_t avail_dsds = 0; @@ -948,21 +948,35 @@ qla24xx_walk_and_build_sglist_no_difb(struct qla_hw_data *ha, srb_t *sp, uint32_t *cur_dsd = dsd; uint16_t used_dsds = tot_dsds; - uint32_t prot_int; + uint32_t prot_int; /* protection interval */ uint32_t partial; struct qla2_sgx sgx; dma_addr_t sle_dma; uint32_t sle_dma_len, tot_prot_dma_len = 0; - struct scsi_cmnd *cmd = GET_CMD_SP(sp); - - prot_int = cmd->device->sector_size; + struct scsi_cmnd *cmd; + struct scsi_qla_host *vha; memset(&sgx, 0, sizeof(struct qla2_sgx)); - sgx.tot_bytes = scsi_bufflen(cmd); - sgx.cur_sg = scsi_sglist(cmd); - sgx.sp = sp; - - sg_prot = scsi_prot_sglist(cmd); + if (sp) { + vha = sp->fcport->vha; + cmd = GET_CMD_SP(sp); + prot_int = cmd->device->sector_size; + + sgx.tot_bytes = scsi_bufflen(cmd); + sgx.cur_sg = scsi_sglist(cmd); + sgx.sp = sp; + + sg_prot = scsi_prot_sglist(cmd); + } else if (tc) { + vha = tc->vha; + prot_int = tc->blk_sz; + sgx.tot_bytes = tc->bufflen; + sgx.cur_sg = tc->sg; + sg_prot = tc->prot_sg; + } else { + BUG(); + return 1; + } while (qla24xx_get_one_block_sg(prot_int, &sgx, &partial)) { @@ -995,10 +1009,18 @@ alloc_and_fill: return 1; } - list_add_tail(&dsd_ptr->list, - &((struct crc_context *)sp->u.scmd.ctx)->dsd_list); + if (sp) { + list_add_tail(&dsd_ptr->list, + &((struct crc_context *) + sp->u.scmd.ctx)->dsd_list); + + sp->flags |= SRB_CRC_CTX_DSD_VALID; + } else { + list_add_tail(&dsd_ptr->list, + &(tc->ctx->dsd_list)); + tc->ctx_dsd_alloced = 1; + } - sp->flags |= SRB_CRC_CTX_DSD_VALID; /* add new list to cmd iocb or last list */ *cur_dsd++ = cpu_to_le32(LSD(dsd_ptr->dsd_list_dma)); @@ -1033,21 +1055,35 @@ alloc_and_fill: return 0; } -static int +int qla24xx_walk_and_build_sglist(struct qla_hw_data *ha, srb_t *sp, uint32_t *dsd, - uint16_t tot_dsds) + uint16_t tot_dsds, struct qla_tgt_cmd *tc) { void *next_dsd; uint8_t avail_dsds = 0; uint32_t dsd_list_len; struct dsd_dma *dsd_ptr; - struct scatterlist *sg; + struct scatterlist *sg, *sgl; uint32_t *cur_dsd = dsd; int i; uint16_t used_dsds = tot_dsds; - struct scsi_cmnd *cmd = GET_CMD_SP(sp); + struct scsi_cmnd *cmd; + struct scsi_qla_host *vha; + + if (sp) { + cmd = GET_CMD_SP(sp); + sgl = scsi_sglist(cmd); + vha = sp->fcport->vha; + } else if (tc) { + sgl = tc->sg; + vha = tc->vha; + } else { + BUG(); + return 1; + } - scsi_for_each_sg(cmd, sg, tot_dsds, i) { + + for_each_sg(sgl, sg, tot_dsds, i) { dma_addr_t sle_dma; /* Allocate additional continuation packets? */ @@ -1076,10 +1112,17 @@ qla24xx_walk_and_build_sglist(struct qla_hw_data *ha, srb_t *sp, uint32_t *dsd, return 1; } - list_add_tail(&dsd_ptr->list, - &((struct crc_context *)sp->u.scmd.ctx)->dsd_list); + if (sp) { + list_add_tail(&dsd_ptr->list, + &((struct crc_context *) + sp->u.scmd.ctx)->dsd_list); - sp->flags |= SRB_CRC_CTX_DSD_VALID; + sp->flags |= SRB_CRC_CTX_DSD_VALID; + } else { + list_add_tail(&dsd_ptr->list, + &(tc->ctx->dsd_list)); + tc->ctx_dsd_alloced = 1; + } /* add new list to cmd iocb or last list */ *cur_dsd++ = cpu_to_le32(LSD(dsd_ptr->dsd_list_dma)); @@ -1102,23 +1145,37 @@ qla24xx_walk_and_build_sglist(struct qla_hw_data *ha, srb_t *sp, uint32_t *dsd, return 0; } -static int +int qla24xx_walk_and_build_prot_sglist(struct qla_hw_data *ha, srb_t *sp, - uint32_t *dsd, - uint16_t tot_dsds) + uint32_t *dsd, uint16_t tot_dsds, struct qla_tgt_cmd *tc) { void *next_dsd; uint8_t avail_dsds = 0; uint32_t dsd_list_len; struct dsd_dma *dsd_ptr; - struct scatterlist *sg; + struct scatterlist *sg, *sgl; int i; struct scsi_cmnd *cmd; uint32_t *cur_dsd = dsd; - uint16_t used_dsds = tot_dsds; + uint16_t used_dsds = tot_dsds; + struct scsi_qla_host *vha; + + if (sp) { + cmd = GET_CMD_SP(sp); + sgl = scsi_prot_sglist(cmd); + vha = sp->fcport->vha; + } else if (tc) { + vha = tc->vha; + sgl = tc->prot_sg; + } else { + BUG(); + return 1; + } - cmd = GET_CMD_SP(sp); - scsi_for_each_prot_sg(cmd, sg, tot_dsds, i) { + ql_dbg(ql_dbg_tgt, vha, 0xe021, + "%s: enter\n", __func__); + + for_each_sg(sgl, sg, tot_dsds, i) { dma_addr_t sle_dma; /* Allocate additional continuation packets? */ @@ -1147,10 +1204,17 @@ qla24xx_walk_and_build_prot_sglist(struct qla_hw_data *ha, srb_t *sp, return 1; } - list_add_tail(&dsd_ptr->list, - &((struct crc_context *)sp->u.scmd.ctx)->dsd_list); + if (sp) { + list_add_tail(&dsd_ptr->list, + &((struct crc_context *) + sp->u.scmd.ctx)->dsd_list); - sp->flags |= SRB_CRC_CTX_DSD_VALID; + sp->flags |= SRB_CRC_CTX_DSD_VALID; + } else { + list_add_tail(&dsd_ptr->list, + &(tc->ctx->dsd_list)); + tc->ctx_dsd_alloced = 1; + } /* add new list to cmd iocb or last list */ *cur_dsd++ = cpu_to_le32(LSD(dsd_ptr->dsd_list_dma)); @@ -1386,10 +1450,10 @@ qla24xx_build_scsi_crc_2_iocbs(srb_t *sp, struct cmd_type_crc_2 *cmd_pkt, if (!bundling && tot_prot_dsds) { if (qla24xx_walk_and_build_sglist_no_difb(ha, sp, - cur_dsd, tot_dsds)) + cur_dsd, tot_dsds, NULL)) goto crc_queuing_error; } else if (qla24xx_walk_and_build_sglist(ha, sp, cur_dsd, - (tot_dsds - tot_prot_dsds))) + (tot_dsds - tot_prot_dsds), NULL)) goto crc_queuing_error; if (bundling && tot_prot_dsds) { @@ -1398,7 +1462,7 @@ qla24xx_build_scsi_crc_2_iocbs(srb_t *sp, struct cmd_type_crc_2 *cmd_pkt, __constant_cpu_to_le16(CF_DIF_SEG_DESCR_ENABLE); cur_dsd = (uint32_t *) &crc_ctx_pkt->u.bundling.dif_address; if (qla24xx_walk_and_build_prot_sglist(ha, sp, cur_dsd, - tot_prot_dsds)) + tot_prot_dsds, NULL)) goto crc_queuing_error; } return QLA_SUCCESS; @@ -1478,8 +1542,8 @@ qla24xx_start_scsi(srb_t *sp) tot_dsds = nseg; req_cnt = qla24xx_calc_iocbs(vha, tot_dsds); if (req->cnt < (req_cnt + 2)) { - cnt = RD_REG_DWORD_RELAXED(req->req_q_out); - + cnt = IS_SHADOW_REG_CAPABLE(ha) ? *req->out_ptr : + RD_REG_DWORD_RELAXED(req->req_q_out); if (req->ring_index < cnt) req->cnt = cnt - req->ring_index; else @@ -1697,8 +1761,8 @@ qla24xx_dif_start_scsi(srb_t *sp) tot_prot_dsds = nseg; tot_dsds += nseg; if (req->cnt < (req_cnt + 2)) { - cnt = RD_REG_DWORD_RELAXED(req->req_q_out); - + cnt = IS_SHADOW_REG_CAPABLE(ha) ? *req->out_ptr : + RD_REG_DWORD_RELAXED(req->req_q_out); if (req->ring_index < cnt) req->cnt = cnt - req->ring_index; else @@ -2825,8 +2889,8 @@ qla2x00_start_bidir(srb_t *sp, struct scsi_qla_host *vha, uint32_t tot_dsds) /* Check for room on request queue. */ if (req->cnt < req_cnt + 2) { - cnt = RD_REG_DWORD_RELAXED(req->req_q_out); - + cnt = IS_SHADOW_REG_CAPABLE(ha) ? *req->out_ptr : + RD_REG_DWORD_RELAXED(req->req_q_out); if (req->ring_index < cnt) req->cnt = cnt - req->ring_index; else diff --git a/drivers/scsi/qla2xxx/qla_isr.c b/drivers/scsi/qla2xxx/qla_isr.c index 95314ef2e505..a56825c73c31 100644 --- a/drivers/scsi/qla2xxx/qla_isr.c +++ b/drivers/scsi/qla2xxx/qla_isr.c @@ -1,6 +1,6 @@ /* * QLogic Fibre Channel HBA Driver - * Copyright (c) 2003-2013 QLogic Corporation + * Copyright (c) 2003-2014 QLogic Corporation * * See LICENSE.qla2xxx for copyright and licensing details. */ @@ -2009,11 +2009,13 @@ qla2x00_status_entry(scsi_qla_host_t *vha, struct rsp_que *rsp, void *pkt) ql_dbg(ql_dbg_io, vha, 0x3017, "Invalid status handle (0x%x).\n", sts->handle); - if (IS_P3P_TYPE(ha)) - set_bit(FCOE_CTX_RESET_NEEDED, &vha->dpc_flags); - else - set_bit(ISP_ABORT_NEEDED, &vha->dpc_flags); - qla2xxx_wake_dpc(vha); + if (!test_bit(ABORT_ISP_ACTIVE, &vha->dpc_flags)) { + if (IS_P3P_TYPE(ha)) + set_bit(FCOE_CTX_RESET_NEEDED, &vha->dpc_flags); + else + set_bit(ISP_ABORT_NEEDED, &vha->dpc_flags); + qla2xxx_wake_dpc(vha); + } return; } @@ -2472,12 +2474,14 @@ void qla24xx_process_response_queue(struct scsi_qla_host *vha, if (pkt->entry_status != 0) { qla2x00_error_entry(vha, rsp, (sts_entry_t *) pkt); - (void)qlt_24xx_process_response_error(vha, pkt); + if (qlt_24xx_process_response_error(vha, pkt)) + goto process_err; ((response_t *)pkt)->signature = RESPONSE_PROCESSED; wmb(); continue; } +process_err: switch (pkt->entry_type) { case STATUS_TYPE: @@ -2494,10 +2498,10 @@ void qla24xx_process_response_queue(struct scsi_qla_host *vha, qla24xx_logio_entry(vha, rsp->req, (struct logio_entry_24xx *)pkt); break; - case CT_IOCB_TYPE: + case CT_IOCB_TYPE: qla24xx_els_ct_entry(vha, rsp->req, pkt, CT_IOCB_TYPE); break; - case ELS_IOCB_TYPE: + case ELS_IOCB_TYPE: qla24xx_els_ct_entry(vha, rsp->req, pkt, ELS_IOCB_TYPE); break; case ABTS_RECV_24XX: @@ -2506,6 +2510,7 @@ void qla24xx_process_response_queue(struct scsi_qla_host *vha, case ABTS_RESP_24XX: case CTIO_TYPE7: case NOTIFY_ACK_TYPE: + case CTIO_CRC2: qlt_response_pkt_all_vps(vha, (response_t *)pkt); break; case MARKER_TYPE: diff --git a/drivers/scsi/qla2xxx/qla_mbx.c b/drivers/scsi/qla2xxx/qla_mbx.c index 2528709c4add..1c33a77db5c2 100644 --- a/drivers/scsi/qla2xxx/qla_mbx.c +++ b/drivers/scsi/qla2xxx/qla_mbx.c @@ -1,6 +1,6 @@ /* * QLogic Fibre Channel HBA Driver - * Copyright (c) 2003-2013 QLogic Corporation + * Copyright (c) 2003-2014 QLogic Corporation * * See LICENSE.qla2xxx for copyright and licensing details. */ @@ -1319,7 +1319,7 @@ qla2x00_get_node_name_list(scsi_qla_host_t *vha, void **out_data, int *out_len) left = 0; - list = kzalloc(dma_size, GFP_KERNEL); + list = kmemdup(pmap, dma_size, GFP_KERNEL); if (!list) { ql_log(ql_log_warn, vha, 0x1140, "%s(%ld): failed to allocate node names list " @@ -1328,7 +1328,6 @@ qla2x00_get_node_name_list(scsi_qla_host_t *vha, void **out_data, int *out_len) goto out_free; } - memcpy(list, pmap, dma_size); restart: dma_free_coherent(&ha->pdev->dev, dma_size, pmap, pmap_dma); } @@ -2644,7 +2643,10 @@ qla24xx_abort_command(srb_t *sp) ql_dbg(ql_dbg_mbx, vha, 0x1090, "Failed to complete IOCB -- completion status (%x).\n", le16_to_cpu(abt->nport_handle)); - rval = QLA_FUNCTION_FAILED; + if (abt->nport_handle == CS_IOCB_ERROR) + rval = QLA_FUNCTION_PARAMETER_ERROR; + else + rval = QLA_FUNCTION_FAILED; } else { ql_dbg(ql_dbg_mbx + ql_dbg_verbose, vha, 0x1091, "Done %s.\n", __func__); @@ -2879,6 +2881,78 @@ qla2x00_read_serdes_word(scsi_qla_host_t *vha, uint16_t addr, uint16_t *data) return rval; } +int +qla8044_write_serdes_word(scsi_qla_host_t *vha, uint32_t addr, uint32_t data) +{ + int rval; + mbx_cmd_t mc; + mbx_cmd_t *mcp = &mc; + + if (!IS_QLA8044(vha->hw)) + return QLA_FUNCTION_FAILED; + + ql_dbg(ql_dbg_mbx + ql_dbg_verbose, vha, 0x1186, + "Entered %s.\n", __func__); + + mcp->mb[0] = MBC_SET_GET_ETH_SERDES_REG; + mcp->mb[1] = HCS_WRITE_SERDES; + mcp->mb[3] = LSW(addr); + mcp->mb[4] = MSW(addr); + mcp->mb[5] = LSW(data); + mcp->mb[6] = MSW(data); + mcp->out_mb = MBX_6|MBX_5|MBX_4|MBX_3|MBX_1|MBX_0; + mcp->in_mb = MBX_0; + mcp->tov = MBX_TOV_SECONDS; + mcp->flags = 0; + rval = qla2x00_mailbox_command(vha, mcp); + + if (rval != QLA_SUCCESS) { + ql_dbg(ql_dbg_mbx, vha, 0x1187, + "Failed=%x mb[0]=%x.\n", rval, mcp->mb[0]); + } else { + ql_dbg(ql_dbg_mbx + ql_dbg_verbose, vha, 0x1188, + "Done %s.\n", __func__); + } + + return rval; +} + +int +qla8044_read_serdes_word(scsi_qla_host_t *vha, uint32_t addr, uint32_t *data) +{ + int rval; + mbx_cmd_t mc; + mbx_cmd_t *mcp = &mc; + + if (!IS_QLA8044(vha->hw)) + return QLA_FUNCTION_FAILED; + + ql_dbg(ql_dbg_mbx + ql_dbg_verbose, vha, 0x1189, + "Entered %s.\n", __func__); + + mcp->mb[0] = MBC_SET_GET_ETH_SERDES_REG; + mcp->mb[1] = HCS_READ_SERDES; + mcp->mb[3] = LSW(addr); + mcp->mb[4] = MSW(addr); + mcp->out_mb = MBX_4|MBX_3|MBX_1|MBX_0; + mcp->in_mb = MBX_2|MBX_1|MBX_0; + mcp->tov = MBX_TOV_SECONDS; + mcp->flags = 0; + rval = qla2x00_mailbox_command(vha, mcp); + + *data = mcp->mb[2] << 16 | mcp->mb[1]; + + if (rval != QLA_SUCCESS) { + ql_dbg(ql_dbg_mbx, vha, 0x118a, + "Failed=%x mb[0]=%x.\n", rval, mcp->mb[0]); + } else { + ql_dbg(ql_dbg_mbx + ql_dbg_verbose, vha, 0x118b, + "Done %s.\n", __func__); + } + + return rval; +} + /** * qla2x00_set_serdes_params() - * @ha: HA context @@ -3660,6 +3734,9 @@ qla25xx_init_req_que(struct scsi_qla_host *vha, struct req_que *req) ql_dbg(ql_dbg_mbx + ql_dbg_verbose, vha, 0x10d3, "Entered %s.\n", __func__); + if (IS_SHADOW_REG_CAPABLE(ha)) + req->options |= BIT_13; + mcp->mb[0] = MBC_INITIALIZE_MULTIQ; mcp->mb[1] = req->options; mcp->mb[2] = MSW(LSD(req->dma)); @@ -3679,7 +3756,7 @@ qla25xx_init_req_que(struct scsi_qla_host *vha, struct req_que *req) /* que in ptr index */ mcp->mb[8] = 0; /* que out ptr index */ - mcp->mb[9] = 0; + mcp->mb[9] = *req->out_ptr = 0; mcp->out_mb = MBX_14|MBX_13|MBX_12|MBX_11|MBX_10|MBX_9|MBX_8|MBX_7| MBX_6|MBX_5|MBX_4|MBX_3|MBX_2|MBX_1|MBX_0; mcp->in_mb = MBX_0; @@ -3688,7 +3765,7 @@ qla25xx_init_req_que(struct scsi_qla_host *vha, struct req_que *req) if (IS_QLA81XX(ha) || IS_QLA83XX(ha) || IS_QLA27XX(ha)) mcp->in_mb |= MBX_1; - if (IS_QLA83XX(ha) || !IS_QLA27XX(ha)) { + if (IS_QLA83XX(ha) || IS_QLA27XX(ha)) { mcp->out_mb |= MBX_15; /* debug q create issue in SR-IOV */ mcp->in_mb |= MBX_9 | MBX_8 | MBX_7; @@ -3697,7 +3774,7 @@ qla25xx_init_req_que(struct scsi_qla_host *vha, struct req_que *req) spin_lock_irqsave(&ha->hardware_lock, flags); if (!(req->options & BIT_0)) { WRT_REG_DWORD(req->req_q_in, 0); - if (!IS_QLA83XX(ha) || !IS_QLA27XX(ha)) + if (!IS_QLA83XX(ha) && !IS_QLA27XX(ha)) WRT_REG_DWORD(req->req_q_out, 0); } spin_unlock_irqrestore(&ha->hardware_lock, flags); @@ -3726,6 +3803,9 @@ qla25xx_init_rsp_que(struct scsi_qla_host *vha, struct rsp_que *rsp) ql_dbg(ql_dbg_mbx + ql_dbg_verbose, vha, 0x10d6, "Entered %s.\n", __func__); + if (IS_SHADOW_REG_CAPABLE(ha)) + rsp->options |= BIT_13; + mcp->mb[0] = MBC_INITIALIZE_MULTIQ; mcp->mb[1] = rsp->options; mcp->mb[2] = MSW(LSD(rsp->dma)); @@ -3740,7 +3820,7 @@ qla25xx_init_rsp_que(struct scsi_qla_host *vha, struct rsp_que *rsp) mcp->mb[4] = rsp->id; /* que in ptr index */ - mcp->mb[8] = 0; + mcp->mb[8] = *rsp->in_ptr = 0; /* que out ptr index */ mcp->mb[9] = 0; mcp->out_mb = MBX_14|MBX_13|MBX_9|MBX_8|MBX_7 diff --git a/drivers/scsi/qla2xxx/qla_mid.c b/drivers/scsi/qla2xxx/qla_mid.c index f0a852257f99..89998244f48d 100644 --- a/drivers/scsi/qla2xxx/qla_mid.c +++ b/drivers/scsi/qla2xxx/qla_mid.c @@ -1,6 +1,6 @@ /* * QLogic Fibre Channel HBA Driver - * Copyright (c) 2003-2013 QLogic Corporation + * Copyright (c) 2003-2014 QLogic Corporation * * See LICENSE.qla2xxx for copyright and licensing details. */ diff --git a/drivers/scsi/qla2xxx/qla_mr.c b/drivers/scsi/qla2xxx/qla_mr.c index 0aaf6a9c87d3..abeb3901498b 100644 --- a/drivers/scsi/qla2xxx/qla_mr.c +++ b/drivers/scsi/qla2xxx/qla_mr.c @@ -1,6 +1,6 @@ /* * QLogic Fibre Channel HBA Driver - * Copyright (c) 2003-2013 QLogic Corporation + * Copyright (c) 2003-2014 QLogic Corporation * * See LICENSE.qla2xxx for copyright and licensing details. */ @@ -527,21 +527,63 @@ qlafx00_soc_cpu_reset(scsi_qla_host_t *vha) struct qla_hw_data *ha = vha->hw; int i, core; uint32_t cnt; + uint32_t reg_val; + + spin_lock_irqsave(&ha->hardware_lock, flags); + + QLAFX00_SET_HBA_SOC_REG(ha, 0x80004, 0); + QLAFX00_SET_HBA_SOC_REG(ha, 0x82004, 0); + + /* stop the XOR DMA engines */ + QLAFX00_SET_HBA_SOC_REG(ha, 0x60920, 0x02); + QLAFX00_SET_HBA_SOC_REG(ha, 0x60924, 0x02); + QLAFX00_SET_HBA_SOC_REG(ha, 0xf0920, 0x02); + QLAFX00_SET_HBA_SOC_REG(ha, 0xf0924, 0x02); + + /* stop the IDMA engines */ + reg_val = QLAFX00_GET_HBA_SOC_REG(ha, 0x60840); + reg_val &= ~(1<<12); + QLAFX00_SET_HBA_SOC_REG(ha, 0x60840, reg_val); + + reg_val = QLAFX00_GET_HBA_SOC_REG(ha, 0x60844); + reg_val &= ~(1<<12); + QLAFX00_SET_HBA_SOC_REG(ha, 0x60844, reg_val); + + reg_val = QLAFX00_GET_HBA_SOC_REG(ha, 0x60848); + reg_val &= ~(1<<12); + QLAFX00_SET_HBA_SOC_REG(ha, 0x60848, reg_val); + + reg_val = QLAFX00_GET_HBA_SOC_REG(ha, 0x6084C); + reg_val &= ~(1<<12); + QLAFX00_SET_HBA_SOC_REG(ha, 0x6084C, reg_val); + + for (i = 0; i < 100000; i++) { + if ((QLAFX00_GET_HBA_SOC_REG(ha, 0xd0000) & 0x10000000) == 0 && + (QLAFX00_GET_HBA_SOC_REG(ha, 0x10600) & 0x1) == 0) + break; + udelay(100); + } /* Set all 4 cores in reset */ for (i = 0; i < 4; i++) { QLAFX00_SET_HBA_SOC_REG(ha, (SOC_SW_RST_CONTROL_REG_CORE0 + 8*i), (0xF01)); - } - - /* Set all 4 core Clock gating control */ - for (i = 0; i < 4; i++) { QLAFX00_SET_HBA_SOC_REG(ha, (SOC_SW_RST_CONTROL_REG_CORE0 + 4 + 8*i), (0x01010101)); } /* Reset all units in Fabric */ - QLAFX00_SET_HBA_SOC_REG(ha, SOC_FABRIC_RST_CONTROL_REG, (0x11F0101)); + QLAFX00_SET_HBA_SOC_REG(ha, SOC_FABRIC_RST_CONTROL_REG, (0x011f0101)); + + /* */ + QLAFX00_SET_HBA_SOC_REG(ha, 0x10610, 1); + QLAFX00_SET_HBA_SOC_REG(ha, 0x10600, 0); + + /* Set all 4 core Memory Power Down Registers */ + for (i = 0; i < 5; i++) { + QLAFX00_SET_HBA_SOC_REG(ha, + (SOC_PWR_MANAGEMENT_PWR_DOWN_REG + 4*i), (0x0)); + } /* Reset all interrupt control registers */ for (i = 0; i < 115; i++) { @@ -564,20 +606,19 @@ qlafx00_soc_cpu_reset(scsi_qla_host_t *vha) QLAFX00_SET_HBA_SOC_REG(ha, SOC_FABRIC_CONTROL_REG, (0x2)); QLAFX00_SET_HBA_SOC_REG(ha, SOC_FABRIC_CONFIG_REG, (0x3)); - spin_lock_irqsave(&ha->hardware_lock, flags); - /* Kick in Fabric units */ QLAFX00_SET_HBA_SOC_REG(ha, SOC_FABRIC_RST_CONTROL_REG, (0x0)); /* Kick in Core0 to start boot process */ QLAFX00_SET_HBA_SOC_REG(ha, SOC_SW_RST_CONTROL_REG_CORE0, (0xF00)); + spin_unlock_irqrestore(&ha->hardware_lock, flags); + /* Wait 10secs for soft-reset to complete. */ for (cnt = 10; cnt; cnt--) { msleep(1000); barrier(); } - spin_unlock_irqrestore(&ha->hardware_lock, flags); } /** @@ -597,7 +638,6 @@ qlafx00_soft_reset(scsi_qla_host_t *vha) ha->isp_ops->disable_intrs(ha); qlafx00_soc_cpu_reset(vha); - ha->isp_ops->enable_intrs(ha); } /** @@ -2675,7 +2715,7 @@ qlafx00_process_response_queue(struct scsi_qla_host *vha, uint16_t lreq_q_out = 0; lreq_q_in = RD_REG_DWORD(rsp->rsp_q_in); - lreq_q_out = RD_REG_DWORD(rsp->rsp_q_out); + lreq_q_out = rsp->ring_index; while (lreq_q_in != lreq_q_out) { lptr = rsp->ring_ptr; @@ -3426,7 +3466,7 @@ qlafx00_fxdisc_iocb(srb_t *sp, struct fxdisc_entry_fx00 *pfxiocb) sp->fcport->vha, 0x3047, (uint8_t *)&fx_iocb, sizeof(struct fxdisc_entry_fx00)); - memcpy((void *)pfxiocb, &fx_iocb, + memcpy_toio((void __iomem *)pfxiocb, &fx_iocb, sizeof(struct fxdisc_entry_fx00)); wmb(); } diff --git a/drivers/scsi/qla2xxx/qla_mr.h b/drivers/scsi/qla2xxx/qla_mr.h index e529dfaeb854..aeaa1b40b1fc 100644 --- a/drivers/scsi/qla2xxx/qla_mr.h +++ b/drivers/scsi/qla2xxx/qla_mr.h @@ -1,6 +1,6 @@ /* * QLogic Fibre Channel HBA Driver - * Copyright (c) 2003-2013 QLogic Corporation + * Copyright (c) 2003-2014 QLogic Corporation * * See LICENSE.qla2xxx for copyright and licensing details. */ @@ -351,6 +351,7 @@ struct config_info_data { #define SOC_FABRIC_RST_CONTROL_REG 0x0020840 #define SOC_FABRIC_CONTROL_REG 0x0020200 #define SOC_FABRIC_CONFIG_REG 0x0020204 +#define SOC_PWR_MANAGEMENT_PWR_DOWN_REG 0x001820C #define SOC_INTERRUPT_SOURCE_I_CONTROL_REG 0x0020B00 #define SOC_CORE_TIMER_REG 0x0021850 diff --git a/drivers/scsi/qla2xxx/qla_nx.c b/drivers/scsi/qla2xxx/qla_nx.c index 5511e24b1f11..58f3c912d96e 100644 --- a/drivers/scsi/qla2xxx/qla_nx.c +++ b/drivers/scsi/qla2xxx/qla_nx.c @@ -1,6 +1,6 @@ /* * QLogic Fibre Channel HBA Driver - * Copyright (c) 2003-2013 QLogic Corporation + * Copyright (c) 2003-2014 QLogic Corporation * * See LICENSE.qla2xxx for copyright and licensing details. */ @@ -848,6 +848,7 @@ qla82xx_rom_lock(struct qla_hw_data *ha) { int done = 0, timeout = 0; uint32_t lock_owner = 0; + scsi_qla_host_t *vha = pci_get_drvdata(ha->pdev); while (!done) { /* acquire semaphore2 from PCI HW block */ @@ -856,17 +857,21 @@ qla82xx_rom_lock(struct qla_hw_data *ha) break; if (timeout >= qla82xx_rom_lock_timeout) { lock_owner = qla82xx_rd_32(ha, QLA82XX_ROM_LOCK_ID); + ql_log(ql_log_warn, vha, 0xb157, + "%s: Simultaneous flash access by following ports, active port = %d: accessing port = %d", + __func__, ha->portnum, lock_owner); return -1; } timeout++; } - qla82xx_wr_32(ha, QLA82XX_ROM_LOCK_ID, ROM_LOCK_DRIVER); + qla82xx_wr_32(ha, QLA82XX_ROM_LOCK_ID, ha->portnum); return 0; } static void qla82xx_rom_unlock(struct qla_hw_data *ha) { + qla82xx_wr_32(ha, QLA82XX_ROM_LOCK_ID, 0xffffffff); qla82xx_rd_32(ha, QLA82XX_PCIE_REG(PCIE_SEM2_UNLOCK)); } @@ -950,6 +955,7 @@ static int qla82xx_rom_fast_read(struct qla_hw_data *ha, int addr, int *valp) { int ret, loops = 0; + uint32_t lock_owner = 0; scsi_qla_host_t *vha = pci_get_drvdata(ha->pdev); while ((qla82xx_rom_lock(ha) != 0) && (loops < 50000)) { @@ -958,8 +964,10 @@ qla82xx_rom_fast_read(struct qla_hw_data *ha, int addr, int *valp) loops++; } if (loops >= 50000) { + lock_owner = qla82xx_rd_32(ha, QLA82XX_ROM_LOCK_ID); ql_log(ql_log_fatal, vha, 0x00b9, - "Failed to acquire SEM2 lock.\n"); + "Failed to acquire SEM2 lock, Lock Owner %u.\n", + lock_owner); return -1; } ret = qla82xx_do_rom_fast_read(ha, addr, valp); @@ -1057,6 +1065,7 @@ static int ql82xx_rom_lock_d(struct qla_hw_data *ha) { int loops = 0; + uint32_t lock_owner = 0; scsi_qla_host_t *vha = pci_get_drvdata(ha->pdev); while ((qla82xx_rom_lock(ha) != 0) && (loops < 50000)) { @@ -1065,8 +1074,9 @@ ql82xx_rom_lock_d(struct qla_hw_data *ha) loops++; } if (loops >= 50000) { + lock_owner = qla82xx_rd_32(ha, QLA82XX_ROM_LOCK_ID); ql_log(ql_log_warn, vha, 0xb010, - "ROM lock failed.\n"); + "ROM lock failed, Lock Owner %u.\n", lock_owner); return -1; } return 0; @@ -2811,12 +2821,14 @@ static void qla82xx_rom_lock_recovery(struct qla_hw_data *ha) { scsi_qla_host_t *vha = pci_get_drvdata(ha->pdev); + uint32_t lock_owner = 0; - if (qla82xx_rom_lock(ha)) + if (qla82xx_rom_lock(ha)) { + lock_owner = qla82xx_rd_32(ha, QLA82XX_ROM_LOCK_ID); /* Someone else is holding the lock. */ ql_log(ql_log_info, vha, 0xb022, - "Resetting rom_lock.\n"); - + "Resetting rom_lock, Lock Owner %u.\n", lock_owner); + } /* * Either we got the lock, or someone * else died while holding it. @@ -2840,47 +2852,30 @@ static int qla82xx_device_bootstrap(scsi_qla_host_t *vha) { int rval = QLA_SUCCESS; - int i, timeout; + int i; uint32_t old_count, count; struct qla_hw_data *ha = vha->hw; - int need_reset = 0, peg_stuck = 1; + int need_reset = 0; need_reset = qla82xx_need_reset(ha); - old_count = qla82xx_rd_32(ha, QLA82XX_PEG_ALIVE_COUNTER); - - for (i = 0; i < 10; i++) { - timeout = msleep_interruptible(200); - if (timeout) { - qla82xx_wr_32(ha, QLA82XX_CRB_DEV_STATE, - QLA8XXX_DEV_FAILED); - return QLA_FUNCTION_FAILED; - } - - count = qla82xx_rd_32(ha, QLA82XX_PEG_ALIVE_COUNTER); - if (count != old_count) - peg_stuck = 0; - } - if (need_reset) { /* We are trying to perform a recovery here. */ - if (peg_stuck) + if (ha->flags.isp82xx_fw_hung) qla82xx_rom_lock_recovery(ha); - goto dev_initialize; } else { - /* Start of day for this ha context. */ - if (peg_stuck) { - /* Either we are the first or recovery in progress. */ - qla82xx_rom_lock_recovery(ha); - goto dev_initialize; - } else - /* Firmware already running. */ - goto dev_ready; + old_count = qla82xx_rd_32(ha, QLA82XX_PEG_ALIVE_COUNTER); + for (i = 0; i < 10; i++) { + msleep(200); + count = qla82xx_rd_32(ha, QLA82XX_PEG_ALIVE_COUNTER); + if (count != old_count) { + rval = QLA_SUCCESS; + goto dev_ready; + } + } + qla82xx_rom_lock_recovery(ha); } - return rval; - -dev_initialize: /* set to DEV_INITIALIZING */ ql_log(ql_log_info, vha, 0x009e, "HW State: INITIALIZING.\n"); @@ -3142,18 +3137,18 @@ qla82xx_check_md_needed(scsi_qla_host_t *vha) if (ql2xmdenable) { if (!ha->fw_dumped) { - if (fw_major_version != ha->fw_major_version || + if ((fw_major_version != ha->fw_major_version || fw_minor_version != ha->fw_minor_version || - fw_subminor_version != ha->fw_subminor_version) { + fw_subminor_version != ha->fw_subminor_version) || + (ha->prev_minidump_failed)) { ql_dbg(ql_dbg_p3p, vha, 0xb02d, - "Firmware version differs " - "Previous version: %d:%d:%d - " - "New version: %d:%d:%d\n", + "Firmware version differs Previous version: %d:%d:%d - New version: %d:%d:%d, prev_minidump_failed: %d.\n", fw_major_version, fw_minor_version, fw_subminor_version, ha->fw_major_version, ha->fw_minor_version, - ha->fw_subminor_version); + ha->fw_subminor_version, + ha->prev_minidump_failed); /* Release MiniDump resources */ qla82xx_md_free(vha); /* ALlocate MiniDump resources */ @@ -3682,8 +3677,10 @@ qla82xx_chip_reset_cleanup(scsi_qla_host_t *vha) for (cnt = 1; cnt < req->num_outstanding_cmds; cnt++) { sp = req->outstanding_cmds[cnt]; if (sp) { - if (!sp->u.scmd.ctx || - (sp->flags & SRB_FCP_CMND_DMA_VALID)) { + if ((!sp->u.scmd.ctx || + (sp->flags & + SRB_FCP_CMND_DMA_VALID)) && + !ha->flags.isp82xx_fw_hung) { spin_unlock_irqrestore( &ha->hardware_lock, flags); if (ha->isp_ops->abort_command(sp)) { diff --git a/drivers/scsi/qla2xxx/qla_nx.h b/drivers/scsi/qla2xxx/qla_nx.h index 1bb93dbbccbb..59c477883a73 100644 --- a/drivers/scsi/qla2xxx/qla_nx.h +++ b/drivers/scsi/qla2xxx/qla_nx.h @@ -1,6 +1,6 @@ /* * QLogic Fibre Channel HBA Driver - * Copyright (c) 2003-2013 QLogic Corporation + * Copyright (c) 2003-2014 QLogic Corporation * * See LICENSE.qla2xxx for copyright and licensing details. */ @@ -333,9 +333,6 @@ #define QLA82XX_ROMUSB_ROM_INSTR_OPCODE (ROMUSB_ROM + 0x0004) #define QLA82XX_ROMUSB_GLB_CAS_RST (ROMUSB_GLB + 0x0038) -/* Lock IDs for ROM lock */ -#define ROM_LOCK_DRIVER 0x0d417340 - #define QLA82XX_PCI_CRB_WINDOWSIZE 0x00100000 /* all are 1MB windows */ #define QLA82XX_PCI_CRB_WINDOW(A) \ (QLA82XX_PCI_CRBSPACE + (A)*QLA82XX_PCI_CRB_WINDOWSIZE) @@ -1186,6 +1183,7 @@ static const int MD_MIU_TEST_AGT_RDDATA[] = { 0x410000A8, 0x410000AC, #define CRB_NIU_XG_PAUSE_CTL_P1 0x8 #define qla82xx_get_temp_val(x) ((x) >> 16) +#define qla82xx_get_temp_val1(x) ((x) && 0x0000FFFF) #define qla82xx_get_temp_state(x) ((x) & 0xffff) #define qla82xx_encode_temp(val, state) (((val) << 16) | (state)) diff --git a/drivers/scsi/qla2xxx/qla_nx2.c b/drivers/scsi/qla2xxx/qla_nx2.c index 86cf10815db0..da9e3902f219 100644 --- a/drivers/scsi/qla2xxx/qla_nx2.c +++ b/drivers/scsi/qla2xxx/qla_nx2.c @@ -1,17 +1,20 @@ /* * QLogic Fibre Channel HBA Driver - * Copyright (c) 2003-2013 QLogic Corporation + * Copyright (c) 2003-2014 QLogic Corporation * * See LICENSE.qla2xxx for copyright and licensing details. */ #include <linux/vmalloc.h> +#include <linux/delay.h> #include "qla_def.h" #include "qla_gbl.h" #include <linux/delay.h> +#define TIMEOUT_100_MS 100 + /* 8044 Flash Read/Write functions */ uint32_t qla8044_rd_reg(struct qla_hw_data *ha, ulong addr) @@ -117,6 +120,95 @@ qla8044_read_write_crb_reg(struct scsi_qla_host *vha, qla8044_wr_reg_indirect(vha, waddr, value); } +static int +qla8044_poll_wait_for_ready(struct scsi_qla_host *vha, uint32_t addr1, + uint32_t mask) +{ + unsigned long timeout; + uint32_t temp; + + /* jiffies after 100ms */ + timeout = jiffies + msecs_to_jiffies(TIMEOUT_100_MS); + do { + qla8044_rd_reg_indirect(vha, addr1, &temp); + if ((temp & mask) != 0) + break; + if (time_after_eq(jiffies, timeout)) { + ql_log(ql_log_warn, vha, 0xb151, + "Error in processing rdmdio entry\n"); + return -1; + } + } while (1); + + return 0; +} + +static uint32_t +qla8044_ipmdio_rd_reg(struct scsi_qla_host *vha, + uint32_t addr1, uint32_t addr3, uint32_t mask, uint32_t addr) +{ + uint32_t temp; + int ret = 0; + + ret = qla8044_poll_wait_for_ready(vha, addr1, mask); + if (ret == -1) + return -1; + + temp = (0x40000000 | addr); + qla8044_wr_reg_indirect(vha, addr1, temp); + + ret = qla8044_poll_wait_for_ready(vha, addr1, mask); + if (ret == -1) + return 0; + + qla8044_rd_reg_indirect(vha, addr3, &ret); + + return ret; +} + + +static int +qla8044_poll_wait_ipmdio_bus_idle(struct scsi_qla_host *vha, + uint32_t addr1, uint32_t addr2, uint32_t addr3, uint32_t mask) +{ + unsigned long timeout; + uint32_t temp; + + /* jiffies after 100 msecs */ + timeout = jiffies + msecs_to_jiffies(TIMEOUT_100_MS); + do { + temp = qla8044_ipmdio_rd_reg(vha, addr1, addr3, mask, addr2); + if ((temp & 0x1) != 1) + break; + if (time_after_eq(jiffies, timeout)) { + ql_log(ql_log_warn, vha, 0xb152, + "Error in processing mdiobus idle\n"); + return -1; + } + } while (1); + + return 0; +} + +static int +qla8044_ipmdio_wr_reg(struct scsi_qla_host *vha, uint32_t addr1, + uint32_t addr3, uint32_t mask, uint32_t addr, uint32_t value) +{ + int ret = 0; + + ret = qla8044_poll_wait_for_ready(vha, addr1, mask); + if (ret == -1) + return -1; + + qla8044_wr_reg_indirect(vha, addr3, value); + qla8044_wr_reg_indirect(vha, addr1, addr); + + ret = qla8044_poll_wait_for_ready(vha, addr1, mask); + if (ret == -1) + return -1; + + return 0; +} /* * qla8044_rmw_crb_reg - Read value from raddr, AND with test_mask, * Shift Left,Right/OR/XOR with values RMW header and write value to waddr. @@ -356,8 +448,8 @@ qla8044_flash_lock(scsi_qla_host_t *vha) lock_owner = qla8044_rd_reg(ha, QLA8044_FLASH_LOCK_ID); ql_log(ql_log_warn, vha, 0xb113, - "%s: flash lock by %d failed, held by %d\n", - __func__, ha->portnum, lock_owner); + "%s: Simultaneous flash access by following ports, active port = %d: accessing port = %d", + __func__, ha->portnum, lock_owner); ret_val = QLA_FUNCTION_FAILED; break; } @@ -1541,7 +1633,7 @@ static void qla8044_need_reset_handler(struct scsi_qla_host *vha) { uint32_t dev_state = 0, drv_state, drv_active; - unsigned long reset_timeout, dev_init_timeout; + unsigned long reset_timeout; struct qla_hw_data *ha = vha->hw; ql_log(ql_log_fatal, vha, 0xb0c2, @@ -1555,84 +1647,78 @@ qla8044_need_reset_handler(struct scsi_qla_host *vha) qla8044_idc_lock(ha); } + dev_state = qla8044_rd_direct(vha, + QLA8044_CRB_DEV_STATE_INDEX); drv_state = qla8044_rd_direct(vha, QLA8044_CRB_DRV_STATE_INDEX); drv_active = qla8044_rd_direct(vha, QLA8044_CRB_DRV_ACTIVE_INDEX); ql_log(ql_log_info, vha, 0xb0c5, - "%s(%ld): drv_state = 0x%x, drv_active = 0x%x\n", - __func__, vha->host_no, drv_state, drv_active); + "%s(%ld): drv_state = 0x%x, drv_active = 0x%x dev_state = 0x%x\n", + __func__, vha->host_no, drv_state, drv_active, dev_state); - if (!ha->flags.nic_core_reset_owner) { - ql_dbg(ql_dbg_p3p, vha, 0xb0c3, - "%s(%ld): reset acknowledged\n", - __func__, vha->host_no); - qla8044_set_rst_ready(vha); + qla8044_set_rst_ready(vha); - /* Non-reset owners ACK Reset and wait for device INIT state - * as part of Reset Recovery by Reset Owner - */ - dev_init_timeout = jiffies + (ha->fcoe_reset_timeout * HZ); + /* wait for 10 seconds for reset ack from all functions */ + reset_timeout = jiffies + (ha->fcoe_reset_timeout * HZ); - do { - if (time_after_eq(jiffies, dev_init_timeout)) { - ql_log(ql_log_info, vha, 0xb0c4, - "%s: Non Reset owner: Reset Ack Timeout!\n", - __func__); - break; - } + do { + if (time_after_eq(jiffies, reset_timeout)) { + ql_log(ql_log_info, vha, 0xb0c4, + "%s: Function %d: Reset Ack Timeout!, drv_state: 0x%08x, drv_active: 0x%08x\n", + __func__, ha->portnum, drv_state, drv_active); + break; + } - qla8044_idc_unlock(ha); - msleep(1000); - qla8044_idc_lock(ha); + qla8044_idc_unlock(ha); + msleep(1000); + qla8044_idc_lock(ha); - dev_state = qla8044_rd_direct(vha, - QLA8044_CRB_DEV_STATE_INDEX); - } while (((drv_state & drv_active) != drv_active) && - (dev_state == QLA8XXX_DEV_NEED_RESET)); + dev_state = qla8044_rd_direct(vha, + QLA8044_CRB_DEV_STATE_INDEX); + drv_state = qla8044_rd_direct(vha, + QLA8044_CRB_DRV_STATE_INDEX); + drv_active = qla8044_rd_direct(vha, + QLA8044_CRB_DRV_ACTIVE_INDEX); + } while (((drv_state & drv_active) != drv_active) && + (dev_state == QLA8XXX_DEV_NEED_RESET)); + + /* Remove IDC participation of functions not acknowledging */ + if (drv_state != drv_active) { + ql_log(ql_log_info, vha, 0xb0c7, + "%s(%ld): Function %d turning off drv_active of non-acking function 0x%x\n", + __func__, vha->host_no, ha->portnum, + (drv_active ^ drv_state)); + drv_active = drv_active & drv_state; + qla8044_wr_direct(vha, QLA8044_CRB_DRV_ACTIVE_INDEX, + drv_active); } else { - qla8044_set_rst_ready(vha); - - /* wait for 10 seconds for reset ack from all functions */ - reset_timeout = jiffies + (ha->fcoe_reset_timeout * HZ); - - while ((drv_state & drv_active) != drv_active) { - if (time_after_eq(jiffies, reset_timeout)) { - ql_log(ql_log_info, vha, 0xb0c6, - "%s: RESET TIMEOUT!" - "drv_state: 0x%08x, drv_active: 0x%08x\n", - QLA2XXX_DRIVER_NAME, drv_state, drv_active); - break; - } - - qla8044_idc_unlock(ha); - msleep(1000); - qla8044_idc_lock(ha); - - drv_state = qla8044_rd_direct(vha, - QLA8044_CRB_DRV_STATE_INDEX); - drv_active = qla8044_rd_direct(vha, - QLA8044_CRB_DRV_ACTIVE_INDEX); - } - - if (drv_state != drv_active) { - ql_log(ql_log_info, vha, 0xb0c7, - "%s(%ld): Reset_owner turning off drv_active " - "of non-acking function 0x%x\n", __func__, - vha->host_no, (drv_active ^ drv_state)); - drv_active = drv_active & drv_state; - qla8044_wr_direct(vha, QLA8044_CRB_DRV_ACTIVE_INDEX, - drv_active); + /* + * Reset owner should execute reset recovery, + * if all functions acknowledged + */ + if ((ha->flags.nic_core_reset_owner) && + (dev_state == QLA8XXX_DEV_NEED_RESET)) { + ha->flags.nic_core_reset_owner = 0; + qla8044_device_bootstrap(vha); + return; } + } - /* - * Clear RESET OWNER, will be set at next reset - * by next RST_OWNER - */ + /* Exit if non active function */ + if (!(drv_active & (1 << ha->portnum))) { ha->flags.nic_core_reset_owner = 0; + return; + } - /* Start Reset Recovery */ + /* + * Execute Reset Recovery if Reset Owner or Function 7 + * is the only active function + */ + if (ha->flags.nic_core_reset_owner || + ((drv_state & drv_active) == QLA8044_FUN7_ACTIVE_INDEX)) { + ha->flags.nic_core_reset_owner = 0; qla8044_device_bootstrap(vha); } } @@ -1655,6 +1741,19 @@ qla8044_set_drv_active(struct scsi_qla_host *vha) qla8044_wr_direct(vha, QLA8044_CRB_DRV_ACTIVE_INDEX, drv_active); } +static int +qla8044_check_drv_active(struct scsi_qla_host *vha) +{ + uint32_t drv_active; + struct qla_hw_data *ha = vha->hw; + + drv_active = qla8044_rd_direct(vha, QLA8044_CRB_DRV_ACTIVE_INDEX); + if (drv_active & (1 << ha->portnum)) + return QLA_SUCCESS; + else + return QLA_TEST_FAILED; +} + static void qla8044_clear_idc_dontreset(struct scsi_qla_host *vha) { @@ -1837,14 +1936,16 @@ qla8044_device_state_handler(struct scsi_qla_host *vha) while (1) { if (time_after_eq(jiffies, dev_init_timeout)) { - ql_log(ql_log_warn, vha, 0xb0cf, - "%s: Device Init Failed 0x%x = %s\n", - QLA2XXX_DRIVER_NAME, dev_state, - dev_state < MAX_STATES ? - qdev_state(dev_state) : "Unknown"); - - qla8044_wr_direct(vha, QLA8044_CRB_DEV_STATE_INDEX, - QLA8XXX_DEV_FAILED); + if (qla8044_check_drv_active(vha) == QLA_SUCCESS) { + ql_log(ql_log_warn, vha, 0xb0cf, + "%s: Device Init Failed 0x%x = %s\n", + QLA2XXX_DRIVER_NAME, dev_state, + dev_state < MAX_STATES ? + qdev_state(dev_state) : "Unknown"); + qla8044_wr_direct(vha, + QLA8044_CRB_DEV_STATE_INDEX, + QLA8XXX_DEV_FAILED); + } } dev_state = qla8044_rd_direct(vha, QLA8044_CRB_DEV_STATE_INDEX); @@ -2017,6 +2118,13 @@ qla8044_watchdog(struct scsi_qla_host *vha) test_bit(FCOE_CTX_RESET_NEEDED, &vha->dpc_flags))) { dev_state = qla8044_rd_direct(vha, QLA8044_CRB_DEV_STATE_INDEX); + if (qla8044_check_fw_alive(vha)) { + ha->flags.isp82xx_fw_hung = 1; + ql_log(ql_log_warn, vha, 0xb10a, + "Firmware hung.\n"); + qla82xx_clear_pending_mbx(vha); + } + if (qla8044_check_temp(vha)) { set_bit(ISP_UNRECOVERABLE, &vha->dpc_flags); ha->flags.isp82xx_fw_hung = 1; @@ -2037,7 +2145,7 @@ qla8044_watchdog(struct scsi_qla_host *vha) qla2xxx_wake_dpc(vha); } else { /* Check firmware health */ - if (qla8044_check_fw_alive(vha)) { + if (ha->flags.isp82xx_fw_hung) { halt_status = qla8044_rd_direct(vha, QLA8044_PEG_HALT_STATUS1_INDEX); if (halt_status & @@ -2073,12 +2181,8 @@ qla8044_watchdog(struct scsi_qla_host *vha) __func__); set_bit(ISP_ABORT_NEEDED, &vha->dpc_flags); - qla82xx_clear_pending_mbx(vha); } } - ha->flags.isp82xx_fw_hung = 1; - ql_log(ql_log_warn, vha, 0xb10a, - "Firmware hung.\n"); qla2xxx_wake_dpc(vha); } } @@ -2286,8 +2390,6 @@ qla8044_minidump_process_rdmem(struct scsi_qla_host *vha, } if (j >= MAX_CTL_CHECK) { - printk_ratelimited(KERN_ERR - "%s: failed to read through agent\n", __func__); write_unlock_irqrestore(&ha->hw_lock, flags); return QLA_SUCCESS; } @@ -2882,6 +2984,231 @@ error_exit: return rval; } +static uint32_t +qla8044_minidump_process_rddfe(struct scsi_qla_host *vha, + struct qla8044_minidump_entry_hdr *entry_hdr, uint32_t **d_ptr) +{ + int loop_cnt; + uint32_t addr1, addr2, value, data, temp, wrVal; + uint8_t stride, stride2; + uint16_t count; + uint32_t poll, mask, data_size, modify_mask; + uint32_t wait_count = 0; + + uint32_t *data_ptr = *d_ptr; + + struct qla8044_minidump_entry_rddfe *rddfe; + rddfe = (struct qla8044_minidump_entry_rddfe *) entry_hdr; + + addr1 = rddfe->addr_1; + value = rddfe->value; + stride = rddfe->stride; + stride2 = rddfe->stride2; + count = rddfe->count; + + poll = rddfe->poll; + mask = rddfe->mask; + modify_mask = rddfe->modify_mask; + data_size = rddfe->data_size; + + addr2 = addr1 + stride; + + for (loop_cnt = 0x0; loop_cnt < count; loop_cnt++) { + qla8044_wr_reg_indirect(vha, addr1, (0x40000000 | value)); + + wait_count = 0; + while (wait_count < poll) { + qla8044_rd_reg_indirect(vha, addr1, &temp); + if ((temp & mask) != 0) + break; + wait_count++; + } + + if (wait_count == poll) { + ql_log(ql_log_warn, vha, 0xb153, + "%s: TIMEOUT\n", __func__); + goto error; + } else { + qla8044_rd_reg_indirect(vha, addr2, &temp); + temp = temp & modify_mask; + temp = (temp | ((loop_cnt << 16) | loop_cnt)); + wrVal = ((temp << 16) | temp); + + qla8044_wr_reg_indirect(vha, addr2, wrVal); + qla8044_wr_reg_indirect(vha, addr1, value); + + wait_count = 0; + while (wait_count < poll) { + qla8044_rd_reg_indirect(vha, addr1, &temp); + if ((temp & mask) != 0) + break; + wait_count++; + } + if (wait_count == poll) { + ql_log(ql_log_warn, vha, 0xb154, + "%s: TIMEOUT\n", __func__); + goto error; + } + + qla8044_wr_reg_indirect(vha, addr1, + ((0x40000000 | value) + stride2)); + wait_count = 0; + while (wait_count < poll) { + qla8044_rd_reg_indirect(vha, addr1, &temp); + if ((temp & mask) != 0) + break; + wait_count++; + } + + if (wait_count == poll) { + ql_log(ql_log_warn, vha, 0xb155, + "%s: TIMEOUT\n", __func__); + goto error; + } + + qla8044_rd_reg_indirect(vha, addr2, &data); + + *data_ptr++ = wrVal; + *data_ptr++ = data; + } + + } + + *d_ptr = data_ptr; + return QLA_SUCCESS; + +error: + return -1; + +} + +static uint32_t +qla8044_minidump_process_rdmdio(struct scsi_qla_host *vha, + struct qla8044_minidump_entry_hdr *entry_hdr, uint32_t **d_ptr) +{ + int ret = 0; + uint32_t addr1, addr2, value1, value2, data, selVal; + uint8_t stride1, stride2; + uint32_t addr3, addr4, addr5, addr6, addr7; + uint16_t count, loop_cnt; + uint32_t poll, mask; + uint32_t *data_ptr = *d_ptr; + + struct qla8044_minidump_entry_rdmdio *rdmdio; + + rdmdio = (struct qla8044_minidump_entry_rdmdio *) entry_hdr; + + addr1 = rdmdio->addr_1; + addr2 = rdmdio->addr_2; + value1 = rdmdio->value_1; + stride1 = rdmdio->stride_1; + stride2 = rdmdio->stride_2; + count = rdmdio->count; + + poll = rdmdio->poll; + mask = rdmdio->mask; + value2 = rdmdio->value_2; + + addr3 = addr1 + stride1; + + for (loop_cnt = 0; loop_cnt < count; loop_cnt++) { + ret = qla8044_poll_wait_ipmdio_bus_idle(vha, addr1, addr2, + addr3, mask); + if (ret == -1) + goto error; + + addr4 = addr2 - stride1; + ret = qla8044_ipmdio_wr_reg(vha, addr1, addr3, mask, addr4, + value2); + if (ret == -1) + goto error; + + addr5 = addr2 - (2 * stride1); + ret = qla8044_ipmdio_wr_reg(vha, addr1, addr3, mask, addr5, + value1); + if (ret == -1) + goto error; + + addr6 = addr2 - (3 * stride1); + ret = qla8044_ipmdio_wr_reg(vha, addr1, addr3, mask, + addr6, 0x2); + if (ret == -1) + goto error; + + ret = qla8044_poll_wait_ipmdio_bus_idle(vha, addr1, addr2, + addr3, mask); + if (ret == -1) + goto error; + + addr7 = addr2 - (4 * stride1); + data = qla8044_ipmdio_rd_reg(vha, addr1, addr3, + mask, addr7); + if (data == -1) + goto error; + + selVal = (value2 << 18) | (value1 << 2) | 2; + + stride2 = rdmdio->stride_2; + *data_ptr++ = selVal; + *data_ptr++ = data; + + value1 = value1 + stride2; + *d_ptr = data_ptr; + } + + return 0; + +error: + return -1; +} + +static uint32_t qla8044_minidump_process_pollwr(struct scsi_qla_host *vha, + struct qla8044_minidump_entry_hdr *entry_hdr, uint32_t **d_ptr) +{ + uint32_t addr1, addr2, value1, value2, poll, mask, r_value; + uint32_t wait_count = 0; + struct qla8044_minidump_entry_pollwr *pollwr_hdr; + + pollwr_hdr = (struct qla8044_minidump_entry_pollwr *)entry_hdr; + addr1 = pollwr_hdr->addr_1; + addr2 = pollwr_hdr->addr_2; + value1 = pollwr_hdr->value_1; + value2 = pollwr_hdr->value_2; + + poll = pollwr_hdr->poll; + mask = pollwr_hdr->mask; + + while (wait_count < poll) { + qla8044_rd_reg_indirect(vha, addr1, &r_value); + + if ((r_value & poll) != 0) + break; + wait_count++; + } + + if (wait_count == poll) { + ql_log(ql_log_warn, vha, 0xb156, "%s: TIMEOUT\n", __func__); + goto error; + } + + qla8044_wr_reg_indirect(vha, addr2, value2); + qla8044_wr_reg_indirect(vha, addr1, value1); + + wait_count = 0; + while (wait_count < poll) { + qla8044_rd_reg_indirect(vha, addr1, &r_value); + + if ((r_value & poll) != 0) + break; + wait_count++; + } + + return QLA_SUCCESS; + +error: + return -1; +} + /* * * qla8044_collect_md_data - Retrieve firmware minidump data. @@ -3089,6 +3416,24 @@ qla8044_collect_md_data(struct scsi_qla_host *vha) if (rval != QLA_SUCCESS) qla8044_mark_entry_skipped(vha, entry_hdr, i); break; + case QLA8044_RDDFE: + rval = qla8044_minidump_process_rddfe(vha, entry_hdr, + &data_ptr); + if (rval != QLA_SUCCESS) + qla8044_mark_entry_skipped(vha, entry_hdr, i); + break; + case QLA8044_RDMDIO: + rval = qla8044_minidump_process_rdmdio(vha, entry_hdr, + &data_ptr); + if (rval != QLA_SUCCESS) + qla8044_mark_entry_skipped(vha, entry_hdr, i); + break; + case QLA8044_POLLWR: + rval = qla8044_minidump_process_pollwr(vha, entry_hdr, + &data_ptr); + if (rval != QLA_SUCCESS) + qla8044_mark_entry_skipped(vha, entry_hdr, i); + break; case QLA82XX_RDNOP: default: qla8044_mark_entry_skipped(vha, entry_hdr, i); @@ -3110,6 +3455,7 @@ skip_nxt_entry: "Dump data mismatch: Data collected: " "[0x%x], total_data_size:[0x%x]\n", data_collected, ha->md_dump_size); + rval = QLA_FUNCTION_FAILED; goto md_failed; } @@ -3134,10 +3480,12 @@ qla8044_get_minidump(struct scsi_qla_host *vha) if (!qla8044_collect_md_data(vha)) { ha->fw_dumped = 1; + ha->prev_minidump_failed = 0; } else { ql_log(ql_log_fatal, vha, 0xb0db, "%s: Unable to collect minidump\n", __func__); + ha->prev_minidump_failed = 1; } } diff --git a/drivers/scsi/qla2xxx/qla_nx2.h b/drivers/scsi/qla2xxx/qla_nx2.h index 2ab2eabab908..ada36057d7cd 100644 --- a/drivers/scsi/qla2xxx/qla_nx2.h +++ b/drivers/scsi/qla2xxx/qla_nx2.h @@ -1,6 +1,6 @@ /* * QLogic Fibre Channel HBA Driver - * Copyright (c) 2003-2013 QLogic Corporation + * Copyright (c) 2003-2014 QLogic Corporation * * See LICENSE.qla2xxx for copyright and licensing details. */ @@ -133,6 +133,7 @@ #define QLA8044_LINK_SPEED(f) (0x36E0+(((f) >> 2) * 4)) #define QLA8044_MAX_LINK_SPEED(f) (0x36F0+(((f) / 4) * 4)) #define QLA8044_LINK_SPEED_FACTOR 10 +#define QLA8044_FUN7_ACTIVE_INDEX 0x80 /* FLASH API Defines */ #define QLA8044_FLASH_MAX_WAIT_USEC 100 @@ -431,6 +432,50 @@ struct qla8044_minidump_entry_pollrd { uint32_t rsvd_1; } __packed; +struct qla8044_minidump_entry_rddfe { + struct qla8044_minidump_entry_hdr h; + uint32_t addr_1; + uint32_t value; + uint8_t stride; + uint8_t stride2; + uint16_t count; + uint32_t poll; + uint32_t mask; + uint32_t modify_mask; + uint32_t data_size; + uint32_t rsvd; + +} __packed; + +struct qla8044_minidump_entry_rdmdio { + struct qla8044_minidump_entry_hdr h; + + uint32_t addr_1; + uint32_t addr_2; + uint32_t value_1; + uint8_t stride_1; + uint8_t stride_2; + uint16_t count; + uint32_t poll; + uint32_t mask; + uint32_t value_2; + uint32_t data_size; + +} __packed; + +struct qla8044_minidump_entry_pollwr { + struct qla8044_minidump_entry_hdr h; + uint32_t addr_1; + uint32_t addr_2; + uint32_t value_1; + uint32_t value_2; + uint32_t poll; + uint32_t mask; + uint32_t data_size; + uint32_t rsvd; + +} __packed; + /* RDMUX2 Entry */ struct qla8044_minidump_entry_rdmux2 { struct qla8044_minidump_entry_hdr h; @@ -516,6 +561,9 @@ static const uint32_t qla8044_reg_tbl[] = { #define QLA8044_DBG_RSVD_ARRAY_LEN 8 #define QLA8044_DBG_OCM_WNDREG_ARRAY_LEN 16 #define QLA8044_SS_PCI_INDEX 0 +#define QLA8044_RDDFE 38 +#define QLA8044_RDMDIO 39 +#define QLA8044_POLLWR 40 struct qla8044_minidump_template_hdr { uint32_t entry_type; diff --git a/drivers/scsi/qla2xxx/qla_os.c b/drivers/scsi/qla2xxx/qla_os.c index afc84814e9bb..d96bfb55e57b 100644 --- a/drivers/scsi/qla2xxx/qla_os.c +++ b/drivers/scsi/qla2xxx/qla_os.c @@ -1,6 +1,6 @@ /* * QLogic Fibre Channel HBA Driver - * Copyright (c) 2003-2013 QLogic Corporation + * Copyright (c) 2003-2014 QLogic Corporation * * See LICENSE.qla2xxx for copyright and licensing details. */ @@ -616,7 +616,7 @@ qla2x00_sp_free_dma(void *vha, void *ptr) if (sp->flags & SRB_CRC_CTX_DSD_VALID) { /* List assured to be having elements */ - qla2x00_clean_dsd_pool(ha, sp); + qla2x00_clean_dsd_pool(ha, sp, NULL); sp->flags &= ~SRB_CRC_CTX_DSD_VALID; } @@ -781,7 +781,7 @@ static int qla2x00_eh_wait_on_command(struct scsi_cmnd *cmd) { #define ABORT_POLLING_PERIOD 1000 -#define ABORT_WAIT_ITER ((10 * 1000) / (ABORT_POLLING_PERIOD)) +#define ABORT_WAIT_ITER ((2 * 1000) / (ABORT_POLLING_PERIOD)) unsigned long wait_iter = ABORT_WAIT_ITER; scsi_qla_host_t *vha = shost_priv(cmd->device->host); struct qla_hw_data *ha = vha->hw; @@ -844,11 +844,8 @@ qla2x00_wait_for_hba_online(scsi_qla_host_t *vha) } /* - * qla2x00_wait_for_reset_ready - * Wait till the HBA is online after going through - * <= MAX_RETRIES_OF_ISP_ABORT or - * finally HBA is disabled ie marked offline or flash - * operations are in progress. + * qla2x00_wait_for_hba_ready + * Wait till the HBA is ready before doing driver unload * * Input: * ha - pointer to host adapter structure @@ -857,35 +854,15 @@ qla2x00_wait_for_hba_online(scsi_qla_host_t *vha) * Does context switching-Release SPIN_LOCK * (if any) before calling this routine. * - * Return: - * Success (Adapter is online/no flash ops) : 0 - * Failed (Adapter is offline/disabled/flash ops in progress) : 1 */ -static int -qla2x00_wait_for_reset_ready(scsi_qla_host_t *vha) +static void +qla2x00_wait_for_hba_ready(scsi_qla_host_t *vha) { - int return_status; - unsigned long wait_online; struct qla_hw_data *ha = vha->hw; - scsi_qla_host_t *base_vha = pci_get_drvdata(ha->pdev); - wait_online = jiffies + (MAX_LOOP_TIMEOUT * HZ); - while (((test_bit(ISP_ABORT_NEEDED, &base_vha->dpc_flags)) || - test_bit(ABORT_ISP_ACTIVE, &base_vha->dpc_flags) || - test_bit(ISP_ABORT_RETRY, &base_vha->dpc_flags) || - ha->optrom_state != QLA_SWAITING || - ha->dpc_active) && time_before(jiffies, wait_online)) + while ((!(vha->flags.online) || ha->dpc_active || + ha->flags.mbox_busy)) msleep(1000); - - if (base_vha->flags.online && ha->optrom_state == QLA_SWAITING) - return_status = QLA_SUCCESS; - else - return_status = QLA_FUNCTION_FAILED; - - ql_dbg(ql_dbg_taskm, vha, 0x8019, - "%s return status=%d.\n", __func__, return_status); - - return return_status; } int @@ -945,7 +922,7 @@ qla2xxx_eh_abort(struct scsi_cmnd *cmd) int ret; unsigned int id, lun; unsigned long flags; - int wait = 0; + int rval, wait = 0; struct qla_hw_data *ha = vha->hw; if (!CMD_SP(cmd)) @@ -974,10 +951,20 @@ qla2xxx_eh_abort(struct scsi_cmnd *cmd) sp_get(sp); spin_unlock_irqrestore(&ha->hardware_lock, flags); - if (ha->isp_ops->abort_command(sp)) { - ret = FAILED; + rval = ha->isp_ops->abort_command(sp); + if (rval) { + if (rval == QLA_FUNCTION_PARAMETER_ERROR) { + /* + * Decrement the ref_count since we can't find the + * command + */ + atomic_dec(&sp->ref_count); + ret = SUCCESS; + } else + ret = FAILED; + ql_dbg(ql_dbg_taskm, vha, 0x8003, - "Abort command mbx failed cmd=%p.\n", cmd); + "Abort command mbx failed cmd=%p, rval=%x.\n", cmd, rval); } else { ql_dbg(ql_dbg_taskm, vha, 0x8004, "Abort command mbx success cmd=%p.\n", cmd); @@ -985,6 +972,12 @@ qla2xxx_eh_abort(struct scsi_cmnd *cmd) } spin_lock_irqsave(&ha->hardware_lock, flags); + /* + * Clear the slot in the oustanding_cmds array if we can't find the + * command to reclaim the resources. + */ + if (rval == QLA_FUNCTION_PARAMETER_ERROR) + vha->req->outstanding_cmds[sp->handle] = NULL; sp->done(ha, sp, 0); spin_unlock_irqrestore(&ha->hardware_lock, flags); @@ -1236,7 +1229,11 @@ qla2xxx_eh_host_reset(struct scsi_cmnd *cmd) ql_log(ql_log_info, vha, 0x8018, "ADAPTER RESET ISSUED nexus=%ld:%d:%d.\n", vha->host_no, id, lun); - if (qla2x00_wait_for_reset_ready(vha) != QLA_SUCCESS) + /* + * No point in issuing another reset if one is active. Also do not + * attempt a reset if we are updating flash. + */ + if (qla2x00_reset_active(vha) || ha->optrom_state != QLA_SWAITING) goto eh_host_reset_lock; if (vha != base_vha) { @@ -2270,6 +2267,13 @@ qla2x00_set_isp_flags(struct qla_hw_data *ha) ha->device_type |= DT_IIDMA; ha->fw_srisc_address = RISC_START_ADDRESS_2400; break; + case PCI_DEVICE_ID_QLOGIC_ISP2271: + ha->device_type |= DT_ISP2271; + ha->device_type |= DT_ZIO_SUPPORTED; + ha->device_type |= DT_FWI2; + ha->device_type |= DT_IIDMA; + ha->fw_srisc_address = RISC_START_ADDRESS_2400; + break; } if (IS_QLA82XX(ha)) @@ -2346,7 +2350,8 @@ qla2x00_probe_one(struct pci_dev *pdev, const struct pci_device_id *id) pdev->device == PCI_DEVICE_ID_QLOGIC_ISP8031 || pdev->device == PCI_DEVICE_ID_QLOGIC_ISPF001 || pdev->device == PCI_DEVICE_ID_QLOGIC_ISP8044 || - pdev->device == PCI_DEVICE_ID_QLOGIC_ISP2071) { + pdev->device == PCI_DEVICE_ID_QLOGIC_ISP2071 || + pdev->device == PCI_DEVICE_ID_QLOGIC_ISP2271) { bars = pci_select_bars(pdev, IORESOURCE_MEM); mem_only = 1; ql_dbg_pci(ql_dbg_init, pdev, 0x0007, @@ -2877,6 +2882,7 @@ skip_dpc: base_vha->flags.init_done = 1; base_vha->flags.online = 1; + ha->prev_minidump_failed = 0; ql_dbg(ql_dbg_init, base_vha, 0x00f2, "Init done and hba is online.\n"); @@ -3136,6 +3142,8 @@ qla2x00_remove_one(struct pci_dev *pdev) base_vha = pci_get_drvdata(pdev); ha = base_vha->hw; + qla2x00_wait_for_hba_ready(base_vha); + set_bit(UNLOADING, &base_vha->dpc_flags); if (IS_QLAFX00(ha)) @@ -3645,6 +3653,7 @@ qla2x00_free_fw_dump(struct qla_hw_data *ha) ha->eft = NULL; ha->eft_dma = 0; ha->fw_dumped = 0; + ha->fw_dump_cap_flags = 0; ha->fw_dump_reading = 0; ha->fw_dump = NULL; ha->fw_dump_len = 0; @@ -4913,12 +4922,13 @@ qla2x00_do_dpc(void *data) if (qlafx00_reset_initialize(base_vha)) { /* Failed. Abort isp later. */ if (!test_bit(UNLOADING, - &base_vha->dpc_flags)) + &base_vha->dpc_flags)) { set_bit(ISP_UNRECOVERABLE, &base_vha->dpc_flags); ql_dbg(ql_dbg_dpc, base_vha, 0x4021, "Reset Recovery Failed\n"); + } } } @@ -5077,8 +5087,10 @@ intr_on_check: ha->isp_ops->enable_intrs(ha); if (test_and_clear_bit(BEACON_BLINK_NEEDED, - &base_vha->dpc_flags)) - ha->isp_ops->beacon_blink(base_vha); + &base_vha->dpc_flags)) { + if (ha->beacon_blink_led == 1) + ha->isp_ops->beacon_blink(base_vha); + } if (!IS_QLAFX00(ha)) qla2x00_do_dpc_all_vps(base_vha); @@ -5325,7 +5337,7 @@ qla2x00_timer(scsi_qla_host_t *vha) #define FW_ISP82XX 7 #define FW_ISP2031 8 #define FW_ISP8031 9 -#define FW_ISP2071 10 +#define FW_ISP27XX 10 #define FW_FILE_ISP21XX "ql2100_fw.bin" #define FW_FILE_ISP22XX "ql2200_fw.bin" @@ -5337,7 +5349,7 @@ qla2x00_timer(scsi_qla_host_t *vha) #define FW_FILE_ISP82XX "ql8200_fw.bin" #define FW_FILE_ISP2031 "ql2600_fw.bin" #define FW_FILE_ISP8031 "ql8300_fw.bin" -#define FW_FILE_ISP2071 "ql2700_fw.bin" +#define FW_FILE_ISP27XX "ql2700_fw.bin" static DEFINE_MUTEX(qla_fw_lock); @@ -5353,7 +5365,7 @@ static struct fw_blob qla_fw_blobs[FW_BLOBS] = { { .name = FW_FILE_ISP82XX, }, { .name = FW_FILE_ISP2031, }, { .name = FW_FILE_ISP8031, }, - { .name = FW_FILE_ISP2071, }, + { .name = FW_FILE_ISP27XX, }, }; struct fw_blob * @@ -5382,8 +5394,8 @@ qla2x00_request_firmware(scsi_qla_host_t *vha) blob = &qla_fw_blobs[FW_ISP2031]; } else if (IS_QLA8031(ha)) { blob = &qla_fw_blobs[FW_ISP8031]; - } else if (IS_QLA2071(ha)) { - blob = &qla_fw_blobs[FW_ISP2071]; + } else if (IS_QLA27XX(ha)) { + blob = &qla_fw_blobs[FW_ISP27XX]; } else { return NULL; } @@ -5714,6 +5726,7 @@ static struct pci_device_id qla2xxx_pci_tbl[] = { { PCI_DEVICE(PCI_VENDOR_ID_QLOGIC, PCI_DEVICE_ID_QLOGIC_ISPF001) }, { PCI_DEVICE(PCI_VENDOR_ID_QLOGIC, PCI_DEVICE_ID_QLOGIC_ISP8044) }, { PCI_DEVICE(PCI_VENDOR_ID_QLOGIC, PCI_DEVICE_ID_QLOGIC_ISP2071) }, + { PCI_DEVICE(PCI_VENDOR_ID_QLOGIC, PCI_DEVICE_ID_QLOGIC_ISP2271) }, { 0 }, }; MODULE_DEVICE_TABLE(pci, qla2xxx_pci_tbl); diff --git a/drivers/scsi/qla2xxx/qla_settings.h b/drivers/scsi/qla2xxx/qla_settings.h index 46ef0ac48f44..2fb7ebfbbc38 100644 --- a/drivers/scsi/qla2xxx/qla_settings.h +++ b/drivers/scsi/qla2xxx/qla_settings.h @@ -1,6 +1,6 @@ /* * QLogic Fibre Channel HBA Driver - * Copyright (c) 2003-2013 QLogic Corporation + * Copyright (c) 2003-2014 QLogic Corporation * * See LICENSE.qla2xxx for copyright and licensing details. */ diff --git a/drivers/scsi/qla2xxx/qla_sup.c b/drivers/scsi/qla2xxx/qla_sup.c index f28123e8ed65..bca173e56f16 100644 --- a/drivers/scsi/qla2xxx/qla_sup.c +++ b/drivers/scsi/qla2xxx/qla_sup.c @@ -1,6 +1,6 @@ /* * QLogic Fibre Channel HBA Driver - * Copyright (c) 2003-2013 QLogic Corporation + * Copyright (c) 2003-2014 QLogic Corporation * * See LICENSE.qla2xxx for copyright and licensing details. */ @@ -1727,11 +1727,8 @@ qla83xx_beacon_blink(struct scsi_qla_host *vha) if (IS_QLA2031(ha)) { led_select_value = qla83xx_select_led_port(ha); - qla83xx_wr_reg(vha, led_select_value, 0x40002000); - qla83xx_wr_reg(vha, led_select_value + 4, 0x40002000); - msleep(1000); - qla83xx_wr_reg(vha, led_select_value, 0x40004000); - qla83xx_wr_reg(vha, led_select_value + 4, 0x40004000); + qla83xx_wr_reg(vha, led_select_value, 0x40000230); + qla83xx_wr_reg(vha, led_select_value + 4, 0x40000230); } else if (IS_QLA8031(ha)) { led_select_value = qla83xx_select_led_port(ha); diff --git a/drivers/scsi/qla2xxx/qla_target.c b/drivers/scsi/qla2xxx/qla_target.c index 0cb73074c199..b1d10f9935c7 100644 --- a/drivers/scsi/qla2xxx/qla_target.c +++ b/drivers/scsi/qla2xxx/qla_target.c @@ -182,6 +182,11 @@ struct scsi_qla_host *qlt_find_host_by_vp_idx(struct scsi_qla_host *vha, void qlt_24xx_atio_pkt_all_vps(struct scsi_qla_host *vha, struct atio_from_isp *atio) { + ql_dbg(ql_dbg_tgt, vha, 0xe072, + "%s: qla_target(%d): type %x ox_id %04x\n", + __func__, vha->vp_idx, atio->u.raw.entry_type, + be16_to_cpu(atio->u.isp24.fcp_hdr.ox_id)); + switch (atio->u.raw.entry_type) { case ATIO_TYPE7: { @@ -236,6 +241,10 @@ void qlt_24xx_atio_pkt_all_vps(struct scsi_qla_host *vha, void qlt_response_pkt_all_vps(struct scsi_qla_host *vha, response_t *pkt) { switch (pkt->entry_type) { + case CTIO_CRC2: + ql_dbg(ql_dbg_tgt, vha, 0xe073, + "qla_target(%d):%s: CRC2 Response pkt\n", + vha->vp_idx, __func__); case CTIO_TYPE7: { struct ctio7_from_24xx *entry = (struct ctio7_from_24xx *)pkt; @@ -1350,13 +1359,42 @@ static int qlt_pci_map_calc_cnt(struct qla_tgt_prm *prm) prm->cmd->sg_mapped = 1; - /* - * If greater than four sg entries then we need to allocate - * the continuation entries - */ - if (prm->seg_cnt > prm->tgt->datasegs_per_cmd) - prm->req_cnt += DIV_ROUND_UP(prm->seg_cnt - - prm->tgt->datasegs_per_cmd, prm->tgt->datasegs_per_cont); + if (cmd->se_cmd.prot_op == TARGET_PROT_NORMAL) { + /* + * If greater than four sg entries then we need to allocate + * the continuation entries + */ + if (prm->seg_cnt > prm->tgt->datasegs_per_cmd) + prm->req_cnt += DIV_ROUND_UP(prm->seg_cnt - + prm->tgt->datasegs_per_cmd, + prm->tgt->datasegs_per_cont); + } else { + /* DIF */ + if ((cmd->se_cmd.prot_op == TARGET_PROT_DIN_INSERT) || + (cmd->se_cmd.prot_op == TARGET_PROT_DOUT_STRIP)) { + prm->seg_cnt = DIV_ROUND_UP(cmd->bufflen, cmd->blk_sz); + prm->tot_dsds = prm->seg_cnt; + } else + prm->tot_dsds = prm->seg_cnt; + + if (cmd->prot_sg_cnt) { + prm->prot_sg = cmd->prot_sg; + prm->prot_seg_cnt = pci_map_sg(prm->tgt->ha->pdev, + cmd->prot_sg, cmd->prot_sg_cnt, + cmd->dma_data_direction); + if (unlikely(prm->prot_seg_cnt == 0)) + goto out_err; + + if ((cmd->se_cmd.prot_op == TARGET_PROT_DIN_INSERT) || + (cmd->se_cmd.prot_op == TARGET_PROT_DOUT_STRIP)) { + /* Dif Bundling not support here */ + prm->prot_seg_cnt = DIV_ROUND_UP(cmd->bufflen, + cmd->blk_sz); + prm->tot_dsds += prm->prot_seg_cnt; + } else + prm->tot_dsds += prm->prot_seg_cnt; + } + } ql_dbg(ql_dbg_tgt, prm->cmd->vha, 0xe009, "seg_cnt=%d, req_cnt=%d\n", prm->seg_cnt, prm->req_cnt); @@ -1377,6 +1415,16 @@ static inline void qlt_unmap_sg(struct scsi_qla_host *vha, BUG_ON(!cmd->sg_mapped); pci_unmap_sg(ha->pdev, cmd->sg, cmd->sg_cnt, cmd->dma_data_direction); cmd->sg_mapped = 0; + + if (cmd->prot_sg_cnt) + pci_unmap_sg(ha->pdev, cmd->prot_sg, cmd->prot_sg_cnt, + cmd->dma_data_direction); + + if (cmd->ctx_dsd_alloced) + qla2x00_clean_dsd_pool(ha, NULL, cmd); + + if (cmd->ctx) + dma_pool_free(ha->dl_dma_pool, cmd->ctx, cmd->ctx->crc_ctx_dma); } static int qlt_check_reserve_free_req(struct scsi_qla_host *vha, @@ -1665,8 +1713,9 @@ static int qlt_pre_xmit_response(struct qla_tgt_cmd *cmd, return QLA_TGT_PRE_XMIT_RESP_CMD_ABORTED; } - ql_dbg(ql_dbg_tgt, vha, 0xe011, "qla_target(%d): tag=%u\n", - vha->vp_idx, cmd->tag); + ql_dbg(ql_dbg_tgt, vha, 0xe011, "qla_target(%d): tag=%u ox_id %04x\n", + vha->vp_idx, cmd->tag, + be16_to_cpu(cmd->atio.u.isp24.fcp_hdr.ox_id)); prm->cmd = cmd; prm->tgt = tgt; @@ -1902,6 +1951,323 @@ skip_explict_conf: /* Sense with len > 24, is it possible ??? */ } + + +/* diff */ +static inline int +qlt_hba_err_chk_enabled(struct se_cmd *se_cmd) +{ + /* + * Uncomment when corresponding SCSI changes are done. + * + if (!sp->cmd->prot_chk) + return 0; + * + */ + switch (se_cmd->prot_op) { + case TARGET_PROT_DOUT_INSERT: + case TARGET_PROT_DIN_STRIP: + if (ql2xenablehba_err_chk >= 1) + return 1; + break; + case TARGET_PROT_DOUT_PASS: + case TARGET_PROT_DIN_PASS: + if (ql2xenablehba_err_chk >= 2) + return 1; + break; + case TARGET_PROT_DIN_INSERT: + case TARGET_PROT_DOUT_STRIP: + return 1; + default: + break; + } + return 0; +} + +/* + * qla24xx_set_t10dif_tags_from_cmd - Extract Ref and App tags from SCSI command + * + */ +static inline void +qlt_set_t10dif_tags(struct se_cmd *se_cmd, struct crc_context *ctx) +{ + uint32_t lba = 0xffffffff & se_cmd->t_task_lba; + + /* wait til Mode Sense/Select cmd, modepage Ah, subpage 2 + * have been immplemented by TCM, before AppTag is avail. + * Look for modesense_handlers[] + */ + ctx->app_tag = __constant_cpu_to_le16(0); + ctx->app_tag_mask[0] = 0x0; + ctx->app_tag_mask[1] = 0x0; + + switch (se_cmd->prot_type) { + case TARGET_DIF_TYPE0_PROT: + /* + * No check for ql2xenablehba_err_chk, as it would be an + * I/O error if hba tag generation is not done. + */ + ctx->ref_tag = cpu_to_le32(lba); + + if (!qlt_hba_err_chk_enabled(se_cmd)) + break; + + /* enable ALL bytes of the ref tag */ + ctx->ref_tag_mask[0] = 0xff; + ctx->ref_tag_mask[1] = 0xff; + ctx->ref_tag_mask[2] = 0xff; + ctx->ref_tag_mask[3] = 0xff; + break; + /* + * For TYpe 1 protection: 16 bit GUARD tag, 32 bit REF tag, and + * 16 bit app tag. + */ + case TARGET_DIF_TYPE1_PROT: + ctx->ref_tag = cpu_to_le32(lba); + + if (!qlt_hba_err_chk_enabled(se_cmd)) + break; + + /* enable ALL bytes of the ref tag */ + ctx->ref_tag_mask[0] = 0xff; + ctx->ref_tag_mask[1] = 0xff; + ctx->ref_tag_mask[2] = 0xff; + ctx->ref_tag_mask[3] = 0xff; + break; + /* + * For TYPE 2 protection: 16 bit GUARD + 32 bit REF tag has to + * match LBA in CDB + N + */ + case TARGET_DIF_TYPE2_PROT: + ctx->ref_tag = cpu_to_le32(lba); + + if (!qlt_hba_err_chk_enabled(se_cmd)) + break; + + /* enable ALL bytes of the ref tag */ + ctx->ref_tag_mask[0] = 0xff; + ctx->ref_tag_mask[1] = 0xff; + ctx->ref_tag_mask[2] = 0xff; + ctx->ref_tag_mask[3] = 0xff; + break; + + /* For Type 3 protection: 16 bit GUARD only */ + case TARGET_DIF_TYPE3_PROT: + ctx->ref_tag_mask[0] = ctx->ref_tag_mask[1] = + ctx->ref_tag_mask[2] = ctx->ref_tag_mask[3] = 0x00; + break; + } +} + + +static inline int +qlt_build_ctio_crc2_pkt(struct qla_tgt_prm *prm, scsi_qla_host_t *vha) +{ + uint32_t *cur_dsd; + int sgc; + uint32_t transfer_length = 0; + uint32_t data_bytes; + uint32_t dif_bytes; + uint8_t bundling = 1; + uint8_t *clr_ptr; + struct crc_context *crc_ctx_pkt = NULL; + struct qla_hw_data *ha; + struct ctio_crc2_to_fw *pkt; + dma_addr_t crc_ctx_dma; + uint16_t fw_prot_opts = 0; + struct qla_tgt_cmd *cmd = prm->cmd; + struct se_cmd *se_cmd = &cmd->se_cmd; + uint32_t h; + struct atio_from_isp *atio = &prm->cmd->atio; + + sgc = 0; + ha = vha->hw; + + pkt = (struct ctio_crc2_to_fw *)vha->req->ring_ptr; + prm->pkt = pkt; + memset(pkt, 0, sizeof(*pkt)); + + ql_dbg(ql_dbg_tgt, vha, 0xe071, + "qla_target(%d):%s: se_cmd[%p] CRC2 prot_op[0x%x] cmd prot sg:cnt[%p:%x] lba[%llu]\n", + vha->vp_idx, __func__, se_cmd, se_cmd->prot_op, + prm->prot_sg, prm->prot_seg_cnt, se_cmd->t_task_lba); + + if ((se_cmd->prot_op == TARGET_PROT_DIN_INSERT) || + (se_cmd->prot_op == TARGET_PROT_DOUT_STRIP)) + bundling = 0; + + /* Compute dif len and adjust data len to incude protection */ + data_bytes = cmd->bufflen; + dif_bytes = (data_bytes / cmd->blk_sz) * 8; + + switch (se_cmd->prot_op) { + case TARGET_PROT_DIN_INSERT: + case TARGET_PROT_DOUT_STRIP: + transfer_length = data_bytes; + data_bytes += dif_bytes; + break; + + case TARGET_PROT_DIN_STRIP: + case TARGET_PROT_DOUT_INSERT: + case TARGET_PROT_DIN_PASS: + case TARGET_PROT_DOUT_PASS: + transfer_length = data_bytes + dif_bytes; + break; + + default: + BUG(); + break; + } + + if (!qlt_hba_err_chk_enabled(se_cmd)) + fw_prot_opts |= 0x10; /* Disable Guard tag checking */ + /* HBA error checking enabled */ + else if (IS_PI_UNINIT_CAPABLE(ha)) { + if ((se_cmd->prot_type == TARGET_DIF_TYPE1_PROT) || + (se_cmd->prot_type == TARGET_DIF_TYPE2_PROT)) + fw_prot_opts |= PO_DIS_VALD_APP_ESC; + else if (se_cmd->prot_type == TARGET_DIF_TYPE3_PROT) + fw_prot_opts |= PO_DIS_VALD_APP_REF_ESC; + } + + switch (se_cmd->prot_op) { + case TARGET_PROT_DIN_INSERT: + case TARGET_PROT_DOUT_INSERT: + fw_prot_opts |= PO_MODE_DIF_INSERT; + break; + case TARGET_PROT_DIN_STRIP: + case TARGET_PROT_DOUT_STRIP: + fw_prot_opts |= PO_MODE_DIF_REMOVE; + break; + case TARGET_PROT_DIN_PASS: + case TARGET_PROT_DOUT_PASS: + fw_prot_opts |= PO_MODE_DIF_PASS; + /* FUTURE: does tcm require T10CRC<->IPCKSUM conversion? */ + break; + default:/* Normal Request */ + fw_prot_opts |= PO_MODE_DIF_PASS; + break; + } + + + /* ---- PKT ---- */ + /* Update entry type to indicate Command Type CRC_2 IOCB */ + pkt->entry_type = CTIO_CRC2; + pkt->entry_count = 1; + pkt->vp_index = vha->vp_idx; + + h = qlt_make_handle(vha); + if (unlikely(h == QLA_TGT_NULL_HANDLE)) { + /* + * CTIO type 7 from the firmware doesn't provide a way to + * know the initiator's LOOP ID, hence we can't find + * the session and, so, the command. + */ + return -EAGAIN; + } else + ha->tgt.cmds[h-1] = prm->cmd; + + + pkt->handle = h | CTIO_COMPLETION_HANDLE_MARK; + pkt->nport_handle = prm->cmd->loop_id; + pkt->timeout = __constant_cpu_to_le16(QLA_TGT_TIMEOUT); + pkt->initiator_id[0] = atio->u.isp24.fcp_hdr.s_id[2]; + pkt->initiator_id[1] = atio->u.isp24.fcp_hdr.s_id[1]; + pkt->initiator_id[2] = atio->u.isp24.fcp_hdr.s_id[0]; + pkt->exchange_addr = atio->u.isp24.exchange_addr; + pkt->ox_id = swab16(atio->u.isp24.fcp_hdr.ox_id); + pkt->flags |= (atio->u.isp24.attr << 9); + pkt->relative_offset = cpu_to_le32(prm->cmd->offset); + + /* Set transfer direction */ + if (cmd->dma_data_direction == DMA_TO_DEVICE) + pkt->flags = __constant_cpu_to_le16(CTIO7_FLAGS_DATA_IN); + else if (cmd->dma_data_direction == DMA_FROM_DEVICE) + pkt->flags = __constant_cpu_to_le16(CTIO7_FLAGS_DATA_OUT); + + + pkt->dseg_count = prm->tot_dsds; + /* Fibre channel byte count */ + pkt->transfer_length = cpu_to_le32(transfer_length); + + + /* ----- CRC context -------- */ + + /* Allocate CRC context from global pool */ + crc_ctx_pkt = cmd->ctx = + dma_pool_alloc(ha->dl_dma_pool, GFP_ATOMIC, &crc_ctx_dma); + + if (!crc_ctx_pkt) + goto crc_queuing_error; + + /* Zero out CTX area. */ + clr_ptr = (uint8_t *)crc_ctx_pkt; + memset(clr_ptr, 0, sizeof(*crc_ctx_pkt)); + + crc_ctx_pkt->crc_ctx_dma = crc_ctx_dma; + INIT_LIST_HEAD(&crc_ctx_pkt->dsd_list); + + /* Set handle */ + crc_ctx_pkt->handle = pkt->handle; + + qlt_set_t10dif_tags(se_cmd, crc_ctx_pkt); + + pkt->crc_context_address[0] = cpu_to_le32(LSD(crc_ctx_dma)); + pkt->crc_context_address[1] = cpu_to_le32(MSD(crc_ctx_dma)); + pkt->crc_context_len = CRC_CONTEXT_LEN_FW; + + + if (!bundling) { + cur_dsd = (uint32_t *) &crc_ctx_pkt->u.nobundling.data_address; + } else { + /* + * Configure Bundling if we need to fetch interlaving + * protection PCI accesses + */ + fw_prot_opts |= PO_ENABLE_DIF_BUNDLING; + crc_ctx_pkt->u.bundling.dif_byte_count = cpu_to_le32(dif_bytes); + crc_ctx_pkt->u.bundling.dseg_count = + cpu_to_le16(prm->tot_dsds - prm->prot_seg_cnt); + cur_dsd = (uint32_t *) &crc_ctx_pkt->u.bundling.data_address; + } + + /* Finish the common fields of CRC pkt */ + crc_ctx_pkt->blk_size = cpu_to_le16(cmd->blk_sz); + crc_ctx_pkt->prot_opts = cpu_to_le16(fw_prot_opts); + crc_ctx_pkt->byte_count = cpu_to_le32(data_bytes); + crc_ctx_pkt->guard_seed = __constant_cpu_to_le16(0); + + + /* Walks data segments */ + pkt->flags |= __constant_cpu_to_le16(CTIO7_FLAGS_DSD_PTR); + + if (!bundling && prm->prot_seg_cnt) { + if (qla24xx_walk_and_build_sglist_no_difb(ha, NULL, cur_dsd, + prm->tot_dsds, cmd)) + goto crc_queuing_error; + } else if (qla24xx_walk_and_build_sglist(ha, NULL, cur_dsd, + (prm->tot_dsds - prm->prot_seg_cnt), cmd)) + goto crc_queuing_error; + + if (bundling && prm->prot_seg_cnt) { + /* Walks dif segments */ + pkt->add_flags |= + __constant_cpu_to_le16(CTIO_CRC2_AF_DIF_DSD_ENA); + + cur_dsd = (uint32_t *) &crc_ctx_pkt->u.bundling.dif_address; + if (qla24xx_walk_and_build_prot_sglist(ha, NULL, cur_dsd, + prm->prot_seg_cnt, cmd)) + goto crc_queuing_error; + } + return QLA_SUCCESS; + +crc_queuing_error: + /* Cleanup will be performed by the caller */ + + return QLA_FUNCTION_FAILED; +} + + /* * Callback to setup response of xmit_type of QLA_TGT_XMIT_DATA and * * QLA_TGT_XMIT_STATUS for >= 24xx silicon @@ -1921,9 +2287,10 @@ int qlt_xmit_response(struct qla_tgt_cmd *cmd, int xmit_type, qlt_check_srr_debug(cmd, &xmit_type); ql_dbg(ql_dbg_tgt, cmd->vha, 0xe018, - "is_send_status=%d, cmd->bufflen=%d, cmd->sg_cnt=%d, " - "cmd->dma_data_direction=%d\n", (xmit_type & QLA_TGT_XMIT_STATUS) ? - 1 : 0, cmd->bufflen, cmd->sg_cnt, cmd->dma_data_direction); + "is_send_status=%d, cmd->bufflen=%d, cmd->sg_cnt=%d, cmd->dma_data_direction=%d se_cmd[%p]\n", + (xmit_type & QLA_TGT_XMIT_STATUS) ? + 1 : 0, cmd->bufflen, cmd->sg_cnt, cmd->dma_data_direction, + &cmd->se_cmd); res = qlt_pre_xmit_response(cmd, &prm, xmit_type, scsi_status, &full_req_cnt); @@ -1941,7 +2308,10 @@ int qlt_xmit_response(struct qla_tgt_cmd *cmd, int xmit_type, if (unlikely(res)) goto out_unmap_unlock; - res = qlt_24xx_build_ctio_pkt(&prm, vha); + if (cmd->se_cmd.prot_op && (xmit_type & QLA_TGT_XMIT_DATA)) + res = qlt_build_ctio_crc2_pkt(&prm, vha); + else + res = qlt_24xx_build_ctio_pkt(&prm, vha); if (unlikely(res != 0)) goto out_unmap_unlock; @@ -1953,7 +2323,8 @@ int qlt_xmit_response(struct qla_tgt_cmd *cmd, int xmit_type, __constant_cpu_to_le16(CTIO7_FLAGS_DATA_IN | CTIO7_FLAGS_STATUS_MODE_0); - qlt_load_data_segments(&prm, vha); + if (cmd->se_cmd.prot_op == TARGET_PROT_NORMAL) + qlt_load_data_segments(&prm, vha); if (prm.add_status_pkt == 0) { if (xmit_type & QLA_TGT_XMIT_STATUS) { @@ -1983,8 +2354,14 @@ int qlt_xmit_response(struct qla_tgt_cmd *cmd, int xmit_type, ql_dbg(ql_dbg_tgt, vha, 0xe019, "Building additional status packet\n"); + /* + * T10Dif: ctio_crc2_to_fw overlay ontop of + * ctio7_to_24xx + */ memcpy(ctio, pkt, sizeof(*ctio)); + /* reset back to CTIO7 */ ctio->entry_count = 1; + ctio->entry_type = CTIO_TYPE7; ctio->dseg_count = 0; ctio->u.status1.flags &= ~__constant_cpu_to_le16( CTIO7_FLAGS_DATA_IN); @@ -1993,6 +2370,11 @@ int qlt_xmit_response(struct qla_tgt_cmd *cmd, int xmit_type, pkt->handle |= CTIO_INTERMEDIATE_HANDLE_MARK; pkt->u.status0.flags |= __constant_cpu_to_le16( CTIO7_FLAGS_DONT_RET_CTIO); + + /* qlt_24xx_init_ctio_to_isp will correct + * all neccessary fields that's part of CTIO7. + * There should be no residual of CTIO-CRC2 data. + */ qlt_24xx_init_ctio_to_isp((struct ctio7_to_24xx *)ctio, &prm); pr_debug("Status CTIO7: %p\n", ctio); @@ -2041,8 +2423,10 @@ int qlt_rdy_to_xfer(struct qla_tgt_cmd *cmd) if (qlt_issue_marker(vha, 0) != QLA_SUCCESS) return -EIO; - ql_dbg(ql_dbg_tgt, vha, 0xe01b, "CTIO_start: vha(%d)", - (int)vha->vp_idx); + ql_dbg(ql_dbg_tgt, vha, 0xe01b, + "%s: CTIO_start: vha(%d) se_cmd %p ox_id %04x\n", + __func__, (int)vha->vp_idx, &cmd->se_cmd, + be16_to_cpu(cmd->atio.u.isp24.fcp_hdr.ox_id)); /* Calculate number of entries and segments required */ if (qlt_pci_map_calc_cnt(&prm) != 0) @@ -2054,14 +2438,19 @@ int qlt_rdy_to_xfer(struct qla_tgt_cmd *cmd) res = qlt_check_reserve_free_req(vha, prm.req_cnt); if (res != 0) goto out_unlock_free_unmap; + if (cmd->se_cmd.prot_op) + res = qlt_build_ctio_crc2_pkt(&prm, vha); + else + res = qlt_24xx_build_ctio_pkt(&prm, vha); - res = qlt_24xx_build_ctio_pkt(&prm, vha); if (unlikely(res != 0)) goto out_unlock_free_unmap; pkt = (struct ctio7_to_24xx *)prm.pkt; pkt->u.status0.flags |= __constant_cpu_to_le16(CTIO7_FLAGS_DATA_OUT | CTIO7_FLAGS_STATUS_MODE_0); - qlt_load_data_segments(&prm, vha); + + if (cmd->se_cmd.prot_op == TARGET_PROT_NORMAL) + qlt_load_data_segments(&prm, vha); cmd->state = QLA_TGT_STATE_NEED_DATA; @@ -2079,6 +2468,143 @@ out_unlock_free_unmap: } EXPORT_SYMBOL(qlt_rdy_to_xfer); + +/* + * Checks the guard or meta-data for the type of error + * detected by the HBA. + */ +static inline int +qlt_handle_dif_error(struct scsi_qla_host *vha, struct qla_tgt_cmd *cmd, + struct ctio_crc_from_fw *sts) +{ + uint8_t *ap = &sts->actual_dif[0]; + uint8_t *ep = &sts->expected_dif[0]; + uint32_t e_ref_tag, a_ref_tag; + uint16_t e_app_tag, a_app_tag; + uint16_t e_guard, a_guard; + uint64_t lba = cmd->se_cmd.t_task_lba; + + a_guard = be16_to_cpu(*(uint16_t *)(ap + 0)); + a_app_tag = be16_to_cpu(*(uint16_t *)(ap + 2)); + a_ref_tag = be32_to_cpu(*(uint32_t *)(ap + 4)); + + e_guard = be16_to_cpu(*(uint16_t *)(ep + 0)); + e_app_tag = be16_to_cpu(*(uint16_t *)(ep + 2)); + e_ref_tag = be32_to_cpu(*(uint32_t *)(ep + 4)); + + ql_dbg(ql_dbg_tgt, vha, 0xe075, + "iocb(s) %p Returned STATUS.\n", sts); + + ql_dbg(ql_dbg_tgt, vha, 0xf075, + "dif check TGT cdb 0x%x lba 0x%llu: [Actual|Expected] Ref Tag[0x%x|0x%x], App Tag [0x%x|0x%x], Guard [0x%x|0x%x]\n", + cmd->atio.u.isp24.fcp_cmnd.cdb[0], lba, + a_ref_tag, e_ref_tag, a_app_tag, e_app_tag, a_guard, e_guard); + + /* + * Ignore sector if: + * For type 3: ref & app tag is all 'f's + * For type 0,1,2: app tag is all 'f's + */ + if ((a_app_tag == 0xffff) && + ((cmd->se_cmd.prot_type != TARGET_DIF_TYPE3_PROT) || + (a_ref_tag == 0xffffffff))) { + uint32_t blocks_done; + + /* 2TB boundary case covered automatically with this */ + blocks_done = e_ref_tag - (uint32_t)lba + 1; + cmd->se_cmd.bad_sector = e_ref_tag; + cmd->se_cmd.pi_err = 0; + ql_dbg(ql_dbg_tgt, vha, 0xf074, + "need to return scsi good\n"); + + /* Update protection tag */ + if (cmd->prot_sg_cnt) { + uint32_t i, j = 0, k = 0, num_ent; + struct scatterlist *sg, *sgl; + + + sgl = cmd->prot_sg; + + /* Patch the corresponding protection tags */ + for_each_sg(sgl, sg, cmd->prot_sg_cnt, i) { + num_ent = sg_dma_len(sg) / 8; + if (k + num_ent < blocks_done) { + k += num_ent; + continue; + } + j = blocks_done - k - 1; + k = blocks_done; + break; + } + + if (k != blocks_done) { + ql_log(ql_log_warn, vha, 0xf076, + "unexpected tag values tag:lba=%u:%llu)\n", + e_ref_tag, (unsigned long long)lba); + goto out; + } + +#if 0 + struct sd_dif_tuple *spt; + /* TODO: + * This section came from initiator. Is it valid here? + * should ulp be override with actual val??? + */ + spt = page_address(sg_page(sg)) + sg->offset; + spt += j; + + spt->app_tag = 0xffff; + if (cmd->se_cmd.prot_type == SCSI_PROT_DIF_TYPE3) + spt->ref_tag = 0xffffffff; +#endif + } + + return 0; + } + + /* check guard */ + if (e_guard != a_guard) { + cmd->se_cmd.pi_err = TCM_LOGICAL_BLOCK_GUARD_CHECK_FAILED; + cmd->se_cmd.bad_sector = cmd->se_cmd.t_task_lba; + + ql_log(ql_log_warn, vha, 0xe076, + "Guard ERR: cdb 0x%x lba 0x%llx: [Actual|Expected] Ref Tag[0x%x|0x%x], App Tag [0x%x|0x%x], Guard [0x%x|0x%x] cmd=%p\n", + cmd->atio.u.isp24.fcp_cmnd.cdb[0], lba, + a_ref_tag, e_ref_tag, a_app_tag, e_app_tag, + a_guard, e_guard, cmd); + goto out; + } + + /* check ref tag */ + if (e_ref_tag != a_ref_tag) { + cmd->se_cmd.pi_err = TCM_LOGICAL_BLOCK_REF_TAG_CHECK_FAILED; + cmd->se_cmd.bad_sector = e_ref_tag; + + ql_log(ql_log_warn, vha, 0xe077, + "Ref Tag ERR: cdb 0x%x lba 0x%llx: [Actual|Expected] Ref Tag[0x%x|0x%x], App Tag [0x%x|0x%x], Guard [0x%x|0x%x] cmd=%p\n", + cmd->atio.u.isp24.fcp_cmnd.cdb[0], lba, + a_ref_tag, e_ref_tag, a_app_tag, e_app_tag, + a_guard, e_guard, cmd); + goto out; + } + + /* check appl tag */ + if (e_app_tag != a_app_tag) { + cmd->se_cmd.pi_err = TCM_LOGICAL_BLOCK_APP_TAG_CHECK_FAILED; + cmd->se_cmd.bad_sector = cmd->se_cmd.t_task_lba; + + ql_log(ql_log_warn, vha, 0xe078, + "App Tag ERR: cdb 0x%x lba 0x%llx: [Actual|Expected] Ref Tag[0x%x|0x%x], App Tag [0x%x|0x%x], Guard [0x%x|0x%x] cmd=%p\n", + cmd->atio.u.isp24.fcp_cmnd.cdb[0], lba, + a_ref_tag, e_ref_tag, a_app_tag, e_app_tag, + a_guard, e_guard, cmd); + goto out; + } +out: + return 1; +} + + /* If hardware_lock held on entry, might drop it, then reaquire */ /* This function sends the appropriate CTIO to ISP 2xxx or 24xx */ static int __qlt_send_term_exchange(struct scsi_qla_host *vha, @@ -2155,18 +2681,36 @@ static void qlt_send_term_exchange(struct scsi_qla_host *vha, rc = __qlt_send_term_exchange(vha, cmd, atio); spin_unlock_irqrestore(&vha->hw->hardware_lock, flags); done: - if (rc == 1) { + /* + * Terminate exchange will tell fw to release any active CTIO + * that's in FW posession and cleanup the exchange. + * + * "cmd->state == QLA_TGT_STATE_ABORTED" means CTIO is still + * down at FW. Free the cmd later when CTIO comes back later + * w/aborted(0x2) status. + * + * "cmd->state != QLA_TGT_STATE_ABORTED" means CTIO is already + * back w/some err. Free the cmd now. + */ + if ((rc == 1) && (cmd->state != QLA_TGT_STATE_ABORTED)) { if (!ha_locked && !in_interrupt()) msleep(250); /* just in case */ + if (cmd->sg_mapped) + qlt_unmap_sg(vha, cmd); vha->hw->tgt.tgt_ops->free_cmd(cmd); } + return; } void qlt_free_cmd(struct qla_tgt_cmd *cmd) { - BUG_ON(cmd->sg_mapped); + ql_dbg(ql_dbg_tgt, cmd->vha, 0xe074, + "%s: se_cmd[%p] ox_id %04x\n", + __func__, &cmd->se_cmd, + be16_to_cpu(cmd->atio.u.isp24.fcp_hdr.ox_id)); + BUG_ON(cmd->sg_mapped); if (unlikely(cmd->free_sg)) kfree(cmd->sg); kmem_cache_free(qla_tgt_cmd_cachep, cmd); @@ -2374,6 +2918,7 @@ static void qlt_do_ctio_completion(struct scsi_qla_host *vha, uint32_t handle, case CTIO_LIP_RESET: case CTIO_TARGET_RESET: case CTIO_ABORTED: + /* driver request abort via Terminate exchange */ case CTIO_TIMEOUT: case CTIO_INVALID_RX_ID: /* They are OK */ @@ -2404,18 +2949,58 @@ static void qlt_do_ctio_completion(struct scsi_qla_host *vha, uint32_t handle, else return; + case CTIO_DIF_ERROR: { + struct ctio_crc_from_fw *crc = + (struct ctio_crc_from_fw *)ctio; + ql_dbg(ql_dbg_tgt_mgt, vha, 0xf073, + "qla_target(%d): CTIO with DIF_ERROR status %x received (state %x, se_cmd %p) actual_dif[0x%llx] expect_dif[0x%llx]\n", + vha->vp_idx, status, cmd->state, se_cmd, + *((u64 *)&crc->actual_dif[0]), + *((u64 *)&crc->expected_dif[0])); + + if (qlt_handle_dif_error(vha, cmd, ctio)) { + if (cmd->state == QLA_TGT_STATE_NEED_DATA) { + /* scsi Write/xfer rdy complete */ + goto skip_term; + } else { + /* scsi read/xmit respond complete + * call handle dif to send scsi status + * rather than terminate exchange. + */ + cmd->state = QLA_TGT_STATE_PROCESSED; + ha->tgt.tgt_ops->handle_dif_err(cmd); + return; + } + } else { + /* Need to generate a SCSI good completion. + * because FW did not send scsi status. + */ + status = 0; + goto skip_term; + } + break; + } default: ql_dbg(ql_dbg_tgt_mgt, vha, 0xf05b, - "qla_target(%d): CTIO with error status " - "0x%x received (state %x, se_cmd %p\n", + "qla_target(%d): CTIO with error status 0x%x received (state %x, se_cmd %p\n", vha->vp_idx, status, cmd->state, se_cmd); break; } - if (cmd->state != QLA_TGT_STATE_NEED_DATA) + + /* "cmd->state == QLA_TGT_STATE_ABORTED" means + * cmd is already aborted/terminated, we don't + * need to terminate again. The exchange is already + * cleaned up/freed at FW level. Just cleanup at driver + * level. + */ + if ((cmd->state != QLA_TGT_STATE_NEED_DATA) && + (cmd->state != QLA_TGT_STATE_ABORTED)) { if (qlt_term_ctio_exchange(vha, ctio, cmd, status)) return; + } } +skip_term: if (cmd->state == QLA_TGT_STATE_PROCESSED) { ql_dbg(ql_dbg_tgt, vha, 0xe01f, "Command %p finished\n", cmd); @@ -2444,7 +3029,8 @@ static void qlt_do_ctio_completion(struct scsi_qla_host *vha, uint32_t handle, "not return a CTIO complete\n", vha->vp_idx, cmd->state); } - if (unlikely(status != CTIO_SUCCESS)) { + if (unlikely(status != CTIO_SUCCESS) && + (cmd->state != QLA_TGT_STATE_ABORTED)) { ql_dbg(ql_dbg_tgt_mgt, vha, 0xf01f, "Finishing failed CTIO\n"); dump_stack(); } @@ -2563,8 +3149,9 @@ static void qlt_do_work(struct work_struct *work) atio->u.isp24.fcp_cmnd.add_cdb_len])); ql_dbg(ql_dbg_tgt, vha, 0xe022, - "qla_target: START qla command: %p lun: 0x%04x (tag %d)\n", - cmd, cmd->unpacked_lun, cmd->tag); + "qla_target: START qla cmd: %p se_cmd %p lun: 0x%04x (tag %d) len(%d) ox_id %x\n", + cmd, &cmd->se_cmd, cmd->unpacked_lun, cmd->tag, data_length, + cmd->atio.u.isp24.fcp_hdr.ox_id); ret = vha->hw->tgt.tgt_ops->handle_cmd(vha, cmd, cdb, data_length, fcp_task_attr, data_dir, bidi); @@ -3527,11 +4114,11 @@ static void qlt_24xx_atio_pkt(struct scsi_qla_host *vha, switch (atio->u.raw.entry_type) { case ATIO_TYPE7: ql_dbg(ql_dbg_tgt, vha, 0xe02d, - "ATIO_TYPE7 instance %d, lun %Lx, read/write %d/%d, " - "add_cdb_len %d, data_length %04x, s_id %x:%x:%x\n", + "ATIO_TYPE7 instance %d, lun %Lx, read/write %d/%d, cdb %x, add_cdb_len %x, data_length %04x, s_id %02x%02x%02x\n", vha->vp_idx, atio->u.isp24.fcp_cmnd.lun, atio->u.isp24.fcp_cmnd.rddata, atio->u.isp24.fcp_cmnd.wrdata, + atio->u.isp24.fcp_cmnd.cdb[0], atio->u.isp24.fcp_cmnd.add_cdb_len, be32_to_cpu(get_unaligned((uint32_t *) &atio->u.isp24.fcp_cmnd.add_cdb[ @@ -3629,11 +4216,13 @@ static void qlt_response_pkt(struct scsi_qla_host *vha, response_t *pkt) tgt->irq_cmd_count++; switch (pkt->entry_type) { + case CTIO_CRC2: case CTIO_TYPE7: { struct ctio7_from_24xx *entry = (struct ctio7_from_24xx *)pkt; - ql_dbg(ql_dbg_tgt, vha, 0xe030, "CTIO_TYPE7: instance %d\n", - vha->vp_idx); + ql_dbg(ql_dbg_tgt, vha, 0xe030, + "CTIO[0x%x] 12/CTIO7 7A/CRC2: instance %d\n", + entry->entry_type, vha->vp_idx); qlt_do_ctio_completion(vha, entry->handle, le16_to_cpu(entry->status)|(pkt->entry_status << 16), entry); @@ -4768,6 +5357,7 @@ qlt_24xx_process_response_error(struct scsi_qla_host *vha, case ABTS_RESP_24XX: case CTIO_TYPE7: case NOTIFY_ACK_TYPE: + case CTIO_CRC2: return 1; default: return 0; diff --git a/drivers/scsi/qla2xxx/qla_target.h b/drivers/scsi/qla2xxx/qla_target.h index ce33d8c26406..f873e10451d2 100644 --- a/drivers/scsi/qla2xxx/qla_target.h +++ b/drivers/scsi/qla2xxx/qla_target.h @@ -293,6 +293,7 @@ struct ctio_to_2xxx { #define CTIO_ABORTED 0x02 #define CTIO_INVALID_RX_ID 0x08 #define CTIO_TIMEOUT 0x0B +#define CTIO_DIF_ERROR 0x0C /* DIF error detected */ #define CTIO_LIP_RESET 0x0E #define CTIO_TARGET_RESET 0x17 #define CTIO_PORT_UNAVAILABLE 0x28 @@ -498,11 +499,12 @@ struct ctio7_from_24xx { #define CTIO7_FLAGS_DONT_RET_CTIO BIT_8 #define CTIO7_FLAGS_STATUS_MODE_0 0 #define CTIO7_FLAGS_STATUS_MODE_1 BIT_6 +#define CTIO7_FLAGS_STATUS_MODE_2 BIT_7 #define CTIO7_FLAGS_EXPLICIT_CONFORM BIT_5 #define CTIO7_FLAGS_CONFIRM_SATISF BIT_4 #define CTIO7_FLAGS_DSD_PTR BIT_2 -#define CTIO7_FLAGS_DATA_IN BIT_1 -#define CTIO7_FLAGS_DATA_OUT BIT_0 +#define CTIO7_FLAGS_DATA_IN BIT_1 /* data to initiator */ +#define CTIO7_FLAGS_DATA_OUT BIT_0 /* data from initiator */ #define ELS_PLOGI 0x3 #define ELS_FLOGI 0x4 @@ -514,6 +516,68 @@ struct ctio7_from_24xx { #define ELS_ADISC 0x52 /* + *CTIO Type CRC_2 IOCB + */ +struct ctio_crc2_to_fw { + uint8_t entry_type; /* Entry type. */ +#define CTIO_CRC2 0x7A + uint8_t entry_count; /* Entry count. */ + uint8_t sys_define; /* System defined. */ + uint8_t entry_status; /* Entry Status. */ + + uint32_t handle; /* System handle. */ + uint16_t nport_handle; /* N_PORT handle. */ + uint16_t timeout; /* Command timeout. */ + + uint16_t dseg_count; /* Data segment count. */ + uint8_t vp_index; + uint8_t add_flags; /* additional flags */ +#define CTIO_CRC2_AF_DIF_DSD_ENA BIT_3 + + uint8_t initiator_id[3]; /* initiator ID */ + uint8_t reserved1; + uint32_t exchange_addr; /* rcv exchange address */ + uint16_t reserved2; + uint16_t flags; /* refer to CTIO7 flags values */ + uint32_t residual; + uint16_t ox_id; + uint16_t scsi_status; + uint32_t relative_offset; + uint32_t reserved5; + uint32_t transfer_length; /* total fc transfer length */ + uint32_t reserved6; + uint32_t crc_context_address[2];/* Data segment address. */ + uint16_t crc_context_len; /* Data segment length. */ + uint16_t reserved_1; /* MUST be set to 0. */ +} __packed; + +/* CTIO Type CRC_x Status IOCB */ +struct ctio_crc_from_fw { + uint8_t entry_type; /* Entry type. */ + uint8_t entry_count; /* Entry count. */ + uint8_t sys_define; /* System defined. */ + uint8_t entry_status; /* Entry Status. */ + + uint32_t handle; /* System handle. */ + uint16_t status; + uint16_t timeout; /* Command timeout. */ + uint16_t dseg_count; /* Data segment count. */ + uint32_t reserved1; + uint16_t state_flags; +#define CTIO_CRC_SF_DIF_CHOPPED BIT_4 + + uint32_t exchange_address; /* rcv exchange address */ + uint16_t reserved2; + uint16_t flags; + uint32_t resid_xfer_length; + uint16_t ox_id; + uint8_t reserved3[12]; + uint16_t runt_guard; /* reported runt blk guard */ + uint8_t actual_dif[8]; + uint8_t expected_dif[8]; +} __packed; + +/* * ISP queue - ABTS received/response entries structure definition for 24xx. */ #define ABTS_RECV_24XX 0x54 /* ABTS received (for 24xx) */ @@ -641,6 +705,7 @@ struct qla_tgt_func_tmpl { int (*handle_cmd)(struct scsi_qla_host *, struct qla_tgt_cmd *, unsigned char *, uint32_t, int, int, int); void (*handle_data)(struct qla_tgt_cmd *); + void (*handle_dif_err)(struct qla_tgt_cmd *); int (*handle_tmr)(struct qla_tgt_mgmt_cmd *, uint32_t, uint8_t, uint32_t); void (*free_cmd)(struct qla_tgt_cmd *); @@ -829,9 +894,9 @@ struct qla_tgt_sess { }; struct qla_tgt_cmd { + struct se_cmd se_cmd; struct qla_tgt_sess *sess; int state; - struct se_cmd se_cmd; struct work_struct free_work; struct work_struct work; /* Sense buffer that will be mapped into outgoing status */ @@ -843,6 +908,7 @@ struct qla_tgt_cmd { unsigned int free_sg:1; unsigned int aborted:1; /* Needed in case of SRR */ unsigned int write_data_transferred:1; + unsigned int ctx_dsd_alloced:1; struct scatterlist *sg; /* cmd data buffer SG vector */ int sg_cnt; /* SG segments count */ @@ -857,6 +923,12 @@ struct qla_tgt_cmd { struct scsi_qla_host *vha; struct atio_from_isp atio; + /* t10dif */ + struct scatterlist *prot_sg; + uint32_t prot_sg_cnt; + uint32_t blk_sz; + struct crc_context *ctx; + }; struct qla_tgt_sess_work_param { @@ -901,6 +973,10 @@ struct qla_tgt_prm { int sense_buffer_len; int residual; int add_status_pkt; + /* dif */ + struct scatterlist *prot_sg; + uint16_t prot_seg_cnt; + uint16_t tot_dsds; }; struct qla_tgt_srr_imm { @@ -976,6 +1052,8 @@ extern void qlt_24xx_atio_pkt_all_vps(struct scsi_qla_host *, extern void qlt_response_pkt_all_vps(struct scsi_qla_host *, response_t *); extern int qlt_rdy_to_xfer(struct qla_tgt_cmd *); extern int qlt_xmit_response(struct qla_tgt_cmd *, int, uint8_t); +extern int qlt_rdy_to_xfer_dif(struct qla_tgt_cmd *); +extern int qlt_xmit_response_dif(struct qla_tgt_cmd *, int, uint8_t); extern void qlt_xmit_tm_rsp(struct qla_tgt_mgmt_cmd *); extern void qlt_free_mcmd(struct qla_tgt_mgmt_cmd *); extern void qlt_free_cmd(struct qla_tgt_cmd *cmd); diff --git a/drivers/scsi/qla2xxx/qla_tmpl.c b/drivers/scsi/qla2xxx/qla_tmpl.c index a804e9b744bb..cb9a0c4bc419 100644 --- a/drivers/scsi/qla2xxx/qla_tmpl.c +++ b/drivers/scsi/qla2xxx/qla_tmpl.c @@ -1,6 +1,6 @@ /* * QLogic Fibre Channel HBA Driver - * Copyright (c) 2003-2013 QLogic Corporation + * Copyright (c) 2003-2014 QLogic Corporation * * See LICENSE.qla2xxx for copyright and licensing details. */ @@ -201,7 +201,6 @@ qla27xx_read_reg(__iomem struct device_reg_24xx *reg, ql_dbg(ql_dbg_misc, NULL, 0xd014, "%s: @%x\n", __func__, offset); } - qla27xx_insert32(offset, buf, len); qla27xx_read32(window, buf, len); } @@ -220,7 +219,7 @@ qla27xx_write_reg(__iomem struct device_reg_24xx *reg, static inline void qla27xx_read_window(__iomem struct device_reg_24xx *reg, - uint32_t base, uint offset, uint count, uint width, void *buf, + uint32_t addr, uint offset, uint count, uint width, void *buf, ulong *len) { void *window = (void *)reg + offset; @@ -229,14 +228,14 @@ qla27xx_read_window(__iomem struct device_reg_24xx *reg, if (buf) { ql_dbg(ql_dbg_misc, NULL, 0xd016, "%s: base=%x offset=%x count=%x width=%x\n", - __func__, base, offset, count, width); + __func__, addr, offset, count, width); } - qla27xx_write_reg(reg, IOBASE_ADDR, base, buf); + qla27xx_write_reg(reg, IOBASE_ADDR, addr, buf); while (count--) { - qla27xx_insert32(base, buf, len); + qla27xx_insert32(addr, buf, len); readn(window, buf, len); window += width; - base += width; + addr++; } } @@ -336,7 +335,8 @@ qla27xx_fwdt_entry_t260(struct scsi_qla_host *vha, ql_dbg(ql_dbg_misc, vha, 0xd204, "%s: rdpci [%lx]\n", __func__, *len); - qla27xx_read_reg(reg, ent->t260.pci_addr, buf, len); + qla27xx_insert32(ent->t260.pci_offset, buf, len); + qla27xx_read_reg(reg, ent->t260.pci_offset, buf, len); return false; } @@ -349,7 +349,7 @@ qla27xx_fwdt_entry_t261(struct scsi_qla_host *vha, ql_dbg(ql_dbg_misc, vha, 0xd205, "%s: wrpci [%lx]\n", __func__, *len); - qla27xx_write_reg(reg, ent->t261.pci_addr, ent->t261.write_data, buf); + qla27xx_write_reg(reg, ent->t261.pci_offset, ent->t261.write_data, buf); return false; } @@ -392,9 +392,9 @@ qla27xx_fwdt_entry_t262(struct scsi_qla_host *vha, goto done; } - if (end < start) { + if (end < start || end == 0) { ql_dbg(ql_dbg_misc, vha, 0xd023, - "%s: bad range (start=%x end=%x)\n", __func__, + "%s: unusable range (start=%x end=%x)\n", __func__, ent->t262.end_addr, ent->t262.start_addr); qla27xx_skip_entry(ent, buf); goto done; @@ -452,17 +452,15 @@ qla27xx_fwdt_entry_t263(struct scsi_qla_host *vha, ql_dbg(ql_dbg_misc, vha, 0xd025, "%s: unsupported atio queue\n", __func__); qla27xx_skip_entry(ent, buf); - goto done; } else { ql_dbg(ql_dbg_misc, vha, 0xd026, "%s: unknown queue %u\n", __func__, ent->t263.queue_type); qla27xx_skip_entry(ent, buf); - goto done; } if (buf) ent->t263.num_queues = count; -done: + return false; } @@ -503,7 +501,7 @@ qla27xx_fwdt_entry_t265(struct scsi_qla_host *vha, ql_dbg(ql_dbg_misc, vha, 0xd209, "%s: pause risc [%lx]\n", __func__, *len); if (buf) - qla24xx_pause_risc(reg); + qla24xx_pause_risc(reg, vha->hw); return false; } @@ -590,7 +588,6 @@ qla27xx_fwdt_entry_t270(struct scsi_qla_host *vha, struct qla27xx_fwdt_entry *ent, void *buf, ulong *len) { struct device_reg_24xx __iomem *reg = qla27xx_isp_reg(vha); - void *window = (void *)reg + 0xc4; ulong dwords = ent->t270.count; ulong addr = ent->t270.addr; @@ -599,10 +596,9 @@ qla27xx_fwdt_entry_t270(struct scsi_qla_host *vha, qla27xx_write_reg(reg, IOBASE_ADDR, 0x40, buf); while (dwords--) { qla27xx_write_reg(reg, 0xc0, addr|0x80000000, buf); - qla27xx_read_reg(reg, 0xc4, buf, len); qla27xx_insert32(addr, buf, len); - qla27xx_read32(window, buf, len); - addr++; + qla27xx_read_reg(reg, 0xc4, buf, len); + addr += sizeof(uint32_t); } return false; @@ -614,12 +610,12 @@ qla27xx_fwdt_entry_t271(struct scsi_qla_host *vha, { struct device_reg_24xx __iomem *reg = qla27xx_isp_reg(vha); ulong addr = ent->t271.addr; + ulong data = ent->t271.data; ql_dbg(ql_dbg_misc, vha, 0xd20f, "%s: wrremreg [%lx]\n", __func__, *len); qla27xx_write_reg(reg, IOBASE_ADDR, 0x40, buf); - qla27xx_read_reg(reg, 0xc4, buf, len); - qla27xx_insert32(addr, buf, len); + qla27xx_write_reg(reg, 0xc4, data, buf); qla27xx_write_reg(reg, 0xc0, addr, buf); return false; @@ -662,9 +658,59 @@ qla27xx_fwdt_entry_t273(struct scsi_qla_host *vha, "%s: failed pcicfg read at %lx\n", __func__, addr); qla27xx_insert32(addr, buf, len); qla27xx_insert32(value, buf, len); - addr += 4; + addr += sizeof(uint32_t); + } + + return false; +} + +static int +qla27xx_fwdt_entry_t274(struct scsi_qla_host *vha, + struct qla27xx_fwdt_entry *ent, void *buf, ulong *len) +{ + uint count = 0; + uint i; + + ql_dbg(ql_dbg_misc, vha, 0xd212, + "%s: getqsh(%x) [%lx]\n", __func__, ent->t274.queue_type, *len); + if (ent->t274.queue_type == T274_QUEUE_TYPE_REQ_SHAD) { + for (i = 0; i < vha->hw->max_req_queues; i++) { + struct req_que *req = vha->hw->req_q_map[i]; + if (req || !buf) { + qla27xx_insert16(i, buf, len); + qla27xx_insert16(1, buf, len); + qla27xx_insert32(req && req->out_ptr ? + *req->out_ptr : 0, buf, len); + count++; + } + } + } else if (ent->t274.queue_type == T274_QUEUE_TYPE_RSP_SHAD) { + for (i = 0; i < vha->hw->max_rsp_queues; i++) { + struct rsp_que *rsp = vha->hw->rsp_q_map[i]; + if (rsp || !buf) { + qla27xx_insert16(i, buf, len); + qla27xx_insert16(1, buf, len); + qla27xx_insert32(rsp && rsp->in_ptr ? + *rsp->in_ptr : 0, buf, len); + count++; + } + } + } else if (ent->t274.queue_type == T274_QUEUE_TYPE_ATIO_SHAD) { + ql_dbg(ql_dbg_misc, vha, 0xd02e, + "%s: unsupported atio queue\n", __func__); + qla27xx_skip_entry(ent, buf); + } else { + ql_dbg(ql_dbg_misc, vha, 0xd02f, + "%s: unknown queue %u\n", __func__, ent->t274.queue_type); + qla27xx_skip_entry(ent, buf); } + if (buf) + ent->t274.num_queues = count; + + if (!count) + qla27xx_skip_entry(ent, buf); + return false; } @@ -709,6 +755,7 @@ static struct qla27xx_fwdt_entry_call ql27xx_fwdt_entry_call_list[] = { { ENTRY_TYPE_WRREMREG , qla27xx_fwdt_entry_t271 } , { ENTRY_TYPE_RDREMRAM , qla27xx_fwdt_entry_t272 } , { ENTRY_TYPE_PCICFG , qla27xx_fwdt_entry_t273 } , + { ENTRY_TYPE_GET_SHADOW , qla27xx_fwdt_entry_t274 } , { -1 , qla27xx_fwdt_entry_other } }; diff --git a/drivers/scsi/qla2xxx/qla_tmpl.h b/drivers/scsi/qla2xxx/qla_tmpl.h index c9d2fff4d964..1967424c8e64 100644 --- a/drivers/scsi/qla2xxx/qla_tmpl.h +++ b/drivers/scsi/qla2xxx/qla_tmpl.h @@ -1,6 +1,6 @@ /* * QLogic Fibre Channel HBA Driver - * Copyright (c) 2003-2013 QLogic Corporation + * Copyright (c) 2003-2014 QLogic Corporation * * See LICENSE.qla2xxx for copyright and licensing details. */ @@ -52,6 +52,7 @@ struct __packed qla27xx_fwdt_template { #define ENTRY_TYPE_WRREMREG 271 #define ENTRY_TYPE_RDREMRAM 272 #define ENTRY_TYPE_PCICFG 273 +#define ENTRY_TYPE_GET_SHADOW 274 #define CAPTURE_FLAG_PHYS_ONLY BIT_0 #define CAPTURE_FLAG_PHYS_VIRT BIT_1 @@ -109,12 +110,12 @@ struct __packed qla27xx_fwdt_entry { } t259; struct __packed { - uint8_t pci_addr; + uint8_t pci_offset; uint8_t reserved[3]; } t260; struct __packed { - uint8_t pci_addr; + uint8_t pci_offset; uint8_t reserved[3]; uint32_t write_data; } t261; @@ -186,6 +187,12 @@ struct __packed qla27xx_fwdt_entry { uint32_t addr; uint32_t count; } t273; + + struct __packed { + uint32_t num_queues; + uint8_t queue_type; + uint8_t reserved[3]; + } t274; }; }; @@ -202,4 +209,8 @@ struct __packed qla27xx_fwdt_entry { #define T268_BUF_TYPE_EXCH_BUFOFF 2 #define T268_BUF_TYPE_EXTD_LOGIN 3 +#define T274_QUEUE_TYPE_REQ_SHAD 1 +#define T274_QUEUE_TYPE_RSP_SHAD 2 +#define T274_QUEUE_TYPE_ATIO_SHAD 3 + #endif diff --git a/drivers/scsi/qla2xxx/qla_version.h b/drivers/scsi/qla2xxx/qla_version.h index e36b94712544..4d2c98cbec4f 100644 --- a/drivers/scsi/qla2xxx/qla_version.h +++ b/drivers/scsi/qla2xxx/qla_version.h @@ -1,13 +1,13 @@ /* * QLogic Fibre Channel HBA Driver - * Copyright (c) 2003-2013 QLogic Corporation + * Copyright (c) 2003-2014 QLogic Corporation * * See LICENSE.qla2xxx for copyright and licensing details. */ /* * Driver version */ -#define QLA2XXX_VERSION "8.07.00.02-k" +#define QLA2XXX_VERSION "8.07.00.08-k" #define QLA_DRIVER_MAJOR_VER 8 #define QLA_DRIVER_MINOR_VER 7 diff --git a/drivers/scsi/qla2xxx/tcm_qla2xxx.c b/drivers/scsi/qla2xxx/tcm_qla2xxx.c index 68fb66fdb757..896cb23adb77 100644 --- a/drivers/scsi/qla2xxx/tcm_qla2xxx.c +++ b/drivers/scsi/qla2xxx/tcm_qla2xxx.c @@ -472,6 +472,11 @@ static int tcm_qla2xxx_write_pending(struct se_cmd *se_cmd) cmd->sg_cnt = se_cmd->t_data_nents; cmd->sg = se_cmd->t_data_sg; + cmd->prot_sg_cnt = se_cmd->t_prot_nents; + cmd->prot_sg = se_cmd->t_prot_sg; + cmd->blk_sz = se_cmd->se_dev->dev_attrib.block_size; + se_cmd->pi_err = 0; + /* * qla_target.c:qlt_rdy_to_xfer() will call pci_map_sg() to setup * the SGL mappings into PCIe memory for incoming FCP WRITE data. @@ -567,8 +572,13 @@ static void tcm_qla2xxx_handle_data_work(struct work_struct *work) return; } - transport_generic_request_failure(&cmd->se_cmd, - TCM_CHECK_CONDITION_ABORT_CMD); + if (cmd->se_cmd.pi_err) + transport_generic_request_failure(&cmd->se_cmd, + cmd->se_cmd.pi_err); + else + transport_generic_request_failure(&cmd->se_cmd, + TCM_CHECK_CONDITION_ABORT_CMD); + return; } @@ -584,6 +594,27 @@ static void tcm_qla2xxx_handle_data(struct qla_tgt_cmd *cmd) queue_work(tcm_qla2xxx_free_wq, &cmd->work); } +static void tcm_qla2xxx_handle_dif_work(struct work_struct *work) +{ + struct qla_tgt_cmd *cmd = container_of(work, struct qla_tgt_cmd, work); + + /* take an extra kref to prevent cmd free too early. + * need to wait for SCSI status/check condition to + * finish responding generate by transport_generic_request_failure. + */ + kref_get(&cmd->se_cmd.cmd_kref); + transport_generic_request_failure(&cmd->se_cmd, cmd->se_cmd.pi_err); +} + +/* + * Called from qla_target.c:qlt_do_ctio_completion() + */ +static void tcm_qla2xxx_handle_dif_err(struct qla_tgt_cmd *cmd) +{ + INIT_WORK(&cmd->work, tcm_qla2xxx_handle_dif_work); + queue_work(tcm_qla2xxx_free_wq, &cmd->work); +} + /* * Called from qla_target.c:qlt_issue_task_mgmt() */ @@ -610,6 +641,11 @@ static int tcm_qla2xxx_queue_data_in(struct se_cmd *se_cmd) cmd->sg = se_cmd->t_data_sg; cmd->offset = 0; + cmd->prot_sg_cnt = se_cmd->t_prot_nents; + cmd->prot_sg = se_cmd->t_prot_sg; + cmd->blk_sz = se_cmd->se_dev->dev_attrib.block_size; + se_cmd->pi_err = 0; + /* * Now queue completed DATA_IN the qla2xxx LLD and response ring */ @@ -1600,6 +1636,7 @@ static void tcm_qla2xxx_update_sess(struct qla_tgt_sess *sess, port_id_t s_id, static struct qla_tgt_func_tmpl tcm_qla2xxx_template = { .handle_cmd = tcm_qla2xxx_handle_cmd, .handle_data = tcm_qla2xxx_handle_data, + .handle_dif_err = tcm_qla2xxx_handle_dif_err, .handle_tmr = tcm_qla2xxx_handle_tmr, .free_cmd = tcm_qla2xxx_free_cmd, .free_mcmd = tcm_qla2xxx_free_mcmd, diff --git a/drivers/scsi/qla4xxx/ql4_83xx.c b/drivers/scsi/qla4xxx/ql4_83xx.c index 2eba35365920..556c1525f881 100644 --- a/drivers/scsi/qla4xxx/ql4_83xx.c +++ b/drivers/scsi/qla4xxx/ql4_83xx.c @@ -249,110 +249,6 @@ void qla4_83xx_rom_lock_recovery(struct scsi_qla_host *ha) qla4_83xx_flash_unlock(ha); } -/** - * qla4_83xx_ms_mem_write_128b - Writes data to MS/off-chip memory - * @ha: Pointer to adapter structure - * @addr: Flash address to write to - * @data: Data to be written - * @count: word_count to be written - * - * Return: On success return QLA_SUCCESS - * On error return QLA_ERROR - **/ -int qla4_83xx_ms_mem_write_128b(struct scsi_qla_host *ha, uint64_t addr, - uint32_t *data, uint32_t count) -{ - int i, j; - uint32_t agt_ctrl; - unsigned long flags; - int ret_val = QLA_SUCCESS; - - /* Only 128-bit aligned access */ - if (addr & 0xF) { - ret_val = QLA_ERROR; - goto exit_ms_mem_write; - } - - write_lock_irqsave(&ha->hw_lock, flags); - - /* Write address */ - ret_val = qla4_83xx_wr_reg_indirect(ha, MD_MIU_TEST_AGT_ADDR_HI, 0); - if (ret_val == QLA_ERROR) { - ql4_printk(KERN_ERR, ha, "%s: write to AGT_ADDR_HI failed\n", - __func__); - goto exit_ms_mem_write_unlock; - } - - for (i = 0; i < count; i++, addr += 16) { - if (!((QLA8XXX_ADDR_IN_RANGE(addr, QLA8XXX_ADDR_QDR_NET, - QLA8XXX_ADDR_QDR_NET_MAX)) || - (QLA8XXX_ADDR_IN_RANGE(addr, QLA8XXX_ADDR_DDR_NET, - QLA8XXX_ADDR_DDR_NET_MAX)))) { - ret_val = QLA_ERROR; - goto exit_ms_mem_write_unlock; - } - - ret_val = qla4_83xx_wr_reg_indirect(ha, MD_MIU_TEST_AGT_ADDR_LO, - addr); - /* Write data */ - ret_val |= qla4_83xx_wr_reg_indirect(ha, - MD_MIU_TEST_AGT_WRDATA_LO, - *data++); - ret_val |= qla4_83xx_wr_reg_indirect(ha, - MD_MIU_TEST_AGT_WRDATA_HI, - *data++); - ret_val |= qla4_83xx_wr_reg_indirect(ha, - MD_MIU_TEST_AGT_WRDATA_ULO, - *data++); - ret_val |= qla4_83xx_wr_reg_indirect(ha, - MD_MIU_TEST_AGT_WRDATA_UHI, - *data++); - if (ret_val == QLA_ERROR) { - ql4_printk(KERN_ERR, ha, "%s: write to AGT_WRDATA failed\n", - __func__); - goto exit_ms_mem_write_unlock; - } - - /* Check write status */ - ret_val = qla4_83xx_wr_reg_indirect(ha, MD_MIU_TEST_AGT_CTRL, - MIU_TA_CTL_WRITE_ENABLE); - ret_val |= qla4_83xx_wr_reg_indirect(ha, MD_MIU_TEST_AGT_CTRL, - MIU_TA_CTL_WRITE_START); - if (ret_val == QLA_ERROR) { - ql4_printk(KERN_ERR, ha, "%s: write to AGT_CTRL failed\n", - __func__); - goto exit_ms_mem_write_unlock; - } - - for (j = 0; j < MAX_CTL_CHECK; j++) { - ret_val = qla4_83xx_rd_reg_indirect(ha, - MD_MIU_TEST_AGT_CTRL, - &agt_ctrl); - if (ret_val == QLA_ERROR) { - ql4_printk(KERN_ERR, ha, "%s: failed to read MD_MIU_TEST_AGT_CTRL\n", - __func__); - goto exit_ms_mem_write_unlock; - } - if ((agt_ctrl & MIU_TA_CTL_BUSY) == 0) - break; - } - - /* Status check failed */ - if (j >= MAX_CTL_CHECK) { - printk_ratelimited(KERN_ERR "%s: MS memory write failed!\n", - __func__); - ret_val = QLA_ERROR; - goto exit_ms_mem_write_unlock; - } - } - -exit_ms_mem_write_unlock: - write_unlock_irqrestore(&ha->hw_lock, flags); - -exit_ms_mem_write: - return ret_val; -} - #define INTENT_TO_RECOVER 0x01 #define PROCEED_TO_RECOVER 0x02 @@ -760,7 +656,7 @@ static int qla4_83xx_copy_bootloader(struct scsi_qla_host *ha) __func__)); /* 128 bit/16 byte write to MS memory */ - ret_val = qla4_83xx_ms_mem_write_128b(ha, dest, (uint32_t *)p_cache, + ret_val = qla4_8xxx_ms_mem_write_128b(ha, dest, (uint32_t *)p_cache, count); if (ret_val == QLA_ERROR) { ql4_printk(KERN_ERR, ha, "%s: Error writing firmware to MS\n", diff --git a/drivers/scsi/qla4xxx/ql4_83xx.h b/drivers/scsi/qla4xxx/ql4_83xx.h index a0de6e25ea5a..775fdf9fcc87 100644 --- a/drivers/scsi/qla4xxx/ql4_83xx.h +++ b/drivers/scsi/qla4xxx/ql4_83xx.h @@ -254,6 +254,50 @@ struct qla83xx_minidump_entry_pollrd { uint32_t rsvd_1; }; +struct qla8044_minidump_entry_rddfe { + struct qla8xxx_minidump_entry_hdr h; + uint32_t addr_1; + uint32_t value; + uint8_t stride; + uint8_t stride2; + uint16_t count; + uint32_t poll; + uint32_t mask; + uint32_t modify_mask; + uint32_t data_size; + uint32_t rsvd; + +} __packed; + +struct qla8044_minidump_entry_rdmdio { + struct qla8xxx_minidump_entry_hdr h; + + uint32_t addr_1; + uint32_t addr_2; + uint32_t value_1; + uint8_t stride_1; + uint8_t stride_2; + uint16_t count; + uint32_t poll; + uint32_t mask; + uint32_t value_2; + uint32_t data_size; + +} __packed; + +struct qla8044_minidump_entry_pollwr { + struct qla8xxx_minidump_entry_hdr h; + uint32_t addr_1; + uint32_t addr_2; + uint32_t value_1; + uint32_t value_2; + uint32_t poll; + uint32_t mask; + uint32_t data_size; + uint32_t rsvd; + +} __packed; + /* RDMUX2 Entry */ struct qla83xx_minidump_entry_rdmux2 { struct qla8xxx_minidump_entry_hdr h; diff --git a/drivers/scsi/qla4xxx/ql4_def.h b/drivers/scsi/qla4xxx/ql4_def.h index 73a502288bde..8f6d0fb2cd80 100644 --- a/drivers/scsi/qla4xxx/ql4_def.h +++ b/drivers/scsi/qla4xxx/ql4_def.h @@ -601,6 +601,7 @@ struct scsi_qla_host { #define DPC_HA_NEED_QUIESCENT 22 /* 0x00400000 ISP-82xx only*/ #define DPC_POST_IDC_ACK 23 /* 0x00800000 */ #define DPC_RESTORE_ACB 24 /* 0x01000000 */ +#define DPC_SYSFS_DDB_EXPORT 25 /* 0x02000000 */ struct Scsi_Host *host; /* pointer to host data */ uint32_t tot_ddbs; diff --git a/drivers/scsi/qla4xxx/ql4_fw.h b/drivers/scsi/qla4xxx/ql4_fw.h index 209853ce0bbc..699575efc9ba 100644 --- a/drivers/scsi/qla4xxx/ql4_fw.h +++ b/drivers/scsi/qla4xxx/ql4_fw.h @@ -1415,6 +1415,9 @@ struct ql_iscsi_stats { #define QLA83XX_DBG_OCM_WNDREG_ARRAY_LEN 16 #define QLA83XX_SS_OCM_WNDREG_INDEX 3 #define QLA83XX_SS_PCI_INDEX 0 +#define QLA8022_TEMPLATE_CAP_OFFSET 172 +#define QLA83XX_TEMPLATE_CAP_OFFSET 268 +#define QLA80XX_TEMPLATE_RESERVED_BITS 16 struct qla4_8xxx_minidump_template_hdr { uint32_t entry_type; @@ -1434,6 +1437,7 @@ struct qla4_8xxx_minidump_template_hdr { uint32_t saved_state_array[QLA8XXX_DBG_STATE_ARRAY_LEN]; uint32_t capture_size_array[QLA8XXX_DBG_CAP_SIZE_ARRAY_LEN]; uint32_t ocm_window_reg[QLA83XX_DBG_OCM_WNDREG_ARRAY_LEN]; + uint32_t capabilities[QLA80XX_TEMPLATE_RESERVED_BITS]; }; #endif /* _QLA4X_FW_H */ diff --git a/drivers/scsi/qla4xxx/ql4_glbl.h b/drivers/scsi/qla4xxx/ql4_glbl.h index b1a19cd8d5b2..5f58b451327e 100644 --- a/drivers/scsi/qla4xxx/ql4_glbl.h +++ b/drivers/scsi/qla4xxx/ql4_glbl.h @@ -274,13 +274,14 @@ int qla4xxx_set_acb(struct scsi_qla_host *ha, uint32_t *mbox_cmd, int qla4xxx_get_acb(struct scsi_qla_host *ha, dma_addr_t acb_dma, uint32_t acb_type, uint32_t len); int qla4_84xx_config_acb(struct scsi_qla_host *ha, int acb_config); -int qla4_83xx_ms_mem_write_128b(struct scsi_qla_host *ha, +int qla4_8xxx_ms_mem_write_128b(struct scsi_qla_host *ha, uint64_t addr, uint32_t *data, uint32_t count); uint8_t qla4xxx_set_ipaddr_state(uint8_t fw_ipaddr_state); int qla4_83xx_get_port_config(struct scsi_qla_host *ha, uint32_t *config); int qla4_83xx_set_port_config(struct scsi_qla_host *ha, uint32_t *config); int qla4_8xxx_check_init_adapter_retry(struct scsi_qla_host *ha); int qla4_83xx_is_detached(struct scsi_qla_host *ha); +int qla4xxx_sysfs_ddb_export(struct scsi_qla_host *ha); extern int ql4xextended_error_logging; extern int ql4xdontresethba; diff --git a/drivers/scsi/qla4xxx/ql4_init.c b/drivers/scsi/qla4xxx/ql4_init.c index 28fbece7e08f..6f12f859b11d 100644 --- a/drivers/scsi/qla4xxx/ql4_init.c +++ b/drivers/scsi/qla4xxx/ql4_init.c @@ -282,6 +282,25 @@ qla4xxx_wait_for_ip_config(struct scsi_qla_host *ha) return ipv4_wait|ipv6_wait; } +static int qla4_80xx_is_minidump_dma_capable(struct scsi_qla_host *ha, + struct qla4_8xxx_minidump_template_hdr *md_hdr) +{ + int offset = (is_qla8022(ha)) ? QLA8022_TEMPLATE_CAP_OFFSET : + QLA83XX_TEMPLATE_CAP_OFFSET; + int rval = 1; + uint32_t *cap_offset; + + cap_offset = (uint32_t *)((char *)md_hdr + offset); + + if (!(le32_to_cpu(*cap_offset) & BIT_0)) { + ql4_printk(KERN_INFO, ha, "PEX DMA Not supported %d\n", + *cap_offset); + rval = 0; + } + + return rval; +} + /** * qla4xxx_alloc_fw_dump - Allocate memory for minidump data. * @ha: pointer to host adapter structure. @@ -294,6 +313,7 @@ void qla4xxx_alloc_fw_dump(struct scsi_qla_host *ha) void *md_tmp; dma_addr_t md_tmp_dma; struct qla4_8xxx_minidump_template_hdr *md_hdr; + int dma_capable; if (ha->fw_dump) { ql4_printk(KERN_WARNING, ha, @@ -326,13 +346,19 @@ void qla4xxx_alloc_fw_dump(struct scsi_qla_host *ha) md_hdr = (struct qla4_8xxx_minidump_template_hdr *)md_tmp; + dma_capable = qla4_80xx_is_minidump_dma_capable(ha, md_hdr); + capture_debug_level = md_hdr->capture_debug_level; /* Get capture mask based on module loadtime setting. */ - if (ql4xmdcapmask >= 0x3 && ql4xmdcapmask <= 0x7F) + if ((ql4xmdcapmask >= 0x3 && ql4xmdcapmask <= 0x7F) || + (ql4xmdcapmask == 0xFF && dma_capable)) { ha->fw_dump_capture_mask = ql4xmdcapmask; - else + } else { + if (ql4xmdcapmask == 0xFF) + ql4_printk(KERN_INFO, ha, "Falling back to default capture mask, as PEX DMA is not supported\n"); ha->fw_dump_capture_mask = capture_debug_level; + } md_hdr->driver_capture_mask = ha->fw_dump_capture_mask; @@ -864,6 +890,8 @@ int qla4xxx_start_firmware(struct scsi_qla_host *ha) if (status == QLA_SUCCESS) { if (test_and_clear_bit(AF_GET_CRASH_RECORD, &ha->flags)) qla4xxx_get_crash_record(ha); + + qla4xxx_init_rings(ha); } else { DEBUG(printk("scsi%ld: %s: Firmware has NOT started\n", ha->host_no, __func__)); diff --git a/drivers/scsi/qla4xxx/ql4_isr.c b/drivers/scsi/qla4xxx/ql4_isr.c index b1925d195f41..081b6b78d2c6 100644 --- a/drivers/scsi/qla4xxx/ql4_isr.c +++ b/drivers/scsi/qla4xxx/ql4_isr.c @@ -1526,7 +1526,7 @@ void qla4xxx_process_aen(struct scsi_qla_host * ha, uint8_t process_aen) int qla4xxx_request_irqs(struct scsi_qla_host *ha) { - int ret; + int ret = 0; int rval = QLA_ERROR; if (is_qla40XX(ha)) @@ -1580,15 +1580,13 @@ try_msi: } } - /* - * Prevent interrupts from falling back to INTx mode in cases where - * interrupts cannot get acquired through MSI-X or MSI mode. - */ +try_intx: if (is_qla8022(ha)) { - ql4_printk(KERN_WARNING, ha, "IRQ not attached -- %d.\n", ret); + ql4_printk(KERN_WARNING, ha, "%s: ISP82xx Legacy interrupt not supported\n", + __func__); goto irq_not_attached; } -try_intx: + /* Trying INTx */ ret = request_irq(ha->pdev->irq, ha->isp_ops->intr_handler, IRQF_SHARED, DRIVER_NAME, ha); diff --git a/drivers/scsi/qla4xxx/ql4_mbx.c b/drivers/scsi/qla4xxx/ql4_mbx.c index 0a6b782d6fdb..0a3312c6dd6d 100644 --- a/drivers/scsi/qla4xxx/ql4_mbx.c +++ b/drivers/scsi/qla4xxx/ql4_mbx.c @@ -2381,7 +2381,7 @@ int qla4_84xx_config_acb(struct scsi_qla_host *ha, int acb_config) ql4_printk(KERN_ERR, ha, "%s: Unable to alloc acb\n", __func__); rval = QLA_ERROR; - goto exit_config_acb; + goto exit_free_acb; } memcpy(ha->saved_acb, acb, acb_len); break; @@ -2395,8 +2395,6 @@ int qla4_84xx_config_acb(struct scsi_qla_host *ha, int acb_config) } memcpy(acb, ha->saved_acb, acb_len); - kfree(ha->saved_acb); - ha->saved_acb = NULL; rval = qla4xxx_set_acb(ha, &mbox_cmd[0], &mbox_sts[0], acb_dma); if (rval != QLA_SUCCESS) @@ -2412,6 +2410,10 @@ exit_free_acb: dma_free_coherent(&ha->pdev->dev, sizeof(struct addr_ctrl_blk), acb, acb_dma); exit_config_acb: + if ((acb_config == ACB_CONFIG_SET) && ha->saved_acb) { + kfree(ha->saved_acb); + ha->saved_acb = NULL; + } DEBUG2(ql4_printk(KERN_INFO, ha, "%s %s\n", __func__, rval == QLA_SUCCESS ? "SUCCEEDED" : "FAILED")); diff --git a/drivers/scsi/qla4xxx/ql4_nx.c b/drivers/scsi/qla4xxx/ql4_nx.c index 63328c812b70..9dbdb4be2d8f 100644 --- a/drivers/scsi/qla4xxx/ql4_nx.c +++ b/drivers/scsi/qla4xxx/ql4_nx.c @@ -14,6 +14,7 @@ #include <asm-generic/io-64-nonatomic-lo-hi.h> +#define TIMEOUT_100_MS 100 #define MASK(n) DMA_BIT_MASK(n) #define MN_WIN(addr) (((addr & 0x1fc0000) >> 1) | ((addr >> 25) & 0x3ff)) #define OCM_WIN(addr) (((addr & 0x1ff0000) >> 1) | ((addr >> 25) & 0x3ff)) @@ -1176,6 +1177,112 @@ qla4_82xx_pinit_from_rom(struct scsi_qla_host *ha, int verbose) return 0; } +/** + * qla4_8xxx_ms_mem_write_128b - Writes data to MS/off-chip memory + * @ha: Pointer to adapter structure + * @addr: Flash address to write to + * @data: Data to be written + * @count: word_count to be written + * + * Return: On success return QLA_SUCCESS + * On error return QLA_ERROR + **/ +int qla4_8xxx_ms_mem_write_128b(struct scsi_qla_host *ha, uint64_t addr, + uint32_t *data, uint32_t count) +{ + int i, j; + uint32_t agt_ctrl; + unsigned long flags; + int ret_val = QLA_SUCCESS; + + /* Only 128-bit aligned access */ + if (addr & 0xF) { + ret_val = QLA_ERROR; + goto exit_ms_mem_write; + } + + write_lock_irqsave(&ha->hw_lock, flags); + + /* Write address */ + ret_val = ha->isp_ops->wr_reg_indirect(ha, MD_MIU_TEST_AGT_ADDR_HI, 0); + if (ret_val == QLA_ERROR) { + ql4_printk(KERN_ERR, ha, "%s: write to AGT_ADDR_HI failed\n", + __func__); + goto exit_ms_mem_write_unlock; + } + + for (i = 0; i < count; i++, addr += 16) { + if (!((QLA8XXX_ADDR_IN_RANGE(addr, QLA8XXX_ADDR_QDR_NET, + QLA8XXX_ADDR_QDR_NET_MAX)) || + (QLA8XXX_ADDR_IN_RANGE(addr, QLA8XXX_ADDR_DDR_NET, + QLA8XXX_ADDR_DDR_NET_MAX)))) { + ret_val = QLA_ERROR; + goto exit_ms_mem_write_unlock; + } + + ret_val = ha->isp_ops->wr_reg_indirect(ha, + MD_MIU_TEST_AGT_ADDR_LO, + addr); + /* Write data */ + ret_val |= ha->isp_ops->wr_reg_indirect(ha, + MD_MIU_TEST_AGT_WRDATA_LO, + *data++); + ret_val |= ha->isp_ops->wr_reg_indirect(ha, + MD_MIU_TEST_AGT_WRDATA_HI, + *data++); + ret_val |= ha->isp_ops->wr_reg_indirect(ha, + MD_MIU_TEST_AGT_WRDATA_ULO, + *data++); + ret_val |= ha->isp_ops->wr_reg_indirect(ha, + MD_MIU_TEST_AGT_WRDATA_UHI, + *data++); + if (ret_val == QLA_ERROR) { + ql4_printk(KERN_ERR, ha, "%s: write to AGT_WRDATA failed\n", + __func__); + goto exit_ms_mem_write_unlock; + } + + /* Check write status */ + ret_val = ha->isp_ops->wr_reg_indirect(ha, MD_MIU_TEST_AGT_CTRL, + MIU_TA_CTL_WRITE_ENABLE); + ret_val |= ha->isp_ops->wr_reg_indirect(ha, + MD_MIU_TEST_AGT_CTRL, + MIU_TA_CTL_WRITE_START); + if (ret_val == QLA_ERROR) { + ql4_printk(KERN_ERR, ha, "%s: write to AGT_CTRL failed\n", + __func__); + goto exit_ms_mem_write_unlock; + } + + for (j = 0; j < MAX_CTL_CHECK; j++) { + ret_val = ha->isp_ops->rd_reg_indirect(ha, + MD_MIU_TEST_AGT_CTRL, + &agt_ctrl); + if (ret_val == QLA_ERROR) { + ql4_printk(KERN_ERR, ha, "%s: failed to read MD_MIU_TEST_AGT_CTRL\n", + __func__); + goto exit_ms_mem_write_unlock; + } + if ((agt_ctrl & MIU_TA_CTL_BUSY) == 0) + break; + } + + /* Status check failed */ + if (j >= MAX_CTL_CHECK) { + printk_ratelimited(KERN_ERR "%s: MS memory write failed!\n", + __func__); + ret_val = QLA_ERROR; + goto exit_ms_mem_write_unlock; + } + } + +exit_ms_mem_write_unlock: + write_unlock_irqrestore(&ha->hw_lock, flags); + +exit_ms_mem_write: + return ret_val; +} + static int qla4_82xx_load_from_flash(struct scsi_qla_host *ha, uint32_t image_start) { @@ -1714,6 +1821,101 @@ void qla4_82xx_rom_lock_recovery(struct scsi_qla_host *ha) qla4_82xx_rom_unlock(ha); } +static uint32_t ql4_84xx_poll_wait_for_ready(struct scsi_qla_host *ha, + uint32_t addr1, uint32_t mask) +{ + unsigned long timeout; + uint32_t rval = QLA_SUCCESS; + uint32_t temp; + + timeout = jiffies + msecs_to_jiffies(TIMEOUT_100_MS); + do { + ha->isp_ops->rd_reg_indirect(ha, addr1, &temp); + if ((temp & mask) != 0) + break; + + if (time_after_eq(jiffies, timeout)) { + ql4_printk(KERN_INFO, ha, "Error in processing rdmdio entry\n"); + return QLA_ERROR; + } + } while (1); + + return rval; +} + +uint32_t ql4_84xx_ipmdio_rd_reg(struct scsi_qla_host *ha, uint32_t addr1, + uint32_t addr3, uint32_t mask, uint32_t addr, + uint32_t *data_ptr) +{ + int rval = QLA_SUCCESS; + uint32_t temp; + uint32_t data; + + rval = ql4_84xx_poll_wait_for_ready(ha, addr1, mask); + if (rval) + goto exit_ipmdio_rd_reg; + + temp = (0x40000000 | addr); + ha->isp_ops->wr_reg_indirect(ha, addr1, temp); + + rval = ql4_84xx_poll_wait_for_ready(ha, addr1, mask); + if (rval) + goto exit_ipmdio_rd_reg; + + ha->isp_ops->rd_reg_indirect(ha, addr3, &data); + *data_ptr = data; + +exit_ipmdio_rd_reg: + return rval; +} + + +static uint32_t ql4_84xx_poll_wait_ipmdio_bus_idle(struct scsi_qla_host *ha, + uint32_t addr1, + uint32_t addr2, + uint32_t addr3, + uint32_t mask) +{ + unsigned long timeout; + uint32_t temp; + uint32_t rval = QLA_SUCCESS; + + timeout = jiffies + msecs_to_jiffies(TIMEOUT_100_MS); + do { + ql4_84xx_ipmdio_rd_reg(ha, addr1, addr3, mask, addr2, &temp); + if ((temp & 0x1) != 1) + break; + if (time_after_eq(jiffies, timeout)) { + ql4_printk(KERN_INFO, ha, "Error in processing mdiobus idle\n"); + return QLA_ERROR; + } + } while (1); + + return rval; +} + +static int ql4_84xx_ipmdio_wr_reg(struct scsi_qla_host *ha, + uint32_t addr1, uint32_t addr3, + uint32_t mask, uint32_t addr, + uint32_t value) +{ + int rval = QLA_SUCCESS; + + rval = ql4_84xx_poll_wait_for_ready(ha, addr1, mask); + if (rval) + goto exit_ipmdio_wr_reg; + + ha->isp_ops->wr_reg_indirect(ha, addr3, value); + ha->isp_ops->wr_reg_indirect(ha, addr1, addr); + + rval = ql4_84xx_poll_wait_for_ready(ha, addr1, mask); + if (rval) + goto exit_ipmdio_wr_reg; + +exit_ipmdio_wr_reg: + return rval; +} + static void qla4_8xxx_minidump_process_rdcrb(struct scsi_qla_host *ha, struct qla8xxx_minidump_entry_hdr *entry_hdr, uint32_t **d_ptr) @@ -1822,7 +2024,7 @@ error_exit: return rval; } -static int qla4_83xx_minidump_pex_dma_read(struct scsi_qla_host *ha, +static int qla4_8xxx_minidump_pex_dma_read(struct scsi_qla_host *ha, struct qla8xxx_minidump_entry_hdr *entry_hdr, uint32_t **d_ptr) { @@ -1899,11 +2101,11 @@ static int qla4_83xx_minidump_pex_dma_read(struct scsi_qla_host *ha, dma_desc.cmd.read_data_size = size; /* Prepare: Write pex-dma descriptor to MS memory. */ - rval = qla4_83xx_ms_mem_write_128b(ha, + rval = qla4_8xxx_ms_mem_write_128b(ha, (uint64_t)m_hdr->desc_card_addr, (uint32_t *)&dma_desc, (sizeof(struct qla4_83xx_pex_dma_descriptor)/16)); - if (rval == -1) { + if (rval != QLA_SUCCESS) { ql4_printk(KERN_INFO, ha, "%s: Error writing rdmem-dma-init to MS !!!\n", __func__); @@ -2359,17 +2561,10 @@ static int qla4_8xxx_minidump_process_rdmem(struct scsi_qla_host *ha, uint32_t *data_ptr = *d_ptr; int rval = QLA_SUCCESS; - if (is_qla8032(ha) || is_qla8042(ha)) { - rval = qla4_83xx_minidump_pex_dma_read(ha, entry_hdr, - &data_ptr); - if (rval != QLA_SUCCESS) { - rval = __qla4_8xxx_minidump_process_rdmem(ha, entry_hdr, - &data_ptr); - } - } else { + rval = qla4_8xxx_minidump_pex_dma_read(ha, entry_hdr, &data_ptr); + if (rval != QLA_SUCCESS) rval = __qla4_8xxx_minidump_process_rdmem(ha, entry_hdr, &data_ptr); - } *d_ptr = data_ptr; return rval; } @@ -2440,6 +2635,227 @@ exit_process_pollrd: return rval; } +static uint32_t qla4_84xx_minidump_process_rddfe(struct scsi_qla_host *ha, + struct qla8xxx_minidump_entry_hdr *entry_hdr, + uint32_t **d_ptr) +{ + int loop_cnt; + uint32_t addr1, addr2, value, data, temp, wrval; + uint8_t stride, stride2; + uint16_t count; + uint32_t poll, mask, data_size, modify_mask; + uint32_t wait_count = 0; + uint32_t *data_ptr = *d_ptr; + struct qla8044_minidump_entry_rddfe *rddfe; + uint32_t rval = QLA_SUCCESS; + + rddfe = (struct qla8044_minidump_entry_rddfe *)entry_hdr; + addr1 = le32_to_cpu(rddfe->addr_1); + value = le32_to_cpu(rddfe->value); + stride = le32_to_cpu(rddfe->stride); + stride2 = le32_to_cpu(rddfe->stride2); + count = le32_to_cpu(rddfe->count); + + poll = le32_to_cpu(rddfe->poll); + mask = le32_to_cpu(rddfe->mask); + modify_mask = le32_to_cpu(rddfe->modify_mask); + data_size = le32_to_cpu(rddfe->data_size); + + addr2 = addr1 + stride; + + for (loop_cnt = 0x0; loop_cnt < count; loop_cnt++) { + ha->isp_ops->wr_reg_indirect(ha, addr1, (0x40000000 | value)); + + wait_count = 0; + while (wait_count < poll) { + ha->isp_ops->rd_reg_indirect(ha, addr1, &temp); + if ((temp & mask) != 0) + break; + wait_count++; + } + + if (wait_count == poll) { + ql4_printk(KERN_ERR, ha, "%s: TIMEOUT\n", __func__); + rval = QLA_ERROR; + goto exit_process_rddfe; + } else { + ha->isp_ops->rd_reg_indirect(ha, addr2, &temp); + temp = temp & modify_mask; + temp = (temp | ((loop_cnt << 16) | loop_cnt)); + wrval = ((temp << 16) | temp); + + ha->isp_ops->wr_reg_indirect(ha, addr2, wrval); + ha->isp_ops->wr_reg_indirect(ha, addr1, value); + + wait_count = 0; + while (wait_count < poll) { + ha->isp_ops->rd_reg_indirect(ha, addr1, &temp); + if ((temp & mask) != 0) + break; + wait_count++; + } + if (wait_count == poll) { + ql4_printk(KERN_ERR, ha, "%s: TIMEOUT\n", + __func__); + rval = QLA_ERROR; + goto exit_process_rddfe; + } + + ha->isp_ops->wr_reg_indirect(ha, addr1, + ((0x40000000 | value) + + stride2)); + wait_count = 0; + while (wait_count < poll) { + ha->isp_ops->rd_reg_indirect(ha, addr1, &temp); + if ((temp & mask) != 0) + break; + wait_count++; + } + + if (wait_count == poll) { + ql4_printk(KERN_ERR, ha, "%s: TIMEOUT\n", + __func__); + rval = QLA_ERROR; + goto exit_process_rddfe; + } + + ha->isp_ops->rd_reg_indirect(ha, addr2, &data); + + *data_ptr++ = cpu_to_le32(wrval); + *data_ptr++ = cpu_to_le32(data); + } + } + + *d_ptr = data_ptr; +exit_process_rddfe: + return rval; +} + +static uint32_t qla4_84xx_minidump_process_rdmdio(struct scsi_qla_host *ha, + struct qla8xxx_minidump_entry_hdr *entry_hdr, + uint32_t **d_ptr) +{ + int rval = QLA_SUCCESS; + uint32_t addr1, addr2, value1, value2, data, selval; + uint8_t stride1, stride2; + uint32_t addr3, addr4, addr5, addr6, addr7; + uint16_t count, loop_cnt; + uint32_t poll, mask; + uint32_t *data_ptr = *d_ptr; + struct qla8044_minidump_entry_rdmdio *rdmdio; + + rdmdio = (struct qla8044_minidump_entry_rdmdio *)entry_hdr; + addr1 = le32_to_cpu(rdmdio->addr_1); + addr2 = le32_to_cpu(rdmdio->addr_2); + value1 = le32_to_cpu(rdmdio->value_1); + stride1 = le32_to_cpu(rdmdio->stride_1); + stride2 = le32_to_cpu(rdmdio->stride_2); + count = le32_to_cpu(rdmdio->count); + + poll = le32_to_cpu(rdmdio->poll); + mask = le32_to_cpu(rdmdio->mask); + value2 = le32_to_cpu(rdmdio->value_2); + + addr3 = addr1 + stride1; + + for (loop_cnt = 0; loop_cnt < count; loop_cnt++) { + rval = ql4_84xx_poll_wait_ipmdio_bus_idle(ha, addr1, addr2, + addr3, mask); + if (rval) + goto exit_process_rdmdio; + + addr4 = addr2 - stride1; + rval = ql4_84xx_ipmdio_wr_reg(ha, addr1, addr3, mask, addr4, + value2); + if (rval) + goto exit_process_rdmdio; + + addr5 = addr2 - (2 * stride1); + rval = ql4_84xx_ipmdio_wr_reg(ha, addr1, addr3, mask, addr5, + value1); + if (rval) + goto exit_process_rdmdio; + + addr6 = addr2 - (3 * stride1); + rval = ql4_84xx_ipmdio_wr_reg(ha, addr1, addr3, mask, + addr6, 0x2); + if (rval) + goto exit_process_rdmdio; + + rval = ql4_84xx_poll_wait_ipmdio_bus_idle(ha, addr1, addr2, + addr3, mask); + if (rval) + goto exit_process_rdmdio; + + addr7 = addr2 - (4 * stride1); + rval = ql4_84xx_ipmdio_rd_reg(ha, addr1, addr3, + mask, addr7, &data); + if (rval) + goto exit_process_rdmdio; + + selval = (value2 << 18) | (value1 << 2) | 2; + + stride2 = le32_to_cpu(rdmdio->stride_2); + *data_ptr++ = cpu_to_le32(selval); + *data_ptr++ = cpu_to_le32(data); + + value1 = value1 + stride2; + *d_ptr = data_ptr; + } + +exit_process_rdmdio: + return rval; +} + +static uint32_t qla4_84xx_minidump_process_pollwr(struct scsi_qla_host *ha, + struct qla8xxx_minidump_entry_hdr *entry_hdr, + uint32_t **d_ptr) +{ + uint32_t addr1, addr2, value1, value2, poll, mask, r_value; + struct qla8044_minidump_entry_pollwr *pollwr_hdr; + uint32_t wait_count = 0; + uint32_t rval = QLA_SUCCESS; + + pollwr_hdr = (struct qla8044_minidump_entry_pollwr *)entry_hdr; + addr1 = le32_to_cpu(pollwr_hdr->addr_1); + addr2 = le32_to_cpu(pollwr_hdr->addr_2); + value1 = le32_to_cpu(pollwr_hdr->value_1); + value2 = le32_to_cpu(pollwr_hdr->value_2); + + poll = le32_to_cpu(pollwr_hdr->poll); + mask = le32_to_cpu(pollwr_hdr->mask); + + while (wait_count < poll) { + ha->isp_ops->rd_reg_indirect(ha, addr1, &r_value); + + if ((r_value & poll) != 0) + break; + + wait_count++; + } + + if (wait_count == poll) { + ql4_printk(KERN_ERR, ha, "%s: TIMEOUT\n", __func__); + rval = QLA_ERROR; + goto exit_process_pollwr; + } + + ha->isp_ops->wr_reg_indirect(ha, addr2, value2); + ha->isp_ops->wr_reg_indirect(ha, addr1, value1); + + wait_count = 0; + while (wait_count < poll) { + ha->isp_ops->rd_reg_indirect(ha, addr1, &r_value); + + if ((r_value & poll) != 0) + break; + wait_count++; + } + +exit_process_pollwr: + return rval; +} + static void qla83xx_minidump_process_rdmux2(struct scsi_qla_host *ha, struct qla8xxx_minidump_entry_hdr *entry_hdr, uint32_t **d_ptr) @@ -2753,6 +3169,24 @@ static int qla4_8xxx_collect_md_data(struct scsi_qla_host *ha) if (rval != QLA_SUCCESS) qla4_8xxx_mark_entry_skipped(ha, entry_hdr, i); break; + case QLA8044_RDDFE: + rval = qla4_84xx_minidump_process_rddfe(ha, entry_hdr, + &data_ptr); + if (rval != QLA_SUCCESS) + qla4_8xxx_mark_entry_skipped(ha, entry_hdr, i); + break; + case QLA8044_RDMDIO: + rval = qla4_84xx_minidump_process_rdmdio(ha, entry_hdr, + &data_ptr); + if (rval != QLA_SUCCESS) + qla4_8xxx_mark_entry_skipped(ha, entry_hdr, i); + break; + case QLA8044_POLLWR: + rval = qla4_84xx_minidump_process_pollwr(ha, entry_hdr, + &data_ptr); + if (rval != QLA_SUCCESS) + qla4_8xxx_mark_entry_skipped(ha, entry_hdr, i); + break; case QLA8XXX_RDNOP: default: qla4_8xxx_mark_entry_skipped(ha, entry_hdr, i); diff --git a/drivers/scsi/qla4xxx/ql4_nx.h b/drivers/scsi/qla4xxx/ql4_nx.h index 14500a0f62cc..337d9fcf6417 100644 --- a/drivers/scsi/qla4xxx/ql4_nx.h +++ b/drivers/scsi/qla4xxx/ql4_nx.h @@ -858,6 +858,9 @@ struct crb_addr_pair { #define QLA83XX_POLLRD 35 #define QLA83XX_RDMUX2 36 #define QLA83XX_POLLRDMWR 37 +#define QLA8044_RDDFE 38 +#define QLA8044_RDMDIO 39 +#define QLA8044_POLLWR 40 #define QLA8XXX_RDROM 71 #define QLA8XXX_RDMEM 72 #define QLA8XXX_CNTRL 98 diff --git a/drivers/scsi/qla4xxx/ql4_os.c b/drivers/scsi/qla4xxx/ql4_os.c index 459b9f7186fd..320206376206 100644 --- a/drivers/scsi/qla4xxx/ql4_os.c +++ b/drivers/scsi/qla4xxx/ql4_os.c @@ -83,12 +83,12 @@ MODULE_PARM_DESC(ql4xsess_recovery_tmo, " Target Session Recovery Timeout.\n" "\t\t Default: 120 sec."); -int ql4xmdcapmask = 0x1F; +int ql4xmdcapmask = 0; module_param(ql4xmdcapmask, int, S_IRUGO); MODULE_PARM_DESC(ql4xmdcapmask, " Set the Minidump driver capture mask level.\n" - "\t\t Default is 0x1F.\n" - "\t\t Can be set to 0x3, 0x7, 0xF, 0x1F, 0x3F, 0x7F"); + "\t\t Default is 0 (firmware default capture mask)\n" + "\t\t Can be set to 0x3, 0x7, 0xF, 0x1F, 0x3F, 0x7F, 0xFF"); int ql4xenablemd = 1; module_param(ql4xenablemd, int, S_IRUGO | S_IWUSR); @@ -1742,6 +1742,9 @@ static int qla4xxx_get_ep_param(struct iscsi_endpoint *ep, struct sockaddr *dst_addr; struct scsi_qla_host *ha; + if (!qla_ep) + return -ENOTCONN; + ha = to_qla_host(qla_ep->host); DEBUG2(ql4_printk(KERN_INFO, ha, "%s: host: %ld\n", __func__, ha->host_no)); @@ -1749,9 +1752,6 @@ static int qla4xxx_get_ep_param(struct iscsi_endpoint *ep, switch (param) { case ISCSI_PARAM_CONN_PORT: case ISCSI_PARAM_CONN_ADDRESS: - if (!qla_ep) - return -ENOTCONN; - dst_addr = (struct sockaddr *)&qla_ep->dst_addr; if (!dst_addr) return -ENOTCONN; @@ -2879,7 +2879,6 @@ static int qla4xxx_conn_get_param(struct iscsi_cls_conn *cls_conn, struct iscsi_conn *conn; struct qla_conn *qla_conn; struct sockaddr *dst_addr; - int len = 0; conn = cls_conn->dd_data; qla_conn = conn->dd_data; @@ -2893,9 +2892,6 @@ static int qla4xxx_conn_get_param(struct iscsi_cls_conn *cls_conn, default: return iscsi_conn_get_param(cls_conn, param, buf); } - - return len; - } int qla4xxx_get_ddb_index(struct scsi_qla_host *ha, uint16_t *ddb_index) @@ -3569,14 +3565,13 @@ static int qla4xxx_copy_from_fwddb_param(struct iscsi_bus_flash_session *sess, if (test_bit(OPT_IPV6_DEVICE, &options)) { conn->ipv6_traffic_class = fw_ddb_entry->ipv4_tos; - conn->link_local_ipv6_addr = kzalloc(IPv6_ADDR_LEN, GFP_KERNEL); + conn->link_local_ipv6_addr = kmemdup( + fw_ddb_entry->link_local_ipv6_addr, + IPv6_ADDR_LEN, GFP_KERNEL); if (!conn->link_local_ipv6_addr) { rc = -ENOMEM; goto exit_copy; } - - memcpy(conn->link_local_ipv6_addr, - fw_ddb_entry->link_local_ipv6_addr, IPv6_ADDR_LEN); } else { conn->ipv4_tos = fw_ddb_entry->ipv4_tos; } @@ -4565,6 +4560,7 @@ static void qla4xxx_timer(struct scsi_qla_host *ha) test_bit(DPC_LINK_CHANGED, &ha->dpc_flags) || test_bit(DPC_HA_UNRECOVERABLE, &ha->dpc_flags) || test_bit(DPC_HA_NEED_QUIESCENT, &ha->dpc_flags) || + test_bit(DPC_SYSFS_DDB_EXPORT, &ha->dpc_flags) || test_bit(DPC_AEN, &ha->dpc_flags)) { DEBUG2(printk("scsi%ld: %s: scheduling dpc routine" " - dpc flags = 0x%lx\n", @@ -4862,9 +4858,6 @@ static int qla4xxx_recover_adapter(struct scsi_qla_host *ha) ha->host_no, __func__)); status = ha->isp_ops->reset_firmware(ha); if (status == QLA_SUCCESS) { - if (!test_bit(AF_FW_RECOVERY, &ha->flags)) - qla4xxx_cmd_wait(ha); - ha->isp_ops->disable_intrs(ha); qla4xxx_process_aen(ha, FLUSH_DDB_CHANGED_AENS); qla4xxx_abort_active_cmds(ha, DID_RESET << 16); @@ -5432,6 +5425,11 @@ dpc_post_reset_ha: qla4xxx_relogin_all_devices(ha); } } + if (test_and_clear_bit(DPC_SYSFS_DDB_EXPORT, &ha->dpc_flags)) { + if (qla4xxx_sysfs_ddb_export(ha)) + ql4_printk(KERN_ERR, ha, "%s: Error exporting ddb to sysfs\n", + __func__); + } } /** @@ -8409,7 +8407,7 @@ exit_ddb_del: * * Export the firmware DDB for all send targets and normal targets to sysfs. **/ -static int qla4xxx_sysfs_ddb_export(struct scsi_qla_host *ha) +int qla4xxx_sysfs_ddb_export(struct scsi_qla_host *ha) { struct dev_db_entry *fw_ddb_entry = NULL; dma_addr_t fw_ddb_entry_dma; @@ -8847,11 +8845,8 @@ skip_retry_init: ql4_printk(KERN_ERR, ha, "%s: No iSCSI boot target configured\n", __func__); - if (qla4xxx_sysfs_ddb_export(ha)) - ql4_printk(KERN_ERR, ha, - "%s: Error exporting ddb to sysfs\n", __func__); - - /* Perform the build ddb list and login to each */ + set_bit(DPC_SYSFS_DDB_EXPORT, &ha->dpc_flags); + /* Perform the build ddb list and login to each */ qla4xxx_build_ddb_list(ha, INIT_ADAPTER); iscsi_host_for_each_session(ha->host, qla4xxx_login_flash_ddb); qla4xxx_wait_login_resp_boot_tgt(ha); diff --git a/drivers/scsi/qla4xxx/ql4_version.h b/drivers/scsi/qla4xxx/ql4_version.h index c6ba0a6b8458..f11eaa773339 100644 --- a/drivers/scsi/qla4xxx/ql4_version.h +++ b/drivers/scsi/qla4xxx/ql4_version.h @@ -5,4 +5,4 @@ * See LICENSE.qla4xxx for copyright and licensing details. */ -#define QLA4XXX_DRIVER_VERSION "5.04.00-k4" +#define QLA4XXX_DRIVER_VERSION "5.04.00-k6" diff --git a/drivers/scsi/scsi_debug.c b/drivers/scsi/scsi_debug.c index f3e9cc038d1d..1328a2621070 100644 --- a/drivers/scsi/scsi_debug.c +++ b/drivers/scsi/scsi_debug.c @@ -130,6 +130,7 @@ static const char * scsi_debug_version_date = "20100324"; #define SCSI_DEBUG_OPT_DIF_ERR 32 #define SCSI_DEBUG_OPT_DIX_ERR 64 #define SCSI_DEBUG_OPT_MAC_TIMEOUT 128 +#define SCSI_DEBUG_OPT_SHORT_TRANSFER 256 /* When "every_nth" > 0 then modulo "every_nth" commands: * - a no response is simulated if SCSI_DEBUG_OPT_TIMEOUT is set * - a RECOVERED_ERROR is simulated on successful read and write @@ -3583,6 +3584,7 @@ int scsi_debug_queuecommand_lck(struct scsi_cmnd *SCpnt, done_funct_t done) int inj_transport = 0; int inj_dif = 0; int inj_dix = 0; + int inj_short = 0; int delay_override = 0; int unmap = 0; @@ -3628,6 +3630,8 @@ int scsi_debug_queuecommand_lck(struct scsi_cmnd *SCpnt, done_funct_t done) inj_dif = 1; /* to reads and writes below */ else if (SCSI_DEBUG_OPT_DIX_ERR & scsi_debug_opts) inj_dix = 1; /* to reads and writes below */ + else if (SCSI_DEBUG_OPT_SHORT_TRANSFER & scsi_debug_opts) + inj_short = 1; } if (devip->wlun) { @@ -3744,6 +3748,10 @@ read: if (scsi_debug_fake_rw) break; get_data_transfer_info(cmd, &lba, &num, &ei_lba); + + if (inj_short) + num /= 2; + errsts = resp_read(SCpnt, lba, num, devip, ei_lba); if (inj_recovered && (0 == errsts)) { mk_sense_buffer(devip, RECOVERED_ERROR, diff --git a/drivers/scsi/scsi_error.c b/drivers/scsi/scsi_error.c index f17aa7aa7879..47a1ffc4c904 100644 --- a/drivers/scsi/scsi_error.c +++ b/drivers/scsi/scsi_error.c @@ -1029,6 +1029,7 @@ retry: rtn = NEEDS_RETRY; } else { timeleft = wait_for_completion_timeout(&done, timeout); + rtn = SUCCESS; } shost->eh_action = NULL; @@ -2306,6 +2307,12 @@ scsi_reset_provider(struct scsi_device *dev, int flag) } scmd = scsi_get_command(dev, GFP_KERNEL); + if (!scmd) { + rtn = FAILED; + put_device(&dev->sdev_gendev); + goto out_put_autopm_host; + } + blk_rq_init(NULL, &req); scmd->request = &req; diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c index a0c95cac91f0..be0d5fad999d 100644 --- a/drivers/scsi/scsi_lib.c +++ b/drivers/scsi/scsi_lib.c @@ -512,68 +512,6 @@ void scsi_run_host_queues(struct Scsi_Host *shost) scsi_run_queue(sdev->request_queue); } -static void __scsi_release_buffers(struct scsi_cmnd *, int); - -/* - * Function: scsi_end_request() - * - * Purpose: Post-processing of completed commands (usually invoked at end - * of upper level post-processing and scsi_io_completion). - * - * Arguments: cmd - command that is complete. - * error - 0 if I/O indicates success, < 0 for I/O error. - * bytes - number of bytes of completed I/O - * requeue - indicates whether we should requeue leftovers. - * - * Lock status: Assumed that lock is not held upon entry. - * - * Returns: cmd if requeue required, NULL otherwise. - * - * Notes: This is called for block device requests in order to - * mark some number of sectors as complete. - * - * We are guaranteeing that the request queue will be goosed - * at some point during this call. - * Notes: If cmd was requeued, upon return it will be a stale pointer. - */ -static struct scsi_cmnd *scsi_end_request(struct scsi_cmnd *cmd, int error, - int bytes, int requeue) -{ - struct request_queue *q = cmd->device->request_queue; - struct request *req = cmd->request; - - /* - * If there are blocks left over at the end, set up the command - * to queue the remainder of them. - */ - if (blk_end_request(req, error, bytes)) { - /* kill remainder if no retrys */ - if (error && scsi_noretry_cmd(cmd)) - blk_end_request_all(req, error); - else { - if (requeue) { - /* - * Bleah. Leftovers again. Stick the - * leftovers in the front of the - * queue, and goose the queue again. - */ - scsi_release_buffers(cmd); - scsi_requeue_command(q, cmd); - cmd = NULL; - } - return cmd; - } - } - - /* - * This will goose the queue request function at the end, so we don't - * need to worry about launching another command. - */ - __scsi_release_buffers(cmd, 0); - scsi_next_command(cmd); - return NULL; -} - static inline unsigned int scsi_sgtable_index(unsigned short nents) { unsigned int index; @@ -625,30 +563,10 @@ static void scsi_free_sgtable(struct scsi_data_buffer *sdb) __sg_free_table(&sdb->table, SCSI_MAX_SG_SEGMENTS, scsi_sg_free); } -static void __scsi_release_buffers(struct scsi_cmnd *cmd, int do_bidi_check) -{ - - if (cmd->sdb.table.nents) - scsi_free_sgtable(&cmd->sdb); - - memset(&cmd->sdb, 0, sizeof(cmd->sdb)); - - if (do_bidi_check && scsi_bidi_cmnd(cmd)) { - struct scsi_data_buffer *bidi_sdb = - cmd->request->next_rq->special; - scsi_free_sgtable(bidi_sdb); - kmem_cache_free(scsi_sdb_cache, bidi_sdb); - cmd->request->next_rq->special = NULL; - } - - if (scsi_prot_sg_count(cmd)) - scsi_free_sgtable(cmd->prot_sdb); -} - /* * Function: scsi_release_buffers() * - * Purpose: Completion processing for block device I/O requests. + * Purpose: Free resources allocate for a scsi_command. * * Arguments: cmd - command that we are bailing. * @@ -659,15 +577,29 @@ static void __scsi_release_buffers(struct scsi_cmnd *cmd, int do_bidi_check) * Notes: In the event that an upper level driver rejects a * command, we must release resources allocated during * the __init_io() function. Primarily this would involve - * the scatter-gather table, and potentially any bounce - * buffers. + * the scatter-gather table. */ void scsi_release_buffers(struct scsi_cmnd *cmd) { - __scsi_release_buffers(cmd, 1); + if (cmd->sdb.table.nents) + scsi_free_sgtable(&cmd->sdb); + + memset(&cmd->sdb, 0, sizeof(cmd->sdb)); + + if (scsi_prot_sg_count(cmd)) + scsi_free_sgtable(cmd->prot_sdb); } EXPORT_SYMBOL(scsi_release_buffers); +static void scsi_release_bidi_buffers(struct scsi_cmnd *cmd) +{ + struct scsi_data_buffer *bidi_sdb = cmd->request->next_rq->special; + + scsi_free_sgtable(bidi_sdb); + kmem_cache_free(scsi_sdb_cache, bidi_sdb); + cmd->request->next_rq->special = NULL; +} + /** * __scsi_error_from_host_byte - translate SCSI error code into errno * @cmd: SCSI command (unused) @@ -725,16 +657,9 @@ static int __scsi_error_from_host_byte(struct scsi_cmnd *cmd, int result) * * Returns: Nothing * - * Notes: This function is matched in terms of capabilities to - * the function that created the scatter-gather list. - * In other words, if there are no bounce buffers - * (the normal case for most drivers), we don't need - * the logic to deal with cleaning up afterwards. - * - * We must call scsi_end_request(). This will finish off - * the specified number of sectors. If we are done, the - * command block will be released and the queue function - * will be goosed. If we are not done then we have to + * Notes: We will finish off the specified number of sectors. If we + * are done, the command block will be released and the queue + * function will be goosed. If we are not done then we have to * figure out what to do next: * * a) We can call scsi_requeue_command(). The request @@ -743,7 +668,7 @@ static int __scsi_error_from_host_byte(struct scsi_cmnd *cmd, int result) * be used if we made forward progress, or if we want * to switch from READ(10) to READ(6) for example. * - * b) We can call scsi_queue_insert(). The request will + * b) We can call __scsi_queue_insert(). The request will * be put back on the queue and retried using the same * command as before, possibly after a delay. * @@ -801,6 +726,8 @@ void scsi_io_completion(struct scsi_cmnd *cmd, unsigned int good_bytes) req->next_rq->resid_len = scsi_in(cmd)->resid; scsi_release_buffers(cmd); + scsi_release_bidi_buffers(cmd); + blk_end_request_all(req, 0); scsi_next_command(cmd); @@ -840,12 +767,25 @@ void scsi_io_completion(struct scsi_cmnd *cmd, unsigned int good_bytes) } /* - * A number of bytes were successfully read. If there - * are leftovers and there is some kind of error - * (result != 0), retry the rest. + * If we finished all bytes in the request we are done now. */ - if (scsi_end_request(cmd, error, good_bytes, result == 0) == NULL) - return; + if (!blk_end_request(req, error, good_bytes)) + goto next_command; + + /* + * Kill remainder if no retrys. + */ + if (error && scsi_noretry_cmd(cmd)) { + blk_end_request_all(req, error); + goto next_command; + } + + /* + * If there had been no error, but we have leftover bytes in the + * requeues just queue the command up again. + */ + if (result == 0) + goto requeue; error = __scsi_error_from_host_byte(cmd, result); @@ -973,7 +913,6 @@ void scsi_io_completion(struct scsi_cmnd *cmd, unsigned int good_bytes) switch (action) { case ACTION_FAIL: /* Give up and fail the remainder of the request */ - scsi_release_buffers(cmd); if (!(req->cmd_flags & REQ_QUIET)) { if (description) scmd_printk(KERN_INFO, cmd, "%s\n", @@ -983,12 +922,11 @@ void scsi_io_completion(struct scsi_cmnd *cmd, unsigned int good_bytes) scsi_print_sense("", cmd); scsi_print_command(cmd); } - if (blk_end_request_err(req, error)) - scsi_requeue_command(q, cmd); - else - scsi_next_command(cmd); - break; + if (!blk_end_request_err(req, error)) + goto next_command; + /*FALLTHRU*/ case ACTION_REPREP: + requeue: /* Unprep the request and put it back at the head of the queue. * A new command will be prepared and issued. */ @@ -1004,6 +942,11 @@ void scsi_io_completion(struct scsi_cmnd *cmd, unsigned int good_bytes) __scsi_queue_insert(cmd, SCSI_MLQUEUE_DEVICE_BUSY, 0); break; } + return; + +next_command: + scsi_release_buffers(cmd); + scsi_next_command(cmd); } static int scsi_init_sgtable(struct request *req, struct scsi_data_buffer *sdb, @@ -1128,15 +1071,7 @@ static struct scsi_cmnd *scsi_get_cmd_from_req(struct scsi_device *sdev, int scsi_setup_blk_pc_cmnd(struct scsi_device *sdev, struct request *req) { - struct scsi_cmnd *cmd; - int ret = scsi_prep_state_check(sdev, req); - - if (ret != BLKPREP_OK) - return ret; - - cmd = scsi_get_cmd_from_req(sdev, req); - if (unlikely(!cmd)) - return BLKPREP_DEFER; + struct scsi_cmnd *cmd = req->special; /* * BLOCK_PC requests may transfer data, in which case they must @@ -1179,15 +1114,11 @@ EXPORT_SYMBOL(scsi_setup_blk_pc_cmnd); */ int scsi_setup_fs_cmnd(struct scsi_device *sdev, struct request *req) { - struct scsi_cmnd *cmd; - int ret = scsi_prep_state_check(sdev, req); - - if (ret != BLKPREP_OK) - return ret; + struct scsi_cmnd *cmd = req->special; if (unlikely(sdev->scsi_dh_data && sdev->scsi_dh_data->scsi_dh && sdev->scsi_dh_data->scsi_dh->prep_fn)) { - ret = sdev->scsi_dh_data->scsi_dh->prep_fn(sdev, req); + int ret = sdev->scsi_dh_data->scsi_dh->prep_fn(sdev, req); if (ret != BLKPREP_OK) return ret; } @@ -1197,16 +1128,13 @@ int scsi_setup_fs_cmnd(struct scsi_device *sdev, struct request *req) */ BUG_ON(!req->nr_phys_segments); - cmd = scsi_get_cmd_from_req(sdev, req); - if (unlikely(!cmd)) - return BLKPREP_DEFER; - memset(cmd->cmnd, 0, BLK_MAX_CDB); return scsi_init_io(cmd, GFP_ATOMIC); } EXPORT_SYMBOL(scsi_setup_fs_cmnd); -int scsi_prep_state_check(struct scsi_device *sdev, struct request *req) +static int +scsi_prep_state_check(struct scsi_device *sdev, struct request *req) { int ret = BLKPREP_OK; @@ -1258,9 +1186,9 @@ int scsi_prep_state_check(struct scsi_device *sdev, struct request *req) } return ret; } -EXPORT_SYMBOL(scsi_prep_state_check); -int scsi_prep_return(struct request_queue *q, struct request *req, int ret) +static int +scsi_prep_return(struct request_queue *q, struct request *req, int ret) { struct scsi_device *sdev = q->queuedata; @@ -1291,18 +1219,44 @@ int scsi_prep_return(struct request_queue *q, struct request *req, int ret) return ret; } -EXPORT_SYMBOL(scsi_prep_return); -int scsi_prep_fn(struct request_queue *q, struct request *req) +static int scsi_prep_fn(struct request_queue *q, struct request *req) { struct scsi_device *sdev = q->queuedata; - int ret = BLKPREP_KILL; + struct scsi_cmnd *cmd; + int ret; - if (req->cmd_type == REQ_TYPE_BLOCK_PC) + ret = scsi_prep_state_check(sdev, req); + if (ret != BLKPREP_OK) + goto out; + + cmd = scsi_get_cmd_from_req(sdev, req); + if (unlikely(!cmd)) { + ret = BLKPREP_DEFER; + goto out; + } + + if (req->cmd_type == REQ_TYPE_FS) + ret = scsi_cmd_to_driver(cmd)->init_command(cmd); + else if (req->cmd_type == REQ_TYPE_BLOCK_PC) ret = scsi_setup_blk_pc_cmnd(sdev, req); + else + ret = BLKPREP_KILL; + +out: return scsi_prep_return(q, req, ret); } -EXPORT_SYMBOL(scsi_prep_fn); + +static void scsi_unprep_fn(struct request_queue *q, struct request *req) +{ + if (req->cmd_type == REQ_TYPE_FS) { + struct scsi_cmnd *cmd = req->special; + struct scsi_driver *drv = scsi_cmd_to_driver(cmd); + + if (drv->uninit_command) + drv->uninit_command(cmd); + } +} /* * scsi_dev_queue_ready: if we can send requests to sdev, return 1 else @@ -1723,6 +1677,7 @@ struct request_queue *scsi_alloc_queue(struct scsi_device *sdev) return NULL; blk_queue_prep_rq(q, scsi_prep_fn); + blk_queue_unprep_rq(q, scsi_unprep_fn); blk_queue_softirq_done(q, scsi_softirq_done); blk_queue_rq_timed_out(q, scsi_times_out); blk_queue_lld_busy(q, scsi_lld_busy); diff --git a/drivers/scsi/sd.c b/drivers/scsi/sd.c index 96af195224f2..e9689d57ccb6 100644 --- a/drivers/scsi/sd.c +++ b/drivers/scsi/sd.c @@ -109,6 +109,8 @@ static int sd_suspend_system(struct device *); static int sd_suspend_runtime(struct device *); static int sd_resume(struct device *); static void sd_rescan(struct device *); +static int sd_init_command(struct scsi_cmnd *SCpnt); +static void sd_uninit_command(struct scsi_cmnd *SCpnt); static int sd_done(struct scsi_cmnd *); static int sd_eh_action(struct scsi_cmnd *, int); static void sd_read_capacity(struct scsi_disk *sdkp, unsigned char *buffer); @@ -503,6 +505,8 @@ static struct scsi_driver sd_template = { .pm = &sd_pm_ops, }, .rescan = sd_rescan, + .init_command = sd_init_command, + .uninit_command = sd_uninit_command, .done = sd_done, .eh_action = sd_eh_action, }; @@ -836,9 +840,9 @@ static int scsi_setup_flush_cmnd(struct scsi_device *sdp, struct request *rq) return scsi_setup_blk_pc_cmnd(sdp, rq); } -static void sd_unprep_fn(struct request_queue *q, struct request *rq) +static void sd_uninit_command(struct scsi_cmnd *SCpnt) { - struct scsi_cmnd *SCpnt = rq->special; + struct request *rq = SCpnt->request; if (rq->cmd_flags & REQ_DISCARD) __free_page(rq->completion_data); @@ -850,18 +854,10 @@ static void sd_unprep_fn(struct request_queue *q, struct request *rq) } } -/** - * sd_prep_fn - build a scsi (read or write) command from - * information in the request structure. - * @SCpnt: pointer to mid-level's per scsi command structure that - * contains request and into which the scsi command is written - * - * Returns 1 if successful and 0 if error (or cannot be done now). - **/ -static int sd_prep_fn(struct request_queue *q, struct request *rq) +static int sd_init_command(struct scsi_cmnd *SCpnt) { - struct scsi_cmnd *SCpnt; - struct scsi_device *sdp = q->queuedata; + struct request *rq = SCpnt->request; + struct scsi_device *sdp = SCpnt->device; struct gendisk *disk = rq->rq_disk; struct scsi_disk *sdkp; sector_t block = blk_rq_pos(rq); @@ -883,12 +879,6 @@ static int sd_prep_fn(struct request_queue *q, struct request *rq) } else if (rq->cmd_flags & REQ_FLUSH) { ret = scsi_setup_flush_cmnd(sdp, rq); goto out; - } else if (rq->cmd_type == REQ_TYPE_BLOCK_PC) { - ret = scsi_setup_blk_pc_cmnd(sdp, rq); - goto out; - } else if (rq->cmd_type != REQ_TYPE_FS) { - ret = BLKPREP_KILL; - goto out; } ret = scsi_setup_fs_cmnd(sdp, rq); if (ret != BLKPREP_OK) @@ -900,11 +890,10 @@ static int sd_prep_fn(struct request_queue *q, struct request *rq) * is used for a killable error condition */ ret = BLKPREP_KILL; - SCSI_LOG_HLQUEUE(1, scmd_printk(KERN_INFO, SCpnt, - "sd_prep_fn: block=%llu, " - "count=%d\n", - (unsigned long long)block, - this_count)); + SCSI_LOG_HLQUEUE(1, + scmd_printk(KERN_INFO, SCpnt, + "%s: block=%llu, count=%d\n", + __func__, (unsigned long long)block, this_count)); if (!sdp || !scsi_device_online(sdp) || block + blk_rq_sectors(rq) > get_capacity(disk)) { @@ -1124,7 +1113,7 @@ static int sd_prep_fn(struct request_queue *q, struct request *rq) */ ret = BLKPREP_OK; out: - return scsi_prep_return(q, rq, ret); + return ret; } /** @@ -1686,12 +1675,12 @@ static int sd_done(struct scsi_cmnd *SCpnt) sshdr.ascq)); } #endif + sdkp->medium_access_timed_out = 0; + if (driver_byte(result) != DRIVER_SENSE && (!sense_valid || sense_deferred)) goto out; - sdkp->medium_access_timed_out = 0; - switch (sshdr.sense_key) { case HARDWARE_ERROR: case MEDIUM_ERROR: @@ -2875,9 +2864,6 @@ static void sd_probe_async(void *data, async_cookie_t cookie) sd_revalidate_disk(gd); - blk_queue_prep_rq(sdp->request_queue, sd_prep_fn); - blk_queue_unprep_rq(sdp->request_queue, sd_unprep_fn); - gd->driverfs_dev = &sdp->sdev_gendev; gd->flags = GENHD_FL_EXT_DEVT; if (sdp->removable) { @@ -3025,8 +3011,6 @@ static int sd_remove(struct device *dev) async_synchronize_full_domain(&scsi_sd_pm_domain); async_synchronize_full_domain(&scsi_sd_probe_domain); - blk_queue_prep_rq(sdkp->device->request_queue, scsi_prep_fn); - blk_queue_unprep_rq(sdkp->device->request_queue, NULL); device_del(&sdkp->dev); del_gendisk(sdkp->disk); sd_shutdown(dev); diff --git a/drivers/scsi/sr.c b/drivers/scsi/sr.c index 40d85929aefe..93cbd36c990b 100644 --- a/drivers/scsi/sr.c +++ b/drivers/scsi/sr.c @@ -79,6 +79,7 @@ MODULE_ALIAS_SCSI_DEVICE(TYPE_WORM); static DEFINE_MUTEX(sr_mutex); static int sr_probe(struct device *); static int sr_remove(struct device *); +static int sr_init_command(struct scsi_cmnd *SCpnt); static int sr_done(struct scsi_cmnd *); static int sr_runtime_suspend(struct device *dev); @@ -94,6 +95,7 @@ static struct scsi_driver sr_template = { .remove = sr_remove, .pm = &sr_pm_ops, }, + .init_command = sr_init_command, .done = sr_done, }; @@ -378,21 +380,14 @@ static int sr_done(struct scsi_cmnd *SCpnt) return good_bytes; } -static int sr_prep_fn(struct request_queue *q, struct request *rq) +static int sr_init_command(struct scsi_cmnd *SCpnt) { int block = 0, this_count, s_size; struct scsi_cd *cd; - struct scsi_cmnd *SCpnt; - struct scsi_device *sdp = q->queuedata; + struct request *rq = SCpnt->request; + struct scsi_device *sdp = SCpnt->device; int ret; - if (rq->cmd_type == REQ_TYPE_BLOCK_PC) { - ret = scsi_setup_blk_pc_cmnd(sdp, rq); - goto out; - } else if (rq->cmd_type != REQ_TYPE_FS) { - ret = BLKPREP_KILL; - goto out; - } ret = scsi_setup_fs_cmnd(sdp, rq); if (ret != BLKPREP_OK) goto out; @@ -517,7 +512,7 @@ static int sr_prep_fn(struct request_queue *q, struct request *rq) */ ret = BLKPREP_OK; out: - return scsi_prep_return(q, rq, ret); + return ret; } static int sr_block_open(struct block_device *bdev, fmode_t mode) @@ -718,7 +713,6 @@ static int sr_probe(struct device *dev) /* FIXME: need to handle a get_capabilities failure properly ?? */ get_capabilities(cd); - blk_queue_prep_rq(sdev->request_queue, sr_prep_fn); sr_vendor_init(cd); disk->driverfs_dev = &sdev->sdev_gendev; @@ -993,7 +987,6 @@ static int sr_remove(struct device *dev) scsi_autopm_get_device(cd->device); - blk_queue_prep_rq(cd->device->request_queue, scsi_prep_fn); del_gendisk(cd->disk); mutex_lock(&sr_ref_mutex); diff --git a/drivers/scsi/sun3_NCR5380.c b/drivers/scsi/sun3_NCR5380.c index 636bbe0ea84c..88220794cc98 100644 --- a/drivers/scsi/sun3_NCR5380.c +++ b/drivers/scsi/sun3_NCR5380.c @@ -364,7 +364,7 @@ static int is_lun_busy(struct scsi_cmnd *cmd, int should_be_tagged) return( 0 ); if (TagAlloc[cmd->device->id][cmd->device->lun].nr_allocated >= TagAlloc[cmd->device->id][cmd->device->lun].queue_size ) { - TAG_PRINTK( "scsi%d: target %d lun %d: no free tags\n", + dprintk(NDEBUG_TAGS, "scsi%d: target %d lun %d: no free tags\n", H_NO(cmd), cmd->device->id, cmd->device->lun ); return( 1 ); } @@ -388,7 +388,7 @@ static void cmd_get_tag(struct scsi_cmnd *cmd, int should_be_tagged) !setup_use_tagged_queuing || !cmd->device->tagged_supported) { cmd->tag = TAG_NONE; hostdata->busy[cmd->device->id] |= (1 << cmd->device->lun); - TAG_PRINTK( "scsi%d: target %d lun %d now allocated by untagged " + dprintk(NDEBUG_TAGS, "scsi%d: target %d lun %d now allocated by untagged " "command\n", H_NO(cmd), cmd->device->id, cmd->device->lun ); } else { @@ -397,7 +397,7 @@ static void cmd_get_tag(struct scsi_cmnd *cmd, int should_be_tagged) cmd->tag = find_first_zero_bit( &ta->allocated, MAX_TAGS ); set_bit( cmd->tag, &ta->allocated ); ta->nr_allocated++; - TAG_PRINTK( "scsi%d: using tag %d for target %d lun %d " + dprintk(NDEBUG_TAGS, "scsi%d: using tag %d for target %d lun %d " "(now %d tags in use)\n", H_NO(cmd), cmd->tag, cmd->device->id, cmd->device->lun, ta->nr_allocated ); @@ -415,7 +415,7 @@ static void cmd_free_tag(struct scsi_cmnd *cmd) if (cmd->tag == TAG_NONE) { hostdata->busy[cmd->device->id] &= ~(1 << cmd->device->lun); - TAG_PRINTK( "scsi%d: target %d lun %d untagged cmd finished\n", + dprintk(NDEBUG_TAGS, "scsi%d: target %d lun %d untagged cmd finished\n", H_NO(cmd), cmd->device->id, cmd->device->lun ); } else if (cmd->tag >= MAX_TAGS) { @@ -426,7 +426,7 @@ static void cmd_free_tag(struct scsi_cmnd *cmd) TAG_ALLOC *ta = &TagAlloc[cmd->device->id][cmd->device->lun]; clear_bit( cmd->tag, &ta->allocated ); ta->nr_allocated--; - TAG_PRINTK( "scsi%d: freed tag %d for target %d lun %d\n", + dprintk(NDEBUG_TAGS, "scsi%d: freed tag %d for target %d lun %d\n", H_NO(cmd), cmd->tag, cmd->device->id, cmd->device->lun ); } } @@ -484,7 +484,7 @@ static __inline__ void initialize_SCp(struct scsi_cmnd *cmd) #include <linux/delay.h> -#if 1 +#if NDEBUG static struct { unsigned char mask; const char * name;} @@ -572,12 +572,6 @@ static void NCR5380_print_phase(struct Scsi_Host *instance) } } -#else /* !NDEBUG */ - -/* dummies... */ -__inline__ void NCR5380_print(struct Scsi_Host *instance) { }; -__inline__ void NCR5380_print_phase(struct Scsi_Host *instance) { }; - #endif /* @@ -618,7 +612,7 @@ static inline void NCR5380_all_init (void) { static int done = 0; if (!done) { - INI_PRINTK("scsi : NCR5380_all_init()\n"); + dprintk(NDEBUG_INIT, "scsi : NCR5380_all_init()\n"); done = 1; } } @@ -681,8 +675,8 @@ static void NCR5380_print_status(struct Scsi_Host *instance) Scsi_Cmnd *ptr; unsigned long flags; - NCR_PRINT(NDEBUG_ANY); - NCR_PRINT_PHASE(NDEBUG_ANY); + NCR5380_dprint(NDEBUG_ANY, instance); + NCR5380_dprint_phase(NDEBUG_ANY, instance); hostdata = (struct NCR5380_hostdata *)instance->hostdata; @@ -928,7 +922,7 @@ static int NCR5380_queue_command_lck(struct scsi_cmnd *cmd, local_irq_restore(flags); - QU_PRINTK("scsi%d: command added to %s of queue\n", H_NO(cmd), + dprintk(NDEBUG_QUEUES, "scsi%d: command added to %s of queue\n", H_NO(cmd), (cmd->cmnd[0] == REQUEST_SENSE) ? "head" : "tail"); /* If queue_command() is called from an interrupt (real one or bottom @@ -998,7 +992,7 @@ static void NCR5380_main (struct work_struct *bl) done = 1; if (!hostdata->connected) { - MAIN_PRINTK( "scsi%d: not connected\n", HOSTNO ); + dprintk(NDEBUG_MAIN, "scsi%d: not connected\n", HOSTNO ); /* * Search through the issue_queue for a command destined * for a target that's not busy. @@ -1012,12 +1006,8 @@ static void NCR5380_main (struct work_struct *bl) for (tmp = (struct scsi_cmnd *) hostdata->issue_queue, prev = NULL; tmp; prev = tmp, tmp = NEXT(tmp) ) { -#if (NDEBUG & NDEBUG_LISTS) if (prev != tmp) - printk("MAIN tmp=%p target=%d busy=%d lun=%d\n", - tmp, tmp->target, hostdata->busy[tmp->target], - tmp->lun); -#endif + dprintk(NDEBUG_LISTS, "MAIN tmp=%p target=%d busy=%d lun=%d\n", tmp, tmp->device->id, hostdata->busy[tmp->device->id], tmp->device->lun); /* When we find one, remove it from the issue queue. */ /* ++guenther: possible race with Falcon locking */ if ( @@ -1047,9 +1037,9 @@ static void NCR5380_main (struct work_struct *bl) * On failure, we must add the command back to the * issue queue so we can keep trying. */ - MAIN_PRINTK("scsi%d: main(): command for target %d " + dprintk(NDEBUG_MAIN, "scsi%d: main(): command for target %d " "lun %d removed from issue_queue\n", - HOSTNO, tmp->target, tmp->lun); + HOSTNO, tmp->device->id, tmp->device->lun); /* * REQUEST SENSE commands are issued without tagged * queueing, even on SCSI-II devices because the @@ -1076,7 +1066,7 @@ static void NCR5380_main (struct work_struct *bl) cmd_free_tag( tmp ); #endif local_irq_restore(flags); - MAIN_PRINTK("scsi%d: main(): select() failed, " + dprintk(NDEBUG_MAIN, "scsi%d: main(): select() failed, " "returned to issue_queue\n", HOSTNO); if (hostdata->connected) break; @@ -1090,10 +1080,10 @@ static void NCR5380_main (struct work_struct *bl) #endif ) { local_irq_restore(flags); - MAIN_PRINTK("scsi%d: main: performing information transfer\n", + dprintk(NDEBUG_MAIN, "scsi%d: main: performing information transfer\n", HOSTNO); NCR5380_information_transfer(instance); - MAIN_PRINTK("scsi%d: main: done set false\n", HOSTNO); + dprintk(NDEBUG_MAIN, "scsi%d: main: done set false\n", HOSTNO); done = 0; } } while (!done); @@ -1130,7 +1120,7 @@ static void NCR5380_dma_complete( struct Scsi_Host *instance ) return; } - DMA_PRINTK("scsi%d: real DMA transfer complete, basr 0x%X, sr 0x%X\n", + dprintk(NDEBUG_DMA, "scsi%d: real DMA transfer complete, basr 0x%X, sr 0x%X\n", HOSTNO, NCR5380_read(BUS_AND_STATUS_REG), NCR5380_read(STATUS_REG)); @@ -1189,27 +1179,27 @@ static irqreturn_t NCR5380_intr (int irq, void *dev_id) int done = 1, handled = 0; unsigned char basr; - INT_PRINTK("scsi%d: NCR5380 irq triggered\n", HOSTNO); + dprintk(NDEBUG_INTR, "scsi%d: NCR5380 irq triggered\n", HOSTNO); /* Look for pending interrupts */ basr = NCR5380_read(BUS_AND_STATUS_REG); - INT_PRINTK("scsi%d: BASR=%02x\n", HOSTNO, basr); + dprintk(NDEBUG_INTR, "scsi%d: BASR=%02x\n", HOSTNO, basr); /* dispatch to appropriate routine if found and done=0 */ if (basr & BASR_IRQ) { - NCR_PRINT(NDEBUG_INTR); + NCR5380_dprint(NDEBUG_INTR, instance); if ((NCR5380_read(STATUS_REG) & (SR_SEL|SR_IO)) == (SR_SEL|SR_IO)) { done = 0; // ENABLE_IRQ(); - INT_PRINTK("scsi%d: SEL interrupt\n", HOSTNO); + dprintk(NDEBUG_INTR, "scsi%d: SEL interrupt\n", HOSTNO); NCR5380_reselect(instance); (void) NCR5380_read(RESET_PARITY_INTERRUPT_REG); } else if (basr & BASR_PARITY_ERROR) { - INT_PRINTK("scsi%d: PARITY interrupt\n", HOSTNO); + dprintk(NDEBUG_INTR, "scsi%d: PARITY interrupt\n", HOSTNO); (void) NCR5380_read(RESET_PARITY_INTERRUPT_REG); } else if ((NCR5380_read(STATUS_REG) & SR_RST) == SR_RST) { - INT_PRINTK("scsi%d: RESET interrupt\n", HOSTNO); + dprintk(NDEBUG_INTR, "scsi%d: RESET interrupt\n", HOSTNO); (void)NCR5380_read(RESET_PARITY_INTERRUPT_REG); } else { @@ -1229,7 +1219,7 @@ static irqreturn_t NCR5380_intr (int irq, void *dev_id) ((basr & BASR_END_DMA_TRANSFER) || !(basr & BASR_PHASE_MATCH))) { - INT_PRINTK("scsi%d: PHASE MISM or EOP interrupt\n", HOSTNO); + dprintk(NDEBUG_INTR, "scsi%d: PHASE MISM or EOP interrupt\n", HOSTNO); NCR5380_dma_complete( instance ); done = 0; // ENABLE_IRQ(); @@ -1238,7 +1228,7 @@ static irqreturn_t NCR5380_intr (int irq, void *dev_id) { /* MS: Ignore unknown phase mismatch interrupts (caused by EOP interrupt) */ if (basr & BASR_PHASE_MATCH) - INT_PRINTK("scsi%d: unknown interrupt, " + dprintk(NDEBUG_INTR, "scsi%d: unknown interrupt, " "BASR 0x%x, MR 0x%x, SR 0x%x\n", HOSTNO, basr, NCR5380_read(MODE_REG), NCR5380_read(STATUS_REG)); @@ -1262,7 +1252,7 @@ static irqreturn_t NCR5380_intr (int irq, void *dev_id) } if (!done) { - INT_PRINTK("scsi%d: in int routine, calling main\n", HOSTNO); + dprintk(NDEBUG_INTR, "scsi%d: in int routine, calling main\n", HOSTNO); /* Put a call to NCR5380_main() on the queue... */ queue_main(); } @@ -1338,8 +1328,8 @@ static int NCR5380_select(struct Scsi_Host *instance, struct scsi_cmnd *cmd, unsigned long flags; hostdata->restart_select = 0; - NCR_PRINT(NDEBUG_ARBITRATION); - ARB_PRINTK("scsi%d: starting arbitration, id = %d\n", HOSTNO, + NCR5380_dprint(NDEBUG_ARBITRATION, instance); + dprintk(NDEBUG_ARBITRATION, "scsi%d: starting arbitration, id = %d\n", HOSTNO, instance->this_id); /* @@ -1385,7 +1375,7 @@ static int NCR5380_select(struct Scsi_Host *instance, struct scsi_cmnd *cmd, && !hostdata->connected); #endif - ARB_PRINTK("scsi%d: arbitration complete\n", HOSTNO); + dprintk(NDEBUG_ARBITRATION, "scsi%d: arbitration complete\n", HOSTNO); if (hostdata->connected) { NCR5380_write(MODE_REG, MR_BASE); @@ -1406,7 +1396,7 @@ static int NCR5380_select(struct Scsi_Host *instance, struct scsi_cmnd *cmd, (NCR5380_read(INITIATOR_COMMAND_REG) & ICR_ARBITRATION_LOST) || hostdata->connected) { NCR5380_write(MODE_REG, MR_BASE); - ARB_PRINTK("scsi%d: lost arbitration, deasserting MR_ARBITRATE\n", + dprintk(NDEBUG_ARBITRATION, "scsi%d: lost arbitration, deasserting MR_ARBITRATE\n", HOSTNO); return -1; } @@ -1421,7 +1411,7 @@ static int NCR5380_select(struct Scsi_Host *instance, struct scsi_cmnd *cmd, hostdata->connected) { NCR5380_write(MODE_REG, MR_BASE); NCR5380_write(INITIATOR_COMMAND_REG, ICR_BASE); - ARB_PRINTK("scsi%d: lost arbitration, deasserting ICR_ASSERT_SEL\n", + dprintk(NDEBUG_ARBITRATION, "scsi%d: lost arbitration, deasserting ICR_ASSERT_SEL\n", HOSTNO); return -1; } @@ -1444,7 +1434,7 @@ static int NCR5380_select(struct Scsi_Host *instance, struct scsi_cmnd *cmd, return -1; } - ARB_PRINTK("scsi%d: won arbitration\n", HOSTNO); + dprintk(NDEBUG_ARBITRATION, "scsi%d: won arbitration\n", HOSTNO); /* * Now that we have won arbitration, start Selection process, asserting @@ -1504,7 +1494,7 @@ static int NCR5380_select(struct Scsi_Host *instance, struct scsi_cmnd *cmd, udelay(1); - SEL_PRINTK("scsi%d: selecting target %d\n", HOSTNO, cmd->device->id); + dprintk(NDEBUG_SELECTION, "scsi%d: selecting target %d\n", HOSTNO, cmd->device->id); /* * The SCSI specification calls for a 250 ms timeout for the actual @@ -1559,7 +1549,7 @@ static int NCR5380_select(struct Scsi_Host *instance, struct scsi_cmnd *cmd, printk(KERN_ERR "scsi%d: weirdness\n", HOSTNO); if (hostdata->restart_select) printk(KERN_NOTICE "\trestart select\n"); - NCR_PRINT(NDEBUG_ANY); + NCR5380_dprint(NDEBUG_ANY, instance); NCR5380_write(SELECT_ENABLE_REG, hostdata->id_mask); return -1; } @@ -1572,7 +1562,7 @@ static int NCR5380_select(struct Scsi_Host *instance, struct scsi_cmnd *cmd, #endif cmd->scsi_done(cmd); NCR5380_write(SELECT_ENABLE_REG, hostdata->id_mask); - SEL_PRINTK("scsi%d: target did not respond within 250ms\n", HOSTNO); + dprintk(NDEBUG_SELECTION, "scsi%d: target did not respond within 250ms\n", HOSTNO); NCR5380_write(SELECT_ENABLE_REG, hostdata->id_mask); return 0; } @@ -1597,7 +1587,7 @@ static int NCR5380_select(struct Scsi_Host *instance, struct scsi_cmnd *cmd, /* Wait for start of REQ/ACK handshake */ while (!(NCR5380_read(STATUS_REG) & SR_REQ)); - SEL_PRINTK("scsi%d: target %d selected, going into MESSAGE OUT phase.\n", + dprintk(NDEBUG_SELECTION, "scsi%d: target %d selected, going into MESSAGE OUT phase.\n", HOSTNO, cmd->device->id); tmp[0] = IDENTIFY(1, cmd->device->lun); @@ -1617,7 +1607,7 @@ static int NCR5380_select(struct Scsi_Host *instance, struct scsi_cmnd *cmd, data = tmp; phase = PHASE_MSGOUT; NCR5380_transfer_pio(instance, &phase, &len, &data); - SEL_PRINTK("scsi%d: nexus established.\n", HOSTNO); + dprintk(NDEBUG_SELECTION, "scsi%d: nexus established.\n", HOSTNO); /* XXX need to handle errors here */ hostdata->connected = cmd; #ifndef SUPPORT_TAGS @@ -1680,12 +1670,12 @@ static int NCR5380_transfer_pio( struct Scsi_Host *instance, */ while (!((tmp = NCR5380_read(STATUS_REG)) & SR_REQ)); - HSH_PRINTK("scsi%d: REQ detected\n", HOSTNO); + dprintk(NDEBUG_HANDSHAKE, "scsi%d: REQ detected\n", HOSTNO); /* Check for phase mismatch */ if ((tmp & PHASE_MASK) != p) { - PIO_PRINTK("scsi%d: phase mismatch\n", HOSTNO); - NCR_PRINT_PHASE(NDEBUG_PIO); + dprintk(NDEBUG_PIO, "scsi%d: phase mismatch\n", HOSTNO); + NCR5380_dprint_phase(NDEBUG_PIO, instance); break; } @@ -1708,24 +1698,24 @@ static int NCR5380_transfer_pio( struct Scsi_Host *instance, if (!((p & SR_MSG) && c > 1)) { NCR5380_write(INITIATOR_COMMAND_REG, ICR_BASE | ICR_ASSERT_DATA); - NCR_PRINT(NDEBUG_PIO); + NCR5380_dprint(NDEBUG_PIO, instance); NCR5380_write(INITIATOR_COMMAND_REG, ICR_BASE | ICR_ASSERT_DATA | ICR_ASSERT_ACK); } else { NCR5380_write(INITIATOR_COMMAND_REG, ICR_BASE | ICR_ASSERT_DATA | ICR_ASSERT_ATN); - NCR_PRINT(NDEBUG_PIO); + NCR5380_dprint(NDEBUG_PIO, instance); NCR5380_write(INITIATOR_COMMAND_REG, ICR_BASE | ICR_ASSERT_DATA | ICR_ASSERT_ATN | ICR_ASSERT_ACK); } } else { - NCR_PRINT(NDEBUG_PIO); + NCR5380_dprint(NDEBUG_PIO, instance); NCR5380_write(INITIATOR_COMMAND_REG, ICR_BASE | ICR_ASSERT_ACK); } while (NCR5380_read(STATUS_REG) & SR_REQ); - HSH_PRINTK("scsi%d: req false, handshake complete\n", HOSTNO); + dprintk(NDEBUG_HANDSHAKE, "scsi%d: req false, handshake complete\n", HOSTNO); /* * We have several special cases to consider during REQ/ACK handshaking : @@ -1746,7 +1736,7 @@ static int NCR5380_transfer_pio( struct Scsi_Host *instance, } } while (--c); - PIO_PRINTK("scsi%d: residual %d\n", HOSTNO, c); + dprintk(NDEBUG_PIO, "scsi%d: residual %d\n", HOSTNO, c); *count = c; *data = d; @@ -1854,7 +1844,7 @@ static int NCR5380_transfer_dma( struct Scsi_Host *instance, } hostdata->dma_len = c; - DMA_PRINTK("scsi%d: initializing DMA for %s, %d bytes %s %p\n", + dprintk(NDEBUG_DMA, "scsi%d: initializing DMA for %s, %d bytes %s %p\n", HOSTNO, (p & SR_IO) ? "reading" : "writing", c, (p & SR_IO) ? "to" : "from", *data); @@ -1931,7 +1921,7 @@ static void NCR5380_information_transfer (struct Scsi_Host *instance) phase = (tmp & PHASE_MASK); if (phase != old_phase) { old_phase = phase; - NCR_PRINT_PHASE(NDEBUG_INFORMATION); + NCR5380_dprint_phase(NDEBUG_INFORMATION, instance); } if(phase == PHASE_CMDOUT) { @@ -1996,7 +1986,7 @@ static void NCR5380_information_transfer (struct Scsi_Host *instance) --cmd->SCp.buffers_residual; cmd->SCp.this_residual = cmd->SCp.buffer->length; cmd->SCp.ptr = SGADDR(cmd->SCp.buffer); - INF_PRINTK("scsi%d: %d bytes and %d buffers left\n", + dprintk(NDEBUG_INFORMATION, "scsi%d: %d bytes and %d buffers left\n", HOSTNO, cmd->SCp.this_residual, cmd->SCp.buffers_residual); } @@ -2088,7 +2078,7 @@ static void NCR5380_information_transfer (struct Scsi_Host *instance) /* Accept message by clearing ACK */ NCR5380_write(INITIATOR_COMMAND_REG, ICR_BASE); - LNK_PRINTK("scsi%d: target %d lun %d linked command " + dprintk(NDEBUG_LINKED, "scsi%d: target %d lun %d linked command " "complete.\n", HOSTNO, cmd->device->id, cmd->device->lun); /* Enable reselect interrupts */ @@ -2113,7 +2103,7 @@ static void NCR5380_information_transfer (struct Scsi_Host *instance) * and don't free it! */ cmd->next_link->tag = cmd->tag; cmd->result = cmd->SCp.Status | (cmd->SCp.Message << 8); - LNK_PRINTK("scsi%d: target %d lun %d linked request " + dprintk(NDEBUG_LINKED, "scsi%d: target %d lun %d linked request " "done, calling scsi_done().\n", HOSTNO, cmd->device->id, cmd->device->lun); #ifdef NCR5380_STATS @@ -2128,7 +2118,7 @@ static void NCR5380_information_transfer (struct Scsi_Host *instance) /* Accept message by clearing ACK */ NCR5380_write(INITIATOR_COMMAND_REG, ICR_BASE); hostdata->connected = NULL; - QU_PRINTK("scsi%d: command for target %d, lun %d " + dprintk(NDEBUG_QUEUES, "scsi%d: command for target %d, lun %d " "completed\n", HOSTNO, cmd->device->id, cmd->device->lun); #ifdef SUPPORT_TAGS cmd_free_tag( cmd ); @@ -2142,7 +2132,7 @@ static void NCR5380_information_transfer (struct Scsi_Host *instance) /* ++Andreas: the mid level code knows about QUEUE_FULL now. */ TAG_ALLOC *ta = &TagAlloc[cmd->device->id][cmd->device->lun]; - TAG_PRINTK("scsi%d: target %d lun %d returned " + dprintk(NDEBUG_TAGS, "scsi%d: target %d lun %d returned " "QUEUE_FULL after %d commands\n", HOSTNO, cmd->device->id, cmd->device->lun, ta->nr_allocated); @@ -2186,7 +2176,7 @@ static void NCR5380_information_transfer (struct Scsi_Host *instance) if ((cmd->cmnd[0] != REQUEST_SENSE) && (status_byte(cmd->SCp.Status) == CHECK_CONDITION)) { scsi_eh_prep_cmnd(cmd, &hostdata->ses, NULL, 0, ~0); - ASEN_PRINTK("scsi%d: performing request sense\n", + dprintk(NDEBUG_AUTOSENSE, "scsi%d: performing request sense\n", HOSTNO); /* this is initialized from initialize_SCp cmd->SCp.buffer = NULL; @@ -2198,7 +2188,7 @@ static void NCR5380_information_transfer (struct Scsi_Host *instance) SET_NEXT(cmd, hostdata->issue_queue); hostdata->issue_queue = (struct scsi_cmnd *) cmd; local_irq_restore(flags); - QU_PRINTK("scsi%d: REQUEST SENSE added to head of " + dprintk(NDEBUG_QUEUES, "scsi%d: REQUEST SENSE added to head of " "issue queue\n", H_NO(cmd)); } else #endif /* def AUTOSENSE */ @@ -2238,7 +2228,7 @@ static void NCR5380_information_transfer (struct Scsi_Host *instance) cmd->device->tagged_supported = 0; hostdata->busy[cmd->device->id] |= (1 << cmd->device->lun); cmd->tag = TAG_NONE; - TAG_PRINTK("scsi%d: target %d lun %d rejected " + dprintk(NDEBUG_TAGS, "scsi%d: target %d lun %d rejected " "QUEUE_TAG message; tagged queuing " "disabled\n", HOSTNO, cmd->device->id, cmd->device->lun); @@ -2255,7 +2245,7 @@ static void NCR5380_information_transfer (struct Scsi_Host *instance) hostdata->connected = NULL; hostdata->disconnected_queue = cmd; local_irq_restore(flags); - QU_PRINTK("scsi%d: command for target %d lun %d was " + dprintk(NDEBUG_QUEUES, "scsi%d: command for target %d lun %d was " "moved from connected to the " "disconnected_queue\n", HOSTNO, cmd->device->id, cmd->device->lun); @@ -2308,13 +2298,13 @@ static void NCR5380_information_transfer (struct Scsi_Host *instance) /* Accept first byte by clearing ACK */ NCR5380_write(INITIATOR_COMMAND_REG, ICR_BASE); - EXT_PRINTK("scsi%d: receiving extended message\n", HOSTNO); + dprintk(NDEBUG_EXTENDED, "scsi%d: receiving extended message\n", HOSTNO); len = 2; data = extended_msg + 1; phase = PHASE_MSGIN; NCR5380_transfer_pio(instance, &phase, &len, &data); - EXT_PRINTK("scsi%d: length=%d, code=0x%02x\n", HOSTNO, + dprintk(NDEBUG_EXTENDED, "scsi%d: length=%d, code=0x%02x\n", HOSTNO, (int)extended_msg[1], (int)extended_msg[2]); if (!len && extended_msg[1] <= @@ -2326,7 +2316,7 @@ static void NCR5380_information_transfer (struct Scsi_Host *instance) phase = PHASE_MSGIN; NCR5380_transfer_pio(instance, &phase, &len, &data); - EXT_PRINTK("scsi%d: message received, residual %d\n", + dprintk(NDEBUG_EXTENDED, "scsi%d: message received, residual %d\n", HOSTNO, len); switch (extended_msg[2]) { @@ -2416,7 +2406,7 @@ static void NCR5380_information_transfer (struct Scsi_Host *instance) break; default: printk("scsi%d: unknown phase\n", HOSTNO); - NCR_PRINT(NDEBUG_ANY); + NCR5380_dprint(NDEBUG_ANY, instance); } /* switch(phase) */ } /* if (tmp * SR_REQ) */ } /* while (1) */ @@ -2458,7 +2448,7 @@ static void NCR5380_reselect (struct Scsi_Host *instance) target_mask = NCR5380_read(CURRENT_SCSI_DATA_REG) & ~(hostdata->id_mask); - RSL_PRINTK("scsi%d: reselect\n", HOSTNO); + dprintk(NDEBUG_RESELECTION, "scsi%d: reselect\n", HOSTNO); /* * At this point, we have detected that our SCSI ID is on the bus, @@ -2580,14 +2570,14 @@ static void NCR5380_reselect (struct Scsi_Host *instance) if (!NCR5380_transfer_pio(instance, &phase, &len, &data) && msg[1] == SIMPLE_QUEUE_TAG) tag = msg[2]; - TAG_PRINTK("scsi%d: target mask %02x, lun %d sent tag %d at " + dprintk(NDEBUG_TAGS, "scsi%d: target mask %02x, lun %d sent tag %d at " "reselection\n", HOSTNO, target_mask, lun, tag); } #endif hostdata->connected = tmp; - RSL_PRINTK("scsi%d: nexus established, target = %d, lun = %d, tag = %d\n", - HOSTNO, tmp->target, tmp->lun, tmp->tag); + dprintk(NDEBUG_RESELECTION, "scsi%d: nexus established, target = %d, lun = %d, tag = %d\n", + HOSTNO, tmp->device->id, tmp->device->lun, tmp->tag); } @@ -2622,7 +2612,7 @@ static int NCR5380_abort(struct scsi_cmnd *cmd) local_irq_save(flags); - ABRT_PRINTK("scsi%d: abort called basr 0x%02x, sr 0x%02x\n", HOSTNO, + dprintk(NDEBUG_ABORT, "scsi%d: abort called basr 0x%02x, sr 0x%02x\n", HOSTNO, NCR5380_read(BUS_AND_STATUS_REG), NCR5380_read(STATUS_REG)); @@ -2635,7 +2625,7 @@ static int NCR5380_abort(struct scsi_cmnd *cmd) if (hostdata->connected == cmd) { - ABRT_PRINTK("scsi%d: aborting connected command\n", HOSTNO); + dprintk(NDEBUG_ABORT, "scsi%d: aborting connected command\n", HOSTNO); /* * We should perform BSY checking, and make sure we haven't slipped * into BUS FREE. @@ -2664,11 +2654,11 @@ static int NCR5380_abort(struct scsi_cmnd *cmd) #endif local_irq_restore(flags); cmd->scsi_done(cmd); - return SCSI_ABORT_SUCCESS; + return SUCCESS; } else { /* local_irq_restore(flags); */ printk("scsi%d: abort of connected command failed!\n", HOSTNO); - return SCSI_ABORT_ERROR; + return FAILED; } } #endif @@ -2686,12 +2676,12 @@ static int NCR5380_abort(struct scsi_cmnd *cmd) SET_NEXT(tmp, NULL); tmp->result = DID_ABORT << 16; local_irq_restore(flags); - ABRT_PRINTK("scsi%d: abort removed command from issue queue.\n", + dprintk(NDEBUG_ABORT, "scsi%d: abort removed command from issue queue.\n", HOSTNO); /* Tagged queuing note: no tag to free here, hasn't been assigned * yet... */ tmp->scsi_done(tmp); - return SCSI_ABORT_SUCCESS; + return SUCCESS; } /* @@ -2707,8 +2697,8 @@ static int NCR5380_abort(struct scsi_cmnd *cmd) if (hostdata->connected) { local_irq_restore(flags); - ABRT_PRINTK("scsi%d: abort failed, command connected.\n", HOSTNO); - return SCSI_ABORT_SNOOZE; + dprintk(NDEBUG_ABORT, "scsi%d: abort failed, command connected.\n", HOSTNO); + return FAILED; } /* @@ -2740,12 +2730,12 @@ static int NCR5380_abort(struct scsi_cmnd *cmd) tmp = NEXT(tmp)) if (cmd == tmp) { local_irq_restore(flags); - ABRT_PRINTK("scsi%d: aborting disconnected command.\n", HOSTNO); + dprintk(NDEBUG_ABORT, "scsi%d: aborting disconnected command.\n", HOSTNO); if (NCR5380_select (instance, cmd, (int) cmd->tag)) - return SCSI_ABORT_BUSY; + return FAILED; - ABRT_PRINTK("scsi%d: nexus reestablished.\n", HOSTNO); + dprintk(NDEBUG_ABORT, "scsi%d: nexus reestablished.\n", HOSTNO); do_abort (instance); @@ -2769,7 +2759,7 @@ static int NCR5380_abort(struct scsi_cmnd *cmd) #endif local_irq_restore(flags); tmp->scsi_done(tmp); - return SCSI_ABORT_SUCCESS; + return SUCCESS; } } @@ -2786,7 +2776,7 @@ static int NCR5380_abort(struct scsi_cmnd *cmd) local_irq_restore(flags); printk(KERN_INFO "scsi%d: warning : SCSI command probably completed successfully before abortion\n", HOSTNO); - return SCSI_ABORT_NOT_RUNNING; + return FAILED; } @@ -2795,7 +2785,7 @@ static int NCR5380_abort(struct scsi_cmnd *cmd) * * Purpose : reset the SCSI bus. * - * Returns : SCSI_RESET_WAKEUP + * Returns : SUCCESS or FAILURE * */ @@ -2804,7 +2794,7 @@ static int NCR5380_bus_reset(struct scsi_cmnd *cmd) SETUP_HOSTDATA(cmd->device->host); int i; unsigned long flags; -#if 1 +#if defined(RESET_RUN_DONE) struct scsi_cmnd *connected, *disconnected_queue; #endif @@ -2826,8 +2816,15 @@ static int NCR5380_bus_reset(struct scsi_cmnd *cmd) * through anymore ... */ (void)NCR5380_read( RESET_PARITY_INTERRUPT_REG ); -#if 1 /* XXX Should now be done by midlevel code, but it's broken XXX */ - /* XXX see below XXX */ + /* MSch 20140115 - looking at the generic NCR5380 driver, all of this + * should go. + * Catch-22: if we don't clear all queues, the SCSI driver lock will + * not be released by atari_scsi_reset()! + */ + +#if defined(RESET_RUN_DONE) + /* XXX Should now be done by midlevel code, but it's broken XXX */ + /* XXX see below XXX */ /* MSch: old-style reset: actually abort all command processing here */ @@ -2857,7 +2854,7 @@ static int NCR5380_bus_reset(struct scsi_cmnd *cmd) */ if ((cmd = connected)) { - ABRT_PRINTK("scsi%d: reset aborted a connected command\n", H_NO(cmd)); + dprintk(NDEBUG_ABORT, "scsi%d: reset aborted a connected command\n", H_NO(cmd)); cmd->result = (cmd->result & 0xffff) | (DID_RESET << 16); cmd->scsi_done( cmd ); } @@ -2869,14 +2866,14 @@ static int NCR5380_bus_reset(struct scsi_cmnd *cmd) cmd->scsi_done( cmd ); } if (i > 0) - ABRT_PRINTK("scsi: reset aborted %d disconnected command(s)\n", i); + dprintk(NDEBUG_ABORT, "scsi: reset aborted %d disconnected command(s)\n", i); /* since all commands have been explicitly terminated, we need to tell * the midlevel code that the reset was SUCCESSFUL, and there is no * need to 'wake up' the commands by a request_sense */ - return SCSI_RESET_SUCCESS | SCSI_RESET_BUS_RESET; + return SUCCESS; #else /* 1 */ /* MSch: new-style reset handling: let the mid-level do what it can */ @@ -2903,11 +2900,11 @@ static int NCR5380_bus_reset(struct scsi_cmnd *cmd) */ if (hostdata->issue_queue) - ABRT_PRINTK("scsi%d: reset aborted issued command(s)\n", H_NO(cmd)); + dprintk(NDEBUG_ABORT, "scsi%d: reset aborted issued command(s)\n", H_NO(cmd)); if (hostdata->connected) - ABRT_PRINTK("scsi%d: reset aborted a connected command\n", H_NO(cmd)); + dprintk(NDEBUG_ABORT, "scsi%d: reset aborted a connected command\n", H_NO(cmd)); if (hostdata->disconnected_queue) - ABRT_PRINTK("scsi%d: reset aborted disconnected command(s)\n", H_NO(cmd)); + dprintk(NDEBUG_ABORT, "scsi%d: reset aborted disconnected command(s)\n", H_NO(cmd)); local_irq_save(flags); hostdata->issue_queue = NULL; @@ -2924,7 +2921,7 @@ static int NCR5380_bus_reset(struct scsi_cmnd *cmd) local_irq_restore(flags); /* we did no complete reset of all commands, so a wakeup is required */ - return SCSI_RESET_WAKEUP | SCSI_RESET_BUS_RESET; + return SUCCESS; #endif /* 1 */ } diff --git a/drivers/scsi/sun3_scsi.c b/drivers/scsi/sun3_scsi.c index e2c009b033ce..9707b7494a89 100644 --- a/drivers/scsi/sun3_scsi.c +++ b/drivers/scsi/sun3_scsi.c @@ -3,6 +3,10 @@ * * Sun3 DMA routines added by Sam Creasey (sammy@sammy.net) * + * VME support added by Sam Creasey + * + * TODO: modify this driver to support multiple Sun3 SCSI VME boards + * * Adapted from mac_scsinew.c: */ /* @@ -45,10 +49,6 @@ * USLEEP - enable support for devices that don't disconnect. Untested. */ -/* - * $Log: sun3_NCR5380.c,v $ - */ - #define AUTOSENSE #include <linux/types.h> @@ -69,23 +69,15 @@ #include <asm/idprom.h> #include <asm/machines.h> -#define NDEBUG 0 - -#define NDEBUG_ABORT 0x00100000 -#define NDEBUG_TAGS 0x00200000 -#define NDEBUG_MERGING 0x00400000 - /* dma on! */ #define REAL_DMA #include "scsi.h" -#include "initio.h" #include <scsi/scsi_host.h> #include "sun3_scsi.h" +#include "NCR5380.h" -static void NCR5380_print(struct Scsi_Host *instance); - -/* #define OLDDMA */ +extern int sun3_map_test(unsigned long, char *); #define USE_WRAPPER /*#define RESET_BOOT */ @@ -101,7 +93,11 @@ static void NCR5380_print(struct Scsi_Host *instance); /* #define SUPPORT_TAGS */ +#ifdef SUN3_SCSI_VME +#define ENABLE_IRQ() +#else #define ENABLE_IRQ() enable_irq( IRQ_SUN3_SCSI ); +#endif static irqreturn_t scsi_sun3_intr(int irq, void *dummy); @@ -123,6 +119,8 @@ module_param(setup_hostid, int, 0); static struct scsi_cmnd *sun3_dma_setup_done = NULL; +#define RESET_RUN_DONE + #define AFTER_RESET_DELAY (HZ/2) /* ms to wait after hitting dma regs */ @@ -136,10 +134,9 @@ static struct scsi_cmnd *sun3_dma_setup_done = NULL; static volatile unsigned char *sun3_scsi_regp; static volatile struct sun3_dma_regs *dregs; -#ifdef OLDDMA -static unsigned char *dmabuf = NULL; /* dma memory buffer */ -#endif +#ifndef SUN3_SCSI_VME static struct sun3_udc_regs *udc_regs = NULL; +#endif static unsigned char *sun3_dma_orig_addr = NULL; static unsigned long sun3_dma_orig_count = 0; static int sun3_dma_active = 0; @@ -159,6 +156,7 @@ static inline void sun3scsi_write(int reg, int value) sun3_scsi_regp[reg] = value; } +#ifndef SUN3_SCSI_VME /* dma controller register access functions */ static inline unsigned short sun3_udc_read(unsigned char reg) @@ -180,6 +178,7 @@ static inline void sun3_udc_write(unsigned short val, unsigned char reg) dregs->udc_data = val; udelay(SUN3_DMA_DELAY); } +#endif /* * XXX: status debug @@ -198,17 +197,32 @@ static struct Scsi_Host *default_instance; * */ -int __init sun3scsi_detect(struct scsi_host_template * tpnt) +static int __init sun3scsi_detect(struct scsi_host_template *tpnt) { - unsigned long ioaddr; + unsigned long ioaddr, irq; static int called = 0; struct Scsi_Host *instance; +#ifdef SUN3_SCSI_VME + int i; + unsigned long addrs[3] = { IOBASE_SUN3_VMESCSI, + IOBASE_SUN3_VMESCSI + 0x4000, + 0 }; + unsigned long vecs[3] = { SUN3_VEC_VMESCSI0, + SUN3_VEC_VMESCSI1, + 0 }; +#endif /* check that this machine has an onboard 5380 */ switch(idprom->id_machtype) { +#ifdef SUN3_SCSI_VME + case SM_SUN3|SM_3_160: + case SM_SUN3|SM_3_260: + break; +#else case SM_SUN3|SM_3_50: case SM_SUN3|SM_3_60: break; +#endif default: return 0; @@ -217,7 +231,11 @@ int __init sun3scsi_detect(struct scsi_host_template * tpnt) if(called) return 0; +#ifdef SUN3_SCSI_VME + tpnt->proc_name = "Sun3 5380 VME SCSI"; +#else tpnt->proc_name = "Sun3 5380 SCSI"; +#endif /* setup variables */ tpnt->can_queue = @@ -234,6 +252,38 @@ int __init sun3scsi_detect(struct scsi_host_template * tpnt) tpnt->this_id = 7; } +#ifdef SUN3_SCSI_VME + ioaddr = 0; + for (i = 0; addrs[i] != 0; i++) { + unsigned char x; + + ioaddr = (unsigned long)sun3_ioremap(addrs[i], PAGE_SIZE, + SUN3_PAGE_TYPE_VME16); + irq = vecs[i]; + sun3_scsi_regp = (unsigned char *)ioaddr; + + dregs = (struct sun3_dma_regs *)(((unsigned char *)ioaddr) + 8); + + if (sun3_map_test((unsigned long)dregs, &x)) { + unsigned short oldcsr; + + oldcsr = dregs->csr; + dregs->csr = 0; + udelay(SUN3_DMA_DELAY); + if (dregs->csr == 0x1400) + break; + + dregs->csr = oldcsr; + } + + iounmap((void *)ioaddr); + ioaddr = 0; + } + + if (!ioaddr) + return 0; +#else + irq = IRQ_SUN3_SCSI; ioaddr = (unsigned long)ioremap(IOBASE_SUN3_SCSI, PAGE_SIZE); sun3_scsi_regp = (unsigned char *)ioaddr; @@ -244,11 +294,6 @@ int __init sun3scsi_detect(struct scsi_host_template * tpnt) printk("SUN3 Scsi couldn't allocate DVMA memory!\n"); return 0; } -#ifdef OLDDMA - if((dmabuf = dvma_malloc_align(SUN3_DVMA_BUFSIZE, 0x10000)) == NULL) { - printk("SUN3 Scsi couldn't allocate DVMA memory!\n"); - return 0; - } #endif #ifdef SUPPORT_TAGS if (setup_use_tagged_queuing < 0) @@ -262,7 +307,7 @@ int __init sun3scsi_detect(struct scsi_host_template * tpnt) default_instance = instance; instance->io_port = (unsigned long) ioaddr; - instance->irq = IRQ_SUN3_SCSI; + instance->irq = irq; NCR5380_init(instance, 0); @@ -283,7 +328,8 @@ int __init sun3scsi_detect(struct scsi_host_template * tpnt) #endif } - printk("scsi%d: Sun3 5380 at port %lX irq", instance->host_no, instance->io_port); + pr_info("scsi%d: %s at port %lX irq", instance->host_no, + tpnt->proc_name, instance->io_port); if (instance->irq == SCSI_IRQ_NONE) printk ("s disabled"); else @@ -300,6 +346,15 @@ int __init sun3scsi_detect(struct scsi_host_template * tpnt) dregs->csr = CSR_SCSI | CSR_FIFO | CSR_INTR; udelay(SUN3_DMA_DELAY); dregs->fifo_count = 0; +#ifdef SUN3_SCSI_VME + dregs->fifo_count_hi = 0; + dregs->dma_addr_hi = 0; + dregs->dma_addr_lo = 0; + dregs->dma_count_hi = 0; + dregs->dma_count_lo = 0; + + dregs->ivect = VME_DATA24 | (instance->irq & 0xff); +#endif called = 1; @@ -367,7 +422,8 @@ static void sun3_scsi_reset_boot(struct Scsi_Host *instance) } #endif -const char * sun3scsi_info (struct Scsi_Host *spnt) { +static const char *sun3scsi_info(struct Scsi_Host *spnt) +{ return ""; } @@ -379,6 +435,10 @@ static irqreturn_t scsi_sun3_intr(int irq, void *dummy) unsigned short csr = dregs->csr; int handled = 0; +#ifdef SUN3_SCSI_VME + dregs->csr &= ~CSR_DMA_ENABLE; +#endif + if(csr & ~CSR_GOOD) { if(csr & CSR_DMA_BUSERR) { printk("scsi%d: bus error in dma\n", default_instance->host_no); @@ -422,31 +482,28 @@ void sun3_sun3_debug (void) /* sun3scsi_dma_setup() -- initialize the dma controller for a read/write */ static unsigned long sun3scsi_dma_setup(void *data, unsigned long count, int write_flag) { -#ifdef OLDDMA - if(write_flag) - memcpy(dmabuf, data, count); - else { - sun3_dma_orig_addr = data; - sun3_dma_orig_count = count; - } -#else void *addr; if(sun3_dma_orig_addr != NULL) dvma_unmap(sun3_dma_orig_addr); -// addr = sun3_dvma_page((unsigned long)data, (unsigned long)dmabuf); +#ifdef SUN3_SCSI_VME + addr = (void *)dvma_map_vme((unsigned long) data, count); +#else addr = (void *)dvma_map((unsigned long) data, count); +#endif sun3_dma_orig_addr = addr; sun3_dma_orig_count = count; -#endif + +#ifndef SUN3_SCSI_VME dregs->fifo_count = 0; sun3_udc_write(UDC_RESET, UDC_CSR); /* reset fifo */ dregs->csr &= ~CSR_FIFO; dregs->csr |= CSR_FIFO; +#endif /* set direction */ if(write_flag) @@ -454,6 +511,17 @@ static unsigned long sun3scsi_dma_setup(void *data, unsigned long count, int wri else dregs->csr &= ~CSR_SEND; +#ifdef SUN3_SCSI_VME + dregs->csr |= CSR_PACK_ENABLE; + + dregs->dma_addr_hi = ((unsigned long)addr >> 16); + dregs->dma_addr_lo = ((unsigned long)addr & 0xffff); + + dregs->dma_count_hi = 0; + dregs->dma_count_lo = 0; + dregs->fifo_count_hi = 0; + dregs->fifo_count = 0; +#else /* byte count for fifo */ dregs->fifo_count = count; @@ -467,17 +535,12 @@ static unsigned long sun3scsi_dma_setup(void *data, unsigned long count, int wri printk("scsi%d: fifo_mismatch %04x not %04x\n", default_instance->host_no, dregs->fifo_count, (unsigned int) count); - NCR5380_print(default_instance); + NCR5380_dprint(NDEBUG_DMA, default_instance); } /* setup udc */ -#ifdef OLDDMA - udc_regs->addr_hi = ((dvma_vtob(dmabuf) & 0xff0000) >> 8); - udc_regs->addr_lo = (dvma_vtob(dmabuf) & 0xffff); -#else udc_regs->addr_hi = (((unsigned long)(addr) & 0xff0000) >> 8); udc_regs->addr_lo = ((unsigned long)(addr) & 0xffff); -#endif udc_regs->count = count/2; /* count in words */ udc_regs->mode_hi = UDC_MODE_HIWORD; if(write_flag) { @@ -501,11 +564,13 @@ static unsigned long sun3scsi_dma_setup(void *data, unsigned long count, int wri /* interrupt enable */ sun3_udc_write(UDC_INT_ENABLE, UDC_CSR); +#endif return count; } +#ifndef SUN3_SCSI_VME static inline unsigned long sun3scsi_dma_count(struct Scsi_Host *instance) { unsigned short resid; @@ -518,6 +583,7 @@ static inline unsigned long sun3scsi_dma_count(struct Scsi_Host *instance) return (unsigned long) resid; } +#endif static inline unsigned long sun3scsi_dma_residual(struct Scsi_Host *instance) { @@ -536,8 +602,23 @@ static inline unsigned long sun3scsi_dma_xfer_len(unsigned long wanted, static inline int sun3scsi_dma_start(unsigned long count, unsigned char *data) { +#ifdef SUN3_SCSI_VME + unsigned short csr; + + csr = dregs->csr; + dregs->dma_count_hi = (sun3_dma_orig_count >> 16); + dregs->dma_count_lo = (sun3_dma_orig_count & 0xffff); + + dregs->fifo_count_hi = (sun3_dma_orig_count >> 16); + dregs->fifo_count = (sun3_dma_orig_count & 0xffff); + +/* if(!(csr & CSR_DMA_ENABLE)) + * dregs->csr |= CSR_DMA_ENABLE; + */ +#else sun3_udc_write(UDC_CHN_START, UDC_CSR); +#endif return 0; } @@ -545,12 +626,46 @@ static inline int sun3scsi_dma_start(unsigned long count, unsigned char *data) /* clean up after our dma is done */ static int sun3scsi_dma_finish(int write_flag) { - unsigned short count; + unsigned short __maybe_unused count; unsigned short fifo; int ret = 0; sun3_dma_active = 0; -#if 1 + +#ifdef SUN3_SCSI_VME + dregs->csr &= ~CSR_DMA_ENABLE; + + fifo = dregs->fifo_count; + if (write_flag) { + if ((fifo > 0) && (fifo < sun3_dma_orig_count)) + fifo++; + } + + last_residual = fifo; + /* empty bytes from the fifo which didn't make it */ + if ((!write_flag) && (dregs->csr & CSR_LEFT)) { + unsigned char *vaddr; + + vaddr = (unsigned char *)dvma_vmetov(sun3_dma_orig_addr); + + vaddr += (sun3_dma_orig_count - fifo); + vaddr--; + + switch (dregs->csr & CSR_LEFT) { + case CSR_LEFT_3: + *vaddr = (dregs->bpack_lo & 0xff00) >> 8; + vaddr--; + + case CSR_LEFT_2: + *vaddr = (dregs->bpack_hi & 0x00ff); + vaddr--; + + case CSR_LEFT_1: + *vaddr = (dregs->bpack_hi & 0xff00) >> 8; + break; + } + } +#else // check to empty the fifo on a read if(!write_flag) { int tmo = 20000; /* .2 sec */ @@ -566,28 +681,8 @@ static int sun3scsi_dma_finish(int write_flag) udelay(10); } } - -#endif count = sun3scsi_dma_count(default_instance); -#ifdef OLDDMA - - /* if we've finished a read, copy out the data we read */ - if(sun3_dma_orig_addr) { - /* check for residual bytes after dma end */ - if(count && (NCR5380_read(BUS_AND_STATUS_REG) & - (BASR_PHASE_MATCH | BASR_ACK))) { - printk("scsi%d: sun3_scsi_finish: read overrun baby... ", default_instance->host_no); - printk("basr now %02x\n", NCR5380_read(BUS_AND_STATUS_REG)); - ret = count; - } - - /* copy in what we dma'd no matter what */ - memcpy(sun3_dma_orig_addr, dmabuf, sun3_dma_orig_count); - sun3_dma_orig_addr = NULL; - - } -#else fifo = dregs->fifo_count; last_residual = fifo; @@ -605,10 +700,23 @@ static int sun3scsi_dma_finish(int write_flag) vaddr[-2] = (data & 0xff00) >> 8; vaddr[-1] = (data & 0xff); } +#endif dvma_unmap(sun3_dma_orig_addr); sun3_dma_orig_addr = NULL; -#endif + +#ifdef SUN3_SCSI_VME + dregs->dma_addr_hi = 0; + dregs->dma_addr_lo = 0; + dregs->dma_count_hi = 0; + dregs->dma_count_lo = 0; + + dregs->fifo_count = 0; + dregs->fifo_count_hi = 0; + + dregs->csr &= ~CSR_SEND; +/* dregs->csr |= CSR_DMA_ENABLE; */ +#else sun3_udc_write(UDC_RESET, UDC_CSR); dregs->fifo_count = 0; dregs->csr &= ~CSR_SEND; @@ -616,6 +724,7 @@ static int sun3scsi_dma_finish(int write_flag) /* reset fifo */ dregs->csr &= ~CSR_FIFO; dregs->csr |= CSR_FIFO; +#endif sun3_dma_setup_done = NULL; diff --git a/drivers/scsi/sun3_scsi.h b/drivers/scsi/sun3_scsi.h index a8da9c710fea..e96a37cf06ac 100644 --- a/drivers/scsi/sun3_scsi.h +++ b/drivers/scsi/sun3_scsi.h @@ -29,12 +29,8 @@ * 1+ (800) 334-5454 */ -/* - * $Log: cumana_NCR5380.h,v $ - */ - -#ifndef SUN3_NCR5380_H -#define SUN3_NCR5380_H +#ifndef SUN3_SCSI_H +#define SUN3_SCSI_H #define SUN3SCSI_PUBLIC_RELEASE 1 @@ -82,8 +78,6 @@ static int sun3scsi_release (struct Scsi_Host *); #define SUN3_SCSI_NAME "Sun3 NCR5380 SCSI" #endif -#ifndef HOSTS_C - #define NCR5380_implementation_fields \ int port, ctrl @@ -108,9 +102,6 @@ static int sun3scsi_release (struct Scsi_Host *); #define NCR5380_dma_read_setup(instance, data, count) sun3scsi_dma_setup(data, count, 0) #define NCR5380_dma_residual sun3scsi_dma_residual -#define BOARD_NORMAL 0 -#define BOARD_NCR53C400 1 - /* additional registers - mainly DMA control regs */ /* these start at regbase + 8 -- directly after the NCR regs */ struct sun3_dma_regs { @@ -191,189 +182,5 @@ struct sun3_udc_regs { #define VME_DATA24 0x3d00 -// debugging printk's, taken from atari_scsi.h -/* Debugging printk definitions: - * - * ARB -> arbitration - * ASEN -> auto-sense - * DMA -> DMA - * HSH -> PIO handshake - * INF -> information transfer - * INI -> initialization - * INT -> interrupt - * LNK -> linked commands - * MAIN -> NCR5380_main() control flow - * NDAT -> no data-out phase - * NWR -> no write commands - * PIO -> PIO transfers - * PDMA -> pseudo DMA (unused on Atari) - * QU -> queues - * RSL -> reselections - * SEL -> selections - * USL -> usleep cpde (unused on Atari) - * LBS -> last byte sent (unused on Atari) - * RSS -> restarting of selections - * EXT -> extended messages - * ABRT -> aborting and resetting - * TAG -> queue tag handling - * MER -> merging of consec. buffers - * - */ - -#include "NCR5380.h" - -#if NDEBUG & NDEBUG_ARBITRATION -#define ARB_PRINTK(format, args...) \ - printk(KERN_DEBUG format , ## args) -#else -#define ARB_PRINTK(format, args...) -#endif -#if NDEBUG & NDEBUG_AUTOSENSE -#define ASEN_PRINTK(format, args...) \ - printk(KERN_DEBUG format , ## args) -#else -#define ASEN_PRINTK(format, args...) -#endif -#if NDEBUG & NDEBUG_DMA -#define DMA_PRINTK(format, args...) \ - printk(KERN_DEBUG format , ## args) -#else -#define DMA_PRINTK(format, args...) -#endif -#if NDEBUG & NDEBUG_HANDSHAKE -#define HSH_PRINTK(format, args...) \ - printk(KERN_DEBUG format , ## args) -#else -#define HSH_PRINTK(format, args...) -#endif -#if NDEBUG & NDEBUG_INFORMATION -#define INF_PRINTK(format, args...) \ - printk(KERN_DEBUG format , ## args) -#else -#define INF_PRINTK(format, args...) -#endif -#if NDEBUG & NDEBUG_INIT -#define INI_PRINTK(format, args...) \ - printk(KERN_DEBUG format , ## args) -#else -#define INI_PRINTK(format, args...) -#endif -#if NDEBUG & NDEBUG_INTR -#define INT_PRINTK(format, args...) \ - printk(KERN_DEBUG format , ## args) -#else -#define INT_PRINTK(format, args...) -#endif -#if NDEBUG & NDEBUG_LINKED -#define LNK_PRINTK(format, args...) \ - printk(KERN_DEBUG format , ## args) -#else -#define LNK_PRINTK(format, args...) -#endif -#if NDEBUG & NDEBUG_MAIN -#define MAIN_PRINTK(format, args...) \ - printk(KERN_DEBUG format , ## args) -#else -#define MAIN_PRINTK(format, args...) -#endif -#if NDEBUG & NDEBUG_NO_DATAOUT -#define NDAT_PRINTK(format, args...) \ - printk(KERN_DEBUG format , ## args) -#else -#define NDAT_PRINTK(format, args...) -#endif -#if NDEBUG & NDEBUG_NO_WRITE -#define NWR_PRINTK(format, args...) \ - printk(KERN_DEBUG format , ## args) -#else -#define NWR_PRINTK(format, args...) -#endif -#if NDEBUG & NDEBUG_PIO -#define PIO_PRINTK(format, args...) \ - printk(KERN_DEBUG format , ## args) -#else -#define PIO_PRINTK(format, args...) -#endif -#if NDEBUG & NDEBUG_PSEUDO_DMA -#define PDMA_PRINTK(format, args...) \ - printk(KERN_DEBUG format , ## args) -#else -#define PDMA_PRINTK(format, args...) -#endif -#if NDEBUG & NDEBUG_QUEUES -#define QU_PRINTK(format, args...) \ - printk(KERN_DEBUG format , ## args) -#else -#define QU_PRINTK(format, args...) -#endif -#if NDEBUG & NDEBUG_RESELECTION -#define RSL_PRINTK(format, args...) \ - printk(KERN_DEBUG format , ## args) -#else -#define RSL_PRINTK(format, args...) -#endif -#if NDEBUG & NDEBUG_SELECTION -#define SEL_PRINTK(format, args...) \ - printk(KERN_DEBUG format , ## args) -#else -#define SEL_PRINTK(format, args...) -#endif -#if NDEBUG & NDEBUG_USLEEP -#define USL_PRINTK(format, args...) \ - printk(KERN_DEBUG format , ## args) -#else -#define USL_PRINTK(format, args...) -#endif -#if NDEBUG & NDEBUG_LAST_BYTE_SENT -#define LBS_PRINTK(format, args...) \ - printk(KERN_DEBUG format , ## args) -#else -#define LBS_PRINTK(format, args...) -#endif -#if NDEBUG & NDEBUG_RESTART_SELECT -#define RSS_PRINTK(format, args...) \ - printk(KERN_DEBUG format , ## args) -#else -#define RSS_PRINTK(format, args...) -#endif -#if NDEBUG & NDEBUG_EXTENDED -#define EXT_PRINTK(format, args...) \ - printk(KERN_DEBUG format , ## args) -#else -#define EXT_PRINTK(format, args...) -#endif -#if NDEBUG & NDEBUG_ABORT -#define ABRT_PRINTK(format, args...) \ - printk(KERN_DEBUG format , ## args) -#else -#define ABRT_PRINTK(format, args...) -#endif -#if NDEBUG & NDEBUG_TAGS -#define TAG_PRINTK(format, args...) \ - printk(KERN_DEBUG format , ## args) -#else -#define TAG_PRINTK(format, args...) -#endif -#if NDEBUG & NDEBUG_MERGING -#define MER_PRINTK(format, args...) \ - printk(KERN_DEBUG format , ## args) -#else -#define MER_PRINTK(format, args...) -#endif - -/* conditional macros for NCR5380_print_{,phase,status} */ - -#define NCR_PRINT(mask) \ - ((NDEBUG & (mask)) ? NCR5380_print(instance) : (void)0) - -#define NCR_PRINT_PHASE(mask) \ - ((NDEBUG & (mask)) ? NCR5380_print_phase(instance) : (void)0) - -#define NCR_PRINT_STATUS(mask) \ - ((NDEBUG & (mask)) ? NCR5380_print_status(instance) : (void)0) - - - -#endif /* ndef HOSTS_C */ -#endif /* SUN3_NCR5380_H */ +#endif /* SUN3_SCSI_H */ diff --git a/drivers/scsi/sun3_scsi_vme.c b/drivers/scsi/sun3_scsi_vme.c index a3dd55d1d2fd..1eeece6e2040 100644 --- a/drivers/scsi/sun3_scsi_vme.c +++ b/drivers/scsi/sun3_scsi_vme.c @@ -1,589 +1,3 @@ - /* - * Sun3 SCSI stuff by Erik Verbruggen (erik@bigmama.xtdnet.nl) - * - * Sun3 DMA routines added by Sam Creasey (sammy@sammy.net) - * - * VME support added by Sam Creasey - * - * Adapted from sun3_scsi.c -- see there for other headers - * - * TODO: modify this driver to support multiple Sun3 SCSI VME boards - * - */ - -#define AUTOSENSE - -#include <linux/types.h> -#include <linux/stddef.h> -#include <linux/ctype.h> -#include <linux/delay.h> - -#include <linux/module.h> -#include <linux/signal.h> -#include <linux/ioport.h> -#include <linux/init.h> -#include <linux/blkdev.h> - -#include <asm/io.h> - -#include <asm/sun3ints.h> -#include <asm/dvma.h> -#include <asm/idprom.h> -#include <asm/machines.h> - #define SUN3_SCSI_VME -#undef SUN3_SCSI_DEBUG - -/* dma on! */ -#define REAL_DMA - -#define NDEBUG 0 - -#define NDEBUG_ABORT 0x00100000 -#define NDEBUG_TAGS 0x00200000 -#define NDEBUG_MERGING 0x00400000 - -#include "scsi.h" -#include "initio.h" -#include <scsi/scsi_host.h> -#include "sun3_scsi.h" - -extern int sun3_map_test(unsigned long, char *); - -#define USE_WRAPPER -/*#define RESET_BOOT */ -#define DRIVER_SETUP - -/* - * BUG can be used to trigger a strange code-size related hang on 2.1 kernels - */ -#ifdef BUG -#undef RESET_BOOT -#undef DRIVER_SETUP -#endif - -/* #define SUPPORT_TAGS */ - -//#define ENABLE_IRQ() enable_irq( SUN3_VEC_VMESCSI0 ); -#define ENABLE_IRQ() - - -static irqreturn_t scsi_sun3_intr(int irq, void *dummy); -static inline unsigned char sun3scsi_read(int reg); -static inline void sun3scsi_write(int reg, int value); - -static int setup_can_queue = -1; -module_param(setup_can_queue, int, 0); -static int setup_cmd_per_lun = -1; -module_param(setup_cmd_per_lun, int, 0); -static int setup_sg_tablesize = -1; -module_param(setup_sg_tablesize, int, 0); -#ifdef SUPPORT_TAGS -static int setup_use_tagged_queuing = -1; -module_param(setup_use_tagged_queuing, int, 0); -#endif -static int setup_hostid = -1; -module_param(setup_hostid, int, 0); - -static struct scsi_cmnd *sun3_dma_setup_done = NULL; - -#define AFTER_RESET_DELAY (HZ/2) - -/* ms to wait after hitting dma regs */ -#define SUN3_DMA_DELAY 10 - -/* dvma buffer to allocate -- 32k should hopefully be more than sufficient */ -#define SUN3_DVMA_BUFSIZE 0xe000 - -/* minimum number of bytes to do dma on */ -#define SUN3_DMA_MINSIZE 128 - -static volatile unsigned char *sun3_scsi_regp; -static volatile struct sun3_dma_regs *dregs; -#ifdef OLDDMA -static unsigned char *dmabuf = NULL; /* dma memory buffer */ -#endif -static unsigned char *sun3_dma_orig_addr = NULL; -static unsigned long sun3_dma_orig_count = 0; -static int sun3_dma_active = 0; -static unsigned long last_residual = 0; - -/* - * NCR 5380 register access functions - */ - -static inline unsigned char sun3scsi_read(int reg) -{ - return( sun3_scsi_regp[reg] ); -} - -static inline void sun3scsi_write(int reg, int value) -{ - sun3_scsi_regp[reg] = value; -} - -/* - * XXX: status debug - */ -static struct Scsi_Host *default_instance; - -/* - * Function : int sun3scsi_detect(struct scsi_host_template * tpnt) - * - * Purpose : initializes mac NCR5380 driver based on the - * command line / compile time port and irq definitions. - * - * Inputs : tpnt - template for this SCSI adapter. - * - * Returns : 1 if a host adapter was found, 0 if not. - * - */ - -static int __init sun3scsi_detect(struct scsi_host_template * tpnt) -{ - unsigned long ioaddr, irq = 0; - static int called = 0; - struct Scsi_Host *instance; - int i; - unsigned long addrs[3] = { IOBASE_SUN3_VMESCSI, - IOBASE_SUN3_VMESCSI + 0x4000, - 0 }; - unsigned long vecs[3] = { SUN3_VEC_VMESCSI0, - SUN3_VEC_VMESCSI1, - 0 }; - /* check that this machine has an onboard 5380 */ - switch(idprom->id_machtype) { - case SM_SUN3|SM_3_160: - case SM_SUN3|SM_3_260: - break; - - default: - return 0; - } - - if(called) - return 0; - - tpnt->proc_name = "Sun3 5380 VME SCSI"; - - /* setup variables */ - tpnt->can_queue = - (setup_can_queue > 0) ? setup_can_queue : CAN_QUEUE; - tpnt->cmd_per_lun = - (setup_cmd_per_lun > 0) ? setup_cmd_per_lun : CMD_PER_LUN; - tpnt->sg_tablesize = - (setup_sg_tablesize >= 0) ? setup_sg_tablesize : SG_TABLESIZE; - - if (setup_hostid >= 0) - tpnt->this_id = setup_hostid; - else { - /* use 7 as default */ - tpnt->this_id = 7; - } - - ioaddr = 0; - for(i = 0; addrs[i] != 0; i++) { - unsigned char x; - - ioaddr = (unsigned long)sun3_ioremap(addrs[i], PAGE_SIZE, - SUN3_PAGE_TYPE_VME16); - irq = vecs[i]; - sun3_scsi_regp = (unsigned char *)ioaddr; - - dregs = (struct sun3_dma_regs *)(((unsigned char *)ioaddr) + 8); - - if(sun3_map_test((unsigned long)dregs, &x)) { - unsigned short oldcsr; - - oldcsr = dregs->csr; - dregs->csr = 0; - udelay(SUN3_DMA_DELAY); - if(dregs->csr == 0x1400) - break; - - dregs->csr = oldcsr; - } - - iounmap((void *)ioaddr); - ioaddr = 0; - } - - if(!ioaddr) - return 0; - -#ifdef SUPPORT_TAGS - if (setup_use_tagged_queuing < 0) - setup_use_tagged_queuing = USE_TAGGED_QUEUING; -#endif - - instance = scsi_register (tpnt, sizeof(struct NCR5380_hostdata)); - if(instance == NULL) - return 0; - - default_instance = instance; - - instance->io_port = (unsigned long) ioaddr; - instance->irq = irq; - - NCR5380_init(instance, 0); - - instance->n_io_port = 32; - - ((struct NCR5380_hostdata *)instance->hostdata)->ctrl = 0; - - if (request_irq(instance->irq, scsi_sun3_intr, - 0, "Sun3SCSI-5380VME", instance)) { -#ifndef REAL_DMA - printk("scsi%d: IRQ%d not free, interrupts disabled\n", - instance->host_no, instance->irq); - instance->irq = SCSI_IRQ_NONE; -#else - printk("scsi%d: IRQ%d not free, bailing out\n", - instance->host_no, instance->irq); - return 0; -#endif - } - - printk("scsi%d: Sun3 5380 VME at port %lX irq", instance->host_no, instance->io_port); - if (instance->irq == SCSI_IRQ_NONE) - printk ("s disabled"); - else - printk (" %d", instance->irq); - printk(" options CAN_QUEUE=%d CMD_PER_LUN=%d release=%d", - instance->can_queue, instance->cmd_per_lun, - SUN3SCSI_PUBLIC_RELEASE); - printk("\nscsi%d:", instance->host_no); - NCR5380_print_options(instance); - printk("\n"); - - dregs->csr = 0; - udelay(SUN3_DMA_DELAY); - dregs->csr = CSR_SCSI | CSR_FIFO | CSR_INTR; - udelay(SUN3_DMA_DELAY); - dregs->fifo_count = 0; - dregs->fifo_count_hi = 0; - dregs->dma_addr_hi = 0; - dregs->dma_addr_lo = 0; - dregs->dma_count_hi = 0; - dregs->dma_count_lo = 0; - - dregs->ivect = VME_DATA24 | (instance->irq & 0xff); - - called = 1; - -#ifdef RESET_BOOT - sun3_scsi_reset_boot(instance); -#endif - - return 1; -} - -int sun3scsi_release (struct Scsi_Host *shpnt) -{ - if (shpnt->irq != SCSI_IRQ_NONE) - free_irq(shpnt->irq, shpnt); - - iounmap((void *)sun3_scsi_regp); - - NCR5380_exit(shpnt); - return 0; -} - -#ifdef RESET_BOOT -/* - * Our 'bus reset on boot' function - */ - -static void sun3_scsi_reset_boot(struct Scsi_Host *instance) -{ - unsigned long end; - - NCR5380_local_declare(); - NCR5380_setup(instance); - - /* - * Do a SCSI reset to clean up the bus during initialization. No - * messing with the queues, interrupts, or locks necessary here. - */ - - printk( "Sun3 SCSI: resetting the SCSI bus..." ); - - /* switch off SCSI IRQ - catch an interrupt without IRQ bit set else */ -// sun3_disable_irq( IRQ_SUN3_SCSI ); - - /* get in phase */ - NCR5380_write( TARGET_COMMAND_REG, - PHASE_SR_TO_TCR( NCR5380_read(STATUS_REG) )); - - /* assert RST */ - NCR5380_write( INITIATOR_COMMAND_REG, ICR_BASE | ICR_ASSERT_RST ); - - /* The min. reset hold time is 25us, so 40us should be enough */ - udelay( 50 ); - - /* reset RST and interrupt */ - NCR5380_write( INITIATOR_COMMAND_REG, ICR_BASE ); - NCR5380_read( RESET_PARITY_INTERRUPT_REG ); - - for( end = jiffies + AFTER_RESET_DELAY; time_before(jiffies, end); ) - barrier(); - - /* switch on SCSI IRQ again */ -// sun3_enable_irq( IRQ_SUN3_SCSI ); - - printk( " done\n" ); -} -#endif - -static const char * sun3scsi_info (struct Scsi_Host *spnt) { - return ""; -} - -// safe bits for the CSR -#define CSR_GOOD 0x060f - -static irqreturn_t scsi_sun3_intr(int irq, void *dummy) -{ - unsigned short csr = dregs->csr; - int handled = 0; - - dregs->csr &= ~CSR_DMA_ENABLE; - - -#ifdef SUN3_SCSI_DEBUG - printk("scsi_intr csr %x\n", csr); -#endif - - if(csr & ~CSR_GOOD) { - if(csr & CSR_DMA_BUSERR) { - printk("scsi%d: bus error in dma\n", default_instance->host_no); -#ifdef SUN3_SCSI_DEBUG - printk("scsi: residual %x count %x addr %p dmaaddr %x\n", - dregs->fifo_count, - dregs->dma_count_lo | (dregs->dma_count_hi << 16), - sun3_dma_orig_addr, - dregs->dma_addr_lo | (dregs->dma_addr_hi << 16)); -#endif - } - - if(csr & CSR_DMA_CONFLICT) { - printk("scsi%d: dma conflict\n", default_instance->host_no); - } - handled = 1; - } - - if(csr & (CSR_SDB_INT | CSR_DMA_INT)) { - NCR5380_intr(irq, dummy); - handled = 1; - } - - return IRQ_RETVAL(handled); -} - -/* - * Debug stuff - to be called on NMI, or sysrq key. Use at your own risk; - * reentering NCR5380_print_status seems to have ugly side effects - */ - -/* this doesn't seem to get used at all -- sam */ -#if 0 -void sun3_sun3_debug (void) -{ - unsigned long flags; - NCR5380_local_declare(); - - if (default_instance) { - local_irq_save(flags); - NCR5380_print_status(default_instance); - local_irq_restore(flags); - } -} -#endif - - -/* sun3scsi_dma_setup() -- initialize the dma controller for a read/write */ -static unsigned long sun3scsi_dma_setup(void *data, unsigned long count, int write_flag) -{ - void *addr; - - if(sun3_dma_orig_addr != NULL) - dvma_unmap(sun3_dma_orig_addr); - -// addr = sun3_dvma_page((unsigned long)data, (unsigned long)dmabuf); - addr = (void *)dvma_map_vme((unsigned long) data, count); - - sun3_dma_orig_addr = addr; - sun3_dma_orig_count = count; - -#ifdef SUN3_SCSI_DEBUG - printk("scsi: dma_setup addr %p count %x\n", addr, count); -#endif - -// dregs->fifo_count = 0; -#if 0 - /* reset fifo */ - dregs->csr &= ~CSR_FIFO; - dregs->csr |= CSR_FIFO; -#endif - /* set direction */ - if(write_flag) - dregs->csr |= CSR_SEND; - else - dregs->csr &= ~CSR_SEND; - - /* reset fifo */ -// dregs->csr &= ~CSR_FIFO; -// dregs->csr |= CSR_FIFO; - - dregs->csr |= CSR_PACK_ENABLE; - - dregs->dma_addr_hi = ((unsigned long)addr >> 16); - dregs->dma_addr_lo = ((unsigned long)addr & 0xffff); - - dregs->dma_count_hi = 0; - dregs->dma_count_lo = 0; - dregs->fifo_count_hi = 0; - dregs->fifo_count = 0; - -#ifdef SUN3_SCSI_DEBUG - printk("scsi: dma_setup done csr %x\n", dregs->csr); -#endif - return count; - -} - -static inline unsigned long sun3scsi_dma_residual(struct Scsi_Host *instance) -{ - return last_residual; -} - -static inline unsigned long sun3scsi_dma_xfer_len(unsigned long wanted, - struct scsi_cmnd *cmd, - int write_flag) -{ - if (cmd->request->cmd_type == REQ_TYPE_FS) - return wanted; - else - return 0; -} - -static int sun3scsi_dma_start(unsigned long count, char *data) -{ - - unsigned short csr; - - csr = dregs->csr; -#ifdef SUN3_SCSI_DEBUG - printk("scsi: dma_start data %p count %x csr %x fifo %x\n", data, count, csr, dregs->fifo_count); -#endif - - dregs->dma_count_hi = (sun3_dma_orig_count >> 16); - dregs->dma_count_lo = (sun3_dma_orig_count & 0xffff); - - dregs->fifo_count_hi = (sun3_dma_orig_count >> 16); - dregs->fifo_count = (sun3_dma_orig_count & 0xffff); - -// if(!(csr & CSR_DMA_ENABLE)) -// dregs->csr |= CSR_DMA_ENABLE; - - return 0; -} - -/* clean up after our dma is done */ -static int sun3scsi_dma_finish(int write_flag) -{ - unsigned short fifo; - int ret = 0; - - sun3_dma_active = 0; - - dregs->csr &= ~CSR_DMA_ENABLE; - - fifo = dregs->fifo_count; - if(write_flag) { - if((fifo > 0) && (fifo < sun3_dma_orig_count)) - fifo++; - } - - last_residual = fifo; -#ifdef SUN3_SCSI_DEBUG - printk("scsi: residual %x total %x\n", fifo, sun3_dma_orig_count); -#endif - /* empty bytes from the fifo which didn't make it */ - if((!write_flag) && (dregs->csr & CSR_LEFT)) { - unsigned char *vaddr; - -#ifdef SUN3_SCSI_DEBUG - printk("scsi: got left over bytes\n"); -#endif - - vaddr = (unsigned char *)dvma_vmetov(sun3_dma_orig_addr); - - vaddr += (sun3_dma_orig_count - fifo); - vaddr--; - - switch(dregs->csr & CSR_LEFT) { - case CSR_LEFT_3: - *vaddr = (dregs->bpack_lo & 0xff00) >> 8; - vaddr--; - - case CSR_LEFT_2: - *vaddr = (dregs->bpack_hi & 0x00ff); - vaddr--; - - case CSR_LEFT_1: - *vaddr = (dregs->bpack_hi & 0xff00) >> 8; - break; - } - - - } - - dvma_unmap(sun3_dma_orig_addr); - sun3_dma_orig_addr = NULL; - - dregs->dma_addr_hi = 0; - dregs->dma_addr_lo = 0; - dregs->dma_count_hi = 0; - dregs->dma_count_lo = 0; - - dregs->fifo_count = 0; - dregs->fifo_count_hi = 0; - - dregs->csr &= ~CSR_SEND; - -// dregs->csr |= CSR_DMA_ENABLE; - -#if 0 - /* reset fifo */ - dregs->csr &= ~CSR_FIFO; - dregs->csr |= CSR_FIFO; -#endif - sun3_dma_setup_done = NULL; - - return ret; - -} - -#include "sun3_NCR5380.c" - -static struct scsi_host_template driver_template = { - .name = SUN3_SCSI_NAME, - .detect = sun3scsi_detect, - .release = sun3scsi_release, - .info = sun3scsi_info, - .queuecommand = sun3scsi_queue_command, - .eh_abort_handler = sun3scsi_abort, - .eh_bus_reset_handler = sun3scsi_bus_reset, - .can_queue = CAN_QUEUE, - .this_id = 7, - .sg_tablesize = SG_TABLESIZE, - .cmd_per_lun = CMD_PER_LUN, - .use_clustering = DISABLE_CLUSTERING -}; - - -#include "scsi_module.c" - -MODULE_LICENSE("GPL"); +#include "sun3_scsi.c" diff --git a/drivers/scsi/t128.c b/drivers/scsi/t128.c index a4abce9d526e..8cc80931df14 100644 --- a/drivers/scsi/t128.c +++ b/drivers/scsi/t128.c @@ -102,10 +102,6 @@ * 15 9-11 */ -/* - * $Log: t128.c,v $ - */ - #include <linux/signal.h> #include <linux/io.h> #include <linux/blkdev.h> diff --git a/drivers/scsi/t128.h b/drivers/scsi/t128.h index 1df82c28e56d..fd68cecc62af 100644 --- a/drivers/scsi/t128.h +++ b/drivers/scsi/t128.h @@ -34,10 +34,6 @@ * 1+ (800) 334-5454 */ -/* - * $Log: t128.h,v $ - */ - #ifndef T128_H #define T128_H @@ -107,8 +103,6 @@ static int t128_bus_reset(struct scsi_cmnd *); #define CAN_QUEUE 32 #endif -#ifndef HOSTS_C - #define NCR5380_implementation_fields \ void __iomem *base @@ -148,6 +142,5 @@ static int t128_bus_reset(struct scsi_cmnd *); #define T128_IRQS 0xc4a8 -#endif /* else def HOSTS_C */ #endif /* ndef ASM */ #endif /* T128_H */ diff --git a/drivers/scsi/ufs/ufs.h b/drivers/scsi/ufs/ufs.h index 721050090520..f42d1cee652a 100644 --- a/drivers/scsi/ufs/ufs.h +++ b/drivers/scsi/ufs/ufs.h @@ -196,9 +196,9 @@ enum { * @dword_2: UPIU header DW-2 */ struct utp_upiu_header { - u32 dword_0; - u32 dword_1; - u32 dword_2; + __be32 dword_0; + __be32 dword_1; + __be32 dword_2; }; /** @@ -207,7 +207,7 @@ struct utp_upiu_header { * @cdb: Command Descriptor Block CDB DW-4 to DW-7 */ struct utp_upiu_cmd { - u32 exp_data_transfer_len; + __be32 exp_data_transfer_len; u8 cdb[MAX_CDB_SIZE]; }; @@ -228,10 +228,10 @@ struct utp_upiu_query { u8 idn; u8 index; u8 selector; - u16 reserved_osf; - u16 length; - u32 value; - u32 reserved[2]; + __be16 reserved_osf; + __be16 length; + __be32 value; + __be32 reserved[2]; }; /** @@ -256,9 +256,9 @@ struct utp_upiu_req { * @sense_data: Sense data field DW-8 to DW-12 */ struct utp_cmd_rsp { - u32 residual_transfer_count; - u32 reserved[4]; - u16 sense_data_len; + __be32 residual_transfer_count; + __be32 reserved[4]; + __be16 sense_data_len; u8 sense_data[18]; }; @@ -286,10 +286,10 @@ struct utp_upiu_rsp { */ struct utp_upiu_task_req { struct utp_upiu_header header; - u32 input_param1; - u32 input_param2; - u32 input_param3; - u32 reserved[2]; + __be32 input_param1; + __be32 input_param2; + __be32 input_param3; + __be32 reserved[2]; }; /** @@ -301,9 +301,9 @@ struct utp_upiu_task_req { */ struct utp_upiu_task_rsp { struct utp_upiu_header header; - u32 output_param1; - u32 output_param2; - u32 reserved[3]; + __be32 output_param1; + __be32 output_param2; + __be32 reserved[3]; }; /** diff --git a/drivers/scsi/ufs/ufshcd.c b/drivers/scsi/ufs/ufshcd.c index 04884d663e4e..0c2877251251 100644 --- a/drivers/scsi/ufs/ufshcd.c +++ b/drivers/scsi/ufs/ufshcd.c @@ -55,6 +55,9 @@ /* Query request timeout */ #define QUERY_REQ_TIMEOUT 30 /* msec */ +/* Task management command timeout */ +#define TM_CMD_TIMEOUT 100 /* msecs */ + /* Expose the flag value from utp_upiu_query.value */ #define MASK_QUERY_UPIU_FLAG_LOC 0xFF @@ -71,9 +74,22 @@ enum { /* UFSHCD states */ enum { - UFSHCD_STATE_OPERATIONAL, UFSHCD_STATE_RESET, UFSHCD_STATE_ERROR, + UFSHCD_STATE_OPERATIONAL, +}; + +/* UFSHCD error handling flags */ +enum { + UFSHCD_EH_IN_PROGRESS = (1 << 0), +}; + +/* UFSHCD UIC layer error flags */ +enum { + UFSHCD_UIC_DL_PA_INIT_ERROR = (1 << 0), /* Data link layer error */ + UFSHCD_UIC_NL_ERROR = (1 << 1), /* Network layer error */ + UFSHCD_UIC_TL_ERROR = (1 << 2), /* Transport Layer error */ + UFSHCD_UIC_DME_ERROR = (1 << 3), /* DME error */ }; /* Interrupt configuration options */ @@ -83,6 +99,18 @@ enum { UFSHCD_INT_CLEAR, }; +#define ufshcd_set_eh_in_progress(h) \ + (h->eh_flags |= UFSHCD_EH_IN_PROGRESS) +#define ufshcd_eh_in_progress(h) \ + (h->eh_flags & UFSHCD_EH_IN_PROGRESS) +#define ufshcd_clear_eh_in_progress(h) \ + (h->eh_flags &= ~UFSHCD_EH_IN_PROGRESS) + +static void ufshcd_tmc_handler(struct ufs_hba *hba); +static void ufshcd_async_scan(void *data, async_cookie_t cookie); +static int ufshcd_reset_and_restore(struct ufs_hba *hba); +static int ufshcd_clear_tm_cmd(struct ufs_hba *hba, int tag); + /* * ufshcd_wait_for_register - wait for register value to change * @hba - per-adapter interface @@ -163,7 +191,7 @@ static inline int ufshcd_is_device_present(u32 reg_hcs) */ static inline int ufshcd_get_tr_ocs(struct ufshcd_lrb *lrbp) { - return lrbp->utr_descriptor_ptr->header.dword_2 & MASK_OCS; + return le32_to_cpu(lrbp->utr_descriptor_ptr->header.dword_2) & MASK_OCS; } /** @@ -176,19 +204,41 @@ static inline int ufshcd_get_tr_ocs(struct ufshcd_lrb *lrbp) static inline int ufshcd_get_tmr_ocs(struct utp_task_req_desc *task_req_descp) { - return task_req_descp->header.dword_2 & MASK_OCS; + return le32_to_cpu(task_req_descp->header.dword_2) & MASK_OCS; } /** * ufshcd_get_tm_free_slot - get a free slot for task management request * @hba: per adapter instance + * @free_slot: pointer to variable with available slot value * - * Returns maximum number of task management request slots in case of - * task management queue full or returns the free slot number + * Get a free tag and lock it until ufshcd_put_tm_slot() is called. + * Returns 0 if free slot is not available, else return 1 with tag value + * in @free_slot. */ -static inline int ufshcd_get_tm_free_slot(struct ufs_hba *hba) +static bool ufshcd_get_tm_free_slot(struct ufs_hba *hba, int *free_slot) { - return find_first_zero_bit(&hba->outstanding_tasks, hba->nutmrs); + int tag; + bool ret = false; + + if (!free_slot) + goto out; + + do { + tag = find_first_zero_bit(&hba->tm_slots_in_use, hba->nutmrs); + if (tag >= hba->nutmrs) + goto out; + } while (test_and_set_bit_lock(tag, &hba->tm_slots_in_use)); + + *free_slot = tag; + ret = true; +out: + return ret; +} + +static inline void ufshcd_put_tm_slot(struct ufs_hba *hba, int slot) +{ + clear_bit_unlock(slot, &hba->tm_slots_in_use); } /** @@ -390,26 +440,6 @@ static inline void ufshcd_copy_sense_data(struct ufshcd_lrb *lrbp) } /** - * ufshcd_query_to_cpu() - formats the buffer to native cpu endian - * @response: upiu query response to convert - */ -static inline void ufshcd_query_to_cpu(struct utp_upiu_query *response) -{ - response->length = be16_to_cpu(response->length); - response->value = be32_to_cpu(response->value); -} - -/** - * ufshcd_query_to_be() - formats the buffer to big endian - * @request: upiu query request to convert - */ -static inline void ufshcd_query_to_be(struct utp_upiu_query *request) -{ - request->length = cpu_to_be16(request->length); - request->value = cpu_to_be32(request->value); -} - -/** * ufshcd_copy_query_response() - Copy the Query Response and the data * descriptor * @hba: per adapter instance @@ -425,7 +455,6 @@ void ufshcd_copy_query_response(struct ufs_hba *hba, struct ufshcd_lrb *lrbp) UPIU_RSP_CODE_OFFSET; memcpy(&query_res->upiu_res, &lrbp->ucd_rsp_ptr->qr, QUERY_OSF_SIZE); - ufshcd_query_to_cpu(&query_res->upiu_res); /* Get the descriptor */ @@ -749,7 +778,7 @@ static void ufshcd_prepare_utp_query_req_upiu(struct ufs_hba *hba, { struct utp_upiu_req *ucd_req_ptr = lrbp->ucd_req_ptr; struct ufs_query *query = &hba->dev_cmd.query; - u16 len = query->request.upiu_req.length; + u16 len = be16_to_cpu(query->request.upiu_req.length); u8 *descp = (u8 *)lrbp->ucd_req_ptr + GENERAL_UPIU_REQUEST_SIZE; /* Query request header */ @@ -766,7 +795,6 @@ static void ufshcd_prepare_utp_query_req_upiu(struct ufs_hba *hba, /* Copy the Query Request buffer as is */ memcpy(&ucd_req_ptr->qr, &query->request.upiu_req, QUERY_OSF_SIZE); - ufshcd_query_to_be(&ucd_req_ptr->qr); /* Copy the Descriptor */ if ((len > 0) && (query->request.upiu_req.opcode == @@ -853,10 +881,25 @@ static int ufshcd_queuecommand(struct Scsi_Host *host, struct scsi_cmnd *cmd) tag = cmd->request->tag; - if (hba->ufshcd_state != UFSHCD_STATE_OPERATIONAL) { + spin_lock_irqsave(hba->host->host_lock, flags); + switch (hba->ufshcd_state) { + case UFSHCD_STATE_OPERATIONAL: + break; + case UFSHCD_STATE_RESET: err = SCSI_MLQUEUE_HOST_BUSY; - goto out; + goto out_unlock; + case UFSHCD_STATE_ERROR: + set_host_byte(cmd, DID_ERROR); + cmd->scsi_done(cmd); + goto out_unlock; + default: + dev_WARN_ONCE(hba->dev, 1, "%s: invalid state %d\n", + __func__, hba->ufshcd_state); + set_host_byte(cmd, DID_BAD_TARGET); + cmd->scsi_done(cmd); + goto out_unlock; } + spin_unlock_irqrestore(hba->host->host_lock, flags); /* acquire the tag to make sure device cmds don't use it */ if (test_and_set_bit_lock(tag, &hba->lrb_in_use)) { @@ -893,6 +936,7 @@ static int ufshcd_queuecommand(struct Scsi_Host *host, struct scsi_cmnd *cmd) /* issue command to the controller */ spin_lock_irqsave(hba->host->host_lock, flags); ufshcd_send_command(hba, tag); +out_unlock: spin_unlock_irqrestore(hba->host->host_lock, flags); out: return err; @@ -1151,7 +1195,7 @@ static int ufshcd_query_flag(struct ufs_hba *hba, enum query_opcode opcode, } if (flag_res) - *flag_res = (response->upiu_res.value & + *flag_res = (be32_to_cpu(response->upiu_res.value) & MASK_QUERY_UPIU_FLAG_LOC) & 0x1; out_unlock: @@ -1170,7 +1214,7 @@ out_unlock: * * Returns 0 for success, non-zero in case of failure */ -int ufshcd_query_attr(struct ufs_hba *hba, enum query_opcode opcode, +static int ufshcd_query_attr(struct ufs_hba *hba, enum query_opcode opcode, enum attr_idn idn, u8 index, u8 selector, u32 *attr_val) { struct ufs_query_req *request; @@ -1195,7 +1239,7 @@ int ufshcd_query_attr(struct ufs_hba *hba, enum query_opcode opcode, switch (opcode) { case UPIU_QUERY_OPCODE_WRITE_ATTR: request->query_func = UPIU_QUERY_FUNC_STANDARD_WRITE_REQUEST; - request->upiu_req.value = *attr_val; + request->upiu_req.value = cpu_to_be32(*attr_val); break; case UPIU_QUERY_OPCODE_READ_ATTR: request->query_func = UPIU_QUERY_FUNC_STANDARD_READ_REQUEST; @@ -1222,7 +1266,7 @@ int ufshcd_query_attr(struct ufs_hba *hba, enum query_opcode opcode, goto out_unlock; } - *attr_val = response->upiu_res.value; + *attr_val = be32_to_cpu(response->upiu_res.value); out_unlock: mutex_unlock(&hba->dev_cmd.lock); @@ -1481,7 +1525,7 @@ EXPORT_SYMBOL_GPL(ufshcd_dme_get_attr); * * Returns 0 on success, non-zero value on failure */ -int ufshcd_uic_change_pwr_mode(struct ufs_hba *hba, u8 mode) +static int ufshcd_uic_change_pwr_mode(struct ufs_hba *hba, u8 mode) { struct uic_command uic_cmd = {0}; struct completion pwr_done; @@ -1701,11 +1745,6 @@ static int ufshcd_make_hba_operational(struct ufs_hba *hba) goto out; } - if (hba->ufshcd_state == UFSHCD_STATE_RESET) - scsi_unblock_requests(hba->host); - - hba->ufshcd_state = UFSHCD_STATE_OPERATIONAL; - out: return err; } @@ -1831,66 +1870,6 @@ static int ufshcd_verify_dev_init(struct ufs_hba *hba) } /** - * ufshcd_do_reset - reset the host controller - * @hba: per adapter instance - * - * Returns SUCCESS/FAILED - */ -static int ufshcd_do_reset(struct ufs_hba *hba) -{ - struct ufshcd_lrb *lrbp; - unsigned long flags; - int tag; - - /* block commands from midlayer */ - scsi_block_requests(hba->host); - - spin_lock_irqsave(hba->host->host_lock, flags); - hba->ufshcd_state = UFSHCD_STATE_RESET; - - /* send controller to reset state */ - ufshcd_hba_stop(hba); - spin_unlock_irqrestore(hba->host->host_lock, flags); - - /* abort outstanding commands */ - for (tag = 0; tag < hba->nutrs; tag++) { - if (test_bit(tag, &hba->outstanding_reqs)) { - lrbp = &hba->lrb[tag]; - if (lrbp->cmd) { - scsi_dma_unmap(lrbp->cmd); - lrbp->cmd->result = DID_RESET << 16; - lrbp->cmd->scsi_done(lrbp->cmd); - lrbp->cmd = NULL; - clear_bit_unlock(tag, &hba->lrb_in_use); - } - } - } - - /* complete device management command */ - if (hba->dev_cmd.complete) - complete(hba->dev_cmd.complete); - - /* clear outstanding request/task bit maps */ - hba->outstanding_reqs = 0; - hba->outstanding_tasks = 0; - - /* Host controller enable */ - if (ufshcd_hba_enable(hba)) { - dev_err(hba->dev, - "Reset: Controller initialization failed\n"); - return FAILED; - } - - if (ufshcd_link_startup(hba)) { - dev_err(hba->dev, - "Reset: Link start-up failed\n"); - return FAILED; - } - - return SUCCESS; -} - -/** * ufshcd_slave_alloc - handle initial SCSI device configurations * @sdev: pointer to SCSI device * @@ -1907,6 +1886,9 @@ static int ufshcd_slave_alloc(struct scsi_device *sdev) sdev->use_10_for_ms = 1; scsi_set_tag_type(sdev, MSG_SIMPLE_TAG); + /* allow SCSI layer to restart the device in case of errors */ + sdev->allow_restart = 1; + /* * Inform SCSI Midlayer that the LUN queue depth is same as the * controller queue depth. If a LUN queue depth is less than the @@ -1934,10 +1916,11 @@ static void ufshcd_slave_destroy(struct scsi_device *sdev) * ufshcd_task_req_compl - handle task management request completion * @hba: per adapter instance * @index: index of the completed request + * @resp: task management service response * - * Returns SUCCESS/FAILED + * Returns non-zero value on error, zero on success */ -static int ufshcd_task_req_compl(struct ufs_hba *hba, u32 index) +static int ufshcd_task_req_compl(struct ufs_hba *hba, u32 index, u8 *resp) { struct utp_task_req_desc *task_req_descp; struct utp_upiu_task_rsp *task_rsp_upiup; @@ -1958,19 +1941,15 @@ static int ufshcd_task_req_compl(struct ufs_hba *hba, u32 index) task_req_descp[index].task_rsp_upiu; task_result = be32_to_cpu(task_rsp_upiup->header.dword_1); task_result = ((task_result & MASK_TASK_RESPONSE) >> 8); - - if (task_result != UPIU_TASK_MANAGEMENT_FUNC_COMPL && - task_result != UPIU_TASK_MANAGEMENT_FUNC_SUCCEEDED) - task_result = FAILED; - else - task_result = SUCCESS; + if (resp) + *resp = (u8)task_result; } else { - task_result = FAILED; - dev_err(hba->dev, - "trc: Invalid ocs = %x\n", ocs_value); + dev_err(hba->dev, "%s: failed, ocs = 0x%x\n", + __func__, ocs_value); } spin_unlock_irqrestore(hba->host->host_lock, flags); - return task_result; + + return ocs_value; } /** @@ -2105,6 +2084,9 @@ ufshcd_transfer_rsp_status(struct ufs_hba *hba, struct ufshcd_lrb *lrbp) case OCS_ABORTED: result |= DID_ABORT << 16; break; + case OCS_INVALID_COMMAND_STATUS: + result |= DID_REQUEUE << 16; + break; case OCS_INVALID_CMD_TABLE_ATTR: case OCS_INVALID_PRDT_ATTR: case OCS_MISMATCH_DATA_BUF_SIZE: @@ -2422,41 +2404,145 @@ out: } /** - * ufshcd_fatal_err_handler - handle fatal errors - * @hba: per adapter instance + * ufshcd_err_handler - handle UFS errors that require s/w attention + * @work: pointer to work structure */ -static void ufshcd_fatal_err_handler(struct work_struct *work) +static void ufshcd_err_handler(struct work_struct *work) { struct ufs_hba *hba; - hba = container_of(work, struct ufs_hba, feh_workq); + unsigned long flags; + u32 err_xfer = 0; + u32 err_tm = 0; + int err = 0; + int tag; + + hba = container_of(work, struct ufs_hba, eh_work); pm_runtime_get_sync(hba->dev); - /* check if reset is already in progress */ - if (hba->ufshcd_state != UFSHCD_STATE_RESET) - ufshcd_do_reset(hba); + + spin_lock_irqsave(hba->host->host_lock, flags); + if (hba->ufshcd_state == UFSHCD_STATE_RESET) { + spin_unlock_irqrestore(hba->host->host_lock, flags); + goto out; + } + + hba->ufshcd_state = UFSHCD_STATE_RESET; + ufshcd_set_eh_in_progress(hba); + + /* Complete requests that have door-bell cleared by h/w */ + ufshcd_transfer_req_compl(hba); + ufshcd_tmc_handler(hba); + spin_unlock_irqrestore(hba->host->host_lock, flags); + + /* Clear pending transfer requests */ + for_each_set_bit(tag, &hba->outstanding_reqs, hba->nutrs) + if (ufshcd_clear_cmd(hba, tag)) + err_xfer |= 1 << tag; + + /* Clear pending task management requests */ + for_each_set_bit(tag, &hba->outstanding_tasks, hba->nutmrs) + if (ufshcd_clear_tm_cmd(hba, tag)) + err_tm |= 1 << tag; + + /* Complete the requests that are cleared by s/w */ + spin_lock_irqsave(hba->host->host_lock, flags); + ufshcd_transfer_req_compl(hba); + ufshcd_tmc_handler(hba); + spin_unlock_irqrestore(hba->host->host_lock, flags); + + /* Fatal errors need reset */ + if (err_xfer || err_tm || (hba->saved_err & INT_FATAL_ERRORS) || + ((hba->saved_err & UIC_ERROR) && + (hba->saved_uic_err & UFSHCD_UIC_DL_PA_INIT_ERROR))) { + err = ufshcd_reset_and_restore(hba); + if (err) { + dev_err(hba->dev, "%s: reset and restore failed\n", + __func__); + hba->ufshcd_state = UFSHCD_STATE_ERROR; + } + /* + * Inform scsi mid-layer that we did reset and allow to handle + * Unit Attention properly. + */ + scsi_report_bus_reset(hba->host, 0); + hba->saved_err = 0; + hba->saved_uic_err = 0; + } + ufshcd_clear_eh_in_progress(hba); + +out: + scsi_unblock_requests(hba->host); pm_runtime_put_sync(hba->dev); } /** - * ufshcd_err_handler - Check for fatal errors - * @work: pointer to a work queue structure + * ufshcd_update_uic_error - check and set fatal UIC error flags. + * @hba: per-adapter instance */ -static void ufshcd_err_handler(struct ufs_hba *hba) +static void ufshcd_update_uic_error(struct ufs_hba *hba) { u32 reg; + /* PA_INIT_ERROR is fatal and needs UIC reset */ + reg = ufshcd_readl(hba, REG_UIC_ERROR_CODE_DATA_LINK_LAYER); + if (reg & UIC_DATA_LINK_LAYER_ERROR_PA_INIT) + hba->uic_error |= UFSHCD_UIC_DL_PA_INIT_ERROR; + + /* UIC NL/TL/DME errors needs software retry */ + reg = ufshcd_readl(hba, REG_UIC_ERROR_CODE_NETWORK_LAYER); + if (reg) + hba->uic_error |= UFSHCD_UIC_NL_ERROR; + + reg = ufshcd_readl(hba, REG_UIC_ERROR_CODE_TRANSPORT_LAYER); + if (reg) + hba->uic_error |= UFSHCD_UIC_TL_ERROR; + + reg = ufshcd_readl(hba, REG_UIC_ERROR_CODE_DME); + if (reg) + hba->uic_error |= UFSHCD_UIC_DME_ERROR; + + dev_dbg(hba->dev, "%s: UIC error flags = 0x%08x\n", + __func__, hba->uic_error); +} + +/** + * ufshcd_check_errors - Check for errors that need s/w attention + * @hba: per-adapter instance + */ +static void ufshcd_check_errors(struct ufs_hba *hba) +{ + bool queue_eh_work = false; + if (hba->errors & INT_FATAL_ERRORS) - goto fatal_eh; + queue_eh_work = true; if (hba->errors & UIC_ERROR) { - reg = ufshcd_readl(hba, REG_UIC_ERROR_CODE_DATA_LINK_LAYER); - if (reg & UIC_DATA_LINK_LAYER_ERROR_PA_INIT) - goto fatal_eh; + hba->uic_error = 0; + ufshcd_update_uic_error(hba); + if (hba->uic_error) + queue_eh_work = true; } - return; -fatal_eh: - hba->ufshcd_state = UFSHCD_STATE_ERROR; - schedule_work(&hba->feh_workq); + + if (queue_eh_work) { + /* handle fatal errors only when link is functional */ + if (hba->ufshcd_state == UFSHCD_STATE_OPERATIONAL) { + /* block commands from scsi mid-layer */ + scsi_block_requests(hba->host); + + /* transfer error masks to sticky bits */ + hba->saved_err |= hba->errors; + hba->saved_uic_err |= hba->uic_error; + + hba->ufshcd_state = UFSHCD_STATE_ERROR; + schedule_work(&hba->eh_work); + } + } + /* + * if (!queue_eh_work) - + * Other errors are either non-fatal where host recovers + * itself without s/w intervention or errors that will be + * handled by the SCSI core layer. + */ } /** @@ -2469,7 +2555,7 @@ static void ufshcd_tmc_handler(struct ufs_hba *hba) tm_doorbell = ufshcd_readl(hba, REG_UTP_TASK_REQ_DOOR_BELL); hba->tm_condition = tm_doorbell ^ hba->outstanding_tasks; - wake_up_interruptible(&hba->ufshcd_tm_wait_queue); + wake_up(&hba->tm_wq); } /** @@ -2481,7 +2567,7 @@ static void ufshcd_sl_intr(struct ufs_hba *hba, u32 intr_status) { hba->errors = UFSHCD_ERROR_MASK & intr_status; if (hba->errors) - ufshcd_err_handler(hba); + ufshcd_check_errors(hba); if (intr_status & UFSHCD_UIC_MASK) ufshcd_uic_cmd_compl(hba, intr_status); @@ -2519,38 +2605,58 @@ static irqreturn_t ufshcd_intr(int irq, void *__hba) return retval; } +static int ufshcd_clear_tm_cmd(struct ufs_hba *hba, int tag) +{ + int err = 0; + u32 mask = 1 << tag; + unsigned long flags; + + if (!test_bit(tag, &hba->outstanding_tasks)) + goto out; + + spin_lock_irqsave(hba->host->host_lock, flags); + ufshcd_writel(hba, ~(1 << tag), REG_UTP_TASK_REQ_LIST_CLEAR); + spin_unlock_irqrestore(hba->host->host_lock, flags); + + /* poll for max. 1 sec to clear door bell register by h/w */ + err = ufshcd_wait_for_register(hba, + REG_UTP_TASK_REQ_DOOR_BELL, + mask, 0, 1000, 1000); +out: + return err; +} + /** * ufshcd_issue_tm_cmd - issues task management commands to controller * @hba: per adapter instance - * @lrbp: pointer to local reference block + * @lun_id: LUN ID to which TM command is sent + * @task_id: task ID to which the TM command is applicable + * @tm_function: task management function opcode + * @tm_response: task management service response return value * - * Returns SUCCESS/FAILED + * Returns non-zero value on error, zero on success. */ -static int -ufshcd_issue_tm_cmd(struct ufs_hba *hba, - struct ufshcd_lrb *lrbp, - u8 tm_function) +static int ufshcd_issue_tm_cmd(struct ufs_hba *hba, int lun_id, int task_id, + u8 tm_function, u8 *tm_response) { struct utp_task_req_desc *task_req_descp; struct utp_upiu_task_req *task_req_upiup; struct Scsi_Host *host; unsigned long flags; - int free_slot = 0; + int free_slot; int err; + int task_tag; host = hba->host; - spin_lock_irqsave(host->host_lock, flags); - - /* If task management queue is full */ - free_slot = ufshcd_get_tm_free_slot(hba); - if (free_slot >= hba->nutmrs) { - spin_unlock_irqrestore(host->host_lock, flags); - dev_err(hba->dev, "Task management queue full\n"); - err = FAILED; - goto out; - } + /* + * Get free slot, sleep if slots are unavailable. + * Even though we use wait_event() which sleeps indefinitely, + * the maximum wait time is bounded by %TM_CMD_TIMEOUT. + */ + wait_event(hba->tm_tag_wq, ufshcd_get_tm_free_slot(hba, &free_slot)); + spin_lock_irqsave(host->host_lock, flags); task_req_descp = hba->utmrdl_base_addr; task_req_descp += free_slot; @@ -2562,18 +2668,15 @@ ufshcd_issue_tm_cmd(struct ufs_hba *hba, /* Configure task request UPIU */ task_req_upiup = (struct utp_upiu_task_req *) task_req_descp->task_req_upiu; + task_tag = hba->nutrs + free_slot; task_req_upiup->header.dword_0 = UPIU_HEADER_DWORD(UPIU_TRANSACTION_TASK_REQ, 0, - lrbp->lun, lrbp->task_tag); + lun_id, task_tag); task_req_upiup->header.dword_1 = UPIU_HEADER_DWORD(0, tm_function, 0, 0); - task_req_upiup->input_param1 = lrbp->lun; - task_req_upiup->input_param1 = - cpu_to_be32(task_req_upiup->input_param1); - task_req_upiup->input_param2 = lrbp->task_tag; - task_req_upiup->input_param2 = - cpu_to_be32(task_req_upiup->input_param2); + task_req_upiup->input_param1 = cpu_to_be32(lun_id); + task_req_upiup->input_param2 = cpu_to_be32(task_id); /* send command to the controller */ __set_bit(free_slot, &hba->outstanding_tasks); @@ -2582,91 +2685,88 @@ ufshcd_issue_tm_cmd(struct ufs_hba *hba, spin_unlock_irqrestore(host->host_lock, flags); /* wait until the task management command is completed */ - err = - wait_event_interruptible_timeout(hba->ufshcd_tm_wait_queue, - (test_bit(free_slot, - &hba->tm_condition) != 0), - 60 * HZ); + err = wait_event_timeout(hba->tm_wq, + test_bit(free_slot, &hba->tm_condition), + msecs_to_jiffies(TM_CMD_TIMEOUT)); if (!err) { - dev_err(hba->dev, - "Task management command timed-out\n"); - err = FAILED; - goto out; + dev_err(hba->dev, "%s: task management cmd 0x%.2x timed-out\n", + __func__, tm_function); + if (ufshcd_clear_tm_cmd(hba, free_slot)) + dev_WARN(hba->dev, "%s: unable clear tm cmd (slot %d) after timeout\n", + __func__, free_slot); + err = -ETIMEDOUT; + } else { + err = ufshcd_task_req_compl(hba, free_slot, tm_response); } + clear_bit(free_slot, &hba->tm_condition); - err = ufshcd_task_req_compl(hba, free_slot); -out: + ufshcd_put_tm_slot(hba, free_slot); + wake_up(&hba->tm_tag_wq); + return err; } /** - * ufshcd_device_reset - reset device and abort all the pending commands + * ufshcd_eh_device_reset_handler - device reset handler registered to + * scsi layer. * @cmd: SCSI command pointer * * Returns SUCCESS/FAILED */ -static int ufshcd_device_reset(struct scsi_cmnd *cmd) +static int ufshcd_eh_device_reset_handler(struct scsi_cmnd *cmd) { struct Scsi_Host *host; struct ufs_hba *hba; unsigned int tag; u32 pos; int err; + u8 resp = 0xF; + struct ufshcd_lrb *lrbp; + unsigned long flags; host = cmd->device->host; hba = shost_priv(host); tag = cmd->request->tag; - err = ufshcd_issue_tm_cmd(hba, &hba->lrb[tag], UFS_LOGICAL_RESET); - if (err == FAILED) + lrbp = &hba->lrb[tag]; + err = ufshcd_issue_tm_cmd(hba, lrbp->lun, 0, UFS_LOGICAL_RESET, &resp); + if (err || resp != UPIU_TASK_MANAGEMENT_FUNC_COMPL) { + if (!err) + err = resp; goto out; + } - for (pos = 0; pos < hba->nutrs; pos++) { - if (test_bit(pos, &hba->outstanding_reqs) && - (hba->lrb[tag].lun == hba->lrb[pos].lun)) { - - /* clear the respective UTRLCLR register bit */ - ufshcd_utrl_clear(hba, pos); - - clear_bit(pos, &hba->outstanding_reqs); - - if (hba->lrb[pos].cmd) { - scsi_dma_unmap(hba->lrb[pos].cmd); - hba->lrb[pos].cmd->result = - DID_ABORT << 16; - hba->lrb[pos].cmd->scsi_done(cmd); - hba->lrb[pos].cmd = NULL; - clear_bit_unlock(pos, &hba->lrb_in_use); - wake_up(&hba->dev_cmd.tag_wq); - } + /* clear the commands that were pending for corresponding LUN */ + for_each_set_bit(pos, &hba->outstanding_reqs, hba->nutrs) { + if (hba->lrb[pos].lun == lrbp->lun) { + err = ufshcd_clear_cmd(hba, pos); + if (err) + break; } - } /* end of for */ + } + spin_lock_irqsave(host->host_lock, flags); + ufshcd_transfer_req_compl(hba); + spin_unlock_irqrestore(host->host_lock, flags); out: + if (!err) { + err = SUCCESS; + } else { + dev_err(hba->dev, "%s: failed with err %d\n", __func__, err); + err = FAILED; + } return err; } /** - * ufshcd_host_reset - Main reset function registered with scsi layer - * @cmd: SCSI command pointer - * - * Returns SUCCESS/FAILED - */ -static int ufshcd_host_reset(struct scsi_cmnd *cmd) -{ - struct ufs_hba *hba; - - hba = shost_priv(cmd->device->host); - - if (hba->ufshcd_state == UFSHCD_STATE_RESET) - return SUCCESS; - - return ufshcd_do_reset(hba); -} - -/** * ufshcd_abort - abort a specific command * @cmd: SCSI command pointer * + * Abort the pending command in device by sending UFS_ABORT_TASK task management + * command, and in host controller by clearing the door-bell register. There can + * be race between controller sending the command to the device while abort is + * issued. To avoid that, first issue UFS_QUERY_TASK to check if the command is + * really issued and then try to abort it. + * * Returns SUCCESS/FAILED */ static int ufshcd_abort(struct scsi_cmnd *cmd) @@ -2675,33 +2775,68 @@ static int ufshcd_abort(struct scsi_cmnd *cmd) struct ufs_hba *hba; unsigned long flags; unsigned int tag; - int err; + int err = 0; + int poll_cnt; + u8 resp = 0xF; + struct ufshcd_lrb *lrbp; host = cmd->device->host; hba = shost_priv(host); tag = cmd->request->tag; - spin_lock_irqsave(host->host_lock, flags); + /* If command is already aborted/completed, return SUCCESS */ + if (!(test_bit(tag, &hba->outstanding_reqs))) + goto out; - /* check if command is still pending */ - if (!(test_bit(tag, &hba->outstanding_reqs))) { - err = FAILED; - spin_unlock_irqrestore(host->host_lock, flags); + lrbp = &hba->lrb[tag]; + for (poll_cnt = 100; poll_cnt; poll_cnt--) { + err = ufshcd_issue_tm_cmd(hba, lrbp->lun, lrbp->task_tag, + UFS_QUERY_TASK, &resp); + if (!err && resp == UPIU_TASK_MANAGEMENT_FUNC_SUCCEEDED) { + /* cmd pending in the device */ + break; + } else if (!err && resp == UPIU_TASK_MANAGEMENT_FUNC_COMPL) { + u32 reg; + + /* + * cmd not pending in the device, check if it is + * in transition. + */ + reg = ufshcd_readl(hba, REG_UTP_TRANSFER_REQ_DOOR_BELL); + if (reg & (1 << tag)) { + /* sleep for max. 200us to stabilize */ + usleep_range(100, 200); + continue; + } + /* command completed already */ + goto out; + } else { + if (!err) + err = resp; /* service response error */ + goto out; + } + } + + if (!poll_cnt) { + err = -EBUSY; goto out; } - spin_unlock_irqrestore(host->host_lock, flags); - err = ufshcd_issue_tm_cmd(hba, &hba->lrb[tag], UFS_ABORT_TASK); - if (err == FAILED) + err = ufshcd_issue_tm_cmd(hba, lrbp->lun, lrbp->task_tag, + UFS_ABORT_TASK, &resp); + if (err || resp != UPIU_TASK_MANAGEMENT_FUNC_COMPL) { + if (!err) + err = resp; /* service response error */ + goto out; + } + + err = ufshcd_clear_cmd(hba, tag); + if (err) goto out; scsi_dma_unmap(cmd); spin_lock_irqsave(host->host_lock, flags); - - /* clear the respective UTRLCLR register bit */ - ufshcd_utrl_clear(hba, tag); - __clear_bit(tag, &hba->outstanding_reqs); hba->lrb[tag].cmd = NULL; spin_unlock_irqrestore(host->host_lock, flags); @@ -2709,6 +2844,129 @@ static int ufshcd_abort(struct scsi_cmnd *cmd) clear_bit_unlock(tag, &hba->lrb_in_use); wake_up(&hba->dev_cmd.tag_wq); out: + if (!err) { + err = SUCCESS; + } else { + dev_err(hba->dev, "%s: failed with err %d\n", __func__, err); + err = FAILED; + } + + return err; +} + +/** + * ufshcd_host_reset_and_restore - reset and restore host controller + * @hba: per-adapter instance + * + * Note that host controller reset may issue DME_RESET to + * local and remote (device) Uni-Pro stack and the attributes + * are reset to default state. + * + * Returns zero on success, non-zero on failure + */ +static int ufshcd_host_reset_and_restore(struct ufs_hba *hba) +{ + int err; + async_cookie_t cookie; + unsigned long flags; + + /* Reset the host controller */ + spin_lock_irqsave(hba->host->host_lock, flags); + ufshcd_hba_stop(hba); + spin_unlock_irqrestore(hba->host->host_lock, flags); + + err = ufshcd_hba_enable(hba); + if (err) + goto out; + + /* Establish the link again and restore the device */ + cookie = async_schedule(ufshcd_async_scan, hba); + /* wait for async scan to be completed */ + async_synchronize_cookie(++cookie); + if (hba->ufshcd_state != UFSHCD_STATE_OPERATIONAL) + err = -EIO; +out: + if (err) + dev_err(hba->dev, "%s: Host init failed %d\n", __func__, err); + + return err; +} + +/** + * ufshcd_reset_and_restore - reset and re-initialize host/device + * @hba: per-adapter instance + * + * Reset and recover device, host and re-establish link. This + * is helpful to recover the communication in fatal error conditions. + * + * Returns zero on success, non-zero on failure + */ +static int ufshcd_reset_and_restore(struct ufs_hba *hba) +{ + int err = 0; + unsigned long flags; + + err = ufshcd_host_reset_and_restore(hba); + + /* + * After reset the door-bell might be cleared, complete + * outstanding requests in s/w here. + */ + spin_lock_irqsave(hba->host->host_lock, flags); + ufshcd_transfer_req_compl(hba); + ufshcd_tmc_handler(hba); + spin_unlock_irqrestore(hba->host->host_lock, flags); + + return err; +} + +/** + * ufshcd_eh_host_reset_handler - host reset handler registered to scsi layer + * @cmd - SCSI command pointer + * + * Returns SUCCESS/FAILED + */ +static int ufshcd_eh_host_reset_handler(struct scsi_cmnd *cmd) +{ + int err; + unsigned long flags; + struct ufs_hba *hba; + + hba = shost_priv(cmd->device->host); + + /* + * Check if there is any race with fatal error handling. + * If so, wait for it to complete. Even though fatal error + * handling does reset and restore in some cases, don't assume + * anything out of it. We are just avoiding race here. + */ + do { + spin_lock_irqsave(hba->host->host_lock, flags); + if (!(work_pending(&hba->eh_work) || + hba->ufshcd_state == UFSHCD_STATE_RESET)) + break; + spin_unlock_irqrestore(hba->host->host_lock, flags); + dev_dbg(hba->dev, "%s: reset in progress\n", __func__); + flush_work(&hba->eh_work); + } while (1); + + hba->ufshcd_state = UFSHCD_STATE_RESET; + ufshcd_set_eh_in_progress(hba); + spin_unlock_irqrestore(hba->host->host_lock, flags); + + err = ufshcd_reset_and_restore(hba); + + spin_lock_irqsave(hba->host->host_lock, flags); + if (!err) { + err = SUCCESS; + hba->ufshcd_state = UFSHCD_STATE_OPERATIONAL; + } else { + err = FAILED; + hba->ufshcd_state = UFSHCD_STATE_ERROR; + } + ufshcd_clear_eh_in_progress(hba); + spin_unlock_irqrestore(hba->host->host_lock, flags); + return err; } @@ -2737,8 +2995,13 @@ static void ufshcd_async_scan(void *data, async_cookie_t cookie) goto out; ufshcd_force_reset_auto_bkops(hba); - scsi_scan_host(hba->host); - pm_runtime_put_sync(hba->dev); + hba->ufshcd_state = UFSHCD_STATE_OPERATIONAL; + + /* If we are in error handling context no need to scan the host */ + if (!ufshcd_eh_in_progress(hba)) { + scsi_scan_host(hba->host); + pm_runtime_put_sync(hba->dev); + } out: return; } @@ -2751,8 +3014,8 @@ static struct scsi_host_template ufshcd_driver_template = { .slave_alloc = ufshcd_slave_alloc, .slave_destroy = ufshcd_slave_destroy, .eh_abort_handler = ufshcd_abort, - .eh_device_reset_handler = ufshcd_device_reset, - .eh_host_reset_handler = ufshcd_host_reset, + .eh_device_reset_handler = ufshcd_eh_device_reset_handler, + .eh_host_reset_handler = ufshcd_eh_host_reset_handler, .this_id = -1, .sg_tablesize = SG_ALL, .cmd_per_lun = UFSHCD_CMD_PER_LUN, @@ -2916,10 +3179,11 @@ int ufshcd_init(struct device *dev, struct ufs_hba **hba_handle, host->max_cmd_len = MAX_CDB_SIZE; /* Initailize wait queue for task management */ - init_waitqueue_head(&hba->ufshcd_tm_wait_queue); + init_waitqueue_head(&hba->tm_wq); + init_waitqueue_head(&hba->tm_tag_wq); /* Initialize work queues */ - INIT_WORK(&hba->feh_workq, ufshcd_fatal_err_handler); + INIT_WORK(&hba->eh_work, ufshcd_err_handler); INIT_WORK(&hba->eeh_work, ufshcd_exception_event_handler); /* Initialize UIC command mutex */ diff --git a/drivers/scsi/ufs/ufshcd.h b/drivers/scsi/ufs/ufshcd.h index 577679a2d189..acf318e338ed 100644 --- a/drivers/scsi/ufs/ufshcd.h +++ b/drivers/scsi/ufs/ufshcd.h @@ -174,15 +174,21 @@ struct ufs_dev_cmd { * @irq: Irq number of the controller * @active_uic_cmd: handle of active UIC command * @uic_cmd_mutex: mutex for uic command - * @ufshcd_tm_wait_queue: wait queue for task management + * @tm_wq: wait queue for task management + * @tm_tag_wq: wait queue for free task management slots + * @tm_slots_in_use: bit map of task management request slots in use * @pwr_done: completion for power mode change * @tm_condition: condition variable for task management * @ufshcd_state: UFSHCD states + * @eh_flags: Error handling flags * @intr_mask: Interrupt Mask Bits * @ee_ctrl_mask: Exception event control mask - * @feh_workq: Work queue for fatal controller error handling + * @eh_work: Worker to handle UFS errors that require s/w attention * @eeh_work: Worker to handle exception events * @errors: HBA errors + * @uic_error: UFS interconnect layer error status + * @saved_err: sticky error mask + * @saved_uic_err: sticky UIC error mask * @dev_cmd: ufs device management command information * @auto_bkops_enabled: to track whether bkops is enabled in device */ @@ -217,21 +223,27 @@ struct ufs_hba { struct uic_command *active_uic_cmd; struct mutex uic_cmd_mutex; - wait_queue_head_t ufshcd_tm_wait_queue; + wait_queue_head_t tm_wq; + wait_queue_head_t tm_tag_wq; unsigned long tm_condition; + unsigned long tm_slots_in_use; struct completion *pwr_done; u32 ufshcd_state; + u32 eh_flags; u32 intr_mask; u16 ee_ctrl_mask; /* Work Queues */ - struct work_struct feh_workq; + struct work_struct eh_work; struct work_struct eeh_work; /* HBA Errors */ u32 errors; + u32 uic_error; + u32 saved_err; + u32 saved_uic_err; /* Device management request data */ struct ufs_dev_cmd dev_cmd; @@ -263,6 +275,8 @@ static inline void check_upiu_size(void) GENERAL_UPIU_REQUEST_SIZE + QUERY_DESC_MAX_SIZE); } +extern int ufshcd_suspend(struct ufs_hba *hba, pm_message_t state); +extern int ufshcd_resume(struct ufs_hba *hba); extern int ufshcd_runtime_suspend(struct ufs_hba *hba); extern int ufshcd_runtime_resume(struct ufs_hba *hba); extern int ufshcd_runtime_idle(struct ufs_hba *hba); diff --git a/drivers/scsi/ufs/ufshci.h b/drivers/scsi/ufs/ufshci.h index 0475c6619a68..9abc7e32b43d 100644 --- a/drivers/scsi/ufs/ufshci.h +++ b/drivers/scsi/ufs/ufshci.h @@ -304,10 +304,10 @@ enum { * @size: size of physical segment DW-3 */ struct ufshcd_sg_entry { - u32 base_addr; - u32 upper_addr; - u32 reserved; - u32 size; + __le32 base_addr; + __le32 upper_addr; + __le32 reserved; + __le32 size; }; /** @@ -330,10 +330,10 @@ struct utp_transfer_cmd_desc { * @dword3: Descriptor Header DW3 */ struct request_desc_header { - u32 dword_0; - u32 dword_1; - u32 dword_2; - u32 dword_3; + __le32 dword_0; + __le32 dword_1; + __le32 dword_2; + __le32 dword_3; }; /** @@ -352,16 +352,16 @@ struct utp_transfer_req_desc { struct request_desc_header header; /* DW 4-5*/ - u32 command_desc_base_addr_lo; - u32 command_desc_base_addr_hi; + __le32 command_desc_base_addr_lo; + __le32 command_desc_base_addr_hi; /* DW 6 */ - u16 response_upiu_length; - u16 response_upiu_offset; + __le16 response_upiu_length; + __le16 response_upiu_offset; /* DW 7 */ - u16 prd_table_length; - u16 prd_table_offset; + __le16 prd_table_length; + __le16 prd_table_offset; }; /** @@ -376,10 +376,10 @@ struct utp_task_req_desc { struct request_desc_header header; /* DW 4-11 */ - u32 task_req_upiu[TASK_REQ_UPIU_SIZE_DWORDS]; + __le32 task_req_upiu[TASK_REQ_UPIU_SIZE_DWORDS]; /* DW 12-19 */ - u32 task_rsp_upiu[TASK_RSP_UPIU_SIZE_DWORDS]; + __le32 task_rsp_upiu[TASK_RSP_UPIU_SIZE_DWORDS]; }; #endif /* End of Header */ diff --git a/drivers/scsi/virtio_scsi.c b/drivers/scsi/virtio_scsi.c index db3b494e5926..d4727b339474 100644 --- a/drivers/scsi/virtio_scsi.c +++ b/drivers/scsi/virtio_scsi.c @@ -73,17 +73,12 @@ struct virtio_scsi_vq { * queue, and also lets the driver optimize the IRQ affinity for the virtqueues * (each virtqueue's affinity is set to the CPU that "owns" the queue). * - * An interesting effect of this policy is that only writes to req_vq need to - * take the tgt_lock. Read can be done outside the lock because: + * tgt_lock is held to serialize reading and writing req_vq. Reading req_vq + * could be done locklessly, but we do not do it yet. * - * - writes of req_vq only occur when atomic_inc_return(&tgt->reqs) returns 1. - * In that case, no other CPU is reading req_vq: even if they were in - * virtscsi_queuecommand_multi, they would be spinning on tgt_lock. - * - * - reads of req_vq only occur when the target is not idle (reqs != 0). - * A CPU that enters virtscsi_queuecommand_multi will not modify req_vq. - * - * Similarly, decrements of reqs are never concurrent with writes of req_vq. + * Decrements of reqs are never concurrent with writes of req_vq: before the + * decrement reqs will be != 0; after the decrement the virtqueue completion + * routine will not use the req_vq so it can be changed by a new request. * Thus they can happen outside the tgt_lock, provided of course we make reqs * an atomic_t. */ @@ -204,7 +199,6 @@ static void virtscsi_complete_cmd(struct virtio_scsi *vscsi, void *buf) set_driver_byte(sc, DRIVER_SENSE); } - mempool_free(cmd, virtscsi_cmd_pool); sc->scsi_done(sc); atomic_dec(&tgt->reqs); @@ -238,38 +232,6 @@ static void virtscsi_req_done(struct virtqueue *vq) int index = vq->index - VIRTIO_SCSI_VQ_BASE; struct virtio_scsi_vq *req_vq = &vscsi->req_vqs[index]; - /* - * Read req_vq before decrementing the reqs field in - * virtscsi_complete_cmd. - * - * With barriers: - * - * CPU #0 virtscsi_queuecommand_multi (CPU #1) - * ------------------------------------------------------------ - * lock vq_lock - * read req_vq - * read reqs (reqs = 1) - * write reqs (reqs = 0) - * increment reqs (reqs = 1) - * write req_vq - * - * Possible reordering without barriers: - * - * CPU #0 virtscsi_queuecommand_multi (CPU #1) - * ------------------------------------------------------------ - * lock vq_lock - * read reqs (reqs = 1) - * write reqs (reqs = 0) - * increment reqs (reqs = 1) - * write req_vq - * read (wrong) req_vq - * - * We do not need a full smp_rmb, because req_vq is required to get - * to tgt->reqs: tgt is &vscsi->tgt[sc->device->id], where sc is stored - * in the virtqueue as the user token. - */ - smp_read_barrier_depends(); - virtscsi_vq_done(vscsi, req_vq, virtscsi_complete_cmd); }; @@ -279,8 +241,6 @@ static void virtscsi_complete_free(struct virtio_scsi *vscsi, void *buf) if (cmd->comp) complete_all(cmd->comp); - else - mempool_free(cmd, virtscsi_cmd_pool); } static void virtscsi_ctrl_done(struct virtqueue *vq) @@ -496,10 +456,9 @@ static int virtscsi_queuecommand(struct virtio_scsi *vscsi, struct virtio_scsi_vq *req_vq, struct scsi_cmnd *sc) { - struct virtio_scsi_cmd *cmd; - int ret; - struct Scsi_Host *shost = virtio_scsi_host(vscsi->vdev); + struct virtio_scsi_cmd *cmd = scsi_cmd_priv(sc); + BUG_ON(scsi_sg_count(sc) > shost->sg_tablesize); /* TODO: check feature bit and fail if unsupported? */ @@ -508,11 +467,6 @@ static int virtscsi_queuecommand(struct virtio_scsi *vscsi, dev_dbg(&sc->device->sdev_gendev, "cmd %p CDB: %#02x\n", sc, sc->cmnd[0]); - ret = SCSI_MLQUEUE_HOST_BUSY; - cmd = mempool_alloc(virtscsi_cmd_pool, GFP_ATOMIC); - if (!cmd) - goto out; - memset(cmd, 0, sizeof(*cmd)); cmd->sc = sc; cmd->req.cmd = (struct virtio_scsi_cmd_req){ @@ -531,13 +485,9 @@ static int virtscsi_queuecommand(struct virtio_scsi *vscsi, if (virtscsi_kick_cmd(req_vq, cmd, sizeof cmd->req.cmd, sizeof cmd->resp.cmd, - GFP_ATOMIC) == 0) - ret = 0; - else - mempool_free(cmd, virtscsi_cmd_pool); - -out: - return ret; + GFP_ATOMIC) != 0) + return SCSI_MLQUEUE_HOST_BUSY; + return 0; } static int virtscsi_queuecommand_single(struct Scsi_Host *sh, @@ -560,12 +510,8 @@ static struct virtio_scsi_vq *virtscsi_pick_vq(struct virtio_scsi *vscsi, spin_lock_irqsave(&tgt->tgt_lock, flags); - /* - * The memory barrier after atomic_inc_return matches - * the smp_read_barrier_depends() in virtscsi_req_done. - */ if (atomic_inc_return(&tgt->reqs) > 1) - vq = ACCESS_ONCE(tgt->req_vq); + vq = tgt->req_vq; else { queue_num = smp_processor_id(); while (unlikely(queue_num >= vscsi->num_queues)) @@ -683,6 +629,7 @@ static struct scsi_host_template virtscsi_host_template_single = { .name = "Virtio SCSI HBA", .proc_name = "virtio_scsi", .this_id = -1, + .cmd_size = sizeof(struct virtio_scsi_cmd), .queuecommand = virtscsi_queuecommand_single, .eh_abort_handler = virtscsi_abort, .eh_device_reset_handler = virtscsi_device_reset, @@ -699,6 +646,7 @@ static struct scsi_host_template virtscsi_host_template_multi = { .name = "Virtio SCSI HBA", .proc_name = "virtio_scsi", .this_id = -1, + .cmd_size = sizeof(struct virtio_scsi_cmd), .queuecommand = virtscsi_queuecommand_multi, .eh_abort_handler = virtscsi_abort, .eh_device_reset_handler = virtscsi_device_reset, diff --git a/drivers/usb/host/ehci-pmcmsp.c b/drivers/usb/host/ehci-pmcmsp.c index af3974a5e7c2..7d75465d97c7 100644 --- a/drivers/usb/host/ehci-pmcmsp.c +++ b/drivers/usb/host/ehci-pmcmsp.c @@ -68,9 +68,6 @@ static void usb_hcd_tdi_set_mode(struct ehci_hcd *ehci) /* set TWI GPIO USB_HOST_DEV pin high */ gpio_direction_output(MSP_PIN_USB0_HOST_DEV, 1); -#ifdef CONFIG_MSP_HAS_DUAL_USB - gpio_direction_output(MSP_PIN_USB1_HOST_DEV, 1); -#endif } /* called during probe() after chip reset completes */ @@ -248,33 +245,6 @@ void usb_hcd_msp_remove(struct usb_hcd *hcd, struct platform_device *dev) usb_put_hcd(hcd); } -#ifdef CONFIG_MSP_HAS_DUAL_USB -/* - * Wrapper around the main ehci_irq. Since both USB host controllers are - * sharing the same IRQ, need to first determine whether we're the intended - * recipient of this interrupt. - */ -static irqreturn_t ehci_msp_irq(struct usb_hcd *hcd) -{ - u32 int_src; - struct device *dev = hcd->self.controller; - struct platform_device *pdev; - struct mspusb_device *mdev; - struct ehci_hcd *ehci = hcd_to_ehci(hcd); - /* need to reverse-map a couple of containers to get our device */ - pdev = to_platform_device(dev); - mdev = to_mspusb_device(pdev); - - /* Check to see if this interrupt is for this host controller */ - int_src = ehci_readl(ehci, &mdev->mab_regs->int_stat); - if (int_src & (1 << pdev->id)) - return ehci_irq(hcd); - - /* Not for this device */ - return IRQ_NONE; -} -#endif /* DUAL_USB */ - static const struct hc_driver ehci_msp_hc_driver = { .description = hcd_name, .product_desc = "PMC MSP EHCI", @@ -283,11 +253,7 @@ static const struct hc_driver ehci_msp_hc_driver = { /* * generic hardware linkage */ -#ifdef CONFIG_MSP_HAS_DUAL_USB - .irq = ehci_msp_irq, -#else .irq = ehci_irq, -#endif .flags = HCD_MEMORY | HCD_USB2 | HCD_BH, /* @@ -334,9 +300,6 @@ static int ehci_hcd_msp_drv_probe(struct platform_device *pdev) return -ENODEV; gpio_request(MSP_PIN_USB0_HOST_DEV, "USB0_HOST_DEV_GPIO"); -#ifdef CONFIG_MSP_HAS_DUAL_USB - gpio_request(MSP_PIN_USB1_HOST_DEV, "USB1_HOST_DEV_GPIO"); -#endif ret = usb_hcd_msp_probe(&ehci_msp_hc_driver, pdev); @@ -351,9 +314,6 @@ static int ehci_hcd_msp_drv_remove(struct platform_device *pdev) /* free TWI GPIO USB_HOST_DEV pin */ gpio_free(MSP_PIN_USB0_HOST_DEV); -#ifdef CONFIG_MSP_HAS_DUAL_USB - gpio_free(MSP_PIN_USB1_HOST_DEV); -#endif return 0; } diff --git a/fs/attr.c b/fs/attr.c index 5d4e59d56e85..6530ced19697 100644 --- a/fs/attr.c +++ b/fs/attr.c @@ -50,14 +50,14 @@ int inode_change_ok(const struct inode *inode, struct iattr *attr) if ((ia_valid & ATTR_UID) && (!uid_eq(current_fsuid(), inode->i_uid) || !uid_eq(attr->ia_uid, inode->i_uid)) && - !inode_capable(inode, CAP_CHOWN)) + !capable_wrt_inode_uidgid(inode, CAP_CHOWN)) return -EPERM; /* Make sure caller can chgrp. */ if ((ia_valid & ATTR_GID) && (!uid_eq(current_fsuid(), inode->i_uid) || (!in_group_p(attr->ia_gid) && !gid_eq(attr->ia_gid, inode->i_gid))) && - !inode_capable(inode, CAP_CHOWN)) + !capable_wrt_inode_uidgid(inode, CAP_CHOWN)) return -EPERM; /* Make sure a caller can chmod. */ @@ -67,7 +67,7 @@ int inode_change_ok(const struct inode *inode, struct iattr *attr) /* Also check the setgid bit! */ if (!in_group_p((ia_valid & ATTR_GID) ? attr->ia_gid : inode->i_gid) && - !inode_capable(inode, CAP_FSETID)) + !capable_wrt_inode_uidgid(inode, CAP_FSETID)) attr->ia_mode &= ~S_ISGID; } @@ -160,7 +160,7 @@ void setattr_copy(struct inode *inode, const struct iattr *attr) umode_t mode = attr->ia_mode; if (!in_group_p(inode->i_gid) && - !inode_capable(inode, CAP_FSETID)) + !capable_wrt_inode_uidgid(inode, CAP_FSETID)) mode &= ~S_ISGID; inode->i_mode = mode; } diff --git a/fs/cifs/cifsfs.c b/fs/cifs/cifsfs.c index 5be1f997ecde..6aaa8112c538 100644 --- a/fs/cifs/cifsfs.c +++ b/fs/cifs/cifsfs.c @@ -87,10 +87,6 @@ extern mempool_t *cifs_mid_poolp; struct workqueue_struct *cifsiod_wq; -#ifdef CONFIG_CIFS_SMB2 -__u8 cifs_client_guid[SMB2_CLIENT_GUID_SIZE]; -#endif - /* * Bumps refcount for cifs super block. * Note that it should be only called if a referece to VFS super block is @@ -251,11 +247,7 @@ cifs_alloc_inode(struct super_block *sb) * server, can not assume caching of file data or metadata. */ cifs_set_oplock_level(cifs_inode, 0); - cifs_inode->delete_pending = false; - cifs_inode->invalid_mapping = false; - clear_bit(CIFS_INODE_PENDING_OPLOCK_BREAK, &cifs_inode->flags); - clear_bit(CIFS_INODE_PENDING_WRITERS, &cifs_inode->flags); - clear_bit(CIFS_INODE_DOWNGRADE_OPLOCK_TO_L2, &cifs_inode->flags); + cifs_inode->flags = 0; spin_lock_init(&cifs_inode->writers_lock); cifs_inode->writers = 0; cifs_inode->vfs_inode.i_blkbits = 14; /* 2**14 = CIFS_MAX_MSGSIZE */ @@ -302,7 +294,7 @@ cifs_show_address(struct seq_file *s, struct TCP_Server_Info *server) struct sockaddr_in *sa = (struct sockaddr_in *) &server->dstaddr; struct sockaddr_in6 *sa6 = (struct sockaddr_in6 *) &server->dstaddr; - seq_printf(s, ",addr="); + seq_puts(s, ",addr="); switch (server->dstaddr.ss_family) { case AF_INET: @@ -314,7 +306,7 @@ cifs_show_address(struct seq_file *s, struct TCP_Server_Info *server) seq_printf(s, "%%%u", sa6->sin6_scope_id); break; default: - seq_printf(s, "(unknown)"); + seq_puts(s, "(unknown)"); } } @@ -324,45 +316,45 @@ cifs_show_security(struct seq_file *s, struct cifs_ses *ses) if (ses->sectype == Unspecified) return; - seq_printf(s, ",sec="); + seq_puts(s, ",sec="); switch (ses->sectype) { case LANMAN: - seq_printf(s, "lanman"); + seq_puts(s, "lanman"); break; case NTLMv2: - seq_printf(s, "ntlmv2"); + seq_puts(s, "ntlmv2"); break; case NTLM: - seq_printf(s, "ntlm"); + seq_puts(s, "ntlm"); break; case Kerberos: - seq_printf(s, "krb5"); + seq_puts(s, "krb5"); break; case RawNTLMSSP: - seq_printf(s, "ntlmssp"); + seq_puts(s, "ntlmssp"); break; default: /* shouldn't ever happen */ - seq_printf(s, "unknown"); + seq_puts(s, "unknown"); break; } if (ses->sign) - seq_printf(s, "i"); + seq_puts(s, "i"); } static void cifs_show_cache_flavor(struct seq_file *s, struct cifs_sb_info *cifs_sb) { - seq_printf(s, ",cache="); + seq_puts(s, ",cache="); if (cifs_sb->mnt_cifs_flags & CIFS_MOUNT_STRICT_IO) - seq_printf(s, "strict"); + seq_puts(s, "strict"); else if (cifs_sb->mnt_cifs_flags & CIFS_MOUNT_DIRECT_IO) - seq_printf(s, "none"); + seq_puts(s, "none"); else - seq_printf(s, "loose"); + seq_puts(s, "loose"); } static void @@ -395,7 +387,7 @@ cifs_show_options(struct seq_file *s, struct dentry *root) cifs_show_cache_flavor(s, cifs_sb); if (cifs_sb->mnt_cifs_flags & CIFS_MOUNT_MULTIUSER) - seq_printf(s, ",multiuser"); + seq_puts(s, ",multiuser"); else if (tcon->ses->user_name) seq_printf(s, ",username=%s", tcon->ses->user_name); @@ -421,16 +413,16 @@ cifs_show_options(struct seq_file *s, struct dentry *root) seq_printf(s, ",uid=%u", from_kuid_munged(&init_user_ns, cifs_sb->mnt_uid)); if (cifs_sb->mnt_cifs_flags & CIFS_MOUNT_OVERR_UID) - seq_printf(s, ",forceuid"); + seq_puts(s, ",forceuid"); else - seq_printf(s, ",noforceuid"); + seq_puts(s, ",noforceuid"); seq_printf(s, ",gid=%u", from_kgid_munged(&init_user_ns, cifs_sb->mnt_gid)); if (cifs_sb->mnt_cifs_flags & CIFS_MOUNT_OVERR_GID) - seq_printf(s, ",forcegid"); + seq_puts(s, ",forcegid"); else - seq_printf(s, ",noforcegid"); + seq_puts(s, ",noforcegid"); cifs_show_address(s, tcon->ses->server); @@ -442,47 +434,47 @@ cifs_show_options(struct seq_file *s, struct dentry *root) cifs_show_nls(s, cifs_sb->local_nls); if (tcon->seal) - seq_printf(s, ",seal"); + seq_puts(s, ",seal"); if (tcon->nocase) - seq_printf(s, ",nocase"); + seq_puts(s, ",nocase"); if (tcon->retry) - seq_printf(s, ",hard"); + seq_puts(s, ",hard"); if (tcon->unix_ext) - seq_printf(s, ",unix"); + seq_puts(s, ",unix"); else - seq_printf(s, ",nounix"); + seq_puts(s, ",nounix"); if (cifs_sb->mnt_cifs_flags & CIFS_MOUNT_POSIX_PATHS) - seq_printf(s, ",posixpaths"); + seq_puts(s, ",posixpaths"); if (cifs_sb->mnt_cifs_flags & CIFS_MOUNT_SET_UID) - seq_printf(s, ",setuids"); + seq_puts(s, ",setuids"); if (cifs_sb->mnt_cifs_flags & CIFS_MOUNT_SERVER_INUM) - seq_printf(s, ",serverino"); + seq_puts(s, ",serverino"); if (cifs_sb->mnt_cifs_flags & CIFS_MOUNT_RWPIDFORWARD) - seq_printf(s, ",rwpidforward"); + seq_puts(s, ",rwpidforward"); if (cifs_sb->mnt_cifs_flags & CIFS_MOUNT_NOPOSIXBRL) - seq_printf(s, ",forcemand"); + seq_puts(s, ",forcemand"); if (cifs_sb->mnt_cifs_flags & CIFS_MOUNT_NO_XATTR) - seq_printf(s, ",nouser_xattr"); + seq_puts(s, ",nouser_xattr"); if (cifs_sb->mnt_cifs_flags & CIFS_MOUNT_MAP_SPECIAL_CHR) - seq_printf(s, ",mapchars"); + seq_puts(s, ",mapchars"); if (cifs_sb->mnt_cifs_flags & CIFS_MOUNT_UNX_EMUL) - seq_printf(s, ",sfu"); + seq_puts(s, ",sfu"); if (cifs_sb->mnt_cifs_flags & CIFS_MOUNT_NO_BRL) - seq_printf(s, ",nobrl"); + seq_puts(s, ",nobrl"); if (cifs_sb->mnt_cifs_flags & CIFS_MOUNT_CIFS_ACL) - seq_printf(s, ",cifsacl"); + seq_puts(s, ",cifsacl"); if (cifs_sb->mnt_cifs_flags & CIFS_MOUNT_DYNPERM) - seq_printf(s, ",dynperm"); + seq_puts(s, ",dynperm"); if (root->d_sb->s_flags & MS_POSIXACL) - seq_printf(s, ",acl"); + seq_puts(s, ",acl"); if (cifs_sb->mnt_cifs_flags & CIFS_MOUNT_MF_SYMLINKS) - seq_printf(s, ",mfsymlinks"); + seq_puts(s, ",mfsymlinks"); if (cifs_sb->mnt_cifs_flags & CIFS_MOUNT_FSCACHE) - seq_printf(s, ",fsc"); + seq_puts(s, ",fsc"); if (cifs_sb->mnt_cifs_flags & CIFS_MOUNT_NOSSYNC) - seq_printf(s, ",nostrictsync"); + seq_puts(s, ",nostrictsync"); if (cifs_sb->mnt_cifs_flags & CIFS_MOUNT_NO_PERM) - seq_printf(s, ",noperm"); + seq_puts(s, ",noperm"); if (cifs_sb->mnt_cifs_flags & CIFS_MOUNT_CIFS_BACKUPUID) seq_printf(s, ",backupuid=%u", from_kuid_munged(&init_user_ns, @@ -1192,10 +1184,6 @@ init_cifs(void) spin_lock_init(&cifs_file_list_lock); spin_lock_init(&GlobalMid_Lock); -#ifdef CONFIG_CIFS_SMB2 - get_random_bytes(cifs_client_guid, SMB2_CLIENT_GUID_SIZE); -#endif - if (cifs_max_pending < 2) { cifs_max_pending = 2; cifs_dbg(FYI, "cifs_max_pending set to min of 2\n"); diff --git a/fs/cifs/cifsfs.h b/fs/cifs/cifsfs.h index 26a754f49ba1..8fe51166d6e3 100644 --- a/fs/cifs/cifsfs.h +++ b/fs/cifs/cifsfs.h @@ -22,20 +22,28 @@ #ifndef _CIFSFS_H #define _CIFSFS_H +#include <linux/hash.h> + #define ROOT_I 2 /* * ino_t is 32-bits on 32-bit arch. We have to squash the 64-bit value down - * so that it will fit. + * so that it will fit. We use hash_64 to convert the value to 31 bits, and + * then add 1, to ensure that we don't end up with a 0 as the value. */ +#if BITS_PER_LONG == 64 static inline ino_t cifs_uniqueid_to_ino_t(u64 fileid) { - ino_t ino = (ino_t) fileid; - if (sizeof(ino_t) < sizeof(u64)) - ino ^= fileid >> (sizeof(u64)-sizeof(ino_t)) * 8; - return ino; + return (ino_t)fileid; } +#else +static inline ino_t +cifs_uniqueid_to_ino_t(u64 fileid) +{ + return (ino_t)hash_64(fileid, (sizeof(ino_t) * 8) - 1) + 1; +} +#endif extern struct file_system_type cifs_fs_type; extern const struct address_space_operations cifs_addr_ops; @@ -67,6 +75,8 @@ extern int cifs_revalidate_dentry_attr(struct dentry *); extern int cifs_revalidate_file(struct file *filp); extern int cifs_revalidate_dentry(struct dentry *); extern int cifs_invalidate_mapping(struct inode *inode); +extern int cifs_revalidate_mapping(struct inode *inode); +extern int cifs_zap_mapping(struct inode *inode); extern int cifs_getattr(struct vfsmount *, struct dentry *, struct kstat *); extern int cifs_setattr(struct dentry *, struct iattr *); @@ -130,5 +140,5 @@ extern long cifs_ioctl(struct file *filep, unsigned int cmd, unsigned long arg); extern const struct export_operations cifs_export_ops; #endif /* CONFIG_CIFS_NFSD_EXPORT */ -#define CIFS_VERSION "2.02" +#define CIFS_VERSION "2.03" #endif /* _CIFSFS_H */ diff --git a/fs/cifs/cifsglob.h b/fs/cifs/cifsglob.h index 30f6e9251a4a..de6aed8c78e5 100644 --- a/fs/cifs/cifsglob.h +++ b/fs/cifs/cifsglob.h @@ -559,6 +559,7 @@ struct TCP_Server_Info { int echo_credits; /* echo reserved slots */ int oplock_credits; /* oplock break reserved slots */ bool echoes:1; /* enable echoes */ + __u8 client_guid[SMB2_CLIENT_GUID_SIZE]; /* Client GUID */ #endif u16 dialect; /* dialect index that server chose */ bool oplocks:1; /* enable oplocks */ @@ -1113,12 +1114,13 @@ struct cifsInodeInfo { __u32 cifsAttrs; /* e.g. DOS archive bit, sparse, compressed, system */ unsigned int oplock; /* oplock/lease level we have */ unsigned int epoch; /* used to track lease state changes */ - bool delete_pending; /* DELETE_ON_CLOSE is set */ - bool invalid_mapping; /* pagecache is invalid */ - unsigned long flags; #define CIFS_INODE_PENDING_OPLOCK_BREAK (0) /* oplock break in progress */ #define CIFS_INODE_PENDING_WRITERS (1) /* Writes in progress */ #define CIFS_INODE_DOWNGRADE_OPLOCK_TO_L2 (2) /* Downgrade oplock to L2 */ +#define CIFS_INO_DELETE_PENDING (3) /* delete pending on server */ +#define CIFS_INO_INVALID_MAPPING (4) /* pagecache is invalid */ +#define CIFS_INO_LOCK (5) /* lock bit for synchronization */ + unsigned long flags; spinlock_t writers_lock; unsigned int writers; /* Number of writers on this inode */ unsigned long time; /* jiffies of last update of inode */ diff --git a/fs/cifs/connect.c b/fs/cifs/connect.c index 8813ff776ba3..20d75b8ddb26 100644 --- a/fs/cifs/connect.c +++ b/fs/cifs/connect.c @@ -2144,6 +2144,9 @@ cifs_get_tcp_session(struct smb_vol *volume_info) sizeof(tcp_ses->srcaddr)); memcpy(&tcp_ses->dstaddr, &volume_info->dstaddr, sizeof(tcp_ses->dstaddr)); +#ifdef CONFIG_CIFS_SMB2 + get_random_bytes(tcp_ses->client_guid, SMB2_CLIENT_GUID_SIZE); +#endif /* * at this point we are the only ones with the pointer * to the struct since the kernel thread not created yet @@ -2225,7 +2228,7 @@ static int match_session(struct cifs_ses *ses, struct smb_vol *vol) vol->username ? vol->username : "", CIFS_MAX_USERNAME_LEN)) return 0; - if (strlen(vol->username) != 0 && + if ((vol->username && strlen(vol->username) != 0) && ses->password != NULL && strncmp(ses->password, vol->password ? vol->password : "", diff --git a/fs/cifs/file.c b/fs/cifs/file.c index 5ed03e0b8b40..208f56eca4bf 100644 --- a/fs/cifs/file.c +++ b/fs/cifs/file.c @@ -335,7 +335,7 @@ cifs_new_fileinfo(struct cifs_fid *fid, struct file *file, spin_unlock(&cifs_file_list_lock); if (fid->purge_cache) - cifs_invalidate_mapping(inode); + cifs_zap_mapping(inode); file->private_data = cfile; return cfile; @@ -392,7 +392,7 @@ void cifsFileInfo_put(struct cifsFileInfo *cifs_file) * again and get at least level II oplock. */ if (cifs_sb->mnt_cifs_flags & CIFS_MOUNT_STRICT_IO) - CIFS_I(inode)->invalid_mapping = true; + set_bit(CIFS_INO_INVALID_MAPPING, &cifsi->flags); cifs_set_oplock_level(cifsi, 0); } spin_unlock(&cifs_file_list_lock); @@ -1529,7 +1529,7 @@ cifs_setlk(struct file *file, struct file_lock *flock, __u32 type, */ if (!CIFS_CACHE_WRITE(CIFS_I(inode)) && CIFS_CACHE_READ(CIFS_I(inode))) { - cifs_invalidate_mapping(inode); + cifs_zap_mapping(inode); cifs_dbg(FYI, "Set no oplock for inode=%p due to mand locks\n", inode); CIFS_I(inode)->oplock = 0; @@ -2218,7 +2218,7 @@ int cifs_strict_fsync(struct file *file, loff_t start, loff_t end, file->f_path.dentry->d_name.name, datasync); if (!CIFS_CACHE_READ(CIFS_I(inode))) { - rc = cifs_invalidate_mapping(inode); + rc = cifs_zap_mapping(inode); if (rc) { cifs_dbg(FYI, "rc: %d during invalidate phase\n", rc); rc = 0; /* don't care about it in fsync */ @@ -2562,7 +2562,7 @@ ssize_t cifs_user_writev(struct kiocb *iocb, const struct iovec *iov, written = cifs_iovec_write(iocb->ki_filp, iov, nr_segs, &pos); if (written > 0) { - CIFS_I(inode)->invalid_mapping = true; + set_bit(CIFS_INO_INVALID_MAPPING, &CIFS_I(inode)->flags); iocb->ki_pos = pos; } @@ -2649,7 +2649,7 @@ cifs_strict_writev(struct kiocb *iocb, const struct iovec *iov, * request comes - break it on the client to prevent reading * an old data. */ - cifs_invalidate_mapping(inode); + cifs_zap_mapping(inode); cifs_dbg(FYI, "Set no oplock for inode=%p after a write operation\n", inode); cinode->oplock = 0; @@ -3112,7 +3112,7 @@ int cifs_file_strict_mmap(struct file *file, struct vm_area_struct *vma) xid = get_xid(); if (!CIFS_CACHE_READ(CIFS_I(inode))) { - rc = cifs_invalidate_mapping(inode); + rc = cifs_zap_mapping(inode); if (rc) return rc; } @@ -3670,7 +3670,7 @@ void cifs_oplock_break(struct work_struct *work) if (!CIFS_CACHE_READ(cinode)) { rc = filemap_fdatawait(inode->i_mapping); mapping_set_error(inode->i_mapping, rc); - cifs_invalidate_mapping(inode); + cifs_zap_mapping(inode); } cifs_dbg(FYI, "Oplock flush inode %p rc %d\n", inode, rc); } diff --git a/fs/cifs/inode.c b/fs/cifs/inode.c index a22d667f1069..a174605f6afa 100644 --- a/fs/cifs/inode.c +++ b/fs/cifs/inode.c @@ -22,6 +22,7 @@ #include <linux/stat.h> #include <linux/slab.h> #include <linux/pagemap.h> +#include <linux/freezer.h> #include <asm/div64.h> #include "cifsfs.h" #include "cifspdu.h" @@ -117,7 +118,7 @@ cifs_revalidate_cache(struct inode *inode, struct cifs_fattr *fattr) cifs_dbg(FYI, "%s: invalidating inode %llu mapping\n", __func__, cifs_i->uniqueid); - cifs_i->invalid_mapping = true; + set_bit(CIFS_INO_INVALID_MAPPING, &cifs_i->flags); } /* @@ -177,7 +178,10 @@ cifs_fattr_to_inode(struct inode *inode, struct cifs_fattr *fattr) else cifs_i->time = jiffies; - cifs_i->delete_pending = fattr->cf_flags & CIFS_FATTR_DELETE_PENDING; + if (fattr->cf_flags & CIFS_FATTR_DELETE_PENDING) + set_bit(CIFS_INO_DELETE_PENDING, &cifs_i->flags); + else + clear_bit(CIFS_INO_DELETE_PENDING, &cifs_i->flags); cifs_i->server_eof = fattr->cf_eof; /* @@ -1121,7 +1125,7 @@ cifs_rename_pending_delete(const char *full_path, struct dentry *dentry, } /* try to set DELETE_ON_CLOSE */ - if (!cifsInode->delete_pending) { + if (!test_bit(CIFS_INO_DELETE_PENDING, &cifsInode->flags)) { rc = CIFSSMBSetFileDisposition(xid, tcon, true, fid.netfid, current->tgid); /* @@ -1138,7 +1142,7 @@ cifs_rename_pending_delete(const char *full_path, struct dentry *dentry, rc = -EBUSY; goto undo_rename; } - cifsInode->delete_pending = true; + set_bit(CIFS_INO_DELETE_PENDING, &cifsInode->flags); } out_close: @@ -1759,23 +1763,62 @@ int cifs_invalidate_mapping(struct inode *inode) { int rc = 0; - struct cifsInodeInfo *cifs_i = CIFS_I(inode); - - cifs_i->invalid_mapping = false; if (inode->i_mapping && inode->i_mapping->nrpages != 0) { rc = invalidate_inode_pages2(inode->i_mapping); - if (rc) { + if (rc) cifs_dbg(VFS, "%s: could not invalidate inode %p\n", __func__, inode); - cifs_i->invalid_mapping = true; - } } cifs_fscache_reset_inode_cookie(inode); return rc; } +/** + * cifs_wait_bit_killable - helper for functions that are sleeping on bit locks + * @word: long word containing the bit lock + */ +static int +cifs_wait_bit_killable(void *word) +{ + if (fatal_signal_pending(current)) + return -ERESTARTSYS; + freezable_schedule_unsafe(); + return 0; +} + +int +cifs_revalidate_mapping(struct inode *inode) +{ + int rc; + unsigned long *flags = &CIFS_I(inode)->flags; + + rc = wait_on_bit_lock(flags, CIFS_INO_LOCK, cifs_wait_bit_killable, + TASK_KILLABLE); + if (rc) + return rc; + + if (test_and_clear_bit(CIFS_INO_INVALID_MAPPING, flags)) { + rc = cifs_invalidate_mapping(inode); + if (rc) + set_bit(CIFS_INO_INVALID_MAPPING, flags); + } + + clear_bit_unlock(CIFS_INO_LOCK, flags); + smp_mb__after_atomic(); + wake_up_bit(flags, CIFS_INO_LOCK); + + return rc; +} + +int +cifs_zap_mapping(struct inode *inode) +{ + set_bit(CIFS_INO_INVALID_MAPPING, &CIFS_I(inode)->flags); + return cifs_revalidate_mapping(inode); +} + int cifs_revalidate_file_attr(struct file *filp) { int rc = 0; @@ -1842,9 +1885,7 @@ int cifs_revalidate_file(struct file *filp) if (rc) return rc; - if (CIFS_I(inode)->invalid_mapping) - rc = cifs_invalidate_mapping(inode); - return rc; + return cifs_revalidate_mapping(inode); } /* revalidate a dentry's inode attributes */ @@ -1857,9 +1898,7 @@ int cifs_revalidate_dentry(struct dentry *dentry) if (rc) return rc; - if (CIFS_I(inode)->invalid_mapping) - rc = cifs_invalidate_mapping(inode); - return rc; + return cifs_revalidate_mapping(inode); } int cifs_getattr(struct vfsmount *mnt, struct dentry *dentry, diff --git a/fs/cifs/ioctl.c b/fs/cifs/ioctl.c index 77492301cc2b..45cb59bcc791 100644 --- a/fs/cifs/ioctl.c +++ b/fs/cifs/ioctl.c @@ -85,7 +85,7 @@ static long cifs_ioctl_clone(unsigned int xid, struct file *dst_file, goto out_fput; } - src_inode = src_file.file->f_dentry->d_inode; + src_inode = file_inode(src_file.file); /* * Note: cifs case is easier than btrfs since server responsible for diff --git a/fs/cifs/netmisc.c b/fs/cifs/netmisc.c index 049884552e76..6834b9c3bec1 100644 --- a/fs/cifs/netmisc.c +++ b/fs/cifs/netmisc.c @@ -795,8 +795,8 @@ cifs_print_status(__u32 status_code) while (nt_errs[idx].nt_errstr != NULL) { if (((nt_errs[idx].nt_errcode) & 0xFFFFFF) == (status_code & 0xFFFFFF)) { - printk(KERN_NOTICE "Status code returned 0x%08x %s\n", - status_code, nt_errs[idx].nt_errstr); + pr_notice("Status code returned 0x%08x %s\n", + status_code, nt_errs[idx].nt_errstr); } idx++; } @@ -941,8 +941,9 @@ cifs_UnixTimeToNT(struct timespec t) return (u64) t.tv_sec * 10000000 + t.tv_nsec/100 + NTFS_TIME_OFFSET; } -static int total_days_of_prev_months[] = -{0, 31, 59, 90, 120, 151, 181, 212, 243, 273, 304, 334}; +static const int total_days_of_prev_months[] = { + 0, 31, 59, 90, 120, 151, 181, 212, 243, 273, 304, 334 +}; struct timespec cnvrtDosUnixTm(__le16 le_date, __le16 le_time, int offset) { diff --git a/fs/cifs/smb2ops.c b/fs/cifs/smb2ops.c index 35ddc3ed119d..787844bde384 100644 --- a/fs/cifs/smb2ops.c +++ b/fs/cifs/smb2ops.c @@ -1047,6 +1047,7 @@ smb2_create_lease_buf(u8 *lease_key, u8 oplock) buf->ccontext.NameOffset = cpu_to_le16(offsetof (struct create_lease, Name)); buf->ccontext.NameLength = cpu_to_le16(4); + /* SMB2_CREATE_REQUEST_LEASE is "RqLs" */ buf->Name[0] = 'R'; buf->Name[1] = 'q'; buf->Name[2] = 'L'; @@ -1073,6 +1074,7 @@ smb3_create_lease_buf(u8 *lease_key, u8 oplock) buf->ccontext.NameOffset = cpu_to_le16(offsetof (struct create_lease_v2, Name)); buf->ccontext.NameLength = cpu_to_le16(4); + /* SMB2_CREATE_REQUEST_LEASE is "RqLs" */ buf->Name[0] = 'R'; buf->Name[1] = 'q'; buf->Name[2] = 'L'; diff --git a/fs/cifs/smb2pdu.c b/fs/cifs/smb2pdu.c index 3802f8c94acc..b0b260dbb19d 100644 --- a/fs/cifs/smb2pdu.c +++ b/fs/cifs/smb2pdu.c @@ -375,7 +375,12 @@ SMB2_negotiate(const unsigned int xid, struct cifs_ses *ses) req->Capabilities = cpu_to_le32(ses->server->vals->req_capabilities); - memcpy(req->ClientGUID, cifs_client_guid, SMB2_CLIENT_GUID_SIZE); + /* ClientGUID must be zero for SMB2.02 dialect */ + if (ses->server->vals->protocol_id == SMB20_PROT_ID) + memset(req->ClientGUID, 0, SMB2_CLIENT_GUID_SIZE); + else + memcpy(req->ClientGUID, server->client_guid, + SMB2_CLIENT_GUID_SIZE); iov[0].iov_base = (char *)req; /* 4 for rfc1002 length field */ @@ -478,7 +483,8 @@ int smb3_validate_negotiate(const unsigned int xid, struct cifs_tcon *tcon) vneg_inbuf.Capabilities = cpu_to_le32(tcon->ses->server->vals->req_capabilities); - memcpy(vneg_inbuf.Guid, cifs_client_guid, SMB2_CLIENT_GUID_SIZE); + memcpy(vneg_inbuf.Guid, tcon->ses->server->client_guid, + SMB2_CLIENT_GUID_SIZE); if (tcon->ses->sign) vneg_inbuf.SecurityMode = @@ -966,6 +972,7 @@ create_durable_buf(void) buf->ccontext.NameOffset = cpu_to_le16(offsetof (struct create_durable, Name)); buf->ccontext.NameLength = cpu_to_le16(4); + /* SMB2_CREATE_DURABLE_HANDLE_REQUEST is "DHnQ" */ buf->Name[0] = 'D'; buf->Name[1] = 'H'; buf->Name[2] = 'n'; @@ -990,6 +997,7 @@ create_reconnect_durable_buf(struct cifs_fid *fid) buf->ccontext.NameLength = cpu_to_le16(4); buf->Data.Fid.PersistentFileId = fid->persistent_fid; buf->Data.Fid.VolatileFileId = fid->volatile_fid; + /* SMB2_CREATE_DURABLE_HANDLE_RECONNECT is "DHnC" */ buf->Name[0] = 'D'; buf->Name[1] = 'H'; buf->Name[2] = 'n'; @@ -1089,6 +1097,7 @@ SMB2_open(const unsigned int xid, struct cifs_open_parms *oparms, __le16 *path, int rc = 0; unsigned int num_iovecs = 2; __u32 file_attributes = 0; + char *dhc_buf = NULL, *lc_buf = NULL; cifs_dbg(FYI, "create/open\n"); @@ -1155,6 +1164,7 @@ SMB2_open(const unsigned int xid, struct cifs_open_parms *oparms, __le16 *path, kfree(copy_path); return rc; } + lc_buf = iov[num_iovecs-1].iov_base; } if (*oplock == SMB2_OPLOCK_LEVEL_BATCH) { @@ -1169,9 +1179,10 @@ SMB2_open(const unsigned int xid, struct cifs_open_parms *oparms, __le16 *path, if (rc) { cifs_small_buf_release(req); kfree(copy_path); - kfree(iov[num_iovecs-1].iov_base); + kfree(lc_buf); return rc; } + dhc_buf = iov[num_iovecs-1].iov_base; } rc = SendReceive2(xid, ses, iov, num_iovecs, &resp_buftype, 0); @@ -1203,6 +1214,8 @@ SMB2_open(const unsigned int xid, struct cifs_open_parms *oparms, __le16 *path, *oplock = rsp->OplockLevel; creat_exit: kfree(copy_path); + kfree(lc_buf); + kfree(dhc_buf); free_rsp_buf(resp_buftype, rsp); return rc; } diff --git a/fs/cifs/smb2pdu.h b/fs/cifs/smb2pdu.h index 2022c542ea3a..69f3595d3952 100644 --- a/fs/cifs/smb2pdu.h +++ b/fs/cifs/smb2pdu.h @@ -183,8 +183,6 @@ struct smb2_symlink_err_rsp { #define SMB2_CLIENT_GUID_SIZE 16 -extern __u8 cifs_client_guid[SMB2_CLIENT_GUID_SIZE]; - struct smb2_negotiate_req { struct smb2_hdr hdr; __le16 StructureSize; /* Must be 36 */ @@ -437,11 +435,15 @@ struct smb2_tree_disconnect_rsp { #define SMB2_CREATE_SD_BUFFER "SecD" /* security descriptor */ #define SMB2_CREATE_DURABLE_HANDLE_REQUEST "DHnQ" #define SMB2_CREATE_DURABLE_HANDLE_RECONNECT "DHnC" -#define SMB2_CREATE_ALLOCATION_SIZE "AlSi" +#define SMB2_CREATE_ALLOCATION_SIZE "AISi" #define SMB2_CREATE_QUERY_MAXIMAL_ACCESS_REQUEST "MxAc" #define SMB2_CREATE_TIMEWARP_REQUEST "TWrp" #define SMB2_CREATE_QUERY_ON_DISK_ID "QFid" #define SMB2_CREATE_REQUEST_LEASE "RqLs" +#define SMB2_CREATE_DURABLE_HANDLE_REQUEST_V2 "DH2Q" +#define SMB2_CREATE_DURABLE_HANDLE_RECONNECT_V2 "DH2C" +#define SMB2_CREATE_APP_INSTANCE_ID 0x45BCA66AEFA7F74A9008FA462E144D74 +#define SVHDX_OPEN_DEVICE_CONTEXT 0x83CE6F1AD851E0986E34401CC9BCFCE9 struct smb2_create_req { struct smb2_hdr hdr; diff --git a/fs/f2fs/acl.c b/fs/f2fs/acl.c index e93e4ec7d165..dbe2141d10ad 100644 --- a/fs/f2fs/acl.c +++ b/fs/f2fs/acl.c @@ -240,7 +240,7 @@ static int __f2fs_set_acl(struct inode *inode, int type, } } - error = f2fs_setxattr(inode, name_index, "", value, size, ipage); + error = f2fs_setxattr(inode, name_index, "", value, size, ipage, 0); kfree(value); if (!error) diff --git a/fs/f2fs/checkpoint.c b/fs/f2fs/checkpoint.c index c405b8f17054..0b4710c1d370 100644 --- a/fs/f2fs/checkpoint.c +++ b/fs/f2fs/checkpoint.c @@ -33,12 +33,12 @@ struct page *grab_meta_page(struct f2fs_sb_info *sbi, pgoff_t index) struct address_space *mapping = META_MAPPING(sbi); struct page *page = NULL; repeat: - page = grab_cache_page_write_begin(mapping, index, AOP_FLAG_NOFS); + page = grab_cache_page(mapping, index); if (!page) { cond_resched(); goto repeat; } - + f2fs_wait_on_page_writeback(page, META); SetPageUptodate(page); return page; } @@ -72,7 +72,7 @@ out: return page; } -inline int get_max_meta_blks(struct f2fs_sb_info *sbi, int type) +static inline int get_max_meta_blks(struct f2fs_sb_info *sbi, int type) { switch (type) { case META_NAT: @@ -154,6 +154,8 @@ static int f2fs_write_meta_page(struct page *page, struct inode *inode = page->mapping->host; struct f2fs_sb_info *sbi = F2FS_SB(inode->i_sb); + trace_f2fs_writepage(page, META); + if (unlikely(sbi->por_doing)) goto redirty_out; if (wbc->for_reclaim) @@ -171,10 +173,7 @@ no_write: return 0; redirty_out: - dec_page_count(sbi, F2FS_DIRTY_META); - wbc->pages_skipped++; - account_page_redirty(page); - set_page_dirty(page); + redirty_page_for_writepage(wbc, page); return AOP_WRITEPAGE_ACTIVATE; } @@ -184,6 +183,8 @@ static int f2fs_write_meta_pages(struct address_space *mapping, struct f2fs_sb_info *sbi = F2FS_SB(mapping->host->i_sb); long diff, written; + trace_f2fs_writepages(mapping->host, wbc, META); + /* collect a number of dirty meta pages and write together */ if (wbc->for_kupdate || get_pages(sbi, F2FS_DIRTY_META) < nr_pages_to_skip(sbi, META)) @@ -367,7 +368,9 @@ void recover_orphan_inodes(struct f2fs_sb_info *sbi) return; sbi->por_doing = true; - start_blk = __start_cp_addr(sbi) + 1; + + start_blk = __start_cp_addr(sbi) + 1 + + le32_to_cpu(F2FS_RAW_SUPER(sbi)->cp_payload); orphan_blkaddr = __start_sum_addr(sbi) - 1; ra_meta_pages(sbi, start_blk, orphan_blkaddr, META_CP); @@ -508,8 +511,11 @@ int get_valid_checkpoint(struct f2fs_sb_info *sbi) unsigned long blk_size = sbi->blocksize; unsigned long long cp1_version = 0, cp2_version = 0; unsigned long long cp_start_blk_no; + unsigned int cp_blks = 1 + le32_to_cpu(F2FS_RAW_SUPER(sbi)->cp_payload); + block_t cp_blk_no; + int i; - sbi->ckpt = kzalloc(blk_size, GFP_KERNEL); + sbi->ckpt = kzalloc(cp_blks * blk_size, GFP_KERNEL); if (!sbi->ckpt) return -ENOMEM; /* @@ -540,6 +546,23 @@ int get_valid_checkpoint(struct f2fs_sb_info *sbi) cp_block = (struct f2fs_checkpoint *)page_address(cur_page); memcpy(sbi->ckpt, cp_block, blk_size); + if (cp_blks <= 1) + goto done; + + cp_blk_no = le32_to_cpu(fsb->cp_blkaddr); + if (cur_page == cp2) + cp_blk_no += 1 << le32_to_cpu(fsb->log_blocks_per_seg); + + for (i = 1; i < cp_blks; i++) { + void *sit_bitmap_ptr; + unsigned char *ckpt = (unsigned char *)sbi->ckpt; + + cur_page = get_meta_page(sbi, cp_blk_no + i); + sit_bitmap_ptr = page_address(cur_page); + memcpy(ckpt + i * blk_size, sit_bitmap_ptr, blk_size); + f2fs_put_page(cur_page, 1); + } +done: f2fs_put_page(cp1, 1); f2fs_put_page(cp2, 1); return 0; @@ -552,14 +575,13 @@ fail_no_cp: static int __add_dirty_inode(struct inode *inode, struct dir_inode_entry *new) { struct f2fs_sb_info *sbi = F2FS_SB(inode->i_sb); - struct list_head *head = &sbi->dir_inode_list; - struct dir_inode_entry *entry; - list_for_each_entry(entry, head, list) - if (unlikely(entry->inode == inode)) - return -EEXIST; + if (is_inode_flag_set(F2FS_I(inode), FI_DIRTY_DIR)) + return -EEXIST; - list_add_tail(&new->list, head); + set_inode_flag(F2FS_I(inode), FI_DIRTY_DIR); + F2FS_I(inode)->dirty_dir = new; + list_add_tail(&new->list, &sbi->dir_inode_list); stat_inc_dirty_dir(sbi); return 0; } @@ -608,31 +630,26 @@ void add_dirty_dir_inode(struct inode *inode) void remove_dirty_dir_inode(struct inode *inode) { struct f2fs_sb_info *sbi = F2FS_SB(inode->i_sb); - struct list_head *head; struct dir_inode_entry *entry; if (!S_ISDIR(inode->i_mode)) return; spin_lock(&sbi->dir_inode_lock); - if (get_dirty_dents(inode)) { + if (get_dirty_dents(inode) || + !is_inode_flag_set(F2FS_I(inode), FI_DIRTY_DIR)) { spin_unlock(&sbi->dir_inode_lock); return; } - head = &sbi->dir_inode_list; - list_for_each_entry(entry, head, list) { - if (entry->inode == inode) { - list_del(&entry->list); - stat_dec_dirty_dir(sbi); - spin_unlock(&sbi->dir_inode_lock); - kmem_cache_free(inode_entry_slab, entry); - goto done; - } - } + entry = F2FS_I(inode)->dirty_dir; + list_del(&entry->list); + F2FS_I(inode)->dirty_dir = NULL; + clear_inode_flag(F2FS_I(inode), FI_DIRTY_DIR); + stat_dec_dirty_dir(sbi); spin_unlock(&sbi->dir_inode_lock); + kmem_cache_free(inode_entry_slab, entry); -done: /* Only from the recovery routine */ if (is_inode_flag_set(F2FS_I(inode), FI_DELAY_IPUT)) { clear_inode_flag(F2FS_I(inode), FI_DELAY_IPUT); @@ -640,26 +657,6 @@ done: } } -struct inode *check_dirty_dir_inode(struct f2fs_sb_info *sbi, nid_t ino) -{ - - struct list_head *head; - struct inode *inode = NULL; - struct dir_inode_entry *entry; - - spin_lock(&sbi->dir_inode_lock); - - head = &sbi->dir_inode_list; - list_for_each_entry(entry, head, list) { - if (entry->inode->i_ino == ino) { - inode = entry->inode; - break; - } - } - spin_unlock(&sbi->dir_inode_lock); - return inode; -} - void sync_dirty_dir_inodes(struct f2fs_sb_info *sbi) { struct list_head *head; @@ -758,6 +755,13 @@ static void do_checkpoint(struct f2fs_sb_info *sbi, bool is_umount) __u32 crc32 = 0; void *kaddr; int i; + int cp_payload_blks = le32_to_cpu(F2FS_RAW_SUPER(sbi)->cp_payload); + + /* + * This avoids to conduct wrong roll-forward operations and uses + * metapages, so should be called prior to sync_meta_pages below. + */ + discard_next_dnode(sbi); /* Flush all the NAT/SIT pages */ while (get_pages(sbi, F2FS_DIRTY_META)) @@ -802,16 +806,19 @@ static void do_checkpoint(struct f2fs_sb_info *sbi, bool is_umount) orphan_blocks = (sbi->n_orphans + F2FS_ORPHANS_PER_BLOCK - 1) / F2FS_ORPHANS_PER_BLOCK; - ckpt->cp_pack_start_sum = cpu_to_le32(1 + orphan_blocks); + ckpt->cp_pack_start_sum = cpu_to_le32(1 + cp_payload_blks + + orphan_blocks); if (is_umount) { set_ckpt_flags(ckpt, CP_UMOUNT_FLAG); ckpt->cp_pack_total_block_count = cpu_to_le32(2 + - data_sum_blocks + orphan_blocks + NR_CURSEG_NODE_TYPE); + cp_payload_blks + data_sum_blocks + + orphan_blocks + NR_CURSEG_NODE_TYPE); } else { clear_ckpt_flags(ckpt, CP_UMOUNT_FLAG); ckpt->cp_pack_total_block_count = cpu_to_le32(2 + - data_sum_blocks + orphan_blocks); + cp_payload_blks + data_sum_blocks + + orphan_blocks); } if (sbi->n_orphans) @@ -837,6 +844,15 @@ static void do_checkpoint(struct f2fs_sb_info *sbi, bool is_umount) set_page_dirty(cp_page); f2fs_put_page(cp_page, 1); + for (i = 1; i < 1 + cp_payload_blks; i++) { + cp_page = grab_meta_page(sbi, start_blk++); + kaddr = page_address(cp_page); + memcpy(kaddr, (char *)ckpt + i * F2FS_BLKSIZE, + (1 << sbi->log_blocksize)); + set_page_dirty(cp_page); + f2fs_put_page(cp_page, 1); + } + if (sbi->n_orphans) { write_orphan_inodes(sbi, start_blk); start_blk += orphan_blocks; diff --git a/fs/f2fs/data.c b/fs/f2fs/data.c index 45abd60e2bff..c1fb6dd10911 100644 --- a/fs/f2fs/data.c +++ b/fs/f2fs/data.c @@ -417,7 +417,7 @@ struct page *find_data_page(struct inode *inode, pgoff_t index, bool sync) if (unlikely(dn.data_blkaddr == NEW_ADDR)) return ERR_PTR(-EINVAL); - page = grab_cache_page_write_begin(mapping, index, AOP_FLAG_NOFS); + page = grab_cache_page(mapping, index); if (!page) return ERR_PTR(-ENOMEM); @@ -455,7 +455,7 @@ struct page *get_lock_data_page(struct inode *inode, pgoff_t index) int err; repeat: - page = grab_cache_page_write_begin(mapping, index, AOP_FLAG_NOFS); + page = grab_cache_page(mapping, index); if (!page) return ERR_PTR(-ENOMEM); @@ -652,8 +652,7 @@ static int get_data_block(struct inode *inode, sector_t iblock, goto put_out; } - end_offset = IS_INODE(dn.node_page) ? - ADDRS_PER_INODE(F2FS_I(inode)) : ADDRS_PER_BLOCK; + end_offset = ADDRS_PER_PAGE(dn.node_page, F2FS_I(inode)); bh_result->b_size = (((size_t)1) << blkbits); dn.ofs_in_node++; pgofs++; @@ -675,8 +674,7 @@ get_next: if (dn.data_blkaddr == NEW_ADDR) goto put_out; - end_offset = IS_INODE(dn.node_page) ? - ADDRS_PER_INODE(F2FS_I(inode)) : ADDRS_PER_BLOCK; + end_offset = ADDRS_PER_PAGE(dn.node_page, F2FS_I(inode)); } if (maxblocks > (bh_result->b_size >> blkbits)) { @@ -710,11 +708,19 @@ out: return err; } +int f2fs_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo, + u64 start, u64 len) +{ + return generic_block_fiemap(inode, fieinfo, start, len, get_data_block); +} + static int f2fs_read_data_page(struct file *file, struct page *page) { struct inode *inode = page->mapping->host; int ret; + trace_f2fs_readpage(page, DATA); + /* If the file has inline data, try to read it directlly */ if (f2fs_has_inline_data(inode)) ret = f2fs_read_inline_data(inode, page); @@ -790,6 +796,8 @@ static int f2fs_write_data_page(struct page *page, .rw = (wbc->sync_mode == WB_SYNC_ALL) ? WRITE_SYNC : WRITE, }; + trace_f2fs_writepage(page, DATA); + if (page->index < end_index) goto write; @@ -798,10 +806,8 @@ static int f2fs_write_data_page(struct page *page, * this page does not have to be written to disk. */ offset = i_size & (PAGE_CACHE_SIZE - 1); - if ((page->index >= end_index + 1) || !offset) { - inode_dec_dirty_dents(inode); + if ((page->index >= end_index + 1) || !offset) goto out; - } zero_user_segment(page, offset, PAGE_CACHE_SIZE); write: @@ -810,7 +816,6 @@ write: /* Dentry blocks are controlled by checkpoint */ if (S_ISDIR(inode->i_mode)) { - inode_dec_dirty_dents(inode); err = do_write_data_page(page, &fio); goto done; } @@ -832,15 +837,16 @@ done: clear_cold_data(page); out: + inode_dec_dirty_dents(inode); unlock_page(page); if (need_balance_fs) f2fs_balance_fs(sbi); + if (wbc->for_reclaim) + f2fs_submit_merged_bio(sbi, DATA, WRITE); return 0; redirty_out: - wbc->pages_skipped++; - account_page_redirty(page); - set_page_dirty(page); + redirty_page_for_writepage(wbc, page); return AOP_WRITEPAGE_ACTIVATE; } @@ -862,12 +868,15 @@ static int f2fs_write_data_pages(struct address_space *mapping, int ret; long diff; + trace_f2fs_writepages(mapping->host, wbc, DATA); + /* deal with chardevs and other special file */ if (!mapping->a_ops->writepage) return 0; if (S_ISDIR(inode->i_mode) && wbc->sync_mode == WB_SYNC_NONE && - get_dirty_dents(inode) < nr_pages_to_skip(sbi, DATA)) + get_dirty_dents(inode) < nr_pages_to_skip(sbi, DATA) && + available_free_memory(sbi, DIRTY_DENTS)) goto skip_write; diff = nr_pages_to_write(sbi, DATA, wbc); @@ -903,6 +912,8 @@ static int f2fs_write_begin(struct file *file, struct address_space *mapping, struct dnode_of_data dn; int err = 0; + trace_f2fs_write_begin(inode, pos, len, flags); + f2fs_balance_fs(sbi); repeat: err = f2fs_convert_inline_data(inode, pos + len); @@ -912,6 +923,10 @@ repeat: page = grab_cache_page_write_begin(mapping, index, flags); if (!page) return -ENOMEM; + + /* to avoid latency during memory pressure */ + unlock_page(page); + *pagep = page; if (f2fs_has_inline_data(inode) && (pos + len) <= MAX_INLINE_DATA) @@ -923,10 +938,18 @@ repeat: f2fs_unlock_op(sbi); if (err) { - f2fs_put_page(page, 1); + f2fs_put_page(page, 0); return err; } inline_data: + lock_page(page); + if (unlikely(page->mapping != mapping)) { + f2fs_put_page(page, 1); + goto repeat; + } + + f2fs_wait_on_page_writeback(page, DATA); + if ((len == PAGE_CACHE_SIZE) || PageUptodate(page)) return 0; @@ -978,6 +1001,8 @@ static int f2fs_write_end(struct file *file, { struct inode *inode = page->mapping->host; + trace_f2fs_write_end(inode, pos, len, copied); + SetPageUptodate(page); set_page_dirty(page); @@ -1022,6 +1047,9 @@ static ssize_t f2fs_direct_IO(int rw, struct kiocb *iocb, if (check_direct_IO(inode, rw, iov, offset, nr_segs)) return 0; + /* clear fsync mark to recover these blocks */ + fsync_mark_clear(F2FS_SB(inode->i_sb), inode->i_ino); + return blockdev_direct_IO(rw, iocb, inode, iov, offset, nr_segs, get_data_block); } @@ -1061,6 +1089,11 @@ static int f2fs_set_data_page_dirty(struct page *page) static sector_t f2fs_bmap(struct address_space *mapping, sector_t block) { + struct inode *inode = mapping->host; + + if (f2fs_has_inline_data(inode)) + return 0; + return generic_block_bmap(mapping, block, get_data_block); } diff --git a/fs/f2fs/dir.c b/fs/f2fs/dir.c index 972fd0ef230f..966acb039e3b 100644 --- a/fs/f2fs/dir.c +++ b/fs/f2fs/dir.c @@ -23,10 +23,10 @@ static unsigned long dir_blocks(struct inode *inode) static unsigned int dir_buckets(unsigned int level, int dir_level) { - if (level < MAX_DIR_HASH_DEPTH / 2) + if (level + dir_level < MAX_DIR_HASH_DEPTH / 2) return 1 << (level + dir_level); else - return 1 << ((MAX_DIR_HASH_DEPTH / 2 + dir_level) - 1); + return MAX_DIR_BUCKETS; } static unsigned int bucket_blocks(unsigned int level) @@ -268,6 +268,8 @@ static void init_dent_inode(const struct qstr *name, struct page *ipage) { struct f2fs_inode *ri; + f2fs_wait_on_page_writeback(ipage, NODE); + /* copy name info. to this inode page */ ri = F2FS_INODE(ipage); ri->i_namelen = cpu_to_le32(name->len); @@ -637,11 +639,17 @@ static int f2fs_readdir(struct file *file, struct dir_context *ctx) struct f2fs_dentry_block *dentry_blk = NULL; struct f2fs_dir_entry *de = NULL; struct page *dentry_page = NULL; + struct file_ra_state *ra = &file->f_ra; unsigned int n = ((unsigned long)ctx->pos / NR_DENTRY_IN_BLOCK); unsigned char d_type = DT_UNKNOWN; bit_pos = ((unsigned long)ctx->pos % NR_DENTRY_IN_BLOCK); + /* readahead for multi pages of dir */ + if (npages - n > 1 && !ra_has_index(ra, n)) + page_cache_sync_readahead(inode->i_mapping, ra, file, n, + min(npages - n, (pgoff_t)MAX_DIR_RA_PAGES)); + for (; n < npages; n++) { dentry_page = get_lock_data_page(inode, n); if (IS_ERR(dentry_page)) diff --git a/fs/f2fs/f2fs.h b/fs/f2fs/f2fs.h index 2ecac8312359..e51c732b0dd9 100644 --- a/fs/f2fs/f2fs.h +++ b/fs/f2fs/f2fs.h @@ -182,6 +182,8 @@ enum { #define F2FS_LINK_MAX 32000 /* maximum link count per file */ +#define MAX_DIR_RA_PAGES 4 /* maximum ra pages of dir */ + /* for in-memory extent cache entry */ #define F2FS_MIN_EXTENT_LEN 16 /* minimum extent length */ @@ -218,6 +220,7 @@ struct f2fs_inode_info { nid_t i_xattr_nid; /* node id that contains xattrs */ unsigned long long xattr_ver; /* cp version of xattr modification */ struct extent_info ext; /* in-memory extent cache entry */ + struct dir_inode_entry *dirty_dir; /* the pointer of dirty dir */ }; static inline void get_extent_info(struct extent_info *ext, @@ -243,6 +246,7 @@ static inline void set_raw_extent(struct extent_info *ext, struct f2fs_nm_info { block_t nat_blkaddr; /* base disk address of NAT */ nid_t max_nid; /* maximum possible node ids */ + nid_t available_nids; /* maximum available node ids */ nid_t next_scan_nid; /* the next nid to be scanned */ unsigned int ram_thresh; /* control the memory footprint */ @@ -323,6 +327,15 @@ struct flush_cmd { int ret; }; +struct flush_cmd_control { + struct task_struct *f2fs_issue_flush; /* flush thread */ + wait_queue_head_t flush_wait_queue; /* waiting queue for wake-up */ + struct flush_cmd *issue_list; /* list for command issue */ + struct flush_cmd *dispatch_list; /* list for command dispatch */ + spinlock_t issue_lock; /* for issue list lock */ + struct flush_cmd *issue_tail; /* list tail of issue list */ +}; + struct f2fs_sm_info { struct sit_info *sit_info; /* whole segment information */ struct free_segmap_info *free_info; /* free segment information */ @@ -353,12 +366,8 @@ struct f2fs_sm_info { unsigned int min_ipu_util; /* in-place-update threshold */ /* for flush command control */ - struct task_struct *f2fs_issue_flush; /* flush thread */ - wait_queue_head_t flush_wait_queue; /* waiting queue for wake-up */ - struct flush_cmd *issue_list; /* list for command issue */ - struct flush_cmd *dispatch_list; /* list for command dispatch */ - spinlock_t issue_lock; /* for issue list lock */ - struct flush_cmd *issue_tail; /* list tail of issue list */ + struct flush_cmd_control *cmd_control_info; + }; /* @@ -755,9 +764,18 @@ static inline unsigned long __bitmap_size(struct f2fs_sb_info *sbi, int flag) static inline void *__bitmap_ptr(struct f2fs_sb_info *sbi, int flag) { struct f2fs_checkpoint *ckpt = F2FS_CKPT(sbi); - int offset = (flag == NAT_BITMAP) ? + int offset; + + if (le32_to_cpu(F2FS_RAW_SUPER(sbi)->cp_payload) > 0) { + if (flag == NAT_BITMAP) + return &ckpt->sit_nat_version_bitmap; + else + return ((unsigned char *)ckpt + F2FS_BLKSIZE); + } else { + offset = (flag == NAT_BITMAP) ? le32_to_cpu(ckpt->sit_ver_bitmap_bytesize) : 0; - return &ckpt->sit_nat_version_bitmap + offset; + return &ckpt->sit_nat_version_bitmap + offset; + } } static inline block_t __start_cp_addr(struct f2fs_sb_info *sbi) @@ -958,6 +976,7 @@ static inline int f2fs_clear_bit(unsigned int nr, char *addr) enum { FI_NEW_INODE, /* indicate newly allocated inode */ FI_DIRTY_INODE, /* indicate inode is dirty or not */ + FI_DIRTY_DIR, /* indicate directory has dirty pages */ FI_INC_LINK, /* need to increment i_nlink */ FI_ACL_MODE, /* indicate acl mode */ FI_NO_ALLOC, /* should not allocate any blocks */ @@ -1071,6 +1090,12 @@ static inline void f2fs_stop_checkpoint(struct f2fs_sb_info *sbi) ((is_inode_flag_set(F2FS_I(i), FI_ACL_MODE)) ? \ (F2FS_I(i)->i_acl_mode) : ((i)->i_mode)) +/* get offset of first page in next direct node */ +#define PGOFS_OF_NEXT_DNODE(pgofs, fi) \ + ((pgofs < ADDRS_PER_INODE(fi)) ? ADDRS_PER_INODE(fi) : \ + (pgofs - ADDRS_PER_INODE(fi) + ADDRS_PER_BLOCK) / \ + ADDRS_PER_BLOCK * ADDRS_PER_BLOCK + ADDRS_PER_INODE(fi)) + /* * file.c */ @@ -1140,8 +1165,10 @@ f2fs_hash_t f2fs_dentry_hash(const char *, size_t); struct dnode_of_data; struct node_info; +bool available_free_memory(struct f2fs_sb_info *, int); int is_checkpointed_node(struct f2fs_sb_info *, nid_t); bool fsync_mark_done(struct f2fs_sb_info *, nid_t); +void fsync_mark_clear(struct f2fs_sb_info *, nid_t); void get_node_info(struct f2fs_sb_info *, nid_t, struct node_info *); int get_dnode_of_data(struct dnode_of_data *, pgoff_t, int); int truncate_inode_blocks(struct inode *, pgoff_t); @@ -1176,9 +1203,12 @@ void destroy_node_manager_caches(void); void f2fs_balance_fs(struct f2fs_sb_info *); void f2fs_balance_fs_bg(struct f2fs_sb_info *); int f2fs_issue_flush(struct f2fs_sb_info *); +int create_flush_cmd_control(struct f2fs_sb_info *); +void destroy_flush_cmd_control(struct f2fs_sb_info *); void invalidate_blocks(struct f2fs_sb_info *, block_t); void refresh_sit_entry(struct f2fs_sb_info *, block_t, block_t); void clear_prefree_segments(struct f2fs_sb_info *); +void discard_next_dnode(struct f2fs_sb_info *); int npages_for_summary_flush(struct f2fs_sb_info *); void allocate_new_segments(struct f2fs_sb_info *); struct page *get_sum_page(struct f2fs_sb_info *, unsigned int); @@ -1221,7 +1251,6 @@ int get_valid_checkpoint(struct f2fs_sb_info *); void set_dirty_dir_page(struct inode *, struct page *); void add_dirty_dir_inode(struct inode *); void remove_dirty_dir_inode(struct inode *); -struct inode *check_dirty_dir_inode(struct f2fs_sb_info *, nid_t); void sync_dirty_dir_inodes(struct f2fs_sb_info *); void write_checkpoint(struct f2fs_sb_info *, bool); void init_orphan_info(struct f2fs_sb_info *); @@ -1242,6 +1271,7 @@ struct page *find_data_page(struct inode *, pgoff_t, bool); struct page *get_lock_data_page(struct inode *, pgoff_t); struct page *get_new_data_page(struct inode *, struct page *, pgoff_t, bool); int do_write_data_page(struct page *, struct f2fs_io_info *); +int f2fs_fiemap(struct inode *inode, struct fiemap_extent_info *, u64, u64); /* * gc.c @@ -1391,5 +1421,6 @@ bool f2fs_may_inline(struct inode *); int f2fs_read_inline_data(struct inode *, struct page *); int f2fs_convert_inline_data(struct inode *, pgoff_t); int f2fs_write_inline_data(struct inode *, struct page *, unsigned int); +void truncate_inline_data(struct inode *, u64); int recover_inline_data(struct inode *, struct page *); #endif diff --git a/fs/f2fs/file.c b/fs/f2fs/file.c index 60e7d5448a1d..9c49c593d8eb 100644 --- a/fs/f2fs/file.c +++ b/fs/f2fs/file.c @@ -19,6 +19,7 @@ #include <linux/compat.h> #include <linux/uaccess.h> #include <linux/mount.h> +#include <linux/pagevec.h> #include "f2fs.h" #include "node.h" @@ -194,6 +195,132 @@ out: return ret; } +static pgoff_t __get_first_dirty_index(struct address_space *mapping, + pgoff_t pgofs, int whence) +{ + struct pagevec pvec; + int nr_pages; + + if (whence != SEEK_DATA) + return 0; + + /* find first dirty page index */ + pagevec_init(&pvec, 0); + nr_pages = pagevec_lookup_tag(&pvec, mapping, &pgofs, PAGECACHE_TAG_DIRTY, 1); + pgofs = nr_pages ? pvec.pages[0]->index: LONG_MAX; + pagevec_release(&pvec); + return pgofs; +} + +static bool __found_offset(block_t blkaddr, pgoff_t dirty, pgoff_t pgofs, + int whence) +{ + switch (whence) { + case SEEK_DATA: + if ((blkaddr == NEW_ADDR && dirty == pgofs) || + (blkaddr != NEW_ADDR && blkaddr != NULL_ADDR)) + return true; + break; + case SEEK_HOLE: + if (blkaddr == NULL_ADDR) + return true; + break; + } + return false; +} + +static loff_t f2fs_seek_block(struct file *file, loff_t offset, int whence) +{ + struct inode *inode = file->f_mapping->host; + loff_t maxbytes = inode->i_sb->s_maxbytes; + struct dnode_of_data dn; + pgoff_t pgofs, end_offset, dirty; + loff_t data_ofs = offset; + loff_t isize; + int err = 0; + + mutex_lock(&inode->i_mutex); + + isize = i_size_read(inode); + if (offset >= isize) + goto fail; + + /* handle inline data case */ + if (f2fs_has_inline_data(inode)) { + if (whence == SEEK_HOLE) + data_ofs = isize; + goto found; + } + + pgofs = (pgoff_t)(offset >> PAGE_CACHE_SHIFT); + + dirty = __get_first_dirty_index(inode->i_mapping, pgofs, whence); + + for (; data_ofs < isize; data_ofs = pgofs << PAGE_CACHE_SHIFT) { + set_new_dnode(&dn, inode, NULL, NULL, 0); + err = get_dnode_of_data(&dn, pgofs, LOOKUP_NODE_RA); + if (err && err != -ENOENT) { + goto fail; + } else if (err == -ENOENT) { + /* direct node is not exist */ + if (whence == SEEK_DATA) { + pgofs = PGOFS_OF_NEXT_DNODE(pgofs, + F2FS_I(inode)); + continue; + } else { + goto found; + } + } + + end_offset = IS_INODE(dn.node_page) ? + ADDRS_PER_INODE(F2FS_I(inode)) : ADDRS_PER_BLOCK; + + /* find data/hole in dnode block */ + for (; dn.ofs_in_node < end_offset; + dn.ofs_in_node++, pgofs++, + data_ofs = pgofs << PAGE_CACHE_SHIFT) { + block_t blkaddr; + blkaddr = datablock_addr(dn.node_page, dn.ofs_in_node); + + if (__found_offset(blkaddr, dirty, pgofs, whence)) { + f2fs_put_dnode(&dn); + goto found; + } + } + f2fs_put_dnode(&dn); + } + + if (whence == SEEK_DATA) + goto fail; +found: + if (whence == SEEK_HOLE && data_ofs > isize) + data_ofs = isize; + mutex_unlock(&inode->i_mutex); + return vfs_setpos(file, data_ofs, maxbytes); +fail: + mutex_unlock(&inode->i_mutex); + return -ENXIO; +} + +static loff_t f2fs_llseek(struct file *file, loff_t offset, int whence) +{ + struct inode *inode = file->f_mapping->host; + loff_t maxbytes = inode->i_sb->s_maxbytes; + + switch (whence) { + case SEEK_SET: + case SEEK_CUR: + case SEEK_END: + return generic_file_llseek_size(file, offset, whence, + maxbytes, i_size_read(inode)); + case SEEK_DATA: + case SEEK_HOLE: + return f2fs_seek_block(file, offset, whence); + } + + return -EINVAL; +} + static int f2fs_file_mmap(struct file *file, struct vm_area_struct *vma) { file_accessed(file); @@ -242,6 +369,9 @@ static void truncate_partial_data_page(struct inode *inode, u64 from) unsigned offset = from & (PAGE_CACHE_SIZE - 1); struct page *page; + if (f2fs_has_inline_data(inode)) + return truncate_inline_data(inode, from); + if (!offset) return; @@ -288,10 +418,7 @@ int truncate_blocks(struct inode *inode, u64 from) return err; } - if (IS_INODE(dn.node_page)) - count = ADDRS_PER_INODE(F2FS_I(inode)); - else - count = ADDRS_PER_BLOCK; + count = ADDRS_PER_PAGE(dn.node_page, F2FS_I(inode)); count -= dn.ofs_in_node; f2fs_bug_on(count < 0); @@ -413,6 +540,7 @@ const struct inode_operations f2fs_file_inode_operations = { .listxattr = f2fs_listxattr, .removexattr = generic_removexattr, #endif + .fiemap = f2fs_fiemap, }; static void fill_zero(struct inode *inode, pgoff_t index, @@ -555,6 +683,7 @@ static int expand_inode_data(struct inode *inode, loff_t offset, i_size_read(inode) < new_size) { i_size_write(inode, new_size); mark_inode_dirty(inode); + f2fs_write_inode(inode, NULL); } return ret; @@ -678,7 +807,7 @@ long f2fs_compat_ioctl(struct file *file, unsigned int cmd, unsigned long arg) #endif const struct file_operations f2fs_file_operations = { - .llseek = generic_file_llseek, + .llseek = f2fs_llseek, .read = do_sync_read, .write = do_sync_write, .aio_read = generic_file_aio_read, diff --git a/fs/f2fs/inline.c b/fs/f2fs/inline.c index 383db1fabcf4..1bba5228c197 100644 --- a/fs/f2fs/inline.c +++ b/fs/f2fs/inline.c @@ -81,8 +81,10 @@ static int __f2fs_convert_inline_data(struct inode *inode, struct page *page) f2fs_lock_op(sbi); ipage = get_node_page(sbi, inode->i_ino); - if (IS_ERR(ipage)) - return PTR_ERR(ipage); + if (IS_ERR(ipage)) { + err = PTR_ERR(ipage); + goto out; + } /* * i_addr[0] is not used for inline data, @@ -90,11 +92,10 @@ static int __f2fs_convert_inline_data(struct inode *inode, struct page *page) */ set_new_dnode(&dn, inode, ipage, NULL, 0); err = f2fs_reserve_block(&dn, 0); - if (err) { - f2fs_unlock_op(sbi); - return err; - } + if (err) + goto out; + f2fs_wait_on_page_writeback(page, DATA); zero_user_segment(page, MAX_INLINE_DATA, PAGE_CACHE_SIZE); /* Copy the whole inline data block */ @@ -118,6 +119,7 @@ static int __f2fs_convert_inline_data(struct inode *inode, struct page *page) sync_inode_page(&dn); f2fs_put_dnode(&dn); +out: f2fs_unlock_op(sbi); return err; } @@ -132,7 +134,7 @@ int f2fs_convert_inline_data(struct inode *inode, pgoff_t to_size) else if (to_size <= MAX_INLINE_DATA) return 0; - page = grab_cache_page_write_begin(inode->i_mapping, 0, AOP_FLAG_NOFS); + page = grab_cache_page(inode->i_mapping, 0); if (!page) return -ENOMEM; @@ -155,6 +157,7 @@ int f2fs_write_inline_data(struct inode *inode, return err; ipage = dn.inode_page; + f2fs_wait_on_page_writeback(ipage, NODE); zero_user_segment(ipage, INLINE_DATA_OFFSET, INLINE_DATA_OFFSET + MAX_INLINE_DATA); src_addr = kmap(page); @@ -175,6 +178,26 @@ int f2fs_write_inline_data(struct inode *inode, return 0; } +void truncate_inline_data(struct inode *inode, u64 from) +{ + struct f2fs_sb_info *sbi = F2FS_SB(inode->i_sb); + struct page *ipage; + + if (from >= MAX_INLINE_DATA) + return; + + ipage = get_node_page(sbi, inode->i_ino); + if (IS_ERR(ipage)) + return; + + f2fs_wait_on_page_writeback(ipage, NODE); + + zero_user_segment(ipage, INLINE_DATA_OFFSET + from, + INLINE_DATA_OFFSET + MAX_INLINE_DATA); + set_page_dirty(ipage); + f2fs_put_page(ipage, 1); +} + int recover_inline_data(struct inode *inode, struct page *npage) { struct f2fs_sb_info *sbi = F2FS_SB(inode->i_sb); @@ -199,6 +222,8 @@ process_inline: ipage = get_node_page(sbi, inode->i_ino); f2fs_bug_on(IS_ERR(ipage)); + f2fs_wait_on_page_writeback(ipage, NODE); + src_addr = inline_data_addr(npage); dst_addr = inline_data_addr(ipage); memcpy(dst_addr, src_addr, MAX_INLINE_DATA); @@ -210,6 +235,7 @@ process_inline: if (f2fs_has_inline_data(inode)) { ipage = get_node_page(sbi, inode->i_ino); f2fs_bug_on(IS_ERR(ipage)); + f2fs_wait_on_page_writeback(ipage, NODE); zero_user_segment(ipage, INLINE_DATA_OFFSET, INLINE_DATA_OFFSET + MAX_INLINE_DATA); clear_inode_flag(F2FS_I(inode), FI_INLINE_DATA); diff --git a/fs/f2fs/inode.c b/fs/f2fs/inode.c index ee829d360468..adc622c6bdce 100644 --- a/fs/f2fs/inode.c +++ b/fs/f2fs/inode.c @@ -12,6 +12,7 @@ #include <linux/f2fs_fs.h> #include <linux/buffer_head.h> #include <linux/writeback.h> +#include <linux/bitops.h> #include "f2fs.h" #include "node.h" @@ -21,20 +22,20 @@ void f2fs_set_inode_flags(struct inode *inode) { unsigned int flags = F2FS_I(inode)->i_flags; - - inode->i_flags &= ~(S_SYNC | S_APPEND | S_IMMUTABLE | - S_NOATIME | S_DIRSYNC); + unsigned int new_fl = 0; if (flags & FS_SYNC_FL) - inode->i_flags |= S_SYNC; + new_fl |= S_SYNC; if (flags & FS_APPEND_FL) - inode->i_flags |= S_APPEND; + new_fl |= S_APPEND; if (flags & FS_IMMUTABLE_FL) - inode->i_flags |= S_IMMUTABLE; + new_fl |= S_IMMUTABLE; if (flags & FS_NOATIME_FL) - inode->i_flags |= S_NOATIME; + new_fl |= S_NOATIME; if (flags & FS_DIRSYNC_FL) - inode->i_flags |= S_DIRSYNC; + new_fl |= S_DIRSYNC; + set_mask_bits(&inode->i_flags, + S_SYNC|S_APPEND|S_IMMUTABLE|S_NOATIME|S_DIRSYNC, new_fl); } static void __get_inode_rdev(struct inode *inode, struct f2fs_inode *ri) @@ -294,4 +295,5 @@ void f2fs_evict_inode(struct inode *inode) sb_end_intwrite(inode->i_sb); no_delete: clear_inode(inode); + invalidate_mapping_pages(NODE_MAPPING(sbi), inode->i_ino, inode->i_ino); } diff --git a/fs/f2fs/namei.c b/fs/f2fs/namei.c index a9409d19dfd4..9138c32aa698 100644 --- a/fs/f2fs/namei.c +++ b/fs/f2fs/namei.c @@ -41,18 +41,9 @@ static struct inode *f2fs_new_inode(struct inode *dir, umode_t mode) } f2fs_unlock_op(sbi); - inode->i_uid = current_fsuid(); - - if (dir->i_mode & S_ISGID) { - inode->i_gid = dir->i_gid; - if (S_ISDIR(mode)) - mode |= S_ISGID; - } else { - inode->i_gid = current_fsgid(); - } + inode_init_owner(inode, dir, mode); inode->i_ino = ino; - inode->i_mode = mode; inode->i_blocks = 0; inode->i_mtime = inode->i_atime = inode->i_ctime = CURRENT_TIME; inode->i_generation = sbi->s_next_generation++; diff --git a/fs/f2fs/node.c b/fs/f2fs/node.c index 57caa6eaf47b..9dfb9a042fd2 100644 --- a/fs/f2fs/node.c +++ b/fs/f2fs/node.c @@ -26,20 +26,26 @@ static struct kmem_cache *nat_entry_slab; static struct kmem_cache *free_nid_slab; -static inline bool available_free_memory(struct f2fs_nm_info *nm_i, int type) +bool available_free_memory(struct f2fs_sb_info *sbi, int type) { + struct f2fs_nm_info *nm_i = NM_I(sbi); struct sysinfo val; unsigned long mem_size = 0; + bool res = false; si_meminfo(&val); - if (type == FREE_NIDS) - mem_size = nm_i->fcnt * sizeof(struct free_nid); - else if (type == NAT_ENTRIES) - mem_size += nm_i->nat_cnt * sizeof(struct nat_entry); - mem_size >>= 12; - - /* give 50:50 memory for free nids and nat caches respectively */ - return (mem_size < ((val.totalram * nm_i->ram_thresh) >> 11)); + /* give 25%, 25%, 50% memory for each components respectively */ + if (type == FREE_NIDS) { + mem_size = (nm_i->fcnt * sizeof(struct free_nid)) >> 12; + res = mem_size < ((val.totalram * nm_i->ram_thresh / 100) >> 2); + } else if (type == NAT_ENTRIES) { + mem_size = (nm_i->nat_cnt * sizeof(struct nat_entry)) >> 12; + res = mem_size < ((val.totalram * nm_i->ram_thresh / 100) >> 2); + } else if (type == DIRTY_DENTS) { + mem_size = get_pages(sbi, F2FS_DIRTY_DENTS); + res = mem_size < ((val.totalram * nm_i->ram_thresh / 100) >> 1); + } + return res; } static void clear_node_page_dirty(struct page *page) @@ -147,6 +153,18 @@ bool fsync_mark_done(struct f2fs_sb_info *sbi, nid_t nid) return fsync_done; } +void fsync_mark_clear(struct f2fs_sb_info *sbi, nid_t nid) +{ + struct f2fs_nm_info *nm_i = NM_I(sbi); + struct nat_entry *e; + + write_lock(&nm_i->nat_tree_lock); + e = __lookup_nat_cache(nm_i, nid); + if (e) + e->fsync_done = false; + write_unlock(&nm_i->nat_tree_lock); +} + static struct nat_entry *grab_nat_entry(struct f2fs_nm_info *nm_i, nid_t nid) { struct nat_entry *new; @@ -179,9 +197,7 @@ retry: write_unlock(&nm_i->nat_tree_lock); goto retry; } - nat_set_blkaddr(e, le32_to_cpu(ne->block_addr)); - nat_set_ino(e, le32_to_cpu(ne->ino)); - nat_set_version(e, ne->version); + node_info_from_raw_nat(&e->ni, ne); } write_unlock(&nm_i->nat_tree_lock); } @@ -243,7 +259,7 @@ int try_to_free_nats(struct f2fs_sb_info *sbi, int nr_shrink) { struct f2fs_nm_info *nm_i = NM_I(sbi); - if (available_free_memory(nm_i, NAT_ENTRIES)) + if (available_free_memory(sbi, NAT_ENTRIES)) return 0; write_lock(&nm_i->nat_tree_lock); @@ -849,8 +865,7 @@ struct page *new_node_page(struct dnode_of_data *dn, if (unlikely(is_inode_flag_set(F2FS_I(dn->inode), FI_NO_ALLOC))) return ERR_PTR(-EPERM); - page = grab_cache_page_write_begin(NODE_MAPPING(sbi), - dn->nid, AOP_FLAG_NOFS); + page = grab_cache_page(NODE_MAPPING(sbi), dn->nid); if (!page) return ERR_PTR(-ENOMEM); @@ -867,6 +882,7 @@ struct page *new_node_page(struct dnode_of_data *dn, new_ni.ino = dn->inode->i_ino; set_node_addr(sbi, &new_ni, NEW_ADDR, false); + f2fs_wait_on_page_writeback(page, NODE); fill_node_footer(page, dn->nid, dn->inode->i_ino, ofs, true); set_cold_node(dn->inode, page); SetPageUptodate(page); @@ -946,8 +962,7 @@ struct page *get_node_page(struct f2fs_sb_info *sbi, pgoff_t nid) struct page *page; int err; repeat: - page = grab_cache_page_write_begin(NODE_MAPPING(sbi), - nid, AOP_FLAG_NOFS); + page = grab_cache_page(NODE_MAPPING(sbi), nid); if (!page) return ERR_PTR(-ENOMEM); @@ -1194,6 +1209,8 @@ static int f2fs_write_node_page(struct page *page, .rw = (wbc->sync_mode == WB_SYNC_ALL) ? WRITE_SYNC : WRITE, }; + trace_f2fs_writepage(page, NODE); + if (unlikely(sbi->por_doing)) goto redirty_out; @@ -1225,10 +1242,7 @@ static int f2fs_write_node_page(struct page *page, return 0; redirty_out: - dec_page_count(sbi, F2FS_DIRTY_NODES); - wbc->pages_skipped++; - account_page_redirty(page); - set_page_dirty(page); + redirty_page_for_writepage(wbc, page); return AOP_WRITEPAGE_ACTIVATE; } @@ -1238,6 +1252,8 @@ static int f2fs_write_node_pages(struct address_space *mapping, struct f2fs_sb_info *sbi = F2FS_SB(mapping->host->i_sb); long diff; + trace_f2fs_writepages(mapping->host, wbc, NODE); + /* balancing f2fs's metadata in background */ f2fs_balance_fs_bg(sbi); @@ -1313,13 +1329,14 @@ static void __del_from_free_nid_list(struct f2fs_nm_info *nm_i, radix_tree_delete(&nm_i->free_nid_root, i->nid); } -static int add_free_nid(struct f2fs_nm_info *nm_i, nid_t nid, bool build) +static int add_free_nid(struct f2fs_sb_info *sbi, nid_t nid, bool build) { + struct f2fs_nm_info *nm_i = NM_I(sbi); struct free_nid *i; struct nat_entry *ne; bool allocated = false; - if (!available_free_memory(nm_i, FREE_NIDS)) + if (!available_free_memory(sbi, FREE_NIDS)) return -1; /* 0 nid should not be used */ @@ -1372,9 +1389,10 @@ static void remove_free_nid(struct f2fs_nm_info *nm_i, nid_t nid) kmem_cache_free(free_nid_slab, i); } -static void scan_nat_page(struct f2fs_nm_info *nm_i, +static void scan_nat_page(struct f2fs_sb_info *sbi, struct page *nat_page, nid_t start_nid) { + struct f2fs_nm_info *nm_i = NM_I(sbi); struct f2fs_nat_block *nat_blk = page_address(nat_page); block_t blk_addr; int i; @@ -1389,7 +1407,7 @@ static void scan_nat_page(struct f2fs_nm_info *nm_i, blk_addr = le32_to_cpu(nat_blk->entries[i].block_addr); f2fs_bug_on(blk_addr == NEW_ADDR); if (blk_addr == NULL_ADDR) { - if (add_free_nid(nm_i, start_nid, true) < 0) + if (add_free_nid(sbi, start_nid, true) < 0) break; } } @@ -1413,7 +1431,7 @@ static void build_free_nids(struct f2fs_sb_info *sbi) while (1) { struct page *page = get_current_nat_page(sbi, nid); - scan_nat_page(nm_i, page, nid); + scan_nat_page(sbi, page, nid); f2fs_put_page(page, 1); nid += (NAT_ENTRY_PER_BLOCK - (nid % NAT_ENTRY_PER_BLOCK)); @@ -1433,7 +1451,7 @@ static void build_free_nids(struct f2fs_sb_info *sbi) block_t addr = le32_to_cpu(nat_in_journal(sum, i).block_addr); nid = le32_to_cpu(nid_in_journal(sum, i)); if (addr == NULL_ADDR) - add_free_nid(nm_i, nid, true); + add_free_nid(sbi, nid, true); else remove_free_nid(nm_i, nid); } @@ -1450,7 +1468,7 @@ bool alloc_nid(struct f2fs_sb_info *sbi, nid_t *nid) struct f2fs_nm_info *nm_i = NM_I(sbi); struct free_nid *i = NULL; retry: - if (unlikely(sbi->total_valid_node_count + 1 >= nm_i->max_nid)) + if (unlikely(sbi->total_valid_node_count + 1 > nm_i->available_nids)) return false; spin_lock(&nm_i->free_nid_list_lock); @@ -1510,7 +1528,7 @@ void alloc_nid_failed(struct f2fs_sb_info *sbi, nid_t nid) spin_lock(&nm_i->free_nid_list_lock); i = __lookup_free_nid_list(nm_i, nid); f2fs_bug_on(!i || i->state != NID_ALLOC); - if (!available_free_memory(nm_i, FREE_NIDS)) { + if (!available_free_memory(sbi, FREE_NIDS)) { __del_from_free_nid_list(nm_i, i); need_free = true; } else { @@ -1532,7 +1550,7 @@ void recover_node_page(struct f2fs_sb_info *sbi, struct page *page, clear_node_page_dirty(page); } -void recover_inline_xattr(struct inode *inode, struct page *page) +static void recover_inline_xattr(struct inode *inode, struct page *page) { struct f2fs_sb_info *sbi = F2FS_SB(inode->i_sb); void *src_addr, *dst_addr; @@ -1557,6 +1575,7 @@ void recover_inline_xattr(struct inode *inode, struct page *page) src_addr = inline_xattr_addr(page); inline_size = inline_xattr_size(inode); + f2fs_wait_on_page_writeback(ipage, NODE); memcpy(dst_addr, src_addr, inline_size); update_inode(inode, ipage); @@ -1612,6 +1631,11 @@ int recover_inode_page(struct f2fs_sb_info *sbi, struct page *page) struct node_info old_ni, new_ni; struct page *ipage; + get_node_info(sbi, ino, &old_ni); + + if (unlikely(old_ni.blk_addr != NULL_ADDR)) + return -EINVAL; + ipage = grab_cache_page(NODE_MAPPING(sbi), ino); if (!ipage) return -ENOMEM; @@ -1619,7 +1643,6 @@ int recover_inode_page(struct f2fs_sb_info *sbi, struct page *page) /* Should not use this inode from free nid list */ remove_free_nid(NM_I(sbi), ino); - get_node_info(sbi, ino, &old_ni); SetPageUptodate(ipage); fill_node_footer(ipage, ino, ino, 0, true); @@ -1645,35 +1668,29 @@ int recover_inode_page(struct f2fs_sb_info *sbi, struct page *page) /* * ra_sum_pages() merge contiguous pages into one bio and submit. - * these pre-readed pages are linked in pages list. + * these pre-readed pages are alloced in bd_inode's mapping tree. */ -static int ra_sum_pages(struct f2fs_sb_info *sbi, struct list_head *pages, +static int ra_sum_pages(struct f2fs_sb_info *sbi, struct page **pages, int start, int nrpages) { - struct page *page; - int page_idx = start; + struct inode *inode = sbi->sb->s_bdev->bd_inode; + struct address_space *mapping = inode->i_mapping; + int i, page_idx = start; struct f2fs_io_info fio = { .type = META, .rw = READ_SYNC | REQ_META | REQ_PRIO }; - for (; page_idx < start + nrpages; page_idx++) { - /* alloc temporal page for read node summary info*/ - page = alloc_page(GFP_F2FS_ZERO); - if (!page) + for (i = 0; page_idx < start + nrpages; page_idx++, i++) { + /* alloc page in bd_inode for reading node summary info */ + pages[i] = grab_cache_page(mapping, page_idx); + if (!pages[i]) break; - - lock_page(page); - page->index = page_idx; - list_add_tail(&page->lru, pages); + f2fs_submit_page_mbio(sbi, pages[i], page_idx, &fio); } - list_for_each_entry(page, pages, lru) - f2fs_submit_page_mbio(sbi, page, page->index, &fio); - f2fs_submit_merged_bio(sbi, META, READ); - - return page_idx - start; + return i; } int restore_node_summary(struct f2fs_sb_info *sbi, @@ -1681,11 +1698,11 @@ int restore_node_summary(struct f2fs_sb_info *sbi, { struct f2fs_node *rn; struct f2fs_summary *sum_entry; - struct page *page, *tmp; + struct inode *inode = sbi->sb->s_bdev->bd_inode; block_t addr; int bio_blocks = MAX_BIO_BLOCKS(max_hw_blocks(sbi)); - int i, last_offset, nrpages, err = 0; - LIST_HEAD(page_list); + struct page *pages[bio_blocks]; + int i, idx, last_offset, nrpages, err = 0; /* scan the node segment */ last_offset = sbi->blocks_per_seg; @@ -1696,29 +1713,31 @@ int restore_node_summary(struct f2fs_sb_info *sbi, nrpages = min(last_offset - i, bio_blocks); /* read ahead node pages */ - nrpages = ra_sum_pages(sbi, &page_list, addr, nrpages); + nrpages = ra_sum_pages(sbi, pages, addr, nrpages); if (!nrpages) return -ENOMEM; - list_for_each_entry_safe(page, tmp, &page_list, lru) { + for (idx = 0; idx < nrpages; idx++) { if (err) goto skip; - lock_page(page); - if (unlikely(!PageUptodate(page))) { + lock_page(pages[idx]); + if (unlikely(!PageUptodate(pages[idx]))) { err = -EIO; } else { - rn = F2FS_NODE(page); + rn = F2FS_NODE(pages[idx]); sum_entry->nid = rn->footer.nid; sum_entry->version = 0; sum_entry->ofs_in_node = 0; sum_entry++; } - unlock_page(page); + unlock_page(pages[idx]); skip: - list_del(&page->lru); - __free_pages(page, 0); + page_cache_release(pages[idx]); } + + invalidate_mapping_pages(inode->i_mapping, addr, + addr + nrpages); } return err; } @@ -1756,9 +1775,7 @@ retry: write_unlock(&nm_i->nat_tree_lock); goto retry; } - nat_set_blkaddr(ne, le32_to_cpu(raw_ne.block_addr)); - nat_set_ino(ne, le32_to_cpu(raw_ne.ino)); - nat_set_version(ne, raw_ne.version); + node_info_from_raw_nat(&ne->ni, &raw_ne); __set_nat_cache_dirty(nm_i, ne); write_unlock(&nm_i->nat_tree_lock); } @@ -1791,7 +1808,6 @@ void flush_nat_entries(struct f2fs_sb_info *sbi) nid_t nid; struct f2fs_nat_entry raw_ne; int offset = -1; - block_t new_blkaddr; if (nat_get_blkaddr(ne) == NEW_ADDR) continue; @@ -1827,11 +1843,7 @@ to_nat_page: f2fs_bug_on(!nat_blk); raw_ne = nat_blk->entries[nid - start_nid]; flush_now: - new_blkaddr = nat_get_blkaddr(ne); - - raw_ne.ino = cpu_to_le32(nat_get_ino(ne)); - raw_ne.block_addr = cpu_to_le32(new_blkaddr); - raw_ne.version = nat_get_version(ne); + raw_nat_from_node_info(&raw_ne, &ne->ni); if (offset < 0) { nat_blk->entries[nid - start_nid] = raw_ne; @@ -1841,7 +1853,7 @@ flush_now: } if (nat_get_blkaddr(ne) == NULL_ADDR && - add_free_nid(NM_I(sbi), nid, false) <= 0) { + add_free_nid(sbi, nid, false) <= 0) { write_lock(&nm_i->nat_tree_lock); __del_from_nat_cache(nm_i, ne); write_unlock(&nm_i->nat_tree_lock); @@ -1869,8 +1881,10 @@ static int init_node_manager(struct f2fs_sb_info *sbi) nat_segs = le32_to_cpu(sb_raw->segment_count_nat) >> 1; nat_blocks = nat_segs << le32_to_cpu(sb_raw->log_blocks_per_seg); + nm_i->max_nid = NAT_ENTRY_PER_BLOCK * nat_blocks; + /* not used nids: 0, node, meta, (and root counted as valid node) */ - nm_i->max_nid = NAT_ENTRY_PER_BLOCK * nat_blocks - 3; + nm_i->available_nids = nm_i->max_nid - 3; nm_i->fcnt = 0; nm_i->nat_cnt = 0; nm_i->ram_thresh = DEF_RAM_THRESHOLD; diff --git a/fs/f2fs/node.h b/fs/f2fs/node.h index 5decc1a375f0..7281112cd1c8 100644 --- a/fs/f2fs/node.h +++ b/fs/f2fs/node.h @@ -59,12 +59,12 @@ struct nat_entry { do { \ ne->checkpointed = false; \ list_move_tail(&ne->list, &nm_i->dirty_nat_entries); \ - } while (0); + } while (0) #define __clear_nat_cache_dirty(nm_i, ne) \ do { \ ne->checkpointed = true; \ list_move_tail(&ne->list, &nm_i->nat_entries); \ - } while (0); + } while (0) #define inc_node_version(version) (++version) static inline void node_info_from_raw_nat(struct node_info *ni, @@ -75,9 +75,18 @@ static inline void node_info_from_raw_nat(struct node_info *ni, ni->version = raw_ne->version; } -enum nid_type { +static inline void raw_nat_from_node_info(struct f2fs_nat_entry *raw_ne, + struct node_info *ni) +{ + raw_ne->ino = cpu_to_le32(ni->ino); + raw_ne->block_addr = cpu_to_le32(ni->blk_addr); + raw_ne->version = ni->version; +} + +enum mem_type { FREE_NIDS, /* indicates the free nid list */ - NAT_ENTRIES /* indicates the cached nat entry */ + NAT_ENTRIES, /* indicates the cached nat entry */ + DIRTY_DENTS /* indicates dirty dentry pages */ }; /* @@ -263,7 +272,7 @@ static inline void set_nid(struct page *p, int off, nid_t nid, bool i) { struct f2fs_node *rn = F2FS_NODE(p); - wait_on_page_writeback(p); + f2fs_wait_on_page_writeback(p, NODE); if (i) rn->i.i_nid[off - NODE_DIR1_BLOCK] = cpu_to_le32(nid); diff --git a/fs/f2fs/recovery.c b/fs/f2fs/recovery.c index b1ae89f0f44e..a112368a4a86 100644 --- a/fs/f2fs/recovery.c +++ b/fs/f2fs/recovery.c @@ -46,15 +46,10 @@ static int recover_dentry(struct page *ipage, struct inode *inode) struct inode *dir, *einode; int err = 0; - dir = check_dirty_dir_inode(F2FS_SB(inode->i_sb), pino); - if (!dir) { - dir = f2fs_iget(inode->i_sb, pino); - if (IS_ERR(dir)) { - err = PTR_ERR(dir); - goto out; - } - set_inode_flag(F2FS_I(dir), FI_DELAY_IPUT); - add_dirty_dir_inode(dir); + dir = f2fs_iget(inode->i_sb, pino); + if (IS_ERR(dir)) { + err = PTR_ERR(dir); + goto out; } name.len = le32_to_cpu(raw_inode->i_namelen); @@ -63,7 +58,7 @@ static int recover_dentry(struct page *ipage, struct inode *inode) if (unlikely(name.len > F2FS_NAME_LEN)) { WARN_ON(1); err = -ENAMETOOLONG; - goto out; + goto out_err; } retry: de = f2fs_find_entry(dir, &name, &page); @@ -73,7 +68,8 @@ retry: einode = f2fs_iget(inode->i_sb, le32_to_cpu(de->ino)); if (IS_ERR(einode)) { WARN_ON(1); - if (PTR_ERR(einode) == -ENOENT) + err = PTR_ERR(einode); + if (err == -ENOENT) err = -EEXIST; goto out_unmap_put; } @@ -87,11 +83,23 @@ retry: goto retry; } err = __f2fs_add_link(dir, &name, inode); + if (err) + goto out_err; + + if (is_inode_flag_set(F2FS_I(dir), FI_DELAY_IPUT)) { + iput(dir); + } else { + add_dirty_dir_inode(dir); + set_inode_flag(F2FS_I(dir), FI_DELAY_IPUT); + } + goto out; out_unmap_put: kunmap(page); f2fs_put_page(page, 0); +out_err: + iput(dir); out: f2fs_msg(inode->i_sb, KERN_NOTICE, "%s: ino = %x, name = %s, dir = %lx, err = %d", @@ -299,10 +307,7 @@ static int do_recover_data(struct f2fs_sb_info *sbi, struct inode *inode, goto out; start = start_bidx_of_node(ofs_of_node(page), fi); - if (IS_INODE(page)) - end = start + ADDRS_PER_INODE(fi); - else - end = start + ADDRS_PER_BLOCK; + end = start + ADDRS_PER_PAGE(page, fi); f2fs_lock_op(sbi); diff --git a/fs/f2fs/segment.c b/fs/f2fs/segment.c index 085f548be7a3..f25f0e07e26f 100644 --- a/fs/f2fs/segment.c +++ b/fs/f2fs/segment.c @@ -25,7 +25,6 @@ #define __reverse_ffz(x) __reverse_ffs(~(x)) static struct kmem_cache *discard_entry_slab; -static struct kmem_cache *flush_cmd_slab; /* * __reverse_ffs is copied from include/asm-generic/bitops/__ffs.h since @@ -200,20 +199,20 @@ void f2fs_balance_fs_bg(struct f2fs_sb_info *sbi) static int issue_flush_thread(void *data) { struct f2fs_sb_info *sbi = data; - struct f2fs_sm_info *sm_i = SM_I(sbi); - wait_queue_head_t *q = &sm_i->flush_wait_queue; + struct flush_cmd_control *fcc = SM_I(sbi)->cmd_control_info; + wait_queue_head_t *q = &fcc->flush_wait_queue; repeat: if (kthread_should_stop()) return 0; - spin_lock(&sm_i->issue_lock); - if (sm_i->issue_list) { - sm_i->dispatch_list = sm_i->issue_list; - sm_i->issue_list = sm_i->issue_tail = NULL; + spin_lock(&fcc->issue_lock); + if (fcc->issue_list) { + fcc->dispatch_list = fcc->issue_list; + fcc->issue_list = fcc->issue_tail = NULL; } - spin_unlock(&sm_i->issue_lock); + spin_unlock(&fcc->issue_lock); - if (sm_i->dispatch_list) { + if (fcc->dispatch_list) { struct bio *bio = bio_alloc(GFP_NOIO, 0); struct flush_cmd *cmd, *next; int ret; @@ -221,47 +220,79 @@ repeat: bio->bi_bdev = sbi->sb->s_bdev; ret = submit_bio_wait(WRITE_FLUSH, bio); - for (cmd = sm_i->dispatch_list; cmd; cmd = next) { + for (cmd = fcc->dispatch_list; cmd; cmd = next) { cmd->ret = ret; next = cmd->next; complete(&cmd->wait); } - sm_i->dispatch_list = NULL; + bio_put(bio); + fcc->dispatch_list = NULL; } - wait_event_interruptible(*q, kthread_should_stop() || sm_i->issue_list); + wait_event_interruptible(*q, + kthread_should_stop() || fcc->issue_list); goto repeat; } int f2fs_issue_flush(struct f2fs_sb_info *sbi) { - struct f2fs_sm_info *sm_i = SM_I(sbi); - struct flush_cmd *cmd; - int ret; + struct flush_cmd_control *fcc = SM_I(sbi)->cmd_control_info; + struct flush_cmd cmd; if (!test_opt(sbi, FLUSH_MERGE)) return blkdev_issue_flush(sbi->sb->s_bdev, GFP_KERNEL, NULL); - cmd = f2fs_kmem_cache_alloc(flush_cmd_slab, GFP_ATOMIC); - cmd->next = NULL; - cmd->ret = 0; - init_completion(&cmd->wait); + init_completion(&cmd.wait); + cmd.next = NULL; - spin_lock(&sm_i->issue_lock); - if (sm_i->issue_list) - sm_i->issue_tail->next = cmd; + spin_lock(&fcc->issue_lock); + if (fcc->issue_list) + fcc->issue_tail->next = &cmd; else - sm_i->issue_list = cmd; - sm_i->issue_tail = cmd; - spin_unlock(&sm_i->issue_lock); + fcc->issue_list = &cmd; + fcc->issue_tail = &cmd; + spin_unlock(&fcc->issue_lock); - if (!sm_i->dispatch_list) - wake_up(&sm_i->flush_wait_queue); + if (!fcc->dispatch_list) + wake_up(&fcc->flush_wait_queue); - wait_for_completion(&cmd->wait); - ret = cmd->ret; - kmem_cache_free(flush_cmd_slab, cmd); - return ret; + wait_for_completion(&cmd.wait); + + return cmd.ret; +} + +int create_flush_cmd_control(struct f2fs_sb_info *sbi) +{ + dev_t dev = sbi->sb->s_bdev->bd_dev; + struct flush_cmd_control *fcc; + int err = 0; + + fcc = kzalloc(sizeof(struct flush_cmd_control), GFP_KERNEL); + if (!fcc) + return -ENOMEM; + spin_lock_init(&fcc->issue_lock); + init_waitqueue_head(&fcc->flush_wait_queue); + fcc->f2fs_issue_flush = kthread_run(issue_flush_thread, sbi, + "f2fs_flush-%u:%u", MAJOR(dev), MINOR(dev)); + if (IS_ERR(fcc->f2fs_issue_flush)) { + err = PTR_ERR(fcc->f2fs_issue_flush); + kfree(fcc); + return err; + } + sbi->sm_info->cmd_control_info = fcc; + + return err; +} + +void destroy_flush_cmd_control(struct f2fs_sb_info *sbi) +{ + struct flush_cmd_control *fcc = + sbi->sm_info->cmd_control_info; + + if (fcc && fcc->f2fs_issue_flush) + kthread_stop(fcc->f2fs_issue_flush); + kfree(fcc); + sbi->sm_info->cmd_control_info = NULL; } static void __locate_dirty_segment(struct f2fs_sb_info *sbi, unsigned int segno, @@ -336,13 +367,26 @@ static void locate_dirty_segment(struct f2fs_sb_info *sbi, unsigned int segno) mutex_unlock(&dirty_i->seglist_lock); } -static void f2fs_issue_discard(struct f2fs_sb_info *sbi, +static int f2fs_issue_discard(struct f2fs_sb_info *sbi, block_t blkstart, block_t blklen) { sector_t start = SECTOR_FROM_BLOCK(sbi, blkstart); sector_t len = SECTOR_FROM_BLOCK(sbi, blklen); - blkdev_issue_discard(sbi->sb->s_bdev, start, len, GFP_NOFS, 0); trace_f2fs_issue_discard(sbi->sb, blkstart, blklen); + return blkdev_issue_discard(sbi->sb->s_bdev, start, len, GFP_NOFS, 0); +} + +void discard_next_dnode(struct f2fs_sb_info *sbi) +{ + struct curseg_info *curseg = CURSEG_I(sbi, CURSEG_WARM_NODE); + block_t blkaddr = NEXT_FREE_BLKADDR(sbi, curseg); + + if (f2fs_issue_discard(sbi, blkaddr, 1)) { + struct page *page = grab_meta_page(sbi, blkaddr); + /* zero-filled page */ + set_page_dirty(page); + f2fs_put_page(page, 1); + } } static void add_discard_addrs(struct f2fs_sb_info *sbi, @@ -1832,7 +1876,6 @@ int build_segment_manager(struct f2fs_sb_info *sbi) { struct f2fs_super_block *raw_super = F2FS_RAW_SUPER(sbi); struct f2fs_checkpoint *ckpt = F2FS_CKPT(sbi); - dev_t dev = sbi->sb->s_bdev->bd_dev; struct f2fs_sm_info *sm_info; int err; @@ -1860,14 +1903,10 @@ int build_segment_manager(struct f2fs_sb_info *sbi) sm_info->nr_discards = 0; sm_info->max_discards = 0; - if (test_opt(sbi, FLUSH_MERGE)) { - spin_lock_init(&sm_info->issue_lock); - init_waitqueue_head(&sm_info->flush_wait_queue); - - sm_info->f2fs_issue_flush = kthread_run(issue_flush_thread, sbi, - "f2fs_flush-%u:%u", MAJOR(dev), MINOR(dev)); - if (IS_ERR(sm_info->f2fs_issue_flush)) - return PTR_ERR(sm_info->f2fs_issue_flush); + if (test_opt(sbi, FLUSH_MERGE) && !f2fs_readonly(sbi->sb)) { + err = create_flush_cmd_control(sbi); + if (err) + return err; } err = build_sit_info(sbi); @@ -1976,10 +2015,10 @@ static void destroy_sit_info(struct f2fs_sb_info *sbi) void destroy_segment_manager(struct f2fs_sb_info *sbi) { struct f2fs_sm_info *sm_info = SM_I(sbi); + if (!sm_info) return; - if (sm_info->f2fs_issue_flush) - kthread_stop(sm_info->f2fs_issue_flush); + destroy_flush_cmd_control(sbi); destroy_dirty_segmap(sbi); destroy_curseg(sbi); destroy_free_segmap(sbi); @@ -1994,17 +2033,10 @@ int __init create_segment_manager_caches(void) sizeof(struct discard_entry)); if (!discard_entry_slab) return -ENOMEM; - flush_cmd_slab = f2fs_kmem_cache_create("flush_command", - sizeof(struct flush_cmd)); - if (!flush_cmd_slab) { - kmem_cache_destroy(discard_entry_slab); - return -ENOMEM; - } return 0; } void destroy_segment_manager_caches(void) { kmem_cache_destroy(discard_entry_slab); - kmem_cache_destroy(flush_cmd_slab); } diff --git a/fs/f2fs/super.c b/fs/f2fs/super.c index c756923a7302..b2b18637cb9e 100644 --- a/fs/f2fs/super.c +++ b/fs/f2fs/super.c @@ -514,7 +514,7 @@ static int f2fs_show_options(struct seq_file *seq, struct dentry *root) { struct f2fs_sb_info *sbi = F2FS_SB(root->d_sb); - if (!(root->d_sb->s_flags & MS_RDONLY) && test_opt(sbi, BG_GC)) + if (!f2fs_readonly(sbi->sb) && test_opt(sbi, BG_GC)) seq_printf(seq, ",background_gc=%s", "on"); else seq_printf(seq, ",background_gc=%s", "off"); @@ -542,7 +542,7 @@ static int f2fs_show_options(struct seq_file *seq, struct dentry *root) seq_puts(seq, ",disable_ext_identify"); if (test_opt(sbi, INLINE_DATA)) seq_puts(seq, ",inline_data"); - if (test_opt(sbi, FLUSH_MERGE)) + if (!f2fs_readonly(sbi->sb) && test_opt(sbi, FLUSH_MERGE)) seq_puts(seq, ",flush_merge"); seq_printf(seq, ",active_logs=%u", sbi->active_logs); @@ -594,6 +594,8 @@ static int f2fs_remount(struct super_block *sb, int *flags, char *data) struct f2fs_sb_info *sbi = F2FS_SB(sb); struct f2fs_mount_info org_mount_opt; int err, active_logs; + bool need_restart_gc = false; + bool need_stop_gc = false; sync_filesystem(sb); @@ -611,7 +613,7 @@ static int f2fs_remount(struct super_block *sb, int *flags, char *data) /* * Previous and new state of filesystem is RO, - * so no point in checking GC conditions. + * so skip checking GC and FLUSH_MERGE conditions. */ if ((sb->s_flags & MS_RDONLY) && (*flags & MS_RDONLY)) goto skip; @@ -625,18 +627,40 @@ static int f2fs_remount(struct super_block *sb, int *flags, char *data) if (sbi->gc_thread) { stop_gc_thread(sbi); f2fs_sync_fs(sb, 1); + need_restart_gc = true; } } else if (test_opt(sbi, BG_GC) && !sbi->gc_thread) { err = start_gc_thread(sbi); if (err) goto restore_opts; + need_stop_gc = true; + } + + /* + * We stop issue flush thread if FS is mounted as RO + * or if flush_merge is not passed in mount option. + */ + if ((*flags & MS_RDONLY) || !test_opt(sbi, FLUSH_MERGE)) { + destroy_flush_cmd_control(sbi); + } else if (test_opt(sbi, FLUSH_MERGE) && + !sbi->sm_info->cmd_control_info) { + err = create_flush_cmd_control(sbi); + if (err) + goto restore_gc; } skip: /* Update the POSIXACL Flag */ sb->s_flags = (sb->s_flags & ~MS_POSIXACL) | (test_opt(sbi, POSIX_ACL) ? MS_POSIXACL : 0); return 0; - +restore_gc: + if (need_restart_gc) { + if (start_gc_thread(sbi)) + f2fs_msg(sbi->sb, KERN_WARNING, + "background gc thread is stop"); + } else if (need_stop_gc) { + stop_gc_thread(sbi); + } restore_opts: sbi->mount_opt = org_mount_opt; sbi->active_logs = active_logs; diff --git a/fs/f2fs/xattr.c b/fs/f2fs/xattr.c index 503c2451131e..8bea941ee309 100644 --- a/fs/f2fs/xattr.c +++ b/fs/f2fs/xattr.c @@ -26,7 +26,7 @@ #include "xattr.h" static size_t f2fs_xattr_generic_list(struct dentry *dentry, char *list, - size_t list_size, const char *name, size_t name_len, int type) + size_t list_size, const char *name, size_t len, int type) { struct f2fs_sb_info *sbi = F2FS_SB(dentry->d_sb); int total_len, prefix_len = 0; @@ -53,11 +53,11 @@ static size_t f2fs_xattr_generic_list(struct dentry *dentry, char *list, return -EINVAL; } - total_len = prefix_len + name_len + 1; + total_len = prefix_len + len + 1; if (list && total_len <= list_size) { memcpy(list, prefix, prefix_len); - memcpy(list + prefix_len, name, name_len); - list[prefix_len + name_len] = '\0'; + memcpy(list + prefix_len, name, len); + list[prefix_len + len] = '\0'; } return total_len; } @@ -108,11 +108,12 @@ static int f2fs_xattr_generic_set(struct dentry *dentry, const char *name, if (strcmp(name, "") == 0) return -EINVAL; - return f2fs_setxattr(dentry->d_inode, type, name, value, size, NULL); + return f2fs_setxattr(dentry->d_inode, type, name, + value, size, NULL, flags); } static size_t f2fs_xattr_advise_list(struct dentry *dentry, char *list, - size_t list_size, const char *name, size_t name_len, int type) + size_t list_size, const char *name, size_t len, int type) { const char *xname = F2FS_SYSTEM_ADVISE_PREFIX; size_t size; @@ -155,9 +156,6 @@ static int f2fs_xattr_advise_set(struct dentry *dentry, const char *name, } #ifdef CONFIG_F2FS_FS_SECURITY -static int __f2fs_setxattr(struct inode *inode, int name_index, - const char *name, const void *value, size_t value_len, - struct page *ipage); static int f2fs_initxattrs(struct inode *inode, const struct xattr *xattr_array, void *page) { @@ -165,9 +163,9 @@ static int f2fs_initxattrs(struct inode *inode, const struct xattr *xattr_array, int err = 0; for (xattr = xattr_array; xattr->name != NULL; xattr++) { - err = __f2fs_setxattr(inode, F2FS_XATTR_INDEX_SECURITY, + err = f2fs_setxattr(inode, F2FS_XATTR_INDEX_SECURITY, xattr->name, xattr->value, - xattr->value_len, (struct page *)page); + xattr->value_len, (struct page *)page, 0); if (err < 0) break; } @@ -241,26 +239,26 @@ const struct xattr_handler *f2fs_xattr_handlers[] = { NULL, }; -static inline const struct xattr_handler *f2fs_xattr_handler(int name_index) +static inline const struct xattr_handler *f2fs_xattr_handler(int index) { const struct xattr_handler *handler = NULL; - if (name_index > 0 && name_index < ARRAY_SIZE(f2fs_xattr_handler_map)) - handler = f2fs_xattr_handler_map[name_index]; + if (index > 0 && index < ARRAY_SIZE(f2fs_xattr_handler_map)) + handler = f2fs_xattr_handler_map[index]; return handler; } -static struct f2fs_xattr_entry *__find_xattr(void *base_addr, int name_index, - size_t name_len, const char *name) +static struct f2fs_xattr_entry *__find_xattr(void *base_addr, int index, + size_t len, const char *name) { struct f2fs_xattr_entry *entry; list_for_each_xattr(entry, base_addr) { - if (entry->e_name_index != name_index) + if (entry->e_name_index != index) continue; - if (entry->e_name_len != name_len) + if (entry->e_name_len != len) continue; - if (!memcmp(entry->e_name, name, name_len)) + if (!memcmp(entry->e_name, name, len)) break; } return entry; @@ -347,6 +345,7 @@ static inline int write_all_xattrs(struct inode *inode, __u32 hsize, if (ipage) { inline_addr = inline_xattr_addr(ipage); + f2fs_wait_on_page_writeback(ipage, NODE); } else { page = get_node_page(sbi, inode->i_ino); if (IS_ERR(page)) { @@ -354,6 +353,7 @@ static inline int write_all_xattrs(struct inode *inode, __u32 hsize, return PTR_ERR(page); } inline_addr = inline_xattr_addr(page); + f2fs_wait_on_page_writeback(page, NODE); } memcpy(inline_addr, txattr_addr, inline_size); f2fs_put_page(page, 1); @@ -374,6 +374,7 @@ static inline int write_all_xattrs(struct inode *inode, __u32 hsize, return PTR_ERR(xpage); } f2fs_bug_on(new_nid); + f2fs_wait_on_page_writeback(xpage, NODE); } else { struct dnode_of_data dn; set_new_dnode(&dn, inode, NULL, NULL, new_nid); @@ -396,42 +397,43 @@ static inline int write_all_xattrs(struct inode *inode, __u32 hsize, return 0; } -int f2fs_getxattr(struct inode *inode, int name_index, const char *name, +int f2fs_getxattr(struct inode *inode, int index, const char *name, void *buffer, size_t buffer_size) { struct f2fs_xattr_entry *entry; void *base_addr; int error = 0; - size_t value_len, name_len; + size_t size, len; if (name == NULL) return -EINVAL; - name_len = strlen(name); - if (name_len > F2FS_NAME_LEN) + + len = strlen(name); + if (len > F2FS_NAME_LEN) return -ERANGE; base_addr = read_all_xattrs(inode, NULL); if (!base_addr) return -ENOMEM; - entry = __find_xattr(base_addr, name_index, name_len, name); + entry = __find_xattr(base_addr, index, len, name); if (IS_XATTR_LAST_ENTRY(entry)) { error = -ENODATA; goto cleanup; } - value_len = le16_to_cpu(entry->e_value_size); + size = le16_to_cpu(entry->e_value_size); - if (buffer && value_len > buffer_size) { + if (buffer && size > buffer_size) { error = -ERANGE; goto cleanup; } if (buffer) { char *pval = entry->e_name + entry->e_name_len; - memcpy(buffer, pval, value_len); + memcpy(buffer, pval, size); } - error = value_len; + error = size; cleanup: kzfree(base_addr); @@ -475,15 +477,15 @@ cleanup: return error; } -static int __f2fs_setxattr(struct inode *inode, int name_index, - const char *name, const void *value, size_t value_len, - struct page *ipage) +static int __f2fs_setxattr(struct inode *inode, int index, + const char *name, const void *value, size_t size, + struct page *ipage, int flags) { struct f2fs_inode_info *fi = F2FS_I(inode); struct f2fs_xattr_entry *here, *last; void *base_addr; int found, newsize; - size_t name_len; + size_t len; __u32 new_hsize; int error = -ENOMEM; @@ -491,11 +493,11 @@ static int __f2fs_setxattr(struct inode *inode, int name_index, return -EINVAL; if (value == NULL) - value_len = 0; + size = 0; - name_len = strlen(name); + len = strlen(name); - if (name_len > F2FS_NAME_LEN || value_len > MAX_VALUE_LEN(inode)) + if (len > F2FS_NAME_LEN || size > MAX_VALUE_LEN(inode)) return -ERANGE; base_addr = read_all_xattrs(inode, ipage); @@ -503,16 +505,23 @@ static int __f2fs_setxattr(struct inode *inode, int name_index, goto exit; /* find entry with wanted name. */ - here = __find_xattr(base_addr, name_index, name_len, name); + here = __find_xattr(base_addr, index, len, name); found = IS_XATTR_LAST_ENTRY(here) ? 0 : 1; - last = here; + if ((flags & XATTR_REPLACE) && !found) { + error = -ENODATA; + goto exit; + } else if ((flags & XATTR_CREATE) && found) { + error = -EEXIST; + goto exit; + } + + last = here; while (!IS_XATTR_LAST_ENTRY(last)) last = XATTR_NEXT_ENTRY(last); - newsize = XATTR_ALIGN(sizeof(struct f2fs_xattr_entry) + - name_len + value_len); + newsize = XATTR_ALIGN(sizeof(struct f2fs_xattr_entry) + len + size); /* 1. Check space */ if (value) { @@ -555,12 +564,12 @@ static int __f2fs_setxattr(struct inode *inode, int name_index, * We just write new entry. */ memset(last, 0, newsize); - last->e_name_index = name_index; - last->e_name_len = name_len; - memcpy(last->e_name, name, name_len); - pval = last->e_name + name_len; - memcpy(pval, value, value_len); - last->e_value_size = cpu_to_le16(value_len); + last->e_name_index = index; + last->e_name_len = len; + memcpy(last->e_name, name, len); + pval = last->e_name + len; + memcpy(pval, value, size); + last->e_value_size = cpu_to_le16(size); new_hsize += newsize; } @@ -583,18 +592,23 @@ exit: return error; } -int f2fs_setxattr(struct inode *inode, int name_index, const char *name, - const void *value, size_t value_len, struct page *ipage) +int f2fs_setxattr(struct inode *inode, int index, const char *name, + const void *value, size_t size, + struct page *ipage, int flags) { struct f2fs_sb_info *sbi = F2FS_SB(inode->i_sb); int err; + /* this case is only from init_inode_metadata */ + if (ipage) + return __f2fs_setxattr(inode, index, name, value, + size, ipage, flags); f2fs_balance_fs(sbi); f2fs_lock_op(sbi); /* protect xattr_ver */ down_write(&F2FS_I(inode)->i_sem); - err = __f2fs_setxattr(inode, name_index, name, value, value_len, ipage); + err = __f2fs_setxattr(inode, index, name, value, size, ipage, flags); up_write(&F2FS_I(inode)->i_sem); f2fs_unlock_op(sbi); diff --git a/fs/f2fs/xattr.h b/fs/f2fs/xattr.h index b21d9ebdeff3..34ab7dbcf5e3 100644 --- a/fs/f2fs/xattr.h +++ b/fs/f2fs/xattr.h @@ -114,18 +114,18 @@ extern const struct xattr_handler f2fs_xattr_security_handler; extern const struct xattr_handler *f2fs_xattr_handlers[]; extern int f2fs_setxattr(struct inode *, int, const char *, - const void *, size_t, struct page *); + const void *, size_t, struct page *, int); extern int f2fs_getxattr(struct inode *, int, const char *, void *, size_t); extern ssize_t f2fs_listxattr(struct dentry *, char *, size_t); #else #define f2fs_xattr_handlers NULL -static inline int f2fs_setxattr(struct inode *inode, int name_index, - const char *name, const void *value, size_t value_len) +static inline int f2fs_setxattr(struct inode *inode, int index, + const char *name, const void *value, size_t size, int flags) { return -EOPNOTSUPP; } -static inline int f2fs_getxattr(struct inode *inode, int name_index, +static inline int f2fs_getxattr(struct inode *inode, int index, const char *name, void *buffer, size_t buffer_size) { return -EOPNOTSUPP; diff --git a/fs/inode.c b/fs/inode.c index 2feb9b69f1be..6eecb7ff0b9a 100644 --- a/fs/inode.c +++ b/fs/inode.c @@ -1839,14 +1839,18 @@ EXPORT_SYMBOL(inode_init_owner); * inode_owner_or_capable - check current task permissions to inode * @inode: inode being checked * - * Return true if current either has CAP_FOWNER to the inode, or - * owns the file. + * Return true if current either has CAP_FOWNER in a namespace with the + * inode owner uid mapped, or owns the file. */ bool inode_owner_or_capable(const struct inode *inode) { + struct user_namespace *ns; + if (uid_eq(current_fsuid(), inode->i_uid)) return true; - if (inode_capable(inode, CAP_FOWNER)) + + ns = current_user_ns(); + if (ns_capable(ns, CAP_FOWNER) && kuid_has_mapping(ns, inode->i_uid)) return true; return false; } diff --git a/fs/lockd/clnt4xdr.c b/fs/lockd/clnt4xdr.c index 00ec0b9c94d1..d3e40db28930 100644 --- a/fs/lockd/clnt4xdr.c +++ b/fs/lockd/clnt4xdr.c @@ -14,6 +14,8 @@ #include <linux/sunrpc/stats.h> #include <linux/lockd/lockd.h> +#include <uapi/linux/nfs3.h> + #define NLMDBG_FACILITY NLMDBG_XDR #if (NLMCLNT_OHSIZE > XDR_MAX_NETOBJ) diff --git a/fs/lockd/clntxdr.c b/fs/lockd/clntxdr.c index 9a55797a1cd4..3e9f7874b975 100644 --- a/fs/lockd/clntxdr.c +++ b/fs/lockd/clntxdr.c @@ -15,6 +15,8 @@ #include <linux/sunrpc/stats.h> #include <linux/lockd/lockd.h> +#include <uapi/linux/nfs2.h> + #define NLMDBG_FACILITY NLMDBG_XDR #if (NLMCLNT_OHSIZE > XDR_MAX_NETOBJ) diff --git a/fs/lockd/svc.c b/fs/lockd/svc.c index de051cb1f553..8f27c93f8d2e 100644 --- a/fs/lockd/svc.c +++ b/fs/lockd/svc.c @@ -622,8 +622,8 @@ static int __init init_nlm(void) err_pernet: #ifdef CONFIG_SYSCTL unregister_sysctl_table(nlm_sysctl_table); -#endif err_sysctl: +#endif return err; } diff --git a/fs/lockd/svcsubs.c b/fs/lockd/svcsubs.c index dc5c75930f0f..b6f3b84b6e99 100644 --- a/fs/lockd/svcsubs.c +++ b/fs/lockd/svcsubs.c @@ -14,12 +14,11 @@ #include <linux/mutex.h> #include <linux/sunrpc/svc.h> #include <linux/sunrpc/addr.h> -#include <linux/nfsd/nfsfh.h> -#include <linux/nfsd/export.h> #include <linux/lockd/lockd.h> #include <linux/lockd/share.h> #include <linux/module.h> #include <linux/mount.h> +#include <uapi/linux/nfs2.h> #define NLMDBG_FACILITY NLMDBG_SVCSUBS diff --git a/fs/lockd/xdr.c b/fs/lockd/xdr.c index 964666c68a86..9340e7e10ef6 100644 --- a/fs/lockd/xdr.c +++ b/fs/lockd/xdr.c @@ -16,6 +16,8 @@ #include <linux/sunrpc/stats.h> #include <linux/lockd/lockd.h> +#include <uapi/linux/nfs2.h> + #define NLMDBG_FACILITY NLMDBG_XDR diff --git a/fs/namei.c b/fs/namei.c index 80168273396b..985c6f368485 100644 --- a/fs/namei.c +++ b/fs/namei.c @@ -332,10 +332,11 @@ int generic_permission(struct inode *inode, int mask) if (S_ISDIR(inode->i_mode)) { /* DACs are overridable for directories */ - if (inode_capable(inode, CAP_DAC_OVERRIDE)) + if (capable_wrt_inode_uidgid(inode, CAP_DAC_OVERRIDE)) return 0; if (!(mask & MAY_WRITE)) - if (inode_capable(inode, CAP_DAC_READ_SEARCH)) + if (capable_wrt_inode_uidgid(inode, + CAP_DAC_READ_SEARCH)) return 0; return -EACCES; } @@ -345,7 +346,7 @@ int generic_permission(struct inode *inode, int mask) * at least one exec bit set. */ if (!(mask & MAY_EXEC) || (inode->i_mode & S_IXUGO)) - if (inode_capable(inode, CAP_DAC_OVERRIDE)) + if (capable_wrt_inode_uidgid(inode, CAP_DAC_OVERRIDE)) return 0; /* @@ -353,7 +354,7 @@ int generic_permission(struct inode *inode, int mask) */ mask &= MAY_READ | MAY_WRITE | MAY_EXEC; if (mask == MAY_READ) - if (inode_capable(inode, CAP_DAC_READ_SEARCH)) + if (capable_wrt_inode_uidgid(inode, CAP_DAC_READ_SEARCH)) return 0; return -EACCES; @@ -2379,7 +2380,7 @@ static inline int check_sticky(struct inode *dir, struct inode *inode) return 0; if (uid_eq(dir->i_uid, fsuid)) return 0; - return !inode_capable(inode, CAP_FOWNER); + return !capable_wrt_inode_uidgid(inode, CAP_FOWNER); } /* diff --git a/fs/nfs/Makefile b/fs/nfs/Makefile index 03192a66c143..4782e0840dcc 100644 --- a/fs/nfs/Makefile +++ b/fs/nfs/Makefile @@ -29,8 +29,6 @@ nfsv4-$(CONFIG_NFS_USE_LEGACY_DNS) += cache_lib.o nfsv4-$(CONFIG_SYSCTL) += nfs4sysctl.o nfsv4-$(CONFIG_NFS_V4_1) += pnfs.o pnfs_dev.o -obj-$(CONFIG_PNFS_FILE_LAYOUT) += nfs_layout_nfsv41_files.o -nfs_layout_nfsv41_files-y := nfs4filelayout.o nfs4filelayoutdev.o - +obj-$(CONFIG_PNFS_FILE_LAYOUT) += filelayout/ obj-$(CONFIG_PNFS_OBJLAYOUT) += objlayout/ obj-$(CONFIG_PNFS_BLOCK) += blocklayout/ diff --git a/fs/nfs/blocklayout/blocklayout.c b/fs/nfs/blocklayout/blocklayout.c index 65d849bdf77a..9b431f44fad9 100644 --- a/fs/nfs/blocklayout/blocklayout.c +++ b/fs/nfs/blocklayout/blocklayout.c @@ -210,7 +210,7 @@ static void bl_end_io_read(struct bio *bio, int err) SetPageUptodate(bvec->bv_page); if (err) { - struct nfs_read_data *rdata = par->data; + struct nfs_pgio_data *rdata = par->data; struct nfs_pgio_header *header = rdata->header; if (!header->pnfs_error) @@ -224,17 +224,17 @@ static void bl_end_io_read(struct bio *bio, int err) static void bl_read_cleanup(struct work_struct *work) { struct rpc_task *task; - struct nfs_read_data *rdata; + struct nfs_pgio_data *rdata; dprintk("%s enter\n", __func__); task = container_of(work, struct rpc_task, u.tk_work); - rdata = container_of(task, struct nfs_read_data, task); + rdata = container_of(task, struct nfs_pgio_data, task); pnfs_ld_read_done(rdata); } static void bl_end_par_io_read(void *data, int unused) { - struct nfs_read_data *rdata = data; + struct nfs_pgio_data *rdata = data; rdata->task.tk_status = rdata->header->pnfs_error; INIT_WORK(&rdata->task.u.tk_work, bl_read_cleanup); @@ -242,7 +242,7 @@ bl_end_par_io_read(void *data, int unused) } static enum pnfs_try_status -bl_read_pagelist(struct nfs_read_data *rdata) +bl_read_pagelist(struct nfs_pgio_data *rdata) { struct nfs_pgio_header *header = rdata->header; int i, hole; @@ -390,7 +390,7 @@ static void bl_end_io_write_zero(struct bio *bio, int err) } if (unlikely(err)) { - struct nfs_write_data *data = par->data; + struct nfs_pgio_data *data = par->data; struct nfs_pgio_header *header = data->header; if (!header->pnfs_error) @@ -405,7 +405,7 @@ static void bl_end_io_write(struct bio *bio, int err) { struct parallel_io *par = bio->bi_private; const int uptodate = test_bit(BIO_UPTODATE, &bio->bi_flags); - struct nfs_write_data *data = par->data; + struct nfs_pgio_data *data = par->data; struct nfs_pgio_header *header = data->header; if (!uptodate) { @@ -423,10 +423,10 @@ static void bl_end_io_write(struct bio *bio, int err) static void bl_write_cleanup(struct work_struct *work) { struct rpc_task *task; - struct nfs_write_data *wdata; + struct nfs_pgio_data *wdata; dprintk("%s enter\n", __func__); task = container_of(work, struct rpc_task, u.tk_work); - wdata = container_of(task, struct nfs_write_data, task); + wdata = container_of(task, struct nfs_pgio_data, task); if (likely(!wdata->header->pnfs_error)) { /* Marks for LAYOUTCOMMIT */ mark_extents_written(BLK_LSEG2EXT(wdata->header->lseg), @@ -438,7 +438,7 @@ static void bl_write_cleanup(struct work_struct *work) /* Called when last of bios associated with a bl_write_pagelist call finishes */ static void bl_end_par_io_write(void *data, int num_se) { - struct nfs_write_data *wdata = data; + struct nfs_pgio_data *wdata = data; if (unlikely(wdata->header->pnfs_error)) { bl_free_short_extents(&BLK_LSEG2EXT(wdata->header->lseg)->bl_inval, @@ -673,7 +673,7 @@ check_page: } static enum pnfs_try_status -bl_write_pagelist(struct nfs_write_data *wdata, int sync) +bl_write_pagelist(struct nfs_pgio_data *wdata, int sync) { struct nfs_pgio_header *header = wdata->header; int i, ret, npg_zero, pg_index, last = 0; @@ -1189,13 +1189,17 @@ bl_pg_init_read(struct nfs_pageio_descriptor *pgio, struct nfs_page *req) pnfs_generic_pg_init_read(pgio, req); } -static bool +/* + * Return 0 if @req cannot be coalesced into @pgio, otherwise return the number + * of bytes (maximum @req->wb_bytes) that can be coalesced. + */ +static size_t bl_pg_test_read(struct nfs_pageio_descriptor *pgio, struct nfs_page *prev, struct nfs_page *req) { if (pgio->pg_dreq != NULL && !is_aligned_req(req, SECTOR_SIZE)) - return false; + return 0; return pnfs_generic_pg_test(pgio, prev, req); } @@ -1241,13 +1245,17 @@ bl_pg_init_write(struct nfs_pageio_descriptor *pgio, struct nfs_page *req) } } -static bool +/* + * Return 0 if @req cannot be coalesced into @pgio, otherwise return the number + * of bytes (maximum @req->wb_bytes) that can be coalesced. + */ +static size_t bl_pg_test_write(struct nfs_pageio_descriptor *pgio, struct nfs_page *prev, struct nfs_page *req) { if (pgio->pg_dreq != NULL && !is_aligned_req(req, PAGE_CACHE_SIZE)) - return false; + return 0; return pnfs_generic_pg_test(pgio, prev, req); } diff --git a/fs/nfs/direct.c b/fs/nfs/direct.c index b8797ae6831f..4ad7bc388679 100644 --- a/fs/nfs/direct.c +++ b/fs/nfs/direct.c @@ -108,6 +108,97 @@ static inline int put_dreq(struct nfs_direct_req *dreq) return atomic_dec_and_test(&dreq->io_count); } +/* + * nfs_direct_select_verf - select the right verifier + * @dreq - direct request possibly spanning multiple servers + * @ds_clp - nfs_client of data server or NULL if MDS / non-pnfs + * @ds_idx - index of data server in data server list, only valid if ds_clp set + * + * returns the correct verifier to use given the role of the server + */ +static struct nfs_writeverf * +nfs_direct_select_verf(struct nfs_direct_req *dreq, + struct nfs_client *ds_clp, + int ds_idx) +{ + struct nfs_writeverf *verfp = &dreq->verf; + +#ifdef CONFIG_NFS_V4_1 + if (ds_clp) { + /* pNFS is in use, use the DS verf */ + if (ds_idx >= 0 && ds_idx < dreq->ds_cinfo.nbuckets) + verfp = &dreq->ds_cinfo.buckets[ds_idx].direct_verf; + else + WARN_ON_ONCE(1); + } +#endif + return verfp; +} + + +/* + * nfs_direct_set_hdr_verf - set the write/commit verifier + * @dreq - direct request possibly spanning multiple servers + * @hdr - pageio header to validate against previously seen verfs + * + * Set the server's (MDS or DS) "seen" verifier + */ +static void nfs_direct_set_hdr_verf(struct nfs_direct_req *dreq, + struct nfs_pgio_header *hdr) +{ + struct nfs_writeverf *verfp; + + verfp = nfs_direct_select_verf(dreq, hdr->data->ds_clp, + hdr->data->ds_idx); + WARN_ON_ONCE(verfp->committed >= 0); + memcpy(verfp, &hdr->verf, sizeof(struct nfs_writeverf)); + WARN_ON_ONCE(verfp->committed < 0); +} + +/* + * nfs_direct_cmp_hdr_verf - compare verifier for pgio header + * @dreq - direct request possibly spanning multiple servers + * @hdr - pageio header to validate against previously seen verf + * + * set the server's "seen" verf if not initialized. + * returns result of comparison between @hdr->verf and the "seen" + * verf of the server used by @hdr (DS or MDS) + */ +static int nfs_direct_set_or_cmp_hdr_verf(struct nfs_direct_req *dreq, + struct nfs_pgio_header *hdr) +{ + struct nfs_writeverf *verfp; + + verfp = nfs_direct_select_verf(dreq, hdr->data->ds_clp, + hdr->data->ds_idx); + if (verfp->committed < 0) { + nfs_direct_set_hdr_verf(dreq, hdr); + return 0; + } + return memcmp(verfp, &hdr->verf, sizeof(struct nfs_writeverf)); +} + +#if IS_ENABLED(CONFIG_NFS_V3) || IS_ENABLED(CONFIG_NFS_V4) +/* + * nfs_direct_cmp_commit_data_verf - compare verifier for commit data + * @dreq - direct request possibly spanning multiple servers + * @data - commit data to validate against previously seen verf + * + * returns result of comparison between @data->verf and the verf of + * the server used by @data (DS or MDS) + */ +static int nfs_direct_cmp_commit_data_verf(struct nfs_direct_req *dreq, + struct nfs_commit_data *data) +{ + struct nfs_writeverf *verfp; + + verfp = nfs_direct_select_verf(dreq, data->ds_clp, + data->ds_commit_index); + WARN_ON_ONCE(verfp->committed < 0); + return memcmp(verfp, &data->verf, sizeof(struct nfs_writeverf)); +} +#endif + /** * nfs_direct_IO - NFS address space operation for direct I/O * @rw: direction (read or write) @@ -168,6 +259,7 @@ static inline struct nfs_direct_req *nfs_direct_req_alloc(void) kref_get(&dreq->kref); init_completion(&dreq->completion); INIT_LIST_HEAD(&dreq->mds_cinfo.list); + dreq->verf.committed = NFS_INVALID_STABLE_HOW; /* not set yet */ INIT_WORK(&dreq->work, nfs_direct_write_schedule_work); spin_lock_init(&dreq->lock); @@ -380,8 +472,7 @@ static ssize_t nfs_direct_read_schedule_segment(struct nfs_pageio_descriptor *de struct nfs_page *req; unsigned int req_len = min_t(size_t, bytes, PAGE_SIZE - pgbase); /* XXX do we need to do the eof zeroing found in async_filler? */ - req = nfs_create_request(dreq->ctx, dreq->inode, - pagevec[i], + req = nfs_create_request(dreq->ctx, pagevec[i], NULL, pgbase, req_len); if (IS_ERR(req)) { result = PTR_ERR(req); @@ -424,7 +515,7 @@ static ssize_t nfs_direct_read_schedule_iovec(struct nfs_direct_req *dreq, size_t requested_bytes = 0; unsigned long seg; - NFS_PROTO(dreq->inode)->read_pageio_init(&desc, dreq->inode, + nfs_pageio_init_read(&desc, dreq->inode, false, &nfs_direct_read_completion_ops); get_dreq(dreq); desc.pg_dreq = dreq; @@ -564,7 +655,7 @@ static void nfs_direct_write_reschedule(struct nfs_direct_req *dreq) dreq->count = 0; get_dreq(dreq); - NFS_PROTO(dreq->inode)->write_pageio_init(&desc, dreq->inode, FLUSH_STABLE, + nfs_pageio_init_write(&desc, dreq->inode, FLUSH_STABLE, false, &nfs_direct_write_completion_ops); desc.pg_dreq = dreq; @@ -603,7 +694,7 @@ static void nfs_direct_commit_complete(struct nfs_commit_data *data) dprintk("NFS: %5u commit failed with error %d.\n", data->task.tk_pid, status); dreq->flags = NFS_ODIRECT_RESCHED_WRITES; - } else if (memcmp(&dreq->verf, &data->verf, sizeof(data->verf))) { + } else if (nfs_direct_cmp_commit_data_verf(dreq, data)) { dprintk("NFS: %5u commit verify failed\n", data->task.tk_pid); dreq->flags = NFS_ODIRECT_RESCHED_WRITES; } @@ -750,8 +841,7 @@ static ssize_t nfs_direct_write_schedule_segment(struct nfs_pageio_descriptor *d struct nfs_page *req; unsigned int req_len = min_t(size_t, bytes, PAGE_SIZE - pgbase); - req = nfs_create_request(dreq->ctx, dreq->inode, - pagevec[i], + req = nfs_create_request(dreq->ctx, pagevec[i], NULL, pgbase, req_len); if (IS_ERR(req)) { result = PTR_ERR(req); @@ -813,13 +903,13 @@ static void nfs_direct_write_completion(struct nfs_pgio_header *hdr) if (dreq->flags == NFS_ODIRECT_RESCHED_WRITES) bit = NFS_IOHDR_NEED_RESCHED; else if (dreq->flags == 0) { - memcpy(&dreq->verf, hdr->verf, - sizeof(dreq->verf)); + nfs_direct_set_hdr_verf(dreq, hdr); bit = NFS_IOHDR_NEED_COMMIT; dreq->flags = NFS_ODIRECT_DO_COMMIT; } else if (dreq->flags == NFS_ODIRECT_DO_COMMIT) { - if (memcmp(&dreq->verf, hdr->verf, sizeof(dreq->verf))) { - dreq->flags = NFS_ODIRECT_RESCHED_WRITES; + if (nfs_direct_set_or_cmp_hdr_verf(dreq, hdr)) { + dreq->flags = + NFS_ODIRECT_RESCHED_WRITES; bit = NFS_IOHDR_NEED_RESCHED; } else bit = NFS_IOHDR_NEED_COMMIT; @@ -829,6 +919,8 @@ static void nfs_direct_write_completion(struct nfs_pgio_header *hdr) spin_unlock(&dreq->lock); while (!list_empty(&hdr->pages)) { + bool do_destroy = true; + req = nfs_list_entry(hdr->pages.next); nfs_list_remove_request(req); switch (bit) { @@ -836,6 +928,7 @@ static void nfs_direct_write_completion(struct nfs_pgio_header *hdr) case NFS_IOHDR_NEED_COMMIT: kref_get(&req->wb_kref); nfs_mark_request_commit(req, hdr->lseg, &cinfo); + do_destroy = false; } nfs_unlock_and_release_request(req); } @@ -874,7 +967,7 @@ static ssize_t nfs_direct_write_schedule_iovec(struct nfs_direct_req *dreq, size_t requested_bytes = 0; unsigned long seg; - NFS_PROTO(inode)->write_pageio_init(&desc, inode, FLUSH_COND_STABLE, + nfs_pageio_init_write(&desc, inode, FLUSH_COND_STABLE, false, &nfs_direct_write_completion_ops); desc.pg_dreq = dreq; get_dreq(dreq); diff --git a/fs/nfs/filelayout/Makefile b/fs/nfs/filelayout/Makefile new file mode 100644 index 000000000000..8516cdffb9e9 --- /dev/null +++ b/fs/nfs/filelayout/Makefile @@ -0,0 +1,5 @@ +# +# Makefile for the pNFS Files Layout Driver kernel module +# +obj-$(CONFIG_PNFS_FILE_LAYOUT) += nfs_layout_nfsv41_files.o +nfs_layout_nfsv41_files-y := filelayout.o filelayoutdev.o diff --git a/fs/nfs/nfs4filelayout.c b/fs/nfs/filelayout/filelayout.c index b9a35c05b60f..d2eba1c13b7e 100644 --- a/fs/nfs/nfs4filelayout.c +++ b/fs/nfs/filelayout/filelayout.c @@ -35,11 +35,11 @@ #include <linux/sunrpc/metrics.h> -#include "nfs4session.h" -#include "internal.h" -#include "delegation.h" -#include "nfs4filelayout.h" -#include "nfs4trace.h" +#include "../nfs4session.h" +#include "../internal.h" +#include "../delegation.h" +#include "filelayout.h" +#include "../nfs4trace.h" #define NFSDBG_FACILITY NFSDBG_PNFS_LD @@ -84,7 +84,7 @@ filelayout_get_dserver_offset(struct pnfs_layout_segment *lseg, loff_t offset) BUG(); } -static void filelayout_reset_write(struct nfs_write_data *data) +static void filelayout_reset_write(struct nfs_pgio_data *data) { struct nfs_pgio_header *hdr = data->header; struct rpc_task *task = &data->task; @@ -105,7 +105,7 @@ static void filelayout_reset_write(struct nfs_write_data *data) } } -static void filelayout_reset_read(struct nfs_read_data *data) +static void filelayout_reset_read(struct nfs_pgio_data *data) { struct nfs_pgio_header *hdr = data->header; struct rpc_task *task = &data->task; @@ -243,7 +243,7 @@ wait_on_recovery: /* NFS_PROTO call done callback routines */ static int filelayout_read_done_cb(struct rpc_task *task, - struct nfs_read_data *data) + struct nfs_pgio_data *data) { struct nfs_pgio_header *hdr = data->header; int err; @@ -270,7 +270,7 @@ static int filelayout_read_done_cb(struct rpc_task *task, * rfc5661 is not clear about which credential should be used. */ static void -filelayout_set_layoutcommit(struct nfs_write_data *wdata) +filelayout_set_layoutcommit(struct nfs_pgio_data *wdata) { struct nfs_pgio_header *hdr = wdata->header; @@ -279,7 +279,7 @@ filelayout_set_layoutcommit(struct nfs_write_data *wdata) return; pnfs_set_layoutcommit(wdata); - dprintk("%s ionde %lu pls_end_pos %lu\n", __func__, hdr->inode->i_ino, + dprintk("%s inode %lu pls_end_pos %lu\n", __func__, hdr->inode->i_ino, (unsigned long) NFS_I(hdr->inode)->layout->plh_lwb); } @@ -305,7 +305,7 @@ filelayout_reset_to_mds(struct pnfs_layout_segment *lseg) */ static void filelayout_read_prepare(struct rpc_task *task, void *data) { - struct nfs_read_data *rdata = data; + struct nfs_pgio_data *rdata = data; if (unlikely(test_bit(NFS_CONTEXT_BAD, &rdata->args.context->flags))) { rpc_exit(task, -EIO); @@ -317,7 +317,7 @@ static void filelayout_read_prepare(struct rpc_task *task, void *data) rpc_exit(task, 0); return; } - rdata->read_done_cb = filelayout_read_done_cb; + rdata->pgio_done_cb = filelayout_read_done_cb; if (nfs41_setup_sequence(rdata->ds_clp->cl_session, &rdata->args.seq_args, @@ -331,7 +331,7 @@ static void filelayout_read_prepare(struct rpc_task *task, void *data) static void filelayout_read_call_done(struct rpc_task *task, void *data) { - struct nfs_read_data *rdata = data; + struct nfs_pgio_data *rdata = data; dprintk("--> %s task->tk_status %d\n", __func__, task->tk_status); @@ -347,14 +347,14 @@ static void filelayout_read_call_done(struct rpc_task *task, void *data) static void filelayout_read_count_stats(struct rpc_task *task, void *data) { - struct nfs_read_data *rdata = data; + struct nfs_pgio_data *rdata = data; rpc_count_iostats(task, NFS_SERVER(rdata->header->inode)->client->cl_metrics); } static void filelayout_read_release(void *data) { - struct nfs_read_data *rdata = data; + struct nfs_pgio_data *rdata = data; struct pnfs_layout_hdr *lo = rdata->header->lseg->pls_layout; filelayout_fenceme(lo->plh_inode, lo); @@ -363,7 +363,7 @@ static void filelayout_read_release(void *data) } static int filelayout_write_done_cb(struct rpc_task *task, - struct nfs_write_data *data) + struct nfs_pgio_data *data) { struct nfs_pgio_header *hdr = data->header; int err; @@ -419,7 +419,7 @@ static int filelayout_commit_done_cb(struct rpc_task *task, static void filelayout_write_prepare(struct rpc_task *task, void *data) { - struct nfs_write_data *wdata = data; + struct nfs_pgio_data *wdata = data; if (unlikely(test_bit(NFS_CONTEXT_BAD, &wdata->args.context->flags))) { rpc_exit(task, -EIO); @@ -443,7 +443,7 @@ static void filelayout_write_prepare(struct rpc_task *task, void *data) static void filelayout_write_call_done(struct rpc_task *task, void *data) { - struct nfs_write_data *wdata = data; + struct nfs_pgio_data *wdata = data; if (test_bit(NFS_IOHDR_REDO, &wdata->header->flags) && task->tk_status == 0) { @@ -457,14 +457,14 @@ static void filelayout_write_call_done(struct rpc_task *task, void *data) static void filelayout_write_count_stats(struct rpc_task *task, void *data) { - struct nfs_write_data *wdata = data; + struct nfs_pgio_data *wdata = data; rpc_count_iostats(task, NFS_SERVER(wdata->header->inode)->client->cl_metrics); } static void filelayout_write_release(void *data) { - struct nfs_write_data *wdata = data; + struct nfs_pgio_data *wdata = data; struct pnfs_layout_hdr *lo = wdata->header->lseg->pls_layout; filelayout_fenceme(lo->plh_inode, lo); @@ -529,7 +529,7 @@ static const struct rpc_call_ops filelayout_commit_call_ops = { }; static enum pnfs_try_status -filelayout_read_pagelist(struct nfs_read_data *data) +filelayout_read_pagelist(struct nfs_pgio_data *data) { struct nfs_pgio_header *hdr = data->header; struct pnfs_layout_segment *lseg = hdr->lseg; @@ -560,6 +560,7 @@ filelayout_read_pagelist(struct nfs_read_data *data) /* No multipath support. Use first DS */ atomic_inc(&ds->ds_clp->cl_count); data->ds_clp = ds->ds_clp; + data->ds_idx = idx; fh = nfs4_fl_select_ds_fh(lseg, j); if (fh) data->args.fh = fh; @@ -568,14 +569,14 @@ filelayout_read_pagelist(struct nfs_read_data *data) data->mds_offset = offset; /* Perform an asynchronous read to ds */ - nfs_initiate_read(ds_clnt, data, - &filelayout_read_call_ops, RPC_TASK_SOFTCONN); + nfs_initiate_pgio(ds_clnt, data, + &filelayout_read_call_ops, 0, RPC_TASK_SOFTCONN); return PNFS_ATTEMPTED; } /* Perform async writes. */ static enum pnfs_try_status -filelayout_write_pagelist(struct nfs_write_data *data, int sync) +filelayout_write_pagelist(struct nfs_pgio_data *data, int sync) { struct nfs_pgio_header *hdr = data->header; struct pnfs_layout_segment *lseg = hdr->lseg; @@ -600,20 +601,18 @@ filelayout_write_pagelist(struct nfs_write_data *data, int sync) __func__, hdr->inode->i_ino, sync, (size_t) data->args.count, offset, ds->ds_remotestr, atomic_read(&ds->ds_clp->cl_count)); - data->write_done_cb = filelayout_write_done_cb; + data->pgio_done_cb = filelayout_write_done_cb; atomic_inc(&ds->ds_clp->cl_count); data->ds_clp = ds->ds_clp; + data->ds_idx = idx; fh = nfs4_fl_select_ds_fh(lseg, j); if (fh) data->args.fh = fh; - /* - * Get the file offset on the dserver. Set the write offset to - * this offset and save the original offset. - */ + data->args.offset = filelayout_get_dserver_offset(lseg, offset); /* Perform an asynchronous write */ - nfs_initiate_write(ds_clnt, data, + nfs_initiate_pgio(ds_clnt, data, &filelayout_write_call_ops, sync, RPC_TASK_SOFTCONN); return PNFS_ATTEMPTED; @@ -637,7 +636,6 @@ filelayout_check_layout(struct pnfs_layout_hdr *lo, struct nfs4_deviceid_node *d; struct nfs4_file_layout_dsaddr *dsaddr; int status = -EINVAL; - struct nfs_server *nfss = NFS_SERVER(lo->plh_inode); dprintk("--> %s\n", __func__); @@ -655,7 +653,7 @@ filelayout_check_layout(struct pnfs_layout_hdr *lo, goto out; } - if (!fl->stripe_unit || fl->stripe_unit % PAGE_SIZE) { + if (!fl->stripe_unit) { dprintk("%s Invalid stripe unit (%u)\n", __func__, fl->stripe_unit); goto out; @@ -692,12 +690,6 @@ filelayout_check_layout(struct pnfs_layout_hdr *lo, goto out_put; } - if (fl->stripe_unit % nfss->rsize || fl->stripe_unit % nfss->wsize) { - dprintk("%s Stripe unit (%u) not aligned with rsize %u " - "wsize %u\n", __func__, fl->stripe_unit, nfss->rsize, - nfss->wsize); - } - status = 0; out: dprintk("--> %s returns %d\n", __func__, status); @@ -850,11 +842,15 @@ filelayout_alloc_commit_info(struct pnfs_layout_segment *lseg, { struct nfs4_filelayout_segment *fl = FILELAYOUT_LSEG(lseg); struct pnfs_commit_bucket *buckets; - int size; + int size, i; if (fl->commit_through_mds) return 0; - if (cinfo->ds->nbuckets != 0) { + + size = (fl->stripe_type == STRIPE_SPARSE) ? + fl->dsaddr->ds_num : fl->dsaddr->stripe_count; + + if (cinfo->ds->nbuckets >= size) { /* This assumes there is only one IOMODE_RW lseg. What * we really want to do is have a layout_hdr level * dictionary of <multipath_list4, fh> keys, each @@ -864,30 +860,36 @@ filelayout_alloc_commit_info(struct pnfs_layout_segment *lseg, return 0; } - size = (fl->stripe_type == STRIPE_SPARSE) ? - fl->dsaddr->ds_num : fl->dsaddr->stripe_count; - buckets = kcalloc(size, sizeof(struct pnfs_commit_bucket), gfp_flags); if (!buckets) return -ENOMEM; - else { - int i; + for (i = 0; i < size; i++) { + INIT_LIST_HEAD(&buckets[i].written); + INIT_LIST_HEAD(&buckets[i].committing); + /* mark direct verifier as unset */ + buckets[i].direct_verf.committed = NFS_INVALID_STABLE_HOW; + } - spin_lock(cinfo->lock); - if (cinfo->ds->nbuckets != 0) - kfree(buckets); - else { - cinfo->ds->buckets = buckets; - cinfo->ds->nbuckets = size; - for (i = 0; i < size; i++) { - INIT_LIST_HEAD(&buckets[i].written); - INIT_LIST_HEAD(&buckets[i].committing); - } - } - spin_unlock(cinfo->lock); - return 0; + spin_lock(cinfo->lock); + if (cinfo->ds->nbuckets >= size) + goto out; + for (i = 0; i < cinfo->ds->nbuckets; i++) { + list_splice(&cinfo->ds->buckets[i].written, + &buckets[i].written); + list_splice(&cinfo->ds->buckets[i].committing, + &buckets[i].committing); + buckets[i].direct_verf.committed = + cinfo->ds->buckets[i].direct_verf.committed; + buckets[i].wlseg = cinfo->ds->buckets[i].wlseg; + buckets[i].clseg = cinfo->ds->buckets[i].clseg; } + swap(cinfo->ds->buckets, buckets); + cinfo->ds->nbuckets = size; +out: + spin_unlock(cinfo->lock); + kfree(buckets); + return 0; } static struct pnfs_layout_segment * @@ -915,47 +917,51 @@ filelayout_alloc_lseg(struct pnfs_layout_hdr *layoutid, /* * filelayout_pg_test(). Called by nfs_can_coalesce_requests() * - * return true : coalesce page - * return false : don't coalesce page + * Return 0 if @req cannot be coalesced into @pgio, otherwise return the number + * of bytes (maximum @req->wb_bytes) that can be coalesced. */ -static bool +static size_t filelayout_pg_test(struct nfs_pageio_descriptor *pgio, struct nfs_page *prev, struct nfs_page *req) { + unsigned int size; u64 p_stripe, r_stripe; - u32 stripe_unit; + u32 stripe_offset; + u64 segment_offset = pgio->pg_lseg->pls_range.offset; + u32 stripe_unit = FILELAYOUT_LSEG(pgio->pg_lseg)->stripe_unit; - if (!pnfs_generic_pg_test(pgio, prev, req) || - !nfs_generic_pg_test(pgio, prev, req)) - return false; + /* calls nfs_generic_pg_test */ + size = pnfs_generic_pg_test(pgio, prev, req); + if (!size) + return 0; - p_stripe = (u64)req_offset(prev); - r_stripe = (u64)req_offset(req); - stripe_unit = FILELAYOUT_LSEG(pgio->pg_lseg)->stripe_unit; + /* see if req and prev are in the same stripe */ + if (prev) { + p_stripe = (u64)req_offset(prev) - segment_offset; + r_stripe = (u64)req_offset(req) - segment_offset; + do_div(p_stripe, stripe_unit); + do_div(r_stripe, stripe_unit); - do_div(p_stripe, stripe_unit); - do_div(r_stripe, stripe_unit); + if (p_stripe != r_stripe) + return 0; + } - return (p_stripe == r_stripe); + /* calculate remaining bytes in the current stripe */ + div_u64_rem((u64)req_offset(req) - segment_offset, + stripe_unit, + &stripe_offset); + WARN_ON_ONCE(stripe_offset > stripe_unit); + if (stripe_offset >= stripe_unit) + return 0; + return min(stripe_unit - (unsigned int)stripe_offset, size); } static void filelayout_pg_init_read(struct nfs_pageio_descriptor *pgio, struct nfs_page *req) { - WARN_ON_ONCE(pgio->pg_lseg != NULL); - - if (req->wb_offset != req->wb_pgbase) { - /* - * Handling unaligned pages is difficult, because have to - * somehow split a req in two in certain cases in the - * pg.test code. Avoid this by just not using pnfs - * in this case. - */ - nfs_pageio_reset_read_mds(pgio); - return; - } - pgio->pg_lseg = pnfs_update_layout(pgio->pg_inode, + if (!pgio->pg_lseg) + pgio->pg_lseg = pnfs_update_layout(pgio->pg_inode, req->wb_context, 0, NFS4_MAX_UINT64, @@ -973,11 +979,8 @@ filelayout_pg_init_write(struct nfs_pageio_descriptor *pgio, struct nfs_commit_info cinfo; int status; - WARN_ON_ONCE(pgio->pg_lseg != NULL); - - if (req->wb_offset != req->wb_pgbase) - goto out_mds; - pgio->pg_lseg = pnfs_update_layout(pgio->pg_inode, + if (!pgio->pg_lseg) + pgio->pg_lseg = pnfs_update_layout(pgio->pg_inode, req->wb_context, 0, NFS4_MAX_UINT64, @@ -1067,6 +1070,7 @@ filelayout_choose_commit_list(struct nfs_page *req, */ j = nfs4_fl_calc_j_index(lseg, req_offset(req)); i = select_bucket_index(fl, j); + spin_lock(cinfo->lock); buckets = cinfo->ds->buckets; list = &buckets[i].written; if (list_empty(list)) { @@ -1080,6 +1084,7 @@ filelayout_choose_commit_list(struct nfs_page *req, } set_bit(PG_COMMIT_TO_DS, &req->wb_flags); cinfo->ds->nwritten++; + spin_unlock(cinfo->lock); return list; } @@ -1176,6 +1181,7 @@ transfer_commit_list(struct list_head *src, struct list_head *dst, return ret; } +/* Note called with cinfo->lock held. */ static int filelayout_scan_ds_commit_list(struct pnfs_commit_bucket *bucket, struct nfs_commit_info *cinfo, @@ -1220,15 +1226,18 @@ static void filelayout_recover_commit_reqs(struct list_head *dst, struct nfs_commit_info *cinfo) { struct pnfs_commit_bucket *b; + struct pnfs_layout_segment *freeme; int i; +restart: spin_lock(cinfo->lock); for (i = 0, b = cinfo->ds->buckets; i < cinfo->ds->nbuckets; i++, b++) { if (transfer_commit_list(&b->written, dst, cinfo, 0)) { - spin_unlock(cinfo->lock); - pnfs_put_lseg(b->wlseg); + freeme = b->wlseg; b->wlseg = NULL; - spin_lock(cinfo->lock); + spin_unlock(cinfo->lock); + pnfs_put_lseg(freeme); + goto restart; } } cinfo->ds->nwritten = 0; @@ -1243,6 +1252,7 @@ alloc_ds_commits(struct nfs_commit_info *cinfo, struct list_head *list) struct nfs_commit_data *data; int i, j; unsigned int nreq = 0; + struct pnfs_layout_segment *freeme; fl_cinfo = cinfo->ds; bucket = fl_cinfo->buckets; @@ -1253,8 +1263,10 @@ alloc_ds_commits(struct nfs_commit_info *cinfo, struct list_head *list) if (!data) break; data->ds_commit_index = i; + spin_lock(cinfo->lock); data->lseg = bucket->clseg; bucket->clseg = NULL; + spin_unlock(cinfo->lock); list_add(&data->pages, list); nreq++; } @@ -1264,8 +1276,11 @@ alloc_ds_commits(struct nfs_commit_info *cinfo, struct list_head *list) if (list_empty(&bucket->committing)) continue; nfs_retry_commit(&bucket->committing, bucket->clseg, cinfo); - pnfs_put_lseg(bucket->clseg); + spin_lock(cinfo->lock); + freeme = bucket->clseg; bucket->clseg = NULL; + spin_unlock(cinfo->lock); + pnfs_put_lseg(freeme); } /* Caller will clean up entries put on list */ return nreq; @@ -1330,7 +1345,7 @@ filelayout_alloc_layout_hdr(struct inode *inode, gfp_t gfp_flags) struct nfs4_filelayout *flo; flo = kzalloc(sizeof(*flo), gfp_flags); - return &flo->generic_hdr; + return flo != NULL ? &flo->generic_hdr : NULL; } static void diff --git a/fs/nfs/nfs4filelayout.h b/fs/nfs/filelayout/filelayout.h index cebd20e7e923..ffbddf2219ea 100644 --- a/fs/nfs/nfs4filelayout.h +++ b/fs/nfs/filelayout/filelayout.h @@ -30,7 +30,7 @@ #ifndef FS_NFS_NFS4FILELAYOUT_H #define FS_NFS_NFS4FILELAYOUT_H -#include "pnfs.h" +#include "../pnfs.h" /* * Default data server connection timeout and retrans vaules. diff --git a/fs/nfs/nfs4filelayoutdev.c b/fs/nfs/filelayout/filelayoutdev.c index b9c61efe9660..44bf0140a4c7 100644 --- a/fs/nfs/nfs4filelayoutdev.c +++ b/fs/nfs/filelayout/filelayoutdev.c @@ -33,9 +33,9 @@ #include <linux/module.h> #include <linux/sunrpc/addr.h> -#include "internal.h" -#include "nfs4session.h" -#include "nfs4filelayout.h" +#include "../internal.h" +#include "../nfs4session.h" +#include "filelayout.h" #define NFSDBG_FACILITY NFSDBG_PNFS_LD diff --git a/fs/nfs/getroot.c b/fs/nfs/getroot.c index 66984a9aafaa..b94f80420a58 100644 --- a/fs/nfs/getroot.c +++ b/fs/nfs/getroot.c @@ -120,7 +120,8 @@ struct dentry *nfs_get_root(struct super_block *sb, struct nfs_fh *mntfh, security_d_instantiate(ret, inode); spin_lock(&ret->d_lock); - if (IS_ROOT(ret) && !(ret->d_flags & DCACHE_NFSFS_RENAMED)) { + if (IS_ROOT(ret) && !ret->d_fsdata && + !(ret->d_flags & DCACHE_NFSFS_RENAMED)) { ret->d_fsdata = name; name = NULL; } diff --git a/fs/nfs/inode.c b/fs/nfs/inode.c index e6f7398d2b3c..c496f8a74639 100644 --- a/fs/nfs/inode.c +++ b/fs/nfs/inode.c @@ -1575,18 +1575,20 @@ static int nfs_update_inode(struct inode *inode, struct nfs_fattr *fattr) inode->i_version = fattr->change_attr; } } else if (server->caps & NFS_CAP_CHANGE_ATTR) - invalid |= save_cache_validity; + nfsi->cache_validity |= save_cache_validity; if (fattr->valid & NFS_ATTR_FATTR_MTIME) { memcpy(&inode->i_mtime, &fattr->mtime, sizeof(inode->i_mtime)); } else if (server->caps & NFS_CAP_MTIME) - invalid |= save_cache_validity & (NFS_INO_INVALID_ATTR + nfsi->cache_validity |= save_cache_validity & + (NFS_INO_INVALID_ATTR | NFS_INO_REVAL_FORCED); if (fattr->valid & NFS_ATTR_FATTR_CTIME) { memcpy(&inode->i_ctime, &fattr->ctime, sizeof(inode->i_ctime)); } else if (server->caps & NFS_CAP_CTIME) - invalid |= save_cache_validity & (NFS_INO_INVALID_ATTR + nfsi->cache_validity |= save_cache_validity & + (NFS_INO_INVALID_ATTR | NFS_INO_REVAL_FORCED); /* Check if our cached file size is stale */ @@ -1608,7 +1610,8 @@ static int nfs_update_inode(struct inode *inode, struct nfs_fattr *fattr) (long long)new_isize); } } else - invalid |= save_cache_validity & (NFS_INO_INVALID_ATTR + nfsi->cache_validity |= save_cache_validity & + (NFS_INO_INVALID_ATTR | NFS_INO_REVAL_PAGECACHE | NFS_INO_REVAL_FORCED); @@ -1616,7 +1619,8 @@ static int nfs_update_inode(struct inode *inode, struct nfs_fattr *fattr) if (fattr->valid & NFS_ATTR_FATTR_ATIME) memcpy(&inode->i_atime, &fattr->atime, sizeof(inode->i_atime)); else if (server->caps & NFS_CAP_ATIME) - invalid |= save_cache_validity & (NFS_INO_INVALID_ATIME + nfsi->cache_validity |= save_cache_validity & + (NFS_INO_INVALID_ATIME | NFS_INO_REVAL_FORCED); if (fattr->valid & NFS_ATTR_FATTR_MODE) { @@ -1627,7 +1631,8 @@ static int nfs_update_inode(struct inode *inode, struct nfs_fattr *fattr) invalid |= NFS_INO_INVALID_ATTR|NFS_INO_INVALID_ACCESS|NFS_INO_INVALID_ACL; } } else if (server->caps & NFS_CAP_MODE) - invalid |= save_cache_validity & (NFS_INO_INVALID_ATTR + nfsi->cache_validity |= save_cache_validity & + (NFS_INO_INVALID_ATTR | NFS_INO_INVALID_ACCESS | NFS_INO_INVALID_ACL | NFS_INO_REVAL_FORCED); @@ -1638,7 +1643,8 @@ static int nfs_update_inode(struct inode *inode, struct nfs_fattr *fattr) inode->i_uid = fattr->uid; } } else if (server->caps & NFS_CAP_OWNER) - invalid |= save_cache_validity & (NFS_INO_INVALID_ATTR + nfsi->cache_validity |= save_cache_validity & + (NFS_INO_INVALID_ATTR | NFS_INO_INVALID_ACCESS | NFS_INO_INVALID_ACL | NFS_INO_REVAL_FORCED); @@ -1649,7 +1655,8 @@ static int nfs_update_inode(struct inode *inode, struct nfs_fattr *fattr) inode->i_gid = fattr->gid; } } else if (server->caps & NFS_CAP_OWNER_GROUP) - invalid |= save_cache_validity & (NFS_INO_INVALID_ATTR + nfsi->cache_validity |= save_cache_validity & + (NFS_INO_INVALID_ATTR | NFS_INO_INVALID_ACCESS | NFS_INO_INVALID_ACL | NFS_INO_REVAL_FORCED); @@ -1662,7 +1669,8 @@ static int nfs_update_inode(struct inode *inode, struct nfs_fattr *fattr) set_nlink(inode, fattr->nlink); } } else if (server->caps & NFS_CAP_NLINK) - invalid |= save_cache_validity & (NFS_INO_INVALID_ATTR + nfsi->cache_validity |= save_cache_validity & + (NFS_INO_INVALID_ATTR | NFS_INO_REVAL_FORCED); if (fattr->valid & NFS_ATTR_FATTR_SPACE_USED) { diff --git a/fs/nfs/internal.h b/fs/nfs/internal.h index dd8bfc2e2464..8b69cba1bb04 100644 --- a/fs/nfs/internal.h +++ b/fs/nfs/internal.h @@ -231,13 +231,20 @@ extern void nfs_destroy_writepagecache(void); extern int __init nfs_init_directcache(void); extern void nfs_destroy_directcache(void); -extern bool nfs_pgarray_set(struct nfs_page_array *p, unsigned int pagecount); extern void nfs_pgheader_init(struct nfs_pageio_descriptor *desc, struct nfs_pgio_header *hdr, void (*release)(struct nfs_pgio_header *hdr)); void nfs_set_pgio_error(struct nfs_pgio_header *hdr, int error, loff_t pos); int nfs_iocounter_wait(struct nfs_io_counter *c); +extern const struct nfs_pageio_ops nfs_pgio_rw_ops; +struct nfs_rw_header *nfs_rw_header_alloc(const struct nfs_rw_ops *); +void nfs_rw_header_free(struct nfs_pgio_header *); +void nfs_pgio_data_release(struct nfs_pgio_data *); +int nfs_generic_pgio(struct nfs_pageio_descriptor *, struct nfs_pgio_header *); +int nfs_initiate_pgio(struct rpc_clnt *, struct nfs_pgio_data *, + const struct rpc_call_ops *, int, int); + static inline void nfs_iocounter_init(struct nfs_io_counter *c) { c->flags = 0; @@ -395,19 +402,11 @@ extern int nfs4_get_rootfh(struct nfs_server *server, struct nfs_fh *mntfh, bool struct nfs_pgio_completion_ops; /* read.c */ -extern struct nfs_read_header *nfs_readhdr_alloc(void); -extern void nfs_readhdr_free(struct nfs_pgio_header *hdr); extern void nfs_pageio_init_read(struct nfs_pageio_descriptor *pgio, - struct inode *inode, + struct inode *inode, bool force_mds, const struct nfs_pgio_completion_ops *compl_ops); -extern int nfs_initiate_read(struct rpc_clnt *clnt, - struct nfs_read_data *data, - const struct rpc_call_ops *call_ops, int flags); extern void nfs_read_prepare(struct rpc_task *task, void *calldata); -extern int nfs_generic_pagein(struct nfs_pageio_descriptor *desc, - struct nfs_pgio_header *hdr); extern void nfs_pageio_reset_read_mds(struct nfs_pageio_descriptor *pgio); -extern void nfs_readdata_release(struct nfs_read_data *rdata); /* super.c */ void nfs_clone_super(struct super_block *, struct nfs_mount_info *); @@ -422,19 +421,10 @@ int nfs_remount(struct super_block *sb, int *flags, char *raw_data); /* write.c */ extern void nfs_pageio_init_write(struct nfs_pageio_descriptor *pgio, - struct inode *inode, int ioflags, + struct inode *inode, int ioflags, bool force_mds, const struct nfs_pgio_completion_ops *compl_ops); -extern struct nfs_write_header *nfs_writehdr_alloc(void); -extern void nfs_writehdr_free(struct nfs_pgio_header *hdr); -extern int nfs_generic_flush(struct nfs_pageio_descriptor *desc, - struct nfs_pgio_header *hdr); extern void nfs_pageio_reset_write_mds(struct nfs_pageio_descriptor *pgio); -extern void nfs_writedata_release(struct nfs_write_data *wdata); extern void nfs_commit_free(struct nfs_commit_data *p); -extern int nfs_initiate_write(struct rpc_clnt *clnt, - struct nfs_write_data *data, - const struct rpc_call_ops *call_ops, - int how, int flags); extern void nfs_write_prepare(struct rpc_task *task, void *calldata); extern void nfs_commit_prepare(struct rpc_task *task, void *calldata); extern int nfs_initiate_commit(struct rpc_clnt *clnt, @@ -447,6 +437,7 @@ extern void nfs_init_commit(struct nfs_commit_data *data, struct nfs_commit_info *cinfo); int nfs_scan_commit_list(struct list_head *src, struct list_head *dst, struct nfs_commit_info *cinfo, int max); +unsigned long nfs_reqs_to_commit(struct nfs_commit_info *); int nfs_scan_commit(struct inode *inode, struct list_head *dst, struct nfs_commit_info *cinfo); void nfs_mark_request_commit(struct nfs_page *req, @@ -492,7 +483,7 @@ static inline void nfs_inode_dio_wait(struct inode *inode) extern ssize_t nfs_dreq_bytes_left(struct nfs_direct_req *dreq); /* nfs4proc.c */ -extern void __nfs4_read_done_cb(struct nfs_read_data *); +extern void __nfs4_read_done_cb(struct nfs_pgio_data *); extern struct nfs_client *nfs4_init_client(struct nfs_client *clp, const struct rpc_timeout *timeparms, const char *ip_addr); diff --git a/fs/nfs/nfs2xdr.c b/fs/nfs/nfs2xdr.c index 62db136339ea..5f61b83f4a1c 100644 --- a/fs/nfs/nfs2xdr.c +++ b/fs/nfs/nfs2xdr.c @@ -103,7 +103,7 @@ static void print_overflow_msg(const char *func, const struct xdr_stream *xdr) /* * typedef opaque nfsdata<>; */ -static int decode_nfsdata(struct xdr_stream *xdr, struct nfs_readres *result) +static int decode_nfsdata(struct xdr_stream *xdr, struct nfs_pgio_res *result) { u32 recvd, count; __be32 *p; @@ -613,7 +613,7 @@ static void nfs2_xdr_enc_readlinkargs(struct rpc_rqst *req, * }; */ static void encode_readargs(struct xdr_stream *xdr, - const struct nfs_readargs *args) + const struct nfs_pgio_args *args) { u32 offset = args->offset; u32 count = args->count; @@ -629,7 +629,7 @@ static void encode_readargs(struct xdr_stream *xdr, static void nfs2_xdr_enc_readargs(struct rpc_rqst *req, struct xdr_stream *xdr, - const struct nfs_readargs *args) + const struct nfs_pgio_args *args) { encode_readargs(xdr, args); prepare_reply_buffer(req, args->pages, args->pgbase, @@ -649,7 +649,7 @@ static void nfs2_xdr_enc_readargs(struct rpc_rqst *req, * }; */ static void encode_writeargs(struct xdr_stream *xdr, - const struct nfs_writeargs *args) + const struct nfs_pgio_args *args) { u32 offset = args->offset; u32 count = args->count; @@ -669,7 +669,7 @@ static void encode_writeargs(struct xdr_stream *xdr, static void nfs2_xdr_enc_writeargs(struct rpc_rqst *req, struct xdr_stream *xdr, - const struct nfs_writeargs *args) + const struct nfs_pgio_args *args) { encode_writeargs(xdr, args); xdr->buf->flags |= XDRBUF_WRITE; @@ -857,7 +857,7 @@ out_default: * }; */ static int nfs2_xdr_dec_readres(struct rpc_rqst *req, struct xdr_stream *xdr, - struct nfs_readres *result) + struct nfs_pgio_res *result) { enum nfs_stat status; int error; @@ -878,7 +878,7 @@ out_default: } static int nfs2_xdr_dec_writeres(struct rpc_rqst *req, struct xdr_stream *xdr, - struct nfs_writeres *result) + struct nfs_pgio_res *result) { /* All NFSv2 writes are "file sync" writes */ result->verf->committed = NFS_FILE_SYNC; diff --git a/fs/nfs/nfs3proc.c b/fs/nfs/nfs3proc.c index db60149c4579..e7daa42bbc86 100644 --- a/fs/nfs/nfs3proc.c +++ b/fs/nfs/nfs3proc.c @@ -795,7 +795,7 @@ nfs3_proc_pathconf(struct nfs_server *server, struct nfs_fh *fhandle, return status; } -static int nfs3_read_done(struct rpc_task *task, struct nfs_read_data *data) +static int nfs3_read_done(struct rpc_task *task, struct nfs_pgio_data *data) { struct inode *inode = data->header->inode; @@ -807,18 +807,18 @@ static int nfs3_read_done(struct rpc_task *task, struct nfs_read_data *data) return 0; } -static void nfs3_proc_read_setup(struct nfs_read_data *data, struct rpc_message *msg) +static void nfs3_proc_read_setup(struct nfs_pgio_data *data, struct rpc_message *msg) { msg->rpc_proc = &nfs3_procedures[NFS3PROC_READ]; } -static int nfs3_proc_read_rpc_prepare(struct rpc_task *task, struct nfs_read_data *data) +static int nfs3_proc_pgio_rpc_prepare(struct rpc_task *task, struct nfs_pgio_data *data) { rpc_call_start(task); return 0; } -static int nfs3_write_done(struct rpc_task *task, struct nfs_write_data *data) +static int nfs3_write_done(struct rpc_task *task, struct nfs_pgio_data *data) { struct inode *inode = data->header->inode; @@ -829,17 +829,11 @@ static int nfs3_write_done(struct rpc_task *task, struct nfs_write_data *data) return 0; } -static void nfs3_proc_write_setup(struct nfs_write_data *data, struct rpc_message *msg) +static void nfs3_proc_write_setup(struct nfs_pgio_data *data, struct rpc_message *msg) { msg->rpc_proc = &nfs3_procedures[NFS3PROC_WRITE]; } -static int nfs3_proc_write_rpc_prepare(struct rpc_task *task, struct nfs_write_data *data) -{ - rpc_call_start(task); - return 0; -} - static void nfs3_proc_commit_rpc_prepare(struct rpc_task *task, struct nfs_commit_data *data) { rpc_call_start(task); @@ -946,13 +940,10 @@ const struct nfs_rpc_ops nfs_v3_clientops = { .fsinfo = nfs3_proc_fsinfo, .pathconf = nfs3_proc_pathconf, .decode_dirent = nfs3_decode_dirent, + .pgio_rpc_prepare = nfs3_proc_pgio_rpc_prepare, .read_setup = nfs3_proc_read_setup, - .read_pageio_init = nfs_pageio_init_read, - .read_rpc_prepare = nfs3_proc_read_rpc_prepare, .read_done = nfs3_read_done, .write_setup = nfs3_proc_write_setup, - .write_pageio_init = nfs_pageio_init_write, - .write_rpc_prepare = nfs3_proc_write_rpc_prepare, .write_done = nfs3_write_done, .commit_setup = nfs3_proc_commit_setup, .commit_rpc_prepare = nfs3_proc_commit_rpc_prepare, diff --git a/fs/nfs/nfs3xdr.c b/fs/nfs/nfs3xdr.c index fa6d72131c19..8f4cbe7f4aa8 100644 --- a/fs/nfs/nfs3xdr.c +++ b/fs/nfs/nfs3xdr.c @@ -953,7 +953,7 @@ static void nfs3_xdr_enc_readlink3args(struct rpc_rqst *req, * }; */ static void encode_read3args(struct xdr_stream *xdr, - const struct nfs_readargs *args) + const struct nfs_pgio_args *args) { __be32 *p; @@ -966,7 +966,7 @@ static void encode_read3args(struct xdr_stream *xdr, static void nfs3_xdr_enc_read3args(struct rpc_rqst *req, struct xdr_stream *xdr, - const struct nfs_readargs *args) + const struct nfs_pgio_args *args) { encode_read3args(xdr, args); prepare_reply_buffer(req, args->pages, args->pgbase, @@ -992,7 +992,7 @@ static void nfs3_xdr_enc_read3args(struct rpc_rqst *req, * }; */ static void encode_write3args(struct xdr_stream *xdr, - const struct nfs_writeargs *args) + const struct nfs_pgio_args *args) { __be32 *p; @@ -1008,7 +1008,7 @@ static void encode_write3args(struct xdr_stream *xdr, static void nfs3_xdr_enc_write3args(struct rpc_rqst *req, struct xdr_stream *xdr, - const struct nfs_writeargs *args) + const struct nfs_pgio_args *args) { encode_write3args(xdr, args); xdr->buf->flags |= XDRBUF_WRITE; @@ -1589,7 +1589,7 @@ out_default: * }; */ static int decode_read3resok(struct xdr_stream *xdr, - struct nfs_readres *result) + struct nfs_pgio_res *result) { u32 eof, count, ocount, recvd; __be32 *p; @@ -1625,7 +1625,7 @@ out_overflow: } static int nfs3_xdr_dec_read3res(struct rpc_rqst *req, struct xdr_stream *xdr, - struct nfs_readres *result) + struct nfs_pgio_res *result) { enum nfs_stat status; int error; @@ -1673,7 +1673,7 @@ out_status: * }; */ static int decode_write3resok(struct xdr_stream *xdr, - struct nfs_writeres *result) + struct nfs_pgio_res *result) { __be32 *p; @@ -1697,7 +1697,7 @@ out_eio: } static int nfs3_xdr_dec_write3res(struct rpc_rqst *req, struct xdr_stream *xdr, - struct nfs_writeres *result) + struct nfs_pgio_res *result) { enum nfs_stat status; int error; diff --git a/fs/nfs/nfs4_fs.h b/fs/nfs/nfs4_fs.h index e1d1badbe53c..f63cb87cd730 100644 --- a/fs/nfs/nfs4_fs.h +++ b/fs/nfs/nfs4_fs.h @@ -337,7 +337,7 @@ nfs4_state_protect(struct nfs_client *clp, unsigned long sp4_mode, */ static inline void nfs4_state_protect_write(struct nfs_client *clp, struct rpc_clnt **clntp, - struct rpc_message *msg, struct nfs_write_data *wdata) + struct rpc_message *msg, struct nfs_pgio_data *wdata) { if (_nfs4_state_protect(clp, NFS_SP4_MACH_CRED_WRITE, clntp, msg) && !test_bit(NFS_SP4_MACH_CRED_COMMIT, &clp->cl_sp4_flags)) @@ -369,7 +369,7 @@ nfs4_state_protect(struct nfs_client *clp, unsigned long sp4_flags, static inline void nfs4_state_protect_write(struct nfs_client *clp, struct rpc_clnt **clntp, - struct rpc_message *msg, struct nfs_write_data *wdata) + struct rpc_message *msg, struct nfs_pgio_data *wdata) { } #endif /* CONFIG_NFS_V4_1 */ diff --git a/fs/nfs/nfs4file.c b/fs/nfs/nfs4file.c index 8de3407e0360..464db9dd6318 100644 --- a/fs/nfs/nfs4file.c +++ b/fs/nfs/nfs4file.c @@ -100,8 +100,7 @@ nfs4_file_fsync(struct file *file, loff_t start, loff_t end, int datasync) break; mutex_lock(&inode->i_mutex); ret = nfs_file_fsync_commit(file, start, end, datasync); - if (!ret && !datasync) - /* application has asked for meta-data sync */ + if (!ret) ret = pnfs_layoutcommit_inode(inode, true); mutex_unlock(&inode->i_mutex); /* diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c index 397be39c6dc8..285ad5334018 100644 --- a/fs/nfs/nfs4proc.c +++ b/fs/nfs/nfs4proc.c @@ -2027,7 +2027,7 @@ static int _nfs4_proc_open(struct nfs4_opendata *data) return status; } if (!(o_res->f_attr->valid & NFS_ATTR_FATTR)) - _nfs4_proc_getattr(server, &o_res->fh, o_res->f_attr, o_res->f_label); + nfs4_proc_getattr(server, &o_res->fh, o_res->f_attr, o_res->f_label); return 0; } @@ -2750,7 +2750,7 @@ static void nfs4_close_context(struct nfs_open_context *ctx, int is_sync) #define FATTR4_WORD1_NFS40_MASK (2*FATTR4_WORD1_MOUNTED_ON_FILEID - 1UL) #define FATTR4_WORD2_NFS41_MASK (2*FATTR4_WORD2_SUPPATTR_EXCLCREAT - 1UL) -#define FATTR4_WORD2_NFS42_MASK (2*FATTR4_WORD2_CHANGE_SECURITY_LABEL - 1UL) +#define FATTR4_WORD2_NFS42_MASK (2*FATTR4_WORD2_SECURITY_LABEL - 1UL) static int _nfs4_server_capabilities(struct nfs_server *server, struct nfs_fh *fhandle) { @@ -4033,12 +4033,12 @@ static bool nfs4_error_stateid_expired(int err) return false; } -void __nfs4_read_done_cb(struct nfs_read_data *data) +void __nfs4_read_done_cb(struct nfs_pgio_data *data) { nfs_invalidate_atime(data->header->inode); } -static int nfs4_read_done_cb(struct rpc_task *task, struct nfs_read_data *data) +static int nfs4_read_done_cb(struct rpc_task *task, struct nfs_pgio_data *data) { struct nfs_server *server = NFS_SERVER(data->header->inode); @@ -4055,7 +4055,7 @@ static int nfs4_read_done_cb(struct rpc_task *task, struct nfs_read_data *data) } static bool nfs4_read_stateid_changed(struct rpc_task *task, - struct nfs_readargs *args) + struct nfs_pgio_args *args) { if (!nfs4_error_stateid_expired(task->tk_status) || @@ -4068,7 +4068,7 @@ static bool nfs4_read_stateid_changed(struct rpc_task *task, return true; } -static int nfs4_read_done(struct rpc_task *task, struct nfs_read_data *data) +static int nfs4_read_done(struct rpc_task *task, struct nfs_pgio_data *data) { dprintk("--> %s\n", __func__); @@ -4077,19 +4077,19 @@ static int nfs4_read_done(struct rpc_task *task, struct nfs_read_data *data) return -EAGAIN; if (nfs4_read_stateid_changed(task, &data->args)) return -EAGAIN; - return data->read_done_cb ? data->read_done_cb(task, data) : + return data->pgio_done_cb ? data->pgio_done_cb(task, data) : nfs4_read_done_cb(task, data); } -static void nfs4_proc_read_setup(struct nfs_read_data *data, struct rpc_message *msg) +static void nfs4_proc_read_setup(struct nfs_pgio_data *data, struct rpc_message *msg) { data->timestamp = jiffies; - data->read_done_cb = nfs4_read_done_cb; + data->pgio_done_cb = nfs4_read_done_cb; msg->rpc_proc = &nfs4_procedures[NFSPROC4_CLNT_READ]; nfs4_init_sequence(&data->args.seq_args, &data->res.seq_res, 0); } -static int nfs4_proc_read_rpc_prepare(struct rpc_task *task, struct nfs_read_data *data) +static int nfs4_proc_pgio_rpc_prepare(struct rpc_task *task, struct nfs_pgio_data *data) { if (nfs4_setup_sequence(NFS_SERVER(data->header->inode), &data->args.seq_args, @@ -4097,14 +4097,14 @@ static int nfs4_proc_read_rpc_prepare(struct rpc_task *task, struct nfs_read_dat task)) return 0; if (nfs4_set_rw_stateid(&data->args.stateid, data->args.context, - data->args.lock_context, FMODE_READ) == -EIO) + data->args.lock_context, data->header->rw_ops->rw_mode) == -EIO) return -EIO; if (unlikely(test_bit(NFS_CONTEXT_BAD, &data->args.context->flags))) return -EIO; return 0; } -static int nfs4_write_done_cb(struct rpc_task *task, struct nfs_write_data *data) +static int nfs4_write_done_cb(struct rpc_task *task, struct nfs_pgio_data *data) { struct inode *inode = data->header->inode; @@ -4121,7 +4121,7 @@ static int nfs4_write_done_cb(struct rpc_task *task, struct nfs_write_data *data } static bool nfs4_write_stateid_changed(struct rpc_task *task, - struct nfs_writeargs *args) + struct nfs_pgio_args *args) { if (!nfs4_error_stateid_expired(task->tk_status) || @@ -4134,18 +4134,18 @@ static bool nfs4_write_stateid_changed(struct rpc_task *task, return true; } -static int nfs4_write_done(struct rpc_task *task, struct nfs_write_data *data) +static int nfs4_write_done(struct rpc_task *task, struct nfs_pgio_data *data) { if (!nfs4_sequence_done(task, &data->res.seq_res)) return -EAGAIN; if (nfs4_write_stateid_changed(task, &data->args)) return -EAGAIN; - return data->write_done_cb ? data->write_done_cb(task, data) : + return data->pgio_done_cb ? data->pgio_done_cb(task, data) : nfs4_write_done_cb(task, data); } static -bool nfs4_write_need_cache_consistency_data(const struct nfs_write_data *data) +bool nfs4_write_need_cache_consistency_data(const struct nfs_pgio_data *data) { const struct nfs_pgio_header *hdr = data->header; @@ -4158,7 +4158,7 @@ bool nfs4_write_need_cache_consistency_data(const struct nfs_write_data *data) return nfs4_have_delegation(hdr->inode, FMODE_READ) == 0; } -static void nfs4_proc_write_setup(struct nfs_write_data *data, struct rpc_message *msg) +static void nfs4_proc_write_setup(struct nfs_pgio_data *data, struct rpc_message *msg) { struct nfs_server *server = NFS_SERVER(data->header->inode); @@ -4168,8 +4168,8 @@ static void nfs4_proc_write_setup(struct nfs_write_data *data, struct rpc_messag } else data->args.bitmask = server->cache_consistency_bitmask; - if (!data->write_done_cb) - data->write_done_cb = nfs4_write_done_cb; + if (!data->pgio_done_cb) + data->pgio_done_cb = nfs4_write_done_cb; data->res.server = server; data->timestamp = jiffies; @@ -4177,21 +4177,6 @@ static void nfs4_proc_write_setup(struct nfs_write_data *data, struct rpc_messag nfs4_init_sequence(&data->args.seq_args, &data->res.seq_res, 1); } -static int nfs4_proc_write_rpc_prepare(struct rpc_task *task, struct nfs_write_data *data) -{ - if (nfs4_setup_sequence(NFS_SERVER(data->header->inode), - &data->args.seq_args, - &data->res.seq_res, - task)) - return 0; - if (nfs4_set_rw_stateid(&data->args.stateid, data->args.context, - data->args.lock_context, FMODE_WRITE) == -EIO) - return -EIO; - if (unlikely(test_bit(NFS_CONTEXT_BAD, &data->args.context->flags))) - return -EIO; - return 0; -} - static void nfs4_proc_commit_rpc_prepare(struct rpc_task *task, struct nfs_commit_data *data) { nfs4_setup_sequence(NFS_SERVER(data->inode), @@ -8432,13 +8417,10 @@ const struct nfs_rpc_ops nfs_v4_clientops = { .pathconf = nfs4_proc_pathconf, .set_capabilities = nfs4_server_capabilities, .decode_dirent = nfs4_decode_dirent, + .pgio_rpc_prepare = nfs4_proc_pgio_rpc_prepare, .read_setup = nfs4_proc_read_setup, - .read_pageio_init = pnfs_pageio_init_read, - .read_rpc_prepare = nfs4_proc_read_rpc_prepare, .read_done = nfs4_read_done, .write_setup = nfs4_proc_write_setup, - .write_pageio_init = pnfs_pageio_init_write, - .write_rpc_prepare = nfs4_proc_write_rpc_prepare, .write_done = nfs4_write_done, .commit_setup = nfs4_proc_commit_setup, .commit_rpc_prepare = nfs4_proc_commit_rpc_prepare, diff --git a/fs/nfs/nfs4state.c b/fs/nfs/nfs4state.c index c0583b9bef71..848f6853c59e 100644 --- a/fs/nfs/nfs4state.c +++ b/fs/nfs/nfs4state.c @@ -1456,7 +1456,7 @@ static int nfs4_reclaim_open_state(struct nfs4_state_owner *sp, const struct nfs * server that doesn't support a grace period. */ spin_lock(&sp->so_lock); - write_seqcount_begin(&sp->so_reclaim_seqcount); + raw_write_seqcount_begin(&sp->so_reclaim_seqcount); restart: list_for_each_entry(state, &sp->so_states, open_states) { if (!test_and_clear_bit(ops->state_flag_bit, &state->flags)) @@ -1519,13 +1519,13 @@ restart: spin_lock(&sp->so_lock); goto restart; } - write_seqcount_end(&sp->so_reclaim_seqcount); + raw_write_seqcount_end(&sp->so_reclaim_seqcount); spin_unlock(&sp->so_lock); return 0; out_err: nfs4_put_open_state(state); spin_lock(&sp->so_lock); - write_seqcount_end(&sp->so_reclaim_seqcount); + raw_write_seqcount_end(&sp->so_reclaim_seqcount); spin_unlock(&sp->so_lock); return status; } diff --git a/fs/nfs/nfs4trace.h b/fs/nfs/nfs4trace.h index 849cf146db30..0a744f3a86f6 100644 --- a/fs/nfs/nfs4trace.h +++ b/fs/nfs/nfs4trace.h @@ -932,7 +932,7 @@ DEFINE_NFS4_IDMAP_EVENT(nfs4_map_gid_to_group); DECLARE_EVENT_CLASS(nfs4_read_event, TP_PROTO( - const struct nfs_read_data *data, + const struct nfs_pgio_data *data, int error ), @@ -972,7 +972,7 @@ DECLARE_EVENT_CLASS(nfs4_read_event, #define DEFINE_NFS4_READ_EVENT(name) \ DEFINE_EVENT(nfs4_read_event, name, \ TP_PROTO( \ - const struct nfs_read_data *data, \ + const struct nfs_pgio_data *data, \ int error \ ), \ TP_ARGS(data, error)) @@ -983,7 +983,7 @@ DEFINE_NFS4_READ_EVENT(nfs4_pnfs_read); DECLARE_EVENT_CLASS(nfs4_write_event, TP_PROTO( - const struct nfs_write_data *data, + const struct nfs_pgio_data *data, int error ), @@ -1024,7 +1024,7 @@ DECLARE_EVENT_CLASS(nfs4_write_event, #define DEFINE_NFS4_WRITE_EVENT(name) \ DEFINE_EVENT(nfs4_write_event, name, \ TP_PROTO( \ - const struct nfs_write_data *data, \ + const struct nfs_pgio_data *data, \ int error \ ), \ TP_ARGS(data, error)) diff --git a/fs/nfs/nfs4xdr.c b/fs/nfs/nfs4xdr.c index 73ce8d4fe2c8..939ae606cfa4 100644 --- a/fs/nfs/nfs4xdr.c +++ b/fs/nfs/nfs4xdr.c @@ -1556,7 +1556,8 @@ static void encode_putrootfh(struct xdr_stream *xdr, struct compound_hdr *hdr) encode_op_hdr(xdr, OP_PUTROOTFH, decode_putrootfh_maxsz, hdr); } -static void encode_read(struct xdr_stream *xdr, const struct nfs_readargs *args, struct compound_hdr *hdr) +static void encode_read(struct xdr_stream *xdr, const struct nfs_pgio_args *args, + struct compound_hdr *hdr) { __be32 *p; @@ -1701,7 +1702,8 @@ static void encode_setclientid_confirm(struct xdr_stream *xdr, const struct nfs4 encode_nfs4_verifier(xdr, &arg->confirm); } -static void encode_write(struct xdr_stream *xdr, const struct nfs_writeargs *args, struct compound_hdr *hdr) +static void encode_write(struct xdr_stream *xdr, const struct nfs_pgio_args *args, + struct compound_hdr *hdr) { __be32 *p; @@ -2451,7 +2453,7 @@ static void nfs4_xdr_enc_readdir(struct rpc_rqst *req, struct xdr_stream *xdr, * Encode a READ request */ static void nfs4_xdr_enc_read(struct rpc_rqst *req, struct xdr_stream *xdr, - struct nfs_readargs *args) + struct nfs_pgio_args *args) { struct compound_hdr hdr = { .minorversion = nfs4_xdr_minorversion(&args->seq_args), @@ -2513,7 +2515,7 @@ static void nfs4_xdr_enc_getacl(struct rpc_rqst *req, struct xdr_stream *xdr, * Encode a WRITE request */ static void nfs4_xdr_enc_write(struct rpc_rqst *req, struct xdr_stream *xdr, - struct nfs_writeargs *args) + struct nfs_pgio_args *args) { struct compound_hdr hdr = { .minorversion = nfs4_xdr_minorversion(&args->seq_args), @@ -5085,7 +5087,8 @@ static int decode_putrootfh(struct xdr_stream *xdr) return decode_op_hdr(xdr, OP_PUTROOTFH); } -static int decode_read(struct xdr_stream *xdr, struct rpc_rqst *req, struct nfs_readres *res) +static int decode_read(struct xdr_stream *xdr, struct rpc_rqst *req, + struct nfs_pgio_res *res) { __be32 *p; uint32_t count, eof, recvd; @@ -5339,7 +5342,7 @@ static int decode_setclientid_confirm(struct xdr_stream *xdr) return decode_op_hdr(xdr, OP_SETCLIENTID_CONFIRM); } -static int decode_write(struct xdr_stream *xdr, struct nfs_writeres *res) +static int decode_write(struct xdr_stream *xdr, struct nfs_pgio_res *res) { __be32 *p; int status; @@ -6636,7 +6639,7 @@ out: * Decode Read response */ static int nfs4_xdr_dec_read(struct rpc_rqst *rqstp, struct xdr_stream *xdr, - struct nfs_readres *res) + struct nfs_pgio_res *res) { struct compound_hdr hdr; int status; @@ -6661,7 +6664,7 @@ out: * Decode WRITE response */ static int nfs4_xdr_dec_write(struct rpc_rqst *rqstp, struct xdr_stream *xdr, - struct nfs_writeres *res) + struct nfs_pgio_res *res) { struct compound_hdr hdr; int status; diff --git a/fs/nfs/objlayout/objio_osd.c b/fs/nfs/objlayout/objio_osd.c index 5457745dd4f1..611320753db2 100644 --- a/fs/nfs/objlayout/objio_osd.c +++ b/fs/nfs/objlayout/objio_osd.c @@ -439,7 +439,7 @@ static void _read_done(struct ore_io_state *ios, void *private) objlayout_read_done(&objios->oir, status, objios->sync); } -int objio_read_pagelist(struct nfs_read_data *rdata) +int objio_read_pagelist(struct nfs_pgio_data *rdata) { struct nfs_pgio_header *hdr = rdata->header; struct objio_state *objios; @@ -487,7 +487,7 @@ static void _write_done(struct ore_io_state *ios, void *private) static struct page *__r4w_get_page(void *priv, u64 offset, bool *uptodate) { struct objio_state *objios = priv; - struct nfs_write_data *wdata = objios->oir.rpcdata; + struct nfs_pgio_data *wdata = objios->oir.rpcdata; struct address_space *mapping = wdata->header->inode->i_mapping; pgoff_t index = offset / PAGE_SIZE; struct page *page; @@ -531,7 +531,7 @@ static const struct _ore_r4w_op _r4w_op = { .put_page = &__r4w_put_page, }; -int objio_write_pagelist(struct nfs_write_data *wdata, int how) +int objio_write_pagelist(struct nfs_pgio_data *wdata, int how) { struct nfs_pgio_header *hdr = wdata->header; struct objio_state *objios; @@ -564,14 +564,22 @@ int objio_write_pagelist(struct nfs_write_data *wdata, int how) return 0; } -static bool objio_pg_test(struct nfs_pageio_descriptor *pgio, +/* + * Return 0 if @req cannot be coalesced into @pgio, otherwise return the number + * of bytes (maximum @req->wb_bytes) that can be coalesced. + */ +static size_t objio_pg_test(struct nfs_pageio_descriptor *pgio, struct nfs_page *prev, struct nfs_page *req) { - if (!pnfs_generic_pg_test(pgio, prev, req)) - return false; + unsigned int size; + + size = pnfs_generic_pg_test(pgio, prev, req); + + if (!size || pgio->pg_count + req->wb_bytes > + (unsigned long)pgio->pg_layout_private) + return 0; - return pgio->pg_count + req->wb_bytes <= - (unsigned long)pgio->pg_layout_private; + return min(size, req->wb_bytes); } static void objio_init_read(struct nfs_pageio_descriptor *pgio, struct nfs_page *req) diff --git a/fs/nfs/objlayout/objlayout.c b/fs/nfs/objlayout/objlayout.c index e4f9cbfec67b..765d3f54e986 100644 --- a/fs/nfs/objlayout/objlayout.c +++ b/fs/nfs/objlayout/objlayout.c @@ -53,10 +53,10 @@ objlayout_alloc_layout_hdr(struct inode *inode, gfp_t gfp_flags) struct objlayout *objlay; objlay = kzalloc(sizeof(struct objlayout), gfp_flags); - if (objlay) { - spin_lock_init(&objlay->lock); - INIT_LIST_HEAD(&objlay->err_list); - } + if (!objlay) + return NULL; + spin_lock_init(&objlay->lock); + INIT_LIST_HEAD(&objlay->err_list); dprintk("%s: Return %p\n", __func__, objlay); return &objlay->pnfs_layout; } @@ -229,11 +229,11 @@ objlayout_io_set_result(struct objlayout_io_res *oir, unsigned index, static void _rpc_read_complete(struct work_struct *work) { struct rpc_task *task; - struct nfs_read_data *rdata; + struct nfs_pgio_data *rdata; dprintk("%s enter\n", __func__); task = container_of(work, struct rpc_task, u.tk_work); - rdata = container_of(task, struct nfs_read_data, task); + rdata = container_of(task, struct nfs_pgio_data, task); pnfs_ld_read_done(rdata); } @@ -241,7 +241,7 @@ static void _rpc_read_complete(struct work_struct *work) void objlayout_read_done(struct objlayout_io_res *oir, ssize_t status, bool sync) { - struct nfs_read_data *rdata = oir->rpcdata; + struct nfs_pgio_data *rdata = oir->rpcdata; oir->status = rdata->task.tk_status = status; if (status >= 0) @@ -266,7 +266,7 @@ objlayout_read_done(struct objlayout_io_res *oir, ssize_t status, bool sync) * Perform sync or async reads. */ enum pnfs_try_status -objlayout_read_pagelist(struct nfs_read_data *rdata) +objlayout_read_pagelist(struct nfs_pgio_data *rdata) { struct nfs_pgio_header *hdr = rdata->header; struct inode *inode = hdr->inode; @@ -312,11 +312,11 @@ objlayout_read_pagelist(struct nfs_read_data *rdata) static void _rpc_write_complete(struct work_struct *work) { struct rpc_task *task; - struct nfs_write_data *wdata; + struct nfs_pgio_data *wdata; dprintk("%s enter\n", __func__); task = container_of(work, struct rpc_task, u.tk_work); - wdata = container_of(task, struct nfs_write_data, task); + wdata = container_of(task, struct nfs_pgio_data, task); pnfs_ld_write_done(wdata); } @@ -324,7 +324,7 @@ static void _rpc_write_complete(struct work_struct *work) void objlayout_write_done(struct objlayout_io_res *oir, ssize_t status, bool sync) { - struct nfs_write_data *wdata = oir->rpcdata; + struct nfs_pgio_data *wdata = oir->rpcdata; oir->status = wdata->task.tk_status = status; if (status >= 0) { @@ -351,7 +351,7 @@ objlayout_write_done(struct objlayout_io_res *oir, ssize_t status, bool sync) * Perform sync or async writes. */ enum pnfs_try_status -objlayout_write_pagelist(struct nfs_write_data *wdata, +objlayout_write_pagelist(struct nfs_pgio_data *wdata, int how) { struct nfs_pgio_header *hdr = wdata->header; diff --git a/fs/nfs/objlayout/objlayout.h b/fs/nfs/objlayout/objlayout.h index 87aa1dec6120..01e041029a6c 100644 --- a/fs/nfs/objlayout/objlayout.h +++ b/fs/nfs/objlayout/objlayout.h @@ -119,8 +119,8 @@ extern void objio_free_lseg(struct pnfs_layout_segment *lseg); */ extern void objio_free_result(struct objlayout_io_res *oir); -extern int objio_read_pagelist(struct nfs_read_data *rdata); -extern int objio_write_pagelist(struct nfs_write_data *wdata, int how); +extern int objio_read_pagelist(struct nfs_pgio_data *rdata); +extern int objio_write_pagelist(struct nfs_pgio_data *wdata, int how); /* * callback API @@ -168,10 +168,10 @@ extern struct pnfs_layout_segment *objlayout_alloc_lseg( extern void objlayout_free_lseg(struct pnfs_layout_segment *); extern enum pnfs_try_status objlayout_read_pagelist( - struct nfs_read_data *); + struct nfs_pgio_data *); extern enum pnfs_try_status objlayout_write_pagelist( - struct nfs_write_data *, + struct nfs_pgio_data *, int how); extern void objlayout_encode_layoutcommit( diff --git a/fs/nfs/pagelist.c b/fs/nfs/pagelist.c index 03ed984ab4d8..b6ee3a6ee96d 100644 --- a/fs/nfs/pagelist.c +++ b/fs/nfs/pagelist.c @@ -24,9 +24,14 @@ #include "internal.h" #include "pnfs.h" +#define NFSDBG_FACILITY NFSDBG_PAGECACHE + static struct kmem_cache *nfs_page_cachep; +static const struct rpc_call_ops nfs_pgio_common_ops; + +static void nfs_free_request(struct nfs_page *); -bool nfs_pgarray_set(struct nfs_page_array *p, unsigned int pagecount) +static bool nfs_pgarray_set(struct nfs_page_array *p, unsigned int pagecount) { p->npages = pagecount; if (pagecount <= ARRAY_SIZE(p->page_array)) @@ -133,11 +138,156 @@ nfs_iocounter_wait(struct nfs_io_counter *c) return __nfs_iocounter_wait(c); } +static int nfs_wait_bit_uninterruptible(void *word) +{ + io_schedule(); + return 0; +} + +/* + * nfs_page_group_lock - lock the head of the page group + * @req - request in group that is to be locked + * + * this lock must be held if modifying the page group list + */ +void +nfs_page_group_lock(struct nfs_page *req) +{ + struct nfs_page *head = req->wb_head; + + WARN_ON_ONCE(head != head->wb_head); + + wait_on_bit_lock(&head->wb_flags, PG_HEADLOCK, + nfs_wait_bit_uninterruptible, + TASK_UNINTERRUPTIBLE); +} + +/* + * nfs_page_group_unlock - unlock the head of the page group + * @req - request in group that is to be unlocked + */ +void +nfs_page_group_unlock(struct nfs_page *req) +{ + struct nfs_page *head = req->wb_head; + + WARN_ON_ONCE(head != head->wb_head); + + smp_mb__before_atomic(); + clear_bit(PG_HEADLOCK, &head->wb_flags); + smp_mb__after_atomic(); + wake_up_bit(&head->wb_flags, PG_HEADLOCK); +} + +/* + * nfs_page_group_sync_on_bit_locked + * + * must be called with page group lock held + */ +static bool +nfs_page_group_sync_on_bit_locked(struct nfs_page *req, unsigned int bit) +{ + struct nfs_page *head = req->wb_head; + struct nfs_page *tmp; + + WARN_ON_ONCE(!test_bit(PG_HEADLOCK, &head->wb_flags)); + WARN_ON_ONCE(test_and_set_bit(bit, &req->wb_flags)); + + tmp = req->wb_this_page; + while (tmp != req) { + if (!test_bit(bit, &tmp->wb_flags)) + return false; + tmp = tmp->wb_this_page; + } + + /* true! reset all bits */ + tmp = req; + do { + clear_bit(bit, &tmp->wb_flags); + tmp = tmp->wb_this_page; + } while (tmp != req); + + return true; +} + +/* + * nfs_page_group_sync_on_bit - set bit on current request, but only + * return true if the bit is set for all requests in page group + * @req - request in page group + * @bit - PG_* bit that is used to sync page group + */ +bool nfs_page_group_sync_on_bit(struct nfs_page *req, unsigned int bit) +{ + bool ret; + + nfs_page_group_lock(req); + ret = nfs_page_group_sync_on_bit_locked(req, bit); + nfs_page_group_unlock(req); + + return ret; +} + +/* + * nfs_page_group_init - Initialize the page group linkage for @req + * @req - a new nfs request + * @prev - the previous request in page group, or NULL if @req is the first + * or only request in the group (the head). + */ +static inline void +nfs_page_group_init(struct nfs_page *req, struct nfs_page *prev) +{ + WARN_ON_ONCE(prev == req); + + if (!prev) { + req->wb_head = req; + req->wb_this_page = req; + } else { + WARN_ON_ONCE(prev->wb_this_page != prev->wb_head); + WARN_ON_ONCE(!test_bit(PG_HEADLOCK, &prev->wb_head->wb_flags)); + req->wb_head = prev->wb_head; + req->wb_this_page = prev->wb_this_page; + prev->wb_this_page = req; + + /* grab extra ref if head request has extra ref from + * the write/commit path to handle handoff between write + * and commit lists */ + if (test_bit(PG_INODE_REF, &prev->wb_head->wb_flags)) + kref_get(&req->wb_kref); + } +} + +/* + * nfs_page_group_destroy - sync the destruction of page groups + * @req - request that no longer needs the page group + * + * releases the page group reference from each member once all + * members have called this function. + */ +static void +nfs_page_group_destroy(struct kref *kref) +{ + struct nfs_page *req = container_of(kref, struct nfs_page, wb_kref); + struct nfs_page *tmp, *next; + + if (!nfs_page_group_sync_on_bit(req, PG_TEARDOWN)) + return; + + tmp = req; + do { + next = tmp->wb_this_page; + /* unlink and free */ + tmp->wb_this_page = tmp; + tmp->wb_head = tmp; + nfs_free_request(tmp); + tmp = next; + } while (tmp != req); +} + /** * nfs_create_request - Create an NFS read/write request. * @ctx: open context to use - * @inode: inode to which the request is attached * @page: page to write + * @last: last nfs request created for this page group or NULL if head * @offset: starting offset within the page for the write * @count: number of bytes to read/write * @@ -146,9 +296,9 @@ nfs_iocounter_wait(struct nfs_io_counter *c) * User should ensure it is safe to sleep in this function. */ struct nfs_page * -nfs_create_request(struct nfs_open_context *ctx, struct inode *inode, - struct page *page, - unsigned int offset, unsigned int count) +nfs_create_request(struct nfs_open_context *ctx, struct page *page, + struct nfs_page *last, unsigned int offset, + unsigned int count) { struct nfs_page *req; struct nfs_lock_context *l_ctx; @@ -180,6 +330,7 @@ nfs_create_request(struct nfs_open_context *ctx, struct inode *inode, req->wb_bytes = count; req->wb_context = get_nfs_open_context(ctx); kref_init(&req->wb_kref); + nfs_page_group_init(req, last); return req; } @@ -237,16 +388,22 @@ static void nfs_clear_request(struct nfs_page *req) } } - /** * nfs_release_request - Release the count on an NFS read/write request * @req: request to release * * Note: Should never be called with the spinlock held! */ -static void nfs_free_request(struct kref *kref) +static void nfs_free_request(struct nfs_page *req) { - struct nfs_page *req = container_of(kref, struct nfs_page, wb_kref); + WARN_ON_ONCE(req->wb_this_page != req); + + /* extra debug: make sure no sync bits are still set */ + WARN_ON_ONCE(test_bit(PG_TEARDOWN, &req->wb_flags)); + WARN_ON_ONCE(test_bit(PG_UNLOCKPAGE, &req->wb_flags)); + WARN_ON_ONCE(test_bit(PG_UPTODATE, &req->wb_flags)); + WARN_ON_ONCE(test_bit(PG_WB_END, &req->wb_flags)); + WARN_ON_ONCE(test_bit(PG_REMOVE, &req->wb_flags)); /* Release struct file and open context */ nfs_clear_request(req); @@ -255,13 +412,7 @@ static void nfs_free_request(struct kref *kref) void nfs_release_request(struct nfs_page *req) { - kref_put(&req->wb_kref, nfs_free_request); -} - -static int nfs_wait_bit_uninterruptible(void *word) -{ - io_schedule(); - return 0; + kref_put(&req->wb_kref, nfs_page_group_destroy); } /** @@ -279,22 +430,249 @@ nfs_wait_on_request(struct nfs_page *req) TASK_UNINTERRUPTIBLE); } -bool nfs_generic_pg_test(struct nfs_pageio_descriptor *desc, struct nfs_page *prev, struct nfs_page *req) +/* + * nfs_generic_pg_test - determine if requests can be coalesced + * @desc: pointer to descriptor + * @prev: previous request in desc, or NULL + * @req: this request + * + * Returns zero if @req can be coalesced into @desc, otherwise it returns + * the size of the request. + */ +size_t nfs_generic_pg_test(struct nfs_pageio_descriptor *desc, + struct nfs_page *prev, struct nfs_page *req) { - /* - * FIXME: ideally we should be able to coalesce all requests - * that are not block boundary aligned, but currently this - * is problematic for the case of bsize < PAGE_CACHE_SIZE, - * since nfs_flush_multi and nfs_pagein_multi assume you - * can have only one struct nfs_page. - */ - if (desc->pg_bsize < PAGE_SIZE) + if (desc->pg_count > desc->pg_bsize) { + /* should never happen */ + WARN_ON_ONCE(1); return 0; + } - return desc->pg_count + req->wb_bytes <= desc->pg_bsize; + return min(desc->pg_bsize - desc->pg_count, (size_t)req->wb_bytes); } EXPORT_SYMBOL_GPL(nfs_generic_pg_test); +static inline struct nfs_rw_header *NFS_RW_HEADER(struct nfs_pgio_header *hdr) +{ + return container_of(hdr, struct nfs_rw_header, header); +} + +/** + * nfs_rw_header_alloc - Allocate a header for a read or write + * @ops: Read or write function vector + */ +struct nfs_rw_header *nfs_rw_header_alloc(const struct nfs_rw_ops *ops) +{ + struct nfs_rw_header *header = ops->rw_alloc_header(); + + if (header) { + struct nfs_pgio_header *hdr = &header->header; + + INIT_LIST_HEAD(&hdr->pages); + spin_lock_init(&hdr->lock); + atomic_set(&hdr->refcnt, 0); + hdr->rw_ops = ops; + } + return header; +} +EXPORT_SYMBOL_GPL(nfs_rw_header_alloc); + +/* + * nfs_rw_header_free - Free a read or write header + * @hdr: The header to free + */ +void nfs_rw_header_free(struct nfs_pgio_header *hdr) +{ + hdr->rw_ops->rw_free_header(NFS_RW_HEADER(hdr)); +} +EXPORT_SYMBOL_GPL(nfs_rw_header_free); + +/** + * nfs_pgio_data_alloc - Allocate pageio data + * @hdr: The header making a request + * @pagecount: Number of pages to create + */ +static struct nfs_pgio_data *nfs_pgio_data_alloc(struct nfs_pgio_header *hdr, + unsigned int pagecount) +{ + struct nfs_pgio_data *data, *prealloc; + + prealloc = &NFS_RW_HEADER(hdr)->rpc_data; + if (prealloc->header == NULL) + data = prealloc; + else + data = kzalloc(sizeof(*data), GFP_KERNEL); + if (!data) + goto out; + + if (nfs_pgarray_set(&data->pages, pagecount)) { + data->header = hdr; + atomic_inc(&hdr->refcnt); + } else { + if (data != prealloc) + kfree(data); + data = NULL; + } +out: + return data; +} + +/** + * nfs_pgio_data_release - Properly free pageio data + * @data: The data to release + */ +void nfs_pgio_data_release(struct nfs_pgio_data *data) +{ + struct nfs_pgio_header *hdr = data->header; + struct nfs_rw_header *pageio_header = NFS_RW_HEADER(hdr); + + put_nfs_open_context(data->args.context); + if (data->pages.pagevec != data->pages.page_array) + kfree(data->pages.pagevec); + if (data == &pageio_header->rpc_data) { + data->header = NULL; + data = NULL; + } + if (atomic_dec_and_test(&hdr->refcnt)) + hdr->completion_ops->completion(hdr); + /* Note: we only free the rpc_task after callbacks are done. + * See the comment in rpc_free_task() for why + */ + kfree(data); +} +EXPORT_SYMBOL_GPL(nfs_pgio_data_release); + +/** + * nfs_pgio_rpcsetup - Set up arguments for a pageio call + * @data: The pageio data + * @count: Number of bytes to read + * @offset: Initial offset + * @how: How to commit data (writes only) + * @cinfo: Commit information for the call (writes only) + */ +static void nfs_pgio_rpcsetup(struct nfs_pgio_data *data, + unsigned int count, unsigned int offset, + int how, struct nfs_commit_info *cinfo) +{ + struct nfs_page *req = data->header->req; + + /* Set up the RPC argument and reply structs + * NB: take care not to mess about with data->commit et al. */ + + data->args.fh = NFS_FH(data->header->inode); + data->args.offset = req_offset(req) + offset; + /* pnfs_set_layoutcommit needs this */ + data->mds_offset = data->args.offset; + data->args.pgbase = req->wb_pgbase + offset; + data->args.pages = data->pages.pagevec; + data->args.count = count; + data->args.context = get_nfs_open_context(req->wb_context); + data->args.lock_context = req->wb_lock_context; + data->args.stable = NFS_UNSTABLE; + switch (how & (FLUSH_STABLE | FLUSH_COND_STABLE)) { + case 0: + break; + case FLUSH_COND_STABLE: + if (nfs_reqs_to_commit(cinfo)) + break; + default: + data->args.stable = NFS_FILE_SYNC; + } + + data->res.fattr = &data->fattr; + data->res.count = count; + data->res.eof = 0; + data->res.verf = &data->verf; + nfs_fattr_init(&data->fattr); +} + +/** + * nfs_pgio_prepare - Prepare pageio data to go over the wire + * @task: The current task + * @calldata: pageio data to prepare + */ +static void nfs_pgio_prepare(struct rpc_task *task, void *calldata) +{ + struct nfs_pgio_data *data = calldata; + int err; + err = NFS_PROTO(data->header->inode)->pgio_rpc_prepare(task, data); + if (err) + rpc_exit(task, err); +} + +int nfs_initiate_pgio(struct rpc_clnt *clnt, struct nfs_pgio_data *data, + const struct rpc_call_ops *call_ops, int how, int flags) +{ + struct rpc_task *task; + struct rpc_message msg = { + .rpc_argp = &data->args, + .rpc_resp = &data->res, + .rpc_cred = data->header->cred, + }; + struct rpc_task_setup task_setup_data = { + .rpc_client = clnt, + .task = &data->task, + .rpc_message = &msg, + .callback_ops = call_ops, + .callback_data = data, + .workqueue = nfsiod_workqueue, + .flags = RPC_TASK_ASYNC | flags, + }; + int ret = 0; + + data->header->rw_ops->rw_initiate(data, &msg, &task_setup_data, how); + + dprintk("NFS: %5u initiated pgio call " + "(req %s/%llu, %u bytes @ offset %llu)\n", + data->task.tk_pid, + data->header->inode->i_sb->s_id, + (unsigned long long)NFS_FILEID(data->header->inode), + data->args.count, + (unsigned long long)data->args.offset); + + task = rpc_run_task(&task_setup_data); + if (IS_ERR(task)) { + ret = PTR_ERR(task); + goto out; + } + if (how & FLUSH_SYNC) { + ret = rpc_wait_for_completion_task(task); + if (ret == 0) + ret = task->tk_status; + } + rpc_put_task(task); +out: + return ret; +} +EXPORT_SYMBOL_GPL(nfs_initiate_pgio); + +/** + * nfs_pgio_error - Clean up from a pageio error + * @desc: IO descriptor + * @hdr: pageio header + */ +static int nfs_pgio_error(struct nfs_pageio_descriptor *desc, + struct nfs_pgio_header *hdr) +{ + set_bit(NFS_IOHDR_REDO, &hdr->flags); + nfs_pgio_data_release(hdr->data); + hdr->data = NULL; + desc->pg_completion_ops->error_cleanup(&desc->pg_list); + return -ENOMEM; +} + +/** + * nfs_pgio_release - Release pageio data + * @calldata: The pageio data to release + */ +static void nfs_pgio_release(void *calldata) +{ + struct nfs_pgio_data *data = calldata; + if (data->header->rw_ops->rw_release) + data->header->rw_ops->rw_release(data); + nfs_pgio_data_release(data); +} + /** * nfs_pageio_init - initialise a page io descriptor * @desc: pointer to descriptor @@ -307,6 +685,7 @@ void nfs_pageio_init(struct nfs_pageio_descriptor *desc, struct inode *inode, const struct nfs_pageio_ops *pg_ops, const struct nfs_pgio_completion_ops *compl_ops, + const struct nfs_rw_ops *rw_ops, size_t bsize, int io_flags) { @@ -320,6 +699,7 @@ void nfs_pageio_init(struct nfs_pageio_descriptor *desc, desc->pg_inode = inode; desc->pg_ops = pg_ops; desc->pg_completion_ops = compl_ops; + desc->pg_rw_ops = rw_ops; desc->pg_ioflags = io_flags; desc->pg_error = 0; desc->pg_lseg = NULL; @@ -328,6 +708,94 @@ void nfs_pageio_init(struct nfs_pageio_descriptor *desc, } EXPORT_SYMBOL_GPL(nfs_pageio_init); +/** + * nfs_pgio_result - Basic pageio error handling + * @task: The task that ran + * @calldata: Pageio data to check + */ +static void nfs_pgio_result(struct rpc_task *task, void *calldata) +{ + struct nfs_pgio_data *data = calldata; + struct inode *inode = data->header->inode; + + dprintk("NFS: %s: %5u, (status %d)\n", __func__, + task->tk_pid, task->tk_status); + + if (data->header->rw_ops->rw_done(task, data, inode) != 0) + return; + if (task->tk_status < 0) + nfs_set_pgio_error(data->header, task->tk_status, data->args.offset); + else + data->header->rw_ops->rw_result(task, data); +} + +/* + * Create an RPC task for the given read or write request and kick it. + * The page must have been locked by the caller. + * + * It may happen that the page we're passed is not marked dirty. + * This is the case if nfs_updatepage detects a conflicting request + * that has been written but not committed. + */ +int nfs_generic_pgio(struct nfs_pageio_descriptor *desc, + struct nfs_pgio_header *hdr) +{ + struct nfs_page *req; + struct page **pages; + struct nfs_pgio_data *data; + struct list_head *head = &desc->pg_list; + struct nfs_commit_info cinfo; + + data = nfs_pgio_data_alloc(hdr, nfs_page_array_len(desc->pg_base, + desc->pg_count)); + if (!data) + return nfs_pgio_error(desc, hdr); + + nfs_init_cinfo(&cinfo, desc->pg_inode, desc->pg_dreq); + pages = data->pages.pagevec; + while (!list_empty(head)) { + req = nfs_list_entry(head->next); + nfs_list_remove_request(req); + nfs_list_add_request(req, &hdr->pages); + *pages++ = req->wb_page; + } + + if ((desc->pg_ioflags & FLUSH_COND_STABLE) && + (desc->pg_moreio || nfs_reqs_to_commit(&cinfo))) + desc->pg_ioflags &= ~FLUSH_COND_STABLE; + + /* Set up the argument struct */ + nfs_pgio_rpcsetup(data, desc->pg_count, 0, desc->pg_ioflags, &cinfo); + hdr->data = data; + desc->pg_rpc_callops = &nfs_pgio_common_ops; + return 0; +} +EXPORT_SYMBOL_GPL(nfs_generic_pgio); + +static int nfs_generic_pg_pgios(struct nfs_pageio_descriptor *desc) +{ + struct nfs_rw_header *rw_hdr; + struct nfs_pgio_header *hdr; + int ret; + + rw_hdr = nfs_rw_header_alloc(desc->pg_rw_ops); + if (!rw_hdr) { + desc->pg_completion_ops->error_cleanup(&desc->pg_list); + return -ENOMEM; + } + hdr = &rw_hdr->header; + nfs_pgheader_init(desc, hdr, nfs_rw_header_free); + atomic_inc(&hdr->refcnt); + ret = nfs_generic_pgio(desc, hdr); + if (ret == 0) + ret = nfs_initiate_pgio(NFS_CLIENT(hdr->inode), + hdr->data, desc->pg_rpc_callops, + desc->pg_ioflags, 0); + if (atomic_dec_and_test(&hdr->refcnt)) + hdr->completion_ops->completion(hdr); + return ret; +} + static bool nfs_match_open_context(const struct nfs_open_context *ctx1, const struct nfs_open_context *ctx2) { @@ -356,18 +824,23 @@ static bool nfs_can_coalesce_requests(struct nfs_page *prev, struct nfs_page *req, struct nfs_pageio_descriptor *pgio) { - if (!nfs_match_open_context(req->wb_context, prev->wb_context)) - return false; - if (req->wb_context->dentry->d_inode->i_flock != NULL && - !nfs_match_lock_context(req->wb_lock_context, prev->wb_lock_context)) - return false; - if (req->wb_pgbase != 0) - return false; - if (prev->wb_pgbase + prev->wb_bytes != PAGE_CACHE_SIZE) - return false; - if (req_offset(req) != req_offset(prev) + prev->wb_bytes) - return false; - return pgio->pg_ops->pg_test(pgio, prev, req); + size_t size; + + if (prev) { + if (!nfs_match_open_context(req->wb_context, prev->wb_context)) + return false; + if (req->wb_context->dentry->d_inode->i_flock != NULL && + !nfs_match_lock_context(req->wb_lock_context, + prev->wb_lock_context)) + return false; + if (req_offset(req) != req_offset(prev) + prev->wb_bytes) + return false; + } + size = pgio->pg_ops->pg_test(pgio, prev, req); + WARN_ON_ONCE(size > req->wb_bytes); + if (size && size < req->wb_bytes) + req->wb_bytes = size; + return size > 0; } /** @@ -381,17 +854,16 @@ static bool nfs_can_coalesce_requests(struct nfs_page *prev, static int nfs_pageio_do_add_request(struct nfs_pageio_descriptor *desc, struct nfs_page *req) { + struct nfs_page *prev = NULL; if (desc->pg_count != 0) { - struct nfs_page *prev; - prev = nfs_list_entry(desc->pg_list.prev); - if (!nfs_can_coalesce_requests(prev, req, desc)) - return 0; } else { if (desc->pg_ops->pg_init) desc->pg_ops->pg_init(desc, req); desc->pg_base = req->wb_pgbase; } + if (!nfs_can_coalesce_requests(prev, req, desc)) + return 0; nfs_list_remove_request(req); nfs_list_add_request(req, &desc->pg_list); desc->pg_count += req->wb_bytes; @@ -421,22 +893,73 @@ static void nfs_pageio_doio(struct nfs_pageio_descriptor *desc) * @desc: destination io descriptor * @req: request * + * This may split a request into subrequests which are all part of the + * same page group. + * * Returns true if the request 'req' was successfully coalesced into the * existing list of pages 'desc'. */ static int __nfs_pageio_add_request(struct nfs_pageio_descriptor *desc, struct nfs_page *req) { - while (!nfs_pageio_do_add_request(desc, req)) { - desc->pg_moreio = 1; - nfs_pageio_doio(desc); - if (desc->pg_error < 0) - return 0; - desc->pg_moreio = 0; - if (desc->pg_recoalesce) - return 0; - } + struct nfs_page *subreq; + unsigned int bytes_left = 0; + unsigned int offset, pgbase; + + nfs_page_group_lock(req); + + subreq = req; + bytes_left = subreq->wb_bytes; + offset = subreq->wb_offset; + pgbase = subreq->wb_pgbase; + + do { + if (!nfs_pageio_do_add_request(desc, subreq)) { + /* make sure pg_test call(s) did nothing */ + WARN_ON_ONCE(subreq->wb_bytes != bytes_left); + WARN_ON_ONCE(subreq->wb_offset != offset); + WARN_ON_ONCE(subreq->wb_pgbase != pgbase); + + nfs_page_group_unlock(req); + desc->pg_moreio = 1; + nfs_pageio_doio(desc); + if (desc->pg_error < 0) + return 0; + desc->pg_moreio = 0; + if (desc->pg_recoalesce) + return 0; + /* retry add_request for this subreq */ + nfs_page_group_lock(req); + continue; + } + + /* check for buggy pg_test call(s) */ + WARN_ON_ONCE(subreq->wb_bytes + subreq->wb_pgbase > PAGE_SIZE); + WARN_ON_ONCE(subreq->wb_bytes > bytes_left); + WARN_ON_ONCE(subreq->wb_bytes == 0); + + bytes_left -= subreq->wb_bytes; + offset += subreq->wb_bytes; + pgbase += subreq->wb_bytes; + + if (bytes_left) { + subreq = nfs_create_request(req->wb_context, + req->wb_page, + subreq, pgbase, bytes_left); + if (IS_ERR(subreq)) + goto err_ptr; + nfs_lock_request(subreq); + subreq->wb_offset = offset; + subreq->wb_index = req->wb_index; + } + } while (bytes_left > 0); + + nfs_page_group_unlock(req); return 1; +err_ptr: + desc->pg_error = PTR_ERR(subreq); + nfs_page_group_unlock(req); + return 0; } static int nfs_do_recoalesce(struct nfs_pageio_descriptor *desc) @@ -535,3 +1058,13 @@ void nfs_destroy_nfspagecache(void) kmem_cache_destroy(nfs_page_cachep); } +static const struct rpc_call_ops nfs_pgio_common_ops = { + .rpc_call_prepare = nfs_pgio_prepare, + .rpc_call_done = nfs_pgio_result, + .rpc_release = nfs_pgio_release, +}; + +const struct nfs_pageio_ops nfs_pgio_rw_ops = { + .pg_test = nfs_generic_pg_test, + .pg_doio = nfs_generic_pg_pgios, +}; diff --git a/fs/nfs/pnfs.c b/fs/nfs/pnfs.c index fd9536e494bc..6fdcd233d6f7 100644 --- a/fs/nfs/pnfs.c +++ b/fs/nfs/pnfs.c @@ -1388,11 +1388,6 @@ pnfs_generic_pg_init_read(struct nfs_pageio_descriptor *pgio, struct nfs_page *r WARN_ON_ONCE(pgio->pg_lseg != NULL); - if (req->wb_offset != req->wb_pgbase) { - nfs_pageio_reset_read_mds(pgio); - return; - } - if (pgio->pg_dreq == NULL) rd_size = i_size_read(pgio->pg_inode) - req_offset(req); else @@ -1417,11 +1412,6 @@ pnfs_generic_pg_init_write(struct nfs_pageio_descriptor *pgio, { WARN_ON_ONCE(pgio->pg_lseg != NULL); - if (req->wb_offset != req->wb_pgbase) { - nfs_pageio_reset_write_mds(pgio); - return; - } - pgio->pg_lseg = pnfs_update_layout(pgio->pg_inode, req->wb_context, req_offset(req), @@ -1434,56 +1424,49 @@ pnfs_generic_pg_init_write(struct nfs_pageio_descriptor *pgio, } EXPORT_SYMBOL_GPL(pnfs_generic_pg_init_write); -void -pnfs_pageio_init_read(struct nfs_pageio_descriptor *pgio, struct inode *inode, - const struct nfs_pgio_completion_ops *compl_ops) -{ - struct nfs_server *server = NFS_SERVER(inode); - struct pnfs_layoutdriver_type *ld = server->pnfs_curr_ld; - - if (ld == NULL) - nfs_pageio_init_read(pgio, inode, compl_ops); - else - nfs_pageio_init(pgio, inode, ld->pg_read_ops, compl_ops, server->rsize, 0); -} - -void -pnfs_pageio_init_write(struct nfs_pageio_descriptor *pgio, struct inode *inode, - int ioflags, - const struct nfs_pgio_completion_ops *compl_ops) -{ - struct nfs_server *server = NFS_SERVER(inode); - struct pnfs_layoutdriver_type *ld = server->pnfs_curr_ld; - - if (ld == NULL) - nfs_pageio_init_write(pgio, inode, ioflags, compl_ops); - else - nfs_pageio_init(pgio, inode, ld->pg_write_ops, compl_ops, server->wsize, ioflags); -} - -bool +/* + * Return 0 if @req cannot be coalesced into @pgio, otherwise return the number + * of bytes (maximum @req->wb_bytes) that can be coalesced. + */ +size_t pnfs_generic_pg_test(struct nfs_pageio_descriptor *pgio, struct nfs_page *prev, struct nfs_page *req) { - if (pgio->pg_lseg == NULL) - return nfs_generic_pg_test(pgio, prev, req); + unsigned int size; + u64 seg_end, req_start, seg_left; + + size = nfs_generic_pg_test(pgio, prev, req); + if (!size) + return 0; /* - * Test if a nfs_page is fully contained in the pnfs_layout_range. - * Note that this test makes several assumptions: - * - that the previous nfs_page in the struct nfs_pageio_descriptor - * is known to lie within the range. - * - that the nfs_page being tested is known to be contiguous with the - * previous nfs_page. - * - Layout ranges are page aligned, so we only have to test the - * start offset of the request. + * 'size' contains the number of bytes left in the current page (up + * to the original size asked for in @req->wb_bytes). + * + * Calculate how many bytes are left in the layout segment + * and if there are less bytes than 'size', return that instead. * * Please also note that 'end_offset' is actually the offset of the * first byte that lies outside the pnfs_layout_range. FIXME? * */ - return req_offset(req) < end_offset(pgio->pg_lseg->pls_range.offset, - pgio->pg_lseg->pls_range.length); + if (pgio->pg_lseg) { + seg_end = end_offset(pgio->pg_lseg->pls_range.offset, + pgio->pg_lseg->pls_range.length); + req_start = req_offset(req); + WARN_ON_ONCE(req_start > seg_end); + /* start of request is past the last byte of this segment */ + if (req_start >= seg_end) + return 0; + + /* adjust 'size' iff there are fewer bytes left in the + * segment than what nfs_generic_pg_test returned */ + seg_left = seg_end - req_start; + if (seg_left < size) + size = (unsigned int)seg_left; + } + + return size; } EXPORT_SYMBOL_GPL(pnfs_generic_pg_test); @@ -1496,7 +1479,7 @@ int pnfs_write_done_resend_to_mds(struct inode *inode, LIST_HEAD(failed); /* Resend all requests through the MDS */ - nfs_pageio_init_write(&pgio, inode, FLUSH_STABLE, compl_ops); + nfs_pageio_init_write(&pgio, inode, FLUSH_STABLE, true, compl_ops); pgio.pg_dreq = dreq; while (!list_empty(head)) { struct nfs_page *req = nfs_list_entry(head->next); @@ -1519,7 +1502,7 @@ int pnfs_write_done_resend_to_mds(struct inode *inode, } EXPORT_SYMBOL_GPL(pnfs_write_done_resend_to_mds); -static void pnfs_ld_handle_write_error(struct nfs_write_data *data) +static void pnfs_ld_handle_write_error(struct nfs_pgio_data *data) { struct nfs_pgio_header *hdr = data->header; @@ -1538,7 +1521,7 @@ static void pnfs_ld_handle_write_error(struct nfs_write_data *data) /* * Called by non rpc-based layout drivers */ -void pnfs_ld_write_done(struct nfs_write_data *data) +void pnfs_ld_write_done(struct nfs_pgio_data *data) { struct nfs_pgio_header *hdr = data->header; @@ -1554,7 +1537,7 @@ EXPORT_SYMBOL_GPL(pnfs_ld_write_done); static void pnfs_write_through_mds(struct nfs_pageio_descriptor *desc, - struct nfs_write_data *data) + struct nfs_pgio_data *data) { struct nfs_pgio_header *hdr = data->header; @@ -1563,11 +1546,11 @@ pnfs_write_through_mds(struct nfs_pageio_descriptor *desc, nfs_pageio_reset_write_mds(desc); desc->pg_recoalesce = 1; } - nfs_writedata_release(data); + nfs_pgio_data_release(data); } static enum pnfs_try_status -pnfs_try_to_write_data(struct nfs_write_data *wdata, +pnfs_try_to_write_data(struct nfs_pgio_data *wdata, const struct rpc_call_ops *call_ops, struct pnfs_layout_segment *lseg, int how) @@ -1589,41 +1572,36 @@ pnfs_try_to_write_data(struct nfs_write_data *wdata, } static void -pnfs_do_multiple_writes(struct nfs_pageio_descriptor *desc, struct list_head *head, int how) +pnfs_do_write(struct nfs_pageio_descriptor *desc, + struct nfs_pgio_header *hdr, int how) { - struct nfs_write_data *data; + struct nfs_pgio_data *data = hdr->data; const struct rpc_call_ops *call_ops = desc->pg_rpc_callops; struct pnfs_layout_segment *lseg = desc->pg_lseg; + enum pnfs_try_status trypnfs; desc->pg_lseg = NULL; - while (!list_empty(head)) { - enum pnfs_try_status trypnfs; - - data = list_first_entry(head, struct nfs_write_data, list); - list_del_init(&data->list); - - trypnfs = pnfs_try_to_write_data(data, call_ops, lseg, how); - if (trypnfs == PNFS_NOT_ATTEMPTED) - pnfs_write_through_mds(desc, data); - } + trypnfs = pnfs_try_to_write_data(data, call_ops, lseg, how); + if (trypnfs == PNFS_NOT_ATTEMPTED) + pnfs_write_through_mds(desc, data); pnfs_put_lseg(lseg); } static void pnfs_writehdr_free(struct nfs_pgio_header *hdr) { pnfs_put_lseg(hdr->lseg); - nfs_writehdr_free(hdr); + nfs_rw_header_free(hdr); } EXPORT_SYMBOL_GPL(pnfs_writehdr_free); int pnfs_generic_pg_writepages(struct nfs_pageio_descriptor *desc) { - struct nfs_write_header *whdr; + struct nfs_rw_header *whdr; struct nfs_pgio_header *hdr; int ret; - whdr = nfs_writehdr_alloc(); + whdr = nfs_rw_header_alloc(desc->pg_rw_ops); if (!whdr) { desc->pg_completion_ops->error_cleanup(&desc->pg_list); pnfs_put_lseg(desc->pg_lseg); @@ -1634,12 +1612,12 @@ pnfs_generic_pg_writepages(struct nfs_pageio_descriptor *desc) nfs_pgheader_init(desc, hdr, pnfs_writehdr_free); hdr->lseg = pnfs_get_lseg(desc->pg_lseg); atomic_inc(&hdr->refcnt); - ret = nfs_generic_flush(desc, hdr); + ret = nfs_generic_pgio(desc, hdr); if (ret != 0) { pnfs_put_lseg(desc->pg_lseg); desc->pg_lseg = NULL; } else - pnfs_do_multiple_writes(desc, &hdr->rpc_list, desc->pg_ioflags); + pnfs_do_write(desc, hdr, desc->pg_ioflags); if (atomic_dec_and_test(&hdr->refcnt)) hdr->completion_ops->completion(hdr); return ret; @@ -1655,7 +1633,7 @@ int pnfs_read_done_resend_to_mds(struct inode *inode, LIST_HEAD(failed); /* Resend all requests through the MDS */ - nfs_pageio_init_read(&pgio, inode, compl_ops); + nfs_pageio_init_read(&pgio, inode, true, compl_ops); pgio.pg_dreq = dreq; while (!list_empty(head)) { struct nfs_page *req = nfs_list_entry(head->next); @@ -1674,7 +1652,7 @@ int pnfs_read_done_resend_to_mds(struct inode *inode, } EXPORT_SYMBOL_GPL(pnfs_read_done_resend_to_mds); -static void pnfs_ld_handle_read_error(struct nfs_read_data *data) +static void pnfs_ld_handle_read_error(struct nfs_pgio_data *data) { struct nfs_pgio_header *hdr = data->header; @@ -1693,7 +1671,7 @@ static void pnfs_ld_handle_read_error(struct nfs_read_data *data) /* * Called by non rpc-based layout drivers */ -void pnfs_ld_read_done(struct nfs_read_data *data) +void pnfs_ld_read_done(struct nfs_pgio_data *data) { struct nfs_pgio_header *hdr = data->header; @@ -1709,7 +1687,7 @@ EXPORT_SYMBOL_GPL(pnfs_ld_read_done); static void pnfs_read_through_mds(struct nfs_pageio_descriptor *desc, - struct nfs_read_data *data) + struct nfs_pgio_data *data) { struct nfs_pgio_header *hdr = data->header; @@ -1718,14 +1696,14 @@ pnfs_read_through_mds(struct nfs_pageio_descriptor *desc, nfs_pageio_reset_read_mds(desc); desc->pg_recoalesce = 1; } - nfs_readdata_release(data); + nfs_pgio_data_release(data); } /* * Call the appropriate parallel I/O subsystem read function. */ static enum pnfs_try_status -pnfs_try_to_read_data(struct nfs_read_data *rdata, +pnfs_try_to_read_data(struct nfs_pgio_data *rdata, const struct rpc_call_ops *call_ops, struct pnfs_layout_segment *lseg) { @@ -1747,41 +1725,35 @@ pnfs_try_to_read_data(struct nfs_read_data *rdata, } static void -pnfs_do_multiple_reads(struct nfs_pageio_descriptor *desc, struct list_head *head) +pnfs_do_read(struct nfs_pageio_descriptor *desc, struct nfs_pgio_header *hdr) { - struct nfs_read_data *data; + struct nfs_pgio_data *data = hdr->data; const struct rpc_call_ops *call_ops = desc->pg_rpc_callops; struct pnfs_layout_segment *lseg = desc->pg_lseg; + enum pnfs_try_status trypnfs; desc->pg_lseg = NULL; - while (!list_empty(head)) { - enum pnfs_try_status trypnfs; - - data = list_first_entry(head, struct nfs_read_data, list); - list_del_init(&data->list); - - trypnfs = pnfs_try_to_read_data(data, call_ops, lseg); - if (trypnfs == PNFS_NOT_ATTEMPTED) - pnfs_read_through_mds(desc, data); - } + trypnfs = pnfs_try_to_read_data(data, call_ops, lseg); + if (trypnfs == PNFS_NOT_ATTEMPTED) + pnfs_read_through_mds(desc, data); pnfs_put_lseg(lseg); } static void pnfs_readhdr_free(struct nfs_pgio_header *hdr) { pnfs_put_lseg(hdr->lseg); - nfs_readhdr_free(hdr); + nfs_rw_header_free(hdr); } EXPORT_SYMBOL_GPL(pnfs_readhdr_free); int pnfs_generic_pg_readpages(struct nfs_pageio_descriptor *desc) { - struct nfs_read_header *rhdr; + struct nfs_rw_header *rhdr; struct nfs_pgio_header *hdr; int ret; - rhdr = nfs_readhdr_alloc(); + rhdr = nfs_rw_header_alloc(desc->pg_rw_ops); if (!rhdr) { desc->pg_completion_ops->error_cleanup(&desc->pg_list); ret = -ENOMEM; @@ -1793,12 +1765,12 @@ pnfs_generic_pg_readpages(struct nfs_pageio_descriptor *desc) nfs_pgheader_init(desc, hdr, pnfs_readhdr_free); hdr->lseg = pnfs_get_lseg(desc->pg_lseg); atomic_inc(&hdr->refcnt); - ret = nfs_generic_pagein(desc, hdr); + ret = nfs_generic_pgio(desc, hdr); if (ret != 0) { pnfs_put_lseg(desc->pg_lseg); desc->pg_lseg = NULL; } else - pnfs_do_multiple_reads(desc, &hdr->rpc_list); + pnfs_do_read(desc, hdr); if (atomic_dec_and_test(&hdr->refcnt)) hdr->completion_ops->completion(hdr); return ret; @@ -1848,7 +1820,7 @@ void pnfs_set_lo_fail(struct pnfs_layout_segment *lseg) EXPORT_SYMBOL_GPL(pnfs_set_lo_fail); void -pnfs_set_layoutcommit(struct nfs_write_data *wdata) +pnfs_set_layoutcommit(struct nfs_pgio_data *wdata) { struct nfs_pgio_header *hdr = wdata->header; struct inode *inode = hdr->inode; diff --git a/fs/nfs/pnfs.h b/fs/nfs/pnfs.h index c3058a076596..4fb309a2b4c4 100644 --- a/fs/nfs/pnfs.h +++ b/fs/nfs/pnfs.h @@ -113,8 +113,8 @@ struct pnfs_layoutdriver_type { * Return PNFS_ATTEMPTED to indicate the layout code has attempted * I/O, else return PNFS_NOT_ATTEMPTED to fall back to normal NFS */ - enum pnfs_try_status (*read_pagelist) (struct nfs_read_data *nfs_data); - enum pnfs_try_status (*write_pagelist) (struct nfs_write_data *nfs_data, int how); + enum pnfs_try_status (*read_pagelist) (struct nfs_pgio_data *nfs_data); + enum pnfs_try_status (*write_pagelist) (struct nfs_pgio_data *nfs_data, int how); void (*free_deviceid_node) (struct nfs4_deviceid_node *); @@ -180,11 +180,6 @@ extern int nfs4_proc_layoutreturn(struct nfs4_layoutreturn *lrp); void pnfs_get_layout_hdr(struct pnfs_layout_hdr *lo); void pnfs_put_lseg(struct pnfs_layout_segment *lseg); -void pnfs_pageio_init_read(struct nfs_pageio_descriptor *, struct inode *, - const struct nfs_pgio_completion_ops *); -void pnfs_pageio_init_write(struct nfs_pageio_descriptor *, struct inode *, - int, const struct nfs_pgio_completion_ops *); - void set_pnfs_layoutdriver(struct nfs_server *, const struct nfs_fh *, u32); void unset_pnfs_layoutdriver(struct nfs_server *); void pnfs_generic_pg_init_read(struct nfs_pageio_descriptor *, struct nfs_page *); @@ -192,7 +187,8 @@ int pnfs_generic_pg_readpages(struct nfs_pageio_descriptor *desc); void pnfs_generic_pg_init_write(struct nfs_pageio_descriptor *pgio, struct nfs_page *req, u64 wb_size); int pnfs_generic_pg_writepages(struct nfs_pageio_descriptor *desc); -bool pnfs_generic_pg_test(struct nfs_pageio_descriptor *pgio, struct nfs_page *prev, struct nfs_page *req); +size_t pnfs_generic_pg_test(struct nfs_pageio_descriptor *pgio, + struct nfs_page *prev, struct nfs_page *req); void pnfs_set_lo_fail(struct pnfs_layout_segment *lseg); struct pnfs_layout_segment *pnfs_layout_process(struct nfs4_layoutget *lgp); void pnfs_free_lseg_list(struct list_head *tmp_list); @@ -217,13 +213,13 @@ bool pnfs_roc(struct inode *ino); void pnfs_roc_release(struct inode *ino); void pnfs_roc_set_barrier(struct inode *ino, u32 barrier); bool pnfs_roc_drain(struct inode *ino, u32 *barrier, struct rpc_task *task); -void pnfs_set_layoutcommit(struct nfs_write_data *wdata); +void pnfs_set_layoutcommit(struct nfs_pgio_data *wdata); void pnfs_cleanup_layoutcommit(struct nfs4_layoutcommit_data *data); int pnfs_layoutcommit_inode(struct inode *inode, bool sync); int _pnfs_return_layout(struct inode *); int pnfs_commit_and_return_layout(struct inode *); -void pnfs_ld_write_done(struct nfs_write_data *); -void pnfs_ld_read_done(struct nfs_read_data *); +void pnfs_ld_write_done(struct nfs_pgio_data *); +void pnfs_ld_read_done(struct nfs_pgio_data *); struct pnfs_layout_segment *pnfs_update_layout(struct inode *ino, struct nfs_open_context *ctx, loff_t pos, @@ -461,18 +457,6 @@ static inline void unset_pnfs_layoutdriver(struct nfs_server *s) { } -static inline void pnfs_pageio_init_read(struct nfs_pageio_descriptor *pgio, struct inode *inode, - const struct nfs_pgio_completion_ops *compl_ops) -{ - nfs_pageio_init_read(pgio, inode, compl_ops); -} - -static inline void pnfs_pageio_init_write(struct nfs_pageio_descriptor *pgio, struct inode *inode, int ioflags, - const struct nfs_pgio_completion_ops *compl_ops) -{ - nfs_pageio_init_write(pgio, inode, ioflags, compl_ops); -} - static inline int pnfs_commit_list(struct inode *inode, struct list_head *mds_pages, int how, struct nfs_commit_info *cinfo) diff --git a/fs/nfs/proc.c b/fs/nfs/proc.c index e55ce9e8b034..c171ce1a8a30 100644 --- a/fs/nfs/proc.c +++ b/fs/nfs/proc.c @@ -578,7 +578,7 @@ nfs_proc_pathconf(struct nfs_server *server, struct nfs_fh *fhandle, return 0; } -static int nfs_read_done(struct rpc_task *task, struct nfs_read_data *data) +static int nfs_read_done(struct rpc_task *task, struct nfs_pgio_data *data) { struct inode *inode = data->header->inode; @@ -594,18 +594,18 @@ static int nfs_read_done(struct rpc_task *task, struct nfs_read_data *data) return 0; } -static void nfs_proc_read_setup(struct nfs_read_data *data, struct rpc_message *msg) +static void nfs_proc_read_setup(struct nfs_pgio_data *data, struct rpc_message *msg) { msg->rpc_proc = &nfs_procedures[NFSPROC_READ]; } -static int nfs_proc_read_rpc_prepare(struct rpc_task *task, struct nfs_read_data *data) +static int nfs_proc_pgio_rpc_prepare(struct rpc_task *task, struct nfs_pgio_data *data) { rpc_call_start(task); return 0; } -static int nfs_write_done(struct rpc_task *task, struct nfs_write_data *data) +static int nfs_write_done(struct rpc_task *task, struct nfs_pgio_data *data) { struct inode *inode = data->header->inode; @@ -614,19 +614,13 @@ static int nfs_write_done(struct rpc_task *task, struct nfs_write_data *data) return 0; } -static void nfs_proc_write_setup(struct nfs_write_data *data, struct rpc_message *msg) +static void nfs_proc_write_setup(struct nfs_pgio_data *data, struct rpc_message *msg) { /* Note: NFSv2 ignores @stable and always uses NFS_FILE_SYNC */ data->args.stable = NFS_FILE_SYNC; msg->rpc_proc = &nfs_procedures[NFSPROC_WRITE]; } -static int nfs_proc_write_rpc_prepare(struct rpc_task *task, struct nfs_write_data *data) -{ - rpc_call_start(task); - return 0; -} - static void nfs_proc_commit_rpc_prepare(struct rpc_task *task, struct nfs_commit_data *data) { BUG(); @@ -734,13 +728,10 @@ const struct nfs_rpc_ops nfs_v2_clientops = { .fsinfo = nfs_proc_fsinfo, .pathconf = nfs_proc_pathconf, .decode_dirent = nfs2_decode_dirent, + .pgio_rpc_prepare = nfs_proc_pgio_rpc_prepare, .read_setup = nfs_proc_read_setup, - .read_pageio_init = nfs_pageio_init_read, - .read_rpc_prepare = nfs_proc_read_rpc_prepare, .read_done = nfs_read_done, .write_setup = nfs_proc_write_setup, - .write_pageio_init = nfs_pageio_init_write, - .write_rpc_prepare = nfs_proc_write_rpc_prepare, .write_done = nfs_write_done, .commit_setup = nfs_proc_commit_setup, .commit_rpc_prepare = nfs_proc_commit_rpc_prepare, diff --git a/fs/nfs/read.c b/fs/nfs/read.c index 411aedda14bb..e818a475ca64 100644 --- a/fs/nfs/read.c +++ b/fs/nfs/read.c @@ -24,85 +24,24 @@ #include "internal.h" #include "iostat.h" #include "fscache.h" +#include "pnfs.h" #define NFSDBG_FACILITY NFSDBG_PAGECACHE -static const struct nfs_pageio_ops nfs_pageio_read_ops; -static const struct rpc_call_ops nfs_read_common_ops; static const struct nfs_pgio_completion_ops nfs_async_read_completion_ops; +static const struct nfs_rw_ops nfs_rw_read_ops; static struct kmem_cache *nfs_rdata_cachep; -struct nfs_read_header *nfs_readhdr_alloc(void) +static struct nfs_rw_header *nfs_readhdr_alloc(void) { - struct nfs_read_header *rhdr; - - rhdr = kmem_cache_zalloc(nfs_rdata_cachep, GFP_KERNEL); - if (rhdr) { - struct nfs_pgio_header *hdr = &rhdr->header; - - INIT_LIST_HEAD(&hdr->pages); - INIT_LIST_HEAD(&hdr->rpc_list); - spin_lock_init(&hdr->lock); - atomic_set(&hdr->refcnt, 0); - } - return rhdr; + return kmem_cache_zalloc(nfs_rdata_cachep, GFP_KERNEL); } -EXPORT_SYMBOL_GPL(nfs_readhdr_alloc); -static struct nfs_read_data *nfs_readdata_alloc(struct nfs_pgio_header *hdr, - unsigned int pagecount) +static void nfs_readhdr_free(struct nfs_rw_header *rhdr) { - struct nfs_read_data *data, *prealloc; - - prealloc = &container_of(hdr, struct nfs_read_header, header)->rpc_data; - if (prealloc->header == NULL) - data = prealloc; - else - data = kzalloc(sizeof(*data), GFP_KERNEL); - if (!data) - goto out; - - if (nfs_pgarray_set(&data->pages, pagecount)) { - data->header = hdr; - atomic_inc(&hdr->refcnt); - } else { - if (data != prealloc) - kfree(data); - data = NULL; - } -out: - return data; -} - -void nfs_readhdr_free(struct nfs_pgio_header *hdr) -{ - struct nfs_read_header *rhdr = container_of(hdr, struct nfs_read_header, header); - kmem_cache_free(nfs_rdata_cachep, rhdr); } -EXPORT_SYMBOL_GPL(nfs_readhdr_free); - -void nfs_readdata_release(struct nfs_read_data *rdata) -{ - struct nfs_pgio_header *hdr = rdata->header; - struct nfs_read_header *read_header = container_of(hdr, struct nfs_read_header, header); - - put_nfs_open_context(rdata->args.context); - if (rdata->pages.pagevec != rdata->pages.page_array) - kfree(rdata->pages.pagevec); - if (rdata == &read_header->rpc_data) { - rdata->header = NULL; - rdata = NULL; - } - if (atomic_dec_and_test(&hdr->refcnt)) - hdr->completion_ops->completion(hdr); - /* Note: we only free the rpc_task after callbacks are done. - * See the comment in rpc_free_task() for why - */ - kfree(rdata); -} -EXPORT_SYMBOL_GPL(nfs_readdata_release); static int nfs_return_empty_page(struct page *page) @@ -114,17 +53,24 @@ int nfs_return_empty_page(struct page *page) } void nfs_pageio_init_read(struct nfs_pageio_descriptor *pgio, - struct inode *inode, + struct inode *inode, bool force_mds, const struct nfs_pgio_completion_ops *compl_ops) { - nfs_pageio_init(pgio, inode, &nfs_pageio_read_ops, compl_ops, - NFS_SERVER(inode)->rsize, 0); + struct nfs_server *server = NFS_SERVER(inode); + const struct nfs_pageio_ops *pg_ops = &nfs_pgio_rw_ops; + +#ifdef CONFIG_NFS_V4_1 + if (server->pnfs_curr_ld && !force_mds) + pg_ops = server->pnfs_curr_ld->pg_read_ops; +#endif + nfs_pageio_init(pgio, inode, pg_ops, compl_ops, &nfs_rw_read_ops, + server->rsize, 0); } EXPORT_SYMBOL_GPL(nfs_pageio_init_read); void nfs_pageio_reset_read_mds(struct nfs_pageio_descriptor *pgio) { - pgio->pg_ops = &nfs_pageio_read_ops; + pgio->pg_ops = &nfs_pgio_rw_ops; pgio->pg_bsize = NFS_SERVER(pgio->pg_inode)->rsize; } EXPORT_SYMBOL_GPL(nfs_pageio_reset_read_mds); @@ -139,7 +85,7 @@ int nfs_readpage_async(struct nfs_open_context *ctx, struct inode *inode, len = nfs_page_length(page); if (len == 0) return nfs_return_empty_page(page); - new = nfs_create_request(ctx, inode, page, 0, len); + new = nfs_create_request(ctx, page, NULL, 0, len); if (IS_ERR(new)) { unlock_page(page); return PTR_ERR(new); @@ -147,7 +93,8 @@ int nfs_readpage_async(struct nfs_open_context *ctx, struct inode *inode, if (len < PAGE_CACHE_SIZE) zero_user_segment(page, len, PAGE_CACHE_SIZE); - NFS_PROTO(inode)->read_pageio_init(&pgio, inode, &nfs_async_read_completion_ops); + nfs_pageio_init_read(&pgio, inode, false, + &nfs_async_read_completion_ops); nfs_pageio_add_request(&pgio, new); nfs_pageio_complete(&pgio); NFS_I(inode)->read_io += pgio.pg_bytes_written; @@ -158,10 +105,16 @@ static void nfs_readpage_release(struct nfs_page *req) { struct inode *d_inode = req->wb_context->dentry->d_inode; - if (PageUptodate(req->wb_page)) - nfs_readpage_to_fscache(d_inode, req->wb_page, 0); + dprintk("NFS: read done (%s/%llu %d@%lld)\n", d_inode->i_sb->s_id, + (unsigned long long)NFS_FILEID(d_inode), req->wb_bytes, + (long long)req_offset(req)); - unlock_page(req->wb_page); + if (nfs_page_group_sync_on_bit(req, PG_UNLOCKPAGE)) { + if (PageUptodate(req->wb_page)) + nfs_readpage_to_fscache(d_inode, req->wb_page, 0); + + unlock_page(req->wb_page); + } dprintk("NFS: read done (%s/%Lu %d@%Ld)\n", req->wb_context->dentry->d_inode->i_sb->s_id, @@ -171,7 +124,12 @@ static void nfs_readpage_release(struct nfs_page *req) nfs_release_request(req); } -/* Note io was page aligned */ +static void nfs_page_group_set_uptodate(struct nfs_page *req) +{ + if (nfs_page_group_sync_on_bit(req, PG_UPTODATE)) + SetPageUptodate(req->wb_page); +} + static void nfs_read_completion(struct nfs_pgio_header *hdr) { unsigned long bytes = 0; @@ -181,21 +139,32 @@ static void nfs_read_completion(struct nfs_pgio_header *hdr) while (!list_empty(&hdr->pages)) { struct nfs_page *req = nfs_list_entry(hdr->pages.next); struct page *page = req->wb_page; + unsigned long start = req->wb_pgbase; + unsigned long end = req->wb_pgbase + req->wb_bytes; if (test_bit(NFS_IOHDR_EOF, &hdr->flags)) { - if (bytes > hdr->good_bytes) - zero_user(page, 0, PAGE_SIZE); - else if (hdr->good_bytes - bytes < PAGE_SIZE) - zero_user_segment(page, - hdr->good_bytes & ~PAGE_MASK, - PAGE_SIZE); + /* note: regions of the page not covered by a + * request are zeroed in nfs_readpage_async / + * readpage_async_filler */ + if (bytes > hdr->good_bytes) { + /* nothing in this request was good, so zero + * the full extent of the request */ + zero_user_segment(page, start, end); + + } else if (hdr->good_bytes - bytes < req->wb_bytes) { + /* part of this request has good bytes, but + * not all. zero the bad bytes */ + start += hdr->good_bytes - bytes; + WARN_ON(start < req->wb_pgbase); + zero_user_segment(page, start, end); + } } bytes += req->wb_bytes; if (test_bit(NFS_IOHDR_ERROR, &hdr->flags)) { if (bytes <= hdr->good_bytes) - SetPageUptodate(page); + nfs_page_group_set_uptodate(req); } else - SetPageUptodate(page); + nfs_page_group_set_uptodate(req); nfs_list_remove_request(req); nfs_readpage_release(req); } @@ -203,95 +172,14 @@ out: hdr->release(hdr); } -int nfs_initiate_read(struct rpc_clnt *clnt, - struct nfs_read_data *data, - const struct rpc_call_ops *call_ops, int flags) +static void nfs_initiate_read(struct nfs_pgio_data *data, struct rpc_message *msg, + struct rpc_task_setup *task_setup_data, int how) { struct inode *inode = data->header->inode; int swap_flags = IS_SWAPFILE(inode) ? NFS_RPC_SWAPFLAGS : 0; - struct rpc_task *task; - struct rpc_message msg = { - .rpc_argp = &data->args, - .rpc_resp = &data->res, - .rpc_cred = data->header->cred, - }; - struct rpc_task_setup task_setup_data = { - .task = &data->task, - .rpc_client = clnt, - .rpc_message = &msg, - .callback_ops = call_ops, - .callback_data = data, - .workqueue = nfsiod_workqueue, - .flags = RPC_TASK_ASYNC | swap_flags | flags, - }; - /* Set up the initial task struct. */ - NFS_PROTO(inode)->read_setup(data, &msg); - - dprintk("NFS: %5u initiated read call (req %s/%llu, %u bytes @ " - "offset %llu)\n", - data->task.tk_pid, - inode->i_sb->s_id, - (unsigned long long)NFS_FILEID(inode), - data->args.count, - (unsigned long long)data->args.offset); - - task = rpc_run_task(&task_setup_data); - if (IS_ERR(task)) - return PTR_ERR(task); - rpc_put_task(task); - return 0; -} -EXPORT_SYMBOL_GPL(nfs_initiate_read); - -/* - * Set up the NFS read request struct - */ -static void nfs_read_rpcsetup(struct nfs_read_data *data, - unsigned int count, unsigned int offset) -{ - struct nfs_page *req = data->header->req; - - data->args.fh = NFS_FH(data->header->inode); - data->args.offset = req_offset(req) + offset; - data->args.pgbase = req->wb_pgbase + offset; - data->args.pages = data->pages.pagevec; - data->args.count = count; - data->args.context = get_nfs_open_context(req->wb_context); - data->args.lock_context = req->wb_lock_context; - - data->res.fattr = &data->fattr; - data->res.count = count; - data->res.eof = 0; - nfs_fattr_init(&data->fattr); -} - -static int nfs_do_read(struct nfs_read_data *data, - const struct rpc_call_ops *call_ops) -{ - struct inode *inode = data->header->inode; - - return nfs_initiate_read(NFS_CLIENT(inode), data, call_ops, 0); -} - -static int -nfs_do_multiple_reads(struct list_head *head, - const struct rpc_call_ops *call_ops) -{ - struct nfs_read_data *data; - int ret = 0; - - while (!list_empty(head)) { - int ret2; - - data = list_first_entry(head, struct nfs_read_data, list); - list_del_init(&data->list); - - ret2 = nfs_do_read(data, call_ops); - if (ret == 0) - ret = ret2; - } - return ret; + task_setup_data->flags |= swap_flags; + NFS_PROTO(inode)->read_setup(data, msg); } static void @@ -311,143 +199,14 @@ static const struct nfs_pgio_completion_ops nfs_async_read_completion_ops = { .completion = nfs_read_completion, }; -static void nfs_pagein_error(struct nfs_pageio_descriptor *desc, - struct nfs_pgio_header *hdr) -{ - set_bit(NFS_IOHDR_REDO, &hdr->flags); - while (!list_empty(&hdr->rpc_list)) { - struct nfs_read_data *data = list_first_entry(&hdr->rpc_list, - struct nfs_read_data, list); - list_del(&data->list); - nfs_readdata_release(data); - } - desc->pg_completion_ops->error_cleanup(&desc->pg_list); -} - -/* - * Generate multiple requests to fill a single page. - * - * We optimize to reduce the number of read operations on the wire. If we - * detect that we're reading a page, or an area of a page, that is past the - * end of file, we do not generate NFS read operations but just clear the - * parts of the page that would have come back zero from the server anyway. - * - * We rely on the cached value of i_size to make this determination; another - * client can fill pages on the server past our cached end-of-file, but we - * won't see the new data until our attribute cache is updated. This is more - * or less conventional NFS client behavior. - */ -static int nfs_pagein_multi(struct nfs_pageio_descriptor *desc, - struct nfs_pgio_header *hdr) -{ - struct nfs_page *req = hdr->req; - struct page *page = req->wb_page; - struct nfs_read_data *data; - size_t rsize = desc->pg_bsize, nbytes; - unsigned int offset; - - offset = 0; - nbytes = desc->pg_count; - do { - size_t len = min(nbytes,rsize); - - data = nfs_readdata_alloc(hdr, 1); - if (!data) { - nfs_pagein_error(desc, hdr); - return -ENOMEM; - } - data->pages.pagevec[0] = page; - nfs_read_rpcsetup(data, len, offset); - list_add(&data->list, &hdr->rpc_list); - nbytes -= len; - offset += len; - } while (nbytes != 0); - - nfs_list_remove_request(req); - nfs_list_add_request(req, &hdr->pages); - desc->pg_rpc_callops = &nfs_read_common_ops; - return 0; -} - -static int nfs_pagein_one(struct nfs_pageio_descriptor *desc, - struct nfs_pgio_header *hdr) -{ - struct nfs_page *req; - struct page **pages; - struct nfs_read_data *data; - struct list_head *head = &desc->pg_list; - - data = nfs_readdata_alloc(hdr, nfs_page_array_len(desc->pg_base, - desc->pg_count)); - if (!data) { - nfs_pagein_error(desc, hdr); - return -ENOMEM; - } - - pages = data->pages.pagevec; - while (!list_empty(head)) { - req = nfs_list_entry(head->next); - nfs_list_remove_request(req); - nfs_list_add_request(req, &hdr->pages); - *pages++ = req->wb_page; - } - - nfs_read_rpcsetup(data, desc->pg_count, 0); - list_add(&data->list, &hdr->rpc_list); - desc->pg_rpc_callops = &nfs_read_common_ops; - return 0; -} - -int nfs_generic_pagein(struct nfs_pageio_descriptor *desc, - struct nfs_pgio_header *hdr) -{ - if (desc->pg_bsize < PAGE_CACHE_SIZE) - return nfs_pagein_multi(desc, hdr); - return nfs_pagein_one(desc, hdr); -} -EXPORT_SYMBOL_GPL(nfs_generic_pagein); - -static int nfs_generic_pg_readpages(struct nfs_pageio_descriptor *desc) -{ - struct nfs_read_header *rhdr; - struct nfs_pgio_header *hdr; - int ret; - - rhdr = nfs_readhdr_alloc(); - if (!rhdr) { - desc->pg_completion_ops->error_cleanup(&desc->pg_list); - return -ENOMEM; - } - hdr = &rhdr->header; - nfs_pgheader_init(desc, hdr, nfs_readhdr_free); - atomic_inc(&hdr->refcnt); - ret = nfs_generic_pagein(desc, hdr); - if (ret == 0) - ret = nfs_do_multiple_reads(&hdr->rpc_list, - desc->pg_rpc_callops); - if (atomic_dec_and_test(&hdr->refcnt)) - hdr->completion_ops->completion(hdr); - return ret; -} - -static const struct nfs_pageio_ops nfs_pageio_read_ops = { - .pg_test = nfs_generic_pg_test, - .pg_doio = nfs_generic_pg_readpages, -}; - /* * This is the callback from RPC telling us whether a reply was * received or some error occurred (timeout or socket shutdown). */ -int nfs_readpage_result(struct rpc_task *task, struct nfs_read_data *data) +static int nfs_readpage_done(struct rpc_task *task, struct nfs_pgio_data *data, + struct inode *inode) { - struct inode *inode = data->header->inode; - int status; - - dprintk("NFS: %s: %5u, (status %d)\n", __func__, task->tk_pid, - task->tk_status); - - status = NFS_PROTO(inode)->read_done(task, data); + int status = NFS_PROTO(inode)->read_done(task, data); if (status != 0) return status; @@ -460,10 +219,10 @@ int nfs_readpage_result(struct rpc_task *task, struct nfs_read_data *data) return 0; } -static void nfs_readpage_retry(struct rpc_task *task, struct nfs_read_data *data) +static void nfs_readpage_retry(struct rpc_task *task, struct nfs_pgio_data *data) { - struct nfs_readargs *argp = &data->args; - struct nfs_readres *resp = &data->res; + struct nfs_pgio_args *argp = &data->args; + struct nfs_pgio_res *resp = &data->res; /* This is a short read! */ nfs_inc_stats(data->header->inode, NFSIOS_SHORTREAD); @@ -480,17 +239,11 @@ static void nfs_readpage_retry(struct rpc_task *task, struct nfs_read_data *data rpc_restart_call_prepare(task); } -static void nfs_readpage_result_common(struct rpc_task *task, void *calldata) +static void nfs_readpage_result(struct rpc_task *task, struct nfs_pgio_data *data) { - struct nfs_read_data *data = calldata; struct nfs_pgio_header *hdr = data->header; - /* Note the only returns of nfs_readpage_result are 0 and -EAGAIN */ - if (nfs_readpage_result(task, data) != 0) - return; - if (task->tk_status < 0) - nfs_set_pgio_error(hdr, task->tk_status, data->args.offset); - else if (data->res.eof) { + if (data->res.eof) { loff_t bound; bound = data->args.offset + data->res.count; @@ -505,26 +258,6 @@ static void nfs_readpage_result_common(struct rpc_task *task, void *calldata) nfs_readpage_retry(task, data); } -static void nfs_readpage_release_common(void *calldata) -{ - nfs_readdata_release(calldata); -} - -void nfs_read_prepare(struct rpc_task *task, void *calldata) -{ - struct nfs_read_data *data = calldata; - int err; - err = NFS_PROTO(data->header->inode)->read_rpc_prepare(task, data); - if (err) - rpc_exit(task, err); -} - -static const struct rpc_call_ops nfs_read_common_ops = { - .rpc_call_prepare = nfs_read_prepare, - .rpc_call_done = nfs_readpage_result_common, - .rpc_release = nfs_readpage_release_common, -}; - /* * Read a page over NFS. * We read the page synchronously in the following case: @@ -592,7 +325,6 @@ static int readpage_async_filler(void *data, struct page *page) { struct nfs_readdesc *desc = (struct nfs_readdesc *)data; - struct inode *inode = page_file_mapping(page)->host; struct nfs_page *new; unsigned int len; int error; @@ -601,7 +333,7 @@ readpage_async_filler(void *data, struct page *page) if (len == 0) return nfs_return_empty_page(page); - new = nfs_create_request(desc->ctx, inode, page, 0, len); + new = nfs_create_request(desc->ctx, page, NULL, 0, len); if (IS_ERR(new)) goto out_error; @@ -654,7 +386,8 @@ int nfs_readpages(struct file *filp, struct address_space *mapping, if (ret == 0) goto read_complete; /* all pages were read */ - NFS_PROTO(inode)->read_pageio_init(&pgio, inode, &nfs_async_read_completion_ops); + nfs_pageio_init_read(&pgio, inode, false, + &nfs_async_read_completion_ops); ret = read_cache_pages(mapping, pages, readpage_async_filler, &desc); @@ -671,7 +404,7 @@ out: int __init nfs_init_readpagecache(void) { nfs_rdata_cachep = kmem_cache_create("nfs_read_data", - sizeof(struct nfs_read_header), + sizeof(struct nfs_rw_header), 0, SLAB_HWCACHE_ALIGN, NULL); if (nfs_rdata_cachep == NULL) @@ -684,3 +417,12 @@ void nfs_destroy_readpagecache(void) { kmem_cache_destroy(nfs_rdata_cachep); } + +static const struct nfs_rw_ops nfs_rw_read_ops = { + .rw_mode = FMODE_READ, + .rw_alloc_header = nfs_readhdr_alloc, + .rw_free_header = nfs_readhdr_free, + .rw_done = nfs_readpage_done, + .rw_result = nfs_readpage_result, + .rw_initiate = nfs_initiate_read, +}; diff --git a/fs/nfs/super.c b/fs/nfs/super.c index 2cb56943e232..084af1060d79 100644 --- a/fs/nfs/super.c +++ b/fs/nfs/super.c @@ -2180,11 +2180,23 @@ out_no_address: return -EINVAL; } +#define NFS_MOUNT_CMP_FLAGMASK ~(NFS_MOUNT_INTR \ + | NFS_MOUNT_SECURE \ + | NFS_MOUNT_TCP \ + | NFS_MOUNT_VER3 \ + | NFS_MOUNT_KERBEROS \ + | NFS_MOUNT_NONLM \ + | NFS_MOUNT_BROKEN_SUID \ + | NFS_MOUNT_STRICTLOCK \ + | NFS_MOUNT_UNSHARED \ + | NFS_MOUNT_NORESVPORT \ + | NFS_MOUNT_LEGACY_INTERFACE) + static int nfs_compare_remount_data(struct nfs_server *nfss, struct nfs_parsed_mount_data *data) { - if (data->flags != nfss->flags || + if ((data->flags ^ nfss->flags) & NFS_MOUNT_CMP_FLAGMASK || data->rsize != nfss->rsize || data->wsize != nfss->wsize || data->version != nfss->nfs_client->rpc_ops->version || @@ -2248,6 +2260,7 @@ nfs_remount(struct super_block *sb, int *flags, char *raw_data) data->nfs_server.addrlen = nfss->nfs_client->cl_addrlen; data->version = nfsvers; data->minorversion = nfss->nfs_client->cl_minorversion; + data->net = current->nsproxy->net_ns; memcpy(&data->nfs_server.address, &nfss->nfs_client->cl_addr, data->nfs_server.addrlen); @@ -2347,18 +2360,6 @@ void nfs_clone_super(struct super_block *sb, struct nfs_mount_info *mount_info) nfs_initialise_sb(sb); } -#define NFS_MOUNT_CMP_FLAGMASK ~(NFS_MOUNT_INTR \ - | NFS_MOUNT_SECURE \ - | NFS_MOUNT_TCP \ - | NFS_MOUNT_VER3 \ - | NFS_MOUNT_KERBEROS \ - | NFS_MOUNT_NONLM \ - | NFS_MOUNT_BROKEN_SUID \ - | NFS_MOUNT_STRICTLOCK \ - | NFS_MOUNT_UNSHARED \ - | NFS_MOUNT_NORESVPORT \ - | NFS_MOUNT_LEGACY_INTERFACE) - static int nfs_compare_mount_options(const struct super_block *s, const struct nfs_server *b, int flags) { const struct nfs_server *a = s->s_fs_info; diff --git a/fs/nfs/write.c b/fs/nfs/write.c index ffb9459f180b..3ee5af4e738e 100644 --- a/fs/nfs/write.c +++ b/fs/nfs/write.c @@ -42,10 +42,10 @@ * Local function declarations */ static void nfs_redirty_request(struct nfs_page *req); -static const struct rpc_call_ops nfs_write_common_ops; static const struct rpc_call_ops nfs_commit_ops; static const struct nfs_pgio_completion_ops nfs_async_write_completion_ops; static const struct nfs_commit_completion_ops nfs_commit_completion_ops; +static const struct nfs_rw_ops nfs_rw_write_ops; static struct kmem_cache *nfs_wdata_cachep; static mempool_t *nfs_wdata_mempool; @@ -70,76 +70,19 @@ void nfs_commit_free(struct nfs_commit_data *p) } EXPORT_SYMBOL_GPL(nfs_commit_free); -struct nfs_write_header *nfs_writehdr_alloc(void) +static struct nfs_rw_header *nfs_writehdr_alloc(void) { - struct nfs_write_header *p = mempool_alloc(nfs_wdata_mempool, GFP_NOIO); - - if (p) { - struct nfs_pgio_header *hdr = &p->header; + struct nfs_rw_header *p = mempool_alloc(nfs_wdata_mempool, GFP_NOIO); + if (p) memset(p, 0, sizeof(*p)); - INIT_LIST_HEAD(&hdr->pages); - INIT_LIST_HEAD(&hdr->rpc_list); - spin_lock_init(&hdr->lock); - atomic_set(&hdr->refcnt, 0); - hdr->verf = &p->verf; - } return p; } -EXPORT_SYMBOL_GPL(nfs_writehdr_alloc); - -static struct nfs_write_data *nfs_writedata_alloc(struct nfs_pgio_header *hdr, - unsigned int pagecount) -{ - struct nfs_write_data *data, *prealloc; - - prealloc = &container_of(hdr, struct nfs_write_header, header)->rpc_data; - if (prealloc->header == NULL) - data = prealloc; - else - data = kzalloc(sizeof(*data), GFP_KERNEL); - if (!data) - goto out; - - if (nfs_pgarray_set(&data->pages, pagecount)) { - data->header = hdr; - atomic_inc(&hdr->refcnt); - } else { - if (data != prealloc) - kfree(data); - data = NULL; - } -out: - return data; -} -void nfs_writehdr_free(struct nfs_pgio_header *hdr) +static void nfs_writehdr_free(struct nfs_rw_header *whdr) { - struct nfs_write_header *whdr = container_of(hdr, struct nfs_write_header, header); mempool_free(whdr, nfs_wdata_mempool); } -EXPORT_SYMBOL_GPL(nfs_writehdr_free); - -void nfs_writedata_release(struct nfs_write_data *wdata) -{ - struct nfs_pgio_header *hdr = wdata->header; - struct nfs_write_header *write_header = container_of(hdr, struct nfs_write_header, header); - - put_nfs_open_context(wdata->args.context); - if (wdata->pages.pagevec != wdata->pages.page_array) - kfree(wdata->pages.pagevec); - if (wdata == &write_header->rpc_data) { - wdata->header = NULL; - wdata = NULL; - } - if (atomic_dec_and_test(&hdr->refcnt)) - hdr->completion_ops->completion(hdr); - /* Note: we only free the rpc_task after callbacks are done. - * See the comment in rpc_free_task() for why - */ - kfree(wdata); -} -EXPORT_SYMBOL_GPL(nfs_writedata_release); static void nfs_context_set_write_error(struct nfs_open_context *ctx, int error) { @@ -211,18 +154,78 @@ static void nfs_set_pageerror(struct page *page) nfs_zap_mapping(page_file_mapping(page)->host, page_file_mapping(page)); } +/* + * nfs_page_group_search_locked + * @head - head request of page group + * @page_offset - offset into page + * + * Search page group with head @head to find a request that contains the + * page offset @page_offset. + * + * Returns a pointer to the first matching nfs request, or NULL if no + * match is found. + * + * Must be called with the page group lock held + */ +static struct nfs_page * +nfs_page_group_search_locked(struct nfs_page *head, unsigned int page_offset) +{ + struct nfs_page *req; + + WARN_ON_ONCE(head != head->wb_head); + WARN_ON_ONCE(!test_bit(PG_HEADLOCK, &head->wb_head->wb_flags)); + + req = head; + do { + if (page_offset >= req->wb_pgbase && + page_offset < (req->wb_pgbase + req->wb_bytes)) + return req; + + req = req->wb_this_page; + } while (req != head); + + return NULL; +} + +/* + * nfs_page_group_covers_page + * @head - head request of page group + * + * Return true if the page group with head @head covers the whole page, + * returns false otherwise + */ +static bool nfs_page_group_covers_page(struct nfs_page *req) +{ + struct nfs_page *tmp; + unsigned int pos = 0; + unsigned int len = nfs_page_length(req->wb_page); + + nfs_page_group_lock(req); + + do { + tmp = nfs_page_group_search_locked(req->wb_head, pos); + if (tmp) { + /* no way this should happen */ + WARN_ON_ONCE(tmp->wb_pgbase != pos); + pos += tmp->wb_bytes - (pos - tmp->wb_pgbase); + } + } while (tmp && pos < len); + + nfs_page_group_unlock(req); + WARN_ON_ONCE(pos > len); + return pos == len; +} + /* We can set the PG_uptodate flag if we see that a write request * covers the full page. */ -static void nfs_mark_uptodate(struct page *page, unsigned int base, unsigned int count) +static void nfs_mark_uptodate(struct nfs_page *req) { - if (PageUptodate(page)) - return; - if (base != 0) + if (PageUptodate(req->wb_page)) return; - if (count != nfs_page_length(page)) + if (!nfs_page_group_covers_page(req)) return; - SetPageUptodate(page); + SetPageUptodate(req->wb_page); } static int wb_priority(struct writeback_control *wbc) @@ -258,12 +261,15 @@ static void nfs_set_page_writeback(struct page *page) } } -static void nfs_end_page_writeback(struct page *page) +static void nfs_end_page_writeback(struct nfs_page *req) { - struct inode *inode = page_file_mapping(page)->host; + struct inode *inode = page_file_mapping(req->wb_page)->host; struct nfs_server *nfss = NFS_SERVER(inode); - end_page_writeback(page); + if (!nfs_page_group_sync_on_bit(req, PG_WB_END)) + return; + + end_page_writeback(req->wb_page); if (atomic_long_dec_return(&nfss->writeback) < NFS_CONGESTION_OFF_THRESH) clear_bdi_congested(&nfss->backing_dev_info, BLK_RW_ASYNC); } @@ -354,10 +360,8 @@ static int nfs_writepage_locked(struct page *page, struct writeback_control *wbc struct nfs_pageio_descriptor pgio; int err; - NFS_PROTO(page_file_mapping(page)->host)->write_pageio_init(&pgio, - page->mapping->host, - wb_priority(wbc), - &nfs_async_write_completion_ops); + nfs_pageio_init_write(&pgio, page->mapping->host, wb_priority(wbc), + false, &nfs_async_write_completion_ops); err = nfs_do_writepage(page, wbc, &pgio); nfs_pageio_complete(&pgio); if (err < 0) @@ -400,7 +404,8 @@ int nfs_writepages(struct address_space *mapping, struct writeback_control *wbc) nfs_inc_stats(inode, NFSIOS_VFSWRITEPAGES); - NFS_PROTO(inode)->write_pageio_init(&pgio, inode, wb_priority(wbc), &nfs_async_write_completion_ops); + nfs_pageio_init_write(&pgio, inode, wb_priority(wbc), false, + &nfs_async_write_completion_ops); err = write_cache_pages(mapping, wbc, nfs_writepages_callback, &pgio); nfs_pageio_complete(&pgio); @@ -425,6 +430,8 @@ static void nfs_inode_add_request(struct inode *inode, struct nfs_page *req) { struct nfs_inode *nfsi = NFS_I(inode); + WARN_ON_ONCE(req->wb_this_page != req); + /* Lock the request! */ nfs_lock_request(req); @@ -441,6 +448,7 @@ static void nfs_inode_add_request(struct inode *inode, struct nfs_page *req) set_page_private(req->wb_page, (unsigned long)req); } nfsi->npages++; + set_bit(PG_INODE_REF, &req->wb_flags); kref_get(&req->wb_kref); spin_unlock(&inode->i_lock); } @@ -452,15 +460,20 @@ static void nfs_inode_remove_request(struct nfs_page *req) { struct inode *inode = req->wb_context->dentry->d_inode; struct nfs_inode *nfsi = NFS_I(inode); + struct nfs_page *head; - spin_lock(&inode->i_lock); - if (likely(!PageSwapCache(req->wb_page))) { - set_page_private(req->wb_page, 0); - ClearPagePrivate(req->wb_page); - clear_bit(PG_MAPPED, &req->wb_flags); + if (nfs_page_group_sync_on_bit(req, PG_REMOVE)) { + head = req->wb_head; + + spin_lock(&inode->i_lock); + if (likely(!PageSwapCache(head->wb_page))) { + set_page_private(head->wb_page, 0); + ClearPagePrivate(head->wb_page); + clear_bit(PG_MAPPED, &head->wb_flags); + } + nfsi->npages--; + spin_unlock(&inode->i_lock); } - nfsi->npages--; - spin_unlock(&inode->i_lock); nfs_release_request(req); } @@ -583,7 +596,7 @@ nfs_clear_request_commit(struct nfs_page *req) } static inline -int nfs_write_need_commit(struct nfs_write_data *data) +int nfs_write_need_commit(struct nfs_pgio_data *data) { if (data->verf.committed == NFS_DATA_SYNC) return data->header->lseg == NULL; @@ -614,7 +627,7 @@ nfs_clear_request_commit(struct nfs_page *req) } static inline -int nfs_write_need_commit(struct nfs_write_data *data) +int nfs_write_need_commit(struct nfs_pgio_data *data) { return 0; } @@ -625,6 +638,7 @@ static void nfs_write_completion(struct nfs_pgio_header *hdr) { struct nfs_commit_info cinfo; unsigned long bytes = 0; + bool do_destroy; if (test_bit(NFS_IOHDR_REDO, &hdr->flags)) goto out; @@ -645,7 +659,7 @@ static void nfs_write_completion(struct nfs_pgio_header *hdr) goto next; } if (test_bit(NFS_IOHDR_NEED_COMMIT, &hdr->flags)) { - memcpy(&req->wb_verf, &hdr->verf->verifier, sizeof(req->wb_verf)); + memcpy(&req->wb_verf, &hdr->verf.verifier, sizeof(req->wb_verf)); nfs_mark_request_commit(req, hdr->lseg, &cinfo); goto next; } @@ -653,7 +667,8 @@ remove_req: nfs_inode_remove_request(req); next: nfs_unlock_request(req); - nfs_end_page_writeback(req->wb_page); + nfs_end_page_writeback(req); + do_destroy = !test_bit(NFS_IOHDR_NEED_COMMIT, &hdr->flags); nfs_release_request(req); } out: @@ -661,7 +676,7 @@ out: } #if IS_ENABLED(CONFIG_NFS_V3) || IS_ENABLED(CONFIG_NFS_V4) -static unsigned long +unsigned long nfs_reqs_to_commit(struct nfs_commit_info *cinfo) { return cinfo->mds->ncommit; @@ -718,7 +733,7 @@ nfs_scan_commit(struct inode *inode, struct list_head *dst, } #else -static unsigned long nfs_reqs_to_commit(struct nfs_commit_info *cinfo) +unsigned long nfs_reqs_to_commit(struct nfs_commit_info *cinfo) { return 0; } @@ -758,6 +773,10 @@ static struct nfs_page *nfs_try_to_update_request(struct inode *inode, if (req == NULL) goto out_unlock; + /* should be handled by nfs_flush_incompatible */ + WARN_ON_ONCE(req->wb_head != req); + WARN_ON_ONCE(req->wb_this_page != req); + rqend = req->wb_offset + req->wb_bytes; /* * Tell the caller to flush out the request if @@ -819,7 +838,7 @@ static struct nfs_page * nfs_setup_write_request(struct nfs_open_context* ctx, req = nfs_try_to_update_request(inode, page, offset, bytes); if (req != NULL) goto out; - req = nfs_create_request(ctx, inode, page, offset, bytes); + req = nfs_create_request(ctx, page, NULL, offset, bytes); if (IS_ERR(req)) goto out; nfs_inode_add_request(inode, req); @@ -837,7 +856,7 @@ static int nfs_writepage_setup(struct nfs_open_context *ctx, struct page *page, return PTR_ERR(req); /* Update file length */ nfs_grow_file(page, offset, count); - nfs_mark_uptodate(page, req->wb_pgbase, req->wb_bytes); + nfs_mark_uptodate(req); nfs_mark_request_dirty(req); nfs_unlock_and_release_request(req); return 0; @@ -863,6 +882,8 @@ int nfs_flush_incompatible(struct file *file, struct page *page) return 0; l_ctx = req->wb_lock_context; do_flush = req->wb_page != page || req->wb_context != ctx; + /* for now, flush if more than 1 request in page_group */ + do_flush |= req->wb_this_page != req; if (l_ctx && ctx->dentry->d_inode->i_flock != NULL) { do_flush |= l_ctx->lockowner.l_owner != current->files || l_ctx->lockowner.l_pid != current->tgid; @@ -990,126 +1011,17 @@ static int flush_task_priority(int how) return RPC_PRIORITY_NORMAL; } -int nfs_initiate_write(struct rpc_clnt *clnt, - struct nfs_write_data *data, - const struct rpc_call_ops *call_ops, - int how, int flags) +static void nfs_initiate_write(struct nfs_pgio_data *data, struct rpc_message *msg, + struct rpc_task_setup *task_setup_data, int how) { struct inode *inode = data->header->inode; int priority = flush_task_priority(how); - struct rpc_task *task; - struct rpc_message msg = { - .rpc_argp = &data->args, - .rpc_resp = &data->res, - .rpc_cred = data->header->cred, - }; - struct rpc_task_setup task_setup_data = { - .rpc_client = clnt, - .task = &data->task, - .rpc_message = &msg, - .callback_ops = call_ops, - .callback_data = data, - .workqueue = nfsiod_workqueue, - .flags = RPC_TASK_ASYNC | flags, - .priority = priority, - }; - int ret = 0; - - /* Set up the initial task struct. */ - NFS_PROTO(inode)->write_setup(data, &msg); - dprintk("NFS: %5u initiated write call " - "(req %s/%llu, %u bytes @ offset %llu)\n", - data->task.tk_pid, - inode->i_sb->s_id, - (unsigned long long)NFS_FILEID(inode), - data->args.count, - (unsigned long long)data->args.offset); + task_setup_data->priority = priority; + NFS_PROTO(inode)->write_setup(data, msg); nfs4_state_protect_write(NFS_SERVER(inode)->nfs_client, - &task_setup_data.rpc_client, &msg, data); - - task = rpc_run_task(&task_setup_data); - if (IS_ERR(task)) { - ret = PTR_ERR(task); - goto out; - } - if (how & FLUSH_SYNC) { - ret = rpc_wait_for_completion_task(task); - if (ret == 0) - ret = task->tk_status; - } - rpc_put_task(task); -out: - return ret; -} -EXPORT_SYMBOL_GPL(nfs_initiate_write); - -/* - * Set up the argument/result storage required for the RPC call. - */ -static void nfs_write_rpcsetup(struct nfs_write_data *data, - unsigned int count, unsigned int offset, - int how, struct nfs_commit_info *cinfo) -{ - struct nfs_page *req = data->header->req; - - /* Set up the RPC argument and reply structs - * NB: take care not to mess about with data->commit et al. */ - - data->args.fh = NFS_FH(data->header->inode); - data->args.offset = req_offset(req) + offset; - /* pnfs_set_layoutcommit needs this */ - data->mds_offset = data->args.offset; - data->args.pgbase = req->wb_pgbase + offset; - data->args.pages = data->pages.pagevec; - data->args.count = count; - data->args.context = get_nfs_open_context(req->wb_context); - data->args.lock_context = req->wb_lock_context; - data->args.stable = NFS_UNSTABLE; - switch (how & (FLUSH_STABLE | FLUSH_COND_STABLE)) { - case 0: - break; - case FLUSH_COND_STABLE: - if (nfs_reqs_to_commit(cinfo)) - break; - default: - data->args.stable = NFS_FILE_SYNC; - } - - data->res.fattr = &data->fattr; - data->res.count = count; - data->res.verf = &data->verf; - nfs_fattr_init(&data->fattr); -} - -static int nfs_do_write(struct nfs_write_data *data, - const struct rpc_call_ops *call_ops, - int how) -{ - struct inode *inode = data->header->inode; - - return nfs_initiate_write(NFS_CLIENT(inode), data, call_ops, how, 0); -} - -static int nfs_do_multiple_writes(struct list_head *head, - const struct rpc_call_ops *call_ops, - int how) -{ - struct nfs_write_data *data; - int ret = 0; - - while (!list_empty(head)) { - int ret2; - - data = list_first_entry(head, struct nfs_write_data, list); - list_del_init(&data->list); - - ret2 = nfs_do_write(data, call_ops, how); - if (ret == 0) - ret = ret2; - } - return ret; + &task_setup_data->rpc_client, msg, data); } /* If a nfs_flush_* function fails, it should remove reqs from @head and @@ -1120,7 +1032,7 @@ static void nfs_redirty_request(struct nfs_page *req) { nfs_mark_request_dirty(req); nfs_unlock_request(req); - nfs_end_page_writeback(req->wb_page); + nfs_end_page_writeback(req); nfs_release_request(req); } @@ -1140,173 +1052,30 @@ static const struct nfs_pgio_completion_ops nfs_async_write_completion_ops = { .completion = nfs_write_completion, }; -static void nfs_flush_error(struct nfs_pageio_descriptor *desc, - struct nfs_pgio_header *hdr) -{ - set_bit(NFS_IOHDR_REDO, &hdr->flags); - while (!list_empty(&hdr->rpc_list)) { - struct nfs_write_data *data = list_first_entry(&hdr->rpc_list, - struct nfs_write_data, list); - list_del(&data->list); - nfs_writedata_release(data); - } - desc->pg_completion_ops->error_cleanup(&desc->pg_list); -} - -/* - * Generate multiple small requests to write out a single - * contiguous dirty area on one page. - */ -static int nfs_flush_multi(struct nfs_pageio_descriptor *desc, - struct nfs_pgio_header *hdr) -{ - struct nfs_page *req = hdr->req; - struct page *page = req->wb_page; - struct nfs_write_data *data; - size_t wsize = desc->pg_bsize, nbytes; - unsigned int offset; - int requests = 0; - struct nfs_commit_info cinfo; - - nfs_init_cinfo(&cinfo, desc->pg_inode, desc->pg_dreq); - - if ((desc->pg_ioflags & FLUSH_COND_STABLE) && - (desc->pg_moreio || nfs_reqs_to_commit(&cinfo) || - desc->pg_count > wsize)) - desc->pg_ioflags &= ~FLUSH_COND_STABLE; - - - offset = 0; - nbytes = desc->pg_count; - do { - size_t len = min(nbytes, wsize); - - data = nfs_writedata_alloc(hdr, 1); - if (!data) { - nfs_flush_error(desc, hdr); - return -ENOMEM; - } - data->pages.pagevec[0] = page; - nfs_write_rpcsetup(data, len, offset, desc->pg_ioflags, &cinfo); - list_add(&data->list, &hdr->rpc_list); - requests++; - nbytes -= len; - offset += len; - } while (nbytes != 0); - nfs_list_remove_request(req); - nfs_list_add_request(req, &hdr->pages); - desc->pg_rpc_callops = &nfs_write_common_ops; - return 0; -} - -/* - * Create an RPC task for the given write request and kick it. - * The page must have been locked by the caller. - * - * It may happen that the page we're passed is not marked dirty. - * This is the case if nfs_updatepage detects a conflicting request - * that has been written but not committed. - */ -static int nfs_flush_one(struct nfs_pageio_descriptor *desc, - struct nfs_pgio_header *hdr) -{ - struct nfs_page *req; - struct page **pages; - struct nfs_write_data *data; - struct list_head *head = &desc->pg_list; - struct nfs_commit_info cinfo; - - data = nfs_writedata_alloc(hdr, nfs_page_array_len(desc->pg_base, - desc->pg_count)); - if (!data) { - nfs_flush_error(desc, hdr); - return -ENOMEM; - } - - nfs_init_cinfo(&cinfo, desc->pg_inode, desc->pg_dreq); - pages = data->pages.pagevec; - while (!list_empty(head)) { - req = nfs_list_entry(head->next); - nfs_list_remove_request(req); - nfs_list_add_request(req, &hdr->pages); - *pages++ = req->wb_page; - } - - if ((desc->pg_ioflags & FLUSH_COND_STABLE) && - (desc->pg_moreio || nfs_reqs_to_commit(&cinfo))) - desc->pg_ioflags &= ~FLUSH_COND_STABLE; - - /* Set up the argument struct */ - nfs_write_rpcsetup(data, desc->pg_count, 0, desc->pg_ioflags, &cinfo); - list_add(&data->list, &hdr->rpc_list); - desc->pg_rpc_callops = &nfs_write_common_ops; - return 0; -} - -int nfs_generic_flush(struct nfs_pageio_descriptor *desc, - struct nfs_pgio_header *hdr) -{ - if (desc->pg_bsize < PAGE_CACHE_SIZE) - return nfs_flush_multi(desc, hdr); - return nfs_flush_one(desc, hdr); -} -EXPORT_SYMBOL_GPL(nfs_generic_flush); - -static int nfs_generic_pg_writepages(struct nfs_pageio_descriptor *desc) -{ - struct nfs_write_header *whdr; - struct nfs_pgio_header *hdr; - int ret; - - whdr = nfs_writehdr_alloc(); - if (!whdr) { - desc->pg_completion_ops->error_cleanup(&desc->pg_list); - return -ENOMEM; - } - hdr = &whdr->header; - nfs_pgheader_init(desc, hdr, nfs_writehdr_free); - atomic_inc(&hdr->refcnt); - ret = nfs_generic_flush(desc, hdr); - if (ret == 0) - ret = nfs_do_multiple_writes(&hdr->rpc_list, - desc->pg_rpc_callops, - desc->pg_ioflags); - if (atomic_dec_and_test(&hdr->refcnt)) - hdr->completion_ops->completion(hdr); - return ret; -} - -static const struct nfs_pageio_ops nfs_pageio_write_ops = { - .pg_test = nfs_generic_pg_test, - .pg_doio = nfs_generic_pg_writepages, -}; - void nfs_pageio_init_write(struct nfs_pageio_descriptor *pgio, - struct inode *inode, int ioflags, + struct inode *inode, int ioflags, bool force_mds, const struct nfs_pgio_completion_ops *compl_ops) { - nfs_pageio_init(pgio, inode, &nfs_pageio_write_ops, compl_ops, - NFS_SERVER(inode)->wsize, ioflags); + struct nfs_server *server = NFS_SERVER(inode); + const struct nfs_pageio_ops *pg_ops = &nfs_pgio_rw_ops; + +#ifdef CONFIG_NFS_V4_1 + if (server->pnfs_curr_ld && !force_mds) + pg_ops = server->pnfs_curr_ld->pg_write_ops; +#endif + nfs_pageio_init(pgio, inode, pg_ops, compl_ops, &nfs_rw_write_ops, + server->wsize, ioflags); } EXPORT_SYMBOL_GPL(nfs_pageio_init_write); void nfs_pageio_reset_write_mds(struct nfs_pageio_descriptor *pgio) { - pgio->pg_ops = &nfs_pageio_write_ops; + pgio->pg_ops = &nfs_pgio_rw_ops; pgio->pg_bsize = NFS_SERVER(pgio->pg_inode)->wsize; } EXPORT_SYMBOL_GPL(nfs_pageio_reset_write_mds); -void nfs_write_prepare(struct rpc_task *task, void *calldata) -{ - struct nfs_write_data *data = calldata; - int err; - err = NFS_PROTO(data->header->inode)->write_rpc_prepare(task, data); - if (err) - rpc_exit(task, err); -} - void nfs_commit_prepare(struct rpc_task *task, void *calldata) { struct nfs_commit_data *data = calldata; @@ -1314,23 +1083,8 @@ void nfs_commit_prepare(struct rpc_task *task, void *calldata) NFS_PROTO(data->inode)->commit_rpc_prepare(task, data); } -/* - * Handle a write reply that flushes a whole page. - * - * FIXME: There is an inherent race with invalidate_inode_pages and - * writebacks since the page->count is kept > 1 for as long - * as the page has a write request pending. - */ -static void nfs_writeback_done_common(struct rpc_task *task, void *calldata) -{ - struct nfs_write_data *data = calldata; - - nfs_writeback_done(task, data); -} - -static void nfs_writeback_release_common(void *calldata) +static void nfs_writeback_release_common(struct nfs_pgio_data *data) { - struct nfs_write_data *data = calldata; struct nfs_pgio_header *hdr = data->header; int status = data->task.tk_status; @@ -1339,34 +1093,46 @@ static void nfs_writeback_release_common(void *calldata) if (test_bit(NFS_IOHDR_NEED_RESCHED, &hdr->flags)) ; /* Do nothing */ else if (!test_and_set_bit(NFS_IOHDR_NEED_COMMIT, &hdr->flags)) - memcpy(hdr->verf, &data->verf, sizeof(*hdr->verf)); - else if (memcmp(hdr->verf, &data->verf, sizeof(*hdr->verf))) + memcpy(&hdr->verf, &data->verf, sizeof(hdr->verf)); + else if (memcmp(&hdr->verf, &data->verf, sizeof(hdr->verf))) set_bit(NFS_IOHDR_NEED_RESCHED, &hdr->flags); spin_unlock(&hdr->lock); } - nfs_writedata_release(data); } -static const struct rpc_call_ops nfs_write_common_ops = { - .rpc_call_prepare = nfs_write_prepare, - .rpc_call_done = nfs_writeback_done_common, - .rpc_release = nfs_writeback_release_common, -}; +/* + * Special version of should_remove_suid() that ignores capabilities. + */ +static int nfs_should_remove_suid(const struct inode *inode) +{ + umode_t mode = inode->i_mode; + int kill = 0; + + /* suid always must be killed */ + if (unlikely(mode & S_ISUID)) + kill = ATTR_KILL_SUID; + /* + * sgid without any exec bits is just a mandatory locking mark; leave + * it alone. If some exec bits are set, it's a real sgid; kill it. + */ + if (unlikely((mode & S_ISGID) && (mode & S_IXGRP))) + kill |= ATTR_KILL_SGID; + + if (unlikely(kill && S_ISREG(mode))) + return kill; + + return 0; +} /* * This function is called when the WRITE call is complete. */ -void nfs_writeback_done(struct rpc_task *task, struct nfs_write_data *data) +static int nfs_writeback_done(struct rpc_task *task, struct nfs_pgio_data *data, + struct inode *inode) { - struct nfs_writeargs *argp = &data->args; - struct nfs_writeres *resp = &data->res; - struct inode *inode = data->header->inode; int status; - dprintk("NFS: %5u nfs_writeback_done (status %d)\n", - task->tk_pid, task->tk_status); - /* * ->write_done will attempt to use post-op attributes to detect * conflicting writes by other clients. A strict interpretation @@ -1376,11 +1142,11 @@ void nfs_writeback_done(struct rpc_task *task, struct nfs_write_data *data) */ status = NFS_PROTO(inode)->write_done(task, data); if (status != 0) - return; - nfs_add_stats(inode, NFSIOS_SERVERWRITTENBYTES, resp->count); + return status; + nfs_add_stats(inode, NFSIOS_SERVERWRITTENBYTES, data->res.count); #if IS_ENABLED(CONFIG_NFS_V3) || IS_ENABLED(CONFIG_NFS_V4) - if (resp->verf->committed < argp->stable && task->tk_status >= 0) { + if (data->res.verf->committed < data->args.stable && task->tk_status >= 0) { /* We tried a write call, but the server did not * commit data to stable storage even though we * requested it. @@ -1396,18 +1162,31 @@ void nfs_writeback_done(struct rpc_task *task, struct nfs_write_data *data) dprintk("NFS: faulty NFS server %s:" " (committed = %d) != (stable = %d)\n", NFS_SERVER(inode)->nfs_client->cl_hostname, - resp->verf->committed, argp->stable); + data->res.verf->committed, data->args.stable); complain = jiffies + 300 * HZ; } } #endif - if (task->tk_status < 0) - nfs_set_pgio_error(data->header, task->tk_status, argp->offset); - else if (resp->count < argp->count) { + + /* Deal with the suid/sgid bit corner case */ + if (nfs_should_remove_suid(inode)) + nfs_mark_for_revalidate(inode); + return 0; +} + +/* + * This function is called when the WRITE call is complete. + */ +static void nfs_writeback_result(struct rpc_task *task, struct nfs_pgio_data *data) +{ + struct nfs_pgio_args *argp = &data->args; + struct nfs_pgio_res *resp = &data->res; + + if (resp->count < argp->count) { static unsigned long complain; /* This a short write! */ - nfs_inc_stats(inode, NFSIOS_SHORTWRITE); + nfs_inc_stats(data->header->inode, NFSIOS_SHORTWRITE); /* Has the server at least made some progress? */ if (resp->count == 0) { @@ -1874,7 +1653,7 @@ int nfs_migrate_page(struct address_space *mapping, struct page *newpage, int __init nfs_init_writepagecache(void) { nfs_wdata_cachep = kmem_cache_create("nfs_write_data", - sizeof(struct nfs_write_header), + sizeof(struct nfs_rw_header), 0, SLAB_HWCACHE_ALIGN, NULL); if (nfs_wdata_cachep == NULL) @@ -1936,3 +1715,12 @@ void nfs_destroy_writepagecache(void) kmem_cache_destroy(nfs_wdata_cachep); } +static const struct nfs_rw_ops nfs_rw_write_ops = { + .rw_mode = FMODE_WRITE, + .rw_alloc_header = nfs_writehdr_alloc, + .rw_free_header = nfs_writehdr_free, + .rw_release = nfs_writeback_release_common, + .rw_done = nfs_writeback_done, + .rw_result = nfs_writeback_result, + .rw_initiate = nfs_initiate_write, +}; diff --git a/fs/nfsd/acl.h b/fs/nfsd/acl.h index b481e1f5eecc..a986ceb6fd0d 100644 --- a/fs/nfsd/acl.h +++ b/fs/nfsd/acl.h @@ -49,7 +49,7 @@ struct svc_rqst; struct nfs4_acl *nfs4_acl_new(int); int nfs4_acl_get_whotype(char *, u32); -__be32 nfs4_acl_write_who(int who, __be32 **p, int *len); +__be32 nfs4_acl_write_who(struct xdr_stream *xdr, int who); int nfsd4_get_nfs4_acl(struct svc_rqst *rqstp, struct dentry *dentry, struct nfs4_acl **acl); diff --git a/fs/nfsd/auth.c b/fs/nfsd/auth.c index 2645be435e75..72f44823adbb 100644 --- a/fs/nfsd/auth.c +++ b/fs/nfsd/auth.c @@ -1,7 +1,6 @@ /* Copyright (C) 1995, 1996 Olaf Kirch <okir@monad.swb.de> */ #include <linux/sched.h> -#include <linux/user_namespace.h> #include "nfsd.h" #include "auth.h" @@ -25,7 +24,6 @@ int nfsd_setuser(struct svc_rqst *rqstp, struct svc_export *exp) struct cred *new; int i; int flags = nfsexp_flags(rqstp, exp); - int ret; validate_process_creds(); @@ -86,8 +84,7 @@ int nfsd_setuser(struct svc_rqst *rqstp, struct svc_export *exp) return 0; oom: - ret = -ENOMEM; abort_creds(new); - return ret; + return -ENOMEM; } diff --git a/fs/nfsd/export.c b/fs/nfsd/export.c index 8513c598fabf..13b85f94d9e2 100644 --- a/fs/nfsd/export.c +++ b/fs/nfsd/export.c @@ -17,17 +17,12 @@ #include <linux/exportfs.h> #include <linux/sunrpc/svc_xprt.h> -#include <net/ipv6.h> - #include "nfsd.h" #include "nfsfh.h" #include "netns.h" #define NFSDDBG_FACILITY NFSDDBG_EXPORT -typedef struct auth_domain svc_client; -typedef struct svc_export svc_export; - /* * We have two caches. * One maps client+vfsmnt+dentry to export options - the export map @@ -73,7 +68,7 @@ static struct svc_expkey *svc_expkey_lookup(struct cache_detail *cd, struct svc_ static int expkey_parse(struct cache_detail *cd, char *mesg, int mlen) { - /* client fsidtype fsid [path] */ + /* client fsidtype fsid expiry [path] */ char *buf; int len; struct auth_domain *dom = NULL; @@ -295,13 +290,19 @@ svc_expkey_update(struct cache_detail *cd, struct svc_expkey *new, static void nfsd4_fslocs_free(struct nfsd4_fs_locations *fsloc) { + struct nfsd4_fs_location *locations = fsloc->locations; int i; + if (!locations) + return; + for (i = 0; i < fsloc->locations_count; i++) { - kfree(fsloc->locations[i].path); - kfree(fsloc->locations[i].hosts); + kfree(locations[i].path); + kfree(locations[i].hosts); } - kfree(fsloc->locations); + + kfree(locations); + fsloc->locations = NULL; } static void svc_export_put(struct kref *ref) @@ -388,6 +389,10 @@ fsloc_parse(char **mesg, char *buf, struct nfsd4_fs_locations *fsloc) int len; int migrated, i, err; + /* more than one fsloc */ + if (fsloc->locations) + return -EINVAL; + /* listsize */ err = get_uint(mesg, &fsloc->locations_count); if (err) @@ -437,13 +442,18 @@ out_free_all: static int secinfo_parse(char **mesg, char *buf, struct svc_export *exp) { - int listsize, err; struct exp_flavor_info *f; + u32 listsize; + int err; + + /* more than one secinfo */ + if (exp->ex_nflavors) + return -EINVAL; - err = get_int(mesg, &listsize); + err = get_uint(mesg, &listsize); if (err) return err; - if (listsize < 0 || listsize > MAX_SECINFO_LIST) + if (listsize > MAX_SECINFO_LIST) return -EINVAL; for (f = exp->ex_flavors; f < exp->ex_flavors + listsize; f++) { @@ -474,6 +484,27 @@ static inline int secinfo_parse(char **mesg, char *buf, struct svc_export *exp) { return 0; } #endif +static inline int +uuid_parse(char **mesg, char *buf, unsigned char **puuid) +{ + int len; + + /* more than one uuid */ + if (*puuid) + return -EINVAL; + + /* expect a 16 byte uuid encoded as \xXXXX... */ + len = qword_get(mesg, buf, PAGE_SIZE); + if (len != EX_UUID_LEN) + return -EINVAL; + + *puuid = kmemdup(buf, EX_UUID_LEN, GFP_KERNEL); + if (*puuid == NULL) + return -ENOMEM; + + return 0; +} + static int svc_export_parse(struct cache_detail *cd, char *mesg, int mlen) { /* client path expiry [flags anonuid anongid fsid] */ @@ -552,18 +583,9 @@ static int svc_export_parse(struct cache_detail *cd, char *mesg, int mlen) while ((len = qword_get(&mesg, buf, PAGE_SIZE)) > 0) { if (strcmp(buf, "fsloc") == 0) err = fsloc_parse(&mesg, buf, &exp.ex_fslocs); - else if (strcmp(buf, "uuid") == 0) { - /* expect a 16 byte uuid encoded as \xXXXX... */ - len = qword_get(&mesg, buf, PAGE_SIZE); - if (len != 16) - err = -EINVAL; - else { - exp.ex_uuid = - kmemdup(buf, 16, GFP_KERNEL); - if (exp.ex_uuid == NULL) - err = -ENOMEM; - } - } else if (strcmp(buf, "secinfo") == 0) + else if (strcmp(buf, "uuid") == 0) + err = uuid_parse(&mesg, buf, &exp.ex_uuid); + else if (strcmp(buf, "secinfo") == 0) err = secinfo_parse(&mesg, buf, &exp); else /* quietly ignore unknown words and anything @@ -649,7 +671,7 @@ static int svc_export_show(struct seq_file *m, if (exp->ex_uuid) { int i; seq_puts(m, ",uuid="); - for (i=0; i<16; i++) { + for (i = 0; i < EX_UUID_LEN; i++) { if ((i&3) == 0 && i) seq_putc(m, ':'); seq_printf(m, "%02x", exp->ex_uuid[i]); @@ -771,7 +793,7 @@ svc_export_update(struct svc_export *new, struct svc_export *old) static struct svc_expkey * -exp_find_key(struct cache_detail *cd, svc_client *clp, int fsid_type, +exp_find_key(struct cache_detail *cd, struct auth_domain *clp, int fsid_type, u32 *fsidv, struct cache_req *reqp) { struct svc_expkey key, *ek; @@ -793,9 +815,9 @@ exp_find_key(struct cache_detail *cd, svc_client *clp, int fsid_type, return ek; } - -static svc_export *exp_get_by_name(struct cache_detail *cd, svc_client *clp, - const struct path *path, struct cache_req *reqp) +static struct svc_export * +exp_get_by_name(struct cache_detail *cd, struct auth_domain *clp, + const struct path *path, struct cache_req *reqp) { struct svc_export *exp, key; int err; @@ -819,11 +841,11 @@ static svc_export *exp_get_by_name(struct cache_detail *cd, svc_client *clp, /* * Find the export entry for a given dentry. */ -static struct svc_export *exp_parent(struct cache_detail *cd, svc_client *clp, - struct path *path) +static struct svc_export * +exp_parent(struct cache_detail *cd, struct auth_domain *clp, struct path *path) { struct dentry *saved = dget(path->dentry); - svc_export *exp = exp_get_by_name(cd, clp, path, NULL); + struct svc_export *exp = exp_get_by_name(cd, clp, path, NULL); while (PTR_ERR(exp) == -ENOENT && !IS_ROOT(path->dentry)) { struct dentry *parent = dget_parent(path->dentry); @@ -844,7 +866,7 @@ static struct svc_export *exp_parent(struct cache_detail *cd, svc_client *clp, * since its harder to fool a kernel module than a user space program. */ int -exp_rootfh(struct net *net, svc_client *clp, char *name, +exp_rootfh(struct net *net, struct auth_domain *clp, char *name, struct knfsd_fh *f, int maxsize) { struct svc_export *exp; diff --git a/include/linux/nfsd/export.h b/fs/nfsd/export.h index 7898c997dfea..cfeea85c5bed 100644 --- a/include/linux/nfsd/export.h +++ b/fs/nfsd/export.h @@ -1,17 +1,16 @@ /* - * include/linux/nfsd/export.h - * - * Public declarations for NFS exports. The definitions for the - * syscall interface are in nfsctl.h - * * Copyright (C) 1995-1997 Olaf Kirch <okir@monad.swb.de> */ #ifndef NFSD_EXPORT_H #define NFSD_EXPORT_H -# include <linux/nfsd/nfsfh.h> +#include <linux/sunrpc/cache.h> #include <uapi/linux/nfsd/export.h> +struct knfsd_fh; +struct svc_fh; +struct svc_rqst; + /* * FS Locations */ @@ -38,6 +37,7 @@ struct nfsd4_fs_locations { * spkm3i, and spkm3p (and using all 8 at once should be rare). */ #define MAX_SECINFO_LIST 8 +#define EX_UUID_LEN 16 struct exp_flavor_info { u32 pseudoflavor; @@ -54,7 +54,7 @@ struct svc_export { int ex_fsid; unsigned char * ex_uuid; /* 16 byte fsid */ struct nfsd4_fs_locations ex_fslocs; - int ex_nflavors; + uint32_t ex_nflavors; struct exp_flavor_info ex_flavors[MAX_SECINFO_LIST]; struct cache_detail *cd; }; diff --git a/fs/nfsd/fault_inject.c b/fs/nfsd/fault_inject.c index d620e7f81429..2ed05c3cd43d 100644 --- a/fs/nfsd/fault_inject.c +++ b/fs/nfsd/fault_inject.c @@ -97,25 +97,14 @@ static ssize_t fault_inject_read(struct file *file, char __user *buf, { static u64 val; char read_buf[25]; - size_t size, ret; + size_t size; loff_t pos = *ppos; if (!pos) nfsd_inject_get(file_inode(file)->i_private, &val); size = scnprintf(read_buf, sizeof(read_buf), "%llu\n", val); - if (pos < 0) - return -EINVAL; - if (pos >= size || !len) - return 0; - if (len > size - pos) - len = size - pos; - ret = copy_to_user(buf, read_buf + pos, len); - if (ret == len) - return -EFAULT; - len -= ret; - *ppos = pos + len; - return len; + return simple_read_from_buffer(buf, len, ppos, read_buf, size); } static ssize_t fault_inject_write(struct file *file, const char __user *buf, diff --git a/fs/nfsd/idmap.h b/fs/nfsd/idmap.h index 66e58db01936..a3f34900091f 100644 --- a/fs/nfsd/idmap.h +++ b/fs/nfsd/idmap.h @@ -56,7 +56,7 @@ static inline void nfsd_idmap_shutdown(struct net *net) __be32 nfsd_map_name_to_uid(struct svc_rqst *, const char *, size_t, kuid_t *); __be32 nfsd_map_name_to_gid(struct svc_rqst *, const char *, size_t, kgid_t *); -__be32 nfsd4_encode_user(struct svc_rqst *, kuid_t, __be32 **, int *); -__be32 nfsd4_encode_group(struct svc_rqst *, kgid_t, __be32 **, int *); +__be32 nfsd4_encode_user(struct xdr_stream *, struct svc_rqst *, kuid_t); +__be32 nfsd4_encode_group(struct xdr_stream *, struct svc_rqst *, kgid_t); #endif /* LINUX_NFSD_IDMAP_H */ diff --git a/fs/nfsd/nfs2acl.c b/fs/nfsd/nfs2acl.c index 11c1fba29312..12b023a7ab7d 100644 --- a/fs/nfsd/nfs2acl.c +++ b/fs/nfsd/nfs2acl.c @@ -182,7 +182,8 @@ static __be32 nfsacld_proc_access(struct svc_rqst *rqstp, struct nfsd3_accessarg static int nfsaclsvc_decode_getaclargs(struct svc_rqst *rqstp, __be32 *p, struct nfsd3_getaclargs *argp) { - if (!(p = nfs2svc_decode_fh(p, &argp->fh))) + p = nfs2svc_decode_fh(p, &argp->fh); + if (!p) return 0; argp->mask = ntohl(*p); p++; @@ -197,7 +198,8 @@ static int nfsaclsvc_decode_setaclargs(struct svc_rqst *rqstp, __be32 *p, unsigned int base; int n; - if (!(p = nfs2svc_decode_fh(p, &argp->fh))) + p = nfs2svc_decode_fh(p, &argp->fh); + if (!p) return 0; argp->mask = ntohl(*p++); if (argp->mask & ~(NFS_ACL|NFS_ACLCNT|NFS_DFACL|NFS_DFACLCNT) || @@ -218,7 +220,8 @@ static int nfsaclsvc_decode_setaclargs(struct svc_rqst *rqstp, __be32 *p, static int nfsaclsvc_decode_fhandleargs(struct svc_rqst *rqstp, __be32 *p, struct nfsd_fhandle *argp) { - if (!(p = nfs2svc_decode_fh(p, &argp->fh))) + p = nfs2svc_decode_fh(p, &argp->fh); + if (!p) return 0; return xdr_argsize_check(rqstp, p); } @@ -226,7 +229,8 @@ static int nfsaclsvc_decode_fhandleargs(struct svc_rqst *rqstp, __be32 *p, static int nfsaclsvc_decode_accessargs(struct svc_rqst *rqstp, __be32 *p, struct nfsd3_accessargs *argp) { - if (!(p = nfs2svc_decode_fh(p, &argp->fh))) + p = nfs2svc_decode_fh(p, &argp->fh); + if (!p) return 0; argp->access = ntohl(*p++); diff --git a/fs/nfsd/nfs3acl.c b/fs/nfsd/nfs3acl.c index adc5f1b1dc26..2a514e21dc74 100644 --- a/fs/nfsd/nfs3acl.c +++ b/fs/nfsd/nfs3acl.c @@ -128,7 +128,8 @@ out: static int nfs3svc_decode_getaclargs(struct svc_rqst *rqstp, __be32 *p, struct nfsd3_getaclargs *args) { - if (!(p = nfs3svc_decode_fh(p, &args->fh))) + p = nfs3svc_decode_fh(p, &args->fh); + if (!p) return 0; args->mask = ntohl(*p); p++; @@ -143,7 +144,8 @@ static int nfs3svc_decode_setaclargs(struct svc_rqst *rqstp, __be32 *p, unsigned int base; int n; - if (!(p = nfs3svc_decode_fh(p, &args->fh))) + p = nfs3svc_decode_fh(p, &args->fh); + if (!p) return 0; args->mask = ntohl(*p++); if (args->mask & ~(NFS_ACL|NFS_ACLCNT|NFS_DFACL|NFS_DFACLCNT) || diff --git a/fs/nfsd/nfs3xdr.c b/fs/nfsd/nfs3xdr.c index de6e39e12cb3..e6c01e80325e 100644 --- a/fs/nfsd/nfs3xdr.c +++ b/fs/nfsd/nfs3xdr.c @@ -278,7 +278,8 @@ void fill_post_wcc(struct svc_fh *fhp) int nfs3svc_decode_fhandle(struct svc_rqst *rqstp, __be32 *p, struct nfsd_fhandle *args) { - if (!(p = decode_fh(p, &args->fh))) + p = decode_fh(p, &args->fh); + if (!p) return 0; return xdr_argsize_check(rqstp, p); } @@ -287,7 +288,8 @@ int nfs3svc_decode_sattrargs(struct svc_rqst *rqstp, __be32 *p, struct nfsd3_sattrargs *args) { - if (!(p = decode_fh(p, &args->fh))) + p = decode_fh(p, &args->fh); + if (!p) return 0; p = decode_sattr3(p, &args->attrs); @@ -315,7 +317,8 @@ int nfs3svc_decode_accessargs(struct svc_rqst *rqstp, __be32 *p, struct nfsd3_accessargs *args) { - if (!(p = decode_fh(p, &args->fh))) + p = decode_fh(p, &args->fh); + if (!p) return 0; args->access = ntohl(*p++); @@ -330,7 +333,8 @@ nfs3svc_decode_readargs(struct svc_rqst *rqstp, __be32 *p, int v; u32 max_blocksize = svc_max_payload(rqstp); - if (!(p = decode_fh(p, &args->fh))) + p = decode_fh(p, &args->fh); + if (!p) return 0; p = xdr_decode_hyper(p, &args->offset); @@ -360,7 +364,8 @@ nfs3svc_decode_writeargs(struct svc_rqst *rqstp, __be32 *p, unsigned int len, v, hdr, dlen; u32 max_blocksize = svc_max_payload(rqstp); - if (!(p = decode_fh(p, &args->fh))) + p = decode_fh(p, &args->fh); + if (!p) return 0; p = xdr_decode_hyper(p, &args->offset); @@ -535,7 +540,8 @@ int nfs3svc_decode_readlinkargs(struct svc_rqst *rqstp, __be32 *p, struct nfsd3_readlinkargs *args) { - if (!(p = decode_fh(p, &args->fh))) + p = decode_fh(p, &args->fh); + if (!p) return 0; args->buffer = page_address(*(rqstp->rq_next_page++)); @@ -558,7 +564,8 @@ int nfs3svc_decode_readdirargs(struct svc_rqst *rqstp, __be32 *p, struct nfsd3_readdirargs *args) { - if (!(p = decode_fh(p, &args->fh))) + p = decode_fh(p, &args->fh); + if (!p) return 0; p = xdr_decode_hyper(p, &args->cookie); args->verf = p; p += 2; @@ -580,7 +587,8 @@ nfs3svc_decode_readdirplusargs(struct svc_rqst *rqstp, __be32 *p, int len; u32 max_blocksize = svc_max_payload(rqstp); - if (!(p = decode_fh(p, &args->fh))) + p = decode_fh(p, &args->fh); + if (!p) return 0; p = xdr_decode_hyper(p, &args->cookie); args->verf = p; p += 2; @@ -605,7 +613,8 @@ int nfs3svc_decode_commitargs(struct svc_rqst *rqstp, __be32 *p, struct nfsd3_commitargs *args) { - if (!(p = decode_fh(p, &args->fh))) + p = decode_fh(p, &args->fh); + if (!p) return 0; p = xdr_decode_hyper(p, &args->offset); args->count = ntohl(*p++); diff --git a/fs/nfsd/nfs4acl.c b/fs/nfsd/nfs4acl.c index f66c66b9f182..d714156a19fd 100644 --- a/fs/nfsd/nfs4acl.c +++ b/fs/nfsd/nfs4acl.c @@ -36,7 +36,6 @@ #include <linux/slab.h> #include <linux/nfs_fs.h> -#include <linux/export.h> #include "nfsfh.h" #include "nfsd.h" #include "acl.h" @@ -920,20 +919,19 @@ nfs4_acl_get_whotype(char *p, u32 len) return NFS4_ACL_WHO_NAMED; } -__be32 nfs4_acl_write_who(int who, __be32 **p, int *len) +__be32 nfs4_acl_write_who(struct xdr_stream *xdr, int who) { + __be32 *p; int i; - int bytes; for (i = 0; i < ARRAY_SIZE(s2t_map); i++) { if (s2t_map[i].type != who) continue; - bytes = 4 + (XDR_QUADLEN(s2t_map[i].stringlen) << 2); - if (bytes > *len) + p = xdr_reserve_space(xdr, s2t_map[i].stringlen + 4); + if (!p) return nfserr_resource; - *p = xdr_encode_opaque(*p, s2t_map[i].string, + p = xdr_encode_opaque(p, s2t_map[i].string, s2t_map[i].stringlen); - *len -= bytes; return 0; } WARN_ON_ONCE(1); diff --git a/fs/nfsd/nfs4idmap.c b/fs/nfsd/nfs4idmap.c index c0dfde68742e..a0ab0a847d69 100644 --- a/fs/nfsd/nfs4idmap.c +++ b/fs/nfsd/nfs4idmap.c @@ -551,44 +551,43 @@ idmap_name_to_id(struct svc_rqst *rqstp, int type, const char *name, u32 namelen return 0; } -static __be32 encode_ascii_id(u32 id, __be32 **p, int *buflen) +static __be32 encode_ascii_id(struct xdr_stream *xdr, u32 id) { char buf[11]; int len; - int bytes; + __be32 *p; len = sprintf(buf, "%u", id); - bytes = 4 + (XDR_QUADLEN(len) << 2); - if (bytes > *buflen) + p = xdr_reserve_space(xdr, len + 4); + if (!p) return nfserr_resource; - *p = xdr_encode_opaque(*p, buf, len); - *buflen -= bytes; + p = xdr_encode_opaque(p, buf, len); return 0; } -static __be32 idmap_id_to_name(struct svc_rqst *rqstp, int type, u32 id, __be32 **p, int *buflen) +static __be32 idmap_id_to_name(struct xdr_stream *xdr, + struct svc_rqst *rqstp, int type, u32 id) { struct ent *item, key = { .id = id, .type = type, }; + __be32 *p; int ret; - int bytes; struct nfsd_net *nn = net_generic(SVC_NET(rqstp), nfsd_net_id); strlcpy(key.authname, rqst_authname(rqstp), sizeof(key.authname)); ret = idmap_lookup(rqstp, idtoname_lookup, &key, nn->idtoname_cache, &item); if (ret == -ENOENT) - return encode_ascii_id(id, p, buflen); + return encode_ascii_id(xdr, id); if (ret) return nfserrno(ret); ret = strlen(item->name); WARN_ON_ONCE(ret > IDMAP_NAMESZ); - bytes = 4 + (XDR_QUADLEN(ret) << 2); - if (bytes > *buflen) + p = xdr_reserve_space(xdr, ret + 4); + if (!p) return nfserr_resource; - *p = xdr_encode_opaque(*p, item->name, ret); - *buflen -= bytes; + p = xdr_encode_opaque(p, item->name, ret); cache_put(&item->h, nn->idtoname_cache); return 0; } @@ -622,11 +621,12 @@ do_name_to_id(struct svc_rqst *rqstp, int type, const char *name, u32 namelen, u return idmap_name_to_id(rqstp, type, name, namelen, id); } -static __be32 encode_name_from_id(struct svc_rqst *rqstp, int type, u32 id, __be32 **p, int *buflen) +static __be32 encode_name_from_id(struct xdr_stream *xdr, + struct svc_rqst *rqstp, int type, u32 id) { if (nfs4_disable_idmapping && rqstp->rq_cred.cr_flavor < RPC_AUTH_GSS) - return encode_ascii_id(id, p, buflen); - return idmap_id_to_name(rqstp, type, id, p, buflen); + return encode_ascii_id(xdr, id); + return idmap_id_to_name(xdr, rqstp, type, id); } __be32 @@ -655,14 +655,16 @@ nfsd_map_name_to_gid(struct svc_rqst *rqstp, const char *name, size_t namelen, return status; } -__be32 nfsd4_encode_user(struct svc_rqst *rqstp, kuid_t uid, __be32 **p, int *buflen) +__be32 nfsd4_encode_user(struct xdr_stream *xdr, struct svc_rqst *rqstp, + kuid_t uid) { u32 id = from_kuid(&init_user_ns, uid); - return encode_name_from_id(rqstp, IDMAP_TYPE_USER, id, p, buflen); + return encode_name_from_id(xdr, rqstp, IDMAP_TYPE_USER, id); } -__be32 nfsd4_encode_group(struct svc_rqst *rqstp, kgid_t gid, __be32 **p, int *buflen) +__be32 nfsd4_encode_group(struct xdr_stream *xdr, struct svc_rqst *rqstp, + kgid_t gid) { u32 id = from_kgid(&init_user_ns, gid); - return encode_name_from_id(rqstp, IDMAP_TYPE_GROUP, id, p, buflen); + return encode_name_from_id(xdr, rqstp, IDMAP_TYPE_GROUP, id); } diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c index d543222babf3..6851b003f2a4 100644 --- a/fs/nfsd/nfs4proc.c +++ b/fs/nfsd/nfs4proc.c @@ -430,12 +430,12 @@ nfsd4_open(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate, goto out; break; case NFS4_OPEN_CLAIM_PREVIOUS: - open->op_openowner->oo_flags |= NFS4_OO_CONFIRMED; status = nfs4_check_open_reclaim(&open->op_clientid, cstate->minorversion, nn); if (status) goto out; + open->op_openowner->oo_flags |= NFS4_OO_CONFIRMED; case NFS4_OPEN_CLAIM_FH: case NFS4_OPEN_CLAIM_DELEG_CUR_FH: status = do_open_fhandle(rqstp, cstate, open); @@ -445,7 +445,6 @@ nfsd4_open(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate, break; case NFS4_OPEN_CLAIM_DELEG_PREV_FH: case NFS4_OPEN_CLAIM_DELEGATE_PREV: - open->op_openowner->oo_flags |= NFS4_OO_CONFIRMED; dprintk("NFSD: unsupported OPEN claim type %d\n", open->op_claim_type); status = nfserr_notsupp; @@ -786,7 +785,6 @@ nfsd4_read(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate, if (!nfsd4_last_compound_op(rqstp)) rqstp->rq_splice_ok = false; - nfs4_lock_state(); /* check stateid */ if ((status = nfs4_preprocess_stateid_op(SVC_NET(rqstp), cstate, &read->rd_stateid, @@ -794,11 +792,8 @@ nfsd4_read(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate, dprintk("NFSD: nfsd4_read: couldn't process stateid!\n"); goto out; } - if (read->rd_filp) - get_file(read->rd_filp); status = nfs_ok; out: - nfs4_unlock_state(); read->rd_rqstp = rqstp; read->rd_fhp = &cstate->current_fh; return status; @@ -937,10 +932,8 @@ nfsd4_setattr(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate, int err; if (setattr->sa_iattr.ia_valid & ATTR_SIZE) { - nfs4_lock_state(); status = nfs4_preprocess_stateid_op(SVC_NET(rqstp), cstate, &setattr->sa_stateid, WR_STATE, NULL); - nfs4_unlock_state(); if (status) { dprintk("NFSD: nfsd4_setattr: couldn't process stateid!\n"); return status; @@ -1006,17 +999,12 @@ nfsd4_write(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate, if (write->wr_offset >= OFFSET_MAX) return nfserr_inval; - nfs4_lock_state(); status = nfs4_preprocess_stateid_op(SVC_NET(rqstp), cstate, stateid, WR_STATE, &filp); if (status) { - nfs4_unlock_state(); dprintk("NFSD: nfsd4_write: couldn't process stateid!\n"); return status; } - if (filp) - get_file(filp); - nfs4_unlock_state(); cnt = write->wr_buflen; write->wr_how_written = write->wr_stable_how; @@ -1072,10 +1060,10 @@ _nfsd4_verify(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate, return nfserr_jukebox; p = buf; - status = nfsd4_encode_fattr(&cstate->current_fh, + status = nfsd4_encode_fattr_to_buf(&p, count, &cstate->current_fh, cstate->current_fh.fh_export, - cstate->current_fh.fh_dentry, &p, - count, verify->ve_bmval, + cstate->current_fh.fh_dentry, + verify->ve_bmval, rqstp, 0); /* * If nfsd4_encode_fattr() ran out of space, assume that's because @@ -1182,9 +1170,7 @@ struct nfsd4_operation { static struct nfsd4_operation nfsd4_ops[]; -#ifdef NFSD_DEBUG static const char *nfsd4_op_name(unsigned opnum); -#endif /* * Enforce NFSv4.1 COMPOUND ordering rules: @@ -1226,6 +1212,8 @@ static inline struct nfsd4_operation *OPDESC(struct nfsd4_op *op) bool nfsd4_cache_this_op(struct nfsd4_op *op) { + if (op->opnum == OP_ILLEGAL) + return false; return OPDESC(op)->op_flags & OP_CACHEME; } @@ -1262,6 +1250,25 @@ static bool need_wrongsec_check(struct svc_rqst *rqstp) return !(nextd->op_flags & OP_HANDLES_WRONGSEC); } +static void svcxdr_init_encode(struct svc_rqst *rqstp, + struct nfsd4_compoundres *resp) +{ + struct xdr_stream *xdr = &resp->xdr; + struct xdr_buf *buf = &rqstp->rq_res; + struct kvec *head = buf->head; + + xdr->buf = buf; + xdr->iov = head; + xdr->p = head->iov_base + head->iov_len; + xdr->end = head->iov_base + PAGE_SIZE - rqstp->rq_auth_slack; + /* Tail and page_len should be zero at this point: */ + buf->len = buf->head[0].iov_len; + xdr->scratch.iov_len = 0; + xdr->page_ptr = buf->pages - 1; + buf->buflen = PAGE_SIZE * (1 + rqstp->rq_page_end - buf->pages) + - rqstp->rq_auth_slack; +} + /* * COMPOUND call. */ @@ -1275,24 +1282,16 @@ nfsd4_proc_compound(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate = &resp->cstate; struct svc_fh *current_fh = &cstate->current_fh; struct svc_fh *save_fh = &cstate->save_fh; - int slack_bytes; - u32 plen = 0; __be32 status; - resp->xbuf = &rqstp->rq_res; - resp->p = rqstp->rq_res.head[0].iov_base + - rqstp->rq_res.head[0].iov_len; - resp->tagp = resp->p; + svcxdr_init_encode(rqstp, resp); + resp->tagp = resp->xdr.p; /* reserve space for: taglen, tag, and opcnt */ - resp->p += 2 + XDR_QUADLEN(args->taglen); - resp->end = rqstp->rq_res.head[0].iov_base + PAGE_SIZE; + xdr_reserve_space(&resp->xdr, 8 + args->taglen); resp->taglen = args->taglen; resp->tag = args->tag; - resp->opcnt = 0; resp->rqstp = rqstp; cstate->minorversion = args->minorversion; - cstate->replay_owner = NULL; - cstate->session = NULL; fh_init(current_fh, NFS4_FHSIZE); fh_init(save_fh, NFS4_FHSIZE); /* @@ -1332,19 +1331,6 @@ nfsd4_proc_compound(struct svc_rqst *rqstp, goto encode_op; } - /* We must be able to encode a successful response to - * this operation, with enough room left over to encode a - * failed response to the next operation. If we don't - * have enough room, fail with ERR_RESOURCE. - */ - slack_bytes = (char *)resp->end - (char *)resp->p; - if (slack_bytes < COMPOUND_SLACK_SPACE - + COMPOUND_ERR_SLACK_SPACE) { - BUG_ON(slack_bytes < COMPOUND_ERR_SLACK_SPACE); - op->status = nfserr_resource; - goto encode_op; - } - opdesc = OPDESC(op); if (!current_fh->fh_dentry) { @@ -1362,9 +1348,13 @@ nfsd4_proc_compound(struct svc_rqst *rqstp, /* If op is non-idempotent */ if (opdesc->op_flags & OP_MODIFIES_SOMETHING) { - plen = opdesc->op_rsize_bop(rqstp, op); /* - * If there's still another operation, make sure + * Don't execute this op if we couldn't encode a + * succesful reply: + */ + u32 plen = opdesc->op_rsize_bop(rqstp, op); + /* + * Plus if there's another operation, make sure * we'll have space to at least encode an error: */ if (resp->opcnt < args->opcnt) @@ -1399,7 +1389,7 @@ encode_op: } if (op->status == nfserr_replay_me) { op->replay = &cstate->replay_owner->so_replay; - nfsd4_encode_replay(resp, op); + nfsd4_encode_replay(&resp->xdr, op); status = op->status = op->replay->rp_status; } else { nfsd4_encode_operation(resp, op); @@ -1438,7 +1428,8 @@ out: #define op_encode_change_info_maxsz (5) #define nfs4_fattr_bitmap_maxsz (4) -#define op_encode_lockowner_maxsz (1 + XDR_QUADLEN(IDMAP_NAMESZ)) +/* We'll fall back on returning no lockowner if run out of space: */ +#define op_encode_lockowner_maxsz (0) #define op_encode_lock_denied_maxsz (8 + op_encode_lockowner_maxsz) #define nfs4_owner_maxsz (1 + XDR_QUADLEN(IDMAP_NAMESZ)) @@ -1470,6 +1461,49 @@ static inline u32 nfsd4_create_rsize(struct svc_rqst *rqstp, struct nfsd4_op *op + nfs4_fattr_bitmap_maxsz) * sizeof(__be32); } +/* + * Note since this is an idempotent operation we won't insist on failing + * the op prematurely if the estimate is too large. We may turn off splice + * reads unnecessarily. + */ +static inline u32 nfsd4_getattr_rsize(struct svc_rqst *rqstp, + struct nfsd4_op *op) +{ + u32 *bmap = op->u.getattr.ga_bmval; + u32 bmap0 = bmap[0], bmap1 = bmap[1], bmap2 = bmap[2]; + u32 ret = 0; + + if (bmap0 & FATTR4_WORD0_ACL) + return svc_max_payload(rqstp); + if (bmap0 & FATTR4_WORD0_FS_LOCATIONS) + return svc_max_payload(rqstp); + + if (bmap1 & FATTR4_WORD1_OWNER) { + ret += IDMAP_NAMESZ + 4; + bmap1 &= ~FATTR4_WORD1_OWNER; + } + if (bmap1 & FATTR4_WORD1_OWNER_GROUP) { + ret += IDMAP_NAMESZ + 4; + bmap1 &= ~FATTR4_WORD1_OWNER_GROUP; + } + if (bmap0 & FATTR4_WORD0_FILEHANDLE) { + ret += NFS4_FHSIZE + 4; + bmap0 &= ~FATTR4_WORD0_FILEHANDLE; + } + if (bmap2 & FATTR4_WORD2_SECURITY_LABEL) { + ret += NFSD4_MAX_SEC_LABEL_LEN + 12; + bmap2 &= ~FATTR4_WORD2_SECURITY_LABEL; + } + /* + * Largest of remaining attributes are 16 bytes (e.g., + * supported_attributes) + */ + ret += 16 * (hweight32(bmap0) + hweight32(bmap1) + hweight32(bmap2)); + /* bitmask, length */ + ret += 20; + return ret; +} + static inline u32 nfsd4_link_rsize(struct svc_rqst *rqstp, struct nfsd4_op *op) { return (op_encode_hdr_size + op_encode_change_info_maxsz) @@ -1500,18 +1534,19 @@ static inline u32 nfsd4_read_rsize(struct svc_rqst *rqstp, struct nfsd4_op *op) if (rlen > maxcount) rlen = maxcount; - return (op_encode_hdr_size + 2) * sizeof(__be32) + rlen; + return (op_encode_hdr_size + 2 + XDR_QUADLEN(rlen)) * sizeof(__be32); } static inline u32 nfsd4_readdir_rsize(struct svc_rqst *rqstp, struct nfsd4_op *op) { + u32 maxcount = svc_max_payload(rqstp); u32 rlen = op->u.readdir.rd_maxcount; - if (rlen > PAGE_SIZE) - rlen = PAGE_SIZE; + if (rlen > maxcount) + rlen = maxcount; - return (op_encode_hdr_size + op_encode_verifier_maxsz) - * sizeof(__be32) + rlen; + return (op_encode_hdr_size + op_encode_verifier_maxsz + + XDR_QUADLEN(rlen)) * sizeof(__be32); } static inline u32 nfsd4_remove_rsize(struct svc_rqst *rqstp, struct nfsd4_op *op) @@ -1526,6 +1561,12 @@ static inline u32 nfsd4_rename_rsize(struct svc_rqst *rqstp, struct nfsd4_op *op + op_encode_change_info_maxsz) * sizeof(__be32); } +static inline u32 nfsd4_sequence_rsize(struct svc_rqst *rqstp, + struct nfsd4_op *op) +{ + return NFS4_MAX_SESSIONID_LEN + 20; +} + static inline u32 nfsd4_setattr_rsize(struct svc_rqst *rqstp, struct nfsd4_op *op) { return (op_encode_hdr_size + nfs4_fattr_bitmap_maxsz) * sizeof(__be32); @@ -1539,7 +1580,7 @@ static inline u32 nfsd4_setclientid_rsize(struct svc_rqst *rqstp, struct nfsd4_o static inline u32 nfsd4_write_rsize(struct svc_rqst *rqstp, struct nfsd4_op *op) { - return (op_encode_hdr_size + op_encode_verifier_maxsz) * sizeof(__be32); + return (op_encode_hdr_size + 2 + op_encode_verifier_maxsz) * sizeof(__be32); } static inline u32 nfsd4_exchange_id_rsize(struct svc_rqst *rqstp, struct nfsd4_op *op) @@ -1607,6 +1648,7 @@ static struct nfsd4_operation nfsd4_ops[] = { [OP_GETATTR] = { .op_func = (nfsd4op_func)nfsd4_getattr, .op_flags = ALLOWED_ON_ABSENT_FS, + .op_rsize_bop = nfsd4_getattr_rsize, .op_name = "OP_GETATTR", }, [OP_GETFH] = { @@ -1676,37 +1718,32 @@ static struct nfsd4_operation nfsd4_ops[] = { [OP_PUTFH] = { .op_func = (nfsd4op_func)nfsd4_putfh, .op_flags = ALLOWED_WITHOUT_FH | ALLOWED_ON_ABSENT_FS - | OP_IS_PUTFH_LIKE | OP_MODIFIES_SOMETHING - | OP_CLEAR_STATEID, + | OP_IS_PUTFH_LIKE | OP_CLEAR_STATEID, .op_name = "OP_PUTFH", .op_rsize_bop = (nfsd4op_rsize)nfsd4_only_status_rsize, }, [OP_PUTPUBFH] = { .op_func = (nfsd4op_func)nfsd4_putrootfh, .op_flags = ALLOWED_WITHOUT_FH | ALLOWED_ON_ABSENT_FS - | OP_IS_PUTFH_LIKE | OP_MODIFIES_SOMETHING - | OP_CLEAR_STATEID, + | OP_IS_PUTFH_LIKE | OP_CLEAR_STATEID, .op_name = "OP_PUTPUBFH", .op_rsize_bop = (nfsd4op_rsize)nfsd4_only_status_rsize, }, [OP_PUTROOTFH] = { .op_func = (nfsd4op_func)nfsd4_putrootfh, .op_flags = ALLOWED_WITHOUT_FH | ALLOWED_ON_ABSENT_FS - | OP_IS_PUTFH_LIKE | OP_MODIFIES_SOMETHING - | OP_CLEAR_STATEID, + | OP_IS_PUTFH_LIKE | OP_CLEAR_STATEID, .op_name = "OP_PUTROOTFH", .op_rsize_bop = (nfsd4op_rsize)nfsd4_only_status_rsize, }, [OP_READ] = { .op_func = (nfsd4op_func)nfsd4_read, - .op_flags = OP_MODIFIES_SOMETHING, .op_name = "OP_READ", .op_rsize_bop = (nfsd4op_rsize)nfsd4_read_rsize, .op_get_currentstateid = (stateid_getter)nfsd4_get_readstateid, }, [OP_READDIR] = { .op_func = (nfsd4op_func)nfsd4_readdir, - .op_flags = OP_MODIFIES_SOMETHING, .op_name = "OP_READDIR", .op_rsize_bop = (nfsd4op_rsize)nfsd4_readdir_rsize, }, @@ -1864,14 +1901,33 @@ static struct nfsd4_operation nfsd4_ops[] = { }, }; -#ifdef NFSD_DEBUG +int nfsd4_max_reply(struct svc_rqst *rqstp, struct nfsd4_op *op) +{ + struct nfsd4_operation *opdesc; + nfsd4op_rsize estimator; + + if (op->opnum == OP_ILLEGAL) + return op_encode_hdr_size * sizeof(__be32); + opdesc = OPDESC(op); + estimator = opdesc->op_rsize_bop; + return estimator ? estimator(rqstp, op) : PAGE_SIZE; +} + +void warn_on_nonidempotent_op(struct nfsd4_op *op) +{ + if (OPDESC(op)->op_flags & OP_MODIFIES_SOMETHING) { + pr_err("unable to encode reply to nonidempotent op %d (%s)\n", + op->opnum, nfsd4_op_name(op->opnum)); + WARN_ON_ONCE(1); + } +} + static const char *nfsd4_op_name(unsigned opnum) { if (opnum < ARRAY_SIZE(nfsd4_ops)) return nfsd4_ops[opnum].op_name; return "unknown_operation"; } -#endif #define nfsd4_voidres nfsd4_voidargs struct nfsd4_voidargs { int dummy; }; diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c index 9a77a5a21557..c0d45cec9958 100644 --- a/fs/nfsd/nfs4state.c +++ b/fs/nfsd/nfs4state.c @@ -81,13 +81,13 @@ static DEFINE_MUTEX(client_mutex); * effort to decrease the scope of the client_mutex, this spinlock may * eventually cover more: */ -static DEFINE_SPINLOCK(recall_lock); +static DEFINE_SPINLOCK(state_lock); -static struct kmem_cache *openowner_slab = NULL; -static struct kmem_cache *lockowner_slab = NULL; -static struct kmem_cache *file_slab = NULL; -static struct kmem_cache *stateid_slab = NULL; -static struct kmem_cache *deleg_slab = NULL; +static struct kmem_cache *openowner_slab; +static struct kmem_cache *lockowner_slab; +static struct kmem_cache *file_slab; +static struct kmem_cache *stateid_slab; +static struct kmem_cache *deleg_slab; void nfs4_lock_state(void) @@ -235,9 +235,9 @@ static void nfsd4_free_file(struct nfs4_file *f) static inline void put_nfs4_file(struct nfs4_file *fi) { - if (atomic_dec_and_lock(&fi->fi_ref, &recall_lock)) { + if (atomic_dec_and_lock(&fi->fi_ref, &state_lock)) { hlist_del(&fi->fi_hash); - spin_unlock(&recall_lock); + spin_unlock(&state_lock); iput(fi->fi_inode); nfsd4_free_file(fi); } @@ -375,7 +375,6 @@ alloc_init_deleg(struct nfs4_client *clp, struct nfs4_ol_stateid *stp, struct sv dp = delegstateid(nfs4_alloc_stid(clp, deleg_slab)); if (dp == NULL) return dp; - dp->dl_stid.sc_type = NFS4_DELEG_STID; /* * delegation seqid's are never incremented. The 4.1 special * meaning of seqid 0 isn't meaningful, really, but let's avoid @@ -418,6 +417,8 @@ nfs4_put_delegation(struct nfs4_delegation *dp) static void nfs4_put_deleg_lease(struct nfs4_file *fp) { + if (!fp->fi_lease) + return; if (atomic_dec_and_test(&fp->fi_delegees)) { vfs_setlease(fp->fi_deleg_file, F_UNLCK, &fp->fi_lease); fp->fi_lease = NULL; @@ -431,18 +432,30 @@ static void unhash_stid(struct nfs4_stid *s) s->sc_type = 0; } +static void +hash_delegation_locked(struct nfs4_delegation *dp, struct nfs4_file *fp) +{ + lockdep_assert_held(&state_lock); + + dp->dl_stid.sc_type = NFS4_DELEG_STID; + list_add(&dp->dl_perfile, &fp->fi_delegations); + list_add(&dp->dl_perclnt, &dp->dl_stid.sc_client->cl_delegations); +} + /* Called under the state lock. */ static void unhash_delegation(struct nfs4_delegation *dp) { + spin_lock(&state_lock); list_del_init(&dp->dl_perclnt); - spin_lock(&recall_lock); list_del_init(&dp->dl_perfile); list_del_init(&dp->dl_recall_lru); - spin_unlock(&recall_lock); - nfs4_put_deleg_lease(dp->dl_file); - put_nfs4_file(dp->dl_file); - dp->dl_file = NULL; + spin_unlock(&state_lock); + if (dp->dl_file) { + nfs4_put_deleg_lease(dp->dl_file); + put_nfs4_file(dp->dl_file); + dp->dl_file = NULL; + } } @@ -645,6 +658,12 @@ static void unhash_lockowner(struct nfs4_lockowner *lo) } } +static void nfs4_free_lockowner(struct nfs4_lockowner *lo) +{ + kfree(lo->lo_owner.so_owner.data); + kmem_cache_free(lockowner_slab, lo); +} + static void release_lockowner(struct nfs4_lockowner *lo) { unhash_lockowner(lo); @@ -699,6 +718,12 @@ static void release_last_closed_stateid(struct nfs4_openowner *oo) } } +static void nfs4_free_openowner(struct nfs4_openowner *oo) +{ + kfree(oo->oo_owner.so_owner.data); + kmem_cache_free(openowner_slab, oo); +} + static void release_openowner(struct nfs4_openowner *oo) { unhash_openowner(oo); @@ -1093,7 +1118,7 @@ static struct nfs4_client *alloc_client(struct xdr_netobj name) return clp; } -static inline void +static void free_client(struct nfs4_client *clp) { struct nfsd_net __maybe_unused *nn = net_generic(clp->net, nfsd_net_id); @@ -1136,13 +1161,13 @@ destroy_client(struct nfs4_client *clp) struct nfsd_net *nn = net_generic(clp->net, nfsd_net_id); INIT_LIST_HEAD(&reaplist); - spin_lock(&recall_lock); + spin_lock(&state_lock); while (!list_empty(&clp->cl_delegations)) { dp = list_entry(clp->cl_delegations.next, struct nfs4_delegation, dl_perclnt); list_del_init(&dp->dl_perclnt); list_move(&dp->dl_recall_lru, &reaplist); } - spin_unlock(&recall_lock); + spin_unlock(&state_lock); while (!list_empty(&reaplist)) { dp = list_entry(reaplist.next, struct nfs4_delegation, dl_recall_lru); destroy_delegation(dp); @@ -1544,6 +1569,7 @@ out_err: void nfsd4_store_cache_entry(struct nfsd4_compoundres *resp) { + struct xdr_buf *buf = resp->xdr.buf; struct nfsd4_slot *slot = resp->cstate.slot; unsigned int base; @@ -1557,11 +1583,9 @@ nfsd4_store_cache_entry(struct nfsd4_compoundres *resp) slot->sl_datalen = 0; return; } - slot->sl_datalen = (char *)resp->p - (char *)resp->cstate.datap; - base = (char *)resp->cstate.datap - - (char *)resp->xbuf->head[0].iov_base; - if (read_bytes_from_xdr_buf(resp->xbuf, base, slot->sl_data, - slot->sl_datalen)) + base = resp->cstate.data_offset; + slot->sl_datalen = buf->len - base; + if (read_bytes_from_xdr_buf(buf, base, slot->sl_data, slot->sl_datalen)) WARN("%s: sessions DRC could not cache compound\n", __func__); return; } @@ -1602,6 +1626,8 @@ nfsd4_replay_cache_entry(struct nfsd4_compoundres *resp, struct nfsd4_sequence *seq) { struct nfsd4_slot *slot = resp->cstate.slot; + struct xdr_stream *xdr = &resp->xdr; + __be32 *p; __be32 status; dprintk("--> %s slot %p\n", __func__, slot); @@ -1610,14 +1636,16 @@ nfsd4_replay_cache_entry(struct nfsd4_compoundres *resp, if (status) return status; - /* The sequence operation has been encoded, cstate->datap set. */ - memcpy(resp->cstate.datap, slot->sl_data, slot->sl_datalen); + p = xdr_reserve_space(xdr, slot->sl_datalen); + if (!p) { + WARN_ON_ONCE(1); + return nfserr_serverfault; + } + xdr_encode_opaque_fixed(p, slot->sl_data, slot->sl_datalen); + xdr_commit_encode(xdr); resp->opcnt = slot->sl_opcnt; - resp->p = resp->cstate.datap + XDR_QUADLEN(slot->sl_datalen); - status = slot->sl_status; - - return status; + return slot->sl_status; } /* @@ -2189,11 +2217,13 @@ nfsd4_sequence(struct svc_rqst *rqstp, struct nfsd4_sequence *seq) { struct nfsd4_compoundres *resp = rqstp->rq_resp; + struct xdr_stream *xdr = &resp->xdr; struct nfsd4_session *session; struct nfs4_client *clp; struct nfsd4_slot *slot; struct nfsd4_conn *conn; __be32 status; + int buflen; struct nfsd_net *nn = net_generic(SVC_NET(rqstp), nfsd_net_id); if (resp->opcnt != 1) @@ -2262,6 +2292,16 @@ nfsd4_sequence(struct svc_rqst *rqstp, if (status) goto out_put_session; + buflen = (seq->cachethis) ? + session->se_fchannel.maxresp_cached : + session->se_fchannel.maxresp_sz; + status = (seq->cachethis) ? nfserr_rep_too_big_to_cache : + nfserr_rep_too_big; + if (xdr_restrict_buflen(xdr, buflen - rqstp->rq_auth_slack)) + goto out_put_session; + svc_reserve(rqstp, buflen); + + status = nfs_ok; /* Success! bump slot seqid */ slot->sl_seqid = seq->seqid; slot->sl_flags |= NFSD4_SLOT_INUSE; @@ -2499,28 +2539,19 @@ static void nfsd4_init_file(struct nfs4_file *fp, struct inode *ino) fp->fi_lease = NULL; memset(fp->fi_fds, 0, sizeof(fp->fi_fds)); memset(fp->fi_access, 0, sizeof(fp->fi_access)); - spin_lock(&recall_lock); + spin_lock(&state_lock); hlist_add_head(&fp->fi_hash, &file_hashtbl[hashval]); - spin_unlock(&recall_lock); -} - -static void -nfsd4_free_slab(struct kmem_cache **slab) -{ - if (*slab == NULL) - return; - kmem_cache_destroy(*slab); - *slab = NULL; + spin_unlock(&state_lock); } void nfsd4_free_slabs(void) { - nfsd4_free_slab(&openowner_slab); - nfsd4_free_slab(&lockowner_slab); - nfsd4_free_slab(&file_slab); - nfsd4_free_slab(&stateid_slab); - nfsd4_free_slab(&deleg_slab); + kmem_cache_destroy(openowner_slab); + kmem_cache_destroy(lockowner_slab); + kmem_cache_destroy(file_slab); + kmem_cache_destroy(stateid_slab); + kmem_cache_destroy(deleg_slab); } int @@ -2529,42 +2560,38 @@ nfsd4_init_slabs(void) openowner_slab = kmem_cache_create("nfsd4_openowners", sizeof(struct nfs4_openowner), 0, 0, NULL); if (openowner_slab == NULL) - goto out_nomem; + goto out; lockowner_slab = kmem_cache_create("nfsd4_lockowners", sizeof(struct nfs4_lockowner), 0, 0, NULL); if (lockowner_slab == NULL) - goto out_nomem; + goto out_free_openowner_slab; file_slab = kmem_cache_create("nfsd4_files", sizeof(struct nfs4_file), 0, 0, NULL); if (file_slab == NULL) - goto out_nomem; + goto out_free_lockowner_slab; stateid_slab = kmem_cache_create("nfsd4_stateids", sizeof(struct nfs4_ol_stateid), 0, 0, NULL); if (stateid_slab == NULL) - goto out_nomem; + goto out_free_file_slab; deleg_slab = kmem_cache_create("nfsd4_delegations", sizeof(struct nfs4_delegation), 0, 0, NULL); if (deleg_slab == NULL) - goto out_nomem; + goto out_free_stateid_slab; return 0; -out_nomem: - nfsd4_free_slabs(); + +out_free_stateid_slab: + kmem_cache_destroy(stateid_slab); +out_free_file_slab: + kmem_cache_destroy(file_slab); +out_free_lockowner_slab: + kmem_cache_destroy(lockowner_slab); +out_free_openowner_slab: + kmem_cache_destroy(openowner_slab); +out: dprintk("nfsd4: out of memory while initializing nfsv4\n"); return -ENOMEM; } -void nfs4_free_openowner(struct nfs4_openowner *oo) -{ - kfree(oo->oo_owner.so_owner.data); - kmem_cache_free(openowner_slab, oo); -} - -void nfs4_free_lockowner(struct nfs4_lockowner *lo) -{ - kfree(lo->lo_owner.so_owner.data); - kmem_cache_free(lockowner_slab, lo); -} - static void init_nfs4_replay(struct nfs4_replay *rp) { rp->rp_status = nfserr_serverfault; @@ -2685,15 +2712,15 @@ find_file(struct inode *ino) unsigned int hashval = file_hashval(ino); struct nfs4_file *fp; - spin_lock(&recall_lock); + spin_lock(&state_lock); hlist_for_each_entry(fp, &file_hashtbl[hashval], fi_hash) { if (fp->fi_inode == ino) { get_nfs4_file(fp); - spin_unlock(&recall_lock); + spin_unlock(&state_lock); return fp; } } - spin_unlock(&recall_lock); + spin_unlock(&state_lock); return NULL; } @@ -2730,6 +2757,7 @@ static void nfsd_break_one_deleg(struct nfs4_delegation *dp) struct nfs4_client *clp = dp->dl_stid.sc_client; struct nfsd_net *nn = net_generic(clp->net, nfsd_net_id); + lockdep_assert_held(&state_lock); /* We're assuming the state code never drops its reference * without first removing the lease. Since we're in this lease * callback (and since the lease code is serialized by the kernel @@ -2766,11 +2794,11 @@ static void nfsd_break_deleg_cb(struct file_lock *fl) */ fl->fl_break_time = 0; - spin_lock(&recall_lock); + spin_lock(&state_lock); fp->fi_had_conflict = true; list_for_each_entry(dp, &fp->fi_delegations, dl_perfile) nfsd_break_one_deleg(dp); - spin_unlock(&recall_lock); + spin_unlock(&state_lock); } static @@ -3047,11 +3075,12 @@ static int nfs4_setlease(struct nfs4_delegation *dp) status = vfs_setlease(fl->fl_file, fl->fl_type, &fl); if (status) goto out_free; - list_add(&dp->dl_perclnt, &dp->dl_stid.sc_client->cl_delegations); fp->fi_lease = fl; fp->fi_deleg_file = get_file(fl->fl_file); atomic_set(&fp->fi_delegees, 1); - list_add(&dp->dl_perfile, &fp->fi_delegations); + spin_lock(&state_lock); + hash_delegation_locked(dp, fp); + spin_unlock(&state_lock); return 0; out_free: locks_free_lock(fl); @@ -3060,33 +3089,21 @@ out_free: static int nfs4_set_delegation(struct nfs4_delegation *dp, struct nfs4_file *fp) { - int status; - if (fp->fi_had_conflict) return -EAGAIN; get_nfs4_file(fp); dp->dl_file = fp; - if (!fp->fi_lease) { - status = nfs4_setlease(dp); - if (status) - goto out_free; - return 0; - } - spin_lock(&recall_lock); + if (!fp->fi_lease) + return nfs4_setlease(dp); + spin_lock(&state_lock); + atomic_inc(&fp->fi_delegees); if (fp->fi_had_conflict) { - spin_unlock(&recall_lock); - status = -EAGAIN; - goto out_free; + spin_unlock(&state_lock); + return -EAGAIN; } - atomic_inc(&fp->fi_delegees); - list_add(&dp->dl_perfile, &fp->fi_delegations); - spin_unlock(&recall_lock); - list_add(&dp->dl_perclnt, &dp->dl_stid.sc_client->cl_delegations); + hash_delegation_locked(dp, fp); + spin_unlock(&state_lock); return 0; -out_free: - put_nfs4_file(fp); - dp->dl_file = fp; - return status; } static void nfsd4_open_deleg_none_ext(struct nfsd4_open *open, int status) @@ -3173,8 +3190,7 @@ nfs4_open_delegation(struct net *net, struct svc_fh *fh, open->op_delegate_type = NFS4_OPEN_DELEGATE_READ; return; out_free: - remove_stid(&dp->dl_stid); - nfs4_put_delegation(dp); + destroy_delegation(dp); out_no_deleg: open->op_delegate_type = NFS4_OPEN_DELEGATE_NONE; if (open->op_claim_type == NFS4_OPEN_CLAIM_PREVIOUS && @@ -3391,8 +3407,7 @@ nfs4_laundromat(struct nfsd_net *nn) struct nfs4_delegation *dp; struct list_head *pos, *next, reaplist; time_t cutoff = get_seconds() - nn->nfsd4_lease; - time_t t, clientid_val = nn->nfsd4_lease; - time_t u, test_val = nn->nfsd4_lease; + time_t t, new_timeo = nn->nfsd4_lease; nfs4_lock_state(); @@ -3404,8 +3419,7 @@ nfs4_laundromat(struct nfsd_net *nn) clp = list_entry(pos, struct nfs4_client, cl_lru); if (time_after((unsigned long)clp->cl_time, (unsigned long)cutoff)) { t = clp->cl_time - cutoff; - if (clientid_val > t) - clientid_val = t; + new_timeo = min(new_timeo, t); break; } if (mark_client_expired_locked(clp)) { @@ -3422,39 +3436,35 @@ nfs4_laundromat(struct nfsd_net *nn) clp->cl_clientid.cl_id); expire_client(clp); } - spin_lock(&recall_lock); + spin_lock(&state_lock); list_for_each_safe(pos, next, &nn->del_recall_lru) { dp = list_entry (pos, struct nfs4_delegation, dl_recall_lru); if (net_generic(dp->dl_stid.sc_client->net, nfsd_net_id) != nn) continue; if (time_after((unsigned long)dp->dl_time, (unsigned long)cutoff)) { - u = dp->dl_time - cutoff; - if (test_val > u) - test_val = u; + t = dp->dl_time - cutoff; + new_timeo = min(new_timeo, t); break; } list_move(&dp->dl_recall_lru, &reaplist); } - spin_unlock(&recall_lock); + spin_unlock(&state_lock); list_for_each_safe(pos, next, &reaplist) { dp = list_entry (pos, struct nfs4_delegation, dl_recall_lru); revoke_delegation(dp); } - test_val = nn->nfsd4_lease; list_for_each_safe(pos, next, &nn->close_lru) { oo = container_of(pos, struct nfs4_openowner, oo_close_lru); if (time_after((unsigned long)oo->oo_time, (unsigned long)cutoff)) { - u = oo->oo_time - cutoff; - if (test_val > u) - test_val = u; + t = oo->oo_time - cutoff; + new_timeo = min(new_timeo, t); break; } release_openowner(oo); } - if (clientid_val < NFSD_LAUNDROMAT_MINTIMEOUT) - clientid_val = NFSD_LAUNDROMAT_MINTIMEOUT; + new_timeo = max_t(time_t, new_timeo, NFSD_LAUNDROMAT_MINTIMEOUT); nfs4_unlock_state(); - return clientid_val; + return new_timeo; } static struct workqueue_struct *laundry_wq; @@ -3654,6 +3664,7 @@ nfs4_preprocess_stateid_op(struct net *net, struct nfsd4_compound_state *cstate, struct svc_fh *current_fh = &cstate->current_fh; struct inode *ino = current_fh->fh_dentry->d_inode; struct nfsd_net *nn = net_generic(net, nfsd_net_id); + struct file *file = NULL; __be32 status; if (filpp) @@ -3665,10 +3676,12 @@ nfs4_preprocess_stateid_op(struct net *net, struct nfsd4_compound_state *cstate, if (ZERO_STATEID(stateid) || ONE_STATEID(stateid)) return check_special_stateids(net, current_fh, stateid, flags); + nfs4_lock_state(); + status = nfsd4_lookup_stateid(stateid, NFS4_DELEG_STID|NFS4_OPEN_STID|NFS4_LOCK_STID, &s, cstate->minorversion, nn); if (status) - return status; + goto out; status = check_stateid_generation(stateid, &s->sc_stateid, nfsd4_has_session(cstate)); if (status) goto out; @@ -3679,8 +3692,8 @@ nfs4_preprocess_stateid_op(struct net *net, struct nfsd4_compound_state *cstate, if (status) goto out; if (filpp) { - *filpp = dp->dl_file->fi_deleg_file; - if (!*filpp) { + file = dp->dl_file->fi_deleg_file; + if (!file) { WARN_ON_ONCE(1); status = nfserr_serverfault; goto out; @@ -3701,16 +3714,20 @@ nfs4_preprocess_stateid_op(struct net *net, struct nfsd4_compound_state *cstate, goto out; if (filpp) { if (flags & RD_STATE) - *filpp = find_readable_file(stp->st_file); + file = find_readable_file(stp->st_file); else - *filpp = find_writeable_file(stp->st_file); + file = find_writeable_file(stp->st_file); } break; default: - return nfserr_bad_stateid; + status = nfserr_bad_stateid; + goto out; } status = nfs_ok; + if (file) + *filpp = get_file(file); out: + nfs4_unlock_state(); return status; } @@ -3726,7 +3743,7 @@ nfsd4_free_lock_stateid(struct nfs4_ol_stateid *stp) * correspondance, and we have to delete the lockowner when we * delete the lock stateid: */ - unhash_lockowner(lo); + release_lockowner(lo); return nfs_ok; } @@ -4896,6 +4913,7 @@ static u64 nfsd_find_all_delegations(struct nfs4_client *clp, u64 max, struct nfs4_delegation *dp, *next; u64 count = 0; + lockdep_assert_held(&state_lock); list_for_each_entry_safe(dp, next, &clp->cl_delegations, dl_perclnt) { if (victims) list_move(&dp->dl_recall_lru, victims); @@ -4911,9 +4929,9 @@ u64 nfsd_forget_client_delegations(struct nfs4_client *clp, u64 max) LIST_HEAD(victims); u64 count; - spin_lock(&recall_lock); + spin_lock(&state_lock); count = nfsd_find_all_delegations(clp, max, &victims); - spin_unlock(&recall_lock); + spin_unlock(&state_lock); list_for_each_entry_safe(dp, next, &victims, dl_recall_lru) revoke_delegation(dp); @@ -4927,11 +4945,11 @@ u64 nfsd_recall_client_delegations(struct nfs4_client *clp, u64 max) LIST_HEAD(victims); u64 count; - spin_lock(&recall_lock); + spin_lock(&state_lock); count = nfsd_find_all_delegations(clp, max, &victims); list_for_each_entry_safe(dp, next, &victims, dl_recall_lru) nfsd_break_one_deleg(dp); - spin_unlock(&recall_lock); + spin_unlock(&state_lock); return count; } @@ -4940,9 +4958,9 @@ u64 nfsd_print_client_delegations(struct nfs4_client *clp, u64 max) { u64 count = 0; - spin_lock(&recall_lock); + spin_lock(&state_lock); count = nfsd_find_all_delegations(clp, max, NULL); - spin_unlock(&recall_lock); + spin_unlock(&state_lock); nfsd_print_count(clp, count, "delegations"); return count; @@ -4983,13 +5001,6 @@ struct nfs4_client *nfsd_find_client(struct sockaddr_storage *addr, size_t addr_ #endif /* CONFIG_NFSD_FAULT_INJECTION */ -/* initialization to perform at module load time: */ - -void -nfs4_state_init(void) -{ -} - /* * Since the lifetime of a delegation isn't limited to that of an open, a * client may quite reasonably hang on to a delegation as long as it has @@ -5160,12 +5171,12 @@ nfs4_state_shutdown_net(struct net *net) nfs4_lock_state(); INIT_LIST_HEAD(&reaplist); - spin_lock(&recall_lock); + spin_lock(&state_lock); list_for_each_safe(pos, next, &nn->del_recall_lru) { dp = list_entry (pos, struct nfs4_delegation, dl_recall_lru); list_move(&dp->dl_recall_lru, &reaplist); } - spin_unlock(&recall_lock); + spin_unlock(&state_lock); list_for_each_safe(pos, next, &reaplist) { dp = list_entry (pos, struct nfs4_delegation, dl_recall_lru); destroy_delegation(dp); diff --git a/fs/nfsd/nfs4xdr.c b/fs/nfsd/nfs4xdr.c index 18881f34737a..2d305a121f37 100644 --- a/fs/nfsd/nfs4xdr.c +++ b/fs/nfsd/nfs4xdr.c @@ -98,11 +98,6 @@ xdr_error: \ status = nfserr_bad_xdr; \ goto out -#define READ32(x) (x) = ntohl(*p++) -#define READ64(x) do { \ - (x) = (u64)ntohl(*p++) << 32; \ - (x) |= ntohl(*p++); \ -} while (0) #define READMEM(x,nbytes) do { \ x = (char *)p; \ p += XDR_QUADLEN(nbytes); \ @@ -248,17 +243,17 @@ nfsd4_decode_bitmap(struct nfsd4_compoundargs *argp, u32 *bmval) bmval[2] = 0; READ_BUF(4); - READ32(bmlen); + bmlen = be32_to_cpup(p++); if (bmlen > 1000) goto xdr_error; READ_BUF(bmlen << 2); if (bmlen > 0) - READ32(bmval[0]); + bmval[0] = be32_to_cpup(p++); if (bmlen > 1) - READ32(bmval[1]); + bmval[1] = be32_to_cpup(p++); if (bmlen > 2) - READ32(bmval[2]); + bmval[2] = be32_to_cpup(p++); DECODE_TAIL; } @@ -270,6 +265,7 @@ nfsd4_decode_fattr(struct nfsd4_compoundargs *argp, u32 *bmval, { int expected_len, len = 0; u32 dummy32; + u64 sec; char *buf; DECODE_HEAD; @@ -278,12 +274,12 @@ nfsd4_decode_fattr(struct nfsd4_compoundargs *argp, u32 *bmval, return status; READ_BUF(4); - READ32(expected_len); + expected_len = be32_to_cpup(p++); if (bmval[0] & FATTR4_WORD0_SIZE) { READ_BUF(8); len += 8; - READ64(iattr->ia_size); + p = xdr_decode_hyper(p, &iattr->ia_size); iattr->ia_valid |= ATTR_SIZE; } if (bmval[0] & FATTR4_WORD0_ACL) { @@ -291,7 +287,7 @@ nfsd4_decode_fattr(struct nfsd4_compoundargs *argp, u32 *bmval, struct nfs4_ace *ace; READ_BUF(4); len += 4; - READ32(nace); + nace = be32_to_cpup(p++); if (nace > NFS4_ACL_MAX) return nfserr_fbig; @@ -305,10 +301,10 @@ nfsd4_decode_fattr(struct nfsd4_compoundargs *argp, u32 *bmval, (*acl)->naces = nace; for (ace = (*acl)->aces; ace < (*acl)->aces + nace; ace++) { READ_BUF(16); len += 16; - READ32(ace->type); - READ32(ace->flag); - READ32(ace->access_mask); - READ32(dummy32); + ace->type = be32_to_cpup(p++); + ace->flag = be32_to_cpup(p++); + ace->access_mask = be32_to_cpup(p++); + dummy32 = be32_to_cpup(p++); READ_BUF(dummy32); len += XDR_QUADLEN(dummy32) << 2; READMEM(buf, dummy32); @@ -330,14 +326,14 @@ nfsd4_decode_fattr(struct nfsd4_compoundargs *argp, u32 *bmval, if (bmval[1] & FATTR4_WORD1_MODE) { READ_BUF(4); len += 4; - READ32(iattr->ia_mode); + iattr->ia_mode = be32_to_cpup(p++); iattr->ia_mode &= (S_IFMT | S_IALLUGO); iattr->ia_valid |= ATTR_MODE; } if (bmval[1] & FATTR4_WORD1_OWNER) { READ_BUF(4); len += 4; - READ32(dummy32); + dummy32 = be32_to_cpup(p++); READ_BUF(dummy32); len += (XDR_QUADLEN(dummy32) << 2); READMEM(buf, dummy32); @@ -348,7 +344,7 @@ nfsd4_decode_fattr(struct nfsd4_compoundargs *argp, u32 *bmval, if (bmval[1] & FATTR4_WORD1_OWNER_GROUP) { READ_BUF(4); len += 4; - READ32(dummy32); + dummy32 = be32_to_cpup(p++); READ_BUF(dummy32); len += (XDR_QUADLEN(dummy32) << 2); READMEM(buf, dummy32); @@ -359,15 +355,16 @@ nfsd4_decode_fattr(struct nfsd4_compoundargs *argp, u32 *bmval, if (bmval[1] & FATTR4_WORD1_TIME_ACCESS_SET) { READ_BUF(4); len += 4; - READ32(dummy32); + dummy32 = be32_to_cpup(p++); switch (dummy32) { case NFS4_SET_TO_CLIENT_TIME: /* We require the high 32 bits of 'seconds' to be 0, and we ignore all 32 bits of 'nseconds'. */ READ_BUF(12); len += 12; - READ64(iattr->ia_atime.tv_sec); - READ32(iattr->ia_atime.tv_nsec); + p = xdr_decode_hyper(p, &sec); + iattr->ia_atime.tv_sec = (time_t)sec; + iattr->ia_atime.tv_nsec = be32_to_cpup(p++); if (iattr->ia_atime.tv_nsec >= (u32)1000000000) return nfserr_inval; iattr->ia_valid |= (ATTR_ATIME | ATTR_ATIME_SET); @@ -382,15 +379,16 @@ nfsd4_decode_fattr(struct nfsd4_compoundargs *argp, u32 *bmval, if (bmval[1] & FATTR4_WORD1_TIME_MODIFY_SET) { READ_BUF(4); len += 4; - READ32(dummy32); + dummy32 = be32_to_cpup(p++); switch (dummy32) { case NFS4_SET_TO_CLIENT_TIME: /* We require the high 32 bits of 'seconds' to be 0, and we ignore all 32 bits of 'nseconds'. */ READ_BUF(12); len += 12; - READ64(iattr->ia_mtime.tv_sec); - READ32(iattr->ia_mtime.tv_nsec); + p = xdr_decode_hyper(p, &sec); + iattr->ia_mtime.tv_sec = sec; + iattr->ia_mtime.tv_nsec = be32_to_cpup(p++); if (iattr->ia_mtime.tv_nsec >= (u32)1000000000) return nfserr_inval; iattr->ia_valid |= (ATTR_MTIME | ATTR_MTIME_SET); @@ -408,13 +406,13 @@ nfsd4_decode_fattr(struct nfsd4_compoundargs *argp, u32 *bmval, if (bmval[2] & FATTR4_WORD2_SECURITY_LABEL) { READ_BUF(4); len += 4; - READ32(dummy32); /* lfs: we don't use it */ + dummy32 = be32_to_cpup(p++); /* lfs: we don't use it */ READ_BUF(4); len += 4; - READ32(dummy32); /* pi: we don't use it either */ + dummy32 = be32_to_cpup(p++); /* pi: we don't use it either */ READ_BUF(4); len += 4; - READ32(dummy32); + dummy32 = be32_to_cpup(p++); READ_BUF(dummy32); if (dummy32 > NFSD4_MAX_SEC_LABEL_LEN) return nfserr_badlabel; @@ -445,7 +443,7 @@ nfsd4_decode_stateid(struct nfsd4_compoundargs *argp, stateid_t *sid) DECODE_HEAD; READ_BUF(sizeof(stateid_t)); - READ32(sid->si_generation); + sid->si_generation = be32_to_cpup(p++); COPYMEM(&sid->si_opaque, sizeof(stateid_opaque_t)); DECODE_TAIL; @@ -457,7 +455,7 @@ nfsd4_decode_access(struct nfsd4_compoundargs *argp, struct nfsd4_access *access DECODE_HEAD; READ_BUF(4); - READ32(access->ac_req_access); + access->ac_req_access = be32_to_cpup(p++); DECODE_TAIL; } @@ -472,7 +470,7 @@ static __be32 nfsd4_decode_cb_sec(struct nfsd4_compoundargs *argp, struct nfsd4_ /* callback_sec_params4 */ READ_BUF(4); - READ32(nr_secflavs); + nr_secflavs = be32_to_cpup(p++); if (nr_secflavs) cbs->flavor = (u32)(-1); else @@ -480,7 +478,7 @@ static __be32 nfsd4_decode_cb_sec(struct nfsd4_compoundargs *argp, struct nfsd4_ cbs->flavor = 0; for (i = 0; i < nr_secflavs; ++i) { READ_BUF(4); - READ32(dummy); + dummy = be32_to_cpup(p++); switch (dummy) { case RPC_AUTH_NULL: /* Nothing to read */ @@ -490,21 +488,21 @@ static __be32 nfsd4_decode_cb_sec(struct nfsd4_compoundargs *argp, struct nfsd4_ case RPC_AUTH_UNIX: READ_BUF(8); /* stamp */ - READ32(dummy); + dummy = be32_to_cpup(p++); /* machine name */ - READ32(dummy); + dummy = be32_to_cpup(p++); READ_BUF(dummy); SAVEMEM(machine_name, dummy); /* uid, gid */ READ_BUF(8); - READ32(uid); - READ32(gid); + uid = be32_to_cpup(p++); + gid = be32_to_cpup(p++); /* more gids */ READ_BUF(4); - READ32(dummy); + dummy = be32_to_cpup(p++); READ_BUF(dummy * 4); if (cbs->flavor == (u32)(-1)) { kuid_t kuid = make_kuid(&init_user_ns, uid); @@ -524,14 +522,14 @@ static __be32 nfsd4_decode_cb_sec(struct nfsd4_compoundargs *argp, struct nfsd4_ "not supported!\n"); READ_BUF(8); /* gcbp_service */ - READ32(dummy); + dummy = be32_to_cpup(p++); /* gcbp_handle_from_server */ - READ32(dummy); + dummy = be32_to_cpup(p++); READ_BUF(dummy); p += XDR_QUADLEN(dummy); /* gcbp_handle_from_client */ READ_BUF(4); - READ32(dummy); + dummy = be32_to_cpup(p++); READ_BUF(dummy); break; default: @@ -547,7 +545,7 @@ static __be32 nfsd4_decode_backchannel_ctl(struct nfsd4_compoundargs *argp, stru DECODE_HEAD; READ_BUF(4); - READ32(bc->bc_cb_program); + bc->bc_cb_program = be32_to_cpup(p++); nfsd4_decode_cb_sec(argp, &bc->bc_cb_sec); DECODE_TAIL; @@ -559,7 +557,7 @@ static __be32 nfsd4_decode_bind_conn_to_session(struct nfsd4_compoundargs *argp, READ_BUF(NFS4_MAX_SESSIONID_LEN + 8); COPYMEM(bcts->sessionid.data, NFS4_MAX_SESSIONID_LEN); - READ32(bcts->dir); + bcts->dir = be32_to_cpup(p++); /* XXX: skipping ctsa_use_conn_in_rdma_mode. Perhaps Tom Tucker * could help us figure out we should be using it. */ DECODE_TAIL; @@ -571,7 +569,7 @@ nfsd4_decode_close(struct nfsd4_compoundargs *argp, struct nfsd4_close *close) DECODE_HEAD; READ_BUF(4); - READ32(close->cl_seqid); + close->cl_seqid = be32_to_cpup(p++); return nfsd4_decode_stateid(argp, &close->cl_stateid); DECODE_TAIL; @@ -584,8 +582,8 @@ nfsd4_decode_commit(struct nfsd4_compoundargs *argp, struct nfsd4_commit *commit DECODE_HEAD; READ_BUF(12); - READ64(commit->co_offset); - READ32(commit->co_count); + p = xdr_decode_hyper(p, &commit->co_offset); + commit->co_count = be32_to_cpup(p++); DECODE_TAIL; } @@ -596,19 +594,19 @@ nfsd4_decode_create(struct nfsd4_compoundargs *argp, struct nfsd4_create *create DECODE_HEAD; READ_BUF(4); - READ32(create->cr_type); + create->cr_type = be32_to_cpup(p++); switch (create->cr_type) { case NF4LNK: READ_BUF(4); - READ32(create->cr_linklen); + create->cr_linklen = be32_to_cpup(p++); READ_BUF(create->cr_linklen); SAVEMEM(create->cr_linkname, create->cr_linklen); break; case NF4BLK: case NF4CHR: READ_BUF(8); - READ32(create->cr_specdata1); - READ32(create->cr_specdata2); + create->cr_specdata1 = be32_to_cpup(p++); + create->cr_specdata2 = be32_to_cpup(p++); break; case NF4SOCK: case NF4FIFO: @@ -618,7 +616,7 @@ nfsd4_decode_create(struct nfsd4_compoundargs *argp, struct nfsd4_create *create } READ_BUF(4); - READ32(create->cr_namelen); + create->cr_namelen = be32_to_cpup(p++); READ_BUF(create->cr_namelen); SAVEMEM(create->cr_name, create->cr_namelen); if ((status = check_filename(create->cr_name, create->cr_namelen))) @@ -650,7 +648,7 @@ nfsd4_decode_link(struct nfsd4_compoundargs *argp, struct nfsd4_link *link) DECODE_HEAD; READ_BUF(4); - READ32(link->li_namelen); + link->li_namelen = be32_to_cpup(p++); READ_BUF(link->li_namelen); SAVEMEM(link->li_name, link->li_namelen); if ((status = check_filename(link->li_name, link->li_namelen))) @@ -668,24 +666,24 @@ nfsd4_decode_lock(struct nfsd4_compoundargs *argp, struct nfsd4_lock *lock) * type, reclaim(boolean), offset, length, new_lock_owner(boolean) */ READ_BUF(28); - READ32(lock->lk_type); + lock->lk_type = be32_to_cpup(p++); if ((lock->lk_type < NFS4_READ_LT) || (lock->lk_type > NFS4_WRITEW_LT)) goto xdr_error; - READ32(lock->lk_reclaim); - READ64(lock->lk_offset); - READ64(lock->lk_length); - READ32(lock->lk_is_new); + lock->lk_reclaim = be32_to_cpup(p++); + p = xdr_decode_hyper(p, &lock->lk_offset); + p = xdr_decode_hyper(p, &lock->lk_length); + lock->lk_is_new = be32_to_cpup(p++); if (lock->lk_is_new) { READ_BUF(4); - READ32(lock->lk_new_open_seqid); + lock->lk_new_open_seqid = be32_to_cpup(p++); status = nfsd4_decode_stateid(argp, &lock->lk_new_open_stateid); if (status) return status; READ_BUF(8 + sizeof(clientid_t)); - READ32(lock->lk_new_lock_seqid); + lock->lk_new_lock_seqid = be32_to_cpup(p++); COPYMEM(&lock->lk_new_clientid, sizeof(clientid_t)); - READ32(lock->lk_new_owner.len); + lock->lk_new_owner.len = be32_to_cpup(p++); READ_BUF(lock->lk_new_owner.len); READMEM(lock->lk_new_owner.data, lock->lk_new_owner.len); } else { @@ -693,7 +691,7 @@ nfsd4_decode_lock(struct nfsd4_compoundargs *argp, struct nfsd4_lock *lock) if (status) return status; READ_BUF(4); - READ32(lock->lk_old_lock_seqid); + lock->lk_old_lock_seqid = be32_to_cpup(p++); } DECODE_TAIL; @@ -705,13 +703,13 @@ nfsd4_decode_lockt(struct nfsd4_compoundargs *argp, struct nfsd4_lockt *lockt) DECODE_HEAD; READ_BUF(32); - READ32(lockt->lt_type); + lockt->lt_type = be32_to_cpup(p++); if((lockt->lt_type < NFS4_READ_LT) || (lockt->lt_type > NFS4_WRITEW_LT)) goto xdr_error; - READ64(lockt->lt_offset); - READ64(lockt->lt_length); + p = xdr_decode_hyper(p, &lockt->lt_offset); + p = xdr_decode_hyper(p, &lockt->lt_length); COPYMEM(&lockt->lt_clientid, 8); - READ32(lockt->lt_owner.len); + lockt->lt_owner.len = be32_to_cpup(p++); READ_BUF(lockt->lt_owner.len); READMEM(lockt->lt_owner.data, lockt->lt_owner.len); @@ -724,16 +722,16 @@ nfsd4_decode_locku(struct nfsd4_compoundargs *argp, struct nfsd4_locku *locku) DECODE_HEAD; READ_BUF(8); - READ32(locku->lu_type); + locku->lu_type = be32_to_cpup(p++); if ((locku->lu_type < NFS4_READ_LT) || (locku->lu_type > NFS4_WRITEW_LT)) goto xdr_error; - READ32(locku->lu_seqid); + locku->lu_seqid = be32_to_cpup(p++); status = nfsd4_decode_stateid(argp, &locku->lu_stateid); if (status) return status; READ_BUF(16); - READ64(locku->lu_offset); - READ64(locku->lu_length); + p = xdr_decode_hyper(p, &locku->lu_offset); + p = xdr_decode_hyper(p, &locku->lu_length); DECODE_TAIL; } @@ -744,7 +742,7 @@ nfsd4_decode_lookup(struct nfsd4_compoundargs *argp, struct nfsd4_lookup *lookup DECODE_HEAD; READ_BUF(4); - READ32(lookup->lo_len); + lookup->lo_len = be32_to_cpup(p++); READ_BUF(lookup->lo_len); SAVEMEM(lookup->lo_name, lookup->lo_len); if ((status = check_filename(lookup->lo_name, lookup->lo_len))) @@ -759,7 +757,7 @@ static __be32 nfsd4_decode_share_access(struct nfsd4_compoundargs *argp, u32 *sh u32 w; READ_BUF(4); - READ32(w); + w = be32_to_cpup(p++); *share_access = w & NFS4_SHARE_ACCESS_MASK; *deleg_want = w & NFS4_SHARE_WANT_MASK; if (deleg_when) @@ -811,7 +809,7 @@ static __be32 nfsd4_decode_share_deny(struct nfsd4_compoundargs *argp, u32 *x) __be32 *p; READ_BUF(4); - READ32(*x); + *x = be32_to_cpup(p++); /* Note: unlinke access bits, deny bits may be zero. */ if (*x & ~NFS4_SHARE_DENY_BOTH) return nfserr_bad_xdr; @@ -825,7 +823,7 @@ static __be32 nfsd4_decode_opaque(struct nfsd4_compoundargs *argp, struct xdr_ne __be32 *p; READ_BUF(4); - READ32(o->len); + o->len = be32_to_cpup(p++); if (o->len == 0 || o->len > NFS4_OPAQUE_LIMIT) return nfserr_bad_xdr; @@ -850,7 +848,7 @@ nfsd4_decode_open(struct nfsd4_compoundargs *argp, struct nfsd4_open *open) open->op_xdr_error = 0; /* seqid, share_access, share_deny, clientid, ownerlen */ READ_BUF(4); - READ32(open->op_seqid); + open->op_seqid = be32_to_cpup(p++); /* decode, yet ignore deleg_when until supported */ status = nfsd4_decode_share_access(argp, &open->op_share_access, &open->op_deleg_want, &dummy); @@ -865,13 +863,13 @@ nfsd4_decode_open(struct nfsd4_compoundargs *argp, struct nfsd4_open *open) if (status) goto xdr_error; READ_BUF(4); - READ32(open->op_create); + open->op_create = be32_to_cpup(p++); switch (open->op_create) { case NFS4_OPEN_NOCREATE: break; case NFS4_OPEN_CREATE: READ_BUF(4); - READ32(open->op_createmode); + open->op_createmode = be32_to_cpup(p++); switch (open->op_createmode) { case NFS4_CREATE_UNCHECKED: case NFS4_CREATE_GUARDED: @@ -904,12 +902,12 @@ nfsd4_decode_open(struct nfsd4_compoundargs *argp, struct nfsd4_open *open) /* open_claim */ READ_BUF(4); - READ32(open->op_claim_type); + open->op_claim_type = be32_to_cpup(p++); switch (open->op_claim_type) { case NFS4_OPEN_CLAIM_NULL: case NFS4_OPEN_CLAIM_DELEGATE_PREV: READ_BUF(4); - READ32(open->op_fname.len); + open->op_fname.len = be32_to_cpup(p++); READ_BUF(open->op_fname.len); SAVEMEM(open->op_fname.data, open->op_fname.len); if ((status = check_filename(open->op_fname.data, open->op_fname.len))) @@ -917,14 +915,14 @@ nfsd4_decode_open(struct nfsd4_compoundargs *argp, struct nfsd4_open *open) break; case NFS4_OPEN_CLAIM_PREVIOUS: READ_BUF(4); - READ32(open->op_delegate_type); + open->op_delegate_type = be32_to_cpup(p++); break; case NFS4_OPEN_CLAIM_DELEGATE_CUR: status = nfsd4_decode_stateid(argp, &open->op_delegate_stateid); if (status) return status; READ_BUF(4); - READ32(open->op_fname.len); + open->op_fname.len = be32_to_cpup(p++); READ_BUF(open->op_fname.len); SAVEMEM(open->op_fname.data, open->op_fname.len); if ((status = check_filename(open->op_fname.data, open->op_fname.len))) @@ -962,7 +960,7 @@ nfsd4_decode_open_confirm(struct nfsd4_compoundargs *argp, struct nfsd4_open_con if (status) return status; READ_BUF(4); - READ32(open_conf->oc_seqid); + open_conf->oc_seqid = be32_to_cpup(p++); DECODE_TAIL; } @@ -976,7 +974,7 @@ nfsd4_decode_open_downgrade(struct nfsd4_compoundargs *argp, struct nfsd4_open_d if (status) return status; READ_BUF(4); - READ32(open_down->od_seqid); + open_down->od_seqid = be32_to_cpup(p++); status = nfsd4_decode_share_access(argp, &open_down->od_share_access, &open_down->od_deleg_want, NULL); if (status) @@ -993,7 +991,7 @@ nfsd4_decode_putfh(struct nfsd4_compoundargs *argp, struct nfsd4_putfh *putfh) DECODE_HEAD; READ_BUF(4); - READ32(putfh->pf_fhlen); + putfh->pf_fhlen = be32_to_cpup(p++); if (putfh->pf_fhlen > NFS4_FHSIZE) goto xdr_error; READ_BUF(putfh->pf_fhlen); @@ -1019,8 +1017,8 @@ nfsd4_decode_read(struct nfsd4_compoundargs *argp, struct nfsd4_read *read) if (status) return status; READ_BUF(12); - READ64(read->rd_offset); - READ32(read->rd_length); + p = xdr_decode_hyper(p, &read->rd_offset); + read->rd_length = be32_to_cpup(p++); DECODE_TAIL; } @@ -1031,10 +1029,10 @@ nfsd4_decode_readdir(struct nfsd4_compoundargs *argp, struct nfsd4_readdir *read DECODE_HEAD; READ_BUF(24); - READ64(readdir->rd_cookie); + p = xdr_decode_hyper(p, &readdir->rd_cookie); COPYMEM(readdir->rd_verf.data, sizeof(readdir->rd_verf.data)); - READ32(readdir->rd_dircount); /* just in case you needed a useless field... */ - READ32(readdir->rd_maxcount); + readdir->rd_dircount = be32_to_cpup(p++); + readdir->rd_maxcount = be32_to_cpup(p++); if ((status = nfsd4_decode_bitmap(argp, readdir->rd_bmval))) goto out; @@ -1047,7 +1045,7 @@ nfsd4_decode_remove(struct nfsd4_compoundargs *argp, struct nfsd4_remove *remove DECODE_HEAD; READ_BUF(4); - READ32(remove->rm_namelen); + remove->rm_namelen = be32_to_cpup(p++); READ_BUF(remove->rm_namelen); SAVEMEM(remove->rm_name, remove->rm_namelen); if ((status = check_filename(remove->rm_name, remove->rm_namelen))) @@ -1062,10 +1060,10 @@ nfsd4_decode_rename(struct nfsd4_compoundargs *argp, struct nfsd4_rename *rename DECODE_HEAD; READ_BUF(4); - READ32(rename->rn_snamelen); + rename->rn_snamelen = be32_to_cpup(p++); READ_BUF(rename->rn_snamelen + 4); SAVEMEM(rename->rn_sname, rename->rn_snamelen); - READ32(rename->rn_tnamelen); + rename->rn_tnamelen = be32_to_cpup(p++); READ_BUF(rename->rn_tnamelen); SAVEMEM(rename->rn_tname, rename->rn_tnamelen); if ((status = check_filename(rename->rn_sname, rename->rn_snamelen))) @@ -1097,7 +1095,7 @@ nfsd4_decode_secinfo(struct nfsd4_compoundargs *argp, DECODE_HEAD; READ_BUF(4); - READ32(secinfo->si_namelen); + secinfo->si_namelen = be32_to_cpup(p++); READ_BUF(secinfo->si_namelen); SAVEMEM(secinfo->si_name, secinfo->si_namelen); status = check_filename(secinfo->si_name, secinfo->si_namelen); @@ -1113,7 +1111,7 @@ nfsd4_decode_secinfo_no_name(struct nfsd4_compoundargs *argp, DECODE_HEAD; READ_BUF(4); - READ32(sin->sin_style); + sin->sin_style = be32_to_cpup(p++); DECODE_TAIL; } @@ -1144,16 +1142,16 @@ nfsd4_decode_setclientid(struct nfsd4_compoundargs *argp, struct nfsd4_setclient if (status) return nfserr_bad_xdr; READ_BUF(8); - READ32(setclientid->se_callback_prog); - READ32(setclientid->se_callback_netid_len); + setclientid->se_callback_prog = be32_to_cpup(p++); + setclientid->se_callback_netid_len = be32_to_cpup(p++); READ_BUF(setclientid->se_callback_netid_len + 4); SAVEMEM(setclientid->se_callback_netid_val, setclientid->se_callback_netid_len); - READ32(setclientid->se_callback_addr_len); + setclientid->se_callback_addr_len = be32_to_cpup(p++); READ_BUF(setclientid->se_callback_addr_len + 4); SAVEMEM(setclientid->se_callback_addr_val, setclientid->se_callback_addr_len); - READ32(setclientid->se_callback_ident); + setclientid->se_callback_ident = be32_to_cpup(p++); DECODE_TAIL; } @@ -1186,7 +1184,7 @@ nfsd4_decode_verify(struct nfsd4_compoundargs *argp, struct nfsd4_verify *verify * nfsd4_proc_verify */ READ_BUF(4); - READ32(verify->ve_attrlen); + verify->ve_attrlen = be32_to_cpup(p++); READ_BUF(verify->ve_attrlen); SAVEMEM(verify->ve_attrval, verify->ve_attrlen); @@ -1204,11 +1202,11 @@ nfsd4_decode_write(struct nfsd4_compoundargs *argp, struct nfsd4_write *write) if (status) return status; READ_BUF(16); - READ64(write->wr_offset); - READ32(write->wr_stable_how); + p = xdr_decode_hyper(p, &write->wr_offset); + write->wr_stable_how = be32_to_cpup(p++); if (write->wr_stable_how > 2) goto xdr_error; - READ32(write->wr_buflen); + write->wr_buflen = be32_to_cpup(p++); /* Sorry .. no magic macros for this.. * * READ_BUF(write->wr_buflen); @@ -1254,7 +1252,7 @@ nfsd4_decode_release_lockowner(struct nfsd4_compoundargs *argp, struct nfsd4_rel READ_BUF(12); COPYMEM(&rlockowner->rl_clientid, sizeof(clientid_t)); - READ32(rlockowner->rl_owner.len); + rlockowner->rl_owner.len = be32_to_cpup(p++); READ_BUF(rlockowner->rl_owner.len); READMEM(rlockowner->rl_owner.data, rlockowner->rl_owner.len); @@ -1278,63 +1276,63 @@ nfsd4_decode_exchange_id(struct nfsd4_compoundargs *argp, return nfserr_bad_xdr; READ_BUF(4); - READ32(exid->flags); + exid->flags = be32_to_cpup(p++); /* Ignore state_protect4_a */ READ_BUF(4); - READ32(exid->spa_how); + exid->spa_how = be32_to_cpup(p++); switch (exid->spa_how) { case SP4_NONE: break; case SP4_MACH_CRED: /* spo_must_enforce */ READ_BUF(4); - READ32(dummy); + dummy = be32_to_cpup(p++); READ_BUF(dummy * 4); p += dummy; /* spo_must_allow */ READ_BUF(4); - READ32(dummy); + dummy = be32_to_cpup(p++); READ_BUF(dummy * 4); p += dummy; break; case SP4_SSV: /* ssp_ops */ READ_BUF(4); - READ32(dummy); + dummy = be32_to_cpup(p++); READ_BUF(dummy * 4); p += dummy; READ_BUF(4); - READ32(dummy); + dummy = be32_to_cpup(p++); READ_BUF(dummy * 4); p += dummy; /* ssp_hash_algs<> */ READ_BUF(4); - READ32(tmp); + tmp = be32_to_cpup(p++); while (tmp--) { READ_BUF(4); - READ32(dummy); + dummy = be32_to_cpup(p++); READ_BUF(dummy); p += XDR_QUADLEN(dummy); } /* ssp_encr_algs<> */ READ_BUF(4); - READ32(tmp); + tmp = be32_to_cpup(p++); while (tmp--) { READ_BUF(4); - READ32(dummy); + dummy = be32_to_cpup(p++); READ_BUF(dummy); p += XDR_QUADLEN(dummy); } /* ssp_window and ssp_num_gss_handles */ READ_BUF(8); - READ32(dummy); - READ32(dummy); + dummy = be32_to_cpup(p++); + dummy = be32_to_cpup(p++); break; default: goto xdr_error; @@ -1342,7 +1340,7 @@ nfsd4_decode_exchange_id(struct nfsd4_compoundargs *argp, /* Ignore Implementation ID */ READ_BUF(4); /* nfs_impl_id4 array length */ - READ32(dummy); + dummy = be32_to_cpup(p++); if (dummy > 1) goto xdr_error; @@ -1350,13 +1348,13 @@ nfsd4_decode_exchange_id(struct nfsd4_compoundargs *argp, if (dummy == 1) { /* nii_domain */ READ_BUF(4); - READ32(dummy); + dummy = be32_to_cpup(p++); READ_BUF(dummy); p += XDR_QUADLEN(dummy); /* nii_name */ READ_BUF(4); - READ32(dummy); + dummy = be32_to_cpup(p++); READ_BUF(dummy); p += XDR_QUADLEN(dummy); @@ -1376,21 +1374,21 @@ nfsd4_decode_create_session(struct nfsd4_compoundargs *argp, READ_BUF(16); COPYMEM(&sess->clientid, 8); - READ32(sess->seqid); - READ32(sess->flags); + sess->seqid = be32_to_cpup(p++); + sess->flags = be32_to_cpup(p++); /* Fore channel attrs */ READ_BUF(28); - READ32(dummy); /* headerpadsz is always 0 */ - READ32(sess->fore_channel.maxreq_sz); - READ32(sess->fore_channel.maxresp_sz); - READ32(sess->fore_channel.maxresp_cached); - READ32(sess->fore_channel.maxops); - READ32(sess->fore_channel.maxreqs); - READ32(sess->fore_channel.nr_rdma_attrs); + dummy = be32_to_cpup(p++); /* headerpadsz is always 0 */ + sess->fore_channel.maxreq_sz = be32_to_cpup(p++); + sess->fore_channel.maxresp_sz = be32_to_cpup(p++); + sess->fore_channel.maxresp_cached = be32_to_cpup(p++); + sess->fore_channel.maxops = be32_to_cpup(p++); + sess->fore_channel.maxreqs = be32_to_cpup(p++); + sess->fore_channel.nr_rdma_attrs = be32_to_cpup(p++); if (sess->fore_channel.nr_rdma_attrs == 1) { READ_BUF(4); - READ32(sess->fore_channel.rdma_attrs); + sess->fore_channel.rdma_attrs = be32_to_cpup(p++); } else if (sess->fore_channel.nr_rdma_attrs > 1) { dprintk("Too many fore channel attr bitmaps!\n"); goto xdr_error; @@ -1398,23 +1396,23 @@ nfsd4_decode_create_session(struct nfsd4_compoundargs *argp, /* Back channel attrs */ READ_BUF(28); - READ32(dummy); /* headerpadsz is always 0 */ - READ32(sess->back_channel.maxreq_sz); - READ32(sess->back_channel.maxresp_sz); - READ32(sess->back_channel.maxresp_cached); - READ32(sess->back_channel.maxops); - READ32(sess->back_channel.maxreqs); - READ32(sess->back_channel.nr_rdma_attrs); + dummy = be32_to_cpup(p++); /* headerpadsz is always 0 */ + sess->back_channel.maxreq_sz = be32_to_cpup(p++); + sess->back_channel.maxresp_sz = be32_to_cpup(p++); + sess->back_channel.maxresp_cached = be32_to_cpup(p++); + sess->back_channel.maxops = be32_to_cpup(p++); + sess->back_channel.maxreqs = be32_to_cpup(p++); + sess->back_channel.nr_rdma_attrs = be32_to_cpup(p++); if (sess->back_channel.nr_rdma_attrs == 1) { READ_BUF(4); - READ32(sess->back_channel.rdma_attrs); + sess->back_channel.rdma_attrs = be32_to_cpup(p++); } else if (sess->back_channel.nr_rdma_attrs > 1) { dprintk("Too many back channel attr bitmaps!\n"); goto xdr_error; } READ_BUF(4); - READ32(sess->callback_prog); + sess->callback_prog = be32_to_cpup(p++); nfsd4_decode_cb_sec(argp, &sess->cb_sec); DECODE_TAIL; } @@ -1437,7 +1435,7 @@ nfsd4_decode_free_stateid(struct nfsd4_compoundargs *argp, DECODE_HEAD; READ_BUF(sizeof(stateid_t)); - READ32(free_stateid->fr_stateid.si_generation); + free_stateid->fr_stateid.si_generation = be32_to_cpup(p++); COPYMEM(&free_stateid->fr_stateid.si_opaque, sizeof(stateid_opaque_t)); DECODE_TAIL; @@ -1451,10 +1449,10 @@ nfsd4_decode_sequence(struct nfsd4_compoundargs *argp, READ_BUF(NFS4_MAX_SESSIONID_LEN + 16); COPYMEM(seq->sessionid.data, NFS4_MAX_SESSIONID_LEN); - READ32(seq->seqid); - READ32(seq->slotid); - READ32(seq->maxslots); - READ32(seq->cachethis); + seq->seqid = be32_to_cpup(p++); + seq->slotid = be32_to_cpup(p++); + seq->maxslots = be32_to_cpup(p++); + seq->cachethis = be32_to_cpup(p++); DECODE_TAIL; } @@ -1511,7 +1509,7 @@ static __be32 nfsd4_decode_reclaim_complete(struct nfsd4_compoundargs *argp, str DECODE_HEAD; READ_BUF(4); - READ32(rc->rca_one_fs); + rc->rca_one_fs = be32_to_cpup(p++); DECODE_TAIL; } @@ -1605,47 +1603,25 @@ nfsd4_opnum_in_range(struct nfsd4_compoundargs *argp, struct nfsd4_op *op) return true; } -/* - * Return a rough estimate of the maximum possible reply size. Note the - * estimate includes rpc headers so is meant to be passed to - * svc_reserve, not svc_reserve_auth. - * - * Also note the current compound encoding permits only one operation to - * use pages beyond the first one, so the maximum possible length is the - * maximum over these values, not the sum. - */ -static int nfsd4_max_reply(u32 opnum) -{ - switch (opnum) { - case OP_READLINK: - case OP_READDIR: - /* - * Both of these ops take a single page for data and put - * the head and tail in another page: - */ - return 2 * PAGE_SIZE; - case OP_READ: - return INT_MAX; - default: - return PAGE_SIZE; - } -} - static __be32 nfsd4_decode_compound(struct nfsd4_compoundargs *argp) { DECODE_HEAD; struct nfsd4_op *op; bool cachethis = false; - int max_reply = PAGE_SIZE; + int auth_slack= argp->rqstp->rq_auth_slack; + int max_reply = auth_slack + 8; /* opcnt, status */ + int readcount = 0; + int readbytes = 0; int i; READ_BUF(4); - READ32(argp->taglen); + argp->taglen = be32_to_cpup(p++); READ_BUF(argp->taglen + 8); SAVEMEM(argp->tag, argp->taglen); - READ32(argp->minorversion); - READ32(argp->opcnt); + argp->minorversion = be32_to_cpup(p++); + argp->opcnt = be32_to_cpup(p++); + max_reply += 4 + (XDR_QUADLEN(argp->taglen) << 2); if (argp->taglen > NFSD4_MAX_TAGLEN) goto xdr_error; @@ -1669,7 +1645,7 @@ nfsd4_decode_compound(struct nfsd4_compoundargs *argp) op->replay = NULL; READ_BUF(4); - READ32(op->opnum); + op->opnum = be32_to_cpup(p++); if (nfsd4_opnum_in_range(argp, op)) op->status = nfsd4_dec_ops[op->opnum](argp, &op->u); @@ -1677,97 +1653,82 @@ nfsd4_decode_compound(struct nfsd4_compoundargs *argp) op->opnum = OP_ILLEGAL; op->status = nfserr_op_illegal; } - - if (op->status) { - argp->opcnt = i+1; - break; - } /* * We'll try to cache the result in the DRC if any one * op in the compound wants to be cached: */ cachethis |= nfsd4_cache_this_op(op); - max_reply = max(max_reply, nfsd4_max_reply(op->opnum)); + if (op->opnum == OP_READ) { + readcount++; + readbytes += nfsd4_max_reply(argp->rqstp, op); + } else + max_reply += nfsd4_max_reply(argp->rqstp, op); + + if (op->status) { + argp->opcnt = i+1; + break; + } } /* Sessions make the DRC unnecessary: */ if (argp->minorversion) cachethis = false; - if (max_reply != INT_MAX) - svc_reserve(argp->rqstp, max_reply); + svc_reserve(argp->rqstp, max_reply + readbytes); argp->rqstp->rq_cachetype = cachethis ? RC_REPLBUFF : RC_NOCACHE; - DECODE_TAIL; -} - -#define WRITE32(n) *p++ = htonl(n) -#define WRITE64(n) do { \ - *p++ = htonl((u32)((n) >> 32)); \ - *p++ = htonl((u32)(n)); \ -} while (0) -#define WRITEMEM(ptr,nbytes) do { if (nbytes > 0) { \ - *(p + XDR_QUADLEN(nbytes) -1) = 0; \ - memcpy(p, ptr, nbytes); \ - p += XDR_QUADLEN(nbytes); \ -}} while (0) - -static void write32(__be32 **p, u32 n) -{ - *(*p)++ = htonl(n); -} + if (readcount > 1 || max_reply > PAGE_SIZE - auth_slack) + argp->rqstp->rq_splice_ok = false; -static void write64(__be32 **p, u64 n) -{ - write32(p, (n >> 32)); - write32(p, (u32)n); + DECODE_TAIL; } -static void write_change(__be32 **p, struct kstat *stat, struct inode *inode) +static __be32 *encode_change(__be32 *p, struct kstat *stat, struct inode *inode) { if (IS_I_VERSION(inode)) { - write64(p, inode->i_version); + p = xdr_encode_hyper(p, inode->i_version); } else { - write32(p, stat->ctime.tv_sec); - write32(p, stat->ctime.tv_nsec); + *p++ = cpu_to_be32(stat->ctime.tv_sec); + *p++ = cpu_to_be32(stat->ctime.tv_nsec); } + return p; } -static void write_cinfo(__be32 **p, struct nfsd4_change_info *c) +static __be32 *encode_cinfo(__be32 *p, struct nfsd4_change_info *c) { - write32(p, c->atomic); + *p++ = cpu_to_be32(c->atomic); if (c->change_supported) { - write64(p, c->before_change); - write64(p, c->after_change); + p = xdr_encode_hyper(p, c->before_change); + p = xdr_encode_hyper(p, c->after_change); } else { - write32(p, c->before_ctime_sec); - write32(p, c->before_ctime_nsec); - write32(p, c->after_ctime_sec); - write32(p, c->after_ctime_nsec); + *p++ = cpu_to_be32(c->before_ctime_sec); + *p++ = cpu_to_be32(c->before_ctime_nsec); + *p++ = cpu_to_be32(c->after_ctime_sec); + *p++ = cpu_to_be32(c->after_ctime_nsec); } + return p; } -#define RESERVE_SPACE(nbytes) do { \ - p = resp->p; \ - BUG_ON(p + XDR_QUADLEN(nbytes) > resp->end); \ -} while (0) -#define ADJUST_ARGS() resp->p = p - /* Encode as an array of strings the string given with components * separated @sep, escaped with esc_enter and esc_exit. */ -static __be32 nfsd4_encode_components_esc(char sep, char *components, - __be32 **pp, int *buflen, - char esc_enter, char esc_exit) +static __be32 nfsd4_encode_components_esc(struct xdr_stream *xdr, char sep, + char *components, char esc_enter, + char esc_exit) { - __be32 *p = *pp; - __be32 *countp = p; + __be32 *p; + __be32 pathlen; + int pathlen_offset; int strlen, count=0; char *str, *end, *next; dprintk("nfsd4_encode_components(%s)\n", components); - if ((*buflen -= 4) < 0) + + pathlen_offset = xdr->buf->len; + p = xdr_reserve_space(xdr, 4); + if (!p) return nfserr_resource; - WRITE32(0); /* We will fill this in with @count later */ + p++; /* We will fill this in with @count later */ + end = str = components; while (*end) { bool found_esc = false; @@ -1789,59 +1750,57 @@ static __be32 nfsd4_encode_components_esc(char sep, char *components, strlen = end - str; if (strlen) { - if ((*buflen -= ((XDR_QUADLEN(strlen) << 2) + 4)) < 0) + p = xdr_reserve_space(xdr, strlen + 4); + if (!p) return nfserr_resource; - WRITE32(strlen); - WRITEMEM(str, strlen); + p = xdr_encode_opaque(p, str, strlen); count++; } else end++; str = end; } - *pp = p; - p = countp; - WRITE32(count); + pathlen = htonl(xdr->buf->len - pathlen_offset); + write_bytes_to_xdr_buf(xdr->buf, pathlen_offset, &pathlen, 4); return 0; } /* Encode as an array of strings the string given with components * separated @sep. */ -static __be32 nfsd4_encode_components(char sep, char *components, - __be32 **pp, int *buflen) +static __be32 nfsd4_encode_components(struct xdr_stream *xdr, char sep, + char *components) { - return nfsd4_encode_components_esc(sep, components, pp, buflen, 0, 0); + return nfsd4_encode_components_esc(xdr, sep, components, 0, 0); } /* * encode a location element of a fs_locations structure */ -static __be32 nfsd4_encode_fs_location4(struct nfsd4_fs_location *location, - __be32 **pp, int *buflen) +static __be32 nfsd4_encode_fs_location4(struct xdr_stream *xdr, + struct nfsd4_fs_location *location) { __be32 status; - __be32 *p = *pp; - status = nfsd4_encode_components_esc(':', location->hosts, &p, buflen, + status = nfsd4_encode_components_esc(xdr, ':', location->hosts, '[', ']'); if (status) return status; - status = nfsd4_encode_components('/', location->path, &p, buflen); + status = nfsd4_encode_components(xdr, '/', location->path); if (status) return status; - *pp = p; return 0; } /* * Encode a path in RFC3530 'pathname4' format */ -static __be32 nfsd4_encode_path(const struct path *root, - const struct path *path, __be32 **pp, int *buflen) +static __be32 nfsd4_encode_path(struct xdr_stream *xdr, + const struct path *root, + const struct path *path) { struct path cur = *path; - __be32 *p = *pp; + __be32 *p; struct dentry **components = NULL; unsigned int ncomponents = 0; __be32 err = nfserr_jukebox; @@ -1872,11 +1831,11 @@ static __be32 nfsd4_encode_path(const struct path *root, components[ncomponents++] = cur.dentry; cur.dentry = dget_parent(cur.dentry); } - - *buflen -= 4; - if (*buflen < 0) + err = nfserr_resource; + p = xdr_reserve_space(xdr, 4); + if (!p) goto out_free; - WRITE32(ncomponents); + *p++ = cpu_to_be32(ncomponents); while (ncomponents) { struct dentry *dentry = components[ncomponents - 1]; @@ -1884,20 +1843,18 @@ static __be32 nfsd4_encode_path(const struct path *root, spin_lock(&dentry->d_lock); len = dentry->d_name.len; - *buflen -= 4 + (XDR_QUADLEN(len) << 2); - if (*buflen < 0) { + p = xdr_reserve_space(xdr, len + 4); + if (!p) { spin_unlock(&dentry->d_lock); goto out_free; } - WRITE32(len); - WRITEMEM(dentry->d_name.name, len); + p = xdr_encode_opaque(p, dentry->d_name.name, len); dprintk("/%s", dentry->d_name.name); spin_unlock(&dentry->d_lock); dput(dentry); ncomponents--; } - *pp = p; err = 0; out_free: dprintk(")\n"); @@ -1908,8 +1865,8 @@ out_free: return err; } -static __be32 nfsd4_encode_fsloc_fsroot(struct svc_rqst *rqstp, - const struct path *path, __be32 **pp, int *buflen) +static __be32 nfsd4_encode_fsloc_fsroot(struct xdr_stream *xdr, + struct svc_rqst *rqstp, const struct path *path) { struct svc_export *exp_ps; __be32 res; @@ -1917,7 +1874,7 @@ static __be32 nfsd4_encode_fsloc_fsroot(struct svc_rqst *rqstp, exp_ps = rqst_find_fsidzero_export(rqstp); if (IS_ERR(exp_ps)) return nfserrno(PTR_ERR(exp_ps)); - res = nfsd4_encode_path(&exp_ps->ex_path, path, pp, buflen); + res = nfsd4_encode_path(xdr, &exp_ps->ex_path, path); exp_put(exp_ps); return res; } @@ -1925,28 +1882,26 @@ static __be32 nfsd4_encode_fsloc_fsroot(struct svc_rqst *rqstp, /* * encode a fs_locations structure */ -static __be32 nfsd4_encode_fs_locations(struct svc_rqst *rqstp, - struct svc_export *exp, - __be32 **pp, int *buflen) +static __be32 nfsd4_encode_fs_locations(struct xdr_stream *xdr, + struct svc_rqst *rqstp, struct svc_export *exp) { __be32 status; int i; - __be32 *p = *pp; + __be32 *p; struct nfsd4_fs_locations *fslocs = &exp->ex_fslocs; - status = nfsd4_encode_fsloc_fsroot(rqstp, &exp->ex_path, &p, buflen); + status = nfsd4_encode_fsloc_fsroot(xdr, rqstp, &exp->ex_path); if (status) return status; - if ((*buflen -= 4) < 0) + p = xdr_reserve_space(xdr, 4); + if (!p) return nfserr_resource; - WRITE32(fslocs->locations_count); + *p++ = cpu_to_be32(fslocs->locations_count); for (i=0; i<fslocs->locations_count; i++) { - status = nfsd4_encode_fs_location4(&fslocs->locations[i], - &p, buflen); + status = nfsd4_encode_fs_location4(xdr, &fslocs->locations[i]); if (status) return status; } - *pp = p; return 0; } @@ -1965,15 +1920,15 @@ static u32 nfs4_file_type(umode_t mode) } static inline __be32 -nfsd4_encode_aclname(struct svc_rqst *rqstp, struct nfs4_ace *ace, - __be32 **p, int *buflen) +nfsd4_encode_aclname(struct xdr_stream *xdr, struct svc_rqst *rqstp, + struct nfs4_ace *ace) { if (ace->whotype != NFS4_ACL_WHO_NAMED) - return nfs4_acl_write_who(ace->whotype, p, buflen); + return nfs4_acl_write_who(xdr, ace->whotype); else if (ace->flag & NFS4_ACE_IDENTIFIER_GROUP) - return nfsd4_encode_group(rqstp, ace->who_gid, p, buflen); + return nfsd4_encode_group(xdr, rqstp, ace->who_gid); else - return nfsd4_encode_user(rqstp, ace->who_uid, p, buflen); + return nfsd4_encode_user(xdr, rqstp, ace->who_uid); } #define WORD0_ABSENT_FS_ATTRS (FATTR4_WORD0_FS_LOCATIONS | FATTR4_WORD0_FSID | \ @@ -1982,31 +1937,28 @@ nfsd4_encode_aclname(struct svc_rqst *rqstp, struct nfs4_ace *ace, #ifdef CONFIG_NFSD_V4_SECURITY_LABEL static inline __be32 -nfsd4_encode_security_label(struct svc_rqst *rqstp, void *context, int len, __be32 **pp, int *buflen) +nfsd4_encode_security_label(struct xdr_stream *xdr, struct svc_rqst *rqstp, + void *context, int len) { - __be32 *p = *pp; + __be32 *p; - if (*buflen < ((XDR_QUADLEN(len) << 2) + 4 + 4 + 4)) + p = xdr_reserve_space(xdr, len + 4 + 4 + 4); + if (!p) return nfserr_resource; /* * For now we use a 0 here to indicate the null translation; in * the future we may place a call to translation code here. */ - if ((*buflen -= 8) < 0) - return nfserr_resource; - - WRITE32(0); /* lfs */ - WRITE32(0); /* pi */ + *p++ = cpu_to_be32(0); /* lfs */ + *p++ = cpu_to_be32(0); /* pi */ p = xdr_encode_opaque(p, context, len); - *buflen -= (XDR_QUADLEN(len) << 2) + 4; - - *pp = p; return 0; } #else static inline __be32 -nfsd4_encode_security_label(struct svc_rqst *rqstp, void *context, int len, __be32 **pp, int *buflen) +nfsd4_encode_security_label(struct xdr_stream *xdr, struct svc_rqst *rqstp, + void *context, int len) { return 0; } #endif @@ -2045,12 +1997,11 @@ static int get_parent_attributes(struct svc_export *exp, struct kstat *stat) /* * Note: @fhp can be NULL; in this case, we might have to compose the filehandle * ourselves. - * - * countp is the buffer size in _words_ */ -__be32 -nfsd4_encode_fattr(struct svc_fh *fhp, struct svc_export *exp, - struct dentry *dentry, __be32 **buffer, int count, u32 *bmval, +static __be32 +nfsd4_encode_fattr(struct xdr_stream *xdr, struct svc_fh *fhp, + struct svc_export *exp, + struct dentry *dentry, u32 *bmval, struct svc_rqst *rqstp, int ignore_crossmnt) { u32 bmval0 = bmval[0]; @@ -2059,12 +2010,13 @@ nfsd4_encode_fattr(struct svc_fh *fhp, struct svc_export *exp, struct kstat stat; struct svc_fh *tempfh = NULL; struct kstatfs statfs; - int buflen = count << 2; - __be32 *attrlenp; + __be32 *p; + int starting_len = xdr->buf->len; + int attrlen_offset; + __be32 attrlen; u32 dummy; u64 dummy64; u32 rdattr_err = 0; - __be32 *p = *buffer; __be32 status; int err; int aclsupport = 0; @@ -2095,8 +2047,8 @@ nfsd4_encode_fattr(struct svc_fh *fhp, struct svc_export *exp, err = vfs_getattr(&path, &stat); if (err) goto out_nfserr; - if ((bmval0 & (FATTR4_WORD0_FILES_FREE | FATTR4_WORD0_FILES_TOTAL | - FATTR4_WORD0_MAXNAME)) || + if ((bmval0 & (FATTR4_WORD0_FILES_AVAIL | FATTR4_WORD0_FILES_FREE | + FATTR4_WORD0_FILES_TOTAL | FATTR4_WORD0_MAXNAME)) || (bmval1 & (FATTR4_WORD1_SPACE_AVAIL | FATTR4_WORD1_SPACE_FREE | FATTR4_WORD1_SPACE_TOTAL))) { err = vfs_statfs(&path, &statfs); @@ -2145,25 +2097,33 @@ nfsd4_encode_fattr(struct svc_fh *fhp, struct svc_export *exp, #endif /* CONFIG_NFSD_V4_SECURITY_LABEL */ if (bmval2) { - if ((buflen -= 16) < 0) + p = xdr_reserve_space(xdr, 16); + if (!p) goto out_resource; - WRITE32(3); - WRITE32(bmval0); - WRITE32(bmval1); - WRITE32(bmval2); + *p++ = cpu_to_be32(3); + *p++ = cpu_to_be32(bmval0); + *p++ = cpu_to_be32(bmval1); + *p++ = cpu_to_be32(bmval2); } else if (bmval1) { - if ((buflen -= 12) < 0) + p = xdr_reserve_space(xdr, 12); + if (!p) goto out_resource; - WRITE32(2); - WRITE32(bmval0); - WRITE32(bmval1); + *p++ = cpu_to_be32(2); + *p++ = cpu_to_be32(bmval0); + *p++ = cpu_to_be32(bmval1); } else { - if ((buflen -= 8) < 0) + p = xdr_reserve_space(xdr, 8); + if (!p) goto out_resource; - WRITE32(1); - WRITE32(bmval0); + *p++ = cpu_to_be32(1); + *p++ = cpu_to_be32(bmval0); } - attrlenp = p++; /* to be backfilled later */ + + attrlen_offset = xdr->buf->len; + p = xdr_reserve_space(xdr, 4); + if (!p) + goto out_resource; + p++; /* to be backfilled later */ if (bmval0 & FATTR4_WORD0_SUPPORTED_ATTRS) { u32 word0 = nfsd_suppattrs0(minorversion); @@ -2175,296 +2135,343 @@ nfsd4_encode_fattr(struct svc_fh *fhp, struct svc_export *exp, if (!contextsupport) word2 &= ~FATTR4_WORD2_SECURITY_LABEL; if (!word2) { - if ((buflen -= 12) < 0) + p = xdr_reserve_space(xdr, 12); + if (!p) goto out_resource; - WRITE32(2); - WRITE32(word0); - WRITE32(word1); + *p++ = cpu_to_be32(2); + *p++ = cpu_to_be32(word0); + *p++ = cpu_to_be32(word1); } else { - if ((buflen -= 16) < 0) + p = xdr_reserve_space(xdr, 16); + if (!p) goto out_resource; - WRITE32(3); - WRITE32(word0); - WRITE32(word1); - WRITE32(word2); + *p++ = cpu_to_be32(3); + *p++ = cpu_to_be32(word0); + *p++ = cpu_to_be32(word1); + *p++ = cpu_to_be32(word2); } } if (bmval0 & FATTR4_WORD0_TYPE) { - if ((buflen -= 4) < 0) + p = xdr_reserve_space(xdr, 4); + if (!p) goto out_resource; dummy = nfs4_file_type(stat.mode); if (dummy == NF4BAD) { status = nfserr_serverfault; goto out; } - WRITE32(dummy); + *p++ = cpu_to_be32(dummy); } if (bmval0 & FATTR4_WORD0_FH_EXPIRE_TYPE) { - if ((buflen -= 4) < 0) + p = xdr_reserve_space(xdr, 4); + if (!p) goto out_resource; if (exp->ex_flags & NFSEXP_NOSUBTREECHECK) - WRITE32(NFS4_FH_PERSISTENT); + *p++ = cpu_to_be32(NFS4_FH_PERSISTENT); else - WRITE32(NFS4_FH_PERSISTENT|NFS4_FH_VOL_RENAME); + *p++ = cpu_to_be32(NFS4_FH_PERSISTENT| + NFS4_FH_VOL_RENAME); } if (bmval0 & FATTR4_WORD0_CHANGE) { - if ((buflen -= 8) < 0) + p = xdr_reserve_space(xdr, 8); + if (!p) goto out_resource; - write_change(&p, &stat, dentry->d_inode); + p = encode_change(p, &stat, dentry->d_inode); } if (bmval0 & FATTR4_WORD0_SIZE) { - if ((buflen -= 8) < 0) + p = xdr_reserve_space(xdr, 8); + if (!p) goto out_resource; - WRITE64(stat.size); + p = xdr_encode_hyper(p, stat.size); } if (bmval0 & FATTR4_WORD0_LINK_SUPPORT) { - if ((buflen -= 4) < 0) + p = xdr_reserve_space(xdr, 4); + if (!p) goto out_resource; - WRITE32(1); + *p++ = cpu_to_be32(1); } if (bmval0 & FATTR4_WORD0_SYMLINK_SUPPORT) { - if ((buflen -= 4) < 0) + p = xdr_reserve_space(xdr, 4); + if (!p) goto out_resource; - WRITE32(1); + *p++ = cpu_to_be32(1); } if (bmval0 & FATTR4_WORD0_NAMED_ATTR) { - if ((buflen -= 4) < 0) + p = xdr_reserve_space(xdr, 4); + if (!p) goto out_resource; - WRITE32(0); + *p++ = cpu_to_be32(0); } if (bmval0 & FATTR4_WORD0_FSID) { - if ((buflen -= 16) < 0) + p = xdr_reserve_space(xdr, 16); + if (!p) goto out_resource; if (exp->ex_fslocs.migrated) { - WRITE64(NFS4_REFERRAL_FSID_MAJOR); - WRITE64(NFS4_REFERRAL_FSID_MINOR); + p = xdr_encode_hyper(p, NFS4_REFERRAL_FSID_MAJOR); + p = xdr_encode_hyper(p, NFS4_REFERRAL_FSID_MINOR); } else switch(fsid_source(fhp)) { case FSIDSOURCE_FSID: - WRITE64((u64)exp->ex_fsid); - WRITE64((u64)0); + p = xdr_encode_hyper(p, (u64)exp->ex_fsid); + p = xdr_encode_hyper(p, (u64)0); break; case FSIDSOURCE_DEV: - WRITE32(0); - WRITE32(MAJOR(stat.dev)); - WRITE32(0); - WRITE32(MINOR(stat.dev)); + *p++ = cpu_to_be32(0); + *p++ = cpu_to_be32(MAJOR(stat.dev)); + *p++ = cpu_to_be32(0); + *p++ = cpu_to_be32(MINOR(stat.dev)); break; case FSIDSOURCE_UUID: - WRITEMEM(exp->ex_uuid, 16); + p = xdr_encode_opaque_fixed(p, exp->ex_uuid, + EX_UUID_LEN); break; } } if (bmval0 & FATTR4_WORD0_UNIQUE_HANDLES) { - if ((buflen -= 4) < 0) + p = xdr_reserve_space(xdr, 4); + if (!p) goto out_resource; - WRITE32(0); + *p++ = cpu_to_be32(0); } if (bmval0 & FATTR4_WORD0_LEASE_TIME) { - if ((buflen -= 4) < 0) + p = xdr_reserve_space(xdr, 4); + if (!p) goto out_resource; - WRITE32(nn->nfsd4_lease); + *p++ = cpu_to_be32(nn->nfsd4_lease); } if (bmval0 & FATTR4_WORD0_RDATTR_ERROR) { - if ((buflen -= 4) < 0) + p = xdr_reserve_space(xdr, 4); + if (!p) goto out_resource; - WRITE32(rdattr_err); + *p++ = cpu_to_be32(rdattr_err); } if (bmval0 & FATTR4_WORD0_ACL) { struct nfs4_ace *ace; if (acl == NULL) { - if ((buflen -= 4) < 0) + p = xdr_reserve_space(xdr, 4); + if (!p) goto out_resource; - WRITE32(0); + *p++ = cpu_to_be32(0); goto out_acl; } - if ((buflen -= 4) < 0) + p = xdr_reserve_space(xdr, 4); + if (!p) goto out_resource; - WRITE32(acl->naces); + *p++ = cpu_to_be32(acl->naces); for (ace = acl->aces; ace < acl->aces + acl->naces; ace++) { - if ((buflen -= 4*3) < 0) + p = xdr_reserve_space(xdr, 4*3); + if (!p) goto out_resource; - WRITE32(ace->type); - WRITE32(ace->flag); - WRITE32(ace->access_mask & NFS4_ACE_MASK_ALL); - status = nfsd4_encode_aclname(rqstp, ace, &p, &buflen); + *p++ = cpu_to_be32(ace->type); + *p++ = cpu_to_be32(ace->flag); + *p++ = cpu_to_be32(ace->access_mask & + NFS4_ACE_MASK_ALL); + status = nfsd4_encode_aclname(xdr, rqstp, ace); if (status) goto out; } } out_acl: if (bmval0 & FATTR4_WORD0_ACLSUPPORT) { - if ((buflen -= 4) < 0) + p = xdr_reserve_space(xdr, 4); + if (!p) goto out_resource; - WRITE32(aclsupport ? + *p++ = cpu_to_be32(aclsupport ? ACL4_SUPPORT_ALLOW_ACL|ACL4_SUPPORT_DENY_ACL : 0); } if (bmval0 & FATTR4_WORD0_CANSETTIME) { - if ((buflen -= 4) < 0) + p = xdr_reserve_space(xdr, 4); + if (!p) goto out_resource; - WRITE32(1); + *p++ = cpu_to_be32(1); } if (bmval0 & FATTR4_WORD0_CASE_INSENSITIVE) { - if ((buflen -= 4) < 0) + p = xdr_reserve_space(xdr, 4); + if (!p) goto out_resource; - WRITE32(0); + *p++ = cpu_to_be32(0); } if (bmval0 & FATTR4_WORD0_CASE_PRESERVING) { - if ((buflen -= 4) < 0) + p = xdr_reserve_space(xdr, 4); + if (!p) goto out_resource; - WRITE32(1); + *p++ = cpu_to_be32(1); } if (bmval0 & FATTR4_WORD0_CHOWN_RESTRICTED) { - if ((buflen -= 4) < 0) + p = xdr_reserve_space(xdr, 4); + if (!p) goto out_resource; - WRITE32(1); + *p++ = cpu_to_be32(1); } if (bmval0 & FATTR4_WORD0_FILEHANDLE) { - buflen -= (XDR_QUADLEN(fhp->fh_handle.fh_size) << 2) + 4; - if (buflen < 0) + p = xdr_reserve_space(xdr, fhp->fh_handle.fh_size + 4); + if (!p) goto out_resource; - WRITE32(fhp->fh_handle.fh_size); - WRITEMEM(&fhp->fh_handle.fh_base, fhp->fh_handle.fh_size); + p = xdr_encode_opaque(p, &fhp->fh_handle.fh_base, + fhp->fh_handle.fh_size); } if (bmval0 & FATTR4_WORD0_FILEID) { - if ((buflen -= 8) < 0) + p = xdr_reserve_space(xdr, 8); + if (!p) goto out_resource; - WRITE64(stat.ino); + p = xdr_encode_hyper(p, stat.ino); } if (bmval0 & FATTR4_WORD0_FILES_AVAIL) { - if ((buflen -= 8) < 0) + p = xdr_reserve_space(xdr, 8); + if (!p) goto out_resource; - WRITE64((u64) statfs.f_ffree); + p = xdr_encode_hyper(p, (u64) statfs.f_ffree); } if (bmval0 & FATTR4_WORD0_FILES_FREE) { - if ((buflen -= 8) < 0) + p = xdr_reserve_space(xdr, 8); + if (!p) goto out_resource; - WRITE64((u64) statfs.f_ffree); + p = xdr_encode_hyper(p, (u64) statfs.f_ffree); } if (bmval0 & FATTR4_WORD0_FILES_TOTAL) { - if ((buflen -= 8) < 0) + p = xdr_reserve_space(xdr, 8); + if (!p) goto out_resource; - WRITE64((u64) statfs.f_files); + p = xdr_encode_hyper(p, (u64) statfs.f_files); } if (bmval0 & FATTR4_WORD0_FS_LOCATIONS) { - status = nfsd4_encode_fs_locations(rqstp, exp, &p, &buflen); + status = nfsd4_encode_fs_locations(xdr, rqstp, exp); if (status) goto out; } if (bmval0 & FATTR4_WORD0_HOMOGENEOUS) { - if ((buflen -= 4) < 0) + p = xdr_reserve_space(xdr, 4); + if (!p) goto out_resource; - WRITE32(1); + *p++ = cpu_to_be32(1); } if (bmval0 & FATTR4_WORD0_MAXFILESIZE) { - if ((buflen -= 8) < 0) + p = xdr_reserve_space(xdr, 8); + if (!p) goto out_resource; - WRITE64(exp->ex_path.mnt->mnt_sb->s_maxbytes); + p = xdr_encode_hyper(p, exp->ex_path.mnt->mnt_sb->s_maxbytes); } if (bmval0 & FATTR4_WORD0_MAXLINK) { - if ((buflen -= 4) < 0) + p = xdr_reserve_space(xdr, 4); + if (!p) goto out_resource; - WRITE32(255); + *p++ = cpu_to_be32(255); } if (bmval0 & FATTR4_WORD0_MAXNAME) { - if ((buflen -= 4) < 0) + p = xdr_reserve_space(xdr, 4); + if (!p) goto out_resource; - WRITE32(statfs.f_namelen); + *p++ = cpu_to_be32(statfs.f_namelen); } if (bmval0 & FATTR4_WORD0_MAXREAD) { - if ((buflen -= 8) < 0) + p = xdr_reserve_space(xdr, 8); + if (!p) goto out_resource; - WRITE64((u64) svc_max_payload(rqstp)); + p = xdr_encode_hyper(p, (u64) svc_max_payload(rqstp)); } if (bmval0 & FATTR4_WORD0_MAXWRITE) { - if ((buflen -= 8) < 0) + p = xdr_reserve_space(xdr, 8); + if (!p) goto out_resource; - WRITE64((u64) svc_max_payload(rqstp)); + p = xdr_encode_hyper(p, (u64) svc_max_payload(rqstp)); } if (bmval1 & FATTR4_WORD1_MODE) { - if ((buflen -= 4) < 0) + p = xdr_reserve_space(xdr, 4); + if (!p) goto out_resource; - WRITE32(stat.mode & S_IALLUGO); + *p++ = cpu_to_be32(stat.mode & S_IALLUGO); } if (bmval1 & FATTR4_WORD1_NO_TRUNC) { - if ((buflen -= 4) < 0) + p = xdr_reserve_space(xdr, 4); + if (!p) goto out_resource; - WRITE32(1); + *p++ = cpu_to_be32(1); } if (bmval1 & FATTR4_WORD1_NUMLINKS) { - if ((buflen -= 4) < 0) + p = xdr_reserve_space(xdr, 4); + if (!p) goto out_resource; - WRITE32(stat.nlink); + *p++ = cpu_to_be32(stat.nlink); } if (bmval1 & FATTR4_WORD1_OWNER) { - status = nfsd4_encode_user(rqstp, stat.uid, &p, &buflen); + status = nfsd4_encode_user(xdr, rqstp, stat.uid); if (status) goto out; } if (bmval1 & FATTR4_WORD1_OWNER_GROUP) { - status = nfsd4_encode_group(rqstp, stat.gid, &p, &buflen); + status = nfsd4_encode_group(xdr, rqstp, stat.gid); if (status) goto out; } if (bmval1 & FATTR4_WORD1_RAWDEV) { - if ((buflen -= 8) < 0) + p = xdr_reserve_space(xdr, 8); + if (!p) goto out_resource; - WRITE32((u32) MAJOR(stat.rdev)); - WRITE32((u32) MINOR(stat.rdev)); + *p++ = cpu_to_be32((u32) MAJOR(stat.rdev)); + *p++ = cpu_to_be32((u32) MINOR(stat.rdev)); } if (bmval1 & FATTR4_WORD1_SPACE_AVAIL) { - if ((buflen -= 8) < 0) + p = xdr_reserve_space(xdr, 8); + if (!p) goto out_resource; dummy64 = (u64)statfs.f_bavail * (u64)statfs.f_bsize; - WRITE64(dummy64); + p = xdr_encode_hyper(p, dummy64); } if (bmval1 & FATTR4_WORD1_SPACE_FREE) { - if ((buflen -= 8) < 0) + p = xdr_reserve_space(xdr, 8); + if (!p) goto out_resource; dummy64 = (u64)statfs.f_bfree * (u64)statfs.f_bsize; - WRITE64(dummy64); + p = xdr_encode_hyper(p, dummy64); } if (bmval1 & FATTR4_WORD1_SPACE_TOTAL) { - if ((buflen -= 8) < 0) + p = xdr_reserve_space(xdr, 8); + if (!p) goto out_resource; dummy64 = (u64)statfs.f_blocks * (u64)statfs.f_bsize; - WRITE64(dummy64); + p = xdr_encode_hyper(p, dummy64); } if (bmval1 & FATTR4_WORD1_SPACE_USED) { - if ((buflen -= 8) < 0) + p = xdr_reserve_space(xdr, 8); + if (!p) goto out_resource; dummy64 = (u64)stat.blocks << 9; - WRITE64(dummy64); + p = xdr_encode_hyper(p, dummy64); } if (bmval1 & FATTR4_WORD1_TIME_ACCESS) { - if ((buflen -= 12) < 0) + p = xdr_reserve_space(xdr, 12); + if (!p) goto out_resource; - WRITE64((s64)stat.atime.tv_sec); - WRITE32(stat.atime.tv_nsec); + p = xdr_encode_hyper(p, (s64)stat.atime.tv_sec); + *p++ = cpu_to_be32(stat.atime.tv_nsec); } if (bmval1 & FATTR4_WORD1_TIME_DELTA) { - if ((buflen -= 12) < 0) + p = xdr_reserve_space(xdr, 12); + if (!p) goto out_resource; - WRITE32(0); - WRITE32(1); - WRITE32(0); + *p++ = cpu_to_be32(0); + *p++ = cpu_to_be32(1); + *p++ = cpu_to_be32(0); } if (bmval1 & FATTR4_WORD1_TIME_METADATA) { - if ((buflen -= 12) < 0) + p = xdr_reserve_space(xdr, 12); + if (!p) goto out_resource; - WRITE64((s64)stat.ctime.tv_sec); - WRITE32(stat.ctime.tv_nsec); + p = xdr_encode_hyper(p, (s64)stat.ctime.tv_sec); + *p++ = cpu_to_be32(stat.ctime.tv_nsec); } if (bmval1 & FATTR4_WORD1_TIME_MODIFY) { - if ((buflen -= 12) < 0) + p = xdr_reserve_space(xdr, 12); + if (!p) goto out_resource; - WRITE64((s64)stat.mtime.tv_sec); - WRITE32(stat.mtime.tv_nsec); + p = xdr_encode_hyper(p, (s64)stat.mtime.tv_sec); + *p++ = cpu_to_be32(stat.mtime.tv_nsec); } if (bmval1 & FATTR4_WORD1_MOUNTED_ON_FILEID) { - if ((buflen -= 8) < 0) + p = xdr_reserve_space(xdr, 8); + if (!p) goto out_resource; /* * Get parent's attributes if not ignoring crossmount @@ -2473,25 +2480,26 @@ out_acl: if (ignore_crossmnt == 0 && dentry == exp->ex_path.mnt->mnt_root) get_parent_attributes(exp, &stat); - WRITE64(stat.ino); + p = xdr_encode_hyper(p, stat.ino); } if (bmval2 & FATTR4_WORD2_SECURITY_LABEL) { - status = nfsd4_encode_security_label(rqstp, context, - contextlen, &p, &buflen); + status = nfsd4_encode_security_label(xdr, rqstp, context, + contextlen); if (status) goto out; } if (bmval2 & FATTR4_WORD2_SUPPATTR_EXCLCREAT) { - if ((buflen -= 16) < 0) + p = xdr_reserve_space(xdr, 16); + if (!p) goto out_resource; - WRITE32(3); - WRITE32(NFSD_SUPPATTR_EXCLCREAT_WORD0); - WRITE32(NFSD_SUPPATTR_EXCLCREAT_WORD1); - WRITE32(NFSD_SUPPATTR_EXCLCREAT_WORD2); + *p++ = cpu_to_be32(3); + *p++ = cpu_to_be32(NFSD_SUPPATTR_EXCLCREAT_WORD0); + *p++ = cpu_to_be32(NFSD_SUPPATTR_EXCLCREAT_WORD1); + *p++ = cpu_to_be32(NFSD_SUPPATTR_EXCLCREAT_WORD2); } - *attrlenp = htonl((char *)p - (char *)attrlenp - 4); - *buffer = p; + attrlen = htonl(xdr->buf->len - attrlen_offset - 4); + write_bytes_to_xdr_buf(xdr->buf, attrlen_offset, &attrlen, 4); status = nfs_ok; out: @@ -2504,6 +2512,8 @@ out: fh_put(tempfh); kfree(tempfh); } + if (status) + xdr_truncate_encode(xdr, starting_len); return status; out_nfserr: status = nfserrno(err); @@ -2513,6 +2523,37 @@ out_resource: goto out; } +static void svcxdr_init_encode_from_buffer(struct xdr_stream *xdr, + struct xdr_buf *buf, __be32 *p, int bytes) +{ + xdr->scratch.iov_len = 0; + memset(buf, 0, sizeof(struct xdr_buf)); + buf->head[0].iov_base = p; + buf->head[0].iov_len = 0; + buf->len = 0; + xdr->buf = buf; + xdr->iov = buf->head; + xdr->p = p; + xdr->end = (void *)p + bytes; + buf->buflen = bytes; +} + +__be32 nfsd4_encode_fattr_to_buf(__be32 **p, int words, + struct svc_fh *fhp, struct svc_export *exp, + struct dentry *dentry, u32 *bmval, + struct svc_rqst *rqstp, int ignore_crossmnt) +{ + struct xdr_buf dummy; + struct xdr_stream xdr; + __be32 ret; + + svcxdr_init_encode_from_buffer(&xdr, &dummy, *p, words << 2); + ret = nfsd4_encode_fattr(&xdr, fhp, exp, dentry, bmval, rqstp, + ignore_crossmnt); + *p = xdr.p; + return ret; +} + static inline int attributes_need_mount(u32 *bmval) { if (bmval[0] & ~(FATTR4_WORD0_RDATTR_ERROR | FATTR4_WORD0_LEASE_TIME)) @@ -2523,8 +2564,8 @@ static inline int attributes_need_mount(u32 *bmval) } static __be32 -nfsd4_encode_dirent_fattr(struct nfsd4_readdir *cd, - const char *name, int namlen, __be32 **p, int buflen) +nfsd4_encode_dirent_fattr(struct xdr_stream *xdr, struct nfsd4_readdir *cd, + const char *name, int namlen) { struct svc_export *exp = cd->rd_fhp->fh_export; struct dentry *dentry; @@ -2576,7 +2617,7 @@ nfsd4_encode_dirent_fattr(struct nfsd4_readdir *cd, } out_encode: - nfserr = nfsd4_encode_fattr(NULL, exp, dentry, p, buflen, cd->rd_bmval, + nfserr = nfsd4_encode_fattr(xdr, NULL, exp, dentry, cd->rd_bmval, cd->rd_rqstp, ignore_crossmnt); out_put: dput(dentry); @@ -2585,9 +2626,12 @@ out_put: } static __be32 * -nfsd4_encode_rdattr_error(__be32 *p, int buflen, __be32 nfserr) +nfsd4_encode_rdattr_error(struct xdr_stream *xdr, __be32 nfserr) { - if (buflen < 6) + __be32 *p; + + p = xdr_reserve_space(xdr, 6); + if (!p) return NULL; *p++ = htonl(2); *p++ = htonl(FATTR4_WORD0_RDATTR_ERROR); /* bmval0 */ @@ -2604,10 +2648,13 @@ nfsd4_encode_dirent(void *ccdv, const char *name, int namlen, { struct readdir_cd *ccd = ccdv; struct nfsd4_readdir *cd = container_of(ccd, struct nfsd4_readdir, common); - int buflen; - __be32 *p = cd->buffer; - __be32 *cookiep; + struct xdr_stream *xdr = cd->xdr; + int start_offset = xdr->buf->len; + int cookie_offset; + int entry_bytes; __be32 nfserr = nfserr_toosmall; + __be64 wire_offset; + __be32 *p; /* In nfsv4, "." and ".." never make it onto the wire.. */ if (name && isdotent(name, namlen)) { @@ -2615,19 +2662,24 @@ nfsd4_encode_dirent(void *ccdv, const char *name, int namlen, return 0; } - if (cd->offset) - xdr_encode_hyper(cd->offset, (u64) offset); + if (cd->cookie_offset) { + wire_offset = cpu_to_be64(offset); + write_bytes_to_xdr_buf(xdr->buf, cd->cookie_offset, + &wire_offset, 8); + } - buflen = cd->buflen - 4 - XDR_QUADLEN(namlen); - if (buflen < 0) + p = xdr_reserve_space(xdr, 4); + if (!p) goto fail; - *p++ = xdr_one; /* mark entry present */ - cookiep = p; + cookie_offset = xdr->buf->len; + p = xdr_reserve_space(xdr, 3*4 + namlen); + if (!p) + goto fail; p = xdr_encode_hyper(p, NFS_OFFSET_MAX); /* offset of next entry */ p = xdr_encode_array(p, name, namlen); /* name length & name */ - nfserr = nfsd4_encode_dirent_fattr(cd, name, namlen, &p, buflen); + nfserr = nfsd4_encode_dirent_fattr(xdr, cd, name, namlen); switch (nfserr) { case nfs_ok: break; @@ -2646,59 +2698,74 @@ nfsd4_encode_dirent(void *ccdv, const char *name, int namlen, */ if (!(cd->rd_bmval[0] & FATTR4_WORD0_RDATTR_ERROR)) goto fail; - p = nfsd4_encode_rdattr_error(p, buflen, nfserr); + p = nfsd4_encode_rdattr_error(xdr, nfserr); if (p == NULL) { nfserr = nfserr_toosmall; goto fail; } } - cd->buflen -= (p - cd->buffer); - cd->buffer = p; - cd->offset = cookiep; + nfserr = nfserr_toosmall; + entry_bytes = xdr->buf->len - start_offset; + if (entry_bytes > cd->rd_maxcount) + goto fail; + cd->rd_maxcount -= entry_bytes; + if (!cd->rd_dircount) + goto fail; + cd->rd_dircount--; + cd->cookie_offset = cookie_offset; skip_entry: cd->common.err = nfs_ok; return 0; fail: + xdr_truncate_encode(xdr, start_offset); cd->common.err = nfserr; return -EINVAL; } -static void -nfsd4_encode_stateid(struct nfsd4_compoundres *resp, stateid_t *sid) +static __be32 +nfsd4_encode_stateid(struct xdr_stream *xdr, stateid_t *sid) { __be32 *p; - RESERVE_SPACE(sizeof(stateid_t)); - WRITE32(sid->si_generation); - WRITEMEM(&sid->si_opaque, sizeof(stateid_opaque_t)); - ADJUST_ARGS(); + p = xdr_reserve_space(xdr, sizeof(stateid_t)); + if (!p) + return nfserr_resource; + *p++ = cpu_to_be32(sid->si_generation); + p = xdr_encode_opaque_fixed(p, &sid->si_opaque, + sizeof(stateid_opaque_t)); + return 0; } static __be32 nfsd4_encode_access(struct nfsd4_compoundres *resp, __be32 nfserr, struct nfsd4_access *access) { + struct xdr_stream *xdr = &resp->xdr; __be32 *p; if (!nfserr) { - RESERVE_SPACE(8); - WRITE32(access->ac_supported); - WRITE32(access->ac_resp_access); - ADJUST_ARGS(); + p = xdr_reserve_space(xdr, 8); + if (!p) + return nfserr_resource; + *p++ = cpu_to_be32(access->ac_supported); + *p++ = cpu_to_be32(access->ac_resp_access); } return nfserr; } static __be32 nfsd4_encode_bind_conn_to_session(struct nfsd4_compoundres *resp, __be32 nfserr, struct nfsd4_bind_conn_to_session *bcts) { + struct xdr_stream *xdr = &resp->xdr; __be32 *p; if (!nfserr) { - RESERVE_SPACE(NFS4_MAX_SESSIONID_LEN + 8); - WRITEMEM(bcts->sessionid.data, NFS4_MAX_SESSIONID_LEN); - WRITE32(bcts->dir); + p = xdr_reserve_space(xdr, NFS4_MAX_SESSIONID_LEN + 8); + if (!p) + return nfserr_resource; + p = xdr_encode_opaque_fixed(p, bcts->sessionid.data, + NFS4_MAX_SESSIONID_LEN); + *p++ = cpu_to_be32(bcts->dir); /* Sorry, we do not yet support RDMA over 4.1: */ - WRITE32(0); - ADJUST_ARGS(); + *p++ = cpu_to_be32(0); } return nfserr; } @@ -2706,8 +2773,10 @@ static __be32 nfsd4_encode_bind_conn_to_session(struct nfsd4_compoundres *resp, static __be32 nfsd4_encode_close(struct nfsd4_compoundres *resp, __be32 nfserr, struct nfsd4_close *close) { + struct xdr_stream *xdr = &resp->xdr; + if (!nfserr) - nfsd4_encode_stateid(resp, &close->cl_stateid); + nfserr = nfsd4_encode_stateid(xdr, &close->cl_stateid); return nfserr; } @@ -2716,12 +2785,15 @@ nfsd4_encode_close(struct nfsd4_compoundres *resp, __be32 nfserr, struct nfsd4_c static __be32 nfsd4_encode_commit(struct nfsd4_compoundres *resp, __be32 nfserr, struct nfsd4_commit *commit) { + struct xdr_stream *xdr = &resp->xdr; __be32 *p; if (!nfserr) { - RESERVE_SPACE(NFS4_VERIFIER_SIZE); - WRITEMEM(commit->co_verf.data, NFS4_VERIFIER_SIZE); - ADJUST_ARGS(); + p = xdr_reserve_space(xdr, NFS4_VERIFIER_SIZE); + if (!p) + return nfserr_resource; + p = xdr_encode_opaque_fixed(p, commit->co_verf.data, + NFS4_VERIFIER_SIZE); } return nfserr; } @@ -2729,15 +2801,17 @@ nfsd4_encode_commit(struct nfsd4_compoundres *resp, __be32 nfserr, struct nfsd4_ static __be32 nfsd4_encode_create(struct nfsd4_compoundres *resp, __be32 nfserr, struct nfsd4_create *create) { + struct xdr_stream *xdr = &resp->xdr; __be32 *p; if (!nfserr) { - RESERVE_SPACE(32); - write_cinfo(&p, &create->cr_cinfo); - WRITE32(2); - WRITE32(create->cr_bmval[0]); - WRITE32(create->cr_bmval[1]); - ADJUST_ARGS(); + p = xdr_reserve_space(xdr, 32); + if (!p) + return nfserr_resource; + p = encode_cinfo(p, &create->cr_cinfo); + *p++ = cpu_to_be32(2); + *p++ = cpu_to_be32(create->cr_bmval[0]); + *p++ = cpu_to_be32(create->cr_bmval[1]); } return nfserr; } @@ -2746,14 +2820,13 @@ static __be32 nfsd4_encode_getattr(struct nfsd4_compoundres *resp, __be32 nfserr, struct nfsd4_getattr *getattr) { struct svc_fh *fhp = getattr->ga_fhp; - int buflen; + struct xdr_stream *xdr = &resp->xdr; if (nfserr) return nfserr; - buflen = resp->end - resp->p - (COMPOUND_ERR_SLACK_SPACE >> 2); - nfserr = nfsd4_encode_fattr(fhp, fhp->fh_export, fhp->fh_dentry, - &resp->p, buflen, getattr->ga_bmval, + nfserr = nfsd4_encode_fattr(xdr, fhp, fhp->fh_export, fhp->fh_dentry, + getattr->ga_bmval, resp->rqstp, 0); return nfserr; } @@ -2761,16 +2834,17 @@ nfsd4_encode_getattr(struct nfsd4_compoundres *resp, __be32 nfserr, struct nfsd4 static __be32 nfsd4_encode_getfh(struct nfsd4_compoundres *resp, __be32 nfserr, struct svc_fh **fhpp) { + struct xdr_stream *xdr = &resp->xdr; struct svc_fh *fhp = *fhpp; unsigned int len; __be32 *p; if (!nfserr) { len = fhp->fh_handle.fh_size; - RESERVE_SPACE(len + 4); - WRITE32(len); - WRITEMEM(&fhp->fh_handle.fh_base, len); - ADJUST_ARGS(); + p = xdr_reserve_space(xdr, len + 4); + if (!p) + return nfserr_resource; + p = xdr_encode_opaque(p, &fhp->fh_handle.fh_base, len); } return nfserr; } @@ -2779,52 +2853,69 @@ nfsd4_encode_getfh(struct nfsd4_compoundres *resp, __be32 nfserr, struct svc_fh * Including all fields other than the name, a LOCK4denied structure requires * 8(clientid) + 4(namelen) + 8(offset) + 8(length) + 4(type) = 32 bytes. */ -static void -nfsd4_encode_lock_denied(struct nfsd4_compoundres *resp, struct nfsd4_lock_denied *ld) +static __be32 +nfsd4_encode_lock_denied(struct xdr_stream *xdr, struct nfsd4_lock_denied *ld) { struct xdr_netobj *conf = &ld->ld_owner; __be32 *p; - RESERVE_SPACE(32 + XDR_LEN(conf->len)); - WRITE64(ld->ld_start); - WRITE64(ld->ld_length); - WRITE32(ld->ld_type); +again: + p = xdr_reserve_space(xdr, 32 + XDR_LEN(conf->len)); + if (!p) { + /* + * Don't fail to return the result just because we can't + * return the conflicting open: + */ + if (conf->len) { + conf->len = 0; + conf->data = NULL; + goto again; + } + return nfserr_resource; + } + p = xdr_encode_hyper(p, ld->ld_start); + p = xdr_encode_hyper(p, ld->ld_length); + *p++ = cpu_to_be32(ld->ld_type); if (conf->len) { - WRITEMEM(&ld->ld_clientid, 8); - WRITE32(conf->len); - WRITEMEM(conf->data, conf->len); - kfree(conf->data); + p = xdr_encode_opaque_fixed(p, &ld->ld_clientid, 8); + p = xdr_encode_opaque(p, conf->data, conf->len); } else { /* non - nfsv4 lock in conflict, no clientid nor owner */ - WRITE64((u64)0); /* clientid */ - WRITE32(0); /* length of owner name */ + p = xdr_encode_hyper(p, (u64)0); /* clientid */ + *p++ = cpu_to_be32(0); /* length of owner name */ } - ADJUST_ARGS(); + return nfserr_denied; } static __be32 nfsd4_encode_lock(struct nfsd4_compoundres *resp, __be32 nfserr, struct nfsd4_lock *lock) { + struct xdr_stream *xdr = &resp->xdr; + if (!nfserr) - nfsd4_encode_stateid(resp, &lock->lk_resp_stateid); + nfserr = nfsd4_encode_stateid(xdr, &lock->lk_resp_stateid); else if (nfserr == nfserr_denied) - nfsd4_encode_lock_denied(resp, &lock->lk_denied); - + nfserr = nfsd4_encode_lock_denied(xdr, &lock->lk_denied); + kfree(lock->lk_denied.ld_owner.data); return nfserr; } static __be32 nfsd4_encode_lockt(struct nfsd4_compoundres *resp, __be32 nfserr, struct nfsd4_lockt *lockt) { + struct xdr_stream *xdr = &resp->xdr; + if (nfserr == nfserr_denied) - nfsd4_encode_lock_denied(resp, &lockt->lt_denied); + nfsd4_encode_lock_denied(xdr, &lockt->lt_denied); return nfserr; } static __be32 nfsd4_encode_locku(struct nfsd4_compoundres *resp, __be32 nfserr, struct nfsd4_locku *locku) { + struct xdr_stream *xdr = &resp->xdr; + if (!nfserr) - nfsd4_encode_stateid(resp, &locku->lu_stateid); + nfserr = nfsd4_encode_stateid(xdr, &locku->lu_stateid); return nfserr; } @@ -2833,12 +2924,14 @@ nfsd4_encode_locku(struct nfsd4_compoundres *resp, __be32 nfserr, struct nfsd4_l static __be32 nfsd4_encode_link(struct nfsd4_compoundres *resp, __be32 nfserr, struct nfsd4_link *link) { + struct xdr_stream *xdr = &resp->xdr; __be32 *p; if (!nfserr) { - RESERVE_SPACE(20); - write_cinfo(&p, &link->li_cinfo); - ADJUST_ARGS(); + p = xdr_reserve_space(xdr, 20); + if (!p) + return nfserr_resource; + p = encode_cinfo(p, &link->li_cinfo); } return nfserr; } @@ -2847,72 +2940,86 @@ nfsd4_encode_link(struct nfsd4_compoundres *resp, __be32 nfserr, struct nfsd4_li static __be32 nfsd4_encode_open(struct nfsd4_compoundres *resp, __be32 nfserr, struct nfsd4_open *open) { + struct xdr_stream *xdr = &resp->xdr; __be32 *p; if (nfserr) goto out; - nfsd4_encode_stateid(resp, &open->op_stateid); - RESERVE_SPACE(40); - write_cinfo(&p, &open->op_cinfo); - WRITE32(open->op_rflags); - WRITE32(2); - WRITE32(open->op_bmval[0]); - WRITE32(open->op_bmval[1]); - WRITE32(open->op_delegate_type); - ADJUST_ARGS(); + nfserr = nfsd4_encode_stateid(xdr, &open->op_stateid); + if (nfserr) + goto out; + p = xdr_reserve_space(xdr, 40); + if (!p) + return nfserr_resource; + p = encode_cinfo(p, &open->op_cinfo); + *p++ = cpu_to_be32(open->op_rflags); + *p++ = cpu_to_be32(2); + *p++ = cpu_to_be32(open->op_bmval[0]); + *p++ = cpu_to_be32(open->op_bmval[1]); + *p++ = cpu_to_be32(open->op_delegate_type); switch (open->op_delegate_type) { case NFS4_OPEN_DELEGATE_NONE: break; case NFS4_OPEN_DELEGATE_READ: - nfsd4_encode_stateid(resp, &open->op_delegate_stateid); - RESERVE_SPACE(20); - WRITE32(open->op_recall); + nfserr = nfsd4_encode_stateid(xdr, &open->op_delegate_stateid); + if (nfserr) + return nfserr; + p = xdr_reserve_space(xdr, 20); + if (!p) + return nfserr_resource; + *p++ = cpu_to_be32(open->op_recall); /* * TODO: ACE's in delegations */ - WRITE32(NFS4_ACE_ACCESS_ALLOWED_ACE_TYPE); - WRITE32(0); - WRITE32(0); - WRITE32(0); /* XXX: is NULL principal ok? */ - ADJUST_ARGS(); + *p++ = cpu_to_be32(NFS4_ACE_ACCESS_ALLOWED_ACE_TYPE); + *p++ = cpu_to_be32(0); + *p++ = cpu_to_be32(0); + *p++ = cpu_to_be32(0); /* XXX: is NULL principal ok? */ break; case NFS4_OPEN_DELEGATE_WRITE: - nfsd4_encode_stateid(resp, &open->op_delegate_stateid); - RESERVE_SPACE(32); - WRITE32(0); + nfserr = nfsd4_encode_stateid(xdr, &open->op_delegate_stateid); + if (nfserr) + return nfserr; + p = xdr_reserve_space(xdr, 32); + if (!p) + return nfserr_resource; + *p++ = cpu_to_be32(0); /* * TODO: space_limit's in delegations */ - WRITE32(NFS4_LIMIT_SIZE); - WRITE32(~(u32)0); - WRITE32(~(u32)0); + *p++ = cpu_to_be32(NFS4_LIMIT_SIZE); + *p++ = cpu_to_be32(~(u32)0); + *p++ = cpu_to_be32(~(u32)0); /* * TODO: ACE's in delegations */ - WRITE32(NFS4_ACE_ACCESS_ALLOWED_ACE_TYPE); - WRITE32(0); - WRITE32(0); - WRITE32(0); /* XXX: is NULL principal ok? */ - ADJUST_ARGS(); + *p++ = cpu_to_be32(NFS4_ACE_ACCESS_ALLOWED_ACE_TYPE); + *p++ = cpu_to_be32(0); + *p++ = cpu_to_be32(0); + *p++ = cpu_to_be32(0); /* XXX: is NULL principal ok? */ break; case NFS4_OPEN_DELEGATE_NONE_EXT: /* 4.1 */ switch (open->op_why_no_deleg) { case WND4_CONTENTION: case WND4_RESOURCE: - RESERVE_SPACE(8); - WRITE32(open->op_why_no_deleg); - WRITE32(0); /* deleg signaling not supported yet */ + p = xdr_reserve_space(xdr, 8); + if (!p) + return nfserr_resource; + *p++ = cpu_to_be32(open->op_why_no_deleg); + /* deleg signaling not supported yet: */ + *p++ = cpu_to_be32(0); break; default: - RESERVE_SPACE(4); - WRITE32(open->op_why_no_deleg); + p = xdr_reserve_space(xdr, 4); + if (!p) + return nfserr_resource; + *p++ = cpu_to_be32(open->op_why_no_deleg); } - ADJUST_ARGS(); break; default: BUG(); @@ -2925,8 +3032,10 @@ out: static __be32 nfsd4_encode_open_confirm(struct nfsd4_compoundres *resp, __be32 nfserr, struct nfsd4_open_confirm *oc) { + struct xdr_stream *xdr = &resp->xdr; + if (!nfserr) - nfsd4_encode_stateid(resp, &oc->oc_resp_stateid); + nfserr = nfsd4_encode_stateid(xdr, &oc->oc_resp_stateid); return nfserr; } @@ -2934,127 +3043,233 @@ nfsd4_encode_open_confirm(struct nfsd4_compoundres *resp, __be32 nfserr, struct static __be32 nfsd4_encode_open_downgrade(struct nfsd4_compoundres *resp, __be32 nfserr, struct nfsd4_open_downgrade *od) { + struct xdr_stream *xdr = &resp->xdr; + if (!nfserr) - nfsd4_encode_stateid(resp, &od->od_stateid); + nfserr = nfsd4_encode_stateid(xdr, &od->od_stateid); return nfserr; } -static __be32 -nfsd4_encode_read(struct nfsd4_compoundres *resp, __be32 nfserr, - struct nfsd4_read *read) +static __be32 nfsd4_encode_splice_read( + struct nfsd4_compoundres *resp, + struct nfsd4_read *read, + struct file *file, unsigned long maxcount) { + struct xdr_stream *xdr = &resp->xdr; + struct xdr_buf *buf = xdr->buf; u32 eof; - int v; - struct page *page; - unsigned long maxcount; - long len; - __be32 *p; + int space_left; + __be32 nfserr; + __be32 *p = xdr->p - 2; - if (nfserr) - return nfserr; - if (resp->xbuf->page_len) + /* + * Don't inline pages unless we know there's room for eof, + * count, and possible padding: + */ + if (xdr->end - xdr->p < 3) return nfserr_resource; - RESERVE_SPACE(8); /* eof flag and byte count */ + nfserr = nfsd_splice_read(read->rd_rqstp, file, + read->rd_offset, &maxcount); + if (nfserr) { + /* + * nfsd_splice_actor may have already messed with the + * page length; reset it so as not to confuse + * xdr_truncate_encode: + */ + buf->page_len = 0; + return nfserr; + } - maxcount = svc_max_payload(resp->rqstp); - if (maxcount > read->rd_length) - maxcount = read->rd_length; + eof = (read->rd_offset + maxcount >= + read->rd_fhp->fh_dentry->d_inode->i_size); + + *(p++) = htonl(eof); + *(p++) = htonl(maxcount); + + buf->page_len = maxcount; + buf->len += maxcount; + xdr->page_ptr += (maxcount + PAGE_SIZE - 1) / PAGE_SIZE; + + /* Use rest of head for padding and remaining ops: */ + buf->tail[0].iov_base = xdr->p; + buf->tail[0].iov_len = 0; + xdr->iov = buf->tail; + if (maxcount&3) { + int pad = 4 - (maxcount&3); + + *(xdr->p++) = 0; + + buf->tail[0].iov_base += maxcount&3; + buf->tail[0].iov_len = pad; + buf->len += pad; + } + + space_left = min_t(int, (void *)xdr->end - (void *)xdr->p, + buf->buflen - buf->len); + buf->buflen = buf->len + space_left; + xdr->end = (__be32 *)((void *)xdr->end + space_left); + + return 0; +} + +static __be32 nfsd4_encode_readv(struct nfsd4_compoundres *resp, + struct nfsd4_read *read, + struct file *file, unsigned long maxcount) +{ + struct xdr_stream *xdr = &resp->xdr; + u32 eof; + int v; + int starting_len = xdr->buf->len - 8; + long len; + int thislen; + __be32 nfserr; + __be32 tmp; + __be32 *p; + u32 zzz = 0; + int pad; len = maxcount; v = 0; - while (len > 0) { - page = *(resp->rqstp->rq_next_page); - if (!page) { /* ran out of pages */ - maxcount -= len; - break; - } - resp->rqstp->rq_vec[v].iov_base = page_address(page); - resp->rqstp->rq_vec[v].iov_len = - len < PAGE_SIZE ? len : PAGE_SIZE; - resp->rqstp->rq_next_page++; + + thislen = (void *)xdr->end - (void *)xdr->p; + if (len < thislen) + thislen = len; + p = xdr_reserve_space(xdr, (thislen+3)&~3); + WARN_ON_ONCE(!p); + resp->rqstp->rq_vec[v].iov_base = p; + resp->rqstp->rq_vec[v].iov_len = thislen; + v++; + len -= thislen; + + while (len) { + thislen = min_t(long, len, PAGE_SIZE); + p = xdr_reserve_space(xdr, (thislen+3)&~3); + WARN_ON_ONCE(!p); + resp->rqstp->rq_vec[v].iov_base = p; + resp->rqstp->rq_vec[v].iov_len = thislen; v++; - len -= PAGE_SIZE; + len -= thislen; } read->rd_vlen = v; - nfserr = nfsd_read_file(read->rd_rqstp, read->rd_fhp, read->rd_filp, - read->rd_offset, resp->rqstp->rq_vec, read->rd_vlen, - &maxcount); - + nfserr = nfsd_readv(file, read->rd_offset, resp->rqstp->rq_vec, + read->rd_vlen, &maxcount); if (nfserr) return nfserr; + xdr_truncate_encode(xdr, starting_len + 8 + ((maxcount+3)&~3)); + eof = (read->rd_offset + maxcount >= read->rd_fhp->fh_dentry->d_inode->i_size); - WRITE32(eof); - WRITE32(maxcount); - ADJUST_ARGS(); - resp->xbuf->head[0].iov_len = (char*)p - - (char*)resp->xbuf->head[0].iov_base; - resp->xbuf->page_len = maxcount; + tmp = htonl(eof); + write_bytes_to_xdr_buf(xdr->buf, starting_len , &tmp, 4); + tmp = htonl(maxcount); + write_bytes_to_xdr_buf(xdr->buf, starting_len + 4, &tmp, 4); - /* Use rest of head for padding and remaining ops: */ - resp->xbuf->tail[0].iov_base = p; - resp->xbuf->tail[0].iov_len = 0; - if (maxcount&3) { - RESERVE_SPACE(4); - WRITE32(0); - resp->xbuf->tail[0].iov_base += maxcount&3; - resp->xbuf->tail[0].iov_len = 4 - (maxcount&3); - ADJUST_ARGS(); - } + pad = (maxcount&3) ? 4 - (maxcount&3) : 0; + write_bytes_to_xdr_buf(xdr->buf, starting_len + 8 + maxcount, + &zzz, pad); return 0; + } static __be32 -nfsd4_encode_readlink(struct nfsd4_compoundres *resp, __be32 nfserr, struct nfsd4_readlink *readlink) +nfsd4_encode_read(struct nfsd4_compoundres *resp, __be32 nfserr, + struct nfsd4_read *read) { - int maxcount; - char *page; + unsigned long maxcount; + struct xdr_stream *xdr = &resp->xdr; + struct file *file = read->rd_filp; + int starting_len = xdr->buf->len; + struct raparms *ra; __be32 *p; + __be32 err; if (nfserr) return nfserr; - if (resp->xbuf->page_len) + + p = xdr_reserve_space(xdr, 8); /* eof flag and byte count */ + if (!p) { + WARN_ON_ONCE(resp->rqstp->rq_splice_ok); return nfserr_resource; - if (!*resp->rqstp->rq_next_page) + } + if (resp->xdr.buf->page_len && resp->rqstp->rq_splice_ok) { + WARN_ON_ONCE(1); return nfserr_resource; + } + xdr_commit_encode(xdr); + + maxcount = svc_max_payload(resp->rqstp); + if (maxcount > xdr->buf->buflen - xdr->buf->len) + maxcount = xdr->buf->buflen - xdr->buf->len; + if (maxcount > read->rd_length) + maxcount = read->rd_length; + + if (!read->rd_filp) { + err = nfsd_get_tmp_read_open(resp->rqstp, read->rd_fhp, + &file, &ra); + if (err) + goto err_truncate; + } + + if (file->f_op->splice_read && resp->rqstp->rq_splice_ok) + err = nfsd4_encode_splice_read(resp, read, file, maxcount); + else + err = nfsd4_encode_readv(resp, read, file, maxcount); + + if (!read->rd_filp) + nfsd_put_tmp_read_open(file, ra); + +err_truncate: + if (err) + xdr_truncate_encode(xdr, starting_len); + return err; +} + +static __be32 +nfsd4_encode_readlink(struct nfsd4_compoundres *resp, __be32 nfserr, struct nfsd4_readlink *readlink) +{ + int maxcount; + __be32 wire_count; + int zero = 0; + struct xdr_stream *xdr = &resp->xdr; + int length_offset = xdr->buf->len; + __be32 *p; - page = page_address(*(resp->rqstp->rq_next_page++)); + if (nfserr) + return nfserr; + p = xdr_reserve_space(xdr, 4); + if (!p) + return nfserr_resource; maxcount = PAGE_SIZE; - RESERVE_SPACE(4); + p = xdr_reserve_space(xdr, maxcount); + if (!p) + return nfserr_resource; /* * XXX: By default, the ->readlink() VFS op will truncate symlinks * if they would overflow the buffer. Is this kosher in NFSv4? If * not, one easy fix is: if ->readlink() precisely fills the buffer, * assume that truncation occurred, and return NFS4ERR_RESOURCE. */ - nfserr = nfsd_readlink(readlink->rl_rqstp, readlink->rl_fhp, page, &maxcount); + nfserr = nfsd_readlink(readlink->rl_rqstp, readlink->rl_fhp, + (char *)p, &maxcount); if (nfserr == nfserr_isdir) - return nfserr_inval; - if (nfserr) + nfserr = nfserr_inval; + if (nfserr) { + xdr_truncate_encode(xdr, length_offset); return nfserr; - - WRITE32(maxcount); - ADJUST_ARGS(); - resp->xbuf->head[0].iov_len = (char*)p - - (char*)resp->xbuf->head[0].iov_base; - resp->xbuf->page_len = maxcount; - - /* Use rest of head for padding and remaining ops: */ - resp->xbuf->tail[0].iov_base = p; - resp->xbuf->tail[0].iov_len = 0; - if (maxcount&3) { - RESERVE_SPACE(4); - WRITE32(0); - resp->xbuf->tail[0].iov_base += maxcount&3; - resp->xbuf->tail[0].iov_len = 4 - (maxcount&3); - ADJUST_ARGS(); } + + wire_count = htonl(maxcount); + write_bytes_to_xdr_buf(xdr->buf, length_offset, &wire_count, 4); + xdr_truncate_encode(xdr, length_offset + 4 + maxcount); + if (maxcount & 3) + write_bytes_to_xdr_buf(xdr->buf, length_offset + 4 + maxcount, + &zero, 4 - (maxcount&3)); return 0; } @@ -3062,47 +3277,52 @@ static __be32 nfsd4_encode_readdir(struct nfsd4_compoundres *resp, __be32 nfserr, struct nfsd4_readdir *readdir) { int maxcount; + int bytes_left; loff_t offset; - __be32 *page, *savep, *tailbase; + __be64 wire_offset; + struct xdr_stream *xdr = &resp->xdr; + int starting_len = xdr->buf->len; __be32 *p; if (nfserr) return nfserr; - if (resp->xbuf->page_len) - return nfserr_resource; - if (!*resp->rqstp->rq_next_page) - return nfserr_resource; - RESERVE_SPACE(NFS4_VERIFIER_SIZE); - savep = p; + p = xdr_reserve_space(xdr, NFS4_VERIFIER_SIZE); + if (!p) + return nfserr_resource; /* XXX: Following NFSv3, we ignore the READDIR verifier for now. */ - WRITE32(0); - WRITE32(0); - ADJUST_ARGS(); - resp->xbuf->head[0].iov_len = ((char*)resp->p) - (char*)resp->xbuf->head[0].iov_base; - tailbase = p; - - maxcount = PAGE_SIZE; - if (maxcount > readdir->rd_maxcount) - maxcount = readdir->rd_maxcount; + *p++ = cpu_to_be32(0); + *p++ = cpu_to_be32(0); + resp->xdr.buf->head[0].iov_len = ((char *)resp->xdr.p) + - (char *)resp->xdr.buf->head[0].iov_base; /* - * Convert from bytes to words, account for the two words already - * written, make sure to leave two words at the end for the next - * pointer and eof field. + * Number of bytes left for directory entries allowing for the + * final 8 bytes of the readdir and a following failed op: + */ + bytes_left = xdr->buf->buflen - xdr->buf->len + - COMPOUND_ERR_SLACK_SPACE - 8; + if (bytes_left < 0) { + nfserr = nfserr_resource; + goto err_no_verf; + } + maxcount = min_t(u32, readdir->rd_maxcount, INT_MAX); + /* + * Note the rfc defines rd_maxcount as the size of the + * READDIR4resok structure, which includes the verifier above + * and the 8 bytes encoded at the end of this function: */ - maxcount = (maxcount >> 2) - 4; - if (maxcount < 0) { - nfserr = nfserr_toosmall; + if (maxcount < 16) { + nfserr = nfserr_toosmall; goto err_no_verf; } + maxcount = min_t(int, maxcount-16, bytes_left); - page = page_address(*(resp->rqstp->rq_next_page++)); + readdir->xdr = xdr; + readdir->rd_maxcount = maxcount; readdir->common.err = 0; - readdir->buflen = maxcount; - readdir->buffer = page; - readdir->offset = NULL; + readdir->cookie_offset = 0; offset = readdir->rd_cookie; nfserr = nfsd_readdir(readdir->rd_rqstp, readdir->rd_fhp, @@ -3110,42 +3330,49 @@ nfsd4_encode_readdir(struct nfsd4_compoundres *resp, __be32 nfserr, struct nfsd4 &readdir->common, nfsd4_encode_dirent); if (nfserr == nfs_ok && readdir->common.err == nfserr_toosmall && - readdir->buffer == page) - nfserr = nfserr_toosmall; + xdr->buf->len == starting_len + 8) { + /* nothing encoded; which limit did we hit?: */ + if (maxcount - 16 < bytes_left) + /* It was the fault of rd_maxcount: */ + nfserr = nfserr_toosmall; + else + /* We ran out of buffer space: */ + nfserr = nfserr_resource; + } if (nfserr) goto err_no_verf; - if (readdir->offset) - xdr_encode_hyper(readdir->offset, offset); + if (readdir->cookie_offset) { + wire_offset = cpu_to_be64(offset); + write_bytes_to_xdr_buf(xdr->buf, readdir->cookie_offset, + &wire_offset, 8); + } - p = readdir->buffer; + p = xdr_reserve_space(xdr, 8); + if (!p) { + WARN_ON_ONCE(1); + goto err_no_verf; + } *p++ = 0; /* no more entries */ *p++ = htonl(readdir->common.err == nfserr_eof); - resp->xbuf->page_len = ((char*)p) - - (char*)page_address(*(resp->rqstp->rq_next_page-1)); - - /* Use rest of head for padding and remaining ops: */ - resp->xbuf->tail[0].iov_base = tailbase; - resp->xbuf->tail[0].iov_len = 0; - resp->p = resp->xbuf->tail[0].iov_base; - resp->end = resp->p + (PAGE_SIZE - resp->xbuf->head[0].iov_len)/4; return 0; err_no_verf: - p = savep; - ADJUST_ARGS(); + xdr_truncate_encode(xdr, starting_len); return nfserr; } static __be32 nfsd4_encode_remove(struct nfsd4_compoundres *resp, __be32 nfserr, struct nfsd4_remove *remove) { + struct xdr_stream *xdr = &resp->xdr; __be32 *p; if (!nfserr) { - RESERVE_SPACE(20); - write_cinfo(&p, &remove->rm_cinfo); - ADJUST_ARGS(); + p = xdr_reserve_space(xdr, 20); + if (!p) + return nfserr_resource; + p = encode_cinfo(p, &remove->rm_cinfo); } return nfserr; } @@ -3153,19 +3380,21 @@ nfsd4_encode_remove(struct nfsd4_compoundres *resp, __be32 nfserr, struct nfsd4_ static __be32 nfsd4_encode_rename(struct nfsd4_compoundres *resp, __be32 nfserr, struct nfsd4_rename *rename) { + struct xdr_stream *xdr = &resp->xdr; __be32 *p; if (!nfserr) { - RESERVE_SPACE(40); - write_cinfo(&p, &rename->rn_sinfo); - write_cinfo(&p, &rename->rn_tinfo); - ADJUST_ARGS(); + p = xdr_reserve_space(xdr, 40); + if (!p) + return nfserr_resource; + p = encode_cinfo(p, &rename->rn_sinfo); + p = encode_cinfo(p, &rename->rn_tinfo); } return nfserr; } static __be32 -nfsd4_do_encode_secinfo(struct nfsd4_compoundres *resp, +nfsd4_do_encode_secinfo(struct xdr_stream *xdr, __be32 nfserr, struct svc_export *exp) { u32 i, nflavs, supported; @@ -3176,6 +3405,7 @@ nfsd4_do_encode_secinfo(struct nfsd4_compoundres *resp, if (nfserr) goto out; + nfserr = nfserr_resource; if (exp->ex_nflavors) { flavs = exp->ex_flavors; nflavs = exp->ex_nflavors; @@ -3197,9 +3427,10 @@ nfsd4_do_encode_secinfo(struct nfsd4_compoundres *resp, } supported = 0; - RESERVE_SPACE(4); + p = xdr_reserve_space(xdr, 4); + if (!p) + goto out; flavorsp = p++; /* to be backfilled later */ - ADJUST_ARGS(); for (i = 0; i < nflavs; i++) { rpc_authflavor_t pf = flavs[i].pseudoflavor; @@ -3207,18 +3438,20 @@ nfsd4_do_encode_secinfo(struct nfsd4_compoundres *resp, if (rpcauth_get_gssinfo(pf, &info) == 0) { supported++; - RESERVE_SPACE(4 + 4 + XDR_LEN(info.oid.len) + 4 + 4); - WRITE32(RPC_AUTH_GSS); - WRITE32(info.oid.len); - WRITEMEM(info.oid.data, info.oid.len); - WRITE32(info.qop); - WRITE32(info.service); - ADJUST_ARGS(); + p = xdr_reserve_space(xdr, 4 + 4 + + XDR_LEN(info.oid.len) + 4 + 4); + if (!p) + goto out; + *p++ = cpu_to_be32(RPC_AUTH_GSS); + p = xdr_encode_opaque(p, info.oid.data, info.oid.len); + *p++ = cpu_to_be32(info.qop); + *p++ = cpu_to_be32(info.service); } else if (pf < RPC_AUTH_MAXFLAVOR) { supported++; - RESERVE_SPACE(4); - WRITE32(pf); - ADJUST_ARGS(); + p = xdr_reserve_space(xdr, 4); + if (!p) + goto out; + *p++ = cpu_to_be32(pf); } else { if (report) pr_warn("NFS: SECINFO: security flavor %u " @@ -3229,7 +3462,7 @@ nfsd4_do_encode_secinfo(struct nfsd4_compoundres *resp, if (nflavs != supported) report = false; *flavorsp = htonl(supported); - + nfserr = 0; out: if (exp) exp_put(exp); @@ -3240,14 +3473,18 @@ static __be32 nfsd4_encode_secinfo(struct nfsd4_compoundres *resp, __be32 nfserr, struct nfsd4_secinfo *secinfo) { - return nfsd4_do_encode_secinfo(resp, nfserr, secinfo->si_exp); + struct xdr_stream *xdr = &resp->xdr; + + return nfsd4_do_encode_secinfo(xdr, nfserr, secinfo->si_exp); } static __be32 nfsd4_encode_secinfo_no_name(struct nfsd4_compoundres *resp, __be32 nfserr, struct nfsd4_secinfo_no_name *secinfo) { - return nfsd4_do_encode_secinfo(resp, nfserr, secinfo->sin_exp); + struct xdr_stream *xdr = &resp->xdr; + + return nfsd4_do_encode_secinfo(xdr, nfserr, secinfo->sin_exp); } /* @@ -3257,41 +3494,47 @@ nfsd4_encode_secinfo_no_name(struct nfsd4_compoundres *resp, __be32 nfserr, static __be32 nfsd4_encode_setattr(struct nfsd4_compoundres *resp, __be32 nfserr, struct nfsd4_setattr *setattr) { + struct xdr_stream *xdr = &resp->xdr; __be32 *p; - RESERVE_SPACE(16); + p = xdr_reserve_space(xdr, 16); + if (!p) + return nfserr_resource; if (nfserr) { - WRITE32(3); - WRITE32(0); - WRITE32(0); - WRITE32(0); + *p++ = cpu_to_be32(3); + *p++ = cpu_to_be32(0); + *p++ = cpu_to_be32(0); + *p++ = cpu_to_be32(0); } else { - WRITE32(3); - WRITE32(setattr->sa_bmval[0]); - WRITE32(setattr->sa_bmval[1]); - WRITE32(setattr->sa_bmval[2]); + *p++ = cpu_to_be32(3); + *p++ = cpu_to_be32(setattr->sa_bmval[0]); + *p++ = cpu_to_be32(setattr->sa_bmval[1]); + *p++ = cpu_to_be32(setattr->sa_bmval[2]); } - ADJUST_ARGS(); return nfserr; } static __be32 nfsd4_encode_setclientid(struct nfsd4_compoundres *resp, __be32 nfserr, struct nfsd4_setclientid *scd) { + struct xdr_stream *xdr = &resp->xdr; __be32 *p; if (!nfserr) { - RESERVE_SPACE(8 + NFS4_VERIFIER_SIZE); - WRITEMEM(&scd->se_clientid, 8); - WRITEMEM(&scd->se_confirm, NFS4_VERIFIER_SIZE); - ADJUST_ARGS(); + p = xdr_reserve_space(xdr, 8 + NFS4_VERIFIER_SIZE); + if (!p) + return nfserr_resource; + p = xdr_encode_opaque_fixed(p, &scd->se_clientid, 8); + p = xdr_encode_opaque_fixed(p, &scd->se_confirm, + NFS4_VERIFIER_SIZE); } else if (nfserr == nfserr_clid_inuse) { - RESERVE_SPACE(8); - WRITE32(0); - WRITE32(0); - ADJUST_ARGS(); + p = xdr_reserve_space(xdr, 8); + if (!p) + return nfserr_resource; + *p++ = cpu_to_be32(0); + *p++ = cpu_to_be32(0); } return nfserr; } @@ -3299,14 +3542,17 @@ nfsd4_encode_setclientid(struct nfsd4_compoundres *resp, __be32 nfserr, struct n static __be32 nfsd4_encode_write(struct nfsd4_compoundres *resp, __be32 nfserr, struct nfsd4_write *write) { + struct xdr_stream *xdr = &resp->xdr; __be32 *p; if (!nfserr) { - RESERVE_SPACE(16); - WRITE32(write->wr_bytes_written); - WRITE32(write->wr_how_written); - WRITEMEM(write->wr_verifier.data, NFS4_VERIFIER_SIZE); - ADJUST_ARGS(); + p = xdr_reserve_space(xdr, 16); + if (!p) + return nfserr_resource; + *p++ = cpu_to_be32(write->wr_bytes_written); + *p++ = cpu_to_be32(write->wr_how_written); + p = xdr_encode_opaque_fixed(p, write->wr_verifier.data, + NFS4_VERIFIER_SIZE); } return nfserr; } @@ -3323,6 +3569,7 @@ static __be32 nfsd4_encode_exchange_id(struct nfsd4_compoundres *resp, __be32 nfserr, struct nfsd4_exchange_id *exid) { + struct xdr_stream *xdr = &resp->xdr; __be32 *p; char *major_id; char *server_scope; @@ -3338,60 +3585,61 @@ nfsd4_encode_exchange_id(struct nfsd4_compoundres *resp, __be32 nfserr, server_scope = utsname()->nodename; server_scope_sz = strlen(server_scope); - RESERVE_SPACE( + p = xdr_reserve_space(xdr, 8 /* eir_clientid */ + 4 /* eir_sequenceid */ + 4 /* eir_flags */ + 4 /* spr_how */); + if (!p) + return nfserr_resource; - WRITEMEM(&exid->clientid, 8); - WRITE32(exid->seqid); - WRITE32(exid->flags); + p = xdr_encode_opaque_fixed(p, &exid->clientid, 8); + *p++ = cpu_to_be32(exid->seqid); + *p++ = cpu_to_be32(exid->flags); - WRITE32(exid->spa_how); - ADJUST_ARGS(); + *p++ = cpu_to_be32(exid->spa_how); switch (exid->spa_how) { case SP4_NONE: break; case SP4_MACH_CRED: /* spo_must_enforce, spo_must_allow */ - RESERVE_SPACE(16); + p = xdr_reserve_space(xdr, 16); + if (!p) + return nfserr_resource; /* spo_must_enforce bitmap: */ - WRITE32(2); - WRITE32(nfs4_minimal_spo_must_enforce[0]); - WRITE32(nfs4_minimal_spo_must_enforce[1]); + *p++ = cpu_to_be32(2); + *p++ = cpu_to_be32(nfs4_minimal_spo_must_enforce[0]); + *p++ = cpu_to_be32(nfs4_minimal_spo_must_enforce[1]); /* empty spo_must_allow bitmap: */ - WRITE32(0); + *p++ = cpu_to_be32(0); - ADJUST_ARGS(); break; default: WARN_ON_ONCE(1); } - RESERVE_SPACE( + p = xdr_reserve_space(xdr, 8 /* so_minor_id */ + 4 /* so_major_id.len */ + (XDR_QUADLEN(major_id_sz) * 4) + 4 /* eir_server_scope.len */ + (XDR_QUADLEN(server_scope_sz) * 4) + 4 /* eir_server_impl_id.count (0) */); + if (!p) + return nfserr_resource; /* The server_owner struct */ - WRITE64(minor_id); /* Minor id */ + p = xdr_encode_hyper(p, minor_id); /* Minor id */ /* major id */ - WRITE32(major_id_sz); - WRITEMEM(major_id, major_id_sz); + p = xdr_encode_opaque(p, major_id, major_id_sz); /* Server scope */ - WRITE32(server_scope_sz); - WRITEMEM(server_scope, server_scope_sz); + p = xdr_encode_opaque(p, server_scope, server_scope_sz); /* Implementation id */ - WRITE32(0); /* zero length nfs_impl_id4 array */ - ADJUST_ARGS(); + *p++ = cpu_to_be32(0); /* zero length nfs_impl_id4 array */ return 0; } @@ -3399,47 +3647,54 @@ static __be32 nfsd4_encode_create_session(struct nfsd4_compoundres *resp, __be32 nfserr, struct nfsd4_create_session *sess) { + struct xdr_stream *xdr = &resp->xdr; __be32 *p; if (nfserr) return nfserr; - RESERVE_SPACE(24); - WRITEMEM(sess->sessionid.data, NFS4_MAX_SESSIONID_LEN); - WRITE32(sess->seqid); - WRITE32(sess->flags); - ADJUST_ARGS(); - - RESERVE_SPACE(28); - WRITE32(0); /* headerpadsz */ - WRITE32(sess->fore_channel.maxreq_sz); - WRITE32(sess->fore_channel.maxresp_sz); - WRITE32(sess->fore_channel.maxresp_cached); - WRITE32(sess->fore_channel.maxops); - WRITE32(sess->fore_channel.maxreqs); - WRITE32(sess->fore_channel.nr_rdma_attrs); - ADJUST_ARGS(); + p = xdr_reserve_space(xdr, 24); + if (!p) + return nfserr_resource; + p = xdr_encode_opaque_fixed(p, sess->sessionid.data, + NFS4_MAX_SESSIONID_LEN); + *p++ = cpu_to_be32(sess->seqid); + *p++ = cpu_to_be32(sess->flags); + + p = xdr_reserve_space(xdr, 28); + if (!p) + return nfserr_resource; + *p++ = cpu_to_be32(0); /* headerpadsz */ + *p++ = cpu_to_be32(sess->fore_channel.maxreq_sz); + *p++ = cpu_to_be32(sess->fore_channel.maxresp_sz); + *p++ = cpu_to_be32(sess->fore_channel.maxresp_cached); + *p++ = cpu_to_be32(sess->fore_channel.maxops); + *p++ = cpu_to_be32(sess->fore_channel.maxreqs); + *p++ = cpu_to_be32(sess->fore_channel.nr_rdma_attrs); if (sess->fore_channel.nr_rdma_attrs) { - RESERVE_SPACE(4); - WRITE32(sess->fore_channel.rdma_attrs); - ADJUST_ARGS(); + p = xdr_reserve_space(xdr, 4); + if (!p) + return nfserr_resource; + *p++ = cpu_to_be32(sess->fore_channel.rdma_attrs); } - RESERVE_SPACE(28); - WRITE32(0); /* headerpadsz */ - WRITE32(sess->back_channel.maxreq_sz); - WRITE32(sess->back_channel.maxresp_sz); - WRITE32(sess->back_channel.maxresp_cached); - WRITE32(sess->back_channel.maxops); - WRITE32(sess->back_channel.maxreqs); - WRITE32(sess->back_channel.nr_rdma_attrs); - ADJUST_ARGS(); + p = xdr_reserve_space(xdr, 28); + if (!p) + return nfserr_resource; + *p++ = cpu_to_be32(0); /* headerpadsz */ + *p++ = cpu_to_be32(sess->back_channel.maxreq_sz); + *p++ = cpu_to_be32(sess->back_channel.maxresp_sz); + *p++ = cpu_to_be32(sess->back_channel.maxresp_cached); + *p++ = cpu_to_be32(sess->back_channel.maxops); + *p++ = cpu_to_be32(sess->back_channel.maxreqs); + *p++ = cpu_to_be32(sess->back_channel.nr_rdma_attrs); if (sess->back_channel.nr_rdma_attrs) { - RESERVE_SPACE(4); - WRITE32(sess->back_channel.rdma_attrs); - ADJUST_ARGS(); + p = xdr_reserve_space(xdr, 4); + if (!p) + return nfserr_resource; + *p++ = cpu_to_be32(sess->back_channel.rdma_attrs); } return 0; } @@ -3448,22 +3703,25 @@ static __be32 nfsd4_encode_sequence(struct nfsd4_compoundres *resp, __be32 nfserr, struct nfsd4_sequence *seq) { + struct xdr_stream *xdr = &resp->xdr; __be32 *p; if (nfserr) return nfserr; - RESERVE_SPACE(NFS4_MAX_SESSIONID_LEN + 20); - WRITEMEM(seq->sessionid.data, NFS4_MAX_SESSIONID_LEN); - WRITE32(seq->seqid); - WRITE32(seq->slotid); + p = xdr_reserve_space(xdr, NFS4_MAX_SESSIONID_LEN + 20); + if (!p) + return nfserr_resource; + p = xdr_encode_opaque_fixed(p, seq->sessionid.data, + NFS4_MAX_SESSIONID_LEN); + *p++ = cpu_to_be32(seq->seqid); + *p++ = cpu_to_be32(seq->slotid); /* Note slotid's are numbered from zero: */ - WRITE32(seq->maxslots - 1); /* sr_highest_slotid */ - WRITE32(seq->maxslots - 1); /* sr_target_highest_slotid */ - WRITE32(seq->status_flags); + *p++ = cpu_to_be32(seq->maxslots - 1); /* sr_highest_slotid */ + *p++ = cpu_to_be32(seq->maxslots - 1); /* sr_target_highest_slotid */ + *p++ = cpu_to_be32(seq->status_flags); - ADJUST_ARGS(); - resp->cstate.datap = p; /* DRC cache data pointer */ + resp->cstate.data_offset = xdr->buf->len; /* DRC cache data pointer */ return 0; } @@ -3471,20 +3729,22 @@ static __be32 nfsd4_encode_test_stateid(struct nfsd4_compoundres *resp, __be32 nfserr, struct nfsd4_test_stateid *test_stateid) { + struct xdr_stream *xdr = &resp->xdr; struct nfsd4_test_stateid_id *stateid, *next; __be32 *p; if (nfserr) return nfserr; - RESERVE_SPACE(4 + (4 * test_stateid->ts_num_ids)); + p = xdr_reserve_space(xdr, 4 + (4 * test_stateid->ts_num_ids)); + if (!p) + return nfserr_resource; *p++ = htonl(test_stateid->ts_num_ids); list_for_each_entry_safe(stateid, next, &test_stateid->ts_stateid_list, ts_id_list) { *p++ = stateid->ts_id_status; } - ADJUST_ARGS(); return nfserr; } @@ -3563,81 +3823,99 @@ static nfsd4_enc nfsd4_enc_ops[] = { }; /* - * Calculate the total amount of memory that the compound response has taken - * after encoding the current operation with pad. - * - * pad: if operation is non-idempotent, pad was calculate by op_rsize_bop() - * which was specified at nfsd4_operation, else pad is zero. - * - * Compare this length to the session se_fmaxresp_sz and se_fmaxresp_cached. + * Calculate whether we still have space to encode repsize bytes. + * There are two considerations: + * - For NFS versions >=4.1, the size of the reply must stay within + * session limits + * - For all NFS versions, we must stay within limited preallocated + * buffer space. * - * Our se_fmaxresp_cached will always be a multiple of PAGE_SIZE, and so - * will be at least a page and will therefore hold the xdr_buf head. + * This is called before the operation is processed, so can only provide + * an upper estimate. For some nonidempotent operations (such as + * getattr), it's not necessarily a problem if that estimate is wrong, + * as we can fail it after processing without significant side effects. */ -__be32 nfsd4_check_resp_size(struct nfsd4_compoundres *resp, u32 pad) +__be32 nfsd4_check_resp_size(struct nfsd4_compoundres *resp, u32 respsize) { - struct xdr_buf *xb = &resp->rqstp->rq_res; - struct nfsd4_session *session = NULL; + struct xdr_buf *buf = &resp->rqstp->rq_res; struct nfsd4_slot *slot = resp->cstate.slot; - u32 length, tlen = 0; + if (buf->len + respsize <= buf->buflen) + return nfs_ok; if (!nfsd4_has_session(&resp->cstate)) - return 0; - - session = resp->cstate.session; - - if (xb->page_len == 0) { - length = (char *)resp->p - (char *)xb->head[0].iov_base + pad; - } else { - if (xb->tail[0].iov_base && xb->tail[0].iov_len > 0) - tlen = (char *)resp->p - (char *)xb->tail[0].iov_base; - - length = xb->head[0].iov_len + xb->page_len + tlen + pad; - } - dprintk("%s length %u, xb->page_len %u tlen %u pad %u\n", __func__, - length, xb->page_len, tlen, pad); - - if (length > session->se_fchannel.maxresp_sz) - return nfserr_rep_too_big; - - if ((slot->sl_flags & NFSD4_SLOT_CACHETHIS) && - length > session->se_fchannel.maxresp_cached) + return nfserr_resource; + if (slot->sl_flags & NFSD4_SLOT_CACHETHIS) { + WARN_ON_ONCE(1); return nfserr_rep_too_big_to_cache; - - return 0; + } + return nfserr_rep_too_big; } void nfsd4_encode_operation(struct nfsd4_compoundres *resp, struct nfsd4_op *op) { + struct xdr_stream *xdr = &resp->xdr; struct nfs4_stateowner *so = resp->cstate.replay_owner; - __be32 *statp; + struct svc_rqst *rqstp = resp->rqstp; + int post_err_offset; + nfsd4_enc encoder; __be32 *p; - RESERVE_SPACE(8); - WRITE32(op->opnum); - statp = p++; /* to be backfilled at the end */ - ADJUST_ARGS(); + p = xdr_reserve_space(xdr, 8); + if (!p) { + WARN_ON_ONCE(1); + return; + } + *p++ = cpu_to_be32(op->opnum); + post_err_offset = xdr->buf->len; if (op->opnum == OP_ILLEGAL) goto status; BUG_ON(op->opnum < 0 || op->opnum >= ARRAY_SIZE(nfsd4_enc_ops) || !nfsd4_enc_ops[op->opnum]); - op->status = nfsd4_enc_ops[op->opnum](resp, op->status, &op->u); + encoder = nfsd4_enc_ops[op->opnum]; + op->status = encoder(resp, op->status, &op->u); + xdr_commit_encode(xdr); + /* nfsd4_check_resp_size guarantees enough room for error status */ - if (!op->status) - op->status = nfsd4_check_resp_size(resp, 0); + if (!op->status) { + int space_needed = 0; + if (!nfsd4_last_compound_op(rqstp)) + space_needed = COMPOUND_ERR_SLACK_SPACE; + op->status = nfsd4_check_resp_size(resp, space_needed); + } + if (op->status == nfserr_resource && nfsd4_has_session(&resp->cstate)) { + struct nfsd4_slot *slot = resp->cstate.slot; + + if (slot->sl_flags & NFSD4_SLOT_CACHETHIS) + op->status = nfserr_rep_too_big_to_cache; + else + op->status = nfserr_rep_too_big; + } + if (op->status == nfserr_resource || + op->status == nfserr_rep_too_big || + op->status == nfserr_rep_too_big_to_cache) { + /* + * The operation may have already been encoded or + * partially encoded. No op returns anything additional + * in the case of one of these three errors, so we can + * just truncate back to after the status. But it's a + * bug if we had to do this on a non-idempotent op: + */ + warn_on_nonidempotent_op(op); + xdr_truncate_encode(xdr, post_err_offset); + } if (so) { + int len = xdr->buf->len - post_err_offset; + so->so_replay.rp_status = op->status; - so->so_replay.rp_buflen = (char *)resp->p - (char *)(statp+1); - memcpy(so->so_replay.rp_buf, statp+1, so->so_replay.rp_buflen); + so->so_replay.rp_buflen = len; + read_bytes_from_xdr_buf(xdr->buf, post_err_offset, + so->so_replay.rp_buf, len); } status: - /* - * Note: We write the status directly, instead of using WRITE32(), - * since it is already in network byte order. - */ - *statp = op->status; + /* Note that op->status is already in network byte order: */ + write_bytes_to_xdr_buf(xdr->buf, post_err_offset - 4, &op->status, 4); } /* @@ -3649,21 +3927,22 @@ status: * called with nfs4_lock_state() held */ void -nfsd4_encode_replay(struct nfsd4_compoundres *resp, struct nfsd4_op *op) +nfsd4_encode_replay(struct xdr_stream *xdr, struct nfsd4_op *op) { __be32 *p; struct nfs4_replay *rp = op->replay; BUG_ON(!rp); - RESERVE_SPACE(8); - WRITE32(op->opnum); + p = xdr_reserve_space(xdr, 8 + rp->rp_buflen); + if (!p) { + WARN_ON_ONCE(1); + return; + } + *p++ = cpu_to_be32(op->opnum); *p++ = rp->rp_status; /* already xdr'ed */ - ADJUST_ARGS(); - RESERVE_SPACE(rp->rp_buflen); - WRITEMEM(rp->rp_buf, rp->rp_buflen); - ADJUST_ARGS(); + p = xdr_encode_opaque_fixed(p, rp->rp_buf, rp->rp_buflen); } int @@ -3720,19 +3999,19 @@ nfs4svc_encode_compoundres(struct svc_rqst *rqstp, __be32 *p, struct nfsd4_compo * All that remains is to write the tag and operation count... */ struct nfsd4_compound_state *cs = &resp->cstate; - struct kvec *iov; + struct xdr_buf *buf = resp->xdr.buf; + + WARN_ON_ONCE(buf->len != buf->head[0].iov_len + buf->page_len + + buf->tail[0].iov_len); + + rqstp->rq_next_page = resp->xdr.page_ptr + 1; + p = resp->tagp; *p++ = htonl(resp->taglen); memcpy(p, resp->tag, resp->taglen); p += XDR_QUADLEN(resp->taglen); *p++ = htonl(resp->opcnt); - if (rqstp->rq_res.page_len) - iov = &rqstp->rq_res.tail[0]; - else - iov = &rqstp->rq_res.head[0]; - iov->iov_len = ((char*)resp->p) - (char*)iov->iov_base; - BUG_ON(iov->iov_len > PAGE_SIZE); if (nfsd4_has_session(cs)) { struct nfsd_net *nn = net_generic(SVC_NET(rqstp), nfsd_net_id); struct nfs4_client *clp = cs->session->se_client; diff --git a/fs/nfsd/nfscache.c b/fs/nfsd/nfscache.c index f8f060ffbf4f..6040da8830ff 100644 --- a/fs/nfsd/nfscache.c +++ b/fs/nfsd/nfscache.c @@ -224,13 +224,6 @@ hash_refile(struct svc_cacherep *rp) hlist_add_head(&rp->c_hash, cache_hash + hash_32(rp->c_xid, maskbits)); } -static inline bool -nfsd_cache_entry_expired(struct svc_cacherep *rp) -{ - return rp->c_state != RC_INPROG && - time_after(jiffies, rp->c_timestamp + RC_EXPIRE); -} - /* * Walk the LRU list and prune off entries that are older than RC_EXPIRE. * Also prune the oldest ones when the total exceeds the max number of entries. @@ -242,8 +235,14 @@ prune_cache_entries(void) long freed = 0; list_for_each_entry_safe(rp, tmp, &lru_head, c_lru) { - if (!nfsd_cache_entry_expired(rp) && - num_drc_entries <= max_drc_entries) + /* + * Don't free entries attached to calls that are still + * in-progress, but do keep scanning the list. + */ + if (rp->c_state == RC_INPROG) + continue; + if (num_drc_entries <= max_drc_entries && + time_before(jiffies, rp->c_timestamp + RC_EXPIRE)) break; nfsd_reply_cache_free_locked(rp); freed++; diff --git a/fs/nfsd/nfsctl.c b/fs/nfsd/nfsctl.c index f34d9de802ab..51844048937f 100644 --- a/fs/nfsd/nfsctl.c +++ b/fs/nfsd/nfsctl.c @@ -1179,7 +1179,6 @@ static int __init init_nfsd(void) retval = nfsd4_init_slabs(); if (retval) goto out_unregister_pernet; - nfs4_state_init(); retval = nfsd_fault_inject_init(); /* nfsd fault injection controls */ if (retval) goto out_free_slabs; diff --git a/fs/nfsd/nfsd.h b/fs/nfsd/nfsd.h index 479eb681c27c..847daf37e566 100644 --- a/fs/nfsd/nfsd.h +++ b/fs/nfsd/nfsd.h @@ -15,11 +15,20 @@ #include <linux/nfs2.h> #include <linux/nfs3.h> #include <linux/nfs4.h> +#include <linux/sunrpc/svc.h> #include <linux/sunrpc/msg_prot.h> -#include <linux/nfsd/debug.h> -#include <linux/nfsd/export.h> -#include <linux/nfsd/stats.h> +#include <uapi/linux/nfsd/debug.h> + +#include "stats.h" +#include "export.h" + +#undef ifdebug +#ifdef NFSD_DEBUG +# define ifdebug(flag) if (nfsd_debug & NFSDDBG_##flag) +#else +# define ifdebug(flag) if (0) +#endif /* * nfsd version @@ -106,7 +115,6 @@ static inline int nfsd_v4client(struct svc_rqst *rq) */ #ifdef CONFIG_NFSD_V4 extern unsigned long max_delegations; -void nfs4_state_init(void); int nfsd4_init_slabs(void); void nfsd4_free_slabs(void); int nfs4_state_start(void); @@ -117,7 +125,6 @@ void nfs4_reset_lease(time_t leasetime); int nfs4_reset_recoverydir(char *recdir); char * nfs4_recoverydir(void); #else -static inline void nfs4_state_init(void) { } static inline int nfsd4_init_slabs(void) { return 0; } static inline void nfsd4_free_slabs(void) { } static inline int nfs4_state_start(void) { return 0; } diff --git a/fs/nfsd/nfsfh.c b/fs/nfsd/nfsfh.c index 3c37b160dcad..ec8393418154 100644 --- a/fs/nfsd/nfsfh.c +++ b/fs/nfsd/nfsfh.c @@ -88,9 +88,8 @@ static __be32 nfsd_setuser_and_check_port(struct svc_rqst *rqstp, /* Check if the request originated from a secure port. */ if (!rqstp->rq_secure && !(flags & NFSEXP_INSECURE_PORT)) { RPC_IFDEBUG(char buf[RPC_MAX_ADDRBUFLEN]); - dprintk(KERN_WARNING - "nfsd: request from insecure port %s!\n", - svc_print_addr(rqstp, buf, sizeof(buf))); + dprintk("nfsd: request from insecure port %s!\n", + svc_print_addr(rqstp, buf, sizeof(buf))); return nfserr_perm; } @@ -169,8 +168,8 @@ static __be32 nfsd_set_fh_dentry(struct svc_rqst *rqstp, struct svc_fh *fhp) data_left -= len; if (data_left < 0) return error; - exp = rqst_exp_find(rqstp, fh->fh_fsid_type, fh->fh_auth); - fid = (struct fid *)(fh->fh_auth + len); + exp = rqst_exp_find(rqstp, fh->fh_fsid_type, fh->fh_fsid); + fid = (struct fid *)(fh->fh_fsid + len); } else { __u32 tfh[2]; dev_t xdev; @@ -385,7 +384,7 @@ static void _fh_update(struct svc_fh *fhp, struct svc_export *exp, { if (dentry != exp->ex_path.dentry) { struct fid *fid = (struct fid *) - (fhp->fh_handle.fh_auth + fhp->fh_handle.fh_size/4 - 1); + (fhp->fh_handle.fh_fsid + fhp->fh_handle.fh_size/4 - 1); int maxsize = (fhp->fh_maxsize - fhp->fh_handle.fh_size)/4; int subtreecheck = !(exp->ex_flags & NFSEXP_NOSUBTREECHECK); @@ -513,7 +512,6 @@ fh_compose(struct svc_fh *fhp, struct svc_export *exp, struct dentry *dentry, */ struct inode * inode = dentry->d_inode; - __u32 *datap; dev_t ex_dev = exp_sb(exp)->s_dev; dprintk("nfsd: fh_compose(exp %02x:%02x/%ld %pd2, ino=%ld)\n", @@ -557,17 +555,16 @@ fh_compose(struct svc_fh *fhp, struct svc_export *exp, struct dentry *dentry, if (inode) _fh_update_old(dentry, exp, &fhp->fh_handle); } else { - int len; + fhp->fh_handle.fh_size = + key_len(fhp->fh_handle.fh_fsid_type) + 4; fhp->fh_handle.fh_auth_type = 0; - datap = fhp->fh_handle.fh_auth+0; - mk_fsid(fhp->fh_handle.fh_fsid_type, datap, ex_dev, + + mk_fsid(fhp->fh_handle.fh_fsid_type, + fhp->fh_handle.fh_fsid, + ex_dev, exp->ex_path.dentry->d_inode->i_ino, exp->ex_fsid, exp->ex_uuid); - len = key_len(fhp->fh_handle.fh_fsid_type); - datap += len/4; - fhp->fh_handle.fh_size = 4 + len; - if (inode) _fh_update(fhp, exp, dentry); if (fhp->fh_handle.fh_fileid_type == FILEID_INVALID) { diff --git a/fs/nfsd/nfsfh.h b/fs/nfsd/nfsfh.h index ad67964d0bb1..2e89e70ac15c 100644 --- a/fs/nfsd/nfsfh.h +++ b/fs/nfsd/nfsfh.h @@ -1,9 +1,58 @@ -/* Copyright (C) 1995, 1996, 1997 Olaf Kirch <okir@monad.swb.de> */ +/* + * Copyright (C) 1995, 1996, 1997 Olaf Kirch <okir@monad.swb.de> + * + * This file describes the layout of the file handles as passed + * over the wire. + */ +#ifndef _LINUX_NFSD_NFSFH_H +#define _LINUX_NFSD_NFSFH_H + +#include <linux/sunrpc/svc.h> +#include <uapi/linux/nfsd/nfsfh.h> + +static inline __u32 ino_t_to_u32(ino_t ino) +{ + return (__u32) ino; +} + +static inline ino_t u32_to_ino_t(__u32 uino) +{ + return (ino_t) uino; +} -#ifndef _LINUX_NFSD_FH_INT_H -#define _LINUX_NFSD_FH_INT_H +/* + * This is the internal representation of an NFS handle used in knfsd. + * pre_mtime/post_version will be used to support wcc_attr's in NFSv3. + */ +typedef struct svc_fh { + struct knfsd_fh fh_handle; /* FH data */ + struct dentry * fh_dentry; /* validated dentry */ + struct svc_export * fh_export; /* export pointer */ + int fh_maxsize; /* max size for fh_handle */ + + unsigned char fh_locked; /* inode locked by us */ + unsigned char fh_want_write; /* remount protection taken */ + +#ifdef CONFIG_NFSD_V3 + unsigned char fh_post_saved; /* post-op attrs saved */ + unsigned char fh_pre_saved; /* pre-op attrs saved */ + + /* Pre-op attributes saved during fh_lock */ + __u64 fh_pre_size; /* size before operation */ + struct timespec fh_pre_mtime; /* mtime before oper */ + struct timespec fh_pre_ctime; /* ctime before oper */ + /* + * pre-op nfsv4 change attr: note must check IS_I_VERSION(inode) + * to find out if it is valid. + */ + u64 fh_pre_change; + + /* Post-op attributes saved in fh_unlock */ + struct kstat fh_post_attr; /* full attrs after operation */ + u64 fh_post_change; /* nfsv4 change; see above */ +#endif /* CONFIG_NFSD_V3 */ -#include <linux/nfsd/nfsfh.h> +} svc_fh; enum nfsd_fsid { FSID_DEV = 0, @@ -215,4 +264,4 @@ fh_unlock(struct svc_fh *fhp) } } -#endif /* _LINUX_NFSD_FH_INT_H */ +#endif /* _LINUX_NFSD_NFSFH_H */ diff --git a/fs/nfsd/nfssvc.c b/fs/nfsd/nfssvc.c index 9a4a5f9e7468..1879e43f2868 100644 --- a/fs/nfsd/nfssvc.c +++ b/fs/nfsd/nfssvc.c @@ -591,12 +591,6 @@ nfsd(void *vrqstp) nfsdstats.th_cnt++; mutex_unlock(&nfsd_mutex); - /* - * We want less throttling in balance_dirty_pages() so that nfs to - * localhost doesn't cause nfsd to lock up due to all the client's - * dirty pages. - */ - current->flags |= PF_LESS_THROTTLE; set_freezable(); /* diff --git a/fs/nfsd/nfsxdr.c b/fs/nfsd/nfsxdr.c index 9c769a47ac5a..1ac306b769df 100644 --- a/fs/nfsd/nfsxdr.c +++ b/fs/nfsd/nfsxdr.c @@ -214,7 +214,8 @@ nfssvc_decode_void(struct svc_rqst *rqstp, __be32 *p, void *dummy) int nfssvc_decode_fhandle(struct svc_rqst *rqstp, __be32 *p, struct nfsd_fhandle *args) { - if (!(p = decode_fh(p, &args->fh))) + p = decode_fh(p, &args->fh); + if (!p) return 0; return xdr_argsize_check(rqstp, p); } @@ -248,7 +249,8 @@ nfssvc_decode_readargs(struct svc_rqst *rqstp, __be32 *p, { unsigned int len; int v; - if (!(p = decode_fh(p, &args->fh))) + p = decode_fh(p, &args->fh); + if (!p) return 0; args->offset = ntohl(*p++); @@ -281,7 +283,8 @@ nfssvc_decode_writeargs(struct svc_rqst *rqstp, __be32 *p, unsigned int len, hdr, dlen; int v; - if (!(p = decode_fh(p, &args->fh))) + p = decode_fh(p, &args->fh); + if (!p) return 0; p++; /* beginoffset */ @@ -355,7 +358,8 @@ nfssvc_decode_renameargs(struct svc_rqst *rqstp, __be32 *p, int nfssvc_decode_readlinkargs(struct svc_rqst *rqstp, __be32 *p, struct nfsd_readlinkargs *args) { - if (!(p = decode_fh(p, &args->fh))) + p = decode_fh(p, &args->fh); + if (!p) return 0; args->buffer = page_address(*(rqstp->rq_next_page++)); @@ -391,7 +395,8 @@ int nfssvc_decode_readdirargs(struct svc_rqst *rqstp, __be32 *p, struct nfsd_readdirargs *args) { - if (!(p = decode_fh(p, &args->fh))) + p = decode_fh(p, &args->fh); + if (!p) return 0; args->cookie = ntohl(*p++); args->count = ntohl(*p++); diff --git a/fs/nfsd/state.h b/fs/nfsd/state.h index 424d8f5f2317..374c66283ac5 100644 --- a/fs/nfsd/state.h +++ b/fs/nfsd/state.h @@ -37,7 +37,6 @@ #include <linux/idr.h> #include <linux/sunrpc/svc_xprt.h> -#include <linux/nfsd/nfsfh.h> #include "nfsfh.h" typedef struct { @@ -123,7 +122,7 @@ static inline struct nfs4_delegation *delegstateid(struct nfs4_stid *s) /* Maximum number of operations per session compound */ #define NFSD_MAX_OPS_PER_COMPOUND 16 /* Maximum session per slot cache size */ -#define NFSD_SLOT_CACHE_SIZE 1024 +#define NFSD_SLOT_CACHE_SIZE 2048 /* Maximum number of NFSD_SLOT_CACHE_SIZE slots per session */ #define NFSD_CACHE_SIZE_SLOTS_PER_SESSION 32 #define NFSD_MAX_MEM_PER_SESSION \ @@ -464,8 +463,6 @@ extern void nfs4_release_reclaim(struct nfsd_net *); extern struct nfs4_client_reclaim *nfsd4_find_reclaim_client(const char *recdir, struct nfsd_net *nn); extern __be32 nfs4_check_open_reclaim(clientid_t *clid, bool sessions, struct nfsd_net *nn); -extern void nfs4_free_openowner(struct nfs4_openowner *); -extern void nfs4_free_lockowner(struct nfs4_lockowner *); extern int set_callback_cred(void); extern void nfsd4_init_callback(struct nfsd4_callback *); extern void nfsd4_probe_callback(struct nfs4_client *clp); diff --git a/fs/nfsd/stats.c b/fs/nfsd/stats.c index 6d4521feb6e3..cd90878a76aa 100644 --- a/fs/nfsd/stats.c +++ b/fs/nfsd/stats.c @@ -24,7 +24,6 @@ #include <linux/seq_file.h> #include <linux/module.h> #include <linux/sunrpc/stats.h> -#include <linux/nfsd/stats.h> #include <net/net_namespace.h> #include "nfsd.h" diff --git a/include/linux/nfsd/stats.h b/fs/nfsd/stats.h index e75b2544ff12..a5c944b771c6 100644 --- a/include/linux/nfsd/stats.h +++ b/fs/nfsd/stats.h @@ -1,12 +1,10 @@ /* - * linux/include/linux/nfsd/stats.h - * * Statistics for NFS server. * * Copyright (C) 1995, 1996 Olaf Kirch <okir@monad.swb.de> */ -#ifndef LINUX_NFSD_STATS_H -#define LINUX_NFSD_STATS_H +#ifndef _NFSD_STATS_H +#define _NFSD_STATS_H #include <uapi/linux/nfsd/stats.h> @@ -42,4 +40,4 @@ extern struct svc_stat nfsd_svcstats; void nfsd_stat_init(void); void nfsd_stat_shutdown(void); -#endif /* LINUX_NFSD_STATS_H */ +#endif /* _NFSD_STATS_H */ diff --git a/fs/nfsd/vfs.c b/fs/nfsd/vfs.c index 16f0673a423c..140c496f612c 100644 --- a/fs/nfsd/vfs.c +++ b/fs/nfsd/vfs.c @@ -820,55 +820,54 @@ static int nfsd_direct_splice_actor(struct pipe_inode_info *pipe, return __splice_from_pipe(pipe, sd, nfsd_splice_actor); } -static __be32 -nfsd_vfs_read(struct svc_rqst *rqstp, struct svc_fh *fhp, struct file *file, - loff_t offset, struct kvec *vec, int vlen, unsigned long *count) +__be32 nfsd_finish_read(struct file *file, unsigned long *count, int host_err) { - mm_segment_t oldfs; - __be32 err; - int host_err; - - err = nfserr_perm; - - if (file->f_op->splice_read && rqstp->rq_splice_ok) { - struct splice_desc sd = { - .len = 0, - .total_len = *count, - .pos = offset, - .u.data = rqstp, - }; - - rqstp->rq_next_page = rqstp->rq_respages + 1; - host_err = splice_direct_to_actor(file, &sd, nfsd_direct_splice_actor); - } else { - oldfs = get_fs(); - set_fs(KERNEL_DS); - host_err = vfs_readv(file, (struct iovec __user *)vec, vlen, &offset); - set_fs(oldfs); - } - if (host_err >= 0) { nfsdstats.io_read += host_err; *count = host_err; - err = 0; fsnotify_access(file); + return 0; } else - err = nfserrno(host_err); - return err; + return nfserrno(host_err); +} + +int nfsd_splice_read(struct svc_rqst *rqstp, + struct file *file, loff_t offset, unsigned long *count) +{ + struct splice_desc sd = { + .len = 0, + .total_len = *count, + .pos = offset, + .u.data = rqstp, + }; + int host_err; + + rqstp->rq_next_page = rqstp->rq_respages + 1; + host_err = splice_direct_to_actor(file, &sd, nfsd_direct_splice_actor); + return nfsd_finish_read(file, count, host_err); } -static void kill_suid(struct dentry *dentry) +int nfsd_readv(struct file *file, loff_t offset, struct kvec *vec, int vlen, + unsigned long *count) { - struct iattr ia; - ia.ia_valid = ATTR_KILL_SUID | ATTR_KILL_SGID | ATTR_KILL_PRIV; + mm_segment_t oldfs; + int host_err; - mutex_lock(&dentry->d_inode->i_mutex); - /* - * Note we call this on write, so notify_change will not - * encounter any conflicting delegations: - */ - notify_change(dentry, &ia, NULL); - mutex_unlock(&dentry->d_inode->i_mutex); + oldfs = get_fs(); + set_fs(KERNEL_DS); + host_err = vfs_readv(file, (struct iovec __user *)vec, vlen, &offset); + set_fs(oldfs); + return nfsd_finish_read(file, count, host_err); +} + +static __be32 +nfsd_vfs_read(struct svc_rqst *rqstp, struct file *file, + loff_t offset, struct kvec *vec, int vlen, unsigned long *count) +{ + if (file->f_op->splice_read && rqstp->rq_splice_ok) + return nfsd_splice_read(rqstp, file, offset, count); + else + return nfsd_readv(file, offset, vec, vlen, count); } /* @@ -922,6 +921,16 @@ nfsd_vfs_write(struct svc_rqst *rqstp, struct svc_fh *fhp, struct file *file, int stable = *stablep; int use_wgather; loff_t pos = offset; + unsigned int pflags = current->flags; + + if (rqstp->rq_local) + /* + * We want less throttling in balance_dirty_pages() + * and shrink_inactive_list() so that nfs to + * localhost doesn't cause nfsd to lock up due to all + * the client's dirty pages or its congested queue. + */ + current->flags |= PF_LESS_THROTTLE; dentry = file->f_path.dentry; inode = dentry->d_inode; @@ -942,10 +951,6 @@ nfsd_vfs_write(struct svc_rqst *rqstp, struct svc_fh *fhp, struct file *file, nfsdstats.io_write += host_err; fsnotify_modify(file); - /* clear setuid/setgid flag after write */ - if (inode->i_mode & (S_ISUID | S_ISGID)) - kill_suid(dentry); - if (stable) { if (use_wgather) host_err = wait_for_concurrent_writes(file); @@ -959,36 +964,33 @@ out_nfserr: err = 0; else err = nfserrno(host_err); + if (rqstp->rq_local) + tsk_restore_flags(current, pflags, PF_LESS_THROTTLE); return err; } -/* - * Read data from a file. count must contain the requested read count - * on entry. On return, *count contains the number of bytes actually read. - * N.B. After this call fhp needs an fh_put - */ -__be32 nfsd_read(struct svc_rqst *rqstp, struct svc_fh *fhp, - loff_t offset, struct kvec *vec, int vlen, unsigned long *count) +__be32 nfsd_get_tmp_read_open(struct svc_rqst *rqstp, struct svc_fh *fhp, + struct file **file, struct raparms **ra) { - struct file *file; struct inode *inode; - struct raparms *ra; __be32 err; - err = nfsd_open(rqstp, fhp, S_IFREG, NFSD_MAY_READ, &file); + err = nfsd_open(rqstp, fhp, S_IFREG, NFSD_MAY_READ, file); if (err) return err; - inode = file_inode(file); + inode = file_inode(*file); /* Get readahead parameters */ - ra = nfsd_get_raparms(inode->i_sb->s_dev, inode->i_ino); + *ra = nfsd_get_raparms(inode->i_sb->s_dev, inode->i_ino); - if (ra && ra->p_set) - file->f_ra = ra->p_ra; - - err = nfsd_vfs_read(rqstp, fhp, file, offset, vec, vlen, count); + if (*ra && (*ra)->p_set) + (*file)->f_ra = (*ra)->p_ra; + return nfs_ok; +} +void nfsd_put_tmp_read_open(struct file *file, struct raparms *ra) +{ /* Write back readahead params */ if (ra) { struct raparm_hbucket *rab = &raparm_hash[ra->p_hindex]; @@ -998,28 +1000,29 @@ __be32 nfsd_read(struct svc_rqst *rqstp, struct svc_fh *fhp, ra->p_count--; spin_unlock(&rab->pb_lock); } - nfsd_close(file); - return err; } -/* As above, but use the provided file descriptor. */ -__be32 -nfsd_read_file(struct svc_rqst *rqstp, struct svc_fh *fhp, struct file *file, - loff_t offset, struct kvec *vec, int vlen, - unsigned long *count) +/* + * Read data from a file. count must contain the requested read count + * on entry. On return, *count contains the number of bytes actually read. + * N.B. After this call fhp needs an fh_put + */ +__be32 nfsd_read(struct svc_rqst *rqstp, struct svc_fh *fhp, + loff_t offset, struct kvec *vec, int vlen, unsigned long *count) { - __be32 err; + struct file *file; + struct raparms *ra; + __be32 err; + + err = nfsd_get_tmp_read_open(rqstp, fhp, &file, &ra); + if (err) + return err; + + err = nfsd_vfs_read(rqstp, file, offset, vec, vlen, count); + + nfsd_put_tmp_read_open(file, ra); - if (file) { - err = nfsd_permission(rqstp, fhp->fh_export, fhp->fh_dentry, - NFSD_MAY_READ|NFSD_MAY_OWNER_OVERRIDE); - if (err) - goto out; - err = nfsd_vfs_read(rqstp, fhp, file, offset, vec, vlen, count); - } else /* Note file may still be NULL in NFSv4 special stateid case: */ - err = nfsd_read(rqstp, fhp, offset, vec, vlen, count); -out: return err; } diff --git a/fs/nfsd/vfs.h b/fs/nfsd/vfs.h index fbe90bdb2214..91b6ae3f658b 100644 --- a/fs/nfsd/vfs.h +++ b/fs/nfsd/vfs.h @@ -70,10 +70,16 @@ __be32 nfsd_commit(struct svc_rqst *, struct svc_fh *, __be32 nfsd_open(struct svc_rqst *, struct svc_fh *, umode_t, int, struct file **); void nfsd_close(struct file *); +struct raparms; +__be32 nfsd_get_tmp_read_open(struct svc_rqst *, struct svc_fh *, + struct file **, struct raparms **); +void nfsd_put_tmp_read_open(struct file *, struct raparms *); +int nfsd_splice_read(struct svc_rqst *, + struct file *, loff_t, unsigned long *); +int nfsd_readv(struct file *, loff_t, struct kvec *, int, + unsigned long *); __be32 nfsd_read(struct svc_rqst *, struct svc_fh *, loff_t, struct kvec *, int, unsigned long *); -__be32 nfsd_read_file(struct svc_rqst *, struct svc_fh *, struct file *, - loff_t, struct kvec *, int, unsigned long *); __be32 nfsd_write(struct svc_rqst *, struct svc_fh *,struct file *, loff_t, struct kvec *,int, unsigned long *, int *); __be32 nfsd_readlink(struct svc_rqst *, struct svc_fh *, diff --git a/fs/nfsd/xdr4.h b/fs/nfsd/xdr4.h index 5ea7df305083..18cbb6d9c8a9 100644 --- a/fs/nfsd/xdr4.h +++ b/fs/nfsd/xdr4.h @@ -58,7 +58,7 @@ struct nfsd4_compound_state { /* For sessions DRC */ struct nfsd4_session *session; struct nfsd4_slot *slot; - __be32 *datap; + int data_offset; size_t iovlen; u32 minorversion; __be32 status; @@ -287,9 +287,8 @@ struct nfsd4_readdir { struct svc_fh * rd_fhp; /* response */ struct readdir_cd common; - __be32 * buffer; - int buflen; - __be32 * offset; + struct xdr_stream *xdr; + int cookie_offset; }; struct nfsd4_release_lockowner { @@ -506,9 +505,7 @@ struct nfsd4_compoundargs { struct nfsd4_compoundres { /* scratch variables for XDR encode */ - __be32 * p; - __be32 * end; - struct xdr_buf * xbuf; + struct xdr_stream xdr; struct svc_rqst * rqstp; u32 taglen; @@ -538,6 +535,9 @@ static inline bool nfsd4_last_compound_op(struct svc_rqst *rqstp) return argp->opcnt == resp->opcnt; } +int nfsd4_max_reply(struct svc_rqst *rqstp, struct nfsd4_op *op); +void warn_on_nonidempotent_op(struct nfsd4_op *op); + #define NFS4_SVC_XDRSIZE sizeof(struct nfsd4_compoundargs) static inline void @@ -563,10 +563,11 @@ int nfs4svc_encode_compoundres(struct svc_rqst *, __be32 *, struct nfsd4_compoundres *); __be32 nfsd4_check_resp_size(struct nfsd4_compoundres *, u32); void nfsd4_encode_operation(struct nfsd4_compoundres *, struct nfsd4_op *); -void nfsd4_encode_replay(struct nfsd4_compoundres *resp, struct nfsd4_op *op); -__be32 nfsd4_encode_fattr(struct svc_fh *fhp, struct svc_export *exp, - struct dentry *dentry, __be32 **buffer, int countp, - u32 *bmval, struct svc_rqst *, int ignore_crossmnt); +void nfsd4_encode_replay(struct xdr_stream *xdr, struct nfsd4_op *op); +__be32 nfsd4_encode_fattr_to_buf(__be32 **p, int words, + struct svc_fh *fhp, struct svc_export *exp, + struct dentry *dentry, + u32 *bmval, struct svc_rqst *, int ignore_crossmnt); extern __be32 nfsd4_setclientid(struct svc_rqst *rqstp, struct nfsd4_compound_state *, struct nfsd4_setclientid *setclid); diff --git a/fs/ocfs2/cluster/tcp.c b/fs/ocfs2/cluster/tcp.c index a68e07a9bd46..681691bc233a 100644 --- a/fs/ocfs2/cluster/tcp.c +++ b/fs/ocfs2/cluster/tcp.c @@ -1799,7 +1799,7 @@ int o2net_register_hb_callbacks(void) /* ------------------------------------------------------------ */ -static int o2net_accept_one(struct socket *sock) +static int o2net_accept_one(struct socket *sock, int *more) { int ret, slen; struct sockaddr_in sin; @@ -1810,6 +1810,7 @@ static int o2net_accept_one(struct socket *sock) struct o2net_node *nn; BUG_ON(sock == NULL); + *more = 0; ret = sock_create_lite(sock->sk->sk_family, sock->sk->sk_type, sock->sk->sk_protocol, &new_sock); if (ret) @@ -1821,6 +1822,7 @@ static int o2net_accept_one(struct socket *sock) if (ret < 0) goto out; + *more = 1; new_sock->sk->sk_allocation = GFP_ATOMIC; ret = o2net_set_nodelay(new_sock); @@ -1919,11 +1921,36 @@ out: return ret; } +/* + * This function is invoked in response to one or more + * pending accepts at softIRQ level. We must drain the + * entire que before returning. + */ + static void o2net_accept_many(struct work_struct *work) { struct socket *sock = o2net_listen_sock; - while (o2net_accept_one(sock) == 0) + int more; + int err; + + /* + * It is critical to note that due to interrupt moderation + * at the network driver level, we can't assume to get a + * softIRQ for every single conn since tcp SYN packets + * can arrive back-to-back, and therefore many pending + * accepts may result in just 1 softIRQ. If we terminate + * the o2net_accept_one() loop upon seeing an err, what happens + * to the rest of the conns in the queue? If no new SYN + * arrives for hours, no softIRQ will be delivered, + * and the connections will just sit in the queue. + */ + + for (;;) { + err = o2net_accept_one(sock, &more); + if (!more) + break; cond_resched(); + } } static void o2net_listen_data_ready(struct sock *sk) diff --git a/fs/ubifs/budget.c b/fs/ubifs/budget.c index e8e01d74dc05..eb997e9c4ab0 100644 --- a/fs/ubifs/budget.c +++ b/fs/ubifs/budget.c @@ -437,7 +437,6 @@ static int calc_dd_growth(const struct ubifs_info *c, */ int ubifs_budget_space(struct ubifs_info *c, struct ubifs_budget_req *req) { - int uninitialized_var(cmt_retries), uninitialized_var(wb_retries); int err, idx_growth, data_growth, dd_growth, retried = 0; ubifs_assert(req->new_page <= 1); diff --git a/fs/ubifs/debug.c b/fs/ubifs/debug.c index 5157b866a853..177b0152fef4 100644 --- a/fs/ubifs/debug.c +++ b/fs/ubifs/debug.c @@ -745,8 +745,10 @@ void ubifs_dump_lprops(struct ubifs_info *c) for (lnum = c->main_first; lnum < c->leb_cnt; lnum++) { err = ubifs_read_one_lp(c, lnum, &lp); - if (err) + if (err) { ubifs_err("cannot read lprops for LEB %d", lnum); + continue; + } ubifs_dump_lprop(c, &lp); } diff --git a/fs/ubifs/file.c b/fs/ubifs/file.c index 4f34dbae823d..0ab7f7dfb98b 100644 --- a/fs/ubifs/file.c +++ b/fs/ubifs/file.c @@ -903,8 +903,9 @@ static int do_writepage(struct page *page, int len) struct ubifs_info *c = inode->i_sb->s_fs_info; #ifdef UBIFS_DEBUG + struct ubifs_inode *ui = ubifs_inode(inode); spin_lock(&ui->ui_lock); - ubifs_assert(page->index <= ui->synced_i_size << PAGE_CACHE_SIZE); + ubifs_assert(page->index <= ui->synced_i_size >> PAGE_CACHE_SHIFT); spin_unlock(&ui->ui_lock); #endif @@ -1525,8 +1526,7 @@ static int ubifs_vm_page_mkwrite(struct vm_area_struct *vma, } wait_for_stable_page(page); - unlock_page(page); - return 0; + return VM_FAULT_LOCKED; out_unlock: unlock_page(page); diff --git a/fs/ubifs/io.c b/fs/ubifs/io.c index e18b9889a51b..2290d5866725 100644 --- a/fs/ubifs/io.c +++ b/fs/ubifs/io.c @@ -988,30 +988,32 @@ int ubifs_read_node(const struct ubifs_info *c, void *buf, int type, int len, return err; if (type != ch->node_type) { - ubifs_err("bad node type (%d but expected %d)", - ch->node_type, type); + ubifs_errc(c, "bad node type (%d but expected %d)", + ch->node_type, type); goto out; } err = ubifs_check_node(c, buf, lnum, offs, 0, 0); if (err) { - ubifs_err("expected node type %d", type); + ubifs_errc(c, "expected node type %d", type); return err; } l = le32_to_cpu(ch->len); if (l != len) { - ubifs_err("bad node length %d, expected %d", l, len); + ubifs_errc(c, "bad node length %d, expected %d", l, len); goto out; } return 0; out: - ubifs_err("bad node at LEB %d:%d, LEB mapping status %d", lnum, offs, - ubi_is_mapped(c->ubi, lnum)); - ubifs_dump_node(c, buf); - dump_stack(); + ubifs_errc(c, "bad node at LEB %d:%d, LEB mapping status %d", lnum, + offs, ubi_is_mapped(c->ubi, lnum)); + if (!c->probing) { + ubifs_dump_node(c, buf); + dump_stack(); + } return -EINVAL; } diff --git a/fs/ubifs/shrinker.c b/fs/ubifs/shrinker.c index f35135e28e96..9a9fb94a41c6 100644 --- a/fs/ubifs/shrinker.c +++ b/fs/ubifs/shrinker.c @@ -128,7 +128,6 @@ static int shrink_tnc(struct ubifs_info *c, int nr, int age, int *contention) freed = ubifs_destroy_tnc_subtree(znode); atomic_long_sub(freed, &ubifs_clean_zn_cnt); atomic_long_sub(freed, &c->clean_zn_cnt); - ubifs_assert(atomic_long_read(&c->clean_zn_cnt) >= 0); total_freed += freed; znode = zprev; } diff --git a/fs/ubifs/super.c b/fs/ubifs/super.c index a81c7b556896..3904c8574ef9 100644 --- a/fs/ubifs/super.c +++ b/fs/ubifs/super.c @@ -1149,6 +1149,9 @@ static int mount_ubifs(struct ubifs_info *c) size_t sz; c->ro_mount = !!(c->vfs_sb->s_flags & MS_RDONLY); + /* Suppress error messages while probing if MS_SILENT is set */ + c->probing = !!(c->vfs_sb->s_flags & MS_SILENT); + err = init_constants_early(c); if (err) return err; @@ -1214,6 +1217,8 @@ static int mount_ubifs(struct ubifs_info *c) if (err) goto out_free; + c->probing = 0; + /* * Make sure the compressor which is set as default in the superblock * or overridden by mount options is actually compiled in. diff --git a/fs/ubifs/tnc.c b/fs/ubifs/tnc.c index 9083bc7ed4ae..8a40cf9c02d7 100644 --- a/fs/ubifs/tnc.c +++ b/fs/ubifs/tnc.c @@ -2859,10 +2859,11 @@ void ubifs_tnc_close(struct ubifs_info *c) { tnc_destroy_cnext(c); if (c->zroot.znode) { - long n; + long n, freed; - ubifs_destroy_tnc_subtree(c->zroot.znode); n = atomic_long_read(&c->clean_zn_cnt); + freed = ubifs_destroy_tnc_subtree(c->zroot.znode); + ubifs_assert(freed == n); atomic_long_sub(n, &ubifs_clean_zn_cnt); } kfree(c->gap_lebs); diff --git a/fs/ubifs/ubifs.h b/fs/ubifs/ubifs.h index e8c8cfe1435c..c1f71fe17cc0 100644 --- a/fs/ubifs/ubifs.h +++ b/fs/ubifs/ubifs.h @@ -51,6 +51,15 @@ #define ubifs_warn(fmt, ...) \ pr_warn("UBIFS warning (pid %d): %s: " fmt "\n", \ current->pid, __func__, ##__VA_ARGS__) +/* + * A variant of 'ubifs_err()' which takes the UBIFS file-sytem description + * object as an argument. + */ +#define ubifs_errc(c, fmt, ...) \ + do { \ + if (!(c)->probing) \ + ubifs_err(fmt, ##__VA_ARGS__); \ + } while (0) /* UBIFS file system VFS magic number */ #define UBIFS_SUPER_MAGIC 0x24051905 @@ -1209,6 +1218,7 @@ struct ubifs_debug_info; * @need_recovery: %1 if the file-system needs recovery * @replaying: %1 during journal replay * @mounting: %1 while mounting + * @probing: %1 while attempting to mount if MS_SILENT mount flag is set * @remounting_rw: %1 while re-mounting from R/O mode to R/W mode * @replay_list: temporary list used during journal replay * @replay_buds: list of buds to replay @@ -1441,6 +1451,7 @@ struct ubifs_info { unsigned int replaying:1; unsigned int mounting:1; unsigned int remounting_rw:1; + unsigned int probing:1; struct list_head replay_list; struct list_head replay_buds; unsigned long long cs_sqnum; diff --git a/fs/xfs/xfs_ioctl.c b/fs/xfs/xfs_ioctl.c index 0b18776b075e..6152cbe353e8 100644 --- a/fs/xfs/xfs_ioctl.c +++ b/fs/xfs/xfs_ioctl.c @@ -1215,7 +1215,7 @@ xfs_ioctl_setattr( * cleared upon successful return from chown() */ if ((ip->i_d.di_mode & (S_ISUID|S_ISGID)) && - !inode_capable(VFS_I(ip), CAP_FSETID)) + !capable_wrt_inode_uidgid(VFS_I(ip), CAP_FSETID)) ip->i_d.di_mode &= ~(S_ISUID|S_ISGID); /* diff --git a/include/linux/ahci_platform.h b/include/linux/ahci_platform.h index 1f16d502600c..6dfd51a04d77 100644 --- a/include/linux/ahci_platform.h +++ b/include/linux/ahci_platform.h @@ -44,6 +44,7 @@ struct ahci_host_priv *ahci_platform_get_resources( int ahci_platform_init_host(struct platform_device *pdev, struct ahci_host_priv *hpriv, const struct ata_port_info *pi_template, + unsigned long host_flags, unsigned int force_port_map, unsigned int mask_port_map); diff --git a/include/linux/amba/xilinx_dma.h b/include/linux/amba/xilinx_dma.h new file mode 100644 index 000000000000..34b98f276ed0 --- /dev/null +++ b/include/linux/amba/xilinx_dma.h @@ -0,0 +1,47 @@ +/* + * Xilinx DMA Engine drivers support header file + * + * Copyright (C) 2010-2014 Xilinx, Inc. All rights reserved. + * + * This is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2 of the License, or + * (at your option) any later version. + */ + +#ifndef __DMA_XILINX_DMA_H +#define __DMA_XILINX_DMA_H + +#include <linux/dma-mapping.h> +#include <linux/dmaengine.h> + +/** + * struct xilinx_vdma_config - VDMA Configuration structure + * @frm_dly: Frame delay + * @gen_lock: Whether in gen-lock mode + * @master: Master that it syncs to + * @frm_cnt_en: Enable frame count enable + * @park: Whether wants to park + * @park_frm: Frame to park on + * @coalesc: Interrupt coalescing threshold + * @delay: Delay counter + * @reset: Reset Channel + * @ext_fsync: External Frame Sync source + */ +struct xilinx_vdma_config { + int frm_dly; + int gen_lock; + int master; + int frm_cnt_en; + int park; + int park_frm; + int coalesc; + int delay; + int reset; + int ext_fsync; +}; + +int xilinx_vdma_channel_set_config(struct dma_chan *dchan, + struct xilinx_vdma_config *cfg); + +#endif diff --git a/include/linux/capability.h b/include/linux/capability.h index a6ee1f9a5018..84b13ad67c1c 100644 --- a/include/linux/capability.h +++ b/include/linux/capability.h @@ -210,7 +210,7 @@ extern bool has_ns_capability_noaudit(struct task_struct *t, struct user_namespace *ns, int cap); extern bool capable(int cap); extern bool ns_capable(struct user_namespace *ns, int cap); -extern bool inode_capable(const struct inode *inode, int cap); +extern bool capable_wrt_inode_uidgid(const struct inode *inode, int cap); extern bool file_ns_capable(const struct file *file, struct user_namespace *ns, int cap); /* audit system wants to get cap info from files as well */ diff --git a/include/linux/cgroup.h b/include/linux/cgroup.h index d60904b9e505..8a111dd42d7a 100644 --- a/include/linux/cgroup.h +++ b/include/linux/cgroup.h @@ -21,6 +21,7 @@ #include <linux/percpu-refcount.h> #include <linux/seq_file.h> #include <linux/kernfs.h> +#include <linux/wait.h> #ifdef CONFIG_CGROUPS @@ -47,21 +48,45 @@ enum cgroup_subsys_id { }; #undef SUBSYS -/* Per-subsystem/per-cgroup state maintained by the system. */ +/* + * Per-subsystem/per-cgroup state maintained by the system. This is the + * fundamental structural building block that controllers deal with. + * + * Fields marked with "PI:" are public and immutable and may be accessed + * directly without synchronization. + */ struct cgroup_subsys_state { - /* the cgroup that this css is attached to */ + /* PI: the cgroup that this css is attached to */ struct cgroup *cgroup; - /* the cgroup subsystem that this css is attached to */ + /* PI: the cgroup subsystem that this css is attached to */ struct cgroup_subsys *ss; /* reference count - access via css_[try]get() and css_put() */ struct percpu_ref refcnt; - /* the parent css */ + /* PI: the parent css */ struct cgroup_subsys_state *parent; - unsigned long flags; + /* siblings list anchored at the parent's ->children */ + struct list_head sibling; + struct list_head children; + + /* + * PI: Subsys-unique ID. 0 is unused and root is always 1. The + * matching css can be looked up using css_from_id(). + */ + int id; + + unsigned int flags; + + /* + * Monotonically increasing unique serial number which defines a + * uniform order among all csses. It's guaranteed that all + * ->children lists are in the ascending order of ->serial_nr and + * used to allow interrupting and resuming iterations. + */ + u64 serial_nr; /* percpu_ref killing and RCU release */ struct rcu_head rcu_head; @@ -70,8 +95,9 @@ struct cgroup_subsys_state { /* bits in struct cgroup_subsys_state flags field */ enum { - CSS_ROOT = (1 << 0), /* this CSS is the root of the subsystem */ + CSS_NO_REF = (1 << 0), /* no reference counting for this css */ CSS_ONLINE = (1 << 1), /* between ->css_online() and ->css_offline() */ + CSS_RELEASED = (1 << 2), /* refcnt reached zero, released */ }; /** @@ -82,8 +108,7 @@ enum { */ static inline void css_get(struct cgroup_subsys_state *css) { - /* We don't need to reference count the root state */ - if (!(css->flags & CSS_ROOT)) + if (!(css->flags & CSS_NO_REF)) percpu_ref_get(&css->refcnt); } @@ -91,35 +116,51 @@ static inline void css_get(struct cgroup_subsys_state *css) * css_tryget - try to obtain a reference on the specified css * @css: target css * - * Obtain a reference on @css if it's alive. The caller naturally needs to - * ensure that @css is accessible but doesn't have to be holding a + * Obtain a reference on @css unless it already has reached zero and is + * being released. This function doesn't care whether @css is on or + * offline. The caller naturally needs to ensure that @css is accessible + * but doesn't have to be holding a reference on it - IOW, RCU protected + * access is good enough for this function. Returns %true if a reference + * count was successfully obtained; %false otherwise. + */ +static inline bool css_tryget(struct cgroup_subsys_state *css) +{ + if (!(css->flags & CSS_NO_REF)) + return percpu_ref_tryget(&css->refcnt); + return true; +} + +/** + * css_tryget_online - try to obtain a reference on the specified css if online + * @css: target css + * + * Obtain a reference on @css if it's online. The caller naturally needs + * to ensure that @css is accessible but doesn't have to be holding a * reference on it - IOW, RCU protected access is good enough for this * function. Returns %true if a reference count was successfully obtained; * %false otherwise. */ -static inline bool css_tryget(struct cgroup_subsys_state *css) +static inline bool css_tryget_online(struct cgroup_subsys_state *css) { - if (css->flags & CSS_ROOT) - return true; - return percpu_ref_tryget(&css->refcnt); + if (!(css->flags & CSS_NO_REF)) + return percpu_ref_tryget_live(&css->refcnt); + return true; } /** * css_put - put a css reference * @css: target css * - * Put a reference obtained via css_get() and css_tryget(). + * Put a reference obtained via css_get() and css_tryget_online(). */ static inline void css_put(struct cgroup_subsys_state *css) { - if (!(css->flags & CSS_ROOT)) + if (!(css->flags & CSS_NO_REF)) percpu_ref_put(&css->refcnt); } /* bits in struct cgroup flags field */ enum { - /* Control Group is dead */ - CGRP_DEAD, /* * Control Group has previously had a child cgroup or a task, * but no longer (only if CGRP_NOTIFY_ON_RELEASE is set) @@ -133,48 +174,37 @@ enum { * specified at mount time and thus is implemented here. */ CGRP_CPUSET_CLONE_CHILDREN, - /* see the comment above CGRP_ROOT_SANE_BEHAVIOR for details */ - CGRP_SANE_BEHAVIOR, }; struct cgroup { + /* self css with NULL ->ss, points back to this cgroup */ + struct cgroup_subsys_state self; + unsigned long flags; /* "unsigned long" so bitops work */ /* * idr allocated in-hierarchy ID. * - * The ID of the root cgroup is always 0, and a new cgroup - * will be assigned with a smallest available ID. + * ID 0 is not used, the ID of the root cgroup is always 1, and a + * new cgroup will be assigned with a smallest available ID. * * Allocating/Removing ID must be protected by cgroup_mutex. */ int id; - /* the number of attached css's */ - int nr_css; - - atomic_t refcnt; - /* - * We link our 'sibling' struct into our parent's 'children'. - * Our children link their 'sibling' into our 'children'. + * If this cgroup contains any tasks, it contributes one to + * populated_cnt. All children with non-zero popuplated_cnt of + * their own contribute one. The count is zero iff there's no task + * in this cgroup or its subtree. */ - struct list_head sibling; /* my parent's children */ - struct list_head children; /* my children */ + int populated_cnt; - struct cgroup *parent; /* my parent */ struct kernfs_node *kn; /* cgroup kernfs entry */ + struct kernfs_node *populated_kn; /* kn for "cgroup.subtree_populated" */ - /* - * Monotonically increasing unique serial number which defines a - * uniform order among all cgroups. It's guaranteed that all - * ->children lists are in the ascending order of ->serial_nr. - * It's used to allow interrupting and resuming iterations. - */ - u64 serial_nr; - - /* The bitmask of subsystems attached to this cgroup */ - unsigned long subsys_mask; + /* the bitmask of subsystems enabled on the child cgroups */ + unsigned int child_subsys_mask; /* Private pointers for each registered subsystem */ struct cgroup_subsys_state __rcu *subsys[CGROUP_SUBSYS_COUNT]; @@ -188,6 +218,15 @@ struct cgroup { struct list_head cset_links; /* + * On the default hierarchy, a css_set for a cgroup with some + * susbsys disabled will point to css's which are associated with + * the closest ancestor which has the subsys enabled. The + * following lists all css_sets which point to this cgroup's css + * for the given subsystem. + */ + struct list_head e_csets[CGROUP_SUBSYS_COUNT]; + + /* * Linked list running through all cgroups that can * potentially be reaped by the release agent. Protected by * release_list_lock @@ -201,12 +240,8 @@ struct cgroup { struct list_head pidlists; struct mutex pidlist_mutex; - /* dummy css with NULL ->ss, points back to this cgroup */ - struct cgroup_subsys_state dummy_css; - - /* For css percpu_ref killing and RCU-protected deletion */ - struct rcu_head rcu_head; - struct work_struct destroy_work; + /* used to wait for offlining of csses */ + wait_queue_head_t offline_waitq; }; #define MAX_CGROUP_ROOT_NAMELEN 64 @@ -250,6 +285,12 @@ enum { * * - "cgroup.clone_children" is removed. * + * - "cgroup.subtree_populated" is available. Its value is 0 if + * the cgroup and its descendants contain no task; otherwise, 1. + * The file also generates kernfs notification which can be + * monitored through poll and [di]notify when the value of the + * file changes. + * * - If mount is requested with sane_behavior but without any * subsystem, the default unified hierarchy is mounted. * @@ -264,6 +305,8 @@ enum { * the flag is not created. * * - blkcg: blk-throttle becomes properly hierarchical. + * + * - debug: disallowed on the default hierarchy. */ CGRP_ROOT_SANE_BEHAVIOR = (1 << 0), @@ -282,6 +325,9 @@ enum { struct cgroup_root { struct kernfs_root *kf_root; + /* The bitmask of subsystems attached to this hierarchy */ + unsigned int subsys_mask; + /* Unique id for this hierarchy. */ int hierarchy_id; @@ -295,7 +341,7 @@ struct cgroup_root { struct list_head root_list; /* Hierarchy-specific flags */ - unsigned long flags; + unsigned int flags; /* IDs for cgroups in this hierarchy */ struct idr cgroup_idr; @@ -342,6 +388,9 @@ struct css_set { */ struct list_head cgrp_links; + /* the default cgroup associated with this css_set */ + struct cgroup *dfl_cgrp; + /* * Set of subsystem states, one for each subsystem. This array is * immutable after creation apart from the init_css_set during @@ -366,6 +415,15 @@ struct css_set { struct cgroup *mg_src_cgrp; struct css_set *mg_dst_cset; + /* + * On the default hierarhcy, ->subsys[ssid] may point to a css + * attached to an ancestor instead of the cgroup this css_set is + * associated with. The following node is anchored at + * ->subsys[ssid]->cgroup->e_csets[ssid] and provides a way to + * iterate through all css's attached to a given cgroup. + */ + struct list_head e_cset_node[CGROUP_SUBSYS_COUNT]; + /* For RCU-protected deletion */ struct rcu_head rcu_head; }; @@ -405,8 +463,7 @@ struct cftype { /* * The maximum length of string, excluding trailing nul, that can - * be passed to write_string. If < PAGE_SIZE-1, PAGE_SIZE-1 is - * assumed. + * be passed to write. If < PAGE_SIZE-1, PAGE_SIZE-1 is assumed. */ size_t max_write_len; @@ -453,19 +510,13 @@ struct cftype { s64 val); /* - * write_string() is passed a nul-terminated kernelspace - * buffer of maximum length determined by max_write_len. - * Returns 0 or -ve error code. - */ - int (*write_string)(struct cgroup_subsys_state *css, struct cftype *cft, - char *buffer); - /* - * trigger() callback can be used to get some kick from the - * userspace, when the actual string written is not important - * at all. The private field can be used to determine the - * kick type for multiplexing. + * write() is the generic write callback which maps directly to + * kernfs write operation and overrides all other operations. + * Maximum write size is determined by ->max_write_len. Use + * of_css/cft() to access the associated css and cft. */ - int (*trigger)(struct cgroup_subsys_state *css, unsigned int event); + ssize_t (*write)(struct kernfs_open_file *of, + char *buf, size_t nbytes, loff_t off); #ifdef CONFIG_DEBUG_LOCK_ALLOC struct lock_class_key lockdep_key; @@ -504,14 +555,24 @@ static inline ino_t cgroup_ino(struct cgroup *cgrp) return 0; } -static inline struct cftype *seq_cft(struct seq_file *seq) +/* cft/css accessors for cftype->write() operation */ +static inline struct cftype *of_cft(struct kernfs_open_file *of) { - struct kernfs_open_file *of = seq->private; - return of->kn->priv; } -struct cgroup_subsys_state *seq_css(struct seq_file *seq); +struct cgroup_subsys_state *of_css(struct kernfs_open_file *of); + +/* cft/css accessors for cftype->seq_*() operations */ +static inline struct cftype *seq_cft(struct seq_file *seq) +{ + return of_cft(seq->private); +} + +static inline struct cgroup_subsys_state *seq_css(struct seq_file *seq) +{ + return of_css(seq->private); +} /* * Name / path handling functions. All are thin wrappers around the kernfs @@ -612,6 +673,9 @@ struct cgroup_subsys { /* link to parent, protected by cgroup_lock() */ struct cgroup_root *root; + /* idr for css->id */ + struct idr css_idr; + /* * List of cftypes. Each entry is the first entry of an array * terminated by zero length name. @@ -627,19 +691,6 @@ struct cgroup_subsys { #undef SUBSYS /** - * css_parent - find the parent css - * @css: the target cgroup_subsys_state - * - * Return the parent css of @css. This function is guaranteed to return - * non-NULL parent as long as @css isn't the root. - */ -static inline -struct cgroup_subsys_state *css_parent(struct cgroup_subsys_state *css) -{ - return css->parent; -} - -/** * task_css_set_check - obtain a task's css_set with extra access conditions * @task: the task to obtain css_set for * @__c: extra condition expression to be passed to rcu_dereference_check() @@ -731,14 +782,14 @@ struct cgroup_subsys_state *css_from_id(int id, struct cgroup_subsys *ss); * @pos: the css * to use as the loop cursor * @parent: css whose children to walk * - * Walk @parent's children. Must be called under rcu_read_lock(). A child - * css which hasn't finished ->css_online() or already has finished - * ->css_offline() may show up during traversal and it's each subsystem's - * responsibility to verify that each @pos is alive. + * Walk @parent's children. Must be called under rcu_read_lock(). * - * If a subsystem synchronizes against the parent in its ->css_online() and - * before starting iterating, a css which finished ->css_online() is - * guaranteed to be visible in the future iterations. + * If a subsystem synchronizes ->css_online() and the start of iteration, a + * css which finished ->css_online() is guaranteed to be visible in the + * future iterations and will stay visible until the last reference is put. + * A css which hasn't finished ->css_online() or already finished + * ->css_offline() may show up during traversal. It's each subsystem's + * responsibility to synchronize against on/offlining. * * It is allowed to temporarily drop RCU read lock during iteration. The * caller is responsible for ensuring that @pos remains accessible until @@ -761,17 +812,16 @@ css_rightmost_descendant(struct cgroup_subsys_state *pos); * @root: css whose descendants to walk * * Walk @root's descendants. @root is included in the iteration and the - * first node to be visited. Must be called under rcu_read_lock(). A - * descendant css which hasn't finished ->css_online() or already has - * finished ->css_offline() may show up during traversal and it's each - * subsystem's responsibility to verify that each @pos is alive. + * first node to be visited. Must be called under rcu_read_lock(). * - * If a subsystem synchronizes against the parent in its ->css_online() and - * before starting iterating, and synchronizes against @pos on each - * iteration, any descendant css which finished ->css_online() is - * guaranteed to be visible in the future iterations. + * If a subsystem synchronizes ->css_online() and the start of iteration, a + * css which finished ->css_online() is guaranteed to be visible in the + * future iterations and will stay visible until the last reference is put. + * A css which hasn't finished ->css_online() or already finished + * ->css_offline() may show up during traversal. It's each subsystem's + * responsibility to synchronize against on/offlining. * - * In other words, the following guarantees that a descendant can't escape + * For example, the following guarantees that a descendant can't escape * state updates of its ancestors. * * my_online(@css) @@ -827,18 +877,34 @@ css_next_descendant_post(struct cgroup_subsys_state *pos, * * Similar to css_for_each_descendant_pre() but performs post-order * traversal instead. @root is included in the iteration and the last - * node to be visited. Note that the walk visibility guarantee described - * in pre-order walk doesn't apply the same to post-order walks. + * node to be visited. + * + * If a subsystem synchronizes ->css_online() and the start of iteration, a + * css which finished ->css_online() is guaranteed to be visible in the + * future iterations and will stay visible until the last reference is put. + * A css which hasn't finished ->css_online() or already finished + * ->css_offline() may show up during traversal. It's each subsystem's + * responsibility to synchronize against on/offlining. + * + * Note that the walk visibility guarantee example described in pre-order + * walk doesn't apply the same to post-order walks. */ #define css_for_each_descendant_post(pos, css) \ for ((pos) = css_next_descendant_post(NULL, (css)); (pos); \ (pos) = css_next_descendant_post((pos), (css))) +bool css_has_online_children(struct cgroup_subsys_state *css); + /* A css_task_iter should be treated as an opaque object */ struct css_task_iter { - struct cgroup_subsys_state *origin_css; - struct list_head *cset_link; - struct list_head *task; + struct cgroup_subsys *ss; + + struct list_head *cset_pos; + struct list_head *cset_head; + + struct list_head *task_pos; + struct list_head *tasks_head; + struct list_head *mg_tasks_head; }; void css_task_iter_start(struct cgroup_subsys_state *css, @@ -849,8 +915,8 @@ void css_task_iter_end(struct css_task_iter *it); int cgroup_attach_task_all(struct task_struct *from, struct task_struct *); int cgroup_transfer_tasks(struct cgroup *to, struct cgroup *from); -struct cgroup_subsys_state *css_tryget_from_dir(struct dentry *dentry, - struct cgroup_subsys *ss); +struct cgroup_subsys_state *css_tryget_online_from_dir(struct dentry *dentry, + struct cgroup_subsys *ss); #else /* !CONFIG_CGROUPS */ diff --git a/include/linux/cgroup_subsys.h b/include/linux/cgroup_subsys.h index 768fe44e19f0..98c4f9b12b03 100644 --- a/include/linux/cgroup_subsys.h +++ b/include/linux/cgroup_subsys.h @@ -7,10 +7,6 @@ SUBSYS(cpuset) #endif -#if IS_ENABLED(CONFIG_CGROUP_DEBUG) -SUBSYS(debug) -#endif - #if IS_ENABLED(CONFIG_CGROUP_SCHED) SUBSYS(cpu) #endif @@ -50,6 +46,13 @@ SUBSYS(net_prio) #if IS_ENABLED(CONFIG_CGROUP_HUGETLB) SUBSYS(hugetlb) #endif + +/* + * The following subsystems are not supported on the default hierarchy. + */ +#if IS_ENABLED(CONFIG_CGROUP_DEBUG) +SUBSYS(debug) +#endif /* * DO NOT ADD ANY SUBSYSTEM WITHOUT EXPLICIT ACKS FROM CGROUP MAINTAINERS. */ diff --git a/include/linux/cpuidle.h b/include/linux/cpuidle.h index c51a436135c4..25e0df6155a4 100644 --- a/include/linux/cpuidle.h +++ b/include/linux/cpuidle.h @@ -84,6 +84,7 @@ struct cpuidle_device { }; DECLARE_PER_CPU(struct cpuidle_device *, cpuidle_devices); +DECLARE_PER_CPU(struct cpuidle_device, cpuidle_dev); /** * cpuidle_get_last_residency - retrieves the last state's residency time diff --git a/include/linux/dmaengine.h b/include/linux/dmaengine.h index 72cb0ddb9678..d2c5cc7c583c 100644 --- a/include/linux/dmaengine.h +++ b/include/linux/dmaengine.h @@ -292,7 +292,7 @@ struct dma_chan_dev { }; /** - * enum dma_slave_buswidth - defines bus with of the DMA slave + * enum dma_slave_buswidth - defines bus width of the DMA slave * device, source or target buses */ enum dma_slave_buswidth { diff --git a/include/linux/f2fs_fs.h b/include/linux/f2fs_fs.h index df53e1753a76..6ff0b0b42d47 100644 --- a/include/linux/f2fs_fs.h +++ b/include/linux/f2fs_fs.h @@ -19,6 +19,7 @@ #define F2FS_LOG_SECTORS_PER_BLOCK 3 /* 4KB: F2FS_BLKSIZE */ #define F2FS_BLKSIZE 4096 /* support only 4KB block */ #define F2FS_MAX_EXTENSION 64 /* # of extension entries */ +#define F2FS_BLK_ALIGN(x) (((x) + F2FS_BLKSIZE - 1) / F2FS_BLKSIZE) #define NULL_ADDR ((block_t)0) /* used as block_t addresses */ #define NEW_ADDR ((block_t)-1) /* used as block_t addresses */ @@ -75,6 +76,7 @@ struct f2fs_super_block { __le16 volume_name[512]; /* volume name */ __le32 extension_count; /* # of extensions below */ __u8 extension_list[F2FS_MAX_EXTENSION][8]; /* extension array */ + __le32 cp_payload; } __packed; /* @@ -146,6 +148,9 @@ struct f2fs_extent { #define ADDRS_PER_BLOCK 1018 /* Address Pointers in a Direct Block */ #define NIDS_PER_BLOCK 1018 /* Node IDs in an Indirect Block */ +#define ADDRS_PER_PAGE(page, fi) \ + (IS_INODE(page) ? ADDRS_PER_INODE(fi) : ADDRS_PER_BLOCK) + #define NODE_DIR1_BLOCK (DEF_ADDRS_PER_INODE + 1) #define NODE_DIR2_BLOCK (DEF_ADDRS_PER_INODE + 2) #define NODE_IND1_BLOCK (DEF_ADDRS_PER_INODE + 3) @@ -391,6 +396,9 @@ typedef __le32 f2fs_hash_t; /* MAX level for dir lookup */ #define MAX_DIR_HASH_DEPTH 63 +/* MAX buckets in one level of dir */ +#define MAX_DIR_BUCKETS (1 << ((MAX_DIR_HASH_DEPTH / 2) - 1)) + #define SIZE_OF_DIR_ENTRY 11 /* by byte */ #define SIZE_OF_DENTRY_BITMAP ((NR_DENTRY_IN_BLOCK + BITS_PER_BYTE - 1) / \ BITS_PER_BYTE) diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index 2018751cad9e..404a686a3644 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -62,9 +62,6 @@ typedef void (*ftrace_func_t)(unsigned long ip, unsigned long parent_ip, * set in the flags member. * * ENABLED - set/unset when ftrace_ops is registered/unregistered - * GLOBAL - set manualy by ftrace_ops user to denote the ftrace_ops - * is part of the global tracers sharing the same filter - * via set_ftrace_* debugfs files. * DYNAMIC - set when ftrace_ops is registered to denote dynamically * allocated ftrace_ops which need special care * CONTROL - set manualy by ftrace_ops user to denote the ftrace_ops @@ -96,15 +93,14 @@ typedef void (*ftrace_func_t)(unsigned long ip, unsigned long parent_ip, */ enum { FTRACE_OPS_FL_ENABLED = 1 << 0, - FTRACE_OPS_FL_GLOBAL = 1 << 1, - FTRACE_OPS_FL_DYNAMIC = 1 << 2, - FTRACE_OPS_FL_CONTROL = 1 << 3, - FTRACE_OPS_FL_SAVE_REGS = 1 << 4, - FTRACE_OPS_FL_SAVE_REGS_IF_SUPPORTED = 1 << 5, - FTRACE_OPS_FL_RECURSION_SAFE = 1 << 6, - FTRACE_OPS_FL_STUB = 1 << 7, - FTRACE_OPS_FL_INITIALIZED = 1 << 8, - FTRACE_OPS_FL_DELETED = 1 << 9, + FTRACE_OPS_FL_DYNAMIC = 1 << 1, + FTRACE_OPS_FL_CONTROL = 1 << 2, + FTRACE_OPS_FL_SAVE_REGS = 1 << 3, + FTRACE_OPS_FL_SAVE_REGS_IF_SUPPORTED = 1 << 4, + FTRACE_OPS_FL_RECURSION_SAFE = 1 << 5, + FTRACE_OPS_FL_STUB = 1 << 6, + FTRACE_OPS_FL_INITIALIZED = 1 << 7, + FTRACE_OPS_FL_DELETED = 1 << 8, }; /* @@ -366,14 +362,12 @@ enum { * IGNORE - The function is already what we want it to be * MAKE_CALL - Start tracing the function * MODIFY_CALL - Stop saving regs for the function - * MODIFY_CALL_REGS - Start saving regs for the function * MAKE_NOP - Stop tracing the function */ enum { FTRACE_UPDATE_IGNORE, FTRACE_UPDATE_MAKE_CALL, FTRACE_UPDATE_MODIFY_CALL, - FTRACE_UPDATE_MODIFY_CALL_REGS, FTRACE_UPDATE_MAKE_NOP, }; @@ -404,6 +398,8 @@ int ftrace_update_record(struct dyn_ftrace *rec, int enable); int ftrace_test_record(struct dyn_ftrace *rec, int enable); void ftrace_run_stop_machine(int command); unsigned long ftrace_location(unsigned long ip); +unsigned long ftrace_get_addr_new(struct dyn_ftrace *rec); +unsigned long ftrace_get_addr_curr(struct dyn_ftrace *rec); extern ftrace_func_t ftrace_trace_function; diff --git a/include/linux/ftrace_event.h b/include/linux/ftrace_event.h index d16da3e53bc7..cff3106ffe2c 100644 --- a/include/linux/ftrace_event.h +++ b/include/linux/ftrace_event.h @@ -38,6 +38,9 @@ const char *ftrace_print_symbols_seq_u64(struct trace_seq *p, *symbol_array); #endif +const char *ftrace_print_bitmask_seq(struct trace_seq *p, void *bitmask_ptr, + unsigned int bitmask_size); + const char *ftrace_print_hex_seq(struct trace_seq *p, const unsigned char *buf, int len); diff --git a/include/linux/gpio_keys.h b/include/linux/gpio_keys.h index a7e977ff4abf..8b622468952c 100644 --- a/include/linux/gpio_keys.h +++ b/include/linux/gpio_keys.h @@ -3,29 +3,53 @@ struct device; +/** + * struct gpio_keys_button - configuration parameters + * @code: input event code (KEY_*, SW_*) + * @gpio: %-1 if this key does not support gpio + * @active_low: %true indicates that button is considered + * depressed when gpio is low + * @desc: label that will be attached to button's gpio + * @type: input event type (%EV_KEY, %EV_SW, %EV_ABS) + * @wakeup: configure the button as a wake-up source + * @debounce_interval: debounce ticks interval in msecs + * @can_disable: %true indicates that userspace is allowed to + * disable button via sysfs + * @value: axis value for %EV_ABS + * @irq: Irq number in case of interrupt keys + */ struct gpio_keys_button { - /* Configuration parameters */ - unsigned int code; /* input event code (KEY_*, SW_*) */ - int gpio; /* -1 if this key does not support gpio */ + unsigned int code; + int gpio; int active_low; const char *desc; - unsigned int type; /* input event type (EV_KEY, EV_SW, EV_ABS) */ - int wakeup; /* configure the button as a wake-up source */ - int debounce_interval; /* debounce ticks interval in msecs */ + unsigned int type; + int wakeup; + int debounce_interval; bool can_disable; - int value; /* axis value for EV_ABS */ - unsigned int irq; /* Irq number in case of interrupt keys */ + int value; + unsigned int irq; }; +/** + * struct gpio_keys_platform_data - platform data for gpio_keys driver + * @buttons: pointer to array of &gpio_keys_button structures + * describing buttons attached to the device + * @nbuttons: number of elements in @buttons array + * @poll_interval: polling interval in msecs - for polling driver only + * @rep: enable input subsystem auto repeat + * @enable: platform hook for enabling the device + * @disable: platform hook for disabling the device + * @name: input device name + */ struct gpio_keys_platform_data { struct gpio_keys_button *buttons; int nbuttons; - unsigned int poll_interval; /* polling interval in msecs - - for polling driver only */ - unsigned int rep:1; /* enable input subsystem auto repeat */ + unsigned int poll_interval; + unsigned int rep:1; int (*enable)(struct device *dev); void (*disable)(struct device *dev); - const char *name; /* input device name */ + const char *name; }; #endif diff --git a/include/linux/i2c/atmel_mxt_ts.h b/include/linux/i2c/atmel_mxt_ts.h index 99e379b74398..3891dc1de21c 100644 --- a/include/linux/i2c/atmel_mxt_ts.h +++ b/include/linux/i2c/atmel_mxt_ts.h @@ -15,35 +15,14 @@ #include <linux/types.h> -/* For key_map array */ -#define MXT_NUM_GPIO 4 - -/* Orient */ -#define MXT_NORMAL 0x0 -#define MXT_DIAGONAL 0x1 -#define MXT_HORIZONTAL_FLIP 0x2 -#define MXT_ROTATED_90_COUNTER 0x3 -#define MXT_VERTICAL_FLIP 0x4 -#define MXT_ROTATED_90 0x5 -#define MXT_ROTATED_180 0x6 -#define MXT_DIAGONAL_COUNTER 0x7 - /* The platform data for the Atmel maXTouch touchscreen driver */ struct mxt_platform_data { const u8 *config; size_t config_length; - - unsigned int x_line; - unsigned int y_line; - unsigned int x_size; - unsigned int y_size; - unsigned int blen; - unsigned int threshold; - unsigned int voltage; - unsigned char orient; + u32 config_crc; unsigned long irqflags; - bool is_tp; - const unsigned int key_map[MXT_NUM_GPIO]; + u8 t19_num_keys; + const unsigned int *t19_keymap; }; #endif /* __LINUX_ATMEL_MXT_TS_H */ diff --git a/include/linux/input-polldev.h b/include/linux/input-polldev.h index ce0b72464eb8..2465182670db 100644 --- a/include/linux/input-polldev.h +++ b/include/linux/input-polldev.h @@ -48,9 +48,12 @@ struct input_polled_dev { /* private: */ struct delayed_work work; + + bool devres_managed; }; struct input_polled_dev *input_allocate_polled_device(void); +struct input_polled_dev *devm_input_allocate_polled_device(struct device *dev); void input_free_polled_device(struct input_polled_dev *dev); int input_register_polled_device(struct input_polled_dev *dev); void input_unregister_polled_device(struct input_polled_dev *dev); diff --git a/include/linux/input/pixcir_ts.h b/include/linux/input/pixcir_ts.h index 7163d91c0373..160cf353aa39 100644 --- a/include/linux/input/pixcir_ts.h +++ b/include/linux/input/pixcir_ts.h @@ -1,10 +1,52 @@ #ifndef _PIXCIR_I2C_TS_H #define _PIXCIR_I2C_TS_H +/* + * Register map + */ +#define PIXCIR_REG_POWER_MODE 51 +#define PIXCIR_REG_INT_MODE 52 + +/* + * Power modes: + * active: max scan speed + * idle: lower scan speed with automatic transition to active on touch + * halt: datasheet says sleep but this is more like halt as the chip + * clocks are cut and it can only be brought out of this mode + * using the RESET pin. + */ +enum pixcir_power_mode { + PIXCIR_POWER_ACTIVE, + PIXCIR_POWER_IDLE, + PIXCIR_POWER_HALT, +}; + +#define PIXCIR_POWER_MODE_MASK 0x03 +#define PIXCIR_POWER_ALLOW_IDLE (1UL << 2) + +/* + * Interrupt modes: + * periodical: interrupt is asserted periodicaly + * diff coordinates: interrupt is asserted when coordinates change + * level on touch: interrupt level asserted during touch + * pulse on touch: interrupt pulse asserted druing touch + * + */ +enum pixcir_int_mode { + PIXCIR_INT_PERIODICAL, + PIXCIR_INT_DIFF_COORD, + PIXCIR_INT_LEVEL_TOUCH, + PIXCIR_INT_PULSE_TOUCH, +}; + +#define PIXCIR_INT_MODE_MASK 0x03 +#define PIXCIR_INT_ENABLE (1UL << 3) +#define PIXCIR_INT_POL_HIGH (1UL << 2) + struct pixcir_ts_platform_data { - int (*attb_read_val)(void); int x_max; int y_max; + int gpio_attb; /* GPIO connected to ATTB line */ }; #endif diff --git a/include/linux/input/touchscreen.h b/include/linux/input/touchscreen.h new file mode 100644 index 000000000000..08a5ef6e8f25 --- /dev/null +++ b/include/linux/input/touchscreen.h @@ -0,0 +1,22 @@ +/* + * Copyright (c) 2014 Sebastian Reichel <sre@kernel.org> + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License version 2 as published by + * the Free Software Foundation. + */ + +#ifndef _TOUCHSCREEN_H +#define _TOUCHSCREEN_H + +#include <linux/input.h> + +#ifdef CONFIG_OF +void touchscreen_parse_of_params(struct input_dev *dev); +#else +static inline void touchscreen_parse_of_params(struct input_dev *dev) +{ +} +#endif + +#endif diff --git a/include/linux/key.h b/include/linux/key.h index 3ae45f09589b..017b0826642f 100644 --- a/include/linux/key.h +++ b/include/linux/key.h @@ -309,6 +309,17 @@ static inline key_serial_t key_serial(const struct key *key) extern void key_set_timeout(struct key *, unsigned); +/* + * The permissions required on a key that we're looking up. + */ +#define KEY_NEED_VIEW 0x01 /* Require permission to view attributes */ +#define KEY_NEED_READ 0x02 /* Require permission to read content */ +#define KEY_NEED_WRITE 0x04 /* Require permission to update / modify */ +#define KEY_NEED_SEARCH 0x08 /* Require permission to search (keyring) or find (key) */ +#define KEY_NEED_LINK 0x10 /* Require permission to link */ +#define KEY_NEED_SETATTR 0x20 /* Require permission to change attributes */ +#define KEY_NEED_ALL 0x3f /* All the above permissions */ + /** * key_is_instantiated - Determine if a key has been positively instantiated * @key: The key to check. diff --git a/include/linux/kprobes.h b/include/linux/kprobes.h index 925eaf28fca9..7bd2ad01e39c 100644 --- a/include/linux/kprobes.h +++ b/include/linux/kprobes.h @@ -355,7 +355,7 @@ static inline void reset_current_kprobe(void) static inline struct kprobe_ctlblk *get_kprobe_ctlblk(void) { - return (&__get_cpu_var(kprobe_ctlblk)); + return this_cpu_ptr(&kprobe_ctlblk); } int register_kprobe(struct kprobe *p); diff --git a/include/linux/lockd/lockd.h b/include/linux/lockd/lockd.h index dcaad79f54ed..219d79627c05 100644 --- a/include/linux/lockd/lockd.h +++ b/include/linux/lockd/lockd.h @@ -17,13 +17,13 @@ #include <linux/fs.h> #include <linux/kref.h> #include <linux/utsname.h> -#include <linux/nfsd/nfsfh.h> #include <linux/lockd/bind.h> #include <linux/lockd/xdr.h> #ifdef CONFIG_LOCKD_V4 #include <linux/lockd/xdr4.h> #endif #include <linux/lockd/debug.h> +#include <linux/sunrpc/svc.h> /* * Version string diff --git a/include/linux/mfd/samsung/rtc.h b/include/linux/mfd/samsung/rtc.h index 3e02b768d537..b6401e7661c7 100644 --- a/include/linux/mfd/samsung/rtc.h +++ b/include/linux/mfd/samsung/rtc.h @@ -18,38 +18,38 @@ #ifndef __LINUX_MFD_SEC_RTC_H #define __LINUX_MFD_SEC_RTC_H -enum sec_rtc_reg { - SEC_RTC_SEC, - SEC_RTC_MIN, - SEC_RTC_HOUR, - SEC_RTC_WEEKDAY, - SEC_RTC_DATE, - SEC_RTC_MONTH, - SEC_RTC_YEAR1, - SEC_RTC_YEAR2, - SEC_ALARM0_SEC, - SEC_ALARM0_MIN, - SEC_ALARM0_HOUR, - SEC_ALARM0_WEEKDAY, - SEC_ALARM0_DATE, - SEC_ALARM0_MONTH, - SEC_ALARM0_YEAR1, - SEC_ALARM0_YEAR2, - SEC_ALARM1_SEC, - SEC_ALARM1_MIN, - SEC_ALARM1_HOUR, - SEC_ALARM1_WEEKDAY, - SEC_ALARM1_DATE, - SEC_ALARM1_MONTH, - SEC_ALARM1_YEAR1, - SEC_ALARM1_YEAR2, - SEC_ALARM0_CONF, - SEC_ALARM1_CONF, - SEC_RTC_STATUS, - SEC_WTSR_SMPL_CNTL, - SEC_RTC_UDR_CON, +enum s5m_rtc_reg { + S5M_RTC_SEC, + S5M_RTC_MIN, + S5M_RTC_HOUR, + S5M_RTC_WEEKDAY, + S5M_RTC_DATE, + S5M_RTC_MONTH, + S5M_RTC_YEAR1, + S5M_RTC_YEAR2, + S5M_ALARM0_SEC, + S5M_ALARM0_MIN, + S5M_ALARM0_HOUR, + S5M_ALARM0_WEEKDAY, + S5M_ALARM0_DATE, + S5M_ALARM0_MONTH, + S5M_ALARM0_YEAR1, + S5M_ALARM0_YEAR2, + S5M_ALARM1_SEC, + S5M_ALARM1_MIN, + S5M_ALARM1_HOUR, + S5M_ALARM1_WEEKDAY, + S5M_ALARM1_DATE, + S5M_ALARM1_MONTH, + S5M_ALARM1_YEAR1, + S5M_ALARM1_YEAR2, + S5M_ALARM0_CONF, + S5M_ALARM1_CONF, + S5M_RTC_STATUS, + S5M_WTSR_SMPL_CNTL, + S5M_RTC_UDR_CON, - SEC_RTC_REG_MAX, + S5M_RTC_REG_MAX, }; enum s2mps_rtc_reg { @@ -88,9 +88,9 @@ enum s2mps_rtc_reg { #define HOUR_12 (1 << 7) #define HOUR_AMPM (1 << 6) #define HOUR_PM (1 << 5) -#define ALARM0_STATUS (1 << 1) -#define ALARM1_STATUS (1 << 2) -#define UPDATE_AD (1 << 0) +#define S5M_ALARM0_STATUS (1 << 1) +#define S5M_ALARM1_STATUS (1 << 2) +#define S5M_UPDATE_AD (1 << 0) #define S2MPS_ALARM0_STATUS (1 << 2) #define S2MPS_ALARM1_STATUS (1 << 1) @@ -101,16 +101,26 @@ enum s2mps_rtc_reg { #define MODEL24_SHIFT 1 #define MODEL24_MASK (1 << MODEL24_SHIFT) /* RTC Update Register1 */ -#define RTC_UDR_SHIFT 0 -#define RTC_UDR_MASK (1 << RTC_UDR_SHIFT) +#define S5M_RTC_UDR_SHIFT 0 +#define S5M_RTC_UDR_MASK (1 << S5M_RTC_UDR_SHIFT) #define S2MPS_RTC_WUDR_SHIFT 4 #define S2MPS_RTC_WUDR_MASK (1 << S2MPS_RTC_WUDR_SHIFT) #define S2MPS_RTC_RUDR_SHIFT 0 #define S2MPS_RTC_RUDR_MASK (1 << S2MPS_RTC_RUDR_SHIFT) #define RTC_TCON_SHIFT 1 #define RTC_TCON_MASK (1 << RTC_TCON_SHIFT) -#define RTC_TIME_EN_SHIFT 3 -#define RTC_TIME_EN_MASK (1 << RTC_TIME_EN_SHIFT) +#define S5M_RTC_TIME_EN_SHIFT 3 +#define S5M_RTC_TIME_EN_MASK (1 << S5M_RTC_TIME_EN_SHIFT) +/* + * UDR_T field in S5M_RTC_UDR_CON register determines the time needed + * for updating alarm and time registers. Default is 7.32 ms. + */ +#define S5M_RTC_UDR_T_SHIFT 6 +#define S5M_RTC_UDR_T_MASK (0x3 << S5M_RTC_UDR_T_SHIFT) +#define S5M_RTC_UDR_T_7320_US (0x0 << S5M_RTC_UDR_T_SHIFT) +#define S5M_RTC_UDR_T_1830_US (0x1 << S5M_RTC_UDR_T_SHIFT) +#define S5M_RTC_UDR_T_3660_US (0x2 << S5M_RTC_UDR_T_SHIFT) +#define S5M_RTC_UDR_T_450_US (0x3 << S5M_RTC_UDR_T_SHIFT) /* RTC Hour register */ #define HOUR_PM_SHIFT 6 diff --git a/include/linux/mlx4/device.h b/include/linux/mlx4/device.h index ba87bd21295a..3447bead9620 100644 --- a/include/linux/mlx4/device.h +++ b/include/linux/mlx4/device.h @@ -401,6 +401,7 @@ struct mlx4_caps { int max_rq_desc_sz; int max_qp_init_rdma; int max_qp_dest_rdma; + u32 *qp0_qkey; u32 *qp0_proxy; u32 *qp1_proxy; u32 *qp0_tunnel; @@ -837,7 +838,7 @@ static inline int mlx4_is_slave(struct mlx4_dev *dev) } int mlx4_buf_alloc(struct mlx4_dev *dev, int size, int max_direct, - struct mlx4_buf *buf); + struct mlx4_buf *buf, gfp_t gfp); void mlx4_buf_free(struct mlx4_dev *dev, int size, struct mlx4_buf *buf); static inline void *mlx4_buf_offset(struct mlx4_buf *buf, int offset) { @@ -874,9 +875,10 @@ int mlx4_mw_enable(struct mlx4_dev *dev, struct mlx4_mw *mw); int mlx4_write_mtt(struct mlx4_dev *dev, struct mlx4_mtt *mtt, int start_index, int npages, u64 *page_list); int mlx4_buf_write_mtt(struct mlx4_dev *dev, struct mlx4_mtt *mtt, - struct mlx4_buf *buf); + struct mlx4_buf *buf, gfp_t gfp); -int mlx4_db_alloc(struct mlx4_dev *dev, struct mlx4_db *db, int order); +int mlx4_db_alloc(struct mlx4_dev *dev, struct mlx4_db *db, int order, + gfp_t gfp); void mlx4_db_free(struct mlx4_dev *dev, struct mlx4_db *db); int mlx4_alloc_hwq_res(struct mlx4_dev *dev, struct mlx4_hwq_resources *wqres, @@ -892,7 +894,8 @@ void mlx4_cq_free(struct mlx4_dev *dev, struct mlx4_cq *cq); int mlx4_qp_reserve_range(struct mlx4_dev *dev, int cnt, int align, int *base); void mlx4_qp_release_range(struct mlx4_dev *dev, int base_qpn, int cnt); -int mlx4_qp_alloc(struct mlx4_dev *dev, int qpn, struct mlx4_qp *qp); +int mlx4_qp_alloc(struct mlx4_dev *dev, int qpn, struct mlx4_qp *qp, + gfp_t gfp); void mlx4_qp_free(struct mlx4_dev *dev, struct mlx4_qp *qp); int mlx4_srq_alloc(struct mlx4_dev *dev, u32 pdn, u32 cqn, u16 xrcdn, @@ -1234,4 +1237,8 @@ int mlx4_phys_to_slave_port(struct mlx4_dev *dev, int slave, int port); int mlx4_get_base_gid_ix(struct mlx4_dev *dev, int slave, int port); int mlx4_config_vxlan_port(struct mlx4_dev *dev, __be16 udp_port); +int mlx4_vf_smi_enabled(struct mlx4_dev *dev, int slave, int port); +int mlx4_vf_get_enable_smi_admin(struct mlx4_dev *dev, int slave, int port); +int mlx4_vf_set_enable_smi_admin(struct mlx4_dev *dev, int slave, int port, + int enable); #endif /* MLX4_DEVICE_H */ diff --git a/include/linux/mlx5/driver.h b/include/linux/mlx5/driver.h index 93cef6313e72..2bce4aad2570 100644 --- a/include/linux/mlx5/driver.h +++ b/include/linux/mlx5/driver.h @@ -427,7 +427,6 @@ struct mlx5_core_mr { u64 size; u32 key; u32 pd; - u32 access; }; struct mlx5_core_srq { diff --git a/include/linux/mmc/card.h b/include/linux/mmc/card.h index b73027298b3a..d424b9de3aff 100644 --- a/include/linux/mmc/card.h +++ b/include/linux/mmc/card.h @@ -63,12 +63,12 @@ struct mmc_ext_csd { unsigned int power_off_longtime; /* Units: ms */ u8 power_off_notification; /* state */ unsigned int hs_max_dtr; + unsigned int hs200_max_dtr; #define MMC_HIGH_26_MAX_DTR 26000000 #define MMC_HIGH_52_MAX_DTR 52000000 #define MMC_HIGH_DDR_MAX_DTR 52000000 #define MMC_HS200_MAX_DTR 200000000 unsigned int sectors; - unsigned int card_type; unsigned int hc_erase_size; /* In sectors */ unsigned int hc_erase_timeout; /* In milliseconds */ unsigned int sec_trim_mult; /* Secure trim multiplier */ @@ -110,6 +110,7 @@ struct mmc_ext_csd { u8 raw_pwr_cl_200_360; /* 237 */ u8 raw_pwr_cl_ddr_52_195; /* 238 */ u8 raw_pwr_cl_ddr_52_360; /* 239 */ + u8 raw_pwr_cl_ddr_200_360; /* 253 */ u8 raw_bkops_status; /* 246 */ u8 raw_sectors[4]; /* 212 - 4 bytes */ @@ -194,6 +195,7 @@ struct sdio_cis { }; struct mmc_host; +struct mmc_ios; struct sdio_func; struct sdio_func_tuple; @@ -250,15 +252,11 @@ struct mmc_card { unsigned int state; /* (our) card state */ #define MMC_STATE_PRESENT (1<<0) /* present in sysfs */ #define MMC_STATE_READONLY (1<<1) /* card is read-only */ -#define MMC_STATE_HIGHSPEED (1<<2) /* card is in high speed mode */ -#define MMC_STATE_BLOCKADDR (1<<3) /* card uses block-addressing */ -#define MMC_STATE_HIGHSPEED_DDR (1<<4) /* card is in high speed mode */ -#define MMC_STATE_ULTRAHIGHSPEED (1<<5) /* card is in ultra high speed mode */ -#define MMC_CARD_SDXC (1<<6) /* card is SDXC */ -#define MMC_CARD_REMOVED (1<<7) /* card has been removed */ -#define MMC_STATE_HIGHSPEED_200 (1<<8) /* card is in HS200 mode */ -#define MMC_STATE_DOING_BKOPS (1<<10) /* card is doing BKOPS */ -#define MMC_STATE_SUSPENDED (1<<11) /* card is suspended */ +#define MMC_STATE_BLOCKADDR (1<<2) /* card uses block-addressing */ +#define MMC_CARD_SDXC (1<<3) /* card is SDXC */ +#define MMC_CARD_REMOVED (1<<4) /* card has been removed */ +#define MMC_STATE_DOING_BKOPS (1<<5) /* card is doing BKOPS */ +#define MMC_STATE_SUSPENDED (1<<6) /* card is suspended */ unsigned int quirks; /* card quirks */ #define MMC_QUIRK_LENIENT_FN0 (1<<0) /* allow SDIO FN0 writes outside of the VS CCCR range */ #define MMC_QUIRK_BLKSZ_FOR_BYTE_MODE (1<<1) /* use func->cur_blksize */ @@ -301,6 +299,7 @@ struct mmc_card { struct sdio_func_tuple *tuples; /* unknown common tuples */ unsigned int sd_bus_speed; /* Bus Speed Mode set for the card */ + unsigned int mmc_avail_type; /* supported device type by both host and card */ struct dentry *debugfs_root; struct mmc_part part[MMC_NUM_PHY_PARTITION]; /* physical partitions */ @@ -353,7 +352,7 @@ struct mmc_fixup { #define CID_OEMID_ANY ((unsigned short) -1) #define CID_NAME_ANY (NULL) -#define END_FIXUP { 0 } +#define END_FIXUP { NULL } #define _FIXUP_EXT(_name, _manfid, _oemid, _rev_start, _rev_end, \ _cis_vendor, _cis_device, \ @@ -418,11 +417,7 @@ static inline void __maybe_unused remove_quirk(struct mmc_card *card, int data) #define mmc_card_present(c) ((c)->state & MMC_STATE_PRESENT) #define mmc_card_readonly(c) ((c)->state & MMC_STATE_READONLY) -#define mmc_card_highspeed(c) ((c)->state & MMC_STATE_HIGHSPEED) -#define mmc_card_hs200(c) ((c)->state & MMC_STATE_HIGHSPEED_200) #define mmc_card_blockaddr(c) ((c)->state & MMC_STATE_BLOCKADDR) -#define mmc_card_ddr_mode(c) ((c)->state & MMC_STATE_HIGHSPEED_DDR) -#define mmc_card_uhs(c) ((c)->state & MMC_STATE_ULTRAHIGHSPEED) #define mmc_card_ext_capacity(c) ((c)->state & MMC_CARD_SDXC) #define mmc_card_removed(c) ((c) && ((c)->state & MMC_CARD_REMOVED)) #define mmc_card_doing_bkops(c) ((c)->state & MMC_STATE_DOING_BKOPS) @@ -430,11 +425,7 @@ static inline void __maybe_unused remove_quirk(struct mmc_card *card, int data) #define mmc_card_set_present(c) ((c)->state |= MMC_STATE_PRESENT) #define mmc_card_set_readonly(c) ((c)->state |= MMC_STATE_READONLY) -#define mmc_card_set_highspeed(c) ((c)->state |= MMC_STATE_HIGHSPEED) -#define mmc_card_set_hs200(c) ((c)->state |= MMC_STATE_HIGHSPEED_200) #define mmc_card_set_blockaddr(c) ((c)->state |= MMC_STATE_BLOCKADDR) -#define mmc_card_set_ddr_mode(c) ((c)->state |= MMC_STATE_HIGHSPEED_DDR) -#define mmc_card_set_uhs(c) ((c)->state |= MMC_STATE_ULTRAHIGHSPEED) #define mmc_card_set_ext_capacity(c) ((c)->state |= MMC_CARD_SDXC) #define mmc_card_set_removed(c) ((c)->state |= MMC_CARD_REMOVED) #define mmc_card_set_doing_bkops(c) ((c)->state |= MMC_STATE_DOING_BKOPS) diff --git a/include/linux/mmc/dw_mmc.h b/include/linux/mmc/dw_mmc.h index 6ce7d2cd3c7a..babaea93bca6 100644 --- a/include/linux/mmc/dw_mmc.h +++ b/include/linux/mmc/dw_mmc.h @@ -248,20 +248,6 @@ struct dw_mci_board { /* delay in mS before detecting cards after interrupt */ u32 detect_delay_ms; - int (*init)(u32 slot_id, irq_handler_t , void *); - int (*get_ro)(u32 slot_id); - int (*get_cd)(u32 slot_id); - int (*get_ocr)(u32 slot_id); - int (*get_bus_wd)(u32 slot_id); - /* - * Enable power to selected slot and set voltage to desired level. - * Voltage levels are specified using MMC_VDD_xxx defines defined - * in linux/mmc/host.h file. - */ - void (*setpower)(u32 slot_id, u32 volt); - void (*exit)(u32 slot_id); - void (*select_slot)(u32 slot_id); - struct dw_mci_dma_ops *dma_ops; struct dma_pdata *data; struct block_settings *blk_settings; diff --git a/include/linux/mmc/host.h b/include/linux/mmc/host.h index cb61ea4d6945..7960424d0bc0 100644 --- a/include/linux/mmc/host.h +++ b/include/linux/mmc/host.h @@ -17,6 +17,7 @@ #include <linux/fault-inject.h> #include <linux/mmc/core.h> +#include <linux/mmc/card.h> #include <linux/mmc/pm.h> struct mmc_ios { @@ -58,13 +59,9 @@ struct mmc_ios { #define MMC_TIMING_UHS_SDR50 5 #define MMC_TIMING_UHS_SDR104 6 #define MMC_TIMING_UHS_DDR50 7 -#define MMC_TIMING_MMC_HS200 8 - -#define MMC_SDR_MODE 0 -#define MMC_1_2V_DDR_MODE 1 -#define MMC_1_8V_DDR_MODE 2 -#define MMC_1_2V_SDR_MODE 3 -#define MMC_1_8V_SDR_MODE 4 +#define MMC_TIMING_MMC_DDR52 8 +#define MMC_TIMING_MMC_HS200 9 +#define MMC_TIMING_MMC_HS400 10 unsigned char signal_voltage; /* signalling voltage (1.8V or 3.3V) */ @@ -136,6 +133,9 @@ struct mmc_host_ops { /* The tuning command opcode value is different for SD and eMMC cards */ int (*execute_tuning)(struct mmc_host *host, u32 opcode); + + /* Prepare HS400 target operating frequency depending host driver */ + int (*prepare_hs400_tuning)(struct mmc_host *host, struct mmc_ios *ios); int (*select_drive_strength)(unsigned int max_dtr, int host_drv, int card_drv); void (*hw_reset)(struct mmc_host *host); void (*card_event)(struct mmc_host *host); @@ -278,6 +278,11 @@ struct mmc_host { #define MMC_CAP2_PACKED_CMD (MMC_CAP2_PACKED_RD | \ MMC_CAP2_PACKED_WR) #define MMC_CAP2_NO_PRESCAN_POWERUP (1 << 14) /* Don't power up before scan */ +#define MMC_CAP2_HS400_1_8V (1 << 15) /* Can support HS400 1.8V */ +#define MMC_CAP2_HS400_1_2V (1 << 16) /* Can support HS400 1.2V */ +#define MMC_CAP2_HS400 (MMC_CAP2_HS400_1_8V | \ + MMC_CAP2_HS400_1_2V) +#define MMC_CAP2_SDIO_IRQ_NOTHREAD (1 << 17) mmc_pm_flag_t pm_caps; /* supported pm features */ @@ -318,6 +323,8 @@ struct mmc_host { int rescan_disable; /* disable card detection */ int rescan_entered; /* used with nonremovable devices */ + bool trigger_card_event; /* card_event necessary */ + struct mmc_card *card; /* device attached to this host */ wait_queue_head_t wq; @@ -391,12 +398,13 @@ static inline void mmc_signal_sdio_irq(struct mmc_host *host) wake_up_process(host->sdio_irq_thread); } +void sdio_run_irqs(struct mmc_host *host); + #ifdef CONFIG_REGULATOR int mmc_regulator_get_ocrmask(struct regulator *supply); int mmc_regulator_set_ocr(struct mmc_host *mmc, struct regulator *supply, unsigned short vdd_bit); -int mmc_regulator_get_supply(struct mmc_host *mmc); #else static inline int mmc_regulator_get_ocrmask(struct regulator *supply) { @@ -409,13 +417,10 @@ static inline int mmc_regulator_set_ocr(struct mmc_host *mmc, { return 0; } - -static inline int mmc_regulator_get_supply(struct mmc_host *mmc) -{ - return 0; -} #endif +int mmc_regulator_get_supply(struct mmc_host *mmc); + int mmc_pm_notify(struct notifier_block *notify_block, unsigned long, void *); static inline int mmc_card_is_removable(struct mmc_host *host) @@ -475,4 +480,32 @@ static inline unsigned int mmc_host_clk_rate(struct mmc_host *host) return host->ios.clock; } #endif + +static inline int mmc_card_hs(struct mmc_card *card) +{ + return card->host->ios.timing == MMC_TIMING_SD_HS || + card->host->ios.timing == MMC_TIMING_MMC_HS; +} + +static inline int mmc_card_uhs(struct mmc_card *card) +{ + return card->host->ios.timing >= MMC_TIMING_UHS_SDR12 && + card->host->ios.timing <= MMC_TIMING_UHS_DDR50; +} + +static inline bool mmc_card_hs200(struct mmc_card *card) +{ + return card->host->ios.timing == MMC_TIMING_MMC_HS200; +} + +static inline bool mmc_card_ddr52(struct mmc_card *card) +{ + return card->host->ios.timing == MMC_TIMING_MMC_DDR52; +} + +static inline bool mmc_card_hs400(struct mmc_card *card) +{ + return card->host->ios.timing == MMC_TIMING_MMC_HS400; +} + #endif /* LINUX_MMC_HOST_H */ diff --git a/include/linux/mmc/mmc.h b/include/linux/mmc/mmc.h index 50bcde3677ca..64ec963ed347 100644 --- a/include/linux/mmc/mmc.h +++ b/include/linux/mmc/mmc.h @@ -325,6 +325,7 @@ struct _mmc_csd { #define EXT_CSD_POWER_OFF_LONG_TIME 247 /* RO */ #define EXT_CSD_GENERIC_CMD6_TIME 248 /* RO */ #define EXT_CSD_CACHE_SIZE 249 /* RO, 4 bytes */ +#define EXT_CSD_PWR_CL_DDR_200_360 253 /* RO */ #define EXT_CSD_TAG_UNIT_SIZE 498 /* RO */ #define EXT_CSD_DATA_TAG_SUPPORT 499 /* RO */ #define EXT_CSD_MAX_PACKED_WRITES 500 /* RO */ @@ -354,18 +355,25 @@ struct _mmc_csd { #define EXT_CSD_CMD_SET_SECURE (1<<1) #define EXT_CSD_CMD_SET_CPSECURE (1<<2) -#define EXT_CSD_CARD_TYPE_26 (1<<0) /* Card can run at 26MHz */ -#define EXT_CSD_CARD_TYPE_52 (1<<1) /* Card can run at 52MHz */ -#define EXT_CSD_CARD_TYPE_MASK 0x3F /* Mask out reserved bits */ +#define EXT_CSD_CARD_TYPE_HS_26 (1<<0) /* Card can run at 26MHz */ +#define EXT_CSD_CARD_TYPE_HS_52 (1<<1) /* Card can run at 52MHz */ +#define EXT_CSD_CARD_TYPE_HS (EXT_CSD_CARD_TYPE_HS_26 | \ + EXT_CSD_CARD_TYPE_HS_52) #define EXT_CSD_CARD_TYPE_DDR_1_8V (1<<2) /* Card can run at 52MHz */ /* DDR mode @1.8V or 3V I/O */ #define EXT_CSD_CARD_TYPE_DDR_1_2V (1<<3) /* Card can run at 52MHz */ /* DDR mode @1.2V I/O */ #define EXT_CSD_CARD_TYPE_DDR_52 (EXT_CSD_CARD_TYPE_DDR_1_8V \ | EXT_CSD_CARD_TYPE_DDR_1_2V) -#define EXT_CSD_CARD_TYPE_SDR_1_8V (1<<4) /* Card can run at 200MHz */ -#define EXT_CSD_CARD_TYPE_SDR_1_2V (1<<5) /* Card can run at 200MHz */ +#define EXT_CSD_CARD_TYPE_HS200_1_8V (1<<4) /* Card can run at 200MHz */ +#define EXT_CSD_CARD_TYPE_HS200_1_2V (1<<5) /* Card can run at 200MHz */ /* SDR mode @1.2V I/O */ +#define EXT_CSD_CARD_TYPE_HS200 (EXT_CSD_CARD_TYPE_HS200_1_8V | \ + EXT_CSD_CARD_TYPE_HS200_1_2V) +#define EXT_CSD_CARD_TYPE_HS400_1_8V (1<<6) /* Card can run at 200MHz DDR, 1.8V */ +#define EXT_CSD_CARD_TYPE_HS400_1_2V (1<<7) /* Card can run at 200MHz DDR, 1.2V */ +#define EXT_CSD_CARD_TYPE_HS400 (EXT_CSD_CARD_TYPE_HS400_1_8V | \ + EXT_CSD_CARD_TYPE_HS400_1_2V) #define EXT_CSD_BUS_WIDTH_1 0 /* Card is in 1 bit mode */ #define EXT_CSD_BUS_WIDTH_4 1 /* Card is in 4 bit mode */ @@ -373,6 +381,11 @@ struct _mmc_csd { #define EXT_CSD_DDR_BUS_WIDTH_4 5 /* Card is in 4 bit DDR mode */ #define EXT_CSD_DDR_BUS_WIDTH_8 6 /* Card is in 8 bit DDR mode */ +#define EXT_CSD_TIMING_BC 0 /* Backwards compatility */ +#define EXT_CSD_TIMING_HS 1 /* High speed */ +#define EXT_CSD_TIMING_HS200 2 /* HS200 */ +#define EXT_CSD_TIMING_HS400 3 /* HS400 */ + #define EXT_CSD_SEC_ER_EN BIT(0) #define EXT_CSD_SEC_BD_BLK_EN BIT(2) #define EXT_CSD_SEC_GB_CL_EN BIT(4) diff --git a/include/linux/mmc/sdhci.h b/include/linux/mmc/sdhci.h index 7be12b883485..08abe9941884 100644 --- a/include/linux/mmc/sdhci.h +++ b/include/linux/mmc/sdhci.h @@ -57,12 +57,8 @@ struct sdhci_host { #define SDHCI_QUIRK_BROKEN_CARD_DETECTION (1<<15) /* Controller reports inverted write-protect state */ #define SDHCI_QUIRK_INVERTED_WRITE_PROTECT (1<<16) -/* Controller has nonstandard clock management */ -#define SDHCI_QUIRK_NONSTANDARD_CLOCK (1<<17) /* Controller does not like fast PIO transfers */ #define SDHCI_QUIRK_PIO_NEEDS_DELAY (1<<18) -/* Controller losing signal/interrupt enable states after reset */ -#define SDHCI_QUIRK_RESTORE_IRQS_AFTER_RESET (1<<19) /* Controller has to be forced to use block size of 2048 bytes */ #define SDHCI_QUIRK_FORCE_BLK_SZ_2048 (1<<20) /* Controller cannot do multi-block transfers */ @@ -147,6 +143,7 @@ struct sdhci_host { bool runtime_suspended; /* Host is runtime suspended */ bool bus_on; /* Bus power prevents runtime suspend */ + bool preset_enabled; /* Preset is enabled */ struct mmc_request *mrq; /* Current request */ struct mmc_command *cmd; /* Current command */ @@ -164,8 +161,7 @@ struct sdhci_host { dma_addr_t adma_addr; /* Mapped ADMA descr. table */ dma_addr_t align_addr; /* Mapped bounce buffer */ - struct tasklet_struct card_tasklet; /* Tasklet structures */ - struct tasklet_struct finish_tasklet; + struct tasklet_struct finish_tasklet; /* Tasklet structures */ struct timer_list timer; /* Timer for timeouts */ @@ -177,6 +173,13 @@ struct sdhci_host { unsigned int ocr_avail_mmc; u32 ocr_mask; /* available voltages */ + unsigned timing; /* Current timing */ + + u32 thread_isr; + + /* cached registers */ + u32 ier; + wait_queue_head_t buf_ready_int; /* Waitqueue for Buffer Read Ready interrupt */ unsigned int tuning_done; /* Condition flag set when CMD19 succeeds */ diff --git a/include/linux/nfs.h b/include/linux/nfs.h index 3e794c12e90a..610af5155ef2 100644 --- a/include/linux/nfs.h +++ b/include/linux/nfs.h @@ -46,6 +46,9 @@ static inline void nfs_copy_fh(struct nfs_fh *target, const struct nfs_fh *sourc enum nfs3_stable_how { NFS_UNSTABLE = 0, NFS_DATA_SYNC = 1, - NFS_FILE_SYNC = 2 + NFS_FILE_SYNC = 2, + + /* used by direct.c to mark verf as invalid */ + NFS_INVALID_STABLE_HOW = -1 }; #endif /* _LINUX_NFS_H */ diff --git a/include/linux/nfs4.h b/include/linux/nfs4.h index 12c2cb947df5..a1e3064a8d99 100644 --- a/include/linux/nfs4.h +++ b/include/linux/nfs4.h @@ -399,8 +399,6 @@ enum lock_type4 { #define FATTR4_WORD2_LAYOUT_BLKSIZE (1UL << 1) #define FATTR4_WORD2_MDSTHRESHOLD (1UL << 4) #define FATTR4_WORD2_SECURITY_LABEL (1UL << 16) -#define FATTR4_WORD2_CHANGE_SECURITY_LABEL \ - (1UL << 17) /* MDS threshold bitmap bits */ #define THRESHOLD_RD (1UL << 0) diff --git a/include/linux/nfs_fs.h b/include/linux/nfs_fs.h index fa6918b0f829..919576b8e2cf 100644 --- a/include/linux/nfs_fs.h +++ b/include/linux/nfs_fs.h @@ -520,7 +520,6 @@ extern int nfs_writepage(struct page *page, struct writeback_control *wbc); extern int nfs_writepages(struct address_space *, struct writeback_control *); extern int nfs_flush_incompatible(struct file *file, struct page *page); extern int nfs_updatepage(struct file *, struct page *, unsigned int, unsigned int); -extern void nfs_writeback_done(struct rpc_task *, struct nfs_write_data *); /* * Try to write back everything synchronously (but check the @@ -553,7 +552,6 @@ nfs_have_writebacks(struct inode *inode) extern int nfs_readpage(struct file *, struct page *); extern int nfs_readpages(struct file *, struct address_space *, struct list_head *, unsigned); -extern int nfs_readpage_result(struct rpc_task *, struct nfs_read_data *); extern int nfs_readpage_async(struct nfs_open_context *, struct inode *, struct page *); diff --git a/include/linux/nfs_page.h b/include/linux/nfs_page.h index 92ce5783b707..7d9096d95d4a 100644 --- a/include/linux/nfs_page.h +++ b/include/linux/nfs_page.h @@ -22,12 +22,17 @@ * Valid flags for a dirty buffer */ enum { - PG_BUSY = 0, - PG_MAPPED, - PG_CLEAN, - PG_NEED_COMMIT, - PG_NEED_RESCHED, - PG_COMMIT_TO_DS, + PG_BUSY = 0, /* nfs_{un}lock_request */ + PG_MAPPED, /* page private set for buffered io */ + PG_CLEAN, /* write succeeded */ + PG_COMMIT_TO_DS, /* used by pnfs layouts */ + PG_INODE_REF, /* extra ref held by inode (head req only) */ + PG_HEADLOCK, /* page group lock of wb_head */ + PG_TEARDOWN, /* page group sync for destroy */ + PG_UNLOCKPAGE, /* page group sync bit in read path */ + PG_UPTODATE, /* page group sync bit in read path */ + PG_WB_END, /* page group sync bit in write path */ + PG_REMOVE, /* page group sync bit in write path */ }; struct nfs_inode; @@ -43,15 +48,29 @@ struct nfs_page { struct kref wb_kref; /* reference count */ unsigned long wb_flags; struct nfs_write_verifier wb_verf; /* Commit cookie */ + struct nfs_page *wb_this_page; /* list of reqs for this page */ + struct nfs_page *wb_head; /* head pointer for req list */ }; struct nfs_pageio_descriptor; struct nfs_pageio_ops { void (*pg_init)(struct nfs_pageio_descriptor *, struct nfs_page *); - bool (*pg_test)(struct nfs_pageio_descriptor *, struct nfs_page *, struct nfs_page *); + size_t (*pg_test)(struct nfs_pageio_descriptor *, struct nfs_page *, + struct nfs_page *); int (*pg_doio)(struct nfs_pageio_descriptor *); }; +struct nfs_rw_ops { + const fmode_t rw_mode; + struct nfs_rw_header *(*rw_alloc_header)(void); + void (*rw_free_header)(struct nfs_rw_header *); + void (*rw_release)(struct nfs_pgio_data *); + int (*rw_done)(struct rpc_task *, struct nfs_pgio_data *, struct inode *); + void (*rw_result)(struct rpc_task *, struct nfs_pgio_data *); + void (*rw_initiate)(struct nfs_pgio_data *, struct rpc_message *, + struct rpc_task_setup *, int); +}; + struct nfs_pageio_descriptor { struct list_head pg_list; unsigned long pg_bytes_written; @@ -63,6 +82,7 @@ struct nfs_pageio_descriptor { struct inode *pg_inode; const struct nfs_pageio_ops *pg_ops; + const struct nfs_rw_ops *pg_rw_ops; int pg_ioflags; int pg_error; const struct rpc_call_ops *pg_rpc_callops; @@ -75,29 +95,33 @@ struct nfs_pageio_descriptor { #define NFS_WBACK_BUSY(req) (test_bit(PG_BUSY,&(req)->wb_flags)) extern struct nfs_page *nfs_create_request(struct nfs_open_context *ctx, - struct inode *inode, struct page *page, + struct nfs_page *last, unsigned int offset, unsigned int count); -extern void nfs_release_request(struct nfs_page *req); +extern void nfs_release_request(struct nfs_page *); extern void nfs_pageio_init(struct nfs_pageio_descriptor *desc, struct inode *inode, const struct nfs_pageio_ops *pg_ops, const struct nfs_pgio_completion_ops *compl_ops, + const struct nfs_rw_ops *rw_ops, size_t bsize, int how); extern int nfs_pageio_add_request(struct nfs_pageio_descriptor *, struct nfs_page *); extern void nfs_pageio_complete(struct nfs_pageio_descriptor *desc); extern void nfs_pageio_cond_complete(struct nfs_pageio_descriptor *, pgoff_t); -extern bool nfs_generic_pg_test(struct nfs_pageio_descriptor *desc, +extern size_t nfs_generic_pg_test(struct nfs_pageio_descriptor *desc, struct nfs_page *prev, struct nfs_page *req); extern int nfs_wait_on_request(struct nfs_page *); extern void nfs_unlock_request(struct nfs_page *req); -extern void nfs_unlock_and_release_request(struct nfs_page *req); +extern void nfs_unlock_and_release_request(struct nfs_page *); +extern void nfs_page_group_lock(struct nfs_page *); +extern void nfs_page_group_unlock(struct nfs_page *); +extern bool nfs_page_group_sync_on_bit(struct nfs_page *, unsigned int); /* * Lock the page of an asynchronous request diff --git a/include/linux/nfs_xdr.h b/include/linux/nfs_xdr.h index 6fb5b2335b59..9a1396e70310 100644 --- a/include/linux/nfs_xdr.h +++ b/include/linux/nfs_xdr.h @@ -489,31 +489,21 @@ struct nfs4_delegreturnres { }; /* - * Arguments to the read call. + * Arguments to the write call. */ -struct nfs_readargs { - struct nfs4_sequence_args seq_args; - struct nfs_fh * fh; - struct nfs_open_context *context; - struct nfs_lock_context *lock_context; - nfs4_stateid stateid; - __u64 offset; - __u32 count; - unsigned int pgbase; - struct page ** pages; +struct nfs_write_verifier { + char data[8]; }; -struct nfs_readres { - struct nfs4_sequence_res seq_res; - struct nfs_fattr * fattr; - __u32 count; - int eof; +struct nfs_writeverf { + struct nfs_write_verifier verifier; + enum nfs3_stable_how committed; }; /* - * Arguments to the write call. + * Arguments shared by the read and write call. */ -struct nfs_writeargs { +struct nfs_pgio_args { struct nfs4_sequence_args seq_args; struct nfs_fh * fh; struct nfs_open_context *context; @@ -521,27 +511,20 @@ struct nfs_writeargs { nfs4_stateid stateid; __u64 offset; __u32 count; - enum nfs3_stable_how stable; unsigned int pgbase; struct page ** pages; - const u32 * bitmask; -}; - -struct nfs_write_verifier { - char data[8]; + const u32 * bitmask; /* used by write */ + enum nfs3_stable_how stable; /* used by write */ }; -struct nfs_writeverf { - struct nfs_write_verifier verifier; - enum nfs3_stable_how committed; -}; - -struct nfs_writeres { +struct nfs_pgio_res { struct nfs4_sequence_res seq_res; struct nfs_fattr * fattr; - struct nfs_writeverf * verf; __u32 count; - const struct nfs_server *server; + int eof; /* used by read */ + struct nfs_writeverf * verf; /* used by write */ + const struct nfs_server *server; /* used by write */ + }; /* @@ -1129,6 +1112,7 @@ struct pnfs_commit_bucket { struct list_head committing; struct pnfs_layout_segment *wlseg; struct pnfs_layout_segment *clseg; + struct nfs_writeverf direct_verf; }; struct pnfs_ds_commit_info { @@ -1264,20 +1248,6 @@ struct nfs_page_array { struct page *page_array[NFS_PAGEVEC_SIZE]; }; -struct nfs_read_data { - struct nfs_pgio_header *header; - struct list_head list; - struct rpc_task task; - struct nfs_fattr fattr; /* fattr storage */ - struct nfs_readargs args; - struct nfs_readres res; - unsigned long timestamp; /* For lease renewal */ - int (*read_done_cb) (struct rpc_task *task, struct nfs_read_data *data); - __u64 mds_offset; - struct nfs_page_array pages; - struct nfs_client *ds_clp; /* pNFS data server */ -}; - /* used as flag bits in nfs_pgio_header */ enum { NFS_IOHDR_ERROR = 0, @@ -1287,19 +1257,22 @@ enum { NFS_IOHDR_NEED_RESCHED, }; +struct nfs_pgio_data; + struct nfs_pgio_header { struct inode *inode; struct rpc_cred *cred; struct list_head pages; - struct list_head rpc_list; + struct nfs_pgio_data *data; atomic_t refcnt; struct nfs_page *req; - struct nfs_writeverf *verf; + struct nfs_writeverf verf; /* Used for writes */ struct pnfs_layout_segment *lseg; loff_t io_start; const struct rpc_call_ops *mds_ops; void (*release) (struct nfs_pgio_header *hdr); const struct nfs_pgio_completion_ops *completion_ops; + const struct nfs_rw_ops *rw_ops; struct nfs_direct_req *dreq; void *layout_private; spinlock_t lock; @@ -1310,30 +1283,24 @@ struct nfs_pgio_header { unsigned long flags; }; -struct nfs_read_header { - struct nfs_pgio_header header; - struct nfs_read_data rpc_data; -}; - -struct nfs_write_data { +struct nfs_pgio_data { struct nfs_pgio_header *header; - struct list_head list; struct rpc_task task; struct nfs_fattr fattr; - struct nfs_writeverf verf; - struct nfs_writeargs args; /* argument struct */ - struct nfs_writeres res; /* result struct */ + struct nfs_writeverf verf; /* Used for writes */ + struct nfs_pgio_args args; /* argument struct */ + struct nfs_pgio_res res; /* result struct */ unsigned long timestamp; /* For lease renewal */ - int (*write_done_cb) (struct rpc_task *task, struct nfs_write_data *data); + int (*pgio_done_cb) (struct rpc_task *task, struct nfs_pgio_data *data); __u64 mds_offset; /* Filelayout dense stripe */ struct nfs_page_array pages; struct nfs_client *ds_clp; /* pNFS data server */ + int ds_idx; /* ds index if ds_clp is set */ }; -struct nfs_write_header { +struct nfs_rw_header { struct nfs_pgio_header header; - struct nfs_write_data rpc_data; - struct nfs_writeverf verf; + struct nfs_pgio_data rpc_data; }; struct nfs_mds_commit_info { @@ -1465,16 +1432,11 @@ struct nfs_rpc_ops { struct nfs_pathconf *); int (*set_capabilities)(struct nfs_server *, struct nfs_fh *); int (*decode_dirent)(struct xdr_stream *, struct nfs_entry *, int); - void (*read_setup) (struct nfs_read_data *, struct rpc_message *); - void (*read_pageio_init)(struct nfs_pageio_descriptor *, struct inode *, - const struct nfs_pgio_completion_ops *); - int (*read_rpc_prepare)(struct rpc_task *, struct nfs_read_data *); - int (*read_done) (struct rpc_task *, struct nfs_read_data *); - void (*write_setup) (struct nfs_write_data *, struct rpc_message *); - void (*write_pageio_init)(struct nfs_pageio_descriptor *, struct inode *, int, - const struct nfs_pgio_completion_ops *); - int (*write_rpc_prepare)(struct rpc_task *, struct nfs_write_data *); - int (*write_done) (struct rpc_task *, struct nfs_write_data *); + int (*pgio_rpc_prepare)(struct rpc_task *, struct nfs_pgio_data *); + void (*read_setup) (struct nfs_pgio_data *, struct rpc_message *); + int (*read_done) (struct rpc_task *, struct nfs_pgio_data *); + void (*write_setup) (struct nfs_pgio_data *, struct rpc_message *); + int (*write_done) (struct rpc_task *, struct nfs_pgio_data *); void (*commit_setup) (struct nfs_commit_data *, struct rpc_message *); void (*commit_rpc_prepare)(struct rpc_task *, struct nfs_commit_data *); int (*commit_done) (struct rpc_task *, struct nfs_commit_data *); diff --git a/include/linux/nfsd/debug.h b/include/linux/nfsd/debug.h deleted file mode 100644 index 19ef8375b577..000000000000 --- a/include/linux/nfsd/debug.h +++ /dev/null @@ -1,19 +0,0 @@ -/* - * linux/include/linux/nfsd/debug.h - * - * Debugging-related stuff for nfsd - * - * Copyright (C) 1995 Olaf Kirch <okir@monad.swb.de> - */ -#ifndef LINUX_NFSD_DEBUG_H -#define LINUX_NFSD_DEBUG_H - -#include <uapi/linux/nfsd/debug.h> - -# undef ifdebug -# ifdef NFSD_DEBUG -# define ifdebug(flag) if (nfsd_debug & NFSDDBG_##flag) -# else -# define ifdebug(flag) if (0) -# endif -#endif /* LINUX_NFSD_DEBUG_H */ diff --git a/include/linux/nfsd/nfsfh.h b/include/linux/nfsd/nfsfh.h deleted file mode 100644 index a93593f1fa4e..000000000000 --- a/include/linux/nfsd/nfsfh.h +++ /dev/null @@ -1,63 +0,0 @@ -/* - * include/linux/nfsd/nfsfh.h - * - * This file describes the layout of the file handles as passed - * over the wire. - * - * Earlier versions of knfsd used to sign file handles using keyed MD5 - * or SHA. I've removed this code, because it doesn't give you more - * security than blocking external access to port 2049 on your firewall. - * - * Copyright (C) 1995, 1996, 1997 Olaf Kirch <okir@monad.swb.de> - */ -#ifndef _LINUX_NFSD_FH_H -#define _LINUX_NFSD_FH_H - -# include <linux/sunrpc/svc.h> -#include <uapi/linux/nfsd/nfsfh.h> - -static inline __u32 ino_t_to_u32(ino_t ino) -{ - return (__u32) ino; -} - -static inline ino_t u32_to_ino_t(__u32 uino) -{ - return (ino_t) uino; -} - -/* - * This is the internal representation of an NFS handle used in knfsd. - * pre_mtime/post_version will be used to support wcc_attr's in NFSv3. - */ -typedef struct svc_fh { - struct knfsd_fh fh_handle; /* FH data */ - struct dentry * fh_dentry; /* validated dentry */ - struct svc_export * fh_export; /* export pointer */ - int fh_maxsize; /* max size for fh_handle */ - - unsigned char fh_locked; /* inode locked by us */ - unsigned char fh_want_write; /* remount protection taken */ - -#ifdef CONFIG_NFSD_V3 - unsigned char fh_post_saved; /* post-op attrs saved */ - unsigned char fh_pre_saved; /* pre-op attrs saved */ - - /* Pre-op attributes saved during fh_lock */ - __u64 fh_pre_size; /* size before operation */ - struct timespec fh_pre_mtime; /* mtime before oper */ - struct timespec fh_pre_ctime; /* ctime before oper */ - /* - * pre-op nfsv4 change attr: note must check IS_I_VERSION(inode) - * to find out if it is valid. - */ - u64 fh_pre_change; - - /* Post-op attributes saved in fh_unlock */ - struct kstat fh_post_attr; /* full attrs after operation */ - u64 fh_post_change; /* nfsv4 change; see above */ -#endif /* CONFIG_NFSD_V3 */ - -} svc_fh; - -#endif /* _LINUX_NFSD_FH_H */ diff --git a/include/linux/omap-dma.h b/include/linux/omap-dma.h index c29a6dee6bec..88e6ea4a5d36 100644 --- a/include/linux/omap-dma.h +++ b/include/linux/omap-dma.h @@ -1,23 +1,6 @@ -/* - * OMAP DMA Engine support - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 2 as - * published by the Free Software Foundation. - */ #ifndef __LINUX_OMAP_DMA_H #define __LINUX_OMAP_DMA_H - -struct dma_chan; - -#if defined(CONFIG_DMA_OMAP) || (defined(CONFIG_DMA_OMAP_MODULE) && defined(MODULE)) -bool omap_dma_filter_fn(struct dma_chan *, void *); -#else -static inline bool omap_dma_filter_fn(struct dma_chan *c, void *d) -{ - return false; -} -#endif +#include <linux/omap-dmaengine.h> /* * Legacy OMAP DMA handling defines and functions diff --git a/include/linux/omap-dmaengine.h b/include/linux/omap-dmaengine.h new file mode 100644 index 000000000000..8e6906c72e90 --- /dev/null +++ b/include/linux/omap-dmaengine.h @@ -0,0 +1,21 @@ +/* + * OMAP DMA Engine support + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ +#ifndef __LINUX_OMAP_DMAENGINE_H +#define __LINUX_OMAP_DMAENGINE_H + +struct dma_chan; + +#if defined(CONFIG_DMA_OMAP) || (defined(CONFIG_DMA_OMAP_MODULE) && defined(MODULE)) +bool omap_dma_filter_fn(struct dma_chan *, void *); +#else +static inline bool omap_dma_filter_fn(struct dma_chan *c, void *d) +{ + return false; +} +#endif +#endif /* __LINUX_OMAP_DMAENGINE_H */ diff --git a/include/linux/percpu-refcount.h b/include/linux/percpu-refcount.h index 0afb48fd449d..5d8920e23073 100644 --- a/include/linux/percpu-refcount.h +++ b/include/linux/percpu-refcount.h @@ -121,6 +121,36 @@ static inline void percpu_ref_get(struct percpu_ref *ref) * percpu_ref_tryget - try to increment a percpu refcount * @ref: percpu_ref to try-get * + * Increment a percpu refcount unless its count already reached zero. + * Returns %true on success; %false on failure. + * + * The caller is responsible for ensuring that @ref stays accessible. + */ +static inline bool percpu_ref_tryget(struct percpu_ref *ref) +{ + unsigned __percpu *pcpu_count; + int ret = false; + + rcu_read_lock_sched(); + + pcpu_count = ACCESS_ONCE(ref->pcpu_count); + + if (likely(REF_STATUS(pcpu_count) == PCPU_REF_PTR)) { + this_cpu_inc(*pcpu_count); + ret = true; + } else { + ret = atomic_inc_not_zero(&ref->count); + } + + rcu_read_unlock_sched(); + + return ret; +} + +/** + * percpu_ref_tryget_live - try to increment a live percpu refcount + * @ref: percpu_ref to try-get + * * Increment a percpu refcount unless it has already been killed. Returns * %true on success; %false on failure. * @@ -128,8 +158,10 @@ static inline void percpu_ref_get(struct percpu_ref *ref) * will fail. For such guarantee, percpu_ref_kill_and_confirm() should be * used. After the confirm_kill callback is invoked, it's guaranteed that * no new reference will be given out by percpu_ref_tryget(). + * + * The caller is responsible for ensuring that @ref stays accessible. */ -static inline bool percpu_ref_tryget(struct percpu_ref *ref) +static inline bool percpu_ref_tryget_live(struct percpu_ref *ref) { unsigned __percpu *pcpu_count; int ret = false; diff --git a/include/linux/percpu.h b/include/linux/percpu.h index 495c6543a8f2..8419053d0f2e 100644 --- a/include/linux/percpu.h +++ b/include/linux/percpu.h @@ -29,7 +29,7 @@ */ #define get_cpu_var(var) (*({ \ preempt_disable(); \ - &__get_cpu_var(var); })) + this_cpu_ptr(&var); })) /* * The weird & is necessary because sparse considers (void)(var) to be diff --git a/include/linux/platform_data/omap4-keypad.h b/include/linux/platform_data/omap4-keypad.h deleted file mode 100644 index 4eef5fb05a17..000000000000 --- a/include/linux/platform_data/omap4-keypad.h +++ /dev/null @@ -1,13 +0,0 @@ -#ifndef __LINUX_INPUT_OMAP4_KEYPAD_H -#define __LINUX_INPUT_OMAP4_KEYPAD_H - -#include <linux/input/matrix_keypad.h> - -struct omap4_keypad_platform_data { - const struct matrix_keymap_data *keymap_data; - - u8 rows; - u8 cols; -}; - -#endif /* __LINUX_INPUT_OMAP4_KEYPAD_H */ diff --git a/include/linux/security.h b/include/linux/security.h index 6478ce3252c7..9c6b9722ff48 100644 --- a/include/linux/security.h +++ b/include/linux/security.h @@ -1708,7 +1708,7 @@ struct security_operations { void (*key_free) (struct key *key); int (*key_permission) (key_ref_t key_ref, const struct cred *cred, - key_perm_t perm); + unsigned perm); int (*key_getsecurity)(struct key *key, char **_buffer); #endif /* CONFIG_KEYS */ @@ -3034,7 +3034,7 @@ static inline int security_path_chroot(struct path *path) int security_key_alloc(struct key *key, const struct cred *cred, unsigned long flags); void security_key_free(struct key *key); int security_key_permission(key_ref_t key_ref, - const struct cred *cred, key_perm_t perm); + const struct cred *cred, unsigned perm); int security_key_getsecurity(struct key *key, char **_buffer); #else @@ -3052,7 +3052,7 @@ static inline void security_key_free(struct key *key) static inline int security_key_permission(key_ref_t key_ref, const struct cred *cred, - key_perm_t perm) + unsigned perm) { return 0; } diff --git a/include/linux/shdma-base.h b/include/linux/shdma-base.h index f92c0a43c54c..abdf1f229dc3 100644 --- a/include/linux/shdma-base.h +++ b/include/linux/shdma-base.h @@ -54,6 +54,7 @@ struct shdma_desc { dma_cookie_t cookie; int chunks; int mark; + bool cyclic; /* used as cyclic transfer */ }; struct shdma_chan { diff --git a/include/linux/sunrpc/svc.h b/include/linux/sunrpc/svc.h index 04e763221246..1bc7cd05b22e 100644 --- a/include/linux/sunrpc/svc.h +++ b/include/linux/sunrpc/svc.h @@ -244,6 +244,7 @@ struct svc_rqst { struct page * rq_pages[RPCSVC_MAXPAGES]; struct page * *rq_respages; /* points into rq_pages */ struct page * *rq_next_page; /* next reply page to use */ + struct page * *rq_page_end; /* one past the last page */ struct kvec rq_vec[RPCSVC_MAXPAGES]; /* generally useful.. */ @@ -254,11 +255,15 @@ struct svc_rqst { u32 rq_prot; /* IP protocol */ unsigned short rq_secure : 1; /* secure port */ + unsigned short rq_local : 1; /* local request */ void * rq_argp; /* decoded arguments */ void * rq_resp; /* xdr'd results */ void * rq_auth_data; /* flavor-specific data */ - + int rq_auth_slack; /* extra space xdr code + * should leave in head + * for krb5i, krb5p. + */ int rq_reserved; /* space on socket outq * reserved for this request */ @@ -454,11 +459,7 @@ char * svc_print_addr(struct svc_rqst *, char *, size_t); */ static inline void svc_reserve_auth(struct svc_rqst *rqstp, int space) { - int added_space = 0; - - if (rqstp->rq_authop->flavour) - added_space = RPC_MAX_AUTH_SIZE; - svc_reserve(rqstp, space + added_space); + svc_reserve(rqstp, space + rqstp->rq_auth_slack); } #endif /* SUNRPC_SVC_H */ diff --git a/include/linux/sunrpc/svc_rdma.h b/include/linux/sunrpc/svc_rdma.h index 0b8e3e6bdacf..5cf99a016368 100644 --- a/include/linux/sunrpc/svc_rdma.h +++ b/include/linux/sunrpc/svc_rdma.h @@ -115,14 +115,13 @@ struct svc_rdma_fastreg_mr { struct list_head frmr_list; }; struct svc_rdma_req_map { - struct svc_rdma_fastreg_mr *frmr; unsigned long count; union { struct kvec sge[RPCSVC_MAXPAGES]; struct svc_rdma_chunk_sge ch[RPCSVC_MAXPAGES]; + unsigned long lkey[RPCSVC_MAXPAGES]; }; }; -#define RDMACTXT_F_FAST_UNREG 1 #define RDMACTXT_F_LAST_CTXT 2 #define SVCRDMA_DEVCAP_FAST_REG 1 /* fast mr registration */ diff --git a/include/linux/sunrpc/svc_xprt.h b/include/linux/sunrpc/svc_xprt.h index b05963f09ebf..7235040a19b2 100644 --- a/include/linux/sunrpc/svc_xprt.h +++ b/include/linux/sunrpc/svc_xprt.h @@ -24,6 +24,7 @@ struct svc_xprt_ops { void (*xpo_release_rqst)(struct svc_rqst *); void (*xpo_detach)(struct svc_xprt *); void (*xpo_free)(struct svc_xprt *); + int (*xpo_secure_port)(struct svc_rqst *); }; struct svc_xprt_class { @@ -63,6 +64,7 @@ struct svc_xprt { #define XPT_DETACHED 10 /* detached from tempsocks list */ #define XPT_LISTENER 11 /* listening endpoint */ #define XPT_CACHE_AUTH 12 /* cache auth info */ +#define XPT_LOCAL 13 /* connection from loopback interface */ struct svc_serv *xpt_server; /* service for transport */ atomic_t xpt_reserved; /* space on outq that is rsvd */ diff --git a/include/linux/sunrpc/xdr.h b/include/linux/sunrpc/xdr.h index 15f9204ee70b..70c6b92e15a7 100644 --- a/include/linux/sunrpc/xdr.h +++ b/include/linux/sunrpc/xdr.h @@ -215,6 +215,9 @@ typedef int (*kxdrdproc_t)(void *rqstp, struct xdr_stream *xdr, void *obj); extern void xdr_init_encode(struct xdr_stream *xdr, struct xdr_buf *buf, __be32 *p); extern __be32 *xdr_reserve_space(struct xdr_stream *xdr, size_t nbytes); +extern void xdr_commit_encode(struct xdr_stream *xdr); +extern void xdr_truncate_encode(struct xdr_stream *xdr, size_t len); +extern int xdr_restrict_buflen(struct xdr_stream *xdr, int newbuflen); extern void xdr_write_pages(struct xdr_stream *xdr, struct page **pages, unsigned int base, unsigned int len); extern unsigned int xdr_stream_pos(const struct xdr_stream *xdr); diff --git a/include/linux/sunrpc/xprt.h b/include/linux/sunrpc/xprt.h index 3876f0f1dfd3..fcbfe8783243 100644 --- a/include/linux/sunrpc/xprt.h +++ b/include/linux/sunrpc/xprt.h @@ -24,6 +24,12 @@ #define RPC_MAX_SLOT_TABLE_LIMIT (65536U) #define RPC_MAX_SLOT_TABLE RPC_MAX_SLOT_TABLE_LIMIT +#define RPC_CWNDSHIFT (8U) +#define RPC_CWNDSCALE (1U << RPC_CWNDSHIFT) +#define RPC_INITCWND RPC_CWNDSCALE +#define RPC_MAXCWND(xprt) ((xprt)->max_reqs << RPC_CWNDSHIFT) +#define RPCXPRT_CONGESTED(xprt) ((xprt)->cong >= (xprt)->cwnd) + /* * This describes a timeout strategy */ diff --git a/include/linux/trace_seq.h b/include/linux/trace_seq.h index a32d86ec8bf2..136116924d8d 100644 --- a/include/linux/trace_seq.h +++ b/include/linux/trace_seq.h @@ -46,6 +46,9 @@ extern int trace_seq_putmem_hex(struct trace_seq *s, const void *mem, extern void *trace_seq_reserve(struct trace_seq *s, size_t len); extern int trace_seq_path(struct trace_seq *s, const struct path *path); +extern int trace_seq_bitmask(struct trace_seq *s, const unsigned long *maskp, + int nmaskbits); + #else /* CONFIG_TRACING */ static inline int trace_seq_printf(struct trace_seq *s, const char *fmt, ...) { @@ -57,6 +60,13 @@ trace_seq_bprintf(struct trace_seq *s, const char *fmt, const u32 *binary) return 0; } +static inline int +trace_seq_bitmask(struct trace_seq *s, const unsigned long *maskp, + int nmaskbits) +{ + return 0; +} + static inline int trace_print_seq(struct seq_file *m, struct trace_seq *s) { return 0; diff --git a/include/linux/tracepoint.h b/include/linux/tracepoint.h index 9d30ee469c2a..2e2a5f7717e5 100644 --- a/include/linux/tracepoint.h +++ b/include/linux/tracepoint.h @@ -185,6 +185,11 @@ extern void syscall_unregfunc(void); static inline void \ check_trace_callback_type_##name(void (*cb)(data_proto)) \ { \ + } \ + static inline bool \ + trace_##name##_enabled(void) \ + { \ + return static_key_false(&__tracepoint_##name.key); \ } /* @@ -230,6 +235,11 @@ extern void syscall_unregfunc(void); } \ static inline void check_trace_callback_type_##name(void (*cb)(data_proto)) \ { \ + } \ + static inline bool \ + trace_##name##_enabled(void) \ + { \ + return false; \ } #define DEFINE_TRACE_FN(name, reg, unreg) diff --git a/include/linux/workqueue.h b/include/linux/workqueue.h index 1b22c42e9c2d..a0cc2e95ed1b 100644 --- a/include/linux/workqueue.h +++ b/include/linux/workqueue.h @@ -56,9 +56,8 @@ enum { WORK_NR_COLORS = (1 << WORK_STRUCT_COLOR_BITS) - 1, WORK_NO_COLOR = WORK_NR_COLORS, - /* special cpu IDs */ + /* not bound to any CPU, prefer the local CPU */ WORK_CPU_UNBOUND = NR_CPUS, - WORK_CPU_END = NR_CPUS + 1, /* * Reserve 7 bits off of pwq pointer w/ debugobjects turned off. @@ -274,13 +273,6 @@ static inline unsigned int work_static(struct work_struct *work) { return 0; } #define delayed_work_pending(w) \ work_pending(&(w)->work) -/** - * work_clear_pending - for internal use only, mark a work item as not pending - * @work: The work item in question - */ -#define work_clear_pending(work) \ - clear_bit(WORK_STRUCT_PENDING_BIT, work_data_bits(work)) - /* * Workqueue flags and constants. For details, please refer to * Documentation/workqueue.txt. @@ -340,6 +332,9 @@ enum { * short queue flush time. Don't queue works which can run for too * long. * + * system_highpri_wq is similar to system_wq but for work items which + * require WQ_HIGHPRI. + * * system_long_wq is similar to system_wq but may host long running * works. Queue flushing might take relatively long. * @@ -358,26 +353,13 @@ enum { * 'wq_power_efficient' is disabled. See WQ_POWER_EFFICIENT for more info. */ extern struct workqueue_struct *system_wq; +extern struct workqueue_struct *system_highpri_wq; extern struct workqueue_struct *system_long_wq; extern struct workqueue_struct *system_unbound_wq; extern struct workqueue_struct *system_freezable_wq; extern struct workqueue_struct *system_power_efficient_wq; extern struct workqueue_struct *system_freezable_power_efficient_wq; -static inline struct workqueue_struct * __deprecated __system_nrt_wq(void) -{ - return system_wq; -} - -static inline struct workqueue_struct * __deprecated __system_nrt_freezable_wq(void) -{ - return system_freezable_wq; -} - -/* equivlalent to system_wq and system_freezable_wq, deprecated */ -#define system_nrt_wq __system_nrt_wq() -#define system_nrt_freezable_wq __system_nrt_freezable_wq() - extern struct workqueue_struct * __alloc_workqueue_key(const char *fmt, unsigned int flags, int max_active, struct lock_class_key *key, const char *lock_name, ...) __printf(1, 6); @@ -587,18 +569,6 @@ static inline bool keventd_up(void) return system_wq != NULL; } -/* used to be different but now identical to flush_work(), deprecated */ -static inline bool __deprecated flush_work_sync(struct work_struct *work) -{ - return flush_work(work); -} - -/* used to be different but now identical to flush_delayed_work(), deprecated */ -static inline bool __deprecated flush_delayed_work_sync(struct delayed_work *dwork) -{ - return flush_delayed_work(dwork); -} - #ifndef CONFIG_SMP static inline long work_on_cpu(int cpu, long (*fn)(void *), void *arg) { diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h index acd825182977..7ccef342f724 100644 --- a/include/rdma/ib_verbs.h +++ b/include/rdma/ib_verbs.h @@ -80,8 +80,8 @@ enum rdma_transport_type { RDMA_TRANSPORT_USNIC_UDP }; -enum rdma_transport_type -rdma_node_get_transport(enum rdma_node_type node_type) __attribute_const__; +__attribute_const__ enum rdma_transport_type +rdma_node_get_transport(enum rdma_node_type node_type); enum rdma_link_layer { IB_LINK_LAYER_UNSPECIFIED, @@ -466,14 +466,14 @@ enum ib_rate { * converted to 2, since 5 Gbit/sec is 2 * 2.5 Gbit/sec. * @rate: rate to convert. */ -int ib_rate_to_mult(enum ib_rate rate) __attribute_const__; +__attribute_const__ int ib_rate_to_mult(enum ib_rate rate); /** * ib_rate_to_mbps - Convert the IB rate enum to Mbps. * For example, IB_RATE_2_5_GBPS will be converted to 2500. * @rate: rate to convert. */ -int ib_rate_to_mbps(enum ib_rate rate) __attribute_const__; +__attribute_const__ int ib_rate_to_mbps(enum ib_rate rate); enum ib_mr_create_flags { IB_MR_SIGNATURE_EN = 1, @@ -604,7 +604,7 @@ struct ib_mr_status { * enum. * @mult: multiple to convert. */ -enum ib_rate mult_to_ib_rate(int mult) __attribute_const__; +__attribute_const__ enum ib_rate mult_to_ib_rate(int mult); struct ib_ah_attr { struct ib_global_route grh; @@ -783,6 +783,7 @@ enum ib_qp_create_flags { IB_QP_CREATE_BLOCK_MULTICAST_LOOPBACK = 1 << 1, IB_QP_CREATE_NETIF_QP = 1 << 5, IB_QP_CREATE_SIGNATURE_EN = 1 << 6, + IB_QP_CREATE_USE_GFP_NOIO = 1 << 7, /* reserve bits 26-31 for low level drivers' internal use */ IB_QP_CREATE_RESERVED_START = 1 << 26, IB_QP_CREATE_RESERVED_END = 1 << 31, diff --git a/include/rdma/iw_portmap.h b/include/rdma/iw_portmap.h new file mode 100644 index 000000000000..928b2775e992 --- /dev/null +++ b/include/rdma/iw_portmap.h @@ -0,0 +1,199 @@ +/* + * Copyright (c) 2014 Intel Corporation. All rights reserved. + * Copyright (c) 2014 Chelsio, Inc. All rights reserved. + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the + * OpenIB.org BSD license below: + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ +#ifndef _IW_PORTMAP_H +#define _IW_PORTMAP_H + +#define IWPM_ULIBNAME_SIZE 32 +#define IWPM_DEVNAME_SIZE 32 +#define IWPM_IFNAME_SIZE 16 +#define IWPM_IPADDR_SIZE 16 + +enum { + IWPM_INVALID_NLMSG_ERR = 10, + IWPM_CREATE_MAPPING_ERR, + IWPM_DUPLICATE_MAPPING_ERR, + IWPM_UNKNOWN_MAPPING_ERR, + IWPM_CLIENT_DEV_INFO_ERR, + IWPM_USER_LIB_INFO_ERR, + IWPM_REMOTE_QUERY_REJECT +}; + +struct iwpm_dev_data { + char dev_name[IWPM_DEVNAME_SIZE]; + char if_name[IWPM_IFNAME_SIZE]; +}; + +struct iwpm_sa_data { + struct sockaddr_storage loc_addr; + struct sockaddr_storage mapped_loc_addr; + struct sockaddr_storage rem_addr; + struct sockaddr_storage mapped_rem_addr; +}; + +/** + * iwpm_init - Allocate resources for the iwarp port mapper + * + * Should be called when network interface goes up. + */ +int iwpm_init(u8); + +/** + * iwpm_exit - Deallocate resources for the iwarp port mapper + * + * Should be called when network interface goes down. + */ +int iwpm_exit(u8); + +/** + * iwpm_valid_pid - Check if the userspace iwarp port mapper pid is valid + * + * Returns true if the pid is greater than zero, otherwise returns false + */ +int iwpm_valid_pid(void); + +/** + * iwpm_register_pid - Send a netlink query to userspace + * to get the iwarp port mapper pid + * @pm_msg: Contains driver info to send to the userspace port mapper + * @nl_client: The index of the netlink client + */ +int iwpm_register_pid(struct iwpm_dev_data *pm_msg, u8 nl_client); + +/** + * iwpm_add_mapping - Send a netlink add mapping request to + * the userspace port mapper + * @pm_msg: Contains the local ip/tcp address info to send + * @nl_client: The index of the netlink client + * + * If the request is successful, the pm_msg stores + * the port mapper response (mapped address info) + */ +int iwpm_add_mapping(struct iwpm_sa_data *pm_msg, u8 nl_client); + +/** + * iwpm_add_and_query_mapping - Send a netlink add and query mapping request + * to the userspace port mapper + * @pm_msg: Contains the local and remote ip/tcp address info to send + * @nl_client: The index of the netlink client + * + * If the request is successful, the pm_msg stores the + * port mapper response (mapped local and remote address info) + */ +int iwpm_add_and_query_mapping(struct iwpm_sa_data *pm_msg, u8 nl_client); + +/** + * iwpm_remove_mapping - Send a netlink remove mapping request + * to the userspace port mapper + * + * @local_addr: Local ip/tcp address to remove + * @nl_client: The index of the netlink client + */ +int iwpm_remove_mapping(struct sockaddr_storage *local_addr, u8 nl_client); + +/** + * iwpm_register_pid_cb - Process the port mapper response to + * iwpm_register_pid query + * @skb: + * @cb: Contains the received message (payload and netlink header) + * + * If successful, the function receives the userspace port mapper pid + * which is used in future communication with the port mapper + */ +int iwpm_register_pid_cb(struct sk_buff *, struct netlink_callback *); + +/** + * iwpm_add_mapping_cb - Process the port mapper response to + * iwpm_add_mapping request + * @skb: + * @cb: Contains the received message (payload and netlink header) + */ +int iwpm_add_mapping_cb(struct sk_buff *, struct netlink_callback *); + +/** + * iwpm_add_and_query_mapping_cb - Process the port mapper response to + * iwpm_add_and_query_mapping request + * @skb: + * @cb: Contains the received message (payload and netlink header) + */ +int iwpm_add_and_query_mapping_cb(struct sk_buff *, struct netlink_callback *); + +/** + * iwpm_mapping_error_cb - Process port mapper notification for error + * + * @skb: + * @cb: Contains the received message (payload and netlink header) + */ +int iwpm_mapping_error_cb(struct sk_buff *, struct netlink_callback *); + +/** + * iwpm_mapping_info_cb - Process a notification that the userspace + * port mapper daemon is started + * @skb: + * @cb: Contains the received message (payload and netlink header) + * + * Using the received port mapper pid, send all the local mapping + * info records to the userspace port mapper + */ +int iwpm_mapping_info_cb(struct sk_buff *, struct netlink_callback *); + +/** + * iwpm_ack_mapping_info_cb - Process the port mapper ack for + * the provided local mapping info records + * @skb: + * @cb: Contains the received message (payload and netlink header) + */ +int iwpm_ack_mapping_info_cb(struct sk_buff *, struct netlink_callback *); + +/** + * iwpm_create_mapinfo - Store local and mapped IPv4/IPv6 address + * info in a hash table + * @local_addr: Local ip/tcp address + * @mapped_addr: Mapped local ip/tcp address + * @nl_client: The index of the netlink client + */ +int iwpm_create_mapinfo(struct sockaddr_storage *local_addr, + struct sockaddr_storage *mapped_addr, u8 nl_client); + +/** + * iwpm_remove_mapinfo - Remove local and mapped IPv4/IPv6 address + * info from the hash table + * @local_addr: Local ip/tcp address + * @mapped_addr: Mapped local ip/tcp address + * + * Returns err code if mapping info is not found in the hash table, + * otherwise returns 0 + */ +int iwpm_remove_mapinfo(struct sockaddr_storage *local_addr, + struct sockaddr_storage *mapped_addr); + +#endif /* _IW_PORTMAP_H */ diff --git a/include/rdma/rdma_netlink.h b/include/rdma/rdma_netlink.h index e38de79eeb48..0790882e0c9b 100644 --- a/include/rdma/rdma_netlink.h +++ b/include/rdma/rdma_netlink.h @@ -43,7 +43,7 @@ int ibnl_remove_client(int index); * Returns the allocated buffer on success and NULL on failure. */ void *ibnl_put_msg(struct sk_buff *skb, struct nlmsghdr **nlh, int seq, - int len, int client, int op); + int len, int client, int op, int flags); /** * Put a new attribute in a supplied skb. * @skb: The netlink skb. @@ -56,4 +56,25 @@ void *ibnl_put_msg(struct sk_buff *skb, struct nlmsghdr **nlh, int seq, int ibnl_put_attr(struct sk_buff *skb, struct nlmsghdr *nlh, int len, void *data, int type); +/** + * Send the supplied skb to a specific userspace PID. + * @skb: The netlink skb + * @nlh: Header of the netlink message to send + * @pid: Userspace netlink process ID + * Returns 0 on success or a negative error code. + */ +int ibnl_unicast(struct sk_buff *skb, struct nlmsghdr *nlh, + __u32 pid); + +/** + * Send the supplied skb to a netlink group. + * @skb: The netlink skb + * @nlh: Header of the netlink message to send + * @group: Netlink group ID + * @flags: allocation flags + * Returns 0 on success or a negative error code. + */ +int ibnl_multicast(struct sk_buff *skb, struct nlmsghdr *nlh, + unsigned int group, gfp_t flags); + #endif /* _RDMA_NETLINK_H */ diff --git a/include/scsi/osd_protocol.h b/include/scsi/osd_protocol.h index 25ac6283b9c7..a2594afe05c7 100644 --- a/include/scsi/osd_protocol.h +++ b/include/scsi/osd_protocol.h @@ -263,16 +263,16 @@ static inline struct osd_cdb_head *osd_cdb_head(struct osd_cdb *ocdb) * Ex name = FORMAT_OSD we have OSD_ACT_FORMAT_OSD && OSDv1_ACT_FORMAT_OSD */ #define OSD_ACT___(Name, Num) \ - OSD_ACT_##Name = __constant_cpu_to_be16(0x8880 + Num), \ - OSDv1_ACT_##Name = __constant_cpu_to_be16(0x8800 + Num), + OSD_ACT_##Name = cpu_to_be16(0x8880 + Num), \ + OSDv1_ACT_##Name = cpu_to_be16(0x8800 + Num), /* V2 only actions */ #define OSD_ACT_V2(Name, Num) \ - OSD_ACT_##Name = __constant_cpu_to_be16(0x8880 + Num), + OSD_ACT_##Name = cpu_to_be16(0x8880 + Num), #define OSD_ACT_V1_V2(Name, Num1, Num2) \ - OSD_ACT_##Name = __constant_cpu_to_be16(Num2), \ - OSDv1_ACT_##Name = __constant_cpu_to_be16(Num1), + OSD_ACT_##Name = cpu_to_be16(Num2), \ + OSDv1_ACT_##Name = cpu_to_be16(Num1), enum osd_service_actions { OSD_ACT_V2(OBJECT_STRUCTURE_CHECK, 0x00) diff --git a/include/scsi/scsi_cmnd.h b/include/scsi/scsi_cmnd.h index dd7c998221b3..e016e2ac38df 100644 --- a/include/scsi/scsi_cmnd.h +++ b/include/scsi/scsi_cmnd.h @@ -133,6 +133,15 @@ struct scsi_cmnd { unsigned char tag; /* SCSI-II queued command tag */ }; +/* + * Return the driver private allocation behind the command. + * Only works if cmd_size is set in the host template. + */ +static inline void *scsi_cmd_priv(struct scsi_cmnd *cmd) +{ + return cmd + 1; +} + /* make sure not to use it with REQ_TYPE_BLOCK_PC commands */ static inline struct scsi_driver *scsi_cmd_to_driver(struct scsi_cmnd *cmd) { diff --git a/include/scsi/scsi_driver.h b/include/scsi/scsi_driver.h index 20fdfc2526ad..36c4114ed9bc 100644 --- a/include/scsi/scsi_driver.h +++ b/include/scsi/scsi_driver.h @@ -4,17 +4,17 @@ #include <linux/device.h> struct module; +struct request; struct scsi_cmnd; struct scsi_device; -struct request; -struct request_queue; - struct scsi_driver { struct module *owner; struct device_driver gendrv; void (*rescan)(struct device *); + int (*init_command)(struct scsi_cmnd *); + void (*uninit_command)(struct scsi_cmnd *); int (*done)(struct scsi_cmnd *); int (*eh_action)(struct scsi_cmnd *, int); }; @@ -31,8 +31,5 @@ extern int scsi_register_interface(struct class_interface *); int scsi_setup_blk_pc_cmnd(struct scsi_device *sdev, struct request *req); int scsi_setup_fs_cmnd(struct scsi_device *sdev, struct request *req); -int scsi_prep_state_check(struct scsi_device *sdev, struct request *req); -int scsi_prep_return(struct request_queue *q, struct request *req, int ret); -int scsi_prep_fn(struct request_queue *, struct request *); #endif /* _SCSI_SCSI_DRIVER_H */ diff --git a/include/trace/events/f2fs.h b/include/trace/events/f2fs.h index 67f38faac589..b983990b4a9f 100644 --- a/include/trace/events/f2fs.h +++ b/include/trace/events/f2fs.h @@ -659,6 +659,66 @@ DEFINE_EVENT_CONDITION(f2fs__submit_bio, f2fs_submit_read_bio, TP_CONDITION(bio) ); +TRACE_EVENT(f2fs_write_begin, + + TP_PROTO(struct inode *inode, loff_t pos, unsigned int len, + unsigned int flags), + + TP_ARGS(inode, pos, len, flags), + + TP_STRUCT__entry( + __field(dev_t, dev) + __field(ino_t, ino) + __field(loff_t, pos) + __field(unsigned int, len) + __field(unsigned int, flags) + ), + + TP_fast_assign( + __entry->dev = inode->i_sb->s_dev; + __entry->ino = inode->i_ino; + __entry->pos = pos; + __entry->len = len; + __entry->flags = flags; + ), + + TP_printk("dev = (%d,%d), ino = %lu, pos = %llu, len = %u, flags = %u", + show_dev_ino(__entry), + (unsigned long long)__entry->pos, + __entry->len, + __entry->flags) +); + +TRACE_EVENT(f2fs_write_end, + + TP_PROTO(struct inode *inode, loff_t pos, unsigned int len, + unsigned int copied), + + TP_ARGS(inode, pos, len, copied), + + TP_STRUCT__entry( + __field(dev_t, dev) + __field(ino_t, ino) + __field(loff_t, pos) + __field(unsigned int, len) + __field(unsigned int, copied) + ), + + TP_fast_assign( + __entry->dev = inode->i_sb->s_dev; + __entry->ino = inode->i_ino; + __entry->pos = pos; + __entry->len = len; + __entry->copied = copied; + ), + + TP_printk("dev = (%d,%d), ino = %lu, pos = %llu, len = %u, copied = %u", + show_dev_ino(__entry), + (unsigned long long)__entry->pos, + __entry->len, + __entry->copied) +); + DECLARE_EVENT_CLASS(f2fs__page, TP_PROTO(struct page *page, int type), @@ -672,6 +732,7 @@ DECLARE_EVENT_CLASS(f2fs__page, __field(int, dir) __field(pgoff_t, index) __field(int, dirty) + __field(int, uptodate) ), TP_fast_assign( @@ -681,14 +742,31 @@ DECLARE_EVENT_CLASS(f2fs__page, __entry->dir = S_ISDIR(page->mapping->host->i_mode); __entry->index = page->index; __entry->dirty = PageDirty(page); + __entry->uptodate = PageUptodate(page); ), - TP_printk("dev = (%d,%d), ino = %lu, %s, %s, index = %lu, dirty = %d", + TP_printk("dev = (%d,%d), ino = %lu, %s, %s, index = %lu, " + "dirty = %d, uptodate = %d", show_dev_ino(__entry), show_block_type(__entry->type), show_file_type(__entry->dir), (unsigned long)__entry->index, - __entry->dirty) + __entry->dirty, + __entry->uptodate) +); + +DEFINE_EVENT(f2fs__page, f2fs_writepage, + + TP_PROTO(struct page *page, int type), + + TP_ARGS(page, type) +); + +DEFINE_EVENT(f2fs__page, f2fs_readpage, + + TP_PROTO(struct page *page, int type), + + TP_ARGS(page, type) ); DEFINE_EVENT(f2fs__page, f2fs_set_page_dirty, @@ -705,6 +783,70 @@ DEFINE_EVENT(f2fs__page, f2fs_vm_page_mkwrite, TP_ARGS(page, type) ); +TRACE_EVENT(f2fs_writepages, + + TP_PROTO(struct inode *inode, struct writeback_control *wbc, int type), + + TP_ARGS(inode, wbc, type), + + TP_STRUCT__entry( + __field(dev_t, dev) + __field(ino_t, ino) + __field(int, type) + __field(int, dir) + __field(long, nr_to_write) + __field(long, pages_skipped) + __field(loff_t, range_start) + __field(loff_t, range_end) + __field(pgoff_t, writeback_index) + __field(int, sync_mode) + __field(char, for_kupdate) + __field(char, for_background) + __field(char, tagged_writepages) + __field(char, for_reclaim) + __field(char, range_cyclic) + __field(char, for_sync) + ), + + TP_fast_assign( + __entry->dev = inode->i_sb->s_dev; + __entry->ino = inode->i_ino; + __entry->type = type; + __entry->dir = S_ISDIR(inode->i_mode); + __entry->nr_to_write = wbc->nr_to_write; + __entry->pages_skipped = wbc->pages_skipped; + __entry->range_start = wbc->range_start; + __entry->range_end = wbc->range_end; + __entry->writeback_index = inode->i_mapping->writeback_index; + __entry->sync_mode = wbc->sync_mode; + __entry->for_kupdate = wbc->for_kupdate; + __entry->for_background = wbc->for_background; + __entry->tagged_writepages = wbc->tagged_writepages; + __entry->for_reclaim = wbc->for_reclaim; + __entry->range_cyclic = wbc->range_cyclic; + __entry->for_sync = wbc->for_sync; + ), + + TP_printk("dev = (%d,%d), ino = %lu, %s, %s, nr_to_write %ld, " + "skipped %ld, start %lld, end %lld, wb_idx %lu, sync_mode %d, " + "kupdate %u background %u tagged %u reclaim %u cyclic %u sync %u", + show_dev_ino(__entry), + show_block_type(__entry->type), + show_file_type(__entry->dir), + __entry->nr_to_write, + __entry->pages_skipped, + __entry->range_start, + __entry->range_end, + (unsigned long)__entry->writeback_index, + __entry->sync_mode, + __entry->for_kupdate, + __entry->for_background, + __entry->tagged_writepages, + __entry->for_reclaim, + __entry->range_cyclic, + __entry->for_sync) +); + TRACE_EVENT(f2fs_submit_page_mbio, TP_PROTO(struct page *page, int rw, int type, block_t blk_addr), diff --git a/include/trace/ftrace.h b/include/trace/ftrace.h index 0a1a4f7caf09..0fd06fef9fac 100644 --- a/include/trace/ftrace.h +++ b/include/trace/ftrace.h @@ -53,6 +53,9 @@ #undef __string #define __string(item, src) __dynamic_array(char, item, -1) +#undef __bitmask +#define __bitmask(item, nr_bits) __dynamic_array(char, item, -1) + #undef TP_STRUCT__entry #define TP_STRUCT__entry(args...) args @@ -128,6 +131,9 @@ #undef __string #define __string(item, src) __dynamic_array(char, item, -1) +#undef __bitmask +#define __bitmask(item, nr_bits) __dynamic_array(unsigned long, item, -1) + #undef DECLARE_EVENT_CLASS #define DECLARE_EVENT_CLASS(call, proto, args, tstruct, assign, print) \ struct ftrace_data_offsets_##call { \ @@ -197,9 +203,22 @@ #define __get_dynamic_array(field) \ ((void *)__entry + (__entry->__data_loc_##field & 0xffff)) +#undef __get_dynamic_array_len +#define __get_dynamic_array_len(field) \ + ((__entry->__data_loc_##field >> 16) & 0xffff) + #undef __get_str #define __get_str(field) (char *)__get_dynamic_array(field) +#undef __get_bitmask +#define __get_bitmask(field) \ + ({ \ + void *__bitmask = __get_dynamic_array(field); \ + unsigned int __bitmask_size; \ + __bitmask_size = __get_dynamic_array_len(field); \ + ftrace_print_bitmask_seq(p, __bitmask, __bitmask_size); \ + }) + #undef __print_flags #define __print_flags(flag, delim, flag_array...) \ ({ \ @@ -322,6 +341,9 @@ static struct trace_event_functions ftrace_event_type_funcs_##call = { \ #undef __string #define __string(item, src) __dynamic_array(char, item, -1) +#undef __bitmask +#define __bitmask(item, nr_bits) __dynamic_array(unsigned long, item, -1) + #undef DECLARE_EVENT_CLASS #define DECLARE_EVENT_CLASS(call, proto, args, tstruct, func, print) \ static int notrace __init \ @@ -372,6 +394,29 @@ ftrace_define_fields_##call(struct ftrace_event_call *event_call) \ #define __string(item, src) __dynamic_array(char, item, \ strlen((src) ? (const char *)(src) : "(null)") + 1) +/* + * __bitmask_size_in_bytes_raw is the number of bytes needed to hold + * num_possible_cpus(). + */ +#define __bitmask_size_in_bytes_raw(nr_bits) \ + (((nr_bits) + 7) / 8) + +#define __bitmask_size_in_longs(nr_bits) \ + ((__bitmask_size_in_bytes_raw(nr_bits) + \ + ((BITS_PER_LONG / 8) - 1)) / (BITS_PER_LONG / 8)) + +/* + * __bitmask_size_in_bytes is the number of bytes needed to hold + * num_possible_cpus() padded out to the nearest long. This is what + * is saved in the buffer, just to be consistent. + */ +#define __bitmask_size_in_bytes(nr_bits) \ + (__bitmask_size_in_longs(nr_bits) * (BITS_PER_LONG / 8)) + +#undef __bitmask +#define __bitmask(item, nr_bits) __dynamic_array(unsigned long, item, \ + __bitmask_size_in_longs(nr_bits)) + #undef DECLARE_EVENT_CLASS #define DECLARE_EVENT_CLASS(call, proto, args, tstruct, assign, print) \ static inline notrace int ftrace_get_offsets_##call( \ @@ -513,12 +558,22 @@ static inline notrace int ftrace_get_offsets_##call( \ __entry->__data_loc_##item = __data_offsets.item; #undef __string -#define __string(item, src) __dynamic_array(char, item, -1) \ +#define __string(item, src) __dynamic_array(char, item, -1) #undef __assign_str #define __assign_str(dst, src) \ strcpy(__get_str(dst), (src) ? (const char *)(src) : "(null)"); +#undef __bitmask +#define __bitmask(item, nr_bits) __dynamic_array(unsigned long, item, -1) + +#undef __get_bitmask +#define __get_bitmask(field) (char *)__get_dynamic_array(field) + +#undef __assign_bitmask +#define __assign_bitmask(dst, src, nr_bits) \ + memcpy(__get_bitmask(dst), (src), __bitmask_size_in_bytes(nr_bits)) + #undef TP_fast_assign #define TP_fast_assign(args...) args @@ -585,7 +640,9 @@ static inline void ftrace_test_probe_##call(void) \ #undef __print_symbolic #undef __print_hex #undef __get_dynamic_array +#undef __get_dynamic_array_len #undef __get_str +#undef __get_bitmask #undef TP_printk #define TP_printk(fmt, args...) "\"" fmt "\", " __stringify(args) @@ -648,9 +705,16 @@ __attribute__((section("_ftrace_events"))) *__event_##call = &event_##call #define __get_dynamic_array(field) \ ((void *)__entry + (__entry->__data_loc_##field & 0xffff)) +#undef __get_dynamic_array_len +#define __get_dynamic_array_len(field) \ + ((__entry->__data_loc_##field >> 16) & 0xffff) + #undef __get_str #define __get_str(field) (char *)__get_dynamic_array(field) +#undef __get_bitmask +#define __get_bitmask(field) (char *)__get_dynamic_array(field) + #undef __perf_addr #define __perf_addr(a) (__addr = (a)) diff --git a/include/uapi/linux/kvm_para.h b/include/uapi/linux/kvm_para.h index 2841f86eae0b..bf6cd7d5cac2 100644 --- a/include/uapi/linux/kvm_para.h +++ b/include/uapi/linux/kvm_para.h @@ -20,6 +20,9 @@ #define KVM_HC_FEATURES 3 #define KVM_HC_PPC_MAP_MAGIC_PAGE 4 #define KVM_HC_KICK_CPU 5 +#define KVM_HC_MIPS_GET_CLOCK_FREQ 6 +#define KVM_HC_MIPS_EXIT_VM 7 +#define KVM_HC_MIPS_CONSOLE_OUTPUT 8 /* * hypercalls use architecture specific diff --git a/include/uapi/linux/nfsd/nfsfh.h b/include/uapi/linux/nfsd/nfsfh.h index 616e3b396476..20391235d088 100644 --- a/include/uapi/linux/nfsd/nfsfh.h +++ b/include/uapi/linux/nfsd/nfsfh.h @@ -1,13 +1,7 @@ /* - * include/linux/nfsd/nfsfh.h - * * This file describes the layout of the file handles as passed * over the wire. * - * Earlier versions of knfsd used to sign file handles using keyed MD5 - * or SHA. I've removed this code, because it doesn't give you more - * security than blocking external access to port 2049 on your firewall. - * * Copyright (C) 1995, 1996, 1997 Olaf Kirch <okir@monad.swb.de> */ @@ -37,7 +31,7 @@ struct nfs_fhbase_old { }; /* - * This is the new flexible, extensible style NFSv2/v3 file handle. + * This is the new flexible, extensible style NFSv2/v3/v4 file handle. * by Neil Brown <neilb@cse.unsw.edu.au> - March 2000 * * The file handle starts with a sequence of four-byte words. @@ -47,14 +41,7 @@ struct nfs_fhbase_old { * * All four-byte values are in host-byte-order. * - * The auth_type field specifies how the filehandle can be authenticated - * This might allow a file to be confirmed to be in a writable part of a - * filetree without checking the path from it up to the root. - * Current values: - * 0 - No authentication. fb_auth is 0 bytes long - * Possible future values: - * 1 - 4 bytes taken from MD5 hash of the remainer of the file handle - * prefixed by a secret and with the important export flags. + * The auth_type field is deprecated and must be set to 0. * * The fsid_type identifies how the filesystem (or export point) is * encoded. @@ -71,14 +58,9 @@ struct nfs_fhbase_old { * 7 - 8 byte inode number and 16 byte uuid * * The fileid_type identified how the file within the filesystem is encoded. - * This is (will be) passed to, and set by, the underlying filesystem if it supports - * filehandle operations. The filesystem must not use the value '0' or '0xff' and may - * only use the values 1 and 2 as defined below: - * Current values: - * 0 - The root, or export point, of the filesystem. fb_fileid is 0 bytes. - * 1 - 32bit inode number, 32 bit generation number. - * 2 - 32bit inode number, 32 bit generation number, 32 bit parent directory inode number. - * + * The values for this field are filesystem specific, exccept that + * filesystems must not use the values '0' or '0xff'. 'See enum fid_type' + * in include/linux/exportfs.h for currently registered values. */ struct nfs_fhbase_new { __u8 fb_version; /* == 1, even => nfs_fhbase_old */ @@ -114,9 +96,9 @@ struct knfsd_fh { #define fh_fsid_type fh_base.fh_new.fb_fsid_type #define fh_auth_type fh_base.fh_new.fb_auth_type #define fh_fileid_type fh_base.fh_new.fb_fileid_type -#define fh_auth fh_base.fh_new.fb_auth #define fh_fsid fh_base.fh_new.fb_auth - +/* Do not use, provided for userspace compatiblity. */ +#define fh_auth fh_base.fh_new.fb_auth #endif /* _UAPI_LINUX_NFSD_FH_H */ diff --git a/include/uapi/rdma/rdma_netlink.h b/include/uapi/rdma/rdma_netlink.h index 8297285b6288..de69170a30ce 100644 --- a/include/uapi/rdma/rdma_netlink.h +++ b/include/uapi/rdma/rdma_netlink.h @@ -4,7 +4,16 @@ #include <linux/types.h> enum { - RDMA_NL_RDMA_CM = 1 + RDMA_NL_RDMA_CM = 1, + RDMA_NL_NES, + RDMA_NL_C4IW, + RDMA_NL_NUM_CLIENTS +}; + +enum { + RDMA_NL_GROUP_CM = 1, + RDMA_NL_GROUP_IWPM, + RDMA_NL_NUM_GROUPS }; #define RDMA_NL_GET_CLIENT(type) ((type & (((1 << 6) - 1) << 10)) >> 10) @@ -22,6 +31,18 @@ enum { RDMA_NL_RDMA_CM_NUM_ATTR, }; +/* iwarp port mapper op-codes */ +enum { + RDMA_NL_IWPM_REG_PID = 0, + RDMA_NL_IWPM_ADD_MAPPING, + RDMA_NL_IWPM_QUERY_MAPPING, + RDMA_NL_IWPM_REMOVE_MAPPING, + RDMA_NL_IWPM_HANDLE_ERR, + RDMA_NL_IWPM_MAPINFO, + RDMA_NL_IWPM_MAPINFO_NUM, + RDMA_NL_IWPM_NUM_OPS +}; + struct rdma_cm_id_stats { __u32 qp_num; __u32 bound_dev_if; @@ -33,5 +54,78 @@ struct rdma_cm_id_stats { __u8 qp_type; }; +enum { + IWPM_NLA_REG_PID_UNSPEC = 0, + IWPM_NLA_REG_PID_SEQ, + IWPM_NLA_REG_IF_NAME, + IWPM_NLA_REG_IBDEV_NAME, + IWPM_NLA_REG_ULIB_NAME, + IWPM_NLA_REG_PID_MAX +}; + +enum { + IWPM_NLA_RREG_PID_UNSPEC = 0, + IWPM_NLA_RREG_PID_SEQ, + IWPM_NLA_RREG_IBDEV_NAME, + IWPM_NLA_RREG_ULIB_NAME, + IWPM_NLA_RREG_ULIB_VER, + IWPM_NLA_RREG_PID_ERR, + IWPM_NLA_RREG_PID_MAX + +}; + +enum { + IWPM_NLA_MANAGE_MAPPING_UNSPEC = 0, + IWPM_NLA_MANAGE_MAPPING_SEQ, + IWPM_NLA_MANAGE_ADDR, + IWPM_NLA_MANAGE_MAPPED_LOC_ADDR, + IWPM_NLA_RMANAGE_MAPPING_ERR, + IWPM_NLA_RMANAGE_MAPPING_MAX +}; + +#define IWPM_NLA_MANAGE_MAPPING_MAX 3 +#define IWPM_NLA_QUERY_MAPPING_MAX 4 +#define IWPM_NLA_MAPINFO_SEND_MAX 3 + +enum { + IWPM_NLA_QUERY_MAPPING_UNSPEC = 0, + IWPM_NLA_QUERY_MAPPING_SEQ, + IWPM_NLA_QUERY_LOCAL_ADDR, + IWPM_NLA_QUERY_REMOTE_ADDR, + IWPM_NLA_RQUERY_MAPPED_LOC_ADDR, + IWPM_NLA_RQUERY_MAPPED_REM_ADDR, + IWPM_NLA_RQUERY_MAPPING_ERR, + IWPM_NLA_RQUERY_MAPPING_MAX +}; + +enum { + IWPM_NLA_MAPINFO_REQ_UNSPEC = 0, + IWPM_NLA_MAPINFO_ULIB_NAME, + IWPM_NLA_MAPINFO_ULIB_VER, + IWPM_NLA_MAPINFO_REQ_MAX +}; + +enum { + IWPM_NLA_MAPINFO_UNSPEC = 0, + IWPM_NLA_MAPINFO_LOCAL_ADDR, + IWPM_NLA_MAPINFO_MAPPED_ADDR, + IWPM_NLA_MAPINFO_MAX +}; + +enum { + IWPM_NLA_MAPINFO_NUM_UNSPEC = 0, + IWPM_NLA_MAPINFO_SEQ, + IWPM_NLA_MAPINFO_SEND_NUM, + IWPM_NLA_MAPINFO_ACK_NUM, + IWPM_NLA_MAPINFO_NUM_MAX +}; + +enum { + IWPM_NLA_ERR_UNSPEC = 0, + IWPM_NLA_ERR_SEQ, + IWPM_NLA_ERR_CODE, + IWPM_NLA_ERR_MAX +}; + #endif /* _UAPI_RDMA_NETLINK_H */ diff --git a/kernel/auditsc.c b/kernel/auditsc.c index f251a5e8d17a..21eae3c05ec0 100644 --- a/kernel/auditsc.c +++ b/kernel/auditsc.c @@ -728,6 +728,22 @@ static enum audit_state audit_filter_task(struct task_struct *tsk, char **key) return AUDIT_BUILD_CONTEXT; } +static int audit_in_mask(const struct audit_krule *rule, unsigned long val) +{ + int word, bit; + + if (val > 0xffffffff) + return false; + + word = AUDIT_WORD(val); + if (word >= AUDIT_BITMASK_SIZE) + return false; + + bit = AUDIT_BIT(val); + + return rule->mask[word] & bit; +} + /* At syscall entry and exit time, this filter is called if the * audit_state is not low enough that auditing cannot take place, but is * also not high enough that we already know we have to write an audit @@ -745,11 +761,8 @@ static enum audit_state audit_filter_syscall(struct task_struct *tsk, rcu_read_lock(); if (!list_empty(list)) { - int word = AUDIT_WORD(ctx->major); - int bit = AUDIT_BIT(ctx->major); - list_for_each_entry_rcu(e, list, list) { - if ((e->rule.mask[word] & bit) == bit && + if (audit_in_mask(&e->rule, ctx->major) && audit_filter_rules(tsk, &e->rule, ctx, NULL, &state, false)) { rcu_read_unlock(); @@ -769,20 +782,16 @@ static enum audit_state audit_filter_syscall(struct task_struct *tsk, static int audit_filter_inode_name(struct task_struct *tsk, struct audit_names *n, struct audit_context *ctx) { - int word, bit; int h = audit_hash_ino((u32)n->ino); struct list_head *list = &audit_inode_hash[h]; struct audit_entry *e; enum audit_state state; - word = AUDIT_WORD(ctx->major); - bit = AUDIT_BIT(ctx->major); - if (list_empty(list)) return 0; list_for_each_entry_rcu(e, list, list) { - if ((e->rule.mask[word] & bit) == bit && + if (audit_in_mask(&e->rule, ctx->major) && audit_filter_rules(tsk, &e->rule, ctx, n, &state, false)) { ctx->current_state = state; return 1; diff --git a/kernel/capability.c b/kernel/capability.c index 84b2bbf443e7..a5cf13c018ce 100644 --- a/kernel/capability.c +++ b/kernel/capability.c @@ -424,23 +424,19 @@ bool capable(int cap) EXPORT_SYMBOL(capable); /** - * inode_capable - Check superior capability over inode + * capable_wrt_inode_uidgid - Check nsown_capable and uid and gid mapped * @inode: The inode in question * @cap: The capability in question * - * Return true if the current task has the given superior capability - * targeted at it's own user namespace and that the given inode is owned - * by the current user namespace or a child namespace. - * - * Currently we check to see if an inode is owned by the current - * user namespace by seeing if the inode's owner maps into the - * current user namespace. - * + * Return true if the current task has the given capability targeted at + * its own user namespace and that the given inode's uid and gid are + * mapped into the current user namespace. */ -bool inode_capable(const struct inode *inode, int cap) +bool capable_wrt_inode_uidgid(const struct inode *inode, int cap) { struct user_namespace *ns = current_user_ns(); - return ns_capable(ns, cap) && kuid_has_mapping(ns, inode->i_uid); + return ns_capable(ns, cap) && kuid_has_mapping(ns, inode->i_uid) && + kgid_has_mapping(ns, inode->i_gid); } -EXPORT_SYMBOL(inode_capable); +EXPORT_SYMBOL(capable_wrt_inode_uidgid); diff --git a/kernel/cgroup.c b/kernel/cgroup.c index ceee0c54c6a4..7868fc3c0bc5 100644 --- a/kernel/cgroup.c +++ b/kernel/cgroup.c @@ -26,6 +26,8 @@ * distribution for more details. */ +#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt + #include <linux/cgroup.h> #include <linux/cred.h> #include <linux/ctype.h> @@ -70,15 +72,6 @@ MAX_CFTYPE_NAME + 2) /* - * cgroup_tree_mutex nests above cgroup_mutex and protects cftypes, file - * creation/removal and hierarchy changing operations including cgroup - * creation, removal, css association and controller rebinding. This outer - * lock is needed mainly to resolve the circular dependency between kernfs - * active ref and cgroup_mutex. cgroup_tree_mutex nests above both. - */ -static DEFINE_MUTEX(cgroup_tree_mutex); - -/* * cgroup_mutex is the master lock. Any modification to cgroup or its * hierarchy must be performed while holding it. * @@ -99,16 +92,21 @@ static DECLARE_RWSEM(css_set_rwsem); #endif /* + * Protects cgroup_idr and css_idr so that IDs can be released without + * grabbing cgroup_mutex. + */ +static DEFINE_SPINLOCK(cgroup_idr_lock); + +/* * Protects cgroup_subsys->release_agent_path. Modifying it also requires * cgroup_mutex. Reading requires either cgroup_mutex or this spinlock. */ static DEFINE_SPINLOCK(release_agent_path_lock); -#define cgroup_assert_mutexes_or_rcu_locked() \ +#define cgroup_assert_mutex_or_rcu_locked() \ rcu_lockdep_assert(rcu_read_lock_held() || \ - lockdep_is_held(&cgroup_tree_mutex) || \ lockdep_is_held(&cgroup_mutex), \ - "cgroup_[tree_]mutex or RCU read lock required"); + "cgroup_mutex or RCU read lock required"); /* * cgroup destruction makes heavy use of work items and there can be a lot @@ -151,6 +149,13 @@ struct cgroup_root cgrp_dfl_root; */ static bool cgrp_dfl_root_visible; +/* some controllers are not supported in the default hierarchy */ +static const unsigned int cgrp_dfl_root_inhibit_ss_mask = 0 +#ifdef CONFIG_CGROUP_DEBUG + | (1 << debug_cgrp_id) +#endif + ; + /* The list of hierarchy roots */ static LIST_HEAD(cgroup_roots); @@ -160,14 +165,13 @@ static int cgroup_root_count; static DEFINE_IDR(cgroup_hierarchy_idr); /* - * Assign a monotonically increasing serial number to cgroups. It - * guarantees cgroups with bigger numbers are newer than those with smaller - * numbers. Also, as cgroups are always appended to the parent's - * ->children list, it guarantees that sibling cgroups are always sorted in - * the ascending serial number order on the list. Protected by - * cgroup_mutex. + * Assign a monotonically increasing serial number to csses. It guarantees + * cgroups with bigger numbers are newer than those with smaller numbers. + * Also, as csses are always appended to the parent's ->children list, it + * guarantees that sibling csses are always sorted in the ascending serial + * number order on the list. Protected by cgroup_mutex. */ -static u64 cgroup_serial_nr_next = 1; +static u64 css_serial_nr_next = 1; /* This flag indicates whether tasks in the fork and exit paths should * check for fork/exit handlers to call. This avoids us having to do @@ -180,17 +184,59 @@ static struct cftype cgroup_base_files[]; static void cgroup_put(struct cgroup *cgrp); static int rebind_subsystems(struct cgroup_root *dst_root, - unsigned long ss_mask); -static void cgroup_destroy_css_killed(struct cgroup *cgrp); + unsigned int ss_mask); static int cgroup_destroy_locked(struct cgroup *cgrp); +static int create_css(struct cgroup *cgrp, struct cgroup_subsys *ss); +static void css_release(struct percpu_ref *ref); +static void kill_css(struct cgroup_subsys_state *css); static int cgroup_addrm_files(struct cgroup *cgrp, struct cftype cfts[], bool is_add); static void cgroup_pidlist_destroy_all(struct cgroup *cgrp); +/* IDR wrappers which synchronize using cgroup_idr_lock */ +static int cgroup_idr_alloc(struct idr *idr, void *ptr, int start, int end, + gfp_t gfp_mask) +{ + int ret; + + idr_preload(gfp_mask); + spin_lock_bh(&cgroup_idr_lock); + ret = idr_alloc(idr, ptr, start, end, gfp_mask); + spin_unlock_bh(&cgroup_idr_lock); + idr_preload_end(); + return ret; +} + +static void *cgroup_idr_replace(struct idr *idr, void *ptr, int id) +{ + void *ret; + + spin_lock_bh(&cgroup_idr_lock); + ret = idr_replace(idr, ptr, id); + spin_unlock_bh(&cgroup_idr_lock); + return ret; +} + +static void cgroup_idr_remove(struct idr *idr, int id) +{ + spin_lock_bh(&cgroup_idr_lock); + idr_remove(idr, id); + spin_unlock_bh(&cgroup_idr_lock); +} + +static struct cgroup *cgroup_parent(struct cgroup *cgrp) +{ + struct cgroup_subsys_state *parent_css = cgrp->self.parent; + + if (parent_css) + return container_of(parent_css, struct cgroup, self); + return NULL; +} + /** * cgroup_css - obtain a cgroup's css for the specified subsystem * @cgrp: the cgroup of interest - * @ss: the subsystem of interest (%NULL returns the dummy_css) + * @ss: the subsystem of interest (%NULL returns @cgrp->self) * * Return @cgrp's css (cgroup_subsys_state) associated with @ss. This * function must be called either under cgroup_mutex or rcu_read_lock() and @@ -203,23 +249,49 @@ static struct cgroup_subsys_state *cgroup_css(struct cgroup *cgrp, { if (ss) return rcu_dereference_check(cgrp->subsys[ss->id], - lockdep_is_held(&cgroup_tree_mutex) || lockdep_is_held(&cgroup_mutex)); else - return &cgrp->dummy_css; + return &cgrp->self; +} + +/** + * cgroup_e_css - obtain a cgroup's effective css for the specified subsystem + * @cgrp: the cgroup of interest + * @ss: the subsystem of interest (%NULL returns @cgrp->self) + * + * Similar to cgroup_css() but returns the effctive css, which is defined + * as the matching css of the nearest ancestor including self which has @ss + * enabled. If @ss is associated with the hierarchy @cgrp is on, this + * function is guaranteed to return non-NULL css. + */ +static struct cgroup_subsys_state *cgroup_e_css(struct cgroup *cgrp, + struct cgroup_subsys *ss) +{ + lockdep_assert_held(&cgroup_mutex); + + if (!ss) + return &cgrp->self; + + if (!(cgrp->root->subsys_mask & (1 << ss->id))) + return NULL; + + while (cgroup_parent(cgrp) && + !(cgroup_parent(cgrp)->child_subsys_mask & (1 << ss->id))) + cgrp = cgroup_parent(cgrp); + + return cgroup_css(cgrp, ss); } /* convenient tests for these bits */ static inline bool cgroup_is_dead(const struct cgroup *cgrp) { - return test_bit(CGRP_DEAD, &cgrp->flags); + return !(cgrp->self.flags & CSS_ONLINE); } -struct cgroup_subsys_state *seq_css(struct seq_file *seq) +struct cgroup_subsys_state *of_css(struct kernfs_open_file *of) { - struct kernfs_open_file *of = seq->private; struct cgroup *cgrp = of->kn->parent->priv; - struct cftype *cft = seq_cft(seq); + struct cftype *cft = of_cft(of); /* * This is open and unprotected implementation of cgroup_css(). @@ -232,9 +304,9 @@ struct cgroup_subsys_state *seq_css(struct seq_file *seq) if (cft->ss) return rcu_dereference_raw(cgrp->subsys[cft->ss->id]); else - return &cgrp->dummy_css; + return &cgrp->self; } -EXPORT_SYMBOL_GPL(seq_css); +EXPORT_SYMBOL_GPL(of_css); /** * cgroup_is_descendant - test ancestry @@ -250,7 +322,7 @@ bool cgroup_is_descendant(struct cgroup *cgrp, struct cgroup *ancestor) while (cgrp) { if (cgrp == ancestor) return true; - cgrp = cgrp->parent; + cgrp = cgroup_parent(cgrp); } return false; } @@ -274,17 +346,30 @@ static int notify_on_release(const struct cgroup *cgrp) * @ssid: the index of the subsystem, CGROUP_SUBSYS_COUNT after reaching the end * @cgrp: the target cgroup to iterate css's of * - * Should be called under cgroup_mutex. + * Should be called under cgroup_[tree_]mutex. */ #define for_each_css(css, ssid, cgrp) \ for ((ssid) = 0; (ssid) < CGROUP_SUBSYS_COUNT; (ssid)++) \ if (!((css) = rcu_dereference_check( \ (cgrp)->subsys[(ssid)], \ - lockdep_is_held(&cgroup_tree_mutex) || \ lockdep_is_held(&cgroup_mutex)))) { } \ else /** + * for_each_e_css - iterate all effective css's of a cgroup + * @css: the iteration cursor + * @ssid: the index of the subsystem, CGROUP_SUBSYS_COUNT after reaching the end + * @cgrp: the target cgroup to iterate css's of + * + * Should be called under cgroup_[tree_]mutex. + */ +#define for_each_e_css(css, ssid, cgrp) \ + for ((ssid) = 0; (ssid) < CGROUP_SUBSYS_COUNT; (ssid)++) \ + if (!((css) = cgroup_e_css(cgrp, cgroup_subsys[(ssid)]))) \ + ; \ + else + +/** * for_each_subsys - iterate all enabled cgroup subsystems * @ss: the iteration cursor * @ssid: the index of @ss, CGROUP_SUBSYS_COUNT after reaching the end @@ -297,22 +382,13 @@ static int notify_on_release(const struct cgroup *cgrp) #define for_each_root(root) \ list_for_each_entry((root), &cgroup_roots, root_list) -/** - * cgroup_lock_live_group - take cgroup_mutex and check that cgrp is alive. - * @cgrp: the cgroup to be checked for liveness - * - * On success, returns true; the mutex should be later unlocked. On - * failure returns false with no lock held. - */ -static bool cgroup_lock_live_group(struct cgroup *cgrp) -{ - mutex_lock(&cgroup_mutex); - if (cgroup_is_dead(cgrp)) { - mutex_unlock(&cgroup_mutex); - return false; - } - return true; -} +/* iterate over child cgrps, lock should be held throughout iteration */ +#define cgroup_for_each_live_child(child, cgrp) \ + list_for_each_entry((child), &(cgrp)->self.children, self.sibling) \ + if (({ lockdep_assert_held(&cgroup_mutex); \ + cgroup_is_dead(child); })) \ + ; \ + else /* the list of cgroups eligible for automatic release. Protected by * release_list_lock */ @@ -360,6 +436,43 @@ struct css_set init_css_set = { static int css_set_count = 1; /* 1 for init_css_set */ +/** + * cgroup_update_populated - updated populated count of a cgroup + * @cgrp: the target cgroup + * @populated: inc or dec populated count + * + * @cgrp is either getting the first task (css_set) or losing the last. + * Update @cgrp->populated_cnt accordingly. The count is propagated + * towards root so that a given cgroup's populated_cnt is zero iff the + * cgroup and all its descendants are empty. + * + * @cgrp's interface file "cgroup.populated" is zero if + * @cgrp->populated_cnt is zero and 1 otherwise. When @cgrp->populated_cnt + * changes from or to zero, userland is notified that the content of the + * interface file has changed. This can be used to detect when @cgrp and + * its descendants become populated or empty. + */ +static void cgroup_update_populated(struct cgroup *cgrp, bool populated) +{ + lockdep_assert_held(&css_set_rwsem); + + do { + bool trigger; + + if (populated) + trigger = !cgrp->populated_cnt++; + else + trigger = !--cgrp->populated_cnt; + + if (!trigger) + break; + + if (cgrp->populated_kn) + kernfs_notify(cgrp->populated_kn); + cgrp = cgroup_parent(cgrp); + } while (cgrp); +} + /* * hash table for cgroup groups. This improves the performance to find * an existing css_set. This hash doesn't (currently) take into @@ -384,6 +497,8 @@ static unsigned long css_set_hash(struct cgroup_subsys_state *css[]) static void put_css_set_locked(struct css_set *cset, bool taskexit) { struct cgrp_cset_link *link, *tmp_link; + struct cgroup_subsys *ss; + int ssid; lockdep_assert_held(&css_set_rwsem); @@ -391,6 +506,8 @@ static void put_css_set_locked(struct css_set *cset, bool taskexit) return; /* This css_set is dead. unlink it and release cgroup refcounts */ + for_each_subsys(ss, ssid) + list_del(&cset->e_cset_node[ssid]); hash_del(&cset->hlist); css_set_count--; @@ -401,10 +518,13 @@ static void put_css_set_locked(struct css_set *cset, bool taskexit) list_del(&link->cgrp_link); /* @cgrp can't go away while we're holding css_set_rwsem */ - if (list_empty(&cgrp->cset_links) && notify_on_release(cgrp)) { - if (taskexit) - set_bit(CGRP_RELEASABLE, &cgrp->flags); - check_for_release(cgrp); + if (list_empty(&cgrp->cset_links)) { + cgroup_update_populated(cgrp, false); + if (notify_on_release(cgrp)) { + if (taskexit) + set_bit(CGRP_RELEASABLE, &cgrp->flags); + check_for_release(cgrp); + } } kfree(link); @@ -453,20 +573,20 @@ static bool compare_css_sets(struct css_set *cset, { struct list_head *l1, *l2; - if (memcmp(template, cset->subsys, sizeof(cset->subsys))) { - /* Not all subsystems matched */ + /* + * On the default hierarchy, there can be csets which are + * associated with the same set of cgroups but different csses. + * Let's first ensure that csses match. + */ + if (memcmp(template, cset->subsys, sizeof(cset->subsys))) return false; - } /* * Compare cgroup pointers in order to distinguish between - * different cgroups in heirarchies with no subsystems. We - * could get by with just this check alone (and skip the - * memcmp above) but on most setups the memcmp check will - * avoid the need for this more expensive check on almost all - * candidates. + * different cgroups in hierarchies. As different cgroups may + * share the same effective css, this comparison is always + * necessary. */ - l1 = &cset->cgrp_links; l2 = &old_cset->cgrp_links; while (1) { @@ -530,14 +650,17 @@ static struct css_set *find_existing_css_set(struct css_set *old_cset, * won't change, so no need for locking. */ for_each_subsys(ss, i) { - if (root->cgrp.subsys_mask & (1UL << i)) { - /* Subsystem is in this hierarchy. So we want - * the subsystem state from the new - * cgroup */ - template[i] = cgroup_css(cgrp, ss); + if (root->subsys_mask & (1UL << i)) { + /* + * @ss is in this hierarchy, so we want the + * effective css from @cgrp. + */ + template[i] = cgroup_e_css(cgrp, ss); } else { - /* Subsystem is not in this hierarchy, so we - * don't want to change the subsystem state */ + /* + * @ss is not in this hierarchy, so we don't want + * to change the css. + */ template[i] = old_cset->subsys[i]; } } @@ -603,10 +726,18 @@ static void link_css_set(struct list_head *tmp_links, struct css_set *cset, struct cgrp_cset_link *link; BUG_ON(list_empty(tmp_links)); + + if (cgroup_on_dfl(cgrp)) + cset->dfl_cgrp = cgrp; + link = list_first_entry(tmp_links, struct cgrp_cset_link, cset_link); link->cset = cset; link->cgrp = cgrp; + + if (list_empty(&cgrp->cset_links)) + cgroup_update_populated(cgrp, true); list_move(&link->cset_link, &cgrp->cset_links); + /* * Always add links to the tail of the list so that the list * is sorted by order of hierarchy creation @@ -629,7 +760,9 @@ static struct css_set *find_css_set(struct css_set *old_cset, struct css_set *cset; struct list_head tmp_links; struct cgrp_cset_link *link; + struct cgroup_subsys *ss; unsigned long key; + int ssid; lockdep_assert_held(&cgroup_mutex); @@ -680,10 +813,14 @@ static struct css_set *find_css_set(struct css_set *old_cset, css_set_count++; - /* Add this cgroup group to the hash table */ + /* Add @cset to the hash table */ key = css_set_hash(cset->subsys); hash_add(css_set_table, &cset->hlist, key); + for_each_subsys(ss, ssid) + list_add_tail(&cset->e_cset_node[ssid], + &cset->subsys[ssid]->cgroup->e_csets[ssid]); + up_write(&css_set_rwsem); return cset; @@ -736,14 +873,13 @@ static void cgroup_destroy_root(struct cgroup_root *root) struct cgroup *cgrp = &root->cgrp; struct cgrp_cset_link *link, *tmp_link; - mutex_lock(&cgroup_tree_mutex); mutex_lock(&cgroup_mutex); BUG_ON(atomic_read(&root->nr_cgrps)); - BUG_ON(!list_empty(&cgrp->children)); + BUG_ON(!list_empty(&cgrp->self.children)); /* Rebind all subsystems back to the default hierarchy */ - rebind_subsystems(&cgrp_dfl_root, cgrp->subsys_mask); + rebind_subsystems(&cgrp_dfl_root, root->subsys_mask); /* * Release all the links from cset_links to this hierarchy's @@ -766,7 +902,6 @@ static void cgroup_destroy_root(struct cgroup_root *root) cgroup_exit_root_id(root); mutex_unlock(&cgroup_mutex); - mutex_unlock(&cgroup_tree_mutex); kernfs_destroy_root(root->kf_root); cgroup_free_root(root); @@ -849,7 +984,7 @@ static struct cgroup *task_cgroup_from_root(struct task_struct *task, * update of a tasks cgroup pointer by cgroup_attach_task() */ -static int cgroup_populate_dir(struct cgroup *cgrp, unsigned long subsys_mask); +static int cgroup_populate_dir(struct cgroup *cgrp, unsigned int subsys_mask); static struct kernfs_syscall_ops cgroup_kf_syscall_ops; static const struct file_operations proc_cgroupstats_operations; @@ -884,79 +1019,95 @@ static umode_t cgroup_file_mode(const struct cftype *cft) if (cft->read_u64 || cft->read_s64 || cft->seq_show) mode |= S_IRUGO; - if (cft->write_u64 || cft->write_s64 || cft->write_string || - cft->trigger) + if (cft->write_u64 || cft->write_s64 || cft->write) mode |= S_IWUSR; return mode; } -static void cgroup_free_fn(struct work_struct *work) +static void cgroup_get(struct cgroup *cgrp) { - struct cgroup *cgrp = container_of(work, struct cgroup, destroy_work); - - atomic_dec(&cgrp->root->nr_cgrps); - cgroup_pidlist_destroy_all(cgrp); - - if (cgrp->parent) { - /* - * We get a ref to the parent, and put the ref when this - * cgroup is being freed, so it's guaranteed that the - * parent won't be destroyed before its children. - */ - cgroup_put(cgrp->parent); - kernfs_put(cgrp->kn); - kfree(cgrp); - } else { - /* - * This is root cgroup's refcnt reaching zero, which - * indicates that the root should be released. - */ - cgroup_destroy_root(cgrp->root); - } + WARN_ON_ONCE(cgroup_is_dead(cgrp)); + css_get(&cgrp->self); } -static void cgroup_free_rcu(struct rcu_head *head) +static void cgroup_put(struct cgroup *cgrp) { - struct cgroup *cgrp = container_of(head, struct cgroup, rcu_head); - - INIT_WORK(&cgrp->destroy_work, cgroup_free_fn); - queue_work(cgroup_destroy_wq, &cgrp->destroy_work); + css_put(&cgrp->self); } -static void cgroup_get(struct cgroup *cgrp) +/** + * cgroup_kn_unlock - unlocking helper for cgroup kernfs methods + * @kn: the kernfs_node being serviced + * + * This helper undoes cgroup_kn_lock_live() and should be invoked before + * the method finishes if locking succeeded. Note that once this function + * returns the cgroup returned by cgroup_kn_lock_live() may become + * inaccessible any time. If the caller intends to continue to access the + * cgroup, it should pin it before invoking this function. + */ +static void cgroup_kn_unlock(struct kernfs_node *kn) { - WARN_ON_ONCE(cgroup_is_dead(cgrp)); - WARN_ON_ONCE(atomic_read(&cgrp->refcnt) <= 0); - atomic_inc(&cgrp->refcnt); + struct cgroup *cgrp; + + if (kernfs_type(kn) == KERNFS_DIR) + cgrp = kn->priv; + else + cgrp = kn->parent->priv; + + mutex_unlock(&cgroup_mutex); + + kernfs_unbreak_active_protection(kn); + cgroup_put(cgrp); } -static void cgroup_put(struct cgroup *cgrp) +/** + * cgroup_kn_lock_live - locking helper for cgroup kernfs methods + * @kn: the kernfs_node being serviced + * + * This helper is to be used by a cgroup kernfs method currently servicing + * @kn. It breaks the active protection, performs cgroup locking and + * verifies that the associated cgroup is alive. Returns the cgroup if + * alive; otherwise, %NULL. A successful return should be undone by a + * matching cgroup_kn_unlock() invocation. + * + * Any cgroup kernfs method implementation which requires locking the + * associated cgroup should use this helper. It avoids nesting cgroup + * locking under kernfs active protection and allows all kernfs operations + * including self-removal. + */ +static struct cgroup *cgroup_kn_lock_live(struct kernfs_node *kn) { - if (!atomic_dec_and_test(&cgrp->refcnt)) - return; - if (WARN_ON_ONCE(cgrp->parent && !cgroup_is_dead(cgrp))) - return; + struct cgroup *cgrp; + + if (kernfs_type(kn) == KERNFS_DIR) + cgrp = kn->priv; + else + cgrp = kn->parent->priv; /* - * XXX: cgrp->id is only used to look up css's. As cgroup and - * css's lifetimes will be decoupled, it should be made - * per-subsystem and moved to css->id so that lookups are - * successful until the target css is released. + * We're gonna grab cgroup_mutex which nests outside kernfs + * active_ref. cgroup liveliness check alone provides enough + * protection against removal. Ensure @cgrp stays accessible and + * break the active_ref protection. */ + cgroup_get(cgrp); + kernfs_break_active_protection(kn); + mutex_lock(&cgroup_mutex); - idr_remove(&cgrp->root->cgroup_idr, cgrp->id); - mutex_unlock(&cgroup_mutex); - cgrp->id = -1; - call_rcu(&cgrp->rcu_head, cgroup_free_rcu); + if (!cgroup_is_dead(cgrp)) + return cgrp; + + cgroup_kn_unlock(kn); + return NULL; } static void cgroup_rm_file(struct cgroup *cgrp, const struct cftype *cft) { char name[CGROUP_FILE_NAME_MAX]; - lockdep_assert_held(&cgroup_tree_mutex); + lockdep_assert_held(&cgroup_mutex); kernfs_remove_by_name(cgrp->kn, cgroup_file_name(cgrp, cft, name)); } @@ -965,7 +1116,7 @@ static void cgroup_rm_file(struct cgroup *cgrp, const struct cftype *cft) * @cgrp: target cgroup * @subsys_mask: mask of the subsystem ids whose files should be removed */ -static void cgroup_clear_dir(struct cgroup *cgrp, unsigned long subsys_mask) +static void cgroup_clear_dir(struct cgroup *cgrp, unsigned int subsys_mask) { struct cgroup_subsys *ss; int i; @@ -973,40 +1124,40 @@ static void cgroup_clear_dir(struct cgroup *cgrp, unsigned long subsys_mask) for_each_subsys(ss, i) { struct cftype *cfts; - if (!test_bit(i, &subsys_mask)) + if (!(subsys_mask & (1 << i))) continue; list_for_each_entry(cfts, &ss->cfts, node) cgroup_addrm_files(cgrp, cfts, false); } } -static int rebind_subsystems(struct cgroup_root *dst_root, - unsigned long ss_mask) +static int rebind_subsystems(struct cgroup_root *dst_root, unsigned int ss_mask) { struct cgroup_subsys *ss; - int ssid, ret; + unsigned int tmp_ss_mask; + int ssid, i, ret; - lockdep_assert_held(&cgroup_tree_mutex); lockdep_assert_held(&cgroup_mutex); for_each_subsys(ss, ssid) { if (!(ss_mask & (1 << ssid))) continue; - /* if @ss is on the dummy_root, we can always move it */ - if (ss->root == &cgrp_dfl_root) - continue; - - /* if @ss has non-root cgroups attached to it, can't move */ - if (!list_empty(&ss->root->cgrp.children)) + /* if @ss has non-root csses attached to it, can't move */ + if (css_next_child(NULL, cgroup_css(&ss->root->cgrp, ss))) return -EBUSY; /* can't move between two non-dummy roots either */ - if (dst_root != &cgrp_dfl_root) + if (ss->root != &cgrp_dfl_root && dst_root != &cgrp_dfl_root) return -EBUSY; } - ret = cgroup_populate_dir(&dst_root->cgrp, ss_mask); + /* skip creating root files on dfl_root for inhibited subsystems */ + tmp_ss_mask = ss_mask; + if (dst_root == &cgrp_dfl_root) + tmp_ss_mask &= ~cgrp_dfl_root_inhibit_ss_mask; + + ret = cgroup_populate_dir(&dst_root->cgrp, tmp_ss_mask); if (ret) { if (dst_root != &cgrp_dfl_root) return ret; @@ -1018,9 +1169,9 @@ static int rebind_subsystems(struct cgroup_root *dst_root, * Just warn about it and continue. */ if (cgrp_dfl_root_visible) { - pr_warning("cgroup: failed to create files (%d) while rebinding 0x%lx to default root\n", - ret, ss_mask); - pr_warning("cgroup: you may retry by moving them to a different hierarchy and unbinding\n"); + pr_warn("failed to create files (%d) while rebinding 0x%x to default root\n", + ret, ss_mask); + pr_warn("you may retry by moving them to a different hierarchy and unbinding\n"); } } @@ -1028,15 +1179,14 @@ static int rebind_subsystems(struct cgroup_root *dst_root, * Nothing can fail from this point on. Remove files for the * removed subsystems and rebind each subsystem. */ - mutex_unlock(&cgroup_mutex); for_each_subsys(ss, ssid) if (ss_mask & (1 << ssid)) cgroup_clear_dir(&ss->root->cgrp, 1 << ssid); - mutex_lock(&cgroup_mutex); for_each_subsys(ss, ssid) { struct cgroup_root *src_root; struct cgroup_subsys_state *css; + struct css_set *cset; if (!(ss_mask & (1 << ssid))) continue; @@ -1051,8 +1201,19 @@ static int rebind_subsystems(struct cgroup_root *dst_root, ss->root = dst_root; css->cgroup = &dst_root->cgrp; - src_root->cgrp.subsys_mask &= ~(1 << ssid); - dst_root->cgrp.subsys_mask |= 1 << ssid; + down_write(&css_set_rwsem); + hash_for_each(css_set_table, i, cset, hlist) + list_move_tail(&cset->e_cset_node[ss->id], + &dst_root->cgrp.e_csets[ss->id]); + up_write(&css_set_rwsem); + + src_root->subsys_mask &= ~(1 << ssid); + src_root->cgrp.child_subsys_mask &= ~(1 << ssid); + + /* default hierarchy doesn't enable controllers by default */ + dst_root->subsys_mask |= 1 << ssid; + if (dst_root != &cgrp_dfl_root) + dst_root->cgrp.child_subsys_mask |= 1 << ssid; if (ss->bind) ss->bind(css); @@ -1070,7 +1231,7 @@ static int cgroup_show_options(struct seq_file *seq, int ssid; for_each_subsys(ss, ssid) - if (root->cgrp.subsys_mask & (1 << ssid)) + if (root->subsys_mask & (1 << ssid)) seq_printf(seq, ",%s", ss->name); if (root->flags & CGRP_ROOT_SANE_BEHAVIOR) seq_puts(seq, ",sane_behavior"); @@ -1092,8 +1253,8 @@ static int cgroup_show_options(struct seq_file *seq, } struct cgroup_sb_opts { - unsigned long subsys_mask; - unsigned long flags; + unsigned int subsys_mask; + unsigned int flags; char *release_agent; bool cpuset_clone_children; char *name; @@ -1101,24 +1262,16 @@ struct cgroup_sb_opts { bool none; }; -/* - * Convert a hierarchy specifier into a bitmask of subsystems and - * flags. Call with cgroup_mutex held to protect the cgroup_subsys[] - * array. This function takes refcounts on subsystems to be used, unless it - * returns error, in which case no refcounts are taken. - */ static int parse_cgroupfs_options(char *data, struct cgroup_sb_opts *opts) { char *token, *o = data; bool all_ss = false, one_ss = false; - unsigned long mask = (unsigned long)-1; + unsigned int mask = -1U; struct cgroup_subsys *ss; int i; - BUG_ON(!mutex_is_locked(&cgroup_mutex)); - #ifdef CONFIG_CPUSETS - mask = ~(1UL << cpuset_cgrp_id); + mask = ~(1U << cpuset_cgrp_id); #endif memset(opts, 0, sizeof(*opts)); @@ -1199,7 +1352,7 @@ static int parse_cgroupfs_options(char *data, struct cgroup_sb_opts *opts) /* Mutually exclusive option 'all' + subsystem name */ if (all_ss) return -EINVAL; - set_bit(i, &opts->subsys_mask); + opts->subsys_mask |= (1 << i); one_ss = true; break; @@ -1211,12 +1364,12 @@ static int parse_cgroupfs_options(char *data, struct cgroup_sb_opts *opts) /* Consistency checks */ if (opts->flags & CGRP_ROOT_SANE_BEHAVIOR) { - pr_warning("cgroup: sane_behavior: this is still under development and its behaviors will change, proceed at your own risk\n"); + pr_warn("sane_behavior: this is still under development and its behaviors will change, proceed at your own risk\n"); if ((opts->flags & (CGRP_ROOT_NOPREFIX | CGRP_ROOT_XATTR)) || opts->cpuset_clone_children || opts->release_agent || opts->name) { - pr_err("cgroup: sane_behavior: noprefix, xattr, clone_children, release_agent and name are not allowed\n"); + pr_err("sane_behavior: noprefix, xattr, clone_children, release_agent and name are not allowed\n"); return -EINVAL; } } else { @@ -1228,7 +1381,7 @@ static int parse_cgroupfs_options(char *data, struct cgroup_sb_opts *opts) if (all_ss || (!one_ss && !opts->none && !opts->name)) for_each_subsys(ss, i) if (!ss->disabled) - set_bit(i, &opts->subsys_mask); + opts->subsys_mask |= (1 << i); /* * We either have to specify by name or by subsystems. (So @@ -1259,14 +1412,13 @@ static int cgroup_remount(struct kernfs_root *kf_root, int *flags, char *data) int ret = 0; struct cgroup_root *root = cgroup_root_from_kf(kf_root); struct cgroup_sb_opts opts; - unsigned long added_mask, removed_mask; + unsigned int added_mask, removed_mask; if (root->flags & CGRP_ROOT_SANE_BEHAVIOR) { - pr_err("cgroup: sane_behavior: remount is not allowed\n"); + pr_err("sane_behavior: remount is not allowed\n"); return -EINVAL; } - mutex_lock(&cgroup_tree_mutex); mutex_lock(&cgroup_mutex); /* See what subsystems are wanted */ @@ -1274,17 +1426,17 @@ static int cgroup_remount(struct kernfs_root *kf_root, int *flags, char *data) if (ret) goto out_unlock; - if (opts.subsys_mask != root->cgrp.subsys_mask || opts.release_agent) - pr_warning("cgroup: option changes via remount are deprecated (pid=%d comm=%s)\n", - task_tgid_nr(current), current->comm); + if (opts.subsys_mask != root->subsys_mask || opts.release_agent) + pr_warn("option changes via remount are deprecated (pid=%d comm=%s)\n", + task_tgid_nr(current), current->comm); - added_mask = opts.subsys_mask & ~root->cgrp.subsys_mask; - removed_mask = root->cgrp.subsys_mask & ~opts.subsys_mask; + added_mask = opts.subsys_mask & ~root->subsys_mask; + removed_mask = root->subsys_mask & ~opts.subsys_mask; /* Don't allow flags or name to change at remount */ if (((opts.flags ^ root->flags) & CGRP_ROOT_OPTION_MASK) || (opts.name && strcmp(opts.name, root->name))) { - pr_err("cgroup: option or name mismatch, new: 0x%lx \"%s\", old: 0x%lx \"%s\"\n", + pr_err("option or name mismatch, new: 0x%x \"%s\", old: 0x%x \"%s\"\n", opts.flags & CGRP_ROOT_OPTION_MASK, opts.name ?: "", root->flags & CGRP_ROOT_OPTION_MASK, root->name); ret = -EINVAL; @@ -1292,7 +1444,7 @@ static int cgroup_remount(struct kernfs_root *kf_root, int *flags, char *data) } /* remounting is not allowed for populated hierarchies */ - if (!list_empty(&root->cgrp.children)) { + if (!list_empty(&root->cgrp.self.children)) { ret = -EBUSY; goto out_unlock; } @@ -1312,7 +1464,6 @@ static int cgroup_remount(struct kernfs_root *kf_root, int *flags, char *data) kfree(opts.release_agent); kfree(opts.name); mutex_unlock(&cgroup_mutex); - mutex_unlock(&cgroup_tree_mutex); return ret; } @@ -1370,14 +1521,22 @@ out_unlock: static void init_cgroup_housekeeping(struct cgroup *cgrp) { - atomic_set(&cgrp->refcnt, 1); - INIT_LIST_HEAD(&cgrp->sibling); - INIT_LIST_HEAD(&cgrp->children); + struct cgroup_subsys *ss; + int ssid; + + INIT_LIST_HEAD(&cgrp->self.sibling); + INIT_LIST_HEAD(&cgrp->self.children); INIT_LIST_HEAD(&cgrp->cset_links); INIT_LIST_HEAD(&cgrp->release_list); INIT_LIST_HEAD(&cgrp->pidlists); mutex_init(&cgrp->pidlist_mutex); - cgrp->dummy_css.cgroup = cgrp; + cgrp->self.cgroup = cgrp; + cgrp->self.flags |= CSS_ONLINE; + + for_each_subsys(ss, ssid) + INIT_LIST_HEAD(&cgrp->e_csets[ssid]); + + init_waitqueue_head(&cgrp->offline_waitq); } static void init_cgroup_root(struct cgroup_root *root, @@ -1400,21 +1559,24 @@ static void init_cgroup_root(struct cgroup_root *root, set_bit(CGRP_CPUSET_CLONE_CHILDREN, &root->cgrp.flags); } -static int cgroup_setup_root(struct cgroup_root *root, unsigned long ss_mask) +static int cgroup_setup_root(struct cgroup_root *root, unsigned int ss_mask) { LIST_HEAD(tmp_links); struct cgroup *root_cgrp = &root->cgrp; struct css_set *cset; int i, ret; - lockdep_assert_held(&cgroup_tree_mutex); lockdep_assert_held(&cgroup_mutex); - ret = idr_alloc(&root->cgroup_idr, root_cgrp, 0, 1, GFP_KERNEL); + ret = cgroup_idr_alloc(&root->cgroup_idr, root_cgrp, 1, 2, GFP_NOWAIT); if (ret < 0) goto out; root_cgrp->id = ret; + ret = percpu_ref_init(&root_cgrp->self.refcnt, css_release); + if (ret) + goto out; + /* * We're accessing css_set_count without locking css_set_rwsem here, * but that's OK - it can only be increased by someone holding @@ -1423,11 +1585,11 @@ static int cgroup_setup_root(struct cgroup_root *root, unsigned long ss_mask) */ ret = allocate_cgrp_cset_links(css_set_count, &tmp_links); if (ret) - goto out; + goto cancel_ref; ret = cgroup_init_root_id(root); if (ret) - goto out; + goto cancel_ref; root->kf_root = kernfs_create_root(&cgroup_kf_syscall_ops, KERNFS_ROOT_CREATE_DEACTIVATED, @@ -1463,7 +1625,7 @@ static int cgroup_setup_root(struct cgroup_root *root, unsigned long ss_mask) link_css_set(&tmp_links, cset, root_cgrp); up_write(&css_set_rwsem); - BUG_ON(!list_empty(&root_cgrp->children)); + BUG_ON(!list_empty(&root_cgrp->self.children)); BUG_ON(atomic_read(&root->nr_cgrps) != 1); kernfs_activate(root_cgrp->kn); @@ -1475,6 +1637,8 @@ destroy_root: root->kf_root = NULL; exit_root_id: cgroup_exit_root_id(root); +cancel_ref: + percpu_ref_cancel_init(&root_cgrp->self.refcnt); out: free_cgrp_cset_links(&tmp_links); return ret; @@ -1497,14 +1661,13 @@ static struct dentry *cgroup_mount(struct file_system_type *fs_type, if (!use_task_css_set_links) cgroup_enable_task_cg_lists(); - mutex_lock(&cgroup_tree_mutex); mutex_lock(&cgroup_mutex); /* First find the desired set of subsystems */ ret = parse_cgroupfs_options(data, &opts); if (ret) goto out_unlock; -retry: + /* look for a matching existing root */ if (!opts.subsys_mask && !opts.none && !opts.name) { cgrp_dfl_root_visible = true; @@ -1536,7 +1699,7 @@ retry: * subsystems) then they must match. */ if ((opts.subsys_mask || opts.none) && - (opts.subsys_mask != root->cgrp.subsys_mask)) { + (opts.subsys_mask != root->subsys_mask)) { if (!name_match) continue; ret = -EBUSY; @@ -1545,28 +1708,27 @@ retry: if ((root->flags ^ opts.flags) & CGRP_ROOT_OPTION_MASK) { if ((root->flags | opts.flags) & CGRP_ROOT_SANE_BEHAVIOR) { - pr_err("cgroup: sane_behavior: new mount options should match the existing superblock\n"); + pr_err("sane_behavior: new mount options should match the existing superblock\n"); ret = -EINVAL; goto out_unlock; } else { - pr_warning("cgroup: new mount options do not match the existing superblock, will be ignored\n"); + pr_warn("new mount options do not match the existing superblock, will be ignored\n"); } } /* - * A root's lifetime is governed by its root cgroup. Zero - * ref indicate that the root is being destroyed. Wait for - * destruction to complete so that the subsystems are free. - * We can use wait_queue for the wait but this path is - * super cold. Let's just sleep for a bit and retry. + * A root's lifetime is governed by its root cgroup. + * tryget_live failure indicate that the root is being + * destroyed. Wait for destruction to complete so that the + * subsystems are free. We can use wait_queue for the wait + * but this path is super cold. Let's just sleep for a bit + * and retry. */ - if (!atomic_inc_not_zero(&root->cgrp.refcnt)) { + if (!percpu_ref_tryget_live(&root->cgrp.self.refcnt)) { mutex_unlock(&cgroup_mutex); - mutex_unlock(&cgroup_tree_mutex); msleep(10); - mutex_lock(&cgroup_tree_mutex); - mutex_lock(&cgroup_mutex); - goto retry; + ret = restart_syscall(); + goto out_free; } ret = 0; @@ -1597,8 +1759,7 @@ retry: out_unlock: mutex_unlock(&cgroup_mutex); - mutex_unlock(&cgroup_tree_mutex); - +out_free: kfree(opts.release_agent); kfree(opts.name); @@ -1617,7 +1778,19 @@ static void cgroup_kill_sb(struct super_block *sb) struct kernfs_root *kf_root = kernfs_root_from_sb(sb); struct cgroup_root *root = cgroup_root_from_kf(kf_root); - cgroup_put(&root->cgrp); + /* + * If @root doesn't have any mounts or children, start killing it. + * This prevents new mounts by disabling percpu_ref_tryget_live(). + * cgroup_mount() may wait for @root's release. + * + * And don't kill the default root. + */ + if (css_has_online_children(&root->cgrp.self) || + root == &cgrp_dfl_root) + cgroup_put(&root->cgrp); + else + percpu_ref_kill(&root->cgrp.self.refcnt); + kernfs_kill_sb(sb); } @@ -1739,7 +1912,7 @@ struct task_struct *cgroup_taskset_next(struct cgroup_taskset *tset) /** * cgroup_task_migrate - move a task from one cgroup to another. - * @old_cgrp; the cgroup @tsk is being migrated from + * @old_cgrp: the cgroup @tsk is being migrated from * @tsk: the task being migrated * @new_cset: the new css_set @tsk is being attached to * @@ -1831,10 +2004,6 @@ static void cgroup_migrate_add_src(struct css_set *src_cset, src_cgrp = cset_cgroup_from_root(src_cset, dst_cgrp->root); - /* nothing to do if this cset already belongs to the cgroup */ - if (src_cgrp == dst_cgrp) - return; - if (!list_empty(&src_cset->mg_preload_node)) return; @@ -1849,13 +2018,14 @@ static void cgroup_migrate_add_src(struct css_set *src_cset, /** * cgroup_migrate_prepare_dst - prepare destination css_sets for migration - * @dst_cgrp: the destination cgroup + * @dst_cgrp: the destination cgroup (may be %NULL) * @preloaded_csets: list of preloaded source css_sets * * Tasks are about to be moved to @dst_cgrp and all the source css_sets * have been preloaded to @preloaded_csets. This function looks up and - * pins all destination css_sets, links each to its source, and put them on - * @preloaded_csets. + * pins all destination css_sets, links each to its source, and append them + * to @preloaded_csets. If @dst_cgrp is %NULL, the destination of each + * source css_set is assumed to be its cgroup on the default hierarchy. * * This function must be called after cgroup_migrate_add_src() has been * called on each migration source css_set. After migration is performed @@ -1866,19 +2036,42 @@ static int cgroup_migrate_prepare_dst(struct cgroup *dst_cgrp, struct list_head *preloaded_csets) { LIST_HEAD(csets); - struct css_set *src_cset; + struct css_set *src_cset, *tmp_cset; lockdep_assert_held(&cgroup_mutex); + /* + * Except for the root, child_subsys_mask must be zero for a cgroup + * with tasks so that child cgroups don't compete against tasks. + */ + if (dst_cgrp && cgroup_on_dfl(dst_cgrp) && cgroup_parent(dst_cgrp) && + dst_cgrp->child_subsys_mask) + return -EBUSY; + /* look up the dst cset for each src cset and link it to src */ - list_for_each_entry(src_cset, preloaded_csets, mg_preload_node) { + list_for_each_entry_safe(src_cset, tmp_cset, preloaded_csets, mg_preload_node) { struct css_set *dst_cset; - dst_cset = find_css_set(src_cset, dst_cgrp); + dst_cset = find_css_set(src_cset, + dst_cgrp ?: src_cset->dfl_cgrp); if (!dst_cset) goto err; WARN_ON_ONCE(src_cset->mg_dst_cset || dst_cset->mg_dst_cset); + + /* + * If src cset equals dst, it's noop. Drop the src. + * cgroup_migrate() will skip the cset too. Note that we + * can't handle src == dst as some nodes are used by both. + */ + if (src_cset == dst_cset) { + src_cset->mg_src_cgrp = NULL; + list_del_init(&src_cset->mg_preload_node); + put_css_set(src_cset, false); + put_css_set(dst_cset, false); + continue; + } + src_cset->mg_dst_cset = dst_cset; if (list_empty(&dst_cset->mg_preload_node)) @@ -1887,7 +2080,7 @@ static int cgroup_migrate_prepare_dst(struct cgroup *dst_cgrp, put_css_set(dst_cset, false); } - list_splice(&csets, preloaded_csets); + list_splice_tail(&csets, preloaded_csets); return 0; err: cgroup_migrate_finish(&csets); @@ -1968,7 +2161,7 @@ static int cgroup_migrate(struct cgroup *cgrp, struct task_struct *leader, return 0; /* check that we can legitimately attach to the cgroup */ - for_each_css(css, i, cgrp) { + for_each_e_css(css, i, cgrp) { if (css->ss->can_attach) { ret = css->ss->can_attach(css, &tset); if (ret) { @@ -1998,7 +2191,7 @@ static int cgroup_migrate(struct cgroup *cgrp, struct task_struct *leader, */ tset.csets = &tset.dst_csets; - for_each_css(css, i, cgrp) + for_each_e_css(css, i, cgrp) if (css->ss->attach) css->ss->attach(css, &tset); @@ -2006,7 +2199,7 @@ static int cgroup_migrate(struct cgroup *cgrp, struct task_struct *leader, goto out_release_tset; out_cancel_attach: - for_each_css(css, i, cgrp) { + for_each_e_css(css, i, cgrp) { if (css == failed_css) break; if (css->ss->cancel_attach) @@ -2065,13 +2258,20 @@ static int cgroup_attach_task(struct cgroup *dst_cgrp, * function to attach either it or all tasks in its threadgroup. Will lock * cgroup_mutex and threadgroup. */ -static int attach_task_by_pid(struct cgroup *cgrp, u64 pid, bool threadgroup) +static ssize_t __cgroup_procs_write(struct kernfs_open_file *of, char *buf, + size_t nbytes, loff_t off, bool threadgroup) { struct task_struct *tsk; const struct cred *cred = current_cred(), *tcred; + struct cgroup *cgrp; + pid_t pid; int ret; - if (!cgroup_lock_live_group(cgrp)) + if (kstrtoint(strstrip(buf), 0, &pid) || pid < 0) + return -EINVAL; + + cgrp = cgroup_kn_lock_live(of->kn); + if (!cgrp) return -ENODEV; retry_find_task: @@ -2137,8 +2337,8 @@ retry_find_task: put_task_struct(tsk); out_unlock_cgroup: - mutex_unlock(&cgroup_mutex); - return ret; + cgroup_kn_unlock(of->kn); + return ret ?: nbytes; } /** @@ -2172,43 +2372,44 @@ int cgroup_attach_task_all(struct task_struct *from, struct task_struct *tsk) } EXPORT_SYMBOL_GPL(cgroup_attach_task_all); -static int cgroup_tasks_write(struct cgroup_subsys_state *css, - struct cftype *cft, u64 pid) +static ssize_t cgroup_tasks_write(struct kernfs_open_file *of, + char *buf, size_t nbytes, loff_t off) { - return attach_task_by_pid(css->cgroup, pid, false); + return __cgroup_procs_write(of, buf, nbytes, off, false); } -static int cgroup_procs_write(struct cgroup_subsys_state *css, - struct cftype *cft, u64 tgid) +static ssize_t cgroup_procs_write(struct kernfs_open_file *of, + char *buf, size_t nbytes, loff_t off) { - return attach_task_by_pid(css->cgroup, tgid, true); + return __cgroup_procs_write(of, buf, nbytes, off, true); } -static int cgroup_release_agent_write(struct cgroup_subsys_state *css, - struct cftype *cft, char *buffer) +static ssize_t cgroup_release_agent_write(struct kernfs_open_file *of, + char *buf, size_t nbytes, loff_t off) { - struct cgroup_root *root = css->cgroup->root; + struct cgroup *cgrp; - BUILD_BUG_ON(sizeof(root->release_agent_path) < PATH_MAX); - if (!cgroup_lock_live_group(css->cgroup)) + BUILD_BUG_ON(sizeof(cgrp->root->release_agent_path) < PATH_MAX); + + cgrp = cgroup_kn_lock_live(of->kn); + if (!cgrp) return -ENODEV; spin_lock(&release_agent_path_lock); - strlcpy(root->release_agent_path, buffer, - sizeof(root->release_agent_path)); + strlcpy(cgrp->root->release_agent_path, strstrip(buf), + sizeof(cgrp->root->release_agent_path)); spin_unlock(&release_agent_path_lock); - mutex_unlock(&cgroup_mutex); - return 0; + cgroup_kn_unlock(of->kn); + return nbytes; } static int cgroup_release_agent_show(struct seq_file *seq, void *v) { struct cgroup *cgrp = seq_css(seq)->cgroup; - if (!cgroup_lock_live_group(cgrp)) - return -ENODEV; + spin_lock(&release_agent_path_lock); seq_puts(seq, cgrp->root->release_agent_path); + spin_unlock(&release_agent_path_lock); seq_putc(seq, '\n'); - mutex_unlock(&cgroup_mutex); return 0; } @@ -2220,6 +2421,320 @@ static int cgroup_sane_behavior_show(struct seq_file *seq, void *v) return 0; } +static void cgroup_print_ss_mask(struct seq_file *seq, unsigned int ss_mask) +{ + struct cgroup_subsys *ss; + bool printed = false; + int ssid; + + for_each_subsys(ss, ssid) { + if (ss_mask & (1 << ssid)) { + if (printed) + seq_putc(seq, ' '); + seq_printf(seq, "%s", ss->name); + printed = true; + } + } + if (printed) + seq_putc(seq, '\n'); +} + +/* show controllers which are currently attached to the default hierarchy */ +static int cgroup_root_controllers_show(struct seq_file *seq, void *v) +{ + struct cgroup *cgrp = seq_css(seq)->cgroup; + + cgroup_print_ss_mask(seq, cgrp->root->subsys_mask & + ~cgrp_dfl_root_inhibit_ss_mask); + return 0; +} + +/* show controllers which are enabled from the parent */ +static int cgroup_controllers_show(struct seq_file *seq, void *v) +{ + struct cgroup *cgrp = seq_css(seq)->cgroup; + + cgroup_print_ss_mask(seq, cgroup_parent(cgrp)->child_subsys_mask); + return 0; +} + +/* show controllers which are enabled for a given cgroup's children */ +static int cgroup_subtree_control_show(struct seq_file *seq, void *v) +{ + struct cgroup *cgrp = seq_css(seq)->cgroup; + + cgroup_print_ss_mask(seq, cgrp->child_subsys_mask); + return 0; +} + +/** + * cgroup_update_dfl_csses - update css assoc of a subtree in default hierarchy + * @cgrp: root of the subtree to update csses for + * + * @cgrp's child_subsys_mask has changed and its subtree's (self excluded) + * css associations need to be updated accordingly. This function looks up + * all css_sets which are attached to the subtree, creates the matching + * updated css_sets and migrates the tasks to the new ones. + */ +static int cgroup_update_dfl_csses(struct cgroup *cgrp) +{ + LIST_HEAD(preloaded_csets); + struct cgroup_subsys_state *css; + struct css_set *src_cset; + int ret; + + lockdep_assert_held(&cgroup_mutex); + + /* look up all csses currently attached to @cgrp's subtree */ + down_read(&css_set_rwsem); + css_for_each_descendant_pre(css, cgroup_css(cgrp, NULL)) { + struct cgrp_cset_link *link; + + /* self is not affected by child_subsys_mask change */ + if (css->cgroup == cgrp) + continue; + + list_for_each_entry(link, &css->cgroup->cset_links, cset_link) + cgroup_migrate_add_src(link->cset, cgrp, + &preloaded_csets); + } + up_read(&css_set_rwsem); + + /* NULL dst indicates self on default hierarchy */ + ret = cgroup_migrate_prepare_dst(NULL, &preloaded_csets); + if (ret) + goto out_finish; + + list_for_each_entry(src_cset, &preloaded_csets, mg_preload_node) { + struct task_struct *last_task = NULL, *task; + + /* src_csets precede dst_csets, break on the first dst_cset */ + if (!src_cset->mg_src_cgrp) + break; + + /* + * All tasks in src_cset need to be migrated to the + * matching dst_cset. Empty it process by process. We + * walk tasks but migrate processes. The leader might even + * belong to a different cset but such src_cset would also + * be among the target src_csets because the default + * hierarchy enforces per-process membership. + */ + while (true) { + down_read(&css_set_rwsem); + task = list_first_entry_or_null(&src_cset->tasks, + struct task_struct, cg_list); + if (task) { + task = task->group_leader; + WARN_ON_ONCE(!task_css_set(task)->mg_src_cgrp); + get_task_struct(task); + } + up_read(&css_set_rwsem); + + if (!task) + break; + + /* guard against possible infinite loop */ + if (WARN(last_task == task, + "cgroup: update_dfl_csses failed to make progress, aborting in inconsistent state\n")) + goto out_finish; + last_task = task; + + threadgroup_lock(task); + /* raced against de_thread() from another thread? */ + if (!thread_group_leader(task)) { + threadgroup_unlock(task); + put_task_struct(task); + continue; + } + + ret = cgroup_migrate(src_cset->dfl_cgrp, task, true); + + threadgroup_unlock(task); + put_task_struct(task); + + if (WARN(ret, "cgroup: failed to update controllers for the default hierarchy (%d), further operations may crash or hang\n", ret)) + goto out_finish; + } + } + +out_finish: + cgroup_migrate_finish(&preloaded_csets); + return ret; +} + +/* change the enabled child controllers for a cgroup in the default hierarchy */ +static ssize_t cgroup_subtree_control_write(struct kernfs_open_file *of, + char *buf, size_t nbytes, + loff_t off) +{ + unsigned int enable = 0, disable = 0; + struct cgroup *cgrp, *child; + struct cgroup_subsys *ss; + char *tok; + int ssid, ret; + + /* + * Parse input - space separated list of subsystem names prefixed + * with either + or -. + */ + buf = strstrip(buf); + while ((tok = strsep(&buf, " "))) { + if (tok[0] == '\0') + continue; + for_each_subsys(ss, ssid) { + if (ss->disabled || strcmp(tok + 1, ss->name) || + ((1 << ss->id) & cgrp_dfl_root_inhibit_ss_mask)) + continue; + + if (*tok == '+') { + enable |= 1 << ssid; + disable &= ~(1 << ssid); + } else if (*tok == '-') { + disable |= 1 << ssid; + enable &= ~(1 << ssid); + } else { + return -EINVAL; + } + break; + } + if (ssid == CGROUP_SUBSYS_COUNT) + return -EINVAL; + } + + cgrp = cgroup_kn_lock_live(of->kn); + if (!cgrp) + return -ENODEV; + + for_each_subsys(ss, ssid) { + if (enable & (1 << ssid)) { + if (cgrp->child_subsys_mask & (1 << ssid)) { + enable &= ~(1 << ssid); + continue; + } + + /* + * Because css offlining is asynchronous, userland + * might try to re-enable the same controller while + * the previous instance is still around. In such + * cases, wait till it's gone using offline_waitq. + */ + cgroup_for_each_live_child(child, cgrp) { + DEFINE_WAIT(wait); + + if (!cgroup_css(child, ss)) + continue; + + cgroup_get(child); + prepare_to_wait(&child->offline_waitq, &wait, + TASK_UNINTERRUPTIBLE); + cgroup_kn_unlock(of->kn); + schedule(); + finish_wait(&child->offline_waitq, &wait); + cgroup_put(child); + + return restart_syscall(); + } + + /* unavailable or not enabled on the parent? */ + if (!(cgrp_dfl_root.subsys_mask & (1 << ssid)) || + (cgroup_parent(cgrp) && + !(cgroup_parent(cgrp)->child_subsys_mask & (1 << ssid)))) { + ret = -ENOENT; + goto out_unlock; + } + } else if (disable & (1 << ssid)) { + if (!(cgrp->child_subsys_mask & (1 << ssid))) { + disable &= ~(1 << ssid); + continue; + } + + /* a child has it enabled? */ + cgroup_for_each_live_child(child, cgrp) { + if (child->child_subsys_mask & (1 << ssid)) { + ret = -EBUSY; + goto out_unlock; + } + } + } + } + + if (!enable && !disable) { + ret = 0; + goto out_unlock; + } + + /* + * Except for the root, child_subsys_mask must be zero for a cgroup + * with tasks so that child cgroups don't compete against tasks. + */ + if (enable && cgroup_parent(cgrp) && !list_empty(&cgrp->cset_links)) { + ret = -EBUSY; + goto out_unlock; + } + + /* + * Create csses for enables and update child_subsys_mask. This + * changes cgroup_e_css() results which in turn makes the + * subsequent cgroup_update_dfl_csses() associate all tasks in the + * subtree to the updated csses. + */ + for_each_subsys(ss, ssid) { + if (!(enable & (1 << ssid))) + continue; + + cgroup_for_each_live_child(child, cgrp) { + ret = create_css(child, ss); + if (ret) + goto err_undo_css; + } + } + + cgrp->child_subsys_mask |= enable; + cgrp->child_subsys_mask &= ~disable; + + ret = cgroup_update_dfl_csses(cgrp); + if (ret) + goto err_undo_css; + + /* all tasks are now migrated away from the old csses, kill them */ + for_each_subsys(ss, ssid) { + if (!(disable & (1 << ssid))) + continue; + + cgroup_for_each_live_child(child, cgrp) + kill_css(cgroup_css(child, ss)); + } + + kernfs_activate(cgrp->kn); + ret = 0; +out_unlock: + cgroup_kn_unlock(of->kn); + return ret ?: nbytes; + +err_undo_css: + cgrp->child_subsys_mask &= ~enable; + cgrp->child_subsys_mask |= disable; + + for_each_subsys(ss, ssid) { + if (!(enable & (1 << ssid))) + continue; + + cgroup_for_each_live_child(child, cgrp) { + struct cgroup_subsys_state *css = cgroup_css(child, ss); + if (css) + kill_css(css); + } + } + goto out_unlock; +} + +static int cgroup_populated_show(struct seq_file *seq, void *v) +{ + seq_printf(seq, "%d\n", (bool)seq_css(seq)->cgroup->populated_cnt); + return 0; +} + static ssize_t cgroup_file_write(struct kernfs_open_file *of, char *buf, size_t nbytes, loff_t off) { @@ -2228,6 +2743,9 @@ static ssize_t cgroup_file_write(struct kernfs_open_file *of, char *buf, struct cgroup_subsys_state *css; int ret; + if (cft->write) + return cft->write(of, buf, nbytes, off); + /* * kernfs guarantees that a file isn't deleted with operations in * flight, which means that the matching css is and stays alive and @@ -2238,9 +2756,7 @@ static ssize_t cgroup_file_write(struct kernfs_open_file *of, char *buf, css = cgroup_css(cgrp, cft->ss); rcu_read_unlock(); - if (cft->write_string) { - ret = cft->write_string(css, cft, strstrip(buf)); - } else if (cft->write_u64) { + if (cft->write_u64) { unsigned long long v; ret = kstrtoull(buf, 0, &v); if (!ret) @@ -2250,8 +2766,6 @@ static ssize_t cgroup_file_write(struct kernfs_open_file *of, char *buf, ret = kstrtoll(buf, 0, &v); if (!ret) ret = cft->write_s64(css, cft, v); - } else if (cft->trigger) { - ret = cft->trigger(css, (unsigned int)cft->private); } else { ret = -EINVAL; } @@ -2328,20 +2842,18 @@ static int cgroup_rename(struct kernfs_node *kn, struct kernfs_node *new_parent, return -EPERM; /* - * We're gonna grab cgroup_tree_mutex which nests outside kernfs + * We're gonna grab cgroup_mutex which nests outside kernfs * active_ref. kernfs_rename() doesn't require active_ref - * protection. Break them before grabbing cgroup_tree_mutex. + * protection. Break them before grabbing cgroup_mutex. */ kernfs_break_active_protection(new_parent); kernfs_break_active_protection(kn); - mutex_lock(&cgroup_tree_mutex); mutex_lock(&cgroup_mutex); ret = kernfs_rename(kn, new_parent, new_name_str); mutex_unlock(&cgroup_mutex); - mutex_unlock(&cgroup_tree_mutex); kernfs_unbreak_active_protection(kn); kernfs_unbreak_active_protection(new_parent); @@ -2379,9 +2891,14 @@ static int cgroup_add_file(struct cgroup *cgrp, struct cftype *cft) return PTR_ERR(kn); ret = cgroup_kn_set_ugid(kn); - if (ret) + if (ret) { kernfs_remove(kn); - return ret; + return ret; + } + + if (cft->seq_show == cgroup_populated_show) + cgrp->populated_kn = kn; + return 0; } /** @@ -2401,7 +2918,7 @@ static int cgroup_addrm_files(struct cgroup *cgrp, struct cftype cfts[], struct cftype *cft; int ret; - lockdep_assert_held(&cgroup_tree_mutex); + lockdep_assert_held(&cgroup_mutex); for (cft = cfts; cft->name[0] != '\0'; cft++) { /* does cft->flags tell us to skip this file on @cgrp? */ @@ -2409,16 +2926,16 @@ static int cgroup_addrm_files(struct cgroup *cgrp, struct cftype cfts[], continue; if ((cft->flags & CFTYPE_INSANE) && cgroup_sane_behavior(cgrp)) continue; - if ((cft->flags & CFTYPE_NOT_ON_ROOT) && !cgrp->parent) + if ((cft->flags & CFTYPE_NOT_ON_ROOT) && !cgroup_parent(cgrp)) continue; - if ((cft->flags & CFTYPE_ONLY_ON_ROOT) && cgrp->parent) + if ((cft->flags & CFTYPE_ONLY_ON_ROOT) && cgroup_parent(cgrp)) continue; if (is_add) { ret = cgroup_add_file(cgrp, cft); if (ret) { - pr_warn("cgroup_addrm_files: failed to add %s, err=%d\n", - cft->name, ret); + pr_warn("%s: failed to add %s, err=%d\n", + __func__, cft->name, ret); return ret; } } else { @@ -2436,11 +2953,7 @@ static int cgroup_apply_cftypes(struct cftype *cfts, bool is_add) struct cgroup_subsys_state *css; int ret = 0; - lockdep_assert_held(&cgroup_tree_mutex); - - /* don't bother if @ss isn't attached */ - if (ss->root == &cgrp_dfl_root) - return 0; + lockdep_assert_held(&cgroup_mutex); /* add/rm files for all cgroups created before */ css_for_each_descendant_pre(css, cgroup_css(root, ss)) { @@ -2508,7 +3021,7 @@ static int cgroup_init_cftypes(struct cgroup_subsys *ss, struct cftype *cfts) static int cgroup_rm_cftypes_locked(struct cftype *cfts) { - lockdep_assert_held(&cgroup_tree_mutex); + lockdep_assert_held(&cgroup_mutex); if (!cfts || !cfts[0].ss) return -ENOENT; @@ -2534,9 +3047,9 @@ int cgroup_rm_cftypes(struct cftype *cfts) { int ret; - mutex_lock(&cgroup_tree_mutex); + mutex_lock(&cgroup_mutex); ret = cgroup_rm_cftypes_locked(cfts); - mutex_unlock(&cgroup_tree_mutex); + mutex_unlock(&cgroup_mutex); return ret; } @@ -2558,6 +3071,9 @@ int cgroup_add_cftypes(struct cgroup_subsys *ss, struct cftype *cfts) { int ret; + if (ss->disabled) + return 0; + if (!cfts || cfts[0].name[0] == '\0') return 0; @@ -2565,14 +3081,14 @@ int cgroup_add_cftypes(struct cgroup_subsys *ss, struct cftype *cfts) if (ret) return ret; - mutex_lock(&cgroup_tree_mutex); + mutex_lock(&cgroup_mutex); list_add_tail(&cfts->node, &ss->cfts); ret = cgroup_apply_cftypes(cfts, true); if (ret) cgroup_rm_cftypes_locked(cfts); - mutex_unlock(&cgroup_tree_mutex); + mutex_unlock(&cgroup_mutex); return ret; } @@ -2596,57 +3112,65 @@ static int cgroup_task_count(const struct cgroup *cgrp) /** * css_next_child - find the next child of a given css - * @pos_css: the current position (%NULL to initiate traversal) - * @parent_css: css whose children to walk + * @pos: the current position (%NULL to initiate traversal) + * @parent: css whose children to walk * - * This function returns the next child of @parent_css and should be called + * This function returns the next child of @parent and should be called * under either cgroup_mutex or RCU read lock. The only requirement is - * that @parent_css and @pos_css are accessible. The next sibling is - * guaranteed to be returned regardless of their states. + * that @parent and @pos are accessible. The next sibling is guaranteed to + * be returned regardless of their states. + * + * If a subsystem synchronizes ->css_online() and the start of iteration, a + * css which finished ->css_online() is guaranteed to be visible in the + * future iterations and will stay visible until the last reference is put. + * A css which hasn't finished ->css_online() or already finished + * ->css_offline() may show up during traversal. It's each subsystem's + * responsibility to synchronize against on/offlining. */ -struct cgroup_subsys_state * -css_next_child(struct cgroup_subsys_state *pos_css, - struct cgroup_subsys_state *parent_css) +struct cgroup_subsys_state *css_next_child(struct cgroup_subsys_state *pos, + struct cgroup_subsys_state *parent) { - struct cgroup *pos = pos_css ? pos_css->cgroup : NULL; - struct cgroup *cgrp = parent_css->cgroup; - struct cgroup *next; + struct cgroup_subsys_state *next; - cgroup_assert_mutexes_or_rcu_locked(); + cgroup_assert_mutex_or_rcu_locked(); /* - * @pos could already have been removed. Once a cgroup is removed, - * its ->sibling.next is no longer updated when its next sibling - * changes. As CGRP_DEAD assertion is serialized and happens - * before the cgroup is taken off the ->sibling list, if we see it - * unasserted, it's guaranteed that the next sibling hasn't - * finished its grace period even if it's already removed, and thus - * safe to dereference from this RCU critical section. If - * ->sibling.next is inaccessible, cgroup_is_dead() is guaranteed - * to be visible as %true here. + * @pos could already have been unlinked from the sibling list. + * Once a cgroup is removed, its ->sibling.next is no longer + * updated when its next sibling changes. CSS_RELEASED is set when + * @pos is taken off list, at which time its next pointer is valid, + * and, as releases are serialized, the one pointed to by the next + * pointer is guaranteed to not have started release yet. This + * implies that if we observe !CSS_RELEASED on @pos in this RCU + * critical section, the one pointed to by its next pointer is + * guaranteed to not have finished its RCU grace period even if we + * have dropped rcu_read_lock() inbetween iterations. * - * If @pos is dead, its next pointer can't be dereferenced; - * however, as each cgroup is given a monotonically increasing - * unique serial number and always appended to the sibling list, - * the next one can be found by walking the parent's children until - * we see a cgroup with higher serial number than @pos's. While - * this path can be slower, it's taken only when either the current - * cgroup is removed or iteration and removal race. + * If @pos has CSS_RELEASED set, its next pointer can't be + * dereferenced; however, as each css is given a monotonically + * increasing unique serial number and always appended to the + * sibling list, the next one can be found by walking the parent's + * children until the first css with higher serial number than + * @pos's. While this path can be slower, it happens iff iteration + * races against release and the race window is very small. */ if (!pos) { - next = list_entry_rcu(cgrp->children.next, struct cgroup, sibling); - } else if (likely(!cgroup_is_dead(pos))) { - next = list_entry_rcu(pos->sibling.next, struct cgroup, sibling); + next = list_entry_rcu(parent->children.next, struct cgroup_subsys_state, sibling); + } else if (likely(!(pos->flags & CSS_RELEASED))) { + next = list_entry_rcu(pos->sibling.next, struct cgroup_subsys_state, sibling); } else { - list_for_each_entry_rcu(next, &cgrp->children, sibling) + list_for_each_entry_rcu(next, &parent->children, sibling) if (next->serial_nr > pos->serial_nr) break; } - if (&next->sibling == &cgrp->children) - return NULL; - - return cgroup_css(next, parent_css->ss); + /* + * @next, if not pointing to the head, can be dereferenced and is + * the next sibling. + */ + if (&next->sibling != &parent->children) + return next; + return NULL; } /** @@ -2662,6 +3186,13 @@ css_next_child(struct cgroup_subsys_state *pos_css, * doesn't require the whole traversal to be contained in a single critical * section. This function will return the correct next descendant as long * as both @pos and @root are accessible and @pos is a descendant of @root. + * + * If a subsystem synchronizes ->css_online() and the start of iteration, a + * css which finished ->css_online() is guaranteed to be visible in the + * future iterations and will stay visible until the last reference is put. + * A css which hasn't finished ->css_online() or already finished + * ->css_offline() may show up during traversal. It's each subsystem's + * responsibility to synchronize against on/offlining. */ struct cgroup_subsys_state * css_next_descendant_pre(struct cgroup_subsys_state *pos, @@ -2669,7 +3200,7 @@ css_next_descendant_pre(struct cgroup_subsys_state *pos, { struct cgroup_subsys_state *next; - cgroup_assert_mutexes_or_rcu_locked(); + cgroup_assert_mutex_or_rcu_locked(); /* if first iteration, visit @root */ if (!pos) @@ -2682,10 +3213,10 @@ css_next_descendant_pre(struct cgroup_subsys_state *pos, /* no child, visit my or the closest ancestor's next sibling */ while (pos != root) { - next = css_next_child(pos, css_parent(pos)); + next = css_next_child(pos, pos->parent); if (next) return next; - pos = css_parent(pos); + pos = pos->parent; } return NULL; @@ -2709,7 +3240,7 @@ css_rightmost_descendant(struct cgroup_subsys_state *pos) { struct cgroup_subsys_state *last, *tmp; - cgroup_assert_mutexes_or_rcu_locked(); + cgroup_assert_mutex_or_rcu_locked(); do { last = pos; @@ -2749,6 +3280,13 @@ css_leftmost_descendant(struct cgroup_subsys_state *pos) * section. This function will return the correct next descendant as long * as both @pos and @cgroup are accessible and @pos is a descendant of * @cgroup. + * + * If a subsystem synchronizes ->css_online() and the start of iteration, a + * css which finished ->css_online() is guaranteed to be visible in the + * future iterations and will stay visible until the last reference is put. + * A css which hasn't finished ->css_online() or already finished + * ->css_offline() may show up during traversal. It's each subsystem's + * responsibility to synchronize against on/offlining. */ struct cgroup_subsys_state * css_next_descendant_post(struct cgroup_subsys_state *pos, @@ -2756,7 +3294,7 @@ css_next_descendant_post(struct cgroup_subsys_state *pos, { struct cgroup_subsys_state *next; - cgroup_assert_mutexes_or_rcu_locked(); + cgroup_assert_mutex_or_rcu_locked(); /* if first iteration, visit leftmost descendant which may be @root */ if (!pos) @@ -2767,12 +3305,36 @@ css_next_descendant_post(struct cgroup_subsys_state *pos, return NULL; /* if there's an unvisited sibling, visit its leftmost descendant */ - next = css_next_child(pos, css_parent(pos)); + next = css_next_child(pos, pos->parent); if (next) return css_leftmost_descendant(next); /* no sibling left, visit parent */ - return css_parent(pos); + return pos->parent; +} + +/** + * css_has_online_children - does a css have online children + * @css: the target css + * + * Returns %true if @css has any online children; otherwise, %false. This + * function can be called from any context but the caller is responsible + * for synchronizing against on/offlining as necessary. + */ +bool css_has_online_children(struct cgroup_subsys_state *css) +{ + struct cgroup_subsys_state *child; + bool ret = false; + + rcu_read_lock(); + css_for_each_child(child, css) { + if (css->flags & CSS_ONLINE) { + ret = true; + break; + } + } + rcu_read_unlock(); + return ret; } /** @@ -2783,27 +3345,36 @@ css_next_descendant_post(struct cgroup_subsys_state *pos, */ static void css_advance_task_iter(struct css_task_iter *it) { - struct list_head *l = it->cset_link; + struct list_head *l = it->cset_pos; struct cgrp_cset_link *link; struct css_set *cset; /* Advance to the next non-empty css_set */ do { l = l->next; - if (l == &it->origin_css->cgroup->cset_links) { - it->cset_link = NULL; + if (l == it->cset_head) { + it->cset_pos = NULL; return; } - link = list_entry(l, struct cgrp_cset_link, cset_link); - cset = link->cset; + + if (it->ss) { + cset = container_of(l, struct css_set, + e_cset_node[it->ss->id]); + } else { + link = list_entry(l, struct cgrp_cset_link, cset_link); + cset = link->cset; + } } while (list_empty(&cset->tasks) && list_empty(&cset->mg_tasks)); - it->cset_link = l; + it->cset_pos = l; if (!list_empty(&cset->tasks)) - it->task = cset->tasks.next; + it->task_pos = cset->tasks.next; else - it->task = cset->mg_tasks.next; + it->task_pos = cset->mg_tasks.next; + + it->tasks_head = &cset->tasks; + it->mg_tasks_head = &cset->mg_tasks; } /** @@ -2829,8 +3400,14 @@ void css_task_iter_start(struct cgroup_subsys_state *css, down_read(&css_set_rwsem); - it->origin_css = css; - it->cset_link = &css->cgroup->cset_links; + it->ss = css->ss; + + if (it->ss) + it->cset_pos = &css->cgroup->e_csets[css->ss->id]; + else + it->cset_pos = &css->cgroup->cset_links; + + it->cset_head = it->cset_pos; css_advance_task_iter(it); } @@ -2846,12 +3423,10 @@ void css_task_iter_start(struct cgroup_subsys_state *css, struct task_struct *css_task_iter_next(struct css_task_iter *it) { struct task_struct *res; - struct list_head *l = it->task; - struct cgrp_cset_link *link = list_entry(it->cset_link, - struct cgrp_cset_link, cset_link); + struct list_head *l = it->task_pos; /* If the iterator cg is NULL, we have no tasks */ - if (!it->cset_link) + if (!it->cset_pos) return NULL; res = list_entry(l, struct task_struct, cg_list); @@ -2862,13 +3437,13 @@ struct task_struct *css_task_iter_next(struct css_task_iter *it) */ l = l->next; - if (l == &link->cset->tasks) - l = link->cset->mg_tasks.next; + if (l == it->tasks_head) + l = it->mg_tasks_head->next; - if (l == &link->cset->mg_tasks) + if (l == it->mg_tasks_head) css_advance_task_iter(it); else - it->task = l; + it->task_pos = l; return res; } @@ -2921,7 +3496,7 @@ int cgroup_transfer_tasks(struct cgroup *to, struct cgroup *from) * ->can_attach() fails. */ do { - css_task_iter_start(&from->dummy_css, &it); + css_task_iter_start(&from->self, &it); task = css_task_iter_next(&it); if (task) get_task_struct(task); @@ -3186,7 +3761,7 @@ static int pidlist_array_load(struct cgroup *cgrp, enum cgroup_filetype type, if (!array) return -ENOMEM; /* now, populate the array */ - css_task_iter_start(&cgrp->dummy_css, &it); + css_task_iter_start(&cgrp->self, &it); while ((tsk = css_task_iter_next(&it))) { if (unlikely(n == length)) break; @@ -3248,7 +3823,7 @@ int cgroupstats_build(struct cgroupstats *stats, struct dentry *dentry) /* * We aren't being called from kernfs and there's no guarantee on - * @kn->priv's validity. For this and css_tryget_from_dir(), + * @kn->priv's validity. For this and css_tryget_online_from_dir(), * @kn->priv is RCU safe. Let's do the RCU dancing. */ rcu_read_lock(); @@ -3260,7 +3835,7 @@ int cgroupstats_build(struct cgroupstats *stats, struct dentry *dentry) } rcu_read_unlock(); - css_task_iter_start(&cgrp->dummy_css, &it); + css_task_iter_start(&cgrp->self, &it); while ((tsk = css_task_iter_next(&it))) { switch (tsk->state) { case TASK_RUNNING: @@ -3390,17 +3965,6 @@ static int cgroup_pidlist_show(struct seq_file *s, void *v) return seq_printf(s, "%d\n", *(int *)v); } -/* - * seq_operations functions for iterating on pidlists through seq_file - - * independent of whether it's tasks or procs - */ -static const struct seq_operations cgroup_pidlist_seq_operations = { - .start = cgroup_pidlist_start, - .stop = cgroup_pidlist_stop, - .next = cgroup_pidlist_next, - .show = cgroup_pidlist_show, -}; - static u64 cgroup_read_notify_on_release(struct cgroup_subsys_state *css, struct cftype *cft) { @@ -3442,7 +4006,7 @@ static struct cftype cgroup_base_files[] = { .seq_stop = cgroup_pidlist_stop, .seq_show = cgroup_pidlist_show, .private = CGROUP_FILE_PROCS, - .write_u64 = cgroup_procs_write, + .write = cgroup_procs_write, .mode = S_IRUGO | S_IWUSR, }, { @@ -3456,6 +4020,27 @@ static struct cftype cgroup_base_files[] = { .flags = CFTYPE_ONLY_ON_ROOT, .seq_show = cgroup_sane_behavior_show, }, + { + .name = "cgroup.controllers", + .flags = CFTYPE_ONLY_ON_DFL | CFTYPE_ONLY_ON_ROOT, + .seq_show = cgroup_root_controllers_show, + }, + { + .name = "cgroup.controllers", + .flags = CFTYPE_ONLY_ON_DFL | CFTYPE_NOT_ON_ROOT, + .seq_show = cgroup_controllers_show, + }, + { + .name = "cgroup.subtree_control", + .flags = CFTYPE_ONLY_ON_DFL, + .seq_show = cgroup_subtree_control_show, + .write = cgroup_subtree_control_write, + }, + { + .name = "cgroup.populated", + .flags = CFTYPE_ONLY_ON_DFL | CFTYPE_NOT_ON_ROOT, + .seq_show = cgroup_populated_show, + }, /* * Historical crazy stuff. These don't have "cgroup." prefix and @@ -3470,7 +4055,7 @@ static struct cftype cgroup_base_files[] = { .seq_stop = cgroup_pidlist_stop, .seq_show = cgroup_pidlist_show, .private = CGROUP_FILE_TASKS, - .write_u64 = cgroup_tasks_write, + .write = cgroup_tasks_write, .mode = S_IRUGO | S_IWUSR, }, { @@ -3483,7 +4068,7 @@ static struct cftype cgroup_base_files[] = { .name = "release_agent", .flags = CFTYPE_INSANE | CFTYPE_ONLY_ON_ROOT, .seq_show = cgroup_release_agent_show, - .write_string = cgroup_release_agent_write, + .write = cgroup_release_agent_write, .max_write_len = PATH_MAX - 1, }, { } /* terminate */ @@ -3496,7 +4081,7 @@ static struct cftype cgroup_base_files[] = { * * On failure, no file is added. */ -static int cgroup_populate_dir(struct cgroup *cgrp, unsigned long subsys_mask) +static int cgroup_populate_dir(struct cgroup *cgrp, unsigned int subsys_mask) { struct cgroup_subsys *ss; int i, ret = 0; @@ -3505,7 +4090,7 @@ static int cgroup_populate_dir(struct cgroup *cgrp, unsigned long subsys_mask) for_each_subsys(ss, i) { struct cftype *cfts; - if (!test_bit(i, &subsys_mask)) + if (!(subsys_mask & (1 << i))) continue; list_for_each_entry(cfts, &ss->cfts, node) { @@ -3527,9 +4112,9 @@ err: * Implemented in kill_css(). * * 2. When the percpu_ref is confirmed to be visible as killed on all CPUs - * and thus css_tryget() is guaranteed to fail, the css can be offlined - * by invoking offline_css(). After offlining, the base ref is put. - * Implemented in css_killed_work_fn(). + * and thus css_tryget_online() is guaranteed to fail, the css can be + * offlined by invoking offline_css(). After offlining, the base ref is + * put. Implemented in css_killed_work_fn(). * * 3. When the percpu_ref reaches zero, the only possible remaining * accessors are inside RCU read sections. css_release() schedules the @@ -3548,11 +4133,37 @@ static void css_free_work_fn(struct work_struct *work) container_of(work, struct cgroup_subsys_state, destroy_work); struct cgroup *cgrp = css->cgroup; - if (css->parent) - css_put(css->parent); + if (css->ss) { + /* css free path */ + if (css->parent) + css_put(css->parent); - css->ss->css_free(css); - cgroup_put(cgrp); + css->ss->css_free(css); + cgroup_put(cgrp); + } else { + /* cgroup free path */ + atomic_dec(&cgrp->root->nr_cgrps); + cgroup_pidlist_destroy_all(cgrp); + + if (cgroup_parent(cgrp)) { + /* + * We get a ref to the parent, and put the ref when + * this cgroup is being freed, so it's guaranteed + * that the parent won't be destroyed before its + * children. + */ + cgroup_put(cgroup_parent(cgrp)); + kernfs_put(cgrp->kn); + kfree(cgrp); + } else { + /* + * This is root cgroup's refcnt reaching zero, + * which indicates that the root should be + * released. + */ + cgroup_destroy_root(cgrp->root); + } + } } static void css_free_rcu_fn(struct rcu_head *rcu_head) @@ -3564,26 +4175,59 @@ static void css_free_rcu_fn(struct rcu_head *rcu_head) queue_work(cgroup_destroy_wq, &css->destroy_work); } +static void css_release_work_fn(struct work_struct *work) +{ + struct cgroup_subsys_state *css = + container_of(work, struct cgroup_subsys_state, destroy_work); + struct cgroup_subsys *ss = css->ss; + struct cgroup *cgrp = css->cgroup; + + mutex_lock(&cgroup_mutex); + + css->flags |= CSS_RELEASED; + list_del_rcu(&css->sibling); + + if (ss) { + /* css release path */ + cgroup_idr_remove(&ss->css_idr, css->id); + } else { + /* cgroup release path */ + cgroup_idr_remove(&cgrp->root->cgroup_idr, cgrp->id); + cgrp->id = -1; + } + + mutex_unlock(&cgroup_mutex); + + call_rcu(&css->rcu_head, css_free_rcu_fn); +} + static void css_release(struct percpu_ref *ref) { struct cgroup_subsys_state *css = container_of(ref, struct cgroup_subsys_state, refcnt); - RCU_INIT_POINTER(css->cgroup->subsys[css->ss->id], NULL); - call_rcu(&css->rcu_head, css_free_rcu_fn); + INIT_WORK(&css->destroy_work, css_release_work_fn); + queue_work(cgroup_destroy_wq, &css->destroy_work); } -static void init_css(struct cgroup_subsys_state *css, struct cgroup_subsys *ss, - struct cgroup *cgrp) +static void init_and_link_css(struct cgroup_subsys_state *css, + struct cgroup_subsys *ss, struct cgroup *cgrp) { + lockdep_assert_held(&cgroup_mutex); + + cgroup_get(cgrp); + + memset(css, 0, sizeof(*css)); css->cgroup = cgrp; css->ss = ss; - css->flags = 0; + INIT_LIST_HEAD(&css->sibling); + INIT_LIST_HEAD(&css->children); + css->serial_nr = css_serial_nr_next++; - if (cgrp->parent) - css->parent = cgroup_css(cgrp->parent, ss); - else - css->flags |= CSS_ROOT; + if (cgroup_parent(cgrp)) { + css->parent = cgroup_css(cgroup_parent(cgrp), ss); + css_get(css->parent); + } BUG_ON(cgroup_css(cgrp, ss)); } @@ -3594,14 +4238,12 @@ static int online_css(struct cgroup_subsys_state *css) struct cgroup_subsys *ss = css->ss; int ret = 0; - lockdep_assert_held(&cgroup_tree_mutex); lockdep_assert_held(&cgroup_mutex); if (ss->css_online) ret = ss->css_online(css); if (!ret) { css->flags |= CSS_ONLINE; - css->cgroup->nr_css++; rcu_assign_pointer(css->cgroup->subsys[ss->id], css); } return ret; @@ -3612,7 +4254,6 @@ static void offline_css(struct cgroup_subsys_state *css) { struct cgroup_subsys *ss = css->ss; - lockdep_assert_held(&cgroup_tree_mutex); lockdep_assert_held(&cgroup_mutex); if (!(css->flags & CSS_ONLINE)) @@ -3622,8 +4263,9 @@ static void offline_css(struct cgroup_subsys_state *css) ss->css_offline(css); css->flags &= ~CSS_ONLINE; - css->cgroup->nr_css--; - RCU_INIT_POINTER(css->cgroup->subsys[ss->id], css); + RCU_INIT_POINTER(css->cgroup->subsys[ss->id], NULL); + + wake_up_all(&css->cgroup->offline_waitq); } /** @@ -3637,111 +4279,102 @@ static void offline_css(struct cgroup_subsys_state *css) */ static int create_css(struct cgroup *cgrp, struct cgroup_subsys *ss) { - struct cgroup *parent = cgrp->parent; + struct cgroup *parent = cgroup_parent(cgrp); + struct cgroup_subsys_state *parent_css = cgroup_css(parent, ss); struct cgroup_subsys_state *css; int err; lockdep_assert_held(&cgroup_mutex); - css = ss->css_alloc(cgroup_css(parent, ss)); + css = ss->css_alloc(parent_css); if (IS_ERR(css)) return PTR_ERR(css); + init_and_link_css(css, ss, cgrp); + err = percpu_ref_init(&css->refcnt, css_release); if (err) goto err_free_css; - init_css(css, ss, cgrp); + err = cgroup_idr_alloc(&ss->css_idr, NULL, 2, 0, GFP_NOWAIT); + if (err < 0) + goto err_free_percpu_ref; + css->id = err; err = cgroup_populate_dir(cgrp, 1 << ss->id); if (err) - goto err_free_percpu_ref; + goto err_free_id; + + /* @css is ready to be brought online now, make it visible */ + list_add_tail_rcu(&css->sibling, &parent_css->children); + cgroup_idr_replace(&ss->css_idr, css, css->id); err = online_css(css); if (err) - goto err_clear_dir; - - cgroup_get(cgrp); - css_get(css->parent); - - cgrp->subsys_mask |= 1 << ss->id; + goto err_list_del; if (ss->broken_hierarchy && !ss->warned_broken_hierarchy && - parent->parent) { - pr_warning("cgroup: %s (%d) created nested cgroup for controller \"%s\" which has incomplete hierarchy support. Nested cgroups may change behavior in the future.\n", - current->comm, current->pid, ss->name); + cgroup_parent(parent)) { + pr_warn("%s (%d) created nested cgroup for controller \"%s\" which has incomplete hierarchy support. Nested cgroups may change behavior in the future.\n", + current->comm, current->pid, ss->name); if (!strcmp(ss->name, "memory")) - pr_warning("cgroup: \"memory\" requires setting use_hierarchy to 1 on the root.\n"); + pr_warn("\"memory\" requires setting use_hierarchy to 1 on the root\n"); ss->warned_broken_hierarchy = true; } return 0; -err_clear_dir: +err_list_del: + list_del_rcu(&css->sibling); cgroup_clear_dir(css->cgroup, 1 << css->ss->id); +err_free_id: + cgroup_idr_remove(&ss->css_idr, css->id); err_free_percpu_ref: percpu_ref_cancel_init(&css->refcnt); err_free_css: - ss->css_free(css); + call_rcu(&css->rcu_head, css_free_rcu_fn); return err; } -/** - * cgroup_create - create a cgroup - * @parent: cgroup that will be parent of the new cgroup - * @name: name of the new cgroup - * @mode: mode to set on new cgroup - */ -static long cgroup_create(struct cgroup *parent, const char *name, - umode_t mode) +static int cgroup_mkdir(struct kernfs_node *parent_kn, const char *name, + umode_t mode) { - struct cgroup *cgrp; - struct cgroup_root *root = parent->root; - int ssid, err; + struct cgroup *parent, *cgrp; + struct cgroup_root *root; struct cgroup_subsys *ss; struct kernfs_node *kn; + int ssid, ret; - /* - * XXX: The default hierarchy isn't fully implemented yet. Block - * !root cgroup creation on it for now. - */ - if (root == &cgrp_dfl_root) - return -EINVAL; + parent = cgroup_kn_lock_live(parent_kn); + if (!parent) + return -ENODEV; + root = parent->root; /* allocate the cgroup and its ID, 0 is reserved for the root */ cgrp = kzalloc(sizeof(*cgrp), GFP_KERNEL); - if (!cgrp) - return -ENOMEM; - - mutex_lock(&cgroup_tree_mutex); - - /* - * Only live parents can have children. Note that the liveliness - * check isn't strictly necessary because cgroup_mkdir() and - * cgroup_rmdir() are fully synchronized by i_mutex; however, do it - * anyway so that locking is contained inside cgroup proper and we - * don't get nasty surprises if we ever grow another caller. - */ - if (!cgroup_lock_live_group(parent)) { - err = -ENODEV; - goto err_unlock_tree; + if (!cgrp) { + ret = -ENOMEM; + goto out_unlock; } + ret = percpu_ref_init(&cgrp->self.refcnt, css_release); + if (ret) + goto out_free_cgrp; + /* * Temporarily set the pointer to NULL, so idr_find() won't return * a half-baked cgroup. */ - cgrp->id = idr_alloc(&root->cgroup_idr, NULL, 1, 0, GFP_KERNEL); + cgrp->id = cgroup_idr_alloc(&root->cgroup_idr, NULL, 2, 0, GFP_NOWAIT); if (cgrp->id < 0) { - err = -ENOMEM; - goto err_unlock; + ret = -ENOMEM; + goto out_cancel_ref; } init_cgroup_housekeeping(cgrp); - cgrp->parent = parent; - cgrp->dummy_css.parent = &parent->dummy_css; - cgrp->root = parent->root; + cgrp->self.parent = &parent->self; + cgrp->root = root; if (notify_on_release(parent)) set_bit(CGRP_NOTIFY_ON_RELEASE, &cgrp->flags); @@ -3752,8 +4385,8 @@ static long cgroup_create(struct cgroup *parent, const char *name, /* create the directory */ kn = kernfs_create_dir(parent->kn, name, mode, cgrp); if (IS_ERR(kn)) { - err = PTR_ERR(kn); - goto err_free_id; + ret = PTR_ERR(kn); + goto out_free_id; } cgrp->kn = kn; @@ -3763,10 +4396,10 @@ static long cgroup_create(struct cgroup *parent, const char *name, */ kernfs_get(kn); - cgrp->serial_nr = cgroup_serial_nr_next++; + cgrp->self.serial_nr = css_serial_nr_next++; /* allocation complete, commit to creation */ - list_add_tail_rcu(&cgrp->sibling, &cgrp->parent->children); + list_add_tail_rcu(&cgrp->self.sibling, &cgroup_parent(cgrp)->self.children); atomic_inc(&root->nr_cgrps); cgroup_get(parent); @@ -3774,107 +4407,66 @@ static long cgroup_create(struct cgroup *parent, const char *name, * @cgrp is now fully operational. If something fails after this * point, it'll be released via the normal destruction path. */ - idr_replace(&root->cgroup_idr, cgrp, cgrp->id); + cgroup_idr_replace(&root->cgroup_idr, cgrp, cgrp->id); - err = cgroup_kn_set_ugid(kn); - if (err) - goto err_destroy; + ret = cgroup_kn_set_ugid(kn); + if (ret) + goto out_destroy; - err = cgroup_addrm_files(cgrp, cgroup_base_files, true); - if (err) - goto err_destroy; + ret = cgroup_addrm_files(cgrp, cgroup_base_files, true); + if (ret) + goto out_destroy; /* let's create and online css's */ for_each_subsys(ss, ssid) { - if (root->cgrp.subsys_mask & (1 << ssid)) { - err = create_css(cgrp, ss); - if (err) - goto err_destroy; + if (parent->child_subsys_mask & (1 << ssid)) { + ret = create_css(cgrp, ss); + if (ret) + goto out_destroy; } } - kernfs_activate(kn); + /* + * On the default hierarchy, a child doesn't automatically inherit + * child_subsys_mask from the parent. Each is configured manually. + */ + if (!cgroup_on_dfl(cgrp)) + cgrp->child_subsys_mask = parent->child_subsys_mask; - mutex_unlock(&cgroup_mutex); - mutex_unlock(&cgroup_tree_mutex); + kernfs_activate(kn); - return 0; + ret = 0; + goto out_unlock; -err_free_id: - idr_remove(&root->cgroup_idr, cgrp->id); -err_unlock: - mutex_unlock(&cgroup_mutex); -err_unlock_tree: - mutex_unlock(&cgroup_tree_mutex); +out_free_id: + cgroup_idr_remove(&root->cgroup_idr, cgrp->id); +out_cancel_ref: + percpu_ref_cancel_init(&cgrp->self.refcnt); +out_free_cgrp: kfree(cgrp); - return err; +out_unlock: + cgroup_kn_unlock(parent_kn); + return ret; -err_destroy: +out_destroy: cgroup_destroy_locked(cgrp); - mutex_unlock(&cgroup_mutex); - mutex_unlock(&cgroup_tree_mutex); - return err; -} - -static int cgroup_mkdir(struct kernfs_node *parent_kn, const char *name, - umode_t mode) -{ - struct cgroup *parent = parent_kn->priv; - int ret; - - /* - * cgroup_create() grabs cgroup_tree_mutex which nests outside - * kernfs active_ref and cgroup_create() already synchronizes - * properly against removal through cgroup_lock_live_group(). - * Break it before calling cgroup_create(). - */ - cgroup_get(parent); - kernfs_break_active_protection(parent_kn); - - ret = cgroup_create(parent, name, mode); - - kernfs_unbreak_active_protection(parent_kn); - cgroup_put(parent); - return ret; + goto out_unlock; } /* * This is called when the refcnt of a css is confirmed to be killed. - * css_tryget() is now guaranteed to fail. + * css_tryget_online() is now guaranteed to fail. Tell the subsystem to + * initate destruction and put the css ref from kill_css(). */ static void css_killed_work_fn(struct work_struct *work) { struct cgroup_subsys_state *css = container_of(work, struct cgroup_subsys_state, destroy_work); - struct cgroup *cgrp = css->cgroup; - mutex_lock(&cgroup_tree_mutex); mutex_lock(&cgroup_mutex); - - /* - * css_tryget() is guaranteed to fail now. Tell subsystems to - * initate destruction. - */ offline_css(css); - - /* - * If @cgrp is marked dead, it's waiting for refs of all css's to - * be disabled before proceeding to the second phase of cgroup - * destruction. If we are the last one, kick it off. - */ - if (!cgrp->nr_css && cgroup_is_dead(cgrp)) - cgroup_destroy_css_killed(cgrp); - mutex_unlock(&cgroup_mutex); - mutex_unlock(&cgroup_tree_mutex); - /* - * Put the css refs from kill_css(). Each css holds an extra - * reference to the cgroup's dentry and cgroup removal proceeds - * regardless of css refs. On the last put of each css, whenever - * that may be, the extra dentry ref is put so that dentry - * destruction happens only after all css's are released. - */ css_put(css); } @@ -3888,9 +4480,18 @@ static void css_killed_ref_fn(struct percpu_ref *ref) queue_work(cgroup_destroy_wq, &css->destroy_work); } -static void __kill_css(struct cgroup_subsys_state *css) +/** + * kill_css - destroy a css + * @css: css to destroy + * + * This function initiates destruction of @css by removing cgroup interface + * files and putting its base reference. ->css_offline() will be invoked + * asynchronously once css_tryget_online() is guaranteed to fail and when + * the reference count reaches zero, @css will be released. + */ +static void kill_css(struct cgroup_subsys_state *css) { - lockdep_assert_held(&cgroup_tree_mutex); + lockdep_assert_held(&cgroup_mutex); /* * This must happen before css is disassociated with its cgroup. @@ -3907,7 +4508,7 @@ static void __kill_css(struct cgroup_subsys_state *css) /* * cgroup core guarantees that, by the time ->css_offline() is * invoked, no new css reference will be given out via - * css_tryget(). We can't simply call percpu_ref_kill() and + * css_tryget_online(). We can't simply call percpu_ref_kill() and * proceed to offlining css's because percpu_ref_kill() doesn't * guarantee that the ref is seen as killed on all CPUs on return. * @@ -3918,36 +4519,14 @@ static void __kill_css(struct cgroup_subsys_state *css) } /** - * kill_css - destroy a css - * @css: css to destroy - * - * This function initiates destruction of @css by removing cgroup interface - * files and putting its base reference. ->css_offline() will be invoked - * asynchronously once css_tryget() is guaranteed to fail and when the - * reference count reaches zero, @css will be released. - */ -static void kill_css(struct cgroup_subsys_state *css) -{ - struct cgroup *cgrp = css->cgroup; - - lockdep_assert_held(&cgroup_tree_mutex); - - /* if already killed, noop */ - if (cgrp->subsys_mask & (1 << css->ss->id)) { - cgrp->subsys_mask &= ~(1 << css->ss->id); - __kill_css(css); - } -} - -/** * cgroup_destroy_locked - the first stage of cgroup destruction * @cgrp: cgroup to be destroyed * * css's make use of percpu refcnts whose killing latency shouldn't be * exposed to userland and are RCU protected. Also, cgroup core needs to - * guarantee that css_tryget() won't succeed by the time ->css_offline() is - * invoked. To satisfy all the requirements, destruction is implemented in - * the following two steps. + * guarantee that css_tryget_online() won't succeed by the time + * ->css_offline() is invoked. To satisfy all the requirements, + * destruction is implemented in the following two steps. * * s1. Verify @cgrp can be destroyed and mark it dying. Remove all * userland visible parts and start killing the percpu refcnts of @@ -3966,12 +4545,10 @@ static void kill_css(struct cgroup_subsys_state *css) static int cgroup_destroy_locked(struct cgroup *cgrp) __releases(&cgroup_mutex) __acquires(&cgroup_mutex) { - struct cgroup *child; struct cgroup_subsys_state *css; bool empty; int ssid; - lockdep_assert_held(&cgroup_tree_mutex); lockdep_assert_held(&cgroup_mutex); /* @@ -3985,127 +4562,68 @@ static int cgroup_destroy_locked(struct cgroup *cgrp) return -EBUSY; /* - * Make sure there's no live children. We can't test ->children - * emptiness as dead children linger on it while being destroyed; - * otherwise, "rmdir parent/child parent" may fail with -EBUSY. + * Make sure there's no live children. We can't test emptiness of + * ->self.children as dead children linger on it while being + * drained; otherwise, "rmdir parent/child parent" may fail. */ - empty = true; - rcu_read_lock(); - list_for_each_entry_rcu(child, &cgrp->children, sibling) { - empty = cgroup_is_dead(child); - if (!empty) - break; - } - rcu_read_unlock(); - if (!empty) + if (css_has_online_children(&cgrp->self)) return -EBUSY; /* * Mark @cgrp dead. This prevents further task migration and child - * creation by disabling cgroup_lock_live_group(). Note that - * CGRP_DEAD assertion is depended upon by css_next_child() to - * resume iteration after dropping RCU read lock. See - * css_next_child() for details. + * creation by disabling cgroup_lock_live_group(). */ - set_bit(CGRP_DEAD, &cgrp->flags); + cgrp->self.flags &= ~CSS_ONLINE; - /* - * Initiate massacre of all css's. cgroup_destroy_css_killed() - * will be invoked to perform the rest of destruction once the - * percpu refs of all css's are confirmed to be killed. This - * involves removing the subsystem's files, drop cgroup_mutex. - */ - mutex_unlock(&cgroup_mutex); + /* initiate massacre of all css's */ for_each_css(css, ssid, cgrp) kill_css(css); - mutex_lock(&cgroup_mutex); - /* CGRP_DEAD is set, remove from ->release_list for the last time */ + /* CSS_ONLINE is clear, remove from ->release_list for the last time */ raw_spin_lock(&release_list_lock); if (!list_empty(&cgrp->release_list)) list_del_init(&cgrp->release_list); raw_spin_unlock(&release_list_lock); /* - * If @cgrp has css's attached, the second stage of cgroup - * destruction is kicked off from css_killed_work_fn() after the - * refs of all attached css's are killed. If @cgrp doesn't have - * any css, we kick it off here. + * Remove @cgrp directory along with the base files. @cgrp has an + * extra ref on its kn. */ - if (!cgrp->nr_css) - cgroup_destroy_css_killed(cgrp); - - /* remove @cgrp directory along with the base files */ - mutex_unlock(&cgroup_mutex); + kernfs_remove(cgrp->kn); - /* - * There are two control paths which try to determine cgroup from - * dentry without going through kernfs - cgroupstats_build() and - * css_tryget_from_dir(). Those are supported by RCU protecting - * clearing of cgrp->kn->priv backpointer, which should happen - * after all files under it have been removed. - */ - kernfs_remove(cgrp->kn); /* @cgrp has an extra ref on its kn */ - RCU_INIT_POINTER(*(void __rcu __force **)&cgrp->kn->priv, NULL); + set_bit(CGRP_RELEASABLE, &cgroup_parent(cgrp)->flags); + check_for_release(cgroup_parent(cgrp)); - mutex_lock(&cgroup_mutex); + /* put the base reference */ + percpu_ref_kill(&cgrp->self.refcnt); return 0; }; -/** - * cgroup_destroy_css_killed - the second step of cgroup destruction - * @work: cgroup->destroy_free_work - * - * This function is invoked from a work item for a cgroup which is being - * destroyed after all css's are offlined and performs the rest of - * destruction. This is the second step of destruction described in the - * comment above cgroup_destroy_locked(). - */ -static void cgroup_destroy_css_killed(struct cgroup *cgrp) -{ - struct cgroup *parent = cgrp->parent; - - lockdep_assert_held(&cgroup_tree_mutex); - lockdep_assert_held(&cgroup_mutex); - - /* delete this cgroup from parent->children */ - list_del_rcu(&cgrp->sibling); - - cgroup_put(cgrp); - - set_bit(CGRP_RELEASABLE, &parent->flags); - check_for_release(parent); -} - static int cgroup_rmdir(struct kernfs_node *kn) { - struct cgroup *cgrp = kn->priv; + struct cgroup *cgrp; int ret = 0; - /* - * This is self-destruction but @kn can't be removed while this - * callback is in progress. Let's break active protection. Once - * the protection is broken, @cgrp can be destroyed at any point. - * Pin it so that it stays accessible. - */ - cgroup_get(cgrp); - kernfs_break_active_protection(kn); + cgrp = cgroup_kn_lock_live(kn); + if (!cgrp) + return 0; + cgroup_get(cgrp); /* for @kn->priv clearing */ - mutex_lock(&cgroup_tree_mutex); - mutex_lock(&cgroup_mutex); + ret = cgroup_destroy_locked(cgrp); + + cgroup_kn_unlock(kn); /* - * @cgrp might already have been destroyed while we're trying to - * grab the mutexes. + * There are two control paths which try to determine cgroup from + * dentry without going through kernfs - cgroupstats_build() and + * css_tryget_online_from_dir(). Those are supported by RCU + * protecting clearing of cgrp->kn->priv backpointer, which should + * happen after all files under it have been removed. */ - if (!cgroup_is_dead(cgrp)) - ret = cgroup_destroy_locked(cgrp); - - mutex_unlock(&cgroup_mutex); - mutex_unlock(&cgroup_tree_mutex); + if (!ret) + RCU_INIT_POINTER(*(void __rcu __force **)&kn->priv, NULL); - kernfs_unbreak_active_protection(kn); cgroup_put(cgrp); return ret; } @@ -4118,15 +4636,15 @@ static struct kernfs_syscall_ops cgroup_kf_syscall_ops = { .rename = cgroup_rename, }; -static void __init cgroup_init_subsys(struct cgroup_subsys *ss) +static void __init cgroup_init_subsys(struct cgroup_subsys *ss, bool early) { struct cgroup_subsys_state *css; printk(KERN_INFO "Initializing cgroup subsys %s\n", ss->name); - mutex_lock(&cgroup_tree_mutex); mutex_lock(&cgroup_mutex); + idr_init(&ss->css_idr); INIT_LIST_HEAD(&ss->cfts); /* Create the root cgroup state for this subsystem */ @@ -4134,7 +4652,21 @@ static void __init cgroup_init_subsys(struct cgroup_subsys *ss) css = ss->css_alloc(cgroup_css(&cgrp_dfl_root.cgrp, ss)); /* We don't handle early failures gracefully */ BUG_ON(IS_ERR(css)); - init_css(css, ss, &cgrp_dfl_root.cgrp); + init_and_link_css(css, ss, &cgrp_dfl_root.cgrp); + + /* + * Root csses are never destroyed and we can't initialize + * percpu_ref during early init. Disable refcnting. + */ + css->flags |= CSS_NO_REF; + + if (early) { + /* allocation can't be done safely during early init */ + css->id = 1; + } else { + css->id = cgroup_idr_alloc(&ss->css_idr, css, 1, 2, GFP_KERNEL); + BUG_ON(css->id < 0); + } /* Update the init_css_set to contain a subsys * pointer to this state - since the subsystem is @@ -4151,10 +4683,7 @@ static void __init cgroup_init_subsys(struct cgroup_subsys *ss) BUG_ON(online_css(css)); - cgrp_dfl_root.cgrp.subsys_mask |= 1 << ss->id; - mutex_unlock(&cgroup_mutex); - mutex_unlock(&cgroup_tree_mutex); } /** @@ -4171,6 +4700,8 @@ int __init cgroup_init_early(void) int i; init_cgroup_root(&cgrp_dfl_root, &opts); + cgrp_dfl_root.cgrp.self.flags |= CSS_NO_REF; + RCU_INIT_POINTER(init_task.cgroups, &init_css_set); for_each_subsys(ss, i) { @@ -4185,7 +4716,7 @@ int __init cgroup_init_early(void) ss->name = cgroup_subsys_name[i]; if (ss->early_init) - cgroup_init_subsys(ss); + cgroup_init_subsys(ss, true); } return 0; } @@ -4204,7 +4735,6 @@ int __init cgroup_init(void) BUG_ON(cgroup_init_cftypes(NULL, cgroup_base_files)); - mutex_lock(&cgroup_tree_mutex); mutex_lock(&cgroup_mutex); /* Add init_css_set to the hash table */ @@ -4214,18 +4744,31 @@ int __init cgroup_init(void) BUG_ON(cgroup_setup_root(&cgrp_dfl_root, 0)); mutex_unlock(&cgroup_mutex); - mutex_unlock(&cgroup_tree_mutex); for_each_subsys(ss, ssid) { - if (!ss->early_init) - cgroup_init_subsys(ss); + if (ss->early_init) { + struct cgroup_subsys_state *css = + init_css_set.subsys[ss->id]; + + css->id = cgroup_idr_alloc(&ss->css_idr, css, 1, 2, + GFP_KERNEL); + BUG_ON(css->id < 0); + } else { + cgroup_init_subsys(ss, false); + } + + list_add_tail(&init_css_set.e_cset_node[ssid], + &cgrp_dfl_root.cgrp.e_csets[ssid]); /* - * cftype registration needs kmalloc and can't be done - * during early_init. Register base cftypes separately. + * Setting dfl_root subsys_mask needs to consider the + * disabled flag and cftype registration needs kmalloc, + * both of which aren't available during early_init. */ - if (ss->base_cftypes) + if (!ss->disabled) { + cgrp_dfl_root.subsys_mask |= 1 << ss->id; WARN_ON(cgroup_add_cftypes(ss, ss->base_cftypes)); + } } cgroup_kobj = kobject_create_and_add("cgroup", fs_kobj); @@ -4308,7 +4851,7 @@ int proc_cgroup_show(struct seq_file *m, void *v) seq_printf(m, "%d:", root->hierarchy_id); for_each_subsys(ss, ssid) - if (root->cgrp.subsys_mask & (1 << ssid)) + if (root->subsys_mask & (1 << ssid)) seq_printf(m, "%s%s", count++ ? "," : "", ss->name); if (strlen(root->name)) seq_printf(m, "%sname=%s", count ? "," : "", @@ -4503,8 +5046,8 @@ void cgroup_exit(struct task_struct *tsk) static void check_for_release(struct cgroup *cgrp) { - if (cgroup_is_releasable(cgrp) && - list_empty(&cgrp->cset_links) && list_empty(&cgrp->children)) { + if (cgroup_is_releasable(cgrp) && list_empty(&cgrp->cset_links) && + !css_has_online_children(&cgrp->self)) { /* * Control Group is currently removeable. If it's not * already queued for a userspace notification, queue @@ -4621,7 +5164,7 @@ static int __init cgroup_disable(char *str) __setup("cgroup_disable=", cgroup_disable); /** - * css_tryget_from_dir - get corresponding css from the dentry of a cgroup dir + * css_tryget_online_from_dir - get corresponding css from a cgroup dentry * @dentry: directory dentry of interest * @ss: subsystem of interest * @@ -4629,8 +5172,8 @@ __setup("cgroup_disable=", cgroup_disable); * to get the corresponding css and return it. If such css doesn't exist * or can't be pinned, an ERR_PTR value is returned. */ -struct cgroup_subsys_state *css_tryget_from_dir(struct dentry *dentry, - struct cgroup_subsys *ss) +struct cgroup_subsys_state *css_tryget_online_from_dir(struct dentry *dentry, + struct cgroup_subsys *ss) { struct kernfs_node *kn = kernfs_node_from_dentry(dentry); struct cgroup_subsys_state *css = NULL; @@ -4646,13 +5189,13 @@ struct cgroup_subsys_state *css_tryget_from_dir(struct dentry *dentry, /* * This path doesn't originate from kernfs and @kn could already * have been or be removed at any point. @kn->priv is RCU - * protected for this access. See destroy_locked() for details. + * protected for this access. See cgroup_rmdir() for details. */ cgrp = rcu_dereference(kn->priv); if (cgrp) css = cgroup_css(cgrp, ss); - if (!css || !css_tryget(css)) + if (!css || !css_tryget_online(css)) css = ERR_PTR(-ENOENT); rcu_read_unlock(); @@ -4669,14 +5212,8 @@ struct cgroup_subsys_state *css_tryget_from_dir(struct dentry *dentry, */ struct cgroup_subsys_state *css_from_id(int id, struct cgroup_subsys *ss) { - struct cgroup *cgrp; - - cgroup_assert_mutexes_or_rcu_locked(); - - cgrp = idr_find(&ss->root->cgroup_idr, id); - if (cgrp) - return cgroup_css(cgrp, ss); - return NULL; + WARN_ON_ONCE(!rcu_read_lock_held()); + return idr_find(&ss->css_idr, id); } #ifdef CONFIG_CGROUP_DEBUG diff --git a/kernel/cgroup_freezer.c b/kernel/cgroup_freezer.c index 345628c78b5b..a79e40f9d700 100644 --- a/kernel/cgroup_freezer.c +++ b/kernel/cgroup_freezer.c @@ -59,7 +59,7 @@ static inline struct freezer *task_freezer(struct task_struct *task) static struct freezer *parent_freezer(struct freezer *freezer) { - return css_freezer(css_parent(&freezer->css)); + return css_freezer(freezer->css.parent); } bool cgroup_freezing(struct task_struct *task) @@ -73,10 +73,6 @@ bool cgroup_freezing(struct task_struct *task) return ret; } -/* - * cgroups_write_string() limits the size of freezer state strings to - * CGROUP_LOCAL_BUFFER_SIZE - */ static const char *freezer_state_strs(unsigned int state) { if (state & CGROUP_FROZEN) @@ -304,7 +300,7 @@ static int freezer_read(struct seq_file *m, void *v) /* update states bottom-up */ css_for_each_descendant_post(pos, css) { - if (!css_tryget(pos)) + if (!css_tryget_online(pos)) continue; rcu_read_unlock(); @@ -404,7 +400,7 @@ static void freezer_change_state(struct freezer *freezer, bool freeze) struct freezer *pos_f = css_freezer(pos); struct freezer *parent = parent_freezer(pos_f); - if (!css_tryget(pos)) + if (!css_tryget_online(pos)) continue; rcu_read_unlock(); @@ -423,20 +419,22 @@ static void freezer_change_state(struct freezer *freezer, bool freeze) mutex_unlock(&freezer_mutex); } -static int freezer_write(struct cgroup_subsys_state *css, struct cftype *cft, - char *buffer) +static ssize_t freezer_write(struct kernfs_open_file *of, + char *buf, size_t nbytes, loff_t off) { bool freeze; - if (strcmp(buffer, freezer_state_strs(0)) == 0) + buf = strstrip(buf); + + if (strcmp(buf, freezer_state_strs(0)) == 0) freeze = false; - else if (strcmp(buffer, freezer_state_strs(CGROUP_FROZEN)) == 0) + else if (strcmp(buf, freezer_state_strs(CGROUP_FROZEN)) == 0) freeze = true; else return -EINVAL; - freezer_change_state(css_freezer(css), freeze); - return 0; + freezer_change_state(css_freezer(of_css(of)), freeze); + return nbytes; } static u64 freezer_self_freezing_read(struct cgroup_subsys_state *css, @@ -460,7 +458,7 @@ static struct cftype files[] = { .name = "state", .flags = CFTYPE_NOT_ON_ROOT, .seq_show = freezer_read, - .write_string = freezer_write, + .write = freezer_write, }, { .name = "self_freezing", diff --git a/kernel/cpuset.c b/kernel/cpuset.c index 130017843899..f6b33c696224 100644 --- a/kernel/cpuset.c +++ b/kernel/cpuset.c @@ -119,7 +119,7 @@ static inline struct cpuset *task_cs(struct task_struct *task) static inline struct cpuset *parent_cs(struct cpuset *cs) { - return css_cs(css_parent(&cs->css)); + return css_cs(cs->css.parent); } #ifdef CONFIG_NUMA @@ -691,11 +691,8 @@ restart: if (nslot == ndoms) { static int warnings = 10; if (warnings) { - printk(KERN_WARNING - "rebuild_sched_domains confused:" - " nslot %d, ndoms %d, csn %d, i %d," - " apn %d\n", - nslot, ndoms, csn, i, apn); + pr_warn("rebuild_sched_domains confused: nslot %d, ndoms %d, csn %d, i %d, apn %d\n", + nslot, ndoms, csn, i, apn); warnings--; } continue; @@ -870,7 +867,7 @@ static void update_tasks_cpumask_hier(struct cpuset *root_cs, bool update_root) continue; } } - if (!css_tryget(&cp->css)) + if (!css_tryget_online(&cp->css)) continue; rcu_read_unlock(); @@ -885,6 +882,7 @@ static void update_tasks_cpumask_hier(struct cpuset *root_cs, bool update_root) /** * update_cpumask - update the cpus_allowed mask of a cpuset and all tasks in it * @cs: the cpuset to consider + * @trialcs: trial cpuset * @buf: buffer of cpu numbers written to this cpuset */ static int update_cpumask(struct cpuset *cs, struct cpuset *trialcs, @@ -1105,7 +1103,7 @@ static void update_tasks_nodemask_hier(struct cpuset *root_cs, bool update_root) continue; } } - if (!css_tryget(&cp->css)) + if (!css_tryget_online(&cp->css)) continue; rcu_read_unlock(); @@ -1600,13 +1598,15 @@ out_unlock: /* * Common handling for a write to a "cpus" or "mems" file. */ -static int cpuset_write_resmask(struct cgroup_subsys_state *css, - struct cftype *cft, char *buf) +static ssize_t cpuset_write_resmask(struct kernfs_open_file *of, + char *buf, size_t nbytes, loff_t off) { - struct cpuset *cs = css_cs(css); + struct cpuset *cs = css_cs(of_css(of)); struct cpuset *trialcs; int retval = -ENODEV; + buf = strstrip(buf); + /* * CPU or memory hotunplug may leave @cs w/o any execution * resources, in which case the hotplug code asynchronously updates @@ -1630,7 +1630,7 @@ static int cpuset_write_resmask(struct cgroup_subsys_state *css, goto out_unlock; } - switch (cft->private) { + switch (of_cft(of)->private) { case FILE_CPULIST: retval = update_cpumask(cs, trialcs, buf); break; @@ -1645,7 +1645,7 @@ static int cpuset_write_resmask(struct cgroup_subsys_state *css, free_trial_cpuset(trialcs); out_unlock: mutex_unlock(&cpuset_mutex); - return retval; + return retval ?: nbytes; } /* @@ -1747,7 +1747,7 @@ static struct cftype files[] = { { .name = "cpus", .seq_show = cpuset_common_seq_show, - .write_string = cpuset_write_resmask, + .write = cpuset_write_resmask, .max_write_len = (100U + 6 * NR_CPUS), .private = FILE_CPULIST, }, @@ -1755,7 +1755,7 @@ static struct cftype files[] = { { .name = "mems", .seq_show = cpuset_common_seq_show, - .write_string = cpuset_write_resmask, + .write = cpuset_write_resmask, .max_write_len = (100U + 6 * MAX_NUMNODES), .private = FILE_MEMLIST, }, @@ -2011,7 +2011,7 @@ static void remove_tasks_in_empty_cpuset(struct cpuset *cs) parent = parent_cs(parent); if (cgroup_transfer_tasks(parent->css.cgroup, cs->css.cgroup)) { - printk(KERN_ERR "cpuset: failed to transfer tasks out of empty cpuset "); + pr_err("cpuset: failed to transfer tasks out of empty cpuset "); pr_cont_cgroup_name(cs->css.cgroup); pr_cont("\n"); } @@ -2149,7 +2149,7 @@ static void cpuset_hotplug_workfn(struct work_struct *work) rcu_read_lock(); cpuset_for_each_descendant_pre(cs, pos_css, &top_cpuset) { - if (cs == &top_cpuset || !css_tryget(&cs->css)) + if (cs == &top_cpuset || !css_tryget_online(&cs->css)) continue; rcu_read_unlock(); @@ -2530,7 +2530,7 @@ int cpuset_mems_allowed_intersects(const struct task_struct *tsk1, /** * cpuset_print_task_mems_allowed - prints task's cpuset and mems_allowed - * @task: pointer to task_struct of some task. + * @tsk: pointer to task_struct of some task. * * Description: Prints @task's name, cpuset name, and cached copy of its * mems_allowed to the kernel log. @@ -2548,7 +2548,7 @@ void cpuset_print_task_mems_allowed(struct task_struct *tsk) cgrp = task_cs(tsk)->css.cgroup; nodelist_scnprintf(cpuset_nodelist, CPUSET_NODELIST_LEN, tsk->mems_allowed); - printk(KERN_INFO "%s cpuset=", tsk->comm); + pr_info("%s cpuset=", tsk->comm); pr_cont_cgroup_name(cgrp); pr_cont(" mems_allowed=%s\n", cpuset_nodelist); @@ -2640,10 +2640,10 @@ out: /* Display task mems_allowed in /proc/<pid>/status file. */ void cpuset_task_status_allowed(struct seq_file *m, struct task_struct *task) { - seq_printf(m, "Mems_allowed:\t"); + seq_puts(m, "Mems_allowed:\t"); seq_nodemask(m, &task->mems_allowed); - seq_printf(m, "\n"); - seq_printf(m, "Mems_allowed_list:\t"); + seq_puts(m, "\n"); + seq_puts(m, "Mems_allowed_list:\t"); seq_nodemask_list(m, &task->mems_allowed); - seq_printf(m, "\n"); + seq_puts(m, "\n"); } diff --git a/kernel/events/core.c b/kernel/events/core.c index 689237a0c5e8..24d35cc38e42 100644 --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -608,7 +608,8 @@ static inline int perf_cgroup_connect(int fd, struct perf_event *event, if (!f.file) return -EBADF; - css = css_tryget_from_dir(f.file->f_dentry, &perf_event_cgrp_subsys); + css = css_tryget_online_from_dir(f.file->f_dentry, + &perf_event_cgrp_subsys); if (IS_ERR(css)) { ret = PTR_ERR(css); goto out; diff --git a/kernel/gcov/base.c b/kernel/gcov/base.c index f45b75b713c0..b358a802fd18 100644 --- a/kernel/gcov/base.c +++ b/kernel/gcov/base.c @@ -85,6 +85,12 @@ void __gcov_merge_ior(gcov_type *counters, unsigned int n_counters) } EXPORT_SYMBOL(__gcov_merge_ior); +void __gcov_merge_time_profile(gcov_type *counters, unsigned int n_counters) +{ + /* Unused. */ +} +EXPORT_SYMBOL(__gcov_merge_time_profile); + /** * gcov_enable_events - enable event reporting through gcov_event() * diff --git a/kernel/gcov/gcc_4_7.c b/kernel/gcov/gcc_4_7.c index 2c6e4631c814..826ba9fb5e32 100644 --- a/kernel/gcov/gcc_4_7.c +++ b/kernel/gcov/gcc_4_7.c @@ -18,7 +18,12 @@ #include <linux/vmalloc.h> #include "gcov.h" +#if __GNUC__ == 4 && __GNUC_MINOR__ >= 9 +#define GCOV_COUNTERS 9 +#else #define GCOV_COUNTERS 8 +#endif + #define GCOV_TAG_FUNCTION_LENGTH 3 static struct gcov_info *gcov_info_head; diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 48e78b657d23..c6b98793d647 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -7669,7 +7669,7 @@ cpu_cgroup_css_alloc(struct cgroup_subsys_state *parent_css) static int cpu_cgroup_css_online(struct cgroup_subsys_state *css) { struct task_group *tg = css_tg(css); - struct task_group *parent = css_tg(css_parent(css)); + struct task_group *parent = css_tg(css->parent); if (parent) sched_online_group(tg, parent); diff --git a/kernel/sched/cpuacct.c b/kernel/sched/cpuacct.c index c143ee380e3a..9cf350c94ec4 100644 --- a/kernel/sched/cpuacct.c +++ b/kernel/sched/cpuacct.c @@ -46,7 +46,7 @@ static inline struct cpuacct *task_ca(struct task_struct *tsk) static inline struct cpuacct *parent_ca(struct cpuacct *ca) { - return css_ca(css_parent(&ca->css)); + return css_ca(ca->css.parent); } static DEFINE_PER_CPU(u64, root_cpuacct_cpuusage); diff --git a/kernel/trace/Kconfig b/kernel/trace/Kconfig index 8639819f6cef..d4409356f40d 100644 --- a/kernel/trace/Kconfig +++ b/kernel/trace/Kconfig @@ -535,6 +535,36 @@ config MMIOTRACE_TEST Say N, unless you absolutely know what you are doing. +config TRACEPOINT_BENCHMARK + bool "Add tracepoint that benchmarks tracepoints" + help + This option creates the tracepoint "benchmark:benchmark_event". + When the tracepoint is enabled, it kicks off a kernel thread that + goes into an infinite loop (calling cond_sched() to let other tasks + run), and calls the tracepoint. Each iteration will record the time + it took to write to the tracepoint and the next iteration that + data will be passed to the tracepoint itself. That is, the tracepoint + will report the time it took to do the previous tracepoint. + The string written to the tracepoint is a static string of 128 bytes + to keep the time the same. The initial string is simply a write of + "START". The second string records the cold cache time of the first + write which is not added to the rest of the calculations. + + As it is a tight loop, it benchmarks as hot cache. That's fine because + we care most about hot paths that are probably in cache already. + + An example of the output: + + START + first=3672 [COLD CACHED] + last=632 first=3672 max=632 min=632 avg=316 std=446 std^2=199712 + last=278 first=3672 max=632 min=278 avg=303 std=316 std^2=100337 + last=277 first=3672 max=632 min=277 avg=296 std=258 std^2=67064 + last=273 first=3672 max=632 min=273 avg=292 std=224 std^2=50411 + last=273 first=3672 max=632 min=273 avg=288 std=200 std^2=40389 + last=281 first=3672 max=632 min=273 avg=287 std=183 std^2=33666 + + config RING_BUFFER_BENCHMARK tristate "Ring buffer benchmark stress tester" depends on RING_BUFFER diff --git a/kernel/trace/Makefile b/kernel/trace/Makefile index 1378e84fbe39..2611613f14f1 100644 --- a/kernel/trace/Makefile +++ b/kernel/trace/Makefile @@ -17,6 +17,7 @@ ifdef CONFIG_TRACING_BRANCHES KBUILD_CFLAGS += -DDISABLE_BRANCH_PROFILING endif +CFLAGS_trace_benchmark.o := -I$(src) CFLAGS_trace_events_filter.o := -I$(src) obj-$(CONFIG_TRACE_CLOCK) += trace_clock.o @@ -62,4 +63,6 @@ endif obj-$(CONFIG_PROBE_EVENTS) += trace_probe.o obj-$(CONFIG_UPROBE_EVENT) += trace_uprobe.o +obj-$(CONFIG_TRACEPOINT_BENCHMARK) += trace_benchmark.o + libftrace-y := ftrace.o diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index 4a54a25afa2f..5b372e3ed675 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -62,7 +62,7 @@ #define FTRACE_HASH_DEFAULT_BITS 10 #define FTRACE_HASH_MAX_BITS 12 -#define FL_GLOBAL_CONTROL_MASK (FTRACE_OPS_FL_GLOBAL | FTRACE_OPS_FL_CONTROL) +#define FL_GLOBAL_CONTROL_MASK (FTRACE_OPS_FL_CONTROL) #ifdef CONFIG_DYNAMIC_FTRACE #define INIT_REGEX_LOCK(opsname) \ @@ -103,7 +103,6 @@ static int ftrace_disabled __read_mostly; static DEFINE_MUTEX(ftrace_lock); -static struct ftrace_ops *ftrace_global_list __read_mostly = &ftrace_list_end; static struct ftrace_ops *ftrace_control_list __read_mostly = &ftrace_list_end; static struct ftrace_ops *ftrace_ops_list __read_mostly = &ftrace_list_end; ftrace_func_t ftrace_trace_function __read_mostly = ftrace_stub; @@ -171,23 +170,6 @@ int ftrace_nr_registered_ops(void) return cnt; } -static void -ftrace_global_list_func(unsigned long ip, unsigned long parent_ip, - struct ftrace_ops *op, struct pt_regs *regs) -{ - int bit; - - bit = trace_test_and_set_recursion(TRACE_GLOBAL_START, TRACE_GLOBAL_MAX); - if (bit < 0) - return; - - do_for_each_ftrace_op(op, ftrace_global_list) { - op->func(ip, parent_ip, op, regs); - } while_for_each_ftrace_op(op); - - trace_clear_recursion(bit); -} - static void ftrace_pid_func(unsigned long ip, unsigned long parent_ip, struct ftrace_ops *op, struct pt_regs *regs) { @@ -237,43 +219,6 @@ static int control_ops_alloc(struct ftrace_ops *ops) return 0; } -static void update_global_ops(void) -{ - ftrace_func_t func = ftrace_global_list_func; - void *private = NULL; - - /* The list has its own recursion protection. */ - global_ops.flags |= FTRACE_OPS_FL_RECURSION_SAFE; - - /* - * If there's only one function registered, then call that - * function directly. Otherwise, we need to iterate over the - * registered callers. - */ - if (ftrace_global_list == &ftrace_list_end || - ftrace_global_list->next == &ftrace_list_end) { - func = ftrace_global_list->func; - private = ftrace_global_list->private; - /* - * As we are calling the function directly. - * If it does not have recursion protection, - * the function_trace_op needs to be updated - * accordingly. - */ - if (!(ftrace_global_list->flags & FTRACE_OPS_FL_RECURSION_SAFE)) - global_ops.flags &= ~FTRACE_OPS_FL_RECURSION_SAFE; - } - - /* If we filter on pids, update to use the pid function */ - if (!list_empty(&ftrace_pids)) { - set_ftrace_pid_function(func); - func = ftrace_pid_func; - } - - global_ops.func = func; - global_ops.private = private; -} - static void ftrace_sync(struct work_struct *work) { /* @@ -301,8 +246,6 @@ static void update_ftrace_function(void) { ftrace_func_t func; - update_global_ops(); - /* * If we are at the end of the list and this ops is * recursion safe and not dynamic and the arch supports passing ops, @@ -314,10 +257,7 @@ static void update_ftrace_function(void) (ftrace_ops_list->flags & FTRACE_OPS_FL_RECURSION_SAFE) && !FTRACE_FORCE_LIST_FUNC)) { /* Set the ftrace_ops that the arch callback uses */ - if (ftrace_ops_list == &global_ops) - set_function_trace_op = ftrace_global_list; - else - set_function_trace_op = ftrace_ops_list; + set_function_trace_op = ftrace_ops_list; func = ftrace_ops_list->func; } else { /* Just use the default ftrace_ops */ @@ -373,6 +313,11 @@ static void update_ftrace_function(void) ftrace_trace_function = func; } +int using_ftrace_ops_list_func(void) +{ + return ftrace_trace_function == ftrace_ops_list_func; +} + static void add_ftrace_ops(struct ftrace_ops **list, struct ftrace_ops *ops) { ops->next = *list; @@ -434,16 +379,9 @@ static int __register_ftrace_function(struct ftrace_ops *ops) if (ops->flags & FTRACE_OPS_FL_DELETED) return -EINVAL; - if (FTRACE_WARN_ON(ops == &global_ops)) - return -EINVAL; - if (WARN_ON(ops->flags & FTRACE_OPS_FL_ENABLED)) return -EBUSY; - /* We don't support both control and global flags set. */ - if ((ops->flags & FL_GLOBAL_CONTROL_MASK) == FL_GLOBAL_CONTROL_MASK) - return -EINVAL; - #ifndef CONFIG_DYNAMIC_FTRACE_WITH_REGS /* * If the ftrace_ops specifies SAVE_REGS, then it only can be used @@ -461,10 +399,7 @@ static int __register_ftrace_function(struct ftrace_ops *ops) if (!core_kernel_data((unsigned long)ops)) ops->flags |= FTRACE_OPS_FL_DYNAMIC; - if (ops->flags & FTRACE_OPS_FL_GLOBAL) { - add_ftrace_list_ops(&ftrace_global_list, &global_ops, ops); - ops->flags |= FTRACE_OPS_FL_ENABLED; - } else if (ops->flags & FTRACE_OPS_FL_CONTROL) { + if (ops->flags & FTRACE_OPS_FL_CONTROL) { if (control_ops_alloc(ops)) return -ENOMEM; add_ftrace_list_ops(&ftrace_control_list, &control_ops, ops); @@ -484,15 +419,7 @@ static int __unregister_ftrace_function(struct ftrace_ops *ops) if (WARN_ON(!(ops->flags & FTRACE_OPS_FL_ENABLED))) return -EBUSY; - if (FTRACE_WARN_ON(ops == &global_ops)) - return -EINVAL; - - if (ops->flags & FTRACE_OPS_FL_GLOBAL) { - ret = remove_ftrace_list_ops(&ftrace_global_list, - &global_ops, ops); - if (!ret) - ops->flags &= ~FTRACE_OPS_FL_ENABLED; - } else if (ops->flags & FTRACE_OPS_FL_CONTROL) { + if (ops->flags & FTRACE_OPS_FL_CONTROL) { ret = remove_ftrace_list_ops(&ftrace_control_list, &control_ops, ops); } else @@ -895,7 +822,7 @@ function_profile_call(unsigned long ip, unsigned long parent_ip, local_irq_save(flags); - stat = &__get_cpu_var(ftrace_profile_stats); + stat = this_cpu_ptr(&ftrace_profile_stats); if (!stat->hash || !ftrace_profile_enabled) goto out; @@ -926,7 +853,7 @@ static void profile_graph_return(struct ftrace_graph_ret *trace) unsigned long flags; local_irq_save(flags); - stat = &__get_cpu_var(ftrace_profile_stats); + stat = this_cpu_ptr(&ftrace_profile_stats); if (!stat->hash || !ftrace_profile_enabled) goto out; @@ -1178,7 +1105,7 @@ struct ftrace_page { static struct ftrace_page *ftrace_pages_start; static struct ftrace_page *ftrace_pages; -static bool ftrace_hash_empty(struct ftrace_hash *hash) +static bool __always_inline ftrace_hash_empty(struct ftrace_hash *hash) { return !hash || !hash->count; } @@ -1625,7 +1552,14 @@ static void __ftrace_hash_rec_update(struct ftrace_ops *ops, in_other_hash = !!ftrace_lookup_ip(other_hash, rec->ip); /* + * If filter_hash is set, we want to match all functions + * that are in the hash but not in the other hash. * + * If filter_hash is not set, then we are decrementing. + * That means we match anything that is in the hash + * and also in the other_hash. That is, we need to turn + * off functions in the other hash because they are disabled + * by this hash. */ if (filter_hash && in_hash && !in_other_hash) match = 1; @@ -1767,19 +1701,15 @@ static int ftrace_check_record(struct dyn_ftrace *rec, int enable, int update) /* * If this record is being updated from a nop, then * return UPDATE_MAKE_CALL. - * Otherwise, if the EN flag is set, then return - * UPDATE_MODIFY_CALL_REGS to tell the caller to convert - * from the non-save regs, to a save regs function. * Otherwise, * return UPDATE_MODIFY_CALL to tell the caller to convert - * from the save regs, to a non-save regs function. + * from the save regs, to a non-save regs function or + * vice versa. */ if (flag & FTRACE_FL_ENABLED) return FTRACE_UPDATE_MAKE_CALL; - else if (rec->flags & FTRACE_FL_REGS_EN) - return FTRACE_UPDATE_MODIFY_CALL_REGS; - else - return FTRACE_UPDATE_MODIFY_CALL; + + return FTRACE_UPDATE_MODIFY_CALL; } if (update) { @@ -1821,6 +1751,42 @@ int ftrace_test_record(struct dyn_ftrace *rec, int enable) return ftrace_check_record(rec, enable, 0); } +/** + * ftrace_get_addr_new - Get the call address to set to + * @rec: The ftrace record descriptor + * + * If the record has the FTRACE_FL_REGS set, that means that it + * wants to convert to a callback that saves all regs. If FTRACE_FL_REGS + * is not not set, then it wants to convert to the normal callback. + * + * Returns the address of the trampoline to set to + */ +unsigned long ftrace_get_addr_new(struct dyn_ftrace *rec) +{ + if (rec->flags & FTRACE_FL_REGS) + return (unsigned long)FTRACE_REGS_ADDR; + else + return (unsigned long)FTRACE_ADDR; +} + +/** + * ftrace_get_addr_curr - Get the call address that is already there + * @rec: The ftrace record descriptor + * + * The FTRACE_FL_REGS_EN is set when the record already points to + * a function that saves all the regs. Basically the '_EN' version + * represents the current state of the function. + * + * Returns the address of the trampoline that is currently being called + */ +unsigned long ftrace_get_addr_curr(struct dyn_ftrace *rec) +{ + if (rec->flags & FTRACE_FL_REGS_EN) + return (unsigned long)FTRACE_REGS_ADDR; + else + return (unsigned long)FTRACE_ADDR; +} + static int __ftrace_replace_code(struct dyn_ftrace *rec, int enable) { @@ -1828,12 +1794,12 @@ __ftrace_replace_code(struct dyn_ftrace *rec, int enable) unsigned long ftrace_addr; int ret; - ret = ftrace_update_record(rec, enable); + ftrace_addr = ftrace_get_addr_new(rec); - if (rec->flags & FTRACE_FL_REGS) - ftrace_addr = (unsigned long)FTRACE_REGS_ADDR; - else - ftrace_addr = (unsigned long)FTRACE_ADDR; + /* This needs to be done before we call ftrace_update_record */ + ftrace_old_addr = ftrace_get_addr_curr(rec); + + ret = ftrace_update_record(rec, enable); switch (ret) { case FTRACE_UPDATE_IGNORE: @@ -1845,13 +1811,7 @@ __ftrace_replace_code(struct dyn_ftrace *rec, int enable) case FTRACE_UPDATE_MAKE_NOP: return ftrace_make_nop(NULL, rec, ftrace_addr); - case FTRACE_UPDATE_MODIFY_CALL_REGS: case FTRACE_UPDATE_MODIFY_CALL: - if (rec->flags & FTRACE_FL_REGS) - ftrace_old_addr = (unsigned long)FTRACE_ADDR; - else - ftrace_old_addr = (unsigned long)FTRACE_REGS_ADDR; - return ftrace_modify_call(rec, ftrace_old_addr, ftrace_addr); } @@ -2115,7 +2075,6 @@ static void ftrace_startup_enable(int command) static int ftrace_startup(struct ftrace_ops *ops, int command) { - bool hash_enable = true; int ret; if (unlikely(ftrace_disabled)) @@ -2128,18 +2087,9 @@ static int ftrace_startup(struct ftrace_ops *ops, int command) ftrace_start_up++; command |= FTRACE_UPDATE_CALLS; - /* ops marked global share the filter hashes */ - if (ops->flags & FTRACE_OPS_FL_GLOBAL) { - ops = &global_ops; - /* Don't update hash if global is already set */ - if (global_start_up) - hash_enable = false; - global_start_up++; - } - ops->flags |= FTRACE_OPS_FL_ENABLED; - if (hash_enable) - ftrace_hash_rec_enable(ops, 1); + + ftrace_hash_rec_enable(ops, 1); ftrace_startup_enable(command); @@ -2148,7 +2098,6 @@ static int ftrace_startup(struct ftrace_ops *ops, int command) static int ftrace_shutdown(struct ftrace_ops *ops, int command) { - bool hash_disable = true; int ret; if (unlikely(ftrace_disabled)) @@ -2166,21 +2115,9 @@ static int ftrace_shutdown(struct ftrace_ops *ops, int command) */ WARN_ON_ONCE(ftrace_start_up < 0); - if (ops->flags & FTRACE_OPS_FL_GLOBAL) { - ops = &global_ops; - global_start_up--; - WARN_ON_ONCE(global_start_up < 0); - /* Don't update hash if global still has users */ - if (global_start_up) { - WARN_ON_ONCE(!ftrace_start_up); - hash_disable = false; - } - } - - if (hash_disable) - ftrace_hash_rec_disable(ops, 1); + ftrace_hash_rec_disable(ops, 1); - if (ops != &global_ops || !global_start_up) + if (!global_start_up) ops->flags &= ~FTRACE_OPS_FL_ENABLED; command |= FTRACE_UPDATE_CALLS; @@ -3524,10 +3461,6 @@ ftrace_set_hash(struct ftrace_ops *ops, unsigned char *buf, int len, struct ftrace_hash *hash; int ret; - /* All global ops uses the global ops filters */ - if (ops->flags & FTRACE_OPS_FL_GLOBAL) - ops = &global_ops; - if (unlikely(ftrace_disabled)) return -ENODEV; @@ -3639,8 +3572,7 @@ int ftrace_set_notrace(struct ftrace_ops *ops, unsigned char *buf, } EXPORT_SYMBOL_GPL(ftrace_set_notrace); /** - * ftrace_set_filter - set a function to filter on in ftrace - * @ops - the ops to set the filter with + * ftrace_set_global_filter - set a function to filter on with global tracers * @buf - the string that holds the function filter text. * @len - the length of the string. * @reset - non zero to reset all filters before applying this filter. @@ -3655,8 +3587,7 @@ void ftrace_set_global_filter(unsigned char *buf, int len, int reset) EXPORT_SYMBOL_GPL(ftrace_set_global_filter); /** - * ftrace_set_notrace - set a function to not trace in ftrace - * @ops - the ops to set the notrace filter with + * ftrace_set_global_notrace - set a function to not trace with global tracers * @buf - the string that holds the function notrace text. * @len - the length of the string. * @reset - non zero to reset all filters before applying this filter. @@ -4443,6 +4374,34 @@ ftrace_ops_test(struct ftrace_ops *ops, unsigned long ip, void *regs) #endif /* CONFIG_DYNAMIC_FTRACE */ +__init void ftrace_init_global_array_ops(struct trace_array *tr) +{ + tr->ops = &global_ops; + tr->ops->private = tr; +} + +void ftrace_init_array_ops(struct trace_array *tr, ftrace_func_t func) +{ + /* If we filter on pids, update to use the pid function */ + if (tr->flags & TRACE_ARRAY_FL_GLOBAL) { + if (WARN_ON(tr->ops->func != ftrace_stub)) + printk("ftrace ops had %pS for function\n", + tr->ops->func); + /* Only the top level instance does pid tracing */ + if (!list_empty(&ftrace_pids)) { + set_ftrace_pid_function(func); + func = ftrace_pid_func; + } + } + tr->ops->func = func; + tr->ops->private = tr; +} + +void ftrace_reset_array_ops(struct trace_array *tr) +{ + tr->ops->func = ftrace_stub; +} + static void ftrace_ops_control_func(unsigned long ip, unsigned long parent_ip, struct ftrace_ops *op, struct pt_regs *regs) @@ -4501,9 +4460,16 @@ __ftrace_ops_list_func(unsigned long ip, unsigned long parent_ip, */ preempt_disable_notrace(); do_for_each_ftrace_op(op, ftrace_ops_list) { - if (ftrace_ops_test(op, ip, regs)) + if (ftrace_ops_test(op, ip, regs)) { + if (WARN_ON(!op->func)) { + function_trace_stop = 1; + printk("op=%p %pS\n", op, op); + goto out; + } op->func(ip, parent_ip, op, regs); + } } while_for_each_ftrace_op(op); +out: preempt_enable_notrace(); trace_clear_recursion(bit); } @@ -4908,7 +4874,6 @@ ftrace_enable_sysctl(struct ctl_table *table, int write, #ifdef CONFIG_FUNCTION_GRAPH_TRACER static int ftrace_graph_active; -static struct notifier_block ftrace_suspend_notifier; int ftrace_graph_entry_stub(struct ftrace_graph_ent *trace) { @@ -5054,13 +5019,6 @@ ftrace_suspend_notifier_call(struct notifier_block *bl, unsigned long state, return NOTIFY_DONE; } -/* Just a place holder for function graph */ -static struct ftrace_ops fgraph_ops __read_mostly = { - .func = ftrace_stub, - .flags = FTRACE_OPS_FL_STUB | FTRACE_OPS_FL_GLOBAL | - FTRACE_OPS_FL_RECURSION_SAFE, -}; - static int ftrace_graph_entry_test(struct ftrace_graph_ent *trace) { if (!ftrace_ops_test(&global_ops, trace->func, NULL)) @@ -5085,6 +5043,10 @@ static void update_function_graph_func(void) ftrace_graph_entry = ftrace_graph_entry_test; } +static struct notifier_block ftrace_suspend_notifier = { + .notifier_call = ftrace_suspend_notifier_call, +}; + int register_ftrace_graph(trace_func_graph_ret_t retfunc, trace_func_graph_ent_t entryfunc) { @@ -5098,7 +5060,6 @@ int register_ftrace_graph(trace_func_graph_ret_t retfunc, goto out; } - ftrace_suspend_notifier.notifier_call = ftrace_suspend_notifier_call; register_pm_notifier(&ftrace_suspend_notifier); ftrace_graph_active++; @@ -5120,7 +5081,10 @@ int register_ftrace_graph(trace_func_graph_ret_t retfunc, ftrace_graph_entry = ftrace_graph_entry_test; update_function_graph_func(); - ret = ftrace_startup(&fgraph_ops, FTRACE_START_FUNC_RET); + /* Function graph doesn't use the .func field of global_ops */ + global_ops.flags |= FTRACE_OPS_FL_STUB; + + ret = ftrace_startup(&global_ops, FTRACE_START_FUNC_RET); out: mutex_unlock(&ftrace_lock); @@ -5138,7 +5102,8 @@ void unregister_ftrace_graph(void) ftrace_graph_return = (trace_func_graph_ret_t)ftrace_stub; ftrace_graph_entry = ftrace_graph_entry_stub; __ftrace_graph_entry = ftrace_graph_entry_stub; - ftrace_shutdown(&fgraph_ops, FTRACE_STOP_FUNC_RET); + ftrace_shutdown(&global_ops, FTRACE_STOP_FUNC_RET); + global_ops.flags &= ~FTRACE_OPS_FL_STUB; unregister_pm_notifier(&ftrace_suspend_notifier); unregister_trace_sched_switch(ftrace_graph_probe_sched_switch, NULL); diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c index 737b0efa1a62..16f7038d1f4d 100644 --- a/kernel/trace/trace.c +++ b/kernel/trace/trace.c @@ -275,7 +275,7 @@ int call_filter_check_discard(struct ftrace_event_call *call, void *rec, } EXPORT_SYMBOL_GPL(call_filter_check_discard); -cycle_t buffer_ftrace_now(struct trace_buffer *buf, int cpu) +static cycle_t buffer_ftrace_now(struct trace_buffer *buf, int cpu) { u64 ts; @@ -599,7 +599,7 @@ static int alloc_snapshot(struct trace_array *tr) return 0; } -void free_snapshot(struct trace_array *tr) +static void free_snapshot(struct trace_array *tr) { /* * We don't free the ring buffer. instead, resize it because @@ -963,27 +963,9 @@ static ssize_t trace_seq_to_buffer(struct trace_seq *s, void *buf, size_t cnt) return cnt; } -/* - * ftrace_max_lock is used to protect the swapping of buffers - * when taking a max snapshot. The buffers themselves are - * protected by per_cpu spinlocks. But the action of the swap - * needs its own lock. - * - * This is defined as a arch_spinlock_t in order to help - * with performance when lockdep debugging is enabled. - * - * It is also used in other places outside the update_max_tr - * so it needs to be defined outside of the - * CONFIG_TRACER_MAX_TRACE. - */ -static arch_spinlock_t ftrace_max_lock = - (arch_spinlock_t)__ARCH_SPIN_LOCK_UNLOCKED; - unsigned long __read_mostly tracing_thresh; #ifdef CONFIG_TRACER_MAX_TRACE -unsigned long __read_mostly tracing_max_latency; - /* * Copy the new maximum trace into the separate maximum-trace * structure. (this way the maximum trace is permanently saved, @@ -1000,7 +982,7 @@ __update_max_tr(struct trace_array *tr, struct task_struct *tsk, int cpu) max_buf->cpu = cpu; max_buf->time_start = data->preempt_timestamp; - max_data->saved_latency = tracing_max_latency; + max_data->saved_latency = tr->max_latency; max_data->critical_start = data->critical_start; max_data->critical_end = data->critical_end; @@ -1048,14 +1030,14 @@ update_max_tr(struct trace_array *tr, struct task_struct *tsk, int cpu) return; } - arch_spin_lock(&ftrace_max_lock); + arch_spin_lock(&tr->max_lock); buf = tr->trace_buffer.buffer; tr->trace_buffer.buffer = tr->max_buffer.buffer; tr->max_buffer.buffer = buf; __update_max_tr(tr, tsk, cpu); - arch_spin_unlock(&ftrace_max_lock); + arch_spin_unlock(&tr->max_lock); } /** @@ -1081,7 +1063,7 @@ update_max_tr_single(struct trace_array *tr, struct task_struct *tsk, int cpu) return; } - arch_spin_lock(&ftrace_max_lock); + arch_spin_lock(&tr->max_lock); ret = ring_buffer_swap_cpu(tr->max_buffer.buffer, tr->trace_buffer.buffer, cpu); @@ -1099,11 +1081,11 @@ update_max_tr_single(struct trace_array *tr, struct task_struct *tsk, int cpu) WARN_ON_ONCE(ret && ret != -EAGAIN && ret != -EBUSY); __update_max_tr(tr, tsk, cpu); - arch_spin_unlock(&ftrace_max_lock); + arch_spin_unlock(&tr->max_lock); } #endif /* CONFIG_TRACER_MAX_TRACE */ -static void default_wait_pipe(struct trace_iterator *iter) +static void wait_on_pipe(struct trace_iterator *iter) { /* Iterators are static, they should be filled or empty */ if (trace_buffer_iter(iter, iter->cpu_file)) @@ -1220,8 +1202,6 @@ int register_tracer(struct tracer *type) else if (!type->flags->opts) type->flags->opts = dummy_tracer_opt; - if (!type->wait_pipe) - type->wait_pipe = default_wait_pipe; ret = run_tracer_selftest(type); if (ret < 0) @@ -1305,22 +1285,71 @@ void tracing_reset_all_online_cpus(void) } } -#define SAVED_CMDLINES 128 +#define SAVED_CMDLINES_DEFAULT 128 #define NO_CMDLINE_MAP UINT_MAX -static unsigned map_pid_to_cmdline[PID_MAX_DEFAULT+1]; -static unsigned map_cmdline_to_pid[SAVED_CMDLINES]; -static char saved_cmdlines[SAVED_CMDLINES][TASK_COMM_LEN]; -static int cmdline_idx; static arch_spinlock_t trace_cmdline_lock = __ARCH_SPIN_LOCK_UNLOCKED; +struct saved_cmdlines_buffer { + unsigned map_pid_to_cmdline[PID_MAX_DEFAULT+1]; + unsigned *map_cmdline_to_pid; + unsigned cmdline_num; + int cmdline_idx; + char *saved_cmdlines; +}; +static struct saved_cmdlines_buffer *savedcmd; /* temporary disable recording */ static atomic_t trace_record_cmdline_disabled __read_mostly; -static void trace_init_cmdlines(void) +static inline char *get_saved_cmdlines(int idx) { - memset(&map_pid_to_cmdline, NO_CMDLINE_MAP, sizeof(map_pid_to_cmdline)); - memset(&map_cmdline_to_pid, NO_CMDLINE_MAP, sizeof(map_cmdline_to_pid)); - cmdline_idx = 0; + return &savedcmd->saved_cmdlines[idx * TASK_COMM_LEN]; +} + +static inline void set_cmdline(int idx, const char *cmdline) +{ + memcpy(get_saved_cmdlines(idx), cmdline, TASK_COMM_LEN); +} + +static int allocate_cmdlines_buffer(unsigned int val, + struct saved_cmdlines_buffer *s) +{ + s->map_cmdline_to_pid = kmalloc(val * sizeof(*s->map_cmdline_to_pid), + GFP_KERNEL); + if (!s->map_cmdline_to_pid) + return -ENOMEM; + + s->saved_cmdlines = kmalloc(val * TASK_COMM_LEN, GFP_KERNEL); + if (!s->saved_cmdlines) { + kfree(s->map_cmdline_to_pid); + return -ENOMEM; + } + + s->cmdline_idx = 0; + s->cmdline_num = val; + memset(&s->map_pid_to_cmdline, NO_CMDLINE_MAP, + sizeof(s->map_pid_to_cmdline)); + memset(s->map_cmdline_to_pid, NO_CMDLINE_MAP, + val * sizeof(*s->map_cmdline_to_pid)); + + return 0; +} + +static int trace_create_savedcmd(void) +{ + int ret; + + savedcmd = kmalloc(sizeof(struct saved_cmdlines_buffer), GFP_KERNEL); + if (!savedcmd) + return -ENOMEM; + + ret = allocate_cmdlines_buffer(SAVED_CMDLINES_DEFAULT, savedcmd); + if (ret < 0) { + kfree(savedcmd); + savedcmd = NULL; + return -ENOMEM; + } + + return 0; } int is_tracing_stopped(void) @@ -1353,7 +1382,7 @@ void tracing_start(void) } /* Prevent the buffers from switching */ - arch_spin_lock(&ftrace_max_lock); + arch_spin_lock(&global_trace.max_lock); buffer = global_trace.trace_buffer.buffer; if (buffer) @@ -1365,7 +1394,7 @@ void tracing_start(void) ring_buffer_record_enable(buffer); #endif - arch_spin_unlock(&ftrace_max_lock); + arch_spin_unlock(&global_trace.max_lock); ftrace_start(); out: @@ -1420,7 +1449,7 @@ void tracing_stop(void) goto out; /* Prevent the buffers from switching */ - arch_spin_lock(&ftrace_max_lock); + arch_spin_lock(&global_trace.max_lock); buffer = global_trace.trace_buffer.buffer; if (buffer) @@ -1432,7 +1461,7 @@ void tracing_stop(void) ring_buffer_record_disable(buffer); #endif - arch_spin_unlock(&ftrace_max_lock); + arch_spin_unlock(&global_trace.max_lock); out: raw_spin_unlock_irqrestore(&global_trace.start_lock, flags); @@ -1461,12 +1490,12 @@ static void tracing_stop_tr(struct trace_array *tr) void trace_stop_cmdline_recording(void); -static void trace_save_cmdline(struct task_struct *tsk) +static int trace_save_cmdline(struct task_struct *tsk) { unsigned pid, idx; if (!tsk->pid || unlikely(tsk->pid > PID_MAX_DEFAULT)) - return; + return 0; /* * It's not the end of the world if we don't get @@ -1475,11 +1504,11 @@ static void trace_save_cmdline(struct task_struct *tsk) * so if we miss here, then better luck next time. */ if (!arch_spin_trylock(&trace_cmdline_lock)) - return; + return 0; - idx = map_pid_to_cmdline[tsk->pid]; + idx = savedcmd->map_pid_to_cmdline[tsk->pid]; if (idx == NO_CMDLINE_MAP) { - idx = (cmdline_idx + 1) % SAVED_CMDLINES; + idx = (savedcmd->cmdline_idx + 1) % savedcmd->cmdline_num; /* * Check whether the cmdline buffer at idx has a pid @@ -1487,22 +1516,24 @@ static void trace_save_cmdline(struct task_struct *tsk) * need to clear the map_pid_to_cmdline. Otherwise we * would read the new comm for the old pid. */ - pid = map_cmdline_to_pid[idx]; + pid = savedcmd->map_cmdline_to_pid[idx]; if (pid != NO_CMDLINE_MAP) - map_pid_to_cmdline[pid] = NO_CMDLINE_MAP; + savedcmd->map_pid_to_cmdline[pid] = NO_CMDLINE_MAP; - map_cmdline_to_pid[idx] = tsk->pid; - map_pid_to_cmdline[tsk->pid] = idx; + savedcmd->map_cmdline_to_pid[idx] = tsk->pid; + savedcmd->map_pid_to_cmdline[tsk->pid] = idx; - cmdline_idx = idx; + savedcmd->cmdline_idx = idx; } - memcpy(&saved_cmdlines[idx], tsk->comm, TASK_COMM_LEN); + set_cmdline(idx, tsk->comm); arch_spin_unlock(&trace_cmdline_lock); + + return 1; } -void trace_find_cmdline(int pid, char comm[]) +static void __trace_find_cmdline(int pid, char comm[]) { unsigned map; @@ -1521,13 +1552,19 @@ void trace_find_cmdline(int pid, char comm[]) return; } - preempt_disable(); - arch_spin_lock(&trace_cmdline_lock); - map = map_pid_to_cmdline[pid]; + map = savedcmd->map_pid_to_cmdline[pid]; if (map != NO_CMDLINE_MAP) - strcpy(comm, saved_cmdlines[map]); + strcpy(comm, get_saved_cmdlines(map)); else strcpy(comm, "<...>"); +} + +void trace_find_cmdline(int pid, char comm[]) +{ + preempt_disable(); + arch_spin_lock(&trace_cmdline_lock); + + __trace_find_cmdline(pid, comm); arch_spin_unlock(&trace_cmdline_lock); preempt_enable(); @@ -1541,9 +1578,8 @@ void tracing_record_cmdline(struct task_struct *tsk) if (!__this_cpu_read(trace_cmdline_save)) return; - __this_cpu_write(trace_cmdline_save, false); - - trace_save_cmdline(tsk); + if (trace_save_cmdline(tsk)) + __this_cpu_write(trace_cmdline_save, false); } void @@ -1746,7 +1782,7 @@ static void __ftrace_trace_stack(struct ring_buffer *buffer, */ barrier(); if (use_stack == 1) { - trace.entries = &__get_cpu_var(ftrace_stack).calls[0]; + trace.entries = this_cpu_ptr(ftrace_stack.calls); trace.max_entries = FTRACE_STACK_MAX_ENTRIES; if (regs) @@ -1995,7 +2031,21 @@ void trace_printk_init_buffers(void) if (alloc_percpu_trace_buffer()) return; - pr_info("ftrace: Allocated trace_printk buffers\n"); + /* trace_printk() is for debug use only. Don't use it in production. */ + + pr_warning("\n**********************************************************\n"); + pr_warning("** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE **\n"); + pr_warning("** **\n"); + pr_warning("** trace_printk() being used. Allocating extra memory. **\n"); + pr_warning("** **\n"); + pr_warning("** This means that this is a DEBUG kernel and it is **\n"); + pr_warning("** unsafe for produciton use. **\n"); + pr_warning("** **\n"); + pr_warning("** If you see this message and you are not debugging **\n"); + pr_warning("** the kernel, report this immediately to your vendor! **\n"); + pr_warning("** **\n"); + pr_warning("** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE **\n"); + pr_warning("**********************************************************\n"); /* Expand the buffers to set size */ tracing_update_buffers(); @@ -3333,7 +3383,7 @@ tracing_cpumask_write(struct file *filp, const char __user *ubuf, mutex_lock(&tracing_cpumask_update_lock); local_irq_disable(); - arch_spin_lock(&ftrace_max_lock); + arch_spin_lock(&tr->max_lock); for_each_tracing_cpu(cpu) { /* * Increase/decrease the disabled counter if we are @@ -3350,7 +3400,7 @@ tracing_cpumask_write(struct file *filp, const char __user *ubuf, ring_buffer_record_enable_cpu(tr->trace_buffer.buffer, cpu); } } - arch_spin_unlock(&ftrace_max_lock); + arch_spin_unlock(&tr->max_lock); local_irq_enable(); cpumask_copy(tr->tracing_cpumask, tracing_cpumask_new); @@ -3592,6 +3642,7 @@ static const char readme_msg[] = " trace_options\t\t- Set format or modify how tracing happens\n" "\t\t\t Disable an option by adding a suffix 'no' to the\n" "\t\t\t option name\n" + " saved_cmdlines_size\t- echo command number in here to store comm-pid list\n" #ifdef CONFIG_DYNAMIC_FTRACE "\n available_filter_functions - list of functions that can be filtered on\n" " set_ftrace_filter\t- echo function name in here to only trace these\n" @@ -3705,55 +3756,153 @@ static const struct file_operations tracing_readme_fops = { .llseek = generic_file_llseek, }; +static void *saved_cmdlines_next(struct seq_file *m, void *v, loff_t *pos) +{ + unsigned int *ptr = v; + + if (*pos || m->count) + ptr++; + + (*pos)++; + + for (; ptr < &savedcmd->map_cmdline_to_pid[savedcmd->cmdline_num]; + ptr++) { + if (*ptr == -1 || *ptr == NO_CMDLINE_MAP) + continue; + + return ptr; + } + + return NULL; +} + +static void *saved_cmdlines_start(struct seq_file *m, loff_t *pos) +{ + void *v; + loff_t l = 0; + + preempt_disable(); + arch_spin_lock(&trace_cmdline_lock); + + v = &savedcmd->map_cmdline_to_pid[0]; + while (l <= *pos) { + v = saved_cmdlines_next(m, v, &l); + if (!v) + return NULL; + } + + return v; +} + +static void saved_cmdlines_stop(struct seq_file *m, void *v) +{ + arch_spin_unlock(&trace_cmdline_lock); + preempt_enable(); +} + +static int saved_cmdlines_show(struct seq_file *m, void *v) +{ + char buf[TASK_COMM_LEN]; + unsigned int *pid = v; + + __trace_find_cmdline(*pid, buf); + seq_printf(m, "%d %s\n", *pid, buf); + return 0; +} + +static const struct seq_operations tracing_saved_cmdlines_seq_ops = { + .start = saved_cmdlines_start, + .next = saved_cmdlines_next, + .stop = saved_cmdlines_stop, + .show = saved_cmdlines_show, +}; + +static int tracing_saved_cmdlines_open(struct inode *inode, struct file *filp) +{ + if (tracing_disabled) + return -ENODEV; + + return seq_open(filp, &tracing_saved_cmdlines_seq_ops); +} + +static const struct file_operations tracing_saved_cmdlines_fops = { + .open = tracing_saved_cmdlines_open, + .read = seq_read, + .llseek = seq_lseek, + .release = seq_release, +}; + static ssize_t -tracing_saved_cmdlines_read(struct file *file, char __user *ubuf, - size_t cnt, loff_t *ppos) +tracing_saved_cmdlines_size_read(struct file *filp, char __user *ubuf, + size_t cnt, loff_t *ppos) { - char *buf_comm; - char *file_buf; - char *buf; - int len = 0; - int pid; - int i; + char buf[64]; + int r; + + arch_spin_lock(&trace_cmdline_lock); + r = sprintf(buf, "%u\n", savedcmd->cmdline_num); + arch_spin_unlock(&trace_cmdline_lock); + + return simple_read_from_buffer(ubuf, cnt, ppos, buf, r); +} + +static void free_saved_cmdlines_buffer(struct saved_cmdlines_buffer *s) +{ + kfree(s->saved_cmdlines); + kfree(s->map_cmdline_to_pid); + kfree(s); +} + +static int tracing_resize_saved_cmdlines(unsigned int val) +{ + struct saved_cmdlines_buffer *s, *savedcmd_temp; - file_buf = kmalloc(SAVED_CMDLINES*(16+TASK_COMM_LEN), GFP_KERNEL); - if (!file_buf) + s = kmalloc(sizeof(struct saved_cmdlines_buffer), GFP_KERNEL); + if (!s) return -ENOMEM; - buf_comm = kmalloc(TASK_COMM_LEN, GFP_KERNEL); - if (!buf_comm) { - kfree(file_buf); + if (allocate_cmdlines_buffer(val, s) < 0) { + kfree(s); return -ENOMEM; } - buf = file_buf; + arch_spin_lock(&trace_cmdline_lock); + savedcmd_temp = savedcmd; + savedcmd = s; + arch_spin_unlock(&trace_cmdline_lock); + free_saved_cmdlines_buffer(savedcmd_temp); - for (i = 0; i < SAVED_CMDLINES; i++) { - int r; + return 0; +} - pid = map_cmdline_to_pid[i]; - if (pid == -1 || pid == NO_CMDLINE_MAP) - continue; +static ssize_t +tracing_saved_cmdlines_size_write(struct file *filp, const char __user *ubuf, + size_t cnt, loff_t *ppos) +{ + unsigned long val; + int ret; - trace_find_cmdline(pid, buf_comm); - r = sprintf(buf, "%d %s\n", pid, buf_comm); - buf += r; - len += r; - } + ret = kstrtoul_from_user(ubuf, cnt, 10, &val); + if (ret) + return ret; - len = simple_read_from_buffer(ubuf, cnt, ppos, - file_buf, len); + /* must have at least 1 entry or less than PID_MAX_DEFAULT */ + if (!val || val > PID_MAX_DEFAULT) + return -EINVAL; - kfree(file_buf); - kfree(buf_comm); + ret = tracing_resize_saved_cmdlines((unsigned int)val); + if (ret < 0) + return ret; - return len; + *ppos += cnt; + + return cnt; } -static const struct file_operations tracing_saved_cmdlines_fops = { - .open = tracing_open_generic, - .read = tracing_saved_cmdlines_read, - .llseek = generic_file_llseek, +static const struct file_operations tracing_saved_cmdlines_size_fops = { + .open = tracing_open_generic, + .read = tracing_saved_cmdlines_size_read, + .write = tracing_saved_cmdlines_size_write, }; static ssize_t @@ -4225,25 +4374,6 @@ tracing_poll_pipe(struct file *filp, poll_table *poll_table) return trace_poll(iter, filp, poll_table); } -/* - * This is a make-shift waitqueue. - * A tracer might use this callback on some rare cases: - * - * 1) the current tracer might hold the runqueue lock when it wakes up - * a reader, hence a deadlock (sched, function, and function graph tracers) - * 2) the function tracers, trace all functions, we don't want - * the overhead of calling wake_up and friends - * (and tracing them too) - * - * Anyway, this is really very primitive wakeup. - */ -void poll_wait_pipe(struct trace_iterator *iter) -{ - set_current_state(TASK_INTERRUPTIBLE); - /* sleep for 100 msecs, and try again. */ - schedule_timeout(HZ / 10); -} - /* Must be called with trace_types_lock mutex held. */ static int tracing_wait_pipe(struct file *filp) { @@ -4255,15 +4385,6 @@ static int tracing_wait_pipe(struct file *filp) return -EAGAIN; } - mutex_unlock(&iter->mutex); - - iter->trace->wait_pipe(iter); - - mutex_lock(&iter->mutex); - - if (signal_pending(current)) - return -EINTR; - /* * We block until we read something and tracing is disabled. * We still block if tracing is disabled, but we have never @@ -4275,6 +4396,15 @@ static int tracing_wait_pipe(struct file *filp) */ if (!tracing_is_on() && iter->pos) break; + + mutex_unlock(&iter->mutex); + + wait_on_pipe(iter); + + mutex_lock(&iter->mutex); + + if (signal_pending(current)) + return -EINTR; } return 1; @@ -5197,7 +5327,7 @@ tracing_buffers_read(struct file *filp, char __user *ubuf, goto out_unlock; } mutex_unlock(&trace_types_lock); - iter->trace->wait_pipe(iter); + wait_on_pipe(iter); mutex_lock(&trace_types_lock); if (signal_pending(current)) { size = -EINTR; @@ -5408,7 +5538,7 @@ tracing_buffers_splice_read(struct file *file, loff_t *ppos, goto out; } mutex_unlock(&trace_types_lock); - iter->trace->wait_pipe(iter); + wait_on_pipe(iter); mutex_lock(&trace_types_lock); if (signal_pending(current)) { ret = -EINTR; @@ -6102,6 +6232,25 @@ static int allocate_trace_buffers(struct trace_array *tr, int size) return 0; } +static void free_trace_buffers(struct trace_array *tr) +{ + if (!tr) + return; + + if (tr->trace_buffer.buffer) { + ring_buffer_free(tr->trace_buffer.buffer); + tr->trace_buffer.buffer = NULL; + free_percpu(tr->trace_buffer.data); + } + +#ifdef CONFIG_TRACER_MAX_TRACE + if (tr->max_buffer.buffer) { + ring_buffer_free(tr->max_buffer.buffer); + tr->max_buffer.buffer = NULL; + } +#endif +} + static int new_instance_create(const char *name) { struct trace_array *tr; @@ -6131,6 +6280,8 @@ static int new_instance_create(const char *name) raw_spin_lock_init(&tr->start_lock); + tr->max_lock = (arch_spinlock_t)__ARCH_SPIN_LOCK_UNLOCKED; + tr->current_trace = &nop_trace; INIT_LIST_HEAD(&tr->systems); @@ -6158,8 +6309,7 @@ static int new_instance_create(const char *name) return 0; out_free_tr: - if (tr->trace_buffer.buffer) - ring_buffer_free(tr->trace_buffer.buffer); + free_trace_buffers(tr); free_cpumask_var(tr->tracing_cpumask); kfree(tr->name); kfree(tr); @@ -6199,8 +6349,7 @@ static int instance_delete(const char *name) event_trace_del_tracer(tr); ftrace_destroy_function_files(tr); debugfs_remove_recursive(tr->dir); - free_percpu(tr->trace_buffer.data); - ring_buffer_free(tr->trace_buffer.buffer); + free_trace_buffers(tr); kfree(tr->name); kfree(tr); @@ -6328,6 +6477,11 @@ init_tracer_debugfs(struct trace_array *tr, struct dentry *d_tracer) trace_create_file("tracing_on", 0644, d_tracer, tr, &rb_simple_fops); +#ifdef CONFIG_TRACER_MAX_TRACE + trace_create_file("tracing_max_latency", 0644, d_tracer, + &tr->max_latency, &tracing_max_lat_fops); +#endif + if (ftrace_create_function_files(tr, d_tracer)) WARN(1, "Could not allocate function filter files"); @@ -6353,11 +6507,6 @@ static __init int tracer_init_debugfs(void) init_tracer_debugfs(&global_trace, d_tracer); -#ifdef CONFIG_TRACER_MAX_TRACE - trace_create_file("tracing_max_latency", 0644, d_tracer, - &tracing_max_latency, &tracing_max_lat_fops); -#endif - trace_create_file("tracing_thresh", 0644, d_tracer, &tracing_thresh, &tracing_max_lat_fops); @@ -6367,6 +6516,9 @@ static __init int tracer_init_debugfs(void) trace_create_file("saved_cmdlines", 0444, d_tracer, NULL, &tracing_saved_cmdlines_fops); + trace_create_file("saved_cmdlines_size", 0644, d_tracer, + NULL, &tracing_saved_cmdlines_size_fops); + #ifdef CONFIG_DYNAMIC_FTRACE trace_create_file("dyn_ftrace_total_info", 0444, d_tracer, &ftrace_update_tot_cnt, &tracing_dyn_info_fops); @@ -6603,18 +6755,19 @@ __init static int tracer_alloc_buffers(void) if (!temp_buffer) goto out_free_cpumask; + if (trace_create_savedcmd() < 0) + goto out_free_temp_buffer; + /* TODO: make the number of buffers hot pluggable with CPUS */ if (allocate_trace_buffers(&global_trace, ring_buf_size) < 0) { printk(KERN_ERR "tracer: failed to allocate ring buffer!\n"); WARN_ON(1); - goto out_free_temp_buffer; + goto out_free_savedcmd; } if (global_trace.buffer_disabled) tracing_off(); - trace_init_cmdlines(); - if (trace_boot_clock) { ret = tracing_set_clock(&global_trace, trace_boot_clock); if (ret < 0) @@ -6629,6 +6782,10 @@ __init static int tracer_alloc_buffers(void) */ global_trace.current_trace = &nop_trace; + global_trace.max_lock = (arch_spinlock_t)__ARCH_SPIN_LOCK_UNLOCKED; + + ftrace_init_global_array_ops(&global_trace); + register_tracer(&nop_trace); /* All seems OK, enable tracing */ @@ -6656,13 +6813,11 @@ __init static int tracer_alloc_buffers(void) return 0; +out_free_savedcmd: + free_saved_cmdlines_buffer(savedcmd); out_free_temp_buffer: ring_buffer_free(temp_buffer); out_free_cpumask: - free_percpu(global_trace.trace_buffer.data); -#ifdef CONFIG_TRACER_MAX_TRACE - free_percpu(global_trace.max_buffer.data); -#endif free_cpumask_var(global_trace.tracing_cpumask); out_free_buffer_mask: free_cpumask_var(tracing_buffer_mask); diff --git a/kernel/trace/trace.h b/kernel/trace/trace.h index 2e29d7ba5a52..9e82551dd566 100644 --- a/kernel/trace/trace.h +++ b/kernel/trace/trace.h @@ -190,7 +190,22 @@ struct trace_array { */ struct trace_buffer max_buffer; bool allocated_snapshot; + unsigned long max_latency; #endif + /* + * max_lock is used to protect the swapping of buffers + * when taking a max snapshot. The buffers themselves are + * protected by per_cpu spinlocks. But the action of the swap + * needs its own lock. + * + * This is defined as a arch_spinlock_t in order to help + * with performance when lockdep debugging is enabled. + * + * It is also used in other places outside the update_max_tr + * so it needs to be defined outside of the + * CONFIG_TRACER_MAX_TRACE. + */ + arch_spinlock_t max_lock; int buffer_disabled; #ifdef CONFIG_FTRACE_SYSCALLS int sys_refcount_enter; @@ -237,6 +252,9 @@ static inline struct trace_array *top_trace_array(void) { struct trace_array *tr; + if (list_empty(ftrace_trace_arrays.prev)) + return NULL; + tr = list_entry(ftrace_trace_arrays.prev, typeof(*tr), list); WARN_ON(!(tr->flags & TRACE_ARRAY_FL_GLOBAL)); @@ -323,7 +341,6 @@ struct tracer_flags { * @stop: called when tracing is paused (echo 0 > tracing_enabled) * @open: called when the trace file is opened * @pipe_open: called when the trace_pipe file is opened - * @wait_pipe: override how the user waits for traces on trace_pipe * @close: called when the trace file is released * @pipe_close: called when the trace_pipe file is released * @read: override the default read callback on trace_pipe @@ -342,7 +359,6 @@ struct tracer { void (*stop)(struct trace_array *tr); void (*open)(struct trace_iterator *iter); void (*pipe_open)(struct trace_iterator *iter); - void (*wait_pipe)(struct trace_iterator *iter); void (*close)(struct trace_iterator *iter); void (*pipe_close)(struct trace_iterator *iter); ssize_t (*read)(struct trace_iterator *iter, @@ -416,13 +432,7 @@ enum { TRACE_FTRACE_IRQ_BIT, TRACE_FTRACE_SIRQ_BIT, - /* GLOBAL_BITs must be greater than FTRACE_BITs */ - TRACE_GLOBAL_BIT, - TRACE_GLOBAL_NMI_BIT, - TRACE_GLOBAL_IRQ_BIT, - TRACE_GLOBAL_SIRQ_BIT, - - /* INTERNAL_BITs must be greater than GLOBAL_BITs */ + /* INTERNAL_BITs must be greater than FTRACE_BITs */ TRACE_INTERNAL_BIT, TRACE_INTERNAL_NMI_BIT, TRACE_INTERNAL_IRQ_BIT, @@ -449,9 +459,6 @@ enum { #define TRACE_FTRACE_START TRACE_FTRACE_BIT #define TRACE_FTRACE_MAX ((1 << (TRACE_FTRACE_START + TRACE_CONTEXT_BITS)) - 1) -#define TRACE_GLOBAL_START TRACE_GLOBAL_BIT -#define TRACE_GLOBAL_MAX ((1 << (TRACE_GLOBAL_START + TRACE_CONTEXT_BITS)) - 1) - #define TRACE_LIST_START TRACE_INTERNAL_BIT #define TRACE_LIST_MAX ((1 << (TRACE_LIST_START + TRACE_CONTEXT_BITS)) - 1) @@ -560,8 +567,6 @@ void trace_init_global_iter(struct trace_iterator *iter); void tracing_iter_reset(struct trace_iterator *iter, int cpu); -void poll_wait_pipe(struct trace_iterator *iter); - void tracing_sched_switch_trace(struct trace_array *tr, struct task_struct *prev, struct task_struct *next, @@ -608,8 +613,6 @@ extern unsigned long nsecs_to_usecs(unsigned long nsecs); extern unsigned long tracing_thresh; #ifdef CONFIG_TRACER_MAX_TRACE -extern unsigned long tracing_max_latency; - void update_max_tr(struct trace_array *tr, struct task_struct *tsk, int cpu); void update_max_tr_single(struct trace_array *tr, struct task_struct *tsk, int cpu); @@ -724,6 +727,8 @@ extern unsigned long trace_flags; #define TRACE_GRAPH_PRINT_PROC 0x8 #define TRACE_GRAPH_PRINT_DURATION 0x10 #define TRACE_GRAPH_PRINT_ABS_TIME 0x20 +#define TRACE_GRAPH_PRINT_IRQS 0x40 +#define TRACE_GRAPH_PRINT_TAIL 0x80 #define TRACE_GRAPH_PRINT_FILL_SHIFT 28 #define TRACE_GRAPH_PRINT_FILL_MASK (0x3 << TRACE_GRAPH_PRINT_FILL_SHIFT) @@ -823,6 +828,10 @@ extern int ftrace_is_dead(void); int ftrace_create_function_files(struct trace_array *tr, struct dentry *parent); void ftrace_destroy_function_files(struct trace_array *tr); +void ftrace_init_global_array_ops(struct trace_array *tr); +void ftrace_init_array_ops(struct trace_array *tr, ftrace_func_t func); +void ftrace_reset_array_ops(struct trace_array *tr); +int using_ftrace_ops_list_func(void); #else static inline int ftrace_trace_task(struct task_struct *task) { @@ -836,6 +845,11 @@ ftrace_create_function_files(struct trace_array *tr, return 0; } static inline void ftrace_destroy_function_files(struct trace_array *tr) { } +static inline __init void +ftrace_init_global_array_ops(struct trace_array *tr) { } +static inline void ftrace_reset_array_ops(struct trace_array *tr) { } +/* ftace_func_t type is not defined, use macro instead of static inline */ +#define ftrace_init_array_ops(tr, func) do { } while (0) #endif /* CONFIG_FUNCTION_TRACER */ #if defined(CONFIG_FUNCTION_TRACER) && defined(CONFIG_DYNAMIC_FTRACE) diff --git a/kernel/trace/trace_benchmark.c b/kernel/trace/trace_benchmark.c new file mode 100644 index 000000000000..40a14cbcf8e0 --- /dev/null +++ b/kernel/trace/trace_benchmark.c @@ -0,0 +1,198 @@ +#include <linux/delay.h> +#include <linux/module.h> +#include <linux/kthread.h> +#include <linux/trace_clock.h> + +#define CREATE_TRACE_POINTS +#include "trace_benchmark.h" + +static struct task_struct *bm_event_thread; + +static char bm_str[BENCHMARK_EVENT_STRLEN] = "START"; + +static u64 bm_total; +static u64 bm_totalsq; +static u64 bm_last; +static u64 bm_max; +static u64 bm_min; +static u64 bm_first; +static u64 bm_cnt; +static u64 bm_stddev; +static unsigned int bm_avg; +static unsigned int bm_std; + +/* + * This gets called in a loop recording the time it took to write + * the tracepoint. What it writes is the time statistics of the last + * tracepoint write. As there is nothing to write the first time + * it simply writes "START". As the first write is cold cache and + * the rest is hot, we save off that time in bm_first and it is + * reported as "first", which is shown in the second write to the + * tracepoint. The "first" field is writen within the statics from + * then on but never changes. + */ +static void trace_do_benchmark(void) +{ + u64 start; + u64 stop; + u64 delta; + u64 stddev; + u64 seed; + u64 last_seed; + unsigned int avg; + unsigned int std = 0; + + /* Only run if the tracepoint is actually active */ + if (!trace_benchmark_event_enabled()) + return; + + local_irq_disable(); + start = trace_clock_local(); + trace_benchmark_event(bm_str); + stop = trace_clock_local(); + local_irq_enable(); + + bm_cnt++; + + delta = stop - start; + + /* + * The first read is cold cached, keep it separate from the + * other calculations. + */ + if (bm_cnt == 1) { + bm_first = delta; + scnprintf(bm_str, BENCHMARK_EVENT_STRLEN, + "first=%llu [COLD CACHED]", bm_first); + return; + } + + bm_last = delta; + + if (delta > bm_max) + bm_max = delta; + if (!bm_min || delta < bm_min) + bm_min = delta; + + /* + * When bm_cnt is greater than UINT_MAX, it breaks the statistics + * accounting. Freeze the statistics when that happens. + * We should have enough data for the avg and stddev anyway. + */ + if (bm_cnt > UINT_MAX) { + scnprintf(bm_str, BENCHMARK_EVENT_STRLEN, + "last=%llu first=%llu max=%llu min=%llu ** avg=%u std=%d std^2=%lld", + bm_last, bm_first, bm_max, bm_min, bm_avg, bm_std, bm_stddev); + return; + } + + bm_total += delta; + bm_totalsq += delta * delta; + + + if (bm_cnt > 1) { + /* + * Apply Welford's method to calculate standard deviation: + * s^2 = 1 / (n * (n-1)) * (n * \Sum (x_i)^2 - (\Sum x_i)^2) + */ + stddev = (u64)bm_cnt * bm_totalsq - bm_total * bm_total; + do_div(stddev, (u32)bm_cnt); + do_div(stddev, (u32)bm_cnt - 1); + } else + stddev = 0; + + delta = bm_total; + do_div(delta, bm_cnt); + avg = delta; + + if (stddev > 0) { + int i = 0; + /* + * stddev is the square of standard deviation but + * we want the actualy number. Use the average + * as our seed to find the std. + * + * The next try is: + * x = (x + N/x) / 2 + * + * Where N is the squared number to find the square + * root of. + */ + seed = avg; + do { + last_seed = seed; + seed = stddev; + if (!last_seed) + break; + do_div(seed, last_seed); + seed += last_seed; + do_div(seed, 2); + } while (i++ < 10 && last_seed != seed); + + std = seed; + } + + scnprintf(bm_str, BENCHMARK_EVENT_STRLEN, + "last=%llu first=%llu max=%llu min=%llu avg=%u std=%d std^2=%lld", + bm_last, bm_first, bm_max, bm_min, avg, std, stddev); + + bm_std = std; + bm_avg = avg; + bm_stddev = stddev; +} + +static int benchmark_event_kthread(void *arg) +{ + /* sleep a bit to make sure the tracepoint gets activated */ + msleep(100); + + while (!kthread_should_stop()) { + + trace_do_benchmark(); + + /* + * We don't go to sleep, but let others + * run as well. + */ + cond_resched(); + } + + return 0; +} + +/* + * When the benchmark tracepoint is enabled, it calls this + * function and the thread that calls the tracepoint is created. + */ +void trace_benchmark_reg(void) +{ + bm_event_thread = kthread_run(benchmark_event_kthread, + NULL, "event_benchmark"); + WARN_ON(!bm_event_thread); +} + +/* + * When the benchmark tracepoint is disabled, it calls this + * function and the thread that calls the tracepoint is deleted + * and all the numbers are reset. + */ +void trace_benchmark_unreg(void) +{ + if (!bm_event_thread) + return; + + kthread_stop(bm_event_thread); + + strcpy(bm_str, "START"); + bm_total = 0; + bm_totalsq = 0; + bm_last = 0; + bm_max = 0; + bm_min = 0; + bm_cnt = 0; + /* These don't need to be reset but reset them anyway */ + bm_first = 0; + bm_std = 0; + bm_avg = 0; + bm_stddev = 0; +} diff --git a/kernel/trace/trace_benchmark.h b/kernel/trace/trace_benchmark.h new file mode 100644 index 000000000000..3c1df1df4e29 --- /dev/null +++ b/kernel/trace/trace_benchmark.h @@ -0,0 +1,41 @@ +#undef TRACE_SYSTEM +#define TRACE_SYSTEM benchmark + +#if !defined(_TRACE_BENCHMARK_H) || defined(TRACE_HEADER_MULTI_READ) +#define _TRACE_BENCHMARK_H + +#include <linux/tracepoint.h> + +extern void trace_benchmark_reg(void); +extern void trace_benchmark_unreg(void); + +#define BENCHMARK_EVENT_STRLEN 128 + +TRACE_EVENT_FN(benchmark_event, + + TP_PROTO(const char *str), + + TP_ARGS(str), + + TP_STRUCT__entry( + __array( char, str, BENCHMARK_EVENT_STRLEN ) + ), + + TP_fast_assign( + memcpy(__entry->str, str, BENCHMARK_EVENT_STRLEN); + ), + + TP_printk("%s", __entry->str), + + trace_benchmark_reg, trace_benchmark_unreg +); + +#endif /* _TRACE_BENCHMARK_H */ + +#undef TRACE_INCLUDE_FILE +#undef TRACE_INCLUDE_PATH +#define TRACE_INCLUDE_PATH . +#define TRACE_INCLUDE_FILE trace_benchmark + +/* This part must be outside protection */ +#include <trace/define_trace.h> diff --git a/kernel/trace/trace_events.c b/kernel/trace/trace_events.c index 3ddfd8f62c05..f99e0b3bca8c 100644 --- a/kernel/trace/trace_events.c +++ b/kernel/trace/trace_events.c @@ -574,6 +574,9 @@ int trace_set_clr_event(const char *system, const char *event, int set) { struct trace_array *tr = top_trace_array(); + if (!tr) + return -ENODEV; + return __ftrace_set_clr_event(tr, NULL, system, event, set); } EXPORT_SYMBOL_GPL(trace_set_clr_event); @@ -2065,6 +2068,9 @@ event_enable_func(struct ftrace_hash *hash, bool enable; int ret; + if (!tr) + return -ENODEV; + /* hash funcs only work with set_ftrace_filter */ if (!enabled || !param) return -EINVAL; @@ -2396,6 +2402,9 @@ static __init int event_trace_enable(void) char *token; int ret; + if (!tr) + return -ENODEV; + for_each_event(iter, __start_ftrace_events, __stop_ftrace_events) { call = *iter; @@ -2442,6 +2451,8 @@ static __init int event_trace_init(void) int ret; tr = top_trace_array(); + if (!tr) + return -ENODEV; d_tracer = tracing_init_dentry(); if (!d_tracer) @@ -2535,6 +2546,8 @@ static __init void event_trace_self_tests(void) int ret; tr = top_trace_array(); + if (!tr) + return; pr_info("Running tests on trace events:\n"); diff --git a/kernel/trace/trace_functions.c b/kernel/trace/trace_functions.c index ffd56351b521..57f0ec962d2c 100644 --- a/kernel/trace/trace_functions.c +++ b/kernel/trace/trace_functions.c @@ -26,8 +26,6 @@ function_trace_call(unsigned long ip, unsigned long parent_ip, static void function_stack_trace_call(unsigned long ip, unsigned long parent_ip, struct ftrace_ops *op, struct pt_regs *pt_regs); -static struct ftrace_ops trace_ops; -static struct ftrace_ops trace_stack_ops; static struct tracer_flags func_flags; /* Our option */ @@ -83,28 +81,24 @@ void ftrace_destroy_function_files(struct trace_array *tr) static int function_trace_init(struct trace_array *tr) { - struct ftrace_ops *ops; - - if (tr->flags & TRACE_ARRAY_FL_GLOBAL) { - /* There's only one global tr */ - if (!trace_ops.private) { - trace_ops.private = tr; - trace_stack_ops.private = tr; - } + ftrace_func_t func; - if (func_flags.val & TRACE_FUNC_OPT_STACK) - ops = &trace_stack_ops; - else - ops = &trace_ops; - tr->ops = ops; - } else if (!tr->ops) { - /* - * Instance trace_arrays get their ops allocated - * at instance creation. Unless it failed - * the allocation. - */ + /* + * Instance trace_arrays get their ops allocated + * at instance creation. Unless it failed + * the allocation. + */ + if (!tr->ops) return -ENOMEM; - } + + /* Currently only the global instance can do stack tracing */ + if (tr->flags & TRACE_ARRAY_FL_GLOBAL && + func_flags.val & TRACE_FUNC_OPT_STACK) + func = function_stack_trace_call; + else + func = function_trace_call; + + ftrace_init_array_ops(tr, func); tr->trace_buffer.cpu = get_cpu(); put_cpu(); @@ -118,6 +112,7 @@ static void function_trace_reset(struct trace_array *tr) { tracing_stop_function_trace(tr); tracing_stop_cmdline_record(); + ftrace_reset_array_ops(tr); } static void function_trace_start(struct trace_array *tr) @@ -199,18 +194,6 @@ function_stack_trace_call(unsigned long ip, unsigned long parent_ip, local_irq_restore(flags); } -static struct ftrace_ops trace_ops __read_mostly = -{ - .func = function_trace_call, - .flags = FTRACE_OPS_FL_GLOBAL | FTRACE_OPS_FL_RECURSION_SAFE, -}; - -static struct ftrace_ops trace_stack_ops __read_mostly = -{ - .func = function_stack_trace_call, - .flags = FTRACE_OPS_FL_GLOBAL | FTRACE_OPS_FL_RECURSION_SAFE, -}; - static struct tracer_opt func_opts[] = { #ifdef CONFIG_STACKTRACE { TRACER_OPT(func_stack_trace, TRACE_FUNC_OPT_STACK) }, @@ -248,10 +231,10 @@ func_set_flag(struct trace_array *tr, u32 old_flags, u32 bit, int set) unregister_ftrace_function(tr->ops); if (set) { - tr->ops = &trace_stack_ops; + tr->ops->func = function_stack_trace_call; register_ftrace_function(tr->ops); } else { - tr->ops = &trace_ops; + tr->ops->func = function_trace_call; register_ftrace_function(tr->ops); } @@ -269,7 +252,6 @@ static struct tracer function_trace __tracer_data = .init = function_trace_init, .reset = function_trace_reset, .start = function_trace_start, - .wait_pipe = poll_wait_pipe, .flags = &func_flags, .set_flag = func_set_flag, .allow_instances = true, diff --git a/kernel/trace/trace_functions_graph.c b/kernel/trace/trace_functions_graph.c index deff11200261..4de3e57f723c 100644 --- a/kernel/trace/trace_functions_graph.c +++ b/kernel/trace/trace_functions_graph.c @@ -38,15 +38,6 @@ struct fgraph_data { #define TRACE_GRAPH_INDENT 2 -/* Flag options */ -#define TRACE_GRAPH_PRINT_OVERRUN 0x1 -#define TRACE_GRAPH_PRINT_CPU 0x2 -#define TRACE_GRAPH_PRINT_OVERHEAD 0x4 -#define TRACE_GRAPH_PRINT_PROC 0x8 -#define TRACE_GRAPH_PRINT_DURATION 0x10 -#define TRACE_GRAPH_PRINT_ABS_TIME 0x20 -#define TRACE_GRAPH_PRINT_IRQS 0x40 - static unsigned int max_depth; static struct tracer_opt trace_opts[] = { @@ -64,11 +55,13 @@ static struct tracer_opt trace_opts[] = { { TRACER_OPT(funcgraph-abstime, TRACE_GRAPH_PRINT_ABS_TIME) }, /* Display interrupts */ { TRACER_OPT(funcgraph-irqs, TRACE_GRAPH_PRINT_IRQS) }, + /* Display function name after trailing } */ + { TRACER_OPT(funcgraph-tail, TRACE_GRAPH_PRINT_TAIL) }, { } /* Empty entry */ }; static struct tracer_flags tracer_flags = { - /* Don't display overruns and proc by default */ + /* Don't display overruns, proc, or tail by default */ .val = TRACE_GRAPH_PRINT_CPU | TRACE_GRAPH_PRINT_OVERHEAD | TRACE_GRAPH_PRINT_DURATION | TRACE_GRAPH_PRINT_IRQS, .opts = trace_opts @@ -1176,9 +1169,10 @@ print_graph_return(struct ftrace_graph_ret *trace, struct trace_seq *s, * If the return function does not have a matching entry, * then the entry was lost. Instead of just printing * the '}' and letting the user guess what function this - * belongs to, write out the function name. + * belongs to, write out the function name. Always do + * that if the funcgraph-tail option is enabled. */ - if (func_match) { + if (func_match && !(flags & TRACE_GRAPH_PRINT_TAIL)) { ret = trace_seq_puts(s, "}\n"); if (!ret) return TRACE_TYPE_PARTIAL_LINE; @@ -1505,7 +1499,6 @@ static struct tracer graph_trace __tracer_data = { .pipe_open = graph_trace_open, .close = graph_trace_close, .pipe_close = graph_trace_close, - .wait_pipe = poll_wait_pipe, .init = graph_trace_init, .reset = graph_trace_reset, .print_line = print_graph_function, diff --git a/kernel/trace/trace_irqsoff.c b/kernel/trace/trace_irqsoff.c index 8ff02cbb892f..9bb104f748d0 100644 --- a/kernel/trace/trace_irqsoff.c +++ b/kernel/trace/trace_irqsoff.c @@ -151,12 +151,6 @@ irqsoff_tracer_call(unsigned long ip, unsigned long parent_ip, atomic_dec(&data->disabled); } - -static struct ftrace_ops trace_ops __read_mostly = -{ - .func = irqsoff_tracer_call, - .flags = FTRACE_OPS_FL_GLOBAL | FTRACE_OPS_FL_RECURSION_SAFE, -}; #endif /* CONFIG_FUNCTION_TRACER */ #ifdef CONFIG_FUNCTION_GRAPH_TRACER @@ -176,7 +170,7 @@ irqsoff_set_flag(struct trace_array *tr, u32 old_flags, u32 bit, int set) for_each_possible_cpu(cpu) per_cpu(tracing_cpu, cpu) = 0; - tracing_max_latency = 0; + tr->max_latency = 0; tracing_reset_online_cpus(&irqsoff_trace->trace_buffer); return start_irqsoff_tracer(irqsoff_trace, set); @@ -303,13 +297,13 @@ static void irqsoff_print_header(struct seq_file *s) /* * Should this new latency be reported/recorded? */ -static int report_latency(cycle_t delta) +static int report_latency(struct trace_array *tr, cycle_t delta) { if (tracing_thresh) { if (delta < tracing_thresh) return 0; } else { - if (delta <= tracing_max_latency) + if (delta <= tr->max_latency) return 0; } return 1; @@ -333,13 +327,13 @@ check_critical_timing(struct trace_array *tr, pc = preempt_count(); - if (!report_latency(delta)) + if (!report_latency(tr, delta)) goto out; raw_spin_lock_irqsave(&max_trace_lock, flags); /* check if we are still the max latency */ - if (!report_latency(delta)) + if (!report_latency(tr, delta)) goto out_unlock; __trace_function(tr, CALLER_ADDR0, parent_ip, flags, pc); @@ -352,7 +346,7 @@ check_critical_timing(struct trace_array *tr, data->critical_end = parent_ip; if (likely(!is_tracing_stopped())) { - tracing_max_latency = delta; + tr->max_latency = delta; update_max_tr_single(tr, current, cpu); } @@ -531,7 +525,7 @@ void trace_preempt_off(unsigned long a0, unsigned long a1) } #endif /* CONFIG_PREEMPT_TRACER */ -static int register_irqsoff_function(int graph, int set) +static int register_irqsoff_function(struct trace_array *tr, int graph, int set) { int ret; @@ -543,7 +537,7 @@ static int register_irqsoff_function(int graph, int set) ret = register_ftrace_graph(&irqsoff_graph_return, &irqsoff_graph_entry); else - ret = register_ftrace_function(&trace_ops); + ret = register_ftrace_function(tr->ops); if (!ret) function_enabled = true; @@ -551,7 +545,7 @@ static int register_irqsoff_function(int graph, int set) return ret; } -static void unregister_irqsoff_function(int graph) +static void unregister_irqsoff_function(struct trace_array *tr, int graph) { if (!function_enabled) return; @@ -559,17 +553,17 @@ static void unregister_irqsoff_function(int graph) if (graph) unregister_ftrace_graph(); else - unregister_ftrace_function(&trace_ops); + unregister_ftrace_function(tr->ops); function_enabled = false; } -static void irqsoff_function_set(int set) +static void irqsoff_function_set(struct trace_array *tr, int set) { if (set) - register_irqsoff_function(is_graph(), 1); + register_irqsoff_function(tr, is_graph(), 1); else - unregister_irqsoff_function(is_graph()); + unregister_irqsoff_function(tr, is_graph()); } static int irqsoff_flag_changed(struct trace_array *tr, u32 mask, int set) @@ -577,7 +571,7 @@ static int irqsoff_flag_changed(struct trace_array *tr, u32 mask, int set) struct tracer *tracer = tr->current_trace; if (mask & TRACE_ITER_FUNCTION) - irqsoff_function_set(set); + irqsoff_function_set(tr, set); return trace_keep_overwrite(tracer, mask, set); } @@ -586,7 +580,7 @@ static int start_irqsoff_tracer(struct trace_array *tr, int graph) { int ret; - ret = register_irqsoff_function(graph, 0); + ret = register_irqsoff_function(tr, graph, 0); if (!ret && tracing_is_enabled()) tracer_enabled = 1; @@ -600,25 +594,37 @@ static void stop_irqsoff_tracer(struct trace_array *tr, int graph) { tracer_enabled = 0; - unregister_irqsoff_function(graph); + unregister_irqsoff_function(tr, graph); } -static void __irqsoff_tracer_init(struct trace_array *tr) +static bool irqsoff_busy; + +static int __irqsoff_tracer_init(struct trace_array *tr) { + if (irqsoff_busy) + return -EBUSY; + save_flags = trace_flags; /* non overwrite screws up the latency tracers */ set_tracer_flag(tr, TRACE_ITER_OVERWRITE, 1); set_tracer_flag(tr, TRACE_ITER_LATENCY_FMT, 1); - tracing_max_latency = 0; + tr->max_latency = 0; irqsoff_trace = tr; /* make sure that the tracer is visible */ smp_wmb(); tracing_reset_online_cpus(&tr->trace_buffer); - if (start_irqsoff_tracer(tr, is_graph())) + ftrace_init_array_ops(tr, irqsoff_tracer_call); + + /* Only toplevel instance supports graph tracing */ + if (start_irqsoff_tracer(tr, (tr->flags & TRACE_ARRAY_FL_GLOBAL && + is_graph()))) printk(KERN_ERR "failed to start irqsoff tracer\n"); + + irqsoff_busy = true; + return 0; } static void irqsoff_tracer_reset(struct trace_array *tr) @@ -630,6 +636,9 @@ static void irqsoff_tracer_reset(struct trace_array *tr) set_tracer_flag(tr, TRACE_ITER_LATENCY_FMT, lat_flag); set_tracer_flag(tr, TRACE_ITER_OVERWRITE, overwrite_flag); + ftrace_reset_array_ops(tr); + + irqsoff_busy = false; } static void irqsoff_tracer_start(struct trace_array *tr) @@ -647,8 +656,7 @@ static int irqsoff_tracer_init(struct trace_array *tr) { trace_type = TRACER_IRQS_OFF; - __irqsoff_tracer_init(tr); - return 0; + return __irqsoff_tracer_init(tr); } static struct tracer irqsoff_tracer __read_mostly = { @@ -668,6 +676,7 @@ static struct tracer irqsoff_tracer __read_mostly = #endif .open = irqsoff_trace_open, .close = irqsoff_trace_close, + .allow_instances = true, .use_max_tr = true, }; # define register_irqsoff(trace) register_tracer(&trace) @@ -680,8 +689,7 @@ static int preemptoff_tracer_init(struct trace_array *tr) { trace_type = TRACER_PREEMPT_OFF; - __irqsoff_tracer_init(tr); - return 0; + return __irqsoff_tracer_init(tr); } static struct tracer preemptoff_tracer __read_mostly = @@ -702,6 +710,7 @@ static struct tracer preemptoff_tracer __read_mostly = #endif .open = irqsoff_trace_open, .close = irqsoff_trace_close, + .allow_instances = true, .use_max_tr = true, }; # define register_preemptoff(trace) register_tracer(&trace) @@ -716,8 +725,7 @@ static int preemptirqsoff_tracer_init(struct trace_array *tr) { trace_type = TRACER_IRQS_OFF | TRACER_PREEMPT_OFF; - __irqsoff_tracer_init(tr); - return 0; + return __irqsoff_tracer_init(tr); } static struct tracer preemptirqsoff_tracer __read_mostly = @@ -738,6 +746,7 @@ static struct tracer preemptirqsoff_tracer __read_mostly = #endif .open = irqsoff_trace_open, .close = irqsoff_trace_close, + .allow_instances = true, .use_max_tr = true, }; diff --git a/kernel/trace/trace_kprobe.c b/kernel/trace/trace_kprobe.c index 903ae28962be..ef2fba1f46b5 100644 --- a/kernel/trace/trace_kprobe.c +++ b/kernel/trace/trace_kprobe.c @@ -1377,6 +1377,9 @@ static __init int kprobe_trace_self_tests_init(void) struct trace_kprobe *tk; struct ftrace_event_file *file; + if (tracing_is_disabled()) + return -ENODEV; + target = kprobe_trace_selftest_target; pr_info("Testing kprobe tracing: "); diff --git a/kernel/trace/trace_nop.c b/kernel/trace/trace_nop.c index 69a5cc94c01a..fcf0a9e48916 100644 --- a/kernel/trace/trace_nop.c +++ b/kernel/trace/trace_nop.c @@ -91,7 +91,6 @@ struct tracer nop_trace __read_mostly = .name = "nop", .init = nop_trace_init, .reset = nop_trace_reset, - .wait_pipe = poll_wait_pipe, #ifdef CONFIG_FTRACE_SELFTEST .selftest = trace_selftest_startup_nop, #endif diff --git a/kernel/trace/trace_output.c b/kernel/trace/trace_output.c index a436de18aa99..f3dad80c20b2 100644 --- a/kernel/trace/trace_output.c +++ b/kernel/trace/trace_output.c @@ -126,6 +126,34 @@ trace_seq_printf(struct trace_seq *s, const char *fmt, ...) EXPORT_SYMBOL_GPL(trace_seq_printf); /** + * trace_seq_bitmask - put a list of longs as a bitmask print output + * @s: trace sequence descriptor + * @maskp: points to an array of unsigned longs that represent a bitmask + * @nmaskbits: The number of bits that are valid in @maskp + * + * It returns 0 if the trace oversizes the buffer's free + * space, 1 otherwise. + * + * Writes a ASCII representation of a bitmask string into @s. + */ +int +trace_seq_bitmask(struct trace_seq *s, const unsigned long *maskp, + int nmaskbits) +{ + int len = (PAGE_SIZE - 1) - s->len; + int ret; + + if (s->full || !len) + return 0; + + ret = bitmap_scnprintf(s->buffer, len, maskp, nmaskbits); + s->len += ret; + + return 1; +} +EXPORT_SYMBOL_GPL(trace_seq_bitmask); + +/** * trace_seq_vprintf - sequence printing of trace information * @s: trace sequence descriptor * @fmt: printf format string @@ -399,6 +427,19 @@ EXPORT_SYMBOL(ftrace_print_symbols_seq_u64); #endif const char * +ftrace_print_bitmask_seq(struct trace_seq *p, void *bitmask_ptr, + unsigned int bitmask_size) +{ + const char *ret = p->buffer + p->len; + + trace_seq_bitmask(p, bitmask_ptr, bitmask_size * 8); + trace_seq_putc(p, 0); + + return ret; +} +EXPORT_SYMBOL_GPL(ftrace_print_bitmask_seq); + +const char * ftrace_print_hex_seq(struct trace_seq *p, const unsigned char *buf, int buf_len) { int i; diff --git a/kernel/trace/trace_sched_wakeup.c b/kernel/trace/trace_sched_wakeup.c index e14da5e97a69..19bd8928ce94 100644 --- a/kernel/trace/trace_sched_wakeup.c +++ b/kernel/trace/trace_sched_wakeup.c @@ -130,15 +130,9 @@ wakeup_tracer_call(unsigned long ip, unsigned long parent_ip, atomic_dec(&data->disabled); preempt_enable_notrace(); } - -static struct ftrace_ops trace_ops __read_mostly = -{ - .func = wakeup_tracer_call, - .flags = FTRACE_OPS_FL_GLOBAL | FTRACE_OPS_FL_RECURSION_SAFE, -}; #endif /* CONFIG_FUNCTION_TRACER */ -static int register_wakeup_function(int graph, int set) +static int register_wakeup_function(struct trace_array *tr, int graph, int set) { int ret; @@ -150,7 +144,7 @@ static int register_wakeup_function(int graph, int set) ret = register_ftrace_graph(&wakeup_graph_return, &wakeup_graph_entry); else - ret = register_ftrace_function(&trace_ops); + ret = register_ftrace_function(tr->ops); if (!ret) function_enabled = true; @@ -158,7 +152,7 @@ static int register_wakeup_function(int graph, int set) return ret; } -static void unregister_wakeup_function(int graph) +static void unregister_wakeup_function(struct trace_array *tr, int graph) { if (!function_enabled) return; @@ -166,17 +160,17 @@ static void unregister_wakeup_function(int graph) if (graph) unregister_ftrace_graph(); else - unregister_ftrace_function(&trace_ops); + unregister_ftrace_function(tr->ops); function_enabled = false; } -static void wakeup_function_set(int set) +static void wakeup_function_set(struct trace_array *tr, int set) { if (set) - register_wakeup_function(is_graph(), 1); + register_wakeup_function(tr, is_graph(), 1); else - unregister_wakeup_function(is_graph()); + unregister_wakeup_function(tr, is_graph()); } static int wakeup_flag_changed(struct trace_array *tr, u32 mask, int set) @@ -184,16 +178,16 @@ static int wakeup_flag_changed(struct trace_array *tr, u32 mask, int set) struct tracer *tracer = tr->current_trace; if (mask & TRACE_ITER_FUNCTION) - wakeup_function_set(set); + wakeup_function_set(tr, set); return trace_keep_overwrite(tracer, mask, set); } -static int start_func_tracer(int graph) +static int start_func_tracer(struct trace_array *tr, int graph) { int ret; - ret = register_wakeup_function(graph, 0); + ret = register_wakeup_function(tr, graph, 0); if (!ret && tracing_is_enabled()) tracer_enabled = 1; @@ -203,11 +197,11 @@ static int start_func_tracer(int graph) return ret; } -static void stop_func_tracer(int graph) +static void stop_func_tracer(struct trace_array *tr, int graph) { tracer_enabled = 0; - unregister_wakeup_function(graph); + unregister_wakeup_function(tr, graph); } #ifdef CONFIG_FUNCTION_GRAPH_TRACER @@ -221,12 +215,12 @@ wakeup_set_flag(struct trace_array *tr, u32 old_flags, u32 bit, int set) if (!(is_graph() ^ set)) return 0; - stop_func_tracer(!set); + stop_func_tracer(tr, !set); wakeup_reset(wakeup_trace); - tracing_max_latency = 0; + tr->max_latency = 0; - return start_func_tracer(set); + return start_func_tracer(tr, set); } static int wakeup_graph_entry(struct ftrace_graph_ent *trace) @@ -350,13 +344,13 @@ static void wakeup_print_header(struct seq_file *s) /* * Should this new latency be reported/recorded? */ -static int report_latency(cycle_t delta) +static int report_latency(struct trace_array *tr, cycle_t delta) { if (tracing_thresh) { if (delta < tracing_thresh) return 0; } else { - if (delta <= tracing_max_latency) + if (delta <= tr->max_latency) return 0; } return 1; @@ -424,11 +418,11 @@ probe_wakeup_sched_switch(void *ignore, T1 = ftrace_now(cpu); delta = T1-T0; - if (!report_latency(delta)) + if (!report_latency(wakeup_trace, delta)) goto out_unlock; if (likely(!is_tracing_stopped())) { - tracing_max_latency = delta; + wakeup_trace->max_latency = delta; update_max_tr(wakeup_trace, wakeup_task, wakeup_cpu); } @@ -587,7 +581,7 @@ static void start_wakeup_tracer(struct trace_array *tr) */ smp_wmb(); - if (start_func_tracer(is_graph())) + if (start_func_tracer(tr, is_graph())) printk(KERN_ERR "failed to start wakeup tracer\n"); return; @@ -600,13 +594,15 @@ fail_deprobe: static void stop_wakeup_tracer(struct trace_array *tr) { tracer_enabled = 0; - stop_func_tracer(is_graph()); + stop_func_tracer(tr, is_graph()); unregister_trace_sched_switch(probe_wakeup_sched_switch, NULL); unregister_trace_sched_wakeup_new(probe_wakeup, NULL); unregister_trace_sched_wakeup(probe_wakeup, NULL); unregister_trace_sched_migrate_task(probe_wakeup_migrate_task, NULL); } +static bool wakeup_busy; + static int __wakeup_tracer_init(struct trace_array *tr) { save_flags = trace_flags; @@ -615,14 +611,20 @@ static int __wakeup_tracer_init(struct trace_array *tr) set_tracer_flag(tr, TRACE_ITER_OVERWRITE, 1); set_tracer_flag(tr, TRACE_ITER_LATENCY_FMT, 1); - tracing_max_latency = 0; + tr->max_latency = 0; wakeup_trace = tr; + ftrace_init_array_ops(tr, wakeup_tracer_call); start_wakeup_tracer(tr); + + wakeup_busy = true; return 0; } static int wakeup_tracer_init(struct trace_array *tr) { + if (wakeup_busy) + return -EBUSY; + wakeup_dl = 0; wakeup_rt = 0; return __wakeup_tracer_init(tr); @@ -630,6 +632,9 @@ static int wakeup_tracer_init(struct trace_array *tr) static int wakeup_rt_tracer_init(struct trace_array *tr) { + if (wakeup_busy) + return -EBUSY; + wakeup_dl = 0; wakeup_rt = 1; return __wakeup_tracer_init(tr); @@ -637,6 +642,9 @@ static int wakeup_rt_tracer_init(struct trace_array *tr) static int wakeup_dl_tracer_init(struct trace_array *tr) { + if (wakeup_busy) + return -EBUSY; + wakeup_dl = 1; wakeup_rt = 0; return __wakeup_tracer_init(tr); @@ -653,6 +661,8 @@ static void wakeup_tracer_reset(struct trace_array *tr) set_tracer_flag(tr, TRACE_ITER_LATENCY_FMT, lat_flag); set_tracer_flag(tr, TRACE_ITER_OVERWRITE, overwrite_flag); + ftrace_reset_array_ops(tr); + wakeup_busy = false; } static void wakeup_tracer_start(struct trace_array *tr) @@ -684,6 +694,7 @@ static struct tracer wakeup_tracer __read_mostly = #endif .open = wakeup_trace_open, .close = wakeup_trace_close, + .allow_instances = true, .use_max_tr = true, }; @@ -694,7 +705,6 @@ static struct tracer wakeup_rt_tracer __read_mostly = .reset = wakeup_tracer_reset, .start = wakeup_tracer_start, .stop = wakeup_tracer_stop, - .wait_pipe = poll_wait_pipe, .print_max = true, .print_header = wakeup_print_header, .print_line = wakeup_print_line, @@ -706,6 +716,7 @@ static struct tracer wakeup_rt_tracer __read_mostly = #endif .open = wakeup_trace_open, .close = wakeup_trace_close, + .allow_instances = true, .use_max_tr = true, }; @@ -716,7 +727,6 @@ static struct tracer wakeup_dl_tracer __read_mostly = .reset = wakeup_tracer_reset, .start = wakeup_tracer_start, .stop = wakeup_tracer_stop, - .wait_pipe = poll_wait_pipe, .print_max = true, .print_header = wakeup_print_header, .print_line = wakeup_print_line, diff --git a/kernel/trace/trace_selftest.c b/kernel/trace/trace_selftest.c index e98fca60974f..5ef60499dc8e 100644 --- a/kernel/trace/trace_selftest.c +++ b/kernel/trace/trace_selftest.c @@ -65,7 +65,7 @@ static int trace_test_buffer(struct trace_buffer *buf, unsigned long *count) /* Don't allow flipping of max traces now */ local_irq_save(flags); - arch_spin_lock(&ftrace_max_lock); + arch_spin_lock(&buf->tr->max_lock); cnt = ring_buffer_entries(buf->buffer); @@ -83,7 +83,7 @@ static int trace_test_buffer(struct trace_buffer *buf, unsigned long *count) break; } tracing_on(); - arch_spin_unlock(&ftrace_max_lock); + arch_spin_unlock(&buf->tr->max_lock); local_irq_restore(flags); if (count) @@ -161,11 +161,6 @@ static struct ftrace_ops test_probe3 = { .flags = FTRACE_OPS_FL_RECURSION_SAFE, }; -static struct ftrace_ops test_global = { - .func = trace_selftest_test_global_func, - .flags = FTRACE_OPS_FL_GLOBAL | FTRACE_OPS_FL_RECURSION_SAFE, -}; - static void print_counts(void) { printk("(%d %d %d %d %d) ", @@ -185,7 +180,7 @@ static void reset_counts(void) trace_selftest_test_dyn_cnt = 0; } -static int trace_selftest_ops(int cnt) +static int trace_selftest_ops(struct trace_array *tr, int cnt) { int save_ftrace_enabled = ftrace_enabled; struct ftrace_ops *dyn_ops; @@ -220,7 +215,11 @@ static int trace_selftest_ops(int cnt) register_ftrace_function(&test_probe1); register_ftrace_function(&test_probe2); register_ftrace_function(&test_probe3); - register_ftrace_function(&test_global); + /* First time we are running with main function */ + if (cnt > 1) { + ftrace_init_array_ops(tr, trace_selftest_test_global_func); + register_ftrace_function(tr->ops); + } DYN_FTRACE_TEST_NAME(); @@ -232,8 +231,10 @@ static int trace_selftest_ops(int cnt) goto out; if (trace_selftest_test_probe3_cnt != 1) goto out; - if (trace_selftest_test_global_cnt == 0) - goto out; + if (cnt > 1) { + if (trace_selftest_test_global_cnt == 0) + goto out; + } DYN_FTRACE_TEST_NAME2(); @@ -269,8 +270,10 @@ static int trace_selftest_ops(int cnt) goto out_free; if (trace_selftest_test_probe3_cnt != 3) goto out_free; - if (trace_selftest_test_global_cnt == 0) - goto out; + if (cnt > 1) { + if (trace_selftest_test_global_cnt == 0) + goto out; + } if (trace_selftest_test_dyn_cnt == 0) goto out_free; @@ -295,7 +298,9 @@ static int trace_selftest_ops(int cnt) unregister_ftrace_function(&test_probe1); unregister_ftrace_function(&test_probe2); unregister_ftrace_function(&test_probe3); - unregister_ftrace_function(&test_global); + if (cnt > 1) + unregister_ftrace_function(tr->ops); + ftrace_reset_array_ops(tr); /* Make sure everything is off */ reset_counts(); @@ -315,9 +320,9 @@ static int trace_selftest_ops(int cnt) } /* Test dynamic code modification and ftrace filters */ -int trace_selftest_startup_dynamic_tracing(struct tracer *trace, - struct trace_array *tr, - int (*func)(void)) +static int trace_selftest_startup_dynamic_tracing(struct tracer *trace, + struct trace_array *tr, + int (*func)(void)) { int save_ftrace_enabled = ftrace_enabled; unsigned long count; @@ -388,7 +393,7 @@ int trace_selftest_startup_dynamic_tracing(struct tracer *trace, } /* Test the ops with global tracing running */ - ret = trace_selftest_ops(1); + ret = trace_selftest_ops(tr, 1); trace->reset(tr); out: @@ -399,7 +404,7 @@ int trace_selftest_startup_dynamic_tracing(struct tracer *trace, /* Test the ops with global tracing off */ if (!ret) - ret = trace_selftest_ops(2); + ret = trace_selftest_ops(tr, 2); return ret; } @@ -802,7 +807,7 @@ out: int trace_selftest_startup_irqsoff(struct tracer *trace, struct trace_array *tr) { - unsigned long save_max = tracing_max_latency; + unsigned long save_max = tr->max_latency; unsigned long count; int ret; @@ -814,7 +819,7 @@ trace_selftest_startup_irqsoff(struct tracer *trace, struct trace_array *tr) } /* reset the max latency */ - tracing_max_latency = 0; + tr->max_latency = 0; /* disable interrupts for a bit */ local_irq_disable(); udelay(100); @@ -841,7 +846,7 @@ trace_selftest_startup_irqsoff(struct tracer *trace, struct trace_array *tr) ret = -1; } - tracing_max_latency = save_max; + tr->max_latency = save_max; return ret; } @@ -851,7 +856,7 @@ trace_selftest_startup_irqsoff(struct tracer *trace, struct trace_array *tr) int trace_selftest_startup_preemptoff(struct tracer *trace, struct trace_array *tr) { - unsigned long save_max = tracing_max_latency; + unsigned long save_max = tr->max_latency; unsigned long count; int ret; @@ -876,7 +881,7 @@ trace_selftest_startup_preemptoff(struct tracer *trace, struct trace_array *tr) } /* reset the max latency */ - tracing_max_latency = 0; + tr->max_latency = 0; /* disable preemption for a bit */ preempt_disable(); udelay(100); @@ -903,7 +908,7 @@ trace_selftest_startup_preemptoff(struct tracer *trace, struct trace_array *tr) ret = -1; } - tracing_max_latency = save_max; + tr->max_latency = save_max; return ret; } @@ -913,7 +918,7 @@ trace_selftest_startup_preemptoff(struct tracer *trace, struct trace_array *tr) int trace_selftest_startup_preemptirqsoff(struct tracer *trace, struct trace_array *tr) { - unsigned long save_max = tracing_max_latency; + unsigned long save_max = tr->max_latency; unsigned long count; int ret; @@ -938,7 +943,7 @@ trace_selftest_startup_preemptirqsoff(struct tracer *trace, struct trace_array * } /* reset the max latency */ - tracing_max_latency = 0; + tr->max_latency = 0; /* disable preemption and interrupts for a bit */ preempt_disable(); @@ -973,7 +978,7 @@ trace_selftest_startup_preemptirqsoff(struct tracer *trace, struct trace_array * } /* do the test by disabling interrupts first this time */ - tracing_max_latency = 0; + tr->max_latency = 0; tracing_start(); trace->start(tr); @@ -1004,7 +1009,7 @@ out: tracing_start(); out_no_start: trace->reset(tr); - tracing_max_latency = save_max; + tr->max_latency = save_max; return ret; } @@ -1057,7 +1062,7 @@ static int trace_wakeup_test_thread(void *data) int trace_selftest_startup_wakeup(struct tracer *trace, struct trace_array *tr) { - unsigned long save_max = tracing_max_latency; + unsigned long save_max = tr->max_latency; struct task_struct *p; struct completion is_ready; unsigned long count; @@ -1083,7 +1088,7 @@ trace_selftest_startup_wakeup(struct tracer *trace, struct trace_array *tr) } /* reset the max latency */ - tracing_max_latency = 0; + tr->max_latency = 0; while (p->on_rq) { /* @@ -1113,7 +1118,7 @@ trace_selftest_startup_wakeup(struct tracer *trace, struct trace_array *tr) trace->reset(tr); tracing_start(); - tracing_max_latency = save_max; + tr->max_latency = save_max; /* kill the thread */ kthread_stop(p); diff --git a/kernel/trace/trace_stack.c b/kernel/trace/trace_stack.c index 21b320e5d163..8a4e5cb66a4c 100644 --- a/kernel/trace/trace_stack.c +++ b/kernel/trace/trace_stack.c @@ -51,11 +51,33 @@ static DEFINE_MUTEX(stack_sysctl_mutex); int stack_tracer_enabled; static int last_stack_tracer_enabled; +static inline void print_max_stack(void) +{ + long i; + int size; + + pr_emerg(" Depth Size Location (%d entries)\n" + " ----- ---- --------\n", + max_stack_trace.nr_entries - 1); + + for (i = 0; i < max_stack_trace.nr_entries; i++) { + if (stack_dump_trace[i] == ULONG_MAX) + break; + if (i+1 == max_stack_trace.nr_entries || + stack_dump_trace[i+1] == ULONG_MAX) + size = stack_dump_index[i]; + else + size = stack_dump_index[i] - stack_dump_index[i+1]; + + pr_emerg("%3ld) %8d %5d %pS\n", i, stack_dump_index[i], + size, (void *)stack_dump_trace[i]); + } +} + static inline void check_stack(unsigned long ip, unsigned long *stack) { - unsigned long this_size, flags; - unsigned long *p, *top, *start; + unsigned long this_size, flags; unsigned long *p, *top, *start; static int tracer_frame; int frame_size = ACCESS_ONCE(tracer_frame); int i; @@ -85,8 +107,12 @@ check_stack(unsigned long ip, unsigned long *stack) max_stack_size = this_size; - max_stack_trace.nr_entries = 0; - max_stack_trace.skip = 3; + max_stack_trace.nr_entries = 0; + + if (using_ftrace_ops_list_func()) + max_stack_trace.skip = 4; + else + max_stack_trace.skip = 3; save_stack_trace(&max_stack_trace); @@ -145,8 +171,12 @@ check_stack(unsigned long ip, unsigned long *stack) i++; } - BUG_ON(current != &init_task && - *(end_of_stack(current)) != STACK_END_MAGIC); + if ((current != &init_task && + *(end_of_stack(current)) != STACK_END_MAGIC)) { + print_max_stack(); + BUG(); + } + out: arch_spin_unlock(&max_stack_lock); local_irq_restore(flags); diff --git a/kernel/workqueue.c b/kernel/workqueue.c index a4bab46cd38e..6203d2900877 100644 --- a/kernel/workqueue.c +++ b/kernel/workqueue.c @@ -65,15 +65,12 @@ enum { * be executing on any CPU. The pool behaves as an unbound one. * * Note that DISASSOCIATED should be flipped only while holding - * manager_mutex to avoid changing binding state while - * create_worker() is in progress. + * attach_mutex to avoid changing binding state while + * worker_attach_to_pool() is in progress. */ - POOL_MANAGE_WORKERS = 1 << 0, /* need to manage workers */ POOL_DISASSOCIATED = 1 << 2, /* cpu can't serve workers */ - POOL_FREEZING = 1 << 3, /* freeze in progress */ /* worker flags */ - WORKER_STARTED = 1 << 0, /* started */ WORKER_DIE = 1 << 1, /* die die die */ WORKER_IDLE = 1 << 2, /* is idle */ WORKER_PREP = 1 << 3, /* preparing to run works */ @@ -124,8 +121,7 @@ enum { * cpu or grabbing pool->lock is enough for read access. If * POOL_DISASSOCIATED is set, it's identical to L. * - * MG: pool->manager_mutex and pool->lock protected. Writes require both - * locks. Reads can happen under either lock. + * A: pool->attach_mutex protected. * * PL: wq_pool_mutex protected. * @@ -163,8 +159,11 @@ struct worker_pool { /* see manage_workers() for details on the two manager mutexes */ struct mutex manager_arb; /* manager arbitration */ - struct mutex manager_mutex; /* manager exclusion */ - struct idr worker_idr; /* MG: worker IDs and iteration */ + struct mutex attach_mutex; /* attach/detach exclusion */ + struct list_head workers; /* A: attached workers */ + struct completion *detach_completion; /* all workers detached */ + + struct ida worker_ida; /* worker IDs for task name */ struct workqueue_attrs *attrs; /* I: worker attributes */ struct hlist_node hash_node; /* PL: unbound_pool_hash node */ @@ -340,16 +339,6 @@ static void copy_workqueue_attrs(struct workqueue_attrs *to, lockdep_is_held(&wq->mutex), \ "sched RCU or wq->mutex should be held") -#ifdef CONFIG_LOCKDEP -#define assert_manager_or_pool_lock(pool) \ - WARN_ONCE(debug_locks && \ - !lockdep_is_held(&(pool)->manager_mutex) && \ - !lockdep_is_held(&(pool)->lock), \ - "pool->manager_mutex or ->lock should be held") -#else -#define assert_manager_or_pool_lock(pool) do { } while (0) -#endif - #define for_each_cpu_worker_pool(pool, cpu) \ for ((pool) = &per_cpu(cpu_worker_pools, cpu)[0]; \ (pool) < &per_cpu(cpu_worker_pools, cpu)[NR_STD_WORKER_POOLS]; \ @@ -375,17 +364,16 @@ static void copy_workqueue_attrs(struct workqueue_attrs *to, /** * for_each_pool_worker - iterate through all workers of a worker_pool * @worker: iteration cursor - * @wi: integer used for iteration * @pool: worker_pool to iterate workers of * - * This must be called with either @pool->manager_mutex or ->lock held. + * This must be called with @pool->attach_mutex. * * The if/else clause exists only for the lockdep assertion and can be * ignored. */ -#define for_each_pool_worker(worker, wi, pool) \ - idr_for_each_entry(&(pool)->worker_idr, (worker), (wi)) \ - if (({ assert_manager_or_pool_lock((pool)); false; })) { } \ +#define for_each_pool_worker(worker, pool) \ + list_for_each_entry((worker), &(pool)->workers, node) \ + if (({ lockdep_assert_held(&pool->attach_mutex); false; })) { } \ else /** @@ -763,13 +751,6 @@ static bool need_to_create_worker(struct worker_pool *pool) return need_more_worker(pool) && !may_start_working(pool); } -/* Do I need to be the manager? */ -static bool need_to_manage_workers(struct worker_pool *pool) -{ - return need_to_create_worker(pool) || - (pool->flags & POOL_MANAGE_WORKERS); -} - /* Do we have too many workers and should some go away? */ static bool too_many_workers(struct worker_pool *pool) { @@ -791,8 +772,8 @@ static bool too_many_workers(struct worker_pool *pool) * Wake up functions. */ -/* Return the first worker. Safe with preemption disabled */ -static struct worker *first_worker(struct worker_pool *pool) +/* Return the first idle worker. Safe with preemption disabled */ +static struct worker *first_idle_worker(struct worker_pool *pool) { if (unlikely(list_empty(&pool->idle_list))) return NULL; @@ -811,7 +792,7 @@ static struct worker *first_worker(struct worker_pool *pool) */ static void wake_up_worker(struct worker_pool *pool) { - struct worker *worker = first_worker(pool); + struct worker *worker = first_idle_worker(pool); if (likely(worker)) wake_up_process(worker->task); @@ -885,7 +866,7 @@ struct task_struct *wq_worker_sleeping(struct task_struct *task, int cpu) */ if (atomic_dec_and_test(&pool->nr_running) && !list_empty(&pool->worklist)) - to_wakeup = first_worker(pool); + to_wakeup = first_idle_worker(pool); return to_wakeup ? to_wakeup->task : NULL; } @@ -1621,70 +1602,6 @@ static void worker_leave_idle(struct worker *worker) list_del_init(&worker->entry); } -/** - * worker_maybe_bind_and_lock - try to bind %current to worker_pool and lock it - * @pool: target worker_pool - * - * Bind %current to the cpu of @pool if it is associated and lock @pool. - * - * Works which are scheduled while the cpu is online must at least be - * scheduled to a worker which is bound to the cpu so that if they are - * flushed from cpu callbacks while cpu is going down, they are - * guaranteed to execute on the cpu. - * - * This function is to be used by unbound workers and rescuers to bind - * themselves to the target cpu and may race with cpu going down or - * coming online. kthread_bind() can't be used because it may put the - * worker to already dead cpu and set_cpus_allowed_ptr() can't be used - * verbatim as it's best effort and blocking and pool may be - * [dis]associated in the meantime. - * - * This function tries set_cpus_allowed() and locks pool and verifies the - * binding against %POOL_DISASSOCIATED which is set during - * %CPU_DOWN_PREPARE and cleared during %CPU_ONLINE, so if the worker - * enters idle state or fetches works without dropping lock, it can - * guarantee the scheduling requirement described in the first paragraph. - * - * CONTEXT: - * Might sleep. Called without any lock but returns with pool->lock - * held. - * - * Return: - * %true if the associated pool is online (@worker is successfully - * bound), %false if offline. - */ -static bool worker_maybe_bind_and_lock(struct worker_pool *pool) -__acquires(&pool->lock) -{ - while (true) { - /* - * The following call may fail, succeed or succeed - * without actually migrating the task to the cpu if - * it races with cpu hotunplug operation. Verify - * against POOL_DISASSOCIATED. - */ - if (!(pool->flags & POOL_DISASSOCIATED)) - set_cpus_allowed_ptr(current, pool->attrs->cpumask); - - spin_lock_irq(&pool->lock); - if (pool->flags & POOL_DISASSOCIATED) - return false; - if (task_cpu(current) == pool->cpu && - cpumask_equal(¤t->cpus_allowed, pool->attrs->cpumask)) - return true; - spin_unlock_irq(&pool->lock); - - /* - * We've raced with CPU hot[un]plug. Give it a breather - * and retry migration. cond_resched() is required here; - * otherwise, we might deadlock against cpu_stop trying to - * bring down the CPU on non-preemptive kernel. - */ - cpu_relax(); - cond_resched(); - } -} - static struct worker *alloc_worker(void) { struct worker *worker; @@ -1693,6 +1610,7 @@ static struct worker *alloc_worker(void) if (worker) { INIT_LIST_HEAD(&worker->entry); INIT_LIST_HEAD(&worker->scheduled); + INIT_LIST_HEAD(&worker->node); /* on creation a worker is in !idle && prep state */ worker->flags = WORKER_PREP; } @@ -1700,12 +1618,68 @@ static struct worker *alloc_worker(void) } /** + * worker_attach_to_pool() - attach a worker to a pool + * @worker: worker to be attached + * @pool: the target pool + * + * Attach @worker to @pool. Once attached, the %WORKER_UNBOUND flag and + * cpu-binding of @worker are kept coordinated with the pool across + * cpu-[un]hotplugs. + */ +static void worker_attach_to_pool(struct worker *worker, + struct worker_pool *pool) +{ + mutex_lock(&pool->attach_mutex); + + /* + * set_cpus_allowed_ptr() will fail if the cpumask doesn't have any + * online CPUs. It'll be re-applied when any of the CPUs come up. + */ + set_cpus_allowed_ptr(worker->task, pool->attrs->cpumask); + + /* + * The pool->attach_mutex ensures %POOL_DISASSOCIATED remains + * stable across this function. See the comments above the + * flag definition for details. + */ + if (pool->flags & POOL_DISASSOCIATED) + worker->flags |= WORKER_UNBOUND; + + list_add_tail(&worker->node, &pool->workers); + + mutex_unlock(&pool->attach_mutex); +} + +/** + * worker_detach_from_pool() - detach a worker from its pool + * @worker: worker which is attached to its pool + * @pool: the pool @worker is attached to + * + * Undo the attaching which had been done in worker_attach_to_pool(). The + * caller worker shouldn't access to the pool after detached except it has + * other reference to the pool. + */ +static void worker_detach_from_pool(struct worker *worker, + struct worker_pool *pool) +{ + struct completion *detach_completion = NULL; + + mutex_lock(&pool->attach_mutex); + list_del(&worker->node); + if (list_empty(&pool->workers)) + detach_completion = pool->detach_completion; + mutex_unlock(&pool->attach_mutex); + + if (detach_completion) + complete(detach_completion); +} + +/** * create_worker - create a new workqueue worker * @pool: pool the new worker will belong to * - * Create a new worker which is bound to @pool. The returned worker - * can be started by calling start_worker() or destroyed using - * destroy_worker(). + * Create a new worker which is attached to @pool. The new worker must be + * started by start_worker(). * * CONTEXT: * Might sleep. Does GFP_KERNEL allocations. @@ -1719,19 +1693,8 @@ static struct worker *create_worker(struct worker_pool *pool) int id = -1; char id_buf[16]; - lockdep_assert_held(&pool->manager_mutex); - - /* - * ID is needed to determine kthread name. Allocate ID first - * without installing the pointer. - */ - idr_preload(GFP_KERNEL); - spin_lock_irq(&pool->lock); - - id = idr_alloc(&pool->worker_idr, NULL, 0, 0, GFP_NOWAIT); - - spin_unlock_irq(&pool->lock); - idr_preload_end(); + /* ID is needed to determine kthread name */ + id = ida_simple_get(&pool->worker_ida, 0, 0, GFP_KERNEL); if (id < 0) goto fail; @@ -1758,33 +1721,14 @@ static struct worker *create_worker(struct worker_pool *pool) /* prevent userland from meddling with cpumask of workqueue workers */ worker->task->flags |= PF_NO_SETAFFINITY; - /* - * set_cpus_allowed_ptr() will fail if the cpumask doesn't have any - * online CPUs. It'll be re-applied when any of the CPUs come up. - */ - set_cpus_allowed_ptr(worker->task, pool->attrs->cpumask); - - /* - * The caller is responsible for ensuring %POOL_DISASSOCIATED - * remains stable across this function. See the comments above the - * flag definition for details. - */ - if (pool->flags & POOL_DISASSOCIATED) - worker->flags |= WORKER_UNBOUND; - - /* successful, commit the pointer to idr */ - spin_lock_irq(&pool->lock); - idr_replace(&pool->worker_idr, worker, worker->id); - spin_unlock_irq(&pool->lock); + /* successful, attach the worker to the pool */ + worker_attach_to_pool(worker, pool); return worker; fail: - if (id >= 0) { - spin_lock_irq(&pool->lock); - idr_remove(&pool->worker_idr, id); - spin_unlock_irq(&pool->lock); - } + if (id >= 0) + ida_simple_remove(&pool->worker_ida, id); kfree(worker); return NULL; } @@ -1800,7 +1744,6 @@ fail: */ static void start_worker(struct worker *worker) { - worker->flags |= WORKER_STARTED; worker->pool->nr_workers++; worker_enter_idle(worker); wake_up_process(worker->task); @@ -1818,8 +1761,6 @@ static int create_and_start_worker(struct worker_pool *pool) { struct worker *worker; - mutex_lock(&pool->manager_mutex); - worker = create_worker(pool); if (worker) { spin_lock_irq(&pool->lock); @@ -1827,8 +1768,6 @@ static int create_and_start_worker(struct worker_pool *pool) spin_unlock_irq(&pool->lock); } - mutex_unlock(&pool->manager_mutex); - return worker ? 0 : -ENOMEM; } @@ -1836,46 +1775,30 @@ static int create_and_start_worker(struct worker_pool *pool) * destroy_worker - destroy a workqueue worker * @worker: worker to be destroyed * - * Destroy @worker and adjust @pool stats accordingly. + * Destroy @worker and adjust @pool stats accordingly. The worker should + * be idle. * * CONTEXT: - * spin_lock_irq(pool->lock) which is released and regrabbed. + * spin_lock_irq(pool->lock). */ static void destroy_worker(struct worker *worker) { struct worker_pool *pool = worker->pool; - lockdep_assert_held(&pool->manager_mutex); lockdep_assert_held(&pool->lock); /* sanity check frenzy */ if (WARN_ON(worker->current_work) || - WARN_ON(!list_empty(&worker->scheduled))) + WARN_ON(!list_empty(&worker->scheduled)) || + WARN_ON(!(worker->flags & WORKER_IDLE))) return; - if (worker->flags & WORKER_STARTED) - pool->nr_workers--; - if (worker->flags & WORKER_IDLE) - pool->nr_idle--; - - /* - * Once WORKER_DIE is set, the kworker may destroy itself at any - * point. Pin to ensure the task stays until we're done with it. - */ - get_task_struct(worker->task); + pool->nr_workers--; + pool->nr_idle--; list_del_init(&worker->entry); worker->flags |= WORKER_DIE; - - idr_remove(&pool->worker_idr, worker->id); - - spin_unlock_irq(&pool->lock); - - kthread_stop(worker->task); - put_task_struct(worker->task); - kfree(worker); - - spin_lock_irq(&pool->lock); + wake_up_process(worker->task); } static void idle_worker_timeout(unsigned long __pool) @@ -1884,7 +1807,7 @@ static void idle_worker_timeout(unsigned long __pool) spin_lock_irq(&pool->lock); - if (too_many_workers(pool)) { + while (too_many_workers(pool)) { struct worker *worker; unsigned long expires; @@ -1892,13 +1815,12 @@ static void idle_worker_timeout(unsigned long __pool) worker = list_entry(pool->idle_list.prev, struct worker, entry); expires = worker->last_active + IDLE_WORKER_TIMEOUT; - if (time_before(jiffies, expires)) + if (time_before(jiffies, expires)) { mod_timer(&pool->idle_timer, expires); - else { - /* it's been idle for too long, wake up manager */ - pool->flags |= POOL_MANAGE_WORKERS; - wake_up_worker(pool); + break; } + + destroy_worker(worker); } spin_unlock_irq(&pool->lock); @@ -2017,44 +1939,6 @@ restart: } /** - * maybe_destroy_worker - destroy workers which have been idle for a while - * @pool: pool to destroy workers for - * - * Destroy @pool workers which have been idle for longer than - * IDLE_WORKER_TIMEOUT. - * - * LOCKING: - * spin_lock_irq(pool->lock) which may be released and regrabbed - * multiple times. Called only from manager. - * - * Return: - * %false if no action was taken and pool->lock stayed locked, %true - * otherwise. - */ -static bool maybe_destroy_workers(struct worker_pool *pool) -{ - bool ret = false; - - while (too_many_workers(pool)) { - struct worker *worker; - unsigned long expires; - - worker = list_entry(pool->idle_list.prev, struct worker, entry); - expires = worker->last_active + IDLE_WORKER_TIMEOUT; - - if (time_before(jiffies, expires)) { - mod_timer(&pool->idle_timer, expires); - break; - } - - destroy_worker(worker); - ret = true; - } - - return ret; -} - -/** * manage_workers - manage worker pool * @worker: self * @@ -2083,8 +1967,6 @@ static bool manage_workers(struct worker *worker) bool ret = false; /* - * Managership is governed by two mutexes - manager_arb and - * manager_mutex. manager_arb handles arbitration of manager role. * Anyone who successfully grabs manager_arb wins the arbitration * and becomes the manager. mutex_trylock() on pool->manager_arb * failure while holding pool->lock reliably indicates that someone @@ -2093,40 +1975,12 @@ static bool manage_workers(struct worker *worker) * grabbing manager_arb is responsible for actually performing * manager duties. If manager_arb is grabbed and released without * actual management, the pool may stall indefinitely. - * - * manager_mutex is used for exclusion of actual management - * operations. The holder of manager_mutex can be sure that none - * of management operations, including creation and destruction of - * workers, won't take place until the mutex is released. Because - * manager_mutex doesn't interfere with manager role arbitration, - * it is guaranteed that the pool's management, while may be - * delayed, won't be disturbed by someone else grabbing - * manager_mutex. */ if (!mutex_trylock(&pool->manager_arb)) return ret; - /* - * With manager arbitration won, manager_mutex would be free in - * most cases. trylock first without dropping @pool->lock. - */ - if (unlikely(!mutex_trylock(&pool->manager_mutex))) { - spin_unlock_irq(&pool->lock); - mutex_lock(&pool->manager_mutex); - spin_lock_irq(&pool->lock); - ret = true; - } - - pool->flags &= ~POOL_MANAGE_WORKERS; - - /* - * Destroy and then create so that may_start_working() is true - * on return. - */ - ret |= maybe_destroy_workers(pool); ret |= maybe_create_worker(pool); - mutex_unlock(&pool->manager_mutex); mutex_unlock(&pool->manager_arb); return ret; } @@ -2314,6 +2168,11 @@ woke_up: spin_unlock_irq(&pool->lock); WARN_ON_ONCE(!list_empty(&worker->entry)); worker->task->flags &= ~PF_WQ_WORKER; + + set_task_comm(worker->task, "kworker/dying"); + ida_simple_remove(&pool->worker_ida, worker->id); + worker_detach_from_pool(worker, pool); + kfree(worker); return 0; } @@ -2361,9 +2220,6 @@ recheck: worker_set_flags(worker, WORKER_PREP, false); sleep: - if (unlikely(need_to_manage_workers(pool)) && manage_workers(worker)) - goto recheck; - /* * pool->lock is held and there's no work to process and no need to * manage, sleep. Workers are woken up only while holding @@ -2440,8 +2296,9 @@ repeat: spin_unlock_irq(&wq_mayday_lock); - /* migrate to the target cpu if possible */ - worker_maybe_bind_and_lock(pool); + worker_attach_to_pool(rescuer, pool); + + spin_lock_irq(&pool->lock); rescuer->pool = pool; /* @@ -2454,6 +2311,11 @@ repeat: move_linked_works(work, scheduled, &n); process_scheduled_works(rescuer); + spin_unlock_irq(&pool->lock); + + worker_detach_from_pool(rescuer, pool); + + spin_lock_irq(&pool->lock); /* * Put the reference grabbed by send_mayday(). @pool won't @@ -3550,9 +3412,10 @@ static int init_worker_pool(struct worker_pool *pool) (unsigned long)pool); mutex_init(&pool->manager_arb); - mutex_init(&pool->manager_mutex); - idr_init(&pool->worker_idr); + mutex_init(&pool->attach_mutex); + INIT_LIST_HEAD(&pool->workers); + ida_init(&pool->worker_ida); INIT_HLIST_NODE(&pool->hash_node); pool->refcnt = 1; @@ -3567,7 +3430,7 @@ static void rcu_free_pool(struct rcu_head *rcu) { struct worker_pool *pool = container_of(rcu, struct worker_pool, rcu); - idr_destroy(&pool->worker_idr); + ida_destroy(&pool->worker_ida); free_workqueue_attrs(pool->attrs); kfree(pool); } @@ -3585,6 +3448,7 @@ static void rcu_free_pool(struct rcu_head *rcu) */ static void put_unbound_pool(struct worker_pool *pool) { + DECLARE_COMPLETION_ONSTACK(detach_completion); struct worker *worker; lockdep_assert_held(&wq_pool_mutex); @@ -3605,18 +3469,24 @@ static void put_unbound_pool(struct worker_pool *pool) /* * Become the manager and destroy all workers. Grabbing * manager_arb prevents @pool's workers from blocking on - * manager_mutex. + * attach_mutex. */ mutex_lock(&pool->manager_arb); - mutex_lock(&pool->manager_mutex); - spin_lock_irq(&pool->lock); - while ((worker = first_worker(pool))) + spin_lock_irq(&pool->lock); + while ((worker = first_idle_worker(pool))) destroy_worker(worker); WARN_ON(pool->nr_workers || pool->nr_idle); - spin_unlock_irq(&pool->lock); - mutex_unlock(&pool->manager_mutex); + + mutex_lock(&pool->attach_mutex); + if (!list_empty(&pool->workers)) + pool->detach_completion = &detach_completion; + mutex_unlock(&pool->attach_mutex); + + if (pool->detach_completion) + wait_for_completion(pool->detach_completion); + mutex_unlock(&pool->manager_arb); /* shut down the timers */ @@ -3662,9 +3532,6 @@ static struct worker_pool *get_unbound_pool(const struct workqueue_attrs *attrs) if (!pool || init_worker_pool(pool) < 0) goto fail; - if (workqueue_freezing) - pool->flags |= POOL_FREEZING; - lockdep_set_subclass(&pool->lock, 1); /* see put_pwq() */ copy_workqueue_attrs(pool->attrs, attrs); @@ -3771,7 +3638,12 @@ static void pwq_adjust_max_active(struct pool_workqueue *pwq) spin_lock_irq(&pwq->pool->lock); - if (!freezable || !(pwq->pool->flags & POOL_FREEZING)) { + /* + * During [un]freezing, the caller is responsible for ensuring that + * this function is called at least once after @workqueue_freezing + * is updated and visible. + */ + if (!freezable || !workqueue_freezing) { pwq->max_active = wq->saved_max_active; while (!list_empty(&pwq->delayed_works) && @@ -4103,17 +3975,13 @@ static void wq_update_unbound_numa(struct workqueue_struct *wq, int cpu, * Let's determine what needs to be done. If the target cpumask is * different from wq's, we need to compare it to @pwq's and create * a new one if they don't match. If the target cpumask equals - * wq's, the default pwq should be used. If @pwq is already the - * default one, nothing to do; otherwise, install the default one. + * wq's, the default pwq should be used. */ if (wq_calc_node_cpumask(wq->unbound_attrs, node, cpu_off, cpumask)) { if (cpumask_equal(cpumask, pwq->pool->attrs->cpumask)) goto out_unlock; } else { - if (pwq == wq->dfl_pwq) - goto out_unlock; - else - goto use_dfl_pwq; + goto use_dfl_pwq; } mutex_unlock(&wq->mutex); @@ -4121,8 +3989,8 @@ static void wq_update_unbound_numa(struct workqueue_struct *wq, int cpu, /* create a new pwq */ pwq = alloc_unbound_pwq(wq, target_attrs); if (!pwq) { - pr_warning("workqueue: allocation failed while updating NUMA affinity of \"%s\"\n", - wq->name); + pr_warn("workqueue: allocation failed while updating NUMA affinity of \"%s\"\n", + wq->name); mutex_lock(&wq->mutex); goto use_dfl_pwq; } @@ -4599,28 +4467,27 @@ static void wq_unbind_fn(struct work_struct *work) int cpu = smp_processor_id(); struct worker_pool *pool; struct worker *worker; - int wi; for_each_cpu_worker_pool(pool, cpu) { WARN_ON_ONCE(cpu != smp_processor_id()); - mutex_lock(&pool->manager_mutex); + mutex_lock(&pool->attach_mutex); spin_lock_irq(&pool->lock); /* - * We've blocked all manager operations. Make all workers + * We've blocked all attach/detach operations. Make all workers * unbound and set DISASSOCIATED. Before this, all workers * except for the ones which are still executing works from * before the last CPU down must be on the cpu. After * this, they may become diasporas. */ - for_each_pool_worker(worker, wi, pool) + for_each_pool_worker(worker, pool) worker->flags |= WORKER_UNBOUND; pool->flags |= POOL_DISASSOCIATED; spin_unlock_irq(&pool->lock); - mutex_unlock(&pool->manager_mutex); + mutex_unlock(&pool->attach_mutex); /* * Call schedule() so that we cross rq->lock and thus can @@ -4660,9 +4527,8 @@ static void wq_unbind_fn(struct work_struct *work) static void rebind_workers(struct worker_pool *pool) { struct worker *worker; - int wi; - lockdep_assert_held(&pool->manager_mutex); + lockdep_assert_held(&pool->attach_mutex); /* * Restore CPU affinity of all workers. As all idle workers should @@ -4671,13 +4537,13 @@ static void rebind_workers(struct worker_pool *pool) * of all workers first and then clear UNBOUND. As we're called * from CPU_ONLINE, the following shouldn't fail. */ - for_each_pool_worker(worker, wi, pool) + for_each_pool_worker(worker, pool) WARN_ON_ONCE(set_cpus_allowed_ptr(worker->task, pool->attrs->cpumask) < 0); spin_lock_irq(&pool->lock); - for_each_pool_worker(worker, wi, pool) { + for_each_pool_worker(worker, pool) { unsigned int worker_flags = worker->flags; /* @@ -4729,9 +4595,8 @@ static void restore_unbound_workers_cpumask(struct worker_pool *pool, int cpu) { static cpumask_t cpumask; struct worker *worker; - int wi; - lockdep_assert_held(&pool->manager_mutex); + lockdep_assert_held(&pool->attach_mutex); /* is @cpu allowed for @pool? */ if (!cpumask_test_cpu(cpu, pool->attrs->cpumask)) @@ -4743,7 +4608,7 @@ static void restore_unbound_workers_cpumask(struct worker_pool *pool, int cpu) return; /* as we're called from CPU_ONLINE, the following shouldn't fail */ - for_each_pool_worker(worker, wi, pool) + for_each_pool_worker(worker, pool) WARN_ON_ONCE(set_cpus_allowed_ptr(worker->task, pool->attrs->cpumask) < 0); } @@ -4776,7 +4641,7 @@ static int workqueue_cpu_up_callback(struct notifier_block *nfb, mutex_lock(&wq_pool_mutex); for_each_pool(pool, pi) { - mutex_lock(&pool->manager_mutex); + mutex_lock(&pool->attach_mutex); if (pool->cpu == cpu) { spin_lock_irq(&pool->lock); @@ -4788,7 +4653,7 @@ static int workqueue_cpu_up_callback(struct notifier_block *nfb, restore_unbound_workers_cpumask(pool, cpu); } - mutex_unlock(&pool->manager_mutex); + mutex_unlock(&pool->attach_mutex); } /* update NUMA affinity of unbound workqueues */ @@ -4887,24 +4752,14 @@ EXPORT_SYMBOL_GPL(work_on_cpu); */ void freeze_workqueues_begin(void) { - struct worker_pool *pool; struct workqueue_struct *wq; struct pool_workqueue *pwq; - int pi; mutex_lock(&wq_pool_mutex); WARN_ON_ONCE(workqueue_freezing); workqueue_freezing = true; - /* set FREEZING */ - for_each_pool(pool, pi) { - spin_lock_irq(&pool->lock); - WARN_ON_ONCE(pool->flags & POOL_FREEZING); - pool->flags |= POOL_FREEZING; - spin_unlock_irq(&pool->lock); - } - list_for_each_entry(wq, &workqueues, list) { mutex_lock(&wq->mutex); for_each_pwq(pwq, wq) @@ -4974,21 +4829,13 @@ void thaw_workqueues(void) { struct workqueue_struct *wq; struct pool_workqueue *pwq; - struct worker_pool *pool; - int pi; mutex_lock(&wq_pool_mutex); if (!workqueue_freezing) goto out_unlock; - /* clear FREEZING */ - for_each_pool(pool, pi) { - spin_lock_irq(&pool->lock); - WARN_ON_ONCE(!(pool->flags & POOL_FREEZING)); - pool->flags &= ~POOL_FREEZING; - spin_unlock_irq(&pool->lock); - } + workqueue_freezing = false; /* restore max_active and repopulate worklist */ list_for_each_entry(wq, &workqueues, list) { @@ -4998,7 +4845,6 @@ void thaw_workqueues(void) mutex_unlock(&wq->mutex); } - workqueue_freezing = false; out_unlock: mutex_unlock(&wq_pool_mutex); } diff --git a/kernel/workqueue_internal.h b/kernel/workqueue_internal.h index 7e2204db0b1a..45215870ac6c 100644 --- a/kernel/workqueue_internal.h +++ b/kernel/workqueue_internal.h @@ -37,6 +37,8 @@ struct worker { struct task_struct *task; /* I: worker task */ struct worker_pool *pool; /* I: the associated pool */ /* L: for rescuers */ + struct list_head node; /* A: anchored at pool->workers */ + /* A: runs through worker->node */ unsigned long last_active; /* L: last active timestamp */ unsigned int flags; /* X: flags */ diff --git a/mm/hugetlb_cgroup.c b/mm/hugetlb_cgroup.c index 595d7fd795e1..493f758445e7 100644 --- a/mm/hugetlb_cgroup.c +++ b/mm/hugetlb_cgroup.c @@ -52,7 +52,7 @@ static inline bool hugetlb_cgroup_is_root(struct hugetlb_cgroup *h_cg) static inline struct hugetlb_cgroup * parent_hugetlb_cgroup(struct hugetlb_cgroup *h_cg) { - return hugetlb_cgroup_from_css(css_parent(&h_cg->css)); + return hugetlb_cgroup_from_css(h_cg->css.parent); } static inline bool hugetlb_cgroup_have_usage(struct hugetlb_cgroup *h_cg) @@ -181,7 +181,7 @@ int hugetlb_cgroup_charge_cgroup(int idx, unsigned long nr_pages, again: rcu_read_lock(); h_cg = hugetlb_cgroup_from_task(current); - if (!css_tryget(&h_cg->css)) { + if (!css_tryget_online(&h_cg->css)) { rcu_read_unlock(); goto again; } @@ -253,15 +253,16 @@ static u64 hugetlb_cgroup_read_u64(struct cgroup_subsys_state *css, return res_counter_read_u64(&h_cg->hugepage[idx], name); } -static int hugetlb_cgroup_write(struct cgroup_subsys_state *css, - struct cftype *cft, char *buffer) +static ssize_t hugetlb_cgroup_write(struct kernfs_open_file *of, + char *buf, size_t nbytes, loff_t off) { int idx, name, ret; unsigned long long val; - struct hugetlb_cgroup *h_cg = hugetlb_cgroup_from_css(css); + struct hugetlb_cgroup *h_cg = hugetlb_cgroup_from_css(of_css(of)); - idx = MEMFILE_IDX(cft->private); - name = MEMFILE_ATTR(cft->private); + buf = strstrip(buf); + idx = MEMFILE_IDX(of_cft(of)->private); + name = MEMFILE_ATTR(of_cft(of)->private); switch (name) { case RES_LIMIT: @@ -271,7 +272,7 @@ static int hugetlb_cgroup_write(struct cgroup_subsys_state *css, break; } /* This function does all necessary parse...reuse it */ - ret = res_counter_memparse_write_strategy(buffer, &val); + ret = res_counter_memparse_write_strategy(buf, &val); if (ret) break; ret = res_counter_set_limit(&h_cg->hugepage[idx], val); @@ -280,17 +281,17 @@ static int hugetlb_cgroup_write(struct cgroup_subsys_state *css, ret = -EINVAL; break; } - return ret; + return ret ?: nbytes; } -static int hugetlb_cgroup_reset(struct cgroup_subsys_state *css, - unsigned int event) +static ssize_t hugetlb_cgroup_reset(struct kernfs_open_file *of, + char *buf, size_t nbytes, loff_t off) { int idx, name, ret = 0; - struct hugetlb_cgroup *h_cg = hugetlb_cgroup_from_css(css); + struct hugetlb_cgroup *h_cg = hugetlb_cgroup_from_css(of_css(of)); - idx = MEMFILE_IDX(event); - name = MEMFILE_ATTR(event); + idx = MEMFILE_IDX(of_cft(of)->private); + name = MEMFILE_ATTR(of_cft(of)->private); switch (name) { case RES_MAX_USAGE: @@ -303,7 +304,7 @@ static int hugetlb_cgroup_reset(struct cgroup_subsys_state *css, ret = -EINVAL; break; } - return ret; + return ret ?: nbytes; } static char *mem_fmt(char *buf, int size, unsigned long hsize) @@ -331,7 +332,7 @@ static void __init __hugetlb_cgroup_file_init(int idx) snprintf(cft->name, MAX_CFTYPE_NAME, "%s.limit_in_bytes", buf); cft->private = MEMFILE_PRIVATE(idx, RES_LIMIT); cft->read_u64 = hugetlb_cgroup_read_u64; - cft->write_string = hugetlb_cgroup_write; + cft->write = hugetlb_cgroup_write; /* Add the usage file */ cft = &h->cgroup_files[1]; @@ -343,14 +344,14 @@ static void __init __hugetlb_cgroup_file_init(int idx) cft = &h->cgroup_files[2]; snprintf(cft->name, MAX_CFTYPE_NAME, "%s.max_usage_in_bytes", buf); cft->private = MEMFILE_PRIVATE(idx, RES_MAX_USAGE); - cft->trigger = hugetlb_cgroup_reset; + cft->write = hugetlb_cgroup_reset; cft->read_u64 = hugetlb_cgroup_read_u64; /* Add the failcntfile */ cft = &h->cgroup_files[3]; snprintf(cft->name, MAX_CFTYPE_NAME, "%s.failcnt", buf); cft->private = MEMFILE_PRIVATE(idx, RES_FAILCNT); - cft->trigger = hugetlb_cgroup_reset; + cft->write = hugetlb_cgroup_reset; cft->read_u64 = hugetlb_cgroup_read_u64; /* NULL terminate the last cft */ diff --git a/mm/memcontrol.c b/mm/memcontrol.c index a9559b91603c..a2c7bcb0e6eb 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -526,18 +526,14 @@ static inline bool mem_cgroup_is_root(struct mem_cgroup *memcg) static inline unsigned short mem_cgroup_id(struct mem_cgroup *memcg) { - /* - * The ID of the root cgroup is 0, but memcg treat 0 as an - * invalid ID, so we return (cgroup_id + 1). - */ - return memcg->css.cgroup->id + 1; + return memcg->css.id; } static inline struct mem_cgroup *mem_cgroup_from_id(unsigned short id) { struct cgroup_subsys_state *css; - css = css_from_id(id - 1, &memory_cgrp_subsys); + css = css_from_id(id, &memory_cgrp_subsys); return mem_cgroup_from_css(css); } @@ -570,7 +566,8 @@ void sock_update_memcg(struct sock *sk) memcg = mem_cgroup_from_task(current); cg_proto = sk->sk_prot->proto_cgroup(memcg); if (!mem_cgroup_is_root(memcg) && - memcg_proto_active(cg_proto) && css_tryget(&memcg->css)) { + memcg_proto_active(cg_proto) && + css_tryget_online(&memcg->css)) { sk->sk_cgrp = cg_proto; } rcu_read_unlock(); @@ -831,7 +828,7 @@ retry: */ __mem_cgroup_remove_exceeded(mz, mctz); if (!res_counter_soft_limit_excess(&mz->memcg->res) || - !css_tryget(&mz->memcg->css)) + !css_tryget_online(&mz->memcg->css)) goto retry; done: return mz; @@ -1073,7 +1070,7 @@ static struct mem_cgroup *get_mem_cgroup_from_mm(struct mm_struct *mm) if (unlikely(!memcg)) memcg = root_mem_cgroup; } - } while (!css_tryget(&memcg->css)); + } while (!css_tryget_online(&memcg->css)); rcu_read_unlock(); return memcg; } @@ -1110,7 +1107,8 @@ skip_node: */ if (next_css) { if ((next_css == &root->css) || - ((next_css->flags & CSS_ONLINE) && css_tryget(next_css))) + ((next_css->flags & CSS_ONLINE) && + css_tryget_online(next_css))) return mem_cgroup_from_css(next_css); prev_css = next_css; @@ -1156,7 +1154,7 @@ mem_cgroup_iter_load(struct mem_cgroup_reclaim_iter *iter, * would be returned all the time. */ if (position && position != root && - !css_tryget(&position->css)) + !css_tryget_online(&position->css)) position = NULL; } return position; @@ -1533,7 +1531,7 @@ static unsigned long mem_cgroup_margin(struct mem_cgroup *memcg) int mem_cgroup_swappiness(struct mem_cgroup *memcg) { /* root ? */ - if (mem_cgroup_disabled() || !css_parent(&memcg->css)) + if (mem_cgroup_disabled() || !memcg->css.parent) return vm_swappiness; return memcg->swappiness; @@ -2769,9 +2767,9 @@ static void __mem_cgroup_cancel_local_charge(struct mem_cgroup *memcg, /* * A helper function to get mem_cgroup from ID. must be called under - * rcu_read_lock(). The caller is responsible for calling css_tryget if - * the mem_cgroup is used for charging. (dropping refcnt from swap can be - * called against removed memcg.) + * rcu_read_lock(). The caller is responsible for calling + * css_tryget_online() if the mem_cgroup is used for charging. (dropping + * refcnt from swap can be called against removed memcg.) */ static struct mem_cgroup *mem_cgroup_lookup(unsigned short id) { @@ -2794,14 +2792,14 @@ struct mem_cgroup *try_get_mem_cgroup_from_page(struct page *page) lock_page_cgroup(pc); if (PageCgroupUsed(pc)) { memcg = pc->mem_cgroup; - if (memcg && !css_tryget(&memcg->css)) + if (memcg && !css_tryget_online(&memcg->css)) memcg = NULL; } else if (PageSwapCache(page)) { ent.val = page_private(page); id = lookup_swap_cgroup_id(ent); rcu_read_lock(); memcg = mem_cgroup_lookup(id); - if (memcg && !css_tryget(&memcg->css)) + if (memcg && !css_tryget_online(&memcg->css)) memcg = NULL; rcu_read_unlock(); } @@ -3365,7 +3363,7 @@ struct kmem_cache *__memcg_kmem_get_cache(struct kmem_cache *cachep, } /* The corresponding put will be done in the workqueue. */ - if (!css_tryget(&memcg->css)) + if (!css_tryget_online(&memcg->css)) goto out; rcu_read_unlock(); @@ -4125,8 +4123,8 @@ void mem_cgroup_uncharge_swap(swp_entry_t ent) memcg = mem_cgroup_lookup(id); if (memcg) { /* - * We uncharge this because swap is freed. - * This memcg can be obsolete one. We avoid calling css_tryget + * We uncharge this because swap is freed. This memcg can + * be obsolete one. We avoid calling css_tryget_online(). */ if (!mem_cgroup_is_root(memcg)) res_counter_uncharge(&memcg->memsw, PAGE_SIZE); @@ -4711,18 +4709,28 @@ static void mem_cgroup_reparent_charges(struct mem_cgroup *memcg) } while (usage > 0); } +/* + * Test whether @memcg has children, dead or alive. Note that this + * function doesn't care whether @memcg has use_hierarchy enabled and + * returns %true if there are child csses according to the cgroup + * hierarchy. Testing use_hierarchy is the caller's responsiblity. + */ static inline bool memcg_has_children(struct mem_cgroup *memcg) { - lockdep_assert_held(&memcg_create_mutex); + bool ret; + /* - * The lock does not prevent addition or deletion to the list - * of children, but it prevents a new child from being - * initialized based on this parent in css_online(), so it's - * enough to decide whether hierarchically inherited - * attributes can still be changed or not. + * The lock does not prevent addition or deletion of children, but + * it prevents a new child from being initialized based on this + * parent in css_online(), so it's enough to decide whether + * hierarchically inherited attributes can still be changed or not. */ - return memcg->use_hierarchy && - !list_empty(&memcg->css.cgroup->children); + lockdep_assert_held(&memcg_create_mutex); + + rcu_read_lock(); + ret = css_next_child(NULL, &memcg->css); + rcu_read_unlock(); + return ret; } /* @@ -4734,11 +4742,6 @@ static inline bool memcg_has_children(struct mem_cgroup *memcg) static int mem_cgroup_force_empty(struct mem_cgroup *memcg) { int nr_retries = MEM_CGROUP_RECLAIM_RETRIES; - struct cgroup *cgrp = memcg->css.cgroup; - - /* returns EBUSY if there is a task or if we come here twice. */ - if (cgroup_has_tasks(cgrp) || !list_empty(&cgrp->children)) - return -EBUSY; /* we call try-to-free pages for make this cgroup empty */ lru_add_drain_all(); @@ -4758,20 +4761,19 @@ static int mem_cgroup_force_empty(struct mem_cgroup *memcg) } } - lru_add_drain(); - mem_cgroup_reparent_charges(memcg); return 0; } -static int mem_cgroup_force_empty_write(struct cgroup_subsys_state *css, - unsigned int event) +static ssize_t mem_cgroup_force_empty_write(struct kernfs_open_file *of, + char *buf, size_t nbytes, + loff_t off) { - struct mem_cgroup *memcg = mem_cgroup_from_css(css); + struct mem_cgroup *memcg = mem_cgroup_from_css(of_css(of)); if (mem_cgroup_is_root(memcg)) return -EINVAL; - return mem_cgroup_force_empty(memcg); + return mem_cgroup_force_empty(memcg) ?: nbytes; } static u64 mem_cgroup_hierarchy_read(struct cgroup_subsys_state *css, @@ -4785,7 +4787,7 @@ static int mem_cgroup_hierarchy_write(struct cgroup_subsys_state *css, { int retval = 0; struct mem_cgroup *memcg = mem_cgroup_from_css(css); - struct mem_cgroup *parent_memcg = mem_cgroup_from_css(css_parent(&memcg->css)); + struct mem_cgroup *parent_memcg = mem_cgroup_from_css(memcg->css.parent); mutex_lock(&memcg_create_mutex); @@ -4802,7 +4804,7 @@ static int mem_cgroup_hierarchy_write(struct cgroup_subsys_state *css, */ if ((!parent_memcg || !parent_memcg->use_hierarchy) && (val == 1 || val == 0)) { - if (list_empty(&memcg->css.cgroup->children)) + if (!memcg_has_children(memcg)) memcg->use_hierarchy = val; else retval = -EBUSY; @@ -4919,7 +4921,8 @@ static int __memcg_activate_kmem(struct mem_cgroup *memcg, * of course permitted. */ mutex_lock(&memcg_create_mutex); - if (cgroup_has_tasks(memcg->css.cgroup) || memcg_has_children(memcg)) + if (cgroup_has_tasks(memcg->css.cgroup) || + (memcg->use_hierarchy && memcg_has_children(memcg))) err = -EBUSY; mutex_unlock(&memcg_create_mutex); if (err) @@ -5021,17 +5024,18 @@ static int memcg_update_kmem_limit(struct mem_cgroup *memcg, * The user of this function is... * RES_LIMIT. */ -static int mem_cgroup_write(struct cgroup_subsys_state *css, struct cftype *cft, - char *buffer) +static ssize_t mem_cgroup_write(struct kernfs_open_file *of, + char *buf, size_t nbytes, loff_t off) { - struct mem_cgroup *memcg = mem_cgroup_from_css(css); + struct mem_cgroup *memcg = mem_cgroup_from_css(of_css(of)); enum res_type type; int name; unsigned long long val; int ret; - type = MEMFILE_TYPE(cft->private); - name = MEMFILE_ATTR(cft->private); + buf = strstrip(buf); + type = MEMFILE_TYPE(of_cft(of)->private); + name = MEMFILE_ATTR(of_cft(of)->private); switch (name) { case RES_LIMIT: @@ -5040,7 +5044,7 @@ static int mem_cgroup_write(struct cgroup_subsys_state *css, struct cftype *cft, break; } /* This function does all necessary parse...reuse it */ - ret = res_counter_memparse_write_strategy(buffer, &val); + ret = res_counter_memparse_write_strategy(buf, &val); if (ret) break; if (type == _MEM) @@ -5053,7 +5057,7 @@ static int mem_cgroup_write(struct cgroup_subsys_state *css, struct cftype *cft, return -EINVAL; break; case RES_SOFT_LIMIT: - ret = res_counter_memparse_write_strategy(buffer, &val); + ret = res_counter_memparse_write_strategy(buf, &val); if (ret) break; /* @@ -5070,7 +5074,7 @@ static int mem_cgroup_write(struct cgroup_subsys_state *css, struct cftype *cft, ret = -EINVAL; /* should be BUG() ? */ break; } - return ret; + return ret ?: nbytes; } static void memcg_get_hierarchical_limit(struct mem_cgroup *memcg, @@ -5083,8 +5087,8 @@ static void memcg_get_hierarchical_limit(struct mem_cgroup *memcg, if (!memcg->use_hierarchy) goto out; - while (css_parent(&memcg->css)) { - memcg = mem_cgroup_from_css(css_parent(&memcg->css)); + while (memcg->css.parent) { + memcg = mem_cgroup_from_css(memcg->css.parent); if (!memcg->use_hierarchy) break; tmp = res_counter_read_u64(&memcg->res, RES_LIMIT); @@ -5097,14 +5101,15 @@ out: *memsw_limit = min_memsw_limit; } -static int mem_cgroup_reset(struct cgroup_subsys_state *css, unsigned int event) +static ssize_t mem_cgroup_reset(struct kernfs_open_file *of, char *buf, + size_t nbytes, loff_t off) { - struct mem_cgroup *memcg = mem_cgroup_from_css(css); + struct mem_cgroup *memcg = mem_cgroup_from_css(of_css(of)); int name; enum res_type type; - type = MEMFILE_TYPE(event); - name = MEMFILE_ATTR(event); + type = MEMFILE_TYPE(of_cft(of)->private); + name = MEMFILE_ATTR(of_cft(of)->private); switch (name) { case RES_MAX_USAGE: @@ -5129,7 +5134,7 @@ static int mem_cgroup_reset(struct cgroup_subsys_state *css, unsigned int event) break; } - return 0; + return nbytes; } static u64 mem_cgroup_move_charge_read(struct cgroup_subsys_state *css, @@ -5322,7 +5327,7 @@ static int mem_cgroup_swappiness_write(struct cgroup_subsys_state *css, if (val > 100) return -EINVAL; - if (css_parent(css)) + if (css->parent) memcg->swappiness = val; else vm_swappiness = val; @@ -5659,7 +5664,7 @@ static int mem_cgroup_oom_control_write(struct cgroup_subsys_state *css, struct mem_cgroup *memcg = mem_cgroup_from_css(css); /* cannot set to root cgroup and only 0 and 1 are allowed */ - if (!css_parent(css) || !((val == 0) || (val == 1))) + if (!css->parent || !((val == 0) || (val == 1))) return -EINVAL; memcg->oom_kill_disable = val; @@ -5705,10 +5710,10 @@ static void kmem_cgroup_css_offline(struct mem_cgroup *memcg) * which is then paired with css_put during uncharge resp. here. * * Although this might sound strange as this path is called from - * css_offline() when the referencemight have dropped down to 0 - * and shouldn't be incremented anymore (css_tryget would fail) - * we do not have other options because of the kmem allocations - * lifetime. + * css_offline() when the referencemight have dropped down to 0 and + * shouldn't be incremented anymore (css_tryget_online() would + * fail) we do not have other options because of the kmem + * allocations lifetime. */ css_get(&memcg->css); @@ -5827,9 +5832,10 @@ static void memcg_event_ptable_queue_proc(struct file *file, * Input must be in format '<event_fd> <control_fd> <args>'. * Interpretation of args is defined by control file implementation. */ -static int memcg_write_event_control(struct cgroup_subsys_state *css, - struct cftype *cft, char *buffer) +static ssize_t memcg_write_event_control(struct kernfs_open_file *of, + char *buf, size_t nbytes, loff_t off) { + struct cgroup_subsys_state *css = of_css(of); struct mem_cgroup *memcg = mem_cgroup_from_css(css); struct mem_cgroup_event *event; struct cgroup_subsys_state *cfile_css; @@ -5840,15 +5846,17 @@ static int memcg_write_event_control(struct cgroup_subsys_state *css, char *endp; int ret; - efd = simple_strtoul(buffer, &endp, 10); + buf = strstrip(buf); + + efd = simple_strtoul(buf, &endp, 10); if (*endp != ' ') return -EINVAL; - buffer = endp + 1; + buf = endp + 1; - cfd = simple_strtoul(buffer, &endp, 10); + cfd = simple_strtoul(buf, &endp, 10); if ((*endp != ' ') && (*endp != '\0')) return -EINVAL; - buffer = endp + 1; + buf = endp + 1; event = kzalloc(sizeof(*event), GFP_KERNEL); if (!event) @@ -5916,8 +5924,8 @@ static int memcg_write_event_control(struct cgroup_subsys_state *css, * automatically removed on cgroup destruction but the removal is * asynchronous, so take an extra ref on @css. */ - cfile_css = css_tryget_from_dir(cfile.file->f_dentry->d_parent, - &memory_cgrp_subsys); + cfile_css = css_tryget_online_from_dir(cfile.file->f_dentry->d_parent, + &memory_cgrp_subsys); ret = -EINVAL; if (IS_ERR(cfile_css)) goto out_put_cfile; @@ -5926,7 +5934,7 @@ static int memcg_write_event_control(struct cgroup_subsys_state *css, goto out_put_cfile; } - ret = event->register_event(memcg, event->eventfd, buffer); + ret = event->register_event(memcg, event->eventfd, buf); if (ret) goto out_put_css; @@ -5939,7 +5947,7 @@ static int memcg_write_event_control(struct cgroup_subsys_state *css, fdput(cfile); fdput(efile); - return 0; + return nbytes; out_put_css: css_put(css); @@ -5964,25 +5972,25 @@ static struct cftype mem_cgroup_files[] = { { .name = "max_usage_in_bytes", .private = MEMFILE_PRIVATE(_MEM, RES_MAX_USAGE), - .trigger = mem_cgroup_reset, + .write = mem_cgroup_reset, .read_u64 = mem_cgroup_read_u64, }, { .name = "limit_in_bytes", .private = MEMFILE_PRIVATE(_MEM, RES_LIMIT), - .write_string = mem_cgroup_write, + .write = mem_cgroup_write, .read_u64 = mem_cgroup_read_u64, }, { .name = "soft_limit_in_bytes", .private = MEMFILE_PRIVATE(_MEM, RES_SOFT_LIMIT), - .write_string = mem_cgroup_write, + .write = mem_cgroup_write, .read_u64 = mem_cgroup_read_u64, }, { .name = "failcnt", .private = MEMFILE_PRIVATE(_MEM, RES_FAILCNT), - .trigger = mem_cgroup_reset, + .write = mem_cgroup_reset, .read_u64 = mem_cgroup_read_u64, }, { @@ -5991,7 +5999,7 @@ static struct cftype mem_cgroup_files[] = { }, { .name = "force_empty", - .trigger = mem_cgroup_force_empty_write, + .write = mem_cgroup_force_empty_write, }, { .name = "use_hierarchy", @@ -6001,7 +6009,7 @@ static struct cftype mem_cgroup_files[] = { }, { .name = "cgroup.event_control", /* XXX: for compat */ - .write_string = memcg_write_event_control, + .write = memcg_write_event_control, .flags = CFTYPE_NO_PREFIX, .mode = S_IWUGO, }, @@ -6034,7 +6042,7 @@ static struct cftype mem_cgroup_files[] = { { .name = "kmem.limit_in_bytes", .private = MEMFILE_PRIVATE(_KMEM, RES_LIMIT), - .write_string = mem_cgroup_write, + .write = mem_cgroup_write, .read_u64 = mem_cgroup_read_u64, }, { @@ -6045,13 +6053,13 @@ static struct cftype mem_cgroup_files[] = { { .name = "kmem.failcnt", .private = MEMFILE_PRIVATE(_KMEM, RES_FAILCNT), - .trigger = mem_cgroup_reset, + .write = mem_cgroup_reset, .read_u64 = mem_cgroup_read_u64, }, { .name = "kmem.max_usage_in_bytes", .private = MEMFILE_PRIVATE(_KMEM, RES_MAX_USAGE), - .trigger = mem_cgroup_reset, + .write = mem_cgroup_reset, .read_u64 = mem_cgroup_read_u64, }, #ifdef CONFIG_SLABINFO @@ -6074,19 +6082,19 @@ static struct cftype memsw_cgroup_files[] = { { .name = "memsw.max_usage_in_bytes", .private = MEMFILE_PRIVATE(_MEMSWAP, RES_MAX_USAGE), - .trigger = mem_cgroup_reset, + .write = mem_cgroup_reset, .read_u64 = mem_cgroup_read_u64, }, { .name = "memsw.limit_in_bytes", .private = MEMFILE_PRIVATE(_MEMSWAP, RES_LIMIT), - .write_string = mem_cgroup_write, + .write = mem_cgroup_write, .read_u64 = mem_cgroup_read_u64, }, { .name = "memsw.failcnt", .private = MEMFILE_PRIVATE(_MEMSWAP, RES_FAILCNT), - .trigger = mem_cgroup_reset, + .write = mem_cgroup_reset, .read_u64 = mem_cgroup_read_u64, }, { }, /* terminate */ @@ -6264,9 +6272,9 @@ static int mem_cgroup_css_online(struct cgroup_subsys_state *css) { struct mem_cgroup *memcg = mem_cgroup_from_css(css); - struct mem_cgroup *parent = mem_cgroup_from_css(css_parent(css)); + struct mem_cgroup *parent = mem_cgroup_from_css(css->parent); - if (css->cgroup->id > MEM_CGROUP_ID_MAX) + if (css->id > MEM_CGROUP_ID_MAX) return -ENOSPC; if (!parent) @@ -6361,7 +6369,7 @@ static void mem_cgroup_css_free(struct cgroup_subsys_state *css) /* * XXX: css_offline() would be where we should reparent all * memory to prepare the cgroup for destruction. However, - * memcg does not do css_tryget() and res_counter charging + * memcg does not do css_tryget_online() and res_counter charging * under the same RCU lock region, which means that charging * could race with offlining. Offlining only happens to * cgroups with no tasks in them but charges can show up @@ -6375,9 +6383,9 @@ static void mem_cgroup_css_free(struct cgroup_subsys_state *css) * lookup_swap_cgroup_id() * rcu_read_lock() * mem_cgroup_lookup() - * css_tryget() + * css_tryget_online() * rcu_read_unlock() - * disable css_tryget() + * disable css_tryget_online() * call_rcu() * offline_css() * reparent_charges() diff --git a/net/core/netclassid_cgroup.c b/net/core/netclassid_cgroup.c index 22931e1b99b4..30d903b19c62 100644 --- a/net/core/netclassid_cgroup.c +++ b/net/core/netclassid_cgroup.c @@ -42,7 +42,7 @@ cgrp_css_alloc(struct cgroup_subsys_state *parent_css) static int cgrp_css_online(struct cgroup_subsys_state *css) { struct cgroup_cls_state *cs = css_cls_state(css); - struct cgroup_cls_state *parent = css_cls_state(css_parent(css)); + struct cgroup_cls_state *parent = css_cls_state(css->parent); if (parent) cs->classid = parent->classid; diff --git a/net/core/netprio_cgroup.c b/net/core/netprio_cgroup.c index 3825f669147b..2f385b9bccc0 100644 --- a/net/core/netprio_cgroup.c +++ b/net/core/netprio_cgroup.c @@ -140,7 +140,7 @@ cgrp_css_alloc(struct cgroup_subsys_state *parent_css) static int cgrp_css_online(struct cgroup_subsys_state *css) { - struct cgroup_subsys_state *parent_css = css_parent(css); + struct cgroup_subsys_state *parent_css = css->parent; struct net_device *dev; int ret = 0; @@ -185,15 +185,15 @@ static int read_priomap(struct seq_file *sf, void *v) return 0; } -static int write_priomap(struct cgroup_subsys_state *css, struct cftype *cft, - char *buffer) +static ssize_t write_priomap(struct kernfs_open_file *of, + char *buf, size_t nbytes, loff_t off) { char devname[IFNAMSIZ + 1]; struct net_device *dev; u32 prio; int ret; - if (sscanf(buffer, "%"__stringify(IFNAMSIZ)"s %u", devname, &prio) != 2) + if (sscanf(buf, "%"__stringify(IFNAMSIZ)"s %u", devname, &prio) != 2) return -EINVAL; dev = dev_get_by_name(&init_net, devname); @@ -202,11 +202,11 @@ static int write_priomap(struct cgroup_subsys_state *css, struct cftype *cft, rtnl_lock(); - ret = netprio_set_prio(css, dev, prio); + ret = netprio_set_prio(of_css(of), dev, prio); rtnl_unlock(); dev_put(dev); - return ret; + return ret ?: nbytes; } static int update_netprio(const void *v, struct file *file, unsigned n) @@ -239,7 +239,7 @@ static struct cftype ss_files[] = { { .name = "ifpriomap", .seq_show = read_priomap, - .write_string = write_priomap, + .write = write_priomap, }, { } /* terminate */ }; diff --git a/net/ipv4/tcp_memcontrol.c b/net/ipv4/tcp_memcontrol.c index d4f015ad6c84..f7a2ec3ac584 100644 --- a/net/ipv4/tcp_memcontrol.c +++ b/net/ipv4/tcp_memcontrol.c @@ -102,17 +102,19 @@ static int tcp_update_limit(struct mem_cgroup *memcg, u64 val) return 0; } -static int tcp_cgroup_write(struct cgroup_subsys_state *css, struct cftype *cft, - char *buffer) +static ssize_t tcp_cgroup_write(struct kernfs_open_file *of, + char *buf, size_t nbytes, loff_t off) { - struct mem_cgroup *memcg = mem_cgroup_from_css(css); + struct mem_cgroup *memcg = mem_cgroup_from_css(of_css(of)); unsigned long long val; int ret = 0; - switch (cft->private) { + buf = strstrip(buf); + + switch (of_cft(of)->private) { case RES_LIMIT: /* see memcontrol.c */ - ret = res_counter_memparse_write_strategy(buffer, &val); + ret = res_counter_memparse_write_strategy(buf, &val); if (ret) break; ret = tcp_update_limit(memcg, val); @@ -121,7 +123,7 @@ static int tcp_cgroup_write(struct cgroup_subsys_state *css, struct cftype *cft, ret = -EINVAL; break; } - return ret; + return ret ?: nbytes; } static u64 tcp_read_stat(struct mem_cgroup *memcg, int type, u64 default_val) @@ -168,17 +170,18 @@ static u64 tcp_cgroup_read(struct cgroup_subsys_state *css, struct cftype *cft) return val; } -static int tcp_cgroup_reset(struct cgroup_subsys_state *css, unsigned int event) +static ssize_t tcp_cgroup_reset(struct kernfs_open_file *of, + char *buf, size_t nbytes, loff_t off) { struct mem_cgroup *memcg; struct cg_proto *cg_proto; - memcg = mem_cgroup_from_css(css); + memcg = mem_cgroup_from_css(of_css(of)); cg_proto = tcp_prot.proto_cgroup(memcg); if (!cg_proto) - return 0; + return nbytes; - switch (event) { + switch (of_cft(of)->private) { case RES_MAX_USAGE: res_counter_reset_max(&cg_proto->memory_allocated); break; @@ -187,13 +190,13 @@ static int tcp_cgroup_reset(struct cgroup_subsys_state *css, unsigned int event) break; } - return 0; + return nbytes; } static struct cftype tcp_files[] = { { .name = "kmem.tcp.limit_in_bytes", - .write_string = tcp_cgroup_write, + .write = tcp_cgroup_write, .read_u64 = tcp_cgroup_read, .private = RES_LIMIT, }, @@ -205,13 +208,13 @@ static struct cftype tcp_files[] = { { .name = "kmem.tcp.failcnt", .private = RES_FAILCNT, - .trigger = tcp_cgroup_reset, + .write = tcp_cgroup_reset, .read_u64 = tcp_cgroup_read, }, { .name = "kmem.tcp.max_usage_in_bytes", .private = RES_MAX_USAGE, - .trigger = tcp_cgroup_reset, + .write = tcp_cgroup_reset, .read_u64 = tcp_cgroup_read, }, { } /* terminate */ diff --git a/net/sunrpc/auth_gss/gss_mech_switch.c b/net/sunrpc/auth_gss/gss_mech_switch.c index 27ce26240932..92d5ab99fbf3 100644 --- a/net/sunrpc/auth_gss/gss_mech_switch.c +++ b/net/sunrpc/auth_gss/gss_mech_switch.c @@ -218,10 +218,8 @@ static struct gss_api_mech *_gss_mech_get_by_pseudoflavor(u32 pseudoflavor) spin_lock(®istered_mechs_lock); list_for_each_entry(pos, ®istered_mechs, gm_list) { - if (!mech_supports_pseudoflavor(pos, pseudoflavor)) { - module_put(pos->gm_owner); + if (!mech_supports_pseudoflavor(pos, pseudoflavor)) continue; - } if (try_module_get(pos->gm_owner)) gm = pos; break; diff --git a/net/sunrpc/auth_gss/svcauth_gss.c b/net/sunrpc/auth_gss/svcauth_gss.c index 0f73f4507746..4ce5eccec1f6 100644 --- a/net/sunrpc/auth_gss/svcauth_gss.c +++ b/net/sunrpc/auth_gss/svcauth_gss.c @@ -1503,6 +1503,7 @@ svcauth_gss_accept(struct svc_rqst *rqstp, __be32 *authp) if (unwrap_integ_data(rqstp, &rqstp->rq_arg, gc->gc_seq, rsci->mechctx)) goto garbage_args; + rqstp->rq_auth_slack = RPC_MAX_AUTH_SIZE; break; case RPC_GSS_SVC_PRIVACY: /* placeholders for length and seq. number: */ @@ -1511,6 +1512,7 @@ svcauth_gss_accept(struct svc_rqst *rqstp, __be32 *authp) if (unwrap_priv_data(rqstp, &rqstp->rq_arg, gc->gc_seq, rsci->mechctx)) goto garbage_args; + rqstp->rq_auth_slack = RPC_MAX_AUTH_SIZE * 2; break; default: goto auth_err; diff --git a/net/sunrpc/cache.c b/net/sunrpc/cache.c index ae333c1845bb..066362141133 100644 --- a/net/sunrpc/cache.c +++ b/net/sunrpc/cache.c @@ -374,7 +374,7 @@ void sunrpc_destroy_cache_detail(struct cache_detail *cd) } return; out: - printk(KERN_ERR "nfsd: failed to unregister %s cache\n", cd->name); + printk(KERN_ERR "RPC: failed to unregister %s cache\n", cd->name); } EXPORT_SYMBOL_GPL(sunrpc_destroy_cache_detail); diff --git a/net/sunrpc/sched.c b/net/sunrpc/sched.c index 25578afe1548..c0365c14b858 100644 --- a/net/sunrpc/sched.c +++ b/net/sunrpc/sched.c @@ -832,7 +832,8 @@ static void rpc_async_schedule(struct work_struct *work) * @size: requested byte size * * To prevent rpciod from hanging, this allocator never sleeps, - * returning NULL if the request cannot be serviced immediately. + * returning NULL and suppressing warning if the request cannot be serviced + * immediately. * The caller can arrange to sleep in a way that is safe for rpciod. * * Most requests are 'small' (under 2KiB) and can be serviced from a @@ -845,7 +846,7 @@ static void rpc_async_schedule(struct work_struct *work) void *rpc_malloc(struct rpc_task *task, size_t size) { struct rpc_buffer *buf; - gfp_t gfp = GFP_NOWAIT; + gfp_t gfp = GFP_NOWAIT | __GFP_NOWARN; if (RPC_IS_SWAPPER(task)) gfp |= __GFP_MEMALLOC; diff --git a/net/sunrpc/sunrpc.h b/net/sunrpc/sunrpc.h index 14c9f6d1c5ff..f2b7cb540e61 100644 --- a/net/sunrpc/sunrpc.h +++ b/net/sunrpc/sunrpc.h @@ -43,6 +43,19 @@ static inline int rpc_reply_expected(struct rpc_task *task) (task->tk_msg.rpc_proc->p_decode != NULL); } +static inline int sock_is_loopback(struct sock *sk) +{ + struct dst_entry *dst; + int loopback = 0; + rcu_read_lock(); + dst = rcu_dereference(sk->sk_dst_cache); + if (dst && dst->dev && + (dst->dev->features & NETIF_F_LOOPBACK)) + loopback = 1; + rcu_read_unlock(); + return loopback; +} + int svc_send_common(struct socket *sock, struct xdr_buf *xdr, struct page *headpage, unsigned long headoffset, struct page *tailpage, unsigned long tailoffset); diff --git a/net/sunrpc/svc_xprt.c b/net/sunrpc/svc_xprt.c index 06c6ff0cb911..b4737fbdec13 100644 --- a/net/sunrpc/svc_xprt.c +++ b/net/sunrpc/svc_xprt.c @@ -597,6 +597,7 @@ static int svc_alloc_arg(struct svc_rqst *rqstp) } rqstp->rq_pages[i] = p; } + rqstp->rq_page_end = &rqstp->rq_pages[i]; rqstp->rq_pages[i++] = NULL; /* this might be seen in nfs_read_actor */ /* Make arg->head point to first page and arg->pages point to rest */ @@ -730,6 +731,8 @@ static int svc_handle_xprt(struct svc_rqst *rqstp, struct svc_xprt *xprt) newxpt = xprt->xpt_ops->xpo_accept(xprt); if (newxpt) svc_add_new_temp_xprt(serv, newxpt); + else + module_put(xprt->xpt_class->xcl_owner); } else if (xprt->xpt_ops->xpo_has_wspace(xprt)) { /* XPT_DATA|XPT_DEFERRED case: */ dprintk("svc: server %p, pool %u, transport %p, inuse=%d\n", @@ -793,7 +796,7 @@ int svc_recv(struct svc_rqst *rqstp, long timeout) clear_bit(XPT_OLD, &xprt->xpt_flags); - rqstp->rq_secure = svc_port_is_privileged(svc_addr(rqstp)); + rqstp->rq_secure = xprt->xpt_ops->xpo_secure_port(rqstp); rqstp->rq_chandle.defer = svc_defer; if (serv->sv_stats) diff --git a/net/sunrpc/svcauth.c b/net/sunrpc/svcauth.c index 2af7b0cba43a..79c0f3459b5c 100644 --- a/net/sunrpc/svcauth.c +++ b/net/sunrpc/svcauth.c @@ -54,6 +54,8 @@ svc_authenticate(struct svc_rqst *rqstp, __be32 *authp) } spin_unlock(&authtab_lock); + rqstp->rq_auth_slack = 0; + rqstp->rq_authop = aops; return aops->accept(rqstp, authp); } diff --git a/net/sunrpc/svcsock.c b/net/sunrpc/svcsock.c index 43bcb4699d69..b507cd327d9b 100644 --- a/net/sunrpc/svcsock.c +++ b/net/sunrpc/svcsock.c @@ -400,6 +400,12 @@ static void svc_sock_setbufsize(struct socket *sock, unsigned int snd, release_sock(sock->sk); #endif } + +static int svc_sock_secure_port(struct svc_rqst *rqstp) +{ + return svc_port_is_privileged(svc_addr(rqstp)); +} + /* * INET callback when data has been received on the socket. */ @@ -678,6 +684,7 @@ static struct svc_xprt_ops svc_udp_ops = { .xpo_prep_reply_hdr = svc_udp_prep_reply_hdr, .xpo_has_wspace = svc_udp_has_wspace, .xpo_accept = svc_udp_accept, + .xpo_secure_port = svc_sock_secure_port, }; static struct svc_xprt_class svc_udp_class = { @@ -842,8 +849,7 @@ static struct svc_xprt *svc_tcp_accept(struct svc_xprt *xprt) * tell us anything. For now just warn about unpriv connections. */ if (!svc_port_is_privileged(sin)) { - dprintk(KERN_WARNING - "%s: connect from unprivileged port: %s\n", + dprintk("%s: connect from unprivileged port: %s\n", serv->sv_name, __svc_print_addr(sin, buf, sizeof(buf))); } @@ -867,6 +873,10 @@ static struct svc_xprt *svc_tcp_accept(struct svc_xprt *xprt) } svc_xprt_set_local(&newsvsk->sk_xprt, sin, slen); + if (sock_is_loopback(newsock->sk)) + set_bit(XPT_LOCAL, &newsvsk->sk_xprt.xpt_flags); + else + clear_bit(XPT_LOCAL, &newsvsk->sk_xprt.xpt_flags); if (serv->sv_stats) serv->sv_stats->nettcpconn++; @@ -1112,6 +1122,7 @@ static int svc_tcp_recvfrom(struct svc_rqst *rqstp) rqstp->rq_xprt_ctxt = NULL; rqstp->rq_prot = IPPROTO_TCP; + rqstp->rq_local = !!test_bit(XPT_LOCAL, &svsk->sk_xprt.xpt_flags); p = (__be32 *)rqstp->rq_arg.head[0].iov_base; calldir = p[1]; @@ -1234,6 +1245,7 @@ static struct svc_xprt_ops svc_tcp_bc_ops = { .xpo_detach = svc_bc_tcp_sock_detach, .xpo_free = svc_bc_sock_free, .xpo_prep_reply_hdr = svc_tcp_prep_reply_hdr, + .xpo_secure_port = svc_sock_secure_port, }; static struct svc_xprt_class svc_tcp_bc_class = { @@ -1272,6 +1284,7 @@ static struct svc_xprt_ops svc_tcp_ops = { .xpo_prep_reply_hdr = svc_tcp_prep_reply_hdr, .xpo_has_wspace = svc_tcp_has_wspace, .xpo_accept = svc_tcp_accept, + .xpo_secure_port = svc_sock_secure_port, }; static struct svc_xprt_class svc_tcp_class = { diff --git a/net/sunrpc/xdr.c b/net/sunrpc/xdr.c index dd97ba3c4456..23fb4e75e245 100644 --- a/net/sunrpc/xdr.c +++ b/net/sunrpc/xdr.c @@ -462,6 +462,7 @@ void xdr_init_encode(struct xdr_stream *xdr, struct xdr_buf *buf, __be32 *p) struct kvec *iov = buf->head; int scratch_len = buf->buflen - buf->page_len - buf->tail[0].iov_len; + xdr_set_scratch_buffer(xdr, NULL, 0); BUG_ON(scratch_len < 0); xdr->buf = buf; xdr->iov = iov; @@ -482,6 +483,73 @@ void xdr_init_encode(struct xdr_stream *xdr, struct xdr_buf *buf, __be32 *p) EXPORT_SYMBOL_GPL(xdr_init_encode); /** + * xdr_commit_encode - Ensure all data is written to buffer + * @xdr: pointer to xdr_stream + * + * We handle encoding across page boundaries by giving the caller a + * temporary location to write to, then later copying the data into + * place; xdr_commit_encode does that copying. + * + * Normally the caller doesn't need to call this directly, as the + * following xdr_reserve_space will do it. But an explicit call may be + * required at the end of encoding, or any other time when the xdr_buf + * data might be read. + */ +void xdr_commit_encode(struct xdr_stream *xdr) +{ + int shift = xdr->scratch.iov_len; + void *page; + + if (shift == 0) + return; + page = page_address(*xdr->page_ptr); + memcpy(xdr->scratch.iov_base, page, shift); + memmove(page, page + shift, (void *)xdr->p - page); + xdr->scratch.iov_len = 0; +} +EXPORT_SYMBOL_GPL(xdr_commit_encode); + +__be32 *xdr_get_next_encode_buffer(struct xdr_stream *xdr, size_t nbytes) +{ + static __be32 *p; + int space_left; + int frag1bytes, frag2bytes; + + if (nbytes > PAGE_SIZE) + return NULL; /* Bigger buffers require special handling */ + if (xdr->buf->len + nbytes > xdr->buf->buflen) + return NULL; /* Sorry, we're totally out of space */ + frag1bytes = (xdr->end - xdr->p) << 2; + frag2bytes = nbytes - frag1bytes; + if (xdr->iov) + xdr->iov->iov_len += frag1bytes; + else + xdr->buf->page_len += frag1bytes; + xdr->page_ptr++; + xdr->iov = NULL; + /* + * If the last encode didn't end exactly on a page boundary, the + * next one will straddle boundaries. Encode into the next + * page, then copy it back later in xdr_commit_encode. We use + * the "scratch" iov to track any temporarily unused fragment of + * space at the end of the previous buffer: + */ + xdr->scratch.iov_base = xdr->p; + xdr->scratch.iov_len = frag1bytes; + p = page_address(*xdr->page_ptr); + /* + * Note this is where the next encode will start after we've + * shifted this one back: + */ + xdr->p = (void *)p + frag2bytes; + space_left = xdr->buf->buflen - xdr->buf->len; + xdr->end = (void *)p + min_t(int, space_left, PAGE_SIZE); + xdr->buf->page_len += frag2bytes; + xdr->buf->len += nbytes; + return p; +} + +/** * xdr_reserve_space - Reserve buffer space for sending * @xdr: pointer to xdr_stream * @nbytes: number of bytes to reserve @@ -495,20 +563,122 @@ __be32 * xdr_reserve_space(struct xdr_stream *xdr, size_t nbytes) __be32 *p = xdr->p; __be32 *q; + xdr_commit_encode(xdr); /* align nbytes on the next 32-bit boundary */ nbytes += 3; nbytes &= ~3; q = p + (nbytes >> 2); if (unlikely(q > xdr->end || q < p)) - return NULL; + return xdr_get_next_encode_buffer(xdr, nbytes); xdr->p = q; - xdr->iov->iov_len += nbytes; + if (xdr->iov) + xdr->iov->iov_len += nbytes; + else + xdr->buf->page_len += nbytes; xdr->buf->len += nbytes; return p; } EXPORT_SYMBOL_GPL(xdr_reserve_space); /** + * xdr_truncate_encode - truncate an encode buffer + * @xdr: pointer to xdr_stream + * @len: new length of buffer + * + * Truncates the xdr stream, so that xdr->buf->len == len, + * and xdr->p points at offset len from the start of the buffer, and + * head, tail, and page lengths are adjusted to correspond. + * + * If this means moving xdr->p to a different buffer, we assume that + * that the end pointer should be set to the end of the current page, + * except in the case of the head buffer when we assume the head + * buffer's current length represents the end of the available buffer. + * + * This is *not* safe to use on a buffer that already has inlined page + * cache pages (as in a zero-copy server read reply), except for the + * simple case of truncating from one position in the tail to another. + * + */ +void xdr_truncate_encode(struct xdr_stream *xdr, size_t len) +{ + struct xdr_buf *buf = xdr->buf; + struct kvec *head = buf->head; + struct kvec *tail = buf->tail; + int fraglen; + int new, old; + + if (len > buf->len) { + WARN_ON_ONCE(1); + return; + } + xdr_commit_encode(xdr); + + fraglen = min_t(int, buf->len - len, tail->iov_len); + tail->iov_len -= fraglen; + buf->len -= fraglen; + if (tail->iov_len && buf->len == len) { + xdr->p = tail->iov_base + tail->iov_len; + /* xdr->end, xdr->iov should be set already */ + return; + } + WARN_ON_ONCE(fraglen); + fraglen = min_t(int, buf->len - len, buf->page_len); + buf->page_len -= fraglen; + buf->len -= fraglen; + + new = buf->page_base + buf->page_len; + old = new + fraglen; + xdr->page_ptr -= (old >> PAGE_SHIFT) - (new >> PAGE_SHIFT); + + if (buf->page_len && buf->len == len) { + xdr->p = page_address(*xdr->page_ptr); + xdr->end = (void *)xdr->p + PAGE_SIZE; + xdr->p = (void *)xdr->p + (new % PAGE_SIZE); + /* xdr->iov should already be NULL */ + return; + } + if (fraglen) { + xdr->end = head->iov_base + head->iov_len; + xdr->page_ptr--; + } + /* (otherwise assume xdr->end is already set) */ + head->iov_len = len; + buf->len = len; + xdr->p = head->iov_base + head->iov_len; + xdr->iov = buf->head; +} +EXPORT_SYMBOL(xdr_truncate_encode); + +/** + * xdr_restrict_buflen - decrease available buffer space + * @xdr: pointer to xdr_stream + * @newbuflen: new maximum number of bytes available + * + * Adjust our idea of how much space is available in the buffer. + * If we've already used too much space in the buffer, returns -1. + * If the available space is already smaller than newbuflen, returns 0 + * and does nothing. Otherwise, adjusts xdr->buf->buflen to newbuflen + * and ensures xdr->end is set at most offset newbuflen from the start + * of the buffer. + */ +int xdr_restrict_buflen(struct xdr_stream *xdr, int newbuflen) +{ + struct xdr_buf *buf = xdr->buf; + int left_in_this_buf = (void *)xdr->end - (void *)xdr->p; + int end_offset = buf->len + left_in_this_buf; + + if (newbuflen < 0 || newbuflen < buf->len) + return -1; + if (newbuflen > buf->buflen) + return 0; + if (newbuflen < end_offset) + xdr->end = (void *)xdr->end + newbuflen - end_offset; + buf->buflen = newbuflen; + return 0; +} +EXPORT_SYMBOL(xdr_restrict_buflen); + +/** * xdr_write_pages - Insert a list of pages into an XDR buffer for sending * @xdr: pointer to xdr_stream * @pages: list of pages diff --git a/net/sunrpc/xprt.c b/net/sunrpc/xprt.c index 89d051de6b3e..c3b2b3369e52 100644 --- a/net/sunrpc/xprt.c +++ b/net/sunrpc/xprt.c @@ -71,24 +71,6 @@ static void xprt_destroy(struct rpc_xprt *xprt); static DEFINE_SPINLOCK(xprt_list_lock); static LIST_HEAD(xprt_list); -/* - * The transport code maintains an estimate on the maximum number of out- - * standing RPC requests, using a smoothed version of the congestion - * avoidance implemented in 44BSD. This is basically the Van Jacobson - * congestion algorithm: If a retransmit occurs, the congestion window is - * halved; otherwise, it is incremented by 1/cwnd when - * - * - a reply is received and - * - a full number of requests are outstanding and - * - the congestion window hasn't been updated recently. - */ -#define RPC_CWNDSHIFT (8U) -#define RPC_CWNDSCALE (1U << RPC_CWNDSHIFT) -#define RPC_INITCWND RPC_CWNDSCALE -#define RPC_MAXCWND(xprt) ((xprt)->max_reqs << RPC_CWNDSHIFT) - -#define RPCXPRT_CONGESTED(xprt) ((xprt)->cong >= (xprt)->cwnd) - /** * xprt_register_transport - register a transport implementation * @transport: transport to register @@ -446,7 +428,15 @@ EXPORT_SYMBOL_GPL(xprt_release_rqst_cong); * @task: recently completed RPC request used to adjust window * @result: result code of completed RPC request * - * We use a time-smoothed congestion estimator to avoid heavy oscillation. + * The transport code maintains an estimate on the maximum number of out- + * standing RPC requests, using a smoothed version of the congestion + * avoidance implemented in 44BSD. This is basically the Van Jacobson + * congestion algorithm: If a retransmit occurs, the congestion window is + * halved; otherwise, it is incremented by 1/cwnd when + * + * - a reply is received and + * - a full number of requests are outstanding and + * - the congestion window hasn't been updated recently. */ void xprt_adjust_cwnd(struct rpc_xprt *xprt, struct rpc_task *task, int result) { diff --git a/net/sunrpc/xprtrdma/rpc_rdma.c b/net/sunrpc/xprtrdma/rpc_rdma.c index 96ead526b125..693966d3f33b 100644 --- a/net/sunrpc/xprtrdma/rpc_rdma.c +++ b/net/sunrpc/xprtrdma/rpc_rdma.c @@ -78,8 +78,7 @@ static const char transfertypes[][12] = { * elements. Segments are then coalesced when registered, if possible * within the selected memreg mode. * - * Note, this routine is never called if the connection's memory - * registration strategy is 0 (bounce buffers). + * Returns positive number of segments converted, or a negative errno. */ static int @@ -102,10 +101,17 @@ rpcrdma_convert_iovs(struct xdr_buf *xdrbuf, unsigned int pos, page_base = xdrbuf->page_base & ~PAGE_MASK; p = 0; while (len && n < nsegs) { + if (!ppages[p]) { + /* alloc the pagelist for receiving buffer */ + ppages[p] = alloc_page(GFP_ATOMIC); + if (!ppages[p]) + return -ENOMEM; + } seg[n].mr_page = ppages[p]; seg[n].mr_offset = (void *)(unsigned long) page_base; seg[n].mr_len = min_t(u32, PAGE_SIZE - page_base, len); - BUG_ON(seg[n].mr_len > PAGE_SIZE); + if (seg[n].mr_len > PAGE_SIZE) + return -EIO; len -= seg[n].mr_len; ++n; ++p; @@ -114,7 +120,7 @@ rpcrdma_convert_iovs(struct xdr_buf *xdrbuf, unsigned int pos, /* Message overflows the seg array */ if (len && n == nsegs) - return 0; + return -EIO; if (xdrbuf->tail[0].iov_len) { /* the rpcrdma protocol allows us to omit any trailing @@ -123,7 +129,7 @@ rpcrdma_convert_iovs(struct xdr_buf *xdrbuf, unsigned int pos, return n; if (n == nsegs) /* Tail remains, but we're out of segments */ - return 0; + return -EIO; seg[n].mr_page = NULL; seg[n].mr_offset = xdrbuf->tail[0].iov_base; seg[n].mr_len = xdrbuf->tail[0].iov_len; @@ -164,15 +170,17 @@ rpcrdma_convert_iovs(struct xdr_buf *xdrbuf, unsigned int pos, * Reply chunk (a counted array): * N elements: * 1 - N - HLOO - HLOO - ... - HLOO + * + * Returns positive RPC/RDMA header size, or negative errno. */ -static unsigned int +static ssize_t rpcrdma_create_chunks(struct rpc_rqst *rqst, struct xdr_buf *target, struct rpcrdma_msg *headerp, enum rpcrdma_chunktype type) { struct rpcrdma_req *req = rpcr_to_rdmar(rqst); struct rpcrdma_xprt *r_xprt = rpcx_to_rdmax(rqst->rq_xprt); - int nsegs, nchunks = 0; + int n, nsegs, nchunks = 0; unsigned int pos; struct rpcrdma_mr_seg *seg = req->rl_segments; struct rpcrdma_read_chunk *cur_rchunk = NULL; @@ -198,12 +206,11 @@ rpcrdma_create_chunks(struct rpc_rqst *rqst, struct xdr_buf *target, pos = target->head[0].iov_len; nsegs = rpcrdma_convert_iovs(target, pos, type, seg, RPCRDMA_MAX_SEGS); - if (nsegs == 0) - return 0; + if (nsegs < 0) + return nsegs; do { - /* bind/register the memory, then build chunk from result. */ - int n = rpcrdma_register_external(seg, nsegs, + n = rpcrdma_register_external(seg, nsegs, cur_wchunk != NULL, r_xprt); if (n <= 0) goto out; @@ -248,10 +255,6 @@ rpcrdma_create_chunks(struct rpc_rqst *rqst, struct xdr_buf *target, /* success. all failures return above */ req->rl_nchunks = nchunks; - BUG_ON(nchunks == 0); - BUG_ON((r_xprt->rx_ia.ri_memreg_strategy == RPCRDMA_FRMR) - && (nchunks > 3)); - /* * finish off header. If write, marshal discrim and nchunks. */ @@ -278,8 +281,8 @@ rpcrdma_create_chunks(struct rpc_rqst *rqst, struct xdr_buf *target, out: for (pos = 0; nchunks--;) pos += rpcrdma_deregister_external( - &req->rl_segments[pos], r_xprt, NULL); - return 0; + &req->rl_segments[pos], r_xprt); + return n; } /* @@ -361,6 +364,8 @@ rpcrdma_inline_pullup(struct rpc_rqst *rqst, int pad) * [1] -- the RPC header/data, marshaled by RPC and the NFS protocol. * [2] -- optional padding. * [3] -- if padded, header only in [1] and data here. + * + * Returns zero on success, otherwise a negative errno. */ int @@ -370,7 +375,8 @@ rpcrdma_marshal_req(struct rpc_rqst *rqst) struct rpcrdma_xprt *r_xprt = rpcx_to_rdmax(xprt); struct rpcrdma_req *req = rpcr_to_rdmar(rqst); char *base; - size_t hdrlen, rpclen, padlen; + size_t rpclen, padlen; + ssize_t hdrlen; enum rpcrdma_chunktype rtype, wtype; struct rpcrdma_msg *headerp; @@ -441,14 +447,10 @@ rpcrdma_marshal_req(struct rpc_rqst *rqst) /* The following simplification is not true forever */ if (rtype != rpcrdma_noch && wtype == rpcrdma_replych) wtype = rpcrdma_noch; - BUG_ON(rtype != rpcrdma_noch && wtype != rpcrdma_noch); - - if (r_xprt->rx_ia.ri_memreg_strategy == RPCRDMA_BOUNCEBUFFERS && - (rtype != rpcrdma_noch || wtype != rpcrdma_noch)) { - /* forced to "pure inline"? */ - dprintk("RPC: %s: too much data (%d/%d) for inline\n", - __func__, rqst->rq_rcv_buf.len, rqst->rq_snd_buf.len); - return -1; + if (rtype != rpcrdma_noch && wtype != rpcrdma_noch) { + dprintk("RPC: %s: cannot marshal multiple chunk lists\n", + __func__); + return -EIO; } hdrlen = 28; /*sizeof *headerp;*/ @@ -474,8 +476,11 @@ rpcrdma_marshal_req(struct rpc_rqst *rqst) headerp->rm_body.rm_padded.rm_pempty[1] = xdr_zero; headerp->rm_body.rm_padded.rm_pempty[2] = xdr_zero; hdrlen += 2 * sizeof(u32); /* extra words in padhdr */ - BUG_ON(wtype != rpcrdma_noch); - + if (wtype != rpcrdma_noch) { + dprintk("RPC: %s: invalid chunk list\n", + __func__); + return -EIO; + } } else { headerp->rm_body.rm_nochunks.rm_empty[0] = xdr_zero; headerp->rm_body.rm_nochunks.rm_empty[1] = xdr_zero; @@ -492,8 +497,7 @@ rpcrdma_marshal_req(struct rpc_rqst *rqst) * on receive. Therefore, we request a reply chunk * for non-writes wherever feasible and efficient. */ - if (wtype == rpcrdma_noch && - r_xprt->rx_ia.ri_memreg_strategy > RPCRDMA_REGISTER) + if (wtype == rpcrdma_noch) wtype = rpcrdma_replych; } } @@ -511,9 +515,8 @@ rpcrdma_marshal_req(struct rpc_rqst *rqst) hdrlen = rpcrdma_create_chunks(rqst, &rqst->rq_rcv_buf, headerp, wtype); } - - if (hdrlen == 0) - return -1; + if (hdrlen < 0) + return hdrlen; dprintk("RPC: %s: %s: hdrlen %zd rpclen %zd padlen %zd" " headerp 0x%p base 0x%p lkey 0x%x\n", @@ -680,15 +683,11 @@ rpcrdma_inline_fixup(struct rpc_rqst *rqst, char *srcp, int copy_len, int pad) rqst->rq_private_buf = rqst->rq_rcv_buf; } -/* - * This function is called when an async event is posted to - * the connection which changes the connection state. All it - * does at this point is mark the connection up/down, the rpc - * timers do the rest. - */ void -rpcrdma_conn_func(struct rpcrdma_ep *ep) +rpcrdma_connect_worker(struct work_struct *work) { + struct rpcrdma_ep *ep = + container_of(work, struct rpcrdma_ep, rep_connect_worker.work); struct rpc_xprt *xprt = ep->rep_xprt; spin_lock_bh(&xprt->transport_lock); @@ -705,13 +704,15 @@ rpcrdma_conn_func(struct rpcrdma_ep *ep) } /* - * This function is called when memory window unbind which we are waiting - * for completes. Just use rr_func (zeroed by upcall) to signal completion. + * This function is called when an async event is posted to + * the connection which changes the connection state. All it + * does at this point is mark the connection up/down, the rpc + * timers do the rest. */ -static void -rpcrdma_unbind_func(struct rpcrdma_rep *rep) +void +rpcrdma_conn_func(struct rpcrdma_ep *ep) { - wake_up(&rep->rr_unbind); + schedule_delayed_work(&ep->rep_connect_worker, 0); } /* @@ -728,7 +729,8 @@ rpcrdma_reply_handler(struct rpcrdma_rep *rep) struct rpc_xprt *xprt = rep->rr_xprt; struct rpcrdma_xprt *r_xprt = rpcx_to_rdmax(xprt); __be32 *iptr; - int i, rdmalen, status; + int rdmalen, status; + unsigned long cwnd; /* Check status. If bad, signal disconnect and return rep to pool */ if (rep->rr_len == ~0U) { @@ -783,6 +785,7 @@ repost: /* from here on, the reply is no longer an orphan */ req->rl_reply = rep; + xprt->reestablish_timeout = 0; /* check for expected message types */ /* The order of some of these tests is important. */ @@ -857,26 +860,10 @@ badheader: break; } - /* If using mw bind, start the deregister process now. */ - /* (Note: if mr_free(), cannot perform it here, in tasklet context) */ - if (req->rl_nchunks) switch (r_xprt->rx_ia.ri_memreg_strategy) { - case RPCRDMA_MEMWINDOWS: - for (i = 0; req->rl_nchunks-- > 1;) - i += rpcrdma_deregister_external( - &req->rl_segments[i], r_xprt, NULL); - /* Optionally wait (not here) for unbinds to complete */ - rep->rr_func = rpcrdma_unbind_func; - (void) rpcrdma_deregister_external(&req->rl_segments[i], - r_xprt, rep); - break; - case RPCRDMA_MEMWINDOWS_ASYNC: - for (i = 0; req->rl_nchunks--;) - i += rpcrdma_deregister_external(&req->rl_segments[i], - r_xprt, NULL); - break; - default: - break; - } + cwnd = xprt->cwnd; + xprt->cwnd = atomic_read(&r_xprt->rx_buf.rb_credits) << RPC_CWNDSHIFT; + if (xprt->cwnd > cwnd) + xprt_release_rqst_cong(rqst->rq_task); dprintk("RPC: %s: xprt_complete_rqst(0x%p, 0x%p, %d)\n", __func__, xprt, rqst, status); diff --git a/net/sunrpc/xprtrdma/svc_rdma_recvfrom.c b/net/sunrpc/xprtrdma/svc_rdma_recvfrom.c index 8d904e4eef15..8f92a61ee2df 100644 --- a/net/sunrpc/xprtrdma/svc_rdma_recvfrom.c +++ b/net/sunrpc/xprtrdma/svc_rdma_recvfrom.c @@ -1,4 +1,5 @@ /* + * Copyright (c) 2014 Open Grid Computing, Inc. All rights reserved. * Copyright (c) 2005-2006 Network Appliance, Inc. All rights reserved. * * This software is available to you under a choice of one of two @@ -69,7 +70,8 @@ static void rdma_build_arg_xdr(struct svc_rqst *rqstp, /* Set up the XDR head */ rqstp->rq_arg.head[0].iov_base = page_address(page); - rqstp->rq_arg.head[0].iov_len = min(byte_count, ctxt->sge[0].length); + rqstp->rq_arg.head[0].iov_len = + min_t(size_t, byte_count, ctxt->sge[0].length); rqstp->rq_arg.len = byte_count; rqstp->rq_arg.buflen = byte_count; @@ -85,7 +87,7 @@ static void rdma_build_arg_xdr(struct svc_rqst *rqstp, page = ctxt->pages[sge_no]; put_page(rqstp->rq_pages[sge_no]); rqstp->rq_pages[sge_no] = page; - bc -= min(bc, ctxt->sge[sge_no].length); + bc -= min_t(u32, bc, ctxt->sge[sge_no].length); rqstp->rq_arg.buflen += ctxt->sge[sge_no].length; sge_no++; } @@ -113,291 +115,265 @@ static void rdma_build_arg_xdr(struct svc_rqst *rqstp, rqstp->rq_arg.tail[0].iov_len = 0; } -/* Encode a read-chunk-list as an array of IB SGE - * - * Assumptions: - * - chunk[0]->position points to pages[0] at an offset of 0 - * - pages[] is not physically or virtually contiguous and consists of - * PAGE_SIZE elements. - * - * Output: - * - sge array pointing into pages[] array. - * - chunk_sge array specifying sge index and count for each - * chunk in the read list - * - */ -static int map_read_chunks(struct svcxprt_rdma *xprt, - struct svc_rqst *rqstp, - struct svc_rdma_op_ctxt *head, - struct rpcrdma_msg *rmsgp, - struct svc_rdma_req_map *rpl_map, - struct svc_rdma_req_map *chl_map, - int ch_count, - int byte_count) +static int rdma_read_max_sge(struct svcxprt_rdma *xprt, int sge_count) { - int sge_no; - int sge_bytes; - int page_off; - int page_no; - int ch_bytes; - int ch_no; - struct rpcrdma_read_chunk *ch; + if (rdma_node_get_transport(xprt->sc_cm_id->device->node_type) == + RDMA_TRANSPORT_IWARP) + return 1; + else + return min_t(int, sge_count, xprt->sc_max_sge); +} - sge_no = 0; - page_no = 0; - page_off = 0; - ch = (struct rpcrdma_read_chunk *)&rmsgp->rm_body.rm_chunks[0]; - ch_no = 0; - ch_bytes = ntohl(ch->rc_target.rs_length); - head->arg.head[0] = rqstp->rq_arg.head[0]; - head->arg.tail[0] = rqstp->rq_arg.tail[0]; - head->arg.pages = &head->pages[head->count]; - head->hdr_count = head->count; /* save count of hdr pages */ - head->arg.page_base = 0; - head->arg.page_len = ch_bytes; - head->arg.len = rqstp->rq_arg.len + ch_bytes; - head->arg.buflen = rqstp->rq_arg.buflen + ch_bytes; - head->count++; - chl_map->ch[0].start = 0; - while (byte_count) { - rpl_map->sge[sge_no].iov_base = - page_address(rqstp->rq_arg.pages[page_no]) + page_off; - sge_bytes = min_t(int, PAGE_SIZE-page_off, ch_bytes); - rpl_map->sge[sge_no].iov_len = sge_bytes; - /* - * Don't bump head->count here because the same page - * may be used by multiple SGE. - */ - head->arg.pages[page_no] = rqstp->rq_arg.pages[page_no]; - rqstp->rq_respages = &rqstp->rq_arg.pages[page_no+1]; +typedef int (*rdma_reader_fn)(struct svcxprt_rdma *xprt, + struct svc_rqst *rqstp, + struct svc_rdma_op_ctxt *head, + int *page_no, + u32 *page_offset, + u32 rs_handle, + u32 rs_length, + u64 rs_offset, + int last); + +/* Issue an RDMA_READ using the local lkey to map the data sink */ +static int rdma_read_chunk_lcl(struct svcxprt_rdma *xprt, + struct svc_rqst *rqstp, + struct svc_rdma_op_ctxt *head, + int *page_no, + u32 *page_offset, + u32 rs_handle, + u32 rs_length, + u64 rs_offset, + int last) +{ + struct ib_send_wr read_wr; + int pages_needed = PAGE_ALIGN(*page_offset + rs_length) >> PAGE_SHIFT; + struct svc_rdma_op_ctxt *ctxt = svc_rdma_get_context(xprt); + int ret, read, pno; + u32 pg_off = *page_offset; + u32 pg_no = *page_no; + + ctxt->direction = DMA_FROM_DEVICE; + ctxt->read_hdr = head; + pages_needed = + min_t(int, pages_needed, rdma_read_max_sge(xprt, pages_needed)); + read = min_t(int, pages_needed << PAGE_SHIFT, rs_length); + + for (pno = 0; pno < pages_needed; pno++) { + int len = min_t(int, rs_length, PAGE_SIZE - pg_off); + + head->arg.pages[pg_no] = rqstp->rq_arg.pages[pg_no]; + head->arg.page_len += len; + head->arg.len += len; + if (!pg_off) + head->count++; + rqstp->rq_respages = &rqstp->rq_arg.pages[pg_no+1]; rqstp->rq_next_page = rqstp->rq_respages + 1; + ctxt->sge[pno].addr = + ib_dma_map_page(xprt->sc_cm_id->device, + head->arg.pages[pg_no], pg_off, + PAGE_SIZE - pg_off, + DMA_FROM_DEVICE); + ret = ib_dma_mapping_error(xprt->sc_cm_id->device, + ctxt->sge[pno].addr); + if (ret) + goto err; + atomic_inc(&xprt->sc_dma_used); - byte_count -= sge_bytes; - ch_bytes -= sge_bytes; - sge_no++; - /* - * If all bytes for this chunk have been mapped to an - * SGE, move to the next SGE - */ - if (ch_bytes == 0) { - chl_map->ch[ch_no].count = - sge_no - chl_map->ch[ch_no].start; - ch_no++; - ch++; - chl_map->ch[ch_no].start = sge_no; - ch_bytes = ntohl(ch->rc_target.rs_length); - /* If bytes remaining account for next chunk */ - if (byte_count) { - head->arg.page_len += ch_bytes; - head->arg.len += ch_bytes; - head->arg.buflen += ch_bytes; - } + /* The lkey here is either a local dma lkey or a dma_mr lkey */ + ctxt->sge[pno].lkey = xprt->sc_dma_lkey; + ctxt->sge[pno].length = len; + ctxt->count++; + + /* adjust offset and wrap to next page if needed */ + pg_off += len; + if (pg_off == PAGE_SIZE) { + pg_off = 0; + pg_no++; } - /* - * If this SGE consumed all of the page, move to the - * next page - */ - if ((sge_bytes + page_off) == PAGE_SIZE) { - page_no++; - page_off = 0; - /* - * If there are still bytes left to map, bump - * the page count - */ - if (byte_count) - head->count++; - } else - page_off += sge_bytes; + rs_length -= len; } - BUG_ON(byte_count != 0); - return sge_no; + + if (last && rs_length == 0) + set_bit(RDMACTXT_F_LAST_CTXT, &ctxt->flags); + else + clear_bit(RDMACTXT_F_LAST_CTXT, &ctxt->flags); + + memset(&read_wr, 0, sizeof(read_wr)); + read_wr.wr_id = (unsigned long)ctxt; + read_wr.opcode = IB_WR_RDMA_READ; + ctxt->wr_op = read_wr.opcode; + read_wr.send_flags = IB_SEND_SIGNALED; + read_wr.wr.rdma.rkey = rs_handle; + read_wr.wr.rdma.remote_addr = rs_offset; + read_wr.sg_list = ctxt->sge; + read_wr.num_sge = pages_needed; + + ret = svc_rdma_send(xprt, &read_wr); + if (ret) { + pr_err("svcrdma: Error %d posting RDMA_READ\n", ret); + set_bit(XPT_CLOSE, &xprt->sc_xprt.xpt_flags); + goto err; + } + + /* return current location in page array */ + *page_no = pg_no; + *page_offset = pg_off; + ret = read; + atomic_inc(&rdma_stat_read); + return ret; + err: + svc_rdma_unmap_dma(ctxt); + svc_rdma_put_context(ctxt, 0); + return ret; } -/* Map a read-chunk-list to an XDR and fast register the page-list. - * - * Assumptions: - * - chunk[0] position points to pages[0] at an offset of 0 - * - pages[] will be made physically contiguous by creating a one-off memory - * region using the fastreg verb. - * - byte_count is # of bytes in read-chunk-list - * - ch_count is # of chunks in read-chunk-list - * - * Output: - * - sge array pointing into pages[] array. - * - chunk_sge array specifying sge index and count for each - * chunk in the read list - */ -static int fast_reg_read_chunks(struct svcxprt_rdma *xprt, +/* Issue an RDMA_READ using an FRMR to map the data sink */ +static int rdma_read_chunk_frmr(struct svcxprt_rdma *xprt, struct svc_rqst *rqstp, struct svc_rdma_op_ctxt *head, - struct rpcrdma_msg *rmsgp, - struct svc_rdma_req_map *rpl_map, - struct svc_rdma_req_map *chl_map, - int ch_count, - int byte_count) + int *page_no, + u32 *page_offset, + u32 rs_handle, + u32 rs_length, + u64 rs_offset, + int last) { - int page_no; - int ch_no; - u32 offset; - struct rpcrdma_read_chunk *ch; - struct svc_rdma_fastreg_mr *frmr; - int ret = 0; + struct ib_send_wr read_wr; + struct ib_send_wr inv_wr; + struct ib_send_wr fastreg_wr; + u8 key; + int pages_needed = PAGE_ALIGN(*page_offset + rs_length) >> PAGE_SHIFT; + struct svc_rdma_op_ctxt *ctxt = svc_rdma_get_context(xprt); + struct svc_rdma_fastreg_mr *frmr = svc_rdma_get_frmr(xprt); + int ret, read, pno; + u32 pg_off = *page_offset; + u32 pg_no = *page_no; - frmr = svc_rdma_get_frmr(xprt); if (IS_ERR(frmr)) return -ENOMEM; - head->frmr = frmr; - head->arg.head[0] = rqstp->rq_arg.head[0]; - head->arg.tail[0] = rqstp->rq_arg.tail[0]; - head->arg.pages = &head->pages[head->count]; - head->hdr_count = head->count; /* save count of hdr pages */ - head->arg.page_base = 0; - head->arg.page_len = byte_count; - head->arg.len = rqstp->rq_arg.len + byte_count; - head->arg.buflen = rqstp->rq_arg.buflen + byte_count; + ctxt->direction = DMA_FROM_DEVICE; + ctxt->frmr = frmr; + pages_needed = min_t(int, pages_needed, xprt->sc_frmr_pg_list_len); + read = min_t(int, pages_needed << PAGE_SHIFT, rs_length); - /* Fast register the page list */ - frmr->kva = page_address(rqstp->rq_arg.pages[0]); + frmr->kva = page_address(rqstp->rq_arg.pages[pg_no]); frmr->direction = DMA_FROM_DEVICE; frmr->access_flags = (IB_ACCESS_LOCAL_WRITE|IB_ACCESS_REMOTE_WRITE); - frmr->map_len = byte_count; - frmr->page_list_len = PAGE_ALIGN(byte_count) >> PAGE_SHIFT; - for (page_no = 0; page_no < frmr->page_list_len; page_no++) { - frmr->page_list->page_list[page_no] = + frmr->map_len = pages_needed << PAGE_SHIFT; + frmr->page_list_len = pages_needed; + + for (pno = 0; pno < pages_needed; pno++) { + int len = min_t(int, rs_length, PAGE_SIZE - pg_off); + + head->arg.pages[pg_no] = rqstp->rq_arg.pages[pg_no]; + head->arg.page_len += len; + head->arg.len += len; + if (!pg_off) + head->count++; + rqstp->rq_respages = &rqstp->rq_arg.pages[pg_no+1]; + rqstp->rq_next_page = rqstp->rq_respages + 1; + frmr->page_list->page_list[pno] = ib_dma_map_page(xprt->sc_cm_id->device, - rqstp->rq_arg.pages[page_no], 0, + head->arg.pages[pg_no], 0, PAGE_SIZE, DMA_FROM_DEVICE); - if (ib_dma_mapping_error(xprt->sc_cm_id->device, - frmr->page_list->page_list[page_no])) - goto fatal_err; + ret = ib_dma_mapping_error(xprt->sc_cm_id->device, + frmr->page_list->page_list[pno]); + if (ret) + goto err; atomic_inc(&xprt->sc_dma_used); - head->arg.pages[page_no] = rqstp->rq_arg.pages[page_no]; - } - head->count += page_no; - - /* rq_respages points one past arg pages */ - rqstp->rq_respages = &rqstp->rq_arg.pages[page_no]; - rqstp->rq_next_page = rqstp->rq_respages + 1; - /* Create the reply and chunk maps */ - offset = 0; - ch = (struct rpcrdma_read_chunk *)&rmsgp->rm_body.rm_chunks[0]; - for (ch_no = 0; ch_no < ch_count; ch_no++) { - int len = ntohl(ch->rc_target.rs_length); - rpl_map->sge[ch_no].iov_base = frmr->kva + offset; - rpl_map->sge[ch_no].iov_len = len; - chl_map->ch[ch_no].count = 1; - chl_map->ch[ch_no].start = ch_no; - offset += len; - ch++; + /* adjust offset and wrap to next page if needed */ + pg_off += len; + if (pg_off == PAGE_SIZE) { + pg_off = 0; + pg_no++; + } + rs_length -= len; } - ret = svc_rdma_fastreg(xprt, frmr); - if (ret) - goto fatal_err; - - return ch_no; - - fatal_err: - printk("svcrdma: error fast registering xdr for xprt %p", xprt); - svc_rdma_put_frmr(xprt, frmr); - return -EIO; -} - -static int rdma_set_ctxt_sge(struct svcxprt_rdma *xprt, - struct svc_rdma_op_ctxt *ctxt, - struct svc_rdma_fastreg_mr *frmr, - struct kvec *vec, - u64 *sgl_offset, - int count) -{ - int i; - unsigned long off; + if (last && rs_length == 0) + set_bit(RDMACTXT_F_LAST_CTXT, &ctxt->flags); + else + clear_bit(RDMACTXT_F_LAST_CTXT, &ctxt->flags); - ctxt->count = count; - ctxt->direction = DMA_FROM_DEVICE; - for (i = 0; i < count; i++) { - ctxt->sge[i].length = 0; /* in case map fails */ - if (!frmr) { - BUG_ON(!virt_to_page(vec[i].iov_base)); - off = (unsigned long)vec[i].iov_base & ~PAGE_MASK; - ctxt->sge[i].addr = - ib_dma_map_page(xprt->sc_cm_id->device, - virt_to_page(vec[i].iov_base), - off, - vec[i].iov_len, - DMA_FROM_DEVICE); - if (ib_dma_mapping_error(xprt->sc_cm_id->device, - ctxt->sge[i].addr)) - return -EINVAL; - ctxt->sge[i].lkey = xprt->sc_dma_lkey; - atomic_inc(&xprt->sc_dma_used); - } else { - ctxt->sge[i].addr = (unsigned long)vec[i].iov_base; - ctxt->sge[i].lkey = frmr->mr->lkey; - } - ctxt->sge[i].length = vec[i].iov_len; - *sgl_offset = *sgl_offset + vec[i].iov_len; + /* Bump the key */ + key = (u8)(frmr->mr->lkey & 0x000000FF); + ib_update_fast_reg_key(frmr->mr, ++key); + + ctxt->sge[0].addr = (unsigned long)frmr->kva + *page_offset; + ctxt->sge[0].lkey = frmr->mr->lkey; + ctxt->sge[0].length = read; + ctxt->count = 1; + ctxt->read_hdr = head; + + /* Prepare FASTREG WR */ + memset(&fastreg_wr, 0, sizeof(fastreg_wr)); + fastreg_wr.opcode = IB_WR_FAST_REG_MR; + fastreg_wr.send_flags = IB_SEND_SIGNALED; + fastreg_wr.wr.fast_reg.iova_start = (unsigned long)frmr->kva; + fastreg_wr.wr.fast_reg.page_list = frmr->page_list; + fastreg_wr.wr.fast_reg.page_list_len = frmr->page_list_len; + fastreg_wr.wr.fast_reg.page_shift = PAGE_SHIFT; + fastreg_wr.wr.fast_reg.length = frmr->map_len; + fastreg_wr.wr.fast_reg.access_flags = frmr->access_flags; + fastreg_wr.wr.fast_reg.rkey = frmr->mr->lkey; + fastreg_wr.next = &read_wr; + + /* Prepare RDMA_READ */ + memset(&read_wr, 0, sizeof(read_wr)); + read_wr.send_flags = IB_SEND_SIGNALED; + read_wr.wr.rdma.rkey = rs_handle; + read_wr.wr.rdma.remote_addr = rs_offset; + read_wr.sg_list = ctxt->sge; + read_wr.num_sge = 1; + if (xprt->sc_dev_caps & SVCRDMA_DEVCAP_READ_W_INV) { + read_wr.opcode = IB_WR_RDMA_READ_WITH_INV; + read_wr.wr_id = (unsigned long)ctxt; + read_wr.ex.invalidate_rkey = ctxt->frmr->mr->lkey; + } else { + read_wr.opcode = IB_WR_RDMA_READ; + read_wr.next = &inv_wr; + /* Prepare invalidate */ + memset(&inv_wr, 0, sizeof(inv_wr)); + inv_wr.wr_id = (unsigned long)ctxt; + inv_wr.opcode = IB_WR_LOCAL_INV; + inv_wr.send_flags = IB_SEND_SIGNALED | IB_SEND_FENCE; + inv_wr.ex.invalidate_rkey = frmr->mr->lkey; + } + ctxt->wr_op = read_wr.opcode; + + /* Post the chain */ + ret = svc_rdma_send(xprt, &fastreg_wr); + if (ret) { + pr_err("svcrdma: Error %d posting RDMA_READ\n", ret); + set_bit(XPT_CLOSE, &xprt->sc_xprt.xpt_flags); + goto err; } - return 0; -} -static int rdma_read_max_sge(struct svcxprt_rdma *xprt, int sge_count) -{ - if ((rdma_node_get_transport(xprt->sc_cm_id->device->node_type) == - RDMA_TRANSPORT_IWARP) && - sge_count > 1) - return 1; - else - return min_t(int, sge_count, xprt->sc_max_sge); + /* return current location in page array */ + *page_no = pg_no; + *page_offset = pg_off; + ret = read; + atomic_inc(&rdma_stat_read); + return ret; + err: + svc_rdma_unmap_dma(ctxt); + svc_rdma_put_context(ctxt, 0); + svc_rdma_put_frmr(xprt, frmr); + return ret; } -/* - * Use RDMA_READ to read data from the advertised client buffer into the - * XDR stream starting at rq_arg.head[0].iov_base. - * Each chunk in the array - * contains the following fields: - * discrim - '1', This isn't used for data placement - * position - The xdr stream offset (the same for every chunk) - * handle - RMR for client memory region - * length - data transfer length - * offset - 64 bit tagged offset in remote memory region - * - * On our side, we need to read into a pagelist. The first page immediately - * follows the RPC header. - * - * This function returns: - * 0 - No error and no read-list found. - * - * 1 - Successful read-list processing. The data is not yet in - * the pagelist and therefore the RPC request must be deferred. The - * I/O completion will enqueue the transport again and - * svc_rdma_recvfrom will complete the request. - * - * <0 - Error processing/posting read-list. - * - * NOTE: The ctxt must not be touched after the last WR has been posted - * because the I/O completion processing may occur on another - * processor and free / modify the context. Ne touche pas! - */ -static int rdma_read_xdr(struct svcxprt_rdma *xprt, - struct rpcrdma_msg *rmsgp, - struct svc_rqst *rqstp, - struct svc_rdma_op_ctxt *hdr_ctxt) +static int rdma_read_chunks(struct svcxprt_rdma *xprt, + struct rpcrdma_msg *rmsgp, + struct svc_rqst *rqstp, + struct svc_rdma_op_ctxt *head) { - struct ib_send_wr read_wr; - struct ib_send_wr inv_wr; - int err = 0; - int ch_no; - int ch_count; - int byte_count; - int sge_count; - u64 sgl_offset; + int page_no, ch_count, ret; struct rpcrdma_read_chunk *ch; - struct svc_rdma_op_ctxt *ctxt = NULL; - struct svc_rdma_req_map *rpl_map; - struct svc_rdma_req_map *chl_map; + u32 page_offset, byte_count; + u64 rs_offset; + rdma_reader_fn reader; /* If no read list is present, return 0 */ ch = svc_rdma_get_read_chunk(rmsgp); @@ -408,122 +384,55 @@ static int rdma_read_xdr(struct svcxprt_rdma *xprt, if (ch_count > RPCSVC_MAXPAGES) return -EINVAL; - /* Allocate temporary reply and chunk maps */ - rpl_map = svc_rdma_get_req_map(); - chl_map = svc_rdma_get_req_map(); + /* The request is completed when the RDMA_READs complete. The + * head context keeps all the pages that comprise the + * request. + */ + head->arg.head[0] = rqstp->rq_arg.head[0]; + head->arg.tail[0] = rqstp->rq_arg.tail[0]; + head->arg.pages = &head->pages[head->count]; + head->hdr_count = head->count; + head->arg.page_base = 0; + head->arg.page_len = 0; + head->arg.len = rqstp->rq_arg.len; + head->arg.buflen = rqstp->rq_arg.buflen; - if (!xprt->sc_frmr_pg_list_len) - sge_count = map_read_chunks(xprt, rqstp, hdr_ctxt, rmsgp, - rpl_map, chl_map, ch_count, - byte_count); + /* Use FRMR if supported */ + if (xprt->sc_dev_caps & SVCRDMA_DEVCAP_FAST_REG) + reader = rdma_read_chunk_frmr; else - sge_count = fast_reg_read_chunks(xprt, rqstp, hdr_ctxt, rmsgp, - rpl_map, chl_map, ch_count, - byte_count); - if (sge_count < 0) { - err = -EIO; - goto out; - } - - sgl_offset = 0; - ch_no = 0; + reader = rdma_read_chunk_lcl; + page_no = 0; page_offset = 0; for (ch = (struct rpcrdma_read_chunk *)&rmsgp->rm_body.rm_chunks[0]; - ch->rc_discrim != 0; ch++, ch_no++) { - u64 rs_offset; -next_sge: - ctxt = svc_rdma_get_context(xprt); - ctxt->direction = DMA_FROM_DEVICE; - ctxt->frmr = hdr_ctxt->frmr; - ctxt->read_hdr = NULL; - clear_bit(RDMACTXT_F_LAST_CTXT, &ctxt->flags); - clear_bit(RDMACTXT_F_FAST_UNREG, &ctxt->flags); + ch->rc_discrim != 0; ch++) { - /* Prepare READ WR */ - memset(&read_wr, 0, sizeof read_wr); - read_wr.wr_id = (unsigned long)ctxt; - read_wr.opcode = IB_WR_RDMA_READ; - ctxt->wr_op = read_wr.opcode; - read_wr.send_flags = IB_SEND_SIGNALED; - read_wr.wr.rdma.rkey = ntohl(ch->rc_target.rs_handle); xdr_decode_hyper((__be32 *)&ch->rc_target.rs_offset, &rs_offset); - read_wr.wr.rdma.remote_addr = rs_offset + sgl_offset; - read_wr.sg_list = ctxt->sge; - read_wr.num_sge = - rdma_read_max_sge(xprt, chl_map->ch[ch_no].count); - err = rdma_set_ctxt_sge(xprt, ctxt, hdr_ctxt->frmr, - &rpl_map->sge[chl_map->ch[ch_no].start], - &sgl_offset, - read_wr.num_sge); - if (err) { - svc_rdma_unmap_dma(ctxt); - svc_rdma_put_context(ctxt, 0); - goto out; - } - if (((ch+1)->rc_discrim == 0) && - (read_wr.num_sge == chl_map->ch[ch_no].count)) { - /* - * Mark the last RDMA_READ with a bit to - * indicate all RPC data has been fetched from - * the client and the RPC needs to be enqueued. - */ - set_bit(RDMACTXT_F_LAST_CTXT, &ctxt->flags); - if (hdr_ctxt->frmr) { - set_bit(RDMACTXT_F_FAST_UNREG, &ctxt->flags); - /* - * Invalidate the local MR used to map the data - * sink. - */ - if (xprt->sc_dev_caps & - SVCRDMA_DEVCAP_READ_W_INV) { - read_wr.opcode = - IB_WR_RDMA_READ_WITH_INV; - ctxt->wr_op = read_wr.opcode; - read_wr.ex.invalidate_rkey = - ctxt->frmr->mr->lkey; - } else { - /* Prepare INVALIDATE WR */ - memset(&inv_wr, 0, sizeof inv_wr); - inv_wr.opcode = IB_WR_LOCAL_INV; - inv_wr.send_flags = IB_SEND_SIGNALED; - inv_wr.ex.invalidate_rkey = - hdr_ctxt->frmr->mr->lkey; - read_wr.next = &inv_wr; - } - } - ctxt->read_hdr = hdr_ctxt; - } - /* Post the read */ - err = svc_rdma_send(xprt, &read_wr); - if (err) { - printk(KERN_ERR "svcrdma: Error %d posting RDMA_READ\n", - err); - set_bit(XPT_CLOSE, &xprt->sc_xprt.xpt_flags); - svc_rdma_unmap_dma(ctxt); - svc_rdma_put_context(ctxt, 0); - goto out; + byte_count = ntohl(ch->rc_target.rs_length); + + while (byte_count > 0) { + ret = reader(xprt, rqstp, head, + &page_no, &page_offset, + ntohl(ch->rc_target.rs_handle), + byte_count, rs_offset, + ((ch+1)->rc_discrim == 0) /* last */ + ); + if (ret < 0) + goto err; + byte_count -= ret; + rs_offset += ret; + head->arg.buflen += ret; } - atomic_inc(&rdma_stat_read); - - if (read_wr.num_sge < chl_map->ch[ch_no].count) { - chl_map->ch[ch_no].count -= read_wr.num_sge; - chl_map->ch[ch_no].start += read_wr.num_sge; - goto next_sge; - } - sgl_offset = 0; - err = 1; } - - out: - svc_rdma_put_req_map(rpl_map); - svc_rdma_put_req_map(chl_map); - + ret = 1; + err: /* Detach arg pages. svc_recv will replenish them */ - for (ch_no = 0; &rqstp->rq_pages[ch_no] < rqstp->rq_respages; ch_no++) - rqstp->rq_pages[ch_no] = NULL; + for (page_no = 0; + &rqstp->rq_pages[page_no] < rqstp->rq_respages; page_no++) + rqstp->rq_pages[page_no] = NULL; - return err; + return ret; } static int rdma_read_complete(struct svc_rqst *rqstp, @@ -595,13 +504,9 @@ int svc_rdma_recvfrom(struct svc_rqst *rqstp) struct svc_rdma_op_ctxt, dto_q); list_del_init(&ctxt->dto_q); - } - if (ctxt) { spin_unlock_bh(&rdma_xprt->sc_rq_dto_lock); return rdma_read_complete(rqstp, ctxt); - } - - if (!list_empty(&rdma_xprt->sc_rq_dto_q)) { + } else if (!list_empty(&rdma_xprt->sc_rq_dto_q)) { ctxt = list_entry(rdma_xprt->sc_rq_dto_q.next, struct svc_rdma_op_ctxt, dto_q); @@ -621,7 +526,6 @@ int svc_rdma_recvfrom(struct svc_rqst *rqstp) if (test_bit(XPT_CLOSE, &xprt->xpt_flags)) goto close_out; - BUG_ON(ret); goto out; } dprintk("svcrdma: processing ctxt=%p on xprt=%p, rqstp=%p, status=%d\n", @@ -644,12 +548,11 @@ int svc_rdma_recvfrom(struct svc_rqst *rqstp) } /* Read read-list data. */ - ret = rdma_read_xdr(rdma_xprt, rmsgp, rqstp, ctxt); + ret = rdma_read_chunks(rdma_xprt, rmsgp, rqstp, ctxt); if (ret > 0) { /* read-list posted, defer until data received from client. */ goto defer; - } - if (ret < 0) { + } else if (ret < 0) { /* Post of read-list failed, free context. */ svc_rdma_put_context(ctxt, 1); return 0; diff --git a/net/sunrpc/xprtrdma/svc_rdma_sendto.c b/net/sunrpc/xprtrdma/svc_rdma_sendto.c index 7e024a51617e..49fd21a5c215 100644 --- a/net/sunrpc/xprtrdma/svc_rdma_sendto.c +++ b/net/sunrpc/xprtrdma/svc_rdma_sendto.c @@ -1,4 +1,5 @@ /* + * Copyright (c) 2014 Open Grid Computing, Inc. All rights reserved. * Copyright (c) 2005-2006 Network Appliance, Inc. All rights reserved. * * This software is available to you under a choice of one of two @@ -49,152 +50,6 @@ #define RPCDBG_FACILITY RPCDBG_SVCXPRT -/* Encode an XDR as an array of IB SGE - * - * Assumptions: - * - head[0] is physically contiguous. - * - tail[0] is physically contiguous. - * - pages[] is not physically or virtually contiguous and consists of - * PAGE_SIZE elements. - * - * Output: - * SGE[0] reserved for RCPRDMA header - * SGE[1] data from xdr->head[] - * SGE[2..sge_count-2] data from xdr->pages[] - * SGE[sge_count-1] data from xdr->tail. - * - * The max SGE we need is the length of the XDR / pagesize + one for - * head + one for tail + one for RPCRDMA header. Since RPCSVC_MAXPAGES - * reserves a page for both the request and the reply header, and this - * array is only concerned with the reply we are assured that we have - * on extra page for the RPCRMDA header. - */ -static int fast_reg_xdr(struct svcxprt_rdma *xprt, - struct xdr_buf *xdr, - struct svc_rdma_req_map *vec) -{ - int sge_no; - u32 sge_bytes; - u32 page_bytes; - u32 page_off; - int page_no = 0; - u8 *frva; - struct svc_rdma_fastreg_mr *frmr; - - frmr = svc_rdma_get_frmr(xprt); - if (IS_ERR(frmr)) - return -ENOMEM; - vec->frmr = frmr; - - /* Skip the RPCRDMA header */ - sge_no = 1; - - /* Map the head. */ - frva = (void *)((unsigned long)(xdr->head[0].iov_base) & PAGE_MASK); - vec->sge[sge_no].iov_base = xdr->head[0].iov_base; - vec->sge[sge_no].iov_len = xdr->head[0].iov_len; - vec->count = 2; - sge_no++; - - /* Map the XDR head */ - frmr->kva = frva; - frmr->direction = DMA_TO_DEVICE; - frmr->access_flags = 0; - frmr->map_len = PAGE_SIZE; - frmr->page_list_len = 1; - page_off = (unsigned long)xdr->head[0].iov_base & ~PAGE_MASK; - frmr->page_list->page_list[page_no] = - ib_dma_map_page(xprt->sc_cm_id->device, - virt_to_page(xdr->head[0].iov_base), - page_off, - PAGE_SIZE - page_off, - DMA_TO_DEVICE); - if (ib_dma_mapping_error(xprt->sc_cm_id->device, - frmr->page_list->page_list[page_no])) - goto fatal_err; - atomic_inc(&xprt->sc_dma_used); - - /* Map the XDR page list */ - page_off = xdr->page_base; - page_bytes = xdr->page_len + page_off; - if (!page_bytes) - goto encode_tail; - - /* Map the pages */ - vec->sge[sge_no].iov_base = frva + frmr->map_len + page_off; - vec->sge[sge_no].iov_len = page_bytes; - sge_no++; - while (page_bytes) { - struct page *page; - - page = xdr->pages[page_no++]; - sge_bytes = min_t(u32, page_bytes, (PAGE_SIZE - page_off)); - page_bytes -= sge_bytes; - - frmr->page_list->page_list[page_no] = - ib_dma_map_page(xprt->sc_cm_id->device, - page, page_off, - sge_bytes, DMA_TO_DEVICE); - if (ib_dma_mapping_error(xprt->sc_cm_id->device, - frmr->page_list->page_list[page_no])) - goto fatal_err; - - atomic_inc(&xprt->sc_dma_used); - page_off = 0; /* reset for next time through loop */ - frmr->map_len += PAGE_SIZE; - frmr->page_list_len++; - } - vec->count++; - - encode_tail: - /* Map tail */ - if (0 == xdr->tail[0].iov_len) - goto done; - - vec->count++; - vec->sge[sge_no].iov_len = xdr->tail[0].iov_len; - - if (((unsigned long)xdr->tail[0].iov_base & PAGE_MASK) == - ((unsigned long)xdr->head[0].iov_base & PAGE_MASK)) { - /* - * If head and tail use the same page, we don't need - * to map it again. - */ - vec->sge[sge_no].iov_base = xdr->tail[0].iov_base; - } else { - void *va; - - /* Map another page for the tail */ - page_off = (unsigned long)xdr->tail[0].iov_base & ~PAGE_MASK; - va = (void *)((unsigned long)xdr->tail[0].iov_base & PAGE_MASK); - vec->sge[sge_no].iov_base = frva + frmr->map_len + page_off; - - frmr->page_list->page_list[page_no] = - ib_dma_map_page(xprt->sc_cm_id->device, virt_to_page(va), - page_off, - PAGE_SIZE, - DMA_TO_DEVICE); - if (ib_dma_mapping_error(xprt->sc_cm_id->device, - frmr->page_list->page_list[page_no])) - goto fatal_err; - atomic_inc(&xprt->sc_dma_used); - frmr->map_len += PAGE_SIZE; - frmr->page_list_len++; - } - - done: - if (svc_rdma_fastreg(xprt, frmr)) - goto fatal_err; - - return 0; - - fatal_err: - printk("svcrdma: Error fast registering memory for xprt %p\n", xprt); - vec->frmr = NULL; - svc_rdma_put_frmr(xprt, frmr); - return -EIO; -} - static int map_xdr(struct svcxprt_rdma *xprt, struct xdr_buf *xdr, struct svc_rdma_req_map *vec) @@ -208,9 +63,6 @@ static int map_xdr(struct svcxprt_rdma *xprt, BUG_ON(xdr->len != (xdr->head[0].iov_len + xdr->page_len + xdr->tail[0].iov_len)); - if (xprt->sc_frmr_pg_list_len) - return fast_reg_xdr(xprt, xdr, vec); - /* Skip the first sge, this is for the RPCRDMA header */ sge_no = 1; @@ -282,8 +134,6 @@ static dma_addr_t dma_map_xdr(struct svcxprt_rdma *xprt, } /* Assumptions: - * - We are using FRMR - * - or - * - The specified write_len can be represented in sc_max_sge * PAGE_SIZE */ static int send_write(struct svcxprt_rdma *xprt, struct svc_rqst *rqstp, @@ -327,23 +177,16 @@ static int send_write(struct svcxprt_rdma *xprt, struct svc_rqst *rqstp, sge_bytes = min_t(size_t, bc, vec->sge[xdr_sge_no].iov_len-sge_off); sge[sge_no].length = sge_bytes; - if (!vec->frmr) { - sge[sge_no].addr = - dma_map_xdr(xprt, &rqstp->rq_res, xdr_off, - sge_bytes, DMA_TO_DEVICE); - xdr_off += sge_bytes; - if (ib_dma_mapping_error(xprt->sc_cm_id->device, - sge[sge_no].addr)) - goto err; - atomic_inc(&xprt->sc_dma_used); - sge[sge_no].lkey = xprt->sc_dma_lkey; - } else { - sge[sge_no].addr = (unsigned long) - vec->sge[xdr_sge_no].iov_base + sge_off; - sge[sge_no].lkey = vec->frmr->mr->lkey; - } + sge[sge_no].addr = + dma_map_xdr(xprt, &rqstp->rq_res, xdr_off, + sge_bytes, DMA_TO_DEVICE); + xdr_off += sge_bytes; + if (ib_dma_mapping_error(xprt->sc_cm_id->device, + sge[sge_no].addr)) + goto err; + atomic_inc(&xprt->sc_dma_used); + sge[sge_no].lkey = xprt->sc_dma_lkey; ctxt->count++; - ctxt->frmr = vec->frmr; sge_off = 0; sge_no++; xdr_sge_no++; @@ -369,7 +212,6 @@ static int send_write(struct svcxprt_rdma *xprt, struct svc_rqst *rqstp, return 0; err: svc_rdma_unmap_dma(ctxt); - svc_rdma_put_frmr(xprt, vec->frmr); svc_rdma_put_context(ctxt, 0); /* Fatal error, close transport */ return -EIO; @@ -397,10 +239,7 @@ static int send_write_chunks(struct svcxprt_rdma *xprt, res_ary = (struct rpcrdma_write_array *) &rdma_resp->rm_body.rm_chunks[1]; - if (vec->frmr) - max_write = vec->frmr->map_len; - else - max_write = xprt->sc_max_sge * PAGE_SIZE; + max_write = xprt->sc_max_sge * PAGE_SIZE; /* Write chunks start at the pagelist */ for (xdr_off = rqstp->rq_res.head[0].iov_len, chunk_no = 0; @@ -472,10 +311,7 @@ static int send_reply_chunks(struct svcxprt_rdma *xprt, res_ary = (struct rpcrdma_write_array *) &rdma_resp->rm_body.rm_chunks[2]; - if (vec->frmr) - max_write = vec->frmr->map_len; - else - max_write = xprt->sc_max_sge * PAGE_SIZE; + max_write = xprt->sc_max_sge * PAGE_SIZE; /* xdr offset starts at RPC message */ nchunks = ntohl(arg_ary->wc_nchunks); @@ -545,7 +381,6 @@ static int send_reply(struct svcxprt_rdma *rdma, int byte_count) { struct ib_send_wr send_wr; - struct ib_send_wr inv_wr; int sge_no; int sge_bytes; int page_no; @@ -559,7 +394,6 @@ static int send_reply(struct svcxprt_rdma *rdma, "svcrdma: could not post a receive buffer, err=%d." "Closing transport %p.\n", ret, rdma); set_bit(XPT_CLOSE, &rdma->sc_xprt.xpt_flags); - svc_rdma_put_frmr(rdma, vec->frmr); svc_rdma_put_context(ctxt, 0); return -ENOTCONN; } @@ -567,11 +401,6 @@ static int send_reply(struct svcxprt_rdma *rdma, /* Prepare the context */ ctxt->pages[0] = page; ctxt->count = 1; - ctxt->frmr = vec->frmr; - if (vec->frmr) - set_bit(RDMACTXT_F_FAST_UNREG, &ctxt->flags); - else - clear_bit(RDMACTXT_F_FAST_UNREG, &ctxt->flags); /* Prepare the SGE for the RPCRDMA Header */ ctxt->sge[0].lkey = rdma->sc_dma_lkey; @@ -590,21 +419,15 @@ static int send_reply(struct svcxprt_rdma *rdma, int xdr_off = 0; sge_bytes = min_t(size_t, vec->sge[sge_no].iov_len, byte_count); byte_count -= sge_bytes; - if (!vec->frmr) { - ctxt->sge[sge_no].addr = - dma_map_xdr(rdma, &rqstp->rq_res, xdr_off, - sge_bytes, DMA_TO_DEVICE); - xdr_off += sge_bytes; - if (ib_dma_mapping_error(rdma->sc_cm_id->device, - ctxt->sge[sge_no].addr)) - goto err; - atomic_inc(&rdma->sc_dma_used); - ctxt->sge[sge_no].lkey = rdma->sc_dma_lkey; - } else { - ctxt->sge[sge_no].addr = (unsigned long) - vec->sge[sge_no].iov_base; - ctxt->sge[sge_no].lkey = vec->frmr->mr->lkey; - } + ctxt->sge[sge_no].addr = + dma_map_xdr(rdma, &rqstp->rq_res, xdr_off, + sge_bytes, DMA_TO_DEVICE); + xdr_off += sge_bytes; + if (ib_dma_mapping_error(rdma->sc_cm_id->device, + ctxt->sge[sge_no].addr)) + goto err; + atomic_inc(&rdma->sc_dma_used); + ctxt->sge[sge_no].lkey = rdma->sc_dma_lkey; ctxt->sge[sge_no].length = sge_bytes; } BUG_ON(byte_count != 0); @@ -627,6 +450,7 @@ static int send_reply(struct svcxprt_rdma *rdma, ctxt->sge[page_no+1].length = 0; } rqstp->rq_next_page = rqstp->rq_respages + 1; + BUG_ON(sge_no > rdma->sc_max_sge); memset(&send_wr, 0, sizeof send_wr); ctxt->wr_op = IB_WR_SEND; @@ -635,15 +459,6 @@ static int send_reply(struct svcxprt_rdma *rdma, send_wr.num_sge = sge_no; send_wr.opcode = IB_WR_SEND; send_wr.send_flags = IB_SEND_SIGNALED; - if (vec->frmr) { - /* Prepare INVALIDATE WR */ - memset(&inv_wr, 0, sizeof inv_wr); - inv_wr.opcode = IB_WR_LOCAL_INV; - inv_wr.send_flags = IB_SEND_SIGNALED; - inv_wr.ex.invalidate_rkey = - vec->frmr->mr->lkey; - send_wr.next = &inv_wr; - } ret = svc_rdma_send(rdma, &send_wr); if (ret) @@ -653,7 +468,6 @@ static int send_reply(struct svcxprt_rdma *rdma, err: svc_rdma_unmap_dma(ctxt); - svc_rdma_put_frmr(rdma, vec->frmr); svc_rdma_put_context(ctxt, 1); return -EIO; } diff --git a/net/sunrpc/xprtrdma/svc_rdma_transport.c b/net/sunrpc/xprtrdma/svc_rdma_transport.c index 25688fa2207f..e7323fbbd348 100644 --- a/net/sunrpc/xprtrdma/svc_rdma_transport.c +++ b/net/sunrpc/xprtrdma/svc_rdma_transport.c @@ -1,4 +1,5 @@ /* + * Copyright (c) 2014 Open Grid Computing, Inc. All rights reserved. * Copyright (c) 2005-2007 Network Appliance, Inc. All rights reserved. * * This software is available to you under a choice of one of two @@ -65,6 +66,7 @@ static void dto_tasklet_func(unsigned long data); static void svc_rdma_detach(struct svc_xprt *xprt); static void svc_rdma_free(struct svc_xprt *xprt); static int svc_rdma_has_wspace(struct svc_xprt *xprt); +static int svc_rdma_secure_port(struct svc_rqst *); static void rq_cq_reap(struct svcxprt_rdma *xprt); static void sq_cq_reap(struct svcxprt_rdma *xprt); @@ -82,6 +84,7 @@ static struct svc_xprt_ops svc_rdma_ops = { .xpo_prep_reply_hdr = svc_rdma_prep_reply_hdr, .xpo_has_wspace = svc_rdma_has_wspace, .xpo_accept = svc_rdma_accept, + .xpo_secure_port = svc_rdma_secure_port, }; struct svc_xprt_class svc_rdma_class = { @@ -160,7 +163,6 @@ struct svc_rdma_req_map *svc_rdma_get_req_map(void) schedule_timeout_uninterruptible(msecs_to_jiffies(500)); } map->count = 0; - map->frmr = NULL; return map; } @@ -336,22 +338,21 @@ static void process_context(struct svcxprt_rdma *xprt, switch (ctxt->wr_op) { case IB_WR_SEND: - if (test_bit(RDMACTXT_F_FAST_UNREG, &ctxt->flags)) - svc_rdma_put_frmr(xprt, ctxt->frmr); + BUG_ON(ctxt->frmr); svc_rdma_put_context(ctxt, 1); break; case IB_WR_RDMA_WRITE: + BUG_ON(ctxt->frmr); svc_rdma_put_context(ctxt, 0); break; case IB_WR_RDMA_READ: case IB_WR_RDMA_READ_WITH_INV: + svc_rdma_put_frmr(xprt, ctxt->frmr); if (test_bit(RDMACTXT_F_LAST_CTXT, &ctxt->flags)) { struct svc_rdma_op_ctxt *read_hdr = ctxt->read_hdr; BUG_ON(!read_hdr); - if (test_bit(RDMACTXT_F_FAST_UNREG, &ctxt->flags)) - svc_rdma_put_frmr(xprt, ctxt->frmr); spin_lock_bh(&xprt->sc_rq_dto_lock); set_bit(XPT_DATA, &xprt->sc_xprt.xpt_flags); list_add_tail(&read_hdr->dto_q, @@ -363,6 +364,7 @@ static void process_context(struct svcxprt_rdma *xprt, break; default: + BUG_ON(1); printk(KERN_ERR "svcrdma: unexpected completion type, " "opcode=%d\n", ctxt->wr_op); @@ -378,29 +380,42 @@ static void process_context(struct svcxprt_rdma *xprt, static void sq_cq_reap(struct svcxprt_rdma *xprt) { struct svc_rdma_op_ctxt *ctxt = NULL; - struct ib_wc wc; + struct ib_wc wc_a[6]; + struct ib_wc *wc; struct ib_cq *cq = xprt->sc_sq_cq; int ret; + memset(wc_a, 0, sizeof(wc_a)); + if (!test_and_clear_bit(RDMAXPRT_SQ_PENDING, &xprt->sc_flags)) return; ib_req_notify_cq(xprt->sc_sq_cq, IB_CQ_NEXT_COMP); atomic_inc(&rdma_stat_sq_poll); - while ((ret = ib_poll_cq(cq, 1, &wc)) > 0) { - if (wc.status != IB_WC_SUCCESS) - /* Close the transport */ - set_bit(XPT_CLOSE, &xprt->sc_xprt.xpt_flags); + while ((ret = ib_poll_cq(cq, ARRAY_SIZE(wc_a), wc_a)) > 0) { + int i; - /* Decrement used SQ WR count */ - atomic_dec(&xprt->sc_sq_count); - wake_up(&xprt->sc_send_wait); + for (i = 0; i < ret; i++) { + wc = &wc_a[i]; + if (wc->status != IB_WC_SUCCESS) { + dprintk("svcrdma: sq wc err status %d\n", + wc->status); - ctxt = (struct svc_rdma_op_ctxt *)(unsigned long)wc.wr_id; - if (ctxt) - process_context(xprt, ctxt); + /* Close the transport */ + set_bit(XPT_CLOSE, &xprt->sc_xprt.xpt_flags); + } - svc_xprt_put(&xprt->sc_xprt); + /* Decrement used SQ WR count */ + atomic_dec(&xprt->sc_sq_count); + wake_up(&xprt->sc_send_wait); + + ctxt = (struct svc_rdma_op_ctxt *) + (unsigned long)wc->wr_id; + if (ctxt) + process_context(xprt, ctxt); + + svc_xprt_put(&xprt->sc_xprt); + } } if (ctxt) @@ -993,7 +1008,11 @@ static struct svc_xprt *svc_rdma_accept(struct svc_xprt *xprt) need_dma_mr = 0; break; case RDMA_TRANSPORT_IB: - if (!(devattr.device_cap_flags & IB_DEVICE_LOCAL_DMA_LKEY)) { + if (!(newxprt->sc_dev_caps & SVCRDMA_DEVCAP_FAST_REG)) { + need_dma_mr = 1; + dma_mr_acc = IB_ACCESS_LOCAL_WRITE; + } else if (!(devattr.device_cap_flags & + IB_DEVICE_LOCAL_DMA_LKEY)) { need_dma_mr = 1; dma_mr_acc = IB_ACCESS_LOCAL_WRITE; } else @@ -1190,14 +1209,7 @@ static int svc_rdma_has_wspace(struct svc_xprt *xprt) container_of(xprt, struct svcxprt_rdma, sc_xprt); /* - * If there are fewer SQ WR available than required to send a - * simple response, return false. - */ - if ((rdma->sc_sq_depth - atomic_read(&rdma->sc_sq_count) < 3)) - return 0; - - /* - * ...or there are already waiters on the SQ, + * If there are already waiters on the SQ, * return false. */ if (waitqueue_active(&rdma->sc_send_wait)) @@ -1207,6 +1219,11 @@ static int svc_rdma_has_wspace(struct svc_xprt *xprt) return 1; } +static int svc_rdma_secure_port(struct svc_rqst *rqstp) +{ + return 1; +} + /* * Attempt to register the kvec representing the RPC memory with the * device. diff --git a/net/sunrpc/xprtrdma/transport.c b/net/sunrpc/xprtrdma/transport.c index 1eb9c468d0c9..66f91f0d071a 100644 --- a/net/sunrpc/xprtrdma/transport.c +++ b/net/sunrpc/xprtrdma/transport.c @@ -149,6 +149,11 @@ static struct ctl_table sunrpc_table[] = { #endif +#define RPCRDMA_BIND_TO (60U * HZ) +#define RPCRDMA_INIT_REEST_TO (5U * HZ) +#define RPCRDMA_MAX_REEST_TO (30U * HZ) +#define RPCRDMA_IDLE_DISC_TO (5U * 60 * HZ) + static struct rpc_xprt_ops xprt_rdma_procs; /* forward reference */ static void @@ -229,7 +234,6 @@ static void xprt_rdma_destroy(struct rpc_xprt *xprt) { struct rpcrdma_xprt *r_xprt = rpcx_to_rdmax(xprt); - int rc; dprintk("RPC: %s: called\n", __func__); @@ -238,10 +242,7 @@ xprt_rdma_destroy(struct rpc_xprt *xprt) xprt_clear_connected(xprt); rpcrdma_buffer_destroy(&r_xprt->rx_buf); - rc = rpcrdma_ep_destroy(&r_xprt->rx_ep, &r_xprt->rx_ia); - if (rc) - dprintk("RPC: %s: rpcrdma_ep_destroy returned %i\n", - __func__, rc); + rpcrdma_ep_destroy(&r_xprt->rx_ep, &r_xprt->rx_ia); rpcrdma_ia_close(&r_xprt->rx_ia); xprt_rdma_free_addresses(xprt); @@ -289,9 +290,9 @@ xprt_setup_rdma(struct xprt_create *args) /* 60 second timeout, no retries */ xprt->timeout = &xprt_rdma_default_timeout; - xprt->bind_timeout = (60U * HZ); - xprt->reestablish_timeout = (5U * HZ); - xprt->idle_timeout = (5U * 60 * HZ); + xprt->bind_timeout = RPCRDMA_BIND_TO; + xprt->reestablish_timeout = RPCRDMA_INIT_REEST_TO; + xprt->idle_timeout = RPCRDMA_IDLE_DISC_TO; xprt->resvport = 0; /* privileged port not needed */ xprt->tsh_size = 0; /* RPC-RDMA handles framing */ @@ -391,7 +392,7 @@ out4: xprt_rdma_free_addresses(xprt); rc = -EINVAL; out3: - (void) rpcrdma_ep_destroy(new_ep, &new_xprt->rx_ia); + rpcrdma_ep_destroy(new_ep, &new_xprt->rx_ia); out2: rpcrdma_ia_close(&new_xprt->rx_ia); out1: @@ -436,10 +437,10 @@ xprt_rdma_connect(struct rpc_xprt *xprt, struct rpc_task *task) schedule_delayed_work(&r_xprt->rdma_connect, xprt->reestablish_timeout); xprt->reestablish_timeout <<= 1; - if (xprt->reestablish_timeout > (30 * HZ)) - xprt->reestablish_timeout = (30 * HZ); - else if (xprt->reestablish_timeout < (5 * HZ)) - xprt->reestablish_timeout = (5 * HZ); + if (xprt->reestablish_timeout > RPCRDMA_MAX_REEST_TO) + xprt->reestablish_timeout = RPCRDMA_MAX_REEST_TO; + else if (xprt->reestablish_timeout < RPCRDMA_INIT_REEST_TO) + xprt->reestablish_timeout = RPCRDMA_INIT_REEST_TO; } else { schedule_delayed_work(&r_xprt->rdma_connect, 0); if (!RPC_IS_ASYNC(task)) @@ -447,23 +448,6 @@ xprt_rdma_connect(struct rpc_xprt *xprt, struct rpc_task *task) } } -static int -xprt_rdma_reserve_xprt(struct rpc_xprt *xprt, struct rpc_task *task) -{ - struct rpcrdma_xprt *r_xprt = rpcx_to_rdmax(xprt); - int credits = atomic_read(&r_xprt->rx_buf.rb_credits); - - /* == RPC_CWNDSCALE @ init, but *after* setup */ - if (r_xprt->rx_buf.rb_cwndscale == 0UL) { - r_xprt->rx_buf.rb_cwndscale = xprt->cwnd; - dprintk("RPC: %s: cwndscale %lu\n", __func__, - r_xprt->rx_buf.rb_cwndscale); - BUG_ON(r_xprt->rx_buf.rb_cwndscale <= 0); - } - xprt->cwnd = credits * r_xprt->rx_buf.rb_cwndscale; - return xprt_reserve_xprt_cong(xprt, task); -} - /* * The RDMA allocate/free functions need the task structure as a place * to hide the struct rpcrdma_req, which is necessary for the actual send/recv @@ -479,7 +463,8 @@ xprt_rdma_allocate(struct rpc_task *task, size_t size) struct rpcrdma_req *req, *nreq; req = rpcrdma_buffer_get(&rpcx_to_rdmax(xprt)->rx_buf); - BUG_ON(NULL == req); + if (req == NULL) + return NULL; if (size > req->rl_size) { dprintk("RPC: %s: size %zd too large for buffer[%zd]: " @@ -503,18 +488,6 @@ xprt_rdma_allocate(struct rpc_task *task, size_t size) * If the allocation or registration fails, the RPC framework * will (doggedly) retry. */ - if (rpcx_to_rdmax(xprt)->rx_ia.ri_memreg_strategy == - RPCRDMA_BOUNCEBUFFERS) { - /* forced to "pure inline" */ - dprintk("RPC: %s: too much data (%zd) for inline " - "(r/w max %d/%d)\n", __func__, size, - rpcx_to_rdmad(xprt).inline_rsize, - rpcx_to_rdmad(xprt).inline_wsize); - size = req->rl_size; - rpc_exit(task, -EIO); /* fail the operation */ - rpcx_to_rdmax(xprt)->rx_stats.failed_marshal_count++; - goto out; - } if (task->tk_flags & RPC_TASK_SWAPPER) nreq = kmalloc(sizeof *req + size, GFP_ATOMIC); else @@ -543,7 +516,6 @@ xprt_rdma_allocate(struct rpc_task *task, size_t size) req = nreq; } dprintk("RPC: %s: size %zd, request 0x%p\n", __func__, size, req); -out: req->rl_connect_cookie = 0; /* our reserved value */ return req->rl_xdr_buf; @@ -579,9 +551,7 @@ xprt_rdma_free(void *buffer) __func__, rep, (rep && rep->rr_func) ? " (with waiter)" : ""); /* - * Finish the deregistration. When using mw bind, this was - * begun in rpcrdma_reply_handler(). In all other modes, we - * do it here, in thread context. The process is considered + * Finish the deregistration. The process is considered * complete when the rr_func vector becomes NULL - this * was put in place during rpcrdma_reply_handler() - the wait * call below will not block if the dereg is "done". If @@ -590,12 +560,7 @@ xprt_rdma_free(void *buffer) for (i = 0; req->rl_nchunks;) { --req->rl_nchunks; i += rpcrdma_deregister_external( - &req->rl_segments[i], r_xprt, NULL); - } - - if (rep && wait_event_interruptible(rep->rr_unbind, !rep->rr_func)) { - rep->rr_func = NULL; /* abandon the callback */ - req->rl_reply = NULL; + &req->rl_segments[i], r_xprt); } if (req->rl_iov.length == 0) { /* see allocate above */ @@ -630,13 +595,12 @@ xprt_rdma_send_request(struct rpc_task *task) struct rpc_xprt *xprt = rqst->rq_xprt; struct rpcrdma_req *req = rpcr_to_rdmar(rqst); struct rpcrdma_xprt *r_xprt = rpcx_to_rdmax(xprt); + int rc; - /* marshal the send itself */ - if (req->rl_niovs == 0 && rpcrdma_marshal_req(rqst) != 0) { - r_xprt->rx_stats.failed_marshal_count++; - dprintk("RPC: %s: rpcrdma_marshal_req failed\n", - __func__); - return -EIO; + if (req->rl_niovs == 0) { + rc = rpcrdma_marshal_req(rqst); + if (rc < 0) + goto failed_marshal; } if (req->rl_reply == NULL) /* e.g. reconnection */ @@ -660,6 +624,12 @@ xprt_rdma_send_request(struct rpc_task *task) rqst->rq_bytes_sent = 0; return 0; +failed_marshal: + r_xprt->rx_stats.failed_marshal_count++; + dprintk("RPC: %s: rpcrdma_marshal_req failed, status %i\n", + __func__, rc); + if (rc == -EIO) + return -EIO; drop_connection: xprt_disconnect_done(xprt); return -ENOTCONN; /* implies disconnect */ @@ -705,7 +675,7 @@ static void xprt_rdma_print_stats(struct rpc_xprt *xprt, struct seq_file *seq) */ static struct rpc_xprt_ops xprt_rdma_procs = { - .reserve_xprt = xprt_rdma_reserve_xprt, + .reserve_xprt = xprt_reserve_xprt_cong, .release_xprt = xprt_release_xprt_cong, /* sunrpc/xprt.c */ .alloc_slot = xprt_alloc_slot, .release_request = xprt_release_rqst_cong, /* ditto */ diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c index 93726560eaa8..13dbd1c389ff 100644 --- a/net/sunrpc/xprtrdma/verbs.c +++ b/net/sunrpc/xprtrdma/verbs.c @@ -48,8 +48,8 @@ */ #include <linux/interrupt.h> -#include <linux/pci.h> /* for Tavor hack below */ #include <linux/slab.h> +#include <asm/bitops.h> #include "xprt_rdma.h" @@ -142,98 +142,139 @@ rpcrdma_cq_async_error_upcall(struct ib_event *event, void *context) } } -static inline -void rpcrdma_event_process(struct ib_wc *wc) +static void +rpcrdma_sendcq_process_wc(struct ib_wc *wc) { - struct rpcrdma_mw *frmr; - struct rpcrdma_rep *rep = - (struct rpcrdma_rep *)(unsigned long) wc->wr_id; + struct rpcrdma_mw *frmr = (struct rpcrdma_mw *)(unsigned long)wc->wr_id; - dprintk("RPC: %s: event rep %p status %X opcode %X length %u\n", - __func__, rep, wc->status, wc->opcode, wc->byte_len); + dprintk("RPC: %s: frmr %p status %X opcode %d\n", + __func__, frmr, wc->status, wc->opcode); - if (!rep) /* send or bind completion that we don't care about */ + if (wc->wr_id == 0ULL) return; - - if (IB_WC_SUCCESS != wc->status) { - dprintk("RPC: %s: WC opcode %d status %X, connection lost\n", - __func__, wc->opcode, wc->status); - rep->rr_len = ~0U; - if (wc->opcode != IB_WC_FAST_REG_MR && wc->opcode != IB_WC_LOCAL_INV) - rpcrdma_schedule_tasklet(rep); + if (wc->status != IB_WC_SUCCESS) return; - } - switch (wc->opcode) { - case IB_WC_FAST_REG_MR: - frmr = (struct rpcrdma_mw *)(unsigned long)wc->wr_id; + if (wc->opcode == IB_WC_FAST_REG_MR) frmr->r.frmr.state = FRMR_IS_VALID; - break; - case IB_WC_LOCAL_INV: - frmr = (struct rpcrdma_mw *)(unsigned long)wc->wr_id; + else if (wc->opcode == IB_WC_LOCAL_INV) frmr->r.frmr.state = FRMR_IS_INVALID; - break; - case IB_WC_RECV: - rep->rr_len = wc->byte_len; - ib_dma_sync_single_for_cpu( - rdmab_to_ia(rep->rr_buffer)->ri_id->device, - rep->rr_iov.addr, rep->rr_len, DMA_FROM_DEVICE); - /* Keep (only) the most recent credits, after check validity */ - if (rep->rr_len >= 16) { - struct rpcrdma_msg *p = - (struct rpcrdma_msg *) rep->rr_base; - unsigned int credits = ntohl(p->rm_credit); - if (credits == 0) { - dprintk("RPC: %s: server" - " dropped credits to 0!\n", __func__); - /* don't deadlock */ - credits = 1; - } else if (credits > rep->rr_buffer->rb_max_requests) { - dprintk("RPC: %s: server" - " over-crediting: %d (%d)\n", - __func__, credits, - rep->rr_buffer->rb_max_requests); - credits = rep->rr_buffer->rb_max_requests; - } - atomic_set(&rep->rr_buffer->rb_credits, credits); - } - /* fall through */ - case IB_WC_BIND_MW: - rpcrdma_schedule_tasklet(rep); - break; - default: - dprintk("RPC: %s: unexpected WC event %X\n", - __func__, wc->opcode); - break; - } } -static inline int -rpcrdma_cq_poll(struct ib_cq *cq) +static int +rpcrdma_sendcq_poll(struct ib_cq *cq, struct rpcrdma_ep *ep) { - struct ib_wc wc; - int rc; + struct ib_wc *wcs; + int budget, count, rc; - for (;;) { - rc = ib_poll_cq(cq, 1, &wc); - if (rc < 0) { - dprintk("RPC: %s: ib_poll_cq failed %i\n", - __func__, rc); + budget = RPCRDMA_WC_BUDGET / RPCRDMA_POLLSIZE; + do { + wcs = ep->rep_send_wcs; + + rc = ib_poll_cq(cq, RPCRDMA_POLLSIZE, wcs); + if (rc <= 0) return rc; - } - if (rc == 0) - break; - rpcrdma_event_process(&wc); + count = rc; + while (count-- > 0) + rpcrdma_sendcq_process_wc(wcs++); + } while (rc == RPCRDMA_POLLSIZE && --budget); + return 0; +} + +/* + * Handle send, fast_reg_mr, and local_inv completions. + * + * Send events are typically suppressed and thus do not result + * in an upcall. Occasionally one is signaled, however. This + * prevents the provider's completion queue from wrapping and + * losing a completion. + */ +static void +rpcrdma_sendcq_upcall(struct ib_cq *cq, void *cq_context) +{ + struct rpcrdma_ep *ep = (struct rpcrdma_ep *)cq_context; + int rc; + + rc = rpcrdma_sendcq_poll(cq, ep); + if (rc) { + dprintk("RPC: %s: ib_poll_cq failed: %i\n", + __func__, rc); + return; } + rc = ib_req_notify_cq(cq, + IB_CQ_NEXT_COMP | IB_CQ_REPORT_MISSED_EVENTS); + if (rc == 0) + return; + if (rc < 0) { + dprintk("RPC: %s: ib_req_notify_cq failed: %i\n", + __func__, rc); + return; + } + + rpcrdma_sendcq_poll(cq, ep); +} + +static void +rpcrdma_recvcq_process_wc(struct ib_wc *wc) +{ + struct rpcrdma_rep *rep = + (struct rpcrdma_rep *)(unsigned long)wc->wr_id; + + dprintk("RPC: %s: rep %p status %X opcode %X length %u\n", + __func__, rep, wc->status, wc->opcode, wc->byte_len); + + if (wc->status != IB_WC_SUCCESS) { + rep->rr_len = ~0U; + goto out_schedule; + } + if (wc->opcode != IB_WC_RECV) + return; + + rep->rr_len = wc->byte_len; + ib_dma_sync_single_for_cpu(rdmab_to_ia(rep->rr_buffer)->ri_id->device, + rep->rr_iov.addr, rep->rr_len, DMA_FROM_DEVICE); + + if (rep->rr_len >= 16) { + struct rpcrdma_msg *p = (struct rpcrdma_msg *)rep->rr_base; + unsigned int credits = ntohl(p->rm_credit); + + if (credits == 0) + credits = 1; /* don't deadlock */ + else if (credits > rep->rr_buffer->rb_max_requests) + credits = rep->rr_buffer->rb_max_requests; + atomic_set(&rep->rr_buffer->rb_credits, credits); + } + +out_schedule: + rpcrdma_schedule_tasklet(rep); +} + +static int +rpcrdma_recvcq_poll(struct ib_cq *cq, struct rpcrdma_ep *ep) +{ + struct ib_wc *wcs; + int budget, count, rc; + + budget = RPCRDMA_WC_BUDGET / RPCRDMA_POLLSIZE; + do { + wcs = ep->rep_recv_wcs; + + rc = ib_poll_cq(cq, RPCRDMA_POLLSIZE, wcs); + if (rc <= 0) + return rc; + + count = rc; + while (count-- > 0) + rpcrdma_recvcq_process_wc(wcs++); + } while (rc == RPCRDMA_POLLSIZE && --budget); return 0; } /* - * rpcrdma_cq_event_upcall + * Handle receive completions. * - * This upcall handles recv, send, bind and unbind events. * It is reentrant but processes single events in order to maintain * ordering of receives to keep server credits. * @@ -242,26 +283,31 @@ rpcrdma_cq_poll(struct ib_cq *cq) * connection shutdown. That is, the structures required for * the completion of the reply handler must remain intact until * all memory has been reclaimed. - * - * Note that send events are suppressed and do not result in an upcall. */ static void -rpcrdma_cq_event_upcall(struct ib_cq *cq, void *context) +rpcrdma_recvcq_upcall(struct ib_cq *cq, void *cq_context) { + struct rpcrdma_ep *ep = (struct rpcrdma_ep *)cq_context; int rc; - rc = rpcrdma_cq_poll(cq); - if (rc) + rc = rpcrdma_recvcq_poll(cq, ep); + if (rc) { + dprintk("RPC: %s: ib_poll_cq failed: %i\n", + __func__, rc); return; + } - rc = ib_req_notify_cq(cq, IB_CQ_NEXT_COMP); - if (rc) { - dprintk("RPC: %s: ib_req_notify_cq failed %i\n", + rc = ib_req_notify_cq(cq, + IB_CQ_NEXT_COMP | IB_CQ_REPORT_MISSED_EVENTS); + if (rc == 0) + return; + if (rc < 0) { + dprintk("RPC: %s: ib_req_notify_cq failed: %i\n", __func__, rc); return; } - rpcrdma_cq_poll(cq); + rpcrdma_recvcq_poll(cq, ep); } #ifdef RPC_DEBUG @@ -493,54 +539,32 @@ rpcrdma_ia_open(struct rpcrdma_xprt *xprt, struct sockaddr *addr, int memreg) ia->ri_dma_lkey = ia->ri_id->device->local_dma_lkey; } - switch (memreg) { - case RPCRDMA_MEMWINDOWS: - case RPCRDMA_MEMWINDOWS_ASYNC: - if (!(devattr.device_cap_flags & IB_DEVICE_MEM_WINDOW)) { - dprintk("RPC: %s: MEMWINDOWS registration " - "specified but not supported by adapter, " - "using slower RPCRDMA_REGISTER\n", - __func__); - memreg = RPCRDMA_REGISTER; - } - break; - case RPCRDMA_MTHCAFMR: - if (!ia->ri_id->device->alloc_fmr) { -#if RPCRDMA_PERSISTENT_REGISTRATION - dprintk("RPC: %s: MTHCAFMR registration " - "specified but not supported by adapter, " - "using riskier RPCRDMA_ALLPHYSICAL\n", - __func__); - memreg = RPCRDMA_ALLPHYSICAL; -#else - dprintk("RPC: %s: MTHCAFMR registration " - "specified but not supported by adapter, " - "using slower RPCRDMA_REGISTER\n", - __func__); - memreg = RPCRDMA_REGISTER; -#endif - } - break; - case RPCRDMA_FRMR: + if (memreg == RPCRDMA_FRMR) { /* Requires both frmr reg and local dma lkey */ if ((devattr.device_cap_flags & (IB_DEVICE_MEM_MGT_EXTENSIONS|IB_DEVICE_LOCAL_DMA_LKEY)) != (IB_DEVICE_MEM_MGT_EXTENSIONS|IB_DEVICE_LOCAL_DMA_LKEY)) { -#if RPCRDMA_PERSISTENT_REGISTRATION dprintk("RPC: %s: FRMR registration " - "specified but not supported by adapter, " - "using riskier RPCRDMA_ALLPHYSICAL\n", - __func__); + "not supported by HCA\n", __func__); + memreg = RPCRDMA_MTHCAFMR; + } else { + /* Mind the ia limit on FRMR page list depth */ + ia->ri_max_frmr_depth = min_t(unsigned int, + RPCRDMA_MAX_DATA_SEGS, + devattr.max_fast_reg_page_list_len); + } + } + if (memreg == RPCRDMA_MTHCAFMR) { + if (!ia->ri_id->device->alloc_fmr) { + dprintk("RPC: %s: MTHCAFMR registration " + "not supported by HCA\n", __func__); +#if RPCRDMA_PERSISTENT_REGISTRATION memreg = RPCRDMA_ALLPHYSICAL; #else - dprintk("RPC: %s: FRMR registration " - "specified but not supported by adapter, " - "using slower RPCRDMA_REGISTER\n", - __func__); - memreg = RPCRDMA_REGISTER; + rc = -ENOMEM; + goto out2; #endif } - break; } /* @@ -552,8 +576,6 @@ rpcrdma_ia_open(struct rpcrdma_xprt *xprt, struct sockaddr *addr, int memreg) * adapter. */ switch (memreg) { - case RPCRDMA_BOUNCEBUFFERS: - case RPCRDMA_REGISTER: case RPCRDMA_FRMR: break; #if RPCRDMA_PERSISTENT_REGISTRATION @@ -563,30 +585,26 @@ rpcrdma_ia_open(struct rpcrdma_xprt *xprt, struct sockaddr *addr, int memreg) IB_ACCESS_REMOTE_READ; goto register_setup; #endif - case RPCRDMA_MEMWINDOWS_ASYNC: - case RPCRDMA_MEMWINDOWS: - mem_priv = IB_ACCESS_LOCAL_WRITE | - IB_ACCESS_MW_BIND; - goto register_setup; case RPCRDMA_MTHCAFMR: if (ia->ri_have_dma_lkey) break; mem_priv = IB_ACCESS_LOCAL_WRITE; +#if RPCRDMA_PERSISTENT_REGISTRATION register_setup: +#endif ia->ri_bind_mem = ib_get_dma_mr(ia->ri_pd, mem_priv); if (IS_ERR(ia->ri_bind_mem)) { printk(KERN_ALERT "%s: ib_get_dma_mr for " - "phys register failed with %lX\n\t" - "Will continue with degraded performance\n", + "phys register failed with %lX\n", __func__, PTR_ERR(ia->ri_bind_mem)); - memreg = RPCRDMA_REGISTER; - ia->ri_bind_mem = NULL; + rc = -ENOMEM; + goto out2; } break; default: - printk(KERN_ERR "%s: invalid memory registration mode %d\n", - __func__, memreg); - rc = -EINVAL; + printk(KERN_ERR "RPC: Unsupported memory " + "registration mode: %d\n", memreg); + rc = -ENOMEM; goto out2; } dprintk("RPC: %s: memory registration strategy is %d\n", @@ -640,6 +658,7 @@ rpcrdma_ep_create(struct rpcrdma_ep *ep, struct rpcrdma_ia *ia, struct rpcrdma_create_data_internal *cdata) { struct ib_device_attr devattr; + struct ib_cq *sendcq, *recvcq; int rc, err; rc = ib_query_device(ia->ri_id->device, &devattr); @@ -659,32 +678,42 @@ rpcrdma_ep_create(struct rpcrdma_ep *ep, struct rpcrdma_ia *ia, ep->rep_attr.srq = NULL; ep->rep_attr.cap.max_send_wr = cdata->max_requests; switch (ia->ri_memreg_strategy) { - case RPCRDMA_FRMR: + case RPCRDMA_FRMR: { + int depth = 7; + /* Add room for frmr register and invalidate WRs. * 1. FRMR reg WR for head * 2. FRMR invalidate WR for head - * 3. FRMR reg WR for pagelist - * 4. FRMR invalidate WR for pagelist + * 3. N FRMR reg WRs for pagelist + * 4. N FRMR invalidate WRs for pagelist * 5. FRMR reg WR for tail * 6. FRMR invalidate WR for tail * 7. The RDMA_SEND WR */ - ep->rep_attr.cap.max_send_wr *= 7; + + /* Calculate N if the device max FRMR depth is smaller than + * RPCRDMA_MAX_DATA_SEGS. + */ + if (ia->ri_max_frmr_depth < RPCRDMA_MAX_DATA_SEGS) { + int delta = RPCRDMA_MAX_DATA_SEGS - + ia->ri_max_frmr_depth; + + do { + depth += 2; /* FRMR reg + invalidate */ + delta -= ia->ri_max_frmr_depth; + } while (delta > 0); + + } + ep->rep_attr.cap.max_send_wr *= depth; if (ep->rep_attr.cap.max_send_wr > devattr.max_qp_wr) { - cdata->max_requests = devattr.max_qp_wr / 7; + cdata->max_requests = devattr.max_qp_wr / depth; if (!cdata->max_requests) return -EINVAL; - ep->rep_attr.cap.max_send_wr = cdata->max_requests * 7; + ep->rep_attr.cap.max_send_wr = cdata->max_requests * + depth; } break; - case RPCRDMA_MEMWINDOWS_ASYNC: - case RPCRDMA_MEMWINDOWS: - /* Add room for mw_binds+unbinds - overkill! */ - ep->rep_attr.cap.max_send_wr++; - ep->rep_attr.cap.max_send_wr *= (2 * RPCRDMA_MAX_SEGS); - if (ep->rep_attr.cap.max_send_wr > devattr.max_qp_wr) - return -EINVAL; - break; + } default: break; } @@ -705,46 +734,51 @@ rpcrdma_ep_create(struct rpcrdma_ep *ep, struct rpcrdma_ia *ia, ep->rep_attr.cap.max_recv_sge); /* set trigger for requesting send completion */ - ep->rep_cqinit = ep->rep_attr.cap.max_send_wr/2 /* - 1*/; - switch (ia->ri_memreg_strategy) { - case RPCRDMA_MEMWINDOWS_ASYNC: - case RPCRDMA_MEMWINDOWS: - ep->rep_cqinit -= RPCRDMA_MAX_SEGS; - break; - default: - break; - } + ep->rep_cqinit = ep->rep_attr.cap.max_send_wr/2 - 1; if (ep->rep_cqinit <= 2) ep->rep_cqinit = 0; INIT_CQCOUNT(ep); ep->rep_ia = ia; init_waitqueue_head(&ep->rep_connect_wait); + INIT_DELAYED_WORK(&ep->rep_connect_worker, rpcrdma_connect_worker); - /* - * Create a single cq for receive dto and mw_bind (only ever - * care about unbind, really). Send completions are suppressed. - * Use single threaded tasklet upcalls to maintain ordering. - */ - ep->rep_cq = ib_create_cq(ia->ri_id->device, rpcrdma_cq_event_upcall, - rpcrdma_cq_async_error_upcall, NULL, - ep->rep_attr.cap.max_recv_wr + + sendcq = ib_create_cq(ia->ri_id->device, rpcrdma_sendcq_upcall, + rpcrdma_cq_async_error_upcall, ep, ep->rep_attr.cap.max_send_wr + 1, 0); - if (IS_ERR(ep->rep_cq)) { - rc = PTR_ERR(ep->rep_cq); - dprintk("RPC: %s: ib_create_cq failed: %i\n", + if (IS_ERR(sendcq)) { + rc = PTR_ERR(sendcq); + dprintk("RPC: %s: failed to create send CQ: %i\n", __func__, rc); goto out1; } - rc = ib_req_notify_cq(ep->rep_cq, IB_CQ_NEXT_COMP); + rc = ib_req_notify_cq(sendcq, IB_CQ_NEXT_COMP); + if (rc) { + dprintk("RPC: %s: ib_req_notify_cq failed: %i\n", + __func__, rc); + goto out2; + } + + recvcq = ib_create_cq(ia->ri_id->device, rpcrdma_recvcq_upcall, + rpcrdma_cq_async_error_upcall, ep, + ep->rep_attr.cap.max_recv_wr + 1, 0); + if (IS_ERR(recvcq)) { + rc = PTR_ERR(recvcq); + dprintk("RPC: %s: failed to create recv CQ: %i\n", + __func__, rc); + goto out2; + } + + rc = ib_req_notify_cq(recvcq, IB_CQ_NEXT_COMP); if (rc) { dprintk("RPC: %s: ib_req_notify_cq failed: %i\n", __func__, rc); + ib_destroy_cq(recvcq); goto out2; } - ep->rep_attr.send_cq = ep->rep_cq; - ep->rep_attr.recv_cq = ep->rep_cq; + ep->rep_attr.send_cq = sendcq; + ep->rep_attr.recv_cq = recvcq; /* Initialize cma parameters */ @@ -754,9 +788,7 @@ rpcrdma_ep_create(struct rpcrdma_ep *ep, struct rpcrdma_ia *ia, /* Client offers RDMA Read but does not initiate */ ep->rep_remote_cma.initiator_depth = 0; - if (ia->ri_memreg_strategy == RPCRDMA_BOUNCEBUFFERS) - ep->rep_remote_cma.responder_resources = 0; - else if (devattr.max_qp_rd_atom > 32) /* arbitrary but <= 255 */ + if (devattr.max_qp_rd_atom > 32) /* arbitrary but <= 255 */ ep->rep_remote_cma.responder_resources = 32; else ep->rep_remote_cma.responder_resources = devattr.max_qp_rd_atom; @@ -768,7 +800,7 @@ rpcrdma_ep_create(struct rpcrdma_ep *ep, struct rpcrdma_ia *ia, return 0; out2: - err = ib_destroy_cq(ep->rep_cq); + err = ib_destroy_cq(sendcq); if (err) dprintk("RPC: %s: ib_destroy_cq returned %i\n", __func__, err); @@ -782,11 +814,8 @@ out1: * Disconnect and destroy endpoint. After this, the only * valid operations on the ep are to free it (if dynamically * allocated) or re-create it. - * - * The caller's error handling must be sure to not leak the endpoint - * if this function fails. */ -int +void rpcrdma_ep_destroy(struct rpcrdma_ep *ep, struct rpcrdma_ia *ia) { int rc; @@ -794,6 +823,8 @@ rpcrdma_ep_destroy(struct rpcrdma_ep *ep, struct rpcrdma_ia *ia) dprintk("RPC: %s: entering, connected is %d\n", __func__, ep->rep_connected); + cancel_delayed_work_sync(&ep->rep_connect_worker); + if (ia->ri_id->qp) { rc = rpcrdma_ep_disconnect(ep, ia); if (rc) @@ -809,13 +840,17 @@ rpcrdma_ep_destroy(struct rpcrdma_ep *ep, struct rpcrdma_ia *ia) ep->rep_pad_mr = NULL; } - rpcrdma_clean_cq(ep->rep_cq); - rc = ib_destroy_cq(ep->rep_cq); + rpcrdma_clean_cq(ep->rep_attr.recv_cq); + rc = ib_destroy_cq(ep->rep_attr.recv_cq); if (rc) dprintk("RPC: %s: ib_destroy_cq returned %i\n", __func__, rc); - return rc; + rpcrdma_clean_cq(ep->rep_attr.send_cq); + rc = ib_destroy_cq(ep->rep_attr.send_cq); + if (rc) + dprintk("RPC: %s: ib_destroy_cq returned %i\n", + __func__, rc); } /* @@ -831,17 +866,20 @@ rpcrdma_ep_connect(struct rpcrdma_ep *ep, struct rpcrdma_ia *ia) if (ep->rep_connected != 0) { struct rpcrdma_xprt *xprt; retry: + dprintk("RPC: %s: reconnecting...\n", __func__); rc = rpcrdma_ep_disconnect(ep, ia); if (rc && rc != -ENOTCONN) dprintk("RPC: %s: rpcrdma_ep_disconnect" " status %i\n", __func__, rc); - rpcrdma_clean_cq(ep->rep_cq); + + rpcrdma_clean_cq(ep->rep_attr.recv_cq); + rpcrdma_clean_cq(ep->rep_attr.send_cq); xprt = container_of(ia, struct rpcrdma_xprt, rx_ia); id = rpcrdma_create_id(xprt, ia, (struct sockaddr *)&xprt->rx_data.addr); if (IS_ERR(id)) { - rc = PTR_ERR(id); + rc = -EHOSTUNREACH; goto out; } /* TEMP TEMP TEMP - fail if new device: @@ -855,35 +893,32 @@ retry: printk("RPC: %s: can't reconnect on " "different device!\n", __func__); rdma_destroy_id(id); - rc = -ENETDOWN; + rc = -ENETUNREACH; goto out; } /* END TEMP */ + rc = rdma_create_qp(id, ia->ri_pd, &ep->rep_attr); + if (rc) { + dprintk("RPC: %s: rdma_create_qp failed %i\n", + __func__, rc); + rdma_destroy_id(id); + rc = -ENETUNREACH; + goto out; + } rdma_destroy_qp(ia->ri_id); rdma_destroy_id(ia->ri_id); ia->ri_id = id; + } else { + dprintk("RPC: %s: connecting...\n", __func__); + rc = rdma_create_qp(ia->ri_id, ia->ri_pd, &ep->rep_attr); + if (rc) { + dprintk("RPC: %s: rdma_create_qp failed %i\n", + __func__, rc); + /* do not update ep->rep_connected */ + return -ENETUNREACH; + } } - rc = rdma_create_qp(ia->ri_id, ia->ri_pd, &ep->rep_attr); - if (rc) { - dprintk("RPC: %s: rdma_create_qp failed %i\n", - __func__, rc); - goto out; - } - -/* XXX Tavor device performs badly with 2K MTU! */ -if (strnicmp(ia->ri_id->device->dma_device->bus->name, "pci", 3) == 0) { - struct pci_dev *pcid = to_pci_dev(ia->ri_id->device->dma_device); - if (pcid->device == PCI_DEVICE_ID_MELLANOX_TAVOR && - (pcid->vendor == PCI_VENDOR_ID_MELLANOX || - pcid->vendor == PCI_VENDOR_ID_TOPSPIN)) { - struct ib_qp_attr attr = { - .path_mtu = IB_MTU_1024 - }; - rc = ib_modify_qp(ia->ri_id->qp, &attr, IB_QP_PATH_MTU); - } -} - ep->rep_connected = 0; rc = rdma_connect(ia->ri_id, &ep->rep_remote_cma); @@ -944,7 +979,8 @@ rpcrdma_ep_disconnect(struct rpcrdma_ep *ep, struct rpcrdma_ia *ia) { int rc; - rpcrdma_clean_cq(ep->rep_cq); + rpcrdma_clean_cq(ep->rep_attr.recv_cq); + rpcrdma_clean_cq(ep->rep_attr.send_cq); rc = rdma_disconnect(ia->ri_id); if (!rc) { /* returns without wait if not connected */ @@ -967,7 +1003,7 @@ rpcrdma_buffer_create(struct rpcrdma_buffer *buf, struct rpcrdma_ep *ep, struct rpcrdma_ia *ia, struct rpcrdma_create_data_internal *cdata) { char *p; - size_t len; + size_t len, rlen, wlen; int i, rc; struct rpcrdma_mw *r; @@ -997,11 +1033,6 @@ rpcrdma_buffer_create(struct rpcrdma_buffer *buf, struct rpcrdma_ep *ep, len += (buf->rb_max_requests + 1) * RPCRDMA_MAX_SEGS * sizeof(struct rpcrdma_mw); break; - case RPCRDMA_MEMWINDOWS_ASYNC: - case RPCRDMA_MEMWINDOWS: - len += (buf->rb_max_requests + 1) * RPCRDMA_MAX_SEGS * - sizeof(struct rpcrdma_mw); - break; default: break; } @@ -1032,32 +1063,29 @@ rpcrdma_buffer_create(struct rpcrdma_buffer *buf, struct rpcrdma_ep *ep, } p += cdata->padding; - /* - * Allocate the fmr's, or mw's for mw_bind chunk registration. - * We "cycle" the mw's in order to minimize rkey reuse, - * and also reduce unbind-to-bind collision. - */ INIT_LIST_HEAD(&buf->rb_mws); r = (struct rpcrdma_mw *)p; switch (ia->ri_memreg_strategy) { case RPCRDMA_FRMR: for (i = buf->rb_max_requests * RPCRDMA_MAX_SEGS; i; i--) { r->r.frmr.fr_mr = ib_alloc_fast_reg_mr(ia->ri_pd, - RPCRDMA_MAX_SEGS); + ia->ri_max_frmr_depth); if (IS_ERR(r->r.frmr.fr_mr)) { rc = PTR_ERR(r->r.frmr.fr_mr); dprintk("RPC: %s: ib_alloc_fast_reg_mr" " failed %i\n", __func__, rc); goto out; } - r->r.frmr.fr_pgl = - ib_alloc_fast_reg_page_list(ia->ri_id->device, - RPCRDMA_MAX_SEGS); + r->r.frmr.fr_pgl = ib_alloc_fast_reg_page_list( + ia->ri_id->device, + ia->ri_max_frmr_depth); if (IS_ERR(r->r.frmr.fr_pgl)) { rc = PTR_ERR(r->r.frmr.fr_pgl); dprintk("RPC: %s: " "ib_alloc_fast_reg_page_list " "failed %i\n", __func__, rc); + + ib_dereg_mr(r->r.frmr.fr_mr); goto out; } list_add(&r->mw_list, &buf->rb_mws); @@ -1082,21 +1110,6 @@ rpcrdma_buffer_create(struct rpcrdma_buffer *buf, struct rpcrdma_ep *ep, ++r; } break; - case RPCRDMA_MEMWINDOWS_ASYNC: - case RPCRDMA_MEMWINDOWS: - /* Allocate one extra request's worth, for full cycling */ - for (i = (buf->rb_max_requests+1) * RPCRDMA_MAX_SEGS; i; i--) { - r->r.mw = ib_alloc_mw(ia->ri_pd, IB_MW_TYPE_1); - if (IS_ERR(r->r.mw)) { - rc = PTR_ERR(r->r.mw); - dprintk("RPC: %s: ib_alloc_mw" - " failed %i\n", __func__, rc); - goto out; - } - list_add(&r->mw_list, &buf->rb_mws); - ++r; - } - break; default: break; } @@ -1105,16 +1118,16 @@ rpcrdma_buffer_create(struct rpcrdma_buffer *buf, struct rpcrdma_ep *ep, * Allocate/init the request/reply buffers. Doing this * using kmalloc for now -- one for each buf. */ + wlen = 1 << fls(cdata->inline_wsize + sizeof(struct rpcrdma_req)); + rlen = 1 << fls(cdata->inline_rsize + sizeof(struct rpcrdma_rep)); + dprintk("RPC: %s: wlen = %zu, rlen = %zu\n", + __func__, wlen, rlen); + for (i = 0; i < buf->rb_max_requests; i++) { struct rpcrdma_req *req; struct rpcrdma_rep *rep; - len = cdata->inline_wsize + sizeof(struct rpcrdma_req); - /* RPC layer requests *double* size + 1K RPC_SLACK_SPACE! */ - /* Typical ~2400b, so rounding up saves work later */ - if (len < 4096) - len = 4096; - req = kmalloc(len, GFP_KERNEL); + req = kmalloc(wlen, GFP_KERNEL); if (req == NULL) { dprintk("RPC: %s: request buffer %d alloc" " failed\n", __func__, i); @@ -1126,16 +1139,16 @@ rpcrdma_buffer_create(struct rpcrdma_buffer *buf, struct rpcrdma_ep *ep, buf->rb_send_bufs[i]->rl_buffer = buf; rc = rpcrdma_register_internal(ia, req->rl_base, - len - offsetof(struct rpcrdma_req, rl_base), + wlen - offsetof(struct rpcrdma_req, rl_base), &buf->rb_send_bufs[i]->rl_handle, &buf->rb_send_bufs[i]->rl_iov); if (rc) goto out; - buf->rb_send_bufs[i]->rl_size = len-sizeof(struct rpcrdma_req); + buf->rb_send_bufs[i]->rl_size = wlen - + sizeof(struct rpcrdma_req); - len = cdata->inline_rsize + sizeof(struct rpcrdma_rep); - rep = kmalloc(len, GFP_KERNEL); + rep = kmalloc(rlen, GFP_KERNEL); if (rep == NULL) { dprintk("RPC: %s: reply buffer %d alloc failed\n", __func__, i); @@ -1145,10 +1158,9 @@ rpcrdma_buffer_create(struct rpcrdma_buffer *buf, struct rpcrdma_ep *ep, memset(rep, 0, sizeof(struct rpcrdma_rep)); buf->rb_recv_bufs[i] = rep; buf->rb_recv_bufs[i]->rr_buffer = buf; - init_waitqueue_head(&rep->rr_unbind); rc = rpcrdma_register_internal(ia, rep->rr_base, - len - offsetof(struct rpcrdma_rep, rr_base), + rlen - offsetof(struct rpcrdma_rep, rr_base), &buf->rb_recv_bufs[i]->rr_handle, &buf->rb_recv_bufs[i]->rr_iov); if (rc) @@ -1179,7 +1191,6 @@ rpcrdma_buffer_destroy(struct rpcrdma_buffer *buf) /* clean up in reverse order from create * 1. recv mr memory (mr free, then kfree) - * 1a. bind mw memory * 2. send mr memory (mr free, then kfree) * 3. padding (if any) [moved to rpcrdma_ep_destroy] * 4. arrays @@ -1194,41 +1205,6 @@ rpcrdma_buffer_destroy(struct rpcrdma_buffer *buf) kfree(buf->rb_recv_bufs[i]); } if (buf->rb_send_bufs && buf->rb_send_bufs[i]) { - while (!list_empty(&buf->rb_mws)) { - r = list_entry(buf->rb_mws.next, - struct rpcrdma_mw, mw_list); - list_del(&r->mw_list); - switch (ia->ri_memreg_strategy) { - case RPCRDMA_FRMR: - rc = ib_dereg_mr(r->r.frmr.fr_mr); - if (rc) - dprintk("RPC: %s:" - " ib_dereg_mr" - " failed %i\n", - __func__, rc); - ib_free_fast_reg_page_list(r->r.frmr.fr_pgl); - break; - case RPCRDMA_MTHCAFMR: - rc = ib_dealloc_fmr(r->r.fmr); - if (rc) - dprintk("RPC: %s:" - " ib_dealloc_fmr" - " failed %i\n", - __func__, rc); - break; - case RPCRDMA_MEMWINDOWS_ASYNC: - case RPCRDMA_MEMWINDOWS: - rc = ib_dealloc_mw(r->r.mw); - if (rc) - dprintk("RPC: %s:" - " ib_dealloc_mw" - " failed %i\n", - __func__, rc); - break; - default: - break; - } - } rpcrdma_deregister_internal(ia, buf->rb_send_bufs[i]->rl_handle, &buf->rb_send_bufs[i]->rl_iov); @@ -1236,6 +1212,33 @@ rpcrdma_buffer_destroy(struct rpcrdma_buffer *buf) } } + while (!list_empty(&buf->rb_mws)) { + r = list_entry(buf->rb_mws.next, + struct rpcrdma_mw, mw_list); + list_del(&r->mw_list); + switch (ia->ri_memreg_strategy) { + case RPCRDMA_FRMR: + rc = ib_dereg_mr(r->r.frmr.fr_mr); + if (rc) + dprintk("RPC: %s:" + " ib_dereg_mr" + " failed %i\n", + __func__, rc); + ib_free_fast_reg_page_list(r->r.frmr.fr_pgl); + break; + case RPCRDMA_MTHCAFMR: + rc = ib_dealloc_fmr(r->r.fmr); + if (rc) + dprintk("RPC: %s:" + " ib_dealloc_fmr" + " failed %i\n", + __func__, rc); + break; + default: + break; + } + } + kfree(buf->rb_pool); } @@ -1299,21 +1302,17 @@ rpcrdma_buffer_put(struct rpcrdma_req *req) int i; unsigned long flags; - BUG_ON(req->rl_nchunks != 0); spin_lock_irqsave(&buffers->rb_lock, flags); buffers->rb_send_bufs[--buffers->rb_send_index] = req; req->rl_niovs = 0; if (req->rl_reply) { buffers->rb_recv_bufs[--buffers->rb_recv_index] = req->rl_reply; - init_waitqueue_head(&req->rl_reply->rr_unbind); req->rl_reply->rr_func = NULL; req->rl_reply = NULL; } switch (ia->ri_memreg_strategy) { case RPCRDMA_FRMR: case RPCRDMA_MTHCAFMR: - case RPCRDMA_MEMWINDOWS_ASYNC: - case RPCRDMA_MEMWINDOWS: /* * Cycle mw's back in reverse order, and "spin" them. * This delays and scrambles reuse as much as possible. @@ -1358,8 +1357,7 @@ rpcrdma_recv_buffer_get(struct rpcrdma_req *req) /* * Put reply buffers back into pool when not attached to - * request. This happens in error conditions, and when - * aborting unbinds. Pre-decrement counter/array index. + * request. This happens in error conditions. */ void rpcrdma_recv_buffer_put(struct rpcrdma_rep *rep) @@ -1498,8 +1496,8 @@ rpcrdma_register_frmr_external(struct rpcrdma_mr_seg *seg, seg1->mr_offset -= pageoff; /* start of page */ seg1->mr_len += pageoff; len = -pageoff; - if (*nsegs > RPCRDMA_MAX_DATA_SEGS) - *nsegs = RPCRDMA_MAX_DATA_SEGS; + if (*nsegs > ia->ri_max_frmr_depth) + *nsegs = ia->ri_max_frmr_depth; for (page_no = i = 0; i < *nsegs;) { rpcrdma_map_one(ia, seg, writing); pa = seg->mr_dma; @@ -1536,10 +1534,6 @@ rpcrdma_register_frmr_external(struct rpcrdma_mr_seg *seg, } else post_wr = &frmr_wr; - /* Bump the key */ - key = (u8)(seg1->mr_chunk.rl_mw->r.frmr.fr_mr->rkey & 0x000000FF); - ib_update_fast_reg_key(seg1->mr_chunk.rl_mw->r.frmr.fr_mr, ++key); - /* Prepare FRMR WR */ memset(&frmr_wr, 0, sizeof frmr_wr); frmr_wr.wr_id = (unsigned long)(void *)seg1->mr_chunk.rl_mw; @@ -1550,7 +1544,16 @@ rpcrdma_register_frmr_external(struct rpcrdma_mr_seg *seg, frmr_wr.wr.fast_reg.page_list_len = page_no; frmr_wr.wr.fast_reg.page_shift = PAGE_SHIFT; frmr_wr.wr.fast_reg.length = page_no << PAGE_SHIFT; - BUG_ON(frmr_wr.wr.fast_reg.length < len); + if (frmr_wr.wr.fast_reg.length < len) { + while (seg1->mr_nsegs--) + rpcrdma_unmap_one(ia, seg++); + return -EIO; + } + + /* Bump the key */ + key = (u8)(seg1->mr_chunk.rl_mw->r.frmr.fr_mr->rkey & 0x000000FF); + ib_update_fast_reg_key(seg1->mr_chunk.rl_mw->r.frmr.fr_mr, ++key); + frmr_wr.wr.fast_reg.access_flags = (writing ? IB_ACCESS_REMOTE_WRITE | IB_ACCESS_LOCAL_WRITE : IB_ACCESS_REMOTE_READ); @@ -1661,135 +1664,6 @@ rpcrdma_deregister_fmr_external(struct rpcrdma_mr_seg *seg, return rc; } -static int -rpcrdma_register_memwin_external(struct rpcrdma_mr_seg *seg, - int *nsegs, int writing, struct rpcrdma_ia *ia, - struct rpcrdma_xprt *r_xprt) -{ - int mem_priv = (writing ? IB_ACCESS_REMOTE_WRITE : - IB_ACCESS_REMOTE_READ); - struct ib_mw_bind param; - int rc; - - *nsegs = 1; - rpcrdma_map_one(ia, seg, writing); - param.bind_info.mr = ia->ri_bind_mem; - param.wr_id = 0ULL; /* no send cookie */ - param.bind_info.addr = seg->mr_dma; - param.bind_info.length = seg->mr_len; - param.send_flags = 0; - param.bind_info.mw_access_flags = mem_priv; - - DECR_CQCOUNT(&r_xprt->rx_ep); - rc = ib_bind_mw(ia->ri_id->qp, seg->mr_chunk.rl_mw->r.mw, ¶m); - if (rc) { - dprintk("RPC: %s: failed ib_bind_mw " - "%u@0x%llx status %i\n", - __func__, seg->mr_len, - (unsigned long long)seg->mr_dma, rc); - rpcrdma_unmap_one(ia, seg); - } else { - seg->mr_rkey = seg->mr_chunk.rl_mw->r.mw->rkey; - seg->mr_base = param.bind_info.addr; - seg->mr_nsegs = 1; - } - return rc; -} - -static int -rpcrdma_deregister_memwin_external(struct rpcrdma_mr_seg *seg, - struct rpcrdma_ia *ia, - struct rpcrdma_xprt *r_xprt, void **r) -{ - struct ib_mw_bind param; - LIST_HEAD(l); - int rc; - - BUG_ON(seg->mr_nsegs != 1); - param.bind_info.mr = ia->ri_bind_mem; - param.bind_info.addr = 0ULL; /* unbind */ - param.bind_info.length = 0; - param.bind_info.mw_access_flags = 0; - if (*r) { - param.wr_id = (u64) (unsigned long) *r; - param.send_flags = IB_SEND_SIGNALED; - INIT_CQCOUNT(&r_xprt->rx_ep); - } else { - param.wr_id = 0ULL; - param.send_flags = 0; - DECR_CQCOUNT(&r_xprt->rx_ep); - } - rc = ib_bind_mw(ia->ri_id->qp, seg->mr_chunk.rl_mw->r.mw, ¶m); - rpcrdma_unmap_one(ia, seg); - if (rc) - dprintk("RPC: %s: failed ib_(un)bind_mw," - " status %i\n", __func__, rc); - else - *r = NULL; /* will upcall on completion */ - return rc; -} - -static int -rpcrdma_register_default_external(struct rpcrdma_mr_seg *seg, - int *nsegs, int writing, struct rpcrdma_ia *ia) -{ - int mem_priv = (writing ? IB_ACCESS_REMOTE_WRITE : - IB_ACCESS_REMOTE_READ); - struct rpcrdma_mr_seg *seg1 = seg; - struct ib_phys_buf ipb[RPCRDMA_MAX_DATA_SEGS]; - int len, i, rc = 0; - - if (*nsegs > RPCRDMA_MAX_DATA_SEGS) - *nsegs = RPCRDMA_MAX_DATA_SEGS; - for (len = 0, i = 0; i < *nsegs;) { - rpcrdma_map_one(ia, seg, writing); - ipb[i].addr = seg->mr_dma; - ipb[i].size = seg->mr_len; - len += seg->mr_len; - ++seg; - ++i; - /* Check for holes */ - if ((i < *nsegs && offset_in_page(seg->mr_offset)) || - offset_in_page((seg-1)->mr_offset+(seg-1)->mr_len)) - break; - } - seg1->mr_base = seg1->mr_dma; - seg1->mr_chunk.rl_mr = ib_reg_phys_mr(ia->ri_pd, - ipb, i, mem_priv, &seg1->mr_base); - if (IS_ERR(seg1->mr_chunk.rl_mr)) { - rc = PTR_ERR(seg1->mr_chunk.rl_mr); - dprintk("RPC: %s: failed ib_reg_phys_mr " - "%u@0x%llx (%d)... status %i\n", - __func__, len, - (unsigned long long)seg1->mr_dma, i, rc); - while (i--) - rpcrdma_unmap_one(ia, --seg); - } else { - seg1->mr_rkey = seg1->mr_chunk.rl_mr->rkey; - seg1->mr_nsegs = i; - seg1->mr_len = len; - } - *nsegs = i; - return rc; -} - -static int -rpcrdma_deregister_default_external(struct rpcrdma_mr_seg *seg, - struct rpcrdma_ia *ia) -{ - struct rpcrdma_mr_seg *seg1 = seg; - int rc; - - rc = ib_dereg_mr(seg1->mr_chunk.rl_mr); - seg1->mr_chunk.rl_mr = NULL; - while (seg1->mr_nsegs--) - rpcrdma_unmap_one(ia, seg++); - if (rc) - dprintk("RPC: %s: failed ib_dereg_mr," - " status %i\n", __func__, rc); - return rc; -} - int rpcrdma_register_external(struct rpcrdma_mr_seg *seg, int nsegs, int writing, struct rpcrdma_xprt *r_xprt) @@ -1819,16 +1693,8 @@ rpcrdma_register_external(struct rpcrdma_mr_seg *seg, rc = rpcrdma_register_fmr_external(seg, &nsegs, writing, ia); break; - /* Registration using memory windows */ - case RPCRDMA_MEMWINDOWS_ASYNC: - case RPCRDMA_MEMWINDOWS: - rc = rpcrdma_register_memwin_external(seg, &nsegs, writing, ia, r_xprt); - break; - - /* Default registration each time */ default: - rc = rpcrdma_register_default_external(seg, &nsegs, writing, ia); - break; + return -1; } if (rc) return -1; @@ -1838,7 +1704,7 @@ rpcrdma_register_external(struct rpcrdma_mr_seg *seg, int rpcrdma_deregister_external(struct rpcrdma_mr_seg *seg, - struct rpcrdma_xprt *r_xprt, void *r) + struct rpcrdma_xprt *r_xprt) { struct rpcrdma_ia *ia = &r_xprt->rx_ia; int nsegs = seg->mr_nsegs, rc; @@ -1847,9 +1713,7 @@ rpcrdma_deregister_external(struct rpcrdma_mr_seg *seg, #if RPCRDMA_PERSISTENT_REGISTRATION case RPCRDMA_ALLPHYSICAL: - BUG_ON(nsegs != 1); rpcrdma_unmap_one(ia, seg); - rc = 0; break; #endif @@ -1861,21 +1725,9 @@ rpcrdma_deregister_external(struct rpcrdma_mr_seg *seg, rc = rpcrdma_deregister_fmr_external(seg, ia); break; - case RPCRDMA_MEMWINDOWS_ASYNC: - case RPCRDMA_MEMWINDOWS: - rc = rpcrdma_deregister_memwin_external(seg, ia, r_xprt, &r); - break; - default: - rc = rpcrdma_deregister_default_external(seg, ia); break; } - if (r) { - struct rpcrdma_rep *rep = r; - void (*func)(struct rpcrdma_rep *) = rep->rr_func; - rep->rr_func = NULL; - func(rep); /* dereg done, callback now */ - } return nsegs; } @@ -1950,7 +1802,6 @@ rpcrdma_ep_post_recv(struct rpcrdma_ia *ia, ib_dma_sync_single_for_cpu(ia->ri_id->device, rep->rr_iov.addr, rep->rr_iov.length, DMA_BIDIRECTIONAL); - DECR_CQCOUNT(ep); rc = ib_post_recv(ia->ri_id->qp, &recv_wr, &recv_wr_fail); if (rc) diff --git a/net/sunrpc/xprtrdma/xprt_rdma.h b/net/sunrpc/xprtrdma/xprt_rdma.h index cc1445dc1d1a..89e7cd479705 100644 --- a/net/sunrpc/xprtrdma/xprt_rdma.h +++ b/net/sunrpc/xprtrdma/xprt_rdma.h @@ -43,6 +43,7 @@ #include <linux/wait.h> /* wait_queue_head_t, etc */ #include <linux/spinlock.h> /* spinlock_t, etc */ #include <linux/atomic.h> /* atomic_t, etc */ +#include <linux/workqueue.h> /* struct work_struct */ #include <rdma/rdma_cm.h> /* RDMA connection api */ #include <rdma/ib_verbs.h> /* RDMA verbs api */ @@ -66,18 +67,21 @@ struct rpcrdma_ia { struct completion ri_done; int ri_async_rc; enum rpcrdma_memreg ri_memreg_strategy; + unsigned int ri_max_frmr_depth; }; /* * RDMA Endpoint -- one per transport instance */ +#define RPCRDMA_WC_BUDGET (128) +#define RPCRDMA_POLLSIZE (16) + struct rpcrdma_ep { atomic_t rep_cqcount; int rep_cqinit; int rep_connected; struct rpcrdma_ia *rep_ia; - struct ib_cq *rep_cq; struct ib_qp_init_attr rep_attr; wait_queue_head_t rep_connect_wait; struct ib_sge rep_pad; /* holds zeroed pad */ @@ -86,6 +90,9 @@ struct rpcrdma_ep { struct rpc_xprt *rep_xprt; /* for rep_func */ struct rdma_conn_param rep_remote_cma; struct sockaddr_storage rep_remote_addr; + struct delayed_work rep_connect_worker; + struct ib_wc rep_send_wcs[RPCRDMA_POLLSIZE]; + struct ib_wc rep_recv_wcs[RPCRDMA_POLLSIZE]; }; #define INIT_CQCOUNT(ep) atomic_set(&(ep)->rep_cqcount, (ep)->rep_cqinit) @@ -124,7 +131,6 @@ struct rpcrdma_rep { struct rpc_xprt *rr_xprt; /* needed for request/reply matching */ void (*rr_func)(struct rpcrdma_rep *);/* called by tasklet in softint */ struct list_head rr_list; /* tasklet list */ - wait_queue_head_t rr_unbind; /* optional unbind wait */ struct ib_sge rr_iov; /* for posting */ struct ib_mr *rr_handle; /* handle for mem in rr_iov */ char rr_base[MAX_RPCRDMAHDR]; /* minimal inline receive buffer */ @@ -159,7 +165,6 @@ struct rpcrdma_mr_seg { /* chunk descriptors */ struct ib_mr *rl_mr; /* if registered directly */ struct rpcrdma_mw { /* if registered from region */ union { - struct ib_mw *mw; struct ib_fmr *fmr; struct { struct ib_fast_reg_page_list *fr_pgl; @@ -207,7 +212,6 @@ struct rpcrdma_req { struct rpcrdma_buffer { spinlock_t rb_lock; /* protects indexes */ atomic_t rb_credits; /* most recent server credits */ - unsigned long rb_cwndscale; /* cached framework rpc_cwndscale */ int rb_max_requests;/* client max requests */ struct list_head rb_mws; /* optional memory windows/fmrs/frmrs */ int rb_send_index; @@ -300,7 +304,7 @@ void rpcrdma_ia_close(struct rpcrdma_ia *); */ int rpcrdma_ep_create(struct rpcrdma_ep *, struct rpcrdma_ia *, struct rpcrdma_create_data_internal *); -int rpcrdma_ep_destroy(struct rpcrdma_ep *, struct rpcrdma_ia *); +void rpcrdma_ep_destroy(struct rpcrdma_ep *, struct rpcrdma_ia *); int rpcrdma_ep_connect(struct rpcrdma_ep *, struct rpcrdma_ia *); int rpcrdma_ep_disconnect(struct rpcrdma_ep *, struct rpcrdma_ia *); @@ -330,11 +334,12 @@ int rpcrdma_deregister_internal(struct rpcrdma_ia *, int rpcrdma_register_external(struct rpcrdma_mr_seg *, int, int, struct rpcrdma_xprt *); int rpcrdma_deregister_external(struct rpcrdma_mr_seg *, - struct rpcrdma_xprt *, void *); + struct rpcrdma_xprt *); /* * RPC/RDMA connection management calls - xprtrdma/rpc_rdma.c */ +void rpcrdma_connect_worker(struct work_struct *); void rpcrdma_conn_func(struct rpcrdma_ep *); void rpcrdma_reply_handler(struct rpcrdma_rep *); diff --git a/scripts/decode_stacktrace.sh b/scripts/decode_stacktrace.sh new file mode 100755 index 000000000000..515c4c00e957 --- /dev/null +++ b/scripts/decode_stacktrace.sh @@ -0,0 +1,126 @@ +#!/bin/bash +# (c) 2014, Sasha Levin <sasha.levin@oracle.com> +#set -x + +if [[ $# != 2 ]]; then + echo "Usage:" + echo " $0 [vmlinux] [base path]" + exit 1 +fi + +vmlinux=$1 +basepath=$2 +declare -A cache + +parse_symbol() { + # The structure of symbol at this point is: + # [name]+[offset]/[total length] + # + # For example: + # do_basic_setup+0x9c/0xbf + + + # Strip the symbol name so that we could look it up + local name=${symbol%+*} + + # Use 'nm vmlinux' to figure out the base address of said symbol. + # It's actually faster to call it every time than to load it + # all into bash. + if [[ "${cache[$name]+isset}" == "isset" ]]; then + local base_addr=${cache[$name]} + else + local base_addr=$(nm "$vmlinux" | grep -i ' t ' | awk "/ $name\$/ {print \$1}" | head -n1) + cache["$name"]="$base_addr" + fi + # Let's start doing the math to get the exact address into the + # symbol. First, strip out the symbol total length. + local expr=${symbol%/*} + + # Now, replace the symbol name with the base address we found + # before. + expr=${expr/$name/0x$base_addr} + + # Evaluate it to find the actual address + expr=$((expr)) + local address=$(printf "%x\n" "$expr") + + # Pass it to addr2line to get filename and line number + # Could get more than one result + if [[ "${cache[$address]+isset}" == "isset" ]]; then + local code=${cache[$address]} + else + local code=$(addr2line -i -e "$vmlinux" "$address") + cache[$address]=$code + fi + + # addr2line doesn't return a proper error code if it fails, so + # we detect it using the value it prints so that we could preserve + # the offset/size into the function and bail out + if [[ $code == "??:0" ]]; then + return + fi + + # Strip out the base of the path + code=${code//$basepath/""} + + # In the case of inlines, move everything to same line + code=${code//$'\n'/' '} + + # Replace old address with pretty line numbers + symbol="$name ($code)" +} + +decode_code() { + local scripts=`dirname "${BASH_SOURCE[0]}"` + + echo "$1" | $scripts/decodecode +} + +handle_line() { + local words + + # Tokenize + read -a words <<<"$1" + + # Remove hex numbers. Do it ourselves until it happens in the + # kernel + + # We need to know the index of the last element before we + # remove elements because arrays are sparse + local last=$(( ${#words[@]} - 1 )) + + for i in "${!words[@]}"; do + # Remove the address + if [[ ${words[$i]} =~ \[\<([^]]+)\>\] ]]; then + unset words[$i] + fi + + # Format timestamps with tabs + if [[ ${words[$i]} == \[ && ${words[$i+1]} == *\] ]]; then + unset words[$i] + words[$i+1]=$(printf "[%13s\n" "${words[$i+1]}") + fi + done + + # The symbol is the last element, process it + symbol=${words[$last]} + unset words[$last] + parse_symbol # modifies $symbol + + # Add up the line number to the symbol + echo "${words[@]}" "$symbol" +} + +while read line; do + # Let's see if we have an address in the line + if [[ $line =~ \[\<([^]]+)\>\] ]]; then + # Translate address to line numbers + handle_line "$line" + # Is it a code line? + elif [[ $line == *Code:* ]]; then + decode_code "$line" + else + # Nothing special in this line, show it as is + echo "$line" + fi +done diff --git a/security/capability.c b/security/capability.c index ad0d4de69944..e76373de3129 100644 --- a/security/capability.c +++ b/security/capability.c @@ -879,7 +879,7 @@ static void cap_key_free(struct key *key) } static int cap_key_permission(key_ref_t key_ref, const struct cred *cred, - key_perm_t perm) + unsigned perm) { return 0; } diff --git a/security/device_cgroup.c b/security/device_cgroup.c index 9134dbf70d3e..d9d69e6930ed 100644 --- a/security/device_cgroup.c +++ b/security/device_cgroup.c @@ -182,7 +182,7 @@ static inline bool is_devcg_online(const struct dev_cgroup *devcg) static int devcgroup_online(struct cgroup_subsys_state *css) { struct dev_cgroup *dev_cgroup = css_to_devcgroup(css); - struct dev_cgroup *parent_dev_cgroup = css_to_devcgroup(css_parent(css)); + struct dev_cgroup *parent_dev_cgroup = css_to_devcgroup(css->parent); int ret = 0; mutex_lock(&devcgroup_mutex); @@ -455,7 +455,7 @@ static bool verify_new_ex(struct dev_cgroup *dev_cgroup, static int parent_has_perm(struct dev_cgroup *childcg, struct dev_exception_item *ex) { - struct dev_cgroup *parent = css_to_devcgroup(css_parent(&childcg->css)); + struct dev_cgroup *parent = css_to_devcgroup(childcg->css.parent); if (!parent) return 1; @@ -476,7 +476,7 @@ static int parent_has_perm(struct dev_cgroup *childcg, static bool parent_allows_removal(struct dev_cgroup *childcg, struct dev_exception_item *ex) { - struct dev_cgroup *parent = css_to_devcgroup(css_parent(&childcg->css)); + struct dev_cgroup *parent = css_to_devcgroup(childcg->css.parent); if (!parent) return true; @@ -587,13 +587,6 @@ static int propagate_exception(struct dev_cgroup *devcg_root, return rc; } -static inline bool has_children(struct dev_cgroup *devcgroup) -{ - struct cgroup *cgrp = devcgroup->css.cgroup; - - return !list_empty(&cgrp->children); -} - /* * Modify the exception list using allow/deny rules. * CAP_SYS_ADMIN is needed for this. It's at least separate from CAP_MKNOD @@ -614,7 +607,7 @@ static int devcgroup_update_access(struct dev_cgroup *devcgroup, char temp[12]; /* 11 + 1 characters needed for a u32 */ int count, rc = 0; struct dev_exception_item ex; - struct dev_cgroup *parent = css_to_devcgroup(css_parent(&devcgroup->css)); + struct dev_cgroup *parent = css_to_devcgroup(devcgroup->css.parent); if (!capable(CAP_SYS_ADMIN)) return -EPERM; @@ -626,7 +619,7 @@ static int devcgroup_update_access(struct dev_cgroup *devcgroup, case 'a': switch (filetype) { case DEVCG_ALLOW: - if (has_children(devcgroup)) + if (css_has_online_children(&devcgroup->css)) return -EINVAL; if (!may_allow_all(parent)) @@ -642,7 +635,7 @@ static int devcgroup_update_access(struct dev_cgroup *devcgroup, return rc; break; case DEVCG_DENY: - if (has_children(devcgroup)) + if (css_has_online_children(&devcgroup->css)) return -EINVAL; dev_exception_clean(devcgroup); @@ -767,27 +760,27 @@ static int devcgroup_update_access(struct dev_cgroup *devcgroup, return rc; } -static int devcgroup_access_write(struct cgroup_subsys_state *css, - struct cftype *cft, char *buffer) +static ssize_t devcgroup_access_write(struct kernfs_open_file *of, + char *buf, size_t nbytes, loff_t off) { int retval; mutex_lock(&devcgroup_mutex); - retval = devcgroup_update_access(css_to_devcgroup(css), - cft->private, buffer); + retval = devcgroup_update_access(css_to_devcgroup(of_css(of)), + of_cft(of)->private, strstrip(buf)); mutex_unlock(&devcgroup_mutex); - return retval; + return retval ?: nbytes; } static struct cftype dev_cgroup_files[] = { { .name = "allow", - .write_string = devcgroup_access_write, + .write = devcgroup_access_write, .private = DEVCG_ALLOW, }, { .name = "deny", - .write_string = devcgroup_access_write, + .write = devcgroup_access_write, .private = DEVCG_DENY, }, { diff --git a/security/integrity/ima/ima_api.c b/security/integrity/ima/ima_api.c index ba9e4d792dd5..d9cd5ce14d2b 100644 --- a/security/integrity/ima/ima_api.c +++ b/security/integrity/ima/ima_api.c @@ -199,6 +199,7 @@ int ima_collect_measurement(struct integrity_iint_cache *iint, struct evm_ima_xattr_data **xattr_value, int *xattr_len) { + const char *audit_cause = "failed"; struct inode *inode = file_inode(file); const char *filename = file->f_dentry->d_name.name; int result = 0; @@ -213,6 +214,12 @@ int ima_collect_measurement(struct integrity_iint_cache *iint, if (!(iint->flags & IMA_COLLECTED)) { u64 i_version = file_inode(file)->i_version; + if (file->f_flags & O_DIRECT) { + audit_cause = "failed(directio)"; + result = -EACCES; + goto out; + } + /* use default hash algorithm */ hash.hdr.algo = ima_hash_algo; @@ -233,9 +240,10 @@ int ima_collect_measurement(struct integrity_iint_cache *iint, result = -ENOMEM; } } +out: if (result) integrity_audit_msg(AUDIT_INTEGRITY_DATA, inode, - filename, "collect_data", "failed", + filename, "collect_data", audit_cause, result, 0); return result; } diff --git a/security/integrity/ima/ima_main.c b/security/integrity/ima/ima_main.c index 52ac6cf41f88..dcc98cf542d8 100644 --- a/security/integrity/ima/ima_main.c +++ b/security/integrity/ima/ima_main.c @@ -214,8 +214,11 @@ static int process_measurement(struct file *file, const char *filename, xattr_ptr = &xattr_value; rc = ima_collect_measurement(iint, file, xattr_ptr, &xattr_len); - if (rc != 0) + if (rc != 0) { + if (file->f_flags & O_DIRECT) + rc = (iint->flags & IMA_PERMIT_DIRECTIO) ? 0 : -EACCES; goto out_digsig; + } pathname = filename ?: ima_d_path(&file->f_path, &pathbuf); diff --git a/security/integrity/ima/ima_policy.c b/security/integrity/ima/ima_policy.c index 93873a450ff7..40a7488f6721 100644 --- a/security/integrity/ima/ima_policy.c +++ b/security/integrity/ima/ima_policy.c @@ -353,7 +353,7 @@ enum { Opt_obj_user, Opt_obj_role, Opt_obj_type, Opt_subj_user, Opt_subj_role, Opt_subj_type, Opt_func, Opt_mask, Opt_fsmagic, Opt_uid, Opt_fowner, - Opt_appraise_type, Opt_fsuuid + Opt_appraise_type, Opt_fsuuid, Opt_permit_directio }; static match_table_t policy_tokens = { @@ -375,6 +375,7 @@ static match_table_t policy_tokens = { {Opt_uid, "uid=%s"}, {Opt_fowner, "fowner=%s"}, {Opt_appraise_type, "appraise_type=%s"}, + {Opt_permit_directio, "permit_directio"}, {Opt_err, NULL} }; @@ -622,6 +623,9 @@ static int ima_parse_rule(char *rule, struct ima_rule_entry *entry) else result = -EINVAL; break; + case Opt_permit_directio: + entry->flags |= IMA_PERMIT_DIRECTIO; + break; case Opt_err: ima_log_string(ab, "UNKNOWN", p); result = -EINVAL; diff --git a/security/integrity/integrity.h b/security/integrity/integrity.h index 2fb5e53e927f..33c0a70f6b15 100644 --- a/security/integrity/integrity.h +++ b/security/integrity/integrity.h @@ -30,6 +30,7 @@ #define IMA_ACTION_FLAGS 0xff000000 #define IMA_DIGSIG 0x01000000 #define IMA_DIGSIG_REQUIRED 0x02000000 +#define IMA_PERMIT_DIRECTIO 0x04000000 #define IMA_DO_MASK (IMA_MEASURE | IMA_APPRAISE | IMA_AUDIT | \ IMA_APPRAISE_SUBMASK) diff --git a/security/keys/internal.h b/security/keys/internal.h index 80b2aac4f50c..5f20da01fd8d 100644 --- a/security/keys/internal.h +++ b/security/keys/internal.h @@ -176,20 +176,11 @@ extern int key_task_permission(const key_ref_t key_ref, /* * Check to see whether permission is granted to use a key in the desired way. */ -static inline int key_permission(const key_ref_t key_ref, key_perm_t perm) +static inline int key_permission(const key_ref_t key_ref, unsigned perm) { return key_task_permission(key_ref, current_cred(), perm); } -/* required permissions */ -#define KEY_VIEW 0x01 /* require permission to view attributes */ -#define KEY_READ 0x02 /* require permission to read content */ -#define KEY_WRITE 0x04 /* require permission to update / modify */ -#define KEY_SEARCH 0x08 /* require permission to search (keyring) or find (key) */ -#define KEY_LINK 0x10 /* require permission to link */ -#define KEY_SETATTR 0x20 /* require permission to change attributes */ -#define KEY_ALL 0x3f /* all the above permissions */ - /* * Authorisation record for request_key(). */ diff --git a/security/keys/key.c b/security/keys/key.c index 6e21c11e48bc..2048a110e7f1 100644 --- a/security/keys/key.c +++ b/security/keys/key.c @@ -714,7 +714,7 @@ static inline key_ref_t __key_update(key_ref_t key_ref, int ret; /* need write permission on the key to update it */ - ret = key_permission(key_ref, KEY_WRITE); + ret = key_permission(key_ref, KEY_NEED_WRITE); if (ret < 0) goto error; @@ -838,7 +838,7 @@ key_ref_t key_create_or_update(key_ref_t keyring_ref, /* if we're going to allocate a new key, we're going to have * to modify the keyring */ - ret = key_permission(keyring_ref, KEY_WRITE); + ret = key_permission(keyring_ref, KEY_NEED_WRITE); if (ret < 0) { key_ref = ERR_PTR(ret); goto error_link_end; @@ -928,7 +928,7 @@ int key_update(key_ref_t key_ref, const void *payload, size_t plen) key_check(key); /* the key must be writable */ - ret = key_permission(key_ref, KEY_WRITE); + ret = key_permission(key_ref, KEY_NEED_WRITE); if (ret < 0) goto error; diff --git a/security/keys/keyctl.c b/security/keys/keyctl.c index cee72ce64222..cd5bd0cef25d 100644 --- a/security/keys/keyctl.c +++ b/security/keys/keyctl.c @@ -111,7 +111,7 @@ SYSCALL_DEFINE5(add_key, const char __user *, _type, } /* find the target keyring (which must be writable) */ - keyring_ref = lookup_user_key(ringid, KEY_LOOKUP_CREATE, KEY_WRITE); + keyring_ref = lookup_user_key(ringid, KEY_LOOKUP_CREATE, KEY_NEED_WRITE); if (IS_ERR(keyring_ref)) { ret = PTR_ERR(keyring_ref); goto error3; @@ -195,7 +195,7 @@ SYSCALL_DEFINE4(request_key, const char __user *, _type, dest_ref = NULL; if (destringid) { dest_ref = lookup_user_key(destringid, KEY_LOOKUP_CREATE, - KEY_WRITE); + KEY_NEED_WRITE); if (IS_ERR(dest_ref)) { ret = PTR_ERR(dest_ref); goto error3; @@ -253,7 +253,7 @@ long keyctl_get_keyring_ID(key_serial_t id, int create) long ret; lflags = create ? KEY_LOOKUP_CREATE : 0; - key_ref = lookup_user_key(id, lflags, KEY_SEARCH); + key_ref = lookup_user_key(id, lflags, KEY_NEED_SEARCH); if (IS_ERR(key_ref)) { ret = PTR_ERR(key_ref); goto error; @@ -334,7 +334,7 @@ long keyctl_update_key(key_serial_t id, } /* find the target key (which must be writable) */ - key_ref = lookup_user_key(id, 0, KEY_WRITE); + key_ref = lookup_user_key(id, 0, KEY_NEED_WRITE); if (IS_ERR(key_ref)) { ret = PTR_ERR(key_ref); goto error2; @@ -365,12 +365,12 @@ long keyctl_revoke_key(key_serial_t id) key_ref_t key_ref; long ret; - key_ref = lookup_user_key(id, 0, KEY_WRITE); + key_ref = lookup_user_key(id, 0, KEY_NEED_WRITE); if (IS_ERR(key_ref)) { ret = PTR_ERR(key_ref); if (ret != -EACCES) goto error; - key_ref = lookup_user_key(id, 0, KEY_SETATTR); + key_ref = lookup_user_key(id, 0, KEY_NEED_SETATTR); if (IS_ERR(key_ref)) { ret = PTR_ERR(key_ref); goto error; @@ -401,7 +401,7 @@ long keyctl_invalidate_key(key_serial_t id) kenter("%d", id); - key_ref = lookup_user_key(id, 0, KEY_SEARCH); + key_ref = lookup_user_key(id, 0, KEY_NEED_SEARCH); if (IS_ERR(key_ref)) { ret = PTR_ERR(key_ref); goto error; @@ -428,7 +428,7 @@ long keyctl_keyring_clear(key_serial_t ringid) key_ref_t keyring_ref; long ret; - keyring_ref = lookup_user_key(ringid, KEY_LOOKUP_CREATE, KEY_WRITE); + keyring_ref = lookup_user_key(ringid, KEY_LOOKUP_CREATE, KEY_NEED_WRITE); if (IS_ERR(keyring_ref)) { ret = PTR_ERR(keyring_ref); @@ -470,13 +470,13 @@ long keyctl_keyring_link(key_serial_t id, key_serial_t ringid) key_ref_t keyring_ref, key_ref; long ret; - keyring_ref = lookup_user_key(ringid, KEY_LOOKUP_CREATE, KEY_WRITE); + keyring_ref = lookup_user_key(ringid, KEY_LOOKUP_CREATE, KEY_NEED_WRITE); if (IS_ERR(keyring_ref)) { ret = PTR_ERR(keyring_ref); goto error; } - key_ref = lookup_user_key(id, KEY_LOOKUP_CREATE, KEY_LINK); + key_ref = lookup_user_key(id, KEY_LOOKUP_CREATE, KEY_NEED_LINK); if (IS_ERR(key_ref)) { ret = PTR_ERR(key_ref); goto error2; @@ -505,7 +505,7 @@ long keyctl_keyring_unlink(key_serial_t id, key_serial_t ringid) key_ref_t keyring_ref, key_ref; long ret; - keyring_ref = lookup_user_key(ringid, 0, KEY_WRITE); + keyring_ref = lookup_user_key(ringid, 0, KEY_NEED_WRITE); if (IS_ERR(keyring_ref)) { ret = PTR_ERR(keyring_ref); goto error; @@ -548,7 +548,7 @@ long keyctl_describe_key(key_serial_t keyid, char *tmpbuf; long ret; - key_ref = lookup_user_key(keyid, KEY_LOOKUP_PARTIAL, KEY_VIEW); + key_ref = lookup_user_key(keyid, KEY_LOOKUP_PARTIAL, KEY_NEED_VIEW); if (IS_ERR(key_ref)) { /* viewing a key under construction is permitted if we have the * authorisation token handy */ @@ -639,7 +639,7 @@ long keyctl_keyring_search(key_serial_t ringid, } /* get the keyring at which to begin the search */ - keyring_ref = lookup_user_key(ringid, 0, KEY_SEARCH); + keyring_ref = lookup_user_key(ringid, 0, KEY_NEED_SEARCH); if (IS_ERR(keyring_ref)) { ret = PTR_ERR(keyring_ref); goto error2; @@ -649,7 +649,7 @@ long keyctl_keyring_search(key_serial_t ringid, dest_ref = NULL; if (destringid) { dest_ref = lookup_user_key(destringid, KEY_LOOKUP_CREATE, - KEY_WRITE); + KEY_NEED_WRITE); if (IS_ERR(dest_ref)) { ret = PTR_ERR(dest_ref); goto error3; @@ -676,7 +676,7 @@ long keyctl_keyring_search(key_serial_t ringid, /* link the resulting key to the destination keyring if we can */ if (dest_ref) { - ret = key_permission(key_ref, KEY_LINK); + ret = key_permission(key_ref, KEY_NEED_LINK); if (ret < 0) goto error6; @@ -727,7 +727,7 @@ long keyctl_read_key(key_serial_t keyid, char __user *buffer, size_t buflen) key = key_ref_to_ptr(key_ref); /* see if we can read it directly */ - ret = key_permission(key_ref, KEY_READ); + ret = key_permission(key_ref, KEY_NEED_READ); if (ret == 0) goto can_read_key; if (ret != -EACCES) @@ -799,7 +799,7 @@ long keyctl_chown_key(key_serial_t id, uid_t user, gid_t group) goto error; key_ref = lookup_user_key(id, KEY_LOOKUP_CREATE | KEY_LOOKUP_PARTIAL, - KEY_SETATTR); + KEY_NEED_SETATTR); if (IS_ERR(key_ref)) { ret = PTR_ERR(key_ref); goto error; @@ -905,7 +905,7 @@ long keyctl_setperm_key(key_serial_t id, key_perm_t perm) goto error; key_ref = lookup_user_key(id, KEY_LOOKUP_CREATE | KEY_LOOKUP_PARTIAL, - KEY_SETATTR); + KEY_NEED_SETATTR); if (IS_ERR(key_ref)) { ret = PTR_ERR(key_ref); goto error; @@ -947,7 +947,7 @@ static long get_instantiation_keyring(key_serial_t ringid, /* if a specific keyring is nominated by ID, then use that */ if (ringid > 0) { - dkref = lookup_user_key(ringid, KEY_LOOKUP_CREATE, KEY_WRITE); + dkref = lookup_user_key(ringid, KEY_LOOKUP_CREATE, KEY_NEED_WRITE); if (IS_ERR(dkref)) return PTR_ERR(dkref); *_dest_keyring = key_ref_to_ptr(dkref); @@ -1315,7 +1315,7 @@ long keyctl_set_timeout(key_serial_t id, unsigned timeout) long ret; key_ref = lookup_user_key(id, KEY_LOOKUP_CREATE | KEY_LOOKUP_PARTIAL, - KEY_SETATTR); + KEY_NEED_SETATTR); if (IS_ERR(key_ref)) { /* setting the timeout on a key under construction is permitted * if we have the authorisation token handy */ @@ -1418,7 +1418,7 @@ long keyctl_get_security(key_serial_t keyid, char *context; long ret; - key_ref = lookup_user_key(keyid, KEY_LOOKUP_PARTIAL, KEY_VIEW); + key_ref = lookup_user_key(keyid, KEY_LOOKUP_PARTIAL, KEY_NEED_VIEW); if (IS_ERR(key_ref)) { if (PTR_ERR(key_ref) != -EACCES) return PTR_ERR(key_ref); @@ -1482,7 +1482,7 @@ long keyctl_session_to_parent(void) struct cred *cred; int ret; - keyring_r = lookup_user_key(KEY_SPEC_SESSION_KEYRING, 0, KEY_LINK); + keyring_r = lookup_user_key(KEY_SPEC_SESSION_KEYRING, 0, KEY_NEED_LINK); if (IS_ERR(keyring_r)) return PTR_ERR(keyring_r); diff --git a/security/keys/keyring.c b/security/keys/keyring.c index 2fb2576dc644..9cf2575f0d97 100644 --- a/security/keys/keyring.c +++ b/security/keys/keyring.c @@ -541,7 +541,7 @@ static int keyring_search_iterator(const void *object, void *iterator_data) /* key must have search permissions */ if (!(ctx->flags & KEYRING_SEARCH_NO_CHECK_PERM) && key_task_permission(make_key_ref(key, ctx->possessed), - ctx->cred, KEY_SEARCH) < 0) { + ctx->cred, KEY_NEED_SEARCH) < 0) { ctx->result = ERR_PTR(-EACCES); kleave(" = %d [!perm]", ctx->skipped_ret); goto skipped; @@ -721,7 +721,7 @@ ascend_to_node: /* Search a nested keyring */ if (!(ctx->flags & KEYRING_SEARCH_NO_CHECK_PERM) && key_task_permission(make_key_ref(key, ctx->possessed), - ctx->cred, KEY_SEARCH) < 0) + ctx->cred, KEY_NEED_SEARCH) < 0) continue; /* stack the current position */ @@ -843,7 +843,7 @@ key_ref_t keyring_search_aux(key_ref_t keyring_ref, return ERR_PTR(-ENOTDIR); if (!(ctx->flags & KEYRING_SEARCH_NO_CHECK_PERM)) { - err = key_task_permission(keyring_ref, ctx->cred, KEY_SEARCH); + err = key_task_permission(keyring_ref, ctx->cred, KEY_NEED_SEARCH); if (err < 0) return ERR_PTR(err); } @@ -973,7 +973,7 @@ struct key *find_keyring_by_name(const char *name, bool skip_perm_check) if (!skip_perm_check && key_permission(make_key_ref(keyring, 0), - KEY_SEARCH) < 0) + KEY_NEED_SEARCH) < 0) continue; /* we've got a match but we might end up racing with diff --git a/security/keys/permission.c b/security/keys/permission.c index efcc0c855a0d..732cc0beffdf 100644 --- a/security/keys/permission.c +++ b/security/keys/permission.c @@ -28,7 +28,7 @@ * permissions bits or the LSM check. */ int key_task_permission(const key_ref_t key_ref, const struct cred *cred, - key_perm_t perm) + unsigned perm) { struct key *key; key_perm_t kperm; @@ -68,7 +68,7 @@ use_these_perms: if (is_key_possessed(key_ref)) kperm |= key->perm >> 24; - kperm = kperm & perm & KEY_ALL; + kperm = kperm & perm & KEY_NEED_ALL; if (kperm != perm) return -EACCES; diff --git a/security/keys/persistent.c b/security/keys/persistent.c index 0ad3ee283781..c9fae5ea89fe 100644 --- a/security/keys/persistent.c +++ b/security/keys/persistent.c @@ -108,7 +108,7 @@ static long key_get_persistent(struct user_namespace *ns, kuid_t uid, return PTR_ERR(persistent_ref); found: - ret = key_task_permission(persistent_ref, current_cred(), KEY_LINK); + ret = key_task_permission(persistent_ref, current_cred(), KEY_NEED_LINK); if (ret == 0) { persistent = key_ref_to_ptr(persistent_ref); ret = key_link(key_ref_to_ptr(dest_ref), persistent); @@ -151,7 +151,7 @@ long keyctl_get_persistent(uid_t _uid, key_serial_t destid) } /* There must be a destination keyring */ - dest_ref = lookup_user_key(destid, KEY_LOOKUP_CREATE, KEY_WRITE); + dest_ref = lookup_user_key(destid, KEY_LOOKUP_CREATE, KEY_NEED_WRITE); if (IS_ERR(dest_ref)) return PTR_ERR(dest_ref); if (key_ref_to_ptr(dest_ref)->type != &key_type_keyring) { diff --git a/security/keys/proc.c b/security/keys/proc.c index 88e9a466940f..d3f6f2fd21db 100644 --- a/security/keys/proc.c +++ b/security/keys/proc.c @@ -218,7 +218,7 @@ static int proc_keys_show(struct seq_file *m, void *v) * - the caller holds a spinlock, and thus the RCU read lock, making our * access to __current_cred() safe */ - rc = key_task_permission(key_ref, ctx.cred, KEY_VIEW); + rc = key_task_permission(key_ref, ctx.cred, KEY_NEED_VIEW); if (rc < 0) return 0; diff --git a/security/keys/sysctl.c b/security/keys/sysctl.c index 8c0af08760c8..b68faa1a5cfd 100644 --- a/security/keys/sysctl.c +++ b/security/keys/sysctl.c @@ -15,7 +15,7 @@ static const int zero, one = 1, max = INT_MAX; -ctl_table key_sysctls[] = { +struct ctl_table key_sysctls[] = { { .procname = "maxkeys", .data = &key_quota_maxkeys, diff --git a/security/security.c b/security/security.c index 8b774f362a3d..31614e9e96e5 100644 --- a/security/security.c +++ b/security/security.c @@ -1425,7 +1425,7 @@ void security_key_free(struct key *key) } int security_key_permission(key_ref_t key_ref, - const struct cred *cred, key_perm_t perm) + const struct cred *cred, unsigned perm) { return security_ops->key_permission(key_ref, cred, perm); } diff --git a/security/selinux/avc.c b/security/selinux/avc.c index fc3e6628a864..a18f1fa6440b 100644 --- a/security/selinux/avc.c +++ b/security/selinux/avc.c @@ -444,11 +444,15 @@ static void avc_audit_post_callback(struct audit_buffer *ab, void *a) avc_dump_query(ab, ad->selinux_audit_data->ssid, ad->selinux_audit_data->tsid, ad->selinux_audit_data->tclass); + if (ad->selinux_audit_data->denied) { + audit_log_format(ab, " permissive=%u", + ad->selinux_audit_data->result ? 0 : 1); + } } /* This is the slow part of avc audit with big stack footprint */ noinline int slow_avc_audit(u32 ssid, u32 tsid, u16 tclass, - u32 requested, u32 audited, u32 denied, + u32 requested, u32 audited, u32 denied, int result, struct common_audit_data *a, unsigned flags) { @@ -477,6 +481,7 @@ noinline int slow_avc_audit(u32 ssid, u32 tsid, u16 tclass, sad.tsid = tsid; sad.audited = audited; sad.denied = denied; + sad.result = result; a->selinux_audit_data = &sad; diff --git a/security/selinux/hooks.c b/security/selinux/hooks.c index 2c7341dbc5d6..83d06db34d03 100644 --- a/security/selinux/hooks.c +++ b/security/selinux/hooks.c @@ -2123,11 +2123,13 @@ static int selinux_bprm_set_creds(struct linux_binprm *bprm) new_tsec->exec_sid = 0; /* - * Minimize confusion: if no_new_privs and a transition is - * explicitly requested, then fail the exec. + * Minimize confusion: if no_new_privs or nosuid and a + * transition is explicitly requested, then fail the exec. */ if (bprm->unsafe & LSM_UNSAFE_NO_NEW_PRIVS) return -EPERM; + if (bprm->file->f_path.mnt->mnt_flags & MNT_NOSUID) + return -EACCES; } else { /* Check for a default transition on this program. */ rc = security_transition_sid(old_tsec->sid, isec->sid, @@ -2770,6 +2772,7 @@ static int selinux_inode_follow_link(struct dentry *dentry, struct nameidata *na static noinline int audit_inode_permission(struct inode *inode, u32 perms, u32 audited, u32 denied, + int result, unsigned flags) { struct common_audit_data ad; @@ -2780,7 +2783,7 @@ static noinline int audit_inode_permission(struct inode *inode, ad.u.inode = inode; rc = slow_avc_audit(current_sid(), isec->sid, isec->sclass, perms, - audited, denied, &ad, flags); + audited, denied, result, &ad, flags); if (rc) return rc; return 0; @@ -2822,7 +2825,7 @@ static int selinux_inode_permission(struct inode *inode, int mask) if (likely(!audited)) return rc; - rc2 = audit_inode_permission(inode, perms, audited, denied, flags); + rc2 = audit_inode_permission(inode, perms, audited, denied, rc, flags); if (rc2) return rc2; return rc; @@ -5722,7 +5725,7 @@ static void selinux_key_free(struct key *k) static int selinux_key_permission(key_ref_t key_ref, const struct cred *cred, - key_perm_t perm) + unsigned perm) { struct key *key; struct key_security_struct *ksec; diff --git a/security/selinux/include/avc.h b/security/selinux/include/avc.h index f53ee3c58d0f..ddf8eec03f21 100644 --- a/security/selinux/include/avc.h +++ b/security/selinux/include/avc.h @@ -102,7 +102,7 @@ static inline u32 avc_audit_required(u32 requested, } int slow_avc_audit(u32 ssid, u32 tsid, u16 tclass, - u32 requested, u32 audited, u32 denied, + u32 requested, u32 audited, u32 denied, int result, struct common_audit_data *a, unsigned flags); @@ -137,7 +137,7 @@ static inline int avc_audit(u32 ssid, u32 tsid, if (likely(!audited)) return 0; return slow_avc_audit(ssid, tsid, tclass, - requested, audited, denied, + requested, audited, denied, result, a, 0); } diff --git a/security/selinux/ss/hashtab.c b/security/selinux/ss/hashtab.c index 933e735bb185..2cc496149842 100644 --- a/security/selinux/ss/hashtab.c +++ b/security/selinux/ss/hashtab.c @@ -6,6 +6,7 @@ #include <linux/kernel.h> #include <linux/slab.h> #include <linux/errno.h> +#include <linux/sched.h> #include "hashtab.h" struct hashtab *hashtab_create(u32 (*hash_value)(struct hashtab *h, const void *key), @@ -40,6 +41,8 @@ int hashtab_insert(struct hashtab *h, void *key, void *datum) u32 hvalue; struct hashtab_node *prev, *cur, *newnode; + cond_resched(); + if (!h || h->nel == HASHTAB_MAX_NODES) return -EINVAL; diff --git a/security/selinux/ss/mls.c b/security/selinux/ss/mls.c index c85bc1ec040c..d307b37ddc2b 100644 --- a/security/selinux/ss/mls.c +++ b/security/selinux/ss/mls.c @@ -492,6 +492,8 @@ int mls_convert_context(struct policydb *oldp, rc = ebitmap_set_bit(&bitmap, catdatum->value - 1, 1); if (rc) return rc; + + cond_resched(); } ebitmap_destroy(&c->range.level[l].cat); c->range.level[l].cat = bitmap; diff --git a/security/smack/smack.h b/security/smack/smack.h index d072fd32212d..020307ef0972 100644 --- a/security/smack/smack.h +++ b/security/smack/smack.h @@ -80,8 +80,8 @@ struct superblock_smack { struct socket_smack { struct smack_known *smk_out; /* outbound label */ - char *smk_in; /* inbound label */ - char *smk_packet; /* TCP peer label */ + struct smack_known *smk_in; /* inbound label */ + struct smack_known *smk_packet; /* TCP peer label */ }; /* @@ -133,7 +133,7 @@ struct smk_port_label { struct list_head list; struct sock *smk_sock; /* socket initialized on */ unsigned short smk_port; /* the port number */ - char *smk_in; /* incoming label */ + struct smack_known *smk_in; /* inbound label */ struct smack_known *smk_out; /* outgoing label */ }; @@ -177,6 +177,14 @@ struct smk_port_label { #define SMACK_CIPSO_MAXCATNUM 184 /* 23 * 8 */ /* + * Ptrace rules + */ +#define SMACK_PTRACE_DEFAULT 0 +#define SMACK_PTRACE_EXACT 1 +#define SMACK_PTRACE_DRACONIAN 2 +#define SMACK_PTRACE_MAX SMACK_PTRACE_DRACONIAN + +/* * Flags for untraditional access modes. * It shouldn't be necessary to avoid conflicts with definitions * in fs.h, but do so anyway. @@ -225,6 +233,7 @@ struct inode_smack *new_inode_smack(char *); */ int smk_access_entry(char *, char *, struct list_head *); int smk_access(struct smack_known *, char *, int, struct smk_audit_info *); +int smk_tskacc(struct task_smack *, char *, u32, struct smk_audit_info *); int smk_curacc(char *, u32, struct smk_audit_info *); struct smack_known *smack_from_secid(const u32); char *smk_parse_smack(const char *string, int len); @@ -244,6 +253,7 @@ extern struct smack_known *smack_net_ambient; extern struct smack_known *smack_onlycap; extern struct smack_known *smack_syslog_label; extern const char *smack_cipso_option; +extern int smack_ptrace_rule; extern struct smack_known smack_known_floor; extern struct smack_known smack_known_hat; diff --git a/security/smack/smack_access.c b/security/smack/smack_access.c index 14293cd9b1e5..c062e9467b62 100644 --- a/security/smack/smack_access.c +++ b/security/smack/smack_access.c @@ -192,20 +192,21 @@ out_audit: } /** - * smk_curacc - determine if current has a specific access to an object + * smk_tskacc - determine if a task has a specific access to an object + * @tsp: a pointer to the subject task * @obj_label: a pointer to the object's Smack label * @mode: the access requested, in "MAY" format * @a : common audit data * - * This function checks the current subject label/object label pair + * This function checks the subject task's label/object label pair * in the access rule list and returns 0 if the access is permitted, - * non zero otherwise. It allows that current may have the capability + * non zero otherwise. It allows that the task may have the capability * to override the rules. */ -int smk_curacc(char *obj_label, u32 mode, struct smk_audit_info *a) +int smk_tskacc(struct task_smack *subject, char *obj_label, + u32 mode, struct smk_audit_info *a) { - struct task_smack *tsp = current_security(); - struct smack_known *skp = smk_of_task(tsp); + struct smack_known *skp = smk_of_task(subject); int may; int rc; @@ -219,7 +220,7 @@ int smk_curacc(char *obj_label, u32 mode, struct smk_audit_info *a) * it can further restrict access. */ may = smk_access_entry(skp->smk_known, obj_label, - &tsp->smk_rules); + &subject->smk_rules); if (may < 0) goto out_audit; if ((mode & may) == mode) @@ -241,6 +242,24 @@ out_audit: return rc; } +/** + * smk_curacc - determine if current has a specific access to an object + * @obj_label: a pointer to the object's Smack label + * @mode: the access requested, in "MAY" format + * @a : common audit data + * + * This function checks the current subject label/object label pair + * in the access rule list and returns 0 if the access is permitted, + * non zero otherwise. It allows that current may have the capability + * to override the rules. + */ +int smk_curacc(char *obj_label, u32 mode, struct smk_audit_info *a) +{ + struct task_smack *tsp = current_security(); + + return smk_tskacc(tsp, obj_label, mode, a); +} + #ifdef CONFIG_AUDIT /** * smack_str_from_perm : helper to transalate an int to a @@ -285,7 +304,10 @@ static void smack_log_callback(struct audit_buffer *ab, void *a) audit_log_untrustedstring(ab, sad->subject); audit_log_format(ab, " object="); audit_log_untrustedstring(ab, sad->object); - audit_log_format(ab, " requested=%s", sad->request); + if (sad->request[0] == '\0') + audit_log_format(ab, " labels_differ"); + else + audit_log_format(ab, " requested=%s", sad->request); } /** diff --git a/security/smack/smack_lsm.c b/security/smack/smack_lsm.c index 14f52be78c75..f2c30801ce41 100644 --- a/security/smack/smack_lsm.c +++ b/security/smack/smack_lsm.c @@ -157,6 +157,74 @@ static int smk_copy_rules(struct list_head *nhead, struct list_head *ohead, return rc; } +/** + * smk_ptrace_mode - helper function for converting PTRACE_MODE_* into MAY_* + * @mode - input mode in form of PTRACE_MODE_* + * + * Returns a converted MAY_* mode usable by smack rules + */ +static inline unsigned int smk_ptrace_mode(unsigned int mode) +{ + switch (mode) { + case PTRACE_MODE_READ: + return MAY_READ; + case PTRACE_MODE_ATTACH: + return MAY_READWRITE; + } + + return 0; +} + +/** + * smk_ptrace_rule_check - helper for ptrace access + * @tracer: tracer process + * @tracee_label: label of the process that's about to be traced, + * the pointer must originate from smack structures + * @mode: ptrace attachment mode (PTRACE_MODE_*) + * @func: name of the function that called us, used for audit + * + * Returns 0 on access granted, -error on error + */ +static int smk_ptrace_rule_check(struct task_struct *tracer, char *tracee_label, + unsigned int mode, const char *func) +{ + int rc; + struct smk_audit_info ad, *saip = NULL; + struct task_smack *tsp; + struct smack_known *skp; + + if ((mode & PTRACE_MODE_NOAUDIT) == 0) { + smk_ad_init(&ad, func, LSM_AUDIT_DATA_TASK); + smk_ad_setfield_u_tsk(&ad, tracer); + saip = &ad; + } + + tsp = task_security(tracer); + skp = smk_of_task(tsp); + + if ((mode & PTRACE_MODE_ATTACH) && + (smack_ptrace_rule == SMACK_PTRACE_EXACT || + smack_ptrace_rule == SMACK_PTRACE_DRACONIAN)) { + if (skp->smk_known == tracee_label) + rc = 0; + else if (smack_ptrace_rule == SMACK_PTRACE_DRACONIAN) + rc = -EACCES; + else if (capable(CAP_SYS_PTRACE)) + rc = 0; + else + rc = -EACCES; + + if (saip) + smack_log(skp->smk_known, tracee_label, 0, rc, saip); + + return rc; + } + + /* In case of rule==SMACK_PTRACE_DEFAULT or mode==PTRACE_MODE_READ */ + rc = smk_tskacc(tsp, tracee_label, smk_ptrace_mode(mode), saip); + return rc; +} + /* * LSM hooks. * We he, that is fun! @@ -165,16 +233,15 @@ static int smk_copy_rules(struct list_head *nhead, struct list_head *ohead, /** * smack_ptrace_access_check - Smack approval on PTRACE_ATTACH * @ctp: child task pointer - * @mode: ptrace attachment mode + * @mode: ptrace attachment mode (PTRACE_MODE_*) * * Returns 0 if access is OK, an error code otherwise * - * Do the capability checks, and require read and write. + * Do the capability checks. */ static int smack_ptrace_access_check(struct task_struct *ctp, unsigned int mode) { int rc; - struct smk_audit_info ad; struct smack_known *skp; rc = cap_ptrace_access_check(ctp, mode); @@ -182,10 +249,8 @@ static int smack_ptrace_access_check(struct task_struct *ctp, unsigned int mode) return rc; skp = smk_of_task(task_security(ctp)); - smk_ad_init(&ad, __func__, LSM_AUDIT_DATA_TASK); - smk_ad_setfield_u_tsk(&ad, ctp); - rc = smk_curacc(skp->smk_known, mode, &ad); + rc = smk_ptrace_rule_check(current, skp->smk_known, mode, __func__); return rc; } @@ -195,23 +260,21 @@ static int smack_ptrace_access_check(struct task_struct *ctp, unsigned int mode) * * Returns 0 if access is OK, an error code otherwise * - * Do the capability checks, and require read and write. + * Do the capability checks, and require PTRACE_MODE_ATTACH. */ static int smack_ptrace_traceme(struct task_struct *ptp) { int rc; - struct smk_audit_info ad; struct smack_known *skp; rc = cap_ptrace_traceme(ptp); if (rc != 0) return rc; - skp = smk_of_task(task_security(ptp)); - smk_ad_init(&ad, __func__, LSM_AUDIT_DATA_TASK); - smk_ad_setfield_u_tsk(&ad, ptp); + skp = smk_of_task(current_security()); - rc = smk_curacc(skp->smk_known, MAY_READWRITE, &ad); + rc = smk_ptrace_rule_check(ptp, skp->smk_known, + PTRACE_MODE_ATTACH, __func__); return rc; } @@ -413,9 +476,11 @@ static int smack_sb_kern_mount(struct super_block *sb, int flags, void *data) * Initialize the root inode. */ isp = inode->i_security; - if (inode->i_security == NULL) { - inode->i_security = new_inode_smack(sp->smk_root); - isp = inode->i_security; + if (isp == NULL) { + isp = new_inode_smack(sp->smk_root); + if (isp == NULL) + return -ENOMEM; + inode->i_security = isp; } else isp->smk_inode = sp->smk_root; @@ -453,7 +518,7 @@ static int smack_sb_statfs(struct dentry *dentry) * smack_bprm_set_creds - set creds for exec * @bprm: the exec information * - * Returns 0 if it gets a blob, -ENOMEM otherwise + * Returns 0 if it gets a blob, -EPERM if exec forbidden and -ENOMEM otherwise */ static int smack_bprm_set_creds(struct linux_binprm *bprm) { @@ -473,7 +538,22 @@ static int smack_bprm_set_creds(struct linux_binprm *bprm) if (isp->smk_task == NULL || isp->smk_task == bsp->smk_task) return 0; - if (bprm->unsafe) + if (bprm->unsafe & (LSM_UNSAFE_PTRACE | LSM_UNSAFE_PTRACE_CAP)) { + struct task_struct *tracer; + rc = 0; + + rcu_read_lock(); + tracer = ptrace_parent(current); + if (likely(tracer != NULL)) + rc = smk_ptrace_rule_check(tracer, + isp->smk_task->smk_known, + PTRACE_MODE_ATTACH, + __func__); + rcu_read_unlock(); + + if (rc != 0) + return rc; + } else if (bprm->unsafe) return -EPERM; bsp->smk_task = isp->smk_task; @@ -880,18 +960,20 @@ static void smack_inode_post_setxattr(struct dentry *dentry, const char *name, return; } - skp = smk_import_entry(value, size); if (strcmp(name, XATTR_NAME_SMACK) == 0) { + skp = smk_import_entry(value, size); if (skp != NULL) isp->smk_inode = skp->smk_known; else isp->smk_inode = smack_known_invalid.smk_known; } else if (strcmp(name, XATTR_NAME_SMACKEXEC) == 0) { + skp = smk_import_entry(value, size); if (skp != NULL) isp->smk_task = skp; else isp->smk_task = &smack_known_invalid; } else if (strcmp(name, XATTR_NAME_SMACKMMAP) == 0) { + skp = smk_import_entry(value, size); if (skp != NULL) isp->smk_mmap = skp; else @@ -938,24 +1020,37 @@ static int smack_inode_removexattr(struct dentry *dentry, const char *name) strcmp(name, XATTR_NAME_SMACKIPOUT) == 0 || strcmp(name, XATTR_NAME_SMACKEXEC) == 0 || strcmp(name, XATTR_NAME_SMACKTRANSMUTE) == 0 || - strcmp(name, XATTR_NAME_SMACKMMAP)) { + strcmp(name, XATTR_NAME_SMACKMMAP) == 0) { if (!smack_privileged(CAP_MAC_ADMIN)) rc = -EPERM; } else rc = cap_inode_removexattr(dentry, name); + if (rc != 0) + return rc; + smk_ad_init(&ad, __func__, LSM_AUDIT_DATA_DENTRY); smk_ad_setfield_u_fs_path_dentry(&ad, dentry); - if (rc == 0) - rc = smk_curacc(smk_of_inode(dentry->d_inode), MAY_WRITE, &ad); - if (rc == 0) { - isp = dentry->d_inode->i_security; + rc = smk_curacc(smk_of_inode(dentry->d_inode), MAY_WRITE, &ad); + if (rc != 0) + return rc; + + isp = dentry->d_inode->i_security; + /* + * Don't do anything special for these. + * XATTR_NAME_SMACKIPIN + * XATTR_NAME_SMACKIPOUT + * XATTR_NAME_SMACKEXEC + */ + if (strcmp(name, XATTR_NAME_SMACK) == 0) isp->smk_task = NULL; + else if (strcmp(name, XATTR_NAME_SMACKMMAP) == 0) isp->smk_mmap = NULL; - } + else if (strcmp(name, XATTR_NAME_SMACKTRANSMUTE) == 0) + isp->smk_flags &= ~SMK_INODE_TRANSMUTE; - return rc; + return 0; } /** @@ -1000,7 +1095,7 @@ static int smack_inode_getsecurity(const struct inode *inode, ssp = sock->sk->sk_security; if (strcmp(name, XATTR_SMACK_IPIN) == 0) - isp = ssp->smk_in; + isp = ssp->smk_in->smk_known; else if (strcmp(name, XATTR_SMACK_IPOUT) == 0) isp = ssp->smk_out->smk_known; else @@ -1367,19 +1462,32 @@ static int smack_file_receive(struct file *file) /** * smack_file_open - Smack dentry open processing * @file: the object - * @cred: unused + * @cred: task credential * * Set the security blob in the file structure. + * Allow the open only if the task has read access. There are + * many read operations (e.g. fstat) that you can do with an + * fd even if you have the file open write-only. * * Returns 0 */ static int smack_file_open(struct file *file, const struct cred *cred) { + struct task_smack *tsp = cred->security; struct inode_smack *isp = file_inode(file)->i_security; + struct smk_audit_info ad; + int rc; - file->f_security = isp->smk_inode; + if (smack_privileged(CAP_MAC_OVERRIDE)) + return 0; - return 0; + smk_ad_init(&ad, __func__, LSM_AUDIT_DATA_PATH); + smk_ad_setfield_u_fs_path(&ad, file->f_path); + rc = smk_access(tsp->smk_task, isp->smk_inode, MAY_READ, &ad); + if (rc == 0) + file->f_security = isp->smk_inode; + + return rc; } /* @@ -1764,7 +1872,7 @@ static int smack_sk_alloc_security(struct sock *sk, int family, gfp_t gfp_flags) if (ssp == NULL) return -ENOMEM; - ssp->smk_in = skp->smk_known; + ssp->smk_in = skp; ssp->smk_out = skp; ssp->smk_packet = NULL; @@ -2004,7 +2112,7 @@ static int smk_ipv6_port_check(struct sock *sk, struct sockaddr_in6 *address, if (act == SMK_RECEIVING) { skp = smack_net_ambient; - object = ssp->smk_in; + object = ssp->smk_in->smk_known; } else { skp = ssp->smk_out; object = smack_net_ambient->smk_known; @@ -2034,9 +2142,9 @@ static int smk_ipv6_port_check(struct sock *sk, struct sockaddr_in6 *address, list_for_each_entry(spp, &smk_ipv6_port_list, list) { if (spp->smk_port != port) continue; - object = spp->smk_in; + object = spp->smk_in->smk_known; if (act == SMK_CONNECTING) - ssp->smk_packet = spp->smk_out->smk_known; + ssp->smk_packet = spp->smk_out; break; } @@ -2076,7 +2184,7 @@ static int smack_inode_setsecurity(struct inode *inode, const char *name, int rc = 0; if (value == NULL || size > SMK_LONGLABEL || size == 0) - return -EACCES; + return -EINVAL; skp = smk_import_entry(value, size); if (skp == NULL) @@ -2100,7 +2208,7 @@ static int smack_inode_setsecurity(struct inode *inode, const char *name, ssp = sock->sk->sk_security; if (strcmp(name, XATTR_SMACK_IPIN) == 0) - ssp->smk_in = skp->smk_known; + ssp->smk_in = skp; else if (strcmp(name, XATTR_SMACK_IPOUT) == 0) { ssp->smk_out = skp; if (sock->sk->sk_family == PF_INET) { @@ -2713,6 +2821,15 @@ static void smack_d_instantiate(struct dentry *opt_dentry, struct inode *inode) * of the superblock. */ if (opt_dentry->d_parent == opt_dentry) { + if (sbp->s_magic == CGROUP_SUPER_MAGIC) { + /* + * The cgroup filesystem is never mounted, + * so there's no opportunity to set the mount + * options. + */ + sbsp->smk_root = smack_known_star.smk_known; + sbsp->smk_default = smack_known_star.smk_known; + } isp->smk_inode = sbsp->smk_root; isp->smk_flags |= SMK_INODE_INSTANT; goto unlockandout; @@ -2726,16 +2843,20 @@ static void smack_d_instantiate(struct dentry *opt_dentry, struct inode *inode) */ switch (sbp->s_magic) { case SMACK_MAGIC: + case PIPEFS_MAGIC: + case SOCKFS_MAGIC: + case CGROUP_SUPER_MAGIC: /* * Casey says that it's a little embarrassing * that the smack file system doesn't do * extended attributes. - */ - final = smack_known_star.smk_known; - break; - case PIPEFS_MAGIC: - /* + * * Casey says pipes are easy (?) + * + * Socket access is controlled by the socket + * structures associated with the task involved. + * + * Cgroupfs is special */ final = smack_known_star.smk_known; break; @@ -2747,13 +2868,6 @@ static void smack_d_instantiate(struct dentry *opt_dentry, struct inode *inode) */ final = ckp->smk_known; break; - case SOCKFS_MAGIC: - /* - * Socket access is controlled by the socket - * structures associated with the task involved. - */ - final = smack_known_star.smk_known; - break; case PROC_SUPER_MAGIC: /* * Casey says procfs appears not to care. @@ -2959,30 +3073,34 @@ static int smack_unix_stream_connect(struct sock *sock, struct sock *other, struct sock *newsk) { struct smack_known *skp; + struct smack_known *okp; struct socket_smack *ssp = sock->sk_security; struct socket_smack *osp = other->sk_security; struct socket_smack *nsp = newsk->sk_security; struct smk_audit_info ad; int rc = 0; - #ifdef CONFIG_AUDIT struct lsm_network_audit net; - - smk_ad_init_net(&ad, __func__, LSM_AUDIT_DATA_NET, &net); - smk_ad_setfield_u_net_sk(&ad, other); #endif if (!smack_privileged(CAP_MAC_OVERRIDE)) { skp = ssp->smk_out; - rc = smk_access(skp, osp->smk_in, MAY_WRITE, &ad); + okp = osp->smk_out; +#ifdef CONFIG_AUDIT + smk_ad_init_net(&ad, __func__, LSM_AUDIT_DATA_NET, &net); + smk_ad_setfield_u_net_sk(&ad, other); +#endif + rc = smk_access(skp, okp->smk_known, MAY_WRITE, &ad); + if (rc == 0) + rc = smk_access(okp, okp->smk_known, MAY_WRITE, NULL); } /* * Cross reference the peer labels for SO_PEERSEC. */ if (rc == 0) { - nsp->smk_packet = ssp->smk_out->smk_known; - ssp->smk_packet = osp->smk_out->smk_known; + nsp->smk_packet = ssp->smk_out; + ssp->smk_packet = osp->smk_out; } return rc; @@ -3014,7 +3132,7 @@ static int smack_unix_may_send(struct socket *sock, struct socket *other) return 0; skp = ssp->smk_out; - return smk_access(skp, osp->smk_in, MAY_WRITE, &ad); + return smk_access(skp, osp->smk_in->smk_known, MAY_WRITE, &ad); } /** @@ -3109,7 +3227,7 @@ static struct smack_known *smack_from_secattr(struct netlbl_lsm_secattr *sap, if (found) return skp; - if (ssp != NULL && ssp->smk_in == smack_known_star.smk_known) + if (ssp != NULL && ssp->smk_in == &smack_known_star) return &smack_known_web; return &smack_known_star; } @@ -3228,7 +3346,7 @@ static int smack_socket_sock_rcv_skb(struct sock *sk, struct sk_buff *skb) * This is the simplist possible security model * for networking. */ - rc = smk_access(skp, ssp->smk_in, MAY_WRITE, &ad); + rc = smk_access(skp, ssp->smk_in->smk_known, MAY_WRITE, &ad); if (rc != 0) netlbl_skbuff_err(skb, rc, 0); break; @@ -3263,7 +3381,7 @@ static int smack_socket_getpeersec_stream(struct socket *sock, ssp = sock->sk->sk_security; if (ssp->smk_packet != NULL) { - rcp = ssp->smk_packet; + rcp = ssp->smk_packet->smk_known; slen = strlen(rcp) + 1; } @@ -3348,7 +3466,7 @@ static void smack_sock_graft(struct sock *sk, struct socket *parent) return; ssp = sk->sk_security; - ssp->smk_in = skp->smk_known; + ssp->smk_in = skp; ssp->smk_out = skp; /* cssp->smk_packet is already set in smack_inet_csk_clone() */ } @@ -3408,7 +3526,7 @@ static int smack_inet_conn_request(struct sock *sk, struct sk_buff *skb, * Receiving a packet requires that the other end be able to write * here. Read access is not required. */ - rc = smk_access(skp, ssp->smk_in, MAY_WRITE, &ad); + rc = smk_access(skp, ssp->smk_in->smk_known, MAY_WRITE, &ad); if (rc != 0) return rc; @@ -3452,7 +3570,7 @@ static void smack_inet_csk_clone(struct sock *sk, if (req->peer_secid != 0) { skp = smack_from_secid(req->peer_secid); - ssp->smk_packet = skp->smk_known; + ssp->smk_packet = skp; } else ssp->smk_packet = NULL; } @@ -3506,11 +3624,12 @@ static void smack_key_free(struct key *key) * an error code otherwise */ static int smack_key_permission(key_ref_t key_ref, - const struct cred *cred, key_perm_t perm) + const struct cred *cred, unsigned perm) { struct key *keyp; struct smk_audit_info ad; struct smack_known *tkp = smk_of_task(cred->security); + int request = 0; keyp = key_ref_to_ptr(key_ref); if (keyp == NULL) @@ -3531,7 +3650,11 @@ static int smack_key_permission(key_ref_t key_ref, ad.a.u.key_struct.key = keyp->serial; ad.a.u.key_struct.key_desc = keyp->description; #endif - return smk_access(tkp, keyp->security, MAY_READWRITE, &ad); + if (perm & KEY_NEED_READ) + request = MAY_READ; + if (perm & (KEY_NEED_WRITE | KEY_NEED_LINK | KEY_NEED_SETATTR)) + request = MAY_WRITE; + return smk_access(tkp, keyp->security, request, &ad); } #endif /* CONFIG_KEYS */ diff --git a/security/smack/smackfs.c b/security/smack/smackfs.c index 3198cfe1dcc6..32b248820840 100644 --- a/security/smack/smackfs.c +++ b/security/smack/smackfs.c @@ -53,6 +53,7 @@ enum smk_inos { SMK_REVOKE_SUBJ = 18, /* set rules with subject label to '-' */ SMK_CHANGE_RULE = 19, /* change or add rules (long labels) */ SMK_SYSLOG = 20, /* change syslog label) */ + SMK_PTRACE = 21, /* set ptrace rule */ }; /* @@ -101,6 +102,15 @@ struct smack_known *smack_onlycap; struct smack_known *smack_syslog_label; /* + * Ptrace current rule + * SMACK_PTRACE_DEFAULT regular smack ptrace rules (/proc based) + * SMACK_PTRACE_EXACT labels must match, but can be overriden with + * CAP_SYS_PTRACE + * SMACK_PTRACE_DRACONIAN lables must match, CAP_SYS_PTRACE has no effect + */ +int smack_ptrace_rule = SMACK_PTRACE_DEFAULT; + +/* * Certain IP addresses may be designated as single label hosts. * Packets are sent there unlabeled, but only from tasks that * can write to the specified label. @@ -1183,7 +1193,7 @@ static ssize_t smk_write_netlbladdr(struct file *file, const char __user *buf, data[count] = '\0'; - rc = sscanf(data, "%hhd.%hhd.%hhd.%hhd/%d %s", + rc = sscanf(data, "%hhd.%hhd.%hhd.%hhd/%u %s", &host[0], &host[1], &host[2], &host[3], &m, smack); if (rc != 6) { rc = sscanf(data, "%hhd.%hhd.%hhd.%hhd %s", @@ -2244,6 +2254,68 @@ static const struct file_operations smk_syslog_ops = { /** + * smk_read_ptrace - read() for /smack/ptrace + * @filp: file pointer, not actually used + * @buf: where to put the result + * @count: maximum to send along + * @ppos: where to start + * + * Returns number of bytes read or error code, as appropriate + */ +static ssize_t smk_read_ptrace(struct file *filp, char __user *buf, + size_t count, loff_t *ppos) +{ + char temp[32]; + ssize_t rc; + + if (*ppos != 0) + return 0; + + sprintf(temp, "%d\n", smack_ptrace_rule); + rc = simple_read_from_buffer(buf, count, ppos, temp, strlen(temp)); + return rc; +} + +/** + * smk_write_ptrace - write() for /smack/ptrace + * @file: file pointer + * @buf: data from user space + * @count: bytes sent + * @ppos: where to start - must be 0 + */ +static ssize_t smk_write_ptrace(struct file *file, const char __user *buf, + size_t count, loff_t *ppos) +{ + char temp[32]; + int i; + + if (!smack_privileged(CAP_MAC_ADMIN)) + return -EPERM; + + if (*ppos != 0 || count >= sizeof(temp) || count == 0) + return -EINVAL; + + if (copy_from_user(temp, buf, count) != 0) + return -EFAULT; + + temp[count] = '\0'; + + if (sscanf(temp, "%d", &i) != 1) + return -EINVAL; + if (i < SMACK_PTRACE_DEFAULT || i > SMACK_PTRACE_MAX) + return -EINVAL; + smack_ptrace_rule = i; + + return count; +} + +static const struct file_operations smk_ptrace_ops = { + .write = smk_write_ptrace, + .read = smk_read_ptrace, + .llseek = default_llseek, +}; + +/** * smk_fill_super - fill the smackfs superblock * @sb: the empty superblock * @data: unused @@ -2296,6 +2368,8 @@ static int smk_fill_super(struct super_block *sb, void *data, int silent) "change-rule", &smk_change_rule_ops, S_IRUGO|S_IWUSR}, [SMK_SYSLOG] = { "syslog", &smk_syslog_ops, S_IRUGO|S_IWUSR}, + [SMK_PTRACE] = { + "ptrace", &smk_ptrace_ops, S_IRUGO|S_IWUSR}, /* last one */ {""} }; diff --git a/tools/testing/selftests/powerpc/Makefile b/tools/testing/selftests/powerpc/Makefile index 316194f26ff4..b3dbe9ef1a40 100644 --- a/tools/testing/selftests/powerpc/Makefile +++ b/tools/testing/selftests/powerpc/Makefile @@ -13,7 +13,7 @@ CFLAGS := -Wall -O2 -flto -Wall -Werror -DGIT_VERSION='"$(GIT_VERSION)"' -I$(CUR export CC CFLAGS -TARGETS = pmu copyloops +TARGETS = pmu copyloops mm endif diff --git a/tools/testing/selftests/powerpc/copyloops/asm/ppc_asm.h b/tools/testing/selftests/powerpc/copyloops/asm/ppc_asm.h index ccd9c84c4e3f..d1dc37425510 100644 --- a/tools/testing/selftests/powerpc/copyloops/asm/ppc_asm.h +++ b/tools/testing/selftests/powerpc/copyloops/asm/ppc_asm.h @@ -46,12 +46,15 @@ #define R20 r20 #define R21 r21 #define R22 r22 +#define R29 r29 +#define R30 r30 +#define R31 r31 #define STACKFRAMESIZE 256 -#define STK_PARAM(i) (48 + ((i)-3)*8) #define STK_REG(i) (112 + ((i)-14)*8) #define _GLOBAL(A) FUNC_START(test_ ## A) +#define _GLOBAL_TOC(A) _GLOBAL(A) #define PPC_MTOCRF(A, B) mtocrf A, B diff --git a/tools/testing/selftests/powerpc/mm/Makefile b/tools/testing/selftests/powerpc/mm/Makefile new file mode 100644 index 000000000000..357ccbd6bad9 --- /dev/null +++ b/tools/testing/selftests/powerpc/mm/Makefile @@ -0,0 +1,18 @@ +noarg: + $(MAKE) -C ../ + +PROGS := hugetlb_vs_thp_test + +all: $(PROGS) + +$(PROGS): ../harness.c + +run_tests: all + @-for PROG in $(PROGS); do \ + ./$$PROG; \ + done; + +clean: + rm -f $(PROGS) + +.PHONY: all run_tests clean diff --git a/tools/testing/selftests/powerpc/mm/hugetlb_vs_thp_test.c b/tools/testing/selftests/powerpc/mm/hugetlb_vs_thp_test.c new file mode 100644 index 000000000000..3d8e5b033e1d --- /dev/null +++ b/tools/testing/selftests/powerpc/mm/hugetlb_vs_thp_test.c @@ -0,0 +1,72 @@ +#include <stdio.h> +#include <sys/mman.h> +#include <unistd.h> + +#include "utils.h" + +/* This must match the huge page & THP size */ +#define SIZE (16 * 1024 * 1024) + +static int test_body(void) +{ + void *addr; + char *p; + + addr = (void *)0xa0000000; + + p = mmap(addr, SIZE, PROT_READ | PROT_WRITE, + MAP_HUGETLB | MAP_ANONYMOUS | MAP_PRIVATE, -1, 0); + if (p != MAP_FAILED) { + /* + * Typically the mmap will fail because no huge pages are + * allocated on the system. But if there are huge pages + * allocated the mmap will succeed. That's fine too, we just + * munmap here before continuing. + */ + munmap(addr, SIZE); + } + + p = mmap(addr, SIZE, PROT_READ | PROT_WRITE, + MAP_ANONYMOUS | MAP_PRIVATE, -1, 0); + if (p == MAP_FAILED) { + printf("Mapping failed @ %p\n", addr); + perror("mmap"); + return 1; + } + + /* + * Either a user or kernel access is sufficient to trigger the bug. + * A kernel access is easier to spot & debug, as it will trigger the + * softlockup or RCU stall detectors, and when the system is kicked + * into xmon we get a backtrace in the kernel. + * + * A good option is: + * getcwd(p, SIZE); + * + * For the purposes of this testcase it's preferable to spin in + * userspace, so the harness can kill us if we get stuck. That way we + * see a test failure rather than a dead system. + */ + *p = 0xf; + + munmap(addr, SIZE); + + return 0; +} + +static int test_main(void) +{ + int i; + + /* 10,000 because it's a "bunch", and completes reasonably quickly */ + for (i = 0; i < 10000; i++) + if (test_body()) + return 1; + + return 0; +} + +int main(void) +{ + return test_harness(test_main, "hugetlb_vs_thp"); +} |